Abstract
This paper reviews a number of recent contributions that demonstrate that a blend of welfare economics and statistical analysis is useful in the evaluation of the citations received by scientific papers in the periodical literature. The paper begins by clarifying the role of citation analysis in the evaluation of research. Next, a summary of results about the citation distributions’ basic features at different aggregation levels is offered. These results indicate that citation distributions share the same broad shape, are highly skewed, and are often crowned by a power law. In light of this evidence, a novel methodology for the evaluation of research units is illustrated by comparing the high- and low-citation impact achieved by the US, the European Union, and the rest of the world in 22 scientific fields. However, contrary to recent claims, it is shown that mean normalization at the sub-field level does not lead to a universal distribution. Nevertheless, among other topics subject to ongoing research, it appears that this lack of universality does not preclude sensible normalization procedures to compare the citation impact of articles in different scientific fields.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Albarrán P, Ruiz-Castillo J (2011) References made and citations received by scientific articles. J Am Soc Inf Sci Technol 62: 40–49
Albarrán P, Crespo J, Ortuño I, Ruiz-Castillo J (2010) A comparison of the scientific performance of the US and Europe at the turn of the 21st century. Scientometrics 85: 329–344
Albarrán P, Ortuño I, Ruiz-Castillo J (2011) The measurement of low- and high-impact in citation distributions: technical results. J Informetr 5: 48–63
Albarrán P, Ortuño I, Ruiz-Castillo J (2011) High- and low-impact citation measures: empirical applications. J Informetr 5: 122–145
Albarrán P, Ortuño I, Ruiz-Castillo J (2011c) Average-based versus high- and low-impact indicators for the evaluation of citation distributions with. In: Research evaluation (forthcoming)
Albarrán P, Crespo J, Ortuño I, Ruiz-Castillo J (2011) The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics 88: 385–397
Alonso S, Cabrerizo FJ, Herrera-Viedma E, Herrera F (2009) h-Index: a review focused in its variants, computation and standardization for different scientific fields. J Informetr 3: 273–289
Bornmann L, Daniel H-D (2008) What do citation counts measure?. J Doc 64: 45–80
Clauset A, Shalizi CR, Newman MEJ (2007) Power-law distributions In empirical data. SIAM Rev 51: 661–703
Cole JR (2000) A short history of the use of citations as a measure of the impact of scientific and scholarly work. In: Cronin B, Atkins HB (eds) The web of knowledge: a festschrisft in honor of Eugene Garfield. Information Today, Medford
Delanghe H, Sloan B, Muldur U (2011) European research policy and bibliometric indicators, 1990–2005. Scientometrics 87: 389–398
Dorogovstev S, Mendes J (2001) Scaling properties of scale-free evolving networks: continuous approach. Phys Rev E 85: 4633–4636
Dosi G, Llerena P, Sylos Labini M (2006) Science-Technology-Industry links and the ‘European Paradox’: some notes on the dynamics of scientific and technological research in Europe. Res Policy 35: 1450–1464
EC (1994) First European Report on Science and Technology Indicators, Directorate-General XII, Science, Research, and Development. Luxembourg: Office for Official Publications of the European Community
EC (2003) Third European Report on Science and Technology Indicators, Directorate-General for Research. Luxembourg: Office for Official Publications of the European Community, http://www.cordis.lu/rtd2002/indicators/home.html
Egghe L (2005) Power laws in the information production process: Lotkaian informetrics. Elsevier, Amsterdam
Foster JE, Greeer J, Thorbecke E (1984) A class of decomposable poverty measures. Econometrica 52: 761–766
Foster JE, Shorrocks A (1991) Subgroup consistent poverty indices. Econometrica 59: 687–709
Glänzel W (2010) The application of characteristics scores and scales to the evaluation and ranking of scientific journals. In: Proceedings of INFO 2010, Havana, Cuba, pp 1–13 (forthcoming)
Glänzel W, Schubert A (2003) A new classification scheme of science fields and subfields designed for scientometric evaluation purposes. Scientometrics 56: 357–367
Hirsch J (2005) An index to quantify an individual’s scientific research output. Proc natl Acad Sci USA 102: 16569–16572
Jackson M, Rogers B (2007) Meeting strangers and friends of friends: how random are social networks?. Am Econ Rev 97: 890–915
Jenkins S, Lambert P (1997) Three ‘I’s of poverty curves, with an analysis of UK poverty trends. Oxf Econ Pap 49: 317–327
King D (2004) The scientific impact of nations. Nature 430: 311–316
Lehmann S, Lautrup B, Jackson AD (2003) Citation networks in high energy physics. Phys Rev E68: 026113–026118
Leydesdorff L, Wagner C (2009) Is the US losing ground in science? a global perspective on the world science system. Scientometrics 78: 23–36
Marchant T (2009) An axiomatic characterization of the ranking based on the h-index and some other bibliometric rankings of authors. Scientometrics 80: 325–342
Mitzenmacher M (2004) A brief history of generative models for power law and lognormal distributions. Internet Math 1: 226–251
Moed HF, Burger WJ, Frankfort JG, van Raan AFJ (1985) The use of bibliometric data for the measurement of university research performance. Res Policy 14: 131–149
Moed HF, van Raan AFJ (1988) Indicators of research performance. In: Raan AFJ (eds) Handbook of quantitative studies of science and technology. North Holland, Amsterdam, pp 177–192
Moed HF, De Bruin RE, van Leeuwen ThN (1995) New bibliometrics tools for the assessment of national research performance: database description, overview of indicators, and first applications. Scientometrics 33: 381–422
Molinari JF, Molinari A (2008) A new methodology for ranking scientific institutions. Scientometrics 75: 163–174
Molinari A, Molinari J-F (2008) Mathematical aspects of a new criterion for ranking scientific institutions based on the h-index. Scientometrics 75: 339–356
Newman MEJ (2005) Power laws, Pareto distributions, and Zipf’s law. Contemp Phys 46: 323–351
Peterson G, Presse S, Dill K (2010) Nonuniversal power law scaling in the probability distribution of scientific citations. PNAS 107: 16023–16027
Quesada A (2009) Monotonicity and the Hirsch index. J Informetr 3: 158–160
Quesada A (2010) More axiomatics for the Hirsch index. Scientometrics 82: 413–418
Radicchi F, Fortunato S, Castellano C (2008) Universality of citation distributions: toward an objective measure of scientific impact. PNAS 105: 17268–17272
Ruiz-Castillo J (2011) The evaluation of citation distributions. Working Paper 11–12, Economics Department, Universidad Carlos III
Schubert A, Glänzel W, Braun T (1987) Subject field characteristic citation scores and scales for assessing research performance. Scientometrics 12: 267–292
Sen A (1976) Poverty: an ordinal approach to measurement. Econometrica 44: 219–230
Tijssen R, Visser M, van Leeuwen T (2002) Benchmarking international scientific excellence: are highly cited research papers an appropriate frame of reference. Scientometrics 54: 381–397
Tijssen RJW, van Leeuwen TN (2003) Bibliometric analyses of world science, extended technical annex to Chapter 5 of the third European Report on science and technology indicators. mimeo, Leiden University
van Leeuwen T, Moed H, Tijssen R, Visser M, van Raan A (2001) Language biases in the coverage of the science citation index and its consequences for international comparisons of national research performance. Scientometrics 51: 335–346
van Leeuwen T, Visser M, Moed H, Nederhof T, van Raan A (2003) The holy grail of science policy: exploring and combining bibliometric tools in search of scientific excellence. Scientometrics 57: 257–280
van Raan AFJ (2004) Measuring Science. In: Moed HF, Glänzel W, Schmoch U (eds) Handbook of quantitative science and technology research. The use of publication and patent statistics in studies of S&T systems, Kluwer, Dordrecht
van Raan AFJ (2005) Fatal attraction: conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics 62: 133–143
Waltman L, van Eck NJ, van Raan AFJ (2011) Universality of citation distributions revisited. Center for science and technological studies, Leiden University, The Netherlands, mimeo, http://arxiv.org/pdf/1105.2934v1
Weingart P (2005) Impact of bibliometrics upon the science system: inadvertent consequences?. Scientometrics 62: 117–131
Woeginger G (2008) An axiomatic characterization of the Hirsch-index. Math Soc Sci 56: 224–232
Woeginger G (2008) A symmetry axiom for scientific impact indices. J Informetr 2: 298–303
Author information
Authors and Affiliations
Corresponding author
Additional information
The author acknowledges financial support from Santander Universities Global Division of Banco Santander, as well as from the Spanish MEC through grant SEJ2007-67436. This paper is part of the SCIFI-GLOW Collaborative Project supported by the European Commission’s Seventh Research Framework Programme, Contract no. SSH7-CT-2008-217436. This paper is the result of the author’s joint work with Pedro Albarrán, Juan A. Crespo, Neus Herranz, and Ignacio Ortuño, whose comments and suggestions are gratefully acknowledged. Comments by Jaime Luque, Jaume Sempere, and Matthew Jackson are also greatly appreciated.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Cite this article
Ruiz-Castillo, J. The evaluation of citation distributions. SERIEs 3, 291–310 (2012). https://doi.org/10.1007/s13209-011-0074-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13209-011-0074-3