Abstract
Background
Besides the number of publications, the number of citations is another key metric often used to compare researchers with each other. While women researchers tend to have fewer publications than their men colleagues, the data is scarce for the number of citations. We aimed to determine whether there is a gender gap in citations.
Methods
We used Web of Science to retrieve the number of citations per year for all research articles and reviews published between January 2015 and December 2019 in fourteen high-impact general medical journals (impact factor > 5). We used Gender API to identify the gender of the first/last authors. We compared the results by gender using multivariable negative binomial regressions (adjusting for intra-cluster correlations within journals).
Results
The gender of the first/last author was determined for 13,218/13,350 (99%) and 11,894/12,026 (99%) articles, respectively. The proportion of women among first/last authors was 40% and 29%, respectively. The median number of citations per year was 5 (IQR = 11.3) for women and 6.8 (IQR = 17.8) for men for first authors (IRR = 1.5 [95% CI = 1.3–1.8], p value < 0.001), and 6 (IQR = 12.4) and 7.5 (IQR = 17.4) for last authors (IRR = 1.3 [95% CI = 1.2–1.5], p value < 0.001). Articles whose first and last authors were women were the least cited and those whose first and last authors were men were the most cited.
Conclusion
In this cross-sectional study, we found that articles authored by women were cited less often than those authored by men. Further studies are needed to explore the reasons for these gender differences in article citations.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
INTRODUCTION
The difficulties that women face in academic medicine because of their gender alone are still a reality, including in general internal medicine. While women represent nearly half of the physicians in Organisation for Economic Co-operation and Development (OECD) countries,1 they remain a minority in senior academic positions.2,3,4
Sociocultural factors likely play an important role in influencing gender differences in work-family balance. Compared to their men counterparts, women researchers generally spend less time at work and more at home, which may be related to personal decisions, but may also reflect persistent cultural norms and stereotypes.5,6,7,8 Gender-based discrimination may not be as prevalent in academic medicine as it was a few decades ago, but some forms of sexism, which may be conscious or unconscious, are certainly still present and can greatly influence women’s academic trajectories.7,9,10,11 These stereotypes can clearly discourage talented women from pursuing their academic careers.
Several studies showed that women were underrepresented as authors of scientific articles,4,12,13,14 but few articles examined whether there were gender differences in the number of article citations. However, in addition to the number of publications, the h-index, which includes the number of citations in its calculation and is defined as the number of publications h that were each cited at least h times,15 is a quantitative measure that is often used to compare researchers with each other. It takes into account both the productivity of researchers (i.e., the number of their publications) and the impact of their research estimated by the number of their citations.16,17 Initiatives such as the DORA declaration (https://sfdora.org/resource/halt-the-h-index/) criticize the use of the h-index for research assessment because it does not account for qualitative indicators of success, such as teaching, research quality, and collaborations. It is, however, often used as the metric of choice in academia for evaluating researchers for job offers, tenure, promotion, grants, and membership in learned societies.17,18,19,20,21 Therefore, both the number of publications and the number of citations (through the h-index) could have a notable impact on the career progression of researchers. A possible gender gap in citations could contribute to gender inequalities in senior academic positions.
In a recent study conducted in Switzerland that assessed the productivity of all hospital-based researchers in general internal medicine (n = 367), we found that articles whose first author was a woman were on average less frequently cited than articles whose first author was a man (median number of citations: 1 vs 6).4 However, this difference was not statistically significant after controlling for various co-factors.
Another recent study that analyzed 5500 publications in general medical journals showed similar results, with articles whose first or last author was a woman being cited significantly less often than other articles (median: 36 vs. 54 citations for first authorship, and 37 vs. 51 citations for last authorship).22 However, the authors included only five journals in their study (New England Journal of Medicine, JAMA, BMJ, Annals of Internal Medicine, and JAMA Internal Medicine), which limits the generalizability of the results. In addition, they used Genderize, a gender-tool that is less efficient than others (e.g., Gender API or NamSor) in terms of inference accuracy,23 and did not perform multivariable analyses.
In the current cross-sectional study, we aimed to determine whether there is a gender gap in citations of articles published in a selection of fourteen high-impact general medical journals.
METHODS
Based on the Journal Citation Reports (JCR) list for the categories “general internal medicine” and “primary health care,” we selected all general medical journals with an impact factor for 2020 greater than five. Then, we used the Web of Science (WoS) resources to retrieve all research articles and reviews published between January 2015 and December 2019. We recorded the name of their authors, and the number of citations and citations per year as of 15 December 2021. The data collection was done with the International Standard Serial Number (ISSN) of the journals because journals can have multiple names in WoS. Finally, we used Gender API to determine the gender of the first and last author of the selected articles. Gender API (https://gender-api.com) is a web-based gender detection tool that infers gender from individuals’ first names. This tool has the great advantage that it can be used even by researchers with little computer knowledge. Its use is indeed extremely simple, since it only requires the uploading of a database in Excel or CSV format (https://gender-api.com/en/excel-and-csv). After processing the data, a “gender” column is added to the initial file. This study complies with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline for cross-sectional studies.
We summarized the data by calculating both the mean (SD) and median (IQR) number of citations and citations per year, stratified by journal and gender. We compared the results by gender using Wilcoxon rank sum test (right-skewed citation distribution) and univariable negative binomial regression (count data with over-dispersion). We also performed multivariable negative binomial regressions, adjusting for year of publication (for citations) and intra-cluster correlations within journals (for citations and citations per year). Finally, we examined the data for gender combinations of first and last authors (women-women, women-men, men-women, men-men) and tested for linear trend across these groups using orthogonal polynomial contrasts.
We repeated all analyses with two additional samples consisting of names whose gender could be determined with ≥ 60% and ≥ 80% accuracy, respectively, to exclude ambiguous names that could skew the gender distribution. Indeed, Gender API provides an additional parameter that estimates the accuracy of the inference (min 0, max 100%). All names whose gender was estimated with an accuracy of < 60% and < 80%, respectively, were considered non-classifications and therefore excluded from the analysis.
The statistical significance was set at a two-sided p value of ≤ 0.05. All analyses were performed with STATA 15.1.
RESULTS
Fourteen general medical journals were included in the study (Table 1). Impact factors ranged from 91.3 (New England Journal of Medicine) to 5.0 (American Journal of Medicine). Of the 14,256 articles retrieved by WoS, 13,350 had first names of authors and 1324 had only one author. Gender API determined the gender of the first and last author for 13,218/13,350 (99%) and 11,894/12,026 (99%) articles, respectively. The proportion of women among first and last authors was 40% and 29%, respectively, and varied by journal from 27 to 54% and from 20 to 41%.
The median number of citations was statistically higher for articles whose first author was a man vs. a woman in five journals including the four with the highest impact factor (Table 1). In contrast, women’s papers were on average more frequently cited than men’s papers in two journals. For the last authors, the median number of citations was statistically higher for men in three journals and for women in one journal.
When analyzing the data overall (Table 2), the unadjusted and adjusted differences in the number of citations and citations per year between women and men were statistically significant for both first and last authors (all p values < 0.001). For example, for first authors, the median number of citations was 31 (IQR = 90) for men and 23 (IQR = 56) for women (adjusted incident rate ratio (IRR) = 1.5 [95% CI = 1.3–1.8], p value < 0.001).
The results were all similar for the sensitivity analyses. For example, for the same indicator (i.e., median number of citations for first authorship), retaining only publications for which gender was determined with 60% and 80% accuracy, the adjusted IRR was 1.5 [(95% CI = 1.3–1.8), p value < 0.001] and 1.6 [(95% CI = 1.3–2.0), p value < 0.001], respectively.
Finally, when analyzing the first/last author combinations (Table 2), there was a linear trend across the groups, with articles with women as first and last authors being cited least often and articles with men as first and last authors being cited most often.
DISCUSSION
In this cross-sectional study, we found that articles published in high-impact general medical journals were on average cited less often when first/last authors were women vs. men. Articles whose both first and last authors were women were cited the least often.
Comparison with Existing Literature
Our results confirm those of the recent study conducted by Chatterjee and Werner on 5500 articles published in high-impact general medical journals (median number of citations: 36 for women vs. 54 for men for first authorship, and 37 vs. 51, respectively, for last authorship).22 In a previous study assessing the productivity of Swiss researchers in general internal medicine, we found that the gender difference in the number of citations per publication (median number of citations: one for women vs six for men) was not statistically significant in multivariable analysis.4 This lack of significance was probably related to the small sample size (N = 367 researchers).
Several hypotheses can in our opinion be considered to explain the gender gap in citations. Compared to men, women may be more restrained in the way they promote their research24 and are less often invited to medical conferences to present their studies.25,26,27 Their research may therefore be less known to the scientific community. It is also possible that women and men researchers differ in the research topics they address and how they address them.28,29 Since the majority of studies are conducted by men, these studies may be more likely to cite other articles authored by men. Because of the balance between career and family aspirations that is socially imposed on women, women researchers often have fewer years of practice behind them, and thus less experience, than their men colleagues of the same age. In addition, women’s research is not as well funded as men’s. With less experience or research projects that receive less funding, it can be hypothesized that women choose research topics or study designs that are considered as less “prestigious” and less valued by the scientific community. The proportion of researchers with an academic affiliation or with a particular type of affiliation is not necessarily the same for women and men. However, depending on the type of affiliation, researchers may have more or less time available to conduct their research. More time may often mean more ambitious studies and therefore of greater potential interest to the scientific community. Finally, men authors may be more inclined to practice self-citation. This technique is a known and opportunistic way to increase the h-index. There are probably still other reasons for these gender-related citation differences, which would certainly require further study to investigate in depth.
We also found that, overall, the proportion of women among first and last authors was 40% and 29% respectively. These results are similar to those from other recent studies conducted with general medical journals. For example, in a study examining 44,000 articles published between 2016 and 2020 in the 100 general medical journals with the highest impact factor, we found that the proportion of women among first authors was 41%.13 In another study, Hart and Perlis found that women were first and last authors of 42% and 32%, respectively, of research articles published in 2017 in a selection of 15 high-impact general medical journals (2016 impact factors ranging from 17.2 to 4.2).12
Various measures may be taken to reduce the gender gap in citations and, more generally, gender inequalities in research. Academic organizations should promote and support women researchers throughout their academic careers, for example, by allowing them to free up time for research, by advocating for gender equity in grants, or by improving the visibility of research done by women (media coverage, communication, conferences). In addition, the way in which research is evaluated should probably be questioned. The h-index only partially reflects the quality of research or the involvement of academics. As researchers, we should value the scientific content of studies more than measures such as the h-index.
Limitations
Our study has a large sample size but has several limitations. We were unable to examine the influence of a number of potentially confounding factors, such as authors’ institutional affiliation and career stage, because these variables were not available. Yet, these variables could explain to some extent the observed differences between the number of article citations and gender. In addition, the determination of the gender of the authors was done with a gender detection tool and not by manual internet search. However, the tool used (Gender API) has been shown to be accurate,23 only 1% of the queries resulted in a non-classification (i.e., undetermined gender), and the results of sensitivity analyses were similar. Finally, using a tool that determines gender on the basis of first names raises ethical considerations, as this method did not allow us to assess non-binary or transgender identity.
CONCLUSION
In this cross-sectional study that examined all research articles and reviews published between January 2015 and December 2019 in sixteen high-impact general medical journals (i.e., with an impact factor greater than five), we found that publications authored by women were cited less often than those authored by men. We also found that publications with women as first and last authors were the least frequently cited, whereas those with men as first and last authors were the most frequently cited.
Further studies with not only quantitative but also qualitative (or mixed) methods would be needed in the future to confirm our hypotheses regarding the reasons behind gender differences in article citations. The results of these studies could also be useful for implementing measures to ensure that publications by women and men researchers are cited more equitably.
Data Availability
The data associated with this article are available in the Open Science Framework (DOI 10.17605/OSF.IO/63QNS).
References
Women in Healthcare (Quick Take). Catalyst. Accessed January 27, 2022. https://www.catalyst.org/research/women-in-healthcare/
2018-2019 The state of women in academic medicine: exploring pathways to equity. AAMC. Accessed May 14, 2021. https://www.aamc.org/data-reports/data/2018-2019-state-women-academic-medicine-exploring-pathways-equity
Richter KP, Clark L, Wick JA, et al. Women physicians and promotion in academic medicine. N Engl J Med. 2020;383(22):2148-2157. doi:https://doi.org/10.1056/NEJMsa1916935
Sebo P, de Lucia S, Vernaz N. Gender gap in medical research: a bibliometric study in Swiss university hospitals. Scientometrics. Published online October 29, 2020. doi:https://doi.org/10.1007/s11192-020-03741-w
Jolly S, Griffith KA, DeCastro R, Stewart A, Ubel P, Jagsi R. Gender Differences in time spent on parenting and domestic responsibilities by high-achieving young physician-researchers. Ann Intern Med. 2014;160(5):344-353. doi:https://doi.org/10.7326/M13-0974
Sieverding M, Eib C, Neubauer AB, Stahl T. Can lifestyle preferences help explain the persistent gender gap in academia? The “mothers work less” hypothesis supported for German but not for U.S. early career researchers. PloS One. 2018;13(8):e0202728. doi:https://doi.org/10.1371/journal.pone.0202728
Ramakrishnan A, Sambuco D, Jagsi R. Women’s participation in the medical profession: insights from experiences in Japan, Scandinavia, Russia, and Eastern Europe. J Womens Health. 2014;23(11):927-934. doi:https://doi.org/10.1089/jwh.2014.4736
Wang MT, Degol JL. Gender Gap in Science, Technology, Engineering, and Mathematics (STEM): current knowledge, implications for practice, policy, and future directions. Educ Psychol Rev. 2017;29(1):119-140. doi:https://doi.org/10.1007/s10648-015-9355-x
Bates C, Gordon L, Travis E, et al. Striving for gender equity in academic medicine careers: a call to action. Acad Med J Assoc Am Med Coll. 2016;91(8):1050-1052. doi:https://doi.org/10.1097/ACM.0000000000001283
Kalaitzi S, Cheung KL, Hiligsmann M, Babich S, Czabanowska K. Exploring women healthcare leaders’ perceptions on barriers to leadership in Greek context. Front Public Health. 2019;7:68. doi:https://doi.org/10.3389/fpubh.2019.00068
Kalaitzi S, Czabanowska K, Azzopardi-Muscat N, et al. Women, healthcare leadership and societal culture: a qualitative study. J Healthc Leadersh. 2019;11:43-59. doi:https://doi.org/10.2147/JHL.S194733
Hart KL, Perlis RH. Trends in proportion of women as authors of medical journal articles, 2008-2018. JAMA Intern Med. 2019;179(9):1285-1287. doi:https://doi.org/10.1001/jamainternmed.2019.0907
Sebo P, Clair C. Gender gap in authorship: a study of 44,000 articles published in 100 high-impact general medical journals. Eur J Intern Med. Published online September 28, 2021:S0953-6205(21)00313-7. doi:https://doi.org/10.1016/j.ejim.2021.09.013
Filardo G, da Graca B, Sass DM, Pollock BD, Smith EB, Martinez MAM. Trends and comparison of female first authorship in high impact medical journals: observational study (1994-2014). BMJ. 2016;352:i847. doi:https://doi.org/10.1136/bmj.i847
Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci. 2005;102(46):16569-16572. doi:https://doi.org/10.1073/pnas.0507655102
Sebo P, de Lucia S. Evaluation of the productivity of hospital-based researchers: comparative study between the h-index and the h(fa)-index. Scientometrics. 2021;126(8):7087-7096. doi:https://doi.org/10.1007/s11192-021-04040-8
Baldock C. The h-index and medical physics. Australas Phys Eng Sci Med. 2008;31(2):xi-xii. doi:https://doi.org/10.1007/bf03178582
Venable GT, Khan NR, Taylor DR, Thompson CJ, Michael LM, Klimo P. A correlation between National Institutes of Health funding and bibliometrics in neurosurgery. World Neurosurg. 2014;81(3-4):468-472. doi:https://doi.org/10.1016/j.wneu.2013.11.013
Zaorsky NG, O’Brien E, Mardini J, Lehrer EJ, Holliday E, Weisman CS. Publication productivity and academic rank in medicine: a systematic review and meta-analysis. Acad Med J Assoc Am Med Coll. 2020;95(8):1274-1282. doi:https://doi.org/10.1097/ACM.0000000000003185
Zaorsky NG, Ahmed AA, Zhu J, et al. Industry funding is correlated with publication productivity of US Academic Radiation Oncologists. J Am Coll Radiol JACR. 2019;16(2):244-251. doi:https://doi.org/10.1016/j.jacr.2018.07.024
Yeh HC, Bertram A, Brancati FL, Cofrancesco J. Perceptions of division directors in general internal medicine about the importance of and support for scholarly work done by clinician-educators. Acad Med J Assoc Am Med Coll. 2015;90(2):203-208. doi:https://doi.org/10.1097/ACM.0000000000000554
Chatterjee P, Werner RM. Gender disparity in citations in high-impact journal articles. JAMA Netw Open. 2021;4(7):e2114509. doi:https://doi.org/10.1001/jamanetworkopen.2021.14509
Sebo P. Performance of gender detection tools: a comparative study of name-to-gender inference services. J Med Libr Assoc JMLA. 2021;109(3):414-421. doi:https://doi.org/10.5195/jmla.2021.1185
Lerchenmueller MJ, Sorenson O, Jena AB. Gender differences in how scientists present the importance of their research: observational study. BMJ. 2019;367:l6573. doi:https://doi.org/10.1136/bmj.l6573
Boiko JR, Anderson AJM, Gordon RA. Representation of women among academic grand rounds speakers. JAMA Intern Med. 2017;177(5):722-724. doi:https://doi.org/10.1001/jamainternmed.2016.9646
Fournier LE, Hopping GC, Zhu L, et al. Females are less likely invited speakers to the international stroke conference: time’s up to address sex disparity. Stroke. 2020;51(2):674-678. doi:https://doi.org/10.1161/STROKEAHA.119.027016
Carley S, Carden R, Riley R, et al. Are there too few women presenting at emergency medicine conferences? Emerg Med J EMJ. 2016;33(10):681-683. doi:https://doi.org/10.1136/emermed-2015-205581
Sebo P, Maisonneuve H, Fournier JP. Gender gap in research: a bibliometric study of published articles in primary health care and general internal medicine. Fam Pract. 2020;37(3):325-331. doi:https://doi.org/10.1093/fampra/cmz091
Ashmos Plowman D, Smith AD. The gendering of organizational research methods: Evidence of gender patterns in qualitative research. Qual Res Organ Manag Int J. 2011;6(1):64-82. doi:https://doi.org/10.1108/17465641111129399
Acknowledgements
None
Funding
Open access funding provided by University of Geneva.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethics Approval
Since this study did not involve the collection of personal health-related data, it did not require ethical review, according to current Swiss law.
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Sebo, P., Clair, C. Gender Inequalities in Citations of Articles Published in High-Impact General Medical Journals: a Cross-Sectional Study. J GEN INTERN MED 38, 661–666 (2023). https://doi.org/10.1007/s11606-022-07717-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11606-022-07717-9