Review Article
Complications Connected to Using the Impact Factor of Journals for the Assessment of Researchers in Higher Education
More Detail
1 Department of Educational Management, University of Calabar, Calabar, NIGERIA* Corresponding Author
Mediterranean Journal of Social & Behavioral Research, 5(1), February 2021, 13-21, https://doi.org/10.30935/mjosbr/10805
Submitted: 07 February 2021, Published: 31 March 2021
OPEN ACCESS 1608 Views 758 Downloads
ABSTRACT
The use of impact factor (IF) in the scientific and academic world is not new. A phenomenon that has gained wide-spread recognition and utilization. However, in modern-day usage, there seems to be a trend in higher education where academics are evaluated based on the impact factor of journals where scholarly works are published. This trend is gradually shifting the paradigm from the assessment of research contents to publication venue. This does not align with the original purpose of IF conceived by Garfield in 1955. One question that has continued to agitate the minds of concerned academics is whether the IF of journals is a dependable measure of research quality. This paper is an attempt to clarify or address this problem. Based on a thorough literature search and filtration, several problems about the use of IF as research quality measure are discussed as well as their implications. Recommendations were also made aimed at providing a way forward in higher education.
CITATION (APA)
Owan, V. J., & Owan, M. V. (2021). Complications Connected to Using the Impact Factor of Journals for the Assessment of Researchers in Higher Education. Mediterranean Journal of Social & Behavioral Research, 5(1), 13-21. https://doi.org/10.30935/mjosbr/10805
REFERENCES
- Abambres, M., Ribeiro, T., Sousa, A., & Lantsoght, E. (2016). Research counts, not the journal. HAL, 1-12. https://hal.archives-ouvertes.fr/hal-02074859v3
- Abramo, G., & D’Angelo, C. A. (2016). Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011-2014). Scientometrics, 109(3), 2053-2065. https://doi.org/10.1007/s11192-016-2153-5
- Agarwal, A., Durairajanayagam, D., Tatagari, S., Esteves, S. C., Harlev, A., Henkel, R., Roychoudhury, S., Homa, S., Puchalt, N. G., Ramasamy, R., Majzoub, A., Dao Ly, K., Tvrda, E., Assidi, M., Kesari, K., Sharma, R., Banihani, S., Ko, E., Abu-Elmagd, M., … Bashiri, A. (2016). Bibliometrics: Tracking research impact by selecting the appropriate metrics. Asian Journal of Andrology, 18(2), 296-309. https://doi.org/10.4103/1008-682X.171582
- Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. SAGE Open, 9(1), 215824401982957. https://doi.org/10.1177/2158244019829575
- Alberts, B. (2013). Impact factor distortions. Science, 340(6134), 787. https://doi.org/10.1126/science.1240319
- Aragon, A. M. (2013). A measure for the impact of research. Scientific Reports, 3, 1649. https://doi.org/10.1038/srep01649
- Bassey, B. A., & Owan, V. J. (2019). Ethical Issues in Educational Research. In P. N. Ololube & G. U. Nwiyi (Eds.), Encyclopedia of institutional leadership, policy, and management: A handbook of research in honour of Professor Ozo-Mekuri Ndimele (Vol 2, pp. 1287-1301). Pearl Publishers International Ltd. https://doi.org/10.13140/RG.2.2.11785.88161
- Baum, J. A. C. (2011). Free-riding on power laws: Questioning the validity of the impact factor as a measure of research quality in organization studies. Organization, 18(4), 449-466. https://doi.org/10.1177/1350508411403531
- Biagioli, M. (2018). Quality to impact, text to metadata: Publication and evaluation in the age of metrics. KNOW A Journal on the Formation of Knowledge, 2(2), 1-27. https://doi.org/10.1086/699152
- Blyth, E., Shardlow, S. M., Masson, H., Lyons, K., Shaw, I., & White, S. (2010). Measuring the quality of peer-reviewed publications in social work: Impact factors-liberation or liability? Social Work Education, 29(2), 120-136. https://doi.org/10.1080/02615470902856705
- Bohannon, J. (2013). Who’s afraid of peer review? Science, 342, 60-65. https://doi.org/10.1126/science.342.6154.60
- Buela-Casal, G., & Zych, I. (2012). What do scientists think about the impact factor? Scientometrics, 92(2), 281-292. https://doi.org/10.1007/s11192-012-0676-y
- Burrows, R. (2012). Living with the h-index? Metric assemblages in the contemporary academy. Sociological Review, 60(2), 355-372.
- Butler, L., & McAllister, I. (2009). Metrics or peer review? Evaluating the 2001 UK research assessment exercise in political science. Political Studies Review, 7(1), 3-17. https://doi.org/10.1111/j.1478-9299.2008.00167.x
- Campbell, P. (2008). Escape from the impact factor. Ethics in Science and Environmental Politics 8, 5-7. https://doi.org/10.3354/esep00078
- Casadevall, A., & Fang, F. C. (2014). Causes for the persistence of impact factor mania. MBio, 5(2), e00064-14. https://doi.org/10.1128/mBio.00064-14
- Chapman, C. A., Bicca-Marques, J. C., Calvignac-Spencer, S., Fan, P., Fashing, P. J., Gogarten, J., Guo, S., Hemingway, C. A., Leendertz, F., Li, B., Matsuda, I., Hou, R., Serio-Silva, J. C., & Stenseth, N. C. (2019). Games academics play and their consequences: How authorship, h-index and journal impact factors are shaping the future of academia. Proceedings of the Royal Society B: Biological Sciences, 286, 2019.2047. https://doi.org/10.1098/rspb.2019.2047
- Cheek, J., Garnham, B., & Quan, J. (2006). What’s in a number? Issues in providing evidence of impact and quality of research(ers). Qualitative Health Research, 16(3), 423-435. https://doi.org/10.1177/1049732305285701
- Craig, I. D., A. M. Plume, M. E. McVeigh, J. Pringle, and M. Amin (2007). Do open-access articles have a greater citation impact? A critical review of the literature. Journal of Informetrics, 1(3), 239-248. https://doi.org/10.1016/j.joi.2007.04.001
- Elliott, D. B. (2014). The impact factor: A useful indicator of journal quality or fatally flawed? Ophthalmic and Physiological Optics, 34(1), 4-7. https://doi.org/10.1111/opo.12107
- Eston, R., (2005). The impact factor: A misleading and flawed measure of research quality. Journal of Sports Sciences, 23(1), 1-3. https://doi.org/10.1080/02640410400014208
- Fallon, L. M., Sanetti, L. M. H., & Johnson, A. H. (2015). Is performance feedback for educators an evidence-based practice? A systematic review and evaluation based on single-case. Exceptional Children, 81(2), 227-246. https://doi.org/10.1177/0014402914551738
- Folly, G., Hajtman, B., Nagy, J. I., & Ruff, I. (1981). Some methodological problems in ranking scientists by citation analysis. Scientometrics, 3, 135-147. https://doi.org/10.1007/BF02025636
- Fu, L. D., Aphinyanaphongs, Y. Y., Wang, L., & Aliferis, C. F. (2011). A comparison of evaluation metrics for biomedical journals, articles, and websites in terms of sensitivity to topic. Journal of Biomedical Informatics, 44(4), 587-594. https://doi.org/10.1016/j.jbi.2011.03.006
- Garfield, E. (1955). Citation indexes for science: A new dimension in documentation through association of ideas. Science, 122(3159), 108-111. https://doi.org/10.1126/science.122.3159.108
- Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471-479. https://doi.org/10.1126/science.178.4060.471
- Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal 161(8), 979-980. https://bit.ly/2WPwNvf
- Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295(1), 90-93. https://doi.org/10.1001/jama.295.1.90
- Grech, V. (2018). Increasing importance of research metrics: Journal impact factor and h-index. International Urogynecology Journal, 29(5), 619-620. https://doi.org/10.1007/s00192-018-3604-8
- Gruber, T. (2015). Academic sell-out: How an obsession with metrics and rankings is damaging academia. Journal of Marketing for Higher Education, 24(2), 165-177. https://doi.org/10.1080/08841241.2014.970248
- Gruber, T., Reppel, A., & Voss, R. (2010). Understanding the characteristics of effective professors: The student’s perspective. Journal of Marketing for Higher Education, 20(2), 175-190. https://doi.org/10.1080/08841241.2010.526356
- Haddawy, P., Hassan, S. U., Asghar, A., & Amin, S. (2016). A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality. Journal of Informetrics, 10(1), 162-173. https://doi.org/10.1016/j.joi.2015.12.005
- Hallberg, L. (2012). Can the impact factor measure the quality of research? International Journal of Qualitative Studies on Health and Well-Being, 7(1), 2-4. https://doi.org/10.3402/qhw.v7i0.19772
- Hammarfelt, B., & Rushforth, A. D. (2017). Indicators as judgment devices: An empirical study of citizen bibliometrics in research evaluation. Research Evaluation, 26(3), 169-180. https://doi.org/10.1093/reseval/rvx018
- Hernan, M. A. (2008). Epidemiologists (of all people) should question journal impact factors. Epidemiology, 19(3), 366-368. https://doi.org/10.1097/EDE.0b013e31816a9e28
- Huggett, S. (2013). Journal bibliometrics indicators and citation ethics: A discussion of current issues. Atherosclerosis, 230(2), 275-277. https://doi.org/10.1016/j.atherosclerosis.2013.07.051
- Ioannidis, J. P., Tatsioni, A., & Karassa, F. B. (2010). Who is afraid of reviewers’ comments? Or, why anything can be published and anything can be cited. Eur J Clin Invest, 40(4), 285-287. https://doi.org/10.1111/j.1365-2362.2010.02272.x
- Jarwal, S. D., Brion, A. M., & King, M. L. (2009). Measuring research quality using the journal impact factor, citations and ‘Ranked Journals’: Blunt instruments or inspired metrics? Journal of Higher Education Policy and Management, 31(4), 289-300. https://doi.org/10.1080/13600800903191930
- Joshi, M. A. (2015). Bibliometric indicators for evaluating the quality of scientific publications. Journal of Contemporary Dental Practice, 15(2), 258-262. https://doi.org/10.5005/jp-journals-10024-1525
- Kieling, C., & Gonçalves, R. R. F. (2007). Assessing the quality of a scientific journal: The case of Revista Brasileira de Psiquiatria. Revista Brasileira de Psiquiatria, 29(2), 177-181. https://doi.org/10.1590/S1516-44462007000200017
- Kiesslich, T., Weineck, S. B., & Koelblinger, D. (2016). Reasons for journal impact factor changes: Influence of changing source items. PLoS ONE, 11(4), e0154199. https://doi.org/10.1371/journal.pone.0154199
- Kochen, M., & Himmel, W. (1998). A critical assessment of the impact factor. European Journal of General Practice, 4(4), 159-163. https://doi.org/10.3109/13814789809160812
- Koya, K., & Chowdhury, G. (2017). Metric-based vs peer-reviewed evaluation of research output: Lesson learnt from UK’s national research assessment exercise. PLoS ONE, 12(7), 1-18. https://doi.org/10.1371/journal.pone.0179722
- Kressel, H. Y. (2014). Beyond the impact factor: Enhancing the impact of imaging research published in radiology. Radiology, 270(1), 3-6. https://doi.org/10.1148/radiol.13132314
- Langfeldt, L. (2001). The decision-making constraints and processes of grant peer review and their effects on the review outcome. Social Studies of Science, 31(6), 820-841. https://doi.org/10.1177/030631201031006002
- Larsen, P. O., & von Ins, M. (2010). The rate of growth in scientific publication and the decline in coverage provided by Science Citation index. Scientometrics, 84(3), 575-603. https://doi.org/10.1007/s11192-010-0202-z
- Lawrence, P. A. (2007). The mismeasurement of science. Current Biology, 17(15), R583-R585. https://doi.org/10.1016/j.cub.2007.06.014
- Leydesdorff, L., Bornmann, L., Comins, J. A., & Milojević, S. (2016). Citations: Indicators of quality? The impact fallacy. Frontiers in Research Metrics and Analytics, 1, 1-15. https://doi.org/10.3389/frma.2016.00001
- Lippi, G., & Mattiuzzi, C. (2017). Scientist impact factor (SIF): A new metric for improving scientists’ evaluation? Annals of Translational Medicine, 5(15), 2-5. https://doi.org/10.21037/atm.2017.06.24
- Mahian, O., & Wongwises, S. (2015). Is it ethical for journals to request self‑citation? Science and Engineering Ethics, 21(2), 531-533. https://doi.org/10.1007/s11948-014-9540-1
- Margolis, J. (1967). Citation indexing and evaluation of scientific papers. Science, 155(3767), 1213-1219. https://doi.org/10.1126/science.155.3767.1213
- Mårtensson, P., Fors, U., Wallin, S. B., Zander, U., & Nilsson, G. H. (2016). Evaluating research: A multidisciplinary approach to assessing research practice and quality. Research Policy, 45(3), 593-603. https://doi.org/10.1016/j.respol.2015.11.009
- Martin, B. R., & Irvine, J. (1983). Assessing basic research: Some partial indicators of scientific progress in radio astronomy. Research Policy, 12(2), 61-90. https://doi.org/10.1016/0048-7333(83)90005-7
- McKiernan, E. C., Schimanski, L. A., Nieves, C. M., Matthias, L., Niles, M. T., & Alperin, J. P. (2019). Use of the journal impact factor in academic review, promotion, and tenure evaluations. ELife, 8, e47338. https://doi.org/10.7554/eLife.47338
- Mingers, J., & Yang, L. (2017). Evaluating journal quality: A review of journal citation indicators and ranking in business and management. European Journal of Operational Research, 257(1), 323-337. https://doi.org/10.1016/j.ejor.2016.07.058
- Misteli, T. (2013). Eliminating the impact of the impact factor. Journal of Cell Biology, 201(5), 651-652. https://doi.org/10.1083/jcb.201304162
- Nature (2003). Coping with peer rejection. Nature, 425(645), 6959. https://doi.org/10.1038/425645a
- Neuberger, J., & Counsell, C. (2002). Impact factors: Uses and abuses. European Journal of Gastroenterology & Hepatology, 14(3), 209-211. https://doi.org/10.1097/00042737-200203000-00001
- Odigwe, F. N., Bassey, B. A., & Owan, V. J. (2020). Data management practices and educational research effectiveness of university lecturers in South-South Nigeria. Journal of Educational and Social Research (JESR), 10(3), 24-34. https://doi.org/10.36941/jesr-2020-0042
- Orduña-Malea, E., Martín-Martín, A., & Delgado-López-Cózar, E. (2016). The next bibliometrics: ALMetrics (Author Level Metrics) and the multiple faces of author impact. Profesional de La Informacion, 25(3), 485-496. https://doi.org/10.3145/epi.2016.may.18
- Owan, V. J., & Bassey, B. A. (2019). Data management practices in Educational Research. In N. P. Ololube & G. U. Nwiyi (Eds.), Encyclopedia of institutional leadership, policy, and management: A handbook of research in honour of Professor Ozo-Mekuri Ndimele (Vol 2, pp. 1251-1265). Pearl Publishers International Ltd. https://doi.org/10.13140/RG.2.2.16819.04647
- Parker, M., & Jary, D. (1995). The McUniversity: Organization, management and academic subjectivity. Organization, 2(2), 319-338. https://doi.org/10.1177/135050849522013
- Quan, W., Chen, B., & Shu, F. (2017). Publish or impoverish: An investigation of the monetary reward system of science in China (1999-2016). Aslib Journal of Information Management, 69(5), 486-502. https://doi.org/10.1108/AJIM-01-2017-0014
- Rousseau, R. (2002). Journal evaluation: Technical and practical issues. Library Trends, 50(3), 418-439. https://bit.ly/2ySdoSm
- Rowlands, I., & Nicholas, D. (2007). The missing link: Journal usage metrics. Aslib Proceedings: New Information Perspectives, 59(3), 222-228. https://doi.org/10.1108/00012530710752025
- Rushforth, A., & de Rijcke, S. (2015). Accounting for impact? The journal impact factor and the making of biomedical research in the Netherlands. Minerva, 53(2), 117-139. https://doi.org/10.1007/s11024-015-9274-5
- Saha, S., Saint, S., & Christakis, D. A. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91(1), 42-46. https://bit.ly/35WGjkr
- Sauder, M., & Espeland W. N. (2009). The discipline of rankings: Tight coupling and organizational change. American Sociological Review, 74(1), 63-82. https://doi.org/10.1177/000312240907400104
- Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628-638. https://doi.org/10.1002/(SICI)1097-4571(199210)43:9%3C628::AID-ASI5%3E3.0.CO;2-0
- Seglen, P. O. (1997a). Citations and journal impact factors: Questionable indicators of research quality. Allergy: European Journal of Allergy and Clinical Immunology, 52(11), 1050-1056. https://doi.org/10.1111/j.1398-9995.1997.tb00175.x
- Seglen, P. O. (1997b). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314(7079), 498-502. https://doi.org/10.1136/bmj.314.7079.497
- Seglen, P. O. (1998). Citation rates and journal impact factors are not suitable for the evaluation of research. Acta Orthopaedica Scandinavica, 69(3), 224-229. https://doi.org/10.3109/17453679809000920
- Sevinc, A. (2004). Manipulating impact factor: An unethical issue or an editor’s choice? Swiss Medical Weekly, 134(27-28), 410. https://doi.org/10.4414/smw.2004.10761
- Shore, C. (2010). Beyond the multiversity: Neoliberalism and the rise of the schizophrenic university. Social Anthropology, 18(1), 15-29. https://doi.org/10.1111/j.1469-8676.2009.00094.x
- Smith, R. (1988). Problems with peer review and alternatives. Research Policy, 296, 774-777. https://doi.org/10.1136/bmj.296.6624.774
- Smith, R. (1997). Journal accused of manipulating impact factor. BMJ, 314(7079), 461. https://doi.org/10.1136/bmj.314.7079.461d
- Solimini, A. G., & Solimini, R. (2011). Impact factor and other metrics for evaluating science: Essentials for public health practitioners. Italian Journal of Public Health, 8(1), 96-103. https://doi.org/10.2427/5650
- Sugimoto, C. R., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science and Technology, 68(9), 2037-2062. https://doi.org/10.1002/asi.23833
- Wall, H. J. (2009). Don’t get skewed over by journal rankings. The B. E. Journal of Economic Analysis and Policy, 9(1), Article 34. https://doi.org/10.2202/1935-1682.2280
- Weale, A. R., Bailey, M., & Lear, P. A. (2004). The level of non-citation of articles within a journal as a measure of quality: A comparison to the impact factor. BMC Medical Research Methodology, 4(1), 14. https://doi.org/10.1186/1471-2288-4-14
- Whitehouse, G. H. (2001). Citation rates and impact factors: Should they matter? British Journal of Radiology, 74(877), 1-3. https://doi.org.10.1259/bjr.74.877.740001
- Willmott, H. (2011). Journal list fetishism and the perversion of scholarship: Reactivity and the ABS list. Organization, 18(4), 429-442. https://doi.org/10.1177/1350508411403532