Review Article

Complications Connected to Using the Impact Factor of Journals for the Assessment of Researchers in Higher Education

Valentine Joseph Owan 1 * , Mercy Valentine Owan 1
More Detail
1 Department of Educational Management, University of Calabar, Calabar, NIGERIA* Corresponding Author
Mediterranean Journal of Social & Behavioral Research, 5(1), February 2021, 13-21, https://doi.org/10.30935/mjosbr/10805
OPEN ACCESS   1265 Views   607 Downloads
Download Full Text (PDF)

ABSTRACT

The use of impact factor (IF) in the scientific and academic world is not new. A phenomenon that has gained wide-spread recognition and utilization. However, in modern-day usage, there seems to be a trend in higher education where academics are evaluated based on the impact factor of journals where scholarly works are published. This trend is gradually shifting the paradigm from the assessment of research contents to publication venue. This does not align with the original purpose of IF conceived by Garfield in 1955. One question that has continued to agitate the minds of concerned academics is whether the IF of journals is a dependable measure of research quality. This paper is an attempt to clarify or address this problem. Based on a thorough literature search and filtration, several problems about the use of IF as research quality measure are discussed as well as their implications. Recommendations were also made aimed at providing a way forward in higher education.
Keywords: evaluation, higher education, impact factor, journal, measure, research

CITATION (APA)

Owan, V. J., & Owan, M. V. (2021). Complications Connected to Using the Impact Factor of Journals for the Assessment of Researchers in Higher Education. Mediterranean Journal of Social & Behavioral Research, 5(1), 13-21. https://doi.org/10.30935/mjosbr/10805

REFERENCES

  1. Abambres, M., Ribeiro, T., Sousa, A., & Lantsoght, E. (2016). Research counts, not the journal. HAL, 1-12. https://hal.archives-ouvertes.fr/hal-02074859v3
  2. Abramo, G., & D’Angelo, C. A. (2016). Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011-2014). Scientometrics, 109(3), 2053-2065. https://doi.org/10.1007/s11192-016-2153-5
  3. Agarwal, A., Durairajanayagam, D., Tatagari, S., Esteves, S. C., Harlev, A., Henkel, R., Roychoudhury, S., Homa, S., Puchalt, N. G., Ramasamy, R., Majzoub, A., Dao Ly, K., Tvrda, E., Assidi, M., Kesari, K., Sharma, R., Banihani, S., Ko, E., Abu-Elmagd, M., … Bashiri, A. (2016). Bibliometrics: Tracking research impact by selecting the appropriate metrics. Asian Journal of Andrology, 18(2), 296-309. https://doi.org/10.4103/1008-682X.171582
  4. Aksnes, D. W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. SAGE Open, 9(1), 215824401982957. https://doi.org/10.1177/2158244019829575
  5. Alberts, B. (2013). Impact factor distortions. Science, 340(6134), 787. https://doi.org/10.1126/science.1240319
  6. Aragon, A. M. (2013). A measure for the impact of research. Scientific Reports, 3, 1649. https://doi.org/10.1038/srep01649
  7. Bassey, B. A., & Owan, V. J. (2019). Ethical Issues in Educational Research. In P. N. Ololube & G. U. Nwiyi (Eds.), Encyclopedia of institutional leadership, policy, and management: A handbook of research in honour of Professor Ozo-Mekuri Ndimele (Vol 2, pp. 1287-1301). Pearl Publishers International Ltd. https://doi.org/10.13140/RG.2.2.11785.88161
  8. Baum, J. A. C. (2011). Free-riding on power laws: Questioning the validity of the impact factor as a measure of research quality in organization studies. Organization, 18(4), 449-466. https://doi.org/10.1177/1350508411403531
  9. Biagioli, M. (2018). Quality to impact, text to metadata: Publication and evaluation in the age of metrics. KNOW A Journal on the Formation of Knowledge, 2(2), 1-27. https://doi.org/10.1086/699152
  10. Blyth, E., Shardlow, S. M., Masson, H., Lyons, K., Shaw, I., & White, S. (2010). Measuring the quality of peer-reviewed publications in social work: Impact factors-liberation or liability? Social Work Education, 29(2), 120-136. https://doi.org/10.1080/02615470902856705
  11. Bohannon, J. (2013). Who’s afraid of peer review? Science, 342, 60-65. https://doi.org/10.1126/science.342.6154.60
  12. Buela-Casal, G., & Zych, I. (2012). What do scientists think about the impact factor? Scientometrics, 92(2), 281-292. https://doi.org/10.1007/s11192-012-0676-y
  13. Burrows, R. (2012). Living with the h-index? Metric assemblages in the contemporary academy. Sociological Review, 60(2), 355-372.
  14. Butler, L., & McAllister, I. (2009). Metrics or peer review? Evaluating the 2001 UK research assessment exercise in political science. Political Studies Review, 7(1), 3-17. https://doi.org/10.1111/j.1478-9299.2008.00167.x
  15. Campbell, P. (2008). Escape from the impact factor. Ethics in Science and Environmental Politics 8, 5-7. https://doi.org/10.3354/esep00078
  16. Casadevall, A., & Fang, F. C. (2014). Causes for the persistence of impact factor mania. MBio, 5(2), e00064-14. https://doi.org/10.1128/mBio.00064-14
  17. Chapman, C. A., Bicca-Marques, J. C., Calvignac-Spencer, S., Fan, P., Fashing, P. J., Gogarten, J., Guo, S., Hemingway, C. A., Leendertz, F., Li, B., Matsuda, I., Hou, R., Serio-Silva, J. C., & Stenseth, N. C. (2019). Games academics play and their consequences: How authorship, h-index and journal impact factors are shaping the future of academia. Proceedings of the Royal Society B: Biological Sciences, 286, 2019.2047. https://doi.org/10.1098/rspb.2019.2047
  18. Cheek, J., Garnham, B., & Quan, J. (2006). What’s in a number? Issues in providing evidence of impact and quality of research(ers). Qualitative Health Research, 16(3), 423-435. https://doi.org/10.1177/1049732305285701
  19. Craig, I. D., A. M. Plume, M. E. McVeigh, J. Pringle, and M. Amin (2007). Do open-access articles have a greater citation impact? A critical review of the literature. Journal of Informetrics, 1(3), 239-248. https://doi.org/10.1016/j.joi.2007.04.001
  20. Elliott, D. B. (2014). The impact factor: A useful indicator of journal quality or fatally flawed? Ophthalmic and Physiological Optics, 34(1), 4-7. https://doi.org/10.1111/opo.12107
  21. Eston, R., (2005). The impact factor: A misleading and flawed measure of research quality. Journal of Sports Sciences, 23(1), 1-3. https://doi.org/10.1080/02640410400014208
  22. Fallon, L. M., Sanetti, L. M. H., & Johnson, A. H. (2015). Is performance feedback for educators an evidence-based practice? A systematic review and evaluation based on single-case. Exceptional Children, 81(2), 227-246. https://doi.org/10.1177/0014402914551738
  23. Folly, G., Hajtman, B., Nagy, J. I., & Ruff, I. (1981). Some methodological problems in ranking scientists by citation analysis. Scientometrics, 3, 135-147. https://doi.org/10.1007/BF02025636
  24. Fu, L. D., Aphinyanaphongs, Y. Y., Wang, L., & Aliferis, C. F. (2011). A comparison of evaluation metrics for biomedical journals, articles, and websites in terms of sensitivity to topic. Journal of Biomedical Informatics, 44(4), 587-594. https://doi.org/10.1016/j.jbi.2011.03.006
  25. Garfield, E. (1955). Citation indexes for science: A new dimension in documentation through association of ideas. Science, 122(3159), 108-111. https://doi.org/10.1126/science.122.3159.108
  26. Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471-479. https://doi.org/10.1126/science.178.4060.471
  27. Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal 161(8), 979-980. https://bit.ly/2WPwNvf
  28. Garfield, E. (2006). The history and meaning of the journal impact factor. Journal of the American Medical Association, 295(1), 90-93. https://doi.org/10.1001/jama.295.1.90
  29. Grech, V. (2018). Increasing importance of research metrics: Journal impact factor and h-index. International Urogynecology Journal, 29(5), 619-620. https://doi.org/10.1007/s00192-018-3604-8
  30. Gruber, T. (2015). Academic sell-out: How an obsession with metrics and rankings is damaging academia. Journal of Marketing for Higher Education, 24(2), 165-177. https://doi.org/10.1080/08841241.2014.970248
  31. Gruber, T., Reppel, A., & Voss, R. (2010). Understanding the characteristics of effective professors: The student’s perspective. Journal of Marketing for Higher Education, 20(2), 175-190. https://doi.org/10.1080/08841241.2010.526356
  32. Haddawy, P., Hassan, S. U., Asghar, A., & Amin, S. (2016). A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality. Journal of Informetrics, 10(1), 162-173. https://doi.org/10.1016/j.joi.2015.12.005
  33. Hallberg, L. (2012). Can the impact factor measure the quality of research? International Journal of Qualitative Studies on Health and Well-Being, 7(1), 2-4. https://doi.org/10.3402/qhw.v7i0.19772
  34. Hammarfelt, B., & Rushforth, A. D. (2017). Indicators as judgment devices: An empirical study of citizen bibliometrics in research evaluation. Research Evaluation, 26(3), 169-180. https://doi.org/10.1093/reseval/rvx018
  35. Hernan, M. A. (2008). Epidemiologists (of all people) should question journal impact factors. Epidemiology, 19(3), 366-368. https://doi.org/10.1097/EDE.0b013e31816a9e28
  36. Huggett, S. (2013). Journal bibliometrics indicators and citation ethics: A discussion of current issues. Atherosclerosis, 230(2), 275-277. https://doi.org/10.1016/j.atherosclerosis.2013.07.051
  37. Ioannidis, J. P., Tatsioni, A., & Karassa, F. B. (2010). Who is afraid of reviewers’ comments? Or, why anything can be published and anything can be cited. Eur J Clin Invest, 40(4), 285-287. https://doi.org/10.1111/j.1365-2362.2010.02272.x
  38. Jarwal, S. D., Brion, A. M., & King, M. L. (2009). Measuring research quality using the journal impact factor, citations and ‘Ranked Journals’: Blunt instruments or inspired metrics? Journal of Higher Education Policy and Management, 31(4), 289-300. https://doi.org/10.1080/13600800903191930
  39. Joshi, M. A. (2015). Bibliometric indicators for evaluating the quality of scientific publications. Journal of Contemporary Dental Practice, 15(2), 258-262. https://doi.org/10.5005/jp-journals-10024-1525
  40. Kieling, C., & Gonçalves, R. R. F. (2007). Assessing the quality of a scientific journal: The case of Revista Brasileira de Psiquiatria. Revista Brasileira de Psiquiatria, 29(2), 177-181. https://doi.org/10.1590/S1516-44462007000200017
  41. Kiesslich, T., Weineck, S. B., & Koelblinger, D. (2016). Reasons for journal impact factor changes: Influence of changing source items. PLoS ONE, 11(4), e0154199. https://doi.org/10.1371/journal.pone.0154199
  42. Kochen, M., & Himmel, W. (1998). A critical assessment of the impact factor. European Journal of General Practice, 4(4), 159-163. https://doi.org/10.3109/13814789809160812
  43. Koya, K., & Chowdhury, G. (2017). Metric-based vs peer-reviewed evaluation of research output: Lesson learnt from UK’s national research assessment exercise. PLoS ONE, 12(7), 1-18. https://doi.org/10.1371/journal.pone.0179722
  44. Kressel, H. Y. (2014). Beyond the impact factor: Enhancing the impact of imaging research published in radiology. Radiology, 270(1), 3-6. https://doi.org/10.1148/radiol.13132314
  45. Langfeldt, L. (2001). The decision-making constraints and processes of grant peer review and their effects on the review outcome. Social Studies of Science, 31(6), 820-841. https://doi.org/10.1177/030631201031006002
  46. Larsen, P. O., & von Ins, M. (2010). The rate of growth in scientific publication and the decline in coverage provided by Science Citation index. Scientometrics, 84(3), 575-603. https://doi.org/10.1007/s11192-010-0202-z
  47. Lawrence, P. A. (2007). The mismeasurement of science. Current Biology, 17(15), R583-R585. https://doi.org/10.1016/j.cub.2007.06.014
  48. Leydesdorff, L., Bornmann, L., Comins, J. A., & Milojević, S. (2016). Citations: Indicators of quality? The impact fallacy. Frontiers in Research Metrics and Analytics, 1, 1-15. https://doi.org/10.3389/frma.2016.00001
  49. Lippi, G., & Mattiuzzi, C. (2017). Scientist impact factor (SIF): A new metric for improving scientists’ evaluation? Annals of Translational Medicine, 5(15), 2-5. https://doi.org/10.21037/atm.2017.06.24
  50. Mahian, O., & Wongwises, S. (2015). Is it ethical for journals to request self‑citation? Science and Engineering Ethics, 21(2), 531-533. https://doi.org/10.1007/s11948-014-9540-1
  51. Margolis, J. (1967). Citation indexing and evaluation of scientific papers. Science, 155(3767), 1213-1219. https://doi.org/10.1126/science.155.3767.1213
  52. Mårtensson, P., Fors, U., Wallin, S. B., Zander, U., & Nilsson, G. H. (2016). Evaluating research: A multidisciplinary approach to assessing research practice and quality. Research Policy, 45(3), 593-603. https://doi.org/10.1016/j.respol.2015.11.009
  53. Martin, B. R., & Irvine, J. (1983). Assessing basic research: Some partial indicators of scientific progress in radio astronomy. Research Policy, 12(2), 61-90. https://doi.org/10.1016/0048-7333(83)90005-7
  54. McKiernan, E. C., Schimanski, L. A., Nieves, C. M., Matthias, L., Niles, M. T., & Alperin, J. P. (2019). Use of the journal impact factor in academic review, promotion, and tenure evaluations. ELife, 8, e47338. https://doi.org/10.7554/eLife.47338
  55. Mingers, J., & Yang, L. (2017). Evaluating journal quality: A review of journal citation indicators and ranking in business and management. European Journal of Operational Research, 257(1), 323-337. https://doi.org/10.1016/j.ejor.2016.07.058
  56. Misteli, T. (2013). Eliminating the impact of the impact factor. Journal of Cell Biology, 201(5), 651-652. https://doi.org/10.1083/jcb.201304162
  57. Nature (2003). Coping with peer rejection. Nature, 425(645), 6959. https://doi.org/10.1038/425645a
  58. Neuberger, J., & Counsell, C. (2002). Impact factors: Uses and abuses. European Journal of Gastroenterology & Hepatology, 14(3), 209-211. https://doi.org/10.1097/00042737-200203000-00001
  59. Odigwe, F. N., Bassey, B. A., & Owan, V. J. (2020). Data management practices and educational research effectiveness of university lecturers in South-South Nigeria. Journal of Educational and Social Research (JESR), 10(3), 24-34. https://doi.org/10.36941/jesr-2020-0042
  60. Orduña-Malea, E., Martín-Martín, A., & Delgado-López-Cózar, E. (2016). The next bibliometrics: ALMetrics (Author Level Metrics) and the multiple faces of author impact. Profesional de La Informacion, 25(3), 485-496. https://doi.org/10.3145/epi.2016.may.18
  61. Owan, V. J., & Bassey, B. A. (2019). Data management practices in Educational Research. In N. P. Ololube & G. U. Nwiyi (Eds.), Encyclopedia of institutional leadership, policy, and management: A handbook of research in honour of Professor Ozo-Mekuri Ndimele (Vol 2, pp. 1251-1265). Pearl Publishers International Ltd. https://doi.org/10.13140/RG.2.2.16819.04647
  62. Parker, M., & Jary, D. (1995). The McUniversity: Organization, management and academic subjectivity. Organization, 2(2), 319-338. https://doi.org/10.1177/135050849522013
  63. Quan, W., Chen, B., & Shu, F. (2017). Publish or impoverish: An investigation of the monetary reward system of science in China (1999-2016). Aslib Journal of Information Management, 69(5), 486-502. https://doi.org/10.1108/AJIM-01-2017-0014
  64. Rousseau, R. (2002). Journal evaluation: Technical and practical issues. Library Trends, 50(3), 418-439. https://bit.ly/2ySdoSm
  65. Rowlands, I., & Nicholas, D. (2007). The missing link: Journal usage metrics. Aslib Proceedings: New Information Perspectives, 59(3), 222-228. https://doi.org/10.1108/00012530710752025
  66. Rushforth, A., & de Rijcke, S. (2015). Accounting for impact? The journal impact factor and the making of biomedical research in the Netherlands. Minerva, 53(2), 117-139. https://doi.org/10.1007/s11024-015-9274-5
  67. Saha, S., Saint, S., & Christakis, D. A. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91(1), 42-46. https://bit.ly/35WGjkr
  68. Sauder, M., & Espeland W. N. (2009). The discipline of rankings: Tight coupling and organizational change. American Sociological Review, 74(1), 63-82. https://doi.org/10.1177/000312240907400104
  69. Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628-638. https://doi.org/10.1002/(SICI)1097-4571(199210)43:9%3C628::AID-ASI5%3E3.0.CO;2-0
  70. Seglen, P. O. (1997a). Citations and journal impact factors: Questionable indicators of research quality. Allergy: European Journal of Allergy and Clinical Immunology, 52(11), 1050-1056. https://doi.org/10.1111/j.1398-9995.1997.tb00175.x
  71. Seglen, P. O. (1997b). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314(7079), 498-502. https://doi.org/10.1136/bmj.314.7079.497
  72. Seglen, P. O. (1998). Citation rates and journal impact factors are not suitable for the evaluation of research. Acta Orthopaedica Scandinavica, 69(3), 224-229. https://doi.org/10.3109/17453679809000920
  73. Sevinc, A. (2004). Manipulating impact factor: An unethical issue or an editor’s choice? Swiss Medical Weekly, 134(27-28), 410. https://doi.org/10.4414/smw.2004.10761
  74. Shore, C. (2010). Beyond the multiversity: Neoliberalism and the rise of the schizophrenic university. Social Anthropology, 18(1), 15-29. https://doi.org/10.1111/j.1469-8676.2009.00094.x
  75. Smith, R. (1988). Problems with peer review and alternatives. Research Policy, 296, 774-777. https://doi.org/10.1136/bmj.296.6624.774
  76. Smith, R. (1997). Journal accused of manipulating impact factor. BMJ, 314(7079), 461. https://doi.org/10.1136/bmj.314.7079.461d
  77. Solimini, A. G., & Solimini, R. (2011). Impact factor and other metrics for evaluating science: Essentials for public health practitioners. Italian Journal of Public Health, 8(1), 96-103. https://doi.org/10.2427/5650
  78. Sugimoto, C. R., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science and Technology, 68(9), 2037-2062. https://doi.org/10.1002/asi.23833
  79. Wall, H. J. (2009). Don’t get skewed over by journal rankings. The B. E. Journal of Economic Analysis and Policy, 9(1), Article 34. https://doi.org/10.2202/1935-1682.2280
  80. Weale, A. R., Bailey, M., & Lear, P. A. (2004). The level of non-citation of articles within a journal as a measure of quality: A comparison to the impact factor. BMC Medical Research Methodology, 4(1), 14. https://doi.org/10.1186/1471-2288-4-14
  81. Whitehouse, G. H. (2001). Citation rates and impact factors: Should they matter? British Journal of Radiology, 74(877), 1-3. https://doi.org.10.1259/bjr.74.877.740001
  82. Willmott, H. (2011). Journal list fetishism and the perversion of scholarship: Reactivity and the ABS list. Organization, 18(4), 429-442. https://doi.org/10.1177/1350508411403532