Citation-tracking databases, such as Scopus, Web of Science, and Google Scholar, are used extensively to collect and report a range of bibliometric measures. Citation-tracking databases are susceptible to limitations based on factors such as:
1 Federation for the Humanities and Social Sciences. (2014). The Impacts of Humanities and Social Science Research: Working Paper. Retrieved from http://www.ideas-idees.ca/sites/default/files/2014-10-03-impact-project-draft-report-english-version-final2.pdf
2 Archambault, E., & Lariviere, V. (2010). The limits of bibliometrics for the analysis of the social sciences and humanities literature. World Social Science Report : Knowledge Divides. (pp. 251-254). Paris: UNESCO Publishing and International Social Science Council.
To be reliable, a large sample size is required in a number of measures, and percentiles may be a more suitable approach with small sample sizes depending on the context.
1 Vieira, E. S., & Gomes, J. A. N. F. (2010). A research impact indicator for institutions. Journal of Informetrics, 4(4), 581-590. doi:10.1016/j.joi.2010.06.006
2 Abramo, G., D’Angelo, C. A., & Viel, F. (2013). The suitability of h and g indexes for measuring the research performance of institutions. Scientometrics, 97(3), 555-570. doi:10.1007/s11192-013-1026-4
3 Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Leiden Manifesto for research metrics. Nature, 520, 429-431.
Citation-tracking databases calculate bibliometric data based on the items they index, and each database has differing coverage and a unique data universe. This means:
A limitation of citation-tracking databases is the different ways authorship is attributed. Problems can stem from data errors, name ambiguity, and how multi-authored articles are attributed.
Citation-tracking databases are also susceptible to gender bias, which limits the reliability and utility of citation-based measures. Evidence shows that:
1 Larivière, Ni, Gingras, Cronin & Sugimoto, 2013
2 Ferber & Brun, 2011
3 Maliniak, Powers, and Walter, 2013
4 Sarsons, 2015
Time affects both a researcher's impact and how their impact is understood.
1 Council of Canadian Academies. Expert Panel on Science Performance and Research Funding (Ed.). (2012). Informing research choices: Indicators and judgement. Ottawa: Council of Canadian Academies.
2 Wang, J. (2013). Citation time window choice for research impact evaluation. Scientometrics, 94(3), 851-872. doi:10.1007/ s11192-012-0775-9