The measurement of research impacts is a complicated business!
In this course the focus lies on the three most commonly used bibliometric tools for the measurements and evaluation of the impact of scholarly publications: via Web of Science, Scopus and Google Scholar / Publish or Perish. Whatever your opinion on the (mis)use of these measurements: because research metrics are not going away, it’s smart to know how these work! As already mentioned in the introduction: it’s important for researchers (and research assessors) to know how they are calculated, the contexts in which they can and can’t be used, which ones can be compared and which definitely can’t and ultimately how you can influence them yourself.
That is the main goal of this course: to get to know the indicators used and its possible pitfalls.
In this part of the course we will:
For more background information: have a look at Maximizing the impacts of your research: a handbook for social scientists. London: London School of Economics and Political Science, Public Policy Group, 2011. Also to be found on their website.
Many activities and outcomes or dimensions of research can be counted. As we have seen, the most basic and common way is counting the number of scholarly (journal) publications. Tracking citations and understanding their trends in contexts is a key to evaluation impacts and influences of research.
But there are limits. Basically these limits concern differences in publication and citation characteristics between disciplines or subject fields:
There are no all-purpose indicators.
No bibliographic indicator should be put to isolated use. Several indicators should always be combined to achieve a more comprehensive picture. Deciding which indicators to use for what purpose, that's the question!
Research performance can be assessed along a number of different dimensions. See for instance this multidimensional matrix, taken from Research Trends issue 23, May 2011, based on the report published in 2010 by an Expert Group on the Assessment of University-Based Research (AUBR), installed by the European Commission:
In this course our focus lies on the most commonly used methods of impact measurement via Web of Science, Scopus and Google Scholar / Publish or Perish.
Alternative methods of impact measurement are being developed, making use of the possibilities the internet - especially the Social Web - has to offer, like taking into account the number of downloads of articles, the number of times a paper was discusses on Twitter or blogs and the number of times an articles was saved in for example Mendeley.
You can read more about these methods in a report commissioned by the SURFfoundation focused on new ways of measuring the quality and impact of scientific output: 'Users, narcissism and control – tracking the impact of scholarly publications in the 21st century'. One of the main conclusions of this report is that these alternatives can't legitimately be used in research assessments yet, because they don't comply with more strict quality criteria.
In the PLOS ONE Altmetrics Collection you can find research on the study and use of altmetrics, including validation.
Selection of (free) alternative metrics or alt-metrics: