Skip to main content

Course Research Impacts: Next generation metrics?

Next generation metrics?

The measurement of research impacts is a complicated business!
In this course the focus lies on the three most commonly used bibliometric tools for the measurements and evaluation of the impact of scholarly publications: via Web of Science, Scopus and Google Scholar / Publish or Perish. Whatever your opinion on the (mis)use of these measurements: because research metrics are not going away, it’s smart to know how these work! As already mentioned in the introduction: it’s important for researchers (and research assessors) to know how they are calculated, the contexts in which they can and can’t be used, which ones can be compared and which definitely can’t and ultimately how you can influence them yourself.
That is the main goal of this course: to get to know the indicators used and its possible pitfalls.

In this part of the course we will:

  • elaborate some more on the limitations of indicators for social sciences and humanities, and
  • list some alternative metrics.

For more background information: have a look at Maximizing the impacts of your research: a handbook for social scientists. London: London School of Economics and Political Science, Public Policy Group, 2011. Also to be found on their website.

Limits of bibliometrical analysis for social sciences and humanities

Many activities and outcomes or dimensions of research can be counted. As we have seen, the most basic and common way is counting the number of scholarly (journal) publications. Tracking citations and understanding their trends in contexts is a key to evaluation impacts and influences of research.

But there are limits. Basically these limits concern differences in publication and citation characteristics between disciplines or subject fields:

  1. less publications in journals and serials for social sciences and humanities and more in books;
  2. the lower coverage of social science and humanities journals compared to for instance biomedical sciences and (in WoS) the almost non-coverage of non-English monographs, reports and juridical comments and annotations;
  3. a different pace of theoretical development and hence the social sciences and humanities literature’s slow ageing rate, and conversely its post-publication citation rate. The rate at which scholarly literature ages and the rapidity with which it is cited have important implications for the way in which impact must be measured in different academic fields. See f.i. the example in the introduction of this course in the chapter 'Differences between disciplines';
  4. the more pronounced national and regional orientation. The target readership may be limited to a country or region. Scholars from these discipline reportedly publish more often in their mother tongue and in journals with a limited distribution. A greater share of publications is directed at a non-scholarly public;
  5. single scholar approach versus team research: characteristically, individual scholars working largely on their own, are engaged in publishing extensive monographs and/or single-authored articles.

There are no all-purpose indicators.

No bibliographic indicator should be put to isolated use. Several indicators should always be combined to achieve a more comprehensive picture. Deciding which indicators to use for what purpose, that's the question!

Research performance can be assessed along a number of different dimensions. See for instance this multidimensional matrix, taken from Research Trends issue 23, May 2011, based on the report published in 2010 by an Expert Group on the Assessment of University-Based Research (AUBR), installed by the European Commission:

Dimensions of research assessment

Alternative metrics

In this course our focus lies on the most commonly used methods of impact measurement via Web of Science, Scopus and Google Scholar / Publish or Perish.

Alternative methods of impact measurement are being developed, making use of the possibilities the internet - especially the Social Web - has to offer, like taking into account the number of downloads of articles, the number of times a paper was discusses on Twitter or blogs and the number of times an articles was saved in for example Mendeley.

You can read more about these methods in a report commissioned by the SURFfoundation focused on new ways of measuring the quality and impact of scientific output: 'Users, narcissism and control – tracking the impact of scholarly publications in the 21st century'. One of the main conclusions of this report is that these alternatives can't legitimately be used in research assessments yet, because they don't comply with more strict quality criteria.

In the PLOS ONE Altmetrics Collection you can find research on the study and use of altmetrics, including validation.

Selection of (free) alternative metrics or alt-metrics:

Article level metrics

The PLoS-journals show 'Article-Level Metrics Information', including the number of citations in Web of Science, Scopus and PubMed Central, but also blog coverage and bookmarks. More information is available here. This example shows the Article-Level Metrics of an article in PLoS Medicine.

Article level metrics in PLOS Medicine