Assessing the Assessment Evaluation Reports: Are They Setting the Example?

Reflections logo

As I write this the Higher Education Funding Council for England (HEFCE) is hosting an invitation-only event titled “REFlections: Evaluation of the Research Excellence Framework (REF) 2014 and a look to the future“.

At the time of writing this line my #REFlections archive has collected more than 1,100 Tweets published today. (I’ll share the archive later after the event).

This is a quick note to refer to two of the reports shared today:

These two reports are available online for free. However, it is no small detail, particularly given both the general context and specific topic of these reports, that none of the reports are available with an open license (I don’t mean CC-BY here, but any license at all).

Both reports indicate they are © Copyright HEFCE 2015 and © HEFCE 2015, which is itself as we know not in contradiction with open licensing (open licenses complement copyright).  However, page 2 of the RAND Corporation report also indicates clearly:

Manville et al, 2015, page 2
“All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the sponsor.”

[I am aware I am most likely infringing copyright law by reproducing this copyright notice here. I invoke fair dealing for educational use].

Unless I totally missed it, the Digital Science and King’s report file does not contain any licensing information telling the reader/user under what conditions the report can be reproduced or re-used or if it can indeed be adapted or enhanced in any way under attribution or any other conditions without having to request previous permission (which takes time, which means resources, which means money).

According to the Open Knowledge Foundation’s Open Definition, the first requirement for a work to qualify as “open” is that

“[t]he work must be available under an open license […]. Any additional terms accompanying the work (such as a terms of use, or patents held by the licensor) must not contradict the terms of the license.”

[See definition of “Open License” in 2.2. in].

Some authors consider this definition of open licensing too permissive, hence unappealing to academic authors or organisations that may have reasons to restrict the conditions under which they publish their work. However, the reports mentioned above do not provide any licensing indication, apart from the copyright notice, and in the case of the RAND report a very clear All Rights Reserved notice.

Open Access and Open Licensing are of course related. The relationship is the object of a long discusssion and it has taken place elsewhere. In this case the reports in question refer to a research assessment exercise that had the Open Access requirement at its core. HEFCE’s “Policy for open access in the post-2014 Research Excellence Framework” indicates that the open access requirement

“applies only to journal articles and conference proceedings with an International Standard Serial Number. It will not apply to monographs, book chapters, other long-form publications, working papers, creative or practice-based research outputs, or data. The policy applies to research outputs accepted for publication after 1 April 2016, but we would strongly urge institutions to implement it now.”

It is clear that the HEFCE policy cannot be applied to the reports mentioned above. They are not journal articles or conference proceedings with an ISSN.

However, I’d like to suggest that in order to engage fully in a transition towards open access to research data and information all stakeholders would need to adopt good practices in open sharing themselves.

The lack of licensing potentially limits the reach and (ironically in this case) the impact of these reports. Certainly users with an awareness of open access, open data and open licensing will notice the lack of open licensing in these reports, and find in this a limitation if not an obstacle.

Since both reports, as far as I understand, were partially or fully funded by a public body, and hence by the taxpayer, and since both reports are available for no economic cost online, noting here that citizens should have been provided with clear open licensing indicating what they may or may not do with the reports seems to me fair.

How can we demand the adoption of open practices if outputs assessing assessment mechanisms (there’s meta for you) based on the mandate to share openly do not adopt open licensing?

The publication and availability of these reports is welcome and worthy of celebration. The fact one of them explicitly forbids any reproduction without previous permission and that none of them contain licensing information is a disappointment.

[Post published 13:35 PM GMT]

14:00 PM GMT Update

This just in:

A #HEFCEmetrics Twitter Archive (Friday 16 January 2015, Warwick)

HEFCE logo

The HEFCE metrics workshop: metrics and the assessment of research quality and impact in the arts and humanities took place on Friday 16 January 2015, 1030 to 1630 GMT at the Scarman Conference Centre, University of Warwick, UK.

I have uploaded a dataset of 821 Tweets tagged with #HEFCEmetrics (case not sensitive):

Priego, Ernesto (2015): A #HEFCEmetrics Twitter Archive (Friday 16 January 2015, Warwick). figshare.

TheTweets in the dataset were publicly published and tagged with #HEFCEmetrics between 16/01/2015 00:35:08 GMT and 16/01/2015 23:19:33 GMT. The collection period corresponds to the day the workshop took place in real time.

The Tweets contained in the file were collected using Martin Hawksey’s TAGS 6.0. The file contains 2 sheets.

Only users with at least 2 followers were included in the archive. Retweets have been included. An initial automatic deduplication was performed but data might require further deduplication.

Please note the data in this file is likely to require further refining and even deduplication. The data is shared as is. The contents of each Tweet are responsibility of the original authors. This dataset is shared to encourage open research into scholarly activity on Twitter. If you use or refer to this data in any way please cite and link back using the citation information above.

For the #HEFCEmetrics Twitter archive corresponding to the one-day workshop hosted by the University of Sussex on Tuesday 7 October 2014, please go to

Priego, Ernesto (2014): A #HEFCEmetrics Twitter Archive. figshare.

You might also be interested in

Priego, Ernesto (2014): The Twelve Days of REF- A #REF2014 Archive. figshare.

The Twelve Days of REF: A #REF2014 Archive

Cirrus word cloud visualisation of a corpus of 23,791 #REF2014 Tweets

I have uploaded a new dataset to figshare:

Priego, Ernesto (2014): The Twelve Days of REF- A #REF2014 Archive. figshare.

The file contains approximately 31,855 unique Tweets published publicly and tagged with #REF2014 during a 12-day period between 08/12/2014 11:18 and 20/12/2014 10:13 GMT.

For some context and an initial partial analysis, please see my previous blog post from 18 December 2014.

As always, this dataset is shared to encourage open research into scholarly activity on Twitter. If you use or refer to this data in any way please cite and link back using the citation information above.

Happy Christmas everybody.

The REF According to Twitter: A #REF2014 Update (18/12/14 16:28 GMT)

As everyone in some way aware of UK higher education knows, the results from the REF 2014 were announced in the first minute of the 18th of december 2014. Two main hashtags have been used to refer to it on Twitter; #REF and the more popular (“official”?) #REF2014.

There’s been of course other variations of these hashtags, including discussion about it not ‘hashing’ the term REF at all. Here I share a quick first look at a sample corpus of  texts from Tweets publicly tagged with #REF2014.

This is just a quick update of a work in progress. No qualitative conclusions are offered, and the quantitative data shared and analysed is provisional. Complete data sets will be published openly once the collection has been completed and the data has been further refined.

The Numbers

I looked at a sample corpus of 23,791 #REF2014 Tweets published by 10,654 unique users between 08/12/2014 11:18 GMT and 18/12/2014 16:32 GMT.

  • The sample corpus only included Tweets from users with a minimum of two followers.
  • The sample corpus consists of 1 document with a total of 454,425 words and 16,968 unique words.
  • The range of Tweets per user varied between 70 and 1, with the average being 2.3 Tweets per user.
  • Only 8 of the total of 10,654 unique users in the corpus published between 50 and 80 Tweets; 30 users published more than 30 Tweets, with 9,473 users publishing between 1 and 5 Tweets only.
  • 6,585 users in the corpus published one Tweet only.

A Quick Text Analysis

Voyant Tools was used to analyse the corpus of 23,791 Tweet texts. A customised English stop words list was applied globally. The most frequent word was “research”, repeated 8,760 times in the corpus; it was included in the stop-word list (as well as, logically, #REF2014).

A word cloud of the whole corpus using the Voyant Cirrus tool looked like this (you can click on the image to enlarge it):

Cirrus word cloud visualisation of a corpus of 23,791 #REF2014 Tweets

#REF2014  Top 50 Most frequent words so far

Word Count
uk 4605
results 4558
top 2784
impact 2091
university 1940
@timeshighered 1790
ranked 1777
world-leading 1314
excellence 1302
universities 1067
world 1040
quality 1012
internationally 933
excellent 931
overall 910
great 827
staff 827
academics 811
proud 794
congratulations 690
rated 690
power 666
@cardiffuni 653
oxford 645
leading 641
best 629
news 616
education 567
5th 561
@gdnhighered 556
@phil_baty 548
ucl 546
number 545
law 544
today 536
table 513
analysis 486
work 482
higher 470
uni 460
result 453
time 447
day 446
cambridge 430
just 428
@ref2014official 427
group 422
science 421
big 420
delighted 410


The map is not the territory. Please note that both research and experience show that the Twitter search API isn’t 100% reliable. Large tweet volumes affect the search collection process. The API might “over-represent the more central users”, not offering “an accurate picture of peripheral activity” (González-Bailón, Sandra, et al. 2012). It is not guaranteed this file contains each and every Tweet tagged with the archived hashtag during the indicated period. Further dedpulication of the dataset will be required to validate this initial look at the data, and it is shared now merely as an update of a work in progress.


Gonzalez-Bailon, Sandra and Wang, Ning and Rivero, Alejandro and Borge-Holthoefer, Javier and Moreno, Yamir, “Assessing the Bias in Samples of Large Online Networks” (December 4, 2012). Forthcoming in Social Networks. Available at SSRN: or

HEFCE Metrics: A one-day workshop hosted by the University of Warwick

University of Warwick Faculty of Arts banner

Metrics and the assessment of research quality and impact in the Arts and Humanities

A one-day workshop hosted by the University of Warwick, as part of the Independent Review of the Role of Metrics in Research Assessment.

Date: Friday 16th January 2015 (10:30 to 16:30)

Location: Scarman Conference Centre, University of Warwick

The workshop will have the following objectives:

1. Offering a clear overview of the progress to date in the development of metrics of relevance to arts and humanities to date and persisting challenges.

2. Exploring the potential benefits and drawbacks of metrics use in research assessment and management from the perspective of disciplines within the arts and humanities.

3. Generating evidence, insights and concrete recommendations that can inform the final report of the independent metrics review.

The workshop will be attended by several members of the metrics review steering group, academics and stakeholders drawn from across the wider HE and research community.

Confirmed speakers include:

  • Prof. Jonathan Adams, King’s College London
  • Prof. Geoffrey Crossick, AHRC Cultural Value Project and Crafts Council
  • Prof. Maria Delgado, Queen Mary, University of London
  • Dr Clare Donovan, Brunel University
  • Dr Martin Eve, University of Lincoln and Open Library of Humanities
  • Prof. Mark Llewellyn, Director of Research, AHRC
  • Dr Alis Oancea, University of Oxford
  • Dr Ernesto Priego, City University London
  • Prof. Mike Thelwall, University of Wolverhampton (member of the HEFCE review steering group)
  • Prof. Evelyn Welch, King’s College London

Please register here.

A #HEFCEmetrics Twitter Archive

#hefcemetrics top tweeters

I have uploaded a new dataset to figshare:
Priego, Ernesto (2014): A #HEFCEmetrics Twitter Archive. figshare.

“In metrics we trust? Prospects & pitfalls of new research metrics” was a one-day workshop hosted by the University of Sussex, as part of the Independent Review of the Role of Metrics in Research Assessment. It took place on Tuesday 7 October 2014 at the Terrace Room, Conference Centre, Bramber House, University of Sussex, UK.

The file contains a dataset of 1178 Tweets tagged with #HEFCEmetrics (case not sensitive). These Tweets were published publicly and tagged with #HEFCEmetrics between 02/10/2014 10:18 and 08/10/2014 00:27 GMT.

The Tweets contained in the file were collected using Martin Hawksey’s TAGS 6.0. The file contains 3 sheets.

Please note the data in this file is likely to require further refining and even deduplication. The data is shared as is. The contents of each Tweet are responsibility of the original authors. This dataset is shared to encourage open research into scholarly activity on Twitter.

For more information refer to the upload itself.

If you use or refer to this data in any way please cite and link back using the citation information above.

On Metrics and Research Assessment

Oh research where art thou?

Next Monday 30th June 2014 at noon is the deadline to reply to the ‘Call for Evidence’ for HEFCE’s Independent review of the role of metrics in research assessment. I share some quick notes on my position as an individual researcher. Needless to say my personal position is based on my experience as a researcher and as the result of reading research in the area. Please excuse the lack of hyperlinked references in the body of the text; however  I have included a bibliography at the end of the post where I link to each reference.

A combination of traditional citation metrics and ‘alternative’ article-level metrics can be used across different academic disciplines to assess the reach (in terms of academic and public ‘impact’) of excellent research undertaken in the higher education sector (Liu and Adie 2013).

If increased international public access and impact are to be key factors in 21st century research assessment, the adoption of metrics, and particularly article-level metrics, is essential. Scholarly outputs published with Digital Object Identifiers can be easily tracked and measured, and as scholars in different fields adopt online methods of dissemination more widely, the data we can obtain from tracking it should not be ignored by assessment panels.

Article-level-metrics on scholarly outputs are already being tested by institutional repositories and publishers across the board. The data is open and facilitates further research, and some evidence for qualitative impact storytelling. Merely on their own, metrics of any kind (understood as mostly quantitative data) cannot and should not be used to assess either impact or ‘excellence’.

However, citation metrics and online mention metrics (“altmetrics”) can provide valuable data that can and should be subject to quantitative and qualitative analyses and review. Qualitative assessment in the form of “impact stories” can be informed by quantitative data provided by alternative metrics providers and methodologies (Neylon 2010; Priego 2012).

The San Francisco Declaration on Research Assessment (DORA) made the general recommendation of not using journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual researcher’s contributions, or in hiring, promotion, or funding decisions.

DORA provides recommendations for institutions, funding agencies, publishers and organisations that supply metrics.  An analysis of available data on individual DORA signers as of June 24, 2013, showed that 10963 individuals and 484 institutions had signed and that 6% were in the humanities and 94% in scientific disciplines; this in itself reflects an important disparity across fields that should be taken into account.

The ‘gaming’ of any kind of metric is possible by definition. It is critical that previous efforts in developing good practice in the measurement and assessment of research are adopted or at least taken into account. DORA makes it explicit that the gaming of metrics will not be tolerated and altmetrics service providers are openly working towards good practice and transparent methodologies (Adie 2013).

Social media adoption by scholars for scholarly dissemination is an important aspect of academic communications. It is wide, varies across disciplines and is still fairly recent (Priem 2011: Adie and Roe 2013;  Sud and Thelwall 2013). Therefore the discovery of correlations between online mentions, downloads and traditional citations is expected to be low since the citation window is still too small. Previous research however demonstrates there are positive yet still low correlations between downloads and citation counts.

Recent and ongoing research shows that Open Access publications can lead to greater number of downloads and social media mentions. Though research looking for possible correlations between Open Access and citation counts exists, the findings vary and the citation window is still too small and more time and research will be needed to determine if positive correlations exist as a general rule (Alperin 2014; Bernal 2013; Costas 2014; Kousha & Thelwall 2007).

It could be predicted that it is likely there will be positive correlations in some cases but not all, as the scholarly, institutional, technological, economic and social variables are multiple and platform and culture-dependent. Likewise, current business models from so-called hybrid publishers that enable Open Access via Article Processing Charges are likely to privilege the dissemination of outputs of those with existing funding schemes to cover them. Similarly a prevalence of academic journals, particularly in the arts and humanities have yet to have a significant, sustainable online presence, and many still lack DOIs to enable their automated and transparent tracking. However, institutional repositories are already embracing altmetrics as a means of both tracking and encouraging engagement with the resources, and the ability to track and measure engagement with grey literature can be a good source of evidence of the role these outputs play in the research and publication life-cycle.

Moreover, some fields privilege the publication of multi-author outputs whilst others prefer single author publications. This clearly puts both those without Open Access funding and single author papers at a quantitative disadvantage. As stated above it is crucial that research assessment employing metrics is based on  qualitative analyses and takes differences in disciplinary cultures into account. Research assessment employing metrics should be conducted on a case-by-case basis even if it is difficult, time-consuming and/or costly.

It is also critical that any assessment of article-level metrics understands how these metrics are possible in the first place and  has an informed awareness of the disparities in social media adoption for scholarly purposes across different disciplinary boundaries in the Higher Education sector. Direct experience and ongoing research shows evidence that at the moment some STEM fields are over-represented online (on blogs, social media and Open Access journals and monographs) while social sciences, arts and humanities outputs are lagging behind.

Traditional citation metrics unfairly benefit those publishing in standard channels and particularly those in the Global North, leaving developing countries scholars at a disadvantage (Alperin 2013; Priego 2014). Alternative metrics more accurately measure the wider reach of scholarly outputs, and might better serve most scholars fostering a research culture that supports national and international research impact objectives.

Even though there is still a bias towards North American and European publications, altmetrics can provide advantages to scholars interested in promoting their research online internationally by addressing public needs and enabling easier discovery and access to research outputs long underrepresented in the traditional literature and databases (Alperin 2014). Moreover, the geolocation data obtainable through altmetrics services offers evidence of both the disparities and international reach of both the production and consumption of research online.

In the internaitonal context some recent and ongoing research suggests that Open Access publications tracked via article-level metrics have a wider international reach and impact; there is a growing body of evidence this is the case in both Latin America and some regions in Africa (see the OpenUCT/ Scholarly Communication in Africa Programme (SCAP) reports as well as Alperin 2014; Priego 2013, 2014; Neylon, Willmers & King 2014).

The success of automated methods to obtain quantitative indicators of the reach, reception and use of scholarly outputs depends on our ability as scholarly communities to realise and develop the potential of the Web for scholarly communications. Developers, adopters and advocates of the use of article-level metrics do not claim that quantitative indicators should be taken at face value. Online publishing offers the unique opportunity to track, measure and evaluate what happens to scholarly outputs once they have been published on the Web. They allow us to make comparisons between dissemination and access models across countries and disciplinary boundaries. More importantly the data they provide is not static, passive quantitative data, but ‘interactive’ as they work as platforms for social interactions between researchers (potentially worldwide, where conditions allow it) enabling the easier, faster discoverability, collecting, exchange and discussion of outputs.

Not embracing article-level metrics or alternative metrics/ altmetrics in research assessment when the 21st century is well underway would be a missed opportunity to push towards a scholarly culture of wider public engagement and adoption of innovative online platforms for scholarly dissemination.

Adopting purely quantitative methods, and even more suggesting that any metric, however large, can equate to “excellence” would be misguided and potentially catastrophic, particularly for those not in STEM areas or without the backing of elite institutions. Only the careful, professional qualitative assessment of live, transparent publishing data will be able to provide evidence of the public and scholarly, local and international reach and reception of excellent research.


Adie, E., & Roe, W. (2013). Enriching scholarly content with article-level discussion and metrics. Learned Publishing, 26(1), 11–17. doi:10.6084/m9.figshare.105851

Adie, E. (2013). Gaming Altmetrics. Altmetric. September 18 2013. Available from

Alperin, J. P. (2013). Ask not what altmetrics can do for you, but what altmetrics can do for developing countries. Bulletin of the American Society for Information Science and Technology, 39(4), 18–21. doi:10.1002/bult.2013.1720390407

Alperin, Juan Pablo (2014): Exploring altmetrics in an emerging country context. figshare.

Bernal, I. (2013). Open Access and the Changing Landscape of Research Impact Indicators: New Roles for Repositories. Publications, 1(2), 56–77. Retrieved from

Costas, R., Zahedi, Z., & Wouters, P. (2014). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective (p. 30). Leiden. Retrieved from

Konkiel, S. (2013, November 5). Altmetrics in Institutional Repositories. Retrieved from

Kousha, K., & Thelwall, M. (2007). The Web impact of open access social science research. Library & Information Science Research, 29(4), 495–507. Retrieved from

Liu, J., & Adie, E. (2013, May 30). Altmetric: Getting Started with Article-Level Metrics. figshare.

Mohammadi, E., & Thelwall, M. (2014). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the American Society for Information Science and Technology. Retrieved from

Neylon, C. (2011). Re-use as Impact: How re-assessing what we mean by “impact” can support improving the return on public investment, develop open research practice, and widen engagement . Altmetrics. Retrieved from

Neylon, C. (2010). Beyond the Impact Factor: Building a community for more diverse measurement of research. Science in the Open. Retrieved November 29, 2010, from

Neylon C, Willmers M and King T (2014). Rethinking Impact: Applying Altmetrics to Southern African Research. Working Paper 1, Scholarly Communication in Africa Programme.

OpenUCT Initiative Publications and SCAP reports. Available from

Priego, E. (2012). Altmetrics’: quality of engagement matters as much as retweets. Guardian Higher Education Network, Friday 24 August 2012. Retrieved from

Priego, E. (2013). Fieldwork: Apples and Oranges? Online Mentions of Papers About the Humanities. Altmetric, January 11 2013. Retrieved from

Priego, E. (2013). Alt-metrics, Digital Opportunity and Africa. Impact of Social Sciences, London School of Economics. February 6 2013. Retrieved from

Priego, E. (2014). The Triple A: Africa, Access, Altmetrics. 22 February 2014. Retrieved from

Priem, J., Hall, M., Hill, C., Piwowar, H., & Waagmeester, A. (2011). Uncovering impacts : CitedIn and total-impact , two new tools for gathering altmetrics . iConference 2012, 9–11. Retrieved from

Priem, J., Piwowar, H. A., & Hemminger, B. H. (n.d.). Altmetrics in the wild: An exploratory study of impact metrics based on social media. Metrics 2011: Symposium on Informetric and Scientometric Research. New Orleans, LA, USA. Retrieved from

Sud, P., & Thelwall, M. (2013). Evaluating altmetrics. Scientometrics. Retrieved from