Metricating #respbib18 and #ResponsibleMetrics: A Comparison

I’m sharing summaries of Twitter numerical data from collecting the following bibliometrics event hashtags:

  • #respbib18 (Responsible use of Bibliometrics in Practice, London, 30 January 2018) and
  • #ResponsibleMetrics (The turning tide: A new culture of responsible metrics for research, London, 8 February 2018).

 

#respbib18 Summary

Event title Responsible use of Bibliometrics in Practice
Date 30-Jan-18
Times 9:00 am – 4:30 pm  GMT
Sheet ID RB
Hashtag #respbib18
Number of links 128
Number of RTs 100
Number of Tweets 360
Unique tweets 343
First Tweet in Archive 23/01/2018 11:44 GMT
Last Tweet in Archive 01/02/2018 16:17 GMT
In Reply Ids 15
In Reply @s 49
Unique usernames 54
Unique users who used tag only once 26 <–for context of engagement

Twitter Activity

#respbib18 twitter activity last three days
CC-BY. Originally published as https://twitter.com/ernestopriego/status/958424112547983363

 

#ResponsibleMetrics Summary

Event title The turning tide: A new culture of responsible metrics for research
Date 08-Feb-18
Times 09:30 – 16:00 GMT
Sheet ID RM
Hashtag #ResponsibleMetrics
Number of links 210
Number of RTs 318
Number of Tweets 796
Unique tweets 795
First Tweet in Archive 05/02/2018 09:31 GMT
Last Tweet in Archive 08/02/2018 16:25 GMT
In Reply Ids 43
In Reply @s 76
Unique usernames 163
Unique usernames who used tag only once 109 <–for context of engagement

Twitter Activity

#responsiblemetrics Twitter activity last three days
CC-BY. Originally published as https://twitter.com/ernestopriego/status/961639382150189058

#respbib18: 30 Most Frequent Terms

 

Term RawFrequency
metrics 141
responsible 89
bibliometrics 32
event 32
data 29
snowball 25
need 24
use 21
policy 18
today 18
looking 17
people 16
rankings 16
research 16
providers 15
forum 14
forward 14
just 14
practice 14
used 14
community 13
different 12
metric 12
point 12
using 12
available 11
know 11
says 11
talks 11
bibliometric 10

#ResponsibleMetrics: 30 Most Frequent Terms

Term RawFrequency
metrics 51
need 36
research 29
indicators 25
panel 16
responsible 15
best 13
different 13
good 13
use 13
index 12
lots 12
people 12
value 12
like 11
practice 11
context 10
linear 10
rankings 10
saying 10
used 10
way 10
bonkers 9
just 9
open 9
today 9
universities 9
coins 8
currency 8
data 8

Methods

Twitter data mined with Tweepy. For robustness and quick charts a parallel collection was done with TAGS. Data was checked and deduplicated with OpenRefine. Text analysis performed with Voyant Tools. Text was anonymised through stoplists; two stoplists were applied (one to each dataset), including usernames and Twitter-specific terms (such as RT, t.co, HTTPS, etc.), including terms in hashtags. Event title keywords were not included in stoplists.

No sensitive, personal nor personally-identifiable data is contained in this data. Any usernames and names of individuals were removed at data refining stage and again from text analysis results if any remained.

Please note that both datasets span different number of days of activity, as indicated in the summary tables. Source data was refined but duplications might have remained, which would logically affect the resulting term raw frequencies, therefore numbers should be interpreted as indicative only and not as exact measurements.  RTs count as Tweets and raw frequencies reflect the repetition of terms implicit in retweeting.

So?

As usual I share this hoping others might find interesting and draw their own conclusions.

A very general insight for me is that we need a wider group engaging with this discussions. At most we are talking about a group of approximately 50 individuals that actively engaged on Twitter on both events.

From the Activity charts it is noticeable that tweeting recedes at breakout times, possibly indicating that most tweeting activity is coming from within the room– when hashtags create wide engagement, activity is more constant and does not exactly reflect the timings of actual real-time activity in the room.

It seems to me that the production, requirement, use and interpretation of metrics for research assessment directly affects everyone in higher education, regardless of their position or role. The topic should not be obscure or limited to bibliometricians and RDM, Research and Enterprise or REF panel people.

Needless to say I do not think everyone ‘engaged’ with these events or topics is or should be actively using the hashtag on Twitter (i.e. we don’t know how many people followed on Twitter). An assumption here is that we cannot detect nor measure anything if there is not a signal– more folks elsewhere might be interested in these events but if they did not use the hashtag they were logically not detected here. That there is no signal measurable with the selected tools does not mean there is not a signal elsewhere, and I’d like this to be a comment on metrics for assessment as well.

In terms of frequent terms it remains apparent (as in other text analyses I have performed on academic Twitter hashtag archives) that frequently tweeted terms remain ‘neutral’ nouns, or adjectives if they are a keyword in the event’s title, subtitle or panel sessions (e.g. ‘responsible’). When a term like ‘snowball’ or ‘bonkers’ appears, it stands out. Due to the lack of more frequent modifiers, it remains hard to distant-read sentiment or critical stances, or even positions. Most frequent terms do come from RTs, not because of consensus in ‘original’ Tweets.

It seems that if we wanted to demonstrate the value added by live-tweeting or using an event’s hashtag remotely, quantifying (metricating?) the active users, tweets over time, days of activity and frequent words would not be the way to go for all events, particularly not for events with relatively low Twitter activity.

As we have seen, automated text analysis is more likely to reveal mostly-neutral keywords, rather than any divergence of opinion on or additions to the official discourse. We would have to look at those words less repeated, and perhaps to replies that did not use the hashtag, but this is not recommended as it would complicate things ethically: though it is generally accepted that RTs do not imply endorsement, less frequent terms in Tweets with the hashtag could single-out individuals, and if a hashtag was not included on a Tweet it should be interpreted the Tweet is not meant to be part of that public discussion/corpus.

 

 

 

 

A 2014 Numeralia

Here an attempt to visualise what I was up to in 2014 publishing, research and teaching engagement wise. I have focused first on how many blog posts I published on this blog per month, how many blog posts I edited and/or authored for the Comics Grid blog, how many outputs I shared on figshare and finally a general numeralia of some main categories of my 2014 activity.

This post is not meant to contribute to heighten already-pervasive anxieties of academic productivity (I’m fully aware most of this activity does not ‘count’ for many anyway), but merely as a humble, personal yet public exercise of reminding myself of the work I’ve done. You can click on the charts to enlarge them.

Happy new year everyone! See you in 2015!

Blog Posts per Month in 2014

Blog Posts per Month in 2014 comicsgrid

Figshare Uploads per Month in 2014

Ernesto Priego Selected Numeralia from 2014

 

My 2014 Blogging in Review

 

An 2014 annual report for this blog courtesy of WordPress.com. Click here to see the complete report.

Here’s an excerpt:

“The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 14,000 times in 2014. If it were a concert at Sydney Opera House, it would take about 5 sold-out performances for that many people to see it.”

Click here to see the complete report.

A #citylis 2014-2015 Term 1 Twitter Archive

#citylis logo

The taught component of Term 1 of the 2014-2015 academic year at the Library and Information Science scheme at City University London has finished today. #citylis is our hashtag and it is used by staff, students and members of the public.

Throughout the term I archived the Tweets tagged with #citylis and I have now uploaded to figshare a spreadsheet containing 4940 Tweets (there’s likely to be some duplicates there, and it includes retweets).

Priego, Ernesto (2014): A #citylis 2014-2015 Term 1 Twitter Archive. figshare.

http://dx.doi.org/10.6084/m9.figshare.1269285

Retrieved 18:14, Dec 12, 2014 (GMT)

All the usual information about collection methods, limitations etc. are included in the ReadMe sheet of the file.

The data is shared as is. This dataset is shared to encourage open research into scholarly activity on Twitter. If you use or refer to this data in any way please cite and link back using the citation information above.

#citylis term 1 twitter actitvity top tweeters

 

Today: Promoting Interdisciplinary Engagement in the Digital Humanities

DH AHRC logo

I am writing this on the train to Oxford, where today James Baker and I will lead a workshop on sharing data for researchers. We have some slides up on figshare here:

Baker, James; Priego, Ernesto (2014): Sharing Data… A Researcher Perspective. figshare.
http://dx.doi.org/10.6084/m9.figshare.1038375

Our session will take place around 11-12 BST but feel free to contribute to the etherpad if you are so inclined (link on slides above).

Our participation is part of the DH Crowdscribe Project‘s Promoting Interdisciplinary Engagement in the Digital Humanities AHRC Collaborative Skills series. The poster is here: http://goo.gl/tHhExE [PDF].

I am very much looking forward to this event. Happy Friday everyone.

Twitter for Engagement with Research. A Survey.

Question 1 answers, Twitter for engagement with research survey, Tuesday 3 December 2013 12:24pm GMT, Chart CC-BY Ernesto Priego

We’ve been conducting a quick (really!) survey on using Twitter for engagement with research.

If you have already responded, thank you. Thank you as well if you have shared it on Twitter.

We are interested in learning more about the ways in which Twitter users engage with academic research. This quick survey contains 15 simple questions and should be pretty quick and straight-forward to complete.

It is a requirement that those answering this survey have an active Twitter account, but we will not ask you to share your username.

We are committed to protecting your personal information and respecting your privacy. We do not ask for your Twitter username, name, email address, telephone numbers or address, but the demographic data we require means we will ask you about your country of residence, age, the name of the institution you work for and main academic area of interest.

In obtaining your cooperation to participate in the survey, we undertake not to mislead you in any way about the nature of the research we are conducting, the way in which the data is collected and the use that will be made of the survey results.

All of the information that you provide will be treated as confidential and will only be used for research purposes. Your comments will not be identified as belonging to you, instead they will be combined with those gathered from other survey participants, and will be analysed as part of a group. We do not use any of the information you provide for direct marketing or other non-research activities. We might use any findings from this survey in an open access research paper or blog post, along the whole dataset.

Your participation is voluntary.

To complete the survey, go here.

An external site for Digital Cultures

Digital Cultures This week I set up a public blog for the Digital Cultures module Lyn Robinson and I are leading (City’s official page for it is here).

Creating a public blog is a simple way of having an external site (apart from the internal Moodle for registered students) where there can be some public record of what we are doing and who else is participating. It also offers, potentially, a way of tracking potential interest from those outside our institution.  You can visit the site and see who else is on board here.

Like all blogs and all courses these are works in progress, so changes are expected.

In the Guardian: On Twitter Engagement

On page 33 of the Guardian's print edition, Tuesday 5 February 2013
On page 33 of the Guardian’s print edition. Education section, Tuesday 5 February 2013

 

Today Nancy Groves featured my post discussing engagement on Twitter in her write-up for the Education section of the Guardian’s print edition. You can read the full post on the Guardian Higher Education Network, here.

Twitter: Towards New, Fluid Rules of Engagement

“Duty Calls”, by Randall Munroe (XKCD)

It’s a conversation, not a lecture“, was the title I had given to this post of mine when I submitted it to the Guardian Higher Education Network back in September 2011. It seems like quite a long time ago now. (I must clarify I’ve always been wary of using the verb “revolutionise” lightly, especially when it comes to social media).

I still believe in the need to ensure that Twitter remains a public, potentially more democratic online space where the playing field is up to a certain extent levelled. Engagement and reciprocity are defining built-in features of the Twitter platform; the @ sign indicating location and response, invocation, evocation, address, acknowledgement and recognition.

Nevertheless, I believe it remains important to clarify what I meant by “a conversation.” I did not mean necessarily, always and at all times, a real-time, chat room-style, synchronic exchange of written messages.

Though I am an enthusiastic believer in the benefits of social media, and Twitter in particular, to foster and strengthen individual and collective scholarly interconnections (2010), I am also very much aware that Twitter, like other Internet-mediated forms of communication, presupposes a particular kind of individual setting, and this setting is defined by a series of variants (what we could call a “real life context”) that are not always visible to those we are interacting with online.

Immediate interaction is one structural element that defines Twitter as a medium and as a tool, but this does not mean that this same capability will erase the external demands that define our circumstances as human individuals working behind a computer or mobile device in real-time and place. Indeed, Twitter challenges in its very structure previous assumptions of hierarchical, vertical communication, and does facilitate and encourage interactions (@ replies, mentions, direct messages) between strangers that defy notions of proximity, intimacy, closeness and even politeness. Twitter deletes the concept of “cold-calling”, but it also demands of its users the common sense to recognise that because the network is all about creating interactions, the focus or driver of activity should not be the individual but the network itself.

This awareness of the immediacy and easiness of interactivity between strangers in a public platform should (ideally) demand acute inter-subjective sensitivity from its users: it might be a good time and place for you to engage in a long and complex dialogue (with the always-already embedded possibility of it getting naturally derailed as others are de facto invited to join in), but it is highly likely it will not be the right time and place for your interlocutor to engage with you.

The 140-character limit of tweets encourages direct, generalizing, simplistic statements that will often get on the nerves of many of those reading (needless to say not all tweets are always simplistic etc., but once again one person’s ‘THIS’ is someone else’s ‘FAIL’). This broadcasting of fragmentary glimpses into often very particular points of view can be interpreted/received as the imposition of a truth that other people don’t believe in, and this type of broadcasting often begs for its immediate interrogation or interpellations. It is very easy, very tempting to get trapped into vicious cycles of misunderstandings. (And yes, sometimes, wonderful coincidences can happen and fruitful, fun, valuable, insightful interactions can take place, especially, in my opinion, if they are brief!).

At the same time, 140 characters will never be enough, no matter how good we are at synthesising our arguments, to add proper nuance and context to arguments fragmented by world-limits and constant ‘interruptions’. In oral communication between two people sharing the same temporal and spatial context it can be hard to agree and to solve misunderstandings. It is easy to see how hard to agree and to avoid misunderstandings can be in written discussions bwtween various people tweeting disjointed elocutions both asynchronically and synchronically from different parts of the world in different time zones and going through often completely disparate situations.

So engaging in a “conversation” on Twitter does not have to mean “real-time chat”. It means being aware that there are others out there who are very different to me and who are going through very different circumstances. Many (especially famous academic Twitterers with lots of followers) tend to simply ignore all @ replies and mentions. Not engaging at all seems to be their solution to avoiding any problems, like headphone-wearing, mobile-reading commuters in a morning rush hour packed train. (I keep seeing similarities between public transportation in big cities and Twitter; I often think of the latter as becoming more a form of public transportation of content than of communication, but that argument is for another post).

I believe there is got to be an alternative to this. We need to develop new, fluid rules of engagement on Twitter, in which we can recognise each other, often directly, often indirectly, respectfully, even when we disagree. These rules need not and in my opinion should not be even written: they should be fluid and abstract; for many of us it’s just about adaptable common sense, but this kind of internationally/interculturally/interlinguistic/interdisciplinary/multitimezone common sense seems to be really scarce in practice, to be honest. Maybe sometimes it’s all about redefining our individual, subjective horizons of expectations. There can be engagement, recognition and reciprocity in forms that I/you do not necessarily recognise as such at first. I love Martin Weller’s notion of ‘shifted reciprocity’ to refer to the reciprocal, but not identical, engagement between different individuals online (The Digital Scholar, 2011).

Shifted reciprocity. This is the kind of conversation I talk about. Redefining the rules of engagement does not mean resorting to ignoring each other (not engaging at all), and engagement is not about expecting any given interlocutors to be always available to spend all waking hours engaged in disjointed exchanges that are very often likely not to take us anywhere. Those kinds of real-time conversations are better carried out elsewhere, to the conference, the workshop, the seminar, the chat room… or the pub.

PS. I’m aware your circumstances might be different and you might disagree or have a completely different experience of the situation I describe above.

At Guardian Higher Ed: ‘Altmetrics’: quality of engagement matters as much as retweets

Guardian Higher Education banner

Today’s been a busy day; today the Guardian Higher Education Network published a piece by me on qualitative sharing, here.

It’s not meant to be a negative critique of altmetrics or statistical cybermetrics (both projects/methodologies I admire), but a call to combine the quantitative and the qualitative when it comes to assessing the correlations between online metrics and the impact of academic outputs.