* (Please note – this title is a quote from the 1967 British TV Series, The Prisoner. There are recognised issues of gender bias in citation and authorship across academia – this title was not intended to reflect that (and yes, the author was male). Please consider yourself equally free to be treated as a number rather than a person – whatever your gender identity – but recognise that those numbers may reflect bias in the practice of authors and reviewers) [ed: 17th July 2019]
Who is citing who?
We often (well, sometimes) get asked by students:-
– How do I know who has cited this work? (How do I do this?)
We more frequently get asked a similar question by our academic colleagues:-
– How do I know who has cited MY work? (How do I do this?)
Is BIGGER always BETTER?
Why would you not want to know who has been citing your research? It may just be to massage your ego, or it might offer an opportunity to re-evaluate your own work in the new light shed by others. It could offer an opportunity for a future collaboration, or a conversation starter with a citing author at an upcoming conference. Sometimes it is just nice to have that (often fleeting) sensation of finally having your value recognised by someone. Or often, sadly, being able to show how often you have been cited is the game you are forced to play for that next academic job application or promotion review.
When it comes to that last reason, the assumption is often that “bigger is better”. Whilst this may often be true, there is a lot of nuance to that question.. not least what might be understood to be “big” from one discipline to another. But, casting your eyes back to the title of this post, do you want ‘quality’ to be measured by a number? The answer to that question might be influenced by whether you’re a STEM or humanities scholar… or just whether you’re the person sitting on an interview panel with a long-list of over 500 applications to get through in far too little time.
But I look much better in Google Scholar!
One thing is common. Most academic authors we speak to will already know, and may prefer, their citation counts listed in Google Scholar compared to their citation counts as listed in Scopus or Web of Science. Why? Because they are BIGGER!
A recent discussion we had with a colleague was around why universities might look at citation data in Scopus or Web of Science, as opposed to that in Google Scholar. The specific example used as an authors publication which listed 33 citations in Google Scholar, compared to only 18 in Scopus. Thirty-three is better than 18, right?
The answer in this instance was… it depends what you are looking for.
Here is the summary of the citations to the article (Ross, Polson & Grosbas (2012); https://doi.org/10.1371/journal.pone.0044815) (open access version) found in Scopus (and Web of Science) and Google Scholar:
| Citations in Scopus | Citations in Web of Science | Citations in Google Scholar | |
| ARTICLE | 16 | 15 | 18 |
| BOOK | 1* | 1* | 1 |
| BOOK CHAPTER | 1 | 1 | 3 |
| DUPLICATE CITATION | 0 | 0 | 1 |
| PREPRINT | 0 | 0 | 1 |
| THESIS / DISSERTATION | 0 | 0 | 9 |
| [Data as at 15/07/19] | 18 | 17 | 33 |
[*Citations from Scopus and Web of Science which were not found in Google Scholar]
As can be seen, the number of citing journal articles are roughly similar. Of the two articles not found in Scopus but found in Google Scholar, one article is likely to appear in Scopus eventually (it just hasn’t been indexed yet) and the other was a non-English language article (language bias in citation patterns, research assessment and scholarly publishing is a whole different discussion).
But (in this unscientifically small-sized sample) the citation inflation seen is primarily seen in the types of citing publications indexed
- theses and pre-prints, which aren’t picked up in Scopus/Web of Science.
- books and book chapters, which are but not as comprehensively.
There was also an example of a ‘duplicate citation’ – the same publication counted twice by Google Scholar – as two separate citing publications.
Out of interest, we looked at two other articles authored by Durham authors (a selection made at random, from articles which appeared in all three sources and which had accrued c.15-40 citations – a manageable number to manually review);
| In Scopus | In Web of Science | In Google Scholar | |
| ARTICLE | 21 | 21 | 19 |
| DUPLICATE CITATION | 0 | 0 | 2 |
| PREPRINT | 0 | 0 | 3 |
| THESIS/DISSERTATION | 0 | 0 | 2 |
| [Data as at 15/07/19] | 21 | 21 | 26 |
[Horner, Miller, Steed & Sutcliffe (2016): https://doi.org/10.1039/C6CS00448B
(open access version)]
| In Scopus | In Web of Science | In Google Scholar | |
| ARTICLE | 19 | 19 | 20 |
| BLOG POSTS | 0 | 0 | 2 |
| BOOK | 1 | 1 | 0 |
| CONFERENCE PAPER | 1 | 1 | 1 |
| DUPLICATE CITATION | 0 | 0 | 3 |
| THESIS/DISSERTATION | 0 | 0 | 9 |
| WORKING PAPER | 0 | 0 | 1 |
| [Data as at 15/07/19] | 21 | 21 | 36 |
[Wiedemann, Burt, Hill & Barton (2015): https://doi.org/10.1098/rsbl.2015.0166 (open access version)]
The above three articles citation data can be summarised in the following figure:
Is this citation better than that citation?
From this (admittedly) very small sample, one apparent observation is that whilst the coverage of citations from journal articles is similar, where Google Scholar offers value is the breadth of coverage from other, non-journal formats: in particular theses and dissertations.
The next question is whether a citation in a Masters dissertation or a doctoral thesis holds the same perceived “value” as a citation from a journal published, peer-reviewed journal article? The answer of course, is “it depends..”
For an author, knowing who has cited your work and where is hopefully pleasing no matter the source of the citation. A citation from a peer-reviewed and published article or book chapter may however offer more value in terms of opportunities for collaboration or a response, than inclusion in a student’s dissertation bibliography.
The other element is that in terms of benchmarking, a clearly defined dataset from which citation data is derived is often preferable – one reason why citation data from Scopus and Web of Science is used as part of various University Ranking and research assessment exercises. It is also an important factor in Durham University’s approach to Responsible Metrics – offering transparency on which data source for any use of citation or publication metrics is clearly communicated.
(See also how journal rankings such as Citescore, SNIP or SJR (from Scopus citation data) or Journal Impact Factor or Eigenfactor (published in the annual JCRs from Web of Science data)
BIGGER is BETTER!!!
But one area where size does matter is sample size if you are to try and draw any meaningful conclusions? Interested to read more? The study published in the Journal of Informetrics in October 2018 may be of interest:
- Alberto Martín-Martín, Enrique Orduna-Malea, Mike Thelwall, Emilio Delgado López-Cózar (2018) ‘Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories’ Journal of Informetrics 12(4) 1160-1177
- https://doi.org/10.1016/j.joi.2018.09.002; Published version [subscription required]
- https://arxiv.org/abs/1808.05053; Updated version [open access – CC-BY-NC-ND]
- https://osf.io/preprints/socarxiv/pqr53/; Supplementary data [open access]
Key conclusions:
- Study looked at 2,448,055 citations to 2,299 English Language documents across 252 Google Scholar subject categories, comparing the citation data provided in Google Scholar, Scopus and Web of Science.
- The study found that for outputs across all subject categories, each service found citations to outputs as follows:
- Google Scholar found 93-96% of citing outputs
- Scopus found 35-77% of citing outputs (GS found 92% of Scopus citations)
- Web of Science found 27-73% of citing outputs (GS found 95% of Web of Science citations)
- 48-65% of the citations found only by Google Scholar were non-journal sources (theses, books, conference papers and unpublished materials).
RSS - Posts
Leave a Reply