Do citations and readership identify seminal publications?
Abstract
Here, this work presents a new approach for analysing the ability of existing research metrics to identify research which has strongly influenced future developments. More specifically, we focus on the ability of citation counts and Mendeley reader counts to distinguish between publications regarded as seminal and publications regarded as literature reviews by field experts. The main motivation behind our research is to gain a better understanding of whether and how well the existing research metrics relate to research quality. For this experiment we have created a new dataset which we call TrueImpactDataset and which contains two types of publications, seminal papers and literature reviews. Using the dataset, we conduct a set of experiments to study how citation and reader counts perform in distinguishing these publication types, following the intuition that causing a change in a field signifies research quality. Finally, our research shows that citation counts work better than a random baseline (by a margin of 10%) in distinguishing important seminal research papers from literature reviews while Mendeley reader counts do not work better than the baseline.
- Authors:
-
- Open Univ., Milton Keynes (United Kingdom); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Open Univ., Milton Keynes (United Kingdom)
- Publication Date:
- Research Org.:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1425336
- Grant/Contract Number:
- AC05-00OR22725
- Resource Type:
- Journal Article: Accepted Manuscript
- Journal Name:
- Scientometrics
- Additional Journal Information:
- Journal Volume: 115; Journal Issue: 1; Journal ID: ISSN 0138-9130
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 99 GENERAL AND MISCELLANEOUS; Information retrieval; Scholarly communication; Publication datasets; Data mining; Research evaluation; Bibliometrics; Altmetrics
Citation Formats
Herrmannova, Drahomira, Patton, Robert M., Knoth, Petr, and Stahl, Christopher G. Do citations and readership identify seminal publications?. United States: N. p., 2018.
Web. doi:10.1007/s11192-018-2669-y.
Herrmannova, Drahomira, Patton, Robert M., Knoth, Petr, & Stahl, Christopher G. Do citations and readership identify seminal publications?. United States. https://doi.org/10.1007/s11192-018-2669-y
Herrmannova, Drahomira, Patton, Robert M., Knoth, Petr, and Stahl, Christopher G. 2018.
"Do citations and readership identify seminal publications?". United States. https://doi.org/10.1007/s11192-018-2669-y. https://www.osti.gov/servlets/purl/1425336.
@article{osti_1425336,
title = {Do citations and readership identify seminal publications?},
author = {Herrmannova, Drahomira and Patton, Robert M. and Knoth, Petr and Stahl, Christopher G.},
abstractNote = {Here, this work presents a new approach for analysing the ability of existing research metrics to identify research which has strongly influenced future developments. More specifically, we focus on the ability of citation counts and Mendeley reader counts to distinguish between publications regarded as seminal and publications regarded as literature reviews by field experts. The main motivation behind our research is to gain a better understanding of whether and how well the existing research metrics relate to research quality. For this experiment we have created a new dataset which we call TrueImpactDataset and which contains two types of publications, seminal papers and literature reviews. Using the dataset, we conduct a set of experiments to study how citation and reader counts perform in distinguishing these publication types, following the intuition that causing a change in a field signifies research quality. Finally, our research shows that citation counts work better than a random baseline (by a margin of 10%) in distinguishing important seminal research papers from literature reviews while Mendeley reader counts do not work better than the baseline.},
doi = {10.1007/s11192-018-2669-y},
url = {https://www.osti.gov/biblio/1425336},
journal = {Scientometrics},
issn = {0138-9130},
number = 1,
volume = 115,
place = {United States},
year = {Sat Feb 10 00:00:00 EST 2018},
month = {Sat Feb 10 00:00:00 EST 2018}
}
Web of Science
Figures / Tables:
Works referenced in this record:
The Journal Impact Factor Denominator: Defining Citable (Counted) Items
journal, September 2009
- McVeigh, Marie E.; Mann, Stephen J.
- JAMA, Vol. 302, Issue 10
Microsoft Academic (Search): a Phoenix arisen from the ashes?
journal, June 2016
- Harzing, Anne-Wil
- Scientometrics, Vol. 108, Issue 3
The Anatomy of Impact: What Makes an Article Influential?
journal, March 1996
- Sternberg, Robert J.; Gordeeva, Tamara
- Psychological Science, Vol. 7, Issue 2
Web indicators for research evaluation. Part 2: Social media metrics
journal, September 2015
- Thelwall, Mike; Kousha, Kayvan
- El Profesional de la Información, Vol. 24, Issue 5
Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web
journal, September 2015
- Thelwall, Mike; Kousha, Kayvan
- El Profesional de la Información, Vol. 24, Issue 5
The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management
null, January 2015
- Wilsdon, James; Allen, Liz; Belfiore, Eleonora
- Unpublished
An interview-based study of the functions of citations in academic writing across two disciplines
journal, March 2009
- Harwood, Nigel
- Journal of Pragmatics, Vol. 41, Issue 3
Differences in impact factor across fields and over time
journal, January 2009
- Althouse, Benjamin M.; West, Jevin D.; Bergstrom, Carl T.
- Journal of the American Society for Information Science and Technology, Vol. 60, Issue 1
What makes articles highly cited?
journal, February 2014
- Antonakis, John; Bastardoz, Nicolas; Liu, Yonghong
- The Leadership Quarterly, Vol. 25, Issue 1
Coverage and adoption of altmetrics sources in the bibliometric community
journal, January 2014
- Haustein, Stefanie; Peters, Isabella; Bar-Ilan, Judit
- Scientometrics, Vol. 101, Issue 2
Problems of citation analysis: A study of uncited and seldom-cited influences
journal, October 2009
- MacRoberts, M. H.; MacRoberts, B. R.
- Journal of the American Society for Information Science and Technology, Vol. 61, Issue 1
Factors affecting citation rates of research articles: Factors Affecting Citation Rates of Research Articles
journal, May 2014
- Onodera, Natsuo; Yoshikane, Fuyuki
- Journal of the Association for Information Science and Technology, Vol. 66, Issue 4
Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics
journal, October 2014
- Bornmann, Lutz
- Journal of Informetrics, Vol. 8, Issue 4
Does quality and content matter for citedness? A comparison with para-textual factors and over time
journal, July 2015
- Bornmann, Lutz; Leydesdorff, Loet
- Journal of Informetrics, Vol. 9, Issue 3
Automatic classification of citation function
conference, January 2006
- Teufel, Simone; Siddharthan, Advaith; Tidhar, Dan
- Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing - EMNLP '06
The invariant distribution of references in scientific articles: The Invariant Distribution of References in Scientific Articles
journal, May 2015
- Bertin, Marc; Atanassova, Iana; Gingras, Yves
- Journal of the Association for Information Science and Technology, Vol. 67, Issue 1
Does the h-index for ranking of scientists really work?
journal, December 2005
- Bornmann, Lutz; Daniel, Hans-Dieter
- Scientometrics, Vol. 65, Issue 3
When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals: When Are Readers as Good as Citers for Bibliometrics? Scopus vs. Mendeley for LIS Journals
journal, January 2015
- Maflahi, Nabeil; Thelwall, Mike
- Journal of the Association for Information Science and Technology, Vol. 67, Issue 1
Measuring Scientific Impact Beyond Citation Counts
journal, September 2016
- Patton, Robert M.; Stahl, Christopher G.; Wells, Jack C.
- D-Lib Magazine, Vol. 22, Issue 9/10
Who reads research articles? An altmetrics analysis of Mendeley user categories
journal, April 2015
- Mohammadi, Ehsan; Thelwall, Mike; Haustein, Stefanie
- Journal of the Association for Information Science and Technology, Vol. 66, Issue 9
Citations versus journal impact factor as proxy of quality: could the latter ever be preferable?
journal, February 2010
- Abramo, Giovanni; D’Angelo, Ciriaco Andrea; Di Costa, Flavia
- Scientometrics, Vol. 84, Issue 3
Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication
journal, September 2008
- Bornmann, Lutz; Nast, Irina; Daniel, Hans-Dieter
- Scientometrics, Vol. 77, Issue 3
Characteristics of highly cited papers
journal, December 2003
- Aksnes, Dag W.
- Research Evaluation, Vol. 12, Issue 3
A review of the literature on citation impact indicators
journal, May 2016
- Waltman, Ludo
- Journal of Informetrics, Vol. 10, Issue 2
When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings
journal, March 2009
- Adler, Nancy J.; Harzing, Anne-Wil
- Academy of Management Learning & Education, Vol. 8, Issue 1
Are all literature citations equally important? Automatic citation strength estimation and its applications: Are Literature Citations Equally Important? Automatic Citation Strength Estimation and Its Applications
journal, April 2014
- Wan, Xiaojun; Liu, Fang
- Journal of the Association for Information Science and Technology, Vol. 65, Issue 9
Problems with Traditional Science Publishing and Finding a Wider Niche for Post-Publication Peer Review
journal, October 2014
- Teixeira da Silva, Jaime A.; Dobránszki, Judit
- Accountability in Research, Vol. 22, Issue 1
Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison
journal, November 2015
- Harzing, Anne-Wil; Alakangas, Satu
- Scientometrics, Vol. 106, Issue 2
Usefulness of altmetrics for measuring the broader impact of research: A case study using data from PLOS and F1000Prime
journal, May 2015
- Bornmann, Lutz
- Aslib Journal of Information Management, Vol. 67, Issue 3
The power of altmetrics on a CV: The Power of Altmetrics on a CV
journal, April 2013
- Piwowar, Heather; Priem, Jason
- Bulletin of the American Society for Information Science and Technology, Vol. 39, Issue 4
To better stand on the shoulder of giants
conference, January 2012
- Yan, Rui; Huang, Congrui; Tang, Jie
- Proceedings of the 12th ACM/IEEE-CS joint conference on Digital Libraries - JCDL '12
Citation Statistics
journal, February 2009
- Adler, Robert; Ewing, John; Taylor, Peter
- Statistical Science, Vol. 24, Issue 1
Mendeley readership counts: An investigation of temporal and disciplinary differences
journal, June 2015
- Thelwall, Mike; Sud, Pardeep
- Journal of the Association for Information Science and Technology, Vol. 67, Issue 12
What do citation counts measure? A review of studies on citing behavior
journal, January 2008
- Bornmann, Lutz; Daniel, Hans‐Dieter
- Journal of Documentation, Vol. 64, Issue 1
Letter to the Editor: About the quality and impact of scientific articles
journal, April 2017
- Ricker, Martin
- Scientometrics, Vol. 111, Issue 3
Priority criteria in peer review of scientific articles
journal, February 2016
- Nedić, Olgica; Dekanski, Aleksandar
- Scientometrics, Vol. 107, Issue 1
The top 100 papers
journal, October 2014
- Van Noorden, Richard; Maher, Brendan; Nuzzo, Regina
- Nature, Vol. 514, Issue 7524
Measuring academic influence: Not all citations are equal: Measuring Academic Influence
journal, May 2014
- Zhu, Xiaodan; Turney, Peter; Lemire, Daniel
- Journal of the Association for Information Science and Technology, Vol. 66, Issue 2
The rise and rise of citation analysis
journal, January 2007
- Meho, Lokman I.
- Physics World, Vol. 20, Issue 1
Using a balanced approach to bibliometrics: quantitative performance measures in the Australian Research Quality Framework
journal, June 2008
- Butler, L.
- Ethics in Science and Environmental Politics, Vol. 8
Which people use which scientific papers? An evaluation of data from F1000 and Mendeley
journal, July 2015
- Bornmann, Lutz; Haunschild, Robin
- Journal of Informetrics, Vol. 9, Issue 3
Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact?
journal, December 2016
- Bornmann, Lutz; Haunschild, Robin
- Scientometrics, Vol. 110, Issue 2
Do editors and referees look for signs of scientific misconduct when reviewing manuscripts?: A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication
text, January 2008
- Bornmann, Lutz; Nast, Irina; Daniel, Hans-Dieter
- ETH Zurich
Does Quality and Content Matter for Citedness? A comparison with para-textual factors and over time
text, January 2015
- Bornmann, Lutz; Leydesdorff, Loet
- figshare
Which people use which scientific papers? An evaluation of data from F1000 and Mendeley
text, January 2015
- Bornmann, Lutz; Haunschild, Robin
- figshare
Coverage and adoption of altmetrics sources in the bibliometric community
text, January 2014
- Stefanie, Haustein,; Isabella, Peters,; Hadas, Shema,
- The University of North Carolina at Chapel Hill University Libraries
Does Quality and Content Matter for Citedness? A comparison with para-textual factors and over time
text, January 2015
- Bornmann, Lutz; Leydesdorff, Loet
- figshare
Does the h-index for ranking of scientists really work?
text, January 2005
- Lutz, Bornmann,; Hans-Dieter, Daniel,
- ETH Zurich
Which people use which scientific papers? An evaluation of data from F1000 and Mendeley
text, January 2015
- Bornmann, Lutz; Haunschild, Robin
- figshare
Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics
preprint, January 2014
- Bornmann, Lutz
- arXiv
Usefulness of altmetrics for measuring the broader impact of research: A case study using data from PLOS and F1000Prime
text, January 2015
- Bornmann, Lutz
- figshare
Why the impact factor of journals should not be used for evaluating research
journal, February 1997
- Seglen, P. O.
- BMJ, Vol. 314, Issue 7079
Simple Yet Effective Methods for Large-Scale Scholarly Publication Ranking
preprint, January 2016
- Herrmannova, Drahomira; Knoth, Petr
- arXiv
Works referencing / citing this record:
Which can better predict the future success of articles? Bibliometric indices or alternative metrics
journal, April 2019
- Wang, Mingyang; Wang, Zhenyu; Chen, Guangsheng
- Scientometrics, Vol. 119, Issue 3
Comparing the topological rank of journals in Web of Science and Mendeley
journal, July 2019
- Katchanov, Yurij L.; Markova, Yulia V.; Shmatko, Natalia A.
- Heliyon, Vol. 5, Issue 7
Peer review vs bibliometrics: which method better predicts the scholarly impact of publications?
text, January 2019
- Abramo, Giovanni; D'Angelo, Ciriaco Andrea; Reale, Emanuela
- arXiv
Figures / Tables found in this record: