Do citations and readership identify seminal publications?

Herrmannova, Drahomira; Patton, Robert M.; Knoth, Petr; Stahl, Christopher G.

doi:10.1007/s11192-018-2669-y

Title: Do citations and readership identify seminal publications?

Journal Article · Sat Feb 10 00:00:00 EST 2018 · Scientometrics

DOI:https://doi.org/10.1007/s11192-018-2669-y· OSTI ID:1425336

^[1];

^[2];

^[3];

^[2]

Open Univ., Milton Keynes (United Kingdom); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Open Univ., Milton Keynes (United Kingdom)

Here, this work presents a new approach for analysing the ability of existing research metrics to identify research which has strongly influenced future developments. More specifically, we focus on the ability of citation counts and Mendeley reader counts to distinguish between publications regarded as seminal and publications regarded as literature reviews by field experts. The main motivation behind our research is to gain a better understanding of whether and how well the existing research metrics relate to research quality. For this experiment we have created a new dataset which we call TrueImpactDataset and which contains two types of publications, seminal papers and literature reviews. Using the dataset, we conduct a set of experiments to study how citation and reader counts perform in distinguishing these publication types, following the intuition that causing a change in a field signifies research quality. Finally, our research shows that citation counts work better than a random baseline (by a margin of 10%) in distinguishing important seminal research papers from literature reviews while Mendeley reader counts do not work better than the baseline.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1425336

Journal Information:: Scientometrics, Vol. 115, Issue 1; ISSN 0138-9130

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 6 works

Citation information provided by
Web of Science

References (54)

The Journal Impact Factor Denominator: Defining Citable (Counted) Items McVeigh, Marie E.; Mann, Stephen J. JAMA, Vol. 302, Issue 10 https://doi.org/10.1001/jama.2009.1301	journal	September 2009
Microsoft Academic (Search): a Phoenix arisen from the ashes? Harzing, Anne-Wil Scientometrics, Vol. 108, Issue 3 https://doi.org/10.1007/s11192-016-2026-y	journal	June 2016
The Anatomy of Impact: What Makes an Article Influential? Sternberg, Robert J.; Gordeeva, Tamara Psychological Science, Vol. 7, Issue 2 https://doi.org/10.1111/j.1467-9280.1996.tb00332.x	journal	March 1996
Web indicators for research evaluation. Part 2: Social media metrics Thelwall, Mike; Kousha, Kayvan El Profesional de la Información, Vol. 24, Issue 5 https://doi.org/10.3145/epi.2015.sep.09	journal	September 2015
Web indicators for research evaluation. Part 1: Citations and links to academic articles from the Web Thelwall, Mike; Kousha, Kayvan El Profesional de la Información, Vol. 24, Issue 5 https://doi.org/10.3145/epi.2015.sep.08	journal	September 2015
The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management Wilsdon, James; Allen, Liz; Belfiore, Eleonora Unpublished https://doi.org/10.13140/RG.2.1.4929.1363	null	January 2015
An interview-based study of the functions of citations in academic writing across two disciplines Harwood, Nigel Journal of Pragmatics, Vol. 41, Issue 3 https://doi.org/10.1016/j.pragma.2008.06.001	journal	March 2009
Differences in impact factor across fields and over time Althouse, Benjamin M.; West, Jevin D.; Bergstrom, Carl T. Journal of the American Society for Information Science and Technology, Vol. 60, Issue 1 https://doi.org/10.1002/asi.20936	journal	January 2009
What makes articles highly cited? Antonakis, John; Bastardoz, Nicolas; Liu, Yonghong The Leadership Quarterly, Vol. 25, Issue 1 https://doi.org/10.1016/j.leaqua.2013.10.014	journal	February 2014
Coverage and adoption of altmetrics sources in the bibliometric community Haustein, Stefanie; Peters, Isabella; Bar-Ilan, Judit Scientometrics, Vol. 101, Issue 2 https://doi.org/10.1007/s11192-013-1221-3	journal	January 2014
Problems of citation analysis: A study of uncited and seldom-cited influences MacRoberts, M. H.; MacRoberts, B. R. Journal of the American Society for Information Science and Technology, Vol. 61, Issue 1 https://doi.org/10.1002/asi.21228	journal	October 2009
Factors affecting citation rates of research articles: Factors Affecting Citation Rates of Research Articles Onodera, Natsuo; Yoshikane, Fuyuki Journal of the Association for Information Science and Technology, Vol. 66, Issue 4 https://doi.org/10.1002/asi.23209	journal	May 2014
Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics Bornmann, Lutz Journal of Informetrics, Vol. 8, Issue 4 https://doi.org/10.1016/j.joi.2014.09.005	journal	October 2014
Does quality and content matter for citedness? A comparison with para-textual factors and over time Bornmann, Lutz; Leydesdorff, Loet Journal of Informetrics, Vol. 9, Issue 3 https://doi.org/10.1016/j.joi.2015.03.001	journal	July 2015
Automatic classification of citation function Teufel, Simone; Siddharthan, Advaith; Tidhar, Dan Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing - EMNLP '06 https://doi.org/10.3115/1610075.1610091	conference	January 2006
The invariant distribution of references in scientific articles: The Invariant Distribution of References in Scientific Articles Bertin, Marc; Atanassova, Iana; Gingras, Yves Journal of the Association for Information Science and Technology, Vol. 67, Issue 1 https://doi.org/10.1002/asi.23367	journal	May 2015
Does the h-index for ranking of scientists really work? Bornmann, Lutz; Daniel, Hans-Dieter Scientometrics, Vol. 65, Issue 3 https://doi.org/10.1007/s11192-005-0281-4	journal	December 2005
When are readership counts as useful as citation counts? Scopus versus Mendeley for LIS journals: When Are Readers as Good as Citers for Bibliometrics? Scopus vs. Mendeley for LIS Journals Maflahi, Nabeil; Thelwall, Mike Journal of the Association for Information Science and Technology, Vol. 67, Issue 1 https://doi.org/10.1002/asi.23369	journal	January 2015
Measuring Scientific Impact Beyond Citation Counts Patton, Robert M.; Stahl, Christopher G.; Wells, Jack C. D-Lib Magazine, Vol. 22, Issue 9/10 https://doi.org/10.1045/september2016-patton	journal	September 2016
Who reads research articles? An altmetrics analysis of Mendeley user categories Mohammadi, Ehsan; Thelwall, Mike; Haustein, Stefanie Journal of the Association for Information Science and Technology, Vol. 66, Issue 9 https://doi.org/10.1002/asi.23286	journal	April 2015
Citations versus journal impact factor as proxy of quality: could the latter ever be preferable? Abramo, Giovanni; D’Angelo, Ciriaco Andrea; Di Costa, Flavia Scientometrics, Vol. 84, Issue 3 https://doi.org/10.1007/s11192-010-0200-1	journal	February 2010
Do editors and referees look for signs of scientific misconduct when reviewing manuscripts? A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication Bornmann, Lutz; Nast, Irina; Daniel, Hans-Dieter Scientometrics, Vol. 77, Issue 3 https://doi.org/10.1007/s11192-007-1950-2	journal	September 2008
Characteristics of highly cited papers Aksnes, Dag W. Research Evaluation, Vol. 12, Issue 3 https://doi.org/10.3152/147154403781776645	journal	December 2003
A review of the literature on citation impact indicators Waltman, Ludo Journal of Informetrics, Vol. 10, Issue 2 https://doi.org/10.1016/j.joi.2016.02.007	journal	May 2016
When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings Adler, Nancy J.; Harzing, Anne-Wil Academy of Management Learning & Education, Vol. 8, Issue 1 https://doi.org/10.5465/AMLE.2009.37012181	journal	March 2009
Are all literature citations equally important? Automatic citation strength estimation and its applications: Are Literature Citations Equally Important? Automatic Citation Strength Estimation and Its Applications Wan, Xiaojun; Liu, Fang Journal of the Association for Information Science and Technology, Vol. 65, Issue 9 https://doi.org/10.1002/asi.23083	journal	April 2014
Problems with Traditional Science Publishing and Finding a Wider Niche for Post-Publication Peer Review Teixeira da Silva, Jaime A.; Dobránszki, Judit Accountability in Research, Vol. 22, Issue 1 https://doi.org/10.1080/08989621.2014.899909	journal	October 2014
Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison Harzing, Anne-Wil; Alakangas, Satu Scientometrics, Vol. 106, Issue 2 https://doi.org/10.1007/s11192-015-1798-9	journal	November 2015
Usefulness of altmetrics for measuring the broader impact of research: A case study using data from PLOS and F1000Prime Bornmann, Lutz Aslib Journal of Information Management, Vol. 67, Issue 3 https://doi.org/10.1108/AJIM-09-2014-0115	journal	May 2015
The power of altmetrics on a CV: The Power of Altmetrics on a CV Piwowar, Heather; Priem, Jason Bulletin of the American Society for Information Science and Technology, Vol. 39, Issue 4 https://doi.org/10.1002/bult.2013.1720390405	journal	April 2013
To better stand on the shoulder of giants Yan, Rui; Huang, Congrui; Tang, Jie Proceedings of the 12th ACM/IEEE-CS joint conference on Digital Libraries - JCDL '12 https://doi.org/10.1145/2232817.2232831	conference	January 2012
Citation Statistics Adler, Robert; Ewing, John; Taylor, Peter Statistical Science, Vol. 24, Issue 1 https://doi.org/10.1214/09-STS285	journal	February 2009
Mendeley readership counts: An investigation of temporal and disciplinary differences Thelwall, Mike; Sud, Pardeep Journal of the Association for Information Science and Technology, Vol. 67, Issue 12 https://doi.org/10.1002/asi.23559	journal	June 2015
What do citation counts measure? A review of studies on citing behavior Bornmann, Lutz; Daniel, Hans‐Dieter Journal of Documentation, Vol. 64, Issue 1 https://doi.org/10.1108/00220410810844150	journal	January 2008
Letter to the Editor: About the quality and impact of scientific articles Ricker, Martin Scientometrics, Vol. 111, Issue 3 https://doi.org/10.1007/s11192-017-2374-2	journal	April 2017
Priority criteria in peer review of scientific articles Nedić, Olgica; Dekanski, Aleksandar Scientometrics, Vol. 107, Issue 1 https://doi.org/10.1007/s11192-016-1869-6	journal	February 2016
The top 100 papers Van Noorden, Richard; Maher, Brendan; Nuzzo, Regina Nature, Vol. 514, Issue 7524 https://doi.org/10.1038/514550a	journal	October 2014
Measuring academic influence: Not all citations are equal: Measuring Academic Influence Zhu, Xiaodan; Turney, Peter; Lemire, Daniel Journal of the Association for Information Science and Technology, Vol. 66, Issue 2 https://doi.org/10.1002/asi.23179	journal	May 2014
The rise and rise of citation analysis Meho, Lokman I. Physics World, Vol. 20, Issue 1 https://doi.org/10.1088/2058-7058/20/1/33	journal	January 2007
Using a balanced approach to bibliometrics: quantitative performance measures in the Australian Research Quality Framework Butler, L. Ethics in Science and Environmental Politics, Vol. 8 https://doi.org/10.3354/esep00077	journal	June 2008
Which people use which scientific papers? An evaluation of data from F1000 and Mendeley Bornmann, Lutz; Haunschild, Robin Journal of Informetrics, Vol. 9, Issue 3 https://doi.org/10.1016/j.joi.2015.04.001	journal	July 2015
Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact? Bornmann, Lutz; Haunschild, Robin Scientometrics, Vol. 110, Issue 2 https://doi.org/10.1007/s11192-016-2200-2	journal	December 2016
Do editors and referees look for signs of scientific misconduct when reviewing manuscripts?: A quantitative content analysis of studies that examined review criteria and reasons for accepting and rejecting manuscripts for publication Bornmann, Lutz; Nast, Irina; Daniel, Hans-Dieter ETH Zurich https://doi.org/10.3929/ethz-b-000010796	text	January 2008
Does Quality and Content Matter for Citedness? A comparison with para-textual factors and over time Bornmann, Lutz; Leydesdorff, Loet figshare https://doi.org/10.6084/m9.figshare.1269182.v3	text	January 2015
Which people use which scientific papers? An evaluation of data from F1000 and Mendeley Bornmann, Lutz; Haunschild, Robin figshare https://doi.org/10.6084/m9.figshare.1375661	text	January 2015
Coverage and adoption of altmetrics sources in the bibliometric community Stefanie, Haustein,; Isabella, Peters,; Hadas, Shema, The University of North Carolina at Chapel Hill University Libraries https://doi.org/10.17615/bzx4-n933	text	January 2014
Does Quality and Content Matter for Citedness? A comparison with para-textual factors and over time Bornmann, Lutz; Leydesdorff, Loet figshare https://doi.org/10.6084/m9.figshare.1269182.v2	text	January 2015
Does the h-index for ranking of scientists really work? Lutz, Bornmann,; Hans-Dieter, Daniel, ETH Zurich https://doi.org/10.3929/ethz-b-000001855	text	January 2005
Which people use which scientific papers? An evaluation of data from F1000 and Mendeley Bornmann, Lutz; Haunschild, Robin figshare https://doi.org/10.6084/m9.figshare.1375661.v1	text	January 2015
Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics Bornmann, Lutz arXiv https://doi.org/10.48550/arxiv.1406.7091	preprint	January 2014
A review of the literature on citation impact indicators Waltman, Ludo arXiv https://doi.org/10.48550/arxiv.1507.02099	preprint	January 2015
Usefulness of altmetrics for measuring the broader impact of research: A case study using data from PLOS and F1000Prime Bornmann, Lutz figshare https://doi.org/10.6084/m9.figshare.1344583	text	January 2015
Why the impact factor of journals should not be used for evaluating research Seglen, P. O. BMJ, Vol. 314, Issue 7079 https://doi.org/10.1136/bmj.314.7079.497	journal	February 1997
Simple Yet Effective Methods for Large-Scale Scholarly Publication Ranking Herrmannova, Drahomira; Knoth, Petr arXiv https://doi.org/10.48550/arxiv.1611.05222	preprint	January 2016

Cited By (3)

Which can better predict the future success of articles? Bibliometric indices or alternative metrics Wang, Mingyang; Wang, Zhenyu; Chen, Guangsheng Scientometrics, Vol. 119, Issue 3 https://doi.org/10.1007/s11192-019-03052-9	journal	April 2019
Comparing the topological rank of journals in Web of Science and Mendeley Katchanov, Yurij L.; Markova, Yulia V.; Shmatko, Natalia A. Heliyon, Vol. 5, Issue 7 https://doi.org/10.1016/j.heliyon.2019.e02089	journal	July 2019
Peer review vs bibliometrics: which method better predicts the scholarly impact of publications? Abramo, Giovanni; D'Angelo, Ciriaco Andrea; Reale, Emanuela arXiv https://doi.org/10.48550/arxiv.1909.06209	text	January 2019

Figures / Tables (20)

Similar Records

Citations and Readership are Poor Indicators of Research Excellence

Conference · Wed Feb 01 00:00:00 EST 2017 · OSTI ID:1425336

Herrmannova, Dasha; Patton, Robert; Knoth, Petr; +1 more

Text and Graph Based Approach for Analyzing Patterns of Research Collaboration: An Analysis of the TrueImpactDataset

Conference · Tue May 01 00:00:00 EDT 2018 · OSTI ID:1425336

Herrmannova, Dasha; Knoth, Petr; Stahl, Christopher; +2 more

Research Collaboration Analysis Using Text and Graph Features

Conference · Thu Mar 01 00:00:00 EST 2018 · OSTI ID:1425336