DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: A Principal Component Analysis of 39 Scientific Impact Measures

Journal Article · · PLoS ONE
 [1];  [2];  [2];  [2]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States); DOE/OSTI
  2. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Background: The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact. Methodology: We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that were calculated on the basis of both citation and usage log data. Conclusions: Our results indicate that the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.

Research Organization:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Organization:
Andrew W. Mellon Foundation; USDOE Office of Science (SC), Biological and Environmental Research (BER). Biological Systems Science Division
OSTI ID:
1627376
Journal Information:
PLoS ONE, Journal Name: PLoS ONE Journal Issue: 6 Vol. 4; ISSN 1932-6203
Publisher:
Public Library of ScienceCopyright Statement
Country of Publication:
United States
Language:
English

References (11)

ISI's impact factor as misnomer: A proposed new measure to assess journal impact journal December 1997
Earlier Web usage statistics as predictors of later citation impact
  • Brody, Tim; Harnad, Stevan; Carr, Leslie
  • Journal of the American Society for Information Science and Technology, Vol. 57, Issue 8 https://doi.org/10.1002/asi.20373
journal January 2006
The R- and AR-indices: Complementing the h-index journal March 2007
Refining dermatology journal impact factors using PageRank journal July 2007
Sense and nonsense about the impact factor journal January 1997
Why the impact factor of journals should not be used for evaluating research journal February 1997
Clickstream Data Yields High-Resolution Maps of Science journal March 2009
Towards Usage-based Impact Metrics: - First Results from the MESUR Project preprint January 2008
"Betweenness Centrality" as an Indicator of the "Interdisciplinarity" of Scientific Journals text January 2009
Toward alternative metrics of journal impact: A comparison of download and citation data preprint January 2005
Journal Status text January 2006

Cited By (100)

Mechanism change in a simulation of peer review: from junk support to elitism journal February 2014
Statistical relationships between journal use and research output at academic institutions in South Korea journal March 2015
Development of computer science disciplines: a social network analysis approach journal April 2011
Lobby index as a network centrality measure journal November 2013
Global Multi-Level Analysis of the ‘Scientific Food Web' journal January 2013
Impact of lexical and sentiment factors on the popularity of scientific papers journal June 2016
Visibility of medical informatics regarding bibliometric indices and databases journal April 2011
Article-Level Metrics and the Evolution of Scientific Impact journal November 2009
The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations journal October 2013
What Can Article-Level Metrics Do for You? journal October 2013
Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level journal September 2016
Where There Is No Health Research: What Can Be Done to Fill the Global Gaps in Health Research? journal April 2012
Integrative Approach to Quality Assessment of Medical Journals Using Impact Factor, Eigenfactor, and Article Influence Scores journal April 2010
Fifty-Year Fate and Impact of General Medical Journals journal September 2010
Self-Selected or Mandated, Open Access Increases Citation Impact for Higher Quality Research journal October 2010
Novel Online Dimensionality Reduction Method with Improved Topology Representing and Radial Basis Function Networks journal July 2015
Does bibliometric research confer legitimacy to research assessment practice? A sociological study of reputational control, 1972-2016 journal June 2018
Measuring researcher independence using bibliometric data: A proposal for a new performance indicator journal March 2019
Validity of Qualis database as a predictor of evidence hierarchy and risk of bias in randomized controlled trials - a case study in dentistry journal January 2011
OpenAccess Statistics: alternative impact measures for Open Access documents? : an examination how to generate interoperable usage information from distributed Open Access services collection January 2010
Open Access, zitationsbasierte und nutzungsbasierte Impact Maße: Einige Befunde collection January 2009
The Journal Impact Factor: Moving Toward an Alternative and Combined Scientometric Approach journal January 2017
Nine Criteria for a Measure of Scientific Output journal January 2011
Learning from open source software projects to improve scientific review journal January 2012
The large-scale structure of journal citation networks preprint January 2011
The citation wake of publications detects Nobel laureates' papers text January 2013
Grand Challenges in Measuring and Characterizing Scholarly Impact text January 2016
Author-level metrics in the new academic profile platforms: The online behaviour of the Bibliometrics community text January 2018
Scholia And Scientometrics With Wikidata text January 2017
Scientific production of Sports Science in Iran: A Scientometric Analysis journal January 2014
Making Use of H-index: the Shape of Science at the University of Sarajevo journal January 2017
Analyzing readerships of International Iranian publications in Mendeley: an altmetrics study dataset January 2015
Usage bibliometrics journal January 2010
A cluster analysis of scholar and journal bibliometric indicators journal October 2009
Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows: Mendeley Readership Altmetrics for the Social Sciences and Humanities: Research Evaluation and Knowledge Flows journal March 2014
Measuring academic influence: Not all citations are equal: Measuring Academic Influence journal May 2014
Who reads research articles? An altmetrics analysis of Mendeley user categories
  • Mohammadi, Ehsan; Thelwall, Mike; Haustein, Stefanie
  • Journal of the Association for Information Science and Technology, Vol. 66, Issue 9 https://doi.org/10.1002/asi.23286
journal April 2015
A relational altmetric? Network centrality on ResearchGate as an indicator of scientific impact: Journal of the Association for Information Science and Technology
  • Hoffmann, Christian Pieter; Lutz, Christoph; Meckel, Miriam
  • Journal of the Association for Information Science and Technology, Vol. 67, Issue 4 https://doi.org/10.1002/asi.23423
journal April 2015
Does the early career ‘publish or perish’ myth represent an opportunity for the publishing industry?: Publish or perish journal January 2019
An Overview of Author-Level Indicators of Research Performance book January 2019
Three Aspects of the Research Impact by a Scientist: Measurement Methods and an Empirical Evaluation book January 2015
Scholia, Scientometrics and Wikidata book January 2017
From Manuscript Evaluation to Article Valuation: The Changing Technologies of Journal Peer Review journal September 2014
The influence of scientific prestige and peer effects on the intention to create university spin-offs journal September 2019
Correlation between the Journal Impact Factor and three other journal citation indices journal June 2010
Impact factor: outdated artefact or stepping-stone to journal certification? journal November 2011
How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications journal March 2014
Do highly cited researchers successfully use the social web? journal June 2014
Successful fish go with the flow: citation impact prediction based on centrality measures for term–document networks journal April 2016
An empirical investigation of the associations among usage, scientific collaboration and citation impact journal March 2017
Quantifying the internationality and multidisciplinarity of authors and journals using ecological statistics journal February 2018
Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: application to the field of Dentistry journal March 2018
#Psychology: a bibliometric analysis of psychological literature in the online media journal March 2018
Rainbow ranking: an adaptable, multidimensional ranking method for publication sets journal April 2018
Scholarly impact assessment: a survey of citation weighting solutions journal December 2018
Two decades of information systems: a bibliometric review journal December 2018
The relationship between usage and citations in an open access mega-journal journal September 2019
Cross-cultural adaptation, psychometric proprieties and factor structure of the Multidimensional Student Life Satisfaction Scale (MSLSS): A study with Palestinian children living in refugee camps journal June 2018
Self-organizing mapping based swarm intelligence for secondary and tertiary proteins classification journal August 2017
Metrics: A profusion of measures journal June 2010
Global Multi-Level Analysis of the ‘Scientific Food Web' journal January 2013
Research Productivity in Top-Ranked Schools in Psychology and Social Work: Does Having a Research Culture Matter? journal January 2015
Top cited research over fifteen years in Sports Biomechanics journal October 2018
Commentary on Ruscio et al.: “Measuring Scholarly Impact Using Modern Citation-Based Indices” journal July 2012
Scholarly reputation journal August 2018
Measuring scientific impact of fisheries and aquaculture research-for-development projects in South East Asia and the Pacific journal August 2019
Impact of lexical and sentiment factors on the popularity of scientific papers journal June 2016
Relative Citation Ratio (RCR): A new metric that uses citation rates to measure influence at the article level posted_content March 2016
Disciplinary differences in the use of academic social networking sites journal August 2015
Using Publication Metrics to Highlight Academic Productivity and Research Impact journal October 2014
Flows of Research Manuscripts Among Scientific Journals Reveal Hidden Submission Patterns journal October 2012
Scholarly Communication: Cultural Contexts, Evolving Models journal October 2013
The citation advantage for open access science journals with and without article processing charges journal March 2019
“3 . . 2 . . 1 . . Impact [Factor]: Target [Academic Career] Destroyed!”: Just Another Statistical Casualty journal November 2012
Geographic trends in research output and citations in veterinary medicine: insight into global research capacity, species specialization, and interdisciplinary relationships journal January 2013
The impact factor of a journal is a poor measure of the clinical relevance of its papers journal March 2014
Article-Level Metrics and the Evolution of Scientific Impact journal November 2009
The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations journal October 2013
What Can Article-Level Metrics Do for You? journal October 2013
Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level journal September 2016
Where There Is No Health Research: What Can Be Done to Fill the Global Gaps in Health Research? journal April 2012
The Altmetrics Collection journal November 2012
An Absolute Index (Ab-index) to Measure a Researcher’s Useful Contributions and Productivity journal December 2013
The Citation Wake of Publications Detects Nobel Laureates' Papers journal December 2014
Does bibliometric research confer legitimacy to research assessment practice? A sociological study of reputational control, 1972-2016 journal June 2018
High-impact and transformative science (HITS) metrics: Definition, exemplification, and comparison journal July 2018
Clustering algorithms: A comparative approach journal January 2019
Colaborações científicas em Zika: identificação dos principais grupos e pesquisadores através da análise de redes sociais journal January 2019
Statistical evaluation of research performance of young university scholars: A case study journal August 2018
Scientometric databases and integration of Ukrainian biological journals into common information space [Наукометричні бази даних та інтеграція української біологічної періодики у міжнародний інформаційний простір] journal January 2014
The relationship between usage and citations in an open access mega-journal text January 2019
The relationship between usage and citations in an open access mega-journal text January 2019
Techniques and Emerging Trends for State of the Art Equipment Maintenance Systems—A Bibliometric Analysis journal June 2018
The impact of scientific journals of communication: Comparing Google Scholar Metrics, Web of Science and Scopus [El impacto de las revistas de comunicación: comparando Google Scholar Metrics, Web of Science y Scopus] journal June 2013
Altmetrics: New indicators for scientific communication in Web 2.0 [Altmetrics: nuevos indicadores para la comunicación científica en la Web 2.0] journal June 2013
Global Multi-Level Analysis of the 'Scientific Food Web' text January 2013
Impact Factor: outdated artefact or stepping-stone to journal certification? text January 2012
Clustering Algorithms: A Comparative Approach preprint January 2016
What Can Article-Level Metrics Do for You? posted_content December 2013
What Can Article-Level Metrics Do for You? posted_content October 2013