Measuring Research Impact

Although generally focused on number of citations articles within a publication receive, more recently, other measures have emerged as useful.  This section will provide an overview of some major players in both citation and non-citation based metrics.  Each has its strengths and weaknesses and should be used in combination with other factors when measuring research impact.

Acceptance Rate

• How to:  Metrics Toolkit: Journal Acceptance Rate
• What is it?:  Percentage of items submitted to a venue that are ultimately accepted for publication. 
• Useful for:  Typically, a lower rate means the journal is more selective, presumably because there is relatively high demand from the research comminuty to showcase their work there.
• Limitations :  Other factors may drive the acceptance rate, such as journal size and age.  It is not always easy to obtain and calculations are not standardized across journals, e.g., some may include all manuscripts while others only include peer-reviewed literature.  There is also variability across disciplines.

Abstracting and Indexing

• How to:  Look up the journal title in Ulrichsweb serials directory and review the "Abstracting & Indexing" section
• What is it?:  Databases that provide summaries and subject heading and/or keyword tagging for publications to aid discovery of the content
• Useful for:  Speaks to the visibility and discoverability of scholarly content to the relevant research communities, as well as presumed vetting performed by article databases.
• Limitations :   Publication types besides journal articles are not well indexed.

Quality Control Measures

• How to:   Review the journal's website for quality control-related protocols, such as guidelines that align with COPE, WAME, or ICMJE  
• What is it?:  The standards and protocols the journal follows, in particular related to publication ethics, research integrity, and peer review, act as scaffolding for responsible reporting of veritable research
• Useful for:   Better quality measures can mean more accurate assessment of research impact by that journal
• Limitations:   Cannot safeguard against all potential maleficence

Journal Citation Reports provides a suite of citation-based metrics including:

Journal Impact Factor (JIF)

• How to:  Quick Guide: JIF
• What is it?:  Measures ratio between citations and recent citable items published in a journal
• Useful for:  Comparing the relative influence of journals within a discipline, as measured by citations
• Limitations :  Scope of journals included may be overly restrictive depending on discipline and some article types, such as reviews, are disproportionately represented

Journal Citation Indicator (JCI)

• How to: Quick Guide: JCI
• What is it?: Uses the mean Category Normalized Citation Impact (CNCI) from the three prior years to measure relative citation impact as the ratio of citations compared to a global baseline
• Useful for:  Comparing journals not necessarily in the same field (although most appropriate for assessing adjacent fields) since it controls for different disciplines, document types, and publication year
• Limitations:  Scores for a single journal influenced by number of subject categories the journal is in. 

Citation Distribution

• How to: Brief Video: Citation Distribution
• What is it?: Shows the frequency with which items published in the year or two years prior were cited
• Useful for: Determining if a small number of articles is driving the majority of citations
• Limitations : Debate as to whether this visual provides enough information without deeper analysis.

Scopus provides a suite of citation-based metrics including:

CiteScore

• How to: CiteScore Journal Metric - FAQs
• What is it?:  Average number of citations per peer-reviewed document (i.e., articles, reviews, conference papers, data papers and book chapters), using four years of data.
• Useful for: Simple comparison of the citation impact of peer-reviewed research within the same field.
• Limitations:  Cannot be used to compare journals from different disciplines.

SNIP (Source Normalized Impact per Paper)

• How to:  Scopus Tips & Tricks: SNIP
• What is it?:  Measures contextual citation impact by weighting citations based on the total number of citations in a subject field. The impact of a single citation is given higher value in subject areas where citations are less likely, and vice versa. 
• Useful for: Comparing journals from different subject fields.
• Limitations:  Does not correct for self citations.

SJR (SCImago Journal Rank)

• How to:  Scopus Tips & Tricks: SJR
• What is it?: Measures the weighted citations of journal articles, book series, and conference proceedings in journal publications during the three previous years.
• Useful for:  Normalizing for the subject field and appraising quality and reputation of the journal in its metrics.
• Limitations:  Difficult to replicate calculations and may not represent interdisciplinary work well.

Google Scholar provides somewhat simpler in design journal-level metrics including:

H5-Index

• How to:  Google Scholar Metrics Overview
• What is it?:  The largest number h such that at least h articles in that publication were cited at least h times each during a five-year period.
• Useful for: Quickly gauging the visibility and influence of literature beyond the reach of Web of Science and/or Scopus.
• Limitations:  Cannot be used to compare journals from different disciplines and journal/citation quality not reviewed.