With Great Power Comes the Responsible Use of Metrics
Download as PPTX, PDF1 like81 views
The document discusses the concept of responsible metrics in research assessment, focusing on the importance of both qualitative and quantitative measures to evaluate scholarly contributions. It highlights the limitations of traditional metrics and introduces frameworks like DORA and the Leiden Manifesto, which advocate for a balanced approach to assessing research quality. The emphasis is on transparency, diversity in metrics, and supporting researchers in the responsible use of these evaluation tools.
With Great Power Comes the Responsible Use of Metrics
- 1. With Great Power Comes the Responsible Use of Metrics Claire Sewell Research Support Skills Coordinator Office of Scholarly Communication April 2019
- 2. Topics 1. Where are we now? 2. The Responsible Metrics Movement 3. Supporting researchers with responsible metrics
- 4. Types of metric Bibliometrics Statistical analysis of outputs Altmetrics Alternative metrics to judge new output types Social media H-Index Journal Impact Factor Eigenfactor Citation counts News stories Public policy Peer review Patents
- 5. Levels of metric Model adapted from Meaningful Metrics / Roemer & Borchardt. Licensed under a CC-BY-NC 4.0 license. Individual scholarly contributions Venues of production Individual authors Groups and institutions
- 6. Levels of metric Model adapted from Meaningful Metrics / Roemer & Borchardt. Licensed under a CC-BY-NC 4.0 license. “Is Galaxy Chocolate Better Than Cadburys?” Journal of Chocolate Studies Dr. Snickers Candy University
- 7. Using metrics • Judging research quality • Publishing decisions • Performance reviews • Career advancement
- 8. Metric limitations Quantitative measures Potential bias Lack of consistency Potential for gaming Fit for purpose?
- 10. DORA (2012) Detail the criteria used to assess both researchers and their work Reinforce the idea that it is the quality of the piece of research that is important as opposed to the metric score of the title that it’s published in Look at the value and impact of all research outputs Review their use of metrics and use a variety of measures to assess researchers and their outputs
- 11. Leiden Manifesto (2015) Base assessment of individual researchers on a qualitative judgement of their portfolio Quantitative evaluation should support qualitative, expert assessment Measure performance against the research missions of the institution, group or researcher Protect excellence in locally relevant research Keep data collection and analytical processes open, transparent and simple Allow those evaluated to verify data and analysis Account for variation by field in publication and citation practices Avoid misplaced concreteness and false precision Recognize the systematic effects of assessment and indicators Scrutinize indicators regularly and update them
- 12. The Metric Tide (2015) Robustness – metrics should be based on the best data available Humility – both qualitative and quantitative measures should be used Transparency – the methods used to calculate metrics should be open and transparent Diversity – metrics need to take into account differences in career stage and discipline Reflexivity – metrics should be responsive to change
- 13. Responsible metrics • Quantitative and qualitative • Openness • Quality • Range • Review
- 14. Supporting researchers with responsible metrics 3