Science audited: Indicator-based systems for research evaluation and resource allocation
Åström, Fredrik LU
(2018)
Scientific Communication and Gatekeeping in Academia in the 21st Century
- Abstract
- This paper presents an ongoing research project investigating the use of bibliometrics for research evaluation and resource allocation purposes, and to investigate the effects of these evaluation models on individual, organizational and political levels.
A main focus has been on bibliometrics based evaluation practices. The use of bibliometrics for local resource allocation at Swedish universities was analyzed (Hammarfelt et.al, 2016; Hammarfelt & Åström, 2015), finding a great degree of variation both in how and where bibliometrics is used; and while the introduction of a bibliometrics based national model for resource allocation in Sweden in 2009 probably had an effect on the increasing use of bibliometrics in local allocation... (More) - This paper presents an ongoing research project investigating the use of bibliometrics for research evaluation and resource allocation purposes, and to investigate the effects of these evaluation models on individual, organizational and political levels.
A main focus has been on bibliometrics based evaluation practices. The use of bibliometrics for local resource allocation at Swedish universities was analyzed (Hammarfelt et.al, 2016; Hammarfelt & Åström, 2015), finding a great degree of variation both in how and where bibliometrics is used; and while the introduction of a bibliometrics based national model for resource allocation in Sweden in 2009 probably had an effect on the increasing use of bibliometrics in local allocation models, there are also great variation in terms of bibliometric indicators and allocation models being in use. But bibliometrics is not just being used in evaluation and resource allocation models at a systemic level. Another important part of the use of bibliometrics in research evaluation practices is how it is being used by academic scholars and scientists serving as external assessors of candidates for academic positions or of research funding proposals. This practice has been studied through analyses of application assessment reports in economics, history, and biomedical research at Swedish universities (Hammarfelt, forthcoming; Hammarfelt & Rushforth, in press).
The other main focus of the project thus far has been on the underlying technologies, or the infrastructure of bibliometrics and research evaluation systems. A main aspect of the ‘infrastructure’ theme is the challenges of understanding complexities and interactions of different technical or auxiliary systems, and their role in evaluation processes. This has been investigated empirically in a minor study of the role of knowledge organization systems in bibliometric analysis and evaluation (Åström et.al, 2016). Another empirical study of infrastructures under development is a study of current research information systems (CRIS). From a theoretical perspective, a conceptual framework has been tentatively developed, describing infrastructures, not so much as specific technical systems, but as systems of relations between hardware, software and people (Åström, 2016). This was operationalized by defining: ‘hardware’ as databases, software for bibliometric analyses, and so on; ‘software’ as the evaluation systems, the bibliometric indicators, and the allocation models; and ‘people’ as the stakeholders involved in evaluation practices.
The second aim of the project is to investigate effects of the use of bibliometric methods for research evaluation, on the research system, and on and the academic researchers. This can be seen from two different points of view. One is how academic researchers themselves use bibliometric methods when evaluating their peers assessing applications for research grants and academic positions. The other is effects along the lines of scholars and scientists changing for instance publication behavior to adjust to evaluation systems and resource allocation models. While the former point of view might be more contextually related to traditions within research fields, the other might be more related to local or national research policies. The interplay between these perspectives is of high theoretical interest for the project to develop further, but has to a certain extent been investigated within a master’s thesis supervised within the project (Nästesjö, 2016); continued in a PhD-project, analyzing the relation between evaluation practices and academic socialization. The master’s thesis, suggest that disciplinary differences, career stage, and academic age are important factors for understanding the relation between research evaluation systems and research practices, where there are potential conflicts between disciplinary norms and political practices, conflicts that affect the behavior of individual scholars, as well as disciplinary norms. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/b9c0bad2-d7e5-4853-8f15-7a9e9f13eef2
- author
- Åström, Fredrik
LU
- organization
- publishing date
- 2018-05-25
- type
- Contribution to conference
- publication status
- unpublished
- subject
- conference name
- Scientific Communication and Gatekeeping in Academia in the 21st Century
- conference location
- Uppsala, Sweden
- conference dates
- 2018-05-24 - 2018-05-25
- project
- "Forskningen granskad": Användningen av prestationsindikatorer för att mäta akademisk forskning, praktiker och effekter
- language
- English
- LU publication?
- yes
- id
- b9c0bad2-d7e5-4853-8f15-7a9e9f13eef2
- date added to LUP
- 2018-11-16 09:52:19
- date last changed
- 2018-11-21 21:43:22
@misc{b9c0bad2-d7e5-4853-8f15-7a9e9f13eef2,
abstract = {{This paper presents an ongoing research project investigating the use of bibliometrics for research evaluation and resource allocation purposes, and to investigate the effects of these evaluation models on individual, organizational and political levels.<br>
A main focus has been on bibliometrics based evaluation practices. The use of bibliometrics for local resource allocation at Swedish universities was analyzed (Hammarfelt et.al, 2016; Hammarfelt & Åström, 2015), finding a great degree of variation both in how and where bibliometrics is used; and while the introduction of a bibliometrics based national model for resource allocation in Sweden in 2009 probably had an effect on the increasing use of bibliometrics in local allocation models, there are also great variation in terms of bibliometric indicators and allocation models being in use. But bibliometrics is not just being used in evaluation and resource allocation models at a systemic level. Another important part of the use of bibliometrics in research evaluation practices is how it is being used by academic scholars and scientists serving as external assessors of candidates for academic positions or of research funding proposals. This practice has been studied through analyses of application assessment reports in economics, history, and biomedical research at Swedish universities (Hammarfelt, forthcoming; Hammarfelt & Rushforth, in press).<br>
The other main focus of the project thus far has been on the underlying technologies, or the infrastructure of bibliometrics and research evaluation systems. A main aspect of the ‘infrastructure’ theme is the challenges of understanding complexities and interactions of different technical or auxiliary systems, and their role in evaluation processes. This has been investigated empirically in a minor study of the role of knowledge organization systems in bibliometric analysis and evaluation (Åström et.al, 2016). Another empirical study of infrastructures under development is a study of current research information systems (CRIS). From a theoretical perspective, a conceptual framework has been tentatively developed, describing infrastructures, not so much as specific technical systems, but as systems of relations between hardware, software and people (Åström, 2016). This was operationalized by defining: ‘hardware’ as databases, software for bibliometric analyses, and so on; ‘software’ as the evaluation systems, the bibliometric indicators, and the allocation models; and ‘people’ as the stakeholders involved in evaluation practices.<br>
The second aim of the project is to investigate effects of the use of bibliometric methods for research evaluation, on the research system, and on and the academic researchers. This can be seen from two different points of view. One is how academic researchers themselves use bibliometric methods when evaluating their peers assessing applications for research grants and academic positions. The other is effects along the lines of scholars and scientists changing for instance publication behavior to adjust to evaluation systems and resource allocation models. While the former point of view might be more contextually related to traditions within research fields, the other might be more related to local or national research policies. The interplay between these perspectives is of high theoretical interest for the project to develop further, but has to a certain extent been investigated within a master’s thesis supervised within the project (Nästesjö, 2016); continued in a PhD-project, analyzing the relation between evaluation practices and academic socialization. The master’s thesis, suggest that disciplinary differences, career stage, and academic age are important factors for understanding the relation between research evaluation systems and research practices, where there are potential conflicts between disciplinary norms and political practices, conflicts that affect the behavior of individual scholars, as well as disciplinary norms.}},
author = {{Åström, Fredrik}},
language = {{eng}},
month = {{05}},
title = {{Science audited: Indicator-based systems for research evaluation and resource allocation}},
url = {{https://lup.lub.lu.se/search/files/54176794/Astrom_SCGA_UU.pdf}},
year = {{2018}},
}