What do practitioners expect from an evaluation report? A qualitative analysis of Dutch crisis management professionals' expectations
Beerens, Ralf Josef Johanna LU
and Haverhoek-Mieremet, Karin
(2020)
In International Journal of Emergency Services
10(1).
p.1-25
- Abstract
Purpose: This paper seeks to overcome the mismatch between evaluation reports and the expectations of the target audience, by identifying crisis management professionals' expectations.
Design/methodology/approach: An adapted stakeholder information analysis was used to survey the expectations of 84 crisis management professionals in the Netherlands. A general inductive analysis was applied to qualitative data, from which five main themes emerged: purpose; object or focus; reasoning and (meta) analysis; result or conclusion, and the overall design of the evaluation.
Findings: Currently, evaluation reports are seen merely as a way to share experience and support thinking about how to avoid repeating mistakes. However, most... (More)
Purpose: This paper seeks to overcome the mismatch between evaluation reports and the expectations of the target audience, by identifying crisis management professionals' expectations.
Design/methodology/approach: An adapted stakeholder information analysis was used to survey the expectations of 84 crisis management professionals in the Netherlands. A general inductive analysis was applied to qualitative data, from which five main themes emerged: purpose; object or focus; reasoning and (meta) analysis; result or conclusion, and the overall design of the evaluation.
Findings: Currently, evaluation reports are seen merely as a way to share experience and support thinking about how to avoid repeating mistakes. However, most respondents expected them to contribute to learning and support improvement. They should provide actionable feedback on what could be done differently or better, and indicate how this can be achieved. Respondents emphasised the need to focus on the human factor and not neglect the context. The wide variety of views underlined that it is difficult to create one evaluation product that meets all expectations.
Research limitations/implications: Although some major themes clearly emerged from the data, it is unclear how they relate to each other, and their relative importance. In addition, no distinction is made between evaluations of real events and simulations.
Practical implications: Users should be encouraged to provide input into the evaluation process by clarifying their needs and how they use evaluation reports.
Originality/value: This research is the first attempt to identify user expectations regarding what constitutes an effective evaluation.
(Less)
- author
- Beerens, Ralf Josef Johanna
LU
and Haverhoek-Mieremet, Karin
- organization
- publishing date
- 2020-08-27
- type
- Contribution to journal
- publication status
- published
- subject
- keywords
- Crisis, Disaster, Emergency, Evaluation, Evaluation design, Evaluation report, Exercise, Expectations, Response, Simulation, Survey, The Netherlands, Usefulness
- in
- International Journal of Emergency Services
- volume
- 10
- issue
- 1
- pages
- 1 - 25
- publisher
- Emerald Group Publishing Limited
- external identifiers
-
- scopus:85089786355
- ISSN
- 2047-0894
- DOI
- 10.1108/IJES-12-2019-0063
- language
- English
- LU publication?
- yes
- additional info
- Ralf Josef Johanna Beerens is a Senior Researcher at the Institute for Safety (IFV) of the Netherlands. He has shared responsibility for the Institute’s Master’s programme in Crisis and Public Order Management (MCPM). In addition, he is a PhD candidate, focusing on multi-organisational emergency response (exercise) evaluation at the Division of Risk Management and Societal Safety at Lund University in Sweden. He is particularly interested in the evaluation of the operational performance of (international) emergency response organizations, teams or modules during exercises and crises, which reflects his professional experience as an evaluator. Ralf Josef Johanna Beerens is the corresponding author and can be contacted at: ralf.beerens@ifv.nl
- id
- 25b2dc99-673f-48ed-9004-2e703554e4e4
- date added to LUP
- 2020-08-31 09:22:28
- date last changed
- 2022-04-19 00:34:06
@article{25b2dc99-673f-48ed-9004-2e703554e4e4,
abstract = {{<p>Purpose: This paper seeks to overcome the mismatch between evaluation reports and the expectations of the target audience, by identifying crisis management professionals' expectations. </p><p>Design/methodology/approach: An adapted stakeholder information analysis was used to survey the expectations of 84 crisis management professionals in the Netherlands. A general inductive analysis was applied to qualitative data, from which five main themes emerged: purpose; object or focus; reasoning and (meta) analysis; result or conclusion, and the overall design of the evaluation. </p><p>Findings: Currently, evaluation reports are seen merely as a way to share experience and support thinking about how to avoid repeating mistakes. However, most respondents expected them to contribute to learning and support improvement. They should provide actionable feedback on what could be done differently or better, and indicate how this can be achieved. Respondents emphasised the need to focus on the human factor and not neglect the context. The wide variety of views underlined that it is difficult to create one evaluation product that meets all expectations. </p><p>Research limitations/implications: Although some major themes clearly emerged from the data, it is unclear how they relate to each other, and their relative importance. In addition, no distinction is made between evaluations of real events and simulations. </p><p>Practical implications: Users should be encouraged to provide input into the evaluation process by clarifying their needs and how they use evaluation reports. </p><p>Originality/value: This research is the first attempt to identify user expectations regarding what constitutes an effective evaluation.</p>}},
author = {{Beerens, Ralf Josef Johanna and Haverhoek-Mieremet, Karin}},
issn = {{2047-0894}},
keywords = {{Crisis; Disaster; Emergency; Evaluation; Evaluation design; Evaluation report; Exercise; Expectations; Response; Simulation; Survey; The Netherlands; Usefulness}},
language = {{eng}},
month = {{08}},
number = {{1}},
pages = {{1--25}},
publisher = {{Emerald Group Publishing Limited}},
series = {{International Journal of Emergency Services}},
title = {{What do practitioners expect from an evaluation report? A qualitative analysis of Dutch crisis management professionals' expectations}},
url = {{http://dx.doi.org/10.1108/IJES-12-2019-0063}},
doi = {{10.1108/IJES-12-2019-0063}},
volume = {{10}},
year = {{2020}},
}