Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta
(2021) In Journal of Trauma and Acute Care Surgery 91(4). p.663-671- Abstract
BACKGROUND Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE. METHODS This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and... (More)
BACKGROUND Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE. METHODS This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard. RESULTS Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed. CONCLUSION Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts.
(Less)
- author
- organization
- publishing date
- 2021-10-01
- type
- Contribution to journal
- publication status
- published
- subject
- keywords
- clinical competence, Resuscitative endovascular balloon occlusion of the aorta, simulation, skills, validation, validity evidence, vascular access, assessment tool
- in
- Journal of Trauma and Acute Care Surgery
- volume
- 91
- issue
- 4
- pages
- 9 pages
- publisher
- Lippincott Williams & Wilkins
- external identifiers
-
- pmid:34225347
- scopus:85116808410
- ISSN
- 2163-0755
- DOI
- 10.1097/TA.0000000000003338
- language
- English
- LU publication?
- yes
- additional info
- Publisher Copyright: © 2021 Wolters Kluwer Health, Inc. All rights reserved.
- id
- 40bbc4d2-e673-4fb9-ba06-fb108c8bed84
- date added to LUP
- 2021-11-12 14:48:37
- date last changed
- 2024-04-20 15:08:21
@article{40bbc4d2-e673-4fb9-ba06-fb108c8bed84,
abstract = {{<p>BACKGROUND Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE. METHODS This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard. RESULTS Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed. CONCLUSION Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts. </p>}},
author = {{Engberg, Morten and Lönn, Lars and Konge, Lars and Mikkelsen, Søren and Hörer, Tal and Lindgren, Hans and Søvik, Edmund and Svendsen, Morten Bo and Frendø, Martin and Taudorf, Mikkel and Russell, Lene}},
issn = {{2163-0755}},
keywords = {{clinical competence; Resuscitative endovascular balloon occlusion of the aorta; simulation; skills; validation; validity evidence; vascular access, assessment tool}},
language = {{eng}},
month = {{10}},
number = {{4}},
pages = {{663--671}},
publisher = {{Lippincott Williams & Wilkins}},
series = {{Journal of Trauma and Acute Care Surgery}},
title = {{Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta}},
url = {{http://dx.doi.org/10.1097/TA.0000000000003338}},
doi = {{10.1097/TA.0000000000003338}},
volume = {{91}},
year = {{2021}},
}