Advanced

The REal Life EVidence AssessmeNt Tool (RELEVANT) : Development of a novel quality assurance asset to rate observational comparative effectiveness research studies

Campbell, Jonathan D.; Perry, Robert; Papadopoulos, Nikolaos G.; Krishnan, Jerry; Brusselle, Guy; Chisholm, Alison; Bjermer, Leif LU ; Thomas, Michael; Van Ganse, Eric and Van Den Berge, Maarten, et al. (2019) In Clinical and Translational Allergy 9(1).
Abstract

Background: Evidence from observational comparative effectiveness research (CER) is ranked below that from randomized controlled trials in traditional evidence hierarchies. However, asthma observational CER studies represent an important complementary evidence source answering different research questions and are particularly valuable in guiding clinical decision making in real-life patient and practice settings. Tools are required to assist in quality appraisal of observational CER to enable identification of and confidence in high-quality CER evidence to inform guideline development. Methods: The REal Life EVidence AssessmeNt Tool (RELEVANT) was developed through a step-wise approach. We conducted an iterative refinement of the tool... (More)

Background: Evidence from observational comparative effectiveness research (CER) is ranked below that from randomized controlled trials in traditional evidence hierarchies. However, asthma observational CER studies represent an important complementary evidence source answering different research questions and are particularly valuable in guiding clinical decision making in real-life patient and practice settings. Tools are required to assist in quality appraisal of observational CER to enable identification of and confidence in high-quality CER evidence to inform guideline development. Methods: The REal Life EVidence AssessmeNt Tool (RELEVANT) was developed through a step-wise approach. We conducted an iterative refinement of the tool based on Task Force member expertise and feedback from pilot testing the tool until reaching adequate inter-rater agreement percentages. Two distinct pilots were conducted - the first involving six members of the Respiratory Effectiveness Group (REG) and European Academy of Allergy and Clinical Immunology (EAACI) joint Task Force for quality appraisal of observational asthma CER; the second involving 22 members of REG and EAACI membership. The final tool consists of 21 quality sub-items distributed across seven methodology domains: Background, Design, Measures, Analysis, Results, Discussion/Interpretation, and Conflict of Interest. Eleven of these sub-items are considered critical and named "primary sub-items". Results: Following the second pilot, RELEVANT showed inter-rater agreement ≥ 70% for 94% of all primary and 93% for all secondary sub-items tested across three rater groups. For observational CER to be classified as sufficiently high quality for future guideline consideration, all RELEVANT primary sub-items must be fulfilled. The ten secondary sub-items further qualify the relative strengths and weaknesses of the published CER evidence. RELEVANT could also be applicable to general quality appraisal of observational CER across other medical specialties. Conclusions: RELEVANT is the first quality checklist to assist in the appraisal of published observational CER developed through iterative feedback derived from pilot implementation and inter-rater agreement evaluation. Developed for a REG-EAACI Task Force quality appraisal of recent asthma CER, RELEVANT also has wider utility to support appraisal of CER literature in general (including pre-publication). It may also assist in manuscript development and in educating relevant stakeholders about key quality markers in observational CER.

(Less)
Please use this url to cite or link to this publication:
author
, et al. (More)
(Less)
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
Assessment tool, Asthma, Comparative effectiveness research (CER), Observational studies, Quality
in
Clinical and Translational Allergy
volume
9
issue
1
publisher
BioMed Central
external identifiers
  • scopus:85063421513
ISSN
2045-7022
DOI
10.1186/s13601-019-0256-9
language
English
LU publication?
yes
id
3f6fb678-4eab-4300-b1c0-f9ebc213b420
date added to LUP
2019-04-08 09:13:10
date last changed
2019-05-21 04:20:20
@article{3f6fb678-4eab-4300-b1c0-f9ebc213b420,
  abstract     = {<p>Background: Evidence from observational comparative effectiveness research (CER) is ranked below that from randomized controlled trials in traditional evidence hierarchies. However, asthma observational CER studies represent an important complementary evidence source answering different research questions and are particularly valuable in guiding clinical decision making in real-life patient and practice settings. Tools are required to assist in quality appraisal of observational CER to enable identification of and confidence in high-quality CER evidence to inform guideline development. Methods: The REal Life EVidence AssessmeNt Tool (RELEVANT) was developed through a step-wise approach. We conducted an iterative refinement of the tool based on Task Force member expertise and feedback from pilot testing the tool until reaching adequate inter-rater agreement percentages. Two distinct pilots were conducted - the first involving six members of the Respiratory Effectiveness Group (REG) and European Academy of Allergy and Clinical Immunology (EAACI) joint Task Force for quality appraisal of observational asthma CER; the second involving 22 members of REG and EAACI membership. The final tool consists of 21 quality sub-items distributed across seven methodology domains: Background, Design, Measures, Analysis, Results, Discussion/Interpretation, and Conflict of Interest. Eleven of these sub-items are considered critical and named "primary sub-items". Results: Following the second pilot, RELEVANT showed inter-rater agreement ≥ 70% for 94% of all primary and 93% for all secondary sub-items tested across three rater groups. For observational CER to be classified as sufficiently high quality for future guideline consideration, all RELEVANT primary sub-items must be fulfilled. The ten secondary sub-items further qualify the relative strengths and weaknesses of the published CER evidence. RELEVANT could also be applicable to general quality appraisal of observational CER across other medical specialties. Conclusions: RELEVANT is the first quality checklist to assist in the appraisal of published observational CER developed through iterative feedback derived from pilot implementation and inter-rater agreement evaluation. Developed for a REG-EAACI Task Force quality appraisal of recent asthma CER, RELEVANT also has wider utility to support appraisal of CER literature in general (including pre-publication). It may also assist in manuscript development and in educating relevant stakeholders about key quality markers in observational CER.</p>},
  articleno    = {21},
  author       = {Campbell, Jonathan D. and Perry, Robert and Papadopoulos, Nikolaos G. and Krishnan, Jerry and Brusselle, Guy and Chisholm, Alison and Bjermer, Leif and Thomas, Michael and Van Ganse, Eric and Van Den Berge, Maarten and Quint, Jennifer and Price, David and Roche, Nicolas},
  issn         = {2045-7022},
  keyword      = {Assessment tool,Asthma,Comparative effectiveness research (CER),Observational studies,Quality},
  language     = {eng},
  month        = {03},
  number       = {1},
  publisher    = {BioMed Central},
  series       = {Clinical and Translational Allergy},
  title        = {The REal Life EVidence AssessmeNt Tool (RELEVANT) : Development of a novel quality assurance asset to rate observational comparative effectiveness research studies},
  url          = {http://dx.doi.org/10.1186/s13601-019-0256-9},
  volume       = {9},
  year         = {2019},
}