Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models

Elfer, Katherine ; Gardecki, Emma ; Garcia, Victor ; Ly, Amy ; Hytopoulos, Evangelos ; Wen, Si ; Hanna, Matthew G ; Peeters, Dieter Je ; Saltz, Joel and Ehinger, Anna LU orcid , et al. (2024) In Modern Pathology : an official journal of the United States and Canadian Academy of Pathology, Inc 37(4).
Abstract

This work advances and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the CONSORT (Consolidated Standards of Reporting Trials) and SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) checklists and the proposed AI extensions to the STARD (Standards for Reporting Diagnostic Accuracy) and TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) checklists. AI for detection and/or diagnostic image analysis requires complete,... (More)

This work advances and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the CONSORT (Consolidated Standards of Reporting Trials) and SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) checklists and the proposed AI extensions to the STARD (Standards for Reporting Diagnostic Accuracy) and TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) checklists. AI for detection and/or diagnostic image analysis requires complete, reproducible, and transparent reporting of the annotations and metadata used in training and testing datasets. Prior work by Wahab et al. proposed an annotation workflow and quality checklist for computational pathology annotations. In this manuscript, we operationalize this workflow into an evaluable quality checklist that applies to any reader-interpreted medical images, and we demonstrate its use for an annotation effort in digital pathology. We refer to this quality framework as CLEARR-AI: The Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence.

(Less)
Please use this url to cite or link to this publication:
author
; ; ; ; ; ; ; ; and , et al. (More)
; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; and (Less)
organization
publishing date
type
Contribution to journal
publication status
published
subject
in
Modern Pathology : an official journal of the United States and Canadian Academy of Pathology, Inc
volume
37
issue
4
article number
100439
publisher
Nature Publishing Group
external identifiers
  • pmid:38286221
  • scopus:85186576010
ISSN
1530-0285
DOI
10.1016/j.modpat.2024.100439
language
English
LU publication?
yes
additional info
Copyright © 2024. Published by Elsevier Inc.
id
d314ef9f-2cd4-4c5d-918e-c1750b089817
date added to LUP
2024-02-29 17:21:04
date last changed
2024-12-20 10:24:02
@article{d314ef9f-2cd4-4c5d-918e-c1750b089817,
  abstract     = {{<p>This work advances and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the CONSORT (Consolidated Standards of Reporting Trials) and SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) checklists and the proposed AI extensions to the STARD (Standards for Reporting Diagnostic Accuracy) and TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) checklists. AI for detection and/or diagnostic image analysis requires complete, reproducible, and transparent reporting of the annotations and metadata used in training and testing datasets. Prior work by Wahab et al. proposed an annotation workflow and quality checklist for computational pathology annotations. In this manuscript, we operationalize this workflow into an evaluable quality checklist that applies to any reader-interpreted medical images, and we demonstrate its use for an annotation effort in digital pathology. We refer to this quality framework as CLEARR-AI: The Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence.</p>}},
  author       = {{Elfer, Katherine and Gardecki, Emma and Garcia, Victor and Ly, Amy and Hytopoulos, Evangelos and Wen, Si and Hanna, Matthew G and Peeters, Dieter Je and Saltz, Joel and Ehinger, Anna and Dudgeon, Sarah N and Li, Xiaoxian and Blenman, Kim Rm and Chen, Weijie and Green, Ursula and Birmingham, Ryan and Pan, Tony and Lennerz, Jochen K and Salgado, Roberto and Gallas, Brandon D}},
  issn         = {{1530-0285}},
  language     = {{eng}},
  month        = {{01}},
  number       = {{4}},
  publisher    = {{Nature Publishing Group}},
  series       = {{Modern Pathology : an official journal of the United States and Canadian Academy of Pathology, Inc}},
  title        = {{Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models}},
  url          = {{http://dx.doi.org/10.1016/j.modpat.2024.100439}},
  doi          = {{10.1016/j.modpat.2024.100439}},
  volume       = {{37}},
  year         = {{2024}},
}