Monitoring with MLOps for Clinical Decision Support
(2024) BMEM01 20241Department of Biomedical Engineering
- Abstract
- The development of Artificial Intelligence(AI) and Machine Learning(ML) has highly increased over the past decades, with numerous research projects developing AI and ML models for clinical decision support. Many of these projects showcase promising results in the lab, however, when set in a real-life clinical setting most models perform significantly worse, resulting in only a few models transitioning from a lab environment to clinical use. One reason of many for the discrepancy between research and commercial application is the absence, and thereof need, for established development processes that deploy and monitor ML models for clinical use in hospital environments. This thesis aimed to implement a model monitoring framework utilizing... (More)
- The development of Artificial Intelligence(AI) and Machine Learning(ML) has highly increased over the past decades, with numerous research projects developing AI and ML models for clinical decision support. Many of these projects showcase promising results in the lab, however, when set in a real-life clinical setting most models perform significantly worse, resulting in only a few models transitioning from a lab environment to clinical use. One reason of many for the discrepancy between research and commercial application is the absence, and thereof need, for established development processes that deploy and monitor ML models for clinical use in hospital environments. This thesis aimed to implement a model monitoring framework utilizing Machine Learning Operations(MLOps), a development process focusing on streamlining development, monitoring, and maintenance, to detect performance variations in clinical decision support systems. The monitoring framework was implemented and evaluated by creating a use-case utilizing the MIMIC-IV-2.2 data set, which comprises real-life patient data from electronic health records, for training, validating, and deploying a novel neural network model that predicts the Length of Stay(LoS) of a patient admitted to the Intensive Care Unit (ICU) of a hospital. The data was split into training- and inference data, based on age groups, where the former was used for model training, and theclatter for evaluating real-life performance via model monitoring. Finally, the performance of the model was evaluated by a set of metrics to determine if performance drift occurred with the inference data. Resultingly, an end-to-end framework was implemented with successful monitoring, able to detect a significant performance drift and loss of performance. The more significant metrics, Mean Square Log Error (MSLE) and Mean Absolute Percentage Error (MAPE) detected a performance loss of 44.64% and 41.13% respectively, compared to model performance on the training data. The MLOps framework used in the thesis fulfilled the use-case’s intentions, had a clear structure, and showed efficiency. However, there are limitations involving no retraining of the ML model implemented in the use-case, the disparity from a real-life clinical environment, and whether the framework used for the thesis is appropriate for similar development processes, illustrating questions that need to be addressed with further research. The MLOps framework however shows promise, employing tools and practices that can further advance the transition, and integration of AI and ML-based technology in clinical environments. (Less)
- Popular Abstract
- A Monitoring Framework for Machine Learning in Healthcare
Developing Machine Learning (ML) models for healthcare is a complicated task. As a result, few ML applications reach the clinic, and fewer show success for their applied purpose. However, novel monitoring frameworks that manage the quirks of ML can apply Model Monitoring, ensuring the effectiveness and safety of healthcare applications. In the future, combinations of novel techniques can further the integration of ML in healthcare.
Please use this url to cite or link to this publication:
http://lup.lub.lu.se/student-papers/record/9151823
- author
- Ekblom, Adam LU
- supervisor
- organization
- course
- BMEM01 20241
- year
- 2024
- type
- H2 - Master's Degree (Two Years)
- subject
- keywords
- MLOPs, Clinical Decision Support, Machine Learning
- language
- English
- additional info
- 2024-08
- id
- 9151823
- date added to LUP
- 2024-05-20 08:47:00
- date last changed
- 2024-05-20 08:47:00
@misc{9151823,
abstract = {{The development of Artificial Intelligence(AI) and Machine Learning(ML) has highly increased over the past decades, with numerous research projects developing AI and ML models for clinical decision support. Many of these projects showcase promising results in the lab, however, when set in a real-life clinical setting most models perform significantly worse, resulting in only a few models transitioning from a lab environment to clinical use. One reason of many for the discrepancy between research and commercial application is the absence, and thereof need, for established development processes that deploy and monitor ML models for clinical use in hospital environments. This thesis aimed to implement a model monitoring framework utilizing Machine Learning Operations(MLOps), a development process focusing on streamlining development, monitoring, and maintenance, to detect performance variations in clinical decision support systems. The monitoring framework was implemented and evaluated by creating a use-case utilizing the MIMIC-IV-2.2 data set, which comprises real-life patient data from electronic health records, for training, validating, and deploying a novel neural network model that predicts the Length of Stay(LoS) of a patient admitted to the Intensive Care Unit (ICU) of a hospital. The data was split into training- and inference data, based on age groups, where the former was used for model training, and theclatter for evaluating real-life performance via model monitoring. Finally, the performance of the model was evaluated by a set of metrics to determine if performance drift occurred with the inference data. Resultingly, an end-to-end framework was implemented with successful monitoring, able to detect a significant performance drift and loss of performance. The more significant metrics, Mean Square Log Error (MSLE) and Mean Absolute Percentage Error (MAPE) detected a performance loss of 44.64% and 41.13% respectively, compared to model performance on the training data. The MLOps framework used in the thesis fulfilled the use-case’s intentions, had a clear structure, and showed efficiency. However, there are limitations involving no retraining of the ML model implemented in the use-case, the disparity from a real-life clinical environment, and whether the framework used for the thesis is appropriate for similar development processes, illustrating questions that need to be addressed with further research. The MLOps framework however shows promise, employing tools and practices that can further advance the transition, and integration of AI and ML-based technology in clinical environments.}},
author = {{Ekblom, Adam}},
language = {{eng}},
note = {{Student Paper}},
title = {{Monitoring with MLOps for Clinical Decision Support}},
year = {{2024}},
}