AbspectroscoPY, a Python toolbox for absorbance-based sensor data in water quality monitoring
(2022) In Environmental Science: Water Research & Technology 8(4). p.836-848- Abstract
- The long-term trend of increasing natural organic matter (NOM) in boreal and north European surface waters represents an economic and environmental challenge for drinking water treatment plants (DWTPs). High-frequency measurements from absorbance-based online spectrophotometers are often used in modern DWTPs to measure the chromophoric fraction of dissolved organic matter (CDOM) over time. These data contain valuable information that can be used to optimise NOM removal at various stages of treatment and/or diagnose the causes of underperformance at the DWTP. However, automated monitoring systems generate large datasets that need careful preprocessing, followed by variable selection and signal processing before interpretation. In this work... (More)
- The long-term trend of increasing natural organic matter (NOM) in boreal and north European surface waters represents an economic and environmental challenge for drinking water treatment plants (DWTPs). High-frequency measurements from absorbance-based online spectrophotometers are often used in modern DWTPs to measure the chromophoric fraction of dissolved organic matter (CDOM) over time. These data contain valuable information that can be used to optimise NOM removal at various stages of treatment and/or diagnose the causes of underperformance at the DWTP. However, automated monitoring systems generate large datasets that need careful preprocessing, followed by variable selection and signal processing before interpretation. In this work we introduce AbspectroscoPY (“Absorbance spectroscopic analysis in
Python”), a Python toolbox for processing time-series datasets collected by in situ spectrophotometers. The toolbox addresses some of the main challenges in data preprocessing by handling duplicates, systematic time shifts, baseline corrections and outliers. It contains automated functions to compute a range of spectral metrics for the time-series data, including absorbance ratios, exponential fits, slope ratios and spectral slope curves. To demonstrate its utility, AbspectroscoPY was applied to 15-month datasets from three online
spectrophotometers in a drinking water treatment plant. Despite only small variations in surface water quality over the time period, variability in the spectrophotometric profiles of treated water could be identified, quantified and related to lake turnover or operational changes in the DWTP. This toolbox
represents a step toward automated early warning systems for detecting and responding to potential threats to treatment performance caused by rapid changes in incoming water quality. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/9b3204a3-b181-4087-b29d-a7284723288a
- author
- Cascone, Claudia ; Murphy, Kathleen R. ; Markensten, H. ; Kern, J.S. ; Schleich, C. ; Keucken, Alexander LU and Köhler, S.J.
- organization
- publishing date
- 2022-02-23
- type
- Contribution to journal
- publication status
- published
- subject
- keywords
- Python, Scientific Computing, NOM removal, Drinking Water
- in
- Environmental Science: Water Research & Technology
- volume
- 8
- issue
- 4
- pages
- 13 pages
- publisher
- Royal Society of Chemistry
- external identifiers
-
- scopus:85129062546
- ISSN
- 2053-1419
- DOI
- 10.1039/D1EW00416F
- language
- English
- LU publication?
- yes
- id
- 9b3204a3-b181-4087-b29d-a7284723288a
- date added to LUP
- 2022-03-03 09:35:25
- date last changed
- 2022-07-01 14:27:50
@article{9b3204a3-b181-4087-b29d-a7284723288a,
abstract = {{The long-term trend of increasing natural organic matter (NOM) in boreal and north European surface waters represents an economic and environmental challenge for drinking water treatment plants (DWTPs). High-frequency measurements from absorbance-based online spectrophotometers are often used in modern DWTPs to measure the chromophoric fraction of dissolved organic matter (CDOM) over time. These data contain valuable information that can be used to optimise NOM removal at various stages of treatment and/or diagnose the causes of underperformance at the DWTP. However, automated monitoring systems generate large datasets that need careful preprocessing, followed by variable selection and signal processing before interpretation. In this work we introduce AbspectroscoPY (“Absorbance spectroscopic analysis in<br/>Python”), a Python toolbox for processing time-series datasets collected by in situ spectrophotometers. The toolbox addresses some of the main challenges in data preprocessing by handling duplicates, systematic time shifts, baseline corrections and outliers. It contains automated functions to compute a range of spectral metrics for the time-series data, including absorbance ratios, exponential fits, slope ratios and spectral slope curves. To demonstrate its utility, AbspectroscoPY was applied to 15-month datasets from three online<br/>spectrophotometers in a drinking water treatment plant. Despite only small variations in surface water quality over the time period, variability in the spectrophotometric profiles of treated water could be identified, quantified and related to lake turnover or operational changes in the DWTP. This toolbox<br/>represents a step toward automated early warning systems for detecting and responding to potential threats to treatment performance caused by rapid changes in incoming water quality.}},
author = {{Cascone, Claudia and Murphy, Kathleen R. and Markensten, H. and Kern, J.S. and Schleich, C. and Keucken, Alexander and Köhler, S.J.}},
issn = {{2053-1419}},
keywords = {{Python, Scientific Computing; NOM removal; Drinking Water}},
language = {{eng}},
month = {{02}},
number = {{4}},
pages = {{836--848}},
publisher = {{Royal Society of Chemistry}},
series = {{Environmental Science: Water Research & Technology}},
title = {{AbspectroscoPY, a Python toolbox for absorbance-based sensor data in water quality monitoring}},
url = {{http://dx.doi.org/10.1039/D1EW00416F}},
doi = {{10.1039/D1EW00416F}},
volume = {{8}},
year = {{2022}},
}