Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Real-time data processing for serial crystallography experiments

White, Thomas ; Schoof, Tim ; Yakubov, Sergey ; Tolstikova, Aleksandra ; Middendorf, Philipp ; Karnevskiy, Mikhail ; Mariani, Valerio ; Henkel, Alessandra ; Klopprogge, Bjarne and Hannappel, Juergen , et al. (2025) In IUCrJ 12(Pt 1). p.97-108
Abstract

We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames (`hits'). The average... (More)

We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames (`hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.

(Less)
Please use this url to cite or link to this publication:
author
; ; ; ; ; ; ; ; and , et al. (More)
; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; and (Less)
publishing date
type
Contribution to journal
publication status
published
subject
keywords
crystallography, synchrotron, serial crystallography, XFEL, data processing
in
IUCrJ
volume
12
issue
Pt 1
pages
12 pages
publisher
International Union of Crystallography
external identifiers
  • pmid:39714221
  • scopus:85214474208
ISSN
2052-2525
DOI
10.1107/S2052252524011837
language
English
LU publication?
no
additional info
open access.
id
79e776c4-7dc3-4fd4-adb5-5b7c6ffb4bf4
date added to LUP
2025-04-21 20:38:46
date last changed
2025-07-15 11:10:46
@article{79e776c4-7dc3-4fd4-adb5-5b7c6ffb4bf4,
  abstract     = {{<p>We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames (`hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.</p>}},
  author       = {{White, Thomas and Schoof, Tim and Yakubov, Sergey and Tolstikova, Aleksandra and Middendorf, Philipp and Karnevskiy, Mikhail and Mariani, Valerio and Henkel, Alessandra and Klopprogge, Bjarne and Hannappel, Juergen and Oberthuer, Dominik and De Gennaro Aquino, Ivan and Egorov, Dmitry and Munke, Anna and Sprenger, Janina and Pompidor, Guillaume and Taberman, Helena and Gruzinov, Andrey and Meyer, Jan and Hakanpää, Johanna and Gasthuber, Martin}},
  issn         = {{2052-2525}},
  keywords     = {{crystallography; synchrotron; serial crystallography; XFEL; data processing}},
  language     = {{eng}},
  month        = {{01}},
  number       = {{Pt 1}},
  pages        = {{97--108}},
  publisher    = {{International Union of Crystallography}},
  series       = {{IUCrJ}},
  title        = {{Real-time data processing for serial crystallography experiments}},
  url          = {{http://dx.doi.org/10.1107/S2052252524011837}},
  doi          = {{10.1107/S2052252524011837}},
  volume       = {{12}},
  year         = {{2025}},
}