Learning Online Multi-sensor Depth Fusion
(2022) 17th European Conference on Computer Vision, ECCV 2022 In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 13692 LNCS. p.87-105- Abstract
Many hand-held or mixed reality devices are used with a single sensor for 3D reconstruction, although they often comprise multiple sensors. Multi-sensor depth fusion is able to substantially improve the robustness and accuracy of 3D reconstruction methods, but existing techniques are not robust enough to handle sensors which operate with diverse value ranges as well as noise and outlier statistics. To this end, we introduce SenFuNet,- a depth fusion approach that learns sensor-specific noise and outlier statistics and combines the data streams of depth frames from different sensors in an online fashion. Our method fuses multi-sensor depth streams regardless of time synchronization and calibration and generalizes well with little... (More)
Many hand-held or mixed reality devices are used with a single sensor for 3D reconstruction, although they often comprise multiple sensors. Multi-sensor depth fusion is able to substantially improve the robustness and accuracy of 3D reconstruction methods, but existing techniques are not robust enough to handle sensors which operate with diverse value ranges as well as noise and outlier statistics. To this end, we introduce SenFuNet,- a depth fusion approach that learns sensor-specific noise and outlier statistics and combines the data streams of depth frames from different sensors in an online fashion. Our method fuses multi-sensor depth streams regardless of time synchronization and calibration and generalizes well with little training data. We conduct experiments with various sensor combinations on the real-world CoRBS and Scene3D datasets, as well as the Replica dataset. Experiments demonstrate that our fusion strategy outperforms traditional and recent online depth fusion approaches. In addition, the combination of multiple sensors yields more robust outlier handling and more precise surface reconstruction than the use of a single sensor. The source code and data are available at https://github.com/tfy14esa/SenFuNet.
(Less)
- author
- Sandström, Erik ; Oswald, Martin R. ; Kumar, Suryansh ; Weder, Silvan ; Yu, Fisher ; Sminchisescu, Cristian LU and Van Gool, Luc
- organization
- publishing date
- 2022
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- host publication
- Computer Vision – ECCV 2022 - 17th European Conference, Proceedings
- series title
- Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
- editor
- Avidan, Shai ; Brostow, Gabriel ; Cissé, Moustapha ; Farinella, Giovanni Maria and Hassner, Tal
- volume
- 13692 LNCS
- pages
- 19 pages
- publisher
- Springer Science and Business Media B.V.
- conference name
- 17th European Conference on Computer Vision, ECCV 2022
- conference location
- Tel Aviv, Israel
- conference dates
- 2022-10-23 - 2022-10-27
- external identifiers
-
- scopus:85144576420
- ISSN
- 1611-3349
- 0302-9743
- ISBN
- 9783031198236
- DOI
- 10.1007/978-3-031-19824-3_6
- language
- English
- LU publication?
- yes
- id
- 075c2f09-9be2-4154-b908-10b511318927
- date added to LUP
- 2023-01-12 10:40:34
- date last changed
- 2024-04-04 15:32:08
@inproceedings{075c2f09-9be2-4154-b908-10b511318927,
abstract = {{<p>Many hand-held or mixed reality devices are used with a single sensor for 3D reconstruction, although they often comprise multiple sensors. Multi-sensor depth fusion is able to substantially improve the robustness and accuracy of 3D reconstruction methods, but existing techniques are not robust enough to handle sensors which operate with diverse value ranges as well as noise and outlier statistics. To this end, we introduce SenFuNet,- a depth fusion approach that learns sensor-specific noise and outlier statistics and combines the data streams of depth frames from different sensors in an online fashion. Our method fuses multi-sensor depth streams regardless of time synchronization and calibration and generalizes well with little training data. We conduct experiments with various sensor combinations on the real-world CoRBS and Scene3D datasets, as well as the Replica dataset. Experiments demonstrate that our fusion strategy outperforms traditional and recent online depth fusion approaches. In addition, the combination of multiple sensors yields more robust outlier handling and more precise surface reconstruction than the use of a single sensor. The source code and data are available at https://github.com/tfy14esa/SenFuNet.</p>}},
author = {{Sandström, Erik and Oswald, Martin R. and Kumar, Suryansh and Weder, Silvan and Yu, Fisher and Sminchisescu, Cristian and Van Gool, Luc}},
booktitle = {{Computer Vision – ECCV 2022 - 17th European Conference, Proceedings}},
editor = {{Avidan, Shai and Brostow, Gabriel and Cissé, Moustapha and Farinella, Giovanni Maria and Hassner, Tal}},
isbn = {{9783031198236}},
issn = {{1611-3349}},
language = {{eng}},
pages = {{87--105}},
publisher = {{Springer Science and Business Media B.V.}},
series = {{Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)}},
title = {{Learning Online Multi-sensor Depth Fusion}},
url = {{http://dx.doi.org/10.1007/978-3-031-19824-3_6}},
doi = {{10.1007/978-3-031-19824-3_6}},
volume = {{13692 LNCS}},
year = {{2022}},
}