Skip to main content

LUP Student Papers

LUND UNIVERSITY LIBRARIES

Enhancing Deep Learning-Based 3D Face Reconstructions with Consumer-Grade Depth Data

Messelt, Oscar LU and Reit, William LU (2025) In Master’s Theses in Mathematical Sciences FMAM05 20251
Mathematics (Faculty of Engineering)
Abstract
Deep learning models have significantly improved the advancement of monocular 3D
face reconstruction models. However, such models often struggle to capture
person-specific high-frequency details. To address this issue, this report proposes a
hybrid reconstruction pipeline that enhances DECA produced 3D face models by
incorporating RGB-D data captured with the consumer grade iPhone TrueDepth
camera. The final pipeline uses DECA output, synchronized RGB-D, face
segmentation, landmark detection, and ICP. Starting from a DECA-generated coarse
mesh, we fit an independent mesh copy to each depth scan, constrain vertex
displacements to the surface normal, and compute the per-vertex mean across all fitted
meshes to obtain a refined,... (More)
Deep learning models have significantly improved the advancement of monocular 3D
face reconstruction models. However, such models often struggle to capture
person-specific high-frequency details. To address this issue, this report proposes a
hybrid reconstruction pipeline that enhances DECA produced 3D face models by
incorporating RGB-D data captured with the consumer grade iPhone TrueDepth
camera. The final pipeline uses DECA output, synchronized RGB-D, face
segmentation, landmark detection, and ICP. Starting from a DECA-generated coarse
mesh, we fit an independent mesh copy to each depth scan, constrain vertex
displacements to the surface normal, and compute the per-vertex mean across all fitted
meshes to obtain a refined, personalized shape. This method allows for correcting 3D
facial models to better reflect individual characteristics. The corrected reconstructions
show an improved detail correspondence to the target person, having converged
towards the mean shape of several depth scans. Since this method only modifies
existing meshes, the resulting personalized mesh is still morphable. (Less)
Popular Abstract
There are Artificial Intelligence (AI) models that, given a picture of a face, can create
3D models of that face. These modes create good results, however, they can be a
bit too general and can miss person specific details. Therefore, during our master’s
thesis, we investigated a method to make the 3D facial model more person specific
using the Face ID camera found on iPhones.
Please use this url to cite or link to this publication:
author
Messelt, Oscar LU and Reit, William LU
supervisor
organization
course
FMAM05 20251
year
type
H2 - Master's Degree (Two Years)
subject
keywords
3D Face Reconstruction, Deep Learning, RGB-D, Consumer-Grade Depth Sensors, iPhone TrueDepth, DECA, FLAME
publication/series
Master’s Theses in Mathematical Sciences
report number
LUTFMA-3579-2025
ISSN
1404-6342
other publication id
2025:E30
language
English
id
9211740
date added to LUP
2025-09-15 11:09:43
date last changed
2025-09-15 11:09:43
@misc{9211740,
  abstract     = {{Deep learning models have significantly improved the advancement of monocular 3D
face reconstruction models. However, such models often struggle to capture
person-specific high-frequency details. To address this issue, this report proposes a
hybrid reconstruction pipeline that enhances DECA produced 3D face models by
incorporating RGB-D data captured with the consumer grade iPhone TrueDepth
camera. The final pipeline uses DECA output, synchronized RGB-D, face
segmentation, landmark detection, and ICP. Starting from a DECA-generated coarse
mesh, we fit an independent mesh copy to each depth scan, constrain vertex
displacements to the surface normal, and compute the per-vertex mean across all fitted
meshes to obtain a refined, personalized shape. This method allows for correcting 3D
facial models to better reflect individual characteristics. The corrected reconstructions
show an improved detail correspondence to the target person, having converged
towards the mean shape of several depth scans. Since this method only modifies
existing meshes, the resulting personalized mesh is still morphable.}},
  author       = {{Messelt, Oscar and Reit, William}},
  issn         = {{1404-6342}},
  language     = {{eng}},
  note         = {{Student Paper}},
  series       = {{Master’s Theses in Mathematical Sciences}},
  title        = {{Enhancing Deep Learning-Based 3D Face Reconstructions with Consumer-Grade Depth Data}},
  year         = {{2025}},
}