Calibration, Positioning and Tracking in a Refractive and Reflective Scene

Abstract:
We propose a framework for calibration, positioning
and tracking in a scene viewed by multiple cameras, through
a flat refractive surface and one or several flat reflective walls.
Refractions are explicitly modeled by Snell’s law and reflections
are handled using virtual points. A novel bundle adjustment
framework is introduced for solving the nonlinear equations
of refractions and the linear equations of reflections, which in
addition enables optimization for calibration and positioning. The
numerical accuracy of the solutions is investigated on synthetic
data, and the influence of noise in image points for several settings
of refractive and reflective planes is presented. The performance of
the framework is evaluated on real data and confirms the validity
of the physical model. Examples of how to use the framework
to back-project image coordinates, forward-project scene points
and estimate the refractive and reflective planes are presented.
Lastly, an application of the system on real data from a biological
experiment on small aquatic organisms is presented.