Towards three-dimensional point cloud reconstruction of fish swimming

Zebrafish is extensively used in behavioral, pharmacological, and neurological studies due to a number of methodological and practical advantages, including genetic and neurobiological homologies with humans and a fully sequenced genome. Critical to a biologically-based understanding of zebrafish behavior is the ability to reconstruct their complex behavioral repertoire in three-dimensions. Toward this aim, several efforts have been made to score their ethogram in three-dimensions, but most of these studies are constrained by a single-view imaging. A promising line of approach to extract refined information about the mechanosensory and perceptual systems of zebrafish is point cloud reconstruction. Here, we provide an initial review of the state of knowledge in zebrafish tracking and we propose a potential methodology that can capture the dynamic three-dimensional geometry of fish swimming. We utilize a stereo vision camera, calibrated with a pinhole camera model with refraction correction to allow for multi-medium imaging. The corrected pinhole camera model accounts for refraction through multiple mediums and allows for more accurate point cloud reconstruction from two cameras. From the point cloud data, we could recreate the three-dimensional geometric model of the fish and analyze its swimming behavior in three dimensions. The extracted dynamic fish geometry should allow for an improved understanding of mechanosensation and perception, which are critical to elucidate how zebrafish process visual cues and perceive flow structures.