Mouse Total Capture: 3D Motion and Expression Capture for the Freely Moving Mouse

Jiachen Zhao1,     Tao Yu1,✉     Liang An1     Qichen Qian1     Fang Deng2, ✉

1Tsinghua University      2Beijing Institute of Technology
Corresponding Author
Mouse Total Capture(MTC) aims to simultaneously capture both the motion and expression of a freely moving mouse. We tackle this problem from three perspectives:
  • System: PanoMouse system with 24 cameras distributed across top-middle-bottom three layers to achieve 360-degree photographic capture.
  • Dataset: PanoMouse dataset with annotation for whole-body 92 keypoints of a freely moving mouse.
  • Algorithm: Structure-Aware Triangulation Network, which takes the system structure and mouse body structure features into account to address the MTC challenges.
  • Abstract

    Natural behavior is the language of brain. Precise capture and quantitative analysis of organisms’ behaviors is indispensable in Neuroscience and Biomedicine. Existing studies mainly focused on body capture of freely moving mice or expression capture under head-fixed conditions. Few works have explored how to simultaneously capture both the motion and expression of a freely moving mouse, i.e. Mouse Total Capture (MTC). The main challenges in MTC stem from the severe self-occlusion caused by the mice’s body structure, substantial size variations across different joint types, and a lack of distinct visual textures. This paper tackles the MTC problem from three perspectives: system, dataset, and algorithm. First, we develop the densest multi-camera system for mice, named the PanoMouse system, featuring 24 cameras distributed across top-middle-bottom three layers to achieve 360-degree photographic capture. We then collect the PanoMouse dataset, the first dataset with annotation for whole-body 92 keypoints of a freely moving mouse. The annotation covers the trunk, limbs, tail, eyes, ears, fingers, and toes, providing a data foundation for fine-grained mouse behavior analysis. Finally, we propose an end-to-end learnable Structure-Aware Triangulation Network, which takes the system structure and mouse body structure features into account to address the MTC challenges. Experimental results demonstrate that our method achieves state-of-the-art MTC performance, with 1.18mm MPJPE and 79.67% PCK.

    Total capture results

    Face capture results

    BibTeX

    @article{Zhao2024MouseTotalCapture,
      author    = {Jiachen Zhao, Tao Yu, Liang An, Qichen Qian, Fang Deng},
      title     = {Mouse Total Capture: 3D Motion and Expression Capture for the Freely Moving Mouse},
      journal   = {Under review},
      year      = {2024},
    }