Link to paper

The full paper is available here.

You can also find the paper on PapersWithCode here.


  • Recent advances in machine learning and computer vision are revolutionizing the field of animal behavior.
  • Large datasets of annotated images of animals for markerless pose tracking are still scarce.
  • A method is proposed that uses a motion capture system to obtain a large amount of annotated data on animal movement and posture.
  • The method extracts the 3D positions of morphological keypoints in reference to the positions of markers attached to the animals.
  • A new dataset - 3D-POP - is offered with approximately 300k annotated frames in the form of videos of groups of one to ten freely moving birds.
  • 3D-POP is the first dataset of flocking birds with accurate keypoint annotations in 2D and 3D.

Paper Content


  • Computer vision and machine learning are revolutionizing research methods
  • Dataset-driven machine learning methods have been successful in animal behavior tasks
  • Automatic methods reduce labor and errors associated with manual coding
  • Data on animal locomotion is used to reverse-engineer behaviors and movements
  • Creating large datasets with animals is difficult
  • Recently, datasets have been created with a focus on animal behavior
  • Most solutions are limited to 2D space
  • Marker-based motion capture technology has been used to create 3D datasets
  • Propose a new motion capture-based approach to create large-scale datasets with a bird species
  • Method enables a large amount of training data to be generated in a semi-automatic manner
  • Able to track a variety of naturalistic behaviors in a flock of up to 10 individuals
  • CNN models trained on dataset are able to predict postures of birds with no markers attached

State of the art

2d posture

  • Animal Kingdom is the largest dataset with 50 hours of video annotations
  • Other datasets contain images and focus on capturing variations in terms of specific taxa
  • Few datasets offer posture annotations for multiple individuals
  • Existing datasets have motivated the development of various methods for posture estimation
  • Manual annotations limit the complexity of datasets

3d posture

  • Obtaining 3D ground truth posture is difficult with a group of animals
  • Popular method is triangulation of 2D postures using multiple views
  • Acinoset, Fly3D, OpenMonkeyStudio use triangulation-based approaches
  • Alternative approach is marker-based motion capture with skeleton tracking
  • Rat 7M dataset uses motion capture with RGB cameras and 20 markers
  • Rat 7M dataset is first with 3D ground truth with more than one animal
  • Motion capture systems offer high accuracy and low noise
  • Marker placement is a limitation for smaller species and wild animals
  • 2D keypoints and silhouettes, synthetic datasets, and toys used to predict 3D posture
  • Computer vision literature focuses on extracting detail
  • Head and body orientations sufficient to quantify key behaviors in groundforaging contexts
  • Measuring head direction in 3D allows gaze reconstruction

Multi-object tracking with identity

  • Identity recognition is a critical problem in biological studies
  • Tracking and identification of multiple individuals in large groups is important
  • Existing solutions perform well with specific perspectives, but not with occlusion
  • Very few datasets offer the possibility of solving all problems simultaneously
  • 3D-POP dataset includes video recordings of 18 unique pigeons from multiple views
  • Ground truth for identity, 2D-3D trajectories, and 2D-3D posture mapping is available
  • Annotations for object detection in the form of bounding boxes are included


Experimental setup

  • Dataset was collected from pigeons moving on a jute fabric
  • Grains were scattered to encourage the birds to feed in that area
  • Mo-cap system was used to track 3D positions of reflective markers
  • 4 high-resolution cameras and an Arduino-based synchronization box were placed at the corners of the feeding area

Animal subjects

  • 18 pigeons were studied over 6 days
  • 10 pigeons were randomly selected each day
  • 4 reflective markers were attached to each pigeon’s head
  • 4 markers were attached to a customized backpack worn by each pigeon
  • Pigeons tolerated markers and quickly habituated to backpacks
  • Unique geometric configuration was used to track individual identities
  • 11 trials were performed each day
  • Total frames and duration of samples are described in Table 1
  • An additional session was recorded with birds without markers

Data annotation pipeline

  • 6-DOF pose of rigid objects can be tracked in 3D space
  • 4 markers attached to head and body of bird used to compute 6-DOF pose
  • Pipeline designed to annotate position of features on head and body
  • Relationship between markers and features does not change during sequence
  • 9 morphological keypoints annotated on 5-10 frames from all view angles
  • 3D positions of keypoints transferred to global coordinate system using 6-DOF pose
  • Bounding box annotations derived from keypoint projections
  • Dataset provides accurate ground truth for 3D keypoints, 2D keypoints, bounding boxes, and individual identities
  • Dataset includes RGB images from 4 high-resolution cameras and up to 6 hours of recordings


  • We released 3D-POPAP, a tool to manipulate annotations of a dataset
  • We designed the annotation approach to allow for easy addition of 2D/3D keypoint annotations
  • There are no datasets available with ground truth on 3D posture of birds

Dataset validation

  • 3D-POP annotations are obtained automatically
  • 3 tests designed to validate accuracy and consistency of annotations
  • First test compares accuracy of 3D features computed with 3D-POP and Kano et al.
  • Second test measures consistency of 3D/2D annotations across dataset
  • Third test checks variation in 3D pose captured in all sequences
  • Ground truth 3D position of eyes and beak compared to 3D position computed with 3D-POP
  • RMSE for all three features is sufficient for pigeons
  • Method alleviates need of using dedicated calibration rigs
  • 2D keypoint detection model trained on 15177 images
  • Outlier analysis used to filter out 2.9% of dataset
  • Consistency check reveals annotations are largely consistent with model predictions
  • Visual examples of outlier frames with mo-cap errors
  • 96.1% of gaps in dataset are less than 30 frames
  • 74,924 unique orientations of head and 14,191 unique orientations of body in dataset


Marker-based + markerless hybrid approach

  • Markerless tracking algorithm trained on 3D-POP can solve 3D tracking when mo-cap fails
  • Hybrid tracking solution has potential applications for future behavior studies
  • Tested on 5 min sequence with 25% of mo-cap tracking data removed
  • Achieved avg. RMS error of 9.2 mm with proposed solution, compared to 52.1 mm with linear interpolation
  • Robust markerless 3D tracking solution needed for biologists to switch from motion-tracking technology

Markerless bird tracking

  • Models trained with 3D-POP dataset can be used to track birds without markers.
  • Experiment works as a “sanity check” to ensure models are not biased.
  • Experiment demonstrates potential contribution of method to develop markerless 3D tracking, posture estimation, and identification.

Manual validation

  • Experiment demonstrated validity of assumption that body keypoints behave like points on a rigid body
  • Compared manual and automatic annotations using PCK05 and PCK10 metrics, average PCK05 of 66% and PCK10 of 94%
  • Visual quantification showed only 2.8% of frames are cases where birds are moving wings, valid in over 97% of dataset

Limitations and future work

  • Assumption that head and body behave as rigid bodies does not hold for certain body parts
  • Proposed approach does not support annotation for flying birds or birds that change shape of body parts
  • Approach depends on tracking accuracy of motion capture system
  • Outlier detection method is effective at identifying noisy annotations
  • Dataset was curated semi-automatically in existing motion tracking setup, limited to indoor environment


  • Introduced novel method to use mocap system for generating large-scale datasets with multiple animals
  • Semi-automated method offers alternative for generating high-quality datasets with animals without manual effort
  • 3D-POP dataset offers ground truth for 3D posture prediction and identity tracking in birds
  • Intrinsic calibration used A0 charuco checkerboard and extrinsic calibration used subject based approach
  • Synchronized RGB action cameras with camera control box and arduino based synchronization device
  • 3D-POP dataset contains ground truth annotation for pigeons with markers attached to their body
  • Experiment 2 shows models trained on 3D-POP dataset work well with pigeons recorded in same arena without markers
  • 3D-POP dataset includes trials of freely moving birds without any marker attachment
  • Post-processing pipeline used to fix mis-labelled frames
  • Technique designed to measure 3D orientation of pigeon body parts relative to each axis
  • Dataset available for download