Scientific article
OA Policy
English

Lightning Pose : improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools

Published inNature methods, vol. 21, no. 7, p. 1316-1328
Publication date2024-07
First online date2024-06-25
Abstract

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.

Keywords
  • Algorithms
  • Animals
  • Bayes Theorem
  • Behavior, Animal
  • Cloud Computing
  • Deep Learning
  • Image Processing, Computer-Assisted / methods
  • Posture / physiology
  • Software
  • Supervised Machine Learning
  • Video Recording / methods
Funding
  • Simons Foundation [543023]
  • Gatsby Charitable Foundation [GAT3708]
  • NINDS NIH HHS [RF1 NS118448]
  • U.S. Department of Health & Human Services | NIH | National Institute of Diabetes and Digestive and Kidney Diseases (National Institute of Diabetes & Digestive & Kidney Diseases) [5R01DK131086-02]
  • Dutch Research Council (NWO) [VI.Veni.212.184]
  • National Science Foundation (NSF) [1707398]
  • U.S. Department of Health & Human Services | NIH | National Institute of Neurological Disorders and Stroke (NINDS) [K99NS128075]
  • Wellcome Trust [216324]
  • NIDDK NIH HHS [R01 DK131086]
Citation (ISO format)
BIDERMAN, Dan et al. Lightning Pose : improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools. In: Nature methods, 2024, vol. 21, n° 7, p. 1316–1328. doi: 10.1038/s41592-024-02319-1
Main files (2)
Article (Published version)
accessLevelRestricted
Article (Accepted version)
accessLevelPublic
Identifiers
Additional URL for this publicationhttps://www.nature.com/articles/s41592-024-02319-1
Journal ISSN1548-7091
14views
25downloads

Technical informations

Creation30/09/2025 12:25:35
First validation02/10/2025 07:08:01
Update02/10/2025 07:08:01
Status update02/10/2025 07:08:01
Last indexation02/10/2025 07:08:02
All rights reserved by Archive ouverte UNIGE and the University of GenevaunigeBlack