multi-animal DeepLabCut: 2.2rc1

multi-animal DeepLabCut is out (rc1)!

• *NEW* 🤩 preprint by Lauer et al

• code

After a (long) beta period (lab moved, covid delays, and many adventures), we learned a lot from our users, and re-tooled the system. This blog post will cover what we learned, what we changed, and the new advanced features of #maDLC!

Firstly, we realized the beta-version needed a lot more domain knowledge that we ultimately wanted for our user base. Although it has the feel of the standard DLC workflow, there are/were some really key decisions that needed to be made. This led to several failures that we as developers could “instantly” see the issue with, but fully appreciate it’s a lot of documentation and without the preprint, it makes the science behind it harder/impossible to grasp. So, we went “back to the drawing board” in the past year and thought about ways we can make it nearly as data-driven as possible….

What emerged is 2.2rc1 - it has novel data-driven methods, a streamlined workflow, and better performance

Here are key decisions & new things related to usability:

• 🥇 Is multi-animal truly good for your problem? We saw many animals that were easily visually distinguishable being used with #maDLC beta, which by default has the strong assumption there should be no unique visual properties. Even two mice with one having a marked tail violates this assumption, unless you use our ID network output (see next point).

• 🐹🐭 If you can distinguish animals, leverage it! One feature we worked on for the last years is using identity (visual features) to learn “who is who.” We did not release this fully (it was a silent feature), as we felt it would complicate beta testing, but at the end of the day, most people need this (see point above). Simply set identity = true in your config before creating a training set & training now!

• 💡 What key points to label? More is Better. Even in our original paper, we should more labels == better performance. I.e., 4 labels on a mouse is better than 2! 8 is better than 4! This is especially true with multiple animals that have occlusions. Have two mice that are on top of each other? Our networks learn who-is-who, and what point-belongs-to-who, but you need enough points! When in doubt, adding a few extra key points will only help - you can ignore them in your analysis later if you want.

• 🗝 The skeleton, you need(ed) to over-connect… And when we mean over-connect, we really meant everything to everything is best. Now, we (1) take this decision away from the user, and (2) introduce a novel fully-data driven approach. Using your data, we now find the ultimate skeleton for your data to get the best performance with the best speed. Super cool, right?

• 📈 Animal Assembly, the crucial step in linking which key points belong to an individual. Here, we have an entirely new algorithm that is ultra fast, more accurate, and done in a data-drive way. This novel approach we introduce in this version is much more robust, and makes tracking even better…

• 🚦Tracking Types. We introduce a new tracker type that beats our prior versions. Basically, use ellipse unless you have a good reason not to.

Now, how to use it!

• 💻 We have re-written many of the docs for a streamlined process. If you go HERE, you will be linked to the new “Doc Hub” that guides you through install, opening the software, and either standard DLC or maDLC.

• 🐭🐭🐭 maDLC will feel very much like standard DLC, and many parts are now entirely data driven. This means your big decision is to get highly quality data! 💪

• 🛠 Future note, we will be releasing a DLC “cook book” to also give both beginner & advanced users worked examples, stay tuned!

What are the major new scientific contributions for multi-animal DLC?

• Introduce four datasets of varying difficulty for benchmarking multi-animal pose estimation networks. 🐭🐒🐠🐁

• A novel multi-task architecture that predicts multiple conditional random fields and therefore can predict keypoints, limbs, as well as animal identity.

• A novel data-driven method for animal assembly that finds the optimal skeleton without user input, and that is state-of-the art (compared to top models on COCO).

• A new tracking module that is locally and globally optimizable.

• We show that one can predict the identity of animals, which is useful to link animals across time when temporally-based tracking fails.

• We extend the open source DeepLabCut software to multi-animal scenarios and provide new graphical user interfaces (GUIs) to allow keypoint annotation and check reconstructed tracks.

left: Example images from the 4 datasets with ground truth and network annotations. right: Performance of ResNets, EfficientNets, and our new multi-fusion, multi-stage DLCRNet!

A little background ….

maDLC started way back in 2018 spearheaded by Alexander plus development from Mackenzie & Tanmay (our Cosyne2019 talk was the first glimpse!). Then, Jessy joined us in Jan 2020 as the CZI DeepLabCut Fellow and made major contributions that to the beta released in April 2020. Now the 2.2 release candidate (rc1) was greatly supported by Jessy and two amazing new students in the Mathis Group and Mathis Lab, Mu and Shaokai.

Fun facts: we tested over 100 network variants in the making of this paper. It includes 1460 commits to the code, nearly every file (>200!) was changed (but our license, still open, still free, just please cite us ;), and our amazing collaborators and students labeled over 8K frames for serious testing. We rigorously benchmarked, and this version of DeepLabCut’s new networks significantly outperform our earlier work (nothing like beating yourself, am I right?), plus we also compared to top networks in human pose estimation on the four multi-animal datasets and had better results (see paper).

Plus, there are numerous people who have contributed to make this paper/code what it is today (see authors & contributors on GitHub)! Last but not least, a big thanks to the amazing labs who helped us with data and testing! Special thanks to the Lauder, Dulac, Feng & V.N.Murthy labs for the very cool data that we present in the paper.

Thank you also to our generous funders (full ack. in the text): The Rowland Institute at Harvard, Chan Zuckerberg Initiative EOSS, Bertarelli Foundation (EPFL).