- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper models Drosophila social behavior generatively: each fly is an agent mapping egocentric sensory inputs (visual depth, field/conspecific touch) to ego-centric motion (velocities, pose changes) as discrete-continuous mixtures, with social behavior emerging from inter-agent interaction rather than centralized modeling. The AnimalPoseForecasting library (composable, invertible Operations) enables autoregressive rollout. On MABe 2022-fly probe tasks, reported F1 exceeds published methods (up to ~3× on Manual); ablations across six variants show agent-centric representations and probabilistic discrete outputs are key, while manual pose-angle decomposition is unnecessary.
Critical strengths
-
This is a genuine and essentially vacant research niche. Literature review confirms no prior fruit-fly generative model at the level of multi-animal, fine-grained pose (19 keypoints, ~9-fly groups), and closed-loop social interaction: the nearest work, Eyjolfsdottir et al. (2016), models only pairwise interactions with coarse features and an RNN; Vaxenburg et al. (2025, Nature) offers single-fly MuJoCo physical simulation; and graphics-oriented animal motion generation (OmniMotionGPT, UniMo, Kirin, etc.) lacks behaviorally meaningful social modeling. The paper's first-principles contribution claim is well founded.
-
The scientific motivation for the agent-centric constraint is clear (Sections 1, 4). Framing the model as a perception-to-motion mapping targets the "algorithmic level" of explanation in neuroethology (per Marr's hierarchy), distinguishing it from purely engineering-based trajectory prediction and naturally supporting closed-loop multi-agent simulation. This framing carries transferable value for the computational ethology community.
Critical weaknesses
-
The ablation supporting the core claim is confounded (6.1, KEYPOINTS): it changes both input (world-frame keypoints vs. sensory features) and output (world-frame keypoints vs. egocentric velocities + skeleton decomposition) — as the authors admit, "more than one component at once" — yet the causal claim that agent-centric representation is key rests largely on this comparison. Without single-factor ablations (e.g., world-frame input with egocentric output, or vice versa), causes like raw keypoint parameterization or missing velocity representation cannot be excluded.
-
The "SOTA on MABe 2022-fly" claim is not directly comparable (6.2): the model trains on augmented full data (150 videos, 72M fly frames, Branson et al., 2026), whereas baselines use only the challenge's User-train (~60% of videos); moreover, whether training respects the User-train/Eval-train/Test split is unstated — if the "full dataset" includes test videos, the evaluation leaks test distribution and the comparison is void.
Questions for authors
-
Model capacity is unmatched across baselines and undiscussed. Per Appendix E.1, d_model=2048 with 10 layers (est. ~190M parameters), far exceeding MABe 2022's lightweight baselines.
-
The RatInABox generalization experiment (Appendix D) is too weak: a single qualitative trajectory plot (Fig. 17b) on synthetic RL-agent data, with no quantitative metrics, yet it grounds the "general-purpose library" claim.
-
Various typos: "a a variety" (Sec. 1), "ohters" (App. A), "deviaties" (near App. B.4), "by by mirroring" (Sec. 2), "into into" (App. C.4.2); Fig. 5a omits the method's exact F1 value, leaving the key claim readable only from the bar chart; and no training compute is reported (~190M params × 500 epochs × multiple variants).
4 comments
Part 2/4
Controlled ablation of the agent-centric representation.
By agent-centric we mean that both the sensory input and the motion output are expressed in the agent's own reference frame. KEYPOINTS was designed to remove both at once, testing the representation as a whole, and it is directly comparable to NO DISCRETIZATION. For clarification, KEYPOINTS inputs the flattened keypoints of all other animals and outputs the continuous velocity of the keypoints. NO DISCRETIZATION inputs the sensory representation of all other animals and outputs the continuous agent-centric velocity.
We agree that it is interesting to understand whether the agent-centric input representation, output representation, or both that contribute to the difference in performance. To answer this, we trained two new ablations:
WorldSensory+AgentMotion changes only the sensory representation compared to NO HANDCRAFTED POSE.
AgentSensory+WorldMotion changes only the output representation compared to NO DISCRETIZATION.
We find that nearly every metric is worse for all three ablations (16/18 comparisons), showing that both the input and the output representation matter. Interestingly, input representation most strongly affects behavior frequency error (0.185 → 0.5); simulated flies walk 32% more than real flies and chase, court and extend their wings 50–60% less. Output representation most strongly affects pose distance (0.004 → 0.010) and probe MCC (0.518 → 0.382). Removing both is worse on every metric.
Results next to the comparable agent-centric reference:
| Model | Sensory input | Pose dist ↓ | Vel. dist ↓ | Real-vs-sim pose ↓ | Real-vs-sim vel. ↓ | Behavior freq. err ↓ | Probe MCC ↑ |
|---|---|---|---|---|---|---|---|
| NO HANDCRAFTED POSE | Agent | 0.005 | 0.056 | 0.650 | 0.950 | 0.185 | 0.504 |
| WorldSensory+AgentMotion | World | 0.006 | 0.054 | 0.670 | 0.954 | 0.500 | 0.496 |
| Model | Sensory input | Motion output | Pose dist ↓ | Vel. dist ↓ | Real-vs-sim pose ↓ | Real-vs-sim vel. ↓ | Behavior freq. err ↓ | Probe MCC ↑ |
|---|---|---|---|---|---|---|---|---|
| NO DISCRETIZATION | Agent | Agent (joint-angle velocities) | 0.004 | 0.107 | 0.650 | 0.950 | 0.263 | 0.518 |
| AgentSensory+WorldMotion | Agent | World (keypoint velocities) | 0.010 | 0.122 | 0.681 | 0.989 | 0.194 | 0.382 |
| KEYPOINTS | World | World (keypoint velocities) | 0.012 | 0.123 | 0.700 | 0.990 | 0.369 | 0.337 |
Difference between each ablation and its reference model (ablation − reference). Bold marks a change where the ablation made the metric worse.
| Ablation | Reference | Pose dist ↓ | Vel. dist ↓ | Real-vs-sim pose ↓ | Real-vs-sim vel. ↓ | Behavior freq. err ↓ | Probe MCC ↑ |
|---|---|---|---|---|---|---|---|
| WorldSensory+AgentMotion | NO HANDCRAFTED POSE | +0.001 | −0.002 | +0.020 | +0.004 | +0.315 | −0.008 |
| AgentSensory+WorldMotion | NO DISCRETIZATION | +0.006 | +0.015 | +0.031 | +0.039 | −0.069 | −0.136 |
| KEYPOINTS | NO DISCRETIZATION | +0.008 | +0.016 | +0.050 | +0.040 | +0.106 | −0.181 |
We will add these results to the paper.
Part 3/4
Generalization of AnimalPoseForecasting library and framework to other datasets.
There is no reason to expect our framework to work only for flies: the sensory inputs, pose representation and motion outputs are built from general operations in our library.
In the paper, we showed qualitatively that the framework models RatInABox data well (Fig. 17b and Supplementary Video 11). To make this more quantitative, we calculated the Wasserstein distance between 20 rollouts sampled from our model and 20 groundtruth RatInABox rollouts from the same 0.8 s initialization, mean over 1,000 held-out episodes. For comparison, we also computed the minimum achievable distance at this sample size based on a second, independent set of 20 RatInABox rollouts from the same initialization, and a ceiling by computing the distance to RatInABox rollouts from another episode initialization. Our model’s distances are close to the minimum achievable distance and far below the different-initialization ceiling. We will add figure versions of these tables to the revised paper.
Position: Wasserstein-1 distance (cm)
| 0.5 s | 1 s | 2 s | 4 s | |
|---|---|---|---|---|
| Ours | 1.11 ± 0.01 | 2.01 ± 0.02 | 2.87 ± 0.03 | 3.94 ± 0.05 |
| Minimum achievable | 1.08 ± 0.01 | 1.92 ± 0.01 | 2.52 ± 0.02 | 3.16 ± 0.04 |
| Different initialization | 24.23 ± 0.44 | 22.45 ± 0.43 | 18.47 ± 0.38 | 12.49 ± 0.27 |
Forward speed: Wasserstein-1 distance (mm/s)
| 0.5 s | 1 s | 2 s | 4 s | |
|---|---|---|---|---|
| Ours | 9.38 ± 0.15 | 9.13 ± 0.15 | 8.78 ± 0.13 | 8.65 ± 0.12 |
| Minimum achievable | 8.37 ± 0.13 | 7.51 ± 0.13 | 6.03 ± 0.10 | 5.66 ± 0.10 |
| Different initialization | 25.03 ± 0.61 | 19.08 ± 0.54 | 18.78 ± 0.55 | 18.08 ± 0.36 |
Sideways speed: Wasserstein-1 distance (mm/s)
| 0.5 s | 1 s | 2 s | 4 s | |
|---|---|---|---|---|
| Ours | 5.51 ± 0.08 | 3.95 ± 0.05 | 3.07 ± 0.04 | 2.56 ± 0.03 |
| Minimum achievable | 4.98 ± 0.07 | 3.57 ± 0.04 | 2.71 ± 0.04 | 2.14 ± 0.03 |
| Different initialization | 17.62 ± 0.35 | 13.39 ± 0.27 | 9.17 ± 0.18 | 6.33 ± 0.12 |
Angular velocity: Wasserstein-1 distance (rad/s)
| 0.5 s | 1 s | 2 s | 4 s | |
|---|---|---|---|---|
| Ours | 0.210 ± 0.003 | 0.157 ± 0.003 | 0.154 ± 0.004 | 0.203 ± 0.004 |
| Minimum achievable | 0.201 ± 0.003 | 0.149 ± 0.003 | 0.139 ± 0.004 | 0.174 ± 0.004 |
| Different initialization | 0.822 ± 0.020 | 0.684 ± 0.017 | 0.619 ± 0.016 | 0.604 ± 0.011 |
Part 4/4
Generalization of AnimalPoseForecasting library and framework to other datasets (continued).
We agree that the paper would be stronger if we could show that the method adapted to real data from another species, thus we used the AnimalPoseForecasting library to apply our framework to the MABe22 mouse triplet trajectory dataset (Sun et al., 2023). The sensory inputs were similar to those for flies: social vision and social touch were nearly identical. Wall vision (distance to closest wall in binned directions) and wall touch (distance from keypoints to closest wall) were added to allow the arena walls to be sensed. For all, hyperparameters were set based on the arena and mouse body sizes. As in NO HANDCRAFTED POSE, the motion output is the mouse's global velocity plus the change in its keypoints in its own reference frame. Results on the MABe22 dataset, compared to SOTA are:
| Method | Exp. day ↓ | Time of day ↓ | Strain ↑ | Movement ↑ | Contact ↑ | Watching ↑ | Lights ↑ | Avg. MSE ↓ | Avg. F1 ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Ignore input** | .0957 | .0921 | .590 | .004 | .187 | .044 | .587 | .0939 | .213 |
| PCA | .0942 | .0946* | .516 | .005 | .169 | .066 | .546 | .0944 | .196 |
| TVAE | .0940 | .0944* | .530 | .008 | .213 | .102 | .568 | .0942 | .227 |
| T-Perceiver | .0933 | .0932* | .698 | .014 | .232 | .164 | .697 | .0933 | .271 |
| T-GPT | .0927 | .0938* | .645 | .012 | .252 | .179 | .654 | .0933 | .274 |
| T-PointNet | .0928 | .0932* | .660 | .036 | .256 | .156 | .672 | .0930 | .281 |
| T-BERT | .0926 | .0928* | .786 | .013 | .266 | .172 | .688 | .0927 | .297 |
| Ours | .0979 | .0979 | .813 | .012 | .255 | .111 | .604 | .0979 | .280 |
*Time of Day MSE reported in the MABe22 paper had values ~0.9. We believe that this was a mistranscription of the results, and that a 0 was missing after the decimal point. We made this correction.
**Ignore input: chance baseline, returns training mean for MSE, positive for F1.
Our model outperforms the baselines (PCA and TVAE) on the categorization tasks (Avg. F1), and its average F1 (0.280) is above four of the six published methods, with best performance on the “Strain” category. It performs worst on the two MSE tasks, but we note that all methods are near chance here. The main complication we noticed with this extension is with noisy trajectories. The agent-centric representation is possibly more sensitive to tracking noise, and we are exploring whether smoothing and automatic data cleaning would help. In addition, the method will also model tracking noise, which is not biologically useful. It was a valuable experiment to be made aware of these issues.
Computational requirements, complexity.
Our newly added 256-d model has fewer parameters, ~6M, compared to the SOTA on MABe22 T-GPT, which has ~170M. Training the 256-d model for 500 epochs took ~45.5 hours on a single H100. Training the full 2048-d model took ~106 H100-hours.
Fig. 5b F1 scores
Figure 13 in the Appendix shows the full confusion matrices. F1 scores, which we can add to the plot:
| Class | Share of prompts | F1, mixed ignored | F1, mixed = wrong |
|---|---|---|---|
| female | 0.79 | 0.970 | 0.947 |
| male | 0.21 | 0.838 | 0.739 |
| courtship | 0.47 | 0.832 | 0.715 |
| not courtship | 0.53 | 0.841 | 0.729 |
| blind | 0.12 | 0.728 | 0.585 |
| not blind | 0.88 | 0.951 | 0.911 |
| 91B01 | 0.10 | 0.724 | 0.642 |
| not 91B01 | 0.90 | 0.980 | 0.971 |
Typos.
Thank you for catching these, we will correct them!
Part 1/4
Thank you for your careful review. We appreciate your recognition that multi-animal, fine-grained generative modeling of fly social behavior is an essentially unexplored research area, and that our agent-centric framing is well motivated. Your concerns are well taken, and we ran new experiments to address them: a more complete ablation study that varies the sensory input and motion output independently, a re-evaluation on the original MABe22 data and splits with a 256-dimensional model, and a new application of our library to the MABe22 mouse triplet data, which tests our library’s generality beyond RatInABox. We respond to these and your other points below.
Fairer comparison with MABe22 published results.
We agree that the comparison would be stronger by exactly matching the dataset and evaluation to the original MABe 2022 challenge splits, and we have performed several new experiments to address this. We note that there was no data leakage: our models were trained only on the User-train split, and the data augmentation kept all videos from the original MABe22 challenge in their original split. This is stricter than the challenge required, since MABe22 allowed training on all the data (excluding labels), as it is an unsupervised task.
We retrained our model using the dataset in the original MABe 2022 fly dataset [Sun et al., 2023]. Evaluation was done using the MABe 2022 evaluation code, with probes trained on the “evaluator train” split, “test1” used for choosing between a few options, and final results reported for “test2”. MABe 2022 also required a 256-dimensional representation, while our original dimensionality was 2048 (the hidden state dimensionality of our model). We tried a few different methods for reducing dimensionality from 2048 to 256, and based on “test1” evaluation, decided to use a narrower model with dimensionality 256. Our updated results, compared to the MABe 2022 SOTA:
*Ignore input: chance baseline; predicts positive for every frame.
Our method continues to outperform SOTA: on average over all tasks, our model (0.514) outperforms all published methods (best: T-GPT, 0.458). Measured from the baseline that ignores the input (0.435), its gain is 3.4× T-GPT's. The largest improvement is on expert-annotated behaviors ("Manual"), where its F1 is 3.1× that of the best published method.
To separate the effects of model size and training data, we also trained a larger model (d_model = 2048) on the original data.
- Dimensionality. Increasing d_model from 256 to 2048 improves the average from 0.514 to 0.536, mainly on fly type (0.382 → 0.456), line category (0.557 → 0.609) and expert-annotated behaviors (0.757 → 0.806).
- Training data. The augmented data does not change the average (0.536 vs. 0.532). It improves line category (0.609 → 0.660) and expert-annotated behaviors (0.806 → 0.876), but lowers fly type (0.456 → 0.355).
Note we re-evaluated the model from the paper (d_model = 2048, trained on the augmented data) with the MABe22 matched protocol, thus results are slightly different from those in Fig. 5a.