acceptodds
Under review as a conference paper at ICLR 2027

Modeling Fly Social Behavior with Agent-Centric Pose Forecasting

Abstract

Understanding animal behavior at an algorithmic level – what animals attend to, how they form internal representations, and how this maps to action – remains a central challenge in neuroethology. We explore how well large generative models capture properties of fly social behavior. We combine transformers with agent-centric representations. Models input egocentric sensory observations and output egocentric movements, mirroring biological constraints. Social behavior emerges from agents independently sensing and responding to one another. We demonstrate that this simple model captures many properties of fly behavior, outperforming the world-centric representation and achieving SOTA on MABe 2022-fly. We systematically evaluate properties of the representation important for capturing the behavior distribution through a series of lesioning experiments. These comparisons were enabled by a new, general-purpose library for animal-centric pose forecasting.

Primary area:
applications to neuroscience & cognitive science
Keywords:
pose forecasting, neuroscience, behavior, multi-agent, library, biology, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UDkBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper models Drosophila social behavior generatively: each fly is an agent mapping egocentric sensory inputs (visual depth, field/conspecific touch) to ego-centric motion (velocities, pose changes) as discrete-continuous mixtures, with social behavior emerging from inter-agent interaction rather than centralized modeling. The AnimalPoseForecasting library (composable, invertible Operations) enables autoregressive rollout. On MABe 2022-fly probe tasks, reported F1 exceeds published methods (up to ~3× on Manual); ablations across six variants show agent-centric representations and probabilistic discrete outputs are key, while manual pose-angle decomposition is unnecessary.

Critical strengths
  1. This is a genuine and essentially vacant research niche. Literature review confirms no prior fruit-fly generative model at the level of multi-animal, fine-grained pose (19 keypoints, ~9-fly groups), and closed-loop social interaction: the nearest work, Eyjolfsdottir et al. (2016), models only pairwise interactions with coarse features and an RNN; Vaxenburg et al. (2025, Nature) offers single-fly MuJoCo physical simulation; and graphics-oriented animal motion generation (OmniMotionGPT, UniMo, Kirin, etc.) lacks behaviorally meaningful social modeling. The paper's first-principles contribution claim is well founded.

  2. The scientific motivation for the agent-centric constraint is clear (Sections 1, 4). Framing the model as a perception-to-motion mapping targets the "algorithmic level" of explanation in neuroethology (per Marr's hierarchy), distinguishing it from purely engineering-based trajectory prediction and naturally supporting closed-loop multi-agent simulation. This framing carries transferable value for the computational ethology community.

Critical weaknesses
  1. The ablation supporting the core claim is confounded (6.1, KEYPOINTS): it changes both input (world-frame keypoints vs. sensory features) and output (world-frame keypoints vs. egocentric velocities + skeleton decomposition) — as the authors admit, "more than one component at once" — yet the causal claim that agent-centric representation is key rests largely on this comparison. Without single-factor ablations (e.g., world-frame input with egocentric output, or vice versa), causes like raw keypoint parameterization or missing velocity representation cannot be excluded.

  2. The "SOTA on MABe 2022-fly" claim is not directly comparable (6.2): the model trains on augmented full data (150 videos, 72M fly frames, Branson et al., 2026), whereas baselines use only the challenge's User-train (~60% of videos); moreover, whether training respects the User-train/Eval-train/Test split is unstated — if the "full dataset" includes test videos, the evaluation leaks test distribution and the comparison is void.

Questions for authors
  1. Model capacity is unmatched across baselines and undiscussed. Per Appendix E.1, d_model=2048 with 10 layers (est. ~190M parameters), far exceeding MABe 2022's lightweight baselines.

  2. The RatInABox generalization experiment (Appendix D) is too weak: a single qualitative trajectory plot (Fig. 17b) on synthetic RL-agent data, with no quantitative metrics, yet it grounds the "general-purpose library" claim.

  3. Various typos: "a a variety" (Sec. 1), "ohters" (App. A), "deviaties" (near App. B.4), "by by mirroring" (Sec. 2), "into into" (App. C.4.2); Fig. 5a omits the method's exact F1 value, leaving the key claim readable only from the bar chart; and no training compute is reported (~190M params × 500 epochs × multiple variants).

4 comments

Thank you for your careful review. We appreciate your recognition that multi-animal, fine-grained generative modeling of fly social behavior is an essentially unexplored research area, and that our agent-centric framing is well motivated. Your concerns are well taken, and we ran new experiments to address them: a more complete ablation study that varies the sensory input and motion output independently, a re-evaluation on the original MABe22 data and splits with a 256-dimensional model, and a new application of our library to the MABe22 mouse triplet data, which tests our library’s generality beyond RatInABox. We respond to these and your other points below.


Fairer comparison with MABe22 published results.
We agree that the comparison would be stronger by exactly matching the dataset and evaluation to the original MABe 2022 challenge splits, and we have performed several new experiments to address this. We note that there was no data leakage: our models were trained only on the User-train split, and the data augmentation kept all videos from the original MABe22 challenge in their original split. This is stricter than the challenge required, since MABe22 allowed training on all the data (excluding labels), as it is an unsupervised task.

We retrained our model using the dataset in the original MABe 2022 fly dataset [Sun et al., 2023]. Evaluation was done using the MABe 2022 evaluation code, with probes trained on the “evaluator train” split, “test1” used for choosing between a few options, and final results reported for “test2”. MABe 2022 also required a 256-dimensional representation, while our original dimensionality was 2048 (the hidden state dimensionality of our model). We tried a few different methods for reducing dimensionality from 2048 to 256, and based on “test1” evaluation, decided to use a narrower model with dimensionality 256. Our updated results, compared to the MABe 2022 SOTA:

MethodFly typeStim. controlStim. aggr.Line categoryMale/femaleManualAll tasks
Ignore input*0.1650.5880.5890.3530.9000.3080.435
PCA0.2820.4660.4840.5530.9900.2300.425
TVAE0.1990.5000.4500.3410.8210.2220.370
T-Perc0.3940.4180.5130.5730.9820.1970.448
T-GPT0.3630.5150.5000.5570.8730.2460.458
Ours, d=256, original data0.3820.5020.4700.5570.9900.7570.514

*Ignore input: chance baseline; predicts positive for every frame.

Our method continues to outperform SOTA: on average over all tasks, our model (0.514) outperforms all published methods (best: T-GPT, 0.458). Measured from the baseline that ignores the input (0.435), its gain is 3.4× T-GPT's. The largest improvement is on expert-annotated behaviors ("Manual"), where its F1 is 3.1× that of the best published method.

To separate the effects of model size and training data, we also trained a larger model (d_model = 2048) on the original data.

MethodFly typeStim. controlStim. aggr.Line categoryMale/femaleManualAll tasks
Ours, d=256, original data0.3820.5020.4700.5570.9900.7570.514
Ours, d=2048, original data0.4560.4990.4460.6090.9960.8060.536
Ours, d=2048, augmented data0.3550.5010.4540.6600.9900.8760.532

- Dimensionality. Increasing d_model from 256 to 2048 improves the average from 0.514 to 0.536, mainly on fly type (0.382 → 0.456), line category (0.557 → 0.609) and expert-annotated behaviors (0.757 → 0.806).
- Training data. The augmented data does not change the average (0.536 vs. 0.532). It improves line category (0.609 → 0.660) and expert-annotated behaviors (0.806 → 0.876), but lowers fly type (0.456 → 0.355).
Note we re-evaluated the model from the paper (d_model = 2048, trained on the augmented data) with the MABe22 matched protocol, thus results are slightly different from those in Fig. 5a.

Controlled ablation of the agent-centric representation.

By agent-centric we mean that both the sensory input and the motion output are expressed in the agent's own reference frame. KEYPOINTS was designed to remove both at once, testing the representation as a whole, and it is directly comparable to NO DISCRETIZATION. For clarification, KEYPOINTS inputs the flattened keypoints of all other animals and outputs the continuous velocity of the keypoints. NO DISCRETIZATION inputs the sensory representation of all other animals and outputs the continuous agent-centric velocity.

We agree that it is interesting to understand whether the agent-centric input representation, output representation, or both that contribute to the difference in performance. To answer this, we trained two new ablations:

WorldSensory+AgentMotion changes only the sensory representation compared to NO HANDCRAFTED POSE.
AgentSensory+WorldMotion changes only the output representation compared to NO DISCRETIZATION.

We find that nearly every metric is worse for all three ablations (16/18 comparisons), showing that both the input and the output representation matter. Interestingly, input representation most strongly affects behavior frequency error (0.185 → 0.5); simulated flies walk 32% more than real flies and chase, court and extend their wings 50–60% less. Output representation most strongly affects pose distance (0.004 → 0.010) and probe MCC (0.518 → 0.382). Removing both is worse on every metric.

Results next to the comparable agent-centric reference:

ModelSensory inputPose dist ↓Vel. dist ↓Real-vs-sim pose ↓Real-vs-sim vel. ↓Behavior freq. err ↓Probe MCC ↑
NO HANDCRAFTED POSEAgent0.0050.0560.6500.9500.1850.504
WorldSensory+AgentMotionWorld0.0060.0540.6700.9540.5000.496
ModelSensory inputMotion outputPose dist ↓Vel. dist ↓Real-vs-sim pose ↓Real-vs-sim vel. ↓Behavior freq. err ↓Probe MCC ↑
NO DISCRETIZATIONAgentAgent (joint-angle velocities)0.0040.1070.6500.9500.2630.518
AgentSensory+WorldMotionAgentWorld (keypoint velocities)0.0100.1220.6810.9890.1940.382
KEYPOINTSWorldWorld (keypoint velocities)0.0120.1230.7000.9900.3690.337

Difference between each ablation and its reference model (ablation − reference). Bold marks a change where the ablation made the metric worse.

AblationReferencePose dist ↓Vel. dist ↓Real-vs-sim pose ↓Real-vs-sim vel. ↓Behavior freq. err ↓Probe MCC ↑
WorldSensory+AgentMotionNO HANDCRAFTED POSE+0.001−0.002+0.020+0.004+0.315−0.008
AgentSensory+WorldMotionNO DISCRETIZATION+0.006+0.015+0.031+0.039−0.069−0.136
KEYPOINTSNO DISCRETIZATION+0.008+0.016+0.050+0.040+0.106−0.181

We will add these results to the paper.

Generalization of AnimalPoseForecasting library and framework to other datasets.

There is no reason to expect our framework to work only for flies: the sensory inputs, pose representation and motion outputs are built from general operations in our library.

In the paper, we showed qualitatively that the framework models RatInABox data well (Fig. 17b and Supplementary Video 11). To make this more quantitative, we calculated the Wasserstein distance between 20 rollouts sampled from our model and 20 groundtruth RatInABox rollouts from the same 0.8 s initialization, mean over 1,000 held-out episodes. For comparison, we also computed the minimum achievable distance at this sample size based on a second, independent set of 20 RatInABox rollouts from the same initialization, and a ceiling by computing the distance to RatInABox rollouts from another episode initialization. Our model’s distances are close to the minimum achievable distance and far below the different-initialization ceiling. We will add figure versions of these tables to the revised paper.

Position: Wasserstein-1 distance (cm)

0.5 s1 s2 s4 s
Ours1.11 ± 0.012.01 ± 0.022.87 ± 0.033.94 ± 0.05
Minimum achievable1.08 ± 0.011.92 ± 0.012.52 ± 0.023.16 ± 0.04
Different initialization24.23 ± 0.4422.45 ± 0.4318.47 ± 0.3812.49 ± 0.27

Forward speed: Wasserstein-1 distance (mm/s)

0.5 s1 s2 s4 s
Ours9.38 ± 0.159.13 ± 0.158.78 ± 0.138.65 ± 0.12
Minimum achievable8.37 ± 0.137.51 ± 0.136.03 ± 0.105.66 ± 0.10
Different initialization25.03 ± 0.6119.08 ± 0.5418.78 ± 0.5518.08 ± 0.36

Sideways speed: Wasserstein-1 distance (mm/s)

0.5 s1 s2 s4 s
Ours5.51 ± 0.083.95 ± 0.053.07 ± 0.042.56 ± 0.03
Minimum achievable4.98 ± 0.073.57 ± 0.042.71 ± 0.042.14 ± 0.03
Different initialization17.62 ± 0.3513.39 ± 0.279.17 ± 0.186.33 ± 0.12

Angular velocity: Wasserstein-1 distance (rad/s)

0.5 s1 s2 s4 s
Ours0.210 ± 0.0030.157 ± 0.0030.154 ± 0.0040.203 ± 0.004
Minimum achievable0.201 ± 0.0030.149 ± 0.0030.139 ± 0.0040.174 ± 0.004
Different initialization0.822 ± 0.0200.684 ± 0.0170.619 ± 0.0160.604 ± 0.011

Generalization of AnimalPoseForecasting library and framework to other datasets (continued).

We agree that the paper would be stronger if we could show that the method adapted to real data from another species, thus we used the AnimalPoseForecasting library to apply our framework to the MABe22 mouse triplet trajectory dataset (Sun et al., 2023). The sensory inputs were similar to those for flies: social vision and social touch were nearly identical. Wall vision (distance to closest wall in binned directions) and wall touch (distance from keypoints to closest wall) were added to allow the arena walls to be sensed. For all, hyperparameters were set based on the arena and mouse body sizes. As in NO HANDCRAFTED POSE, the motion output is the mouse's global velocity plus the change in its keypoints in its own reference frame. Results on the MABe22 dataset, compared to SOTA are:

MethodExp. day ↓Time of day ↓Strain ↑Movement ↑Contact ↑Watching ↑Lights ↑Avg. MSE ↓Avg. F1 ↑
Ignore input**.0957.0921.590.004.187.044.587.0939.213
PCA.0942.0946*.516.005.169.066.546.0944.196
TVAE.0940.0944*.530.008.213.102.568.0942.227
T-Perceiver.0933.0932*.698.014.232.164.697.0933.271
T-GPT.0927.0938*.645.012.252.179.654.0933.274
T-PointNet.0928.0932*.660.036.256.156.672.0930.281
T-BERT.0926.0928*.786.013.266.172.688.0927.297
Ours.0979.0979.813.012.255.111.604.0979.280

*Time of Day MSE reported in the MABe22 paper had values ~0.9. We believe that this was a mistranscription of the results, and that a 0 was missing after the decimal point. We made this correction.
**Ignore input: chance baseline, returns training mean for MSE, positive for F1.

Our model outperforms the baselines (PCA and TVAE) on the categorization tasks (Avg. F1), and its average F1 (0.280) is above four of the six published methods, with best performance on the “Strain” category. It performs worst on the two MSE tasks, but we note that all methods are near chance here. The main complication we noticed with this extension is with noisy trajectories. The agent-centric representation is possibly more sensitive to tracking noise, and we are exploring whether smoothing and automatic data cleaning would help. In addition, the method will also model tracking noise, which is not biologically useful. It was a valuable experiment to be made aware of these issues.


Computational requirements, complexity.
Our newly added 256-d model has fewer parameters, ~6M, compared to the SOTA on MABe22 T-GPT, which has ~170M. Training the 256-d model for 500 epochs took ~45.5 hours on a single H100. Training the full 2048-d model took ~106 H100-hours.

Fig. 5b F1 scores
Figure 13 in the Appendix shows the full confusion matrices. F1 scores, which we can add to the plot:

ClassShare of promptsF1, mixed ignoredF1, mixed = wrong
female0.790.9700.947
male0.210.8380.739
courtship0.470.8320.715
not courtship0.530.8410.729
blind0.120.7280.585
not blind0.880.9510.911
91B010.100.7240.642
not 91B010.900.9800.971

Typos.
Thank you for catching these, we will correct them!

Reviewer_gq5COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an agent-centric generative framework for modeling fruit fly social behavior using causal transformers. Each fly is modeled as an independent agent that receives egocentric sensory observations and predicts egocentric movements, allowing social interactions to emerge through closed-loop simulation. The authors introduce the AnimalPoseForecasting library to support composable transformations between sensory inputs, pose representations, and predicted actions.

The paper systematically evaluates several representation choices, including agent-centric versus world-centric representations, continuous versus discretized predictions, and different pose parameterizations. Experiments on the MABe 2022 fly dataset demonstrate realistic behavioral patterns, improved distribution matching, and strong performance on behavior representation learning benchmarks. Additional experiments using RatInABox illustrate the framework's applicability to another domain.

Critical strengths
  1. Biologically motivated agent-centric modeling. The paper introduces a biologically motivated framework in which each fly independently processes sensory information and generates movements. This decentralized formulation provides a meaningful alternative to conventional world-centric multi-agent forecasting and enables plausible social interactions through closed-loop simulation.
  2. Systematic evaluation of representation choices. The authors investigate several important modeling decisions, including output discretization, pose parameterization, and temporal context length. The experiments provide useful insights into the importance of probabilistic prediction for multimodal behaviors and the trade-offs between matching marginal distributions and maintaining coherent long-term trajectories.
  3. Strong empirical results and reusable infrastructure. The framework reproduces several recognizable fly behaviors, including walking, chasing, and wing extension, and achieves strong results on MABe 2022 representation learning tasks. The accompanying AnimalPoseForecasting library provides reusable infrastructure for implementing and evaluating alternative representations, with an additional demonstration on RatInABox.
Critical weaknesses
  1. The agent-centric versus world-centric comparison is not sufficiently controlled. The KEYPOINTS baseline simultaneously changes multiple components, including sensory inputs, pose representations, and prediction targets. Although the authors compare it against the NO DISCRETIZATION variant, the experiments do not independently isolate the effects of the agent-centric sensory representation and the egocentric action representation. Consequently, the evidence supporting the central claim that agent-centric modeling is responsible for the improvements is not fully conclusive.
  2. The state-of-the-art comparison is potentially unfair due to differences in training data. The results in Section 6.2 and Figure 5 are compared against previously published MABe 2022 methods, but the proposed model is trained using an augmented unsupervised dataset. Without controlling for the amount and composition of training data, it is difficult to determine how much of the reported improvement comes from the proposed representation rather than additional training data.
  3. Long-horizon simulation fidelity remains limited. Despite promising qualitative examples, the generated trajectories remain distinguishable from real behavior. In particular, the velocity-based discriminator achieves approximately 0.92 accuracy even within the first four simulated frames (Appendix B.3). Furthermore, the authors primarily evaluate behavior frequencies and feature distributions over the first 64 frames to avoid severe rollout degradation. These results suggest that the model has not yet fully captured important aspects of the real behavioral distribution, especially over longer horizons.
Questions for authors
  1. Can the authors provide a more controlled ablation of the agent-centric representation? Specifically, can the sensory input representation and action output representation be varied independently while keeping the architecture, prediction distribution, and training protocol fixed? This would help establish which components are responsible for the improvements over world-centric modeling.
  2. How much of the MABe 2022 performance improvement can be attributed to the additional training data? Could the authors provide results using only the original benchmark training data, or compare against baselines retrained on the same augmented dataset? Such a comparison would help clarify the contribution of the proposed modeling framework.
  3. What are the main sources of the discrepancy between real and simulated behavior, particularly in the velocity distributions? Given the high discriminator accuracy even at very short horizons, could the authors further analyze whether these discrepancies arise from independent sampling of output dimensions, deterministic prediction of continuous features, or accumulated rollout errors? Quantifying their relative contributions would help assess the model's current limitations.
3 comments

Thank you for your careful and constructive review. We appreciate your recognition of the biologically motivated, decentralized formulation, the systematic evaluation of representation choices, and the reusable library. Your concerns are well taken, and we ran new experiments to address them: a more complete ablation study that varies the sensory input and motion output independently, a re-evaluation on the original MABe22 data and splits with a 256-dimensional model, and a new model that samples output features jointly rather than independently, which makes simulated movement much harder to distinguish from real movement. We respond to each point below.


Fairer comparison with MABe22 published results.
We agree that the comparison would be stronger by exactly matching the dataset and evaluation to the original MABe 2022 challenge splits, and we have performed several new experiments to address this. We note that there was no data leakage: our models were trained only on the User-train split, and the data augmentation kept all videos from the original MABe22 challenge in their original split. This is stricter than the challenge required, since MABe22 allowed training on all the data (excluding labels), as it is an unsupervised task.

We retrained our model using the dataset in the original MABe 2022 fly dataset [Sun et al., 2023]. Evaluation was done using the MABe 2022 evaluation code, with probes trained on the “evaluator train” split, “test1” used for choosing between a few options, and final results reported for “test2”. MABe 2022 also required a 256-dimensional representation, while our original dimensionality was 2048 (the hidden state dimensionality of our model). We tried a few different methods for reducing dimensionality from 2048 to 256, and based on “test1” evaluation, decided to use a narrower model with dimensionality 256. Our updated results, compared to the MABe 2022 SOTA:

MethodFly typeStim. controlStim. aggr.Line categoryMale/femaleManualAll tasks
Ignore input*0.1650.5880.5890.3530.9000.3080.435
PCA0.2820.4660.4840.5530.9900.2300.425
TVAE0.1990.5000.4500.3410.8210.2220.370
T-Perc0.3940.4180.5130.5730.9820.1970.448
T-GPT0.3630.5150.5000.5570.8730.2460.458
Ours, d=256, original data0.3820.5020.4700.5570.9900.7570.514

*Ignore input: chance baseline; predicts positive for every frame.

Our method continues to outperform SOTA: on average over all tasks, our model (0.514) outperforms all published methods (best: T-GPT, 0.458). Measured from the baseline that ignores the input (0.435), its gain is 3.4× T-GPT's. The largest improvement is on expert-annotated behaviors ("Manual"), where its F1 is 3.1× that of the best published method.

To separate the effects of model size and training data, we also trained a larger model (d_model = 2048) on the original data.

MethodFly typeStim. controlStim. aggr.Line categoryMale/femaleManualAll tasks
Ours, d=256, original data0.3820.5020.4700.5570.9900.7570.514
Ours, d=2048, original data0.4560.4990.4460.6090.9960.8060.536
Ours, d=2048, augmented data0.3550.5010.4540.6600.9900.8760.532

- Dimensionality. Increasing d_model from 256 to 2048 improves the average from 0.514 to 0.536, mainly on fly type (0.382 → 0.456), line category (0.557 → 0.609) and expert-annotated behaviors (0.757 → 0.806).
- Training data. The augmented data does not change the average (0.536 vs. 0.532). It improves line category (0.609 → 0.660) and expert-annotated behaviors (0.806 → 0.876), but lowers fly type (0.456 → 0.355).
Note we re-evaluated the model from the paper (d_model = 2048, trained on the augmented data) with the MABe22 matched protocol, thus results are slightly different from those in Fig. 5a.

Controlled ablation of the agent-centric representation.

By agent-centric we mean that both the sensory input and the motion output are expressed in the agent's own reference frame. KEYPOINTS was designed to remove both at once, testing the representation as a whole, and it is directly comparable to NO DISCRETIZATION. For clarification, KEYPOINTS inputs the flattened keypoints of all other animals and outputs the continuous velocity of the keypoints. NO DISCRETIZATION inputs the sensory representation of all other animals and outputs the continuous agent-centric velocity.

We agree that it is interesting to understand whether the agent-centric input representation, output representation, or both that contribute to the difference in performance. To answer this, we trained two new ablations:

WorldSensory+AgentMotion changes only the sensory representation compared to NO HANDCRAFTED POSE.
AgentSensory+WorldMotion changes only the output representation compared to NO DISCRETIZATION.

We find that nearly every metric is worse for all three ablations (16/18 comparisons), showing that both the input and the output representation matter. Interestingly, input representation most strongly affects behavior frequency error (0.185 → 0.5); simulated flies walk 32% more than real flies and chase, court and extend their wings 50–60% less. Output representation most strongly affects pose distance (0.004 → 0.010) and probe MCC (0.518 → 0.382). Removing both is worse on every metric.

Results next to the comparable agent-centric reference:

ModelSensory inputPose dist ↓Vel. dist ↓Real-vs-sim pose ↓Real-vs-sim vel. ↓Behavior freq. err ↓Probe MCC ↑
NO HANDCRAFTED POSEAgent0.0050.0560.6500.9500.1850.504
WorldSensory+AgentMotionWorld0.0060.0540.6700.9540.5000.496
ModelSensory inputMotion outputPose dist ↓Vel. dist ↓Real-vs-sim pose ↓Real-vs-sim vel. ↓Behavior freq. err ↓Probe MCC ↑
NO DISCRETIZATIONAgentAgent (joint-angle velocities)0.0040.1070.6500.9500.2630.518
AgentSensory+WorldMotionAgentWorld (keypoint velocities)0.0100.1220.6810.9890.1940.382
KEYPOINTSWorldWorld (keypoint velocities)0.0120.1230.7000.9900.3690.337

Difference between each ablation and its reference model (ablation − reference). Bold marks a change where the ablation made the metric worse.

AblationReferencePose dist ↓Vel. dist ↓Real-vs-sim pose ↓Real-vs-sim vel. ↓Behavior freq. err ↓Probe MCC ↑
WorldSensory+AgentMotionNO HANDCRAFTED POSE+0.001−0.002+0.020+0.004+0.315−0.008
AgentSensory+WorldMotionNO DISCRETIZATION+0.006+0.015+0.031+0.039−0.069−0.136
KEYPOINTSNO DISCRETIZATION+0.008+0.016+0.050+0.040+0.106−0.181

We will add these results to the paper.

Long horizon fidelity.
Our simulations are autoregressive: the model predicts a velocity, we integrate it to obtain the next pose, and that pose is fed back as input. Prediction errors therefore compound over time, and integration can produce poses outside the training distribution, which degrades later predictions further. This is a known problem in behavior cloning (Ross et al., 2011). Errors in long simulations could thus come from the model failing to capture long-timescale structure, or from this accumulation of errors. Several approaches could reduce compounding error within our framework: training on the model's own rollouts (Ross et al., 2011; Bengio et al., 2015), fine-tuning with reinforcement learning using a learned discriminator as reward (Ho & Ermon, 2016; Lu et al., 2023), and searching over sampled futures to avoid low-likelihood states (Janner et al., 2021; Igl et al., 2022). These are outside the scope of this work, but we plan to explore them in the future. In this work, our main metrics therefore use the first 64 simulated frames, and Figs. 8 and 10 show how discriminator accuracy, behavior frequencies and feature means change over the full 512-frame simulations. We also note that, since the end goal is to interrogate the model, reducing error accumulation during simulation may not improve the model.

Real vs sim discrepancy.
We believe the discrepancy comes from both sampling output features independently and from the continuous features. Over 64 frames of simulation, a discriminator trained on any single discretized feature is near chance (0.52–0.53; Figs. 3c, 9), so the distribution of each feature is well matched. Trained on the discretized features jointly, it reaches 0.67 (0.63 for the two wing angles alone), which shows that sampling these features independently does not capture how they are coordinated. The continuous features account for most of the discrepancy: trained on them jointly, the discriminator reaches 0.96, the same score as with all features.

As part of our ongoing work, we have developed a new architecture, our "flattened" model, which discretizes all output features and samples them one at a time, each conditioned on the features already sampled for that frame. With this model that both discretizes all features and conditionally samples features, our real-vs-sim discrimination score is 0.618, a large improvement (REFERENCE model: 0.960, best model on this metric DISCRETIZE_ALL: 0.860, chance: 0.5).

We realize that this model is a substantial addition to the paper. We would be happy to add it to the final version, but could also leave it to future work if reviewers prefer. Briefly, our model follows other approaches that discretize each dimension and predict the dimensions autoregressively (Janner et al., 2021; Reed et al., 2022; Brohan et al., 2023). Our model differs from these in that the main transformer has one token per time point, and a separate small transformer decoder head samples the features within each frame one at a time, similar to the RQ-Transformer (Lee et al., 2022) and MEGABYTE (Yu et al., 2023). Here is the summary table of results with the new FLATTENED model added

ModelPose distance ↓Velocity distance ↓Real-vs-sim pose ↓Real-vs-sim velocity ↓Behavior freq. error ↓Probe score ↑
Flattened0.0030.0150.6110.6180.1560.540
Reference0.0050.0560.6800.9600.1980.512
Shorter context0.0060.0420.6900.9400.2250.426
No discretization0.0040.1070.6500.9500.2630.518
Discretize all0.0120.0270.7900.8600.2900.536
Static local pose0.0080.0640.7000.9700.2160.476
No handcrafted pose0.0050.0560.6500.9500.1850.504
Keypoints0.0120.1230.7000.9900.3690.337
Reviewer_DMwEOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a software library for training and simulating agent-centric models of animal behavior, using a transformer to map a history of sensory inputs to future actions. This library is used to instantiate agent-centric models that test a wide range of sensory/action representations. These representations are evaluated via comparison of feature distributions, a real-vs-simulated discriminator, and behavior class frequencies.

Critical strengths

This paper addresses an important problem in behavioral and social neuroscience, namely: how do we begin to study the algorithmic description of animal behavior? Detailed biomechanical models offer one level of description, but the modeling approach presented here offers a more abstract level and complementary level of description.

The choice of input/output representations is a crucial step towards making such models work in practice, and this paper performs thorough comparisons across a range of options, and provides intuition for why certain representations work better than others.

The introduction of the AnimalPoseForecasting software library is laudable and will hopefully kick-start more work in this field by lowering the barrier to entry.

Critical weaknesses

I wouldn't say this is critical, but I found the figures small and hard to parse.

Questions for authors

One of the clear paths to advancement for LLMs has been the scaling laws. Given the authors have access to large amounts of data, I'm curious if they have explored performance of their, say, best model as a function of model size and training data. Is there a steady improvement across all evaluation metrics, or are there step changes? Even a hint at some results here could be useful for new entrants into the field.

I'd like to authors to speculate on how this modeling approach will transfer to other species, say mice. Most of the input/output representations seem straightforward to map onto a mouse skeleton. Do the authors foresee any additional complications?

2 comments

Thank you for your encouraging review, and for seeing this work as a step toward an algorithmic description of animal behavior. We are glad you found the comparison of representations informative, and we share your hope that the AnimalPoseForecasting library will lower the barrier to entry for this field.

Generalization of AnimalPoseForecasting library and framework to other datasets.
There is no reason to expect our framework to work only for flies: the sensory inputs, pose representation and motion outputs are built from general operations in our library.

In the paper, we showed qualitatively that the framework models RatInABox data well (Fig. 17b and Supplementary Video 11). To make this more quantitative, we calculated the Wasserstein distance between 20 rollouts sampled from our model and 20 groundtruth RatInABox rollouts from the same 0.8 s initialization, mean over 1,000 held-out episodes. For comparison, we also computed the minimum achievable distance at this sample size based on a second, independent set of 20 RatInABox rollouts from the same initialization, and a ceiling by computing the distance to RatInABox rollouts from another episode initialization. Our model’s distances are close to the minimum achievable distance and far below the different-initialization ceiling. We will add figure versions of these tables to the revised paper.

Position: Wasserstein-1 distance (cm)

0.5 s1 s2 s4 s
Ours1.11 ± 0.012.01 ± 0.022.87 ± 0.033.94 ± 0.05
Minimum achievable1.08 ± 0.011.92 ± 0.012.52 ± 0.023.16 ± 0.04
Different initialization24.23 ± 0.4422.45 ± 0.4318.47 ± 0.3812.49 ± 0.27

Forward speed: Wasserstein-1 distance (mm/s)

0.5 s1 s2 s4 s
Ours9.38 ± 0.159.13 ± 0.158.78 ± 0.138.65 ± 0.12
Minimum achievable8.37 ± 0.137.51 ± 0.136.03 ± 0.105.66 ± 0.10
Different initialization25.03 ± 0.6119.08 ± 0.5418.78 ± 0.5518.08 ± 0.36

Sideways speed: Wasserstein-1 distance (mm/s)

0.5 s1 s2 s4 s
Ours5.51 ± 0.083.95 ± 0.053.07 ± 0.042.56 ± 0.03
Minimum achievable4.98 ± 0.073.57 ± 0.042.71 ± 0.042.14 ± 0.03
Different initialization17.62 ± 0.3513.39 ± 0.279.17 ± 0.186.33 ± 0.12

Angular velocity: Wasserstein-1 distance (rad/s)

0.5 s1 s2 s4 s
Ours0.210 ± 0.0030.157 ± 0.0030.154 ± 0.0040.203 ± 0.004
Minimum achievable0.201 ± 0.0030.149 ± 0.0030.139 ± 0.0040.174 ± 0.004
Different initialization0.822 ± 0.0200.684 ± 0.0170.619 ± 0.0160.604 ± 0.011

Generalization of AnimalPoseForecasting library and framework to other datasets (continued).

We agree that the paper would be stronger if we could show that the method adapted to real data from another species, thus we used the AnimalPoseForecasting library to apply our framework to the MABe22 mouse triplet trajectory dataset (Sun et al., 2023). The sensory inputs were similar to those for flies: social vision and social touch were nearly identical. Wall vision (distance to closest wall in binned directions) and wall touch (distance from keypoints to closest wall) were added to allow the arena walls to be sensed. For all, hyperparameters were set based on the arena and mouse body sizes. As in NO HANDCRAFTED POSE, the motion output is the mouse's global velocity plus the change in its keypoints in its own reference frame. Results on the MABe22 dataset, compared to SOTA are:

MethodExp. day ↓Time of day ↓Strain ↑Movement ↑Contact ↑Watching ↑Lights ↑Avg. MSE ↓Avg. F1 ↑
Ignore input**.0957.0921.590.004.187.044.587.0939.213
PCA.0942.0946*.516.005.169.066.546.0944.196
TVAE.0940.0944*.530.008.213.102.568.0942.227
T-Perceiver.0933.0932*.698.014.232.164.697.0933.271
T-GPT.0927.0938*.645.012.252.179.654.0933.274
T-PointNet.0928.0932*.660.036.256.156.672.0930.281
T-BERT.0926.0928*.786.013.266.172.688.0927.297
Ours.0979.0979.813.012.255.111.604.0979.280

*Time of Day MSE reported in the MABe22 paper had values ~0.9. We believe that this was a mistranscription of the results, and that a 0 was missing after the decimal point. We made this correction.
**Ignore input: chance baseline, returns training mean for MSE, positive for F1.

Our model outperforms the baselines (PCA and TVAE) on the categorization tasks (Avg. F1), and its average F1 (0.280) is above four of the six published methods, with best performance on the “Strain” category. It performs worst on the two MSE tasks, but we note that all methods are near chance here. The main complication we noticed with this extension is with noisy trajectories. The agent-centric representation is possibly more sensitive to tracking noise, and we are exploring whether smoothing and automatic data cleaning would help. In addition, the method will also model tracking noise, which is not biologically useful. It was a valuable experiment to be made aware of these issues.

Other concerns. We are interested in exploring the effects of model and training data size to see if scaling parallels results with LLMs, but do not yet have conclusive results. We will enlarge the figures in the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal summary

We thank all the reviewers for their time and thoughtful feedback. Reviewers found the problem important (DMwE) and the research area essentially unexplored (UDkB), and all three found our agent-centric approach well motivated as a biologically grounded, algorithmic-level description of behavior (DMwE, gq5C, UDkB). Two reviewers noted the thoroughness of our evaluation of representation choices and the value of the AnimalPoseForecasting library (DMwE, gq5C). The reviewers raised several good questions, which we addressed with new experiments and analyses:

  • MABe22 probe benchmark (gq5C, UDkB). We improved our comparison to the MABe22 benchmark results by using the original dataset and the specified dimensionality limit. Our method continues to outperform the SOTA.

  • Agent-centric ablations (gq5C, UDkB): Two new models vary the input and output representations independently. Removing the agent-centric representation from either one makes nearly every metric worse, so both matter.

  • Other species (DMwE, UDkB). We used our AnimalPoseForecasting library to adapt our approach to a real mouse dataset, the MABe22 mouse triplet dataset, and report results competitive with SOTA.

  • RatInABox (UDkB). We added a quantitative evaluation of the RatInABox experiments, and show our model nears the minimum achievable Wasserstein distance in capturing the groundtruth distributions.

  • Real-vs-simulated discrepancy (gq5C). A new model that samples output features jointly rather than independently reduces velocity discriminator accuracy from 0.86 to 0.62.

These results are in the comments to these individual reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.