acceptodds
Under review as a conference paper at ICLR 2027

EthoBehavior: Learning Compositional Ethogram for Primate Behavior Understanding

Abstract

Primate behavior understanding seeks to identify and temporally characterize behavioral events and states from video, providing scalable measurements for research in cognition and sociality. Existing methods operate within independent ethograms whose categories differ in granularity and annotation scope, limiting the comparability of behavioral measurements across studies. Although Video-Language Models (VLMs) capture fine-grained visual cues, they typically treat ethogram categories as atomic concepts and remain agnostic to the structured behavioral semantics. To address these limitations, we introduce Compositional Ethogram Factorization, a shared representation that models primate behaviors as structured combinations of reusable behavioral predicates and typed attributes. We further instantiate it in PrInstruct-34K, a multi-task video instruction dataset comprising 34K examples from four primate behavior benchmarks. It retains 58 ethogram categories and standardizes their operational definitions into three complementary tasks: multiple-choice question answering, video captioning, and multi-label temporal parsing. Building on this, we develop EthoBehavior, a VLM with an Ethogram Composition Adapter that estimates factor distributions from video under partial supervision and encodes them as continuous Ethogram Factor Tokens for free-form behavior description and structured behavior analysis. Experiments across all three tasks and on unseen factor compositions show that EthoBehavior consistently outperforms instruction-tuned baselines and large open-source VLMs including GLM-5.3-Flash and DeepSeek-V4.1-Flash. Together, these results establish compositional ethogram representations as a principled basis for generalizable and comparable primate behavior analysis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Behaviour Understanding, Video-Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vk1bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the fact that primate behaviour datasets use different ethograms whose categories differ in granularity and semantics. It proposes Compositional Ethogram Factorization (CEF), which decomposes each dataset's categories into partial assignments over eight typed behavioural factors. The eight factors are predicate, locomotion mode, direction, posture, target, social role, interaction mode and signal mode, with 37 values in total. Factors not determined by a category definition are treated as unspecified rather than negative, so that factor-level supervision can be shared across datasets without merging their label spaces. On this basis the authors convert the annotations of PanAf20K, PanAf500, BaboonLand and ChimpACT into PrInstruct-34K, a set of 34,626 instruction instances over 58 categories in three tasks: multiple-choice QA with 17,362 instances, behaviour captioning with 15,542, and multi-label temporal parsing with 1,722. Captions are generated by Qwen3.5-397B conditioned on the annotated categories and definitions and then filtered for consistency. The model, EthoBehavior, adds an Ethogram Composition Adapter to Qwen3-VL-4B/8B. Pooled visual tokens plus a task embedding yield per-value factor probabilities trained with partially supervised binary cross-entropy. These probabilities are encoded as 16 static factor tokens, selected as the two most probable values per factor and gated by a threshold, and as 8 dynamic residual codebook tokens, and they also modulate the visual tokens through a low-rank branch before the language model. Training combines the generation loss with factor, alignment and regularisation terms. Experiments report that EthoBehavior outperforms zero-shot general-purpose and video-language models on the three PrInstruct-34K tasks, achieves the highest mAP on ChimpACT under the PriVi protocol, improves over Qwen3-VL on unseen factor compositions, and transfers poorly to Animal Kingdom.

Critical strengths
  1. The problem is real and the formalisation is careful. Ethograms of different primate datasets genuinely do not map one-to-one, and sharing factor-level supervision through partial assignments, with unspecified values explicitly distinguished from negatives, is a well-considered design; the negative-inference rule in Appendix A is clearly stated.
  2. The dataset construction is documented. Four benchmarks are converted into three task formats, all instances of a video share a partition, overlapping PanAf20K/PanAf500 videos are re-partitioned by hash, and the caption filtering rules and per-source rejection rates are reported, with an overall rejection rate of 10.5%.
  3. The evaluation is broad and does not hide negative results. Beyond the three generative tasks, the paper reports ChimpACT mAP under the PriVi protocol, designs an unseen-factor-composition test, and reports candidly that zero-shot transfer to Animal Kingdom does not yield an advantage.
Critical weaknesses

W1. The factorization rewrites coarse categories as equally coarse factor assignments, so information lost at annotation time is not recovered; and the complete category-to-factor mapping for the 58 categories is not provided. The information in question is lost at annotation time. The introduction's own example is that one ethogram encodes locomotion as a higher-level category while another distinguishes specific motor patterns such as walking or jumping. In a clip annotated as locomotion the animal may be walking, running, jumping or climbing; the annotator recorded only that it moved, and which motor pattern occurred can no longer be recovered from the category. CEF maps each category to factor clauses from its operational definition alone, without revisiting the videos: Section 4.2 states that the four benchmarks' annotations are reformulated without changing their semantics, and no new human annotation is reported. A coarse category such as locomotion, or travel in PanAf20K, is therefore assigned only the predicate move, with locomotion mode, direction and posture left unspecified; the coarse category is rewritten as an equally coarse factor assignment. Without re-annotation, such clips are no finer in the factor space than in their native ethogram, so the shared space does not add resolution beyond the original categories, and what the factorization changes is the form of the supervision rather than its content. Under the paper's own motivation, clips carrying coarse categories would need to be re-annotated at the factor level, at least for a subset. Section 3 further specifies that a category defined with alternatives, such as BaboonLand's walking or running, entails only move, while walk and run are possible values that receive no supervision, so in these cases the factor supervision is coarser than the original category. In my view, as long as the source annotation consists of coarse categories, the information that the factorization can add is limited, and this bears directly on the value of the core contribution. Second, the complete mapping that specifies which factor values each of the 58 categories decomposes into, which categories have more than one clause, and which values are inferred as confirmed negatives is not given in the paper or the appendix; only four worked examples are provided, namely locomotion, walking, climbing up, and walking or running. For categories defined by function, context or a sequence of actions, such as displaying in ChimpACT or display, aggression, social interaction and cross species interaction in PanAf20K, it is unclear whether a conjunction of a predicate with mode and direction values can express them, or how the authors decomposed them in practice. Since the mapping is the foundation on which the method rests, it should be provided in full.

W2. The paper is motivated by the limited comparability of behavioral measurements across studies and concludes that CEF provides a basis for comparable primate behavior analysis, but the comparability problem is not yet solved: the shared factor space exists only at the level of definitions and is never read out, validated or used for any comparison. The research question posed in the introduction is how heterogeneous ethograms can be represented in a shared behavioral space while preserving the ethogram-specific semantics of their categories. The CEF mapping answers this question by construction: categories from any two ethograms can be aligned on the factor values they both entail, for example travel and walking both entail move. This remains, however, a scheme at the level of definitions, and the paper does not provide experiments showing that it can be used in practice. The only quantities in the model that live in the shared space are the factor probabilities estimated by the ECA, and these are never reported: no accuracy for factor-level predictions is given, no factor-level comparison across datasets is made, and no experiment trains on one ethogram and reads out another ethogram's categories through the factors. Factor-level accuracy would be straightforward to evaluate, since ground truth for entailed values follows directly from the mapping. As Sections 4.1 and 5.1 and Appendix D state, every dataset is evaluated in its native ethogram. That is a legitimate evaluation choice, since ground truth exists only in the native categories, but it means that the claim of comparability has no experimental support; even the unseen-factor-composition experiment, the one experiment framed in terms of factors, is scored by question-answering F1 on native-category questions rather than by factor predictions. Since comparability is the paper's stated goal, such a comparison should be carried out if it is feasible, and if it is not, the paper should explain why. Moreover, under the formulation in Section 5.1 the factor probabilities depend on the task embedding, so the same video yields different factor probabilities under different tasks; a measurement intended for comparison across studies should not change with the question asked. As the paper stands, therefore, the comparability problem it sets out to address remains unsolved.

W3. Two baseline rows of Table 1 are identical, several counts are inconsistent, and the data sources are heavily unbalanced. The Qwen3-VL-4B and Qwen3-VL-8B rows of Table 1 are identical in nine columns: question-answering F1 is 23.21 in both, recall is 24.26 in both, the four captioning metrics are 14.66, 18.85, 33.96 and 70.52 in both, and the three temporal-parsing metrics are 8.93, 25.34 and 33.35 in both, with only precision and accuracy differing. Identical rows for two models of different size are unexpected and should be rechecked, particularly because the reported gain over the matched backbone rests on these rows. The number of factor values is also inconsistent: Figure 1 gives 37, whereas Appendix C.6 gives K+ = 39, although K+ by definition counts only values with at least one confirmed positive and cannot exceed the vocabulary size. The per-source category counts in Table 4 are 18, 9, 13 and 23, which sum to 63, while the total is given as 58, without explanation. The dataset is also heavily unbalanced across sources: PanAf20K contributes 29,858 of the 34,626 instances, which is 86%, and ChimpACT only 482. Finally, although Appendix D states that every metric is computed separately for each dataset in its native category space, Table 1 does not state how the four datasets are aggregated into a single number.

Questions for authors
  1. Please recheck the Qwen3-VL-4B and Qwen3-VL-8B rows of Table 1, which are identical in nine columns. How are the per-dataset metrics aggregated into the single numbers in Table 1?
  2. Please provide the complete CEF mapping from the 58 categories to factor clauses and state: how many categories map to a single predicate with all other factors unspecified; how categories such as displaying, display, aggression, social interaction and cross species interaction are decomposed; whether there are 37 or 39 factor values; and why the per-source category counts sum to 63 while the total is 58.
  3. Could the authors report the accuracy of the factor-level predictions against the entailed factor labels, and show at least one cross-ethogram use of the shared space, for example comparing the predicted factor distributions of two datasets, or training on PanAf and reading out BaboonLand or ChimpACT categories through the factors? If such a comparison cannot be carried out, please explain why. Since the factor probabilities depend on the task embedding, which task's output is meant to serve as the cross-study measurement? Could the unseen-composition experiment include a backbone fine-tuned with the same recipe but without factors?
  4. Have the authors considered re-annotating, at least for a subset, the videos carrying coarse categories such as travel or locomotion at the factor level?
Reviewer_f8sBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Compositional Ethogram Factorization, which maps dataset-specific primate behavior categories to partial assignments over eight shared behavioral factors. It constructs PrInstruct-34K from four existing benchmarks for multi-label question answering, captioning, and temporal parsing. EthoBehavior augments Qwen3-VL with an adapter that predicts factor probabilities, converts them into explicit and residual factor tokens, and modulates the visual tokens. The model is trained with generation, partially observed factor, alignment, and regularization objectives. Evaluation includes the three instruction tasks, ChimpACT recognition, held-out factor/value groups, and transfer to Animal Kingdom.

Critical strengths
  1. Sharing partial semantics across heterogeneous ethograms addresses a useful problem. Retaining dataset-specific categories and distinguishing unknown factors from confirmed negatives are sensible choices.

  2. PrInstruct-34K combines four benchmarks across recognition, description, and temporal parsing. The matched-backbone gains and component ablations support the practical utility of factor-guided learning.

Critical weaknesses
  1. The connection to cross-ethogram comparability is incomplete. CEF assumes a predefined vocabulary and category-to-factor mapping; retaining dataset-specific labels prevents conflation but does not establish which operational distinctions the shared factors preserve. The scope and semantic fidelity of this mapping need clearer justification.

  2. The compositional claim is stronger than the implemented factor structure. ECA predicts video-level factors without binding them to individuals or intervals: “A grooms B” and “B grooms A” can share the same factor inventory. The visual backbone may recover the relation, but the proposed factor pathway does not explicitly represent it.

Questions for authors
  1. Which differences between ethogram definitions are represented by the shared factors, and which remain only in the original category labels? Please provide the complete mapping, its construction criteria, and how semantic consistency was checked.

  2. Is compositionality intended as factor-level semantic sharing or as modeling relations among actions, participants, and intervals? How does the proposed factor pathway retain clause structure or participant-role binding beyond what the original visual tokens provide?

  3. Can you isolate factor recombination using held-out combinations whose constituent values remain represented in training, with an otherwise comparable category-based representation? The current split removes the constituent values as well.

Reviewer_UuNWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the issue of inconsistent behavioral categories across different primate behavior datasets. The authors propose CEF, a framework that decomposes behavioral categories into eight reusable behavioral factors. This approach enables the sharing of behavioral semantics across diverse datasets while preserving the original category definitions. Based on this representation, the authors constructed PrInstruct-34K and introduced EthoBehavior, which integrates behavioral factor information into vision-language models via ECA and EFTs. The experimental evaluation covers behavioral question answering, video description, temporal parsing, and action recognition, alongside component ablation studies and generalization testing. The results demonstrate good performance in question answering, description generation, and certain action recognition metrics, but performance in temporal parsing and cross domain generalization remains inconsistent.

Critical strengths
  1. Definitions and the level of granularity for behavioral categories vary across different ethograms, which complicates behavioral analysis when working with multiple datasets. Decomposing behavioral categories into shared factors represents a promising approach to addressing this issue.
  2. This method distinguishes between confirmed positive cases, negative cases, and unknown factors. To avoid the pitfall of treating unlabeled behaviors as negative cases and it is suitable for scenarios where behavioral labeling is incomplete.
  3. The paper covers three action understanding tasks and includes additional validation on ChimpACT. Ablation studies also demonstrate that ECA, EFT, and factor supervision contribute to performance.
Critical weaknesses
  1. The current approach incorporates factor supervision, additional labels, a dynamic codebook, and visual feature modulation. Existing ablation studies cannot fully disentangle whether the performance gains stem from the semantic structure provided by the CEF or from the additional model modules and training objectives.
  2. Section 6.5 excludes both the target factor combinations and the values ​​of their constituent factors. Therefore, the experiment cannot independently assess the model’s ability to recombine previously learned factors. And this is the one of the paper’s core claims
  3. In Table 1, the Frame-F1 scores for EthoBehavior-4B and 8B are only 11.64 and 23.06, so both lower than the 33.35 baseline set by Qwen3-VL. Furthermore, the segmental F1 score for the 8B model is lower than that of the 4B model. This stands in contrast to the improvements in temporal behavior understanding emphasized in the paper.
Questions for authors
  1. Could an ablation study be added in which CEF is replaced by standard learnable tokens or class-level representations, while keeping the model parameter count and training configuration consistent?
  2. In combinatorial generalization experiments where individual factor values ​​are retained during training. If specific combinations of these factors are excluded, does EthoBehavior still outperform the baselines?
  3. Why is the Frame-F1 for EthoBehavior significantly lower than the baseline, while the segmental F1 shows an improvement? Could this discrepancy be related to the prediction of temporal boundaries or the evaluation methodology?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.