- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the fact that primate behaviour datasets use different ethograms whose categories differ in granularity and semantics. It proposes Compositional Ethogram Factorization (CEF), which decomposes each dataset's categories into partial assignments over eight typed behavioural factors. The eight factors are predicate, locomotion mode, direction, posture, target, social role, interaction mode and signal mode, with 37 values in total. Factors not determined by a category definition are treated as unspecified rather than negative, so that factor-level supervision can be shared across datasets without merging their label spaces. On this basis the authors convert the annotations of PanAf20K, PanAf500, BaboonLand and ChimpACT into PrInstruct-34K, a set of 34,626 instruction instances over 58 categories in three tasks: multiple-choice QA with 17,362 instances, behaviour captioning with 15,542, and multi-label temporal parsing with 1,722. Captions are generated by Qwen3.5-397B conditioned on the annotated categories and definitions and then filtered for consistency. The model, EthoBehavior, adds an Ethogram Composition Adapter to Qwen3-VL-4B/8B. Pooled visual tokens plus a task embedding yield per-value factor probabilities trained with partially supervised binary cross-entropy. These probabilities are encoded as 16 static factor tokens, selected as the two most probable values per factor and gated by a threshold, and as 8 dynamic residual codebook tokens, and they also modulate the visual tokens through a low-rank branch before the language model. Training combines the generation loss with factor, alignment and regularisation terms. Experiments report that EthoBehavior outperforms zero-shot general-purpose and video-language models on the three PrInstruct-34K tasks, achieves the highest mAP on ChimpACT under the PriVi protocol, improves over Qwen3-VL on unseen factor compositions, and transfers poorly to Animal Kingdom.
Critical strengths
- The problem is real and the formalisation is careful. Ethograms of different primate datasets genuinely do not map one-to-one, and sharing factor-level supervision through partial assignments, with unspecified values explicitly distinguished from negatives, is a well-considered design; the negative-inference rule in Appendix A is clearly stated.
- The dataset construction is documented. Four benchmarks are converted into three task formats, all instances of a video share a partition, overlapping PanAf20K/PanAf500 videos are re-partitioned by hash, and the caption filtering rules and per-source rejection rates are reported, with an overall rejection rate of 10.5%.
- The evaluation is broad and does not hide negative results. Beyond the three generative tasks, the paper reports ChimpACT mAP under the PriVi protocol, designs an unseen-factor-composition test, and reports candidly that zero-shot transfer to Animal Kingdom does not yield an advantage.
Critical weaknesses
W1. The factorization rewrites coarse categories as equally coarse factor assignments, so information lost at annotation time is not recovered; and the complete category-to-factor mapping for the 58 categories is not provided. The information in question is lost at annotation time. The introduction's own example is that one ethogram encodes locomotion as a higher-level category while another distinguishes specific motor patterns such as walking or jumping. In a clip annotated as locomotion the animal may be walking, running, jumping or climbing; the annotator recorded only that it moved, and which motor pattern occurred can no longer be recovered from the category. CEF maps each category to factor clauses from its operational definition alone, without revisiting the videos: Section 4.2 states that the four benchmarks' annotations are reformulated without changing their semantics, and no new human annotation is reported. A coarse category such as locomotion, or travel in PanAf20K, is therefore assigned only the predicate move, with locomotion mode, direction and posture left unspecified; the coarse category is rewritten as an equally coarse factor assignment. Without re-annotation, such clips are no finer in the factor space than in their native ethogram, so the shared space does not add resolution beyond the original categories, and what the factorization changes is the form of the supervision rather than its content. Under the paper's own motivation, clips carrying coarse categories would need to be re-annotated at the factor level, at least for a subset. Section 3 further specifies that a category defined with alternatives, such as BaboonLand's walking or running, entails only move, while walk and run are possible values that receive no supervision, so in these cases the factor supervision is coarser than the original category. In my view, as long as the source annotation consists of coarse categories, the information that the factorization can add is limited, and this bears directly on the value of the core contribution. Second, the complete mapping that specifies which factor values each of the 58 categories decomposes into, which categories have more than one clause, and which values are inferred as confirmed negatives is not given in the paper or the appendix; only four worked examples are provided, namely locomotion, walking, climbing up, and walking or running. For categories defined by function, context or a sequence of actions, such as displaying in ChimpACT or display, aggression, social interaction and cross species interaction in PanAf20K, it is unclear whether a conjunction of a predicate with mode and direction values can express them, or how the authors decomposed them in practice. Since the mapping is the foundation on which the method rests, it should be provided in full.
W2. The paper is motivated by the limited comparability of behavioral measurements across studies and concludes that CEF provides a basis for comparable primate behavior analysis, but the comparability problem is not yet solved: the shared factor space exists only at the level of definitions and is never read out, validated or used for any comparison. The research question posed in the introduction is how heterogeneous ethograms can be represented in a shared behavioral space while preserving the ethogram-specific semantics of their categories. The CEF mapping answers this question by construction: categories from any two ethograms can be aligned on the factor values they both entail, for example travel and walking both entail move. This remains, however, a scheme at the level of definitions, and the paper does not provide experiments showing that it can be used in practice. The only quantities in the model that live in the shared space are the factor probabilities estimated by the ECA, and these are never reported: no accuracy for factor-level predictions is given, no factor-level comparison across datasets is made, and no experiment trains on one ethogram and reads out another ethogram's categories through the factors. Factor-level accuracy would be straightforward to evaluate, since ground truth for entailed values follows directly from the mapping. As Sections 4.1 and 5.1 and Appendix D state, every dataset is evaluated in its native ethogram. That is a legitimate evaluation choice, since ground truth exists only in the native categories, but it means that the claim of comparability has no experimental support; even the unseen-factor-composition experiment, the one experiment framed in terms of factors, is scored by question-answering F1 on native-category questions rather than by factor predictions. Since comparability is the paper's stated goal, such a comparison should be carried out if it is feasible, and if it is not, the paper should explain why. Moreover, under the formulation in Section 5.1 the factor probabilities depend on the task embedding, so the same video yields different factor probabilities under different tasks; a measurement intended for comparison across studies should not change with the question asked. As the paper stands, therefore, the comparability problem it sets out to address remains unsolved.
W3. Two baseline rows of Table 1 are identical, several counts are inconsistent, and the data sources are heavily unbalanced. The Qwen3-VL-4B and Qwen3-VL-8B rows of Table 1 are identical in nine columns: question-answering F1 is 23.21 in both, recall is 24.26 in both, the four captioning metrics are 14.66, 18.85, 33.96 and 70.52 in both, and the three temporal-parsing metrics are 8.93, 25.34 and 33.35 in both, with only precision and accuracy differing. Identical rows for two models of different size are unexpected and should be rechecked, particularly because the reported gain over the matched backbone rests on these rows. The number of factor values is also inconsistent: Figure 1 gives 37, whereas Appendix C.6 gives K+ = 39, although K+ by definition counts only values with at least one confirmed positive and cannot exceed the vocabulary size. The per-source category counts in Table 4 are 18, 9, 13 and 23, which sum to 63, while the total is given as 58, without explanation. The dataset is also heavily unbalanced across sources: PanAf20K contributes 29,858 of the 34,626 instances, which is 86%, and ChimpACT only 482. Finally, although Appendix D states that every metric is computed separately for each dataset in its native category space, Table 1 does not state how the four datasets are aggregated into a single number.
Questions for authors
- Please recheck the Qwen3-VL-4B and Qwen3-VL-8B rows of Table 1, which are identical in nine columns. How are the per-dataset metrics aggregated into the single numbers in Table 1?
- Please provide the complete CEF mapping from the 58 categories to factor clauses and state: how many categories map to a single predicate with all other factors unspecified; how categories such as displaying, display, aggression, social interaction and cross species interaction are decomposed; whether there are 37 or 39 factor values; and why the per-source category counts sum to 63 while the total is 58.
- Could the authors report the accuracy of the factor-level predictions against the entailed factor labels, and show at least one cross-ethogram use of the shared space, for example comparing the predicted factor distributions of two datasets, or training on PanAf and reading out BaboonLand or ChimpACT categories through the factors? If such a comparison cannot be carried out, please explain why. Since the factor probabilities depend on the task embedding, which task's output is meant to serve as the cross-study measurement? Could the unseen-composition experiment include a backbone fine-tuned with the same recipe but without factors?
- Have the authors considered re-annotating, at least for a subset, the videos carrying coarse categories such as travel or locomotion at the factor level?