acceptodds
Under review as a conference paper at ICLR 2027

Beyond Actions and Objects: Video Classification in the Wild

Abstract

Existing video classification datasets are largely built around predefined actions, objects, or their combinations. However, video classification in the wild is more challenging: videos within the same category can vary widely in subjects, scenes, and appearance, while different categories can be visually similar yet semanti-cally distinct. To study this problem, we introduce EventTree, a human-in-the-loop framework that discovers and organizes event categories from large-scale video collections through top-down taxonomy construction. Applying EventTree yields WildEvent-1K, a hierarchical event-centric video classification dataset with 538K videos and 1, 040 fine-grained event categories in a three-level taxonomy. WildEvent-1K exhibits substantial intra-class variation, subtle semantic differences between neighboring categories, and a long-tailed distribution, providing a challenging testbed for real-world classification. These characteristics make conventional video encoders with direct classification heads less effective, motivating classification through next-token prediction with a VLM pretrained across diverse visual domains. To accommodate the hierarchical structure of WildEvent-1K, we propose Generative Hierarchical Video Classification (GHVC), which predicts categories level by level, restricting each prediction to valid children of its predicted parent. Experiments reveal a clear contrast with conventional datasets such as Kinetics-400: specialized video encoders remain strong in action recognition but are substantially less effective on WildEvent-1K, where VLM-based classification performs better. GHVC further improves it through hierarchical next-token prediction, demonstrating its effectiveness for video classification in the wild. Models, code, and data will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
Video classification, Event-centric video understanding, Video datasets, Generative classification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uxuSOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WildEvent-1K organizes diverse videos into a hierarchical label space defined by visible events, using EventTree to combine model-generated descriptions, candidate categories, and human refinement. The benchmark compares specialized video encoders with adapted VLMs and reports a reversal of their relative performance from Kinetics-400. GHVC uses category tokens and parent-constrained generation to predict a valid path through the event taxonomy.

Critical strengths
  1. The event taxonomy makes fine-grained semantic distinctions inspectable. The supplementary cards specify observable evidence, exclusions, and precedence when events coexist. For example, tug, fetch, and human-controlled object play are separated by the interaction structure. Together with the human refinement and residual-discovery examples, this gives the benchmark value beyond a collection of class names.

  2. Separating frozen representations from adaptation reveals an informative change in model behavior. Table 1 shows that encoders retain an advantage under frozen probing, while adapted VLMs perform better on the event benchmark. I find this distinction useful for studying how pretrained models adapt to different label spaces.

Critical weaknesses
  1. Labels were finalized by VLM agreement, so the encoder versus VLM finding has not been checked independently. Model-proposed labels are reasonable for a label space this fine-grained; my concern is the acceptance step. A video is kept only if the GPT branch (reading Qwen3-VL descriptions) and the Qwen3-VL branch agree and Gemma confirms (App. A.5); the rest is discarded. Humans saw 22,500 of 538,539 videos, and in the Figure 2 branch only 27,833 of 47,810 survive. The videos the VLMs could not agree on are the hard cases "in the wild" refers to, and the VLM in Table 1 shares a family with the labeler. I am not asking for relabeling: a human-verified test split with Table 1 rerun on it, plus the discard rate per L1 family, would settle this.

  2. The hierarchical part of GHVC adds little. A flat category token reaches 73.78 leaf accuracy against 73.92 for GHVC in one run (Table 2), and logit adjustment on the linear head beats GHVC on tail macro-F1 (65.4 vs 63.6). The paper [1] find the opposite ranking of discriminative and generative MLLM classifiers and are not compared. GHVC reads better as a benchmark baseline.

  3. Only Qwen VLMs are tested, and Qwen3-VL is also part of the labeling pipeline. One other VLM family would help.

  4. X-CLIP or ViFi-CLIP style classifiers are absent from Table 1. They also use class-name semantics, so I wonder how much of the VLM gain they would recover.

[1] Z. Pang, D. Chatterjee, F. Sener, A. Yao, On Discriminative vs. Generative Classifiers: Rethinking MLLMs for Action Understanding, ICLR 2026

Questions for authors
  1. How many of the 22,500 audited videos were flagged and corrected, and what was the agreement among the three reviewers? The per-reviewer rule in App. A.6 accepts up to 3 errors in a sample of 20, which a class with a 20% error rate passes about 40% of the time, so the observed counts would say much more about label quality than the pass rule.

  2. How was the test split drawn? Spoken Communication makes up 32.2% of training videos but about 13.5% of test videos (Table 6b), and the Table 6 note refers to "uncapped" training frequency.

  3. Which InternVideo2 and X-CLIP checkpoints are used in Figure 6? If they were trained or fine-tuned with Kinetics labels, tighter Kinetics-400 classes are expected and the comparison says less about WildEvent-1K itself.

4 comments

We thank the reviewer for the thoughtful review and suggestions. We respond to the weaknesses below and to the questions in the next comment.

 

W1. Human evaluation, hard cases, and per-L1 retention

Human-verified subset. Three annotators independently labeled 5,000 retained test videos (all 12 L1 families) from the original videos and finalized event cards, without access to released labels or model candidates. After adjudication, 97.4% of the released labels agreed. This is separate from the 22,500-video audit.

Leaf accuracy under these human labels (full-test accuracy in parentheses; encoders: full FT): Video Swin-L 58.0% (58.33%), VideoMAE-H 59.6% (60.24%), InternVideo2-1B 67.9% (68.46%), Qwen3.5-2B linear head 72.2% (72.83%), Flat Category Token 73.5% (73.78%), GHVC 73.8% (73.92%). The encoder–VLM ranking is unchanged.

Hard cases. Of 500 branch-disagreement and 500 Gemma rejections reviewed by humans without access to model predictions, 326 and 264 were assigned valid labels under the finalized taxonomy. On these 590 videos, InternVideo2-1B, Flat Category Token, and GHVC achieved 53.4%, 58.6%, and 59.0%, preserving the encoder–VLM ranking.

L1 familyInitial L1-routedFinal retainedReduction
Spoken Communication219,685164,30725.21%
Life and Society137,58263,99653.49%
Object Display109,951100,4008.69%
Making, Repair and Operation85,35240,79752.20%
Food and Beverage79,55033,62157.74%
Sports and Fitness63,13726,05158.74%
Human–Mammal/Bird Interaction47,81027,83341.78%
Vehicle Movement41,92033,19320.82%
Mammals and Birds38,97113,30665.86%
Human–Aquatic/Other Interaction27,60414,03149.17%
Scenes and Environments21,56313,63236.78%
Aquatic and Other17,6367,37258.20%
Total890,761538,53939.54%

These reductions span taxonomy construction, human refinement, and model-based filtering, not model disagreement or Gemma verification alone. Assignment to an L1 family does not establish that a video satisfies a finalized leaf-event definition: candidates without coherent definitions, distinguishable boundaries, or sufficient supporting examples were revised or removed, and videos not covered by the finalized event cards were not retained merely to increase coverage.

Manual inspection found concentrated recurring content in Spoken Communication, Object Display, and Vehicle Movement, which account for 41.71% of initially routed videos but only 17.02% of leaf categories and retain 80.18%, vs. 46.35% for the other nine. We acknowledge that agreement-based filtering may still exclude valid videos that the construction models find difficult to recognize.

 

W2. Hierarchical contribution, long-tail trade-off, and Pang et al. [1]

We agree that the hierarchical part of GHVC adds only a small gain over Flat Category Token (+0.14% leaf accuracy and +0.33% macro-F1 on 2B) and that GHVC reads better as a benchmark baseline. The revision presents it as such.

Beyond one run. With three seeds, GHVC improves over Flat Category Token on 2B (73.94% vs. 73.75%, +0.19%), 4B (75.54 ± 0.05% vs. 75.28 ± 0.07%, +0.26%), and 9B (76.36 ± 0.05% vs. 76.02 ± 0.08%, +0.34%), so the small gain is consistent.

Logit adjustment. It raises the linear head's tail macro-F1 to 65.4%, above GHVC's 63.6%, at the cost of head accuracy (−1.4%) and overall leaf accuracy (−0.3%). GHVC improves tail macro-F1 over the unadjusted linear head (56.6% → 63.6%) while raising overall leaf accuracy by 1.09%, without class-frequency priors.

Pang et al. [1]. Their discriminative classifier reads out a learned classification token, unlike our linear head with video-token pooling. Under matched training and data, classifiers in the style of their discriminative and GAD designs reach 73.37% and 73.51%, vs. 73.92% for GHVC, so the generative classifier remains ahead on WildEvent-1K.

 

W3. Beyond the Qwen family

Molmo2-O-7B, whose Olmo-based language backbone is not used in the annotation pipeline, reaches 72.04% with a linear head, 73.57% with Flat Category Token, and 73.86% with GHVC. The same ranking holds outside the Qwen family. Its linear head remains above InternVideo2-1B (68.46%), so the encoder versus VLM finding does not rely on a VLM from the labeler's family.

 

W4. X-CLIP and ViFi-CLIP

With the complete L1–L2–leaf category paths and their respective fine-tuning protocols, X-CLIP-B/16 reaches 65.20% leaf accuracy / 62.30% macro-F1 and ViFi-CLIP-B/16 65.81% / 62.91%, vs. 72.83% / 71.22% for the Qwen3.5-2B linear head and 73.92% / 73.19% for GHVC. They remain 8.72% and 8.11% below GHVC in leaf accuracy. Even the linear head, which uses no class-name semantics, outperforms both, so the advantage of adapted VLMs is not explained by class-name semantics alone.

Revision (highlighted). Abstract, contribution list, conclusion; Table 1; Sections 5.2, 5.3; Related Work; Appendices A.5–A.7, C.2, C.8, C.9.

Q1. Audit error counts, adjudication, and reviewer agreement

We agree that the observed audit outcomes are more informative than the acceptance rule. In the original category-stratified audit, all three reviewers independently examined the same 22,500 videos across 1,040 leaf categories.

Audit metricResult
Initial flags and rates, Reviewers 1 / 2 / 3496 (2.20%) / 510 (2.27%) / 490 (2.18%)
Unique videos flagged by at least one reviewer738 (3.28%)
Pre-adjudication agreement on assignment correctness98.18% unanimous; Fleiss’ κ = 0.72
Adjudication-confirmed errors, before correction422 (1.88% of audited videos)
Reassigned / removed after adjudication367 / 55

Initial flags indicate suspected errors; confirmed errors are those upheld after joint adjudication. All adjudication-confirmed errors were reassigned or removed before release. The 15% threshold was an acceptance rule for partially sampled categories, not a guarantee of label accuracy. The confirmed error fraction describes the stratified audit sample before correction, not the remaining error rate after correction or a dataset-wide error estimate or bound.

Manuscript revision. These audit outcomes and agreement statistics have been added to Appendix A.6.

 

Q2. Test-set sampling and “uncapped” training frequency

We agree that the original description of the test-set sampling strategy was insufficiently clear.

Test-set sampling. WildEvent-1K follows a long-tailed class distribution. Strictly proportional sampling would leave many rare categories with very few or even no test examples, whereas a fully class-balanced test set would discard the natural frequency structure and move most videos of small categories out of training. There is no single correct choice here, and long-tailed benchmarks differ on it (for example, ImageNet-LT and iNaturalist 2018 are evaluated on class-balanced test or validation sets). We chose an intermediate design: the number of test videos drawn from each leaf grows sublinearly with the size of the leaf, approximately in proportion to the square root of its number of videos, for a total of about 50K test videos, and all clips from the same source video are kept in the same split to prevent train–test leakage. Frequent categories therefore still contribute more test videos than rare ones, but the imbalance is compressed, and every one of the 1,040 leaves has test examples. This explains why Spoken Communication accounts for 32.2% of training videos but only 13.5% of test videos. Because leaf accuracy is weighted by this test distribution whereas macro-F1 weights all leaves equally, we report both, together with head/medium/tail F1, so that the conclusions do not depend on a single weighting.

Uncapped training frequency. This means that we use all available training videos without truncating or downsampling frequent categories. The head, medium, and tail groups are defined using these complete per-class training counts (≥101, 20–100, and ≤19, respectively). Thus, the rescaling applies only to test-set sampling, while training retains the full long-tailed distribution without class-frequency caps.

Manuscript revision. We have clarified the test-set sampling strategy and the meaning of “uncapped training frequency” in Appendix C.1 and Table 6, with the changes highlighted in the revised manuscript.

 

Q3. Figure 6 checkpoints and possible Kinetics-supervision bias

We agree that Kinetics-supervised features may favor K400 class separability. The frozen checkpoints used in Figure 6 were:

  • InternVideo2-1B: OpenGVLab/InternVideo2-Stage1-1B-224p-K400 (1B_ft_k710_ft_k400_f16.pth), pretrained on K-Mash and subsequently fine-tuned on Kinetics-710 and Kinetics-400.
  • X-CLIP-B/16: microsoft/xclip-base-patch16-16-frames (corresponding to the k400_16_16.pth configuration), trained with Kinetics-400 supervision.

To address the potential bias from Kinetics supervision, we repeated the same 50-draw, 400-class geometry analysis using frozen DINOv2 features without Kinetics-label fine-tuning. The mean intra-class cosine distance increased from 0.36 on K400 to 0.46 on WildEvent-1K, while nearest-class centroid similarity increased from 0.71 to 0.79.

The same trend therefore persists in a representation without Kinetics-specific label supervision: WildEvent-1K exhibits greater intra-class variation and less separable class boundaries.

Manuscript revision. We have documented the exact checkpoints and their supervision histories in Appendix C.2, added the DINOv2 analysis to Appendix C.8, and highlighted the revised interpretation in Section 5.5.

Thanks for the rebuttal. The authors significantly address my concerns, so I'm raising my rating to 4.

We appreciate your support for our work.

Reviewer_ijtPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EventTree, a human-in-the-loop process for discovering an event taxonomy from video descriptions, structured semantic attributes, model-proposed categories, and human inspection of the original videos. Applying it to VidGen-1M produces WildEvent-1K, with 538,539 videos and a three-level taxonomy containing 12 event families, 129 intermediate categories, and 1,040 leaves. Model-assisted assignment and verification are followed by a category-stratified human audit. The paper also proposes GHVC, which adapts a VLM to generate a coarse-to-fine category path and restricts each inference step to valid children of the predicted parent. Experiments compare specialized video encoders with Qwen3.5 models, study several classification formulations, and report accuracy, macro-F1, hierarchical metrics, and long-tail behavior.

Critical strengths
  1. The dataset is a substantial and potentially useful contribution. Its broad event vocabulary, explicit hierarchy, and long-tailed distribution provide a setting that differs meaningfully from conventional action classification. The construction description includes concrete category-boundary rules, examples of human revision, and residual analysis rather than treating model-generated labels as ground truth. Appendix A.6 reports independent review of the same 22,500 videos by three reviewers across all 1,040 categories, including complete inspection of categories with at most 50 videos. Source-grouped splitting and near-duplicate removal in Appendix C.1 are also valuable safeguards.

  2. The experiments distinguish several sources of improvement and provide a useful benchmark result. Table 1 separates frozen probing from downstream adaptation and evaluates several model sizes. Table 2 compares a linear head, name generation, flat semantic tokens, and hierarchical prediction using the same 2B backbone. GHVC improves macro-F1 over the linear head at 2B, 4B, and 9B, with larger gains on rare categories. These results make the benchmark useful even if the method is viewed as a straightforward structured prediction baseline.

Critical weaknesses
  1. The annotation audit lacks the quantitative outcomes needed to assess the remaining label uncertainty and model-selection bias. The audit protocol is detailed, but Appendices A.5–A.7 do not report the observed pre-adjudication error rates, reviewer agreement, numbers of corrected or removed examples, or the audit's overlap with the test set. The paper correctly notes that the 15% per-category sample threshold is not an estimate of dataset-wide error; the missing observed outcomes remain important. Moreover, retention depends on Qwen-generated descriptions, direct Qwen assignment, and candidate-conditioned Gemma verification. Human review mitigates this dependence, but it does not quantify whether the retained benchmark disproportionately favors distinctions recognized by the construction models. This is an evaluation limitation, not evidence that the labels are incorrect or that contamination has occurred.

  2. The additional contribution of hierarchical generation is small and not yet supported by uncertainty estimates or a clear model-selection protocol. On the matched 2B comparison in Table 2, GHVC improves over flat category tokens by only 0.14 percentage points in leaf accuracy and 0.33 points in macro-F1. Both approaches already produce fully valid paths, and GHVC's throughput is slightly lower (34.1 versus 35.1 videos/s in Table 3). Most of the improvement over the linear head therefore comes from changing the classification formulation rather than introducing the hierarchy. No repeated-run uncertainty is reported for this comparison. Appendix C.1 also states that there is no validation split, leaving unclear how checkpoints and reported design choices were selected without using the test set. These concerns limit the methodological claim more than the value of the dataset.

Questions for authors
  1. Can you report the existing audit records: each reviewer's initial error rate, agreement or disagreement statistics, correction/removal counts, and how many audited examples belong to the test set? Please distinguish values before and after adjudication.
  2. Are retention counts available for the major filtering stages, particularly model disagreement and candidate verification? What evidence from the existing human audit helps assess systematic errors shared with the construction models?
  3. How were hyperparameters, checkpoints, and classification variants selected without a validation split? Were these choices fixed before test evaluation? Are repeated runs or paired prediction records available to assess the small GHVC–flat-token difference?
2 comments

We thank the reviewer for the detailed and constructive review. We respond to the weaknesses below and to the questions in the next comment.

 

W1. Annotation reliability and potential model-selection bias

We agree that the original submission did not sufficiently quantify the annotation audit or the potential bias from model-based filtering.

Audit outcomes. The audit covered 22,500 videos across all 1,040 leaf categories, each inspected independently by three reviewers. Before adjudication, the three reviewers flagged 2.20% / 2.27% / 2.18% of the audited videos (738 unique videos; Fleiss' κ = 0.72). Adjudication confirmed 422 errors (1.88% of audited videos), all of which were corrected (367) or removed (55); 6,152 audited videos belong to the final test split. The full reviewer-level table is given in our response to Q1.

Model-based filtering. We agree that retention depends on the construction models' recognition of the required evidence. Stage-wise retention counts are given in our response to Q2. To quantify the effect on evaluation, we conducted two independent human evaluations.

Retained test videos. Three annotators independently reviewed 5,000 test videos using the original videos and finalized event cards, without access to released labels or model predictions. After adjudication, 97.4% of the released labels agreed with the human annotations.

Rejected videos. We also reviewed 500 videos rejected due to disagreement between the two assignment branches and 500 rejected by Gemma, excluding source-video overlap with training data. Without seeing the model predictions, annotators found that 326 and 264 videos, respectively, could be assigned valid categories under the finalized taxonomy.

We evaluated representative models against the independent human annotations in both subsets:

ModelRetained test subset (5,000)Valid rejected subset (590)
InternVideo2-1B67.9%53.4%
Qwen3.5-2B + Flat Category Token73.5%58.6%
Qwen3.5-2B + GHVC73.8%59.0%

The encoder–VLM ranking is unchanged in both independently evaluated subsets.

Manuscript revision. We have added the audit statistics, filtering counts, and independent human evaluations to Appendices A.5–A.7 and C.9, with a brief summary in Section 5.2.

 

W2. Hierarchy-specific gain, uncertainty, and the absent validation split

We agree that GHVC's gain over Flat Category Token is modest (+0.14% leaf accuracy and +0.33% macro-F1 on 2B), that both formulations already produce valid paths, and that GHVC's throughput is slightly lower (34.1 vs. 35.1 videos/s). We also agree that most of the improvement over the linear head comes from the generative classification formulation rather than from the hierarchy, and that this limits the methodological claim. We have revised the abstract and Section 5.3 accordingly: hierarchical prediction is presented as adding a small further gain on top of the flat formulation.

Repeated runs. We repeated the 2B comparison with three random training seeds and extended the comparison to Qwen3.5-4B and 9B.

BackboneFlat Category Token Acc.GHVC Acc.Improvement
Qwen3.5-2B (Table 2 of the paper, single run)73.78%73.92%+0.14%
Qwen3.5-2B (3 seeds)73.75%73.94%+0.19%
Qwen3.5-4B (3 seeds)75.28 ± 0.07%75.54 ± 0.05%+0.26%
Qwen3.5-9B (3 seeds)76.02 ± 0.08%76.36 ± 0.05%+0.34%

GHVC outperforms Flat Category Token at all three scales, by a modest but consistent margin.

Model selection without a validation split. Our training hyperparameters followed a recipe carried over from our earlier projects rather than being tuned on WildEvent-1K. We used a fixed two-epoch schedule and the final checkpoint, with no test-based model selection (details in Q3).

Manuscript revision. We have updated Sections 5.2–5.3 with the additional experiments and analysis, clarified the training and checkpoint selection procedures in Appendices C.1 and C.5, and highlighted the changes.

Q1. Reviewer-level flags, agreement, adjudication, and test overlap

The audit covered 22,500 videos across all 1,040 categories, with three reviewers independently examining each video. The existing records are summarized below.

Audit metricResult
Initial flags and rates (Reviewer 1 / 2 / 3)496 (2.20%) / 510 (2.27%) / 490 (2.18%)
Unique videos flagged by at least one reviewer738 (3.28%)
Pre-adjudication agreement on assignment correctness98.18% unanimous; Fleiss' κ = 0.72
Adjudication-confirmed errors, before correction422 (1.88% of audited videos)
Reassigned / removed after adjudication367 / 55
Audited videos retained in the final test split6,152 of 49,999

Initial flags reflect individual reviewers' judgments before adjudication, whereas confirmed errors are those upheld through joint adjudication, before correction. All adjudication-confirmed errors were reassigned or removed. The confirmed error fraction describes the stratified audit sample, not the remaining error rate after correction or an estimate of the dataset-wide error rate.

Manuscript revision. These audit statistics have been added to Appendix A.6.

 

Q2. Filtering-stage retention and errors shared with construction models

The stage-wise retention counts are:

Filtering stage (unique video IDs)Videos retained
Videos routed to the 12 L1 event families890,761
Both assignment branches produce high-confidence labels613,264
Both branches agree on the leaf label590,826
Gemma confirms the candidate label538,594
Final WildEvent-1K benchmark538,539

The first step (890,761 → 613,264) removes videos without a supported leaf assignment under the finalized event cards; such videos are excluded rather than forced into a category (Appendix A.7). Among the 613,264 videos receiving two high-confidence predictions, 22,438 were excluded because the branches disagreed; Gemma rejected a further 52,232 of the 590,826 agreed assignments.

Evidence from the existing human audit. The audit examined original videos that had passed both assignment branches and Gemma verification. Among the 422 adjudication-confirmed errors, 238 concerned category boundaries or co-occurring events, 145 involved temporal or participant relationships, and 39 lacked sufficient visual evidence. These are the error types that persisted despite model agreement and candidate verification.

Manuscript revision. We have added the stage-wise retention counts and the audit error analysis to Appendices A.5–A.7.

 

Q3. Model selection without a validation split; repeated runs

Training and model selection. Although no separate validation split was released, we did not tune on WildEvent-1K test results: the VLM training recipe was carried over from our earlier projects and fixed at the start of this work. As documented in Appendix C.5, it uses AdamW, cosine learning-rate decay, 3% warm-up, a global batch size of 64, learning rates of 1 × 10⁻⁴ for LoRA and 1 × 10⁻⁵ for full fine-tuning, and a fixed two-epoch schedule, all commonly used default values. Video encoders follow their published model-specific training recipes. We used the final-epoch checkpoint under this fixed schedule. The linear head, Flat Name Generation, Flat Category Token, and GHVC formulations were evaluated as planned comparisons of classification designs, not selected to maximize test accuracy.

To answer the question directly: yes, the hyperparameters, the two-epoch schedule, the final-checkpoint rule, and the set of formulations were all fixed before any test evaluation.

Repeated runs. Table 2 of the paper compares the classification formulations on the same 49,895 test videos. We additionally trained Flat Category Token and GHVC with three random training seeds on 2B, 4B, and 9B (see the table in our response to W2). All three scales show positive mean gains across seeds: +0.19%, +0.26%, and +0.34%.

Manuscript revision. We have clarified the training recipe and the checkpoint rule and the choice of classification formulations in Appendices C.1/C.5, added the multi-seed results to Appendix C.8, and highlighted the revised interpretation in Section 5.3.

Reviewer_mtXWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates video event classification that goes beyond predefined actions and object categories. The authors propose EventTree, a human-machine collaborative system, which constructs WildEvent-1K from VidGen-1M through model description, category discovery, and human annotation. The dataset comprises 538,539 videos and a three-level classification tree consisting of 12 top-level categories, 129 intermediate categories, and 1,040 leaf categories. The authors further propose GHVC, which uses semantically initialized category tokens for autoregressive prediction of classification paths and ensures valid outputs through parent-child node constraints. Experiments show that, on this dataset, a fine-tuned Qwen3.5 outperforms the specialized video encoders it was compared against, while GHVC further improves performance relative to a baseline linear classification head, particularly enhancing performance for tail categories.

Critical strengths
  1. A large-scale event classification resource with potential value for the research community. WildEvent-1K extends the classification targets to include participant relationships, interaction processes, and contextual semantics, and provides fine-grained hierarchical labels. The event cards in EventTree clearly specify inclusion criteria, exclusion criteria, category boundaries, and priority rules for event co-occurrence; the appendix also presents examples of manually decomposed categories and identified missing categories. These design choices give the dataset value beyond simple automated labeling, which I believe is the primary reason this paper merits consideration for acceptance. The authors also report 22,500 video reviews covering all categories, independently completed by three reviewers, as well as measures for splitting the training and testing sets by original video source.

  2. Meaningful model comparisons and observations on long-tail classification are provided. Table 1 shows that, under the reported training configuration, Qwen3.5-2B with a linear head achieved a leaf accuracy of 72.83%, higher than the 68.46% achieved by InternVideo2-1B; GHVC achieved further accuracy gains of approximately 1.1–1.4 percentage points on the 2B, 4B, and 9B models, respectively. Figure 5 reports F1 gains for tail classes of approximately 6.9–7.9 percentage points. These results suggest that this benchmark may reveal capability differences not fully captured by traditional action classification evaluations and provides a practical baseline for semantically rich, long-tail video classification.

Critical weaknesses
  1. The reliability of dataset annotations and bias in model selection have not yet been fully quantified. Appendix A.6 details the review process but does not report the actual number of errors found, inter-reviewer consistency, the results of the corrected independent review, or the annotation quality of the test set itself. The 15% figure cited in the paper represents the acceptance threshold for category sampling and should not be interpreted as the actual error rate or an upper bound on the overall error rate. Furthermore, both annotation branches rely on Qwen’s visual understanding, with Gemma verifying the candidate labels, and the VLMs primarily evaluated also belong to the Qwen family. This process may improve annotation accuracy while retaining samples that better align with the annotation model’s judgments. The current manual review mitigates this risk, but its impact on model ranking has not yet been quantified.

  2. GHVC improved the accuracy by only 0.14 percentage points and the F1 score by 0.33 percentage points compared to the flat category token; no multiple training fluctuations or paired statistical tests were reported for these differences; corresponding flat generative controls on larger models were also not provided. Appendix C.3 also notes that Flat Name Generation itself generates third-level category names; therefore, Table 2 does not fully disentangle the effects of hierarchical sequences, category tokens, and decoding constraints. The linear head uses video token pooling, and differences caused by readout methods have not been ruled out.

Questions for authors
  1. Is the improvement of GHVC over the Flat Category Token stable across different training seeds, and does it still hold true on the 4B/9B datasets? Since Flat Name Generation outputs three-level labels, how much do the category token and parent-child constraints each contribute to the difference between it and GHVC?
  2. Is there a test subset that was independently annotated by humans without revealing the model’s candidate labels? On this subset, do the relative rankings of the dedicated video encoder, Qwen’s flat classification, and GHVC hold? This would directly impact my assessment of the model family comparison.
  3. What criteria are used to select the parameters for the learning rate, number of training iterations, checkpoints, and long-tail logit adjustment? Are these parameters selected without using the test set at all?
2 comments

We thank the reviewer for the careful and constructive review. We respond to the weaknesses below and to the questions in the next comment.

 

W1: Annotation reliability and model-selection bias

We agree that the audit outcomes and the potential model-selection bias were not sufficiently quantified.

Audit outcomes. The original audit covered 22,500 videos across all 1,040 leaf categories, each examined by three independent human reviewers. Before adjudication, the three reviewers flagged 496 / 510 / 490 videos, 738 unique videos in total (98.18% unanimous agreement; Fleiss’ κ = 0.72). Adjudication confirmed 422 labeling errors, of which 367 were reassigned and 55 removed. A total of 6,152 audited videos belong to the released test set. The 15% rule is a category-level acceptance threshold, neither the observed error rate nor an upper bound on the dataset-wide error.

Test-set quality and model ranking. We had 5,000 retained test videos independently annotated by humans without revealing their existing labels or model predictions; after adjudication, 97.4% of the released labels agreed with the human annotations. On this subset, InternVideo2-1B, Flat Category Token, and GHVC (both on Qwen3.5-2B) achieved leaf accuracies of 67.9%, 73.5%, and 73.8%, respectively (see Q2).

Videos removed by model-based filtering. To quantify the effect of the filtering on the model ranking, we also examined rejected videos. Without seeing model predictions, annotators reviewed 500 videos rejected due to disagreement between the two assignment branches and 500 rejected by Gemma, excluding source-video overlap with training data; 326 and 264 of them, respectively, could be assigned valid categories under the finalized taxonomy. On these 590 videos, InternVideo2-1B, Flat Category Token, and GHVC achieved 53.4%, 58.6%, and 59.0%, so the encoder–VLM ranking is unchanged.

Beyond the Qwen family. Because the evaluated Qwen3.5 models share a family with the Qwen3-VL model used in construction, we evaluated Molmo2-O-7B, whose Olmo-based language backbone was not used in the annotation pipeline. Its linear head reaches 72.04% leaf accuracy (compared with 68.46% for InternVideo2-1B in Table 1), Flat Category Token 73.57%, and GHVC 73.86%.

These results support the model ranking under independent human annotations, on the sampled rejected videos, and outside the Qwen family.

Manuscript revision. We have expanded Appendices A.5–A.6 with the observed audit outcomes, added the evaluations on independently annotated test videos, on rejected videos, and on another VLM family to Appendix C.9, and highlighted the changes in the manuscript.

 

W2: Small GHVC increment; missing robustness and disentanglement

We agree that GHVC's gain over Flat Category Token on 2B is modest (+0.14% leaf accuracy and +0.33% macro-F1 in Table 2). We provide results with multiple training seeds, the flat generative controls on larger models, and a readout comparison below; the disentanglement question is answered in Q1.

Training seeds and larger models. We repeated the 2B comparison with three random training seeds and added the Flat Category Token control on Qwen3.5-4B and 9B. On 2B, GHVC averaged 73.94% leaf accuracy, compared with 73.75% for Flat Category Token (+0.19%). On 4B, GHVC achieved 75.54 ± 0.05%, compared with 75.28 ± 0.07% (+0.26%). On 9B, GHVC achieved 76.36 ± 0.05%, compared with 76.02 ± 0.08% (+0.34%).

Readout. We agree that the linear head with video-token pooling leaves the effect of the readout open. Under matched training and data, replacing video-token pooling with a learned classification token as the readout (following Pang et al., 2026) gives 73.37% leaf accuracy, compared with 72.83% for the linear head with video-token pooling. This remains below Flat Category Token (73.78%) and GHVC (73.92%), so the readout alone does not account for the advantage of the generative formulations.

Manuscript revision. We have added these results to Section 5.3 and Appendix C.8, with the changes highlighted.

Q1. Seed stability, 4B/9B results, and component contributions

Stability across seeds and model scales. With three random training seeds, GHVC averaged 73.94% vs. 73.75% for Flat Category Token on 2B (+0.19%), with per-seed gains ranging from +0.10% to +0.25%. On 4B and 9B, GHVC achieved 75.54 ± 0.05% vs. 75.28 ± 0.07% (+0.26%) and 76.36 ± 0.05% vs. 76.02 ± 0.08% (+0.34%). The improvement is therefore consistent across training seeds and model scales.

Component contributions. We agree that Table 2 does not isolate every factor. To separate them, we additionally evaluated the 2B GHVC checkpoint with the parent–child constraint disabled at inference:

Formulation (Qwen3.5-2B)OutputCategory tokensConstraint at inferenceLeaf acc.
Flat Name GenerationThree category names as free textNoNone73.74%
GHVC without the parent–child constraint (new)Three category tokensYesNone73.76%
Flat Category TokenLeaf tokenYesLegal leaf-token set73.78%
GHVCThree category tokensYesValid children of the predicted parent73.92%

As the reviewer points out, Flat Name Generation already outputs all three levels (Appendix C.3). The “Hier. ✗” entry in Table 2 refers to the absence of parent-conditioned decoding, not to whether all three levels are generated; we have revised the table and its caption to make this explicit.

Category tokens. They are used for efficiency: they replace variable-length names with fixed-length targets, raising throughput from 21.3 to 35.1 videos/s (Table 3) with accuracy comparable to name generation.

Decoding constraints. Enabling the parent–child constraint at inference raises leaf accuracy by 0.16% (73.76% → 73.92%).

Hierarchical sequence. The coarse-to-fine sequence is what makes the parent–child constraint applicable, since a single leaf token has no predicted parent to condition on. Hierarchical prediction, i.e., the sequence together with the constraint, improves over Flat Category Token at all three scales; on 2B, the ablation above shows that this gain is realized through the constraint.

In answer to the reviewer's specific question: the accuracy difference between Flat Name Generation and GHVC on 2B is mainly attributable to the parent–child constraint (+0.16%), while category tokens provide the efficiency gain.

 

Q2. Independent human annotation and model ranking

Yes. Three annotators independently assigned leaf labels to a randomly sampled 5,000-video subset of the test split, using the original videos and complete event cards, without seeing the released labels or any model predictions. Disagreements were resolved through human adjudication, and 97.4% of the released labels agreed with the adjudicated annotations. This experiment is separate from the original 22,500-video audit of proposed labels.

We reevaluated the following models on the same 5,000 videos against the adjudicated human labels:

ModelLeaf accuracy
InternVideo2-1B (full fine-tuning)67.9%
Qwen3.5-2B + linear head72.2%
Qwen3.5-2B + Flat Category Token73.5%
Qwen3.5-2B + GHVC73.8%

The relative rankings under the adjudicated human labels are the same as on the full test set.

Manuscript revision. We have added the sampling and annotation protocol, adjudication results, and model evaluation to Appendix C.9 and summarized the findings in Section 5.2, with the changes highlighted.

 

Q3: Training hyperparameters, checkpoints, logit adjustment, and test-set independence

Learning rate and number of training iterations. We did not tune hyperparameters for WildEvent-1K. The VLM training recipe was carried over from our earlier projects and fixed at the start of this work. As detailed in Appendix C.5, we use AdamW with cosine decay, a 3% warm-up ratio, a global batch size of 64, learning rates of 1×10⁻⁴ for LoRA and 1×10⁻⁵ for full fine-tuning, and a fixed two-epoch schedule; these are commonly used default values rather than benchmark-specific choices. The number of training iterations follows from the dataset size and batch configuration. Video encoders follow their respective published training recipes.

Checkpoints. We used the final checkpoint after the fixed training schedule, without early stopping or checkpoint selection.

Logit adjustment. It is applied only to the linear-head baseline in the long-tail analysis of Section 5.3. It uses class-frequency priors from the training set with the standard setting τ = 1 (Menon et al., 2021); no adjustment parameter was tuned. The wording “tuned correction” in Section 5.3 was imprecise and has been corrected.

No setting was selected using the test set; all of the above were fixed before any test evaluation.

Manuscript revision. We have clarified these settings and selection criteria in Appendices C.1 and C.5, with the changes highlighted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.