- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WildEvent-1K organizes diverse videos into a hierarchical label space defined by visible events, using EventTree to combine model-generated descriptions, candidate categories, and human refinement. The benchmark compares specialized video encoders with adapted VLMs and reports a reversal of their relative performance from Kinetics-400. GHVC uses category tokens and parent-constrained generation to predict a valid path through the event taxonomy.
Critical strengths
-
The event taxonomy makes fine-grained semantic distinctions inspectable. The supplementary cards specify observable evidence, exclusions, and precedence when events coexist. For example, tug, fetch, and human-controlled object play are separated by the interaction structure. Together with the human refinement and residual-discovery examples, this gives the benchmark value beyond a collection of class names.
-
Separating frozen representations from adaptation reveals an informative change in model behavior. Table 1 shows that encoders retain an advantage under frozen probing, while adapted VLMs perform better on the event benchmark. I find this distinction useful for studying how pretrained models adapt to different label spaces.
Critical weaknesses
-
Labels were finalized by VLM agreement, so the encoder versus VLM finding has not been checked independently. Model-proposed labels are reasonable for a label space this fine-grained; my concern is the acceptance step. A video is kept only if the GPT branch (reading Qwen3-VL descriptions) and the Qwen3-VL branch agree and Gemma confirms (App. A.5); the rest is discarded. Humans saw 22,500 of 538,539 videos, and in the Figure 2 branch only 27,833 of 47,810 survive. The videos the VLMs could not agree on are the hard cases "in the wild" refers to, and the VLM in Table 1 shares a family with the labeler. I am not asking for relabeling: a human-verified test split with Table 1 rerun on it, plus the discard rate per L1 family, would settle this.
-
The hierarchical part of GHVC adds little. A flat category token reaches 73.78 leaf accuracy against 73.92 for GHVC in one run (Table 2), and logit adjustment on the linear head beats GHVC on tail macro-F1 (65.4 vs 63.6). The paper [1] find the opposite ranking of discriminative and generative MLLM classifiers and are not compared. GHVC reads better as a benchmark baseline.
-
Only Qwen VLMs are tested, and Qwen3-VL is also part of the labeling pipeline. One other VLM family would help.
-
X-CLIP or ViFi-CLIP style classifiers are absent from Table 1. They also use class-name semantics, so I wonder how much of the VLM gain they would recover.
[1] Z. Pang, D. Chatterjee, F. Sener, A. Yao, On Discriminative vs. Generative Classifiers: Rethinking MLLMs for Action Understanding, ICLR 2026
Questions for authors
-
How many of the 22,500 audited videos were flagged and corrected, and what was the agreement among the three reviewers? The per-reviewer rule in App. A.6 accepts up to 3 errors in a sample of 20, which a class with a 20% error rate passes about 40% of the time, so the observed counts would say much more about label quality than the pass rule.
-
How was the test split drawn? Spoken Communication makes up 32.2% of training videos but about 13.5% of test videos (Table 6b), and the Table 6 note refers to "uncapped" training frequency.
-
Which InternVideo2 and X-CLIP checkpoints are used in Figure 6? If they were trained or fine-tuned with Kinetics labels, tighter Kinetics-400 classes are expected and the comparison says less about WildEvent-1K itself.
4 comments
Q1. Audit error counts, adjudication, and reviewer agreement
We agree that the observed audit outcomes are more informative than the acceptance rule. In the original category-stratified audit, all three reviewers independently examined the same 22,500 videos across 1,040 leaf categories.
| Audit metric | Result |
|---|---|
| Initial flags and rates, Reviewers 1 / 2 / 3 | 496 (2.20%) / 510 (2.27%) / 490 (2.18%) |
| Unique videos flagged by at least one reviewer | 738 (3.28%) |
| Pre-adjudication agreement on assignment correctness | 98.18% unanimous; Fleiss’ κ = 0.72 |
| Adjudication-confirmed errors, before correction | 422 (1.88% of audited videos) |
| Reassigned / removed after adjudication | 367 / 55 |
Initial flags indicate suspected errors; confirmed errors are those upheld after joint adjudication. All adjudication-confirmed errors were reassigned or removed before release. The 15% threshold was an acceptance rule for partially sampled categories, not a guarantee of label accuracy. The confirmed error fraction describes the stratified audit sample before correction, not the remaining error rate after correction or a dataset-wide error estimate or bound.
Manuscript revision. These audit outcomes and agreement statistics have been added to Appendix A.6.
Q2. Test-set sampling and “uncapped” training frequency
We agree that the original description of the test-set sampling strategy was insufficiently clear.
Test-set sampling. WildEvent-1K follows a long-tailed class distribution. Strictly proportional sampling would leave many rare categories with very few or even no test examples, whereas a fully class-balanced test set would discard the natural frequency structure and move most videos of small categories out of training. There is no single correct choice here, and long-tailed benchmarks differ on it (for example, ImageNet-LT and iNaturalist 2018 are evaluated on class-balanced test or validation sets). We chose an intermediate design: the number of test videos drawn from each leaf grows sublinearly with the size of the leaf, approximately in proportion to the square root of its number of videos, for a total of about 50K test videos, and all clips from the same source video are kept in the same split to prevent train–test leakage. Frequent categories therefore still contribute more test videos than rare ones, but the imbalance is compressed, and every one of the 1,040 leaves has test examples. This explains why Spoken Communication accounts for 32.2% of training videos but only 13.5% of test videos. Because leaf accuracy is weighted by this test distribution whereas macro-F1 weights all leaves equally, we report both, together with head/medium/tail F1, so that the conclusions do not depend on a single weighting.
Uncapped training frequency. This means that we use all available training videos without truncating or downsampling frequent categories. The head, medium, and tail groups are defined using these complete per-class training counts (≥101, 20–100, and ≤19, respectively). Thus, the rescaling applies only to test-set sampling, while training retains the full long-tailed distribution without class-frequency caps.
Manuscript revision. We have clarified the test-set sampling strategy and the meaning of “uncapped training frequency” in Appendix C.1 and Table 6, with the changes highlighted in the revised manuscript.
Q3. Figure 6 checkpoints and possible Kinetics-supervision bias
We agree that Kinetics-supervised features may favor K400 class separability. The frozen checkpoints used in Figure 6 were:
- InternVideo2-1B:
OpenGVLab/InternVideo2-Stage1-1B-224p-K400(1B_ft_k710_ft_k400_f16.pth), pretrained on K-Mash and subsequently fine-tuned on Kinetics-710 and Kinetics-400. - X-CLIP-B/16:
microsoft/xclip-base-patch16-16-frames(corresponding to thek400_16_16.pthconfiguration), trained with Kinetics-400 supervision.
To address the potential bias from Kinetics supervision, we repeated the same 50-draw, 400-class geometry analysis using frozen DINOv2 features without Kinetics-label fine-tuning. The mean intra-class cosine distance increased from 0.36 on K400 to 0.46 on WildEvent-1K, while nearest-class centroid similarity increased from 0.71 to 0.79.
The same trend therefore persists in a representation without Kinetics-specific label supervision: WildEvent-1K exhibits greater intra-class variation and less separable class boundaries.
Manuscript revision. We have documented the exact checkpoints and their supervision histories in Appendix C.2, added the DINOv2 analysis to Appendix C.8, and highlighted the revised interpretation in Section 5.5.
Thanks for the rebuttal. The authors significantly address my concerns, so I'm raising my rating to 4.
We appreciate your support for our work.
We thank the reviewer for the thoughtful review and suggestions. We respond to the weaknesses below and to the questions in the next comment.
W1. Human evaluation, hard cases, and per-L1 retention
Human-verified subset. Three annotators independently labeled 5,000 retained test videos (all 12 L1 families) from the original videos and finalized event cards, without access to released labels or model candidates. After adjudication, 97.4% of the released labels agreed. This is separate from the 22,500-video audit.
Leaf accuracy under these human labels (full-test accuracy in parentheses; encoders: full FT): Video Swin-L 58.0% (58.33%), VideoMAE-H 59.6% (60.24%), InternVideo2-1B 67.9% (68.46%), Qwen3.5-2B linear head 72.2% (72.83%), Flat Category Token 73.5% (73.78%), GHVC 73.8% (73.92%). The encoder–VLM ranking is unchanged.
Hard cases. Of 500 branch-disagreement and 500 Gemma rejections reviewed by humans without access to model predictions, 326 and 264 were assigned valid labels under the finalized taxonomy. On these 590 videos, InternVideo2-1B, Flat Category Token, and GHVC achieved 53.4%, 58.6%, and 59.0%, preserving the encoder–VLM ranking.
These reductions span taxonomy construction, human refinement, and model-based filtering, not model disagreement or Gemma verification alone. Assignment to an L1 family does not establish that a video satisfies a finalized leaf-event definition: candidates without coherent definitions, distinguishable boundaries, or sufficient supporting examples were revised or removed, and videos not covered by the finalized event cards were not retained merely to increase coverage.
Manual inspection found concentrated recurring content in Spoken Communication, Object Display, and Vehicle Movement, which account for 41.71% of initially routed videos but only 17.02% of leaf categories and retain 80.18%, vs. 46.35% for the other nine. We acknowledge that agreement-based filtering may still exclude valid videos that the construction models find difficult to recognize.
W2. Hierarchical contribution, long-tail trade-off, and Pang et al. [1]
We agree that the hierarchical part of GHVC adds only a small gain over Flat Category Token (+0.14% leaf accuracy and +0.33% macro-F1 on 2B) and that GHVC reads better as a benchmark baseline. The revision presents it as such.
Beyond one run. With three seeds, GHVC improves over Flat Category Token on 2B (73.94% vs. 73.75%, +0.19%), 4B (75.54 ± 0.05% vs. 75.28 ± 0.07%, +0.26%), and 9B (76.36 ± 0.05% vs. 76.02 ± 0.08%, +0.34%), so the small gain is consistent.
Logit adjustment. It raises the linear head's tail macro-F1 to 65.4%, above GHVC's 63.6%, at the cost of head accuracy (−1.4%) and overall leaf accuracy (−0.3%). GHVC improves tail macro-F1 over the unadjusted linear head (56.6% → 63.6%) while raising overall leaf accuracy by 1.09%, without class-frequency priors.
Pang et al. [1]. Their discriminative classifier reads out a learned classification token, unlike our linear head with video-token pooling. Under matched training and data, classifiers in the style of their discriminative and GAD designs reach 73.37% and 73.51%, vs. 73.92% for GHVC, so the generative classifier remains ahead on WildEvent-1K.
W3. Beyond the Qwen family
Molmo2-O-7B, whose Olmo-based language backbone is not used in the annotation pipeline, reaches 72.04% with a linear head, 73.57% with Flat Category Token, and 73.86% with GHVC. The same ranking holds outside the Qwen family. Its linear head remains above InternVideo2-1B (68.46%), so the encoder versus VLM finding does not rely on a VLM from the labeler's family.
W4. X-CLIP and ViFi-CLIP
With the complete L1–L2–leaf category paths and their respective fine-tuning protocols, X-CLIP-B/16 reaches 65.20% leaf accuracy / 62.30% macro-F1 and ViFi-CLIP-B/16 65.81% / 62.91%, vs. 72.83% / 71.22% for the Qwen3.5-2B linear head and 73.92% / 73.19% for GHVC. They remain 8.72% and 8.11% below GHVC in leaf accuracy. Even the linear head, which uses no class-name semantics, outperforms both, so the advantage of adapted VLMs is not explained by class-name semantics alone.
Revision (highlighted). Abstract, contribution list, conclusion; Table 1; Sections 5.2, 5.3; Related Work; Appendices A.5–A.7, C.2, C.8, C.9.