- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ActionLens, a diagnostic benchmark for whether video VLMs can bind the right action to the right person at the right moment. It contains 6,701 multiple-choice questions generated deterministically from AVA v2.2 annotations and split into five diagnostics: transition sensitivity, actor disambiguation, concurrent binding, directed interaction reasoning, and gaze detection. Across 18 open models and two closed models, the best model reaches 68.8% on the full set and 65.9% on a 1,194-item subset where pooled humans reach 91.0%. Gaze detection stays near chance, and most models pick the other person's action more often than chance when they are wrong.
Critical strengths
- The binding-trap design is a good idea, and it produces the paper's most useful finding. Because the other person's action always appears as a distractor in actor disambiguation, errors become interpretable. 13 of 16 models choose that distractor more often than the 33.3% random rate, up to 53.4% (Table 9). This is a failure that ordinary video QA accuracy hides.
- Ground truth is derived from annotations rather than from an LLM. Answers are a direct function of AVA's per-second, per-person labels, answer positions are balanced to within 5.1% of uniform (Fig. 2b), and distractors are filtered for physical compatibility.
- The statistical analysis is careful. Paired bootstrap over the 64 source videos, explicit intervals for model gaps (e.g., Qwen3.5-4B vs. InternVL3.5-38B tied at [−1.74, 1.59]), equal-video weighting, a box-color swap, and a reference-interface control that separates numeric coordinates, relational descriptions and tracked boxes.
Critical weaknesses
- The diagnostics do not cleanly measure binding, which is the paper's central claim. The paper says each diagnostic isolates "exactly one spatial-temporal binding operation" (L159–160) and that the benchmark "cannot be passed by exploiting visual scene biases or other shortcuts" (L79), but its own results do not support this. Concurrent binding is solvable from a single frame (71.4% for InternVL3.5-8B, §4.3), so it behaves more like multi-label recognition. Gaze is a perception task, and the paper itself lists "spatial-binding diagnostics and gaze detection" (L100) and "multi-actor binding, and gaze inference" (L484) as separate things. Transition sensitivity is partly answerable from text alone, and the paper admits it is "less clean" (§4.4). In practice, only actor disambiguation and interaction reasoning clearly test binding.
- The data, human reference and evaluation settings are not solid enough to support the headline numbers. The "6,701 video clips" (L79) are 5-second segments from only 64 Hollywood films (L273), and the paper neither says which AVA split is used nor discusses whether evaluated models may have seen AVA in training. The 25-point human gap (L81–82) rests on four raters with one response per item and no agreement statistic (L497, App. Q), on items filtered over 14 rounds to be "solvable by humans" (L255–257), so the 91.0% is partly a product of that filtering. The evaluation settings are also underspecified. For open models §4 gives only the frame count and prompt (L288–289), not whether thinking modes were on or how answers were parsed, and for GPT-5.2 and Gemini 3 Flash no settings are given at all. Some results look like setup failures rather than capability limits (Gemini 3 Flash 27.6% on interaction reasoning vs. 26.7% random; Gemma-3-27B 33.5% overall), which weakens the claims about scale and about closed versus open models (L299–301, L314).
- The contribution is limited for a benchmark paper. The questions are a re-packaging of existing AVA labels into multiple-choice form, and the main findings are largely observational: models are worse than humans, gaze is hard, and the target-specification format matters. Binding failures in VLMs are already well documented in images, and the paper does not go further to explain why models fail or test whether any intervention, such as cropping or tracking the target person, helps. Part of what is listed as contribution is process and engineering rather than scientific results: the 14-round quality engineering, presented as a "methodological contribution" is mostly a log of pipeline fixes.
Questions for authors
- Could you release per-sample outputs for all models together with the exact model arguments used (frame count, Qwen3.5 thinking on or off, max output tokens), and the call settings for GPT-5.2 and Gemini 3 Flash (frames or video upload, reasoning settings, output length)? What are the answer-parsing failure rates, especially for Gemini 3 Flash on interaction reasoning and for Gemma-3-27B?
- What exactly is the "blank context" in the text-only control, and why does Gemma-4-31B do better on gaze without video? Could you report single-frame and shuffled-frame baselines for all five diagnostics, and the answer distribution on gaze?
- How does ActionLens relate to region-referring video benchmarks (e.g., VideoRefer-Bench) and recent fine-grained motion and temporal benchmarks (e.g., TemporalBench, MotionBench, Vinoground, TOMATO, FAVOR-Bench)? Table 1 claims ActionLens is "the only" benchmark with per-person grounding, but it only compares against general video QA benchmarks. Separately, have you tried simple interventions, such as cropping the target person or providing track coordinates, to test whether the diagnosed failures are actionable?
Thank you for the detailed response and additional experiments, particularly fixing the parsing bug and adding the three-rater agreement study.
While these updates address several evaluation and reliability concerns, my reservations about the overall contribution remain. With the overstated binding claims stepped back, ActionLens still reads mainly as heterogeneous MCQ templates repackaged from existing AVA labels, with largely observational findings. I therefore maintain my score.