acceptodds
Under review as a conference paper at ICLR 2027

ActionLens: Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models

Abstract

Video-capable vision-language models score above 80% on popular benchmarks, yet they systematically fail at something more fundamental: spatial-temporal binding, i.e., associating the right action with the right person at the right moment. We introduce ActionLens, a diagnostic benchmark of 6,701 multiple-choice video questions that decomposes this capability into five orthogonal diagnostics: transition detection, actor-specific identification, concurrent action binding, directed interaction reasoning, and gaze detection. Every ground-truth answer is derived deterministically from AVAv2.2's 1.58 million per-second, per-person annotations, no language model generated our labels. Fourteen rounds of human quality engineering raised answer clarity from 53% to above 90% human accuracy. Evaluating 20 VLMs reveals a sobering gap: the best open-weight model achieves 68.8% weighted average accuracy against 91.0% for humans, with gaze detection collapsing near chance despite 98.6% human accuracy. A coordinate-variant ablation, replacing visual bounding-box overlays with text coordinates, exposes up to a 19.2-point grounding deficit concentrated exclusively on diagnostics that require person-identity binding. A binding-trap analysis shows 13 of 16 models systematically select the wrong actor's action on D2. ActionLens is not another leaderboard; it is a probe: each diagnostic score is an independent measurement of a capability a model either has or does not. We release all data, code, and evaluation scripts at https://anonymous.4open.science/r/lmms-eval-2276, with ActionLens integrated into lmms-eval for one-command reproduction across any supported VLM.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Video-language models, Vision-laguage models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cBZKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ActionLens, a diagnostic benchmark for whether video VLMs can bind the right action to the right person at the right moment. It contains 6,701 multiple-choice questions generated deterministically from AVA v2.2 annotations and split into five diagnostics: transition sensitivity, actor disambiguation, concurrent binding, directed interaction reasoning, and gaze detection. Across 18 open models and two closed models, the best model reaches 68.8% on the full set and 65.9% on a 1,194-item subset where pooled humans reach 91.0%. Gaze detection stays near chance, and most models pick the other person's action more often than chance when they are wrong.

Critical strengths
  1. The binding-trap design is a good idea, and it produces the paper's most useful finding. Because the other person's action always appears as a distractor in actor disambiguation, errors become interpretable. 13 of 16 models choose that distractor more often than the 33.3% random rate, up to 53.4% (Table 9). This is a failure that ordinary video QA accuracy hides.
  2. Ground truth is derived from annotations rather than from an LLM. Answers are a direct function of AVA's per-second, per-person labels, answer positions are balanced to within 5.1% of uniform (Fig. 2b), and distractors are filtered for physical compatibility.
  3. The statistical analysis is careful. Paired bootstrap over the 64 source videos, explicit intervals for model gaps (e.g., Qwen3.5-4B vs. InternVL3.5-38B tied at [−1.74, 1.59]), equal-video weighting, a box-color swap, and a reference-interface control that separates numeric coordinates, relational descriptions and tracked boxes.
Critical weaknesses
  1. The diagnostics do not cleanly measure binding, which is the paper's central claim. The paper says each diagnostic isolates "exactly one spatial-temporal binding operation" (L159–160) and that the benchmark "cannot be passed by exploiting visual scene biases or other shortcuts" (L79), but its own results do not support this. Concurrent binding is solvable from a single frame (71.4% for InternVL3.5-8B, §4.3), so it behaves more like multi-label recognition. Gaze is a perception task, and the paper itself lists "spatial-binding diagnostics and gaze detection" (L100) and "multi-actor binding, and gaze inference" (L484) as separate things. Transition sensitivity is partly answerable from text alone, and the paper admits it is "less clean" (§4.4). In practice, only actor disambiguation and interaction reasoning clearly test binding.
  2. The data, human reference and evaluation settings are not solid enough to support the headline numbers. The "6,701 video clips" (L79) are 5-second segments from only 64 Hollywood films (L273), and the paper neither says which AVA split is used nor discusses whether evaluated models may have seen AVA in training. The 25-point human gap (L81–82) rests on four raters with one response per item and no agreement statistic (L497, App. Q), on items filtered over 14 rounds to be "solvable by humans" (L255–257), so the 91.0% is partly a product of that filtering. The evaluation settings are also underspecified. For open models §4 gives only the frame count and prompt (L288–289), not whether thinking modes were on or how answers were parsed, and for GPT-5.2 and Gemini 3 Flash no settings are given at all. Some results look like setup failures rather than capability limits (Gemini 3 Flash 27.6% on interaction reasoning vs. 26.7% random; Gemma-3-27B 33.5% overall), which weakens the claims about scale and about closed versus open models (L299–301, L314).
  3. The contribution is limited for a benchmark paper. The questions are a re-packaging of existing AVA labels into multiple-choice form, and the main findings are largely observational: models are worse than humans, gaze is hard, and the target-specification format matters. Binding failures in VLMs are already well documented in images, and the paper does not go further to explain why models fail or test whether any intervention, such as cropping or tracking the target person, helps. Part of what is listed as contribution is process and engineering rather than scientific results: the 14-round quality engineering, presented as a "methodological contribution" is mostly a log of pipeline fixes.
Questions for authors
  1. Could you release per-sample outputs for all models together with the exact model arguments used (frame count, Qwen3.5 thinking on or off, max output tokens), and the call settings for GPT-5.2 and Gemini 3 Flash (frames or video upload, reasoning settings, output length)? What are the answer-parsing failure rates, especially for Gemini 3 Flash on interaction reasoning and for Gemma-3-27B?
  2. What exactly is the "blank context" in the text-only control, and why does Gemma-4-31B do better on gaze without video? Could you report single-frame and shuffled-frame baselines for all five diagnostics, and the answer distribution on gaze?
  3. How does ActionLens relate to region-referring video benchmarks (e.g., VideoRefer-Bench) and recent fine-grained motion and temporal benchmarks (e.g., TemporalBench, MotionBench, Vinoground, TOMATO, FAVOR-Bench)? Table 1 claims ActionLens is "the only" benchmark with per-person grounding, but it only compares against general video QA benchmarks. Separately, have you tried simple interventions, such as cropping the target person or providing track coordinates, to test whether the diagnosed failures are actionable?
1 comment

Thank you for the detailed response and additional experiments, particularly fixing the parsing bug and adding the three-rater agreement study.

While these updates address several evaluation and reliability concerns, my reservations about the overall contribution remain. With the overstated binding claims stepped back, ActionLens still reads mainly as heterogeneous MCQ templates repackaged from existing AVA labels, with largely observational findings. I therefore maintain my score.

Reviewer_tURXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper constructs a new benchmark from existing AVA annotations. It introduces five diagnostics for measure video understanding.

D1: The model must identify action transition, and eval samples are created by looking at transitions in annotated actions and picking paris with certain criteria (e.g. actions don't overlap, other actors do not do same action etc..) D2: The model must identify what the actor in blue box is doing when screen has two boxes. Annotated action is present along with 3 other actions including action of red box actor. D3: The model must identify two annotated actions of an actor. Distractor options include pairs of actions that are feasible. D4: Model must identify correctly how two people are interacting. Other options include switched roles or actions. D5: Model must identify who is looking at whom.

Across these dimensions, authors identify substantial gap in human vs model understanding. A secondary interesting finding of the paper is that while annotated boxes are the best way to point a model to an actor/part of the video, different model families behave differently to other approaches (coordinates provided, relational language).

Critical strengths
  • The benchmark construction approach is general enough that it can be applied to build other more sophisticated benchmarks from simpler annotations. This is a valuable contribution of the paper
  • The human vs model understanding gaps are real (despite weaknesses below) and highlight specific dimensions the models need to get better in terms of video understanding. Providing the training teams with direct feedback.
  • The secondary findings around how different model families behave on different means of pointing at video/actor is valuable for teams building agentic vision pipelines especially in situations where bounding boxes are not necessarily available
Critical weaknesses
  • The names of six axes are more general while the tasks are very specific. While the gap is real, it is not clear that getting better at these tasks for models will generalize to the model becoming better at video understanding as a whole. No rationale or proof is offered.
  • It is not clear to me that distractor options are truly wrong, and in some places the claims seem stronger than what the details back up e.g. in D3 it says "(a) neither distractor action is a true action of the highlighted person" whereas it looks like simply absence of annotation is assumed to qualify. That is not proof of (a) however. For example a "breathing" action will rarely be annotated, but a distractor action containing "walking and breathing" is a perfectly valid/correct option.
  • Construciton of distractors could end up being systematically adverse to models in synthetic ways that it is not for humand after many iterations of construction criteria. A changelog of iterations would be helpful.
  • No study on inter annotator agreement. A small sample study would be valuable. Again, mitigated by 91% rate of humans agreeing with ground truth.
Questions for authors
  • How humans were recruited, their familiarity with AVA or other datasets
  • What knowledge of the task, ground truth, ava annotations, they had, if any at the time of annotations.
  • What are common reasons for humans to differ from ground truth on the tasks where they do?
3 comments

Thank you for the thorough response and for adding the inter-annotator agreement study, the 14-round engineering changelog, and the rater protocol details. These additions resolve my earlier soundness concerns.

However the framing concern remains unresolved and the paper still uses fairly generalized terms for diagnostics, and the overall framing remains broad. The paper should frame ActionLens as a diagnostic probe for specific person-action binding errors, rather than a broad "spatial-temporal binding" benchmark. My score remains unchanged.

Thank you for revisiting our response and for acknowledging that the additional evidence resolves your soundness concerns. We appreciate your clarification of the remaining framing concern. We see the value in this distinction and have revised the manuscript accordingly:

  • Title and Abstract: Reframed ActionLens around person-centered video understanding failures, rather than broadly diagnosing spatial-temporal binding.
  • Introduction and Contributions: Sharpened the central contribution as identifying concrete, interpretable failure modes that aggregate VideoQA scores obscure.
  • Section 3.1 (Design Philosophy): Explicitly distinguished the five operational capabilities.

These revisions address the distinction you highlighted: ActionLens diagnoses specific, interpretable failures in person-centered video understanding without claiming that performance on these diagnostics establishes general video understanding ability. The narrower framing preserves our central empirical findings, including systematic wrong-actor confusion and sensitivity to target-reference interfaces, while making their scope explicit.

We believe this directly addresses your remaining concern while preserving the contributions you recognized in the original review. With the soundness concerns now resolved and the framing revised to address your remaining concern, we would appreciate your reconsideration of the overall assessment.

We welcome any further discussion and thank you again for your constructive feedback.

Thanks for addressing the remaining concerns. I am raising my score to 3

Reviewer_3YXBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ActionLens introduces 6,701 multiple-choice questions derived from AVA annotations to evaluate five aspects of person-centered video understanding: action transitions, actor–action association, concurrent actions, interaction roles, and watching behavior. The paper evaluates 20 VLMs and provides controls for reference interfaces, frame budgets, textual priors, prompting, and box colors, together with human-reference results and video-clustered uncertainty estimates.

Critical strengths
  • The paper addresses an important problem: recognizing actions at the scene level does not ensure correctly associating them with particular people.
  • The evaluation includes useful controls beyond aggregate accuracy. In particular, the relational-reference condition improves the interpretation of coordinate-based results, while video-clustered intervals appropriately account for the concentration of questions in 64 source videos.
  • The documented construction revisions and independent pooled human reference provide valuable evidence of answerability. The appendix also candidly discusses several limitations and negative findings.
Critical weaknesses
  • Some diagnostics do not yet establish the necessity of their intended operation. On D1, InternVL3.5-38B achieves 69.6% without video versus 68.0% with video, indicating substantial action-sequence priors. On D5, Gemma-4-31B similarly achieves 46.0% without video versus 42.0% with video. These results warrant stronger shortcut controls than answer-position balance. For D3, the stated exclusion of both true actions from distractor pairs appears to permit selecting the correct pair after recognizing only one component; the actual frequency of such items should be quantified.
  • The validity of D4 as directed interaction reasoning needs stronger support. Per-person talk/listen labels do not independently establish a directed relation between the two highlighted people. Adding distinct visible co-actions does not resolve this annotation gap or independently validate communication roles in muted video. The high human accuracy is encouraging, but targeted role and relation adjudication is needed.
Questions for authors
  • Can D1 be evaluated using paired original and reversed videos with corresponding answer changes? Can option permutations and options-only controls clarify the D5 text-only result?
  • What fraction of D3 questions requires recognizing both actions? How do results change with distractors that each preserve one correct component?
  • When will the benchmark be released?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions, new experiments, and discussion status

Guided by the reviews from 3YXB, tURX, and cBZK, we narrowed the manuscript's title and framing to ActionLens: Diagnosing Person-Centered Video Understanding Failures in Vision-Language Models, conducted five new shortcut and intervention experiments, performed a shared-item inter-annotator study, and released per-sample evaluation logs and a verification script for all 20 evaluated models. Following these revisions, Reviewers 3YXB and tURX both updated their ratings from Weak Rejection to Weak Acceptance (2 → 3). We have also followed through on cBZK's experimental, auditing, and reproducibility requests in the revised manuscript, discussion, and anonymous repository.

Main takeaways

  • Models often recognize what is happening, yet attach it to the wrong person. By placing a co-occurring actor's true action among D2's distractors, ActionLens shows that 13 of 16 models select another person's action significantly above the 33.3% uniform-error rate, accounting for up to 53.4% of errors (Table 9), a failure mode hidden by aggregate VideoQA accuracy.
  • In-scene visual grounding works, whereas naive target cropping backfires. Across 3,611 D2/D4 probes, relational text recovers 5.55 to 13.25 points over static coordinates on D2, and full-scene tracked boxes add another 1.15 to 6.50 points across all five tested models (Sec. 4.2). Conversely, the 2,000-item target-cropping experiment suggested by cBZK lowers Qwen3.5-27B D2 accuracy from 67.70% to 64.45% (95% CI for the difference: [−5.37, −1.22] points). Together, these results demonstrate how ActionLens can assess both the effectiveness and limitations of practical interventions.
  • Raw annotations are not sufficient to build a reliable diagnostic benchmark. Direct MCQ conversion of AVA v2.2 validation labels reached only 53% human accuracy in our 281-item pilot. Reaching 91.0% pooled human accuracy required 14 human-audit rounds and 22 systematic filters (Appendix C). On the same 1,194 questions, the leading open-weight model reaches only 65.9%. We further establish annotation reliability through a new 250-item three-rater study (Fleiss' κ = 0.836, 98.4% majority-vote accuracy, Table 18) and demonstrate the construction pipeline's portability through a 500-candidate MultiSports pilot (Appendix M).

Summary of review concerns and additional evidence

Review areaNew evidence and manuscript updates
Scope and framing (tURX, cBZK, 3YXB)Retitled the paper and narrowed Secs. 1 and 3.1 to five concrete person-centered probes. Expanded Sec. 2 and Table 1 with all six requested benchmarks.
Temporal and shortcut controls (3YXB, cBZK)Added paired 194-item D1 reversal (27.3–40.7% correct in both directions), full 6,701-item frame shuffling (D1 −7.22, D2 −4.45 points), exhaustive 2,000-item D3 distractor audit, four cyclic D5 option rotations, and options-only controls (24.0–25.5%, near chance).
D4 interaction boundary (3YXB)Renamed D4 to Interaction Reasoning. Established high agreement on composite choices (κ = 0.893, 50/50 majority-vote accuracy) while explicitly distinguishing composite assignment from independent validation of communication roles.
Scoring audit, IAA and reproducibility (cBZK, tURX)Added the 250-item three-rater study and annotation protocol. Audited all 20 models' outputs, corrected Gemini 3 Flash's Markdown parsing artifact (D4: 27.6% → 59.2%; Avg.: 44.6% → 61.2%), removed the misplaced Gemma-4-31B text-only result, and released all per-sample logs and a verification script.
Actionability and novelty (cBZK)Conducted the suggested target-cropping intervention, expanded the benchmark comparison, and highlighted the reusable construction methodology and MultiSports portability study.

Beyond addressing the concerns raised in the reviews, we pursued the suggested intervention and released the complete evaluation logs and verification script in our anonymous repository. While 3YXB and tURX have acknowledged the additional evidence and reconsidered their recommendations, we have not yet received cBZK's assessment of our responses.

We believe ActionLens makes a meaningful contribution by going beyond measuring how often models fail: it makes specific person-centered errors identifiable, measurable, and amenable to targeted interventions.

Thank you to all three reviewers and the Area Chair for the time and care invested in evaluating our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.