- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Questions in previous video benchmarks are typically annotated over different video clip and different objects. The authors propose AnchorGroup, which organizes questions from 598 videos into 625 QA group, each containing perception, reasoning, and summarization questions targeting the same object, with reasoning and summarization additionally sharing evidence segments. To build this dataset, the authors propose A3Loop, which constructs questions through video-grounded generation, verification, and revision, followed by additional filtering, distractor construction, and human review. The paper evaluates 35 models and reports per-task accuracy, the rate of answering all three questions correctly, and correlations conditioned on perception correctness.
Critical strengths
- S1. Offers a new and interesting evaluation perspective for video. Questions in prior benchmarks are usually annotated over different clips and objects. This paper is the first to propose an evaluation format AnchorGroup based on question groups targeting the same object, enabling a fairer assessment of models' perception, reasoning, and summarization, and providing a new angle for understanding them.
- S2. The main experiments are large in scale and the results are solid. The main experiments in §5 and Table 2 cover multiple closed-source and open-source models spanning a wide range of architectures and sizes, strengthening the evidence behind the paper's main conclusions.
- S3. The paper is well presented. The writing is clear and easy to follow, and the figures effectively help readers grasp the core contributions and methods.
Critical weaknesses
- W1. The value of the new evaluation perspective remains underexploited. Grouping perception, reasoning, and summarization tasks for the same video and object is an interesting design, but the paper does not adequately demonstrate what new model weaknesses or valuable insights this evaluation reveals compared to conventional ones. Whether it can produce such insights might determine the value and necessity of this evaluation design.
- W2. The rationale for the generation order in A3Loop lacks experimental validation. The authors generate Summarization first, then Reasoning, and finally Perception, and offer a qualitative explanation, but provide no simple preliminery experiments or ablations to substantite the value of this annotation recipe. The motivation for this design choice therefore remains unsupported by evidence.
- W3. The current multiple-choice construction may not be well suited to evaluating summarization ability. The questions are mostly multiple-choice; A.5 explicitly keeps the event structure and temporal order identical across the four options, changing only two binary factual attributes. In Fig. 7, all four options are already fairly complete summaries of the events, differing only in the prize type and bubble color. Intuitively, the main difference among the options is whether the model can perceive these details, so it is questionable whether the task measures summarization or perception.
Questions for authors
- Q1. Why are perception scores the lowest? The main results show that models score lowest on perception, which contradicts intuition, since reasoning and summary would seem to require integrating more information. Is this caused by the annotation order in A3Loop, or by some other factor?
- Q2. What is the main value of the new evaluation perspective? Prior benchmarks are indeed built over different clips and objects, and the authors do offer a new perspective. But how exactly can this framework enable deeper analysis of model capabilities? Demonstrating the advantages and value of this design still requires further study.
7 comments
Thanks and response to Reviewer hLys (2/5)
Response to W1 & Q2 (continued): Two examples
Analyzing tasks. Task scores give the accuracy of each Reasoning type, while the conditional gap shows what its errors depend on. As shown in Table 2, Causal Implication and Intent Inference have a much larger than , so their errors are tied mainly to event understanding. Training that improves event understanding may help them most. Emotion Inference shows neither gap. Its errors lie in the inference itself, a weakness that task scores hide, so it may need more targeted inference data.
Analyzing model updates. Task scores show how much an update raises each accuracy, while the conditional gap shows where the gain lands. A correct Perception answer shows that the model recognizes the object, and a correct Summarization answer shows that it understands the object's events, so we regard an object as understood () when both are correct. As shown in Table 3, MiMo-VL-7B keeps its Reasoning accuracy after RL, but its accuracy rises sharply on understood objects and drops slightly on the others. Qwen3.6-35B shows the same shift over Qwen3.5-35B. Both updates make the models reason best on understood objects, which points to better perception as the next step of iteration. From Gemini 3.6 Flash to 3.7 Flash, Perception and Summarization rise more than Reasoning, while both conditional gaps stay unchanged. Part of the Reasoning gain therefore comes from stronger Perception and Summarization, and the conditional gaps separate this indirect gain from changes in reasoning itself.
In summary, the object-anchored design turns separate task scores into diagnoses of what limits each skill and what each update changes.
Table 2. Reasoning accuracy by Reasoning type and by the Perception and Summarization answers about the same object (35 models, %). Notation follows the first part of this response. Brackets give 95% bootstrap CIs over videos.
| Reasoning type | Groups | ||||||
|---|---|---|---|---|---|---|---|
| Causal Implication | 46 | 47.67 | 50.54 | 62.65 | 76.41 | +7.98 [−4.14, 20.53] | +18.32 [4.47, 32.34] |
| Intent Inference | 193 | 58.11 | 62.74 | 66.58 | 68.49 | +2.42 [−2.97, 8.10] | +6.70 [1.52, 12.31] |
| Emotion Inference | 95 | 52.09 | 54.83 | 52.87 | 54.09 | +1.69 [−6.28, 10.27] | −0.13 [−8.53, 8.07] |
| All types | 625 | 53.14 | 58.20 | 59.45 | 64.86 | +5.44 [2.36, 8.55] | +6.64 [3.33, 9.91] |
Table 3. Three version pairs on the same 625 groups (%). Each other cell lists the older value first. marks understood objects. , , and match the main table.
| Version pair | |||||||
|---|---|---|---|---|---|---|---|
| MiMo-VL-7B SFT → RL | 32.96 → 34.40 | 42.88 → 42.40 | 38.72 → 42.88 | +4.83 → +7.69 | +4.20 → +11.35 | 42.53 → 56.57 | 42.94 → 39.73 |
| Qwen3.5-35B → Qwen3.6-35B | 54.72 → 53.60 | 57.12 → 62.08 | 58.24 → 59.04 | +1.07 → +3.24 | +3.34 → +5.90 | 56.81 → 67.49 | 57.28 → 59.48 |
| Gemini 3.6 Flash → 3.7 Flash | 67.20 → 73.28 | 75.04 → 78.56 | 77.44 → 84.32 | +10.77 → +10.78 | +12.64 → +12.09 | 81.35 → 82.44 | 68.12 → 71.98 |
Thanks and response to Reviewer hLys (3/5)
Response to W2: Generation order of A³Loop
We thank the reviewer for the suggestion. Section 4.2 explains the backward order qualitatively. Writing Summarization last would force it to fit around the evidence already chosen for Reasoning, so A³Loop writes Summarization first and lets Reasoning pick one of its events. To support this rationale with evidence, we add a preliminary experiment to Section 5.4.
We sample 100 inputs with a fixed seed and run two simplified A³Loops on them. The backward loop writes Summarization, Reasoning and Perception in this order, as in A³Loop, and the forward loop uses the opposite order. Both loops use the same model (Qwen3.8-Flash-Next), generation prompt, revision rules and checks, so only the order differs. For reference, we also run a single pass that writes all three questions in one call without revision.
As shown in Table 4, the backward loop passes more inputs and needs fewer calls per passed group. It also keeps gaining passes in later rounds, while the forward loop stops gaining after round 5. Only 3 inputs pass under the single pass, so both orders benefit from revision, and the backward order adds a further gain on top of it.
These results fit the rationale in Section 4.2 and support the backward order used in A³Loop.
Table 4. Generation order on the same 100 inputs. The single pass writes all three questions in one call without revision. counts revision rounds (: no revision). No input uses more than 9 of the 12 allowed rounds.
| Variant | Passed | Calls per input | Calls per passed group | Passed within rounds, – |
|---|---|---|---|---|
| Single pass | 3/100 | 10.8 | 360.7 | – |
| Loop, forward () | 12/100 | 34.7 | 289.5 | 5, 6, 7, 9, 9, 12, 12, 12, 12, 12 |
| Loop, backward () | 16/100 | 37.2 | 232.4 | 3, 5, 8, 10, 12, 13, 14, 14, 15, 16 |
Thanks and response to Reviewer hLys (1/5)
Thank you for your valuable and insightful comments. We deeply appreciate your input and have made every effort to address your concerns, as detailed below.
Response to W1 & Q2: Value of AnchorGroup
We thank the reviewer for the question. The reviewer asks how this framework enables deeper analysis of model capabilities. Like conventional benchmarks, the paper reports task scores in Section 5.2, which show how strong each model is at each task. AnchorGroup goes one step further because of its shared-object design. This design lets us compute the conditional gap defined in Section 3.3 and we use this gap for further analysis.
To confirm that this gap relies on the shared object, we run a simple control and add it to Section 5.4. As shown in Table 1, when we replace the Perception question with one about another object in the same video, the gap disappears and even turns negative. The gap therefore comes from the shared object, so benchmarks whose questions concern different objects cannot capture it. Building on the conditional gap, AnchorGroup can show what limits each skill and what a model update actually changes. These insights go beyond task scores and can guide training and model iteration. We give two examples below and add them to Section 5.3.
Notation for Tables 1–3. P, R, and S denote Perception, Reasoning, and Summarization. P+ and P− denote groups with correct and incorrect Perception answers, respectively. S+ and S− are defined in the same way. ΔP→R is the difference between R∣P+ and R∣P−, reported in percentage points. The other Δ are defined in the same way.
Table 1. Same-video, different-object control (19 open-source models, 142 pairs matched in difficulty, %). In the second row, P+ and P− use the other object's Perception question to predict the same Reasoning and Summarization questions.