acceptodds
Under review as a conference paper at ICLR 2027

AnchorGroup: Object-Anchored Video Understanding Benchmark with an Agentic Annotation Loop

Abstract

Video Question Answering (VideoQA) has seen advances in both Multimodal Large Language Models (MLLMs) and evaluation benchmarks. Existing benchmarks typically evaluate independent QA pairs and report performance for each task. Questions from different tasks may concern different objects, even within the same video. Separate task scores therefore provide limited insight into whether models succeed across tasks involving the same object. To address this gap, we introduce AnchorGroup, a video understanding benchmark organized into object-anchored QA groups. Each QA group contains perception, reasoning, and summarization questions about the same object. The shared anchor provides a common reference for comparing task performance and examining associations between task outcomes. Existing annotation pipelines primarily focus on individual QA pairs rather than shared-anchor constraints across tasks. We therefore introduce A3Loop, an anchor-guided agentic annotation loop that constructs QA groups with direct access to video clips. The loop generates QA pairs from summarization to reasoning and then perception. During construction, verifier feedback guides generators in revising QA pairs to meet task-specific and group-level requirements. Revised QA pairs are then verified again. Evaluations on AnchorGroup show that perception has the lowest average accuracy among the three tasks. Correct perception answers are generally associated with higher reasoning and summarization accuracy within the same QA groups. We release the dataset and annotation framework to support reproducible evaluation of video understanding across tasks within shared object contexts.

Primary area:
datasets and benchmarks
Keywords:
Video Understanding, Multimodel Large Language Models, Agentic Annotation Loop

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_hLysOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Questions in previous video benchmarks are typically annotated over different video clip and different objects. The authors propose AnchorGroup, which organizes questions from 598 videos into 625 QA group, each containing perception, reasoning, and summarization questions targeting the same object, with reasoning and summarization additionally sharing evidence segments. To build this dataset, the authors propose A3Loop, which constructs questions through video-grounded generation, verification, and revision, followed by additional filtering, distractor construction, and human review. The paper evaluates 35 models and reports per-task accuracy, the rate of answering all three questions correctly, and correlations conditioned on perception correctness.

Critical strengths
  • S1. Offers a new and interesting evaluation perspective for video. Questions in prior benchmarks are usually annotated over different clips and objects. This paper is the first to propose an evaluation format AnchorGroup based on question groups targeting the same object, enabling a fairer assessment of models' perception, reasoning, and summarization, and providing a new angle for understanding them.
  • S2. The main experiments are large in scale and the results are solid. The main experiments in §5 and Table 2 cover multiple closed-source and open-source models spanning a wide range of architectures and sizes, strengthening the evidence behind the paper's main conclusions.
  • S3. The paper is well presented. The writing is clear and easy to follow, and the figures effectively help readers grasp the core contributions and methods.
Critical weaknesses
  • W1. The value of the new evaluation perspective remains underexploited. Grouping perception, reasoning, and summarization tasks for the same video and object is an interesting design, but the paper does not adequately demonstrate what new model weaknesses or valuable insights this evaluation reveals compared to conventional ones. Whether it can produce such insights might determine the value and necessity of this evaluation design.
  • W2. The rationale for the generation order in A3Loop lacks experimental validation. The authors generate Summarization first, then Reasoning, and finally Perception, and offer a qualitative explanation, but provide no simple preliminery experiments or ablations to substantite the value of this annotation recipe. The motivation for this design choice therefore remains unsupported by evidence.
  • W3. The current multiple-choice construction may not be well suited to evaluating summarization ability. The questions are mostly multiple-choice; A.5 explicitly keeps the event structure and temporal order identical across the four options, changing only two binary factual attributes. In Fig. 7, all four options are already fairly complete summaries of the events, differing only in the prize type and bubble color. Intuitively, the main difference among the options is whether the model can perceive these details, so it is questionable whether the task measures summarization or perception.
Questions for authors
  • Q1. Why are perception scores the lowest? The main results show that models score lowest on perception, which contradicts intuition, since reasoning and summary would seem to require integrating more information. Is this caused by the annotation order in A3Loop, or by some other factor?
  • Q2. What is the main value of the new evaluation perspective? Prior benchmarks are indeed built over different clips and objects, and the authors do offer a new perspective. But how exactly can this framework enable deeper analysis of model capabilities? Demonstrating the advantages and value of this design still requires further study.
7 comments
AuthorsOpenReview ↗

Thanks and response to Reviewer hLys (1/5)

Thank you for your valuable and insightful comments. We deeply appreciate your input and have made every effort to address your concerns, as detailed below.

Response to W1 & Q2: Value of AnchorGroup

We thank the reviewer for the question. The reviewer asks how this framework enables deeper analysis of model capabilities. Like conventional benchmarks, the paper reports task scores in Section 5.2, which show how strong each model is at each task. AnchorGroup goes one step further because of its shared-object design. This design lets us compute the conditional gap defined in Section 3.3 and we use this gap for further analysis.

To confirm that this gap relies on the shared object, we run a simple control and add it to Section 5.4. As shown in Table 1, when we replace the Perception question with one about another object in the same video, the gap disappears and even turns negative. The gap therefore comes from the shared object, so benchmarks whose questions concern different objects cannot capture it. Building on the conditional gap, AnchorGroup can show what limits each skill and what a model update actually changes. These insights go beyond task scores and can guide training and model iteration. We give two examples below and add them to Section 5.3.

Notation for Tables 1–3. PP, RR, and SS denote Perception, Reasoning, and Summarization. P+P^+ and P−P^- denote groups with correct and incorrect Perception answers, respectively. S+S^+ and S−S^- are defined in the same way. ΔP→R\Delta_{P\to R} is the difference between R∣P+R\mid P^+ and R∣P−R\mid P^-, reported in percentage points. The other Δ\Delta are defined in the same way.

Table 1. Same-video, different-object control (19 open-source models, 142 pairs matched in difficulty, %). In the second row, P+P^+ and P−P^- use the other object's Perception question to predict the same Reasoning and Summarization questions.

Perception itemPPRRSSR∣P+R\mid P^+R∣P−R\mid P^-ΔP→R\Delta_{P\to R}S∣P+S\mid P^+S∣P−S\mid P^-ΔP→S\Delta_{P\to S}
Same object43.3455.6554.7258.6153.22+5.3957.8852.73+5.15
Different object43.4255.6554.7253.0157.61−4.6053.9555.18−1.23
AuthorsOpenReview ↗

Thanks and response to Reviewer hLys (2/5)

Response to W1 & Q2 (continued): Two examples

Analyzing tasks. Task scores give the accuracy of each Reasoning type, while the conditional gap shows what its errors depend on. As shown in Table 2, Causal Implication and Intent Inference have a much larger ΔS→R\Delta_{S\to R} than ΔP→R\Delta_{P\to R}, so their errors are tied mainly to event understanding. Training that improves event understanding may help them most. Emotion Inference shows neither gap. Its errors lie in the inference itself, a weakness that task scores hide, so it may need more targeted inference data.

Analyzing model updates. Task scores show how much an update raises each accuracy, while the conditional gap shows where the gain lands. A correct Perception answer shows that the model recognizes the object, and a correct Summarization answer shows that it understands the object's events, so we regard an object as understood (P+S+P^+S^+) when both are correct. As shown in Table 3, MiMo-VL-7B keeps its Reasoning accuracy after RL, but its accuracy rises sharply on understood objects and drops slightly on the others. Qwen3.6-35B shows the same shift over Qwen3.5-35B. Both updates make the models reason best on understood objects, which points to better perception as the next step of iteration. From Gemini 3.6 Flash to 3.7 Flash, Perception and Summarization rise more than Reasoning, while both conditional gaps stay unchanged. Part of the Reasoning gain therefore comes from stronger Perception and Summarization, and the conditional gaps separate this indirect gain from changes in reasoning itself.

In summary, the object-anchored design turns separate task scores into diagnoses of what limits each skill and what each update changes.

Table 2. Reasoning accuracy by Reasoning type and by the Perception and Summarization answers about the same object (35 models, %). Notation follows the first part of this response. Brackets give 95% bootstrap CIs over videos.

Reasoning typeGroupsR∣P−S−R\mid P^-S^-R∣P+S−R\mid P^+S^-R∣P−S+R\mid P^-S^+R∣P+S+R\mid P^+S^+ΔP→R\Delta_{P\to R}ΔS→R\Delta_{S\to R}
Causal Implication4647.6750.5462.6576.41+7.98 [−4.14, 20.53]+18.32 [4.47, 32.34]
Intent Inference19358.1162.7466.5868.49+2.42 [−2.97, 8.10]+6.70 [1.52, 12.31]
Emotion Inference9552.0954.8352.8754.09+1.69 [−6.28, 10.27]−0.13 [−8.53, 8.07]
All types62553.1458.2059.4564.86+5.44 [2.36, 8.55]+6.64 [3.33, 9.91]

Table 3. Three version pairs on the same 625 groups (%). Each other cell lists the older value first. P+S+P^+S^+ marks understood objects. PP, RR, SS and ΔP→R\Delta_{P\to R} match the main table.

Version pairPPRRSSΔP→R\Delta_{P\to R}ΔS→R\Delta_{S\to R}R∣P+S+R\mid P^+S^+R∣¬(P+S+)R\mid\neg(P^+S^+)
MiMo-VL-7B SFT → RL32.96 → 34.4042.88 → 42.4038.72 → 42.88+4.83 → +7.69+4.20 → +11.3542.53 → 56.5742.94 → 39.73
Qwen3.5-35B → Qwen3.6-35B54.72 → 53.6057.12 → 62.0858.24 → 59.04+1.07 → +3.24+3.34 → +5.9056.81 → 67.4957.28 → 59.48
Gemini 3.6 Flash → 3.7 Flash67.20 → 73.2875.04 → 78.5677.44 → 84.32+10.77 → +10.78+12.64 → +12.0981.35 → 82.4468.12 → 71.98
AuthorsOpenReview ↗

Thanks and response to Reviewer hLys (3/5)

Response to W2: Generation order of A³Loop

We thank the reviewer for the suggestion. Section 4.2 explains the backward order qualitatively. Writing Summarization last would force it to fit around the evidence already chosen for Reasoning, so A³Loop writes Summarization first and lets Reasoning pick one of its events. To support this rationale with evidence, we add a preliminary experiment to Section 5.4.

We sample 100 inputs with a fixed seed and run two simplified A³Loops on them. The backward loop writes Summarization, Reasoning and Perception in this order, as in A³Loop, and the forward loop uses the opposite order. Both loops use the same model (Qwen3.8-Flash-Next), generation prompt, revision rules and checks, so only the order differs. For reference, we also run a single pass that writes all three questions in one call without revision.

As shown in Table 4, the backward loop passes more inputs and needs fewer calls per passed group. It also keeps gaining passes in later rounds, while the forward loop stops gaining after round 5. Only 3 inputs pass under the single pass, so both orders benefit from revision, and the backward order adds a further gain on top of it.

These results fit the rationale in Section 4.2 and support the backward order used in A³Loop.

Table 4. Generation order on the same 100 inputs. The single pass writes all three questions in one call without revision. kk counts revision rounds (k=0k=0: no revision). No input uses more than 9 of the 12 allowed rounds.

VariantPassedCalls per inputCalls per passed groupPassed within kk rounds, k=0k=0–99
Single pass3/10010.8360.7–
Loop, forward (P→R→SP\to R\to S)12/10034.7289.55, 6, 7, 9, 9, 12, 12, 12, 12, 12
Loop, backward (S→R→PS\to R\to P)16/10037.2232.43, 5, 8, 10, 12, 13, 14, 14, 15, 16
Reviewer_irEmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that existing benchmarks typically evaluate questions from different tasks concerning different objects and therefore cannot provide insight into whether models succeed across tasks involving the same object. To address this gap, the paper builds a new video QA benchmark, AnchorGroup, consisting of object-anchored QA groups. AnchorGroup covers 137.9 hours of video and contains 1,875 QA pairs in 625 QA groups. The authors also propose A3LOOP, which constructs questions in the order of summarization, reasoning, and perception through video retrieval, generation and verification agents, iterative feedback, and human review. Evaluations of 35 video MLLMs report task accuracy, joint correctness across all three questions (All3), and perception-conditioned accuracy gaps. Perception has the lowest average accuracy, while correct perception answers are generally associated with better reasoning and summarization performance.

Critical strengths
  • The shared-object grouping is valuable. The benchmark enables the assessment to go beyond evaluating how well each task is performed and instead provides a deeper insight into how various tasks are performed on the same object. The corresponding evaluation metric All3 can complement task-level accuracy by measuring whether models answer all three questions correctly for the same object.
  • Extensive experimental evaluation and analyses are broad. The main experiments in Table2 cover 35 models and report task-level results and conditional results. The finding that All3 can rise while perception-conditioned gaps shrink is useful. It shows why these metrics need to be read separately. In addition, text-only results are also included.
Critical weaknesses
  • It seems that the boundary between perception and reasoning is not reliably established. Perception asks for directly observable actions, interactions, physical states, or spatial arrangements involving the anchor, while reasoning requires an inference about an intention, cause, or underlying state. The released questions leave some uncertainty about how consistently this distinction is applied. For example, in ucs-bench_000476-G004, the reasoning question asks about a woman’s hair, with options describing it as wet, dishevelled, braided, or neatly styled. The answer is dishevelled. Another example, in video_mme_000887-G001, the reasoning question asks about the jeans’ functional state, but the options describe different visible openings or forms of wear, and the answer is vertical slits. As written, both questions appear to ask about visible attributes. Their distinction from perception questions is difficult to justify.
  • The experiments do not isolate what A3LOOP’s components contribute. A3LOOP is one of the contributions of the paper. As a proposed data pipeline method, the paper lacks comparisons with simpler pipelines under the same budget, as well as ablations of generation order, feedback, and verification. Only 625 of 23,415 candidate inputs are ultimately retained, and 550 of those groups require human revision. These numbers make it difficult to separate the contribution of the loop from later filtering and manual work, or to judge whether the added complexity pays off in quality and cost.
Questions for authors
  • How was the perception/reasoning distinction checked in the final dataset? Section 3.2 and Table 4 distinguish directly observable facts from conclusions inferred from visual evidence. Could you explain the practical criterion used to assign these labels, and show how it applies to the following examples? In ucs-bench_000476-G004, the reasoning question asks about the woman’s hair, with options such as dishevelled and neatly styled. In video_mme_000887-G001, the options describe visible openings or wear in the jeans. For each example, what evidence must the model combine, and what conclusion must it infer beyond recognizing the visible attribute? How was task membership (perception/reasoning/summarization) checked during generation, verification, and final human review? In particular, was the perception/reasoning distinction reviewed separately? If an independent review of task labels was conducted, please describe the procedure, the disagreements found, and how borderline cases were resolved. These details matter for interpreting the reported accuracy as distinct abilities.
  • What kinds of changes were made to the 550 groups accepted after human revision? How many revisions were limited to words, and how many changed the questions or distractors? Do the existing records include a quality assessment of A3Loop’s output before additional model/human review? These details would help clarify how much of the final dataset’s quality can be attributed to the automated loop.
  • Which finding best demonstrates the value of object-anchored grouping? The paper reports several observations, including differences between average-accuracy and All3 rankings, and variation in perception-conditioned accuracy gaps across models. Which finding most clearly depends on the questions referring to the same object? Could you point to the strongest result in the current experiments and explain what it allows to learn beyond separate task scores?
12 comments
AuthorsOpenReview ↗

Thanks and response to Reviewer irEm (1/6)

Thank you for your valuable and insightful comments. We deeply appreciate your input and have made every effort to address your concerns, as detailed below.

Response to W1 & Q1: Boundary between Perception and Reasoning

We thank the reviewer for the specific cases. We first explain how task labels are assigned and checked, and then apply this process to the two examples.

How labels are assigned. A³Loop has three stages, and each stage produces the question of one level. As described in Section 4.2, the loop writes the Summarization question first, uses one of its events as the visible premise of the Reasoning question, and writes the Perception question last. The criterion follows the definitions in Section 3.2. In practice, we ask whether the answer can be copied or paraphrased from a description of what the video shows. A Perception answer can, since it states a visible fact. A Reasoning answer cannot, and it must follow from the visible premise in one step.

How task membership is checked. Because the levels test different abilities, we check task membership at generation, verification and human review, with separate checks for the Perception/Reasoning distinction. At generation, the Summarization generator must cover several events, the Reasoning generator must add one conclusion that the video does not show, and the Perception generator must ask for a directly visible fact. At verification, the Reasoning verifier rejects an answer that can be copied or paraphrased from what the clip shows, and a pair verifier rejects a conclusion that only paraphrases its premise. The Perception verifier confirms that each Perception answer is directly visible. At human review, separate checklist items ask whether the Perception question is direct perception and whether the Reasoning question makes a one-step inference.

Disagreements and borderline cases. As shown in Table 1, the Reasoning checks reject drafts that restate what the video shows or says in about one fifth of the 625 final groups. The Perception verifier never rejects a Perception answer for needing inference. A rejected draft returns to the Reasoning generator, which usually takes another event as the premise. When no event is left, the loop rebuilds the Summarization question. At human review, a borderline Reasoning question is rewritten into a valid inference if the video allows it, and its group is dropped otherwise.

Table 1. Perception/Reasoning disagreements found by the separate checks in the 625 final groups. A rejection is counted only when its reason states this cause.

TotalCount
Reasoning verifier checks judging the answer to be direct observation1,347124 (9.2%)
– visible action, state or object89
– read from subtitles, dialogue or on-screen text35
Pair verifier checks judging the conclusion to be a paraphrase of the premise1,13860 (5.3%)
Perception verifier checks judging the answer to need inference9070
Groups with at least one rejection625131 (21.0%)
Next step after the 184 rejections: another event as the premise125
Next step: new Summarization question or evidence25
Next step: same premise, another attempt34
AuthorsOpenReview ↗

Thanks and response to Reviewer irEm (2/6)

Response to W1 & Q1 (continued): The two examples

ucs-bench_000476-G004. To answer the Reasoning question, the model must find the scene where the woman walks across the room and adjusts her hair (5:29–5:37), and combine this gesture with the common knowledge that people fix their hair when it is out of place. Beyond recognizing the gesture, it must infer the state behind it, which is that her hair is disheveled. A Perception question about this scene would ask what she does with her hair. The answer "disheveled" cannot be copied from a description of the gesture, so the question counts as Reasoning.

Figure 1. A³Loop on ucs-bench_000476-G004.

┌────────────────┐    ┌─────────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│ Summarization  │───>│      Reasoning      │───>│    Perception   │───>│ Multiple choice │
│                │    │                     │    │                 │    │                 │
│ talks with a   │    │ draft 1, another    │    │ Q: what is she  │    │ wet /           │
│ man, walks     │    │ event: receive a    │    │ doing at the    │    │ disheveled /    │
│ across the     │    │ box                 │    │ outdoor table?  │    │ braided /       │
│ room, adjusts  │    │ x restates action   │    │ (2:50-2:56)     │    │ neatly styled   │
│ her hair       │    │                     │    │ A: sorting      │    └─────────────────┘
└────────────────┘    │ draft 2, the hair   │    │ items into a    │             |
                      │ gesture 5:29-5:37   │    │ plastic bag     │             v
                      │ Q: condition of     │    │ + directly      │    ┌─────────────────┐
                      │    her hair?        │    │   visible       │    │   Human review  │
                      │ A: disheveled       │    └─────────────────┘    │                 │
                      │ + infers the state  │                           │ kept without    │
                      │   behind gesture    │                           │ edits           │
                      │ + pair: 5 passes,   │                           └─────────────────┘
                      │   1 paraphrase flag │
                      └─────────────────────┘

+ check passed   x check rejected

video_mme_000887-G001. The question asks for the functional state of the jeans, not just how they look. To answer it, the model must combine the levitation trick (0:16–0:30) with the close-up of the jeans (1:43–2:40). The close-up shows more than one of the listed modifications, so perception alone cannot pick the answer. The model must infer which modification makes the trick work. This choice cannot be copied from a description of the close-up, so the question counts as Reasoning.

Figure 2. A³Loop on video_mme_000887-G001.

┌────────────────┐    ┌─────────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│ Summarization  │───>│      Reasoning      │───>│    Perception   │───>│ Multiple choice │
│                │    │                     │    │                 │    │                 │
│ levitation     │    │ premise: the reveal │    │ Q: what does he │    │ horizontal      │
│ illusion, then │    │ (1:43-2:40)         │    │ do with the     │    │ rips /          │
│ the secret:    │    │ Q: functional state │    │ patterned       │    │ vertical        │
│ modified shoes │    │    of his jeans?    │    │ cloth?          │    │ slits /         │
│ and jeans, and │    │ A: vertical slits   │    │ (0:43-0:55)     │    │ frayed hems /   │
│ other tricks   │    │    letting the legs │    │ A: holds it up  │    │ large knee      │
└────────────────┘    │    fold into a      │    │ in front of his │    │ tears           │
                      │    hidden support   │    │ body            │    └─────────────────┘
                      │ + infers purpose    │    │ + directly      │             |
                      │   of the slits      │    │   visible       │             v
                      │ + pair: functional  │    └─────────────────┘    ┌─────────────────┐
                      │   inference         │                           │   Human review  │
                      └─────────────────────┘                           │                 │
                                                                        │ Summarization   │
                                                                        │ edited,         │
                                                                        │ Reasoning kept  │
                                                                        └─────────────────┘

+ check passed   x check rejected

In summary, each level comes from its own stage, and the Perception/Reasoning distinction is checked separately at generation, verification and human review.

AuthorsOpenReview ↗

Thanks and response to Reviewer irEm (3/6)

Response to W2: Contribution of A³Loop

We thank the reviewer for the suggestion. Section 4.2 of the paper motivates the components of A³Loop, and Appendices A.4 and A.6 report how many groups the loop and the later steps keep. To isolate what each component contributes, we compare a single pass with two simplified loops. The single pass writes all three questions in one call and gets no feedback from the verifiers. The two loops send each rejection back to the generator as feedback for a rewrite, and they write the questions in opposite orders.

For experiment settings, we test on the same 100 inputs sampled with a fixed seed from the 23,415 candidates. All variants use the same model (Qwen3.8-Flash-Next), generation prompt and checks. A group passes only if it passes every check, including the text-only probe and the final review. All passed groups therefore meet the same quality bar, and we compare the variants by yield and cost. Results are shown in Table 2.

Feedback and verification. All variants keep the same verifiers, since they define the quality bar that every passed group must meet, so only the feedback differs. As shown in Tables 2 and 3, feedback raises the pass rate four to five times on the 100 inputs, and most groups in the production run pass only after feedback. Most rejections come from the text-only checks, which remove questions answerable without the video.

Generation order. Backward (S→R→PS\to R\to P, as in A³Loop) passes more inputs than forward (P→R→SP\to R\to S), with fewer calls per passed group. It also keeps gaining passes in later rounds, while forward levels off early. This supports the backward order used in A³Loop. Since every passed group meets the same quality bar, both design choices pay off as more passed groups at a lower cost per group, and feedback gives the larger gain. We add this ablation to Section 5.4 of the revised paper, with the results in Table 6.

Loop vs. later steps. The comparisons above stop at the loop's own checks, so they measure the loop alone. Appendices A.4 and A.6 of the paper report how many groups each step keeps, and Table 3 adds what each step contributes. The loop's effect carries into the final data, since most final groups need its feedback. Later steps mainly fix the multiple-choice conversion and refine the wording, and only a small share of human edits correct facts. We add these statistics to Appendices A.4 and A.6.

In summary, feedback from the verifiers raises the pass rate the most, and backward generation adds a smaller gain. Most final groups exist because of the loop, while later steps mainly fix the multiple-choice conversion and refine the wording.

Table 2. A³Loop design choices on the same 100 inputs. kk counts revision rounds (k=0k=0: no revision). No input uses more than 9 of the 12 allowed rounds.

VariantPassedCalls per inputCalls per passed groupPassed within kk rounds, k=0k=0–99
Single pass3/10010.8360.7–
Loop, forward (P→R→SP\to R\to S)12/10034.7289.55, 6, 7, 9, 9, 12, 12, 12, 12, 12
Loop, backward (S→R→PS\to R\to P)16/10037.2232.43, 5, 8, 10, 12, 13, 14, 14, 15, 16

Table 3. Contribution of each step in the production run. GPT-5.6 flags come from a frame-based audit of the 1,095 groups before human review. Error and edit shares are estimated by comparing each final group with its version before review.

StepGroups keptContribution
A³Loop8,392 of 23,4157,312 pass only after feedback
Conversion, GPT-5.6 review and screening1,095 of 8,392GPT-5.6 serious flags: content 8.2%, options 28.0%, answer exposure 35.3%
Human review625 of 1,095470 removed, about 8% of edits correct facts
Final data625543 needed loop feedback, about 10% had a factual error before review
Reviewer_cpHNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AnchorGroup, a video understanding benchmark that groups perception, reasoning, and summarization questions around the same object instance within a video.

Its main contributions are:

(1) a benchmark containing 1,875 questions organized into 625 groups;

(2) an annotation pipeline that iteratively generates and verifies QA groups;

(3) an evaluation of 35 MLLMs, examining task-level accuracy, joint correctness, and cross-task associations.

Critical strengths
  1. AnchorGroup includes perception, reasoning, and summarization questions about the same object in a video. This allows comparison across tasks within a shared object context. It also shows whether a model succeeds across all three tasks, which separate task averages do not capture.

  2. The automatically generated QAs are manually checked and revised. This helps catch errors missed by automated verification and improves the reliability of the benchmark.

Critical weaknesses
  1. The reported cross-task associations may partly reflect video difficulty. In an easy video, questions about different objects may also be answered correctly together. The paper lacks a comparison between same-object and different-object question pairs within the same video. Comparing their conditional accuracy gaps would help clarify whether the shared object matters.

  2. The paper does not directly assess QA quality after automated pipeline. In particular, the remaining factual error rate is unclear. It is therefore difficult to judge whether the automated pipeline produces reliable annotations, and how much the final quality depends on human review.

Questions for authors
  1. How do the conditional accuracy gaps change when the perception and reasoning questions refer to different objects in the same video? Could you provide this comparison on a small subset, using questions with comparable task types and difficulty? This would help clarify the role of the shared object in the reported associations.

  2. Could you report the factual error rate after all automated checks and filtering, but before human review?

  3. Minor suggestion: Some text in Figures 2 and 3 is too small. The authors could increase the font size to improve readability.

6 comments
AuthorsOpenReview ↗

Thanks and response to Reviewer cpHN (1/3)

Thank you for your valuable and insightful comments. We deeply appreciate your input and have made every effort to address your concerns, as detailed below.

Response to W1 & Q1: Same object vs. different object

We thank the reviewer for this suggestion. Section 3.3 of the paper defines the conditional gaps as associations between task outcomes, and Sections 5.2 and 5.3 report them for each model. To check whether they come from the shared object rather than from video difficulty, we add a same-video, different-object experiment.

We annotate about 200 new Perception questions. Each asks about another object in the same video as an existing group. We condition the group's Reasoning and Summarization answers on either its own Perception question or the new one. To compare at equal difficulty, we keep 142 pairs with nearly the same Perception accuracy (43.34% vs. 43.42%). Most pairs (88 of 142) also share the same Perception task type.

As shown in Table 1, the gaps drop when the object changes. Across 19 open-source models, the conditional gap for Reasoning (ΔP→R\Delta_{P\to R}) falls from +5.39 to −4.60. The conditional gap for Summarization (ΔP→S\Delta_{P\to S}) falls from +5.15 to −1.23. The new question comes from the same video, yet it does not predict the group's answers. The link therefore comes from the shared object rather than from video difficulty. We add this analysis to Section 5.4 of the revised paper, with the results in Table 5.

Notation for Table 1. PP, RR, and SS denote Perception, Reasoning, and Summarization accuracy. R∣P+R\mid P^+ and R∣P−R\mid P^- are Reasoning accuracies in groups with correct and incorrect Perception answers, and ΔP→R\Delta_{P\to R} is their difference in percentage points. S∣P+S\mid P^+, S∣P−S\mid P^- and ΔP→S\Delta_{P\to S} are defined in the same way.

Table 1. Same-video, different-object control (19 open-source models, 142 pairs, %). Both rows predict the same Reasoning and Summarization questions.

Perception itemPPRRSSR∣P+R\mid P^+R∣P−R\mid P^-ΔP→R\Delta_{P\to R}S∣P+S\mid P^+S∣P−S\mid P^-ΔP→S\Delta_{P\to S}
Same object43.3455.6554.7258.6153.22+5.3957.8852.73+5.15
Different object43.4255.6554.7253.0157.61−4.6053.9555.18−1.23
AuthorsOpenReview ↗

Thanks and response to Reviewer cpHN (2/3)

Response to W2 & Q2: Factual errors before human review

We thank the reviewer for this question. After all automated checks and filtering, 1,095 groups enter human review, and Appendix A.6 of the paper reports how many of them are kept, revised or removed. To measure the error rate before review, we compare each final group with its version before review. As shown in Table 2(a), about 10% of the final groups contain a factual error before review, mostly in Perception. A frame-based GPT-5.6 audit of the 1,095 groups before review flags content errors in 8.2% of them. The automated pipeline therefore produces factually correct content for about 90% of the final groups.

Human review removes 470 groups that need heavy correction. As shown in Table 2(b), most revisions in the kept groups refine wording and details, and only a small share correct facts. Human review therefore mainly removes weak groups and refines wording, while most final groups are already factually correct before review. We add these statistics to Appendix A.6.

Table 2. Quality before human review. Counts and shares are estimated by comparing each final group with its version before review.

(a) Final groups whose pre-review version has a wrong correct option, out of 625.

PerceptionReasoningSummarizationAny question in the group
42 (6.7%)9 (1.4%)9 (1.4%)61 (9.8%)

(b) Edit types of the 900 revised stems and correct options.

Add or remove detailsRemove ambiguityPolish wordingCorrect factsChange the asked fact
48%19%23%8%1%
AuthorsOpenReview ↗

Thanks and response to Reviewer cpHN (3/3)

Response to Q3: Font size

We thank the reviewer for pointing this out. We enlarge the text in Figures 2 and 3 in the revised paper.

Reviewer_bWDKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AnchorGroup is a VQA benchmark designed to evaluate common video model capabilities indexed by object instead of by task. The benchmark contains 625 groups of three multiple-choice questions (perception, reasoning, and summarization), covering 598 videos and 137.9 hours of content. All questions in a group concern the same object instance.

The authors propose A3 Loop, an agentic annotation framework that generates questions in the order summarization → reasoning → perception, with iterative verification, revision, and video retrieval. The resulting annotations undergo additional automated screening and human quality control.

The paper evaluates 35 video MLLMs and examines task-level accuracy, joint correctness (All3), and conditional associations between perception correctness and performance on the other tasks. The main findings are that perception is generally the lowest-performing task and that correct perception answers tend to correlate with higher reasoning and summarization accuracy.

Critical strengths
  1. Useful structure for studying cross-task consistency

Organizing questions around the same object is a reasonable and potentially valuable improvement over evaluating independent QA pairs. It makes it possible to analyze whether models succeed consistently on different questions involving the same entity rather than relying exclusively on aggregate task-level accuracy. I particularly appreciate that the framework imposes shared-evidence constraints between reasoning and summarization.

  1. Broad empirical evaluation and transparent annotation statistics

The evaluation covers 35 models across several families and sizes. The paper also reports detailed statistics about its annotation pipeline, including rejection rates, agent invocation counts, manual quality control, and sensitivity to the number of sampled frames. This provides useful information for understanding how the benchmark was constructed and its potential limitations

Critical weaknesses
  1. Empirical results do not convincingly demonstrate much added value of object-anchored grouping

The central motivation is that anchoring multiple questions to the same object should enable more informative analysis of model capabilities. However, the experiments primarily report joint correctness and conditional accuracy gaps, without demonstrating that the observed relationships arise specifically from shared-object grounding.

There are several issues:

  • The reported associations may reflect shared video difficulty, model ability, or question difficulty rather than a meaningful relationship between perception and reasoning about the same object. Since the benchmark has 625 groups across 598 videos, nearly all videos have only one group, making it difficult to disentangle object-specific effects from video-level effects.
  • All3 is strongly determined by marginal task accuracies. For example, Gemini 3.7 Flash achieves 51.84% All3, while the product of its three task accuracies is approximately 48.54%. A comparison with an appropriate independence or shuffled-group baseline is needed to establish how much additional information the grouping provides.
  • The perception question is not necessarily a prerequisite for reasoning or summarization. Indeed, the design requires shared evidence between reasoning and summarization, but not between perception and reasoning. Consequently, the conditional accuracy gap does not establish that successful reasoning depends on accurate perception of the relevant evidence.

The reported mean conditional gaps (+5.44 percentage points for reasoning and +4.61 for summarization) are interesting, but modest and variable across models. Statistical uncertainty is also not reported.

I would find the contribution substantially more convincing with a controlled comparison against groups containing different objects from the same video, along with permutation baselines and confidence intervals.

This concern also affects novelty. Video-MME-v2 already evaluates related questions using grouped metrics, while previous work has explored object-centric video understanding. The specific shared-anchor constraint is a useful distinction, but its importance needs clearer experimental support.

  1. The effectiveness and scalability of A3 Loop are insufficiently validated

The annotation pipeline is presented as a major methodological contribution, but there is no controlled comparison with simpler alternatives, such as independent QA generation followed by group-level verification, a non-iterative annotation pipeline, or a different task-generation order.

This matters particularly because the reported construction process is expensive and relies heavily on downstream filtering and human intervention.

Appendix A.4 reports that 23,415 candidate inputs result in 8,392 groups passing A3 Loop, with only 625 groups surviving the subsequent screening and manual review. Table 7 reports more than 1.16 million agent calls across the production run. Furthermore, Appendix A.6 states that only 75 of the final 625 groups were accepted by human reviewers without modification, while 550 required revisions.

These figures do not invalidate the approach, but they make it difficult to assess the actual contribution of the agentic loop. In particular, it is unclear whether the eventual dataset quality comes primarily from A3 Loop or from aggressive filtering and manual correction.

I would expect an ablation quantifying the effect of iterative verification, on-demand video retrieval, the summarization-first ordering, and final human intervention. Ideally, these comparisons would report both final annotation quality and total production cost.

Without such evidence, the agentic annotation methodology appears to be an engineering composition of existing generation-and-verification techniques rather than a demonstrated methodological advance.

Questions for authors
  1. What improvement is attributable specifically to sharing an object anchor?

Could the authors compare the reported conditional associations and joint accuracy against an appropriate baseline with different object anchors but the same video, or another controlled grouping strategy? This would help isolate the contribution of shared-object context from general video difficulty.

  1. How statistically robust are the cross-task associations?

Could the authors report bootstrap confidence intervals for the conditional gaps and compare All3 against independence or shuffled-group baselines? It would also be useful to determine whether the observed associations persist after controlling for video and question difficulty.

  1. How much does A3 Loop improve annotation quality or efficiency over simpler alternatives?

A comparison with independent QA construction and post-hoc group verification, ideally under a matched computational or annotation budget, would be particularly informative. How much benefit comes from iterative feedback and summarization-first generation, and how much from subsequent filtering and human correction?

  1. How robust is benchmark quality to shortcut effects and the evaluation protocol?

Table 2 shows that some strong models perform substantially above chance without video input, despite the text-only leakage filtering. What kinds of questions remain solvable from text alone, and do the main findings hold after removing these questions?

Additionally, the uniform 64-frame evaluation can be sparse for videos lasting up to 60 minutes. Appendix A.7 demonstrates that increasing frame counts has a substantial effect on accuracy, particularly perception. To what extent does the conclusion that perception is the primary bottleneck depend on this sampling protocol?

It would also strengthen confidence in the dataset to report an independent human audit or inter-annotator agreement, since Appendix A.6 describes the final groups as being partitioned among three reviewers rather than independently labeled by multiple annotators.

6 comments
AuthorsOpenReview ↗

Thanks and response to Reviewer bWDK (1/4)

Thank you for your valuable and insightful comments. We deeply appreciate your input and have made every effort to address your concerns, as detailed below.

Response to W1 & Q1: Why does the shared object matter?

We thank the reviewer for this suggestion. Sections 1 and 3.2 of the paper discuss the value of object-anchored grouping qualitatively, and Section 5.3 shows that All3 and the conditional gaps reveal differences that task averages miss. To show that these results come from the shared object rather than from the video, we add a same-video, different-object experiment.

We annotate about 200 new Perception questions. Each asks about another object in the same video as an existing group. We condition the group's Reasoning and Summarization answers on either its own Perception question or the new one. To compare at equal difficulty, we keep 142 pairs with nearly the same Perception accuracy.

As shown in Table 1, the Perception question of the same object predicts Reasoning. The question about the other object does not. The two conditional gaps for Reasoning (ΔP→R\Delta_{P\to R}) differ by 10.0 points. Summarization (ΔP→S\Delta_{P\to S}) points the same way. Joint accuracy shows the same pattern. The two pairings have nearly the same independence baseline, since their Perception questions are equally hard. All3 exceeds this baseline only with the same object's Perception question. The two excesses differ by 1.8 points, with a 95% CI of [0.1, 3.6]. The association is therefore tied to the object rather than to the video, and capturing it requires object-anchored grouping.

This experiment also supports the shared-anchor constraint. Video-MME-v2 groups questions by capability or by reasoning chain. AnchorGroup groups them by object. The control experiment shows that the object links the answers. We add this analysis to Section 5.4 of the revised paper, with the results in Table 5.

Notation for all tables. PP, RR, and SS denote Perception, Reasoning, and Summarization accuracy. R∣P+R\mid P^+ and R∣P−R\mid P^- are Reasoning accuracies in groups with correct and incorrect Perception answers, and ΔP→R\Delta_{P\to R} is their difference in percentage points. S∣P+S\mid P^+, S∣P−S\mid P^- and ΔP→S\Delta_{P\to S} are defined in the same way.

Table 1. Same-video, different-object control experiment (19 open-source models, 142 pairs, %). Both rows predict the same Reasoning and Summarization questions. All3 is the share of groups with all three answers right. Indep. is the All3 expected if the three answers were independent. Excess is All3 minus Indep. The last row gives the difference between the two rows, with 95% bootstrap CIs over videos.

Perception itemPPRRSSR∣P+R\mid P^+R∣P−R\mid P^-ΔP→R\Delta_{P\to R}S∣P+S\mid P^+S∣P−S\mid P^-ΔP→S\Delta_{P\to S}All3Indep.Excess
Same object43.3455.6554.7258.6153.22+5.3957.8852.73+5.1515.6314.23+1.40
Different object43.4255.6554.7253.0157.61−4.6053.9555.18−1.2313.8714.29−0.41
Difference [95% CI]−0.08––––+10.00 [0.26, 20.03]––+6.38 [−1.02, 13.50]+1.76 [−0.09, 3.58]−0.06+1.82 [0.06, 3.61]
AuthorsOpenReview ↗

Thanks and response to Reviewer bWDK (2/4)

Response to W2 & Q3: Effectiveness of A³Loop

We thank the reviewer for the suggestion. Section 4.2 of the paper motivates the two design choices of A³Loop, iterative feedback and summarization-first generation, and Appendix A.4 reports the yield and cost of the production run. To measure what the loop itself contributes, we compare a single pass with two simplified loops. The single pass writes all three questions in one call without revision, which is close to the suggested post-hoc group verification. The two loops revise rejected questions and write them in opposite orders.

For experiment settings, we test on the same 100 inputs sampled with a fixed seed from the 23,415 candidates. The three variants use the same model (Qwen3.8-Flash-Next), generation prompt and checks. A group passes only if it passes every check, including the text-only probe and the final review. All passed groups therefore meet the same quality bar, and we compare the variants by yield and cost. Results are in Table 2.

Iterative revision. As shown in Table 2(a), the single pass passes only 3 of 100 inputs, and most of the rest are rejected as answerable without the video. Revision raises the pass rate in both loops at a lower cost per passed group.

Generation order. Backward (S→R→PS\to R\to P, as in A³Loop) passes more inputs than forward (P→R→SP\to R\to S) with fewer calls per passed group, and it keeps gaining passes in later rounds. The full A³Loop extends the backward loop and passes the most inputs at the fewest calls per passed group. We add this ablation to Section 5.4 of the revised paper, with the results in Table 6.

Filtering and human review. Table 2(b) traces the production run. Most loop groups pass only after revision. Screening and human review then select groups, and human review mainly refines their wording and details. We add these statistics to Appendices A.4 and A.6.

In summary, iterative feedback and summarization-first generation decide how many usable groups exist, and filtering and human correction decide how clean they are. Revision lifts the pass rate from 3% to 12–16%, while human correction fixes facts in about 10% of the final groups.

Table 2. A³Loop design choices and the production run.

(a) Variants on the same 100 inputs. kk counts revision rounds (k=0k=0: no revision). No input uses more than 9 of the 12 allowed rounds. The last row reports the production run on the same inputs with optimized generation prompts.

VariantPassedCalls per inputCalls per passed groupPassed within kk rounds, k=0k=0–99
Single pass3/10010.8360.7–
Loop, forward (P→R→SP\to R\to S)12/10034.7289.55, 6, 7, 9, 9, 12, 12, 12, 12, 12
Loop, backward (S→R→PS\to R\to P)16/10037.2232.43, 5, 8, 10, 12, 13, 14, 14, 15, 16
Full A³Loop (reference)33/10049.8151.0–

(b) Groups kept at each stage of the production run and what each stage adds.

StageGroups keptWhat the stage adds
A³Loop8,392 of 23,415 inputs87% pass only after revision
Screening1,095Keeps groups that pass GPT-5.6 review and further checks
Human review625Fixes facts in about 10% of groups and refines most others
Final data625Text-only accuracy of the generator falls from 31.05% before human review to 26.24% (chance 25%)
AuthorsOpenReview ↗

Thanks and response to Reviewer bWDK (3/4)

Response to Q2 and the related issues in W1: Statistical robustness

We thank the reviewer for the suggestions. Sections 5.2 and 5.3 of the paper report positive average conditional gaps and the All3 of each model. To test these results, we examine the confidence intervals of the gaps, the All3 baselines and whether the gaps reflect a relationship about the same object.

Confidence intervals. As shown in Table 3, both Perception-conditioned gaps stay above zero under 95% bootstrap confidence intervals. We resample videos with replacement, since groups from the same video are not independent. The gaps are modest but statistically reliable.

All3 baselines. As shown in Table 4, All3 exceeds both the independence baseline and the shuffled-group baseline. The excess over the shuffled baseline is significant on average. It is also significant for Gemini 3.7 Flash, the example raised by the reviewer. Most of All3 follows the marginal accuracies, but the grouping adds a small and reliable part beyond them.

Relationship about the same object. We check the three alternatives raised by the reviewer.

  • Video difficulty. As shown in our response to W1 & Q1, a Perception question about another object in the same video does not predict the group's Reasoning or Summarization answers, even at the same difficulty.
  • Model ability. Each gap is computed within one model, so differences in ability between models do not produce it.
  • Question difficulty. We measure the difficulty of every question by the accuracy of the other 34 models. Among groups that match in the difficulty of both questions, we shuffle the pairing of each model's two answers. As shown in Table 3, this shuffle reproduces most of each gap, so the gap comes mainly from question difficulty. That is, the questions of a group tend to be hard or easy together for all models. A Perception question about another object in the same video does not follow this pattern, so the shared difficulty is tied to the same object.

The gap therefore reflects a relationship between perception and reasoning about the same object, rather than video difficulty, model ability or difficulty unrelated to the object.

We add these results to Appendix A.8 of the revised paper.

Table 3. Perception-conditioned gaps (35 models). Notation follows our response to W1 & Q1. Brackets give 95% bootstrap confidence intervals from 2,000 resamples of videos. "From difficulty" is the gap produced by shuffling each model's answers among questions of similar difficulty. "Model-specific" is the observed gap minus this part.

Gap [95% CI]From difficultyModel-specific
ΔP→R\Delta_{P\to R}+5.44 [2.36, 8.55]5.020.42
ΔP→S\Delta_{P\to S}+4.61 [1.78, 7.27]3.850.76

Table 4. All3 against two baselines (%). The independence baseline multiplies the three task accuracies. The shuffled baseline pairs each model's Perception answers with the Reasoning and Summarization answers of another group. The excess is over the shuffled baseline. Brackets give 95% bootstrap confidence intervals from resampling videos.

All3IndependenceShuffled groupsExcess [95% CI]
Average of 35 models25.7723.5224.26+1.51 [0.83, 2.14]
Gemini 3.7 Flash51.8448.5449.69+2.15 [0.45, 3.74]

Comments on the paper

3 comments
AuthorsOpenReview ↗

Thanks and General Response (1/3)

Dear AC,

We thank all reviewers for their careful reading and constructive comments. Below, we summarize our work, the strengths noted by the reviewers and the score changes, and then respond to the common concerns. Part 3/3 summarizes each review and our response.

Our work. AnchorGroup is a video understanding benchmark organized into object-anchored QA groups. Each group asks a Perception, a Reasoning and a Summarization question about the same object, so a model's answers can be compared across tasks on the same object rather than only through separate task scores. To build these groups, we design A³Loop, an anchor-guided agentic annotation loop that writes the questions from Summarization to Reasoning and then Perception, and revises them with verifier feedback. AnchorGroup contains 625 groups from 598 videos, and we evaluate 35 video MLLMs on it.

Strengths. The reviewers find the object-anchored design valuable and interesting. Reviewer irEm writes that "the shared-object grouping is valuable" and "provides a deeper insight into how various tasks are performed on the same object". Reviewer hLys describes it as "a new and interesting evaluation perspective for video", notes that the paper "is the first to propose" such an evaluation format, and after the discussion finds the setting "interesting and valuable". Reviewer bWDK regards it as "a reasonable and potentially valuable improvement over evaluating independent QA pairs", and Reviewer cpHN notes that it shows "whether a model succeeds across all three tasks, which separate task averages do not capture". The reviewers also appreciate the broad evaluation of 35 models (bWDK, irEm, hLys), the transparent annotation statistics (bWDK), the human verification (cpHN) and the clear presentation (hLys).

Score changes. The ratings of bWDK, cpHN, irEm and hLys changed from 2, 2, 2, 2 to 2, 3, 2, 3, with confidence 3, 2, 2, 3. Reviewer cpHN raised the rating, noting that their concerns about the role of the shared object and the quality before human review are addressed, and Reviewer hLys also raised the rating. Reviewer irEm is convinced by our explanation of the Perception/Reasoning distinction and by the diagnostic analyses that show the value of AnchorGroup, and finds A³Loop useful in building the benchmark. Reviewer bWDK has not replied yet.

Common concerns. The reviewers raise four common concerns. We address them with a same-video, different-object control, two diagnostic analyses, same-budget and blind-retry variants of A³Loop, and a comparison of the final groups with their versions before review. We respond to each below and refer to the revised paper, where changes are highlighted in blue.

Common concernRaised byReviewers who consider it addressed
C1. Role of the shared objectbWDK (W1, Q1), cpHN (W1, Q1)cpHN (rating raised)
C2. Value of the object-anchored designbWDK (W1), irEm (Q3), hLys (W1, Q2)irEm, hLys (rating raised)
C3. Effectiveness of A³LoopbWDK (W2, Q3), irEm (W2), hLys (W2)hLys (rating raised), irEm ("useful in this paper")
C4. Quality before human reviewbWDK (W2, Q3), cpHN (W2, Q2), irEm (W2, Q2)cpHN (rating raised)
AuthorsOpenReview ↗

Thanks and General Response (2/3)

C1: The cross-task associations come from the shared object. These associations could also reflect video difficulty, since questions about the same video may be easy or hard together. We annotate about 200 new Perception questions, each about another object in the same video as an existing group, and keep 142 pairs matched in difficulty. As shown in Table C1, the Reasoning and Summarization gaps appear only with the same object and turn negative with the other object, and All3 exceeds the independence baseline only for the same object (Section 5.4).

Table C1. Same-video, different-object control. The gaps compare groups with correct and incorrect Perception answers (points). Indep. is the All3 expected if the three answers were independent, and Excess is All3 minus Indep. (points).

Perception question aboutReasoning gapSummarization gapAll3 (%)Indep. (%)Excess
The same object+5.39+5.1515.6314.23+1.40
Another object in the same video−4.60−1.2313.8714.29−0.41

C2: The object-anchored design reveals what limits each skill and what a model update changes. Separate task scores show how well a model performs each task, but not why. We add two diagnostic analyses based on the conditional gaps, one across Reasoning types and one across three model version pairs (Section 5.3, Table C2). The errors of Causal Implication and Intent Inference are tied mainly to event understanding, while those of Emotion Inference lie in the inference itself. After RL, MiMo-VL-7B keeps its Reasoning accuracy, but its accuracy rises sharply on objects it both perceives and summarizes correctly. Such diagnoses can guide training and model iteration.

Table C2. Diagnostic analyses.

(a) Reasoning gaps conditioned on the Perception or Summarization answer, by Reasoning type (35 models, points).

Reasoning typePerception-conditionedSummarization-conditioned
Causal Implication+7.98+18.32
Intent Inference+2.42+6.70
Emotion Inference+1.69−0.13

(b) Reasoning accuracy before → after an update (625 groups, %). Understood objects have correct Perception and Summarization answers.

Version pairAll objectsUnderstood objectsOther objects
MiMo-VL-7B SFT → RL42.88 → 42.4042.53 → 56.5742.94 → 39.73
Qwen3.5-35B → Qwen3.6-35B57.12 → 62.0856.81 → 67.4957.28 → 59.48
Gemini 3.6 Flash → 3.7 Flash75.04 → 78.5681.35 → 82.4468.12 → 71.98

C3: A³Loop builds more valid groups at a lower cost than simpler designs. A simpler pipeline or other design choices might do as well under the same budget. We compare A³Loop with a single pass, best-of-kk single passes, a forward-order loop and a blind-retry loop on the same 100 inputs, with the same model, prompt and verifiers (Table C3). With as many calls as the loop or more, best-of-4 and best-of-5 pass only 5 and 7 inputs, while the backward loop passes 16. The backward order and the verifier feedback each yield more passed inputs at fewer calls per passed group than the forward order and the blind retry.

Table C3. A³Loop design choices on the same 100 inputs.

VariantPassedCalls per inputCalls per passed group
Single pass3/10010.8360.7
Single pass, best-of-45/10038.8775.4
Single pass, best-of-57/10048.8697.3
Loop, forward (P→R→SP\to R\to S)12/10034.7289.5
Blind-retry loop, backward13/10039.8306.2
Loop, backward (S→R→PS\to R\to P)16/10037.2232.4

C4: The questions and correct answers of AnchorGroup come from A³Loop. Since only 625 of 23,415 inputs are kept and 550 of the 625 groups are revised, the final quality might come from filtering and human review rather than from the loop. We compare each final group with its version before human review and evaluate the same 625 groups before and after review (Table C4). About 90% of the final groups are already factually correct before review and only 8% of the human edits correct facts (Appendix A.6), and accuracy with or without video barely changes after review. Filtering removes weak groups and replaces weak distractors without changing any question or correct answer, and human review mainly removes groups with shortcuts or factual errors and refines the wording.

Table C4. Quality of the loop output before human review.

MeasureResult
Final groups with a factual error before reviewabout 10%
Human edits to stems and correct options that correct facts8% of 900
Final stems and correct options unchanged by review76%
Accuracy with video on the same 625 groups, before → after review58.6% → 57.5%
Accuracy without video on the same 625 groups, before → after review24.5% → 24.8%
AuthorsOpenReview ↗

Thanks and General Response (3/3)

Reviewer bWDK (rating 2, no reply yet)

  • W1 & Q1: The added value of object-anchored grouping, and whether the cross-task associations come from the shared object rather than from video difficulty. The gaps appear only with the same object (C1), and the conditional gaps show what limits each skill (C2). Reviewer cpHN considers the same question addressed.
  • W2 & Q3: Effectiveness of A³Loop compared with simpler alternatives, and how much quality comes from filtering and human correction. With as many calls or more, best-of-k passes at most 7 of 100 inputs against 16 for the loop (C3), and about 90% of the final groups are factually correct before review (C4).
  • Q2: Statistical robustness of the associations. Both gaps stay above zero under 95% bootstrap CIs over videos, All3 exceeds both the independence and the shuffled-group baselines, and a difficulty-matched shuffle ties the shared difficulty to the same object (Appendix A.8).
  • Q4: Text shortcuts and the 64-frame protocol. Perception stays at chance without video, both gaps stay significant after removing the groups each model answers without video, and on average Perception remains the lowest level at 64, 128 and 256 frames.

Reviewer cpHN (rating 2 → 3)

  • W1 & Q1: Same-object vs. different-object conditional gaps. The Reasoning gap falls from +5.39 to −4.60 points when the Perception question concerns another object (C1).
  • W2 & Q2: Factual error rate after automated checks and before human review. About 10% of the final groups, as reported in C4.
  • Q3: Font size in Figures 2 and 3. We enlarge the text in the revised paper.

The reviewer writes that "these additions address my concerns" and raised the rating.

Reviewer irEm (rating 2)

  • W1 & Q1: Whether the Perception/Reasoning boundary is applied consistently, with two examples. We explain the labeling criterion and the separate checks at generation, verification and human review, and walk through both examples. The reviewer is "convinced by the explanation of the perception/reasoning distinction".
  • W2: Component ablations of A³Loop, and its contribution separate from filtering and manual work. The loop outperforms same-budget simpler variants (C3), and the questions and correct answers come from the loop (C4).
  • Q2: Revision types and quality before review. Only 8% of the 900 human edits correct facts, and accuracy barely changes after review (C4).
  • Q3: Which finding best shows the value of object-anchored grouping. The conditional gap, which appears only for the same object, and the diagnostic analyses built on it (C1, C2). The reviewer is "convinced by ... the diagnostic analyses".
  • Follow-up 1: Whether A³Loop is suitable as an independent contribution. Each design choice of A³Loop is effective under the same budget (C3), and the content of the final data comes from the loop (C4).
  • Follow-up 2: A³Loop is useful in this paper, but an independent method should be tested on other datasets. We agree that A³Loop is a part of building AnchorGroup, as Sections 1 and 4.2 introduce it, and revise the abstract, the second contribution and the conclusion accordingly.

Reviewer hLys (rating 2 → 3)

  • W1 & Q2: Value of the new evaluation perspective. The gap depends on the shared object (C1), and the conditional gaps show what limits each skill and what an update changes (C2).
  • W2: Experimental support for the generation order. The backward order passes 16 vs. 12 inputs at 232.4 vs. 289.5 calls per passed group (C3).
  • W3: Whether multiple-choice Summarization tests perception. Multiple choice is common for summarization in video benchmarks such as HourVideo, LVBench and Video-MME. All options keep complete events and stay plausible without video (28.3% text-only accuracy), and most distractors differ from the correct option in at least two events.
  • Q1: Why Perception scores the lowest. This agrees with prior findings (MINERVA, MLVU), the generation order does not change what Perception asks, and Perception stays near chance without video and, on average, the lowest at all frame counts.

The reviewer finds the setting "interesting and valuable" and raised the rating.

Revisions in the revised paper (highlighted in blue).

  • Abstract, Sec. 1 and Sec. 6: A³Loop is presented as designed to build AnchorGroup, with other datasets left for future work.
  • Sec. 5.3 and 5.4: diagnostic analyses (Tables 3 and 4), same-video, different-object control (Table 5) and A³Loop design choices (Table 6).
  • Appendices A.4, A.6 and A.8: loop yield, human edits and quality before review, and statistical robustness (Tables 15 and 16).
  • Figures 2 and 3: enlarged text.

We thank the AC and all reviewers again for their time and constructive feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.