acceptodds
Withdrawn from ICLR 2027

ASSEMBLE: Atomic Skills for Evidence-Grounded Video Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Complex video reasoning often depends on evidence scattered across distant moments, entities, and events, yet a correct answer alone does not reveal whether a model relied on the right parts of the video. We introduce ASSEMBLE, a framework that makes supporting evidence explicit throughout long-video reasoning. ASSEMBLE organizes local observations and cross-clip narratives into timestamped evidence catalogs traceable to the source video. A grounding-aware reader then composes question-specific atomic skills whose structured outputs contain explicit evidence references and support assessments. We use correctness-gated citation alignment as a direct grounding signal: after teacher-supervised fine-tuning, Group Relative Policy Optimization (GRPO) jointly optimizes answer correctness and citation alignment. This produces inspectable intermediate traces while keeping final predictions linked to explicit supporting evidence. Using a 9B reader supervised by a 235B teacher and shared precomputed evidence catalogs, ASSEMBLE achieves 59.2% macro-averaged answer accuracy across three long-video reasoning benchmarks, compared with 58.3% for Gemini-2.5-Pro, while improving macro-averaged overlap-based Grounded accuracy by 6.7%, with gains on all three benchmarks. Ablations further show that, with the same post-trained reader and inference budget, structured skill inference improves Grounded accuracy over free-form reasoning. Together, these results show that explicit evidence grounding can be integrated directly into long-video reasoning without sacrificing answer accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long-video reasoning, evidence grounding, video question answering, atomic skills, reinforcement learning, citation alignment, multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nYouOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ASSEMBLE: each video is first turned into a timestamped textual "evidence catalog", and a Qwen3.5-9B reader answers from it using 23 prompt-defined "atomic skills", citing catalog entries. Training is LoRA SFT on Qwen3-VL-235B teacher responses (4,416), then GRPO on Video-Holmes only, with a correctness-gated citation term A·P_cite in the reward. On Video-Holmes, VRBench and CG-Bench-mini, ASSEMBLE reports 59.2% macro accuracy (Gemini-2.5-Pro 58.3%) and 25.4% Grounded accuracy (vs 18.7%) (Table 1).

Critical strengths
  1. The correctness-gated citation reward is supported by a clean, controlled ablation. On Video-Holmes, SFT and outcome-only GRPO raise accuracy (43.44 → 51.99 → 54.44). This shows that answer-level RL does not by itself produce evidence-aligned citations.
  2. The analysis is transparent and self-critical in places. The authors report interventions (Table 10), citation breakdowns (Tables 6–8), often unfavorable transfer results (Table 9) and failure cases (Figs. 8–10).
Critical weaknesses
  1. No comparison with models that take the raw video as input. All comparisons, including Gemini-2.5-Pro, read the same precomputed textual catalogs (Section 5; Table 1 caption; Appendix F), and the paper reports no result for any model given the raw video. "Matching Gemini-2.5-Pro" therefore shows comparable reading of textual evidence, not comparable video reasoning.
  2. The Grounded metric is too lenient to support the headline grounding claim. Under standard localization metrics the advantage over Gemini largely disappears (Table 2): Union mIoU is 1.40 vs. 1.41 on Video-Holmes and lower than Gemini on VRBench (11.76 vs. 17.73), and Acc@GQA stays near zero on Video-Holmes (0.16 vs. 0.11) and CG-Bench (1.71 vs. 0.99). The "+6.7% Grounded" therefore mainly reflects overlap with coarse entries rather than localization.
Questions for authors
  1. Please report results for Gemini-2.5-Pro and Qwen3-VL with the raw video as input, to quantify the gain or loss introduced by catalog construction itself, and revise the comparison with Gemini in the abstract accordingly.
  2. Are all GRPO ablation rows in Table 1 evaluated with the same inference procedure as ASSEMBLE (full), i.e., the five-call Dynamic Skills pipeline? On VRBench, whose default catalog contains no narrative entries, "+GRPO (full reward; w/o narrative)" (75.96 / 30.10) and "ASSEMBLE (full)" (78.99 / 35.15) appear to describe the same configuration but differ substantially. Please clarify what distinguishes these rows.
  3. How are graph-oriented skills such as retrieve by relation ("query graph paths or relation edges") executed on the flat (text, interval) catalog? Please report per-skill invocation frequencies on each benchmark, and an ablation with a reduced library (e.g., only the skills that appear in the traces).
  4. What prompts and citation-parsing rules were used for the external baselines, and what fraction of their responses contain at least one valid citation? A fairer control would be a baseline fine-tuned on the same SFT data and format.
  5. Dynamic skill selection shows no clear advantage over a fixed skill sequence (Table 3). With the reader and catalog held fixed, Dynamic Skills and Fixed Skills reach nearly the same accuracy on all three benchmarks (54.06 vs. 53.78 on Video-Holmes, 78.99 vs. 76.57 on VRBench, 44.59 vs. 44.54 on CG-Bench). The Grounded differences on VRBench (35.15 vs. 31.31) and CG-Bench (24.50 vs. 23.90) are within the reported confidence intervals; only Video-Holmes shows a clear gap (16.49 vs. 12.63). Question-specific skill composition, the central idea of the method, is therefore not shown to matter beyond a fixed localize-then-assess template.
  6. The skill library does not match the evidence representation. The 23 atomic skills are presented as a main contribution, but several are described as graph operations: retrieve by relation "query[ies] graph paths or relation edges", and retrieve by event returns "event or evidence nodes" (Table 5). Nodes, edges and paths are knowledge-graph or scene-graph concepts; these operations only have something to act on once the video has been built into a graph (e.g., person A —talks to→ person B). The catalog defined in Section 4.1 and Appendix A, however, is a flat list of (text, interval) entries with no nodes, edges or paths, so "querying relation edges" has nothing to retrieve. In practice, the 9B reader presumably reads the text descriptions and improvises according to the skill name. In addition, per-skill invocation frequency is not reported, and the qualitative traces use almost only a handful of skills such as localize_clue and score_hypothesis_support. This suggests the library has not been carefully designed or validated, and that many skills may be unused.
  7. The skill chain confirms a pre-selected answer rather than reasoning toward one.
    • The planning call already proposes a candidate answer ("The planner proposes C" and similar in the traces), and the subsequent skills assess this pre-determined candidate, which invites confirmation bias.
    • The authors acknowledge that the support assessment is advisory and does not constrain the final answer: in Fig. 9 the claim is assessed as unsupported, yet the final answer is still D.
  8. The external baselines are not adapted to the evaluation format. The video-reasoning baselines (Video-R1, VideoChat-R1, GRPO-CARE) are all built on Qwen2.5-VL-7B, a generation behind the reader, and none is adapted to catalog reading or the "(clip i)" citation format. The ASSEMBLE reader, by contrast, is trained with SFT and a citation reward on exactly this format. The near-zero Grounded scores of these baselines (e.g., 0.00 for Video-R1 on VRBench) more likely reflect that they were never fine-tuned on this output format than poor localization ability.
Reviewer_P4CzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ASSEMBLE presents an evidence-grounded framework for complex long-video reasoning. It constructs reusable evidence catalogs containing timestamped local descriptions and longer-range narratives. A Qwen3.5-9B reader dynamically composes 23 atomic skills for evidence retrieval, hypothesis assessment, and reasoning. Each skill produces structured outputs with evidence references and support assessments. The reader is trained with teacher-supervised fine-tuning followed by GRPO, using answer correctness and correctness-gated citation alignment. Across Video-Holmes, VRBench, and CG-Bench, ASSEMBLE achieves 59.2% average answer accuracy and 25.4% Grounded accuracy. It outperforms Gemini-2.5-Pro by 6.7 percentage points in Grounded accuracy under a shared-catalog evaluation protocol.

Critical strengths
  1. Explicit and reusable evidence representation. The timestamped catalogs and structured skill outputs make intermediate reasoning steps inspectable. They also allow evidence reuse across different questions.

  2. Grounding-aware reinforcement learning. The correctness-gated citation reward encourages both correct answers and relevant evidence references. Reward ablations demonstrate its benefits over answer-only optimization.

Critical weaknesses
  1. Weak grounding metric. Grounded accuracy counts any temporal overlap, even for overly broad citations. On Video-Holmes, citations cover 52.45% of video duration on average, but citation IoU-F1 is only 1.42%.

  2. Unclear contribution of atomic skills. The 23 skills are predefined prompts without direct training. Current ablations do not clearly show whether the gains come from dynamic skill selection or structured reasoning.

  3. Limited benchmark coverage. The evaluation mainly focuses on evidence-grounded VideoQA. Broader tests on LVBench, Video-MME-v2, and VideoZeroBench are needed to assess generalization and precise evidence grounding.

  4. Heavy reliance on generated evidence. The 235B model generates narratives that already contain causal and intentional reasoning. It is unclear how much of the reasoning is performed by the 9B reader itself.

  5. Missing comparisons with closely related video reasoning methods. The experiments lack comparisons with recent thinking-with-videos methods, such as Video-o3, VideoTemp-o3, Open-o3-Video, and VideoMind. These methods also explore evidence retrieval, structured reasoning, or grounding-aware training, making it difficult to assess the advantages of ASSEMBLE over existing approaches.

Questions for authors
  1. What is the total cost of building the evidence catalog and running the five-call inference pipeline? How much does catalog reuse reduce the cost per question?

  2. Does the temporal reward improve temporal reasoning beyond simply sorting citations by timestamp? Can the authors compare it with a citation-sorting baseline and report results on temporal reasoning questions?

Reviewer_yE7eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ASSEMBLE, a framework for evidence-grounded video reasoning. It combines timestamped evidence catalogs, atomic-skill-based reasoning, and grounding-aware post-training using SFT and GRPO. Experiments on three video reasoning benchmarks show improvements in answer accuracy and evidence grounding.

Critical strengths
  1. The paper addresses an important problem in video reasoning, where correct answers do not necessarily imply reliable evidence grounding.
  2. The framework is clearly presented, and the combination of evidence catalogs, structured skills, and grounding-aware training is practically useful.
  3. The experimental results are encouraging. The authors provide comparisons across three benchmarks, reward ablations, and controlled inference-policy experiments.
Critical weaknesses
  1. Limited methodological novelty. The atomic skills are essentially prompt-defined operations rather than learned modules. The overall framework combines existing techniques, including structured prompting, SFT, and GRPO.
  2. The contribution of dynamic skill selection is not fully isolated. Table 3 shows improved Grounded accuracy, but the comparison with free-form reasoning also changes the intermediate reasoning structure. The improvement over fixed skills is relatively modest, particularly in answer accuracy.
  3. Grounding evaluation has limitations. The main metric only requires temporal overlap with annotated evidence and does not directly measure semantic support. Although additional metrics are reported, the temporal-localization results remain mixed.
Questions for authors

Please see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.