acceptodds
Withdrawn from ICLR 2027

SAVE: Self-Compressing Agents for Long Video Evidence via Multi-Question Pressure

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-video understanding agents are increasingly used to tackle complex long video question answering tasks through extensible tools and multi-step interactions. However, active evidence search in long videos introduces two key bottlenecks: high inference cost caused by repeated tool calls and lengthy observation sequences, and reasoning distraction caused by irrelevant or redundant evidence. Existing methods typically compress visual tokens or textual chains of thought, but overlook tool-mediated evidence search, which dominates the cost of long video agents. We propose (elf-compressing gents for Long ideo vidence), a training-time self-compression framework for long video agents. SAVE uses another question from the same video as a Pressure Question and jointly solves it with the Main Question within a shared multi-turn video-tool rollout. The resulting implicit resource pressure encourages the agent to allocate evidence-search steps more selectively. SAVE then converts the resulting joint rollouts into single-question supervision by retaining target-relevant and shared tool interactions while removing pressure-question-specific searches and answers. SAVE supports both supervised fine-tuning and reinforcement learning. Experiments on multiple long video benchmarks show that SAVE reduces tool calls and token consumption while maintaining competitive accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Understanding, Tool-Augmented Agents, Evidence Search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_oLoPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAVE, a self-compression framework for long-video agentic QA that reduces redundant tool-mediated evidence search. The key idea is to solve a Main Question jointly with a same-video Pressure Question under a shared interaction budget to induce “multi-question pressure,” then extract target-centric single-question supervision by keeping Main- and shared-evidence tool interactions while removing Pressure-only steps; the authors further refine with GRPO where rollouts still include both questions but rewards depend only on the Main Question. Across LVBench, VideoMME (with/without subtitles), and MLVU, SAVE reduces tool calls and token usage by 5–19% with minimal accuracy change, and occasionally improves accuracy (e.g., +2.6 points on MLVU) after RL.

Critical strengths
  1. The method addresses a meaningful deployment cost and transfers to single-question inference. SAVE targets unnecessary tool-mediated evidence acquisition, which can reduce both repeated model inputs and tool calls. Its deployment interface requires neither an auxiliary question nor an additional compressor. Table 1 reports consistent tool-call and token reductions across all four evaluation settings, with relatively small accuracy losses in three settings and an improvement on MLVU. This makes the proposed approach practically relevant.

  2. The behavioral analysis provides useful evidence about where savings occur. Figure 3 shows reductions concentrated in the long-search tail and in fine-grained caption retrieval, while medium-level retrieval and local video-QA usage remain approximately stable on jointly correct examples. This is more informative than reporting average trajectory length alone and is consistent with reduced repeated probing, although it does not establish the causal mechanism by itself.

Critical weaknesses
  1. The trajectory extraction rules need more detail. The authors use the baseline trajectory as a reference, but do not clearly explain how they assign evidence roles or handle useful evidence absent from that trajectory. The coherence evaluation does not establish that the extracted trajectories retain the necessary evidence. Since this extraction step produces SAVE’s supervision, these missing details make it difficult to assess and reproduce a central part of the method.

  2. The experiments do not establish the specific benefit of multi-question pressure. The paper lacks matched single-question SFT and GRPO controls, so additional training or data selection could explain some of the gains. During RL, the objective does not reward Pressure-Question correctness, and the authors do not quantify how much search budget that question consumes. These gaps weaken the evidence for the pressure mechanism, which forms the paper’s main contribution and strongly affects my recommendation.

Questions for authors
  1. How did you choose the 500 MLVU questions? Does this subset reflect the task distribution of the full development set?
  2. What exact rules do you use to assign evidence roles? How do you handle useful evidence that does not appear in the baseline trajectory?
  3. Do you have matched single-question SFT and GRPO results using the same training questions and comparable training budgets?
  4. During RL, how much search budget does the Pressure Question consume? Its accuracy alone does not show whether the agent actively searches for evidence to answer it.
  5. Do the reductions in tool calls and tokens lead to lower end-to-end inference latency? Can you also report the total training compute, including rollout generation?
Reviewer_hsHpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SAVE, a training-time self-compression framework for long-video question-answering agents. To reduce redundant tool calls and excessive context consumption during evidence search, SAVE introduces a Pressure Question from the same video as the Main Question during training-trajectory generation. The agent is asked to answer both questions under a shared interaction budget, creating implicit compression pressure. Target-centric trajectory extraction then removes steps that serve only the Pressure Question, producing compact single-question data for supervised fine-tuning. The model is further trained with joint GRPO, using rewards based solely on Main-Question answer correctness and format validity, without directly rewarding Pressure-Question correctness. Experiments on LVBench, VideoMME, and MLVU show that SAVE reduces tool calls and token consumption.

Critical strengths
  1. The paper introduces same-video multi-question pressure into evidence search for long-video agents. Through target-centric trajectory extraction, it converts joint search trajectories into training data suitable for single-question deployment. This is an interesting design that enables the agent to learn a more compact search policy without explicit penalties on tool calls or trajectory length. The LVBench analysis in Figure 3 shows that tool-call reductions are concentrated in longer search trajectories and low-level caption retrieval, providing empirical support for reduced redundant probing.
  2. SAVE does not require an external teacher model to prune trajectories or rewrite reasoning, simplifying the construction of compact supervision. Table 3 shows lower external-teacher API overhead than the teacher-based pruning methods implemented in the paper, particularly WebClipper. This offers practical value, although the comparison excludes the cost of generating source-agent trajectories.
Critical weaknesses

1.Although SAVE (RL) reduces tool calls and token consumption, its accuracy falls below the Baseline in three of the four evaluation settings. For example, accuracy drops from 55.9% to 54.1% on VideoMME without subtitles and from 57.0% to 56.4% on LVBench. These results indicate that the efficiency gains come at the expense of task performance in most evaluated settings. This trade-off may limit the method’s practical value in applications where answer accuracy is a priority.

2.The paper does not sufficiently explain why the model continues to answer the Pressure Question during GRPO optimization. Although the reward includes only Main-Question correctness and answer-format validity, the diagnostics in Appendix D suggest that the model does not completely ignore the Pressure Question. However, the analysis does not explain why this behavior persists without a direct correctness reward. The mechanism behind the joint training design, and its stability over longer training runs, therefore remain unclear.

Questions for authors

1.Could the authors report the average input and output token counts for each method in Table 1, along with their reductions relative to the Baseline, and clarify whether these statistics include token consumption by the tool models? If the logs permit, a further breakdown into tool observations, agent reasoning, and other generated content would help identify the main sources of savings.

2.GRPO rewards only Main-Question correctness and answer-format validity, while the prompt still asks the model to answer both questions. Why does the model continue attempting to answer the Pressure Question, which receives no direct reward, rather than gradually learning to ignore it? Could the authors explain the mechanism that sustains this behavior and provide experimental evidence?

3.The paper selects a different question from the same video to create shared interaction pressure, but does not examine how the relationship between the Main and Pressure Questions affects extraction quality. If the two questions require evidence from entirely disjoint temporal segments—for example, the first and last minutes of the video—can the agent still learn a transferable, compact search policy? During target-centric trajectory extraction, could tool calls across these distant segments instead introduce noise into the Main-Question trajectory?

Reviewer_v8r5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SAVE aims to reduce tool calls and tokens of long-video agents (instantiated from LongVideo-R1 with caption-tree and local video-QA tools). For each training question Q_m, a second question Q_p about the same video is added, and the source agent answers both in one rollout under a shared interaction budget. From rollouts with a correct Q_m answer, SAVE removes Q_p-specific interactions and reasoning ("target-centric extraction") and fine-tunes on the projected single-question trajectories. It then runs GRPO with joint Q_m/Q_p prompts and a reward on Q_m only. On LVBench, Video-MME (with and without subtitles), and a 500-question MLVU subset, SAVE reduces tool calls by 5–13% and total tokens by 6–19%. Accuracy changes range from −3.2 to +3.7 points.

Critical strengths
  1. Practical problem. Tool-call and context cost dominate long-video agents.
  2. No external teacher. The method needs no teacher model and no explicit length penalty.
  3. Relevant baselines. The comparison includes ConPress, WebClipper, and teacher pruning, plus a cross-video negative control.
  4. Useful behavioral analysis. Savings concentrate in the long-search tail (Fig. 3b) and in low-level caption calls (Fig. 3c).
Critical weaknesses

W1. The accuracy–efficiency trade-off is modest, and the presentation obscures it. Aggregating Table 1 with the same sample weighting as Table 2 (3,849 items) gives:

  • Baseline: 60.86% accuracy, 8.61 tool calls, 98.6k tokens.
  • SAVE (RL): 60.30% (−0.56 pt), 7.66 tool calls (−11%), 82.6k tokens (−16%).

Table 2 omits the Baseline row, which would make this visible. Accuracy drops in 3 of 4 settings, and RL is below SFT in 3 of 4. Table 1 has no confidence intervals, although Fig. 3 uses paired bootstrap. The aggregate also counts the same 900 Video-MME questions twice (with and without subtitles).

W2. Missing key controls. (a) Self-distillation without pressure. Sample K single-question rollouts from the same LongVideo-R1 agent on the same CG-Bench questions and fine-tune on correct ones, or on the shortest correct one. Filtering for correctness under a budget already biases toward shorter, more decisive trajectories. Without this control, the benefit cannot be attributed to multi-question pressure. (b) Single-question GRPO. Run GRPO from the same SFT checkpoint with single-question prompts. The current RL step adds only +0.28 pt of aggregate accuracy, so it is unclear whether the joint-prompt design matters.

W3. Coherence of extracted trajectories and the source of token savings.

  • Extraction removes "residual auxiliary-question reasoning".
  • Logical quality drops from 3.37 to 3.14 (Table 4, judged by DeepSeek-V4-Flash on 100 pairs, without human validation).
  • In the App. G cases, SAVE shows "(No reasoning text was emitted.)" for most steps.

Part of the token reduction may therefore come from suppressed reasoning rather than more selective search. Please report input vs. output tokens separately for Table 1, the fraction of empty-reasoning steps, and accuracy on questions that require multi-step verification.

W4. RL train–deploy mismatch. GRPO optimizes a policy conditioned on two questions, but deployment is single-question. Why should behavior learned under joint prompts transfer? The "shared interaction budget" is never quantified: what is the budget, and does it equal the single-question budget? Appendix Fig. 5 shows that Q_p stays active but, as the authors note, does not measure the budget Q_p consumes. Table 5 (4.26 + 2.03 tool calls in joint rollouts vs. 7.62 single) suggests the "pressure" is mostly a truncation effect.

W5. Baseline instantiations look weak.

  • T-Pruned increases tool calls and tokens in 3 of 4 settings despite training on pruned trajectories. The ≥3-tool-call filter likely biases its corpus toward long searches, or something is off in the implementation.
  • WebClipper uses only 462 trajectories for 20 epochs and yields 44 empty predictions.
  • ConPress is evaluated only at N = 2.

These make the claimed advantage over pruning hard to interpret.

W6. Novelty. The mechanism (multi-question contextual pressure, transfer to single-question inference, multi-problem GRPO without a length penalty) is ConPress and BCR. The video-specific contribution is the projection step, which W3 notes is under-specified.

Questions for authors
  1. Please give the exact algorithm, with thresholds, for assigning ℓ_t, and its agreement with a manual audit on a sample.
  2. What do correctness-filtered single-question self-distillation and single-question GRPO achieve under the same budget?
  3. What is the shared interaction budget, and how is it split between Q_m and Q_p after RL?
  4. How are the token reductions split between input and output tokens?
  5. How was the MLVU 500-question subset chosen?
Reviewer_VmJbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SAVE aims to reduce the cost of long-video agents at inference time. During data collection, the agent answers a main question and another question about the same video under a shared interaction budget. The method then keeps tool interactions that support the main question and removes content used only for the second question. These extracted trajectories are used for single-question SFT.

SAVE further applies GRPO with both questions in the rollout, while rewarding only the main answer and its format. At test time, the agent receives one question. Experiments on LVBench, VideoMME, and a 500-question MLVU subset show lower tool and token use, with some loss in accuracy in most settings.

Critical strengths

1. The method reduces inference cost without adding a test-time component.

SAVE needs neither a second question nor a separate compression model at deployment. SAVE-SFT reduces total tokens by 6.0–17.7% across the four evaluation settings, with accuracy decreases of 0.2–1.2 percentage points. SAVE-RL achieves larger token savings and improves accuracy on the MLVU subset.

2. The paper examines search behavior beyond average cost.

It provides a tool-type breakdown, paired trajectory examples, and an assessment of local reasoning quality. The analysis on jointly correct LVBench examples shows fewer low-level caption requests while other tool use changes little. This provides useful evidence that at least part of the saving comes from reducing repeated searches.

Critical weaknesses

1. The experiments do not isolate the effect of multi-question pressure.

There is no matched control that continues SFT on correct single-question trajectories, or applies single-question GRPO from the same SAVE-SFT checkpoint. These comparisons are needed to separate pressure from data selection and extra training. The cross-video control also changes the task: it explicitly tells the agent not to discuss or answer the second question and to stop after the main answer. It therefore does not create the same competition for resources as SAVE. During RL, nonzero second-question accuracy alone does not show that the agent spends a meaningful share of its tool budget on that question.

2. The central extraction procedure is not specified well enough to assess its reliability.

Section 3.3 assigns tool interactions to questions using temporal matches to a baseline trajectory and whether they support either question. However, it does not give the matching rule, explain how support is decided, or describe how mixed reasoning is edited after deletion. A useful search path may differ from the baseline path, and an earlier step may be needed to justify a later action even when it does not directly answer the main question. The 100-pair model assessment is helpful, but its average score does not establish how often these failures occur. The reported ConPress comparison also uses different numbers of training examples, making the effect of extraction harder to separate from changes in the training data.

Questions for authors
  1. Could you add matched single-question controls: SFT on correct single-question trajectories, and single-question GRPO starting from SAVE-SFT? Please keep the training examples or training budget comparable.

  2. How much of the shared budget is actually used for the pressure question during RL? Please report its answer-attempt rate and tool use, together with the numerical interaction budget, GRPO group size, learning rate, KL coefficient, and format-reward weight.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.