acceptodds
Under review as a conference paper at ICLR 2027

Teaching Video Agents How to Look via Counterfactual Information Gain Policy Optimization

Abstract

Video agents are increasingly equipped with tools that actively retrieve visual evidence during reasoning. However, video retrieval can be costly, as each retrieved video evidence can introduce thousands of context tokens, and noisy, irrelevant frames can hinder model performance. In this work, we focus on developing efficient video agents that make intelligent tool decisions about when to look (whether calling a tool is necessary) and where to look (what video interval to retrieve to maximize relevant information). Existing reinforcement learning frameworks for video agents typically use outcome rewards or dense turn-level rewards, which encourage spurious tool calls and struggle to learn stopping conditions and temporal localization. To address this challenge, we introduce Counterfactual Information Gain Policy Optimization (CIGO), an RL framework that incentivizes tool calls based on the model’s intrinsic information gain compared to directly answering the question. To accurately attribute information gain to the tool calls, we sample two types of counterfactual actions branching from the same reasoning prefix: (1) thinking and answering directly, and (2) issuing a single tool call before answering. By using this counterfactual construction, the model learns the stopping condition, while variance within the tool-call subgroup grounds localization. Extensive experiments across multiple benchmarks demonstrate improved overall performance on complex video reasoning tasks compared to existing agents. Beyond aggregate performance, CIGO learns more adaptive tool use and retrieves higher-quality visual evidence than existing video agents.

Primary area:
reinforcement learning
Keywords:
long video understanding, agentic RL, information gain reward

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_51g3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Counterfactual Information Gain Policy Optimization (CIGO) to improve credit assignment for tool-using video agents, focusing on when and where to retrieve additional frames. CIGO combines counterfactual STOP/LOOK rollouts, a likelihood-based information reward, and a replay buffer for training multi-step retrieval decisions. Starting from Qwen3-VL-8B-Thinking, the method uses 4.8K RL samples without SFT and reports gains of 0.5–4.8 points across long-video QA and temporal grounding benchmarks. Additional analyses examine tool-call behavior, retrieval quality, and performance on blind versus non-blind questions.

Critical strengths
  1. Well-motivated problem. CIGO addresses the limitations of outcome-only rewards by evaluating the state-dependent value of retrieval decisions.
  2. Simple and efficient training. Counterfactual rollouts provide informative learning signals even when correctness rewards are identical. Training requires only 4.8K samples without SFT or teacher-generated trajectories.
  3. Useful behavioral analysis. The paper examines tool-call distributions, retrieval quality, and blind/non-blind performance, providing insights beyond aggregate accuracy.
  4. Good reproducibility. Training configurations, prompt templates, and an anonymized code repository are provided.
Critical weaknesses
  1. Small gains and unfair comparisons. Improvements over the base model are only 0.5–1.9 points on four of five QA benchmarks, without statistical significance analysis. Most competing agents use weaker base models and different retrieval budgets, making direct comparisons difficult.
  2. Weak ablation baselines and limited evidence of adaptive retrieval. GRPO variants perform substantially worse than the untrained base model, suggesting CIGO may primarily learn to avoid harmful retrieval. Moreover, tool-call allocation shows no adaptation on LVBench (1.0× necessary/unnecessary ratio), weakening the central claim.
  3. Unclear reward design and implementation. The self-referential information reward may conflate retrieved evidence with generated reasoning, potentially encouraging reward hacking. The STOP/LOOK training mechanism, discount factor γ, and likelihood computation for temporal grounding are insufficiently explained.
  4. Limited novelty and missing efficiency analysis. The main components closely resemble prior work (CaRT, IGPO, CIGPO), without controlled comparisons. Despite emphasizing retrieval efficiency, the paper reports no token or latency measurements. Minor presentation issues, including an empty appendix and inconsistent notation, remain.
Questions for authors
  1. Could you report results for the zero-shot base model with tool access, along with CIGO's evaluation-time tool-call distribution and accuracy conditioned on the number of calls? Why do the GRPO baselines perform substantially worse than the no-tool base model?
  2. Could you provide confidence intervals or results across multiple seeds, and compare CIGO with IGPO- or CaRT-style objectives using the same base model and training data?
  3. How are STOP/LOOK continuations sampled or enforced during training? How is γ applied, and how is the information reward computed for temporal grounding? Have you investigated potential reward hacking through generated reasoning?
  4. Why does adaptive retrieval show no improvement on LVBench? Could you also report per-question token consumption and latency to substantiate the claimed efficiency benefits?
5 comments

Dear Reviewer 51g3,

Thank you for your thorough review and your insightful feedback. We answer each point below; each answer gives the key results and refers to the General Response (GR) where it adds more detail, and all new results are in the revised PDF.

"Improvements over the base model are only 0.5–1.9 points … without statistical significance analysis. Most competing agents use weaker base models and different retrieval budgets."

Over 3 seeds, the gain exceeds one standard deviation on LVBench, LongVideoBench and Charades-STA. For a controlled comparison, Table 2 of the paper trains every variant from the same base model on the same data and call budget. We also add the base model with and without tools (table below). We also reproduce IGPO on the same base model, data and budget (results under the novelty question below).

ModelVideo-MMEMLVULVBenchLongVideoBenchVideo-MMMUCharades-STA
Base, no tools67.472.045.162.367.549.9
Base, with tools (zero-shot)67.471.144.661.163.226.1
CIGO (3 seeds)68.1 ± 1.172.7 ± 1.147.9 ± 1.163.8 ± 1.268.4 ± 1.357.5 ± 1.3

Bold: the gain over Base exceeds one SD. Tool access alone does not improve the base model on any of the 6 benchmarks.

Table 1 now reports the seed statistics, and Table 2 adds the base model with tools.

"GRPO variants perform substantially worse than the untrained base model, suggesting CIGO may primarily learn to avoid harmful retrieval."

As shown in Sec. 4.4, CIGO calls more when the information is insufficient than when it is sufficient (1.33× on LVBench and 4.41× on ReXTime), whereas existing agents call a similar number of times regardless. We note that CIGO not only avoids unnecessary tool calls, but also improves tool-call quality: its retrieved windows are the most accurate and the most helpful to a reference model (Sec. 4.4, Table 4).

We also re-evaluated all our models and reproduced baselines under a unified evaluation protocol, with frame resizing disabled in the evaluation framework following Qwen3-VL’s suggested practice. Under this protocol, GRPO is within 1.6 points of the base model on long-video QA and above it on Video-MMMU and Charades-STA:

ModelVideo-MMEMLVULVBenchLongVideoBenchVideo-MMMUCharades-STA
Base (no tools)67.472.045.162.367.549.9
GRPO66.171.443.561.667.853.8

"Tool-call allocation shows no adaptation on LVBench (1.0× necessary/unnecessary ratio)."

The original “unnecessary” label (whether the ground truth evidence improves the reference model's success rate) mixes 2 groups: (a) questions the overview already answers, and (b) questions that even adding the ground-truth frames cannot reliably solve. In the latter case, it is still reasonable to call the tool, since the existing information is insufficient for answering the question. Split by whether the overview suffices, CIGO calls 1.33× more on insufficient questions on LVBench and 4.41× on ReXTime; it is the only agent whose 95% CI excludes 1 on LVBench.

AgentLVBenchReXTime
Video-o31.10׆1.07×
VideoTemp-o31.01׆1.00׆
ParaVT1.16׆0.98׆
CIGO1.33×4.41×

Ratio: mean calls on insufficient questions / mean calls on sufficient questions. †: the 95% bootstrap CI includes 1.

We updated Table 3 (Sec. 4.4) with this split; GR Q4 gives the definitions of the three groups and the mean number of calls in each.

"The self-referential information reward may conflate retrieved evidence with generated reasoning, potentially encouraging reward hacking."

In the proposed method (post-rationale information potential), the rationale is unlikely to lead to reward hacking or inflation due to 2 designs: (1) Φ is scored right before the final answer, and all continuations are sampled from the same fixed prefix. Therefore the model must make decisions that lead to more information gain before the answer, either it is directly answering or calling the tool. (2) we normalize within the valid options, so Φ measures the belief shift among the options rather than the probability of emitting an answer at that point.

We implement the CIGO variant that assigns information reward post-retrieval before further reasoning, and observe that gain still persists if we score Φ post-retrieval. Our full analysis is in GR Q6. We argue that measuring evidence-only information gain is another effective instantiation for our proposed training framework, and we have added this post-retrieval variant to the paper (App. D.2, Table 10).

Φ scoredVideo-MMEMLVULVBenchLongVideoBenchVideo-MMMUCharades-STATool calls at evaluation (%): 0 / 1 / 2 / 3
Base model67.472.045.162.367.549.9100 / – / – / –
Post-rationale (default)68.172.747.963.868.457.565.6 / 13.6 / 17.3 / 3.6
Post-retrieval68.973.148.764.469.659.112.5 / 85.1 / 1.2 / 1.3

"The STOP/LOOK training mechanism, discount factor γ, and likelihood computation for temporal grounding are insufficiently explained."

Please refer to our full response in GR Q8 (branch sampling), GR Q7 (γ) and GR Q5 (Φ) for detailed clarifications. We have also clarified all three in Sec. 3 of the revised paper. In short:

  • Branch sampling. From each prefix, the policy samples 8 continuations, each of which either answers directly or calls the tool once and then answers. Branches are sampled freely; forcing both branches lowers performance on all six benchmarks.
  • Discount γ. γ multiplies only the correctness reward of LOOK continuations. CIGO uses γ = 1.0; with γ < 1, we observe that the policy converge to a no-call behavior.
  • Φ for temporal grounding. We score only the timestamp digits, renormalize each digit over 0–9, and average over digit positions.

"The main components closely resemble prior work (CaRT, IGPO, CIGPO), without controlled comparisons."

IGPO. We reproduce IGPO on the same base model, data and budget (Table 2). CIGO outperforms it on all six benchmarks (+0.6 to +2.8). Like the other standard-rollout methods, IGPO calls the tool exactly once on 92.6% of questions.

MethodVideo-MMEMLVULVBenchLongVideoBenchVideo-MMMUCharades-STAExactly 1 call (%)
Base, no tools67.472.045.162.367.549.9–
IGPO65.671.545.162.667.856.792.6
CIGO68.172.747.963.868.457.513.6

CaRT. We thank the reviewer for request the additional of this meaningful baseline. Unfortunately, due to time and resource limitations, we could not reproduce CaRT's counterfactual rollout sampling for supervised fine-tuning. We provide a table to specify how our proposed CIGO distinguish from existing related methods:

MethodRollout startWhat is comparedTraining signal
IGPOQuestionConsecutive turns of one trajectoryOutcome + per-turn information gain
CIGPOQuestionConsecutive evidence sets (text QA)Outcome + information gain
CaRTPrefixes selected offline where the decision changes the outcomeEdited prefixes (counterfactual states)SFT on stop/continue labels from an external model; RL variant rewards agreement with the labels
CIGOQuestion, or a policy-sampled prefix from a replay bufferSTOP vs. LOOK from the same prefix (counterfactual actions)Correctness + counterfactual information gain; no SFT

We added this comparison to App. C.1 (Table 7) and the IGPO reproduction to App. C.2 (Table 8).

"The paper reports no token or latency measurements."

We provide a detailed response in GR Q3 and the full analysis on all six benchmarks in Sec. 4.5 (Fig. 4) of the revised paper. Below we summarize the key takeaways and numbers on three benchmarks.

Takeaway. CIGO lies on the accuracy–token frontier on all three benchmarks and is the most accurate agent on Video-MME and Video-MMMU at a similar token count. Its latency is higher, mainly inherited from its Thinking base model, which already takes 23.2 s per question on Video-MME without tools.

AgentVideo-MME: acc. / tokens / sVideo-MMMU: acc. / tokens / sCharades-STA: mIoU / tokens / s
Base (no tools), reduced65.0 / 7.2k / 20.762.5 / 10.3k / 78.845.7 / 2.3k / 7.6
Base (no tools), default67.4 / 19.8k / 23.267.5 / 22.6k / 87.549.9 / 4.7k / 9.3
Video-o3-7B66.5 / 18.0k / 9.051.7 / 19.2k / 10.660.7 / 6.0k / 2.8
VideoTemp-o3-7B64.5 / 22.7k / 23.253.2 / 34.4k / 37.657.8 / 8.3k / 4.7
CIGO, reduced65.1 / 8.8k / 29.862.4 / 10.7k / 66.654.9 / 3.3k / 12.1
CIGO, default68.1 / 21.5k / 35.868.4 / 23.0k / 75.557.5 / 5.6k / 13.5

Default: the setting of Sec. 4.1; reduced: 256 frames at 224². Tokens and latency are per question; latency on one L40S GPU, batch size 1.

"Minor presentation issues, including an empty appendix and inconsistent notation."

We completed the appendix, including method and training details, comparison with related methods, additional ablations, tool-call and inference-cost analyses, and have fixed previous the notation inconsistencies in Sec. 3.

Thank you again for your insightful comments, which have helped improve the paper. We hope these results address your concerns, and we are happy to answer any further questions.

Reviewer_ndw4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CIGO trains video agents using STOP and single-retrieval LOOK continuations from a shared prefix. It combines answer correctness with ground-truth-answer information potential and reuses intermediate states through a replay buffer. Experiments report improvements over the base model and RL baselines, alongside analyses of tool allocation and retrieved evidence.

Critical strengths
  1. Well-motivated action comparison. Comparing stopping with one additional retrieval from the same prefix is more directly aligned with the decision being learned than assigning one terminal reward to an entire trajectory.
  2. Evidence beyond aggregate accuracy. The matched-backbone RL comparisons and forced-retrieval evaluation show better retrieved evidence than the tested alternatives (Tables 2 and 4), providing useful support for the approach.
Critical weaknesses
  1. The core training signal is not fully specified or isolated. Figure 2 identifies information potential as log-likelihood, but multi-token normalization and the branch-sampling procedure are unclear. The discount in Eq. (3) is absent from Eq. (8). Furthermore, scoring after newly generated reasoning conflates evidence quality with the rationale's effect on answer likelihood.
  2. The efficiency and adaptive-stopping evidence is limited. Evaluation starts with 1,024 frames on four benchmarks, so three additional calls alone do not establish low total cost. QA gains over the base model are modest, without reported run variability. Table 3 shows essentially identical call allocation for necessary and unnecessary LVBench questions, limiting the generality of the stopping claim.
Questions for authors
  1. Method clarification: Does the potential scorer track the updated policy, how is it normalized for temporal-grounding answers, how are STOP/LOOK branches sampled, and where is the retrieval discount applied?
  2. Causal clarification: Does the gain persist when potential is scored before post-retrieval reasoning? What exactly is the “CT” row, and does it provide a counterfactual-only control?
  3. Efficiency clarification: What are inference-time total visual/text tokens and latency, and do repeated runs confirm the small QA gains?
3 comments

Dear Reviewer ndw4,

Thank you for your insightful comments and effort to review. We answer each of your points below; each answer gives the key results and refers to the General Response (GR) where it adds more detail, and all new results are in the revised PDF.

"Does the potential scorer track the updated policy?"

No. Φ is computed with the frozen reference model (Qwen3-VL-8B-Thinking), so the reward does not drift as the policy updates. We now state this explicitly in Sec. 3.3.

"How is it normalized for temporal-grounding answers?" / "multi-token normalization … unclear."

We score only the timestamp digits, renormalize each digit over 0–9, and average over digit positions; template tokens are teacher-forced but not scored. For multiple choice, Φ is the log-probability of the correct option renormalized over the options.

Please refer to GR Q5 for the full formulation. We added the full definition to Sec. 3.3 (Eq. (4)).

"How are STOP/LOOK branches sampled?"

Please refer to our GR Q8 for detailed answers in the sampling procedure.

From each prefix, the policy samples 8 continuations, each of which either answers directly or calls the tool once and then answers; we use a prompt cue after the tool output enforces the one-call budget during training. Branches are sampled freely, no forcing that ensures both LOOK and STOP exists in every group. If all eight continuations choose the same action, the group is still used.

We clarified this procedure in Sec. 3.2 and App. A.1 and added the forcing ablation to App. D.3 (Table 11).

"Where is the retrieval discount applied?" / "The discount in Eq. (3) is absent from Eq. (8)."

Please refer to our GR Q7 for detailed clarification for discount factor. In short, γ multiplies only the correctness reward of LOOK continuation, and the information gain is not discounted. CIGO training uses γ = 1.0 (no discounting); Below we provide the ablation on different values of γ. With γ < 1, we observe that the model converges to a zero-call policy.

SettingVideo-MMELVBenchCharades-STA0 calls (%)Mean calls
Base, no tools67.445.149.9100–
γ = 1.0 (CIGO)68.147.957.565.60.59
γ = 0.9567.646.855.799.10.01
γ = 0.8567.246.554.41000.00

We added γ to Eq. (6), its rationale to App. A.2 and this ablation to App. D.1 (Table 9).

"Does the gain persist when potential is scored before post-retrieval reasoning?"

Yes. We provide a detailed response in GR Q6. We implemented this evidence-only variant and report the results below. We observe that it matches or exceeds the post-rationale model on all six benchmarks, while it learned a policy that allocates more tool calls.

Φ scoredVideo-MMEMLVULVBenchLongVideoBenchVideo-MMMUCharades-STATool calls at evaluation (%): 0 / 1 / 2 / 3
Base model67.472.045.162.367.549.9100 / – / – / –
Post-rationale (default)68.172.747.963.868.457.565.6 / 13.6 / 17.3 / 3.6
Post-retrieval68.973.148.764.469.659.112.5 / 85.1 / 1.2 / 1.3

"What exactly is the 'CT' row, and does it provide a counterfactual-only control?"

Yes. It uses counterfactual STOP/LOOK rollouts with the correctness reward only and no information reward. We renamed it CF and clarified it in Table 2.

"What are inference-time total visual/text tokens and latency?"

We provide a detailed response in GR Q3 and the full analysis on all six benchmarks in Sec. 4.5 (Fig. 4) of the revised paper. Below we summarize the key takeaways and numbers on three benchmarks.

Takeaway. CIGO lies on the accuracy–token frontier on all three benchmarks and is the most accurate agent on Video-MME and Video-MMMU at a similar token count. Its latency is higher, mainly inherited from its Thinking base model, which already takes 23.2 s per question on Video-MME without tools.

AgentVideo-MME: acc. / tokens / sVideo-MMMU: acc. / tokens / sCharades-STA: mIoU / tokens / s
Base (no tools), reduced65.0 / 7.2k / 20.762.5 / 10.3k / 78.845.7 / 2.3k / 7.6
Base (no tools), default67.4 / 19.8k / 23.267.5 / 22.6k / 87.549.9 / 4.7k / 9.3
Video-o3-7B66.5 / 18.0k / 9.051.7 / 19.2k / 10.660.7 / 6.0k / 2.8
VideoTemp-o3-7B64.5 / 22.7k / 23.253.2 / 34.4k / 37.657.8 / 8.3k / 4.7
CIGO, reduced65.1 / 8.8k / 29.862.4 / 10.7k / 66.654.9 / 3.3k / 12.1
CIGO, default68.1 / 21.5k / 35.868.4 / 23.0k / 75.557.5 / 5.6k / 13.5

Default: the setting of Sec. 4.1; reduced: 256 frames at 224². Tokens and latency are per question; latency on one L40S GPU, batch size 1.

"Do repeated runs confirm the small QA gains?"

To assess the robustness against repeated runs, we retrain the model using three random seeds and report the mean and standard deviation of performance. Over 3 seeds, the gain exceeds one standard deviation on LVBench, LongVideoBench and Charades-STA:

ModelVideo-MMEMLVULVBenchLongVideoBenchVideo-MMMUCharades-STA
Base67.472.045.162.367.549.9
CIGO (3 seeds)68.1 ± 1.172.7 ± 1.147.9 ± 1.163.8 ± 1.268.4 ± 1.357.5 ± 1.3
Δ+0.7+0.7+2.8+1.5+0.9+7.6

Bold: the gain exceeds one SD.

Table 1 now reports the mean and standard deviation over the 3 seeds.

"Table 3 shows essentially identical call allocation for necessary and unnecessary LVBench questions."

Please refer to GR Q4 for the detailed analysis. The original “unnecessary” label (whether the ground truth evidence improves the reference model's success rate) mixes 2 groups: (a) questions the overview already answers, and (b) questions that even adding the ground-truth frames cannot reliably solve. In the latter case, it is still reasonable to call the tool, since the existing information is insufficient for answering the question. Split by whether the overview suffices, CIGO calls 1.33× more on insufficient questions on LVBench and 4.41× on ReXTime; it is the only agent whose 95% CI excludes 1 on LVBench.

AgentLVBenchReXTime
Video-o31.10׆1.07×
VideoTemp-o31.01׆1.00׆
ParaVT1.16׆0.98׆
CIGO1.33×4.41×

Ratio: mean calls on insufficient questions / mean calls on sufficient questions. †: the 95% bootstrap CI includes 1. We updated Table 3 (Sec. 4.4) with this split.

Thank you again for your detailed review, which helped us improve the paper. We hope these results address your concerns. Please feel free to let us know for any further questions, and we are happy to answer them quickly.

Dear reviewer ndw4,

Thank you again for your time and thoughtful feedback! As the discussion period is coming to an end, we wanted to kindly follow up and see whether our responses and additional experiments have addressed your concerns. If there are any remaining questions or points you would like us to clarify, we would be very happy to discuss them further!

Reviewer_ZFFfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Counterfactual Information Gain Policy Optimization (CIGO), a reinforcement learning framework that trains video agents to decide when additional visual evidence is necessary and where to retrieve it. The method constructs counterfactual rollouts from the same reasoning prefix, comparing direct answering (STOP) against retrieving an additional video segment before answering (LOOK). It combines an information-potential reward, measured through the model's confidence in the ground-truth answer, with an outcome-based reward to train retrieval decisions and temporal localization. The authors train CIGO on 4,800 examples using Qwen3-VL-8B-Thinking and evaluate it on six video understanding benchmarks. CIGO consistently improves over the base model, with gains ranging from 0.5 to 4.8 points.

Critical strengths
  1. Well-motivated counterfactual formulation for retrieval credit assignment. The paper addresses an important limitation of outcome-based RL for tool-using agents: successful trajectories do not necessarily imply that every retrieval action was useful. Comparing STOP and LOOK continuations from the same reasoning prefix provides a meaningful training signal for deciding whether additional evidence is needed.

  2. Controlled ablations and evidence-quality analyses support the proposed method. The experiments go beyond final-answer accuracy to analyze the behavior of the retrieval policy. Table 2 shows that counterfactual training consistently improves performance over GRPO with information rewards across all six benchmarks and avoids the fixed one-call behavior exhibited by several baselines.

Critical weaknesses
  1. The improvements over the base model are relatively small, and their statistical robustness is unclear.

  2. The claimed adaptive stopping and retrieval efficiency are not fully substantiated by the evaluation. A central motivation is to reduce unnecessary retrieval and visual-context costs. However, the tool-call distributions in Table 2 primarily describe training rollouts rather than inference-time behavior, and the paper does not provide a systematic comparison of actual inference latency, processed visual tokens, or total computational cost. This matters because CIGO already receives 1,024 initial frames on several benchmarks, in addition to its retrieval budget, making tool-call counts alone an insufficient efficiency metric.

Questions for authors
  1. How exactly is the information potential (\Phi(s)) computed and normalized, particularly for temporal grounding with variable-length timestamp answers?

  2. Could the authors report the actual distribution of retrieval calls, total processed frames or visual tokens, and inference latency on each evaluation benchmark? A matched-budget efficiency comparison would help establish whether CIGO reduces computational cost in practice. It would also be useful to clarify why the necessity-based retrieval analysis shows little separation on LVBench.

3 comments

Dear Reviewer ZFFf,

Thank you for your insightful feedback and your effort for review. We answer each of your points below, with new results updated in the revised PDF.

"The improvements over the base model are relatively small, and their statistical robustness is unclear."

We provide a detailed response in GR Q1 and summarize the key takeaways and numbers below. To study CIGO’s robustness under run variability, we retrain the model using three random seeds and report the mean and standard deviation of performance. The gain exceeds one standard deviation on LVBench (+2.8), LongVideoBench (+1.5) and Charades-STA (+7.6). These results characterize the robustness of the improvements while showing that their magnitude varies across benchmarks. We have updated Table 1 with these results.

ModelVideo-MMEMLVULVBenchLongVideoBenchVideo-MMMUCharades-STA
Base67.472.045.162.367.549.9
CIGO (3 seeds)68.1 ± 1.172.7 ± 1.147.9 ± 1.163.8 ± 1.268.4 ± 1.357.5 ± 1.3
Δ+0.7+0.7+2.8+1.5+0.9+7.6

"The tool-call distributions in Table 2 primarily describe training rollouts rather than inference-time behavior."

We have updated Table 2 to reflect the inference-time tool call distributions. We observe that CIGO directly answers the questions 65.6% of times on average, whereas the variants without the information reward or without counterfactual rollouts converge to a one-call retrieval policy.

Method0 calls1 call2 calls3 calls
Base (w/ tool, zero-shot)54.843.40.90.9
GRPO3.394.11.11.5
GRPO + AR3.794.10.91.2
GRPO + IR7.591.10.60.8
IGPO5.492.60.91.2
CF8.686.11.04.3
CIGO (CF + IR)65.613.617.33.6

We also added the accuracy by number of calls on all six benchmarks to App. E.1 (Table 12); GR Q2 shows that CIGO's gains come mainly from the questions where it calls the tool.

"Could the authors report … total processed frames or visual tokens, and inference latency …? A matched-budget efficiency comparison would help."

We provide a detailed response in GR Q3 and the full analysis on all six benchmarks in Sec. 4.5 (Fig. 4) of the revised paper. Below we summarize the key takeaways and numbers on three benchmarks.

Takeaway. CIGO lies on the accuracy–token frontier on all three benchmarks and is the most accurate agent on Video-MME and Video-MMMU at a similar token count. Its latency is higher, mainly inherited from its Thinking base model, which already takes 23.2 s per question on Video-MME without tools.

AgentVideo-MME: acc. / tokens / sVideo-MMMU: acc. / tokens / sCharades-STA: mIoU / tokens / s
Base (no tools), reduced65.0 / 7.2k / 20.762.5 / 10.3k / 78.845.7 / 2.3k / 7.6
Base (no tools), default67.4 / 19.8k / 23.267.5 / 22.6k / 87.549.9 / 4.7k / 9.3
Video-o3-7B66.5 / 18.0k / 9.051.7 / 19.2k / 10.660.7 / 6.0k / 2.8
VideoTemp-o3-7B64.5 / 22.7k / 23.253.2 / 34.4k / 37.657.8 / 8.3k / 4.7
CIGO, reduced65.1 / 8.8k / 29.862.4 / 10.7k / 66.654.9 / 3.3k / 12.1
CIGO, default68.1 / 21.5k / 35.868.4 / 23.0k / 75.557.5 / 5.6k / 13.5

Default: the setting of Sec. 4.1; reduced: 256 frames at 224². Tokens and latency are per question; latency on one L40S GPU, batch size 1.

"How exactly is the information potential Φ(s) computed and normalized, particularly for temporal grounding with variable-length timestamp answers?"

We provide a detailed response in GR Q5 and summarize the key points below. Φ is scored by the frozen reference model right before the answer. For temporal grounding, we score only the whole-second timestamp digits (drop decimal digits), renormalize each digit over 0–9, and average over digit positions, so the scale does not depend on the number of digits; template tokens are teacher-forced but not scored. For multiple choice, Φ is the log-probability of the correct option renormalized over all valid options. We have added the full definition to Sec. 3.3 (Eq. (4)).

"Clarify why the necessity-based retrieval analysis shows little separation on LVBench."

The original “unnecessary” label (whether the ground truth evidence improves the reference model's success rate) mixes 2 groups: (a) questions the overview already answers, and (b) questions that even adding the ground-truth frames cannot reliably solve. In the latter case, it is still reasonable to call the tool, since the existing information is insufficient for answering the question. Split by whether the overview suffices, CIGO calls 1.33× more on insufficient questions on LVBench and 4.41× on ReXTime; it is the only agent whose 95% CI excludes 1 on LVBench.

AgentLVBenchReXTime
Video-o31.10׆1.07×
VideoTemp-o31.01׆1.00׆
ParaVT1.16׆0.98׆
CIGO1.33×4.41×

Ratio: mean calls on insufficient questions / mean calls on sufficient questions. †: the 95% bootstrap CI includes 1.

We updated Table 3 (Sec. 4.4) with this split; GR Q4 gives the definitions of the three groups and the mean number of calls in each.

Thank you again for your detailed review, which helped us improve the paper. We hope these results address your concerns, and we are happy to answer any further questions.

Dear reviewer ZFFf,

Thank you again for your time and your insightful review! As the discussion period is coming to an end, we wanted to kindly follow up and see whether our responses and additional experiments have addressed your concerns. If there are any remaining questions or points you would like us to clarify, we would be very happy to discuss them further!

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response: Shared Clarifications and New Experiments

We thank all reviewers for their insightful and constructive feedback. Most reviewers found the proposed counterfactual STOP/LOOK comparison well motivated (ZFFf, ndw4, 51g3), the tool decision analyses beyond aggregated accuracy useful (ZFFf, ndw4, 51g3), and the training framework simple and reproducible (51g3).

In this response, we focus on providing shared clarifications and a summary of additional experiments for questions that are raised by more than one reviewer (Q1–Q8). In our individual response for the reviewers, we refer to the shared message for the full details.

Apart from the concerns, we re-evaluated all our models and reproduced baselines under a unified evaluation protocol, with frame resizing disabled in the evaluation framework following Qwen3-VL’s suggested practice; Tables 1–3 now contain the updated numbers.

#ConcernRaised byWhat we addedRevisions in PDF
Q1Significance of gains over the base modelZFFf, ndw4, 51g3Standard deviation over 3 seedsTable 1
Q2Tool call distributions during evaluationZFFf, 51g3Call distribution and accuracy by number of calls on all six benchmarksTable 2, App. E.1
Q3CIGO’s retrieval efficiency in terms of tokens, latencyZFFf, ndw4, 51g3Accuracy vs. tokens and latency on the same hardwareFig. 4
Q4Why LVBench doesn’t show adaptive calling based on tool benefitZFFf, ndw4, 51g3Allocation split by whether the overview sufficesTable 3
Q5How information potential is computedZFFf, ndw4, 51g3Full definitionSec. 3.3
Q6Whether reasoning inside Φ lead to reward hackingndw4, 51g3Evidence-only variant of ΦApp. D.2, Table 10
Q7Where the discount γ is appliedndw4, 51g3Revised Eq. (6); γ ablationApp. A.2, D.1
Q8How STOP/LOOK continuations are sampledndw4, 51g3Procedure for training and inferenceSec. 3.2, App. A.1, D.3

Q1. Are the gains over the base model significant? [ZFFf, ndw4, 51g3]

To study CIGO’s robustness under run variability, we retrain the model using three random seeds and report the mean and standard deviation of performance. The gain exceeds one standard deviation on LVBench (+2.8), LongVideoBench (+1.5) and Charades-STA (+7.6). These results characterize the robustness of the improvements while showing that their magnitude varies across benchmarks. We have updated Table 1 with these results.

ModelVideo-MMEMLVULVBenchLongVideoBenchVideo-MMMUCharades-STA
Base67.472.045.162.367.549.9
CIGO (3 seeds)68.1 ± 1.172.7 ± 1.147.9 ± 1.163.8 ± 1.268.4 ± 1.357.5 ± 1.3
Δ+0.7+0.7+2.8+1.5+0.9+7.6

Q2.1 Zero-shot base model with tool access

We report the results for the zero-shot base model with tool access in Table 2. We observe that (1) tool access alone does not improve performance on any of the 6 datasets. On Charades-STA it drops to 26.1 mIoU. (2) CIGO improvements come from CIGO’s RL training, not from zero-shot tool usage.

ModelVideo-MMEMLVULVBenchLongVideoBenchVideo-MMMUCharades-STAQuestions with ≥1 call (%)
Base, no tools67.472.045.162.367.549.90
Base, with tools (zero-shot)67.471.144.661.163.226.145.2
CIGO68.172.747.963.868.457.534.4

Q2. Tool call distribution on all evaluation benchmarks [ZFFf, 51g3]

We report the inference time tool call distribution on all the benchmarks

BenchmarkCallsShareCIGOBaseΔ
Video-MME070.472.974.6-1.7
18.359.453.8+5.6
219.255.649.6+6.0
32.156.140.4+15.7
All10068.167.4+0.7
LVBench043.643.150.2-7.1
113.453.441.0+12.4
231.448.141.6+6.5
311.758.639.8+18.8
All10047.945.1+2.8
Charades-STA077.654.651.1+3.5
114.670.547.8+22.7
27.361.941.4+20.5
30.5†67.838.2+29.6
All10057.549.9+7.6
MLVU060.376.176.4-0.3
113.876.074.1+1.9
219.465.463.2+2.2
36.555.353.9+1.4
All10072.772.0+0.7
LongVideoBench045.864.270.3-6.1
125.068.957.2+11.7
226.260.655.4+5.2
33.143.943.90.0
All10063.862.3+1.5
Video-MMMU082.472.971.4+1.5
17.548.547.0+1.5
29.747.150.6-3.5
30.5†25.050.0-25.0
All10068.467.5+0.9

Share: % of questions. Δ: CIGO − Base. †: groups with less than 1% of questions, whose scores are noisy.

We make the following observations:

The model exhibits inference-time adaptive calling. Overall, CIGO answers directly for 44–82% of the examples and can call more tools on harder questions (when the base model achieves lower performance on the examples) instead of a fixed number of tools.

CIGO’s gain mainly comes from questions with tool calls. We observe that CIGO outperforms the base model on questions with at least one tool call in 5 / 6 benchmarks (+1.9 to +22.1), whereas on directly answered questions the difference varies from −7.1 (LVBench) to +3.5 (Charades-STA).

We have added this analysis in the Appendix E.1 (Table 12).

Q3. Inference cost: tokens and latency, with a reduced frame budget [ZFFf, ndw4, 51g3]

We added a full inference-cost analysis in Sec. 4.5 of the revised PDF: Fig. 4 compares accuracy against total tokens, latency and assistant turns for CIGO, the base model and five open-weight agents (Video-o3, VideoTemp-o3, ParaVT, VideoZoomer and Video-MTR) on all six benchmarks. CIGO lies on the accuracy–token Pareto frontier of the agents we compare. Below we report three representative benchmarks; all six are in Fig. 4.

Setup. We run all open-weight baselines in our codebase with their original settings and measure latency on the same hardware (one L40S GPU). Since in the default eval setting, the overview frames dominate the visual tokens, we also evaluate CIGO and the base model with 256 frames at 224², which provides an additional lower-budget setting for comparison with the baselines. All numbers below are per-question averages on three representative benchmarks.

AgentVideo-MME: acc. / tokens / sVideo-MMMU: acc. / tokens / sCharades-STA: mIoU / tokens / s
Base (no tools), reduced65.0 / 7.2k / 20.762.5 / 10.3k / 78.845.7 / 2.3k / 7.6
Base (no tools), default67.4 / 19.8k / 23.267.5 / 22.6k / 87.549.9 / 4.7k / 9.3
Video-o3-7B66.5 / 18.0k / 9.051.7 / 19.2k / 10.660.7 / 6.0k / 2.8
VideoTemp-o3-7B64.5 / 22.7k / 23.253.2 / 34.4k / 37.657.8 / 8.3k / 4.7
CIGO, reduced65.1 / 8.8k / 29.862.4 / 10.7k / 66.654.9 / 3.3k / 12.1
CIGO, default68.1 / 21.5k / 35.868.4 / 23.0k / 75.557.5 / 5.6k / 13.5

Default: the evaluation setting of Sec. 4.1 (1,024 frames at 224² for Video-MME; 256 frames at 384² for Video-MMMU and Charades-STA). Reduced: 256 frames at 224². Tokens: total processed tokens per question (input and generated). Latency: seconds per question on one L40S GPU, batch size 1.

  • Accuracy–token trade-off. CIGO lies on the accuracy–token frontier on all three benchmarks. At the default budget, it is the most accurate agent on Video-MME and Video-MMMU while adding only 0.4k–1.7k tokens per question over its base model; with the reduced budget, it uses 1.8–3.2× fewer tokens than Video-o3 and VideoTemp-o3.
  • Latency. CIGO's latency is higher because it generates long reasoning, inherited from the Thinking base model, while most existing baselines are post-trained on instruct (e.g. Qwen2.5-VL-7B-Instruct) model. Relative to its base model, CIGO adds 12.6 s on Video-MME and 4.2 s on Charades-STA, and is 12.0 s faster on Video-MMMU.

Q4. Why the necessity analysis shows little separation on LVBench [ZFFf, ndw4, 51g3]

The "unnecessary" group combined two kinds of questions that call for opposite behavior. Split by whether the overview suffices, CIGO calls 1.33x more tools when the information is insufficient than when information is sufficient.

In the original definition, we define a question as “necessary” when adding ground-truth evidence increases reference-model success rate: SR₁ > SR₀. The original “unnecessary” label (SR₁ ≤ SR₀) therefore mixes 2 groups: (a) questions the overview already answers, and (b) questions that even adding the ground-truth frames cannot reliably solve. In the latter case, it is still reasonable to call the tool, since the existing information is insufficient for answering the question.

To address this, we add another axis (information sufficiency, measured by whether with only the video overview, the reference model reaches a success rate of at least κ = 0.75), and partition the datasets into 3 groups: SR₀ = success rate of the reference model with the video alone, SR₁ = with the ground-truth window added, each measured over 16 rollouts:

  1. Evidence helps (insufficient): SR₁ > SR₀.
  2. Overview suffices (sufficient): SR₁ ≤ SR₀ and SR₀ ≥ κ.
  3. Neither suffices (insufficient): SR₁ ≤ SR₀ and SR₀ < κ.

And we compute the adaptive tool allocation ratio = mean calls on insufficient questions (Evidence helps + Neither suffices) / mean calls on sufficient questions (Overview suffices).

AgentOverview sufficesEvidence helpsNeither sufficesRatio ↑
LVBench20.5%45.5%34.0%
Video-o30.931.021.021.10×
VideoTemp-o31.381.411.381.01×
ParaVT2.522.813.061.16×
CIGO1.031.301.461.33×
ReXTime58.0%27.0%15.0%
Video-o30.941.021.001.07×
VideoTemp-o30.160.180.121.00×
ParaVT4.314.134.440.98×
CIGO0.090.490.234.41×

We observe that CIGO calls 1.33x (LVBench) and 4.41× (ReXTime) more tools when the tool is helpful or the information is insufficient than when information is sufficient and the tool is unhelpful.

Q5. How the information potential Φ is computed [ZFFf, ndw4, 51g3]

Φ is the log-likelihood that the frozen reference model assigns to the ground-truth answer, restricted to valid answer tokens. We have added the full definition we use in Sec. 3.3, Eq. (4).

In practice, we instantiate Φ(s)\Phi(s) using length-normalized token probabilities that are restricted to some valid option set C\mathcal{C} according to the specific task:

\begin{cases} \log \bar{\pi}_{\mathcal{O}}(y^{\ast} \mid s), & \text{MCQ},\\\\[6pt] \dfrac{1}{\lvert\mathcal{D}\rvert} \displaystyle\sum_{j \in \mathcal{D}} \log \bar{\pi}_{\mathcal{V}}\big(y^{\ast}_j \mid s,\\, y^{\ast}_{\lt j}\big), & \text{TG}, \end{cases} \qquad \bar{\pi}_{\mathcal{C}}(c \mid \cdot) = \frac{\pi(c \mid \cdot)}{\sum_{c' \in \mathcal{C}} \pi(c' \mid \cdot)}.$$ For MCQ, $y^{\ast}$ is a single option token from the set of valid options $\mathcal{O}$. For TG, $y^{\ast}$ is the ground-truth interval written as "The event happens in $T_s$ - $T_e$ seconds." $\mathcal{D}$ includes only the whole-second digit positions of $T_s$ and $T_e$, and $\mathcal{V} = \\{0, \dots, 9\\}$. ### Q6. Does scoring Φ before the reasoning (post-retrieval) lead to similar improvements? [ndw4, 51g3] We implement the CIGO variant that assigns information reward post-retrieval before further reasoning. We only normalize within the LOOK branches, and STOP branches by default receive 0 information advantage. In this formulation, the LOOK over STOP then comes only from the counterfactual task reward, while the counterfactual information gain teaches where to look. We observe that gain still persists if we score Φ post-retrieval: this variant scores higher than post-rationale on all six benchmarks (+0.4 to +1.6) but also increases the average number of tool calls (it answers directly on only 12.5% of questions, while post-rationale answers 65.6% of questions directly). We argue that measuring evidence-only information gain is another effective instantiation for our proposed training framework while the default post-rationale variant uses the tool more selectively. We add this post-retrieval variant to the paper (App. D.2, Table 10). | Φ scored | Video-MME | MLVU | LVBench | LongVideoBench | Video-MMMU | Charades-STA | Tool calls at evaluation (%): 0 / 1 / 2 / 3 | |-|-|-|-|-|-|-|-| | Base model | 67.4 | 72.0 | 45.1 | 62.3 | 67.5 | 49.9 | 100 / – / – / – | | Post-rationale | 68.1 | 72.7 | 47.9 | 63.8 | 68.4 | 57.5 | 65.6 / 13.6 / 17.3 / 3.6 | | Post-retrieval | **68.9** | **73.1** | **48.7** | **64.4** | **69.6** | **59.1** | 12.5 / 85.1 / 1.2 / 1.3 |

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.