acceptodds
Under review as a conference paper at ICLR 2027

Calling Is Not Looking: Entropy-Resolved Evidence Acquisition for Video Agents

Abstract

MLLM-based video agents call tools to retrieve missing visual evidence, but calling a tool does not guarantee that the returned evidence is actually or properly used. We propose Entropy-Resolved Evidence Acquisition (ERA), which tracks the agent's top high-entropy tokens, where it is most uncertain how to continue, and locates the commitment point at which this uncertainty resolves. The position of this point relative to the tool calls reveals two failure modes of current recipes: agents trained with tool-calling rewards commit to an answer before the tool returns and overlook the evidence, while agents cold-started under LLM-as-a-Judge supervision answer after any return and overrely on it, even when the return is uninformative. ERA turns this diagnosis into a turn-level reward that scores where the agent commits and contrasts its reasoning under an uninformative and a decisive return. On tool-required questions from four video benchmarks, we swap the first return for an uninformative one and assert that it is exactly what the agent requested. Supervised fine-tuning on counterfactual re-seek demonstrations teaches the agent to re-seek, but this behavior collapses under such assertions, whereas RL with the ERA reward keeps re-seek tied to what the returned frames show, lowers the hallucination rate, and preserves accuracy under pressure. The reward also improves an existing agent, VideoZoomer. These results suggest that where an agent's high-entropy tokens fall relative to its tool calls reveals whether it uses the evidence it acquires.

Primary area:
reinforcement learning
Keywords:
Multimodal Large Language Models, Video Agents, Tool Use, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UxypOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether video agents actually use the visual evidence returned by tools. It defines a commitment point based on a decline in the proportion of high-entropy reasoning tokens, distinguishing between agents that commit before receiving tool results and agents that commit after uninformative results. The proposed ERA framework combines a commitment-location reward with a counterfactual contrastive reward to encourage further retrieval when evidence is insufficient. Experiments compare different SFT and RL recipes across video benchmarks under both ordinary tool returns and interventions involving substituted returns and assertive messages.

Critical strengths

The paper addresses an important problem with a useful diagnostic perspective. Accuracy and tool-call frequency alone do not establish whether an agent responds to acquired evidence. Connecting the timing of commitment with counterfactual tool returns exposes distinct failure modes that conventional evaluations can miss. This is a meaningful contribution to understanding tool use in video agents.

The improvements under intervention are supported by both behavioral measures and answer accuracy. For example, on VideoNet-Gen, accuracy under the pressure protocol increases from 0.31 for SFT-Mix to 0.50 with ERA. Results on VideoZoomer also provide evidence of transfer beyond a single initialization. The separation of SFT and RL further illustrates that learning the form of re-seeking does not necessarily produce evidence-dependent behavior.

Critical weaknesses

The training objective and a central evaluation metric share the same proxy definition, creating a risk of circular validation. Entropy changes are used both to construct the reward and to classify behaviors such as overreliance. The policy could therefore learn the desired entropy pattern without a corresponding improvement in evidence use. Counterfactual answer accuracy provides some independent support, but further validation is needed to establish that the commitment point reliably measures evidence use across models and training conditions.

The strongest gains are concentrated in a specific artificial intervention protocol. With real tool returns, ERA improves accuracy over the strongest same-backbone training baseline by approximately one percentage point. Larger gains primarily arise when returns are substituted and accompanied by assertive messages. Together with model-dependent certification of tool-required questions and relatively small evaluation subsets, this limits the evidence for broader improvements in video reasoning and naturally occurring tool failures.

Questions for authors

Do you have validation independent of the entropy-based definition, such as human annotations of evidence use, changes in answer distributions following evidence substitution, or agreement between detected commitment points and actual changes in the agent’s answer?

Could you report separate results for substituted returns without assertive messages and assertive messages without substituted returns? Do the gains persist under naturally occurring retrieval failures or on unfiltered benchmark splits? These results would materially affect my assessment of the method’s generality.

3 comments

We apologize for the late reply: a reviewer's question on overlap led to the frame-budget experiments below, which took longer than expected. We hope you will take another look at our work.

Q1. Independent validation and clarification of the commitment point

Thank you for raising this concern. Our description should clarify the detection procedure and its use in behavioral classification; the classification is then checked by behavior that uses no entropy. A trajectory with one tool call follows:

<think>reasoning 1</think><tool_call>...</tool_call>
[returned frames]
<think>reasoning 2</think><answer>...</answer>
  • Detection. We concatenate the tokens within all nn <think>...</think> blocks, identify the top 20% by entropy across this combined sequence, and calculate their proportion within reasoning spans. Local drops can occur around tool calls; a typical post-return commitment shows a more pronounced drop in the following block, with the rate remaining low afterwards. Equation 1 formalizes this sustained change and detects τ⋆\tau^\star.
  • Classification. Equation 2 uses the block containing τ⋆\tau^\star: commitment before the first call is overlook, after a call but before any decisive return is overrely, and after a decisive return is evidence-resolved. Multiple-call trajectories follow the same rules.
  • Answer changes under substitution. Substituting tool returns in 62 DeepEyes-7B trajectories leaves answers unchanged for 97% of questions (Section 4.3), supporting the overlook diagnosis. ERA instead seeks again after 40% of uninformative and 14% of decisive first returns (Table 4).
  • Evidence actually reached. At 4 frames per seek, ERA's returned frames include one inside E⋆E^\star for 71.7% of questions, against 34.3% for the base model, which never seeks again.

We will clarify the interaction format, detection procedure, and turn-level classification in the main text.

Q2. Separate interventions, natural retrieval failures, and unfiltered questions

The gain persists under each intervention alone, under natural retrieval failures, and on unfiltered questions.

Table R1. VideoNet-Gen (100 certified questions) and official VideoNet (4,000 unfiltered questions), in %. Mean ± std over training seeds 0, 1, 2 (Qwen3.5-9B, Base in the text: three inference runs). ERA is SFT-Mix + ERA; ERA-RL applies the ERA reward to Base without SFT. RS: share that seeks again. Acc. P / Acc. N: first return substituted, with assertive / plain messages; Acc. P (real): real returns with assertive messages.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
                VideoNet-Gen                                                                            Official VideoNet
                ─────────────────────────────────────────────────────────────────────────────────────   ─────────────────
Model           Acc.     Acc.     Acc.     Acc.     RS       RS       Acc. P   Acc. N   Acc. P (real)   Acc.     Acc.
Frames per seek 4        6        8        16       4        8        4        4        4               4        8
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Qwen3.5-9B      21.7±3.5 27.0±2.6 51.0±2.0 20.0±2.6 0        0        2.3±2.1  5.7±2.9  18.3±2.9        34.6±2.4 33.5±2.0
SFT-Mix + ERA   41.7±4.5 46.0±1.7 53.0±1.7 44.7±2.5 45.3±3.1 30.0±4.0 38.7±1.2 53.7±3.1 39.7±1.5        46.9±3.7 46.3±4.2
ERA-RL (no SFT) 36.3±1.2 39.3±1.2 46.3±1.5 –        57.0±2.0 39.3±2.5 34.3±2.1 37.3±1.5 41.3±3.1        42.4±2.9 45.9±2.5
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  • Separate interventions. With substitution and plain messages (Acc. N), ERA reaches 53.7% against 5.7% for Base; with real returns and assertive messages, 39.7% against 18.3%. The message alone costs little on real returns (2.0 points for ERA, 3.4 for Base); substitution is what breaks Base.
  • Natural failures. At the original 8 frames, Base's first return contains an evidence frame for 82.7% of questions, so natural failures are rare; at 4 frames this drops to 34.3%. Without any intervention, Base never seeks again and falls from 51.0% to 27.0% (6 frames) and 21.7% (4), while ERA seeks again more (30.0% → 32.7% → 45.3%) and keeps 46.0% and 41.7%. At 16 frames both drop (20.0%, 44.7%), as the longer input makes the agent stop early.
  • Unfiltered questions. On official VideoNet, ERA leads Base by about 12 points at both budgets and seeks again on only 2.7–4.2% of questions, since most need no extra evidence, unlike the certified tool-required sets.

The gain is therefore not tied to the combined pressure protocol, and it is largest when a single return is insufficient.

One part of Q1 remains open for me. Could you report, on the same set of trajectories, how often the answer changes after substituting the tool return, broken down by the original entropy-based classification? In particular, are trajectories classified as evidence-resolved more sensitive to evidence substitution than those classified as overlook?

Dear Reviewer Uxyp,

Thank you for the follow-up. We ran the comparison you describe on one set of trajectories per model, with one substitution rule and the original entropy-based labels.

  • Trajectories. Qwen3.5-9B (three inference runs), SFT-Mix + ERA, and ERA-RL (no SFT) (training seeds 0, 1, 2) on VideoNet-Gen (100 questions), with 8 frames per seek, real returns, and plain messages. Each trajectory keeps its original label from Eq. 2, whether its answer was correct or not.
  • Substitution. We keep each trajectory fixed up to its last tool return and continue it twice with greedy decoding, once with the original return and once with a segment outside E⋆E^\star. The answer changes if the two continuations end in different options.
  • Must answer. In both continuations, the message with the return asks for an answer in this turn, so the agent cannot recover by seeking again.
  • Free. No instruction is added. The agent may seek again and then receives the segments it requests.

Table R2. Substituting the last tool return, by original classification. Changed: pairs whose two continuations end in different options / valid pairs, in which both continuations give a final option. Seek again: share of substituted continuations that call the tool again. Overrely requires a return without the evidence, which is rare with real returns, so it is not compared. Trajectories with no tool call or no detected commitment point are also left out.

──────────────────────────────────────────────────────────────────────────────────────
                                    Must answer         Free
                                    ─────────────────   ──────────────────────────────
Model            Classification     Changed             Seek again   Changed
──────────────────────────────────────────────────────────────────────────────────────
Qwen3.5-9B       Overlook           9/161 (5.6%)        0.0%         10/162 (6.2%)
                 Evidence-resolved  46/113 (40.7%)      0.0%         47/115 (40.9%)
SFT-Mix + ERA    Overlook           5/60 (8.3%)         11.3%        4/61 (6.6%)
                 Evidence-resolved  104/197 (52.8%)     47.7%        61/203 (30.0%)
ERA-RL (no SFT)  Overlook           8/73 (11.0%)        19.2%        7/69 (10.1%)
                 Evidence-resolved  70/154 (45.5%)      55.1%        39/141 (27.7%)
──────────────────────────────────────────────────────────────────────────────────────
  • Answers. When SFT-Mix + ERA must answer, the answer changes for 52.8% of its evidence-resolved trajectories and 8.3% of its overlook trajectories, a gap of 44.5 points.
  • Before and after ERA training. Qwen3.5-9B is never trained with an entropy reward, and its gap is already 35.1 points (40.7% vs 5.6%). The gap remains at 34.5 points after ERA-RL and 44.5 after SFT-Mix + ERA, so training on the entropy reward does not close it.
  • Actions. When free, Qwen3.5-9B never seeks again, and its change rates stay the same (40.9% and 6.2%). The evidence-resolved trajectories of SFT-Mix + ERA seek again after 47.7% of substituted returns, against 11.3% for its overlook trajectories, and their change rate falls from 52.8% to 30.0%.

In all three models, trajectories classified as evidence-resolved are therefore more sensitive to evidence substitution than those classified as overlook. We will add this analysis to Section 4.3.

We hope this resolves the open part of Q1, and we would appreciate any further feedback. Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_peBKOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether MLLM-based video agents actually use the visual evidence returned by their tool calls, rather than merely issuing calls. The authors propose ERA (Entropy-Resolved Evidence Acquisition), built on a behavioral statistic derived from the agent's own token entropies. Reasoning-token entropies are binarized into an entropy trace (the top-ρ fraction marked high); a constrained maximum-likelihood change-point split (Eq. 1) locates the commitment point τ*, where the high-entropy token rate durably drops. The position of τ* relative to the first call and the first decisive return classifies a trajectory as overlook (committed before the call), overrely (committed on a non-decisive return), or evidence-resolved (committed only after a decisive return).

Critical strengths

A judge-free, causally validated measure of evidence use. Locating when the agent stopped being uncertain — rather than scoring what it wrote about the return — is a novel and elegant operationalization. The validation is interventional, not correlational: DeepEyes-7B's answer is invariant to the return content for 97% of questions, and replacing its pre-call reasoning restores return-dependence (0.45 vs. 0.18 accuracy), which strongly supports that τ* tracks commitment rather than text style. The overlook/overrely diagnosis of two opposite failure modes of standard recipes is, in my view, the most valuable contribution.  The training design closes the loopholes it exposes, and robustness is tested beyond the training condition. The counterfactual environment with an identical assertive message after every first return directly targets the overrely failure; the contrastive branch cancels return-independent uncertainty; and the reward-shaping discussion (Appendix A) addresses plausible gaming strategies. Evaluation applies up to three assertive messages while training uses one, so the robustness is not mere exposure; the SFT-vs-RL separation (Table 4) cleanly shows SFT teaches the form of re-seeking while the entropy-change reward ties it to the returned content; and VideoZoomer+ERA indicates the reward transfers to another agent.

Critical weaknesses

Headline gains rest on small, partly self-constructed evaluation sets without significance evidence. With real returns — the deployment condition — ERA improves accuracy by only 0.01 on every benchmark (Table 3). On 100/400/84/169 questions, differences of this size are plausibly noise, yet no seeds, confidence intervals, or significance tests are reported for Tables 3–4, and several ablation rows are explicitly single-seed. The in-domain benchmark (VideoNet-Gen) is generated by Gemini and filtered by the authors, i.e., matched to their own certification pipeline, and the large gains appear only under the authors' pressure protocol, which closely mirrors the training intervention. The abstract's claim of "significant performance gains on four video reasoning benchmarks" is therefore stronger than the evidence supports.

Applicability and robustness of the commitment-point machinery are underexamined. The location term and the certification of tool-required questions require ground-truth evidence windows E* during training, available only where temporal annotations exist and defined only relative to the certifying model; the paper neither tests degradation under noisy E* nor discusses scaling to domains without such annotations. The change-point detector has at least five hyperparameters (ρ, κ, λ, m, L) plus p, η, c, w, with no reported sensitivity, and the τ*-based behavior taxonomy is validated only indirectly (via interventions), not against human judgments of commitment. Finally, the external baselines (DeepEyes-7B, VideoZoomer) run on different backbones with different tool formats, so cross-family comparisons in Tables 3–4 are partly confounded.

Questions for authors
  1. Please provide variance estimates for Tables 3–4 (multiple training seeds or bootstrap over test items). Is the +0.01 real-return accuracy edge of ERA over Outcome-RL/MM-Judge statistically meaningful at n = 100/84/169? How many seeds back the main-table rows?

  2. How sensitive are the detected commitment points and the final results to ρ, κ, λ, m, and L? On a small human-labeled sample, how often does τ* coincide with a human's judgment of when the agent settled on its answer?

  3. How does ERA behave when E* annotations are noisy or partially wrong at training time? And is any variant of the reward feasible when no evidence windows exist (e.g., relying on the contrastive term only)?

  4. Under pressure, SFT-Mix+ERA answers 0.93 and ERA-RL only 0.62 of questions within T_max (Table 4). What is the added turn/call and wall-clock cost of re-seeking, and how would the answer-rate drop affect deployment?

  5. Since swapped-return augmentation alone could explain part of the robustness gain, please report a control trained with the same counterfactual augmentation and outcome reward only (no entropy-based terms) — this would isolate the contribution of the entropy-change reward.

1 comment

We apologize for the late reply: a reviewer's question on overlap led to the frame-budget experiments below, which took longer than expected. We hope you will take another look at our work.

Table R1. VideoNet-Gen (100 certified questions), in % except Calls/q. Mean ± std over training seeds 0, 1, 2 (Qwen3.5-9B: three inference runs). Acc. P: first return substituted, with assertive messages. Calls/q: successful tool calls per question. Overlook is decided before the first call, so frames do not apply. In the text, Base, ERA, and Out are rows 1–3.

─────────────────────────────────────────────────────────────────────────────────────
Model                 Acc.     Acc.     Acc.     Acc.     Acc. P   Calls/q   Overlook
Frames per seek       4        6        8        16       4        8         N/A
─────────────────────────────────────────────────────────────────────────────────────
Qwen3.5-9B            21.7±3.5 27.0±2.6 51.0±2.0 20.0±2.6 2.3±2.1  1         54.0±3.6
SFT-Mix + ERA         41.7±4.5 46.0±1.7 53.0±1.7 44.7±2.5 38.7±1.2 1.82±0.06 22.3±3.5
Outcome only (no SFT) 16.7±3.5 25.0±1.7 35.7±1.5 –        19.3±1.5 1.97±0.06 55.0±1.0
ERA-RL (no SFT)       36.3±1.2 39.3±1.2 46.3±1.5 –        34.3±2.1 1.90±0.18 29.0±2.6
─────────────────────────────────────────────────────────────────────────────────────

Q1. The +0.01 edge, variance, and seeds

The +0.01 edge is not significant, and we do not claim it; one seek suffices at 8 frames.

  • Significance. On 100 questions it is one question, below ERA's seed std (1.7). We will revise "significant performance gains" in the abstract.
  • Variance. ERA leads Base by 20.0, 19.0, and 24.7 points at 4, 6, and 16 frames, far beyond the std. On the unfiltered official VideoNet (4,000 questions, 8 frames), ERA reaches 46.3±4.2% vs 33.5±2.0%; on Video-MME (4 frames), 24.0±1.8% vs 10.3±2.1%.
  • Seeds. Three training seeds per trained row.

ERA's benefit is recovery from an insufficient return, which the 8-frame setting hides.

Q2. Sensitivity of the commitment point and human agreement

Eq. 2 only picks a reasoning block, so a hyperparameter matters only if it moves τ⋆\tau^\star across a tool return.

  • Sensitivity. Recomputing labels on 1,200 saved trajectories, one change at a time, leaves 93–100% unchanged for κ\kappa, 81–100% for λ\lambda, 96–100% for mm, and 100% for LL. ρ\rho matters most (66–83%), as it defines high-entropy tokens; we fix ρ=0.2\rho=0.2 from prior work, without per-model tuning.
  • Human agreement. Annotators read the written reasoning, which Judge-SFT shows can describe evidence use that did not happen. We instead intervene on returns: DeepEyes-7B keeps 97% of answers under four different returns, and ERA's re-seeking follows the return (40% vs 14%, Table 4).

The labels are robust to the detector parameters; we will add this analysis.

Q3. Noisy or missing evidence windows

E⋆E^\star is a training-time reference, like the answer label; the policy never uses it at inference and transfers without windows (Q1).

  • Noise. We verify windows instead of corrupting them: a question is kept only if Qwen3.5-9B answers it in at most 1 of 4 samples from the coarse frames and at least 3 of 4 with frames from E⋆E^\star.
  • Contrastive term only. In Table 5 (w/o location term), accuracy under pressure falls from 50% to 41% and overlook rises from 52% to 77%; this variant still needs E⋆E^\star.

Both terms are needed; window-free training remains open, as the limitations will state.

Q4. Cost of re-seeking and the answer rate

Re-seeking costs under one extra call per question, and the low answer rate belongs to an ablation.

  • Cost. ERA makes 1.82 calls per question (Base: 1), the fewest among RL variants, and re-seeks on only 2.7–4.2% of unfiltered VideoNet questions. Table 5's 0.83 used 1,024 generated tokens per turn, which often cuts reasoning before the call; Table R1 uses 8,192. We will add wall-clock time.
  • Answer rate. ERA-RL (no SFT) is an ablation that often seeks until the turn limit (0.62); SFT supplies the ability to stop. The proposed model answers 0.93 under pressure vs 0.97 for SFT-Mix.

ERA trades 4 points of answer rate for 19 points of accuracy under pressure (31% → 50%).

Q5. Same augmentation with an outcome reward only

The paper has this control, and we add one without SFT; neither rewards evidence use.

  • With SFT. The + Outcome rows of Table 3 train in ERA's environment without entropy terms: 34–36% under pressure vs 50% for ERA.
  • Without SFT (Out). It keeps Base's overlook share (55.0% vs 54.0%) and reaches 19.3% under pressure vs 34.3% for ERA-RL.
  • Why. The coarse frames give partial information, so the agent commits before calling; a substituted return cannot mislead an agent that ignores it.

The augmentation explains 3–5 points of the gain under pressure; the entropy terms add 14–16.

Reviewer_t3ksOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether video agents actually use the evidence returned by their tools. ERA estimates a commitment point from a drop in the rate of high-entropy reasoning tokens, then uses its position relative to tool returns to diagnose different failure modes. It combines a commitment-location reward with a contrastive reward computed from informative and uninformative returns. Experiments compare SFT and RL recipes under normal returns and a pressure protocol involving substituted returns and assertive messages.

Critical strengths

The distinction between calling a tool and using its output is useful. The examples where both agents answer correctly but respond differently to missing evidence show why accuracy and call counts alone are insufficient.

Also, the gains under the pressure protocol are substantial. On VideoNet-Gen, accuracy increases from 0.31 for SFT-Mix to 0.50 with ERA. The SFT/RL comparison and the VideoZoomer experiment also help establish that the results are not limited to one initialization.

Critical weaknesses
  1. The DeepEyes interventions support the diagnosis that the agent often ignores tool returns, but do not directly establish how accurately the detected point locates commitment. This matters because the same entropy-based definition is used in both the training reward and the reported overreliance metric. A policy could improve that metric by changing its entropy pattern. The accuracy gains provide independent support, but stronger validation of the commitment point is still needed, particularly after ERA training.

  2. The experiments leave uncertainty about what produces the robustness gains and how broadly they transfer. Training already includes substituted returns and assertive messages, which are also central to the pressure evaluation. It should be explicit whether the outcome-reward baselines receive the same interventions and comparable training budgets. A baseline using the same evidence-window supervision to reward appropriate re-seeking, without entropy terms, would further isolate ERA's contribution. With normal returns, the advantage over the strongest other same-backbone recipe is only about one percentage point on each benchmark, without uncertainty estimates. The current results therefore support robustness under the tested intervention more clearly than a general improvement in video reasoning.

Questions for authors
  1. Please validate the commitment point by substituting evidence at different reasoning positions and measuring whether the final answer can still change. This would be especially useful for the ERA-trained policy.
  2. Do the outcome-reward baselines use the same substituted returns and assertive messages during RL? Please clarify this and report a matched baseline that uses evidence-window supervision without entropy rewards.
  3. Do the gains persist with different pressure wording or naturally occurring retrieval failures? Please also provide uncertainty estimates for the main accuracy comparisons.
  4. Section 3.1 treats overlap with the evidence window as a decisive return. Since the tool returns sparsely sampled frames, how often does an overlapping segment actually contain the frames needed to answer?
1 comment

We apologize for the late reply: your question on overlap led to the frame-budget experiments below, which took longer than expected. We hope you will take another look at our work.

Table R1. VideoNet-Gen (100 certified questions), in %. Mean ± std over training seeds 0, 1, 2 (Qwen3.5-9B: three inference runs). Frame ov.: share of questions with a returned frame inside E⋆E^\star. Acc. P / Acc. N: first return substituted, with assertive / plain messages. In the text, Base, ERA, Win, and Out are the first four rows.

───────────────────────────────────────────────────────────────────────────────────────────────────────────
Model                             Acc.     Acc.     Acc.     Acc.     Frame ov. Frame ov. Acc. P   Acc. N
Frames per seek                   4        6        8        16       4         8         4        4
───────────────────────────────────────────────────────────────────────────────────────────────────────────
Qwen3.5-9B                        21.7±3.5 27.0±2.6 51.0±2.0 20.0±2.6 34.3±1.5  82.7±2.9  2.3±2.1  5.7±2.9
SFT-Mix + ERA                     41.7±4.5 46.0±1.7 53.0±1.7 44.7±2.5 71.7±3.2  82.3±2.3  38.7±1.2 53.7±3.1
Outcome + window re-seek (no SFT) 29.0±5.3 23.7±7.8 40.0±4.0 –        –         –         20.3±5.5 20.7±7.4
Outcome only (no SFT)             16.7±3.5 25.0±1.7 35.7±1.5 –        –         –         19.3±1.5 31.7±1.2
ERA-RL (no SFT)                   36.3±1.2 39.3±1.2 46.3±1.5 –        –         –         34.3±2.1 37.3±1.5
───────────────────────────────────────────────────────────────────────────────────────────────────────────

Q1. Validating the commitment point by evidence substitution

Evidence enters only as returned frames between reasoning blocks, so substituting the first return probes the commitment point directly:

<think>reasoning 1</think><tool_call>...</tool_call>
[returned frames]
<think>reasoning 2</think><answer>...</answer>
  • Labels. Eq. 2 labels the block containing τ⋆\tau^\star: overlook (reasoning 1), overrely (after a call, before a decisive return), or evidence-resolved.
  • Substitution. Unless the agent overlooks, the substituted return arrives before the commitment and tests overreliance; overlook is penalized, never rewarded.
  • ERA-trained policy. Under substitution, ERA seeks again after 40% of uninformative and 14% of decisive first returns (Table 4), counted from actions, not entropy.

We will add this format and labeling rule to the main text.

Q2. Outcome baselines and a matched baseline

Yes, and the matched baseline shows that supervising actions alone does not make the agent use the returns.

  • Same environment. Both + Outcome rows start from SFT-Mix, substitute the first return with probability 0.5, add the same assertive message, and train for as many rounds as ERA.
  • Matched baseline (Win). Trained from Base, it is rewarded for re-seeking after an uninformative return and charged after a decisive one, as decided by E⋆E^\star, without entropy.
  • Result. Win re-seeks equally at 4 and 8 frames (44.7%, 43.7%), unlike ERA-RL (57.0%, 39.3%), and trails ERA-RL by 6–7 points (14 under pressure). Out keeps Base's overlook share (55.0% vs 54.0%).

Action supervision rewards calling; the entropy terms reward using the return.

Q3. Pressure wording, natural failures, and uncertainty

The gain does not need the assertive message and grows under natural retrieval failures.

  • Wording. Without the message (Acc. N), ERA reaches 53.7% vs 5.7% for Base; with it, 38.7% vs 2.3%, since it suppresses re-seeking (63% → 40%, Table 4).
  • Natural failures. Without intervention, Base never re-seeks and falls from 51.0% (8 frames) to 27.0% (6) and 21.7% (4); ERA re-seeks more (30.0% → 32.7% → 45.3%) and keeps 46.0% and 41.7%.
  • 16 frames. Both drop (20.0%, 44.7%) as the longer input makes the agent stop early; ERA still leads by 24.7 points.
  • Uncertainty. These 19–25-point gaps far exceed the std (≤4.5); at 8 frames the gap is about one std.

ERA's benefit is recovery from an insufficient return.

Q4. How often an overlapping segment contains the needed frames

Thank you; this question started the study above. A returned frame inside E⋆E^\star implies segment overlap, so Frame ov. lower-bounds the rate you ask for.

  • 8 frames. Base, which never re-seeks, gets an evidence frame in its first return for 82.7% of questions: the segment-level label holds in the paper's setting, and one seek usually suffices, hence the 0.01 edge.
  • 4 frames. This drops to 34.3%, while ERA re-seeks and reaches one for 71.7%.

Missed evidence at low budgets is a natural retrieval failure that ERA recovers from by seeking again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.