acceptodds
Under review as a conference paper at ICLR 2027

Information-Gain Calibrated On-Policy Distillation for Deep Search Agents

Abstract

On-policy distillation (OPD) has recently emerged as a promising approach for transferring the capabilities of large agents to compact students. In deep search, where agents iteratively retrieve external evidence, a teacher's preference among plausible queries before retrieval may not predict which queries will return useful evidence. Standard OPD nevertheless supervises each student query from the preceding context alone, inducing a failure mode we term evidence-blind supervision (EBS). To understand this, we regroup the OPD objective by interaction turns and reveal two implicit priors: a budget prior that allocates supervision by token volume, and a direction prior that favors tokens preferred by the teacher, neither of which accounts for the evidence returned by the current query. Our empirical analysis shows that both priors contribute to EBS in practice: the first search turn accounts for 51.1% of total absolute information gain but receives only 18.0% of the supervision budget, while 43.7% of search queries have teacher rewards that disagree in sign with post-retrieval information utility. To mitigate this, we propose \ours, which uses post-retrieval information gain (IG) to calibrate both priors: reallocating supervision toward information-critical turns while calibrating query preferences using the utility of retrieved evidence. Experiments on six multimodal search benchmarks show that \ours improves average accuracy by 3.4 and 1.7 percentage points for 2B and 4B students over standard OPD, enabling the 4B student to surpass its 8B teacher on MMSearch and BrowseComp-VL.

Primary area:
foundation or frontier models, including LLMs
Keywords:
search agents, on-policy distillation, information gain, posterior calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nppLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation (OPD) for multimodal deep-search agents and identifies a failure mode called evidence-blind supervision (EBS): the per-token OPD signal is computed only from the pre-retrieval prefix and never reflects what a query actually retrieves. Regrouping the OPD objective by interaction turn, the authors expose two implicit priors: a budget prior that allocates supervision by token length, and a direction prior that pushes tokens toward the teacher’s pre-retrieval preference. A diagnosis on MMSearch shows both are misaligned with post-retrieval information gain. The proposed IG-OPD uses IGPO-style information gain, defined as the change in the teacher’s gold-answer log-likelihood after each observation, to correct both priors. ITC reweights per-turn supervision budgets by IG magnitude, and IQC adds a signed, cosine-ramped IG correction to the action-critical tokens of each tool call; the authors interpret IQC as distillation toward an answer-tilted “power posterior” of the teacher. With an 8B OpenSearch-VL teacher and Qwen3-VL 2B/4B students, IG-OPD improves the six-benchmark average over vanilla OPD by 3.4 and 1.7 points, and the 4B student surpasses its teacher on MMSearch and BrowseComp-VL. Training overhead is about 11% and inference cost is unchanged.

Critical strengths
  1. Clear, well-motivated problem. In search, a query’s value is only known after retrieval, while the OPD signal is fixed before it.
  2. Method is lightweight and practical. IG-OPD needs only one extra teacher-forced scoring pass per turn, with no additional rollouts. Training cost rises about 11.6% in time and 10.6% in memory, inference is unchanged, and the method drops into existing OPD trainers.
  3. Realistic setting with consistent gains. Experiments use a multimodal agent with a rich tool set in a live web environment, six benchmarks, and two student scales. Gains are largest on search-intensive benchmarks (MMSearch, BrowseComp-VL), which is consistent with the paper’s hypothesis.
Critical weaknesses
  1. Most of the gain is not from IG. Equal per-turn weighting alone (λ = 0) gives +2.6, the largest single gain in Table 2, and uses no IG. With λ = 0.1, the IG term changes turn weights by at most 1.1×, and Figure 4a shows a nearly uniform budget. ITC’s benefit therefore comes mainly from length normalization, which is close to existing turn-aware OPD work.
  2. IQC is essentially an IGPO reward added to OPD. Adding αₖgₜ to the advantage makes the method an OPD+RL hybrid; at α_max = 10, the correction is about half of the OPD signal. Comparing against IGPO alone does not isolate the contribution; baselines such as OPD + tuned IG reward or OPD + outcome reward are needed.
  3. The rollout and compute settings for GRPO and IGPO are not given, and the “mean across candidate rollouts” conflicts with using one rollout per prompt. Relevant OPD variants that the paper cites (SCOPE, TIP, SOD, SAGE-OPD, TurnOPD) are not compared. Beating the author-enhanced teacher on two benchmarks by 1–2 points, without variance estimates, is weak evidence.
  4. The OOD claim is inaccurate. Training includes the FVQA train split, yet FVQA test is presented as out-of-distribution. Overlap between the other training sources and the evaluation sets should also be checked.
Questions for authors

see above

Reviewer_WfkaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates evidence-blind supervision in on-policy distillation (OPD) for multimodal search agents, where teacher preferences may not reflect the actual utility of retrieved evidence. The authors identify two limitations in standard OPD: supervision allocation based on token length and query preference based on teacher likelihoods. To address these, they propose IG-OPD, which uses post-retrieval information gain to recalibrate turn-level supervision weights (ITC) and query-level preferences (IQC). Experiments across six multimodal search benchmarks show improvements of 3.4 and 1.7 percentage points over OPD for 2B and 4B students, respectively. Additional ablations examine the contributions of individual components and the reliability of information gain.

Critical strengths
  1. Clear motivation and problem formulation. The paper identifies a meaningful limitation of OPD in search-agent training where teacher preference may not align with the usefulness of the retrieved evidence.
  2. Consistent empirical improvements. The proposed IG-OPD outperforms standard OPD across six benchmarks on average, with particularly notable improvements on search-intensive tasks. Ablations support the effectiveness of the proposed components.
Critical weaknesses
  1. Limited methodological novelty. While the problem formulation is interesting, the proposed solution primarily combines turn-level loss reweighting with an information-gain-based correction to the distillation signal. Both mechanisms are relatively straightforward and closely related to existing adaptive weighting and reward-shaping approaches. Although their integration into OPD is sensible, the methodological distinction from prior work could be more clearly established.
  2. Unclear relative contribution of information gain. The component ablation shows that uniform per-turn normalization alone improves MMSearch accuracy by 2.6 points, accounting for a substantial portion of the overall 6.3-point gain. Furthermore, the default ITC coefficient (λ=0.1)( \lambda=0.1) restricts the information-dependent variation in turn weights to a relatively narrow range. Although additional improvements are observed with IG, it remains unclear whether comparable gains could be achieved using simpler weighting or uncertainty-based signals. More direct comparisons are needed to establish the necessity of information gain.
Questions for authors
  1. Information gain is measured through changes in the teacher's confidence in the gold answer, which may not necessarily correspond to the actual utility of retrieved evidence. Although Section 5.5 examines its correlation with final-answer correctness, how sensitive is the method to teacher miscalibration or misleading evidence? Could the authors compare IG against simpler uncertainty-based or retrieval-quality signals?
Reviewer_WNthOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies OPD for multimodal search agents. It argues that OPD scores each search query only from the pre-retrieval context, which it calls EBS. Splitting OPD by turn exposes a length-based budget prior and a direction prior set by the teacher's preference. The proposed IG-OPD uses information gain on the gold answer to calibrate both. ITC reallocates the turn budget by IG magnitude, and IQC adds a signed IG bonus to tool-call tokens. With Qwen3-VL-2B/4B students and an 8B teacher, IG-OPD improves average accuracy over OPD by 3.4 and 1.7 points on six benchmarks.

Critical strengths
  1. Rewriting the OPD objective per turn is simple and grounded. It makes the budget and direction priors explicit and gives a useful lens on multi-turn OPD.
  2. Each component targets empirical diagnosis, and the paper re-measures symptoms after calibration.
  3. The analyses are informative. They include a step-by-step ablation, hyperparameter sensitivity, a check of how well IG predicts the final outcome, and a study of search suppression that motivates the cosine ramp.
Critical weaknesses
  1. IG needs the gold answer, so IG-OPD adds a label-based turn-level reward that vanilla OPD does not use during distillation. The label-using baselines are pure RL without dense teacher signal, so the key comparison is missing.
  2. The EBS diagnosis is partly circular. It defines utility as IG and then shows that OPD disagrees with IG, but a token-level imitation log-ratio is not meant to measure retrieval utility. Table 3 also shows that positive IG on later searches predicts a correct answer only 54.3% of the time, close to chance, so IG is a weak signal on exactly those turns.
  3. Much of the gain does not come from IG. In Table 2, plain uniform per-turn weighting, with no IG involved, already gives +2.6 of the total +6.3. Turn-normalized OPD should appear as a baseline in Table 1.
Questions for authors
  1. How does IG-OPD compare with OPD + outcome-reward methods and with simply adding an IGPO turn reward to the OPD advantage, all under the same label access?
  2. Given that IG on later searches is a weak predictor of the final outcome, why apply the signed IG bonus on later turns?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.