acceptodds
Under review as a conference paper at ICLR 2027

IGSD: Environment-Verified Hindsight Self-Distillation for Search Agents

Abstract

On-policy self-distillation densifies agent training without external teachers: a policy conditioned on privileged hindsight provides step-level guidance for its own unprivileged rollouts. For search agents, however, hindsight can make the teacher prefer a query that does not improve retrieval from the student's state. Existing methods either distill this preference directly or filter it with model-internal scores, but neither strategy verifies the query's executed retrieval consequence. We propose Information-Gain-Gated Self-Distillation (IGSD), which verifies on-policy token proposals with environment feedback before distilling them. Treating each query token as a micro action, IGSD completes the teacher's token proposal and the student's sampled token into matched queries and executes both from the same failed state with the same retriever. Shared random-document controls account for query-conditioned shifts in answer likelihood, so their difference, the executed paired information gain, provides a relative utility contrast for the retrieved documents. IGSD uses this contrast as a positive-only soft weight for candidate-pair distillation, while leaving the GRPO objective unchanged and confining verification to training. Across seven single-hop and multi-hop QA benchmarks, IGSD reaches macro-average exact-match accuracies of 42.8% and 47.0% with 3B and 7B policies, respectively, without inference-time verification or additional search cost. These results support environment-verified hindsight as an effective approach to reliable action-level supervision for search agents.

Primary area:
reinforcement learning
Keywords:
search agents, retrieval-augmented generation, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aq9FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy hindsight self-distillation (HSD) for search agents. The authors argue that a query preferred by a hindsight-conditioned teacher may not retrieve useful evidence from the student’s actual state. They propose IGSD: On the last search turn of a failed trajectory, IGSD completes each query-token disagreement between the hindsight teacher and the student into two queries and executes both. It uses the difference in how much each branch’s retrieved documents raise the gold-answer likelihood, net of a random-document control, as a positive-only soft gate. Only teacher tokens verified as more useful are distilled, and the GRPO objective and inference stay unchanged.

Critical strengths
  1. The gap between a hindsight teacher’s preference and the action’s actual utility from the student’s state is a real weakness of HSD methods. Case B (Freud) and the newspaper case illustrate redundant or mis-anchored teacher proposals well.
  2. The two branches share the state, prefix, completion policy, decoding, and retriever, and differ only in one token, which makes attribution clean. Using shared random documents to control for query-phrasing effects on answer likelihood is a sensible consideration.
  3. Verification is training-only. Tables 10–11 show inference-time search calls nearly identical to Search-R1. Training cost is 1.25×, below SD-Search’s 2.18×.
Critical weaknesses
  1. “Environment-verified” overstates the contribution, the signal fundamentally depends on the gold answer. The core of ΔIG=log⁡π(a∣h,Q,D)\Delta IG=\log\pi(a|h,Q,D), where a is the reference answer. The environment only returns documents; whether they are “useful” is judged by gold-answer likelihood.
  2. The evaluation is limited. Only on QA benchmarks, some other search-based benchmarks should be used, such as deep-research like benchmark
  3. The models used in the experiments are outdated. The author should at least use Qwen3 family and test on some other model families to evaluate the generalization
  4. Which baselines were reproduced locally in SearchAgent-Zero, and which numbers are copied from the original papers? Do IGPO, GiGPO, CRAFT, and LOTAPO use the same retriever, corpus, number of steps, and number of rollouts?
  5. At 7B, only Search-R1 and SD-Search are compared. The closest method, IGPO, is missing.
  6. All methods are trained for 400 steps. IGSD uses 1.25× the compute; was Search-R1 or any other baseline given equivalent extra compute? Figure 2 partially addresses this, but only for Search-R1.
Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer aq9F [1/4]

We thank the reviewer for the careful reading and for the constructive suggestions. We are glad to address the points raised below.

W1: "Environment-verified" overstates the contribution, the signal fundamentally depends on the gold answer. The environment only returns documents; whether they are "useful" is judged by gold-answer likelihood.

We thank the reviewer for raising this point. We justify that "environment-verified" is appropriate for our method.

(1) The term fits our setting. In search agents, the environment is the information source that returns documents in response to a query [1, 2, 3]. Verification, in turn, means executing an action and judging its consequence against a reference. In RLVR, for example, the reference answer judges the model's output; similarly, in code agents, predefined unit tests judge the output of an executed program. Following this pattern, IGSD executes each candidate query in the environment, and the reference answer judges the returned evidence. Moreover, this answer already defines the GRPO reward, so IGSD adds no supervision beyond training and none at inference.

(2) Execution produces the gate signal; the answer alone does not. The answer only judges evidence, so it cannot replace the evidence obtained by executing the query. IGSD executes both candidate queries from the same state and measures the information gain of the evidence the environment returns. SD-Search [4] and SDAR [5] do not execute the proposed query: one distills the hindsight preference directly, and the other filters it by a model-internal score. Execution with only answer string matching does not yield this signal either, because string matching cannot tell whether the retrieved documents actually support the answer. The table below compares these strategies.

Table 1. Verification strategies compared. "Executes query" indicates whether the candidate query is executed for verification.

VariantWhat it doesExecutes queryAvg.
Search-R1No hindsight teacher or verificationNo0.394
Answer coverage gate (Appendix E.3)Scores by string matching to the answerYes0.409
IGSDComputes the evidence's information gainYes0.428

W2: The evaluation is limited. Only on QA benchmarks, some other search-based benchmarks should be used, such as deep-research like benchmark

We thank the reviewer for suggesting broader evaluation. Our main evaluation follows the standard setting of recent work such as Search-R1 [1], SD-Search [4], and Search-E1 [8], and is therefore not limited to a custom setup. Nevertheless, we added BrowseComp-Plus [6], a deep-research benchmark derived from BrowseComp. We choose BrowseComp-Plus because it provides a stable, fixed corpus that makes evaluation and comparison across methods straightforward. We train Qwen3-8B on the ASearcher [7] training data for 100 steps, with a train batch size of 64, a maximum response length of 36,864 tokens, on 8 H200 GPUs. Evaluation uses Qwen3-Embedding-8B as the retriever with a search budget of 100 turns, following the official BrowseComp-Plus protocol. Full training and evaluation settings of our method will be added to the revised paper.

Table 2. BrowseComp-Plus results with Qwen3-8B trained on ASearcher data, under the official evaluation setting.

MethodAccuracyEvidence recallSearch calls
Search-R121.08%29.15%7.82
IGSD w/o Gate21.69%29.07%8.05
IGSD22.65%31.82%8.61

IGSD gains 1.57 points over Search-R1 and 0.96 points over the ungated control. Evidence recall rises from 29.15% to 31.82%, a 2.67-point gain, which suggests that verified queries retrieve more of the evidence needed to answer. Execution-verified distillation therefore transfers to deep-research.

AuthorsOpenReview ↗

Response to Reviewer aq9F [2/4]

W3: The models used in the experiments are outdated. The author should at least use Qwen3 family and test on some other model families to evaluate the generalization

We thank the reviewer for this suggestion. We keep the Qwen2.5 family as the main setting because it is the standard configuration in this line of work. Early work such as Search-R1 [1] and recent work such as SD-Search [4], Search-E1 [8], IGPO [9], IG-Search [10] and SmartSearch [11] all use the Qwen2.5 family, so replacing it would break comparability with our baselines. To test whether the method generalizes beyond this family, we add experiments on Qwen3-4B-Instruct-2507 and Llama-3.2-3B-Instruct, covering both a newer Qwen generation and a different model family.

Table 3. Exact match on the seven benchmarks of Section 4.1. Single-hop is the average of NQ, TriviaQA and PopQA. Multi-hop is the average of HotpotQA, 2WikiMultiHopQA, MuSiQue and Bamboogle. Qwen3-4B is trained for 400 steps and Llama-3.2-3B for 300 steps. All other settings, including the training data, retriever and optimization hyperparameters, are identical to those used for Qwen2.5-3B.

ModelMethodSingle-hopMulti-hopAvg.
Qwen2.5-3BSearch-R10.4840.3260.394
IGSD w/o Gate0.5090.3480.417
IGSD0.5130.3630.428
Qwen2.5-7BSearch-R10.5350.3780.445
IGSD w/o Gate0.5390.3890.453
IGSD0.5430.4160.470
Qwen3-4B-Instruct-2507Search-R10.5280.4140.463
IGSD w/o Gate0.5380.4170.469
IGSD0.5460.4250.477
Llama-3.2-3B-InstructSearch-R10.5240.3460.422
IGSD w/o Gate0.5230.3540.426
IGSD0.5270.3690.437

IGSD improves over both Search-R1 and the ungated control on Qwen3-4B and Llama-3.2-3B. On Qwen3-4B it reaches 0.477, above 0.469 for the ungated control and 0.463 for Search-R1. On Llama-3.2-3B it reaches 0.437, above 0.426 and 0.422. The gains hold on both single-hop and multi-hop benchmarks. The method therefore generalizes beyond the Qwen2.5 family used in our main experiments.

We also evaluate on FRAMES [12] to test generalization to out-of-distribution questions, which requires composing evidence across multiple Wikipedia articles per question. We use the December 2018 Wikipedia dump as the corpus, E5-base-v2 as the retriever with a search budget of 10 turns, and Qwen3-32B as the judge, following the official protocol. The detailed evaluation protocol will be added to the revised paper.

Table 4. FRAMES accuracy with the December 2018 Wikipedia corpus and Qwen3-32B as judge. Search calls are the average number of search invocations per question.

ModelMethodFRAMES AccSearch calls
Qwen3-4B-Instruct-2507Search-R123.18%3.282
IGSD w/o Gate24.27%3.449
IGSD25.00%3.399
Llama-3.2-3BSearch-R118.08%3.106
IGSD w/o Gate20.05%3.059
IGSD20.75%3.108

IGSD also improves on FRAMES. On Qwen3-4B it reaches 25.00%, above 24.27% for the ungated control and 23.18% for Search-R1. On Llama-3.2-3B it reaches 20.75%, above 20.05% and 18.08%. The gains come with comparable search usage, around 3.4 calls per question. This shows that the method also generalizes to out-of-distribution questions.

AuthorsOpenReview ↗

Response to Reviewer aq9F [3/4]

W4: Which baselines were reproduced locally in SearchAgent-Zero, and which numbers are copied from the original papers? Do IGPO, GiGPO, CRAFT, and LOTAPO use the same retriever, corpus, number of steps, and number of rollouts?

Direct Generation, Vanilla RAG and Search-o1 are taken from the Search-R1 paper, whose evaluation setting matches ours. All other baselines are reproduced locally under the same training setting: the same number of steps, learning rate, reward, retriever, corpus and number of rollouts. We also reimplemented Search-R1 on the newer veRL framework (version 0.8.0.dev) [13], since the older version caused training instability. Under this matched setup, Search-R1 itself is a strong baseline that matches several process-level methods.

W5: At 7B, only Search-R1 and SD-Search are compared. The closest method, IGPO, is missing.

We thank the reviewer for pointing this out. We have added IGPO [9] at 7B.

Table 5. Results with Qwen2.5-7B-Instruct, including the added IGPO baseline.

MethodNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAvg.
Search-R10.4770.6470.4800.4520.4350.2020.4240.445
SD-Search0.4920.6520.4690.4720.4380.2280.4400.456
IGPO0.4900.6420.4660.4610.4300.2230.4400.450
IGSD w/o Gate0.4890.6550.4720.4650.4360.2160.4400.453
IGSD0.4960.6560.4760.4780.4890.2320.4640.470

IGPO reaches 0.450 at 7B, below IGSD at 0.470 and SD-Search at 0.456. IGSD remains the strongest method at this scale.

W6: All methods are trained for 400 steps. IGSD uses 1.25× the compute; was Search-R1 or any other baseline given equivalent extra compute?

As shown in Figure 2, Search-R1 plateaus after roughly 200 steps. We therefore trained compute-matched variants of Search-R1 and the ungated control, giving them comparable training time to IGSD. We also measured the per-step cost of IGSD and its ungated control on 8 H200 GPUs. The results are in Table 6.

Table 6. Compute-matched comparison with the Qwen2.5-3B-Instruct policy. Relative to IGSD is the wall-clock training time relative to IGSD at 400 steps. Search-R1 at 500 steps and the ungated control at 450 steps receive comparable training time to IGSD.

MethodStepsRelative to IGSDAvg.
Search-R14000.80x0.394
Search-R1, compute-matched5001.00x0.392
IGSD w/o Gate4000.90x0.417
IGSD w/o Gate, compute-matched4501.01x0.419
SD-Search4001.74x0.415
IGSD4001.00x0.428

The compute-matched Search-R1 reaches 0.392, so additional compute does not improve it. The ungated control at 400 steps reaches 0.417, and at 450 steps it reaches 0.419, a slight increase of 0.002 that remains within its fluctuation range.

Verification accounts for only 9.9% of the step time, and the rest comes from the teacher forward pass and the distillation loss, which IGSD shares with the ungated control. IGSD is also more affordable than SD-Search, which costs 1.74 times our training time and reaches 0.415. Giving the baselines more compute therefore does not close the gap.

Reviewer_tbTDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Information-Gain-Gated Self-Distillation (IGSD), a training-time mechanism for improving hindsight self-distillation in search agents. Previous hindsight self-distillation methods may introduce unreliable teacher guidance, as the teacher's preferred queries do not necessarily improve retrieval from the student's current state. IGSD executes both teacher-induced and student-induced queries through the same retriever and estimates their relative utility using paired information gain. This signal is then used to gate an auxiliary distillation loss alongside the GRPO objective, without introducing additional inference-time verification overhead. IGSD achieves the best performance among the evaluated methods across seven QA benchmarks, with ablations showing improvements of 1.1–1.7 percentage points over the ungated variant.

Critical strengths
  1. The paper identifies an important reliability issue in hindsight self-distillation and proposes an intuitive and technically coherent mechanism to filter unreliable teacher supervision. The method is well motivated, and the experiments include systematic ablation studies supporting the contribution of its individual components.
Critical weaknesses
  1. Comparison with simpler retrieval-quality signals. The proposed executed information gain estimates retrieval utility using the frozen policy's reference-answer likelihood. Although the authors demonstrate improvements over model-internal gating signals, it is unclear whether simpler retrieval-quality measures, such as answer coverage or passage relevance, could achieve comparable results with lower computational overhead. Such comparisons would help establish the necessity of the proposed information-gain formulation.

  2. Relatively modest gains from verification. Although IGSD improves over its ungated variant, the verification mechanism contributes relatively modest gains of 1.1 and 1.7 percentage points for the 3B and 7B models, respectively. Given the additional training-time computation required for paired verification, further analysis of its cost-effectiveness would strengthen the empirical contribution.

Questions for authors
  1. The proposed executed information gain uses the frozen policy's reference-answer likelihood to estimate retrieval utility. Have the authors considered simpler retrieval-based gating signals, such as answer coverage or passage relevance? How would these alternatives compare in terms of downstream performance and computational cost?
5 comments
AuthorsOpenReview ↗

Response to Reviewer tbTD [1/2]

We thank the reviewer for the careful reading and for the positive assessment of our work. We are glad to address the points raised below.

W1: it is unclear whether simpler retrieval-quality measures, such as answer coverage or passage relevance, could achieve comparable results with lower computational overhead

We compare ΔIG against four alternative signals. Answer coverage records whether the retrieved documents contain the reference answer string. Passage relevance uses E5 question-passage similarity, and BM25 scores the retrieved documents against the query. The log-probability gap is a model-internal signal. We include it for comparison. Each signal is computed as a paired difference, like ΔIG. We then use each signal in place of ΔIG in Eq. 6. Pairs with a score at or below zero get zero weight, as in the default gate, and every other setting follows Section 4.1.

Table 1. Comparison of gate signals with the Qwen2.5-3B-Instruct policy under the recipe of Section 4.1. AUC is the terminal-winner ROC-AUC on the audit pairs of Section 4.3, where 0.5 is the random baseline. Each signal is computed as a paired difference and then used in place of ΔIG in Eq. 6. The No gate row uses no gate signal and is included as the reference.

Gate signalAUCAvg.
ΔIG (IGSD)0.7080.428
Answer coverage0.6020.409
Teacher-student log-probability gap0.5520.409
BM25 over retrieved documents0.5210.391
E5 question-passage relevance0.5300.402
No gaten/a0.417

Table 1 supports three conclusions.

(1) ΔIG outperforms all alternatives. Passage relevance and BM25 stay close to chance at 0.530 and 0.521. Answer coverage is higher at 0.602, but it comes from a small fraction of pairs. On 88% of pairs the two branches retrieve the same number of documents containing the answer string, so coverage gives them an identical score. Its AUC therefore comes almost entirely from the remaining 12%, and it provides no signal for the other 88%.

(2) All alternatives hurt final accuracy. Answer coverage reaches 0.409, the model-internal log-probability gap 0.409, E5 relevance 0.402 and BM25 0.391, against 0.417 for the ungated control and 0.428 for IGSD.

(3) The computational saving is small. A simpler signal removes only the scoring stage, which accounts for 10.6s of 134.7s per step, or 7.9% (Table 2 below).

AuthorsOpenReview ↗

Response to Reviewer tbTD [2/2]

W2: Relatively modest gains from verification. Although IGSD improves over its ungated variant, the verification mechanism contributes relatively modest gains of 1.1 and 1.7 percentage points for the 3B and 7B models, respectively. Given the additional training-time computation required for paired verification, further analysis of its cost-effectiveness would strengthen the empirical contribution.

We measured the time of each training step of IGSD and its ungated control on 8 H200 GPUs, with all other settings identical to Section 4.1. The ungated control has the same teacher forward pass and distillation loss. The difference between the two rows is therefore the cost of verification.

Table 2. Per-step training time in seconds on H200. The three rows under Verification break down the 13.4s it adds.

StageSearch-R1IGSD w/o GateIGSD
Rollout (include tool calls)71.870.870.6
Log-prob, actor update, weight sync36.540.440.5
Hindsight teacher and distillation scoringn/a9.910.2
Verificationn/an/a13.4
↳ IG log-probability forward9.0
↳ Branch completion and retrieval2.8
↳ Counterfactual construction1.6
Total108.3121.1134.7
Relative to the ungated controln/a1.00x1.11x

Table 2 gives three results.

(1) Verification costs 9.9% of a step. IGSD therefore costs 1.11 times the w/o gate control and 1.24 times Search-R1. IGSD is therefore more affordable than SD-Search, which costs 2.18 times Search-R1 and reaches 0.416, below IGSD at 0.427.

(2) Verification contributes 45% of the gain. Over three seeds, IGSD reaches 0.427, the ungated control 0.414, and Search-R1 0.398. Verification therefore contributes 0.013 of the 0.029 total gain.

(3) Verification also stabilizes training. As shown in Figure 2, the ungated control fluctuates across the schedule, since unverified hindsight supervision helps or hurts depending on the current state. IGSD instead improves across the full schedule.

Q1: Have the authors considered simpler retrieval-based gating signals, such as answer coverage or passage relevance? How would these alternatives compare in terms of downstream performance and computational cost?

Please see w1 and Table 1 for the detailed results. In summary:

  • All simpler signals underperform ΔIG. The log-probability gap reaches AUC 0.552, E5 relevance 0.530 and BM25 0.521, against 0.708 for ΔIG. Answer coverage is higher at 0.602, but on 88% of pairs the two branches retrieve the same number of documents containing the answer string, so answer coverage gives them an identical score. Its AUC therefore comes almost entirely from the remaining 12%.
  • All simpler signals hurt final accuracy. Downstream accuracy drops to 0.409 for answer coverage and the log-probability gap, 0.402 for E5 relevance and 0.391 for BM25, all below 0.417 for the ungated control and 0.428 for IGSD.
  • The cost saving is small. Replacing ΔIG with a simpler signal saves 7.9% of per-step time.

We thank the reviewer again for these concrete suggestions. We have added these results to the revised PDF, and we welcome further questions from the reviewer.

AuthorsOpenReview ↗

Response to Reviewer tbTD

Dear Reviewer, thank you again for your thoughtful feedback. As the discussion period is coming to a close, we would appreciate your thoughts on whether our responses have addressed your concerns. Please let us know if any further clarification would be helpful.

Reviewer_b9BWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

IGSD augments outcome-based reinforcement learning with hindsight self-distillation for search agents. A successful sibling rollout supplies a query-based hint to a frozen self-teacher. At disagreement positions within a failed trajectory's last search query, teacher and sampled student tokens are completed into alternative queries using the same student policy. The method executes both queries and compares reference-answer likelihood gains relative to shared counterfactual documents. Positive gains weight candidate-pair Jensen-Shannon distillation. Evaluation includes seven QA benchmarks, two model scales, component ablations, repeated runs, and training-cost measurements.

Critical strengths
  1. The paired construction is carefully controlled: both branches share the student state, continuation policy, and retriever. Ablations of the sampled-token branch and counterfactual control directly test important parts of this construction.

  2. The paper provides a relatively comprehensive empirical package. In addition to benchmark improvements, it reports repeated-run results, inference search usage, training time, and an audit connecting the proposed signal to downstream terminal outcomes.

Critical weaknesses
  1. The verification signal remains partly model-dependent. Executing retrieval grounds the comparison in actual documents, but document utility is measured through the same frozen policy's reference-answer likelihood. This may reflect answer compatibility or lexical cues rather than reliable evidence support. The distinction matters for interpreting environment-verified supervision and its expected robustness.

  2. The mechanistic audit is suggestive but limited: the terminal-winner AUC uses 72 discordant pairs from 70 questions, and the confidence intervals are fairly wide. Furthermore, the non-positive-gain group still contains more teacher-only successes than student-only successes. Thus, the audit supports enrichment of useful proposals, but does not fully explain why zero is the best rejection threshold.

Questions for authors
  1. Does the gain signal remain predictive when utility is scored by a different frozen model, or when passages containing answer strings without supporting evidence are used as controls?

  2. How does the final result change under a mildly negative acceptance threshold or a smoother weighting rule that retains some non-positive-gain pairs? A larger, independently sampled audit would also help establish the reliability of the signal comparison.

4 comments
AuthorsOpenReview ↗

Response to Reviewer b9BW [1/3]

We thank the reviewer for the careful reading and for the positive assessment of our work. We are glad to address the points raised below.

W1: The verification signal remains partly model-dependent. Executing retrieval grounds the comparison in actual documents, but document utility is measured through the same frozen policy's reference-answer likelihood. This may reflect answer compatibility or lexical cues rather than reliable evidence support.

We thank the reviewer for raising this concern. We use the policy to score the retrieved documents for two reasons. First, it has been trained in this environment, so it reads retrieved evidence more reliably and is familiar with the tool-response and <answer> format of our trajectories. Second, adding another frozen model would keep a second set of weights in GPU memory for all of training.

To test these concerns, we recompute ΔIG on the audit pairs of Section 4.3 and report its terminal-winner ROC-AUC under two changes. First, we replace the scorer with three untrained models. Second, we replace the random control with passages that contain the answer string without supporting it.

  • Scorer (the frozen model computing the answer log-likelihood of Eq. 4): the rollout policy, as in our default setting, and three off-the-shelf models never trained in our pipeline.
  • Counterfactual control: three settings. Random CF is the default of Section 3.3, using passages drawn at random from the corpus. Answer-bearing CF uses corpus passages containing the answer string that rank lowest by retriever similarity. Injected CF inserts the answer string at a random position into randomly drawn passages.

Table 1. Terminal-winner ROC-AUC of executed paired information gain under four scorers and three counterfactual-control constructions. Only the scorer and the control construction change. An AUC of 0.5 is the random baseline. For reference, the strongest model-internal signal in this setting reaches 0.552 (Figure 3c).

Scorer in Eq. 4Random CF (default)Answer-bearing CFInjected CF
Policy (default)0.7080.7110.702
Qwen2.5-3B-Instruct (untrained)0.5680.5760.560
Qwen2.5-7B-Instruct (untrained)0.6850.7240.717
Qwen3-4B-instruct-2507 (untrained)0.6170.5980.604

Table 1 gives two results.

(1) The signal generalizes beyond the policy. An untrained Qwen2.5-7B reaches 0.685, and among untrained scorers the AUC rises with scale (3B 0.568, 4B 0.617, 7B 0.685). The policy reaches 0.708 under the default control, the highest in that setting, because it has been trained on this exact format.

(2) ΔIG carries more information than the answer string. Using answer-bearing controls changes the AUC only slightly. Eq. 5 subtracts the same control from both branches. Any contribution shared by the two branches therefore cancels. What remains is the difference between the evidence they retrieved.

We also compare ΔIG against a purely lexical signal on the same pairs.

Answer coverage cannot rank most pairs. We score each pair by answer coverage, the difference in the number of retrieved documents containing the answer string. It reaches AUC 0.602, but on 88% of pairs the two branches give the same count. Coverage therefore leaves 88% of pairs without any signal, and its AUC comes almost entirely from the remaining 12%. ΔIG instead separates all pairs by how much the retrieved evidence supports the answer.

AuthorsOpenReview ↗

Response to Reviewer b9BW [2/3]

W2-1: The terminal-winner AUC uses 72 discordant pairs from 70 questions, and the confidence intervals are fairly wide.

We sincerely thank the reviewer for this suggestion and have extended the audit to a larger sample. Using the same protocol as Section 4.3, we sampled an independent set of candidate pairs. This gives 200 terminal-discordant pairs from 200 different questions. Confidence intervals follow Figure 3c.

Table 2. Terminal-winner ROC-AUC with 95% question-cluster bootstrap confidence intervals, on the original audit (72 discordant pairs from 70 questions) and the enlarged independent audit (200 discordant pairs from 200 distinct questions).

SignalAUC, 72 pairs (Figure 3c)AUC, 200 pairs (new)
Executed paired information gain (ΔIG)0.708 [0.568, 0.836]0.711 [0.621, 0.787]
Teacher-student log-probability gap0.552 [0.404, 0.706]0.547 [0.456, 0.640]
Teacher preference gap0.428 [0.272, 0.579]0.435 [0.338, 0.519]

As shown in Table 2, the point estimate for executed gain moves only from 0.708 to 0.711, while the interval narrows from width 0.268 to 0.166. The separation from the log-probability gap also sharpens: the two intervals now overlap by only 0.019, about one tenth of either interval, against 0.138 on the original audit.

W2-2: The non-positive-gain group still contains more teacher-only successes than student-only successes. Thus, the audit supports enrichment of useful proposals, but does not fully explain why zero is the best rejection threshold.

We appreciate the reviewer raising this point. Zero has a clear meaning here: under the matched comparison, ΔIG > 0 means the teacher's query retrieved better evidence for the answer than the student's own query. It is the natural decision boundary of the contrast, not a tuned hyperparameter.

As the reviewer notes, teacher-only successes outnumber student-only successes when ΔIG ≤ 0. This follows from the base rate of the hindsight teacher, which wins 73.6% of all discordant pairs. The gate changes the composition: the teacher wins 85.0% of retained pairs but only 59.4% of rejected ones, close to chance. Distilling the rejected pairs would therefore add little useful supervision.

To verify this, we train three variants that differ from IGSD only in the weight assigned to each candidate pair. Candidate construction, the teacher target, and the candidate-pair JSD loss are unchanged, as is the recipe of Section 4.1.

  • ΔIG ≤ 0 (reversed gate): w = tanh(β/2 · |ΔIG|) for ΔIG ≤ 0, else 0. The exact mirror of Eq. 6.
  • ΔIG ≥ -0.2: w = max(0, tanh(β/2 · (ΔIG + 0.2))). Relaxes the threshold and gives pairs in (-0.2, 0] a smooth non-zero weight.
  • Sigmoid: w = σ(β · ΔIG). No cut-off; a pair at ΔIG = 0 gets weight 0.5 and negative-gain pairs decay smoothly toward 0.

Table 3. Gate ablation with the Qwen2.5-3B-Instruct policy under the recipe of Section 4.1. Single-hop is the average of NQ, TriviaQA and PopQA. Multi-hop is the average of HotpotQA, 2WikiMultiHopQA, MuSiQue and Bamboogle. Only the weighting of the auxiliary loss changes across rows. The rows for Binary gate, w/o Gate and Search-R1 are reproduced from Table 16 and Table 1 of the paper.

VariantSingle-hopMulti-hopAvg.
ΔIG ≤ 0 (reversed gate)0.4790.3320.395
ΔIG ≥ -0.20.5000.3600.420
Sigmoid0.5050.3580.421
Binary gate0.5060.3580.421
w/o Gate0.5090.3480.417
IGSD0.5130.3630.428
Search-R1, no distillation0.4840.3260.394

Table 3 shows three findings.

(1) Rejected pairs give no gain. The reversed gate reaches 0.395, and the ungated control 0.417, against 0.428 for the positive-only gate. Supervision drawn from the rejected group contributes little on its own. When combined with the positive-gain pairs, it lowers the overall result.

(2) Relaxing the gate does not help. A threshold at -0.2 reaches 0.420 and a sigmoid weight that retains all non-positive pairs reaches 0.421, both short of 0.428 and close to the ungated control. The gain comes from excluding the non-positive region, not from down-weighting it.

(3) Continuous weights beat binary ones. The binary gate reaches 0.421 while accepting exactly the same pairs as IGSD, so the continuous weight contributes beyond the accept-or-reject decision.

AuthorsOpenReview ↗

Response to Reviewer b9BW [3/3]

Q1: Does the gain signal remain predictive when utility is scored by a different frozen model, or when passages containing answer strings without supporting evidence are used as controls?

Please see w1 and Table 1 for the detailed results. In summary:

  • The signal generalizes beyond the policy. An untrained Qwen2.5-7B reaches 0.685, while the policy reaches 0.708, the highest. The signal is therefore not specific to the policy, and the policy is the best scorer.
  • Answer-bearing controls do not reduce the AUC. The same control is subtracted from both branches, so the comparison measures the evidence each branch retrieved.

Q2: How does the final result change under a mildly negative acceptance threshold or a smoother weighting rule that retains some non-positive-gain pairs? A larger, independently sampled audit would also help establish the reliability of the signal comparison.

Please see w2-1 and w2-2 for the detailed results. In summary:

  • Relaxing the gate does not help. A mildly negative threshold reaches 0.420 and a smooth weighting rule 0.421, both below the positive-only gate at 0.428 (Table 3). Distilling only the rejected pairs reaches 0.395.
  • The enlarged audit confirms the signal. On 200 discordant pairs from 200 questions, the point estimate moves slightly to 0.711 and the interval narrows (Table 2).

We thank the reviewer again for these concrete suggestions. We have added these results to the revised PDF, and we sincerely look forward to further questions from the reviewer.

AuthorsOpenReview ↗

Response to Reviewer b9BW

Dear Reviewer, thank you again for your valuable feedback. As the discussion period draws to a close, we would appreciate your thoughts on whether our responses have addressed your concerns. We would be happy to provide further clarification on any remaining points.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised PDF Uploaded

We thank all reviewers and the Area Chair for their careful reading and constructive suggestions. We have uploaded a revised PDF that includes additional experiments and clarifications addressing your comments. All changes are marked in blue for easy reference. We sincerely welcome any further questions or feedback.

AuthorsOpenReview ↗

Summary of Revisions and Responses

Dear Area Chair and Reviewers,

We thank you for your positive assessment and constructive feedback. Below we summarize the contribution of IGSD and the updates made during the discussion.

Contribution. Hindsight self-distillation for search agents suffers from a teacher-supervision reliability gap: the hindsight teacher often proposes queries that fail to improve retrieval from the student's actual state, and model-internal scores reflect preference rather than actual retrieval utility. IGSD addresses this with environment-verified hindsight self-distillation. It executes matched teacher and student queries from the same state, scores the retrieved evidence with executed paired information gain (ΔIG), and uses it as a positive-only soft weight on distillation, leaving GRPO unchanged with zero inference overhead.

Key updates during the discussion.

  • Signal reliability. We clarified that our verification follows standard RLVR practice, executing candidate queries and judging retrieved evidence with the reference answer. We tested alternative scorers, answer-bearing controls, and simpler signals such as answer coverage and passage relevance. ΔIG outperforms these alternatives.
  • Audit and gate design. We enlarged the terminal audit to 200 discordant pairs from distinct questions, confirming ΔIG's clear advantage over model-internal signals (0.711 vs. 0.547 AUC). We also tested relaxed thresholds, smooth weighting, and a reversed gate, and the positive-only gate remains best.
  • Cost-effectiveness. We measured per-step training cost on H200 GPUs and trained compute-matched baselines. Verification accounts for only 9.9% of a step, and additional compute does not close the gap to IGSD.
  • Generalization. While our main setting follows the standard configuration of recent related work for fair comparison, we further added evaluations on newer backbones (Qwen3-4B/8B and Llama-3.2-3B), OOD Benchmark (FRAMES), and deep research (BrowseComp-Plus). IGSD consistently outperforms the baselines across all these settings.
  • Baseline reproduction. We clarified which baselines are reproduced locally and added IGPO at 7B, where IGSD maintains a clear lead (0.470 vs. 0.450).

Reviewer tbTD has confirmed that the additional evidence adequately addressed their main concerns. All new results and clarifications have been incorporated into the revised PDF (marked in blue). We sincerely welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.