- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies on-policy hindsight self-distillation (HSD) for search agents. The authors argue that a query preferred by a hindsight-conditioned teacher may not retrieve useful evidence from the student’s actual state. They propose IGSD: On the last search turn of a failed trajectory, IGSD completes each query-token disagreement between the hindsight teacher and the student into two queries and executes both. It uses the difference in how much each branch’s retrieved documents raise the gold-answer likelihood, net of a random-document control, as a positive-only soft gate. Only teacher tokens verified as more useful are distilled, and the GRPO objective and inference stay unchanged.
Critical strengths
- The gap between a hindsight teacher’s preference and the action’s actual utility from the student’s state is a real weakness of HSD methods. Case B (Freud) and the newspaper case illustrate redundant or mis-anchored teacher proposals well.
- The two branches share the state, prefix, completion policy, decoding, and retriever, and differ only in one token, which makes attribution clean. Using shared random documents to control for query-phrasing effects on answer likelihood is a sensible consideration.
- Verification is training-only. Tables 10–11 show inference-time search calls nearly identical to Search-R1. Training cost is 1.25×, below SD-Search’s 2.18×.
Critical weaknesses
- “Environment-verified” overstates the contribution, the signal fundamentally depends on the gold answer. The core of , where a is the reference answer. The environment only returns documents; whether they are “useful” is judged by gold-answer likelihood.
- The evaluation is limited. Only on QA benchmarks, some other search-based benchmarks should be used, such as deep-research like benchmark
- The models used in the experiments are outdated. The author should at least use Qwen3 family and test on some other model families to evaluate the generalization
- Which baselines were reproduced locally in SearchAgent-Zero, and which numbers are copied from the original papers? Do IGPO, GiGPO, CRAFT, and LOTAPO use the same retriever, corpus, number of steps, and number of rollouts?
- At 7B, only Search-R1 and SD-Search are compared. The closest method, IGPO, is missing.
- All methods are trained for 400 steps. IGSD uses 1.25× the compute; was Search-R1 or any other baseline given equivalent extra compute? Figure 2 partially addresses this, but only for Search-R1.
Questions for authors
See weakness
5 comments
Response to Reviewer aq9F [2/4]
W3: The models used in the experiments are outdated. The author should at least use Qwen3 family and test on some other model families to evaluate the generalization
We thank the reviewer for this suggestion. We keep the Qwen2.5 family as the main setting because it is the standard configuration in this line of work. Early work such as Search-R1 [1] and recent work such as SD-Search [4], Search-E1 [8], IGPO [9], IG-Search [10] and SmartSearch [11] all use the Qwen2.5 family, so replacing it would break comparability with our baselines. To test whether the method generalizes beyond this family, we add experiments on Qwen3-4B-Instruct-2507 and Llama-3.2-3B-Instruct, covering both a newer Qwen generation and a different model family.
Table 3. Exact match on the seven benchmarks of Section 4.1. Single-hop is the average of NQ, TriviaQA and PopQA. Multi-hop is the average of HotpotQA, 2WikiMultiHopQA, MuSiQue and Bamboogle. Qwen3-4B is trained for 400 steps and Llama-3.2-3B for 300 steps. All other settings, including the training data, retriever and optimization hyperparameters, are identical to those used for Qwen2.5-3B.
| Model | Method | Single-hop | Multi-hop | Avg. |
|---|---|---|---|---|
| Qwen2.5-3B | Search-R1 | 0.484 | 0.326 | 0.394 |
| IGSD w/o Gate | 0.509 | 0.348 | 0.417 | |
| IGSD | 0.513 | 0.363 | 0.428 | |
| Qwen2.5-7B | Search-R1 | 0.535 | 0.378 | 0.445 |
| IGSD w/o Gate | 0.539 | 0.389 | 0.453 | |
| IGSD | 0.543 | 0.416 | 0.470 | |
| Qwen3-4B-Instruct-2507 | Search-R1 | 0.528 | 0.414 | 0.463 |
| IGSD w/o Gate | 0.538 | 0.417 | 0.469 | |
| IGSD | 0.546 | 0.425 | 0.477 | |
| Llama-3.2-3B-Instruct | Search-R1 | 0.524 | 0.346 | 0.422 |
| IGSD w/o Gate | 0.523 | 0.354 | 0.426 | |
| IGSD | 0.527 | 0.369 | 0.437 |
IGSD improves over both Search-R1 and the ungated control on Qwen3-4B and Llama-3.2-3B. On Qwen3-4B it reaches 0.477, above 0.469 for the ungated control and 0.463 for Search-R1. On Llama-3.2-3B it reaches 0.437, above 0.426 and 0.422. The gains hold on both single-hop and multi-hop benchmarks. The method therefore generalizes beyond the Qwen2.5 family used in our main experiments.
We also evaluate on FRAMES [12] to test generalization to out-of-distribution questions, which requires composing evidence across multiple Wikipedia articles per question. We use the December 2018 Wikipedia dump as the corpus, E5-base-v2 as the retriever with a search budget of 10 turns, and Qwen3-32B as the judge, following the official protocol. The detailed evaluation protocol will be added to the revised paper.
Table 4. FRAMES accuracy with the December 2018 Wikipedia corpus and Qwen3-32B as judge. Search calls are the average number of search invocations per question.
| Model | Method | FRAMES Acc | Search calls |
|---|---|---|---|
| Qwen3-4B-Instruct-2507 | Search-R1 | 23.18% | 3.282 |
| IGSD w/o Gate | 24.27% | 3.449 | |
| IGSD | 25.00% | 3.399 | |
| Llama-3.2-3B | Search-R1 | 18.08% | 3.106 |
| IGSD w/o Gate | 20.05% | 3.059 | |
| IGSD | 20.75% | 3.108 |
IGSD also improves on FRAMES. On Qwen3-4B it reaches 25.00%, above 24.27% for the ungated control and 23.18% for Search-R1. On Llama-3.2-3B it reaches 20.75%, above 20.05% and 18.08%. The gains come with comparable search usage, around 3.4 calls per question. This shows that the method also generalizes to out-of-distribution questions.
Response to Reviewer aq9F [3/4]
W4: Which baselines were reproduced locally in SearchAgent-Zero, and which numbers are copied from the original papers? Do IGPO, GiGPO, CRAFT, and LOTAPO use the same retriever, corpus, number of steps, and number of rollouts?
Direct Generation, Vanilla RAG and Search-o1 are taken from the Search-R1 paper, whose evaluation setting matches ours. All other baselines are reproduced locally under the same training setting: the same number of steps, learning rate, reward, retriever, corpus and number of rollouts. We also reimplemented Search-R1 on the newer veRL framework (version 0.8.0.dev) [13], since the older version caused training instability. Under this matched setup, Search-R1 itself is a strong baseline that matches several process-level methods.
W5: At 7B, only Search-R1 and SD-Search are compared. The closest method, IGPO, is missing.
We thank the reviewer for pointing this out. We have added IGPO [9] at 7B.
Table 5. Results with Qwen2.5-7B-Instruct, including the added IGPO baseline.
| Method | NQ | TriviaQA | PopQA | HotpotQA | 2Wiki | MuSiQue | Bamboogle | Avg. |
|---|---|---|---|---|---|---|---|---|
| Search-R1 | 0.477 | 0.647 | 0.480 | 0.452 | 0.435 | 0.202 | 0.424 | 0.445 |
| SD-Search | 0.492 | 0.652 | 0.469 | 0.472 | 0.438 | 0.228 | 0.440 | 0.456 |
| IGPO | 0.490 | 0.642 | 0.466 | 0.461 | 0.430 | 0.223 | 0.440 | 0.450 |
| IGSD w/o Gate | 0.489 | 0.655 | 0.472 | 0.465 | 0.436 | 0.216 | 0.440 | 0.453 |
| IGSD | 0.496 | 0.656 | 0.476 | 0.478 | 0.489 | 0.232 | 0.464 | 0.470 |
IGPO reaches 0.450 at 7B, below IGSD at 0.470 and SD-Search at 0.456. IGSD remains the strongest method at this scale.
W6: All methods are trained for 400 steps. IGSD uses 1.25× the compute; was Search-R1 or any other baseline given equivalent extra compute?
As shown in Figure 2, Search-R1 plateaus after roughly 200 steps. We therefore trained compute-matched variants of Search-R1 and the ungated control, giving them comparable training time to IGSD. We also measured the per-step cost of IGSD and its ungated control on 8 H200 GPUs. The results are in Table 6.
Table 6. Compute-matched comparison with the Qwen2.5-3B-Instruct policy. Relative to IGSD is the wall-clock training time relative to IGSD at 400 steps. Search-R1 at 500 steps and the ungated control at 450 steps receive comparable training time to IGSD.
| Method | Steps | Relative to IGSD | Avg. |
|---|---|---|---|
| Search-R1 | 400 | 0.80x | 0.394 |
| Search-R1, compute-matched | 500 | 1.00x | 0.392 |
| IGSD w/o Gate | 400 | 0.90x | 0.417 |
| IGSD w/o Gate, compute-matched | 450 | 1.01x | 0.419 |
| SD-Search | 400 | 1.74x | 0.415 |
| IGSD | 400 | 1.00x | 0.428 |
The compute-matched Search-R1 reaches 0.392, so additional compute does not improve it. The ungated control at 400 steps reaches 0.417, and at 450 steps it reaches 0.419, a slight increase of 0.002 that remains within its fluctuation range.
Verification accounts for only 9.9% of the step time, and the rest comes from the teacher forward pass and the distillation loss, which IGSD shares with the ungated control. IGSD is also more affordable than SD-Search, which costs 1.74 times our training time and reaches 0.415. Giving the baselines more compute therefore does not close the gap.
Response to Reviewer aq9F [1/4]
We thank the reviewer for the careful reading and for the constructive suggestions. We are glad to address the points raised below.
We thank the reviewer for raising this point. We justify that "environment-verified" is appropriate for our method.
(1) The term fits our setting. In search agents, the environment is the information source that returns documents in response to a query [1, 2, 3]. Verification, in turn, means executing an action and judging its consequence against a reference. In RLVR, for example, the reference answer judges the model's output; similarly, in code agents, predefined unit tests judge the output of an executed program. Following this pattern, IGSD executes each candidate query in the environment, and the reference answer judges the returned evidence. Moreover, this answer already defines the GRPO reward, so IGSD adds no supervision beyond training and none at inference.
(2) Execution produces the gate signal; the answer alone does not. The answer only judges evidence, so it cannot replace the evidence obtained by executing the query. IGSD executes both candidate queries from the same state and measures the information gain of the evidence the environment returns. SD-Search [4] and SDAR [5] do not execute the proposed query: one distills the hindsight preference directly, and the other filters it by a model-internal score. Execution with only answer string matching does not yield this signal either, because string matching cannot tell whether the retrieved documents actually support the answer. The table below compares these strategies.
Table 1. Verification strategies compared. "Executes query" indicates whether the candidate query is executed for verification.
We thank the reviewer for suggesting broader evaluation. Our main evaluation follows the standard setting of recent work such as Search-R1 [1], SD-Search [4], and Search-E1 [8], and is therefore not limited to a custom setup. Nevertheless, we added BrowseComp-Plus [6], a deep-research benchmark derived from BrowseComp. We choose BrowseComp-Plus because it provides a stable, fixed corpus that makes evaluation and comparison across methods straightforward. We train Qwen3-8B on the ASearcher [7] training data for 100 steps, with a train batch size of 64, a maximum response length of 36,864 tokens, on 8 H200 GPUs. Evaluation uses Qwen3-Embedding-8B as the retriever with a search budget of 100 turns, following the official BrowseComp-Plus protocol. Full training and evaluation settings of our method will be added to the revised paper.
Table 2. BrowseComp-Plus results with Qwen3-8B trained on ASearcher data, under the official evaluation setting.
IGSD gains 1.57 points over Search-R1 and 0.96 points over the ungated control. Evidence recall rises from 29.15% to 31.82%, a 2.67-point gain, which suggests that verified queries retrieve more of the evidence needed to answer. Execution-verified distillation therefore transfers to deep-research.