- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies reward design for RL-trained search agents, where outcome rewards give the same credit to an answer grounded in retrieved evidence and one recalled from parametric knowledge after unnecessary searches. The authors propose Search-G1, which trains two lightweight readouts on the policy's hidden states, one predicting whether the policy can answer without retrieval and the other predicting whether a searched answer changes when the evidence is removed. These scores form a reward that scales the credit for evidence use by how much retrieval is needed, rewards direct answers otherwise, and penalizes repeated searches, and both readouts are refit periodically as the policy changes. With Qwen2.5-3B/7B-Instruct on four QA benchmarks and ten trained baselines, Search-G1 reports the best NQ EM (0.453/0.512), along with the best grounding and search-count scores, which are measured on NQ only.
Critical strengths
- Careful experimental setup. The paper uses four seeds, disjoint calibration, validation and test splits, ten trained baselines, and hierarchical bootstrap tests with Holm correction (Table A1). It also adds a human evaluation of answer support (Tables A10 and A18).
Critical weaknesses
-
The advantages over existing methods are not supported. The significance tests cover NQ only (Table A1), and there the significant EM gains are only against Search-R1, an adapted version with a fixed direct-answer bonus and the lowest NQ EM among the trained baselines. Against the strongest baseline per metric, EM is not significant at either scale, which the authors also acknowledge in Appendix A.2.1. D-Inv, which the authors treat as the primary grounding metric, improves by only 0.004 at 3B (not significant) and 0.005 at 7B, and the search reduction is not significant at 3B. The one metric that is significant at both scales, TC, gives full credit to correct direct answers, which is exactly what the Search-G1 reward encourages. In the human evaluation on 600 shared NQ questions, Search-G1 is on par with the GPT-5 Judge baseline (0.360 vs. 0.355).
-
The value of the readouts is not shown. The main idea of Search-G1 is to approximate two intervention signals, closed-book correctness and evidence-deletion sensitivity, with readouts on hidden states. However, the paper never compares this with using the intervention labels directly as rewards, and Appendix A.2.1 argues for the readouts through lower decoding cost and graded scores without testing them against the labels. The ablations in Table 3 remove reward terms or the refresh, the shuffled-gate control (Table 3) shows that the necessity score must be paired with the right question, and the permutation control (Table A19) shows that reliance credit tracks human-judged support. None of these show that a readout works better than the label it approximates. Given that the gains over the strongest baselines are marginal, the added complexity of periodic target regeneration and refitting is not justified.
Questions for authors
- How does Search-G1 compare with rewarding the closed-book and evidence-deletion labels directly, without the readouts?
- How does Search-G1 compare with the strongest baselines (Search-E1, KbPO, SAAS) under the unified accounting of Table A14, which currently includes only Prompted Search and Search-R1?
- Why is Search/Q on 3B 2Wiki about 45% lower for all three methods in Table A14 than in Table 2 (1.580 vs. 2.812 for Search-G1)?