acceptodds
Under review as a conference paper at ICLR 2027

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search

Abstract

Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Yet terminal correctness assigns equal credit to direct answers from parametric knowledge, evidence-grounded answers, and answers recalled after redundant search. Process supervision and LLM judges provide richer feedback but can require high-cost annotation or inference; low-cost, non-contrastive policy confidence signals do not directly identify whether retrieved evidence changes an answer. We propose Search-G1, a reward framework that couples retrieval necessity with evidence reliance through two intervention-calibrated hidden-state readouts for selective grounded search. A prompt readout predicts closed-book sufficiency, whose complement proxies policy-relative retrieval necessity; an answer-commit readout predicts evidence reliance as sensitivity to evidence deletion conditional on the realized reasoning prefix. A correctness-first reward gates evidence-reliance credit by retrieval necessity, rewards sufficient direct answers, and penalizes repeated search. After calibration, scoring needs neither process annotations nor LLM-judge inference. As policy-relative targets, trajectory distributions, and representations may drift, Search-G1 periodically regenerates targets and refits its readouts. Across four QA benchmarks and two scales, Search-G1 achieves competitive task accuracy, with stronger counterfactual evidence dependence and fewer searches in the main comparisons, complemented by independent human audits of answer support.

Primary area:
reinforcement learning
Keywords:
Search-Augmented Language Agents, Representation-Based Intrinsic Rewards, GRPO, Retrieval Necessity, Evidence Reliance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_pULROpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies reward design for RL-trained search agents, where outcome rewards give the same credit to an answer grounded in retrieved evidence and one recalled from parametric knowledge after unnecessary searches. The authors propose Search-G1, which trains two lightweight readouts on the policy's hidden states, one predicting whether the policy can answer without retrieval and the other predicting whether a searched answer changes when the evidence is removed. These scores form a reward that scales the credit for evidence use by how much retrieval is needed, rewards direct answers otherwise, and penalizes repeated searches, and both readouts are refit periodically as the policy changes. With Qwen2.5-3B/7B-Instruct on four QA benchmarks and ten trained baselines, Search-G1 reports the best NQ EM (0.453/0.512), along with the best grounding and search-count scores, which are measured on NQ only.

Critical strengths
  • Careful experimental setup. The paper uses four seeds, disjoint calibration, validation and test splits, ten trained baselines, and hierarchical bootstrap tests with Holm correction (Table A1). It also adds a human evaluation of answer support (Tables A10 and A18).
Critical weaknesses
  • The advantages over existing methods are not supported. The significance tests cover NQ only (Table A1), and there the significant EM gains are only against Search-R1, an adapted version with a fixed direct-answer bonus and the lowest NQ EM among the trained baselines. Against the strongest baseline per metric, EM is not significant at either scale, which the authors also acknowledge in Appendix A.2.1. D-Inv, which the authors treat as the primary grounding metric, improves by only 0.004 at 3B (not significant) and 0.005 at 7B, and the search reduction is not significant at 3B. The one metric that is significant at both scales, TC, gives full credit to correct direct answers, which is exactly what the Search-G1 reward encourages. In the human evaluation on 600 shared NQ questions, Search-G1 is on par with the GPT-5 Judge baseline (0.360 vs. 0.355).

  • The value of the readouts is not shown. The main idea of Search-G1 is to approximate two intervention signals, closed-book correctness and evidence-deletion sensitivity, with readouts on hidden states. However, the paper never compares this with using the intervention labels directly as rewards, and Appendix A.2.1 argues for the readouts through lower decoding cost and graded scores without testing them against the labels. The ablations in Table 3 remove reward terms or the refresh, the shuffled-gate control (Table 3) shows that the necessity score must be paired with the right question, and the permutation control (Table A19) shows that reliance credit tracks human-judged support. None of these show that a readout works better than the label it approximates. Given that the gains over the strongest baselines are marginal, the added complexity of periodic target regeneration and refitting is not justified.

Questions for authors
  • How does Search-G1 compare with rewarding the closed-book and evidence-deletion labels directly, without the readouts?
  • How does Search-G1 compare with the strongest baselines (Search-E1, KbPO, SAAS) under the unified accounting of Table A14, which currently includes only Prompted Search and Search-R1?
  • Why is Search/Q on 3B 2Wiki about 45% lower for all three methods in Table A14 than in Table 2 (1.580 vs. 2.812 for Search-G1)?
Reviewer_qyNROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reward design for search-augmented agents. Retrieval should be used when the current policy cannot answer without it, and correct searched answers should depend on retrieved evidence. Search-G1 fits two hidden-state readouts to intervention-derived labels: closed-book answer sufficiency and answer sensitivity to deleting retrieved observations while retaining the realized reasoning prefix. A correctness-first reward combines necessity-gated evidence-reliance credit, a sufficiency-dependent direct-answer bonus, and a bounded repeated-search penalty. The evaluator snapshot and readouts are periodically refreshed during masked GRPO training. Experiments cover four QA benchmarks and Qwen2.5-3B/7B, with component ablations, intervention controls, and human-support audits. The results support competitive accuracy and improved search/grounding diagnostics in the evaluated setting.

Critical strengths
  1. A coherent reward design that distinguishes useful retrieval from merely correct answers. Equations (4)–(7) couple necessity and reliance while retaining correct-over-incorrect reward ordering. Proposition 1 explicitly identifies the homogeneous-group case in which a positive gate cannot change credit direction, rather than assuming that every reward modification survives GRPO normalization. Table 3 provides relevant downstream controls, removing the searched gate reduces 2Wiki EM from 0.368 to 0.347 at 3B and from 0.475 to 0.444 at 7B, while worsening TC and increasing search count. The design and analysis together offer useful guidance for agent reward construction.

  2. An extensive evaluation that goes beyond a single learned or perturbation-based metric. The two-scale, four-benchmark results are complemented by decoding-mismatch controls, stricter deletion, surface-feature controls, and independent human annotation (Tables A6, A10, A15–A19). The reported NQ comparisons against Search-R1 improve all four endpoints in Table A1, while the authors explicitly acknowledge that EM superiority over the strongest per-metric comparator is not established. Table A18 further checks support on common question IDs: Search-G1's UGA is 0.360, close to the GPT-5 Judge comparator's 0.355. This triangulation supports the method's utility without requiring a claim of universal grounding superiority.

Critical weaknesses
  1. Table 3 removes reward components or refresh, but does not compare the same necessity-gated objective using the intervention scores directly or a suitable contrastive likelihood scorer. Related mechanisms are discussed in Sections 2.1–2.2, yet the comparison does not establish the quality–cost trade-off attributable specifically to Search-G1's readouts. Appendix A.2.1 gives decode-count estimates and explicitly matches on-policy exposure rather than total compute; the extra frozen-snapshot forward pass and refresh costs are not summarized in measured training-time comparisons. This limits the strength of the contribution.

  2. Equation (7) depends on the numerical values of both readouts, not just their rankings. However, architecture/layer selection and most diagnostics use AUC. Logistic-regression candidates use class balancing, and Table A17 reports score distributions on the calibration slice. Good AUC does not establish probability calibration or stable direct-versus-search reward preferences under refitting. Closed-book inclusion labels differ from exact-match labels on 6.90% of NQ examples, so calibration should first be assessed against the declared inclusion target, with its mismatch to EM kept separate. The ablations support the chosen implementation, but do not establish that its operating boundary is robust to score calibration. Held-out reliability diagnostics and a controlled recalibration check would strengthen the mechanistic interpretation.

Questions for authors
  1. Can you provide a comparison that keeps the necessity gate, direct bonus, cost term, data, and update budget fixed, while replacing the readouts with directly computed intervention scores or one justified contrastive scoring alternative?

  2. Which readout architectures are actually selected at representative refits, and what are their Brier scores or reliability curves on disjoint questions? Under held-out probability recalibration, how often do the direct-versus-search ordering in Equation (7) and the within-group advantage signs change?

Reviewer_bfbFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Search-G1, a reinforcement learning framework designed to improve selective and grounded retrieval in LLM-based search agents. The central motivation is that conventional outcome-based rewards cannot distinguish between whether retrieval is actually necessary and whether the agent genuinely relies on retrieved evidence when producing an answer.To address this issue, the authors introduce two policy-dependent signals: Retrieval Necessity, estimated through closed-book answer sufficiency, and Evidence Reliance, estimated through counterfactual evidence deletion. Two hidden-state readout models are trained to predict these signals, and their outputs are combined into a necessity-gated reward. The resulting reward is incorporated into GRPO, with periodic recalibration of the readout models to accommodate policy distribution shifts. The framework also includes a direct-answer bonus and a penalty for repeated searches.The method is evaluated using Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct on four QA benchmarks: NQ, HotpotQA, 2WikiMultiHopQA, and MuSiQue. The experiments include comparisons with multiple retrieval-oriented RL baselines, component ablations, representation probing, counterfactual robustness checks, cross-dataset transfer, and human evaluations of evidence support.The results are promising: Search-G1 achieves competitive or improved answer accuracy, reduces search frequency, and obtains better grounding-related metrics in many settings. The authors also provide extensive analyses of the proposed readouts and reward components. Overall, the paper presents a coherent and carefully implemented framework.

Critical strengths

1. Clear motivation and a coherent end-to-end framework.

I appreciate the paper's central distinction between whether an agent needs to retrieve information and whether it actually uses the retrieved evidence. This is a meaningful problem in retrieval-augmented agents, and the authors translate it into a relatively intuitive framework. The overall pipeline, from counterfactual supervision to hidden-state readouts and necessity-gated policy optimization, is clearly organized. Even though I have reservations about the validity of some components, the research motivation itself is easy to understand and well communicated.

2. Extensive empirical investigation and meaningful engineering results.

The paper contains a substantial experimental effort, including four QA benchmarks, two model scales, multiple baselines, component ablations, robustness analyses, and human evaluations. I've read appendix in detail and particularly appreciate the attempt to evaluate evidence support independently of the proposed readout models, as shown in the human annotation experiments. The results suggest that Search-G1 can improve the practical behavior of retrieval agents under the evaluated settings. The authors have clearly invested considerable effort in validating and analyzing their framework.

Critical weaknesses

1. The validity of the two central operational definitions is insufficiently established.

This is my most significant concern. The paper treats answer changes after evidence deletion as Evidence Reliance, and closed-book greedy decoding outcomes as Retrieval Necessity. However, neither operation necessarily measures the intended underlying construct.

  • For Evidence Reliance, deleting retrieved documents changes the model's input context and may introduce distribution shifts. An answer change therefore indicates sensitivity to the intervention, but does not necessarily establish that the original answer was causally grounded in the retrieved evidence. Conversely, an unchanged answer does not imply that the model did not use the evidence, since parametric knowledge and retrieved information may be redundant.

  • Similarly, a failure under one closed-book greedy decoding protocol does not establish that retrieval is fundamentally necessary. The outcome may depend on prompt design, decoding behavior, answer verification, or reasoning variability. I appreciate the additional robustness analyses in Tables A6 and A15, but these mainly demonstrate consistency across related intervention and decoding protocols. They do not fully establish the construct validity of the proposed signals.

As a result, the paper appears to make a substantial interpretive leap from behavioral sensitivity and closed-book sufficiency to causal evidence reliance and retrieval necessity.

2. The proposed objective is largely heuristic, while the experimental evidence does not fully validate its intended interpretation.

The central reward is intuitive, but I am not convinced that its particular multiplicative form has a sufficiently principled justification. Why should the two predicted signals be multiplied rather than combined through another function? Since their numerical values directly affect the reward, how reliable is their probability calibration? The binary repeated-search penalty also appears to be a design choice rather than a consequence of a clearly established optimization objective.

Proposition 1 provides a useful analysis of how the reward modification interacts with GRPO normalization, but it does not establish that the proposed reward corresponds to a principled retrieval objective or leads to genuinely better evidence-grounded behavior.

More importantly, a substantial portion of the experimental evidence validates the proposed proxy system internally: the readouts predict intervention-derived labels, the reward optimizes those predictions, and related metrics subsequently improve. This makes it difficult to distinguish genuine improvement in grounding from successful optimization of the chosen proxies. The human evaluations in Tables A10 and A18 are valuable attempts to address this issue, but the sampling and evaluation designs, together with the non-significant difference in UGA against GPT-5 Judge, leave the independent evidence less conclusive than the overall experimental volume might suggest.

Finally, while the combination of existing ideas is useful, the methodological novelty appears incremental. The individual ingredients—closed-book sufficiency, evidence deletion, representation probing, reward shaping, GRPO, and periodic recalibration—are relatively natural choices. The paper would be more compelling if it established a deeper insight into why their particular combination is necessary, rather than primarily demonstrating that the resulting engineering framework works.

Questions for authors

1. How can the authors distinguish genuine evidence reliance from sensitivity to the deletion intervention? I am particularly uncertain about the interpretation of Evidence Reliance. Removing retrieved documents changes the model's context, and the resulting answer change may reflect distribution shift rather than causal use of the original evidence. The additional controls in Tables A6 and A15 are helpful, but I am not sure they fully resolve this concern.

Could the authors clarify how they distinguish these possibilities? In particular, do they have evidence from controlled evidence replacement or contradiction interventions showing that the model's answers change in a semantically appropriate direction when the evidence content changes, rather than simply changing because part of the context has been removed?

2. Why should closed-book greedy decoding be considered a reliable measure of Retrieval Necessity? I understand the intuition behind using closed-book answer sufficiency as a proxy. However, I am not fully convinced that failure under a single deterministic decoding protocol means that retrieval is necessary for the problem. The result could depend on the prompt, decoding strategy, or the model's reasoning variability.

Could the authors clarify whether the retrieval-necessity labels remain stable under multiple closed-book samples or alternative decoding protocols? More importantly, how do they separate the contribution of external information from the additional reasoning process induced by searching?

3. What justifies the particular reward composition, and how does the necessity gate affect GRPO optimization in practice? The multiplicative form is intuitive, but I would appreciate a clearer explanation of why this specific composition is preferable to alternatives such as additive, minimum-based, or learned combinations. Since the predicted values directly determine reward magnitudes, calibration errors may also matter more than the reported AUC alone suggests.

I am also interested in the practical implications of Proposition 1. Since the necessity gate can be canceled by group normalization under certain homogeneous rollout conditions, could the authors report how frequently the gate actually changes the relative advantages or their signs during training? This would help clarify whether the proposed mechanism plays a substantial role in policy optimization or primarily affects a relatively limited subset of rollout groups.

4. How should we interpret the independent grounding evidence and the claimed generalization? I appreciate the human evaluation in Table A10 and the common-question analysis in Table A18. These are among the most valuable experiments in the paper because they go beyond the proposed internal readout signals. However, A10 uses separately sampled correct trajectories for different methods, while the UGA difference between Search-G1 and GPT-5 Judge in A18 is relatively small, with substantially overlapping confidence intervals. Furthermore, the correlation in A19 is encouraging, but correlation alone does not establish that optimizing the reward causes better evidence support.

Could the authors clarify how strongly these results support the claim of improved grounding? I am also curious about the cross-dataset results: given that the gains over Search-R1 are generally small and often statistically inconclusive in transfer settings, what level of cross-domain generalization does the paper intend to claim?

If the authors can provide convincing clarifications and additional evidence on these points, I would be open to reconsidering my score.

Reviewer_3UvTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Search-G1 introduces a reward framework for selective search in language agents. Two periodically refreshed hidden-state readouts estimate closed-book sufficiency and answer sensitivity to evidence deletion, with the latter conditioned on the realized reasoning prefix. These signals support a reward that favors correct direct answers when retrieval appears unnecessary, rewards evidence-sensitive searched answers when retrieval is needed, and penalizes repeated searches. Policy optimization uses masked GRPO, while periodic target regeneration and readout refitting accommodate changes in the policy. Experiments across four QA benchmarks and two model scales show competitive accuracy, reduced search usage in the main comparisons, and improved counterfactual evidence dependence, complemented by independent human assessments of answer support.

Critical strengths
  1. Clear and interpretable decomposition of the search objective. Separating retrieval necessity from evidence reliance addresses two distinct aspects of search behavior: whether external information is needed and whether a searched answer depends on it. The reward provides an explicit comparison between correct direct and searched trajectories while preserving the ordering of correct, incorrect, and invalid outputs. Periodic recalibration also accounts for changes in policy capability. The analysis makes the reward structure transparent, including the conditions under which group normalization limits the effect of necessity gating.

  2. Broad empirical evaluation with useful validity checks. The evaluation combines task accuracy, retrieval cost, component ablations, and readout-refresh studies across multiple benchmarks and model scales. Independent human support assessments and evaluations on shared question sets complement the counterfactual metrics. Controls for retained reasoning, decoding mismatch, and surface features further strengthen the empirical analysis. Together, these results provide meaningful evidence that the framework can improve the accuracy–search-cost trade-off in the evaluated settings.

Critical weaknesses
  1. Necessity gating can be largely neutralized by group normalization, while sufficient direct-answer exploration is not ensured. When all trajectories in a group are correct, use retrieval, and incur the same search cost, GRPO normalization removes the necessity factor in the zero-stabilizer limit and can leave it with little effect under the implemented stabilizer. Consequently, a small estimated retrieval necessity need not produce a correspondingly small auxiliary advantage. Small differences between reliance scores can still receive substantial normalized credit on questions that are likely answerable without retrieval. The intended comparison between direct answering and searching also depends on sampling useful representatives of both branches. This leaves uncertainty about whether unnecessary retrieval receives an effective corrective signal on high-sufficiency questions.

  2. The search-cost objective provides no marginal penalty beyond the second retrieval. The cost (c(N)=1[N>1]c(N)=\mathbf{1}[N>1]) assigns the same penalty to two, three, or four searches. Thus, trajectories with identical correctness and reliance scores receive the same reward regardless of how many additional calls they make after the second search. Incorrect but valid trajectories also receive no search-cost penalty. The objective therefore does not directly discourage additional retrieval in these regimes. This limitation matters for multi-step stopping: fewer searches on average under a four-round cap do not establish that the agent stops when sufficient information has been acquired. The improvement could primarily reflect a shift toward single-search trajectories, while later stopping decisions remain weakly constrained. A bounded per-call penalty could preserve the same correctness-first reward tiers while distinguishing successive calls, so the reward-ordering constraint does not require the current saturation.

Questions for authors
  1. How frequently does necessity gating meaningfully affect the optimization signal, particularly on high-sufficiency questions?
  2. How does the saturated search penalty compare with a bounded per-call penalty that preserves the same reward tiers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.