acceptodds
Under review as a conference paper at ICLR 2027

EG-GRPO: Evidence-Guided Step-level Credit Assignment for Deep Search Agents

11 views

Abstract

LLM-based search agents demonstrate strong capabilities in long-horizon information seeking and reasoning, typically trained via reinforcement learning. However, existing approaches predominantly rely on trajectory-level outcome rewards, providing limited supervision for complex search and reasoning processes. Although step-level credit assignment offers a promising solution, obtaining reliable and task-grounded supervision for open-ended search remains challenging. In this paper, we propose Evidence-Guided Group Relative Policy Optimization (EG-GRPO), a framework that derives step-level supervision from the evidence annotations available in synthetic deep search data. EG-GRPO identifies the first supporting citation for each required evidence claim and uses the newly acquired evidence to construct step-level process rewards. By representing the accumulated evidence as an evolving knowledge state, EG-GRPO further performs state-conditioned relative advantage estimation and combines the resulting step-level signal with the standard trajectory-level outcome advantage. Experiments on five challenging deep search benchmarks show that EG-GRPO generally outperforms standard GRPO and existing reward-enhanced and step-level RL baselines across different model scales. These results demonstrate that task-grounded evidence annotations can provide effective process supervision for credit assignment in deep search RL.

Primary area:
reinforcement learning
Keywords:
Agentic RL, Credit Assignment, Deep Search, Long Horizon Agent

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_KjQ1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses sparse outcome rewards in deep search agents. It proposes EG-GRPO, which converts evidence claims from synthetic multi-hop data into dense step-level rewards, uses an LLM judge to attribute each claim to its first supporting citation, defines cumulative knowledge states, and applies three-level fallback normalization to step advantages before fusing them with GRPO trajectory advantages. On 5 deep search benchmarks and two model scales, EG-GRPO reports average gains of 4.96 and 3.64 points over GRPO.

Critical strengths
  1. Novel and practical use of synthetic evidence annotations as step-level supervision, avoiding additional human process annotation or reward-model training.
  2. The author gives clear formulation of cumulative knowledge states and state-conditioned advantage normalization, with a three-level fallback for sparse exact states.
  3. The author performed extensive evaluations, five deep search benchmarks, two model scales, and comparisons against multiple baselines.
Critical weaknesses
  1. LLM-judge reliability is verified only on 10 trajectories and 67 evidence claims, which is small for a method whose rewards depend on judge attribution. And there is no direct evidence found in the manuscript for broader judge validation. The paper also does not investigate the impact of different judge models on the performance of the search agent.
  2. There exists a risk of reward hacking, the agent retrieves texts containing evidential statements, yet such texts do not support the final answer. This phenomenon is not directly quantified, nor are any illustrative comparisons provided.
  3. Computational cost of LLM-judge calls for outcome scoring and evidence matching is not quantified.
Questions for authors
  1. Can the LLM-judge validation be expanded beyond 10 trajectories and 67 claims, including false-positive evidence matches?
  2. Can author report judge-call token and latency costs, and include them in training-cost comparisons?
  3. Suggestions for future work: Evaluate on real-user or non-synthetic queries to test distribution transfer.
3 comments

We are grateful for your insightful review and constructive feedback on our manuscript. We address your concerns below.

1. W1 and Q1: Judge validation

Can the LLM-judge validation be expanded beyond 10 trajectories and 67 claims, including false-positive evidence matches?

Thank you for your question. We sampled 10 trajectories generated by the Qwen3-30B-A3B SFT model and manually assessed all 67 associated evidence claims.

We limited this initial validation to 10 trajectories because reviewing long-horizon deep-search rollouts is time-intensive. Each trajectory contains multiple retrieval steps and substantial retrieved content, requiring annotators to verify each claim and trace its first supporting citation. The validation achieves 91.0% support-detection accuracy and 94.0% first-citation accuracy, as reported in Appendix E.

We will specify the sampling source, metric denominators, and error breakdown in Appendix E and present the results as an initial reliability assessment. We used Qwen-Flash throughout training and have not compared training outcomes under different judge models; we will state this limitation.

2. W2: Evidence acquisition and reward hacking

The agent retrieves texts containing evidential statements, yet such texts do not support the final answer.

Thank you for your question. The process reward checks whether retrieved content supports a query-specific reference claim. The judge is instructed to require explicit factual support; topical overlap, entity mentions, and the agent’s own unsupported inferences do not qualify. Each claim receives credit only at its first supported occurrence, so repeated retrieval of the same evidence does not accumulate additional reward.

A trajectory can acquire useful evidence and still produce an incorrect final answer. Rewarding the productive retrieval steps in such trajectories is an intended part of our credit-assignment design.

Final-answer correctness is retained through a separate outcome advantage, and the process advantage is clipped before the two signals are combined. We have not separately quantified how often agents increase evidence coverage without improving answer correctness. We will clarify this distinction and the scope of the current analysis.

3. W3 and Q2: Judge-call costs

Can the author report judge-call token and latency costs, and include them in training-cost comparisons?

Thank you for your suggestion. Evidence matching evaluates all annotated claims in one Qwen-Flash call per trajectory. With at most 64K input tokens and negligible output, the estimated cost is up to approximately CNY 0.01 per trajectory. Step attribution, discounted returns, and grouping reuse that output without further judge calls. We will include this in the training-cost discussion.

4. Q3: Transfer to real-user queries

Evaluate on real-user or non-synthetic queries to test distribution transfer.

Thank you for your valuable suggestion. We train on synthesized queries and evaluate on BrowseComp, BrowseComp-ZH, GAIA, and the two xBench-DS versions, testing transfer to external benchmarks. Real-user queries would provide a further test of generalization, and we will include this direction in the future-work discussion.

I appreciate the authors' rebuttal, which clarifies the computational costs and the rationale behind the reward design. However, part of my concerns remain unaddressed: validating the LLM-as-a-judge on only 10 trajectories lacks statistical power. Authors have not provided expanded validation data or experiments with alternative judge models (e.g., cost-approximation models like Qwen3.5-Flash). Therefore, I maintain my current rating for now and welcome further clarification from authors.

Thank you very much for your constructive feedback and perspective! We appreciate your acknowledgment of the clarifications on computational costs and reward design. We understand your remaining concerns about the judge-validation sample size and the lack of comparisons with alternative judges. Your suggestions will help guide further validation and improvements to the manuscript. Thank you again for your time and careful consideration.

Reviewer_gp8YOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the challenge of long-horizon credit assignment in Deep Research Agents, particularly in evaluating effective search exploration and reasoning steps. To address this, the paper proposes EG-GRPO, which leverages ground-truth evidence claims to construct step-level process rewards and introduces knowledge-state-based grouping for fine-grained credit assignment. Experiments validate the effectiveness of evidence-guided rewards through evidence coverage analysis and reward ablations, assess the contribution of knowledge-state-based credit assignment through grouping ablations, and demonstrate improved exploration behavior and overall performance across five Deep Search benchmarks.

Critical strengths
  1. Evidence-Guided Process Rewards: The paper proposes an evidence-guided reward mechanism that leverages ground-truth evidence claims to provide step-level supervision, addressing the sparse reward problem in long-horizon Deep Search RL.

  2. Robust Advantage Estimation: The paper introduces knowledge-state-based grouping with hierarchical fallback normalization to address sparse state matching and improve the reliability of step-level advantage estimation.

Critical weaknesses
  1. The methodological novelty is limited, as evidence-based credit assignment and state-conditioned advantage estimation have already been explored in STAMP[1] and GiGPO[2]. The proposed method primarily combines and extends these existing ideas.

  2. The experimental evaluation lacks comparisons with earlier Deep Search RL methods, such as E-GRPO[3] and IGPO[4]. Moreover, EG-GRPO introduces additional LLM-based evidence evaluation costs, but the paper does not investigate whether the performance improvements justify the extra computation compared with alternative reward mechanisms under comparable computational budgets.

[1] Xu, Ke, et al. "STAMP: Provenance-Guided Credit Assignment for Deep Search Agents." arXiv preprint arXiv:2607.11172 (2026).

[2] Feng, Lang, et al. "Group-in-group policy optimization for llm agent training." Advances in Neural Information Processing Systems 38 (2026): 46375-46408.

[3] Zhao, Yida, et al. "Repurposing synthetic data for fine-grained search agent supervision." International Conference on Learning Representations. Vol. 2026. 2026.

[4] Wang, Guoqing, et al. "Information gain-based policy optimization: A simple and effective approach for multi-turn search agents." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  1. Could the authors compare EG-GRPO with a trajectory-level evidence reward baseline using the same evidence annotations? This would help distinguish the benefits of step-level credit assignment from those of additional evidence-based supervision.

  2. Could the authors provide controlled comparisons with earlier methods such as E-GRPO and IGPO, and report the additional computational cost of LLM-based evidence matching? In particular, how do the performance gains compare under similar reward evaluation budgets?

4 comments

We sincerely appreciate your thorough review and valuable feedback on our paper. Your concerns are addressed as follows:

1. W1: Contribution relative to STAMP and GiGPO

The proposed method primarily combines and extends these existing ideas.

Thank you for your question. EG-GRPO uses evidence claims to define both process rewards and knowledge states: newly supported claims provide step rewards, while cumulative supported claims determine which steps are compared during advantage estimation.

The main distinction from STAMP is how evidence credit enters optimization. STAMP applies sign-preserving modulation to the trajectory advantage. EG-GRPO computes a separate evidence-based process advantage and combines it with the outcome advantage. When all rollouts receive identical outcome rewards, differences in evidence returns within a normalization group can still produce nonzero process advantages.

We build on GiGPO’s state-conditioned comparison principle, using cumulative supported claims to represent search progress. Hierarchical fallback preserves exact-state comparisons when enough matches exist and broadens the group when matches are sparse.

We developed EG-GRPO independently and became aware of STAMP after completing our method design. We subsequently included it as a baseline and will clarify the methodological differences in the revised manuscript.

2. Q1: Comparison with trajectory-level evidence supervision

Could the authors compare EG-GRPO with a trajectory-level evidence reward baseline using the same evidence annotations?

Thank you for your valuable suggestion. We added this baseline on Qwen3-30B-A3B, using the same evidence annotations, matching judge, SFT initialization, RL data, and training budget as EG-GRPO. Its reward combines outcome correctness with the fraction of distinct annotated claims supported by retrieved content:

Ri=Rioutcome+0.5Ricoverage.R_i=R_i^{\mathrm{outcome}}+0.5R_i^{\mathrm{coverage}}.

Standard GRPO normalization produces one advantage per trajectory, applied uniformly to decision tokens. The baseline uses no step-level returns or knowledge-state grouping.

MethodBCBC-ZHGAIAxBench-2505xBench-2510Average
GRPO10.322.442.757.620.130.62
GRPO + trajectory-level evidence reward10.322.441.657.720.330.46
EG-GRPO11.126.149.164.220.834.26

EG-GRPO improves over this baseline on all five benchmarks, with an unweighted average gain of 3.8 percentage points. This comparison supports the benefit of the step-level framework beyond adding evidence coverage to a trajectory-level reward. Due to the limited time available during the response period, the corresponding Qwen3-4B experiment is still running; we will include these results in the revised manuscript.

3. W2 and Q2: Comparisons with IGPO and E-GRPO

Could the authors provide controlled comparisons with earlier methods such as E-GRPO and IGPO?

Thank you for your suggestion.

IGPO. We evaluated our reproduction of IGPO on Qwen3-4B before submission:

MethodBCBC-ZHGAIAxBench-2505xBench-2510
IGPO5.714.836.650.016.0

We used the method-specific hyperparameters from the IGPO paper within our training pipeline. We initially focused the main comparison on stronger-performing baselines and omitted these results. We report them here and will add the implementation and training details to the revised manuscript.

E-GRPO. We prioritized C-GRPO because it uses citation-aware rubric supervision and outperformed E-GRPO in its reported controlled comparisons. Our computational budget led us to reproduce C-GRPO rather than both methods.

4. W2 and Q2: Evidence-matching cost and evaluation budgets

How do the performance gains compare under similar reward evaluation budgets?

Thank you for your question. Evidence matching evaluates all annotated claims in one Qwen-Flash call per trajectory. With at most 64K input tokens and negligible output tokens, the estimated cost is up to approximately CNY 0.01 per trajectory. Step attribution and grouping reuse the matching output and require no further judge calls. The trajectory-level evidence baseline uses the same matching procedure and one-call-per-trajectory protocol. We will include the call count and cost estimate in the revised manuscript.

AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer,

Thank you once again for your thoughtful and valuable comments! As the deadline for the rebuttal discussion approaches, we would greatly appreciate knowing if our responses have adequately addressed your inquiries. Should you require any further clarifications or have any additional concerns, please do not hesitate to reach out to us.

Your feedback is immensely valuable to us.

Warm regards, Authors.

Thank you for your detailed rebuttal and the additional experimental efforts. I appreciate the authors' efforts to address my concerns, and I acknowledge that the new results provide stronger evidence for the effectiveness of the proposed method. My only remaining concern is the limited methodological novelty relative to prior work. Nevertheless, given the additional evidence and clarifications, I am raising my rating to weak accept.

Thank you for reviewing our responses and additional experiments, and for updating your assessment. We appreciate your recognition of the new results and understand your remaining concern about methodological novelty. We will refine the discussion of prior work to make the contribution and its relationship to existing methods clearer. Thank you again for your time and constructive feedback!

Reviewer_W6R2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Evidence-Guided Group Relative Policy Optimization (EG-GRPO) for training deep search agents with step-level credit assignment.

The method uses evidence claims available from synthetic data generation to identify when each claim is first supported by retrieved content.

Experiments on two Qwen3 model scales and five deep search benchmarks show improvements over standard GRPO and generally competitive or superior results against reward-enhanced and step-level baselines.

Critical strengths
  1. EG-GRPO converts existing synthetic evidence annotations into step-level rewards without requiring additional human process annotations or reward-model training.

  2. The evidence-state representation and hierarchical fallback grouping provide a practical way to compare intermediate search steps across trajectories.

  3. EG-GRPO improves over GRPO across all five benchmarks at both model scales, with particularly substantial gains on GAIA.

Critical weaknesses
  1. The experiments lack a trajectory-level evidence-coverage reward baseline using the same annotations and judge, leaving the specific benefit of step-level attribution unclear.

  2. The evidence judge is validated on only 10 trajectories and 67 claims, providing limited evidence of its reliability.

  3. The paper omits variability estimates and key implementation details, including M1M_1 and the handling of zero-variance groups.

Questions for authors
  1. How does EG-GRPO compare with GRPO using a trajectory-level evidence-coverage reward computed from the same claims and judge?

  2. How were the judge-validation samples selected, and how were support-detection and first-citation accuracy calculated?

  3. What value of M1M_1 and zero-variance handling were used, and are variability estimates across training seeds or evaluation repetitions available?

3 comments

Thank you for the positive assessment and suggestions. We address the three questions below. For clarity, we denote Weaknesses with W and Questions with Q.

1. W1 and Q1: Trajectory-level evidence supervision

Thank you for your valuable suggestion. We added this baseline on Qwen3-30B-A3B using the same evidence claims, matching judge, SFT initialization, RL data, and training budget. For each trajectory, the evidence reward is the fraction of annotated claims supported by retrieved content. We combine it with outcome correctness:

Ri=Rioutcome+0.5Ricoverage.R_i=R_i^{\mathrm{outcome}}+0.5R_i^{\mathrm{coverage}}.

Standard GRPO normalization is applied to this scalar reward, and the resulting trajectory advantage is broadcast uniformly across decision tokens. The baseline uses no step-level returns or knowledge-state grouping.

MethodBCBC-ZHGAIAxBench-2505xBench-2510Average
GRPO10.322.442.757.620.130.62
GRPO + trajectory-level evidence reward10.322.441.657.720.330.46
EG-GRPO11.126.149.164.220.834.26

EG-GRPO improves over the trajectory-level evidence baseline on all five benchmarks, with an unweighted average gain of 3.8 percentage points. Using the same evidence annotations and judge, these results support the benefit of our step-level framework beyond adding evidence coverage to a trajectory-level reward. We will incorporate this comparison into the revised manuscript.

2. W2 and Q2: Judge-validation sampling and accuracy definitions

Thank you for your comments. We sampled 10 trajectories from rollouts generated by the Qwen3-30B-A3B-SFT model and manually assessed all 67 associated evidence claims. For each claim, human annotators determined whether the retrieved content explicitly supported it and, if supported, identified the earliest supporting citation.

Support-detection accuracy measures agreement between the judge’s supported/unsupported labels and human annotations across all 67 claims, yielding 91.0%. First-citation accuracy is calculated over all human-supported claims and requires an exact match with the human-identified earliest supporting citation. Missed support detections are counted as incorrect, yielding an accuracy of 94.0%.

These results provide an initial assessment on search trajectories. We agree that broader validation would strengthen the reliability assessment. We will specify the sampling source and metric definitions in Appendix E and revise the reliability claim to reflect this validation scope.

3. W3 and Q3: Grouping threshold and zero-variance handling

Thank you for your questions. The minimum group size is M1=4M_1=4 for both exact-state and state-size grouping. Query-global statistics are computed when at least two step samples are available. We stabilize the normalization denominator with ϵ=10−6\epsilon=10^{-6}. Groups with identical returns therefore yield zero process advantages, leaving the trajectory-level advantage intact. Fallback between grouping levels is triggered by insufficient sample counts.

Each configuration was trained with a single seed due to the computational and API costs of long-horizon agentic RL. All reported scores are averaged over 3 independent evaluation runs.

AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer,

Thank you once again for your thoughtful and valuable comments. As the deadline for the rebuttal discussion approaches, we would greatly appreciate learning whether our responses have adequately addressed your questions. Should you require any further clarification or have additional concerns, please do not hesitate to let us know.

Your feedback is valuable to us.

Warm regards, Authors

Thank you for your reply.

Reviewer_ctqhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a way to leverage the artificial data used for search agent RL post training training to provide additional step level reward signals that are incorporated with the full rollout level reward. In traditional RL post training algorithms like GRPO and C-GRPO, one binary rollout level award is given. The authors propose that steps that contribute more towards obtaining the correct evidence that could lead to the answer should be rewarded more.

The authors use the knowledge evidence used to design the query for RL post training to provide supplementary step level rewards. Each step between two retrieval calls that surfaces more evidence is awarded more reward. The step level reward advantage is normalized based on the history state defined by the cumulative set of facts obtained by that step. If no other steps have the same state, then the grouping is done based on the number of evidence, and if even that is not available, then all step's average advantage. This step level reward is added with a weight w to the original rollout level reward. The resulting algorithm is called EG-GRPO (Evidence Grounded-GRPO).

The authors run experiments on 5 benchmarks, BC (1266 q), BC-ZH (289 q), GAIA (103 q), XBench-DS-2505 and , XBench-DS-2510 and show consistent improvement over other RL baselines such as GRPO, C-GRPO, GiGPO, STAMP.

The authors run experiments on 5 benchmarks, BC (1266 q), BC-ZH (289 q), GAIA (103 q), XBench-DS-2505 and , XBench-DS-2510 and show consistent improvement over other RL baselines such as GRPO, C-GRPO, GiGPO, STAMP.

Critical strengths
  • EG-GRPO is a cheap way to utilize the additional information present in the synthetic training examples used for deep research agent training. Giving step level advantages for finding additional information based on concrete evidence, weighted by how far in the future the information is revealed, is a reasonable way to measure each step’s relative contribution. Normalizing the advantages based on grouping based on a tiered system is also a novel approach.
  • The algorithm shows consistent improvement over baselines across datasets and across two model sizes. Although I do think these results are too close and without standard error estimations.
Critical weaknesses

A lot of the key ideas of the paper are not necessarily novel and mostly a combination of ideas proposed by earlier works. Using the evidence that comes from multi hop data has been seen earlier (E-GRPO,C-GRPO)

  • Giving credit based one when a new fact is revealed is done in STAMP
  • Step level reward based on how much new information is revealed in this step and then using weighted combination of these is also proposed in IGPO (Wang et al, ICLR 2026) (although the information gain in IGPO is measured by the probability of outputting the answer instead of facts)
  • Discounted step rewards: GiGPO, and IGPO also use discounted reward
  • Adding a weighted sum of step reward and rollout reward is also in proposed in GiGPO
  • Normalizing across step level rewards based on the anchor state is proposed in GiGPO (although their state is env state, here they use cumulative facts as state) Using the query level step statistics to normalize is proposed in IGPO

So the main new ideas that this paper is contributing:

  1. Using a tiered system for step level reward normalization when grouping based on evidence state, evidence size, or query.
  2. A reward based on the numbers of evidences surfaced and combining other techniques to get good eval performance.

The grouping by the tiered system is not sufficiently shown to help in the experiments. Table 4 compares the tiering levels with exact state grouping (GiGPO) and query level (IGPO) with the tiered method. The results are very close, on three of the benchmarks, query level and tiered differ by only one question, tied in one, and for the other dataset, its also within expected noise level. Since there are no standard error reports or how many random seeds were run to obtain the results, there isn’t evidence to show the tiered grouping helped.

For the overall accuracy numbers compared with baselines as well, no measure of number of runs or standard deviations is given. And the difference between EG-GRPO and STAMP in 30B, is again very close. In the 4B case as well, the difference between EG-GRPO and other baselines is too close to be confident without repeated random seed runs.

Table 3 shows that the accuracy also varies with the weight w, the the results reported in comparison with baselines are actually tuned on the test benchmarks itself as the w=0.5 is selected based on the model performance. Given that the differences between baselines and the method is small, it’s not clear if the accuracy gains are consistent or by cherry picking the best results from a sample of results. It’s also not clear if the baselines were also tuned on the test test or not? It also seems that GRPO gains over SFT are also much smaller than what other works have reported ( For eg in STAMP, for Qwen30B on BC goes from SFT 11.1, GRPO takes 18.9 but in this work, it only goes from 9.3 to 10.3, and C-GRPO takes it to 18.3 but only 10.4 in this work. The setups are different so numbers are not exactly comparable, but it does point that the training setup in the paper might not be sufficient to achieve the most that is possible from the baselines.

STAMP’s original paper shows STAMP can be added to E-GRPO, GRPO, C-GRPO, I assume the paper only compares GRPO + STAMP, but it’s not clear. For the GRPO+STAMP setup the original paper claimed BC accuracy going from 11.1 to 20.9 but this work only shows 9.3 to 10.8. STAMP is a very similar baseline as it does step level reward scaling based on information gained. I believe such a small gap compared to the original author’s results again makes me doubt if STAMP was tuned correctly. Important baselines such as IGPO and E-GRPO are also missing.

So overall even though the method is an interesting approach to utilize the evidence in training signals, I believe it’s mostly a combination of existing ideas, and the new ideas that the work contributes are not supported with sufficient evidence.

Questions for authors
  1. How many trained seeds and evaluation runs were used for Tables 2, 3, and 4. Can you report mean +- standard deviation?
  2. Can you please consider adding IGPO as a baseline?
  3. Was w=0.5 chosen based on the test benchmark performance? Were any other parameters also tuned based on the test performance? If not, how did you tune them? For example, how was the discount factor and clipping threshold selected?
  4. Why are the gains for GRPO and STAMP so small compared to STAMP paper’s reported numbers?
  5. How were the baselines tuned?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1)

We sincerely appreciate your detailed review and constructive feedback. Your concerns are addressed as follows:

1. Contribution beyond existing methods

A lot of the key ideas of the paper are not necessarily novel and mostly a combination of ideas proposed by earlier works.

Thank you for your question. EG-GRPO uses evidence acquisition in two connected ways: newly supported claims define step rewards, and cumulative acquired claims define the knowledge states used to compare step returns.

This produces a different learning signal from the closest methods. E-GRPO uses entity-based trajectory rewards, while EG-GRPO assigns supported claim acquisition to individual steps. IGPO measures changes in the policy’s likelihood of the reference answer; EG-GRPO measures explicitly supported evidence acquisition. Building on GiGPO’s state-conditioned comparison principle, we use cumulative acquired claims to represent search progress and hierarchical fallback to handle sparse state matches.

We developed EG-GRPO independently and became aware of STAMP in July 2026, after completing our method design. Given the overlap in evidence attribution, we included STAMP as a baseline and focus the comparison on how the two methods construct the learning signal. The key distinction from STAMP is how evidence credit enters optimization. STAMP applies sign-preserving modulation to the trajectory advantage. EG-GRPO independently estimates an evidence-based process advantage and combines it with the outcome advantage. Even when all rollouts receive identical outcome rewards, variation in evidence returns within a normalization group can still produce nonzero process advantages.

To test whether additional evidence supervision alone explains the gains, we added a trajectory-level evidence-reward baseline on Qwen3-30B-A3B, using the same evidence annotations and judge, with the reward

Ri=Rioutcome+0.5Ricoverage,R_i=R_i^{\mathrm{outcome}}+0.5R_i^{\mathrm{coverage}},

where coverage is the fraction of distinct annotated claims supported by retrieved content. Standard GRPO normalization produces a single trajectory advantage, which is broadcast uniformly across decision tokens.

MethodBCBC-ZHGAIAxBench-2505xBench-2510Average
GRPO10.322.442.757.620.130.62
GRPO + trajectory-level evidence reward10.322.441.657.720.330.46
EG-GRPO11.126.149.164.220.834.26

EG-GRPO outperforms the trajectory-level evidence baseline on all five benchmarks, with an average improvement of 3.8 percentage points. This supports the benefit of the step-level framework beyond adding evidence coverage to a trajectory-level reward.

2. Evidence supporting hierarchical fallback grouping

The grouping by the tiered system is not sufficiently shown to help in the experiments...

Thank you for your insightful question. Table 4 measures the incremental contribution of grouping within EG-GRPO. Relative to query-global normalization, the full hierarchy gives an average gain of 0.66 points. Its average gain over exact-state-only grouping is 2.20 points.

The gain over query-global normalization is modest and is not uniform across benchmarks. The hierarchy is designed to retain knowledge-state conditioning when enough matches are available and broaden the comparison group when exact matches are sparse.

The “Level 1 only” and “Level 3 only” variants retain the other components of EG-GRPO and vary only the grouping rule, so Table 4 isolates the effect of grouping within our framework.

3. Training seeds and evaluation repetitions

The difference between EG-GRPO and other baselines is too close to be confident without repeated random seed runs.

How many trained seeds and evaluation runs were used for Tables 2, 3, and 4.

Due to the extreme computational intensity of long-horizon deep search RL, where a single training run for the 30B model consumes approximately 1,536 H800 GPU-hours plus substantial API/tool interaction costs, each model was trained using a single seed. This cost limited our ability to repeat training across seeds.

All reported evaluation scores are averages over three independent evaluation runs, which capture evaluation variability. We will state the training and evaluation counts explicitly in the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2)

4. Missing IGPO and E-GRPO baselines

Important baselines such as IGPO and E-GRPO are also missing.

We appreciate your insightful suggestion.

Regarding IGPO: We evaluated our reproduction of IGPO on Qwen3-4B before submission, obtaining the following results:

MethodBrowseCompBrowseComp-ZHGAIAxBench-DS-2505xBench-DS-2510
IGPO (Qwen3-4B)5.714.836.650.016.0

We initially prioritized the stronger-performing baselines for the main comparison and omitted these results. We have included these reproduced IGPO results here and will incorporate them along with their implementation details into the revised manuscript.

Regarding E-GRPO: We prioritized C-GRPO as our reward-enhanced baseline because it uses citation-aware rubric supervision and outperformed E-GRPO in its reported controlled comparisons. Given our computational budget, we therefore selected C-GRPO rather than reproducing both methods.

5. Hyperparameter selection

Was w=0.5 chosen based on the test benchmark performance? Were any other parameters also tuned based on the test performance?

Thank you for your valuable question. We set the discount factor γ=0.95\gamma=0.95 following GiGPO and fixed the step-advantage clipping threshold C=5C=5 in advance. We selected w=0.5w=0.5 from the five values evaluated in Table 3, w∈[0,0.25,0.5,0.75,1.0]w\in[0,0.25,0.5,0.75,1.0], based on its performance on the reported benchmarks. Table 3 also shows that all four nonzero values of ww improve average accuracy over w=0w=0. We will make this selection procedure explicit in the revised manuscript.

6. Smaller GRPO and STAMP gains

Why are the gains for GRPO and STAMP so small compared to STAMP paper’s reported numbers?

Thank you for this insightful question. Our experiments use a different training pipeline and a paper-based reimplementation of STAMP.

Our pipeline uses 1,000 SFT trajectories selected from OpenSeeker and 2,316 RL queries, with Qwen-Flash as the judge. The original STAMP study uses 1,500 SFT trajectories, 2,600 RL queries, and GPT-5.1 as the judge. Differences in initialization, training data, reward judges, and implementation may contribute to the different gains.

Within our study, the methods share the same SFT initialization, RL data, training budget, and evaluation protocol, allowing a controlled comparison within this pipeline.

7. STAMP configuration

I assume the paper only compares GRPO + STAMP, but it’s not clear.

Yes, the STAMP row in Table 2 denotes GRPO + STAMP, using outcome-only GRPO as the underlying algorithm. We selected this configuration because it achieved the strongest overall performance among GRPO + STAMP, C-GRPO + STAMP, and E-GRPO + STAMP in the original STAMP paper’s controlled comparisons. It also allows us to compare STAMP and EG-GRPO on top of the same GRPO foundation. We will explicitly label the row “GRPO + STAMP” and clarify this configuration in the experimental setup.

How were the baselines tuned?

We adopted each baseline’s method-specific hyperparameters from its original paper. Shared settings—including SFT initialization, RL data, and training budget—followed our experimental pipeline. We did not conduct additional baseline-specific hyperparameter searches. We will document the adopted values and their sources in the revised manuscript.

AuthorsOpenReview ↗

Kindly Reminder

Dear Reviewer,

Thank you once again for your insightful and meticulous comments! As the deadline for the rebuttal discussion approaches, we would be grateful to receive your feedback on whether our responses have satisfactorily addressed your inquiries. Should you require any additional clarifications or have any further concerns, please do not hesitate to reach out to us. Your assistance is truly appreciated.

Warm regards, Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.