- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses sparse outcome rewards in deep search agents. It proposes EG-GRPO, which converts evidence claims from synthetic multi-hop data into dense step-level rewards, uses an LLM judge to attribute each claim to its first supporting citation, defines cumulative knowledge states, and applies three-level fallback normalization to step advantages before fusing them with GRPO trajectory advantages. On 5 deep search benchmarks and two model scales, EG-GRPO reports average gains of 4.96 and 3.64 points over GRPO.
Critical strengths
- Novel and practical use of synthetic evidence annotations as step-level supervision, avoiding additional human process annotation or reward-model training.
- The author gives clear formulation of cumulative knowledge states and state-conditioned advantage normalization, with a three-level fallback for sparse exact states.
- The author performed extensive evaluations, five deep search benchmarks, two model scales, and comparisons against multiple baselines.
Critical weaknesses
- LLM-judge reliability is verified only on 10 trajectories and 67 evidence claims, which is small for a method whose rewards depend on judge attribution. And there is no direct evidence found in the manuscript for broader judge validation. The paper also does not investigate the impact of different judge models on the performance of the search agent.
- There exists a risk of reward hacking, the agent retrieves texts containing evidential statements, yet such texts do not support the final answer. This phenomenon is not directly quantified, nor are any illustrative comparisons provided.
- Computational cost of LLM-judge calls for outcome scoring and evidence matching is not quantified.
Questions for authors
- Can the LLM-judge validation be expanded beyond 10 trajectories and 67 claims, including false-positive evidence matches?
- Can author report judge-call token and latency costs, and include them in training-cost comparisons?
- Suggestions for future work: Evaluate on real-user or non-synthetic queries to test distribution transfer.
3 comments
I appreciate the authors' rebuttal, which clarifies the computational costs and the rationale behind the reward design. However, part of my concerns remain unaddressed: validating the LLM-as-a-judge on only 10 trajectories lacks statistical power. Authors have not provided expanded validation data or experiments with alternative judge models (e.g., cost-approximation models like Qwen3.5-Flash). Therefore, I maintain my current rating for now and welcome further clarification from authors.
Thank you very much for your constructive feedback and perspective! We appreciate your acknowledgment of the clarifications on computational costs and reward design. We understand your remaining concerns about the judge-validation sample size and the lack of comparisons with alternative judges. Your suggestions will help guide further validation and improvements to the manuscript. Thank you again for your time and careful consideration.
We are grateful for your insightful review and constructive feedback on our manuscript. We address your concerns below.
1. W1 and Q1: Judge validation
Thank you for your question. We sampled 10 trajectories generated by the Qwen3-30B-A3B SFT model and manually assessed all 67 associated evidence claims.
We limited this initial validation to 10 trajectories because reviewing long-horizon deep-search rollouts is time-intensive. Each trajectory contains multiple retrieval steps and substantial retrieved content, requiring annotators to verify each claim and trace its first supporting citation. The validation achieves 91.0% support-detection accuracy and 94.0% first-citation accuracy, as reported in Appendix E.
We will specify the sampling source, metric denominators, and error breakdown in Appendix E and present the results as an initial reliability assessment. We used Qwen-Flash throughout training and have not compared training outcomes under different judge models; we will state this limitation.
2. W2: Evidence acquisition and reward hacking
Thank you for your question. The process reward checks whether retrieved content supports a query-specific reference claim. The judge is instructed to require explicit factual support; topical overlap, entity mentions, and the agent’s own unsupported inferences do not qualify. Each claim receives credit only at its first supported occurrence, so repeated retrieval of the same evidence does not accumulate additional reward.
A trajectory can acquire useful evidence and still produce an incorrect final answer. Rewarding the productive retrieval steps in such trajectories is an intended part of our credit-assignment design.
Final-answer correctness is retained through a separate outcome advantage, and the process advantage is clipped before the two signals are combined. We have not separately quantified how often agents increase evidence coverage without improving answer correctness. We will clarify this distinction and the scope of the current analysis.
3. W3 and Q2: Judge-call costs
Thank you for your suggestion. Evidence matching evaluates all annotated claims in one Qwen-Flash call per trajectory. With at most 64K input tokens and negligible output, the estimated cost is up to approximately CNY 0.01 per trajectory. Step attribution, discounted returns, and grouping reuse that output without further judge calls. We will include this in the training-cost discussion.
4. Q3: Transfer to real-user queries
Thank you for your valuable suggestion. We train on synthesized queries and evaluate on BrowseComp, BrowseComp-ZH, GAIA, and the two xBench-DS versions, testing transfer to external benchmarks. Real-user queries would provide a further test of generalization, and we will include this direction in the future-work discussion.