acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking and Training Reward Models for Cross-Source Evidence Grounding in Search Agents

Abstract

Existing approaches to training and evaluating search agents focus heavily on final-answer quality, largely neglecting how generated answers map back to retrieved evidence. Recent methods incorporate evidence citation tracking to enforce grounding, but citation support alone does not establish source independence or whether an answer appropriately handles conflicting evidence. In this paper, we argue that credibility in open-web retrieval depends on principled cross-source evidence grounding that accounts for source independence and systematically attributes and adjudicates conflicting evidence, while hierarchical judgments of these grounding decisions provide a unified signal for training and evaluation. To this end, we propose an evaluation framework, **Cross-Source Evidence Grounding (CSEG)**, that employs a fine-grained, six-layer decision tree over the complete search trajectory. We introduce **CSEG-Bench** and trainable reward models to provide targeted supervision following this framework. Our method offers the key advantage of eliminating the severe data skew of natural trajectories by providing targeted coverage over complex, deep-layer decision paths, such as cross-source conflict resolution and nuanced attribution, that standard rollouts fail to capture. Specifically, (1) we define the hierarchy at label, node, and path levels; (2) we construct 1,160 verified trajectories over nine canonical paths by rewriting answers on real retrieved document pools, with independent frontier judges reproducing the entire gold path and a human expert deciding inclusion; and (3) we train **CSEG-RM**, which reaches 83.9% exact accuracy on our benchmark while lying on the performance–token efficiency Pareto frontier. Experiments that use this reward alone to supervise a frozen-retrieval answer policy raise the independently judged mean label from 2.83 to 4.19 and show that learning the hierarchy generalizes across layers at a fraction of frontier-judge cost.

Primary area:
datasets and benchmarks
Keywords:
datasets and benchmarks, search agents, attribution, reward models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_AnqHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a problem: prior methods evaluate the quality of search agents only on the final answer, ignoring citation supports or source independence. The author proposes the CSEG framework to classify trajectories and a reward model.

Critical strengths

The paper identifies the gap in evidence grounding in search agents.

Critical weaknesses
  • Although the performance has improved on CSEG-Bench, it is not clear that it can transfer to other benchmarks. It lacks experiments to show its transferability.
  • The improvement on CSEG-Bench seems obvious to me: training and testing on the same benchmark.
  • The major claim in the paper is not fully addressed. The paper and the experiments do not provide evidence to support the claim.
Questions for authors
  • What is the preliminary setup mentioned in the introduction & RQ1? What is the dataset used, and how did the authors obtain the results?
  • How are the trajectories generated using the 6-layer decision tree? Is it rule-based or LLM-as-a-Judge?
Reviewer_wdT8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CSEG, a six-layer evaluation framework that judges whether a search agent's answer is grounded across sources, covering citation presence, multi-source support, evidence-root independence, conflict handling, and adjudication by authority and recency. Built on outcome-passing trajectories from a production search service, CSEG-Bench contains 1,160 trajectories over nine canonical paths, constructed by injecting evidence and rewriting answers on real retrieved pools. The authors also distill 4B and 8B reward models that reach 83.9% exact accuracy and raise a search agent's judged mean label from 2.83 to 4.19 when used as the only reward.

Critical strengths
  • Shows that outcome rewards miss grounding failures, as 43.8% of outcome-passing production trajectories fail by L3, which justifies a grounding benchmark.
  • A controlled comparison with natural rollouts under the same GRPO recipe turns the need for constructed deep-path data into evidence, with gains of 12.6 and 20.2 points.
Critical weaknesses
  • CSEG assumes credibility requires two independent roots, but a single official filing stops at L2 while two blogs pass L3, so the gate blocks answers before the L6 authority check, leaving open whether grounding should rest on source quality.
  • CSEG assumes pool grounding implies trustworthiness, but a biased pool can ground a wrong answer and the policy has no factuality reward, so the gain from 2.83 to 4.19 says nothing about correctness, leaving the trade-off between grounding and truth open.
Questions for authors

Clarification

  • Your own L6 ordering ranks an official regulatory document above a personal blog, so why should an answer resting on one filing fail at L2 while one citing two blogs moves on? If authority counts only after corroboration, what does CSEG take credibility to be?
  • With no factuality reward, what stops the policy from raising its label from 2.83 to 4.19 by citing sources that agree rather than sources that are right? When the pool is biased, should a well-grounded wrong answer outscore a weakly grounded correct one?

Future work

  • Injected documents were detectable before surface controls, so what does the construction pipeline leave behind that natural failures do not, and could a reward model trained and tested on the same pipeline be learning that trace instead of grounding?
Reviewer_DFozOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a cross-source evidence-grounding framework for search agents, CSEG, together with CSEG-Bench and trainable reward models. Rather than focusing solely on final-answer quality, it evaluates how answers are grounded in retrieved evidence, including source independence and the handling of conflicting evidence. The framework uses a six-layer decision hierarchy and evaluates predictions at label, node, and path levels. The paper validates the benchmark through comparisons across open-weight models, analyzes thinking effort, and evaluates trained reward models and downstream answer-policy optimization. The reward-model contribution and downstream validation were added to this summary following the manuscript check.

Critical strengths
  1. The grounding construct captures an important limitation of answer correctness and citation presence: several URLs may still represent one evidence root. The production audit reports 85 such failures among 256 outcome-passing trajectories, and the hierarchy distinguishes missed conflict from attribution and adjudication errors. This provides a meaningful diagnostic contribution beyond a flat citation-support score.
  2. The benchmark and reward-model study are substantial. Full-path verification, expert inclusion decisions, question-based splitting, multiple training routes, and the matched constructed-versus-natural data comparison support the learnability of the hierarchy. The 4B result of 83.9% exact accuracy with approximately 2.1k completion tokens is a useful operational result within the reported benchmark distribution.
Critical weaknesses
  1. The reward model is mainly validated under the same construction and selection process used to build its training and evaluation data. Rewriting, injected evidence conditions, and trajectory selection can leave systematic cues; question-disjoint splitting alone does not test performance on unmodified natural trajectories. The 256-case production audit motivates the task but does not report trained-RM accuracy against independent human decisions on those trajectories.
  2. The aggregate downstream improvement is not sufficient to establish improved honest grounding. Section 4.3 proposes a pool-specific honest-ceiling analysis, since no truthful answer on a single-root pool can exceed label 3, yet Table 5 reports only aggregate labels without quantifying above-ceiling outputs. This distinction is central when the policy is trained exclusively on the hierarchy's reward: a higher label can reflect better evidence use or erroneous independence claims that a judge accepts. An independent final judge reduces direct self-scoring but does not resolve this construct-level concern. Appendix G.1 also states that the 512 evaluation questions are RM training-data seeds, although policy-training questions are disjoint; this is not direct test leakage, but the assertion that it necessarily understates generalization is not established.
Questions for authors

1.Could you further explain the differences between Hy3 and DeepSeek-V4-Flash in Figure 3(a), and the rationale for the models selected for that illustration? I understand that the paper already discusses model-family differences and that similarly named effort settings do not imply equal reasoning budgets across models. A clearer interpretation of the observed differences would be helpful. 2. Appendix G.1 identifies the downstream evaluation questions as RM training-data seeds, while Section 4.3 concerns disjoint policy-training questions. Can the downstream result also be checked on questions unseen by both trained components, and can the claimed direction of any bias from the current choice be justified? 3.How accurately does the trained RM score a human-adjudicated sample of unmodified trajectories, especially source-independence and conflict nodes? 4.For single-root and multi-root pools separately, what are the before/after label distributions and the fraction of outputs exceeding the supported ceiling? For above-ceiling cases, can human inspection distinguish valid evidence use from erroneous independence claims or judge errors?

Reviewer_spzfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CSEG, a structured reward for how search agents use evidence. It checks whether an answer cites relevant and independent sources and handles conflicting information correctly. The authors build a benchmark with 1,160 examples and train smaller reward models to follow this framework. They also use the reward to train a search model and show that its CSEG score improves. The structured reward is clear and the experiments are fairly thorough. However, the setting feels quite niche, and most of the benchmark is constructed and evaluated using LLMs. The final experiment also does not show that the answers become more correct or useful.

Critical strengths
  1. The paper found an important limitation of answer level and citation level rewards. Multiple URLs can repeat the same evidence, and a cited answer can ignore or mishandle disagreement. The explicit hierarchy makes these failures inspectable.

  2. The experimental design is very careful about split discipline, path coverage, and possible data shortcuts. The paper validates complete paths meticulously.

Critical weaknesses
  1. The main contribution is the structured reward hierarchy, but it seems to be designed only for a niche use case. It is currently for single-turn answers over a fixed set of documents, and it is unclear how useful it is for more general search or reasoning tasks.

  2. The benchmark is constructed and labeled mainly using LLMs. Humans seem to decide which examples to keep, but 468 of the 1,160 examples have only one-judge agreement. It is unclear whether the models learn evidence grounding or just the rules used to create the benchmark.

  3. The final experiment only shows that the model gets a higher CSEG score. It does not show that the answers become more correct or useful, and to add to it, the evaluation is still done by an LLM using the same framework.

  4. The paper is hard to read. The language is dense and repetitive. Many simple ideas are explained in a much more complicated way than necessary.

Questions for authors
  1. Can human annotators independently label a representative subset without seeing the target path or the LLM judges’ decisions? Could consider providing agreement for each layer and separate results for natural and constructed examples.

  2. After training with CSEG, are the answers still as correct and useful as before according to an answer-based metric and human evaluation? Maybe also test on completely new questions and document sets, then check whether the gain comes from longer answers or just from adding more citations.

Comments on the paper

1 comment
Area_Chair_69BWOpenReview ↗

Author-Reviewer Discussion

Dear Reviewers,

The authors have provided the rebuttal. Please have a look and let the authors know if you need further clarification or still have follow-up questions.

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.