acceptodds
Under review as a conference paper at ICLR 2027

Evidence in the Loop: Training Search Agents from Knowledge Graphs

Abstract

Training search agents increasingly relies on synthetic data to reduce human annotation costs. Most prior methods generate training data offline, leaving the task distribution fixed and unable to adapt to the agent's evolving capabilities. Self-evolving frameworks address this limitation by synthesizing data online as the agent improves. However, optimization remains largely outcome-driven, resulting in sparse supervision for intermediate search behavior.Recent work therefore incorporates sampled knowledge graph (KG) paths into self-evolving training, using relational structure to guide question generation and path entities as intermediate supervision. However, this entity-centric design can weaken both the effectiveness of training data and the supervision signal: generated questions may still be answerable even when parts of the factual chain behind it are not explicitly supported by the target text corpus, while entity-based rewards can conflate mentioning path entities with genuine search progress, even when no relevant evidence is actually retrieved to support the solution. To this end, we propose Evidence-in-the-Loop, a self-evolving framework that grounds both question generation and process supervision in retrievable corpus evidence. We sample multi-hop KG paths and retain only those whose triplets are supported by verified corpus evidence. The retained paths guide online question generation, while the associated evidence is reused to supervise the agent's search process. Specifically, we introduce two process rewards: evidence acquisition, which rewards retrieving verified supporting passages, and evidence utilization, which rewards using newly acquired evidence to guide subsequent search toward additional relevant evidence. Experiments on knowledge-intensive QA benchmarks show consistent improvements across models and initialization settings, with particularly strong gains on multi-hop QA. The trained Proposer also produces higher-quality questions, as reflected in both their validity and downstream training utility.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Search Agents, Knowledge Graphs, Reinforcement Learning, Process Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8DuZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Evidence-in-the-Loop, a framework for training search agents through knowledge-graph-guided self-play. It verifies that every relation in a sampled KG path is supported by retrievable corpus passages before using the path for question generation. These passages also provide intermediate supervision through two rewards: evidence acquisition, which measures retrieval of verified supporting passages, and evidence utilization, which measures consecutive retrieval transitions involving newly discovered bridge entities. Process rewards provide partial credit only for incorrect answers, while correct answers receive full reward.

Critical strengths
  1. A concrete improvement to intermediate search supervision. Associating KG relations with verified corpus passages ties rewards to observable retrieval behavior rather than entity mentions alone. Applying partial rewards only to incorrect trajectories also preserves full credit for alternative successful search strategies.
  2. Consistent results with relevant baselines and a useful transfer experiment. The method improves average scores across five configurations compared with SSP and KG-Path. Training a fresh Solver on equally sized generated datasets with outcome-only rewards further supports the value of the generated questions independently of the proposed process rewards
Critical weaknesses
  1. Evidence acquisition may miss valid supporting passages and evidence utilization is not sufficiently validated. The reward matches a small set of predefined passage IDs, potentially overlooking alternative evidence that supports the same relations. This may bias supervision toward selected passages rather than evidence relevance. Entity reuse in consecutive queries does not establish actual dependence on retrieved evidence. The utilization reward adds only 0.3 points in average EM and F1, without repeated-run results or uncertainty estimates.
  2. Limited comparison with stronger search agents. Comparisons focus on SSP and KG-Path, while continued-training experiments use only Search-R1, but Search-R1 is not a robust search agent. Broader evaluation with more capable search-agent frameworks under matched budgets is needed to establish practical significance.
  3. Limited novelty beyond existing frameworks. The method primarily extends SSP [1] and KG-Path [2] with corpus verification and evidence-based rewards. The incremental contribution requires stronger empirical support, particularly for the proposed utilization reward.

[1] Lu et al. Search Self-Play: Pushing the Frontier of Agent Capability without Supervision. ICLR 2026.
[2] Wu et al. Knowledge-Graph Paths as Intermediate Supervision for Self-Evolving Search Agents. 2026.

Questions for authors
  1. How sensitive is the acquisition reward to the predefined passage set? How often do trajectories retrieve valid supporting evidence that receives no credit because its passage ID is not included?

  2. Can you provide behavioral evidence that the utilization reward captures actual dependence on retrieved information, rather than entity repetition? Is its 0.3-point gain reproducible across training seeds?

  3. Do the improvements persist for larger models or more recent model generations? How does the benefit vary with the underlying model’s search capability?

  4. The continued-training evaluation relies solely on Search-R1, leaving applicability to stronger search agents unclear. Can you evaluate stronger search-agent checkpoints and additional search frameworks under matched retrieval and training budgets?

  5. Relative to SSP and KG-Path, what benefit does the full framework provide beyond evidence-acquisition reward alone? Can a comparison using identical KG paths and generated questions isolate the contributions of corpus verification and evidence utilization?

Reviewer_6tWAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Evidence-in-the-Loop, a self-evolving framework for training search agents with knowledge graphs and corpus evidence. The method retains KG paths whose relations are supported by retrieved passages and uses these paths to guide question generation. The verified passages also support two process rewards for evidence acquisition and evidence utilization. These rewards distinguish incorrect trajectories, while correct answers receive full credit. Experiments across seven QA benchmarks and five model configurations show improvements in average performance, particularly on multi-hop questions. A separate experiment also evaluates the training value of the generated questions.

Critical strengths
  1. Grounding intermediate supervision in supporting passages addresses a meaningful limitation of entity-based rewards, since mentioning a relevant entity does not necessarily indicate successful retrieval.
  2. The evaluation covers multiple model families and initialization settings, with consistent improvements in average EM and F1 over KG-Path.
  3. The experiment using only outcome rewards provides useful evidence that the generated questions improve downstream training. With equally sized datasets, average multi-hop F1 increases from 30.4 for SSP-generated questions to 35.3.
Critical weaknesses
  1. The reliability of evidence verification is insufficiently established. Section 4.1 states that retrieved passages are verified for relational support, but the verification procedure and acceptance criteria remain unclear. No independent assessment of verification accuracy is reported. This matters because the resulting labels determine both path selection and process rewards. The 70.9% validity rate in Table 3 measures acceptance by the training filter and does not independently establish factual correctness or unique answerability.

  2. The contribution of evidence grounding to question generation is not clearly isolated. The proposed pipeline also introduces difficulty filtering, curriculum scheduling, and a revised Proposer prompt. The comparison with KG-Path therefore evaluates several changes simultaneously. Table 4 supports the usefulness of the generated questions relative to SSP, but omits KG-Path, the closest baseline. A comparison that controls these additional changes would better establish the specific benefit of grounding.

  3. The interpretation of the process rewards needs stronger empirical support. Among incorrect trajectories, partial credit depends on matching a small set of predefined passages and reusing an entity alias in the immediately following query. These criteria can miss useful progress through alternative evidence or less direct search sequences. The paper does not quantify this limitation or whether the rewards reduce the illustrated failure patterns. Moreover, evidence utilization adds only 0.3 points in average EM and F1 over acquisition alone, with no repeated runs or uncertainty estimates.

  4. Important experimental details remain unspecified. Appendix C reports approximately 150 to 200 optimization steps without stating the exact budget for each method, despite the claim of strictly matched training settings. Evaluation splits and sample sizes are also unclear. In addition, the costs of evidence grounding, verification, and difficulty filtering are not reported. These omissions limit reproducibility and assessment of the improvements under comparable computational budgets.

Questions for authors

Please refer to the summary of weaknesses.

Reviewer_WcmGOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an evidence-grounded self-evolving framework for training search agents. It verifies KG paths against a text corpus and introduces evidence acquisition and utilization rewards to provide intermediate supervision.

Critical strengths

The paper identifies a meaningful limitation of entity-based supervision.

The method connects structured KG relations with retrievable textual evidence.

Critical weaknesses
  1. Passage matching is an overly restrictive proxy for evidence acquisition.

Semantically equivalent evidence from other passages receives no credit, while retrieving a designated passage does not ensure that the agent correctly interprets it. T

  1. The utilization reward does not establish actual evidence use.

The reward relies on consecutive retrievals linked by an intermediate entity appearing in the subsequent query. However, this structural correspondence does not establish that the preceding evidence actually informs the next search

Questions for authors

Why should mentioning an intermediate entity in the next query be sufficient to establish genuine evidence utilization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.