acceptodds
Under review as a conference paper at ICLR 2027

SYNERGATE-RAG: Learning Graph-Grounded Process Rewards for Multi-Hop Question Answering

Abstract

Multi-hop retrieval-augmented generation requires a reader to connect facts across evidence sources, yet final-answer supervision provides little information about the validity of individual reasoning steps. A trajectory with a correct final answer can therefore receive full credit despite unsupported intermediate claims or disconnected evidence use. Knowledge graphs provide explicit factual constraints, but incomplete coverage limits their reliability as process supervision. We introduce SynerGate-RAG, which learns continuous graph utility from source matching, citation relevance, and validated contradiction witnesses. Unlike graph process reward models (PRMs) based on algorithmic correctness or implicit outcome signals, our framework learns explicit, source-grounded utility and controls its contribution to graph-text process rewards. The labels preserve partial support and separate unavailable evidence from contradiction. An evidence-conditioned PRM learns this utility, while a stepwise gate combines graph and text rewards under a source-availability mask. The resulting rewards guide reader training alongside answer correctness. Experiments on multi-hop question answering benchmarks show improved answer quality and support the complementary roles of reward learning and adaptive allocation. SynerGate-RAG thus turns incomplete structured knowledge into selective process supervision without reward-model calls at inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Multi-Hop Question Answering, Knowledge Graphs, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JozoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SynerGate-RAG, a framework for multi-hop question answering that combines graph-grounded and text-based process rewards. It constructs continuous graph-utility labels and trains a graph process reward model, then uses an adaptive gate to combine graph and text feedback for PPO training. Experiments on three multi-hop QA benchmarks demonstrate improved answer quality over chosen baselines.

Critical strengths
  1. The method achieves strong results across three multi-hop QA benchmarks.
  2. The controlled reward ablations and independent evaluations of the PRM and gate provide useful empirical evidence.
Critical weaknesses
  1. The method is overly complicated and relies heavily on heuristic design choices. In particular, the graph-utility target based on the product of citation matching and relevance rates, and the gate trained to predict the graph-utility value, lack sufficiently convincing justification. Moreover, treating unavailable graph evidence as neutral does not fundamentally address the challenge of verifying reasoning under incomplete KGs.
  2. The overall contribution is difficult to disentangle from the complexity of the system, which involves multiple supervision stages and reward models. While the experiments demonstrate effectiveness, it remains unclear whether the benefits justify this complexity. The presentation is also a little bit difficult to follow, making the central methodological contribution harder to appreciate.
Questions for authors
  1. What is the end-to-end training and evidence-preparation cost of the complete pipeline, including teacher annotation and PRM training? This would help assess whether the improvements justify the additional system complexity.
  2. Can the authors provide comparisons with relevant methods that you introduced in related works such as Search-R1 and GraphRAG-R1, or clarify why these baselines are not directly comparable?
2 comments

Thank you for recognizing the answer-quality improvements and the controlled evaluations. We appreciate your concern that the contribution and its practical cost should be easier to assess. The revision connects each design choice to a control and adds resource and related-system comparisons.

1. Design choices and the central contribution.

The method learns signed process utility from incomplete source evidence and uses it for stepwise graph-text reward allocation during reader training. Appendix H.1 maps its components to existing ablations: macro F1 is 56.6 for the full method, versus 49.0 for direct graph rules, 50.8 for a hard-label PRM, 49.9 for fixed mixing, and 52.6 for terminal-only reward placement. These controls distinguish learned scoring, continuous supervision, adaptive allocation, and temporal credit assignment.

We also agree that the product target and gate require more than an intuitive explanation. Table H5 finds product-positive/joint-zero cases in 0.62% of 5,997 eligible steps (37 cases). In Table H6, product and joint targets yield reader macro F1 of 57.1 and 56.8, with slightly better process metrics for the joint target. We present this as sensitivity to the target choice, not an optimality claim. The gate predicts ∣zt∣|z_t| rather than signed utility, preserving both supportive and contradictory feedback. Table H7 adds a development-tuned constant: learned allocation achieves 85.1% ranking accuracy and 69.4% top-1 EM, versus 82.1% and 67.0%. Table H8 separately examines channel error.

2. What the method addresses under incomplete graphs.

Your scope concern is well taken. The mechanism prevents a missing relation alone from creating a negative supervision target; it does not reconstruct missing facts. When no admissible source slot remains, graph weight becomes zero and textual feedback remains available. Negative offline targets require accepted contradiction witnesses. The revised discussion frames this as selective process supervision under incomplete evidence, and the Limitations section explains the boundaries imposed by coverage, linking, and weak-label quality.

3. End-to-end preparation and training cost (Q1).

Tables H9-H11 separate the pipeline stages. Shared index/source preparation costs 21.4 GPU-hours and question-specific evidence preparation costs 8.6, totaling 30.0. Reader SFT, graph PRM training, gate features/fitting, and PPO/replay cost 12.8, 8.1, 2.3, and 41.7 GPU-hours, totaling 64.9; development evaluation adds 4.3. The overall recorded GPU budget is 99.2 GPU-hours. Teacher generation uses 11,842,000 input and 3,914,000 output API tokens; semantic annotation uses 7,906,000 input and 986,000 output tokens. Stage wall times are reported separately, without treating their sum as end-to-end elapsed time.

Table H10 now agrees with this full-method ledger. Updated baseline cost totals were not supplied, so we leave them unreported rather than infer a cost ratio. The supplemental answer-quality comparison is 57.1 macro F1 for the full method, 50.2 for fixed mixture, and 40.6 for answer-only; these differ from the main-table three-run means. Deployment requires no reward-model calls. Table H11 reports full-method median latency of 6.30 s cold and 4.66 s warm, versus 6.02/4.39 s for answer-only and 6.19/4.55 s for fixed mixture. Figure 3(b) measures rollout-budget efficiency, excluding offline preparation and supervision costs.

4. Search-R1 and GraphRAG-R1 (Q2).

We agree that both are relevant comparisons; the existing R1-Searcher row is not a substitute. Table H12 now reports macro F1 of 57.1 for SynerGate-RAG, 54.4 for Search-R1, and 55.5 for GraphRAG-R1. Both Search-R1 and GraphRAG-R1 use Qwen2.5-7B, while SynerGate-RAG uses Llama-3-8B-Instruct. We distinguish these system-level results from Table 2's matched reward ablations: backbones, retrieval policies, and system configurations can contribute to the gaps, whereas Table 2 fixes evidence, initialization, and rollout budget to isolate reward design. This distinction allows the additional comparisons to inform practical performance without attributing every system-level difference to our reward mechanism.

Thank you for the detailed rebuttal and the additional experiments. I appreciate the new analyses, baseline comparisons, and cost breakdown, which help clarify and strengthen the empirical evaluation. However, my main concern about the method relies on a complex pipeline with several heuristic design choices, still remain. Thus, I would like to maintain my original rating.

Reviewer_Ta4iOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SynerGate-RAG, which learns continuous graph-grounded process rewards and uses an adaptive gate to combine graph and text feedback during reader training.

Critical strengths

The method explicitly distinguishes missing evidence from contradiction.

The combination of process reward learning and adaptive reward allocation is clearly formulated.

Critical weaknesses
  1. The graph-utility target does not guarantee joint citation support.

The proposed target multiplies the marginal citation-verification rate by the marginal relevance rate. These rates can both be positive even when no citation is simultaneously verified and relevant.

  1. The gate does not directly estimate the relative reliability of graph and text rewards.

The gate predicts graph-verdict magnitude using graph density, entity-link resolvability, and reference-model surprisal. However, these features do not establish whether graph feedback is more reliable than textual feedback for a particular step.

Questions for authors

Why use the product of marginal verification and relevance rates instead of a joint citation-level support measure?

1 comment

Thank you for identifying these two important distinctions. We agree that joint citation support and relative channel reliability should not be inferred from the target definitions alone. The revision clarifies the objectives and adds targeted comparisons.

1. Why use marginal verification times relevance?

The product combines two step-level summaries: how much of the declared citation set matches supplied source facts, and how much serves a semantic role in the current step. It provides graded credit and becomes zero if either summary is zero. This is a composite weak utility, not a probability of correctness or a guarantee that the same citation passes both checks. Your counterexample is valid. The joint target jt=∑iVtiLtimax⁡(1,nt)j_t=\frac{\sum_i V_{ti}L_{ti}}{\max(1,n_t)} is a natural alternative and also retains graded credit; no independence assumption is used to equate the two.

Appendix H.3 now examines both the incidence and the practical effect of this distinction. Among 5,997 source-ready, non-contradictory steps, 37 are product-positive but joint-zero: 0.62% of eligible steps and 0.84% of the 4,398 positive-product steps (Table H5). These rates quantify this particular failure pattern in the audited population, rather than establishing that the product is an exact support measure.

Table H6 compares the two targets. Product versus joint gives 83.6% versus 83.9% step-preference accuracy, 74.8% versus 75.0% first-error accuracy, and 0.110 versus 0.108 utility MAE against the same independent support ratings. On the supplemental evaluation, reader macro F1 is 57.1 versus 56.8; these are separate from the main-table three-run means. The joint target is slightly better on process metrics, while the product is slightly better on answer quality. We therefore treat this as sensitivity evidence, not proof that the product is intrinsically superior. The revised main text and Appendix B explicitly state the target's semantics.

2. What justifies graph-verdict magnitude as an allocation proxy?

The gate is supervised by ∣zt∣|z_t|, not signed ztz_t. An accepted contradiction has zt=−1z_t=-1 and ∣zt∣=1|z_t|=1, so informative negative evidence can receive substantial weight. The source mask determines whether graph feedback is admissible; the gate estimates its strength. We agree that strength alone does not establish relative graph-text reliability, and Section 3.4 now makes this distinction explicit.

The empirical justification has two parts. First, Table H7 compares allocation rules on the same frozen candidate bank. The learned gate achieves 85.1% correct-wrong ranking accuracy and 69.4% top-1 EM, compared with 82.1% and 67.0% for a development-tuned constant, and 77.9% and 64.6% for shuffled learned weights. Table 2 separately evaluates reader training, with macro F1 of 56.6 for the full method and 49.9 for the fixed mixture.

Second, Table H8 directly relates gate magnitude to channel error using a common independent rating scale and development calibration. The graph channel has lower error on 30%, 53%, and 76% of steps in the low-, middle-, and high-gate strata, respectively. These observations support the proxy's usefulness while showing why it should not be described as a calibrated reliability probability. Thank you for prompting us to separate that interpretation from the measured allocation benefit.

Reviewer_SK3pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SynerGate-RAG, a method for training multi-hop RAG readers with step-level process rewards. A PRM learns signed graph-utility labels derived from Wikidata citation checks. A gate then mixes this graph reward with a text-likelihood reward during PPO training. The reader is Llama-3-8B, evaluated on HotpotQA, 2WikiMultiHopQA and MuSiQue, with a reported macro EM/F1 of 45.9/56.6.

Critical strengths

The paper treats incomplete graph evidence carefully. A missing relation never produces a negative label, so negative labels require a validated contradiction.

The ablations in Table 2 are controlled: all variants share the same initialization, evidence and budget.

Critical weaknesses
  1. Weak motivation, not supported by experiments: The claim that correct answers can conceal disconnected reasoning (lines 34–35) is supported only by a 2020 XLNet result, with no evidence from the datasets used. The paper also never shows that the trained reader produces fewer unsupported steps. Table 3 evaluates the PRM, not the reader, and all reader results are EM/F1 only.

  2. Limited experimental scope: Only one reader, Llama-3-8B, is tested, so generalization to other scales or model families is unclear. There is also no analysis of computational cost. Figure 3(b) excludes offline supervision, and the paper reports no training compute, annotation cost or inference latency for the full pipeline.

  3. Writing: The writing is dense and hard to follow. GG, sis_i (lines 116–117), VtiV_{ti} and LtiL_{ti} (line 194) are not formally defined in the main text.

  4. Case studies: The case studies in Appendix D are three simple, correct, retrospectively selected examples, so they are inconclusive. I would like to see examples where the proposed algorithm succeeded and the baselines failed; and the proposed algorithm also failed.

Questions for authors
  1. Please formally define GG and sis_i in y=(s1,…,sT,a)y = (s_1, \dots, s_T, a), and state the values VtiV_{ti} and LtiL_{ti} can take.
  2. Can you give a real example from your datasets where a correct answer conceals disconnected reasoning?
  3. Can you evaluate whether the trained reader produces fewer unsupported steps than the baselines?
  4. What is the total computational cost of the pipeline compared with the baselines?
  5. Can you add case studies where a prominent baseline fails and your method succeeds, and where your method fails?
2 comments

Thank you for your careful review and for recognizing the treatment of missing graph evidence and the controlled ablations. Your comments helped us distinguish reader reasoning quality from reward-model quality and make the presentation more concrete.

1. Definitions and presentation (Q1).

Section 3.1 now includes a notation-and-domain table. GG is a finite set of source fact records with canonical entities/relations, typed values, qualifiers, and source-revision identifiers. Each step is st=(ρt,Ct,κt)s_t=(\rho_t,C_t,\kappa_t): reasoning text, declared citations, and an intermediate conclusion. Vti,Lti∈{0,1}V_{ti},L_{ti}\in\{0,1\} are indicators of deterministic source matching and accepted relevance, respectively. IRRELEVANT and UNKNOWN give Lti=0L_{ti}=0; technical annotation failures receive no training target. This makes the distinction between absent evidence, semantic uncertainty, and annotation failure explicit.

2. Reader-level evidence and motivation (Q2-Q3).

We agree that the original Table 3 evaluates the PRM, not whether the trained reader produces fewer unsupported steps. Appendix H.2 now reports a reader-output audit on 100 questions from each of HotpotQA, 2WikiMultiHopQA, and MuSiQue. Dataset-macro unsupported-step rates are 25.3% for answer-only training, 17.6% for the fixed mixture, and 11.7% for SynerGate-RAG (Table H2): reductions of 13.6 and 5.9 percentage points, respectively. Table H3 reports the same assessable coverage across methods within each dataset and slightly more steps per full-method output, so the improvement is not accompanied by fewer generated steps or lower assessable coverage.

Table H4 also compares each baseline and our method on questions both answer correctly. Unsupported-step rates are 17.0% versus 6.9% for answer-only versus SynerGate-RAG, and 11.8% versus 7.2% for fixed mixture versus SynerGate-RAG. Thus, final-answer correctness can coexist with unsupported intermediate claims, and the reader-level comparison remains informative within correct-answer subsets.

3. Computational cost and model scope (Q4).

Appendix H.5 separates evidence preparation, teacher generation, annotation, SFT, PRM/gate fitting, and PPO. Table H9 gives 30.0 GPU-hours for preparation, 64.9 for training, and 4.3 for development evaluation: 99.2 GPU-hours overall. Teacher generation and annotation use 19,748,000 input and 4,900,000 output API tokens. Updated baseline cost totals were not supplied alongside this ledger, so Table H10 leaves those cells unreported and we do not claim matched training costs. Figure 3(b) measures rollout-budget efficiency, excluding offline preparation and supervision costs.

Table H11 reports full-method end-to-end median latency of 6.30 s cold and 4.66 s warm, versus 6.02 s and 4.39 s for answer-only and 6.19 s and 4.55 s for fixed mixture. Deployment uses evidence preparation and the reader, without reward-model calls. Table H13 additionally evaluates Qwen3-8B: macro F1 is 54.3 for SynerGate-RAG versus 48.1 for fixed mixture and 39.6 for answer-only. This extends the evidence to another model family at the same 8B scale.

4. Case studies (Q2, Q5).

Appendix D now presents three concrete examples: a correct answer with an unsupported studio-location assertion (D4), contrasting date-ordering responses (D5), and an incorrect inference from a director's work location to birthplace (D6). These illustrate the distinction between answer correctness and intermediate support, together with a failure that can remain under incomplete evidence. Tables H2-H4 provide the reader-level evidence on the three datasets.

Thank you for your rebuttal. I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.