acceptodds
Under review as a conference paper at ICLR 2027

Rewarding the Graph Behind the Chain: Topology-Aware Process Supervision for RL and Reasoning Distillation

Abstract

Long-form reasoning is produced as a linear token sequence, even when a conclusion depends on multiple non-adjacent premises. Outcome-only rewards do not expose this support structure, while scalar process scores do not explicitly identify which earlier premises support a later conclusion. We introduce TopoPRM, a hierarchical process reward model that recovers typed support graphs from ordinary reasoning text. The same dependency representation drives two forms of supervision: asymmetric credit estimation (ACE) assigns structural credit under final-answer correctness constraints, and topology-guided distillation (TGD) directs a fixed teacher to revise the current student's reasoning. Fresh student rollouts close the loop between structural diagnosis and online supervision. The deployed policy generates ordinary text without graph decoding or a verifier. Across four mathematics benchmarks, the complete pipeline improves pass@1 by up to 13.3 percentage points over outcome-only training while reducing mean response length by 3.8%-24.0%. Against the distillation-matched w/o ACE control, it gains 2.6 points in average pass@1 and uses 14.8% fewer tokens. These results connect correctness-constrained structural credit with more compact reasoning, using a shared representation for reward and targeted revision.

Primary area:
reinforcement learning
Keywords:
Process Reward Models, Reasoning Topology, Reinforcement Learning from Verifiable Rewards, Credit Assignment, On-Policy Distillation, Mathematical Reasoning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eXXtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes TopoPRM, a pipeline that rewards the dependency structure behind a reasoning chain rather than only the final answer. A frozen rule-based extractor turns ordinary reasoning text into a typed support graph, from which two scores are derived: how well the conclusion is supported by earlier premises, including non-adjacent ones, and whether consecutive steps connect locally. These scores modulate the outcome, format, and length reward. During RL, an asymmetric credit estimation step adjusts credit separately within correct and incorrect groups, and clips the advantage so the structural term only pushes in the allowed direction. The same diagnostics are then used for distillation: the pipeline locates a defective step, asks a fixed teacher of the same size to revise it, and distills the accepted revisions with reverse KL. At deployment the student only generates ordinary text. Pass@5 results across three post-training lineages, controlled accuracy-length comparisons, an edge audit, and a reranking study against a semantic PRM are reported. A matched comparison of targeted and generic teacher revision is also provided.

Critical strengths
  1. The support relations are recovered from ordinary text, so the deployed policy interface is unchanged, and the same graph supplies inspectable locations for teacher feedback. The Full vs. w/o ACE comparison in table 2 keeps TGD in both conditions, which makes the reported accuracy-length trade-off more informative than an outcome-only comparison alone.

  2. The targeted-revision evidence in appendix C.3 is convincing: TGD and generic revision share the Stage-II initialization, teacher, acceptance gate, question sequence, and attempt budget. On the 675-item OlympiadBench set, TGD solves 169 questions against 143, with a paired question-bootstrap interval of [1.19, 6.67] points. The paper is also transparent about its boundaries, reporting moderate extractor F1, unsuccessful PRM fusion, and mixed transfer rather than treating topology as a semantic verifier.

Critical weaknesses
  1. The main comparisons do not yet isolate the structural contribution. In table 2, Full includes TGD while Outcome Only and the topology/continuity removals stop after Stage II; the removals also redistribute reward weights, and w/o Continuity cuts the completion cap from 4,096 to 1,024 tokens in appendix B. I appreciate that these differences are disclosed, but the large drops cannot establish that topology or continuity is individually necessary. Full vs. w/o ACE is the closest matched comparison, yet it removes correctness clipping and the structural adjustment together, and does not distinguish ACE from the cited SCAE approach or from avoiding saturation in the scalar reward. This matters because one-sided auxiliary credit stratified by correctness is already present in SCAE, as the paper acknowledges.

  2. The evidence for the proposed mechanism and its generalization remains limited. Online topology scores reduce to 0, 10/13, or 1 in appendix B.2, while extractor F1 is about 0.51–0.53 against model references. In appendix C.3, 218 of 336 accepted TGD outputs repeat the student text exactly, only five improve continuity, and none improves the raw topology score. These facts do not rule out useful supervision, but the accuracy gain alone does not establish dependency repair as the mechanism. The matched TGD comparison also increases response length on all four tasks, most sharply on AIME at 439 vs. 783 tokens. In table 2, the rounded 2.6-point macro-average gain over w/o ACE is dominated by three additional AIME answers, while the other three tasks change by only 0–0.3 points. Appendix E reports that the ordering is not stable under stochastic decoding, and that MATH-500 accuracy on Llama drops from 46.5 to 42.5. Together with the unreported per-item records behind table 1 and the absence of any uncertainty estimate across training seeds, the evidence is insufficient for a strong, general claim for structural supervision.

Questions for authors
  1. Can you provide matched comparisons isolating each structural source, such as outcome+format+length vs. the complete structural reward at Stage II, or TGD with vs. without each source, with caps, training data, and update budgets fixed? The current controls change several factors at once, so it is hard to tell which component drives the gains.

  2. What does eq. 4 add over SCAE, correctness clipping alone, or a non-saturating scalar reward built from the same structural signals? Appendix C.3 shows coefficient differences but not their downstream effects, so when ACE actually differs from SCAE in ways that matter for the gains remains unclear.

  3. In the 118 accepted TGD revisions that change the text, how often is the flagged defect actually repaired, judged independently of the extraction rules? A breakdown into unchanged text, answer correction, and localized repair would clarify where the supervision comes from; also note that TGD and generic revision accept 336 vs. 379 attempts, so matched attempts do not mean matched numbers of supervised examples.

  4. Can you release paired per-item results and uncertainty estimates across training seeds for the main Full vs. w/o ACE comparison + exact training data sources, counts, split and decontamination details, checkpoint identifiers, and evaluation configs for the distinct suites? I could not verify the anonymous artifact in this review environment; this is a request for accessible evidence, not a claim that the artifact is absent. A breakdown of training compute and teacher tokens would also help separate inference savings from total training cost.

Reviewer_ZYfoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TopoPRM extracts a typed, forward dependency graph from ordinary reasoning text and uses conclusion support and local continuity to shape outcome-anchored rewards. Asymmetric credit estimation (ACE) constrains update signs by answer correctness and retains structural preferences when scalar rewards tie. Topology-guided distillation (TGD) asks a fixed teacher to revise localized defects in current student traces. The deployed student requires neither graph decoding nor a verifier. Evaluation includes broad pass@5 comparisons, a controlled Qwen3.5-9B pass@1 study, graph-edge audits, and supplementary robustness and teacher-revision comparisons. In Table 2, Full improves average pass@1 from 49.4 to 52.0 relative to the TGD-matched w/o ACE control while reducing average response length from 1,893 to 1,612 tokens.

Critical strengths
  1. The shared representation serves two concrete training functions. Using extracted dependencies both for reward assignment and for localized teacher feedback is a coherent contribution. Proposition 1 correctly identifies a useful boundary case: with two correct tied-reward traces, ACE can reinforce the structurally preferred trace when ordinary group-relative reward gives no signal. The logged example with nonzero ACE credit make this mechanism more concrete.

  2. There is meaningful controlled evidence and unusually explicit boundary reporting. The Full/w/o ACE contrast retains TGD in both arms. The supplementary matched teacher-revision experiment reports 169 versus 143 correct OlympiadBench answers out of 675, with a paired question-bootstrap interval of [1.19, 6.67] percentage points for TGD minus generic revision. The moderate edge F1, semantic-PRM fusion tie, and mixed cross-backbone results are also disclosed rather than hidden.

Critical weaknesses
  1. The specific contribution of graph topology is not adequately separated from other training changes. In Appendix B.2, the online topology score is only 0 for an empty graph, 1 for full conclusion support, or 10/13 otherwise; acyclicity and forward direction are guaranteed by construction, and reference-edge F1 has zero online weight. This is a coarse conclusion-support signal, not a graded measure of the full dependency topology. The w/o Topology and w/o Continuity rows also omit TGD and redistribute reward weights; the latter reduces the completion cap from 4,096 to 1,024. These are acknowledged pipeline-sensitivity experiments, but cannot establish the necessity of graph structure. A matched simple support/continuity heuristic or shuffled-edge control is needed, especially with extractor F1 around 0.51–0.53.

  2. The strongest controlled accuracy claim is fragile and incompletely replicated. Against w/o ACE, the improvements on GSM8K, MATH-500, and OlympiadBench are only 0.3, 0.2, and 0.0 percentage points; AIME contributes three additional solved problems and most of the macro-average gain. Appendix E shows that the greedy ranking does not survive decoding variation on the DeepSeek comparison and that Llama MATH accuracy drops. The evidence supports promising checkpoint-level efficiency gains light strongly than a general accuracy improvement.

Questions for authors
  1. Attribution Ablation: Under identical TGD, reward weights, data, completion caps, and training budgets, how does the full method compare with a simple conclusion-support/continuity reward and with graph edges randomized while preserving node and edge counts?
  2. Mechanism clarification: How often do graph-derived locations produce revisions different from a local continuity detector or generic correctness/length feedback? Can the matched TGD experiment be repeated with only the localization signal changed? This would directly test the value of the shared dependency representation.
Reviewer_LxdsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Nice paper extracting support graphs from mathematical reasoning traces.
  • Uses the graphs for reinforcement-learning rewards and localized teacher revision.
  • Reports shorter responses and mixed accuracy gains, with separate extraction and revision audits.
Critical strengths
  • Interesting and promising idea to get structural representation of reasoning, which seems useful.
  • The paper connects inspectable structural rewards with localized teacher feedback while retaining ordinary student inference.
Critical weaknesses

Main: -- The controls do not isolate graph information from correctness-sign clipping, coarse structural scores or additional teacher training.

  • Almost all of the primary macro accuracy gain comes from a few additional AIME answers. -- Authors don't have a metric for the potential interpretability or robustness you would get from the structured reasoning representation.

Secondary

  • Most accepted teacher revisions repeat the original text, and few improve the measured structure.
  • The online topology score is coarse, and transfer results are mixed.
Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.