- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes TopoPRM, a pipeline that rewards the dependency structure behind a reasoning chain rather than only the final answer. A frozen rule-based extractor turns ordinary reasoning text into a typed support graph, from which two scores are derived: how well the conclusion is supported by earlier premises, including non-adjacent ones, and whether consecutive steps connect locally. These scores modulate the outcome, format, and length reward. During RL, an asymmetric credit estimation step adjusts credit separately within correct and incorrect groups, and clips the advantage so the structural term only pushes in the allowed direction. The same diagnostics are then used for distillation: the pipeline locates a defective step, asks a fixed teacher of the same size to revise it, and distills the accepted revisions with reverse KL. At deployment the student only generates ordinary text. Pass@5 results across three post-training lineages, controlled accuracy-length comparisons, an edge audit, and a reranking study against a semantic PRM are reported. A matched comparison of targeted and generic teacher revision is also provided.
Critical strengths
-
The support relations are recovered from ordinary text, so the deployed policy interface is unchanged, and the same graph supplies inspectable locations for teacher feedback. The Full vs. w/o ACE comparison in table 2 keeps TGD in both conditions, which makes the reported accuracy-length trade-off more informative than an outcome-only comparison alone.
-
The targeted-revision evidence in appendix C.3 is convincing: TGD and generic revision share the Stage-II initialization, teacher, acceptance gate, question sequence, and attempt budget. On the 675-item OlympiadBench set, TGD solves 169 questions against 143, with a paired question-bootstrap interval of [1.19, 6.67] points. The paper is also transparent about its boundaries, reporting moderate extractor F1, unsuccessful PRM fusion, and mixed transfer rather than treating topology as a semantic verifier.
Critical weaknesses
-
The main comparisons do not yet isolate the structural contribution. In table 2, Full includes TGD while Outcome Only and the topology/continuity removals stop after Stage II; the removals also redistribute reward weights, and w/o Continuity cuts the completion cap from 4,096 to 1,024 tokens in appendix B. I appreciate that these differences are disclosed, but the large drops cannot establish that topology or continuity is individually necessary. Full vs. w/o ACE is the closest matched comparison, yet it removes correctness clipping and the structural adjustment together, and does not distinguish ACE from the cited SCAE approach or from avoiding saturation in the scalar reward. This matters because one-sided auxiliary credit stratified by correctness is already present in SCAE, as the paper acknowledges.
-
The evidence for the proposed mechanism and its generalization remains limited. Online topology scores reduce to 0, 10/13, or 1 in appendix B.2, while extractor F1 is about 0.51–0.53 against model references. In appendix C.3, 218 of 336 accepted TGD outputs repeat the student text exactly, only five improve continuity, and none improves the raw topology score. These facts do not rule out useful supervision, but the accuracy gain alone does not establish dependency repair as the mechanism. The matched TGD comparison also increases response length on all four tasks, most sharply on AIME at 439 vs. 783 tokens. In table 2, the rounded 2.6-point macro-average gain over w/o ACE is dominated by three additional AIME answers, while the other three tasks change by only 0–0.3 points. Appendix E reports that the ordering is not stable under stochastic decoding, and that MATH-500 accuracy on Llama drops from 46.5 to 42.5. Together with the unreported per-item records behind table 1 and the absence of any uncertainty estimate across training seeds, the evidence is insufficient for a strong, general claim for structural supervision.
Questions for authors
-
Can you provide matched comparisons isolating each structural source, such as outcome+format+length vs. the complete structural reward at Stage II, or TGD with vs. without each source, with caps, training data, and update budgets fixed? The current controls change several factors at once, so it is hard to tell which component drives the gains.
-
What does eq. 4 add over SCAE, correctness clipping alone, or a non-saturating scalar reward built from the same structural signals? Appendix C.3 shows coefficient differences but not their downstream effects, so when ACE actually differs from SCAE in ways that matter for the gains remains unclear.
-
In the 118 accepted TGD revisions that change the text, how often is the flagged defect actually repaired, judged independently of the extraction rules? A breakdown into unchanged text, answer correction, and localized repair would clarify where the supervision comes from; also note that TGD and generic revision accept 336 vs. 379 attempts, so matched attempts do not mean matched numbers of supervised examples.
-
Can you release paired per-item results and uncertainty estimates across training seeds for the main Full vs. w/o ACE comparison + exact training data sources, counts, split and decontamination details, checkpoint identifiers, and evaluation configs for the distinct suites? I could not verify the anonymous artifact in this review environment; this is a request for accessible evidence, not a claim that the artifact is absent. A breakdown of training compute and teacher tokens would also help separate inference savings from total training cost.