- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes dependency-grounded process supervision for mathematical and physical reasoning. SaT-3L-SM proposes candidate text spans, which Qwen3.8-27B groups into reasoning steps while extracting inputs, assumptions, outputs, and dependencies. A subsequent attribution stage distinguishes newly introduced mistakes from inherited errors, including mixed cases. These annotations train a Qwen3-4B process reward model to predict correctness, error role, and intrinsic error type. Evaluations cover PRM800K, an annotated physics subset, and ProcessBench. In the controlled shared-label study, full-trace supervision improves detection over first-error masking. The hierarchy achieves similar Error AP to full-trace binary supervision while improving role macro-F1 over a correctness-based sequential rule. Additional analyses examine reasoning length and error density, alongside downstream answer voting and SFT data selection, with both improvements and regressions across settings. The PRM800K and physics evaluations use reference labels generated by the proposed annotation pipeline.
Critical strengths
-
The dependency-based distinction between consumed invalid content and locally introduced mistakes is well motivated and more expressive than assigning propagation solely by step order. The thin-film example clearly illustrates correct use of an unaffected output, inherited invalidity, and a mixed error.
-
The experimental analysis is unusually transparent. The paper includes shared-label controls, length and error-density slices, role confusion matrices, explicit negative findings, class counts, training configurations, score mappings, and bootstrap procedures. It also honestly reports the sparse-error degradation and GSM8K localization loss.
-
The shared-label comparison provides quantitative support for improved error-role diagnosis. The learned role head raises macro-F1 from 66.24% to 70.80% over the correctness-based sequential rule (+4.56 points, 95% CI [3.37, 5.72]), while the hierarchical and full-trace binary models have similar detection AP (Table 2). This supports the value of explicit role supervision beyond correctness scores and step order.
Critical weaknesses
-
The practical value of the type head remains insufficiently demonstrated. In Table 2(a), adding type supervision to correctness+role leaves Error AP essentially unchanged (86.59% versus 86.58%) and increases role macro-F1 by only 0.08 percentage points. Although the head recognizes some error categories, its macro-F1 over the six types with test support is 26.67%. On this test set, F1 is reported as 0.00% for assumption/condition and answer mismatch errors, with 56 and 13 reference examples, respectively (Table 14). These scores are conditional on reference new errors with known types. The downstream studies neither use type predictions directly nor isolate the contribution of type supervision. The authors acknowledge that type prediction needs improvement and leave type-guided correction to future work, but the current results provide limited evidence for the practical benefit of this component.
-
The downstream voting results do not establish a consistent advantage over the three-class PRM. Figure 4(a) and Table 19 show lower accuracy for the hierarchical model at N = 2–5 and small gains at N = 6–8, while the prose in Section 5.4 emphasizes N = 8. A more balanced discussion across budgets would be helpful. Although these results support the utility of correctness scores for voting, a matched downstream ablation is needed to isolate the contribution of role and type supervision.
-
The organization of the results section makes it difficult to connect individual findings with their experimental settings. For example, the analyses of reasoning length and first-error position in Table 4 are discussed in separate sections, with other comparisons in between. Details on checkpoint selection, scoring rules, and related analyses are distributed across Appendices C–E, requiring repeated cross-referencing to interpret the results. Grouping related analyses and providing a consolidated summary of the main experimental settings would improve the coherence and readability of the presentation.
Questions for authors
-
What fraction of traces or steps were rejected as unverifiable, malformed, or otherwise ineligible? Please provide a human audit of dependency extraction, role attribution, and intrinsic types, including agreement rates and common failure modes.
-
Could the authors report how frequently mixed errors and correct uses of unaffected outputs occur in the evaluation sets, and how the model performs on these cases? This would help connect the aggregate results to the distinctions illustrated in Figure 2.
-
Appendix E.3 leaves open whether correctness-only decoding would recover the GSM8K loss. Could the authors compare the hierarchical model's current correctness–role decoder with decoding based only on its error probability, using the same checkpoint, inputs, and threshold of 0.5? Reporting localization accuracy, correct-trace acceptance, and F1 on all four ProcessBench subsets would help isolate the decoder's contribution.
-
The threshold is tuned on the validation set (Appendix E.4), but the rule still treats only the first predicted error as new. Could the authors also test a rule that uses the error probabilities and how they change across steps, and allows later errors to be labeled as new? It would be useful to see whether the role head still has an advantage over this less restrictive baseline.
5 comments
Response to Reviewer Xynp (2/3)
Comment 2 · Data screening and independent review
Q1. Screening and audit. These exclusion rates use the source-checked pools. Evaluation conclusions apply to the retained, fully evaluable traces.
| Pool | Excluded | Recorded exclusions |
|---|---|---|
| PRM800K train/validation, before splitting | 31,327/73,797 (42.45%) | 29,493 hard rejections, 1,674 held for review, 160 without usable attribution |
| PRM800K test | 1,282/2,659 (48.21%) | 19 unverifiable, 1,183 other hard rejections, 75 held for review, five without usable attribution |
| Physics | 32/667 (4.80%) | Eight hard rejections, ten held for review, 13 held after cessation review, one without usable attribution |
Of the five PRM800K test traces without usable attribution, two failed output validation and three timed out.
In the independent audit of 100 PRM800K questions, the two annotators agreed on 97.08% of four-way roles (κ = 0.954, 616 jointly determinate steps). On 617 final determinate steps, the automatic reference agreed with humans on 84.28% of correctness decisions and 75.85% of four-way roles. Of the 61 human-identified mixed errors, the automatic reference labeled 3 as mixed, 7 as pure new, 34 as propagated, and 17 as correct. Under the training mapping, mixed and pure new both receive the new-error role, so 10/61 receive a new-error label. Human type agreement was 133/153 (86.93%) when both annotators found a local error. On the final 153 human-typed steps, the automatic reference supplied a type for 69 and matched 41 of those. One recurring mismatch was assigning derivation/logic to method/formula errors.
In a stratified blind audit of 80 physics steps, two reviewers agreed on 71/75 jointly determinate four-way roles (94.67%, κ = 0.947). After adjudication, automatic labels agreed with humans on 61/76 correctness decisions (80.26%) and 51/76 four-way roles (67.11%). The sets of source steps matched exactly on 61/78 steps with determinate source references. The cases in Q2 illustrate why matching the source step alone can miss an error in the extracted content or its causal attribution.
Q2. Mixed errors and unaffected outputs. The automatic PRM800K reference marks 295/9,073 steps (3.25%) as mixed, across 208/1,377 traces. The hierarchy flags 283 as errors and predicts new for 99. The independent mathematics audit identifies 61/617 determinate steps as mixed. On these steps, direct and explicit-IO training give 68.85% and 71.31% recall as new (+2.46 points [−0.94, 5.93]). In a targeted audit with premise links and local judgments fixed, false propagation predictions on unaffected uses are 3/3 for the step-level rule and 2/3 for the output-level rule. Misses on genuine propagation are 0/11 and 1/11. We restrict the mathematics rule comparison to the targeted audit, whose cases were selected for the comparison. An evaluation-wide frequency would require a representative sample.
In physics, the human audit identified 13 mixed errors. The automatic reference labeled six as mixed, six as pure propagation, and one as correct. The hierarchy flags 12/13 as errors and predicts new for 3/13. The audit found four correct reuses of unaffected outputs, three in the screen-positive stratum and one in the screen-negative stratum. This gives an exploratory inverse-sampling-weighted estimate of 3.52% in the physics evaluation set. Treating unresolved judgments as all absent or all present gives 3.50–4.00%. This range captures sensitivity to unresolved adjudications alone. Each evaluated PRM accepts 2/4 of these steps.
In U058, the extractor bundled a valid oscillation claim with the incorrect claim that the object never stops at the endpoints. The target used the oscillation claim alone, but the automatic label marked it as propagated. In U118, the extractor correctly isolated the conclusion that 8 m/s² was unattainable, which the target reused. The automatic attribution still marked that valid conclusion as the source of propagation.
Response to Reviewer Xynp (3/3)
Comment 3 · Same-model decoding and a stronger score-only rule
Q3. Same-model ProcessBench decoding. Both decoders use the same hierarchical checkpoint and inputs and flag the first step with a score strictly above 0.5. Correctness-only uses the error probability. Joint decoding weights it by the probability of new conditional on the two error roles. Joint decoding increases correct-trace acceptance on all four subsets. All four localization intervals include zero. Each cell gives correctness-only → joint (%) and the paired difference in points with a 95% interval. We resampled question clusters within each subset 2,000 times.
| Subset | Exact first-error localization | Fully correct trace acceptance | F1 |
|---|---|---|---|
| GSM8K | 55.56 → 54.11 (−1.45 [−3.85, 0.49]) | 91.19 → 91.71 (+0.52 [0.00, 1.90]) | 69.05 → 68.06 (−0.99 [−2.94, 0.55]) |
| MATH | 58.92 → 57.91 (−1.01 [−2.73, 0.67]) | 75.86 → 77.83 (+1.97 [0.74, 3.36]) | 66.33 → 66.41 (+0.08 [−1.12, 1.28]) |
| OlympiadBench | 54.16 → 54.46 (+0.30 [−0.80, 1.49]) | 53.69 → 59.00 (+5.31 [3.07, 7.74]) | 53.92 → 56.64 (+2.72 [1.43, 4.13]) |
| Omni-MATH | 54.55 → 54.55 (0.00 [−1.33, 1.34]) | 55.19 → 58.09 (+2.90 [0.90, 5.14]) | 54.86 → 56.26 (+1.40 [0.21, 2.70]) |
| Unweighted mean | 55.80 → 55.26 (−0.54 [−1.40, 0.28]) | 68.98 → 71.66 (+2.68 [1.79, 3.62]) | 61.04 → 61.84 (+0.80 [0.11, 1.53]) |
Q4. Score-only rule for later new errors. The rule predicts new at the start of a flagged segment or when the error score jumps by at least γ. We jointly select the flagging threshold and γ by maximizing three-way role macro-F1 on 13,930 validation steps for each seed. The jump grid runs from 0.05 to 0.50 in increments of 0.05. The flagging grid runs from 0 to 1 in increments of 0.01 and includes the original sequential threshold. Both seeds select γ=0.35, with flagging thresholds of 0.44 and 0.39. We freeze them before test scoring. The role head's gains over this rule on the human reference are positive in both seeds, with both intervals including zero. Values below are role macro-F1 (%), with paired 95% intervals in points.
| Reference, seed | Sequential | Score-only | Role head | Score-only − sequential | Head − score-only |
|---|---|---|---|---|---|
| Automatic, 42 | 66.24 | 66.45 | 70.80 | +0.21 [−0.57, 0.98] | +4.35 [3.34, 5.33] |
| Automatic, 43 | 65.50 | 66.20 | 70.87 | +0.70 [0.03, 1.39] | +4.68 [3.75, 5.61] |
| Human, 42 | 61.51 | 64.13 | 66.95 | +2.62 [−0.19, 5.54] | +2.82 [−0.66, 6.10] |
| Human, 43 | 62.61 | 63.30 | 65.23 | +0.69 [−2.07, 3.35] | +1.92 [−1.77, 5.54] |
Response to Reviewer Xynp (1/3)
Thank you for the detailed review and for asking us to clarify our evaluation protocol and what the results support.
Comment 1 · Type diagnosis, voting, and experimental settings
W1. What the type head adds. We present type prediction as an auxiliary diagnostic target in the revision. Adding role supervision to correctness + type improves conditional type macro-F1 on 153 independently typed steps by +2.96 [0.59, 5.48] and +3.85 [0.58, 6.78] points across two seeds. Metrics are conditioned on reference local-error steps with known types. The practical benefit of using predicted types for repair remains untested.
W2. Voting at every budget. Both PRMs beat majority voting on the same 1,319 GSM8K questions. The hierarchy trails the three-class PRM at N=2–5 and leads slightly at N=6–8, with all seven paired intervals including zero. In the matched Table 2 replay, differences from full-trace binary range from −0.38 to +0.08 points, with all intervals including zero. The revision limits the voting claim to correctness-score utility and reports the full budget breakdown in the appendix.
W3. Evaluation settings. We grouped trace length and first-error position in RQ3 and added a single selection and decoding table. These PRM800K slices use the existing three-class and hierarchical models at scores ≥0.5, with length cutoffs of ≤6, 7, and ≥8 steps and position cutoffs at 1/3 and 2/3 on erroneous traces. The existing hierarchy was selected by validation balanced accuracy. The shared-label and explicit-IO models were selected by validation Error AP. ProcessBench uses the same application hierarchy for both decoders at a strict >0.5 threshold.