- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies first-error localization in multi-step LLM reasoning. It introduces Progression-aware Reasoning Origin (PRO), a training-free pipeline combing incoming and outgoing verification, mismatch-triggered refinement, and detector-conditioned candidate attribution through local rewriting and reranking. The experiments cover GSM8K subset of ProcessBench, a constructed medical reasoning benchmark, and Number Triangle Summation.
Critical strengths
- It is useful to distinguish detecting failures and identifying its origin. Authors do not simply assume that the first rejected block must be the first erroneous block. Instead, they treat detection as evidence to a separate attribution stage. I believe this makes a lot of sense, because the final decision can reconsider earlier steps rather than being attached to the detector's initial judgment.
- The evaluation is decently comprehensive. It covers multiple reasoning settings and multiple backbones.
Critical weaknesses
- I believe the headline claim (source-attribution) is not completely established by the experimental design or formal analysis. The results feel like concerning a structural-exposure target rather than the annotated first-error problem. In particular, the impossibility statement refers to the rules that only observe the binary incoming verdicts, not every verifier with access to the full prefix. The authors acknowledge the difference between exposure and source localization, but the connection between its theoretical results and the latter task remains underexplored. Altho there are ablation studies, they still seems not isolate the contribution of rewrite evidence while keeping candidate proposal and reranking intact.
- The empirical advantage is modest on ProcessBench. The avg ProcessBench improvement over GoV is ~0.6 F1. On Qwen2.5-32B, both methods have 68.12% Error Acc, while PRO has slightly lower Correct Acc. Using the paper's own definition, this would give ~79.10 for PRO and 79.28 for GoV.
Questions for authors
Q1: Clarification of benchmark validity. How were the unmodified medical traces and injected first-error labels validated?
Q2: Validation of the current method. Does PRO retain its advantage under comparable inference budgets? A comparison that retains candidate proposal reranking but without rewriting evidence would help isolate the claimed contribution. Please also explain how the ablation (all components disabled) differs from GoV on the same backbone.
Q3: Clarification of repaired semantics. When evaluating a rewritten source candidate, is the subsequent reasoning kept unchanged, regenerated, or assessed under a different consistency standard? How does the method handle a correct source repair that conflicts with later downstream steps because those steps propagated the original error?
Q4: Explanation and clarification of the theoretical claim. Are the incoming signal intended to represent ideal semantic support or LLM judgements? Can you provide a concrete trace with an explicitly justified first-error label that showcases the claimed source-exposure gap under the intended interpretation?
3 comments
Response to Reviewer ZH7a(2/3)
Q1: How were the unmodified medical traces and injected first-error labels validated?
Response: Thank you for your question. Clean traces are MedReason reference solutions kept verbatim (Wu et al., 2025). Error traces are built by a MedPRMBench-style blueprint injection (Wu et al., 2026): an error is inserted at a chosen step using one of five templates (factual/drug/disease/anatomy swap, omitted safety check, redundancy, skipped prerequisite, omitted differential). The base chain is clean and the error lands at exactly one known step, so the first-error label is that step's index by construction, and there is no annotation to validate. The injection type and step index are recorded for auditing, and label/prediction semantics are checked throughout the pipeline.
Caveat: template injection is a controlled proxy for natural errors. It guarantees the label but does not sample the full distribution of organic mistakes. That is why we also report ProcessBench and Number-Triangle, which carry their own labels.
Q2: Does PRO retain its advantage under comparable inference budgets?
Response: Thank you for your question. The "all-disabled" ablation is not GoV. They are different detectors. GoV is pure-forward, presumes the prefix is correct ([Correct Preceding Steps]), and stops at the first rejection. Our detector-stage reports , the earliest incoming- or outgoing-side exposure (forward and backward fused, with edge refinement). The two numbers measure different targets:
Table 1. Detector-stage vs. GoV.
| Method | Error Acc | F1 |
|---|---|---|
| GoV (forward-only) | 68.12% | 79.28% |
| PRO detector-stage (disabled) | 56.28% | 68.64% |
| PRO (full) | 68.12% | 79.10% |
The gap is entirely Error Acc (56.28 vs 68.12). Reporting the earliest exposure instead of a first rejection produces exactly this gap, and it is not a weaker detector.
On budget, we have not matched GoV's call budget (PRO costs ~2.7x GoV on MedReason). Our claim is that the attribution stage drives the gain, not that PRO wins at equal compute. The ablation asymmetry supports this: removing rerank costs −21.3 F1, removing backward −9.8. On ProcessBench PRO and GoV tie (79.10 vs 79.28); the +9.98 gain appears on MedReason, where the source–exposure gap is largest (forward GoV misses the source on 72.4% of erroneous traces). That concentration is what attribution predicts, not generic compute.
On isolating rewrite, we have not separated rewrite from rerank. "w/o top-k rerank" removes both together (57.78) and only brackets the contribution. A "rerank w/o rewrite" run, which keeps proposal and rerank while withholding rewrite evidence, isolates it, and we will add it to Section 5.6.
References
Wu, J., Deng, W., Li, X., Liu, S., Mi, T., Peng, Y., Xu, Z., Liu, Y., Cho, H., Choi, C.-I., Cao, Y., Ren, H., Li, X., Li, X., & Zhou, Y. (2025). MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs. arXiv preprint arXiv:2504.00993.
Wu, L., Zheng, X., Zhai, W., Wang, W., Ren, X., Zhang, Z., Wei, H., & Zhao, B. (2026). MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning. arXiv preprint arXiv:2604.17282.
Response to Reviewer ZH7a(3/3)
Q3: When evaluating a rewritten source candidate, is the subsequent reasoning kept unchanged?
Response: Thank you for your question. Only the suspicious step is rewritten (rewrite_candidate: [Verified Preceding Steps] + [Suspicious Step]); subsequent steps are never regenerated. The rewritten step is then judged by repair_backward, which checks the step's own correctness independently of the still-wrong downstream. The prompt says not to mark the rewrite incorrect merely because the original next step still carries the old value. A correct source repair that conflicts with downstream propagated errors is therefore evaluated on its own merits, not against the corrupted continuation.
Q4: Are the incoming signals ideal semantic support or LLM judgments?
Response: Thank you for your question. with semantic-logical support, and (Section 3). These are ideal indicators; the LLM verifier is the empirical approximation. Theorem 2 is about the ideal case: two traces can be indistinguishable from binary incoming verdicts yet differ in structural exposure.
A real ProcessBench trace (id gsm8k-7, label = 1, GoV rejection = 3):
Problem. The Adams family made 7995 cookies, of which 2595 are rainbow and 3075 are oatmeal; how many chocolate chip? Step 0. …subtract rainbow and oatmeal from the total. (correct) Step 1. . ← first error (should be ) Step 2. . (consistent with the wrong ) Step 3. Therefore the family made chocolate chip cookies. ← GoV rejects here
The error is made at Step 1, stays locally plausible to a forward checker (the operation is right; only the value is off), and propagates into Steps 2–3. Forward verification first flags Step 3, so the error seen is not the error made. That is the mismatch set : steps that pass incoming checking but fail to support their continuation.
Response to Reviewer ZH7a(1/3)
We thank the reviewer for the careful read. The central concern, which is that PRO may measure structural exposure rather than the annotated first error, is fair. We answer the four questions directly.
Response: Thank you for your comment. We agree the headline must be read precisely, and we have tightened it. Source attribution is an algorithmic contribution, not something the theory proves: Theorems 1–2 characterize structural exposure (earlier than forward rejection; not recoverable from binary incoming verdicts alone), and Section 3 explicitly separates τexp from the final source target f(q,B). The impossibility statement is scoped to detectors that observe only binary incoming verdicts, not to real LLM verifiers with full-prefix access. For the ablation, we agree "w/o top-k rerank" removes rewrite and rerank together; we will add a "rerank w/o rewrite" run to isolate the rewrite evidence in Section 5.6.
Response: Thank you for your comment. We report this transparently. On ProcessBench the advantage is modest (79.10 vs 79.28 F1, equal 68.12% Error Acc) because the source–exposure gap there is small, since forward GoV misses the true source on only 31.9% of erroneous traces. On MedReason, where the gap is large and forward GoV misses the true source on 72.4% of erroneous traces, PRO improves by +9.98 F1 (51.20 vs 41.22). The improvement tracks the size of the source–exposure gap, which is exactly what progression-aware attribution predicts rather than generic compute.