acceptodds
Under review as a conference paper at ICLR 2027

The Error You See Is Not the Error You Made: Progression-aware Reasoning Origin for Reasoning Error Localization

Abstract

Verifying multi-step LLM reasoning requires more than determining whether a trace is correct: a useful verifier should identify where the reasoning first goes wrong. However, existing holistic methods provide little positional evidence, while forward sequential verification often treats the first rejected step as the error source. Under error propagation, this assumption can fail, since an earlier mistake may remain locally plausible and become observable only through its downstream consequences. We therefore rethink reasoning verification as a progression-aware error-source localization problem: rather than asking only where a reasoning trace first appears inconsistent, we ask which earlier step best explains how that inconsistency emerges along the trajectory. Based on this view, we propose Progression-aware Reasoning Origin (PRO), a training-free framework for first-error localization. PRO jointly models incoming support from the preceding context and outgoing compatibility with subsequent reasoning, selectively refines regions where these signals disagree, and finally performs detector-conditioned source attribution with intervention-based evidence to distinguish the true error origin from its propagated manifestations. We further formalize the gap between forward rejection and structural exposure, showing why incoming-side evidence alone is insufficient for reliable localization under error propagation. Experiments across open-form, medical, and structured reasoning tasks demonstrate consistent improvements over strong verification baselines, supporting progression-aware source attribution as a more faithful formulation of reasoning verification.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Reasoning Verification, First-Error Localization, Graph Reasoning, Training-free Verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZH7aOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies first-error localization in multi-step LLM reasoning. It introduces Progression-aware Reasoning Origin (PRO), a training-free pipeline combing incoming and outgoing verification, mismatch-triggered refinement, and detector-conditioned candidate attribution through local rewriting and reranking. The experiments cover GSM8K subset of ProcessBench, a constructed medical reasoning benchmark, and Number Triangle Summation.

Critical strengths
  • It is useful to distinguish detecting failures and identifying its origin. Authors do not simply assume that the first rejected block must be the first erroneous block. Instead, they treat detection as evidence to a separate attribution stage. I believe this makes a lot of sense, because the final decision can reconsider earlier steps rather than being attached to the detector's initial judgment.
  • The evaluation is decently comprehensive. It covers multiple reasoning settings and multiple backbones.
Critical weaknesses
  • I believe the headline claim (source-attribution) is not completely established by the experimental design or formal analysis. The results feel like concerning a structural-exposure target rather than the annotated first-error problem. In particular, the impossibility statement refers to the rules that only observe the binary incoming verdicts, not every verifier with access to the full prefix. The authors acknowledge the difference between exposure and source localization, but the connection between its theoretical results and the latter task remains underexplored. Altho there are ablation studies, they still seems not isolate the contribution of rewrite evidence while keeping candidate proposal and reranking intact.
  • The empirical advantage is modest on ProcessBench. The avg ProcessBench improvement over GoV is ~0.6 F1. On Qwen2.5-32B, both methods have 68.12% Error Acc, while PRO has slightly lower Correct Acc. Using the paper's own definition, this would give ~79.10 for PRO and 79.28 for GoV.
Questions for authors

Q1: Clarification of benchmark validity. How were the unmodified medical traces and injected first-error labels validated?

Q2: Validation of the current method. Does PRO retain its advantage under comparable inference budgets? A comparison that retains candidate proposal reranking but without rewriting evidence would help isolate the claimed contribution. Please also explain how the ablation (all components disabled) differs from GoV on the same backbone.

Q3: Clarification of repaired semantics. When evaluating a rewritten source candidate, is the subsequent reasoning kept unchanged, regenerated, or assessed under a different consistency standard? How does the method handle a correct source repair that conflicts with later downstream steps because those steps propagated the original error?

Q4: Explanation and clarification of the theoretical claim. Are the incoming signal intended to represent ideal semantic support or LLM judgements? Can you provide a concrete trace with an explicitly justified first-error label that showcases the claimed source-exposure gap under the intended interpretation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZH7a(1/3)

We thank the reviewer for the careful read. The central concern, which is that PRO may measure structural exposure rather than the annotated first error, is fair. We answer the four questions directly.

W1: The headline source-attribution claim is not fully established; the results concern structural exposure rather than the annotated first error.

Response: Thank you for your comment. We agree the headline must be read precisely, and we have tightened it. Source attribution is an algorithmic contribution, not something the theory proves: Theorems 1–2 characterize structural exposure (earlier than forward rejection; not recoverable from binary incoming verdicts alone), and Section 3 explicitly separates τexp\tau_\mathrm{exp} from the final source target f(q,B)f(q,\mathcal{B}). The impossibility statement is scoped to detectors that observe only binary incoming verdicts, not to real LLM verifiers with full-prefix access. For the ablation, we agree "w/o top-k rerank" removes rewrite and rerank together; we will add a "rerank w/o rewrite" run to isolate the rewrite evidence in Section 5.6.


W2: The empirical advantage is modest on ProcessBench (~0.6 F1; equal 68.12% Error Acc).

Response: Thank you for your comment. We report this transparently. On ProcessBench the advantage is modest (79.10 vs 79.28 F1, equal 68.12% Error Acc) because the source–exposure gap there is small, since forward GoV misses the true source on only 31.9% of erroneous traces. On MedReason, where the gap is large and forward GoV misses the true source on 72.4% of erroneous traces, PRO improves by +9.98 F1 (51.20 vs 41.22). The improvement tracks the size of the source–exposure gap, which is exactly what progression-aware attribution predicts rather than generic compute.

AuthorsOpenReview ↗

Response to Reviewer ZH7a(2/3)

Q1: How were the unmodified medical traces and injected first-error labels validated?

Response: Thank you for your question. Clean traces are MedReason reference solutions kept verbatim (Wu et al., 2025). Error traces are built by a MedPRMBench-style blueprint injection (Wu et al., 2026): an error is inserted at a chosen step using one of five templates (factual/drug/disease/anatomy swap, omitted safety check, redundancy, skipped prerequisite, omitted differential). The base chain is clean and the error lands at exactly one known step, so the first-error label is that step's index by construction, and there is no annotation to validate. The injection type and step index are recorded for auditing, and label/prediction semantics are checked throughout the pipeline.

Caveat: template injection is a controlled proxy for natural errors. It guarantees the label but does not sample the full distribution of organic mistakes. That is why we also report ProcessBench and Number-Triangle, which carry their own labels.


Q2: Does PRO retain its advantage under comparable inference budgets?

Response: Thank you for your question. The "all-disabled" ablation is not GoV. They are different detectors. GoV is pure-forward, presumes the prefix is correct ([Correct Preceding Steps]), and stops at the first rejection. Our detector-stage reports τexp\tau_\mathrm{exp}, the earliest incoming- or outgoing-side exposure (forward and backward fused, with edge refinement). The two numbers measure different targets:

Table 1. Detector-stage vs. GoV.

MethodError AccF1
GoV (forward-only)68.12%79.28%
PRO detector-stage (disabled)56.28%68.64%
PRO (full)68.12%79.10%

The gap is entirely Error Acc (56.28 vs 68.12). Reporting the earliest exposure instead of a first rejection produces exactly this gap, and it is not a weaker detector.

On budget, we have not matched GoV's call budget (PRO costs ~2.7x GoV on MedReason). Our claim is that the attribution stage drives the gain, not that PRO wins at equal compute. The ablation asymmetry supports this: removing rerank costs −21.3 F1, removing backward −9.8. On ProcessBench PRO and GoV tie (79.10 vs 79.28); the +9.98 gain appears on MedReason, where the source–exposure gap is largest (forward GoV misses the source on 72.4% of erroneous traces). That concentration is what attribution predicts, not generic compute.

On isolating rewrite, we have not separated rewrite from rerank. "w/o top-k rerank" removes both together (57.78) and only brackets the contribution. A "rerank w/o rewrite" run, which keeps proposal and rerank while withholding rewrite evidence, isolates it, and we will add it to Section 5.6.


References

Wu, J., Deng, W., Li, X., Liu, S., Mi, T., Peng, Y., Xu, Z., Liu, Y., Cho, H., Choi, C.-I., Cao, Y., Ren, H., Li, X., Li, X., & Zhou, Y. (2025). MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs. arXiv preprint arXiv:2504.00993.

Wu, L., Zheng, X., Zhai, W., Wang, W., Ren, X., Zhang, Z., Wei, H., & Zhao, B. (2026). MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning. arXiv preprint arXiv:2604.17282.

AuthorsOpenReview ↗

Response to Reviewer ZH7a(3/3)

Q3: When evaluating a rewritten source candidate, is the subsequent reasoning kept unchanged?

Response: Thank you for your question. Only the suspicious step is rewritten (rewrite_candidate: [Verified Preceding Steps] + [Suspicious Step]); subsequent steps are never regenerated. The rewritten step is then judged by repair_backward, which checks the step's own correctness independently of the still-wrong downstream. The prompt says not to mark the rewrite incorrect merely because the original next step still carries the old value. A correct source repair that conflicts with downstream propagated errors is therefore evaluated on its own merits, not against the corrupted continuation.


Q4: Are the incoming signals ideal semantic support or LLM judgments?

Response: Thank you for your question. Ii=1[(q,Pi−1)⊨bi]I _ i = \mathbf{1}[(q,\mathcal{P}_{i-1}) \models b _ i] with ⊨\models semantic-logical support, and Oi=1[bi⊢bi+1]O _ i = \mathbf{1}[b _ i \vdash b _{i+1}] (Section 3). These are ideal indicators; the LLM verifier is the empirical approximation. Theorem 2 is about the ideal case: two traces can be indistinguishable from binary incoming verdicts yet differ in structural exposure.

A real ProcessBench trace (id gsm8k-7, label = 1, GoV rejection = 3):

Problem. The Adams family made 7995 cookies, of which 2595 are rainbow and 3075 are oatmeal; how many chocolate chip? Step 0. …subtract rainbow and oatmeal from the total. (correct) Step 1. 7995−2595=45007995 - 2595 = 4500. ← first error (should be 54005400) Step 2. 4500−3075=14254500 - 3075 = 1425. (consistent with the wrong 45004500) Step 3. Therefore the family made 1425\boxed{1425} chocolate chip cookies. ← GoV rejects here

The error is made at Step 1, stays locally plausible to a forward checker (the operation is right; only the value is off), and propagates into Steps 2–3. Forward verification first flags Step 3, so the error seen is not the error made. That is the mismatch set M={i∣Ii=1,Oi=0}\mathcal{M} = \{i \mid I_i=1, O_i=0\}: steps that pass incoming checking but fail to support their continuation.

Reviewer_4ur8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Progression-aware Reasoning Origin (PRO), a training-free framework for first-error localization in multi-step LLM reasoning. Unlike existing methods that identify errors based on the first rejected step, PRO considers error propagation and aims to locate the original reasoning mistake rather than its downstream manifestations. Specifically, the method jointly models incoming support from preceding steps and outgoing compatibility with subsequent reasoning, selectively refines regions where these signals disagree, and employs intervention-based evidence to identify the true error source. The paper also provides a theoretical analysis of the limitations of forward sequential verification under error propagation. Experiments across open-form, medical, and structured reasoning tasks demonstrate the effectiveness of the proposed method over existing verification baselines.

Critical strengths
  1. The paper highlights that the first detected error may not be the actual source of the error. This distinction helps provide a more accurate understanding of how errors propagate through multi-step reasoning.
  2. PRO combines forward verification evidence, compatibility with subsequent reasoning, local refinement, and error-source attribution, offering a new perspective on traditional step-by-step verification.
  3. The proposed method mainly relies on inference-time verification and candidate attribution without requiring additional training of a verifier model, making it potentially applicable to different LLMs.
Critical weaknesses
  1. The proposed method assumes that considering both incoming and outgoing evidence can improve the identification of the original reasoning error. However, an inconsistency between consecutive reasoning steps does not necessarily indicate which step introduces the error. The authors should provide further analysis to clarify how bidirectional verification contributes to identifying the true error origin rather than merely detecting inconsistencies.
  2. Theorem 1 and Theorem 2 mainly demonstrate the limitations of forward-only verification in identifying structural exposure points. However, detecting an inconsistency earlier does not necessarily imply more accurate identification of the original error. The connection between these theoretical results and the final source-localization objective requires further clarification.
  3. The proposed method identifies potential error sources by rewriting candidate reasoning steps and examining whether local consistency can be restored. However, repairing a downstream step may also resolve an observed inconsistency, even when that step is not the original error source. Therefore, successful local repair does not necessarily establish causal responsibility for the initial error. Additional controlled experiments are needed to validate this assumption.
  4. The ablation results indicate that the Top-k reranking stage makes a substantial contribution to overall performance. It remains unclear whether the improvements mainly arise from the proposed progression-aware evidence modeling or from additional LLM-based candidate selection and reranking. Comparisons under matched inference budgets would help clarify the contribution of each component.
Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4ur8(1/2)

We thank the reviewer for the careful read. The four concerns reduce to one question: does PRO's improvement come from progression-aware source attribution, or from extra LLM compute and inconsistency detection? We answer with existing evidence, not new claims, plus two textual clarification (Theorem 2).

W1: How does bidirectional verification identify the true error origin rather than merely detecting inconsistencies?

Response: We agree, and PRO does not assume that a mismatch names the origin. The paper states this explicitly: a progression mismatch "does not yet reveal whether the underlying error lies in bib_i, in bi+1b_{i+1}, or in a finer reasoning step." Bidirectional verification serves source identification through a three-stage separation, which proceeds from detection to refinement to attribution.

The first step is to localize the failure to a transition rather than a step. Incoming evidence checks whether bib _ i is admissible from its preceding context, Ii=1[(q,Pi−1)⊨bi]I _ i=\mathbf{1}[(q,\mathcal{P}_{i-1})\models b _ i]; outgoing evidence checks whether bib _ i provides a valid basis for the next block, Oi=1[bi⊢bi+1]O _ i=\mathbf{1}[b _ i\vdash b _{i+1}]. Their combination yields three states: R\mathrm{R} (Ii=0I _ i=0), A\mathrm{A} (Ii=Oi=1I _ i=O _ i=1), and M\mathrm{M} (Ii=1I _ i=1, Oi=0O _ i=0). A forward-only checker cannot produce M\mathrm{M} because it never asks whether a locally plausible step still supports what follows, so it flags a step without seeing the transition where the trace first loses coherence. The M\mathrm{M} set is exactly where forward verification is insufficient, and it is what Theorems 1–2 show incoming-only signals cannot recover.

A mismatch at bi→bi+1b_i\to b_{i+1} still does not say which side of the boundary is wrong, so the next step is refinement: PRO decomposes the mismatched window (bi,bi+1)(b_i,b_{i+1}) into finer units and re-runs the same incoming/outgoing checks. If the inconsistency survives finer inspection, the surviving position becomes the exposure anchor τdet\tau_{\mathrm{det}}; if it dissolves, the original mismatch was a coarse-decomposition artifact. This turns "the failure is near here" into a concrete exposure point.

Even that anchor can sit downstream of the true error, because a wrong step can pass both checks locally (right operation, wrong value) and only break coherence several steps later, so the origin is decided by intervention rather than read off the exposure. PRO rewrites each shortlisted candidate under its preceding context and re-evaluates it with the same incoming/outgoing evidence; a candidate whose repair restores the downstream consistency is upstream of the exposure, and the rerank selects the earliest candidate that both repairs cleanly and best explains the failure. The bidirectional evidence is the signal against which this intervention is scored, so it is used to identify the origin, not merely to flag an inconsistency.

The ablations and stratification confirm this separation. The ablation (Table 1) shows attribution carries the localization: removing the backward signal (−9.8) removes the check that creates M\mathrm{M}, while removing the top-k rerank (−21.3) removes the attribution itself. The advantage over a forward-only baseline concentrates where the gap is large: PRO ties GoV on ProcessBench (79.10 vs 79.28), where the source–exposure gap is small, and gains +9.98 F1 on MedReason (51.20 vs 41.22), where forward GoV misses the source on 72.4% of erroneous traces. The same distance, measured per trace, bounds how well even PRO can localize. Accuracy falls from 85.7% at |d| = 0 to 28.6% at |d| ≥ 3 (Table 2), because the backward evidence must bridge more propagation steps.

Table 1. Ablation on ProcessBench (Qwen2.5-32B).

AblationF1Δ F1 (ablation − full)
Full PRO79.10—
w/o backward (inconsistency detection)69.28−9.8
w/o refinement64.51−14.6
w/o top-k rerank (source attribution)57.78−21.3

Table 2. Source-localization accuracy by source–exposure distance.

Distance |d| (exposure − source)SamplesPRO after rerank hits source
010585.7%
15877.6%
2944.4%
≥3728.6%
AuthorsOpenReview ↗

Response to Reviewer 4ur8(2/2)

W2: The connection between the theoretical results and the source-localization objective requires clarification.

Response: Thank you for your comment. Theorems 1–2 do prove properties of structural exposure (earlier than forward rejection; not recoverable from binary incoming verdicts alone), not source attribution. The paper already states this: Section 3 notes that "τexp\tau_\mathrm{exp} is not yet the final first-error localization f(q,B)f(q,\mathcal{B})" and that "these results do not by themselves recover the final source target f(q,B)f(q,\mathcal{B})." Source attribution is an algorithmic contribution, namely the top-k rerank with rewrite evidence. The ablations above validate it, and it is not a claim that follows from the theory.

The only change is a one-line clarification in Theorem 2: the impossibility concerns detectors that observe only the binary incoming-decision signals Ik∈{0,1}I_k\in\{0,1\} (already implied by the definition of IkI_k). This makes the abstraction explicit without altering the result.


W3: Does successful local repair establish causal responsibility for the initial error?

Response: Thank you for your comment. Rewriting is not a standalone discriminator in PRO. In the source-rerank stage, a candidate's "successfully repaired" verdict is auxiliary evidence only; the rerank prompt instructs the model that a candidate whose rewrite becomes correct does not automatically make it the first error. The final source selection is made over the full trace, preferring the earliest candidate that both admits a valid repair and best explains the downstream inconsistency.

A "rerank w/o rewrite" run, which keeps candidate proposal and rerank while withholding rewrite evidence, isolates this contribution, and we will add it to Section 5.6.


W4: Is the improvement from progression-aware modeling or from additional LLM-based candidate selection and reranking?

Response: Thank you for your comment. We agree the Top-k rerank contributes substantially, but its contribution is progression-aware rather than generic LLM compute. The rerank is the attribution step: it scores each candidate source by how well it explains the downstream inconsistency under the bidirectional incoming/outgoing evidence. It is not an independent reranker bolted onto a fixed detector, because the progression-aware evidence is what the reranker consumes, so the reviewer's two alternatives are not separable. The ablation structure supports this: removing the rerank (−21.3) removes the attribution mechanism itself, while removing the backward (outgoing) evidence (−9.8) removes the very signal the rerank relies on. A generic "extra compute" reranker would instead produce an unstructured, uniform drop.

I appreciate the detailed explanations and experiments provided by the authors, which help in further understanding the proposed method. However, my primary concern regarding the validity of the error source attribution remains not fully addressed.

AuthorsOpenReview ↗

Response to Reviewer 4ur8

We thank the reviewer for the continued engagement. In our MedReason evaluation set the first-error label is defined by construction rather than by human annotation, so it measures source attribution directly. Its corrupted traces are produced by MedPRMBench-style blueprint injection (Wu et al., 2026) applied to clean MedReason reference solutions kept verbatim (Wu et al., 2025), with one or two errors inserted at recorded steps under the MedPRMBench error-code taxonomy (fourteen error types grouped into Simplicity, Soundness, and Sensitivity). Because every injected step is recorded and the base chain is otherwise clean, the first-error label is the smallest injected step index, so hitting the label is hitting the genuine source.

The result on this benchmark is exactly what separates PRO from forward-only GoV. PRO reaches 51.20 F1 against 41.22 for GoV (+9.98), and the gap concentrates in source identification: GoV misses the true source on 72.4% of erroneous traces, since a forward checker flags the first step it rejects, which under error propagation is typically downstream of the injected error. The bidirectional evidence together with the top-k rerank recovers that source.

Reviewer_gorWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper states that existing failure attribution methods either assess the reasoning trace as a whole or locate errors through sequential verification based only on preceding steps, potentially overlooking the true origins of errors. To address this limitation, the authors propose PRO, a progression-aware error localization framework that combines incoming admissibility and outgoing compatibility to identify where errors originate.

Critical strengths
  1. The motivation is plausible and addresses an important problem: some errors may not be immediately apparent but can mislead subsequent reasoning.
Critical weaknesses
  1. The paper is difficult to follow. The introduction presents the motivation in a convoluted manner, and the notation is not consistently explained. For example, Section 3.2 introduces tautau starwithout explicitly defining it or clarifying its relationship to tauexptau_{exp}. The evaluation metrics are also insufficiently explained, making it difficult to interpret the results and assess the trade-off between localization accuracy and false positives.
  2. The framework relies heavily on LLM-as-a-judge evaluations for consistency check , without sufficiently examining whether these judgments introduce additional errors or hallucinations. Both consistency checking and repair assessment depend on potentially unreliable model judgments, yet the paper provides little independent validation of their accuracy or analysis of how their errors propagate through the pipeline.
Questions for authors
  1. The theoretical results establish properties of structural exposure rather than first-error attribution. How do these results support the claim that PRO identifies the true error origin more reliably, beyond detecting an inconsistency earlier?

  2. How reliable are the LLM judgments used for consistency checking, repair verification, and final attribution? Have the authors independently evaluated these judgments or examined how judge errors and hallucinations affect the final localization results?

  3. Could the authors clarify the context used for repair verification? The main text describes incoming/outgoing checks over local transitions, whereas the appendix case study also mentions consistency with the final conclusion. Does verification consider only adjacent blocks or more distant downstream reasoning?

  4. Beyond averaging F1 improvements across backbones in Section 5.4, can the authors provide a more detailed analysis of when PRO succeeds or fails? In particular, does its advantage increase with the distance between the true error source and the point where the error becomes observable?

  5. Although PRO’s relative advantage over GoV increases with reasoning depth, its absolute performance drops substantially. How well does the method scale to substantially longer reasoning traces, both in localization accuracy and computational cost?

5 comments
AuthorsOpenReview ↗

Response to Reviewer gorW(1/3)

We thank the reviewer for the careful read. We fixed the notation, added per-stage judge calibration, and report cost numbers; responses are keyed to each comment.

W1: The notation and evaluation metrics are insufficiently explained.

Response: Thank you for your comment. On notation, we agree and have fixed the inconsistency. The submitted version used two symbols for the same quantity: τexp\tau_\mathrm{exp} was defined in Section 3.1 as the earliest detector-stage exposure, but Section 3.2 (Theorems 1–2) introduced τ⋆\tau_{\star} without defining it, leaving its relationship to τexp\tau_\mathrm{exp} unclear. We have unified the notation: the quantity is now consistently written as τexp\tau_\mathrm{exp} throughout Section 3, defined once in Section 3.1 together with its relationship to the forward-rejection index τseq\tau_\mathrm{seq} and to the final source target f(q,B)f(q,\mathcal{B}), with an explicit note that τexp\tau_\mathrm{exp} is not the final localization. Theorem 2's scope is likewise stated as conditioning only on the binary incoming-decision signals {Ik}\{I_k\}.

On the motivation, we revised the opening contribution statement to state directly why incoming-side evidence alone is insufficient and why outgoing-side evidence exposes errors earlier.

On the metrics, the trade-off between localization accuracy and false positives is captured by our primary metric F1, the harmonic mean of Correct Accuracy and Error Accuracy. F1 penalizes both a missed localization and a false flag on a correct trace, so the trade-off is measured rather than left implicit. The F1-trends figures for ProcessBench and MedReason report F1 directly across backbones and serve as the evidence for this trade-off.


W2: The framework relies heavily on LLM-as-a-judge evaluations.

Response: Thank you for your comment. We address all three parts, covering accuracy, hallucination, and error propagation. PRO does rely on several LLM judges, so we report per-stage calibration measured against the ground-truth first-error label:

Table 1. Per-stage judge calibration.

Judge stageQuantitative evidenceValue
Candidate generationTrue error source appears in top-5 set84.1%
RerankTrue source ranked top-1 / top-3 (cumulative)63.8% / 91.4%
Rerank final selectionSelection matches true source68.12%
Repair vs. rerankSelection agrees with "successfully repaired" verdict89.4%
DetectorMiss rate on erroneous traces13.5%

On accuracy, every number in Table 1 is measured against the annotated source rather than self-reported. The top-5 recall of 84.1%, the rerank top-3 coverage of 91.4%, and the final-selection accuracy of 68.12% all compare a judge's output to the ground-truth first-error label. The one internal figure, the repair-vs-rerank agreement of 89.4%, cross-checks two judges that answer different questions, so no single judge's error goes unmeasured.

On hallucination, each judge makes a constrained discriminative decision rather than free-form generation, asking whether a step follows from the accepted prefix, whether it supports the next step, or whether a rewrite fixes it. Concretely, the prompt fixes a rigid output format: the binary judges must put their final conclusion ('correct' or 'incorrect') inside \boxed{} and nothing else, the reranker outputs exactly one integer index in \boxed{}, and the rewriter emits only the corrected step between <rewrite> and </rewrite>. The parser reads only these fields, so the failure mode is misjudgment, not fabrication. The detector's miss rate of 13.5% on erroneous traces is a measured, bounded error.

On propagation, a judge error does not propagate linearly to the final output, because source-rerank aggregates several independent signals and selects the candidate that best explains the downstream inconsistency. When one stage errs, the remaining evidence usually still recovers the source. Concretely, even when the detector misses, rerank still places the true source in its top-3 for 91.4% of erroneous traces.

AuthorsOpenReview ↗

Response to Reviewer gorW(2/3)

Q1: How do the theoretical results support the claim that PRO identifies the true error origin more reliably?

Response: Thank you for your question. We agree Theorems 1–2 are statements about structural exposure, not a proof of attribution. Their support for the attribution claim is a design rationale rather than a guarantee: Theorem 1 shows exposure can precede rejection, and Theorem 2 shows incoming-only evidence cannot recover it. Together they imply that any forward-only rule provably stops at the exposure rather than the source, so reaching the true origin requires outgoing-side evidence, which is exactly what PRO's bidirectional checking and source reranking add. The theory thus explains why forward baselines underperform attribution and why the extra evidence is necessary; the experiments (the distance-stratified results in Q4) then show this evidence does recover the source across the gap. We have also scoped the contribution bullet to match this, stating that the theory shows why incoming-side evidence alone is insufficient and why outgoing-side evidence exposes errors earlier.


Q2: How reliable are the LLM judgments used for consistency checking, repair verification, and final attribution?

Response: Thank you for your question. The three judges of consistency checking, repair verification, and final attribution are calibrated in Table 1 against the ground-truth first-error label. On how judge errors affect the final localization: it does not rest on any single judge. Source-rerank scores each candidate with several independent signals and picks the one that best explains the downstream inconsistency, so when one judge misjudges a step, the remaining evidence usually still recovers the true source. The detector's miss rate is 13.5%, yet rerank still places the true source in its top-3 for 91.4% of erroneous traces, so an upstream judge error is absorbed rather than amplified by the later stage.


Q3: What context is used for repair verification, and does verification consider only adjacent blocks or more distant downstream reasoning?

Response: Thank you for your question. On repair verification, it uses the same local context as the main checks: the repaired candidate is verified forward against the already-accepted preceding steps and backward against the original immediate next step, never the final answer. The backward repair judge is explicitly instructed to judge whether the rewritten step itself is a valid correction, and not to mark it incorrect merely because the original next step still reflects the old value. On the adjacent-vs-distant question, the appendix case study now says the backward check tests the immediate next step, not the final conclusion. This matches the implementation: the backward verifier only checks whether the current step supports the immediately following step, and never uses the final answer.

AuthorsOpenReview ↗

Response to Reviewer gorW(3/3)

Q4: Does its advantage increase with the distance between the true error source and the point where the error becomes observable?

Response: Thank you for your question. On the 207 erroneous ProcessBench traces, PRO localizes the true source on 141 (68.12%). The 66 failures split into two disjoint causes. First, the source never reaches the candidate set in 33 traces (15.9%): the detector misses the trace entirely in 28 of them (13.5%) and candidate generation drops the source in the remaining 5. Second, the source is among the top-5 candidates but rerank mis-picks in 33 traces (19.0% of the 174 covered traces).

The second failure mode is driven by the source–exposure distance d, which counts the number of steps the error propagates before it becomes observable:

Table 2. Source-localization accuracy by source–exposure distance.

Distance |d| (exposure − source)SamplesPRO after rerank hits source
010585.7%
15877.6%
2944.4%
≥3728.6%

PRO succeeds at small-to-moderate gaps (85.7% at |d| = 0, 77.6% at |d| = 1) and fails increasingly as the gap widens (44.4% at |d| = 2, 28.6% at |d| ≥ 3), because the backward evidence must bridge more propagation steps and downstream noise obscures the origin. The benefit is therefore largest at small gaps and erodes as the gap widens.


Q5: How well does the method scale to substantially longer reasoning traces, both in localization accuracy and computational cost?

Response: Thank you for your question. On accuracy, absolute localization does decline with depth, which is inherent to the task: longer traces propagate an error through more steps, so the first-error step is harder to recover. What scales well is PRO's advantage over GoV, which widens with depth because the source–exposure gap, where backward evidence and attribution help, grows with the number of downstream steps. On cost:

Table 3. Inference cost factor relative to GoV.

BenchmarkAverage factorCorrect samplesErroneous samples
MedReason2.66x2.24x3.08x
NTS N=22.71x1.50x3.92x
NTS N=43.27x1.78x5.26x
ProcessBench3.33x1.97x5.40x
NTS N=63.52x1.95x5.78x

Cost is on-demand and concentrates on the samples that need localization. Correct samples (the majority) run only the forward and backward detector, at 1.5x to 2.2x GoV, while erroneous samples run candidate rewriting and rerank, at 3.1x to 5.8x.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to PCs, SACs, ACs, and Reviewers

Dear PCs, SACs, ACs, and Reviewers,

We sincerely appreciate your diligent efforts and insightful feedback. Your comments have greatly helped us improve the quality and clarity of our manuscript. To assist the AC and help reduce the workload, we provide below a summary of the key points from the reviews and the reviewer-author discussions.

We are pleased to note that the reviewers recognized several strengths of our work, including:

  • The motivation addresses an important problem: some errors may not be immediately apparent, yet can mislead subsequent reasoning (Reviewer gorW).
  • The distinction between detecting a failure and identifying its origin, treating detection as evidence for a separate attribution stage rather than assuming the first rejected block is the first erroneous block (Reviewers ZH7a, 4ur8).
  • The training-free design, which relies on inference-time verification and candidate attribution without requiring additional training of a verifier model, making it applicable across different LLMs (Reviewer 4ur8).
  • The comprehensiveness of the evaluation, covering open-form, medical, and structured reasoning settings and multiple backbones (Reviewers ZH7a, 4ur8).

Following the valuable suggestions from the reviewers, we have provided detailed responses to all comments and made the following major improvements:

  • We unified the notation and scoped the theoretical results to the binary incoming-decision signals.
  • We reported per-stage judge calibration against the ground-truth first-error labels.
  • We analyzed localization accuracy as a function of the source–exposure distance.
  • We reported inference cost relative to GoV across benchmarks.
  • We conducted a "rerank w/o rewrite" ablation to isolate the rewrite evidence.
  • We clarified the repair-verification context and documented the MedReason benchmark construction.

Above, we have faithfully summarized the reviewer comments and our corresponding responses, hoping that this will assist the AC in evaluating our work.

We are deeply grateful to the reviewers, ACs, SACs and PCs for their time, insightful feedback, and dedicated efforts throughout this process. We believe that this review round has greatly improved the quality of our manuscript.

Best regards,

The Authors

Reviewers, please respond to the authors if you have not done so already.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.