acceptodds
Under review as a conference paper at ICLR 2027

Error Localization and Steering of Language Model Quantitative Reasoning

Abstract

A single reasoning error in a large language model (LLM) can propagate through subsequent steps and lead to consequential failures in quantitative tasks. Existing correction methods rely largely on textual feedback, while formal verification can provide precise diagnostics without directly translating them into effective model interventions. We introduce our approach, a neuro-symbolic framework that turns formal diagnosis into targeted inference-time correction. A typed factor-graph intermediate representation grounds variables, relations, and dependencies in the source text, enabling a Satisfiability Modulo Theories (SMT) solver to localize the first conflicting reasoning step and identify the constraints responsible for the error. Our approach then converts this diagnosis into steering signals applied through the model's Key-Value (KV) cache, preserving the valid reasoning prefix while regenerating only the erroneous step with the same frozen model. Across four LLM backbones and four quantitative reasoning datasets, our approach improves both first-error localization and correction accuracy over three state-of-the-art baselines. Crucially, it provides effective guidance even for smaller models that struggle to act on textual critiques, showing that verifier-grounded steering can enhance corrections where language-only feedback falls short.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Large language model reasoning, Neuro-symbolic AI, Inference-Time Correction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2unDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how errors propagate along subsequent steps in quantitative reasoning. The authors propose a neuro-symbolic pipeline: extract the problem and reasoning steps into a typed factor graph, use an SMT solver to check step by step whether each step is entailed by the problem and the accepted prefix, thereby locating the first conflicting step; then derive correction conditions from the conflict, feed those conditions into the same frozen model, and rewrite only the erroneous step while preserving the valid prefix. Conditions are first rendered as text, with corresponding tokens appended to the prefix KV cache; after the new step passes verification, the system removes these diagnostic tokens and rebuilds the cache from the visible text.

Critical strengths

Locating the first conflicting step with checkable constraints and preserving the valid prefix accordingly. The paper splits each step into definitions and assertions, committing a step only when the whole step is entailed by the accepted state; otherwise it stops and returns conflicting factors. This prevents erroneous premises from entering later state and is more targeted than regenerating the entire solution at the first failure point.

Critical weaknesses
  1. KV steering is not distinguished from the same diagnostics given as plain visible input, and removing diagnostics afterward changes the conditions for subsequent generation. Diagnostic results are written as strings and appended to the cache via normal decoding; the appendix states that under exact computation this yields the same logits as computing “prefix + diagnostic text” from scratch. The paper does not run a control with the same correction conditions as a visible prompt, so it cannot show that the cache form itself provides benefit. Repair steps are generated while diagnostics are visible, but later text is generated after diagnostics are deleted. The authors acknowledge this changes the context for subsequent decoding but do not measure its impact on coherence and final recovery rate.

  2. The method is limited to problems solvable by SMT. The proposed pipeline focuses on problems SMT can solve, while the solver can already obtain most target values directly on fixed formulas and SatLM also returns solver answers. The paper argues that LLM rewriting matters for keeping intermediate textual steps valid, but answer recovery rate does not check whether those steps are valid. The paper also does not test the pipeline on tasks SMT cannot fully solve, so existing results do not yet show broader applicability to model reasoning ability.

Questions for authors
  1. Please compare three inputs using exactly the same correction conditions: hidden KV conditioning, visible plain-text prompts, and continuing generation after fixing the current step with diagnostics removed. How do the latter two differ in final answers and coherence of subsequent steps?

  2. On the same formalization, what are final accuracy and intermediate-step validity rates for directly returning the SMT answer, only telling the model the SMT answer, and the current stepwise rewriting?

  3. After removing manually checked representations and externally selected correction records, can extraction, minimal correction-set selection, and rewriting run automatically? Do failures mainly come from extraction, undecided solver cases, or the model not executing the correction conditions?

  4. For reasoning tasks where only part of the constraints can be formalized or SMT cannot decide, which parts of this localization-and-rewrite pipeline still apply?

4 comments

We thank the reviewer for the careful assessment and constructive comments. We address the concerns point by point below.

Response to W1 & Q1

KV-cache execution is the prefix-preserving execution mechanism of our verification-and-repair loop rather than our sole claimed novelty. While hidden KV conditioning and visible plain-text prompts with identical tokens yield the same effect, our core contribution lies in factor-graph/MCS localization (where to steer) and EGS certified feedback (how to steer). Rather than relying on the caching operation to alter single-step logits in isolation, KV-cache execution serves to roll back directly to the localized error without re-prefilling admitted steps, while EGS certified feedback drives a statistically significant +11.30 percentage-point recovery gain (Table 3).

Because hidden KV and visible plain-text conditioning yield same effect, we conduct an experiment to compare whether diagnostic are removed or retained as visible tokens after step repair. Across 209 cases under identical initial states and seeds, both policies achieve virtually identical final-answer accuracy:

  1. Continuing generation with diagnostics removed: 183/209 (87.56%) final answers correct.
  2. Continuing generation with visible diagnostic token retained: 184/209 (88.04%) final answers correct.

Crucially, the two policies differ by only a single case (0.48 percentage points) out of 209, which is well within normal sampling variance and confirms that removing diagnostics after step repair does not degrade downstream accuracy. We remove the diagnostic in order to keep subsequent reasoning steps clean, coherent and free of accumulated diagnostic clutter across multi-step repairs.

Response to W2:

SMT is broadly used across quantitative analysis, including planning and optimization. SMT serves as an established formal foundation for reasoning, as also adopted by SatLM. Rather than relying on a narrow domain or solver-specific heuristics, our factor-graph intermediate representation compiles into standard, solver-agnostic constraints.

An SMT solver can obtain the target values only when given a valid, conflict-free formalization, which is precisely the bottleneck when a model's reasoning contains errors. SatLM fails to generate correct answer when the formulation contains erroneous premises or arithmetic inconsistencies. Feedback-driven baselines (Logic-LM and Reflexion) struggle to isolate and resolve the underlying conflict. This bottleneck directly motivates Algorithm 1: localizing the exact step and intra-step factor where reasoning breaks down and repairing the derivation until both a valid formalization and a verified natural-language reasoning trace are achieved.

Response to Q2

In Table 2, directly returning the SMT answer (SatLM) achieves 60.51% final accuracy on average across all four backbones. Feeding solver feedback back to the model (Logic-LM) achieves 46.24% accuracy. Our stepwise rewriting reaches 84.29% accuracy across all four backbones.

Specifically, SatLM relies on initial formalization correctness and returns only the solver's output without repairing or verifying the model's erroneous derivation. Therefore, whenever the formalization contains an error, the solver cannot recover a valid answer. Meanwhile, coarse solver conflict messages in Logic-LM do not specify which step is wrong, leaving small language models struggling to construct a sound repair from scratch. Our advantage comes directly from leveraging factor-graph and MCS intra-step localization to identify where to steer and Entailment-Guided Steering (EGS) to determine how to steer, which together recover accurate final answers alongside sound, step-by-step verified natural-language derivations.

Response to Q3:

After removing manually checked representations and externally selected correction records, can extraction, minimal correction-set selection, and rewriting run automatically?

None of the 209 evaluated symbolic representations were manually edited by humans, and all three stages, symbolic extraction, MCS selection, and step rewriting, run as a 100% automated pipeline:

  1. Automated symbolic extraction converts natural-language reasoning steps into typed factor-graph updates without human intervention. On the 91 fully unassisted direct-extraction cases, our automatic pipeline achieves 98.90% first-error localization accuracy (90/91: 61/61 on GSM8K, 18/19 on SVAMP, and 11/11 on ASDiv) and achieves 71.4% to 94.5% final-answer accuracy across our four evaluated backbones.

  2. Minimal Correction Set (MCS) selection is 100% deterministic and fully automated across all evaluated cases. Our pipeline deterministically selects the admissible MCS candidate with the fewest deleted factors.

  3. KV-cache rollback, EGS-conditioned regeneration, and online SMT re-verification execute automatically in a closed loop. Once EGS derives the certified entailment conditions from the selected MCS, the generation backbone automatically rewrites the localized error step from the retained KV-cache checkpoint and submits the candidate step to the online SMT verification gate.

In inference time, the pipeline is automated. Only dataset construction requires "checked representations". The "checked representations" refers to the process that screens for genuinely erroneous traces and verifies schema/solver admissibility before evaluation. Across the 209 evaluated cases, 91 were produced entirely through our automatic pipeline without any context supplementation (61 in GSM8K, 19 in SVAMP, and 11 in ASDiv). The remaining 118 cases (26 in GSM8K, 9 in SVAMP, 10 in ASDiv, and 73 in GSM-Symbolic) used automated context supplementation to resolve implicit domain assumptions in natural language word problems.

Do failures mainly come from extraction, undecided solver cases, or the model not executing the correction conditions?

In most cases our steering succeeds, and remaining failures stem primarily from running out of retry budget and formating issues. Among the 131 observed failure cases, 69/131 (52.7%) occur when the generation backbone proposes a replacement step that still violates the certified symbolic constraints and exhausts the retry budget. For these failures, our online verification gate can still correctly detect and reject to prevent unverified reasoning from propagating. There are 37/131 (28.2%) failures stem from formatting issues. Across all evaluated runs, none of solver checks returned undecided.

Response to Q4

Our framework can still apply over the subset of the formalizable constraints. Given successful extraction of the relevant variables, a factor graph, by definition, encodes the relationships among variables as factors, which directly form the symbolic constraints. Consequently, any extracted relationship is guaranteed to obtain a corresponding formalization. Because the factor graph explicitly links each constraint to its source text while preserving the admitted prefix, our framework operates directly on the formalized constraints without affecting the unformalized reasoning context.

When an SMT query cannot be decided, the pipeline still preserves formal soundness. The pipeline will stop rather than injecting unverified MCS/EGS steering signals. Empirically, across all evaluated runs in our benchmark (including all 112 cross-solver checks across Z3, cvc5, and dReal in Table 4), zero solver queries returned undecided. Extending the guarantees to partially formalizable tasks is future work.

We thank the reviewer for the constructive discussion. We have thoroughly considered your suggestions and made substantial revisions to the manuscript in response. All corresponding revisions are highlighted in blue in the revised manuscript:

  • W1 & Q1: Section 4.2 explains KV-based execution for prefix reuse, localized regeneration, and temporary diagnostic conditioning. Appendix F adds the 209-case comparison showing virtually identical accuracy when removing versus retaining correction conditions after step repair, and Appendix F (Table 3) reports the feedback gain (+11.30 percentage points) with its 95% confidence interval ([+3.48, +19.13]).
  • W2: Appendix F (Table 4) confirms 100% cross-solver verdict agreement across Z3, cvc5, and dReal with zero undecided queries, and Section 7 states our evaluation scope and SMT-based assumptions.
  • Q2: Section 5 clarifies the comparison against SatLM and Logic-LM on shared error traces.
  • Q3: Section 1, Section 4.1, Algorithm 1, and Appendix B.1 specify deterministic, automatic selection of the admissible MCS candidate with the fewest deleted factors. Appendix E.1 clarifies the 118 reference-annotated cases in dataset construction, Appendix E.3 adds the 91 unassisted cases and distinguishes closed-loop recovery (Table 2) from the single-pass ablation (Table 3), and Appendix E.4 adds a failure analysis breaking down retry-budget exhaustion, formatting issues, and solver compatibility.
  • Q4: Section 3.2 clarifies how the verification gate preserves soundness when only part of a step is formalizable or when a solver query is undecided.

The discussion period will be closing soon on the 18th. We truly appreciate your perspective and invite you to review our rebuttal along with the updated manuscript. We hope that our modifications address your concerns. Please feel free to reach out if you need any further clarification or wish to discuss the changes in more detail.

Reviewer: please respond to the authors, as they have spent a lot of effort replying to you. If you have any remaining concerns, you should raise them now.

Reviewer_NmEpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a neuro-symbolic framework for correcting LLM quantitative reasoning errors. It uses typed factor graphs and SMT verification to locate the first invalid reasoning step, derives correction conditions through MUS/MCS analysis, and regenerates the erroneous step using KV-cache-based conditioning while preserving the valid prefix.

Experiments on 209 incorrect solutions across four datasets report 94.55% first-error localization accuracy and 73.81–96.11% answer recovery across four LLM backbones.

Critical strengths
  1. Meaningful problem. Localizing and correcting reasoning errors without discarding valid preceding steps is a useful research direction.

  2. Technically grounded formulation. The symbolic verification framework is relatively well specified, distinguishing inconsistent from unsupported assertions and providing conditional correctness guarantees.

  3. Promising results. The reported improvements are substantial. Table 3 also shows that correction feedback improves recovery from 83.48% to 94.78% under a controlled prefix-preserving setting.

Critical weaknesses
  1. Questionable novelty of KV-cache steering. Section 4.2 describes converting verified constraints into tokens and appending them to the retained KV cache. This appears equivalent to ordinary prefix-preserving generation with additional textual conditioning, rather than a novel steering mechanism. No comparison isolates this claimed contribution.

  2. Reliance on oracle-assisted representations. Symbolic representations are extracted using GPT-5.6 and include reviewed, oracle-assisted inputs (Appendix E.3). Since faithful extraction is arguably the most challenging part of the framework, these conditions substantially weaken the end-to-end claims.

  3. Insufficient ablation studies. The experiments do not isolate the contributions of factor graphs, MUS/MCS enumeration, and KV-cache execution. Table 3 establishes the benefit of feedback, but not the necessity of the proposed mechanisms. Table 4 merely compares solver backends.

  4. Missing closely related baselines. The paper discusses VeriCoT and VERGE but excludes them from evaluation. Reflexion, SatLM, and Logic-LM are less directly aligned with the proposed verification-and-repair task. Stronger comparisons are necessary to establish novelty.

  5. Mismatch between the algorithm and evaluation. Appendix E.3 states that recovery results combine closed-loop and one-shot runs, scoring final answers regardless of verification-gate acceptance. Consequently, Table 2 does not establish the end-to-end effectiveness of Algorithm 1.

  6. Limited evaluation and incomplete automation. All 209 erroneous traces originate from Qwen2.5-7B, with only 21–28 examples in two datasets. Furthermore, Appendix B.1 indicates that repair records require external selection. The proposed fully automatic correction workflow is therefore not adequately demonstrated.

    • Repetitive presentation: The manuscript repeatedly emphasizes bridging formal diagnosis and actionable steering without clearly distinguishing its mechanisms from existing approaches.
Questions for authors
  1. How does the proposed KV-cache steering differ from conventional prompting with identical solver-derived constraints and a preserved prefix?

  2. What is the end-to-end performance using fully automatic, unreviewed symbolic extraction?

  3. Why are VeriCoT and VERGE excluded from the experimental comparisons?

  4. Can the authors provide ablations isolating factor graphs, MUS/MCS enumeration, and KV-cache execution against simpler solver-assisted prompting?

  5. What is the actual closed-loop success rate of Algorithm 1, including automatic repair selection and verification acceptance?

9 comments

We thank the reviewer for the careful assessment and constructive comments. We address the concerns point by point below.

Response to W1 & Q1

KV-cache steering is the execution engine of our integrated diagnostic loop—its novelty lies in enabling intra-step factor-level repair (preserving valid premises in 68.3% of cases) and clean context continuation without history pollution, rather than low-level token caching alone.

How does proposed KV-cache steering differ from conventional prompting?

Under exact arithmetic, identical token sequences naturally produce identical logits. Our methodological novelty does not rest on the caching operation in isolation, but on the integrated verification-and-repair framework where factor-graph diagnosis determines where to steer, and Entailment-Guided Steering (EGS) derives how to steer. This establishes three decisive structural differences from conventional prompting:

  1. Intra-step factor diagnosis preserves valid intermediate premises within the rejected step, unlike conventional prompting that treats entire reasoning steps as indivisible black boxes (in 142 of 209 evaluated cases, or 67.9%, our Minimal Correction Set isolates partial factor conflicts, retaining verified premises within the localized error step and deriving certified entailment conditions via EGS).

  2. Temporary diagnostic conditioning ensures clean context continuation without biasing subsequent reasoning, pruning diagnostic tokens once the replacement step passes verification to restore a pristine generation context, whereas conventional prompt appending permanently pollutes the context history with verbal critiques.

  3. Prefix preservation and rollback eliminate redundant prefill computation over admitted steps, allowing the generator to branch directly from the localized error step across iterative multi-step repairs.

Table 3 (Appendix F) directly shows our formally derived EGS steering demonstrates a significant +11.30 percentage-point gain in answer recovery (from 83.48% to 94.78%). This controlled ablation confirms the necessity of EGS feedback.

Response to W2 & Q2

Reliance on oracle-assisted representations... reviewed, oracle-assisted inputs substantially weaken the end-to-end claims.

Exactly none of the 209 evaluated cases were manually edited by humans after extraction. The inference of manual rewriting is a misunderstanding. During the inference time, the full pipeline is automated on the 209 Cases. We leverage an external model (GPT-5.6) to perform automated extraction. Across all 209 evaluated cases with repair records, the minimal correction set (MCS) is chosen deterministically by taking the admissible candidate with the fewest deleted factors. In the revised manuscript, we will clarify in Section 7 and Appendix E.3.

In inference time, the pipeline is automated. Only dataset construction requires "oracle-assist" and "reviewed". "oracle-assist" refers to supplementation of implicit domain assumptions in word problems. "reviewed" refers to the process that screens for genuinely erroneous traces and verifies schema/solver admissibility before evaluation.

We decouple symbolic extraction from diagnosis to evaluate conflict localization and steering independently of parsing errors. Because smaller models (e.g., 7B) could misparse problem statements when translating text into formal constraints, having each generation backbone extract its own formulas would conflate parsing failures with diagnostic failures. Using a parser (GPT-5.6) cleanly isolates our factor-graph localization (MUS/MCS) and entailment-guided steering (EGS) across all four evaluated backbones.

What is the end-to-end performance using fully automatic, unreviewed symbolic extraction?

Across the 209 evaluated cases, 91 were produced completely through our direct, unassisted automated pipeline without any context supplementation (61 in GSM8K, 19 in SVAMP, and 11 in ASDiv). The remaining 118 cases (26 in GSM8K, 9 in SVAMP, 10 in ASDiv, and 73 in GSM-Symbolic) used automated context supplementation to resolve implicit domain assumptions in natural language word problems.

On this 91-case fully unassisted subset, first-error localization achieves 98.90% accuracy (90/91 correct: 61/61 on GSM8K, 18/19 on SVAMP, and 11/11 on ASDiv), surpassing the 94.55% overall benchmark average. Furthermore, answer recovery under steering on these identical unassisted cases reaches 71.4% for Qwen2.5-7B, 79.1% for Llama3.1-8B, 87.9% for GPT-OSS-20B, and 94.5% for Qwen3.8-27B. Downstream repair selection is likewise 100% automatic: the Minimal Correction Set (MCS) is chosen deterministically by taking the admissible candidate with the fewest deleted factors, with zero human selection. These results directly confirm that high localization precision and steering efficacy are intrinsic capabilities of our factor-graph diagnostic algorithm.

Response to W3 & Q4

Factor graphs, MUS/MCS enumeration, EGS, and KV-cache execution form a strictly coupled causal pipeline where each stage resolves a specific failure bottleneck that simpler solver-assisted prompting cannot address.

The experiments do not isolate the contributions of factor graphs, MUS/MCS enumeration, and KV-cache execution against simpler solver-assisted prompting.

Factor graphs, MUS/MCS enumeration, and EGS each resolve a distinct failure bottleneck that conventional solver-assisted prompting cannot address, forming a tightly coupled diagnostic-and-repair pipeline:

  1. Factor graphs explicitly bind natural language quantities and assertions to formal symbolic factors to enable intra-step error localization, overcoming standard solver prompting (e.g., SatLM or raw SMT encoding) that treats an entire reasoning step as an indivisible black box.

  2. Minimal Correction Set (MCS) search isolates minimal conflicting factors while preserving valid intermediate premises within the rejected step, unlike naive solver interfaces that only emit coarse UNSAT signals forcing whole-step or whole-trace discarding (in 142 of 208 evaluated cases, or 68.3%, our error is a partial factor deletion where correct premises are retained).

  3. Entailment-Guided Steering (EGS) derives formally certified survivor entailment conditions around retained factors, bridging the gap where raw SMT solvers merely certify satisfiability or output numerical assignments without providing actionable linguistic repair boundaries.

Controlled empirical comparisons directly confirm that simpler solver-assisted prompting alternatives fail to achieve consistent derivation repair:

  1. Coarse verbal critique (Reflexion) achieves only 9.20% recovery on GSM8K with Qwen2.5-7B, demonstrating that ungrounded natural language reflection fails to resolve arithmetic breakdowns without factor-level diagnosis.

  2. Supplying solver-certified final answers directly (SATLM-SOL) still fails to produce a valid derivation in 82.1% of cases (55/67), proving that knowing the target answer does not teach the model how to correct its intermediate reasoning steps.

  3. Isolating EGS feedback under identical verified prefixes and seeds (Table 3) yields a statistically significant net gain of +11.30 percentage points (raising recovery from 83.48% to 94.78%, paired 95% CI: [+3.48, +19.13] pp).

Together, the synergy of factor-graph localization, MCS intra-step isolation, and EGS certified steering, enables our framework to achieve an 86.21% recovery rate where simpler alternatives collapse.

Response to W4 & Q3

Neither VeriCoT nor VERGE has an open-sourced codebase available, while our evaluated baselines (SatLM and Logic-LM) already comprehensively cover their underlying verification-and-repair mechanisms.

The paper discusses VeriCoT and VERGE but excludes them from evaluation... Stronger comparisons are necessary to establish novelty.

Neither VeriCoT nor VERGE has an open-source implementation or public artifacts available. Rather than relying on speculative third-party re-implementations that could introduce confounding evaluation discrepancies, we benchmarked against established, reproducible open-source baselines that directly represent the core paradigms underlying both frameworks:

SatLM directly embodies the formalization-and-solving paradigm of VeriCoT, translating natural language reasoning into a solver-facing formal representation and querying an external solver for verification.

Logic-LM directly captures the iterative constraint-refinement paradigm of VERGE and ARc, formalizing assertions and utilizing solver conflict feedback to guide iterative prompt correction.

Reflexion serves as the recognized gold standard for ungrounded natural language reflection, benchmarking multi-turn self-correction without external formal assistance.

Crucially, our framework departs fundamentally from all these frameworks through a localized, intra-step verification-and-repair loop that whole-trace methods cannot achieve. Prior verifiable-reasoning methods either discard the entire generation and restart from scratch under natural language critiques or delegate the solution entirely to an external solver. In sharp contrast, our framework preserves the verified prefix via KV-cache rollback, isolates intra-step factor conflicts via Minimal Correction Sets (retaining valid premises within the rejected step in 68.3% of cases), and steers the replacement step with formally certified survivor entailment conditions via EGS.

In the revised manuscript, we will explicitly detail this mechanistic mapping between our evaluated baselines and these frameworks in Section 2 (Related Work) and Appendix E.

Response to W5 & Q5

Recovery results combine closed-loop and one-shot runs, scoring final answers regardless of verification-gate acceptance.

Closed-loop runs and single-pass runs are never pooled into a composite metric. They are reported separately in Table 2 (Section 5) and Table 3 (Appendix F) for distinct analytical scopes. Table 2 evaluates end-to-end closed-loop recovery across different generation backbones under iterative step-level verification and repair, whereas Table 3 is a dedicated, controlled ablation specifically designed to isolate the single-turn impact of EGS feedback under strictly identical prefixes and random seeds. These two modes are reported in their respective tables rather than pooled. In the revision, we will make section headings and table captions even more explicit to eliminate any ambiguity.

Consequently, Table 2 does not establish the end-to-end effectiveness of Algorithm 1. What is the actual closed-loop success rate?

Under 100% deterministic automatic repair selection, Algorithm 1's actual closed-loop recovery rate is directly reported in Table 2. Our approach achieves 73.81% to 96.11% across all four evaluated backbones. Specifically, repair selection deterministically chooses the admissible Minimal Correction Set with the fewest deleted factors with zero human intervention, and reporting final-answer recovery in Table 2 ensures an objective, apple-to-apple comparison against external baselines (Reflexion, SatLM, and Logic-LM) that lack an online step-level formal verification gate.

Response to W6

Repair selection is 100% deterministic and fully automated with zero external intervention, sourcing initial error traces from Qwen2.5-7B establishes a controlled benchmark for evaluating steering across four distinct backbones, and the 21 ASDiv and 28 SVAMP cases reflect the complete natural failure pool (currently being scaled via higher-temperature sampling).

All 209 erroneous traces originate from Qwen2.5-7B, with only 21–28 examples in two datasets.

We source all 209 erroneous trajectories from Qwen2.5-7B to establish the exact same starting point for all evaluated methods and repair backbones. Rather than testing different models on different mistakes, using a shared pool of genuine errors enables a controlled, apple-to-apple comparison across all four repair backbones (Qwen2.5-7B, Llama3.1-8B, GPT-OSS-20B, and Qwen3.8-27B), all of which show consistent recovery gains.

The 21 ASDiv and 28 SVAMP cases represent Qwen2.5-7B's complete natural error rather than an arbitrary downsampling. Because existing error-label benchmarks lack step-level symbolic representations and model-specific error traces, our benchmark must be built directly from genuine model failures, and Qwen2.5-7B already solves most ASDiv/SVAMP problems correctly in one pass. To further scale sample diversity, we are actively expanding both subsets via higher-temperature sampling. We have completed 3,305 generations across both datasets (SVAMP and ASDiv) and confirmed a more 836 erroneous cases (275 on SVAMP and 561 on ASDiv). We will update the manuscript with experiment results on the expanded dataset.

Appendix B.1 indicates that repair records require external selection. The proposed fully automatic correction workflow is therefore not adequately demonstrated.

Repair selection is 100% deterministic and fully automated across all evaluated cases, requiring zero external or human selection. While Appendix B.1 described the loop receiving a repair record without explicitly stating the selection rule, our actual implementation deterministically chooses the admissible Minimal Correction Set (MCS) candidate with the fewest deleted factors. We will make this rule explicit in Algorithm 1 and Appendix B.1.

Response to W7

We thank the reviewer for this observation on presentation clarity. In the revision, we will streamline repetitive motivation statements and present our pipeline once upfront to contrast directly with prior whole-trace paradigms:

  1. Fine-grained factor-graph binding enables intra-step error localization.

  2. Minimal Correction Set (MCS) enumeration isolates minimal conflicting factors while preserving valid intermediate premises within the rejected step, avoiding coarse UNSAT discards (partial deletion in 68.3% of cases).

  3. Entailment-Guided Steering (EGS) derives formally certified entailment conditions, providing rigorous mathematical repair boundaries.

Reviewer_WtZ5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a neuro-symbolic pipeline to repair arithmetic word-problem solutions. It parses the problem and the model's steps into a typed factor-graph IR, uses an SMT solver to localize the first invalid step, then uses KV-cache steering to regenerate only that step with the same frozen model. The main evaluation covers 209 incorrect answers produced by Qwen2.5-7B-Instruct, using four backbones and four datasets. Reported macro-average localization accuracy is 94.55%. Its macro-average answer recovery ranges from 73.81% to 96.11% across the four backbones.

Critical strengths

(1) I found the formal setup genuinely useful. Definition 1 gives a crisp inductive notion of the first invalid step, and the paper separates structural reachability from semantic validity carefully enough that the correctness claims are actually checkable.

(2) Table 3 is the one experiment I trust most. Both conditions resume from the same pre-error prefix with the same seed, so the controlled difference is the presence of the correction conditions. The jump from 83.48% to 94.78% on GSM8K plus SVAMP with Qwen3.8-27B is concrete evidence that the steering signal does something beyond restarting from a valid prefix.

(3) The appendix is also transparent about implementation gaps, which I appreciate. Appendix A.4 admits the checker can accept non-conservative definitions like y:=1/x, and Appendix C documents a historical KV-cache mismatch that remains unexplained. That is more honest than what the main text does.

Critical weaknesses

(1) My main concern is that the headline numbers are not end-to-end numbers. Appendix E.3 says symbolic representations are extracted by GPT-5.6 and that the evaluated representations include reviewed oracle-assisted inputs. This conditioning is acknowledged only in a single sentence in the Limitations; the abstract and the experimental sections present the numbers as if they were automatic. As a result, Tables 1 and 2 measure localization and steering given externally produced, human-reviewed symbolic representations rather than fully automatic extraction. This needs to be disclosed wherever those numbers first appear.

(2) The evaluation is also too small. ASDiv has 21 cases and SVAMP has 28; a single case moves ASDiv accuracy by about 4.8 points and SVAMP by about 3.6. There are no confidence intervals or seed variance, so the macro-averages are hard to interpret. Table 4 is uninformative for the same reason: 28 SVAMP cases per backbone cannot support a strong claim about solver portability.

(3) I am not convinced the baselines were given a fair chance. Reflexion at Qwen2.5-7B recovers only 9.20% on GSM8K in Table 2, which is suspiciously low even accounting for the different task from Table 1. The retry budget and feedback template are not reported, so it is impossible to tell whether Reflexion was allowed enough attempts. The paper also does not compare against the verifiable-reasoning frameworks it cites in related work, such as ARc, VeriCoT, or VERGE, even though those are more directly relevant than generic self-correction baselines.

(4) Several smaller issues suggest an unfinished submission. MOS is used throughout Appendix E and Table 3 but never defined in the main text. What the paper calls Qwen3.8-27B is described in Appendix E.3 as a deployment label whose configuration declares qwen3_5, which I cannot map to a public checkpoint. There is also a leftover reviewer-guidelines template paragraph in the introduction.

Questions for authors

(1) What fraction of the 209 cases were manually edited after GPT-5.6 extraction, and what kinds of edits were made? I would like a per-dataset breakdown. This determines whether the 94.55% localization score is a property of the system or of the system plus oracle input.

(2) What exactly is Qwen3.8-27B? If the deployment label declares qwen3_5, the checkpoint identity and serving configuration should be reported precisely. Also, why are saved closed-loop and one-shot runs combined in the recovery numbers, and are the underlying trajectories comparable?

(3) What retry budget and feedback template were used for Reflexion? Would the 9.20% GSM8K recovery accuracy improve substantially if more attempts were allowed?

If the authors can provide in the rebuttal the percentage of manually edited cases and a fully automatic end-to-end evaluation subset, I would be glad to raise my rating.

6 comments

We thank the reviewer for the careful assessment and constructive comments. We address the concerns point by point below.

Response to W1 & Q1:

Human review is not a stage of our method. None of the 209 evaluated symbolic representations were manually edited by humans, and our framework operates as a fully automated pipeline achieving 98.90% localization accuracy on the 91 unassisted direct-extraction cases.

What fraction of the 209 cases were manually edited after GPT-5.6 extraction, and what kinds of edits were made?

Exactly none of the 209 evaluated cases were manually edited by humans after extraction. The inference of manual rewriting is a misunderstanding. During the inference time, the full pipeline is automated on the 209 Cases. We leverage an external model (GPT-5.6) to perform automated extraction. Across all 209 evaluated cases with repair records, the minimal correction set (MCS) is chosen deterministically by taking the admissible candidate with the fewest deleted factors. In the revised manuscript, we will clarify in Section 7 and Appendix E.3.

If the authors can provide in the rebuttal the percentage of manually edited cases and a fully automatic end-to-end evaluation subset, I would be glad to raise my rating.

In inference time, the pipeline is automated. Only dataset construction requires "oracle-assist" and "reviewed". "oracle-assist" refers to supplementation of implicit domain assumptions in word problems. "reviewed" refers to the process that screens for genuinely erroneous traces and verifies schema/solver admissibility before evaluation. Across the 209 evaluated cases, 91 were produced entirely through our automatic pipeline without any context supplementation (61 in GSM8K, 19 in SVAMP, and 11 in ASDiv). The remaining 118 cases (26 in GSM8K, 9 in SVAMP, 10 in ASDiv, and 73 in GSM-Symbolic) used automated context supplementation to resolve implicit domain assumptions in natural language word problems.

On this 91-case fully unassisted subset, first-error localization achieves 98.90% accuracy (90/91 correct: 61/61 on GSM8K, 18/19 on SVAMP, and 11/11 on ASDiv), surpassing the 94.55% overall benchmark average. Furthermore, on these unassisted cases, answer recovery under steering reaches 71.4% for Qwen2.5-7B, 79.1% for Llama3.1-8B, 87.9% for GPT-OSS-20B, and 94.5% for Qwen3.8-27B. These results directly confirm that high localization precision and repair efficacy are intrinsic properties of our factor-graph diagnostic algorithm.

Response to W2:

The evaluation is also too small. ASDiv has 21 cases and SVAMP has 28... There are no confidence intervals or seed variance

The 21 ASDiv and 28 SVAMP cases represent the model's complete natural error pool rather than an arbitrary subsampling. Because existing error-label benchmarks provide neither step-level symbolic representations nor the erroneous trajectories of evaluated models, test cases must be constructed directly from genuine model failures. Since Qwen2.5-7B already solves the vast majority of these elementary word problems correctly in a standard pass, these cases represent all natural errors produced by the model.

Our claims do not rely on ASDiv or SVAMP in isolation, but are validated across the full 209-case benchmark, achieving a first-error localization accuracy of 95.2% (199/209) with a 95% confidence interval of [91.4%, 97.4%] (with individual intervals of [71.1%, 97.3%] on ASDiv and [82.3%, 99.4%] on SVAMP). To further scale sample diversity, we are actively expanding both subsets via higher-temperature sampling. We have completed 3,305 generations across both datasets (SVAMP and ASDiv) and confirmed a more 836 erroneous cases (275 on SVAMP and 561 on ASDiv). We will update the manuscript with experiment results on the expanded dataset.

Table 4 is uninformative for the same reason: 28 SVAMP cases per backbone cannot support a strong claim about solver portability.

Table 4 is designed to verify solver backend compatibility rather than dataset-level accuracy variance. Across all 112 symbolic formulations generated on SVAMP (28 cases ×\times 4 backbones), Z3, cvc5, and dReal achieve 100% verdict agreement (107 satisfiable with certified target values, 5 unsatisfiable, and 0 timeouts). This cross-backend consistency demonstrates that our approach is compatible across SMT solvers.

Response to W3 & Q3:

Reflexion's 9.20% recovery stems from the intrinsic self-correction bottleneck of 7B models under coarse verbal critique rather than budget limits, while our evaluated baselines already mechanistically cover ARc, VeriCoT, and VERGE.

Reflexion at Qwen2.5-7B recovers only 9.20% on GSM8K in Table 2,... What retry budget and feedback template were used?

In our implementation, Reflexion follows sequential execution: Reflection (max 1024 tokens) →\rightarrow Thought (max 2048 tokens) →\rightarrow Finish (max 256 tokens) with up to 2 format retries per stage as retry budget. The feedback template follows the description in the reference paper. Specifically, it prompts the model to diagnose its prior error (Reflection), re-derive a step-by-step solution conditioned on that diagnosis (Thought), and output the final answer (Finish).

Allowing more attempts does not close the gap because 7B models suffer from an intrinsic self-correction bottleneck under ungrounded verbal feedback. As established in recent literature (e.g., Huang et al., 2023), small models lack the internal verifier needed to identify where an arithmetic breakdown occurred from coarse outcome critiques, causing sequential retries to repeat invalid premises or hallucinate new inconsistencies.

In contrast, our framework raises answer recovery on the exact same failure traces from 9.20% to 86.21% (75/87) through two decisive capabilities, (i) Minimal Correction Set (MCS) search isolates errors at the intra-step factor level; (ii) Entailment-Guided Steering (EGS) derives formally certified entailment conditions via SMT.

The paper also does not compare against... ARc, VeriCoT, or VERGE

Neither ARc, VeriCoT, nor VERGE has an open-sourced codebase available, whereas our evaluated baselines already comprehensively cover their underlying verification-and-repair mechanisms. Hence, we benchmarked against open-source baselines:

  • SatLM directly represents the formalization-and-solving paradigm embodied in VeriCoT, mapping natural language reasoning into a solver-facing formal representation for external verification.
  • Logic-LM directly captures the iterative constraint-refinement paradigm embodied in VERGE and ARc, formalizing assertions and using solver conflict checks to iteratively refine the prompt.

Crucially, our framework departs from these whole-trace approaches through a fundamentally distinct, localized repair mechanism. Rather than restarting whole-trace generation under natural language critiques or delegating the solution entirely to an external solver, our approach preserves the verified prefix via KV-cache rollback, isolates intra-step factor conflicts, and steers the replacement step with certified entailment conditions.

Response to W4 & Q2

What exactly is Qwen3.8-27B?

The model referenced as Qwen3.8-27B is the publicly accessible open-weight checkpoint released by Qwen under the qwen3_5 architecture family (available on Hugging Face). In the revised version, we will replace this deployment label in Appendix E.

why are saved closed-loop and one-shot runs combined in the recovery numbers, and are the underlying trajectories comparable?

Closed-loop runs and single-pass runs are reported in separate tables serving distinct analytical purposes. Specifically, Table 2 in Section 5 evaluates end-to-end closed-loop recovery across different generation backbones where models undergo multi-step iterative verification and repair. Table 3 in Appendix F is a dedicated, controlled ablation specifically designed to isolate the impact of EGS feedback on single-pass recovery. In the revised manuscript, we will adjust section headings and table captions to make this distinction more clear.

"MOS is used throughout Appendix E and Table 3 but never defined... leftover reviewer-guidelines template paragraph in the introduction."

"MOS" was an internal development name that inadvertently persisted in Appendix E and Table 3. We will replace "MOS" with unified terminology ("our framework" / "our approach") across all tables and appendices. Furthermore, the reviewer-guidelines text in the Introduction might be an automated placeholder added during review phase and was completely absent from our authored LaTeX source files.

We thank the reviewer for the constructive feedback. We have thoroughly considered your suggestions and made substantial revisions to the manuscript in response. All corresponding revisions are highlighted in blue in the revised manuscript:

  • W1 & Q1: Section 1, Section 4.1, Algorithm 1, and Appendix B.1 specify the fully automatic pipeline and the deterministic MCS selection rule to pick the admissible candidate with the fewest deleted factors. Appendix E.1 clarifies the 118 reference-annotated cases in dataset construction. Appendix E.3 reports localization (98.90%) and recovery results on the 91 unassisted cases.
  • W2: Section 5 adds the 95% confidence interval ([91.4%, 97.4%]) for overall first-error localization accuracy across all 209 cases. Appendix F clarifies that Table 4 evaluates cross-solver verdict agreement across Z3, cvc5, and dReal on 112 fixed symbolic formulations.
  • W3 & Q3: Appendix E.2 specifies Reflexion’s stage-wise token limits, retry budget, and feedback template. Section 5 and Appendix G explain how SatLM and Logic-LM cover the core mechanisms of ARc, VeriCoT, and VERGE.
  • W4 & Q2: Appendix E.3 documents the Qwen3.8-27B open-weight checkpoint and explicitly separates closed-loop recovery (Table 2) from the single-pass ablation (Table 3).

The discussion period will be closing soon on the 18th. We truly appreciate your perspective and invite you to review our rebuttal along with the updated manuscript. We hope that our modifications address your concerns. Please feel free to reach out if you need any further clarification or wish to discuss the changes in more detail.

Comments on the paper

2 comments

Reviewers, please respond to the authors if you have not done so already. Let them know if there are any remaining concerns.

AuthorsOpenReview ↗

General Response: Summary of Contributions and Revisions

Dear Area Chair and Reviewers,

We sincerely thank Reviewers WtZ5, NmEp, and 2unD for their thoughtful assessments and constructive feedback. All three reviewers recognize one of our core contributions: a neuro-symbolic intermediate representation that models evolving LLM reasoning states as a typed factor graph. Combined with formal SMT entailment checking, this representation localizes the root cause of reasoning errors and prevents cascading error propagation.

  • Reviewer WtZ5 acknowledged that our formal setup is genuinely useful, separating structural reachability from semantic validity and making correctness claims checkable.
  • Reviewer NmEp acknowledged that our symbolic verification framework is well specified, distinguishes inconsistent from unsupported assertions, and provides conditional correctness guarantees.
  • Reviewer 2unD noted our design of splitting each step into definitions and assertions, committing only fully entailed steps, and returning conflicting factors to prevent invalid premises from propagating.

During the rebuttal, we resolved every concern through new experiments and clarifications, with all manuscript revisions highlighted in blue:

  • Inference-Time Automation and Unassisted Evaluation (WtZ5, NmEp, 2unD): We clarified that zero evaluated symbolic representations were manually edited by humans. Repair selection via Minimal Correction Sets (MCS) is also 100% deterministic and automatic, selecting the admissible candidate with the fewest deleted factors. On the 136-case fully unassisted subset, our pipeline reaches 98.53% (134/136) localization accuracy and 72.79%–95.59% recovery across all four backbones, confirming that high precision is intrinsic to our approach.
  • Dataset Size (WtZ5, NmEp): We expanded SVAMP (28 → 50) and ASDiv (21 → 50), increasing the benchmark from 209 to 260 cases. We re-evaluated all methods across all four backbones, and added 95% confidence intervals. All conclusions hold with the expanded experiment.
  • Baseline Implementations and Coverage (WtZ5, NmEp): We documented Reflexion's exact retry budget and feedback template, and clarified how our open-source baselines (SatLM, Logic-LM, Reflexion) cover the formalization-and-solving and iterative-refinement paradigms of closed-source methods (VeriCoT, VERGE, ARc).
  • Evaluation Protocols (WtZ5, NmEp): We clarified that closed-loop runs and single-pass runs are reported in separate tables serving distinct analytical purposes rather than pooled. Table 2 reports closed-loop final-answer recovery across all four datasets and four backbones. Table 3 reports a controlled single-pass ablation that isolates the effect of solver-certified feedback by holding the verified prefix and random seed fixed.
  • Coupled Component Roles and Empirical Impact (NmEp): We clarified that our framework operates as a coupled system built on the neuro-symbolic intermediate representation. MCS isolates intra-step factor conflicts and preserves valid premises in 69.26% of rejected steps. Entailment-Guided Steering (EGS) then derives certified entailment conditions to guide localized regeneration via KV-cache rollback. Together, this pipeline achieves 75.30%, 84.96%, 88.93%, and 96.30% four-dataset macro-average closed-loop recovery across the four backbones, outperforming the strongest baseline by +38.41% on Qwen2.5-7B and +30.27% on Llama3.1-8B (Table 2). Under identical verified prefixes (Table 3), EGS feedback alone yields a statistically significant +11.30 percentage-point recovery gain.
  • Impact of Removing Diagnostics on Downstream Generation (2unD): In a controlled ablation over 209 cases, removing diagnostic tokens after step repair achieves 87.56% (183/209) accuracy, matching retaining them (88.04%, 184/209). The two settings differ by a single case, while removal keeps downstream context clean.

We hope these clarifications and expanded results resolve all remaining concerns and highlight our contribution to neuro-symbolic representation learning and reliable LLM reasoning. We appreciate the Area Chair and Reviewers' consideration of these updates in the final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.