acceptodds
Under review as a conference paper at ICLR 2027

Consistent but Wrong: Analyzing What LLM Reasoning Lacks via State Decoding

Abstract

Language models are increasingly used for multi-step reasoning. They decide at the token level, and every later step builds on what they have written. Sampling-and-selection (best-of-N, majority voting) chooses among chains already written. A *consistent error* occurs when all sampled chains that reach a decisive step write the same wrong successor state, so the pool holds no correct candidate at that step. We use *state decoding* to analyze what the model's reasoning lacks at such a step. Its *support* supplies the candidate states at each step, the *scoring* ranks them, and the *search* assembles the decisions. An enumerated support contains the correct successor whatever probability the model assigns to it, and with an exact search every remaining error is a scoring error. With Qwen3-4B, we run SAVI (State-Aware Viterbi Inference), which enumerates the candidates and searches exactly on *MuSR* (belief tracking) and *Belief-R* (belief revision). In both tasks, most failures that more samples do not repair are consistent errors. Once the candidates contain the correct state, the model's likelihood picks a belief table on MuSR by the sentence that asserts the move, not by who saw it. On Belief-R it picks a reading by its wording. When the prompt states the correct reading, the model answers correctly, so it fails at choosing the reading. For Qwen3-4B, only signals outside the likelihood pick the correct candidate above chance. Of 19 MuSR items on which voting stays wrong, a verifier built from the annotated facts fixes 15 when it scores enumerated states and 3 when it scores the sampled pool. On *Countdown* the chains disagree, and frequency does not track target reachability: of 45 items on which every sampled chain fails, state decoding solves 9 with frequency scoring and 44 with an exact verifier. With uniform proposals and the same verifier, it solves 39 to 42. Separating the candidates without labels or an exact verifier remains open.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs reasoning, decoding, consistent errors, state decoding, belief tracking, test-time scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1e9iOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes why repeated sampling and selection can still fail to correct LLM reasoning errors. It decomposes decoding into candidate support, scoring, and search, and uses SAVI, a state-aware Viterbi decoder, to isolate these factors across several reasoning tasks.

Critical strengths
  1. The paper provides a clear diagnostic framework that decomposes reasoning failures into candidate support, scoring, and search, and uses controlled state-decoding interventions to localize which component is responsible for a failure.
  2. The experiments are careful within the studied settings. Exact and oracle controls are used explicitly to isolate different failure sources, revealing qualitatively different bottlenecks across Countdown, MuSR, and Belief-R.
Critical weaknesses
  1. The conceptual novelty appears limited relative to closely related prior work. e.g., generation-versus-selection failure analyses [1]; the studied self-consistent errors [2]; and to separate search errors from model/scoring errors [3]. SAVI itself mainly combines canonical states, edge scoring, and Viterbi inference, so I view its main contribution as unifying these ideas rather than introducing a substantially new algorithmic principle.
  2. TThe empirical evidence is narrow relative to the broad framing about LLM reasoning. Most core decoding and failure-attribution results use Qwen3-4B, while the additional Qwen2.5-7B and Llama-3.1-8B results are primarily representation-probing analyses rather than full replications of the decoding framework. Moreover, all three studied tasks admit manually specified discrete canonical states, and the main MuSR study focuses on only 19 certified hard cases.
  3. The strongest improvements depend heavily on oracle information. These experiments diagnose what would be sufficient to repair the failures, but leave automatic candidate construction and reliable non-oracle scoring largely unresolved.

[1] Fei Yu et al., “Scaling Flaws of Verifier-Guided Search in Mathematical Reasoning,” [2] Hexiang Tan et al., “Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs,” EMNLP 2025. [3] Felix Stahlberg and Bill Byrne, “On NMT Search Errors and Model Errors: Cat Got Your Tongue?”, EMNLP-IJCNLP 2019.

Questions for authors
  1. Novelty clarification. How do the authors distinguish the support/scoring decomposition from prior generation-versus-selection analyses, and consistent errors from prior self-consistent-error work? A clearer statement of what is new specifically at the state level would materially affect my assessment.
  2. Generality. Can the authors provide evidence that the same support/scoring failure patterns occur for another model family or on a less manually structured reasoning benchmark? The current full decoding analysis is largely concentrated on Qwen3-4B and task-specific canonical states.
  3. Non-oracle performance. How much of the oracle gain remains when both candidate construction and scoring are obtained without annotations, exact solvers, or human labels? This would clarify whether state decoding is primarily a diagnostic framework or can form the basis of a practical inference method.
3 comments
AuthorsOpenReview ↗

State Decoding: Analytical Contributions, Generality, and Practical Scope

We are grateful for your comments. Reasoning requires using the problem’s information to decide which step follows. Our central insight is that a correct candidate can be available while the model still prefers a step unsupported by that information. We extended the state-level analysis across models and another benchmark, tested ways to improve selection, and clarified the framework’s scope.

Q1/W1. State-level findings relative to prior work. Reasoning requires deciding what follows from the given information. Even when a correct candidate is available, the model must use that information to distinguish it from competing candidates. Our state-level analysis locates choices that fail to reflect the problem's information and tests which changes improve them. In Belief-R, an added premise can be a requirement or another route to the conclusion. Spelling out this relation yields 99.4% accuracy on the BU modus-ponens subset. Separately, likelihood selection between the two relations yields BA 0.500 under the sufficiency wording. On 11/19 certified MuSR items, all eligible chains have the correct queried belief before the decisive step but retain its old, now incorrect value when an update is required. With correct-answer paths supplied, exact search finds the highest-scoring path, yet likelihood scoring still yields 10/19 (52.6%). We locate the specific failed update, extending answer-level consistent-error analysis (Tan et al., 2025). Building on generation/selection analysis (Yu et al., 2025), we supply alternatives and use exact-search diagnosis (Stahlberg and Byrne, 2019) to test the model's own scores. The fixed-pool MuSiQue control (Trivedi et al., 2022), detailed in Q2, shows how changing the same model's scoring target improves selection.

Q2/W2. Evidence across models and benchmarks. To address your concern about model coverage, we repeated MuSR likelihood decoding on larger models and another model family, using the same items and candidate graphs. The results are close to Qwen3-4B’s 10/19 (Llama-3.1-8B: 9/19; Qwen3-8B: 8/19; Qwen3-14B: 10/19). Likelihood-based selection failures therefore persist across model sizes and families. We also extended the analysis to MuSiQue, a multi-hop question answering benchmark. Following the provided decompositions, Qwen3-4B extracts candidate answers from model-selected passages and records the accumulated subquestion–answer pairs as states. The resulting pool contains a correct answer on 363/600 (60.5%) items, but path scoring by summed paragraph-selection probabilities selects one on only 252/600 (42.0%). To examine this selection gap, we rescored the fixed pool using the same model’s probability that the selected passage supports the candidate answer. Correct selections rise to 295/600 (49.2%), a net gain of 7.2 percentage points, but 68 items still receive a wrong answer despite having a correct candidate. This extends the diagnosis to model-generated answer states and shows that changing the same model's scoring target can reduce the selection gap.

Q3/W3. Oracle dependence and automation. Regarding your question about oracle gains, the paper reports component comparisons on the same MuSR candidate graph: likelihood scoring repairs 4/19 items with content-free placeholders and 9/19 with model-written event sentences; annotated sentences with the fact verifier reach 15/19. Model-generated event information thus enables partial repair, while oracle inputs provide further gains. As for automation, SAVI was introduced as an analysis framework. A fully automated, oracle-free pipeline for MuSR and Belief-R remains open. Our component tests provide a basis for developing such methods by distinguishing failures in candidate generation from failures in selection and testing which changes improve state selection.

AuthorsOpenReview ↗

Updated PDF with Revisions

Thank you again for your suggestions on the state-level contribution, generality, and oracle dependence. We have updated the PDF, revising the wording of several statements and adding the new experimental results; the changes are shown in blue. We hope our response has clarified these points and the scope of the framework. If any questions remain, we are available for further discussion.

Thank you for the detailed rebuttal and additional experiments, which address some of my concerns. However, I remain unconvinced that the state-level framework offers sufficient conceptual novelty beyond existing analyses of generation-versus-selection and reasoning failures. Therefore, I will maintain my original rating.

Reviewer_dnreOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper seeks to explore how Qwen3-4B inference fails on three concrete tasks: Countdown, MusR, and Belief-R. The authors examined the following possible sources of errors (Figure 3):

  • Greedy inference may produce an answer with lower likelihood score than the majority vote of N samples
  • The majority vote of N samples may still produce the wrong answer although the correct answer might also be sampled
  • Drawing N samples may not cover the correct answer altogether

The authors found that the majority of error originates from the last two categories. Further, they observe that Qwen3-4B frequently make the same the mistake at certain steps, which they call "consistent error" at a "decisive step".

Because the tasks being evaluated can be easily cast as state search problems, they further examined whether exhaustive search (Algorithm 1) using the model score can lead to better answers. The authors then found

  • Model likelihood scores alone cannot find better answers even with exhaustive search
  • Exact verifiers along with model likelihood scores can help
Critical strengths

None.

Critical weaknesses
  • Experiment setup: This paper purports to explore LLM reasoning, yet the experiments were conducted on Qwen3-4B with thinking disabled (lines 748, 1050, and 1124). The setup on Countdown is particularly questionable: the model was not allowed trial-and-error in direct generation. Such a mode of inference is far from the common definition of "reasoning".
  • Main finding: The main finding of this paper can be summarized as "if a model does not assign high likelihood scores to the right answer, no practical decoding method will find the right answer" (and the model consistently making the same mistake is ostensibly also a result of poor log-likelihood scores). This rather obvious finding unfortunately does not provide actionable insights for improving LLM reasoning.
Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Thinking-Mode Experiments and Reasoning Insights

Thank you for your feedback. Reasoning involves working through possibilities and deciding what follows from the given information. Our central insight is that making correct candidates available does not ensure that this information guides their selection. We added thinking-mode experiments across all three tasks and controls to test how information and scoring affect selection.

W1. Thinking-mode results across three tasks.

Reasoning concerns drawing conclusions from the information given; thinking mode provides more scope to work through, check, and revise those inferences. To address your concern about trial and error, we added thinking-mode experiments on all three tasks, allowing free-form attempts and revisions on Countdown. On Countdown, majority voting over 16 free-form thinking responses per item solves 37/45 (82.2%) items, while none of the original 64 samples per item was correct on these selected items.

On the 19 certified MuSR items, direct thinking-mode generation with 1,024 responses per item produces a correct answer on 14/19 (73.7%) items, but majority voting solves only 2/19 (10.5%). On Belief-R, generated reading choices remain near chance (BA 0.508 versus 0.522 in the matched non-thinking control). These tests show a substantial improvement on Countdown, while the MuSR and Belief-R failures persist with thinking enabled.

W2. Insights for improving reasoning.

Reasoning requires deciding what follows from the given information. Having a correct candidate available does not establish that the model uses that information to select it over alternatives. We use state-level controls to locate selection failures and test ways to reduce them. Exact search finds the highest-scoring path, letting us examine the scores with search error controlled. Belief-R asks whether an added premise is a requirement or another route to the conclusion. The model answers accurately on the BU modus-ponens subset when this relation is spelled out; in a separate evaluation, likelihood does not reliably select the relation from the original text.

In MuSR, who sees a move determines whose belief should change. With correct-answer paths and exact search, likelihood decoding repairs 10/19 (52.6%) items; all 10 are also solved by a rule that updates everyone regardless of who saw the move. Annotating who saw each move raises repairs to 17/19 (89.5%) with model and search fixed. To test whether a different scoring target can also improve selection, we rescored a fixed pool of answers extracted from model-selected MuSiQue passages (Trivedi et al., 2022). Replacing paragraph-selection probabilities with the same model's probability that the passage supports the candidate answer raises correct selections from 252/600 (42.0%) to 295/600 (49.2%), a net gain of 7.2 percentage points.

Both controls improve selection with the model and alternatives fixed. These results suggest using process supervision (Lightman et al., 2024) to distinguish correct from common wrong successors of the same preceding state. Evaluation should check whether changing who saw a move changes whose belief is updated, rather than only whether the final answer is correct.

AuthorsOpenReview ↗

Updated PDF with Revisions

Thank you again for your comments on the reasoning settings and the implications of our findings. We have updated the PDF, revising the wording of several statements and adding the new experimental results; the changes are shown in blue. We hope the added thinking-mode experiments and clarifications have addressed your concerns. If any questions remain, we are available for further discussion.

Thanks for the additional results and clarification! However, my second concern remains valid. I do not find the techniques employed in improving the MuSR results are practical or generally applicable. For that reason, my rating remains unchanged.

Reviewer_2VWWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether multi-step LLM reasoning fails because the correct intermediate state never appears among the candidates (support) or because it appears but is ranked below a wrong one (scoring). It defines a consistent error, in which every sampled chain that reaches a decisive step writes the same wrong successor, so the correct successor is absent from the sampled pool and no selection over that pool can choose it at that step. It also proposes a four-term accuracy decomposition and SAVI, a Viterbi search over a state trellis that is exact when all states are kept (on MuSR and Belief-R; Countdown uses a K=8 beam), so that remaining errors can be attributed to support or scoring.

Using Qwen3-4B, the paper reports three findings. On Countdown, frequency scoring fails, a learned head helps modestly, and an exact verifier solves nearly all hard items, even with uniform proposals. On 19 selected MuSR items, the model's likelihood does not significantly outperform a uniform draw, and falls below it in two of four settings, even when the correct belief table is enumerated, whereas an annotated-fact verifier fixes 15/19. On Belief-R, the likelihood is at chance and flips with wording, while a probe trained on human labels does better. The paper concludes that consistent errors dominate unrepaired failures and that only signals external to the likelihood select the correct state.

Critical strengths

1. An experimental design that separates "cannot generate" from "cannot recognize." Enumerating the correct state into the candidates under exact search isolates scoring failures from support failures. On MuSR, the likelihood then does not significantly outperform a uniform draw and follows the cue sentence rather than who saw the move. This is the paper's most informative result and the strength that weighs most in its favor.

2. Transparent and carefully controlled reporting. Uniform-draw and model-free baselines, paired intervals, ORACLE marking and disclosed pre-registration departures make the narrow findings credible, though not the broader claims.

Critical weaknesses

1. The key inferences go beyond what the experiments establish.

The §6 conclusion that only signals external to the likelihood can select the correct state rests on one likelihood construction: a single fixed-template rendering per state, without the previously generated reasoning and without calibration for surface-form competition [5]. Belief-R already shows its preference flipping across wordings (AUC 0.526 vs. 0.380), and calibrated scoring, thinking mode and decoding with any model other than Qwen3-4B were not tested, so the failure may belong to this scorer rather than to the model's likelihood. The strongest repairs come from oracles, while the one learned substitute helps only modestly (0.20 → 0.40 on 45 Countdown items selected for sampling failure), and the claim that consistent errors dominate unrepaired failures is measured on 19 MuSR items selected for voting failure. The MuSR attribution is also unverified: §4.3 places the failure in extracting who saw each move, yet App. C.1 states that the self-extracted facts were never checked against the annotation, whereas Jung et al. [4] find that models infer who perceives an event well but struggle to infer beliefs from it. Finally, the Belief-R probe is outperformed by a character-length rule on all rows (BREU 0.746 vs. 0.684); on the length-matched subset it scores higher (0.596 vs. 0.517) but not significantly, and that subset's criterion was relaxed after the results were seen.

2. Limited novelty over closely related prior work.

The step-level consistent error extends answer-level self-consistent errors [1], support vs. scoring failure mirrors generation vs. selection failure [2], and attributing errors via exact search follows [3]. The paper cites these, but the added value of its state-level diagnostic is modest: the four-term split is an accounting identity over greedy, maj@N and pass@N, and Viterbi decoding over a merged trellis is standard. The wording sensitivity of the likelihood is a known effect of surface-form competition [5], and the conclusion that external signals are needed parallels findings that LLMs struggle to self-correct reasoning without external feedback [6]. No oracle-free way to obtain such signals is offered, which limits the practical contribution. The text is also hard to read: dense task-specific terminology and heavy qualification obscure the main claim, and an important detail, the post hoc relaxation of the pre-registered criterion, appears only in App. D.

References

  1. Tan et al. Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs.
  2. Yu et al. Scaling Flaws of Verifier-Guided Search in Mathematical Reasoning.
  3. Stahlberg & Byrne. On NMT Search Errors and Model Errors: Cat Got Your Tongue?
  4. Jung et al. Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models.
  5. Holtzman et al. Surface Form Competition: Why the Highest Probability Answer Isn't Always Right.
  6. Huang et al. Large Language Models Cannot Self-Correct Reasoning Yet.
Questions for authors
  1. Is the likelihood failure robust to a stronger likelihood scorer? Beyond the three Belief-R wordings and the probed models, does the MuSR and Belief-R failure persist under calibrated scoring, such as PMI or domain-conditional normalization [5], aggregation over paraphrases of each state, or a thinking-mode or larger model? Relatedly, decoding scores each step without the previously generated reasoning (the story remains in the prompt), which App. A.1 says was not tested: does restoring that history change the MuSR conclusions? If the failure persists, the central claim is much stronger; if not, it concerns this scorer rather than the model.

  2. How much does the MuSR annotated-fact verifier filter? Per item, how many enumerated terminal states survive it, and what is the expected accuracy of a uniform draw over only the surviving states? If that is close to 15/19, the gain over the sampled pool may come mainly from the enumerated support and oracle filtering, leaving the added value of likelihood ranking and path search unclear.

  3. How prevalent are decisive consistent errors? App. C.1 reports unanimous-wrong steps on 32 of 40 judgeable items, but checks neither decisiveness nor naming and event alignment. Could you report the rate of consistent errors after these checks on those 40 items? The title-level claim depends on this prevalence, which the 19 certified items cannot establish.

  4. Does the Belief-R probe add information beyond sentence length? Please state the original and relaxed length-matched criteria and the result under the frozen split, and test whether the probe predicts the reading beyond a length-only model, for example with features residualized on the character-count difference.

  5. Was the extraction failure verified? Could you score the model's self-extracted facts (who moved each object and who saw each move) against the annotation, to confirm that the MuSR failure lies in extraction rather than in updating beliefs from correctly extracted facts?

3 comments
AuthorsOpenReview ↗

Robustness, Failure Attribution, and Analytical Contributions

We appreciate your review. Reasoning requires choosing steps supported by the given information. Our central insight is that correct candidates and exact search do not ensure that the model makes such choices. State-level controls let us examine this gap. We added scorer checks, probe controls and a fact audit.

Q1. Likelihood failures under stronger scorers. On MuSR, calibrating with PMI, aggregating state renderings, or adding previously decoded state tables to the scoring prompt yields 8–11/19 repairs, close to the original 10/19. Larger and different-family models give similar results (Llama-3.1-8B: 9/19; Qwen3-8B: 8/19; Qwen3-14B: 10/19). Thinking-conditioned scoring increases repairs from 10/19 to 11/19. On Belief-R, despite a modest gain for Qwen3-8B under sufficiency (BA 0.533, 95% CI [0.514, 0.552]), direct selection by mean token log-likelihood still shows no reliable above-chance advantage in the other five tested conditions. The gains are partial: likelihood-based state-selection failures remain.

Q2. Contributions of filtering, ranking, and search. With the graph and likelihood scores fixed, annotated-fact filtering accounts for most of the gain; global search adds one repair. Terminal counts fall by 33.3–66.7% on 9/19 items and are unchanged on the other 10. Uniform selection among the surviving terminal states repairs 12/19 in expectation. Here, likelihood ranking shows no reliable additional gain.

DecodingWithout fact filterWith fact filter
Greedy9/1914/19
Viterbi10/1915/19

Q3. Frequency of consistent errors. Of 40 unfiltered items, 18 fail majority voting. After matching object and location name variants, 9 of these failures are judgeable (≥64 chains with a correct preceding queried belief). On 7/9 (77.8%) judgeable failures, all eligible chains write the same wrong value at the decisive step (the queried belief’s last annotated update), with no later recovery.

Q4. Probe information beyond sentence length. As suggested, we regressed probe features on character-count difference within training folds and retrained the probe after subtracting the fitted component. Held-out BA is 0.615 [0.553, 0.679], 0.597 [0.536, 0.663] and 0.618 [0.561, 0.675] for Qwen3-4B, Qwen2.5-7B and Llama-3.1-8B, respectively. These results support a reading signal beyond this linear length component. On the probe criteria, the paper’s length-matched check retained a 95% CI excluding 0.5 but removed BA ≥0.60. The CI criterion supports the narrower claim that a signal remains beyond length; BA ≥0.60 additionally requires a specified level of predictive accuracy. Qwen3-4B’s frozen-split result remains 0.596 [0.522, 0.674].

Q5. Self-extracted facts and failure attribution. We audited the original extract-then-answer control (F3) against the annotations. Witness information was reported for 83/114 (72.8%) event–observer pairs; 71/83 (85.5%) matched the annotations. On 8 items, the extracted facts matched the annotations for the checked initial location, move details and decisive witnesses, yet all 8 still failed F3 majority voting. F3 tests answer repair by self-extraction, without isolating the cause. We will retain the screen but replace “The failure therefore lies in this extraction” with “Self-extraction prompting did not reliably fix these items.”

W2. Contribution relative to prior work. Reasoning requires deciding what follows from the given information. With a correct candidate available, the model must still use this information to reject wrong ones; we locate failures and test what helps. This builds on answer-level consistent errors (Tan et al., 2025), generation/selection (Yu et al., 2025) and exact search (Stahlberg and Byrne, 2019). The four-term split separates missing correct answers from those missed by voting; Viterbi finds MuSR's highest-scoring path exactly to test the scores. In MuSR, only witnesses should update. The story is provided, yet likelihood paths mostly retain old beliefs without event sentences (51/57 move layers) or update everyone's belief with annotated move sentences (49/57). Both patterns treat observers and non-observers alike; corresponding rules also solve the repaired items. Annotating who saw each move raises repairs from 10/19 (52.6%) to 17/19 (89.5%) with model and search fixed. This information control complements work on wording effects (Holtzman et al., 2021) and self-correction limits (Huang et al., 2024). A learned update score should reflect who saw the move. For oracle-free signals, Countdown learns reachability from verifier labels without test-time exact verification; comparable automatic signals for MuSR and Belief-R remain open. For readability, we will foreground our central insight and simplify the terminology.

AuthorsOpenReview ↗

Updated PDF with Revisions

Thank you again for your detailed suggestions on scoring robustness and failure attribution. We have updated the PDF, revising the wording of several statements and adding the new experimental results; the changes are shown in blue. We hope the additional controls and clarifications have addressed your concerns. If any questions remain, we are available for further discussion.

Thank you for the clarifications and additional experiments. While I appreciate the new results, my primary concerns regarding the limited novelty over prior work and the reliance on oracle signals over small subsets remain unresolved, so I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.