- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyzes why repeated sampling and selection can still fail to correct LLM reasoning errors. It decomposes decoding into candidate support, scoring, and search, and uses SAVI, a state-aware Viterbi decoder, to isolate these factors across several reasoning tasks.
Critical strengths
- The paper provides a clear diagnostic framework that decomposes reasoning failures into candidate support, scoring, and search, and uses controlled state-decoding interventions to localize which component is responsible for a failure.
- The experiments are careful within the studied settings. Exact and oracle controls are used explicitly to isolate different failure sources, revealing qualitatively different bottlenecks across Countdown, MuSR, and Belief-R.
Critical weaknesses
- The conceptual novelty appears limited relative to closely related prior work. e.g., generation-versus-selection failure analyses [1]; the studied self-consistent errors [2]; and to separate search errors from model/scoring errors [3]. SAVI itself mainly combines canonical states, edge scoring, and Viterbi inference, so I view its main contribution as unifying these ideas rather than introducing a substantially new algorithmic principle.
- TThe empirical evidence is narrow relative to the broad framing about LLM reasoning. Most core decoding and failure-attribution results use Qwen3-4B, while the additional Qwen2.5-7B and Llama-3.1-8B results are primarily representation-probing analyses rather than full replications of the decoding framework. Moreover, all three studied tasks admit manually specified discrete canonical states, and the main MuSR study focuses on only 19 certified hard cases.
- The strongest improvements depend heavily on oracle information. These experiments diagnose what would be sufficient to repair the failures, but leave automatic candidate construction and reliable non-oracle scoring largely unresolved.
[1] Fei Yu et al., “Scaling Flaws of Verifier-Guided Search in Mathematical Reasoning,” [2] Hexiang Tan et al., “Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs,” EMNLP 2025. [3] Felix Stahlberg and Bill Byrne, “On NMT Search Errors and Model Errors: Cat Got Your Tongue?”, EMNLP-IJCNLP 2019.
Questions for authors
- Novelty clarification. How do the authors distinguish the support/scoring decomposition from prior generation-versus-selection analyses, and consistent errors from prior self-consistent-error work? A clearer statement of what is new specifically at the state level would materially affect my assessment.
- Generality. Can the authors provide evidence that the same support/scoring failure patterns occur for another model family or on a less manually structured reasoning benchmark? The current full decoding analysis is largely concentrated on Qwen3-4B and task-specific canonical states.
- Non-oracle performance. How much of the oracle gain remains when both candidate construction and scoring are obtained without annotations, exact solvers, or human labels? This would clarify whether state decoding is primarily a diagnostic framework or can form the basis of a practical inference method.
3 comments
Updated PDF with Revisions
Thank you again for your suggestions on the state-level contribution, generality, and oracle dependence. We have updated the PDF, revising the wording of several statements and adding the new experimental results; the changes are shown in blue. We hope our response has clarified these points and the scope of the framework. If any questions remain, we are available for further discussion.
Thank you for the detailed rebuttal and additional experiments, which address some of my concerns. However, I remain unconvinced that the state-level framework offers sufficient conceptual novelty beyond existing analyses of generation-versus-selection and reasoning failures. Therefore, I will maintain my original rating.
State Decoding: Analytical Contributions, Generality, and Practical Scope
We are grateful for your comments. Reasoning requires using the problem’s information to decide which step follows. Our central insight is that a correct candidate can be available while the model still prefers a step unsupported by that information. We extended the state-level analysis across models and another benchmark, tested ways to improve selection, and clarified the framework’s scope.
Q1/W1. State-level findings relative to prior work. Reasoning requires deciding what follows from the given information. Even when a correct candidate is available, the model must use that information to distinguish it from competing candidates. Our state-level analysis locates choices that fail to reflect the problem's information and tests which changes improve them. In Belief-R, an added premise can be a requirement or another route to the conclusion. Spelling out this relation yields 99.4% accuracy on the BU modus-ponens subset. Separately, likelihood selection between the two relations yields BA 0.500 under the sufficiency wording. On 11/19 certified MuSR items, all eligible chains have the correct queried belief before the decisive step but retain its old, now incorrect value when an update is required. With correct-answer paths supplied, exact search finds the highest-scoring path, yet likelihood scoring still yields 10/19 (52.6%). We locate the specific failed update, extending answer-level consistent-error analysis (Tan et al., 2025). Building on generation/selection analysis (Yu et al., 2025), we supply alternatives and use exact-search diagnosis (Stahlberg and Byrne, 2019) to test the model's own scores. The fixed-pool MuSiQue control (Trivedi et al., 2022), detailed in Q2, shows how changing the same model's scoring target improves selection.
Q2/W2. Evidence across models and benchmarks. To address your concern about model coverage, we repeated MuSR likelihood decoding on larger models and another model family, using the same items and candidate graphs. The results are close to Qwen3-4B’s 10/19 (Llama-3.1-8B: 9/19; Qwen3-8B: 8/19; Qwen3-14B: 10/19). Likelihood-based selection failures therefore persist across model sizes and families. We also extended the analysis to MuSiQue, a multi-hop question answering benchmark. Following the provided decompositions, Qwen3-4B extracts candidate answers from model-selected passages and records the accumulated subquestion–answer pairs as states. The resulting pool contains a correct answer on 363/600 (60.5%) items, but path scoring by summed paragraph-selection probabilities selects one on only 252/600 (42.0%). To examine this selection gap, we rescored the fixed pool using the same model’s probability that the selected passage supports the candidate answer. Correct selections rise to 295/600 (49.2%), a net gain of 7.2 percentage points, but 68 items still receive a wrong answer despite having a correct candidate. This extends the diagnosis to model-generated answer states and shows that changing the same model's scoring target can reduce the selection gap.
Q3/W3. Oracle dependence and automation. Regarding your question about oracle gains, the paper reports component comparisons on the same MuSR candidate graph: likelihood scoring repairs 4/19 items with content-free placeholders and 9/19 with model-written event sentences; annotated sentences with the fact verifier reach 15/19. Model-generated event information thus enables partial repair, while oracle inputs provide further gains. As for automation, SAVI was introduced as an analysis framework. A fully automated, oracle-free pipeline for MuSR and Belief-R remains open. Our component tests provide a basis for developing such methods by distinguishing failures in candidate generation from failures in selection and testing which changes improve state selection.