- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies when repeated sampling and self-consistency help, and when voting instead reinforces a model's wrong preferred answer. It introduces a systematic-stochastic mixture (SSM) in which some queries are recoverable, while others always produce the same wrong answer, termed the “lure.” Under this model, the paper derives a ceiling for voting accuracy and an oracle allocation rule for distributing samples across queries. It also proposes Correlation-Aware Self-Consistency (CASC), which uses labeled calibration data to score a current answer and the risk of a systematic error, then decides whether to answer, sample again, or abstain. The paper's analysis is interesting, but its empirical comparison is not conclusive.
Critical strengths
-
The paper clearly illustrates that a sample pool can contain a correct answer without plurality selecting it. In modular addition, 40 samples contain a correct response for 95.34% of instances at the reported transition checkpoint, while the plurality vote is correct on 62.30%. This is a useful way to characterize the limits of self-consistency and motivates separating the value of another sample from confidence in the current leader.
-
Under the SSM assumptions and known query parameters, the paper derives diminishing gains from adding sample pairs and an optimal greedy allocation across equally weighted query types.
-
Modular addition and multiplication let the authors inspect complete answer distributions and compare observed voting behavior with exact limits. The public GSM8K answer banks add a complementary, if limited, check on language-model outputs.
Critical weaknesses
-
CASC is not compared with stronger test-time scaling strategies. The central practical claim concerns deciding whether another sample is worthwhile, but the empirical baselines are self-consistency, agreement-based stopping rules, entropy stopping, and greedy decoding. The related work briefly mentions verifier-guided allocation (Snell et al., 2024), but CASC is not empirically compared with methods that use an external verifier to select or score candidates. Such comparisons are important because verifiers provide a signal different from answer agreement or a calibrated population model, and can change both accuracy and cost. Separately, the paper should position its analysis and allocation strategy against recent work such as Feng et al. (ICML 2026), which studies improving self-consistency, scaling laws, and allocation strategies; this is relevant related work. Comparisons should account for total compute or latency, verifier calls where applicable, sample budgets, and access to labeled calibration data. Without stronger comparisons, the results establish how CASC fares against the selected stopping rules, but not its relative value among broader test-time scaling strategies.
-
The SSM and calibration pipeline are not explained sufficiently in the main text. The paper defines the “lure” as an instance-specific wrong answer repeated on every draw for a systematic instance, but important details needed to assess this model and its use appear in the appendices. These include how the SSM parameters and wrong-answer concentration are estimated from finite labeled streams, how the fit is assessed, and how the implementation's approximate CASC scores are derived. Since the SSM underpins the sampling ceiling, oracle allocation, and CASC, readers should not have to reconstruct its assumptions and empirical adequacy from supplementary material. The main text should give a simple example, explain what the parameters mean, summarize the fit diagnostics and limitations, and distinguish conclusions that hold generally from those requiring SSM. CASC does use labeled calibration data but does not require correctness labels at inference; the source of calibration labels, split procedure, threshold selection, and expected transfer to new data should also be clear. This matters especially because the practical CASC scores are heuristic: the formal systematic-error guarantee described in the paper does not apply to CASC itself.
-
The framing of agreement-based confidence risks overstating the point. Section 3.3 shows that high agreement can coexist with a wrong answer under the SSM. This is a valid caution about interpreting agreement-based confidence as correctness confidence, but self-consistency is a voting procedure and does not itself claim that consensus is a calibrated estimate of correctness. The paper should position this result specifically as a limitation of stopping rules that interpret vote concentration as correctness confidence, rather than as a shortcoming of self-consistency per se. The more substantive question is whether CASC's additional population-level calibration signal improves decisions enough to justify the calibration requirements and complexity.
-
The language-model evidence is narrow, offline, and model-dependent. The GSM8K evaluation uses fixed public answer banks from GPT-3.5, GPT-4, and Llama-2 7B rather than live inference. The relative gains are inconsistent across models: on GPT-4, default CASC uses fewer answers than ESC with a statistically inconclusive accuracy difference, whereas on GPT-3.5, ASC/Beta is both more accurate and cheaper than CASC. The authors appropriately note that model versions are unknown and calibration may not transfer. This is useful initial evidence, but it is not enough to establish deployment performance, especially for a stopping policy whose benefits depend on the answer distribution and calibration population. Results on additional tasks, model families, and fresh inference would strengthen the claims.
Questions for authors
-
Can CASC be compared against test-time scaling that uses an external verifier, with matched total compute or latency and clearly specified access to calibration labels? Separately, how does its analysis and allocation strategy relate to Feng et al. (ICML 2026) on self-consistency improvements and sample allocation?
-
Could the main text summarize the SSM's assumptions, parameter interpretation, finite-sample fitting, and goodness-of-fit diagnostics currently relegated to the appendices? Can it also give an end-to-end account of calibration labels and threshold selection, and distinguish which guarantees apply to the oracle versus implemented CASC?
-
Can the authors clarify the intended role of the modular-arithmetic transformers: are they a mechanistic testbed for exposing exact answer distributions, or evidence intended to support claims about language-model deployment? It seems that the authors didn't fully exploit the possibilities offered by this rather toy testbed.