acceptodds
Withdrawn from ICLR 2027

The Sampling Ceiling: Why Test-Time Self-Consistency Saturates and an Oracle Rule for Allocating Samples

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-consistency can sample a correct answer and still vote for a wrong one. We study how confidence in the current answer differs from the value of drawing more samples. In modular addition, 40 samples contain a correct answer on 95.34% of instances, yet voting accuracy converges to 63.32% (three-seed means). In our systematic-stochastic mixture, voting can remove random errors on some instances, while on the rest the model always repeats one wrong answer. Each extra pair of samples then adds less majority-vote accuracy than the last, so a greedy oracle allocation is optimal for equally weighted queries with known per-query mixture parameters. When these are unknown, Correlation-Aware Self-Consistency (CASC) uses labeled population data to score the current answer's correctness and the risk of a systematic error. On public Grade School Math 8K (GSM8K) answer sets, the resulting tradeoffs depend on the model: default CASC uses 3.67 answers versus Early-stopping Self-Consistency's 8.24 on GPT-4, with a paired 95% interval for the accuracy difference that includes zero. Removing CASC's systematic-error term leaves its GPT-4 predictions and stopping times unchanged; Adaptive-Consistency's Beta rule is more accurate and cheaper on GPT-3.5. The oracle ranks queries by the gain from additional samples, and the stopping comparisons show that scores for the current answer do not measure that gain.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
test-time compute, self-consistency, correlated sampling, adaptive stopping, compute allocation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zcjaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when repeated sampling and self-consistency help, and when voting instead reinforces a model's wrong preferred answer. It introduces a systematic-stochastic mixture (SSM) in which some queries are recoverable, while others always produce the same wrong answer, termed the “lure.” Under this model, the paper derives a ceiling for voting accuracy and an oracle allocation rule for distributing samples across queries. It also proposes Correlation-Aware Self-Consistency (CASC), which uses labeled calibration data to score a current answer and the risk of a systematic error, then decides whether to answer, sample again, or abstain. The paper's analysis is interesting, but its empirical comparison is not conclusive.

Critical strengths
  1. The paper clearly illustrates that a sample pool can contain a correct answer without plurality selecting it. In modular addition, 40 samples contain a correct response for 95.34% of instances at the reported transition checkpoint, while the plurality vote is correct on 62.30%. This is a useful way to characterize the limits of self-consistency and motivates separating the value of another sample from confidence in the current leader.

  2. Under the SSM assumptions and known query parameters, the paper derives diminishing gains from adding sample pairs and an optimal greedy allocation across equally weighted query types.

  3. Modular addition and multiplication let the authors inspect complete answer distributions and compare observed voting behavior with exact limits. The public GSM8K answer banks add a complementary, if limited, check on language-model outputs.

Critical weaknesses
  1. CASC is not compared with stronger test-time scaling strategies. The central practical claim concerns deciding whether another sample is worthwhile, but the empirical baselines are self-consistency, agreement-based stopping rules, entropy stopping, and greedy decoding. The related work briefly mentions verifier-guided allocation (Snell et al., 2024), but CASC is not empirically compared with methods that use an external verifier to select or score candidates. Such comparisons are important because verifiers provide a signal different from answer agreement or a calibrated population model, and can change both accuracy and cost. Separately, the paper should position its analysis and allocation strategy against recent work such as Feng et al. (ICML 2026), which studies improving self-consistency, scaling laws, and allocation strategies; this is relevant related work. Comparisons should account for total compute or latency, verifier calls where applicable, sample budgets, and access to labeled calibration data. Without stronger comparisons, the results establish how CASC fares against the selected stopping rules, but not its relative value among broader test-time scaling strategies.

  2. The SSM and calibration pipeline are not explained sufficiently in the main text. The paper defines the “lure” as an instance-specific wrong answer repeated on every draw for a systematic instance, but important details needed to assess this model and its use appear in the appendices. These include how the SSM parameters and wrong-answer concentration are estimated from finite labeled streams, how the fit is assessed, and how the implementation's approximate CASC scores are derived. Since the SSM underpins the sampling ceiling, oracle allocation, and CASC, readers should not have to reconstruct its assumptions and empirical adequacy from supplementary material. The main text should give a simple example, explain what the parameters mean, summarize the fit diagnostics and limitations, and distinguish conclusions that hold generally from those requiring SSM. CASC does use labeled calibration data but does not require correctness labels at inference; the source of calibration labels, split procedure, threshold selection, and expected transfer to new data should also be clear. This matters especially because the practical CASC scores are heuristic: the formal systematic-error guarantee described in the paper does not apply to CASC itself.

  3. The framing of agreement-based confidence risks overstating the point. Section 3.3 shows that high agreement can coexist with a wrong answer under the SSM. This is a valid caution about interpreting agreement-based confidence as correctness confidence, but self-consistency is a voting procedure and does not itself claim that consensus is a calibrated estimate of correctness. The paper should position this result specifically as a limitation of stopping rules that interpret vote concentration as correctness confidence, rather than as a shortcoming of self-consistency per se. The more substantive question is whether CASC's additional population-level calibration signal improves decisions enough to justify the calibration requirements and complexity.

  4. The language-model evidence is narrow, offline, and model-dependent. The GSM8K evaluation uses fixed public answer banks from GPT-3.5, GPT-4, and Llama-2 7B rather than live inference. The relative gains are inconsistent across models: on GPT-4, default CASC uses fewer answers than ESC with a statistically inconclusive accuracy difference, whereas on GPT-3.5, ASC/Beta is both more accurate and cheaper than CASC. The authors appropriately note that model versions are unknown and calibration may not transfer. This is useful initial evidence, but it is not enough to establish deployment performance, especially for a stopping policy whose benefits depend on the answer distribution and calibration population. Results on additional tasks, model families, and fresh inference would strengthen the claims.

Questions for authors
  1. Can CASC be compared against test-time scaling that uses an external verifier, with matched total compute or latency and clearly specified access to calibration labels? Separately, how does its analysis and allocation strategy relate to Feng et al. (ICML 2026) on self-consistency improvements and sample allocation?

  2. Could the main text summarize the SSM's assumptions, parameter interpretation, finite-sample fitting, and goodness-of-fit diagnostics currently relegated to the appendices? Can it also give an end-to-end account of calibration labels and threshold selection, and distinguish which guarantees apply to the oracle versus implemented CASC?

  3. Can the authors clarify the intended role of the modular-arithmetic transformers: are they a mechanistic testbed for exposing exact answer distributions, or evidence intended to support claims about language-model deployment? It seems that the authors didn't fully exploit the possibilities offered by this rather toy testbed.

Reviewer_4bNCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explains why self-consistency stops improving even when a correct answer is often sampled. On some instances the model always repeats the same wrong answer. Modeling this as a systematic-stochastic mixture, the authors show that voting accuracy is capped at 1−α1-\alpha (the share of instances that can be fixed), that each extra pair of samples helps less, and that a greedy oracle allocation is optimal when per-query parameters are known. They also propose CASC, a stopping rule that flags likely repeated errors. In modular addition, 40 samples contain a correct answer on 95.34% of instances, yet voting is capped at 63.32%. On GSM8K, CASC's cost–accuracy tradeoff depends on the model.

Critical strengths
  1. Clean theory explaining why voting saturates. SSM gives an exact binary-majority curve and ceiling. It also shows that, on systematic instances, the Beta stopping rule commits after only 4 samples, at nominal 95% confidence, to an answer that is always wrong. This separates confidence in the current answer from the value of sampling more.
  2. Controlled and transparent evaluation. All methods use the same data splits, draw cap and answer sequence for each instance, and on GSM8K the same answer parser and five answer orders. The authors openly report negative results, e.g., CASC's systematic-error term gives no detectable accuracy benefit, and ASC/Beta is more accurate and cheaper than CASC on GPT-3.5.
Critical weaknesses
  1. The proposed method does not deliver. CASC's distinctive component, the systematic-error term, "mainly delays decisions" with no detectable accuracy benefit. On GPT-3.5, ASC/Beta is more accurate and cheaper than CASC (85.86% / 10.13 vs. 85.13% / 20.40 draws; Table 2). On GPT-4, CASC uses fewer answers than ESC (3.67 vs. 8.24), but its accuracy gain has a 95% CI of [−0.18, 1.00] points, including 0.
  2. SSM does not fit the data. At the transition checkpoints, instances "fail SSM's third-moment test", and 15.89%/12.25% of instances have a correct mode with probability at most 1/2. The authors state that SSM "does not exactly fit the learned populations", so the diminishing-gain result and oracle guarantee "do not extend to arbitrary plurality distributions".
  3. Limited novelty of the main finding. That coverage grows while selected-answer accuracy plateaus is already reported by Brown et al. (2024), and Bay & Yearick (2026) study mode and correlation ceilings and their implications for allocation. The stated difference is only that the paper derives exact gains and an oracle allocation "for one latent mixture", which, given Weakness 3, does not fit the learned models.
  4. Narrow experimental scope. Experiments use one-layer transformers on modular arithmetic with single-token answers and one text dataset, GSM8K, via three previously released answer banks whose exact model versions are unknown. Results vary widely across seeds (e.g., 62.4 ± 36.6% for CASC on addition; Table 1), and "three-seed means cannot rank methods statistically". CASC also needs labeled calibration data, unlike ESC and ASC/Beta.
Questions for authors
  1. Given that the systematic-error term gives no accuracy benefit, in what setting does CASC offer a clear advantage over ASC/Beta or ESC? Or what subsets might CASC be useful for?
  2. At the transition checkpoints, every correct-answer probability is positive, so no instance is systematic in SSM's sense. What error does CASC's α\alpha then capture, and on GSM8K how many questions have all 40 answers agreeing on a wrong answer, the case the systematic-error term is meant for?
  3. The voting limit equals Greedy accuracy whenever each instance has a unique mode, without SSM, and the ceiling 1−α1-\alpha follows from defining systematic instances as never correct. Which result gives an insight not already in Brown et al. (2024) or Bay & Yearick (2026)?
  4. Can you evaluate current reasoning models and more benchmarks (e.g., MATH, AIME) with freshly generated samples?
Reviewer_S56nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper distinguishes confidence in the current vote from the value of additional sampling. It derives sampling gains and an oracle allocation rule under a systematic–stochastic mixture model, proposes CASC as a practical stopping heuristic, and evaluates voting limits, calibration, and sampling costs on arithmetic tasks and public GSM8K responses.

Critical strengths
  • The distinction between current confidence and future sampling value is useful. The arithmetic experiments illustrate the underlying selection problem: 40 samples achieve 95.34% answer availability, while the plurality limit is 63.32%.

  • The experiments include meaningful controls: shared response streams, separate calibration and evaluation splits, tuned baselines, and component ablations. The third-moment diagnostic also directly tests the model’s assumptions.

  • The negative results are informative. In particular, calibration improvements do not necessarily improve stopping decisions, and the proposed systematic-error component often adds cost without a detectable accuracy benefit. The authors report these findings clearly and distinguish CASC from the theoretical oracle.

Critical weaknesses
  • The allocation theory needs clearer positioning against PETS (Liu et al., 2026), which the paper already cites. Theorem 2’s gain matches PETS Lemma C.1 up to the factor 1−α1-\alpha, and Theorem 3 uses the same diminishing-returns allocation structure. The mixture interpretation and diagnostics carry more novelty than the basic allocation principle.

  • The SSM assumptions are restrictive: recoverable instances require q>1/2q>1/2, while systematic instances always repeat one wrong answer. The measured distribution mismatch limits how directly the oracle results explain practical plurality voting.

  • CASC’s additional component has limited demonstrated benefit. Removing it leaves GPT-4 decisions unchanged and reduces GPT-3.5 mean draws from 20.40 to 5.12 with an unresolved accuracy change. Generalization beyond the fixed response banks also remains uncertain.

Questions for authors
  • Under the binary SSM, with at least one sample per query and no abstention, mode agreement appears to equal Ai(n)+αiA_i(n)+\alpha_i. Since the added term is independent of allocation, what additional allocation result follows from changing the stated objective?

  • Do the existing results identify conditions where retaining the systematic-error term improves the accuracy–cost tradeoff?

  • Can the measured SSM discrepancy be related to errors in predicted sampling gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.