acceptodds
Under review as a conference paper at ICLR 2027

Depth-Entropy Guided Sampling for Training-Free LLM Reasoning

Abstract

Reinforcement learning (RL) has become the dominant paradigm for improving the reasoning capabilities of large language models, but it requires expensive training, curated data, and reward signals. Recent work shows that sampling from sharpened base-model distributions at test time recovers much of the RL gain, yet existing methods rely solely on output-layer likelihoods and ignore the transformer’s internal forward-pass dynamics. We introduce Depth-Entropy Guided Sampling (DEGS), a training-free, test-time method that exploits layer-wise entropy collapse as an intrinsic quality signal. We observe that stronger reasoners—including RL-posttrained variants—exhibit a distinctive “late collapse”: logit-lens–decoded entropy stays elevated until deeper layers before converging. We define a per-sequence collapse depth and a joint objective that combines sequence likelihood with this depth-entropy structure, instantiated inside an MCMC power-sampling framework (DEGS-MCMC). Across three open-weight models and four reasoning benchmarks, this near-chance per-candidate signal compounds over the sampling trajectory into state-of-the-art training-free accuracy, with gains largest out of domain and on the harder splits—exactly where likelihood alone falls short—at single-digit-percent wall-clock overhead. DEGS narrowly trails an in-house GRPO reference on the math splits GRPO was trained for, yet surpasses it out of domain on GPQA for all three models, without any training, reward model, or labeled data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reasoning, training-free decoding, test-time computation, depth-entropy guided sampling, MCMC power sampling, layer-wise entropy, logit lens

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tuHYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Recent studies have found a useful test-time effect. If we sharpen the output probability distribution of a base model, the model becomes more likely to choose answers that it already considers probable. This can achieve many effects similar to those brought by reinforcement learning.

Based on these findings, this work proposes Depth-Entropy Guided Sampling (DEGS). DEGS measures how internal entropy changes during generation. It then computes a sequence-level “collapse depth” D(x)D(x). This depth is combined with the standard sequence probability. Together, they can be used to judge the quality of a response.

The authors also combine DEGS with an MCMC power-sampling framework. This gives DEGS-MCMC. It uses two signals to guide sampling. One signal is probability. The other signal is collapse depth. With these signals, the sampling process gradually becomes more likely to find higher-quality answers.

Critical strengths

The paper finds that the depth of internal entropy collapse in the model is correlated with response quality.

The paper proposes a training-free sampling optimization method that is effective on Qwen2.5-7B.

Critical weaknesses

Using internal entropy to analyze response quality is not a very novel idea. Moreover, although the paper finds that collapse depth needs to be combined with logits to effectively judge response quality, it does not specifically analyze how reliable this metric actually is. This further weakens the contribution of the work.

All experiments in the paper are conducted on the Qwen2.5-7B architecture, so it cannot be ensured that the findings of this work are generalizable.

Questions for authors

Are collapse depths correlated with response quality on architectures such as LLaMA3, Qwen3, and Qwen3.5, as well as at other parameter scales? Do Best-of-N-Entropy and DEGS-MCMC remain effective?

When π(x)\pi(x) is used as a per-candidate discriminator, what are its AUC, Spearman correlation, and pairwise-ranking accuracy?

What differences are there between responses with higher π(x)\pi(x) and those with lower π(x)\pi(x) in terms of reasoning patterns, generation length, and other behaviors? Are there any stable patterns?

Some high-entropy tokens usually appear in responses. What impact do these tokens have on collapse depth? How do different values of τ\tau affect the collapse depth at these positions?

Under more hyperparameter settings involving random seeds, temperature, and max length, can Best-of-N-Entropy and DEGS-MCMC still achieve stable improvements?

Is sequence-level collapse depth related to response length? Are there differences in collapse depth among responses of different lengths?

9 comments
AuthorsOpenReview ↗

Official Response to Reviewer tuHY (Part 1/7)

Thank you for the questions on novelty, reliability and generalization. We present the contribution and the corresponding experimental evidence below.

About the Contribution and Evaluated Models

All experiments in the paper are conducted on the Qwen2.5-7B architecture, so it cannot be ensured that the findings of this work are generalizable.

Novelty and reliability. Best-of-N-Entropy selects complete responses by final-layer entropy; DEGS-MCMC adds normalized earliest entropy-crossing depth to a likelihood-guided sequence target, π(x)∝p(x)αexp⁡(βD(x))\pi(x)\propto p(x)^\alpha\exp(\beta D(x)). Both use the existing model without parameter updates or a trained verifier. The auxiliary DEGS reranker is a separate control.

The distinction from LED (Tan et al., ICML 2026) is the statistic and its use: LED constructs token-level exploration distributions from intermediate posteriors, whereas DEGS scores complete sequences by first-crossing depth and incorporates this score into MH acceptance. This completion-level statistic and sequence-sampling objective define the methodological contribution. Figure 2 quantifies the signal's reliability, and Table 2 tests its combined use with likelihood within each decoding mode.

Evaluated models. Table 1 includes Qwen2.5-7B, Qwen2.5-Math-7B and DeepSeek-Math-7B. Appendix E specifies 28-layer Qwen and 30-layer DeepSeek probing, with grids {4,8,12,16,20,24,28} and {4,9,14,18,22,26,30}. Depths are normalized by the number of layers. The submitted DeepSeek results are:

DeepSeek-Math-7B, submitted Table 1Likelihood Best-of-N %Best-of-N-Entropy %Likelihood MCMC %DEGS-MCMC %
MATH50042.042.342.447.5
DeepMind50027.629.830.733.6
HumanEval Pass@143.345.347.050.8
GPQA33.834.134.539.2

Both proposed decoders improve over their corresponding likelihood baselines across these four tasks. Parts 2 and 7 add the parameter-scale and Qwen3 comparisons, respectively.

AuthorsOpenReview ↗

Official Response to Reviewer tuHY (Part 2/7)

About the Architecture and Parameter Scale

Are collapse depths correlated with response quality on architectures such as LLaMA3, Qwen3, and Qwen3.5, as well as at other parameter scales? Do Best-of-N-Entropy and DEGS-MCMC remain effective?

We add a comparison of Qwen2.5-Math-1.5B and 7B using the same first 100 MATH500 problems, generation seeds 0/1/2 and first 16 candidates per problem/seed (4,800/model). We use T=.25, top-p=1, disabled top-k, max 768 tokens, alpha=4, beta=5, tau=.25 nats, probes {4,8,12,16,20,24,28} and final RMSNorm.

ModelSelectorAccuracy; contrast vs same-model likelihood
1.5BLikelihood60.00%; reference
1.5BFinal entropy75.00%; +15.00 pp
1.5BDEGS reranking62.00%; +2.00 pp
1.5BMajority voting72.00%; +12.00 pp
7BLikelihood67.00%; reference
7BFinal entropy82.00%; +15.00 pp
7BDEGS reranking68.00%; +1.00 pp
7BMajority voting79.00%; +12.00 pp

Final entropy minimizes mean predictive entropy; DEGS reranking maximizes 4 log p+5D with post-token depth. The prompt, parser and grader are shared. Majority voting groups normalized extracted answers and breaks ties by earliest candidate; invalid groups are excluded, and all-invalid pools count incorrect. Accuracy averages all problems and generation seeds.

Best-of-N-Entropy improves on likelihood selection by 15.00 pp and majority voting by 3.00 pp at both scales. This comparison evaluates fixed-pool selection at 1.5B/7B. Part 7 evaluates both proposed decoders on Qwen3-8B-Base; the submitted Table 1 provides the DeepSeek comparison. Together, these comparisons cover both parameter scales and the additional Qwen3 and DeepSeek checkpoints.

AuthorsOpenReview ↗

Official Response to Reviewer tuHY (Part 3/7)

About the Discrimination and Reasoning Behavior

what are its AUC, Spearman correlation, and pairwise-ranking accuracy?

Per-candidate discrimination. Figure 2 reports all three metrics for collapse depth on Qwen2.5-Math-7B. Each dataset contains 500 problems ×16 responses at T=.8/top-p=.9:

Submitted Fig. 2AUCSpearman rho(D,correct)Pair-ranking accuracy
MATH500.595.150.571
DeepMind500.612.146.592

The lowest/highest-D buckets have accuracies of 17%/41% on MATH500 and 11%/29% on DeepMind500. D complements likelihood, and Tables 1–2 evaluate their combined use. The combined score S(x)=αlog⁡p(x)+βD(x)S(x)=\alpha\log p(x)+\beta D(x) has the same ranking as π(x)\pi(x). The joint-matching analysis in our response to Reviewer fjAd compares this combined score directly with likelihood and likelihood plus final entropy; for EOS-ended, jointly matched pairs, its ranking accuracy is 65.92%, versus 51.93% for likelihood.

Scale comparison. For the first-100, three-seed, N=16 pools in Part 2, we compare correct and incorrect candidates within the same problem across seeds, giving ties half credit and weighting pairs equally:

ModelD AUC / within-problem pair accuracy
Qwen2.5-Math-1.5B0.649/0.650
Qwen2.5-Math-7B0.650/0.649

The longer-horizon random-100 configuration in Part 5 gives:

T / capAUC / rho(D,correct) / pair accuracy / rho(D,length)
.25/30720.600/0.171/0.600/0.100

Pooled AUC and within-problem pair accuracy measure complementary aspects of discrimination. The latter compares candidates for the same question; the joint-matching analysis additionally controls likelihood, length and final-layer entropy.

Reasoning behavior. D summarizes the evolution of internal confidence across layers through the average normalized first-crossing depth. Figure 3 illustrates the entropy trajectories motivating this statistic, while Part 6 examines response length explicitly. Table 7 also links probe placement to downstream accuracy: later-layer probes yield 76.6%/42.6% on MATH500/GPQA, compared with 72.8%/38.5% for the earliest probes in the four-probe comparison.

Reviewer_fjAdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Depth-Entropy Guided Sampling (DEGS), a training-free decoding method that augments sequence likelihood with an internal entropy-based score. The authors define collapse depth as the earliest probed layer at which logit-lens entropy falls below a threshold, aggregate it across generated tokens, and incorporate the resulting score into a Metropolis–Hastings power-sampling objective. Experiments across three base models and four benchmarks show consistently positive improvements over the corresponding likelihood-only MCMC baseline. The idea of exploiting intermediate-layer uncertainty is interesting. However, the compute accounting, sampler-matched ablations, and evidence supporting the proposed mechanism require clarification before the main conclusions are sufficiently established.

Critical strengths
  1. A simple and testable internal scoring signal. Collapse depth is explicitly defined and can be computed without training an auxiliary verifier or updating model parameters.
  2. Consistent within-framework improvements. Comparing positive depth weight against β=0\beta=0 within the same MCMC framework is a useful experimental design. The positive accuracy differences across all twelve model–benchmark combinations warrant further investigation.
  3. Appropriately qualified predictive analysis. The paper acknowledges that collapse depth is only a weak standalone correctness predictor, with AUC around 0.6, rather than overstating its discriminative strength.
  4. Evaluation beyond mathematical reasoning. HumanEval and GPQA broaden the empirical scope. Reporting the case where low-temperature decoding outperforms DEGS also helps characterize the method’s limitations.
Critical weaknesses

Weaknesses

  1. The compute accounting does not clearly match the described implementation. Section 5.3 and Table 6 charge layer-subset probing as approximately K/LK/L of a full forward pass. However, Table 4 and Appendix E describe a separate teacher-forced forward with hidden-state hooks. Reading only selected layers does not eliminate the Transformer computation needed to produce those states, and each probed layer additionally requires vocabulary projection and entropy computation. Proposition 5 likewise states that a separate scoring pass costs Cfwd+KclensC_{\mathrm{fwd}}+Kc_{\mathrm{lens}}.

  2. The key ablations confound the scoring signal with the sampling mechanism. Table 2 compares final-layer entropy under one-shot reranking with collapse depth under MCMC. This changes both the signal and the search procedure, so it does not isolate the advantage of depth-wise information over output-layer entropy. Similarly, depth-only reranking versus the combined MCMC objective does not cleanly establish the necessity of both score components.

  3. The predictive analysis does not establish the signal’s incremental value in the relevant candidate regime. Figure 2 uses candidates sampled at temperature 0.8 with top-p 0.9, whereas the main MCMC experiments use substantially lower-temperature, untruncated proposals. The higher-temperature analysis is reasonable for obtaining varied candidates, but the operative claim concerns whether collapse depth distinguishes correctness among the near-equal-likelihood candidates actually encountered during decoding.

  4. The interpretation of trajectory-level “compounding” exceeds the current evidence. Proposition 8 establishes a bounded perturbation of pairwise scores, but does not show that repeatedly applying a weak signal necessarily improves correctness. For the fixed target distribution, the relative tilt remains proportional to exp⁡(βD(x))\exp(\beta D(x)); it does not automatically strengthen with the number of MH steps.

Questions for authors
  1. Is entropy probing fused with likelihood evaluation or performed through a separate full forward? Can you provide corrected compute accounting and comparisons at equal wall-clock or FLOP budgets?
  2. Can you evaluate final-layer-entropy guidance within the identical MCMC sampler to isolate the contribution of depth-wise information?
  3. Does collapse depth provide incremental correctness information within problems and near-equal-likelihood candidate groups after controlling for length and final-layer entropy? Does this hold on actual MCMC proposals?
  4. Can you report performance versus MH steps and analyze acceptance decisions changed specifically by the depth term? How does the convergence claim account for blockwise execution and EOS stopping?
  5. Can you clarify Table 7’s decoding method, the GRPO training data, and hyperparameter selection, and provide repeated-seed results with paired confidence intervals?
8 comments
AuthorsOpenReview ↗

Official Response to Reviewer fjAd (Part 1/7)

Thank you for the detailed review and questions on computational cost, matched comparisons and the depth signal. We respond to the specific topics below.

About the Probing Cost and Compute Budgets

Is entropy probing fused with likelihood evaluation or performed through a separate full forward? Can you provide corrected compute accounting and comparisons at equal wall-clock or FLOP budgets?

Table 4/Appendix E describe a separate teacher-forced scoring pass with hidden-state hooks, final RMSNorm and vocabulary-chunked entropy. Its cost is Cfwd(n)+Kclens(n)C_{\mathrm{fwd}}(n)+Kc_{\mathrm{lens}}(n) (Proposition 5). Selecting layers reduces projection and entropy work; the backbone forward is still required. This expression accounts for the complete scoring pass. Known positions are processed together, and the implementation caches the current score and reuses an accepted proposal's score.

On Qwen2.5-Math-7B/MATH500, Table 6 reports 228.1/230.5/235.6 s for likelihood/final-entropy/depth reranking and 330.5/359.5 s for likelihood/depth MCMC. Entropy Best-of-N gains +0.3 pp (68.4→68.7) for +1.1% time, and DEGS-MCMC gains +1.9 pp (74.8→76.7) for +8.8% time (Tables 2/6). The cross-decoder ratio is 1.576 times likelihood Best-of-16, so we also compare methods under matched decision deadlines.

E7 protocol and results. We use Qwen2.5-Math-7B, the first 100 MATH500 problems in file order, seeds 0/1/2, T=.25, top-p=1, no top-k and a 768-token cap. DEGS uses four 192-token blocks, ten MH steps/block, α=4, β=5 and τ=.25. On the same H800, each problem-seed's DEGS latency defines the likelihood/majority decision deadline. After common warm-up, timing includes tokenization, generation, required scoring, parsing and selection; loading, warm-up and ground-truth grading are excluded. Only fully processed candidates completed by the deadline count. Greedy is timed separately.

MethodAccuracy %Δ vs DEGS, pp
DEGS-MCMC82.00Reference
Likelihood by deadline67.00-15.00
Majority by deadline79.00-3.00
Greedy68.00-14.00

The selectors share a candidate pool, so its cost is counted once. Overrun candidates are excluded from selection, with their work accounted for separately. The reported budget is the time to an available decision, with overrun work accounted for separately. E7 adds timing around E1's ten-step DEGS outputs, hence the shared 82.00% accuracy. DEGS exceeds likelihood and majority by 15.00 and 3.00 pp, respectively, under this protocol.

AuthorsOpenReview ↗

Official Response to Reviewer fjAd (Part 2/7)

About the Ablations Within the Same Sampler

Can you evaluate final-layer-entropy guidance within the identical MCMC sampler to isolate the contribution of depth-wise information?

Table 2 compares score components within the same selection procedure. Its submitted Qwen2.5-Math-7B results are:

Score / modeMATH500 %DeepMind500 %GPQA %
Likelihood / rerank68.446.534.3
Depth only / rerank63.241.432.8
Final entropy / rerank68.747.936.9
Likelihood + depth / rerank71.654.836.2
Likelihood / MCMC74.857.438.9
Likelihood + depth / MCMC76.761.942.6

Within reranking, adding likelihood to depth-only gains 8.4/13.4/3.4 pp, and adding depth to likelihood gains 3.2/8.3/1.9 pp. We additionally evaluate your requested final-entropy control within the identical MCMC sampler.

E1 protocol and results. We use Qwen2.5-Math-7B, the first 100 MATH500 problems, seeds 0/1/2, T=.25, top-p=1/no top-k, four 192-token blocks and a 768-token cap. Arms share the prompt, parser/grader, fixed-horizon suffix proposal, EOS convention and final-chain-state selection. Budgets of 1/5/10 steps per block are separate runs.

The scores are S_L(x)=4 log p(x), S_D(x)=4 log p(x)+5D(x) and S_F(x)=4 log p(x)+5F(x), where F(x)=−mean(H_final)/log|V|. L/D/F label the scoring arms. D uses τ=.25 nats, probes {4,8,12,16,20,24,28}, final RMSNorm and generated non-EOS tokens. D and F use the same post-token positions; likelihood sums completion-token log probabilities without length normalization. Depth-only uses 5D (α=0). F-guided MCMC is distinct from one-shot Best-of-N-Entropy. The score weights are held fixed across these sampler-matched comparisons.

Steps/blockScoreAccuracy %Δ vs L, pp
1L63.00Reference
1D76.00+13.00
1F64.00+1.00
5L62.00Reference
5D80.00+18.00
5F64.00+2.00
10L63.00Reference
10D82.00+19.00
10F65.00+2.00
10Depth only61.00-2.00

Accuracy averages all 100 problems and three seeds; deltas are computed before display rounding. With the sampler held fixed, depth guidance outperforms likelihood and final-entropy guidance at every tested budget. At ten steps, the combined score also outperforms depth-only. Collapse depth records when uncertainty falls across layers: trajectories with the same final entropy can cross the threshold at different depths, giving a distinct statistic for this comparison.

AuthorsOpenReview ↗

Official Response to Reviewer fjAd (Part 3/7)

About the Incremental Predictive Information

Does collapse depth provide incremental correctness information within problems and near-equal-likelihood candidate groups after controlling for length and final-layer entropy? Does this hold on actual MCMC proposals?

Figure 2 reports association in diverse T=.8/top-p=.9 pools: MATH500 AUC/rho/pair accuracy .595/.150/.571; DeepMind500 .612/.146/.592. To address the candidate regime used in decoding, we analyze actual low-temperature proposals separately.

E3 protocol and results. We analyze accepted and rejected proposals from E1's ten-step DEGS trajectories: Qwen2.5-Math-7B, the first 100 MATH500 problems, seeds 0/1/2, T=.25, α=4, β=5, τ=.25 and four 192-token blocks/cap 768. EOS-ended and final-cap proposals are analyzed separately; unfinished prefixes are excluded, and missing parsed answers count incorrect.

Five-fold GroupKFold holds out entire problems. Fixed L2 logistic regression (C=1), with training-fold standardization, compares [summed log-likelihood, log-length, post-token final entropy] against the same controls plus D. This evaluates prediction on held-out questions. The regression is a diagnostic; DEGS itself directly uses likelihood and depth during decoding.

CohortD AUCOOF AUC controls→+DΔAUCΔlog loss ↓
EOS0.6500.699→0.750+0.051-0.049
Capped0.6000.650→0.700+0.050-0.032

The corresponding OOF loss levels are 0.449→0.400 (EOS) and 0.251→0.220 (cap); deltas use values before display rounding.

We also enumerate same-problem correct/incorrect pairs across seeds, with half credit for D ties and equal pair weight. Near-likelihood means |Δlog p|≤1.25 nats. Part 4 adds simultaneous length and entropy matching.

Pair cohortD ranking accuracy
EOS, all0.651
EOS, near likelihood0.651
Capped, all0.600
Capped, near likelihood0.600

Adding D improves held-out AUC and log loss after the stated controls. The pair analysis provides complementary within-problem evidence on actual MCMC proposals. These are predictive associations; the jointly matched analysis below examines the deployed combined score more directly.

Reviewer_SR3mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper is based on the empirical observation that “depth entropy collapse” (weakly) correlates with LLM accuracy. Specifically, for a given generated sentence x, depth entropy collapse (D(x)) is defined as the first layer in which the model’s entropy is lower than a given threshold, averaged over all the tokens of x. Exploiting this correlation between D(x) and accuracy, the authors propose to modify the power distribution proposed in (Karan and Du, 2025) including D(x) as a depth entropy term. Finally, the modified power distribution is used in a Metropolis–Hastings sampling algorithm.

Critical strengths

The empirical correlation between depth entropy collapse and accuracy is interesting.

The experimental results are good.

Critical weaknesses
  1. The presentation of the paper is not fully self-contained, as several terms and concepts are left undefined or are used before they are defined. This makes it difficult to understand the proposed approach. The following is a non-exhaustive list of examples.
  • Not all readers are familiar with Metropolis–Hastings sampling. Providing the standard algorithm (at least in the appendix) would help a lot.
  • How q() is defined in Eq. 6?
  • Why you divide the sequence into blocks of size B (lines 257-258)?
  • What are the “twelve cells” (lines 81-82)?
  • The unembedding matrix U (line 132) is the same used in the Language Modeling head?
  • What is N in line 227?
  • What is “n=8000 candidate pool” in line 321?
  • In Tab. 1, it would be preferable to use bold text for the best (training-free) models (and, for instance, underlining for the second-best ones), since using bold text for one's own proposed models creates the impression that they always achieve the best results.
  1. Although the experimental results shown in Tab. 1 are good, the experimental protocol has some weaknesses:
  • In line 269 you mention that you also used “DeepMind500, the latter a held-out subset we construct from the DeepMind”. However, the use of a benchmark that is not publicly available makes it difficult to reproduce your results.
  • In Tab. 1, you should include Majority Voting (Wang et al., “Self-consistency improves chain of thought reasoning in language models”, ICLR, 2023), a simple and widely used training-free method which can help determine whether, given the same number of forward passes, your proposal can improve upon simply selecting the majority-voted answer.
  • In lines 443-446 you state that your method introduces a small computational overhead compared to other methods based on MCMC sampling. However, Tab. 1 also includes comparisons with methods that do not use sampling, such as base decoding. In Tab. 6 (Appendix E), the wall-clock time for base decoding is not reported. Consequently, it is unclear whether the proposed method is effective when compared on the basis of equal execution time. For instance, the timings in Tab. 6 show that DEGS-MCMC is 1.57 times slower than “Best-of-N (log-likelihood)”. Therefore, a fairer comparison would likely involve Best-of-N (log-likelihood) using a larger value for N (e.g., N = 16 × 1.57 = 25).
  • In Tab. 8 you report the grid search range you used for selecting the critical threshold \tau, but I cannot find the accuracy values corresponding to using these different values.

Minor points:

  1. The statement “DEGS is the first method to use the depth-wise evolution of internal uncertainty—rather than output-layer signals alone—as a test-time selection criterion for reasoning” (lines 89-90) is not completely true. For instance, layer-based entropy collapse is exploited in this training-free decoding method:
  • Tan et al., “Restoring Exploration after Post-Training: Latent Exploration Decoding for Large Reasoning Models”, ICML, 2026.
  1. There are some typos. E.g., posttrained -> post-trained.
Questions for authors

Please, see the Critical Weaknesses.

8 comments
AuthorsOpenReview ↗

Official Response to Reviewer SR3m (Part 1/7)

Thank you for the careful reading and suggestions on presentation and experimental comparisons. We respond to the specific points below.

About the MH Sampler and Proposal Distribution

How q() is defined in Eq. 6?

  • Standard Metropolis–Hastings. Starting from x, propose x′∼q(⋅∣x)x'\sim q(\cdot\mid x) and accept with

A(x′,x)=min⁡{1,π(x′)q(x∣x′)/[π(x)q(x′∣x)]}.A(x',x)=\min\{1,\pi(x')q(x\mid x')/[\pi(x)q(x'\mid x)]\}.

Otherwise retain x. The unknown target normalizer cancels. DEGS uses π(x)∝p(x)αexp⁡(βD(x))\pi(x)\propto p(x)^\alpha\exp(\beta D(x)); beta=0 recovers likelihood-only power sampling. For each generated non-EOS token, d is the first probed layer whose vocabulary entropy is at most tau, or L if none crosses. D averages d/L over those tokens. Appendix E specifies tau=.25 nats, final RMSNorm and exclusion of prompt/EOS positions.

  • q in Eq. 6. Section 4.4 uses random-cut suffix resampling. Choose an editable completion position, retain its preceding prefix and regenerate the suffix from

rT(v∣h)=p(v∣h)1/T/∑wp(w∣h)1/T.r_T(v\mid h)=p(v\mid h)^{1/T}/\sum_w p(w\mid h)^{1/T}.

Let Qi(y∣x)Q_i(y\mid x) denote suffix generation conditional on cut i, including a prefix-match indicator, and wi(x)w_i(x) its cut probability. The marginal proposal is q(y∣x)=∑iwi(x)Qi(y∣x)q(y\mid x)=\sum_i w_i(x)Q_i(y\mid x); reverse probabilities use the same construction. In a cut-conditional implementation with uniform cuts, the reverse/forward cut factor is ℓ(x)/ℓ(y)\ell(x)/\ell(y) when EOS changes editable length. Here ell includes generated EOS, whereas D excludes it. D changes the acceptance target; the temperature proposal remains the base-model proposal. As Proposition 2 explains, T=1/alpha generally does not sample the sequence distribution proportional to p(x)^alpha, because token-level normalizers depend on the prefix. MH corrects this proposal–target difference.

  • Why blocks? Blocks initialize and refine a growing completion before extending its horizon. This schedule lets the sampler begin refinement before generating the entire response. Only the prompt is fixed: later cuts may revisit earlier generated blocks (Algorithm 1). The default is sixteen 192-token blocks and ten MH steps/block. Here, b=192 denotes block length and J=16 denotes block count, distinguishing the two quantities used in Section 4.4 and Tables 4/8.
AuthorsOpenReview ↗

Official Response to Reviewer SR3m (Part 2/7)

About the Definitions and Presentation

What are the “twelve cells” (lines 81-82)?

  • “Twelve cells.” Table 1 contains three checkpoints × four benchmarks: Qwen2.5-7B, Qwen2.5-Math-7B and DeepSeek-Math-7B on MATH500, DeepMind500, HumanEval and GPQA.

  • Unembedding U. Yes, this is the LM-head unembedding. Appendix E applies the model's final RMSNorm before vocabulary projection; no separately trained head is introduced.

  • N at line 227. N is the candidate budget, set to 16 in Section 4.2 and Algorithm 2. This candidate budget is distinct from the MH-step count and sequence length.

  • “n=8000 candidate pool.” Section 5.1 uses 500 problems × 16 responses, separately for each diagnostic dataset. T=.8/top-p=.9 provides varied candidates for this analysis. The additional proposal-level analysis in our response to fjAd examines the lower-temperature, untruncated MCMC regime directly.

  • Bold/underlined entries. We agree with highlighting the best training-free result in bold and the second-best with underlining, with GRPO identified separately. This makes the per-setting ranking explicit.

The two proposed decoders use different uncertainty statistics: Best-of-N-Entropy uses final-layer entropy for fixed-pool selection; DEGS-MCMC uses likelihood plus collapse depth in sampling. The DEGS reranker is an additional control. This distinction is relevant to the majority-voting comparison below.

AuthorsOpenReview ↗

Official Response to Reviewer SR3m (Part 3/7)

About the Reproducibility and Majority Voting

In Tab. 1, you should include Majority Voting

  • DeepMind500. Section 5/Table 5 identifies a 500-problem held-out subset of the public DeepMind mathematics dataset. Appendix E specifies the zero-shot CoT prompt, last-boxed-answer extraction and symbolic/numeric grader shared with MATH500. We will publicly release the exact evaluation subset and evaluation instructions so that the reported comparisons can be reproduced.

  • Majority voting and larger N. We evaluated Qwen2.5-Math-7B on the first 100 MATH500 problems in file order, with generation seeds 0/1/2, T=.25, top-p=1, no top-k and max 768 new tokens. Each problem/seed receives 25 candidates; N=16 uses the same pool's prefix. All selectors share the prompt, parser and grader. Likelihood sums base log probabilities; Best-of-N-Entropy minimizes mean predictive final-layer entropy; DEGS reranking maximizes 4 log p+5D with tau=.25, probes {4,8,12,16,20,24,28}, final RMSNorm and post-token D. Majority voting groups normalized answers, excludes invalid groups and breaks ties by earliest candidate; all-invalid pools count incorrect.

MethodNAccuracy; Δ vs likelihood N=16
Greedy168.00%; +1.00 pp
Likelihood1667.00%; Reference
Likelihood2568.00%; +1.00 pp
Majority1679.00%; +12.00 pp
Majority2582.00%; +15.00 pp
Final entropy1682.00%; +15.00 pp
DEGS reranking1668.00%; +1.00 pp
Final entropy2583.00%; +16.00 pp
DEGS reranking2569.00%; +2.00 pp

Accuracy averages all 100 problems and three generation seeds; deltas are computed before display rounding. Best-of-N-Entropy exceeds majority voting by 3.00 pp at N=16 and 1.00 pp at N=25. These comparisons isolate the selection rule on shared candidates. DEGS-MCMC is evaluated separately under matched decision deadlines in Part 4.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their detailed feedback. Our contribution is a practical use of model uncertainty for training-free reasoning. Best-of-N-Entropy selects complete responses using final-layer entropy; DEGS-MCMC combines normalized first-crossing depth with sequence likelihood in MH sampling. Both use the existing model without parameter updates or a trained verifier. The DEGS reranker is an additional control.

Table 1 evaluates three checkpoints on MATH500, DeepMind500, HumanEval and GPQA. Both proposed decoders improve over their corresponding likelihood-only baselines in all twelve settings. DEGS-MCMC exceeds the listed Power-SMC baseline in all twelve and gives the best listed training-free accuracy in eleven. Table 2 compares score components within each decoder, while Figure 2 reports the depth signal's discrimination metrics.

The responses below add majority-voting and decision-deadline comparisons, final-entropy guidance within the same MCMC sampler, prediction on actual proposals, joint matching for likelihood/length/final entropy, and temperature, length and model comparisons. We explain the full-forward-plus-lens cost of separate entropy scoring, the fixed-horizon MH argument, and the specific sequence-level statistic and sampling objective that define the contribution.

We will publicly release the exact DeepMind500 evaluation subset and evaluation instructions, and incorporate the notation, presentation and related-work clarifications into the revision.

AuthorsOpenReview ↗

Note to the Area Chair

We summarize the evidence addressing the reviewers' main concerns.

Table 1 evaluates three checkpoints on four benchmarks. Both proposed decoders improve over their respective likelihood-only baselines in all twelve settings; DEGS-MCMC exceeds the listed Power-SMC baseline in all twelve and achieves the best listed training-free accuracy in eleven. Table 2 compares score components within the same decoding modes, Table 6 reports measured computational costs, and Figure 2 gives AUC, Spearman correlation and pair-ranking accuracy. The submitted experiments include DeepSeek-Math-7B as well as the two Qwen checkpoints.

The rebuttal adds majority-voting and decision-deadline comparisons, final-entropy guidance within the same MCMC sampler, and conditional ranking analyses on actual proposals. Temperature/length, parameter-scale and Qwen3 comparisons further address robustness and transfer. The detailed protocols and results are provided in the reviewer-specific responses.

We explain the full-backbone-plus-lens scoring cost and the fixed-horizon MH argument, and define the contribution as the completion-level first-crossing statistic and its likelihood-coupled sampling objective. The reviewer-specific replies provide the corresponding notation, computational accounting and related-work explanations.

We appreciate the reviewers' feedback and the Area Chair's consideration of the submitted evidence and the additional results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.