acceptodds
Under review as a conference paper at ICLR 2027

Truncation Samplers Are Nested: Comparing Decoding Rules Is a Measurement Problem

17 views

Abstract

Settling one comparison between two truncation samplers has cost six thousand A100-hours, and nothing says in advance how far apart two samplers can be. We observe that almost every deployed truncation rule keeps a prefix of the probability-sorted vocabulary, so any two such rules are nested at every decoding step. Nesting yields a closed-form per-step distance and a coupling under which both samplers emit identical text until they first disagree. Accumulated over a generation, this gives a separation certificate that bounds the difference between two samplers on any bounded outcome, is computable from logs a comparison already produces, and equals their sequence-level total variation distance exactly when neither rule ever overtakes the other. Two consequences follow without measurement: exactness is fragile, and certifying a small difference requires the two retained masses to agree to very high precision at every step. On six open-weight models over four math and science benchmarks, the certificate for every hyperparameter grid we tested is at least , so a bound alone rarely settles a comparison, although two members of one family can be far closer. What remains useful is the coupling, which gives an unbiased paired estimator that cuts variance – against prompt-level pairing. Read against benchmark sizes at the default pair's variance and our point estimate of its between-prompt component, resolving a one-point difference needs to samples per prompt on GSM8K, MATH-500 and GPQA-Diamond and about on AIME 2025; at that component's upper confidence bound GPQA-Diamond needs more than and AIME 2025 cannot be resolved at any sample count. Choosing a truncation sampler is a measurement problem at least as much as a design one.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
truncation sampling, separation certificate, coupling, total variation distance, evaluation methodology, statistical power

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ygCWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares nested truncation samplers. It gives a separation bound and a paired estimator.

Critical strengths

GSM8K tests show variance reductions of 1.3–3.8 times against prompt-level pairing. This improves measurement efficiency.

Critical weaknesses

Certificate and outcome runs use different stopping rules. This limits their connection. Sample requirements for other benchmarks rely on GSM8K estimates. These requirements need further validation.

Questions for authors

Do results hold with the same stopping rule? Does the sample analysis target fixed benchmarks or new prompts?

6 comments

We thank the reviewer for finding the paper clear and easy to follow, and for highlighting the 1.3–3.8× variance reductions and the gain in measurement efficiency.

Below we quote each concern and add new evidence. Both questions led to new results.

"Certificate and outcome runs use different stopping rules" (Q1: "Do results hold with the same stopping rule?"): We reran with the chat template (thinking disabled), with every arm and the certificate stopping at the same EOS; 98% of generations end before the cap. For the default pair, the results hold:

Default pair, chat format, shared EOS stoppingValue
Variance reduction5.4× [3.8, 8.3] (raw completion: 4.7×)
Certificate0.17 (raw completion: 0.72); bounds every measured gap
Prop 7 guarantee vs measured≥2.1× guaranteed, 12.3× measured

Most of the certificate's drop comes from the chat format rather than from stopping: it is already 0.06 at T=64, before any generation stops. The two pairs whose certificates stay above 1/2 show smaller, non-significant reductions (1.19×, 1.25×); the theory guarantees nothing there. This closes the gap between the certificate and the outcome runs: with one stopping rule for both, the certificate bounds every measured gap, and Prop 7's variance guarantee holds on a real cross-family pair.

"Sample requirements for other benchmarks rely on GSM8K estimates": the paper states this. Sec. 5 notes that "this extrapolation assumes the τ² measured on GSM8K", and Table 3 is captioned "(default pair, GSM8K variance decomposition)". We now use each benchmark's own variance. For fixed benchmarks the conclusions hold:

  • GSM8K: k=4.6.
  • MATH-500: k=23, within the paper's "by 32".
  • AIME 2025: k≈305, against the GSM8K-based ≈210.

GPQA-Diamond's estimate rests on 5 nonzero outcomes, and we say so. This validates the sample-size conclusions on each benchmark's own data: the larger benchmarks are resolved with tens of samples per prompt, and AIME 2025 needs hundreds.

Q2 ("Does the sample analysis target fixed benchmarks or new prompts?"): new prompts. We thank the reviewer for this question, it sharpens the paper. For a fixed benchmark, τ² drops out and k = Cσ_w²/N. For new prompts, the upper bounds from 100 and 30 prompts leave MATH-500 and AIME unresolved, and the camera-ready version will state this. This makes the sizing directly usable: for a fixed benchmark, every benchmark we study has a finite number of samples per prompt that resolves a one-point difference.

As we have addressed both weaknesses and both questions, and the new results strengthen our claims, we kindly ask the reviewer to consider an updated assessment.

Dear Reviewer ygCW,

Thank you again for your review and your two questions. As the discussion period closes on November 18, we wanted to check whether our response answered them.

In brief:

  • In the chat format, with one stopping rule for both the certificate and the outcomes, the default pair keeps a 5.4× variance reduction, and its certificate (0.17) bounds every measured gap.
  • The sample analysis targets new prompts. For fixed benchmarks, we now give each benchmark's requirement from its own variance.

We would be glad to clarify anything that remains. If our response resolves your concerns, we would be grateful if you would consider updating your assessment.

Dear Reviewer ygCW,

The discussion period closes on November 18 AoE, so we would like to ask you directly to check our rebuttal. Both of your questions now have direct answers:

  • You asked whether the results hold with the same stopping rule. We ran exactly that experiment. With the certificate and the outcomes stopping at the same EOS, the default pair keeps a 5.4× variance reduction. Its certificate (0.17) bounds every measured gap, and Prop 7's guarantee holds (at least 2.1× guaranteed, 12.3× measured).
  • You asked whether the sample analysis targets fixed benchmarks or new prompts. New prompts. For fixed benchmarks we now give the requirement directly, k = Cσ_w²/N.
  • You noted that the requirements relied on GSM8K estimates. Each benchmark now uses its own variance: GSM8K 4.6, MATH-500 23 and AIME 2025 about 305 samples per prompt. The paper's conclusions hold.

Could you let us know before the deadline whether this answers your questions, or what remains? We will respond right away. If the new results resolve them, we ask that your assessment reflect this.

Reviewer_x5vtOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2(+1)
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper focuses on a relatively interesting problem i.e. the variance of the estimated quality difference between a pair of two samplers (such as top k, min p, etc). The paper proposes a simple solution by using one sampler’s random draw first. It then checks whether that sampled token is also allowed by the other sampler; thereby reducing the variation. This gives an exact one-step total-variation distance and a coupling that keeps both generations identical until their first disagreement. Repeating the coupling across a generation yields an upper bound on differences in any bounded outcome and a paired estimator of sampler performance. The paper reports that broad sampler grids usually have nearly maximal bounds, while the coupling reduces estimator variance and can lower the number of generations required to compare two samplers.

Critical strengths
  1. The proposed method for variance-reduced estimation of a pair of two samplers is inexpensive and straightforward to implement, and seems to be backed by theory.

  2. The paper tests its formal identities extensively on synthetic models and separates several descriptive results from inferential claims.

Critical weaknesses
  1. Beyond reporting/comparison, I don't see a practical utility offered by this paper. The whole point of standard benchmarking in Language modeling is to see how an LLM performs when used by practitioners. Enforcing some sort of coupled runs reduces the variance between the scores but that is not what/how the practitioners use in real world. While the problem of high variance in individual samplers is real and significant, this proposed method is not solving that core issue of variance of an individual sampler. Instead it is relying on shared sampling and claiming that that reduces the variance, which is of course, tautological (in fact, most of the theoretical results are tautological); more importantly, such coupled sampling is not the truly representative of how the individual sampler performance. Further, how does the algorithm work if there are more than two samplers; the current simplistic pairing doesn't work.

  2. The paper reports estimator variance reduction but no compute-normalized comparison. The coupled decoder runs at batch size one, performs additional probability and cache operations, and generates two continuations after divergence. Since the reported separation probabilities are nearly one, most pairs diverge and share limited computation. The relevant quantity is GPU time required to reach a fixed confidence interval, not variance per sample. This comparison is absent, and the proposed repeated-sampling requirements may be more expensive than ordinary evaluation.

The final manuscript retains layers of unnecessary terminology around a simple coupling argument. The same probability is variously called a “separation certificate,” “divergence probability,” “grid certificate,” “pairwise certificate,” and “instrument,” while related analyses receive labels such as “exactness is fragile,” “predictive validity,” “gate controls,” “estimator, hardened,” and “dispersion lower bound bracket.” Phrases such as “exactness and fairness pull against each other” replace precise mathematical statements with rhetorical language. The supplementary code also contains internal drafting references such as “OpenReview LLM feedback” and dated review notes. These are not isolated stylistic preferences. The terminology repeatedly forces the reader to translate paper-specific labels back into standard concepts such as maximal coupling, total variation, paired estimation, and statistical power. Substantial author-led rewriting is needed to state the contribution in established language and separate the standard results from the genuinely new truncation-specific observation.

Questions for authors
  1. The paired estimator assumes that each sampler generates exactly T new tokens. If the samplers disagree, the released code instead stops when the combined prompt and completion reach length T. A three-token prompt with a four-token generation limit therefore produces between one and three new tokens after disagreement, while a pair that never disagrees produces four. I reproduced this behavior using the supplied implementation. Because completion length depends on when the samplers disagree, the two outputs do not follow the fixed-length sampler distributions assumed by Proposition 7. The reported variance reductions and accuracy differences must be regenerated after correcting this error.

  2. The paper evaluates 200 prompts with five random generations per prompt. These 1,000 outcomes are not independent because the five generations belonging to one prompt share the same question and expected difficulty. The paper describes its procedure as a prompt-cluster bootstrap, which should resample 200 prompts while keeping each prompt’s five generations together. The released code instead flattens the data and resamples 1,000 outcomes separately. This can make the confidence intervals too narrow and affects the significance claims in Table 2, Table 5, and Figure 4.

  3. All scored tasks are short-answer mathematics or science, although truncation samplers target open-ended generation; the configured MT-Bench experiment has no reported result. Why is no open-ended generation result reported when the paper states that truncation samplers were designed for that setting?

  4. The supplement promises result files and retained-mass logs but contains none, preventing verification of the real-model results against the released code. Am I missing something here?

6 comments

We thank the reviewer for finding the method inexpensive, simple to implement, and for noting the extensive synthetic verification and the separation of descriptive from inferential claims.

New result (prompt-level CIs)Value
Variance reduction, GSM8K default pair3.8× → 4.7× [3.8, 6.0]; all 3 pairs significant after Holm
Precision per GPU-hour, measured6.1×, 1.2×, 1.5× (GSM8K); 1.8× (MATH-500)

Q1 thank you, this is fixed now. Every arm now generates exactly T tokens, and exact enumeration confirms each follows its sampler's law. We regenerated every number: 1.2× and 1.5× on the other GSM8K pairs (were 1.3×), 1.6× on MATH-500 (was 1.7×). The central result survives the fix and is larger on the default pair.

Q2 corrected to resampling prompts. On the rerun, the predictiveness test the submission called inconclusive (p≈0.055) is significant (p=0.012). The evidence that the certificate tracks measured gaps is now stronger.

"not what/how the practitioners use … not the truly representative": each coupled arm is an exact sample from its own sampler (Lemma 2), so the estimator is unbiased for "the same estimand independent runs target" (Prop 7). On the real model, accuracy inside the coupled run shows no significant difference from accuracy run alone in 16/16 comparisons. With the same 1,000 samples, coupling resolves a 3.6-point top-k vs top-p gap (CI [−6.9, −0.4]) that prompt-level pairing misses ([−7.2, +0.6]). The method targets differences between samplers, not each sampler's own variance. Practitioners get the same answer with less noise.

"GPU time required to reach a fixed confidence interval": measured. Coupling needs 6.1× [4.9, 7.8] less GPU time than prompt-level pairing on the default pair, and 1.2–1.8× less on the others. A fork costs 2.9 ms (<0.1 forward pass). Batched, on the default pair, the two cost within 8% of each other at batch sizes 1–64 and coupling keeps 4.1× better precision per GPU-hour; for top-k vs top-p, the batched coupled run took 25% longer and gained nothing. On the default pair, the variance reduction is a 6.1× saving in GPU time.

"tautological": the short proofs follow from the nesting observation; Thm 3(ii), Prop 5 and Prop 8 need real arguments. The variance reduction is empirical: Prop 7 guarantees nothing above Sep_T=1/2, and there we observe both outcomes (significant 1.24× and 1.47× in raw completion; non-significant 1.19× and 1.25× under the chat format), while the default pair (certificate 0.17) gains 5.4×. A result that depends on the pair is not a tautology.

"how does the algorithm work if there are more than two samplers": Cor. 4 states that "All of G can be coupled simultaneously" (p. 4). The camera-ready version will add a construction giving every pair its own two-rule coupling (agreement 1 − Sep_T) at once, for any number of rules; exact enumeration verifies it (4 rules, nesting order changing on 54% of paths). The coupling and its bound extend to any number of samplers.

"layers of unnecessary terminology": the labels name different things, and most occur once. Standard terms are already used, e.g. Lemma 2's "standard maximal coupling for nested supports". We will use one name per quantity in the camera-ready version. The paper already speaks in standard terms; the remaining labels mark distinct quantities.

Q4 correct. But this does not affect the results, we will release them with the camera-ready version.

Q3 Sec. 7 lists this limitation. MT-Bench has no gold answer, so it gives certificates, not accuracy gaps: 0.86 at T=256 against 0.71 on GSM8K, and 0.65 under the chat template. Open-ended text separates samplers even more, reinforcing that a bound alone rarely settles a comparison.

As we have addressed each concern, and the new results support and strengthen the paper's claims, we kindly ask the reviewer to consider an updated assessment.

Dear Reviewer x5vt,

Thank you again for your careful review. As the discussion period closes on November 18, we wanted to check whether our response addressed your concerns.

In brief:

  • Both errors you identified are fixed, and every number has been regenerated. The default-pair variance reduction is now 4.7×.
  • Coupling needs 6.1× less GPU time than prompt-level pairing at a fixed CI.
  • Each sampler's accuracy inside the coupled run shows no significant difference from its accuracy run alone in 16/16 comparisons.

We would be glad to clarify anything that remains. If our response resolves your concerns, we would be grateful if you would consider updating your assessment.

Dear Reviewer x5vt,

The discussion period closes on November 18th AoE, so we would like to ask you directly to check our rebuttal. Each point you raised now has a direct answer:

  • You found the generation-length bug and asked us to regenerate the results. Fixed, and every number has been regenerated. The default-pair variance reduction is 4.7× (was 3.8×), and all three GSM8K pairs remain significant after Holm correction.
  • You asked for a prompt-level bootstrap. Done. The certificate-versus-gap test, inconclusive in the submission, is now significant (p=0.012).
  • You asked for the GPU time to reach a fixed CI. Measured: coupling needs 6.1× less GPU time on the default pair, and costs the same as pairing when batched.
  • You asked whether coupled runs represent each sampler. Each sampler's accuracy inside the coupled run shows no significant difference from its accuracy run alone in 16/16 comparisons.

These two errors were the basis of your soundness concern; both are fixed, and the conclusions hold. Could you let us know before the deadline whether this resolves your concerns, or which points remain? We will respond right away. If the new results resolve them, we ask that your assessment reflect this.

Reviewer_RZfxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to compare truncation sampling strategies for LLM decoding. The authors observe that most truncation samplers retain a prefix of the probability-sorted vocabulary, making their retained token sets nested. This property enables an exact characterization of per-step total variation distance and a sequence-level separation certificate that upper-bounds differences in bounded evaluation metrics. The authors further leverage this structure to construct a maximal coupling method for paired evaluation, which preserves each sampler's output distribution while encouraging shared generations. Experiments across several open-weight models and reasoning benchmarks show that the separation certificates are often too loose to meaningfully bound accuracy differences, but the coupled estimator reduces evaluation variance by approximately 1.2–3.8× in the main comparisons.

Critical strengths
  1. Interesting and important problem. Comparing decoding strategies reliably and with lower computational cost is an important evaluation problem for LLMs, particularly given the randomness inherent in text generation.

  2. Potentially useful evaluation methodology. The proposed coupling provides an unbiased estimator for comparing sampler performance and improves statistical efficiency. This approach may also offer useful insights for broader LLM evaluation beyond truncation samplers.

Critical weaknesses
  1. Limited practical utility of the theoretical bound. Although the separation certificate is theoretically valid, it is nearly vacuous in most experiments (often exceeding 0.99), while observed accuracy differences are much smaller. Consequently, the bound rarely provides useful guidance on whether two samplers perform similarly or whether further evaluation is necessary.

  2. Limited novelty of the coupling method. Variance reduction through common random numbers is well established (Yang & Nelson, 1991; Glasserman & Yao, 1992), as is maximal coupling (Lindvall, 2002). Moreover, the authors acknowledge that their coupling reduces to the accept-and-resample mechanism of speculative decoding (Leviathan et al., 2023) for nested distributions. The contribution therefore appears primarily to be a specialization of existing techniques, and its methodological novelty could be better justified.

  3. Lack of compute-normalized efficiency comparisons. The paper demonstrates variance reductions relative to traditional prompt-level pairing, but does not evaluate statistical power or estimation precision under equal computational budgets. Since coupled evaluation still generates two responses, lower variance per sample does not necessarily translate into lower GPU cost.

  4. Limited empirical coverage. Despite considering a grid of 39 sampler configurations (741 pairs), the paper does not evaluate all pairwise certificates, and its accuracy-gap analysis covers only eight pairs on a single model and benchmark. Furthermore, experiments focus exclusively on math and science QA, leaving the effectiveness of the method on open-ended generation tasks where truncation samplers are widely used unclear.

Questions for authors

N/A

4 comments

We thank the reviewer for finding the problem interesting and important, and the coupled estimator a useful, unbiased methodology whose insights may extend beyond truncation samplers.

Three of the four weaknesses restate points the paper itself makes. Below we quote each and add new evidence. For the fourth, compute, we ran new experiments and now measure it.

"The separation certificate … is nearly vacuous … rarely provides useful guidance": this is our main finding. The abstract says "a bound alone rarely settles a comparison" (p. 1); the conclusion says "a comparison has to be measured rather than bounded" (Sec. 7). A large certificate is not a loose bound: when the nesting order never changes, it equals the exact total variation (Thm 3(ii); Sec. 1: "this is also their exact total variation distance"). In 8/8 new runs, it matched the divergence rate observed in real coupled decoding (within 0.01). The samplers really are that far apart as distributions while their accuracies are close, which is why we measure.

New results show where the certificate is informative. Under the chat template with shared EOS stopping, the default pair's certificate is 0.17. It bounds every measured gap and makes Prop 7's guarantee non-vacuous (≥2.1× guaranteed; 12.3× measured). At T=256, 20 of 109 logged pairs of two different truncation rules fall below 0.5. This supports our central claim: the certificate faithfully measures how far two samplers diverge, and because they can diverge widely while accuracy barely moves, a comparison has to be measured, which our estimator makes efficient.

"Limited novelty of the coupling method": our contribution is not the coupling construction, which is standard (Sec. 6, App. C). It is that truncation rules make the maximal coupling closed-form and rejection-free at every step, and what follows from that:

  • the observation that deployed truncation rules are nested (0 violations in 997 million logged cutoffs);
  • a certificate computable from logs, with its exactness condition (Thm 3);
  • the structural results (Prop 5, Cor 6);
  • the measurements, which tell practitioners when a comparison must be measured and how many samples it needs.

The variance gain is not generic common random numbers either: the maximal coupling has 3.2× [2.6, 4.0] lower variance than shared seeds and 1.5× [1.2, 1.9] lower than a shared-uniform coupling. The contribution therefore rests on what nesting makes possible, a closed-form, log-computable distance between any two truncation samplers and a coupling whose gain is specific to that structure.

"Lack of compute-normalized efficiency comparisons": we measured it. At a fixed CI, coupling needs 6.1× less GPU time than prompt-level pairing on the default pair. It needs 1.2× and 1.5× less on the other GSM8K pairs, and 1.8× less on MATH-500. A fork costs under a tenth of a forward pass. In batched decoding on the default pair, the two methods cost within 8% of each other from 1 to 64 pairs per batch, and coupling keeps 4.1× better precision per GPU-hour. On top-k vs top-p, which forks within about 5 tokens, the batched gain disappears. This turns the variance reduction into a measured saving in GPU time, so the estimator's practical value no longer rests on variance per sample.

"Limited empirical coverage": each part is listed in our Limitations (Sec. 7). These does not limit the validity, and we extended the evidence:

  • Table 5: rerun for all eight pairs. No gap exceeds its certificate; r=0.91, now significant against a pure-noise null (p=0.012; submitted p≈0.055).
  • Pair logs: certificates for 145 of the 741 grid pairs.
  • Open-ended text: MT-Bench certificates (0.65 under the chat template). We have no outcome-level result on open-ended text.

With the rerun, the evidence that the certificate tracks measured gaps is now significant rather than inconclusive, so this part of the paper is also stronger.

As we have addressed each concern, and the new results strengthen both the estimator and the certificate, we kindly ask the reviewer to consider an updated assessment.

Dear Reviewer RZfx,

Thank you again for your review. As the discussion period closes on November 18, we wanted to check whether our response addressed your concerns.

In brief:

  • Coupling needs 6.1× less GPU time than prompt-level pairing at a fixed CI.
  • In the chat format with shared EOS stopping, the default pair's certificate is 0.17 and bounds every measured gap.
  • The certificate-versus-gap test is now significant (p=0.012).

We would be glad to clarify anything that remains. If our response resolves your concerns, we would be grateful if you would consider updating your assessment.

Dear Reviewer RZfx,

The discussion period closes on November 18 AoE, so we would like to ask you directly to check our rebuttal. Each weakness you listed now has a direct answer:

  • You asked for a compute-normalized comparison. Measured: at a fixed CI, coupling needs 6.1× less GPU time than prompt-level pairing on the default pair, and 1.2–1.8× less on the others.
  • You found the certificate nearly vacuous. This is our main finding, and the certificate is not loose. It matches the divergence rate observed in real coupled decoding in 8/8 runs, and in the chat format with shared EOS stopping it falls to 0.17 and bounds every measured gap.
  • You questioned the novelty of the coupling. We do not claim the construction (Sec. 6). The gain is specific to nesting: the maximal coupling has 3.2× lower variance than shared seeds and 1.5× lower than a shared-uniform coupling.
  • You noted limited coverage. The certificate-versus-gap test, rerun on all eight pairs, is now significant (p=0.012).

Could you let us know before the deadline whether this resolves your concerns, or which points remain? We will respond right away. If the new results resolve them, we ask that your assessment reflect this.

Dear Reviewer RZfx,

We have written to you three times and no reply, and the author discussion closes in 24 hours.

Your major concerns were that the certificate is nearly vacuous, that the coupling lacks novelty, that there is no compute-normalized comparison, and that the empirical coverage is limited. We have resolved all four, and the results strengthen the paper:

  • Compute: at a fixed confidence interval, coupling needs 6.1× less GPU time than prompt-level pairing (Table 2).
  • Certificate: it matches the divergence observed in real decoding in 8 of 8 runs. In the chat format with shared EOS stopping, it falls to 0.17 and bounds every measured gap (Table 6).
  • Novelty: the gain is specific to the maximal coupling, with 3.2× lower variance than shared seeds and 1.5× lower than a shared-uniform coupling. Sec. 6 separates the standard results from what nesting adds.
  • Coverage: the certificate-versus-gap test is significant (p = 0.012), and certificates for 145 grid pairs and for MT-Bench are reported.

Every change is in blue in the revised PDF, and in the supplement.

ICLR's reviewer guidelines state that "it is crucial that you are actively engaged during this phase" and ask reviewers to "update your review, taking into account the new information collected during the discussion phase and any revisions to the submission." Your score predates all of this evidence. Please update your assessment, or tell us today which concern remains.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revision

We have uploaded a revised PDF. Every change from the submitted version is shown in blue. The main changes are below.

Corrected generation length and regenerated results. We fixed the generation length in the estimator: every arm now generates exactly T new tokens. We regenerated every estimator number with the corrected code, including Table 2, Table 5, Figure 4, the baselines and the negative control (App. E). The results hold. On GSM8K the three variance reductions are 1.2–4.7×, all significant after Holm correction, and the default pair's reduction is now 4.7× (it was 3.8×).

Prompt-level confidence intervals. Every interval and test now resamples prompts, keeping all seeds of a prompt together. With this, the test of whether the certificate tracks the measured gap is significant (p = 0.012; it was ≈0.055).

Compute-normalized comparison. We added a GPU-time column to Table 2 and a new paragraph in Sec. 4.1. At a fixed confidence-interval width, coupling needs 6.1× less GPU time than prompt-level pairing on the default pair and 1.2–1.8× less on the other pairs. We also report a batched implementation, including the pair where the batched gain disappears (App. F).

One stopping rule for the certificate and the outcomes. We added an experiment with the chat template in which the certificate and every outcome arm stop at the same end-of-sequence token (Sec. 4.1, Table 6). On the default pair the variance reduction is 5.4×, the certificate is 0.17 and bounds every measured gap, and Proposition 7's guarantee holds (at least 2.1× guaranteed, 12.3× measured).

Coupled runs represent each sampler. We added Table 7 and a sentence in the introduction. In 16 of 16 comparisons, a sampler's accuracy inside the coupled run shows no significant difference from its accuracy when run alone.

Sample sizes. We rewrote Sec. 5 to separate claims about a fixed benchmark from claims about new prompts, and rebuilt Table 3 from each benchmark's own variance rather than GSM8K's. A fixed benchmark needs about 4.6 samples per prompt on GSM8K, 23 on MATH-500 and 305 on AIME 2025.

More than two samplers. We extended Corollary 4: one joint run gives every pair of samplers its own maximal coupling. The proof is in App. A, with a numerical check.

Standard versus new. We added a passage in Sec. 6 that maps each result to its standard counterpart and states what is specific to nesting. We also replaced several paper-specific labels with plain descriptions.

Limitations. We now report MT-Bench certificates and the pair logs for 145 of the 741 grid pairs, and we withdrew the suggestion that reasoning chains inflate certificates.

Smaller fixes. Figure 2 and its caption now describe the 64-token trace actually shown. The single-run model-scale reductions from an earlier pipeline are no longer used, and Table 3 replaces the sample-size figure based on GSM8K's variance. We also corrected the AIME stop rate at the longest cap (0.93, not 1.00), identified the "independent" arm as the prompt-paired run, and updated the compute accounting.

We hope the revision makes each answer easy to check against the paper.

AuthorsOpenReview ↗

Follow-up for the Area Chair

Dear Area Chair,

With the author discussion closing on November 18, we write to summarize where the discussion stands and to ask for your help.

The reviewers' major concerns were clear. Reviewer x5vt found a generation-length error in our estimator code and a bootstrap that resampled outcomes rather than prompts, asked for a compute-normalized comparison, and questioned whether coupled runs represent each sampler. Reviewer RZfx found the certificate nearly vacuous, questioned the novelty of the coupling, and also asked for a compute-normalized comparison. Reviewer ygCW asked whether the results hold when the certificate and the outcomes share one stopping rule, and whether our sample-size analysis targets fixed benchmarks or new prompts.

We have answered every one of these. We fixed the code and regenerated every number, and the results hold and are stronger: the default pair's variance reduction rises from 3.8× to 4.7×, and the certificate-versus-gap test, inconclusive in the submission, is now significant (p = 0.012). We measured compute directly: at a fixed confidence interval, coupling needs 6.1× less GPU time. We ran the shared-stopping-rule experiment: the reduction is 5.4×, the certificate falls to 0.17 and bounds every measured gap, and Proposition 7's guarantee holds. In all 16 comparisons, coupling leaves each sampler's accuracy unchanged. We also showed that the gain is specific to the maximal coupling, which has 3.2× and 1.5× lower variance than shared seeds and a shared-uniform coupling. We separated fixed-benchmark from new-prompt sample sizes using each benchmark's own variance, and we extended Corollary 4 to any number of samplers. The revised PDF shows every change in blue. The revised supplement contains the corrected code, every result file and the scripts that recompute the revised tables.

We must also note that most of the weaknesses in two reviews restate what the submitted paper itself disclosed. Three of Reviewer RZfx's four weaknesses repeat the paper's own statements: that the certificate is large (our main finding, stated in the abstract), that the coupling construction is standard (Sec. 6 and App. C, citing the same references), and the coverage limits (listed almost word for word in our Limitations, Sec. 7). Both of Reviewer ygCW's weaknesses repeat disclosures in the paper: App. B states that the stopping rules differ and that this "is not resolved here", and Sec. 5 states that the extrapolation "assumes the τ² measured on GSM8K". Counting a paper's own findings and disclosed limitations as weaknesses against it is not acceptable, since it penalizes exactly the transparency reviewing should reward. We have nonetheless answered every one of these concerns with new evidence, as above.

The reviewers themselves recognized the value of this work. Reviewer RZfx called it an "interesting and important problem" and noted that the estimator "improves statistical efficiency". Reviewer x5vt found the method "inexpensive and straightforward to implement, and backed by theory". Reviewer ygCW wrote that "this improves measurement efficiency."

Yet the discussion has not caught up with the evidence. Reviewer x5vt acknowledged the revision, wrote that it "now answers compute and sample-size questions" and raised the Presentation and overall scores. The one concern x5vt kept, that the corrected code was not available, is resolved by the revised supplement. We have written to Reviewers RZfx and ygCW three times each, once with our full response and twice with follow-up reminders, and we have posted the revised paper. Neither has responded. We have done everything in our power to engage them, the next step is no longer in our hands.

We therefore ask you to take immediate action. Please ask Reviewers RZfx and ygCW to read the revision and respond before the discussion closes. ICLR's reviewer guidelines state that "it is crucial that you are actively engaged during this phase" and ask reviewers to "update your review, taking into account the new information collected during the discussion phase and any revisions to the submission." If they do not respond, we ask you to weigh the revised paper, these answers and the points above in your evaluation, since those scores predate all of this evidence.

Thank you for your time and attention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.