acceptodds
Under review as a conference paper at ICLR 2027

How Far Can Shallow Evaluation See? The Limits of Best-of-K Prediction

Abstract

Can evaluations with few candidates per prompt predict Best-of-K success at larger budgets? Under a common rank-correctness response, complete shallow, score-ordered joint label distributions leave the same worst-case ambiguity as rank marginals alone. We characterize this gap exactly at every finite deployment budget and construct collisions attaining it. In a slope-bounded monotone response class, worst whole-curve ambiguity scales inversely with the square of evaluation depth; smooth monotonicity without a common slope bound can still permit nearly maximal ambiguity as deployment grows at fixed evaluation depth. With a common response and arbitrary unknown atomless group-specific score distributions of unrestricted complexity, numerical scores do not improve minimax risk. By contrast, one unknown conditional score distribution shared across prompts enables unbiased pooling despite heterogeneous responses. For declared small uniform departures from sharing, we derive risk bounds that match on the logarithmic scale. Within-prompt comparisons reduce mismatch bias, but noise limits the correction order finite data support. At equal total cost, two designs can both be consistent yet differ in whether they eventually meet a shrinking error tolerance. Together, these results separate irreducible information loss from finite-sample error and show why prediction range and consistency do not determine attainable precision.

Primary area:
learning theory
Keywords:
Best-of-K, test-time scaling, model evaluation, statistical learning theory, identifiability, minimax estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_pjt8OpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper tackles the question of whether a few candidate evaluations per prompt predict Best-of-K success at larger budgets. The paper provides many theoretical and empirical investigations, yet lacks a clear motivation and explanation of the problems at hand. As such, following the arguments is difficult, and the exact impact of each theoretical result is not clearly articulated. I find the current paper's write-up unsuitable for this venue, as a clear communication of what problem is addressed, why it is addressed, and how it is addressed is difficult to understand from the current manuscript.

Critical strengths

[S1] Appears Theoreticaly Rigorous: The main theoretical write-up, whilst complicated to follow, does appear very rigorous. Yet I do not have the expertise of verifying those, as I am not from this particular field, and the presentation and write-up make it hard to follow the arguments (see weaknesses).

Critical weaknesses

Summary: As stated in my confidence, I am not a specialist in this particular research field, yet I believe a paper should still be understandable and motivating for researchers willing to delve into these topics. The overall assessment and my verdict of „clear reject“ are based on the missing motivation, unclear statement of contributions, what practical takeaways exist, how the theory is linked to the experiments, and the overall presentation. As such I find the paper At its current state, it is not acceptable. [Update: After the first response, I have updated my score to "weak reject". See my comment for details] [Update2: After the second response, my core problems have been resolved.]

[W1] Motivation of Contributions: The paper does not clearly indicate which problem is tackled. The abstract seems to present the theoretical investigation, but does not specify the settings in which it is used. A motivation for the problem at hand is lacking, and the reader who is not deeply immersed in the topic quickly becomes lost. The paper simply starts with a question, which is not clearly motivated at all. Also, the question at the beginning of the paper cannot be clearly understood: what does „evaluations with a few candidates per prompt“ have to do with „large budgets“, as the latter does not seem to be motivated. A revision should make it clearer exactly what is being tackled and why the problem is interesting to study.

[W2] Theory and Notation: Within the first page of theory, more than 20 different symbols for notations are introduced alongside various concepts. For example, the term „Chebyshev“ polynomials is introduced without clearly stating exactly why they might be needed or what their motivation is. Theorem 3.1 then directly shows an equality for those polynomials, so at least an explanation of that would be necessary. Also, the results are not put into context but are simply stated, and then the paper moves on. Overall, this makes it difficult to follow the theoretical storyline. A clear revision, alongside a consistent storyline, is necessary to make the understanding clear.

[W3] Experimental Evidence: The experimental section is currently built solely from paragraphs describing the main results of a figure in the appendix. That put aside, it is not clear to what extent the theoretical insights drive empirical gains. Datasets and setup are not discussed in the main sections at all, and are hard to understand from the paper's appendix paragraph. The experiments seem to be run on small-scale models, and I could not find a discussion of exactly why. Overall, the experimental paragraphs do not clearly explain which insights from the theory they support or investigate.

[W4] Experimental Setup: In Appendix D, it is stated that the experimental setup can be understood. There, the ranking models are listed, but details such as exact protocols are not stated. No code is available to check for protocols, further hindering understanding of exactly what has been run. This does not increase the trust in the experimental findings.

[W5] Writeup and Storyline: In my opinion, the paper lacks a clear storyline and requires substantial rewriting. The problem at hand, „Best-of-K prediction, “ is not well motivated, but simply used as a setting for investigation. Core concepts such as „shallow evidence“ are not introduced at the beginning, but assumed to be known throughout. Theoretical results are not put into context, and their implications are not clearly stated.

Questions for authors

[Q1] What specifically are the contributions of this work? What implications do the theoretical results have, and how have they been addressed empirically? Could you give a brief context of what setting the papers are in, which problems exist, how this paper tackles them, and how the theory and experiments relate to this?

[Q2] What is the answer to the stated question, „Can evaluations with few candidates per prompt predict Best-of-K success at larger budgets? Which theoretical and empirical evidence support the answer?

[Q3] What exactly is Figure 1 showing us? Can you give a brief explanation of what belongs to what?

[Q4] Could you explain the following concepts briefly and put them into context of this work: fibres; shallow evaluations; joint information; joint pattern and larger budgets

6 comments
AuthorsOpenReview ↗

Response to Reviewer pjt8 (1/2): Q1, Q2, W1 and W5

Thank you for a frank and constructive review. We agree that the submitted version was written for specialists. Your review names five issues, and the revised PDF (changes in blue) addresses each:

  • Motivation. The abstract and introduction now open with the deployment setting and the problem (p. 1).
  • Contributions. The introduction then states our answer and four contributions (pp. 1–3).
  • Takeaways. Each theorem is followed by one sentence on what it means for an evaluator.
  • Theory and experiments. Table 2 in Section 6 maps every study to the claim it tests (p. 10).
  • Presentation. Table 31 gathers the notation (p. 70), and Figure 1 has a reading guide (p. 2).

Q1 / W1 / W5: setting, problem, contributions

Setting. Best-of-KK is a standard way to use a reward model or verifier at inference time. Sample KK answers to a prompt from a language model, score each with the verifier, and return the highest-scoring one. A deployment cares about its accuracy θK\theta _ K, the probability that the returned answer is correct. Depending on the scorer, θK\theta _ K can rise or fall as KK grows, so it must be judged at the KK actually deployed.

Problem. Measuring θK\theta _ K directly requires generating and scoring KK candidates per evaluation prompt: K/BK/B times the generation and scoring of a shallow evaluation with BB labeled candidates per prompt. Reward-model benchmarks often report pairwise accuracy (B=2B=2). We ask when extrapolating from such shallow evaluations is justified: what does a cheap evaluation with BB candidates per prompt tell us about Best-of-KK accuracy at a larger KK?

The key point. The obstacle is not sampling noise. As KK grows, Best-of-KK accuracy depends increasingly on how often the very highest-scored answers are correct, and even exact knowledge of every shallow correctness pattern leaves this undetermined; joint patterns leave the same worst-case gap as per-rank accuracies alone.

Contributions.

  1. A limit that more shallow data cannot remove. Two systems with identical complete shallow label records can differ in Best-of-KK accuracy by ΔB,K\Delta _ {B,K}, and by no more. We compute ΔB,K\Delta _ {B,K} in closed form for every BB and KK; it approaches 100 points once KK is a few times B2B^2 (Theorem 3.1). In the worst case, numerical scores do not help (Theorem 3.2).
  2. What extra assumptions buy. A bound LL on how fast correctness can change with score rank reduces the worst-case ambiguity to at most 4L/B24L/B^2 (Theorem 4.2). For monotone cubic responses, no prediction from per-rank accuracies alone is consistent over the whole class; one added joint statistic gives one (Section 4.2). A score distribution shared across prompts makes cross-prompt pooling unbiased (Theorem 5.1).
  3. Why consistency is not precision. When this sharing is only approximate, pooling is biased, and finite data can correct the bias only up to a limited order. So two designs can both be consistent, or predict equally far in KK, yet differ in attainable precision (Theorem 5.2; Section 5.4).
  4. Experiments test these mechanisms and their assumptions on public reward-model benchmarks and new MATH-500 score banks, including a prospective test whose predictions were frozen before the reference data existed.

Practical takeaway. Before extrapolating a shallow evaluation, check which of these assumptions the system can justify. If none can, evaluate at the target budget directly, or report the range of Best-of-KK accuracy that the observed per-rank accuracies allow (App. D.1), whose width Theorem 3.1 bounds.

Q2: the answer to the opening question

In general, no: shallow records alone leave a worst-case gap, which we compute exactly (Theorem 3.1). Under specific added structure, yes. With B=3B=3, two systems with identical records, including every joint pattern, have Best-of-12 accuracy 85.1% and 14.9% (Figure 1, centre). With B=4B=4, the worst-case gap is 60 points at K=16K=16 and 99.7 points at K=64K=64. The empirical evidence:

  • Across 17 reward models, pairwise accuracy and Best-of-16 rank the scorers alike on some benchmarks and only partly alike on others (Kendall τ=.71\tau=.71–11), and shallow data cannot tell which: a bootstrap screen that allows for the range per-rank accuracies leave open (Theorem 3.1) confirms none of these orderings from up to four candidates per prompt (Section 6).
  • In simulations enforcing score sharing (five populations, B=4B=4, K=16K=16), pooled error falls as M−1/2M^{-1/2} (Theorem 5.1) to .0025–.0038 at M=32768M=32768, the lowest of the tested rules in four of five (Table 15).
  • On natural scores, a label-free test (Section 5.1) rules out the sharing that pooling needs; as it warns, pooling over-predicted Best-of-16 by .20 in a prospective test, while extrapolation that does not rely on sharing was within .02 (Table 2).
AuthorsOpenReview ↗

Response to Reviewer pjt8 (2/2): Q3, Q4 and W2-W4

Q3: Figure 1

The figure has one worked example (top left and centre); the other panels are separate results under their own assumptions, not steps of one method.

  • Top left. Per prompt, three candidates are sampled, sorted by score and labeled (filled = correct): 8 possible patterns ("words"), each with the same probability under the blue and the red system.
  • Centre. Best-of-12 accuracy weights each score percentile uu by the winner density 12u1112u^{11}. The blue system places its correct answers where this weight is large and reaches 85.07%; the red system reaches 14.93%. The 70.14-point gap is the largest possible for Best-of-12 at depth three (Theorem 3.1).
  • Right column. Numerical scores do not help in the worst case (Section 3); at depth three a linear target is identifiable only for a quadratic weight (1,u,u21, u, u^2), a slope cap shrinks ambiguity to order 1/B21/B^2, and smooth monotonicity alone still allows a 96-point gap as K→∞K\to\infty (Section 4).
  • Left column and bottom. One joint statistic restores recovery for monotone cubics (Section 4.2); a shared score distribution makes pooling unbiased (Section 5.1); under approximate sharing, a paid pilot and same-prompt comparisons correct the bias up to an order DD that finite data support, and designs with the same horizon can differ in precision (Sections 5.2–5.4).

The caption now has this guide.

Q4: concepts

  • Shallow evaluation. An evaluation that samples, scores and labels only a few candidates per prompt, BB of them (B=2B=2 for pairwise accuracy).
  • Larger budgets. The value K>BK>B used in deployment (Best-of-KK).
  • Joint pattern (word). For one prompt, the correctness labels of its BB candidates from lowest to highest score, e.g. (wrong, right, right).
  • Joint information. The full distribution over these 2B2^B patterns: all the label information of a shallow evaluation, more than per-rank accuracies.
  • Fibre. The set of all systems that produce the same joint information. A shallow evaluation cannot tell them apart, so the spread of Best-of-KK accuracy across a fibre is what it leaves undetermined; Theorem 3.1 computes the largest spread.

W2: notation and Chebyshev polynomials

Shallow records determine a weighted average of the response across a whole response class only when the weight is a polynomial of degree below BB (Theorem 4.1). Best-of-KK uses the weight KuK−1Ku^{K-1}, which for K>BK>B is not such a polynomial, so the undetermined part is set by how far KuK−1Ku^{K-1} is from them (Theorem 3.1). The error of the best approximation changes sign where a Chebyshev polynomial does, and that sign pattern gives the worst-case pair: two systems with identical shallow records, whose Best-of-KK accuracies differ most. The revision says this before Theorem 3.1, opens Section 3 with the worked example and moves the measure-theoretic setup and proof-only symbols to the appendix.

W3 / W4: experiments, models, protocols, code

The experiments test the theory's mechanisms and assumptions, and two of its outputs pay off directly. The label-free test of Section 5.1 rules out sharing in every natural bank we examined, and pooling over-predicts in most of them. The order DD of Theorem 5.2, computable before sampling, says when to correct: in 2,880 new simulated cells fixed in advance, no correction beat pooling at D<2D<2, and correcting only at D≥3D\ge3 cut the mean error by 14%. Table 2 (Section 6) gives each study's claim, data and models, estimator, budget and outcome.

  • Data and models. The public Best-of-KK sets of Frick et al. (2025): four domains, 32 labeled candidates per prompt from GPT-4o mini, Claude 3 Haiku, Llama-3-8B or Gemma-2-9B, scored by 17 reward models (0.6B–8B). MATH-500 banks from Qwen2.5-7B-Instruct and Mistral-7B-Instruct-v0.3, scored by nine 3B–8B reward models.
  • Scale. Model scale is not what we study, and the conclusions do not depend on it. On all 164 public-set banks (GPT-4o mini, Claude 3 Haiku included), the label-free test rules out sharing, and rank-based extrapolation from four candidates tracks Best-of-16 within a median .04 (pooling: .09). New sealed prospective runs with Qwen2.5-32B-Instruct and Mistral-Small-24B-Instruct repeat the 7B result under both scorers: pooling over-predicts by .15–.21; plateau and inverse-budget extrapolation stay within .04 (App. F.4, F.6).
  • Protocols. Tables 4–5 give decoding settings, scorer input format, labeling rule, estimator, budget, repetitions and seeds.
  • Code. The anonymized package linked from the supplementary material holds the certificate scripts and recomputes the numbers we quote from App. D.2.4–D.2.6, D.3.5, D.3.7 and F.

Your questions gave the revision its outline: the introduction now follows Q1 and Q2, and Figure 1 has the reading guide Q3 asked for. We are glad to clarify any passage that remains hard to follow.

Thank you for the detailed response. The revision helped me understand the problem, and I now see it as a refinement of previous work (Dorner et al., 2026). That makes it harder for me to judge its significance. I still have concerns about presentation and experiments.

Summary The revision makes a real effort to address the points I raised, and I recognize the time spent making the paper accessible to readers outside the subfield. As the authors have put effort into addressing my main concerns with the presentation, I have increased my score from 1 to 2. However, I remain unconvinced that the paper is in an acceptable state. I would only be willing to change my assessment if other reviewers found that the theoretical insights outweigh the empirical findings, which I do not currently see.

Presentation [RW1] The revised abstract is dense with technical detail, and I found the answers in the rebuttal much easier to follow. I would suggest aligning the abstract with the core findings, in one or two sentences each:

  • Best-of-K is widely used in practice.
  • Evaluating at the deployed K is expensive, so systems are often evaluated with only B < K candidates per prompt.
  • The question is how far such shallow evaluations can be extrapolated.
  • The theory finds …
  • The experiments show …

This would convey the paper’s whole message much more clearly.

I still find Figure 1 too confusing and would argue against keeping it in its current form. As the reading guide itself shows, it combines many separate results in a single figure. The storyline is hard to follow and requires constant switching between figure and caption. An overview of the different assumptions is reasonable. But I think it matters more to state clearly which theorems change the result, and in which direction, than to depict the theory with schematic curves.

Empirical evidence [RW2]

The new Table 2 mostly summarizes information already in the appendix, and I do not find it suitable as an experiment section. A reader cannot tell which findings are central and which are details. All rows except the last two are simulated studies that check the theory under its own assumptions. The last two rows use natural scores and come closest to the motivating problem. There, the recovery methods derived from the theory perform worst, while simple extrapolation rules perform best. Reviewer f9TX raised a similar point. I would ask the authors to clarify what practitioners should take from these results. I would rely on reviewer f9TX’s judgment as to whether these findings are of sufficient interest to the community. To me, they suggest that the paper is better framed as purely theoretical work, which I am not in the best position to assess.

Reviewer_f9TXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a cheap shallow evaluation can predict if the top-scoring candidate is correct at a larger sampling budget K. In a shared-response model it derives a tight worst-case uncertainty bound (an L1 distance to low-degree polynomials), identifies when linear targets are recoverable, and shows how a slope bound and a joint label statistic control error. A second numerical-score model allows heterogeneous responses under exact or approximate distribution sharing, giving unbiased pooling and logarithmic minimax risk bounds; simulations and natural-score experiments confirm the mechanisms but also reveal violations of the sharing assumption.

Critical strengths

S1. Theorem 3.1 goes beyond showing that early accuracies can approximately agree while later behavior differs: it matches the entire shallow ordered-label law and attains the maximal gap at each finite target budget. Combining ordered cancellation with the approximation certificate is a substantive step. The shape results further explain why qualitative monotonicity and a quantitative slope constraint have different consequences (Theorems 4.1–4.2).

S2. Theorem 5.1 preserves the original same-prompt target when pooling heterogeneous responses and explicitly accounts for prompt dependence in variance. Theorem 5.2 charges the pilot and handles dependent comparisons, while Appendix C proves the empirical-mean lower bound using the laws of the actual summaries, not merely equal expectations. These distinctions strengthen the statistical interpretation.

Critical weaknesses

W1. The practical scope of the approximate-sharing recovery theory remains unestablished. Theorem 5.2 matches risk only on a logarithmic scale and does not yield a calibrated finite-budget choice of correction order. The experiments do not implement its complete construction; the grid in Appendix D.2.5 does not isolate the proposed effective order D from depth B, and pooling outperforms the best tested correction in 129 of 140 local-regime cells. Natural-score studies exclude the assumed sharing regime, and direct evaluation or simple rank-based rules are competitive or better. These findings do not refute the theorem, and the authors acknowledge them, but they substantially limit the evidence for using the recovery results to guide an actual evaluation design.

W2. The breadth of the presentation makes the most technically ambitious claims difficult to independently audit. Theorem 5.2 combines approximation lemmas, two different correction constructions, pilot concentration, a multi-regime argument, and an identity-preserving lower bound across Appendices B.3–B.6. Theorem C.1 and the exact-arithmetic certificate in Appendix A.7 introduce further substantial machinery. A compact theorem-to-assumption and proof-dependency map, together with executable verification artifacts, would materially improve assessability. The reproducibility statement currently promises public experimental code only upon acceptance.

Questions for authors

Q1. Can you provide an explicit finite-budget instantiation of the full Theorem 5.2 construction, including all tuning constants and pilot costs, for which its stated error bound is informative? If none is available at the studied budgets, what concrete design conclusion should a practitioner draw from D beyond its asymptotic interpretation? This would affect my assessment of the recovery contribution’s significance.

Q2. Can you provide an anonymized executable package for the Appendix A.7 rational/interval certificates and the Appendix D experiments, and identify which experimental rules implement which parts of the theoretical construction? This would directly improve confidence in the certificate-based and empirical claims.

5 comments
AuthorsOpenReview ↗

Response to Reviewer f9TX (1/3): Q1 and W1

Thank you for a precise and fair review; S1 and S2 name the distinctions we most hoped a reader would see. In brief:

  • Q2 / W2. The supplementary material links to the executable package, and a new appendix maps the proof dependencies.
  • Q1 / W1. New Proposition B.7 gives the order-two rule we simulate explicit finite-budget constants and a strict worst-case separation from pooling; at our budgets, the rule of Theorem 5.2 itself is pooling, with the explicit bound (15). DD gives a rule fixed before sampling: correction never beat pooling at D<2D<2, in 173 cells of three studies and in 1,152 new cells fixed in advance (Part 3); at D≥3D\ge3, order two won all 44 grid cells and 78% of the new ones.

Q1: a finite-budget instantiation

The full Theorem 5.2: at our budgets, its pooling branch. The proof uses the correction branches only for D≥D0>39D\ge D _ 0>39 (our grid: D≤4.6D\le4.6), so at our budgets the theorem's rule is pooling and its upper bound is (15) (App. B.6): every constant is explicit, there is no pilot, and the bound is informative for small KδK\delta (for example .049.049 at B=8B=8, M=4096M=4096, K=32K=32, δ=1/2048\delta=1/2048, where the measured error is .010.010). The threshold D0D _ 0 is a limit of our proof's constants, not a lower bound for every estimator; Section 5.2 now states this.

New: explicit constants for the rule we simulate. The grid's order-two rule is exactly rule (95) of Theorem B.6 at j=2j=2. New Proposition B.7 (App. B.8) bounds its worst-case error over the declared-δ\delta class (atomless scores), charging the pilot and all MBMB candidates and labels: MAE ≤\le RMSE ≤(β2+A/n+P)1/2\le(\beta^2+A/n+P)^{1/2}, with explicit integrals (bias, analysis and shared-pilot variance), evaluated exactly by a package script. At the grid cell B=8B=8, M=218M=2^{18}, K=32K=32, Kδ=1/16K\delta=1/16, this is .033.033, where (15) gives .127.127. In the introduction's setting (B=4B=4, K=16K=16, M=218M=2^{18}, δ=1/256\delta=1/256), it is below .026.026, while pooling's error exceeds .028.028 on a valid two-type population. These separate worst-case risks, not two upper bounds, among rules reading the same depth-BB records.

What DD tells a practitioner at finite budgets

Post hoc, on the frozen grid (Table 18; 320 cells with δ>0\delta>0), fixed order two against pooling:

DD<2<2[2,2.5)[2,2.5)[2.5,3)[2.5,3)≥3\ge3
order two beats pooling0 of 14623 of 8041 of 5044 of 44
median RMSE, order two / pooling4.361.35.54.26

(a) Whether to correct. In all 173 cells with D<2D<2 (the grid and two earlier studies, App. D.2.4 and D.2.6), no tested correction beats pooling. A paired bootstrap within seed blocks, with approximate simultaneous 95% intervals over all 320 cells, resolves 43 of the 44 wins at D≥3D\ge3 and all 146 losses at D<2D<2. So D<2D<2 is a signal to pool, and D≥3D\ge3 a signal that order two can pay. How often it does depends on the response: quadratic responses, which leave pooling little bias to remove, gain least (none of 10 such cells with D≥2.5D\ge2.5 in the two other studies, where bit and threshold win 16 of 20; 60% of the new cells with D≥3D\ge3, Part 3). This accounts for the 129 of 140 you cite: 112 of those local-regime cells have D<2D<2, and all 11 where correction wins are among the 12 with the largest DD.

(b) When more prompts pay. At fixed (B,K,δ)(B,K,\delta) and response, raising MM from 2102^{10} to 2182^{18} lifts order two from 4 to 42 wins in 64 trajectories (38 losses become wins, none the reverse). At fixed depth, more prompts can thus make correction pay.

(c) How far to extrapolate. The fixed-error horizon is Θ(min⁡{MB,D2/δ})\Theta(\min\lbrace MB,D^2/\delta\rbrace), Eq. (18). Once D2/δD^2/\delta binds, more prompts help only through D≤BD\le B, which grows like log⁡M/log⁡log⁡M\log M/\log\log M (App. B.7); reducing δ\delta or adding depth is then the lever.

(a) and (b) are simulation patterns, tested prospectively in Part 3; (c) follows from Eq. (18).

W1: DD versus depth BB

You are right that the headline correlation does not isolate DD from BB; a stratified re-analysis of the frozen outputs does.

  • Within every depth, DD separates wins of order two from losses. The probability that a winning cell has a larger DD than a losing one (AUC) is .88,.98,.99,.98.88,.98,.99,.98 for B=2,4,8,16B=2,4,8,16, and .97.97 over the whole grid, against .72.72 for depth itself.
  • Within every depth, the best order also rises with DD (ρ(j∗,D)=.48,.76,.45,.38\rho(j^\ast,D)=.48,.76,.45,.38). Depth keeps an association of its own (partial ρ(j∗,B∣D)=.69\rho(j^\ast,B\mid D)=.69), consistent with a finite-sample effect of the one-score pilot.

App. D.2.5 now reports these findings and states which parts of the construction the experiments validate (the comparison kernel and the pilot accounting).

AuthorsOpenReview ↗

Response to Reviewer f9TX (2/3): W1 (continued), Q2 and W2

W1: logarithmic matching

The match in (17) is up to constants whenever the sampling floor dominates: if ρ≥e−cE∗\rho\ge e^{-cE _ \ast}, then cρ≤Rδ≤2Cρc\rho\le R _ \delta\le2C\rho. Only the mismatch term is matched in its exponent, as e−Θ(E∗)e^{-\Theta(E _ \ast)} with unidentified constants. The revision states this after the theorem.

W1: natural scores

On natural scores the theory guides the design through its label-free test, which says before any label is spent that pooling is not justified. Sharing-radius lower bounds (δ≥.268\delta\ge.268 at 95% confidence under each of nine scorers, App. D.3.5) rule the sharing route out, and the warning is borne out: prospective Best-of-16 discrepancies are .20 and .80 for pooling and order two, versus .02 and .0006 for plateau and inverse-budget extrapolation, which do not rely on sharing (Table 27). The same pattern holds in sealed reruns with 32B and 24B generators and on 164 public-set banks, where the certificate excludes sharing in every bank and plateau or inverse-budget extrapolation is closer than pooling in 132 (App. F.3, F.4, F.6). These studies concern the recovery results of Section 5, not the exact limits of Section 3 or the response-structure results of Section 4, which S1 describes; the revision states this scope in the introduction.

Q2 / W2: executable package and auditability

The anonymized package, linked from the supplementary material, needs no GPU or network access; it runs in about six minutes on a laptop, and every script prints each check next to the paper's value.

  • certificates/: verify_A7.py re-derives Appendix A.7 from the data printed there, in exact rational and interval arithmetic using only the standard library: the rank-6 rational 10×2010\times20 matrix, the cubic identity for all 16 word differences, the root bracket of QQ, the enclosure .02921638735166<G(t∗)<.02921638735168.02921638735166 < G(t _ \ast)<.02921638735168, and the contraction certificate for the marginal maximizer. order_two_bound.py evaluates Proposition B.7 exactly.
  • experiments/: App. D.2.5, D.3.5 and D.3.7, with analysis code, seeds and saved raw outputs; one command per study recomputes its tables and the numbers we quote. The order grid also reruns in a reduced configuration, matching the stored estimates to 10−1510^{-15}, and has the analyses of Part 1. App. D.2.4 and D.2.6: saved outputs, the zone check of Part 1, regeneration from seeds.
  • MAP.md: paper item → script → theoretical object.

The revised reproducibility statement points to this package instead of promising code upon acceptance.

Which experimental rules implement which parts of the theory

Rule (App. D; Table 6 of the revision)Theoretical objectCorrespondence
Pooled subset-winner estimator, Eq. (11)Theorem 5.1; pooling bound (15)exact
Ungated order-jj rule, one-score pilot (D.2.4–D.2.5; Eq. (110) in D.2.7, D.3.7)upper rule (95) of Theorem B.6exact
Centered-gated order-jj rule (84), all pilot scores, T=8T=8 (D.2.4–D.2.6)centered comparisons and tail-local pilot (App. B.5.2, Lemma B.4)estimator exact; j≤4j\le4 and T=8T=8 fixed instead of the proof's j=⌊γD⌋j=\lfloor\gamma D\rfloor, T=12a+2E∗T=12a+2E _ \ast
Order-jj rule with the true marginal CDF (D.2.5)bias analysis with known HHdiagnostic only
Depth-two counting decision rule (D.2.7)three-band family indistinguishable at depth oneanalytic finite-NN bounds; not part of Theorem 5.2
Four-counter inverse (105) (D.2.1)estimator of Theorem C.1exact
Canonical depth-two rule (D.2.2)App. A.6.4, from Theorem 4.2exact
Plateau, inverse budget, block winners, rank fitsbaselines, not from our theoryn/a

Not implemented: the coarse gated estimator (77) of App. B.5.1 and the branch selection of App. B.6, both used by the proof only when D≥D0>39D\ge D _ 0>39; and the rank-additive estimator of the dense regime log⁡(eM)≤Bδ\log(eM)\le B\delta, which no grid cell reaches.

Dependency map (W2)

Appendix E of the revision gives a theorem-to-assumption and proof-dependency map (Tables 29–30). For each main result it lists the observation model, the response class, the lemmas used and where they are proved, classical versus new steps, and whether constants are explicit. For Theorem 5.2:

  • Upper bound. The pooling coupling (63) of App. B.3 in the bounded regime; the rank-additive estimator in the dense regime; Lemma B.2 (interval residual, giving E∗E _ \ast); coarse localization (App. B.5.1) and Lemmas B.4–B.5 (App. B.5.2; tail-local pilot, only two Hoeffding orders remain) for the correction branches; and the case split of App. B.6.
  • Lower bound. Two-band moment concealment (67) and the frozen-identity lift (66), both in App. B.3, plus Lemma B.3 (App. B.4; uniform endpoint approximation).
  • Independent results. Theorem C.1 and the A.7 certificates use none of these.
AuthorsOpenReview ↗

Response to Reviewer f9TX (3/3): a prospective check of Q1 and W1

Parts 1–2 rest on post hoc analyses of the frozen grid. To test their conclusions on cells they were not read from, we fixed protocols before running anything (rules, families, budgets and analyses) and simulated 3,060 new cells with fresh seeds. They vary what the grid held fixed: four score families with a declared Kolmogorov radius δ\delta (the grid's tail warp, a broad warp, a smooth warp, an endpoint rescaling), three responses (bit, threshold, quadratic), K=16,32,64K=16,32,64, depth B=2B=2–3232 and M=210M=2^{10}–2182^{18}, with the pilot and all MBMB candidates and labels charged and at least 8,192 repetitions per cell. New App. F reports them, and two package commands recompute every count and error below (summarize_prospective.py, about a second; order_two_map.py, about a minute).

DD<2<2[2,2.5)[2,2.5)[2.5,3)[2.5,3)≥3\ge3
order two beats pooling (cells with δ>0\delta>0)0 of 1152156 of 756183 of 456402 of 516
median RMSE, order two / pooling4.712.121.19.53
  • (a) holds out of sample. In none of the 1,152 cells with D<2D<2 did a correction beat pooling, whether its pilot used one score or all, whether it was tail-gated, and even with the true marginal CDF. With 16 random perturbation shapes drawn before running (2,946 more cells, 2,880 with δ>0\delta>0; K=16K=16–6464), the grid's order-two rule again never did (0 of 1,008 cells with D<2D<2). At D≥3D\ge3, order two won 78% of the four-family cells (91%, 79%, 66% at K=16,32,64K=16,32,64; 60% for quadratic responses, which leave pooling little bias to remove) and 62% of the random-shape cells.
  • A rule from DD. "Order two if D≥3D\ge3, otherwise pool" needs only (B,M,K,δ)(B,M,K,\delta), so it is fixed before sampling; with the threshold read off the frozen grid, it was no worse than pooling in 96% of the four-family cells and cut the mean RMSE over cells by 14% (from .030 to .026; the better of the two in each cell: .023). Where it lost to pooling, its median excess RMSE was .004; where it won, its median gain was .022. Each branch carries an explicit bound, (15) or Proposition B.7, and the latter is never weaker where D≥3D\ge3.
  • (b) holds. Along 576 trajectories at fixed (K,B,δ)(K,B,\delta), family and response, raising MM from 2102^{10} to 2182^{18} lifts order two from 25 to 296 wins; 271 losses become wins and no win becomes a loss.
  • W1: DD versus depth. Within each depth from 2 to 32, DD separates wins from losses (AUC .84–.93); over all cells the AUC is .90 for DD and .64 for depth. At D<2D<2, correction loses at every depth.
  • Q1: where Proposition B.7 is informative (App. F.2). Evaluated exactly over 686 budgets (B=2B=2–3232, K=8K=8–6464, M=210M=2^{10}–2222^{22}, Kδ=1/128K\delta=1/128–1/21/2), the bound is below .05 for Kδ≤1/16K\delta\le1/16 from M=216M=2^{16} (K=16K=16), 2182^{18} (K=32K=32) or 2202^{20} (K=64K=64) at B=4B=4, and it separates order two strictly from pooling's worst case in 62 budgets (D=2.0D=2.0–3.93.9). For M≥218M\ge2^{18} it certifies an expected error of .05 at radii 3–5 times those (15) allows. In all 6,080 simulated cells with δ>0\delta>0 of the grid and of these checks, measured order-two RMSE stays below the bound (largest ratio .39). Pooling's error likewise stays below its bound (15) in all 6,006 new cells (largest ratio .81), so the branch of Theorem 5.2 that our budgets instantiate holds as stated.

Together, these checks show that the conclusions of Part 1 do not rest on the grid's family, budgets or response pair.

In sum, your questions shaped the revision's main additions: Q1 led to an explicit finite-budget bound for the rule we simulate (Proposition B.7, mapped over 686 budgets) and to a rule from DD that is fixed before sampling and was tested on new cells; W1 led to the analysis of DD within every depth; Q2 and W2 led to the package and the dependency map. Each clause of W1 has its own answer: the logarithmic match, the implemented constructions and the natural scores in Part 2; the 129 of 140 cells and DD versus depth in Part 1; the choice between pooling and correction in the rule above. We are glad to clarify any point, and we will fix at once any script that does not run as documented.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank the three reviewers for their careful reading. They credit the paper with an exact characterization of the shallow-to-deep gap with a matching construction, which strengthens prior collision-existence results (f9TX S1, p59T), unusual care about scope (p59T), and apparent rigor (pjt8 S1).

Short answer to the paper's question (pjt8 Q2). In general, a shallow evaluation does not determine Best-of-KK accuracy at larger budgets, and we compute exactly how much it leaves open, with a pair of systems that attains it. With B=4B=4 candidates per prompt, even perfect knowledge of every shallow correctness pattern leaves a range of up to 60 percentage points for Best-of-16 accuracy, and of up to 99.7 points for Best-of-64 (Theorem 3.1). Narrowing this worst case requires additional structure, and the paper establishes what each kind buys: a bound on how fast correctness changes with score rank (ambiguity at most 4L/B24L/B^2); one retained joint statistic (monotone cubic responses); a score distribution shared across prompts (unbiased pooling), where approximate sharing limits precision through DD.

What each review asked and how it improved the paper. The PDF is uploaded with this response; changes are in blue, and all numbers in our replies refer to it. Items marked new are new results; the PDF proves the new statements, and scripts in the anonymized package recompute the new numbers.

  1. Accessibility outside this subfield (pjt8). The abstract states the deployment setting. The introduction starts from a concrete scenario (evaluate with B=4B=4 candidates per prompt, deploy Best-of-KK at K=16K=16–6464, where direct measurement needs 4–16 times more generation and scoring), then gives the answer above and four contributions with their implications. Section 3 opens with the worked example of Figure 1, which gets a reading guide, and each theorem is followed by one sentence on what it means for an evaluator. Table 2 maps every study to its claim, data, models, estimator and outcome; datasets and models are now in the main text, and Tables 4–5 give the protocols.
  2. What the approximate-sharing theory implies at finite budgets (f9TX Q1/W1).
    • New: Proposition B.7 gives the order-two rule we simulate an explicit finite-budget bound, with the pilot charged: in the introduction's setting (B=4B=4, K=16K=16), with 2182^{18} prompts and a declared δ=1/256\delta=1/256, its worst-case error is below .026.026, while pooling's exceeds .028.028. At our budgets, the rule of Theorem 5.2 itself is pooling, with the explicit bound (15); Section 5.2 now states this.
    • New: no correction beat pooling in any cell whose data-supported order is D<2D<2: neither in 173 such cells of three frozen studies nor in 1,152 new ones fixed in advance (four score families, three responses, K=16K=16–6464, B=2B=2–3232). At D≥3D\ge3, order two won 78% of the new cells, and within each depth DD separates wins from losses (AUC .84.84–.93.93). DD is computable before sampling; correcting only when D≥3D\ge3 cut pooling's mean error by 14%.
  3. Auditability (f9TX Q2/W2; pjt8 W4). New Appendix E maps every main result to its assumptions, the lemmas it uses, classical versus new steps and explicit constants; Table 6 maps each experimental rule to its theoretical object. New: a standard-library Python script re-derives every claim of the Appendix A.7 certificate from the data printed in the paper, in exact arithmetic, in about a second. The anonymized code package, linked from the supplementary material, also recomputes the tables of App. D.2.5, D.3.5, D.3.7 and F and the numbers we quote from App. D.2.4–D.2.6, from saved raw outputs and seeds, with a map from each paper item to its script.
  4. Three scope questions (p59T).
    • New: Theorem 4.2 holds for every slope bound, 2cmin⁡{L,2}/B2≤WBword,curve(L)≤min⁡{1,4L/B2}2c\min\lbrace L,2\rbrace/B^2\le W^{\rm word,curve} _ B(L)\le\min\lbrace1,4L/B^2\rbrace for all L>0L>0.
    • New: Proposition A.5 gives an explicit finite-KK bracket for the monotone class at every (B,K)(B,K), certified at the 25 pairs of Table 3 and narrowed by new certified witnesses (App. F.5).
    • Section 5.1 gives a three-step workflow for declaring and stress-testing δ\delta.

Natural scores (f9TX W1). The label-free test of Section 5.1 warns an evaluator, before any label is spent, when pooling is not justified. It rules sharing out on every natural bank we examined, including 164 public banks (GPT-4o mini, Claude 3 Haiku) and new sealed runs with 32B and 24B generators. As it warns, pooling then over-predicts Best-of-16 in most banks; in the sealed runs it does so by .15.15–.21.21, while plateau and inverse-budget extrapolation, which do not rely on sharing, stay within .04.04 (App. F). This concerns the recovery results of Section 5, not the exact limits of Section 3 or the response-structure results of Section 4.

We reply to each review individually and welcome follow-up questions.

AuthorsOpenReview ↗

Response to Reviewer p59T (1/2): Q1 and Q2

Thank you for the precise questions; each pointed at a place where the paper could say more, and the revised PDF now does:

  • Q1. Theorem 4.2 covers every L>0L>0 (p. 6).
  • Q2. Section 5.1 gives a three-step workflow for δ\delta (p. 8).
  • Q3. Proposition A.5 (App. A.7, p. 28; summarized in Section 4.2) bounds the monotone-class width at every finite (B,K)(B,K); certified witnesses (App. F.5) narrow it.

Q1: Theorem 4.2 for L>1L>1

Short answer. You are right that the range [0,1][0,1] is what limits the displayed pair. It affects only the dependence on LL, not the rate: B−2B^{-2} holds for every fixed L>0L>0, while the linear law L/B2L/B^2 is proved up to L=2L=2 and cannot continue for large LL, since the width is at most 1.

  • Upper bound. The derivative-duality bound (46) holds for every L>0L>0 (App. A.6.1). Hence WBword,curve(L)≤WBmar,curve(L)≤LλB≤4L/B2W^{\rm word,curve} _ B(L)\le W^{\rm mar,curve} _ B(L)\le L\lambda _ B\le4L/B^2.
  • Linear lower bound up to L=2L=2. The displayed pair keeps slopes in [L/4,3L/4][L/4,3L/4] to be strictly increasing, which costs range. Lemma A.2 with s=1s=1 applies equally to (L/2)(u±Jh2B(u))(L/2)(u\pm Jh _ {2B}(u)), which has slopes in {0,L}\lbrace0,L\rbrace and range [0,L/2][0,L/2]. Its depth-BB words agree and its gap is LΔ2B,K+1/(K+1)L\Delta _ {2B,K+1}/(K+1), twice that of the displayed pair. Hence WBword,curve(L)≥2cL/B2W^{\rm word,curve} _ B(L)\ge2cL/B^2 for 0<L≤20 < L\le2, with c=(1−2e−2)/36c=(1-2e^{-2})/36 from (8).
  • Larger LL. ML\mathcal M _ L grows with LL, so the value at L=2L=2 remains a lower bound.

Together, 2cmin⁡{L,2}B2≤WBword,curve(L)≤min⁡{1,4LB2}(L>0, B≥1),\frac{2c \min\lbrace L,2\rbrace}{B^2} \le W^{\rm word,curve} _ B(L) \le \min\Big\lbrace1,\frac{4L}{B^2}\Big\rbrace\qquad(L>0,\ B\ge1), which is ΘL(B−2)\Theta _ L(B^{-2}) for each fixed LL. The submitted App. A.6.2 already stated this order for L>1L>1, with the weaker lower bound c/B2c/B^2. For an evaluator, the cap need not be small: at B=8B=8, a declared L=2L=2 still leaves at most 12.5 points of ambiguity at every KK.

Beyond L=2L=2. Linear growth cannot persist: the upper bound is vacuous once L≥B2/4L\ge B^2/4, and WBword,curve(L)→1W^{\rm word,curve} _ B(L)\to1 as L→∞L\to\infty at fixed BB, since each pair in (9) (with κ=0\kappa=0) has bounded slope. Whether the width grows linearly on 2<L≲B22 < L\lesssim B^2 is open, as the revision now states.

Q2: how to select δ\delta

Short answer. Treat δ\delta as a sensitivity parameter, like Γ\Gamma in Rosenbaum-style sensitivity analysis: refute the radii the data rule out, compute the largest radius the target tolerance allows, and report the guarantee as conditional across the range in between; if no range remains, evaluate at depth KK. Only the recovery results of Section 5 depend on δ\delta; the characterizations of Sections 3–4 do not use it.

We agree: without further assumptions on the prompt population, δ\delta cannot be certified from data. We see this as a property of the problem, not a gap in the analysis. Repeated-score comparisons identify the average discrepancy Dcal≤δ2\mathcal D _ {\rm cal}\le\delta^2, not the uniform radius (Prop. B.1, App. B.2). In the two-type example there, Dcal→0\mathcal D _ {\rm cal}\to0 while the best uniform radius stays 1/81/8. Like the slope cap LL of Theorem 4.2, δ\delta is a modeling claim to be justified outside the data; the theory makes explicit what it buys, and the data can only refute it. The workflow:

  1. Refute. From nn unlabeled scores per prompt, the DKW–Bonferroni bound of App. B.2 gives a lower confidence bound δLCB≥0\delta _ {\rm LCB}\ge0. It refutes any smaller declared δ\delta, needs no labels, and is nontrivial only when n>2log⁡(2M/α)n>2\log(2M/\alpha).
  2. Tolerance. Compute the largest radius at which a bound still certifies expected error ϵ\epsilon. For pooling, (15) gives δϵ={ϵ−12(M−1+K/(MB))1/2}/(2K)\delta _ \epsilon=\lbrace\epsilon-\tfrac12(M^{-1}+K/(MB))^{1/2}\rbrace/(2K). This says how small δ\delta must be, not that it is; if δϵ<0\delta _ \epsilon<0, (15) cannot certify ϵ\epsilon even at exact sharing. For order two, new Proposition B.7 raises it: at B=4B=4, K=16K=16, M=218M=2^{18}, ϵ=.05\epsilon=.05, from Kδ=.023K\delta=.023 to .112.112.
  3. Decide.
    • If δLCB>δϵ\delta _ {\rm LCB}>\delta _ \epsilon, (15) cannot certify pooling at ϵ\epsilon, though pooling may still be accurate; evaluate at depth KK or use the ranges of Sections 3–4, which need no sharing. This is the case for our natural score banks: δ≥.268\delta\ge.268 at 95% confidence under each of nine scorers (App. D.3.5). This warning was borne out: in a post hoc comparison there, pooling over-predicts Best-of-16 accuracy by .15.15–.24.24 under seven of the nine scorers.
    • Otherwise, report the guarantee as conditional on δ≤δϵ\delta\le\delta _ \epsilon and show the results across [δLCB,δϵ][\delta _ {\rm LCB},\delta _ \epsilon], a sensitivity range, not a confidence interval. For example, with B=8B=8, M=4096M=4096, K=32K=32 and a declared δ=1/2048\delta=1/2048, (15) certifies an expected error of at most .049.049 with no pilot; on our grid, where δ\delta is known, the measured error there is .010.010 (App. D.2.5).
AuthorsOpenReview ↗

Response to Reviewer p59T (2/2): Q3

Q3: finite KK in the full monotone class

Short answer. Yes. Finite-KK statements hold for every (B,K)(B,K): exact at depth two and, beyond it, an explicit bracket whose lower end new certified witnesses raise considerably. Only the sharp large-KK value needs the limit (9).

Equation (9) is a K→∞K\to\infty statement for strictly increasing responses. At finite KK, the following holds for the full nondecreasing class, with W(B,K)W(B,K) and EBE _ B as in App. A.7.

  1. W(B,K)=0W(B,K)=0 for K≤BK\le B, and W(B,K)>0W(B,K)>0 for every K>BK>B. Theorem 4.1 gives exact word collisions in Hκ,L⊂H↑\mathcal H _ {\kappa,L}\subset\mathcal H _ \uparrow with unequal θK\theta _ K, though not quantitatively.
  2. Depth two is exact at every KK: W(2,K)=2E2(uK)W(2,K)=2E _ 2(u^K) (Corollary A.4), because depth-two words are equivalent to rank means.
  3. For all BB and KK there is an explicit bracket ℓB,K≤W(B,K)≤2EB(uK),ℓB,K=max⁡0<a≤12E[Δ2B,N+1N+1],N∼Bin(K,a).\ell _ {B,K} \le W(B,K) \le 2E _ B(u^K),\qquad \ell _ {B,K}=\max _ {0 < a\le1}2 \mathbb E\Big[\frac{\Delta _ {2B,N+1}}{N+1}\Big],\quad N\sim\mathrm{Bin}(K,a). The upper end is the equal-marginal benchmark. The lower end uses the nondecreasing pair equal to 00 below 1−a1-a and to v±Jh2B(v)v\pm Jh _ {2B}(v), v=(u−1+a)/av=(u-1+a)/a, above it. At a=1a=1, Lemma A.2 (s=1s=1) gives equal depth-BB words and the gap 2Δ2B,K+1/(K+1)2\Delta _ {2B,K+1}/(K+1), positive for K≥2BK\ge2B. For a<1a<1, the words stay equal: given that jj candidates fall above 1−a1-a, they are iid uniform there, so their labels follow the depth-jj word law of v±Jh2B(v)v\pm Jh _ {2B}(v), the same for both signs since j≤Bj\le B, and all labels below are zero; the gap is the corresponding Bin-average, as in (35). For B<K<2BB < K<2B the explicit lower end is zero; positivity there comes from item 1. Localizing keeps ℓB,K\ell _ {B,K} from decaying when K≫B2K\gg B^2.
  4. The construction behind (9) is itself quantitative at finite KK. With κ=0\kappa=0, its ramp near u=1u=1 gives W(B,K)≥(1−3ε){1−(1−a/2)K}−O(a2−K)W(B,K)\ge(1-3\varepsilon)\lbrace1-(1-a/2)^K\rbrace-O(a2^{-K}) for every aa below a threshold. The threshold and the OO-constant depend on BB and ε\varepsilon through a non-explicit contraction argument. Hence W(B,K)→1W(B,K)\to1 as K→∞K\to\infty, as does the upper end, but without an explicit rate.

To sharpen the explicit bracket of item 3, we also certified explicit witnesses (new App. F.5). For each pair, a numerical search found two nondecreasing step responses (at most 60 steps) with equal depth-BB word laws; a Krawczyk (interval Newton) test in outward-rounded arithmetic proves that an exact equal-law pair lies within 10−3010^{-30} of them, and encloses the gap. Equal laws follow from equal signature coordinates on Lyndon words of length ≤B\le B (shuffle identities). Where the width is known (depth two), the witnesses come within .003 of it. Lower ends and witnesses are truncated; upper ends are rounded up, pinned to 10−2210^{-22} by a de la Vallée Poussin alternation.

(B,K)(B,K)(3,16)(4,16)(4,64)(4,128)(5,32)
explicit lower end.0251.0044.0228.0227.0065
certified witness.1479.0331.2490.4131.0349
2EB(uK)2E _ B(u^K).2950.1553.5121.6677.2171

At (4,16)(4,16) the witness improves the A.7 certificate (.0292.0292). For an evaluator, restricting to monotone responses thus cuts the worst-case ambiguity at (4,16)(4,16) from 60 points (Theorem 3.1) to between 3.3 and 15.6; at depth three it stays between 14.7 and 29.5 points at K=16K=16.

The revision states items 1–4 as Proposition A.5, adds the bracket for the 25 pairs B=2,…,6B=2,\ldots,6, K∈{8,16,32,64,128}K\in\lbrace8,16,32,64,128\rbrace (Table 3) and their certified witnesses (Table 37), and summarizes the bracket in one sentence in Section 4.2. Our anonymized package, linked from the supplementary material, reproduces both tables in exact and interval arithmetic, and checks to 50 digits, for B=2,…,5B=2,\ldots,5 at several parameter values, that the pairs used here and in Q1 have equal depth-BB word laws, unequal depth-(B+1)(B+1) laws and the stated gaps.

Presentation and takeaways. Beyond your questions, the revision (changes in blue) targets the passages that require effort. The introduction opens with the deployment setting and states the paper's answer and four contributions. Each theorem is followed by one sentence on what it means for an evaluator. A table in Section 6 maps every study to the claim it tests, and a new appendix gives a proof roadmap with a dependency map.

Thank you again for these questions; answering them made the paper's scope statements more complete: Theorem 4.2 now holds for every L>0L>0, Section 5.1 gives a workflow for δ\delta, and Proposition A.5 brackets the monotone-class width at every finite (B,K)(B,K), with certified witnesses at 25 pairs. We are glad to clarify any of these further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.