acceptodds
Under review as a conference paper at ICLR 2027

Coverage Is Not Selection: Exactly Attributing the Gains of Test-Time Scaling

Abstract

Test-time scaling methods (self-consistency, best-of- under a trained verifier, adaptive sampling) are compared by one number: accuracy at budget . Per question that number is an exact product: coverage , the probability a correct answer exists among samples, times selection, the probability of returning it given it exists; the first depends on the sampler, the second on sampler and rule. We turn it into an audit instrument: a remainder-free identity splitting any accuracy difference into base-rate, headroom and selection channels; an efficiency normalized between a content-blind floor and the oracle ceiling; and unbiased estimators pricing any rule at any budget from one pool of cached samples. On 149,000 generations from six models it does three things accuracy alone cannot. It attributes: a token cap and 63 hours at a fixed configuration leave accuracy unresolved at yet move the selection channel (cap: ; 63 hours: at , , and , , at ; one unreplicated pair). It prices verifiers: a 7B process reward model is the best selector we measure, yet selectors lose efficiency as the budget grows on all 9 (cell, selector) pairs (6 resolved, uncorrected), and how fast a vote loses it tracks how concentrated its per-sample weights are (Spearman with effective sample size, 17 pairs). It audits adaptive schedules at matched expected budget: adaptive-consistency gains on the three cells with ample coverage ( to ), though which channel pays is unresolved. All code, per-cell results and cached pools behind every number here are released at https://github.com/xxxxx/xxxxx (URL anonymized for review; public upon acceptance); code and results also accompany this submission, the pools being too large for it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time scaling, best-of-n, self-consistency, verifiers, coverage, selection efficiency, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_a6sbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies test-time scaling, and argues that the accuracy at budget n is not sufficient to paint the full picture on its own. It then proposes a decomposition of the accuracy gap into three components called base rate, headroom and selection channels, each identifying different causes of the gain. They argue that this decomposition allows attributing the gains to different components, it allows evaluating the verifiers and compares adaptive schedules for an expected budget.

Critical strengths

Analysing the test-time scaling behaviour to better understand the impact of different components is certainly valuable. While it is not surprising that looking at a single average number quantifying the average accuracy of a method does not provide complete information, the paper tries to identify different components and quantify their gains separately. There have been works aiming to better understand the interactions between the generator, verifier and sampler in the pipeline, but I do not know if there are other prior works following the similar approach in the paper.

Critical weaknesses

The main critical weakness of the paper is that it is very difficult to read and understand. I believe the text has been heavily AI-generated. The authors acknowledge the use of AI in polishing their text, and that they have reviewed all AI-generated content, but it looks to me the other way around - at least this is the impression I got from reading it over and over again.

There are many sentences that are hard to decipher, and some variables and terms are not defined. I'll give some examples, but there are many more. The Abstract states: "We turn it into an audit instrument: a remainder-free identity splitting any accuracy difference into base-rate, headroom and selection channels; an efficiency σR(n) normalized between a content-blind floor and the oracle ceiling; and unbiased estimators pricing any rule at any budget n≤N from one pool of N cached samples. " At this point, it's not clear what the identity is, or what the base-rate, headroom or selection channels refer to; these have not been defined yet. "content-blind floor" or "oracle ceiling" are kind of understandable once one reads through the paper, but they're not obvious to a reader at this point. Also, "the rule" is not defined yet, it becomes obvious much later in the paper as the "selection rule". What are the "cached samples" and estimating and pricing what exactly? The following sentences summarize the results, but I really cannot understand what they're trying to convey: it writes "move the selection channel" while the channel has not been defined, it uses variables \sigma and Q, but I don't know what they represent, it refers to 63 hours, but I don't know what that time means in this context. There are many many more.. Basically, after reading the Abstract, I had no idea what the paper really does.

Some of it became more clear as I read the rest of the paper, but not much more. Introduction is similarly reads very heavily AI-generated. Second paragraph of Introduction says: "A reported accuracy difference can therefore be dominated by a channel other than the one the claim is about, and in configurations we exhibit it is: T=0.6 →1.0 at fixed rule and budget moves accuracy +0.2 points, a null superposing a 1.4-point base-rate loss, +1.8 points of headroom and−0.2 of selection (§7). This paper turns that observation into a measurement theory with three parts."

This reads to me very much like Claude-generated, which is fine in itself, as long as I could understand what it said: What is a channel? What is the configuration? What is T? What is "null"? base-rate loss, headroom and selection have not been defined yet.

Next paragraph: "Per-question accuracy factorizes exactly as AR,q(n) = Cq(n) sR,q(n). Writing aggregate accuracy as A= B+ σH, with a content-blind floor B, headroom H(n) = C(n)−B, and a normalized selection efficiency σR(n), 0 for any rule ignoring sample contents and 1 for an oracle, gives a symmetric, remainder-free identity splitting the accuracy difference between any two configurations into base-rate, headroom and selection channels (Theorem 1). The algebra is elementary; only the third channel is evidence about selection. "

None of the variables above have yet been defined. What is the "third channel"?

I can continue like this for the whole paper. This style not only makes it very difficult to understand what the paper actually does, but also puts the reader off early on. By the end of Introduction, I already had little motivation to continue.

I did continue reading, however, and things got a bit clearer when I read the Setup, and managed to decipher the setting to some extent. Even there, there are so many undefined variables, terms and concepts. As yet another example, I believe one of the main quantities, which is later reported in the Tables, A_{MV}(32), has never been defined in the paper. One has to make many assumptions, and go back-and-forth in the paper to understand most of the sentences and claims.

In the end, based on what I gathered, the paper proposes to decompose the gain in test-time-accuracy in order to be able to attribute it to different components, e.g., the sampler, verifier, and selector. While this is certainly valuable, it doesn't seem very novel. Moreover, I wasn't really able to understand what is actually being presented as results. There is just too many unclear terms and convoluted Claude-style sentences even in the Results section.

I believe the paper needs to be completely rewritten in human-understandable language, with the aim of making its setup and core contributions clear to any reader.

Questions for authors

I don't have specific technical questions. I could list all the undefined variables and terms in the paper and ask "What is A_MV?", "What is wrong-mode rate?", "What is a cluster?", "What is \bar{\sigma}?", but there are just too many of them.

3 comments

Thank you for persisting with the paper and for the specific examples; they were the most useful guide to the rewrite. We agree the submitted text was too dense and used terms before defining them. The main text is rewritten.

A short reading path:

  1. Pages 1–2 state the problem.
  2. Table 1 works one example by hand.
  3. Table 2 defines every term.
  4. Table 4 and Figure 2 carry the two main new results.

Where each term you list is now defined:

  • "rule": the procedure that returns one of the n samples (§1, Table 2).
  • "cached samples" / "pricing": a pool of N samples per question, generated once (Table 2). "Pricing" is now "evaluating": one pool evaluates every rule at every budget n ≤ N (§4).
  • "content-blind floor" / "oracle ceiling": the accuracy of a random pick (B) and of an oracle (C(n)) (Table 2, Lemma 1).
  • "channels" / "third channel": the base-rate, headroom and selection components of an accuracy difference, defined in eq. (2) with what each measures.
  • σ, Q: σ is the selection efficiency, the share of the gap between a random and an oracle pick that a rule closes (eq. 1, Table 1); Q is the number of questions.
  • "63 hours": the interval between two runs of the same configuration against a served API (§6.1).
  • T: temperature.
  • "null":replaced by plain wording (“not significant”, “finds nothing”). The word now appears only in “null hypothesis” and the standard deviation under it.
  • A_MV(32): accuracy of majority voting at budget 32.
  • "wrong-mode rate": share of solvable questions whose most frequent answer is wrong.
  • "cluster": samples whose final answers are mathematically equivalent.
  • σ̄: the average of σ over the two configurations being compared (§3).

Novelty. We do not claim novelty for the conditional-probability factorization itself. Coverage (Brown et al., 2024) and the gap between coverage and voting accuracy (Bay & Yearick, 2026) are prior work, and we build on both. Our contribution is the audit and estimation framework built on them:

  • a calibrated efficiency;
  • an exact split of any accuracy difference;
  • a split of efficiency changes into within-question and reweighting parts;
  • unbiased pool-based estimation for any rule, with exact computation for majority voting.

We also report findings the framework makes explicit. A verifier's efficiency carries over across models, so its gain can be predicted before running it. And the reward model's loss of efficiency with budget is mostly a reweighting effect.

We hope the revised version makes these readable.

Thank you again for the detailed examples. With the discussion closing tomorrow, we would be grateful if you could look at the rewritten first three pages: the abstract, the introduction, and Tables 1 and 2. Every term you listed is now defined there before it is used.

If any passage is still hard to follow, a pointer to it would help us a great deal. If the text now reads clearly, we would appreciate it if you would consider updating your assessment.

I thank the authors for their responses. While the writing has substantially improved, I should say that the overall language still sounds very much AI-style. I believe the revision was also mostly done with AI. This is fine in itself, but the result is still not a very smooth read.

However, more importantly, I find the overall claim and the results very limited in novelty. It's quite clear that the overall accuracy depends on multiple factors, which the paper highlights; but it doesn't really provide any guidelines for any improvements in practice. It's not clear why Theorem 3.1 is a theorem. It just makes some simple statements. Overall, I don't think the paper makes the cut for publication at this venue. I will retain my score.

Reviewer_hzdROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper tries to understand where accuracy gains in test-time scaling actually come from. Its central observation is that when a model answers a question n times and a rule must return one answer, the system can fail either because no sample is correct or because a correct sample exists and the rule fails to pick it. The author introduce a rather simple process of determining a selection productivity or sigma that measures the fraction of the gap between random picking and an oracle that basically is the best conversion. They then give an identity that splits the accuracy difference between any two configurations exactly into a base-rate, a headroom and a selection component. They also provide unbiased estimators that price any rule at any budget. The authors run experiments over Math benchmarks and find that selection methods and base models behave quite differently on their metric. Verifier-based selectors also lose efficiency as the budget grows, whereas majority voting does not.

Critical strengths
  1. Separating whether a correct answer is generated from whether it is selected is sensible. The pool-reuse estimators let every selection rule be evaluated at every budget from one set of generations at no extra cost. This is the most practically valuable part of the paper and could benefit evaluation practice.

  2. The experiments are rigorous and honest. They use permutation tests, bootstrap intervals, and multiple-testing corrections.They openly state their own limits. Usually this is rare annd should be anppreciated.

Critical weaknesses
  1. In my opinion this is a diagnostic metric, which is good in itself but its utility is narrow. sigma does not change any decision a practitioner would make. To choose a model, budget, temperature or rule, one still picks the configuration with the highest A, and sigma never affects or impacts that choice. In the appendix e3 they admit this that their machinery adds little. The paper into is also over-formalized relative to what it delivers. Theory is not always bad, but in this paper it acts against it because much of the heavy lifting is done the original sigma formulation and the Shapley based observations actually do not add much value. Rather the paper could have explored other metrics of priductiveity with simpler formulation, rather than over formalizing.

  2. The paper is very hard to read and needs to be rewritten. Terms are used before they are defined even in the intro. The paper general is heavily edited by AI and it is distracting with all the formalized. The paper should lead with the simple diagnostic and a small worked example (one question, a few samples, showing a,,b,c and sigma), state each finding cleanly with its strength of evidence, and move or cut the heavier theory. In my opinion it should be rewritten heavily for better understabaility.

Questions for authors
  1. Can you show a concrete case where your selection efficiency, or the three-way split, leads a practitioner to a different choice of model? Without such a case, I find it hard to see the practical utility of the diagnostic beyond accounting.

  2. By your own analysis in the appendix, the decomposition never detects a change that the simple differences in base rate, coverage and the coverage–accuracy gap do not already reveal. What does the Shapley-based framing contribute beyond reporting these three differences directly?

2 comments

Thank you for the frank review. We agree with much of it and have changed the paper accordingly.

"σ does not change any decision." For choosing among configurations one has measured, at the budget one measured, we agree. At a fixed sampler and budget, σ ranks rules exactly as accuracy does, and the revision says so in §1 and §2. The measurements matter when a decision depends on something not yet measured.

Q1, a concrete case. Should a practitioner pay to run a 7B PRM over all samples of a new model? Scoring costs GPU time comparable to generation.

  • On GPT-4.1-nano and Qwen-turbo the PRM adds 0.1–0.8 points at n = 16; on Llama-3.2-3B, Llama-3.1-8B and Phi-4-mini it adds 4–10 points.
  • The PRM's σ is similar across models (0.59–0.70 on 12 of 13 pools), unlike accuracy. So B + σ̄_V·H − A_MV, with σ̄_V its mean efficiency on other models and B, H, A_MV from the new model's labeled but unscored samples, predicts the gain.
  • Leaving one model out at a time, this prediction has a mean absolute error of 1.6 points, against 2.6–3.7 for three alternatives.
  • The best alternative is the gain as a share of the gap C − A_MV. It equals (σ_V − σ_MV)/(1 − σ_MV), so it also normalizes by headroom. With each model counted once, the margin over it is small (1.6 against 1.8).
  • Both normalized predictors make all 13 deploy calls at a 2-point threshold. At 1, 3 and 5 points, σ is right on 13, 12 and 10 pools and the share of the gap on 11, 9 and 5. Unnormalized predictors, such as the average gain, would have recommended scoring both API models.
  • The prediction fails for Mistral-7B (6.7 points predicted, 2.2 observed) and for Math-Shepherd (Table 4, App. F.8).

You asked specifically about the choice of model. In our pools no model pair changes order between voting and PRM selection (108 matched-pool comparisons: 36 pairs of pools from different models, each at n = 4, 8 and 16), so we cannot show a reversal of model choice. The decision the measurement informs is whether scoring is worth its cost.

Q2 (beyond ΔB, ΔC and Δ(C − A)). Like any function of (B, C, A), the aggregate split cannot detect a change that the raw differences miss. What it adds is how ΔA divides among them.

  • The gap equals H(1 − σ), so its change mixes headroom and selection. In the token-cap contrast the gap grows by 15.2 points: 7.4 from more headroom at unchanged efficiency and 7.8 from lower efficiency. The gap alone does not distinguish the two.
  • The new split (Theorem 1c) uses per-question values, which the aggregate triple does not contain. It shows, for instance, that reweighting accounts for most of the PRM's loss of efficiency with budget.
  • The Shapley discussion is now one sentence in the main text, with the argument in App. A.

Over-formalization and simpler metrics. We followed your suggestion: the paper now leads with the diagnostic and a worked example (Table 1, two questions with b, c, a and σ). Each finding is stated once with its evidence, and the heavier theory is in the appendix. σ = (A − B)/(C − B) is the simplest summary that is 0 for a random pick and 1 for an oracle. The gap C − A and the ratio A/C take the values C − B and B/C for a random pick, so they move with the sampler even when the rule ignores the samples (§2).

Thank you again for the frank review. With the discussion closing tomorrow, we would like to ask whether the revision addresses your two main concerns.

  • Utility: §6.4 and Table 4 give a decision the measurement informs, namely whether to pay for scoring a new model with a verifier. Leaving one model out at a time, the σ-based prediction has a mean absolute error of 1.6 points and makes the right call on all 13 pools at a 2-point threshold.
  • Readability: as you suggested, the paper now leads with the diagnostic and a worked example (Table 1), and the Shapley argument and heavier theory are in the appendix.

If the revision changes your view, we would be grateful if you would consider updating your score. If not, we would value knowing what still falls short.

Reviewer_YNhNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper separates accuracy into a single-sample floor, coverage headroom, and selection efficiency. It derives an attribution identity and estimators based on cached response pools, including exact plurality evaluation. Experiments audit decoding changes, endpoint variation, verifier behavior, and adaptive sampling.

Critical strengths
  • The distinctive contribution is the accounting framework. It helps explain how configuration changes can have opposing effects: in the cap experiment, coverage rises from 0.383 to 0.583 while selection efficiency falls from 0.60 to 0.33, despite an unresolved accuracy increase.

  • The estimation methodology adds useful statistical machinery. Unbiased subset estimation, the UMVUE argument, and exact dynamic programming for count-only plurality provide more than the general practice of fitting budget curves from response pools.

  • The empirical and statistical checks are thoughtful. They include PRM interface validation, matched-budget adaptive comparisons, supplementary non-mathematical tasks, ratio-bias diagnostics, and appropriate treatment of cases with little headroom.

Critical weaknesses
  • The paper should discuss PETS (Liu et al., 2026). Its finite-budget population-vote objective and heterogeneous allocation framework are relevant to §5 and §7.2. These are shared principles; this paper’s separate contribution is its correctness-based accounting and estimation framework.

  • The mathematical ingredients largely draw on established factorization, symmetric decomposition, and statistical estimation tools. The contribution is therefore primarily their integration and empirical use. Its value depends on the additional interpretation or decisions that this integration enables.

  • The attribution is descriptive rather than inherently causal. Selection efficiency can decline with an unchanged selector because the sampler adds harder problems to the available headroom. A negative selection channel alone does not establish deterioration of the selection rule.

  • Some evidence remains exploratory or difficult to verify. Endpoint drift rests on one pair of sessions, several selector trends depend on the budget window, and the unavailable raw pools and scorer outputs prevent independent reproduction of key tests.

Questions for authors
  • How should practitioners interpret a negative selection channel when the selector is fixed but coverage expands to harder questions?

  • Can the authors give a concrete decision that the framework changes relative to inspecting accuracy, coverage, and their raw gap together?

  • Can anonymized pools, scorer outputs, and timestamps for the central cap and drift contrasts be made available during review?

4 comments
AuthorsOpenReview ↗

Reviewer YNhN (1/2)

Thank you for the detailed and constructive review.

Q1 and "descriptive, not causal." You are right, and the revision builds this in. Theorem 1(c) (the Kitagawa rate–composition split) writes σ as a headroom-weighted average of per-question efficiencies. Any change in σ then splits exactly into (i) changes in efficiency on individual questions and (ii) headroom moving between questions.

A negative selection component should be read together with this split. Neither part is causal by itself:

  • At a fixed sampler, part (i) describes how the rule's choices change on the same questions.
  • When the sampler changes, each question's answer distribution changes too. Part (i) can then move while the rule stays the same.

The split distinguishes three cases in the paper:

  • Token cap (nano). 13 questions gain headroom, most of them wrong-moded. At n = 16 the composition part is −0.24 [−0.40, −0.09], and the within part is not significant (−0.03 [−0.17, +0.08]).
  • 63-hour drift (same configuration). At n = 16 the within part is significantly negative (−0.081 [−0.161, −0.003]); the composition part, of comparable size (−0.094), is not significant.
  • Budget trends. Reweighting lowers σ in 54 of 56 (pool, rule) cases, 48 significantly (uncorrected). The PRM selector's within part is never significantly negative, and voting's rises by 0.09–0.35. Reweighting therefore accounts for most of the PRM's decline.

PETS. Thank you for the pointer. We discuss PETS in §7 and evaluate an offline re-implementation of its online allocator in §6.3. Following PETS, a difficulty prior is learned from other questions (here, the other questions of the cell); the reported budget excludes those samples, which if anything favors PETS.

  • At the same mean budget as fixed voting, it gains 0.6–1.0 points on the three high-coverage MATH cells.
  • On low-coverage cells, neither adaptive-consistency nor PETS changes accuracy by more than 0.7 points.
  • The headroom component is significant on all three cells. The selection component is of similar size but not significant, so the split does not settle which component pays.
  • PETS raises agreement with the full-pool plurality (a finite-pool stand-in for its target) by 1.3–2.7 points, more than its accuracy gain. Agreement can rise on questions whose plurality is wrong, where it earns no accuracy.

As you note, PETS optimizes allocation, while our framework accounts for where accuracy comes from.

"Primarily integration." We agree. §6.4 gives three uses of the measurements; the verifier-gain prediction (Q2 below) is tested out of sample.

AuthorsOpenReview ↗

Reviewer YNhN (2/2)

Exploratory evidence.

(a) Drift on one pair. GPT-4.1-nano has been retired, so the exact pair cannot be repeated. We ran GPT-4.1-mini on the same AIME configuration with 64 samples per problem per session. This allows the within-session control at the same budget, on the same kind of cell, that the original design lacked.

  • Within session D, we detect no significant difference between the earliest and latest 32 samples (Δσ = +0.008, p = 0.97). Its null s.d. (0.181) is larger than that of the original nano comparison (0.133).
  • Nor between D and a second session 39 hours later (64 + 64 samples; Δσ_MV(32) = +0.073, p = 0.44; null s.d. 0.092). On that scale, nano's −0.36 would lie 3.9 null s.d. from zero.

We therefore do not claim that served models drift routinely. The nano change is one observed event, which we present as a reason to re-measure the sampler when comparing across time (§6.1, §8).

The replication also shows that the token-cap effect is model-dependent. On mini the longer cap raises accuracy by 16–19 points, with no significant change in σ, because the newly solvable problems are mostly right-moded.

(b) Window dependence. App. F.7 reports the sign over all 21 budget pairs and one fixed slope summary, which we chose after seeing the windowed results. The direction of the selector trends holds over most windows on every pool (at least 16 of 21 for the PRM). Their size depends on the window, and the (8, 16) family contrast is not significant on any cell.

(c) Reproduction. See Q3.

Q2 (a decision beyond A, C and the gap). The clearest decision is whether to pay for scoring a new model with a verifier. The gap C − A bounds what any verifier can add. The verifier's σ, measured on other models, predicts what it will add. At n = 16:

  • The PRM adds 0.1–0.8 points on GPT-4.1-nano and Qwen-turbo, and 4–10 points on the Llama and Phi models.
  • The σ prediction matches both. Its mean absolute error is 1.6 points, the lowest of four predictors, and it makes the right call on all 13 pools.
  • The share of the gap, which also normalizes by headroom, comes close (1.8 points when models are weighted equally). Unnormalized predictors have about twice the error.

Q3 (data during review). Yes. The updated supplementary material includes:

  • every AIME pool behind the cap and drift contrasts, with per-request timestamps;
  • the GPT-4.1-mini sessions;
  • all PRM, Math-Shepherd and judge score files, with a manifest and checksums.

drift_permutation.py, composition_split.py and mini_replication.py re-run the central tests from raw samples on a CPU in a few minutes each, at the permutation counts given in the README (the paper's 10^5 permutations take about 35 minutes). The MATH pools (about 270 MB) exceed the limit and will be released with the camera-ready. Every MATH number is in the shipped results.

Thank you again for the constructive review. As the discussion period closes tomorrow, we would like to check whether the revision answers your three questions:

  • how to read a negative selection component when coverage expands: the within-question/reweighting split (Theorem 1(c), Figure 2);
  • a decision beyond accuracy, coverage and the gap: whether to pay for scoring a new model with a verifier, tested out of sample (§6.4, Table 4);
  • data during review: the AIME pools with timestamps and all score files are in the supplementary material, with scripts that re-run the cap and drift tests from raw samples, and the drift test is repeated on GPT-4.1-mini (App. F.5).

If anything is still unclear or missing, we are glad to answer before the deadline. If these address your concerns, we would appreciate it if you would consider updating your assessment.

Thank you for the detailed rebuttal and substantial revisions. I appreciate the additional theoretical analysis, empirical validation, and discussion of PETS, which have largely addressed my previous concerns.

However, I remain concerned about the limited novelty. In particular, the central insight of analyzing where test-time scaling gains come from has already been explored in related work such as PETS, which studies the relationship between finite-budget voting, population-level behavior, and correctness. The proposed coverage–selection decomposition appears to be a relatively modest extension of these existing insights, rather than a substantial conceptual advance. While the verifier-gain prediction demonstrates some practical utility, its improvement over simpler baselines is also limited.

Overall, I appreciate the authors' efforts and will maintain my current rating of Weak Accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers for careful and candid reviews. Two concerns were shared: the paper was hard to read, with terms used before they were defined, and it was unclear what the decomposition adds to accuracy. Two reviewers asked for a rewrite, and we have uploaded a revised PDF and supplementary material.

The original measurements are unchanged. New analyses refine their interpretation; in particular, the reward model's aggregate loss of efficiency as the budget grows turns out to be mostly a reweighting of headroom toward harder questions. New content (analyses, results, tables, figures, definitions and the worked example) is marked in blue. Because the main text was rephrased throughout, rephrasing alone is not marked.

Where the original sections went.

  • §1 Introduction → §1, with a worked example (Table 1).
  • §2 Related work → §7 (PETS added).
  • §3 Decomposition → §2–3, with a table of terms (Table 2) and Theorem 1(c).
  • §4 Estimation → §4.
  • §5 Vote asymptotics → App. D.
  • §6 Experimental setup → §5.
  • §7 Results → §6.1 (sampler changes), §6.2 (verifiers), §6.3 (adaptive allocation); §6.4 (three uses of the measurements) is new.
  • §8 Discussion → §8.

Readability. The abstract and introduction now define every term before using it. Table 1 works one example by hand: two questions, with single-sample accuracy, coverage, voting accuracy and σ at n = 4 and 16. Each finding is stated in one sentence before its evidence, and theory that carries no finding is in the appendix. Two reviewers found the prose compressed and machine-like. We agree it was over-compressed, and we rewrote it in plain sentences. A consistency scan shipped with the code (code/consistency_scan.py) checks the revised PDF against the results files; run from the supplementary material it reports 243 passed checks and 2 skipped because their inputs are not shipped.

Data. The supplementary material now includes every AIME sample pool with request timestamps, covering the token-cap and 63-hour contrasts and the new GPT-4.1-mini sessions. It also includes every PRM, Math-Shepherd and judge score file, with a manifest and checksums. The MATH pools (about 270 MB) exceed the supplementary limit and will be released with the camera-ready.

What the measurements add (hzdR, YNhN). At a fixed sampler and budget, σ ranks rules exactly as accuracy does, and we now say so in §1. §6.4 describes three uses for settings one has not measured. The first is tested out of sample.

A process reward model's σ lies between 0.59 and 0.70 on 12 of the 13 MATH pools it scores. Its gain on a model it has not scored can therefore be predicted from that model's labeled but unscored samples, as B + σ̄_V·H − A_MV, with σ̄_V the verifier's mean efficiency on the other models. Leaving one model out at a time:

  • The mean absolute error is 1.6 points, against 2.6–3.7 for three predictors that do not use σ.
  • The best of those three is the gain as a share of the gap C − A_MV, which also normalizes by headroom. When each model counts once, the margin over it is small (1.6 against 1.8).
  • At a 2-point threshold, both normalized predictors make the right deploy decision on all 13 pools. At 1, 3 and 5 points, the σ prediction is right on 13, 12 and 10 pools, and the share of the gap on 11, 9 and 5.
  • The prediction fails for Mistral-7B (6.7 points predicted, 2.2 observed) and for Math-Shepherd, whose σ is not stable across samplers (Table 4, App. F.8).

Within-question changes and reweighting (YNhN). Theorem 1(c), the rate–composition split of Kitagawa (1955), divides any change in σ into two parts: changes in efficiency on individual questions, and headroom moving between questions. Applied to the budget trends, the split shows that reweighting accounts for most of the PRM selector's loss of efficiency. Its headroom-weighted within-question part is never significantly negative, while majority voting's rises by 0.09–0.35 (Figure 2, App. Table 14).

Other additions.

  • PETS is discussed (§7) and evaluated at matched budget with the decomposition (§6.3, Table 5, App. F.11).
  • Budget trends are now summarized over all 21 budget pairs and by one fixed slope (App. F.7).
  • The token-cap contrast and the session test are replicated on GPT-4.1-mini, since GPT-4.1-nano has been retired (§6.1, App. F.5):
    • Within one session, we detect no significant difference between the earliest and latest 32 samples per problem (Δσ = +0.008, p = 0.97).
    • Nor between two sessions 39 hours apart (Δσ = +0.073, p = 0.44).
    • On mini, the longer token limit raises accuracy by 16–19 points, through the base rate and headroom, with no significant change in σ. The problems it makes solvable are mostly right-moded for mini (wrong-mode rate 0.17 → 0.16) and were mostly wrong-moded for nano (0.22 → 0.46).

The main text is within the 10-page limit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.