- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies test-time scaling, and argues that the accuracy at budget n is not sufficient to paint the full picture on its own. It then proposes a decomposition of the accuracy gap into three components called base rate, headroom and selection channels, each identifying different causes of the gain. They argue that this decomposition allows attributing the gains to different components, it allows evaluating the verifiers and compares adaptive schedules for an expected budget.
Critical strengths
Analysing the test-time scaling behaviour to better understand the impact of different components is certainly valuable. While it is not surprising that looking at a single average number quantifying the average accuracy of a method does not provide complete information, the paper tries to identify different components and quantify their gains separately. There have been works aiming to better understand the interactions between the generator, verifier and sampler in the pipeline, but I do not know if there are other prior works following the similar approach in the paper.
Critical weaknesses
The main critical weakness of the paper is that it is very difficult to read and understand. I believe the text has been heavily AI-generated. The authors acknowledge the use of AI in polishing their text, and that they have reviewed all AI-generated content, but it looks to me the other way around - at least this is the impression I got from reading it over and over again.
There are many sentences that are hard to decipher, and some variables and terms are not defined. I'll give some examples, but there are many more. The Abstract states: "We turn it into an audit instrument: a remainder-free identity splitting any accuracy difference into base-rate, headroom and selection channels; an efficiency σR(n) normalized between a content-blind floor and the oracle ceiling; and unbiased estimators pricing any rule at any budget n≤N from one pool of N cached samples. " At this point, it's not clear what the identity is, or what the base-rate, headroom or selection channels refer to; these have not been defined yet. "content-blind floor" or "oracle ceiling" are kind of understandable once one reads through the paper, but they're not obvious to a reader at this point. Also, "the rule" is not defined yet, it becomes obvious much later in the paper as the "selection rule". What are the "cached samples" and estimating and pricing what exactly? The following sentences summarize the results, but I really cannot understand what they're trying to convey: it writes "move the selection channel" while the channel has not been defined, it uses variables \sigma and Q, but I don't know what they represent, it refers to 63 hours, but I don't know what that time means in this context. There are many many more.. Basically, after reading the Abstract, I had no idea what the paper really does.
Some of it became more clear as I read the rest of the paper, but not much more. Introduction is similarly reads very heavily AI-generated. Second paragraph of Introduction says: "A reported accuracy difference can therefore be dominated by a channel other than the one the claim is about, and in configurations we exhibit it is: T=0.6 →1.0 at fixed rule and budget moves accuracy +0.2 points, a null superposing a 1.4-point base-rate loss, +1.8 points of headroom and−0.2 of selection (§7). This paper turns that observation into a measurement theory with three parts."
This reads to me very much like Claude-generated, which is fine in itself, as long as I could understand what it said: What is a channel? What is the configuration? What is T? What is "null"? base-rate loss, headroom and selection have not been defined yet.
Next paragraph: "Per-question accuracy factorizes exactly as AR,q(n) = Cq(n) sR,q(n). Writing aggregate accuracy as A= B+ σH, with a content-blind floor B, headroom H(n) = C(n)−B, and a normalized selection efficiency σR(n), 0 for any rule ignoring sample contents and 1 for an oracle, gives a symmetric, remainder-free identity splitting the accuracy difference between any two configurations into base-rate, headroom and selection channels (Theorem 1). The algebra is elementary; only the third channel is evidence about selection. "
None of the variables above have yet been defined. What is the "third channel"?
I can continue like this for the whole paper. This style not only makes it very difficult to understand what the paper actually does, but also puts the reader off early on. By the end of Introduction, I already had little motivation to continue.
I did continue reading, however, and things got a bit clearer when I read the Setup, and managed to decipher the setting to some extent. Even there, there are so many undefined variables, terms and concepts. As yet another example, I believe one of the main quantities, which is later reported in the Tables, A_{MV}(32), has never been defined in the paper. One has to make many assumptions, and go back-and-forth in the paper to understand most of the sentences and claims.
In the end, based on what I gathered, the paper proposes to decompose the gain in test-time-accuracy in order to be able to attribute it to different components, e.g., the sampler, verifier, and selector. While this is certainly valuable, it doesn't seem very novel. Moreover, I wasn't really able to understand what is actually being presented as results. There is just too many unclear terms and convoluted Claude-style sentences even in the Results section.
I believe the paper needs to be completely rewritten in human-understandable language, with the aim of making its setup and core contributions clear to any reader.
Questions for authors
I don't have specific technical questions. I could list all the undefined variables and terms in the paper and ask "What is A_MV?", "What is wrong-mode rate?", "What is a cluster?", "What is \bar{\sigma}?", but there are just too many of them.
3 comments
Follow-up
Thank you again for the detailed examples. With the discussion closing tomorrow, we would be grateful if you could look at the rewritten first three pages: the abstract, the introduction, and Tables 1 and 2. Every term you listed is now defined there before it is used.
If any passage is still hard to follow, a pointer to it would help us a great deal. If the text now reads clearly, we would appreciate it if you would consider updating your assessment.
I thank the authors for their responses. While the writing has substantially improved, I should say that the overall language still sounds very much AI-style. I believe the revision was also mostly done with AI. This is fine in itself, but the result is still not a very smooth read.
However, more importantly, I find the overall claim and the results very limited in novelty. It's quite clear that the overall accuracy depends on multiple factors, which the paper highlights; but it doesn't really provide any guidelines for any improvements in practice. It's not clear why Theorem 3.1 is a theorem. It just makes some simple statements. Overall, I don't think the paper makes the cut for publication at this venue. I will retain my score.
Thank you for persisting with the paper and for the specific examples; they were the most useful guide to the rewrite. We agree the submitted text was too dense and used terms before defining them. The main text is rewritten.
A short reading path:
Where each term you list is now defined:
Novelty. We do not claim novelty for the conditional-probability factorization itself. Coverage (Brown et al., 2024) and the gap between coverage and voting accuracy (Bay & Yearick, 2026) are prior work, and we build on both. Our contribution is the audit and estimation framework built on them:
We also report findings the framework makes explicit. A verifier's efficiency carries over across models, so its gain can be predicted before running it. And the reward model's loss of efficiency with budget is mostly a reweighting effect.
We hope the revised version makes these readable.