acceptodds
Under review as a conference paper at ICLR 2027

How Much Can Language Models Gain from Test-Time Computation?

13 views

Abstract

How much can test-time computation improve a language model, and at what cost? Test-time scaling is widely proposed as a substitute for larger models, but existing comparisons mostly evaluate one domain at a time and rarely charge selection to the budget. We introduce SELF-POT, a benchmark and evaluation framework that measures the test-time potential of a model across competition mathematics, competitive programming, and agentic workflows. SELF-POT separates candidate coverage from final accuracy on static tasks, tracks correctness transitions under revision, and measures protocol completion alongside task success in agentic environments. Under a unified budget rule, it compares Direct inference with parallel sampling and self-revision under fixed multiples of the Direct budget, and charges every model call, including selection and critique, in dollars. This supports comparisons of both within-model gains and lower-cost models with additional inference against a stronger model. Across five low-cost reasoning models on 350 sealed tasks, with Claude Opus 5.5 Direct as the reference, the returns depend on the domain, selection rule, and failure handling. Replaying retained programming pools, public-example selection raises correct submissions from 376 to 453 of 500 scheduled cells while saving 12-49% of logical API cost across models; simply retaining an available candidate when judging fails recovers 61 submissions at unchanged cost. On identical mathematics pools, judging with fallback yields 186 correct submissions versus 182 for voting, which saves 12-21% of logical API cost. These controlled replays show how selection and failure handling change the gains realized from the same generated candidates and quantify the marginal value of a model judge.

Primary area:
datasets and benchmarks
Keywords:
Test-time scaling, Large language models, Benchmarking and evaluation, Inference-time compute

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L3UtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SELF-POT, an evaluation framework that systematically assesses the extent to which additional computation at test time can improve language model performance, and whether such improvements are justified when considering the total inference cost. Instead of proposing a new test-time scaling algorithm, the framework evaluates several existing inference methods within a unified accounting structure: direct inference, parallel sampling with rule-based or model-based selection, and self-revision. Notably, all computational calls-including those for generation, selection, critique, planning, and execution-are included in the inference budget, and performance metrics are reported alongside the actual dollar cost.

The study evaluates five cost-effective reasoning models across 350 closed tasks, encompassing competition mathematics, competitive programming, AppWorld workflows, and Terminal-Bench. Claude Opus 5.5 Direct is used as a more capable reference model. The paper differentiates candidate coverage from final accuracy on static tasks, analyzes changes in correctness following revisions, and examines whether agent protocols effectively achieve their intended objectives.

The primary empirical finding is that computational requirements at test time are strongly influenced by the task domain, selection rule, and failure management strategies. For instance, Parallel@2 improves programming performance overall, while the more resource-intensive Parallel@4 configuration, which uses the same model for judging, performs significantly worse under the original submission rule. When programming candidate pools are replayed with a simple fallback or a selection method based on public examples, most of the lost performance is recovered. This indicates that candidate generation and candidate conversion should be evaluated separately. In mathematics tasks, model-based judging provides only marginal improvements over less expensive voting when applied to the same candidate pools. The agent experiments also demonstrate that protocol or interface failures can substantially affect measured performance, making it difficult to attribute failures specifically to planning or reasoning quality.

The paper provides a valuable perspective on test-time scaling by jointly considering candidate generation, selection, failure handling, and monetary inference cost. However, I remain uncertain whether the current experimental design is comprehensive and statistically robust enough to support generalizable conclusions about test-time scaling across different models and domains.

Critical strengths

A significant strength is that the evaluation includes not only candidate generation but also the model-based stages of selection, criticism, planning, and review. This approach avoids the common issue where a method appears more efficient because auxiliary inference calls are excluded from the budget. Additionally, comparing results with a stronger model that provides a single answer is practically relevant: achieving a positive gain within the same model does not necessarily justify allocating additional inference to a less expensive model instead of using a more capable one.

Distinguishing between budget allowance and actual monetary cost is also valuable. As illustrated in Figure 5, the point of highest gain within a model does not necessarily correspond to the optimal deployment configuration.

The paper convincingly demonstrates that relying solely on final accuracy can obscure failures occurring during test-time computation. For example, in programming tasks, the Parallel@4 candidate pools often contain a correct program even when the ultimately selected submission is incorrect or missing. When a candidate is retained due to a failed selection process, this significantly affects the observed mean result, while selection based on public examples recovers nearly all of the observed candidate coverage.

This diagnostic approach is valuable because it distinguishes failures related to sampling, selection, interface reliability, and answer preservation, rather than aggregating them under a single "test-time scaling failed" outcome.

Critical weaknesses

Each of the model–task–protocol cells is selected only once. As a result, the uncertainty stated captures the variation between tasks but not the considerable stochastic variation which may occur during candidate generation, self-selection, critique, revision, or agent execution.

This issue is particularly significant because the paper compares inference strategies that exhibit inherently stochastic behavior. A protocol's performance may fluctuate simply due to different candidate samples. Distinguishing stable protocol effects from sampling noise would require repeated trials on at least a representative subset of tasks.

In certain domains, the number of tasks is relatively small: 60 for mathematics and only 30 for Terminal-Bench. The authors correctly note that a single terminal task represents 3.3 percentage points, and that the reported panel intervals should not be interpreted as universal scaling laws. Consequently, the general claims about the benefits of test-time computation for language models are not fully substantiated by the current evidence.

The same collection of protocols is not assessed by all the models and domains in the experimental matrix. Due to limitations on the context window, only two of the models undergo revision, and for GPT-OSS and MiniMax on Terminal-Bench adaptive omission of the test-time scaling takes place after it is observed that there is zero Direct performance. Thus the cross-model comparison that results is not based on a fully crossed experimental design.

More importantly, the agent's experiments usually do not assess the intended inference strategy when it has been successfully carried out. For instance, the paper gives very high figures for workflow stage failures with respect to GPT-OSS and MiniMax, the failures often taking place before the chosen plan is actually carried out. The authors are right to point out that these entries refer to the combination of model–protocol–interface and do not indicate the quality of a working multi-plan strategy.

This substantially weakens one of the paper's central claims: that it provides a unified cross-domain evaluation of test-time scaling. While the static-task experiments enable meaningful comparisons between sampling and selection strategies, much of the agent-related results focus on interface reliability and protocol completion. I believe it would be preferable to first validate the agent and then conduct repeated end-to-end runs of protocols that are confirmed to be executable.

Questions for authors
  1. Since each cell is only sampled once, could the authors give data from repeated runs for at least a representative subset of the combinations of model, task, and protocol? Specifically, what is the extent of within-task variance in comparison with the differences reported between Direct, Parallel@2, Parallel@4, and Revise@4? It appears that this is important when interpreting differences of just a few percentage points.

  2. Could the authors give a more completely crossed protocol matrix? Specifically, isn't it possible to assess the revision under smaller but matched feasible budgets for models whose context windows are not able to support the present Revise@4 setup, rather than excluding those models altogether? And similarly, what is the reason for omitting terminal test-time scaling on the basis of the observed Direct results rather than keeping it as part of the original evaluation intention?

  3. In the case of the workflow experiments, might the authors re-run the parallel protocols after having corrected or verified the plan/selection interface so that most of the runs actually get to execution? At present, the high rate of stage failures makes it difficult to assess whether plan-level parallelism actually helps or hinders agent performance.

  4. To what extent do the cost–accuracy frontiers vary when alternative price assumptions are made beyond those of the DeepSeek half-price analysis? Given that the conclusions regarding deployment depend on the prices charged by the providers as well as on the caching policies and the runtime configuration, it would be useful to provide either the normalized token-level frontiers or a sensitivity analysis covering a broader range of plausible prices.

  5. been retained. Could the authors confirm their findings by carrying out prospective end-to-end runs on a new held-out task set? Although the present replay is useful for identifying selection effects, it does not check whether the proposed alternative policy would produce the same gains if it were to be used prospectively.

4 comments
AuthorsOpenReview ↗

Response to Reviewer L3Ut [1/2]

We thank the reviewer for the careful reading and the concrete suggestions. We answer each question with new end-to-end experiments or analyses of the saved records; the revised PDF adds §3.5 (Table 2, Fig. 6) and Appendix M. Tables R1–R3 are also referenced in our other responses.

Q1/W1: repeated runs. We reran Direct, P@2, and P@4 (shared candidates) on all 60 mathematics tasks five more times for the version-pinned Qwen3-Next and GPT-OSS (six runs including the frozen one; 18 samples per task), and six times on DeepSeek Flash's current endpoint; the provider replaced the model in October and no longer serves the frozen version, so these runs are a separate model, never pooled with the frozen Flash runs.

Table R1. Mathematics accuracy (%), mean ± SD over six runs.

ModelDirectP@2P@4P@4 − Direct [95% CI]runs with P@4 > Direct
Qwen3-Next49.4 ± 4.249.4 ± 4.250.3 ± 2.7+0.8 [−2.2, +4.2]4/6
GPT-OSS52.8 ± 1.753.3 ± 1.861.1 ± 2.9+8.3 [+3.9, +13.3]6/6
Flash (Oct. update)85.6 ± 1.785.8 ± 1.788.1 ± 2.5+2.5 [−0.6, +5.8]5/6

The stochastic variation the reviewer asked about amounts to a run-to-run SD of 1.7–4.2 pp in accuracy. The frozen runs lie inside the replicate spread (|z| ≤ 1.15), and every protocol conclusion holds: P@4 improves GPT-OSS in all six runs, P@2 matches Direct, and Qwen3-Next stays at its Direct level (+0.8 pp). Programming behaves the same way: a new seed of the frozen code configuration reproduces Direct/P@2/P@4 within 2 pp (GLM-5 66/81/55 → 67/80/54; Qwen3-Next 76/82/75 → 76/81/77). Details: App. M.1.

Q5: prospective held-out evaluation. Before any model call, we pre-registered the hypotheses and analysis and sealed 46 AtCoder tasks never shown to any model (same eligibility check and sealing procedure). GLM-5 and Qwen3-Next then ran Direct, P@2, P@4-judge, and P@4-public end to end on shared candidates (2 seeds, 92 pools per model).

Table R2. Held-out programming accuracy (%).

ModelDirectP@2P@4-judgeP@4-publicpublic − judge (W–L)conversion: public / judge
GLM-575.091.365.294.6+29.3 (27–0 pools; 21–0 tasks, p = 9.5e-7)100% / 69%
Qwen3-Next90.291.393.593.50.0 (0–0)97.7% / 97.7%

All pre-registered hypotheses hold: the replay's conclusion transfers prospectively to unseen tasks, and public-example selection converts every covered GLM-5 pool (App. M.6).

Q3/W5: validated agent protocols. Using development tasks only, we traced the stage failures to the interface: the planning prompt did not state that the next stage can call the Python tool, and the parser accepted only a reply that is exactly SELECT: N. We fixed both under a rule set in advance (a variant is adopted only if no model gets worse on development tasks), added Plan@1 (one plan, no selection) to isolate explicit planning, and reran the 160 sealed AppWorld test tasks with fresh episodes.

Table R3. AppWorld test (160 tasks, new seed; success %).

Modelvalid selection (frozen)DirectPlan@1P@2P@2 − Direct [95% CI]
GPT-OSS99.4% (27%)6.25.67.5+1.2 [−3.1, +5.6]
MiniMax80.0% (0.6%)10.68.86.2−4.4 [−10.0, +1.2]
Opus 5.5100% (–)93.194.393.10.0 [−3.1, +3.1]

With the protocols executing, plan-level parallelism stays within sampling error of Direct for all three models, including the strong anchor. Even Direct, which has no planning or selection stage, solves only 6.2% (GPT-OSS) and 10.6% (MiniMax) of tasks against 93.1% for Opus 5.5, so the gap to the stronger model is not an interface effect. Fresh episodes also bound resetting: an oracle that keeps the best of Flash's four independent AppWorld episodes solves 80.0% within the anchor's cost ($0.097/task), still below the anchor's 92.5% (Terminal-Bench: 20.0% vs. 43.3%). App. M.8–M.9.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer L3Ut [2/2]

Q2: a more completely crossed matrix.

(a) Revision at a feasible budget. For MiniMax and GLM-5, excluded from full-budget Revise@4 by their context windows, we ran revision at 2× on the 60 mathematics tasks: the initial answer is the Direct call itself (so the comparison is paired on the same draft), critique ≤ 64K and revision ≤ 32K tokens, with the frozen Self-Refine prompts. It fits both windows without any context fitting and improves both models: MiniMax 46.7 → 50.0% (+3.3 pp, 2–0) and GLM-5 60.0 → 61.7% (+1.7 pp, 4–3), at 1.6–1.7× the Direct cost, i.e., gains of the same size as their P@4 gains. GPT-OSS's 131K-token window is already filled by one full-allowance answer, so revising it would require shortening the shared Direct draft (App. M.7).

(b) Terminal TTS for GPT-OSS and MiniMax. These configurations were omitted after both models scored 0/30 with Direct and no terminal protocol had improved on Direct for the three models run first (Fig. 3); §3.1 now states this. Across all independent terminal episodes, GLM-5 and Qwen3-Next solve 0/30, and even an oracle over Flash's four episodes reaches 20.0%, far below the anchor's 43.3% (App. M.9).

Q4: price assumptions. We provide both suggested analyses (Table R7 in our response to Reviewer W3ZP; App. M.4). Token-level frontiers use no prices: by output tokens, Opus Direct alone forms the mathematics and terminal frontiers. Under independent 0.25–4× price changes for every model (2,000 draws), every member of the measured-price frontiers stays on its frontier in a majority of draws (53–100%) and no other configuration appears in more than a third; removing cache discounts, billing output only, or quadrupling input prices leaves the mathematics and terminal frontiers unchanged and Flash Direct on the programming frontier. Outside programming no low-cost configuration reaches the anchor's accuracy, so that comparison holds at any price. App. M.4 also lists every configuration's cost as a multiple of the anchor's.

W3: task counts. The follow-up evidence adds 18 samples per mathematics task and 46 held-out programming tasks (about 7,300 new cells in total). The run-to-run SD of 1.7–4.2 pp is smaller than the main protocol effects (GPT-OSS P@4 +8.3 pp, positive in all six runs), and 16-sample voting leaves the two version-pinned models 24.5–45.1 pp below the anchor.

Central claim (unified cross-domain evaluation). With the validated interface, the agent results now measure executing strategies (Table R3) together with an oracle reset bound, alongside the interface-reliability finding of the frozen matrix. The one-factor experiments requested by Reviewer HqNP (larger pools, judge with and without execution, programming without feedback; Tables R4–R6 in that response) further show that the static findings are mechanistic: coverage grows faster than conversion as pools grow, and the programming judge's losses arise in tool-enabled selection.

We believe these end-to-end results resolve the soundness concerns, and we are happy to provide any further detail.

Dear Reviewer L3Ut,

As the discussion period is approaching its end, we would greatly appreciate any feedback on our rebuttal and revised manuscript. We conducted the repeated evaluations, prospective held-out tests, and validated agent experiments you specifically requested, along with the additional controls detailed in our responses.

We would be grateful to know whether these new results address your main concerns, or if any questions remain. Thank you for your time and consideration!

Dear Reviewer L3Ut,

As the discussion period approaches its end, we would like to briefly highlight the new evidence directly addressing your main concerns:

  • Statistical robustness: Six complete mathematics runs confirm the protocol trends, including GPT-OSS's consistent +8.3 pp gain (6/6 runs).
  • Prospective validation: On 46 new held-out programming tasks, public-example selection outperforms the same-model judge by +29.3 pp for GLM-5 (21–0 at the task level, p < 10⁻⁶).
  • Agent validity and experimental coverage: We validated the planning/selection interface, conducted fresh end-to-end AppWorld evaluations, and added feasible-budget revision experiments and price sensitivity analyses.

These results are detailed in our responses and revised manuscript (§3.5, Appendix M). We believe they directly resolve the principal soundness concerns raised in your review.

We would greatly appreciate your assessment of this new evidence before the discussion closes, and would be happy to clarify any remaining questions. Thank you again for your careful consideration.

Reviewer_HqNPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is an empirical study of how much low-cost reasoning models gain from test-time scaling (TTS) and at what cost. It asks which parallel and sequential protocols turn extra computation into accuracy and in which domains, where the opportunity is lost (generating a correct candidate, selecting it, or preserving it under revision), and whether a cheap model with extra computation can match a stronger model answering once. The study constructs a benchmark SELF-POT, which consists of 350 tasks from existing benchmarks (APEX and Omni-MATH for mathematics, AtCoder problems from LiveCodeBench, AppWorld workflows and Terminal-Bench). Five low-cost models answer each task once (Direct) and under three TTS protocols that may use up to two times (Parallel@2) or four times (Parallel@4, Revise@4) the tokens of a single answer. The design covers the main factors in TTS: the domain (whether the solver gets feedback while solving, and whether its actions can be undone), the budget, parallel sampling against sequential revision, the choice rule (majority vote, sample tests on programming problems, or a same-model judge), and what happens when the selection step returns nothing. For mathematics and programming, it also records whether any generated candidate was correct, separately from whether the submitted one was, and re-scores the saved candidates under other ways of choosing among them without new model calls. It finds that gains depend on domain, choice rule and failure handling, that submitting the program that passes the most sample tests gives a correct submission in 453 of the 457 programming cases where a correct candidate existed, and that a same-model judge does no better than majority voting on mathematics.

Critical strengths

(++) The measurement design is careful and charges costs honestly. Every call (judging, critique, planning) counts against the same budget and is priced in dollars, and the split between coverage and conversion shows where a protocol loses, for example GLM-5 producing a correct program for 88 tasks and submitting 55.

(+) Two questions get concrete answers. On static tasks, losses come mostly from selection and submission, since submitting the program that passes the most sample tests gives a correct submission in 453 of the 457 programming cases where a correct candidate existed. No cheaper model with extra computation reaches the accuracy of Claude Opus 5.5 on mathematics or on either agent domain, and at peak prices the best mathematics configurations also cost more, which is a useful negative result for practitioners.

Critical weaknesses

(---) The budget stops at four times one answer (three candidates on static tasks), too small to show how gains grow with more computation.

(--) No result is broken down by task difficulty, although half of the model–domain pairs sit near 0% or 100% under Direct and every task has a difficulty label.

(---) Protocols change several things at once (Parallel@2 and Parallel@4 differ in both candidate count and choice rule), and the replays vary only the choice rule and the fallback.

(-) Mathematics gets no tools while programming can run its sample tests, so domain and external verification are confounded.

(-) Agent episodes are never reset and no agent-side component is varied, so the agent results say little about test-time computation.

(-) The writing is dense with coined terms ("cells", "pools", "conversion", "anchor") and long sentences such as RQ2, and "@4" means three candidates on static tasks but four plans on agent tasks.

Questions for authors
  • Can you report each protocol's gain over Direct by the published difficulty tiers (easy/medium/hard), or by how many of a model's three Parallel@4 candidates are correct on each task? This would show whether extra computation mainly helps on tasks of medium difficulty for a given model, and little on tasks that are too easy or too hard for it.
  • How do accuracy and the share of tasks with a correct candidate change with larger budgets, for example 8 or 16 candidates chosen by sample tests on programming and by majority vote on mathematics?
  • Would programming without execution feedback, where candidates cannot be run on the sample tests, behave like mathematics, and would mathematics with code execution behave like programming? Either setting would separate the domain from the effect of external verification (tools or environments).
  • Can you vary one part of a protocol at a time, for example the number of candidates with the choice rule fixed, the judge with and without running the sample tests, and revision with and without test results as feedback? This would show which part causes each gain or loss.
5 comments
AuthorsOpenReview ↗

Response to Reviewer HqNP [1/2]

We thank the reviewer for the positive assessment and the precise questions; each is answered below by a new experiment. The revised PDF adds §3.5 (Table 2, Fig. 6) and Appendix M.

Q1: gains by difficulty. We report both stratifications (App. M.3). By model-specific difficulty, i.e., the number k of a model's three P@4 candidates that are correct (five models pooled):

Table R4. Correct submissions (%) by k.

kMath poolsDirectJudgeVote@3Code poolsDirectJudgePublic@3
083000402.500
13333452125323688
24060859873667199
31369910010035810088100

This matches the reviewer's hypothesis: extra computation acts on tasks of intermediate difficulty for the given model (k = 1 or 2: 73 mathematics and 98 programming pools). No rule rescues a k = 0 pool, and a k = 3 pool can only be lost, as 43 programming pools are under the frozen judge. By published tiers, P@2 gains on programming in every tier (easy/medium/hard +3.3/+7.9/+7.1 pp; 31–0 overall), while the judge loses in every tier (−7.3 to −9.3 pp), so its losses are not a difficulty effect; the mathematics P@4 gain concentrates on APEX (+4.7 pp, 15–7).

Q2: 8 or 16 candidates. With 18 samples per mathematics task (six complete runs; Table R1 in our response to Reviewer L3Ut) and eight shared programs per programming task (Qwen3-Next), Table R5 and Fig. 6 show selection and coverage as the pool grows (coverage and programming selection exact over all k-subsets; mathematics voting over 4,000 random orders per task).

Table R5. Accuracy (%) by pool size k.

k=124816
Qwen3-Next math: vote / coverage48.6 / 48.648.6 / 55.850.4 / 62.350.7 / 68.149.9 / 73.9
GPT-OSS math: vote / coverage53.8 / 53.854.2 / 65.260.4 / 74.465.4 / 81.470.5 / 87.5
Flash (Oct.) math: vote / coverage85.4 / 85.485.5 / 90.288.9 / 92.691.1 / 93.891.7 / 94.8
Qwen3-Next code: public / coverage77.6 / 77.680.6 / 83.282.6 / 86.884.0 / 89.0–

Coverage keeps rising with the budget, whereas majority voting saturates (Qwen3-Next stays near 50%) or grows more slowly, and public-example selection keeps converting additional programs. The coverage–conversion gap, SELF-POT's central diagnostic, therefore widens with budget, and no pool size brings voting to the anchor's 95.0%.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer HqNP [2/2]

Q3: programming without execution feedback. Since either setting separates the domain from external verification, we used programming, where execution can be removed without changing the task: on the 100 sealed tasks, Arm B removes run_code from every stage (public-example selection remains available as a rule), next to the frozen Arm A.

Table R6. Programming accuracy (%) and judge conversion (correct / covered pools).

ModelArmDirectP@4 judgeP@4 judge, no execP@4 publicjudge conversion
GLM-5A (frozen setup, new seed)67.054.088.087.054/89
GLM-5B (no feedback)88.092.0–90.092/93
Qwen3-NextA (frozen setup, new seed)76.077.080.082.077/85
Qwen3-NextB (no feedback)79.083.0–81.083/87

Without execution feedback, programming behaves like mathematics in the judge's effect: the same-model judge gains +4 pp over Direct for both models (the mathematics panel gain is +4.0 pp) and converts 92/93 and 83/87 covered pools. On identical candidates, removing execution from the judge alone raises GLM-5 from 54% to 88% (Q4). The programming-specific judge losses in the paper thus arise from tool-enabled selection, the conversion stage that SELF-POT isolates, rather than from the domain.

Q4: one factor at a time.

  • Number of candidates, rule fixed (identical frozen pools, nested): vote@1/2/3 = 172/175/182 of 300 mathematics cells; public examples @1/2/3 = 416/448/453 of 500 programming cells; extended to 16 and 8 candidates in Table R5.
  • Judge with vs. without running the sample tests (shared candidates; Table R6, Arm A): GLM-5 54.0 → 88.0% (+34.0 pp, 34–0, p = 1.2e-10; 32–0 on the 96 pools whose programs are identical across selectors), Qwen3-Next 77 → 80%. With execution, 37 of GLM-5's 100 judge replies end without a selection line, the "no choice" failure of Fig. 4; without it, the judge converts 87/89 covered pools, as many as public-example selection.
  • Revision with vs. without test results (same initial programs, Qwen3-Next): identical outcomes on all 86 tasks where both variants complete.

These runs identify selection, not generation, as the stage where programming candidates are lost, and tool-enabled judging as the mechanism for GLM-5, the model with the largest coverage–conversion gap in the paper (App. M.5).

W5: agent episodes and agent-side components. We varied the agent side directly: a validated plan interface, Plan@1 versus P@2 (a second plan with selection), and fresh episodes (Table R3 in our response to Reviewer L3Ut). Independent episodes also bound what resetting could achieve: keeping the best of Flash's four fresh AppWorld episodes with an oracle reaches 80.0% at the anchor's cost (anchor 92.5%), and the best of six episodes reaches 23.1% for GPT-OSS and 31.2% for MiniMax. Resets add headroom, but realizing it requires a verifier of final environment states, and even a perfect one stays below the stronger model (App. M.9).

W6: terminology. The revision defines cell and pool at first use (§2.1) and states in §2.2 that @k denotes a budget multiple: on static tasks P@4 spends 3B on three candidates and B on selection, and on agent tasks it compares four plans.

We believe these experiments answer each question and are happy to provide further detail.

Dear Reviewer HqNP,

Thank you again for your constructive and encouraging review. We have added the larger-budget experiments, difficulty analyses, and controlled ablations you suggested, including the execution-feedback comparisons.

As the discussion period is nearing its end, we would greatly appreciate any feedback on these new results and whether they address your questions. We would be happy to clarify anything further. Thank you for your time!

Reviewer_W3ZPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SELF-POT, a benchmark and evaluation framework for measuring how much language models benefit from additional test-time computation under explicit inference budgets. It evaluates five lower-cost reasoning models against Claude Opus 5.5 Direct across 350 tasks spanning competition mathematics, competitive programming, AppWorld, and Terminal-Bench.

The authors compare Direct inference, parallel sampling with rule-based selection, parallel sampling with same-model judging, and sequential self-revision. They account for inference costs across generation, selection, and critique, and distinguish candidate coverage from final task success.

The main findings are that additional computation does not consistently improve performance, selection and failure handling substantially affect realized gains, and cheaper models with additional inference do not necessarily outperform stronger models on the accuracy–cost frontier.

Critical strengths

S1. Important research problem. The paper tackles an important and timely question: how much capability can be unlocked from low-cost language models through additional test-time computation? Understanding the potential of cheaper intelligence and whether it can compete with stronger, more expensive models is highly relevant to both research and practical deployment.

Critical weaknesses

W1. Limited model and benchmark coverage.

The paper evaluates only five low-cost models and one stronger reference model across 350 tasks. The coverage is further restricted by the incomplete experimental matrix, with self-revision evaluated on only two models and some agentic configurations omitted. The relatively small number of tasks in certain domains, particularly mathematics and Terminal-Bench, also limits the generalizability of the findings.

W2. Conflation of model capabilities and inference costs.

The current evaluation setting does not sufficiently disentangle model capabilities from inference costs. API pricing is influenced by factors beyond model capability and can change substantially as models and serving infrastructure become more cost-efficient.

W3. Narrow evaluation of test-time scaling strategies.

The paper evaluates a limited set of predefined scaling protocols, which may not adequately characterize the potential gains from additional computation. In particular, the scaling strategies used for agentic tasks differ substantially from those used for static reasoning tasks, making cross-domain comparisons difficult to interpret.

W4. Presentation and organization could be improved.

The paper contains extensive experimental results and implementation details, but the overall narrative could be clearer. I

Questions for authors

Q1. Why not allocate a fixed fraction or multiple of the stronger model's inference budget to each low-cost model? Q2. Why are different test-time scaling strategies used for static and agentic tasks? Q3. How robust are the conclusions to changes in model pricing?

5 comments
AuthorsOpenReview ↗

Response to Reviewer W3ZP

We thank the reviewer for recognizing the importance of the question. The revision adds end-to-end follow-up experiments (§3.5, Table 2, Fig. 6; Appendix M; about 7,300 new cells) that address each weakness.

W1: coverage of models, tasks, and protocols. The follow-up evidence extends every axis the reviewer lists:

  • six complete runs of Direct, P@2, and P@4 in mathematics (18 samples per task) and a new seed for programming and AppWorld (Table R1 in our response to Reviewer L3Ut);
  • 46 pre-registered held-out programming tasks (Table R2, ibid.);
  • revision for two more models at a feasible budget, and validated agent protocols with Opus 5.5 added (Table R3, ibid.);
  • pools of up to 16 candidates and one-factor ablations of candidates, choice rule, and execution feedback (Tables R5–R6 in our response to Reviewer HqNP).

All conclusions hold: the run-to-run SD is 1.7–4.2 pp, smaller than the main protocol effects (GPT-OSS P@4 +8.3 pp, positive in all six runs), and 16-sample voting leaves the version-pinned models 24.5–45.1 pp below the anchor.

The revision also adds new findings: without execution feedback, programming behaves like mathematics (the judge gains +4 pp); the programming judge's losses come from tool-enabled selection (GLM-5: 54% → 88% when the judge cannot run code); the gap between coverage and conversion widens with budget; and public-example selection beats the same-model judge prospectively on unseen tasks (+29.3 pp for GLM-5).

W2/Q3: capability vs. price. As in any cost-aware evaluation of LLMs, dollar costs follow provider prices. SELF-POT therefore reports tokens next to dollars and separates the comparisons that prices cannot change from those they can (App. M.4):

Table R7. Price robustness (2,000 draws, each model's prices independently ×0.25–4).

Domainlow-cost configs reaching the anchor's accuracymeasured-price frontier (share of draws kept)frontier by output tokens only
MathnoneOpus Direct 100%, Flash Direct 71%, GPT-OSS Direct 64%, Flash Revise@4 53%Opus Direct only
WorkflownoneOpus Direct 100%, GPT-OSS P@2 96%, Flash P@2 90%, MiniMax P@2 76%, GPT-OSS Direct 57%Opus Direct, GPT-OSS P@2, MiniMax P@2
TerminalnoneOpus Direct 100%, Flash Direct 94%Opus Direct only
CodeFlash Direct (99%; cheaper unless its prices rise 2.07×)Flash Direct 100%Flash Direct, Opus Direct

The capability comparison is price-free: outside programming, no low-cost configuration reaches the anchor's accuracy at any price. The dollar frontiers are stable too: no configuration outside them appears in more than a third of draws, and cache-free, output-only, or 4× input pricing leaves the mathematics and terminal frontiers unchanged and Flash Direct on the programming frontier.

Q1: a multiple of the stronger model's budget. Within-model budget multiples answer RQ1 (how a model converts its own extra computation); RQ3 compares configurations against the anchor at measured cost. The new larger-pool runs give the anchor-budget view directly: at the anchor's cost per task (interpolated between pool sizes), majority voting reaches 48.8% (Qwen3-Next), 60.7% (GPT-OSS), and 87.1% (Flash Oct.) on mathematics against the anchor's 95.0%, and even at 3.8–7.2× the anchor's cost (16 candidates) it reaches at most 91.7% (Table R5 in our response to Reviewer HqNP; Fig. 6). On AppWorld, four fresh Flash episodes cost no more than the anchor ($0.097/task) and, even with an oracle picking the best, solve 80.0% against 92.5%. App. M.4 lists every configuration's cost as a multiple of the anchor's.

W3/Q2: different strategies for static and agentic tasks. In an agent task, actions change the environment, so keeping one of several executed trajectories requires a verifier of final states, which the hidden native evaluator does not provide at test time. Parallel sampling with selection therefore operates before execution (plan-level parallelism), the counterpart of drawing several static answers and choosing one, and Revise@4 maps to periodic review. The revision adds the agent counterparts the comparison needs: a validated plan interface with Plan@1 versus P@2, isolating a second plan with selection, and fresh-episode resets with an oracle choice as an upper bound (Table R3 in our response to Reviewer L3Ut). Both confirm the cross-domain conclusion: outside programming, no tested amount of sampling, selection, or resetting reaches the stronger model. On static tasks, the protocol space now spans pools of up to 16, three choice rules, execution feedback on and off, and two revision budgets.

W4: presentation. The new §3.5 consolidates the follow-up evidence in one table, one figure, and Finding 5; §2 defines the coined terms and @k at first use.

We believe the added evidence addresses each concern and would be glad to clarify any remaining point.

Dear Reviewer W3ZP,

As the discussion period is approaching its end, we would greatly appreciate your feedback on our detailed rebuttal and revised manuscript. We added substantial new experiments addressing the concerns about evaluation coverage, scaling budgets, protocol comparisons, and pricing robustness.

We hope these additional results help clarify the scope and contributions of our work, and would be grateful to hear whether any major concerns remain. Thank you for your time and consideration!

Dear Reviewer W3ZP,

As the discussion period approaches its end, we would like to briefly revisit the substantial new evidence addressing your concerns about evaluation breadth, scaling strategies, and cost comparisons.

Our revision adds approximately 7,300 experimental cells, including six-run mathematics evaluations, candidate pools up to 16, new held-out programming tasks, controlled execution-feedback ablations, and validated agent protocols. Importantly, these experiments provide new mechanistic evidence: removing tool access from the GLM-5 judge improves accuracy from 54% to 88%, while prospective public-example selection outperforms judging by 29.3 pp on unseen tasks. We also report anchor-normalized costs and robustness under 0.25–4× price variations.

Together, these results strengthen both the empirical foundation and the central coverage–conversion diagnosis of SELF-POT (§3.5, Appendix M).

We would sincerely appreciate your reconsideration of the revised evidence and welcome any remaining questions. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.