- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SELF-POT, an evaluation framework that systematically assesses the extent to which additional computation at test time can improve language model performance, and whether such improvements are justified when considering the total inference cost. Instead of proposing a new test-time scaling algorithm, the framework evaluates several existing inference methods within a unified accounting structure: direct inference, parallel sampling with rule-based or model-based selection, and self-revision. Notably, all computational calls-including those for generation, selection, critique, planning, and execution-are included in the inference budget, and performance metrics are reported alongside the actual dollar cost.
The study evaluates five cost-effective reasoning models across 350 closed tasks, encompassing competition mathematics, competitive programming, AppWorld workflows, and Terminal-Bench. Claude Opus 5.5 Direct is used as a more capable reference model. The paper differentiates candidate coverage from final accuracy on static tasks, analyzes changes in correctness following revisions, and examines whether agent protocols effectively achieve their intended objectives.
The primary empirical finding is that computational requirements at test time are strongly influenced by the task domain, selection rule, and failure management strategies. For instance, Parallel@2 improves programming performance overall, while the more resource-intensive Parallel@4 configuration, which uses the same model for judging, performs significantly worse under the original submission rule. When programming candidate pools are replayed with a simple fallback or a selection method based on public examples, most of the lost performance is recovered. This indicates that candidate generation and candidate conversion should be evaluated separately. In mathematics tasks, model-based judging provides only marginal improvements over less expensive voting when applied to the same candidate pools. The agent experiments also demonstrate that protocol or interface failures can substantially affect measured performance, making it difficult to attribute failures specifically to planning or reasoning quality.
The paper provides a valuable perspective on test-time scaling by jointly considering candidate generation, selection, failure handling, and monetary inference cost. However, I remain uncertain whether the current experimental design is comprehensive and statistically robust enough to support generalizable conclusions about test-time scaling across different models and domains.
Critical strengths
A significant strength is that the evaluation includes not only candidate generation but also the model-based stages of selection, criticism, planning, and review. This approach avoids the common issue where a method appears more efficient because auxiliary inference calls are excluded from the budget. Additionally, comparing results with a stronger model that provides a single answer is practically relevant: achieving a positive gain within the same model does not necessarily justify allocating additional inference to a less expensive model instead of using a more capable one.
Distinguishing between budget allowance and actual monetary cost is also valuable. As illustrated in Figure 5, the point of highest gain within a model does not necessarily correspond to the optimal deployment configuration.
The paper convincingly demonstrates that relying solely on final accuracy can obscure failures occurring during test-time computation. For example, in programming tasks, the Parallel@4 candidate pools often contain a correct program even when the ultimately selected submission is incorrect or missing. When a candidate is retained due to a failed selection process, this significantly affects the observed mean result, while selection based on public examples recovers nearly all of the observed candidate coverage.
This diagnostic approach is valuable because it distinguishes failures related to sampling, selection, interface reliability, and answer preservation, rather than aggregating them under a single "test-time scaling failed" outcome.
Critical weaknesses
Each of the model–task–protocol cells is selected only once. As a result, the uncertainty stated captures the variation between tasks but not the considerable stochastic variation which may occur during candidate generation, self-selection, critique, revision, or agent execution.
This issue is particularly significant because the paper compares inference strategies that exhibit inherently stochastic behavior. A protocol's performance may fluctuate simply due to different candidate samples. Distinguishing stable protocol effects from sampling noise would require repeated trials on at least a representative subset of tasks.
In certain domains, the number of tasks is relatively small: 60 for mathematics and only 30 for Terminal-Bench. The authors correctly note that a single terminal task represents 3.3 percentage points, and that the reported panel intervals should not be interpreted as universal scaling laws. Consequently, the general claims about the benefits of test-time computation for language models are not fully substantiated by the current evidence.
The same collection of protocols is not assessed by all the models and domains in the experimental matrix. Due to limitations on the context window, only two of the models undergo revision, and for GPT-OSS and MiniMax on Terminal-Bench adaptive omission of the test-time scaling takes place after it is observed that there is zero Direct performance. Thus the cross-model comparison that results is not based on a fully crossed experimental design.
More importantly, the agent's experiments usually do not assess the intended inference strategy when it has been successfully carried out. For instance, the paper gives very high figures for workflow stage failures with respect to GPT-OSS and MiniMax, the failures often taking place before the chosen plan is actually carried out. The authors are right to point out that these entries refer to the combination of model–protocol–interface and do not indicate the quality of a working multi-plan strategy.
This substantially weakens one of the paper's central claims: that it provides a unified cross-domain evaluation of test-time scaling. While the static-task experiments enable meaningful comparisons between sampling and selection strategies, much of the agent-related results focus on interface reliability and protocol completion. I believe it would be preferable to first validate the agent and then conduct repeated end-to-end runs of protocols that are confirmed to be executable.
Questions for authors
-
Since each cell is only sampled once, could the authors give data from repeated runs for at least a representative subset of the combinations of model, task, and protocol? Specifically, what is the extent of within-task variance in comparison with the differences reported between Direct, Parallel@2, Parallel@4, and Revise@4? It appears that this is important when interpreting differences of just a few percentage points.
-
Could the authors give a more completely crossed protocol matrix? Specifically, isn't it possible to assess the revision under smaller but matched feasible budgets for models whose context windows are not able to support the present Revise@4 setup, rather than excluding those models altogether? And similarly, what is the reason for omitting terminal test-time scaling on the basis of the observed Direct results rather than keeping it as part of the original evaluation intention?
-
In the case of the workflow experiments, might the authors re-run the parallel protocols after having corrected or verified the plan/selection interface so that most of the runs actually get to execution? At present, the high rate of stage failures makes it difficult to assess whether plan-level parallelism actually helps or hinders agent performance.
-
To what extent do the cost–accuracy frontiers vary when alternative price assumptions are made beyond those of the DeepSeek half-price analysis? Given that the conclusions regarding deployment depend on the prices charged by the providers as well as on the caching policies and the runtime configuration, it would be useful to provide either the normalized token-level frontiers or a sensitivity analysis covering a broader range of plausible prices.
-
been retained. Could the authors confirm their findings by carrying out prospective end-to-end runs on a new held-out task set? Although the present replay is useful for identifying selection effects, it does not check whether the proposed alternative policy would produce the same gains if it were to be used prospectively.
4 comments
Response to Reviewer L3Ut [2/2]
Q2: a more completely crossed matrix.
(a) Revision at a feasible budget. For MiniMax and GLM-5, excluded from full-budget Revise@4 by their context windows, we ran revision at 2× on the 60 mathematics tasks: the initial answer is the Direct call itself (so the comparison is paired on the same draft), critique ≤ 64K and revision ≤ 32K tokens, with the frozen Self-Refine prompts. It fits both windows without any context fitting and improves both models: MiniMax 46.7 → 50.0% (+3.3 pp, 2–0) and GLM-5 60.0 → 61.7% (+1.7 pp, 4–3), at 1.6–1.7× the Direct cost, i.e., gains of the same size as their P@4 gains. GPT-OSS's 131K-token window is already filled by one full-allowance answer, so revising it would require shortening the shared Direct draft (App. M.7).
(b) Terminal TTS for GPT-OSS and MiniMax. These configurations were omitted after both models scored 0/30 with Direct and no terminal protocol had improved on Direct for the three models run first (Fig. 3); §3.1 now states this. Across all independent terminal episodes, GLM-5 and Qwen3-Next solve 0/30, and even an oracle over Flash's four episodes reaches 20.0%, far below the anchor's 43.3% (App. M.9).
Q4: price assumptions. We provide both suggested analyses (Table R7 in our response to Reviewer W3ZP; App. M.4). Token-level frontiers use no prices: by output tokens, Opus Direct alone forms the mathematics and terminal frontiers. Under independent 0.25–4× price changes for every model (2,000 draws), every member of the measured-price frontiers stays on its frontier in a majority of draws (53–100%) and no other configuration appears in more than a third; removing cache discounts, billing output only, or quadrupling input prices leaves the mathematics and terminal frontiers unchanged and Flash Direct on the programming frontier. Outside programming no low-cost configuration reaches the anchor's accuracy, so that comparison holds at any price. App. M.4 also lists every configuration's cost as a multiple of the anchor's.
W3: task counts. The follow-up evidence adds 18 samples per mathematics task and 46 held-out programming tasks (about 7,300 new cells in total). The run-to-run SD of 1.7–4.2 pp is smaller than the main protocol effects (GPT-OSS P@4 +8.3 pp, positive in all six runs), and 16-sample voting leaves the two version-pinned models 24.5–45.1 pp below the anchor.
Central claim (unified cross-domain evaluation). With the validated interface, the agent results now measure executing strategies (Table R3) together with an oracle reset bound, alongside the interface-reliability finding of the frozen matrix. The one-factor experiments requested by Reviewer HqNP (larger pools, judge with and without execution, programming without feedback; Tables R4–R6 in that response) further show that the static findings are mechanistic: coverage grows faster than conversion as pools grow, and the programming judge's losses arise in tool-enabled selection.
We believe these end-to-end results resolve the soundness concerns, and we are happy to provide any further detail.
Dear Reviewer L3Ut,
As the discussion period is approaching its end, we would greatly appreciate any feedback on our rebuttal and revised manuscript. We conducted the repeated evaluations, prospective held-out tests, and validated agent experiments you specifically requested, along with the additional controls detailed in our responses.
We would be grateful to know whether these new results address your main concerns, or if any questions remain. Thank you for your time and consideration!
Dear Reviewer L3Ut,
As the discussion period approaches its end, we would like to briefly highlight the new evidence directly addressing your main concerns:
- Statistical robustness: Six complete mathematics runs confirm the protocol trends, including GPT-OSS's consistent +8.3 pp gain (6/6 runs).
- Prospective validation: On 46 new held-out programming tasks, public-example selection outperforms the same-model judge by +29.3 pp for GLM-5 (21–0 at the task level, p < 10⁻⁶).
- Agent validity and experimental coverage: We validated the planning/selection interface, conducted fresh end-to-end AppWorld evaluations, and added feasible-budget revision experiments and price sensitivity analyses.
These results are detailed in our responses and revised manuscript (§3.5, Appendix M). We believe they directly resolve the principal soundness concerns raised in your review.
We would greatly appreciate your assessment of this new evidence before the discussion closes, and would be happy to clarify any remaining questions. Thank you again for your careful consideration.
Response to Reviewer L3Ut [1/2]
We thank the reviewer for the careful reading and the concrete suggestions. We answer each question with new end-to-end experiments or analyses of the saved records; the revised PDF adds §3.5 (Table 2, Fig. 6) and Appendix M. Tables R1–R3 are also referenced in our other responses.
Q1/W1: repeated runs. We reran Direct, P@2, and P@4 (shared candidates) on all 60 mathematics tasks five more times for the version-pinned Qwen3-Next and GPT-OSS (six runs including the frozen one; 18 samples per task), and six times on DeepSeek Flash's current endpoint; the provider replaced the model in October and no longer serves the frozen version, so these runs are a separate model, never pooled with the frozen Flash runs.
Table R1. Mathematics accuracy (%), mean ± SD over six runs.
The stochastic variation the reviewer asked about amounts to a run-to-run SD of 1.7–4.2 pp in accuracy. The frozen runs lie inside the replicate spread (|z| ≤ 1.15), and every protocol conclusion holds: P@4 improves GPT-OSS in all six runs, P@2 matches Direct, and Qwen3-Next stays at its Direct level (+0.8 pp). Programming behaves the same way: a new seed of the frozen code configuration reproduces Direct/P@2/P@4 within 2 pp (GLM-5 66/81/55 → 67/80/54; Qwen3-Next 76/82/75 → 76/81/77). Details: App. M.1.
Q5: prospective held-out evaluation. Before any model call, we pre-registered the hypotheses and analysis and sealed 46 AtCoder tasks never shown to any model (same eligibility check and sealing procedure). GLM-5 and Qwen3-Next then ran Direct, P@2, P@4-judge, and P@4-public end to end on shared candidates (2 seeds, 92 pools per model).
Table R2. Held-out programming accuracy (%).
All pre-registered hypotheses hold: the replay's conclusion transfers prospectively to unseen tasks, and public-example selection converts every covered GLM-5 pool (App. M.6).
Q3/W5: validated agent protocols. Using development tasks only, we traced the stage failures to the interface: the planning prompt did not state that the next stage can call the Python tool, and the parser accepted only a reply that is exactly
SELECT: N. We fixed both under a rule set in advance (a variant is adopted only if no model gets worse on development tasks), added Plan@1 (one plan, no selection) to isolate explicit planning, and reran the 160 sealed AppWorld test tasks with fresh episodes.Table R3. AppWorld test (160 tasks, new seed; success %).
With the protocols executing, plan-level parallelism stays within sampling error of Direct for all three models, including the strong anchor. Even Direct, which has no planning or selection stage, solves only 6.2% (GPT-OSS) and 10.6% (MiniMax) of tasks against 93.1% for Opus 5.5, so the gap to the stronger model is not an interface effect. Fresh episodes also bound resetting: an oracle that keeps the best of Flash's four independent AppWorld episodes solves 80.0% within the anchor's cost ($0.097/task), still below the anchor's 92.5% (Terminal-Bench: 20.0% vs. 43.3%). App. M.8–M.9.
(continued below)