- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how the performance of LLM agents scales with increasing test-time compute, particularly when continuing iterative refinement within a single agent session becomes less efficient than running multiple independent sessions. The authors further propose a strategy based on the Scaling Inflection Point to determine how a fixed token budget should be allocated across independent sessions.
Experiments are conducted with four general-purpose agent systems on 14 open-ended optimization tasks across four benchmarks. The results show that increasing test-time compute can substantially improve agent performance at early stages, but the marginal gains diminish at larger budgets. The compute-allocation experiments suggest that an appropriate number of independent sessions can outperform both a single long session and many short sessions.
Critical strengths
-
How to allocate a fixed test-time compute budget between iterative refinement and independent exploration is an important research topic. The proposed Elo-per-token framework provides a way to measure agent scaling behavior across tasks with different evaluation metrics. It captures how solution quality improves throughout long agent trajectories and offers a criterion for deciding how to balance continued refinement and independent exploration.
-
The paper evaluates four frontier agent systems. The experiments reveal interesting patterns of diminishing marginal returns in long-running agent sessions. More importantly, the authors attempt to translate these observations into a practical compute-allocation rule. The results on Polyomino Packing demonstrate that selecting an appropriate number of independent sessions can significantly improve performance under a fixed total token budget.
Critical weaknesses
-
The observation that model performance eventually exhibits diminishing marginal returns as test-time compute increases is not entirely new. Furthermore, the trade-off between continuing an existing optimization trajectory and restarting independent searches has been extensively studied, including classical Restart Strategies and Bet-and-Run methods.
-
Therefore, the main novelty of this paper appears to lie in the Elo-per-token formulation and the resulting Inflection-point Allocation Rule. Although the authors theoretically derive a criterion for allocating tokens to individual sessions, the experimental design does not sufficiently validate its practical reliability. For example, the authors could first estimate the optimal session budget (b) using a set of pilot runs, and then compare it against alternative budget allocations on independent held-out runs to demonstrate that the predicted allocation achieves competitive performance.
Questions for authors
Q1. What is the unique advantage of Elo-per-token over existing scaling analyses and restart strategies?
Q2. How robust is the 400-Elo-based Inflection Point in practice? For example, could the authors estimate the Inflection Point using one set of pilot sessions and then evaluate the predicted budget allocation on completely independent sessions?
4 comments
Thank the authors for clarifying my concerns. But for Q2, since the calculated K is either 1 or 2, why not add a comparison between K=1 and K=2, or include other K values such as 3, instead of directly jumping to K=10?
Thank you for the follow-up. Every held-out pool already contains these allocations. As in Sec. 5, each pool uses the nested design (five sessions each at 100M, 50M, 33.3M, 25M and 20M, and twenty-five at 10M), so one tournament rates K = 1, 2, 3, 4, 5 and 10 at the same total budget of 100M tokens (App. C.6). Our earlier table summarized only the gains over the two extremes. The full results are below (joint Elo at 100M; best allocation in bold):
| System (harness + model) | Inflection point → K | K = 1 | K = 2 | K = 3 | K = 4 | K = 5 | K = 10 |
|---|---|---|---|---|---|---|---|
| Gemini CLI 0.67.0 + carbon | K = 1 | 2568 | 2487 | 2260 | 2220 | 2253 | 2169 |
| Claude Code 2.1.319 + DeepSeek-V4.1-Flash | K = 2 | 1899 | 2183 | 2106 | 2110 | 2130 | 2056 |
| dsh 0.2.2 + DeepSeek-V4.1-Flash | K = 2 | 1886 | 2082 | – | – | – | 2039 |
| Claude Code 2.1.319 + Claude Opus 5.5 | K = 2 | 2411 | 2445 | 2397 | 2351 | 2275 | 2154 |
Each row is a separate fit, so only differences within a row are meaningful. The dsh harness ablation runs three arms (K = 1, 2 and 10) with the allocation measured in Claude Code (App. C.6). Gains quoted in the text are computed from unrounded ratings and may differ by ±1 from differences of the rounded values in the table.
Comparing the K = 1 and K = 2 columns gives the direct comparison: where the rule gives K = 2, two sessions beat one by 284, 196 and 33 Elo; for Gemini, where it gives K = 1, one session beats two by 80 Elo. The predicted allocation ranks first in every row, also against K = 3 to 5. Where self-Elo runs parallel to the sampling reference, neighbouring allocations rate alike, as the robustness paragraph of App. C.6 derives: for Claude Code + DeepSeek-V4.1-Flash, K = 2 to 5 lie within 77 Elo of each other.
We report K = 10 because it is the opposite extreme. K = 1 continues a single session, while K = 10 runs the many short independent sessions of our design, the closest it comes to repeated sampling. Gains over both show that the predicted allocation beats each of the two default strategies.
Okay, I will raise my score.
We thank Reviewer for the thoughtful feedback and for recognizing the usefulness of our Elo-per-token framework and allocation findings. Your suggestions on differentiating from classical restart strategies and validating the inflection point on independent held-out runs significantly strengthened the paper. We have incorporated these new analyses and experiments into the revision and detail our responses below:
1. Advantage of Elo-per-token over existing strategies: As far as we know, Elo-per-token is the first principled metric for analyzing how research agents scale at test time without being skewed by arbitrary score scales. As Reviewer SYWZ noted, our theoretical baseline is exact rather than an approximation: for independent trials with continuous scores, best-of-n beats best-of-m with probability n/(n+m), which translates directly to 400 Elo per 10× compute. This gives us a clear yardstick to measure diminishing returns in long agent sessions and compare them against independent sampling. Practically, it also yields a simple rule for deciding when restarting a session is more token-efficient than letting it continue.
2. Inflection Point in Practice: Thanks for pointing out. We now report held-out results with pilot runs under your suggestion on FrontierCS P0, along with evaluations on three new harness–model pairs using the latest models and harnesses (App. C.6). Specifically, the allocation rule is fitted on pilot sessions and evaluated exclusively on fresh, held-out sessions. These results provide further empirical validation of our allocation rule. Moreover, we demonstrate that the rule can transfer across harnesses: for DSH + DeepSeek-V4.1-Flash, we used the inflection point profiled by Claude Code with the same model and still observed significant Elo gains. We hope these results address your concerns regarding our inflection point robustness and practical applicability raised in Weakness 2 and Question 2.
Thank you again for suggesting the validation protocol, which has made the paper stronger. We hope the results address your concerns, and if so, we would be grateful if you would consider raising your score.