acceptodds
Under review as a conference paper at ICLR 2027

When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis

154 views

Abstract

Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alternatives, and decide when to stop. This test-time strategy makes it difficult to measure how agent performance scales. We study open-ended tasks that provide continuous scores for intermediate submissions, making progress observable throughout long trajectories. We propose Elo-per-token analysis, which tracks the best solution found at each token budget and uses a Bradley–Terry model to aggregate within-task orderings into Elo ratings across tasks with different score scales. We apply it to four general-purpose agents on four open-ended benchmarks, with sessions of up to M tokens, and to three feedback-driven LLM optimization harnesses in controlled single-task interventions. Independent sampling provides a theoretically characterized reference, for which Elo grows linearly with log compute. Against this reference, agents can initially convert tokens into Elo faster than independent sampling, but their marginal gains diminish and eventually fall below the reference. In contrast, the strongest historical human contestants improve superlinearly over contest time on shared AtCoder Heuristic Contest tasks, providing evidence of continual learning and substantial headroom after agents slow down. We define the scaling inflection point as the per-session budget where marginal Elo gains match the independent-sampling reference. Using this point as the per-session budget, we split 100M tokens across parallel sessions on FrontierCS Polyomino Packing, gaining +264 Elo over one long session and +355 over ten short sessions.

Primary area:
generative models
Keywords:
LLM Agents, Test-time Scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WvCAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how the performance of LLM agents scales with increasing test-time compute, particularly when continuing iterative refinement within a single agent session becomes less efficient than running multiple independent sessions. The authors further propose a strategy based on the Scaling Inflection Point to determine how a fixed token budget should be allocated across independent sessions.

Experiments are conducted with four general-purpose agent systems on 14 open-ended optimization tasks across four benchmarks. The results show that increasing test-time compute can substantially improve agent performance at early stages, but the marginal gains diminish at larger budgets. The compute-allocation experiments suggest that an appropriate number of independent sessions can outperform both a single long session and many short sessions.

Critical strengths
  1. How to allocate a fixed test-time compute budget between iterative refinement and independent exploration is an important research topic. The proposed Elo-per-token framework provides a way to measure agent scaling behavior across tasks with different evaluation metrics. It captures how solution quality improves throughout long agent trajectories and offers a criterion for deciding how to balance continued refinement and independent exploration.

  2. The paper evaluates four frontier agent systems. The experiments reveal interesting patterns of diminishing marginal returns in long-running agent sessions. More importantly, the authors attempt to translate these observations into a practical compute-allocation rule. The results on Polyomino Packing demonstrate that selecting an appropriate number of independent sessions can significantly improve performance under a fixed total token budget.

Critical weaknesses
  1. The observation that model performance eventually exhibits diminishing marginal returns as test-time compute increases is not entirely new. Furthermore, the trade-off between continuing an existing optimization trajectory and restarting independent searches has been extensively studied, including classical Restart Strategies and Bet-and-Run methods.

  2. Therefore, the main novelty of this paper appears to lie in the Elo-per-token formulation and the resulting Inflection-point Allocation Rule. Although the authors theoretically derive a criterion for allocating tokens to individual sessions, the experimental design does not sufficiently validate its practical reliability. For example, the authors could first estimate the optimal session budget (b) using a set of pilot runs, and then compare it against alternative budget allocations on independent held-out runs to demonstrate that the predicted allocation achieves competitive performance.

Questions for authors

Q1. What is the unique advantage of Elo-per-token over existing scaling analyses and restart strategies?

Q2. How robust is the 400-Elo-based Inflection Point in practice? For example, could the authors estimate the Inflection Point using one set of pilot sessions and then evaluate the predicted budget allocation on completely independent sessions?

4 comments

We thank Reviewer for the thoughtful feedback and for recognizing the usefulness of our Elo-per-token framework and allocation findings. Your suggestions on differentiating from classical restart strategies and validating the inflection point on independent held-out runs significantly strengthened the paper. We have incorporated these new analyses and experiments into the revision and detail our responses below:

1. Advantage of Elo-per-token over existing strategies: As far as we know, Elo-per-token is the first principled metric for analyzing how research agents scale at test time without being skewed by arbitrary score scales. As Reviewer SYWZ noted, our theoretical baseline is exact rather than an approximation: for independent trials with continuous scores, best-of-nn beats best-of-mm with probability n/(n+m)n/(n+m), which translates directly to 400 Elo per 10× compute. This gives us a clear yardstick to measure diminishing returns in long agent sessions and compare them against independent sampling. Practically, it also yields a simple rule for deciding when restarting a session is more token-efficient than letting it continue.

While our primary focus is characterizing agents' test-time scaling free from task-specific score distortions, we agree that the resulting allocation rule overlaps with classical optimization strategies. The key distinction lies in how the switching point is found: existing restart methods tune it empirically or depend on task-specific score metrics, whereas our framework derives it directly from the observed curve and theoretical analysis.

2. Inflection Point in Practice: Thanks for pointing out. We now report held-out results with pilot runs under your suggestion on FrontierCS P0, along with evaluations on three new harness–model pairs using the latest models and harnesses (App. C.6). Specifically, the allocation rule is fitted on pilot sessions and evaluated exclusively on fresh, held-out sessions. These results provide further empirical validation of our allocation rule. Moreover, we demonstrate that the rule can transfer across harnesses: for DSH + DeepSeek-V4.1-Flash, we used the inflection point profiled by Claude Code with the same model and still observed significant Elo gains. We hope these results address your concerns regarding our inflection point robustness and practical applicability raised in Weakness 2 and Question 2.

SystemPilotInflection point → KElo Gain over K = 1 (higher the better)Elo Gain over K = 10 (higher the better)
Gemini CLI 0.67.0 + Carbon5 ×\times 200MK = 1+0+399
Claude Code 2.1.319 + DeepSeek-V4.1-Flash10 ×\times 100MK = 2+284+127
DSH + DeepSeek-V4.1-Flash10 ×\times 100M (Claude Code)K = 2+196+43
Claude Code 2.1.319 + Claude Opus 5.510 ×\times 100MK = 2+33+291

Thank you again for suggesting the validation protocol, which has made the paper stronger. We hope the results address your concerns, and if so, we would be grateful if you would consider raising your score.

Thank the authors for clarifying my concerns. But for Q2, since the calculated K is either 1 or 2, why not add a comparison between K=1 and K=2, or include other K values such as 3, instead of directly jumping to K=10?

Thank you for the follow-up. Every held-out pool already contains these allocations. As in Sec. 5, each pool uses the nested design (five sessions each at 100M, 50M, 33.3M, 25M and 20M, and twenty-five at 10M), so one tournament rates K = 1, 2, 3, 4, 5 and 10 at the same total budget of 100M tokens (App. C.6). Our earlier table summarized only the gains over the two extremes. The full results are below (joint Elo at 100M; best allocation in bold):

System (harness + model)Inflection point → KK = 1K = 2K = 3K = 4K = 5K = 10
Gemini CLI 0.67.0 + carbonK = 1256824872260222022532169
Claude Code 2.1.319 + DeepSeek-V4.1-FlashK = 2189921832106211021302056
dsh 0.2.2 + DeepSeek-V4.1-FlashK = 218862082–––2039
Claude Code 2.1.319 + Claude Opus 5.5K = 2241124452397235122752154

Each row is a separate fit, so only differences within a row are meaningful. The dsh harness ablation runs three arms (K = 1, 2 and 10) with the allocation measured in Claude Code (App. C.6). Gains quoted in the text are computed from unrounded ratings and may differ by ±1 from differences of the rounded values in the table.

Comparing the K = 1 and K = 2 columns gives the direct comparison: where the rule gives K = 2, two sessions beat one by 284, 196 and 33 Elo; for Gemini, where it gives K = 1, one session beats two by 80 Elo. The predicted allocation ranks first in every row, also against K = 3 to 5. Where self-Elo runs parallel to the sampling reference, neighbouring allocations rate alike, as the robustness paragraph of App. C.6 derives: for Claude Code + DeepSeek-V4.1-Flash, K = 2 to 5 lie within 77 Elo of each other.

We report K = 10 because it is the opposite extreme. K = 1 continues a single session, while K = 10 runs the many short independent sessions of our design, the closest it comes to repeated sampling. Gains over both show that the predicted allocation beats each of the two default strategies.

Okay, I will raise my score.

Reviewer_bFgdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Elo-per-token, an evaluation framework that uses pairwise comparisons of best-so-far solutions and Bradley–Terry ratings to characterize how agent performance scales with test-time token budgets. It derives a theoretical independent-sampling reference of 400 Elo per tenfold increase in compute. Experiments with four agent systems across fourteen tasks show diminishing marginal gains at large budgets, while historical human contestants exhibit accelerating Elo gains in log contest time. The paper further uses the estimated slope crossing to guide allocation across independent sessions, reporting improvements over a single long session on Polyomino Packing.

Critical strengths
  1. Relevant perspective on agent efficiency. The paper examines the marginal value of extending an agent session, going beyond final performance at a fixed budget. This perspective helps characterize when additional computation yields diminishing improvements.
  2. Trajectory-level empirical analysis. The evaluation tracks intermediate submissions from four agent systems across fourteen tasks, with budgets up to 100M tokens per session. These records provide a view of how solution quality evolves and document diminishing returns under the evaluated settings.
Critical weaknesses
  1. The analysis remains primarily descriptive. Elo-per-token provides a framework for measuring test-time scaling, and the experiments document diminishing returns at large budgets. However, the study does not experimentally identify which agent behaviors or system components drive this pattern. Although possible explanations are discussed, the paper does not establish specific design insights whose effectiveness in improving long-horizon agent performance is validated.
  2. The practical value of the allocation rule is not sufficiently established. The scaling threshold is estimated and the resulting allocations are evaluated using the same session pool, without held-out validation. The cost of prior task-level profiling is also not incorporated into the reported gains. With allocation experiments covering only two tasks and one agent system, the evidence does not establish whether the threshold reliably guides finite-budget decisions or provides a cost-effective advantage on new tasks.
Questions for authors
  1. Added value of Elo-per-token. What substantive conclusions or decisions does Elo-per-token enable beyond per-task score curves and direct pairwise win-rate comparisons? Can the authors demonstrate a case where the proposed framework provides additional insight that these simpler analyses do not?
  2. Explanatory findings about agent behavior. Beyond observing diminishing returns, which competing explanations for agent slowdown can the analysis distinguish? What experimentally supported finding leads to a testable implication for improving agent strategies?
  3. Predictive value of the allocation rule. Can the proposed threshold predict effective allocations on independent runs or new tasks, rather than characterize allocations retrospectively? After accounting for profiling costs, what consistent advantage does it offer over simple allocation and restart heuristics?
  4. Scope of the claimed scaling pattern. What evidence distinguishes a general property of agent test-time scaling from behavior specific to the evaluated tasks and execution protocols? Do the central findings persist when compute accounting and termination handling are varied?
2 comments

We thank the reviewer for the insightful questions, and for recognizing the value of our agent-efficiency perspective and trajectory-level analysis. Directly addressing your central request, we have added rigorous held-out validation experiments in both the general response and the revision, complemented by new analyses on context compaction and profiling cost. We address each point below:

1. Advantage of Elo-per-token over existing strategies: As far as we know, Elo-per-token is the first principled metric for analyzing how research agents scale at test time without being skewed by arbitrary score scales. Furthermore, compared to direct pairwise win-rate comparisons, Elo ratings account for opponent strength, distinguishing the significance of outperforming strong submissions versus weak ones. More importantly, it enables us to construct a continuous scaling curve for direct comparison against repeated sampling. If we use a pairwise win rate, the reference curves for repeated sampling will be nonlinear, so we cannot derive the inflection point from them.

Moreover, as Reviewer SYWZ noted, our theoretical baseline is exact rather than an approximation: for independent trials with continuous scores, best-of-$n$ beats best-of-$m$ with probability $n/(n+m)$, which translates directly to 400 Elo per 10× compute. This gives us a clear yardstick to measure diminishing returns in long agent sessions and compare them against independent sampling. Practically, it also yields a simple rule for deciding when restarting a session is more token-efficient than letting it continue. 

2. Reasons behind Diminishing Gain We appreciate this point. To further analyze the failure reasons, we conducted an additional study examining the relationship between context compaction and Elo performance gain (details in App. A.2), yielding the results below:

WindowMedian GainMean Gain
Window 1+835+837
Window 2+0+59
Window 300

As shown in the table, flattening already occurs within the first window, and compacting into a fresh window fails to restore the slope, ruling out simple context exhaustion.

More broadly, we trace this flattening to how harnesses handle accumulated context (Sec. 1, Sec. 5, App. A.1–A.2). Once past the inflection point, independent branches overtake a continuous session (post-inflection harness-failure theorem) because of a "sticky-basin" effect—the longer the context grows, the harder it becomes for the agent to break away from early solution paths. We also show that this isn't an artifact of any single component: the plateau holds up across four harnesses, outer loops (AdaEvolve, GEPA), test-time training (TTT-Discover), and four Qwen3.5 model sizes (Sec. 4, App. C.2, C.5). Taken together (App. A.2, Table A), this gives a clear takeaway: restart or branch out whenever the slope drops below 400 Elo per decade, which we validate on both allocation and held-out experiments.

3. Inflection Point in Practice: Thanks for pointing this out. We now report held-out results with pilot runs under your suggestion on FrontierCS P0, along with evaluations on three new harness–model pairs using the latest models and harnesses (App. C.6). Specifically, the allocation rule is fitted on pilot sessions and evaluated exclusively on fresh, held-out sessions. These results provide further empirical validation of our allocation rule. Moreover, we demonstrate that the rule can transfer across harnesses: for DSH + DeepSeek-V4.1-Flash, we used the inflection point profiled by Claude Code with the same model and still observed significant Elo gains. We also note that our appendix already includes evaluations across other distinct tasks, further demonstrating the applicability of our framework beyond a single benchmark.** We hope these results address your concerns regarding our inflection point robustness and practical applicability raised in Weakness 2 and Question 2.

SystemPilotInflection point → KElo Gain over K = 1 (higher the better)Elo Gain over K = 10 (higher the better)
Gemini CLI 0.67.0 + Carbon5 ×\times 200MK = 1+0+399
Claude Code 2.1.319 + DeepSeek-V4.1-Flash10 ×\times 100MK = 2+284+127
DSH + DeepSeek-V4.1-Flash10 ×\times 100M (Claude Code)K = 2+196+43
Claude Code 2.1.319 + Claude Opus 5.510 ×\times 100MK = 2+33+291

(Continued in the next comment.)

4. Cost We agree that measuring the inflection point has a cost, and the revision states it (App. C.6). Two numbers put it in proportion. Under the independent-sampling reference, a gain of +264 Elo is worth a 4.6$\times$ larger budget 10264/40010^{264/400} . A pilot of about 8–10 times the budget already yields a stable K: on the original pool, a pilot of two of the five groups gives K = 2 or 3 in every subset. Profiling therefore pays off when the same task, or a family of similar tasks, is solved repeatedly, after about two to three uses.

We hope these answers and the new held-out results address your questions. If they do, we would be grateful if you would consider raising your score, and we would be glad to clarify anything further.

Reviewer_SYwzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether LLM agents' adaptive test-time strategies turn extra tokens into quality faster or slower than independent restarts. The proposed Elo-per-token analysis aggregates within-task orderings of best-so-far scores into Bradley–Terry Elo ratings at each token budget; for i.i.d. equal-cost attempts, best-of-n sampling provably gains 400 Elo per decade, a distribution-free reference. The central claim is that the agents' fitted slopes can initially exceed this reference but their pooled slopes eventually fall below it, whereas the strongest human contestants improve superlinearly in log contest time. Kimi Code, Codex, Claude Code, and Gemini CLI run five sessions of up to 100M cache-inclusive tokens on each of fourteen tasks from four open-ended benchmarks; the authors also analyze AtCoder Heuristic Contest histories and propose a “scaling inflection point” rule for splitting a fixed budget across parallel sessions.

Critical strengths

The sampling reference is exact, not approximate: for i.i.d. continuous scores, best-of-n beats best-of-m with probability n/(n+m), itself a Bradley–Terry form, so 400 Elo per decade is the model's own null. Local self-Elo slopes therefore test, independently of score units, whether continuing a session still pays more per token than restarting.

Critical weaknesses
  1. My primary concern is the practical value of this analysis. Claude Code and Codex run only as single long sessions, resumed by a watchdog whenever they exit early (App. B.2.1); CLI versions and sub-agent settings are unreported, and the allocation rule is tested only with Kimi Code. Would the findings change how practitioners configure such harnesses built in sub-agents versus K independent sessions, and does the flattening persist when the harness itself decides when to stop, compact context, or delegate?

  2. The +264 Elo allocation gain is in-sample on a single task (“no held-out runs,” Sec. 5), and its 1×100M baseline is always one of the five sessions filling the 100M slots, which already trail at 50M (mean 0.820 versus 0.862 for the 50M-slot sessions; Fig. C).

Questions for authors

N/A

1 comment

We thank the reviewer for the detailed and constructive feedback, and for recognizing the exactness of our 400-Elo-per-decade sampling reference. Your suggestions helped us significantly improve the empirical rigor and clarity of our work. We address each point in detail below:

1. Practical value of analysis: Thanks for pointing this out. With additional experiments in the revision (also as shown in the table in weakness 2), we demonstrate that flattening persists across multiple model–harness pairs and that the allocation rule generalizes beyond Kimi + Kimi Code to other agents, including those powered by the latest models, such as Claude Opus 5.5, Gemini 4 Carbon, and DeepSeek V4.1 Flash 1017.

Regarding the setting, the revision adds the harness versions and settings to the App. B.2.1 (Table B): Claude Code 2.1.197 (Claude Opus 4.8), Codex CLI 0.142.5 (GPT-5.5), Gemini CLI 0.55.1 (Gemini 3.5 Flash), and Kimi Code 0.37.0 (Kimi K2.7). Sub-agents and context compaction were left at each harness's defaults; we disabled neither. 

The main difference between subagents and our allocation strategy is that subagents are not independent of the main session. In our experiments, Kimi Code and Gemini CLI used more subagents than other harnesses, such as Claude Code and Codex, yet they exhibited similarly flattening Elo-per-token curves. This suggests that our findings also hold in settings involving subagents.

Regarding termination criteria determined by the harness itself, we compared specialized harnesses such as AdaEvolve, GEPA, and TTT-Discover (which also performs **model weight updates** at test time) in the paper. All of them have their own stopping criteria (e.g., AdaEvolve using an UCB-based algorithm to decide when to change or flash the context), yet exhibit similar diminishing returns in their Elo-per-token curves.

2. No Held-out Runs: Thanks for the suggestion. We now report held-out results following the protocol suggested by WvCA on FrontierCS P0, along with evaluations on three new harness–model pairs using the latest models and harnesses (App. C.6). Specifically, the allocation rule is fitted on pilot sessions and evaluated exclusively on fresh, held-out sessions. These results provide further empirical validation of our allocation rule. Moreover, we demonstrate that the rule can transfer across harnesses: for DSH + DeepSeek-V4.1-Flash, we used the inflection point profiled by Claude Code with the same model and still observed significant Elo gains.

SystemPilotInflection point → KElo Gain over K = 1 (higher the better)Elo Gain over K = 10 (higher the better)
Gemini CLI 0.67.0 + Carbon5 ×\times 200MK = 1+0+399
Claude Code 2.1.319 + DeepSeek-V4.1-Flash10 ×\times 100MK = 2+284+127
DSH + DeepSeek-V4.1-Flash10 ×\times 100M (Claude Code)K = 2+196+43
Claude Code 2.1.319 + Claude Opus 5.510 ×\times 100MK = 2+33+291

We hope these clarifications and the new held-out results address your concerns. If so, we would be grateful if you would consider raising your score, and we are happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.