acceptodds
Under review as a conference paper at ICLR 2027

Long-Horizon Scaling: How Model Capabilities Shape the Returns to Computation

Abstract

Long-horizon agents improve solutions through sustained interaction, execution, and task feedback. Scaling studies relate performance to resources and capabilities, yet how existing capabilities shape returns to extended interaction remains less understood. To address this gap, we analyze AutoLab and EdgeBench, two long-horizon benchmarks. We find that starting performance and subsequent growth are associated with different capabilities: within a task category, similar early scores can precede different later gains. To formalize this finding, we model capability–time scaling with category-specific logistic power laws shared across models. Fitted to early trajectories, these curves extrapolate the observed models' category-average scores to later computation. However, rising average scores mask narrowing improvement opportunities: later gains concentrate among fewer improving models. High final scores and continued improvement also have distinct capability profiles. Predicted mean gains estimate each model's fraction of improving tasks; averaging these estimates forecasts the average share of improving models. These uneven returns motivate deciding whether a specific run should continue. We therefore derive a continuation policy to save time and compute with limited score loss. The policy conditions growth predictions on the run's observed progress and weighs immediate and delayed gains against computation costs. In replay with training and price calibration based on other models' histories, the policy saves roughly one-third of full-run time, with relative score losses of on AutoLab individual runs and on EdgeBench published mean curves.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling laws, long-horizon agents, test-time compute

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_FUs8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how model capabilities relate to performance growth in long-horizon agent tasks, using AutoLab and EdgeBench (10 categories, 10 models, 71 tasks). The authors project 29 external benchmark scores onto five principal components and, for each category, use two separate linear readouts of these coordinates to set the starting level and the growth rate of a logistic power law in log time. Fitted on early checkpoints, the curves predict later category-mean scores (mean R² = 0.844, RMSE 2.31). The paper then shows that while average scores keep rising, the share of models still improving on each task falls by 35.2 percentage points from the first to the last interval, and relates this to predicted mean gains and to external benchmark profiles. Finally, it proposes a continuation policy that conditions the growth curve on the current run and decides whether to buy one more interval; in replay it saves about one-third of full-run time with 2.4% (AutoLab) and 3.3% (EdgeBench) relative score loss.

Critical strengths
  1. Separating the starting level from later growth is a useful idea. Modeling the two with separate readouts is a natural way to express that similar early scores can be followed by different later gains. The observation that average scores rise while fewer models keep improving is useful for long-horizon evaluation, and it holds in all ten categories.
  2. The question is practically relevant. Deciding when to stop a long run can save substantial compute, and the authors plan to release the AutoLab trajectories and code.
Critical weaknesses
  1. With so few models and tasks, it is hard to tell how reliable many conclusions are. Each category has only 4 to 5 models, and several categories have only 2 to 5 tasks. Some conclusions are sensitive to small changes: removing a single model changes the final-score versus terminal-frequency difference from 0.268 to anywhere between 0.089 and 0.355, and DeepSWE's correlation in Scientific & ML goes from +1.00 in the default setting to −0.77 when only the final interval and a 0.5-point threshold are used. Appendix B.3 calls these associations exploratory, but the main text presents them as findings and contributions. The paper should separate its core claims from exploratory observations.
  2. Many formulas and components are introduced, but their necessity is often unclear. In particular, the capability representation, which is central to the framing, plays a limited role. In Section 3, Appendix A.4 notes that with 4 to 5 models the intercept-plus-PC design has full row rank, so the PCs mainly determine the direction of regularization rather than constrain the fit. Section 4.3 uses raw benchmarks instead of the PCs, and the improvement-frequency model gains little over using time alone (Brier 0.2071 vs 0.2187).
Questions for authors
  1. The extrapolation is short. Category forecasts are fit up to 60% budget (AutoLab) or 8 hours (EdgeBench) and predict at most 1.67x or 1.5x more compute. How do the forecasts behave with a shorter fitting window, for example fitting 20 to 40% and predicting 60 to 100%? Relatedly, the continuation policy forecasts endpoint gains from checkpoints as early as 5% of budget. How accurate and calibrated are these endpoint forecasts?
  2. The first and last intervals differ in log time: the first AutoLab interval (20 to 40%) doubles compute, while the last (80 to 100%) adds only 1.25x; EdgeBench is similar (2 to 4 hours versus 10 to 12 hours). How much of the participation decline remains with intervals of equal log length, and after excluding tasks that are saturated or have terminated?
Reviewer_QmTGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the performance of LLM agents improves over long-horizon tasks and when an agent run should be stopped. The authors use trajectories from AutoLab and EdgeBench across 71 tasks in 10 categories with 4-5 models per category. The authors represent each model with 5 principal components from 29 external benchmark scores and fit a logistic curve in log time for each category where the starting score and growth depend on the model components. The curve is fit on early checkpoints and used to predict the extrapolation on later checkpoints. The authors observe that the fraction of models that still improve drops towards the end and benchmarks with final score are not the same as benchmarks with continued improvement. The authors also propose a policy that predicts remaining gain from the current run and stops if the gain is not meaningful for the time spent. The results show that this policy saves more time than a fixed budget for the same score on both AutoLab and EdgeBench.

Critical strengths

S1. Autonomous coding agents are used for intensive long-horizon research and engineering tasks taking days to weeks. It is extremely important to understand how much extra time is worth it for the success of an agent and when it should be terminated. This work tries to explore this question.

S2. The idea of a shared capability space with category-specific readouts is really simple and makes it easier to track specific benchmarks related to early performance and other ones improvements over time. The forecasting setup fits the coefficients and regularization using early checkpoints and compares against baselines such as the last score, fitting independent curves for each model and using model indicators.

S3. The paper shows an interesting observation that only a few models keep improving towards the end. This suggests that later gains come from fewer models instead of being spread unevenly across the same models. This seems to hold true even when using more tasks or different improvement thresholds.

S4. I find the policy to terminate an agent quite practical and it doesn't depend on the target model, which is interesting for adaptation across different models. The results show that this policy retains a much higher performance with 50% time savings compared to a fixed budget.

Critical weaknesses

W1. There are very few models in each category (only 4 or 5) for the proposed model. The authors mention that with this many models, the 5 principal components can fit each model exactly and work similarly to model indicators. I think the forecasting results only show that the curve can predict later checkpoints for the same models. It is not clear if the external benchmarks can predict the curve for a new model. It would be more convincing to keep a model as held out and predict its curve only from its benchmark scores.

W2. The number of components is chosen partly based on the errors on the later checkpoints which are also used for evaluation. If only the early checkpoints are used for this choice then the model picks 19 components and the error is almost the same as fitting a separate curve for each model. The results are currently not consistent across categories.

W3. It may be likely that higher correlations are by chance due to fewer models and many benchmarks share the exact same ranking. The results have a lot of variance when either of the models is dropped. I think these results should be treated as exploratory and the claims about which benchmarks relate to later improvement should be more grounded.

W4. It may be likely that the drop in the number of improving models is a ceiling effect as models which already have higher scores early on have little room for improvement later. It is unclear if the claim about the drop in models will hold after calibrating for how much room each model has left.

W5. There is a marginal improvement in following policy for continuation/termination compared to heuristics for EdgeBench. I think the estimations for EdgeBench are computed using the smoothed trajectories and not on realistic scenarios factoring in run-to-run variation.

Questions for authors
  1. How does the capability-time model perform when a whole model is held out and its trajectory is predicted only from its external benchmark scores?

  2. How do the forecasting results change when the number of principal components is selected using only early checkpoints?

  3. Why does the budget target result in different realized costs on AutoLab? Is there a way to calibrate the price so that the realized cost is closer to the target?

Reviewer_frM4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper studies how model capabilities affect the benefits of scaling compute in long-horizon agent tasks, such as AutoLab and EdgeBench. The authors describe each model using principal components derived from its performance on 29 external benchmarks, then fit a logistic power law for each task category. The fitted curves can predict later performance using data from earlier checkpoints. The authors also find that although average performance continues to improve with more compute, fewer models show progress on individual tasks. Based on these findings, they propose a continue/stop strategy that uses the progress made during a run to update the predicted growth curve. The strategy continues execution only when the expected performance gain justifies the compute cost, reducing runtime by about one-third.

Critical strengths
  1. The paper addresses a meaningful and practical problem.

  2. The main modeling approach is simple and well motivated.

  3. The experiments are carefully designed and cover several aspects of the problem.

Critical weaknesses
  1. Each task category contains only 4–5 models, while the capability vector has five dimensions plus an intercept. This means the model can fit the observed starting performance and growth rates exactly. The appendix also acknowledges that the design matrix has full rank and that the regularizer determines the fitted coefficients. As a result, it is difficult to draw meaningful conclusions about which capabilities actually drive performance growth. This is a major concern because understanding that relationship is one of the paper's main goals.

  2. The capability profiles are based on 290 Spearman correlations, each computed using only about five models. With five models, even random rankings produce an absolute correlation of at least 0.7 roughly 23% of the time. The external benchmarks are also highly correlated with one another. These issues raise concerns about how reliable the estimated capability profiles are.

  3. The paper is unnecessarily difficult to follow. It introduces many new terms, and several explanations are packed into long, complicated sentences. I often had to refer back to the appendix to understand the main text.

Questions for authors

Given the small number of models and the number of fitted parameters, how can the authors determine whether the estimated coefficients reflect meaningful relationships between capabilities and performance growth, rather than simply fitting the observed data?

Reviewer_JsXROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how external model capabilities relate to performance growth during long-horizon agent execution. It represents models using principal components of 29 external benchmark scores and fits category-specific logistic power laws whose starting levels and temporal coefficients depend on these coordinates. Using AutoLab and EdgeBench trajectories, the authors evaluate temporal extrapolation, characterize declining participation in later improvements, and relate improvement patterns to external benchmark profiles. They further introduce a continuation policy that conditions reference growth on observed run progress and compares predicted immediate and remaining-budget gains against computation costs. Held-out-model replay reports roughly one-third time savings with small average score losses.

Critical strengths
  1. The paper connects trajectory analysis to a practically relevant allocation decision. It distinguishes high final performance from continued improvement and evaluates whether further execution is worthwhile for a particular run. The continuation rule considers both immediate and delayed gains, which is appropriate when useful progress can follow a period of stagnation. The fixed-plan comparison and sign-error bound give a clear theoretical interpretation under the stated assumptions.

  2. The evaluation includes meaningful safeguards and useful sensitivity analyses. The category panels maintain fixed task and model membership across time. The authors compare against persistence, independent time curves, and model indicators; examine alternative improvement thresholds and panel definitions; and report source-calibrated operating points separately from swept frontiers. In continuation replay, target-model histories are excluded from training, and AutoLab uses individual replicated runs. These choices strengthen the empirical study.

Critical weaknesses
  1. The predictive value of external capabilities is not sufficiently established. Each category contains only four or five models, while the five-PC representation plus an intercept can express arbitrary model-specific levels and slopes (Appendix A.4). Moreover, the five-PC choice uses later-checkpoint errors. With strictly early-only dimension selection, RMSE is 2.434–2.486, close to 2.52 for independent time curves. These results support temporal extrapolation more clearly than a transferable capability–growth relationship. Held-out-model forecasting and permuted-feature controls would strengthen this claim.

  2. The continuation gains are not isolated from observed progress and task history. The policy combines capability-based growth with seven prefix features and learned task effects, but lacks a matched learned baseline without capability inputs. It is therefore unclear how much the capability representation contributes. Evaluation also permits source histories on the same tasks, while EdgeBench uses averaged trajectories rather than individual runs. These choices limit evidence for deployment on unseen tasks and stochastic individual runs.

Questions for authors
  1. Can you report the main forecasting results with all representation choices selected exclusively from training data, together with uncertainty relative to the independent-curve and model-indicator controls? Does the capability representation improve prediction of future increments, rather than primarily predicting differences in score levels?

  2. Can you evaluate forecasting on held-out models and compare against permuted capability profiles or random embeddings of the same dimension? These controls would help establish whether the external coordinates contain useful growth information beyond their ability to identify the small model panel.

  3. How does continuation perform when the capability-dependent reference is replaced with a category-only reference, while retaining the same prefix features, task effects, training procedure, and price calibration? An ablation of task effects and a learned gain predictor without the proposed scaling structure would further clarify the source of the improvement.

  4. Can you use AutoLab to compare stopping on individual runs against stopping on their averaged curves, thereby quantifying the effect of the EdgeBench evaluation protocol? How much performance is retained when tasks, rather than only models, are held out?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.