- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how model capabilities relate to performance growth in long-horizon agent tasks, using AutoLab and EdgeBench (10 categories, 10 models, 71 tasks). The authors project 29 external benchmark scores onto five principal components and, for each category, use two separate linear readouts of these coordinates to set the starting level and the growth rate of a logistic power law in log time. Fitted on early checkpoints, the curves predict later category-mean scores (mean R² = 0.844, RMSE 2.31). The paper then shows that while average scores keep rising, the share of models still improving on each task falls by 35.2 percentage points from the first to the last interval, and relates this to predicted mean gains and to external benchmark profiles. Finally, it proposes a continuation policy that conditions the growth curve on the current run and decides whether to buy one more interval; in replay it saves about one-third of full-run time with 2.4% (AutoLab) and 3.3% (EdgeBench) relative score loss.
Critical strengths
- Separating the starting level from later growth is a useful idea. Modeling the two with separate readouts is a natural way to express that similar early scores can be followed by different later gains. The observation that average scores rise while fewer models keep improving is useful for long-horizon evaluation, and it holds in all ten categories.
- The question is practically relevant. Deciding when to stop a long run can save substantial compute, and the authors plan to release the AutoLab trajectories and code.
Critical weaknesses
- With so few models and tasks, it is hard to tell how reliable many conclusions are. Each category has only 4 to 5 models, and several categories have only 2 to 5 tasks. Some conclusions are sensitive to small changes: removing a single model changes the final-score versus terminal-frequency difference from 0.268 to anywhere between 0.089 and 0.355, and DeepSWE's correlation in Scientific & ML goes from +1.00 in the default setting to −0.77 when only the final interval and a 0.5-point threshold are used. Appendix B.3 calls these associations exploratory, but the main text presents them as findings and contributions. The paper should separate its core claims from exploratory observations.
- Many formulas and components are introduced, but their necessity is often unclear. In particular, the capability representation, which is central to the framing, plays a limited role. In Section 3, Appendix A.4 notes that with 4 to 5 models the intercept-plus-PC design has full row rank, so the PCs mainly determine the direction of regularization rather than constrain the fit. Section 4.3 uses raw benchmarks instead of the PCs, and the improvement-frequency model gains little over using time alone (Brier 0.2071 vs 0.2187).
Questions for authors
- The extrapolation is short. Category forecasts are fit up to 60% budget (AutoLab) or 8 hours (EdgeBench) and predict at most 1.67x or 1.5x more compute. How do the forecasts behave with a shorter fitting window, for example fitting 20 to 40% and predicting 60 to 100%? Relatedly, the continuation policy forecasts endpoint gains from checkpoints as early as 5% of budget. How accurate and calibrated are these endpoint forecasts?
- The first and last intervals differ in log time: the first AutoLab interval (20 to 40%) doubles compute, while the last (80 to 100%) adds only 1.25x; EdgeBench is similar (2 to 4 hours versus 10 to 12 hours). How much of the participation decline remains with intervals of equal log length, and after excluding tasks that are saturated or have terminated?