acceptodds
Under review as a conference paper at ICLR 2027

Frontier Scaling Laws for Agentic Cost: Measuring Diminishing Returns in AI Agents

Abstract

The last few years have seen a dramatic acceleration in AI capabilities, most notably in the time horizon metric, which relates the success rate of an AI agent to the number of hours it would take a human to complete the same task. When measured in human-hours, the tasks an AI agent can complete 50% of the time have been doubling approximately every 4 months for the last three years. At the same time the direct inference-time cost of running an AI agent has also experienced rapid growth. Thus, the question remains: which of these two is growing more quickly? That is, for every doubling of time horizon, by what factor does inference-time cost grow? Building on methods from item response theory, we show that at the price-performance frontier there are significant diminishing returns to paying more for AI inference. In particular, we find that cost scales as a superquadratic power law in time horizon.

Primary area:
foundation or frontier models, including LLMs
Keywords:
frontier scaling laws, item response theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Hn5XOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper connects the time horizon of tasks (in the software domain mostly) following METR's papers on human hours per task with the inference cost in dollars, using scaling laws fitted on benchmark results and IRT. The authors claim there are diminishing returns.

Critical strengths
  • The paper is very well-written and uses state-of-the-art data and the fits seem robust.
  • Gives useful information for understanding trends, in the line of published results by key third-parte evaluation organisations such as Epoch and METR.
Critical weaknesses

Misleading interpretation of task length: The time horizon is not "directly" comparable to a "human baseline via hourly wages", as they say in the introduction. You pay by hour, but not necessarily by continuous work on the same task. There is no given evidence that wages per hour are related to how long the tasks take. I imagine that a worker doing short repetitive tasks would be paid less per hour than one that does long projects, but a football player never has a task that lasts more than two hours and can earn a lot of money. Similarly for people at the boards of big multinationals. The paper doesn't depend on this assumption (longer tasks higher human cost per hour), but putting this prominently in the introduction is misleading. However, what does affect the whole study is the use of the volume of the task (not only by METR) as the key element of capability. First, because other capabilities also matter for the task, and some may be more important than others for the cost of the task, but most especially because METR's hours per task has to naturally saturate. Many meaningful tasks of hundreds of hours can be split into smaller tasks.

A bit circular: There's a very simple explanation of the curves in Kinniment et al. and METR's results: a simple survival analysis of the per-step probability of success, given the exponential distribution. As the probability of success for "chunks" increases then everything else increases, with formulas that derive easily by just assuming an independence of the chunks (the messiness in the original METR paper). But this is circular if then you use this to understand the capability. It's a good change of view, as there's a scale, in time, with units, hours, which is a big difference from aggregate benchmarks or populational-derived indicators as in Epoch's ECI, but it's what you expect to get.

Diminishing returns? I think the statement of diminishing returns is assuming we won't split the tasks (this is hinted at the end, suggesting human intervention, but there could be orchestrators with high executive functions and planning capabilities doing this splitting, like a software project manager). The long-term horizon is a good predictor, because it's simply the exponential distribution as in survival analysis, but that doesn't mean that tasks in the future should be composed of hundreds of hours.

Contributions: while doing experimental work with Epoch's and other contributor's data is useful for the community, this is a portfolio of data postprocessing with little technical contribution to ICLR in terms of theory, ideas, etc., or new experiments. Despite the influential papers from METR and Epoch (one because of the infamous title of 'Rosetta Stone'), IRT has been used in AI for at least a decade, and volume of a task is a well-known factor of complexity for planning, memory and other executive-function capabilities. But here it is just the exponental distribution of Prob(success)= p^k, without a clear understanding by the METR paper of the dependencies between the components of a long task. There are no new contributions in these areas beyond the simple (or even simplistic) models from METR and Epoch.

Benchmark-level: Another limitation of this work, shared with a big proportion of papers using IRT these days, is that the analysis is at the level of benchmarks, with saturation issues among others, rather than putting all items together and performing the analysis at the level of items. I understand that collecting item-level results is harder, but it could lead to different results.

Questions for authors
4 comments

We are glad to hear that you found that our paper is very well-written, uses state-of-the-art data, gives useful information for understanding trends, and provides robust results. We appreciate you valuing the presentation, quality and soundness of our paper highly.

  1. "While doing experimental work with Epoch's and other contributor's data is useful for the community, this is a portfolio of data postprocessing with little technical contribution to ICLR in terms of theory, ideas, etc"

Our paper’s main thesis is an entirely new idea that stands in direct contrast to the standard conclusion drawn from Epoch and METR data (i.e. the conclusion that AI capabilities are accelerating exponentially with no end in sight). Our analysis shows that the cost to achieve a given capability level grows super-quadratically, and this relationship is stable over time. This is a new idea: that we should measure AI progress by directly comparing the relative growth rate of time-horizon as a function of the actual economic cost to run the model.

  1. "I think the statement of diminishing returns is assuming we won't split the tasks (this is hinted at the end, suggesting human intervention, but there could be orchestrators with high executive functions and planning capabilities doing this splitting, like a software project manager)"

Splitting tasks into subproblems orchestrated by an AI model is already allowed in some of the benchmarks included in our analysis e.g. FontierSWE and CursorBench both utilize the Claude Code and OpenAI Codex harnesses when testing Anthropic and OpenAI models. Thus, splitting into subproblems orchestrated by AI is already considered in our analysis.

Furthermore, the main question we set out to study is if and when AI systems will be able to autonomously do large portions of economically productive work. Looking only at ECI data or METR time horizons, a person might assume that this will happen quite soon. However, if costs increase quadratically in time horizon, then there will rapidly be a limit where it is not worth it to replace many aspects of human labor with AI.

  1. "How does this paper relate to McFayden et al.? (https://arxiv.org/abs/2606.17930) (see also https://www.aisi.gov.uk/blog/more-compute-more-capability-why-ai-agent-evals-need-to-account-for-test-time-compute). This paper appears earlier and seems more complete to me in coverage and extension of inference to high costs."

We think you have missed the point of our paper, which is to examine the precise functional relationship between economic cost and time horizon, along with its change over time. The paper of McFayden et al does neither of these things, it just focuses on giving larger token budgets to a small subset of models, and seeing if it improves performance. In contrast, the economic cost that we measure takes into account model size (more FLOPs), inference budget, the falling cost of tokens as data centers scale up and new hardware becomes available etc. Despite all these moving factors, we find a very stable power-law exponent over time, which indicates a deeper connection between inputs as measured in dollars, and outputs as measured by time horizon. In particular, it shows that on the current scaling trend, as time horizon grows inference costs could rapidly exceed the economic returns to AI.

McFayden et al. [1] do not ever ask the same questions as us (how does economic cost grow as a function of time horizon), and they study a much smaller subset of benchmarks and models for a completely different question. It is factually incorrect to say that this paper [1] is "more complete in coverage."

[1] Jessica McFadyen, Ole Jorgensen, Harry Coppock, Kevin Wei, Cozmin Ududec. How Inference Compute Shapes Frontier LLM Evaluation, ArXiv July 2026.

  1. "Do the authors have any data that wage per hour depends on how long the tasks are?"

We believe you have misunderstood our argument. The point we are making is that human software engineers are paid an hourly wage, say it is X dollars per hour. If you ask a human SWE to complete a large software engineering task that takes 5 months = 108 working days = 864 hours, then you must pay them 864×X864\times X dollars. If instead one wants to completely automate this 5 months of human labor with AI, our superquadratic power law scaling predicts a cost that grows faster than 8642=746496864^2 = 746496. This is a different scaling relationship than the human cost, which scales linearly in the number of hours. Thus, while it might be economically efficient to automate shorter tasks with AI, one cannot fully automate the human SWE without paying a prohibitive cost. This is the point we are trying to make: despite exponential gains in capabilities, the exponential growth in inference cost is even larger. Thus on the current scaling trend, it will not be economically feasible to fully automate large chunks of human labor.

Thank you for the precise responses. They (and reading the other reviews and responses) help clarify some misunderstandings. One final question: how would things change if you did the analysis at 95% instead of 50%? My intuition is that with a higher threshold here, the increase of the horizon can build much more on previous steps than in the hugely hazardous case of 50%, and the laws may be very different.

Thank you for your response. We re-ran the full analysis pipeline upon your suggestion this time using cost to achieve 95% accuracy, rather than 50% accuracy. The result was a scaling exponent of γ=2.68\gamma = 2.68 with R2=0.75R^2 =0.75. This is quite similar to the results for 50% accuracy costs, demonstrating the robustness of these results.

Wow, that was a quick reply, really appreciated. I must admit that this is a bit counterintuitive to me so there are some things I'm not understanding well. I'll have another read through some parts of your paper and see if that makes change my assessment. Thank you.

Reviewer_tTkwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors study how the inference cost required for 50% task success scales with task time horizon. They combine benchmark performance and cost data using a hierarchical item response theory model, then use METR evaluations to convert latent benchmark difficulty into human-equivalent time. They find that cost scales superquadratically with task horizon, suggesting diminishing returns to additional inference spending on longer tasks.

Critical strengths
  1. The paper studies an important and practical question, as understanding how inference cost changes with task time horizon could be useful for forecasting the economic impact and practical limits of AI agents.
  2. The paper uses a broad collection of model evaluations and includes several robustness checks. In particular, it reports bootstrap confidence intervals, stability over time, and leave-one-benchmark-out analysis rather than relying only on the main regression.
Critical weaknesses
  1. It is unclear whether the method estimates the minimum-cost frontier. Equations 2 and 4 fit an average cost-capability relationship rather than the cheapest achievable cost. Figure 3 also uses the same cost data used to fit the model, so it does not appear to be a held-out validation.
  2. The human-hours conversion relies on a strong assumption. Figure 4 fits capability against METR software-task horizons, but the paper then applies this relationship to benchmarks in physics, cybersecurity, formal proof, GUI use, and other domains without validating their inferred human completion times.
Questions for authors

Clarification Requests:

  1. Why are Equations 2 and 4 interpreted as estimating the minimum-cost frontier? The method uses ordinary least squares to fit an average cost-capability relationship, and Figure 3 uses benchmark costs that were also used to fit the model. Is any part of this validation held out?

  2. For Figure 4 and Equation 12, why should the relationship between capability and METR software-task horizons also apply to benchmarks in physics, cybersecurity, formal proof, and GUI use? Do any of these benchmarks have measured human completion times that validate this conversion?

Suggestions for Future Work:

  1. Can the authors report the scaling exponent using only the 10 benchmarks where at least one model reaches 50% success? This would show how much γ=2.81\gamma=2.81 depends on the four benchmarks without an observed model reaching 50%.

  2. A held-out cost evaluation or explicit lower-frontier fit would help show whether the method predicts minimum cost rather than average cost conditional on capability.

2 comments

We are glad to see that you found that our paper studies an important and practical question on the economic impact and practical limits of AI agents, uses a broad collection of model evaluations and includes several robustness checks reporting bootstrap confidence intervals, stability over time, and leave-one-benchmark-out analysis.

  1. "Why are Equations 2 and 4 interpreted as estimating the minimum-cost frontier? The method uses ordinary least squares to fit an average cost-capability relationship, and Figure 3 uses benchmark costs that were also used to fit the model. Is any part of this validation held out?"

The method estimates the average cost to achieve at least 50% success. We can clarify this in the paper.

To answer your question regarding Figure 3, we performed 5-fold cross validation to measure the accuracy of these 50% success-rate costs. We partitioned the models into 5 groups of equal size. For each held-out group of 20% of the models, we fit the 50% solver cost predictions Cj∗C^{*}_j using the other 80% of the models, and then validated this prediction on the actual minimum-cost empirical >50% solver from the 20% held-out set Cj,held−outC_{j,held-out}. To do this validation we fit an OLS regression log⁡10(Cheld−out)=α+βlog⁡10(C∗)\log_{10}(C_{held-out}) = \alpha + \beta \log_{10}(C^{*}). This results in a slope =1.12=1.12 and intercept =−0.178=-0.178, with R2=0.816R^2=0.816. Given the slope close to 1 and intercept value close to 0, our 50% success-rate cost-extrapolation methodology clearly gives quite good predictions on empirical minimum cost to exceed 50% accuracy for held-out models.

  1. "For Figure 4 and Equation 12, why should the relationship between capability and METR software-task horizons also apply to benchmarks in physics, cybersecurity, formal proof, and GUI use? Do any of these benchmarks have measured human completion times that validate this conversion?"

Cybersecurity and formal proof tasks are still essentially software engineering in the sense that the final output is some code that has to be run and pass held-out unit tests, and the agent can do some thinking and run some tests to figure out how to produce the final code. The argument for the original METR software-task horizons also directly applies here. In particular, the assumption is that long-horizon tasks decompose into subtasks, all of which have to be done correctly to succeed at the overall task. Thus failures compound multiplicatively, and a logistic function is therefore appropriate to relate log success rate to log time horizon.

We would argue that all tasks are like this, including physics and GUI use: they decompose into simpler subtasks and you have to get them all correct in order to solve the overall task. We do agree that ground-truth human completion time on physics might be on a different scale than for software tasks, and so this is something that would be important to take up in follow-up work. We emphasize however that in our leave-one-out analysis in Figure 6 of the appendix, removing the physics benchmark or the GUI benchmark has a negligible effect on the estimated exponent. This can be explained by recalling that the IRT methodology assumes that some latent "general intelligence" factor is responsible for a large majority of the variation in performance across all tasks, and so including these benchmarks and associated costs still allows for more accurate estimation of θ\theta and associated benchmark difficulties.

  1. "Can the authors report the scaling exponent using only the 10 benchmarks where at least one model reaches 50% success? This would show how much \gamma = 2.81 depends on the four benchmarks without an observed model reaching 50%."

We refit our model on the 10 benchmarks with observed minimum costs to achieve greater than 50% solve rates. The result was entirely consistent with our full model: the time scaling exponent was γ=2.60\gamma=2.60 with bootstrap confidence interval [1.30,4.07][1.30,4.07]. The wider bootstrap confidence interval is almost entirely due to regression uncertainty in step 5 of our hierarchical bootstrap analysis via student's t posterior (equation 17), because we have 10 benchmarks with reported minimum cost solvers out of 14 total cost-reporting benchmarks.

Thank you for the response and additional cross-validation. The results help address the held-out validation concern, but the method estimates average cost rather than the minimum-cost frontier. Additionally, the cross-domain human-hours conversion remains unvalidated. I therefore maintain my rating.

Reviewer_LsLJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how inference costs increase with agentic task difficulty. It combines benchmark-level item response modeling, a shared-slope log-cost regression, and a calibration from latent model capability to METR time horizons. The resulting regression across 14 cost-reporting benchmarks has exponent 2.81 and reported 95% interval [1.96, 3.58]. The authors interpret it as a minimum-cost frontier with implications for long-horizon autonomy and human labor.

The question is important and the descriptive relationship is potentially useful. My main concern is that the analysis does not yet establish the stated estimand: a currently attainable minimum cost for completing tasks of a given human duration. Direct frontier estimation and narrower interpretation would substantially strengthen the paper without requiring a large new experiment campaign.

Critical strengths
  • Connecting capability evaluations to actual inference costs addresses a practical deployment question.
  • The data integration and explicit equations make the measurement pipeline understandable.
  • The robustness evidence is real. Model-cluster resampling preserves within-model dependence across benchmarks, and leave-one-benchmark-out exponents remain between 2.64 and 2.99 (Table 2), so no single benchmark drives the trend.
  • Figures 3 and 4 show strong descriptive associations that motivate further investigation.
Critical weaknesses

Main concerns

1. The fitted cost function is not a minimum-cost frontier, and its absolute calibration is problematic. Equations (2), (4), and (5), pp. 4–5, regress log cost on capability with benchmark intercepts and evaluate that regression at θ=b_j. Least squares estimates a mean relationship in log space, not the least expensive feasible system. Filtering to systems with ≥50% success does not make the fit a lower envelope.

Figure 3 reports log10 C_min = −0.73 + 1.11 log10 C*. On this line, predicted costs of $1 and $100 correspond to observed minima of about $0.19 and $31. The high R² therefore does not establish identity calibration, and the discussion of a near-unit slope overlooks the intercept. A constant multiplicative bias would shift the intercept rather than the exponent. The problem lies with the "minimum cost" interpretation and the absolute economic comparisons.

Please refit the scaling relation using actual minimum qualifying costs for the ten benchmarks with observed ≥50% solvers, and report absolute calibration against an identity line on held-out systems or time periods. Figure 3 draws its minimum qualifying costs from the same data used to fit the cost model, so its R² is not independent validation. Please also specify the pricing dates and availability protocol. Pooled historical cost observations do not identify today's attainable frontier, because prices and systems change.

2. The horizon and success-probability interpretations need validation at a consistent task granularity. Equations (6)–(7) and (11)–(12) calibrate model capability against METR horizons, then map benchmark difficulty onto that coordinate. This yields a METR-equivalent horizon for a hypothetical threshold solver. It does not establish the human completion time of the physics, visual-reasoning, GUI, proof, and software tasks in the cost analysis. A common capability scale is not a universal labor-time scale.

There is also a definitional gap between the binary outcome Y_ij and the normalized benchmark scores (p. 4, lines 175–194). A score of 0.5 may represent partial credit or subtest completion rather than a 50% probability of completing an entire job. Please specify each benchmark's success event, score transformation, cost denominator, and human-time unit. Until there is direct human-time validation, "METR-equivalent horizon" is the appropriate term, and the wage comparisons should be framed as conditional illustrations or removed.

3. The paper relies on extrapolation without establishing its accuracy. For The Agent Company, ExploitBench, OSWorld 2.0, and CritPt, no observed system reaches 50% (p. 7, lines 371–374). Under the stated rule, only the best-performing system supplies cost data. With one observation and a free benchmark intercept,

log10 C*_j = log10 C_rj + η(b_j−θ_r).

The threshold cost for these benchmarks is therefore a transfer of η from other benchmarks, not an available solver price. Please distinguish these predictions from observed feasible frontiers, and add a sensitivity analysis that excludes all four benchmarks jointly. The single-deletion analysis answers a different question.

The manuscript uses 270 models for capability fitting, 146 for the cost dataset, and 37 for METR calibration, but the final cross-benchmark regression has only 14 units. The constructed benchmark horizons span roughly 29 minutes to 57.9 hours, and the 80-hour example lies beyond that range. The longest calibration observation in Figure 4 appears near the 12-hour tick, so the upper benchmark horizons also exceed the observed METR calibration range. Please report the exact calibration limits.

Quarterly refitting and deletion sensitivity are not a test of predictive extrapolation. A held-out test that trains on earlier or weaker systems and predicts the frontier for later or stronger ones would give much more direct support for the proposed scaling law.

4. The final relationship partly reflects shared construction, and the strongest conclusions exceed the observational evidence. The plotted quantities satisfy

log10 T_j=(b_j−κ0)/κ, and log10 C*_j=c_j+ηb_j.

Hence the cross-benchmark slope obeys γ=κ[η+Cov(b,c)/Var(b)]. The covariance term is data-dependent, but the common cost slope, the benchmark cost structure, and shared errors in the difficulty estimates all enter the final result. Please report η̂, which the paper does not provide, together with this decomposition. Please also test whether the result persists with directly observed frontier costs and heterogeneous cost slopes.

A cross-task association does not identify the effect of increasing the inference budget on a fixed task. The claims about a fundamental property of the reasoning paradigm, and about the need for a new paradigm to change labor-substitution outcomes, should be narrowed. A stable exponent combined with falling intercepts would still change the economic comparison.

Statistical clarifications and minor corrections

The interval [1.96, 3.58] includes 2, so the usual two-sided 95% criterion does not exclude quadratic scaling. That 96.8% of bootstrap draws exceed 2 indicates a direction, but it is neither a posterior probability nor a calibrated one-sided test. Section 4.5's claim that subquadratic scaling can be confidently rejected needs a specified inference procedure or weaker wording.

Lines 461–463 state a temporal range of 2.81–3.08, but Table 1 gives 2.33–3.08 for the same period. Please correct this. Table 1's ± values appear to be κ times the OLS slope standard errors rather than full hierarchical 95% intervals; please label them accordingly.

Appendix B should explain how the refitted IRT coordinates remain jointly consistent with the full-data METR parameter draws, and what uncertainty the additional t draws represent. Coverage validation or a clear generative justification would help. Please also explain how the latent scale is fixed in resamples that lack a normalization anchor. For each historical cutoff, please report whether it uses contemporaneous evaluations, pricing, and calibration. Consistent affine normalization of the IRT scale is fine; the question is whether any calibration, evaluation, or price uses information that was unavailable at the forecast cutoff.

Eq. (1) implies logit p_ij=a_j(θ_i−b_j), not the benchmark-independent θ_i∝log p_i interpretation used to motivate Eq. (3). The cost relation can stand as an empirical assumption, but this argument should be rewritten. Appendix A's GBAEval cost, taken at the first checkpoint that reaches the final score, uses retrospective information; please distinguish it from the cost of an executable online stopping policy. Please also release the frozen dataset and code, and document the cost normalization, model/harness matching, and post-filter sample counts.

Questions for authors
  1. What exponent, uncertainty, and held-out absolute error result from fitting the observed minimum qualifying costs?
  2. Does the relationship persist when all four unachieved-threshold benchmarks are removed together, and when only well-defined binary-success metrics are retained?
  3. What evidence aligns the cost-task unit, the success event, and actual human completion time outside METR?
  4. Can data from earlier or weaker models predict the cost frontier of later or stronger models without refitting on the test set?
  5. Can the authors clarify joint uncertainty propagation, correct the temporal range, and separate observed, interpolated, and extrapolated estimates?
1 comment

We are glad to hear that you found the question we ask important, the descriptive relationship we reveal useful, and our paper clear and easy to follow with real robustness evidence and strong descriptive associations that motivate further investigation. We appreciate you valuing the presentation, informativeness and soundness of our paper highly.

  1. ”What exponent, uncertainty, and held-out absolute error result from fitting the observed minimum qualifying costs?”

We refit our model on the 10 benchmarks with observed minimum costs to achieve greater than 50% solve rates. The result was entirely consistent with our full model: the time scaling exponent was γ=2.60\gamma = 2.60 with bootstrap confidence interval [1.30,4.07][1.30,4.07]. The wider bootstrap confidence interval is almost entirely due to regression uncertainty in step 5 of our hierarchical bootstrap analysis via student's t posterior (equation 17), because we have 10 benchmarks with reported minimum cost solvers instead of 14.

  1. ”Does the relationship persist when all four unachieved-threshold benchmarks are removed together, and when only well-defined binary-success metrics are retained?”

Yes the relationship persists when we remove these four benchmarks. Please see our answer for question 1 above for the exact numerical results. All of our results use well-defined binary success metrics based on subtask completion, following the identical methodology used by Epoch AI to fit the ECI latent capability model.

  1. ”What evidence aligns the cost-task unit, the success event, and actual human completion time outside METR?”

To answer this question, we performed 5-fold cross validation on our analysis methodology. We partitioned the models into 5 groups of equal size. For each held-out group of 20% of the models, we fit the 50% solver cost predictions Cj∗C^{*}_j using the other 80% of the models, and then validated this prediction on the actual minimum cost empirical 50% solver from the 20% held-out set Cj,held−outC_{j,held-out}. To do this validation we fit an OLS regression log⁡10(Cheld−out)=α+βlog⁡10(C∗)\log_{10}(C_{held-out}) = \alpha + \beta \log_{10}(C^{*}). This results in a slope β=1.12\beta = 1.12 and intercept α=−0.178\alpha = -0.178, with R2=0.816R^2 = 0.816. Given the slope close to 1 and intercept value close to 0, our minimum task cost-extrapolation methodology clearly gives quite good predictions on minimum 50% solver cost for held-out models.

  1. ”Can data from earlier or weaker models predict the cost frontier of later or stronger models without refitting on the test set?”

Yes it can. To answer this question, we performed temporal out-of-sample validation as follows. We selected a training set of all evaluations before March 31, 2026 (248 evaluations of 99 models) and a test set of 186 evaluations of 45 models that were released after March 31, 2026. We fit our cost extrapolation model on the training data to predict the 50% solver cost Cj∗C^{*}_j, and then measured the actual minimum 50% solver cost Cj,min−testC_{j,min-test} on the held-out test data. We then fit OLS regression to obtain log⁡10(Cmin−test)=−0.762+0.918log⁡10(C∗)\log_{10}(C_{min-test}) = -0.762 + 0.918 \log_{10}(C^{*}) with R2=0.726R^2 = 0.726. The slope is close to one with a similar intercept as in Figure 3, indicating good predictive accuracy when fitting on the earlier and weaker models, and then predicting on the later stronger models.

  1. ”Can the authors clarify joint uncertainty propagation, correct the temporal range, and separate observed, interpolated, and extrapolated estimates?”

The joint uncertainty propagation is detailed in our description of the non-parametric bootstrap estimates we used in Appendix B. Our answers to questions 1, 3, and 4 detail validation of the observed vs extrapolated estimates in the paper.

Reviewer_xQctOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Paper Summary: The paper offers an analysis that aims to “measure diminishing returns in AI Agents” — particularly the relationship between task horizon (the length of task completed by the AI agent at 50% success probability, as measured in time it would take a human to complete it) the task completion cost (Figure 1).

In order to achieve this task horizon and task cost need to be tied over different times and a diversity of models and tasks (and task domains)

Critical strengths

Critical Strengths:

The paper is very well written with commendable effort from the authors to make it easy to follow and understand. There are very minor typos and editorial notes — but in an age rife with AI slop and over polishing, reading this paper is a breath of fresh air from a clarity and thoughtfulness perspective — it allows a human reviewer to fully engage with the methodologies and motivations without undue friction.

The core question that the paper tackles is also of significant importance — how does inference cost scale versus task length (in human equivalent time, which bears economic meaning), at a given performance level (50% in this case).

The methodology is straightforward and well specified.

Critical weaknesses

Critical Weaknesses:

The core weakness of the paper is that it does not rigorously establish well its rather large claim — namely that there is a strong power law relationship between task length and the inference compute needed in order to solve it at 50% success probability.

  1. The projection to task horizon (on the SW / METR measured domain) from latent difficulty (in other domains) implies at best that the correlation holds in the narrow domain of SW. One has no evidence that in other domains capability does translate to task length and there are strong reasons to suspect it does not. → If using this cross domain methodology, then one needs to show that indeed the task length is tied to latent difficulty across domains. Does this hold?

To be clear this is material not from the perspective of generalization (which is also important and missing) but in order to support the main claim which ties cost and task length (but uses different domains to do so).

  1. Even if one were to narrow the claim such that it holds only for the SW domain — i.e. such that one concludes that only in the SW domain where the correlation between latent capability and task horizon is established and interpretable as true time — the use of averages hides a key falsification test of the claim:

In the SW domain, if one measures the true per-task inference compute versus the task duration — does the relationship hold? This is not demonstrated.

  1. The rapid decrease in inference costs for a given performance level is recognized in the paper preface, but is not accounted for. Explicitly does inference cost at the frontier increase with task horizon? I.e. if one tasks the first models to reach increasing horizons, does one find that this cost increases? Or is a skew in cost per horizon at-a-given-moment dominated by the algorithmic progress compressing costs off frontier? Decoupling and quantifying the magnitude of both these phenomena materially changes the economic interpretation and implications.

Analysis separating and quantifying these phenomena (first-to-achieve frontier cost over time vs lowest cost at a given time, or more generally the frontier cost increase (?) rate vs iso-performance cost decrease rate) is thus required.

  1. The analysis and conclusion does not account for The paper reproducibility is not made easy — with no shared repo. As such significant effort was required in the review in order to check the soundness of the results. A reproduction effort exposed what seems to be a potential serious bug: The ~2.8 exponent is approximately reproduced but only if using cost data total-run costs instead of what should be average per task costs. By correcting this suspected bug, the high “superquadratic” exponent does not survive

Non-rigor related: The authors do well to include related work including good work in calling out “original” work linages — but fall to actually cite the actual original work: specifically in the scaling law literature [Hastiness et al 2017 first observes power law trends in deep learning and Rosenfeld et al were the first to show scaling laws as we now know them]

Questions for authors

Questions For Authors: The questions mirror the above weaknesses:

  1. Can you show that task length and model capability are related in different domains (as necessary in order to move from cross domain latent difficulties to in-domain duration horizon projection)
  2. Can you show that the main claim survives within a domain at the task level? I.e. that cost increases super-quadratically with task horizon within a domain?
  3. Can you show what are the dominant effects quantitatively that govern performance cost per task — separating the algorithmic progress associated induced skew from a true cost increase at the frontier?
  4. It is highly advisable to provide a reproducible repo — and in particular here: is the suspected bug true? Does the analysis use the total (bug) or per task average (intended) costs?

Finally — I would commend the authors for very clear and interesting work that is important and addresses impactful questions and encourage to increase its rigor. If the above are well addressed such a submission may move to be a strong contribution depending on the findings and what survives and I will be happy to update and potentially increase my assessment of the work accordingly.

1 comment

We are truly glad to hear that you found reading our paper a breath of fresh air from a clarity and thoughtfulness perspective and that you highlighted that our paper is very well written with commendable effort, and that the core question that our paper tackles is of significant importance with a straightforward and well specified methodology.

  1. ”Can you show that task length and model capability are related in different domains (as necessary in order to move from cross domain latent difficulties to in-domain duration horizon projection)”

Of our 14 cost-reporting benchmarks, 10 of them involve writing code or software engineering, while 4 (ARC-AGI-1, ARC-AGI-2, OSWorld 2.0, and CritPt) are from completely different domains. We acknowledge that the time-horizon to latent difficulty mapping is less well-justified across domains, so we re-ran our full analysis pipeline on the 10 coding benchmarks. This still yields a superquadratic scaling exponent of γ=2.36\gamma = 2.36 with R2=0.79R^2 = 0.79.

As an aside: a recent report on CritPt revealed very severe methodological flaws that completely change the performance picture for this benchmark (e.g. some models go from 30% to 80% accuracy). Thus, we think it should likely be excluded from any analysis in any domain.

  1. ”Can you show that the main claim survives within a domain at the task level? I.e. that cost increases super-quadratically with task horizon within a domain?”

Yes, as stated above for question 1, for the 10 benchmarks in the coding domain our analysis yields a superquadratic scaling exponent of γ=2.36\gamma = 2.36.

We further found the average human time estimates in the original papers that introduced each of these 10 benchmarks. We then re-ran our pipeline using these estimates in place of our IRT-predicted human time horizon. Using the actual empirical human time estimates also yields a superquadratic scaling exponent of γ=2.40\gamma = 2.40.

Finally, we measured the Nash-Sutcliffe Efficiency to determine how well our IRT-predicted time horizons match the measured empirical human time horizon on each benchmark. This measures how well the predicted times vs empirical times plot matches the exact y=x identity line. The result was R2=0.91R^2 = 0.91, meaning that the IRT predictions are quite close to exact for the empirical measured times.

  1. ”Can you show what are the dominant effects quantitatively that govern performance cost per task — separating the algorithmic progress associated induced skew from a true cost increase at the frontier?”

To answer this question, we re-ran our analysis using only the cost of the first model to ever achieve greater than 50% accuracy on each benchmark. Running this on the 8 out of 10 coding benchmarks where >50% accuracy has ever been achieved by any model, we get γ=2.32\gamma = 2.32. This is quite close to the γ=2.36\gamma = 2.36 value estimated using our IRT cost model on the original 10 coding benchmarks, suggesting that our method has little overestimation of the scaling exponent due to cost deflation over time. This also makes sense mathematically, because our cost-estimation procedure uses all models with >50% accuracy to estimate the cost.

To understand this issue further, we also re-ran the analysis pipeline on these same 8 benchmarks using only the minimum cost model with >50% accuracy. This results in an exponent of γ=2.81\gamma = 2.81. Thus, using minimum costs directly seems to inflate the exponent by approximately 0.4, likely due to cost deflation off the frontier. In contrast our IRT cost estimation procedure does not appear to inflate the estimates much, if at all.

  1. ”It is highly advisable to provide a reproducible repo — and in particular here: is the suspected bug true? Does the analysis use the total (bug) or per task average (intended) costs?”

Thank you for pointing this out. There was a bug in per-task normalization that affected 4 benchmarks (Surface Evolver Bench, ExploitBench, OSWorld 2.0, CritPt), two of which are coding-domain benchmarks. Our analysis for the 10 coding-domain benchmarks in questions 1 and 2 correctly normalized the costs per-task, yielding the superquadratic scaling exponent of γ=2.36\gamma = 2.36.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

We truly thank you for investing your time to provide a review for our paper. We are truly glad that

Reviewer xQct found reading our paper a breath of fresh air from a clarity and thoughtfulness perspective, and highlighted that our paper is very well written with commendable effort, and emphasised the core question that our paper tackles is of significant importance with a straightforward and well specified methodology.

Reviewer LsLJ found the question we ask important, the descriptive relationship we reveal useful, and found our paper clear and easy to follow with real robustness evidence and strong descriptive associations that motivate further investigation. We also appreciate Reviewer LsLJ valuing the presentation, informativeness and soundness of our paper highly.

Reviewer tTkw found that our paper studies an important and practical question on the economic impact and practical limits of AI agents, uses a broad collection of model evaluations, and includes several robustness checks reporting bootstrap confidence intervals, stability over time, and leave-one-benchmark-out analysis.

Reviewer Hn5X found that our paper is very well-written, uses state-of-the-art data, gives useful information for understanding trends, and provides robust results. We also appreciate Reviewer Hn5X valuing the presentation, quality and soundness of our paper highly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.