- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper connects the time horizon of tasks (in the software domain mostly) following METR's papers on human hours per task with the inference cost in dollars, using scaling laws fitted on benchmark results and IRT. The authors claim there are diminishing returns.
Critical strengths
- The paper is very well-written and uses state-of-the-art data and the fits seem robust.
- Gives useful information for understanding trends, in the line of published results by key third-parte evaluation organisations such as Epoch and METR.
Critical weaknesses
Misleading interpretation of task length: The time horizon is not "directly" comparable to a "human baseline via hourly wages", as they say in the introduction. You pay by hour, but not necessarily by continuous work on the same task. There is no given evidence that wages per hour are related to how long the tasks take. I imagine that a worker doing short repetitive tasks would be paid less per hour than one that does long projects, but a football player never has a task that lasts more than two hours and can earn a lot of money. Similarly for people at the boards of big multinationals. The paper doesn't depend on this assumption (longer tasks higher human cost per hour), but putting this prominently in the introduction is misleading. However, what does affect the whole study is the use of the volume of the task (not only by METR) as the key element of capability. First, because other capabilities also matter for the task, and some may be more important than others for the cost of the task, but most especially because METR's hours per task has to naturally saturate. Many meaningful tasks of hundreds of hours can be split into smaller tasks.
A bit circular: There's a very simple explanation of the curves in Kinniment et al. and METR's results: a simple survival analysis of the per-step probability of success, given the exponential distribution. As the probability of success for "chunks" increases then everything else increases, with formulas that derive easily by just assuming an independence of the chunks (the messiness in the original METR paper). But this is circular if then you use this to understand the capability. It's a good change of view, as there's a scale, in time, with units, hours, which is a big difference from aggregate benchmarks or populational-derived indicators as in Epoch's ECI, but it's what you expect to get.
Diminishing returns? I think the statement of diminishing returns is assuming we won't split the tasks (this is hinted at the end, suggesting human intervention, but there could be orchestrators with high executive functions and planning capabilities doing this splitting, like a software project manager). The long-term horizon is a good predictor, because it's simply the exponential distribution as in survival analysis, but that doesn't mean that tasks in the future should be composed of hundreds of hours.
Contributions: while doing experimental work with Epoch's and other contributor's data is useful for the community, this is a portfolio of data postprocessing with little technical contribution to ICLR in terms of theory, ideas, etc., or new experiments. Despite the influential papers from METR and Epoch (one because of the infamous title of 'Rosetta Stone'), IRT has been used in AI for at least a decade, and volume of a task is a well-known factor of complexity for planning, memory and other executive-function capabilities. But here it is just the exponental distribution of Prob(success)= p^k, without a clear understanding by the METR paper of the dependencies between the components of a long task. There are no new contributions in these areas beyond the simple (or even simplistic) models from METR and Epoch.
Benchmark-level: Another limitation of this work, shared with a big proportion of papers using IRT these days, is that the analysis is at the level of benchmarks, with saturation issues among others, rather than putting all items together and performing the analysis at the level of items. I understand that collecting item-level results is harder, but it could lead to different results.
Questions for authors
-
How does this paper relate to McFayden et al.? (https://arxiv.org/abs/2606.17930) (see also https://www.aisi.gov.uk/blog/more-compute-more-capability-why-ai-agent-evals-need-to-account-for-test-time-compute). This paper appears earlier and seems more complete to me in coverage and extension of inference to high costs.
-
Do the authors have any data that wage per hour depends on how long the tasks are?
4 comments
Thank you for the precise responses. They (and reading the other reviews and responses) help clarify some misunderstandings. One final question: how would things change if you did the analysis at 95% instead of 50%? My intuition is that with a higher threshold here, the increase of the horizon can build much more on previous steps than in the hugely hazardous case of 50%, and the laws may be very different.
Thank you for your response. We re-ran the full analysis pipeline upon your suggestion this time using cost to achieve 95% accuracy, rather than 50% accuracy. The result was a scaling exponent of with . This is quite similar to the results for 50% accuracy costs, demonstrating the robustness of these results.
Wow, that was a quick reply, really appreciated. I must admit that this is a bit counterintuitive to me so there are some things I'm not understanding well. I'll have another read through some parts of your paper and see if that makes change my assessment. Thank you.
We are glad to hear that you found that our paper is very well-written, uses state-of-the-art data, gives useful information for understanding trends, and provides robust results. We appreciate you valuing the presentation, quality and soundness of our paper highly.
Our paper’s main thesis is an entirely new idea that stands in direct contrast to the standard conclusion drawn from Epoch and METR data (i.e. the conclusion that AI capabilities are accelerating exponentially with no end in sight). Our analysis shows that the cost to achieve a given capability level grows super-quadratically, and this relationship is stable over time. This is a new idea: that we should measure AI progress by directly comparing the relative growth rate of time-horizon as a function of the actual economic cost to run the model.
Splitting tasks into subproblems orchestrated by an AI model is already allowed in some of the benchmarks included in our analysis e.g. FontierSWE and CursorBench both utilize the Claude Code and OpenAI Codex harnesses when testing Anthropic and OpenAI models. Thus, splitting into subproblems orchestrated by AI is already considered in our analysis.
Furthermore, the main question we set out to study is if and when AI systems will be able to autonomously do large portions of economically productive work. Looking only at ECI data or METR time horizons, a person might assume that this will happen quite soon. However, if costs increase quadratically in time horizon, then there will rapidly be a limit where it is not worth it to replace many aspects of human labor with AI.
We think you have missed the point of our paper, which is to examine the precise functional relationship between economic cost and time horizon, along with its change over time. The paper of McFayden et al does neither of these things, it just focuses on giving larger token budgets to a small subset of models, and seeing if it improves performance. In contrast, the economic cost that we measure takes into account model size (more FLOPs), inference budget, the falling cost of tokens as data centers scale up and new hardware becomes available etc. Despite all these moving factors, we find a very stable power-law exponent over time, which indicates a deeper connection between inputs as measured in dollars, and outputs as measured by time horizon. In particular, it shows that on the current scaling trend, as time horizon grows inference costs could rapidly exceed the economic returns to AI.
McFayden et al. [1] do not ever ask the same questions as us (how does economic cost grow as a function of time horizon), and they study a much smaller subset of benchmarks and models for a completely different question. It is factually incorrect to say that this paper [1] is "more complete in coverage."
[1] Jessica McFadyen, Ole Jorgensen, Harry Coppock, Kevin Wei, Cozmin Ududec. How Inference Compute Shapes Frontier LLM Evaluation, ArXiv July 2026.
We believe you have misunderstood our argument. The point we are making is that human software engineers are paid an hourly wage, say it is X dollars per hour. If you ask a human SWE to complete a large software engineering task that takes 5 months = 108 working days = 864 hours, then you must pay them 864×X dollars. If instead one wants to completely automate this 5 months of human labor with AI, our superquadratic power law scaling predicts a cost that grows faster than 8642=746496. This is a different scaling relationship than the human cost, which scales linearly in the number of hours. Thus, while it might be economically efficient to automate shorter tasks with AI, one cannot fully automate the human SWE without paying a prohibitive cost. This is the point we are trying to make: despite exponential gains in capabilities, the exponential growth in inference cost is even larger. Thus on the current scaling trend, it will not be economically feasible to fully automate large chunks of human labor.