acceptodds
Under review as a conference paper at ICLR 2027

The Declining Hazard Reported for AI Agents Is a Pooling Artefact

Abstract

Published fits to METR's time-horizon data put the Weibull shape k below one, so the failure rate falls as tasks get longer, and read it as agents getting better as a task goes on. That decline is an artefact of pooling unequally difficult tasks: a mixed batch of light bulbs shows a falling burnout rate while each wears out. Modelling task difficulty reverses the sign: the shape moves from 0.63 to 1.85, and 17 of 20 models exclude k ≤ 1, a count no null simulation reached. Across admissible ways of pooling the shape spans 1.56 to 1.85; the data reject the only one consistent with constant hazard. The sign holds on every clock; the magnitude does not: the same runs clocked by tokens, rated task length or elapsed time have median shapes 1.700, 2.045 and 2.625; the ordering repeats on a benchmark sharing no tasks, agents or harness. Humans on the same tasks and clock are indistinguishable from constant hazard; agents exceed them by Δk = 1.225 [0.825, 1.725]. A blind reimplementation from the specification reproduced both headline values. METR's pooled horizons stand. At fixed task difficulty an agent's 80%-reliability length is about half its 50% length; the published shape implies a sixth. Report the sign; report the magnitude as a range across clocks, or not at all.

Primary area:
datasets and benchmarks
Keywords:
agentic benchmarks, agent benchmarks, evaluation methodology, LLM agents, time horizon, survival analysis, frailty models, unobserved heterogeneity, task heterogeneity, measurement validity, hazard rate, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FcUZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that AI agents' failure rate rises as tasks get longer, the opposite of earlier analyses. METR fits a logistic curve of success against log task length. Later re-analyses of METR's public data (Hamilton, 2026; Ord, 2026) fit a Weibull survival curve instead, found a shape parameter kk below one, and concluded that agents become less likely to fail as a run goes on. This paper argues that k<1k < 1 comes from fitting one curve to tasks of different difficulty. Easy-to-fail tasks drop out early, so the pooled failure rate can fall even when each task's rate rises.

To correct for this, the authors give each task its own difficulty. They estimate how difficulty is spread across tasks (the frailty distribution GG, mainly with an NPMLE) from the fact that benchmarks run each task several times per model. With this correction, kk on METR's second release goes from 0.63 to 1.85, or 1.56 to 1.85 depending on how models are pooled.

The authors then show that kk depends on how run length is measured. On the same runs, counting tokens, using the task's rated human time, or using wall-clock time gives different values. They test three explanations for this (study design bias, tokens growing as a power of time, noise in token counts). Each covers at most a quarter of the gap. Finally, they compare agents with humans on the same tasks. Humans fail at a roughly constant rate over wall-clock time. Agents' failure rate rises faster.

Critical strengths
  • S1: The published k<1k < 1 does not show that agents' failure rate falls.

    • When the authors simulate METR's tasks with a constant failure rate, the uncorrected fit still returns kk = 0.57 or 0.43 (§5, Fig. 2, App. C.1). So pooling alone produces k<1k < 1. This holds without the paper's own correction.
    • The "agents get better as a task goes on" reading has been repeated in AI forecasting discussions (Ord, 2026).
  • S2: How run length is measured changes kk about as much as the correction does.

    • On the same METR runs, kk is 1.70 on tokens and 2.63 on wall-clock time. Wall-clock time gives the higher value for 17 of 18 models on METR. On Terminal-Bench, which shares no tasks or agents with METR, it does so for 50 of 51 agent–model pairs (§6.2).
    • Benchmarks log different units, so their kk values cannot be compared. The gap remains when the Weibull is replaced by a Cox model.
Critical weaknesses
  • W1: The corrected sign depends on the most flexible estimator, which might be biased.

    • With the standard gamma frailty model, kk stays below one (0.90, Fig. 1A). Only the log-normal model and the NPMLE push it above one. In the authors' simulations the NPMLE overestimates kk. At a true kk of 1 it is 0.115 too high on average. With difficulty taken from a real dataset, it returns 1.15 at a true kk of 1 and 1.90 at 1.65. The grid used for GG alone moves the estimate from 1.64 to 1.87.
    • The paper attempts to address this bias with the "17 of 20 models" test. In simulated data with a constant failure rate, 17 models never had intervals above one. The paper does not say how spread out task difficulty was in that simulation. If METR has a lot of near-impossible tasks, 17 is more likely by chance than the simulation suggests.
    • The estimate also depends on how many times each task was run. On SWE-bench, going from 2 to 10 runs per task lowered kk by a median of 0.85, in all 14 settings (App. C.4). On METR, 80–83% of model–task pairs are all-pass or all-fail. The paper does not check how 1.85 changes with the number of runs on METR.
    • Rerunning the simulation with the difficulty spread measured from real data would test this. Comparing how well the gamma, log-normal and NPMLE models fit the METR data would show whether gamma is the wrong model there. Refitting METR with repeats dropped at random would show how much kk depends on the number of runs.
  • W2: The agent-versus-human contrast uses the metric that the paper is questioning.

    • The agent–human gap (Δk=1.225\Delta k = 1.225) is measured on wall-clock time. By the paper's own numbers this is the least reliable measurement/clock. A model's tokens per minute vary by a median factor of 19.7 between its slower and faster runs. Wall-clock time includes queueing and retries. The extra kk on this clock is unexplained. Agents likely wait on servers, unlike humans. The authors' fallback puts agents on tokens and humans on wall-clock time, which compares two different measures.
    • A human failure likely means the person gave up, which differs from an agent failing. On rated task length the humans' kk is 0.60 (0.67 corrected), below one. The abstract mentions neither point.
    • Refitting agents on time that excludes waiting (for example, wall-clock time minus the gaps between steps) would address the first point.
  • W3: The paper is very hard to follow.

    • The paper is full of jargons even from the abstract.
    • Most sentences are very terse; very hard to parse.
    • Most result sentences are packed with numbers (and jargons).
    • Key terms are used before they are defined.
    • There is a chance that I misunderstood the paper (the weaknesses listed above) because it is presented poorly.
Questions for authors

The abstract says that, at equal difficulty, the task length an agent completes 80% of the time is about half the length it completes 50% of the time. This needs Assumption 1 (the spread of difficulty does not change with task length). Is there evidence for it? If not, the abstract might need to be revised.

Reviewer_QHkJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that previously reported declining failure hazards in AI agents arise from pooling tasks with heterogeneous difficulty. Using a Weibull frailty model fitted to METR's repeated-run data, the authors find that the median shape parameter changes when heterogeneity is explicitly modeled.

They also find that the estimated hazard depends substantially on whether task length is measured by human-rated duration, elapsed time, or tokens generated. A comparison with human trajectories suggests substantially lower hazard growth for humans.

Critical strengths
  • It’s an interesting problem to dive deeper into the popular METR Time Horizon graph
  • Statistical tests and rigour, for the pertinent scope, are thorough
  • Datapoints considered overall is significant and robustly analysed
Critical weaknesses
  • The writing and language throughout the paper is unfortunately incorrigible. It was hard to understand what the authors were claiming, their motivations and the design of experiments.
  • For a simple example, the abstract immediately assumes the reader understands Weibull distributions, shape parameters, hazard functions, and METR's time-horizon methodology.
  • The writing is extremely dense to parse or understand anything. There is significant emphasis on statistical machinery and tests, but much less on the problem and motivation for studying this, and the conclusion as well.
  • From a broader methodological lens: the METR time graph has become highly influential over the past year, however, I’m not sure whether it is relevant in today’s time. Even a few months ago they stopped publishing updates simply for the reason that at 16 hours or beyond their statistical reliability was very less due to less data samples, and the infeasibility of that kind of human data.
  • Further it is evident that the strength of models today is enough to complete work requiring much longer time horizons than what might take a single human 1-2 days.
  • So in effect: I’m not super confident that the claims made by the paper hold to frontier models today and whether the bottleneck is METR’s statistical reliability or simply lack of data.
Questions for authors
  • I’m more curious as to what was the author’s intention for this work, could you please articulate in simple language, what you think are the broad takeaways for a reader?
  • The authors themselves say: “A single replacement value of k is not what this paper supplies.“ However the research questions, abstract and discussion was very convoluted for me to digest, and only thing I could takeaway was a very narrow insight that this paper advocates for a shift in one of the benchmark curve’s shape, which is a narrow finding for an ICLR paper, in my opinion.
Reviewer_cUEbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper provides a method to rectify agent results on long-running tasks. The paper aruges that agentic benchmarks have moved from a single score to a curve. Specifically, the paper investiages METR data and finds that we need to account for task difficulty when reporting the results. Otherwise, we will see the failure rate falls as tasks get longer. The paper conducts several experiments to verify this.

Critical strengths
  1. The paper provides a concrete, decision-relevant result for anyone using METR horizons to reason about agent deployment reliability. It uses rigorous statistical method to verify the argument.
Critical weaknesses
  1. The paper's title provides a general claim but the analysis is only done on the METR's time-horizon data can it's unclear how this finding generalize to other scenarios and how this finding impacts agent evaluation.
  2. The methodological contribution is standard statistical methods applied to a new domain. The paper explicitly mentions that "We claim none of the machinery". So besides the specific finding on METR results, what is the contribution to the ICLR community? Is it just using a statistical method to analyze an AI benchmark result? The contribution of the paper needs to be better justified.
  3. The paper is very hard to read. There are many short sections and many parts of the paper feel AI-generated, especially for the abstract (e.g., many symbols undefined in the abstract, and a sentence like "Report the sign; report the magnitude as a range across clocks, or not at all"). I think the quality of the writing does not match the bar of ICLR.
Questions for authors
  1. I don't get the "how reliably depend on the shape" argument. I think the way METR report time-horizon data is still one specific case; for most of agent benchmarks, people still report aggregated results. I don't understand what "the shape" means in those situations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.