- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper argues that AI agents' failure rate rises as tasks get longer, the opposite of earlier analyses. METR fits a logistic curve of success against log task length. Later re-analyses of METR's public data (Hamilton, 2026; Ord, 2026) fit a Weibull survival curve instead, found a shape parameter below one, and concluded that agents become less likely to fail as a run goes on. This paper argues that comes from fitting one curve to tasks of different difficulty. Easy-to-fail tasks drop out early, so the pooled failure rate can fall even when each task's rate rises.
To correct for this, the authors give each task its own difficulty. They estimate how difficulty is spread across tasks (the frailty distribution , mainly with an NPMLE) from the fact that benchmarks run each task several times per model. With this correction, on METR's second release goes from 0.63 to 1.85, or 1.56 to 1.85 depending on how models are pooled.
The authors then show that depends on how run length is measured. On the same runs, counting tokens, using the task's rated human time, or using wall-clock time gives different values. They test three explanations for this (study design bias, tokens growing as a power of time, noise in token counts). Each covers at most a quarter of the gap. Finally, they compare agents with humans on the same tasks. Humans fail at a roughly constant rate over wall-clock time. Agents' failure rate rises faster.
Critical strengths
-
S1: The published does not show that agents' failure rate falls.
- When the authors simulate METR's tasks with a constant failure rate, the uncorrected fit still returns = 0.57 or 0.43 (§5, Fig. 2, App. C.1). So pooling alone produces . This holds without the paper's own correction.
- The "agents get better as a task goes on" reading has been repeated in AI forecasting discussions (Ord, 2026).
-
S2: How run length is measured changes about as much as the correction does.
- On the same METR runs, is 1.70 on tokens and 2.63 on wall-clock time. Wall-clock time gives the higher value for 17 of 18 models on METR. On Terminal-Bench, which shares no tasks or agents with METR, it does so for 50 of 51 agent–model pairs (§6.2).
- Benchmarks log different units, so their values cannot be compared. The gap remains when the Weibull is replaced by a Cox model.
Critical weaknesses
-
W1: The corrected sign depends on the most flexible estimator, which might be biased.
- With the standard gamma frailty model, stays below one (0.90, Fig. 1A). Only the log-normal model and the NPMLE push it above one. In the authors' simulations the NPMLE overestimates . At a true of 1 it is 0.115 too high on average. With difficulty taken from a real dataset, it returns 1.15 at a true of 1 and 1.90 at 1.65. The grid used for alone moves the estimate from 1.64 to 1.87.
- The paper attempts to address this bias with the "17 of 20 models" test. In simulated data with a constant failure rate, 17 models never had intervals above one. The paper does not say how spread out task difficulty was in that simulation. If METR has a lot of near-impossible tasks, 17 is more likely by chance than the simulation suggests.
- The estimate also depends on how many times each task was run. On SWE-bench, going from 2 to 10 runs per task lowered by a median of 0.85, in all 14 settings (App. C.4). On METR, 80–83% of model–task pairs are all-pass or all-fail. The paper does not check how 1.85 changes with the number of runs on METR.
- Rerunning the simulation with the difficulty spread measured from real data would test this. Comparing how well the gamma, log-normal and NPMLE models fit the METR data would show whether gamma is the wrong model there. Refitting METR with repeats dropped at random would show how much depends on the number of runs.
-
W2: The agent-versus-human contrast uses the metric that the paper is questioning.
- The agent–human gap () is measured on wall-clock time. By the paper's own numbers this is the least reliable measurement/clock. A model's tokens per minute vary by a median factor of 19.7 between its slower and faster runs. Wall-clock time includes queueing and retries. The extra on this clock is unexplained. Agents likely wait on servers, unlike humans. The authors' fallback puts agents on tokens and humans on wall-clock time, which compares two different measures.
- A human failure likely means the person gave up, which differs from an agent failing. On rated task length the humans' is 0.60 (0.67 corrected), below one. The abstract mentions neither point.
- Refitting agents on time that excludes waiting (for example, wall-clock time minus the gaps between steps) would address the first point.
-
W3: The paper is very hard to follow.
- The paper is full of jargons even from the abstract.
- Most sentences are very terse; very hard to parse.
- Most result sentences are packed with numbers (and jargons).
- Key terms are used before they are defined.
- There is a chance that I misunderstood the paper (the weaknesses listed above) because it is presented poorly.
Questions for authors
The abstract says that, at equal difficulty, the task length an agent completes 80% of the time is about half the length it completes 50% of the time. This needs Assumption 1 (the spread of difficulty does not change with task length). Is there evidence for it? If not, the abstract might need to be revised.