- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces AgentTime, a benchmark evaluating three aspects of agents’ awareness and control of their own runtime: following a requested working duration, forecasting task runtime, and retrospectively estimating elapsed time. The suite contains 222 tasks drawn from 18 benchmarks, covering question answering, software engineering, computer use, and automated research. Agents are evaluated in their native harnesses while retaining the underlying benchmarks’ task scores. The main experiments report on time termination rates of 88% for GPT-6 Astra, 62% for GPT-5.6 Sol, and 16% for Fable 5.1. Within task analyses and harness swapping experiments further investigate these differences. Transcript audits show that on time termination can involve rechecking or explicit waiting. Forecasting experiments find that agents generally overestimate their natural runtimes, while retrospective experiments show that removing temporal cues reduces estimation accuracy.
Critical strengths
-
The evaluation addresses a practically important capability with substantial empirical coverage. The paper evaluates runtime control separately while retaining native task scores, exposing behaviors that task-success evaluation alone can miss. The 222 tasks across 18 benchmarks and multiple requested durations provide a substantial empirical basis. I particularly appreciate that the authors explicitly examine waiting behavior and acknowledge that rechecking can be useful work and that gaps in visible output do not establish an absence of reasoning. These qualifications improve the credibility of the interpretation.
-
The principal model differences are supported by targeted analyses of plausible confounders. The Astra–Fable gap also persists under harness swapping. Together, these analyses support the central observation and make task composition or the default harness less plausible as complete explanations.
Critical weaknesses
-
The evidence for sustained productive work and the value of additional time is weaker than the evidence for termination time control. The behavioral analysis in Figure 3 relies on subsets of classifiable transcripts, with 231, 126, and 158 runs for the three models. The main text does not sufficiently establish how these subsets were selected or how reliable the annotations are. The authors acknowledge the small and concentrated samples, but these limitations still constrain the generalizability of the reported behavioral proportions.
-
How were the transcript subsets in Figure 3b selected? Were task and duration distributions comparable across models? This part is not clear enough to judge.
Questions for authors
-
Does the forecasting evaluation include unsuccessful or abnormally terminated runs?
-
Were the trajectories underlying Figure 5 obtained from natural runs without requested durations or from duration-following runs?
2 comments
Response to the Rebuttal
Thanks for your careful rebuttal. My concerns are addressed and I am willing to maintain my positive score to support this paper to be accepted.
Reply to Reviewer 1q1r
Hi, thank you for your review! We’ve addressed your points in the paper. (New text is in blue. Figure numbers moved down by one, so your Figures 3 and 5 are now Figures 2 and 4.)
Regarding how the Figure 3b (now 2b) transcripts were selected: nothing was sampled. The panel uses every run with a kept timestamped transcript (252 runs, 153 agentic, primarily CORE-Bench, TUA-Bench and Terminal-Bench, plus GPQA Diamond and HLE questions). On the 47 task-request pairs where all three agents have timestamped transcripts, the order is unchanged: Fable 4%, Sol 38%, Astra 53% on time (Appendix F.2).
Regarding the reliability of the annotations: the submitted labels were made by a language model (Sonnet 5). We have now updated labeling with the majority of three blind labelers: GPT-6 Luna, Claude Sonnet 5.5 and Gemini 4 Hydrogen. They agree on whether an agentic run kept working in 89 to 98% of runs (Fleiss’ kappa 0.78), and Transect (a UK AISI transcript reading method) agrees with the majority in about 95% of agentic runs (Appendix F.2, Table 11).
Questions:
No, all forecasting runs ran normally. Runs with a wrong answer are included, since they still have a runtime (Section 4.2). Some duration-following runs ran past our hidden cutoff at twice the task’s longest request (29 runs, which we keep; Section 4.1). For the six newer agents, 97% of natural runs ended cleanly (Appendix F.5).
From natural runs, with no requested duration (Section 4.3). Figure 4 now shows nine agents: with the six added agents and their natural runs (Appendix F.6, Table 14). Agents in their native harnesses still struggle to recollect how long they spent on a task.