acceptodds
Under review as a conference paper at ICLR 2027

AgentTime: Can Agents Estimate and Control Their Own Runtime?

Abstract

An essential aspect of controlling AI agents is their ability to manage runtime. This ability requires a sense of time-awareness, to predict and estimate wall-clock time and to control their own reasoning and actions. Prior work has focused on time-awareness, but duration-following and control in native agent harnesses remain unexplored. We present AgentTime, a benchmark for testing whether agents can work for a requested duration, predict their runtime, and estimate elapsed time afterward. It comprises 222 tasks from 18 sources spanning coding, computer use, agentic work, and automated research. Duration-following experiments append a single instruction specifying how long to work, with requests ranging from about a minute to multiple days. Accuracy on these instructions varies substantially: Fable 5.1 in Claude Code deviates from requested runtimes by a typical factor of 2.9, compared with only 1.2 for GPT-6 Astra in Codex. However, matching the requested runtime does not, by itself, establish continued work on the task. Among 158 reviewed Astra runs with classifiable transcripts, 14 explicitly slept after appearing to finish. In forecasting experiments, predictions tend to overestimate natural runtimes. In retrospective experiments, removing temporal information more than doubles deviation for Sol and Astra and nearly doubles it for Fable. An agent's ability to complete a task does not guarantee that it can control its own time or work for the whole requested duration. For agents to run reliably, safely, and autonomously over long horizons, we require the evaluation of both.

Primary area:
datasets and benchmarks
Keywords:
Evaluations, Time, Time-awareness, Benchmark, self-control, multi-agents

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_1q1rOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AgentTime, a benchmark evaluating three aspects of agents’ awareness and control of their own runtime: following a requested working duration, forecasting task runtime, and retrospectively estimating elapsed time. The suite contains 222 tasks drawn from 18 benchmarks, covering question answering, software engineering, computer use, and automated research. Agents are evaluated in their native harnesses while retaining the underlying benchmarks’ task scores. The main experiments report on time termination rates of 88% for GPT-6 Astra, 62% for GPT-5.6 Sol, and 16% for Fable 5.1. Within task analyses and harness swapping experiments further investigate these differences. Transcript audits show that on time termination can involve rechecking or explicit waiting. Forecasting experiments find that agents generally overestimate their natural runtimes, while retrospective experiments show that removing temporal cues reduces estimation accuracy.

Critical strengths
  1. The evaluation addresses a practically important capability with substantial empirical coverage. The paper evaluates runtime control separately while retaining native task scores, exposing behaviors that task-success evaluation alone can miss. The 222 tasks across 18 benchmarks and multiple requested durations provide a substantial empirical basis. I particularly appreciate that the authors explicitly examine waiting behavior and acknowledge that rechecking can be useful work and that gaps in visible output do not establish an absence of reasoning. These qualifications improve the credibility of the interpretation.

  2. The principal model differences are supported by targeted analyses of plausible confounders. The Astra–Fable gap also persists under harness swapping. Together, these analyses support the central observation and make task composition or the default harness less plausible as complete explanations.

Critical weaknesses
  1. The evidence for sustained productive work and the value of additional time is weaker than the evidence for termination time control. The behavioral analysis in Figure 3 relies on subsets of classifiable transcripts, with 231, 126, and 158 runs for the three models. The main text does not sufficiently establish how these subsets were selected or how reliable the annotations are. The authors acknowledge the small and concentrated samples, but these limitations still constrain the generalizability of the reported behavioral proportions.

  2. How were the transcript subsets in Figure 3b selected? Were task and duration distributions comparable across models? This part is not clear enough to judge.

Questions for authors
  1. Does the forecasting evaluation include unsuccessful or abnormally terminated runs?

  2. Were the trajectories underlying Figure 5 obtained from natural runs without requested durations or from duration-following runs?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer 1q1r

Hi, thank you for your review! We’ve addressed your points in the paper. (New text is in blue. Figure numbers moved down by one, so your Figures 3 and 5 are now Figures 2 and 4.)

  1. Regarding how the Figure 3b (now 2b) transcripts were selected: nothing was sampled. The panel uses every run with a kept timestamped transcript (252 runs, 153 agentic, primarily CORE-Bench, TUA-Bench and Terminal-Bench, plus GPQA Diamond and HLE questions). On the 47 task-request pairs where all three agents have timestamped transcripts, the order is unchanged: Fable 4%, Sol 38%, Astra 53% on time (Appendix F.2).

  2. Regarding the reliability of the annotations: the submitted labels were made by a language model (Sonnet 5). We have now updated labeling with the majority of three blind labelers: GPT-6 Luna, Claude Sonnet 5.5 and Gemini 4 Hydrogen. They agree on whether an agentic run kept working in 89 to 98% of runs (Fleiss’ kappa 0.78), and Transect (a UK AISI transcript reading method) agrees with the majority in about 95% of agentic runs (Appendix F.2, Table 11).

Questions:

  1. No, all forecasting runs ran normally. Runs with a wrong answer are included, since they still have a runtime (Section 4.2). Some duration-following runs ran past our hidden cutoff at twice the task’s longest request (29 runs, which we keep; Section 4.1). For the six newer agents, 97% of natural runs ended cleanly (Appendix F.5).

  2. From natural runs, with no requested duration (Section 4.3). Figure 4 now shows nine agents: with the six added agents and their natural runs (Appendix F.6, Table 14). Agents in their native harnesses still struggle to recollect how long they spent on a task.

Reviewer_1q1rOpenReview ↗

Response to the Rebuttal

Thanks for your careful rebuttal. My concerns are addressed and I am willing to maintain my positive score to support this paper to be accepted.

Reviewer_LsXyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AgentTime, 222 tasks drawn from 18 existing benchmarks (Table 2). It evaluates three temporal skills of agents running in their native CLI harnesses (Claude Code, Codex), with native time caps removed and runtime measured by an external timer (Sec. 3):

  • duration-following: working for a requested time;
  • forecasting: predicting one's own natural runtime;
  • retrospection: estimating the elapsed time of a finished run.

Duration-following (Sec. 4.1). Requests range from ~1 minute to 60 hours, with three requests per task. Results (Table 3, Figs. 1–3):

Agent (harness)On timeTypical deviation
GPT-6 Astra (Codex)88%1.18×1.18\times
GPT-5.6 Sol (Codex)62%1.77×1.77\times
Fable 5.1 (Claude Code)16%2.86×2.86\times
  • Within-task slopes are 0.90, 0.58 and 0.24 respectively.
  • A harness swap indicates that the gap comes mainly from the model.
  • On-time runs are not necessarily working: among 158 classifiable Astra transcripts, 60% re-checked earlier work and 14 explicitly slept.
  • A ~16×16\times longer request left the native score unchanged on ~62–64% of tasks.

Forecasting (Sec. 4.2). Forecasts are mostly too high (63–83%), with typical misses of 2.62.6–3.8×3.8\times.

Retrospection (Sec. 4.3). Estimates are accurate when time cues are present (1.031.03–1.33×1.33\times) but degrade to 2.62.6–5.4×5.4\times when cues are scrubbed (Fig. 5).

Critical strengths
  1. A new, practically important evaluation axis, measured at realistic scale.
    • 1,991 duration-following runs (Fig. 2) spanning a minute to 60 hours;
    • native harnesses and native graders;
    • ~1,700 agent-hours and ~$93k of compute (Sec. 6).
  2. Analyses that rule out obvious confounds and separate "on time" from "working".
    • The within-task slope removes the harder-task-gets-longer-request confound: pooled slopes of 0.59/0.71/0.94 become within-task slopes of 0.24/0.58/0.90.
    • The harness swap (Table 3, lower block) attributes the gap mainly to the model.
    • Transcript review (Fig. 3b) shows that ending on time can mean re-checking or sleeping.
Critical weaknesses
  1. Each condition is a single run, so request effects are confounded with run-to-run variance. "Each agent ran each task once per requested duration" (Sec. 6), and natural runtimes for forecasting are also single runs. Within-task slopes are fit to three single-run points per task. The small provider check in Appendix B, which repeats 35 Fable question runs, already shows noticeable run-to-run movement (6 vs 8 on time, 3.78×3.78\times vs 2.95×2.95\times). The "more time rarely changes the score" result (Fig. 3c, 62–64% "same") is especially fragile: 11 of 18 benchmarks are pass/fail, so tasks that are trivially solved or unsolvable at both durations score "same" automatically.
  2. The headline metric's normative target is ambiguous, and the transcript labelling is not specified.
    • The prompt asks the agent to "work on this task for a full {N} {unit}" while doing its best (Sec. 4.1). For short QA (GPQA + HLE = 52 of 222 tasks), padding or sleeping is the only way to be "on time", while returning early is arguably more useful. The on-time ranking (Astra ≫\gg Fable) partly rewards padding.
    • The Working/Re-checked/Slept labels (Fig. 3b), which carry the "on time ≠\neq working" claim, are defined only by a one-line rule in the Fig. 3 caption ("the result changed in the last fifth of the run, or a needed job ran"). It is not stated who applied them or how they were validated. The AI-use statement implies model-produced labels ("seven model labels ... were replaced"). The samples are also small (64 Astra and 23 Sol on-time agentic runs).
Questions for authors
  1. Can you add repeated runs (e.g. 2–3 per condition on a 30–50 task subset) to quantify variance in runtime and native score, and break down Fig. 3c by whether the task was already solved at the shortest request?
  2. How were the Working/Re-checked/Slept labels assigned (humans or a model; which rubric), and what is the inter-annotator agreement?
  3. Section 4.1 gives Fable's open-ended slope (0.77) and Appendix Table 6 splits the swap subset into questions and agentic tasks. Can you report the full Table 3 (on-time rate, deviation, slope for all three agents) separately for the open-ended tasks and the closed QA tasks over all 222 tasks? What behavior do you consider correct when a task finishes well before the requested time?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer LsXy

Thank you for your review! Your points on variance, labels and padding were very helpful, and we’ve added all three to the paper. (New text is in blue; your Figure 3 is now Figure 2.)

  1. Regarding single runs: 251 runs already ran 2 to 10 times under identical conditions, and even pessimistic 95% ranges keep the agents apart: Astra 50-67%, Sol 26-42%, Fable 1-7% (Appendix F.1, Table 9).

  2. Regarding Figure 3c (now 2c): agreed. 37% of tasks were already at the top score at the shortest request, and panel (c) now hatches them. On the rest, a longer request raised the score about three times as often as it lowered it (Appendix F.4).

  3. Regarding padding on short QA: on open-ended tasks, where padding is never needed, the order holds: Astra was on time in 98% of runs, Sol in 40%, and Fable in 4% (Appendix F.3, Table 12).

Questions:

  1. Yes we can and we did! We ran two fresh copies on 40 agentic tasks with Astra, GPT 6.1 Sol and Claude Opus 5.5 (720 runs in total). The order holds in every copy, slopes move by at most 0.03, and the two fresh copies get the same native score in 84 to 87% of runs (Appendix F.1, Table 9). Forecasts, natural runtimes and retrospection got the same check (Table 10).

  2. In the original version the labels were made by a language model (Sonnet 5). We have updated labeling with three blind labelers under one written rule, one model from each of three makers: GPT-6 Luna, Claude Sonnet 5.5 and Gemini 4 Hydrogen. Each run takes the label at least two of them give (Appendix F.2). They agree on whether an agentic run kept working in 89 to 98% of runs (Fleiss’ kappa 0.78), and two other methods, UK AISI’s Transect and a model-free timestamp rule, agree with the majority in about 95% of agentic runs. Questions agree less, since the agents’ reasoning is mostly hidden (Table 11).

  3. Yes: Table 12 splits Table 3 into closed questions, open-ended and other agentic tasks for all nine agents. The paper’s original three agents keep their order in every group, and GPT 6.1 Sol stays ahead of all the new agents added (Appendix F.3). For many open-ended tasks (PostTrainBench, ProgramBench, and PaperBench), we think agent behavior should be to keep checking or improving it instead of sleeping. In closed or early finished tasks, sleeping is a correct kind of behavior according to the prompt. However an agent should disclose this to the user.

Reviewer_LsXyOpenReview ↗

Follow-up after rebuttal

Thanks for the detailed response and additional experiments. The repeated runs, the open-ended versus closed-QA breakdown, and the improved transcript labelling address my main concerns, so I am keeping my rating of 3 and continue to recommend acceptance.

Reviewer_gtSvOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AgentTIme is a benchmark that includes 222 tasks from 18 known benchmarks. It's goal is to test the following in a harness' ability to work with time: work for a required amount of time, forecast execution time, and estimating elapsed time afterwards. The authors apply it on Astra and Fable and report the results. Some interesting observations include that being on time is not the same as working AND having more time does not necessarily improve score significant across many tasks. To get good introspection, one need needs good cues (e.g., timestamps in the logs. Model forecasts are pessimistic, unlike human forecasts that tend to be shorter.

Critical strengths
  • A benchmark that evaluates wall clock duration control in AI model and their harnesses.
  • Pretty good analysis: thousands of runs, careful treatment of the task length, separation of model vs tool impact, honest presentation of shortcomings.
  • I found the observation about on time Vs actually working to be very useful.
Critical weaknesses
  • the paper measures but does not propose any fixes. It feels rather thin in that sense.
  • coverage is narro: 3 agents from 2 companies, single framing of instructions.
  • unclear if the results are repeatable: each task is run once, so how do we know what is an important observation vs random variation?
Questions for authors
  • Have you done any experiments to estimate per-task variance and its impact on your findings?

  • What is the source and quality of transcript labels?

  • Can you explain how serving speed factors in here? A model that is served at a slower pace (TPS) will take longer so it may look like poor time control while the issue is speed/overload.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer gtSv

Thank you for your review! All your questions and concerns were crucial. And we’ve added them to our paper! (New text is in blue; your Figure 3 is now Figure 2.)

Regarding your first point on proposing fixes, we believe that including more timestamps in SFT training data, RL environments that test for duration control, and giving agents clock access in the harness would allow them to have better awareness and control over their own runtime, while telling them their throughput speed shifts forecasts shorter and helps overestimators (Section 5, Appendix F.7). A clock tool already helps: on 32 closed-book questions at 4x forecast, it brought GPT 6.1 Sol from 2.2x to 1.04x the requested time (0% to 66% on time).

To your second point, we have added 6 more agents from five makers, and they can be found in the revised paper (Figures 1, 3 and 4, Table 3, Appendix F.5). As well as an open-weight model at two throughput speeds (MiMo V2.6 Pro & MiMo V2.6 Pro UltraSpeed). Only GPT 6.1 Sol follows a request as well as Astra; the other five are on time in at most 11% of runs. We also added an ablation study on repeating tasks: 251 runs repeated 2 to 10 times (950 extra runs; 1% median variation for Astra, 3% for Sol) plus two fresh copies on a 40-task agentic sample (720 runs in total; Appendix F.1, Tables 9 and 10). The agents keep the same order in every copy.

To your question on the source and quality of transcript labels, the first labels came from a language model (Sonnet 5). We have updated labeling with three blind labelers, one model from each of three makers, for inter-agreement scores: GPT-6 Luna, Claude Sonnet 5.5 and Gemini 4 Hydrogen (Appendix F.2, Table 11). Each run takes the majority label, and the three agree on whether an agentic run kept working in 89 to 98% of runs (Fleiss’ kappa 0.78). Two checks with other methods, Transect from a recent UK AISI paper and a model-free timestamp rule, agree with the majority in about 95% of agentic runs.

Regarding serving speed factors, we tested open-weight MiMo V2.6 Pro in its coding agent environment fed through the API at 36 tokens per second and 140 tokens per second. The faster tier worked 0.40 to 0.44x as long, kept forecasts nearly unchanged (1.09x), and was no more often on time (3% vs. 4%, stopping early on 80% of runs; Table 3, Appendix F.5).

We also added an ablation study where we tell models their exact throughput speed as reported by Artificial Analysis, and we see that on some models, forecasts improve! And generally agents predict faster runtimes when they know their own throughput speed: forecasts got about a quarter shorter, which helped the two agents that overestimate the most, GPT 6.1 Sol and Muse Wonder 1.0 (Section 4.2, Appendix F.7, Figure 17).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our revisions according to reviews

We thank all the reviewers! We’ve provided edits in the paper (both the main paper and the appendix) in blue. We’ve used a stricter on-time rule (within 5% of the request, not 0.8 to 1.25x), and Figures 2 to 5 are now 1 to 4. None of the rankings have changed.

All three reviewers noted these concerns and we have addressed them all below.

  1. Coverage. We added six more agents (3,600 more runs), including an open-weight model and the same model with different throughput speeds. They ran on a revised set of 200 instead of 222 tasks (we removed tasks that our own and Epoch AI’s benchmark reviews found broken), with each task's shortest and longest request. Figures 1, 3 and 4 now show all nine: duration following, forecasts and retrospection on those 200 tasks. GPT models scale their runtime best with the request, the others do not. Appendix F.8 (Table 15) shows every version: the original three agents keep their order in each.

  2. Repeats. We had run 251 runs 2-10 times (950 extra runs). A repeat landed within 1% of the first run for Astra, 3% for Sol and 1.3x for Fable. The difference between agents remains significant: Astra 50-67%, Sol 26-42%, Fable 1-7%. We also ran repeats on 40 agentic tasks multiple times and they also kept the order (720 runs in total, Astra 48-58%, GPT 6.1 Sol 59-60%, Opus 5.5 0-3%). In repeated runs, forecasts typically differ by 1.14 to 1.5x, natural runs by 1.15 to 1.24x on the 40-task set, and retrospective answers are usually identical.

  3. Labels. We used LLMs for labeling transcripts, we initially used Sonnet 5. We have now updated with three blind labelers, one model from each of three makers (GPT-6 Luna, Claude Sonnet 5.5 and Gemini 4 Hydrogen), and each run takes the majority label. For inter-label agreement: they agree with each other on whether an agentic run kept working in 89 to 98% of runs (Fleiss’ kappa 0.78), and in 90 to 100% of Astra’s 49 on-time agentic runs (Table 11). UK AISI’s Transect, a different method, agrees with the majority in 94%.

  4. Further ablations. We also told four agents their own output speed (tokens per second) to see whether their forecasts would change. Asked without reference, models under-estimate their own tokens per second compared with the speeds Artificial Analysis reports. So when given to them, all four shortened their forecasts, by 17-31%. This helped GPT 6.1 Sol and Muse Wonder, as they generally forecast too long, but hurt Opus 5.5 and Gemini 4 Argon as they already forecast too short (Section 4.2, Section 7, Appendix F.7, Figure 17).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.