acceptodds
Under review as a conference paper at ICLR 2027

Can LLM Agents Estimate Their Own Task-Completion Time Horizon?

Abstract

The increasing reliability of LLM agents has led to soaring demand for persistent assistants and autonomously running agents. Yet an overlooked limitation of current systems is their inability to reliably model how long their own runs will take. Despite its practical importance for agent scheduling, planning, and real-time coordination, this phenomenon has not been systematically studied. To bridge this gap, we propose an evaluation framework that decomposes run-time self-estimation into four categories: pre-run, post-run, relative, and compositional estimation. We further investigate prompting- and training-based calibration methods to reduce estimation error. We find that models appear to inherit general knowledge about task duration from training data, yet they anticipate poorly how much work their own runs will involve and how the deployment that executes them affects run time. These results expose an important weakness in how modern LLMs anticipate their own run time, with practical implications for persistent agents, time-sensitive applications, and real-time human–AI coordination.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLMs, agents, time, awareness, task completion, time horizon, time perception, temporal grounding, temporal self-estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V51zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether LLM agents can accurately estimate their own task-completion time. The authors introduce MetaTime, a benchmark comprising 182 tasks across single-turn, short agentic, and long-horizon settings, and evaluate four aspects of runtime estimation: prospective, retrospective, comparative, and compositional estimation. Through extensive experiments on multiple LLMs, the paper finds that agents generally struggle to predict their execution time, particularly for long-horizon tasks, while retrospective estimates are substantially more accurate. The authors further investigate the effects of deployment configurations, task composition, and incentives on estimation accuracy, and explore prompting-based and training-based calibration methods. The results suggest that current LLM agents have limited awareness of their own execution workloads and deployment-dependent runtime, while existing calibration methods only partially address these limitations.

Critical strengths

Important and underexplored research problem. This paper investigates an important yet relatively underexplored problem: whether LLM agents can accurately estimate their own task-completion time. This capability is highly relevant to practical agentic systems, especially for long-horizon planning, resource allocation, and multi-agent coordination. I appreciate the authors' effort to systematically formulate this problem and decompose it into four complementary evaluation settings.

Comprehensive empirical evaluation. The paper introduces MetaTime, a benchmark covering 182 tasks across different execution horizons, and conducts extensive experiments across multiple models and evaluation settings. Beyond measuring estimation accuracy, the authors investigate several relevant factors, including deployment sensitivity, compositional estimation, incentive effects, and calibration strategies. Overall, the experimental effort is substantial and provides a useful empirical characterization of current LLM agents' runtime estimation capabilities.

Critical weaknesses

Limited depth of empirical insights. Despite the extensive experiments, the paper's main findings are largely descriptive and often consistent with intuitive expectations. While the results establish the limitations of current LLM agents in runtime estimation, they offer relatively few insights beyond quantifying these limitations. Consequently, the scientific insights gained from the empirical evaluation appear somewhat limited relative to its scope.

Insufficient analysis of the underlying failure mechanisms. Although the paper explores several factors affecting runtime estimation, it does not systematically disentangle their contributions to estimation errors. In particular, the relative importance of workload prediction errors, deployment-related factors, and execution variability remains unclear. A more systematic analysis of these error sources would strengthen the paper's explanatory value and provide clearer directions for improving runtime estimation.

Questions for authors

I find this paper somewhat of a missed opportunity, as I genuinely appreciate the importance of this underexplored problem and its potential to provide valuable guidance for agent efficiency, but for an ICLR submission, I would have expected a more comprehensive and rigorously validated benchmark, along with deeper experimental analyses and insights.

1. Benchmark diversity and representativeness. Given the relatively limited number of tasks and the concentration of long-horizon tasks in a few existing benchmarks, how well do the findings generalize across different task domains and execution settings? Could the authors provide more fine-grained analyses to better establish the representativeness and robustness of MetaTime?

2. Sources of estimation errors. The paper identifies several factors affecting runtime estimation, but their relative contributions remain unclear. Can the authors provide a more systematic analysis to distinguish errors arising from workload prediction, deployment characteristics, and execution variability? Which factors are the primary bottlenecks across different task horizons?

3. Effectiveness beyond simple baselines. Some simple statistical baselines achieve comparable or even better performance than LLM-based estimation. To what extent do LLMs provide additional predictive value beyond task-level priors and execution statistics? Could the authors further investigate this under controlled, matched-information comparisons?

4. Definition and reliability of task-completion time. The paper measures runtime until an agent attempt terminates, regardless of whether the task is successfully completed. How does this definition affect the interpretation of task-completion time estimation? Are the main findings consistent when successful and unsuccessful runs are analyzed separately, particularly when execution is terminated by resource or time limits?

5. Practical implications for agent efficiency. While the paper motivates runtime estimation through scheduling, planning, and resource allocation, the evaluation primarily focuses on estimation accuracy. Can the authors demonstrate whether improvements in runtime estimation actually translate into more efficient agent execution or better resource allocation decisions? Such evidence would strengthen the practical significance of the benchmark.

Reviewer_G6JSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies runtime self-estimation: whether an LLM agent can predict its own wall-clock execution time. By constructing MetaTime, a 182-task benchmark spanning three regimes(long-horizon, short and single-turn), the author evaluates agent's estimation in four settings, with prompting- and training-based calibration. The experimental results reveal the weakness in LLMs' anticipating their own runtime. However, current self-estimation is weakly established: the estimates may largely reflect pre-trained duration priors plus work counting rather than knowledge specific to the model itself. Additionally, some experimental designs need careful refinements, such as the limited evaluated model diversity, the narrow calibration-by-training study, and possible timing leakage into the retrospective context.

Critical strengths
  1. Novel, practically relevant question with a clean decomposition. Runtime self-estimation hasn't been studied systematically. The author splits duration into work (calls, tools, tokens) and rate (deployment speed), and probing prospective/retrospective/comparative/compositional estimation separately, provides a clear categorization and makes the failures interpretable.
Critical weaknesses
  1. "Self"-estimation is weakly established; estimates look like pre-trained priors plus work counting.

    • Uncontrolled deployment: The long-horizon runs are released benchmark records whose deployment (snapshot, provider, rate limits, concurrency) is not under the author's control. At estimation time, the model is possibly different deployment than the one that produced the run.
    • Retrospective estimates may reduce to counting work and converting it to seconds: With actions visible in the transcript, three counts (requests, tool calls, output tokens) match or beat the full transcript, and a fixed counts × rate rule with no model does just as well (Table 8). No self-specific knowledge of speed is needed to explain the retrospective accuracy.
    • Counting and prior matching are not separated, and contamination is untested: Most long-horizon tasks come from public benchmarks (SWE-bench Pro/Verified in Harbor-Index, METR) whose discussions, leaderboards, and released trajectories may appear in training data, so "inherited knowledge" may include memorized durations for these specific tasks.
  2. Limited evaluation.

    • Only two frontier models are evaluated in the main study, both at medium effort.
    • Training is done only on single-turn tasks with Qwen3-8B on a local A100. Whether training helps for agentic runtime forecasting, the paper's main motivation, is not tested.
Questions for authors
  1. Self-estimation

    • Q1: Can we introduce a controlled deployment setting for evaluation? Under this setting, does a model estimate its own runs more accurately than another model estimates those same runs?
    • Q2: Can the authors test whether retrospective estimates reflect model-specific runtime knowledge by varying transcript work while keeping the task fixed?
    • Q3: How do estimates compare between public benchmark tasks and private custom tasks, where memorized task durations are less likely?
  2. Evaluation

    • Q4: Do the main findings hold across more models and controlled deployment settings?
    • Q5: Does multi-run training keep or improve runtime estimates? How about the variance of the runtime statistics?
Reviewer_cCnsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether LLM agents can accurately estimate their own task-completion time. The authors introduce MetaTime, a benchmark consisting of 182 tasks across long-horizon, short agentic, and single-turn settings, and evaluate four types of time estimation: prospective, retrospective, comparative, and compositional estimation. The authors show that LLM agents generally struggle to estimate their own execution time, particularly before executing long-horizon tasks, while retrospective estimation is substantially more accurate. They further investigate deployment sensitivity, calibration methods, and how the intended use of time estimates influences model predictions.

Critical strengths
  • I believe the paper addresses an interesting and practically important problem that has not been sufficiently explored in the context of long-running LLM agents. As LLM agents become increasingly capable of executing complex tasks, accurately estimating their own execution time is important for scheduling, resource allocation, and multi-agent coordination. I appreciate that the authors investigate this problem systematically across different task horizons and estimation settings.

  • The paper provides several interesting empirical findings. In particular, I find it surprising that models struggle to estimate their own execution time before starting a task, while they perform substantially better after observing the completed execution. I also appreciate the finding that simple statistical baselines can perform similarly to or better than LLM-based estimation, and that models do not adequately account for deployment-specific inference speed. Overall, I think the experimental analysis is comprehensive and provides useful insights into the limitations of current LLM agents.

Critical weaknesses
  • My main concern is that the paper does not sufficiently disentangle the effects of task complexity, agent scaffolding, and deployment configurations on execution-time estimation. Since the benchmark combines tasks from different sources and execution environments, it is unclear whether poor estimation performance primarily reflects limitations in the models' understanding of their own task requirements or uncertainty introduced by the execution environments. Although the authors provide some benchmark-specific and deployment analyses, I believe more controlled experiments with repeated executions would strengthen the conclusions.
  • I also think the evaluation could better reflect realistic time estimation for continuously running agents. The paper primarily focuses on predictions before and after task execution, while agents in practice may need to update their remaining-time estimates as they make progress. Although the authors provide a smaller experiment on intermediate time estimation, it is not investigated as extensively as the main evaluation. I believe a more detailed analysis of progress-aware remaining-time estimation would make the work more useful for real-world agent scheduling and coordination.
Questions for authors
  1. Can the authors provide a more controlled analysis of execution-time estimation across repeated runs of the same tasks? In particular, how much of the observed estimation error can be attributed to variation in agent execution, tool latency, or deployment configurations rather than the model's inability to anticipate its own workload? I think in order to really determine the "estimated runtime", the experiments need to be more rigorously designed.
  2. Can the authors further investigate remaining-time estimation during ongoing task execution? E.g., how does estimation accuracy change as an agent completes more subtasks, and can explicitly providing execution progress, elapsed time, or remaining steps improve its predictions?
  3. In the incentive experiment, how do the authors distinguish actual strategic estimation from prompt-framing effects? Would similar behavior emerge when deadlines are actually enforced rather than merely described to the model?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

Dear Area Chair,

We thank the reviewers for a constructive discussion, and summarise it below:

The reviewers recognised our work as an important and underexplored problem, while highlighting the comprehensive evaluation across tasks and estimation settings, the surprising findings, and the clarity of our presentation. To our understanding, no reviewer has an outstanding objection to the correctness of the results.

Critical concerns and resolution

  1. Does the error come from the execution environment rather than the estimates? We ran 171 controlled runs (19 tasks x 3 seeds x 3 models) with pinned snapshots, first-party serving, dedicated machine and no concurrency. Repeats of a task differ by about 1.1x, while the before-run estimates are off by 3.8-7.1x, ruling out run-to-run variation as the error source.

  2. What causes the error? The workload forecast is 2-3x too high while the implied time per token is within 20% of the truth, so models misjudge how much work they will do, not how fast they are served.

New experiments

  1. Progress-aware estimation (cCns) We stop each run at 25%, 50% and 75% and ask how much time remains, for three frontier models (4,813 estimates on 132 runs), models are still off by about 2x halfway through, with growing errors as the run goes on.

  2. Causal attribution by intervention (cCns, V51z, G6JS) We changed one factor at a time and measured both run time and estimate

    • Deployment: adding 10s to every command
    • Workload: two tasks in one session
    • Visible work: doubling or halving the work in a transcript
    • Incentive: with identical wording and a cost added for reserved time

Overall, findings indicate that the estimates follow the tasks, and are largely insensitive to the run’s execution dynamics

  1. Broader coverage (G6JS, V51z) We evaluated two newer frontier models and provided a breakdown over five workflow categories , both of which reproduced our main findings

Points clarified from existing material

Several concerns were already covered in the submission, and the reviewers accepted the clarifications where they responded

  • Timing leakage (G6JS): clock readings are removed from the transcripts (L407-409)
  • Memorised durations (G6JS): the ground-truth runs postdate each model's release, and private tasks are estimated no worse than public ones
  • Framing (cCns): the approval condition in the paper is already a framing control (L347, Table 6)
  • Successful vs failed runs (V51z): these are already analysed separately and estimated alike (Table 17)
  • Offline simulation (V51z): the enforced-deadline experiments are live runs, not replays

We will ensure that the revised version of the paper:

  • Adds new experiments, the settings table and per-task statistics
  • States the limitations on long-horizon repeats, GUI tasks and agentic training
  • Fixes all presentation issues

We believe the discussion has made the central claim more precise and better supported.

We thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.