- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether LLM agents can accurately estimate their own task-completion time. The authors introduce MetaTime, a benchmark comprising 182 tasks across single-turn, short agentic, and long-horizon settings, and evaluate four aspects of runtime estimation: prospective, retrospective, comparative, and compositional estimation. Through extensive experiments on multiple LLMs, the paper finds that agents generally struggle to predict their execution time, particularly for long-horizon tasks, while retrospective estimates are substantially more accurate. The authors further investigate the effects of deployment configurations, task composition, and incentives on estimation accuracy, and explore prompting-based and training-based calibration methods. The results suggest that current LLM agents have limited awareness of their own execution workloads and deployment-dependent runtime, while existing calibration methods only partially address these limitations.
Critical strengths
Important and underexplored research problem. This paper investigates an important yet relatively underexplored problem: whether LLM agents can accurately estimate their own task-completion time. This capability is highly relevant to practical agentic systems, especially for long-horizon planning, resource allocation, and multi-agent coordination. I appreciate the authors' effort to systematically formulate this problem and decompose it into four complementary evaluation settings.
Comprehensive empirical evaluation. The paper introduces MetaTime, a benchmark covering 182 tasks across different execution horizons, and conducts extensive experiments across multiple models and evaluation settings. Beyond measuring estimation accuracy, the authors investigate several relevant factors, including deployment sensitivity, compositional estimation, incentive effects, and calibration strategies. Overall, the experimental effort is substantial and provides a useful empirical characterization of current LLM agents' runtime estimation capabilities.
Critical weaknesses
Limited depth of empirical insights. Despite the extensive experiments, the paper's main findings are largely descriptive and often consistent with intuitive expectations. While the results establish the limitations of current LLM agents in runtime estimation, they offer relatively few insights beyond quantifying these limitations. Consequently, the scientific insights gained from the empirical evaluation appear somewhat limited relative to its scope.
Insufficient analysis of the underlying failure mechanisms. Although the paper explores several factors affecting runtime estimation, it does not systematically disentangle their contributions to estimation errors. In particular, the relative importance of workload prediction errors, deployment-related factors, and execution variability remains unclear. A more systematic analysis of these error sources would strengthen the paper's explanatory value and provide clearer directions for improving runtime estimation.
Questions for authors
I find this paper somewhat of a missed opportunity, as I genuinely appreciate the importance of this underexplored problem and its potential to provide valuable guidance for agent efficiency, but for an ICLR submission, I would have expected a more comprehensive and rigorously validated benchmark, along with deeper experimental analyses and insights.
1. Benchmark diversity and representativeness. Given the relatively limited number of tasks and the concentration of long-horizon tasks in a few existing benchmarks, how well do the findings generalize across different task domains and execution settings? Could the authors provide more fine-grained analyses to better establish the representativeness and robustness of MetaTime?
2. Sources of estimation errors. The paper identifies several factors affecting runtime estimation, but their relative contributions remain unclear. Can the authors provide a more systematic analysis to distinguish errors arising from workload prediction, deployment characteristics, and execution variability? Which factors are the primary bottlenecks across different task horizons?
3. Effectiveness beyond simple baselines. Some simple statistical baselines achieve comparable or even better performance than LLM-based estimation. To what extent do LLMs provide additional predictive value beyond task-level priors and execution statistics? Could the authors further investigate this under controlled, matched-information comparisons?
4. Definition and reliability of task-completion time. The paper measures runtime until an agent attempt terminates, regardless of whether the task is successfully completed. How does this definition affect the interpretation of task-completion time estimation? Are the main findings consistent when successful and unsuccessful runs are analyzed separately, particularly when execution is terminated by resource or time limits?
5. Practical implications for agent efficiency. While the paper motivates runtime estimation through scheduling, planning, and resource allocation, the evaluation primarily focuses on estimation accuracy. Can the authors demonstrate whether improvements in runtime estimation actually translate into more efficient agent execution or better resource allocation decisions? Such evidence would strengthen the practical significance of the benchmark.
Discussion Summary
Dear Area Chair,
We thank the reviewers for a constructive discussion, and summarise it below:
The reviewers recognised our work as an important and underexplored problem, while highlighting the comprehensive evaluation across tasks and estimation settings, the surprising findings, and the clarity of our presentation. To our understanding, no reviewer has an outstanding objection to the correctness of the results.
Critical concerns and resolution
Does the error come from the execution environment rather than the estimates? We ran 171 controlled runs (19 tasks x 3 seeds x 3 models) with pinned snapshots, first-party serving, dedicated machine and no concurrency. Repeats of a task differ by about 1.1x, while the before-run estimates are off by 3.8-7.1x, ruling out run-to-run variation as the error source.
What causes the error? The workload forecast is 2-3x too high while the implied time per token is within 20% of the truth, so models misjudge how much work they will do, not how fast they are served.
New experiments
Progress-aware estimation (cCns) We stop each run at 25%, 50% and 75% and ask how much time remains, for three frontier models (4,813 estimates on 132 runs), models are still off by about 2x halfway through, with growing errors as the run goes on.
Causal attribution by intervention (cCns, V51z, G6JS) We changed one factor at a time and measured both run time and estimate
Overall, findings indicate that the estimates follow the tasks, and are largely insensitive to the run’s execution dynamics
Points clarified from existing material
Several concerns were already covered in the submission, and the reviewers accepted the clarifications where they responded
We will ensure that the revised version of the paper:
We believe the discussion has made the central claim more precise and better supported.
We thank you for your time and consideration.