- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper evaluates how well LLMs understand time. The authors answer this question by proposing 4 different kinds of experiments, each experiment answering a different question. They are: (1) do models know basic rules about time, (2) whether they can estimate how long a task takes, (3) LLMs’ duration judgments are compared with human judgments and (4) whether they consider waiting time when making choices. They find that the models are good at the basic rules but very bad at estimating how long their own tasks take, often by huge amounts.
Critical strengths
I feel the problem this paper tries to formulate and understand is very timely. Evaluating how LLMs understand and process time, which they never experience directly, matters more and more as agents schedule work and run long tasks. The evaluation setup is rigorous and well designed as they evaluate across four levels progressing from rules to decisions. I also like the organization of the paper. It is also easy to read and well organized: each experiment is built to answer one clear research question which makes the findings easy to follow.
Critical weaknesses
In Experiment 2, the authors use only 16 tasks from the METR benchmark to test how well LLMs estimate completion time. Most of these tasks involve coding and research. But LLMs are also being used for other tasks too so it be interesting to know whther these results generalize to other tasks too.
Questions for authors
- How do the duration estimated by LLMs compare with simple baselines such as using the average or median completion time of similar tasks?
- Is there any relationship between performance across the four experiments? What i mean is that if being good at one type of time tasks make LLM better at any other task?
4 comments
Rebuttal to Reviewer dCkq
3. “Is there any relationship between performance across the four experiments? What i mean is that if being good at one type of time tasks make LLM better at any other task?”
Yes, our evaluations are conceptually structured such that the tested capabilities of later experiments rely on competence from earlier ones. In order, they go from the most basic intuitive rules of time (exp 1) to simple quantitative measurements (i.e., estimation, exp 2) that depend on these rules, to biases that influence these measurements (exp 3), and finally to decisions (exp 4, the bread-and-butter of agents) which depend on estimations.
Exp 1 measures models’ knowledge about rules of time, which are a prerequisite to estimation (exp 2, 3). For instance, without knowledge of how skill affects task duration (Axiom 1), models would struggle at estimating how long a person or agent takes on tasks. Similarly, without knowledge about how durations compose (Theorems 2–5), one cannot properly estimate the duration of a complex task that breaks down into subtasks.
Estimation (exp 2, 3) is then a pre-requisite to competent decision making (exp 4). Good decisions rely on having proper estimates of how long tasks will take. We illustrate this through a new ablation to exp 4 (see response to reviewer B5TY point 2, second experiment). TLDR, if we provide models with the exact times for how long they take both using and not using the tool, models become significantly more sensitive to time costs in their decisions than if they had to estimate those durations themselves. The takeaway is that an agent that cannot estimate durations well will make decisions that are not in accordance with their utility profile if they had more complete information.
To help guide future readers, we have added sentences describing the connections between our experiments at the start and end of each experiment section. Thank you for your feedback!
Any remaining questions?
Dear Reviewer dCkq,
Thank you again for your kind review. As there are less than 24 hours left in the discussion period, we would like to ask if you have any lingering questions or comments.
Cheers, Authors
Thank you for detailed response. I think it addresses all my concern and i will maintain my positive score
Rebuttal to Reviewer dCkq
Dear Reviewer dCkq, Thank you for your kind review. We respond to your points below.
1. “But LLMs are also being used for other tasks too so it be interesting to know whther these results generalize to other tasks too.”
Completely agree! We conduct a new experiment on whether models can estimate response times to users’ everyday queries. We filtered WildChat-1M (Zhao et. al., 2024) for english, non-toxic, and non-redacted user requests, and then randomly sampled 500. We have GPT-5.6 Sol, Claude Opus 5, and Gemini 3.7 Flash estimate the duration it takes to respond back to the user, including API latency, five times per user request. We compare these against ground truth runtimes that we collect by directly querying the model through said API.
Results are shown in the table below. Average estimate-to-runtime ratios were 0.50–0.94, indicating underestimation, consistent with previous findings in experiment 2.
Table: WildChat response-time estimation results. A bias factor below one indicates underestimation; the absolute error factor summarizes error magnitude irrespective of direction. Within-2× and within-4× report the proportion of model estimates in the corresponding multiplicative ranges of the ground truth.
2. “How do the duration estimated by LLMs compare with simple baselines such as using the average or median completion time of similar tasks?”
We find that using naive mean and median runtimes from other tasks in the same category as estimates performs surprisingly well: both achieve 100% within-2x accuracy, meaning that the estimates are all between half and double the ground truth value. Mean multiplicative absolute error values are between 1.06–1.40x (with 1 indicating perfect estimation). These greatly outperform models’ original estimates (see table below).
Because the runtimes of similar tasks are so informative, we additionally run an experiment that provides these runtimes in context (i.e., few-shot) to see if they can better calibrate LLMs. For each task, we provide ground truth runtimes for 1–2 tasks from the same task category, along with their instructions and environment. We do this for all 42 same-category task pairs and all 64 same-category triplets, resulting in 106 few-shot estimation problems. We repeat each estimation 10 times and average across estimates per task.
Our results indicate that few-shot calibration is very effective. Estimates are more accurate, particularly for Claude and Gemini, but GPT continues to greatly underestimate times. See table below, as well as figure: https://imgur.com/VtWyun4. We plan to add this to the main paper as a potential solution for exp 2. We thank the reviewer for suggesting the analysis that contributed towards this improvement.
Error factor is a multiplicative absolute-error metric (1× is perfect); coverage is the proportion of tasks whose ground truth is in the estimated interval.