acceptodds
Under review as a conference paper at ICLR 2027

In Search of Lost Time: Temporal Competence in Large Language Models

Abstract

Today's large language models (LLMs) increasingly interact with time: AI agents schedule tasks, coordinate with people and other AI systems, and even decide how long to think. Yet LLMs only ever encounter time secondhand—in descriptions rather than experience. Does this hinder their reasoning and decisions that involve time? Our paper traces this question from a model's knowledge to its actions through four successive evaluations. We find that models know the rules of time, which can be read, but struggle to predict durations, which must be lived; while they are highly accurate (91.9–95.8%) on a collection of questions built from the theorems of various time-related literatures, they underestimate their own completion times by up to 99.5% and fail to identify which tasks will take them longer to run. And while they are better at estimating durations of human tasks, their duration judgments magnify certain human biases by 1.3 simply based on how an event is worded. In practice, we find that some LLMs disregard time costs completely when answering user queries, ignoring a 1,200-fold increase in wait time entirely. Together, our experiments reveal new challenges posed to LLMs as they navigate a dimension they cannot directly perceive, and highlight how blind spots in LLMs can even be simple things that humans take for granted.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Time, Large Language Models, Cognitive Science, Evaluations

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dCkqOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper evaluates how well LLMs understand time. The authors answer this question by proposing 4 different kinds of experiments, each experiment answering a different question. They are: (1) do models know basic rules about time, (2) whether they can estimate how long a task takes, (3) LLMs’ duration judgments are compared with human judgments and (4) whether they consider waiting time when making choices. They find that the models are good at the basic rules but very bad at estimating how long their own tasks take, often by huge amounts.

Critical strengths

I feel the problem this paper tries to formulate and understand is very timely. Evaluating how LLMs understand and process time, which they never experience directly, matters more and more as agents schedule work and run long tasks. The evaluation setup is rigorous and well designed as they evaluate across four levels progressing from rules to decisions. I also like the organization of the paper. It is also easy to read and well organized: each experiment is built to answer one clear research question which makes the findings easy to follow.

Critical weaknesses

In Experiment 2, the authors use only 16 tasks from the METR benchmark to test how well LLMs estimate completion time. Most of these tasks involve coding and research. But LLMs are also being used for other tasks too so it be interesting to know whther these results generalize to other tasks too.

Questions for authors
  1. How do the duration estimated by LLMs compare with simple baselines such as using the average or median completion time of similar tasks?
  2. Is there any relationship between performance across the four experiments? What i mean is that if being good at one type of time tasks make LLM better at any other task?
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer dCkq

Dear Reviewer dCkq, Thank you for your kind review. We respond to your points below.

1. “But LLMs are also being used for other tasks too so it be interesting to know whther these results generalize to other tasks too.”

Completely agree! We conduct a new experiment on whether models can estimate response times to users’ everyday queries. We filtered WildChat-1M (Zhao et. al., 2024) for english, non-toxic, and non-redacted user requests, and then randomly sampled 500. We have GPT-5.6 Sol, Claude Opus 5, and Gemini 3.7 Flash estimate the duration it takes to respond back to the user, including API latency, five times per user request. We compare these against ground truth runtimes that we collect by directly querying the model through said API.

Results are shown in the table below. Average estimate-to-runtime ratios were 0.50–0.94, indicating underestimation, consistent with previous findings in experiment 2.

ModelNMBias factorAbs. error factorWithin 2×Within 4×ρ
Claude Opus 54970.941.9652.5%95.6%0.61
GPT-5.6 Sol5000.701.6173.4%96.8%0.86
Gemini 3.7 Flash4980.502.0446.2%99.4%0.82

Table: WildChat response-time estimation results. A bias factor below one indicates underestimation; the absolute error factor summarizes error magnitude irrespective of direction. Within-2× and within-4× report the proportion of model estimates in the corresponding multiplicative ranges of the ground truth.

 

2. “How do the duration estimated by LLMs compare with simple baselines such as using the average or median completion time of similar tasks?”

We find that using naive mean and median runtimes from other tasks in the same category as estimates performs surprisingly well: both achieve 100% within-2x accuracy, meaning that the estimates are all between half and double the ground truth value. Mean multiplicative absolute error values are between 1.06–1.40x (with 1 indicating perfect estimation). These greatly outperform models’ original estimates (see table below).

ModelMethodBias factorAbsolute error factor ↓Within 2×Within 4×
ClaudeOriginal prediction0.6972.11×64.3%85.7%
Naive mean1.1241.13×100%100%
Naive median1.4011.40×100%100%
GPTOriginal prediction0.005197.58×0%0%
Naive mean1.0261.15×100%100%
Naive median1.0321.15×100%100%
GeminiOriginal prediction0.02148.26×0%0%
Naive mean1.0071.06×100%100%
Naive median0.9931.07×100%100%

Because the runtimes of similar tasks are so informative, we additionally run an experiment that provides these runtimes in context (i.e., few-shot) to see if they can better calibrate LLMs. For each task, we provide ground truth runtimes for 1–2 tasks from the same task category, along with their instructions and environment. We do this for all 42 same-category task pairs and all 64 same-category triplets, resulting in 106 few-shot estimation problems. We repeat each estimation 10 times and average across estimates per task.

Our results indicate that few-shot calibration is very effective. Estimates are more accurate, particularly for Claude and Gemini, but GPT continues to greatly underestimate times. See table below, as well as figure: https://imgur.com/VtWyun4. We plan to add this to the main paper as a potential solution for exp 2. We thank the reviewer for suggesting the analysis that contributed towards this improvement.

EstimatorReferencesError factor50% coverage80% coverage95% coverage
Claude02.31×26.7%66.7%80.0%
Claude11.31×92.5%95.0%97.5%
Claude21.18×95.3%96.9%100%
GPT0197.58×0%0%0%
GPT112.19×19.0%33.3%33.3%
GPT25.93×23.4%42.2%48.4%
Gemini048.26×0%0%0%
Gemini11.17×45.2%85.7%92.9%
Gemini21.12×56.2%92.2%100%

Error factor is a multiplicative absolute-error metric (1× is perfect); coverage is the proportion of tasks whose ground truth is in the estimated interval.

AuthorsOpenReview ↗

Rebuttal to Reviewer dCkq

3. “Is there any relationship between performance across the four experiments? What i mean is that if being good at one type of time tasks make LLM better at any other task?”

Yes, our evaluations are conceptually structured such that the tested capabilities of later experiments rely on competence from earlier ones. In order, they go from the most basic intuitive rules of time (exp 1) to simple quantitative measurements (i.e., estimation, exp 2) that depend on these rules, to biases that influence these measurements (exp 3), and finally to decisions (exp 4, the bread-and-butter of agents) which depend on estimations.

Exp 1 measures models’ knowledge about rules of time, which are a prerequisite to estimation (exp 2, 3). For instance, without knowledge of how skill affects task duration (Axiom 1), models would struggle at estimating how long a person or agent takes on tasks. Similarly, without knowledge about how durations compose (Theorems 2–5), one cannot properly estimate the duration of a complex task that breaks down into subtasks.

Estimation (exp 2, 3) is then a pre-requisite to competent decision making (exp 4). Good decisions rely on having proper estimates of how long tasks will take. We illustrate this through a new ablation to exp 4 (see response to reviewer B5TY point 2, second experiment). TLDR, if we provide models with the exact times for how long they take both using and not using the tool, models become significantly more sensitive to time costs in their decisions than if they had to estimate those durations themselves. The takeaway is that an agent that cannot estimate durations well will make decisions that are not in accordance with their utility profile if they had more complete information.

To help guide future readers, we have added sentences describing the connections between our experiments at the start and end of each experiment section. Thank you for your feedback!

AuthorsOpenReview ↗

Any remaining questions?

Dear Reviewer dCkq,

Thank you again for your kind review. As there are less than 24 hours left in the discussion period, we would like to ask if you have any lingering questions or comments.

Cheers, Authors

Thank you for detailed response. I think it addresses all my concern and i will maintain my positive score

Reviewer_B5TYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
3(+1)
Soundness
2(+1)
Read review
Paper summary

Large language models do not encounter time through true experience. This fact motivates the central question posed by this paper: Do secondhand experiences of time hinder language model reasoning and decisions that involve time? To explore this, the paper proposes a systematic evaluation of langauge model temporal coherence across a number of levels: qualitative understanding (do models understand the rules of itme), quantitative calibration (can models estimate human, agent and their own time usage accurately), and decision-making (when time is a factor can language models make good decisions).

The paper first defines a set of structures that govern the passage of time, presenting five axioms that determine how events can edit temporal state and five principles entailed by the axioms that provide rules governing the temporal impacts of decision making. Then, the paper defines 4 experiments evaluating LLM temporal understanding. Experiment 1 evaluates qualitative understanding by testing how a model understands the impact of singular deviations on time to complete a task, showing general competence. The second experiment evaluates LLMs as estimators of time, showing per-task disambiguation but poor calibration. Experiment 3 compares LLM esitmations to humans, showing predictions are aligned with human means but exhibit less inter-agent prediction variance than human predictions. Experiment 4 evaluates LLM's understanding of the impact of choices with explicitly stated temporal consumption, showing a general disregard of time. The findings of the paper are interesting and timely--even without the explicit strategies humans use to monitor and understand the passage of time (external monitoring, experiential calibration) their base understanding of time is somewhat aligned with human behaviors, though other deficiencies exist (miscalibration, lack of temporal regard).

Critical strengths

The motivation are clear and compelling: LLMs do not inherently possess the skills humans use that are not purely cognitive, and the formalism of the study, while motivated by a substantial body of work, is relatively unique.

The individual experiments are designed rigorously. The proposed axioms and theorems leverage interdisciplinary work across psychology and economics that, while debatable, provide a strong foundation for the study of temporal competencies in large language models, and are adaptable to domains in which the psychology and economic foundations are irrelevant.

Some of the experimental results are compelling. Experiment 1 provides strong evidence suggesting competency in understanding the laws of time while also illuminating some of the decision making tendencies across models (e.g. GPT's break insensitivity, Claude's choice when considering rest vs rust). Experiment 2 shows compelling evidence that models are heavily miscalibrated when given no historical data.

Critical weaknesses

Regarding experiment 2: prior evidence has suggested that large language models struggle to generate accurate confidence intervals without historical calibration; because task duration differences align with true differences, the underestimation tendency could presumably be solved by providing the model with a few calibrated examples.

Regarding experiment 4: while the result show that models almost always believe tool use will improve success and that language models leverage these tools even when extreme wait times will be incurred, the prompt design fails to convey explicit consequences of tool use times--a 20 minute wait may be reasonable in long-running agentic tasks; The observed tendency to disregard time may be largely eliminated when temporal consequences are clear.

The paper reads as 4 loosely related experiments combined into one paper: the axioms and theorems presented in experiment 1 seem irrelevant to further experiments; experiments 2 and 3 explore similar factors but leverage different datasets with different output formats that confound the source of error; experiment 4 provides time as an observation in two additional unrelated scenarios; The paper's stated goal of evaluating temporal competence and providing clear failure attribution is not achieved with the disjoint experimental design. Aligning each of the four experiments with similar task design/ouptut format would require a significant reworking of the paper.

Lastly, minor changes to writing structure and clarity would improve readability; some examples: naming 3 components of state but only labeling 2; stating 4 levels of exploration but only clearly separating 3.

Questions for authors
  1. Can you provide an ablation for experiment 2 showing how behavior changes when few-shot estimates of time are leveraged for calibration?
  2. Can you provide an ablation for experiment 4 providing an understanding or consequences of time to show true disregard?
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer B5TY

Dear Reviewer B5TY,

Thank you so much for your careful review. Each of your points were very insightful, and we do our best to address them below.

1. “Can you provide an ablation for experiment 2 showing how behavior changes when few-shot estimates of time are leveraged for calibration?”

This is a great point. We run a new experiment on how models’ estimates change with calibrated examples. For each task, we provide ground truth runtimes for 1–2 tasks from the same task category, along with their instructions and environment. We do this for all 42 same-category task pairs and all 64 same-category triplets, resulting in 106 few-shot estimation problems. We repeat each estimation 10 times and average across estimates per task.

Our results match the reviewer’s intuition: few-shot calibration substantially improves estimates, particularly for Claude and Gemini, but GPT continues to greatly underestimate times. We are happy to add this to the main paper as a potential solution for exp 2. See table below, as well as figure: https://imgur.com/VtWyun4.

EstimatorReferencesError factor50% coverage80% coverage95% coverage
Claude02.31×26.7%66.7%80.0%
Claude11.31×92.5%95.0%97.5%
Claude21.18×95.3%96.9%100%
GPT0197.58×0%0%0%
GPT112.19×19.0%33.3%33.3%
GPT25.93×23.4%42.2%48.4%
Gemini048.26×0%0%0%
Gemini11.17×45.2%85.7%92.9%
Gemini21.12×56.2%92.2%100%

Error factor is a multiplicative absolute-error metric (1× is perfect); coverage is the proportion of tasks whose ground truth is in the estimated interval.

 

2. “a 20 minute wait may be reasonable in long-running agentic tasks… Can you provide an ablation for experiment 4 providing an understanding or consequences of time to show true disregard?”

We approach the reviewer’s question in two ways. We agree that a 20 min wait is not necessarily problematic. To truly test insensitivity for GPT-5.6 Sol, we first run additional time delays t=1 hour, 1 day, 1 week, and 1 month, which are normatively much less acceptable. We find that GPT continues to be insensitive to time even when wait times are extreme, representing a value profile that is much less defensible (see table).

ConditionNo delay1 s5 s10 s20 s30 s3 min20 min1 hr1 day1 wk1 moSlope, 7 delaysSlope, 11 delays
Cubics, default15.313.315.012.712.713.012.714.714.711.711.79.7+0.1 ± 0.6−0.4 ± 0.3
Cubics, none61.051.755.751.350.353.754.353.755.355.751.749.7+0.5 ± 0.8−0.1 ± 0.4
FreshQA, default97.796.094.794.794.394.093.396.395.792.794.792.30.0 ± 0.3−0.3 ± 0.3
FreshQA, none100.0100.099.799.398.799.3100.099.799.799.399.799.70.0 ± 0.10.0 ± 0.1

GPT-5.6 Sol tool-use rate (%) by stated time cost. Each cell has 300 trials. Slopes are percentage points per 10× increase in time, with task fixed effects and task-clustered std errors.

Second, we run a new experiment that further localizes the insensitivity: in the previous design, models still needed to estimate how long they would take without the tool, thus being subject to the estimation issues identified in exp 2. To do this, we instead provide the total time the model previously took both using and not using the tool.

We find that in this condition, models become more sensitive to time. See figure for full results: https://imgur.com/a/AVPOXjo. GPT changes from nearly flat lines to negative slopes between −1.9% and −13.5% per 10× increase in time. On search with reasoning, GPT’s tool use at t = 20 min falls from 96% to 49%. Both Gemini and Claude’s slopes roughly double (e.g., on search with reasoning, Gemini −10.5% → −22.0%; Claude −2.4% → −5.3%). Notably, this means that when provided with more complete information, models’ decision profiles change. This means that poor estimations are a likely culprit for why models can be insensitive, and providing exact durations greatly help models reason about the consequences of spending time.

We believe both these results strengthen our paper’s claims, and will definitely include them in the final version. We thank the reviewer for pushing us to explore along this dimension.

AuthorsOpenReview ↗

Rebuttal to Reviewer B5TY

3. “The paper reads as 4 loosely related experiments combined into one paper... Aligning each of the four experiments with similar task design/ouptut format would require a significant reworking of the paper”

We thank the reviewer for their candid feedback, and would like to clarify that our experiments are structurally tied together in the concepts that they measure, rather than in domain or scenario. Specifically, they go from the most basic intuitive rules of time (exp 1) to simple quantitative measurements (i.e., estimation, exp 2) that depend on these rules, to biases that act upon these measurements (exp 2 & 3), and finally to decisions (the bread-and-butter of agents, exp 4) which depend on estimations.

Exp 1 measures models’ knowledge about rules of time, which are a prerequisite to estimation (exp 2, 3). For instance, without knowledge of how skill affects task duration (Axiom 1), models would struggle at estimating how long a person or agent takes on tasks. Similarly, without knowledge about how durations compose (Theorems 2–5), one cannot properly estimate the duration of a complex task that breaks down into subtasks.

Estimation (exp 2, 3) is then a pre-requisite to competent decision making (exp 4). As we illustrate in our second experiment in the previous point, an agent that cannot estimate durations well will make decisions that are not in accordance with their utility profile compared to if they had more complete information.

We have added sentences describing these connections at the start and end of each section to better guide the reader through why these experiments are connected together. We would like to acknowledge the reviewer for pushing us to deeper analyze exp 4, which yielded a much nicer experiment localizing models’ insensitivity to mis-estimation and providing a stronger connection to exp 2, making our paper more cohesive in nature.

One further experiment we would like to run before the camera ready deadline—if accepted—is how wording bias (exp 3) interacts with decision-making (exp 4). Given that models’ decisions depend on their estimates, we will next test how their decisions change when a task they need to estimate is phrased in a way that magnifies its duration. This would provide an additional connection between experiments to make our paper even more cohesive.

Finally, we want to note that experimental domains were not chosen haphazardly; Exp 1 and exp 3 match literature and human subjects experiments respectively, with exp 3 limited to written durations (most time estimation human studies use videos or sound). Exp 2 used METR tasks that match today’s agent paradigm, and we also added a new domain where models estimate response times to everyday user queries (see response to Reviewer dCkq, point 1). Lastly, exp 4 needed a common event that 1) the model has control over, 2) meaningfully changes accuracy, and 3) has arbitrary length (in order to measure the trade-off at different points). Tool use is one of the very few candidates that fits this perfectly. We needed to find domains where tools are useful but not required to solve a task, among which the most common were math problems and search queries.

 

4. “minor changes to writing”

We have edited the writing to state 4 levels of exploration clearly in the intro (https://imgur.com/a/iEQZCYS), and also label all three components of state (https://imgur.com/a/JZ7ayse). Thank you for this feedback on writing, it is much appreciated.

Thank you for the additional work and revisions. The new ablations have answered my concern regarding few shot calibration, and the expansions to experiment 4 help clarify the connection between time estimation and decision making.

However, I believe a limitation remains regarding Experiment 4 and the consequences of time, which was my original concern. Within the prompt, no context or directive is provided regarding the consequences of waiting, and thus inflating the delay to 1 month tests sensitivity to duration rather than true temporal cost, and in many long-running systems, waiting long periods of time for a tool to complete accurately can be seen as optimal behavior. Nevertheless, I believe this can be explored in future work and noted as a limitation in the final text.

As the revisions have addressed the bulk of my concerns, I will raise my presentation and informativeness scores to 3 and soundness to 2, and my overall rating to 3.

AuthorsOpenReview ↗

Response to Reviewer B5TY

Dear Reviewer B5TY,

Thank you so much for your continued engagement and for increasing your scores! We are glad to hear that our new ablations have addressed the bulk of your concerns.

We agree that exp 4 doesn't explicitly quantify the consequences of time spent. This is partially intentional, as we want to measure the value that LLMs ascribe to time spent---rather than the value ascribed to the associated consequences. However, we recognize that this value is greatly sensitive to contextual details such as user preferences, so measuring the default value that LLMs ascribe to time is only the first step. We are happy to add this limitation to the discussion of the paper, and explore how LLMs' values of time change with respect to more defined consequences in future work. Thank you for this piece of feedback!

Sincerely, Authors

Reviewer_NoD4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether large language models (LLMs) can account for time costs when performing agentic tasks, given that time information is explicitly marked within the text. This research motivation is both significant and timely for the deployment of autonomous agents; furthermore, the use of real-world agent execution data—provided by METR—ensures high relevance to actual, practical scenarios.

Critical strengths

This paper is highly significant for LLM research labs; for instance, when climbing evaluation benchmarks like TerminalBench, time is a highly sensitive factor. Furthermore, achieving a Pareto-optimal balance between performance and time cost is of critical importance.

Critical weaknesses
  1. The paper does not define the extent to which users value time; consequently, there is no benchmark for an "optimal choice" against which others can be compared. Given this, the decision to "disregard a 1200-fold increase in waiting time" is not necessarily incorrect—a user might prefer to wait 20 minutes to receive a correct answer. Furthermore, users may enter a "sleep mode" of sorts; for instance, they might be insensitive to delays in certain tasks, choosing instead to sleep or switch to other activities. Admittedly, for routine Q&A tasks, users are likely to be more time-sensitive.

  2. With accuracy rates as high as 91.9%–95.8%—and reaching 100% in many instances—there is very little room for these metrics to differentiate between various models. I wonder whether the authors have investigated whether these metrics (or the proxies used for evaluation) have reached a saturation point. The axioms proposed in the paper—such as learning effects, fatigue, recovery, and task switching—describe how humans perform tasks, which differs significantly from how AI agents actually consume time. For example, agents do not experience fatigue, and they learn extremely rapidly; provided they have the necessary context, they can complete a task in a single "prefill" pass. It is worth questioning whether the simulation approach employed in the paper can accurately model these processes.

  3. The abstract somewhat exaggerates certain research findings. For instance, the claim that "certain human biases are amplified by a factor of 1.3" refers to a metric measuring how sensitive a model’s judgments are to specific linguistic features (compared to human sensitivity). However, some of these features—such as predicate types—may inherently contain genuine information regarding duration. Therefore, higher sensitivity does not necessarily equate to a stronger "bias."

Questions for authors

please refer to the weakness section

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer NoD4

Dear Reviewer NoD4,

Thank you so much for your careful review. To respond to your questions:

1a. “ignoring a 1,200-fold increase in wait time is not necessarily wrong”

We agree that waiting 20 min for better accuracy is not necessarily wrong—users may have this preference. To truly test insensitivity for GPT-5.6 Sol, we run additional time delays t=1 hour, 1 day, 1 week, and 1 month, which are normatively much less acceptable for a user to wait. GPT continues to be insensitive even at these extreme waits, representing a much less defensible value profile. We will add this to our main results. We believe this finding strengthens our paper’s claims, and thank the reviewer for pushing us to examine it closely.

Condition1 s30 s3 min20 min1 hr1 day1 wk1 moSlope, previousSlope, new
Cubics, default13.313.012.714.714.711.711.79.70.1 ± 0.6−0.4 ± 0.3
Cubics, none51.753.754.353.755.355.751.749.70.5 ± 0.8−0.1 ± 0.4
FreshQA, default96.094.093.396.395.792.794.792.30.0 ± 0.3−0.3 ± 0.3
FreshQA, none100.099.3100.099.799.799.399.799.70.0 ± 0.10.0 ± 0.1

GPT-5.6 Sol tool-use rate (%) by stated time cost. Each cell has 300 trials. Slopes are percentage points per 10× increase in time, with task fixed effects and task-clustered std errors.

 

1b. “The delay is only stated in the prompt; the model never actually waits”

As LLMs are turn-based, they cannot distinguish the duration between two turns without scaffolds. Prior to testing, we confirmed that they were unable to do so through the API. Thus, our experiment setting is identical to one where the model actually waits. Furthermore, actually waiting in the 20 min setting alone would total to over 30 days in wall-clock time.

 

1c. “Real agents face time costs over multi-turn loops“

Great point. To measure how models adjust their choices in response to past decisions, we run an additional experiment where models 5 decisions in the same context, one after another. Including tool use, this equals up to 10 LLM turns. We test GPT and Claude on 20 randomly sampled 5-question trajectories per condition. We only test the search domain due to cost.

We find that tool use does not change as time costs add up across turns. Search rates on the fifth question are nearly identical to the original single-turn rates on the same questions, with no significant trend from the first to the fifth question. This suggests that our results generalize to multi-turn settings, and we will add this to the main paper.

ModelTime costQ1Q2Q3Q4Q5
GPT-5.6 Sol, default1 s95 / 94.0100 / 96.5100 / 95.0100 / 96.5100 / 100
30 s95 / 91.0100 / 93.0100 / 91.0100 / 96.0100 / 100
20 min95 / 95.5100 / 95.5100 / 97.5100 / 96.5100 / 99.0
GPT-5.6 Sol, none1 s100 / 100100 / 10090 / 100100 / 10095 / 100
30 s100 / 99.5100 / 99.090 / 100100 / 99.5100 / 99.0
20 min100 / 100100 / 99.590 / 100100 / 100100 / 99.0
Claude Opus 5, default1 s95 / 95.090 / 95.095 / 10095 / 95.095 / 100
30 s95 / 95.090 / 94.595 / 10095 / 95.090 / 99.0
20 min90 / 89.580 / 88.590 / 90.595 / 94.090 / 90.0
Claude Opus 5, none1 s100 / 100100 / 100100 / 100100 / 100100 / 100
30 s95 / 95.0100 / 95.0100 / 10095 / 95.0100 / 100
20 min95 / 95.090 / 95.0100 / 10095 / 95.0100 / 100

Search rate (%) at stated time costs of 1 s and 20 mins. Each cell lists multi-turn rates / original rates.

AuthorsOpenReview ↗

Rebuttal to Reviewer NoD4

2a. “For exp 1, accuracy is 91.9–95.8%, so this level offers little room to separate models”.

We clarify that the purpose of exp 1 is to confirm that LLMs understand basic rules of time. This is a pre-requisite for the capabilities tested in exp 2–4: for example, when estimating the duration of a coding task, one must be able to combine the durations of writing and testing separate functions.

Moreover, by comparing these results against the lower accuracies of later experiments, we can identify exactly where models start to go wrong. With exp 1, we safely rule out “knowing the rules of time” as a potential cause for poor estimations, localizing the estimation failures in exp 2 to models not having the correct priors for task durations.

 

2b. “ Axioms and vignettes describe a human doing tasks… Grounding the axioms in software or agent workflows would make Exp 1 more relevant to the paper's agent motivation.”

The axioms are built from a collection of literature that studies humans. There is no guarantee that they would generalize to agents. As a counterexample, consider fatigue: unlike humans, agents do not tire when doing tasks, but instead have context rot, which has a different underlying mechanism. To leverage the existing literature on time, we must focus on human tasks.

However, human task durations are still incredibly relevant to agents. LLMs are increasingly used as co-work partners that would benefit from knowing how long human partners take to complete tasks.

 

3. Features such as predicate type may carry real information about duration, so stronger sensitivity is not necessarily a stronger bias. This also sits uneasily with the finding that models agree with human ratings better than a single annotator does.

We thank the reviewer for their feedback and will soften our claims in the final revision. However, the features we measure, such as whether an event is described as a state or an action, do carry information about how long an event lasts.

When an event is described as a state rather than an action, annotators raise their duration estimate by 1.4 steps on the 11-point scale, while the models raise their duration estimate by about 2.8 steps. To see whether that extra response helps or hurts, we set each model's response to these features back to the human level, left the rest of its judgment of the event unchanged, and compared its answers to the human average again. The adjustment was computed on half the events and applied to the other half.

ModelDistance from human answersOrdering events by length
Claude Opus 50.07 closer0.03 worse
GPT-5.6 Sol0.02 closer0.03 worse
Gemini 3.7 Flash0.10 closer0.02 worse
Qwen2.5-32B0.02 further0.02 worse

Distance is the mean absolute difference from the average human label in scale steps. Ordering is the Spearman correlation with that average.

We find that removing the extra response brings three of the four models closer to the human average, but makes all four worse at ordering events by length. This suggests that the models are treating these features as real signals and weighting them too heavily, and we will describe the effect as an overshoot in magnitude rather than a magnified bias.

We would also like to note that the ten features explain about 16% of the variation in human labels. Most of what separates a long event from a short one is the event itself, where models and annotators agree, and this is what the correlation picks up. A model can therefore track the human average closely while still responding too strongly to wording. We will add this analysis to Appendix C.4.

AuthorsOpenReview ↗

Response to Changed Review

Dear Reviewer NoD4,

We just noticed that you changed your review right before we posted our response. We do our best to map existing responses to the new review, and respond to newly added details:

1a. "the decision to disregard a 1200-fold increase in waiting time is not necessarily incorrect"

This maps to original response 1a cleanly.

1b. "users may enter a "sleep mode" of sorts; for instance, they might be insensitive to delays in certain tasks."

It's true that users may decide at any point to context switch to a different activity (including sleep). However, unless the user tells the model that they are planning to switch, the model should not assume this, and should continue to be sensitive to time costs. Furthermore, our additional experiment in response 1a includes tool durations (1 day, 1 week, 1 month) longer than sleep or another activity, and we find that GPT continues to use the tool at nearly identical rates, representing insensitivity to time even beyond a night's rest.

2a. "With accuracy rates as high as 91.9%–95.8%—and reaching 100% in many instances—there is very little room for these metrics to differentiate between various models."

This maps to original response 2a cleanly.

2b. "I wonder whether the authors have investigated whether these metrics (or the proxies used for evaluation) have reached a saturation point."

As we state in response 2a, the purpose of exp 1 is to confirm that LLMs understand basic rules of time. With respect to this purpose, saturation isn't necessarily a concern. Exp 1 is still valuable because estimation (exp 2 & 3) is a task that depends on said rules; high performance in exp 1 rules out models not understanding the rules of time as a potential cause for the poor estimations in exp 2.

2c. "The axioms proposed in the paper describe how humans perform tasks, which differs significantly from how AI agents actually consume time."

This maps to original response 2b cleanly. We previously considered doing axioms for agents rather than people, but there isn't enough literature on agent task time patterns to synthesize generalizable rules from. Two further concerns impact the feasibility of this approach: 1) the hetergeneous design of agents themselves, where processes (e.g., compressing memory) impact task times but can be arbitrarily frequent, and 2) the jaggedness of agent behavior across different domains of tasks, where agent runtimes may fit certain rules in one domain, but no longer fit those rules in another domain.

2d. "It is worth questioning whether the simulation approach employed in the paper can accurately model these processes."

Our paper does not take a simulation approach; exp 1 builds vignettes to test models' understanding of the rules of time for human processes. We do not claim that the axioms or theorems generalize to agent behavior (they clearly don't), and we don't try to simulate agent runtimes or model any agent processes.

3a. "some of these features—such as predicate types—may inherently contain genuine information regarding duration. Therefore, higher sensitivity does not necessarily equate to a stronger bias."

This maps to original response 3 cleanly.

Please let us know if we can help clarify our experiments further, or if you have any lingering questions that we can respond to before the discussion period ends.

AuthorsOpenReview ↗

Discussion phase ending soon

Dear Reviewer NoD4,

Thank you again for your review. As there are less than 12 hours left in the discussion period, we would like to check in and see if you have any lingering questions or comments.

Sincerely, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.