acceptodds
Under review as a conference paper at ICLR 2027

Agentic Myopia: Probing the Temporal Awareness of Agents in Simulated Multi-Turn Coding Sessions

Abstract

Coding agents increasingly operate over long, multi-turn sessions, in which a user and an agent collaborate on a large software task. As these sessions lengthen, agents are delegated increasing autonomy, completing substantial subtasks with limited user oversight. This delegation introduces a coordination cost: a user who intends to plan around, budget for, or intervene in the agent's work requires an accurate estimate of the work that remains. We term an agent's capacity to produce this estimate of remaining work (measured in outstanding turns, wall-clock time, and generated tokens) its temporal awareness. This capacity remains largely unexamined in the multi-turn setting in which coding agents are now deployed, where requirements emerge over the course of a session rather than being fixed at its start. We study coding agent temporal awareness in five frontier-hard software environments, in which a simulated user reveals task instruction to a coding agent incrementally. Using a carefully-designed fork-and-discard probing protocol to estimate five quantities of remaining work in a session, we find that frontier agents largely lack temporal awareness. Their answers are myopic: estimates of remaining work barely move as a session runs down. Given the full list of requirements still to come, agents size the remaining turns almost exactly, with the ratio of predicted to actual rising from 0.27 to 0.97. Curiously, the same list pushes wall-clock estimates to three times the truth. Token estimates climb only from 3% of the truth to about a quarter of it, so the two costs of the remaining work err in opposite directions. An agent that knows how much is left can still misjudge what it will cost, and coding agents cannot yet be trusted to budget the rest of a session.

Primary area:
foundation or frontier models, including LLMs
Keywords:
user simulation, coding agents, LLM agents, introspective probing, multi-turn agent evaluation, calibration and overconfidence, interactive coding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hqv8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether coding agents know how much work is left (in turns, time, and tokens) in a multi-turn session when requirements are revealed one by one. The authors use a fork-and-discard probing method on disposable branches to ask the agent remaining effort without messing up the main conversation. They find agents are myopic: the agent always predicts around 2 turns left no matter where it is in the session. Giving the full list of requirements fixes the turn count, but overestimates wall-clock time by 3x and still underestimates tokens by 4x.

Critical strengths

The paper asks an interesting and practical question. Evaluating whether coding agents can estimate their own remaining work and time is timely and useful for real-world user-agent coordination.

The fork-and-discard probing idea is neat to test agent progress without interfering with the live conversation context

Critical weaknesses
  1. Narrow and Synthetic Empirical Scope (Only 5 Tasks): The entire paper only tests 5 greenfield synthetic tasks across 100 sessions. This leaves the core conclusion questionable.

  2. Over-formalized packaging and no solution or mechanism analysis. The paper uses heavy econometric jargon to wrap simple observations. The paper is like an interesting report, but either a partial solution or some mechnism level explnation are expected so we can understand more about what is going on, rather than presenting one pattern.

Questions for authors

suggestions for future work.

  1. Can a simple fix be done by a few-shot prompt?
  2. What is the underlying mechanism behind this failure, and do you have any concrete solution or mitigation method to propose?
2 comments

We thank the reviewer for the close reading. We respond to each point below.

Concern 1: Are five synthetic tasks sufficient?

The tasks are greenfield, as we state in Appendix A, but they are not synthetic. They are drawn from an existing pool of terminal-based coding environments, each shipping a reference solution and a rubric-scored verifier that we keep unmodified. We author only the decomposition into requirements and the staging of those requirements. We will make this explicit in the final version.

On scale, we use five agents, four repetitions each, probed under both information conditions with a five-question battery at every turn, together with seven controls: probe wording, repeated sampling, previous-answer feedback, full tool-history reconstruction, scope framing, reveal policy and simulator strength. While this does not establish universality (and we do not claim it), this is a significant amount of computation spend: 100 sessions carrying 113,156 scored probe answers.

Concern 2: Is the presentation overly formalized?

We agree. The calibration slope, the observed-to-expected ratio and the skill score come from forecast evaluation, but the paper does not need the surrounding vocabulary. We can reduce the formalism and state the findings plainly.

Question 1: Could few-shot prompting fix the problem?

It does not. We ran a few-shot prompting arm on one task across three agents, four repetitions each, prefacing every probe with three worked examples of the same question answered correctly on other tasks and changing nothing else. Median per-session calibration slope:

MeasureLocalFew-shot (k=3)Global
Requirements+0.01+0.02+1.00
User turns+0.05+0.13+0.94
Agent turns+0.02+0.06+0.94
Wall clock+0.09+0.01+4.34

Few-shot prompting raises the level of the estimate, but do not improve temporal awareness without making it track: O:E rises (requirements 0.24 to 0.36) and skill improves while staying negative (-0.58 to -0.35), but the slope stays near zero against the +1.00 that global context reaches on the same sessions. This is one task and three agents, and we can extend it in the final paper.

Question 2: What mechanism explains the result, and what mitigation is supported?

While we do not claim to answer this in our work, and future work could identify several avenues for improving temporal awareness, we do provide some answers about what mechanisms do not explain the results in the paper. It is not missing information about scope (Section 3.2). It is not capability (Section 3.3). And it is not how well the session itself is going (Section 3.3, Figure 2).

We suspect the harness is a primary inhibitor. Agents have no native access to elapsed time – the harnesses we test expose no timestamps or per-call latency. We will present this important question as future work.

Thanks for the response and the new few-shot ablation showing that prompting alone does not fix the calibration slope. However, because the evaluation remains limited to 5 greenfield tasks and both mechanism validation and mitigation are deferred to future work, I will maintain my current rating.

Reviewer_9fUCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether coding agents can estimate the work remaining in multi-turn sessions where a simulated user reveals requirements incrementally. Five agent configurations complete five coding tasks with four repetitions each, producing 100 main sessions. A fork-and-discard protocol reconstructs historical contexts and elicits predictions without altering the original trajectories. The authors compare local context with privileged access to the complete requirement list and current position. They report nearly constant local forecasts, substantial improvements in turn-count estimates with global context, and persistent errors in time and token estimates. Additional experiments examine prompt wording, previous predictions, tool-history reconstruction, reveal policies, and simulator strength.

Critical strengths
  1. Controlled, non-interfering comparisons. Probing the same saved histories under different information conditions avoids changing the retained execution trajectory. This provides a useful design for studying how access to future requirements changes reported estimates.

  2. Substantial checks on the measurement procedure. The paper examines wording, resampling, previous-answer feedback, full tool histories, and simulator strength. The batched-reveal experiment is particularly informative because it exposes the dependence of accurate turn counting on the reveal policy.

Critical weaknesses
  1. Cost measurements need explicit definitions. Specify which visible/reasoning tokens, participants, and timing components are counted, and what timing information the off-harness probes receive. Matched accounting and an observed-rate baseline are needed to interpret the headline cost biases.

  2. The capability-independence analysis is insufficient. Figure 2 correlates reward with signed log prediction/truth ratios. Near-zero correlation with directional bias does not establish independence from error magnitude; absolute-error and within-measure analyses are needed.

  3. Stopping and exclusion accounting remain unclear. Sessions can stop at (2n) turns, while probes ask about user satisfaction; 58% of attempts are also rerolled. Report termination reasons, probe coverage, and analysis inclusion, and check results on fully settled sessions.

  4. The methodological novelty needs clearer positioning. BAGEN already re-queries logged prefixes for remaining-budget prediction without altering rollouts. The paper should clarify what fork-and-discard adds and distinguish its contribution through hidden incremental requirements and the information intervention.

Questions for authors

See Critical Weaknesses above.

4 comments

I would appreciate it if the authors could provide a response to my comments, particularly the last three questions, as their clarification will be important for my final evaluation.

Thank you for the comprehensive review and for recognizing the importance and practicality of the work. While we have been motivated by the economic and psychometric fields in our user simulation work, we agree we can improve the clarity of the paper.

  1. Cost definitions, visible timing information, and observed-rate baselines.

We agree and will state these in §2.5. The definitions are

Tokens: tokens the session still has to generate: the agent’s output tokens plus the simulator’s completion tokens. Input tokens are excluded.

Wall clock: the summed duration of the remaining agent turns, including tool execution.

Probe context: probes see only the rebuilt message history (App. B).

With the full list, agents count remaining turns almost exactly (O:E 0.97) but put wall clock at 2.99× the truth and tokens at 0.23×. The myopic conclusion is based on the slope, which is near zero for both cost measures (+0.11 and +0.00). Because the estimates barely change over a session, the slope remains near zero regardless of which tokens or timing components are counted.

  1. Does Figure 2 establish independence from capability?

We reran Fig. 2 separately for each measure, using the absolute error rather than the signed one. Error size is not correlated with reward on any measure except agent turns, where higher-reward sessions err more, not less (r = +0.26). At the agent level, qwen3.5-27b has the lowest reward but is the only agent that matches the baseline on wall clock (Table 2). Higher-reward sessions do track progress slightly better without the list (r ≤ 0.27), but the gap closes once the list is given.

  1. Termination, exclusions, coverage, and fully settled sessions.

Of the 100 sessions, 93 met every requirement, and 7 reached the 2n cap (6 Claude-Opus-5, 1 Claude-Sonnet-4.5); none ended in an error. Truth is read from where each session ended. Attempts that failed containment or ended in an API error were re-rolled, not analyzed (App. E). Answers that are not a number are dropped (App. C.2), leaving 1,412–1,477 usable answers per measure, and every session is probed both with and without the list. The cap ends only 7 sessions early, which shortens their true remaining turns. This can only raise O:E, so the reported underprediction (0.27) is conservative.

  1. What does the method add beyond BAGEN?

Re-querying a logged prefix is not new in itself. Our novelty comes from:

  • An exact referent while requirements are still hidden. BAGEN fixes the task up front.
  • Paired information interventions on one frozen state. The list moves the turn O:E (observed-to-expected) ratio from 0.27 to 0.97, while framing alone reaches only 0.51 (§3.3). Missing information explains the count but not the cost.
  • A within-session slope that tests tracking, not just level: +0.02 locally against +0.85 with the list.

We sincerely thank the reviewer for following up. We have posted a clarifying response to your comments and questions and look forward to any further feedback.

Thank you for the response and additional analyses.

However, the analysis restricted to fully settled sessions is not reported, while the potential selection effects of the substantial reroll rate remain unclear. Although the contribution is now more clearly positioned, I still find its novelty relative to prior work limited.

I therefore maintain my rating of 2.

Reviewer_jhrcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether coding agents can estimate how much work remains in a multi-turn session where a simulated user reveals requirements one at a time. It converts five coding tasks into incremental sessions and introduces a fork-and-discard protocol. The protocol asks an agent for its estimate of remaining work (user turns, agent turns, wall-clock minutes, tokens) on a discarded branch of a rebuilt context, so the question doesn't affect the session. Five agents are evaluated.

  • Local context (session only): estimates barely move as the session progresses.
  • Global context (the full requirement list plus the current position): turn estimates become accurate, while wall-clock estimates over-shoot (~3×) and token estimates under-shoot (~0.23×).
Critical strengths
  1. An important and under-studied capability. As agents work over longer horizons and collaborate inside organisations, users need running estimates of remaining work. The paper clearly identifies two obstacles to measuring this: having no referent to compare against, and the probe interfering with the session.
  2. Probing that allows controlled variation. Fork-and-discard lets the same point in a session be probed repeatedly, varying both the question and the information available. This makes paired comparisons such as local vs. global context possible.
Critical weaknesses
  1. The setup doesn't separate temporal awareness from the impossibility of predicting unrevealed scope.
    • The agent has little basis for predicting. Under local context it has no way to know how much work the user will still assign, except from priors. Under sequential reveal, each settled requirement is replaced by a new one of similar size, so the agent's view barely changes between probes. Repeating the same estimate (F.3, p.24) is then the consistent answer.
    • The answer format and probe conditions force a flat answer. Agents must give a single number ("COUNT: followed by a single number", p.18), with no way to express confidence or a range. They can't consult tools or the workspace when probed. A flat estimate is therefore reasonable, and a slope near zero doesn't by itself show a lack of awareness.
    • Global context makes counting close to trivial. It supplies the full list and "requirement k of n" (p.19), and the authors acknowledge the turn estimate partly reflects a one-requirement-per-turn convention (p.6).
    • The key prediction isn't shown. No figure shows how well agents predict how much work the user will assign.
  2. The probing protocol and measures are underspecified.
    • The authors' own justification undercuts fork-and-discard. The protocol relies on reconstructing the agent's state, yet the authors say the workspace is "not faithfully reconstructable" (p.24), tool outputs are truncated or absent (p.17), and probes are answered outside the harness (p.9). The probed model is a reader of a lossy transcript, not the agent in its working state.
    • What the probed model can see over time isn't described. This matters especially for wall clock, which is never defined: it's unclear whether it includes simulator time, API or network latency, or retries. The agent should have access to these to estimate time.
    • "Conversation progress" is undefined, though the calibration slope depends on it.
    • Sessions are short. Durations of roughly 7–40 min (Fig. 1) are hard to square with the "next hour" motivation. Why Opus 5 and GPT-5 mini run much longer isn't explained.

Other comments

  • Five tasks is a very small benchmark.
  • Fig. 1: the y-scales differ across panels, which makes them hard to compare.
  • Batched-reveal citation: the result (p.6) is cited to "Appendix D", but it's unclear what in Appendix D it refers to.
  • Novelty of fork-and-discard: replaying and branching from a saved state has precedent, as the authors note (p.8).
  • Related work placement: consider moving it before the methods.
Questions for authors
  1. (Clarification) What exactly can the probed model see, in particular any temporal information (timestamps, elapsed time, latencies)? What does wall clock include, and how is "conversation progress" normalised?
  2. (Clarification) Why is fork-and-discard valid if the agent's state can't be faithfully reconstructed? Can you add a control where the probe has tool and workspace access?
  3. (Clarification) Can agents express a range or confidence instead of a single number, and does that change the "myopia" result?
  4. (Clarification) Why do Opus 5 and GPT-5 mini sessions take much longer?
  5. (Future work) How do estimates change immediately before vs. after a new requirement is revealed? And with users who sometimes add work and sometimes drop it, do estimates move in the expected direction?
  6. (Future work) How would humans estimate remaining work in the same setting?
1 comment

Thank you for the detailed review. We agree that hidden scope changes how flat forecasts should be interpreted. The paper’s strongest evidence comes from comparing forecasts at the same session state while changing the information supplied.

  1. What can probes see? What does wall clock include, and how is progress normalized?

We wish to clarify that the probed model sees the probe question and the complete reconstructed conversation through the current user message, before the agent replies. Lines 883-898 in the Appendix were included in error from an earlier version of the manuscript and will be removed, we apologize for any confusion.

Lines 935-940 highlight the wall clock probing questions asked to the agent. Progress is normalized by dividing each probe's step index by the session's maximum step, so that every session is placed on a common 0 to 1 grid.

  1. Why is fork-and-discard valid without faithful state reconstruction? Can probes access tools and the workspace?

We again highlight that the probed model sees a faithful reconstruction, including tool calls. We did not test interactive workspace access, but the results support comparison of forecasts from reconstructed histories, rather than equivalence to in-harness estimation.

  1. Would ranges or confidence estimates change the result?

Our experiments measure point forecasts, which are relevant for scheduling and budgeting. We do not test interval calibration directly, and we note that BAGEN (Lin et al., 2026) elicits intervals for a related quantity and reports coverage capping at 47% after SFT and RL, so intervals do not by themselves rescue calibration. Adding confidence intervals would add an important dimension to the evaluation, we can add this to the final manuscript.

  1. Why do Opus 5 and GPT-5 mini sessions take longer?

The main difference is time spent at each turn. Median turn counts range from 13 to 16 across agents, while median session durations range from 15.4 to 64.6 minutes. Within the same harness, both longer-running agents make substantially more tool calls per turn: Opus 5 makes 6.5 versus Sonnet 4.5’s 2.7, and GPT-5 mini makes 4.1 versus GPT-5.6-sol’s 1.8. These comparisons associate longer sessions with more tool use within each exchange.

Aggregate wall time per tool call is similar within these pairs, but this measure includes other session overhead and does not isolate tool-execution latency. The evidence therefore supports more within-turn activity as an explanation, without identifying its precise timing components.

  1. What happens before and after requirements are revealed, added, or dropped?

We do not report a paired before-and-after test or scope withdrawals. We do test a related intervention: batching three requirements per turn changes global turn estimates from approximately correct to approximately twofold overprediction. This result is reported in Section 3.2.

  1. How would humans perform?

We have not measured human forecasts and do not make any claim about a human–agent gap and view this as future work.

Does the experiment distinguish temporal awareness from uncertainty about hidden requirements?

Temporal awareness in collaborative coding agents requires agents to understand hidden requirements. To distinguish temporal awareness from uncertainty about hidden requirements, we also evaluate the scenario where no requirements are hidden. We perform this global context conditioning which supplies the full requirement list and the current position, removing the hidden scope. Even in this scenario agents perform poorly at wall clock estimation. The cost results therefore do not depend on scope being hidden.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.