- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether coding agents know how much work is left (in turns, time, and tokens) in a multi-turn session when requirements are revealed one by one. The authors use a fork-and-discard probing method on disposable branches to ask the agent remaining effort without messing up the main conversation. They find agents are myopic: the agent always predicts around 2 turns left no matter where it is in the session. Giving the full list of requirements fixes the turn count, but overestimates wall-clock time by 3x and still underestimates tokens by 4x.
Critical strengths
The paper asks an interesting and practical question. Evaluating whether coding agents can estimate their own remaining work and time is timely and useful for real-world user-agent coordination.
The fork-and-discard probing idea is neat to test agent progress without interfering with the live conversation context
Critical weaknesses
-
Narrow and Synthetic Empirical Scope (Only 5 Tasks): The entire paper only tests 5 greenfield synthetic tasks across 100 sessions. This leaves the core conclusion questionable.
-
Over-formalized packaging and no solution or mechanism analysis. The paper uses heavy econometric jargon to wrap simple observations. The paper is like an interesting report, but either a partial solution or some mechnism level explnation are expected so we can understand more about what is going on, rather than presenting one pattern.
Questions for authors
suggestions for future work.
- Can a simple fix be done by a few-shot prompt?
- What is the underlying mechanism behind this failure, and do you have any concrete solution or mitigation method to propose?
2 comments
Thanks for the response and the new few-shot ablation showing that prompting alone does not fix the calibration slope. However, because the evaluation remains limited to 5 greenfield tasks and both mechanism validation and mitigation are deferred to future work, I will maintain my current rating.
We thank the reviewer for the close reading. We respond to each point below.
The tasks are greenfield, as we state in Appendix A, but they are not synthetic. They are drawn from an existing pool of terminal-based coding environments, each shipping a reference solution and a rubric-scored verifier that we keep unmodified. We author only the decomposition into requirements and the staging of those requirements. We will make this explicit in the final version.
On scale, we use five agents, four repetitions each, probed under both information conditions with a five-question battery at every turn, together with seven controls: probe wording, repeated sampling, previous-answer feedback, full tool-history reconstruction, scope framing, reveal policy and simulator strength. While this does not establish universality (and we do not claim it), this is a significant amount of computation spend: 100 sessions carrying 113,156 scored probe answers.
We agree. The calibration slope, the observed-to-expected ratio and the skill score come from forecast evaluation, but the paper does not need the surrounding vocabulary. We can reduce the formalism and state the findings plainly.
It does not. We ran a few-shot prompting arm on one task across three agents, four repetitions each, prefacing every probe with three worked examples of the same question answered correctly on other tasks and changing nothing else. Median per-session calibration slope:
Few-shot prompting raises the level of the estimate, but do not improve temporal awareness without making it track: O:E rises (requirements 0.24 to 0.36) and skill improves while staying negative (-0.58 to -0.35), but the slope stays near zero against the +1.00 that global context reaches on the same sessions. This is one task and three agents, and we can extend it in the final paper.
While we do not claim to answer this in our work, and future work could identify several avenues for improving temporal awareness, we do provide some answers about what mechanisms do not explain the results in the paper. It is not missing information about scope (Section 3.2). It is not capability (Section 3.3). And it is not how well the session itself is going (Section 3.3, Figure 2).
We suspect the harness is a primary inhibitor. Agents have no native access to elapsed time – the harnesses we test expose no timestamps or per-call latency. We will present this important question as future work.