acceptodds
Under review as a conference paper at ICLR 2027

Correction Has a Window: How Long a False Premise Stays Fixable in Tool-Using Agents

Abstract

A tool-using agent reads a record, the record is wrong, and later the truth arrives. We ask how much the timing of that correction matters. We overwrite one field in a tool result that a -bench retail agent has already read, let the agent act on it for live turns, and then delete the false record and append the true one. Across ten policy families from five vendors (30,942 episodes with a simulated customer), when the false value licenses a refusal, a correction one turn in buys back 0.678 [0.602, 0.754] of the task success the false premise cost and four turns in only 0.254 [0.132, 0.376] (pooled over the nine families the fault damages, on the same units). This decline, the correction window (success corrected at minus ), is positive in all six API-served families, where it pools to +0.287 [+0.217, +0.358] and to +0.294 [+0.210, +0.377] net of the same edit applied to a context with nothing false in it; net of that edit, none is detected in the four open-weight families we served locally. Withholding the same record until the same turn shows that lateness is part of the window but not all of it. Among episodes that reach the late correction the window is smaller under a second simulated customer, though not when every scheduled episode is counted. Late resets of the agent's side do not reliably restore success, and an explicit retraction adds no reliable benefit over silent correction. All code, data and episode traces are in the supplementary material and will be released at https://github.com/xxx/xxx upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tool-using agents, llm agents, false premise, error recovery, context repair, agent reliability, tau-bench

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_y7kgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the timing of correcting a false tool observation affects eventual task success in a retail tool-agent environment. It corrupts a field in a replayed read-only investigation prefix, lets the agent act for a specified number of steps, and then supplies the true record. Controls examine delayed information without an initial false record, transcript-editing effects, explicit retraction, customer simulators, and late transcript/database resets. The principal finding is a substantial early-versus-late correction difference in the API-served families, with no surgery-adjusted window detected in the locally served families.

Critical strengths
  1. The question is practically relevant, and the evaluation measures final task success rather than only the immediate response to corrected information. Distinguishing an initially false observation from information that merely arrives late is particularly useful.

  2. The study includes thoughtful controls for context editing, notice content, customer behavior, and state resets, and reports heterogeneous and negative findings. These controls meaningfully constrain interpretations of the observed timing effect.

Critical weaknesses
  1. The central phenomenon is strongly model-dependent, without a sufficiently explanatory account of that dependence. A substantial correction window is observed in the API-served pool, whereas no surgery-adjusted window is detected in the locally served pool. These groups differ in model properties and serving configurations, so the split does not identify its cause. The heterogeneous results are valuable, but the paper does not yet establish which model or behavioral properties predict a window, or a mechanism that explains its presence and absence. This limits the generalizable insight beyond a careful measurement of conditional effects.

  2. The study's scope and explanatory resolution limit the significance of the contribution. Most evidence comes from a retail setting with a corrupted order-status field and a replayed investigation prefix; the second field family is a small replication. The working open/sealed-plan account organizes the observations but is not decisively distinguished from interaction-history and trajectory-selection explanations. Customer substitutions and late-reset experiments leave these explanations unresolved. Consequently, the study does not yet provide a sufficiently predictive account of recoverability or demonstrate a recovery strategy derived from its findings. A new algorithm is not necessary for an empirical contribution, but stronger mechanism or transfer evidence would make these observations substantially more informative.

Questions for authors
  1. Which measurable model or trajectory properties predict whether a correction window exists? Can matched comparisons separate model behavior from serving configuration and explain the API/local heterogeneity without treating serving mode as a causal factor?

  2. What evidence most clearly distinguishes the proposed plan-commitment account from alternative explanations involving interaction history, termination, and remaining recovery opportunities? Can the resulting account predict recoverability in a held-out task or fault setting, preferably under append-only correction?

1 comment

Thank you again for your review. As the discussion closes on 18 November, we wanted to check whether our response is useful on your two questions. On Q1, the window appears in four held-out models in six configurations, including open weights served locally (+0.173 [+0.075, +0.271] net of the edit), so local serving alone does not remove it; the predictor we froze in advance failed its criterion, and §3.5 now says so rather than claiming predictive power. On Q2, the step cap shows little sign of driving the window, the window holds with every scheduled episode kept, and only resetting the customer restored success after a late correction, which is consistent with part of the loss running through the interaction; we now present the open/committed account as a description these tests constrain, not an established mechanism. If a further analysis on the existing data would help, we can still run it before the deadline. If this addresses your concerns, we would be grateful if you would consider updating your assessment.

Reviewer_GaymOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the scenarios in which an agent with tools reads a wrong result and is then provided with the accurate result. The authors aim to measure how long can a "window" stay open where the correction is able to recover the usefulness. The authors use tau-bench retail as their environment to test 10 models.

In tau-bench retail, there are several tasks where it is required to read an order record so the authors perturb this record with different 3 levels of perturbation intensity. Then, the agent gets 1, 2, 4 turns with the false information before the accurate information is supplied. Finally, the task success is measured. The experimental setup is sound in answering the question motivated by the authors and the findings are novel enough to contribute towards real world design of agent harnesses. The paper also uses several controls to inform their findings. However, I think the paper in its current form is not as readable as I've seen with other ICLR papers and the uneven model task list pushed me towards a weak reject instead of acceptance-leaning scores.

Critical strengths

The problem is real and this is a timely research direction which real world applications in coding agents and in the memory/personalization domain. Outlined related works position this paper well towards answering the question of stale documentation and rapidly evolving information while an agent is running over multiple turns. The paper does a good job of connecting the real world problem to its contributions as listed.

The empirical study in this paper uses 10 models on various tasks (see Q1) giving nearly 31K episodes, showing technical rigor in their experiment. Multiple types of interventions are tested and the findings from it can be applied to real world harness design. The controls used in this paper are well positioned to its central framing such as the "Surgery control" is a good question to ask and the findings form it prevent overconfident conclusions (such as in the case of Qwen3-30B-A3B)

Critical weaknesses

Section 2 explicitly states that "The customer is gpt-4o-mini at temperature 0 in every API-served arm, the gpt-4o-mini policy's included" (lines 90–92), meaning that only one default customer is evaluated in nine of the ten families making it inconsistent with the "working model" as described in P5 "if the seal is in the interaction, a different customer changes the window." Section 2 (lines 215–216). The paper neither evaluates more customers nor has human annotators check the inter-rater rate with the gpt-4o-mini model.

Sentences like "controls and tests, three of which (a placebo, an equivalence test and a floor check) overturned conclusions we would otherwise have drawn (Appendix B)" in the introduction interrupt the clarity, and such can be found multiple times throughout the paper so simplifying the writing and leaving caveats to a limitations paragraph or a good transition sentence in such places will greatly help with clarity and emphasis on the central question the paper is trying to answer.

Overall, there's a lot of hedging throughout the paper showing a lack of confidence in their own methods and results. The writing is very dense, which is not fatal to the paper and I hope the authors commit to improving it. Specifically, the use of unintuitive terms is slightly annoying since there are widely used terms for the same things. For instance, terms like "rung", "ladder", "surgery-adjusted window", "sealed plan", "Survivor window", "Commitment channel"

Questions for authors

Q1: Why are the number of tasks (or "units") uneven across the models? Several conclusions rely on only 3 out of the 10 models because of this unevenness

Q2: The correction range of 1,2,4 is small since most agents encountering this problem are long-horizon so they work with hundreds of tool calls in a session. I understand this constraint comes from tau-bench's retail episode lengths being short but it would be interesting to measure on longer tasks. At the same time I understand that budget is a constraint, so I would like to hear from the authors on why using j = 1,2,4 is enough for their findings.

Q3: I haven't seen a lot of real world agents have interventions inserted by the harnesses, it's more so that new information is retrieved and the model corrects its reasoning. I could have under-explored this so I'd like to ask the authors to include a note with either agent framework documentation or real world examples of harness editing an agent's context on the fly.

2 comments

Thank you again for your review. Before the discussion closes on 18 November, we wanted to check whether the revision addresses your points. On W1, we added a customer simulator from a third vendor (no detectable change in the window: −0.006 [−0.108, +0.096] counting every scheduled episode) and the human check you asked for: three annotators, blind to the condition, rated 60 simulator conversations (Fleiss' κ = 0.45 on contradiction; their majority agrees with the stricter LLM auditor, κ = 0.67 on overall faithfulness; Table 19). On W2–W3, the terms you listed are replaced throughout, caveats are gathered in one Limitations section, and a coverage table (Appendix A.4) shows which families carry each analysis (Q1). If anything still reads as unclear, we would be glad to fix it. If the revision addresses your concerns, we would be grateful if you would consider updating your assessment.

I appreciate your efforts in rewriting it, and the paper's writing is better now.

I am not fully convinced by the explanation for uneven units. That is not something I observe in ICLR papers, and only for that reason I am inclining on not resolving that weakness. Your explanations for Q2 and Q3 makes sense, happy to resolve those two.

I re-read the updated PDF document and there have been a lot of changes. A lot of them do improve readability, as I mentioned, but the paper is still dense with technical details and results. I hoped that the paper would have a better direction than before, and I don't think that is fully resolved yet.

Overall, I am inclined to keep my score of a weak rejection because there's still a lot of hedging in the paper and I'm not fully confident in the methods and experimental setup. I do think the problem is real and relevant to the real world. But this paper can certainly use more time and a more coherent direction to address it.

Reviewer_NxY3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how correction timing affects tool-using agents after they receive an incorrect tool observation. Using τ-bench retail and ten policy families, the authors inject a false record and supply the correct record after different numbers of agent actions. The experiments compare correction, withdrawal, explicit retraction, delayed access to missing information, and several state-reset interventions. In the six API-served families studied, earlier correction recovers more task success. The paper also identifies substantial variation across models and shows how transcript editing and intervention-induced damage affect the interpretation of recovery.

Critical strengths
  • Earlier correction outperforms later correction in all six API-served families studied. The effect persists after accounting for the same transcript edit on a clean prefix and when episodes ending before the scheduled correction are retained. These checks make the result consequential: the observed decline cannot be attributed solely to the editing operation or to excluding early-ending episodes.

  • The paper tests explanations that would otherwise support misleading conclusions. For example, the Qwen3-30B-A3B raw window is not retained after adjustment for the editing control, and the database-reset experiment exposes an apparent benefit caused by damaging clean episodes. These findings strengthen the measurement methodology and provide useful guidance for evaluating agent-repair interventions.

Critical weaknesses
  • The claimed lower bound on the additional cost of falsity is not established by the current argument. Section 3.2 assumes that error messages cause at least as much harm by turn four as by turn one, whereas Appendix A.9 states the opposite condition. The direction matters for the bound. Beyond this inconsistency, the false-record and withheld-record arms differ in service-error feedback, prefix content, and deletion at correction time. The post-treatment comparison between episodes receiving errors and those receiving none does not establish the required assumption. This weakness affects a central contribution: separating the cost of prior false information from the cost of receiving information late.

  • The deployment interpretation extends beyond the interaction setting directly tested. The main experiments use a nine-step prefix assembled from ground-truth entities and presented as the agent’s own previous tool calls. A matched clean control does not establish whether the timing effect depends on this construction. The authors also test append-only correction, but its pooled Hartung–Knapp interval includes zero, and it lacks a dedicated clean-prefix control. The extension to naturally generated trajectories and append-only correction therefore remains unresolved. This matters because the paper draws practical conclusions about correction latency in deployed agents.

Questions for authors

Existing evidence for generalization: Do the existing experiments or supplementary materials contain timing comparisons using prefixes generated by the agent’s own actions, with append-only correction? Such evidence would directly address my concern about the scope of the main result. If these experiments are unavailable, please clarify which conclusions are intended to apply specifically to the constructed-prefix intervention.

1 comment

Thank you again for your review. As the discussion closes on 18 November, we wanted to check whether our response addresses your two concerns. Briefly: (1) the lower bound is withdrawn (§3.3, Appendix C); with the withheld arm's read rule matched and all cells run together at three seeds, the excess is +0.057 [−0.111, +0.225], so we no longer claim it. (2) The experiment your question asked about is now in §3.2: on the agent's own trajectories, with the true record only appended and the identical append on clean trajectories as the control, the window is +0.223 [+0.064, +0.383] and +0.231 [+0.077, +0.385], and it holds with every scheduled episode kept. If any point is still unclear, or you would like a further check on the existing data, we can still provide it before the deadline. If these changes address your concerns, we would be grateful if you would consider updating your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.