- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how the timing of correcting a false tool observation affects eventual task success in a retail tool-agent environment. It corrupts a field in a replayed read-only investigation prefix, lets the agent act for a specified number of steps, and then supplies the true record. Controls examine delayed information without an initial false record, transcript-editing effects, explicit retraction, customer simulators, and late transcript/database resets. The principal finding is a substantial early-versus-late correction difference in the API-served families, with no surgery-adjusted window detected in the locally served families.
Critical strengths
-
The question is practically relevant, and the evaluation measures final task success rather than only the immediate response to corrected information. Distinguishing an initially false observation from information that merely arrives late is particularly useful.
-
The study includes thoughtful controls for context editing, notice content, customer behavior, and state resets, and reports heterogeneous and negative findings. These controls meaningfully constrain interpretations of the observed timing effect.
Critical weaknesses
-
The central phenomenon is strongly model-dependent, without a sufficiently explanatory account of that dependence. A substantial correction window is observed in the API-served pool, whereas no surgery-adjusted window is detected in the locally served pool. These groups differ in model properties and serving configurations, so the split does not identify its cause. The heterogeneous results are valuable, but the paper does not yet establish which model or behavioral properties predict a window, or a mechanism that explains its presence and absence. This limits the generalizable insight beyond a careful measurement of conditional effects.
-
The study's scope and explanatory resolution limit the significance of the contribution. Most evidence comes from a retail setting with a corrupted order-status field and a replayed investigation prefix; the second field family is a small replication. The working open/sealed-plan account organizes the observations but is not decisively distinguished from interaction-history and trajectory-selection explanations. Customer substitutions and late-reset experiments leave these explanations unresolved. Consequently, the study does not yet provide a sufficiently predictive account of recoverability or demonstrate a recovery strategy derived from its findings. A new algorithm is not necessary for an empirical contribution, but stronger mechanism or transfer evidence would make these observations substantially more informative.
Questions for authors
-
Which measurable model or trajectory properties predict whether a correction window exists? Can matched comparisons separate model behavior from serving configuration and explain the API/local heterogeneity without treating serving mode as a causal factor?
-
What evidence most clearly distinguishes the proposed plan-commitment account from alternative explanations involving interaction history, termination, and remaining recovery opportunities? Can the resulting account predict recoverability in a held-out task or fault setting, preferably under append-only correction?
Thank you again for your review. As the discussion closes on 18 November, we wanted to check whether our response is useful on your two questions. On Q1, the window appears in four held-out models in six configurations, including open weights served locally (+0.173 [+0.075, +0.271] net of the edit), so local serving alone does not remove it; the predictor we froze in advance failed its criterion, and §3.5 now says so rather than claiming predictive power. On Q2, the step cap shows little sign of driving the window, the window holds with every scheduled episode kept, and only resetting the customer restored success after a late correction, which is consistent with part of the loss running through the interaction; we now present the open/committed account as a description these tests constrain, not an established mechanism. If a further analysis on the existing data would help, we can still run it before the deadline. If this addresses your concerns, we would be grateful if you would consider updating your assessment.