acceptodds
Under review as a conference paper at ICLR 2027

Beyond Tool Responses: Effect Verification and Residual Recovery under Response–Effect Mismatch

Abstract

Tool-using agents rely on responses that may not faithfully reflect execution effects. A tool may report success without producing the intended effects, or return an ambiguous response after those effects have occurred. We term this discrepancy response-effect mismatch, which creates a dilemma: blind trust can leave required effects missing, while blind retry can duplicate effects. Resolving this mismatch hinges on two capabilities: effect verification and residual recovery. Effect verification uses agent-accessible observations to determine which intended effects have occurred; residual recovery completes missing effects while preserving those already realized, rather than re-executing the entire operation. We propose EGR-Lite, a training-free framework that unifies these capabilities using verification evidence to guide recovery. It controls verification cost by selectively acquiring evidence based on response status and predefined tool properties. We construct an AppWorld-based benchmark that separately controls tool responses and execution effects, combining full-task evaluation with controlled continuations. We evaluate two language models on 147 train/dev and 168 held-out tasks. On held-out tasks under mixed mismatches, EGR-Lite records lower measured unsafe-event rates than the verification baselines, with competitive task success. On controlled partial-execution continuations, EGR-Lite records 40% fewer episodes with duplicate effects than direct retry, with similar aggregate task success. Across the held-out injected conditions, it uses 20–36% fewer additional verification queries than always-verify. Gains vary by condition, revealing a verification cost–coverage trade-off.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tool-Using Agents, Execution Verification, Response–Effect Mismatch, Fault Injection, Stateful Environments

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_thJuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies tool calls where the response doesn't align with what actually happened. This can be ambiguous responses or missing data replies. To fix for it, the authors:

  1. Decide whether to check or not for a given reply. 2, If it checks, read the state back.
  2. If something is missing, just fix that part.
  3. Re-verify.

This per-API knowledge lives in effect contracts which record what each api should change, how to check it, and how to repair it.

Critical strengths
  1. The problem is real, and formulated clearly.
  2. The paper reports raw counts, tables, and admits non-significant results, which is useful.
Critical weaknesses
  1. Under partial execution, in about 4/5 of episodes the faulty call's effects are incomplete.
  2. So in about four fifths of these episodes the effects of the faulty call are still incomplete at the end.

Mainly, the paper does not analyse where the pipeline stalls

Questions for authors
  1. How often does EGR-Lite actually reach a recovery decision under partial execution?
  2. Why is EGR-Lite's EC lowest under Qwen None/Ambiguous in every seed?
  3. What is the exact ambiguous-response template?
5 comments
AuthorsOpenReview ↗

Thank you for your careful review and valuable suggestions. Your feedback has helped us clarify the method and strengthen the experimental analysis. Below, we address each weakness (W) and question (Q), providing additional explanations and experimental results. We hope these responses address your concerns and welcome any further points that need clarification.

Response to Q1: How often EGR-Lite reaches recovery under Partial execution

In the original 244 Partial episodes, EGR-Lite reached a recovery decision in 58 cases (23.77%). Recovery actions were executed in 50 of these cases, and 43 ultimately achieved effect closure, giving an EC rate of 86.0% among episodes in which recovery was executed.

The table gives the number of episodes reaching each stage for both models under Success and Ambiguous feedback, with 61 episodes per group.

Table 1: Recovery under each condition in the original controlled Partial experiment (61 episodes per group; EC checked using a common scoring criterion).

ModelFeedbackMissing effects identifiedRecovery selectedRecovery executedFinal EC after recovery
QwenSuccess19161513
QwenAmbiguous1313119
GLMSuccess20161311
GLMAmbiguous13131110
Total244 episodes65585043

One direct reason for low overall EC is that many episodes never enter targeted recovery of the missing effects. Recovery is selected in only about a quarter of episodes and actually executed in about a fifth. Among episodes with executed recovery, however, 86.0% achieve effect closure. This shows that recovery can effectively complete missing effects in these cases. Improving overall EC also requires more episodes with a feasible recovery path to receive a clear diagnosis and reach recovery.

Response to W1–W2: Explaining and improving EC under Partial execution

One direct reason for low EC is that many episodes never reach recovery. Of the original 192 incomplete episodes, 158 (82.3%) stopped at verification and diagnosis. Tool limitations are an important reason: 147 incomplete episodes (76.6%) involved such limitations, including 126 that stopped at this stage. We also refined verification and recovery: paired reruns improve overall EC from 29.10% to 35.66%, and EC after excluding APIs with these limitations from 66.67% to 86.90%.

1. Where does the pipeline stall?

Table 2: Analysis of the original 192 episodes without effect closure.

StageObservationEpisodes
Verification and diagnosisNo qualified contract enabled31
Verification and diagnosisCore effects checked, other missing effects overlooked44
Verification and diagnosisEffect status remained unresolved83
RecoveryMissing effects identified, but no target recovery executed14
RecoveryRecovery executed, but final effects remained incomplete7
EC scoringCompleted effects incorrectly scored as incomplete13
Total192

Table 2 shows that most incomplete episodes never obtained a complete, clear effect diagnosis and therefore did not enter targeted recovery. Improving overall EC also requires more episodes with a safe recovery path to receive a clear diagnosis and reach recovery.

Tool limitations make it difficult to progress from verification and diagnosis to recovery. API inspection and AppWorld tests identified two limitations:

  • Missing information: an expense notification goes to other participants, but the caller's expense and notification queries cannot confirm receipt. The queries examined cannot establish whether resending is necessary or would create duplicates.
  • No separate recovery action: after a payment request is rejected, the API prevents rejecting it again and provides no action to resend a missing rejection notification.

Agents can use only the provided APIs, without directly modifying the database. Thus, even detecting a Partial failure may not provide the information or actions needed for safe recovery.

We excluded the same APIs with these limitations from both methods and compared the remaining common subset of 84 episodes, using identical, corrected EC scoring.

Table 3: Original recovery trajectories, with and without APIs with these limitations.

Recovery methodFull-set EC ↑EC excluding APIs with these limitations ↑Subset Dup. (episodes) ↓
Direct retry67/244 (27.46%)51/84 (60.71%)13
EGR-Lite65/244 (26.64%)52/84 (61.90%)0

Original EGR-Lite achieves EC of 61.90% on this subset, slightly above direct retry, while avoiding its 13 duplicate-effect episodes. The difference between full-set and subset EC illustrates the importance of the environment's available information and actions.


2. Targeted refinements and evaluation

Our trajectory analysis also identified verification omissions and recovery actions that deviated from the original target. Of the 83 unresolved diagnoses, 24 already had sufficient query evidence to determine the core effects' status. For example, sent-mail and draft queries showed that an email had been sent while its original draft remained, identifying draft cleanup as the remaining effect.

We therefore expanded effect coverage, improved interpretation of existing queries, and instructed replanning to prioritize the original call's missing effects. Recovery still uses agent-accessible evidence and existing tools.

We conducted paired reruns on the original 61 shared prefixes, with 244 episodes per method. Both methods use identical, corrected EC scoring and the same 84-episode API subset.

Table 4: Paired controlled Partial experiment (244 episodes per method).

MethodTS ↑EC ↑EC excluding APIs with these limitations ↑Dup. (episodes) ↓
EGR-Lite (paired rerun)58.61%71/244 (29.10%)56/84 (66.67%)27
Improved method65.98%87/244 (35.66%)73/84 (86.90%)25

Compared with the original method rerun in the same experiment, overall EC improves by 6.56 percentage points, and EC after excluding APIs with these limitations improves by 20.24 percentage points. TS increases from 58.61% to 65.98%, while duplicate-effect episodes decrease from 27 to 25. We will include the analysis and results in the revised paper.

Response to Q2: Why EC is low under Qwen None/Ambiguous, and targeted refinements

Analysis of the original trajectories identifies failure to consistently recover the original call's intended effects after verification as an important reason EGR-Lite has lower EC than the baselines under Qwen None/Ambiguous. We therefore made the recovery goal explicit and conducted a paired evaluation at 125 recovery nodes from the original three seeds. EC increased from 76.00% to 88.00%, and TS from 60.80% to 72.00%, without increasing the total numbers of duplicate-effect episodes or unsafe events.

Under None/Ambiguous, the original call produces no effects, so repeating it can often complete recovery. In the original trajectories, Vanilla subsequently repeated the original call successfully in 239 of its 297 fault-triggered episodes (80.47%). ATV, Always-verify, and EGR-Lite all verify ambiguous responses when the call is verifiable, but differ in how they proceed afterward: the verification baselines provide the results to the model to continue execution, whereas EGR-Lite additionally selects a recovery method. Some EGR replanning still uses the entire task as its goal, without prioritizing the missing effects of the current call. We examined the 40 episodes in the original experiment that entered replanning but did not achieve effect closure, as summarized below.

Table 5: Trajectory analysis of the 40 episodes without effect closure after replanning in the original Qwen None/Ambiguous experiment.

Observation in the trajectoryOriginal tool available?EpisodesProportion
Acted on other objects without completing the original targetYes3177.5%
Modified the original object but did not write the required contentYes615.0%
The original tool was absent from the recovery tool listNo37.5%
Total40100%

In 37 cases (92.5%), the original tool remained available, but the recovery actions did not complete the original target. For example, in a task requiring several friends to be added, the model detected that the first friend had not been added but proceeded to add others, ultimately omitting the first. Thus, even after detecting missing effects, EGR can leave more effects incomplete than baselines that repeat the original call.

We therefore refined the recovery goal to complete the publicly diagnosed missing effects of the original call before continuing the remaining task. The model still selects actions using agent-accessible evidence and existing tool permissions; it can use alternative operations that achieve the same effects, without being required to retry every ambiguous call. To evaluate this change, we continued execution from 125 recovery nodes publicly diagnosed as NONE across the original three seeds. The pre-recovery state, interaction history, diagnostic evidence, available tools, and remaining budget were held fixed, with only the recovery goal wording changed.

Table 6: Paired evaluation at original Qwen None/Ambiguous recovery nodes (125 episodes per method).

MethodEC ↑TS ↑Dup. (episodes) ↓UE (episodes) ↓
EGR-Lite (original recovery goal)95/125 (76.00%)76/125 (60.80%)01
EGR-Lite (explicit recovery goal)110/125 (88.00%)90/125 (72.00%)01

Making the recovery goal explicit improves EC by 12.00 percentage points and TS by 11.20 percentage points, with net gains across all three seeds. A paired bootstrap analysis clustered by task ID gives a 95% confidence interval of [3.88,20.97] percentage points for the EC improvement. These results support translating effect diagnosis into a clear recovery goal for the original call to reduce target drift and improve effect closure. The results above come from paired continuations at original recovery nodes; we will further evaluate this refinement on the full None/Ambiguous benchmark and report the results.

Reviewer_h4pvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the mismatch between tool responses and their actual execution effects. The authors propose EGR-Lite, a training-free framework that verifies what a tool call has actually accomplished and recovers only the missing effects, rather than blindly retrying the whole operation. The authors also build an AppWorld-based benchmark with controlled execution failures and evaluate the method on two LLMs. The results show that EGR-Lite can reduce unsafe events and verification overhead while maintaining comparable task success rates.

Critical strengths
  1. A successful tool response does not always mean the operation was completed, while retrying it may cause duplicate effects. The paper clearly explains this problem and proposes a reasonable way to handle it.
  2. Besides end-to-end task evaluation, the authors use controlled continuations to compare recovery strategies from the same starting state. They also measure duplicate effects and unsafe events rather than relying on task success alone, which makes the evaluation more informative.
Critical weaknesses
  1. Limited novelty and reliance on predefined effect contracts. Postcondition verification, safe retries, and recovery based on execution state have been explored in prior work. EGR-Lite brings these ideas together, but its distinct technical contribution is not entirely clear. The method also relies on predefined effect contracts that specify expected effects, verification procedures, and recovery actions. The paper does not give enough detail about the effort required to build these contracts or how easily the approach would scale to new tools.
  2. The results for residual recovery are not fully convincing. The paper reports a 40% reduction in duplicate-effect episodes compared with direct retry, which is encouraging. However, the paired tests in Appendix K.3 do not show statistically significant differences in any of the four settings. Also, although effect closure improves, the absolute rates remain quite low. This raises questions about how reliable the recovery process is in practice.
Questions for authors
  1. What is the main technical difference between EGR-Lite and existing approaches to tool verification and safe retries? How much effort does it take to build the effect contracts, and how easily can the method be applied to new tools?
  2. The reduction in duplicate effects is not statistically significant in Appendix K.3. Could the authors provide more evidence to support this result? Also, why does effect closure remain relatively low even with EGR-Lite?
5 comments
AuthorsOpenReview ↗

Thank you for your careful review and valuable suggestions. Your feedback has helped us clarify the method and strengthen the experimental analysis. Below, we address each weakness (W) and question (Q), providing additional explanations and experimental results. We hope these responses address your concerns and welcome any further points that need clarification.

Response to W1.1 & Q1.1: Technical differences from prior tool verification and safe-retry methods

Our main contributions are: (1) moving from operation-level retries to effect-level residual recovery; (2) selective effect verification based on tool responses and operation properties; and (3) a response–effect mismatch benchmark that separately evaluates selective verification and residual recovery. The first two are methodological contributions, and the third provides the corresponding evaluation basis.

  1. Moving from operation-level retry decisions to effect-level residual recovery. ToolGate (Liu et al., 2026) uses preconditions and postconditions to constrain tool calls and acceptance of their results. Verified Tool Calls (Mansoor et al., 2026) uses operation-level postconditions to decide whether to retry the original call and employs idempotency keys to reduce duplicate-execution risk. SagaLLM (Chang & Geng, 2025) uses transaction compensation to restore workflow consistency, while AgentRewind (Zhuang et al., 2026) restores the agent context and controlled environment to a selected checkpoint before continuing execution. EGR-Lite diagnoses the individual effects of a call as confirmed, missing, or unresolved, and uses the effect contract to translate the diagnosis into recovery targets and effects to preserve. Realized effects must be preserved; missing effects become recovery targets; unresolved effects are not assumed missing. For example, if a comment exists but its attachment is missing, recovery should attach the file to the original comment rather than post another comment.

  2. Selective effect verification based on tool responses and operation properties. Algorithm 1 of VTC verifies ambiguous responses and directly returns on SUCCESS. EGR-Lite uses both responses and predefined operation properties to decide whether and when to verify, enabling it to check calls that report success despite incomplete effects. It also reuses existing observations and defers some checks, balancing verification coverage against query cost.

  3. A response–effect mismatch benchmark that separately evaluates selective verification and residual recovery. In AppWorld, we independently intervene on tool feedback and realized database changes, covering no execution, partial execution, and full execution. The benchmark supports comparisons of reliability and query cost across verification strategies on complete tasks. It also supports comparisons of recovery methods from the same pre-call state, interaction history, and partial-execution state. Task success, effect closure, and duplicate effects are recorded separately, distinguishing task completion, completion of operation effects, and duplicate writes.

References

  • Chang, E. Y., & Geng, L. (2025). SagaLLM: Context Management, Validation, and Transaction Guarantees for Multi-Agent LLM Planning. Proceedings of the VLDB Endowment, 18(12), 4874–4886. Paper · DOI
  • Liu, Y., Peng, X., Cao, J., Wang, X., Deng, S., Chen, J., Yin, J., & Zhang, X. (2026). ToolGate: Contract-Grounded and Verified Tool Execution for LLMs. Findings of the Association for Computational Linguistics: ACL 2026, 9653–9684. Paper · DOI
  • Mansoor, I. K., Phadke, A., & Rana, P. (2026). Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures. arXiv preprint, arXiv:2608.02645. Paper · DOI
  • Zhuang, Y., Chen, K., Duan, Y., Zheng, S., Li, J., & Zhang, X.-Y. (2026). AgentRewind: Recoverable Execution for Long-Horizon LLM Agents. arXiv preprint, arXiv:2608.14380. Paper · DOI

Response to W1.2 & Q1.2: Contract construction effort and adaptation to new tools

Contract construction primarily uses an automated offline process, with humans mainly designing and refining that process through interaction with Codex. When adding new tools, the core verification, diagnosis, and recovery framework and the contract construction process can be reused; the tool interfaces, effect semantics, and observation and recovery capabilities require adaptation.

Specifically, the process automatically extracts evidence from tool specifications, API documentation, and implementation source code. Codex generates candidate contracts under a common specification, followed by structural checks, semantic review, and controlled execution tests for validation and refinement. Capabilities that pass the checks are frozen and enabled before task execution, and contracts are reused across tasks rather than rewritten for each task. The process is described in Appendix C (Effect Contract Construction). Because human working hours were not systematically recorded, we cannot provide a precise person-hour estimate.

The same process can be applied to new tools: their documentation, source code, and public interfaces establish which effects to verify, how to observe them, and which recovery actions are safe, before integration into the general framework. Tool-specific knowledge therefore needs adaptation, but the overall verification and recovery mechanisms do not need to be redesigned for each tool.

Our current experiments cover multiple applications and tools in AppWorld. We have also begun adapting EGR-Lite to τ²-bench, initially reviewing which APIs are suitable for injecting the studied failures in a way consistent with their actual execution. Given the time constraints, we do not expect to complete the full transfer evaluation during this rebuttal period. We plan to report contract construction effort, human involvement, recovery coverage, and execution results in a subsequent version of the paper.


Response to W2.1 & Q2.1: Statistical support for the reduction in duplicate effects

We added a paired bootstrap analysis with shared task prefixes as the resampling unit. The results support a statistically significant reduction in duplicate effects overall in the original controlled experiment: the duplicate-effect rate decreases from 16.39% (40/244) with direct retry to 9.84% (24/244) with EGR-Lite, an absolute reduction of 6.56 percentage points, with a 95% confidence interval of [1.64, 12.70] percentage points, excluding zero. The corresponding relative reduction is 40%.

This analysis directly addresses the paper's claim of a "40% reduction overall," estimating the average improvement across both models and feedback conditions. The four subgroup tests in Appendix K.3 instead ask whether each condition shows a significant improvement. Directly estimating the overall difference and its confidence interval is better aligned with that overall claim than requiring all four subgroup comparisons to be individually significant.

Specifically, we performed 50,000 paired bootstrap resamples over the 61 shared prefixes. Whenever a prefix is sampled, we retain all outcomes for both methods, both models, and both feedback conditions under that prefix. This preserves both method pairing and the dependence across models and feedback conditions sharing a prefix. The interval therefore reflects the experimental design, rather than treating the 244 episodes as 244 independent tasks.

Response to W2.2 & Q2.2: Why effect closure remains low under partial execution, and targeted refinements

An important reason EC remains low even with EGR-Lite in the original controlled Partial experiment is the environment's and tools' limitations: 147 of the 192 incomplete episodes (76.6%), more than three quarters, involve such limitations. We checked their correspondence to the actual missing effects through episode-level trajectory analysis, API source inspection, and additional tests in AppWorld.

In this AppWorld setting, the agent can use only the provided APIs; it cannot directly modify the database or add tool functionality. There are two main limitations:

  • Insufficient tool information to confirm some effects. Some effects, including certain notifications, cannot be confirmed through the agent-accessible queries we examined. This makes it difficult to determine whether another action is needed or whether it would cause duplicates.
  • No separate action for completing residual effects. After part of an operation has completed, a tool may not support completing only a missing notification, record, or cleanup step. Re-executing the entire operation may duplicate existing effects. For example, once a payment request has been rejected but its rejection notification is missing, the API does not allow rejecting it again and provides no separate public action to resend that notification.

As a supplementary analysis, we excluded the same APIs with these limitations from both EGR-Lite and direct retry and compared them on the remaining common subset of 84 episodes.

Table 1: Original recovery trajectories after excluding APIs with these limitations (84 episodes per method).

Recovery methodEC ↑Dup. (episodes) ↓
Direct retry51/84 (60.71%)13
EGR-Lite52/84 (61.90%)0

EGR-Lite achieves slightly higher EC than direct retry while producing fewer episodes with duplicate effects.

We further analyzed the remaining 45 incomplete episodes. Of these, 32 (71.1%) stopped at verification and diagnosis without entering recovery; inspection of recovery trajectories also identified actions that deviated from the original call's target. We therefore expanded the effects covered by verification, made fuller use of existing query evidence, and explicitly prioritized completing the missing effects of the original call. We then conducted paired reruns of the original and refined methods on the original 61 shared prefixes, with the following results.

Table 2: Complete paired controlled Partial experiment for the original and improved methods (244 episodes per method).

All methods use the same EC scoring, and EC after excluding APIs with these limitations uses the same subset of 84 episodes.

MethodTS ↑ (%)EC ↑EC (excluding APIs with these limitations) ↑Dup. (episodes) ↓
EGR-Lite (paired rerun)58.6171/244 (29.10%)56/84 (66.67%)27
Improved method65.9887/244 (35.66%)73/84 (86.90%)25

Relative to the original method rerun in the same experiment, the improved method increases EC by 6.56 percentage points across all 244 episodes and by 20.24 percentage points to 86.90% on the common subset of 84 episodes excluding APIs with these limitations. These results show that refining effect verification, diagnosis, and recovery targets improves recovery under the available interfaces. Full effect closure remains constrained by the observation and action capabilities provided by the environment.

Reviewer_pPt6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies cases where a tool’s response does not reliably indicate what actually happened. It proposes EGR-Lite, which checks execution effects and recovers only the missing parts. Experiments in AppWorld show fewer duplicate effects than direct retry and fewer verification queries than always checking.The problem is practical and worth studying. The observation that a task can pass its evaluator while still producing duplicate effects is particularly useful. Separating confirmed, missing, and unresolved effects is a sensible design choice. It gives recovery a clearer basis than simply trusting the response or retrying the whole operation. The controlled continuation experiments are well motivated. Starting from the same state and retaining the same verification process makes the recovery comparison informative.

Critical strengths

The problem is practical and worth studying. The observation that a task can pass its evaluator while still producing duplicate effects is particularly useful. Separating confirmed, missing, and unresolved effects is a sensible design choice. It gives recovery a clearer basis than simply trusting the response or retrying the whole operation. The controlled continuation experiments are well motivated. Starting from the same state and retaining the same verification process makes the recovery comparison informative.

Critical weaknesses

My main concern is how much of the improvement comes from the framework versus the detailed, tool-specific contracts. The direct-retry comparison is useful, but a replanning baseline given the same effect evidence and recovery interfaces would better establish the value of the proposed recovery mechanism.

Generalization remains unclear. The held-out tasks use the same AppWorld environment and preconstructed contracts. A small study on a new service, including the effort needed to build its contracts, would strengthen the practical case. Delayed or unreliable verification reads would also be an important test.

Questions for authors

a. How much manual work was needed to construct and validate the contracts, and how many evaluated APIs support residual recovery? b. How does the method distinguish a missing effect from one that has occurred but is not yet visible to verification queries? c. Could you provide an uncertainty estimate for the pooled duplicate reduction, accounting for the shared task prefixes?

4 comments
AuthorsOpenReview ↗

Thank you for your careful review and valuable suggestions. Your feedback has helped us clarify the method and strengthen the experimental analysis. Below, we address each weakness (W) and question (Q), providing additional explanations and experimental results. We hope these responses address your concerns and welcome any further points that need clarification.

Response to W1: Framework contribution and replanning with shared effect evidence

We added an LLM replanning baseline on the controlled Partial subset, giving it the same types of effect evidence and recovery interfaces. The results show that EGR-Lite's benefits are not limited to recovery actions predefined in contracts: the framework can improve task success and reduce unsafe events even when the LLM plans recovery autonomously. In the controlled Partial experiment, LLM replanning and EGR-Lite perform similarly, and both produce fewer episodes with duplicate effects than direct retry. With LLM replanning held fixed, EGR Selective still improves task success and reduces unsafe-event rates.

We hold EGR Selective verification fixed and compare direct retry, LLM replanning, and EGR-Lite's contract-guided recovery. All three share effect contracts, verification and diagnosis mechanisms, and public recovery tools. LLM replanning receives confirmed, missing, and unresolved effects, query results, and the original call arguments, then selects recovery actions autonomously. All experiments below use seed 4090.

First, in the controlled Partial experiment with the original 61 shared prefixes, all three methods continue from the same pre-call state, interaction history, and pending call, facing the same partial-execution failure. Pooling both models and feedback conditions gives 244 episodes per method, with the following results.

Table 1: Controlled Partial recovery with shared effect evidence and recovery interfaces.

Recovery methodTS ↑ (%)Dup. ↓ (%)UE ↓ (%)
Direct retry55.7416.3946.72
LLM replanning54.929.4335.25
EGR-Lite (contract-guided recovery)56.159.8435.25

LLM replanning and EGR-Lite produce duplicate effects in 23/244 and 24/244 episodes, respectively, both below direct retry's 40/244. Both record unsafe events in 86/244 episodes. This shows that, given effect diagnosis and the same public recovery tools, an LLM can autonomously plan targeted recovery while retaining the reduction in duplicate effects.

We also compare these three recovery methods on the 168 official AppWorld Mixed tasks, using the same tasks and seed 4090 for every model and method.

Table 2: Recovery methods on AppWorld Mixed with EGR Selective held fixed.

Recovery methodQwen TS ↑ (%)UE ↓ (%)GLM TS ↑ (%)UE ↓ (%)
Direct retry51.1942.1126.7936.84
LLM replanning50.6032.9830.3632.94
EGR-Lite (contract-guided recovery)52.9839.8029.7634.07

Contract-guided recovery achieves higher TS and lower UE than direct retry on both models, while exhibiting trade-offs relative to LLM replanning.

To further test the framework's contribution when the LLM plans recovery autonomously, we hold LLM replanning fixed and compare ATV with EGR Selective.

Table 3: Verification strategies on AppWorld Mixed with LLM replanning held fixed.

ModelVerification strategyTS ↑ (%)UE ↓ (%)VQ/task ↓
QwenATV48.2148.960.696
QwenEGR Selective50.6032.983.405
GLMATV28.5746.070.631
GLMEGR Selective30.3632.942.583

With the same LLM recovery method, EGR Selective increases TS by 2.38 and 1.79 percentage points and reduces UE by 15.98 and 13.13 percentage points relative to ATV on Qwen and GLM, respectively, at the cost of more verification queries.

EGR-Lite therefore contributes more than a set of tool-specific recovery rules: it provides a unified framework that uses realized-effect evidence to organize verification, diagnosis, and recovery. Effect contracts provide tool effect semantics and available interfaces, allowing the framework to determine which effects are complete and which still require recovery. This framework can improve reliability even when subsequent actions are planned autonomously by the LLM.

Response to W2: Adaptation to new tools and services

EGR-Lite separates tool-specific effect knowledge from the general verification, diagnosis, and recovery process. Adapting to a new service requires specifying its effect semantics, agent-accessible observation interfaces, and recovery capabilities; the core framework and automated construction process can be reused. This provides a basis for extending the method to different tools and tasks without redesigning the agent's verification and recovery mechanisms for each service.

Our current experiments cover multiple applications and tools within AppWorld, but remain within the same evaluation environment. We have begun adapting EGR-Lite to τ²-bench. The initial work focuses on reviewing which tool APIs are suitable for injecting the failures studied here, so that the interventions follow actual execution behavior and simulate plausible failures. Given the time available in this discussion period, we do not expect to complete the full transfer evaluation during rebuttal. We plan to include contract construction effort, human involvement, recovery coverage, and execution results in a subsequent version of the paper to assess transfer across environments.

Overall, EGR-Lite's extensibility comes from separating the general framework from tool-specific knowledge and automating contract construction. This reduces the manual burden of adapting individual tools while allowing tools with different observation and recovery capabilities to share the same effect-guided verification, diagnosis, and recovery mechanisms.

Response to Qa: Manual effort for contract construction and residual recovery coverage

Contract construction primarily uses an automated offline process, with limited human involvement. Regarding recovery coverage, 123 of the 274 evaluated mutation APIs (44.89%) have recovery routes guided by effect evidence that can be enabled under the corresponding conditions.

Human involvement mainly consists of designing and refining the construction process through interaction with Codex. Once established, the process operates largely automatically without further manual intervention. Specifically, it automatically extracts evidence from tool specifications, API documentation, and implementation source code; Codex then generates candidate contracts under a common specification. Structural checks, semantic review, and controlled execution tests assess observation and recovery capabilities and guide revisions. Capabilities that pass these checks are frozen and enabled before task execution. The resulting contracts are reused across tasks, rather than rewritten for each task. The process is described in Appendix C (Effect Contract Construction). Because human working hours were not systematically recorded, we cannot provide a precise person-hour estimate.

Here, residual recovery means using effect diagnosis to complete intended effects that remain unrealized. Under NONE, the residual includes all intended effects; under PARTIAL, recovery must preserve confirmed effects while completing the missing ones. The 123 APIs above use this scope of recovery. EGR-Lite does not require a dedicated deterministic recovery rule for every tool: it can apply suitable contract rules or guide LLM replanning using effect diagnosis and public tool interfaces.


Response to Qb: Distinguishing missing effects from delayed visibility

We distinguish these cases according to whether agent-accessible queries provide sufficient evidence of absence: an effect is labeled MISSING when its absence can be confirmed; if a read fails, results are incomplete, or the target is ambiguous, the effect remains UNRESOLVED rather than being treated as missing. If a query confirms realization, the effect is labeled CONFIRMED.

For example, a Todoist comment query returns at most 20 comments. If the target comment is not found:

  • If 20 comments are returned: the target may be on a later page, so the effect remains UNRESOLVED.
  • If fewer than 20 are returned: provided that the query already reflects the write, its completeness condition supports a MISSING diagnosis.

For UNRESOLVED effects, the framework does not directly assume absence and perform recovery. Instead, it seeks further agent-accessible observations within the query budget. The diagnosis and recovery rules are detailed in Appendices D and E.


Response to Qc: Uncertainty in the overall duplicate reduction

We added a paired bootstrap analysis with shared task prefixes as the resampling unit. Accounting for dependence induced by shared prefixes, EGR-Lite reduces the duplicate-effect rate relative to direct retry by 6.56 percentage points, with a 95% confidence interval of [1.64, 12.70] percentage points. The corresponding relative reduction is 40.0%, with a 95% confidence interval of [12.5%, 65.0%]. The intervals exclude zero, supporting an overall reduction in duplicate effects in the original controlled experiment.

The original experiment includes 61 shared task prefixes, each evaluated under both models and feedback conditions, giving 244 episodes per method. Direct retry produces duplicate effects in 40/244 episodes (16.39%), compared with 24/244 (9.84%) for EGR-Lite.

We resample the 61 prefixes with replacement 50,000 times, keeping the outcomes for both models, both feedback conditions, and both recovery methods together within each sampled prefix. We then calculate the overall reduction and its 95% confidence interval. This preserves method pairing and accounts for dependence across conditions sharing a prefix, rather than treating the 244 episodes as independent samples.

Thank you for your time and thoughtful consideration. We sincerely appreciate your valuable feedback.

Reviewer_3HEKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies response–effect mismatch in tool-using agents: a tool response may indicate success even though the intended effects did not occur, or may be ambiguous after some or all effects have already occurred. Such mismatches can lead agents to leave operations incomplete or repeat effects. The authors propose EGR-Lite, a training-free framework based on preconstructed effect contracts. It selectively verifies intended effects using agent-accessible observations, identifies missing effects, and performs residual recovery while preserving confirmed effects. The authors also construct an AppWorld-based benchmark that independently controls execution effects and tool responses. The reported results show lower measured unsafe-event rates in several settings and fewer duplicate effects in controlled partial-execution continuations, with condition-dependent task success and verification cost.

Critical strengths
  1. A useful effect-level evaluation perspective. The paper shows that a task can be judged successful while an execution leaves duplicate effects. In the controlled recovery comparison, all 40 direct-retry episodes with measured duplicates also satisfy the task evaluator. This is a concrete reason to report side effects alongside task completion.
  2. A controlled comparison of recovery decisions. The partial-execution comparison restores the same pre-call state, history, and pending operation for both strategies. Direct retry retains the verification and diagnosis process up to the recovery choice. This design provides a meaningful comparison between targeting the diagnosed residual and repeating the original operation.
Critical weaknesses
  1. The benefits over existing verification methods and the overall cost remain unclear. Verified Tool Calls already verifies effects before retrying, but the paper does not directly compare against it. It is also unclear whether ATV and Always-verify use the same diagnosis and recovery mechanisms as EGR-Lite, making the benefit of selective verification difficult to isolate. Moreover, the paper reports verification-query counts but not total calls, tokens, latency, or contract costs, so the net cost of the framework remains unknown.
  2. The benchmark’s label validity and real-world generalizability need stronger evidence. In the recovery results, 137 episodes are judged successful, while only 52 achieve effect closure. The authors should clarify whether the missing effects are required by the task and how valid alternative completions are assessed. Partial execution is evaluated on only 61 controlled tasks, without end-to-end partial-failure experiments or sufficient detail on sample selection and the realism of cut points. The reported 40% reduction in duplicates aggregates these tasks across models and conditions; the individual comparisons are not statistically significant. The injected faults also do not establish how frequently these failures occur in practice.
Questions for authors
  1. How does EGR-Lite compare with Verified Tool Calls, and do ATV and Always-verify use the same diagnosis and recovery mechanisms?
  2. How do the authors explain the gap between task success and effect closure, and how representative are the 61 partial-execution tasks of real-world failures?
  3. Do the reliability gains outweigh the framework’s total overhead, including tool calls, token usage, latency, and contract construction?
6 comments
AuthorsOpenReview ↗

Thank you for your careful review and valuable suggestions. Your feedback has helped us clarify the method and strengthen the experimental analysis. Below, we address each weakness (W) and question (Q), providing additional explanations and experimental results. We hope these responses address your concerns and welcome any further points that need clarification.

Response to W1.1 & Q1.1: Differences from and comparison with Verified Tool Calls

We added a comparison based on the verify-before-retry procedure in Verified Tool Calls (VTC): across both models, EGR-Lite achieves higher task success and lower unsafe-event rates with limited additional tool calls and token usage. EGR-Lite extends VTC in two main respects:

  1. Verification timing. Algorithm 1 of VTC directly accepts successful responses and verifies ambiguous responses. EGR-Lite also selectively verifies successful responses when effects may be missing, addressing calls that report success without completing the intended effects.
  2. Recovery granularity. VTC uses operation-level postconditions to decide whether to accept, wait, or retry the original operation with an idempotency key. EGR-Lite distinguishes realized and missing effects individually, preserves confirmed effects, and recovers the missing effects.

We have not found an official implementation publicly released by the VTC authors. We therefore constructed a VTC-inspired baseline on AppWorld Mixed from the paper's description: it verifies only after ambiguous responses and retries the original operation when agent-accessible evidence confirms missing effects. This baseline implements VTC's core idea, but does not include backoff waiting or server-side idempotency keys. Each setting evaluates 168 tasks per model with seed 4090. The table reports task success (TS), unsafe-event rate (UE), tool calls, token usage, and additional verification queries (VQ).

Table 1: EGR-Lite versus the VTC-inspired baseline on AppWorld Mixed.

ModelMethodTS ↑ (%)UE ↓ (%)Tool calls/task ↓Tokens/task ↓ (thousands)VQ/task ↓
QwenVTC-inspired48.8145.7425.01378.500.595
QwenEGR-Lite52.9839.8026.39385.583.452
GLMVTC-inspired26.1946.9924.63350.820.613
GLMEGR-Lite29.7634.0724.70371.653.012

Relative to VTC-inspired, EGR-Lite increases TS by 4.17 and 3.57 percentage points and reduces UE by 5.95 and 12.92 percentage points on Qwen and GLM, respectively. Total tool calls increase by 5.55% and 0.29%, and token usage increases by 1.87% and 5.94%, respectively.

Response to W1.2 & Q1.2: Baseline mechanisms and the contribution of selective verification

In the original submission, ambiguity-triggered verification (ATV), Always-verify, and EGR-Lite share effect verification and diagnosis based on effect contracts and public read-back, but differ in recovery: ATV and Always-verify primarily rely on the large language model (LLM) to continue planning, while EGR-Lite also uses contract-guided recovery. To isolate the contribution of selective verification, we added a 3×3 factorial comparison crossing three verification strategies (ATV, Always-verify, and EGR Selective) with three recovery methods (direct retry, LLM replanning, and contract-guided recovery), allowing verification strategies to be compared with recovery held fixed. Across all three recovery methods, EGR Selective achieves higher TS and lower UE than ATV, and uses fewer verification queries than Always-verify, illustrating the trade-off between reliability and verification cost.

The experiment uses Qwen and GLM on AppWorld Mixed, with 168 tasks per combination and seed 4090. All combinations share effect contracts, diagnosis rules, and tool budgets. The complete results for both models are shown below.

Table 2: Verification strategy × recovery method on AppWorld Mixed with Qwen.

Recovery methodVerification strategyTS ↑ (%)UE ↓ (%)Tool calls/task ↓Tokens/task ↓ (thousands)VQ/task ↓
Direct retryATV48.8145.7425.01378.500.595
Always-verify55.9532.9927.86395.054.607
EGR Selective51.1942.1127.05402.353.363
LLM replanningATV48.2148.9624.58382.570.696
Always-verify52.9836.4627.92440.284.494
EGR Selective50.6032.9826.72401.703.405
Contract-guided recoveryATV50.6045.3623.84352.920.619
Always-verify52.3834.6927.47395.934.458
EGR Selective52.9839.8026.39385.583.452

Table 3: Verification strategy × recovery method on AppWorld Mixed with GLM.

Recovery methodVerification strategyTS ↑ (%)UE ↓ (%)Tool calls/task ↓Tokens/task ↓ (thousands)VQ/task ↓
Direct retryATV26.1946.9924.63350.820.613
Always-verify30.3634.0429.15421.744.488
EGR Selective26.7936.8426.15375.982.756
LLM replanningATV28.5746.0724.85365.530.631
Always-verify30.9538.2028.34426.284.185
EGR Selective30.3632.9426.88409.392.583
Contract-guided recoveryATV26.7952.6922.98301.410.649
Always-verify26.1932.9728.18419.674.226
EGR Selective29.7634.0724.70371.653.012

Compared with ATV, broader verification coverage improves reliability. With each of the three recovery methods held fixed, EGR Selective achieves higher TS and lower UE on both models, while using more verification queries. This shows the value of checking successful responses that may conceal missing effects, beyond verifying ambiguous responses alone.

Compared with Always-verify, selective verification reduces query cost. EGR Selective uses fewer verification queries in all six comparisons across the two models and three fixed recovery methods, with reductions of 22.6%–38.6%.

Averaging equally across the three recovery methods, EGR Selective increases TS by 2.38 and 1.79 percentage points and reduces UE by 8.39 and 13.96 percentage points relative to ATV on Qwen and GLM, respectively. Compared with Always-verify, it reduces verification queries by 24.6% and 35.3%, respectively. Both the individual comparisons and the averages show the trade-off between reliability and verification cost.

Response to W2.1 & Q2.1: Task success and effect closure in controlled partial-execution recovery

1. Are the missing effects required by the task?

Some missing effects are ancillary effects of the original operation and are not required for the TS evaluator to judge the task successful. Among the 137 episodes that passed TS, 85 did not achieve EC; in 49 of these (57.6%), the requested content was complete, while effects such as notifications or invitation cleanup were missing. For example, a Splitwise expense's amount and allocation may both satisfy the request, so TS passes, but a missing notification leaves EC at 0.


2. How are valid alternative completions assessed?

We assess the final database state, without requiring the original call to be repeated. TS checks whether the task objective is satisfied, while EC checks whether all intended effects of the original operation are complete. Different tools or execution paths are accepted when their final states satisfy the corresponding requirements.


3. How do we explain the gap between TS and EC?

Besides the different scopes of task objectives and operation effects, environment and tool limitations are an important reason for low overall EC: 147 of the original 192 incomplete episodes (76.6%) involve such limitations. Trajectory analysis, API inspection, and tests in AppWorld identified two constraints:

  • Insufficient query evidence for some effects, making it difficult to determine whether another action is needed or would cause duplicates.
  • No separate action to complete residual effects, while repeating the entire operation may duplicate completed effects.

The agent can use only the provided APIs, rather than directly modify the database. Detecting missing effects therefore does not always make safe recovery possible. After excluding the same APIs with these limitations from both methods, original EGR-Lite achieves EC of 61.90% on the common subset of 84 episodes, while producing fewer duplicate effects than direct retry.

Table 4: Common subset supported by the available recovery APIs (84 episodes per method).

Recovery methodEC ↑Dup. (episodes) ↓
Direct retry51/84 (60.71%)13
EGR-Lite52/84 (61.90%)0

Tables 4 and 5 uniformly use corrected matching of generated objects and related records, addressing effects missed by the original scoring without changing the effect requirements.

We further analyzed the remaining 45 incomplete episodes and found that 32 (71.1%) stopped at verification and diagnosis without entering recovery. Inspection of recovery trajectories also revealed actions that deviated from the original call's target. Based on these findings, we expanded verification coverage and made fuller use of existing query results to determine whether effects were complete; during recovery, we explicitly prioritized the original call's missing effects. We then conducted paired reruns of the original and refined methods on the original 61 shared prefixes to test whether these changes bring actual benefits, as shown below.

Table 5: Complete controlled Partial experiment before and after refinement (244 episodes per method).

MethodTS ↑EC ↑EC (common 84-episode subset) ↑Dup. (episodes) ↓
EGR-Lite (paired rerun)58.61%71/244 (29.10%)56/84 (66.67%)27
Improved method65.98%87/244 (35.66%)73/84 (86.90%)25

The refinements improve overall EC by 6.56 percentage points and EC on the same common subset by 20.24 percentage points to 86.90%, alongside higher TS and fewer duplicate effects.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.