- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies tool calls where the response doesn't align with what actually happened. This can be ambiguous responses or missing data replies. To fix for it, the authors:
- Decide whether to check or not for a given reply. 2, If it checks, read the state back.
- If something is missing, just fix that part.
- Re-verify.
This per-API knowledge lives in effect contracts which record what each api should change, how to check it, and how to repair it.
Critical strengths
- The problem is real, and formulated clearly.
- The paper reports raw counts, tables, and admits non-significant results, which is useful.
Critical weaknesses
- Under partial execution, in about 4/5 of episodes the faulty call's effects are incomplete.
- So in about four fifths of these episodes the effects of the faulty call are still incomplete at the end.
Mainly, the paper does not analyse where the pipeline stalls
Questions for authors
- How often does EGR-Lite actually reach a recovery decision under partial execution?
- Why is EGR-Lite's EC lowest under Qwen None/Ambiguous in every seed?
- What is the exact ambiguous-response template?
5 comments
Response to W1–W2: Explaining and improving EC under Partial execution
One direct reason for low EC is that many episodes never reach recovery. Of the original 192 incomplete episodes, 158 (82.3%) stopped at verification and diagnosis. Tool limitations are an important reason: 147 incomplete episodes (76.6%) involved such limitations, including 126 that stopped at this stage. We also refined verification and recovery: paired reruns improve overall EC from 29.10% to 35.66%, and EC after excluding APIs with these limitations from 66.67% to 86.90%.
1. Where does the pipeline stall?
Table 2: Analysis of the original 192 episodes without effect closure.
| Stage | Observation | Episodes |
|---|---|---|
| Verification and diagnosis | No qualified contract enabled | 31 |
| Verification and diagnosis | Core effects checked, other missing effects overlooked | 44 |
| Verification and diagnosis | Effect status remained unresolved | 83 |
| Recovery | Missing effects identified, but no target recovery executed | 14 |
| Recovery | Recovery executed, but final effects remained incomplete | 7 |
| EC scoring | Completed effects incorrectly scored as incomplete | 13 |
| Total | 192 |
Table 2 shows that most incomplete episodes never obtained a complete, clear effect diagnosis and therefore did not enter targeted recovery. Improving overall EC also requires more episodes with a safe recovery path to receive a clear diagnosis and reach recovery.
Tool limitations make it difficult to progress from verification and diagnosis to recovery. API inspection and AppWorld tests identified two limitations:
- Missing information: an expense notification goes to other participants, but the caller's expense and notification queries cannot confirm receipt. The queries examined cannot establish whether resending is necessary or would create duplicates.
- No separate recovery action: after a payment request is rejected, the API prevents rejecting it again and provides no action to resend a missing rejection notification.
Agents can use only the provided APIs, without directly modifying the database. Thus, even detecting a Partial failure may not provide the information or actions needed for safe recovery.
We excluded the same APIs with these limitations from both methods and compared the remaining common subset of 84 episodes, using identical, corrected EC scoring.
Table 3: Original recovery trajectories, with and without APIs with these limitations.
| Recovery method | Full-set EC ↑ | EC excluding APIs with these limitations ↑ | Subset Dup. (episodes) ↓ |
|---|---|---|---|
| Direct retry | 67/244 (27.46%) | 51/84 (60.71%) | 13 |
| EGR-Lite | 65/244 (26.64%) | 52/84 (61.90%) | 0 |
Original EGR-Lite achieves EC of 61.90% on this subset, slightly above direct retry, while avoiding its 13 duplicate-effect episodes. The difference between full-set and subset EC illustrates the importance of the environment's available information and actions.
2. Targeted refinements and evaluation
Our trajectory analysis also identified verification omissions and recovery actions that deviated from the original target. Of the 83 unresolved diagnoses, 24 already had sufficient query evidence to determine the core effects' status. For example, sent-mail and draft queries showed that an email had been sent while its original draft remained, identifying draft cleanup as the remaining effect.
We therefore expanded effect coverage, improved interpretation of existing queries, and instructed replanning to prioritize the original call's missing effects. Recovery still uses agent-accessible evidence and existing tools.
We conducted paired reruns on the original 61 shared prefixes, with 244 episodes per method. Both methods use identical, corrected EC scoring and the same 84-episode API subset.
Table 4: Paired controlled Partial experiment (244 episodes per method).
| Method | TS ↑ | EC ↑ | EC excluding APIs with these limitations ↑ | Dup. (episodes) ↓ |
|---|---|---|---|---|
| EGR-Lite (paired rerun) | 58.61% | 71/244 (29.10%) | 56/84 (66.67%) | 27 |
| Improved method | 65.98% | 87/244 (35.66%) | 73/84 (86.90%) | 25 |
Compared with the original method rerun in the same experiment, overall EC improves by 6.56 percentage points, and EC after excluding APIs with these limitations improves by 20.24 percentage points. TS increases from 58.61% to 65.98%, while duplicate-effect episodes decrease from 27 to 25. We will include the analysis and results in the revised paper.
Response to Q2: Why EC is low under Qwen None/Ambiguous, and targeted refinements
Analysis of the original trajectories identifies failure to consistently recover the original call's intended effects after verification as an important reason EGR-Lite has lower EC than the baselines under Qwen None/Ambiguous. We therefore made the recovery goal explicit and conducted a paired evaluation at 125 recovery nodes from the original three seeds. EC increased from 76.00% to 88.00%, and TS from 60.80% to 72.00%, without increasing the total numbers of duplicate-effect episodes or unsafe events.
Under None/Ambiguous, the original call produces no effects, so repeating it can often complete recovery. In the original trajectories, Vanilla subsequently repeated the original call successfully in 239 of its 297 fault-triggered episodes (80.47%). ATV, Always-verify, and EGR-Lite all verify ambiguous responses when the call is verifiable, but differ in how they proceed afterward: the verification baselines provide the results to the model to continue execution, whereas EGR-Lite additionally selects a recovery method. Some EGR replanning still uses the entire task as its goal, without prioritizing the missing effects of the current call. We examined the 40 episodes in the original experiment that entered replanning but did not achieve effect closure, as summarized below.
Table 5: Trajectory analysis of the 40 episodes without effect closure after replanning in the original Qwen None/Ambiguous experiment.
| Observation in the trajectory | Original tool available? | Episodes | Proportion |
|---|---|---|---|
| Acted on other objects without completing the original target | Yes | 31 | 77.5% |
| Modified the original object but did not write the required content | Yes | 6 | 15.0% |
| The original tool was absent from the recovery tool list | No | 3 | 7.5% |
| Total | 40 | 100% |
In 37 cases (92.5%), the original tool remained available, but the recovery actions did not complete the original target. For example, in a task requiring several friends to be added, the model detected that the first friend had not been added but proceeded to add others, ultimately omitting the first. Thus, even after detecting missing effects, EGR can leave more effects incomplete than baselines that repeat the original call.
We therefore refined the recovery goal to complete the publicly diagnosed missing effects of the original call before continuing the remaining task. The model still selects actions using agent-accessible evidence and existing tool permissions; it can use alternative operations that achieve the same effects, without being required to retry every ambiguous call. To evaluate this change, we continued execution from 125 recovery nodes publicly diagnosed as NONE across the original three seeds. The pre-recovery state, interaction history, diagnostic evidence, available tools, and remaining budget were held fixed, with only the recovery goal wording changed.
Table 6: Paired evaluation at original Qwen None/Ambiguous recovery nodes (125 episodes per method).
| Method | EC ↑ | TS ↑ | Dup. (episodes) ↓ | UE (episodes) ↓ |
|---|---|---|---|---|
| EGR-Lite (original recovery goal) | 95/125 (76.00%) | 76/125 (60.80%) | 0 | 1 |
| EGR-Lite (explicit recovery goal) | 110/125 (88.00%) | 90/125 (72.00%) | 0 | 1 |
Making the recovery goal explicit improves EC by 12.00 percentage points and TS by 11.20 percentage points, with net gains across all three seeds. A paired bootstrap analysis clustered by task ID gives a 95% confidence interval of [3.88,20.97] percentage points for the EC improvement. These results support translating effect diagnosis into a clear recovery goal for the original call to reduce target drift and improve effect closure. The results above come from paired continuations at original recovery nodes; we will further evaluate this refinement on the full None/Ambiguous benchmark and report the results.
Thank you for your careful review and valuable suggestions. Your feedback has helped us clarify the method and strengthen the experimental analysis. Below, we address each weakness (W) and question (Q), providing additional explanations and experimental results. We hope these responses address your concerns and welcome any further points that need clarification.
Response to Q1: How often EGR-Lite reaches recovery under Partial execution
In the original 244 Partial episodes, EGR-Lite reached a recovery decision in 58 cases (23.77%). Recovery actions were executed in 50 of these cases, and 43 ultimately achieved effect closure, giving an EC rate of 86.0% among episodes in which recovery was executed.
The table gives the number of episodes reaching each stage for both models under Success and Ambiguous feedback, with 61 episodes per group.
Table 1: Recovery under each condition in the original controlled Partial experiment (61 episodes per group; EC checked using a common scoring criterion).
One direct reason for low overall EC is that many episodes never enter targeted recovery of the missing effects. Recovery is selected in only about a quarter of episodes and actually executed in about a fifth. Among episodes with executed recovery, however, 86.0% achieve effect closure. This shows that recovery can effectively complete missing effects in these cases. Improving overall EC also requires more episodes with a feasible recovery path to receive a clear diagnosis and reach recovery.