acceptodds
Under review as a conference paper at ICLR 2027

Source-MSIS: Tracing Source Errors in Interactive Agents

Abstract

Accurately tracing root causes is key to helping interactive agents recover from failures in long-horizon tasks. Observations at failure may reveal only the cascading effects of earlier, unnoticed errors. Existing methods can flag suspicious steps in execution traces, but struggle to verify root causes and derive precise, nonredundant recovery plans. We propose the Source-indexed Minimal Sufficient Intervention Sequence (Source-MSIS) framework for tracing source errors and verifying recovery. The framework uses execution records from failed tasks to identify earlier errors that may block later steps and proposes targeted repairs for these errors. It then replays the task to check whether the repairs remove the corresponding blockages. If the task remains incomplete, feedback from the new execution guides the search for other errors and additional repairs. Once the task is completed, an independent verifier removes one or more repairs and replays the remaining repairs in their original order under the same conditions. A plan passes verification only if the full plan completes the task and every reduced plan fails to do so. Across three interactive benchmarks with publicly available base models, Source-MSIS achieves an overall recovery success rate of 65.3%, compared with 5.7% for the strongest baseline, and reaches 100% on ALFWorld. These results show that tracing source errors and verifying recovery through replay provide concrete targets for repair, helping agents complete previously failed tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM Agents, Failure Attribution, Failure Recovery, Counterfactual Replay

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pK2uOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method that attempts to identify the location of errors in a long trajectory and fix them for a follow-up run. In particular, it proposes three core steps:

  1. error identification
  2. counterfactual tests
  3. probes for minimality of the repair sequence
Critical strengths
  1. With a very big caveat (see below), the paper generally cites and compares with previous work.
  2. The method seems reasonably well described in section 3.
Critical weaknesses

I mainly had two critical questions that lead me to not be willing to accept the paper in its current form:

  1. Major questions about the correctness of the experimental results. In particular, on ScienceWorld and AlfWorld, the proposed method gets extremely strong scores (including 110/110), but the baselines get extremely low scores, under 5%. In a reasonable comparison, this would not happen. Digging a bit deeper into the text, it seems that auxiliary contributions such as the task parser, repair constructor, action interface, and continuation policy may be making a bigger difference here. I would want to see experiments comparing with the baselines keeping all of these other auxiliary elements fixed.

  2. I am not sure about the feasibility of using this method in real situations due to potential additional cost with respect to time and money. As a practitioner, I would want to see a graph of accuracy on the y axis and monetary/time cost on the y axis to be able to better understand whether such methods would be useful in practical settings.

Questions for authors

Would it be possible to get answers to the weaknesses above?

2 comments
AuthorsOpenReview ↗

[1/2] Author Response

Note to the Area Chair, Senior Area Chair, and Reviewer pK2u: The 1: Clear rejection rating and Soundness: 1 score in this review rest on two unsubstantiated claims: (1) a speculative assertion that baseline recovery rates under 5% "would not happen" in a reasonable comparison and must be driven by auxiliary components, which overlooks both the 7,140 stage-by-stage execution records in Appendix B and the controlled fixed-solver ablations in Appendix C, and (2) an out-of-scope demand for a plot of "accuracy on the y axis and monetary/time cost on the y axis" [sic], a metric absent from prior work in this area and contradicted by our empirical cost data. Below we refute both claims across Part 1/2 and Part 2/2.


1. Controlled Experiments and Long-Horizon Task Theory Refute the Claim That Auxiliary Components Drive Performance (Addressing Weakness 1)

The reviewer speculates from the text alone that "auxiliary contributions such as the task parser, repair constructor, action interface, and continuation policy may be making a bigger difference here," without checking the controlled ablations or execution logs in the paper. When all of these auxiliary elements are held strictly fixed, empirical data prove that our core source-tracing loop, multi-source composition, and dependency ordering drive the performance gains. Moreover, the sub-5% baseline recovery rate is the inevitable theoretical outcome of applying single-error attribution to long-horizon tasks with multiple interdependent prerequisites.

A. Direct empirical proof when holding all auxiliary elements strictly fixed In Section 4.3 (Tables 2 and 3, page 9, lines 433 to 466) and Appendix C (pages 21 to 24), we hold all auxiliary elements constant across two sets of controlled experiments:

1) Fixed-solver and fixed-auxiliary ablations across all three benchmarks (Table 2, page 9, and Appendices C.3 and C.6, pages 21 to 24): We hold the exact same task parser, exploration procedure, native repair constructor, action interface, continuation policy, and budgets fixed across ALFWorld, Habitat, and ScienceWorld, and ablate only our core mechanisms:

Core Mechanism Ablated (Fixed Auxiliary Modules)ALFWorld (7B, N=110)Habitat (7B, N=30)ScienceWorld (7B, N=30)Two-Stage (N=27)
Full Source-MSIS110/110 (100.0%)20/30 (66.7%)15/30 (50.0%)27/27 (100.0%)
Without Multi-Source Composition5/110 (4.5%)0/30 (0.0%)1/30 (3.3%)–
Without Dependency Ordering23/110 (20.9%)0/30 (0.0%)1/30 (3.3%)–
Without Residual Feedback Iteration110/110 (100.0%)11/30 (36.7%)15/30 (50.0%)0/27 (0.0%)

Removing multi-source composition or dependency ordering causes recovery to collapse by -79.1 to -95.5 percentage points in ALFWorld (two-sided exact McNemar p<10−25p < 10^{-25}, Tables 14 and 15, page 21), -66.7 percentage points in Habitat (Holm-adjusted p=1.14×10−5p = 1.14 \times 10^{-5}, Tables 18 and 19, page 23), and -46.7 percentage points in ScienceWorld (p=4.88×10−4p = 4.88 \times 10^{-4}). Removing residual feedback iteration drops recovery by -30.0 percentage points in Habitat (p=0.00781p = 0.00781) and -100.0 percentage points on two-stage cases (p=1.49×10−8p = 1.49 \times 10^{-8}, Table 16, page 22).

2) Controlled 2x2 mechanism experiment in Habitat (Table 3, page 9, and Appendix C.7, page 24): On 16 development cases in Habitat (Qwen2.5-7B), we hold the failed policy, action interface, continuation rule, and auxiliary recovery modules constant (sharing the exact same public-state compiler in the enhanced setting) while switching only our explicit source-tracing loop on or off:

Auxiliary Recovery Capability (Fixed Within Row)Without Source LoopWith Source Loop (Source-MSIS)Net Gain from Source Loop
Basic auxiliary recovery3/16 (18.8%)15/16 (93.8%)+75.0 percentage points
Enhanced auxiliary recovery (shared compiler)9/16 (56.3%)16/16 (100.0%)+43.7 percentage points

Without the explicit source loop, multi-source composition, and dependency ordering of Source-MSIS, the auxiliary modules alone cannot recover these tasks. (Continued in Part 2/2.)

AuthorsOpenReview ↗

[2/2] Author Response

Response to Reviewer pK2u (Part 2/2)

B. Why the drop from 149 evaluable replays to 1 completed task in baselines is theoretically expected (Weakness 1, continued) The reviewer claims that "In a reasonable comparison, this would not happen," conflating executing a replayed trajectory with completing all prerequisites of a long-horizon task:

  • Baselines execute valid replays, but fail at final task completion: In Appendix B.2 (Table 8, page 18, lines 919 to 938) across all 7,140 runs, AgentDebug, AgentDebugX, Who&When, and CausalFlow (7B) execute 149 to 150 evaluable environment replays out of 150 cases in ScienceWorld (and 105 to 107 out of 110 in ALFWorld), yet only 0 to 1 case passes the official task evaluator (Ω(τ)=1\Omega(\tau) = 1).
  • Theoretical reason for this drop: As explained in Section 1 (pages 1 to 2, lines 048 to 065), auxiliary interfaces only provide local state access and syntax formatting, whereas ScienceWorld and ALFWorld require satisfying multiple chained prerequisites in strict dependency order. Existing baselines perform single-point attribution or one-shot reflection without composing repairs across multiple source errors, ordering repairs by task dependencies, or verifying local milestone restoration via replay. Fixing one error while leaving even one downstream prerequisite unmet yields a task score of 0.
  • In Table 2 (page 9), restricting Source-MSIS to a single source (Without Multi-Source Composition) likewise drops ALFWorld recovery from 100.0% (110/110) to 4.5% (5/110), matching the <5% baseline regime. Dismissing these results from the text alone without checking Appendices B and C is unfounded.

2. Cost Curves Are Outside the Scope of This Field, and Empirical Data Refute the Assumption (Addressing Weakness 2)

The reviewer questions feasibility and asks for a graph with "accuracy on the y axis and monetary/time cost on the y axis" [sic]. This objection fails on both literature standards and empirical facts:

A. Out of scope and absent across prior work in this area This paper studies how to trace source errors and verify recovery via counterfactual replay (Section 1, page 2, lines 079 to 087), not commercial API pricing or latency optimization. Across the literature on agent failure attribution and intervention-based debugging, including Who&When (ICML 2025), AgenTracer (Zhang et al., ICLR 2026), DoVer (Ma et al., ICLR 2026), OpenRCA (ICLR 2025), AgentDebug (Zhu et al., 2025), AgentDebugX (2026), REFLECT (2026), and CausalFlow (2026), none reports commercial monetary cost curves or wall-clock latency trade-off plots. DoVer (Ma et al., 2026, Section 7) notes that strict latency and cost constraints lie outside intervention-based debugging evaluation, while AgentDebug (Zhu et al., 2025, Sections 4.2 to 5.1) and AgenTracer (Zhang et al., 2026, Sections 5.2 and 5.3) evaluate attribution and recovery rather than monetary or time curves (Appendix E.5, page 30, lines 1592 to 1599).

B. Empirical runtime and token measurements contradict the reviewer's assumption Measured time and token usage across 2,048, 4,096, and 8,192 token caps (Appendix E.5, Tables 27 to 30, Figures 5 and 6, pages 28 to 31) show the opposite of what the reviewer assumed (Table 27, page 28):

Environment & Method (Qwen2.5-7B, N=8)Rec.CallsInput / Output TokensStepsReplaysTime (s)
ALFWorld: Who&When0/860.2539517.50 / 1124.6276.755.0025.70
ALFWorld: AgenTracer0/88.127460.00 / 362.5016.623.126.94
ALFWorld: AgentDebug0/864.0047751.88 / 2007.3877.625.0034.06
ALFWorld: CausalFlow0/85.505916.25 / 169.258.751.384.11
ALFWorld: DoVer0/855.3856716.50 / 1428.6269.252.6227.39
ALFWorld: Source-MSIS8/81.001826.00 / 158.7525.3813.0013.37
Habitat: All 5 baselines (range)0/81.00–7.121830.88–15196.88 / 224.12–1758.129.38–36.252.25–5.005.37–22.41
Habitat: Source-MSIS4/89.7522793.12 / 2493.1220.502.1229.00
  • On ALFWorld, Source-MSIS search uses fewer calls (1.00) and fewer tokens (1826.00 / 158.75) than all five baselines, runs 1.9x to 2.5x faster than multi-round baselines (13.37 s vs. 25.70 s to 34.06 s), and is the only method to recover the tasks (8/8 vs. 0/8).
  • On Habitat, baselines spend 5.37 s to 22.41 s for 0/8 recovery, whereas Source-MSIS recovers 4/8 (50%) in 29.00 s (scaling from 2/8 to 3/8 to 4/8 across the three token caps in Table 29, Figures 5 and 6, pages 29 to 31). Post-recovery minimality verification (Table 28, page 29, adding 103.29 s for 116.66 s total) is accounted for separately in Section A.6 (page 16) and Table 25 (page 27) for on-demand auditing.
Reviewer_HieTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Source-MSIS, an intervention-based framework for diagnosing and recovering from failures in long-horizon interactive agents. It identifies source errors from failed execution trajectories, constructs targeted interventions, and iteratively replays tasks to discover and address subsequent errors. The framework further introduces a minimality verification procedure to determine whether each intervention is necessary for successful recovery. Experiments across Habitat, ScienceWorld, and ALFWorld evaluate its recovery performance against existing debugging and failure-attribution methods.

Critical strengths

S1. Important Problem of Intervention-Based Agent Debugging

The paper addresses the important problem of diagnosing and recovering from failures in long-horizon interactive agents. Its intervention-based approach is particularly valuable, as it goes beyond identifying suspicious execution steps to testing potential root causes through targeted repairs and environment replay.

S2. Analysis of Failure Cases and Recovery Limitations

The paper provides useful analyses of why the proposed method fails to recover certain tasks, particularly in Habitat. By examining incomplete action sequences, omitted task goals, and rejected actions across different model configurations, the authors offer insights into the limitations of their approach and potential directions for improving agent recovery.

Critical weaknesses

W1. Insufficient Justification for the Necessity and Practical Implications of Minimality

The paper emphasizes minimal sufficient intervention sequences (MSIS) as a central contribution, but their necessity and practical implications remain insufficiently justified. Since minimality verification is performed only after successful task recovery, it is unclear how enforcing minimality contributes to improved recovery performance or more accurate source-error attribution, beyond identifying redundant repairs. The experiments lack a dedicated analysis demonstrating the benefits of minimality, such as improved diagnosis quality, intervention efficiency, or recovery robustness. Given the exponential verification cost and incomplete certification coverage, this also raises concerns about the novelty and practical value of minimality as a core contribution.

Questions for authors

Q1. Necessity and Contribution of Minimal Sufficient Interventions

The paper emphasizes minimal sufficient intervention sequences (MSIS), but their necessity and practical benefits remain unclear. Since minimality verification is performed after successful recovery, how does it improve recovery outcomes or source-error diagnosis beyond identifying redundant repairs? Could the authors provide an ablation or analysis demonstrating the added value of minimality, particularly considering its exponential verification cost?

Q2. Baseline Comparisons and Sources of Performance Gains

The proposed method achieves substantially higher recovery rates than the baselines, particularly on ScienceWorld and ALFWorld, where baseline success rates are nearly zero. What are the key mechanisms responsible for these large improvements? In particular, how much of the gain comes from environment-specific dependency construction and recovery planning, versus the proposed source-error tracing and iterative intervention framework? Could the authors provide more controlled comparisons using comparable recovery capabilities across methods?

Q3. Generalizability and Evaluation on More Recent Benchmarks

The framework relies on executable replay, task dependencies, and environment-specific intervention libraries. How readily can it generalize to more realistic/recent or open-ended agent environments where these assumptions may not hold? The adopted benchmarks seem to be a bit old (2021 or 2022), not sure about the method’s performance on recent benchmarks.

3 comments
AuthorsOpenReview ↗

[1/2] Author Response

Thank you for recognizing the importance of intervention-based agent debugging and our failure analysis. We have uploaded a revised manuscript (highlighted in blue) incorporating all new experiments and analyses. Below we address Q1 in Part 1/2 and Q2 to Q3 in Part 2/2.


Q1: Beyond identifying redundant repairs, how does minimality verification improve source-error diagnosis and recovery outcomes?

A: Enforcing minimality after task recovery provides three concrete benefits for source-error diagnosis, long-term experience reuse, and execution stability:

1. Eliminating false-positive root causes in source-error diagnosis In Source-MSIS, every repair ri∈Qr_i \in Q is bound to a specific candidate source error ci∈Zc_i \in Z (Section 3.3, page 5, lines 238 to 245). If a full sequence QQ completes the task (Sufficiency) but still succeeds after removing rir_i, then cic_i was not an actual bottleneck blocking task completion, meaning cic_i is a false-positive root-cause diagnosis. As shown in Section 4.2 (page 8, lines 407 to 410) and Appendix E.2 (Table 24, page 27), although all 330 ALFWorld runs recover the task (110/110), minimality verification discovers that 51 of the 330 successful sequences (15.5%) contain redundant repairs (2, 19, and 30 redundant sequences under 3B, 7B, and 12B, respectively, reaching 27.3% on 12B), certifying only the remaining 279 sequences (Certified) alongside 229 certified sequences in ScienceWorld (Table 23, page 26). Additional deletion tests in Appendix E.4 (Table 26, pages 27 to 28) likewise certify 16/16 Habitat controlled sequences (32 subsequence tests, 0 successes) and 8/8 ALFWorld verification sequences (40 subsequence tests, 0 successes). Minimality verification thus directly improves diagnosis accuracy by pruning false-positive root causes that sufficiency alone cannot detect.

2. Lowering the cost of long-term experience reuse and agent memory maintenance Interactive agent frameworks increasingly store successful recovery trajectories in memory or skill libraries to reuse on similar future tasks. While minimality verification requires up to 2k2^k checks, the number of declared repairs kk within a task window is small in practice: Appendix E.3 (Table 25, page 27) shows a mean of k=1.00k = 1.00 repair (2.00 replays) in ScienceWorld and k=2.55k = 2.55 repairs (7.22 physical executions) in ALFWorld. Moreover, for non-minimal plans, our early-rejection rule (Section 3.4, page 6, lines 276 to 278, and Appendix E.4, page 28, lines 1472 to 1477) stops immediately as soon as any proper subsequence succeeds, skipping the remaining subset tests. Paying this small one-time verification cost guarantees that the stored recovery experience is free of superfluous steps. When that verified recovery experience is repeatedly retrieved and reused across future episodes, removing redundant steps continuously saves model tokens, preserves the agent's limited context window, and conserves environment action budgets.

3. Reducing compounding uncertainty and execution risk in long-horizon tasks In realistic embodied and autonomous tasks, execution uncertainty is always present and no two runs are identical. Every unnecessary action executed during recovery introduces additional state transitions and the risk of unintended side effects. If an agent reuses a learned recovery routine that contains even one redundant manipulation step, executing that unnecessary step in a slightly shifted environment state can trigger cascading errors and cause the entire task to fail. Verifying minimality strips away unnecessary interventions, directly reducing compounding execution risk and improving the robustness of learned recovery behaviors.

AuthorsOpenReview ↗

[2/2] Author Response

Q2: How much of the gain comes from environment-specific dependency construction and recovery planning, versus the proposed source-error tracing and iterative intervention framework?

A: Controlled experiments in Section 4.3 (Tables 2 and 3, page 9, lines 433 to 466) and Appendix C (pages 21 to 24) isolate these two factors directly and show that the vast majority of the recovery gains (+43.7 to +95.5 percentage points) come from our proposed source-error tracing, multi-source composition, dependency ordering, and iterative feedback mechanisms, rather than from environment-specific planning modules.

1. Direct separation of environment-specific recovery planning vs. our source-tracing loop (Table 3, page 9, and Appendix C.7, page 24) On 16 controlled cases in Habitat (Qwen2.5-7B), we cross environment-specific recovery planning capability with our explicit source-tracing loop under identical models, tasks, interfaces, and budgets:

Recovery Planning CapabilityPlanning Alone (No Source Loop)With Source-MSIS LoopNet Gain from Source Loop
Basic recovery planning3/16 (18.8%)15/16 (93.8%)+75.0 percentage points
Enhanced state & dependency compiler9/16 (56.3%)16/16 (100.0%)+43.7 percentage points

Even with the enhanced state compiler and iterative replay, relying on recovery planning without our source-tracing loop recovers only 9/16 (56.3%) cases (and 3/16 [18.8%] under basic planning). Adding our explicit source-tracing loop boosts recovery to 16/16 (100.0%) and 15/16 (93.8%), respectively.

2. Keeping all environment-specific dependency constructors and planners 100% fixed across all three benchmarks (Table 2, page 9, and Appendices C.3 and C.6, pages 21 to 24) In Table 2 (page 9), we keep the entire environment-specific task parser, dependency constructor, action interface, and native recovery planner fixed across ALFWorld, Habitat, and ScienceWorld, and ablate only our core framework mechanisms:

Core Mechanism Ablated (Fixed Planner)ALFWorld (7B, N=110)Habitat (7B, N=30)ScienceWorld (7B, N=30)Two-Stage (N=27)
Full Source-MSIS Framework110/110 (100.0%)20/30 (66.7%)15/30 (50.0%)27/27 (100.0%)
Without Multi-Source Composition5/110 (4.5%)0/30 (0.0%)1/30 (3.3%)–
Without Dependency Ordering23/110 (20.9%)0/30 (0.0%)1/30 (3.3%)–
Without Residual Feedback Iteration110/110 (100.0%)11/30 (36.7%)15/30 (50.0%)0/27 (0.0%)
  • Without multi-source composition or dependency ordering, the environment-specific planners alone recover only 4.5% to 20.9% in ALFWorld (p<10−25p < 10^{-25}), 0.0% in Habitat (p=1.14×10−5p = 1.14 \times 10^{-5}), and 3.3% in ScienceWorld (p=4.88×10−4p = 4.88 \times 10^{-4}).
  • Without residual feedback iteration, recovery drops by 30.0 percentage points in Habitat (20/30 vs. 11/30, p=0.00781p = 0.00781, recovering 9 additional cases via replay feedback) and by 100.0 percentage points on two-stage cases where the second error becomes visible only after the first repair takes effect (27/27 vs. 0/27, p=1.49×10−8p = 1.49 \times 10^{-8}, Table 16, page 22).

Q3: How readily can the framework generalize to more recent or open-ended agent environments?

A: We evaluated Source-MSIS and all five original baselines on the open-ended coding and REST API benchmark AppWorld using Qwen3-14B and the official task evaluator, reported in Section 5 (page 10, lines 516 to 518) and Appendix F (Table 31, page 32, lines 1674 to 1717):

Method (Qwen3-14B on AppWorld, N=12)RecoveredRSR
Who&When / AgenTracer / AgentDebug / CausalFlow / DoVer (All 5 Baselines)0/120.0%
Source-MSIS1/128.3%

In AppWorld, agents generate multi-line Python code blocks interacting with complex REST APIs rather than discrete embodied actions. Under the shared execution adapter and identical budgets (24 model calls, 24,576 tokens, 3 recovery attempts), all five baselines fail to recover any task (0/12), whereas Source-MSIS is the only method to achieve end-to-end task recovery (1/12).

As summarized in Section 5 (page 10, lines 511 to 518), Source-MSIS is a training-free algorithmic framework whose effectiveness scales directly with the accuracy of prerequisite dependency extraction and action-interface grounding in the target environment. While open-ended Python/API state spaces make dependency extraction harder than in structured embodied simulators, this result demonstrates that our core loop (evidence extraction, local repair binding, and replay verification) transfers across domains without task-specific training, and will directly benefit from future advances in code and API dependency modeling.

Thank you for the detailed rebuttal and clarifications. My concerns are mostly addressed. Will raise the score from 2 to 3.

Reviewer_yPjCOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Source-MSIS, a framework for tracing source errors in failed agent trajectories and verifying recovery. It matches trajectory evidence against task dependencies to propose candidate source errors, binds each to a local repair, and replays the task with previously retained repairs.

Critical strengths
  1. The topic is important and the use case is valuable. Turning failure localization into executable, verified repairs moves beyond flagging suspicious steps.
  2. The paper is well written and has a good baseline comparison. Five recent baselines are run on the same failed cases under three model configurations, with paired McNemar tests and bootstrap intervals (Appendix B).
  3. The results are quite impressive compared with the baselines: 77–79/150 vs. at most 1/150 on ScienceWorld, and 110/110 vs. at most 5/110 on ALFWorld (Table 1).
Critical weaknesses
  1. The generality of the results could be strengthened with more base models, harnesses and ablations.
    • All experiments use relatively small models (Qwen2.5-3B/7B, Gemma3-12B), and each environment uses a single harness. Results with stronger or more recent models and with different agent harnesses would show whether the gains hold.
    • The component ablation (Table 2) is reported only for ALFWorld with 7B, plus constructed two-stage cases. Ablations on Habitat and ScienceWorld, where performance varies most, would be informative.
    • Comparison with the most recent failure-diagnosis and recovery baselines (e.g., AgentDebugX, alongside others) would also help.
Questions for authors
  1. How efficient is the method compared with the baselines? Minimality verification requires 2^k replays (Appendix A.6). Appendix E.3 reports about 7.2 executions and roughly 260–300 model calls per verification run on ALFWorld, on top of a search budget of up to 64 replays and 64 model calls per case. However, no comparable cost figures (environment steps, model calls, tokens, wall-clock time) are given for the baselines. Could you provide a side-by-side cost comparison? Are there ways to improve efficiency, for example by pruning the deletion tests or stopping verification early?
7 comments
AuthorsOpenReview ↗

[1/2] Author Response

Thank you for your positive assessment of our problem formulation, executable repair verification, and baseline comparisons. We have uploaded a revised manuscript (highlighted in blue) incorporating all requested experiments. Below we address Q1 in Part 1/2 and Q2 to Q4 in Part 2/2, with exact locations in the revised PDF.


Q1: How efficient is the method compared with the baselines, and can minimality verification be sped up?

A: We added a unified side-by-side cost comparison and budget-RSR curves for Source-MSIS and all five original baselines (Who&When, AgenTracer, AgentDebug, CausalFlow, and DoVer) in Appendix E.5 (Tables 27 to 30 and Figures 5 and 6, pages 28 to 31, lines 1478 to 1670), referenced in Section 4.2 (page 8, line 410).

The experiment uses Qwen2.5-7B on 8 common failed cases in Habitat and 8 in ALFWorld, executed sequentially on the same local GPU setup under generation caps of 2,048, 4,096, and 8,192 tokens (cap of 64 model calls during search). At the 8,192-token budget (Table 27, page 28), Source-MSIS recovery search on ALFWorld uses fewer model calls (1.00) and fewer input and output tokens (1826.00 / 158.75) than all five baselines, runs 1.9x to 2.5x faster than multi-round baselines (13.37 s vs. 25.70 s to 34.06 s), and recovers 8/8 cases vs. 0/8. On Habitat, Source-MSIS recovers 4/8 (50%) in 29.00 s while all baselines recover 0/8:

1. Recovery search costs in Habitat (N = 8, Qwen2.5-7B)

MethodRecoveredCallsInput TokensOutput TokensStepsReplaysSeconds
Who&When0/86.0012101.75660.5036.255.0012.66
AgenTracer0/81.001830.88270.7520.882.385.69
AgentDebug0/87.1215196.881758.1231.385.0022.41
CausalFlow0/81.002097.00224.1217.502.505.37
DoVer0/84.009946.62794.259.382.2510.49
Source-MSIS4/89.7522793.122493.1220.502.1229.00

2. Recovery search costs in ALFWorld (N = 8, Qwen2.5-7B)

MethodRecoveredCallsInput TokensOutput TokensStepsReplaysSeconds
Who&When0/860.2539517.501124.6276.755.0025.70
AgenTracer0/88.127460.00362.5016.623.126.94
AgentDebug0/864.0047751.882007.3877.625.0034.06
CausalFlow0/85.505916.25169.258.751.384.11
DoVer0/855.3856716.501428.6269.252.6227.39
Source-MSIS8/81.001826.00158.7525.3813.0013.37

Results across all three token caps (2,048, 4,096, and 8,192 tokens) are provided in Tables 29 and 30 (pages 29 to 30) and plotted in Figures 5 and 6 (page 31).

3. Minimality verification cost and early rejection (Appendix E.4 and Table 28, pages 28 to 29) Full verification on the same 8 ALFWorld cases requires a mean of 7 additional physical executions, 307.88 model calls, and 103.29 s. Including the search phase (summing separately timed phases), the total costs are:

Source-MSIS Cost Scope on ALFWorld (N=8)RecoveredCallsInput TokensOutput TokensStepsReplaysSeconds
Recovery search only8/81.001826.00158.7525.3813.0013.37
Recovery search + full enumeration8/8308.88170087.503788.50357.3820.00116.66
Recovery search + early rejection8/8308.88170514.123823.50357.3820.00114.48
  • Early-rejection pruning: A program with kk declared repairs has 2k−12^k - 1 proper order-preserving subsequences. With early rejection (Appendix E.4, page 28, lines 1472 to 1477), once any proper subsequence completes the task, the verifier immediately classifies the program as Non-Minimal and skips all remaining deletion tests (for example, on the 51 Non-Minimal ALFWorld sequences in Appendix E.2, early discovery of a successful subsequence prunes the remaining subset replays). Across the 8 certified minimal cases above (Table 26, page 28), no proper subsequence succeeds, so both policies check all proper subsets (7 verification executions and 307.88 calls).
  • On-demand verification: Because verification runs after recovery (Section 3.4, page 6), task recovery requires only the search phase (13.37 s on ALFWorld and 29.00 s on Habitat), while verification is invoked on demand for root-cause certification.
AuthorsOpenReview ↗

[2/2] Author Response

Q2: Why use smaller models, and do the results hold with stronger models and different agent harnesses?

A: We selected Qwen2.5-3B, Qwen2.5-7B, and Gemma3-12B to reflect deployment constraints in embodied systems, where on-board computation, memory, and response latency limits require smaller local execution models (Section 4.1, page 7, lines 328 to 329). Across these two model families and three configurations (Table 1, page 8, lines 392 to 393), Source-MSIS shows a clear scaling trend as base model capability increases:

Base ModelHabitatScienceWorldALFWorld
Qwen2.5-3B5/80 (6.3%)77/150 (51.3%)110/110 (100.0%)
Qwen2.5-7B43/80 (53.8%)78/150 (52.0%)110/110 (100.0%)
Gemma3-12B54/80 (67.5%)79/150 (52.7%)110/110 (100.0%)

To further evaluate a stronger, more recent model and different agent harnesses, we added two experiments:

  1. Stronger model (Qwen3-14B) + code/API agent harness on AppWorld (Section 5, page 10, lines 516 to 518, and Appendix F, Table 31, page 32): Across 12 common tasks in AppWorld using Qwen3-14B and a shared Python code-block/REST API execution adapter, all five original baselines recover 0/12 (0.0%) tasks, while Source-MSIS recovers 1/12 (8.3%).
  2. Different recovery harnesses in Habitat (Table 3 in Section 4.3, page 9, lines 452 to 466, and Appendix C.7, page 24): On 16 controlled cases in Habitat (Qwen2.5-7B), we compare a Basic harness (direct model action generation) with an Enhanced harness (public-state navigation and action compiler). Adding the Source-MSIS loop raises recovery from 3/16 (18.8%) to 15/16 (93.8%) under the Basic harness, and from 9/16 (56.3%) to 16/16 (100.0%) under the Enhanced harness.

Q3: What are the component ablation results on Habitat and ScienceWorld?

A: We added 300 formal ablation runs in Habitat (N=30N=30) and ScienceWorld (N=30N=30) with Qwen2.5-7B (5 conditions per formal case, excluding 4 pilot cases per environment) directly into Table 2 in Section 4.3 (page 9, lines 433 to 466) and Appendix C.6 (Tables 18 and 19, pages 22 to 24, lines 1180 to 1253). All conditions share the native auxiliary recovery constructor, action interface, and maximum budgets:

Component Group / Control (Qwen2.5-7B, N=30 Each)Habitat Recoveries (RSR)Habitat Adjusted ppScienceWorld Recoveries (RSR)ScienceWorld Adjusted pp
Full Source-MSIS20/30 (66.7%)–15/30 (50.0%)–
Multi-source composition (First repair unit only)0/30 (0.0%)1.14×10−51.14 \times 10^{-5}1/30 (3.3%)4.88×10−44.88 \times 10^{-4}
Dependency ordering (Reverse repair units)0/30 (0.0%)1.14×10−51.14 \times 10^{-5}1/30 (3.3%)4.88×10−44.88 \times 10^{-4}
Residual feedback (Without source residual iteration)11/30 (36.7%)0.007810.0078115/30 (50.0%)1.0001.000
  • Repair composition and dependency ordering are critical in both environments: Removing either component drops successful recoveries from 20/30 to 0/30 in Habitat (two-sided exact McNemar test paired by task with Holm correction across the six comparisons: p=1.14×10−5p = 1.14 \times 10^{-5}) and from 15/30 to 1/30 in ScienceWorld (p=4.88×10−4p = 4.88 \times 10^{-4}, Table 19, page 23).
  • Residual feedback iteration provides a +30.0 percentage point gain in Habitat: Disabling post-program source residual iteration in Habitat drops recoveries from 20/30 to 11/30 (Holm-adjusted p=0.00781p = 0.00781). Runtime records (Appendix C.6, page 23, lines 1237 to 1241) confirm that the full method makes 35 residual feedback model calls across these cases (vs. 0 when disabled) to recover 9 additional cases whose subsequent blockages become observable only after the initial repair program executes.

Q4: How does Source-MSIS compare with newer baselines such as AgentDebugX?

A: We evaluated AgentDebugX (LLMJudgeAnalyzer, AllAtOnceAttributor, and ReflexionSuggestion) across all 340 failed cases and three models (1,020 runs), and integrated it directly into Table 1 and Section 4.2 (page 7, lines 374 to 377, and page 8, lines 379 to 406) as well as Appendices B and E (Tables 7, 8, 10, and 21 to 24, pages 16 to 27):

Base ModelMethodHabitat (N=80)ScienceWorld (N=150)ALFWorld (N=110)
Qwen2.5-3BAgentDebugX3/80 (3.8%)0/150 (0.0%)0/110 (0.0%)
Qwen2.5-3BSource-MSIS5/80 (6.3%)77/150 (51.3%)110/110 (100.0%)
Qwen2.5-7BAgentDebugX2/80 (2.5%)0/150 (0.0%)0/110 (0.0%)
Qwen2.5-7BSource-MSIS43/80 (53.8%)78/150 (52.0%)110/110 (100.0%)
Gemma3-12BAgentDebugX32/80 (40.0%)0/150 (0.0%)3/110 (2.7%)
Gemma3-12BSource-MSIS54/80 (67.5%)79/150 (52.7%)110/110 (100.0%)

Source-MSIS consistently outperforms AgentDebugX across every benchmark and model configuration.

Thanks for your response! The results look very impressive.

I have one question about where the performance gains are actually coming from.

My understanding is that one of the key contributions is the minimality verification step, which identifies which interventions are truly necessary for successful recovery.

This is based on the successful fix already so won't affect the performance as I understand.

However, the earlier stages seem conceptually similar to existing approaches such as AgentDebug and AgentRx, where the system identifies potential errors, proposes fixes, and iteratively refines them based on execution feedback.

I'm curious what you see as the main mechanism driving the substantial improvement in recovery success rate. Is it primarily the iterative search over candidate fixes, the dependency-aware ordering of interventions, or something fundamentally different from previous methods?

I'd love to understand this distinction more clearly.

Comments on the paper

1 comment
Area_Chair_2tUeOpenReview ↗

check authors response

Dear reviewers,

The authors have posted their responses. Please read them and update your scores if they address your concerns. If they don't change your view, a short note on what remains unresolved would help the discussion.

Thank you for your time. AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.