- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a method that attempts to identify the location of errors in a long trajectory and fix them for a follow-up run. In particular, it proposes three core steps:
- error identification
- counterfactual tests
- probes for minimality of the repair sequence
Critical strengths
- With a very big caveat (see below), the paper generally cites and compares with previous work.
- The method seems reasonably well described in section 3.
Critical weaknesses
I mainly had two critical questions that lead me to not be willing to accept the paper in its current form:
-
Major questions about the correctness of the experimental results. In particular, on ScienceWorld and AlfWorld, the proposed method gets extremely strong scores (including 110/110), but the baselines get extremely low scores, under 5%. In a reasonable comparison, this would not happen. Digging a bit deeper into the text, it seems that auxiliary contributions such as the task parser, repair constructor, action interface, and continuation policy may be making a bigger difference here. I would want to see experiments comparing with the baselines keeping all of these other auxiliary elements fixed.
-
I am not sure about the feasibility of using this method in real situations due to potential additional cost with respect to time and money. As a practitioner, I would want to see a graph of accuracy on the y axis and monetary/time cost on the y axis to be able to better understand whether such methods would be useful in practical settings.
Questions for authors
Would it be possible to get answers to the weaknesses above?
2 comments
[2/2] Author Response
Response to Reviewer pK2u (Part 2/2)
B. Why the drop from 149 evaluable replays to 1 completed task in baselines is theoretically expected (Weakness 1, continued) The reviewer claims that "In a reasonable comparison, this would not happen," conflating executing a replayed trajectory with completing all prerequisites of a long-horizon task:
- Baselines execute valid replays, but fail at final task completion: In Appendix B.2 (Table 8, page 18, lines 919 to 938) across all 7,140 runs, AgentDebug, AgentDebugX, Who&When, and CausalFlow (7B) execute 149 to 150 evaluable environment replays out of 150 cases in ScienceWorld (and 105 to 107 out of 110 in ALFWorld), yet only 0 to 1 case passes the official task evaluator ().
- Theoretical reason for this drop: As explained in Section 1 (pages 1 to 2, lines 048 to 065), auxiliary interfaces only provide local state access and syntax formatting, whereas ScienceWorld and ALFWorld require satisfying multiple chained prerequisites in strict dependency order. Existing baselines perform single-point attribution or one-shot reflection without composing repairs across multiple source errors, ordering repairs by task dependencies, or verifying local milestone restoration via replay. Fixing one error while leaving even one downstream prerequisite unmet yields a task score of 0.
- In Table 2 (page 9), restricting Source-MSIS to a single source (
Without Multi-Source Composition) likewise drops ALFWorld recovery from 100.0% (110/110) to 4.5% (5/110), matching the<5%baseline regime. Dismissing these results from the text alone without checking Appendices B and C is unfounded.
2. Cost Curves Are Outside the Scope of This Field, and Empirical Data Refute the Assumption (Addressing Weakness 2)
The reviewer questions feasibility and asks for a graph with "accuracy on the y axis and monetary/time cost on the y axis" [sic]. This objection fails on both literature standards and empirical facts:
A. Out of scope and absent across prior work in this area This paper studies how to trace source errors and verify recovery via counterfactual replay (Section 1, page 2, lines 079 to 087), not commercial API pricing or latency optimization. Across the literature on agent failure attribution and intervention-based debugging, including Who&When (ICML 2025), AgenTracer (Zhang et al., ICLR 2026), DoVer (Ma et al., ICLR 2026), OpenRCA (ICLR 2025), AgentDebug (Zhu et al., 2025), AgentDebugX (2026), REFLECT (2026), and CausalFlow (2026), none reports commercial monetary cost curves or wall-clock latency trade-off plots. DoVer (Ma et al., 2026, Section 7) notes that strict latency and cost constraints lie outside intervention-based debugging evaluation, while AgentDebug (Zhu et al., 2025, Sections 4.2 to 5.1) and AgenTracer (Zhang et al., 2026, Sections 5.2 and 5.3) evaluate attribution and recovery rather than monetary or time curves (Appendix E.5, page 30, lines 1592 to 1599).
B. Empirical runtime and token measurements contradict the reviewer's assumption Measured time and token usage across 2,048, 4,096, and 8,192 token caps (Appendix E.5, Tables 27 to 30, Figures 5 and 6, pages 28 to 31) show the opposite of what the reviewer assumed (Table 27, page 28):
| Environment & Method (Qwen2.5-7B, N=8) | Rec. | Calls | Input / Output Tokens | Steps | Replays | Time (s) |
|---|---|---|---|---|---|---|
| ALFWorld: Who&When | 0/8 | 60.25 | 39517.50 / 1124.62 | 76.75 | 5.00 | 25.70 |
| ALFWorld: AgenTracer | 0/8 | 8.12 | 7460.00 / 362.50 | 16.62 | 3.12 | 6.94 |
| ALFWorld: AgentDebug | 0/8 | 64.00 | 47751.88 / 2007.38 | 77.62 | 5.00 | 34.06 |
| ALFWorld: CausalFlow | 0/8 | 5.50 | 5916.25 / 169.25 | 8.75 | 1.38 | 4.11 |
| ALFWorld: DoVer | 0/8 | 55.38 | 56716.50 / 1428.62 | 69.25 | 2.62 | 27.39 |
| ALFWorld: Source-MSIS | 8/8 | 1.00 | 1826.00 / 158.75 | 25.38 | 13.00 | 13.37 |
| Habitat: All 5 baselines (range) | 0/8 | 1.00–7.12 | 1830.88–15196.88 / 224.12–1758.12 | 9.38–36.25 | 2.25–5.00 | 5.37–22.41 |
| Habitat: Source-MSIS | 4/8 | 9.75 | 22793.12 / 2493.12 | 20.50 | 2.12 | 29.00 |
- On ALFWorld, Source-MSIS search uses fewer calls (1.00) and fewer tokens (1826.00 / 158.75) than all five baselines, runs 1.9x to 2.5x faster than multi-round baselines (13.37 s vs. 25.70 s to 34.06 s), and is the only method to recover the tasks (8/8 vs. 0/8).
- On Habitat, baselines spend 5.37 s to 22.41 s for 0/8 recovery, whereas Source-MSIS recovers 4/8 (50%) in 29.00 s (scaling from 2/8 to 3/8 to 4/8 across the three token caps in Table 29, Figures 5 and 6, pages 29 to 31). Post-recovery minimality verification (Table 28, page 29, adding 103.29 s for 116.66 s total) is accounted for separately in Section A.6 (page 16) and Table 25 (page 27) for on-demand auditing.
[1/2] Author Response
Note to the Area Chair, Senior Area Chair, and Reviewer pK2u: The
1: Clear rejectionrating andSoundness: 1score in this review rest on two unsubstantiated claims: (1) a speculative assertion that baseline recovery rates under 5% "would not happen" in a reasonable comparison and must be driven by auxiliary components, which overlooks both the 7,140 stage-by-stage execution records in Appendix B and the controlled fixed-solver ablations in Appendix C, and (2) an out-of-scope demand for a plot of "accuracy on the y axis and monetary/time cost on the y axis" [sic], a metric absent from prior work in this area and contradicted by our empirical cost data. Below we refute both claims across Part 1/2 and Part 2/2.1. Controlled Experiments and Long-Horizon Task Theory Refute the Claim That Auxiliary Components Drive Performance (Addressing Weakness 1)
The reviewer speculates from the text alone that "auxiliary contributions such as the task parser, repair constructor, action interface, and continuation policy may be making a bigger difference here," without checking the controlled ablations or execution logs in the paper. When all of these auxiliary elements are held strictly fixed, empirical data prove that our core source-tracing loop, multi-source composition, and dependency ordering drive the performance gains. Moreover, the sub-5% baseline recovery rate is the inevitable theoretical outcome of applying single-error attribution to long-horizon tasks with multiple interdependent prerequisites.
A. Direct empirical proof when holding all auxiliary elements strictly fixed In Section 4.3 (Tables 2 and 3, page 9, lines 433 to 466) and Appendix C (pages 21 to 24), we hold all auxiliary elements constant across two sets of controlled experiments:
1) Fixed-solver and fixed-auxiliary ablations across all three benchmarks (Table 2, page 9, and Appendices C.3 and C.6, pages 21 to 24): We hold the exact same task parser, exploration procedure, native repair constructor, action interface, continuation policy, and budgets fixed across ALFWorld, Habitat, and ScienceWorld, and ablate only our core mechanisms:
Removing multi-source composition or dependency ordering causes recovery to collapse by -79.1 to -95.5 percentage points in ALFWorld (two-sided exact McNemar p<10−25, Tables 14 and 15, page 21), -66.7 percentage points in Habitat (Holm-adjusted p=1.14×10−5, Tables 18 and 19, page 23), and -46.7 percentage points in ScienceWorld (p=4.88×10−4). Removing residual feedback iteration drops recovery by -30.0 percentage points in Habitat (p=0.00781) and -100.0 percentage points on two-stage cases (p=1.49×10−8, Table 16, page 22).
2) Controlled 2x2 mechanism experiment in Habitat (Table 3, page 9, and Appendix C.7, page 24): On 16 development cases in Habitat (Qwen2.5-7B), we hold the failed policy, action interface, continuation rule, and auxiliary recovery modules constant (sharing the exact same public-state compiler in the enhanced setting) while switching only our explicit source-tracing loop on or off:
Without the explicit source loop, multi-source composition, and dependency ordering of Source-MSIS, the auxiliary modules alone cannot recover these tasks. (Continued in Part 2/2.)