- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Experience Harness to improve experience reuse in LLM agents. It extracts reusable skills from successful trajectories and failure patterns from failed ones across four bottleneck dimensions. Skills guide task execution from the start, while detected failure patterns trigger targeted corrections that are removed once resolved. Experiments on AppWorld and BFCL-V3 show consistent improvements over existing experience-memory baselines.
Critical strengths
-
The mechanism for leveraging experience is fully designed. The successful experiences are used as guidance at the beginning of the task, the failed experiences are used for targeted error correction during the execution process, and the error correction prompts are removed after the problem is solved, balancing the reuse of experience with the flexibility of the execution stage.
-
The experimental verification is quite thorough. Multiple basic models were evaluated on two benchmarks, AppWorld and BFCL-V3, and the effectiveness of components such as successful skills, failure patterns, and runtime detection was verified through ablation experiments.
Critical weaknesses
- The method initially divides the experience into four dimensions. However, the main bottlenecks of different tasks may vary, and a fixed classification system may not be suitable for new task types. This limits the flexibility and generalization ability of the method.
- Compared with MNL (https://aclanthology.org/2026.findings-acl.719.pdf), which clusters recurring failures into reusable error notes, Experience Harness additionally organizes failure modes by bottleneck dimensions and activates corrective guidance during execution. However, the paper does not directly compare these mechanisms with MNL, leaving their unique advantages insufficiently established. Moreover, although ReMe is included as a baseline, the paper does not separately quantify the contribution of bottleneck-aware decomposition, making it difficult to determine how much improvement comes specifically from this design.
Questions for authors
- The ablation experiments show that classifying experiences into four dimensions is indeed helpful, but it has not been proven why these four categories must be used. What if a different classification method is adopted, or if the model is allowed to discover the task bottlenecks on its own? Would the results be better?
- The failure mode analysis on AppWorld (Figure 4(b)) reveals that some correction suggestions actually reduce the performance of the weaker models. The authors need to further explain under what circumstances the corrections are likely to fail and how to minimize the impact of errors.
6 comments
Response to Reviewer Dx3N (2/2)
W4. Failure Cases of Runtime Corrections and Possible Mitigations:
Thank you for highlighting this issue.
We noticed this issue while analyzing trajectories during our experiments. In some Qwen3-32B trajectories, the agent focused excessively on local corrections while overlooking other requirements of the original task. The two modes with negative net improvement—Invalid API Call and Text-in-Code Error—both concern local execution correctness: API validity and executable-code syntax. These observations suggest that attention to local execution details can sometimes come at the expense of overall task completion.
Our current mechanism withdraws corrective experience once the corresponding failure mode is resolved, limiting its continued influence. To further address this issue, we propose prompting the agent after each injection to revisit the original goal and outstanding requirements before proceeding. We agree that this is an interesting problem and hope to investigate it further in future work.
Q1. Distinction from MNL The authors clarify the conceptual differences between Experience Harness and MNL, but the empirical advantages remain unclear. Could the authors further clarify what measurable benefits bottleneck-aware organization provides beyond MNL's semantic clustering and memory updates?
Q2. Runtime Correction Failures The authors propose prompting the agent to revisit the original task goal after each correction injection. Could the authors clarify whether this mitigation has been implemented and whether it helps reduce the negative effects of incorrect corrections?
Response to Reviewer Dx3N (1/2)
W1. Rationale and Flexibility of the Four-Dimension Organization:
Thank you for your valuable feedback.
Our four dimensions are inspired by Norman’s action cycle and cover key functions in task execution: understanding the goal, planning actions, using tools, and grounding evidence and outputs. We intend these dimensions to serve as an organization prior that covers essential processes in task execution, rather than an exhaustive taxonomy of all possible bottlenecks.
Our central claim is that experience should be extracted from complementary perspectives to reduce redundant emphasis and overlooked needs, while corrective experience should be introduced when execution reveals a relevant problem and withdrawn once it is resolved. The current four dimensions provide one concrete implementation of this approach. We acknowledge that these four dimensions may not be the only or the best way to organize experience. Exploring alternative classifications and allowing models to discover bottlenecks, including new bottleneck types, are therefore valuable directions for future work. We will clarify this scope and discuss these extensions in the paper.
W2. Relationship to MNL and Distinct Contributions:
MNL clusters failures by semantic subject into reusable mistake notes and accepts memory updates when batch performance improves. Our work instead emphasizes complementary experience across bottleneck dimensions and state-dependent activation and withdrawal of corrective experience during execution. The two methods share an interest in learning from failures but differ in how experience is organized and controlled.
Our evaluation already includes related methods, such as ReMe for learning from successful and failed trajectories and Skill-Pro for runtime interventions. These comparisons cover related experience-learning and runtime-use mechanisms.
We have added MNL to the related work and clarified these distinctions. (Revised manuscript: §2.)
W3. Contribution of Bottleneck-Aware Decomposition:
We added a direct-extraction ablation to examine the contribution of bottleneck-aware experience construction. (Revised manuscript: §5.5, Table 2; Appendix C, Table 6.) Using the same training trajectories, we compare two variants:
Bottleneck-aware construction: Our original pipeline constructs success skills and failure-mode experiences from trajectories through bottleneck-aware decomposition.
Direct extraction: The LLM directly extracts success experiences and failure-mode experiences from trajectories, without intermediate atoms or predefined bottleneck dimensions.
For each task, both variants use the same selection LLM to globally select and inject the same number of success experiences as the full method, without dimension labels or per-dimension quotas. The experiences are selected from the respective memories produced by the two construction methods. Both variants retain the same runtime detection, injection, and withdrawal mechanisms for failure-mode experiences. This comparison tests the contribution of experience construction while keeping the success-experience count, selection strategy, and runtime-use mechanisms consistent.
The additional AppWorld results are shown in Table 1:
Table 1. Experience-construction ablation on AppWorld Test-Normal under count-matched global selection.
Bottleneck-aware construction improves all three metrics on both models, with Avg@3 gains of 1.59 and 1.58 percentage points for Qwen3-32B and Qwen3-235B, respectively. These results support the additional value of bottleneck-aware decomposition and experience construction beyond the downstream dimension-aware selection mechanism.