acceptodds
Under review as a conference paper at ICLR 2027

Multifaceted Guidance, Targeted Correction: Experience Harness for LLM Agents

Abstract

LLM agents can improve future behavior by reusing experience from past task executions. However, such reuse becomes challenging in realistic complex tasks involving sequentially dependent bottlenecks that differ in focus and are progressively revealed during execution. Consequently, generic reuse may overemphasize certain bottleneck focuses while overlooking others, or fail to provide timely intervention when errors emerge during execution. In this paper, we introduce Experience Harness, a specialized external control method that organizes experience from successful and failed trajectories and regulates how that experience guides and corrects an LLM agent. Specifically, it decomposes trajectories into bottleneck-aware atoms along four dimensions, inducing success skills for multifaceted task-start guidance and failure modes that trigger targeted corrective intervention during execution. This design preserves both the bottleneck focus of each experience and the stage at which it should take effect. Across multiple base agents on the AppWorld and BFCL-V3 benchmarks, our method consistently outperforms existing experience-memory baselines. Detailed experimental analyses further confirm the value of multifaceted guidance across bottleneck focuses and targeted correction at appropriate execution stages.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Experience Harness, Stage-Aware Experience

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Dx3NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Experience Harness to improve experience reuse in LLM agents. It extracts reusable skills from successful trajectories and failure patterns from failed ones across four bottleneck dimensions. Skills guide task execution from the start, while detected failure patterns trigger targeted corrections that are removed once resolved. Experiments on AppWorld and BFCL-V3 show consistent improvements over existing experience-memory baselines.

Critical strengths
  1. The mechanism for leveraging experience is fully designed. The successful experiences are used as guidance at the beginning of the task, the failed experiences are used for targeted error correction during the execution process, and the error correction prompts are removed after the problem is solved, balancing the reuse of experience with the flexibility of the execution stage.

  2. The experimental verification is quite thorough. Multiple basic models were evaluated on two benchmarks, AppWorld and BFCL-V3, and the effectiveness of components such as successful skills, failure patterns, and runtime detection was verified through ablation experiments.

Critical weaknesses
  1. The method initially divides the experience into four dimensions. However, the main bottlenecks of different tasks may vary, and a fixed classification system may not be suitable for new task types. This limits the flexibility and generalization ability of the method.
  2. Compared with MNL (https://aclanthology.org/2026.findings-acl.719.pdf), which clusters recurring failures into reusable error notes, Experience Harness additionally organizes failure modes by bottleneck dimensions and activates corrective guidance during execution. However, the paper does not directly compare these mechanisms with MNL, leaving their unique advantages insufficiently established. Moreover, although ReMe is included as a baseline, the paper does not separately quantify the contribution of bottleneck-aware decomposition, making it difficult to determine how much improvement comes specifically from this design.
Questions for authors
  1. The ablation experiments show that classifying experiences into four dimensions is indeed helpful, but it has not been proven why these four categories must be used. What if a different classification method is adopted, or if the model is allowed to discover the task bottlenecks on its own? Would the results be better?
  2. The failure mode analysis on AppWorld (Figure 4(b)) reveals that some correction suggestions actually reduce the performance of the weaker models. The authors need to further explain under what circumstances the corrections are likely to fail and how to minimize the impact of errors.
6 comments
AuthorsOpenReview ↗

Response to Reviewer Dx3N (1/2)

W1. Rationale and Flexibility of the Four-Dimension Organization:

Thank you for your valuable feedback.

Our four dimensions are inspired by Norman’s action cycle and cover key functions in task execution: understanding the goal, planning actions, using tools, and grounding evidence and outputs. We intend these dimensions to serve as an organization prior that covers essential processes in task execution, rather than an exhaustive taxonomy of all possible bottlenecks.

Our central claim is that experience should be extracted from complementary perspectives to reduce redundant emphasis and overlooked needs, while corrective experience should be introduced when execution reveals a relevant problem and withdrawn once it is resolved. The current four dimensions provide one concrete implementation of this approach. We acknowledge that these four dimensions may not be the only or the best way to organize experience. Exploring alternative classifications and allowing models to discover bottlenecks, including new bottleneck types, are therefore valuable directions for future work. We will clarify this scope and discuss these extensions in the paper.

W2. Relationship to MNL and Distinct Contributions:

MNL clusters failures by semantic subject into reusable mistake notes and accepts memory updates when batch performance improves. Our work instead emphasizes complementary experience across bottleneck dimensions and state-dependent activation and withdrawal of corrective experience during execution. The two methods share an interest in learning from failures but differ in how experience is organized and controlled.

Our evaluation already includes related methods, such as ReMe for learning from successful and failed trajectories and Skill-Pro for runtime interventions. These comparisons cover related experience-learning and runtime-use mechanisms.

We have added MNL to the related work and clarified these distinctions. (Revised manuscript: §2.)

W3. Contribution of Bottleneck-Aware Decomposition:

We added a direct-extraction ablation to examine the contribution of bottleneck-aware experience construction. (Revised manuscript: §5.5, Table 2; Appendix C, Table 6.) Using the same training trajectories, we compare two variants:

  • Bottleneck-aware construction: Our original pipeline constructs success skills and failure-mode experiences from trajectories through bottleneck-aware decomposition.

  • Direct extraction: The LLM directly extracts success experiences and failure-mode experiences from trajectories, without intermediate atoms or predefined bottleneck dimensions.

For each task, both variants use the same selection LLM to globally select and inject the same number of success experiences as the full method, without dimension labels or per-dimension quotas. The experiences are selected from the respective memories produced by the two construction methods. Both variants retain the same runtime detection, injection, and withdrawal mechanisms for failure-mode experiences. This comparison tests the contribution of experience construction while keeping the success-experience count, selection strategy, and runtime-use mechanisms consistent.

The additional AppWorld results are shown in Table 1:

Table 1. Experience-construction ablation on AppWorld Test-Normal under count-matched global selection.

ModelExperience constructionAvg@3 (%)Pass@3 (%)Score@3 (%)
Qwen3-32BBottleneck-aware construction33.9346.4262.40
Qwen3-32BDirect extraction32.3444.6460.20
Qwen3-235BBottleneck-aware construction55.9566.6780.90
Qwen3-235BDirect extraction54.3765.4879.20

Bottleneck-aware construction improves all three metrics on both models, with Avg@3 gains of 1.59 and 1.58 percentage points for Qwen3-32B and Qwen3-235B, respectively. These results support the additional value of bottleneck-aware decomposition and experience construction beyond the downstream dimension-aware selection mechanism.

AuthorsOpenReview ↗

Response to Reviewer Dx3N (2/2)

W4. Failure Cases of Runtime Corrections and Possible Mitigations:

Thank you for highlighting this issue.

We noticed this issue while analyzing trajectories during our experiments. In some Qwen3-32B trajectories, the agent focused excessively on local corrections while overlooking other requirements of the original task. The two modes with negative net improvement—Invalid API Call and Text-in-Code Error—both concern local execution correctness: API validity and executable-code syntax. These observations suggest that attention to local execution details can sometimes come at the expense of overall task completion.

Our current mechanism withdraws corrective experience once the corresponding failure mode is resolved, limiting its continued influence. To further address this issue, we propose prompting the agent after each injection to revisit the original goal and outstanding requirements before proceeding. We agree that this is an interesting problem and hope to investigate it further in future work.

Q1. Distinction from MNL The authors clarify the conceptual differences between Experience Harness and MNL, but the empirical advantages remain unclear. Could the authors further clarify what measurable benefits bottleneck-aware organization provides beyond MNL's semantic clustering and memory updates?

Q2. Runtime Correction Failures The authors propose prompting the agent to revisit the original task goal after each correction injection. Could the authors clarify whether this mitigation has been implemented and whether it helps reduce the negative effects of incorrect corrections?

Reviewer_mFAbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes "Experience Harness," a memory method for LLM agents. Training trajectories are broken into "atoms" along four fixed dimensions: task understanding, tool use, strategy planning, and evidence grounding/output. The atoms are grouped through an LLM-judged merge graph into success skills and failure modes. At test time, success skills are injected once at task start. Failure modes are activated and removed by an LLM detector that runs every K steps. Results on AppWorld (test-normal) and a 120-task BFCL-V3 subset show consistent Avg@3 gains over six memory baselines across four base models.

Critical strengths

Table 2 and Table 4 show the same pattern on three base models: removing success skills, failure modes, runtime detection, or dimension-aware selection each lowers Avg@3. The "w/o RT Detection" variant is especially useful. Injecting all failure modes from the start does worse than having no failure modes at all for Qwen3-32B (33.13 vs. 35.11). That is direct evidence that when experience is injected matters.

Critical weaknesses
  1. The evaluation can't separate transferable experience from benchmark-specific memorization, and the margins are small. AppWorld test-normal uses the same apps and APIs as the training split. Several of the learned failure modes are literal API facts. For example, "use order_id, not order_num, in FoodDelivery.get_order" and "Wrong Parameter Name" appear among the top-activated modes (Fig. 4b). Gains of that kind would mostly disappear on unseen apps, yet test-challenge is not evaluated. On top of that, the margin over the best baseline is often small: about 2 points Avg@3 for Qwen3-235B on AppWorld (roughly 3 tasks out of 168) and about 1 to 3 points on the 120-task BFCL subset. The results come from 3 runs with no standard deviation or significance test. Pass@3 is sometimes lower than Skill-Pro's (Qwen3-235B AppWorld, Kimi BFCL), so the abstract's "consistently outperforms" claim is too strong.
  2. The central motivation is asserted, not tested, and part of the supporting analysis is circular. The paper's framing (heterogeneous, sequentially dependent, progressively revealed bottlenecks) is the reason given for the four dimensions and for stage-aware use. But the only evidence that real tasks have this structure is Fig. 1 and one case study. The four dimensions are hand-picked, "inspired by" Norman's action cycle. Section 5.4 is meant to show the problem is alleviated, but it relies on GPT-5.5-annotated "aspects." The paper doesn't say whether those aspects line up with the method's own four dimensions; if they do, the metric favors the method by construction. The Aspect Diversity Ratio is also normalized by the number of injected experiences, so a method that injects fewer items (Table 6) scores higher regardless of coverage. Finally, the dimension-level ablation (Table 5) partly cuts against the story. Tool Use holds the most atoms and skills, yet removing it barely affects Qwen3-32B: Avg@3 drops from 36.70 to 35.80 and Score@3 actually rises. Without stronger evidence, the dimensional organization looks like a reasonable prompt-engineering choice rather than a validated principle.
Questions for authors
  1. Which model runs the extra LLM calls? That covers atom extraction, canonicalization, merge judging, skill selection, and the runtime detector. Is it always the base agent model, or a stronger fixed model? If it's a stronger model, the comparison with no-memory and some baselines is not like-for-like.
  2. Failure atoms come from up to 5 repair-guided retries per training task, judged with the official benchmark evaluator. Did the baselines get the same number of training rollouts and the same access to evaluator feedback? If not, part of the gain may simply come from more training signal.
6 comments
AuthorsOpenReview ↗

Response to Reviewer mFAb (1/3)

W1. Transferable Experience versus Benchmark-Specific API Memorization:

Thank you for your valuable feedback.

W1.1 Experience content.

Literal API facts are only part of our experience memory. For example, “Wrong Parameter Name” in Fig. 4(b) names an error category; its corrective experience in Fig. 17 instructs the agent to verify endpoint-specific parameter names rather than guess by analogy. Other experiences address verifying identity from sufficient evidence (Fig. 18), distinguishing item-level from group-level task objectives (Fig. 20), and collecting complete information before making changes (Fig. 21). These extend beyond memorizing specific API facts.

W1.2 Transfer to unseen applications.

We evaluated Qwen3-235B and GLM-4.6 on AppWorld Test-Challenge, directly reusing their previously constructed experience memories without rebuilding them for Challenge tasks. (Revised manuscript: Appendix I, Table 9.)

Table 1. AppWorld Test-Challenge results.

ModelMethodAvg@3 (%)Pass@3 (%)Score@3 (%)
Qwen3-235BNo Memory26.8641.2560.43
Qwen3-235BMetaFlowLLM29.4244.6061.45
Qwen3-235BSkill-Pro31.4945.3363.73
Qwen3-235BOurs35.7348.6865.36
GLM-4.6No Memory48.2057.7979.57
GLM-4.6MetaFlowLLM49.2459.7180.69
GLM-4.6Skill-Pro51.8860.4382.54
GLM-4.6Ours55.6462.5985.02

Our method outperforms both baselines across all three metrics on both models. Its Avg@3 gains over No Memory are 8.87/7.44 percentage points, and those over Skill-Pro are 4.24/3.76 points, for Qwen3-235B/GLM-4.6. These results support the usefulness of the learned experience beyond the familiar application setting of Test-Normal.

W2. Small Gains and Missing Variability Analysis:

We conducted three runs with Qwen3-235B, repeating memory construction and test-task execution in each run; No Memory repeated test-task execution only. Within each run, every test task was executed three times to compute Avg@3, Pass@3, and, for AppWorld, Score@3. AppWorld used fixed training and test sets, while BFCL used a newly sampled training/test split in each run, shared by all methods.

Table 2 reports mean ± sample standard deviation across three runs. (Revised manuscript: Appendix J, Table 10.)

Table 2. Performance across three runs with Qwen3-235B (mean ± sample standard deviation).

DatasetMethodAvg@3 (%)Pass@3 (%)Score@3 (%)
AppWorldNo Memory49.27 ± 1.0066.47 ± 0.9176.72 ± 0.54
AppWorldMetaFlowLLM54.76 ± 0.7267.66 ± 0.9180.16 ± 0.17
AppWorldSkill-Pro54.96 ± 1.0569.25 ± 0.9180.70 ± 1.04
AppWorldOurs58.73 ± 0.5370.24 ± 0.6083.32 ± 0.62
BFCLNo Memory49.26 ± 0.9860.83 ± 1.67—
BFCLMetaFlowLLM53.05 ± 0.7462.78 ± 0.48—
BFCLSkill-Pro53.70 ± 0.6963.89 ± 0.48—
BFCLOurs58.15 ± 0.5866.39 ± 0.48—

Our method achieves the highest mean on every reported metric and leads in Avg@3 in every run on both datasets. Its mean Avg@3 gains over Skill-Pro are approximately 3.8 and 4.4 percentage points, with standard deviations below 0.6 points. These results support the stability of the Avg@3 gains under repeated memory construction and test-task execution.

W3. Overly Broad “Consistently Outperforms” Claim

Thank you for pointing this out. We agree that “consistently outperforms” is too broad without specifying the metric. Our method achieves the highest Avg@3 across all evaluated models on both benchmarks, but does not lead in every Pass@3 comparison. We have revised the abstract to state that our method “consistently outperforms existing experience-memory baselines in Avg@3” and clarified the Pass@3 exceptions in the results discussion. (Revised manuscript: Abstract, §1, §5.2, and §6.)

AuthorsOpenReview ↗

Response to Reviewer mFAb (2/3)

W4. Design Motivation and Validation of Task-Relevant Experience Diversity:

W4.1 Design motivation and validation.

Our design aims to provide complementary experience for different task needs and timely correction as execution unfolds. The four dimensions, inspired by Norman’s action cycle, serve as an organization prior. To further validate the benefits of organizing experience by bottleneck dimension and providing corrections during execution, we added two controlled ablations:

  • Without dimension-aware selection: Using the same candidate pool and selection LLM, we remove dimension labels and per-dimension quotas and globally select the same number of success skills as the full method for each task. This tests dimension-aware selection while controlling the number of injected skills.

  • Without runtime detection: At task start, an LLM selects task-relevant failure-mode experiences from the same failure memory pool, matching the number injected by the full method during execution. These selected experiences are injected once at task start, without subsequent runtime detection, injection, or withdrawal. This tests runtime intervention against count-matched, task-relevant static guidance.

The additional AppWorld results are shown in Table 3. (Revised manuscript: §5.5, Table 2; Appendix C, Table 6.)

Table 3. Count-matched ablations on AppWorld Test-Normal.

ModelVariantAvg@3 (%)Pass@3 (%)Score@3 (%)
Qwen3-32BFull method36.7049.4063.25
Qwen3-32BCount-matched global skill selection33.9346.4262.40
Qwen3-32BCount-matched static failure selection34.1247.0261.50
Qwen3-235BFull method58.1369.6482.61
Qwen3-235BCount-matched global skill selection55.9566.6780.90
Qwen3-235BCount-matched static failure selection55.7567.8581.45

The full method performs better across all three metrics. Its Avg@3 gains over global skill selection and static failure selection are approximately 2.8/2.2 and 2.6/2.4 percentage points, respectively, for Qwen3-32B/235B. These results provide additional evidence that preserving bottleneck dimensions during selection and using execution context for corrective intervention improve task performance beyond memory-count differences.

W4.2 Aspect annotation.

GPT-5.5 identified aspects from the experience content without being given our four predefined bottleneck dimensions as annotation categories. Thus, the annotation did not require experiences to align with our taxonomy. We have clarified this procedure in the paper. (Revised manuscript: §5.4.)

W4.3 Meaning of Aspect Diversity Ratio.

Thank you for raising this concern. We would like to clarify that our goal is task-relevant complementary coverage, rather than maximizing the number of aspects for its own sake or ensuring exhaustive coverage. The ratio measures distinct relevant aspects per injected experience, serving as a proxy for complementarity and reduced redundancy. A higher ratio does not necessarily imply more aspects covered in absolute terms, and tasks requiring fewer experiences should not be encouraged to receive unnecessary ones. We have clarified the intended meaning and scope of this metric in the paper. (Revised manuscript: §5.4.)

W5. Limited Tool Use Gains and Metric Trade-offs Weaken the Evidence for Dimensional Organization:

We acknowledge that Tool Use provides a modest marginal benefit on Qwen3-32B: including it improves Avg@3 from 35.80 to 36.70, while Score@3 decreases from 63.75 to 63.25. Having the most atoms and skills does not imply that a dimension must contribute the largest performance gain.

The benefits can also differ across models and metrics. For Qwen3-235B, including Tool Use improves all three metrics. Our motivation is to provide complementary experience for different task needs; it does not require every dimension to yield equally large or uniformly positive gains across models and metrics.

We believe that the modest marginal benefit of Tool Use on Qwen3-32B does not, by itself, negate the broader benefits of dimension-aware selection. The count-matched ablations in Table 3 support these benefits: dimension-aware selection improves Avg@3 over global selection by approximately 2.8 and 2.2 percentage points for Qwen3-32B and Qwen3-235B, respectively. (Revised manuscript: §5.5, Table 2; Appendix C, Table 6.)

AuthorsOpenReview ↗

Response to Reviewer mFAb (3/3)

W6. Fairness of Auxiliary Models and Training Interaction Budgets:

W6.1 Models for auxiliary calls.

Atom extraction, canonicalization, merge judging, repair validation, skill selection, and runtime detection all use the same model as the agent performing the tasks. No stronger model is used for experience construction or deployment. (Revised manuscript: §5.1; Appendix A.3.)

W6.2 Training tasks and interaction budgets.

All methods use the same training tasks and the same maximum training interaction budget: one initial execution plus up to five additional retries or sampling attempts per task. We retain each method’s original trajectory-generation and experience-construction mechanisms. Thus, the repair retries used by our method fall within the shared budget rather than providing additional training attempts. These details have been added to the implementation appendix. (Revised manuscript: §5.1; Appendix A.3, Table 4.)

W6.3 Evaluator feedback.

All methods receive the same evaluator feedback, limited to whether the final task outcome is correct. Our method receives no additional evaluator information for repair or experience construction. (Revised manuscript: Appendix A.3.)

Reviewer_bCzpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Experience Harness, which organizes successful and failed trajectories into four predefined bottleneck dimensions, then induces reusable success skills and failure modes. Success skills provide initial guidance; failure modes trigger runtime corrections that are removed once resolved. Experiments on AppWorld Test Normal and a BFCL-V3 Base Multi-Turn subset report improved Avg@3 across multiple backbones, supplemented by component ablations and token-cost analyses.

Critical strengths

Consistent reported improvements across backbones. Tables 1 and 3 show Avg@3 improvements across four backbones and both datasets, including comparisons with runtime-intervention methods such as MetaFlowLLM and Skill-Pro. This provides meaningful evidence that the overall pipeline can improve experience reuse within the evaluated settings.

Critical weaknesses

The incremental contribution is insufficiently established.

  • The four dimensions and the division between initial guidance and runtime correction are manually specified. Meta-Harness style of work (https://arxiv.org/abs/2603.28052) offers a relevant alternative through automated optimization of memory, retrieval, and control logic. The paper does not establish whether its prescribed structure offers better efficiency or generalization under comparable optimization budgets.
  • ReasoningBank (https://arxiv.org/abs/2509.25140), which extracts multiple transferable memory items from both successful and failed trajectories, is neither discussed nor compared. This is a particularly relevant omission.
  • Table 2 does not cleanly isolate these design choices: removing runtime detection also removes contextual filtering by injecting all failure memories initially, while removing dimensions changes selection to global top-5. Timing, relevance, and memory quantity are therefore confounded. These comparisons do not sufficiently identify the proposed mechanism’s contribution.

The evaluation leaves generality, fairness, and reliability unresolved.

  • Experiments use older-generation datasets and models! AppWorld Normal, and only 120 BFCL-V3 Base test tasks. This does not establish whether the method remains useful for stronger contemporary models or tasks that more directly stress progressively revealed, dependent bottlenecks.
  • Following the old model part, many corrections concern basic tool-use mistakes. Their value relative to simply using a stronger model remains unclear without an accuracy–cost comparison.
  • Some gains are small—for example, GLM-4.6 improves BFCL Avg@3 by 1.11 percentage points over the strongest baseline. Three executions per task do not characterize variability from memory construction or dataset sampling. Without confidence intervals, decoding settings, and independent construction runs, the stability of the ranking is unclear.
Questions for authors
  • Do all methods receive comparable training trajectories and interaction budgets, including repair retries? Which models and decoding settings are used for extraction, validation, selection, and runtime detection?
4 comments
AuthorsOpenReview ↗

Response to Reviewer bCzp (1/2)

W1. Relationship to Meta-Harness and ReasoningBank:

Thank you for your suggestions!

We believe that Meta-Harness is orthogonal to our work. ReasoningBank is relevant, and our evaluation includes related competing methods.

As outlined in Introduction and §4, our work focuses on: (1) extracting and selecting experience across different bottleneck dimensions to reduce redundant emphasis and overlooked needs; and (2) managing whether intervention is needed, what role it should play, and when its influence should end.

Meta-Harness automatically optimizes harness code through outer-loop search. Our mechanism, which specifies how experience is organized and deployed during execution, is complementary and could be incorporated into this optimization.

ReasoningBank learns from successes and failures, but does not explicitly address complementary bottleneck coverage or dynamic memory injection and withdrawal during execution. We already compared against related methods, including ReMe for learning from successes and failures and Skill-Pro for runtime interventions. These comparisons cover related experience-learning and runtime-use mechanisms.

We have added a discussion of both Meta-Harness and ReasoningBank to the related work. (Revised manuscript: §2.)

W2. Confounded Ablation Comparisons:

Thank you for your valuable feedback. We added two ablation experiments for your concerns. (Revised manuscript: §5.5, Table 2; Appendix C, Table 6.)

  • Runtime detection: To isolate the effect of runtime detection from the loss of contextual filtering, we use an LLM to select task-relevant failure experiences at task start and match their number to the full method, rather than injecting all failure memories.

  • Dimension-aware selection: We separate dimension-aware selection from the change to a fixed memory count. Using the same candidate pool and selection LLM, we select the same number of skills as the full method, but globally without dimension labels or per-dimension quotas.

The additional AppWorld results are shown in Table 1:

Table 1. Count-matched ablations on AppWorld Test-Normal.

ModelVariantAvg@3 (%)Pass@3 (%)Score@3 (%)
Qwen3-32BFull method36.7049.4063.25
Qwen3-32BCount-matched static failure selection34.1247.0261.50
Qwen3-32BCount-matched global skill selection33.9346.4262.40
Qwen3-235BFull method58.1369.6482.61
Qwen3-235BCount-matched static failure selection55.7567.8581.45
Qwen3-235BCount-matched global skill selection55.9566.6780.90

The full method remains better across all three metrics. Its Avg@3 advantages are approximately 2.6/2.4 points over static failure selection and 2.8/2.2 points over global skill selection for 32B/235B.

These controls support the benefits of runtime detection and dimension-aware selection beyond differences in memory count.

W3. Evaluation Coverage and Generalization:

We extended our evaluation to AppWorld Test-Challenge, which includes unseen applications and more complex tasks, using Qwen3-235B and GLM-4.6 with two competitive runtime-intervention baselines. (Revised manuscript: Appendix I, Table 9.)

Table 2. AppWorld Test-Challenge results.

ModelMethodAvg@3 (%)Pass@3 (%)Score@3 (%)
Qwen3-235BNo Memory26.8641.2560.43
Qwen3-235BMetaFlowLLM29.4244.6061.45
Qwen3-235BSkill-Pro31.4945.3363.73
Qwen3-235BOurs35.7348.6865.36
GLM-4.6No Memory48.2057.7979.57
GLM-4.6MetaFlowLLM49.2459.7180.69
GLM-4.6Skill-Pro51.8860.4382.54
GLM-4.6Ours55.6462.5985.02

Our method outperforms both baselines across all three metrics on both models. Compared with Skill-Pro, it improves Avg@3 by 4.24/3.76 percentage points, Pass@3 by 3.35/2.16 points, and Score@3 by 1.63/2.48 points for Qwen3-235B/GLM-4.6. Its Avg@3 gains over No Memory are 8.87/7.44 points, respectively. These results extend the evidence beyond Test-Normal and support the usefulness of our approach across models under more challenging conditions involving unseen applications.

AuthorsOpenReview ↗

Response to Reviewer bCzp (2/2)

W4. Performance Stability and Accuracy–Cost Trade-offs:

W4.1 Performance stability.

We conducted three runs with Qwen3-235B, repeating memory construction and test-task execution in each run; No Memory repeated test-task execution only. Within each run, every test task was executed three times to compute Avg@3, Pass@3, and, for AppWorld, Score@3. We report the mean and sample standard deviation across the three runs. AppWorld used fixed training and test sets, while BFCL used a newly sampled training/test split in each run, shared by all methods.

Table 3 reports mean ± sample standard deviation across three runs. (Revised manuscript: Appendix J, Table 10.)

Table 3. Performance across three runs with Qwen3-235B (mean ± sample standard deviation).

DatasetMethodAvg@3 (%)Pass@3 (%)Score@3 (%)
AppWorldNo Memory49.27 ± 1.0066.47 ± 0.9176.72 ± 0.54
AppWorldMetaFlowLLM54.76 ± 0.7267.66 ± 0.9180.16 ± 0.17
AppWorldSkill-Pro54.96 ± 1.0569.25 ± 0.9180.70 ± 1.04
AppWorldOurs58.73 ± 0.5370.24 ± 0.6083.32 ± 0.62
BFCLNo Memory49.26 ± 0.9860.83 ± 1.67—
BFCLMetaFlowLLM53.05 ± 0.7462.78 ± 0.48—
BFCLSkill-Pro53.70 ± 0.6963.89 ± 0.48—
BFCLOurs58.15 ± 0.5866.39 ± 0.48—

Our method achieves the highest mean on every reported metric and leads in Avg@3 in every run on both datasets. Its mean Avg@3 gains over Skill-Pro are approximately 3.8 and 4.4 percentage points, with standard deviations below 0.6 points. These results support the stability of the Avg@3 gains under repeated memory construction and test-task execution.

W4.2 Accuracy–cost trade-offs.

Our evaluation spans multiple model sizes and families. With Qwen3-235B, experience reuse improves Avg@3 from 49.27 to 58.73 on Test-Normal, averaged across runs, and from 26.86 to 35.73 on Test-Challenge. Section 5.6 counts all inference LLM calls, including runtime detection, and reports token usage comparable to other memory methods with the same backbone. These results support the value of experience reuse with more capable models.

W5. Fairness and Transparency of the Experimental Protocol:

W5.1 Training tasks and interaction budgets.

All methods use the same training tasks and the same maximum training interaction budget: one initial execution plus up to five additional retries or sampling attempts per task. We retain each method’s original trajectory-generation and experience-construction mechanisms. (Revised manuscript: §5.1; Appendix A.3, Table 4.)

W5.2 Models for auxiliary calls.

Experience extraction, validation, selection, and runtime detection all use the same model as the agent performing the tasks. No stronger model is used for experience construction or deployment. (Revised manuscript: §5.1; Appendix A.3.)

W5.3 Decoding settings.

The model versions, sampling parameters, and context limits used in our experiments are listed in Table 4. These details have been added to the implementation appendix. (Revised manuscript: Appendix A.3, Table 5.)

Table 4. Model versions, decoding parameters, and context limits used in the experiments.

ModelTemperatureTop-pTop-kContext limit used
Qwen3-32B0.10.82032k tokens
Qwen3-235B-A22B-Instruct-25070.10.820256k tokens
GLM-4.60.10.820128k tokens
Kimi-K2-Instruct-09050.10.820128k tokens
AuthorsOpenReview ↗

Follow-up on Our Response and Revisions

Thank you for your constructive feedback on our paper. As the discussion period draws to a close, we would appreciate your thoughts on whether our responses and revisions address your concerns. In particular, we have:

  • Clarified the relationship to Meta-Harness and ReasoningBank, including their different emphases and the related mechanisms covered by our existing baselines, and added both works to §2.
  • Added two count-matched ablations addressing the memory-count and filtering confounds. The full method outperforms task-relevant static failure selection and global skill selection across all three metrics on both Qwen models (§5.5; Appendix C).
  • Expanded evaluation to AppWorld Test-Challenge with Qwen3-235B and GLM-4.6, where our method outperforms the compared baselines across all three metrics (Appendix I).
  • Added three-run stability experiments on AppWorld and BFCL, repeating memory construction and test-task execution, with three executions per test task within each run. We report mean ± standard deviation and observe consistent Avg@3 advantages across runs (Appendix J).
  • Clarified the experimental protocol, including training tasks, maximum interaction budgets, auxiliary-model usage, and decoding settings (§5.1; Appendix A.3).

If any points remain unresolved or require further clarification, we would greatly appreciate your feedback and the opportunity to respond before the discussion closes. Thank you again for your time and careful review.

AuthorsOpenReview ↗

Further Follow-up on Our Response

Thank you again for your time and consideration. We would appreciate any feedback on our responses, additional experiments, and revised manuscript. Are there any remaining concerns or further questions that we could help clarify? We would welcome the opportunity to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.