acceptodds
Under review as a conference paper at ICLR 2027

ExpHarness: Model-Agnostic Experience Learning through a Trainable Harness

Abstract

Large language model (LLM) agents increasingly operate within a harness, the scaffolding that determines what enters the executor’s context, yet the experi- ence they accumulate across tasks rarely flows back into this harness. Existing approaches include executor fine-tuning and external memory retrieval, but combin- ing task-adaptive retrieval with experience reuse across frozen executors remains challenging. To this end, we introduce ExpHarness, a learnable experience harness that improves frozen and replaceable LLM executors without modifying their parameters. Specifically, ExpHarness distills trajectories into reusable skills and failure lessons within a self-evolving experience graph, and trains a lightweight retrieval copilot that decides, per task, how broadly to explore the graph and how strongly to favor historically useful experiences over merely similar ones. The copilot is optimized with reinforcement learning from a utility-grounded reward combining the with/without-experience score difference and an absolute- performance term; the same reward updates the graph during training. Extensive experiments on ExpSuite, spanning 10 static benchmarks and 2 agentic environ- ments, show that ExpHarness improves over the strongest baseline by 12.1% and 4.5% on static tasks and by 21.4% and 12.7% on agentic tasks with the smaller and larger executors, respectively, while reducing interaction steps by up to 21.6%. Transfer experiments further examine reuse of the learned harness across executors of different scales and reasoning capabilities, with joint graph and copilot transfer performing closest to target-specific training among the evaluated transfer variants.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Harness, Experience Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zsMaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ExpHarness moves experience learning for LLM agents from the executor to a trainable harness. The system has two components. An experience graph distills historical trajectories into skills and lessons, links related entries, and is searched by semantic seeding, personalized PageRank diffusion, and a UCB-style utility-aware ranking. A retrieval copilot, a lightweight LLM trained with PPO, outputs two per-task controls: the locality of graph diffusion and the trade-off between semantic relevance and historical utility. The reward combines the executor's score difference with and without retrieved experience and an absolute-performance term, and the same signal updates the utility estimates of retrieved nodes through an EMA, so the graph and copilot co-evolve while the executor stays frozen.

Critical strengths
  1. The problem positioning is practically meaningful and the evaluation is broad. A frozen, replaceable executor with a portable harness matches how agents are built on closed or rapidly evolving models, and the paper provides a unified learning interface for this setting. Twelve baselines, ten static benchmarks, two agentic environments, three transfer directions, and four ablations show consistent gains across settings, with interaction steps reduced in the agentic environments at the same time.
  2. The calibration of claims is above the norm for this area. The analysis explicitly notes how relative improvements are computed, that the Flat Experience ablation conflates two changes, that radar-chart axes differ so areas are not comparable, and that model identity and reasoning capability change together. This transparency increases confidence in the reported comparisons.
Critical weaknesses
  1. There is no statistical uncertainty reporting anywhere in the paper. Across 66 pages there is not a single standard deviation, seed count, or significance test. GPQA-Diamond has 198 questions, and many benchmark margins are a few points, so a number of gaps in Tables 2 and 3 cannot be distinguished from noise. The abstract states relative improvements in percent (12.1%, 21.4%), which, without variance information, can inflate the impression of the actual effect size.
  2. The reward and the necessity of the training machinery are under-justified. First, with η = 1 the reward r = (s_with - s_without) + η·s_with stays positive even when retrieval changes nothing, so the copilot is partly rewarded for the executor's own ability; the paper acknowledges this in Section 3.4 but provides no ablation on η, including the pure counterfactual difference at η = 0. Second, the same set-level reward is assigned to all retrieved nodes and folded into an EMA, which is coarse credit assignment (also acknowledged), and its interaction with the UCB exploration bonus is not analyzed. Third, the copilot is a 3B model trained with PPO and a critic to emit two integers; a contextual bandit, a small MLP, or even a grid search over fixed (R, W) could be equivalent alternatives. Without such controls the necessity of RL is not established, and the training cost (two executor calls per task plus PPO) cannot be judged reasonable.
Questions for authors
  1. In the static setting, is the experience graph trained and evaluated per benchmark, or shared across the ten benchmarks? If per benchmark, the static gains measure in-benchmark adaptation; please add cross-task transfer (e.g., graph trained on GSM8K, evaluated on MATH) and discuss potential contamination between executor pretraining memory and public test sets.
  2. Are the results single runs? Please report mean and standard deviation of the weighted averages over at least three seeds, especially for small test sets such as GPQA-Diamond.
  3. Please compare against lightweight controllers: how far do a two-layer MLP, a contextual bandit, or a grid search over fixed (R, W) get? If they match the trained copilot, the PPO machinery is not justified.
7 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the thorough assessment and valuable suggestions on statistical reliability, reward design, experience graph learning, and the necessity of the proposed retrieval controller. We have substantially strengthened our evaluation with multi-seed experiments, additional component-level ablations, and comparisons against lightweight retrieval controllers. We address these concerns in detail below.

[W1, Q2]: Statistical uncertainty, effect sizes, and robustness across random seeds

W1: Missing statistical uncertainty reporting. There is no statistical uncertainty reporting anywhere in the paper. Across 66 pages there is not a single standard deviation, seed count, or significance test. GPQA-Diamond has 198 questions, and many benchmark margins are a few points, so a number of gaps in Tables 2 and 3 cannot be distinguished from noise. The abstract states relative improvements in percent (12.1%, 21.4%), which, without variance information, can inflate the impression of the actual effect size.

Q2: Statistical significance and multiple seeds. Are the results single runs? Please report mean and standard deviation of the weighted averages over at least three seeds, especially for small test sets such as GPQA-Diamond.

Response. Thank you for raising this concern. Tables 2 and 3 in §4.1 report a single run. We now report mean ± standard deviation across three seeds for every method’s benchmark scores and test-instance-weighted average scores, in Appendix I, “Statistical Uncertainty Across Three Seeds”. Table 110 reports ExpSuite-Static, and Table 111 reports ExpSuite-Agentic. For GPQA-Diamond, where our test split contains 60 questions and one answer changes accuracy by 1.67 percentage points (Appendix E, Table 8), ExpHarness scores 26.67 ± 1.67 and 29.44 ± 0.96 with the small and large executors, respectively. Due to space constraints, we show the results for only a subset of baselines in the tables below. The complete results are provided in the Appendix: Table 110 reports ExpSuite-Static, and Table 111 reports ExpSuite-Agentic.

Effect sizes. For the single run summarized in the abstract, the relative improvements of 12.1% and 4.5% on ExpSuite-Static correspond to absolute gains of 7.51 and 3.42 percentage points for the small and large executors. On ExpSuite-Agentic, the corresponding relative improvements of 21.4% and 12.7% represent 9.4 and 7.0 percentage points.

Statistical tests. The final column of each appendix table compares each baseline’s weighted score with ExpHarness at the same executor size. We use two-sided Welch’s tt-tests, treating the three run-level scores per method as independent. The tests are exploratory: the pp-values are computed from the reported means and sample SDs and are unadjusted for multiple comparisons. We interpret individual benchmark results descriptively.

Conclusion. Across three seeds, ExpHarness retains a statistically significant gain in weighted score over the highest-scoring baseline in all four suite–executor settings at the nominal 0.05 level. On ExpSuite-Static, the gains are 7.76 percentage points over MemRL for the small executor (p=0.0003p=0.0003) and 4.02 points over S3 for the large executor (p=0.0112p=0.0112). On ExpSuite-Agentic, the gains over S3 are 9.1 points for the small executor (p=0.0003p=0.0003) and 6.5 points for the large executor (p=0.0054p=0.0054).

AuthorsOpenReview ↗

Response 2

Continuation of our response to [W1, Q2] from the previous page:

Table 110 (abridged). ExpSuite-Static results across all ten benchmarks. For each executor size, we show ExpHarness and the two baselines with the highest weighted average score. Scores are percentages reported as mean ± SD across three seeds. The weighted average uses the number of test questions in each benchmark. For all methods, see Appendix I, Table 110 of the revised manuscript.

ExecutorMethodARC-CCSQAGPQA-DMMLUOBQAGSM8KGSM-SymMATHHumanEval+MBPP+Weighted Avg.pp vs. ExpHarness (weighted avg.)
SmallMemRL67.56±1.6860.15±1.1225.56±1.9239.33±0.8967.26±0.1384.67±1.5468.30±0.9354.74±2.0646.15±2.5657.08±1.9162.12±0.780.0003
SmallS364.81±0.7860.00±2.7822.22±0.9643.19±1.8163.85±0.6882.52±0.5674.67±0.5953.19±1.4547.86±3.9255.00±1.2562.06±1.000.0006
SmallExpHarness74.37±2.1463.70±0.5626.67±1.6760.30±1.1475.33±1.3384.67±0.5981.78±1.2457.26±0.2656.41±2.5662.50±1.2569.88±0.84—
LargeS382.81±1.2274.96±1.3426.67±1.6765.04±0.7181.19±0.8489.78±1.9286.52±2.1653.56±0.8058.97±2.5672.92±1.9175.09±1.100.0112
LargeIRCoT83.70±1.2275.70±0.4630.00±1.6767.04±0.1382.96±1.1088.67±1.3383.78±2.1252.30±1.1450.43±1.4871.25±2.5075.05±0.830.0087
LargeExpHarness86.15±0.6874.81±0.7129.44±0.9669.04±0.9087.11±2.7394.52±0.6890.52±1.9459.04±1.6170.09±2.9679.17±1.9179.11±1.11—

Table 111 (abridged). ExpSuite-Agentic results across all four evaluation splits. For each executor size, we show ExpHarness and the two baselines with the highest weighted average score. Values are mean ± SD across three seeds. ALFWorld reports success rate (SR, %), and AppWorld reports pass rate (PR, %). Average scores and interaction steps are weighted by the number of test tasks. For all methods, see Appendix I, Table 111 of the revised manuscript.

ExecutorMethodALF-Seen SRStepsALF-Unseen SRStepsAppWorld Test-N PRStepsAppWorld Test-C PRStepsAvg. Score ↑Avg. Steps ↓pp vs. ExpHarness (avg. score)
SmallS360.0±1.423.5±1.042.8±2.621.4±1.054.0±1.29.3±1.535.3±1.315.4±1.644.1±0.916.4±1.20.0003
SmallIRCoT34.3±0.738.0±1.327.1±0.939.2±1.853.6±0.612.3±1.236.1±2.217.5±1.637.8±1.323.2±1.40.0001
SmallExpHarness69.8±1.120.3±1.775.4±1.518.1±0.756.7±0.710.0±1.939.2±1.015.0±1.453.2±1.015.4±1.4—
LargeS377.4±1.121.9±1.880.6±0.721.4±0.752.0±2.817.6±1.441.4±1.321.7±0.755.5±1.420.9±1.00.0054
LargeSearch-o175.7±1.222.0±0.878.4±0.723.5±1.351.8±1.021.1±1.939.7±1.124.5±1.354.0±0.723.3±1.30.0044
LargeExpHarness85.2±2.518.7±2.288.1±3.016.4±1.156.5±1.013.1±1.348.0±1.014.2±0.762.0±1.515.0±0.9—
AuthorsOpenReview ↗

Response 3

W2: Under-justified reward design.

The reward and the necessity of the training machinery are under-justified. First, with η = 1 the reward r = (s_with - s_without) + η·s_with stays positive even when retrieval changes nothing, so the copilot is partly rewarded for the executor's own ability; the paper acknowledges this in Section 3.4 but provides no ablation on η, including the pure counterfactual difference at η = 0. Second, the same set-level reward is assigned to all retrieved nodes and folded into an EMA, which is coarse credit assignment (also acknowledged), and its interaction with the UCB exploration bonus is not analyzed.

Response. Thank you for raising these questions. We address the choice of training reward in §4.3, Table 5 and the contribution of the node-utility update and exploration bonus in the new Appendix K, “Node Utility Updates and UCB Exploration,” Table 113.

Contribution of the two reward terms

We agree that the full reward gives some credit when the executor already knows the answer. With binary scores and η=1\eta=1, a correct answer both with and without experience earns a reward of 1. Turning an incorrect answer into a correct one earns 2. For a given input, swithouts_{\mathrm{without}} is constant across retrieval choices. Across inputs, it distinguishes these two kinds of success. That distinction also enters the graph’s utility estimates and can influence future retrieval.

Setting. We vary only the training reward in §3.4. Each run starts from the same cold-start experience graph and Qwen2.5-3B-Instruct retrieval copilot, then independently trains the copilot and updates its graph. We compare Relative only (swith−swithouts_{\mathrm{with}}-s_{\mathrm{without}}, η=0\eta=0), Absolute only (swiths_{\mathrm{with}}), and the Full reward ((swith−swithout)+swith(s_{\mathrm{with}}-s_{\mathrm{without}})+s_{\mathrm{with}}, η=1\eta=1).

Table 5. Test scores by training reward (§4.3).

ExecutorTraining rewardQAReasoningCodingALFWorldAppWorld
SmallRelative only65.3271.7857.1469.3441.37
SmallAbsolute only66.0273.7059.6671.1743.42
SmallFull reward66.4574.6760.5072.6344.44
LargeRelative only75.9179.2672.2783.9448.72
LargeAbsolute only76.7780.4473.1185.0449.40
LargeFull reward77.3781.0474.7986.5050.94

Results. The full reward scores highest in all five domains at both executor sizes. It improves on the pure counterfactual reward by 1.13–3.36 points and on the absolute-only reward by 0.43–1.68 points. These gains measure the joint effect on the trained copilot and graph. The full-versus-absolute comparison changes both the coefficient of swiths_{\mathrm{with}} and the inclusion of swithouts_{\mathrm{without}}, so the observed difference reflects both changes.

Shared node utility and UCB exploration

The executor uses the retrieved experiences together to produce one answer or trajectory. Several nodes can supply complementary information, while the observed reward applies to the set as a whole. Our update draws on the idea in cooperative reinforcement learning that a shared outcome can provide useful feedback when individual contributions are difficult to observe [1]. We update each retrieved node’s utility using an exponential moving average (EMA) of the set reward, uv←(1−β)uv+βru_v \leftarrow (1-\beta)u_v+\beta r. Thus, uvu_v tracks recent rewards for retrieval sets containing node vv.

Our original w/o Utility Ranking ablation in §4.3, Figure 3 ranked candidates by semantic similarity alone, removing both historical utility and UCB exploration from ranking. Following the reviewer’s suggestion, we added two ablations that remove each component separately. The EMA records outcomes for previously retrieved nodes. The UCB bonus gives less frequently retrieved nodes more opportunities to receive feedback.

Setting. w/o Utility Update fixes uv=0u_v=0 for every node and retains the UCB bonus (c=1c=1). w/o UCB retains the EMA utility update (β=0.1\beta=0.1) and sets c=0c=0. ExpHarness uses both components (β=0.1\beta=0.1, c=1c=1).

Table 113. Ablations of the EMA utility update and UCB exploration bonus (Appendix K).

ExecutorConfigurationQAReasoningCodingALFWorldAppWorld
Smallw/o Utility Update64.2570.4457.1467.5240.34
Smallw/o UCB65.0572.9658.8270.4443.08
SmallExpHarness66.4574.6760.5072.6344.44
Largew/o Utility Update74.3077.4169.7581.0245.98
Largew/o UCB76.2479.7873.1184.3148.72
LargeExpHarness77.3781.0474.7986.5050.94
Reviewer_ujBMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ExpHarness, a trainable system for retrieving previous experiences to be used as few-shot examples. Past experiences/trajectories are summarized, successful runs become "skills", failed runs become "lessons". These are stored as nodes in a graph, with nearest neighbours linked with edges based on embedding similarity. A separate model, trained with RL, predicts two parameters that control the behaviour and ranking of the algorithm that retrieves memories from this graph. Evaluation is performed on ExpSuite-Static and ExpSuite-Agentic benchmarks, showing relative improvements over the baselines.

Critical strengths

The paper makes a strong emirical contribution: The premise is interesting, to develop a trainable retrieval algorithm that would balance relevance and usefulness. The experimental improvements over the provided baseline numbers are noteworthy. Evaluation on 10 static benchmarks and two agentic environments is good to see.

Critical weaknesses

Some open questions and concerns remain:

  1. The reported baseline numbers seem suspiciously low. For example, the official Llama-3.1-8B-Instruct model card reports performance on ARC-C (83.4), MMLU (69.4-73.0), GSM-8K (84.5), MATH (51.9). The "no memory" numbers reported for the same model in this paper are much lower: ARC-C (70.22), MMLU (53.11), GSM-8K (70.89), MATH (39.78). The difference is considerable and concerning. If the baseline models are poorly prompted, then it may be trivial to achieve relative improvements over them, either by fixing the prompt or showing any examples of the required format.

  2. The appendix seems to show that the baselines are prompted differently compared to the ExpHarness model. For example, for ExpSuite the executor template in Table 13 requires the format in <answer> tags, whereas the prompt for No Memory asks for "#### [number]" format instead. This introduces unnecessary confounding factors and it is difficult to determine where the performance differences are coming from. At minimum, there should be a baseline configuration included that runs ExpHarness, using the ExpHarness prompts, but with an empty experience memory.

  3. The data use and splitting is not sufficiently specified. What data was used for generating the examples? What data was used for training the copilot? What data was used for tuning the hyperparameters? Is the copilot training all shared between the different benchmarks or done separately for each benchmark? And is the memory bank shared between benchmarks or separate from each of them? And are these data choices done the same way for all the baselines?

  4. There are some ablation experiments, which is great to see. But it is not fully clear which specific components these ablate, as they seem to be affecting many at the same time. The introduced model is quite intricate, therefore ablation experiments would be needed that ablate individual model choices to test their actual benefit. The paper also needs a baseline that replaces the proposed pagerank-based graph retrieval with a simple kNN that retrieves stored experiences as few-shot examples.

  5. The main premise of the paper is that training a task-adaptive retrieval policy is important, but the trained copilot is never ablated to show that is the case. A baseline for demonstrating that could use fixed values for R and W tuned along with the other hyperparameters.

  6. The method includes quite a few hyperparameters. What data were these tuned on? Did that include results on the test data? And was similar tuning performed for the baselines?


Update after author response:

I disagree that showing the baselines zero few-shot examples (which results in substantially lower performance compared to the official performance of these models) is "fair". The proposed method is also using few-shot examples, the novelty part is about the selection of these examples. Please include baseline results with few-shot examples in the final version. You can always just report both (zero-shot and few-shot).

The new provided results strengthen the paper. I do feel that this is an exceptional amount of new experiments introduced during the author response period, many of which should have been included in the original paper. Nevertheless, since this seems to be currently allowed I am increasing my score.

Questions for authors

Please see above.

9 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the detailed and constructive feedback on baseline performance, prompting consistency, data splitting, experimental fairness, and component-level ablations. We have carefully addressed these concerns through additional controlled experiments, more comprehensive ablation studies, and further clarification of our experimental protocols, as detailed below.

W1: Suspiciously low baseline performance.

The reported baseline numbers seem suspiciously low. For example, the official Llama-3.1-8B-Instruct model card reports performance on ARC-C (83.4), MMLU (69.4-73.0), GSM-8K (84.5), MATH (51.9). The "no memory" numbers reported for the same model in this paper are much lower: ARC-C (70.22), MMLU (53.11), GSM-8K (70.89), MATH (39.78). The difference is considerable and concerning. If the baseline models are poorly prompted, then it may be trivial to achieve relative improvements over them, either by fixing the prompt or showing any examples of the required format.

Response. Thank you for raising this concern. Two differences are important when interpreting the gap between our No-Memory scores and the model-card results. First, to keep the benchmarks comparable in size within ExpSuite-Static, we sample 1,500 instances from each of the four benchmarks and use the same 5:2:3 train/validation/test split. Each reported score therefore comes from our own 450 held-out questions, whose composition can differ from that of the official evaluation set (Appendix E, Dataset Statistics). Second, our evaluation uses no manually curated few-shot demonstrations. By comparison, Meta reports GSM8K at 84.5 with eight worked examples and MMLU at 69.4 with five. Meta also reports zero-shot results for ARC-C, MATH, and MMLU with chain-of-thought, so the absence of few-shot examples explains a protocol difference for some comparisons, but cannot by itself account for all four gaps.

We chose this protocol because our research question is how effectively different methods use memory from past interactions. Manually supplying worked examples would introduce another source of experience: an executor could benefit from those examples, from retrieved memory, or from their overlap. For a fair comparison, we apply the same rule to No-Memory, every memory baseline, and ExpHarness. Each memory method supplies the experience selected by its own mechanism.

We respectfully disagree that the lower scores, by themselves, establish that the baseline was poorly prompted. The No-Memory prompts explicitly ask the executor to reason step by step. For multiple-choice questions, the prompt asks it to analyze each option and shows the final-answer format <answer> A </answer>. For math, it asks the executor to show its calculations and gives #### 42 as a formatting example (Appendix F, Executor and Baseline Answering Prompts). The executor therefore sees the required format even in the No-Memory condition. In the prompt-matched control reported under W2 (Appendix F.3, Table 15), switching between the two No-Memory prompts changes the ExpSuite-Static weighted average by less than one point for either executor. With the prompt held fixed, adding retrieved experience improves the weighted average by 18.38 points for the small executor and 17.60 points for the large executor.

AuthorsOpenReview ↗

Response 2

W2: Inconsistent prompting between baselines and ExpHarness.

The appendix seems to show that the baselines are prompted differently compared to the ExpHarness model. For example, for ExpSuite the executor template in Table 13 requires the format in <answer> tags, whereas the prompt for No Memory asks for "#### [number]" format instead. This introduces unnecessary confounding factors and it is difficult to determine where the performance differences are coming from. At minimum, there should be a baseline configuration included that runs ExpHarness, using the ExpHarness prompts, but with an empty experience memory.

Response. Thank you for pointing out this difference in the executor prompts. As discussed in our response to W1, No-Memory (original) includes reasoning instructions and an example of the required answer format. For mathematical reasoning, its executor prompt asks for the final answer after ####, while the ExpHarness executor prompt asks for it inside <answer>...</answer>. We agree that the original comparison leaves the effect of this difference unresolved.

We added No-Memory (unified prompt), the empty-memory control requested by the reviewer. On every ExpSuite-Static test item, No-Memory (unified prompt) and ExpHarness use the same frozen executor, executor prompt, and decoding configuration. The only difference in the executor input is the retrieved-experience field: No-Memory (unified prompt) leaves it empty, and ExpHarness populates it with retrieved experience.

Table 1. Prompt-matched empty-memory control on ExpSuite-Static. Small and large denote Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct, respectively. The average is weighted by the number of test items in each benchmark.

ExecutorMethodARC-CCSQAGPQA-DMMLUOBQAGSM8KGSM-SymMATHHumanEval+MBPP+Weighted avg.
SmallNo-Memory (original)51.5654.4418.3342.8954.2269.5660.4437.5643.5938.7551.88
SmallNo-Memory (unified prompt)50.8954.0018.3342.2253.7868.4459.7836.6743.5937.5051.19
SmallExpHarness74.0063.1126.6760.0074.0084.8982.0057.1156.4162.5069.57
LargeNo-Memory (original)70.2263.1123.3353.1169.7870.8962.6739.7843.5958.7560.41
LargeNo-Memory (unified prompt)70.6763.7823.3354.0070.4472.0063.3340.6743.5960.0061.16
LargeExpHarness86.0075.1128.3368.8986.0095.1190.0058.0066.6778.7578.76

Switching from No-Memory (original) to No-Memory (unified prompt) changes the weighted average by −0.69 points for the small executor and +0.75 points for the large executor. With the executor prompt held fixed, adding retrieved experience improves the weighted average by 18.38 and 17.60 points, respectively, with gains on all ten benchmarks for both executors. These results show that the choice between the two tested prompts has little effect on the weighted average, while access to retrieved experience accounts for the large improvement observed with ExpHarness. We report the full results in Appendix F.3, “Executor and Baseline Answering Prompts,” Table 15.

AuthorsOpenReview ↗

Response 3

[W3, W6]: Data splits, memory sharing, hyperparameter selection, and potential test-data leakage

W3: Insufficient specification of data use and splitting.

The data use and splitting is not sufficiently specified. What data was used for generating the examples? What data was used for training the copilot? What data was used for tuning the hyperparameters? Is the copilot training all shared between the different benchmarks or done separately for each benchmark? And is the memory bank shared between benchmarks or separate from each of them? And are these data choices done the same way for all the baselines?

W6: Hyperparameter tuning and potential test-data leakage.

The method includes quite a few hyperparameters. What data were these tuned on? Did that include results on the test data? And was similar tuning performed for the baselines?

Response. Thank you for asking us to clarify how we use the data and which components are shared across benchmarks. We have added these details to Appendix E (Dataset Statistics).

Experience generation and copilot training. To initialize the experience graph, we run the frozen executor on a subset of training tasks with an empty memory. We summarize high-scoring trajectories into skills and low-scoring trajectories into failure lessons. We then train the retrieval copilot with PPO on training tasks and update the graph using the resulting trajectories. For final evaluation, we freeze the selected copilot checkpoint and its corresponding graph.

Sharing across benchmarks and executor sizes. We train a separate retrieval copilot and maintain a separate experience graph for each executor size. Within each size, all ten ExpSuite-Static benchmarks share one copilot and one graph. In ExpSuite-Agentic, ALFWorld has its own copilot and graph shared across Seen and Unseen. AppWorld has its own copilot and graph shared across Test-N and Test-C.

Hyperparameters and checkpoint selection. In the main comparison, we matched the number of training updates and rollouts wherever applicable. For ExpHarness and all baselines, we used the remaining method-specific defaults without tuning them on validation or test results. For ExpSuite-Static, we use the validation split in Table 8 (a 5:2:3 train/validation/test split). For ExpSuite-Agentic, we hold out 20% of the ALFWorld training environments and 20% of the AppWorld training tasks for validation. We use validation performance to select trainable checkpoints and the fixed retrieval parameters in §4.3. We reserve the test splits for final evaluation (Appendix E).

Baseline memory construction. All methods use the same task splits. Where applicable, retrieval-centric baselines receive the same training trajectories as ExpHarness. Each baseline builds its memory bank using its default procedure because memory construction is part of the baseline method. For a fair comparison, ExpHarness and every memory-based baseline use the same upper limit on the number of experiences they can store.

Reviewer_YTFwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

They proposes ExpHarness, a framework that improves frozen LLM agents by learning an external experience harness. It builds an experience graph from past trajectories and trains a retrieval copilot to select useful experiences based on task relevance and historical utility. The authors evaluate the method on 10 static benchmarks and two agentic environments, showing improvements. They also study transfer across different models.

Critical strengths
  1. good experimental evaluation. The paper evaluates the method on a wide range of benchmarks and different executor models. The performance improvements are consistent.

  2. Interesting transfer experiments. The authors study transferring the experience graph and retrieval copilot across different executors. These experiments are useful for understanding the potential of model-agnostic experience learning.

Critical weaknesses
  1. algorithmic novelty. The method mainly combines existing techniques, including graph-based memory, personalized PageRank, utility-aware retrieval, and PPO. The retrieval copilot essentially learns two retrieval hyperparameters. While the combination is useful, I think the technical novelty is relatively limited.

  2. important mechanism ablations. The paper does not directly compare the trained copilot against fixed or tuned retrieval parameters. Since the no-experience score is independent of the retrieval choice for a given task, its additional contribution to learning is unclear.

Questions for authors
  1. Could you compare with fixed or tuned retrieval parameters, and ablate the different reward terms?
  2. Could yous provide a more complete cost comparison, including copilot training, retrieval overhead, and executor calls?
5 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the thoughtful assessment and constructive suggestions on algorithmic novelty, retrieval parameter optimization, reward design, and computational efficiency. We address these concerns below and provide additional experiments and analyses to further clarify the methodological contributions and practical effectiveness of ExpHarness.

W1: Limited algorithmic novelty.

The method mainly combines existing techniques, including graph-based memory, personalized PageRank, utility-aware retrieval, and PPO. The retrieval copilot essentially learns two retrieval hyperparameters. While the combination is useful, I think the technical novelty is relatively limited.

Response. Thank you for recognizing the practical value of ExpHarness and for raising the question of algorithmic novelty. We appreciate the opportunity to clarify what ExpHarness contributes to experience learning for LLM agents.

Many of these works we compared and discussed in our paper have appeared at leading conferences and helped shape research on how agents learn from experience. They creatively combine established techniques such as retrieval (Zhao et al., 2024), reinforcement learning (Zhang et al., 2026), and episodic memory (Shinn et al., 2023). We believe their impact illustrates a widely recognized form of innovation in agent experience learning: identifying a problem and showing how existing techniques, thoughtfully adapted to it, can give agents a new capability.

ExpHarness addresses a gap between finding experiences that resemble the current task and finding experiences that help the executor solve it. For example, a lesson from a past failure may apply to the current task even when the two tasks are worded differently. Its historical utility matters only if the retriever can find it. As Table 1 shows, what remains unresolved across these methods is how to use task outcomes to learn when to search beyond the nearest semantic matches and which of the newly reached experiences to trust. A useful lesson can be missed before ranking even begins, while broader search can introduce experiences that are related but unhelpful. ExpHarness addresses both problems together: for each input, the copilot controls graph expansion and the weight given to historical utility, and the same execution feedback updates both the retrieval policy and the utility estimates of the selected experiences.

Compared with the strongest method in each setting, ExpHarness achieves relative gains of 12.1% and 4.5% on ExpSuite-Static and 21.4% and 12.7% on ExpSuite-Agentic for the small and large executors, respectively (§4.1). In small-to-large transfer, transferring the graph and copilot together outperforms transferring either alone (§4.2). All four ablated variants perform below the full method across the evaluated domains, supporting the value of each tested design choice (§4.3). Taken together, these methodological advances over prior work, the gains across static and interactive tasks, and the ability to transfer the learned harness across executors demonstrate both the novelty and the practical value of ExpHarness.

AuthorsOpenReview ↗

Response 2

[W2, Q1]: Fixed retrieval parameters and reward ablations

W2: The paper does not directly compare the trained copilot against fixed or tuned retrieval parameters. Since the no-experience score is independent of the retrieval choice for a given task, its additional contribution to learning is unclear.

Q1: Could you compare with fixed or tuned retrieval parameters, and ablate the different reward terms?

Response. Thank you for raising these questions. We first test whether the learned retrieval copilot improves on a fixed configuration selected by validation search (Table 1). We then examine how each reward term contributes to performance (Table 2).

Trained copilot vs. validation-tuned retrieval parameters (R,W)(R,W)

Setting. Within each suite and executor size, all three settings use the same training, validation, and test splits, start from the same cold-start experience graph, and use the same graph capacity of 2,000 nodes (Appendix B). The two fixed baselines use the same graph update procedure and identical hyperparameters apart from (R,W)(R,W). Appendix E describes how we construct the validation sets.

  • Fixed uses the prespecified pair (R,W)=(50,50)(R,W)=(50,50) throughout training and testing.
  • Validation-tuned fixed searches all 25 pairs with R,W∈{0,25,50,75,100}R,W \in \{0,25,50,75,100\}, separately on the ExpSuite-Static and ExpSuite-Agentic validation sets. Each pair stays fixed throughout its training run while its graph evolves. We freeze the resulting graphs, evaluate each pair with its own graph on the suite’s validation set, and use the pair and graph with the highest weighted validation score at test time.
  • ExpHarness trains the retrieval copilot with PPO to choose (R,W)(R,W) for each task as the graph evolves. We select the copilot checkpoint on the same validation set used for the grid search, then freeze the copilot and its associated graph for testing.

Table 1. Test scores with fixed, validation-tuned, and task-adaptive retrieval controls. QA, Reasoning, and Coding are ExpSuite-Static domains. ALFWorld and AppWorld are ExpSuite-Agentic domains.

ExecutorMethodQAReasoningCodingALFWorldAppWorld
SmallFixed (50, 50)61.0266.8153.7852.9234.36
SmallValidation-tuned fixed63.4470.1557.1460.9540.17
SmallExpHarness66.4574.6760.5072.6344.44
LargeFixed (50, 50)72.8572.4468.0775.5538.46
LargeValidation-tuned fixed75.5477.5671.4381.3945.30
LargeExpHarness77.3781.0474.7986.5050.94

Results. Validation tuning improves on (50,50)(50,50) in all five domains for both executor sizes. ExpHarness outperforms the validation-tuned fixed pair in all ten comparisons, by 3.01–11.68 points with the small executor and 1.83–5.64 points with the large executor. The largest gain for the small executor is on ALFWorld, from 60.95 to 72.63 (+11.68 points). For the large executor, it is on AppWorld, from 45.30 to 50.94 (+5.64 points).

Conclusion. At the same graph capacity, the learned retrieval copilot achieves higher scores by choosing (R,W)(R,W) for each task than either fixed baseline achieves, including the best pair selected by the 25-pair grid search. These end-to-end gains include how the copilot’s retrieval choices shape the experience graph during training. We describe the protocol in §4.3 and report the results in Table 4.

Reward ablations:

Setting. We vary only the training reward in §3.4. All three runs start from the same cold-start experience graph and Qwen2.5-3B-Instruct retrieval copilot. Each run independently trains the copilot and updates the graph with its assigned reward. We compare Relative only (swith−swithouts_{\mathrm{with}}-s_{\mathrm{without}}, η=0\eta=0), Absolute only (swiths_{\mathrm{with}}), and the full ExpHarness reward (2swith−swithout2s_{\mathrm{with}}-s_{\mathrm{without}}, η=1\eta=1).

Table 2. Test scores with relative-only, absolute-only, and full training rewards.

ExecutorTraining rewardQAReasoningCodingALFWorldAppWorld
SmallRelative only65.3271.7857.1469.3441.37
SmallAbsolute only66.0273.7059.6671.1743.42
SmallFull reward66.4574.6760.5072.6344.44
LargeRelative only75.9179.2672.2783.9448.72
LargeAbsolute only76.7780.4473.1185.0449.40
LargeFull reward77.3781.0474.7986.5050.94

Results. The full reward scores highest in all five domains for both executor sizes. Across the ten comparisons, it improves on Relative only by 1.13–3.36 points and on Absolute only by 0.43–1.68 points.

AuthorsOpenReview ↗

Response 3

Continuation of our response to [W2, Q1] from the previous page:

For a given task, subtracting swithouts_{\mathrm{without}} leaves the ordering of retrieval choices unchanged. The full and absolute-only rewards also differ in the coefficient of swiths_{\mathrm{with}}, so this comparison cannot isolate the contribution of subtracting swithouts_{\mathrm{without}} alone. Across tasks with binary scores, the full reward assigns 2 when retrieval turns a failure into a success and 1 when the executor succeeds with or without retrieval. These rewards update the graph’s utility estimates during training. We report this ablation in §4.3, Table 5 of the revised manuscript.


Q2: Could you provide a more complete cost comparison, including copilot training, retrieval overhead, and executor calls?

Response. Thank you for asking about the cost of learning to retrieve experience. Table 3 compares ExpHarness with MemRL and S3 on ExpSuite-Static and on ALFWorld for ExpSuite-Agentic. In the revised manuscript, we added the experimental setting, cost measurements, and analysis to Appendix B (Implementation Details), under “Training cost and retrieval overhead,” immediately after “Infrastructure.” Table 6 reports the measurements.

Setting. The ExpHarness retrieval copilot and the S3 search policy are both trained from the same Qwen2.5-3B-Instruct checkpoint. All methods run on the same hardware platform with four NVIDIA A6000 GPUs (48 GB each). We use the best-performing training hyperparameters reported in each baseline’s original paper. For retrieval timing, all methods use FAISS with the same memory capacity, number of indexed entries, and experience budget per executor input.

Table 3. Training cost and mean retrieval overhead. Training times are wall-clock hours per run. API calls are reported in thousands. ALFWorld call counts are shown for the small and large executors, respectively.

SettingMethodTrainingTraining executor API callsMean retrieval overhead
ExpSuite-StaticMemRLCPU Q-value learning8.0k0.12 s/task
ExpSuite-StaticS330 h13.5k2.84 s/task
ExpSuite-StaticExpHarness20 h13.5k1.07 s/task
ExpSuite-Agentic (ALFWorld)MemRLCPU Q-value learning325.8k / 227.4k0.16 s/retrieval
ExpSuite-Agentic (ALFWorld)S355 h334.2k / 273.7k3.26 s/retrieval
ExpSuite-Agentic (ALFWorld)ExpHarness40 h306.3k / 239.8k1.23 s/retrieval

ExpSuite-Static. MemRL achieves the highest weighted average among the comparison methods. ExpHarness improves on it by 12.1% with the small executor and 4.5% with the large executor (§4.1), with 13.5k training executor calls versus 8.0k for MemRL.

ExpSuite-Agentic. S3 achieves the highest weighted average among the comparison methods. ExpHarness improves on S3 by 21.4% with the small executor and 12.7% with the large executor across the full suite (§4.1). On ALFWorld, where we measure cost, both methods train the same Qwen2.5-3B-Instruct policy backbone. ExpHarness trains in 40 h versus 55 h (27% less time), uses 8.3% and 12.4% fewer training executor calls for the small and large executors, respectively, and reduces mean retrieval overhead from 3.26 to 1.23 s per retrieval (62% lower).

Retrieval at inference. A single FAISS lookup takes milliseconds, with negligible differences across methods. ExpHarness predicts its controls once, retrieves semantic seeds, expands them through graph diffusion, and reranks the candidates. S3 can make another model decision and issue another search after inspecting an intermediate result. On both ExpSuite-Static and ALFWorld, ExpHarness retrieves faster than S3 but slower than MemRL. On ExpSuite-Static, ExpHarness scores higher than MemRL and uses more training executor calls (13.5k vs. 8.0k). On ALFWorld, it uses fewer training executor calls than S3 at both executor sizes.

We would like to thank the reviewer once again for their valuable time and thoughtful feedback. We sincerely hope that our responses and revisions have adequately addressed the concerns raised and further improved the clarity and overall quality of the manuscript. We would greatly appreciate the reviewer's further assessment of these improvements and warmly welcome any additional feedback or discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Appreciation to the Area Chair and Reviewers: Summary of Our Rebuttal and Key Improvements

We sincerely thank the Area Chair and all reviewers for their thoughtful and constructive feedback. During the rebuttal, we conducted additional controlled experiments, strengthened our ablation studies, clarified evaluation protocols, and expanded our analysis of statistical reliability and computational efficiency. Below, we summarize the major concerns and our responses.

DimensionKey ConcernsOur Main ActionsKey Findings
Novelty & ContributionWhat distinguishes ExpHarness from existing memory and retrieval methods?Clarified the joint learning of task-adaptive graph exploration and utility-aware experience selection, with a frozen and replaceable executor.ExpHarness learns how to retrieve useful experiences, rather than relying solely on semantic similarity or fixed retrieval strategies.
Experimental FairnessAre improvements caused by inconsistent prompts, data splits, or baseline configurations?Added prompt-matched No-Memory controls, clarified train/validation/test splits, memory sharing, and hyperparameter selection.Under identical executor prompts, retrieved experiences improve weighted scores by 18.38 and 17.60 points for small and large executors.
Component EffectivenessAre the learned copilot, graph retrieval, reward design, and utility updates necessary?Added comparisons with 25-pair validation-tuned retrieval, MLP contextual bandits, and simple kNN, alongside separate reward, EMA utility-update, and UCB ablations.ExpHarness consistently outperforms these alternatives, supporting the effectiveness of its task-adaptive retrieval and experience-learning mechanisms.
Statistical Reliability & GeneralizationAre the improvements robust, and how are experiences shared across tasks?Added three-seed evaluations with mean ± SD and statistical tests, clarified shared experience graphs, and discussed cross-executor transfer.ExpHarness maintains significant weighted-score improvements over the strongest baselines across all four suite–executor settings at the nominal 0.05 level.
Computational EfficiencyWhat are the training costs, executor-call requirements, and retrieval overhead?Added detailed training-time, executor-call, and retrieval-latency comparisons against MemRL and S3.On ALFWorld, ExpHarness reduces training time by 27% and retrieval overhead by 62% compared with S3, while achieving higher task performance.

Overall Assessment

The additional experiments strengthen three central conclusions:

  1. Methodological effectiveness: Task-adaptive retrieval and joint graph–copilot learning provide consistent benefits over fixed and lightweight alternatives.
  2. Experimental rigor: Prompt-controlled comparisons, targeted ablations, and multi-seed evaluations provide stronger evidence that the reported improvements arise from the proposed mechanisms.
  3. Practical value: ExpHarness delivers consistent improvements across static and interactive tasks while enabling experience reuse across frozen executors with competitive computational costs.

Appreciation

We sincerely appreciate the reviewers' valuable feedback, which has helped improve the rigor, clarity, and completeness of our work. We hope these additional results and clarifications provide a stronger basis for evaluating the contributions and practical significance of ExpHarness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.