- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ExpHarness moves experience learning for LLM agents from the executor to a trainable harness. The system has two components. An experience graph distills historical trajectories into skills and lessons, links related entries, and is searched by semantic seeding, personalized PageRank diffusion, and a UCB-style utility-aware ranking. A retrieval copilot, a lightweight LLM trained with PPO, outputs two per-task controls: the locality of graph diffusion and the trade-off between semantic relevance and historical utility. The reward combines the executor's score difference with and without retrieved experience and an absolute-performance term, and the same signal updates the utility estimates of retrieved nodes through an EMA, so the graph and copilot co-evolve while the executor stays frozen.
Critical strengths
- The problem positioning is practically meaningful and the evaluation is broad. A frozen, replaceable executor with a portable harness matches how agents are built on closed or rapidly evolving models, and the paper provides a unified learning interface for this setting. Twelve baselines, ten static benchmarks, two agentic environments, three transfer directions, and four ablations show consistent gains across settings, with interaction steps reduced in the agentic environments at the same time.
- The calibration of claims is above the norm for this area. The analysis explicitly notes how relative improvements are computed, that the Flat Experience ablation conflates two changes, that radar-chart axes differ so areas are not comparable, and that model identity and reasoning capability change together. This transparency increases confidence in the reported comparisons.
Critical weaknesses
- There is no statistical uncertainty reporting anywhere in the paper. Across 66 pages there is not a single standard deviation, seed count, or significance test. GPQA-Diamond has 198 questions, and many benchmark margins are a few points, so a number of gaps in Tables 2 and 3 cannot be distinguished from noise. The abstract states relative improvements in percent (12.1%, 21.4%), which, without variance information, can inflate the impression of the actual effect size.
- The reward and the necessity of the training machinery are under-justified. First, with η = 1 the reward r = (s_with - s_without) + η·s_with stays positive even when retrieval changes nothing, so the copilot is partly rewarded for the executor's own ability; the paper acknowledges this in Section 3.4 but provides no ablation on η, including the pure counterfactual difference at η = 0. Second, the same set-level reward is assigned to all retrieved nodes and folded into an EMA, which is coarse credit assignment (also acknowledged), and its interaction with the UCB exploration bonus is not analyzed. Third, the copilot is a 3B model trained with PPO and a critic to emit two integers; a contextual bandit, a small MLP, or even a grid search over fixed (R, W) could be equivalent alternatives. Without such controls the necessity of RL is not established, and the training cost (two executor calls per task plus PPO) cannot be judged reasonable.
Questions for authors
- In the static setting, is the experience graph trained and evaluated per benchmark, or shared across the ten benchmarks? If per benchmark, the static gains measure in-benchmark adaptation; please add cross-task transfer (e.g., graph trained on GSM8K, evaluated on MATH) and discuss potential contamination between executor pretraining memory and public test sets.
- Are the results single runs? Please report mean and standard deviation of the weighted averages over at least three seeds, especially for small test sets such as GPQA-Diamond.
- Please compare against lightweight controllers: how far do a two-layer MLP, a contextual bandit, or a grid search over fixed (R, W) get? If they match the trained copilot, the PPO machinery is not justified.
7 comments
Response 2
Continuation of our response to [W1, Q2] from the previous page:
Table 110 (abridged). ExpSuite-Static results across all ten benchmarks. For each executor size, we show ExpHarness and the two baselines with the highest weighted average score. Scores are percentages reported as mean ± SD across three seeds. The weighted average uses the number of test questions in each benchmark. For all methods, see Appendix I, Table 110 of the revised manuscript.
| Executor | Method | ARC-C | CSQA | GPQA-D | MMLU | OBQA | GSM8K | GSM-Sym | MATH | HumanEval+ | MBPP+ | Weighted Avg. | vs. ExpHarness (weighted avg.) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Small | MemRL | 67.56±1.68 | 60.15±1.12 | 25.56±1.92 | 39.33±0.89 | 67.26±0.13 | 84.67±1.54 | 68.30±0.93 | 54.74±2.06 | 46.15±2.56 | 57.08±1.91 | 62.12±0.78 | 0.0003 |
| Small | S3 | 64.81±0.78 | 60.00±2.78 | 22.22±0.96 | 43.19±1.81 | 63.85±0.68 | 82.52±0.56 | 74.67±0.59 | 53.19±1.45 | 47.86±3.92 | 55.00±1.25 | 62.06±1.00 | 0.0006 |
| Small | ExpHarness | 74.37±2.14 | 63.70±0.56 | 26.67±1.67 | 60.30±1.14 | 75.33±1.33 | 84.67±0.59 | 81.78±1.24 | 57.26±0.26 | 56.41±2.56 | 62.50±1.25 | 69.88±0.84 | — |
| Large | S3 | 82.81±1.22 | 74.96±1.34 | 26.67±1.67 | 65.04±0.71 | 81.19±0.84 | 89.78±1.92 | 86.52±2.16 | 53.56±0.80 | 58.97±2.56 | 72.92±1.91 | 75.09±1.10 | 0.0112 |
| Large | IRCoT | 83.70±1.22 | 75.70±0.46 | 30.00±1.67 | 67.04±0.13 | 82.96±1.10 | 88.67±1.33 | 83.78±2.12 | 52.30±1.14 | 50.43±1.48 | 71.25±2.50 | 75.05±0.83 | 0.0087 |
| Large | ExpHarness | 86.15±0.68 | 74.81±0.71 | 29.44±0.96 | 69.04±0.90 | 87.11±2.73 | 94.52±0.68 | 90.52±1.94 | 59.04±1.61 | 70.09±2.96 | 79.17±1.91 | 79.11±1.11 | — |
Table 111 (abridged). ExpSuite-Agentic results across all four evaluation splits. For each executor size, we show ExpHarness and the two baselines with the highest weighted average score. Values are mean ± SD across three seeds. ALFWorld reports success rate (SR, %), and AppWorld reports pass rate (PR, %). Average scores and interaction steps are weighted by the number of test tasks. For all methods, see Appendix I, Table 111 of the revised manuscript.
| Executor | Method | ALF-Seen SR | Steps | ALF-Unseen SR | Steps | AppWorld Test-N PR | Steps | AppWorld Test-C PR | Steps | Avg. Score ↑ | Avg. Steps ↓ | vs. ExpHarness (avg. score) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Small | S3 | 60.0±1.4 | 23.5±1.0 | 42.8±2.6 | 21.4±1.0 | 54.0±1.2 | 9.3±1.5 | 35.3±1.3 | 15.4±1.6 | 44.1±0.9 | 16.4±1.2 | 0.0003 |
| Small | IRCoT | 34.3±0.7 | 38.0±1.3 | 27.1±0.9 | 39.2±1.8 | 53.6±0.6 | 12.3±1.2 | 36.1±2.2 | 17.5±1.6 | 37.8±1.3 | 23.2±1.4 | 0.0001 |
| Small | ExpHarness | 69.8±1.1 | 20.3±1.7 | 75.4±1.5 | 18.1±0.7 | 56.7±0.7 | 10.0±1.9 | 39.2±1.0 | 15.0±1.4 | 53.2±1.0 | 15.4±1.4 | — |
| Large | S3 | 77.4±1.1 | 21.9±1.8 | 80.6±0.7 | 21.4±0.7 | 52.0±2.8 | 17.6±1.4 | 41.4±1.3 | 21.7±0.7 | 55.5±1.4 | 20.9±1.0 | 0.0054 |
| Large | Search-o1 | 75.7±1.2 | 22.0±0.8 | 78.4±0.7 | 23.5±1.3 | 51.8±1.0 | 21.1±1.9 | 39.7±1.1 | 24.5±1.3 | 54.0±0.7 | 23.3±1.3 | 0.0044 |
| Large | ExpHarness | 85.2±2.5 | 18.7±2.2 | 88.1±3.0 | 16.4±1.1 | 56.5±1.0 | 13.1±1.3 | 48.0±1.0 | 14.2±0.7 | 62.0±1.5 | 15.0±0.9 | — |
Response 3
W2: Under-justified reward design.
The reward and the necessity of the training machinery are under-justified. First, with η = 1 the reward r = (s_with - s_without) + η·s_with stays positive even when retrieval changes nothing, so the copilot is partly rewarded for the executor's own ability; the paper acknowledges this in Section 3.4 but provides no ablation on η, including the pure counterfactual difference at η = 0. Second, the same set-level reward is assigned to all retrieved nodes and folded into an EMA, which is coarse credit assignment (also acknowledged), and its interaction with the UCB exploration bonus is not analyzed.
Response. Thank you for raising these questions. We address the choice of training reward in §4.3, Table 5 and the contribution of the node-utility update and exploration bonus in the new Appendix K, “Node Utility Updates and UCB Exploration,” Table 113.
Contribution of the two reward terms
We agree that the full reward gives some credit when the executor already knows the answer. With binary scores and , a correct answer both with and without experience earns a reward of 1. Turning an incorrect answer into a correct one earns 2. For a given input, is constant across retrieval choices. Across inputs, it distinguishes these two kinds of success. That distinction also enters the graph’s utility estimates and can influence future retrieval.
Setting. We vary only the training reward in §3.4. Each run starts from the same cold-start experience graph and Qwen2.5-3B-Instruct retrieval copilot, then independently trains the copilot and updates its graph. We compare Relative only (, ), Absolute only (), and the Full reward (, ).
Table 5. Test scores by training reward (§4.3).
| Executor | Training reward | QA | Reasoning | Coding | ALFWorld | AppWorld |
|---|---|---|---|---|---|---|
| Small | Relative only | 65.32 | 71.78 | 57.14 | 69.34 | 41.37 |
| Small | Absolute only | 66.02 | 73.70 | 59.66 | 71.17 | 43.42 |
| Small | Full reward | 66.45 | 74.67 | 60.50 | 72.63 | 44.44 |
| Large | Relative only | 75.91 | 79.26 | 72.27 | 83.94 | 48.72 |
| Large | Absolute only | 76.77 | 80.44 | 73.11 | 85.04 | 49.40 |
| Large | Full reward | 77.37 | 81.04 | 74.79 | 86.50 | 50.94 |
Results. The full reward scores highest in all five domains at both executor sizes. It improves on the pure counterfactual reward by 1.13–3.36 points and on the absolute-only reward by 0.43–1.68 points. These gains measure the joint effect on the trained copilot and graph. The full-versus-absolute comparison changes both the coefficient of and the inclusion of , so the observed difference reflects both changes.
Shared node utility and UCB exploration
The executor uses the retrieved experiences together to produce one answer or trajectory. Several nodes can supply complementary information, while the observed reward applies to the set as a whole. Our update draws on the idea in cooperative reinforcement learning that a shared outcome can provide useful feedback when individual contributions are difficult to observe [1]. We update each retrieved node’s utility using an exponential moving average (EMA) of the set reward, . Thus, tracks recent rewards for retrieval sets containing node .
Our original w/o Utility Ranking ablation in §4.3, Figure 3 ranked candidates by semantic similarity alone, removing both historical utility and UCB exploration from ranking. Following the reviewer’s suggestion, we added two ablations that remove each component separately. The EMA records outcomes for previously retrieved nodes. The UCB bonus gives less frequently retrieved nodes more opportunities to receive feedback.
Setting. w/o Utility Update fixes for every node and retains the UCB bonus (). w/o UCB retains the EMA utility update () and sets . ExpHarness uses both components (, ).
Table 113. Ablations of the EMA utility update and UCB exploration bonus (Appendix K).
| Executor | Configuration | QA | Reasoning | Coding | ALFWorld | AppWorld |
|---|---|---|---|---|---|---|
| Small | w/o Utility Update | 64.25 | 70.44 | 57.14 | 67.52 | 40.34 |
| Small | w/o UCB | 65.05 | 72.96 | 58.82 | 70.44 | 43.08 |
| Small | ExpHarness | 66.45 | 74.67 | 60.50 | 72.63 | 44.44 |
| Large | w/o Utility Update | 74.30 | 77.41 | 69.75 | 81.02 | 45.98 |
| Large | w/o UCB | 76.24 | 79.78 | 73.11 | 84.31 | 48.72 |
| Large | ExpHarness | 77.37 | 81.04 | 74.79 | 86.50 | 50.94 |
Response 1
We sincerely thank the reviewer for the thorough assessment and valuable suggestions on statistical reliability, reward design, experience graph learning, and the necessity of the proposed retrieval controller. We have substantially strengthened our evaluation with multi-seed experiments, additional component-level ablations, and comparisons against lightweight retrieval controllers. We address these concerns in detail below.
Response. Thank you for raising this concern. Tables 2 and 3 in §4.1 report a single run. We now report mean ± standard deviation across three seeds for every method’s benchmark scores and test-instance-weighted average scores, in Appendix I, “Statistical Uncertainty Across Three Seeds”. Table 110 reports ExpSuite-Static, and Table 111 reports ExpSuite-Agentic. For GPQA-Diamond, where our test split contains 60 questions and one answer changes accuracy by 1.67 percentage points (Appendix E, Table 8), ExpHarness scores 26.67 ± 1.67 and 29.44 ± 0.96 with the small and large executors, respectively. Due to space constraints, we show the results for only a subset of baselines in the tables below. The complete results are provided in the Appendix: Table 110 reports ExpSuite-Static, and Table 111 reports ExpSuite-Agentic.
Effect sizes. For the single run summarized in the abstract, the relative improvements of 12.1% and 4.5% on ExpSuite-Static correspond to absolute gains of 7.51 and 3.42 percentage points for the small and large executors. On ExpSuite-Agentic, the corresponding relative improvements of 21.4% and 12.7% represent 9.4 and 7.0 percentage points.
Statistical tests. The final column of each appendix table compares each baseline’s weighted score with ExpHarness at the same executor size. We use two-sided Welch’s t-tests, treating the three run-level scores per method as independent. The tests are exploratory: the p-values are computed from the reported means and sample SDs and are unadjusted for multiple comparisons. We interpret individual benchmark results descriptively.
Conclusion. Across three seeds, ExpHarness retains a statistically significant gain in weighted score over the highest-scoring baseline in all four suite–executor settings at the nominal 0.05 level. On ExpSuite-Static, the gains are 7.76 percentage points over MemRL for the small executor (p=0.0003) and 4.02 points over S3 for the large executor (p=0.0112). On ExpSuite-Agentic, the gains over S3 are 9.1 points for the small executor (p=0.0003) and 6.5 points for the large executor (p=0.0054).