- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HExA improves a frozen LLM agent by turning its experimental experience into reusable skills and mistake corrections. An actor collects trajectories, an evolver updates the skill bank, and a retriever supplies skills for later tasks. The paper also introduces Interphyre, a physics environment with experimentation tools. Evaluations cover physical reasoning, games, reasoning/planning, and physics-law discovery, with gains over memoryless and episodic-memory baselines and some transfer across tasks.
Critical strengths
-
The gains on unseen instances are large. With a frozen bank, HExA reaches 82.7%/96.0% on the two hard Interphyre levels, versus ReAct's 11.0%/20.0% and Episodic RAG's 62.9%/60.0% (Table 8). This makes experience distillation a useful approach even without weight updates.
-
The feedback ablation is informative. With the same bank, extended reasoning without feedback reaches 42%/48%, while outcome-only feedback reaches 90%/90% (Table 14, single-run results). It shows why stored experience still needs testing on the current instance.
Critical weaknesses
-
The methodological novelty is not sufficiently established, and the specific contribution remains unclear. HExA’s process of distilling interaction trajectories into skills, updating a skill bank, and guiding subsequent tasks overlaps substantially with existing skill-learning methods.
-
Replacement discards the previous skill bank, while Offline stops updating it. The gains may therefore come from retaining knowledge and continued learning, rather than hierarchical composition.
Questions for authors
-
How does HExA compare with a representative automatic skill-building method under the same actor, experience budget, and context budget?
-
Does hierarchical composition improve held-out performance over cumulative skill distillation when both retain comparable experience and knowledge?
-
Please reconcile the GRPO/HExA interaction counts, clarify Figure 20's axis, and confirm whether both methods use the same evaluation instances.
8 comments
Response to Reviewer eFFR (Part II/V)
W2: “Replacement discards the previous skill bank, while Offline stops updating it. The gains may therefore come from retaining knowledge and continued learning, rather than hierarchical composition.”;
Q1: “How does HExA compare with a representative automatic skill-building method under the same actor, experience budget, and context budget?”; AND
Q2: “Does hierarchical composition improve held-out performance over cumulative skill distillation when both retain comparable experience and knowledge?”
We want to clarify the Iterative Replacement configuration, it does not discard earlier knowledge completely. Since new trajectories are rolled out conditioned on previous skill bank, earlier skills do affect the contents in the new bank because the latter is synthesised from these trajectories. We agree, however, that neither Replacement nor the Offline ablation, which stops updating the bank, fully isolates the gains of HExA. The point of using these ablations was to show usefulness of skill conditioned data collection via this setup. But as we have shown the optimal setup is to do this on top of the existing skill bank, which is our final Evolution setup in HExA (Figure 6 in paper). We have therefore added more controlled ablations that retain all accumulated experience and keep learning, but replace HExA’s skill evolution and hierarchical composition with simpler alternatives.
We have also clarified how skill evolution and hierarchical skill formation both are complementary (updated in Paper in Section 3.4). Repeated updates to the skills via experimentation is what we call skill evolution: skills are refined, corrected and improved under reward. Hierarchical composition refers to building increasingly complex skills from previously acquired ones, which we test across levels. We now describe in detail all these ablations.
Within a level: evolution vs. cumulative distillation. All the ablations below use the same actor, evolver model, training seeds and number of trajectories as HExA, and all retain the accumulated experience.
HExA vs Accumulated Summary Ablations
| Configuration | Retains accumulated lessons | Injected context | Held-out success (%) | Turns per seed |
|---|---|---|---|---|
| Summary bank | Yes | ~1.9k tokens | 62.7 ± 0.9 | 17.6 |
| Append-only bank | Yes | Same capacity as HExA | 48.0 ± 12.8 | 18.3 |
| HExA | Yes | 1.5–2.3k tokens | 82.7 ± 9.3 | 12.5 |
- The summary bank answers Q2 directly. The evolver distills all experience into a cumulative summary, from the same trajectories and at a matched context budget. It reaches 62.7%, 20.0 points below HExA. Append-only updates retain every new lesson without revising earlier ones and loses 34.7 points.
Skill Based Experimentation
| Configuration | What changes | Held-out success (%) | Turns per seed | Drop vs. HExA |
|---|---|---|---|---|
| Skill-free collection | Bank hidden during collection | 63.3 ± 3.3 | 15.3 | 19.4 |
| Earliest skill versions | Skill text reverted to earliest versions | 56.7 ± 16.5 | 12.5 | 26.0 |
| HExA | — | 82.7 ± 9.3 | 12.5 | — |
Two further ablations in the table above isolate HExA's components (Section 3.3, Table 2; Appendix H.2.3, Table 17.):
- Earliest skill versions: keeping the same skills but reverting their content to its first version costs 26.0 points, so the gain comes from how skills are revised, not from which knowledge is kept.
- Skill-free collection: hiding the bank during in-context training costs 19.4 points, so skills also improve the experience they collect later and what we later learn from them.
Across levels: hierarchical skill composition vs retained knowledge.
- Our main test is per-level transfer. It retains exactly the same knowledge as HExA: the same three source skill banks, with the same actor, evolver and output budget, and no target trajectories. Each source skill bank is adapted to the target level separately and are combined, so the source skills are carried over but never composed with skills from other levels. HExA instead composes new target skills from skills across the three source level skill banks . The only difference between the two is therefore whether hierarchical composition across levels takes place.
Hierarchical composition raises held-out success from 25.3% to 42.7% on catapult, a gain of 17.4 points with identical retained knowledge. The synthesized skills do combine across levels: the nine target skills draw on 22 source skills, and six of the nine combine skills from two or more source levels (Appendix H.4.1).
Details in table below.
Response to Reviewer eFFR (Part III/V)
Across levels: hierarchical composition vs retained knowledge. (continued)
Hierarchical Skill Ablations
| Configuration | Knowledge available | Composition across levels | Held-out success (%) |
|---|---|---|---|
| HExA | Three source banks | Explicit: new target skills composed from source skills | 42.7 ± 1.9 |
| Invocation-based hierarchical skills | Three source banks | Implicit: the actor composes source skills in its context | 34.7 ± 6.4 |
| Per-level transfer | Three source banks | No cross-level synthesis. Each bank is adapted separately. | 25.3 ± 10.9 |
| Description-only synthesis | Task description only | — | 18.0 ± 2.0 |
| ReAct | None | — | 11.0 ± 3.0 |
We show additional ablations to support the benefit of HExA’s hierarchical composition :
- Invocation-based hierarchical skills: the source skills are placed unchanged in the actor's context, and the actor invokes and composes them on the target task. This is implicit hierarchical composition, and it reaches 34.7%, 9.4 points above per-level transfer. This shows the source skills already work as reusable sub-skills on an unseen task. Making the composition explicit, by synthesizing new target skills from them, adds a further 8.0 points as shown with HExA finally.
- Description-only synthesis: with the same synthesis procedure but no source banks, the bank reaches only 18.0%, 24.7 points below HExA. The gain therefore comes from composing the learned skills hierarchically, not from the model's prior knowledge of the task.
With the retained knowledge held fixed, hierarchical composition improves held-out performance (Section 3.4, Table 3; Appendix H.2.3, Table 18).
- Comparison with automatic skill-building methods (Q1). We added two recent agentic memory methods that automatically distill experience into reusable strategies: ACE [1] and Reasoning Bank [2] details in Appendix (G.7 and G.8) . Each uses the same actor for all roles, the same 5 warm-start trajectories, the same 50 training seeds with one episode per seed, and a frozen memory on the same 50 held-out seeds. For ReasoningBank we use top-5 retrieval so its injected context is close to HExA’s (~2.0k vs ~2.5k tokens) and ACE injects its full, growing playbook, up to 4.2× HExA’s context (Table 10).
ACE and ReasoningBank with Claude Sonnet 4.6
| Level | Method | Success (%) | Avg iters | Tokens/ep. |
|---|---|---|---|---|
| catapult | ACE (playbook) | 50.0 ± 3.0 | 17.5 ± 1.6 | 9,960 |
| catapult | ReasoningBank | 56.0 ± 10.7 | 16.3 ± 2.1 | 2,085 |
| catapult | HExA | 82.7 ± 9.3 | 12.5 ± 0.9 | 2,392 |
| pass_the_parcel | ACE (playbook) | 84.0 ± 10.0 | 10.1 ± 2.1 | 4,243 |
| pass_the_parcel | ReasoningBank | 80.0 ± 13.0 | 8.0 ± 1.9 | 1,989 |
| pass_the_parcel | HExA | 96.0 ± 2.8 | 5.8 ± 0.8 | 2,631 |
ACE and ReasoningBank with Qwen2.5 3B and 7B
| Method | 7B DtE | 7B TBP | 3B DtE | 3B TBP |
|---|---|---|---|---|
| ReAct (no memory) | 63.3 ± 3.4 | 12.0 ± 0.0 | 8.3 ± 0.9 | 7.3 ± 0.9 |
| ACE (playbook) | 62.7 ± 9.4 | 26.0 ± 7.5 | 2.7 ± 2.5 | 1.3 ± 0.9 |
| ReasoningBank | 77.3 ± 5.7 | 28.7 ± 8.1 | 11.3 ± 1.9 | 8.0 ± 4.3 |
| HExA | 82.7 ± 5.2 | 50.0 ± 3.3 | 23.3 ± 0.9 | 12.0 ± 0.0 |
HExA has the highest success in all six settings, at similar context budgets compared to both baselines. ACE and ReasoningBank accumulate and summarize lessons, but they do not revise and compose skills under reward. Together with the other ablations above, this shows that HExA's gains not only comes from retaining more experience or more context, but also from how skills are evolved and hierarchically composed enabling both within level improvement and cross-task transfer. (updated Figure 3, Tables 10–11).
Q3: “Please reconcile the GRPO/HExA interaction counts, clarify Figure 20's axis, and confirm whether both methods use the same evaluation instances.”
- Axes: The x-axis shows GRPO training steps. With batch size 1 and n=4 rollouts per step, 50 steps make one full pass over the 50 GRPO training seeds. The later steps are further interactions over the same seeds. The y-axis is the success rate (25-turn cap) on 50 held-out test seeds that were not used during training. We have updated the axis label and caption to say this explicitly.
- Interaction budgets: The matched interaction budget line (step 50) marks the point where GRPO has made one pass over its training instances (50 gradient updates from 200 rollouts). The "x-more-interactions" annotations show how much further GRPO must train before its test success rate reaches HExA's matched-budget result.
- Evaluation instances: Both methods are scored on the same 50 held-out seeds of the same level, with the same 25-turn cap and the same binary simulator success criterion as every other method in the paper. In all cases the test seeds are disjoint from the seeds that method trained on, so neither result reflects memorized instances.
Response to Reviewer eFFR (Part I/V)
We thank the reviewer for the constructive feedback and for asking clarifying questions. We provide new experiments and ablations to address all of them.
W1: “The methodological novelty is not sufficiently established, and the specific contribution remains unclear. HExA’s process of distilling interaction trajectories into skills, updating a skill bank, and guiding subsequent tasks overlaps substantially with existing skill-learning methods.”
Clarifying methodological novelty and contributions:
We want to clarify that the loop the reviewer describes of distilling interaction trajectories into skills and reusing them is not our primary methodological contribution. While several recent and concurrent methods accumulate lessons from trajectories, these often remain closely tied to individual experiences, limiting their generalization and transfer. We have additionally included two recent agentic memory baselines, ACE [1] and ReasoningBank [2], which substantially underperform HExA and do not explicitly support hierarchical cross-task skill synthesis. Specifically, ReasoningBank distills each episode into at most three memory items, each derived from a single trajectory; MemRL stores (intent, experience, utility) triplets associated with past episodes; and ACE maintains an itemized playbook with local refinement and consolidation. Unlike HExA, none of these methods explicitly evolve higher-level, reward-grounded compositional skills by integrating evidence across multiple episodes. Our comparisons use the same actor, interaction limits, training experience, and memory-injection interface, with context budgets controlled or explicitly accounted for.
We first propose and formalize the hypothesis driven experimentation setup as a general in-context learning to learn and transfer framework and to establish the importance of learning from interactions and exploration guided by experiment grounded rewards and feedback. This grounded reward and feedback is essential and enables skill learning and evolution and enables us to work with a compressed efficient skill bank without continuously appending and increasing bank size without any reward guidance as done in ACE and ReasoningBank. These framework details are very crucial as they enable us to very efficiently learn on settings where the base model has no capabilities and also enables HExA to be the first method to instantiate in-context hierarchical skill learning.
The skill evolution is guided by reward obtained through active experimentation. The actor actively tests hypotheses in the environment, so every trajectory carries a reward. The evolver uses these scores to decide which entries to compose, merge, refine or prune, and the retriever uses them to rank what the actor sees. In our ablations, we find that reward free evolution labels lowers success as shown in Figure 7 and Table 14. The skill bank is bounded, and new evidence therefore cannot simply be appended. It must refine an earlier skill, narrow its applicability, combine with it, or replace it as a result of which very efficient, generalizable and hierarchical skill learning emerges.
This in-turn enables skill transfer and zero-shot self-improvement on unseen harder tasks, and this inner-loop self-improvement and outer-loop transfer ability lets HExA perform a form of in-context meta-learning. In its inner loop, HExA adapts to one level by building that level's skill bank. Each meta-episode consists of n trajectories collected under the current bank, after which the evolver updates the bank. We instantiate n = 3 and run 17 meta-episodes per level, with each trajectory limited to 25 interaction turns. In its outer loop, the banks learned on the source levels are synthesized into a single bank for a level the agent has never seen, with no trajectories from that level. What is carried across levels is not a task specific solution to any instance but knowledge of how to learn and improve via active experimentation. We clarify these in the updated paper Figure 1 and method.
We also introduce Interphyre to test agents and perform controlled experiments to design algorithms for learning via interaction and experimentation. As agents become more general and are deployed in interactive, real-world settings, static benchmarks are no longer sufficient: they test recall and reasoning over given information, not the ability to acquire missing information by acting. Evaluating and improving algorithms for this setting requires benchmarks built around experimentation. We therefore introduce Interphyre, a procedural 2D physics benchmark that gives the agent tools to inspect the scene, simulate, intervene, experiment and observe intermediate dynamics.