acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Experimentalist Agents

19 views

Abstract

Large language models (LLMs) are increasingly used to act in the real world and augment human decision-making, yet most systems rely on parametric knowledge acquired through imitation, optionally improved by post-training, retrieval, or search over fixed data. This paradigm struggles in novel domains or on complex queries that require information unavailable from prior knowledge alone; knowing the laws of physics, for example, does not by itself enable reliable reasoning or long-horizon actions in a complex physical system. We argue that agents therefore need *active experimentation*: the ability to gather targeted query-specific evidence, discover general principles of unseen environments, and convert interaction experience into reusable skills. We introduce **Hierarchical Experimentalist Agents (HExA)**, an in-context, experiment-centric self-improvement framework that (1) iteratively designs and refines query-relevant experiments, (2) incrementally distills experience into reusable and composable skills that accelerate experimentation within and across tasks, and (3) integrates experimental evidence to act or answer queries. HExA can work with any LLM including frontier black-box models, and requires no external supervision, oracle solutions, or offline data. We evaluate HExA on multi-step reasoning, interactive partially observable games, and physics law discovery tasks. To further evaluate when active experimentation is especially necessary, we introduce Interphyre, a set of hard procedural physics tasks where we enable tool-calling and intervention APIs. Current frontier LLMs including GPT-6 Astra fail in these settings: on the most challenging Interphyre tasks, we show that HExA can enable such models to near 100% success. Moreover, HExA improves token efficiency compared to naive agentic and memory baselines and enables generalization via hierarchical skill transfer from easier to harder tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Experimentalist Agent, Exploration, Hierarchical Skill Learning, Transfer Learning, In-Context RL, Self-Improvement

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eFFROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

HExA improves a frozen LLM agent by turning its experimental experience into reusable skills and mistake corrections. An actor collects trajectories, an evolver updates the skill bank, and a retriever supplies skills for later tasks. The paper also introduces Interphyre, a physics environment with experimentation tools. Evaluations cover physical reasoning, games, reasoning/planning, and physics-law discovery, with gains over memoryless and episodic-memory baselines and some transfer across tasks.

Critical strengths
  1. The gains on unseen instances are large. With a frozen bank, HExA reaches 82.7%/96.0% on the two hard Interphyre levels, versus ReAct's 11.0%/20.0% and Episodic RAG's 62.9%/60.0% (Table 8). This makes experience distillation a useful approach even without weight updates.

  2. The feedback ablation is informative. With the same bank, extended reasoning without feedback reaches 42%/48%, while outcome-only feedback reaches 90%/90% (Table 14, single-run results). It shows why stored experience still needs testing on the current instance.

Critical weaknesses
  1. The methodological novelty is not sufficiently established, and the specific contribution remains unclear. HExA’s process of distilling interaction trajectories into skills, updating a skill bank, and guiding subsequent tasks overlaps substantially with existing skill-learning methods.

  2. Replacement discards the previous skill bank, while Offline stops updating it. The gains may therefore come from retaining knowledge and continued learning, rather than hierarchical composition.

Questions for authors
  1. How does HExA compare with a representative automatic skill-building method under the same actor, experience budget, and context budget?

  2. Does hierarchical composition improve held-out performance over cumulative skill distillation when both retain comparable experience and knowledge?

  3. Please reconcile the GRPO/HExA interaction counts, clarify Figure 20's axis, and confirm whether both methods use the same evaluation instances.

8 comments
AuthorsOpenReview ↗

Response to Reviewer eFFR (Part I/V)

We thank the reviewer for the constructive feedback and for asking clarifying questions. We provide new experiments and ablations to address all of them.

W1: “The methodological novelty is not sufficiently established, and the specific contribution remains unclear. HExA’s process of distilling interaction trajectories into skills, updating a skill bank, and guiding subsequent tasks overlaps substantially with existing skill-learning methods.”


Clarifying methodological novelty and contributions:

  • We want to clarify that the loop the reviewer describes of distilling interaction trajectories into skills and reusing them is not our primary methodological contribution. While several recent and concurrent methods accumulate lessons from trajectories, these often remain closely tied to individual experiences, limiting their generalization and transfer. We have additionally included two recent agentic memory baselines, ACE [1] and ReasoningBank [2], which substantially underperform HExA and do not explicitly support hierarchical cross-task skill synthesis. Specifically, ReasoningBank distills each episode into at most three memory items, each derived from a single trajectory; MemRL stores (intent, experience, utility) triplets associated with past episodes; and ACE maintains an itemized playbook with local refinement and consolidation. Unlike HExA, none of these methods explicitly evolve higher-level, reward-grounded compositional skills by integrating evidence across multiple episodes. Our comparisons use the same actor, interaction limits, training experience, and memory-injection interface, with context budgets controlled or explicitly accounted for.

  • We first propose and formalize the hypothesis driven experimentation setup as a general in-context learning to learn and transfer framework and to establish the importance of learning from interactions and exploration guided by experiment grounded rewards and feedback. This grounded reward and feedback is essential and enables skill learning and evolution and enables us to work with a compressed efficient skill bank without continuously appending and increasing bank size without any reward guidance as done in ACE and ReasoningBank. These framework details are very crucial as they enable us to very efficiently learn on settings where the base model has no capabilities and also enables HExA to be the first method to instantiate in-context hierarchical skill learning.

  • The skill evolution is guided by reward obtained through active experimentation. The actor actively tests hypotheses in the environment, so every trajectory carries a reward. The evolver uses these scores to decide which entries to compose, merge, refine or prune, and the retriever uses them to rank what the actor sees. In our ablations, we find that reward free evolution labels lowers success as shown in Figure 7 and Table 14. The skill bank is bounded, and new evidence therefore cannot simply be appended. It must refine an earlier skill, narrow its applicability, combine with it, or replace it as a result of which very efficient, generalizable and hierarchical skill learning emerges.

  • This in-turn enables skill transfer and zero-shot self-improvement on unseen harder tasks, and this inner-loop self-improvement and outer-loop transfer ability lets HExA perform a form of in-context meta-learning. In its inner loop, HExA adapts to one level by building that level's skill bank. Each meta-episode consists of n trajectories collected under the current bank, after which the evolver updates the bank. We instantiate n = 3 and run 17 meta-episodes per level, with each trajectory limited to 25 interaction turns. In its outer loop, the banks learned on the source levels are synthesized into a single bank for a level the agent has never seen, with no trajectories from that level. What is carried across levels is not a task specific solution to any instance but knowledge of how to learn and improve via active experimentation. We clarify these in the updated paper Figure 1 and method.

  • We also introduce Interphyre to test agents and perform controlled experiments to design algorithms for learning via interaction and experimentation. As agents become more general and are deployed in interactive, real-world settings, static benchmarks are no longer sufficient: they test recall and reasoning over given information, not the ability to acquire missing information by acting. Evaluating and improving algorithms for this setting requires benchmarks built around experimentation. We therefore introduce Interphyre, a procedural 2D physics benchmark that gives the agent tools to inspect the scene, simulate, intervene, experiment and observe intermediate dynamics.

AuthorsOpenReview ↗

Response to Reviewer eFFR (Part II/V)

W2: “Replacement discards the previous skill bank, while Offline stops updating it. The gains may therefore come from retaining knowledge and continued learning, rather than hierarchical composition.”;

Q1: “How does HExA compare with a representative automatic skill-building method under the same actor, experience budget, and context budget?”; AND

Q2: “Does hierarchical composition improve held-out performance over cumulative skill distillation when both retain comparable experience and knowledge?”


We want to clarify the Iterative Replacement configuration, it does not discard earlier knowledge completely. Since new trajectories are rolled out conditioned on previous skill bank, earlier skills do affect the contents in the new bank because the latter is synthesised from these trajectories. We agree, however, that neither Replacement nor the Offline ablation, which stops updating the bank, fully isolates the gains of HExA. The point of using these ablations was to show usefulness of skill conditioned data collection via this setup. But as we have shown the optimal setup is to do this on top of the existing skill bank, which is our final Evolution setup in HExA (Figure 6 in paper). We have therefore added more controlled ablations that retain all accumulated experience and keep learning, but replace HExA’s skill evolution and hierarchical composition with simpler alternatives.

We have also clarified how skill evolution and hierarchical skill formation both are complementary (updated in Paper in Section 3.4). Repeated updates to the skills via experimentation is what we call skill evolution: skills are refined, corrected and improved under reward. Hierarchical composition refers to building increasingly complex skills from previously acquired ones, which we test across levels. We now describe in detail all these ablations.

Within a level: evolution vs. cumulative distillation. All the ablations below use the same actor, evolver model, training seeds and number of trajectories as HExA, and all retain the accumulated experience.

HExA vs Accumulated Summary Ablations

ConfigurationRetains accumulated lessonsInjected contextHeld-out success (%)Turns per seed
Summary bankYes~1.9k tokens62.7 ± 0.917.6
Append-only bankYesSame capacity as HExA48.0 ± 12.818.3
HExAYes1.5–2.3k tokens82.7 ± 9.312.5
  • The summary bank answers Q2 directly. The evolver distills all experience into a cumulative summary, from the same trajectories and at a matched context budget. It reaches 62.7%, 20.0 points below HExA. Append-only updates retain every new lesson without revising earlier ones and loses 34.7 points.

Skill Based Experimentation

ConfigurationWhat changesHeld-out success (%)Turns per seedDrop vs. HExA
Skill-free collectionBank hidden during collection63.3 ± 3.315.319.4
Earliest skill versionsSkill text reverted to earliest versions56.7 ± 16.512.526.0
HExA—82.7 ± 9.312.5—

Two further ablations in the table above isolate HExA's components (Section 3.3, Table 2; Appendix H.2.3, Table 17.):

  • Earliest skill versions: keeping the same skills but reverting their content to its first version costs 26.0 points, so the gain comes from how skills are revised, not from which knowledge is kept.
  • Skill-free collection: hiding the bank during in-context training costs 19.4 points, so skills also improve the experience they collect later and what we later learn from them.

Across levels: hierarchical skill composition vs retained knowledge.

  • Our main test is per-level transfer. It retains exactly the same knowledge as HExA: the same three source skill banks, with the same actor, evolver and output budget, and no target trajectories. Each source skill bank is adapted to the target level separately and are combined, so the source skills are carried over but never composed with skills from other levels. HExA instead composes new target skills from skills across the three source level skill banks . The only difference between the two is therefore whether hierarchical composition across levels takes place.

Hierarchical composition raises held-out success from 25.3% to 42.7% on catapult, a gain of 17.4 points with identical retained knowledge. The synthesized skills do combine across levels: the nine target skills draw on 22 source skills, and six of the nine combine skills from two or more source levels (Appendix H.4.1).

Details in table below.

AuthorsOpenReview ↗

Response to Reviewer eFFR (Part III/V)

Across levels: hierarchical composition vs retained knowledge. (continued)

Hierarchical Skill Ablations

ConfigurationKnowledge availableComposition across levelsHeld-out success (%)
HExAThree source banksExplicit: new target skills composed from source skills42.7 ± 1.9
Invocation-based hierarchical skillsThree source banksImplicit: the actor composes source skills in its context34.7 ± 6.4
Per-level transferThree source banksNo cross-level synthesis. Each bank is adapted separately.25.3 ± 10.9
Description-only synthesisTask description only—18.0 ± 2.0
ReActNone—11.0 ± 3.0

We show additional ablations to support the benefit of HExA’s hierarchical composition :

  • Invocation-based hierarchical skills: the source skills are placed unchanged in the actor's context, and the actor invokes and composes them on the target task. This is implicit hierarchical composition, and it reaches 34.7%, 9.4 points above per-level transfer. This shows the source skills already work as reusable sub-skills on an unseen task. Making the composition explicit, by synthesizing new target skills from them, adds a further 8.0 points as shown with HExA finally.
  • Description-only synthesis: with the same synthesis procedure but no source banks, the bank reaches only 18.0%, 24.7 points below HExA. The gain therefore comes from composing the learned skills hierarchically, not from the model's prior knowledge of the task.

With the retained knowledge held fixed, hierarchical composition improves held-out performance (Section 3.4, Table 3; Appendix H.2.3, Table 18).

  • Comparison with automatic skill-building methods (Q1). We added two recent agentic memory methods that automatically distill experience into reusable strategies: ACE [1] and Reasoning Bank [2] details in Appendix (G.7 and G.8) . Each uses the same actor for all roles, the same 5 warm-start trajectories, the same 50 training seeds with one episode per seed, and a frozen memory on the same 50 held-out seeds. For ReasoningBank we use top-5 retrieval so its injected context is close to HExA’s (~2.0k vs ~2.5k tokens) and ACE injects its full, growing playbook, up to 4.2× HExA’s context (Table 10).

ACE and ReasoningBank with Claude Sonnet 4.6

LevelMethodSuccess (%)Avg itersTokens/ep.
catapultACE (playbook)50.0 ± 3.017.5 ± 1.69,960
catapultReasoningBank56.0 ± 10.716.3 ± 2.12,085
catapultHExA82.7 ± 9.312.5 ± 0.92,392
pass_the_parcelACE (playbook)84.0 ± 10.010.1 ± 2.14,243
pass_the_parcelReasoningBank80.0 ± 13.08.0 ± 1.91,989
pass_the_parcelHExA96.0 ± 2.85.8 ± 0.82,631

ACE and ReasoningBank with Qwen2.5 3B and 7B

Method7B DtE7B TBP3B DtE3B TBP
ReAct (no memory)63.3 ± 3.412.0 ± 0.08.3 ± 0.97.3 ± 0.9
ACE (playbook)62.7 ± 9.426.0 ± 7.52.7 ± 2.51.3 ± 0.9
ReasoningBank77.3 ± 5.728.7 ± 8.111.3 ± 1.98.0 ± 4.3
HExA82.7 ± 5.250.0 ± 3.323.3 ± 0.912.0 ± 0.0

HExA has the highest success in all six settings, at similar context budgets compared to both baselines. ACE and ReasoningBank accumulate and summarize lessons, but they do not revise and compose skills under reward. Together with the other ablations above, this shows that HExA's gains not only comes from retaining more experience or more context, but also from how skills are evolved and hierarchically composed enabling both within level improvement and cross-task transfer. (updated Figure 3, Tables 10–11).


Q3: “Please reconcile the GRPO/HExA interaction counts, clarify Figure 20's axis, and confirm whether both methods use the same evaluation instances.”

  • Axes: The x-axis shows GRPO training steps. With batch size 1 and n=4 rollouts per step, 50 steps make one full pass over the 50 GRPO training seeds. The later steps are further interactions over the same seeds. The y-axis is the success rate (25-turn cap) on 50 held-out test seeds that were not used during training. We have updated the axis label and caption to say this explicitly.
  • Interaction budgets: The matched interaction budget line (step 50) marks the point where GRPO has made one pass over its training instances (50 gradient updates from 200 rollouts). The "x-more-interactions" annotations show how much further GRPO must train before its test success rate reaches HExA's matched-budget result.
  • Evaluation instances: Both methods are scored on the same 50 held-out seeds of the same level, with the same 25-turn cap and the same binary simulator success criterion as every other method in the paper. In all cases the test seeds are disjoint from the seeds that method trained on, so neither result reflects memorized instances.
Reviewer_igG2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents HExA, a non-parametric adaptation framework for frozen LLM agents. An actor interacts with an environment, while an evolver distills successful and failed trajectories into a bounded bank of reusable strategies and mistake corrections. Retrieved skills guide subsequent experiments and are progressively refined or combined. Source-task banks can also be synthesized into target-task guidance without target trajectories during construction. The paper additionally introduces Interphyre, a PHYRE-based physics benchmark with experimentation interfaces. Evaluation covers physical puzzles, interactive games, reasoning and planning, and physics-law discovery, using held-out instances with frozen skill banks, comparisons against memoryless and memory-based agents, component and feedback ablations, and cross-task transfer. Results generally show improved task success with smaller injected memory contexts than episodic-memory baselines.

Critical strengths
  1. The paper is well written. The article is very well-structured and coherent.

  2. The author does extensive and well-designed experiments to show the effectiveness of HExA.

Critical weaknesses
  1. The word“hierarchical” is not so appropriate. The claim that“later trajectories are generated while conditioned on previously learned skills, allowing subsequent skills to refine, compose, delimit, or supersede earlier ones.”does not mean the method is hierarchical. I think it is more like evolutionary or reflective.

  2. The paper lacks novelty because some concepts are very old for classical RL. However, considering the substantial effort involved in this article, I do not think the lack of novelty will affect the score too much.

Questions for authors

See weaknesses. I will consider raise my score if all my concerns are solved.

7 comments
AuthorsOpenReview ↗

Response to Reviewer igG2 (Part I/IV)

We thank the reviewer for the constructive feedback and for helpful clarifying questions. We provide new experiments and ablations to address all of them.

W1: “The word “hierarchical” is not so appropriate. The claim that“later trajectories are generated while conditioned on previously learned skills, allowing subsequent skills to refine, compose, delimit, or supersede earlier ones.”does not mean the method is hierarchical. I think it is more like evolutionary or reflective.”


We thank the reviewer for pointing this out, and we have updated the paper clarifying the difference. HExA performs hierarchical skill creation: higher level skills are constructed explicitly from previously learned sub-skills, or sub-skills are used implicitly by invocation inside actors context . We added ablations that distinguish both and show that skill evolution and hierarchical skill learning are complementary and each improves performance.

Hierarchical skill formation. Beyond skill evolution, HExA can reason over and build increasingly complex hierarchical skills as well. It does so either explicitly, by using learned sub-skills or implicitly by referencing and invoking them from previous time-steps and tasks. This hierarchical nature of HExA, enables creation of more general new skills across temporal and task abstractions that is crucial for cross-task transfer and generalization. We show additional results in the updated Table 3 (Table 18 in Appendix H.2.3) tests this on catapult with no target trajectories and every ablation uses the same evolver, prompt structure and output budget.

  • The new invocation-based hierarchical skills ablation enables the actor to directly invoke, inject and use source-level skills hierarchically without constructing explicit hierarchical skills. This already setup itself reaches 34.7%, showing that HExA’s hierarchical operationalization enables previously acquired skills to be used to support the actor on a harder unseen task.
  • Our final instantiation of HExA further synthesises hierarchical skills and improves this to 42.7%, an additional 8.0 points. It also exceeds independently adapting and concatenating the same source banks (25.3%) by 17.4 points, providing evidence for the benefit of integrating knowledge across sources. Table 30 (Appendix H.4.1) traces the synthesized Catapult skills to their source entries.
  • Thus, HExA exhibits hierarchical skill use in two complementary forms: the actor can directly reuse earlier skills, and the evolver can explicitly combine constituent skills into higher-level, target-grounded strategies. The new experiments establish the existence and utility of hierarchical skill formation concretely in the learned banks (Updated in paper Section 3.4, new Table 3; Appendix H.2.3 and H.4).
  • We also performed an ablation of skill creation using only the target-task description to study and help show how the importance of the source skills and the hierarchical skill creation process beyond the model’s existing knowledge and evolution abilities. As shown in the table below and in Table 3 in the paper and in the table below, Target-task description alone using the model's existing knowledge of skill evolution is insufficient to create good generalizable and transferable skills. This target task description-only ablation uses the same actor, evolver, target description, shared environment information, skill schema, and output/injection token budget. This setup leads to a 18.0 ± 2.0 success rate on the held-out test seeds

Hierarchical Skill Ablations

ConfigurationWhat changesHeld-out success (%)
HExABuild Hierarchical skills explicitly from three source banks42.7 ± 1.9
Invocation-based hierarchical skillsSkills from the three source banks referenced and invoked hierarchically34.7 ± 6.4
Per-level transferEach source adapted separately25.3 ± 10.9
Description-only synthesisNo source skill banks18.0 ± 2.0
ReActNo skills11.0 ± 3.0

AuthorsOpenReview ↗

Response to Reviewer igG2 (Part II/IV)

Within-level skill evolution. Previously acquired skills guide the actor's experiments, whose outcomes allow the evolver to refine, correct, and integrate earlier knowledge. Table 32 (Appendix H.4.2) gives an example in which the evolver merges two previously learned skills into a single skill. We isolate these mechanisms through new ablations.

  • Removing skill conditioning during training reduces held-out success by 19.4 points, showing that earlier skills improve subsequent experience collection. Reverting evolved skills to their earliest versions reduces success by 26.0 points, demonstrating the importance of progressively developing their content.

Skill Based Experimentation

ConfigurationWhat changesHeld-out success (%)Turns per seedDrop vs. HExA (pp)
Skill-free collectionBank hidden during collection63.3 ± 3.315.319.4
Earliest skill versionsSkill text reverted to earliest versions56.7 ± 16.512.526.0
HExA—82.7 ± 9.312.5—
  • Moreover, cumulative summarization (62.7%) and append-only updates (48.0%) remain substantially below full HExA, despite both retain accumulated experience. These controls support the combined benefit of skill-conditioned experimentation, revision, and integration over mere memory accumulation (updated in paper Section 3.3, Table 2; Appendix H.2.3, Table 17).

HExA vs Accumulated Summary Ablations

ConfigurationRetains accumulated lessonsInjected contextHeld-out success (%)Turns per seed
Summary bankYes~1.9k tokens62.7 ± 0.917.6
Append-only bankYesSame capacity as HExA48.0 ± 12.818.3
HExAYes1.5–2.3k tokens82.7 ± 9.312.5
  • We further compare against two recent memory-based agents, ACE [1] and ReasoningBank [2]. Both work by accumulating and summarizing lessons, without any explicit evolution abilities, and they perform significantly worse under the same experience budget: 50.0 ± 3.0% and 56.0 ± 10.7% on catapult against 82.7 ± 9.3% for HExA. We provide the additional results here as well as in the updated paper in Figure 3 and Table 10 and 11.

ACE and ReasoningBank with Claude Sonnet 4.6

LevelMethodSuccess (%)Avg itersTokens/ep.
catapultACE (playbook)50.0 ± 3.017.5 ± 1.69,960
catapultReasoningBank56.0 ± 10.716.3 ± 2.12,085
catapultHExA82.7 ± 9.312.5 ± 0.92,392
pass_the_parcelACE (playbook)84.0 ± 10.010.1 ± 2.14,243
pass_the_parcelReasoningBank80.0 ± 13.08.0 ± 1.91,989
pass_the_parcelHExA96.0 ± 2.85.8 ± 0.82,631

ACE and ReasoningBank with Qwen2.5 3B and 7B

Method7B DtE7B TBP3B DtE3B TBP
ReAct (no memory)63.3 ± 3.412.0 ± 0.08.3 ± 0.97.3 ± 0.9
ACE (playbook)62.7 ± 9.426.0 ± 7.52.7 ± 2.51.3 ± 0.9
ReasoningBank77.3 ± 5.728.7 ± 8.111.3 ± 1.98.0 ± 4.3
HExA82.7 ± 5.250.0 ± 3.323.3 ± 0.912.0 ± 0.0
AuthorsOpenReview ↗

Response to Reviewer igG2 (Part III/IV)

W2: “The paper lacks novelty because some concepts are very old for classical RL. However, considering the substantial effort involved in this article, I do not think the lack of novelty will affect the score too much.”


Learning through trial-and-error interaction, hierarchical reinforcement learning, and skill-learning are well-established ideas in classical RL. Although training pure RL agents to any general settings is very hard and inefficient and also using existing classical RL algorithms for skill learning and HRL to better train LLM agents is very hard to scale. We do not claim novelty in these foundational principles, and we appreciate the reviewer highlighting that we use these principles as inspiration in our method. We use them to propose and build hypothesis driven experimentation as a framework to build generalist LLM agents and HExA as a method to instantiate this framework of learning.

  • We first propose and formalize the hypothesis driven experimentation setup as a general in-context learning to learn and transfer framework and to establish the importance of learning from interactions and exploration guided by experiment grounded rewards and feedback. This grounded reward and feedback is essential and enables skill learning and evolution and enables us to work with a compressed efficient skill bank without continuously appending and increasing bank size without any reward guidance as done in ACE [1] and ReasoningBank [2] . These framework details are very crucial as they enable us to very efficiently learn on settings where the base model has no capabilities and also enables HExA to be the first method to instantiate in-context hierarchical skill learning. This creates a closed-loop learning process in which accumulated knowledge improves not only future decisions but also the experiments through which further knowledge is acquired.

  • This in-turn enables skill transfer and zero-shot self-improvement on unseen harder tasks, with the inner loop being skill-conditioned experimentation and an outer loop of skill evolution and cross-task knowledge synthesis, all without updating the LLM's parameters. This lets HExA perform a form of in-context meta-learning. In its inner loop, HExA adapts to one level by building that level's skill bank. Each meta-episode consists of n trajectories collected under the current bank, after which the evolver updates the bank. We instantiate n = 3 and run 17 meta-episodes per level, with each trajectory limited to 25 interaction turns. In its outer loop, the banks learned on the source levels are synthesized into a single bank for a level the agent has never seen, with no trajectories from that level. What is carried across levels is not a specific solution to any instance but knowledge of how to learn and improve via active experimentation. We clarify these in the updated paper Figure 1 and method.

  • Finally, we introduce Interphyre, an experimentation-centric physical reasoning benchmark designed to carefully evaluate these learning mechanisms in settings where static reasoning alone is insufficient and agents must actively acquire instance-specific evidence through interaction.


Applying HExA to Improve on Robot Manipulation

We also show HExA working very well in an agentic robot learning setup on LIBERO-PRO [3], outperforming Pigey [4], a React like physical agent orchestration method that had state of the art performance beating VLAs trained trained on Libero like π_0.5 [5]. HExA improves further, on average 44.2% increase relative to Pigey over all six LIBERO-PRO evaluation settings. This shows the generality of HExA and learning via experimentation. All details in Figure 5 and Table 23 in the updated paper.

MethodGoal taskGoal swapObject taskObject swapSpatial taskSpatial swap
π_0.524.436.410.416.856.447.6
Pigey30.443.228.026.070.451.6
HExA56.873.653.635.677.263.2
(HExA − Pigey)+26.4+30.4+25.6+9.6+6.8+11.6

[1] Zhang, Qizheng, et al. "Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. doi: 10.48550." arXiv preprint arXiv.2510.04618 (2025).

[2] Ouyang, Siru, et al. "Reasoningbank: Scaling agent self-evolving with reasoning memory." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Zhou, Xueyang, et al. "Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization." arXiv preprint arXiv:2510.03827 (2025).

[4] Galanti, Liane, et al. "Addressing the orchestration gap in generalist robots via physical agency." arXiv preprint arXiv:2607.21725 (2026).

[5] Intelligence, Physical, et al. "$\pi_ {0.5} $: a Vision-Language-Action Model with Open-World Generalization." arXiv preprint arXiv:2504.16054 (2025).

Reviewer_9fF9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HExA, a framework in which an LLM experiments with an environment, distills successful and failed attempts into a skill bank, and uses those skills to guide later experiments. The model’s weights remain fixed. The authors also introduce Interphyre, a set of interactive physics tasks, and evaluate on games, reasoning problems, and physics-law discovery. HExA improves performance on held-out instances while reducing interaction and token costs relative to memory baselines. Additional experiments examine the skill bank’s contents and transfer skills between tasks.

Critical strengths

The paper addresses an important problem and presents the idea clearly. The distinction between collecting experience, extracting useful lessons, and applying those lessons to new instances is easy to follow. On the two hardest Interphyre levels, HExA reaches approximately 83% and 96% success with Claude Sonnet, compared with 63% and 60% for episodic retrieval.

I liked that the experiments go beyond comparing final scores. Removing skills or mistake records, replacing relevant skills with wrong-task skills, and changing the available feedback help explain where the gains come from. The cost analysis also includes building the memory, rather than counting only its use at evaluation. Together with results across several task families, these experiments provide convincing support for the practical value of distilling experience into compact and reusable guidance.

Critical weaknesses

The benefit of hierarchical composition is less clearly established than the benefit of the skill bank overall. Rebuilding a bank from only the newest trajectories discards earlier experience, so it does not provide a strong comparison with a simpler summary that retains accumulated lessons. The transfer results are promising, but the evolver also receives the target-task description. A bank generated from that description alone would help show how much the source skills contribute beyond the model’s existing knowledge.

Questions for authors

How does HExA compare with a compact summary of accumulated lessons that uses the same experience and context budget, without the proposed skill-composition procedure?

For cross-task transfer, how much improvement remains over a bank generated from the target description alone, using the same evolver and output budget?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 9fF9 (Part I/IV)

We thank the reviewer for the constructive feedback and for helpful clarifying questions. We provide new experiments and ablations to address all of them.


W1: “The benefit of hierarchical composition is less clearly established than the benefit of the skill bank overall. ”;

W3: “The transfer results are promising, but the evolver also receives the target-task description. A bank generated from that description alone would help show how much the source skills contribute beyond the model’s existing knowledge.”;

AND

Q2: “For cross-task transfer, how much improvement remains over a bank generated from the target description alone, using the same evolver and output budget?”


We clarify the benefits of the hierarchical nature of HExA’s skill creation with additional experiments and ablations. In the updated paper, we also clarify the difference between skill evolution and hierarchical skill formation.

Within each level, skills evolve over training rounds, updates refine, merge or replace existing skills in the light of new experiments. Table 2 in the main paper (Table 17 in Appendix H.2.3) shows that this evolution gives the gains within a level and not merely having a skill bank only. We also have new baselines like ACE [1] and ReasoningBank [2] which distill lessons into growing memories without any explicit evolution abilities, and they perform worse. We show details of these in updated Figure 3 and Table 10, 11.

Hierarchical skill formation across tasks. HExA can reason over and build increasingly complex hierarchical skills as well. It does so either explicitly, by using learned sub-skills or implicitly by referencing and invoking them from previous time-steps and tasks. This hierarchical nature of HExA, enables creation of more general new skills across temporal and task abstractions that is crucial for cross-task transfer and generalization. We show additional results in the updated Table 3 (Table 18 in Appendix H.2.3) tests this on catapult with no target trajectories and every ablation uses the same evolver, prompt structure and output budget.

  • As suggested by you, we performed an ablation of skill creation using only the target-task description to study and help show how much the source skills contribute in hierarchical skill creation beyond the model’s existing knowledge. As shown in the table below and in Table 3 in the paper, Target-task description alone using the model's existing knowledge is insufficient to create good generalizable skill. This target task description-only ablation uses the same actor, evolver, target description, shared environment information, skill schema, and output/injection token budget.This setup leads to a 18.0 ± 2.0 success rate on the held-out test seeds, compared to HExA with explicit hierarchical skill creation leading to 42.7 ± 1.9 success rate. This clarifies the point reviewer made in W3 and Q2 beyond the evolver’s proper knowledge which is insufficient for transfer and generalization..
  • We also add another ablation to show the importance of proper hierarchical skill formation. In the Per-level transfer experiment, each source skill bank is adapted to the target task separately, so no target can hierarchically combine sub-skills from different levels. This drops performance to **25.3 ± 10.9 ** from 42.7 ± 1.9 for HExA, suggesting that the ability to build complex skills from sub-skills across levels is what makes transferred skills effective.
  • We also perform another ablation to show how implicit hierarchical skill creation itself is also very useful. To test whether the source skills can also be used hierarchically by the actor itself, we invoke and inject them as it is to actor's context (invocation-based hierarchical skills in table), where the actor invokes them directly while solving the task. This already reaches 34.7 ± 6.4, nearly twice the task description-only bank, so the actor can build on source skills without them being rewritten into target-level skills; HExA's explicit hierarchical skill synthesis adds a further 8.0 points.

Hierarchical Skill Ablations

ConfigurationWhat changesHeld-out success (%)
HExABuild Hierarchical skills explicitly from three source banks42.7 ± 1.9
Invocation-based hierarchical skillsSkills from the three source banks referenced and invoked hierarchically34.7 ± 6.4
Per-level transferEach source adapted separately25.3 ± 10.9
Description-only synthesisNo source skill banks18.0 ± 2.0
ReActNo skills11.0 ± 3.0
AuthorsOpenReview ↗

Response to Reviewer 9fF9 (Part II/IV)

W2: “Rebuilding a bank from only the newest trajectories discards earlier experience, so it does not provide a strong comparison with a simpler summary that retains accumulated lessons.”;

AND

Q1: “How does HExA compare with a compact summary of accumulated lessons that uses the same experience and context budget, without the proposed skill-composition procedure?”


We want to clarify that in the Iterative Replacement ablation, HExA does not discard earlier experience entirely, as these new rollouts are generated by an agent conditioned on the previous skill bank, which helps in better data generation (ie. the new rollouts) using which we build the new skill bank from scratch. This setup still leads to gains over other baselines and we wanted to highlight the usefulness of skill conditioned data collection via this setup. But as we have shown the optimal setup is to do this on top of the existing skill bank, which is our final Evolution setup in HExA.

To further clarify W2 and Q1 and to show strong comparison with a simpler summary that retains accumulated lessons, we added four new ablations in Table 2 updated in the paper also (with more details in Table 17 in Appendix H.2.3) to show the benefits of evolving, revising and composing skills in HExA over accumulating and summarizing lessons.

  • To distinguish evolution from merely accumulating experience, we compare against a summary bank that retains the same trajectory history and an append-only skill bank that cannot revise, merge or compose existing entries. These reach 62.7 and 48.0 respectively, despite continued learning under comparable memory budgets. Thus, HExA's gains arise from skill-conditioned experimentation and the progressive revision and integration of acquired knowledge, rather than experience retention alone. These controlled experiments establish the combined benefits of HExA.

HExA vs Accumulated Summary ablations

ConfigurationRetains accumulated lessonsInjected contextHeld-out success (%)Turns per seed
Summary bankYes~1.9k tokens62.7 ± 0.917.6
Append-only bankYesSame capacity as HExA48.0 ± 12.818.3
HExAYes1.5–2.3k tokens82.7 ± 9.312.5
  • To show that skill-conditioned experimentation and progressive refinement of skills both matter we also compare against skill-free collection, where the skill bank still evolver but is not given to the actor while collecting experience, and earliest skill versions where final skill bank skills are replaced by their earliest version from previous rounds. These reach 63.3 and 56.7 respectively.

Skill Based Experimentation

ConfigurationWhat changesHeld-out success (%)Turns per seedDrop vs. HExA (pp)
Skill-free collectionBank hidden during collection63.3 ± 3.315.319.4
Earliest skill versionsSkill text reverted to earliest versions56.7 ± 16.512.526.0
HExA—82.7 ± 9.312.5—
  • We also add two more memory baselines - ACE [1] and ReasoningBank [2] two recent relevant existing baselines which also work by accumulating and summarizing lessons, give the same picture under the same experience budget: 50.0 ± 3.0% and 56.0 ± 10.7% on catapult against 82.7 ± 9.3% for HExA. We provide the additional results here as well as in the updated paper in Figure 3 and Table 10 and 11.

ACE and ReasoningBank with Claude Sonnet 4.6

LevelMethodSuccess (%)Avg itersTokens/ep.
catapultACE (playbook)50.0 ± 3.017.5 ± 1.69,960
catapultReasoningBank56.0 ± 10.716.3 ± 2.12,085
catapultHExA82.7 ± 9.312.5 ± 0.92,392
pass_the_parcelACE (playbook)84.0 ± 10.010.1 ± 2.14,243
pass_the_parcelReasoningBank80.0 ± 13.08.0 ± 1.91,989
pass_the_parcelHExA96.0 ± 2.85.8 ± 0.82,631

ACE and ReasoningBank with Qwen2.5 3B and 7B

Method7B DtE7B TBP3B DtE3B TBP
ReAct (no memory)63.3 ± 3.412.0 ± 0.08.3 ± 0.97.3 ± 0.9
ACE (playbook)62.7 ± 9.426.0 ± 7.52.7 ± 2.51.3 ± 0.9
ReasoningBank77.3 ± 5.728.7 ± 8.111.3 ± 1.98.0 ± 4.3
HExA82.7 ± 5.250.0 ± 3.323.3 ± 0.912.0 ± 0.0
AuthorsOpenReview ↗

Response to Reviewer 9fF9 (Part III/IV)

HExA easily transfers and works in to new domain - Robot Manipulation

We also show HExA working very well in an agentic robot learning setup on a complex robot manipulation benchmark LIBERO-PRO [3], outperforming Pigey [4], a React like physical agent orchestration method that had state of the art performance on LIBERO-PRO beating VLAs trained on Libero like π_0.5 [5]. HExA leads to significant gains further on top of Pigey [4], on average 44.2% increase relative to Pigey over all six LIBERO-PRO evaluation settings. This shows the generality of HExA and learning via experimentation. All details in Figure 5 and Table 23 in the updated paper.

Performance on Robot Manipulation Benchmark LIBERO-PRO

MethodGoal taskGoal swapObject taskObject swapSpatial taskSpatial swap
π_0.524.436.410.416.856.447.6
Pigey30.443.228.026.070.451.6
HExA56.873.653.635.677.263.2
(HExA − Pigey)+26.4+30.4+25.6+9.6+6.8+11.6

[1] Zhang, Qizheng, et al. "Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. doi: 10.48550." arXiv preprint arXiv.2510.04618 (2025).

[2] Ouyang, Siru, et al. "Reasoningbank: Scaling agent self-evolving with reasoning memory." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Zhou, Xueyang, et al. "Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization." arXiv preprint arXiv:2510.03827 (2025).

[4] Galanti, Liane, Dhruv Shah, and Tri Dao. "Addressing the orchestration gap in generalist robots via physical agency." arXiv preprint arXiv:2607.21725 (2026).

[5] Intelligence, Physical, et al. "$\pi_ {0.5} $: a Vision-Language-Action Model with Open-World Generalization." arXiv preprint arXiv:2504.16054 (2025).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Rebuttal and Summary (I/II)

We thank all reviewers for their time, constructive feedback, and active discussions. We are happy that the reviewers found the paper well written and clearly structured, and that it addresses an important problem (9fF9,igG2). They highlighted HExA’s large gains on Interphyre’s hardest levels and on unseen instances against recent relevant baselines (9fF9,eFFR). They also valued the extensive, well-designed experiment including diverse ablations, feedback and cost analyses that explain where the improvements come from (9fF9, igG2, eFFR). We also appreciate that our rebuttals helped substantially addresse and solve their concerns and strengthens their assessment (9fF9, igG2, eFFR).

We thank reviewer 9fF9 for giving us review rating of 3 and for both reviewers igG2 and eFFR for increasing their review ratings to 3.

We would like to summarize our rebuttals, additional ablations and updated experiments here:


1. Hierarchical skill formation and skill evolution.

We have updated the paper to clarify the difference between skill evolution and hierarchical skill formation (Section 3.4), and we test both empirically with new ablations.

  • Skill evolution. Within each level, skills evolve over training rounds, updates refine, merge or replace existing skills in light of new experiments.
  • Hierarchical skill formation. HExA can reason over and build increasingly complex hierarchical skills from previously acquired skills. It does this either explicitly, by synthesizing new target skills from learned sub-skills, or implicitly, with the actor invoking source-level skills inside its own context.

The new cross-level ablations keep the same source knowledge, actor, evolver and output budget, use no target trajectories, and vary only how the skills are composed. Success increases with the degree of composition: none (25.3%), implicit (34.7%), explicit (42.7%). A bank generated from the target-task description alone reaches only 18.0%, so the model's existing knowledge is not sufficient for transfer. We show ablations below (also in Section 3.4, Table 3, Appendix H.2.3).

Hierarchical Skill Ablations

ConfigurationKnowledge availableComposition across levelsHeld-out success (%)
HExAThree source banksExplicit: new target skills composed from source skills42.7 ± 1.9
Invocation-based hierarchical skillsThree source banksImplicit: the actor composes source skills in its context34.7 ± 6.4
Per-level transferThree source banksNone: each bank adapted separately25.3 ± 10.9
Description-only synthesisTask description only—18.0 ± 2.0
ReActNone—11.0 ± 3.0

2. Comparison with retaining accumulated experience

To show the benefits of evolving, revising and composing skills in HExA over accumulating and summarising lessons we add more controlled ablations that replace HExA’s skill evolution with simpler alternatives.

  • Summary bank (same trajectories, matched context) and append-only bank (no revision) both retain all experience, yet reach only 62.7% and 48.0%, i.e., 20.0 and 34.7 points below HExA.
  • Hiding the bank during training (skill-free collection) costs 19.4 points and reverting skills to their earliest versions costs 26.0 points, showing that both skill-conditioned experimentation and progressive revision drive HExA's gains.

Within-level Ablations

ConfigurationWhat changesHeld-out success (%)Drop vs. HExA (pp)
Summary bankCumulative summary of lessons, ~1.9k tokens62.7 ± 0.920.0
Append-only bankNo revision of existing skills48.0 ± 12.834.7
Skill-free collectionBank hidden during collection63.3 ± 3.319.4
Earliest skill versionsSkill text reverted to earliest versions56.7 ± 16.526.0
HExA—82.7 ± 9.3—

(Updated in paper Section 3.3, Table 2; Appendix H.2.3, Table 17.)

New Baselines: ACE and Reasoning Bank

We added two recent agentic memory methods that automatically distill experience into reusable strategies, ACE [1] and ReasoningBank [2]. We run them with the same actor, warm-start and training seeds, and the same held-out seeds for evaluation. HExA achieves the highest success across all tested models and levels (updated Figure 3, Tables 10-11)

ACE and ReasoningBank with Claude Sonnet 4.6

LevelMethodSuccess (%)Avg itersTokens/ep.
catapultACE (playbook)50.0 ± 3.017.5 ± 1.69,960
catapultReasoningBank56.0 ± 10.716.3 ± 2.12,085
catapultHExA82.7 ± 9.312.5 ± 0.92,392
pass_the_parcelACE (playbook)84.0 ± 10.010.1 ± 2.14,243
pass_the_parcelReasoningBank80.0 ± 13.08.0 ± 1.91,989
pass_the_parcelHExA96.0 ± 2.85.8 ± 0.82,631
AuthorsOpenReview ↗

Common Rebuttal and Summary (II/II)

ACE and ReasoningBank with Qwen2.5 3B and 7B

Method7B DtE7B TBP3B DtE3B TBP
ReAct (no memory)63.3 ± 3.412.0 ± 0.08.3 ± 0.97.3 ± 0.9
ACE (playbook)62.7 ± 9.426.0 ± 7.52.7 ± 2.51.3 ± 0.9
ReasoningBank77.3 ± 5.728.7 ± 8.111.3 ± 1.98.0 ± 4.3
HExA82.7 ± 5.250.0 ± 3.323.3 ± 0.912.0 ± 0.0

3. Methodological Novelty and Contributions

We use ideas of learning through interaction and trial-and-error, hierarchical RL and skill learning from classical RL to draw inspiration and propose hypothesis-driven experimentation as a framework for building generalist LLM agents, and HExA as a method to instantiate this framework.

Skill-conditioned experimentation: HExA’s skill bank guides the actor to hypothesise and act, so accumulated knowledge also improves the experiments through which new knowledge is acquired.

Reward-guided evolution: rewards help keeping the knowledge accumulated bounded by deciding which skills are refined, merged and pruned.

Hierarchical composition and zero-shot transfer: skills learned on easier tasks are composed into skills for an unseen task without any target trajectories, a form of in-context meta-learning.

Interphyre: released an experiment-centric benchmark which gives acting agents tools to inspect the scene, simulate, intervene and observe dynamics, in settings where static reasoning alone is insufficient.

4. Robot manipulation on LIBERO-PRO

To show the generality of learning through experimentation, we also apply HExA to agentic robot manipulation on LIBERO-PRO [3]. HExA improves over Pigey [4], a recent LLM-orchestration method for physical agents that outperforms VLAs such as π0.5 [5], in all six evaluation settings, raising average success from 41.6% to 60.0%, a 44.2% relative gain (updated in paper Figure 5 and Table 23).

MethodGoal taskGoal swapObject taskObject swapSpatial taskSpatial swap
0.524.436.410.416.856.447.6
Pigey30.443.228.026.070.451.6
HExA56.873.653.635.677.263.2
(HExA − Pigey)+26.4+30.4+25.6+9.6+6.8+11.6

We thank all the reviewers for their positive assessment and review scores and for discussions during the rebuttal that helped improve the paper.

[1] Zhang, Qizheng, et al. "Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. doi: 10.48550." arXiv preprint arXiv.2510.04618 (2025).

[2] Ouyang, Siru, et al. "Reasoningbank: Scaling agent self-evolving with reasoning memory." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Zhou, Xueyang, et al. "Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization." arXiv preprint arXiv:2510.03827 (2025).

[4] Galanti, Liane, et al. "Addressing the orchestration gap in generalist robots via physical agency." arXiv preprint arXiv:2607.21725 (2026).

[5] Intelligence, Physical, et al. "$\pi_ {0.5} $: a Vision-Language-Action Model with Open-World Generalization." arXiv preprint arXiv:2504.16054 (2025).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.