acceptodds
Under review as a conference paper at ICLR 2027

HERAKLES: Hierarchical Skill Compilation for LLM Agents in Compositional Goal Spaces

Abstract

LLM agents fine-tuned with reinforcement learning exploit common-sense knowledge to act in interactive environments, but typically decide every elementary action. In compositional goal spaces, where reaching a goal requires reaching others first, this forces credit assignment over horizons that grow with goal depth. This in turn leads to considerable decrease of sample efficiency as goals get more complex. We posit that the compositionality of behaviours follows that of their linguistic descriptions, so that an LLM is best used to reason over goals, while low-level routines run without language. Existing hierarchies built on this division rely on skills supplied by humans, or grow them without direction. We introduce HERAKLES, in which an LLM planner and a lightweight controller acting from raw observations are trained concurrently by RL, without any pre-trained skill library. The goal space is given, but its prerequisite structure is not: the planner selects intermediate goals among those the controller is estimated to master, and trajectories reaching a goal are compiled into the controller, turning mastered goals into options for harder ones and amortizing past learning. On Crafter, HERAKLES keeps improving where flat and hierarchical baselines plateau, and degrades markedly less on compositional variants of known goals.

Primary area:
reinforcement learning
Keywords:
llm agent, skill composition, sparse-reward Learning, hierarchical rl, language-guided planning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CB1pOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HERAKLES, a hierarchical framework that jointly trains an LLM-based high-level planner and a lightweight low-level controller via reinforcement learning without predefined skill libraries. The method employs adaptive subgoal filtering to align the planner with the controller's current competence and compiles successful trajectories into reusable skills to amortize past learning. Experiments in the Crafter environment demonstrate that HERAKLES achieves superior sample efficiency and robust generalization to compositional and synonymous goals compared to existing flat and hierarchical baselines.

Critical strengths
  1. The selection of related work is relatively comprehensive.
  2. The writing is relatively clear.
Critical weaknesses
  1. While Appendix E.3 mentions mixing trajectories from other goals (βsample=0.2\beta_{sample}=0.2), this simple uniform strategy may be insufficient as the goal space expands. I suggest the authors provide experiments showing that success rates for early, simple skills (e.g., "collect wood") remain stable during late-stage training on complex goals, rather than just reporting the overall score.
  2. The paper frames the coupling as a Stackelberg-like problem but relies on heuristic update frequencies without explaining why this specific asynchronous schedule ensures stable convergence. A discussion on the sensitivity of performance to the ratio of high-level to low-level updates would strengthen the methodological rigor.
  3. The use of an oracle for trajectory relabeling in the BOSS baseline (Appendix H.3) introduces external supervision. Does this lead to an unfair comparison with the fully self-supervised HERAKLES? I suggest adding an additional baseline variant for a fairer comparison.
  4. I suggest the authors include visualizations of failure cases to provide a more detailed analysis of why HERAKLES succeeds or fails in specific scenarios.
Questions for authors

Please refer to "Critical Weaknesses".

7 comments

We thank the reviewer for the attention and time spent on our paper. Below, we present additional experiments and clarifications to address the questions raised and provide a fuller understanding of our method and results.

Weakness 1 + question 1:

I suggest the authors provide experiments showing that success rates for early, simple skills (e.g., "collect wood") remain stable during late-stage training on complex goals

We thank the reviewer for this suggestion and ran the experiment. On Crafter with Qwen3.5-4B (2 seeds, 24 epochs, about 200k environment steps), we took the low-level policy saved at every epoch and tested it on its own, without the LLM, on three early skills: “go to tree”, “collect wood” and “place table”. Every checkpoint was tested on the same 200 new worlds, starting from an empty inventory.

We added these new results to a new Appendix B.5 and made them also available at this link for easier reference.

In late training (epochs 16–23), two thirds of the low-level updates go to harder skills, yet the success of the simple skills does not decrease. From the first epochs after a skill is unlocked to the end of training, “go to tree” goes from 0.84 to 0.89, “collect wood” from 0.67 to 0.73, and “place table” stays stable at 0.23–0.22. “place table” scores low here because the policy must also collect two pieces of wood within 64 steps. Forgetting does not occur because early skills are never frozen. They remain prerequisites of the complex goals, so the high level keeps calling them and the shared low-level policy keeps training on them.

Weakness 2 + question 2:

A discussion on the sensitivity of performance to the ratio of high-level to low-level updates would strengthen the methodological rigor.

In order to address the point raised by the reviewer we ran an additional experiment. We varied the high-level PPO update period (k) and the low-level update interval (m), pooling decisions between high-level updates and using (m) times as many AWR passes per low-level update to preserve the low-level gradient budget. We tested four schedules: the default ((k,m)=(1,1)), a slower low level ((1,4)), both levels slower ((2,2)), and a slower high level ((4,1)). These correspond to nominal low-level/high-level update ratios of (1), (1/4), (1), and (4), respectively. Each schedule was evaluated with five seeds using Qwen3-0.6B for 24 epochs, yielding 20 runs in total.

We added these new results to a new Appendix B.4. We made them also available at these links for easier reference.

In the previous plots, the default schedule achieved the highest mean Crafter score, 10.9, compared with 7.1 when slowing the low level, 8.0when slowing both levels, and 9.3 when slowing the high level. Performance sensitivity is therefore asymmetric: slowing the low level causes the largest decrease, whereas slowing the high level has a smaller effect on the mean and increases variability across seeds. The lower score when both levels are slowed, despite preserving their nominal ratio, also indicates that absolute update frequency matters alongside the ratio.

To better understand this sensitivity, we examined the realized update counts. Across runs, Crafter score was positively correlated with the realized number of low-level updates per high-level update (Spearman (\rho=0.70), (p=0.001)). All three slower schedules also ended with fewer subgoals admitted to the high level’s action space than the default ((p\leq0.016) for each comparison). This association does not imply that increasing the nominal LL/HL ratio always improves performance: the default still outperforms the schedule with a slower high level.

Together, these observations are consistent with the two-timescale interpretation of the Stackelberg-like coupling: the high level acts as the slow leader, while the low level and its success estimates adapt between high-level updates and determine which subgoals are available to the leader. Timely low-level adaptation is therefore particularly important.** HERAKLES implements this coupling through epoch-wise high-level updates and data-triggered low-level updates, without explicitly tuning a fixed update ratio**; this default provides the best mean performance among the schedules tested.

Weakness 3 + question 3:

I suggest adding an additional baseline [BOSS without relabeling oracle] variant for a fairer comparison.

We thank the reviewer for its suggestion and conducted an experiment using a new baseline called BOSS-LLM. In BOSS-LLM, each chain of successfully executed skills is relabelled by the same LLM that acts as the high-level policy (Qwen3.5-4B) instead of the hand-written oracle, with greedy constrained decoding (a token lattice over the goal names) that forces the label to be one of the 23 goals in the oracle's label table, each of which has a low-level goal embedding. The few-shot prompt (10 worked examples) shows the chain's skill names and a state-change line computed from the same first and last textual observations the oracle reads: the items gained, the objects placed, and the object faced at the end of the chain. Because BOSS counts every executed elementary action as a successful step even when it changes nothing, the prompt tells the model to judge a chain by what it achieved rather than by its step names, and to answer "go to X" only when nothing was gained or placed, following the oracle's rule that the most advanced achievement wins.

We added these new results to a new Appendix I.4 and made them also available at this link for easier reference.[

BOSS-LLM scores lower because the LLM relabeller makes two kinds of mistakes, both of which steer BOSS's skill library the wrong way:

  • It invents skills the agent never achieved. For instance, in about 1% of chains, a craft step ran but crafted nothing, and the LLM still labels the chain "make X pickaxe". BOSS adds every new label as a skill, so both runs ended up with making wood/stone/iron pickaxe. The high level then spent 22–24% of its skill calls, about 21% of all environment steps, on these three skills. They never succeeded once, and each failure ended the chain of skills.
  • It misses the real progress. Only about 40–55% of the chains that actually collected wood were labelled "collect wood"; the rest mostly became "go to grass" (even if it is the last step of the trajectory it is not representative of what has been done in the trajectory). As a result, BOSS-LLM never learned a "place table" skill, the one that made BOSS oracle's scores jump.
Reviewer_MW81OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HERAKLES, a hierarchical RL framework that enables LLM-based agents to progressively acquire, compile, and reuse skills for solving increasingly complex tasks.

The central idea is that an LLM should focus on high-level goal planning, while a lightweight neural network learns to execute those goals through primitive actions. Unlike existing hierarchical methods that require predefined or pretrained skills, HERAKLES automatically develops reusable skills through RL.

Its key innovation is hierarchical skill compilation: once the agent successfully completes a complex goal by composing simpler skills, the entire successful behavior is used to train the low-level controller to accomplish that goal directly. The newly acquired skill can then be reused when pursuing even more complex goals.

Critical strengths
  1. The most significant strength is the proposed mechanism for progressively compiling successful hierarchical trajectories into reusable low-level skills.

Rather than repeatedly planning and executing the same prerequisite sequences, HERAKLES trains its low-level controller to directly accomplish goals previously achieved through hierarchical composition. This progressively reduces the effective planning horizon for increasingly complex tasks.

  1. HERAKLES jointly trains an LLM-based high-level planner and a lightweight goal-conditioned low-level controller, without requiring a pretrained skill library or expert demonstrations. This allows both components to improve collaboratively, where the high-level planner learns to compose available skills, while the low-level controller gradually internalizes successful hierarchical behaviors.

  2. The paper introduces a learned competence estimator that predicts the low-level controller's ability to execute individual skills from the current observation. This creates an effective interface between planning and execution, reducing the risk of selecting skills the low-level controller cannot reliably perform.

Critical weaknesses
  1. The central mechanism—relabeling successful hierarchical trajectories to train a goal-conditioned low-level policy—is closely related to existing techniques such as Hindsight Experience Replay (HER), skill distillation, and BOSS. So, the novelty primarily comes from how these mechanisms are integrated into a continually expanding skill-learning system, rather than from a fundamentally new algorithmic principle.

  2. Authors do not provide sufficient ablation study to validate the core contribution. This is one of the most significant experimental weaknesses. The paper includes an ablation on mixing training samples from different goals, but it does not systematically isolate its three central contributions.

Possible missing ablation study could be: 1) train the low-level policy only on subgoal trajectories, without relabeling complete hierarchical trajectories; 2) freeze the high-level LLM and train only the low-level controller.

  1. The evaluation is restricted to a modified version of Crafter containing only 17 predefined goals. Furthermore, the authors simplify the original environment by removing survival-related challenges, including hostile entities and the need to eat or drink. So, experiments on one relatively constrained environment are insufficient to establish the method's effectiveness in general compositional goal spaces.

  2. The proposed compilation mechanism trains the low-level controller to execute a complex goal directly using trajectories generated by the hierarchical policy. However, these trajectories may be substantially longer than the maximum execution horizon allowed for an individual low-level skill.

For example, a complex goal may require several consecutive subgoals, each involving many primitive actions. Simply relabeling the complete trajectory does not ensure that the low-level controller can achieve the same goal within its limited execution horizon.

  1. Although the authors emphasize sample efficiency, HERAKLES introduces substantial computational overhead from jointly training the LLM planner, low-level controller, and competence-estimation components. As reported by authors, the proposed method requires approximately 3.3 times the GPU training time of POAD and twice that of BOSS under the reported configurations.

The improvements in environment sample efficiency may come at a substantial computational cost. The paper does not adequately analyze this trade-off or demonstrate scalability to large goal spaces.

Questions for authors
  1. What is the specific algorithmic novelty of hierarchical skill compilation compared with existing trajectory relabeling and skill bootstrapping methods such as HER and BOSS? Since HERAKLES relabels successful hierarchical trajectories to train the low-level controller, can the authors clarify what fundamentally distinguishes this mechanism from existing approaches?

  2. Given that both policies change during training and the high-level action space depends on the low-level controller's competence, can the authors provide empirical stability analyses or theoretical justification for the proposed timescale separation?

  3. Since hierarchical trajectories may be substantially longer than the maximum allowed low-level execution horizon, can the authors quantify how frequently compilation produces executable skills and evaluate the effectiveness of the trajectory-filtering heuristic?

  4. Since the evaluation is limited to 17 predefined goals in modified Crafter, can the authors demonstrate scalability to larger skill repertoires, deeper prerequisite chains, or environments with multiple alternative ways to achieve a goal?

  5. Does HERAKLES genuinely discover and exploit hidden prerequisite relationships, or does its performance primarily depend on the pretrained LLM's commonsense knowledge? Can the authors evaluate the method when prerequisite relationships differ from common linguistic expectations, or compare against a planner with explicitly provided prerequisite information?

12 comments

We thank the reviewer for their appreciation of our methodological contribution. In the following, we would like to address all questions raised by the reviewer and summarize several additions we made to the latest revision of our work:

Weakness 1 + question 1:

The central mechanism [...] is closely related to existing techniques such as Hindsight Experience Replay (HER), skill distillation, and BOSS.

Can the authors clarify what fundamentally distinguishes this mechanism [HERAKLES] from existing approaches [HER, BOSS]?

HERAKLES, HER, and BOSS share the idea of reusing collected experience to learn goals or skills. They tackle different parts of the problem, however. HER is a relabelling mechanism that better exploits the learning signal contained in failed trajectories. BOSS grows a skill library bottom-up. HERAKLES addresses, within a single architecture, which goals to pursue, how to compose skills to reach them, and how to compile successful compositions into new skills. Their main differences therefore lie in how goals are selected, how training trajectories are generated, and how they are labelled.

HER relabels experience with goals that were actually achieved, allowing an attempt that fails at its original goal to provide useful supervision for another. This improves sample efficiency and can induce an implicit curriculum. HER does not, however, address goal selection, and it is typically combined with uniform goal sampling. In large, structured goal spaces, where most goals are out of reach early in training, uniform sampling spends most episodes on goals the agent cannot yet achieve. HERAKLES instead uses an explicit curriculum (MAGELLAN), which focuses training on the goals with the highest learning progress. HER also provides no hierarchy to reduce the complexity of achieving goals: the number of actions needed to reach a goal grows with its depth. In HERAKLES, automatically compiled skills keep the number of high-level decisions small as goals deepen; for instance, "make wood pickaxe" can be achieved with a single high-level decision covering 57 environment steps (Appendix B.9). Finally, relabelling alone does not provide a mechanism for composing skills to reach increasingly complex goals. In Crafter, an attempt to "go to tree" is unlikely to produce a successful trajectory for "make wood pickaxe" without additional coordinated behavior. The challenge is therefore to generate experience that reaches such goals, so that useful supervision becomes available.

Both BOSS and HERAKLES address this challenge through hierarchical skill composition: an LLM chains skills executed by a low-level policy, and the resulting experience is used to train that policy on more complex behaviors. Their composition procedures nevertheless follow different principles. During skill bootstrapping, BOSS follows a bottom-up approach: it selects an initial skill using its value function, then uses the LLM to propose plausible continuations conditioned on the skills executed so far. It subsequently assigns a language description to the composite behavior by summarizing the constituent skill instructions and uses the collected experience for training.

HERAKLES follows a top-down approach: it starts from an explicit target goal and composes available skills to achieve it. When the composition succeeds, the resulting trajectory is compiled into a low-level training example labeled with that target goal. The target thus guides both the generation of the trajectory and the supervision used to learn from it.

This explicit goal conditioning can focus exploration on a small but relevant region of the state space. It allows the planner to select intermediate skills for their contribution to the intended objective, including sequences that may be unlikely under a continuation model conditioned only on the preceding skill history. We hypothesize that this distinction contributes to the pattern observed in Figure 3: in our Crafter experiments, BOSS does not master goals beyond “place table,” whereas HERAKLES progresses to “make stone pickaxe.” These results are consistent with the benefit of directing skill composition toward specific, increasingly complex goals.

In short, HER improves how collected trajectories are exploited, and BOSS grows a library of skills. HERAKLES combines goal selection, hierarchical composition and skill compilation in a single architecture, so that what it learns on one goal directly serves the next ones. Thus, the use of HER as a relabeler of failed trajectories is orthogonal to the use of HERAKLES.

Weakness 2:

Possible missing ablation study could be: 1) train the low-level policy only on subgoal trajectories, without relabeling complete hierarchical trajectories; 2) freeze the high-level LLM and train only the low-level controller.

Following the advice of the reviewer we conducted two additional ablations to further our comprehension of the HERAKLES method. We use Queen 3.5-4B and trained HERAKLES and the ablation for 200,000200,000 environment steps.

We added these new results to a new Appendix B.3 and made them also available at this link for easier reference.

In a first ablation we froze the high level resulting in a drop in the crafter score from 11.511.5 for HERAKLES to 7.47.4 with untrained LLM. Looking at the high level calls it appears that even if HERAKLES and the high level frozen ablation are calling the low level to execute some skills as often (respectively 16% and 18% of the calls for HERAKLES and the HL frozen ablation over the last five epochs) the ablation tends to call too often the wrong skills. For instance when herakles and the frozen baseline shared the same skill set the high level of the latter calls the skill “go to tree” twice as much as HERAKLES does. Thus through training the high level can discover and learn efficient combinations of subgoals that were possibly unlikely from its initial pretrained distribution.

For the ablation without relabelling we had to adapt the skill selection mechanism as the HERAKLES’s one was set such that a skill could not be proposed if it has not been updated at least two times, without relabelling a naive adaptation would have prevented the ablation to have access to any skill. In the no-hindsight ablation all the skills are always callable by the high level. When we look at the crafter score after 200,000200,000 environment steps it is at 10.710.7, below the HERAKLES one. Investigating further, the no-hindsight baseline fails differently from the frozen high level. Its high level learns which skills to call, but the skills are weaker and each decision costs about twice as many env steps. Indeed, observing the share of skill calls stopped at the 64-step limit HERAKLES is at 3535% and the no-hindsight baseline at 4444% resulting in 7070% of environment steps spent in failed skill calls. So at 200k steps no-hindsight baseline has had far fewer high-level decisions to learn from.

Question 2:

Can the authors provide empirical stability analyses or theoretical justification for the proposed timescale separation?

We thank the reviewers for these questions and ran a dedicated experiment. We varied the high-level PPO period (k ∈ {1, 2, 4} epochs, decisions pooled between updates) and the low-level update interval (m ∈ {1, 2, 4}, with m times more AWR passes per update, so gradient budgets are kept). This gave four schedules with nominal LL/HL update ratios of 1/4, 1 (default), 2/2 (both slower) and 4 low-level updates per high-level update. Each schedule ran with 5 seeds (20 runs, Qwen3-0.6B, 24 epochs).

We added these new results to a new Appendix B.4. We made them also available at these links for easier reference.

In the previous plots, we observe that the default schedule obtains the best mean score (10.9, against 7.1 with a 4× slower low level, 8.0 with both slower and 9.3 with a 4× slower high level). To better understand the mechanism we plot the crafter score against the total low level updates and observe that across runs the score rises with the realized number of low-level updates per high-level update (Spearman 0.70, p = 0.001). Moreover, looking at the size of the number of subgoals admitted in the high level’s action space it appears that every slower schedule ends with fewer subgoals (p ≤ 0.016) . Sensitivity is thus asymmetric. Slowing the low level, which is the fast follower that defines the leader's action space, costs the most; slowing the high level adds variance. This matches the two-timescale reading of the Stackelberg-like coupling: the high level is the slow leader, and the low level and its success estimates adapt in between. HERAKLES obtains this ordering without a tuned ratio by updating the high level every epoch and triggering low-level updates on collected data.

Question 3:

Can the authors quantify how frequently compilation produces executable skills and evaluate the effectiveness of the trajectory-filtering heuristic?

We measured how many relabeled trajectories are necessary to unlock a skill. We observed that for a skill to be unlocked at least 1616 successful trajectories have to be compiled (median 1818). Moreover, once unlocked, a skill works almost immediately. Indeed, 1212 of the 1313 skills that ever became reliable (≥5050% success over ≥2020 calls) did so 11–22 epochs after unlock.

The reviewer was also curious about the effectiveness of the trajectory-filtering heuristic. This heuristic keeps keep all the successful trajectories and drops a failed hierarchical trajectory if it is longer than 64/(1−ALP)64 / (1 − ALP) steps, the idea being that a failed trajectory form a skill on which we have a lot of training progress is more valuable (closer to a valid trajectory) than a skill on which we have no progress at all (empirically very long and often locked in loop trajectories). Under the filtering heuristic the share of failed-episode environment steps kept is of 1111%. Moreover, 3131–3535% of successful hierarchical trajectories are longer than the 6464 step, and they make up 8484–8989% of the successful training steps. Thus, the filtering contributes to transform compiled data that is 80–90% failures into data that is mostly successes.

To further the analysis we ablated this filter in two ways with Qwen3-0.6B. (i) Accept all keeps every trajectory, whatever its length or outcome. (ii) Reject > 64 steps keeps only trajectories of at most 64 steps, successful or not.

We added these new results to a new Appendix F.5 and made them also available at this link for easier reference.

Both ablations lower the Crafter score at epoch 23: HERAKLES reaches 13.4 (mean of its 3 best of 5 seeds), against 8.0 for accept all and 9.2 for reject > 64 steps (2 seeds each). Without the filter, long failed trajectories dominate the low-level data. Consistent with our previous measurement, the success on “collect stone” falls from 0.30 to 0.08. A length-only cut has the opposite problem: it also discards the long successful trajectories, which are most of the positive data for multi-step skills. With this cut, the success on “make wood pickaxe” falls from 0.50 to 0.18. Both parts of the filter therefore contribute: keeping long successes and discarding long failures.

Weakness 4:

The proposed compilation mechanism trains the low-level controller to execute a complex goal directly using trajectories generated by the hierarchical policy. However, these trajectories may be substantially longer than the maximum execution horizon allowed for an individual low-level skill.

The reviewer raises a critical point in the method about the compilation of skills. Indeed, successful hierarchical trajectories which are used to compile new skills in the low level network can be theoretically much longer than the maximal authorised length for a low level skill. However two mechanisms are at play such that learning these trajectories is beneficial overall. First, Even if the low level learned a skill from compiled trajectories longer than its execution horizon, the success-rate estimator controls when the skill is offered. Within a few epochs after unlocking, it learns to propose the skill mainly in states where it can complete within the allowed steps. For example, "go-to" skills are offered with probability ≈0.850.85 when the target is within 22 steps, against ≈0.120.12 when it is not visible. As a result, successful executions are short (median 66 steps), although a third of the successful compiled trajectories exceed 64 steps. A second phenomenon is that the hierarchical trajectories are often longer than optimal trajectories, the agent through the discounted reward signal of the different trajectories can learn a more efficient policy than the one that has generated them, maximizing the total reward (and thus shorter trajectories).

Reviewer_7TaMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HERAKLES starts with basic actions while jointly training an LLM planner and a controller that utilizes visual information within a given list of goals. It learns reusable skills by associating final goals with records of multi-step actions and determines which skills the planner can invoke based on estimates of success probability. Evaluated on the modified Crafter with 250,000 environment steps and five seeds, the system demonstrates continuous learning and transferability to variations of existing goals.

Critical strengths
  1. A useful contribution is the design that enables the planner to learn and create skills by retrieving records of actions taken to achieve goals. Unlike BOSS, which uses a fixed LLM to suggest the next skill during skill bootstrapping, HERAKLES trains the planner and creates skills based on records of actions taken to achieve the current goal. It starts with basic actions in the beginning.

  2. The results from repeated data collection concretely demonstrate the practical benefits of reuse. When switching from collecting one tree to collecting four, HERAKLES maintained a success rate of 0.92±0.05 to 0.91±0.06, while the modified BOSS dropped from 0.88 to 0.55 and POAD from 0.80 to 0.07. These results from five seeds provide positive evidence; however, comparisons between methods also require verification of the execution budget, as shown below.

Critical weaknesses
  1. It is not sufficiently clear how much of a benefit comes from training the planner alongside the other components. Since HERAKLES and BOSS differ in how they build skills, it is difficult to isolate the contribution of planner training based solely on a comparison of the two methods. A comparison in which all other components remain unchanged but the planner is not trained separately would help clarify this contribution.

  2. The evidence supporting transfer to other goals solved through similar behavioral processes is limited. Since the success rates for making a wooden sword are 0.40 ± 0.37 for HERAKLES and 0.33 ± 0.05 for BOSS, the statement in the text that the baselines fail at such transfer does not align with these results. Comparing a few additional goals would help determine whether HERAKLES’s advantage is consistent beyond this specific case.

Questions for authors
  1. Since BOSS also accumulates new skills by combining existing ones, I would like to better understand the additional benefits HERAKLES gains by training the planner as well. Could you compare this to a scenario where the planner is not trained separately, while maintaining the same configuration and learning interaction budget?

  2. In addition to crafting a wooden sword, could you compare the methods across several new tasks that reuse similar action sequences? Since Table 1 shows that the modified BOSS also succeeds at crafting a wooden sword, examining additional examples within the same Crafter framework would help us determine the range within which HERAKLES’s transfer benefits remain consistent.

  3. Could you confirm whether the same number of basic environmental actions per goal was allowed for each method in the evaluations shown in Figure 3 and Table 1? If the restrictions differed, a representative comparison under the same restrictions would help us fairly assess performance differences.

  4. SayCan combines the language model’s selection score with the feasibility of a skill. While maintaining the same planner and controller training methods, could you compare these selection rules with the proposed candidate skill filtering? This would help assess the additional benefits provided by filtering in a scenario where skills are continuously learned.

7 comments

We thank the reviewer for their interest in our method. We made several additions to the paper in response to their feedback and tried to address each one of their concerns, as summarized in our rebuttal below:

Weakness 1 + question 1:

A comparison in which all other components remain unchanged but the planner is not trained separately would help clarify this contribution.

I would like to better understand the additional benefits HERAKLES gains by training the planner as well.

To assess the contribution of planner training, we conducted an ablation comparing HERAKLES with a variant in which the high-level planner remains frozen throughout training. Both variants use Qwen3.5-4B and were trained for 200,000200,000 environment steps. Freezing the planner reduces final performance by 4.54.5 points in crafter score, supporting the importance of adapting the high-level policy during training.

We added these new results to a new Appendix B.3 and made them also available at this link for easier reference.

In a first ablation we froze the high level resulting in a drop in the crafter score from 11.511.5 for HERAKLES to 7.47.4 with untrained LLM. Looking at the high level calls it appears that even if HERAKLES and the high level frozen ablation are calling the low level to execute some skills as often (respectively 16% and 18% of the calls for HERAKLES and the HL frozen ablation over the last five epochs) the ablation tends to call too often the wrong skills. For instance when herakles and the frozen baseline shared the same skill set the high level of the latter calls the skill “go to tree” twice as much as HERAKLES does. Thus through training the high level can discover and learn efficient combinations of subgoals that were possibly unlikely from its initial pretrained distribution.

Weakness 2 + question 2:

Comparing a few additional goals would help determine whether HERAKLES’s advantage is consistent beyond this specific case.

Could you compare the methods across several new tasks that reuse similar action sequences?

To test whether the transfer benefits extend beyond crafting a wooden sword, we evaluated both methods on three additional compositional tasks, each expressed in two ways, giving six evaluation instructions in total. Each task requires the agent to reach a target while holding an item that it must first obtain. The shortest solutions involve chaining five or six training skills, combining resource collection, crafting, and navigation. For each task, the two instructions have exactly the same success condition. The explicit version names the required item, such as “Go to stone with wood pickaxe.” The implicit version states only the purpose, such as “Go to stone ready to mine,” requiring the agent to infer the necessary preparation.

Both methods were trained on the 17 standard Crafter goals and evaluated on these new compositions without further training. Both use Qwen3.5-4B as the high-level model. The table reports success rates (%) ± standard error.

Goal instructionPhrasingMinimum skillsHERAKLESBOSS
Go to table with woodExplicit578.8 ± 4.613.8 ± 3.9
Go to table ready to craftImplicit558.8 ± 5.515.0 ± 4.0
Go to stone with wood pickaxeExplicit626.3 ± 4.95.0 ± 2.4
Go to stone ready to mineImplicit612.5 ± 3.71.3 ± 1.2
Go to coal with wood pickaxeExplicit627.5 ± 5.01.3 ± 1.2
Go to coal ready to mineImplicit611.3 ± 3.50.0
Average across instructions35.86.0

HERAKLES achieves higher mean success on all three tasks under both phrasings, with an average success rate of 35.8%, compared with 6.0% for BOSS.

The paired instructions also reveal a limitation. Leaving the prerequisite implicit reduces HERAKLES’s success. The rollout logs suggest that the difficulty lies in identifying and carrying out the appropriate preparation. The agent continues to invoke navigation skills (and does it more frequently under the implicit instructions) but changes how it gathers and uses resources. Finally, HERAKLES shows higher mean success across all six tested instructions.

Question 3:

Could you confirm whether the same number of basic environmental actions per goal was allowed for each method in the evaluations shown in Figure 3 and Table 1?

In Figure 3 and Table 1, Herakles and its baselines (PPO, BOSS and FuN) were subjected to the same number of basic environmental actions (250,000250,000).

Question 4:

[SayCan] Could you compare these selection rules with the proposed candidate skill filtering?

Following the reviewer’s request, we implemented SayCan on top of the same skill-learning pipeline as HERAKLES, so the two methods differ only in how the high level selects actions. The low-level skills and their success estimator are trained exactly as in HERAKLES (same architecture, hindsight relabelling, skill-unlocking rule and Qwen3.5-4B LLM, about 200k environment steps, 2 seeds); the success estimator plays the role of SayCan's affordance function. At each high-level step, every elementary action and every unlocked skill is scored by its LLM probability multiplied by its estimated success (1 for elementary actions), and the action is sampled from this distribution.

We added these new results to a new Appendix B.7 and made them also available at this link for easier reference.

At evaluation in the plot above, SayCan reaches a Crafter score of 7.6, against 11.5 for HERAKLES. It matches HERAKLES on single-step tasks (collect wood: 0.94 vs 0.96) but falls behind as soon as skills must be chained (place table 0.59 vs 0.86, make wood pickaxe 0.44 vs 0.65, collect stone 0.05 vs 0.29).

We thank the reviewer once again for their insightful comments and would welcome any further questions or discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment

We thank the reviewers for their time and constructive feedback. We address each reviewer's questions in the individual responses. Following a request by Reviewer MW81, we tested our method in NetHack, a very difficult learning environment with  a wide range of entities to interact with or to combine (we have more than 1000 goals). The main changes are listed below and are shown in blue in the revised paper.

  • Role of planner training and of the LLM prior (Appendix B.3). Freezing the high-level LLM lowers the Crafter score from 11.5 to 7.4. The gap grows with prerequisite depth: success drops by 21% on goals with 1–2 prerequisite steps and by 90% on goals that require a pickaxe. RL on the planner turns the LLM's one-step knowledge into executable multi-step plans.

  • Ablation of skill compilation (Appendix B.3). Training the low-level policy without relabelled hierarchical trajectories lowers the score to 10.7. Skills are weaker, and each high-level decision costs about twice as many environment steps.

  • Comparison with SayCan (Appendix B.2). We implemented SayCan's selection rule on the same skill-learning pipeline, with the pretrained LLM at evaluation. It scores 7.6, against 11.5 for HERAKLES, and falls behind as soon as skills must be chained.

  • BOSS with LLM relabelling (Appendix I.4). Without the hand-written oracle, BOSS-LLM scores 3.4, against 6.1 for BOSS and 9.3 for HERAKLES at equal environment steps. Its relabeller invents skills the agent never achieved and misses real progress.

  • Generalization to new compositional goals (Appendix B.7). On six new instructions that name the required item or leave it implicit, HERAKLES reaches 35.8% average success, against 6.0% for BOSS.

  • Scaling to NetHack (Appendix C). In a goal space of 1,373 possible goals, HERAKLES achieves 91 and 75 distinct goals over two seeds and reaches deeper dungeon levels than BOSS.

  • Stability and update timescales (Appendices B.4 and B.5). Across 20 runs with four update schedules, no run diverges, and the default schedule obtains the best mean score. Slowing the low level costs the most, consistent with the two-timescale reading of the coupling. Early skills do not degrade during late training.

  • Compilation and trajectory filtering (Appendix F.5). We quantify how compiled trajectories become executable skills and ablate both parts of the filter. We also explain why compiled trajectories longer than the skill horizon remain useful.

  • Relation to HER and BOSS (Appendix D.3). We discuss the differences between HER, BOSS and HERAKLES

  • Qualitative analysis (Appendix B.9). We show key transitions of HERAKLES and BOSS in identical test worlds, including failure cases, and characterize where each method fails.

  • Computational cost (Appendix K.1). With the optimized implementation, HERAKLES needs about 47 GPU-hours for 200k environment steps, against about 82–85 for BOSS. With Qwen3-0.6B, it reaches a similar score in about 4.4 GPU-hours.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.