- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HERAKLES, a hierarchical framework that jointly trains an LLM-based high-level planner and a lightweight low-level controller via reinforcement learning without predefined skill libraries. The method employs adaptive subgoal filtering to align the planner with the controller's current competence and compiles successful trajectories into reusable skills to amortize past learning. Experiments in the Crafter environment demonstrate that HERAKLES achieves superior sample efficiency and robust generalization to compositional and synonymous goals compared to existing flat and hierarchical baselines.
Critical strengths
- The selection of related work is relatively comprehensive.
- The writing is relatively clear.
Critical weaknesses
- While Appendix E.3 mentions mixing trajectories from other goals (), this simple uniform strategy may be insufficient as the goal space expands. I suggest the authors provide experiments showing that success rates for early, simple skills (e.g., "collect wood") remain stable during late-stage training on complex goals, rather than just reporting the overall score.
- The paper frames the coupling as a Stackelberg-like problem but relies on heuristic update frequencies without explaining why this specific asynchronous schedule ensures stable convergence. A discussion on the sensitivity of performance to the ratio of high-level to low-level updates would strengthen the methodological rigor.
- The use of an oracle for trajectory relabeling in the BOSS baseline (Appendix H.3) introduces external supervision. Does this lead to an unfair comparison with the fully self-supervised HERAKLES? I suggest adding an additional baseline variant for a fairer comparison.
- I suggest the authors include visualizations of failure cases to provide a more detailed analysis of why HERAKLES succeeds or fails in specific scenarios.
Questions for authors
Please refer to "Critical Weaknesses".
7 comments
Weakness 2 + question 2:
A discussion on the sensitivity of performance to the ratio of high-level to low-level updates would strengthen the methodological rigor.
In order to address the point raised by the reviewer we ran an additional experiment. We varied the high-level PPO update period (k) and the low-level update interval (m), pooling decisions between high-level updates and using (m) times as many AWR passes per low-level update to preserve the low-level gradient budget. We tested four schedules: the default ((k,m)=(1,1)), a slower low level ((1,4)), both levels slower ((2,2)), and a slower high level ((4,1)). These correspond to nominal low-level/high-level update ratios of (1), (1/4), (1), and (4), respectively. Each schedule was evaluated with five seeds using Qwen3-0.6B for 24 epochs, yielding 20 runs in total.
We added these new results to a new Appendix B.4. We made them also available at these links for easier reference.
- Crafter score by update schedule
- Crafter score against the total low level updates
- Size of the number of subgoals admitted
In the previous plots, the default schedule achieved the highest mean Crafter score, 10.9, compared with 7.1 when slowing the low level, 8.0when slowing both levels, and 9.3 when slowing the high level. Performance sensitivity is therefore asymmetric: slowing the low level causes the largest decrease, whereas slowing the high level has a smaller effect on the mean and increases variability across seeds. The lower score when both levels are slowed, despite preserving their nominal ratio, also indicates that absolute update frequency matters alongside the ratio.
To better understand this sensitivity, we examined the realized update counts. Across runs, Crafter score was positively correlated with the realized number of low-level updates per high-level update (Spearman (\rho=0.70), (p=0.001)). All three slower schedules also ended with fewer subgoals admitted to the high level’s action space than the default ((p\leq0.016) for each comparison). This association does not imply that increasing the nominal LL/HL ratio always improves performance: the default still outperforms the schedule with a slower high level.
Together, these observations are consistent with the two-timescale interpretation of the Stackelberg-like coupling: the high level acts as the slow leader, while the low level and its success estimates adapt between high-level updates and determine which subgoals are available to the leader. Timely low-level adaptation is therefore particularly important.** HERAKLES implements this coupling through epoch-wise high-level updates and data-triggered low-level updates, without explicitly tuning a fixed update ratio**; this default provides the best mean performance among the schedules tested.
Weakness 3 + question 3:
I suggest adding an additional baseline [BOSS without relabeling oracle] variant for a fairer comparison.
We thank the reviewer for its suggestion and conducted an experiment using a new baseline called BOSS-LLM. In BOSS-LLM, each chain of successfully executed skills is relabelled by the same LLM that acts as the high-level policy (Qwen3.5-4B) instead of the hand-written oracle, with greedy constrained decoding (a token lattice over the goal names) that forces the label to be one of the 23 goals in the oracle's label table, each of which has a low-level goal embedding. The few-shot prompt (10 worked examples) shows the chain's skill names and a state-change line computed from the same first and last textual observations the oracle reads: the items gained, the objects placed, and the object faced at the end of the chain. Because BOSS counts every executed elementary action as a successful step even when it changes nothing, the prompt tells the model to judge a chain by what it achieved rather than by its step names, and to answer "go to X" only when nothing was gained or placed, following the oracle's rule that the most advanced achievement wins.
BOSS-LLM scores lower because the LLM relabeller makes two kinds of mistakes, both of which steer BOSS's skill library the wrong way:
- It invents skills the agent never achieved. For instance, in about 1% of chains, a craft step ran but crafted nothing, and the LLM still labels the chain "make X pickaxe". BOSS adds every new label as a skill, so both runs ended up with making wood/stone/iron pickaxe. The high level then spent 22–24% of its skill calls, about 21% of all environment steps, on these three skills. They never succeeded once, and each failure ended the chain of skills.
- It misses the real progress. Only about 40–55% of the chains that actually collected wood were labelled "collect wood"; the rest mostly became "go to grass" (even if it is the last step of the trajectory it is not representative of what has been done in the trajectory). As a result, BOSS-LLM never learned a "place table" skill, the one that made BOSS oracle's scores jump.
We thank the reviewer for the attention and time spent on our paper. Below, we present additional experiments and clarifications to address the questions raised and provide a fuller understanding of our method and results.
Weakness 1 + question 1:
We thank the reviewer for this suggestion and ran the experiment. On Crafter with Qwen3.5-4B (2 seeds, 24 epochs, about 200k environment steps), we took the low-level policy saved at every epoch and tested it on its own, without the LLM, on three early skills: “go to tree”, “collect wood” and “place table”. Every checkpoint was tested on the same 200 new worlds, starting from an empty inventory.
We added these new results to a new Appendix B.5 and made them also available at this link for easier reference.
In late training (epochs 16–23), two thirds of the low-level updates go to harder skills, yet the success of the simple skills does not decrease. From the first epochs after a skill is unlocked to the end of training, “go to tree” goes from 0.84 to 0.89, “collect wood” from 0.67 to 0.73, and “place table” stays stable at 0.23–0.22. “place table” scores low here because the policy must also collect two pieces of wood within 64 steps. Forgetting does not occur because early skills are never frozen. They remain prerequisites of the complex goals, so the high level keeps calling them and the shared low-level policy keeps training on them.