- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes BEAT, which jointly adapts the skill organization and the low-level controllers of a hierarchical embodied agent, represented as a control tree of skills composed by sequential (SeqAND) and conditional (XOR) nodes. BEAT identifies a bottleneck skill from closed-loop rollouts and first trains it with PPO from restored skill-entry states; when learning plateaus, GPT-5.6 proposes a local SeqAND/XOR refinement, which is accepted or rolled back after matched evaluation against the incumbent. On a 30-task Minecraft suite with a frozen STEVE-1 controller, BEAT reaches 58.9% average success versus 51.1% for the strongest baseline. In Craftax, with controllers trained from scratch within 800M steps, it reaches 23.93% versus 20.41% on four goals.
Critical strengths
S1. Well-motivated problem with a local, inspectable refinement mechanism
The paper convincingly argues that skill granularity should match the current controller's capability, since overly coarse skills are hard to learn and overly fine skills burden the planner. BEAT replaces only the bottleneck leaf with a SeqAND/XOR subtree, and the case studies in Appendix F show readable refinements. This makes the method's behavior inspectable, which I consider a meaningful advantage over approaches that regenerate the whole plan.
S2. Consistent gains across three settings without LLM calls at test time
BEAT achieves the best average in every Minecraft tier with a frozen controller, the best four-goal average in Craftax from scratch, and gains on three of four Language Table tasks. The LLM is used only during development (three calls per Minecraft task, four for Lantern) and not at evaluation.
Critical weaknesses
W1. The gains are not attributable to structural evolution
BEAT trains skills with LLM-written local rewards and restores skill-entry states for about 90% of its budget, but the paper does not state whether the baselines receive the same shaping and resets, and apart from Language Table, where BEAT also revises its shaping, there is no KEEP-only ablation that disables only the refinement. In Minecraft, RL is added only after the structure is fixed, so co-evolution is not tested there either. Since shaping and resets alone can substantially help in sparse long-horizon tasks, the contribution of co-evolving structure and controllers is not isolated.
W2. The headline margins are statistically fragile
Craftax results appear to come from a single run, and 75% of the average gain over ARM-FM comes from CollectDiamond, while the other three goals differ by at most five successes out of 256 episodes. On CollectDiamond, the paper's own Strong-prior PPO ends slightly higher (Fig. 4) but is omitted from Table 2, and in Minecraft BEAT does not significantly outperform the Python-code variant under equal feedback (61.5% vs. 55.7% on six tasks, p$\approx$0.25). Together with W1, this leaves open whether the improvements are robust and specific to the SeqAND/XOR refinement.
Questions for authors
Q1. What reward, reset, and hyperparameter settings do PPO, PQN, and the Weak/Strong-prior variants use? In particular, do they receive the skill-entry resets and specification shaping that BEAT uses? For reference, prior work on Full Craftax (e.g., Scalar) reports considerably higher PPO success on EnterDungeon than the 42.97% reported here.
Q2. Section 3.3 refers to Appendix C for the acceptance criteria, but Appendix C.5 only states that GPT-5.6 receives the two estimates and the memory and returns PROMOTE/ROLLBACK. How are decisions made when both candidate and incumbent have near-zero success, as in the early phase of training in Fig. 4? How many proposals were made, accepted, and rolled back per goal, and what do the final trees look like?
Q3. How many seeds were run for Craftax, and what is the variance across seeds and across planner runs, given that the model is not pinned and no sampling seed is set (App. C.6)?
Q4. What adaptation budget does a candidate receive before matched evaluation, and are the adaptation and evaluation steps counted in the 800M budget?
Q5. How were the six representative Minecraft tasks chosen, what are the per-task results for the remaining 24, and what development budget (LLM calls, rollouts) did each baseline receive?
2 comments
Response to Reviewer kbkw (Part 2)
Q2/Q4: Acceptance, adaptation, and accounting. After candidate adaptation, both agents are frozen and evaluated on the same 256 fresh starts, with identical task objectives and episode limits. We promote a valid candidate if its complete-task success rate matches or exceeds the incumbent’s; otherwise, we roll it back. Zero–zero ties are accepted, allowing structural exploration before successful complete-task episodes emerge.
Each affected skill trains until consecutive non-improving blocks, where is skill depth, or its configured phase limit, within the remaining run budget. The 800M Craftax budget includes all training and development evaluations, including rejected-candidate adaptation and both sides of paired comparisons. Rollback does not reset the counter.
The complete Jukebox example contains eight executable leaves:
SeqAND ├── Diamond preparation: XOR │ ├── Diamond already held → Equip axe │ ├── Else, diamond block held → Unpack block and equip axe │ ├── Else, diamond ore detected → Acquire diamond via ore route │ └── Otherwise → Acquire diamond via block route ├── Wood acquisition: SeqAND │ ├── Obtain the first oak log │ └── XOR │ ├── Oak log detected within 32 blocks → Gather second log │ └── Otherwise → Search for and gather second log └── Convert logs to planks and craft the jukebox
Both second-log routes finish with at least two oak logs and hand control to the crafting routine. The Craftax example separates floor-1 combat from descent: after clearing eight hostiles, control passes to the skill responsible for reaching floor 2.
We will include per-goal proposal, acceptance, and rollback counts in the camera-ready version.
Q5: Task selection and development budgets. We have selected representative tasks. Specifically, for the focused ablation, we use two tasks from each difficulty tier: Bucket and Iron Nugget (Simple), Shield and Tripwire Hook (Medium), and Jukebox and Crossbow (Hard). These cover increasingly complex acquisition and crafting chains, from short sequences to multiple-resource coordination and longer dependencies. The main comparison evaluates all 30 tasks, with 32 held-out test episodes per task; the reported tier and overall averages include all tasks.
All methods use gpt-5.6-sol/xhigh with the same per-task development allocation: one initial specification and up to two revision requests, plus one additional revision for Lantern. This gives a cap of 91 top-level specification requests per method across the 30-task suite. Controller-training and development-evaluation interaction budgets are also matched per task.
We have incorporated the specific success rate of the task into the newly uploaded version.
If our responses have addressed your concerns, we hope that you could consider raising the score. Thank you for your time!
Response to Reviewer kbkw (Part 1)
Thank you for the careful review and your questions about experimental controls and statistical robustness.
W1/Q1: Shaping, resets, and structural adaptation. Strong Prior receives restoration: its SCALAR-derived chains use reset probability 0.9, curriculum threshold 0.3, 1,024 environments, 64 rollout steps, four PPO epochs, eight minibatches, and initial learning rate 2e-4. BEAT starts with one broad skill, using approximately 90% restored local learning and 10% fresh execution; Diamond uses specification shaping, other goals native-plus-specification reward.
BEAT's initial Craftax PPO uses learning rate 2e-4, gamma 0.99, GAE 0.8, clip 0.2, entropy 0.01, and separate actor/critic networks with three width-512 tanh layers (first shared across skills). Methods share observations, native success predicates, and 800M development interactions/run. We will tabulate PPO/PQN/Weak-Prior reward, reset, and hyperparameter settings in the camera-ready appendix; identical settings across methods are not assumed.
For the Minecraft ablation, we also correct the description of submitted Table 3: Point, Chain, Code, and full BEAT all trained controllers; full BEAT continued refinement. Controller-training protocols and interaction budgets are matched; Chain/Code/BEAT also share feedback/revision budgets.
Six-task success rates (%):
Full BEAT gains 17.2 percentage points over Code and improves on five of six tasks. These comparisons are descriptive; Minecraft SDs measure across-task variation. Point fixes one skill; a same-multiskill-tree KEEP-only control is missing, so a topology-only effect is not isolated.
W2/Q3: Repeated seeds and Strong Prior. To address the full-loop concern, we reran the 30-task Minecraft evaluation after submission with joint controller learning and structural revision. Full BEAT reaches 58.6%, versus 52.8% without RL and 53.6% MineEvolve. Testing uses 32 episodes/task, disjoint development/test seeds, common STEVE-1 and observation/GUI interfaces, and frozen agents. The six-task ablation uses separately developed agents. Minecraft SDs are across tasks; independent development-run variance is not established.
To assess repeatability, we also reran Craftax after submission with three training seeds per method/goal. Within 800M development interactions/run, 256 fresh-start development episodes select checkpoints; each selected policy is tested on 1,024 independent fresh-start episodes.
SDs are approximate sample SDs across three rounded seed results. BEAT's four-goal mean is 40.13% vs 22.68% for PPO-RNN, the strongest baseline by overall mean. Deeper-goal variability remains. We will incorporate these Minecraft and Craftax rerun results into the camera-ready version, with the full Craftax per-seed tables and learning curves in the appendix.