acceptodds
Withdrawn from ICLR 2027

BEAT: BOTTLENECK-GUIDED SELF-EVOLUTION OF EMBODIED AGENTS FOR LONG-HORIZON TASKS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon embodied tasks require skill organization that matches controller capabilities, yet skill decompositions are often fixed in advance. We introduce BEAT, a bottleneck-guided self-evolution framework that jointly adapts the planner and controller through execution feedback. Starting from a coarse control tree, BEAT identifies bottleneck skills through end-to-end rollouts and first improves their controllers with targeted online reinforcement learning. Fresh rollouts continually refresh skill-entry states, keeping local training aligned with the evolving agent. When local learning plateaus while failures persist, the planner refines skill granularity and task organization through sequential decomposition or conditional branching. Candidate structures are evaluated against the incumbent after controller adaptation using matched end-to-end rollouts. Across 30 Minecraft long-horizon tasks, BEAT achieves 58.9% average success, compared with 51.1% for the strongest evaluated baseline. Targeted controller adaptation further increases average success from 61.5% to 72.9% on a six-task subset. In Craftax, where controllers are trained from scratch, BEAT achieves 23.93% average success across four goals within an 800M-step budget, compared with 20.41% for the strongest evaluated baseline. These results support execution-driven adaptation of skill organization and controller capabilities for long-horizon tasks.

Primary area:
reinforcement learning
Keywords:
Embodied agents, Long-horizon tasks, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kbkwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BEAT, which jointly adapts the skill organization and the low-level controllers of a hierarchical embodied agent, represented as a control tree of skills composed by sequential (SeqAND) and conditional (XOR) nodes. BEAT identifies a bottleneck skill from closed-loop rollouts and first trains it with PPO from restored skill-entry states; when learning plateaus, GPT-5.6 proposes a local SeqAND/XOR refinement, which is accepted or rolled back after matched evaluation against the incumbent. On a 30-task Minecraft suite with a frozen STEVE-1 controller, BEAT reaches 58.9% average success versus 51.1% for the strongest baseline. In Craftax, with controllers trained from scratch within 800M steps, it reaches 23.93% versus 20.41% on four goals.

Critical strengths

S1. Well-motivated problem with a local, inspectable refinement mechanism
The paper convincingly argues that skill granularity should match the current controller's capability, since overly coarse skills are hard to learn and overly fine skills burden the planner. BEAT replaces only the bottleneck leaf with a SeqAND/XOR subtree, and the case studies in Appendix F show readable refinements. This makes the method's behavior inspectable, which I consider a meaningful advantage over approaches that regenerate the whole plan.

S2. Consistent gains across three settings without LLM calls at test time
BEAT achieves the best average in every Minecraft tier with a frozen controller, the best four-goal average in Craftax from scratch, and gains on three of four Language Table tasks. The LLM is used only during development (three calls per Minecraft task, four for Lantern) and not at evaluation.

Critical weaknesses

W1. The gains are not attributable to structural evolution
BEAT trains skills with LLM-written local rewards and restores skill-entry states for about 90% of its budget, but the paper does not state whether the baselines receive the same shaping and resets, and apart from Language Table, where BEAT also revises its shaping, there is no KEEP-only ablation that disables only the refinement. In Minecraft, RL is added only after the structure is fixed, so co-evolution is not tested there either. Since shaping and resets alone can substantially help in sparse long-horizon tasks, the contribution of co-evolving structure and controllers is not isolated.

W2. The headline margins are statistically fragile
Craftax results appear to come from a single run, and 75% of the average gain over ARM-FM comes from CollectDiamond, while the other three goals differ by at most five successes out of 256 episodes. On CollectDiamond, the paper's own Strong-prior PPO ends slightly higher (Fig. 4) but is omitted from Table 2, and in Minecraft BEAT does not significantly outperform the Python-code variant under equal feedback (61.5% vs. 55.7% on six tasks, p$\approx$0.25). Together with W1, this leaves open whether the improvements are robust and specific to the SeqAND/XOR refinement.

Questions for authors

Q1. What reward, reset, and hyperparameter settings do PPO, PQN, and the Weak/Strong-prior variants use? In particular, do they receive the skill-entry resets and specification shaping that BEAT uses? For reference, prior work on Full Craftax (e.g., Scalar) reports considerably higher PPO success on EnterDungeon than the 42.97% reported here.

Q2. Section 3.3 refers to Appendix C for the acceptance criteria, but Appendix C.5 only states that GPT-5.6 receives the two estimates and the memory and returns PROMOTE/ROLLBACK. How are decisions made when both candidate and incumbent have near-zero success, as in the early phase of training in Fig. 4? How many proposals were made, accepted, and rolled back per goal, and what do the final trees look like?

Q3. How many seeds were run for Craftax, and what is the variance across seeds and across planner runs, given that the model is not pinned and no sampling seed is set (App. C.6)?

Q4. What adaptation budget does a candidate receive before matched evaluation, and are the adaptation and evaluation steps counted in the 800M budget?

Q5. How were the six representative Minecraft tasks chosen, what are the per-task results for the remaining 24, and what development budget (LLM calls, rollouts) did each baseline receive?

2 comments
AuthorsOpenReview ↗

Response to Reviewer kbkw (Part 1)

Thank you for the careful review and your questions about experimental controls and statistical robustness.

W1/Q1: Shaping, resets, and structural adaptation. Strong Prior receives restoration: its SCALAR-derived chains use reset probability 0.9, curriculum threshold 0.3, 1,024 environments, 64 rollout steps, four PPO epochs, eight minibatches, and initial learning rate 2e-4. BEAT starts with one broad skill, using approximately 90% restored local learning and 10% fresh execution; Diamond uses specification shaping, other goals native-plus-specification reward.

BEAT's initial Craftax PPO uses learning rate 2e-4, gamma 0.99, GAE 0.8, clip 0.2, entropy 0.01, and separate actor/critic networks with three width-512 tanh layers (first shared across skills). Methods share observations, native success predicates, and 800M development interactions/run. We will tabulate PPO/PQN/Weak-Prior reward, reset, and hyperparameter settings in the camera-ready appendix; identical settings across methods are not assumed.

For the Minecraft ablation, we also correct the description of submitted Table 3: Point, Chain, Code, and full BEAT all trained controllers; full BEAT continued refinement. Controller-training protocols and interaction budgets are matched; Chain/Code/BEAT also share feedback/revision budgets.

Six-task success rates (%):

VariantBucketIron NuggetShieldTripwire HookJukeboxCrossbowMean
Point9.40.00.00.00.00.01.6
Chain87.578.162.531.318.843.853.6
Code81.396.956.325.037.537.555.7
BEAT without RL90.690.668.837.543.837.561.5
Full BEAT87.590.684.459.465.650.072.9

Full BEAT gains 17.2 percentage points over Code and improves on five of six tasks. These comparisons are descriptive; Minecraft SDs measure across-task variation. Point fixes one skill; a same-multiskill-tree KEEP-only control is missing, so a topology-only effect is not isolated.

W2/Q3: Repeated seeds and Strong Prior. To address the full-loop concern, we reran the 30-task Minecraft evaluation after submission with joint controller learning and structural revision. Full BEAT reaches 58.6%, versus 52.8% without RL and 53.6% MineEvolve. Testing uses 32 episodes/task, disjoint development/test seeds, common STEVE-1 and observation/GUI interfaces, and frozen agents. The six-task ablation uses separately developed agents. Minecraft SDs are across tasks; independent development-run variance is not established.

To assess repeatability, we also reran Craftax after submission with three training seeds per method/goal. Within 800M development interactions/run, 256 fresh-start development episodes select checkpoints; each selected policy is tested on 1,024 independent fresh-start episodes.

GoalBEAT mean +/- SD (%)Best baseline mean (%)
CollectDiamond24.35 +/- 5.4712.01 (Strong Prior)
EnterDungeon90.07 +/- 2.4878.84 (PPO-RNN)
UnlockGnomishLadder30.11 +/- 14.409.41 (Strong Prior)
EnterGnomishMines15.98 +/- 6.601.24 (Strong Prior)

SDs are approximate sample SDs across three rounded seed results. BEAT's four-goal mean is 40.13% vs 22.68% for PPO-RNN, the strongest baseline by overall mean. Deeper-goal variability remains. We will incorporate these Minecraft and Craftax rerun results into the camera-ready version, with the full Craftax per-seed tables and learning curves in the appendix.

AuthorsOpenReview ↗

Response to Reviewer kbkw (Part 2)

Q2/Q4: Acceptance, adaptation, and accounting. After candidate adaptation, both agents are frozen and evaluated on the same 256 fresh starts, with identical task objectives and episode limits. We promote a valid candidate if its complete-task success rate matches or exceeds the incumbent’s; otherwise, we roll it back. Zero–zero ties are accepted, allowing structural exploration before successful complete-task episodes emerge.

Each affected skill trains until 2+2d2+2d consecutive non-improving blocks, where dd is skill depth, or its configured phase limit, within the remaining run budget. The 800M Craftax budget includes all training and development evaluations, including rejected-candidate adaptation and both sides of paired comparisons. Rollback does not reset the counter.

The complete Jukebox example contains eight executable leaves:

SeqAND ├── Diamond preparation: XOR │ ├── Diamond already held → Equip axe │ ├── Else, diamond block held → Unpack block and equip axe │ ├── Else, diamond ore detected → Acquire diamond via ore route │ └── Otherwise → Acquire diamond via block route ├── Wood acquisition: SeqAND │ ├── Obtain the first oak log │ └── XOR │ ├── Oak log detected within 32 blocks → Gather second log │ └── Otherwise → Search for and gather second log └── Convert logs to planks and craft the jukebox

Both second-log routes finish with at least two oak logs and hand control to the crafting routine. The Craftax example separates floor-1 combat from descent: after clearing eight hostiles, control passes to the skill responsible for reaching floor 2.

We will include per-goal proposal, acceptance, and rollback counts in the camera-ready version.

Q5: Task selection and development budgets. We have selected representative tasks. Specifically, for the focused ablation, we use two tasks from each difficulty tier: Bucket and Iron Nugget (Simple), Shield and Tripwire Hook (Medium), and Jukebox and Crossbow (Hard). These cover increasingly complex acquisition and crafting chains, from short sequences to multiple-resource coordination and longer dependencies. The main comparison evaluates all 30 tasks, with 32 held-out test episodes per task; the reported tier and overall averages include all tasks.

All methods use gpt-5.6-sol/xhigh with the same per-task development allocation: one initial specification and up to two revision requests, plus one additional revision for Lantern. This gives a cap of 91 top-level specification requests per method across the 30-task suite. Controller-training and development-evaluation interaction budgets are also matched per task.

We have incorporated the specific success rate of the task into the newly uploaded version.

If our responses have addressed your concerns, we hope that you could consider raising the score. Thank you for your time!

Reviewer_idfYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper proposes BEAT: a hierarchical framework where both the skill structure and the skill controllers improve from execution feedback. The agent is an executable tree with skills as leaves, each having a conpletion condition, local reward and time limit. In rollout, the skill that limit task success is identified and trained with PPO.

  • Result shows that adding RL for bottleneck skills raises the average by about 10% on the measured tasks.

Critical strengths
  • Well-motivated problem: coarseness of skills matters.

  • Evaluation design is sound. There is a diverse range of experiments spanning various settings (frozen pretrained controller, learning from scratch, continuous control) and baselines.

Critical weaknesses
  • Incremental novelty. Compared to papers that introduced each ingredient (LLM generating/updating behavior trees, revising skills or reward from experience, training from saved states), the new parts (RL first, restructure on plateau) are more marginal and are not evaluated with targeted ablation to showcase the necessity and contribution of each component (see last bullet point).

  • The headline Minecraft result doesn’t test the joint planner–controller evolution the paper claims. In the main 30-task comparison the controller is frozen so it's just the LLM plan revision from execution feedback in a SeqAnd/XOR format, same as Reflexion. The controller half is tested separately. The two parts are only jointly evolved in Craftax and the appendix Language Table study. The 58.9% vs 51.1% result mainly supports the tree representation but not the coupled training loop.

  • No variance across runs for the main results, and several margins are small. Neither Minecraft nor Craftax reports repeated runs. In Craftax, 75% of the 3.5-point average lead comes from CollectDiamond (62 vs 35 successes out of 256); without it BEAT averages 23.8% vs 22.7% for ARM-FM. The other Craftax goals differ by a few episodes (EnterGnomishMines 3 vs 2; UnlockGnomishLadder 15 vs 12; EnterDungeon 165 vs 166 for SkillRL).

  • All Minecraft baselines are the authors’ own adaptations to MineStudio and STEVE-1. Also it’s unclear whether they get the same budget as BEAT. The goal-only GPT-5.6 baseline (9.4%) gets no task decomposition, which makes it a weak reference point.

  • Need more ablation to show the contribution of each key components:

    • 90/10 split of restored local runs vs fresh runs, compared with training from the start state only
    • doing RL before restructuring, compared with restructuring immediately
    • picking bottleneck skill with various strategy. The paper claims that having LLM pick it from learning curves and failure trajectories is "more informative" than selecting a leaf solely by its raw failure count. But this is never tested. Simple rules like failure/success rate should be tested to show it is necessary to have LLM pick the bottleneck skill. the accept/reject step is also untested: GPT-5.6 decides from the two success rates whether to keep a structural change. A fixed threshold, or a paired statistical test on the matched runs, should be compared.
Questions for authors
  • line 344 typo: "Reflection" should be "Reflexion".

  • What is the LLM sampling seed and temperature? What are the token counts for each experiment?

  • A more general question: does the approach work without some kind of structured state? (e.g. in this paper, Minecraft tasks start in a flat 17×17 area, with tools, crafting stations and fuel supplied where needed. Crafting and smelting are done by scripted GUI skills, not learned. Craftax uses symbolic observations rather than pixels. )

  • Checkpoint selection is rather optimistic. In Craftax, each method is scored at the checkpoint with the highest training-time success anywhere in its 800M steps, then re-evaluated on 256 new episodes. The fresh re-evaluation is good and is consistently applied to every method, however final-checkpoint results would be more standard. It would be interesting to see how much of the performance came from this optimistic checkpoint selection.

2 comments
AuthorsOpenReview ↗

Response to reviewer idfY (Part 1)

W1: Contribution. BEAT's key idea is to adapt skill definitions to what the controllers can learn, while training controllers to execute the revised skills. Both sides respond to each other, beyond revising plans with fixed controllers or training controllers under a fixed organization.

For example, a logically valid subgoal may exceed the current controller's capabilities; learning feedback then guides the planner to split or redefine it. Conversely, changing skill boundaries or handoff conditions changes what the affected controllers must learn. Our contribution is this bidirectional adaptation framework. RL-first and plateau triggering coordinate this interaction.

The revised paper supports this argument through adaptive versus fixed specifications in Craftax, complementary organization/controller ablations in Minecraft, and robotics experiments. W5 below summarizes these three groups of evidence.

W2/W3: Full-loop results and seed variability. The revised 30-task Minecraft experiment directly tests joint structural revision and controller learning. In this post-submission rerun, full BEAT reaches 58.6%, versus 52.8% without RL and 53.6% for MineEvolve (Table 1). Controllers learn while the structure evolves; both are frozen for final testing. Testing uses 32 episodes/task, disjoint development/test seeds, and shared STEVE-1 and observation/GUI interfaces. The six-task ablation uses separately developed agents.

The Craftax reruns measure variation across three training seeds per method/goal, with gains on all four goals. Each run has an 800M-interaction development budget. Checkpoints are selected using 256 fresh-start development episodes and tested on 1,024 independent fresh-start episodes.

GoalBEAT mean +/- SD (%)Best baseline mean (%)
CollectDiamond24.35 +/- 5.4712.01 (Strong Prior)
EnterDungeon90.07 +/- 2.4878.84 (PPO-RNN)
UnlockGnomishLadder30.11 +/- 14.409.41 (Strong Prior)
EnterGnomishMines15.98 +/- 6.601.24 (Strong Prior)

BEAT's four-goal mean is 40.13% versus 22.68% for PPO-RNN, the strongest baseline overall. Excluding Diamond, the means remain 45.39% versus 26.29%. The SDs above are approximate sample SDs from three rounded seed results; Minecraft's SDs describe variation across tasks. Full Craftax per-seed results and learning curves appear in Table 11 and Figure 3.

W4/Q2: Baselines, models, and development cost. All Minecraft methods share the same LLM, execution interfaces, and development budgets. Every main-comparison and organization-ablation method uses gpt-5.6-sol / xhigh, including BAR-static, PSN, and MineEvolve (Appendices C.6/E.1). Their adaptations preserve their planning logic within MineStudio/STEVE-1, with common observations, GUI skills, test starts, and episode limits. Controller and GUI actions both count toward the interaction budget.

Each method receives one initialization and at most two revision requests per task; Lantern permits one extra revision. This is a cap of 91 specification requests per method across 30 tasks, excluding transport retries and earlier pilots. Controller-training and development-evaluation budgets are also matched per task; methods apply their own update rules within these limits. The six-task ablation matches training protocols and interaction budgets; Chain, Code, and BEAT also share feedback/revision budgets. The goal-only model (11.1% in Table 1) is a no-decomposition reference. The main evidence also includes stronger decomposing baselines and controlled ablations.

We did not set an LLM sampling seed or override temperature/top-p. Full Craftax uses gpt-6.1-sol / xhigh; model identifiers are not pinned to dated snapshots (Appendix C.6). Logged usage is approximately 1.1-3.8 million tokens per Minecraft task or single-seed Craftax run, incurred during development. Accounting separates uncached input, cached input, and billed output, including reasoning. Final testing executes frozen agents with zero high-level LLM calls.

W5: Component evidence and design choices. Our core contribution is the joint adaptation of skill organization and controllers, while the remaining ablation suggestions concern secondary design choices. Section 4.3 and the appendix provide complementary evidence for this contribution:

  • Adaptive skill specifications (Craftax): adaptive organization and shaping versus fixed coarse and multi-skill priors, with policy learning enabled.
  • Skill organization (Minecraft): Point, Chain, Code, and SeqAND/XOR structures under matched controller training.
  • Controller adaptation (Minecraft): structural refinement with and without controller learning.
  • Robotics: adaptive versus fixed task specifications.
AuthorsOpenReview ↗

Response to reviewer idfY (Part 2)

Q1: Typo. Thank you for your correction.

Q3: Scope and structured observations. Structured representation is a commonly adopted design choice that can reduce the learning burden. By doing this, our method focuses on learning the coevolution of planner and controller. In Craftax and LanguageTable, the main information still comes from the pixel observations of the environment. Related works such as DEPS and Voyager also use structured environmental information to assist in planning. Learning directly from pixels introduces additional perceptual complexity and may require more data and computing resources.

Q4: Checkpoint selection.

The budget counts ordinary training, adaptation of accepted and rejected candidates, both sides of candidate comparisons, and checkpoint development evaluations. All methods follow the same selection protocol.

If our responses have addressed your concerns, we hope that you could consider raising the score. Thank you for your time!

Reviewer_4PA4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BEAT, a framework for embodied agents to complete complex long-horizon tasks. It combines a large language model for task decomposition with reinforcement learning for skill improvement, using execution feedback to adapt how tasks are broken down to the agent's capabilities. The authors evaluate the framework in Minecraft, Craftax, and Language Table and report improvements over several baselines.

Critical strengths

The paper addresses an important problem: adapting task plans to the capabilities of embodied agents. Its integration of language-model planning and reinforcement learning is interesting. Its reported performance in Minecraft and Crafta improves over several baselines.

Critical weaknesses
  1. The added value of the prescribed revision strategy is insufficiently demonstrated. Failure-driven decomposition already appears in Agentic Skill Discovery cited in the paper. BEAT trains a selected bottleneck and considers local tree edits when learning plateaus. Figure 4 compares against fixed decompositions, while Table 3 compares organization variants with frozen controllers and separately adds RL with organization fixed. These experiments do not isolate the benefit of plateau-triggered revision over simpler schedules, such as periodic revision or revision when a skill exhausts its training budget without reaching a target success rate (similar to SCALAR), under matched feedback and budgets. Missing planner prompts and schemas also make the revision process difficult to assess and reproduce.

  2. Evidence for the full RL/refinement loop and its repeatability is limited. The 30-task Minecraft comparison freezes controllers and bypasses RL-plateau detection; its six-task RL ablation fixes task organization. Full-loop evidence therefore comes mainly from four Craftax goals, with a supplementary four-task Language Table study showing mixed results. Independent training/development-run counts and variability are not specified for the main Craftax and Minecraft results; Minecraft's SD instead measures variation across tasks. Figure 3 also shows pronounced fluctuations on the deeper Craftax goals. These gaps leave uncertainty about the robustness of the proposed method.

Questions for authors
  1. How does plateau-triggered revision compare with periodic revision or revision after a skill exhausts its training budget below a target success rate?
  2. Can you provide the planner prompts/schemas and one complete refinement trace?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 4PA4 (Part 1)

Thank you for the constructive review and for highlighting the need for clearer protocols and stronger evidence of repeatability.

W1/Q1: Joint adaptation and revision timing. Plateau triggering is a scheduling design choice, not our core contribution. Our core contribution is the co-evolution of planner structure and controller capabilities to keep them aligned. The rollout allocation, RL-first schedule, bottleneck selection, and acceptance rules are design choices for implementing this principle.

Section 4.3 and the appendix provide the following ablations:

  • Adaptive skill specifications (Craftax): adaptive organization and shaping versus human-designed and SCALAR-based fixed priors.
  • Skill organization (Minecraft): Point, Chain, Code, and SeqAND/XOR structures with fixed controllers.
  • Controller adaptation (Minecraft): targeted RL for bottleneck skills while holding the high-level organization fixed.
  • Robotics (appendix): adaptive versus fixed task specifications.

These studies support the complementary roles of skill organization and controller adaptation in our framework. Alternative implementation choices may further improve their coordination; we leave their systematic exploration to future work.

We also correct an error in the description of submitted Table 3: Point, Chain, Code, and full BEAT all trained controllers, and full BEAT continued structural revision during training. Controller-training protocols and interaction budgets were matched; Chain, Code, and BEAT also shared feedback and revision budgets. The six-task results are:

Six-task ablationSuccess (%)
Point1.6
Chain53.6
Code55.7
BEAT without RL61.5
Full BEAT72.9

Compared to the code, the 17.2% improvement supports the value of the control tree.

The complete planner prompts are now provided in Appendix H of the newly submitted version.

W2: Full-loop results and repeatability. To address the full-loop concern, we reran the 30-task Minecraft evaluation after submission with joint controller learning and structural revision. Full BEAT reaches 58.6%, versus 52.8% without RL and 53.6% MineEvolve. Testing uses 32 episodes/task, disjoint development/test seeds, common STEVE-1 and observation/GUI interfaces, and frozen agents. The six-task ablation uses separately developed agents. Minecraft SDs are across tasks; independent development-run variance is not established.

To assess repeatability, we also reran Craftax after submission with three training seeds per method/goal. Within 800M development interactions/run, 256 fresh-start development episodes select checkpoints; each selected policy is tested on 1,024 independent fresh-start episodes.

GoalBEAT mean +/- SD (%)Best baseline mean (%)
CollectDiamond24.35 +/- 5.4712.01 (Strong Prior)
EnterDungeon90.07 +/- 2.4878.84 (PPO-RNN)
UnlockGnomishLadder30.11 +/- 14.409.41 (Strong Prior)
EnterGnomishMines15.98 +/- 6.601.24 (Strong Prior)

SDs are approximate sample SDs across three rounded seed results. BEAT's four-goal mean is 40.13% vs 22.68% for PPO-RNN, the strongest baseline by overall mean. Deeper-goal variability remains. We will incorporate these Minecraft and Craftax rerun results into the camera-ready version, with the full Craftax per-seed tables and learning curves in the appendix.

AuthorsOpenReview ↗

Response to Reviewer 4PA4 (Part 2)

Q2: Planner interface and refinement example. The frozen LLM uses the following three prompt templates, populated with the current task, interfaces, execution evidence, and memory.

Skill Initialization Prompt

You are the high-level planner of an embodied agent. Construct an executable control tree for the supplied task using the available controllers and tools.

Task context. You receive the task goal, observable state and action interfaces, available controllers and GUI routines, interaction limits, and the native task-success criterion.

Instructions. Start with a coarse skill that expresses the task objective. For each leaf, specify its goal, controller or tool, executable completion predicate, local training reward, and interaction horizon. Use only observable information. Distinguish the condition that transfers control from the reward that trains the skill. Preserve the native complete-task success criterion.

Response. Return an executable JSON tree specification and a short rationale. Assign stable skill identifiers so that subsequent training and refinement can refer to the same controllers.

Bottleneck Refinement Prompt

You are revising the current control tree after success at the selected bottleneck has plateaued and execution failures persist.

Evidence. You receive the current tree, bottleneck skill, local learning history and patience status, recent skill-entry states, successful and failed trajectories, downstream handoff outcomes, and memory of earlier proposals and evaluations.

Decision. Choose Keep, SeqAnd, or XOR. Keep the current organization when the evidence does not support a useful change. Use SeqAnd when the bottleneck contains successive behaviors with distinct handoffs. Use XOR when observable entry conditions require different strategies.

Candidate specification. Replace only the selected leaf. Define each child's goal, controller initialization, completion predicate, reward, and horizon. Order sequential children explicitly. For XOR, provide an executable selector evaluated at entry; every child must have sufficient observed support, with at least 32 entry contexts per child. Preserve unaffected controllers and their training state. Revise shaping or learning controls through the supported interfaces listed below. Keep the task-level objective unchanged.

Response. Return the action, revised executable JSON specification, and a short explanation linking the change to the observed failure. Identify the controllers that must adapt before matched complete-task evaluation.

Candidate Selection Prompt

Compare the incumbent with the candidate after the affected controllers have adapted. You receive complete-task success estimates from the same 256 fresh initial states, revised-subtree and upstream success statistics, the proposal, and refinement memory. Apply the numerical acceptance gates listed below using the shared task-level objective and interaction limit. Return Promote or Rollback, with a brief rationale grounded in these outcomes. Record the decision and its evidence in refinement memory.

Controls. XOR commits its entry-time choice for each invocation. Full Craftax exposes learning rate, discount, GAE, entropy, dense-reward scale, health penalty, kill-progress scale, completion bonus, native-reward inclusion, frontier-reset probability, and internal skill-entry states.

Acceptance. Comparison uses the same 256 fresh starts per agent (512 episodes total), with fixed trees/controllers. Lower task success is rejected; Minecraft accepts ties. Other executors require strict improvement when either succeeds. If both score exactly zero, local progress Q must improve with no regression >0.05 on unchanged upstream phases having sufficient attempts on both sides. For phase attempts n_i and completions s_i, q_i=s_i/max(1,n_i); Q is q_b for a leaf, the product of q_i for SeqAND, and sum(s_i)/max(1,sum(n_i)) for XOR. These are selection gates, not significance tests.

Jukebox example. The complete structure is SeqAND(D,W,C): D selects among four observable routes (diamond already held: equip axe; diamond block held: unpack/equip; ore route; block route); W=SeqAND(first log, XOR(nearby second log, search for second log)); C converts planks and crafts. W's selector tests for a detected log within 32 blocks; both routes hand at least two logs to C. This gives eight leaves.

The 1 -> 3 -> 6 -> 8 expansion is a structural walkthrough, not a recorded call trace. We will add a machine-validatable JSON schema, a complete annotated refinement trace, and tree/handoff diagrams to the camera-ready appendix.

If our responses have addressed your concerns, we hope that you could consider raising the score. Thank you for your time!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chair and Reviewers,

Thank you for your thoughtful feedback. We have uploaded a revised manuscript with two main experimental updates.

  • Minecraft (Table 1). We evaluated our method on all 30 tasks with targeted controller learning enabled during planner refinement. Average success increased from 52.8% with frozen controllers to 58.6% with controller learning, compared with 53.6% for MineEvolve, the strongest baseline overall.
  • Craftax (Tables 2 and 11). We repeated the experiments across three training seeds and evaluated each selected checkpoint on 1,024 held-out episodes. Our method achieved 40.13% mean success across the four tasks, compared with 22.68% for PPO-RNN, the strongest baseline overall. It also outperformed every evaluated baseline on each of the four tasks for all three seeds.

We revised the Introduction to clarify how the planner and controllers adapt to one another. When progress on a skill stalls and failures persist, the planner revises the skill decomposition, and the affected controllers then learn the revised skills. The appendix now documents the experimental settings and evaluation protocols and includes refinement case studies and planner prompts.

We expect the code and accompanying assets to be ready for release by the end of November. We welcome any further questions or feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.