acceptodds
Under review as a conference paper at ICLR 2027

Coding Agents as Task Compilers: From Specifications to State-Centric Visual Control Specialists

12 views

Abstract

In visual control, a coding agent can build the controller rather than be the controller. Using a general-purpose coding agent as the online controller requires repeated model calls during each episode, incurring latency and inference cost. We instead concentrate coding-agent computation in development and formulate task compilation: compiling a task or task family into a reusable specialist that thereafter executes independently through a fixed runtime interface. We instantiate this idea with State-Centric Task Compilation (SCTC), which organizes the specialist around a task-state and memory module , a visual encoder , and a control policy . The state interface separates the information estimated from images, retained across time, and used for control, providing a compatibility boundary for goal-conditioned reuse and localized recompilation. Across the evaluated manipulation tasks and games, coding agents construct effective frozen specialists. In Craftax, a specialist compiled by Astra achieves similar mean reward to Astra playing the game directly, with about shorter episode runtime. In object pushing, updating only the visual encoder restores success under visual perturbations, while adding a generalization requirement to the specification improves performance on unseen shapes. In Minecraft, frozen specialists use screenshots and keyboard/mouse actions to craft a stone pickaxe within five minutes in unseen worlds. These results show that coding agents can turn task specifications into reusable visual control specialists with low execution overhead.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
task compilation, coding agents, visual control, state-centric interfaces, reusable specialists

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jDYYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses coding agents to construct frozen visual controllers organized around perception, state, and control. Experiments in Craftax, pushing, and Minecraft demonstrate efficient execution, visual repair, and some generalization to unseen tasks

Critical strengths

The deployment results are compelling: the selected Craftax specialist achieves comparable reward to direct Astra at roughly 70× shorter episode runtime.

Adding a generalization requirement improves mean unseen-shape success from 0.08% to 57.33% across three T-only constructions.

Critical weaknesses

The major concern is that the methodology has limited novelty, the performance gain main stems from the stronger foundations model.

The methodological advance over existing agent-generated controllers is unclear. RHO already moves policy construction into development, and agentic Push-T already generates visual, state-based controllers outside the LLM control loop. (Section 5 acknowledges these closely related approaches, but the experiments do not compare against them.)

The results show that the state interface permits local repair, but not that explicitly imposing this organization improves construction or repair. Ordinary modular controllers could exhibit the same behavior. (Table 2a starts all branches from the same ZEP specialist, with no unconstrained construction baseline; Table 9 shows that Full often makes only perception-related edits.)

The Craftax comparison with RL confounds construction method with access to memory. Remembering off-screen resources is itself a major source of improvement. (Appendix B.3 gives both PPO baselines only the current frame, while Section 4.1 attributes specialist gains to persistent maps and resource memory.)

Questions for authors

What does SCTC add over the same coding agent developing a controller without the prescribed ZEP decomposition?

5 comments
AuthorsOpenReview ↗

Response to Reviewer jDYY (1/2)

Thank you for the valuable review, and for recognizing the deployment results.

Main concern. "…the methodology has limited novelty, the performance gain main stems from the stronger foundations model."

We agree the model matters, and Table 1 says so in its title: "The choice of coding agent matters for specialist quality." Our contribution is the formulation of task compilation and its state-centric instantiation. With the same model, the compiled specialist matches direct control in Reward (20.16 vs 19.48, Table 7) while running episodes about 70× faster with no online API calls, and the state interface lets an encoder be repaired with state and control frozen (99.88% vs 99.92%, Table 2a).

W1. "The methodological advance over existing agent-generated controllers is unclear. RHO already moves policy construction into development, and agentic Push-T already generates visual, state-based controllers outside the LLM control loop."

RHO, GaP, and agentic Push-T are concurrent arXiv preprints (June–August 2026), cited in Sec. 5. Moving construction into development is the pattern they share, which Sec. 2 formalizes as task compilation, and they can be understood as concrete task-compilation approaches. Their repository search, skill composition, and procedural control are complementary to the ZEP interface organization. SCTC separates current visual evidence, persistent state, and action selection, which makes perception independently revisable as long as it keeps supplying the information the unchanged state updater and policy require.

SystemAs a task-compilation approachInput
RHOA construction procedure: reflective evolutionary search over multi-file policy repositoriesSupplied perception, planning, and control primitives
GaPA graph-structured specialist: skill nodes assembled from a library and refined in simulationRGB-D through a supplied library of 51 skills
Agentic Push-TA specialist for pushing: a procedural controller built by iterative generation, execution, and revisionGround-truth state, or RGB through color segmentation
SCTCA specialist organized around the state interface ZRaw RGB; the agent builds perception; no LLM or VLM calls at run time

An example with the paper's stone-pickaxe task. A controller split by task step might look like this:

find_tree():     screenshot → detect log texture → turn, walk
chop_tree():     screenshot → check crosshair on log → hold LMB → read inventory
craft_planks():  open inventory → template-match slots → click
mine_stone():    screenshot → detect stone texture → mine

Organized around the state interface:

E(frame)       → {log candidates, crosshair target, HUD, inventory counts}
Z.update(e, a) → {target tree, inventory, task phase, heading, resources seen}
P(z)           → choose the action for the current phase; reads only z

A new resource pack touches every skill with its own detector in the first layout, and only E in the second, provided the retained state and control remain compatible. SCTC makes the second layout a requirement.

AuthorsOpenReview ↗

Response to Reviewer jDYY (2/2)

W2 / Q1. "…not that explicitly imposing this organization improves construction or repair. Ordinary modular controllers could exhibit the same behavior." / "What does SCTC add over the same coding agent developing a controller without the prescribed ZEP decomposition?"

Our original claim emphasizes localized recompilation that preserves an existing control solution, rather than improved initial task performance: the interface provides "a compatibility boundary for goal-conditioned reuse and localized recompilation" (Abstract), and Table 2 tests that (E-only repair 99.88% vs Full 99.92% with state and control frozen; frozen reuse on unseen shapes).

Following your suggestion, we ran matched ZEP/no-ZEP comparisons. For a fair comparison, both conditions use the same model, effort, and a fixed development time; only the prescription differs.

  • Craftax: gpt-6.1-sol at xhigh effort, a 1-hour development budget per compilation, three compilations per condition, 100 test worlds.
  • Minecraft: GPT-6-Astra at high effort, reward objective with diamond as the goal, a 10-hour development budget per compilation, two compilations per condition, the same 20 unseen seeds with 30-minute episodes, F3 allowed in all. A diamond requires the full technology tree (log → planks → stick → crafting table → wooden pickaxe → cobblestone → stone pickaxe → furnace → iron ore → iron ingot → iron pickaxe → diamond), which makes this a very long-horizon task. Reward follows the MineRL ObtainDiamond schedule, a sum of one-time milestone rewards that grow along this tree (log 1, planks 2, stick 4, crafting table 4, wooden pickaxe 8, cobblestone 16, stone pickaxe 32, furnace 32, iron ore 64, iron ingot 128, iron pickaxe 256, diamond 1024; maximum 1571). With 10 hours of development plus the 20-seed test per compilation, we could run two per condition in the time available.
CraftaxRun 1Run 2Run 3Mean Reward
With ZEP20.5020.6218.9620.03
Without ZEP20.5720.0818.6119.75
Minecraftno ZEP #1no ZEP #2ZEP #1ZEP #2no ZEP pooledZEP pooled
Mean reward250.70255.30337.25440.10253.00388.68
Wooden pickaxe18/2015/2019/2011/2033/4030/40
Iron ore5/2010/2012/209/2015/4021/40
Iron pickaxe5/208/2010/208/2013/4018/40
Diamond1/200/200/204/201/404/40

With the same agent, ZEP costs nothing on Craftax and progresses further on the long-horizon Minecraft task: higher mean reward in both compilations (pooled 388.68 vs 253.00), with more iron pickaxes (45.0% vs 32.5%) and diamonds (10.0% vs 2.5%).

On repair, the prescription makes "E-only" a checked operation: it leaves the validated state and control unchanged, whereas Full also changed control settings in the Noise branch (Table 9). New PushT experiments show E-only and P-only repairs reaching comparable success to Full at lower mean cost and time (our reply to Reviewer HeDa, W3). An ordinary modular controller has this property only when a module boundary coincides with the observation mechanism; splitting by task step does not guarantee that (example under W1).

W3. "The Craftax comparison with RL confounds construction method with access to memory."

Agreed. The submission notes the difference ("stateful compiled specialists additionally retain episode history", Appendix B.3), and we now control for it on the RL side with three memory-enabled baselines.

RL baseline (one run each; same 100 test worlds)Reward, mean ± SDCraftax scoreTraining steps
CNN-PPO16.08 ± 2.9755.2149.61M
CNN + GRU16.86 ± 2.1763.6081.81M
Qwen3-VL, single frame10.08 ± 2.0217.306.78M
Qwen3-VL + GRU10.42 ± 2.3017.215.75M
Qwen3-VL, four frames10.86 ± 1.9319.166.78M
Qwen3-VL + MemAgent-style text memory4.42 ± 1.214.251.75M
Astra specialists (Table 1)17.03 / 18.45 / 19.4591.03—

With memory, the best RL baseline reaches 16.86; the Astra-compiled specialists reach 17.03–19.45.

Dear Reviewer jDYY,

The author-reviewer discussion period is approaching the end, and we would appreciate if you could let us know whether our responses are satisfactory, and whether there are any further questions or comments. If our answers resolve your concerns, we kindly ask you to consider raising the rating of our work. Thank you very much for your time and efforts!

Best,

The authors

Reviewer_HeDaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces State-Centric Task Compilation (SCTC), a framework that uses coding agents to build reusable visual control specialists instead of making decisions directly at runtime. The main idea is to move expensive LLM reasoning into the development stage, allowing the compiled controllers to run independently without repeated LLM calls. The authors organize each specialist into three components: a visual encoder (E), a task-state and memory module (Z), and a control policy (P). They evaluate the framework on Craftax, object-pushing tasks, and Minecraft, showing promising results in runtime efficiency, task performance, and component reuse.

Critical strengths
  1. The proposed Z–E–P framework provides a clear separation between perception, task state, and control.

  2. The paper includes experiments across several environments, which helps demonstrate the practical potential of the approach. The runtime improvements are particularly impressive, especially the approximately 70× shorter execution time reported in Craftax.

Critical weaknesses
  1. While the overall framework is practical, I feel that many of its key ideas, such as maintaining task states, iteratively refining control programs, and deploying reusable policies, have already been explored in previous work. I think authors should claim the differences/novelty and adapt them on your task to make comparison. ([1-3])

  2. The experiments show that coding agents can build effective task-specific controllers, but it's unclear how much of the improvement actually comes from the proposed Z–E–P architecture. A strong coding agent might achieve similar results by iteratively developing a controller without following this specific structure.

  3. The E-only recompilation results are promising, as updating the encoder achieves similar performance to full recompilation. However, I'm wondering whether this actually reduces the development cost. How much development time, LLM inference cost, and computational cost does E-only recompilation actually save compared to full recompilation?

  4. While the paper evaluates tasks that require multi-step planning and memory, I'm wondering how well the proposed state-centric framework performs over longer task horizons. Since the specialist continuously updates its internal state, errors in state estimation could accumulate over time and affect later decisions. Could the authors provide more analysis of how task success rate and state consistency change as the task horizon increases? I think this would help better demonstrate the benefits of the proposed state and memory module.

[1]. RHO: Your Coding Agent is Secretly a Roboticist [2]. Revisiting the "Push-T" Robot Manipulation Task with Agentic Robotics [3]. GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks

Questions for authors

See in Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer HeDa (1/3)

Thank you for the valuable review and the pointers to related work, and for noting the clear separation of perception, state, and control.

W1. "…many of its key ideas … have already been explored in previous work. I think authors should claim the differences/novelty and adapt them on your task to make comparison. ([1-3])"

[1–3] are concurrent arXiv preprints (June–August 2026), cited in Sec. 5, and can be understood as concrete task-compilation approaches. Their repository search, skill composition, and procedural control are complementary to the ZEP interface organization. SCTC separates current visual evidence, persistent state, and action selection, which makes perception independently revisable as long as it keeps supplying the information the unchanged state updater and policy require.

SystemAs a task-compilation approachInput
RHO [1]A construction procedure: reflective evolutionary search over multi-file policy repositoriesSupplied perception, planning, and control primitives
GaP [3]A graph-structured specialist: skill nodes assembled from a library and refined in simulationRGB-D through a supplied library of 51 skills
Agentic Push-T [2]A specialist for pushing: a procedural controller built by iterative generation, execution, and revisionGround-truth state, or RGB through color segmentation
SCTCA specialist organized around the state interface ZRaw RGB; the agent builds perception; no LLM or VLM calls at run time

An example with the paper's stone-pickaxe task. A controller split by task step might look like this:

find_tree():     screenshot → detect log texture → turn, walk
chop_tree():     screenshot → check crosshair on log → hold LMB → read inventory
craft_planks():  open inventory → template-match slots → click
mine_stone():    screenshot → detect stone texture → mine

Organized around the state interface:

E(frame)       → {log candidates, crosshair target, HUD, inventory counts}
Z.update(e, a) → {target tree, inventory, task phase, heading, resources seen}
P(z)           → choose the action for the current phase; reads only z

A new resource pack touches every skill with its own detector in the first layout, and only E in the second, provided the retained state and control remain compatible. SCTC makes the second layout a requirement.

AuthorsOpenReview ↗

Response to Reviewer HeDa (2/3)

W2. "…it's unclear how much of the improvement actually comes from the proposed Z–E–P architecture. A strong coding agent might achieve similar results by iteratively developing a controller without following this specific structure."

Our original claim emphasizes localized recompilation that preserves an existing control solution, rather than improved initial task performance: the interface provides "a compatibility boundary for goal-conditioned reuse and localized recompilation" (Abstract), and Table 2 tests that (E-only repair 99.88% vs Full 99.92% with state and control frozen; frozen reuse on unseen shapes).

Following your suggestion, we ran matched ZEP/no-ZEP comparisons. For a fair comparison, both conditions use the same model, effort, and a fixed development time; only the prescription differs.

  • Craftax: gpt-6.1-sol at xhigh effort, a 1-hour development budget per compilation, three compilations per condition, 100 test worlds.
  • Minecraft: GPT-6-Astra at high effort, reward objective with diamond as the goal, a 10-hour development budget per compilation, two compilations per condition, the same 20 unseen seeds with 30-minute episodes, F3 allowed in all. A diamond requires the full technology tree (log → planks → stick → crafting table → wooden pickaxe → cobblestone → stone pickaxe → furnace → iron ore → iron ingot → iron pickaxe → diamond), which makes this a very long-horizon task. Reward follows the MineRL ObtainDiamond schedule, a sum of one-time milestone rewards that grow along this tree (log 1, planks 2, stick 4, crafting table 4, wooden pickaxe 8, cobblestone 16, stone pickaxe 32, furnace 32, iron ore 64, iron ingot 128, iron pickaxe 256, diamond 1024; maximum 1571). With 10 hours of development plus the 20-seed test per compilation, we could run two per condition in the time available.
CraftaxRun 1Run 2Run 3Mean Reward
With ZEP20.5020.6218.9620.03
Without ZEP20.5720.0818.6119.75
Minecraftno ZEP #1no ZEP #2ZEP #1ZEP #2no ZEP pooledZEP pooled
Mean reward250.70255.30337.25440.10253.00388.68
Wooden pickaxe18/2015/2019/2011/2033/4030/40
Iron ore5/2010/2012/209/2015/4021/40
Iron pickaxe5/208/2010/208/2013/4018/40
Diamond1/200/200/204/201/404/40

Similar on Craftax, as you expected; greater progress through Minecraft's long-horizon crafting chain with ZEP: higher mean reward in both compilations (pooled 388.68 vs 253.00), and more frequent iron-pickaxe completion (45.0% vs 32.5%) and diamond acquisition (10.0% vs 2.5%). The interface appears to help most when visual parsing is hard and the task chain is long.

AuthorsOpenReview ↗

Response to Reviewer HeDa (3/3)

W3. "How much development time, LLM inference cost, and computational cost does E-only recompilation actually save compared to full recompilation?"

We measured it in new repair experiments on PushT (gpt-6.1-sol, xhigh). Each scope is repeated three times from the same frozen specialist and tested on 100 held-out seeds.

Local scopeFull scope
Visual noise (σ = 30): E-only vs Full. Test success100%100%
Mean API cost per repair0.24 USD0.38 USD
Mean development time412 s660 s
Object mass ×20: P-only vs Full (unrepaired: 46%). Test success88.7%90.3%
Mean time to the development target253 s369 s
Mean API cost per repair0.20 USD0.23 USD

On PushT, local recompilation reaches comparable success at lower mean cost and time: 38% lower API cost for E-only, and the development target reached 31% sooner for P-only. Full-scope cost also varies more (0.22–0.65 USD, against 0.19–0.27 for E-only).

W4. "…how well the proposed state-centric framework performs over longer task horizons."

We tested this on a much longer task. The submitted Minecraft task ends at a stone pickaxe within 5 minutes; the new one asks for a diamond within 30 minutes. A diamond requires the full technology tree (log → planks → stick → crafting table → wooden pickaxe → cobblestone → stone pickaxe → furnace → iron ore → iron ingot → iron pickaxe → diamond), where each tool tier depends on the one before it.

The ZEP specialists progress further along this chain: iron pickaxes in 18 of 40 episodes and diamonds in 4, against 13 and 1 without ZEP (table under W2).

Reviewer_hjFVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes State-Centric Task Compilation (SCTC), which uses coding agents to build reusable specialists comprising visual perception (E), task state and memory (Z), and control (P). Evaluated in Craftax, PushT/PushObj, and Minecraft, the approach achieves approximately 70× faster execution in Craftax, restores PushObj success to 99.88% through encoder-only recompilation under visual perturbations, improves transfer to unseen object shapes to 68.67% macro-average success, and completes a Minecraft stone-pickaxe task in 60% of 30 held-out worlds. The results highlight the potential of reusable specialists and targeted recompilation for efficient task execution.

Critical strengths

1.The paper addresses the cost of repeatedly using coding agents for tasks that must be executed many times. 2. Separating perception, state, and control provides a clear mechanism for updating one part of a specialist without rebuilding everything. 3. The PushObj experiment provides convincing evidence that encoder-only recompilation can recover performance under the tested visual perturbations. 4. The reported speedups suggest that compiling specialists can be worthwhile for repeated execution. 5. Experiments in three environments provide evidence that the approach is not limited to a single task setting.

Critical weaknesses
  1. The paper does not convincingly distinguish SCTC from existing coding-agent approaches that generate and reuse executable skills. The authors need to identify precisely what the state-centric decomposition enables that prior methods do not.
  2. The results show that the overall system can work, but do not clearly establish that the ZEP decomposition itself produces the improvements. More direct comparisons are needed to support this central claim.
  3. The reported execution speedups do not, by themselves, establish an overall cost advantage. The cost of specialist construction and recompilation must be considered alongside deployment savings.
  4. Although performance is demonstrated on specific tasks and perturbations, evidence regarding reliability across distinct tasks or more significant environmental variations remains limited. The experiments demonstrate performance on selected tasks and perturbations, but provide limited evidence about reliability across different tasks and more substantial environmental changes.
  5. The use of the F3 overlay in Minecraft may provide additional state information. The results therefore do not fully establish performance under ordinary visual-only gameplay.
Questions for authors
  1. What is the main technical distinction between SCTC and prior systems that generate and reuse executable skills?
  2. The RL vs SCTC is not fair since RL receives no history, but SCTC has a memory module. Can ablations be run to compare the performance of SCTC both with and without the memory module?
  3. What experimental evidence demonstrates that the ZEP decomposition is necessary, rather than simply one workable implementation?
  4. Do the efficiency gains remain substantial after including specialist construction and recompilation costs?
  5. Under what conditions does encoder-only recompilation fail, requiring changes to the state representation or control policy?
  6. How much does the F3 overlay contribute to the Minecraft results?
8 comments
AuthorsOpenReview ↗

Response to Reviewer hjFV (1/3)

Thank you for the detailed review, and for recognizing that the PushObj experiment is convincing evidence for encoder-only recompilation.

W1 / Q1. "What is the main technical distinction between SCTC and prior systems that generate and reuse executable skills?"

RHO, GaP, and agentic Push-T are concurrent arXiv preprints (June–August 2026), cited in Sec. 5, and can be understood as concrete task-compilation approaches. Their repository search, skill composition, and procedural control are complementary to the ZEP interface organization. SCTC separates current visual evidence, persistent state, and action selection, which makes perception independently revisable as long as it keeps supplying the information the unchanged state updater and policy require.

SystemAs a task-compilation approachInput
RHOA construction procedure: reflective evolutionary search over multi-file policy repositoriesSupplied perception, planning, and control primitives
GaPA graph-structured specialist: skill nodes assembled from a library and refined in simulationRGB-D through a supplied library of 51 skills
Agentic Push-TA specialist for pushing: a procedural controller built by iterative generation, execution, and revisionGround-truth state, or RGB through color segmentation
SCTCA specialist organized around the state interface ZRaw RGB; the agent builds perception; no LLM or VLM calls at run time

An example with the paper's stone-pickaxe task. A controller split by task step might look like this:

find_tree():     screenshot → detect log texture → turn, walk
chop_tree():     screenshot → check crosshair on log → hold LMB → read inventory
craft_planks():  open inventory → template-match slots → click
mine_stone():    screenshot → detect stone texture → mine

Organized around the state interface:

E(frame)       → {log candidates, crosshair target, HUD, inventory counts}
Z.update(e, a) → {target tree, inventory, task phase, heading, resources seen}
P(z)           → choose the action for the current phase; reads only z

A new resource pack touches every skill with its own detector in the first layout, and only E in the second, provided the retained state and control remain compatible. SCTC makes the second layout a requirement.

AuthorsOpenReview ↗

Response to Reviewer hjFV (2/3)

W2 / Q3. "…do not clearly establish that the ZEP decomposition itself produces the improvements." / "…necessary, rather than simply one workable implementation?"

Our original claim emphasizes localized recompilation that preserves an existing control solution, rather than improved initial task performance: the interface provides "a compatibility boundary for goal-conditioned reuse and localized recompilation" (Abstract), and Table 2 tests that (E-only repair 99.88% vs Full 99.92% with state and control frozen; frozen reuse on unseen shapes).

Following your suggestion, we ran matched ZEP/no-ZEP comparisons. For a fair comparison, both conditions use the same model, effort, and a fixed development time; only the prescription differs.

  • Craftax: gpt-6.1-sol at xhigh effort, a 1-hour development budget per compilation, three compilations per condition, 100 test worlds.
  • Minecraft: GPT-6-Astra at high effort, reward objective with diamond as the goal, a 10-hour development budget per compilation, two compilations per condition, the same 20 unseen seeds with 30-minute episodes, F3 allowed in all. A diamond requires the full technology tree (log → planks → stick → crafting table → wooden pickaxe → cobblestone → stone pickaxe → furnace → iron ore → iron ingot → iron pickaxe → diamond), which makes this a very long-horizon task. Reward follows the MineRL ObtainDiamond schedule, a sum of one-time milestone rewards that grow along this tree (log 1, planks 2, stick 4, crafting table 4, wooden pickaxe 8, cobblestone 16, stone pickaxe 32, furnace 32, iron ore 64, iron ingot 128, iron pickaxe 256, diamond 1024; maximum 1571). With 10 hours of development plus the 20-seed test per compilation, we could run two per condition in the time available.
CraftaxRun 1Run 2Run 3Mean Reward
With ZEP20.5020.6218.9620.03
Without ZEP20.5720.0818.6119.75
Minecraftno ZEP #1no ZEP #2ZEP #1ZEP #2no ZEP pooledZEP pooled
Mean reward250.70255.30337.25440.10253.00388.68
Wooden pickaxe18/2015/2019/2011/2033/4030/40
Iron ore5/2010/2012/209/2015/4021/40
Iron pickaxe5/208/2010/208/2013/4018/40
Diamond1/200/200/204/201/404/40

Similar on Craftax; greater progress through Minecraft's long-horizon crafting chain with ZEP: higher mean reward in both compilations (pooled 388.68 vs 253.00), and more frequent iron-pickaxe completion (45.0% vs 32.5%) and diamond acquisition (10.0% vs 2.5%). The interface appears to help most when visual parsing is hard and the task chain is long.

AuthorsOpenReview ↗

Response to Reviewer hjFV (3/3)

Q2. "The RL vs SCTC is not fair since RL receives no history, but SCTC has a memory module."

We added three RL baselines with memory.

RL baseline (one run each; same 100 test worlds)Reward, mean ± SDCraftax scoreTraining steps
CNN-PPO16.08 ± 2.9755.2149.61M
CNN + GRU16.86 ± 2.1763.6081.81M
Qwen3-VL, single frame10.08 ± 2.0217.306.78M
Qwen3-VL + GRU10.42 ± 2.3017.215.75M
Qwen3-VL, four frames10.86 ± 1.9319.166.78M
Qwen3-VL + MemAgent-style text memory4.42 ± 1.214.251.75M
Astra specialists (Table 1)17.03 / 18.45 / 19.4591.03—

With memory, the best RL baseline reaches 16.86; the Astra-compiled specialists reach 17.03–19.45.

W3 / Q4. "Do the efficiency gains remain substantial after including specialist construction and recompilation costs?"

Yes, on recorded API cost and wall-clock time.

CraftaxAPI costWall-clock
Direct Astra, per episode (Table 7)16.16 USD2,732 s
Compiled specialist, per episode038 s
Construction, all three Astra builds (Table 1)73.86 USD3.43 h
Construction, selected Astra-231.06 USD1.58 h
Local recompilation (new; 30 repairs on PushT and Craftax, gpt-6.1-sol)0.10–0.85 USD≤ 30 min

Construction is recovered after about 5 episodes of direct play counting all three builds, or 2 for the selected one. A local recompilation costs under 1 USD.

W4. "…evidence regarding reliability across distinct tasks or more significant environmental variations remains limited."

We added two settings.

  • A much longer-horizon task. The submitted Minecraft task ends at a stone pickaxe within 5 minutes. The new one asks for a diamond within 30 minutes, which requires the full technology tree: log → planks → stick → crafting table → wooden pickaxe → cobblestone → stone pickaxe → furnace → iron ore → iron ingot → iron pickaxe → diamond. Compiled specialists reach iron pickaxes in up to 10 of 20 episodes and diamonds in up to 4 (table under W2).
  • New shift types. A dynamics change in PushT (object mass ×20), repaired with P-only recompilation, and appearance changes in Craftax (palette and render scale), where E-only repairs reached the recovery target in 6 of 6 runs. See details in rebuttal to HeDa W3.

The paper now covers reuse across worlds, across goals and shapes, and under repaired perception or control.

Q5. "Under what conditions does encoder-only recompilation fail…?"

E-only is enough when the change is in appearance and the needed evidence can still be recovered (Table 2a). Beyond that, Sec. 3.3 says: "a new objective, a goal outside the compiled task family, or a newly required decision-relevant distinction may require changing Z and the control components that depend on it as well." We now test one such case: under a dynamics change in PushT (object mass ×20, rendering unchanged), the encoder has nothing to repair, and a P-only repair with E and Z frozen raises Test success from 46% to 88.7%.

W5 / Q6. "How much does the F3 overlay contribute to the Minecraft results?"

Table 3 already reports the specialist without F3.

Policy (30 shared worlds)Stone pickaxes / gesture
SCTC with F360.0%0.374
SCTC without F330.0%0.382
Direct Astra, medium (no F3)30.0%3.655
Direct Astra, xhigh (no F3)13.3%4.368

Without F3, the specialist matches direct Astra at medium effort and beats xhigh, about ten times faster per gesture.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Responses

We thank all the reviewers for their valuable feedback. We would like summarize our answer to two common questions below.

1. Does prescribing Z–E–P improve performance over the same agent without it? (hjFV W2/Q3, HeDa W2, jDYY W2/Q1)

We need to clarify that our original claim mainly emphasizes its advantage on localized recompilation that preserves an existing perception or control module (Abstract; Table 2). Following the reviewers' suggestion, we also ran matched ZEP/no-ZEP comparisons, giving both conditions the same fixed development time:

Same agent, only the prescription differsWithout ZEPWith ZEP
Craftax mean Reward (3 compilations each)19.7520.03
Minecraft mean reward (2 compilations each)250.70 / 255.30337.25 / 440.10
Minecraft iron pickaxe, pooled13/4018/40

Craftax performance is similar. Minecraft shows greater progress with ZEP through its long-horizon crafting chain (wooden, stone, and iron, then diamond): higher mean reward (pooled 388.68 vs 253.00), and more frequent iron-pickaxe completion (45.0% vs 32.5%) and diamond acquisition (10.0% vs 2.5%). Each Minecraft compilation takes a 10-hour budget plus a test on 20 unseen seeds with 30-minute episodes; due to the time limit, we compiled twice per condition.

2. How does SCTC differ from RHO, GaP, and agentic Push-T? (hjFV W1/Q1, HeDa W1, jDYY W1)

All three are concurrent arXiv preprints (June–August 2026) and are concrete task-compilation approaches, built on repository search (RHO), skill composition (GaP), and procedural control (agentic Push-T). SCTC's contribution is orthogonal: an explicit state interface that acts as a compatibility boundary, so perception can be revised on its own while the existing state updater and policy stay unchanged (E-only repair: 99.88% vs 99.92% for Full, Table 2a). Our specialists also build perception from raw RGB, where RHO and GaP use supplied primitives or skill libraries.

Also added

  • RL baselines with memory (hjFV Q2, jDYY W3): we added recurrent and text-memory RL baselines. Giving the RL policies memory does not close the gap to the compiled specialists.
  • Cost including construction (hjFV W3/Q4, HeDa W3): the efficiency gain holds after counting construction. Its API cost is recovered within a few episodes of direct play, and a local recompilation costs under 1 USD.
AuthorsOpenReview ↗

Summary of Revision

Dear Reviewers,

We have uploaded a revised PDF, with changes marked in blue.

  • New Sec. 4.4 and Table 4 compare the same coding agent with and without the ZEP prescription; Appendix E.1 gives the protocols and all runs. Sec. 3.2 now states that the interface is prescribed for reuse and local recompilation. (hjFV, HeDa, jDYY)
  • Related work (Sec. 5) now discusses RHO, GaP, and agentic Push-T as concurrent task-compilation approaches, with a comparison in Table 5. (hjFV, HeDa, jDYY)
  • Table 1(b) adds RL baselines with memory; Appendix E.3 gives their settings. (hjFV, jDYY)
  • Sec. 4.1 adds the construction break-even and the cost of local recompilation. (hjFV, HeDa)
  • Sec. 4.2 adds repairs under a dynamics shift (P-only) and Craftax appearance shifts; Appendix E.2 reports their cost and time. (hjFV, HeDa)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.