acceptodds
Withdrawn from ICLR 2027

Attacca: Goal-Directed Control under State Continuity for Long-Horizon Embodied Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A central capability of embodied agents is to accomplish complex objectives through sequences of interdependent tasks. Yet existing visual goal-conditioned policies underlying these agents are typically evaluated on isolated interactions where the target is already visible, and thus do not capture the conditions that arise during continuous long-horizon task execution. In such settings, each task begins from the state left by the previous one: the agent may end at a different position and orientation, the world may have been modified, and the next interaction target may lie outside the current field of view. As a result, agents relying on such policies may struggle to proceed to the next task when they cannot ground their target in the current observation. To address this challenge, we propose Attacca, a new approach that trains visual goal-conditioned policies on complete exploration-to-interaction trajectories using goal images decoupled from the execution environment. Attacca uses context-decoupled goal sampling to pair each demonstration with a class-compatible masked goal image from another world, removing direct scene and pose correspondence. It learns dense current-view grounding through a target-mask prediction head, providing auxiliary supervision beyond action imitation. We further introduce behavioral-phase conditioning that teaches the policy to distinguish Search, Approach, and Interact stages and adapt its control as execution progresses. We evaluate Attacca on multiple short- and long-horizon embodied tasks in Minecraft. Our method achieves 39.0–47.5% clean success, improving over the strongest baseline by 1.7–2.4×. On long-horizon tasks, it attains 54%, 30%, and 28% completion, yielding up to a 7× improvement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Embodied AI, Visual Goal-Conditioned Policies, Exemplar-Based Goal Specification, Visual Search and Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oayjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Attacca trains a goal-conditioned visuomotor policy under an imitation-learning setting. Traditional approaches typically obtain the goal image from the same demonstration trajectory or the same world. In contrast, for each demonstration trajectory, Attacca requires the goal image to come from another trajectory with the same task-target category, thereby decoupling the goal-associated object from the execution context.

Attacca is a direct improvement over ROCKET-2 on the Minecraft tasks. Compared with ROCKET-1, ROCKET-2 replaces per-frame dynamic goal mask conditioning, while using per-frame target masks only as privileged training supervision for centroid and visibility prediction. Attacca does not remove this train-time annotation requirement; instead, it introduces a new goal-image construction scheme and strengthens ROCKET-2's centroid/visibility supervision into dense segmentation supervision, together with an additional behavioral-phase objective.

Overall, Attacca achieves substantial improvements over ROCKET-2 on long-horizon tasks, but the ablation studies are not sufficiently comprehensive, and it remains unclear how transferable the proposed design choices are to tasks beyond Minecraft.

Critical strengths

Attacca addresses two failure modes relevant to long-horizon goal-conditioned control: the inability of current-view goal-construction pipelines to instantiate a valid goal when the target is initially out of view, and the context sensitivity of ROCKET-2-style reference-goal policies when the goal view is decoupled from the execution scene.

Critical weaknesses
  1. The different-world goal-image design appears to primarily address a limitation of ROCKET-2 under the specific Minecraft setup. In more general goal-conditioned control problems, the goal may need to specify not only a target object category, but also the states and relations of multiple objects within a particular environment. In such cases, deliberately decoupling the goal image from the execution context may no longer be appropriate. The generality of this design choice is therefore unclear.
  2. It is already well established that richer and more fine-grained supervision can improve representation learning and downstream control. From this perspective, the observation that replacing ROCKET-2-style centroid/visibility supervision with denser segmentation supervision, together with an additional behavioral-stage classification objective, improves performance is not by itself a sufficiently informative finding. A stronger analysis is needed to establish what new capability or principle is enabled by these supervision signals beyond simply providing more privileged annotations.

Overall, although the paper motivates its method by the challenges of goal construction and subtask transitions in long-horizon tasks, it remains unclear what new insights or broadly applicable design principles the work provides for general long-horizon embodied control beyond the particular Minecraft setting.

Questions for authors
  1. Could the authors provide a full incremental ablation that isolates the contribution of each newly introduced component, for example: ROCKET-2 with same-world goal sampling → + different-world goal sampling → + dense-segmentation auxiliary loss → + phase auxiliary loss → + Attacca feedback/conditioning?
  2. Is there direct evidence that fully disentangle the contribution of replacing ROCKET-2-style centroid/visibility supervision with dense segmentation? How much additional parameter and latency overhead is introduced by the complete grounding pathway? Can the authors compare dense-mask supervision against a ROCKET-2-style centroid+visibility objective under a matched backbone and comparable auxiliary-model capacity?
  3. ROCKET-2 already demonstrates cross-view goal specification in which the target need not be visible in the agent's current observation. Could the authors clarify how Attacca's motivation differs from these existing capabilities? Is the primary motivation for Attacca's different-world goal images to reduce context sensitivity during training, or to avoid failures of test-time goal construction when the target is outside the current field of view?
  4. Is the different-world goal a particular form of task embedding? If so, the same motivation seems to admit more general task-embedding designs, such as scale- and background-invariant embeddings that jointly represent the target object and the interaction type.
  5. Is the fixed phase classification essentially a Minecraft-specific structural prior, with the ground-truth phase labels generated according to predefined rules? If so, does this narrow the task formulation to tasks that naturally conform to this predefined stage structure?
  6. Could the authors discuss what more general conclusions or design principles for goal-conditioned visuomotor policies can be drawn from the proposed method beyond the Minecraft tasks?
6 comments
AuthorsOpenReview ↗

Author Response to Reviewer oayj (1/4)

We thank the reviewer for the thoughtful review and the precise questions. All line, table, figure, and section numbers below refer to our revision.

W1. Generality beyond Minecraft.

The different-world goal-image design appears to primarily address a limitation of ROCKET-2 under the specific Minecraft setup. [...] the goal may need to specify not only a target object category, but also the states and relations of multiple objects within a particular environment.

We believe this concern may stem from a misunderstanding of our method. Attacca is not an incremental fix for a limitation of ROCKET-2 in Minecraft. It addresses a clearly defined problem, goal-directed control under state continuity, in which each task starts from the state left by the previous one and the next target is often out of view (Lines 15-18, 47-50, and 187-189). Any embodied agent that executes a long-horizon objective as a sequence of tasks faces this problem, and our training method is not tied to Minecraft.

Goals that specify the states and relations of multiple objects, such as "arrange the objects on the desk" or a goal image of a fully set table, define a different problem. Our setting specifies the object to interact with in each task, not the goal state of a scene. Embodied control is studied as such separate problems, as navigation, locomotion, and manipulation are in robotics. A single policy that handles every type of goal does not yet exist and remains a fundamental research agenda of embodied AI.

State continuity, however, arises inside such goals as well. Consider "place the blue cup to the right of the red plate." The agent must (1) find and pick up the blue cup, (2) find the red plate, and (3) place the cup to its right. Picking up the cup leaves the agent with a different camera pose, from which it must search for the red plate. A policy must do this reliably from whatever state the previous task leaves. This robustness under state continuity is the goal of our work, and our long-horizon chains test it directly (Table 2).

The problem is also not specific to Minecraft. In UnrealZoo, a photorealistic environment with a target class that never appears in Minecraft, Attacca reaches the red car in 9 of 20 episodes against 3 for ROCKET-2 without further training (Fig. 6c in our revision). Its mask head, trained only on Minecraft, activates on the car once it enters the view, and its predicted phase switches from Search to Approach at that moment (Fig. 6a and 6b, Lines 522-527).

W2. What the method enables beyond more supervision.

[...] the observation that replacing ROCKET-2-style centroid/visibility supervision with denser segmentation supervision, together with an additional behavioral-stage classification objective, improves performance is not by itself a sufficiently informative finding.

This is a misunderstanding. Our gain does not come from richer supervision. In our ablation (Table 3 in our revision), the parametric and dense rows share the backbone, data, goal sampling, and training recipe, and both take their labels from the same renderer masks. The parametric row trains the visibility, centroid, and bounding-box head of ROCKET-2, and the dense row replaces all three losses with our mask loss. This switch changes success by at most 0.035. The gain comes from feeding the policy's own predictions back into control. Residual feedback lets the policy act on its estimate of where the target is visible, and phase conditioning lets it act on its estimate of whether it should search, approach, or interact. On the same dense supervision, residual feedback and phase conditioning account for 59%, 79%, and 55% of the gain over behavior cloning (please see Q1 and Table 3 in our revision).

AuthorsOpenReview ↗

Author Response to Reviewer oayj (2/4)

All line, table, figure, and section numbers below refer to our revision.

Q1. Full incremental ablation.

Reflecting the reviewer's constructive comments, we have further performed additional analyses during the rebuttal period. We expanded the ablation to four worlds per chain (Fig. 18 in our revision) with 50 seeds each. The first two rows fine-tune ROCKET-2 with its three auxiliary losses, with and without our goal sampling. They use classifier-free guidance with coefficient k=1, as in the main results. The remaining rows add the components of Attacca one at a time, starting from behavior cloning, without guidance as in Attacca (chain success ± standard error, n=200). The phase head is added together with its conditioning.

PolicyDiamond PickaxeWolf FeedingNether Portal
ROCKET-20.160 ± 0.0260.170 ± 0.0270.135 ± 0.024
+ context-decoupled goal sampling0.310 ± 0.0330.140 ± 0.0250.115 ± 0.023
Behavior cloning0.275 ± 0.0320.105 ± 0.0220.010 ± 0.007
+ context-decoupled goal sampling0.375 ± 0.0340.140 ± 0.0250.095 ± 0.021
++ parametric target supervision0.410 ± 0.0350.120 ± 0.0230.080 ± 0.019
++ dense target supervision (replaces parametric)0.410 ± 0.0350.155 ± 0.0260.110 ± 0.022
+++ residual feedback pathway0.500 ± 0.0350.245 ± 0.0300.135 ± 0.024
++++ phase conditioning (Ours)0.605 ± 0.0350.340 ± 0.0330.230 ± 0.030

With our goal sampling, ROCKET-2 nearly doubles on Diamond Pickaxe (0.160 to 0.310) but not on Wolf Feeding and Nether Portal. ROCKET-2 runs with guidance, whose effect differs from task to task (Table 9 in our revision), so its success reflects guidance as well as training. Without guidance, context-decoupled goal sampling raises success on all three chains (0.275 to 0.375, 0.105 to 0.140, and 0.010 to 0.095). Attacca still roughly doubles ROCKET-2 trained with our goal sampling on every chain, so goal sampling alone does not explain the gain.

Q2. Dense masks against centroid and visibility supervision, and overhead.

Yes. The parametric and dense rows above share the backbone and all other settings and differ only in the target supervision. Target supervision alone, parametric or dense, changes success by at most 0.035 (Table 3 in our revision). Our mask head reads the patch features before the causal Transformer, whereas the parametric head of ROCKET-2 is a linear head on the Transformer output. Therefore, only the dense prediction can be fed back into the Transformer input. This feedback through the residual pathway raises success by 0.090, 0.090, and 0.025, and phase conditioning adds another 0.105, 0.095, and 0.095.

Both heads are small next to the shared backbone of about 187M parameters (7,175 for the parametric head and 262,657 for the dense mask head), so head capacity does not drive this comparison. The complete grounding pathway adds 2.11M parameters (+1.13%), 0.21 GFLOPs per step (+0.3%), and 3.8% latency. ROCKET-2 with classifier-free guidance runs two forward passes at every step, so Attacca runs at about twice the FPS of ROCKET-2 (65 against 30 FPS, Table 1 in our revision).

AuthorsOpenReview ↗

Author Response to Reviewer oayj (3/4)

All line, table, figure, and section numbers below refer to our revision.

Q3. Different-world goals and relation to ROCKET-2.

Q3. [...] Is the primary motivation for Attacca's different-world goal images to reduce context sensitivity during training, or to avoid failures of test-time goal construction when the target is outside the current field of view?

Difference from ROCKET-2. ROCKET-2's cross-view goal alignment assumes an accompanying human who captures the goal view in advance, in the very world the agent will act in. An agent that explores an unknown world on its own cannot have such a goal image (Lines 155-157). In our setting, the goal image is taken from a different world to show the target's appearance. The world in the goal image does not match the agent's current world, and the agent performs object-centric control from the target's appearance alone. In addition, ROCKET-2 does not evaluate the case in which the target is not visible in the agent's current observation. Its quantitative benchmark starts every task with the target in view. In all our short-horizon tasks, the target is not visible from the start, and in our long-horizon task chains each new target must be found from wherever the previous task ended (Tables 1 and 2).

Primary motivation. Both motivations follow from this setting. At test time, a goal cannot be built from the current view while the target is out of view (Lines 353-355). A policy trained with each goal image taken from the imitated demonstration itself learns to depend on the shared scene, and at test time it degrades with goals from other worlds (Fig. 3b). Context-decoupled goal sampling removes this dependence during training.

Attacca also works when a goal image taken in the agent's own world is available, as in InstanceImageNav (Krantz et al., 2023). It reaches 0.89 with goal images taken in the execution world and 0.81 with goal images from another world (Table 5).

Q4. Goal images as visual task embeddings.

We agree that the masked goal image can be seen as a visual task embedding, one that specifies the target by its appearance. The reviewer's suggestion then raises two questions, whether this embedding should be made invariant to background and scale, and whether it should also carry the interaction type. We tested both on Mine with the 200 episodes of Table 1. We compared our goal with two formats that are background-invariant or scale- and background-invariant by design, and for each format we also merged the interaction type into the goal embedding. In the merged variants, the interaction-type embedding is concatenated to each goal token and projected back to the token dimension by a linear layer initialized to pass the goal token through, and the separate interaction-type input of the Transformer is set to a fixed null value. Each variant retrains Attacca with the same data and training setup and changes only the goal input, in training and at test (see Fig. 9 in our revision).

Goal formatID (n=140)OOD (n=60)ID, mergedOOD, merged
Ours (masked goal image)0.4500.2500.4070.183
Background removed (white background, native scale)0.3360.1170.3290.117
Object crop (rescaled, white padding)0.3210.1000.3140.033

Both hand-designed invariant formats stay below Ours on trained and held-out classes, so learning the invariance from varied scenes works better than removing the scene by hand. Merging the interaction type into the goal embedding does not improve any format, and it lowers held-out success for Ours (0.250 to 0.183) and for the object crop (0.100 to 0.033). Our policy already uses both the target appearance and the interaction type, as the goal image and a separate interaction-type input to the causal Transformer (Lines 192-201), and keeping them separate works better than merging them into a single embedding.

Reviewer_28oxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Attacca, a visual goal-conditioned policy designed to handle search-to-interact behaviors under state continuity in Minecraft. Building upon the ROCKET-2 architecture and pretrained checkpoint, the method introduces three components: (1) context-decoupled goal sampling, which pairs demonstrations with target images from different worlds; (2) dense target-mask prediction with residual feedback to the policy; and (3) behavioral-phase conditioning that distinguishes Search, Approach, and Interact.

The paper evaluates Attacca on three short-horizon tasks and three long-horizon task chains. Compared with a fine-tuned ROCKET-2 baseline, Attacca improves short-horizon success rates from 23.5/18.5/22.5% to 39.0/47.5/47.0%, and long-horizon chain success from 8/6/4% to 54/30/28%.

Critical strengths
  • The paper addresses a relevant problem in goal-conditioned embodied control. And the context-decoupled goal sampling method is intuitive.
  • The reported improvements are substantial, and the failure analysis provides useful observations.
Critical weaknesses
  1. Limited methodological novelty. Attacca largely retains the ROCKET-2 architecture, including the visual encoders, cross-view fusion, temporal Transformer, and action decoder. The main architectural changes are a dense mask-prediction head with residual feedback and a lightweight phase-conditioning module. Furthermore, ROCKET-2 has already investigated cross-view goal conditioning, target occlusion, and cross-episode goal alignment. The proposed approach appears to be primarily a combination of modified goal sampling, additional supervision, and relatively minor architectural extensions. I find the methodological contribution rather incremental.

  2. The ablations do not convincingly support the proposed architectural contributions. In Table 3, removing both residual feedback and phase conditioning still achieves 50% success on Diamond Pickaxe, compared with 54% for the full method and only 8% for fine-tuned ROCKET-2. Thus, almost the entire improvement on this task is already present without the two proposed action-conditioning modules. This raises the question of whether the gains mainly come from goal relabeling and additional supervision, rather than the proposed architectural innovations. The current ablations do not adequately disentangle these factors.

  3. Insufficiently controlled ROCKET-2 comparison. Since Attacca is initialized from ROCKET-2, a carefully controlled fine-tuning baseline is essential. However, important implementation details are unclear. The original ROCKET-2 uses auxiliary visibility, point, and bounding-box prediction losses. The Attacca implementation disables these losses and introduces different supervision. It is unclear whether the fine-tuned ROCKET-2 baseline retains its original auxiliary losses or uses the same goal-sampling distribution. Additionally, ROCKET-2 supports classifier-free guidance, whereas the released Attacca evaluation uses a guidance scale of zero. The paper does not clearly report the CFG settings used for the ROCKET-2 baselines.

Questions for authors
  • Q1. ROCKET-2 fine-tuning protocol. Could the authors provide the exact training configuration for the fine-tuned ROCKET-2 baseline, including loss functions, auxiliary-loss weights, label construction, learning rate, optimization steps, and checkpoint selection? In particular, were the original ROCKET-2 visibility, point, and bounding-box auxiliary losses retained?
  • Q2. Controlled attribution of performance gains. Could the authors evaluate the original ROCKET-2 architecture trained with the same context-decoupled goal sampling as Attacca, while retaining its native auxiliary objectives? Ideally, a separate comparison would isolate the effect of replacing these objectives with dense target-mask supervision. Given the results in Table 3, this seems essential for understanding the source of the large improvement.
  • Q3. Classifier-free guidance. What CFG coefficients were used for the released and fine-tuned ROCKET-2 baselines? Was the unconditional branch retained during fine-tuning? Could the authors report performance under several guidance strengths, particularly including zero and the nonzero configurations supported by the original ROCKET-2 implementation?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer 28ox (1/3)

We thank the reviewer for the detailed and constructive review. We are especially grateful for the point on classifier-free guidance. The submission evaluated ROCKET-2 without guidance, and this comment led us to re-evaluate it with the default guidance of the official implementation, which gives a stronger baseline and a fairer comparison. All line, table, figure, and section numbers below refer to our revision.

W1. Novelty and relation to ROCKET-2.

ROCKET-2 has already investigated cross-view goal conditioning, target occlusion, and cross-episode goal alignment.

We respectfully disagree with the reviewer's opinion that our novelty is limited. Our work makes three contributions.

  1. Problem. We study goal-directed control under state continuity. Each task starts from the state left by the previous task, so the next target is often out of view (Lines 15-18 and 47-50). Every embodied agent that executes a long-horizon objective as a sequence of tasks faces this condition, yet existing goal-conditioned policies are largely developed and evaluated on short tasks with the target already visible from the start (Lines 74-77 and 116-119).
  2. Method. Attacca trains the policy to act on its own estimates of where the target is and which behavioral phase it is in. Its components are context-decoupled goal sampling, goal-conditioned target grounding with residual feedback, and behavioral-phase conditioning (Secs. 3.2 to 3.4). Each component raises chain success on all three chains (Table 3 in our revision).
  3. Analysis. We show by experiment why existing policies fail under state continuity. The fine-tuned ROCKET-2 fails at target-directed control, losing the target from view even after it enters the view (Fig. 4, Lines 457-467). Trained without context-decoupled goals, it acts by reproducing the goal view rather than by pursuing the masked target (Fig. 5 and Table 5, Lines 470-478). Attacca is trained with context-decoupled goals and feeds its own grounding into action computation, so it keeps the target in view through approach and interaction. This control carries over without further training to an unseen photorealistic world, where Attacca reaches the target in 9 of 20 episodes against 3 for ROCKET-2 (Sec. 4.6 in our revision).

ROCKET-2 does investigate the three topics below, but it does not resolve them, and our setting evaluates them directly.

  • Target occlusion. The quantitative evaluation of ROCKET-2, the Minecraft Interaction Benchmark, starts every task with the target in view. In all our short-horizon tasks the target starts out of view, and in our long-horizon tasks each new target must be found from wherever the previous stage ended.
  • Cross-episode goals. ROCKET-2 provides two qualitative examples of cross-episode generalization without reporting quantitative results. Our benchmark measures this setting quantitatively, since every goal image comes from a different episode in a different world (Tables 1 and 2).
  • Cross-view goal alignment. ROCKET-2 assumes that the goal view is taken in the agent's own world, for example by an accompanying human, and the agent relies on visual landmarks shared between that view and its own. An agent that explores an unknown world on its own has no such view. To generalize to unseen worlds, it must pursue the masked target of a goal image taken in any world regardless of its background.

In this setting, the fine-tuned ROCKET-2 drops sharply when the goal image comes from another world (Fig. 3b). With the same data and initialization, Attacca reaches chain success of 0.605, 0.340, and 0.230, against 0.160, 0.170, and 0.135 for the fine-tuned ROCKET-2 with classifier-free guidance.

AuthorsOpenReview ↗

Author Response to Reviewer 28ox (2/3)

All line, table, figure, and section numbers below refer to our revision.

W2 and Q2. Attribution of the gains.

In Table 3, removing both residual feedback and phase conditioning still achieves 50% success on Diamond Pickaxe, compared with 54% for the full method and only 8% for fine-tuned ROCKET-2. [...]

Q2. Could the authors evaluate the original ROCKET-2 architecture trained with the same context-decoupled goal sampling as Attacca, while retaining its native auxiliary objectives? Ideally, a separate comparison would isolate the effect of replacing these objectives with dense target-mask supervision.

Reflecting the reviewer's constructive comments, we have further performed additional analyses during the rebuttal period. The submitted Table 3 only toggled residual feedback and phase conditioning, in one world per chain with 50 seeds (standard error up to 0.07). We expanded the ablation to four worlds per chain (Fig. 18 in our revision) with 50 seeds each (standard error up to 0.035), and we ran the experiment of Q2. The first two rows fine-tune ROCKET-2 with its three auxiliary losses, with and without our goal sampling, and use guidance (k=1) as in the main results. The remaining rows add the components of Attacca one at a time, starting from behavior cloning, without guidance as in Attacca (chain success ± standard error, n=200).

PolicyDiamond PickaxeWolf FeedingNether Portal
ROCKET-20.160 ± 0.0260.170 ± 0.0270.135 ± 0.024
+ context-decoupled goal sampling0.310 ± 0.0330.140 ± 0.0250.115 ± 0.023
Behavior cloning0.275 ± 0.0320.105 ± 0.0220.010 ± 0.007
+ context-decoupled goal sampling0.375 ± 0.0340.140 ± 0.0250.095 ± 0.021
++ parametric target supervision0.410 ± 0.0350.120 ± 0.0230.080 ± 0.019
++ dense target supervision (replaces parametric)0.410 ± 0.0350.155 ± 0.0260.110 ± 0.022
+++ residual feedback pathway0.500 ± 0.0350.245 ± 0.0300.135 ± 0.024
++++ phase conditioning (Ours)0.605 ± 0.0350.340 ± 0.0330.230 ± 0.030

For the first part of Q2, our goal sampling nearly doubles ROCKET-2 on Diamond Pickaxe (0.160 to 0.310) but not on Wolf Feeding and Nether Portal. ROCKET-2 runs with guidance, whose effect differs from task to task (Table 9 in our revision), so its success reflects guidance as well as training. Without guidance, context-decoupled goal sampling raises success on all three chains (0.275 to 0.375, 0.105 to 0.140, and 0.010 to 0.095). Attacca still roughly doubles ROCKET-2 trained with our goal sampling on every chain, so goal sampling alone does not explain the gain.

For the second part of Q2, residual feedback and phase conditioning, not target supervision, are the primary drivers of Attacca's gains (Lines 404-413). Target supervision alone changes success by at most 0.035, whether it uses the parametric losses of ROCKET-2 or our dense mask loss. Our mask head reads the patch features before the causal Transformer, whereas the parametric head of ROCKET-2 sits after it. Therefore, only the dense mask prediction can be fed back into the Transformer input (Lines 407-409). This feedback through the residual pathway raises success by 0.090, 0.090, and 0.025, and phase conditioning adds another 0.105, 0.095, and 0.095. Residual feedback and phase conditioning account for 59%, 79%, and 55% of the gain over behavior cloning.

AuthorsOpenReview ↗

Author Response to Reviewer 28ox (3/3)

All line, table, figure, and section numbers below refer to our revision.

W3, Q1, and Q3. Fine-tuning protocol and classifier-free guidance.

Q1. [...] In particular, were the original ROCKET-2 visibility, point, and bounding-box auxiliary losses retained?

Yes. The fine-tuned ROCKET-2 keeps all three auxiliary losses and the same-episode goal sampling of ROCKET-2. Appendix D.1 (Table 8) in our revision lists its configuration. It starts from the released ROCKET-2 checkpoint and uses the loss weights of the official implementation. Its auxiliary labels come from the renderer masks in our training dataset (Appendix B), the same masks that supervise Attacca. It is trained with the same data and optimization settings as Attacca (learning rate 2e-5, 200 warmup steps, batch 12, and 1,350 steps). Both models are evaluated at the final training step, with no checkpoint selection.

Q3. What CFG coefficients were used for the released and fine-tuned ROCKET-2 baselines? Was the unconditional branch retained during fine-tuning? Could the authors report performance under several guidance strengths [...]?

The guided logits are (1+k) times the conditional logits minus k times the unconditional logits, so k=0 means no guidance. We swept k on the short-horizon tasks (Table 9 in our revision).

PolicykMineHuntPlace
Fine-tuned ROCKET-200.2350.1850.225
0.50.2550.3200.350
1.00.2500.3400.325
1.50.2800.3100.285
Released ROCKET-200.0050.1000.000
0.50.0050.1450.000
1.00.0450.0850.005
1.50.0300.0850.010

Guidance helps the fine-tuned ROCKET-2, raising Hunt from 0.185 to 0.340 and Place from 0.225 to 0.325 at k=1, but the best k differs by task. We therefore use k=1, the default of the official implementation, for both the released and fine-tuned models, and we re-evaluated every quantitative ROCKET-2 result in the main paper with it (Tables 1, 2, and 5, Figs. 3b and 4, and Sec. 4.6 in our revision). Even with the best k for each task (0.280, 0.340, and 0.350), the fine-tuned ROCKET-2 stays below Attacca (0.390, 0.475, and 0.470). Guidance also needs two forward passes at every step, so ROCKET-2 runs at 30 to 31 FPS, against 65 FPS for Attacca, which runs without guidance (Table 1).

The unconditional branch is trained during fine-tuning with the same protocol as the official implementation. It uses no explicit goal dropout and instead gives a null goal, an empty image and mask, to training samples in which the target is never visible.

We thank the reviewer again for these questions, which made the comparison with ROCKET-2 more controlled, and we would be grateful if the reviewer could reconsider the assessment.

Reviewer_mTViOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes that instead of using the goal image from the current trajectory for your policy, you should give it the same target from another trajectory to ... help it generalize? It otherwise uses a straight forward architecture and demonstrates within Minecraft.

Critical strengths

The key insight is "goal images decoupled from the execution environment" and the paper appears to show strong rules in the tables compared to baselines.

Critical weaknesses

To be totally transparent, I don't really understand what the paper is trying to show or teach me. If the argument is simply about the choice of goal representation then shouldn't language be the best representation since it won't have any visual artifacts? if the visual scene is necessary then what is the appropriate way to choose a comparison? Is hard negative mining necessary? There should probably be curves showing how some properties of samples are better/worse than others.

The results seem to indicate text is not the answer but why is not clear. The precise impacts of architectural and training details that change from baselines are also unclear -- perhaps the best version of this paper would be a side-by-side of all differences between the architecture presented and ROCKET-2, with a breakdown of exactly how each decision plays into the results. Minimally we need ROCKET-2 trained with the same goal image swapping as Attacca?

This concern is further amplified by, Table 3 which demonstrates that most of the gains are not related to the residual or phase? This makes it even harder to understand where the gains come from.

Questions for authors

I have concerns about generality or use of the term visuomotor but these are all minor compared to the issues raised above.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer mTVi (1/2)

We thank the reviewer for raising these concerns. All line, table, figure, and section numbers below refer to our revision.

Research Challenges and Contributions. The central question of our paper is how to train a visual goal-conditioned policy to execute successive tasks under state continuity (Lines 15-18, 47-50, and 187-189). Each task starts from the state left by the previous one, and its target is often out of view. This setting requires target acquisition and interaction without direct scene correspondence between the reference and the agent's observations (Lines 155-157 and 181-183). This is a practical embodied setup, yet challenging to recent strong baselines. At inference they build the goal from the current view, which presumes a visible target. During training they take the goal from the demonstration itself, which shares its scene (Lines 116-120). Attacca addresses it by training the policy with context-decoupled goal sampling, goal-conditioned target grounding with residual feedback, and behavioral-phase conditioning (Lines 83-90).

Our experiments provide three findings, the first of which answers the question on language goals.

shouldn't language be the best representation since it won't have any visual artifacts? [...] The results seem to indicate text is not the answer but why is not clear.

  1. Text goals generalize poorly because they do not show the target's appearance. A text goal only names the target, so the policy must learn from its demonstrations what each class looks like, and it has no appearance to match for a class absent from training. JARVIS-VLA, the strongest text agent trained on our demonstrations, reaches only 0.080, 0.165, and 0.085 clean success on Mine, Hunt, and Place (Table 1). On held-out classes it falls to 0.000, 0.083, and 0.000, and it completes no long-horizon chain (Table 2). STEVE-1 is likewise limited by weak object-level grounding across environments (Lines 349-351). A masked goal image instead shows the target's appearance, so the policy only has to find a matching instance in its own view, for unseen classes as well.

  2. Existing visual goal-conditioned policies depend on the context of the reference or need a visible target. STEVE-1 with video goals and GROOT follow reference gameplay clips, which entangle the target with the demonstrator's behavior and the surrounding layout, so they depend heavily on the context of the reference (Lines 109-115). ROCKET-1 and ROCKET-2 with a current-view goal need a target mask in the current observation, so they obtain a valid goal only when the target is visible (Lines 115-119 and 353-355). Building this mask online with a VLM and SAM 2 also slows them to 14 and 20 FPS, against 65 FPS for Attacca (Lines 356-357). ROCKET-2 with a masked goal image, trained with each goal image taken from the imitated demonstration itself, relies on the background of the goal image rather than the masked target and acts by reproducing the goal view (Fig. 5, Lines 210-213 and 470-478). In our tasks, STEVE-1 with video goals and GROOT reach at most 0.040 clean success, the current-view pipelines at most 0.180, and the fine-tuned ROCKET-2 0.250 to 0.340, against 0.390 to 0.475 for Attacca (Table 1). On the long-horizon chains, no baseline exceeds 0.17 chain success, against 0.23 to 0.60 for Attacca (Table 2). We aim for a policy that finds and pursues the target itself, not its context.

  3. Feeding the policy's own grounding and phase estimates into control is the key. On the same dense target supervision, residual feedback raises chain success by 0.090, 0.090, and 0.025, and phase conditioning adds another 0.105, 0.095, and 0.095 (Table 3 in our revision, Lines 404-413). With both, Attacca achieves the highest success on every short-horizon task and long-horizon chain (Tables 1 and 2).

Goal sampling and hard negatives. No mining is involved, and Eq. 3 is the only rule. Each time a trajectory is sampled, we draw a random demonstration of the same class from a different world and use a target-visible frame with its mask as the goal. There is no contrastive loss, hence no negatives. Varying the scene while fixing the class is what matters. A single fixed goal image per class would act as a class label that the policy never needs to look at. As for how properties of the goal image affect success, Table 5 and Fig. 8 vary its world, capture distance, and background, and Attacca stays stable across all of them.

AuthorsOpenReview ↗

Author Response to Reviewer mTVi (2/2)

All line, table, figure, and section numbers below refer to our revision.

Where the gains come from.

Minimally we need ROCKET-2 trained with the same goal image swapping as Attacca?

Reflecting the reviewer's constructive comments, we have further performed additional analyses during the rebuttal period. The submitted Table 3 only toggled residual feedback and phase conditioning, in one world per chain with 50 seeds (standard error up to 0.07). We expanded the ablation to four worlds per chain (Fig. 18 in our revision) with 50 seeds each (standard error up to 0.035). The first block below fine-tunes ROCKET-2 with its native visibility, centroid, and bounding-box losses, with and without our goal sampling. The second block adds the components of Attacca one at a time (chain success ± standard error, n=200).

PolicyDiamond PickaxeWolf FeedingNether Portal
First block: ROCKET-2
Fine-tuned ROCKET-20.160 ± 0.0260.170 ± 0.0270.135 ± 0.024
+ context-decoupled goal sampling0.310 ± 0.0330.140 ± 0.0250.115 ± 0.023
Second block: components of Attacca
Behavior cloning0.275 ± 0.0320.105 ± 0.0220.010 ± 0.007
+ context-decoupled goal sampling0.375 ± 0.0340.140 ± 0.0250.095 ± 0.021
++ parametric target supervision0.410 ± 0.0350.120 ± 0.0230.080 ± 0.019
++ dense target supervision (replaces parametric)0.410 ± 0.0350.155 ± 0.0260.110 ± 0.022
+++ residual feedback pathway0.500 ± 0.0350.245 ± 0.0300.135 ± 0.024
++++ phase conditioning (Ours)0.605 ± 0.0350.340 ± 0.0330.230 ± 0.030

With our goal sampling, ROCKET-2 nearly doubles on Diamond Pickaxe (0.160 to 0.310) but not on Wolf Feeding and Nether Portal. ROCKET-2 runs with classifier-free guidance (k=1), whose effect differs from task to task (Table 9 in our revision), so its success reflects guidance as well as training. Without guidance, as in the second block, goal sampling raises success on all three chains (0.275 to 0.375, 0.105 to 0.140, and 0.010 to 0.095). Either way, goal sampling alone does not explain the gain of Attacca, which roughly doubles ROCKET-2 trained with our goal sampling on every chain (0.605, 0.340, and 0.230 against 0.310, 0.140, and 0.115).

Table 3 which demonstrates that most of the gains are not related to the residual or phase?

We believe this concern may stem from a misunderstanding of the distinction between target supervision and feedback in Table 3. Contrary to the reviewer's interpretation, the ablation results demonstrate that residual feedback and phase conditioning are the primary drivers of Attacca's performance gains (Lines 404-413). In contrast, neither parametric nor dense target supervision alone changes the success rate by more than 0.035. Our mask head reads the patch features before the causal Transformer, whereas the parametric head of ROCKET-2 sits after it. Therefore, only the dense mask prediction can be fed back into the Transformer input (Lines 407-409). This feedback through the residual pathway raises success by 0.090, 0.090, and 0.025, and phase conditioning adds another 0.105, 0.095, and 0.095. Residual feedback and phase conditioning account for 59%, 79%, and 55% of the gain over behavior cloning.

Generality and "visuomotor".

I have concerns about generality or use of the term visuomotor

In the new Sec. 4.6, Attacca, without further training, reaches a red car in the photorealistic UnrealZoo from a goal image taken in another world in 9 of 20 episodes, against 3 for ROCKET-2. We used "visuomotor" as ROCKET-2 does in its title, for a policy that maps pixels to low-level actions. To avoid suggesting robot experiments, the revision no longer uses the term and describes Attacca as a visual goal-conditioned policy (Sec. 1).

We hope this clarifies the paper and would be grateful if the reviewer could reconsider the assessment.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer mTVi,

We sincerely appreciate your service as a reviewer. Your comments have helped us improve the completeness of our paper. When your time permits, could you please read our responses and check whether your concerns are addressed?

During the rebuttal period, we made every effort to address your concerns as follows.

  • Source of the performance gains. As you suggested, we trained ROCKET-2 with our goal sampling. We also expanded the ablation to add the components of Attacca one at a time (n=200, Table 3 in our revision). Attacca reaches about twice the chain success of this ROCKET-2 on all three chains, and residual feedback and phase conditioning account for 59%, 79%, and 55% of the gain over behavior cloning.
  • Generality and "visuomotor". We added an evaluation in the photorealistic UnrealZoo without further training (Sec. 4.6 in our revision), and the revision no longer uses the term "visuomotor".
  • Research challenges and contributions. We clarified the problem that the paper studies, why text goals generalize poorly, and how the goal image is sampled (Eq. 3).

We strongly believe that our responses and the additional results have addressed your concerns. We would be grateful if you could reconsider the rating in light of these clarifications and additional results. If any points remain unclear, or if further experiments would help, please let us know. We remain fully committed to the discussion and will make every effort to address any remaining concerns thoroughly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment: Summary of Revisions

We thank all reviewers for their constructive feedback. We have uploaded a revision in which the changes are marked in green. All table, figure, and section numbers below refer to our revision.

  1. Stronger ROCKET-2 baseline (Reviewer 28ox). Following the suggestion of Reviewer 28ox, we now evaluate ROCKET-2 with the classifier-free guidance of its official implementation (k=1). This setup needs two forward passes per step, so it runs at about half the speed of Attacca (30 against 65 FPS), but it gives a stronger baseline. Every quantitative ROCKET-2 result uses it (Tables 1, 2, and 5, Figs. 3b, 4, and 8, and Sec. 4.6), and Table 9 reports a sweep over k. Against this stronger baseline, Attacca still achieves the highest success on every short-horizon task (0.390 to 0.475 against 0.250 to 0.340) and on every long-horizon chain (0.605, 0.340, and 0.230 against 0.160, 0.170, and 0.135).

  2. Expanded ablation (Reviewers mTVi, 28ox, and oayj). Table 3 now adds the components of Attacca one at a time and is evaluated in four worlds per chain (n=200). Residual feedback and phase conditioning account for 59%, 79%, and 55% of the gain over behavior cloning. The individual responses also report ROCKET-2 trained with our goal sampling.

  3. Larger long-horizon evaluation. The rows of Table 2 with different-world goal images are now evaluated in four worlds per chain (n=200, Fig. 18).

  4. Generality beyond Minecraft (Reviewers mTVi and oayj). The new Sec. 4.6 evaluates Attacca in the photorealistic UnrealZoo without further training. It reaches the target in 9 of 20 episodes, against 3 for ROCKET-2.

  5. Additional details. Appendix D.1 and Table 8 list the fine-tuning settings of ROCKET-2 (Reviewer 28ox). Appendix A compares our goal image with background-invariant goal formats (Reviewer oayj). The paper no longer uses the term "visuomotor" (Reviewer mTVi).

The numbers in the abstract and introduction are updated accordingly. Attacca improves over the strongest baseline by 1.4 to 1.6 times on the short-horizon tasks and by 1.7 to 3.8 times on the long-horizon chains, and our conclusions are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.