- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Attacca trains a goal-conditioned visuomotor policy under an imitation-learning setting. Traditional approaches typically obtain the goal image from the same demonstration trajectory or the same world. In contrast, for each demonstration trajectory, Attacca requires the goal image to come from another trajectory with the same task-target category, thereby decoupling the goal-associated object from the execution context.
Attacca is a direct improvement over ROCKET-2 on the Minecraft tasks. Compared with ROCKET-1, ROCKET-2 replaces per-frame dynamic goal mask conditioning, while using per-frame target masks only as privileged training supervision for centroid and visibility prediction. Attacca does not remove this train-time annotation requirement; instead, it introduces a new goal-image construction scheme and strengthens ROCKET-2's centroid/visibility supervision into dense segmentation supervision, together with an additional behavioral-phase objective.
Overall, Attacca achieves substantial improvements over ROCKET-2 on long-horizon tasks, but the ablation studies are not sufficiently comprehensive, and it remains unclear how transferable the proposed design choices are to tasks beyond Minecraft.
Critical strengths
Attacca addresses two failure modes relevant to long-horizon goal-conditioned control: the inability of current-view goal-construction pipelines to instantiate a valid goal when the target is initially out of view, and the context sensitivity of ROCKET-2-style reference-goal policies when the goal view is decoupled from the execution scene.
Critical weaknesses
- The different-world goal-image design appears to primarily address a limitation of ROCKET-2 under the specific Minecraft setup. In more general goal-conditioned control problems, the goal may need to specify not only a target object category, but also the states and relations of multiple objects within a particular environment. In such cases, deliberately decoupling the goal image from the execution context may no longer be appropriate. The generality of this design choice is therefore unclear.
- It is already well established that richer and more fine-grained supervision can improve representation learning and downstream control. From this perspective, the observation that replacing ROCKET-2-style centroid/visibility supervision with denser segmentation supervision, together with an additional behavioral-stage classification objective, improves performance is not by itself a sufficiently informative finding. A stronger analysis is needed to establish what new capability or principle is enabled by these supervision signals beyond simply providing more privileged annotations.
Overall, although the paper motivates its method by the challenges of goal construction and subtask transitions in long-horizon tasks, it remains unclear what new insights or broadly applicable design principles the work provides for general long-horizon embodied control beyond the particular Minecraft setting.
Questions for authors
- Could the authors provide a full incremental ablation that isolates the contribution of each newly introduced component, for example: ROCKET-2 with same-world goal sampling → + different-world goal sampling → + dense-segmentation auxiliary loss → + phase auxiliary loss → + Attacca feedback/conditioning?
- Is there direct evidence that fully disentangle the contribution of replacing ROCKET-2-style centroid/visibility supervision with dense segmentation? How much additional parameter and latency overhead is introduced by the complete grounding pathway? Can the authors compare dense-mask supervision against a ROCKET-2-style centroid+visibility objective under a matched backbone and comparable auxiliary-model capacity?
- ROCKET-2 already demonstrates cross-view goal specification in which the target need not be visible in the agent's current observation. Could the authors clarify how Attacca's motivation differs from these existing capabilities? Is the primary motivation for Attacca's different-world goal images to reduce context sensitivity during training, or to avoid failures of test-time goal construction when the target is outside the current field of view?
- Is the different-world goal a particular form of task embedding? If so, the same motivation seems to admit more general task-embedding designs, such as scale- and background-invariant embeddings that jointly represent the target object and the interaction type.
- Is the fixed phase classification essentially a Minecraft-specific structural prior, with the ground-truth phase labels generated according to predefined rules? If so, does this narrow the task formulation to tasks that naturally conform to this predefined stage structure?
- Could the authors discuss what more general conclusions or design principles for goal-conditioned visuomotor policies can be drawn from the proposed method beyond the Minecraft tasks?
6 comments
Author Response to Reviewer oayj (2/4)
All line, table, figure, and section numbers below refer to our revision.
Q1. Full incremental ablation.
Reflecting the reviewer's constructive comments, we have further performed additional analyses during the rebuttal period. We expanded the ablation to four worlds per chain (Fig. 18 in our revision) with 50 seeds each. The first two rows fine-tune ROCKET-2 with its three auxiliary losses, with and without our goal sampling. They use classifier-free guidance with coefficient k=1, as in the main results. The remaining rows add the components of Attacca one at a time, starting from behavior cloning, without guidance as in Attacca (chain success ± standard error, n=200). The phase head is added together with its conditioning.
| Policy | Diamond Pickaxe | Wolf Feeding | Nether Portal |
|---|---|---|---|
| ROCKET-2 | 0.160 ± 0.026 | 0.170 ± 0.027 | 0.135 ± 0.024 |
| + context-decoupled goal sampling | 0.310 ± 0.033 | 0.140 ± 0.025 | 0.115 ± 0.023 |
| Behavior cloning | 0.275 ± 0.032 | 0.105 ± 0.022 | 0.010 ± 0.007 |
| + context-decoupled goal sampling | 0.375 ± 0.034 | 0.140 ± 0.025 | 0.095 ± 0.021 |
| ++ parametric target supervision | 0.410 ± 0.035 | 0.120 ± 0.023 | 0.080 ± 0.019 |
| ++ dense target supervision (replaces parametric) | 0.410 ± 0.035 | 0.155 ± 0.026 | 0.110 ± 0.022 |
| +++ residual feedback pathway | 0.500 ± 0.035 | 0.245 ± 0.030 | 0.135 ± 0.024 |
| ++++ phase conditioning (Ours) | 0.605 ± 0.035 | 0.340 ± 0.033 | 0.230 ± 0.030 |
With our goal sampling, ROCKET-2 nearly doubles on Diamond Pickaxe (0.160 to 0.310) but not on Wolf Feeding and Nether Portal. ROCKET-2 runs with guidance, whose effect differs from task to task (Table 9 in our revision), so its success reflects guidance as well as training. Without guidance, context-decoupled goal sampling raises success on all three chains (0.275 to 0.375, 0.105 to 0.140, and 0.010 to 0.095). Attacca still roughly doubles ROCKET-2 trained with our goal sampling on every chain, so goal sampling alone does not explain the gain.
Q2. Dense masks against centroid and visibility supervision, and overhead.
Yes. The parametric and dense rows above share the backbone and all other settings and differ only in the target supervision. Target supervision alone, parametric or dense, changes success by at most 0.035 (Table 3 in our revision). Our mask head reads the patch features before the causal Transformer, whereas the parametric head of ROCKET-2 is a linear head on the Transformer output. Therefore, only the dense prediction can be fed back into the Transformer input. This feedback through the residual pathway raises success by 0.090, 0.090, and 0.025, and phase conditioning adds another 0.105, 0.095, and 0.095.
Both heads are small next to the shared backbone of about 187M parameters (7,175 for the parametric head and 262,657 for the dense mask head), so head capacity does not drive this comparison. The complete grounding pathway adds 2.11M parameters (+1.13%), 0.21 GFLOPs per step (+0.3%), and 3.8% latency. ROCKET-2 with classifier-free guidance runs two forward passes at every step, so Attacca runs at about twice the FPS of ROCKET-2 (65 against 30 FPS, Table 1 in our revision).
Author Response to Reviewer oayj (3/4)
All line, table, figure, and section numbers below refer to our revision.
Q3. Different-world goals and relation to ROCKET-2.
Q3. [...] Is the primary motivation for Attacca's different-world goal images to reduce context sensitivity during training, or to avoid failures of test-time goal construction when the target is outside the current field of view?
Difference from ROCKET-2. ROCKET-2's cross-view goal alignment assumes an accompanying human who captures the goal view in advance, in the very world the agent will act in. An agent that explores an unknown world on its own cannot have such a goal image (Lines 155-157). In our setting, the goal image is taken from a different world to show the target's appearance. The world in the goal image does not match the agent's current world, and the agent performs object-centric control from the target's appearance alone. In addition, ROCKET-2 does not evaluate the case in which the target is not visible in the agent's current observation. Its quantitative benchmark starts every task with the target in view. In all our short-horizon tasks, the target is not visible from the start, and in our long-horizon task chains each new target must be found from wherever the previous task ended (Tables 1 and 2).
Primary motivation. Both motivations follow from this setting. At test time, a goal cannot be built from the current view while the target is out of view (Lines 353-355). A policy trained with each goal image taken from the imitated demonstration itself learns to depend on the shared scene, and at test time it degrades with goals from other worlds (Fig. 3b). Context-decoupled goal sampling removes this dependence during training.
Attacca also works when a goal image taken in the agent's own world is available, as in InstanceImageNav (Krantz et al., 2023). It reaches 0.89 with goal images taken in the execution world and 0.81 with goal images from another world (Table 5).
Q4. Goal images as visual task embeddings.
We agree that the masked goal image can be seen as a visual task embedding, one that specifies the target by its appearance. The reviewer's suggestion then raises two questions, whether this embedding should be made invariant to background and scale, and whether it should also carry the interaction type. We tested both on Mine with the 200 episodes of Table 1. We compared our goal with two formats that are background-invariant or scale- and background-invariant by design, and for each format we also merged the interaction type into the goal embedding. In the merged variants, the interaction-type embedding is concatenated to each goal token and projected back to the token dimension by a linear layer initialized to pass the goal token through, and the separate interaction-type input of the Transformer is set to a fixed null value. Each variant retrains Attacca with the same data and training setup and changes only the goal input, in training and at test (see Fig. 9 in our revision).
| Goal format | ID (n=140) | OOD (n=60) | ID, merged | OOD, merged |
|---|---|---|---|---|
| Ours (masked goal image) | 0.450 | 0.250 | 0.407 | 0.183 |
| Background removed (white background, native scale) | 0.336 | 0.117 | 0.329 | 0.117 |
| Object crop (rescaled, white padding) | 0.321 | 0.100 | 0.314 | 0.033 |
Both hand-designed invariant formats stay below Ours on trained and held-out classes, so learning the invariance from varied scenes works better than removing the scene by hand. Merging the interaction type into the goal embedding does not improve any format, and it lowers held-out success for Ours (0.250 to 0.183) and for the object crop (0.100 to 0.033). Our policy already uses both the target appearance and the interaction type, as the goal image and a separate interaction-type input to the causal Transformer (Lines 192-201), and keeping them separate works better than merging them into a single embedding.
Author Response to Reviewer oayj (1/4)
We thank the reviewer for the thoughtful review and the precise questions. All line, table, figure, and section numbers below refer to our revision.
W1. Generality beyond Minecraft.
We believe this concern may stem from a misunderstanding of our method. Attacca is not an incremental fix for a limitation of ROCKET-2 in Minecraft. It addresses a clearly defined problem, goal-directed control under state continuity, in which each task starts from the state left by the previous one and the next target is often out of view (Lines 15-18, 47-50, and 187-189). Any embodied agent that executes a long-horizon objective as a sequence of tasks faces this problem, and our training method is not tied to Minecraft.
Goals that specify the states and relations of multiple objects, such as "arrange the objects on the desk" or a goal image of a fully set table, define a different problem. Our setting specifies the object to interact with in each task, not the goal state of a scene. Embodied control is studied as such separate problems, as navigation, locomotion, and manipulation are in robotics. A single policy that handles every type of goal does not yet exist and remains a fundamental research agenda of embodied AI.
State continuity, however, arises inside such goals as well. Consider "place the blue cup to the right of the red plate." The agent must (1) find and pick up the blue cup, (2) find the red plate, and (3) place the cup to its right. Picking up the cup leaves the agent with a different camera pose, from which it must search for the red plate. A policy must do this reliably from whatever state the previous task leaves. This robustness under state continuity is the goal of our work, and our long-horizon chains test it directly (Table 2).
The problem is also not specific to Minecraft. In UnrealZoo, a photorealistic environment with a target class that never appears in Minecraft, Attacca reaches the red car in 9 of 20 episodes against 3 for ROCKET-2 without further training (Fig. 6c in our revision). Its mask head, trained only on Minecraft, activates on the car once it enters the view, and its predicted phase switches from Search to Approach at that moment (Fig. 6a and 6b, Lines 522-527).
W2. What the method enables beyond more supervision.
This is a misunderstanding. Our gain does not come from richer supervision. In our ablation (Table 3 in our revision), the parametric and dense rows share the backbone, data, goal sampling, and training recipe, and both take their labels from the same renderer masks. The parametric row trains the visibility, centroid, and bounding-box head of ROCKET-2, and the dense row replaces all three losses with our mask loss. This switch changes success by at most 0.035. The gain comes from feeding the policy's own predictions back into control. Residual feedback lets the policy act on its estimate of where the target is visible, and phase conditioning lets it act on its estimate of whether it should search, approach, or interact. On the same dense supervision, residual feedback and phase conditioning account for 59%, 79%, and 55% of the gain over behavior cloning (please see Q1 and Table 3 in our revision).