acceptodds
Under review as a conference paper at ICLR 2027

Look Ahead Together, Act Locally: Learning a Shared Predictive Brain for Multi-Robot Manipulation

Abstract

Multi-robot manipulation requires robots to coordinate their actions while reliably performing their individual operations. Existing approaches often face a trade-off between decentralized policies that preserve per-robot controllers but lack global coordination context, and centralized policies that model team interactions but tightly couple coordination with joint action generation. In this paper, we introduce a multi-robot manipulation framework built on the principle of Look Ahead Together, Act Locally. Our method separates coordination from control through a shared predictive Brain and individual Action Experts. Given all robots' observations and a language instruction, the Brain learns a shared prediction of the collaborative future through alignment with future multi-view 3D features, and derives corresponding guidance for each Action Expert. The frozen Brain then guides independently adapted single-robot policies, enabling coordinated action while preserving their modularity. Experiments on RoboFactory demonstrate an 85.5% average success rate across six tasks, consistently outperforming decentralized and centralized baselines, while preserving a modular interface that enables individual experts to be replaced without retraining the Brain or other controllers.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Multi-robot manipulation, Robot learning, Feature alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gfHKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a multi-robot manipulation framework called "Look Ahead Together, Act Locally." The framework decouples shared coordination mechanisms from the process of individual action generation. It introduces a shared, predictive "brain" that processes observations from all robots and generates robot-specific guidance by aligning these inputs with future multi-view 3D features extracted from a frozen foundation model. In six RoboFactory tasks involving 2 to 4 robots, this method achieved an average success rate of 85.5%, outperforming several baselines. The authors also conducted ablation studies—covering "future feature alignment," "multi-view observations," and the "3D teacher model"—which yielded promising results.

Critical strengths

This paper addresses a key challenge in multi-robot manipulation: how to facilitate collaboration through shared information while maintaining the flexibility for individual robot controllers to adapt independently. The proposed architecture decouples a "shared predictive Brain" from "robot-specific Action Experts," offering a viable alternative to both fully decentralized policies and centralized joint-action models. Experiments involving the substitution of local experts (Section 4.5) provide preliminary evidence for the modular nature of this architecture.

The experimental results are compelling and strongly support the concept of "coordination with future awareness." The evaluation covers six collaborative manipulation tasks, which span a range of coordination requirements—from synchronous operations to tasks involving sequential dependencies. As shown in Table 1, the proposed method outperforms the listed baselines across all metrics, achieving an average success rate of 85.5%. Table 2 presents a comparison between "future-state alignment" and "current-frame alignment," with a striking result: omitting "future-state alignment" causes the average success rate to drop to 68.0%.

Critical weaknesses

1. Insufficient control in comparative experiments makes it difficult to attribute the reported performance gains to the proposed coordination mechanism.

In Section 4.1 (lines 317–323), baseline results are cited directly from prior literature rather than being reproduced under a unified training setup. Due to differences in model initialization, training pipelines, and observation/action horizons, it is difficult to determine the extent to which the significant performance improvements in Table 1 are attributable to the proposed "shared prediction coordination" mechanism.

Base on above, i question is "predictive supervision" methods, it raises a critical question: which component is primarily driving the substantial performance gains over existing methods?

Figure 4 presents a comparison of execution times for the "No Brain" setting but lacks corresponding success rate evaluations across the six tasks under matched training conditions.

2. Ablation experiments fail to adequately justify the necessity of several key methodological choices.

The proposed framework integrates future feature prediction, multi-depth guidance, and 3D representation alignment. Consequently, clearly identifying the specific contributions of this component combination is crucial for evaluating the method's innovation. First, the alignment ablation experiment introduces a confounding variable into the training pipeline. In Section 4.4 (lines 424–426), the "w/o alignment" variant jointly trains the "Brain" and "Action Experts" using only the action loss; in contrast, the full method trains and freezes the "Brain" separately prior to expert adaptation. Therefore, the drastic performance drop from 85.5% to 9.8% cannot be attributed solely to the removal of feature alignment; it may also reflect differences in optimization stability or the benefits of staged training.

Questions for authors

Q1. Contribution of the shared "Brain." Could the authors report the success rates of "Action Experts" trained using the same pre-trained initialization, demonstration data, and comparable adaptation processes, but without guidance from the "Brain"? Does the "No Brain" configuration in Figure 4 correspond to such a baseline?

Q2. Isolating the impact of feature alignment. In Section 4.4, the "w/o alignment" variant also alters the training process by switching to the joint optimization of the "Brain" and the "Action Expert." Could the authors explain why this change, and whether the effect of alignment could be evaluated while keeping the optimization strategy comparable?

Q3. Necessity of multi-depth and geometric supervision. Have the authors evaluated variants that use only single-depth guidance, or schemes involving alignment with non-geometric future visual features?

Q4. Scope of the modularity claim. All six RoboFactory tasks share the same "Brain" checkpoint, or is a separate "Brain" trained for each task?

5 comments

Response to Reviewer gfHK (1/2)

(W1a, Q1) Baseline results in Table 1 are cited from prior literature rather than obtained under a matched setup with the same pretrained Experts and adaptation process, and it is unclear whether the “No Brain” curve in Fig. 4 corresponds to such a baseline.

Thank you for raising these concerns. In the submitted paper, the baseline scores in Table 1 were cited from prior literature (L322–323). To separate the Brain's contribution from pretraining and the training pipeline, we added the baseline the reviewer describes, Decentralized-DP with the same single-robot pretraining, demonstrations, and Expert hyperparameters as our method (L350–353). We trained it and our method three times each and evaluated every run on 300 episodes per task. This control reaches 62.0%, and adding our Brain raises success to 86.8% (+24.8 points). Without pretraining, the same Decentralized-DP reaches 33.7%, so both pretraining and the Brain contribute substantially. The pretrained VLM and its frozen DINOv2 and SigLIP encoders explain only part of the Brain's gain. With the same VLM and encoders but without alignment training, the Brain reaches 72.8%, and alignment adds a further 14.0 points (see W2a).

MethodSingle-robot pretrainingBrainAvg. success (%)
Decentralized-DP✗✗33.7
Decentralized-DP✓✗62.0 ± 0.8
Ours✓✓86.8 ± 3.0

Figure 4 compares our method with Decentralized-DP without single-robot pretraining, as stated in the updated figure legend. It therefore illustrates the from-scratch setting, whereas the matched pretrained no-Brain control reaches 62.0% (Table 3). We have updated the figure legend and added the from-scratch results to Table 1 (Sec. 4.2) and the matched pretrained controls to Table 3 (Sec. 4.4) of the revised paper.

(W1b) Removing multi-view inputs in the original Table 2 lowers the average success rate only from 85.5% to 80.8%, raising the question of which component primarily drives the performance gains.

Thank you for raising this question. The gains over existing methods come from both single-robot pretraining and the predictive Brain, and within the Brain they come primarily from predicting the team's future rather than from shared current views at inference. Our matched controls separate these gains into three steps (W1a, W2a). Single-robot pretraining raises Decentralized-DP from 33.7% to 62.0%, and a frozen VLM Brain that sees all robots' views raises it to 72.8%. Training the Brain to predict the team's future then adds a further 14.0 points, reaching 86.8%. In the original Table 2 as well, where all variants share the same 100-episode evaluation, aligning to current instead of future features costs 17.5 points, much more than the 4.7 points from removing shared views.

Removing shared views costs less because the alignment targets in this ablation are unchanged. The VGGT teacher still processes all robots' future views jointly (L450–451), so the Brain still learns team-level information during training and only loses access to other views at inference. This access matters most for larger teams, as success on the four-robot Take Photo task drops from 72% to 60% in the original Table 2. We have revised Sec. 4.4 (“Multi-View versus Single-View Brain Inputs”) of the revised paper to clarify that the teacher still processes all robots’ future views jointly and that this ablation tests the value of shared current views at inference time.

(W2a, Q2) In Sec. 4.4, the “w/o alignment” variant (9.8%) also switches to joint optimization of the Brain and Action Experts; why was this change made, and can alignment be evaluated while keeping the optimization strategy comparable?

Thank you for this insightful observation. The original variant changes the training pipeline as well as removing alignment. Without alignment, the Brain has no Stage-1 objective, so we trained it jointly with the Experts in one stage using only the action loss (L424–426). Training it longer, to 200 epochs, still averages only 5.8%, suggesting that this joint training is hard to optimize. To isolate alignment, we added a control that keeps the full method's two-stage pipeline (Sec. 3.4) and removes only the alignment training, leaving the Brain frozen at its pretrained VLM weights. This control reaches 72.8%, whereas the aligned Brain reaches 86.8% (+14.0 points), showing that alignment training accounts for this gain under the same training pipeline. We have added the frozen-VLM control alongside the original joint-training variant in Table 3 (originally Table 2) and clarified their different training procedures in Sec. 4.4 of the revised paper.

MethodAvg. success (%)
Frozen VLM Brain72.8 ± 0.8
Ours86.8 ± 3.0

Response to Reviewer gfHK (2/2)

(W2b, Q3) Have the authors evaluated variants that use only single-depth guidance, or schemes involving alignment with non-geometric future visual features?

Thank you for suggesting these comparisons. To test multi-depth guidance, we trained a single-depth variant that aligns and reads guidance only from the last Brain block (block 23) instead of blocks 8, 13, 18, and 23 (Eq. 2, L345), keeping the rest of the pipeline unchanged. This variant reaches 81.2% average success (Table 3).

To address the reviewer’s question about non-geometric future visual features, we replaced the VGGT teacher with future DINOv2 features [1], extracted independently from each robot’s view. This variant reaches 83.8% average success. DINOv2’s frozen features support both semantic segmentation and monocular depth estimation [1, 2], capturing object semantics alongside spatial structure. These complementary cues can provide informative future targets for anticipating changes in objects and robot configurations, helping explain its strong performance.

This variant also retains our shared Brain’s access to all robots’ current views and its future-feature prediction objective. Together with the matched controls in W1a and W2a, these results support our central design of learning a shared predictive Brain to anticipate collaborative scene outcomes and guide individual Experts. We have added both variants and their discussion to Sec. 4.4 (Table 3, originally Table 2) of the revised paper.

VariantAvg. success (%)
Ours (Table 3)86.8 ± 3.0
w/o multi-depth (block 23)81.2
Alignment w/ DINOv283.8

(Q4) Do all six RoboFactory tasks share the same Brain checkpoint, or is a separate Brain trained for each task?

Thank you for asking for this clarification. All six RoboFactory tasks share a single Brain checkpoint, trained jointly on their demonstrations with teams of two, three, and four robots, which is why each training batch groups samples with the same number of views (L728–730). Each robot in each task keeps its own Expert (L746–747), while all of them read guidance from this one frozen Brain. The Expert replacement in Sec. 4.5 therefore leaves a Brain shared by all six tasks unchanged, and the same checkpoint also guides the replacement Experts on Two Robots Stack Cube and Place Food (see our response to Reviewer rtaY, W2c). We have clarified in Sec. 4.1 (“Implementation Details”) of the revised paper that all six tasks share a single Brain trained jointly on their demonstrations.

We thank the reviewer again for the detailed questions, which led to the matched controls, the clean alignment control, and the depth and teacher ablations. We hope these responses address your concerns.

Note: Line numbers (L###) refer to the original submission, while section, table, and figure numbers refer to the revised paper, where new content is highlighted in blue.

References

[1] DINOv2: Learning Robust Visual Features without Supervision. TMLR 2024.

[2] Probing the 3D Awareness of Visual Foundation Models. CVPR 2024.

Dear Reviewer,

We sincerely appreciate your time and thoughtful consideration of our work.

As the discussion period is coming to a close, we would be grateful for any additional feedback you may have on our responses. We would be glad to address any remaining questions or concerns before the discussion concludes.

Thank you once again for your valuable time and consideration.

Best regards, Paper 21783 Authors

Reviewer_JgJMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a "look ahead together, act locally" policy for multi-robot manipulation tasks. In such a task setup, there are multiple robot arms collaborating to finish a task. The proposed policy have a world model like network trunk shared with multiple independent policy action head, one for each robot. The policy is evaluated in robofactory and show advantages against previous methods along this line.

Critical strengths

The proposed architecture is very clear and easy to understand, and makes sense for a policy that being deployed for multiple robot arm. The network trunk perceives all observations and therefore has the strongest representation power, and each action head handles the nuances for each robot arm.

Critical weaknesses

My main complaint is about technical novelty. The proposed architecture can be regarded as a very common choice in policy learning, except that existing policies (classical imitation policies, VLA, WAM, etc) may predict different modalities or action granularity in different heads, and this method predict actions of different robots. I do not learn much from the proposed technique.

My second concern is about the solidness of the evaluation. Table 1 shows the proposed method outperforms existing methods by a large margin. But most of those baselines get very low success rate and DP (diffusion policy) gets 0%, this looks not good to me and it is likely the baseline settings are skewed? I am a little skeptical here. But again I believe this task is a rather new direction, so I could be mis-informed.

Questions for authors

See weaknesses

3 comments

Response to Reviewer JgJM

(W1) The proposed architecture is a common policy-learning design (as in imitation policies, VLAs, and WAMs), except that its heads predict actions for different robots rather than different modalities or granularities.

Thank you for this comment. A shared trunk with one action head per robot is a form of the centralized joint-action policy we contrast with in Fig. 1b, where one network perceives all robots' views and its heads are trained together through the action loss (L075–081). Our framework departs from this design by separating coordination from control (Fig. 1c). The Brain is never trained with an action loss. It learns only to predict all robots' future views and is then frozen (Sec. 3.2). Each robot runs a complete single-robot policy instead of a head, with its own encoder, observation history, and proprioception, initialized from a pretrained single-robot policy and reading the Brain's guidance through zero-initialized adapters (Sec. 3.3). Because each Expert is adapted independently, one robot's Expert can be replaced by a different architecture, such as GR00T, without retraining the Brain or the other Experts (Sec. 4.5, with more tasks in our response to Reviewer rtaY, W2c). This is not straightforward when all heads depend on one jointly trained trunk.

Our results show that separating coordination from control outperforms coupling them in one network, especially when the Brain is trained to predict the team's future. With single-robot pretraining, a centralized joint-action policy (Centralized-DP) reaches 42.9%, below Decentralized-DP (62.0%). Adding our Brain to the same decentralized Experts raises success to 72.8% when the Brain is kept at its pretrained VLM weights, and to 86.8% when it is trained to predict the team's future (three training runs, 300 episodes per task, as in our response to Reviewer rtaY, W1a). We have added these matched pretrained controls, including the Frozen VLM Brain, to Sec. 4.4 (Table 3, originally Table 2) of the revised paper.

(W2) Most baselines in Table 1 get very low success rates and Diffusion Policy gets 0%, raising concern that the baseline settings may be skewed.

Thank you for raising this concern. The baseline scores in the submitted Table 1 were published results from RoboFactory, GauDP, and CLS-DP under the official RoboFactory protocol (L322–323). To check them, we retrained Centralized-DP and Decentralized-DP following our setting. Without pretraining, they still reach only 15.0% and 33.7%, so the low scores are reproducible rather than an artifact of the baseline settings. These tasks are hard to learn because an episode fails whenever any robot fails its part. Most 0% entries in the submitted Table 1, including both of Centralized-DP's, occur on the three- and four-robot tasks. For fair comparison purposes, we additionally consider Decentralized-DP with the same single-robot pretraining as our Experts (L096–097), which reaches 62.0%, still 24.8 points below our method (W1). We have added the retrained baselines to Table 1 (Sec. 4.2) and the matched pretrained controls to Sec. 4.4 (Table 3, originally Table 2) of the revised paper.

MethodPublished (submitted Table 1)Retrained by us
Centralized-DP5.015.0
Decentralized-DP26.533.7

We thank the reviewer again for the comments, which helped us clarify how our design differs from a shared trunk with per-robot heads and check the baseline results. We hope these responses address your concerns.

Note: Line numbers (L###) refer to the original submission, while section, table, and figure numbers refer to the revised paper, where new content is highlighted in blue.

Dear Reviewer,

We sincerely appreciate your time and thoughtful consideration of our work.

As the discussion period is coming to a close, we would be grateful for any additional feedback you may have on our responses. We would be glad to address any remaining questions or concerns before the discussion concludes.

Thank you once again for your valuable time and consideration.

Best regards, Paper 21783 Authors

Dear Reviewer,

Thank you for your thoughtful and constructive feedback.

We would be grateful for any feedback you may wish to share on our rebuttal.

Thank you again for your valuable time and consideration.

Best regards, Paper 21783 Authors

Reviewer_rtaYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a "Look Ahead Together, Act Locally" framework for multi-robot manipulation that separates coordination from control. A shared predictive Brain (a Qwen2.5-0.5B VLM with per-robot learnable queries) takes all robots' current views and the instruction as input. It is trained only by aligning its hidden states from multiple layers with frozen VGGT 3D features of future multi-view observations, and it outputs robot-specific guidance. The frozen Brain then guides a separate Action Expert for each robot. Each Expert is initialized from a pretrained single-robot diffusion policy and reads the guidance through zero-initialized residual cross-attention adapters, so individual Experts can be replaced independently. On six RoboFactory tasks with 2–4 robots, the method reaches an 85.5% average success rate, compared with 38% for CLS-DP and 19.7% for GauDP. Ablations cover future versus current alignment, shared versus local views, and the choice of 3D teacher, and an experiment shows that swapping two Experts for GR00T works without retraining the Brain or the other Experts.

Critical strengths
  1. A centralized predictive Brain combined with independently adaptable per-robot Experts addresses a real tension between decentralized and centralized multi-robot policies. The modular interface is shown to work in practice: two Experts are replaced with GR00T without retraining the Brain or the other Experts.
  2. The method is best on all six tasks and more than doubles the average success of the strongest baseline. On Three Robots Stack Cube and Take Photo it reaches 84% and 72%, while no baseline exceeds 22%, which supports the claim that shared anticipation helps when robots must build on their teammates' progress.
  3. Future alignment beats current-frame alignment (H=0), with the drop concentrated on stacking tasks. Shared multi-view input beats local input, especially in the 4-robot task. Using a different 3D teacher (DA3) still beats all baselines. The t-SNE analysis further shows that alignment produces robot-specific guidance.
Critical weaknesses
  1. All baseline numbers are copied from prior papers rather than re-run under a matched setup, and the results come from a single run of 100 episodes per task with no seeds or variance. The proposed system also uses much more prior knowledge than the baselines: a pretrained VLM (Qwen plus DINOv2 and SigLIP), a VGGT teacher, and Experts pretrained on additional single-robot tasks. There is no control baseline, such as a centralized or decentralized policy using the same pretrained encoders and single-robot pretraining, so it is hard to tell how much of the 47-point gain comes from the "predictive Brain" itself. The "w/o alignment" variant (9.8%) scoring far below plain Decentralized-DP (26.5%) also suggests an optimization issue in that setting rather than a clean ablation.
  2. All results come from one simulation benchmark, with no real-robot experiments. "Act locally" still requires every robot's views to be gathered by a central Brain at every step, so the communication, latency and failure-mode costs of this centralized perception go undiscussed. Modularity is shown only by swapping the Experts within one fixed task. Generalization to new team sizes, new tasks or new robot compositions is not tested, and these would likely require retraining the Brain. Finally, the core technique (future or 3D feature alignment across multiple layers) builds closely on single-robot work such as FLARE, Spatial Forcing and ROCKET, so the methodological novelty lies mainly in applying it to the multi-robot setting.
Questions for authors

see the weaknesses.

7 comments

Dear authors,

I have carefully read the comments from other reviewers, and I remain positive about this work at this stage. Looking forward to your response to the concerns I raised.

Best

Response to Reviewer rtaY (1/2)

(W1a) There is no control baseline under a matched setup (same pretrained encoders and single-robot pretraining) to show how much gain comes from the "predictive Brain" itself, and results have no seeds or variance.

Thank you for raising these concerns. We added three controls under a matched setup and found that the predictive Brain itself accounts for the gain. While the baseline scores in Table 1 are published results from RoboFactory [1], GauDP [2], and CLS-DP [3] (L322–323) that cannot isolate this contribution, our three controls share the same single-robot pretraining, demonstrations, and Expert hyperparameters (L350–353). The first two controls, pretrained Centralized-DP and Decentralized-DP, reach only 42.9% and 62.0%, whereas our method reaches 86.8% (+24.8 points over Decentralized-DP). The third, Frozen VLM Brain, keeps our full architecture with the same pretrained VLM (Qwen2.5-0.5B with DINOv2 and SigLIP) seeing all robots' views and guiding the same per-robot Experts through the same adapters, but omits future alignment (unlike Table 2's 9.8% "w/o alignment" row, clarified in W1b); even with this pretrained VLM, success is 72.8%, and our future alignment adds a further 14.0 points to reach 86.8%. Table 2 also shows that the VGGT [4] teacher alone does not explain the gain, as aligning to current instead of future features drops success from 85.5% to 68.0%.

To account for run-to-run variance, each model was trained three times and evaluated on 300 episodes per task (the 100 official seeds plus 200 new ones). Our method improves over the controls in every run, and its 86.8% ± 3.0% average is consistent with the 85.5% in Table 1, confirming that the reported gains are reliable. We will add these controls, with per-task results, to the revised paper.

MethodPerceptionActionVLM BrainFuture alignmentAvg. success (%)
Centralized-DPall robots' viewsone joint policy✗✗42.9 ± 1.0
Decentralized-DPown viewper-robot policies✗✗62.0 ± 0.8
Frozen VLM Brainall robots' viewsper-robot policies✓✗72.8 ± 0.8
Oursall robots' viewsper-robot policies✓✓86.8 ± 3.0

All rows use the same single-robot pretraining. Mean ± SD over three training runs, each evaluated on 300 episodes per task.

(W1b) The "w/o alignment" variant (9.8%) scores far below plain Decentralized-DP (26.5%), suggesting an optimization issue rather than a clean ablation.

Thank you for this observation. We would like to clarify that without alignment, the Brain has no Stage-1 objective, so we trained it jointly with the Experts in one stage using only the action loss (L424–426), which changes the training pipeline as well as removing alignment. Training it longer, to 200 epochs, still averages only 5.8%. The clean control is the frozen VLM Brain in W1a, which keeps the full method's two-stage pipeline (Sec. 3.4) and removes only the alignment training. It reaches 72.8%, whereas the aligned Brain reaches 86.8% (+14.0 points), showing that alignment training is what makes the Brain effective. We will replace the original row with this control in the revised paper.

(W2a) All results come from one simulation benchmark, with no real-robot experiments.

Thank you for raising this concern. Like the concurrent work CLS-DP [3], we evaluate in simulation rather than on real robots. To test whether the framework generalizes beyond RoboFactory [1], we add a second benchmark, RoboTwin 2.0 [5], using four dual-arm tasks with each arm treated as a separate robot (as in TwinVLA [6]). We train our method, a Decentralized-DP control, and GauDP [2] with the same demonstrations and evaluation seeds. Our method reaches 73.5% average success, outperforming both Decentralized-DP (55.0%, +18.5 points) and GauDP (53.0%, +20.5 points) across all four tasks.

MethodHandover BlockPut Object CabinetPlace Dual ShoesLift PotAvg.
Decentralized-DP4246389455.0
GauDP [2]5240249653.0
Ours7762569973.5

Success rate (%), one training run per method, 100 episodes per task.

Ref:

[1] RoboFactory: Exploring Embodied Agent Collaboration with Compositional Constraints. ICCV 2025.

[2] GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies. NeurIPS 2025.

[3] Distilling Collaborative Dynamics into Latent Space for Implicit Coordination in Decentralized Multi-Agent Manipulation. IROS 2026.

[4] VGGT: Visual Geometry Grounded Transformer. CVPR 2025.

[5] RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation. ICML 2026.

[6] TwinVLA: Data-Efficient Bimanual Manipulation with Twin Single-Arm Vision-Language-Action Models. ICLR 2026.

Response to Reviewer rtaY (2/2)

(W2b) "Act locally" still requires every robot's views to be gathered by a central Brain at every step, so the communication, latency, and failure-mode costs of this centralized perception go undiscussed.

Thank you for raising this point. We first clarify that the Brain does not run at every control step. It runs once per replanning update, every eight control steps, on the robots' current RGB images only (L297–298, L353–354). Each robot keeps its observation history, proprioception, and action generation local, and reuses the same guidance throughout denoising.

Centralized methods such as Centralized-DP and GauDP [2] also gather these views, and the cost is modest. Per update, the four robots in Take Photo upload 900 KB of uncompressed RGB and receive 224 KB of FP32 guidance in total, and the Brain takes only 72 ms (12%) of the 599 ms compute latency on one H200. Decentralized methods avoid this step, but with comparable latency to CLS-DP [3] (599 vs. 488 ms), our method achieves far higher success than all baselines in Table 1, and leads Decentralized-DP under the same pretraining by 24.8 points on RoboFactory [1] (W1a) and 18.5 points on RoboTwin 2.0 [5] (W2a). Regarding failure modes, the Brain is the only shared component. If its guidance is delayed or unavailable, each robot still runs its own Expert locally and can reuse the most recent guidance, whereas a centralized joint policy stops all robots when its single model fails. We will add these measurements to the revised paper.

MethodShared Brain (ms)Local Brain (ms)Action generation (ms)Total (ms)Success in Table 1 (%)
CLS-DP [3]—24.08463.47487.8438.0
GauDP [2]425.03—423.17848.2619.7
Ours72.37—526.52599.1585.5

(W2c) Modularity is shown only by swapping the Experts within one fixed task, without testing generalization to new team sizes, tasks, or robot compositions or whether these require retraining the Brain.

Thank you for suggesting a broader evaluation of expert replacement. A single Brain checkpoint already serves all six RoboFactory [1] tasks, with teams of two, three, and four robots (L311–312).

To test replacement beyond Take Photo, we extended the GR00T N1.7 [7] replacement experiments to Two Robots Stack Cube and Place Food, evaluating each on 300 episodes. Following Sec. 4.5, we adapt only the replacement Experts while keeping the Brain and any remaining Experts fixed. Across two- and four-robot teams, with both full replacement and mixed DP/GR00T teams, the same frozen Brain guides the replacement Experts. New tasks or team sizes would be added through Brain training, which needs no action labels (Sec. 3.2). We will add these results to the revised paper.

TaskRobotsExperts replacedOurs, DP Experts (%)Ours, Replacement (%)
Two Robots Stack Cube2All75.784.7
Place Food2All83.389.0
Take Photo (Sec. 4.5)4Arms 0 and 172.078.0

(W2d) The core technique (future or 3D feature alignment across multiple layers) builds closely on single-robot work such as FLARE, Spatial Forcing, and ROCKET, so the methodological novelty lies mainly in applying it to the multi-robot setting.

Thank you for pointing out these connections. Our alignment objective builds on these works, and the difference lies in where alignment is applied and what it serves (L139–146). FLARE [8], Spatial Forcing [9], and ROCKET [10] add alignment as an auxiliary loss inside a single policy, improving that policy's representation from its own observations. We instead make future alignment the only training objective of a separate Brain that sees all robots' views, so it predicts how the whole team's scene will evolve, including teammates and shared objects. The Brain is then frozen and guides independently adapted single-robot policies, which can be reused or replaced (Sec. 4.5, W2c). The ablations in Table 2 show that these multi-robot parts carry the gain: aligning to current instead of future features lowers the average from 85.5% to 68.0%, and giving the Brain only each robot's own view lowers it to 80.8%.

We thank the reviewer again for the constructive comments, which led to the new controls, the second benchmark, and the broader replacement experiments. We hope these responses address your concerns.

References

[7] GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv 2025.

[8] FLARE: Robot Learning with Implicit World Modeling. CoRL 2025.

[9] Spatial Forcing: Implicit Spatial Representation Alignment for Vision-Language-Action Model. ICLR 2026.

[10] ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Models. arXiv 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.