- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a multi-robot manipulation framework called "Look Ahead Together, Act Locally." The framework decouples shared coordination mechanisms from the process of individual action generation. It introduces a shared, predictive "brain" that processes observations from all robots and generates robot-specific guidance by aligning these inputs with future multi-view 3D features extracted from a frozen foundation model. In six RoboFactory tasks involving 2 to 4 robots, this method achieved an average success rate of 85.5%, outperforming several baselines. The authors also conducted ablation studies—covering "future feature alignment," "multi-view observations," and the "3D teacher model"—which yielded promising results.
Critical strengths
This paper addresses a key challenge in multi-robot manipulation: how to facilitate collaboration through shared information while maintaining the flexibility for individual robot controllers to adapt independently. The proposed architecture decouples a "shared predictive Brain" from "robot-specific Action Experts," offering a viable alternative to both fully decentralized policies and centralized joint-action models. Experiments involving the substitution of local experts (Section 4.5) provide preliminary evidence for the modular nature of this architecture.
The experimental results are compelling and strongly support the concept of "coordination with future awareness." The evaluation covers six collaborative manipulation tasks, which span a range of coordination requirements—from synchronous operations to tasks involving sequential dependencies. As shown in Table 1, the proposed method outperforms the listed baselines across all metrics, achieving an average success rate of 85.5%. Table 2 presents a comparison between "future-state alignment" and "current-frame alignment," with a striking result: omitting "future-state alignment" causes the average success rate to drop to 68.0%.
Critical weaknesses
1. Insufficient control in comparative experiments makes it difficult to attribute the reported performance gains to the proposed coordination mechanism.
In Section 4.1 (lines 317–323), baseline results are cited directly from prior literature rather than being reproduced under a unified training setup. Due to differences in model initialization, training pipelines, and observation/action horizons, it is difficult to determine the extent to which the significant performance improvements in Table 1 are attributable to the proposed "shared prediction coordination" mechanism.
Base on above, i question is "predictive supervision" methods, it raises a critical question: which component is primarily driving the substantial performance gains over existing methods?
Figure 4 presents a comparison of execution times for the "No Brain" setting but lacks corresponding success rate evaluations across the six tasks under matched training conditions.
2. Ablation experiments fail to adequately justify the necessity of several key methodological choices.
The proposed framework integrates future feature prediction, multi-depth guidance, and 3D representation alignment. Consequently, clearly identifying the specific contributions of this component combination is crucial for evaluating the method's innovation. First, the alignment ablation experiment introduces a confounding variable into the training pipeline. In Section 4.4 (lines 424–426), the "w/o alignment" variant jointly trains the "Brain" and "Action Experts" using only the action loss; in contrast, the full method trains and freezes the "Brain" separately prior to expert adaptation. Therefore, the drastic performance drop from 85.5% to 9.8% cannot be attributed solely to the removal of feature alignment; it may also reflect differences in optimization stability or the benefits of staged training.
Questions for authors
Q1. Contribution of the shared "Brain." Could the authors report the success rates of "Action Experts" trained using the same pre-trained initialization, demonstration data, and comparable adaptation processes, but without guidance from the "Brain"? Does the "No Brain" configuration in Figure 4 correspond to such a baseline?
Q2. Isolating the impact of feature alignment. In Section 4.4, the "w/o alignment" variant also alters the training process by switching to the joint optimization of the "Brain" and the "Action Expert." Could the authors explain why this change, and whether the effect of alignment could be evaluated while keeping the optimization strategy comparable?
Q3. Necessity of multi-depth and geometric supervision. Have the authors evaluated variants that use only single-depth guidance, or schemes involving alignment with non-geometric future visual features?
Q4. Scope of the modularity claim. All six RoboFactory tasks share the same "Brain" checkpoint, or is a separate "Brain" trained for each task?
5 comments
Response to Reviewer gfHK (2/2)
(W2b, Q3) Have the authors evaluated variants that use only single-depth guidance, or schemes involving alignment with non-geometric future visual features?
Thank you for suggesting these comparisons. To test multi-depth guidance, we trained a single-depth variant that aligns and reads guidance only from the last Brain block (block 23) instead of blocks 8, 13, 18, and 23 (Eq. 2, L345), keeping the rest of the pipeline unchanged. This variant reaches 81.2% average success (Table 3).
To address the reviewer’s question about non-geometric future visual features, we replaced the VGGT teacher with future DINOv2 features [1], extracted independently from each robot’s view. This variant reaches 83.8% average success. DINOv2’s frozen features support both semantic segmentation and monocular depth estimation [1, 2], capturing object semantics alongside spatial structure. These complementary cues can provide informative future targets for anticipating changes in objects and robot configurations, helping explain its strong performance.
This variant also retains our shared Brain’s access to all robots’ current views and its future-feature prediction objective. Together with the matched controls in W1a and W2a, these results support our central design of learning a shared predictive Brain to anticipate collaborative scene outcomes and guide individual Experts. We have added both variants and their discussion to Sec. 4.4 (Table 3, originally Table 2) of the revised paper.
| Variant | Avg. success (%) |
|---|---|
| Ours (Table 3) | 86.8 ± 3.0 |
| w/o multi-depth (block 23) | 81.2 |
| Alignment w/ DINOv2 | 83.8 |
(Q4) Do all six RoboFactory tasks share the same Brain checkpoint, or is a separate Brain trained for each task?
Thank you for asking for this clarification. All six RoboFactory tasks share a single Brain checkpoint, trained jointly on their demonstrations with teams of two, three, and four robots, which is why each training batch groups samples with the same number of views (L728–730). Each robot in each task keeps its own Expert (L746–747), while all of them read guidance from this one frozen Brain. The Expert replacement in Sec. 4.5 therefore leaves a Brain shared by all six tasks unchanged, and the same checkpoint also guides the replacement Experts on Two Robots Stack Cube and Place Food (see our response to Reviewer rtaY, W2c). We have clarified in Sec. 4.1 (“Implementation Details”) of the revised paper that all six tasks share a single Brain trained jointly on their demonstrations.
We thank the reviewer again for the detailed questions, which led to the matched controls, the clean alignment control, and the depth and teacher ablations. We hope these responses address your concerns.
Note: Line numbers (L###) refer to the original submission, while section, table, and figure numbers refer to the revised paper, where new content is highlighted in blue.
References
[1] DINOv2: Learning Robust Visual Features without Supervision. TMLR 2024.
[2] Probing the 3D Awareness of Visual Foundation Models. CVPR 2024.
Dear Reviewer,
We sincerely appreciate your time and thoughtful consideration of our work.
As the discussion period is coming to a close, we would be grateful for any additional feedback you may have on our responses. We would be glad to address any remaining questions or concerns before the discussion concludes.
Thank you once again for your valuable time and consideration.
Best regards, Paper 21783 Authors
Response to Reviewer gfHK (1/2)
(W1a, Q1) Baseline results in Table 1 are cited from prior literature rather than obtained under a matched setup with the same pretrained Experts and adaptation process, and it is unclear whether the “No Brain” curve in Fig. 4 corresponds to such a baseline.
Thank you for raising these concerns. In the submitted paper, the baseline scores in Table 1 were cited from prior literature (L322–323). To separate the Brain's contribution from pretraining and the training pipeline, we added the baseline the reviewer describes, Decentralized-DP with the same single-robot pretraining, demonstrations, and Expert hyperparameters as our method (L350–353). We trained it and our method three times each and evaluated every run on 300 episodes per task. This control reaches 62.0%, and adding our Brain raises success to 86.8% (+24.8 points). Without pretraining, the same Decentralized-DP reaches 33.7%, so both pretraining and the Brain contribute substantially. The pretrained VLM and its frozen DINOv2 and SigLIP encoders explain only part of the Brain's gain. With the same VLM and encoders but without alignment training, the Brain reaches 72.8%, and alignment adds a further 14.0 points (see W2a).
Figure 4 compares our method with Decentralized-DP without single-robot pretraining, as stated in the updated figure legend. It therefore illustrates the from-scratch setting, whereas the matched pretrained no-Brain control reaches 62.0% (Table 3). We have updated the figure legend and added the from-scratch results to Table 1 (Sec. 4.2) and the matched pretrained controls to Table 3 (Sec. 4.4) of the revised paper.
(W1b) Removing multi-view inputs in the original Table 2 lowers the average success rate only from 85.5% to 80.8%, raising the question of which component primarily drives the performance gains.
Thank you for raising this question. The gains over existing methods come from both single-robot pretraining and the predictive Brain, and within the Brain they come primarily from predicting the team's future rather than from shared current views at inference. Our matched controls separate these gains into three steps (W1a, W2a). Single-robot pretraining raises Decentralized-DP from 33.7% to 62.0%, and a frozen VLM Brain that sees all robots' views raises it to 72.8%. Training the Brain to predict the team's future then adds a further 14.0 points, reaching 86.8%. In the original Table 2 as well, where all variants share the same 100-episode evaluation, aligning to current instead of future features costs 17.5 points, much more than the 4.7 points from removing shared views.
Removing shared views costs less because the alignment targets in this ablation are unchanged. The VGGT teacher still processes all robots' future views jointly (L450–451), so the Brain still learns team-level information during training and only loses access to other views at inference. This access matters most for larger teams, as success on the four-robot Take Photo task drops from 72% to 60% in the original Table 2. We have revised Sec. 4.4 (“Multi-View versus Single-View Brain Inputs”) of the revised paper to clarify that the teacher still processes all robots’ future views jointly and that this ablation tests the value of shared current views at inference time.
(W2a, Q2) In Sec. 4.4, the “w/o alignment” variant (9.8%) also switches to joint optimization of the Brain and Action Experts; why was this change made, and can alignment be evaluated while keeping the optimization strategy comparable?
Thank you for this insightful observation. The original variant changes the training pipeline as well as removing alignment. Without alignment, the Brain has no Stage-1 objective, so we trained it jointly with the Experts in one stage using only the action loss (L424–426). Training it longer, to 200 epochs, still averages only 5.8%, suggesting that this joint training is hard to optimize. To isolate alignment, we added a control that keeps the full method's two-stage pipeline (Sec. 3.4) and removes only the alignment training, leaving the Brain frozen at its pretrained VLM weights. This control reaches 72.8%, whereas the aligned Brain reaches 86.8% (+14.0 points), showing that alignment training accounts for this gain under the same training pipeline. We have added the frozen-VLM control alongside the original joint-training variant in Table 3 (originally Table 2) and clarified their different training procedures in Sec. 4.4 of the revised paper.