acceptodds
Under review as a conference paper at ICLR 2027

MM-ABC: Towards Generalist Mobile Manipulation via Seeing, Coordinating and Imagining

Abstract

Mobile manipulation lets a robot control the extent of its own workspace, but it couples two challenges that fixed-base policies can largely avoid: perception must remain spatially grounded under continuous ego-motion, and heterogeneous arm and base actions must be coordinated. We present Goku, a vision-language-action model built around seeing, coordinating, and imagining arm-base collaboration. Goku conditions its action expert on sparse multilevel VLM features, supervises the shared perceptual representation with a training-only branch that predicts future features from a frozen geometric teacher, and generates manipulation and body actions with MM-APT, a two-stream transformer that couples the streams through masked near-far joint attention and directly predicts clean action chunks. A toy experiment motivates clean-action prediction for coupled action streams. In the full policy, replacing clean-action prediction with velocity prediction reduces success on RoboCasa365 composite-seen tasks from 32.8% to 29.2%. We pretrain Goku on more than 5,000 hours of robot data spanning over 400K episodes, 12 datasets, and 17 embodiments, including MM-30, a new multi-view dual-arm mobile manipulation dataset. Goku achieves 44.71% success on EBench, 61.2% on RoboCasa365, the highest mean success across three ManiSkill-HAB suites, 99.1% on LIBERO, 82.8% on LIBERO-Plus without perturbation training, and 83% on five long-horizon real-world tasks, 12 percentage points above pi_0.5.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Mobile Manipulation, Spatial Understanding, Arm-Base Collaboration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TwLfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dedicated policy architecture for loco-manipulation with bimanual arm and a moving base. The policy has careful attention mechanism design, and VGGT feature alignment learning for spatial understanding. The policy is verified on a variety of benchmarks such as LIBERO, robocasa, ebench, libero-plus and real world tasks.

Critical strengths

Even though similar technical concepts such as x-prediction, attention masking for different experts, feature alignments, have been explored by previous work, this work demonstrate sufficient technical strength with a policy that outperforms many well-established sota.
The supplementary video shows very cool robot rollouts and reinforce my confidence.

Critical weaknesses

The x-prediction analysis on figure 2 is not clear to me at the plots, axis labels, the caption text and paragraph as well. I am not convinced why x-pred can be better than v-pred in this case after reading.

The attention mask design (at figure 1) is very interesting, especially the separation for steps 1-32 and 33-64. I believe this corresponds to the initial motivation that "manipulation over a workspace that the policy itself reconfigures". I would appreciate more in-depth explanation about this design intuition and how the two sub chunks of both loco and manips are used in deployment, and some ablation studies on this if possible.

Questions for authors

See Weakness

1 comment

We thank the reviewer for the encouraging assessment and helpful questions. We have clarified Figure 2 and Sec. 3.5, expanded the explanation of near–far attention, and added corresponding ablations (Table 5; Appendix H).

W1 / Q1: Why can clean-action prediction outperform velocity prediction?

The intuition is that predicting a clean action and predicting how to remove noise can have different learning difficulties, despite being mathematically equivalent for an ideal predictor.

Valid arm–base trajectories are highly structured, even in a high-dimensional action space. An x-head directly predicts this clean structure, whereas a v-head predicts v=x−ϵv=x-\epsilon, which also contains random noise across all action dimensions. Recovering the same clean action therefore requires the v-head to represent and cancel these noise components. This can be harder for our finite-capacity decoder, which has no direct noisy-input skip connection.

To examine this, we constructed a synthetic two-stream task with a known low-dimensional action structure and Bayes-optimal denoiser. Both variants share the same architecture, training loss, and sampling procedure. Figure 2 measures four aspects, with detailed definitions in Appendix C:

  • (a) Target geometry: Singular-value decomposition of centered action chunks shows that clean actions occupy a rank-13 subspace, while velocity targets contain full-dimensional noise.
  • (b) Endpoint variance: For each fixed clean chunk, we generate 24 independently noised inputs and measure the variance of their predicted clean endpoints, normalized by clean-chunk variance. At t=0.01t=0.01, x-prediction achieves 0.023 versus 0.54 for v-prediction (23× lower).
  • (c) Excess risk: We subtract the Bayes-optimal conditional mean's MSE from each model's endpoint MSE to isolate approximation error. The v/x excess-risk ratio reaches 1.64 at t=0.01t=0.01.
  • (d) Task allocation: We measure the distance between generated arm–body contributions and the nearest valid task allocation. At five Euler steps, the errors are 0.0090 (x) versus 0.0106 (v).

These findings suggest that x-prediction is easier to approximate in our setting, particularly near the high-noise start of sampling. The full-policy experiments support this trend: x-prediction improves RoboCasa365 success from 29.2% to 32.8% from scratch and 57.8% to 61.2% after full pretraining and post-training (Tables 4–5).

W2 / Q2: Motivation and deployment of near–far attention.

We aim for one model to learn both immediate actions and how they develop into a longer plan. For example, later reaching actions should account for earlier base repositioning. During 64-step pretraining, the far segment (steps 33–64) attends to the near segment (steps 1–32), allowing future predictions to build on earlier planned actions and provide additional supervision to near-action representations.

Reviewer_onbiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MM-ABC, a vision-language-action model for robotic mobile manipulation that combines multilevel vision-language features, auxiliary future geometric representations supervision, and near-far joint attention for coordinated arm-base action generation. This approach integrates robot demonstrations across embodiments and introduces MM-30, an additional mobile-manipulation dataset. Experiments on five simulation benchmarks and five real-world tasks demonstrate strong performance of the method.

Critical strengths
  1. This paper explicitly addresses coordination between the mobile base and the manipulator. The proposed near-far joint attention mechanism offers a well-motivated approach to coordinating whole-body action generation.

  2. MM-30 contributes a valuable data resource for training mobile-manipulation policies, particularly given the relative scarcity of datasets containing coordinated base and arm motions.

  3. This paper compares velocity-field prediction and clean-action prediction under a common training objective. The controlled synthetic experiment provides useful insights into the choice of action output parameterization.

  4. Experiments span multiple simulation benchmarks and real-world mobile-manipulation tasks, providing strong support for the effectiveness of the method.

Critical weaknesses
  1. The reported experiments do not sufficiently isolate the contribution of the proposed coordination mechanism. In particular, the benefits of the dual-branch architecture are not disentangled from those of near-far joint attention. Consequently, it remains unclear which architectural choices account for the observed performance gains.

  2. Although Table 4 provides useful component ablations under a common training configuration, these experiments omit large-scale robot-data pretraining. These findings therefore do not directly establish whether the same components provide comparable benefits under the pretraining and post-training pipeline used for the main results.

Questions for authors
  1. Could the authors disentangle the contributions of the dual-branch architecture and near-far joint attention?

  2. Table 4 evaluates individual components without large-scale robot-data pretraining, whereas the main results are obtained through post-training pretrained models. Could the authors provide evidence that the relative benefits of these components persist under the full pretraining and post-training pipeline?

  3. Could the authors clarify the complete data-mixing configuration and sampling procedures at each training stage, including the resulting dataset-level sampling probabilities and any changes between pretraining and post-training?

1 comment

We thank the reviewer for the helpful suggestions. We have added coordination ablations, extended the component evaluation to the full training pipeline, and clarified the data-mixing configuration.

W1 / Q1: Dual-stream coordination and near–far attention.

Arm and base actions serve different but interdependent roles, motivating separate streams with explicit communication. On 16 RoboCasa365 composite-seen tasks, success improves from 21.0% (single stream) to 27.0% (two streams without communication) and 32.8% (communicating streams), demonstrating the benefits of stream specialization and information exchange (Table 5, left).

Near–far attention further allows one model to learn near-term actions and their longer-term continuation. The far segment reads earlier planned actions, providing additional supervision to near-action representations, while near actions remain independent of the future tail. With both models predicting 32 steps and executing the first 16, near–far achieves 34.3% versus 31.0% with unrestricted attention (Table 5; Appendix H).

W2 / Q2: Component effectiveness after full pretraining.

We repeated the key component ablations using the complete 5,000+-hour pretraining corpus, followed by RoboCasa365 post-training under the same protocol as the main experiments (Table 5, right).

ConfigurationAverage SRDifference
MM-ABC61.2%—
Without future supervision51.5%−9.7
Final-layer-only conditioning54.2%−7.0
Velocity prediction57.8%−3.4

The improvements remain consistent with the from-scratch results in Table 4. In particular, future-geometry supervision improves composite-unseen success from 38.5% to 50.3%, supporting its effectiveness even though the auxiliary branch is removed during inference.

Q3: Data mixing and sampling probabilities across training stages.

We have clarified the data-mixing configuration and the differences between pretraining and post-training in the revised paper.

During pretraining, we use 12 datasets totaling 5,166.2 cleaned hours. Figure 6 directly presents the actual dataset-level sampling probabilities, including ABC-130k (19.94%), AgiBotWorld 2026 (15.97%), InternData-A1 (11.73%), and MM-30 (1.49%), with all 12 datasets labeled. Figure 7(a) separately reports their cleaned data volumes and duration shares. Section 4 and Appendix D describe the dataset composition and sampling procedure, while Appendix E details the data cleaning pipeline.

During post-training, we use only the demonstrations from each target benchmark or task, without mixing in the pretraining datasets. The corresponding training steps, batch sizes, and action horizons are provided in Section 5.1 and Appendix B (Table 6).

These details clarify both the pretraining mixture and its transition to benchmark-specific post-training.

Reviewer_1VcQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents MM-ABC, a vision–language–action model for mobile manipulation. The method combines sparse multilevel VLM features, separate manipulation and body action streams connected through asymmetric near–far attention, and a training-only objective that predicts future geometric features from a frozen teacher.

Critical strengths
  1. A coherent design for mobile manipulation. The paper explicitly considers the coupling between base motion, reachability, and manipulation. Separate action streams, shared perceptual context, and temporal visibility constraints form a reasonable architecture for this setting. Removing the future-prediction branch at deployment also makes the auxiliary supervision practically attractive.

  2. Broad empirical evaluation in sim. The evaluation spans mobile manipulation, fixed-base manipulation, visual perturbations, and physical execution.

Critical weaknesses
  1. The central coordination architecture is not directly ablated. MM-APT’s two-stream structure and asymmetric near–far attention are central to the proposed method, but Table 4 varies only the conditioning interface, future supervision, and prediction target. It does not compare against a single-stream model, two streams without cross-stream communication, or two streams with unrestricted attention.

  2. Limited real-world evaluation. The five tasks each use 200 training demonstrations, but the paper does not clarify what objects, layouts, or scenes are held out, making generalization difficult to assess. It also lacks a breakdown of failure causes and an evaluation of autonomous recovery. Can the robot recover from missed grasps or displaced objects without human intervention?

Questions for authors

See weakness 1&2

  1. Please clarify whether the 5 real-world tasks, object instances, or physical scenes are included in pretraining data MM-30. If yes, the comparison is unfair, new tasks without seen object or scenes during pretrain should be tested.
3 comments

We thank the reviewer for the constructive feedback. We have added direct coordination ablations (Sec. 5.4, Table 5; Appendix H) and expanded the real-world evaluation protocol and failure analysis (Sec. 5.5; Appendix I).

W1 / Q1: Contribution of the coordination architecture.

Arm and base actions have distinct control roles but must coordinate closely, motivating separate streams with cross-stream communication. For near–far attention, our intuition is to train one model to generate near-term actions while learning how they extend into longer trajectories. Far-action predictions build on the near plan and provide additional supervision to its representations, while the near segment remains independent of potentially unexecuted future actions. This also facilitates adaptation to different downstream action horizons.

We added five ablations on 16 RoboCasa365 composite-seen tasks under the same training budget (Table 5):

ConfigurationDescriptionSR
Single streamOne shared head for arm and body actions21.0%
No communicationSeparate streams without cross-stream attention27.0%
Flat 16-stepCommunicating streams, predicting 16 steps32.8%
Full attentionUnrestricted attention, predicting 32 and executing 16 steps31.0%
Near–farFar reads near, predicting 32 and executing 16 steps34.3%

Cross-stream communication improves success by 5.8 points, while near–far attention outperforms unrestricted attention by 3.3 points under the same prediction and execution horizons. These results support the benefits of both explicit arm–base coordination and structured temporal attention.

W2 / Q2–Q3: Real-world generalization and recovery.

None of the five evaluation tasks, exact object instances, or physical rooms appears in MM-30 or other pretraining sources. All methods are subsequently fine-tuned on the same 200 demonstrations per task and evaluated on matched layouts.

All five tasks include initial-pose variations, typically 10–20 cm, while retaining the same rooms and objects used during fine-tuning. For example, the cake in Birthday Party Setup is placed outside its demonstrated position range, with an additional 20–30° candle rotation. Similar positional variations are applied across the other tasks.

We also added a detailed failure analysis in Appendix I. Failures primarily involve unstable grasps, handovers, and precise placements. MM-ABC can sometimes recover missed candle or small-part grasps autonomously within the same rollout, although severe failures such as dropped cakes or tipped bins remain challenging.

Overall, MM-ABC achieves 83% mean success across five long-horizon tasks, compared with 71% for π0.5 and 54% for StarVLA-GR00T (Fig. 5).

Thank you for the additional experiments and clarifications. The new coordination ablations and clarifications address my main concern about the architecture and experiment results. The comparison between near–far and unrestricted attention under matched prediction and execution horizons is helpful. Overall, I will raise my score to positive.

Thank you for taking the time to review our revisions and additional experiments. We appreciate your positive feedback and your willingness to reconsider the rating. We are glad that the new results and clarifications have addressed your main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.