acceptodds
Withdrawn from ICLR 2027

RxBrain: Embodied Foundation Model with Joint Language-Visual Subgoal Planning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce RxBrain, an embodied foundation model that combines embodied understanding and reasoning with joint language-visual subgoal planning. It interleaves textual subgoals with goal images, pairing task decomposition and constraints with intended intermediate and final physical states. Built on vision-language model, RxBrain adopts a Mixture-of-Transformers architecture, with general image generation pretraining followed by interleaved text-image planning training. We introduce an inference-aware visual state construction strategy to enhance joint planning. Our automatic pipeline converts over 50,000 hours of embodied video into 21.51M trainable segments, aligning textual steps with visual state transitions. Experiments demonstrate competitive performance on embodied understanding and reasoning benchmarks, alongside general image generation and joint language-visual subgoal planning capabilities. Its joint subgoals substantially improve downstream policy performance on long-horizon manipulation tasks, while an action-generation model built on RxBrain enables effective continuous action generation on real robots without additional action pretraining.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Unified Multimodal Models, Subgoal Planning, Robotic Manipulation, Interleaved Text-Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_h5PSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RxBrain, a multimodal foundation model aimed at embodied intelligence.

Critical strengths

It unifies embodied understanding, reasoning, image generation, and language-visual joint subgoal planning into a single framework through a Mixture-of-Transformers architecture.

Critical weaknesses
  1. The paper has not fully explained the core technical differences of RxBrain compared to existing multimodal foundation models, embodied visual-language models, VLA models, and world models, and it also needs to further clarify the unique advantages brought by its unified modeling framework.
  2. The method mainly addresses the problem of goal state representation and cannot fully cover embodied planning such as modeling, decision-making, and environmental uncertainty. Therefore, the problem definition and conclusions need to be reconsidered.
  3. Currently, experiments mostly show overall performance improvement of RxBrain, but do not sufficiently prove that the improvement mainly comes from visual subgoal planning itself. There is a lack of in-depth ablation experiments on text-only planning, visual-only planning, and the contributions of different modules.
  4. Currently, the comparative methods are still not comprehensive, lacking fair comparisons with the latest VLA, world models, and stronger embodied foundation models. Hence, it cannot yet be fully verify that it achieves the state-of-the-art in the field.
  5. The paper introduces the model structure and training process mainly from an engineering perspective but lacks deeper theoretical explanations on why language-visual joint subgoal representation can effectively promote long-term planning.
  6. The paper defines RxBrain as an embodied foundation model, but the current method is more focused on the integration of multimodal understanding, visual generation, and subgoal planning, and has not yet clearly explained the essential differences from existing embodied VLMs, world models, and VLA models.
Questions for authors

None.

Reviewer_Y8vbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RxBrain, an embodied foudnation model for robotics with a particular emphasis on joint language-visual subgoal planning.

Essentially, what is new here in the model is that they use a MoE architecture to generate seqs of textual subgoals (which we, as robotics researchers, have understood the limitation of textual subgoals for a long time, but the authors seems not aware of these textual planning limitations).

And they hope to have better performance for long-horizon robotic tasks. Experiments have slightly shown these evidence but with a couple of weak baselines.

Critical strengths
  • This is a very well-written paper, with impressive teamwork and a huge amount of engineering effort (and cost! LOL).

  • The authors collected a large amount of data, with around 50k hours of videos in the training dataset, and conducted large-scale pretraining. Looking at the training configurations in Appendix E, I can imagine how much computational resources, engineering work, and cost went into this project. I really appreciate the authors' effort and commitment to this research direction.

  • What I found particularly interesting is the experiments on data mixtures and training configurations. These experiments are very expensive, and most academic labs simply cannot afford to run them at this scale. I think sharing these findings is quite valuable, as it can help other researchers make better decisions about dataset composition and training strategies, and save a lot of experimentation costs.

  • I am glad to know that joint language–visual subgoal planning can improve downstream manipulation performance, especially for long-horizon planning, although it seems quite obvious for researchers in this community. This is a nice observation, although I would like to see more analysis on why it helps and how much improvement actually comes from this design.

  • My main concern is that the scientific contribution seems rather limited compared to the huge engineering effort. The authors invested substantial resources into data collection, pretraining, and system development, but it is not very clear what new methodological insights we can learn from these experiments. And more importantly, the performance gains are not significantly strong!

  • I enjoyed reading this paper. It is quite an big research project, and I appreciate the amount of work behind it.

  • However, I find the novelty not particularly strong, and the performance is not very convincing either, especially considering the resources invested.

  • Quite an interesting research investment, but I am less convinced by its scientific contribution.

Critical weaknesses

Just follow my concerns in the previous section, here are the more details:

1 very limited scientific novelty and the performance gain is not impressive

(1) My primary concern is the lack of significant scientific novelty, which I consider particularly important for a conference like ICLR.

  • The proposed approach largely combines existing techniques, including the MoE architecture, flow-matching-based image generation, and hierarchical subgoal planning. The overall architecture and training recipe appear to be relatively straightforward combinations of existing techniques.

(2) Even these contribution can be considered as "novel", the results are not convicing though!

  • considering the large scale of data collection, engineering, and computational resources invested, I would expect a more convincing demonstration of long-horizon manipulation capabilities.
  • But the experiments primarily involve nine RMBench tasks, joint-planning trajectories of up to five (correct me if I am wrong) steps in the reported benchmark, and three real-world manipulation tasks which seems quite simple does not show "long" horizon task planning rather than simply 3~5 sub tasks.
  • Therefore, these evaluations do not sufficiently demonstrate that the proposed method can address substantially more challenging long-horizon problems involving complex dependencies, dynamic environments, and unexpected execution failures.

And, the empirical results are not consistently compelling:

  • Limited improvement in real-world action generation: RxBrain achieves 87% average success compared with 82% for π0.5, across only three real-world tasks. This five-percentage-point improvement is relatively modest considering the scale of the proposed model and training effort. It is also unclear whether the difference is robust across a broader range of tasks.
  • On RMBench, several tasks remain challenging, including Blocks Ranking Try (27%), Battery Try (36%), and Press Button (0%). These results raise questions about the if the proposed model can work reliably for any tasks.
  • Overall, these exp results, seem to me, is just saying proving joint text–image subgoals to a downstream policy can improve performance, but does not necessarily establish the pros of the proposed unified planning architecture.

(3) weak baselines comparison

Another major concern is the choice of experimental baselines.

  • The paper aims to address long-horizon manipulation through subgoal planning. However, there are alternative approaches to this problem, particularly modular agentic AI systems that combine pretrained vision-language models, task decomposition, environment feedback, and closed-loop replanning.
  • These systems may achieve similar long-horizon planning capabilities without requiring a dedicated foundation model trained on tens of millions of embodied planning segments.
  • The authors do include Qwen-Agent and Cosmos-3-Nano agent baselines in their joint-planning benchmark, which I appreciate. They do not investigate how such agentic approaches would perform when integrated with the same downstream VLA policy.
  • A more informative experiment would compare RxBrain against a modular agentic planner connected to the same VLA policy.
  • Given the significant resources invested in RxBrain, I believe this comparison is important. Without it, the paper does not convincingly establish whether its proposed unified architecture is a better solution than existing modular approaches.

(4) failed to address more fundamental challenge in long-horizon robotic task planning.

This part is more of my personal thoughts, and does not affect my rating

In robotics research, regarding sub-goal planning, we know the limits of textual sub-goals and image based sub-goals, simply because the assumption that "manipulation tasks can be decomposed into textual subgoals and corresponding visual goal states" does not establish in many more general robotic tasks, for example, thoese involve contact dynamics, force control, partial observability, and continuous interaction, which may not be adequately represented by textual instructions or goal images alone.

This raises several fundamental questions: When are language–visual subgoals sufficient for manipulation? When do they become inadequate? How should the system represent subgoals involving forces, contacts, or partially observable physical states?

For any researcher in this field, should understand this limitations. But I did not find any discussion about this limitation.

I would have appreciated a deeper investigation of these limitations.

(5) it's terrible to read the related works section

I also feel the related-works section is somewhat superficial. The paper mostly summarizes what previous methods do, without sufficiently analyzing how they relate to the proposed contributions or identifying the specific scientific gaps that RxBrain addresses. For example, BagelVLA and π0.7 have already explored closely related ideas involving language-based task decomposition, visual-state prediction, and long-horizon manipulation. Although the authors cite these works, the discussion does not clearly explain what fundamental problem remains unsolved by these approaches or how RxBrain advances beyond them.

(6) lastly, it's still a good work for the research community, I want to provide some encouragements to the authors

This is a well-written paper. I could imaging huge amount of engineering work and compute invested in this project. The community will definitely learn from their exp configuraitons. I particularely appreciate the authors listed the detailed dataset consctruciton, how they mixture these datasets.

Given the high standard of ICLR, I will give more chance to other papers that truely can advance our understanding of these foundation models.

Questions for authors

Pls check some realted questions I raised in the critical weakness section.

Reviewer_C2XyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RxBrain, which combines embodied understanding and reasoning, general image generation, and joint language–visual subgoal planning. The authors construct large-scale joint planning training data and introduce RxBrain-Bench-JointPlan to evaluate models’ joint language–visual planning capabilities, filling an evaluation gap in embodied joint text–image subgoal planning. Experiments further evaluate joint subgoal conditioned control and direct action generation.

Critical strengths
  1. Substantial work and comprehensive implementation. The work covers data construction, model training, and RxBrain-Bench-JointPlan, providing a complete pipeline.

  2. Visually appealing figures and clear, fluent writing. The architecture, data construction pipeline, and experimental results are presented clearly and are easy to follow.

Critical weaknesses
  1. Insufficient discussion and ablation of the model architecture. For a model presented as an embodied foundation model, I would expect more discussion and experimental analysis of its architectural choices. The paper lacks comparisons between shared and separate attention projections and FFNs for visual understanding and generation, leaving the benefits of these choices unclear.
  2. The sources of real-robot performance improvements remain unclear. RxBrain, π0, and π0.5 are fine-tuned on the same real-world robot data, but the paper does not explain whether the preceding training data overlap with the real-robot task data. Consequently, the contributions of data, the base model, model architecture, and training methods cannot be distinguished.
  3. Insufficient real-robot evaluation for an embodied foundation model. Given the large-scale embodied training data and the paper’s positioning as an embodied foundation model, evaluating generalization is necessary. The real-robot experiments cover only three tasks across two robot embodiments, without evaluation on unseen tasks or scenes. Such evaluation is needed to demonstrate whether the large-scale training leads to generalization beyond the tasks and scenes seen during training.
  4. Duplicated references. Fang et al., 2026a and 2026b refer to the same MolmoAct2 paper and should be merged.
Questions for authors
  1. Please see the Weaknesses above.

  2. I will increase/maintain or decrease my score based on whether, and to what extent, the authors address the concerns raised in the Weaknesses.

Reviewer_2GmyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RxBrain, an embodied foundation model that integrates understanding and reasoning, image generation, and joint language–visual planning. An automated annotation pipeline derives planning supervision from over 50,000 hours of embodied video, while inference-aware visual state construction aligns how visual histories are formed during training and inference. Experiments show competitive results across understanding, generation, and planning benchmarks. The model also improves long-horizon manipulation through joint textual and visual subgoals and can be extended to generate actions for real-robot control.

Critical strengths
  1. A well-developed subtask annotation pipeline. The pipeline combines adaptive keyframe sampling, temporal step localization, bidirectional boundary refinement, quality and consistency checks, and L0-L3 supervision at multiple levels of granularity. It converts large-scale embodied video into supervision that aligns textual steps with visual state transitions. The systematic treatment of step boundaries and annotation quality is a valuable data-engineering contribution supporting large-scale joint-planning training.

  2. Substantial success-rate gains in a controlled comparison. On RMBench, average π0.5 success increases from 12.6% to 60.6% with joint text-image subgoals, a gain of 48.0 percentage points, and exceeds the 23.6% achieved with text-only subgoals. Appendix G.1 reports matched training data, initialization and training settings. These results provide strong evidence for the effectiveness of visual subgoal conditioning and the practical feasibility of the approach on the evaluated tasks.

Critical weaknesses
  1. The methodological novelty relative to prior work appears limited. The main contribution lies in adapting and integrating existing multimodal understanding and generation techniques for embodied subgoal planning, while a substantial methodological advance over existing frameworks is not clearly established.

  2. Overstated methodological significance of inference-aware visual state construction. Since RxBrain uses VAE-decoded goal images as visual history at inference, reconstructing training history frames through the same VAE is a natural implementation choice for maintaining train-inference consistency. The ablation in Table 5 adequately supports this choice. My concern is its positioning as a distinct methodological contribution: the procedure addresses an input mismatch introduced by the chosen architecture, and its benefit does not by itself establish a distinctive advantage of the proposed modeling approach. I would regard it as a practical training detail and suggest moderating the novelty claim.

  3. Missing comparison with the HY-Embodied-0.5 initialization. RxBrain inherits its understanding backbone from HY-Embodied-0.5, yet Table 1 does not include the original model. Comparisons with other VLMs establish competitiveness, but do not directly substantiate preservation of the original understanding and reasoning capabilities after generation and joint-planning training. A comparison with the exact initialization checkpoint under the same evaluation protocol is needed to identify improvements, retention or degradation. Its absence limits assessment of capability preservation and the incremental changes over the starting model.

Questions for authors
  1. Please clarify the RxBrain-π0.5 integration on RMBench. Although Appendix G.1 specifies matched training conditions, how are the generated textual subgoals and goal images supplied to the policy? How are they aligned with demonstration action chunks, and how are inconsistencies between generated goals and demonstrated actions handled? Please specify the observations and history available to the planner, along with subgoal completion, switching and replanning rules. The supplementary code omits this integration pipeline. These details would materially affect the reproducibility and interpretation of the main success-rate improvement.

  2. Please identify the models and report the efficiency of the annotation pipeline. Which MLLM/LLM models and versions are used for annotation, boundary refinement, verification and hierarchical label construction? Please provide the key prompts and inference settings, together with annotation cost, computational or API resource usage, and processing throughput. Appendices A and C describe the procedures, but the supplementary materials omit the data-pipeline implementation. These details are important for assessing the reproducibility and practical scalability of the central data contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.