acceptodds
Under review as a conference paper at ICLR 2027

UniW(A)M: A Unified Mixture-of-Experts Framework for World and World-Action Models

Abstract

World models (WMs) and world-action models (WAMs) focus on future world prediction and executable action generation, respectively. Although these models learn shared physical dynamics and interaction semantics from embodied trajectories, they are typically trained in isolation, which restricts knowledge reuse between prediction and control. The central challenge is therefore to develop a unified learning framework that shares these dynamics while preserving the distinct modeling capabilities required by each paradigm. We propose UniW(A)M, a mixture-of-experts world-(action) model built upon diffusion Transformers that unifies action-conditioned world modeling and diverse WAM variants. Shared experts capture common dynamics, while atomic capability experts are reused across interfaces so that each interface keeps its own conditional pathway. At deployment, a plug-in adaptive interface router (PAIR) learns from a small subset of target-task chunk scenarios to select a suitable WAM interface for each execution chunk, balancing task success and execution efficiency while keeping the backbone frozen. We evaluate UniW(A)M on LIBERO, RoboTwin 2.0, and three real-world manipulation tasks, where it achieves 88.3% average real-world success versus 76.7% for the strongest baseline. On WorldArena, it leads action-conditioned world modeling on all three tasks. AC-WM co-training contributes to action generation on every action interface and PAIR raises OOD success from 63.3% to 76.7%. Code is available at https://anonymous.4open.science/r/UniW-A-M-B8C3/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, World Action Model, Mixture-of-Experts, Unifed Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HSQKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniW(A)M unifies action-conditioned world modeling and four world-action interfaces through a mixture-of-expert structure with shared experts, capability-specific experts, and attention masks. PAIR selects an interface per action chunk while keeping the backbone frozen.

Critical strengths

This submission has a clear motivation for sharing knowledge between world prediction and action generation while supporting different inference costs. Its evaluation covers simulation and real robots, with strong RoboTwin C2R performance and promising real-world results. In experiments, AC-WM ablations consistently support positive transfer to all four action-generation interfaces.

Critical weaknesses
  • The conceptual advance over existing unified models remains limited. Prior work already combines world prediction, action generation, and expert specialization. UniW(A)M offers a useful integration, but the paper does not sufficiently establish a distinct modeling principle or capability beyond these existing directions.
  • The broader value of unifying multiple interfaces is not convincingly demonstrated. AC-WM co-training provides evidence of positive transfer to action generation. However, this does not establish why jointly maintaining four action interfaces is preferable to a simpler unified model or a well-chosen single interface.
  • PAIR’s contribution to generalizable embodied control remains narrow. Its gains rely on target-task supervision, with validation focused on variations within three manipulation tasks. These results support task-specific adaptation, but leave the broader scaling issue of learned interface/experts unverified (Do more experts delivers better compositional generalization?).
Questions for authors

How does PAIR compare with simple phase-based routing or lightweight policy fine-tuning using identical OOD supervision?

3 comments
AuthorsOpenReview ↗

Response (1/2): Novelty and Cross-Interface Learning

W1. Conceptual novelty

We thank the reviewer for encouraging us to further clarify the novelty of UniW(A)M.

UWM couples video and action diffusion with flexible conditional inference, Motus integrates understanding, video generation, and action experts, and Worldscape-MoE explores shared and specialized experts for heterogeneous world modeling. UniW(A)M organizes video–action conditional prediction into reusable atomic capabilities, allowing different interfaces to share these capabilities while preserving their distinct conditional dependencies.

Specifically, the atomic experts are organized around prediction roles rather than assigned independently to individual interfaces. VGM serves video tokens across Joint, IDM, and Action-Only, while IDM serves action tokens in both Joint and IDM. Thus, the interfaces are not merely separate predictors collected within one network: they reuse capability-specific computation, enabling supervision from different interfaces to improve common prediction capabilities.

Crucially, expert reuse does not imply identical conditioning information. Attention masks specify what information each interface can access, whereas CATR determines which capability-aligned experts its token groups can use and learns their mixture with shared computation. Masks preserve interface-specific conditional dependencies, while CATR implements cross-interface capability reuse and specialized computation within a single backbone. CATR therefore operationalizes the proposed capability decomposition.

PAIR further connects this shared learning framework to adaptive deployment. It learns a lightweight router for target-domain adaptation, selecting an interface for each action chunk while keeping the unified backbone and individual action predictors frozen. This enables context-dependent use of the jointly learned capabilities.

Our controlled routing and expert-removal ablations support the functional value of this organization, while the cross-interface learning results discussed in W2 address the benefits of jointly maintaining multiple interfaces. We thank the reviewers for their suggestions, which have helped clarify our core contributions.

W2. Benefits of multiple interfaces

We thank the reviewer for raising this important question. The benefits of jointly maintaining multiple WAM interfaces can be understood from two complementary perspectives: cross-interface knowledge transfer during training and adaptive interface selection during inference.

(1) Cross-interface knowledge transfer. We conduct progressive co-training ablations on Folding Clothes to investigate whether jointly training multiple interfaces provides additional benefits. Each configuration is independently trained for 150K total steps from the same initialization and training dataset. Every reported interface is evaluated on 20 matched test scenarios.

Table 1. Progressive interface co-training on Folding Clothes (success rate, %).

Training ConfigurationJointIDMAction-CentricAction-OnlyAverage
Joint50.0———50.0
+ AC-WM60.0———60.0
+ IDM75.060.0——67.5
+ Action-Centric75.085.060.0—73.3
+ Action-Only90.090.085.085.087.5

As additional interfaces are incorporated into training, Joint improves from 60.0% to 90.0%, IDM from 60.0% to 90.0%, and Action-Centric from 60.0% to 85.0%. These improvements demonstrate the additional benefits of multi-interface co-training under a fixed total training budget. Importantly, incorporating new interfaces improves the performance of existing interfaces, providing evidence of positive cross-interface knowledge transfer beyond AC-WM co-training alone.

(2) Adaptive interface selection during inference. Jointly maintaining multiple WAM interfaces also naturally enables our proposed PAIR mechanism, which adaptively selects the most suitable interface for each action chunk without requiring separately deployed models. By training only a lightweight router while keeping the unified backbone frozen, PAIR improves average OOD success from 63.3% to 76.7% across three manipulation tasks under target-domain adaptation. This demonstrates the practical benefit of maintaining complementary interfaces that can be adaptively utilized according to the current execution context.

Together, these results support the value of our multi-interface design in both training and inference: joint training promotes knowledge transfer across interfaces, while PAIR enables adaptive utilization of their complementary capabilities.

AuthorsOpenReview ↗

Response (2/2): Generalization, Heuristic Routing, and LoRA

W3. Generalizability and expert scalability

We thank the reviewer for raising this important concern regarding generalizability and expert scalability. Our current OOD evaluation focuses on variations within three specific manipulation tasks, where PAIR improves average success from 63.3% to 76.7% through supervised target-domain adaptation while keeping the WAM backbone frozen. These results demonstrate the reuse of pretrained capabilities under task-specific distribution shifts.

Regarding expert scalability, our four atomic experts are designed according to the distinct video–action conditional prediction roles described in Section 3.1 and Appendix A.2, rather than simply increasing the number of experts. Our motivation is to organize complementary prediction capabilities for reuse across interfaces, rather than assuming that adding more experts necessarily improves generalization.

More broadly, we believe stronger cross-task and compositional generalization would benefit from substantially larger and more diverse embodied training data. Our current real-world training dataset is relatively limited in scale, containing only three manipulation tasks:

Table 2. Real-world training dataset.

TaskEpisodesFramesDuration (h)
Tabletop Cleaning2,4261,528,83114.16
Folding Box1,2171,768,22016.37
Folding Clothes6053,110,73228.80
Total4,2486,407,78359.33

Despite this task-focused data scale, UniW(A)M achieves an average success rate of 88.3% across the three real-world tasks, compared with 76.7% for the baseline, benefiting from pretrained initialization and unified world–action learning. We consider achieving strong manipulation performance with 59.33 hours of task-specific real-world data an encouraging result. While broader generalization remains an important direction, our findings highlight the effectiveness and data efficiency of jointly learning reusable world–action capabilities in the evaluated setting.

Q1. Heuristic routing and LoRA comparisons

Following your suggestion, we compare PAIR with phase-based heuristic routing and four interface-specific LoRA baselines on the Folding Clothes OOD task.

The heuristic follows the fixed mapping in Appendix B.3: Reach → Action-Only, Contact → Joint, and Core → IDM. Qwen3-VL-8B identifies the current phase based on robot observations and subtask information. In contrast, PAIR learns context-dependent interface selection for each action chunk without relying on predefined phase-to-interface rules.

We independently fine-tune all four WAM interfaces using the same 2,000 OOD training chunks as PAIR, without dynamic routing. Each LoRA adapter uses rank 16, alpha 32, and Kaiming initialization (init_lora_weights="kaiming"), and is injected into the Attention Q/K/V/O projections and FFN layers. Each adapter is trained for 10K optimization steps. The four adapted interfaces and PAIR are evaluated over 20 OOD trials each under the same evaluation protocol.

Table 3. OOD adaptation on Folding Clothes (success rate, %).

MethodBefore LoRAAfter LoRA
Best Fixed Interface65.0—
Phase-based Heuristic45.0—
Joint-WAM65.065.0
IDM-WAM65.065.0
Action-Centric-WAM50.055.0
Action-Only-WAM50.050.0
PAIR (Ours)75.0—

Under the same target-domain supervision, LoRA fine-tuning leaves the success rates of Joint, IDM, and Action-Only unchanged, while improving Action-Centric from 50.0% to 55.0%. PAIR achieves 75.0%, compared with 65.0% for both the best fixed interface and the best LoRA-adapted interface, and 45.0% for heuristic routing.

The heuristic results suggest that phase-level preferences can capture general trends but may not provide optimal interface assignments for individual action chunks. In contrast, PAIR selects interfaces based on the current execution context. Its observed 10-percentage-point advantage over the best LoRA baseline supports adapting interface selection as an alternative to fine-tuning individual action predictors under limited target-domain supervision, while keeping the pretrained backbone frozen in this evaluation.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer HSQK,

Thank you again for your thoughtful review. In our responses above, we have clarified the conceptual novelty of UniW(A)M, added progressive co-training ablations showing the value of multiple interfaces, and compared PAIR with phase-based routing and LoRA fine-tuning under identical OOD supervision.

As the discussion period is drawing to a close, we would greatly appreciate your feedback. If our responses have resolved your concerns, we would be sincerely grateful if you could reconsider your score in light of the new results. If any point remains unclear, we would be glad to clarify further.

Best regards,

The Authors

Reviewer_gD5hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents UniW(A)M, motivated by the potential for action generation and future-world prediction to support each other through joint learning. It unifies action-conditioned world modeling and four action-generation interfaces within a diffusion Transformer, using a shared expert and four capability-specific experts to balance shared dynamics knowledge with interface-specific specialization. Capability-Aligned Token Routing (CATR) assigns token groups to appropriate experts. The paper also introduces PAIR for few-shot adaptation of interface selection while keeping the backbone frozen. Experiments on LIBERO, RoboTwin 2.0, real-world manipulation tasks, and WorldArena evaluate action generation, world prediction, and OOD adaptation, with ablations supporting the benefits of AC-WM co-training and CATR.

Critical strengths
  1. Well-motivated architecture. The paper is motivated by the complementary dynamics knowledge shared between action generation and future-world prediction, alongside the need to preserve their distinct conditional dependencies. Its shared and capability-specific experts provide a reasonable balance between knowledge sharing and specialization, which may encourage richer representations. Ablations support the effectiveness of capability-aligned routing and AC-WM co-training.

  2. Few-shot adaptation to OOD scenarios. PAIR enables few-shot adaptation of interface selection while keeping the backbone frozen, improving performance in OOD environments. These results suggest that the backbone retains reusable representations and relevant world knowledge that can be accessed through adaptive interface selection.

  3. Strong baseline comparisons. The evaluation includes strong VLA, WAM, and action-conditioned world-model baselines across simulation, real-world control, and future-world prediction, providing a broad assessment of the proposed framework.

Critical weaknesses

1. Insufficient isolation of the central architectural contribution. The ablations demonstrate benefits from AC-WM co-training and the overall CATR design, but do not establish the necessity of jointly training four WAM interfaces or the proposed capability-specific expert decomposition. There are no controlled interface-removal or expert-removal/merging ablations to determine which components provide complementary knowledge. Moreover, the dense baseline uses a different backbone and initialization, complicating attribution of the large performance gap to the proposed architecture. Consequently, the evidence does not sufficiently distinguish the benefits of this particular decomposition from those of a more general shared multi-task model, weakening the central contribution.

Questions for authors

refer to weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer: Architectural Ablations

We thank the reviewer for these comments. Our controlled experiments demonstrate the advantages of capability-decomposed co-training: improved individual interfaces through cross-interface transfer, complementary contributions from world-prediction and action experts, and higher performance than Dense under matched backbone and initialization. We establish these points through interface- and expert-removal ablations, followed by a matched Dense comparison, and will include these results in the revised manuscript.

(1) Co-training and complementary knowledge. To address your concern about the need for multiple WAM interfaces, we conduct controlled interface-removal ablations on Folding Clothes, comparing subsets of jointly trained interfaces with the same initialization, data, and total training budget (Table 1).

Table 1. Progressive co-training on Folding Clothes (success rate, %). Each configuration is independently trained for 150K total steps and evaluated on 20 matched scenarios per interface. Average covers the available WAM interfaces.

Training ConfigurationJointIDMAction-CentricAction-OnlyAverage
Joint50.0———50.0
+ AC-WM60.0———60.0
+ IDM75.060.0——67.5
+ Action-Centric75.085.060.0—73.3
+ Action-Only90.090.085.085.087.5

As WAM interfaces are added, Joint and IDM each improve from 60.0% to 90.0%, and Action-Centric improves from 60.0% to 85.0%. With AC-WM already present and the total training budget fixed, these gains show that adding WAM supervision improves the existing interfaces themselves, supporting cross-interface knowledge transfer.

Having established the benefit of joint training at the interface level, we next examine the contributions of the underlying world-prediction and action capabilities. We individually remove the corresponding atomic experts (Table 2); both intact and ablated variants start from the same trained Folding Clothes checkpoint and continue for 20K steps with identical data and optimization settings.

Table 2. Expert removal with matched continued training. WAM rows report success rate (%); AC-WM reports EWMScore-P. Δ Drop is the absolute difference in the corresponding metric.

InterfaceRemoved ExpertFull (Continued)w/o ExpertΔ Drop
JointVGM90.070.020.0
JointIDM90.020.070.0
IDMVGM95.050.045.0
IDMIDM95.035.060.0
Action-CentricWM85.055.030.0
Action-CentricVLA85.030.055.0
Action-OnlyVGM85.060.025.0
Action-OnlyVLA85.025.060.0
AC-WMWM61.7431.2030.54

Removing action experts (IDM/VLA) reduces WAM success by 55–70 percentage points, while removing world-prediction experts (VGM/WM) causes 20–45-point drops. Removing WM also lowers AC-WM's EWMScore-P from 61.74 to 31.20. Thus, world-prediction experts contribute to action generation as well as video prediction, alongside the substantial contributions of action experts. Together, Tables 1 and 2 support our co-training rationale: additional interfaces improve existing predictors, while the underlying world-prediction and action experts provide complementary capabilities.

(2) Dense baseline with matched backbone and initialization. The preceding ablations support the benefits of complementary learning within UniW(A)M. We next test whether its advantage over dense co-training persists after aligning the backbone and initialization. The original Wan2.1-14B Dense baseline averages only 7.5% success on Folding Clothes; we supplement it with a Dense baseline using the same Wan2.2-TI2V-5B pretrained checkpoint as UniW(A)M, with matched training data, optimization settings, and total training budget (Table 3).

Table 3. Architecture comparison on Folding Clothes (success rate, %). Average is the mean across the four interfaces. The new 5B Dense baseline uses eight evaluation trials per interface.

ArchitectureJointIDMAction-CentricAction-OnlyAverage
Dense (Wan2.1-14B)10.020.00.00.07.5
Dense (Wan2.2-5B, New)12.50.00.00.03.1
UniW(A)M (Full)90.090.085.085.087.5

The new 5B Dense baseline exhibits the same joint-learning difficulty: three interfaces achieve 0% success, and its average is 3.1%, lower overall than the original 14B baseline. The performance gap therefore persists after aligning the backbone and pretrained initialization source.

Compared with dense co-training, UniW(A)M reuses common knowledge through shared representations while preserving paradigm-specific computation through atomic experts and CATR, enabling more effective joint learning of different video–action dependencies.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer gD5h,

Thank you again for your positive assessment. In our response above, we addressed your concern about isolating the architectural contribution with progressive interface co-training, expert-removal ablations with matched continued training, and a Dense baseline with matched Wan2.2-TI2V-5B initialization.

As the discussion period is drawing to a close, we would greatly appreciate your feedback. If you find that these results address your concern, we would be sincerely grateful if you could share your view in the discussion with the other reviewers.

Best regards,

The Authors

Reviewer_4eaMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper proposes UniW(A)M, which co-trains an action-conditioned world model (AC-WM) and four world-action model interfaces (Joint, IDM, Action-Centric, Action-Only) on a single diffusion transformer backbone, distinguishing them through video-action visibility masks. The FFN is replaced by an MoE consisting of one shared expert and four atomic experts (VGM/VLA/WM/IDM), and CATR learns mixing weights within the expert set assigned to each token group. At deployment, the backbone is frozen and PAIR is trained on target-domain data to select a WAM interface for each chunk. Results are reported on three real-world tasks (88.3% vs. 76.7%), LIBERO (97.5%), RoboTwin 2.0 (C2R 66.6%), WorldArena (leading EWMScore-P on three tasks), ablations on co-training and routing, and OOD results with PAIR (76.7% vs. 63.3%).

Critical strengths

S1. The evaluation is broad, and the evaluation protocol and measurement uncertainty are reported in detail. Three real-robot tasks are each run for 20 trials, LIBERO, RoboTwin 2.0, and WorldArena are also evaluated, and the main ablations are conducted on the same real-world tasks. Per-task Wilson intervals (Tables 8–9), the range of latency measurements (B.2, C), and the size of the PAIR training data (B.1) are stated explicitly, so the measurement conditions and the uncertainty of the results can be assessed.

S2. The controlled ablations demonstrating the effect of CATR and of co-training with the AC-WM are convincing. CATR and sample-level routing share the same number of experts, architecture, and parameters, and the success rates of Action-Centric and Action-Only improve from 10% to 81.7% and 80.0%, respectively. While this is not solely an effect of the pooling granularity, it shows the validity of the CATR design. The comparison with and without the AC-WM also controls for initialization, data, and training budget, and all four interfaces improve even though fewer updates are allocated to each WAM.

Critical weaknesses

W1: Weak motivation for the necessity of unified training. The Abstract and Related Work claim that training the AC-WM and WAMs independently limits knowledge reuse and data efficiency, and that capabilities are not shared across WAM interfaces. However, concrete evidence and citations are lacking on why these limitations are problematic and why co-training is necessary. The Introduction also derives the value of shared modeling and modular specialization from Motus, UWM, and Worldscape-MoE, but does not sufficiently argue for the necessity. The trade-off between fast action generation and visual prediction is a reason to use the interfaces selectively, not a reason to co-train them on a single backbone.

W2: Overinterpretation of the OOD results. The claim that “PAIR raises OOD success” in the Abstract, Section 1, and Section 4.3 should be qualified as a result of adaptation on target OOD-domain data. The phrase “small subset of target-task chunk scenarios” in the Abstract also fails to convey the actual data scale. According to B.1, 2,000 chunks were extracted from 500 OOD episodes per task, and the 0.0274% figure is a sampling ratio relative to the overlapping window candidates.

Questions for authors

Q1. Can router weights, expert utilization, or swap/removal ablations be provided to show the functional specialization of the atomic capability experts?

Q2. What are the co-training dataset, the number of episodes, the robot types, and the data scale for each real-world task? What is the source of the checkpoint used for each benchmark?

Q3. How would Table 15 change if Dense were initialized identically from Wan2.2-TI2V-5B?

5 comments
AuthorsOpenReview ↗

Response (1/3): Co-training motivation and OOD adaptation

W1. Motivation for unified co-training

We thank the reviewer for the opportunity to elaborate on the motivation for unified training. Below, we explain its rationale and practical benefits from two perspectives: developments in unified world–action modeling and evidence from our controlled experiments. We will also expand the discussion of this motivation in the main text.

(1) Motivation

Our motivation for co-training is to make complementary supervision from the same trajectories transferable across interfaces: AC-WM learns the visual consequences of actions, while WAMs learn action generation under different video–action dependencies. Independent training confines these signals to separate parameter sets; co-training lets them jointly improve shared representations and reusable capability experts, while visibility masks preserve each interface's conditioning requirements. UWM, Motus, and Worldscape-MoE demonstrate the feasibility and promise of improving policy learning or world prediction within unified models through dynamics supervision, heterogeneous pretraining, and shared computation with expert specialization, respectively.

(2) Controlled experimental evidence

Our ablations support the transfer of complementary supervision both from world prediction to action generation and across WAM interfaces. With training data and total optimization steps fixed, adding AC-WM improves average success across three real-world tasks by 15.0, 15.0, 31.7, and 43.3 percentage points for Joint, IDM, Action-Centric, and Action-Only, respectively, despite fewer expected updates per WAM. The latter two interfaces generate no future video at deployment, showing that world-model training benefits action generation itself.

We further evaluate progressive co-training on Folding Clothes (Table 1). Each configuration is trained independently from the same initialization and underlying data for 150K total steps, matching our main model's budget. Each reported interface is evaluated on 20 matched test scenarios.

Table 1. Progressive co-training on Folding Clothes (success rate, %).

Training ConfigurationJointIDMAction-CentricAction-OnlyAverage
Joint50.0———50.0
+ AC-WM60.0———60.0
+ IDM75.060.0——67.5
+ Action-Centric75.085.060.0—73.3
+ Action-Only90.090.085.085.087.5

We base cross-interface comparisons on the same interface: Joint improves from 50.0% to 60.0% after adding AC-WM and then to 90.0% as more WAM interfaces are included, while IDM and Action-Centric improve from 60.0% to 90.0% and from 60.0% to 85.0%, respectively. These gains in individually evaluated interfaces, without PAIR, support more effective use of the same trajectory supervision through co-training at a fixed total training budget.

(3) Co-training versus interface selection

We appreciate the reviewer's recognition of the value of adaptive interface selection in balancing efficiency and performance. We would like to clarify that co-training is motivated by complementary learning signals and knowledge transfer across interfaces, while the shared backbone also enables PAIR to adaptively select among the learned interfaces. We will make this distinction explicit in the revised Introduction.

W2. OOD adaptation and supervision scale

We thank the reviewer for these comments and will revise the related statements in the Abstract, Introduction, and Section 4.3. First, regarding your concern about overinterpretation, Sections 3.3 and 4.3 describe target-domain adaptation of interface selection with a frozen backbone; the reported gains are within this setting.

Second, to address your concern about “small subset,” we will replace it with the absolute counts of 2,000 chunks from 500 OOD episodes per task, clarifying that 0.0274% is the sampling ratio relative to overlapping window candidates.

PAIR's benefit under this supervision budget is adapting only a lightweight router to reuse complementary pretrained capabilities. Our supplementary Folding Clothes comparison supports this: PAIR achieves 75.0% success versus 65.0% for the best LoRA-adapted interface using the same target-domain data.

AuthorsOpenReview ↗

Response (2/3): Contributions of atomic experts

Q1. Functional contributions of atomic capability experts

Below, we first analyze expert utilization using router weights, then examine the functional contributions of atomic capability experts through newly added expert-removal ablations. These results will be incorporated into the final revised manuscript.

(1) Gate-weighted expert utilization

First, we quantify expert utilization by reporting the gate weights assigned to shared and atomic experts for each interface and token group (Table 2).

Table 2. Gate-weighted expert utilization (%).

InterfaceToken GroupAtomic ExpertSharedAtomic
JointVideoVGM78.421.6
JointActionIDM63.236.8
IDMVideoVGM74.925.1
IDMActionIDM66.733.3
Action-CentricVideoWM81.318.7
Action-CentricActionVLA59.540.5
Action-OnlyVideoVGM72.127.9
Action-OnlyActionVLA65.834.2
AC-WMVideoWM69.630.4

Shared experts receive 59.5%–81.3% of the gate weight. Within each WAM interface, action tokens assign higher weights to their atomic experts than video tokens do. CATR defines which capability-aligned experts each token group can access and learns their relative weights. These patterns show how CATR balances shared computation with capability-specific computation within its eligible expert sets.

(2) Expert removal with matched continued training

Second, we assess the functional contributions of atomic experts through removal ablations (Table 3). Both intact and ablated variants start from the same trained Folding Clothes checkpoint and continue for 20K steps with identical training data and optimization settings.

Table 3. Expert removal with matched continued training.

InterfaceRemoved ExpertFull (Continued)w/o ExpertΔ Drop
JointVGM90.070.020.0
JointIDM90.020.070.0
IDMVGM95.050.045.0
IDMIDM95.035.060.0
Action-CentricWM85.055.030.0
Action-CentricVLA85.030.055.0
Action-OnlyVGM85.060.025.0
Action-OnlyVLA85.025.060.0
AC-WMWM61.7431.2030.54

The metric is success rate (%) for WAM interfaces and EWMScore-P for AC-WM. Δ Drop is the absolute difference in the corresponding metric. Both variants receive the same 20K additional steps.

Removing IDM/VLA lowers action success by 55–70 percentage points, consistent with the higher atomic-expert weights on action tokens. Removing VGM/WM causes 20–45-point drops in WAM success, while removing WM reduces AC-WM's EWMScore-P from 61.74 to 31.20. These ablations support substantial contributions from action experts and show that world-prediction experts benefit both video prediction and action generation. Together with the routing patterns, these results support UniW(A)M's design of combining shared computation with complementary capability-specific experts to improve world–action modeling.

AuthorsOpenReview ↗

Response (3/3): Training data and matched initialization

Q2. Training data and checkpoint provenance

We have provided further details on the co-training data, robot platform, and sources of model checkpoints and baseline results, and will incorporate this description into the revised manuscript. Specifically, our AgileX dual-arm dataset contains 4,248 episodes, 6.41M frames, and 59.33 hours across three real-world tasks.

Table 4. Real-world training dataset.

TaskEpisodesFramesDuration (h)
Tabletop Cleaning2,4261,528,83114.16
Folding Box1,2171,768,22016.37
Folding Clothes6053,110,73228.80
Total4,2486,407,78359.33

(1) Initialization and co-training budget

Durations use 30 FPS. UniW(A)M starts from Wan2.2-TI2V-5B, whose pretrained FFN weights initialize both shared and atomic experts. The main model jointly trains five paradigms (AC-WM and four WAM interfaces) in one backbone, with approximately 30K steps per paradigm and 150K steps in total. Table 1 also fixes the total budget at 150K for every configuration, rather than allocating 150K to each paradigm.

(2) Baseline and benchmark sources

Real-world baselines are independently trained and evaluated using official implementations, available pretrained checkpoints or recommended backbone initializations, with matched task data, training steps, GPU configurations, and batch sizes.

On LIBERO and RoboTwin 2.0, we train and evaluate UniW(A)M using official datasets and recommended configurations, obtaining 97.5% and 66.6% C2R, respectively. Baseline scores come from published benchmark results.

On WorldArena, UniW(A)M in AC-WM mode and all baselines are independently trained and evaluated by us. All reported results come from our experiments.

We will document these data, initialization, and baseline sources in the revised manuscript.

Q3. Dense baseline with matched initialization

First, the original Wan2.1-14B Dense baseline (Table 15) shows that jointly training the paradigms in a single dense network fails to produce effective policies under our training setup. On Folding Clothes, it averages only 7.5% success across the four interfaces, with Action-Centric and Action-Only both at 0%.

Second, to directly address your question about initialization, we additionally train a Dense baseline from Wan2.2-TI2V-5B, matching UniW(A)M's initialization source, training data, optimization settings, and total training budget. This baseline exhibits the same joint-learning difficulty: three interfaces achieve 0% success, and its average is 3.1%, below the 14B baseline's 7.5% (Table 5).

Table 5. Architecture comparison on Folding Clothes (success rate, %).

ArchitectureJointIDMAction-CentricAction-OnlyAverage
Dense (Wan2.1-14B)10.020.00.00.07.5
Dense (Wan2.2-5B, New)12.50.00.00.03.1
UniW(A)M (Full)90.090.085.085.087.5

The performance gap therefore persists with matched pretrained initialization. Compared with dense co-training, UniW(A)M reuses common knowledge through shared representations while preserving paradigm-specific computation through atomic experts and CATR, enabling more effective joint learning of different video–action dependencies.

Our Sample-Level Routing vs. CATR ablation provides complementary evidence with matched expert architecture and parameter count. We will include the new results and clarify these comparison conditions in the revised manuscript.

Comments on the paper

3 comments

Dear Reviewers and Area Chair,

Thank you again for your thoughtful comments and constructive suggestions. We have uploaded a revised manuscript incorporating the additional experiments and clarifications discussed in our responses, with all revised content highlighted in blue. The main updates include:

(1) clarified motivation and progressive co-training evidence for unified training; (2) expert utilization and expert-removal analyses; (3) a Dense baseline with matched initialization; (4) qualified OOD claims and comparisons with heuristic routing and LoRA adaptation; (5) training data details, a clarified novelty statement, and a discussion of generalization scope.

We sincerely appreciate the time and care you have devoted to reviewing our paper, and hope the revised manuscript addresses your concerns.

Area_Chair_SFRDOpenReview ↗

Discussion

Dear reviewers,

We have mixed signals from the review ratings.

  • 4eaM: 3: Weak acceptance
  • HSQK: 2: Weak rejection
  • gD5h: 3: Weak acceptance

Could you take a look at each other's reviews and the author responses and initiate a discussion to reach consensus?
Were your concerns resolved and questions answered?
How do you weigh the strengths and weaknesses?
Are there any issues or agenda you want to highlight?

Please note that ICLR 2027 has Policy on LLM use.

Best, Your AC

AuthorsOpenReview ↗

Summary of Rebuttal Discussion and Manuscript Revisions

Dear Area Chair and Reviewers,

Thank you for your thoughtful reviews and constructive discussion. UniW(A)M studies whether world prediction and multiple action-generation interfaces can be learned in one backbone so that each improves the others, and whether a robot can then select among them at deployment. The discussion helped us clarify why co-training is needed, which components of the capability decomposition carry the gains, and the scope of PAIR's OOD results. All additional experiments and clarifications are incorporated into the revised manuscript, highlighted in blue.

1. Co-training improves the interfaces themselves

Reviewers 4eaM and HSQK asked why AC-WM and multiple WAM interfaces should be trained jointly. With data and total steps fixed, adding AC-WM improves real-world success by 15.0, 15.0, 31.7, and 43.3 points for Joint, IDM, Action-Centric, and Action-Only, despite fewer updates per WAM; the latter two generate no video at deployment. In new progressive co-training on Folding Clothes (150K total steps per configuration), adding interfaces raises Joint from 60.0% to 90.0%, IDM from 60.0% to 90.0%, and Action-Centric from 60.0% to 85.0%, showing cross-interface transfer beyond AC-WM alone.

Reviewer 4eaM: "The controlled ablations demonstrating the effect of CATR and of co-training with the AC-WM are convincing."

2. The capability decomposition, not generic sharing, drives the gains

Reviewer gD5h asked us to isolate the architectural contribution; Reviewer 4eaM asked for evidence of expert specialization and a matched Dense baseline.

  • Expert utilization: shared experts receive 59.5%–81.3% of gate weight, and action tokens consistently weight their atomic experts more than video tokens do.
  • Expert removal (same checkpoint, matched 20K-step continued training): removing IDM/VLA lowers success by 55–70 points, and removing VGM/WM by 20–45 points; removing WM also reduces AC-WM's EWMScore-P from 61.74 to 31.20.
  • Matched Dense: initialized from the same Wan2.2-TI2V-5B checkpoint with matched data and budget, Dense reaches 3.1% average success (7.5% for Wan2.1-14B Dense) vs. 87.5% for UniW(A)M.
  • Routing granularity: with identical experts and parameters, CATR raises Action-Centric and Action-Only from 10.0% under sample-level routing to 81.7% and 80.0%.

3. PAIR: qualified claims and simpler alternatives

Following Reviewers 4eaM and HSQK, we now describe PAIR's OOD gains as target-domain adaptation with a frozen backbone, and state the absolute supervision scale (2,000 chunks from 500 OOD episodes per task). Under identical OOD supervision on Folding Clothes, PAIR achieves 75.0%, vs. 65.0% for the best fixed interface, 65.0% for the best of four LoRA-adapted interfaces, and 45.0% for phase-based heuristic routing with Qwen3-VL-8B. We also added a discussion of generalization scope: our evaluation covers shifts within three manipulation tasks, and the atomic experts are defined by prediction roles rather than by scaling expert count; broader compositional generalization remains future work.

4. Data and reproducibility

The revision documents the real-world dataset (4,248 AgileX dual-arm episodes, 6.41M frames, 59.33 hours), Wan2.2-TI2V-5B initialization with a 150K-step budget, and the sources of all baseline and benchmark results. Real-world baselines use matched data, steps, GPUs, and batch sizes. We will fully open-source UniW(A)M.

Reviewer 4eaM, after the rebuttal: "The response has resolved all of my concerns. I am therefore raising my score."

We sincerely appreciate the time and care the reviewers and Area Chair have devoted to UniW(A)M.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.