acceptodds
Under review as a conference paper at ICLR 2027

MuxWAM: A Compact World–Action Model with a Neural Multiplexer

Abstract

World–Action Models (WAMs) extend action-centric Vision–Language–Action (VLA) policies with explicit foresight by jointly predicting future world states and robot actions. However, dense future-image prediction in WAMs incurs inference overhead, while recursive decoding and re-encoding can compound prediction errors. To address these challenges, we introduce MuxWAM, a compact WAM whose unified multiplexer-inspired world constructor (MUX) maps dense visual patches to a fixed set of task-conditioned world tokens per view. MUX uses task-indexed routing over shared experts to refine visual summaries, with normalized global fusion completing the construction. The model jointly predicts future world tokens and actions through flow matching, without future-image decoding at inference. We evaluate MuxWAM on LIBERO, LIBERO-Plus, Meta-World, and physical ALOHA manipulation tasks. With only two world tokens per view, MuxWAM achieves 98.8% mean success on LIBERO, 83.76% averaged equally over Meta-World's four difficulty groups, and 78.3% across three physical tasks under normal conditions. These results demonstrate the effectiveness of task-conditioned compact world representations for robot control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, VLM, embodied intelligence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ydz2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MuxWAM compresses visual patches into task-conditioned world tokens and jointly predicts future tokens and robot actions through flow matching, without decoding future images during deployment. Using two tokens per view, it reports 98.8% mean LIBERO success and demonstrates the approach on physical grasping, folding, and assembly tasks.

Critical strengths
  1. Two world tokens per view support joint world–action prediction without future-image decoding at deployment.
  2. Task-conditioned routing improves LIBERO-Long success from 92.8% to 98.4% while retaining the same expert-bank capacity.
  3. Token-budget sweeps, perturbation diagnostics, and physical folding and assembly experiments test several aspects of compact representations.
Critical weaknesses
  1. LIBERO-Plus task-ID mappings and adaptation logs are unavailable, leaving the reported generalization protocol incompletely verifiable.
  2. Cosine-error diagnostics do not establish that predicted future tokens causally improve actions; no action-only or future-token intervention control is reported.
  3. Compact token counts are not accompanied by measured latency or memory, and every expert is evaluated despite Top-2 weighting.
Questions for authors
  1. Can you release variant-to-task mappings and adaptation records, then verify frozen-checkpoint LIBERO-Plus results with original task IDs and a language-derived routing control?
  2. Can you compare against an action-only model with matched capacity and mask or shuffle predicted future tokens to quantify their contribution to success?
  3. Can you report batch-one latency, peak memory, and success across token budgets, including encoder, MUX, all expert evaluations, and joint denoising costs?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ydz2(Q1 & W1)

Thank you for the constructive questions. We provide the task mappings, control ablations, and measured deployment costs below.

W1 / Q1 — LIBERO-Plus mapping, adaptation and language routing

We appreciate the reviewer's careful attention to reproducibility and the valuable suggestion to make the task mapping and routing protocol more explicit.

The table below lists all 40 original operations and their expected router indices from the corresponding training metadata. Variant counts sum to 10,030 mapping entries.

Table R1. LIBERO / LIBERO-Plus task-to-router mapping

Note: Each entry denotes LIBERO router → Plus router (number of variants). ID refers to the original task ID within each suite.

IDSpatialObjectGoalLong
04→7 (231)4→0 (242)9→2 (268)5→8 (212)
17→3 (198)2→4 (173)7→7 (156)7→1 (210)
28→2 (267)6→9 (264)4→1 (270)3→2 (281)
35→8 (246)3→8 (292)2→6 (324)8→5 (304)
41→0 (280)1→2 (214)8→5 (243)0→0 (287)
52→6 (256)8→6 (251)5→9 (270)9→9 (237)
60→1 (232)7→3 (273)3→0 (291)1→4 (283)
73→5 (269)5→1 (272)6→8 (232)4→3 (194)
86→4 (180)9→5 (257)0→3 (235)6→7 (233)
99→9 (243)0→7 (280)1→4 (302)2→6 (278)

Reviewer-requested routing control with the original instruction held fixed: Camera and Background. We evaluate closed-loop control with language-derived routing under these two visual perturbation categories while keeping the original instruction unchanged.

We evaluate every Camera and Background variant in all four LIBERO-Plus suites, totaling 2,675 episodes. We use Plus-trained models, with one predefined initial state per variant. Both the language selector and action policy receive the unchanged original task instruction. The selector retrieves among the ten known task descriptions using cosine similarity of masked-mean vocabulary embeddings; the client supplies an invalid task-ID placeholder, and the server uses the retrieved router ID.

Table R2. New language-routing measurements: successful / total episodes and success rate.

SuiteCameraBackground
Spatial343/376 (91.22%)244/258 (94.57%)
Object368/396 (92.93%)236/248 (95.16%)
Goal346/408 (84.80%)248/281 (88.26%)
Long367/419 (87.59%)262/289 (90.66%)

The suite-equal mean success rates are 89.14% for Camera and 92.16% for Background.

Table R3. Paper Appendix Table 9 (p. 21): Plus-trained original-ID success rates (%). We reproduce the Camera and Background columns from the "Trained on LIBERO-Plus (In-domain Evaluation)" block.

SuiteCamera (%)Background (%)
Spatial98.4098.84
Object99.4999.60
Goal90.493.6
Long96.995.5
Suite-equal average96.396.9

Conclusion. On known tasks with unchanged original instructions, language-derived routing achieves suite-equal success rates of 89.14% on Camera and 92.16% on Background. This additional evaluation provides control evidence for MuxWAM's compact world representation under these visual perturbations.

Table R4. Supplementary evaluation of the additional control: original instructions on standard LIBERO-Long. We also evaluate the newly added language selector on all ten tasks with 50 initial states per task (500 episodes), using the original benchmark instructions and the frozen Long-38K checkpoint. Language retrieval selects among the ten known task descriptions using frozen policy vocabulary embeddings; the policy server receives an invalid task-ID placeholder and replaces it with the language-selected router. Evaluation uses 4 GPUs and 12 rollout processes, with no training.

RoutingResult sourceSuccessful / total episodesSuccess %
Original task IDPaper Table 2; Appendix A.6492/50098.4
Language retrievalNew full LIBERO-Long evaluation478/50095.6

The original-ID reference uses the 98.4% result in paper Table 2 (p. 7); the count of 492/500 is reported in Appendix A.6 (p. 24).

AuthorsOpenReview ↗

Response to Reviewer ydz2(Q2 & W2)

W2 / Q2 — Do future tokens contribute to actions?

We thank the reviewer for this insightful question, which addresses the key connection between future-world prediction and action control. We add a training ablation and inference-time interventions to directly test the contribution of future-world tokens to action generation.

Experimental setup. For the training ablation, we retain world supervision and retrain MuxWAM while blocking action queries from reading the keys and values of future-world tokens during both training and inference. For the inference-time interventions, we freeze existing models and, during joint denoising, separately block this attention pathway, replace future tokens with current tokens, or reverse their temporal order. Each inference-time intervention is evaluated on all four LIBERO suites, with 500 episodes per suite.

Table R5. Contribution of future-world tokens to control. Success rates (%).

Method / settingSpatialObjectGoalLongAvg.
MuxWAM (Full, paper Table 2)98.8100.097.898.498.8
Block future-world-to-action attention (training + inference)95.296.496.294.695.6
Block future-world-to-action attention (inference only)0.20.03.20.00.85
Replace future tokens with current tokens (inference only)48.886.671.857.066.05
Reverse future-token temporal order (inference only)94.898.898.095.296.7

Results and analysis. Blocking future-world-to-action interaction during both training and inference reduces success across all four suites, from 98.8% to 95.6% on average. On the frozen models, blocking the pathway and replacing future tokens yield mean success rates of 0.85% and 66.05%, respectively, while temporal reversal achieves 96.7%. Blocking or replacing future-world information affects control more strongly than changing its temporal order.

Together, these results indicate that future-world tokens provide useful information for action generation during joint denoising, and the future-world-to-action pathway contributes to control performance.

AuthorsOpenReview ↗

Response to Reviewer ydz2(Q3 & W3)

W3 / Q3 — Batch-one latency, memory and token budgets

We appreciate this valuable suggestion from a deployment perspective, which helps provide a more complete assessment of the computational benefits of compact representations.

Table R6. New measurements: batch-one cost across K=1, 2, 4, 8, 64, 128 and 256. We profile all budgets on the same exclusive NVIDIA L20, with two 224×224 views, a ten-action chunk and ten Euler updates. K is the number of world tokens per view. The frozen Object-30K checkpoint trained with K=2 is reused throughout.

K per viewMean msP50 msP95 msPeak live JAX GiBLatency / K=2
170.89470.35373.7626.6020.96×
273.80173.35277.2816.6021.00×
477.48577.47978.6406.5281.05×
886.66386.45390.7556.5061.17×
64216.818216.394220.1466.6022.94×
128605.646605.513608.6266.8178.21×
2561807.3821806.8521813.9228.19624.49×

At K=2, end-to-end inference is 24.49× faster and peak live memory is 19.4% lower than at K=256 in this sweep. These measurements quantify the deployment cost of the world-token budget with encoder and expert computation included.

Table R7. Paper Appendix Table 10 (p. 23): token-budget success (%), H=10.

SuiteK=1K=2K=4K=8
Spatial97.098.898.296.8
Object98.6100.099.098.0
Goal97.897.897.097.2
Long96.098.498.295.4

Table R8. Independent native K=2 repeat. Under the same deployment configuration, a separate run uses 100 warm-up calls and 500 measured calls.

CheckpointWarm-upCallsMean msP50 msP95 msPeak live JAX GiB
K=2; Object-30K10050073.10572.81975.3576.602
Reviewer_WzpQOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MuxWAM fine-tunes π0 so that each camera view is represented by only K=2 tokens produced by a constructor, MUX. MUX fuses a linear patch-pooling branch with a learned-query MoE branch whose router is selected by a closed-set task ID, over shared experts. The action expert jointly flow-matches the action chunk and the future compressed tokens. Targets come from the same encoder + MUX applied to future frames, and the predicted tokens are discarded at inference. Two propositions give a low-rank sufficiency condition and a Lipschitz bound on decode/re-encode error.

Critical strengths
  1. Strong results with an aggressive bottleneck. Running π0's VLM on only 4 visual tokens (two per view) still reaches 98.8% on LIBERO and 83.8% on Meta-World (Tab. 2, Tab. 3).

  2. Large real-robot gains on two hard bimanual tasks. Clothes folding reaches 65% and phone assembly 70%, versus ≤25% and ≤15% for π0, π0.5 and Cosmos Policy (Tab. 4). The physical protocol (A.2) has fixed success predicates and safety stops counted as failures.

Critical weaknesses

W1 [Novelty] The framework closely reproduces prior work, and the remaining delta is also known. OneWM-VLA ("One Token Per Frame: Reconsidering Visual Bandwidth in World Models for VLA Policy", Tang et al., arXiv 2605.07931, May 2026) is cited here only inside a list (Sec. 1, Sec. 2). It already proposes the same framework: a π0 backbone; attention pooling that compresses each camera view into one (or a few) tokens; joint conditional flow matching of future compact latents and actions; future observations used only as training targets, not needed at inference. It evaluates on the same benchmarks (MetaWorld MT50, LIBERO 98.1%, and real-robot cloth folding on a Piper arm, where π0 goes from 20% to 60%) and includes a 1-vs-3-tokens-per-view study. MuxWAM's interface (Definition 1, Sec. 4.2), training targets and deployment path are essentially this design with K=2.

The remaining delta, the "multiplexer", is a task-specific router over a shared expert pool with Top-k selection. That is exactly the mechanism of Sparse Diffusion Policy (Wang et al., CoRL 2024; uncited), which uses task-specific routers so that different tasks select different shared experts, and of M3ViT (cited) for multi-task vision.

The paper needs a direct, same-recipe comparison with OneWM-VLA (and ideally a CogVLA/Compressor-style instruction-conditioned compressor), and must state precisely what is new beyond "OneWM-VLA + task-indexed MoE routing".

W2 [Soundness/Positioning] The motivation rests on a pipeline that current WAMs do not use, and the benefit of world prediction is not shown.

(a) Codec-cycle motivation. The abstract, introduction and Proposition 2 motivate compact tokens by the error that accumulates when predicted images are decoded to pixels and re-encoded. The paper never identifies an existing WAM that uses such a loop; the introduction states it only conditionally (“if predicted images are decoded and re-encoded…”), and no such baseline is evaluated. Current latent WAMs (e.g., Wan- or Cosmos-based) roll out directly in latent space, so this error channel does not arise in practice, and Proposition 2 addresses a problem the compared systems do not have.

(b) Role of world prediction. There is no ablation without the future-token loss (λ_world = 0). It is therefore unclear whether predicting future tokens helps control at all, or whether the gains come from the compressed, task-routed context alone. The diagnostics do not resolve this: the prediction error is nearly flat from the first step (Fig. 3a,b), and without a copy-current or mean-token reference its quality cannot be judged.

Questions for authors
  1. What exactly is new relative to OneWM-VLA (Tang et al., 2026)? Can you provide a same-backbone, same-data comparison?

  2. How does MUX's task-indexed routing differ from the task-specific routers of Sparse Diffusion Policy (CoRL 2024)?

  3. What happens with the future-token loss removed (λ_world = 0)? What are the copy-current/mean-token baseline errors?

4.Which WAM baseline actually decodes predicted latents to pixels and re-encodes them? If none, what is the empirical relevance of Proposition 2?

4 comments
AuthorsOpenReview ↗

Response to Reviewer WzpQ (1/4)

Q1 & W1. Contribution Relative to OneWM-VLA

We appreciate the reviewer’s interest in the relationship between MuxWAM and OneWM-VLA, which we cite in Section 2 (Related Work). Both MuxWAM and the closed-source OneWM-VLA were developed based on OpenPI. This shared foundation may have given the impression that MuxWAM is simply “OneWM-VLA + task-indexed MoE routing.” We would like to clarify this point, they address different research questions and introduce distinct technical designs.

Motivation. OneWM-VLA focuses on predictive visual bandwidth: how many visual tokens should be retained per view and future timestep to balance long-horizon control performance against generation cost. Its motivation is that increasing the number of visual features enlarges the joint generation sequence without necessarily improving control.

In contrast, MuxWAM investigates how to construct task-conditioned visual representations under a compact world–action modeling interface. As shown in Figure 1 and Table 4 of the paper, We jointly trained one policy on demonstrations from three physical tasks and evaluated the same checkpoint on all three. Rather than studying the optimal number of visual tokens to retain, we focus on constructing informative representations that support task-conditioned prediction under a fixed token budget.

Methodological differences. We acknowledge that both methods are developed upon OpenPI, which may contribute to their architectural similarities. However, their visual construction mechanisms is fundamentally different, as summarized below.

DimensionOneWM-VLAMuxWAM (Ours)
Visual constructionAdaptive Attention Pooling (AAP)MUX: linear aggregation and dual learned-query groups
Task conditioningLanguage/policy-context conditioningExplicit task-indexed routing
Expert organizationNo task-indexed MoETwo independent expert banks for query groups
Shared informationAAP-based visual aggregationLinear bypass without expert refinement
Feature fusionLearned AAP fusionHierarchical RMS-normalized global fusion
Routing optimizationNo MoE routing objectiveTask-wise expert load-balancing regularization

While sharing the compact joint world–action prediction paradigm, MuxWAM introduces task-conditioned visual construction, pathway-specific expert organization, hierarchical normalized fusion, and asymmetric world–action attention under a fixed K-token interface.

Distinct contributions and empirical evidence. Our contributions are threefold:

  1. Task-conditioned visual construction under an extreme compression constraint. Inspired by digital multiplexers, we introduce MUX to construct compact visual representations while preserving task-conditioned specialization. Unlike conventional task-indexed MoE routing, MUX integrates multiple complementary visual aggregation mechanisms into a fixed compact interface. We provide theoretical analysis and empirical evidence to investigate representation sufficiency under extreme compression.

  2. Task-conditioned multi-task learning with real-robot deployment. MuxWAM supports mixed-task training and single-model inference across multiple real-world manipulation tasks under the same compact visual interface. This is particularly important for real-robot deployment, where maintaining a single model across tasks is practically valuable. Our real-robot results, including the multi-task inputs illustrated in the main figure, demonstrate this capability.

  3. Strong benchmark performance. MuxWAM achieves competitive or superior performance on LIBERO and LIBERO-Plus, and substantially outperforms OneWM-VLA on Meta-World. The following comparisons use the same backbone and training data.

LIBERO success rate (%). As shown in Table R1, MuxWAM achieves an average success rate of 98.8%, outperforming OneWM-VLA (98.1%).

Table R1. LIBERO Performance Comparison.

MethodSpatialObjectGoalLongAvg.
OneWM-VLA98.299.699.095.698.1
MuxWAM98.8100.097.898.498.8

Meta-World success rate (%). The average equally weights the four difficulty groups. The advantage is more pronounced on Meta-World. As reported in Table R2, MuxWAM achieves 83.76% average success, substantially outperforming OneWM-VLA (61.53%) across all four difficulty groups.Besides it, MuxWAM also achieves an overall success rate of 81.1% on LIBERO-Plus.

Table R2. Meta-World Performance Comparison.

MethodEasyMediumHardVery HardAvg.
OneWM-VLA79.2941.8265.0060.0061.53
MuxWAM92.2175.6478.8088.4083.76

MuxWAM’s contribution lies in task-conditioned visual representation construction for compact world–action modeling, with strong control performance across real multiple manipulation tasks.

AuthorsOpenReview ↗

Response to Reviewer WzpQ (2/4)

Q2. Relationship to Sparse Diffusion Policy (SDP)

We thank the reviewer for pointing out the missing citation and the relationship to Sparse Diffusion Policy (SDP). We apologize for this omission and will add the relevant reference (Wang et al., 2024) and clarify the distinction in the revised manuscript.

We acknowledge SDP as prior work on task-indexed expert routing. However, MuxWAM differs in both the placement of expert computation and how task-conditioned experts are organized and integrated into compact world representation construction.

StructureSDPMuxWAM (Ours)
MoE insertionDiffusion Policy Transformer layersVisual representation constructor
Routing granularityTask-specific router per MoE layerIndependent task-router banks per query group
Expert organizationShared expert pool within each MoE layerTwo independent expert banks, each shared across tasks
Expert inputPolicy hidden statesTwo complementary learned-query representations
Expert functionRefine policy features for action generationRefine aggregated visual evidence for world representation
Shared pathwayPolicy Transformer residual pathwayLinear summary bypasses expert refinement
Expert computationSparse expert activationAll four expert responses computed, followed by Top-2 weighted aggregation
FusionIntegrated within policy layersHierarchical RMS-normalized global fusion
OutputPolicy hidden representationsFixed K-token predictive world representation

Importantly, SDP applies task-indexed sparse expert routing to policy Transformer features, whereas MuxWAM performs task-conditioned processing during visual representation construction. Its output is a compact predictive world representation that is subsequently integrated into joint world–action prediction.

Thus, although both methods employ task-indexed routing, they differ in their architectural placement, expert organization, information pathways, and prediction interfaces.

AuthorsOpenReview ↗

Response to Reviewer WzpQ (3/4)

Q3 & W2b. Role of World Prediction and copy-current/mean-token baseline errors

Future-token supervision. Setting λworld=0\lambda_{\mathrm{world}} = 0 reduced LIBERO-Long success from 98.4% to 0.0% in one training run. This removes direct supervision of future tokens, while MUX can still receive gradients from the action loss.

Action access to future tokens. We retained world supervision and retrained MuxWAM with an attention mask that blocks action queries from reading future-world keys and values during both training and inference. Success fell across all four LIBERO suites, from 98.8% to 95.6% on average. This supports the contribution of the future-world-to-action pathway to control.

Table R3. LIBERO success rates (%).

MethodSpatialObjectGoalLongAvg.
MuxWAM (Full)98.8100.097.898.498.8
MuxWAM (w/o future-world-to-action attention)95.296.496.294.695.6

LIBERO-Plus robustness under different perturbations.

We further examine this effect under distribution shifts. Table R4 shows that the full model achieves an 81.1% macro average, compared with 72.5% for the attention-masked variant, with particularly notable differences under lighting, background, and noise perturbations.

Table R4. Robustness under LIBERO-Plus Perturbations.

ModelOriginalCameraRobotLanguageLightBackgroundNoiseLayoutMacro Avg.
MuxWAM (Full)98.876.1 ↓22.749.5 ↓49.397.4 ↓1.495.6 ↓3.290.1 ↓8.781.7 ↓17.177.3 ↓21.581.1
MuxWAM (Mask World)95.668.3 ↓27.346.6 ↓49.092.8 ↓2.883.6 ↓12.080.4 ↓15.263.6 ↓32.072.2 ↓23.472.5

The full model achieves an 81.1% macro average, compared with 72.5% for the masked variant. The largest differences appear under lighting, background, and noise perturbations. These results suggest that future-world information is particularly beneficial for maintaining robustness under visual distribution shifts, beyond its contribution to standard task execution.

Copy-current and task-mean diagnostic baselines. We compared all three predictors on the same 400 LIBERO episodes (6,891 complete action chunks), measuring cosine error against the tokens encoded from the actual observation at step 10. Copy-current reuses the tokens from the start of each chunk. Task-mean uses a fixed mean token for each task, camera view, and token slot, computed from about 270K non-initial frames across 1,693 training demonstrations.

Table R6. Step-10 cosine error on LIBERO (lower is better).

LIBERO suiteComplete chunksMuxWAMCopy-currentTask-mean
Object1,2880.359810.3980.411
Spatial9580.368510.4230.409
Goal1,2790.355840.3870.395
Long3,3660.366420.3920.401
Equal-weight mean6,8910.362640.4000.404

MuxWAM achieves lower step-10 cosine error than both baselines in all four suites. Averaged equally across suites, its error is 9.3% lower than copy-current and 10.2% lower than task-mean.

AuthorsOpenReview ↗

Response to Reviewer WzpQ (4/4)

Q4 / W2(a). Recursive Error Propagation and Latent Imagination

We thank the reviewer for raising this concern. Proposition 2, which we state and prove in Section A.8.2 of the supplementary material, analyzes recursive prediction-error propagation under a pixel-feedback interface, rather than merely the error introduced by a single decode–encode operation. In recursive imagination for model-based RL, if predicted images are re-encoded as inputs for subsequent predictions, each codec residual can propagate through later transitions. We clarify that this conditional interface is not established as the inference pipeline of our compared latent-WAM baselines.

Codec-cycle verification. We conducted a controlled experiment using a frozen Cosmos Policy, comparing direct latent feedback with single and repeated codec cycles. Across 100 LIBERO-Long segments, the one-step codec residual was 0.1705 (95% CI [0.1703, 0.1707]). At depth 8, prediction errors were 0.5780 (Direct), 0.5655 (Single-cycle), and 0.4139 (Every-cycle) (Table R7). The lower realized error under repeated cycles is consistent with Proposition 2 being an upper bound rather than a claim of inevitable error growth. These inserted cycles are diagnostic interventions, not Cosmos Policy’s original inference pipeline.

Table R7. Recursive prediction error under three feedback strategies in a frozen Cosmos Policy. Brackets give 95% confidence intervals. Lower error is better.

Feedback strategyDepth 1Depth 2Depth 4Depth 8
Direct0.2396 [0.2292, 0.2506]0.3335 [0.3238, 0.3446]0.4542 [0.4458, 0.4631]0.5780 [0.5669, 0.5896]
Single-cycle0.2037 [0.1917, 0.2162]0.2914 [0.2806, 0.3035]0.4227 [0.4143, 0.4314]0.5655 [0.5547, 0.5769]
Every-cycle0.2037 [0.1917, 0.2162]0.2652 [0.2538, 0.2776]0.3327 [0.3232, 0.3425]0.4139 [0.4040, 0.4241]

Recursive latent imagination with MuxWAM. We further evaluated direct latent feedback using a frozen MuxWAM (K=2K=2 tokens per view). Across all ten LIBERO-Long tasks, cosine error remained relatively stable from 0.37804 at depth 1 to 0.38262 at depth 8 (Table R8).

Table R8. Recursive latent prediction error on LIBERO-Long. Cosine error is averaged equally across the ten tasks and two camera views.

Imagination depth (chunks)Cosine error ↓Change from depth 1Valid rollouts
10.37804—100
20.38458+0.00653100
40.38386+0.00582100
80.38262+0.00458100

Downstream RL utility. We used the MuxWAM checkpoint saved at step 15,000 of few-shot training as the base policy. Following Wagenmaker et al. (2025), we used diffusion steering via reinforcement learning (DSRL) to train a policy that selects MuxWAM’s initial action noise. With a 50/50 mix of real samples and samples from MuxWAM’s recursive latent rollouts, a depth of two chunks yielded the highest observed mean success rate, 86.0%, compared with 80.0% for real-only RL (Table R9). At the same sampling ratio, extending the rollouts to eight chunks reduced mean success to 76.0%.

Table R9. Effect of imagination depth on noise-space RL. Success rates are averaged across training seeds. Real-sample share is the proportion of training samples drawn from real experience.

RL settingImagination depth (chunks)Real-sample shareMean success rate (%)Difference from real-only (pp)
Real-only RL—100%80.0—
MuxWAM + RL150%84.0+4.0
MuxWAM + RL250%86.0+6.0
MuxWAM + RL450%82.0+2.0
MuxWAM + RL850%76.0−4.0

Together, these experiments distinguish conditional codec-related error propagation from direct compact-latent imagination, while demonstrating the downstream utility and depth sensitivity of MuxWAM’s predicted worlds.

Reviewer_QMEPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents MuxWAM, which uses a task-conditioned MUX module to compress visual features into a fixed number of world tokens per view. Flow matching is used to jointly predict future world tokens and actions. MUX combines shared visual aggregation with expert refinement controlled by task-indexed routers, without requiring future-image decoding at inference. Using two world tokens per view, the model reports a mean success rate of 98.8% on LIBERO, an equally weighted average of 83.76% across the four Meta-World difficulty groups, and 78.3% across three real-world tasks under normal conditions. The paper also includes ablations on construction branches, task routing, token budgets, and prediction horizons, along with an analysis of future-token prediction errors.

Critical strengths
  1. The paper demonstrates strong control performance with a small visual token budget across multiple settings. The two-token-per-view design achieves high success rates on LIBERO, Meta-World, and three real-world tasks under normal conditions. Appendix Table 10 compares different token budgets across the four LIBERO suites, providing empirical support for the default setting. These results show that compact visual representations can support control on the evaluated tasks.
  2. The benefit of task-conditioned routing is supported by a direct comparison. Table 5(c) compares task-conditioned routing with shared routing while keeping the token budget and expert-bank capacity fixed. Success on LIBERO-Long increases from 92.8% to 98.4%, with improvements on seven tasks, ties on two, and a decline on one.
Critical weaknesses
  1. The practical computational benefits of the compact representation are not quantified. The paper is motivated by reducing the overhead of dense future-image prediction, but it does not report comparisons of end-to-end inference latency, memory usage, or computational cost. Fewer visual tokens do not necessarily imply lower overall computational cost. Therefore, the trade-off between success rate and actual deployment cost remains unclear.
  2. Missing task-mapping records limit the verification of LIBERO-Plus robustness results. The LIBERO-Plus evaluation does not provide the mapping from perturbed variants to original task IDs or the run-level adaptation records, although task identity directly determines which router parameters are selected. These omissions limit the reproducibility and interpretation of the reported results.
  3. The effect of task count on routing gains is not studied. Although the paper includes ablations over (K), it does not examine how routing gains change with the number of tasks or how these changes relate to the token budget. The method’s scalability with a fixed shared expert capacity therefore remains unclear.
Questions for authors
  1. Proposition 2 analyzes errors introduced by the decoding and re-encoding cycle, but it does not establish that compression reduces the overall error. How do the authors assess the potential loss of control-relevant information caused by compression?
  2. Could the small increases in prediction error in Figure 3 reflect world tokens being insensitive to task-relevant changes?
  3. The method uses closed-set task IDs to select routers. How does the current system select a router for an unseen operation task? Can existing routers be reused, or must a new router be added and trained? If training is required, which modules need to be updated?
5 comments

I have yet to receive a response from the authors. As my concerns remain open at this stage, I will retain my original score.

AuthorsOpenReview ↗

Response to Reviewer QMEP(Q1 & Q2)

Thank you for the careful review. We use the paper's original success tables and identify additional measurements explicitly.

Q1 — Control-Relevant Information Retention under Compression

We thank the reviewer for raising this important question. To directly evaluate whether MUX preserves control-relevant information under compression, we conducted a task-relevant information retention probe using our K=2 checkpoint trained for 30K steps.

Experimental setup. We froze the entire MuxWAM and trained lightweight probes to predict target-relative positions and actions from three representations: dense SigLIP features (256 tokens/view), compact MUX features (2 tokens/view), and language with robot state only (0 visual tokens). All probes used matched readout capacity and training budgets. Probe errors are reported as mean ± standard deviation over three random seeds.

Table R1. New probe experiment: Control-Relevant Information Retention on LIBERO-10 (K=2, 30K checkpoint).

RepresentationTokens/ViewRelative Position MAE (cm) ↓Action MAE ↓LIBERO-10 Success ↑
Dense SigLIP2561.335 ± 0.0210.06093 ± 0.00034—
MUX (K=2)21.329 ± 0.0260.05948 ± 0.0004798.6%
Language + Robot State03.025 ± 0.1400.11024 ± 0.00417—

Results and analysis. Despite a 128× reduction in visual token count, MUX achieves probe accuracy comparable to dense SigLIP features for both relative positions (1.329 vs. 1.335 cm) and actions (0.05948 vs. 0.06093). Compared with language and robot state alone, MUX reduces relative-position error by 56.1% and action prediction error by 46.0%. The 98.6% LIBERO-10 success is measured on this K=2, 30K checkpoint; the paper's 98.4% LIBERO-Long result is reported in Table 2 (p. 7).

The token-budget ablation in paper Appendix Table 10 (p. 23) provides complementary control-level evidence across K=1/2/4/8.

Core finding. These results indicate that MUX preserves control-relevant spatial and action information in compact world representations, with probe accuracy comparable to dense features at a substantially reduced visual token budget, without requiring dense visual features in the downstream world–action model.

Q2 — Could small error changes indicate insensitivity?

We appreciate the reviewer's perceptive observation about the prediction-error diagnostics, which motivates a more direct examination of responses to task-relevant changes.

Small cosine-error changes alone are ambiguous. We supplement the paper's control results with direct changes to the input state.

Table R2. New frozen sensitivity measurement: 1,000 input pairs from 100 states across ten Object tasks. Target and distractor translations are ±2/±4 cm in world-x; lighting uses factors 0.7/1.3. Proprioception and sampling noise are held fixed. All cosine distances below are in units of 10⁻³; action RMS uses the three translation-command channels in controller units.

PerturbationPairsDense distanceCurrent MUXFuture tokensTranslation-action RMS
Distractor translation4005.22650.44140.03740.01050
Target translation4007.31460.92680.11620.01644
Lighting2000.71290.07300.00720.00511

These results demonstrate that MUX learns task-relevant world representations, with predicted future tokens exhibiting a 3.11× stronger response to target-object displacement than to distractor displacement, accompanied by a 1.57× larger action response.

Table R3. New small closed-loop check: one paired initial state per Object task, ten episodes per condition.

Closed-loop conditionSuccessful episodes / total
Original captured state10/10
Target +4 cm in world-x9/10
Distractor +4 cm in world-x10/10
Lighting ×0.710/10

In these short rollout tests, the policy succeeds in 9/10 trials after target-object shifts and 10/10 trials under each of the distractor and lighting conditions. These results complement the prediction-error analysis in Figure 3 (p. 9) by showing that the policy can still complete tasks under the tested perturbations.

AuthorsOpenReview ↗

Response to Reviewer QMEP(Q3 & W1)

Q3 — Selecting a router for an unseen operation

We thank the reviewer for this practically relevant question, which helps clarify the relationship between routing for known tasks and adaptation to new operations.

The current system selects routers by task ID. For new operations, we test router reuse and expansion, each followed by full-model continued training. Known-task results are listed below.

Table R4. Known-task routing settings and evidence.

SituationRouter choiceUpdated modulesEvidence
Paper setting: known operation, supplied identityReuse mapped routerNoneActual router matches in 2,518 new Object episodes
Additional control: original benchmark instructionNew frozen language selectorNoneFull LIBERO-Long: 500/500 router matches; 95.6% success
Additional control: Plus-trained, fixed instruction, Camera/BackgroundLanguage retrievalNoneCamera 89.14%; Background 92.16% (suite-equal)

Table R5. New experiments: adaptation from LIBERO-Spatial to LIBERO-Long. Both conditions start from Spatial-25K, trained without Long demonstrations, and continue on Long for 30K steps. Reuse follows Spatial's task-ID mapping: Long task i reuses the router assigned to Spatial task i, r_Long(i) = r_Spatial(i); expansion adds routing parameters for ten Long tasks, with two router groups per task. Both update the encoder, MUX routers and shared experts, and world/action modules, and are evaluated on the full LIBERO-Long suite.

SettingInitializationAdditional training stepsUpdated modulesLong success %
Paper Table 2: MuxWAM———98.4
New experiment: reuse existing routersSpatial-25K30KFull model96.4
New experiment: add Long task routersSpatial-25K30KFull model95.8

Reusing and adding routers achieve 96.4% and 95.8% success, respectively, 2.0 and 2.6 percentage points below the 98.4% reference in paper Table 2 (p. 7). Both strategies support adaptation to new operations through full-model continued training.

W1 — Practical computational benefits

We appreciate the reviewer's valuable emphasis on practical computational cost, which is essential for assessing the method's deployment value.

Table R6. New measurements: batch-one cost across K=1, 2, 4, 8, 64, 128 and 256. We profile all budgets on the same exclusive NVIDIA L20, with two 224×224 views, a ten-action chunk and ten Euler updates. K is the number of world tokens per view. The frozen Object-30K checkpoint trained with K=2 is reused throughout;

K per viewMean msP50 msP95 msPeak live JAX GiBLatency / K=2
170.89470.35373.7626.6020.96×
273.80173.35277.2816.6021.00×
477.48577.47978.6406.5281.05×
886.66386.45390.7556.5061.17×
64216.818216.394220.1466.6022.94×
128605.646605.513608.6266.8178.21×
2561807.3821806.8521813.9228.19624.49×

At K=2, end-to-end inference is 24.49× faster and peak live memory is 19.4% lower than at K=256 in this sweep. These measurements quantify the deployment cost of the world-token budget with encoder and expert computation included.

Table R7. Independent native K=2 repeat. Under the same deployment configuration, a separate run uses 100 warm-up calls and 500 measured calls.

CheckpointWarm-upCallsMean msP50 msP95 msPeak live JAX GiB
K=2; Object-30K10050073.10572.81975.3576.602

The success rates for K=1/2/4/8 are reported in paper Appendix Table 10 (p. 23); paper Table 2 (p. 7) reports 98.8% mean LIBERO success at K=2.

Table R8. New five-call GPU trace: kernel-active attribution.

AttributionGPU kernel-active ms/call
Encoder7.667
Prefix transformer11.003
Joint denoising22.118
Attributed MUX non-expert kernels0.190
Attributed expert-MLP kernels0.049
Cross-stage fused kernels1.973
Unattributed kernels3.625
Device copies2.278

These are kernel-active times, with cross-stage fusion and unattributed work retained. Expert-specific attribution covers only identifiable kernels; the end-to-end latency above includes every executed expert.

Comments on the paper

1 comment

Dear reviewer,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.