- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MuxWAM compresses visual patches into task-conditioned world tokens and jointly predicts future tokens and robot actions through flow matching, without decoding future images during deployment. Using two tokens per view, it reports 98.8% mean LIBERO success and demonstrates the approach on physical grasping, folding, and assembly tasks.
Critical strengths
- Two world tokens per view support joint world–action prediction without future-image decoding at deployment.
- Task-conditioned routing improves LIBERO-Long success from 92.8% to 98.4% while retaining the same expert-bank capacity.
- Token-budget sweeps, perturbation diagnostics, and physical folding and assembly experiments test several aspects of compact representations.
Critical weaknesses
- LIBERO-Plus task-ID mappings and adaptation logs are unavailable, leaving the reported generalization protocol incompletely verifiable.
- Cosine-error diagnostics do not establish that predicted future tokens causally improve actions; no action-only or future-token intervention control is reported.
- Compact token counts are not accompanied by measured latency or memory, and every expert is evaluated despite Top-2 weighting.
Questions for authors
- Can you release variant-to-task mappings and adaptation records, then verify frozen-checkpoint LIBERO-Plus results with original task IDs and a language-derived routing control?
- Can you compare against an action-only model with matched capacity and mask or shuffle predicted future tokens to quantify their contribution to success?
- Can you report batch-one latency, peak memory, and success across token budgets, including encoder, MUX, all expert evaluations, and joint denoising costs?
3 comments
Response to Reviewer ydz2(Q2 & W2)
W2 / Q2 — Do future tokens contribute to actions?
We thank the reviewer for this insightful question, which addresses the key connection between future-world prediction and action control. We add a training ablation and inference-time interventions to directly test the contribution of future-world tokens to action generation.
Experimental setup. For the training ablation, we retain world supervision and retrain MuxWAM while blocking action queries from reading the keys and values of future-world tokens during both training and inference. For the inference-time interventions, we freeze existing models and, during joint denoising, separately block this attention pathway, replace future tokens with current tokens, or reverse their temporal order. Each inference-time intervention is evaluated on all four LIBERO suites, with 500 episodes per suite.
Table R5. Contribution of future-world tokens to control. Success rates (%).
| Method / setting | Spatial | Object | Goal | Long | Avg. |
|---|---|---|---|---|---|
| MuxWAM (Full, paper Table 2) | 98.8 | 100.0 | 97.8 | 98.4 | 98.8 |
| Block future-world-to-action attention (training + inference) | 95.2 | 96.4 | 96.2 | 94.6 | 95.6 |
| Block future-world-to-action attention (inference only) | 0.2 | 0.0 | 3.2 | 0.0 | 0.85 |
| Replace future tokens with current tokens (inference only) | 48.8 | 86.6 | 71.8 | 57.0 | 66.05 |
| Reverse future-token temporal order (inference only) | 94.8 | 98.8 | 98.0 | 95.2 | 96.7 |
Results and analysis. Blocking future-world-to-action interaction during both training and inference reduces success across all four suites, from 98.8% to 95.6% on average. On the frozen models, blocking the pathway and replacing future tokens yield mean success rates of 0.85% and 66.05%, respectively, while temporal reversal achieves 96.7%. Blocking or replacing future-world information affects control more strongly than changing its temporal order.
Together, these results indicate that future-world tokens provide useful information for action generation during joint denoising, and the future-world-to-action pathway contributes to control performance.
Response to Reviewer ydz2(Q3 & W3)
W3 / Q3 — Batch-one latency, memory and token budgets
We appreciate this valuable suggestion from a deployment perspective, which helps provide a more complete assessment of the computational benefits of compact representations.
Table R6. New measurements: batch-one cost across K=1, 2, 4, 8, 64, 128 and 256. We profile all budgets on the same exclusive NVIDIA L20, with two 224×224 views, a ten-action chunk and ten Euler updates. K is the number of world tokens per view. The frozen Object-30K checkpoint trained with K=2 is reused throughout.
| K per view | Mean ms | P50 ms | P95 ms | Peak live JAX GiB | Latency / K=2 |
|---|---|---|---|---|---|
| 1 | 70.894 | 70.353 | 73.762 | 6.602 | 0.96× |
| 2 | 73.801 | 73.352 | 77.281 | 6.602 | 1.00× |
| 4 | 77.485 | 77.479 | 78.640 | 6.528 | 1.05× |
| 8 | 86.663 | 86.453 | 90.755 | 6.506 | 1.17× |
| 64 | 216.818 | 216.394 | 220.146 | 6.602 | 2.94× |
| 128 | 605.646 | 605.513 | 608.626 | 6.817 | 8.21× |
| 256 | 1807.382 | 1806.852 | 1813.922 | 8.196 | 24.49× |
At K=2, end-to-end inference is 24.49× faster and peak live memory is 19.4% lower than at K=256 in this sweep. These measurements quantify the deployment cost of the world-token budget with encoder and expert computation included.
Table R7. Paper Appendix Table 10 (p. 23): token-budget success (%), H=10.
| Suite | K=1 | K=2 | K=4 | K=8 |
|---|---|---|---|---|
| Spatial | 97.0 | 98.8 | 98.2 | 96.8 |
| Object | 98.6 | 100.0 | 99.0 | 98.0 |
| Goal | 97.8 | 97.8 | 97.0 | 97.2 |
| Long | 96.0 | 98.4 | 98.2 | 95.4 |
Table R8. Independent native K=2 repeat. Under the same deployment configuration, a separate run uses 100 warm-up calls and 500 measured calls.
| Checkpoint | Warm-up | Calls | Mean ms | P50 ms | P95 ms | Peak live JAX GiB |
|---|---|---|---|---|---|---|
| K=2; Object-30K | 100 | 500 | 73.105 | 72.819 | 75.357 | 6.602 |
Response to Reviewer ydz2(Q1 & W1)
Thank you for the constructive questions. We provide the task mappings, control ablations, and measured deployment costs below.
W1 / Q1 — LIBERO-Plus mapping, adaptation and language routing
We appreciate the reviewer's careful attention to reproducibility and the valuable suggestion to make the task mapping and routing protocol more explicit.
The table below lists all 40 original operations and their expected router indices from the corresponding training metadata. Variant counts sum to 10,030 mapping entries.
Table R1. LIBERO / LIBERO-Plus task-to-router mapping
Note: Each entry denotes LIBERO router → Plus router (number of variants). ID refers to the original task ID within each suite.
Reviewer-requested routing control with the original instruction held fixed: Camera and Background. We evaluate closed-loop control with language-derived routing under these two visual perturbation categories while keeping the original instruction unchanged.
We evaluate every Camera and Background variant in all four LIBERO-Plus suites, totaling 2,675 episodes. We use Plus-trained models, with one predefined initial state per variant. Both the language selector and action policy receive the unchanged original task instruction. The selector retrieves among the ten known task descriptions using cosine similarity of masked-mean vocabulary embeddings; the client supplies an invalid task-ID placeholder, and the server uses the retrieved router ID.
Table R2. New language-routing measurements: successful / total episodes and success rate.
The suite-equal mean success rates are 89.14% for Camera and 92.16% for Background.
Table R3. Paper Appendix Table 9 (p. 21): Plus-trained original-ID success rates (%). We reproduce the Camera and Background columns from the "Trained on LIBERO-Plus (In-domain Evaluation)" block.
Conclusion. On known tasks with unchanged original instructions, language-derived routing achieves suite-equal success rates of 89.14% on Camera and 92.16% on Background. This additional evaluation provides control evidence for MuxWAM's compact world representation under these visual perturbations.
Table R4. Supplementary evaluation of the additional control: original instructions on standard LIBERO-Long. We also evaluate the newly added language selector on all ten tasks with 50 initial states per task (500 episodes), using the original benchmark instructions and the frozen Long-38K checkpoint. Language retrieval selects among the ten known task descriptions using frozen policy vocabulary embeddings; the policy server receives an invalid task-ID placeholder and replaces it with the language-selected router. Evaluation uses 4 GPUs and 12 rollout processes, with no training.
The original-ID reference uses the 98.4% result in paper Table 2 (p. 7); the count of 492/500 is reported in Appendix A.6 (p. 24).