- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents SemA-WAM, a world action model that uses training-time semantic supervision to improve robotic manipulation under distribution shifts. Building on Fast-WAM’s video–action architecture, it combines future semantic-mask prediction, gated mask-to-action attention, and a compact semantic bank integrating SigLIP2, DINOv3, and mask features. These semantic pathways are jointly trained with the action model and removed at inference. The authors report success rates of 98.3% on standard LIBERO and 77.3% on LIBERO-Plus, with the latter exceeding Fast-WAM-Joint by 8.9 percentage points. Real-world Franka experiments also report improved success over the evaluated Fast-WAM baselines. Component ablations, attention visualizations, and trajectory statistics provide additional analysis of the proposed design.
Critical strengths
- The method incorporates semantic cues during training and removes the auxiliary semantic pathways at inference, offering a practical design for robust control.
- SemA-WAM achieves 77.3% on LIBERO-Plus, outperforming FAST-WAM-JOINT by 8.9 percentage points, while maintaining 98.3% on standard LIBERO.
- Evaluation covers simulation, diverse perturbations, and real-world manipulation, supported by component-level ablations.
Critical weaknesses
- The overall approach follows semantic-mask future modeling in MWM and complementary visual-semantic feature fusion in methods such as SemanticVLA. Although the paper identifies concrete gated-coupling and semantic-routing designs, the current evidence does not sufficiently establish their distinctive methodological value beyond combining existing techniques.
- Tables 4 and 6 demonstrate benefits from the bank as a whole, but neither the individual semantic sources nor the specific Q/K/V routing are evaluated against simpler fusion alternatives. Moreover, the 1.0-percentage-point overall improvement on LIBERO-Plus lacks repeated-training results, leaving its stability unclear.
- The attribution to semantic internalization in the action pathway lacks direct evidence. Attention visualizations and aggregate trajectory statistics indicate behavioral changes, but do not specifically attribute the gains to semantic competence internalized within Action DiT. As described in Section 3.5, the shared Video DiT also receives semantic-training gradients, so improved visual representations remain an alternative explanation.
Questions for authors
- What distinctive mechanism or generalizable insight do the proposed gated coupling and semantic routing provide relative to MWM, SemanticVLA, and related work? Please clarify the incremental contribution beyond combining established techniques.
- Could the authors provide source-specific ablations and simpler fusion controls using the same features and comparable capacity, together with means and variability across independent training runs for the key comparisons?
- Could a control retaining the same semantic-prediction auxiliary training but removing direct mask/bank-to-action injection distinguish shared-backbone improvements from action-pathway injection? If this mechanism cannot be isolated, the semantic-internalization claims should be qualified accordingly.
6 comments
Rebuttal for Reviewer aN1L [2/3]
W2 & Q2: Semantic-Source Ablations, Fusion Controls, and Training Variability.
Under the same training budget, semantic-routing structure, and complete 10,030-episode LIBERO-Plus evaluation protocol, DINOv3-only achieves 75.0% and SigLIP2-only achieves 76.5%, compared with 77.3% for the complete model. Both source-specific variants retain the same mid-depth mask features as keys. The labels DINOv3-only and SigLIP2-only therefore specify the foundation-model source used for the queries and values, rather than removal of the mask keys.
DINOv3-only is stronger on Robot Initial States, reflecting its detailed visual representation, while SigLIP2-only is stronger on Light Conditions and Language Instructions, reflecting its vision–language semantic alignment. Combining the two sources produces the strongest overall LIBERO-Plus result and improves Camera Viewpoints, Objects Layout, Background Textures, and Language Instructions.
| Variant | Camera | Robot | Layout | Light | BG | Lang. | Noise | Avg. |
|---|---|---|---|---|---|---|---|---|
| Channel-Concat | 40.4 | 52.3 | 70.4 | 84.2 | 61.9 | 73.0 | 52.0 | 61.0 |
| mask tokens w/o α | 24.5 | 58.4 | 69.4 | 82.0 | 60.0 | 75.2 | 47.0 | 58.3 |
| w/o Semantic Bank | 48.9 | 81.5 | 81.6 | 92.6 | 77.3 | 94.7 | 63.7 | 76.3 |
| DINOv3 only | 49.3 | 84.1 | 79.5 | 93.0 | 71.7 | 90.8 | 62.1 | 75.0 |
| SigLIP2 only | 50.3 | 82.0 | 80.6 | 97.2 | 73.6 | 93.5 | 64.6 | 76.5 |
| SEMA-WAM (ours) | 55.0 | 81.0 | 81.1 | 97.2 | 73.8 | 94.8 | 63.8 | 77.3 |
The standard LIBERO results show the same complementarity. DINOv3-only is stronger on Spatial and Object, while the complete model achieves the best results on Goal and LIBERO-10 and reaches the highest overall success rate among these variants.
| Variant | Spatial | Object | Goal | LIBERO-10 | Overall |
|---|---|---|---|---|---|
| DINOv3 only | 98.60 | 99.60 | 90.00 | 98.00 | 96.55 |
| SigLIP2 only | 98.20 | 99.20 | 95.40 | 96.40 | 97.30 |
| SEMA-WAM (ours) | 98.00 | 99.20 | 97.20 | 98.40 | 98.30 |
The existing controls further evaluate the routing and coupling design. Channel-Concat reaches 61.0% on LIBERO-Plus, while direct mask-token injection without the learnable gate α reaches 58.3%. These results show that early RGB–mask fusion and ungated mask injection are less effective than the complete structured design under the evaluated settings.
Together, these completed experiments evaluate two key aspects of the design: the complementarity of the semantic sources and the routing and coupling strategy. Due to the limited rebuttal time, we were unable to complete the additional training required for a strictly feature- and capacity-matched fusion control using the same SigLIP2 and DINOv3 features or for independent multi-seed runs. The reported comparisons are therefore based on single training runs, with each checkpoint evaluated on the complete 10,030-episode LIBERO-Plus benchmark. The source-specific ablations and routing controls provide direct evidence for the complementary semantic sources and the benefit of the gated coupling design.
We added the source-specific analysis and clarified the existing routing controls in the revised manuscript (Section 4, Lines 392–400 and 432–443; Appendix A, Lines 646–688).
Rebuttal for Reviewer aN1L [3/3]
W3 & Q3: Shared-Backbone Learning versus Direct Semantic Injection.
To isolate the two effects, we conducted a controlled ablation using the same data, training budget, and future-mask prediction objective as the complete model. This control retains the mask stream and shared Video DiT during training, but fixes every mask-to-action gate to α=0 and disables the semantic bank. Semantic supervision can therefore affect the policy only through shared-backbone learning. Both settings use the same RGB-only deployment protocol, without masks or semantic-bank features at inference.
On the complete LIBERO-Plus benchmark, the control without direct injection achieves 77.0%, improving over the 49.9% FAST-WAM baseline by 27.1 percentage points. This result shows that semantic auxiliary training substantially strengthens the shared backbone and that these benefits are retained by the deployed RGB-only policy. The complete model further reaches 77.3% overall.
| Setting | Camera | Robot | Layout | Light | BG | Lang. | Noise | Avg. |
|---|---|---|---|---|---|---|---|---|
| w/o Direct Injection | 52.1 | 82.1 | 81.8 | 97.3 | 70.7 | 93.2 | 66.9 | 77.0 |
| Full SEMA-WAM | 55.0 | 81.0 | 81.1 | 97.2 | 73.8 | 94.8 | 63.8 | 77.3 |
The direct semantic pathway provides additional gains on the shifts most dependent on preserving object and instruction semantics: +2.9 points on Camera Viewpoints, +3.1 points on Background Textures, and +1.6 points on Language Instructions. Across the corresponding 4,212 episodes, the weighted success rate increases from 71.8% to 74.3% (+2.5 points).
| Distribution shift | w/o Direct Injection | Full | Δ |
|---|---|---|---|
| Camera Viewpoints | 52.1 | 55.0 | +2.9 |
| Background Textures | 70.7 | 73.8 | +3.1 |
| Language Instructions | 93.2 | 94.8 | +1.6 |
These results separate two complementary effects. Shared-backbone semantic learning accounts for the broad performance improvement, while direct mask/bank-to-action injection provides additional robustness on semantic-sensitive distribution shifts. Accordingly, semantic internalization refers to the semantic capability retained by the RGB-only deployment model through shared-backbone auxiliary learning, while direct semantic injection serves as a complementary training-time mechanism for targeted robustness.
Rebuttal for Reviewer aN1L [1/3]
We appreciate Reviewer aN1L for the careful assessment and constructive questions. We particularly appreciate the recognition that SEMA-WAM provides a practical training-only semantic design, improves LIBERO-Plus by 8.9 points over FAST-WAM-Joint while retaining 98.3% on standard LIBERO, and is evaluated across simulation, diverse perturbations, and real-world manipulation. Below, we clarify the methodological distinction, provide source-specific ablations and routing-level controls, and separate shared-backbone learning from direct semantic injection.
W1 & Q1: Methodological Distinction from MWM and SemanticVLA.
We appreciate the pioneering roles of MWM and SemanticVLA in semantic world modeling and visual–semantic fusion. SEMA-WAM addresses a different training–deployment question: whether semantic supervision can improve the action model during training without requiring auxiliary semantic modules at inference.
MWM retains its mask-dynamics backbone to provide predictive features during inference, while SemanticVLA retains its SigLIP–DINO pruning and fusion modules for online perception. In contrast, SEMA-WAM removes the mask stream, frozen semantic encoders, and semantic bank after training, leaving only the RGB–action backbone with language and proprioceptive inputs.
The distinctive contribution is therefore the complete training-only mechanism rather than any individual concatenation, attention, or slot operation. Joint future modeling supplies structured semantic supervision, asymmetric routing controls how semantic information enters the action model, and layer-wise gates adapt its influence across network depth. This supports a general training principle: auxiliary semantic pathways can shape a robust action representation without becoming a permanent deployment dependency.
We clarified this methodological positioning and deployment distinction in the revised manuscript (Introduction, Lines 46–107; Related Work, Lines 108–135).