acceptodds
Under review as a conference paper at ICLR 2027

SemA-WAM: Hierarchical Semantic Distillation for Robust Action Generation

16 views

Abstract

Leveraging spatiotemporal priors from pretrained video generators, World Action Models (WAMs) jointly forecast future observations and robot actions so that anticipated scene evolution can inform policy learning. Yet pixel-centric supervision typically overlooks high-level semantics needed for robust decision-making, limiting the semantic grounding of learned actions. Prior work either predicts action-relevant features from pretrained visual encoders as auxiliary targets or injects them as conditioning, leaving their effect on the core action representation largely indirect. Going beyond such indirect formulations, we propose , a hierarchical semantics-aware Mixture-of-Transformers that internalizes multi-source semantic knowledge in the action pathway. comprises (1) a multi-source semantic bank that fuses SigLIP2, semantic masks, and DINOv3 for action-relevant perception; (2) a hierarchical injection that couples dense mask structure with compressed foundation-model tokens, aligning fine spatial detail with high-level context; and (3) an action-centric distillation that transfers this hierarchy into a compact action pathway, discarding the mask stream and bank at inference. achieves competitive performance, reaching on standard LIBERO and on LIBERO-Plus, outperforming by points on LIBERO-Plus.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, World Action Models, Semantic Future Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aN1LOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SemA-WAM, a world action model that uses training-time semantic supervision to improve robotic manipulation under distribution shifts. Building on Fast-WAM’s video–action architecture, it combines future semantic-mask prediction, gated mask-to-action attention, and a compact semantic bank integrating SigLIP2, DINOv3, and mask features. These semantic pathways are jointly trained with the action model and removed at inference. The authors report success rates of 98.3% on standard LIBERO and 77.3% on LIBERO-Plus, with the latter exceeding Fast-WAM-Joint by 8.9 percentage points. Real-world Franka experiments also report improved success over the evaluated Fast-WAM baselines. Component ablations, attention visualizations, and trajectory statistics provide additional analysis of the proposed design.

Critical strengths
  1. The method incorporates semantic cues during training and removes the auxiliary semantic pathways at inference, offering a practical design for robust control.
  2. SemA-WAM achieves 77.3% on LIBERO-Plus, outperforming FAST-WAM-JOINT by 8.9 percentage points, while maintaining 98.3% on standard LIBERO.
  3. Evaluation covers simulation, diverse perturbations, and real-world manipulation, supported by component-level ablations.
Critical weaknesses
  1. The overall approach follows semantic-mask future modeling in MWM and complementary visual-semantic feature fusion in methods such as SemanticVLA. Although the paper identifies concrete gated-coupling and semantic-routing designs, the current evidence does not sufficiently establish their distinctive methodological value beyond combining existing techniques.
  2. Tables 4 and 6 demonstrate benefits from the bank as a whole, but neither the individual semantic sources nor the specific Q/K/V routing are evaluated against simpler fusion alternatives. Moreover, the 1.0-percentage-point overall improvement on LIBERO-Plus lacks repeated-training results, leaving its stability unclear.
  3. The attribution to semantic internalization in the action pathway lacks direct evidence. Attention visualizations and aggregate trajectory statistics indicate behavioral changes, but do not specifically attribute the gains to semantic competence internalized within Action DiT. As described in Section 3.5, the shared Video DiT also receives semantic-training gradients, so improved visual representations remain an alternative explanation.
Questions for authors
  1. What distinctive mechanism or generalizable insight do the proposed gated coupling and semantic routing provide relative to MWM, SemanticVLA, and related work? Please clarify the incremental contribution beyond combining established techniques.
  2. Could the authors provide source-specific ablations and simpler fusion controls using the same features and comparable capacity, together with means and variability across independent training runs for the key comparisons?
  3. Could a control retaining the same semantic-prediction auxiliary training but removing direct mask/bank-to-action injection distinguish shared-backbone improvements from action-pathway injection? If this mechanism cannot be isolated, the semantic-internalization claims should be qualified accordingly.
6 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer aN1L [1/3]

We appreciate Reviewer aN1L for the careful assessment and constructive questions. We particularly appreciate the recognition that SEMA-WAM provides a practical training-only semantic design, improves LIBERO-Plus by 8.9 points over FAST-WAM-Joint while retaining 98.3% on standard LIBERO, and is evaluated across simulation, diverse perturbations, and real-world manipulation. Below, we clarify the methodological distinction, provide source-specific ablations and routing-level controls, and separate shared-backbone learning from direct semantic injection.

W1 & Q1: Methodological Distinction from MWM and SemanticVLA.

We appreciate the pioneering roles of MWM and SemanticVLA in semantic world modeling and visual–semantic fusion. SEMA-WAM addresses a different training–deployment question: whether semantic supervision can improve the action model during training without requiring auxiliary semantic modules at inference.

MWM retains its mask-dynamics backbone to provide predictive features during inference, while SemanticVLA retains its SigLIP–DINO pruning and fusion modules for online perception. In contrast, SEMA-WAM removes the mask stream, frozen semantic encoders, and semantic bank after training, leaving only the RGB–action backbone with language and proprioceptive inputs.

The distinctive contribution is therefore the complete training-only mechanism rather than any individual concatenation, attention, or slot operation. Joint future modeling supplies structured semantic supervision, asymmetric routing controls how semantic information enters the action model, and layer-wise gates adapt its influence across network depth. This supports a general training principle: auxiliary semantic pathways can shape a robust action representation without becoming a permanent deployment dependency.

We clarified this methodological positioning and deployment distinction in the revised manuscript (Introduction, Lines 46–107; Related Work, Lines 108–135).

AuthorsOpenReview ↗

Rebuttal for Reviewer aN1L [2/3]

W2 & Q2: Semantic-Source Ablations, Fusion Controls, and Training Variability.

Under the same training budget, semantic-routing structure, and complete 10,030-episode LIBERO-Plus evaluation protocol, DINOv3-only achieves 75.0% and SigLIP2-only achieves 76.5%, compared with 77.3% for the complete model. Both source-specific variants retain the same mid-depth mask features as keys. The labels DINOv3-only and SigLIP2-only therefore specify the foundation-model source used for the queries and values, rather than removal of the mask keys.

DINOv3-only is stronger on Robot Initial States, reflecting its detailed visual representation, while SigLIP2-only is stronger on Light Conditions and Language Instructions, reflecting its vision–language semantic alignment. Combining the two sources produces the strongest overall LIBERO-Plus result and improves Camera Viewpoints, Objects Layout, Background Textures, and Language Instructions.

VariantCameraRobotLayoutLightBGLang.NoiseAvg.
Channel-Concat40.452.370.484.261.973.052.061.0
mask tokens w/o α24.558.469.482.060.075.247.058.3
w/o Semantic Bank48.981.581.692.677.394.763.776.3
DINOv3 only49.384.179.593.071.790.862.175.0
SigLIP2 only50.382.080.697.273.693.564.676.5
SEMA-WAM (ours)55.081.081.197.273.894.863.877.3

The standard LIBERO results show the same complementarity. DINOv3-only is stronger on Spatial and Object, while the complete model achieves the best results on Goal and LIBERO-10 and reaches the highest overall success rate among these variants.

VariantSpatialObjectGoalLIBERO-10Overall
DINOv3 only98.6099.6090.0098.0096.55
SigLIP2 only98.2099.2095.4096.4097.30
SEMA-WAM (ours)98.0099.2097.2098.4098.30

The existing controls further evaluate the routing and coupling design. Channel-Concat reaches 61.0% on LIBERO-Plus, while direct mask-token injection without the learnable gate α reaches 58.3%. These results show that early RGB–mask fusion and ungated mask injection are less effective than the complete structured design under the evaluated settings.

Together, these completed experiments evaluate two key aspects of the design: the complementarity of the semantic sources and the routing and coupling strategy. Due to the limited rebuttal time, we were unable to complete the additional training required for a strictly feature- and capacity-matched fusion control using the same SigLIP2 and DINOv3 features or for independent multi-seed runs. The reported comparisons are therefore based on single training runs, with each checkpoint evaluated on the complete 10,030-episode LIBERO-Plus benchmark. The source-specific ablations and routing controls provide direct evidence for the complementary semantic sources and the benefit of the gated coupling design.

We added the source-specific analysis and clarified the existing routing controls in the revised manuscript (Section 4, Lines 392–400 and 432–443; Appendix A, Lines 646–688).

AuthorsOpenReview ↗

Rebuttal for Reviewer aN1L [3/3]

W3 & Q3: Shared-Backbone Learning versus Direct Semantic Injection.

To isolate the two effects, we conducted a controlled ablation using the same data, training budget, and future-mask prediction objective as the complete model. This control retains the mask stream and shared Video DiT during training, but fixes every mask-to-action gate to α=0 and disables the semantic bank. Semantic supervision can therefore affect the policy only through shared-backbone learning. Both settings use the same RGB-only deployment protocol, without masks or semantic-bank features at inference.

On the complete LIBERO-Plus benchmark, the control without direct injection achieves 77.0%, improving over the 49.9% FAST-WAM baseline by 27.1 percentage points. This result shows that semantic auxiliary training substantially strengthens the shared backbone and that these benefits are retained by the deployed RGB-only policy. The complete model further reaches 77.3% overall.

SettingCameraRobotLayoutLightBGLang.NoiseAvg.
w/o Direct Injection52.182.181.897.370.793.266.977.0
Full SEMA-WAM55.081.081.197.273.894.863.877.3

The direct semantic pathway provides additional gains on the shifts most dependent on preserving object and instruction semantics: +2.9 points on Camera Viewpoints, +3.1 points on Background Textures, and +1.6 points on Language Instructions. Across the corresponding 4,212 episodes, the weighted success rate increases from 71.8% to 74.3% (+2.5 points).

Distribution shiftw/o Direct InjectionFullΔ
Camera Viewpoints52.155.0+2.9
Background Textures70.773.8+3.1
Language Instructions93.294.8+1.6

These results separate two complementary effects. Shared-backbone semantic learning accounts for the broad performance improvement, while direct mask/bank-to-action injection provides additional robustness on semantic-sensitive distribution shifts. Accordingly, semantic internalization refers to the semantic capability retained by the RGB-only deployment model through shared-backbone auxiliary learning, while direct semantic injection serves as a complementary training-time mechanism for targeted robustness.

Reviewer_BrW7OpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SEMA-WAM, a World Action Model built on the Mixture-of-Transformers architecture of FAST-WAM to internalize semantic representations into action generation without adding inference overhead. During training, it co-predicts RGB and semantic mask futures in Video DiT via sequence-level concatenation, couples mask tokens to Action DiT through a gated residual branch, and injects a multi-source semantic bank (fusing SigLIP2, DINOv3, and mid-depth mask features) into the action context. At test time, both the mask stream and the semantic bank are discarded. Experiments on LIBERO, LIBERO-Plus, and a real-world Franka robot demonstrate improved robustness under visual distribution shifts.

Critical strengths
  • Using the mask stream and semantic bank strictly as training-only privileged pathways allows the model to improve semantic grounding while keeping the exact inference speed and architecture of the RGB-only baseline at deployment.
  • Tokenizing RGB and masks separately with the frozen Wan VAE and coupling them via a learnable residual gate preserves pretrained video priors, outperforming early channel concatenation and ungated mask injection in the ablation study.
  • The empirical gains on the zero-shot LIBERO-Plus benchmark over the joint RGB baseline are supported by real-robot evaluations, attention visualizations, and closed-loop trajectory metrics.
Critical weaknesses
  • The methodological novelty is somewhat incremental, as the framework closely follows FAST-WAM and recent privileged or mask-based WAMs by combining sequence concatenation, gated cross-attention, and Perceiver-style slot attention over frozen foundation models. Framing the method as "semantic distillation" is also inaccurate since the training objective simply sums standard flow-matching losses without any teacher-student distillation loss, relying purely on multi-task co-training.
  • The multi-source semantic bank is overly complex (requiring frozen SigLIP2, DINOv3, and multi-stage cross-attention routing) relative to its marginal empirical benefit. Removing the entire semantic bank leaves standard LIBERO performance unchanged and yields only a very minor difference on LIBERO-Plus, while even slightly degrading performance on several perturbation axes such as background, robot appearance, and layout. Fine-grained ablations isolating SigLIP2, DINOv3, and the mid-depth mask keys are also missing.
  • The inference protocol is inconsistent between the main text and the appendix, and risks a train-test distribution mismatch. While the main text repeatedly states that the entire mask stream and semantic bank are discarded at test time, the appendix states that the first-frame mask is forced to zero during evaluation, implying the mask stream is still executed under a zero condition. Moreover, neither the semantic bank nor the mask residual branch uses dropout during training, so abruptly removing them at test time introduces an unaddressed representation shift in Action DiT.
Questions for authors
  • At inference, is the Video DiT mask stream completely skipped, or is it still executed with a zeroed first-frame mask condition as described in the appendix? Without training-time dropout on the semantic bank and the residual gate, how does Action DiT avoid representation shift when they are removed at test time, and what do the learned gate values look like across layers?
  • Given the minimal performance change on LIBERO-Plus when removing the semantic bank, can the authors provide ablations for individual components inside the bank (SigLIP2, DINOv3, and mid-depth mask keys) and report the extra training memory and time overhead?
  • In the semantic bank routing, how does linear resampling preserve spatial alignment between frame-level DINOv3 tokens and spatiotemporal mask latents, and how is the horizontal view splitting described in the method adapted to the multi-view camera mosaic used in the Franka experiments?
5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer BrW7 [1/3]

We appreciate Reviewer BrW7 for the careful assessment and constructive questions. We particularly appreciate the recognition that SEMA-WAM uses semantic pathways only during training while preserving an efficient RGB-only deployment model, that its gated coupling outperforms simpler fusion strategies, and that its robustness gains are supported across simulation and real-world evaluations. Below, we clarify the central training–deployment principle, provide component-level evidence, and explain the inference and routing protocols.

W1: Methodological Distinction and Scope of Semantic Distillation.

SEMA-WAM uses auxiliary semantic information during training to strengthen the action model, while removing all semantic modules at deployment. The deployed model therefore uses only RGB observations, language instructions, and proprioception.

We have narrowed and explicitly distinguished the related terminology in the revised manuscript. We use semantic internalization only to describe the observed persistence of training-time semantic benefits in the RGB-conditioned action pathway after the auxiliary mask stream and semantic bank are removed. We use action-centric privilege distillation to describe this training–deployment separation, rather than a classical knowledge-distillation objective based on logit matching, feature matching, or KL divergence.

Our central contribution is therefore a training-only semantic internalization framework that strengthens the RGB–action model without retaining any auxiliary semantic pathways at inference.

We clarified the training–deployment mechanism and terminology in the revised manuscript (Lines 17–25 and 76–94).

AuthorsOpenReview ↗

Rebuttal for Reviewer BrW7 [2/3]

W2 & Q2: Semantic-Source Ablations and Training Overhead.

To isolate the two foundation-model sources, we conducted DINOv3-only and SigLIP2-only ablations using the same training data, training budget, semantic-routing structure, and complete 10,030-episode LIBERO-Plus evaluation protocol. Both ablations retain the same mid-depth mask features as keys, so the measured differences isolate DINOv3 and SigLIP2.

DINOv3-only achieves 75.0% on LIBERO-Plus and provides stronger performance on Robot Initial States, reflecting its detailed visual representation. SigLIP2-only reaches 76.5% and performs better on Light Conditions and Language Instructions, demonstrating the benefit of vision–language semantic alignment. Combining both sources enables the complete SEMA-WAM model to achieve the strongest overall result of 77.3%, with further improvements on Camera Viewpoints, Objects Layout, Background Textures, and Language Instructions.

VariantCameraRobotLayoutLightBGLang.NoiseAvg.
DINOv3 only49.384.179.593.071.790.862.175.0
SigLIP2 only50.382.080.697.273.693.564.676.5
SEMA-WAM (ours)55.081.081.197.273.894.863.877.3

The standard LIBERO results show the same complementarity. DINOv3-only is stronger on Spatial and Object, while the complete SEMA-WAM model achieves the best results on Goal and LIBERO-10 and reaches the highest overall success rate of 98.3%. Together, these results show that combining detailed visual representations with language-aligned semantic guidance produces the strongest overall performance.

VariantSpatialObjectGoalLIBERO-10Overall
DINOv3 only98.6099.6090.0098.0096.55
SigLIP2 only98.2099.2095.4096.4097.30
SEMA-WAM (ours)98.0099.2097.2098.4098.30

The overall average does not fully capture the semantic bank's targeted contribution. Relative to removing the semantic bank, the complete model improves Camera Viewpoints from 48.9% to 55.0% (+6.1 points) and Light Conditions from 92.6% to 97.2% (+4.6 points). These gains support its intended role in preserving task-relevant information under substantial visual changes.

We further conducted matched training-cost profiling with the semantic bank enabled and disabled, while keeping the data, random seed, base model, mask stream and loss, and all other settings fixed. The experiment used four NVIDIA A100 80GB GPUs with bf16, batch size 2 per GPU, and gradient accumulation of 16. After two warm-up updates, we averaged 64 complete optimizer updates for each setting, corresponding to 1,024 measured forward–backward iterations. Enabling the bank increased peak allocated memory per GPU from 50.54 to 51.53 GiB (+0.98 GiB, +1.95%) and mean optimizer-update time from 10.766 ± 0.135 to 11.930 ± 0.159 s (+1.163 s, +10.80%). The memory measurement covers the forward pass, backward pass, optimizer step, and optimizer state. Because the semantic bank is removed after training, it introduces no inference-memory or inference-time overhead.

Due to the limited rebuttal time, we were unable to complete the additional training required for an independent ablation of the mid-depth mask keys. The two completed ablations retain identical mask keys and therefore specifically isolate the respective contributions of DINOv3 and SigLIP2.

We added the source-specific analysis and ablation definitions to the revised manuscript (Section 4, Lines 392–400; Appendix A, Lines 649–688).

AuthorsOpenReview ↗

Rebuttal for Reviewer BrW7 [3/3]

W3 & Q1: Inference Protocol, Representation Shift, and Learned Gates.

The appendix statement that the first-frame mask is set to zero was a wording error and does not describe the actual evaluation protocol. At inference, the Video DiT mask stream is completely skipped, the mask residual branch is not executed, and the semantic bank and frozen semantic encoders are removed. No mask input or semantic-bank feature is processed by the deployed model. All reported evaluation results were obtained under this RGB-only deployment setting.

During training, dropout is applied only to the first-frame mask condition with probability 0.5. This condition-level dropout does not remove the mask pathway: future-mask prediction and the subsequent mask tokens remain active during training. The semantic bank and mask residual branch are not independently dropped as complete modules. Nevertheless, the reported evaluation results, all obtained after fully removing these pathways, do not show a detrimental performance shift. The gate values are therefore reported only to analyze how the semantic pathway shapes different network depths during training; they do not participate in action prediction at inference.

At the final training checkpoint, the gates range from −0.1279 to 0.2969, with an overall mean of −0.0084. Their mean increases from −0.0910 in layers 1–10 to −0.0302 in layers 11–20 and 0.0959 in layers 21–30. This bounded depth-wise transition indicates that the semantic pathway primarily provides corrective modulation in early layers and increasingly positive semantic enhancement in deeper layers.

LayerαLayerαLayerα
1−0.004911−0.103521−0.0294
2−0.048312−0.0688220.0688
3−0.080113−0.084023−0.0942
4−0.096214−0.0767240.1069
5−0.106415−0.055425−0.0041
6−0.095216−0.0305260.1982
7−0.127917−0.0337270.2637
8−0.115718−0.0447280.2969
9−0.1270190.0159290.0215
10−0.1079200.1797300.1309

We corrected and unified the inference protocol and added the depth-wise gate analysis in the revised manuscript (Section 3.2, Lines 202–211; Section 4, Lines 406–413 and 432–443; Appendix A, Lines 660–670, 690–691, and 714–717).

Q3: Spatial Alignment in Semantic Routing and Multi-View Adaptation.

Linear resampling is used as a deterministic sequence-length matching operation rather than to enforce exact patch-to-patch geometric alignment. DINOv3 tokens are already contextualized by 12 ViT layers, while the mask keys are extracted after 15 DiT layers of self-attention and text cross-attention. Both therefore represent contextual semantic regions rather than isolated local patches.

The router aggregates these features into 16 global semantic slots for the action context instead of producing a dense mask or pixel-level prediction. It consequently requires stable coarse correspondence rather than exact local alignment. Precise spatial structure is maintained by the main RGB–mask backbone, where both streams use the same canvas, frozen VAE, and DiT patch grid. The resampling rule remains fixed throughout training and introduces no stochastic token permutation.

For the real-world Franka setting, we recover the three regions of the 320×384 RobotWin camera mosaic rather than applying equal-width horizontal splitting: a 96×384 wrist-camera view at the top and two 224×192 third-person views at the bottom left and right. Each view is encoded independently, and the resulting tokens are concatenated in a fixed camera order. Equal-width horizontal splitting is used only for the two-camera layout.

We clarified the semantic-bank construction and real-robot camera routing in the revised manuscript (Section 3.4, Lines 237–269; Appendix A, Lines 693–699).

Reviewer_L4y8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to augment the World Action Model with an additional mask generation objective, and to improve the action generation with a mixed semantic bank of SigLIP2 and DINOv3 features. All these privileged pathways are discarded during inference.

Critical strengths

The authors honestly demonstrate the limitations of their approach: it is not the top‑performing method on standard LIBERO (Tab.1). They do not deliberately omit stronger baselines.

Critical weaknesses
  1. In Fig. 2, the VAE is illustrated as trainable, while it is described to be frozen in Eq. 1 (line 190).
  2. The necessity of building the multi-source semantic bank in Fig. 2 is unclear. Maybe it is just sufficient to use either SigLip2 or DINOv3. Additionally, this sophesticated design only leads to marginal performance improvement (Tab.4 and Tab.6), raising questions about the necessity of such design.
  3. The real-world experiment is superficial and does not validate that the proposed method can improve generalization across spatial and environmental conditions, instead, they merely offer basic performance comparisons. I would recommend a better setup including additional environmental complexity during test time via visual distractors, varied table cloths, and changing lighting.
Questions for authors
  1. What is the data distribution of the learnable per-layer gate αl\alpha_{l} in Eq. 7?
  2. How to assign attention positions for hdualh^{dual} in Eq. 3? The RGB latent is always located at 2n2n temporarily while the semantic frame is always located at the 2n+12n + 1?
5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer L4y8 [1/3]

We appreciate Reviewer L4y8 for the careful assessment and constructive questions. The comments center on the clarity of several methodological details, the empirical support for the multi-source semantic mechanism, and real-world generalization. Below, we address these points with precise clarifications, new ablations, and expanded evaluations.

W1 & Q2: VAE Training Status and Dual-Stream Token Positions.

We appreciate the reviewer for identifying these two points that require clearer presentation. First, the trainable indicator attached to the VAE in Fig. 2 was an annotation error. The VAE remains frozen throughout training, consistent with Eq. 1, and Fig. 2 has been corrected accordingly. This correction appears in Fig. 2 and Lines 179–183 of the revised manuscript.

Second, Eq. 3 uses block-wise rather than alternating concatenation. All RGB tokens are placed first, followed by all semantic tokens. A corresponding RGB–semantic pair therefore has sequence indices nn and N+nN+n, rather than 2n2n and 2n+12n+1. The paired tokens share the same local 3D RoPE coordinates, preserving their temporal and spatial alignment. The two modalities are distinguished through their fixed sequence blocks, input content, asymmetric attention routing, and separate prediction heads. We clarified the token ordering and shared RoPE coordinates in Section 3.2, Lines 187–198 of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal for Reviewer L4y8 [2/3]

W2 & Q1: Semantic-Source Contributions and Layer-Wise Gate Values.

We appreciate the reviewer's focus on both the necessity of the multi-source semantic bank and the behavior of its layer-wise coupling.

Complementary semantic sources. To isolate the contribution of each semantic source, we conducted DINOv3-only and SigLIP2-only ablations using the same training data, training budget, semantic-routing structure, and complete 10,030-episode LIBERO-Plus evaluation protocol. Both settings retain the same mid-depth mask features as keys, thereby isolating the two foundation-model sources.

The DINOv3-only model achieves 75.0% on LIBERO-Plus and performs better on Robot Initial States, reflecting the value of its detailed visual representation. The SigLIP2-only model reaches 76.5% and performs better on Light Conditions and Language Instructions, demonstrating the benefit of vision–language semantic alignment. Combining the two sources enables the complete SemA-WAM model to reach the strongest overall result of 77.3%. These results support the semantic bank design: DINOv3 contributes detailed visual information, while SigLIP2 supplies language-aligned semantic guidance.

Although the overall gain over removing the semantic bank is 1.0 point, its contribution is concentrated on challenging semantic-sensitive shifts: it improves Camera Viewpoints by 6.1 points and Light Conditions by 4.6 points. This indicates that the bank primarily provides targeted robustness rather than uniform gains across all perturbation categories.

VariantCameraRobotLayoutLightBGLang.NoiseAvg.
DINOv3 only49.384.179.593.071.790.862.175.0
SigLIP2 only50.382.080.697.273.693.564.676.5
SEMA-WAM (ours)55.081.081.197.273.894.863.877.3

The standard LIBERO results further demonstrate this complementarity. DINOv3-only performs better on Spatial and Object, while the complete SEMA-WAM model achieves the strongest results on Goal and LIBERO-10 and the highest overall success rate of 98.3%. Together, the two benchmarks show that the two semantic sources provide complementary information and that their combination produces the strongest overall performance.

VariantSpatialObjectGoalLIBERO-10Overall
DINOv3 only98.6099.6090.0098.0096.55
SigLIP2 only98.2099.2095.4096.4097.30
SEMA-WAM (ours)98.0099.2097.2098.4098.30

Depth-wise gate behavior. The learned gates further reveal how semantic information is modulated across network depth. At the final training checkpoint, the gate values range from −0.1279 to 0.2969, with 21 negative and 9 positive values. Their mean increases from −0.0910 in layers 1–10 to −0.0302 in layers 11–20 and 0.0959 in layers 21–30. This transition indicates that semantic features primarily provide corrective modulation in early layers and increasingly enhance high-level semantics in deeper layers.

These values characterize the training-time semantic coupling. At inference, the mask residual branch is completely skipped, and the learned gates do not participate in action prediction.

LayerαLayerαLayerα
1−0.004911−0.103521−0.0294
2−0.048312−0.0688220.0688
3−0.080113−0.084023−0.0942
4−0.096214−0.0767240.1069
5−0.106415−0.055425−0.0041
6−0.095216−0.0305260.1982
7−0.127917−0.0337270.2637
8−0.115718−0.0447280.2969
9−0.1270190.0159290.0215
10−0.1079200.1797300.1309

We added the source-specific ablations and depth-wise gate analysis to the revised manuscript (Section 4, Lines 392–400 and 406–413; Appendix A, Lines 649–691).

AuthorsOpenReview ↗

Rebuttal for Reviewer L4y8 [3/3]

W3: Real-World Generalization under OOD Perturbations.

We appreciate the reviewer's emphasis on evaluating the model under more challenging real-world conditions. We added OOD evaluations covering background, lighting, and initial-state shifts. For each Franka task, we conducted 50 trials: 20 trials varied the background color or material, 20 trials changed the lighting conditions, and 10 trials modified the initial scene state. The task instructions and success criteria remained unchanged across all conditions.

Across the three tasks, SemA-WAM achieves the highest aggregate OOD success counts: 7/50 for Stack Blocks, 16/50 for Open/Close Drawer, and 25/50 for Pick and Place. These results extend the original evaluation from standard task settings to controlled spatial and environmental distribution shifts.

MethodID Stack BlocksID Open/Close DrawerID Pick and PlaceOOD Stack BlocksOOD Open/Close DrawerOOD Pick and Place
FAST-WAM6/5028/5021/502/506/5016/50
FAST-WAM-Joint13/5024/5024/505/509/5020/50
SemA-WAM (ours)17/5035/5032/507/5016/5025/50

The detailed breakdown below shows that SemA-WAM achieves the strongest or tied-strongest result in eight of the nine OOD settings and the second-best result in the remaining setting. Its gains therefore extend across background, lighting, and initial-state changes rather than arising from a single perturbation type.

MethodStack BGStack LightStack InitialDrawer BGDrawer LightDrawer InitialPick BGPick LightPick Initial
FAST-WAM1/200/201/101/204/201/103/207/206/10
FAST-WAM-Joint0/201/204/104/203/202/105/2010/205/10
SemA-WAM (ours)1/203/203/106/208/202/105/2013/207/10

Together, these real-world results validate the central design principle of SemA-WAM: training-time semantic supervision improves the robustness of the deployed RGB-only policy to environmental and initial-state disturbances without retaining auxiliary semantic pathways at inference.

We added the real-robot OOD protocol and results to the revised manuscript (Section 4, Lines 401–404; Appendix A, Lines 703–732).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.