SemA-WAM: Hierarchical Semantic Distillation for Robust Action Generation

Leveraging spatiotemporal priors from pretrained video generators, World Action Models (WAMs) jointly forecast future observations and robot actions so that anticipated scene evolution can inform policy learning. Yet pixel-centric supervision typically overlooks high-level semantics needed for robust decision-making, limiting the semantic grounding of learned actions. Prior work either predicts action-relevant features from pretrained visual encoders as auxiliary targets or injects them as conditioning, leaving their effect on the core action representation largely indirect. Going beyond such indirect formulations, we propose , a hierarchical semantics-aware Mixture-of-Transformers that internalizes multi-source semantic knowledge in the action pathway. comprises (1) a multi-source semantic bank that fuses SigLIP2, semantic masks, and DINOv3 for action-relevant perception; (2) a hierarchical injection that couples dense mask structure with compressed foundation-model tokens, aligning fine spatial detail with high-level context; and (3) an action-centric distillation that transfers this hierarchy into a compact action pathway, discarding the mask stream and bank at inference. achieves competitive performance, reaching $98.3\%$ on standard LIBERO and $77.3\%$ on LIBERO-Plus, outperforming by $8.9$ points on LIBERO-Plus.