acceptodds
Withdrawn from ICLR 2027

CSWAM: Better Causal Semantic Representations for Out-of-Distribution Generalization in World Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

FastWAM-style world action models enable efficient action-only inference, but generalize poorly under visual distribution shifts. Their reconstruction-oriented representations emphasize appearance-specific details, limiting generalization to unseen scenes and objects. Without observation history, the model also lacks temporal evidence for robustly identifying task-relevant state changes and motion in unfamiliar visual conditions. To address these limitations, we present the Causal Semantic World Action Model (CSWAM), which augments FastWAM with a causal semantic expert built on V-JEPA 2.1. V-JEPA provides temporally grounded representations of semantic state changes and motion with less dependence on appearance-specific details. The expert learns their future evolution from a sparse history of current and past observations and shares the history-derived context with both the video and action streams through causal attention. At inference, CSWAM conditions action denoising on the current video state and observed semantic history, retaining efficient action-only inference. We conduct simulation and real-robot experiments to evaluate generalization under distribution shifts. With embodied pretraining, CSWAM raises Randomized success on RoboTwin 2.0 Clean-to-Randomized transfer from 10.16% to 45.18%, a gain of 35.02 percentage points over FastWAM. Across two real-robot tasks and three OOD difficulty levels, CSWAM improves average success over FastWAM by 42.5 percentage points, from 27.5% to 70.0%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action model, vjpea, out of distribution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_duapOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem that FastWAM-style world action models generalize poorly under visual distribution shifts. The authors argue that FastWAM struggles with unseen scenes and objects for two reasons: its reconstruction-oriented representations emphasize appearance-specific details, and it lacks observation history, which deprives it of temporal evidence.

To address this, they propose the Causal Semantic World Action Model (CSWAM), which adds a causal semantic expert built on a frozen V-JEPA 2.1 encoder to the existing FastWAM architecture. The expert predicts future semantic features from a sparse history of observations, aiming to make the model less dependent on appearance-specific details.

At inference, action denoising is conditioned on both the current video state and the observed semantic history, thereby retaining action-only inference.

The method is evaluated on RoboTwin 2.0 under both a mixed-training protocol and a Clean-to-Randomized transfer protocol, as well as on two real-robot tasks across three OOD difficulty levels, with ablations on the video expert, the observation history, and the semantic encoder.

The authors report that, with embodied pretraining, CSWAM improves Clean-to-Randomized success over FastWAM from 10.16% to 45.18%, and that it raises average real-robot success in the Hard and Extreme settings from 10% to 55%.

Critical strengths

S1. The Clean-to-Randomized evaluation protocol is stricter than the common mixed-training protocol.This setting measures genuine transfer, making it the paper’s most convincing evidence for its OOD generalization claim. Evidence: §4.1 and Table 3.

S2. CSWAM outperforms FastWAM in every real-robot task–difficulty combination.The gains remain consistent under compounded shifts in real-world robot experiments. I therefore find the main results credible and did not assign a low overall score. Evidence: Table 5.

Critical weaknesses

W1: It is not clear whether OOD gains come from the V-JEPA representation, the history frames, or the increased capacity of the additional expert. OOD generalization is the paper’s central claim, but the source of the gains remains unclear, making its explanation of why the method works unconvincing. This is my main reason for assigning a Soundness score of 1.

W2: The efficiency and scale mentioned in the paper are not supported by exact data. There's no number for latency, frequency, FLOPs or parameter count.The paper emphasizes efficiency, but provides insufficient data to verify this claim. If CSWAM is substantially larger, model capacity would further confound the attribution of gains discussed in W1, making that concern more serious.

Questions for authors
  1. Section 4.1 states that CSWAM uses the embodied-pretrained initialization on LIBERO-Plus, but Table 4 marks CSWAM's "Emb. PT" as "No". Which is correct? In either case, were FastWAM and ST-WAM in Table 4 initialized in the same way as CSWAM?
  2. Could the authors report the inference latency, control frequency, and parameter count of CSWAM and FastWAM, measured on the same hardware and under the same settings?
  3. Were the baseline results in Table 2 reproduced by the authors or taken from the original papers?
  4. In Fig. 2, why do the observed-context queries primarily attend to the action token (a_1) while masking out the semantic token (j_0) during inference? Is this a figure error, or does it reflect the attention pattern implemented in this work? If the latter, how can this be reconciled with the training mask and the reuse of the fixed observation context during denoising?
  5. I suggest reporting the same aggregate in the abstract and the conclusion. The abstract reports the average improvement over all three difficulty levels (from 27.5% to 70.0%), whereas the conclusion includes only the Hard and Extreme settings, with different numbers (from 10% to 55%).
4 comments
AuthorsOpenReview ↗

Response for Submission20013 review

W1: To clarify the sources of the OOD gains, we extend the ablations in Tables 6–8 from RoboTwin-Full to Clean-to-Randomized transfer. As in CSWAM with Embodied PT., all retained video/action experts use the same pretrained FastWAM weights, while additional expert DiTs are initialized from Wan2.2 video DiT weights via dimension-wise interpolation and rescaling to match their reduced hidden and FFN widths. The results show:

VariantCleanRandomizedAverage
w/o video expert64.113.038.5
FastWAM76.710.243.4
0-frame V-JEPA80.132.556.3
4-frame V-JEPA82.037.259.6
4-frame DINOv383.432.357.9
8-frame V-JEPA84.545.264.8
  • Representation choice. With the same additional expert architecture and downstream token budget, 8-frame V-JEPA outperforms 4-frame DINOv3 by 12.9 points on Randomized scenes and 6.9 points on average. With the same four input frames, V-JEPA still improves Randomized and average success by 4.9 and 1.7 points, respectively. DINOv3 produces twice as many downstream tokens in this setting and takes longer to train in our measurements. These comparisons support V-JEPA’s benefit for OOD transfer.
  • Temporal history. To isolate the contribution of temporal context, we vary history length while keeping expert architecture and parameter count fixed. Increasing history from 0→4→8 frames improves Clean success by 1.9 and 2.5 points, and Randomized success by 4.7 and 8.0 points. These gains support the value of longer history, particularly for OOD generalization.
  • Complementary pathways. The semantic–action variant without the video expert still exceeds FastWAM by 2.8 points on Randomized scenes, showing an OOD advantage in this reduced configuration. Retaining the video expert further improves Clean/Randomized success by 20.4/32.2 points, supporting the complementary value of the two pathways.
  • Additional capacity. The history-length variants retain identical model capacity, while the encoder variants retain the same additional expert architecture. Their different OOD outcomes show that adding an expert alone is insufficient to explain the gains: representation choice and temporal information also matter. Our original main generalization evaluation additionally compares CSWAM with the structurally related ST-WAM on LIBERO-Plus, where CSWAM achieves a 4.3-point improvement (Table 4). We do not rule out a contribution from increased capacity, but the fixed-architecture controls show that capacity alone cannot explain the observed differences. These new ablations directly examine the sources of improvement under OOD transfer and strengthen the evidence beyond the original mixed-training analysis.

We will include the detailed results, settings, and analyses in the revised manuscript.

W2Q2. We agree that the original manuscript did not provide sufficient quantitative evidence for efficiency and model scale. We will add the following latency, throughput, and parameter measurements, together with the hardware and inference settings. Both models generate 32 actions per inference call.

MetricFastWAMCSWAM
End-to-end inference latency192 ms200 ms
Inference rate, calculated from latency5.21 Hz5.00 Hz
Predicted-action throughput166.7 actions/s160.0 actions/s
Total parameters, including frozen T512.406B13.520B
Parameters excluding frozen T56.725B7.839B
Trainable parameters6.021B7.048B

CSWAM adds 8 ms of latency (+4.2%), while increasing total parameters by 1.114B and trainable parameters by 1.027B. Our efficiency claim therefore concerns retaining inference latency close to FastWAM through action-only denoising.

We do not rule out a contribution from increased capacity. To address this concern, the new C2R ablations discussed in W1. We will report the parameter increase explicitly and clarify the scope of our efficiency claim.

Q1: Thank you for catching this error. Table 4 is correct: CSWAM was evaluated on LIBERO-Plus without embodied pretraining. CSWAM, ST-WAM, and FastWAM all use the same embodied pretraining setup. The sentence in Sec. 4.1 was inadvertently carried over from the RoboTwin/real-robot setup and will be corrected in the newly submitted version.

Q3: Baseline results in Table 2 are taken from the original papers.

Q4: Thank you for catching this error. The inference mask in Fig. 2 contains a drawing mistake: observed-context queries attend to the semantic token j0j_0, while the action token a1a_1 should be masked out. We will correct Fig. 2; the reported results are unaffected.

Q5: Thank you for pointing this out. We will revise the conclusion to match the abstract, reporting average success across both tasks and all three difficulty levels: 27.5% for FastWAM and 70.0% for CSWAM. The Hard-and-Extreme aggregate will remain separately labeled in the results section.

AuthorsOpenReview ↗

Updated Manuscript and Summary of Revisions

We have uploaded a revised manuscript incorporating additional experiments, clarifications, and corrections in response to the reviewers' feedback.

The main updates include:

  • Section 3.3 (Fig. 2): Clarified the attention mask and cross-stream information flow.
  • Section 4.2: Expanded the comparison with ST-WAM and the analysis of OOD robustness on LIBERO-Plus.
  • Section 4.4 (Tables 6–8): Added ablations on the complementary video and semantic pathways, V-JEPA versus DINOv3, and temporal history length.
  • Section 4.5: Added an inference latency comparison with FastWAM.
  • Additional corrections: Corrected a typo regarding embodied pretraining in the LIBERO-Plus experiments and aligned the presentation of results in the Conclusion with the Abstract.

We thank the reviewers and Area Chair for their valuable feedback and welcome further questions or discussion.

Dear Reviewer,

We hope this message finds you well. As the discussion period is nearing its end with less than 24 hours remaining, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you’d like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Dear Reviewer duap,

Please respond to the author soon and engage in the discussion.

Best,

AC

Reviewer_wWN6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CSWAM, which augments FastWAM with a V-JEPA 2.1-based causal semantic expert to extract temporal semantic information from observation history. Future semantic prediction and cross-expert attention enhance video and action representations. Simulation and real-robot experiments demonstrate improved manipulation performance under visual distribution shifts.The method addresses the sensitivity of reconstruction-oriented representations to appearance changes and the lack of temporal evidence in current observations. Incorporating video semantic features and historical context while retaining action-only inference is a reasonable design.

Critical strengths

The method addresses the sensitivity of reconstruction-oriented representations to appearance changes and the lack of temporal evidence in current observations. Incorporating video semantic features and historical context while retaining action-only inference is a reasonable design.

Critical weaknesses
  1. The term “causal” primarily refers to using only current and past observations and preventing future information leakage through attention masks. This ensures temporal causality but does not directly establish that the model learns task-relevant causal factors.

  2. Although ST-WAM and JEPA-WAM are discussed in the related work, the key Clean2Random experiments compare only against FastWAM. These comparisons are insufficient to establish the distinct advantages of the proposed semantic expert and cross-stream interaction design.

  3. Although the DINOv3 comparison matches the downstream token count, DINOv3 receives four input frames while V-JEPA receives eight. The performance difference may therefore partly reflect differences in the amount of observed information.

Questions for authors
  1. Does “causal” refer solely to temporal causality, or is there evidence that the model learns task-relevant causal factors?
  2. Could you compare against ST-WAM and JEPA-WAM under the same Clean2Random protocol to clarify the advantages of your design?
  3. Does V-JEPA retain its advantage over DINOv3 when input frame counts and temporal coverage are matched?
4 comments
AuthorsOpenReview ↗

Response for Submission20013 review

W1Q1: We agree that temporal causality does not establish causal-factor learning. In this work, “causal” refers specifically to the temporal information-flow constraint: the semantic expert uses observations up to the current time, and attention masks prevent future-target leakage into action prediction. We do not claim to identify task-relevant causal factors. We will clarify this distinction in the paper and describe the learned features as history-conditioned semantic context.

W2Q2: Our main OOD evaluation already includes ST-WAM on LIBERO-Plus (Table 4), where CSWAM improves success by 4.3 percentage points. To further clarify the component contributions under Clean-to-Randomized transfer, we extend the ablations in Tables 6–8 to this setting. As in CSWAM with Embodied PT., all retained video/action experts use the same pretrained FastWAM weights, while additional expert DiTs are initialized from Wan2.2 video DiT weights via dimension-wise interpolation and rescaling to match their reduced hidden and FFN widths.

VariantCleanRandomizedAverage
w/o video expert64.113.038.5
FastWAM76.710.243.4
0-frame V-JEPA80.132.556.3
4-frame V-JEPA82.037.259.6
4-frame DINOv383.432.357.9
8-frame V-JEPA84.545.264.8
  • Complementary video and semantic pathways. We remove the video expert to assess whether the semantic pathway can replace it. The reduced model retains a 2.8-point Randomized advantage over FastWAM, while combining both pathways improves Randomized success by a further 32.2 points. This supports their complementary roles.
  • Encoder choice. Motivated by prior work (Bardes et al., 2024; Assran et al., 2025), we hypothesize that V-JEPA’s joint spatiotemporal encoding is better suited to capturing state changes and motion from history than framewise DINOv3 features. On C2R Randomized, the V-JEPA-based variant outperforms its DINOv3, counterpart under both matched downstream token budgets (+12.9 points) and matched input frame counts (+4.9 points), consistent with this hypothesis.
  • Temporal history. To isolate the contribution of temporal context, we vary history length while keeping expert architecture and parameter count fixed. Increasing history from 0→4→8 frames improves Clean success by 1.9 and 2.5 points, and Randomized success by 4.7 and 8.0 points. These gains support the value of longer history, particularly for OOD generalization.

These results provide evidence for the contributions of encoder choice, temporal history, future-semantic supervision, and the complementary video pathway. They strengthen the explanation of our OOD gains.

JEPA-WAM uses a Qwen-based shared predictor and a different policy architecture, so it is not a controlled comparison for isolating CSWAM’s semantic expert or cross-stream interaction.

W3Q3: Table 8 shows that V-JEPA with 4 input frames still outperforms DINOv3 with 4 input frames in table 7. Table 7 matches downstream token budgets for fair comparison and computational efficiency, while Table 8 additionally matches input frame counts. In our extended Clean-to-Randomized experiments, V-JEPA outperforms DINOv3 by 1.1/12.9/6.9 points on Clean/Randomized/average success under matched token budgets, and by 4.9/1.7 points on Randomized/average success with matched input frames. These results support our encoder choice. We will add the comparisons and analysis to Section 4.5.

AuthorsOpenReview ↗

Updated Manuscript and Summary of Revisions

We have uploaded a revised manuscript incorporating additional experiments, clarifications, and corrections in response to the reviewers' feedback.

The main updates include:

  • Section 4.2: Expanded the comparison with ST-WAM and the analysis of OOD robustness on LIBERO-Plus.
  • Section 4.4 (Tables 6–8): Added ablations on the complementary video and semantic pathways, V-JEPA versus DINOv3, and temporal history length.
  • Section 4.5: Added an inference latency comparison with FastWAM.
  • Additional corrections: Corrected a typo regarding embodied pretraining in the LIBERO-Plus experiments.

We thank the reviewers and Area Chair for their valuable feedback and welcome further questions or discussion.

Dear Reviewer,

We hope this message finds you well. As the discussion period is nearing its end with less than 24 hours remaining, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you’d like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Dear Reviewer wWN6,

Please respond to the author soon and engage in the discussion. Best,

AC

Reviewer_MdzsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CSWAM extends FastWAM with a frozen V-JEPA 2.1 encoder and a history-conditioned semantic expert. Joint future-video, future-semantic and action prediction train the model, while deployment uses observed context for action-only denoising. The paper reports improved robustness on RoboTwin Clean-to-Randomized transfer, LIBERO-Plus and two real-robot tasks.

Critical strengths
  1. The reported gains are substantial: Clean-to-Randomized success improves from 10.16% to 45.18%, and average real-robot success from 27.5% to 70.0%, supporting the practical promise of the approach.

  2. The evaluation explicitly separates mixed-training performance from OOD transfer, making the main generalization claim more meaningful.

Critical weaknesses
  1. The source of the OOD gains and the incremental contribution remain unclear. Tables 6–8 use mixed-training Full, without isolating JEPA task relevance, history placement or future-semantic supervision under OOD conditions. ST-WAM (Wang et al) already combines semantic prediction, history and action-only inference, making controlled comparison particularly important.

  2. Comparison fairness and efficiency are unresolved. Section 4.1 says LIBERO-Plus uses embodied pretraining, but Table 4 says No. No end-to-end latency is reported, so action-only denoising does not establish FastWAM-like speed after adding history encoding and another expert.

Questions for authors
  1. What evidence shows stronger task-state sensitivity and nuisance invariance than VAE/video-DiT features under matched observations? Table 7 uses eight V-JEPA frames versus four DINO frames.

  2. On Clean2Random, how does CSWAM compare with history supplied directly to the video branch or both branches, and with the same semantic expert and history but zero future-semantic loss? These controls would separate history and prediction benefits.

  3. Which LIBERO-Plus pretraining description is correct? Please specify the initialization and data, and clarify whether comparisons with ST-WAM and FastWAM match embodied pretraining.

  4. What is end-to-end latency versus FastWAM on identical hardware, action horizons and denoising steps, including history encoding, context processing and action denoising?

4 comments
AuthorsOpenReview ↗

Response for Submission20013 review

W1Q2: To clarify the sources of the OOD gains, we extend the ablations in Tables 6–8 from RoboTwin-Full to Clean-to-Randomized transfer. As in CSWAM with Embodied PT., all retained video/action experts use the same pretrained FastWAM weights, while additional expert DiTs are initialized from Wan2.2 video DiT weights via dimension-wise interpolation and rescaling to match their reduced hidden and FFN widths. The results show:

VariantCleanRandomizedAverage
w/o video expert64.113.038.5
FastWAM76.710.243.4
0-frame V-JEPA80.132.556.3
4-frame V-JEPA82.037.259.6
4-frame DINOv383.432.357.9
8-frame V-JEPA84.545.264.8
  • Complementary video and semantic pathways. We remove the video expert to assess whether the semantic pathway can replace it. The reduced model retains a 2.8-point Randomized advantage over FastWAM, while combining both pathways improves Randomized success by a further 32.2 points. This supports their complementary roles.
  • Encoder choice. Motivated by prior work (Bardes et al., 2024; Assran et al., 2025), we hypothesize that V-JEPA’s joint spatiotemporal encoding is better suited to capturing state changes and motion from history than framewise DINOv3 features. On C2R Randomized, the V-JEPA-based variant outperforms its DINOv3 counterpart under both matched downstream token budgets (+12.9 points) and matched input frame counts (+4.9 points), consistent with this hypothesis.
  • Temporal history. To isolate the contribution of temporal context, we vary history length while keeping expert architecture and parameter count fixed. Increasing history from 0→4→8 frames improves Clean success by 1.9 and 2.5 points, and Randomized success by 4.7 and 8.0 points. These gains support the value of longer history, particularly for OOD generalization.
  • History placements. We considered directly adding history to the video branch, but this would alter FastWAM’s pretrained temporal conditioning interface. We therefore use a separate semantic pathway. To evaluate our history integration design at the system level, we compare with ST-WAM (Wang et al.), which queries DINO-encoded history using the current image and instruction. CSWAM’s separate semantic pathway achieves 4.3 points higher success on LIBERO-Plus (Table 4), supporting the OOD effectiveness of our integrated design.
  • Future-semantic supervision. Removing the future-semantic loss yields performance comparable to FastWAM, further supporting the importance of this predictive objective.

We will include detailed ablations, settings, and analyses in the revised manuscript.

W2Q3: 1. Thank you for catching this error. Table 4 is correct: CSWAM was evaluated on LIBERO-Plus without embodied pretraining. CSWAM, ST-WAM, and FastWAM all use the same embodied pretraining setup. The sentence in Sec. 4.1 was inadvertently carried over from the RoboTwin/real-robot setup and will be corrected in the newly submitted version. 2. For CSWAM in No Embodied PT tasks, the video expert was initialized with pretrained Wan DiT weights, while ActionDiT and the causal semantic expert were initialized from the pretrained Wan2.2 video DiT by applying dimension-wise linear interpolation and rescaling to accommodate its smaller hidden and FFN widths. 3. Under identical hardware (the same 5090), 32 action horizons, and 10 denoising steps, we measured 192 ms for FastWAM versus 200 ms for CSWAM, showing that CSWAM incurs only ~8 ms of additional latency. We will upadate in our newly revise.

Q1: Motivated by prior work (Bardes et al., 2024; Assran et al., 2025), we adopt V-JEPA for its temporal-semantic representations learned through feature prediction rather than pixel reconstruction. We aim to demonstrate its effectiveness within FastWAM, rather than establish superiority over Wan-VAE in task-state sensitivity or nuisance invariance. Table 8 shows that V-JEPA with 4 input frames still outperforms DINOv3 with 4 input frames in table 7. Table 7 matches downstream token budgets for fair comparison and computational efficiency, while Table 8 additionally matches input frame counts. In our extended Clean-to-Randomized experiments, V-JEPA outperforms DINOv3 by 1.1/12.9/6.9 points on Clean/Randomized/average success under matched token budgets, and by 4.9/1.7 points on Randomized/average success with matched input frames. These results support our encoder choice. We will add the comparisons and analysis to Section 4.5.

Q4: Under identical hardware (the same 5090), 32 action horizons, and 10 denoising steps, we measured 192 ms for FastWAM versus 200 ms for CSWAM, showing that CSWAM incurs only ~8 ms of additional latency. We will upadate in our newly revise.

AuthorsOpenReview ↗

Updated Manuscript and Summary of Revisions

We have uploaded a revised manuscript incorporating additional experiments, clarifications, and corrections in response to the reviewers' feedback.

The main updates include:

  • Section 4.2: Expanded the comparison with ST-WAM, including differences in history placement and integration, and analyzed OOD robustness on LIBERO-Plus.
  • Section 4.4 (Tables 6–8): Added ablations on the complementary video and semantic pathways, V-JEPA versus DINOv3, and temporal history length.
  • Section 4.5: Added an inference latency comparison with FastWAM.
  • Additional corrections: Corrected a typo regarding embodied pretraining in the LIBERO-Plus experiments.

We thank the reviewers and Area Chair for their valuable feedback and welcome further questions or discussion.

Dear Reviewer,

We hope this message finds you well. As the discussion period is nearing its end with less than 24 hours remaining, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you’d like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Dear Reviewer Mdzs,

Please respond to the author soon and engage in the discussion. Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.