acceptodds
Under review as a conference paper at ICLR 2027

Causality-Inspired World Action Model via Contrastive Representation Learning

Abstract

World-action models (WAMs) transfer predictive visual priors to robotic manipulation, but robust control requires invariance to task-irrelevant appearance changes that remain relevant to video prediction. Action supervision specifies the desired behavior for each observation without explicitly identifying which visually different observations share the same manipulation state. We introduce CausalWAM, a post-training method that supervises state correspondence through *action-preserving visual interventions*. We vary scene appearance while preserving robot–object geometry, task instructions, and demonstrated actions, changing the observation mechanism without changing the underlying manipulation state. A contrastive objective aligns alternative renderings of the same state and distinguishes different states within each task. A shared content reader connects this supervision to action learning, supplying compact tokens alongside the action expert’s native visual conditioning. The same design supports interface learning over a frozen video backbone and joint video–action adaptation. With action demonstrations restricted to Clean scenes, CausalWAM-Joint increases RoboTwin 2.0 Randomized success from 1.9% for FastWAM to 34.7%. It achieves 79.4% mean success across seven unseen LIBERO-Plus perturbation types and improves real-robot success under visual disturbance from 15.6% to 46.7%. Across 50 RoboTwin tasks, CausalWAM-Interface improves cross-appearance state retrieval by 9.42 percentage points. These results demonstrate the value of intervention-defined state correspondence for adapting predictive world representations to visually robust manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Robot Manipulation, Causal Representation Learning, Domain Generalization, Contrastive Learning, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mYq6OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

World–action models learn from video prediction and robot-action demonstrations. Predicting future images encourages these models to retain details such as background, texture, and lighting, even when those details are irrelevant to the robot’s actions. The authors propose CausalWAM, which trains the representation used for action prediction to recognize the same physical state across appearance changes. They render identical robot–object configurations with different backgrounds and textures, then use contrastive learning to bring their representations together while separating different physical states. A small attention module extracts the resulting features and supplies them to the action predictor, improving robustness to visual changes.

Critical strengths
  • CausalWAM-Joint substantially improves robustness to visual changes, increasing success from 51.5% to 79.4% on LIBERO-Plus and from 15.6% to 46.7% in real-robot experiments with visual disturbances, while preserving strong performance on standard LIBERO. These gains address an important weakness of world–action models and are the strongest evidence supporting acceptance. However, further controlled comparisons are needed to establish which components drive the improvements.

  • The content reader connects explicit cross-appearance state correspondence to action prediction while retaining the model’s original visual pathway. Its frozen-backbone variant also improves state retrieval and control, suggesting that useful invariances can be extracted from existing predictive representations without retraining the video backbone. This makes the approach a potentially useful adaptation strategy beyond the particular model evaluated.

Critical weaknesses
  • The paper is missing two key comparisons: training the baseline on the same randomized images using the action labels, and testing the Joint model without contrastive learning in the Clean-to-Randomized setting, where the largest improvement occurs. Without these tests, it is hard to tell how much of the gain comes from contrastive learning and how much comes from seeing more varied images, adding the new modules, or fine-tuning the backbone. This makes the main result less convincing.

  • Using controlled appearance changes to learn representations that focus on physical state is already an established idea. The main addition here is bringing this approach into a world–action model through the content reader and adapter. The causal framing explains why the training pairs make sense, but does not introduce causal discovery. This could still be a worthwhile contribution, but the missing comparisons make it hard to judge how much the proposed design adds over existing approaches.

Questions for authors
  • How does CausalWAM compare with FastWAM trained on the same randomized images using their preserved action labels, with a comparable training budget? This would help determine whether contrastive state correspondence adds value beyond appearance augmentation.

  • What is the Joint model’s performance without contrastive learning in the Clean-to-Randomized setting of Table 3, keeping the architecture and training setup unchanged? This would clarify whether the proposed objective drives the largest reported gain.

6 comments
AuthorsOpenReview ↗

Response to Reviewer mYq6 - Thank you for reviewing our paper!

Thank you for your thoughtful comments! Below, we address the comparison with FastWAM trained on the same paired observations, report the matched Joint ablation without contrastive learning, and explain how CausalWAM uses state-correspondence supervision in a world-action model. The changes are highlighted in blue in the revised manuscript.

[W1] Matched comparisons to isolate the contribution of CausalWAM

We thank the reviewer for highlighting the need for these two comparisons. We conducted both experiments to examine the contribution of contrastive supervision. With the same paired images and inherited action labels, FastWAM achieves 5.46% Random success, while CausalWAM-Joint without contrastive learning achieves 6.24%. Enabling contrastive learning raises Joint's success to 34.90%. In the Clean-to-Randomized setting, Joint also improves from 4.80% to 34.74% when contrastive learning is enabled, a gain of 29.94 percentage points.

We detail the matched FastWAM comparison in Q1 and the Clean-to-Randomized ablation in Q2.

[Q1] FastWAM trained on the same paired observations

We train FastWAM and CausalWAM-Joint using the same paired observations and their inherited action labels. The results are:

ModelAction-supervision dataContrastive lossClean (%)Random (%)
FastWAMOfficial Clean + paired observations-78.625.46
CausalWAM-JointOfficial CleanOff79.244.80
CausalWAM-JointOfficial CleanOn82.8234.74
CausalWAM-JointOfficial Clean + paired observationsOff78.306.24
CausalWAM-JointOfficial Clean + paired observationsOn81.6234.90

FastWAM and CausalWAM-Joint without contrastive learning achieve similar Random success (5.46% and 6.24%, respectively) when trained on the same paired images with action supervision. With contrastive learning, Joint reaches 34.90%, a gain of 28.66 percentage points over its counterpart. This comparison shows that the improvement goes beyond directly adding action-labelled appearance variations.

[Q2] CausalWAM-Joint without contrastive learning in the Clean-to-Randomized setting

We additionally evaluate Joint without contrastive learning under the Clean-only protocol of Table 3, keeping the architecture and training setup unchanged.

CausalWAM-JointClean (%)Random (%)
Without contrastive loss79.244.80
With contrastive loss82.8234.74
Improvement (pp)+3.58+29.94

Contrastive supervision improves Random success by 29.94 percentage points, from 4.80% to 34.74%. The architecture, initialization, trainable modules, official Clean demonstrations, and training schedule remain unchanged. The paired observations contribute only through the contrastive loss.

[W2] Contribution beyond existing appearance-invariant learning methods

Thank you for this thoughtful comment! The key contribution of CausalWAM is to make cross-appearance state correspondence an explicit training objective for the action-facing representation in a world–action model. This allows the action expert to distinguish changes in physical state from variations in scene appearance, while retaining the original visual conditioning.

Several experiments highlight the role of this objective. In the Clean-to-Randomized setting, FastWAM achieves only 1.9% Random success, compared with 7.5% for CausalWAM-Interface and 34.7% for CausalWAM-Joint, as reported in Table 3. Table 5 further separates the effects of action and contrastive supervision: adding paired observations as action targets alone improves Random success only from 4.8% to 6.2%, whereas contrastive supervision raises it to 34.7%. Combining both yields 34.9%.

The contribution of contrastive learning is also reflected in other ablations. In Table 4, removing the contrastive objective reduces CausalWAM-Joint's Random success from 93.62% to 92.18%. Table 6 compares two representation objectives under the same architecture and training data:

Representation lossClean (%)Random (%)
Paired-feature MSE82.2625.48
SupCon (Ours)82.8234.74

Although both objectives achieve similar Clean performance, SupCon improves Random success by 9.26 percentage points over paired-feature MSE.

These results show that explicitly learning correspondence between different appearances of the same physical state provides an important supervisory signal for action prediction in WAMs. We have further clarified the relationship to prior work on causal representation learning and contrastive domain randomization in Section 2, and formalized our action-preserving intervention and state-correspondence objective in Section 3.

Reviewer_mYq6OpenReview ↗

Main Concerns Addressed

Thank you for your detailed responses and additional experiments. The new comparisons substantially strengthen the paper and convincingly demonstrate the benefit of contrastive supervision beyond appearance augmentation. The comparison with paired-feature MSE further supports the proposed objective.

I have two minor remaining comments:

Please include the matched FastWAM result (5.46%) in Table 5 or the appendix.

Please reconcile the real-robot baseline success rate of 15.6% in the main paper with the 13.3% aggregate from the three evaluation rounds.

Overall, my main concerns have been addressed, and the revisions have strengthened my assessment of the paper.

Reviewer_RnDrOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CausalWAM improves the visual robustness of world-action models through contrastive learning on state-preserving appearance variations, using a content reader to connect the learned representations to action generation.

Critical strengths
  1. Meaningful problem and clear motivation. The paper addresses the tension between preserving appearance information for visual prediction and learning invariance to task-irrelevant appearance changes for control.

  2. Intuitive method with a integration design. Same-state observations under different appearances provide explicit correspondence supervision. The content reader incorporates this supervision into action generation while retaining the original visual pathway.

  3. Encouraging empirical results. The improvements in visually perturbed control and cross-appearance state retrieval support the approach, while the standard benchmark results suggest compatibility with strong task performance.

Critical weaknesses
  1. Clarification of training data and a matched action-supervision baseline. Please first clarify the relationship between the randomized action demonstrations and the paired renderings used for contrastive learning. Let $ C $ denote the Clean demonstrations, RorigR_{\mathrm{orig}} the original Random demonstrations released by RoboTwin, and RpairR_{\mathrm{pair}} the additional paired randomized renderings. Are RorigR_{\mathrm{orig}} and RpairR_{\mathrm{pair}} distinct or overlapping? Specifically:

    • Table 1: Is action supervision applied to C∪RorigC \cup R_{\mathrm{orig}}, while contrastive supervision uses C∪RpairC \cup R_{\mathrm{pair}}?
    • Table 3: Action supervision is explicitly restricted to CC. Does contrastive supervision use C∪RpairC \cup R_{\mathrm{pair}}?

    If this interpretation is correct, the key missing baseline is to use RpairR_{\mathrm{pair}}, with inherited action labels, directly for action supervision:

    • Table 1: Train on C∪Rorig∪RpairC \cup R_{\mathrm{orig}} \cup R_{\mathrm{pair}}.
    • Table 3: Train on C∪RpairC \cup R_{\mathrm{pair}}.

    For each setting, compare this baseline without and with the contrastive loss, keeping initialization and training budgets comparable. This would distinguish the benefit of explicit correspondence supervision from that of additional randomized observations. For rebuttal, I would prioritize the Table 3 comparison, particularly for Joint.

  2. The Clean2Random training protocol needs clarification. The paper restricts action supervision to Clean demonstrations, but it is unclear whether randomized scenes also contribute to the video-prediction loss. Please explicitly specify the data used for each loss in this experiment. If video prediction uses randomized scenes, a baseline with the same video-training exposure but without the contrastive loss is needed to isolate its contribution.

Minor Point

  1. Table 1 omits CausalWAM-Joint under RoboTwin Clean+Random training. Reporting this result, or explaining its omission, would improve completeness. This is a lower-priority concern and would not substantially affect my rating.
  2. The bold formatting in Table 4 is potentially misleading: it appears to highlight the default configuration rather than the best result in each column. Please clarify the convention in the caption or use shading for the default configuration and reserve bold for column-wise best results.
Questions for authors

Please address the main questions raised in the weaknesses: clarify the training data used for each loss in Tables 1 and 3, and compare direct action supervision on the paired randomized observations with and without the contrastive loss. These clarifications and comparisons would materially affect my assessment.

3 comments
AuthorsOpenReview ↗

Response to Reviewer RnDr - Thank you for reviewing our paper!

Thank you for your careful review and helpful suggestions! We appreciate the suggestion to distinguish the official Random demonstrations from the additional paired renderings and to compare their use for action supervision. The changes are highlighted in blue in the revised manuscript.

[W1, Q1] Training data and action supervision on paired observations

Thank you for raising this important question. In our method, RorigR_{\mathrm{orig}} and RpairR_{\mathrm{pair}} serve different purposes and are generated separately. RorigR_{\mathrm{orig}} denotes the original Random demonstrations released by RoboTwin, whereas RpairR_{\mathrm{pair}} consists of additional renderings generated from Clean training states under different appearances. Each paired rendering inherits the action label of its corresponding Clean state.

As described in Section 3 and Appendix A.1, these renderings preserve the physical state and demonstrated action while varying scene appearance. Section 4.2 introduces the paired stream, and Section 4.4 specifies how it is used alongside the original demonstration stream. The data assignment in our experiments is summarized below:

Training settingAction supervisionContrastive supervisionVideo-prediction supervision
Clean+RandomC∪RorigC\cup R_{\mathrm{orig}}C∪RpairC\cup R_{\mathrm{pair}}C∪RorigC\cup R_{\mathrm{orig}}
Clean+Random + paired actionC∪Rorig∪RpairC\cup R_{\mathrm{orig}}\cup R_{\mathrm{pair}}C∪RpairC\cup R_{\mathrm{pair}}C∪RorigC\cup R_{\mathrm{orig}}
Clean-onlyCCC∪RpairC\cup R_{\mathrm{pair}}CC
Clean-only + paired actionC∪RpairC\cup R_{\mathrm{pair}}C∪RpairC\cup R_{\mathrm{pair}}CC

Our paired-view data contain 1,500 Clean training trajectories across 50 tasks, yielding 411,389 physical-state samples, each rendered under four appearances (C/R1/R2/R3). The 1,500 trajectories correspond to 5.45% of the 27,500 official trajectories used in Clean+Random training. Clean-only training uses 2,500 official Clean trajectories.

Following your suggestion, we further examine whether these paired views benefit from direct action supervision. We add their inherited action labels to the Joint training objective and compare the results with the original contrastive-only setting, under both Clean-only and Clean+Random training.

Clean-only setting (Table 3):

Action-supervision dataContrastive lossClean (%)Random (%)
CCOff79.244.80
CCOn82.8234.74
C∪RpairC\cup R_{\mathrm{pair}}Off78.306.24
C∪RpairC\cup R_{\mathrm{pair}}On81.6234.90

Direct action supervision on RpairR_{\mathrm{pair}} alone brings Random success from 4.80% to 6.24%. With contrastive learning, it reaches 34.90%, close to the 34.74% achieved by our original setting, where paired views are used only for contrastive supervision.

Clean+Random setting (Table 1):

Action-supervision dataContrastive lossClean (%)Random (%)
C∪RorigC\cup R_{\mathrm{orig}}Off92.0292.16
C∪RorigC\cup R_{\mathrm{orig}}On93.1893.74
C∪Rorig∪RpairC\cup R_{\mathrm{orig}}\cup R_{\mathrm{pair}}Off92.0892.20
C∪Rorig∪RpairC\cup R_{\mathrm{orig}}\cup R_{\mathrm{pair}}On93.2293.72

Here, adding paired-image action supervision has little effect, while contrastive learning improves Random success by about 1.5 pp with or without that additional action supervision. Within each setting, initialization, architecture, training schedule, and video-prediction data are matched. Appendix B.1 gives the training settings.

[W2] Whether paired randomized observations enter video prediction

The paired C/R1/R2/R3 observations do not enter the video-prediction loss. They are used only for contrastive learning. In Joint, the contrastive gradients also update VideoDiT, but the paired observations are not used as video-prediction targets.

For Clean-only training, both action and video-prediction losses use official Clean demonstrations. For Clean+Random training, both losses use official Clean and Random demonstrations. These official Random demonstrations are separate from the paired observations used for contrastive learning.

The Clean-only Joint ablation keeps the official Clean demonstrations and video-prediction objective unchanged. The results are summarized below.

CausalWAM-JointClean (%)Random (%)
Without contrastive loss79.244.80
With contrastive loss82.8234.74

Random success increases from 4.80% to 34.74%, a gain of 29.94 percentage points. This improvement comes from contrastive supervision without introducing additional randomized video-prediction targets. We have clarified the data assignment for each training objective in Section 4.4 and provided the corresponding training settings in Appendix B.1.

[M1] CausalWAM-Joint under Clean+Random training

We have added the Joint results to Table 1:

ModelClean (%)Random (%)Average (%)
CausalWAM-Interface93.2493.6293.43
CausalWAM-Joint93.1893.7493.46

The two variants perform similarly under Clean+Random training, with Joint slightly higher on Random and Interface slightly higher on Clean.

[M2] Formatting of Table 4

We have revised Table 4 accordingly. An asterisk now marks the default setting, while bold indicates the best result in each column. We have also clarified in the caption that each subtable varies one factor at a time. In the current results, the default setting also gives the best scores.

As the discussion period is drawing to a close, we would be willing to address any remaining questions or concerns about our work. If there is anything further we can clarify, we would greatly appreciate your feedback before the discussion closes.

Thank you again for your time and consideration.

Best wishes,

Authors of Submission #18615

Reviewer_r5e5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper presents CausalWAM, an approach that uses contrastive representation learning to make WAMs more robust to visual variation. Their training uses correspondence action pairs, in which the scene's appearance is changed while the physical manipulation states, instructions and the demonstrated actions remain fixed. With their contrastive loss, representations of the same state are pulled together across different appearances and pushed apart from those of other states in the same task. The learned tokens are then passed to a shared content reader as supplementary features to the existing visual conditions. The authors introduce two training settings, namely, finetuning only the adaptation parts while keeping the video backbone frozen, or jointly finetuning video and action components together. The results show that across RoboTwin 2.0, LIBERO, LIBERO-Plus, and three real-robot tasks, the method performs better under visual perturbations without sacrificing performance on the standard benchmarks. Ablations and experiments retrieving matching states across appearances offer further support for both the learned representations and the contrastive objective.

Critical strengths
  1. The learning signal is well-motivated for visual robustness. The paper addresses the mismatch between retaining appearance details for video prediction and learning invariances for robot control by introducing paired renderings and adding a contrastive loss to the training recipe, inspired by the causal analysis of the standard WAM learning scheme. By rendering the same physical state under different appearances, the paired data makes explicit which observations should share a representation. The ablation supports this motivation, showing that with the architecture unchanged, introducing contrastive supervision alone lifts success on randomized RoboTwin from 92.18% to 93.62%.

  2. Converging evidence from both control and representation analyses. On LIBERO-Plus, CausalWAM raises success to 79.4% compared with 51.5% for FastWAM, and it does so without giving up performance on standard LIBERO, where it reaches 97.8%. The gains extend to cross-appearance retrieval, which climbs from 75.22% to 84.64%, and to real-robot trials under visual disturbance, where success rises from 15.6% to 46.7%. These complementary results indicate that appearance-consistent representations deliver practical benefits, rather than gains confined to one benchmark.

Critical weaknesses
  1. Incomplete ablations to support one of the main results. Table 3 reports a substantial improvement in randomized success from 1.9% to 34.7%, but does not isolate the contribution of contrastive supervision within CausalWAM-Joint. Table 4 provides a no-contrastive-loss ablation only for CausalWAM-Interface. A matched CausalWAM-Joint experiment with lambda = 0 under Table 3's Clean-only action-supervision protocol, with initialization, architecture and training budget fixed, would clarify how much of the gain comes from the contrastive loss.

  2. Limited intervention scope and an incomplete connection between representation alignment and policy invariance. The contrastive objective aligns a projected embedding across appearance variants, while the policy takes the full content tokens since the existing visual conditioning is kept. As a result, the alignment does not guarantee invariance of the resulting actions or generalization to unseen perturbations, which can be seen in the results of Table 2. For example, since the interventions primarily vary appearance, the strong background results the paper is highlighting are not surprising and consistent with the training signal. On the other hand, transferring to robot initial states, instruction paraphrases, and object layouts is not directly encouraged by that signal, so no obvious gains are observed. This limits the broader implications of the robustness results. More diverse, carefully designed interventions, or an explicit mechanism connecting representation invariance to action consistency, would strengthen the contribution.

Questions for authors
  1. Could you provide a matched CausalWAM-Joint ablation with lambda = 0 under the same Clean-only action-supervision protocol? Table 4 addresses CausalWAM-Interface under Clean+Random training, but it does not explain how much of the 1.9% to 34.7% improvement comes specifically from contrastive supervision.
  2. Since contrastive supervision acts on a pooled projection while the policy receives full content tokens, is there evidence that matched appearances produce more consistent action predictions? Better image matching does not necessarily mean the robot's actions become less sensitive to appearance. This would help connect the improved state-retrieval results to the policy's robustness.
  3. The method describes background and table appearance interventions, while the introduction also mentions task-irrelevant clutter. Could you specify the factors actually varied in each experiment and the randomizations enabled during Table 3's evaluation? For the real-robot experiments, how are appearance-matched training pairs obtained, and which evaluation disturbances are represented in those pairs?
  4. Could you report variability across training seeds for the key comparisons, particularly Table 4's 1.44 percentage-point improvement, and uncertainty estimates for the real-robot results? These would help assess whether the gains are reliable given the limited physical evaluation.
5 comments
AuthorsOpenReview ↗

Response to Reviewer r5e5 - Thank you for reviewing our paper!

Thank you for your careful review! Following your comments, we have added a matched ablation for CausalWAM-Joint, an action-level analysis, and repeated evaluations for the training and real-robot results. We have also clarified how the paired observations are generated and used. The changes are highlighted in blue in the revised manuscript.

[W1, Q1] Contribution of contrastive supervision in CausalWAM-Joint

Following your suggestion, we trained CausalWAM-Joint with the contrastive loss disabled, keeping the architecture, initialization, trainable modules, native training data, and schedule unchanged. The results are:

Training demonstrationsContrastive lossClean %Random %
Clean onlyOff79.244.80
Clean onlyOn82.8234.74
Clean + RandomOff92.0292.16
Clean + RandomOn93.1893.74

In the Clean-only setting of Table 3, contrastive supervision improves Random success from 4.80% to 34.74% +299˙4percentagepoints+29\.94 percentage points. In the Clean+Random setting, it improves Random success from 92.16% to 93.74% +15˙8points+1\.58 points. These results show that the contrastive objective contributes in both settings. The larger gain in the Clean-only setting is expected: without randomized demonstrations for native action and video training, the paired views provide the model with an explicit signal that the same physical state can appear under different backgrounds and tabletop appearances. The training objectives are defined in Section 4.4, and the matched training settings are detailed in Appendix B.1.

[W2, Q2] Generalization beyond appearance changes and action consistency

We thank the reviewer for raising these two important questions. We discuss generalization beyond the training interventions and the connection between representation alignment and action consistency below.

(1) Generalization beyond appearance changes.

As described in Section 3, our training objective uses observations of the same physical state under different appearances as positive pairs, while observations of different physical states within the same task serve as negatives. This encourages the content reader to capture manipulation-relevant information while reducing sensitivity to appearance variations. Meanwhile, the action expert retains its original visual conditioning and proprioceptive inputs.

The benefits extend beyond the appearance changes used during training. On LIBERO-Plus, which evaluates seven unseen perturbation types, CausalWAM-Joint improves average success from 51.5% to 79.4%, as reported in Table 2. Gains are also observed under Layout, Language, and Robot changes, with success increasing from 60.7% to 80.5%, 68.9% to 76.9%, and 44.5% to 46.3%, respectively. On real robots, CausalWAM-Joint further improves success under visual disturbances from 15.6% to 46.7% in Fig. 6.

(2) From representation alignment to action consistency.

Our original results in Fig. 4 show that CausalWAM-Interface improves cross-appearance physical-state retrieval across 50 RoboTwin tasks, raising mean R@1 from 75.22% to 84.64%, a gain of 9.42 percentage points. Following your suggestion, we further added Fig. 5 to examine whether this improvement is reflected in action predictions.

We compare predictions for matched observations of the same physical states across appearances, keeping the instruction, robot state, and sampling noise fixed. The results are summarized in Fig. 5(b):

MetricFastWAMCausalWAM-InterfaceImprovement
Action discrepancy (×10⁻⁴) ↓3.8292.345−38.75%
Expert action error (×10⁻²) ↓6.2755.642−10.07%
Random success (%) ↑91.7893.62+1.84 pp

Fig. 5(a) further shows the per-task results across 50 RoboTwin tasks. CausalWAM-Interface generally achieves lower action discrepancy and expert action error, with both metrics improving together.

Taken together, Fig. 4 and Fig. 5 connect improved cross-appearance state representations with more consistent and accurate action predictions. The results in Table 2 and Fig. 6 further demonstrate the benefits of CausalWAM under diverse perturbations in simulation and real-world manipulation.

[Q3] Appearance-pair construction and evaluation settings

For RoboTwin, we obtain official Clean and Random demonstrations through trajectory replay. Clean-only training uses Clean demonstrations, while Clean+Random training uses both for action and video prediction. Separately, we restore states from Clean training trajectories and re-render them with different wall and table textures, producing four matched views (C/R1/R2/R3) with identical physical states, instructions, and action labels. These paired views provide contrastive supervision in our original training protocol, as described in Section 4.4.

For LIBERO, we similarly re-render recorded simulator states with changes to backgrounds, tabletop materials, or lighting. For real-robot training, we generate matched views through mask-constrained image editing, varying backgrounds and tabletop textures or adding distractors outside task-relevant regions while preserving the original robot–object configuration and action labels. For real-world experiments, we generate training pairs through masked edits of backgrounds, textures, and distractors while preserving task-relevant regions, and evaluate these variations across three manipulation tasks in Fig. 6 and Section 5.4.

For Table 3, we follow RoboTwin 2.0's Clean and Randomized evaluation protocols. The Randomized setting varies wall and table textures, tabletop clutter, lighting, and table height, with dynamic lighting applied at a probability of 2%. Head-camera position and robot embodiment remain fixed. The evaluation thus covers both the texture variations used in paired training and additional perturbations not included in paired-view construction.

[Q4] Variation across training seeds and real-robot trials

We ran CausalWAM-Interface with three independent training seeds. The results in Table 4 are reported as mean ± standard deviation:

CausalWAM-InterfaceClean %Random %
Without contrastive loss λ=0λ = 092.46 ± 0.0892.18 ± 0.06
With contrastive loss λ=05˙λ = 0\.593.24 ± 0.0493.62 ± 0.06

The Random improvement is 1.44 percentage points. The standard deviations across the three runs are small.

We also repeated each real-robot task for three rounds, with 15 trials per round. The results below report the mean and standard deviation across rounds.

TaskModelRound 1Round 2Round 3Success %
OOD PnP FruitsFastWAM3/151/154/1517.78 ± 10.18
OOD PnP FruitsCausalWAM-Joint13/1513/1512/1584.44 ± 3.85
OOD Pen to HolderFastWAM2/151/152/1511.11 ± 3.85
OOD Pen to HolderCausalWAM-Joint4/153/155/1526.67 ± 6.67
OOD Fold and Flatten TowelFastWAM2/153/150/1511.11 ± 10.18
OOD Fold and Flatten TowelCausalWAM-Joint4/155/154/1528.89 ± 3.85

CausalWAM-Joint improves success in all three OOD tasks. We have added these repeated results to Fig. 5 and the revised real-robot evaluation.

As the discussion period is drawing to a close, we would be willing to address any remaining questions or concerns about our work. If there is anything further we can clarify, we would greatly appreciate your feedback before the discussion closes.

Thank you again for your time and consideration.

Best wishes,

Authors of Submission #18615

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions. We have conducted additional experiments and revised the manuscript to address the questions raised during the review.

1. Summary of Recognized Strengths

  • Motivation: Addressing the tension between appearance-sensitive video prediction and appearance-invariant robot control is a meaningful and well-motivated problem (r5e5, RnDr).
  • Method: The action-preserving intervention design provides explicit state correspondence across appearances, while the shared content reader incorporates this supervision into action generation without replacing the original visual pathway (r5e5, RnDr, mYq6).
  • Experimental evidence: The improvements on LIBERO-Plus (51.5% to 79.4%), cross-appearance state retrieval (75.22% to 84.64%), and real-robot manipulation provide complementary evidence of visual robustness (r5e5, RnDr, mYq6).

2. Summary of Key Concerns and Our Responses

  • Contribution of contrastive supervision (r5e5, mYq6): We added a matched CausalWAM-Joint ablation with contrastive learning disabled. Under Clean-only training, enabling contrastive learning raises Random success from 4.80% to 34.74% (+29.94 pp), with the architecture, initialization, action/video supervision, and training schedule unchanged.

  • Paired-image augmentation versus contrastive learning (RnDr, mYq6): We trained FastWAM and Joint on the same paired images with inherited action labels. FastWAM achieves 5.46% Random success, compared with 6.24% for Joint without contrastive learning and 34.90% with contrastive learning. These matched comparisons show that the improvement cannot be explained by additional action-labeled images alone.

  • Training data and supervision protocols (RnDr): We distinguished official Random demonstrations from paired renderings of Clean states and clarified the data used for each loss. In Clean-only training, both action and video prediction use official Clean demonstrations; paired views provide contrastive supervision and are not video-prediction targets. We also compared paired-image action supervision under both Clean-only and Clean+Random training.

  • Representation alignment and action consistency (r5e5): We added action-level analysis across 50 RoboTwin tasks. CausalWAM-Interface reduces cross-appearance action discrepancy by 38.75% and expert action error by 10.07% (Fig. 5), complementing the state-retrieval improvements in Fig. 4.

  • Choice of representation objective (mYq6): We compared SupCon with paired-feature MSE using matched training settings. SupCon achieves 34.74% Random success versus 25.48% for MSE, supporting the benefit of distinguishing physical states rather than only aligning paired features.

  • Generalization and evaluation reliability (r5e5): We clarified that RoboTwin paired training varies wall and table textures, while Randomized evaluation additionally includes clutter, lighting, and table-height changes. We also reported three independent training seeds for the Interface ablations and three evaluation rounds for each real-robot task. The aggregated real-robot OOD success improves from 13.3% to 46.7%.

3. Summary of Manuscript Revisions

Section / LocationAdded Content and Revisions
Section 4.4 and Appendix B.1Clarified the distinction between official demonstrations and paired renderings, their use in action/video/contrastive supervision, and the matched training settings.
Tables 1 and 4Added Joint results under Clean+Random training, three-seed Interface ablations, and clarified the default-setting and best-result formatting.
Tables 5 and 6Added matched Joint ablations, paired-image action-supervision controls, the FastWAM baseline, and the SupCon versus paired-feature MSE comparison.
Figures 4–5 and Appendix CAdded action-level consistency analysis alongside cross-appearance state retrieval, with detailed evaluation protocols.
Figure 6, Section 5.4, and Appendix Table 9Updated real-robot results using three OOD evaluation rounds, including per-round statistics and consistent aggregate success rates.

These additional experiments directly address the central concerns raised during the review. We also appreciate the positive follow-up assessments from reviewers r5e5 and mYq6, who acknowledged the value of the new controlled comparisons.

The corresponding changes are highlighted in blue in the revised manuscript.

Thank you again for your time and consideration.

Best regards,

Authors of Submission #18615

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.