acceptodds
Withdrawn from ICLR 2027

RAC-WAM: Region-Wise Adversarial Consistency for Visual Robustness in World-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World action models (WAMs) jointly predict robot actions and future visual states, with future prediction providing supervision beyond action labels. Yet their performance still degrades notably under visual distribution shifts such as changes in camera viewpoint or scene appearance. Adversarial consistency training offers an effective way to improve robustness, but existing methods typically use action-only consistency and perform global perturbation search over the full observation. These choices are not fully aligned with WAMs: action-only consistency may miss vulnerabilities exposed by the future-video branch, while global search may repeatedly exploit the most sensitive evidence, leaving other semantic regions underexplored. To address these issues, we propose RAC-WAM, a region-wise adversarial consistency framework for robust WAMs. RAC-WAM jointly leverages action and future-video discrepancies to guide adversarial search. To broaden perturbation coverage across camera views and semantic regions, RAC-WAM performs hierarchical region-wise search by sampling a camera scope and constructing separate perturbations for robot, object, and background regions. To account for regional sensitivity differences under a conserved perturbation budget, perturbation magnitudes are allocated across regions according to local sensitivity. Experiments on LIBERO-Plus, RoboTwin 2.0, and a Franka robot demonstrate the effectiveness of RAC-WAM: it improves zero-shot success on LIBERO-Plus from 72.2% to 82.8% over a same-backbone baseline while preserving in-distribution performance, and delivers consistent robustness gains in bimanual and real-world manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Robustness, Adversarial Training, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zBEcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RAC-WAM, a training procedure for improving world-action models under visual distribution shifts. Starting from Fast-WAM-Joint, it penalizes changes in action and future-video flow-velocity predictions between clean and perturbed observation latents while holding other conditioning variables fixed. It samples an all-camera, head-camera, or wrist-camera scope, separately searches for perturbations in robot, object, and background regions, and allocates their RMS radii using local gradient sensitivity under an area-weighted budget. Robot-region perturbations use video consistency alone; object and background perturbations use both branches. An auxiliary future-mask prediction loss supplies additional semantic supervision. These additions are removed at deployment. The reported evaluation covers LIBERO/LIBERO-Plus, six randomized RoboTwin 2.0 tasks, and three physical manipulation tasks. The appendix adds regional sensitivity measurements, future-video fidelity results, and a random-latent perturbation control.

Critical strengths

A substantial controlled robustness improvement with useful breadth of validation. Table 1 compares against the direct Fast-WAM-Joint baseline and reports a 10.6 percentage-point LIBERO-Plus improvement, including 17.9 points for camera shifts, 27.2 for backgrounds, and 20.5 for noise, alongside preserved standard LIBERO performance. Positive results in RoboTwin and physical manipulation (Tables 2-3) make the contribution more convincing than a single corruption benchmark. Removing the masks, auxiliary head, and adversarial search at deployment makes these gains practically relevant.

The paper includes controls that meaningfully test the training ingredients. Table 4 makes clear that mask supervision already contributes 5.3 points, avoiding attribution of the entire gain to adversarial search. Table 8 also measures future-video fidelity rather than inferring video-branch robustness solely from action success. These controls provide substantive evidence beyond the main benchmark table.

Critical weaknesses

The evidence is insufficiently precise to establish the smaller gains claimed for the central design choices. Adding regional coverage and sensitivity allocation raises the aggregate success from 81.1% to 81.9% to 82.8% (Table 4); the full method exceeds action-only consistency by 1.5 points (Table 5). The paper reports no training-seed variation or uncertainty intervals for these comparisons. The action-only row also does not specify its robot action coefficient, although that coefficient changes the objective and potentially the sensitivity-based allocation. Thus, the results support the overall training recipe more strongly than they establish a robust incremental benefit from each proposed mechanism. This matters because semantic mask supervision and action consistency account for much of the headline improvement.

Regional search changes training computation as well as perturbation coverage, leaving an important alternative explanation uncontrolled. Equations (12)-(14) explicitly optimize and evaluate each active region separately, with three ascent steps per region (Table 10). The regional method therefore performs multiple adversarial searches when several regions are present, whereas the global-search ablation does not have this same structure. The conserved perturbation-energy identity in Eq. (11) controls magnitude, not the number of searches or training cost. The paper gives hardware and epochs, but no runtime/memory overhead or comparison to a global baseline given an equivalent search budget. Consequently, Table 4 does not isolate whether the regional gain comes from its semantic coverage or from the changed amount of search and regularization. The random-direction control addresses whether model-guided directions help, but does not resolve this particular comparison. This limits both attribution and assessment of the method's practical training tradeoff.

Questions for authors

How many independently trained seeds and evaluation rollouts underlie the LIBERO-Plus ablations, and are rollout initializations paired across methods? Please provide uncertainty for the 81.1/81.9/82.8 comparison and the 81.3/82.8 action-only/full comparison. In Table 5, what is alpha_robot for the action-only row? If it is zero, how are the zero robot objective/sensitivity and associated allocation handled; if it is one, what evidence separates the effect of adding video consistency from removing robot action consistency? These answers could change the assessment of the main design-specific contribution.

What are the exact projected-ascent update, step size, and gradient normalization, and what are the measured training runtime and memory overhead relative to the global joint-consistency variant? Has a global variant with a comparable number of searches/forward-backward evaluations been evaluated? A comparison at matched search computation would materially clarify the interpretation of Table 4; this is a control for the present claim, not a request for a new research direction.

What data or validation protocol selected the perturbation budget, video weight, sensitivity exponent, and αrob\alpha_{rob} rule in Table 10 and Table 5? In particular, were the reported LIBERO-Plus test conditions used for these choices? This matters for interpreting the result as zero-shot robustness to unseen shifts rather than robustness tuned on the reported evaluation set.

Reviewer_SJpJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the robustness of World-Action Models (WAMs) under visual distribution shifts such as camera viewpoint, background, lighting, and sensor noise. RAC-WAM proposes dual-branch discrepancy, camera/semantic region-wise perturbation search, and sensitivity-guided perturbation budget allocation. In addition, the method introduces auxiliary future-mask supervision.

Critical strengths
  1. The paper not only demonstrates significant improvements on LIBERO-Plus but also includes dual-arm operation experiments using RoboTwin 2.0 and experiments with a real Franka robot.

  2. Experiments involving multiple VLA backbones were conducted.

Critical weaknesses

W1. Insufficient Evidence for the Independent Contribution of Region-Wise Adversarial Consistency.

Although region-wise adversarial consistency is presented as a central contribution, Table 4 suggests that much of the improvement comes from mask supervision and non-region-wise adversarial consistency, while region coverage and sensitivity-guided allocation contribute only 0.8 and 0.9 pp, respectively. More critically, the proposed method performs independent PGD searches for each active semantic region, potentially requiring substantially more gradient evaluations and adversarial forward passes than the global baseline. Without compute-matched global PGD and random or non-semantic partition baselines, it remains unclear whether the gains arise from semantic region decomposition or simply additional optimization and perturbation diversity. Moreover, the sequential ablation does not fully isolate component interactions. These limitations make it difficult to attribute the reported improvements to the paper's central methodological contribution, substantially weakening its empirical justification.

W2. Insufficient Justification and Empirical Validation of Region-Specific Consistency Objectives.

The region-specific consistency objectives lack a convincing theoretical justification. In particular, Eq. (9) sets (\alpha_{\mathrm{rob}}=0), disabling action consistency for robot-region perturbations. However, these perturbations are applied in the observation latent space without changing the underlying physical robot configuration or proprioceptive inputs. It is therefore unclear why action invariance should no longer be encouraged, or how the perturbed visual representation remains consistent with unchanged proprioception. Furthermore, Table 5 shows that video-only consistency slightly underperforms mask supervision, while the full method improves upon action-only consistency by only 1.5 pp without multi-seed validation. Together, these issues raise substantial concerns about whether the proposed region-specific objectives are conceptually well-founded and whether their reported benefits are robust and reproducible.

Questions for authors

As weaknesses

Reviewer_6hB3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAC-WAM, a training-time method that perturbs semantic regions of a world-action model and matches action and future-video predictions under those perturbations. Inference is unchanged. The reported gain is better zero-shot robustness than the same backbone.

Critical strengths
  1. The authors improve LIBERO-Plus over a same-backbone baseline while keeping in-distribution LIBERO performance.
  2. The authors leave the inference pipeline and its cost unchanged.
  3. The authors measure a clear sensitivity gap across robot, object, and background regions before building the method.
Critical weaknesses
  1. The authors claim the 72.2% to 82.8% gain for region-wise action--video consistency, but Table~4 gives +5.3 to mask supervision and +3.6 to non-regional consistency, and only +1.7 to region coverage plus sensitivity allocation. The titled method is a small add-on to mask loss and standard adversarial consistency.
  2. The authors say global search over-uses sensitive regions, then set Equation~11 (ψ=0.5\psi=0.5) to give those regions a larger budget. The rule contradicts the motivation and is worth only +0.9, with camera dropping from 69.0% to 67.2%.
  3. The authors treat the video branch as necessary, but video-only consistency falls to 77.1% and joint consistency with αrob=1\alpha_{\mathrm{rob}}=1 loses to action-only (80.8% vs.\ 81.3%). αrob=0\alpha_{\mathrm{rob}}=0 is a hand-set switch on a stop-gradient consistency loss, and proprioception is never attacked.
  4. The authors do not show that the method is robust outside one benchmark. Real-robot cells use 20 trials with no intervals, clean success jumps from 61.7% to 81.7%, and Table~3's ``noise'' is Gaussian blur. RoboTwin moves only from 94.5% to 97.8% with no ablation. Mask error is never tested, and cited mask-based and adversarial VLA baselines are missing.
Questions for authors
  1. Table~4 gives +5.3 and +3.6 to mask loss and non-regional consistency, but only +1.7 to the titled components. Why is 72.2% to 82.8% credited to region-wise action--video consistency?
  2. Why does Equation~11 (ψ=0.5\psi=0.5) give more budget to the more sensitive region? What happens for ψ=0\psi=0 and ψ<0\psi<0?
  3. Video-only falls to 77.1%, and αrob=1\alpha_{\mathrm{rob}}=1 loses to action-only (80.8% vs.\ 81.3%). What are the scales of Δact\Delta^{\mathrm{act}} and Δvid\Delta^{\mathrm{vid}}, and why set αrob=0\alpha_{\mathrm{rob}}=0?
  4. How does RAC-WAM compare with MaskWAM and global adversarial VLA training on this backbone? On Franka, what is the robust gap after controlling for the clean jump from 61.7% to 81.7%?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.