acceptodds
Under review as a conference paper at ICLR 2027

PICA-WAM: Perturbation Invariance via Cross-Appearance Self-Alignment in World Action Models

Abstract

World Action Models (WAMs) jointly learn video prediction and action generation, yet remain fragile under visual perturbations that alter appearance without changing the task semantics or required actions. We identify a key source of this fragility in the different representation required by the two objectives: video prediction favors fidelity to visual appearance, whereas reliable action prediction requires invariance to task-irrelevant appearance changes. Existing approaches introduce external pretrained representation encoders. However, external feature alignment do not explicitly enforce consistency between the representations induced by nominal and perturbed observations. To address this gap, we propose PICA-WAM, a cross-appearance self-alignment objective that promotes perturbation invariance in WAMs. PICA-WAM aligns earlier-layer representations induced by perturbed observations with stable targets from later-layer representations induced by the corresponding nominal observations, explicitly enforcing cross-appearance consistency during training with no inference-time overhead. Experiments across diverse benchmarks demonstrate consistent robustness gains under visual perturbations, without compromising nominal performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action models, embodied manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Y1mWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose PICA-WAM, a self-supervised training method to improve the robustness of WAM under visual perturbations. An EMA copy of the model reads the nominal view at a lower noise level and provides a later-layer action feature as the target; the online model reads a perturbed view of the same state and aligns its earlier-layer action features to this target. The teacher and projection head are discarded after training, so there is no extra cost at inference.

Critical strengths
  1. To my knowledge, this is the first work to introduce cross-appearance self-alignment for WAMs, without relying on an external pretrained encoder.

  2. The paired diagnostic protocol (layer-wise sensitivity, layer-band intervention, trajectory drift) clearly explains why augmentation-only training fails, so the motivation is supported by experiments rather than intuition.

Critical weaknesses
  1. Some key hyperparameters (eg. m, n) are only given in the appendix without any sensitivity analysis.

  2. The ablation of the alignment design is incomplete. Table 4 only shows the effect of removing the appearance gap as a whole in "+SRA", without verifying the individual contributions of the noise-level gap and the layer gap in the WAM setting, or analyzing how EMA teacher quality on nominal views affects the alignment signal.

Questions for authors
  1. How sensitive is PICA-WAM to the choice of student layer m, teacher layer n, noise offset, and loss weight? And does aligning deeper action-stream layers corresponding to the main leakage region work better?

  2. What are the individual contributions of the noise-level gap and the layer gap when keeping the appearance gap in the WAM setting?

  3. Can the authors provide an analysis of the EMA teacher quality on nominal views and how different teacher training stages or qualities affect the alignment benefit?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for your thoughtful comments and questions.

W1. Some key hyperparameters (eg. m, n) are only given in the appendix without any sensitivity analysis.

Q1. How sensitive is PICA-WAM to the choice of student layer m, teacher layer n, noise offset, and loss weight? And does aligning deeper action-stream layers corresponding to the main leakage region work better?

The layer ablations support aligning an early student layer to a later teacher layer, with gains that are not confined to our default layer pair. Across all six tested pairs, success ranges from 83.36% to 86.10%, compared with 81.09% for augmentation-only training. We evaluated each configuration on the same 10,030 LIBERO-Plus tasks.

First, we varied the student and teacher layers while fixing the maximum noise gap at 0.2 and the alignment weight at 0.05:

Student layer mTeacher layer nOverall success (%)
42083.49
82085.17
142083.36
81486.10
82685.69
202683.63

The results favor an early, but not necessarily the earliest, student alignment layer: with teacher layer 20 fixed, student layer 8 outperforms both layers 4 and 14. With teacher layer 26 fixed, moving the student from layer 8 to layer 20 reduces success from 85.69% to 83.63% (−2.06 pp). Thus, directly placing the loss deeper in the action stream corresponding to the main leakage region does not improve control in these comparisons, supporting our choice to supervise earlier action features. The exact teacher depth is more flexible: with student layer 8 fixed, all three later teacher layers achieve 85.17–86.10%. A useful alignment target therefore does not require the particular teacher layer used in our default configuration.

We next examined the noise gap and alignment weight with student/teacher layers fixed at 8/20. The table summarizes selected noise-gap settings with weight 0.05, and the weight sweep with maximum noise gap 0.2.

Parameter variedValues, in orderOverall success (%), in the same order
Maximum sampled noise gap0 / 0.2 / 0.484.81 / 85.17 / 86.13
Alignment weight0.01 / 0.05 / 0.184.90 / 85.17 / 81.50

The weight results support using alignment as a moderate auxiliary constraint. A change from 0.01 to 0.05 changes success by only 0.28 pp, with both settings substantially outperforming augmentation-only training (+3.81 and +4.09 pp). Increasing the weight further to 0.1 reduces the gain, so stronger alignment is not necessarily better.

For the noise gap, both 0.2 and 0.4 yield substantial gains over augmentation-only training (+4.09 and +5.04 pp), showing that the benefit is not tied to our default value. Even with zero noise gap, cross-appearance alignment retains a +3.72 pp gain. These comparisons distinguish the contribution of cross-appearance supervision from the additional benefit of the noise gap: the former remains useful without a noise offset, while the latter can further improve success, as demonstrated by the 0.4 setting.

We have added this analysis to Section 4.4, “Layer and hyperparameter sensitivity,” with the explicit configurations and success rates in Appendix C, Table 10.

W2a. The ablation of the alignment design is incomplete. ... without verifying the individual contributions of the noise-level gap and the layer gap in the WAM setting.

Q2. What are the individual contributions of the noise-level gap and the layer gap when keeping the appearance gap in the WAM setting?

Removing either gap alone retains most of the overall success, but reduces robustness on particular evaluation dimensions, and removing both gives a larger overall decrease. We kept the appearance gap in all four conditions and fixed the alignment weight at 0.05 and EMA decay at 0.9999. The evaluation protocol is the same as above.

Alignment configurationStudent / teacher layersMaximum noise gapOverall (%)Camera (%)Sensor Noise (%)
Both gaps (default)8 / 200.285.1764.6794.63
No noise gap8 / 20084.8163.2391.63
No layer gap8 / 80.284.4559.9791.19
Only appearance gap8 / 8083.7760.2390.01

With the layer gap retained, adding the noise-level gap improves Sensor Noise success by 3.00 pp and overall success by 0.37 pp. With the noise-level gap retained, adding the layer gap improves Camera success by 4.69 pp, Sensor Noise success by 3.44 pp, and overall success by 0.73 pp. Using both gaps achieves the highest overall success among these four configurations, improving on appearance-only alignment from 83.77% to 85.17% (+1.41 pp). These results support retaining both gaps: the noise-level gap helps sensor-noise robustness, while the layer gap benefits both camera-viewpoint and sensor-noise robustness.

We have added these findings and results to Section 4.4, Table 5.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Thank you for your thoughtful comments and questions.

W2b. The ablation of the alignment design is incomplete. ... analyzing how EMA teacher quality on nominal views affects the alignment signal.

Q3. Can the authors provide an analysis of the EMA teacher quality on nominal views and how different teacher training stages or qualities affect the alignment benefit?

To examine how teacher quality and construction affect alignment on the nominal view, we compare three teacher regimes on the same 10,030 LIBERO-Plus tasks: using the online live student without EMA smoothing, using a frozen converged +Aug checkpoint as teacher, and using our default EMA teacher with decay 0.9999.

Teacher configurationOverall success (%)Gain vs. +Aug (pp)
+Aug, no alignment81.09+0.00
Alignment with live teacher, no EMA83.63+2.54
Alignment with frozen +Aug teacher83.40+2.31
Default EMA teacher, decay 0.999985.17+4.09

Both alternative teachers still improve substantially over unaligned +Aug, reaching 83.63% (+2.54 pp) with a live online teacher and 83.40% (+2.31 pp) with a frozen converged +Aug teacher. Alignment therefore does not depend on EMA smoothing to remain effective, and even a fixed converged +Aug teacher supplies a useful nominal-view target. At the same time, the default co-evolving EMA teacher reaches 85.17% (+4.09 pp), outperforming both the live teacher and the frozen +Aug teacher. Together, they show that nominal-view supervision from either an online or converged teacher improves robustness, while slowly updated EMA targets deliver the strongest gain.

We have added these teacher-regime controls and their scope to Section 4.4 and Appendix C.

Thank you for the thorough follow-up. Your responses and added experiments clarified my remaining questions, so I will maintain my support for the paper.

Reviewer_mh7aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper addresses making world action models more robust across a series of perturbations to the environment. The core claim in this paper seeks to make action representations robust to visual perturbations by introducing a range of data augmentations during training and the general idea is to make the action prediction invariant to these perturbations while making video generation sensitive to it. The authors then show this is not enough to prevent action invariant information leakage into action prediction. The authors then verify these claimss through emperical experiments showing influence of action invariant information on action prediction. To prevent this authors propose cross appearance selff alignment objective. Paper shows significant gains in sim and real tasks.

Critical strengths
  1. Paper in general is easy to follow. The claims made are mostly clear on first reading and experriments to verify the claims are clearly stated .
  2. Ablations are clear and mostly present clearly the benifit of alignment and augmentations usage.
  3. The training recipe shows decent gains across robotwin and Libero-Plus benchmark
Critical weaknesses
  1. The core claim on wich the effectiveness of paper rests - "video prediction favors fidelity to visual appearance, whereas reliable action prediction requires invariance to task-irrelevant appearance changes". The paper does not identify clearly what are those task invariant changes. The listed augmentation adds perturbations to hue,light and other visual aspects. My main issue summarized with a toy example - "Suppose we have to blocks one blue and one dark blue and the task is "Pick dark blue block". Adding visual perturbations to hue could potentially change the appearance of blue to dark blue and dark blue to black which would require changes in actions, contradictory to the core claim of the paper. Appendix A implements T using brightness, contrast, saturation, hue, gamma, exposure, crop/rotation, blur, and fog (p. 14, lines 704-713). Keeping a simulator state fixed does not by itself establish that every resulting image preserves an identifiable referent for the instruction. As the rotation invarriance would depend highly on the reference frame in which actions are predicted which is not stated. Since no study is provided on how perturbations still maintain these features. Ddue to this paper lacks significant potion of verifying this.

  2. The unseen family and OOD interpretation iss not separated from augmentation applied during training making some of the pertubations seen during training.The diagnostic calls glass blur, lighting, background, and camera viewpoint unseen families (p. 6, lines 297-299), and the introduction and conclusion generalize the results to perturbation families unseen during training (p. 2, lines 080-086; p. 9, lines 483-485). Yet the training distribution contains brightness, hue, gamma, exposure, crop/rotation, and several optical corruptions. . It limits the interpretation of blanket unseen-family and cross-method OOD comparisons.

  3. The paper normalizes nominal/perturbed feature distance by between-state distance and states that this removes cross-layer and cross-model scale differences (p. 6, lines 290-293; p. 15, lines 770-776). This is a useful reference scale, but its interpretation is not unique. The ratio can fall because perturbation distance decreases, because the state-distance denominator increases, or both. As a counterexample the learned representation can retain an appearance component unchanged while increasing separation along task-state dimensions by a large factor. Its ratio then decreases without losing that appearance information.

  4. The intervention and drift measurements establish limited forms of influence and consistency, not necessarily correlation beteen leakage and worse downstream performance.

  5. Real world evaluations are really limited both in terrms of tasks and perturbations for unseen evaluations which are aligned with the perturbations introudced in the paper.

Questions for authors
  1. How is semantic/task preservation established for the actual augmentation ranges. Can the model retain instruction-relevant color and relation distinctions while rejecting nuisance changes
  2. Can you provide an exact train/test perturbation map and disaggregated results for genuinely held-out families, including a distinction between training photometric augmentation and test lighting?
  3. Equation 3 does not establish the full-representation invariance described in Equation 2.The text says matching the teacher requires the student's action features to discard what T changes, which is identified with Equation 2 (p. 5, lines 224-229). However, Equation 2 compares unprojected features of the same student, layer, and noise level across appearances (p. 4, lines 187-191), whereas Equation 3 compares a learned projection of an earlier student feature with a later EMA-teacher feature at a different video noise level
9 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/4)

Thank you for your thoughtful comments and questions.

W1a. The core claim on wich the effectiveness of paper rests - "video prediction favors fidelity to visual appearance, whereas reliable action prediction requires invariance to task-irrelevant appearance changes". The paper does not identify clearly what are those task invariant changes. ... Keeping a simulator state fixed does not by itself establish that every resulting image preserves an identifiable referent for the instruction.

Q1a. How is semantic/task preservation established for the actual augmentation ranges.

We define a perturbation T as task-irrelevant when (1) the ground-truth action trajectory remains valid for the perturbed observation and instruction, and (2) the target referent remains distinguishable from distractors and background under T. We agree that fixing the simulator state alone does not establish referent preservation for arbitrary transforms: unbounded hue rotations, channel permutations, or extreme contrast collapse would violate (2) and fall outside our claim. Our claim is therefore scoped to the bounded training transforms in T.

Specifically, our training photometric augmentation shifts hue within ±0.04, scales brightness and contrast in [0.7, 1.3], saturation in [0.75, 1.25], gamma in [0.8, 1.25], exposure within ±0.25 EV, and adds Gaussian noise with σ = 0.015, alongside the bounded spatial, blur, and fog transforms in T detailed in W1b and W2 below. Testing these exact ranges shows:

First, to test the reviewer's toy example directly, we construct a grey-table scene with a blue block and a dark-blue block, covering both pure RGB shades and softer rendered-like blue shades, and check whether photometric augmentation turns dark blue black or shrinks their color gap below one just-noticeable difference [1], the standard human perceptual threshold where two colors become barely distinguishable at CIELAB Euclidean distance ΔE < 2.3. Across all min/max combinations of our 6 photometric parameters, dark blue never turns black, keeping its brightest RGB channel at least 0.179 compared with 0 for black, and never becomes lighter than blue (0 lightness inversions). Across 20,000 random training draws, color separation drops below ΔE = 2.3 in only 0.06% of pure-RGB draws (1 lightness inversion) and 0.00% of rendered-like draws (0 inversions).

Second, to verify semantic preservation in real scenes under full T, including blur, fog, and crop/rotation, we audit all 40 standard LIBERO tasks using per-object segmentation masks warped identically under crop/rotation, comparing each target referent against all other objects in the scene. Under random training draws of T, only 1.31% of originally color-distinct pairs with ΔE ≥ 2.3 drop below ΔE = 2.3. Spatial cues also remain intact: crop/rotation reverses clear left-right ordering in only 0.07% and up-down ordering in 0.14%.

Together, these results establish semantic preservation under our actual training augmentation ranges: bounded T retains the target referent's color and spatial distinguishability across both synthetic color extremes and all 40 LIBERO scenes.

We have updated Section 3.2 to define task-irrelevant perturbations and scope our invariance claim to bounded training transforms, and added this stress test and 40-task LIBERO scene audit to Appendix E.

[1] M. Mahy, L. Van Eycken, and A. Oosterlinck. Evaluation of uniform color spaces developed after the adoption of CIELAB and CIELUV. Color Research & Application, 19(2):105–121, 1994. DOI: 10.1111/j.1520-6378.1994.tb00070.x

W1b. As the rotation invarriance would depend highly on the reference frame in which actions are predicted which is not stated.

Both benchmarks predict actions in robot/world coordinates rather than camera coordinates: LIBERO outputs 7-D world-frame OSC delta end-effector poses, and RoboTwin 2.0 outputs 16-D dual-arm end-effector poses. We apply spatial augmentation only with a probability of 0.25 per camera clip: crop to 92–100% of the original side length, resize back to the input resolution, and rotate within ±8°. The robot and objects undergo the same image transformation while their physical relationship stays unchanged, so the required action in world/robot coordinates remains the same. By contrast, predicting actions in camera coordinates would require equivariant transforms of the target actions.

We have added these benchmark action reference frames and the camera-frame equivariance distinction to Section 4.4, Appendix A, and Appendix D.

AuthorsOpenReview ↗

Official Comment by Authors (2/4)

Thank you for your thoughtful comments and questions.

W1c. Since no study is provided on how perturbations still maintain these features. Ddue to this paper lacks significant potion of verifying this.

Q1b. Can the model retain instruction-relevant color and relation distinctions while rejecting nuisance changes.

Beyond showing in W1a and Q1a that the training perturbations T preserve referents in the input, we further test whether PICA-WAM rejects nuisance visual changes while keeping instruction and color distinctions in two studies.

First, to test whether alignment suppresses instruction sensitivity alongside visual noise, we evaluate 170 initial states across 17 LIBERO scenes, spanning LIBERO-10 color-named mugs, LIBERO-Goal, and LIBERO-Object. We compare two edits: applying nuisance perturbations via eight random draws of T on the same scene and instruction, versus applying an instruction swap that keeps the exact same image while replacing the instruction with another valid task or target in that scene. We measure the shift from nominal at the aligned action-stream Layer 8 using relative L2 feature distance ∥h′−h∥2/∥h∥2\lVert h' - h \rVert_2 / \lVert h \rVert_2, reported in percent. We then compute selectivity on each state as the instruction-swap shift divided by the nuisance-T shift, where values above 1 indicate that instruction changes outweigh visual noise.

Method across 170 LIBERO statesLayer 8 shift: nuisance T ↓Layer 8 shift: instr. swap ↑Median selectivity at Layer 8 ↑States with instr > T at Layer 8 ↑
Baseline31.05%18.41%0.5118.8%
+Aug8.59%21.02%3.3890.6%
PICA-WAM3.03%25.15%9.29100.0%

On Baseline, instruction swaps outweigh nuisance perturbations in only 18.8% of states, giving a median selectivity of 0.51. Relative to +Aug, PICA-WAM cuts Layer 8 nuisance shift nearly threefold from 8.59% to 3.03% while increasing instruction-swap sensitivity from 21.02% to 25.15%, raising median selectivity from 3.38 to 9.29 so instruction changes outweigh visual noise in all 170 states (100.0%).

Second, to test in closed-loop control whether alignment preserves color distinctions when shape and position cannot separate targets, we evaluate Fast-WAM with +Aug and with PICA-WAM on two RoboTwin 2.0 block tasks: blocks_ranking_rgb and stack_blocks_three. We choose these two tasks because both use geometrically identical red, green, and blue blocks with randomized initial positions, so color is the only cue for deciding which block goes where. Each task is evaluated over 100 trials in both the Clean and Randomized splits.

RoboTwin 2.0 color tasks (Fast-WAM, 100 trials/split)+Aug (%)PICA-WAM (%)Gain
blocks_ranking_rgb — Clean97.099.0+2.0 pp
blocks_ranking_rgb — Randomized27.060.0+33.0 pp
stack_blocks_three — Clean96.097.0+1.0 pp
stack_blocks_three — Randomized44.056.0+12.0 pp

On both color tasks, PICA-WAM lifts Randomized success by +33.0 pp on blocks_ranking_rgb and +12.0 pp on stack_blocks_three. Cross-appearance alignment thus preserves the RGB color cues needed for the task.

Together, these studies show that PICA-WAM rejects nuisance visual changes while keeping instruction selectivity and color-based manipulation.

We have added these selectivity and color-retention results to Section 4.2, Section 4.3, and Appendix F, Table 14.

AuthorsOpenReview ↗

Official Comment by Authors (3/4)

Thank you for your thoughtful comments and questions.

W2. The unseen family and OOD interpretation iss not separated from augmentation applied during training making some of the pertubations seen during training...

Q2. Can you provide an exact train/test perturbation map and disaggregated results for genuinely held-out families, including a distinction between training photometric augmentation and test lighting?

We have checked the manuscript and confirmed that we do not describe these experiments as an out-of-distribution (OOD) evaluation. Our claim in the submitted manuscript is that PICA-WAM “reduces both the appearance sensitivity of video features and its propagation into actions, including under perturbation families unseen during training.” Here, “unseen families” refers to perturbation-generating mechanisms absent from our training augmentation T, and it does not imply that the resulting appearances are OOD relative to the augmented training data.

Our training augmentation T operates on recorded images, whereas the Lighting, Camera, and Background evaluations modify simulator light parameters, 3D camera poses, and scene textures, respectively. These simulator operations are absent from T, although image augmentation may cover some of their visual effects.

For example, training photometric augmentation applies brightness/contrast factors of 0.7–1.3, saturation factors of 0.75–1.25, hue shifts within ±0.04, gamma values of 0.8–1.25, and exposure shifts within ±0.25 EV to recorded pixels. Test lighting instead changes the simulator's light positions, directions, and diffuse/specular RGB coefficients before rendering. Both can alter image brightness and color, but the simulator light-source changes are not part of T.

The table maps the eight visual families used in our diagnostics to T and reports their closed-loop success rates. These results are disaggregated from the same 10,030 LIBERO-Plus tasks.

Test familyCorresponding training operationTasks+Aug (%)PICA-WAM (%)
Gaussian blurGaussian blur, sigma 0.4–5 pixels, same family34189.4494.43
Motion blurDirectional blur, length 3–21 pixels, same family33688.1095.83
Zoom blurMaximum zoom 1.02–1.20, same family28884.0394.44
FogVeil strength 0.3–2.5, same family27284.1992.65
Glass blurLocal pixel shuffling absent, Gaussian smoothing overlaps36488.7495.33
LightingSimulator light changes absent, global photometry is used1,14293.2694.13
Camera viewpointSimulator camera-pose changes absent, 2D crop/rotation is used1,59962.1064.67
Background textureTexture replacement absent1,07674.1679.28

On the three simulator-generated families, PICA-WAM improves over +Aug by 0.88 pp for lighting, 2.56 pp for camera viewpoint, and 5.11 pp for background texture. These gains show that alignment trained through image augmentation also benefits perturbations introduced by changing the simulator.

We have specified the training augmentation in Section 4.1, with the complete augmentation ranges in Appendix A, and added the detailed train/test map and disaggregated results in Appendix D, Tables 11–12.

W3. The paper normalizes nominal/perturbed feature distance by between-state distance and states that this removes cross-layer and cross-model scale differences...

We agree that the ratio alone cannot establish removal of appearance information. The revised analysis shows that PICA-WAM reduces the relative appearance-induced feature change itself, so the lower ratio is not explained solely by increased cross-state distance.

We separate the original metric S = A/D into A, the relative feature change between perturbed and nominal views of the same state, and D, the relative feature distance between nominal views of different states within the same episode. Both are computed as L2 feature differences divided by the reference nominal feature norm.

MethodAction L8: A (%) ↓Action L8: D (%)Action L8: S (%) ↓Video L22: A (%) ↓Video L22: D (%)Video L22: S (%) ↓
Baseline4.947.3667.1534.1540.7883.75
+Aug2.8110.8725.8241.9640.12104.60
+Aug+SRA2.2719.8811.4231.9944.5471.82
PICA-WAM1.8117.3510.4326.5639.9366.51

At action layer 8, PICA-WAM reduces A by 35.5% compared with +Aug, and 20.3% compared with +Aug+SRA. At video layer 22, A falls from 41.96% for +Aug to 26.56% for PICA-WAM. Thus, our analysis of reduced appearance sensitivity also holds when using the relative feature change A directly, without normalization by the cross-state distance D.

We have updated Section 4.2, Table 1 with these results, and added their definitions and averaging protocol to Appendix B.

Reviewer_4mXPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The work tackles World Action Models (WAMs), which remain brittle under task-irrelevant visual perturbations because generative video prediction demands detailed appearance reconstruction, whereas control requires appearance invariance.

Standard data augmentation trains on clean and perturbed views independently, allowing visual noise to rebound in late video layers and leak into action representations.

To resolve this, the authors introduce PICA-WAM, a self-supervised framework where an EMA teacher processing nominal views at lower noise supervises an online student processing perturbed views.

Alignment is applied strictly to the action stream and incurs zero inference overhead since the teacher and projection heads are discarded post-training.

Across various benchmarks including LIBERO-Plus, RoboTwin 2.0, and an ARX-Lift2 dual-arm robot, PICA-WAM substantially improves against visual shifts while preserving nominal task performance.

Critical strengths

The paper pinpoints the core issue between video prediction (which requires appearance detail) and control (which demands invariance), resolving it by enforcing self-alignment strictly on the action stream. This setup eliminates appearance leakage into control without degrading video fidelity or introducing any inference overhead.

Instead of relying merely on task success, the authors design a three-tier diagnostic protocol that mechanistically shows how standard augmentation fails in late video layers and how their method stops this leakage. This diagnostic depth makes the substantial gains across both simulation benchmarks and physical robot experiments particularly credible and consequential.

Critical weaknesses
  • PICA-WAM trains by applying 2D image edits to recorded videos while forcing the robot's action representation to stay identical. While valid for lighting and blur, 2D edits on a single video cannot synthesize true 3D camera-angle changes, and large 2D image shifts under a fixed camera would teach the policy to ignore pixel-position cues needed to aim at moved objects; accordingly, Camera viewpoint remains PICA-WAM's lowest LIBERO-Plus score (64.7% vs. 78.4%) and the exact training edits deferred to Appendix A rather than specified in the main text.

  • PICA-WAM combines visual data augmentation with its self-alignment loss. However, an augmentation-only baseline is evaluated only on LIBERO (Section 4.1). On RoboTwin 2.0 (29.8% to 57.0%) and the real robot under colored lights (33.1% to 67.5%), PICA-WAM is compared only against unaugmented base models (LingBot-VA and OpenWAM-a). Without augmentation controls on those two benchmarks, we cannot tell how much of the headline real-robot and RoboTwin gains come from simple visual augmentation versus the alignment loss itself.

Questions for authors
  • How sensitive is training to the noise gap δ\delta? Did you run into any representation collapse in the projection head during early training before the teacher stabilized?

  • Since training needs passes through both the student and the teacher, what is the actual increase in training time and GPU memory compared to standard training and regular data augmentation?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for your thoughtful comments and questions.

W1. PICA-WAM trains by applying 2D image edits to recorded videos while forcing the robot's action representation to stay identical. While valid for lighting and blur, 2D edits on a single video cannot synthesize true 3D camera-angle changes, and large 2D image shifts under a fixed camera would teach the policy to ignore pixel-position cues needed to aim at moved objects; accordingly, Camera viewpoint remains PICA-WAM's lowest LIBERO-Plus score (64.7% vs. 78.4%) and the exact training edits deferred to Appendix A rather than specified in the main text.

We found that mild crop/rotation improves robustness to actual 3D camera changes, although it does not generate new 3D views. The results also show that the alignment does not make the policy ignore task-relevant object positions. We apply these edits only with a probability of 0.25 per camera clip: crop to 92–100% of the original side length, resize back to the input resolution, and rotate within ±8°.

To measure their contribution, we removed only crop/rotation while keeping all other augmentations. The four configurations below were evaluated on the same 10,030 LIBERO-Plus tasks.

Training geometryMethodCamera (%)Object Layout (%)
Crop/rotation, probability 0.25+Aug62.1078.30
Crop/rotation, probability 0.25PICA-WAM64.6782.56
None+Aug48.2280.39
NonePICA-WAM47.2279.67

Keeping crop/rotation increases Camera success by 13.88 pp for +Aug and 17.45 pp for PICA-WAM. With the full augmentation recipe, adding alignment further improves Camera success from 62.10% to 64.67%. These comparisons support using crop/rotation together with alignment to improve viewpoint robustness.

The Object Layout evaluation changes physical object positions, requiring the policy to adjust its actions accordingly. With the same full augmentation recipe, adding alignment increases success from 78.30% to 82.56%. These results indicate that alignment does not make the policy ignore task-relevant object positions.

The robot and objects undergo the same image transformation while their physical relationship stays unchanged, so the required action in world coordinates remains the same. We align only the nominal and transformed observations of the same state, while different states retain their own action targets in the action-prediction loss. Thus, alignment encourages consistency under visual perturbation, while action loss continues to supervise how the robot should respond to different object positions.

We have specified the training edits and their bounds in Section 4.1, reported the ablation in Section 4.4 and Appendix D, Table 13, and clarified the action coordinates and the distinction between image edits and 3D camera changes in Appendix D.

W2. PICA-WAM combines visual data augmentation with its self-alignment loss. However, an augmentation-only baseline is evaluated only on LIBERO (Section 4.1). On RoboTwin 2.0 (29.8% to 57.0%) and the real robot under colored lights (33.1% to 67.5%), PICA-WAM is compared only against unaugmented base models (LingBot-VA and OpenWAM-a). Without augmentation controls on those two benchmarks, we cannot tell how much of the headline real-robot and RoboTwin gains come from simple visual augmentation versus the alignment loss itself.

We agree that PICA-WAM should be compared against baselines trained with data augmentation on both benchmarks.

On the real robot, both OpenWAM-α and PICA-WAM are fine-tuned from the same pretrained OpenWAM-α checkpoint with the same 2,000-step schedule and the exact same training augmentation T. The two models differ only in whether the cross-appearance alignment loss is added.

On RoboTwin 2.0, we did not retrain a separate LingBot-VA (+Aug) baseline during the rebuttal due to limited compute and time. Instead, we ran a controlled +Aug versus PICA-WAM comparison on the Fast-WAM backbone under the same RoboTwin 2.0 protocol.

RoboTwin 2.0 (Fast-WAM)+Aug (%)PICA-WAM (%)Gain
Clean84.688.0+3.4 pp
Randomized29.639.6+10.0 pp
Avg.57.163.8+6.7 pp

Although this control uses Fast-WAM rather than LingBot-VA, together with the OpenWAM-α real-robot comparison and the LIBERO ablation in Section 4.4, it shows across three benchmarks that alignment provides clear gains beyond image augmentation.

We have updated Section 4.5, Table 6, and Appendix A to state explicitly that the OpenWAM-α real-robot baseline uses the same augmentation T, and updated Section 4.3, Table 2, and Appendix A to group RoboTwin 2.0 by embodied pretraining and include both Fast-WAM (+Aug) and PICA-WAM based on Fast-WAM.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Thank you for your thoughtful comments and questions.

Q1a. How sensitive is training to the noise gap δ?

We examined the noise gap δ on the same 10,030 LIBERO-Plus tasks with student and teacher layers fixed at 8 and 20 and alignment weight fixed at 0.05, where augmentation-only training reaches 81.09%.

Maximum sampled noise gapOverall success (%)Gain vs. +Aug (pp)
084.81+3.72
0.285.17+4.09
0.486.13+5.04

For the noise gap, both 0.2 and 0.4 yield substantial gains over augmentation-only training (+4.09 and +5.04 pp), showing that the benefit is not tied to our default value. Even with zero noise gap, cross-appearance alignment retains a +3.72 pp gain.

We have added this noise-gap analysis to Section 4.4, “Layer and hyperparameter sensitivity,” and Appendix C, Table 10.

Q1b. Did you run into any representation collapse in the projection head during early training before the teacher stabilized?

For early projection head analysis, we replayed the first 500 training steps of PICA-WAM on 8 A100 GPUs and probed 64 fixed states at steps 0, 1, 10, 50, 100, 250, and 500, using the same Gaussian noise for all states so that diversity cannot come from noise. We found no constant or low-rank collapse. The centered rank of the projected features stays at the maximum of 63 at every probe point and noise level, and their mean norm grows from 8.03 to 15.50 at the lowest noise level. The fixed-probe action loss falls from 0.92 to 0.13 at the lowest noise level and from 3.01 to 0.20 at the middle level. After removing the mean over states, the projected feature of each state becomes positively correlated with the teacher feature of the same state, with cosine rising from 0.003 to 0.205 at the lowest noise level, whereas the cosine between different states stays near zero at −0.003.

We have added this early-training analysis to Appendix G, Table 17.

Q2. Since training needs passes through both the student and the teacher, what is the actual increase in training time and GPU memory compared to standard training and regular data augmentation?

We measured all three training settings on the same allocation of 8 A100 GPUs with bf16, DeepSpeed ZeRO-1, and 16 samples per GPU. Timing covers the forward and backward passes, gradient clipping, the optimizer, the scheduler, and the EMA update.

MethodTime per step (s)Peak allocated memory (GiB)
Baseline2.5333.93
+Aug3.4034.09
PICA-WAM3.7039.80

Relative to +Aug, alignment adds 8.8% training time per step and 5.71 GiB of peak memory. The extra cost comes from one gradient-free teacher forward pass on the nominal view that stops at Layer 20, together with the EMA teacher weights and the activations of the nominal view. Relative to the Baseline, most of the additional time of PICA-WAM comes from constructing and encoding augmented inputs, which adds 34.5% on its own. The teacher and the projection head are used only during training, so inference cost is unchanged.

We have added this cost comparison to Appendix G, Table 16.

Thanks for the experiments and clarifications on the real-robot setup and the Fast-WAM RoboTwin 2.0 control. I have a follow-up question on the new tables:

In your W1 rebuttal table, PICA-WAM with crop/rotation gets 64.67% on Camera and 82.56% on Layout, matching PICA-WAM in Table 4 of the paper. However, +Aug with crop/rotation gets 62.10% on Camera in your rebuttal table, whereas +Aug in Table 4 of the paper got only 45.72% (close to your no-crop/rotation row at 48.22%). Was the +Aug baseline in Table 4 (and in Section 4.2's diagnostics) trained without crop/rotation while PICA-WAM included it, and if so, what are the full Table 4 results when all methods use the exact same augmentations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.