- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates visual representation selection and action generation in World Action Models (WAMs). The authors first theoretically analyze and empirically compare different pretrained visual encoders for action generation, finding that I-JEPA performs best among the evaluated encoders. Based on this finding, they propose LeWAM, which uses frozen I-JEPA features as visual inputs and future-state prediction targets, jointly learning action generation and future-feature prediction with a shared Transformer. They also introduce AdaFuse to fuse visual features from different encoder layers. In addition, the authors propose Demonstration-Guided DPO (DemoDPO), which constructs offline preference pairs based on the mean squared error (MSE) between generated actions and demonstrated actions to further refine the action policy. The evaluation covers 50 tasks on RoboTwin 2.0, three real-world robotic manipulation tasks, visual representation analysis, and inference efficiency. LeWAM achieves an average success rate of 92.28% on RoboTwin 2.0, while also demonstrating lower inference latency and GPU memory consumption.
Critical strengths
A major strength of this paper is its systematic investigation of visual representation selection, together with an efficient WAM architecture. The authors compare multiple frozen visual encoders under a consistent downstream action-generation setting and find that I-JEPA representations achieve strong manipulation performance. LeWAM further avoids relying on a large video diffusion backbone for feature extraction and uses future-feature prediction to provide additional training supervision. The evaluation includes 50 simulation tasks, real-world robotic manipulation, and inference efficiency measurements, providing evidence beyond a single offline metric.
Critical weaknesses
The independent contribution of the core architecture is insufficiently validated. A key design feature of LeWAM is the use of a shared Transformer to jointly learn action generation and future-feature prediction. However, the existing ablations do not directly compare the shared predictor with separate predictors under otherwise comparable settings. Consequently, the current experiments do not establish whether the shared architecture itself provides meaningful benefits beyond those of future-feature prediction and feature fusion.
The methodological novelty and validity of DemoDPO remain insufficiently established. The method largely follows existing DPO/Flow-DPO objectives, with its main additional design being the construction of preference pairs based on the MSE between generated actions and demonstrated actions. However, actions closer to demonstrations are not necessarily more likely to succeed at the task. The paper does not sufficiently establish when this distance-based preference signal reliably reflects action quality, which limits the current evidence for DemoDPO's independent contribution.
The evidence for performance superiority over existing methods is limited. LeWAM exceeds the strongest reported baseline by only 0.08 percentage points in average success rate, with a slightly higher score on clean scenes but a slightly lower score on randomized scenes. This small margin is not inherently problematic, but its significance remains unclear. Moreover, some baseline results are taken from prior papers, and the comparability of their training and evaluation protocols is not fully established. Further clarification of the evaluation settings and uncertainty in the reported results would help substantiate the performance claims.
Questions for authors
Core architecture: Could the authors compare the shared Transformer with separate predictors for action generation and future-feature prediction under the same visual encoder, prediction objectives, and training settings? Such a comparison would help determine whether the shared architecture provides independent benefits beyond those of the other components.
DemoDPO: Why do the authors consider the MSE between candidate actions and demonstrated actions a reliable basis for ranking action quality? When multiple distinct action trajectories can achieve the same goal, how might this preference construction affect the resulting policy? Do the current results support the applicability of this assumption to the evaluated tasks?
Benchmark comparisons: Given the small average success-rate difference between LeWAM and the strongest reported baseline, and the fact that some baseline results are taken from prior papers, could the authors clarify the comparability of the methods in terms of training data, evaluation protocols, and randomized scenarios? Are results from multiple independent runs or uncertainty estimates available to establish whether this difference is meaningful?
3 comments
Response to Reviewer LFAH (Part 2 / 3)
(2) How does distance-based preference construction affect the retained supervision?
Recall that DemoDPO applies a threshold when constructing preference pairs, which changes the distribution of retained training samples. A sample is labeled grasp or release if a gripper closing or opening motion starts or ends within its 32-step demonstrated chunk. Samples with no such event are labeled non-event.
| 0 | ||||
|---|---|---|---|---|
| Grasp | 22.48 | 24.73 | 35.80 (1.59×) | 42.41 |
| Release | 8.89 | 10.53 | 11.02 (1.24×) | 14.69 |
| Non-event | 68.64 | 64.73 | 53.17 (0.77×) | 42.90 |
| Success rate | 71.69 | 73.56 | 74.50 | 67.88 |
As shown in the table, filtering shifts the training distribution toward grasp and release events, giving these critical moments more preference supervision and improving task success.
However, an excessively large threshold reduces coverage of non-event actions, which may explain the decline in performance. Based on this analysis, we choose to emphasize critical gripper events while retaining coverage of non-event actions.
(3) How does this preference construction relate to multiple valid trajectories?
We agree that a single demonstration cannot represent every trajectory that may complete the same task. However, in the current setting, the training data covers diverse scene configurations but provides only one demonstrated mode per scene. Preferring candidates close to the demonstration for that scene keeps the policy within the demonstrated data distribution. This is particularly meaningful around contact events, where successful grasping and placement actions require precise spatial and temporal coordination and therefore admit fewer effective alternatives.
The current benchmark evaluates task completion rather than trajectory diversity. Thus, learning one successful mode is sufficient for the criterion used here. Moreover, evaluation uses held-out seeds and unseen scene configurations, and DemoDPO improves success on these scenes, showing that the policy generalizes the demonstrated behavior rather than simply replaying training trajectories.
For tasks where trajectory diversity itself is important, multiple demonstrations for the same state would provide a stronger basis for learning alternative valid modes. We will explore this direction in future work.
In summary, DemoDPO contributes a demonstration-guided preference construction strategy. While standard DPO can use all labeled pairwise comparisons within a candidate group for training, DemoDPO selects only the minimum- and maximum-MSE candidates and retains their pair only when the distance gap meets the threshold. This selective construction reshapes the training distribution toward critical grasp and release events, strengthening their preference supervision.
Our analyses have explained how this change in the training distribution improves task success, highlighting a contribution in preference construction beyond the underlying DPO objective.
Response to Reviewer LFAH (Part 3 / 3)
Q3 & W3: How comparable are the benchmark comparisons, and how should the performance difference be interpreted?
Thank you for raising this question. The compared methods use 2,500 clean and 25,000 randomized demonstrations across 50 RoboTwin 2.0 tasks for benchmark-specific training, as described in Fast-WAM, Motus, LingBot-VA, GigaWorld-Policy, and LaWAM. We evaluate 100 scenes per task per setting under the official clean and randomized configurations. The latter varies background textures, table clutter, table height, and lighting.
We use Fast-WAM as our baseline because it does not use embodied pretraining, avoiding the confounding effects of different and potentially unreleased pretraining datasets while reducing training cost. We match its training data, evaluation protocol, and evaluation seeds. To contextualize comparisons with other models, the table below summarizes their differences in embodied pretraining, task conditioning, and visual input processing alongside their performance.
In the table, resolutions are given as before concatenation. Embodied PT. indicates pretraining on additional robot data. Avg. SR averages the clean and randomized success rates.
| Method | Embodied PT. | Task conditioning | Head | Left | Right | Multi-view concatenation | Avg. SR |
|---|---|---|---|---|---|---|---|
| ✓ | Language → PaliGemma | Per-view encoding → token concatenation | 79.75% | ||||
| GigaWorld-Policy | ✓ | Language → UMT5-XXL | T-shaped image → VAE | 85.70% | |||
| Motus | ✓ | Language → UMT5-XXL + Qwen3-VL | T-shaped image → pad to → encoders | 87.84% | |||
| LaWAM | ✓ | Language → Qwen3-VL | Per-view encoding → token concatenation | 91.22% | |||
| LingBot-VA | ✓ | Language → UMT5-XXL | Per-view VAE → T-shaped latent concatenation | 92.20% | |||
| Fast-WAM | ✗ | Language → UMT5-XXL | T-shaped image → VAE | 91.83% | |||
| LeWAM | ✗ | Task ID → embedding + AdaLN | T-shaped image → resize to → I-JEPA | 92.28% |
T-shaped concatenation arranges the full-width head view above the two wrist views placed side by side. LingBot-VA reverses this vertical order, placing the wrist latents above the head latent.
As shown above, LeWAM achieves competitive performance with LingBot-VA using lower-resolution visual inputs, without additional embodied pretraining or language-derived semantic information.
For a fairer comparison with language-conditioned baselines, we additionally evaluate LeWAM (Text), which incorporates task instructions encoded by UMT5-XXL, reducing the conditioning mismatch with LingBot-VA. We project the text embeddings to the predictor dimension and prepend them to the visual and action tokens, reusing the existing self-attention layers to limit the increase in trainable parameters. Total parameters, including frozen visual and text encoders, are 1.0B for LeWAM and 6.7B for LeWAM (Text).
To quantify scene-sampling uncertainty, we use a stratified bootstrap, adapting the task-stratified approach used in prior robotic manipulation work [1]. Each task–setting combination contains 100 distinct, independently sampled scenes, with one rollout per scene, allowing us to resample outcomes within each combination and recompute aggregate success rates while keeping the checkpoint and task set fixed. Bootstrap standard deviations provide standard errors (SE), and the 2.5th and 97.5th percentiles give 95% confidence intervals (CI).
| Method | Setting | Success rate (%) | SE (pp) | 95% CI (%) |
|---|---|---|---|---|
| LeWAM | Clean | 93.14 | 0.34 | [92.45, 93.82] |
| LeWAM | Rand | 91.42 | 0.37 | [90.67, 92.16] |
| LeWAM | Avg. | 92.28 | 0.25 | [91.78, 92.77] |
| LeWAM (Text) | Clean | 93.68 | 0.33 | [93.01, 94.35] |
| LeWAM (Text) | Rand | 92.96 | 0.36 | [92.25, 93.67] |
| LeWAM (Text) | Avg. | 93.32 | 0.24 | [92.83, 93.81] |
LeWAM (Text) achieves 93.32% overall success, a 1.12-percentage-point improvement over LingBot-VA's reported 92.20%. Its entire 95% CI of [92.83%, 93.81%] lies above that reported score, providing stronger evidence for the performance advantage under language conditioning.
[1] Seo, Younggyo, et al. "Multi-view masked world models for visual robotic manipulation." International Conference on Machine Learning (ICML). PMLR, 2023.
Response to Reviewer LFAH (Part 1 / 3)
We deeply appreciate your constructive review and recognition of our visual-encoder comparison, efficient WAM architecture, and simulation and real-world evaluation. We address your concerns through additional experiments and analyses below.
Q1 & W1: Does the shared Transformer provide benefits beyond those of the other components, compared with separate predictors for action generation and future-feature prediction?
To address whether a shared Transformer offers benefits over separate predictors, we design LeWAM-MoT, following the two-expert architecture of π0.5. Specifically, its action-generation and future-feature-prediction branches use independent parameters and interact through joint self-attention under the same attention mask as LeWAM.
We keep the frozen visual encoder, training data, input sequence, prediction targets, loss weights, optimization schedule, and evaluation protocol identical to LeWAM. We choose the MoT branch widths to approximately match LeWAM’s trainable parameter budget while retaining 12 layers per branch, a head dimension of 64, and power-of-two hidden and FFN widths (1024 and 2048).
As shown in the table, with comparable trainable parameter budgets, LeWAM outperforms LeWAM-MoT in both clean and randomized settings before DemoDPO post-training. This supports a shared Transformer for jointly learning action generation and future-feature prediction.
Q2 & W2: Why is demonstration MSE a reliable basis for ranking action quality, and how does it affect tasks with multiple valid trajectories?
We design DemoDPO’s preference construction as a local greedy strategy based on demonstration MSE, which works well in our experiments. We agree that being closer to the demonstration does not by itself make an action more likely to succeed, and DemoDPO does not depend on such a graded relationship. It compares only the two ends of each candidate group. When the closest and the farthest candidates are separated by a large MSE gap, we treat the closest one as preferred, and we make no judgment about candidates in between or about pairs with a small gap. Our original submission did not analyze this strategy in detail, and we thank you for highlighting this gap. Below, we address your concern through a closed-loop evaluation and a discussion of tasks with multiple valid trajectories.
(1) Closed-loop Evaluation
We conduct a closed-loop evaluation to assess whether demonstration MSE provides a useful criterion for candidate selection during task execution. In this evaluation, all paired rollouts start from the same initial simulator states. At each replanning step, the RoboTwin expert generates a successful continuation from the current state, and its next action chunk defines the local reference. The policy samples G candidate action chunks and selects the candidate with the smallest, median, or largest distance to that reference. The median rule selects the candidate ranked ⌊G/2⌋-th in ascending order of distance. When G=1, all three rules select the sole candidate. After the selected chunk is executed, the expert replans from the resulting observation. We evaluate G∈{1,4,8}, with G=4 matching the DemoDPO setting. Because repeated expert replanning makes these rollouts time-consuming, we evaluate the eight tasks from the DemoDPO ablation on the first 10% of the original validation cases for each task, yielding 20 scenes per task.
The resulting task success rates are as follows:
As shown in the table, the smallest-MSE rule consistently outperforms the largest-MSE rule, while median selection remains close to the single-candidate baseline. These results support demonstration MSE as a local greedy criterion for candidate selection on the evaluated tasks.
Moreover, the gain from increasing G from 4 to 8 remains limited, consistent with the DemoDPO ablation in Figure 5(c). We therefore choose G=4 to balance performance and candidate-sampling cost.