- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the selection, integration, and composition of visual representations in VAE-based world action models for robotic manipulation. Under a shared backbone and training protocol, the authors compare 13 pretrained visual encoders, five integration mechanisms, and multiple feature combinations. The results suggest that general-purpose representations perform better than specialized representations under different types of perturbations, channel-wise joint prediction achieves the highest average success rate, and the benefit of additional features depends on the existing combination. Based on these findings, the authors construct R2-WAM by combining DINOv3, UniTok, and V-JEPA 2.1 through channel-wise concatenation. Experimental results show that R2-WAM improves performance over VAE-only baselines across multiple WAM formulations, simulation benchmarks, and real-robot tasks.
Critical strengths
1.The paper compares 13 visual encoders, five integration mechanisms, and multiple feature combinations using a common WAM backbone and shared training protocol. These comparisons provide useful guidance for selecting, integrating, and combining visual features in WAMs.
2.R2-WAM improves upon the VAE-only baselines across multiple WAM formulations, simulation benchmarks, and real-robot tasks. It increases the success rate from 78.8% to 84.7% on RoboCasa and from 90.6% to 92.8% on RoboTwin 2.0.
Critical weaknesses
1.The paper presents an effective configuration of existing visual representations and provides prediction ablations and sensitivity analysis. However, these analyses offer limited explanations for why different representations and integration mechanisms lead to performance differences. R²-WAM mainly combines existing encoders through channel-wise concatenation, while concurrent work such as ReWAM([2609.38163] Rethinking Representations for World-Action Modeling) adapts frozen DINO features to WAMs through a trainable temporal bottleneck supervised by the action prediction loss. R2-WAM’s contribution therefore lies mainly in empirical guidance for selecting and combining existing components, with limited method novelty.
2.On LIBERO-Plus, adding DINOv3 alone accounts for most of the improvement over VAE-only, increasing success from 74.7% to 84.0%, while the full R2-WAM reaches 87.3%. However, the RoboCasa and RoboTwin 2.0 experiments (Table 3) and the four real-robot tasks (Table 2(b)) do not include a VAE + DINOv3 baseline. This comparison is needed to determine whether UniTok and V-JEPA 2.1 provide additional benefits in these settings.
3.The robustness findings are mainly based on LIBERO-Plus. The real-robot experiments evaluate task performance but do not report controlled tests under changes in viewpoint, background, or lighting. Such tests would help assess whether the robustness findings also hold on physical robots.
Questions for authors
Please refer to the weakness
4 comments
Response to Reviewer Ns4d: Single-source and multi-source comparisons
W2. Single-source and multi-source comparisons
We thank the reviewer for suggesting this comparison. The additional comparisons distinguish the substantial benefit of adding DINOv3 from the further gains provided by multi-source composition. Across different evaluation settings, especially those testing generalization, the additional representations provide benefits beyond the DINOv3-only configuration.
We compare VAE-only, VAE + DINOv3, and the full R²-WAM on LIBERO-Plus, RoboCasa, RoboTwin 2.0, and the four real-robot tasks. For the supplementary RoboTwin comparison, we adopt a clean-to-random (C2R) protocol, training all three models exclusively on clean demonstrations and evaluating them in both clean and randomized environments. The full-data protocol reported in Table 5 of the revised manuscript includes randomized demonstrations during training, whereas C2R reserves these conditions for evaluation and thus more directly examines generalization beyond the clean training distribution. Full-dataset training also takes approximately one week on 32 PPU accelerators in our setup, further motivating the C2R protocol for this supplementary comparison.
Table R3a. Comparison of VAE-only, VAE + DINOv3, and the full R²-WAM across simulation benchmarks and real-robot tasks (%).
| Benchmark / test condition | VAE-only | VAE + DINOv3 | Full R²-WAM |
|---|---|---|---|
| LIBERO-Plus | 74.7 | 84.0 | 87.3 |
| RoboCasa GR1 Tabletop | 78.8 | 83.0 | 84.7 |
| RoboTwin C2R — Clean | 82.38 | 85.64 | 85.32 |
| RoboTwin C2R — Randomized | 4.12 | 10.50 | 12.16 |
| Galbot G1 — Air-fryer food preparation | 50.0 | 65.0 | 60.0 |
| Galbot G1 — Instructed pick-and-place | 70.0 | 70.0 | 80.0 |
| Galbot G1 — Sweeping, debris collection rate | 48.3 | 52.5 | 66.7 |
| Galbot G1 — Cup pushing | 65.0 | 75.0 | 75.0 |
Benchmark evaluation protocols and real-robot evaluation criteria are detailed in Appendix C.2 and Appendix C.3, respectively.
The results confirm that DINOv3 provides a substantial part of the improvement. Adding the two further representations increases performance by 3.3 percentage points on LIBERO-Plus and 1.7 points on RoboCasa. On RoboTwin, the full model improves randomized-environment performance from 10.50% to 12.16%.
The full model’s real-robot gains are consistent with the added representations’ pretraining objectives. UniTok’s image–text alignment provides language-associated visual semantics relevant to instructed pick-and-place, where success increases from 70.0% to 80.0%. V-JEPA 2.1’s predictive video pretraining captures temporal relationships relevant to tool–debris interactions in the long-horizon Sweep Rubbish task, where collection rate increases from 52.5% to 66.7%.
Response to Reviewer Ns4d: Real-robot robustness under visual changes
W3. Real-robot robustness under visual changes
We thank the reviewer for suggesting these experiments. We have added controlled physical-robot evaluations under background, viewpoint, and lighting changes. The results extend the robustness evidence beyond LIBERO-Plus and show that representation augmentation improves performance under the tested visual changes, with the full R²-WAM achieving the strongest overall performance. The experimental setup and results are provided in Appendix C.3 and Table R8, with the visual conditions illustrated in Figure 7.
We evaluate VAE-only, VAE + DINOv3, and the full R²-WAM on all four real-robot tasks. For background variation, we replace the white tabletop appearance with a wood-textured surface. For viewpoint variation, we rotate the robot’s head 10° to the right while keeping its pitch fixed. For lighting variation, we physically change the workspace lighting. Each model is evaluated with 20 trials per task and reported condition; the lighting evaluation comprises 10 trials under low light and 10 under bright light. Sweeping retains the debris collection-rate metric, while the other three tasks use task success.
Table R8. Real-robot generalization across four tasks.
| Task | Condition | VAE-only | VAE + DINOv3 | Full R²-WAM |
|---|---|---|---|---|
| Air-fryer food preparation | Reference | 50.0 (10/20) | 65.0 (13/20) | 60.0 (12/20) |
| Air-fryer food preparation | Background | 30.0 (6/20) | 65.0 (13/20) | 65.0 (13/20) |
| Air-fryer food preparation | Viewpoint | 10.0 (2/20) | 25.0 (5/20) | 30.0 (6/20) |
| Air-fryer food preparation | Lighting | 50.0 (10/20) | 60.0 (12/20) | 65.0 (13/20) |
| Instructed pick-and-place | Reference | 70.0 (14/20) | 70.0 (14/20) | 80.0 (16/20) |
| Instructed pick-and-place | Background | 55.0 (11/20) | 70.0 (14/20) | 75.0 (15/20) |
| Instructed pick-and-place | Viewpoint | 35.0 (7/20) | 45.0 (9/20) | 50.0 (10/20) |
| Instructed pick-and-place | Lighting | 60.0 (12/20) | 70.0 (14/20) | 75.0 (15/20) |
| Tool sweeping | Reference | 48.3 (58/120) | 52.5 (63/120) | 66.7 (80/120) |
| Tool sweeping | Background | 32.5 (39/120) | 48.3 (58/120) | 61.7 (74/120) |
| Tool sweeping | Viewpoint | 12.5 (15/120) | 30.0 (36/120) | 40.0 (48/120) |
| Tool sweeping | Lighting | 45.8 (55/120) | 54.2 (65/120) | 65.0 (78/120) |
| Cup pushing | Reference | 65.0 (13/20) | 75.0 (15/20) | 75.0 (15/20) |
| Cup pushing | Background | 40.0 (8/20) | 60.0 (12/20) | 65.0 (13/20) |
| Cup pushing | Viewpoint | 10.0 (2/20) | 35.0 (7/20) | 40.0 (8/20) |
| Cup pushing | Lighting | 60.0 (12/20) | 75.0 (15/20) | 80.0 (16/20) |
Entries are percentages, with successful trials/total trials in parentheses for air-fryer preparation, instructed pick-and-place, and cup pushing. Sweeping reports mean debris collection rate, with collected items/total items in parentheses (20 trials, six items per trial). Reference denotes the original evaluation condition. Best scores within each row are bold.
The results in Table R8 are consistent with our findings on LIBERO-Plus. All three models generalize well to the tested lighting changes, maintaining performance close to their respective reference conditions. Under background changes, VAE-only exhibits a pronounced performance drop, whereas VAE + DINOv3 and the full R²-WAM largely preserve their reference performance. Viewpoint changes pose the greatest challenge and reduce performance for all three models. Nevertheless, both augmented models retain clear advantages over VAE-only, and the full R²-WAM achieves the best performance on all four tasks under this condition. These physical-robot results support the benefits of representation augmentation and the additional gains from multi-source composition.
Response Letter
Dear Authors,
Thanks for your efforts to address my concerns.
However, considering some comments provided by other reviewers, I decided to maintain my rating.
Response to Reviewer Ns4d: Contribution and explanatory depth
W1. Contribution and explanatory depth
We thank the reviewer for recognizing the value of our systematic comparisons and the improvements of R²-WAM across WAM formulations, simulation benchmarks, and real-robot tasks. Our paper is a systematic study of visual representation design for WAMs. The central contribution is to establish an empirical understanding of how visual representation choices, integration mechanisms, and feature combinations affect WAM performance, and to translate this understanding into actionable design principles.
Comparing 13 visual encoders, five integration mechanisms, and multiple feature combinations within a shared WAM framework allows us to study relationships that individual model proposals do not resolve. R²-WAM translates this understanding into a practical configuration, whose evaluation across WAM formulations and environments demonstrates the usefulness of the resulting design principles (Section 4.5).
To address the concern about explanatory depth, we have strengthened the analyses of both integration mechanisms and feature composition. These additions build on the prediction ablations, feature visualizations, and action-sensitivity analysis in Section 4.3 and Figure 3.
For integration mechanisms, Table 2 in Section 4.3 retains each mechanism’s future-feature supervision while restricting the action stream’s access to future visual information. Under these restrictions, the success-rate range across the three mechanisms narrows from 76.1–84.0% to 75.8–78.5%. Channel-wise concatenation shows the largest decrease, from 84.0% to 78.5%, compared with changes of 0.3 and 1.5 percentage points for token-wise concatenation and feature expert. These results suggest that the information pathways enabled by the integration mechanisms play a more important role than auxiliary feature-prediction supervision alone in explaining their performance differences.
For feature composition, Section 4.4 and Figure 4 examine how an additional representation’s benefit depends on the features already present. We further introduce controls for auxiliary-loss coefficients and channel count in Table 3, with the objectives and feature construction detailed in Appendix E.4. Matching the sum of auxiliary-loss coefficients preserves multi-source advantages of 0.81, 2.83, and 2.30 percentage points for two, three, and four sources. Matching the auxiliary width at 16 channels also preserves a 1.97-point advantage for DINOv3 + UniTok over DINOv3 alone. Together, these controls and the source-composition results support the interpretation that the gains primarily arise from the additional task-relevant information contributed by complementary representations.
ReWAM’s representation-adaptation approach and our systematic investigation of representation selection, integration, and composition are complementary. We recognize ReWAM as a valuable concurrent contribution to world-action modeling and will add its citation and discuss its relationship to our study in the revised manuscript.