acceptodds
Withdrawn from ICLR 2027

What Visual Representations Do World Action Models Need?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. Recent studies show that external visual representations improve manipulation performance and robustness. However, these methods select representations based on presumed capability gaps and use different integration mechanisms, making it difficult to disentangle the effects of representation choice and integration. To better understand how external representations strengthen WAMs, we conduct a systematic study through controlled comparisons of multiple visual representations, five integration mechanisms, and feature combinations. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert's effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R²-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action models, visual representation learning, robot manipulation, systematic study

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ns4dOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the selection, integration, and composition of visual representations in VAE-based world action models for robotic manipulation. Under a shared backbone and training protocol, the authors compare 13 pretrained visual encoders, five integration mechanisms, and multiple feature combinations. The results suggest that general-purpose representations perform better than specialized representations under different types of perturbations, channel-wise joint prediction achieves the highest average success rate, and the benefit of additional features depends on the existing combination. Based on these findings, the authors construct R2-WAM by combining DINOv3, UniTok, and V-JEPA 2.1 through channel-wise concatenation. Experimental results show that R2-WAM improves performance over VAE-only baselines across multiple WAM formulations, simulation benchmarks, and real-robot tasks.

Critical strengths

1.The paper compares 13 visual encoders, five integration mechanisms, and multiple feature combinations using a common WAM backbone and shared training protocol. These comparisons provide useful guidance for selecting, integrating, and combining visual features in WAMs.

2.R2-WAM improves upon the VAE-only baselines across multiple WAM formulations, simulation benchmarks, and real-robot tasks. It increases the success rate from 78.8% to 84.7% on RoboCasa and from 90.6% to 92.8% on RoboTwin 2.0.

Critical weaknesses

1.The paper presents an effective configuration of existing visual representations and provides prediction ablations and sensitivity analysis. However, these analyses offer limited explanations for why different representations and integration mechanisms lead to performance differences. R²-WAM mainly combines existing encoders through channel-wise concatenation, while concurrent work such as ReWAM([2609.38163] Rethinking Representations for World-Action Modeling) adapts frozen DINO features to WAMs through a trainable temporal bottleneck supervised by the action prediction loss. R2-WAM’s contribution therefore lies mainly in empirical guidance for selecting and combining existing components, with limited method novelty.

2.On LIBERO-Plus, adding DINOv3 alone accounts for most of the improvement over VAE-only, increasing success from 74.7% to 84.0%, while the full R2-WAM reaches 87.3%. However, the RoboCasa and RoboTwin 2.0 experiments (Table 3) and the four real-robot tasks (Table 2(b)) do not include a VAE + DINOv3 baseline. This comparison is needed to determine whether UniTok and V-JEPA 2.1 provide additional benefits in these settings.

3.The robustness findings are mainly based on LIBERO-Plus. The real-robot experiments evaluate task performance but do not report controlled tests under changes in viewpoint, background, or lighting. Such tests would help assess whether the robustness findings also hold on physical robots.

Questions for authors

Please refer to the weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ns4d: Contribution and explanatory depth

W1. Contribution and explanatory depth

We thank the reviewer for recognizing the value of our systematic comparisons and the improvements of R²-WAM across WAM formulations, simulation benchmarks, and real-robot tasks. Our paper is a systematic study of visual representation design for WAMs. The central contribution is to establish an empirical understanding of how visual representation choices, integration mechanisms, and feature combinations affect WAM performance, and to translate this understanding into actionable design principles.

Comparing 13 visual encoders, five integration mechanisms, and multiple feature combinations within a shared WAM framework allows us to study relationships that individual model proposals do not resolve. R²-WAM translates this understanding into a practical configuration, whose evaluation across WAM formulations and environments demonstrates the usefulness of the resulting design principles (Section 4.5).

To address the concern about explanatory depth, we have strengthened the analyses of both integration mechanisms and feature composition. These additions build on the prediction ablations, feature visualizations, and action-sensitivity analysis in Section 4.3 and Figure 3.

For integration mechanisms, Table 2 in Section 4.3 retains each mechanism’s future-feature supervision while restricting the action stream’s access to future visual information. Under these restrictions, the success-rate range across the three mechanisms narrows from 76.1–84.0% to 75.8–78.5%. Channel-wise concatenation shows the largest decrease, from 84.0% to 78.5%, compared with changes of 0.3 and 1.5 percentage points for token-wise concatenation and feature expert. These results suggest that the information pathways enabled by the integration mechanisms play a more important role than auxiliary feature-prediction supervision alone in explaining their performance differences.

For feature composition, Section 4.4 and Figure 4 examine how an additional representation’s benefit depends on the features already present. We further introduce controls for auxiliary-loss coefficients and channel count in Table 3, with the objectives and feature construction detailed in Appendix E.4. Matching the sum of auxiliary-loss coefficients preserves multi-source advantages of 0.81, 2.83, and 2.30 percentage points for two, three, and four sources. Matching the auxiliary width at 16 channels also preserves a 1.97-point advantage for DINOv3 + UniTok over DINOv3 alone. Together, these controls and the source-composition results support the interpretation that the gains primarily arise from the additional task-relevant information contributed by complementary representations.

ReWAM’s representation-adaptation approach and our systematic investigation of representation selection, integration, and composition are complementary. We recognize ReWAM as a valuable concurrent contribution to world-action modeling and will add its citation and discuss its relationship to our study in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Ns4d: Single-source and multi-source comparisons

W2. Single-source and multi-source comparisons

We thank the reviewer for suggesting this comparison. The additional comparisons distinguish the substantial benefit of adding DINOv3 from the further gains provided by multi-source composition. Across different evaluation settings, especially those testing generalization, the additional representations provide benefits beyond the DINOv3-only configuration.

We compare VAE-only, VAE + DINOv3, and the full R²-WAM on LIBERO-Plus, RoboCasa, RoboTwin 2.0, and the four real-robot tasks. For the supplementary RoboTwin comparison, we adopt a clean-to-random (C2R) protocol, training all three models exclusively on clean demonstrations and evaluating them in both clean and randomized environments. The full-data protocol reported in Table 5 of the revised manuscript includes randomized demonstrations during training, whereas C2R reserves these conditions for evaluation and thus more directly examines generalization beyond the clean training distribution. Full-dataset training also takes approximately one week on 32 PPU accelerators in our setup, further motivating the C2R protocol for this supplementary comparison.

Table R3a. Comparison of VAE-only, VAE + DINOv3, and the full R²-WAM across simulation benchmarks and real-robot tasks (%).

Benchmark / test conditionVAE-onlyVAE + DINOv3Full R²-WAM
LIBERO-Plus74.784.087.3
RoboCasa GR1 Tabletop78.883.084.7
RoboTwin C2R — Clean82.3885.6485.32
RoboTwin C2R — Randomized4.1210.5012.16
Galbot G1 — Air-fryer food preparation50.065.060.0
Galbot G1 — Instructed pick-and-place70.070.080.0
Galbot G1 — Sweeping, debris collection rate48.352.566.7
Galbot G1 — Cup pushing65.075.075.0

Benchmark evaluation protocols and real-robot evaluation criteria are detailed in Appendix C.2 and Appendix C.3, respectively.

The results confirm that DINOv3 provides a substantial part of the improvement. Adding the two further representations increases performance by 3.3 percentage points on LIBERO-Plus and 1.7 points on RoboCasa. On RoboTwin, the full model improves randomized-environment performance from 10.50% to 12.16%.

The full model’s real-robot gains are consistent with the added representations’ pretraining objectives. UniTok’s image–text alignment provides language-associated visual semantics relevant to instructed pick-and-place, where success increases from 70.0% to 80.0%. V-JEPA 2.1’s predictive video pretraining captures temporal relationships relevant to tool–debris interactions in the long-horizon Sweep Rubbish task, where collection rate increases from 52.5% to 66.7%.

AuthorsOpenReview ↗

Response to Reviewer Ns4d: Real-robot robustness under visual changes

W3. Real-robot robustness under visual changes

We thank the reviewer for suggesting these experiments. We have added controlled physical-robot evaluations under background, viewpoint, and lighting changes. The results extend the robustness evidence beyond LIBERO-Plus and show that representation augmentation improves performance under the tested visual changes, with the full R²-WAM achieving the strongest overall performance. The experimental setup and results are provided in Appendix C.3 and Table R8, with the visual conditions illustrated in Figure 7.

We evaluate VAE-only, VAE + DINOv3, and the full R²-WAM on all four real-robot tasks. For background variation, we replace the white tabletop appearance with a wood-textured surface. For viewpoint variation, we rotate the robot’s head 10° to the right while keeping its pitch fixed. For lighting variation, we physically change the workspace lighting. Each model is evaluated with 20 trials per task and reported condition; the lighting evaluation comprises 10 trials under low light and 10 under bright light. Sweeping retains the debris collection-rate metric, while the other three tasks use task success.

Table R8. Real-robot generalization across four tasks.

TaskConditionVAE-onlyVAE + DINOv3Full R²-WAM
Air-fryer food preparationReference50.0 (10/20)65.0 (13/20)60.0 (12/20)
Air-fryer food preparationBackground30.0 (6/20)65.0 (13/20)65.0 (13/20)
Air-fryer food preparationViewpoint10.0 (2/20)25.0 (5/20)30.0 (6/20)
Air-fryer food preparationLighting50.0 (10/20)60.0 (12/20)65.0 (13/20)
Instructed pick-and-placeReference70.0 (14/20)70.0 (14/20)80.0 (16/20)
Instructed pick-and-placeBackground55.0 (11/20)70.0 (14/20)75.0 (15/20)
Instructed pick-and-placeViewpoint35.0 (7/20)45.0 (9/20)50.0 (10/20)
Instructed pick-and-placeLighting60.0 (12/20)70.0 (14/20)75.0 (15/20)
Tool sweepingReference48.3 (58/120)52.5 (63/120)66.7 (80/120)
Tool sweepingBackground32.5 (39/120)48.3 (58/120)61.7 (74/120)
Tool sweepingViewpoint12.5 (15/120)30.0 (36/120)40.0 (48/120)
Tool sweepingLighting45.8 (55/120)54.2 (65/120)65.0 (78/120)
Cup pushingReference65.0 (13/20)75.0 (15/20)75.0 (15/20)
Cup pushingBackground40.0 (8/20)60.0 (12/20)65.0 (13/20)
Cup pushingViewpoint10.0 (2/20)35.0 (7/20)40.0 (8/20)
Cup pushingLighting60.0 (12/20)75.0 (15/20)80.0 (16/20)

Entries are percentages, with successful trials/total trials in parentheses for air-fryer preparation, instructed pick-and-place, and cup pushing. Sweeping reports mean debris collection rate, with collected items/total items in parentheses (20 trials, six items per trial). Reference denotes the original evaluation condition. Best scores within each row are bold.

The results in Table R8 are consistent with our findings on LIBERO-Plus. All three models generalize well to the tested lighting changes, maintaining performance close to their respective reference conditions. Under background changes, VAE-only exhibits a pronounced performance drop, whereas VAE + DINOv3 and the full R²-WAM largely preserve their reference performance. Viewpoint changes pose the greatest challenge and reduce performance for all three models. Nevertheless, both augmented models retain clear advantages over VAE-only, and the full R²-WAM achieves the best performance on all four tasks under this condition. These physical-robot results support the benefits of representation augmentation and the additional gains from multi-source composition.

Reviewer_Ns4dOpenReview ↗

Response Letter

Dear Authors,

Thanks for your efforts to address my concerns.

However, considering some comments provided by other reviewers, I decided to maintain my rating.

Reviewer_fuTJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to select, integrate, and combine external visual representations for video-generation-based World-Action Models. Under a shared WAM backbone and training protocol, the authors systematically compare frozen visual encoders, feature integration mechanisms, and multi-source feature combinations starting from DINOv3. The five integration mechanisms include REPA, gated cross-attention, channel concatenation, token concatenation, and a separate feature expert.

Critical strengths
  1. Representation selection and integration mechanisms often vary simultaneously across existing methods, making it difficult to isolate the sources of performance improvements. By evaluating 13 encoders and five integration mechanisms under a shared backbone, dataset, base observation input, and training configuration, this work provides reliable evidence for practical design choices.

  2. The paper provides meaningful ablation studies on future-feature supervision and multi-source feature combinations, helping identify where the improvements arise and how different representations contribute to performance.

  3. The authors distinguish evaluation seeds from training seeds, acknowledge that local gradient measurements do not necessarily imply causal contributions, and clarify that feature combinations are explored only along a selected path. They also recognize that conditional performance gains alone cannot distinguish feature redundancy from optimization interference.

Critical weaknesses
  1. The conclusion that general-purpose representations outperform specialized representations requires further qualification. The evaluated encoders differ in their pretraining data, model capacity, feature dimensionality, and potentially optimal feature extraction layers. These differences introduce confounding factors that make it difficult to attribute the observed performance differences solely to representation generality.

  2. The comparison of integration mechanisms involves multiple coupled design differences. The five mechanisms differ not only in where or how features are integrated, but also in their information pathways, supervision signals, and computational structures. Consequently, performance differences cannot be attributed solely to the feature integration strategy itself.

Questions for authors
  1. How much do the five integration mechanisms differ in terms of additional trainable parameters, computational overhead, and auxiliary loss configurations? Could these differences partially explain the observed performance gaps?

  2. How does performance change when the future-feature training objective is retained, but the action stream's access to future features is restricted? Such an experiment could help clarify whether the observed gains primarily arise from the additional representation-learning supervision or from the information pathways enabled by the integration mechanism.

3 comments
AuthorsOpenReview ↗

Response to Reviewer fuTJ: Encoder selection and comparison scope

Thank you for your careful reading and for suggesting a way to distinguish the supervision provided by future features from the action stream’s access to them. We ran the proposed attention-mask experiment and measured the training and inference costs of all five integration mechanisms.

Response to Reviewer fuTJ — W1

We thank the reviewer for pointing out the need to clarify the scope of this conclusion. One of the central goals of our study is to provide practical guidance on selecting pretrained visual encoders—developed using different datasets and learning objectives—to augment WAMs. Under a common downstream protocol, our results support strong general-purpose representations as the default starting point for this selection.

In Table 1, general-purpose representations achieve a higher average LIBERO-Plus success rate than specialized representations (82.2% versus 80.0%), and a general-purpose representation achieves the highest observed mean success in each of the seven perturbation categories.

With the aim of making the comparison across visual representations as fair and consistent as possible, we prioritized encoder variants with comparable parameter counts and used a common protocol for feature extraction and dimensionality reduction. We also discuss pretraining data as an important source of the encoders’ learned capabilities.

First, encoder selection takes model capacity and public checkpoint availability into account. We primarily use Large variants, with most feature-producing modules containing approximately 303–446M parameters. HRP and DynaFLIP use Base variants because their official releases provide the corresponding pretrained Base checkpoints without a pretrained Large counterpart. VGGT-Ω is similarly constrained by its public checkpoints: the released variants use a 1B-scale architecture, with differences in resolution and training rather than a comparable Large alternative.

Table R5. Encoder configurations used in the RQ1 comparison.

EncoderVariant usedParameters (M)Feature extractionNative dimension
DINOv3ViT-L/16303.13Normalized patch tokens after layer 241,024
LingBot-VisionViT-L/16303.23Normalized patch tokens after layer 241,024
VC-1ViT-L/16303.30Normalized patch tokens after layer 241,024
SigLIP 2ViT-L/16-256303.36Final visual hidden states after layer 241,024
VideoMAE V2InternVideo-L, UCF101-EDL (mae-l)303.86Layer-24 tokens with token-wise normalization, before pooling1,024
V-JEPA 2.1ViT-L/16, distilled-384304.68Normalized spatiotemporal tokens after layer 241,024
UniTokLarge / ViTamin-Large343.46Final encoder features followed by quantization and post-quantization projection1,024
C-RADIOv4SO400M431.24Dense spatial tokens after layer 271,152
SAM 3ViT/14, 32 layers446.24Final visual-backbone hidden states after layer 321,024
HRPHRP-IN, ViT-B/1685.80Normalized patch tokens after layer 12768
DynaFLIPBase, DINOv2 ViT-B/14 backbone86.58Normalized visual patch tokens after layer 12768
VGGT-ΩVGGT-Omega-1B-512908.14Patch tokens from the final aggregation stage2,048
Qwen3.5Qwen3.5-4B4,539.27Final VLM hidden states at visual-token positions after layer 322,560

Second, feature extraction and dimensionality reduction follow a consistent protocol. We extract token features from the final output of each encoder, with the precise readouts listed above. We agree that the optimal extraction layer may differ across encoders; using the final output establishes a consistent comparison rule, without assuming that this readout is individually optimal for every model. For the channel-wise comparison in Table 1, every source is reduced to eight channels using PCA before integration into the WAM. Thus, differences in native feature dimensionality do not translate into different numbers of auxiliary feature channels.

Third, pretraining data and objectives are important sources of representational capability. Data coverage, supervisory signals, and learning objectives jointly shape what an encoder captures and how its features transfer. We retain each encoder’s original pretrained weights and compare the resulting features using the same downstream WAM backbone, training data, and evaluation protocol. The pretraining datasets and budgets themselves are not matched. Providing practical guidance on selecting effective visual representations for WAMs from these differently pretrained encoders is a central goal and contribution of our study.

AuthorsOpenReview ↗

Response to Reviewer fuTJ: Integration costs and auxiliary losses

Q1. How do the five mechanisms differ in parameter count, computational cost, and auxiliary loss, and could these differences explain the performance gaps?

For each of the five mechanisms, Table R6 lists the modules added or modified and the auxiliary loss, alongside the additional trainable parameters (0.2M–1.02B), training time (2.26–2.74 s per update), and inference latency (0.71–1.09 s per action chunk). Success does not consistently increase with parameter count or computational cost, and the type of auxiliary loss alone does not explain the ranking.

Table R6. Integration costs and LIBERO-Plus success with DINOv3.

MechanismAdded params (M)Main module added or modifiedTrain (s/update)Inference (ms/chunk)Auxiliary feature lossSuccess (%)
VAE-only (reference)0—2.245708.5None74.7
REPA12.59Alignment MLP2.261710.9Cosine alignment78.1
Gated cross-attention409.59Visual projections and adapters in action blocks2.739868.4None73.9
Channel-wise0.20Wider video input and output projections2.264757.5Flow matching84.0
Token-wise25.18Feature-token input projection and prediction head2.6521,000.3Flow matching76.1
Feature expert1,022.97Separate 30-block Transformer2.3581,091.8Flow matching78.7

Channel-wise concatenation achieves the highest success with the fewest added parameters. Gated cross-attention adds 409.59M parameters and takes longer than channel-wise concatenation in both training and inference, yet has the lowest success rate (73.9%).

The auxiliary loss follows from how each mechanism uses the features. REPA aligns hidden states with DINOv3 features using cosine similarity, while gated cross-attention conditions on current-frame features without an auxiliary loss. Even among the three mechanisms that predict future features by flow matching, channel-wise concatenation achieves the highest success at 84.0%, compared with 76.1% for token-wise concatenation and 78.7% for the feature expert.

Table R6 measurement details. Training times are per update on 16 PPU-ZW810E accelerators with a global batch size of 128 and exclude offline feature extraction. For inference, we report the median latency on a single PPU-ZW810E at batch size 1 to generate a 32-action chunk in 10 denoising steps.

AuthorsOpenReview ↗

Response to Reviewer fuTJ: Future-feature supervision and information pathways

Q2. How does performance change when the future-feature training objective is retained, but the action stream's access to future features is restricted?

We thank the reviewer for suggesting this control. Our additional experiments suggest that the information pathways enabled by the integration mechanism play a more important role than auxiliary feature-prediction supervision in explaining the performance differences across mechanisms.

We focus on channel-wise concatenation, token-wise concatenation, and feature expert, because all three explicitly predict future external features whose information can reach the action stream through their respective integration pathways. In contrast, REPA applies feature-alignment supervision to intermediate hidden states of the video expert, without providing external future-feature tokens for the action stream to access directly. Gated cross-attention injects current-frame external features and does not involve access to predicted future features.

For each of the three mechanisms, we retain its original future-feature prediction objective, and compare performance with and without restrictions on the action stream’s access to future features.

Table R7. LIBERO-Plus success with original and restricted future-feature access.

Integration mechanismOriginal information pathwaysRestricted access to future features
Channel-wise concatenation84.0%78.5%
Token-wise concatenation76.1%75.8%
Feature expert78.7%77.2%

First, the performance differences between mechanisms become substantially smaller when future-feature access is restricted, even though the auxiliary prediction supervision is retained. Their success rates span 75.8%–78.5% under the restrictions, compared with 76.1%–84.0% with their original pathways. The maximum gap therefore decreases from 7.9 to 2.7 percentage points. This pattern suggests that differences in auxiliary prediction supervision alone are insufficient to explain the larger separation between the original mechanisms; the pathways through which predicted information reaches the action stream are a more important distinguishing factor.

Second, channel-wise concatenation shows a much larger benefit from its original information pathways. Its success rate increases from 78.5% under restricted access to 84.0%, a gain of 5.5 percentage points. The corresponding differences for token-wise concatenation and feature expert are approximately 0.3 and 1.5 points. With the original pathways, channel-wise concatenation also outperforms these alternatives by 7.9 and 5.3 points, respectively. These observations support the interpretation that channel-wise concatenation enables the action stream to use future visual information more effectively, accounting for an important part of its performance advantage.

Reviewer_LPncOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies visual representation choice, integration mechanisms, and feature composition within a common world action model framework. It recommends starting with strong general-purpose representations, jointly predicting future features, and selecting additional sources carefully. The resulting WAM recipe is evaluated on multiple robot benchmarks and real-robot tasks.

Critical strengths
  1. Comparing 13 encoders and five integration mechanisms addresses a practical design question. The common framework is more informative than comparing results across independently developed systems, and the paper documents implementation and computational differences.
  2. Several experiments support the main findings.
Critical weaknesses
  1. Uncertainty in fine-grained rankings is not fully quantified. Appendices C.2 and E.3 use one trained checkpoint per configuration; repeated evaluations measure evaluation variability rather than variability across independent training runs. For example, the difference between 87.30% for three sources and 87.73% for four sources is not established as reproducible across training runs. This limits close rankings and composition interactions, rather than invalidating the much larger 9.3-point gain from adding DINOv3.
  2. The composition results do not by themselves identify semantic complementarity or redundancy as the cause. Equation 9 in Appendix E.1 sums source-specific auxiliary losses, so adding sources also changes the auxiliary objective and channel dimensionality. These factors may affect gains across source counts. The authors already qualify the interpretation of performance interactions and local gradient diagnostics in Appendices E.2 and D.3. The main-text interpretation should retain those qualifications, or be supported by a focused sensitivity check with fixed total auxiliary-loss weight.
Questions for authors
  1. Which conclusions depend on differences below one percentage point? Please qualify those rankings, or provide independent training repetitions for the small set of configurations that determine the main design choices.
  2. Does the key single-source to multi-source trend persist when the total auxiliary-loss weight is comparable? If this evidence is unavailable, please present complementarity as an interpretation consistent with the results rather than an identified cause.
  3. Why are RoboTwin 2.0 clean-to-random (C2R) results not reported? Appendix C.1 (Table 7) uses both clean and randomized demonstrations, so the randomized evaluation in Appendix G (Table 12) does not measure C2R generalization. Given the paper’s emphasis on robustness, could the authors train both R²-WAM and the VAE-only baseline on clean demonstrations only and report clean and randomized success rates separately? This would complement the LIBERO-Plus evidence and help distinguish robustness gained from the representations from the benefit of exposure to randomized training data.
3 comments
AuthorsOpenReview ↗

Response to Reviewer LPnc: Main conclusions and training repetitions

Thank you for your careful reading and for suggesting concrete tests of our conclusions. In response, we independently retrained the three- and four-source models, compared visual sources at matched auxiliary-loss budgets, and tested clean-to-random transfer on RoboTwin 2.0.

Q1. Which conclusions depend on differences below one percentage point?

Our three main conclusions are supported by the broader comparisons summarized below. The small difference between the three- and four-source models provides an additional observation, without being essential to these conclusions.

  1. Representation choice. Across the encoders tested, general-purpose representations average 82.2% success and specialized representations average 80.0%. The recommendation to start with a general-purpose encoder rests on this 2.2-point group difference.

  2. Integration mechanism. Across 13 encoders, channel-wise concatenation averages 81.5%, compared with 76.4% for the next-best mechanism, the feature expert. Its lead is 5.1 points.

  3. Feature composition. Our recommendation is to build on a strong visual representation and select additional sources for complementarity. DINOv3 alone improves LIBERO-Plus success by 9.3 percentage points over VAE-only. The broader composition results show why source selection matters: with exactly two sources, success ranges from 78.59% for DINOv3 + C-RADIOv4 to 86.12% for DINOv3 + UniTok. Moreover, adding VGGT-Ω to DINOv3 + UniTok + V-JEPA 2.1 reduces success from 87.30% to 84.94% (Figure 4 and Table 13). These comparisons show that additional sources can improve or degrade performance depending on the composition, supporting selective combination based on the contribution to existing features. This design recommendation does not require resolving the much smaller 0.43-point difference between the three- and four-source models. We additionally report independent training repetitions of these two configurations in Table R1.

Table R1. LIBERO-Plus success across training seeds for the three- and four-source models.

Training seedThree sources (%)Four sources (%)Δ from adding HRP (pp)
4287.3087.73+0.43
086.4986.77+0.28
AuthorsOpenReview ↗

Response to Reviewer LPnc: Matched auxiliary-loss controls

Q2. Does the single-to-multi-source gain persist at the same total auxiliary-loss weight?

The multi-source advantage persists when the sum of the auxiliary-loss coefficients is matched between the single-source and multi-source models. Increasing the single-source auxiliary-loss weight narrows the performance gap, but does not eliminate the multi-source advantage. At every matched budget, distributing the weight across visual sources yielded higher success than assigning it all to DINOv3. The multi-source advantage persisted when the sum of the auxiliary-loss coefficients was held fixed.

Specifically, for a source set K\mathcal{K} containing KK sources, we use:

Multi-source model:

L=LWAM+λaux∑k∈KLauxk\mathcal{L} = \mathcal{L}_{\mathrm{WAM}} + \lambda_{\mathrm{aux}} \sum_{k\in\mathcal{K}} \mathcal{L}_{\mathrm{aux}}^{k}

Single-source control:

L=LWAM+KλauxLauxDINOv3\mathcal{L} = \mathcal{L}_{\mathrm{WAM}} + K\lambda_{\mathrm{aux}}\mathcal{L}_{\mathrm{aux}}^{\mathrm{DINOv3}}

At the 3× budget, the DINOv3-only control achieved 84.47% success. Splitting the same weight equally among DINOv3, UniTok, and V-JEPA 2.1 raised success to 87.30%, a gain of 2.83 percentage points. The corresponding gains at 2× and 4× were 0.81 and 2.30 percentage points (Table R2).

Table R2. LIBERO-Plus success at matched auxiliary-loss budgets.

Total auxiliary budgetDINOv3 only (%)Multi-source inputsMulti-source (%)Δ (pp)
1×84.00———
2×85.31DINOv3 + UniTok86.12+0.81
3×84.47DINOv3 + UniTok + V-JEPA 2.187.30+2.83
4×85.43Three-source recipe + HRP87.73+2.30

To examine the effect of increasing the number of auxiliary channels, we held the total at 16. Both models used two equally weighted eight-channel prediction targets. For the single-source control, we projected DINOv3 patch features onto their first 16 principal components and used components 1–8 and 9–16 as the two targets. The DINOv3-only model achieved 84.15% success on LIBERO-Plus. The dual-source model achieved 86.12%. So an increase in the number of auxiliary channels alone cannot explain the performance gap.

Table R3. LIBERO-Plus success with 16 auxiliary channels.

Auxiliary features (16 channels total)LIBERO-Plus success (%)
DINOv3 PCA16 (8 + 8)84.15
DINOv3 PCA8 + UniTok PCA886.12
AuthorsOpenReview ↗

Response to Reviewer LPnc: Clean-to-random transfer

Q3. How well do policies trained on clean demonstrations transfer to randomized RoboTwin 2.0 scenes?

The original RoboTwin experiment included randomized demonstrations during training. To test whether external visual representations improve robustness to scene variations unseen during training, we trained three policies on clean demonstrations only. We then evaluated each policy separately on clean and randomized scenes. On randomized scenes, success increased from 4.12% for VAE-only to 10.50% with DINOv3 and 12.16% with the full R²-WAM.

Table R4. RoboTwin 2.0 success after clean-only training.

ModelClean success (%)Randomized success (%)Avg success (%)
VAE-only82.384.1243.25
VAE + DINOv385.6410.5048.07
R²-WAM85.3212.1648.74

Training configuration. We trained all three policies for 10 epochs on a common set of 2,500 clean demonstrations (50 per task across 50 tasks). We used a global batch size of 256, a base learning rate of 0.0001 with cosine decay.

Evaluation protocol. Following the RoboTwin 2.0 benchmark protocol, we evaluated each policy on all 50 RoboTwin tasks, with 100 trials per task in each of the two conditions (clean and randomized).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

Dear Area Chair and Reviewers,

Thank you for your constructive feedback. We have revised the manuscript to incorporate the following additional experiments and analyses:

  • Information pathways and future-feature supervision: Section 4.3, Table 2, and Appendix D.4.
  • Auxiliary-loss weight and channel-count controls for feature composition: Section 4.4, Table 3, and Appendix E.4.
  • Real-robot generalization under background, viewpoint, and lighting changes: Appendix C.3, Table 10, and Figure 7.
  • Single-source and multi-source comparison on RoboCasa: Appendix F, Table 15 compares VAE-only, VAE + DINOv3, and the full R²-WAM.
  • Single-source and multi-source comparison on RoboTwin under clean-only training: Appendix G, Table 17 reports the three models’ performance in clean and randomized environments under the clean-to-random (C2R) protocol.
  • Related-work discussion: Section 2 now cites ReWAM and clarifies its relationship to our systematic study.

The corresponding additions and substantive changes are highlighted in red in the revised PDF. Detailed results and discussions are provided in our responses to the individual reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.