- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper evaluates 20 pre-trained visual encoders across 24 dual-arm manipulation tasks within the RoboCasa-GR1 and RoboTwin 2.0 frameworks, employing a shared patch-conditioned policy architecture. It investigates the impacts of fine-tuning, task sharing, data volume, model scale, and specific feature layers. Furthermore, the study analyzes token statistics, probing for state and object localization, the use of Inverse Dynamics Models (IDM), and the correlation between standard visual benchmarks and closed-loop control performance, while also examining combinations of metrics and consistency across different experimental settings. The authors propose a two-stage selection process: first, using an IDM to filter out weaker candidates, followed by closed-loop experiments to finalize the encoder choice--emphasizing that standard visual benchmarks and model scale are not direct substitutes for control-based evaluation.
Critical strengths
-
The paper features an extensive experimental scope, encompassing multiple encoder families, two dual-arm manipulation benchmarks, and a variety of training and adaptation configurations
-
By utilizing a consistent patch-conditioned policy backbone, while keeping data, camera configurations, optimization protocols, and evaluation pipelines fixed across benchmarks, the paper enhances the comparability of different visual representations.
-
The paper provides a comprehensive analysis of the experimental results, covering representation interfaces, fine-tuning, data and model scale, as well as the relationship between diagnostic metrics and success rates. This offers valuable insights into the applicability of various encoders.
Critical weaknesses
- The robotic control evaluations in this paper are derived exclusively from two simulation suites; consequently, the extent to which these findings generalize to the visual discrepancies encountered in real-world environments remains unverified. Real-world experiments are critical for robotic tasks. The authors should, at a minimum, incorporate open-loop probing evaluations using real-world robotic data, and, if feasible, include closed-loop results from physical robot experiments. Alternatively, drawing on Reference [1], the authors could leverage SimplerEnv and its corresponding real-world tasks to assess whether simulation-based metrics or rankings can effectively predict performance in real-world control scenarios.
[1] Capturing Visual Environment Structure Correlates with Control Performance (ICLR 2026) https://arxiv.org/abs/2602.04880
- The novelty of the diagnostic metrics proposed in this paper is somewhat limited: Reference [2] has already utilized IDMs to assess action recoverability, while Reference [1] has explored proxy metrics such as state decoding, segmentation, and depth estimation. Although the subjects of evaluation differ, the paper should more clearly articulate its specific, novel contributions to the field. While the paper emphasizes the effectiveness of IDM, there are counterexamples in the rankings, such as V-JEPA 2.1 and VGGT-Ω, where the method fails to align with performance. Consequently, beyond overall correlation, I would like to see a clearer delineation of the method’s capabilities: is IDM merely capable of filtering out weak candidates, or can it reliably identify the optimal encoder?
[2] Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models, https://arxiv.org/abs/2605.06388
- Although the paper provides extensive diagnostic analysis, the results still do not offer clear guidance on the actual selection of encoders. Existing experiments of leaving encoder regression and candidate screening support the value of the indicators for prediction and preliminary screening, but have not directly explained whether probe-assisted screening can reduce the cost of closed-loop evaluation while retaining high-performance candidates compared to not using these indicators. In particular, the authors also pointed out that IDM has limited sorting ability for strong candidates, and the final selection still depends on closed-loop experiments. Therefore, it is recommended to further clarify the method's ability boundary between excluding weak candidates and identifying strong candidates, and through the comparison of selection quality and evaluation cost, to more directly reflect its practical guiding value.
Questions for authors
- What is the specific novelty of this paper’s egocentric setting, will the analysis results change for non-egocentric settings?
- What is the rationale for introducing 32 additional "query tokens" into the action head? How do these differ from standard "action tokens," and what is the basis for choosing this specific number? While the appendix explains how these tokens participate in flow-matching velocity prediction via attention, I would like to see ablation studies, removing these query tokens or varying their count, to justify this design choice and assess its impact on encoder comparisons.
- Different encoders produce varying numbers of visual tokens (ranging from 64 to 576) and feature different widths. Simply standardizing the number of training steps and the policy backbone does not equate to matching the FLOPs in the cross-attention layers; furthermore, variations in context width alter the parameter counts of the projection layers and the action decoder. Could the authors report the downstream parameter counts and computational costs for each configuration?
Thank you for asking what the benchmark adds and when its diagnostics help choose an encoder.
Weaknesses
W1. Do the findings extend to physical control?
We added physical closed-loop evaluation on an ARX A7 bimanual robot with an egocentric RealSense D405. Three encoders share multi-task training with 50 demonstrations per task and 100 trials per encoder/task: 900 trials (App. D).
DINOv2 leads throughout: 49.67% on average versus 41.00% and 17.00%, preserving the simulation ordering. The real-robot policies are trained with state input. The consistent ordering adds physical evidence for the encoder comparison.
W2. What is new beyond the two cited studies?
Thank you for pointing us to these studies. Related Work and Table 1 now credit their contributions and clarify how our evaluation extends this line of work:
Our contribution is comprehensive evaluation of this policy setting and how diagnostic associations change with training. Your V-JEPA/VGGT counterexamples show why correlation need not identify the best encoder. On Twin, IDM–success correlation is −0.80 across frozen/adapted checkpoints, but −0.40 within adapted checkpoints. Improving on IDM across settings remains an open problem.
W3. Does screening beat random selection at the same budget?
We compare five lowest-IDM candidates with five random candidates and the full cohort. Random is the exact expected best over uniformly sampled subsets.
On Twin, five instead of twenty policy runs retain success within 0.45 points of the best. On both Casa settings IDM loses to random; state and single-frame probes also lose on Casa ST. Even on Twin, IDM-3 gives 45.17%, below random-three's 47.33%. The practical benefit therefore depends on the setting and shortlist size. We report policy-run budgets here, with probe training as an additional cost (Sec. 5.4, App. E).
Questions
Q1. Why egocentric patches, and would other views change the answer?
Moving head/wrist views bring occlusion and fine local contact detail. Pooling patches lowers DINOv2/DINOv3 Casa success from 32.28/31.22% to 9.94/6.44%. Keeping spatial tokens therefore matters to the evaluation itself. This motivates evaluating the spatial interface increasingly used by egocentric policies. A matched camera-swap study would help determine which findings also hold under third-person views.
Q2. Why 32 queries rather than only action tokens?
The 16 action tokens represent the noisy action chunk and are decoded into velocity. The 32 queries provide extra learned attention positions without action targets; their count follows the common head. Removing them lowers Casa success from 32.28→27.78% for DINOv2 and 31.22→23.50% for DINOv3. The queries help both encoders and preserve their order (App. C.5). We retained 32 from the shared architecture; optimizing their count remains a separate design question.
Q3. Are downstream capacity and compute matched?
Appendix A reports all 20 encoders' counts. Thirteen share a 138.669M action head; the largest is 156.478M (12.8% more). Downstream totals add external projections and include the allocated state MLP even when unused; pretrained encoders are excluded. Table 3 reports per-encoder training costs: DINOv2 uses 54.84 TFLOPs/0.662 s per Casa MT step, VGGT 113.35/0.869, and VLMs about 195.35/1.12–1.14. Encoder size, token interfaces, and context-dependent projections all contribute to the observed compute differences. Equal training steps therefore do not imply equal computational cost.