- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes FakeVLA, a framework for evaluating visual representations for robotic manipulation without training a downstream action predictor. FakeVLA uses nearest-neighbor retrieval in the feature space of a frozen vision encoder to select and execute actions from demonstrations. The authors also introduce StateNN–VisionNN Drift, a metric that measures the physical-state discrepancy between demonstrations retrieved using visual features and ground-truth robot states. Experiments across different visual encoders, environmental perturbations, and downstream action predictors show that lower Drift is strongly correlated with manipulation success, while representation robustness varies across visual shifts. The paper further compares probe performance with full VLA policies and investigates sim-to-real transfer, finding that strong within-domain performance does not necessarily translate to cross-domain retrieval.
Critical strengths
- The paper presents an interesting and novel perspective on evaluating visual representations for robotic manipulation. By using nearest-neighbor action retrieval instead of training a downstream policy, FakeVLA provides a simple and intuitive way to examine the manipulation-relevant information captured by different vision encoders.
- The paper provides useful empirical insights into visual representations for robotic manipulation. The authors systematically compare different vision encoders under various environmental perturbations and investigate their relationships with downstream VLA performance. The analysis of representation robustness, action decodability, and sim-to-real transfer provides a more comprehensive understanding of the strengths and limitations of existing visual representations.
Critical weaknesses
-
Insufficient state representation in StateNN. StateNN relies only on the end-effector pose and gripper state at a single timestep, with the distance largely dominated by the end-effector position. I find this representation too limited to meaningfully characterize manipulation states. Similar end-effector positions do not necessarily imply similar task states or appropriate actions, especially when object poses, robot configurations, and motion histories differ. A more comprehensive state representation incorporating these factors would make StateNN a more convincing reference.
-
The experimental evaluation is limited in task diversity, and some important experimental details are missing.
- 2.1. Limited experimental validation. The experiments only consider tabletop pick-and-place tasks with two objects and fixed pick-and-place locations, while the real-world evaluation is restricted to a single object. Such a constrained setting may be particularly favorable for nearest-neighbor action retrieval. The limited task diversity makes it difficult to assess the practical applicability of FakeVLA or whether the observed correlation between Drift and manipulation success would hold in more complex and diverse manipulation scenarios.
- 2.2. Insufficient experimental details. Some important details of the experimental setup are missing, including the robot hardware, camera viewpoints, number of cameras, and image resolution. For a paper focusing on the evaluation of visual representations, I would expect to see example observations from both simulation and real-world environments, as well as visualizations of the different environmental perturbations. Without these, it is difficult to understand the actual visual inputs or assess the severity of camera-pose and texture shifts, as well as the sim-to-real domain gap. The implementation of texture randomization is also not sufficiently described. These missing details make it difficult to interpret the experimental results and assess their reproducibility.
Questions for authors
-
How would a more comprehensive state representation affect the proposed Drift metric? Have the authors considered incorporating object poses, robot joint configurations, or temporal state histories into StateNN? If the correlation remains strong under these more comprehensive settings, it would provide more convincing evidence that Drift is a meaningful metric for evaluating manipulation-relevant visual representations.
-
How sensitive is FakeVLA to the size and diversity of the demonstration memory? Since FakeVLA relies on nearest-neighbor retrieval, its performance may depend heavily on the coverage of the demonstration dataset. An ablation study varying the size and diversity of the demonstration memory would be valuable to understand their impact on manipulation success and StateNN–VisionNN Drift.
-
Why is StateNN–VisionNN Drift preferred over direct state retrieval error? The proposed metric measures the discrepancy between the states retrieved by StateNN and VisionNN, but both retrieved states could be far from the actual query state. Would directly measuring the physical-state distance between the current state and the VisionNN-retrieved state provide a more meaningful evaluation of representation quality?
-
Can FakeVLA reliably predict the performance ranking of visual encoders in end-to-end policy learning? Although the paper reports moderate correlations between FakeVLA and full-VLA performance, the compared VLA models differ in their policy architectures and training procedures. A more convincing evaluation would be to train the same downstream policy architecture with different frozen vision encoders, and compare the resulting policy success rates with the FakeVLA rankings.
5 comments
Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns.
However, I still have a concern about the effectiveness of FakeVLA as a probe for evaluating visual representations. As I understand it, the motivation is to evaluate vision encoders without the need to train downstream policies. However, visually similar observations do not necessarily require similar actions, especially across different tasks or manipulation stages. Therefore, poor nearest-neighbor action retrieval performance does not necessarily indicate a poor visual representation.
More importantly, the additional results in Appendix D.7 show weak agreement between FakeVLA and full-VLA performance rankings. This raises a question about how reliably FakeVLA can reflect the usefulness of a vision encoder for downstream manipulation tasks. While I understand that FakeVLA is intended to analyze local feature geometry rather than directly predict full-VLA performance, I am still not fully convinced of its practical value as a general probe for evaluating manipulation-relevant visual representations. For example, given the weak agreement between FakeVLA and full-VLA rankings, how should practitioners interpret and use the evaluation results when comparing different vision encoders? What practical conclusions about an encoder's suitability for downstream manipulation can be drawn from its FakeVLA performance?
Unfortunately, this concern remains significant to my overall assessment, so I will maintain my original rating at this stage.
Reply To Reviewer sFxW
Thank you for clarifying this concern.
- We would like to highlight that, for OpenVLA and GR00T, our results do suggest strong correlation between FakeVLA policy and VLA performance: Across six conditions, grasp/placement correlations are 0.90/0.71 for OpenVLA and 0.89/0.86 for GR00T.
- Lower correlation for π0.5 and X-VLA is likely due to the differences in how these models use visual features: OpenVLA-OFT and GR00T pass extracted VLM features to separate action modules. Both π0.5 and X-VLA use attention to condition action generation on visual information. π0.5’s action expert accesses VLM features across transformer layers, while X-VLA combines different camera-processing paths with learned domain prompts. These mechanisms may help policies use information that nearest-neighbour retrieval does not readily access.
Our central claim is the following: When building a VLA, FakeVLA can help assess what frozen visual features support before training the full policy. Its results guide decisions about feature use, robustness, and data collection coverage. They may not determine which encoder will produce the best final VLA for some VLA design choices.
We find the following practical benefits helpful for the community:
- Training-free Policy: The features support successful manipulation through a simple action-retrieval mechanism. This establishes a working baseline without training an action head. This can especially be helpful in static environments with conditions against which the visual features are robust.
- FakeVLA tests whether action retrieval remains reliable when visual conditions change. Comparing it with an MLP on the same frozen features shows that strong performance under familiar conditions can hide different weaknesses under changes in lighting, camera pose, or texture.
- FakeVLA measures how demonstration coverage affects action retrieval. Increasing memory from 60 to 240 demonstrations improved grasp and placement for all nine encoders, showing why retrieval failures cannot be attributed solely to encoder quality.
- FakeVLA helps distinguish poor retrieval from information that a learned predictor can use. When an MLP succeeds on the same frozen features where NN struggles, practitioners have evidence to investigate the action mapping before replacing the encoder. Our experiments demonstrate this distinction under familiar observations.
- FakeVLA makes action selection inspectable. Each selected action can be traced to a specific demonstration image and state. Practitioners can inspect whether retrieval matches the robot pose, object arrangement, or manipulation stage, alongside task success and positional error.
A practitioner should therefore use FakeVLA to test specific properties and identify weaknesses during VLA development. The final encoder choice should be validated with the intended policy architecture. While weak agreement on full-VLA rankings is a limitation for this particular use of the probe; it does not remove the value of the controlled comparisons it provides, as discussed above.
Reply To Reviewer sFxW
We thank the reviewer for recognizing the simplicity and novelty of nearest-neighbour action retrieval as a probe, and the breadth of the analysis across visual robustness, action prediction and sim-to-real transfer.
W1 / Q1. Using a richer state reference. Following the reviewer’s suggestion, we added object poses, seven joint angles and short motion histories to StateNN, individually and together, while keeping the visual match fixed for each query. Adding object poses reduced the state reference’s apple-position error from 10.1 to 4.0 mm. Adding joint angles reduced joint error from 0.058 to 0.024 rad. With normalized richer references, mean Drift increased from 5.4 mm to 7.2–12.0 mm. A reference could therefore match the scene more closely while selecting a demonstration farther from the visual match.
Across 431 closed-loop trials, adding the new inputs changed the correlation with grasp success by at most 0.04. We clarified this dependence on the state reference and reported the richer-reference results alongside direct error. See Common Response C2 and Appendix D.5 for the state variants and full results. The association between Drift and success is therefore not an artefact of the simple state reference, and we present Drift as a consistency measure next to the direct error.
W2.1. Task diversity. In simulation, we distributed the apple and bowl positions across nine layouts and reused the same ten robot starts across layouts. With 240 demonstrations, placement success ranged from 11% to 29% across nine encoders. We also evaluated FakeVLA on all ten LIBERO-Spatial tasks, with 200 rollouts per encoder and condition. Under the standard setup, task success ranged from 53.5% to 69.5% across the four VLA encoders. See Common Response C1 and Appendices D.1 and D.9 for both evaluations. These experiments extended the evaluation to more object arrangements and tasks and revealed differences in how the encoders handled them.
W2.2. Experimental details. We expanded the description of the Franka Panda setup and its front and wrist RGB cameras. The real and simulated robot used Intel RealSense D435 cameras. We documented image capture at 1280 × 720, resizing of demonstration frames to 224 × 224, and each encoder’s preprocessing. We also corrected the camera-perturbation description. The front view was shifted around a nominal pose about 15 cm above the training view, with offsets of ±2 cm per axis, ±15° in roll and ±5° in pitch and yaw. See Appendix B.1 for the camera setup, lighting and texture settings, and examples from simulation and the real robot.
Q2. Demonstration size and diversity. We tested memories of 60, 120 and 240 demonstrations. All encoders used the same demonstrations within each configuration, and we tested three sampled memories at each smaller size. Increasing memory from 60 to 240 demonstrations improved grasp and placement success for all nine encoders and reduced Drift for eight. For example, placement rose from 15% to 29% for π0.5 and from 9% to 28% for DINO-SigLIP. In a separate test on recorded observations, increasing memory from 12 to 60 demonstrations reduced direct positional error on held-out layouts from 20.2 to 12.0 mm. At a fixed size of 30 demonstrations, covering six layouts instead of three increased correct layout retrieval from 50% to 78%. See Common Response C4 and Appendices D.1 and D.6 for the sampling protocol and full results.
Q3. Drift and direct error. We retained Drift as our primary retrieval metric and added direct positional error as a complementary check. It measures how far apart the gripper positions are in the demonstrations selected by VisionNN and StateNN. StateNN uses the current robot state to select its match. Direct error measures how far the visually retrieved gripper position is from the robot’s current gripper position. On the same held-out observations, mean Drift was 7.02 mm and mean direct error was 7.06 mm. Both measures showed similar associations with grasp success in the rollout analysis (Appendices D.4 and D.5).
Q4. Using the same action predictor across encoders. Experiment 2 used the same MLP architecture, trained separately on each frozen encoder’s features. We recomputed the correlations with full-policy performance using the existing results for all four models, including OpenVLA. The pooled MLP–VLA correlations were 0.627 for grasping and 0.572 for placement, compared with 0.570 and 0.396 for NN–VLA. We also ranked the models separately in each test setting, such as unseen robot starts or a shift in camera pose. In that analysis, the NN probe and the full VLA policies showed weak agreement on model rankings (Kendall’s τ ≈ 0.1). Evaluating frozen encoders with a shared action architecture at full VLA scale is a direction for future work. See Common Response C3 and Appendix D.7 for more details.