acceptodds
Under review as a conference paper at ICLR 2027

FAKEVLA: A PROBE FOR ANALYZING VISUAL REPRESENTATIONS FOR ROBOT ACTIONS

Abstract

Vision encoders are commonly evaluated frozen through downstream probes that reveal what information is encoded and how readily it can be used for prediction. In robotic manipulation, however, visual representations are typically evaluated within complete visuomotor policies, making it difficult to separate representation quality from downstream control. We propose FakeVLA, a framework for probing frozen visual representations through manipulation-specific tests that connect feature geometry to action selection. FakeVLA uses nearest-neighbor (NN) retrieval over demonstrations to test whether local visual neighborhoods recover manipulation-relevant physical states, finding that stronger state recovery is closely associated with higher Grasp and Place success. Further lightweight action predictors are used to test whether useful information remains decodable beyond local NN geometry. We evaluate generic, vision-language, and VLAderived representations under controlled state and visual shifts, and compare probe trends with corresponding full-VLA policies whose vision modules remain frozen. Lower Drift is strongly associated with higher manipulation success, while texture causes the largest robustness degradation. Nonlinear predictors perform strongly under familiar observations but degrade more sharply under visual shifts than NN retrieval, showing that action decodability and local geometry capture distinct properties. Probe performance is also positively associated with full-VLA performance. Finally, strong within-domain performance does not imply sim-to-real alignment: cross-domain retrieval remains substantially weaker, although VLA pretraining and real-domain adaptation improve SimReal transfer.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Representation Learning, Computer Vision, Robotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sFxWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes FakeVLA, a framework for evaluating visual representations for robotic manipulation without training a downstream action predictor. FakeVLA uses nearest-neighbor retrieval in the feature space of a frozen vision encoder to select and execute actions from demonstrations. The authors also introduce StateNN–VisionNN Drift, a metric that measures the physical-state discrepancy between demonstrations retrieved using visual features and ground-truth robot states. Experiments across different visual encoders, environmental perturbations, and downstream action predictors show that lower Drift is strongly correlated with manipulation success, while representation robustness varies across visual shifts. The paper further compares probe performance with full VLA policies and investigates sim-to-real transfer, finding that strong within-domain performance does not necessarily translate to cross-domain retrieval.

Critical strengths
  1. The paper presents an interesting and novel perspective on evaluating visual representations for robotic manipulation. By using nearest-neighbor action retrieval instead of training a downstream policy, FakeVLA provides a simple and intuitive way to examine the manipulation-relevant information captured by different vision encoders.
  2. The paper provides useful empirical insights into visual representations for robotic manipulation. The authors systematically compare different vision encoders under various environmental perturbations and investigate their relationships with downstream VLA performance. The analysis of representation robustness, action decodability, and sim-to-real transfer provides a more comprehensive understanding of the strengths and limitations of existing visual representations.
Critical weaknesses
  1. Insufficient state representation in StateNN. StateNN relies only on the end-effector pose and gripper state at a single timestep, with the distance largely dominated by the end-effector position. I find this representation too limited to meaningfully characterize manipulation states. Similar end-effector positions do not necessarily imply similar task states or appropriate actions, especially when object poses, robot configurations, and motion histories differ. A more comprehensive state representation incorporating these factors would make StateNN a more convincing reference.

  2. The experimental evaluation is limited in task diversity, and some important experimental details are missing.

  • 2.1. Limited experimental validation. The experiments only consider tabletop pick-and-place tasks with two objects and fixed pick-and-place locations, while the real-world evaluation is restricted to a single object. Such a constrained setting may be particularly favorable for nearest-neighbor action retrieval. The limited task diversity makes it difficult to assess the practical applicability of FakeVLA or whether the observed correlation between Drift and manipulation success would hold in more complex and diverse manipulation scenarios.
  • 2.2. Insufficient experimental details. Some important details of the experimental setup are missing, including the robot hardware, camera viewpoints, number of cameras, and image resolution. For a paper focusing on the evaluation of visual representations, I would expect to see example observations from both simulation and real-world environments, as well as visualizations of the different environmental perturbations. Without these, it is difficult to understand the actual visual inputs or assess the severity of camera-pose and texture shifts, as well as the sim-to-real domain gap. The implementation of texture randomization is also not sufficiently described. These missing details make it difficult to interpret the experimental results and assess their reproducibility.
Questions for authors
  1. How would a more comprehensive state representation affect the proposed Drift metric? Have the authors considered incorporating object poses, robot joint configurations, or temporal state histories into StateNN? If the correlation remains strong under these more comprehensive settings, it would provide more convincing evidence that Drift is a meaningful metric for evaluating manipulation-relevant visual representations.

  2. How sensitive is FakeVLA to the size and diversity of the demonstration memory? Since FakeVLA relies on nearest-neighbor retrieval, its performance may depend heavily on the coverage of the demonstration dataset. An ablation study varying the size and diversity of the demonstration memory would be valuable to understand their impact on manipulation success and StateNN–VisionNN Drift.

  3. Why is StateNN–VisionNN Drift preferred over direct state retrieval error? The proposed metric measures the discrepancy between the states retrieved by StateNN and VisionNN, but both retrieved states could be far from the actual query state. Would directly measuring the physical-state distance between the current state and the VisionNN-retrieved state provide a more meaningful evaluation of representation quality?

  4. Can FakeVLA reliably predict the performance ranking of visual encoders in end-to-end policy learning? Although the paper reports moderate correlations between FakeVLA and full-VLA performance, the compared VLA models differ in their policy architectures and training procedures. A more convincing evaluation would be to train the same downstream policy architecture with different frozen vision encoders, and compare the resulting policy success rates with the FakeVLA rankings.

5 comments
AuthorsOpenReview ↗

Reply To Reviewer sFxW

We thank the reviewer for recognizing the simplicity and novelty of nearest-neighbour action retrieval as a probe, and the breadth of the analysis across visual robustness, action prediction and sim-to-real transfer.

W1 / Q1. Using a richer state reference. Following the reviewer’s suggestion, we added object poses, seven joint angles and short motion histories to StateNN, individually and together, while keeping the visual match fixed for each query. Adding object poses reduced the state reference’s apple-position error from 10.1 to 4.0 mm. Adding joint angles reduced joint error from 0.058 to 0.024 rad. With normalized richer references, mean Drift increased from 5.4 mm to 7.2–12.0 mm. A reference could therefore match the scene more closely while selecting a demonstration farther from the visual match.

Across 431 closed-loop trials, adding the new inputs changed the correlation with grasp success by at most 0.04. We clarified this dependence on the state reference and reported the richer-reference results alongside direct error. See Common Response C2 and Appendix D.5 for the state variants and full results. The association between Drift and success is therefore not an artefact of the simple state reference, and we present Drift as a consistency measure next to the direct error.

W2.1. Task diversity. In simulation, we distributed the apple and bowl positions across nine layouts and reused the same ten robot starts across layouts. With 240 demonstrations, placement success ranged from 11% to 29% across nine encoders. We also evaluated FakeVLA on all ten LIBERO-Spatial tasks, with 200 rollouts per encoder and condition. Under the standard setup, task success ranged from 53.5% to 69.5% across the four VLA encoders. See Common Response C1 and Appendices D.1 and D.9 for both evaluations. These experiments extended the evaluation to more object arrangements and tasks and revealed differences in how the encoders handled them.

W2.2. Experimental details. We expanded the description of the Franka Panda setup and its front and wrist RGB cameras. The real and simulated robot used Intel RealSense D435 cameras. We documented image capture at 1280 × 720, resizing of demonstration frames to 224 × 224, and each encoder’s preprocessing. We also corrected the camera-perturbation description. The front view was shifted around a nominal pose about 15 cm above the training view, with offsets of ±2 cm per axis, ±15° in roll and ±5° in pitch and yaw. See Appendix B.1 for the camera setup, lighting and texture settings, and examples from simulation and the real robot.

Q2. Demonstration size and diversity. We tested memories of 60, 120 and 240 demonstrations. All encoders used the same demonstrations within each configuration, and we tested three sampled memories at each smaller size. Increasing memory from 60 to 240 demonstrations improved grasp and placement success for all nine encoders and reduced Drift for eight. For example, placement rose from 15% to 29% for π0.5 and from 9% to 28% for DINO-SigLIP. In a separate test on recorded observations, increasing memory from 12 to 60 demonstrations reduced direct positional error on held-out layouts from 20.2 to 12.0 mm. At a fixed size of 30 demonstrations, covering six layouts instead of three increased correct layout retrieval from 50% to 78%. See Common Response C4 and Appendices D.1 and D.6 for the sampling protocol and full results. 

Q3. Drift and direct error. We retained Drift as our primary retrieval metric and added direct positional error as a complementary check. It measures how far apart the gripper positions are in the demonstrations selected by VisionNN and StateNN. StateNN uses the current robot state to select its match. Direct error measures how far the visually retrieved gripper position is from the robot’s current gripper position. On the same held-out observations, mean Drift was 7.02 mm and mean direct error was 7.06 mm. Both measures showed similar associations with grasp success in the rollout analysis (Appendices D.4 and D.5).

Q4. Using the same action predictor across encoders. Experiment 2 used the same MLP architecture, trained separately on each frozen encoder’s features. We recomputed the correlations with full-policy performance using the existing results for all four models, including OpenVLA. The pooled MLP–VLA correlations were 0.627 for grasping and 0.572 for placement, compared with 0.570 and 0.396 for NN–VLA. We also ranked the models separately in each test setting, such as unseen robot starts or a shift in camera pose. In that analysis, the NN probe and the full VLA policies showed weak agreement on model rankings (Kendall’s τ ≈ 0.1). Evaluating frozen encoders with a shared action architecture at full VLA scale is a direction for future work. See Common Response C3 and Appendix D.7 for more details.

Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns.

However, I still have a concern about the effectiveness of FakeVLA as a probe for evaluating visual representations. As I understand it, the motivation is to evaluate vision encoders without the need to train downstream policies. However, visually similar observations do not necessarily require similar actions, especially across different tasks or manipulation stages. Therefore, poor nearest-neighbor action retrieval performance does not necessarily indicate a poor visual representation.

More importantly, the additional results in Appendix D.7 show weak agreement between FakeVLA and full-VLA performance rankings. This raises a question about how reliably FakeVLA can reflect the usefulness of a vision encoder for downstream manipulation tasks. While I understand that FakeVLA is intended to analyze local feature geometry rather than directly predict full-VLA performance, I am still not fully convinced of its practical value as a general probe for evaluating manipulation-relevant visual representations. For example, given the weak agreement between FakeVLA and full-VLA rankings, how should practitioners interpret and use the evaluation results when comparing different vision encoders? What practical conclusions about an encoder's suitability for downstream manipulation can be drawn from its FakeVLA performance?

Unfortunately, this concern remains significant to my overall assessment, so I will maintain my original rating at this stage.

AuthorsOpenReview ↗

Reply To Reviewer sFxW

Thank you for clarifying this concern. 

  • We would like to highlight that, for OpenVLA and GR00T, our results do suggest strong correlation between FakeVLA policy and VLA performance: Across six conditions, grasp/placement correlations are 0.90/0.71 for OpenVLA and 0.89/0.86 for GR00T. 
  • Lower correlation for π0.5 and X-VLA is likely due to the differences in how these models use visual features: OpenVLA-OFT and GR00T pass extracted VLM features to separate action modules. Both π0.5 and X-VLA use attention to condition action generation on visual information. π0.5’s action expert accesses VLM features across transformer layers, while X-VLA combines different camera-processing paths with learned domain prompts. These mechanisms may help policies use information that nearest-neighbour retrieval does not readily access. 

Our central claim is the following: When building a VLA, FakeVLA can help assess what frozen visual features support before training the full policy. Its results guide decisions about feature use, robustness, and data collection coverage. They may not determine which encoder will produce the best final VLA for some VLA design choices.

We find the following practical benefits helpful for the community:

  • Training-free Policy: The features support successful manipulation through a simple action-retrieval mechanism. This establishes a working baseline without training an action head. This can especially be helpful in static environments with conditions against which the visual features are robust.
  • FakeVLA tests whether action retrieval remains reliable when visual conditions change. Comparing it with an MLP on the same frozen features shows that strong performance under familiar conditions can hide different weaknesses under changes in lighting, camera pose, or texture.
  • FakeVLA measures how demonstration coverage affects action retrieval. Increasing memory from 60 to 240 demonstrations improved grasp and placement for all nine encoders, showing why retrieval failures cannot be attributed solely to encoder quality.
  • FakeVLA helps distinguish poor retrieval from information that a learned predictor can use. When an MLP succeeds on the same frozen features where NN struggles, practitioners have evidence to investigate the action mapping before replacing the encoder. Our experiments demonstrate this distinction under familiar observations.
  • FakeVLA makes action selection inspectable. Each selected action can be traced to a specific demonstration image and state. Practitioners can inspect whether retrieval matches the robot pose, object arrangement, or manipulation stage, alongside task success and positional error.

A practitioner should therefore use FakeVLA to test specific properties and identify weaknesses during VLA development. The final encoder choice should be validated with the intended policy architecture. While weak agreement on full-VLA rankings is a limitation for this particular use of the probe; it does not remove the value of the controlled comparisons it provides, as discussed above.

Reviewer_F3t9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FakeVLA studies how to assess robot visual representations beyond complete-policy success. It uses frozen-feature nearest-neighbor retrieval to reuse demonstrated actions. Its StateNN-VisionNN Drift metric measures the distance between demonstration states selected by visual and state-based retrieval. Validation includes controlled visual perturbations, linear and MLP action probes, full VLA comparisons, and transfer between simulation and reality. The contribution is a diagnostic framework and empirical findings showing that action prediction performance, local neighborhood structure, and robustness capture different aspects of representation quality in the evaluated pick-and-place settings.

Critical strengths
  1. Useful diagnostic design. Sections 3-4 compare NN and learned predictors on the same frozen features. This tests what the features support with and without a trained action head, complementing full-policy evaluation.
  2. Meaningful empirical insight. MLP achieves higher average Place success than NN under familiar observations but lower success under visual shifts (Figure 4, Appendix Table 3). This shows why one decoder alone can give an incomplete picture of representation quality.
Critical weaknesses
  1. The diagnostic interpretation of Drift needs stronger support. The paper uses Drift’s correlation with success to support its value for evaluating visual representations. However, Drift measures the distance between visually retrieved and state-retrieved demonstration states, not their distances to the current robot state (Sections 3.2 and 4.2). Therefore, the agreement does not ensure that either is close to the current state. Also, Drift is measured on each encoder’s own rollouts, where different actions can change the states visited and the retrieved neighbors. The analysis does not clearly separate retrieval quality from these rollout effects, which limits confidence in what Drift reveals about representation quality.
  2. The relationship between probe performance and full-VLA performance needs clarification. Section 4.5 reports correlations for three models without explaining why OpenVLA, reported in Appendix Table 3, is excluded. Including all four models yields an NN-VLA Place correlation of approximately 0.396, versus 0.627 for the reported subset. The association remains positive, while pooling multiple conditions may capture both systems performing worse under harder conditions. This does not establish agreement on encoder rankings within the same condition. Including OpenVLA and separating these relationships would clarify what the probe can tell us about full-policy behavior.
Questions for authors
  1. In StateNN’s L1 distance, what units are used for position and orientation? Are any state components rescaled? Using the same held-out observations across encoders, can you compare Drift with the positional distances from the current robot state to (i) the VisionNN-selected state and (ii) the StateNN-selected state?
  2. Why was OpenVLA excluded from the correlation analysis in Section 4.5? Please report results for all four models. For each model, do NN and full-VLA success rates rise and fall together across conditions? Within each condition, do the two evaluations give similar model rankings?
  3. For NN, Linear, MLP, and full VLA, what do the action values represent? Please explain how they become robot commands, including any coordinate conversion or rescaling. How many actions are retrieved or predicted at once, and how many are executed before a new observation is used? Which policies, if any, also receive measured robot states as inputs?
  4. Section 4.1 fixes the pick and place locations. Do any existing tests move the object or placement target while keeping the robot’s starting pose similar, and show successful grasping or placement at the new location? If not, please clarify that the spatial-understanding claim in Section 1 has only been tested at fixed locations.
  5. In Section 4.6, are actions retrieved from the other domain executed unchanged or converted? Do existing tests compare visual and state-based retrieval using the same controller, or check cross-domain visual retrieval at matched physical states? If neither test is available, what evidence links the performance drop in Figure 5 specifically to poor visual alignment between simulation and reality?
5 comments
AuthorsOpenReview ↗

Reply To Reviewer F3t9

We thank the reviewer for recognizing the value of comparing retrieval and learned predictors on the same frozen features. We also appreciate their observation that the MLP’s advantage in placement under familiar observations reverses under visual shifts.

W1 / Q1. What Drift measures. We clarified the distinction between Drift and direct error. Drift measures the distance between the end-effector positions in the demonstrations selected by VisionNN and StateNN. Direct error measures the distance from the robot’s current end-effector position to the VisionNN match. Both are Euclidean distances reported in millimetres. StateNN selects its neighbour using position in metres in the robot base frame, wrapped roll, pitch and yaw differences in radians, and gripper state encoded as ±1.

We ran an offline comparison using DINOv2-CLS on a fixed set of 1,215 held-out observations. The VisionNN and StateNN matches were, on average, 7.06 mm and 2.99 mm from the current end-effector position, respectively. Mean Drift was 7.02 mm. We retained Drift as the primary retrieval metric and added direct error to show how close the visual match was to the current position. The similar means supported agreement between the measures on these observations. Using the same recorded queries also removed differences caused by the policies visiting different states.

W2 / Q2. Including OpenVLA and comparing model rankings. We originally excluded OpenVLA-OFT after observing that its action expert, which uses an MLP action head, fine-tuned faster on our small demonstration set than the policies with diffusion or flow-matching action experts. We recomputed the correlations from the original results with all four models included. The pooled NN–VLA correlations were 0.570 for grasping and 0.396 for placement. We then examined whether performance rose and fell together across test settings, such as changes in lighting or camera pose. This relationship was stronger for OpenVLA and GR00T than for π0.5 and X-VLA. We also ranked the models separately in each test setting. Those rankings showed weak agreement between NN and the full policies, with Kendall’s τ of 0.12 for grasping and 0.08 for placement. We clarified this distinction in the revised analysis. See Common Response C3 and Appendix D.7 for the full comparison. FakeVLA thus indicates which conditions are hard for a representation.

Q3. From actions to robot commands. We added the action format and execution details. Each action contained an end-effector position change in metres in the robot base frame, a relative rotation in radians, and the next gripper state. NN retrieved four recorded actions. The MLP predicted four actions and converted them back to physical units using the demonstration normalization statistics. Both probes used front and wrist RGB images and refreshed the action queue at every control step. We divided the pose changes by 0.1 s to obtain twist commands for the 100 Hz Jacobian controller and mapped the gripper command to a 0–4 cm opening. The full VLAs retained their own inference code with chunk size four. See Appendix B.1 for the action and execution settings. NN and MLP shared the same action format and controller, which controlled the execution settings in that comparison.

Q4. Changing the position of the object or placement target. The original experiments varied the robot’s starting pose with fixed object locations. In the new simulation experiment, we distributed the apple and bowl positions across nine layouts and reused each of ten robot starts across all layouts. On the three layouts absent from the demonstration memory, OpenVLA achieved 29% placement success and DINO-SigLIP achieved 20%. We also fixed the robot’s start pose and moved the apple and bowl around the nine layouts. Placement success ranged from 10% to 52% near demonstrated layouts. On held-out layouts, eight encoders achieved 0% and SigLIP achieved 7%. We revised the spatial claim to reflect these results. See Common Response C1 and Appendices D.1 and D.2 for the setup and results across all nine encoders. The experiments showed successful retrieval when objects positions varied, while exposing limited generalization to new combinations.

Q5. Cross-domain action execution. NN retrieved raw recorded actions from the source-domain demonstrations and passed them directly to the same controller, without coordinate or calibration conversion. We recreated the physical scene in simulation and aligned the robot configuration and object poses to facilitate comparisons at similar physical states (Appendix D.10). The reported cross-domain success rates reflect the combined effects of visual retrieval and action execution across domains.

AuthorsOpenReview ↗

Reply To Reviewer F3t9

Dear Reviewer F3t9,

Thank you again for your constructive review. As the discussion period comes to a close, we wanted to briefly follow up on our rebuttal. We would be very grateful to hear whether our rebuttal resolves your concerns. We are happy to answer any further questions. Thank you for your time.

Thank you for the detailed response and additional experiments. Including OpenVLA and separating pooled correlations from within-condition rankings addresses my concern about the original correlation analysis. The direct-error comparison, clearer action interface, and new object-layout and LIBERO-Spatial experiments also strengthen the paper.

Some limitations remain. The common-query analysis covers DINOv2 only, so it does not fully resolve the concern about rollout-dependent comparisons across encoders. I also share the concern raised in the discussion about encoder selection. The current evidence does not establish that a visual encoder that performs better with FakeVLA will also perform better in a full VLA.

Nevertheless, I find the distinction between local action retrieval and learned action decodability informative. The revised paper makes the diagnostic scope clearer and provides useful evidence about robustness and demonstration coverage. These contributions remain valuable despite the limitations above. I will therefore maintain my rating of 3 (Weak acceptance).

Reviewer_NKhTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper propose a probing method to estimate the performance of vision encoders in VLAs. Specifically, they use the nearest-neighbor retrieval from the vision feature to get the corresponding action and evaluate the success rate on that. Compared with traditional training-evaluation pipeline, the evaluation is much faster. In their experiments, extensive ablations are provided. However, the scale of the benchmark is a big concern for the conclusion's effectiveness.

Critical strengths
  • The question studied is interesting overall. With complex models like VLA/Diffusion, isolating the advantage from vision encoder becomes a hard problem and it will be helpful to better understand the problem. This paper provides a reasonable aspect.
  • The encoder comparison is broad and covers SSL, VLM and VLA encoders. Also there are some interesting emperical findings like NN is actually better than MLP for retrieval. More study or emperical analysis on this will be helpful for better understanding.
  • It is good to see real results for the study.
Critical weaknesses
  • The biggest concern is the significant narrow tasks evaluated in this paper. According to the experiments, only apple/banana pick-and-place tasks are used and 100 episodes of each are provided. In addition, the pick-and-place location is fixed, and only the robot initial state is randomized. In this case, all the observations are very similar and what the method actually probes is just the global image embedding's correlation with the robot arm states,, which is usually provided in policies. To support the claims, much more tasks and much larger domain randomization are needed. Existing benchmarks like RoboCasa or LIBERO may be options.
  • The rollout number is also too small. Specifically, 25 rollouts are used in Exp. 1 and Exp. 2 and 10 rollouts in Exp. 3. With such a smaller number, the variance should be very high. Thus, error bars is necessary to understand the significance. Also, the OpenVLA correlation is excluded form the main results.
  • Experiment details are missing, like camera, resolution, etc.
Questions for authors

Currenlty the texture randomness includes the robot, which may be too difficult since the robot state is the most important factor. What if we only change the other parts?

3 comments
AuthorsOpenReview ↗

Reply To Reviewer NKhT

We thank the reviewer for recognizing the value of studying visual representations separately from full VLA policies, the breadth of our encoder comparison, and the real-robot evaluation. In response to the concerns about benchmark scope, we added experiments with moving objects, results on LIBERO-Spatial, a demonstration-memory ablation, and the requested texture comparison.

W1. Fixed locations and the scope of the benchmark. During the rebuttal, we ran a new simulation experiment with the apple and bowl placed across nine layouts. Six layouts appeared in the demonstration memory. The other three combined familiar apple and bowl positions in arrangements absent from memory. We used the same robot starts across layouts to compare changes in object position while holding the starting arm pose fixed.With 240 demonstrations, π0.5’s placement success fell from 38% on demonstrated layouts to 12% on new layouts. OpenVLA achieved similar placement rates on both sets of layouts, at 28% and 29%, respectively. The robot successfully retrieved actions for new object arrangements, but generalization varied across encoders. We observe that a better performance on the demonstrated layouts do not necessarily translate to better generalization to unseen object arrangements. Common response C1 and Appendix D.1 provide the setup and full results, and C4 reports how the number of demonstrations affects performance.

Evaluation on LIBERO-Spatial. In the revision, we added results from all ten tasks in the LIBERO-Spatial simulation benchmark. FakeVLA used frozen visual features to retrieve demonstration actions, with 200 (20 rollouts for each of the 10 tasks) rollouts per encoder and condition. Under the standard setup, OpenVLA-OFT and X-VLA achieved 69.5% and 65.5% task success, respectively. GR00T achieved 64.5% and π0.5 achieved 53.5%. **These results show that action retrieval could complete tasks on an established benchmark. ** Appendix D.9 reports the evaluation setup and full results, including changes to the robot’s starting pose, lighting, camera pose and textures.

W2. Rollout counts, uncertainty and the OpenVLA comparison. In the revision, we added trial counts and uncertainty intervals to the submitted simulation results (Tables 3 and 4) and summarized the evaluation protocols in Table 6. The original evaluation used 25 rollouts per encoder and condition. Dataset Init. and Unseen Init. each used 25 distinct starts; the remaining conditions used five starts with five repetitions each. During the rebuttal, we reran Exp. 1 on Apple in simulation with 100 rollouts per encoder and condition. Table 15 reports these new results with 95% confidence intervals obtained by resampling groups of rollouts from the same start.

We also recomputed the correlations from the original full-policy results with OpenVLA included. Its original exclusion followed our observation that its MLP-based action head fine-tuned faster on our small dataset than the diffusion or flow-matching action experts. With all four models included, the pooled NN–VLA correlations were 0.570 for grasping and 0.396 for placement. Common response C3 gives the full comparison. The negative association between Drift and grasp success remained strong in the 100-rollout rerun, with correlation −0.93 similar to −0.88 originally.

W3. Experimental details. We expanded the description of the Franka Panda setup, cameras and image processing. Both simulation and the real robot used front and wrist RGB views. The simulated and real cameras were Intel RealSense D435s. We recorded images at 1280 × 720 and resized demonstration frames to 224 × 224. VisionNN concatenated the two camera features and applied L2 normalization. See Appendix B.1 for the camera setup, encoder preprocessing, and examples of simulation and real-robot observations and visual perturbations. 

Q. Changing textures while preserving the robot’s appearance. We ran the requested comparison, in simulation, with 100 rollouts per condition. We applied the same texture changes to the scene surfaces while either changing or preserving the robot’s materials. The bowl kept its original appearance throughout. Preserving the robot’s appearance increased GR00T’s grasp success from 9% to 15% and placement success from 4% to 10%, while Drift stayed roughly the same. For X-VLA, grasp success rose from 52% to 65%, placement success from 19% to 21%, and Drift fell from 142 to 76 mm. Grasp success increased for seven of the nine encoders. See Appendix D.3 for the full results, uncertainty intervals, and surfaces changed. These results show that changes to the robot’s appearance contributed to the difficulty of the texture condition, with different effects across encoders.

AuthorsOpenReview ↗

Reply To Reviewer NKhT

Dear Reviewer NKhT,

Thank you again for your constructive review. As the discussion period comes to a close, we wanted to briefly follow up on our rebuttal. We would be very grateful to hear whether our rebuttal resolves your concerns. We are happy to answer any further questions. Thank you for your time.

I appreaciate the detailed response and additional experiments made during the rebuttal, which resolve most of my concerns. I will increase my accessment to weak accept.

Comments on the paper

4 comments
AuthorsOpenReview ↗

List of Changes

We thank all reviewers for their detailed evaluation of the manuscript. We have revised the paper to address the comments, questions and suggestions. All additions and changes, including new tables, table cells and captions, are marked in blue in the revised PDF. The main improvements are a broader experimental evaluation (moving objects, a fixed robot start, memory size, LIBERO-Spatial, more rollouts) and a clearer definition of what the probe and its metrics measure. We believe that the revised paper, together with our replies, addresses the main points raised in the reviews.

Highlights

The experimental evaluation has been extended in three main ways, following the reviewers' suggestions:

  1. Spatial diversity and memory size. Nine frozen encoders are now evaluated on nine object layouts (three apple × three bowl positions), three of which are absent from memory, with memories of 60, 120 and 240 demonstrations (Appendix D.1). All nine encoders improve in both Grasp and Place from 60 to 240 demonstrations, and unseen layouts remain hard for most encoders. A complementary test with a fixed robot start and moved objects (Appendix D.2) shows 10–52% Place on demonstrated layouts and close to 0% on new layouts.
  2. What Drift measures. Drift is now defined explicitly as a retrieval-consistency measure. We add the direct positional error between the current robot state and the visually retrieved state as the secondary proximity measure. On 1,215 common held-out queries, mean Drift and direct error are similar (7.02 vs. 7.06 mm; Appendix D.4). Richer state references (Appendix D.5) and an offline memory-size study (Appendix D.6) complement this analysis.
  3. More tasks and more rollouts. VisionNN is now evaluated on LIBERO-Spatial (10 tasks, 200 episodes per cell; Appendix D.9). Exp. 1 is repeated with 100 rollouts per cell (20 starts × 5 runs) and start-cluster bootstrap confidence intervals (Appendix D.8); it gives the same picture as the 25-rollout results in the main text. A protocol table (Table 6) now lists the memory, evaluation starts, chunk length, re-query interval and controller of every experiment.
AuthorsOpenReview ↗

Common Response

We thank the chairs and reviewers for their feedback. The revision adds simulation results on LIBERO-Spatial, held-out object layouts, and demonstration memory size and coverage. We also compare Drift with direct positional error and reanalyze the original full-policy results with OpenVLA included.

C1. Evaluation across different object positions. We evaluated nine layouts in simulation (three apple positions × three bowl positions). Six layouts appeared in the demonstration memory. The other three paired familiar positions in combinations absent from memory. We reused the same ten new robot starts across all layouts.

Results. With 240 demonstrations, π0.5 and DINO-SigLIP remained among the strongest encoders for placement. Eight of nine encoders placed the apple less often on held-out layouts (e.g., π0.5: 38% to 12%). OpenVLA achieved 28% on familiar layouts and 29% on held-out layouts, ranking first on the latter. In a separate test with one fixed robot start and jittered object positions, placement reached 10–52% near layouts represented in memory. On held-out layouts, eight encoders achieved 0% and SigLIP achieved 7%. See Appendices D.1 and D.2.

C2. Clarifying what Drift measures. Drift is the distance between the end-effector positions of the visual and state-based matches. Direct error is the distance from the current end-effector position to the visual match. On 1,215 recorded frames from 10 held-out Apple demonstrations, using DINOv2-CLS, mean Drift was 7.02 mm, mean direct error was 7.06 mm, and the state-based match was 2.99 mm from the current position on average (Appendix D.4).

In the Apple simulation rerun with 100 rollouts per encoder and condition, Drift correlated negatively with grasp success within all six conditions (r = −0.60 to −0.95, and r = −0.86 after removing condition means). Direct positional error also correlated negatively with grasp success across the 54 encoder–condition pairs (r = −0.91, Appendix D.8). In a separate rollout analysis, the association was weak over the first ≈0.4 s, suggesting that later policy behaviour contributes to the rollout-level relationship (Appendix D.5).

C3. Comparing FakeVLA with full VLA policies, including OpenVLA. The submitted paper reported full-policy results for all four models (Table 3 in the submission, Table 4 in the revision). Including OpenVLA in the analysis, the pooled nearest-neighbour correlations with full-policy success changed from 0.641 to 0.570 for grasping and from 0.627 to 0.396 for placement. The corresponding four-model MLP correlations were 0.627 and 0.572.

Across conditions, nearest-neighbour and full-policy success tracked each other more closely for OpenVLA and GR00T than for π0.5 and X-VLA. When comparing model rankings within the same condition, mean Kendall’s τ was 0.12 for grasping and 0.08 for placement. With the 100-rollout nearest-neighbour rerun, these rose to 0.32 and 0.47, respectively, indicating partial ranking agreement. See Appendices D.7 and D.8.

C4. Effects of demonstration memory size and coverage. In C1, we ran closed-loop trials with memories of 60, 120, and 240 demonstrations, testing three sampled memories at each smaller size. All encoders used the same demonstrations within each configuration.

Results. From 60 to 240 demonstrations, grasp and placement success increased for all nine encoders, and Drift decreased for eight. Placement rose from 15% to 29% for π0.5 and from 9% to 28% for DINO-SigLIP. In an offline test with fixed queries, increasing memory from 12 to 60 demonstrations reduced mean direct positional error on held-out layouts from 20.2 to 12.0 mm. At a fixed size of 30 demonstrations, covering six layouts instead of three increased correct-layout retrieval from 50% to 78% on seen-layout queries. See Appendices D.1 and D.6.

AuthorsOpenReview ↗

Detailed List of Changes

  • Moving objects (Sec. 4.4, Appendix D.1). We evaluate nine encoders (81 policies, 7,255 trials) on nine layouts with 240-, 120- and 60-demonstration memories, using the same memory frames across encoders. New tables report Grasp and Place on seen and unseen layouts, Drift and direct error, with start-cluster bootstrap intervals. The policies use 4-action chunks with a new query at every step. Matched comparisons use the 82 scenes completed by all policies. Rankings on seen layouts do not carry over to unseen layouts (e.g., π0.5 38% vs. 12% Place; OpenVLA 28% vs. 29%). Unseen layouts require retrieving useful actions from demonstrations recorded with different object arrangements, and most encoders struggle with this transfer. The spatial-understanding claim in the Abstract and Introduction is now limited to the evaluated pick-and-place settings.
  • Fixed robot start with moved objects (Sec. 4.4, Appendix D.2). The robot always starts from the same pose while the apple and bowl are moved around the nine layouts (192-demonstration memory, 90 test scenes × 9 encoders). Place reaches 10–52% on demonstrated layouts but is zero on new layouts for eight of nine encoders (SigLIP 7%). Zero-success cells report counts with exact intervals. With the arm start fixed, retrieval follows the object layout but does not generalize to unseen layouts.
  • Texture Randomization without changing the robot texture (Sec. 4.4, Appendix D.3). We separate texture changes on the scene from texture changes on the robot, with 100 rollouts per cell for nine encoders. Leaving the robot's appearance unchanged raises Grasp for six of nine encoders and roughly halves Drift for four (e.g., DINOv2 272 → 131 mm, X-VLA 142 → 76 mm); for the other five, Drift changes by at most 30 mm.
  • Drift and direct error (Sec. 4.2, Appendices D.4–D.6). Drift is defined as the end-effector distance (mm) between the VisionNN and StateNN retrievals, with failed trials included. The Drift is the primary measure and direct error E(t) reported as complementary. Both measures use the query state; their similar means on the recorded queries show agreement in this setting. Appendix D.5 compares richer state references (R0–R5). Appendix D.6 shows that direct error on unseen layouts falls from 20.2 to 14.7 to 12.0 mm as the memory grows from 12 to 30 to 60 demonstrations.
  • StateNN and VisionNN definitions (Sec. 3). The StateNN distance is stated in full (position + rotation + gripper, k = 1, gripper in {−1, +1}). StateNN is described as an offline reference, not a policy. VisionNN concatenates front and wrist features and applies L2 normalization.
  • Probe–VLA correlation with all four models (Sec. 4.5, Appendix D.7). OpenVLA is now included. Pooled NN–VLA correlations become 0.57 (Grasp) and 0.40 (Place), and MLP–VLA correlations 0.63 and 0.57. The reason for the original OpenVLA exclusion is stated in Appendix D.7.
  • Statistical reporting and 100-rollout rerun (Sec. 4.2, Appendices B and D.8). Trial counts are reported for every cell, and each table states its interval method. The submitted 25-rollout tables give 95% Wilson intervals for Grasp and Place; we note that these treat repeated runs from the same start as independent and are likely too narrow. Drift intervals in Table 3 are approximations derived from the rerun's per-trial variability. Exp. 1 is repeated with 100 rollouts per cell for all nine encoders (Dataset Init.: 100 starts × 1; Unseen Init.: 25 unseen poses × 4; other conditions: 20 starts × 5), reporting Grasp, Place, Drift and direct error with start-cluster bootstrap intervals (whole start groups resampled, 10,000 resamples). The rerun gives the same picture as the 25-rollout results: mean Grasp/Place at Dataset Init. 75.9/56.7% and under Texture 16.7/8.6%, texture remains the most disruptive condition, and the Drift–Grasp correlation is −0.93 (vs. −0.88). Banana cells keep the submitted 25-rollout values; a 100-rollout Banana rerun is planned.
  • Evaluation protocol (Appendix B, Table 6). Dataset Init. uses 25 dataset starts × 1 run; Unseen Init. uses 25 fixed start positions that are not in the dataset and lie at least 6–8 cm from the dataset start poses (× 1 run); the other conditions use 5 starts × 5 runs. The full VLAs use the same seen and unseen start schedule as VisionNN. The closed-loop Apple experiments retrieve from a 100-demonstration memory; the offline common-query analysis uses all 100 demonstrations with 10 held out.
Area_Chair_wfHUOpenReview ↗

Rebuttal Discussion

Dear Reviewers,

The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.

Thank you very much for your time and valuable contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.