- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper focuses on the problem of linear probes overfitting to variables which are simply correlated with the objective. For example one of the settings identified in the paper is the robot opening a drawer in a single scene. If you train a probe to output the position of the drawer you may expect that rather than learning to accurately reflect the state of the drawer you may overfit to the robot's position. The authors create a set of counterfactual data samples across two scenes to evaluate this across several VLAs and pretrained image encoders. Unsurprisingly, given the limited nature of the dataset the probes overfit to other visual aspects of the scene. Overall the main finding of this paper is well understood in probing and machine learning at large. Additionally the paper does not provide any fixes, nor test on any higher diversity scenes which may not suffer from this issue.
Critical strengths
This work is highly reproducible, the appendix is highly detailed and the code is included in the supplementary material.
Critical weaknesses
This paper is extremely difficult to understand and parse. This includes the presentation of the method and results in which the core contributions/findings of the paper are quite unclear. I feel this paper is not suitable for publication in its current state.
The experimental settings are quite limited in nature. Only applied to 2 simulated scenes. This is a major limitation of the paper. If the authors are trying to make general claims about the performance of linear probes a much much larger diversity of tasks is needed. Furthermore it is unclear if these models are adapted at all for these scenes. Nothing is reported on their natural success rates or if they are fine-tuned and this specific manipulator type.
It is confusing that the features/embeddings used to train the ridge regression and probes for task variables are trained only on pooled image activations from each of these models. Why should we expect that task relevant information lives in the image tokens rather than across the prompt or proprioception tokens ? For most VLAs, including the ones used in this paper the action expert attends to both image, prompt and proprioception tokens.
Questions for authors
Can you explain what checkpoints you use for the VLA models and if those models are able to complete the tasks that you collect your data from at a high success rate ?
Why do the authors only probe on pooled image tokens rather than other potentially more relevant action tokens ? Could this explain part of the findings of shortcut learning ?
3 comments
Thank you for the response. I appreciate the additional features beyond pooled image tokens. However I do not feel that my primary concerns on diversity of experimental settings were fully addressed. I therefore maintain my original score.
Additional Microwave Pilot and Task Coverage
Thank you for clarifying the remaining concern. We agree that broader evaluation across tasks and environments is needed, as stated in Section 5.2. Our current contribution is the controlled evidence about readout properties and their training dependence within the evaluated settings.
To investigate your concern further, we completed an exploratory microwave-opening study using frozen π0.5-LIBERO features. All 40 controlled observations across eight initial conditions passed the specified physical and visual checks. For a 0.15-rad change in door angle, median absolute prediction changes were 0.0417 and 0.0644 rad in the two robot contexts. Changing robot context while holding the door angle fixed produced median prediction changes of 0.2797 and 0.3248 rad. Repeating the baseline produced identical predictions in all eight conditions.
Natural-development MAE was 0.0616 rad, compared with 0.6199 rad for the training-mean constant. Baseline error on the separate controlled cohort was substantially larger: 3.6070 rad versus 0.5625 rad for the constant. We report the attenuated state responses and context-dependent predictions together with this accuracy loss as preliminary evidence from an additional task.
The updated supplementary archive includes the complete protocol, all observation-level results, and reproducible numerical summaries in Additional_Task_Pilots.pdf and its accompanying files. Broader task validation remains an important direction for extending the scoped findings of this work.
Response to Reviewer wATB
We thank the reviewer for the detailed comments and for recognizing the reproducibility of our work. Your feedback helped us clarify the empirical contribution, reorganize the presentation, and add analyses connecting state predictions to the policy's action-generation features and behavior.
1. Clearer contributions and presentation
The known risk that probes learn correlated cues motivates our study. We examine how predictions respond when measured object state and robot pose are changed separately in physically and visually validated observations, and how predictor training changes these responses.
The revised manuscript separates three properties: accuracy during task execution, response to state changes at a fixed robot pose, and stability across robot poses at a fixed state. The additional comparisons show why they must be evaluated together:
These findings show how predictor training changes distinct prediction properties while VLA features remain fixed. The evaluation also includes nonlinear RBF and MLP predictors (Section 4.2; B.2).
We also clarified the practical implications of the existing training experiments. In one Faucet-SmolVLA development comparison, changing the allocation of 36 training observations reduced natural MAE from 0.5629 to 0.4693 rad (~17%) and controlled MAE from 0.3954 to 0.2648 rad (~33%). This provides a concrete example of improving both errors through training-data design (Section 4.5; C.2).
We rewrote the abstract, introduction, Figure 1, and methods around these distinctions. Sections 4.1-4.4 connect each finding to its evidence, Section 4.5 explains the observed improvements, and Section 5.2 states the scope of the conclusions.
2. Task coverage, checkpoints, and actual performance (Q1)
We now distinguish feature evaluation from autonomous policy evaluation.
We distinguish Drawer/Faucet's central findings, Mug/Stove's boundary cases, and the added Plate analysis. Mug does not reproduce the accurate-natural/weak-response combination; Stove shows weak responsiveness with high success. Table 1 identifies each task's role. Broader validation across independent tasks remains future work (Section 5.2).
3. Features beyond pooled images (Q2)
We do not assume that image tokens contain all task-relevant information. We expanded the analysis to OpenVLA's final prompt features before the first action-token logits, and π0.5 prefix and action-expert features. Action-expert features are taken immediately before the output projection at the first vector-field call. We specify the actual inputs: the evaluated π0.5 prefix contains images and task text, without direct state or history tokens (Appendices I.3-I.4).
At one Plate development initial condition, six ridge/RBF predictors across visual, prefix, and action-expert features had predicted-to-actual state-change ratios from -0.015 to 0.303, where 1 is ideal. The action-expert ridge ratio was 0.166. The same frozen policy reached the goal in all four controlled and two natural branches from that condition.
These results show that weak state-readout responses are not confined to pooled image features in this setting. We report feature-specific responses alongside actual behavior, without treating a weak readout as evidence that the policy ignores physical state (Section 4.4; I.3-I.4).