- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SAE-VLA, a framework for interpreting and steering frozen VLA policies using sparse autoencoder features. The authors name features from multimodal evidence, verify their descriptions, and test how feature interventions affect motor outputs. Experiments across five policies on two backbones identify useful action-proximal features and show improvements on selected real-world subtasks through language-assisted steering.
Critical strengths
-
I think the paper studies an interesting problem: whether we can understand the hidden activations of an existing VLA and use those features to guide its behavior through language. Distinguishing interpretable descriptions from actual control effects is a useful perspective.
-
The experimental analysis is extensive and includes comparisons with raw residual dimensions, PCA components, random directions, and reconstruction-only controls. These comparisons help assess what the sparse features contribute beyond simpler representations.
Critical weaknesses
-
The method aims to turn verified concepts into usable control handles, but the main robot demonstration does not clearly follow the stated qualification procedure. Section 3.4 requires intervention specificity Rspec > 1, whereas five of the seven selected feature rows in Figure 6a have reported values below this threshold. The authors should clarify why these features were selected and how the actual combined intervention was validated. Without this clarification, the task-success gains do not fully establish the reliability of the proposed qualification framework.
-
It is unclear whether the final readout evaluation is independent of the episodes used for feature selection and name verification. In addition, recognizing the requested subtask may reflect information already provided by the instruction rather than the policy’s actual behavior. The paper should clarify the evaluation split and provide a clearly independent assessment of the completed, frozen concept table, with readout accuracy measured against the policy’s generated or executed actions.
-
Although the feature-level study covers multiple policies and two backbones, the task-level language-guided correction results focus on two bread-handling subtasks from one policy. These results are promising, but provide limited evidence of how reliably the complete correction procedure generalizes to other task types and correction requests. Evaluating the full language-guided correction pipeline across additional manipulation scenarios and policies would strengthen the practical contribution.
Questions for authors
See weakness
3 comments
Response (2/2): W2, W3
W2. Independence of the readout, and the instruction. Overlap. Yes: the 4,771 frames come from the 63 held-out episodes that also supplied the verifier's windows. We cross-fitted by episode (two folds alternating episode ids within each subtask; no new model calls): a name counts as verified using only verifier items from one fold, and the readout is scored on the other. This gives 97.6% (arm), 96.0% (subtask) and 98.6% (gripper events; names shuffled 83.3%), against 98.6 / 97.0 / 100% in Table 2. Over 20 random splits the means are 96.5 / 95.9 / 98.9%.
Instruction. You are right that the instruction carries much of this. Each subtask has its own instruction and usual arm, so an instruction-only readout names the moving arm on 96.4% of single-arm frames; the 86.5% majority baseline of Table 2 understates this. The table adds information beyond the instruction on the 131 single-arm frames (14 episodes) where the moving arm is not the subtask's usual one: the cross-fit readout is right on 101 (77%), the instruction on none. Within a subtask, where the instruction is constant, frames within 0.5 s of a grasp or release are ranked above the rest with AUROC 0.77 (cross-fit; 0.50 for any instruction-only readout), but an elapsed-time prior fitted on training episodes reaches 0.76. The event readout therefore identifies which gripper acts (names shuffled: 83.3%) rather than adding timing precision. The subtask readout cannot beat the instruction, and the revision no longer presents it as evidence of behavior tracking.
Executed actions. Table 2 scores the policy's activations on held-out demonstrations against the demonstrated motion, not against its own rollouts. The robot trials log the SAE codes at every call, but we could not score them against the executed actions in time, so the revision removes the unquantified sentence on hardware readout at the end of Sec. 5.2.
W3. Generality of language-guided correction. Agreed. The correction covers two subtasks of one policy, and only one of them uses qualified rows. We could not add tasks or policies during the discussion period, and the revision states this scope in the Scope paragraph and narrows the abstract accordingly. The cross-policy evidence in the paper is for what the correction builds on: the property chain on five policies and two backbones, and gated steering by name on the π0.5 LIBERO policy, a second dictionary and OpenVLA-OFT. For the main LIBERO handle we also ran 400 new paired episodes on initial states never used before: success changes by pp , and the descent still starts steps earlier .
All new numbers are in Appendix L of the revised PDF (readout: Table 27; LIBERO blocks: Table 26).
Thank you for the clarifications and additional analyses, which partially address Weaknesses 1 and 2 and appropriately narrow the claims. However, Weakness 3 remains unresolved, as the complete language-guided correction pipeline has not been evaluated beyond the original two subtasks of one policy. I therefore maintain my current score for now.
Response (1/2): W1
Thank you for the careful review. You are right on the first two points. Below we give the exact protocol, new analyses, and the changes in the revised PDF.
W1. Rows with Rspec≤1 in Fig. 6a, and validation of the combined intervention. Correct: the place-bread correction does not follow the qualification of Sec. 3.4. In steering by language, Eq. (4) applies every row listed under a ticked name; the screen result is shown to the planner and the operator but is not used as a filter, and we should have said so. For place bread, the seven rows under the three ticked names (12126, 139, 1122, 9375, 10715, 15852, 13190; Rspec 0.80, 0.21, 1.88, 1.19, 0.98, 0.85, 0.85) have measured signs that agree with their names, but the five verified ones fail the screen and the two that pass it (1122, 9375) are unverified. The names and λ were fixed after [[N]] pilot trials on the non-demonstrated placements.
To test whether this matters, we ran an open-loop attribution control (new):
Within the robot experiment, rows that pass the table's tests give a targeted combined edit and rows that fail them do not. We therefore attribute only the load-bread gain to the table (13/60 to 51/60 on its three non-demonstrated leans, Table 22) and report place bread as an operator-tuned demonstration that the table does not explain.
No combination was gated in closed loop. Both requests use λ=4, i.e. 32σj per row, on top of the table dose 8σj transferred from the LIBERO gate. The only in-distribution check of the full combinations is post hoc: on the demonstrated placements the same configurations give 95% (place; frozen 80%) and 80% (load; frozen 80%) over 20 paired trials each. The revision states all of this in Sec. 5.5 and Fig. 6, and adds Appendix L: Table 24 lists every applied row with its σj, verification and screen result, and Table 25 gives the attribution control.