acceptodds
Under review as a conference paper at ICLR 2027

Not All Concepts Are Causal: Sparse Autoencoders Reveal a Motor Layer in VLAs

Abstract

Vision-language-action (VLA) policies hide which internal features merely describe a task and which produce the action. We introduce SAE-VLA, which turns a frozen VLA's activations into a concept lookup table for reading out and steering its actions. A sparse autoencoder provides the rows, a vision-language model names them, and a blinded verifier checks every name on held-out episodes. A signed intervention records whether a row moves the action its name implies, at a dose certified by a closed-loop gate. Across five policies on two backbones, where the table is built decides whether it works. In , language features receive plausible motor names that verify half as often and barely act. The action expert instead holds a motor layer of grasp, release, hand-height, and wrist-tilt features. On the breakfast policy, verified names identify the moving arm on of held-out single-arm frames and contain the current subtask on , by lookup alone. A name indicates where to steer but not whether, so the table's causal column defines what is steerable. On the LIBERO policy, a gated verified row makes the arm descend earlier with success unchanged. Gated handles replicate with a second dictionary and on the OpenVLA-OFT LIBERO policy. Steering by language lets the frozen breakfast policy recover bread placements beyond its demonstrations, raising task success from to without new data or retraining. Not every named concept is causal; the table records which ones are.

Primary area:
interpretability and explainable AI
Keywords:
robot manipulation, vision-language-action models, sparse autoencoders, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RbWAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SAE-VLA, a framework for interpreting and steering frozen VLA policies using sparse autoencoder features. The authors name features from multimodal evidence, verify their descriptions, and test how feature interventions affect motor outputs. Experiments across five policies on two backbones identify useful action-proximal features and show improvements on selected real-world subtasks through language-assisted steering.

Critical strengths
  1. I think the paper studies an interesting problem: whether we can understand the hidden activations of an existing VLA and use those features to guide its behavior through language. Distinguishing interpretable descriptions from actual control effects is a useful perspective.

  2. The experimental analysis is extensive and includes comparisons with raw residual dimensions, PCA components, random directions, and reconstruction-only controls. These comparisons help assess what the sparse features contribute beyond simpler representations.

Critical weaknesses
  1. The method aims to turn verified concepts into usable control handles, but the main robot demonstration does not clearly follow the stated qualification procedure. Section 3.4 requires intervention specificity Rspec > 1, whereas five of the seven selected feature rows in Figure 6a have reported values below this threshold. The authors should clarify why these features were selected and how the actual combined intervention was validated. Without this clarification, the task-success gains do not fully establish the reliability of the proposed qualification framework.

  2. It is unclear whether the final readout evaluation is independent of the episodes used for feature selection and name verification. In addition, recognizing the requested subtask may reflect information already provided by the instruction rather than the policy’s actual behavior. The paper should clarify the evaluation split and provide a clearly independent assessment of the completed, frozen concept table, with readout accuracy measured against the policy’s generated or executed actions.

  3. Although the feature-level study covers multiple policies and two backbones, the task-level language-guided correction results focus on two bread-handling subtasks from one policy. These results are promising, but provide limited evidence of how reliably the complete correction procedure generalizes to other task types and correction requests. Evaluating the full language-guided correction pipeline across additional manipulation scenarios and policies would strengthen the practical contribution.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Response (1/2): W1

Thank you for the careful review. You are right on the first two points. Below we give the exact protocol, new analyses, and the changes in the revised PDF.

W1. Rows with Rspec≤1R_\mathrm{spec}\le 1 in Fig. 6a, and validation of the combined intervention. Correct: the place-bread correction does not follow the qualification of Sec. 3.4. In steering by language, Eq. (4) applies every row listed under a ticked name; the screen result is shown to the planner and the operator but is not used as a filter, and we should have said so. For place bread, the seven rows under the three ticked names (12126, 139, 1122, 9375, 10715, 15852, 13190; RspecR_\mathrm{spec} 0.80, 0.21, 1.88, 1.19, 0.98, 0.85, 0.85) have measured signs that agree with their names, but the five verified ones fail the screen and the two that pass it (1122, 9375) are unverified. The names and λ\lambda were fixed after [[N]] pilot trials on the non-demonstrated placements.

To test whether this matters, we ran an open-loop attribution control (new):

  • We applied each robot configuration exactly as on the robot (32σj32\sigma_j per row, all rows at once) on 518 held-out frames of the two bread subtasks and compared it with 32 random combinations of equally many table rows, rescaled to the same residual-space norm.
  • Load bread (three verified, screen-passing rows: 7724, 10565, 12396) moves the right gripper more than all 32 random combinations and is more right-arm-specific than 29 of 32.
  • Place bread is less right-arm-specific than 27 of 32 random combinations and moves the left arm about twice as much as the right arm.

Within the robot experiment, rows that pass the table's tests give a targeted combined edit and rows that fail them do not. We therefore attribute only the load-bread gain to the table (13/60 to 51/60 on its three non-demonstrated leans, Table 22) and report place bread as an operator-tuned demonstration that the table does not explain.

No combination was gated in closed loop. Both requests use λ=4\lambda=4, i.e. 32σj32\sigma_j per row, on top of the table dose 8σj8\sigma_j transferred from the LIBERO gate. The only in-distribution check of the full combinations is post hoc: on the demonstrated placements the same configurations give 95% (place; frozen 80%) and 80% (load; frozen 80%) over 20 paired trials each. The revision states all of this in Sec. 5.5 and Fig. 6, and adds Appendix L: Table 24 lists every applied row with its σj\sigma_j, verification and screen result, and Table 25 gives the attribution control.

AuthorsOpenReview ↗

Response (2/2): W2, W3

W2. Independence of the readout, and the instruction. Overlap. Yes: the 4,771 frames come from the 63 held-out episodes that also supplied the verifier's windows. We cross-fitted by episode (two folds alternating episode ids within each subtask; no new model calls): a name counts as verified using only verifier items from one fold, and the readout is scored on the other. This gives 97.6% (arm), 96.0% (subtask) and 98.6% (gripper events; names shuffled 83.3%), against 98.6 / 97.0 / 100% in Table 2. Over 20 random splits the means are 96.5 / 95.9 / 98.9%.

Instruction. You are right that the instruction carries much of this. Each subtask has its own instruction and usual arm, so an instruction-only readout names the moving arm on 96.4% of single-arm frames; the 86.5% majority baseline of Table 2 understates this. The table adds information beyond the instruction on the 131 single-arm frames (14 episodes) where the moving arm is not the subtask's usual one: the cross-fit readout is right on 101 (77%), the instruction on none. Within a subtask, where the instruction is constant, frames within 0.5 s of a grasp or release are ranked above the rest with AUROC 0.77 (cross-fit; 0.50 for any instruction-only readout), but an elapsed-time prior fitted on training episodes reaches 0.76. The event readout therefore identifies which gripper acts (names shuffled: 83.3%) rather than adding timing precision. The subtask readout cannot beat the instruction, and the revision no longer presents it as evidence of behavior tracking.

Executed actions. Table 2 scores the policy's activations on held-out demonstrations against the demonstrated motion, not against its own rollouts. The robot trials log the SAE codes at every call, but we could not score them against the executed actions in time, so the revision removes the unquantified sentence on hardware readout at the end of Sec. 5.2.

W3. Generality of language-guided correction. Agreed. The correction covers two subtasks of one policy, and only one of them uses qualified rows. We could not add tasks or policies during the discussion period, and the revision states this scope in the Scope paragraph and narrows the abstract accordingly. The cross-policy evidence in the paper is for what the correction builds on: the property chain on five policies and two backbones, and gated steering by name on the π0.5 LIBERO policy, a second dictionary and OpenVLA-OFT. For the main LIBERO handle we also ran 400 new paired episodes on initial states never used before: success changes by −1.0-1.0 pp [−4.0,+2.0][-4.0, +2.0], and the descent still starts 1.181.18 steps earlier [−1.39,−0.89][-1.39, -0.89].

All new numbers are in Appendix L of the revised PDF (readout: Table 27; LIBERO blocks: Table 26).

Thank you for the clarifications and additional analyses, which partially address Weaknesses 1 and 2 and appropriately narrow the claims. However, Weakness 3 remains unresolved, as the complete language-guided correction pipeline has not been evaluated beyond the original two subtasks of one policy. I therefore maintain my current score for now.

Reviewer_FXYnOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SAE-VLA, a framework for interpreting and steering frozen vision-language-action policies through a concept lookup table. Sparse autoencoders extract features, a vision-language model assigns names, and held-out verification, open-loop interventions, and closed-loop dose selection assess their interpretability and steerability. The central finding is that reliable semantic names do not necessarily identify effective steering directions. Across five policies and two architectures, the authors compare language-side and action-side representations, identifying concentrations of motor-related features and demonstrating action readout, motor-event modulation, and human-assisted language steering. In real-robot experiments covering four placements outside the demonstrations, the reported aggregate success rate on two breakfast subtasks increases from 17.5% to 87.5% without retraining the policy.

Critical strengths
  1. Clear distinction between interpretability and steerability. The paper provides systematic evidence that verified feature names do not reliably predict intervention effects, offering a useful evaluation framework for VLA interpretability.

  2. Broad experimental controls. Comparisons include reconstruction controls, neurons, PCA, random directions, and multiple dictionary seeds. Real-robot experiments demonstrate practical potential, while negative results help clarify the method’s limitations.

Critical weaknesses
  1. Unclear connection between dose qualification and final steering. The stated 3-point success-loss margin is difficult to reconcile with the −8.3-point hardware results and the unexecuted hardware dose ladder. Final correction also uses stronger feature combinations containing features below the selectivity threshold, without clearly documented qualification of the complete intervention.

  2. Insufficient attribution of task-success gains. The final experiments mainly compare the complete system against the frozen policy. Matched reconstruction-only, simple steering, and planner-free controls are needed to distinguish the benefits of the validated concept table from those of generic intervention and operator selection.

Questions for authors
  1. Does the closed-loop gate use the point estimate of the success-rate difference or a confidence bound? Are dose selection and final evaluation performed on completely separate rollouts and initial states?

  2. Why are features with R_spec≤1 included in the correction shown in Figure 6a? Was the complete combination at λ=4 independently calibrated in closed loop? Please report the selected features, signs, individual and combined doses, and the number of calibration trials conducted before freezing the configuration.

  3. Do the 4,771 readout-evaluation frames in Table 2 overlap with the trajectories used to select the 188 verified features? If so, can the readout results be reported on trajectories that played no role in name verification or feature selection?

3 comments
AuthorsOpenReview ↗

Response (1/2): Q1, W1, Q2

Thank you for the precise questions; they identify real gaps. Below are the exact protocol and new analyses on existing data and in simulation. Q1. Point estimate or confidence bound? Separate rollouts and initial states?

  • Statistic. Point estimate: the gate takes the largest grid dose whose paired Δ\DeltaSR vs SAE-in-loop is ≥−3\ge -3 pp on 200 LIBERO episodes (initial states 0–4): +4σj+4\sigma_j: 0.0, +8σj+8\sigma_j: +0.5+0.5, +16σj+16\sigma_j: −4.5-4.5 pp, so αcl=8σj\alpha_\mathrm{cl}=8\sigma_j. A lower-confidence-bound rule would pass no positive dose at this nn (+4σj+4\sigma_j: [−4.5,+4.5][-4.5, +4.5]). The revision calls the gate a point-estimate tolerance check and drops "certified".
  • Separation in the paper. Not complete: the reported n=400n=400 adds 200 episodes (initial states 5–9) to the 200 used for selection. Those 200 were run after the dose was frozen in a time-stamped rule file (Sep 21, 10:19 UTC; runs from 11:06 UTC) and give −2.5-2.5 pp [−7.0,+2.0][-7.0, +2.0], with the descent 1.401.40 steps earlier [−1.65,−1.18][-1.65, -1.18].
  • New fresh confirmation. During the discussion we ran 400 further paired episodes on initial states 10–19, never used for selection or reporting: Δ\DeltaSR −1.0-1.0 pp [−4.0,+2.0][-4.0, +2.0] (McNemar p=0.62p=0.62), descent 1.181.18 steps earlier [−1.39,−0.89][-1.39, -0.89], and 74% of episodes with the predicted sign of the height command. Over all 600 post-freeze episodes, Δ\DeltaSR is −1.5-1.5 pp [−4.0,+1.0][-4.0, +1.0]. The point estimate holds on fresh rollouts, but even n=600n=600 cannot exclude a loss slightly beyond 3 pp. The gate selects on success only, so the event shift is not a selection criterion. The two replications (Table 20) report the same cells they were gated on. W1. The 3 pp margin, the pp hardware results, and the dose ladder. No gate and no dose ladder were run on the robot. The hardware rows labeled +αcl+\alpha_\mathrm{cl} in Table 3 use 8σj8\sigma_j transferred from the LIBERO gate. With 12 paired trials they test whether target and event reverse with the sign of the dose; they do for all three screened rows (67–83% of trials consistent; random and language-locus controls 50%). One lost success is already 8.3 pp, so these trials cannot test a 3 pp margin. The revision relabels these rows "+8σj+8\sigma_j, transferred, not gated on the robot". Q2. Features, signs, doses and calibration trials of the language-steering runs.
  • Place bread: 12126, 139, 1122 (grasp-bread-right), 9375 (hold-toasted-bread-right), 10715, 15852, 13190 (release-bread-right). RspecR_\mathrm{spec} on the named channel: 0.80, 0.21, 1.88, 1.19, 0.98, 0.85, 0.85. Measured signs agree with the names; 1122 and 9375 pass the screen but are unverified, and the other five are verified but fail it.
  • Load bread: 7724 (approach-bread-rack-right), 10565 (grasp-bread-right), 12396 (hold-bread-right), all verified and passing the screen (RspecR_\mathrm{spec} 2.15, 2.60, 2.24). For 10565 the measured gripper direction is opposite to its name.
  • Doses: sign +1+1 for every row; table dose 8σj8\sigma_j times λ=4\lambda=4, i.e. 32σj32\sigma_j per row, all rows at once (σj\sigma_j values in the revised appendix). Neither the rows at 32σj32\sigma_j nor the combinations were calibrated in closed loop.
  • Calibration trials: the requests, ticked names and λ\lambda were fixed after [[N]] pilot trials on the non-demonstrated placements; the 20 paired trials per placement in Table 22 were run afterwards and do not include them.
  • Why rows with : Eq. (4) applies every row under a ticked name; this mode shows the screen result but does not enforce it. We should have stated that this departs from Sec. 3.4, and the revision does.
AuthorsOpenReview ↗

Response (2/2): W2, Q3

W2. Attribution of the task-success gains. We agree that Table 22 compares only with the frozen policy. We could not run reconstruction-only, simple-steering or planner-free controls on the robot during the discussion period, but we ran the attribution test that is possible offline.

  • Generic-intervention control (new, open loop). On 518 held-out frames of the two bread subtasks (the table's screen protocol), each robot combination was applied exactly as on the robot and compared with 32 random combinations of equally many table rows, rescaled to the same residual-space norm and drawn from all active rows or from right-arm-named rows.
    • Load bread (all rows qualified): right-gripper change 0.28, above all 32 random combinations (mean 0.08) and 30 of 32 right-arm-named ones; right-arm/left-arm ratio 2.5, above 29 of 32 (mean 1.4).
    • Place bread (no row both verified and screen-passing): ratio 0.52, below 27 of 32 (mean 1.3); its left-arm change is about twice its right-arm change.
  • Reading. The qualified load-bread rows are distinguishable from a generic edit of the same size; the place-bread rows are not. We therefore count only load bread (13/60 to 51/60 on three non-demonstrated leans) as evidence for steering with qualified rows, and report place bread as operator-tuned and not explained by the table.
  • Other evidence. Reconstruction alone is neutral on the robot in distribution (SAE-in-loop −2.5±6.5-2.5\pm6.5 pp over 80 paired trials). At the transferred single-row dose, a random expert row and a language-locus row stay at chance (50% consistent), while the three screened rows reverse target and event with the sign of the dose.

These are open-loop or single-row results, not matched task-level controls, and the pilot trials shaped the configuration. The revision presents steering by language as an operator-in-the-loop demonstration and lists the missing robot controls in the Scope paragraph.

Q3. Do the 4,771 readout frames overlap with feature selection? Yes: all verifier windows come from the same 63 episodes, so the 188 verified names were selected partly on them. We cross-fitted with two folds that alternate episode ids within each subtask (rule fixed before any number was computed; no new model calls): a name counts as verified using only verifier items from one fold, and the readout is scored on the other.

armsubtaskgripper events (top 3)
Table 2 (in-sample)98.697.0100 (shuffle 80.4)
cross-fit97.696.098.6 (shuffle 83.3)
all names91.493.999.3 (shuffle 82.3)

Over 20 random subtask-stratified splits the cross-fit gives 96.5% (range 93.2–98.7), 95.9% and 98.9%. Caveat: each fold keeps on average about 1.5 forced-choice pairs and 2.8 top windows per name.

We also checked whether the readout only echoes the instruction. An instruction-only readout names the moving arm on 96.4% of single-arm frames; on the 131 frames where it is wrong, the cross-fit readout is right on 101 (77%). Within a subtask, the matching rows rank frames near a grasp or release event with AUROC 0.77 (cross-fit), against 0.50 for any instruction-only readout and 0.76 for an elapsed-time prior, so their timing adds little beyond typical event times. The revision no longer presents the subtask readout as evidence of behavior tracking, and the appendix reports both analyses.

Thank you for the detailed clarification and additional experiments. The independent rollouts strengthen the evidence for a reproducible motor effect, and the cross-fitted readout analysis partially addresses the evaluation-overlap concern. I also appreciate the proposed corrections to the gating claims.

However, my main concern remains the connection between the validation framework and the robot task improvements. The load-bread features pass naming verification and open-loop screening, but their deployment at 32σ per feature and in combination was not qualified in closed loop. The new matched-norm controls provide useful open-loop evidence, but do not establish task-level attribution or isolate the planner’s contribution. The instruction-only and elapsed-time baselines also substantially narrow the incremental value of the readout results.

Moreover, the author left an unresolved “[[N]]” placeholder to the actual pilot-trial count. The revision should describe the configuration-selection procedure, and also consistently distinguish screen-qualified features from closed-loop-qualified interventions, including in the abstract and main claims.

Overall, the response supports a narrower contribution, but the remaining attribution gap keeps me at weak rejection.

Reviewer_X5kcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SAE-VLA, a framework for interpreting and steering frozen vision-language-action policies using sparse autoencoders. It extracts features from internal activations, assigns them language descriptions, verifies those descriptions on held-out episodes, and tests whether feature interventions selectively change robot actions at strengths validated through closed-loop evaluation. Across five policies spanning π0.5 and OpenVLA-OFT, the authors find that representations closer to action generation contain more useful motor-control features. The resulting concept table supports action readout and language corrections without retraining the policy, including improving success on an object placement task.

Critical strengths
  • Makes SAE-based interpretability practically useful: The paper goes beyond identifying interpretable features by combining held-out verification, causal testing, and closed-loop calibration into a pipeline that supports real-robot failure correction.
  • Clear distinction between interpretability and causality: Separates whether a feature has a plausible, verifiable description from whether intervening on it produces the corresponding action, demonstrating that neither property guarantees the other.
  • Practical connection between interpretation and control: Goes beyond feature visualization to demonstrate action readout and real-robot failure recovery through operator-guided corrections, with closed-loop calibration to limit performance degradation.
Critical weaknesses
  • Missing baselines for the failure-recovery results. The feature-level experiments compare SAEs against neurons, PCA, and random directions, but the bread-recovery experiments do not establish whether the full SAE pipeline improves task success over simpler correction methods. Please include comparisons with approaches such as revised instructions, direct action offsets, or dense-direction steering, using comparable tuning effort.
  • Limited evaluation of language-guided failure recovery. The paper studies representations across several policies, but the strongest practical result, correcting failures through language, is evaluated only on two bread-related subtasks of the breakfast policy. Please extend this evaluation to additional failure types and at least one other policy or task domain. This would help establish whether the method provides a broadly useful correction mechanism beyond the demonstrated examples.
  • Unclear validation of combined interventions. The closed-loop gate evaluates individual features, but language steering combines several features and scales their strengths. Validation of individual features alone does not establish that their amplified combination preserves performance. Please clarify or address experimentally.
  • Terminology should more closely match the evidence (minor). Terms such as “safe” and “certified” suggest stronger guarantees than the empirical gate establishes. Please define them explicitly in terms of the tested conditions and distinguish changing a motor channel from causally implementing a named semantic concept. Simplifying the terminology around features, table rows, and steering handles would also improve readability.
Questions for authors

The weaknesses are framed as addressable points for the authors. I don't have any further clarifications to request here.

2 comments
AuthorsOpenReview ↗

Response to Reviewer X5kc

Thank you for the constructive review. We agree with all four points. Below are the exact protocol, new analyses on existing data and in simulation, and the revisions.

W1. Baselines for the failure-recovery results. Agreed: Table 22 compares SAE-VLA only with the frozen policy. We could not run revised instructions, direct action offsets, dense-direction steering or reconstruction-only controls on the robot during the discussion period, so the experiment does not show that the table beats simpler corrections at comparable tuning effort. We now disclose the tuning effort: the requests, ticked names and λ\lambda were fixed after [[N]] pilot trials on the non-demonstrated placements, and the 20 paired trials per placement were run afterwards. What we could test is whether the table's rows matter compared with a generic edit of the same size. On 518 held-out frames of the two bread subtasks, we applied each robot combination exactly as on the robot and compared it with 32 random combinations of equally many table rows at the same residual-space norm:

  • Load bread (three rows, all verified and passing the screen): the right-gripper change exceeds all 32 random combinations, and the right-arm/left-arm ratio (2.5) exceeds 29 of 32.
  • Place bread (seven rows, none both verified and screen-passing): the ratio (0.52) is below 27 of 32, and the left arm moves about twice as much as the right.

Qualified rows give a targeted edit and unqualified rows do not. We therefore count only the load-bread correction (13/60 to 51/60 on three non-demonstrated leans) as evidence for steering with qualified rows, and report place bread as an operator-tuned demonstration. On dense directions in general: on the π0.5 folding and flipping policies, 77–97% of the most aligned expert features steer their channel selectively in open loop (median RspecR_\mathrm{spec} 2.3–3.1), while aligned raw dimensions, PCA components and matched-energy random directions stay near 1 (Table 14). The revision lists the missing task-level controls in the Scope paragraph.

W2. Limited evaluation of language-guided recovery. Agreed: two subtasks of one policy, and only one correction uses qualified rows. We could not add failure types or policies during the discussion period; the revision states this scope in the abstract and Sec. 5.5. Beyond this policy, the closed-loop evidence is for steering by name: the π0.5 LIBERO policy, a second dictionary, and OpenVLA-OFT. We also confirmed the gated LIBERO dose on 400 new paired episodes from initial states never used before: success changes by −1.0-1.0 pp [−4.0,+2.0][-4.0, +2.0] and the descent starts 1.18 steps earlier.

W3. Validation of combined interventions. You are right, and the gap is larger than single-row gating: no closed-loop gate was run on the robot at all. Both requests apply every row under the ticked names with sign +1+1 at λ=4\lambda=4 times the table dose 8σj8\sigma_j (transferred from the LIBERO gate), i.e. 32σj32\sigma_j per row, simultaneously. The only closed-loop check of the full combinations is post hoc: on the demonstrated placements, the same configurations give 95% (place; frozen 80%) and 80% (load; frozen 80%) over 20 paired trials each. The open-loop control above adds that the load combination stays targeted while the place combination does not. The revision restricts claims about gated doses to steering by name.

W4. Terminology. Adopted.

  • Safe becomes tolerated: at the tested dose, the point estimate of paired success stays within 3 pp of SAE-in-loop on held-out in-distribution rollouts at the stated nn. This is weaker than a guarantee: over 600 LIBERO episodes run after the dose was frozen, the change is −1.5-1.5 pp [−4.0,+1.0][-4.0, +1.0].
  • Certified becomes selected by the closed-loop gate.
  • We distinguish "moves the motor channel its name implies", which the tests show, from "causally implements the named concept", which they do not.
  • Sec. 3.1 adds a glossary of feature, name, row and handle.

Thanks to the authors for acknowledging and responding to my feedback and critique. While the response is appreciated, the authors acknowledge the validity of the critique but were unable to provide new results given the short timeframe of the discussion period. This is understandable, but suggests the best course of action may be to incorporate the new results and details and resubmit. I have no doubt that the submission will be stronger after this process.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of New Analyses and Revisions

Summary of responses, new analyses, and revisions

We thank the reviewers. All three identify the separation between verified names and causal effects, and the breadth of matched controls, as strengths. The shared concern is that the robot language-steering result (Fig. 6, Table 22) did not follow the qualification procedure the paper describes. We agree. We answer with the exact protocol, new analyses on existing data and in simulation (no new robot trials), and narrower claims. Each reviewer thread is self-contained.

1. What the robot runs were. Both requests apply Eq. (4) to every row under a ticked name, with sign +1+1, at λ=4\lambda=4 times the table dose 8σj8\sigma_j (the LIBERO gate value, transferred), i.e. 32σj32\sigma_j per row. No dose ladder or closed-loop gate was run on the robot, for single rows or for combinations. The requests, ticked names and λ\lambda were fixed after [[N]] pilot trials on the non-demonstrated placements; the 20 paired trials per placement were run afterwards. Load bread applies three rows that are verified and pass the open-loop screen; place bread applies seven rows, none of which is both verified and screen-passing.

2. New analyses.

  • Attribution (open loop). On 518 held-out frames of the two bread subtasks, we compared each robot combination with 32 random combinations of equally many table rows at the same residual-space norm. The load combination moves the right gripper more than all 32 and is more right-arm-specific than 29 of 32. The place combination is less right-arm-specific than 27 of 32 and moves the left arm about twice as much as the right. Within the robot experiment, rows that pass the table's causal column give a targeted combined edit and rows that fail it do not. We therefore count only the load-bread correction (13/60 to 51/60 on three non-demonstrated leans) as evidence for steering with qualified rows.
  • Gate on fresh episodes. On 400 new paired LIBERO episodes (initial states 10–19, never used before), the gated dose changes success by −1.0-1.0 pp [−4.0,+2.0][-4.0, +2.0] and still moves the descent 1.18 steps earlier [−1.39,−0.89][-1.39, -0.89].
  • Readout. Verification and readout shared the 63 held-out episodes. Verifying names on one half and scoring the other gives 97.6% (arm), 96.0% (subtask) and 98.6% (gripper events), about one point below Table 2. An instruction-only readout already reaches 96.4% on the arm; on the 131 frames where it names the wrong arm, the cross-fitted readout is right on 77%.

3. Revisions. Steering by language is presented as an operator-in-the-loop demonstration, and claims about gated doses are restricted to steering by name. The hardware rows of Table 3 are relabeled 8σj8\sigma_j (transferred, not gated on the robot). "Safe" and "certified" are replaced by explicit definitions, with a glossary of terms. Sec. 5.2 states the readout overlap and the instruction-only baseline. The Scope paragraph lists the missing task-level robot controls (reconstruction-only on the new placements, revised instructions, direct action offsets, dense-direction steering) and the single-policy scope of the correction. The new analyses are added to the appendix.

Dear Reviewers,

Could you please take a look at the rebuttal and respond to it? The author-reviewer discussion phase is ending soon.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.