acceptodds
Under review as a conference paper at ICLR 2027

Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models

Abstract

Can better feature discovery improve control over robot behavior through internal interventions and training data? Recent work in vision and language models reveals multidimensional features, motivating block-sparse featurizers (BSFs), which represent each feature in a low-dimensional subspace. We investigate whether recovering this structure provides an interpretable interface for steering vision-language-action models and shaping the data from which they learn. We show that BSFs recover several multi-dimensional task-relevant features underlying the model's computation (e.g., perception, language, proprioceptive state, episode phase, and control variables). We causally validate these features through fine-grained steering: targeted changes in visual perception, semantic representation, and spatial localization produce predictable changes in downstream action prediction. Finally, we use feature-based data attribution to isolate where a specific behavior (failure recovery) arises in the training dataset, identifying fewer than 1.4% of training tokens whose masking is sufficient to remove the targeted behavior. Overall, our results show how matching featurizers to representation geometry can provide more effective tools for understanding and changing robot behavior.

Primary area:
applications to robotics, autonomy, planning
Keywords:
mechanistic interpretability, vision-language-action models, robot manipulation, block-sparse featurizers, sparse autoencoders, activation steering, data attribution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_veo2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether block-sparse featurizers (BSFs) are a better tool than sparse autoencoders (SAEs) for understanding and steering vision-language-action (VLA) policies. A BSF represents each feature as a small subspace (b = 3) rather than a single direction. The authors train BSFs and matched TopK SAEs (the b = 1 case of the same pipeline) on roughly 50M unpooled token activations per model and layer from MolmoAct2, π0.5 and MolmoBot. They also train a Fisher-weighted BSF variant on MolmoAct2.

The contributions are:

  1. Comparison and phenomenology. A quantitative comparison on MolmoAct2 covers reconstruction, change in action likelihood, per-feature ablation effect and single-feature decodability (Fig. 3). Qualitative examples include gripper, end-effector, proprioceptive and cross-modal object features.

  2. Causal validation by steering. Feature edits at three sites redirect the predicted reach: the L3 noun token, the L7 image tokens and the L23 action-readout token (Fig. 4). In closed loop, BSF edits complete 56/60 redirected tasks versus 45/60 for SAE edits (Table 1).

  3. Data shaping. A π0.5 "failure-recovery" feature selects DROID windows whose loss is then masked (1.38% of action tokens). This raises the time to reopen a manually closed gripper from 1.89 s to 14.25 s, with no observed loss of task success on 20 unperturbed trials per method (Table 2).

Critical strengths
  1. A controlled comparison, backed by behaviour. The SAE is the b = 1 case of the same pipeline and data, with matched coordinate budgets (21 three-dimensional blocks versus 63 scalar latents; App. D). The shared pipeline and matched total width and active-coordinate budgets make the vanilla BSF–SAE comparison a meaningful control for implementation and overall representation size. The unpooled activation corpus is a real step up from the pooled or small corpora used in prior VLA SAE work. The closed-loop result is the strongest evidence in the paper. Feature sets were frozen before evaluation, and both featurizers were calibrated by the same rule (≥75% recovery, cosine ≥ 0.90; App. C.2). BSF edits complete 56/60 tasks versus 45/60 and lead on task completion in all six pair-directions (Table 3). This provides direct behavioural evidence beyond the per-feature metrics, and it is why I take the paper seriously.

  2. Linking a discovered feature to training data with an actual retraining test. The data-shaping experiment re-fine-tunes π0.5 from base and compares against alternatives matched on selected raw-frame budget (TRAK influence, kNN, difference of means, random; App. B). Unperturbed task success shows no drop (20/20 versus 18/20 for the unmodified policy). Showing that a feature-selected ~1.4% of action tokens changes one specific behaviour is a useful demonstration that interpretability can guide robot data curation.

Critical weaknesses

1. The comparisons do not fully isolate the benefit of block geometry from block capacity. Budgets are matched at the featurizer level. But the per-feature metrics behind the claim that geometry is useful "rather than simply excess representational capacity" (p. 2) compare a 3-dimensional unit with a 1-dimensional one.

  • Fig. 3 (right): per-feature ablation removes a 3-D block versus a single latent, with the individual effects averaged over the top 7 blocks or top 21 latents. If the single-feature probe uses all three block coordinates, decodability also changes the probe's input dimension; the paper does not say which input is used. Neither right-hand panel has error bars.
  • Fig. 3 (left, layer 13): the gain is stated at fixed block sparsity k. For the vanilla featurizers, at matched active coordinates (k·b = 64), R² reads approximately 0.64 for the SAE versus 0.63 / 0.63 / 0.62 for b = 2 / 4 / 8.
  • Fig. 4: "more compact semantic and perceptual interventions" (p. 6) holds in blocks but not in nominal edited latent-coordinate count. Semantic: 6.45 vs 3.50. Visual: 61.2 vs 36.25, where the SAE also transfers better (97.3% vs 84.6%), as the paper notes.
  • Fisher weighting: a Fisher-weighted SAE is trained (Fig. 3 left) but absent from Fig. 3 (right) and Fig. 4. So the largest gains (Fisher BSF, approximately 50 mm at L15) cannot be credited specifically to the blocks rather than the loss. Eq. 7 defines the Fisher term, but the full Fisher training objective is not given.

As submitted, the advantage attributable specifically to block geometry remains insufficiently isolated, limiting the support for contribution 1. The controls are feasible, which is why I rate the concerns as fixable.

2. The application claims rest on limited or under-specified evidence.

Data shaping:

  • The 14.25 s headline is a mean over five perturbed trials, with no spread, from one reported training seed (p. 14), and no rule for trials that never reopen.
  • Random masking (windows from unflagged episodes) already moves reopening from 1.89 s to 5.00 s. Concept masking produces a larger mean delay, but the small sample and missing variability leave the reliability of its advantage unclear.
  • Table 2 has no SAE-based selector, although Fig. 7 lists six SAE latents that tile F1515. The selector uses summed block norms and active-token counts. Without a matched SAE-based selector, this experiment does not establish whether learning multidimensional blocks improves data selection over scalar features.
  • The abstract says masking is "sufficient to remove the targeted behavior", but the result, as the conclusion says, is a delay.

Steering:

  • The open-loop recovery metric (App. C.1) projects the endpoint shift onto the patching shift, so 100% does not imply "the same direction" as stated: a shift of (1, 10) against a patch of (1, 0) scores 100%. Closed-loop calibration does require cosine ≥ 0.90, but alignment is not reported for the open-loop held-out results.
  • More broadly, "held out" means a mirrored layout or the other instruction in the same scene for two of the three sites. Steering is on MolmoAct2 only, despite "across multiple models" (p. 2). The closed-loop robot and the real-vs-simulated setting are not stated.
Questions for authors

Clarifications (answers could change my rating):

  1. Closed loop (Table 1, App. C.2): Please confirm the model, and state the robot and whether the study was real or simulated. What are the unsteered and directly instructed success rates?

  2. Fisher variant: What is the full training objective? How is the Eq. 7 term combined with reconstruction and AuxK, with what coefficients, and on which data?

  3. Figs. 1 and 3–7: Is a block probed with its three coordinates or with its norm? What are the endpoint error and cosine alignment for each open-loop comparison? How are the "SAE assignments" in Figs. 1 and 5–7 computed, and what is "none"? What are the dictionary widths?

  4. Table 2: What are the per-trial reopening times and the cap rule? How were F1515 and the thresholds E ≥ 150, C ≥ 50 chosen? Beyond the five top-scoring examples per method in Fig. 10, what is the precision of the selected DROID windows on a random labelled sample?

Experiments that would raise my rating (feasible with the existing models):

  1. Dimension-matched controls, reported with variability:

    • One block versus the best training-selected group of three SAE latents, for both probing and ablation.
    • Joint ablation of the top 7 blocks versus the top 21 latents.
    • A Fisher SAE in Fig. 3 (right) and Fig. 4.
  2. Stronger Table 2 evidence:

    • An SAE-latent selector, and a simple gripper-state heuristic, at the same frame budget.
    • More perturbed trials, or a second fine-tuning seed.

Future work (does not affect my rating): Featurizer-free steering baselines at matched dimension (difference of means, DAS); steering on π0.5; a formal irreducibility test of the blocks (Engels et al., ICLR 2025).

Minor comments (do not affect my rating):

  • π0.5 (Black et al., 2025) and MolmoAct2 are not cited.
  • Related work on multi-dimensional features (Engels et al., ICLR 2025) and on output-preserving dictionary training (Braun et al., NeurIPS 2024) is missing.
  • "Strictly outperform" (p. 6) fails at layer 5 decodability, and Fisher BSF steers worse at two sites (Fig. 4).
  • Please clarify the relationship between the edited and reference-patched token sets at each site: 63 edited vs "all 196 exo-image tokens", and "all 79 text tokens".
  • Fig. 3's panels are hard to read at print size.
  • Typos: "one 1 H200", "limtations", "propriocetion", "elminate", "implicitely".
2 comments

We thank the reviewer for their thoughtful review and feedback. We appreciate you pointing out that the closed-loop and retraining experiments provide direct behavioral evidence. We are especially grateful for being transparent with what affected your score. We address the weaknesses and questions below.

W1 Geometry versus capacity

“The comparisons do not fully isolate the benefit of block geometry from block capacity. … the per-feature metrics … compare a 3-dimensional unit with a 1-dimensional one.”

We agree and we have worked hard in this rebuttal to compare on an even basis SAE and BSF features. To more adequately compare them, we form groups of 3 SAE features, randomly and using activation correlation and compare these to the BSF and fisher BSF features (block size = 3). We also trained a Fisher-weighted SAE at every layer. Fig. 3 (right) now includes these triplets and has error bars.

  • Ablation. One Vanilla BSF block moves the end effector 1.8 mm more than one SAE triplet in layers 9–19 (95% CI [1.0, 2.7], on about 16 mm), while the difference is reduced when the top seven units are ablated together (21 dimensions; +2.3 mm [−1.8, 7.1]) and in later layers. This shows that BSF blocks do shift robot behavior more when ablated, relative to dimension matched SAE baselines.
  • Fisher weighting. The Fisher objective we propose raises the ablation effect of both SAE triplets and of BSFs blocks, showing how this technique can be applied to improve the causal efficacy of different featurizers.
  • Decodability. Our original probes used all three block coordinates versus one coordinate for SAEs. However, compared to SAE triplets we match the block protocol exactly. We find that BSF features better decode robot-state variables.

“Fig. 3 (left, layer 13): the gain is stated at fixed block sparsity k. For the vanilla featurizers, at matched active coordinates (k·b = 64), R² reads approximately 0.64 for the SAE versus 0.63 / 0.63 / 0.62 for b = 2 / 4 / 8.”

This is correct. At matched active coordinates (k·b = 64) the SAE reconstructs slightly better: R² 0.636 against 0.627 / 0.627 / 0.616 for b = 2 / 4 / 8. We argue that BSFs tradeoff improved structure for pure reconstruction loss. Prior work Fel et al. (2025) has shown that high reconstruction is not necessarily tied to high quality features.

Fel et al. (2025), “Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models”, ICML 2025.

“ Fig. 4: "more compact semantic and perceptual interventions" (p. 6) holds in blocks but not in nominal edited latent-coordinate count. Semantic: 6.45 vs 3.50. Visual: 61.2 vs 36.25, where the SAE also transfers better (97.3% vs 84.6%), as the paper notes.”

The steering table in the appendix (Table 4) now reports edited coordinates alongside features (semantic 6.5 vs 3.5; visual 61 vs 36, where the SAE also transfers better) and our matched SAE triplet dictionaries for more fair comparisons. When we group the SAE's latents into triplets, so that each unit has three coordinates like a block, most of the feature-count difference disappears.

“Fisher weighting: a Fisher-weighted SAE is trained (Fig. 3 left) but absent from Fig. 3 (right) and Fig. 4 .”

We train a Fisher-weighted SAE at every layer with the same recipe as the Fisher BSF, and add it to Fig. 3 (right) and to the steering table in the appendix (Table 4). We find that our Fisher-weighting improves the causal significance of the learned features and overall can be applied to improve the causal efficacy of different types featurizers.

W2. Application claims

"The 14.25 s headline is a mean over five perturbed trials … Random masking … already moves reopening from 1.89 s to 5.00 s."

We ran 10 more real-robot trials per policy (15 each). Concept masking gives 12.93 s against 3.13 s unmodified and 4.34 s for random masking (Mann–Whitney, Holm-adjusted p0.018 against every policy). We also built a photorealistic simulated replica of our setup in NVIDIA RoboLab and ran over 4000 additional induced-failure trials over 17 fine-tuned policies and eight additional tasks (App. D.3). We run a scaling study over the thresholds (Fig. 13). Scaling EE and CC together to mask between 0.70% and 5.5% of action tokens slows recovery monotonically, from 9.0 s to 29.1 s, while masking random windows or our baselines at the same budgets does not have an effect.

"Table 2 has no SAE-based selector …"

We add one and evaluate it (App. D.3). We use F4050, a latent that tiles F1515. It also delays recovery. However the BSF block is more selective: at 5.5% the F1515 BSF-policy keeps ordinary task success (41/48 episodes vs 40/48 unmodified) while the SAE-latent policy drops to 25/48 (Fisher's exact p<0.001; Fig. 14), and F1515's windows are more precise (Table 3).

"The abstract says … 'remove the targeted behavior' …"

Thank you, we have corrected the abstract.

"The open-loop recovery metric … projects the endpoint shift onto the patching shift …"

This is correct. We now report alignment (App. C.1). The median cosine with the patch shift is at least 0.98 at every site, and exceeds 0.90 in 39–44 of 52 held-out comparisons at the semantic site, 46–48 at the visual and 50 at the geometric. The median endpoint error is 7–8%, 22–23% and 16–21% of the patch shift.

"'Held out' means a mirrored layout or the other instruction … Steering is on MolmoAct2 only … The closed-loop robot … not stated."

We now state each hold-out type (semantic: mirrored layout; visual: other instruction; geometric: leave-one-object-pair-out). We also correct "across multiple models" as we perform steering on MolmoAct2 and data shaping on Pi05. For a much harder hold-out, feature sets found on the real robot and applied in simulation without recalibration redirect the policy in 68% (BSF) and 54% (SAE) of rollouts. (App. D.2, over 13,000 rollouts). Overall this shows that BSF features provide and effective basis for steering, and transfer better and with less effect on closed loop success rate than SAEs.

Clarifications

  1. "Closed loop: … model, robot, real or simulated? … unsteered and directly instructed success rates?"

We use MolmoAct2 on a real Franka FR3 with a Robotiq gripper (DROID setup), which is now stated in the paper with more experimental details including images in Figure 11 (Appendix C). For the closed loop steering results unsteered the policy gets 100% success rate across 15 trials for each object in each object pair. In simulation, the unsteered policy picks the correct object with a 80%.

  1. "Fisher variant: full training objective …"

The complete objective is now in App. E ("Fisher-weighted featurizers").

  1. "Probe input, alignment, SAE assignments, dictionary widths"
  • Blocks are probed with their three coordinates.
  • SAE assignments: each SAE latent is scored by the share of its decoder direction inside the block's subspace times its (non-negative) firing correlation with the block; the top K are kept.
  • Widths: an expansion factor of 4 per coordinate, i.e. 3,413 blocks or 10,240 latents for MolmoAct2 and MolmoBot, and 2,731 blocks or 8,192 latents for π0.5, with 63 active coordinates for both featurizers. All are now in App. E.
  1. "Per-trial reopening times and cap rule; choice of F1515 and thresholds; precision of selected windows"
  • All 90 real-robot trials reopened (longest 34.3 s), so no cap was needed; we add the per-trial times. In simulation, trials that never reopen count as 40 s. F1515 was selected with the same labelled examples every baseline uses (81 recovery and 25 successful-grasp timesteps; it fires on 56/81 and 0/25)
  • The thresholds were fixed before fine-tuning: E≥150E \geq 150 at the start of the high-energy tail of F1515's per-episode activation on 2,000 DROID episodes, and C≥50C \geq 50 so that a frame needs broad multi-token firing rather than a few active tokens.
  • Across 40 random windows per selector (Table 3), 87% of F1515's windows show a recovery, against 64% for the SAE latent and 8–32% for the other learned selectors.

Requested experiments

  1. "Dimension-matched controls, reported with variability"

We complete this experiment using SAE triplets for probing and ablation and add the results to (Fig. 3), joint ablation of the top 7 blocks, top 21 latents and top 7 triplets is added (App. E.1), and a Fisher SAE in Fig. 3 and Table 4.

  1. "An SAE-latent selector, a simple gripper-state heuristic, more perturbed trials or a second seed"

On the real robot each policy now has 15 perturbed trials instead of 5. We also built a simulated replica of our setup (NVIDIA RoboLab) and fine-tuned 11 additional policies with the same recipe, evaluating all 17 on over 4000 induced-failure trials across eight pick-and-place tasks (App. D.3):

  • Seeds. The result replicates across three independent fine-tuning seeds (9.3, 12.5 and 10.1 s, against 5.4 s unmodified).
  • Scaling study. A four-point dose sweep of the F1515 thresholds shows a monotonic dose-response, from 9.0 s at 0.70% of action tokens to 29.1 s at 5.5% (Fig. 13).
  • Baselines at four times the budget. At 5.5%, random, kNN and influence-based masking still have no effect (3.3–4.3 s), and difference of means only a small one (8.7 s).
  • SAE-latent selector we complete at both budgets, with ordinary task success for every policy
  • We evaluated a gripper-state heuristics but given its very low intersection with our concept based attribution (Jaccard 0.05) we do not train a policy on it.
Reviewer_iyReOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether block-sparse featurizers (BSFs), which represent each feature as a low-dimensional subspace, provide a better interface for understanding and controlling vision-language-action models than scalar sparse autoencoder (SAE) features. Using large corpora of individual token activations, the authors identify features associated with objects, robot state, spatial position, and task progress. They evaluate these features through action-steering experiments, including closed-loop object redirection, and demonstrate feature-based data selection: masking approximately 1.4% of fine-tuning action tokens substantially delays recovery from an induced grasp failure while preserving performance on an unperturbed pick-and-place task.

Critical strengths
  • Well-motivated representation choice: Multi-dimensional features are a natural fit for continuous robot variables and related semantic concepts. The visualizations help explain how a BSF block can group structure that an SAE distributes across several features.
  • Substantial effort toward stronger SAE baselines: Both approaches use large, unpooled activation corpora and a shared training pipeline, with matched total latent widths and active-coordinate budgets in key comparisons.
  • Connects interpretation to measurable behavior: The paper evaluates interventions rather than relying solely on feature visualizations. Closed-loop semantic steering achieves 93.3% task completion compared with 75.0% for SAE steering across the tested object pairs.
  • Interesting application to training-data selection: The data-masking experiment connects internal features to behavior learned during fine-tuning, with comparisons against influence-based attribution, representation-based selection, and random masking.
Critical weaknesses
  • Fairness of the BSF–SAE comparison: A BSF feature contains multiple coordinates, so fewer edited features or larger per-feature effects do not alone establish an advantage. Compare against groups of SAE features with matched intervention size.
  • Limited closed-loop evaluation: Successful steering is demonstrated for semantic redirection on three object pairs. Broader tasks and another model would strengthen the claim of a generally useful control interface.
  • Insufficient support for the data-shaping claim: Five recovery trials show delayed recovery, rather than removal of the behavior. More trials and failure types, together with an SAE-based masking baseline, would better establish the contribution.
  • Claims should reflect the mixed results: SAEs achieve better held-out recovery for visual steering, and geometric steering is comparable. The paper should state more precisely where BSFs help: grouping related variation, some semantic interventions, and the demonstrated closed-loop setting.
Questions for authors

All feedback and questions have been expressed in the weaknesses section. I have no other clarifications to request of the authors.

2 comments

We thank the reviewer for their thoughtful review and feedback. We appreciate the recognition that multi-dimensional features suit robot variables, our effort toward strong SAE baselines, and that the paper connects interpretation to measurable behavior through closed-loop steering and training-data selection. We address the weaknesses and questions below.

Fairness of the BSF–SAE comparison

"A BSF feature contains multiple coordinates, so fewer edited features or larger per-feature effects do not alone establish an advantage. Compare against groups of SAE features with matched intervention size."

Thank you for your suggestion, we have implemented SAE grouping as a more fair comparison with BSF blocks. We group SAE feature into sets of three (by activation correlation and at random). We do the same for a Fisher-weighted SAE trained (Fig. 3, Table 4, App. E.1).

  • Intervention size. At matched size for blocks and SAE latents triplets, one block moves the end effector slightly more than one SAE triplet in the early and middle layers (+1.8 mm [1.0, 2.7] on about 16 mm in layers 9–19), while the difference is reduced when seven units are ablated together and in later layers (App. E.1).
  • Decodability. BSFs outperform SAE triplets in decoding task relevant variables. From layer 9 the best block decodes robot-state variables better than the best SAE triplet

These results support our view that the features in these models are genuinely multi-dimensional. What blocks organize related variation into single features.

Claims should reflect the mixed results

"SAEs achieve better held-out recovery for visual steering, and geometric steering is comparable. The paper should state more precisely where BSFs help: grouping related variation, some semantic interventions, and the demonstrated closed-loop setting."

We agree and will revise the claims of the paper to be consistent with this. We highlight three main advantages of BSFs over SAEs

  • Decodability: a learned block decodes robot-state variables better than the SAEs

  • Steering: BSF feature sets steer the real robot more reliably and precisely in closed loop (93.3% vs 75.0% task completion, Fisher's exact p=0.011p = 0.011) and transfer better from the real robot to simulation (68% target selection, against 54% for SAE latents)

  • Data shaping: BSF features are more selective data filters than SAE latents covering the same feature.

Limited closed-loop evaluation

"Successful steering is demonstrated for semantic redirection on three object pairs. Broader tasks and another model would strengthen the claim of a generally useful control interface."

We added a large simulated closed-loop evaluation in a photo-real replica of our real-robot setup (NVIDIA RoboLab; App. D.2). We run over 13,000 rollouts on 14 new object pairs and 32 rollouts per direction. We find that feature sets found on the real robot redirect the policy in simulation with no recalibration (68% for BSF and 54% for SAE). We find that steering is robust to changes of table colour, lighting and background, to scattered object placements and to distractor objects. For our real robot results we use two models, MolmoAct2 for steering and Pi05 for data shaping.

Data shaping

"Five recovery trials show delayed recovery, rather than removal of the behavior. More trials and failure types, together with an SAE-based masking baseline, would better establish the contribution."

We agree our method does not fully remove failure recovery beahavior and will update the paper to be consistent with this. We have substantially expanded this evaluation.

  • More real-robot trials. Each policy now has 15 induced-failure trials instead of 5. Concept-based masking gives 12.93 s against 3.13–5.74 s for every other policy (Holm-adjusted p ≤ 0.018).
  • Many more policies and trials in simulation (App. D.3). We fine-tuned 11 additional policies and evaluated all 17 on 4,352 additional trials across eight tasks. The result replicates across three training seeds, slows recovery monotonically as more feature-selected data is masked (9.0 to 29.1 s).
  • SAE-based masking baseline. Masking with an SAE latent that covers F1515 also delays recovery. However, the SAE-latent policy also loses ordinary task success (25/48 episodes, against 41/48 for F1515 and 40/48 unmodified), and its selected windows are less precise (64% vs 87%, Table 3).

I thank the authors for their detailed response. The reply responds to and resolves many of the critiques of the work. I raise my score accordingly.

Reviewer_YEMROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper applies block-sparse featurizers (BSFs), which model features as low-dimensional subspaces rather than single directions, to vision-language-action (VLA) robot foundation models. Trained on a very large corpus of unpooled token activations, BSFs recover multi-dimensional features spanning perception, language, proprioception, task progress, and control variables; they are causally validated via fine-grained semantic, visual, and geometric steering and further used for feature-guided dataset attribution to suppress a specific behavior (failure recovery) by masking ≈1.4% of training tokens. Quantitatively, BSFs match or exceed strong SAE baselines on several metrics (functional fidelity, per-feature causal effect, single-feature decodability), and qualitatively recover coherent manifolds that SAEs fragment, enabling more compact and targeted interventions in several settings.

Critical strengths

The feature-based data attribution experiment (Section 4) is the paper’s standout contribution. Identifying a single BSF feature (F1515) that tracks failure recovery, using it to flag 6.5% of DROID episodes, and showing that masking just 1.38% of action tokens produces a sevenfold increase in gripper reopening time while maintaining perfect task success is compelling and practically useful. The careful comparison against TRAK, kNN, and difference-of-means baselines with matched budgets strengthens this claim. The multi-level steering framework (semantic/visual/geometric across layers) also provides unusually fine-grained causal evidence for feature validity in VLAs.

Critical weaknesses

BSF advantages over SAEs are inconsistent, undermining the central thesis. SAEs achieve higher held-out recovery in visual steering (97.3% vs. 84.6-88.3%) and outperform on some quantitative metrics. The paper acknowledges this but does not adequately explain when or why BSFs should be preferred. Second, only semantic steering is evaluated closed-loop. Visual and geometric steering remain open-loop, which is a significant gap for a robotics paper claiming fine-grained behavioral control. The authors note that closed-loop visual/geometric steering requires tracking image regions across observations, but this limitation leaves the most practically relevant evaluation incomplete.

Additionally, the steering evaluation lacks comparison to lightweight and very popular steering baselines such as Contrastive Activation Addition (linear steering) or conceptor-based subspace methods (both have been applied to robotics and conceptor-based approach has been shown to outperform SAE-steering), making it unclear whether the improvements over SAEs justify training a full BSF when simpler closed-form approaches might achieve comparable behavioral control. You will need to cite those relevant papers as well.

Questions for authors
  1. When should practitioners prefer BSFs over SAEs? SAEs outperform BSFs on visual steering held-out recovery (97.3% vs. 84.6-88.3%) and on some quantitative metrics, while BSFs win on compactness and closed-loop task completion. Can you provide clearer guidance, or even a decision rule, for when the block-sparse inductive bias is actually worth the added complexity?
  2. How robust is the data attribution result to the feature and threshold choices? The failure-recovery experiment hinges on a single feature (F1515) and specific activation/count thresholds (E ≥ 150, C ≥ 50). How sensitive is the sevenfold reopening delay to these thresholds?
  3. What is the path to closed-loop visual and geometric steering? You note that robust closed-loop evaluation for these modalities requires tracking image regions across changing observations. Do you have preliminary results or a concrete proposed method for this? Without closed-loop validation, the practical value of visual and geometric steering for deployed VLA policies remains uncertain.
2 comments

We thank the reviewer for their thoughtful review and feedback. We appreciate the recognition of feature-based data attribution as the paper’s standout contribution and of the multi-level steering framework as fine-grained causal evidence for feature validity. We address the weaknesses and questions below.

BSF and SAE comparison

"BSF advantages over SAEs are inconsistent, undermining the central thesis. SAEs achieve higher held-out recovery in visual steering (97.3% vs. 84.6–88.3%) and outperform on some quantitative metrics. The paper acknowledges this but does not adequately explain when or why BSFs should be preferred."

We agree that the original comparison did not properly emphasize the strong contributions of BSFs. We have ran additional experiments to showcase this. A BSF block has three coordinates while a SAE latent only has one. For improved fairness we now compare blocks with SAE triplets which are latents of the same SAE grouped after training into sets of three and evaluated exactly like blocks (Fig. 3, Table 4, App. E.1). Here are a few of the key advantages of BSFs:

  • Decodability: a learned block decodes robot-state variables better than the SAE triplets

  • Steering: BSF feature sets steer the real robot more reliably and precisely in closed loop (93.3% vs 75.0% task completion, Fisher's exact p=0.011p = 0.011) and transfer better from the real robot to simulation (68% target selection, against 54% for SAE latents)

  • Data shaping: BSF features are more selective data filters than SAE latents covering the same feature.

"Only semantic steering is evaluated closed-loop. Visual and geometric steering remain open-loop, which is a significant gap for a robotics paper claiming fine-grained behavioral control."

"What is the path to closed-loop visual and geometric steering? … Do you have preliminary results or a concrete proposed method for this?"

We see the steering in this paper primarily as a means to validate our features, not as something which is useful in deployment. We argue that if a small set of features can precisely change the behavior of the policy then those features form an effective sparse basis. We test this in our open-loop evaluation and closed-loop semantic steering. We have further strengthened this validation in this rebuttal. We introduce a new simulated replica of our setup (App. D) and run over 13,000 additional trials, across more objects and other visual and scene level perturbations. We hope that the most direct practical application of the features is our data shaping (Sec. 4, App. D.3) experiments.

Steering baselines

"The steering evaluation lacks comparison to lightweight and very popular steering baselines such as Contrastive Activation Addition (linear steering) or conceptor-based subspace methods (both have been applied to robotics and conceptor-based approach has been shown to outperform SAE-steering)… You will need to cite those relevant papers as well."

We now include both of these baselines (Table 6, App. D.2) as well as citing both papers. A difference-of-means direction or a conceptor edits the full residual vector and is typically fit separately for each contrast. The goal of our featurizer is to express the same edit with a sparse set of units learned in an unsupervised manner. We find that our steering results nearly matching these baselines demonstrates how effectively they learn an effective basis for which to steer the model. We report the full results in Table 6 but treat the dense edits as an upper bound on performance.

Robustness of the data attribution result

"How robust is the data attribution result to the feature and threshold choices? … How sensitive is the sevenfold reopening delay to these thresholds?"

We run an additional 10 trials per policy. Concept masking gives a mean reopening time of 12.93 s against 3.13 s for the unmodified policy and 3.45–5.74 s for every baseline selector (Holm-adjusted p≤0.018p \leq 0.018). We then built a photo-real simulated replica of our setup in NVIDIA RoboLab. We run over 400 additional trials over 17 fine-tuned policies and eight tasks (App. D.3). We run a scaling study over the thresholds (Fig. 13). Scaling EE and CC together to mask between 0.70% and 5.5% of action tokens slows recovery monotonically, from 9.0 s to 29.1 s, while masking random windows at the same budgets has no effect (4.0 and 4.3 s, against 5.4 s unmodified).

Thanks for the response. I have read it and considered it and my assessment remains the same.

Reviewer_uHpQOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether block-sparse featurizers (BSFs) provide more interpretable and controllable representations than traditional sparse autoencoders (SAEs) in vision-language-action models.

The authors apply BSFs to MolmoAct2, π₀.₅, and MolmoBot, recovering features related to perception, object semantics, proprioception, and robot control. They demonstrate feature-based steering and use data attribution to identify training examples associated with failure recovery. The main results show improved closed-loop task completion (93.3% versus 75% for SAEs) and that masking 1.38% of action tokens substantially delays failure recovery.

In addition they add a “Fisher-weighted” variant, which weights reconstruction error by the action log-likelihood gradient.

Critical strengths
  1. Useful extension to robotics. Applying BSFs to VLA models is well motivated. Equations 2–5 clearly define the block-sparse representation, and Figures 1 and 6 provide interesting examples of interpretable robot-state features.

  2. Behavioral validation beyond visualization. The closed-loop experiments in Table 1 demonstrate that discovered features can influence actual robot actions. The comparison with SAEs using the same activation corpus is also valuable.

  3. The data-shaping experiment is a concrete test of whether a feature can find training data.

Critical weaknesses
  1. Limited evidence for robust behavioral control. Table 1 reports results from only three object pairs and 60 trials. Although BSFs achieve 93.3% task completion, the evaluation is too narrow to establish generalizable robot steering.

  2. The data-shaping headline is thin, and nothing ties it to block geometry. Table 2 has 5 trials per method and a single fine-tuning seed (“All runs share the same seed”, App. B). There are no intervals.

  3. The data-attribution claim needs stronger validation. Section 4 and Table 2 claim that masking only 1.38% of action tokens selectively suppresses failure recovery. However, recovery is evaluated over just five trials per method, with no variance or repeated training seeds.

Questions for authors

Along with weaknesses please answer the followings.

  1. In Figure 4, why does SAE achieve better visual steering recovery despite BSFs using fewer features?

  2. How was the failure-recovery feature F1515 selected, and how sensitive is the attribution result to its activation threshold (Appendix B)?

  3. What are the off-target effects of steering, for example on the non-target object or on unrelated tasks?

3 comments

We thank the reviewer for their thoughtful review and feedback. We appreciate the recognition that applying BSFs to VLAs is well motivated and that our steering and data-shaping experiments test whether features influence and explain robot behavior, beyond visualization. We address the weaknesses and questions below.

Stronger experimental validation

“Limited evidence for robust behavioral control. Table 1 reports results from only three object pairs and 60 trials.” We agree that 60 trials over three object pairs was not sufficient to establish generalizable steering, and we have scaled this evaluation up substantially.

  • We conduct a large simulated evaluation (App. D.2).We built a photo-real simulated replica of our setup (NVIDIA RoboLab) and ran over 13,000 additional closed-loop rollouts: 14 object pairs instead of 3, and 32 rollouts per direction instead of 10.
    • We observe good transfer from the real robot. Feature sets found on real-robot data, applied in simulation, redirect the policy to the other object in 68% of rollouts (BSF) and 54% (SAE).
    • This performance generalizes to new scenes. Under four visual perturbations (table colour, lighting, background), steering continues working (73–86%), even with scattered and distractor objects
  • We add intervals on the real-robot results. BSF feature sets complete the redirected task in 56/60 trials (Wilson 95% CI 84–97%) and SAE feature sets in 45/60 (63–84%); Fisher’s exact p = 0.011.

“The data-attribution claim needs stronger validation….However, recovery is evaluated over just five trials per method, with no variance or repeated training seeds.”

We agree, and have strengthened this result on the real robot and in simulation.

  • Real robot. We ran 10 more induced-failure trials per policy for 90 trials in total. Every trial reopens, so we do not apply a cap. Concept-based masking gives a mean reopening time of 12.93 s (bootstrap 95% CI [8.3, 18.0]) against 3.13 s ([2.1, 4.3]) for the unmodified policy and 3.45–5.74 s for our other baselines. Ordinary task success is unchanged (100% vs 90% for the unmodified policy over 20 trials; 11.79 vs 12.13 s).
  • Simulation (App. D.3). We fine-tuned 11 additional policies and evaluated all 17 on over 4000 additional trials over eight different tasks tasks
    • Replication across seeds. The ordering of the real-robot result is reproduced (9.3 s vs 4.0–5.4 s, p ≤ 1.2×10⁻⁵) and holds across three different fine-tuning seeds (9.3, 12.5 and 10.1 s).
    • Dose-response. Masking more F1515-selected data slows recovery monotonically: 9.0, 9.3, 21.0 and 29.1 s at 0.70%, 1.38%, 2.75% and 5.5% of action tokens.
    • Baselines. At 5.5% of action tokens, random, kNN and influence-based selection have no effect (3.3–4.3 s), and difference of means has a small effect (8.7 s).

“The data-shaping headline is thin, and nothing ties it to block geometry.”

In order to tie our data-shaping results to block geometry we run an additional SAE baseline. Block-F1515 (the feature related to failure recovery) is tiled by a small number of SAE latents (Fig. 8). We perform a similar concept based attribution experiment with one of them: we use SAE-F4050 at the same budget and training procedure (App. D.3). It also delays recovery, but slightly less than Block-F1515 at the 1.38% masking (8.1 vs 9.3 s, p = 0.027) and equally at 5.5% (29.4 vs 29.1 s). However two methods differ significantly in the side effects of the data shaping. At 5.5%, the Block-F1515 policy keeps ordinary task success the same (41/48 episodes vs 40/48 unmodified), while the SAE-latent policy drops to 25/48. Block-F1515 also has a higher precision. In 40 randomly sampled windows, 87% of F1515’s windows show a failure recovery, against 64% for the SAE latent and 8–32% for the other learned selectors (Table 3). Overall this shows that BSFs geometry provides a more effective basis for data shaping compared to just SAE features and ties our results more strongly to block geometry.

Additional Questions

“In Figure 4, why does SAE achieve better visual steering recovery despite BSFs using fewer features?”

When the SAE's latents are grouped into triplets (such that we can compare BSFs and SAEs on equal footing), the two perform roughly the same. The sole SAE gap versus the BSF may also come from reconstruction quality. We find that the SAE reconstructs layer-7 image tokens better than the BSF (relative error 0.29 vs 0.35).

“How was the failure-recovery feature F1515 selected, and how sensitive is the attribution result to its activation threshold (Appendix B)”

F1515 was selected with the same set of labelled examples that our baselines use. This is ~80 failed-grasp recovery timesteps and ~25 successful-grasp timesteps from our rollouts. The thresholds were selected before fine-tuning based on visual inspection of F1515's activation histogram over the first 2,000 DROID episodes. We add an additional scaling study in Figure 13. We show that changes these thresholds together to mask between 0.70% and 5.5% of action tokens slows recovery monotonically (9.0, 9.3, 21.0 and 29.1 s), while masking random windows at the same budgets has no effect (4.0 and 4.3 s, against 5.4 s unmodified).

“What are the off-target effects of steering, for example on the non-target object or on unrelated tasks?”

Our interventions change the targeted behavior while leaving the rest of the policy intact, which we see as a key improvement over prior VLA featurizer work. During our real robot experiments, the BSF-steered policy goes for the target object in every trial (60/60) and still completes the full pick-and-place task in 56/60, while with SAE feature steering leads task completions indicating more off target effects (45/60, from 53/60 selections). Prior VLA SAE work (Swann et al., 2026) can also steer a policy toward a specific object, but in doing so it disrupts the policy's ability to grasp, showing strong off target effects.

We also see this same selectivity separating the BSFs from SAEs in data shaping. Masking the training data selected by failure recovery feature suppresses failure recovery while ordinary task success is unchanged, both on the real robot (100% vs 90% for the unmodified policy) and in simulation, even when up to 5.5% of action tokens are masked (41/48 episodes vs 40/48 unmodified). Our new SAE latent baseline that covers the same feature suppresses recovery as strongly at that budget, but the policy also loses ordinary task success (25/48; Fisher's exact p < 0.001), and it has less precision in its selections (64% vs 87%).

Thank you for the thorough rebuttal. The scaled-up evidence addresses most of my concerns:

The 13,000-rollout simulation (BSF 68% vs SAE 54%), the real-robot confidence intervals, and the 90-trial recovery study with intervals all make the empirical claims more convincing. The replication across seeds, the dose-response, and the SAE-latent baseline now tie the data-shaping result to block structure.

I am raising my confidence to 3. I keep my original rating.

Reviewer_Ns1uOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies an existing method, block-sparse featurizers (BSF), to VLA models and compares it with sparse autoencoders (SAE). The authors show that BSFs can capture multidimensional features related to visual perception, semantics, and robot states. They then use these features to steer robot actions, such as redirecting the robot to grasp a different object. Finally, they demonstrate that BSF features can help identify training data associated with failure-recovery behavior, allowing this behavior to be suppressed through targeted data masking.

Critical strengths

S1. Solid experimental design. The paper provides a systematic empirical evaluation, including carefully controlled SAE baselines, quantitative feature comparisons, causal steering experiments, and downstream behavior shaping.

S2. Effective behavioral steering. The paper demonstrates that modifying the identified features can directly change robot behavior. In particular, the closed-loop experiments show that BSF-based interventions can reliably redirect the robot to pick up a different object.

Critical weaknesses

W1. Limited novelty. BSF is directly adopted from Fel et al. (2026a), which already demonstrated its advantages over SAEs in capturing multidimensional visual concepts. Similar advantages in VLA representations are therefore largely expected. Furthermore, SAE-based feature discovery and steering have already been studied in VLAs (Swann et al., 2026; Grant et al., 2026; Jin et al., 2026). While the steering and data-shaping experiments provide useful applications, the incremental contribution beyond applying an existing method to VLAs remains unclear.

W2. Limited scale and diversity of steering evaluation. The steering experiments are mainly limited to two-object target-redirection tasks. Open-loop evaluation covers 52 scene–instruction comparisons, while closed-loop evaluation includes only 60 trials across three object pairs. Larger-scale experiments in simulation, covering more diverse tasks and scenarios, would provide stronger evidence for the method's generalizability and robustness.

Questions for authors

Given that Fel et al. (2026a) already demonstrated the advantages of BSFs over SAEs in capturing multidimensional visual concept manifolds, and that SAE-based feature discovery and steering have been explored in VLAs, could the authors clarify what new VLA-specific insights this work establishes beyond applying an existing method to VLA models?

3 comments

We thank the reviewer for their thoughtful review and feedback. We appreciate the recognition of our controlled SAE baselines, systematic evaluation, and closed-loop steering results. We address the weaknesses and questions below.

Novelty beyond applying BSFs to VLAs

“What new VLA-specific insights this work establishes beyond applying an existing method to VLA models?”

Applying BSFs to VLAs is not a direct transfer of prior BSF work. Prior BSF work studies vision-only models, whereas a VLA processes image, language, proprioceptive and action tokens together. We find multi-dimensional features that span these modalities, to our knowledge these are the first multi-modal features found with BSFs. Across the three VLAs we study, this yields a plethora of new findings. We list our findings about VLA representations and our further contributions below.

1. New findings about VLA representations.

  • Features that span modalities. Fel et al. 2026 study vision-only models. We find single blocks that fire for the same concept on exterior-camera, wrist-camera and text tokens (Fig. 6). We also find blocks that hold robot and scene geometry, Pi0.5 computes Cartesian end-effector position from joint angles and stores it in unused image tokens (Fig. 7, middle), MolmoBot encodes the camera’s pose relative to the robot (Fig. 7, right), and MolmoAct2 state tokens encode value and component index in one block (Fig. 8). To our knowledge these are the first cross-modal multi-dimensional features reported with any featurizer and the first mutli-model features reported in robotics.
  • VLA features are inherently multi-dimensional. We find many cases where a block’s extra dimensions carry how a concept varies. One gripper block arranges its codes in an X whose arms are the two sides of the robot gripper and whose position along each arm is how far the gripper is open (Fig 6.) One pi0.5 block traces the end-effector’s x, y and z position through an episode (Fig. 6). One MolmoAct2 block orders proprioceptive tokens by numeric value along curves, and by which joint they describe across curves (Fig 7). One block places “bowl”, “bin” and “basket” in distinct regions of the same subspace on both image and text tokens (Fig. 5). We show that many VLA features are best represented as multiple dimensions. This finding is backed up by our steering experiments where we show that steering with BSF features more precisely shapes behavior compared to SAEs.

2. An action-grounded training objective. Vision featurizers have no action likelihood to use. Weighting reconstruction by the Fisher information of the action log-likelihood roughly doubles the causal effect of a feature in layers 9–19.

3. Scale and unpooled tokens, relative to prior VLA SAEs. We train our featurizers on about 52M per-token activations from 45,344 episodes. Swann et al. use 567k mean-pooled timestep vectors and Grant et al. 500k action-token activations, both about 100× fewer. While prior VLA SAE steering reports whether the robot reaches for the target (because the coarse grain steering destroys the policy), ours also reports whether it finishes the task.

4. Feature-based curation of robot training data. Prior VLA featurizer work uses the features at inference time only. We show masking 1.38% of training data identified by a failure recovery feature can significantly effect that specific behavior of the policy while leaving others intact across more than 4,352 simulated trials and three seeds.

Scale and diversity of the evaluation

“Larger-scale experiments in simulation, covering more diverse tasks and scenarios, would provide stronger evidence for the method's generalizability and robustness.” We agree that additional experimental results would strengthen the paper, and have expanded both the real-robot and the simulated evaluations to make the best use of the short rebuttal time period.

  • Real robot. We run an additional 60 trials for our Table 2 behavior shaping, up to 15 trials per policy. Concept based attribution takes 12.93 s to recover versus 3.13–5.74 s for the baselines (Holm-adjusted p ≤ 0.018). This demonstrates that BSF attribution outperforms our baselines (including an SAE baseline we added) at effective data shaping.
  • Simulated data shaping (new, App. D). We run over 4,000 additional trials over 17 fine-tuned policies across three fine-tuning seeds. We find that increasing the sensitivity of our attribution monotonitcally slows recovery (9.0 → 29.1 s), while random, kNN and influence selection have no effect. This further reinforces our original hardware results, but compares to more baselines at different masking percentages across more tasks.
  • Simulated steering (new, App. D). We rerun the closed-loop protocol from Table 1 in photo-real DROID simulator (NVIDIA RoboLab ) and ran over 13,000 rollouts: 14 object pairs instead of 3, 32 rollouts per direction instead of 10 and four visual shifts (table colour, lighting, background, combined). We can apply the paper’s real-robot recipes, unchanged to the simulation and find that they make it select that object in 68% (BSF) and 54% (SAE) of rollouts. This dramatically increases the sample size of our work and demonstrates that BSFs more robustly transfer from real to sim.

Thank you for the clarification. I appreciate the additional contributions highlighted by the authors. However, I still share concerns about the limited novelty and significance of the work, so I will maintain my original rating.

Comments on the paper

1 comment

We thank all five reviewers for their constructive feedback, which has substantially improved the paper. Below we summarize the main changes made to the paper.

A coordinate-matched comparison of BSFs and SAEs (R4, R5). To fairly compare SAEs and BSFs we now compare BSF blocks (block size of three) with SAE triplets: latents of the same SAE grouped after training into sets of three (by activation correlation, or at random) and evaluated in the same manner as blocks. This improves the fairness of our comparisons between BSFs and SAEs. Overall we show that BSFs are better than coordinate matched SAEs at task relevant variable decodability, steering and data shaping.

Large-scale simulated evaluation (R1, R2, R4). We built a photo-real simulated replica of our real-robot setup in NVIDIA RoboLab (Isaac Sim, DROID Franka; App. D.1) and run over 17,000 additional trials in both closed loop steering and data shaping (including 11 additional fine-tuned policies). This includes both

  • Closed-loop steering (App. D.2): over 13,000 rollouts across 14 object pairs, four visual shifts, scattered layouts and distractor objects.
  • Data shaping (App. D.3): 11 additional fine-tuned policies, over 4,000 additional trials. We further replicate the real-robot result across three training seeds. Recovery slows monotonically as more F1515-selected data is masked (9.0 to 29.1 s), while random, kNN and influence-based selection have no effect even at four times the token %. This shows that BSFs features outperform baselines (including SAEs) at concept based data attribution.

More real-robot trials (R2, R5). Failure recovery now has 15 real robot trials per policy instead of 5. Concept-based masking raises the mean reopening time to 12.93 s, against 3.13–5.74 s for our baselines. This increases the significance of our results and complements the 4,000 additional trials we run in simulation.

We respond specifically to each reviewer’s points individually below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.