- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether block-sparse featurizers (BSFs) are a better tool than sparse autoencoders (SAEs) for understanding and steering vision-language-action (VLA) policies. A BSF represents each feature as a small subspace (b = 3) rather than a single direction. The authors train BSFs and matched TopK SAEs (the b = 1 case of the same pipeline) on roughly 50M unpooled token activations per model and layer from MolmoAct2, π0.5 and MolmoBot. They also train a Fisher-weighted BSF variant on MolmoAct2.
The contributions are:
-
Comparison and phenomenology. A quantitative comparison on MolmoAct2 covers reconstruction, change in action likelihood, per-feature ablation effect and single-feature decodability (Fig. 3). Qualitative examples include gripper, end-effector, proprioceptive and cross-modal object features.
-
Causal validation by steering. Feature edits at three sites redirect the predicted reach: the L3 noun token, the L7 image tokens and the L23 action-readout token (Fig. 4). In closed loop, BSF edits complete 56/60 redirected tasks versus 45/60 for SAE edits (Table 1).
-
Data shaping. A π0.5 "failure-recovery" feature selects DROID windows whose loss is then masked (1.38% of action tokens). This raises the time to reopen a manually closed gripper from 1.89 s to 14.25 s, with no observed loss of task success on 20 unperturbed trials per method (Table 2).
Critical strengths
-
A controlled comparison, backed by behaviour. The SAE is the b = 1 case of the same pipeline and data, with matched coordinate budgets (21 three-dimensional blocks versus 63 scalar latents; App. D). The shared pipeline and matched total width and active-coordinate budgets make the vanilla BSF–SAE comparison a meaningful control for implementation and overall representation size. The unpooled activation corpus is a real step up from the pooled or small corpora used in prior VLA SAE work. The closed-loop result is the strongest evidence in the paper. Feature sets were frozen before evaluation, and both featurizers were calibrated by the same rule (≥75% recovery, cosine ≥ 0.90; App. C.2). BSF edits complete 56/60 tasks versus 45/60 and lead on task completion in all six pair-directions (Table 3). This provides direct behavioural evidence beyond the per-feature metrics, and it is why I take the paper seriously.
-
Linking a discovered feature to training data with an actual retraining test. The data-shaping experiment re-fine-tunes π0.5 from base and compares against alternatives matched on selected raw-frame budget (TRAK influence, kNN, difference of means, random; App. B). Unperturbed task success shows no drop (20/20 versus 18/20 for the unmodified policy). Showing that a feature-selected ~1.4% of action tokens changes one specific behaviour is a useful demonstration that interpretability can guide robot data curation.
Critical weaknesses
1. The comparisons do not fully isolate the benefit of block geometry from block capacity. Budgets are matched at the featurizer level. But the per-feature metrics behind the claim that geometry is useful "rather than simply excess representational capacity" (p. 2) compare a 3-dimensional unit with a 1-dimensional one.
- Fig. 3 (right): per-feature ablation removes a 3-D block versus a single latent, with the individual effects averaged over the top 7 blocks or top 21 latents. If the single-feature probe uses all three block coordinates, decodability also changes the probe's input dimension; the paper does not say which input is used. Neither right-hand panel has error bars.
- Fig. 3 (left, layer 13): the gain is stated at fixed block sparsity k. For the vanilla featurizers, at matched active coordinates (k·b = 64), R² reads approximately 0.64 for the SAE versus 0.63 / 0.63 / 0.62 for b = 2 / 4 / 8.
- Fig. 4: "more compact semantic and perceptual interventions" (p. 6) holds in blocks but not in nominal edited latent-coordinate count. Semantic: 6.45 vs 3.50. Visual: 61.2 vs 36.25, where the SAE also transfers better (97.3% vs 84.6%), as the paper notes.
- Fisher weighting: a Fisher-weighted SAE is trained (Fig. 3 left) but absent from Fig. 3 (right) and Fig. 4. So the largest gains (Fisher BSF, approximately 50 mm at L15) cannot be credited specifically to the blocks rather than the loss. Eq. 7 defines the Fisher term, but the full Fisher training objective is not given.
As submitted, the advantage attributable specifically to block geometry remains insufficiently isolated, limiting the support for contribution 1. The controls are feasible, which is why I rate the concerns as fixable.
2. The application claims rest on limited or under-specified evidence.
Data shaping:
- The 14.25 s headline is a mean over five perturbed trials, with no spread, from one reported training seed (p. 14), and no rule for trials that never reopen.
- Random masking (windows from unflagged episodes) already moves reopening from 1.89 s to 5.00 s. Concept masking produces a larger mean delay, but the small sample and missing variability leave the reliability of its advantage unclear.
- Table 2 has no SAE-based selector, although Fig. 7 lists six SAE latents that tile F1515. The selector uses summed block norms and active-token counts. Without a matched SAE-based selector, this experiment does not establish whether learning multidimensional blocks improves data selection over scalar features.
- The abstract says masking is "sufficient to remove the targeted behavior", but the result, as the conclusion says, is a delay.
Steering:
- The open-loop recovery metric (App. C.1) projects the endpoint shift onto the patching shift, so 100% does not imply "the same direction" as stated: a shift of (1, 10) against a patch of (1, 0) scores 100%. Closed-loop calibration does require cosine ≥ 0.90, but alignment is not reported for the open-loop held-out results.
- More broadly, "held out" means a mirrored layout or the other instruction in the same scene for two of the three sites. Steering is on MolmoAct2 only, despite "across multiple models" (p. 2). The closed-loop robot and the real-vs-simulated setting are not stated.
Questions for authors
Clarifications (answers could change my rating):
-
Closed loop (Table 1, App. C.2): Please confirm the model, and state the robot and whether the study was real or simulated. What are the unsteered and directly instructed success rates?
-
Fisher variant: What is the full training objective? How is the Eq. 7 term combined with reconstruction and AuxK, with what coefficients, and on which data?
-
Figs. 1 and 3–7: Is a block probed with its three coordinates or with its norm? What are the endpoint error and cosine alignment for each open-loop comparison? How are the "SAE assignments" in Figs. 1 and 5–7 computed, and what is "none"? What are the dictionary widths?
-
Table 2: What are the per-trial reopening times and the cap rule? How were F1515 and the thresholds E ≥ 150, C ≥ 50 chosen? Beyond the five top-scoring examples per method in Fig. 10, what is the precision of the selected DROID windows on a random labelled sample?
Experiments that would raise my rating (feasible with the existing models):
-
Dimension-matched controls, reported with variability:
- One block versus the best training-selected group of three SAE latents, for both probing and ablation.
- Joint ablation of the top 7 blocks versus the top 21 latents.
- A Fisher SAE in Fig. 3 (right) and Fig. 4.
-
Stronger Table 2 evidence:
- An SAE-latent selector, and a simple gripper-state heuristic, at the same frame budget.
- More perturbed trials, or a second fine-tuning seed.
Future work (does not affect my rating): Featurizer-free steering baselines at matched dimension (difference of means, DAS); steering on π0.5; a formal irreducibility test of the blocks (Engels et al., ICLR 2025).
Minor comments (do not affect my rating):
- π0.5 (Black et al., 2025) and MolmoAct2 are not cited.
- Related work on multi-dimensional features (Engels et al., ICLR 2025) and on output-preserving dictionary training (Braun et al., NeurIPS 2024) is missing.
- "Strictly outperform" (p. 6) fails at layer 5 decodability, and Fisher BSF steers worse at two sites (Fig. 4).
- Please clarify the relationship between the edited and reference-patched token sets at each site: 63 edited vs "all 196 exo-image tokens", and "all 79 text tokens".
- Fig. 3's panels are hard to read at print size.
- Typos: "one 1 H200", "limtations", "propriocetion", "elminate", "implicitely".
2 comments
"The open-loop recovery metric … projects the endpoint shift onto the patching shift …"
This is correct. We now report alignment (App. C.1). The median cosine with the patch shift is at least 0.98 at every site, and exceeds 0.90 in 39–44 of 52 held-out comparisons at the semantic site, 46–48 at the visual and 50 at the geometric. The median endpoint error is 7–8%, 22–23% and 16–21% of the patch shift.
"'Held out' means a mirrored layout or the other instruction … Steering is on MolmoAct2 only … The closed-loop robot … not stated."
We now state each hold-out type (semantic: mirrored layout; visual: other instruction; geometric: leave-one-object-pair-out). We also correct "across multiple models" as we perform steering on MolmoAct2 and data shaping on Pi05. For a much harder hold-out, feature sets found on the real robot and applied in simulation without recalibration redirect the policy in 68% (BSF) and 54% (SAE) of rollouts. (App. D.2, over 13,000 rollouts). Overall this shows that BSF features provide and effective basis for steering, and transfer better and with less effect on closed loop success rate than SAEs.
Clarifications
- "Closed loop: … model, robot, real or simulated? … unsteered and directly instructed success rates?"
We use MolmoAct2 on a real Franka FR3 with a Robotiq gripper (DROID setup), which is now stated in the paper with more experimental details including images in Figure 11 (Appendix C). For the closed loop steering results unsteered the policy gets 100% success rate across 15 trials for each object in each object pair. In simulation, the unsteered policy picks the correct object with a 80%.
- "Fisher variant: full training objective …"
The complete objective is now in App. E ("Fisher-weighted featurizers").
- "Probe input, alignment, SAE assignments, dictionary widths"
- Blocks are probed with their three coordinates.
- SAE assignments: each SAE latent is scored by the share of its decoder direction inside the block's subspace times its (non-negative) firing correlation with the block; the top K are kept.
- Widths: an expansion factor of 4 per coordinate, i.e. 3,413 blocks or 10,240 latents for MolmoAct2 and MolmoBot, and 2,731 blocks or 8,192 latents for π0.5, with 63 active coordinates for both featurizers. All are now in App. E.
- "Per-trial reopening times and cap rule; choice of F1515 and thresholds; precision of selected windows"
- All 90 real-robot trials reopened (longest 34.3 s), so no cap was needed; we add the per-trial times. In simulation, trials that never reopen count as 40 s. F1515 was selected with the same labelled examples every baseline uses (81 recovery and 25 successful-grasp timesteps; it fires on 56/81 and 0/25)
- The thresholds were fixed before fine-tuning: at the start of the high-energy tail of F1515's per-episode activation on 2,000 DROID episodes, and so that a frame needs broad multi-token firing rather than a few active tokens.
- Across 40 random windows per selector (Table 3), 87% of F1515's windows show a recovery, against 64% for the SAE latent and 8–32% for the other learned selectors.
Requested experiments
- "Dimension-matched controls, reported with variability"
We complete this experiment using SAE triplets for probing and ablation and add the results to (Fig. 3), joint ablation of the top 7 blocks, top 21 latents and top 7 triplets is added (App. E.1), and a Fisher SAE in Fig. 3 and Table 4.
- "An SAE-latent selector, a simple gripper-state heuristic, more perturbed trials or a second seed"
On the real robot each policy now has 15 perturbed trials instead of 5. We also built a simulated replica of our setup (NVIDIA RoboLab) and fine-tuned 11 additional policies with the same recipe, evaluating all 17 on over 4000 induced-failure trials across eight pick-and-place tasks (App. D.3):
- Seeds. The result replicates across three independent fine-tuning seeds (9.3, 12.5 and 10.1 s, against 5.4 s unmodified).
- Scaling study. A four-point dose sweep of the F1515 thresholds shows a monotonic dose-response, from 9.0 s at 0.70% of action tokens to 29.1 s at 5.5% (Fig. 13).
- Baselines at four times the budget. At 5.5%, random, kNN and influence-based masking still have no effect (3.3–4.3 s), and difference of means only a small one (8.7 s).
- SAE-latent selector we complete at both budgets, with ordinary task success for every policy
- We evaluated a gripper-state heuristics but given its very low intersection with our concept based attribution (Jaccard 0.05) we do not train a policy on it.
We thank the reviewer for their thoughtful review and feedback. We appreciate you pointing out that the closed-loop and retraining experiments provide direct behavioral evidence. We are especially grateful for being transparent with what affected your score. We address the weaknesses and questions below.
W1 Geometry versus capacity
We agree and we have worked hard in this rebuttal to compare on an even basis SAE and BSF features. To more adequately compare them, we form groups of 3 SAE features, randomly and using activation correlation and compare these to the BSF and fisher BSF features (block size = 3). We also trained a Fisher-weighted SAE at every layer. Fig. 3 (right) now includes these triplets and has error bars.
This is correct. At matched active coordinates (k·b = 64) the SAE reconstructs slightly better: R² 0.636 against 0.627 / 0.627 / 0.616 for b = 2 / 4 / 8. We argue that BSFs tradeoff improved structure for pure reconstruction loss. Prior work Fel et al. (2025) has shown that high reconstruction is not necessarily tied to high quality features.
Fel et al. (2025), “Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models”, ICML 2025.
The steering table in the appendix (Table 4) now reports edited coordinates alongside features (semantic 6.5 vs 3.5; visual 61 vs 36, where the SAE also transfers better) and our matched SAE triplet dictionaries for more fair comparisons. When we group the SAE's latents into triplets, so that each unit has three coordinates like a block, most of the feature-count difference disappears.
We train a Fisher-weighted SAE at every layer with the same recipe as the Fisher BSF, and add it to Fig. 3 (right) and to the steering table in the appendix (Table 4). We find that our Fisher-weighting improves the causal significance of the learned features and overall can be applied to improve the causal efficacy of different types featurizers.
W2. Application claims
We ran 10 more real-robot trials per policy (15 each). Concept masking gives 12.93 s against 3.13 s unmodified and 4.34 s for random masking (Mann–Whitney, Holm-adjusted p0.018 against every policy). We also built a photorealistic simulated replica of our setup in NVIDIA RoboLab and ran over 4000 additional induced-failure trials over 17 fine-tuned policies and eight additional tasks (App. D.3). We run a scaling study over the thresholds (Fig. 13). Scaling E and C together to mask between 0.70% and 5.5% of action tokens slows recovery monotonically, from 9.0 s to 29.1 s, while masking random windows or our baselines at the same budgets does not have an effect.
We add one and evaluate it (App. D.3). We use F4050, a latent that tiles F1515. It also delays recovery. However the BSF block is more selective: at 5.5% the F1515 BSF-policy keeps ordinary task success (41/48 episodes vs 40/48 unmodified) while the SAE-latent policy drops to 25/48 (Fisher's exact p<0.001; Fig. 14), and F1515's windows are more precise (Table 3).
Thank you, we have corrected the abstract.