- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The author proposed a sparse representation-based analysis forπ0.5 VLA policy by adapting timestep-conditioned sparse transcoders (STCs). The learned representations are used for feature discovery, attribution-based circuit tracing, open-loop ablation and steering. In a selected case study, ablating a traced 16-node circuit changes predicted actions more than ablating the target feature alone or matched random circuits. Separately, the author used non-time-conditioned TopK transcoders to examine visual features in the policy’s PaliGemma language-prefix MLPs. Although many selected features retain visual selectivity on held-out frames, sparse interventions have limited action effects, and attribution-based restoration does not improve closed-loop success over occlusion-only or random-restoration controls.
Critical strengths
- The author presented a valuable explanation adaptation to VLA. Applying STCs across action-expert MLPs is meaningful engineering and empirical extension of sparse circuit analysis to a flow-matching policy. The training setup, loss functions, intervention protocols and results are informative.
- The author did not equate visually selective features with open-loop action effects from behavioral recovery, which is commendable in the field of explanability. In particular, they report that attribution-based closed-loop restoration fails to outperform the occlusion-only and random-restoration controls, while full-MLP restoration succeeds. This negative result makes the paper more informative and credible than a report limited to successful examples.
Critical weaknesses
- Limited technical novelty. Sparse-transcoder, timestep-conditioning, circuit-tracing, and counterfactual intervention are existing techniques. The paper adapts these techniques to a flow-matching VLA policy. The methodological advance appears incremental. Meanwhile the empirical results do not yet clearly compensate for the incremental methodological advance.
- Limited validation. The main action-expert circuit result centers on a single discovery-selected feature and its traced 16-node circuit. Although the ablations outperform matched random controls on evaluated open-loop examples, the paper does not show whether applying the same feature-selection, tracing, and intervention procedure to other candidate features would yield similarly coherent and intervention-sensitive circuits. Nor is it clear how consistently such results would hold across tasks and independently held-out episodes. Consequently, the evidence supports a promising case study more strongly than a generalizable account of action-expert computation.
- Limited experimental support for behavioral claims. The action-expert ablation and steering experiments show changes in predicted actions, but do not establish whether the identified features and circuits causally affect closed-loop task performance. In the separate PaliGemma counterfactual experiments, attribution-based restoration does not improve closed-loop success over occlusion-only or random-restoration controls, despite the strong full-MLP positive control. Together, these results leave the behavioral significance of the sparse explanations uncertain. The visual-audit result should not, however, be interpreted as a direct failure of the action-expert circuit.
Questions for authors
- The paper presents the F9970 circuit as a case study but also describes the tracing procedure as a general workflow. Beyond this selected feature, what evidence suggests that the same feature-selection, tracing, and intervention procedure would identify similarly intervention-sensitive circuits for other action-expert features?
- The introduction lists four contributions and acknowledges prior work on sparse-transcoder circuit tracing and timestep conditioning. To be more specific, what is the technical advance in adapting these methods to the flow-matching action expert, beyond applying the existing pipeline to a new model? Which results should instead be viewed as empirical findings of this case study and the separate visual audit?
- Please report the computational cost of training the STCs, including hardware, GPU-hours, peak memory, and other necessary costs. It would also be useful to report the runtime of feature discovery and circuit tracing, distinguishing one-time costs from costs that scale with the number of analyzed features or episodes.
- Circuit tracing uses STCs in replacement mode, but the reported MLP reconstruction losses do not establish end-to-end fidelity. Please report action-chunk differences from the original policy across tasks and flow timesteps. Closed-loop success under replacement, compared with the original policy, would also be informative.
- Presentation. Figure 1 and several other figures are difficult to read at their current resolution. Please improve their legibility and check if they are vector graphs. Please also check the notation throughout, such as “π” at the bottom of Figure 1 is wrongly written as “Pi”
- The appendix provides several diagnostics of the failed closed-loop restoration. Do they identify the main reason attribution-based sparse restoration fails to improve success, or do feature selection and intervention coverage remain unresolved explanations?