- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Bilinear autoencoders are introduced to find manifold structure in neural activations. The resulting non-linear representation is demonstrated to achieve improved reconstruction and interpretable geometric structure in language model representations compared to representations restricted to linear features.
Critical strengths
-
An interesting and explicit architecture for capturing nonlinear structure that goes beyond standard linear representation is introduced. The quadratic forms are not only more expressive, they are designed to be analyzable through geometric tools and matrix decompositions. This can be a impactful step towards better interpretable nonlinear representations.
-
The evaluation of the methods includes real LLM activations and not only toy examples.
Critical weaknesses
-
The evaluation of the interpretability remains somewhat quantitative. Compelling visual examples are shown, but it is not clear to me whether the discovered manifolds correspond to the structures underlying the model and can be used to gain a deeper understanding of the model.
-
Recent work, (Bhalla et al., 2026), suggests ordinary sparse autoencoders can already contain manifold information distributed across sets of features. It is not clear to me whether bilinear autoencoders discover genuinely new structure or just extract structure that existing SAEs already encode.
-
It is not clear to me how well the approach scales to more large and complex data. The quadratic representations can become expensive to compute and more difficult to analyze when latent dimensions grows.
Questions for authors
For every manifold detected by the bilinear autoencoder, can the same manifold also be reconstructed from a standard SAE using grouped feature analysis, clustering, or a low-dimensional embedding of the feature activations? If not, can you provide examples where the manifold is identifiable only in the bilinear representation?
2 comments
Thank you for the detailed response. The new experiments concerning W2 strengthen the paper. They indicate that bilinear latents capture manifold structure more efficiently than standard SAEs. Concerning W1, the additional analysis is interesting but I still see the weakness remaining. Concerning W3, the provided analysis and training times provide more insight into the performs at the scale consider. However, it is still unclear how well the approach scales to substantially larger model and dictionary sizes. Overall, the rebuttal strengthens the paper in particular with respect to W2 but W1 and W3 remain. So my recommendation is still a weak accept.
We thank the reviewer for their efforts and address their remarks in order. We are currently running additional experiments and will upload the revised paper soon.
Weaknesses
W1 Bilinear latents match TopK under autointerpretability (App. H) and sparse probing (App. G). The viewer already characterises each latent by effective rank, support, importance and the variance captured in 3D. To further characterise their geometry, we will report the distributions of these statistics, plus definiteness, across all latents compared with an untrained dictionary. Whether the model uses these representational properties downstream requires causal validation, which we leave to future work. Quadratic interventions act on energy landscapes instead of fixed directions. This makes them fundamentally different and potentially less prone to overshooting. We believe they deserve a study of their own.
W2 SAE latents may span the same subspaces as ours. Sec. 4.3 shows that different decompositions can span the same subspace, and concurrent work finds that SAE subspaces are reproducible across seeds even when individual latents are not (Gerasimov et al., 2026). However, the manifold structure itself is not in the SAE. Bhalla et al. (2026) recover it post hoc from activation statistics, which they call "possible but unreliable … a workaround, not a solution", and they call for featurisers that "treat geometric objects as the primitive". Our contribution is such a featuriser. A composite captures a quadric manifold in the weights of a single latent. To compare, we added an experiment that measures, for each composite, how many TopK latents are needed to span its subspace. A few TopK latents capture over 50% of it, but capturing 90% typically takes more than 128. This suggests that bilinear autoencoders find coherent structure that TopK SAEs fragment.
W3 For input dimension d, k≥d latents and n≥k tokens per batch, a training step of the composite model costs:
At a fixed expansion factor, both methods scale as O(nd2) with model size. The extra cost is quadratic in the dictionary size k, and all our models train in about 15 minutes on one RTX 4080 (App. B). Analysis cost depends on a latent's rank, not on d. Composites stay low-rank (Fig. 5) and remain competitive in reconstruction and autointerpretability. Therefore, we see dictionary size, not rank, as the main scaling factor. We will add this to Sec. 2.2.
Questions
Q1 Not necessarily (see W2). Post-hoc grouping is only as good as the underlying tiling (Bhalla et al., 2026). Fig. 3 gives an example. The difference of two rank-1 atoms yields a hyperboloid that fires positively on w1 and negatively on w2. These regions are disjoint, so the SAE latents tiling them rarely co-activate and are unlikely to be grouped. The experiment in W2 quantifies how many TopK latents such a composite requires.
We hope this addresses the reviewer's concerns and would be grateful if they would consider raising their score. We welcome any further suggestions.