- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that distances between representations are not intrinsically tied to downstream functionality, and argues for a principled notion of statistical indistinguishability. The authors do so by giving representations volumes, instead of simply considering them to be points, and they do so with noise jittering, and justify this choice with the geometry induced by the LayerNorm.
The authors test this idea on transformers by distilling models into their "stochastic LayerNorm" version in both vision and language. They show how their method can localize the layer/attention head responsible for distinguishing progressively corrupted inputs in the case of vision, and two 'similar' sentences in the case of language.
Critical strengths
- The identified problem is interesting: tying representations to function is a principled way to study representational similarity. Furthermore, considering volumes instead of points could be particularly interesting to assess the sensitivity of interpretability tools (such as probes) to noise.
- The method is well grounded and mathematically sensible.
- The method is lightweight and simple to implement, making it potentially widely applicable.
Critical weaknesses
My main problem with this paper is its evaluation.
- It identifies a problem: "distances between point-valued representations are not intrinsically tied to down-stream function", but it never shows examples of this actually being an issue in practice. I think that the paper would be greatly strengthened if the authors could (i) show occurrences of this problem with standard point-wise metrics such as L2 distance or cosine similarity, and (ii) show how the proposed method fixes it.
- As acknowledged by the authors, all statements made about the distilled noisy models are not guaranteed to transfer to the teacher model. The paper would benefit from an interventional study on both the teacher and the student models. For example, if the method identifies a particular architectural component to be responsible for the separation of two inputs, can an intervention verify this causally in the distilled model? If the same intervention also cleanly transfers to the teacher, this would provide evidence that the analysis conducted on the student model also transfers to the teacher model.
- The narrative is a bit hard to follow, and I had to read the paper several times to understand what is the research gap the authors are addressing. I think that this could be better exposed by incorporating the first point above and by making clearer what each experiment is intended to show and how it fits the narrative that representations should have volume.
Questions for authors
- Can the authors provide examples where standard point-wise metrics fail while this framework succeeds at appropriately measuring representational similarity?
- Can the authors provide an interventional study to assess the framework's validity both within the student and teacher models?
- Can the authors clarify the relationship between their method and quantization? They seem to be quite related as they both convert/construct lower precision models.
- Looking at the equation for BC (lines 200-206), if is fixed, BC only depends on the cosine similarity between the means and their cosine similarity. Can the authors discuss how their method relates to first do noise jittering and then compute the cosine similarities of the means?
- Transformers are quite complicated architectures, but from my understanding this method is applicable to any architecture with LayerNorm, including simple MLPs. Do the authors think that evaluating the framework on simpler architectures could highlight its features and potentially reduce confounding factors coming from architectural specifics?
We thank the reviewer for the helpful feedback, which directly led to changes in the revision. The revision centers around reframing and two new key results.
First, we had placed too much emphasis on the distinguishability of a pair of inputs in the submission. We think that the analyses on a single processing stream provide clearer novelty compared to the deterministic geometry scenario (W1), and have reworked several parts of the text to this end, including a new subsection in the Methods (Tracing stochastic variability within representations). Simply via multiple (valid) forward passes, we can trace the origin of variation at different locations (Fig. 3) and probe the structure across streams (Fig. 4); analogous analyses would require interventions or manufactured ensembles for deterministic nets.
Second, faithfulness was a central issue in the submission (W2, Q2). How much do insights about the stochastic models matter if the degree of rewiring is unknown? To this end, we introduce a new Fig. 2 and Sec. 3.1 measuring the component-level faithfulness of the spectrum of GPT-2 models. We ablated each component of each model and measured the change in log probabilities across ~30,000 tokens. The correlation of these changes quantifies the degree to which the components have matching causal effects, and we find a smooth degradation as the total information rate decreases. Please take a look at the new Fig. 2 if you have a chance!
Remaining questions:
Q3, we agree that quantization and stochastic normalization share the property of imposing finite resolution on representations. Deterministic quantization partitions representation space into discrete regions, within which states become indistinguishable to downstream computation. Stochastic normalization instead induces lossiness through overlapping distributions, providing a graded notion of distinguishability without hard boundaries between regions. Moreover, the stochastic construction affords a second avenue of analysis: variability within a representation serves as a tracer of computation, whose propagation and mixing can be tracked across depth and position (something we have emphasized more heavily in the revision). Stochastic quantization could potentially support similar analyses.
Our objective also differs from that of conventional quantization: rather than reducing the computational or memory costs of model execution, we seek to constrain and measure the information accessible to individual components for purposes of interpretability of the processing. The normalization geometry enables us to parameterize these constraints through continuously adjustable channel rates and optimize their allocation across the network.
Q4, yes the BC is a monotonic function of the cosine similarity of the vectors before sampling. We have added text about this in the methods: Note that BC$(p_i,p_j)$ is a monotonic function of the cosine similarity between normalized states. Unlike cosine similarity, however, it incorporates the learned resolution of the read, allowing comparisons across channels with different noise scales and supplying an operational interpretation in terms of downstream distinguishability. Importantly, the residual stream reorganizes such that this relationship holds; there is no explicit incentive to organize in this way in the deterministic model.
Q5 Demonstration of the core mechanism in a simpler system is a nice idea and seems like it might connect to the presentation issues raised in W3. Our express goal for this method is to port into much larger language models (if interested, please see our new results on Gemma 3 1B in Figs. 1 and 3). That does not preclude a more careful analysis in the other direction, of the interplay between the information restriction and the hyperspherical geometry in simpler settings, something we leave to future work.