acceptodds
Under review as a conference paper at ICLR 2027

Tracing Information Processing in Transformers with Stochastic Normalization

Abstract

Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied to downstream function: nearby states may produce different behaviors, while distant states may behave similarly. We instead give representations volume, turning similarity into statistical distinguishability. Overlapping stochastic representations necessarily induce overlapping downstream distributions, grounding latent comparison in model function and bringing it under information-theoretic tools such as the data-processing inequality. We realize this idea in pretrained transformers through a light-touch modification to normalization: at each residual-stream read, normalize the state, add isotropic Gaussian noise, and renormalize. During distillation fine-tuning, one learned allocation parameter per read distributes a fixed global rate budget across the processing stack. The resulting model can be viewed as transformer blocks reading the residual stream with learned finite precision. This stochastic construction supports complementary views of transformer computation: tracing which input distinctions remain accessible to downstream components, and tracking how variability introduced at different locations propagates and mixes through the network. Across vision and language transformers, these views reveal structured, component-specific patterns of information preservation and processing. Together, our results establish finite-precision stochastic representations as a functionally grounded lens on transformer computation.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, information theory, representational similarity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1KbwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that distances between representations are not intrinsically tied to downstream functionality, and argues for a principled notion of statistical indistinguishability. The authors do so by giving representations volumes, instead of simply considering them to be points, and they do so with noise jittering, and justify this choice with the geometry induced by the LayerNorm.

The authors test this idea on transformers by distilling models into their "stochastic LayerNorm" version in both vision and language. They show how their method can localize the layer/attention head responsible for distinguishing progressively corrupted inputs in the case of vision, and two 'similar' sentences in the case of language.

Critical strengths
  1. The identified problem is interesting: tying representations to function is a principled way to study representational similarity. Furthermore, considering volumes instead of points could be particularly interesting to assess the sensitivity of interpretability tools (such as probes) to noise.
  2. The method is well grounded and mathematically sensible.
  3. The method is lightweight and simple to implement, making it potentially widely applicable.
Critical weaknesses

My main problem with this paper is its evaluation.

  1. It identifies a problem: "distances between point-valued representations are not intrinsically tied to down-stream function", but it never shows examples of this actually being an issue in practice. I think that the paper would be greatly strengthened if the authors could (i) show occurrences of this problem with standard point-wise metrics such as L2 distance or cosine similarity, and (ii) show how the proposed method fixes it.
  2. As acknowledged by the authors, all statements made about the distilled noisy models are not guaranteed to transfer to the teacher model. The paper would benefit from an interventional study on both the teacher and the student models. For example, if the method identifies a particular architectural component to be responsible for the separation of two inputs, can an intervention verify this causally in the distilled model? If the same intervention also cleanly transfers to the teacher, this would provide evidence that the analysis conducted on the student model also transfers to the teacher model.
  3. The narrative is a bit hard to follow, and I had to read the paper several times to understand what is the research gap the authors are addressing. I think that this could be better exposed by incorporating the first point above and by making clearer what each experiment is intended to show and how it fits the narrative that representations should have volume.
Questions for authors
  1. Can the authors provide examples where standard point-wise metrics fail while this framework succeeds at appropriately measuring representational similarity?
  2. Can the authors provide an interventional study to assess the framework's validity both within the student and teacher models?
  3. Can the authors clarify the relationship between their method and quantization? They seem to be quite related as they both convert/construct lower precision models.
  4. Looking at the equation for BC (lines 200-206), if κ\kappa is fixed, BC only depends on the cosine similarity between the means μi,μj\mu_i,\mu_j and their cosine similarity. Can the authors discuss how their method relates to first do noise jittering and then compute the cosine similarities of the means?
  5. Transformers are quite complicated architectures, but from my understanding this method is applicable to any architecture with LayerNorm, including simple MLPs. Do the authors think that evaluating the framework on simpler architectures could highlight its features and potentially reduce confounding factors coming from architectural specifics?
1 comment

We thank the reviewer for the helpful feedback, which directly led to changes in the revision. The revision centers around reframing and two new key results.

First, we had placed too much emphasis on the distinguishability of a pair of inputs in the submission. We think that the analyses on a single processing stream provide clearer novelty compared to the deterministic geometry scenario (W1), and have reworked several parts of the text to this end, including a new subsection in the Methods (Tracing stochastic variability within representations). Simply via multiple (valid) forward passes, we can trace the origin of variation at different locations (Fig. 3) and probe the structure across streams (Fig. 4); analogous analyses would require interventions or manufactured ensembles for deterministic nets.

Second, faithfulness was a central issue in the submission (W2, Q2). How much do insights about the stochastic models matter if the degree of rewiring is unknown? To this end, we introduce a new Fig. 2 and Sec. 3.1 measuring the component-level faithfulness of the spectrum of GPT-2 models. We ablated each component of each model and measured the change in log probabilities across ~30,000 tokens. The correlation of these changes quantifies the degree to which the components have matching causal effects, and we find a smooth degradation as the total information rate decreases. Please take a look at the new Fig. 2 if you have a chance!

Remaining questions:

Q3, we agree that quantization and stochastic normalization share the property of imposing finite resolution on representations. Deterministic quantization partitions representation space into discrete regions, within which states become indistinguishable to downstream computation. Stochastic normalization instead induces lossiness through overlapping distributions, providing a graded notion of distinguishability without hard boundaries between regions. Moreover, the stochastic construction affords a second avenue of analysis: variability within a representation serves as a tracer of computation, whose propagation and mixing can be tracked across depth and position (something we have emphasized more heavily in the revision). Stochastic quantization could potentially support similar analyses.

Our objective also differs from that of conventional quantization: rather than reducing the computational or memory costs of model execution, we seek to constrain and measure the information accessible to individual components for purposes of interpretability of the processing. The normalization geometry enables us to parameterize these constraints through continuously adjustable channel rates and optimize their allocation across the network.

Q4, yes the BC is a monotonic function of the cosine similarity of the vectors before sampling. We have added text about this in the methods: Note that BC$(p_i,p_j)$ is a monotonic function of the cosine similarity between normalized states. Unlike cosine similarity, however, it incorporates the learned resolution of the read, allowing comparisons across channels with different noise scales and supplying an operational interpretation in terms of downstream distinguishability. Importantly, the residual stream reorganizes such that this relationship holds; there is no explicit incentive to organize in this way in the deterministic model.

Q5 Demonstration of the core mechanism in a simpler system is a nice idea and seems like it might connect to the presentation issues raised in W3. Our express goal for this method is to port into much larger language models (if interested, please see our new results on Gemma 3 1B in Figs. 1 and 3). That does not preclude a more careful analysis in the other direction, of the interplay between the information restriction and the hyperspherical geometry in simpler settings, something we leave to future work.

Reviewer_qnZKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Most model interpretability and analysis tools that operate on model hidden states, do so in a pointwise manner. This can be misleading because natural distances between representations such as Euclidean distance or cosine similarity, may be only loosely related to differences in downstream model behavior that these states correspond to. To address this, this paper proposes a new approach to studying transformer internals. This is done by adding a notion of volume to representations via points in blocks where noise is injected into the information flow. After defining the method, the paper describes ways that this new architecture can be analyzed. Specifically, because of the stochastic feature of the new architecture, representations can be treated as distributions rather than individual points. The remainder of the paper is a description of various experiments that are run using this tool. These involve GPT2 and a ViT. Examples include an experiment that generates response curves for different types and severities of augmented input to a ViT. The results show how representations of an image and its augmented version become increasingly distinguishable with depth. Another experiment looks at distinguishability at the level of attention heads in a GPT2 model. The paper connects the results with heads that are known to play specific roles in GPT2 (e.g., induction heads). The paper ends by discussing impacts and limitations.

Critical strengths
  • The paper tries to address a real challenge in model interpretability: Spanning the divide between the geometric properties of internal representations and their resulting behavior is a continuing issue in interpretability. The approach proposed by this paper is interesting and principled. One could imagine it being expanded substantially to either incorporate more theoretical analysis or to use this tool on address existing questions in the field.
Critical weaknesses
  • The tools largely confirm existing understanding rather than revealing new insights: There are now a vast number of analytic tools that have been proposed to better understand transformers. At best, most of these only confirm what we already know. There is nothing wrong with this, but it does limit their impact. The approach proposed in this paper does seem to elevate certain aspects of transformer information processing that may be inaccessible to other methods, however, the paper does not provide examples that support this. In addition, it was unclear to this reviewer how we should use several of the insights (e.g., augmented input for ViTs). Perhaps providing additional discussion of the results, the ‘so what?’, would be helpful.
  • The method requires training a surrogate model: The biggest downside to this method is that it requires post hoc training of a surrogate. This will likely limit use relative to other tools that can be applied without training. It is also unclear how well this method can scale. This is important since many of the most interesting transformers are quite large relative to ViTs and GPT2s. Finally, because a surrogate model is being analyzed, there is always a question of whether the behavior we see might not be a consequence of the additional training. To the reviewer’s memory, this was not addressed in the work.
Questions for authors
  • The paper may have addressed it, but what happens if these analyses are performed without the finetuning? How much does the finetuning change the model?
  • Is ‘tap’ standard terminology in the transformer literature? This reviewer does not remember seeing it used before.
  • On line 243, what does ‘protected’ mean?
  • Can the authors provide further explanation of the experiment analyzed on line 257-269?
1 comment

We are grateful for the feedback. We focused our revision efforts on highlighting new insights (W1) and the scalability+faithfulness of the surrogate models (W2).

Re new insights (W1): we think that the new insights are easiest to see in the single-stream analyses of the current Figs. 3 and 4. Our original writing emphasized the distinguishability of different inputs, but the newly available insights are most apparent in what can be done by simply re-sampling forward passes of a single input. We’ve added a Gemma 3 1B series in response to scaling concerns (as you raised in W2), and in the updated Fig. 3 we compare Gemma and GPT2 in terms of where variation about earlier repetition enters the processing. This sort of analysis requires no intervention nor contrasting ensemble -- and it’s all because the model was fine tuned to work with voluminous representations. We have reframed portions of the text to make this more clear, for instance by balancing the methods section with a Tracing stochastic variability within representations part on the variance decomposition.

Re scaling (W2): we agree that the method is more costly than post hoc analyses, but we feel that the benefits strongly outweigh the costs. The fine tuning is cheap compared to other methods with more significant architectural modifications (eg information bottlenecks), and, once finished, the models are easily shared since they have no extra modules beyond a modified LayerNorm/RMSNorm. We anticipate the spectra of models parameterized by information usage will be fruitful for comparative analyses similar to those done on the Pythia family (as a function of training step).

Re faithfulness (W2): Thank you for raising this. We have added a new faithfulness measurement based on ablating components in the student and teacher models and then evaluating the average correlation of changes to the output distribution. We realized that the insights related to the stochastic models need firmer grounding in the teacher behavior, and think that the new results are important enough to be at the front of the experiments section (Sec. 3.1, Fig. 2). Please take a look if you have a chance.

Specific questions: 1) Noise without fine-tuning: the model breaks quickly! It has no reason to be robust to the injected noise, so we end up probing OOD behavior. The finetuning involves serious restructuring to the residual stream. We are working on a characterizastion for the appendix but wanted to get the current revision out with some time before the end of the discussion period. 2) Thank you, we have replaced “tap” with “read”. 3) “Consistently, the final readout was the most protected.” --> by protected, we mean that the noise level was kept low. The optimization allocates noise to every tap/read, and it seemed to protect the final readout.
4) Expt on L257-269: Let’s say you ran the same input sequence of text through one of the stochastic GPT2s 10 times. Each time there would be a different path, sampled from the stochasticity at each read, and they should all be roughly equivalent computations. How much structure is there in a single path? We broke that structure by sampling representations at a given depth from different ~equivalent paths, effectively sampling states from the marginal distributions at each read instead of the joint distribution across reads. This is a probe of the structure in the processing that has no counterpart in the deterministic models.

Reviewer_P4FQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper examines informational distance in the residual stream of a transformer, drawing a contrast to geometric distance. It implements this by injecting noise into the layerNorm step, which calibrates the noise to a small region on a hypersphere around the original point. With this noise, the Bhattacharyya overlap, rather than the geometric distance, evaluates the proximity of internal states in the residual stream. This information measure benefits from the data processing inequality, which indicates that if two points can't be distinguished, their downstream effects are also indistinguishable. Additionally, the randomness can be used as a tracer dye for the flow of information in the neural network using the covariance of noise.

The paper applies these techniques to two models: a vision transformer (ViT-S) and GPT-2 small. In the spirit of the information bottleneck, they examine how performance and precision trade off by training a twin to replicate the target transformers but constraining the information in the layerNorms. Below 10^4 nats of information, they find that the performance falls off. For the vision model, the informational distance is most sensitive to blur and least sensitive to contrast fading. Swapping out a noun in GPT-2 identifies seven heads, six of which are already known to be involved in induction. This indicates that the informational distance recovers known structure.

Critical strengths
  1. The paper is well motivated in recognizing the limitations of geometric distance measures and proposing informational distance measures as an alternative. This is an interesting idea that deserves exploration.

  2. Using noise as a tracer dye is clever, and I'm interested to see how this technique can be used to determine the logical architecture of the transformer.

Critical weaknesses
  1. While the paper is trying to advocate for the advantage of informational distance over geometric distance, it fails to make that comparison numerically. A straightforward test would be to compare the Bhattacharyya overlap to the Euclidean distance and see which features each identifies. Since the paper does not make this comparison, it doesn't effectively argue for information over geometry. It simply shows that information can reproduce some known results.

  2. The object of study is not the original model, but a twin whose weights have undergone a new learning procedure to optimize its performance subject to informational constraints. Therefore, it seems that the twin could have rewired its function in this training process and may have a different structure than the original.

Questions for authors
  1. What is the mathematical relationship between the Bhattacharyya distance and Euclidean distance when noise is added to the layerNorm? This seems like it would have a clean functional relationship, based on the regular structure of the noise, and could give cleaner insight into how the geometric and informational pictures coincide.

  2. Is there a result in the paper that the tool (the informational distance) is needed to obtain? It's not clear to me that geometric distance can't do the work.

2 comments

We thank the reviewer for the incredibly helpful feedback and appreciate the suggestive analogy to tracer dye.

Our revision is focused on a reframing and two sets of additional results, in direct response to points raised by all reviewers: a Gemma 3 1B series of stochastic models and a new faithfulness measurement across the GPT2 spectrum of models.

Re W1 and Q2: we realized our framing around representational similarity had been too narrow -- focused on pairs of inputs where the distinction from geometry is smaller. (Indeed, in the stochastic models, Bhattacharyya distance is a monotonic function of cosine similarity -- good catch in Q1).

A broader view includes the analyses of a single processing stream as distinct paths with similar functionality engrained during fine-tuning. This better accounts for the “tracer dye” variance decomposition (Fig 3). By simply running a bunch of forward passes on the same input and then decomposing the variance, we measure sensitivity between loci in the processing. If you have bandwidth, please look at the new Gemma results in the updated figure 3: the decomposition shows the retrieved information entering strongly at the global attention layers (L5, L11, L17). The two models are readily compared in the same units; a comparable geometric analysis on their deterministic counterparts would have more complicating factors such as OOD interventions or arbitrary ensembles.

Re W2, the faithfulness concern is important. As the information budget decreases, the faithfulness must degrade; does it do so gracefully or all at once? The components may or may not remain faithful because the distillation loss only incorporated model outputs. We mean-ablated each component in the stochastic and deterministic models and measured the correlation of changes to the log probability (following 2025 Anthropic work measuring faithfulness of replacement models). Intuitively, if L5H0 has the same effect in both models, ablating it should cause highly correlated changes in the output distributions. We find that the overall faithfulness degrades gracefully across the noise spectrum and is also heterogeneous in the allocation across components, and have added a new figure with these results. Faithfulness to the deterministic model is centrally important, so it's now Fig. 2 with a dedicated Section 3.1 -- please give a look if possible.

All told, we think that the reframing, the new Gemma series, and the faithfulness measurements make the paper stronger, and we appreciate the constructive criticism.

Reviewer_P4FQOpenReview ↗

Reply to authors

I would like to thank the authors for the detailed reply and the work that went into the revision. The new faithfulness experiment is exactly the kind of test I was hoping for, and I appreciate that you show where it breaks down. My worry is what it shows: agreement with the original model fades steadily as the noise grows, and the paper doesn't say where on that curve the main analysis sits. My reading is that it's in the middle, where the twin only loosely tracks the original. If so, my central question is still open: are we learning about the model we care about, or about a retrained cousin of it? The test also covers only GPT-2, not Gemma, and it checks each component's overall effect on the output, not whether information is routed between positions the same way, which is what your tracer and head-selectivity results are about.

On the comparison to geometry, I read the authors' reply as agreeing with me for the pairwise results: at a given read, the informational distance and cosine similarity order every pair the same way, so the informational view adds nothing there. The direct comparison I asked for is still missing. If a purely geometric measure on the original model picks out the same heads, the stochastic machinery isn't doing the work. I agree that using noise as a tracer is the more original idea, with no real counterpart in a deterministic model. But in GPT-2 it confirms what we already knew, and the Gemma finding, the genuinely new one, has no causal check. The paper is also still split between the old framing and the new one.

I'm raising soundness from 1 to 2, since faithfulness is now measured instead of assumed, and keeping my overall rating.

Comments on the paper

1 comment

The feedback from all reviewers was thoughtful and constructive. There was broad consensus about the strengths and weaknesses, which was helpful. Our revision focused on three main changes, addressing reviewer concerns:

  • A new evaluation of component-wise faithfulness between stochastic and deterministic models (Sec. 3.1, Fig. 2)
  • A Gemma 3 1B series of stochastic models, with new results in Fig. 1 and Fig. 3.
  • A re-balancing of the paper’s focus on distinguishability between different inputs’ representations, and the spread of representations within a single representation

In case it’s helpful, we’ve included a diff.pdf in the supp material (since we’re not currently able to access diff pdfs through openreview). Below we respond to your individual reviews, and would welcome any further discussion in the remaining ~40 hours.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.