- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper investigates what parts of a transformer’s internal state in the residual stream matter for its outputs. To study this, the authors compress the residual stream, dropping the subspaces that the model does not use for the output, while ensuring that the output remains reproducible. The authors then show that the studied compression function is as strong as it can be, as any changes to the residual stream can be identified from the probability ratios of two tokens. This allows them to determine the minimum state size of the model and lossy compression error below that size.
Critical strengths
- While I am not able to verify them, the statements seem rigorous and well-backed
- The high-level aim of the paper, understanding the fundamentals of transformer interpretability, is very relevant and a topic that warrants theoretical treatment
Critical weaknesses
- The paper is overwhelmingly dense. Admittedly, I am not an expert on the topics covered, but I struggled a lot when trying to understand even the high-level ideas of the contributions. This was my main issue with the paper, as it made it, despite the interesting topic, hard to read and follow.
- The writing is, as perhaps expected for a theoretical contribution, very dry and to the point, but particularly because of my lack of familiarity with the field, I found the very direct exposition hard to follow. For example, many claims and crucial parts appear without prior justification (e.g., Assumption 2 begins Section 4 with no context given).
- Similarly, many concepts are used without explanation (again, it might be clear to someone more familiar with the topics), for example (among others), “native execution”, “cache-write suppression”, “probe reachability”, etc.
- Altogether, this led to me struggling to understand what the different claims and results in the paper are trying to achieve and, most importantly, how they contribute to the main goal of understanding the interpretability of transformers. Experiments seem to aim to apply the derived results to real-world models, but, similarly, it was difficult to discern what they contribute.
- To the best of my understanding, the strict requirements lead to the conclusion that everything in the model matters (i.e., the compressibility is minimal); experiments confirm this in actual models. Thus, I am not sure about the usefulness of the techniques.
Questions for authors
- The introduction starts by motivating the work by trying to understand the computation of a transformer. What do the results imply about the steerability of transformer LMs? What would be the implications for, e.g., mechanistic interpretability, if any?