- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This study shows experimentally that although each prediction of prefix depends on only a few directions, preserving output distributions across many prefixes requires a much broader directions. The authors keep the model weights fixed, insert the same low-rank reconstruction at the final state or an intermediate layer for every prefix, and measure the KL divergence from the original model's output distribution. They also find that compression causes larger KL divergence for prefixes followed by rare tokens and shifts generation to frequent tokens.
Critical strengths
The idea is clear, the paper is easy to read, and the experimental setting and results support its claims. The theoretical claim in this paper is also insightful.
Critical weaknesses
- Regarding the impact of the contribution: As noted in Related Work, Gurnee et al. (2026) showed that different directions are active at different positions, and that these directions together span a broad space. Although this paper measures a different quantity, what new insight do its results provide compared with that earlier work? The conclusion that a broad shared basis is needed seems predictable from the earlier findings.
Questions for authors
- The authors argue that preserving predictions across prefixes requires 67–83% of the dimensions, which they describe as high. High compared with what, and by how much? One could also say that the rank can be reduced by 17–33% even across prefixes.
- This paper argues that a shared linear reconstruction struggles to preserve the KL divergence of the output distributions across prefixes because different directions matter for different prefixes. Could the authors explain in more detail how this negative result can inform the design and evaluation of LLMs?
- Figure2b suggests that intermediate states are easier to compress in the middle of the network than in later layers. However, the paper only tests layers from 1/4 depth onward. Compressing at earlier layers would remove information about the prefix itself, so I would expect the effect of compression to be more harmful. Could compressibility therefore vary nonmonotonically across layers? (I am not necessarily asking for another experiment. I mean this as a possible suggestion for improving the paper or for future work.)
First, thank you for your time and helpful review.
We agree that Gurnee et al. (2026) already showed that narrow local subspaces can together span a broad space. Our revision adds two findings.
First, a shared map can fail even when its subspace contains accurate reconstructions. In Appendix E.3, we keep a half-width subspace fixed and meet the KL tolerance by fitting coordinates separately for each prefix, where shared encoders fail.
Second, a rough prediction can identify which additional directions to use. A shared map followed by a prefix-specific correction meets the same tolerance with 23–25% fewer linear projections than fitted fixed maps, counting both stages (Section 4.3 and Figure 4). The correction still needs the original state, so this does not yet reduce model width.
The lesson is that adding dimensions is not the only response to a failed reconstruction. We can also improve how those dimensions are used or select additional directions for each prefix. We have clarified this distinction in the introduction, Related Work and Section 6.
We mean "high" compared with the few directions that matter around one prediction. Using the same 90%-Fisher-trace criterion, the median rises from 1.5% of the width for one prefix to 55.7% across the corpus sample. These sensitivity measures differ from KL-based reconstruction ranks, so the comparison does not establish a compression bound. The fitted ranks also depend on training. In the revision, we increase from 800 to 4,900 steps, with about 49× more sampled examples which lowered the passing range from 67–83% to 58–80% of width.
What we show is that concentrated sensitivity around individual predictions does not imply that one equally narrow reconstruction can preserve predictions across prefixes, and conversely, it is not because indiviually narrow reconstruction spans a large space, that we cannot exploit better the space for representation.
First, a failed reconstruction does not always mean we need more dimensions. In Appendix E.3, choosing coordinates separately for each prefix meets the KL tolerance in a half-width subspace where the shared map fails. The space is sufficient, but the map does not use it well.
Second, a rough prediction can tell us which additional directions to use. Our two-stage method identifies likely next tokens, then uses the original state to correct their relative probabilities. It meets the same KL tolerance with 23–25% fewer linear projections, including the correction. Training the map for the corrected output improves the result, even when its initial prediction becomes worse. The first stage should therefore be trained for its role in the complete procedure (Section 4.3).
Third, check how generation changes, not just the average error. Low-rank reconstruction favors frequent tokens, while noise matched to its average KL on held-out text favors infrequent tokens. KL alone misses this difference, so evaluation should also examine which tokens the model generates more or less often (Section 5.2 and Appendix F.1).
These results suggest testing better ways to use the retained dimensions before adding more. At the moment, our correction still needs the original full-width state, so it does not yet make the model narrower. Another aspect not explored yet in the manuscript is to guide distillation (do we really need to increase studen's width?) or to detect and correct some bias that are invisible to KL (the shift toward frequent tokens).
We added a test at 1/8th depth, using PCA reconstruction at half width. In most models, KL is lowest at one-quarter or one-half depth and higher both earlier and later in the network. This supports your suggestion that intermediate layers can tolerate rank reduction better than earlier and later layers. Our results show a related depth dependence when reducing the rank of states rather than removing layers (Men et al., 2025) We now state this finding in Section 3.3 and report the additional check in Appendix D.1.
This is actually what we plan for future work. Right now, the two-stage method is limited by the fact we need the full state state of the current layer, but if could somehow use the previous state rather, this might open the door to layer skip using the low-rank map + deterministic correction.