acceptodds
Under review as a conference paper at ICLR 2027

The Dimensional Cost of Context in Language Models

Abstract

Low-rank structure in language-model activations is often taken as evidence that the residual stream can be narrowed. However, a subspace that works for one prediction may not work for another. We define the context requirement dimension as the smallest rank of a (linear) map that preserves the model's output distributions within a fixed tolerance. We show that a prediction may depend on few directions, but these directions vary across contexts and together cover most of the residual width. At the output, preserving the distributions within our tolerance requires roughly 67–83% of the residual width across 21 models from nine families. The requirement is lower inside the network but generally increases towards the output. During generation, narrowing the stream shifts output towards frequent tokens. Therefore, local low dimensionality gives a misleading estimate of the width needed across a corpus.

Primary area:
foundation or frontier models, including LLMs
Keywords:
representation dimensionality, low-rank compression, contextual prediction, information geometry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W3ZiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This study shows experimentally that although each prediction of prefix depends on only a few directions, preserving output distributions across many prefixes requires a much broader directions. The authors keep the model weights fixed, insert the same low-rank reconstruction at the final state or an intermediate layer for every prefix, and measure the KL divergence from the original model's output distribution. They also find that compression causes larger KL divergence for prefixes followed by rare tokens and shifts generation to frequent tokens.

Critical strengths

The idea is clear, the paper is easy to read, and the experimental setting and results support its claims. The theoretical claim in this paper is also insightful.

Critical weaknesses
  • Regarding the impact of the contribution: As noted in Related Work, Gurnee et al. (2026) showed that different directions are active at different positions, and that these directions together span a broad space. Although this paper measures a different quantity, what new insight do its results provide compared with that earlier work? The conclusion that a broad shared basis is needed seems predictable from the earlier findings.
Questions for authors
  • The authors argue that preserving predictions across prefixes requires 67–83% of the dimensions, which they describe as high. High compared with what, and by how much? One could also say that the rank can be reduced by 17–33% even across prefixes.
  • This paper argues that a shared linear reconstruction struggles to preserve the KL divergence of the output distributions across prefixes because different directions matter for different prefixes. Could the authors explain in more detail how this negative result can inform the design and evaluation of LLMs?
  • Figure2b suggests that intermediate states are easier to compress in the middle of the network than in later layers. However, the paper only tests layers from 1/4 depth onward. Compressing at earlier layers would remove information about the prefix itself, so I would expect the effect of compression to be more harmful. Could compressibility therefore vary nonmonotonically across layers? (I am not necessarily asking for another experiment. I mean this as a possible suggestion for improving the paper or for future work.)
1 comment

First, thank you for your time and helpful review.

Although this paper measures a different quantity, what new insight do its results provide compared with that earlier work? The conclusion that a broad shared basis is needed seems predictable from the earlier findings.

We agree that Gurnee et al. (2026) already showed that narrow local subspaces can together span a broad space. Our revision adds two findings.

First, a shared map can fail even when its subspace contains accurate reconstructions. In Appendix E.3, we keep a half-width subspace fixed and meet the KL tolerance by fitting coordinates separately for each prefix, where shared encoders fail.

Second, a rough prediction can identify which additional directions to use. A shared map followed by a prefix-specific correction meets the same tolerance with 23–25% fewer linear projections than fitted fixed maps, counting both stages (Section 4.3 and Figure 4). The correction still needs the original state, so this does not yet reduce model width.

The lesson is that adding dimensions is not the only response to a failed reconstruction. We can also improve how those dimensions are used or select additional directions for each prefix. We have clarified this distinction in the introduction, Related Work and Section 6.

The authors argue that preserving predictions across prefixes requires 67–83% of the dimensions, which they describe as high. High compared with what, and by how much? One could also say that the rank can be reduced by 17–33% even across prefixes.

We mean "high" compared with the few directions that matter around one prediction. Using the same 90%-Fisher-trace criterion, the median rises from 1.5% of the width for one prefix to 55.7% across the corpus sample. These sensitivity measures differ from KL-based reconstruction ranks, so the comparison does not establish a compression bound. The fitted ranks also depend on training. In the revision, we increase from 800 to 4,900 steps, with about 49× more sampled examples which lowered the passing range from 67–83% to 58–80% of width.

What we show is that concentrated sensitivity around individual predictions does not imply that one equally narrow reconstruction can preserve predictions across prefixes, and conversely, it is not because indiviually narrow reconstruction spans a large space, that we cannot exploit better the space for representation.

Could the authors explain in more detail how this negative result can inform the design and evaluation of LLMs?

First, a failed reconstruction does not always mean we need more dimensions. In Appendix E.3, choosing coordinates separately for each prefix meets the KL tolerance in a half-width subspace where the shared map fails. The space is sufficient, but the map does not use it well.

Second, a rough prediction can tell us which additional directions to use. Our two-stage method identifies likely next tokens, then uses the original state to correct their relative probabilities. It meets the same KL tolerance with 23–25% fewer linear projections, including the correction. Training the map for the corrected output improves the result, even when its initial prediction becomes worse. The first stage should therefore be trained for its role in the complete procedure (Section 4.3).

Third, check how generation changes, not just the average error. Low-rank reconstruction favors frequent tokens, while noise matched to its average KL on held-out text favors infrequent tokens. KL alone misses this difference, so evaluation should also examine which tokens the model generates more or less often (Section 5.2 and Appendix F.1).

These results suggest testing better ways to use the retained dimensions before adding more. At the moment, our correction still needs the original full-width state, so it does not yet make the model narrower. Another aspect not explored yet in the manuscript is to guide distillation (do we really need to increase studen's width?) or to detect and correct some bias that are invisible to KL (the shift toward frequent tokens).

Could compressibility therefore vary nonmonotonically across layers?

We added a test at 1/8th depth, using PCA reconstruction at half width. In most models, KL is lowest at one-quarter or one-half depth and higher both earlier and later in the network. This supports your suggestion that intermediate layers can tolerate rank reduction better than earlier and later layers. Our results show a related depth dependence when reducing the rank of states rather than removing layers (Men et al., 2025) We now state this finding in Section 3.3 and report the additional check in Appendix D.1.

This is actually what we plan for future work. Right now, the two-stage method is limited by the fact we need the full state state of the current layer, but if could somehow use the previous state rather, this might open the door to layer skip using the low-rank map + deterministic correction.

Reviewer_sXGNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors have probed the minimum dimensions of individual layer that can preserve the language models’ functions and used 4 dimension reduction algorithms, PCA, output-Fisher-weighted PCA and 2 variant KL-trained reconstructions to reduce the dimensions of the modules on layer-by-layer basis. They found that a large part of width is necessary to reconstruct the original outputs faithfully.

Critical strengths

The authors address an interesting topic, the low-dimensional structure of activation space observed in language models (LMs), using a simple but effective method. Furthermore, they evaluate the correlations between LM layers’ effective dimensions and contexts and find that directions are sensitive to contexts. The results, which are not surprising or ground-breaking, can be of interest to those who study in low-dimensional subspace of activation space in LMs and guide future studies.

Critical weaknesses

Their analyses suggest that a large part of width is necessary to reconstruct the original outputs faithfully, which seems to be conflicting with the earlier studies (e.g., linear representation hypothesis). The authors suggest that these conflicting results may be reconciled by the fact that the earlier studies studied activation space associated with a single prefix only. To address this issue, they probed the low-dimensional subspaces depending on prefixes, but they did not explain how they define each prefix here. In lines 309-310, the authors mentioned 1-64 token types and 4096 corpus prefixes. How did the authors define types 64 token types and 4096 prefixes?

Questions for authors

Did the authors use contexts and prefixes interchangeably in this manuscript? Or does the meaning of contexts differ from prefixes in their study?

2 comments

First of all, thank you for your time the valuable comments.

Their analyses suggest that a large part of width is necessary to reconstruct the original outputs faithfully, which seems to be conflicting with the earlier studies (e.g., linear representation hypothesis). The authors suggest that these conflicting results may be reconciled by the fact that the earlier studies studied activation space associated with a single prefix only. To address this issue, they probed the low-dimensional subspaces depending on prefixes, but they did not explain how they define each prefix here.

There are three different claims to separate:

  1. linear representation hypothesis which tells about individual features and concepts but tells nothing about how many feature direction the model uses at inference
  2. how many fixed directions are needed to preserve the original distribution at inference time
  3. can we lower than number of fixed direction by a small prefix-dependent correction (for the same KL tolerance on the predictions)

The linear representation hypothesis concerns the first question. This paper is about 2. and 3. which is perfectly compatible mathematically with 1.: information can be represented linearly without being recoverable through one small, fixed linear reconstruction.

During this revision, we have added experiments about point 3. (new Section 4.3 and Appendix E.3) to precisely distinguish two reasons for that reconstruction failure: insufficient dimensions or an ineffective use of the retained dimensions.

To address this issue, they probed the low-dimensional subspaces depending on prefixes, but they did not explain how they define each prefix here. In lines 309-310, the authors mentioned 1-64 token types and 4096 corpus prefixes. How did the authors define types 64 token types and 4096 prefixes?

Thank you for pointing out this ambiguity. A prefix is the text from the start of a sampled sequence up to the position immediately before the next token. A token type means a distinct token in the model's vocabulary, rather than an individual occurrence. "1-64 token types" refers to the number of distinct observed next tokens represented in a group of prefixes. It does not describe prefix length. We keep the number of prefixes fixed and sample equally many per token type.

Example of groups with 2 prefixes instead of 256: Group A with the same continuation token "cat": "The animal purring on the sofa is a" -> cat "She filled the food bowl for her" -> cat

Another group of prefixes with different continuation: "The animal purring on the sofa is a" -> cat "The puppy grew into a large" -> dog

We average the Fisher matrices in each group and compare how many directions contain 90% of the sensitivity. Does including different next tokens require more directions, even though the number of prefixes stays unchanged? And the answer we found is: yes, knowing the next token does not identify one small subspace that serves all its occurrences. And, less surprising: including different next tokens broadens this space further.

"4096 corpus prefixes" refers to a separate random sample of prefixes, without restricting their next-token identities. We sample these without replacement from 20 480 WikiText-103 positions. This comparison tests whether sensitive directions vary even when the next token stays the same, and how much further they spread when different next tokens are included. We have clarified the definitions in Section 4.2 and the sampling procedure in Appendix E.2. and we will release the code of the experiments after the review process.

Did the authors use contexts and prefixes interchangeably in this manuscript? Or does the meaning of contexts differ from prefixes in their study?

Yes, we did. We have changed that throughout the manuscript to use only prefix for clarity, while retaining "context requirement dimension" as the name of our measurement.

During this revision, we added a new experiment (Section 4.3) where we test training a shared map followed by a prefix-specific correction, showing this lower the overal rank for the shared map. We have retrained the maps at larger scale (4900 steps vs 800, 512 states samples per step vs 64) with about 49x more sampled training examples which is why we found slightly lower passing rank (58-80% vs 67-83%). We've also reworked the introduction and the conclusion to present more what is at stakes and how this work positions itself w.r.t. the existing literature. Other sections received only cosmetic updates or additional control and information on the protocol.

The authors have provided more explanation and addressed my concerns.

Reviewer_mugDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper concerns the intrinsic dimensionality of language models. It asks whether the observation that many LM behaviors can be explained by low rank approximations implies that the entire LM behavior can be approximated with a low rank projection. The answer is basically, no. They define some metrics for preserving KL within some tolerance, and run experiments across many models using next work prediction on wikitext. They find that by and large one can only prune 20-40% of the dimensionality at best to preserve model outputs.

Critical strengths

The paper tests a lot of models so the results are robust. Questions about the effect rank of AI models are interesting and timely.

Critical weaknesses

I’m trying to figure out what exactly we learn from this study. Perhaps I am just missing something/not the intended audience. But from my understanding, the results can basically be summarized as: more dimensions are required to fix more components of the dataset. Some of the main findings are, for example, that different prefixes rely on different subspaces to predict the same next word, and that more dimensions are needed to output rarer words – I would have been shocked if this were not the case! This seems like the bread-and-butter of how dimensionality reduction works, whether in an LLM or any other model.

When I read the title/abstract I assumed the paper would address a more live question about the intrinsic dimensionality of tasks that the model is being deployed to perform. LoRA finetuning, for example, is the norm. It might be interesting to ask how LoRA dimensions across related tasks are shared. Intrinsic dimensionality is also an active topic in interpretability, for example in work on SAEs. There might be interesting implications to this line of work that relate to our ability to understand/interpret models in general. But as written, when the work is just about how many dimensions are required to fit the language modeling task, I’m struggling to see what the significance of the findings is.

Perhaps I am missing something, but if that is the case, the paper intro/discussion could be substantially rewritten to make clear what the open questions are and how this work contributes. The current related work mostly reads as a list of papers and one line summaries (feels very AI generated) and doesn’t do a great job situating the work. For publication, the paper should really explain where the knowledge gap is, why it matters that we fill it (“what is at stake”) and then proceed to connect the experiments to that bigger-picture framing.

Questions for authors

see weaknesses

1 comment

Thank you for your time and helpful review!

This seems like the bread-and-butter of how dimensionality reduction works, whether in an LLM or any other model.

We agree that different prefixes relying on different directions is not, by itself, surprising. Our original version did not explain clearly enough what we learn beyond that observation.

The question is what to conclude when a low-rank reconstruction fails. Do we need more dimensions, or could we use the retained dimensions better? Preserving most activation variance does not settle this question, because small changes in the state can still affect the next-token prediction.

The revision adds two experiments.

First, in Appendix E.3, we keep a half-width subspace fixed and fit coordinates separately for each evaluation prefix. These reconstructions meet the mean-KL tolerance where learned shared encoders fail. The subspace contains accurate reconstructions, but the shared maps do not find them.

Second, in Section 4.3, a rough prediction identifies likely next tokens, and a correction uses the original state to recover their relative probabilities. This meets the same tolerance with 23–25% fewer linear projections than fitted fixed maps, counting both stages. Training the map for the corrected prediction improves the result even when its initial prediction becomes less accurate. The correction still needs the original state, so this is not yet usable to narrow a model but it can at least be used as a diagnostic at the mometn. The finding is that adding fixed directions is not the only way to improve prediction accuracy. We can also improve how coordinates are chosen or select additional directions for the current prefix.

It might be interesting to ask how LoRA dimensions across related tasks are shared. Intrinsic dimensionality is also an active topic in interpretability, for example in work on SAEs.

Originally, our focus was the geometry of the trained model itself, rather than how fine-tuning changes it. To put it in perspective, the questions we tried to answer are:

  1. How many fixed state directions are needed to preserve predictions across prefixes?
  2. Can choosing additional directions for each prefix preserve predictions with fewer projections?

With LoRA, the corresponding questions would rather be:

  1. how many weight-update directions are needed to adapt the model to a task?
  2. When adapting the same model to different tasks, do the updates use overlapping directions or different ones?

We don't think the connection is direct: adapting a model may require a small update even if preserving its existing computation requires many state directions.

The connection to SAEs seems closer. Our fixed-subspace experiment also suggests a diagnostic for SAE reconstruction errors: keep the dictionary and sparsity budget fixed, then test whether better feature activations preserve predictions more accurately. This could help distinguish limitations of the dictionary from the limitations of the encoder. We have not tested this application yet.

For evaluation, our generation experiment provides a concrete diagnostic tool. Low-rank reconstruction and noise matched to its average KL on held-out text shift token frequencies in opposite directions. When an approximation replaces activations, average KL alone does not describe how generation changes. This can be helpful notably during distillation to check changes in generated-token frequencies alongside average KL.

For publication, the paper should really explain where the knowledge gap is, why it matters that we fill it (“what is at stake”) and then proceed to connect the experiments to that bigger-picture framing.

Definitely agree. We have revised the abstract, Introduction, Related Work and Discussion around the question above: when reconstruction fails, should we add dimensions or change how they are used? Section 4.3 and Figure 4 introduce the prefix-specific correction. Appendices E.3–E.5 report the fixed-subspace diagnostic, error analysis and adaptive comparisons. Section 3.2 and Appendix C give the updated fitting results and protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.