acceptodds
Under review as a conference paper at ICLR 2027

The Value of Memory in Language Models: Joint Limits on Language Model Expressivity

Abstract

The value of memory in a language model depends on the model that must use it. Prompt information matters only if it is both retained and usable by the model’s fixed parameters. We study this interaction through prompt accessibility, asking which outputs a frozen model can be prompted to generate. We place a causal cut between prompt processing and generation, so all prompt-derived information available afterward must pass through the retained state. This gives a joint bound in which prompt budget, retained memory, and compatible model-side constraints all limit the same accessible outputs, with the tightest applicable constraint governing. We then sharpen the memory-side bound under additional assumptions on the update dynamics and reachable-state geometry, and show how mechanism-specific structure can yield tighter bounds, using Gated Linear Attention (GLA) as a concrete example. Controlled experiments on two frozen GLA checkpoints show that increasing retained memory moves the discovered output frontier. On GLA-1.3B, increasing the retained payload from 12 to 24 to 36 MiB raises the evaluated 50% frontier from 8192 to 10240 to 12288 tokens. Controls further show that memory usefulness depends on both prompt budget and what information is retained. A same-weight sliding-window comparison reproduces the qualitative memory-frontier response, while Mamba provides complementary evidence from fixed-size recurrent memory. Together, these results support treating prompt budget, memory, and the learned model as coupled resources, motivating memory and architecture to be matched rather than scaled independently.

Primary area:
learning theory
Keywords:
language model expressivity, prompt accessibility, memory mechanisms, test-time training, long-context language models, associative memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ffyaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how prompt budget, retained memory capacity, and model expressivity jointly constrain the outputs accessible to a frozen language model. The authors introduce a causal-cut formulation and derive theoretical accessibility bounds based on memory capacity, state-update dynamics, and reachable-state geometry, with further specialization to Gated Linear Attention (GLA). Experiments on pretrained GLA models, together with supporting analyses of Sliding-Window Attention and Mamba, examine how memory size and prompt budget affect exact target-sequence recovery.

Critical strengths
  1. Interesting theoretical perspective. The paper provides a unified perspective on the relationship between prompt information, retained memory, and model expressivity. The theoretical analysis of memory dynamics and reachable-state geometry offers useful insights into the limitations of memory mechanisms.

  2. Controlled experimental investigation. The experiments systematically examine how different memory capacities and prompt budgets affect output accessibility. The additional comparisons involving Sliding-Window Attention and Mamba provide complementary evidence across different memory architectures.

Critical weaknesses
  1. Limited validation of the theoretical bounds. The theoretical analysis relies on assumptions about memory-update stability, contraction, and reachable-state geometry. However, the experiments do not sufficiently establish whether these assumptions hold in the evaluated pretrained models or how tight the derived bounds are in practice. Consequently, the connection between the theoretical results and the observed model behavior remains largely qualitative.

  2. Insufficient experimental evidence for the central joint-limit claim. The experiments mainly demonstrate that increasing retained memory can improve exact sequence recovery under optimized continuous prompts. However, they do not directly establish when model expressivity, rather than memory capacity or prompt budget, becomes the limiting factor. Furthermore, the primary GLA experiments use relatively small sets of synthetic target sequences, and the findings are not validated on practical language understanding or reasoning tasks. This limits the broader applicability of the conclusions.

Questions for authors
  1. Can the authors provide empirical evidence that the assumptions underlying the stability-based and geometry-sensitive bounds hold for the evaluated pretrained GLA models? How tight are these theoretical bounds relative to the observed accessibility frontiers?

  2. Can the authors demonstrate an experimental setting in which model expressivity, rather than retained memory capacity, becomes the dominant bottleneck? This would provide more direct validation of the proposed joint-limit framework. Additionally, how well do the findings extend to practical language modeling or reasoning tasks beyond optimized target-sequence reconstruction?

7 comments

Thank you for recognizing our joint perspective. We are glad you found it interesting. We agree deeper empirical validation would be helpful, but we were also limited by available compute.

First, we want to clarify the central goal. We want to establish joint limits and demonstrate why prompt budget, retained memory, and fixed-model capacity must be considered together. Identifying the precise trade-off in a pretrained model is important future work.

In our view, the main practical value is to recognize memory and model capacity as resources to balance when designing language models. People already do this implicitly. Most models have context-window limits reflecting practical constraints, even if these are not absolute expressivity ceilings. Meanwhile, recurrent and hybrid sliding-window or linear-attention models restrict retained memory for efficiency while preserving useful performance. The intuition behind our paper already exists and has been arrived at organically in practice, but these two perspectives have largely been treated separately. We formalize this intuition, unifying these two perspectives. We hope this motivates researchers to explicitly study the balance between memory and model capacity.

Q1. Do stability and geometry assumptions hold in pretrained GLA? How tight are the bounds?

We agree that we did not directly test how well the theoretical assumptions describe pretrained GLA. We have conducted several additional experiments.

Stability: We tested whether small differences in a prompt gradually disappear from the model's memory. We took two slightly different prompts, then fed the model identical subsequent tokens and measured how far apart their memory states remained. After 1,024 tokens, the difference had decreased by about 25% on average, with 191–192 out of 192 trials showing a decrease. This provides evidence of memory contraction under these conditions.

Geometry: We tested whether the memory states occupy a restricted region rather than filling the entire available space. We fitted an ellipsoid using separate calibration data and found that it contained all 48 held-out prompt-end states. Furthermore, preserving just 64 informative directions reconstructed these states with 3.57% average relative error, compared with 69.36% using random directions. This provides empirical evidence that GLA memory has a highly nonuniform structure.

Tightness: We do not claim that our bounds are tight for pretrained GLA, but rather that incorporating memory dynamics and reachable-state geometry allows us to derive tighter bounds than more general capacity arguments. We believe obtaining tighter, model-specific bounds is an important direction for future work. In particular, introducing additional assumptions about data distributions and using information-theoretic techniques such as mutual information could provide more refined, distribution-dependent bounds.

Q2. Can model capacity become limiting? Do the results extend beyond cramming?

We agree this deserves more direct tests. Given available compute and rebuttal time, we completed a deliberately restricted Qwen3.5 single-head, single-active-layer ablation, replacing the remaining blocks with identities. Under a fixed 10,000-step optimization budget, the discovered exact-recovery frontiers (four targets per prompt length) were:

Prompt tokensOutput-length 50%-recovery bracket
5,0007,168–7,680
10,0008,704–9,216
15,0009,728–10,240
20,00010,752–11,264
25,00013,312–13,824
30,00012,288–12,800

The frontier initially grows but shows clear signs of plateauing, with no gain from 25K to 30K prompt tokens.

Beyond synthetic cramming, we added a 500-example SQuAD-completion experiment on frozen Qwen3-1.7B. We vary the sliding-window size in all 28 layers during ingestion and generation. Accuracy measures answer containment in generated completions, not standard SQuAD exact match.

WindowCorrect / 500Accuracy
16153.0%
32448.8%
649719.4%
12817334.6%
25624448.8%
51225350.6%
102425250.4%
Full25250.4%

These are relatively short-context examples: 99% of inputs are at most 512 tokens, and all input-plus-output sequences fit within 1,024 tokens. This shows how sliding window size directly translates to QA performance.

In response, we will make the following revisions:

  1. Include experimental controls validating key assumptions.
  2. Provide small scale experiments demonstrating the existence of the model capacity limitation.
  3. Further clarify the intention behind our work and include some real world experiments on QA reasoning beyond cramming.

We hope this appropriately addresses your concerns. Please let us know if you have any further questions

I think the theoretical part of this paper is very good.

The analysis is solid, and the main paper and appendix provide detailed derivations. I particularly appreciate the analysis of memory-update dynamics and reachable-state geometry. 

Thank you for adding the experiments in the rebuttal.

The additional GLA experiments help connect the theoretical analysis to pretrained models, and the SQuAD results extend the evaluation beyond synthetic sequence recovery.

Overall, I still think the experimental part needs to be strengthened.

In the new Qwen3.5 experiment, increasing the prompt length from 25K to 30K did not improve recovery under the fixed optimization budget. It is still unclear whether this is due to a limit in model expressivity or a difficulty in optimizing the longer prompt. Stronger evidence on this point would make the paper’s main conclusion about the joint limits more convincing.

I will reconsider rating.

Thank you for your thoughtful follow-up and positive comments! We agree that the previous experiment could not clearly distinguish model limitations from optimization difficulty.

To address this, we extended our Qwen3.5 single-head, single-active-layer experiment to 40K prompt tokens and 40K optimization steps, using five paired targets. We measure the output-length frontier at which at least 3/5 targets can still be recovered exactly. All brackets have been refined to 512-token resolution, with exact recoveries verified through native greedy generation.

Prompt tokens20K steps30K steps40K steps
10,0009,728–10,24011,264–11,77611,264–11,776
20,00012,800–13,31212,800–13,31213,312–13,824
25,00013,312–13,82413,312–13,82413,824–14,336
30,00013,312–13,82415,360–15,87215,360–15,872
35,00015,360–15,87215,360–15,87215,360–15,872
40,00014,848–15,36015,360–15,87215,360–15,872

We observe two important things.

First, additional optimization initially improves recovery, but the improvements begin to level off. For instance, with a 30K-token prompt, the recovery frontier increases from approximately 13.5K to 15.5K output tokens when increasing optimization from 20K to 30K steps. However, increasing optimization further from 30K to 40K steps produces no additional shift for the 30K, 35K, and 40K prompts. The endpoint recovery counts also remain unchanged. This gives us greater confidence that the observed flattening is not simply due to insufficient optimization at earlier budgets.

Second, the frontier shows clear signs of plateauing with increasing prompt length. At both 30K and 40K optimization steps, all three largest prompt lengths (30K, 35K, and 40K) yield the same 15,360–15,872 recovery bracket, despite the additional prompt capacity.

We recognize that further optimization could potentially still improve recovery, and our available compute during the rebuttal limits how extensively we can investigate this. Nevertheless, these experiments strengthen the evidence for diminishing returns under restricted model capacity, with the plateau persisting across both larger prompt lengths and optimization budgets.

We will include this additional experiment and its analysis in the final version of our paper.

We appreciate the reviewer raising this point, and hope these additional experiments address your remaining concerns.

Reviewer_PkzPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how prompt budget, retained memory, and fixed model capabilities jointly constrain the expressive capacity of language models. The authors introduce a causal-cut framework for prompt accessibility and derive theoretical upper bounds based on packing numbers, contractive memory dynamics, reachable-state geometry, and Gated Linear Attention (GLA) structure. Experiments on frozen GLA checkpoints, Qwen3-0.6B, and Mamba models demonstrate that retained memory size and representation influence discovered exact-recovery frontiers, motivating the joint consideration of memory and model architecture.

Critical strengths
  1. This paper introduces a coherent causal-cut formulation that jointly considers prompt budget, retained memory, and fixed model capabilities when analyzing accessible outputs. Its extension to stability-based, geometry-sensitive, and GLA-specific memory bounds provides a useful perspective on the limitations of memory mechanisms. This is important because it addresses a fundamental problem in language-model architecture design and offers a potentially generalizable framework for understanding memory-expressivity trade-offs across different architectures.

  2. The authors have employed frozen model weights, paired targets, controlled memory interventions, and strict saved-state evaluation to investigate how memory affects exact output recovery. Particularly important is the equal-payload comparison in which a distribution-informed memory representation recovers 23/24 targets, compared with only 1/24 for a random representation. This is important because it provides strong empirical evidence that memory usefulness depends not merely on storage capacity but also on the information retained, supporting the paper's central motivation.

Critical weaknesses
  1. The theory establishes robustness under prompt and state perturbations, but the experiments only evaluate exact recovery via finite prompt optimization. As they neither test robustness at nonzero perturbation levels nor validate the derived bounds, the results provide only qualitative support for memory retention rather than direct empirical confirmation of the theoretical capacity limits. Robustness evaluations or numerical bound validation would strengthen the work.

  2. The prompt-independent memory bound requires uniform contraction, while the geometry-sensitive bounds assume certified reachability of all states. The manuscript does not verify these assumptions for the evaluated pretrained models, and contraction of a recurrent component may not extend to the full multilayer system. As a result, the practical applicability of the strongest theoretical results remains unclear. A verified numerical example would strengthen the work.

  3. Theorem 4.1 largely follows from causal-cut factorization and standard packing-number arguments, and the manuscript does not clearly distinguish its contributions from prior prompt-accessibility theory. Furthermore, the smallest upper-bound term need not correspond to the actual accessibility bottleneck. The authors should better clarify the novelty of their theoretical results and demonstrate settings where the proposed bounds are nontrivial and informative.

Questions for authors
  1. Can the authors verify that the assumptions of Theorem 4.2 or Proposition 4.3 hold for at least one representative model and provide numerical bounds comparable to empirical accessibility measurements?

  2. What new theoretical insights does Theorem 4.1 provide beyond existing prompt-accessibility results and standard packing-number arguments? Specifically, how can the framework identify a true memory bottleneck rather than merely the smallest available memory-side upper bound?

  3. Whether the observed memory-frontier improvements persist when prompt optimization is fixed and memory compression is applied afterward?

3 comments

Thank you for your careful reading and for recognizing the value of the joint framework and the GLA experiments.

Q1. Do the assumptions hold in pretrained GLA? Can we obtain numerical bounds?

Robustness: We perturbed 23 successful GLA-340M prompts and their saved memory states, then checked whether they still generated the entire 4,096-token target. At a perturbation size of 1% of a fixed reference norm, 98.9% of prompt perturbations and 90.8% of dense-state perturbations preserved exact recovery. This demonstrates empirical stability of successful recoveries.

Stability: We tested whether small prompt-induced differences in memory decay when the model receives the same subsequent tokens. After 1,024 tokens, state differences fell by approximately 25% on average, and 191–192 of 192 pairs became closer. This supports finite-time forgetting in pretrained GLA.

Geometry: We tested whether pretrained GLA's memory states occupy the entire available space or are concentrated in particular directions. We fitted an ellipsoid using separate calibration data and found that it contained all 48 held-out memory states measured at the end of prompts. We also found that preserving just 64 carefully selected directions allowed us to reconstruct these states with only 3.57% average error, compared with 69.36% when using random directions. This suggests that the model's memory occupies a highly structured region rather than being spread evenly across all dimensions, supporting the motivation for our geometry-sensitive bounds.

Numerical bounds: We also evaluated our bounds directly on pretrained GLA-340M. Using its normalization and frozen projection weights, we derived a conservative reachable-state enclosure, giving a log-packing bound of 50.84 million bits at 1% state resolution relative to a fixed reference norm. Accounting for the observed memory geometry reduces the empirical bound expression from 27.16 million to 9.43 million bits, although these fitted geometric estimates are not globally certified.

While these bounds remain too loose to explain pretrained GLA's recovery frontier, we also tested a controlled two-dimensional memory model where the exact capacity is known. Here, 65 outputs are accessible, compared with upper bounds of 169 using our geometry-sensitive approach and 25,921 using a conventional ball bound. This demonstrates how incorporating memory geometry can substantially sharpen the theoretical bounds.

Q2. What is new beyond standard packing, and when is a bottleneck truly active?

We agree that taking the minimum of several packing bounds in Theorem 4.1 is mathematically simple. Our contribution is introducing the complete retained memory as an independent constraint, alongside existing prompt and model-side bounds. This lets us derive new memory-specific limitations based on memory dynamics, reachable-state geometry, and architectural properties such as GLA's rank and retention mechanisms.

We also agree that the smallest upper bound does not necessarily identify the actual bottleneck. Our theorem establishes necessary constraints, rather than guaranteeing which one is attained. However, we think the underlying intuition is important and already appears implicitly in practice.

For example, models have context-window limits reflecting practical constraints on how much information they can effectively use. Meanwhile, recent recurrent and hybrid attention architectures deliberately restrict retained memory to improve efficiency while preserving useful performance. These trade-offs already exist and have been arrived at organically in practice, but the different perspectives have largely been treated separately. Theorem 4.1 formalizes this joint perspective by establishing necessary constraints from prompt budget, retained memory, and fixed-model capacity.

Q3. Do the observed memory-frontier improvements persist when prompt optimization is fixed and memory compression is applied afterward?

We already tested this in Appendix E.2.6. We selected the 45/48 dense-optimized trajectories that recovered their targets exactly and then compressed their saved states through four tested channels, leaving the prompts unchanged. None of the 45 remained exact after compression. In contrast, when we optimize prompts for the informed rank-64 compressed interface, we recover 23/24 targets. Thus the compressed memory accessibility is not simply obtained by compressing an already optimized dense prompt.

In response, we will make the following revisions:

  1. Include experimental controls validating key assumptions.
  2. Highlight the novelty and meaning behind our key theorems.
  3. Emphasize the controls in the appendix showing the effect of introducing memory controls before and after optimization.

We hope these additional results and clarifications address your concerns.

Thank you for the detailed response. The additional robustness, stability, geometry, and numerical-bound evaluations substantially address my concerns regarding the connection between the theoretical assumptions and the experiments. The controlled two-dimensional example also helps demonstrate when the geometry-sensitive bound is informative. I appreciate the clarification regarding the scope of Theorem 4.1 and the post-hoc compression control. Some limitations remain, particularly the looseness of the pretrained-model bounds and the lack of global certification for the fitted geometry, but these are now clearly acknowledged. Overall, the response has strengthened my assessment of the paper.

Thank you for your kind comments, we are glad we were able to appropriately resolve your concerns.

Reviewer_fGoUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper characterize the limits of language model expressivity by distinct information bottleneck and its induced retained memory capacity. The authors propose a new concept called "causal cut" as a information bottleneck between the prompt processing and text generation phases, forcing all prompt-derived information to pass through a fixed internal state. This enables to derive a joint bound: a model's ability to generate diverse outputs is constrained by the minimum of the prompt budget, the model's fixed read/write capacity, and the capacity of the retained memory itself. Even if one of them is quite large, other quantities can become as a bottleneck, which gives more fine grained understanding of the LLM's capability. As a theoretical bound on the capacity, they derived a general packing number type bound and then they also derived a tighter bound for concrete models by using each model's geometric property. The theoretical framework is empirically tested on Gated Linear Attention (GLA) models, where continuous prompts are optimized to recover synthetic target sequences from compressed memory states.

Critical strengths

I think this paper gives an interesting piece to understand the throughput capability of LLMs based on their proposed concept:

Novelty of formulation: The concept of the causal cut provides a first-principles framework to isolate memory as an independent bottleneck, successfully integrating it with existing prompt- and model-side constraints.

Sound theoretical investigation : Incorporating the geometry of the reachable state space (e.g., narrow vs. wide variation directions) and the stability of token generation dynamics to tighten packing number bounds is a nice contribution that improves naive dimension-counting.

Careful experimental control: The methodology for testing exact recovery is carefully designed to prevent data leakage, ensuring that the generation phase strictly relies only on the causally cut memory state.

Critical weaknesses

My biggest concern is its separation from real world practice as summarized follows:

  • Disconnect between theory and empirical investigation: The theoretical analysis provides (ϵ,ρ)(\epsilon, \rho)-robust accessibility, but the experiments rely on optimizing continuous prompts to perfectly reconstruct random word sequences. A failure to recover a target might merely indicate an optimization failure rather than a true theoretical memory limit.
  • Lack of practical task validation: The evaluation is entirely based on an extreme, synthetic memorization task (cramming). There is no demonstration of how these memory bounds manifest in or degrade performance on actual downstream LLM tasks like reasoning, summarization, or in-context learning.
  • Artificial memory constraints: In the GLA experiments, the memory footprint is reduced by projecting an already-computed dense state into a lower rank, rather than natively restricting the architecture. This makes it difficult to draw practical conclusions about actual inference efficiency or memory savings.

I am also not fully convinced by the appropriateness of the deterministic packing number type argument. In the usual information theoretic analysis, the average code length is considered which takes "importance" of each items into account. On the other hand, this paper considers just counting the number of possible resulting targets. Such an evaluation metric looks a bit naive and apart from practical demands.

Questions for authors
  • Can you design an experimental setting in which we can see the practical performance of the model based on the causal cut argument.
  • Is it possible to provide more "practical" insight of the proposed metric in more real world datasets and problem settings?
  • I want to see more deep discussions about connection to statistical/information-theoretic quantities (such as mutual information) and the pros-and-cons compared with those classic and well-known approaches. I think this is quite important to clarify the position of this paper's proposal.
3 comments

Thank you so much for recognizing the novelty of our work and for your insightful comments. You bring up points that we wished to express better but maybe didn't fully.

Q1. Can we demonstrate practical performance under the causal-cut framework?

We added a 500-example SQuAD-completion experiment on frozen Qwen3-1.7B. We vary the sliding-window size in all 28 layers during ingestion and generation. Accuracy measures answer containment in the generated completions, not standard SQuAD exact match.

WindowCorrect / 500Accuracy
16153.0%
32448.8%
649719.4%
12817334.6%
25624448.8%
51225350.6%
102425250.4%
Full25250.4%

These are relatively short-context examples: 99% of inputs are at most 512 tokens, and all input-plus-output sequences fit within 1,024 tokens. This explains why the 1,024-window outputs match full attention exactly on all 500 examples. This shows how sliding window size directly translates to QA performance.

Concern: Projected GLA memory is artificial We acknowledge that the GLA experiment projects the memory matrix to lower rank for a cleaner controlled setting. However, Section 6.3 also includes a same-weight SWA comparison, which directly changes the attention window and retained memory.

Q2. What practical insight does the metric provide for real-world models?

In our view, the primary practical value of this paper is to explicitly recognize memory and model architecture (including parameter capacity) as key resources to balance when constructing language models. People already do this implicitly in two different ways. First, almost all models have explicit context-window limits. They reflect practical constraints that developers have encountered through training and evaluation. On the other hand, many recent models explore memory-saving techniques through recurrent and hybrid sliding-window or linear-attention architectures in the name of efficiency. These approaches have found that limiting retained memory can still achieve good performance at lower inference costs. The intuition behind our paper already exists and has been arrived at organically in practice, but these two perspectives have largely been treated separately.

We formalize this intuition, unifying these two perspectives. This opens up questions about designing architectures that make this trade-off most efficient, developing memory mechanisms that maximize usable expressivity relative to parameter and memory costs, or adaptively limiting retained memory depending on the input.

Q3. Comparison with Information-theoretic quantities.

Put simply for us the packing argument is a worst case scenario, mutual information gives an "average" case scenario. The naive nature of the packing argument is actually something we consider integral to our work. This approach allows us to derive bounds primarily from properties of the model and its retained state, without assuming a particular distribution over the target sequences.

We see information-theoretic approaches as complementary rather than competing. Our goal in this paper is to first establish that these memory and model-capacity limitations exist independently of the target distribution, under our stated assumptions. We think establishing this general principle is important before studying more specific settings with stronger assumptions. Introducing additional stronger assumptions about the data could help refine these bounds. We consider this an important direction for future work.

Concern: Optimization and accessibility

We agree that failing to recover a target through prompt optimization does not prove definitively it is inaccessible. Our pretrained-model experiments therefore measure discovered accessibility, not absolute capacity limits.

To address this distinction, we constructed a simple two-dimensional linear memory model:

Mt=0.95Mt−1+diag⁡(a,aα)ut,∣ut∣2≤1.M_t=0.95M_{t-1}+\operatorname{diag}(a,a\alpha)u_t,\qquad \\|u_t\\|_2\leq1.

A fixed decoder maps the memory state to discrete output labels. Since the reachable memory states can be characterized exactly, we can count all robustly accessible outputs without relying on prompt optimization. We find that increasing prompt length initially raises the number of accessible outputs from 3 to 65, after which it saturates. Changing the geometry of the memory also reduces the accessible count from 3,241 to 65, even with the same memory dimension and maximum radius.

In response, we will be making the following changes:

  1. Conduct a more in depth discussion in the paper regarding the practical meaning of our work.
  2. Provide a clearer comparison with information theoretic techniques to qualify our angle.
  3. Include the experiments extending our work past the purely cramming setting.

We hope this appropriately addresses your concerns. Please let us know if you have any further questions!

Thank you very much for your response. The additional experiment somehow demonstrates the relation to practical situation. However, I am still not convinced that this approach is indeed useful to investigate how information is processed inside the model. I agree that seeking a general approach is important but when we consider high-dimensional models/data, this approach would be pessimistic. Therefore, I would like to maintain my score.

Thank you for your follow-up. We agree that general theoretical bounds can be pessimistic for high-dimensional models, and that developing tighter bounds is an important direction.

However, we believe that establishing these theoretical limitations is itself an important first step, even before the bounds can be made tight. Our contribution is to introduce retained memory as an independent constraint on expressivity and develop a framework for understanding how it interacts with prompt budget and fixed-model capacity. We believe these initial bounds provide a foundation for sharper results through additional assumptions and more refined mathematical analysis, potentially applicable to broad classes of models.

We also think this question has direct relevance to practical architectures. In our GLA experiments, changing how information is retained under the same memory budget changes exact recovery from 1/24 to 23/24 targets. Our SWA experiments further demonstrate how restricting the actual attention window affects accessibility, while the additional SQuAD results show that memory restrictions also affect natural-language task performance.

As smaller models and personalized agents become increasingly capable, we believe understanding how much information a model can retain and effectively use will become an important architectural question. Our work aims to establish a theoretical foundation for studying this trade-off, with tighter theoretical characterization as a natural next step. We appreciate your feedback and will clarify this motivation in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.