acceptodds
Under review as a conference paper at ICLR 2027

Rank Is the Bottleneck: Measuring and Training Pluggable Memory for LLMs

Abstract

Pluggable memory offers a general and portable way to provide language models with reusable context: source text is encoded once into a memory matrix that can be inserted directly into the decoder's input. Yet memory is commonly characterized by its embedding count or encoding rate, which describes the allocated budget without revealing whether those embeddings provide distinct, task-useful information. We identify rank as a key constraint on this interface. For an idealized linear attention layer, the rank of the memory matrix bounds both the feature directions that attention can read and the space of score patterns across memory positions. This perspective leads to our proposed MemTower, a two-stage framework designed to make available memory directions predictive through variable-rate continuation pretraining, explicit budget announcement, rank regularization, and fixed-rate task specialization. Across multi-passage QA and reasoning benchmarks, MemTower improves over embedding-based memory baselines; at a x4 encoding rate, its 8B configuration outperforms the best baseline on all four reasoning tasks, averaging 0.79 versus 0.72. Ablations and rank interventions further show that budget announcement and regularization improve both accuracy and the use of retained directions. MemTower preserves at least 98% of its reasoning accuracy with 16 directions, whereas removing either component requires 32 despite achieving lower full-memory accuracy. Together, these results expose a distinction overlooked by memory size alone: how much representational capacity is available, and how effectively that capacity supports prediction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Memory, Context Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YUpROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the representation efficiency of compressed pluggable memory in large language models, with a focus on the rank of the memory matrix. It asks how the available representation space can be used more effectively under a given memory budget. The authors analyze the redundancy among memory vectors from the perspective of memory-matrix rank. They argue that, compared with only looking at the size of the memory matrix, rank can further reflect the effective dimension of the memory representation space, and it bounds the independent feature directions that the later attention layer can read. The paper then proposes MemTower, which improves downstream task performance through variable-rate continuation pretraining, explicit budget announcement, and rank regularization. In my view, the problem studied here is important and meaningful, and using matrix rank to analyze the capacity of compressed memory is an intuitive and inspiring idea. The experiments show that the proposed method achieves some performance gains on several QA and reasoning tasks.

Critical strengths

The paper addresses an important and practical problem in LLMs about how to make good use of a limited pluggable memory capacity. The authors propose to analyze representation redundancy from the memory-matrix rank perspective. Although analyzing redundancy with rank is a common idea in traditional machine learning, applying it to the memory matrix of an LLM is still intuitive and insightful. MemTower has a reasonable technical motivation, the experiments are fairly comprehensive, and the method achieves meaningful performance improvements on several downstream tasks. In addition to standard benchmarks, the authors also analyze the memory representations through rank interventions and spectral analysis.

Critical weaknesses

The technical contribution of this paper is relatively limited. The theoretical analysis mainly relies on standard properties of linear algebra and does not develop substantially deeper theoretical results. The proposed method mainly combines existing training strategies and representation regularization techniques within the pluggable memory framework, resulting in limited technical novelty. Although the authors provide fairly complete performance comparisons and representation analysis, the relationship among rank, memory budget, and actual task performance is not studied in depth. In particular, a systematic comparison with other representation-diversity regularization methods is missing, and the actual computation cost is not analyzed. The theoretical analysis mainly targets the local linear attention operation, while its effectiveness in a real nonlinear Transformer is supported mostly by experiments. On top of this, the paper does not release implementation code, which to some extent limits the reproducibility of the results. For these reasons, I think the authors need to answer the Questions For Authors properly before the paper can reach the acceptance level.

Questions for authors
  1. The main theoretical result of the paper shows that the rank of the memory matrix gives an upper bound on both the independent feature directions readable by attention and the dimension of the pre-softmax score patterns. This conclusion is reasonable and meaningful, but the proof mainly relies on standard matrix-rank properties of linear algebra, so the mathematical derivation is fairly direct. Representation redundancy, low-rank structure, and effective representation dimension have also been studied for a long time in machine learning and pattern recognition. Could the authors further explain what new theoretical insight this analysis provides compared with existing work on rank and representation redundancy?
  2. Compared with covariance regularization, orthogonality constraints, or other spectral regularization methods, what is the advantage of the log-determinant regularization used here? Why are these common representation-diversity regularization methods not used as baselines under the same experimental setup?
  3. Table 3 shows that, after removing rank regularization, the average accuracy drops by 1.6% at the 4× compression rate and by 7.6% at 16×. This result is interesting because it suggests that efficient use of representation dimensions may matter more at higher compression rates. Could the authors further analyze how rank regularization works under different memory budgets?
  4. MemTower requires training and running a Memory Tower with up to 8B parameters, and this cost has an important effect on the practical value of the method. In particular, a larger Memory Tower may improve memory representation quality, but its encoding cost can also be higher. Could the authors report the actual encoding and inference cost?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer YUpR (part i)

We sincerely appreciate the reviewer's thoughtful feedback and for recognizing the practical importance of pluggable memory. We first address the reviewer's specific questions as below.

Q1. New theoretical insight, and applicability to a nonlinear Transformer.

  • Our analysis gives representation redundancy an operational meaning at the memory interface. Proposition 1 identifies that adding linearly dependent embeddings can increase storage and processing cost without expanding these independent dimensions. This establishes a distinction between the number of memory slots and the capacity they expose to the decoder, motivating us to examine how effectively a given memory budget is used.

  • This distinction motivates usable rank, which relates retained memory directions to the task accuracy they sustain. RankMe assesses representation quality through a spectral statistic, while Seq-VCR regularizes internal Transformer representations [1,2]. Our intervention instead measures the utilization of each context's compressed memory by a separate decoder. Such paradigm connects the capacity constraint to task utility, which the spectrum alone does not establish.

  • The paper's "linear interface" analyzes the projections and weighted sums used to read memory. The value bound also holds when softmax determines the weights: the output remains a combination of the fixed value vectors and cannot leave their span. These are local constraints that we model theoretically on memory access; their relevance to the general Transformers is empirically tested by removing memory directions and measuring accuracy with the full nonlinear Transformer unchanged.

Q2. Choice of log-determinant and comparison with diversity regularizers.

  • We choose the normalized log-determinant because it acts on the spectrum of each context's memory matrix: it is invariant to rotations of the embedding coordinates and does not reward larger embedding magnitudes. Its diminishing returns favor distributing variation across directions, while we also adopt normalization to make the value comparable across memory budgets. These properties suit our variable-budget memory training, where we build on established covariance and coding-rate objectives [3,4].

  • We agree with the reviewer that covariance regularization and orthogonality constraints offer related ways to encourage representation diversity. However, penalizing off-diagonal covariance alone can still leave variation concentrated in a few dimensions; our normalized log-determinant additionally rewards spreading that variation across directions. Together with its scale and budget normalization, this makes it a natural surrogate for our memory-capacity analysis. Our ablations therefore focus on whether this spectral objective improves memory utilization within the complete training recipe.

Q3. Rank regularization under different memory budgets.

  • For a 1024-token source context, x4 provides 256 memory slots, whereas x16 provides 64. Redundant directions therefore consume a larger fraction of the smaller budget. Consistent with this, removing regularization causes a 1.6% accuracy drop at x4, increasing to 7.6% at x16.

  • To compare geometry across budgets, we apply the same spectral diagnostic to the existing x4 runs, obtaining 0.88 without / 0.93 with regularization, compared with the reported 0.80 / 0.95 at x16. Normalization accounts for the different numbers of available directions, which shows a larger increase in spectral spread at x16 (+0.15 vs. +0.05), consistent with the larger accuracy drop when regularization is removed at that rate. Such a pattern supports regularization's role in using a tighter memory budget more effectively.

AuthorsOpenReview ↗

Author Response to Reviewer YUpR (part ii)

Q4. Encoding and inference cost.

  • We additionally perform detailed analysis regarding MemTower's costs and savings, as in our response to Reviewer eo5Q's Q2. Overall, the savings come from two complementary sources: reusing a memory amortizes its encoding cost, and decoupling representation from generation permits a smaller encoder. For 1k source tokens at x4, a 64-token query, a 128-token answer, and an 8B decoder, we estimate model-weight computation using 2P2P FLOPs per token.
AccuracyEncodingFirst use, total10 uses, total
Full context, 8B85%-19.46 (100%)194.56 (100%)
MemTower, 8B→8B79%20.4827.65 (142.1%)92.16 (47.4%)
MemTower, 1.7B→8B76%4.3511.52 (59.2%)76.03 (39.1%)
MemTower, 0.6B→8B76%1.548.70 (44.7%)73.22 (37.6%)
  • Reuse recovers the cost of even an 8B encoder on the second use. Each query costs 7.17 versus 19.46 TFLOPs for full context, amortizing the initial encoding cost.

  • Smaller encoders make savings possible from the first use. Our additional 0.6B→8B model achieves 76% accuracy while reducing first-use computation by 55.3% relative to full context under this approximation.

Lastly, regarding contribution and reproducibility:

  • Our central contribution is a rank-based perspective on how effectively pluggable memory uses its limited capacity. Adding memory embeddings increases the allocated budget, but redundancy can prevent that increase from exposing more independent directions to attention. We make this distinction concrete through the rank bound, then connect it to prediction through usable rank, measuring the accuracy sustained by retained memory directions. This perspective motivates our design of MemTower, which trains the encoder and decoder to allocate predictive content across the available directions as the memory budget changes. Its improvements in both task accuracy and performance under rank truncation demonstrate the value of connecting memory geometry, decoder utilization, and training design.

Our scripts for model, training and evaluation will be released along with the camera-ready version to ensure reproducibility. Our trained models will be released in multiple settings, including 8B→8B, 1.7B→8B, and 0.6B→8B.

We sincerely thank for the reviewer's suggestions again for clarifying the paper's contribution and its practical value. We will incorporate these additional insights into the final version, and please let us know if there are any remaining questions or concerns.

[1] Garrido et al., RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank, ICML 2023.

[2] Arefin et al., Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning, ICLR 2025.

[3] Bardes et al., VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning, ICLR 2022.

[4] Yu et al., Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction, NeurIPS 2020.

Thank you for your response. It has addressed most of my concerns. My main remaining concern is still whether the constraint imposed by the rank of the memory matrix remains meaningful in the presence of the complex nonlinear operations in Transformers. Although the authors' current discussion does not fully resolve this question, it provides some theoretical justification and empirical evidence supporting the rank-based analysis. Overall, I recognize the value and completeness of this work, and I will raise my score.

We sincerely thank the reviewer for the engagement and positive assessment of our work!

Mathematical-wise, it is indeed difficult to perform theoretical analysis on full nonlinear operations. Rather, in this work we establish a rank-based perspective for understanding how effectively pluggable memory utilizes its limited representational capacity, supported by both tractable theoretical analysis in the linear setting and empirical validation on real-world LLMs. We greatly appreciate the reviewer’s recognition of the value of this perspective and the constructive feedback throughout the discussion, which we believe will make a better version of the final paper.

Reviewer_nsyhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the role of memory rank in embedding-based context compression for LLMs. It introduces usable rank through principal-component truncation and proposes MemTower, combining variable-rate pretraining, budget conditioning, and spectral regularization. Experiments on question answering and reasoning benchmarks report improvements over memory baselines, supported by component ablations and analyses of memory geometry.

Critical strengths
  1. The distinction between memory size and task-useful capacity is well motivated. The accuracy–rank analysis provides a useful diagnostic beyond compression rate, and the paper appropriately distinguishes spectral spread from downstream utility.

  2. Budget conditioning addresses a concrete limitation of causal memory readouts. At ×4 compression, MemTower improves over the reimplemented PISCO on all four reasoning benchmarks, increasing reported average accuracy from 0.72 to 0.79. Ablations support the effectiveness of the combined training recipe.

Critical weaknesses
  1. The evaluation leaves questions about attribution and scope. Table 1 combines substantial training–evaluation overlap with different backbones, limiting the interpretation of its small average gain. The reasoning comparison is more informative, but the PISCO reimplementation’s training conditions and the reasoning mixture’s provenance and split handling remain insufficiently specified. Moreover, encoding complete MMLU/GSM8K problems assesses task-conditioned compression of relatively short inputs. While useful for testing information retention, this provides limited evidence for the stated motivation of encoding context once and reusing it across subsequent queries.

  2. The rank-based explanation is not yet sufficiently established. Proposition 1 provides a valid bound under an idealized interface, but does not show that rank actively limits the evaluated models. Truncation removes information as well as reducing rank, while the regularization ablation does not separate spectral spreading from anti-collapse. Usable rank also depends on each model’s reference accuracy and a coarse rank grid. The evidence therefore supports the combined training recipe more strongly than a rank-specific mechanism.

Questions for authors
  1. Which training conditions were matched between PISCO and MemTower, particularly pretraining, trainable parameters, teacher responses, and optimization budget?

  2. What are the exact sources, splits, and unique-question counts for the CSQA and MMLU entries in Table 7? Was the complete reasoning training mixture checked for overlap with the evaluation sets?

  3. Does spectral spreading improve performance beyond the anti-collapse term alone or a simple decorrelation regularizer under matched training conditions?

  4. Does the reported 16-versus-32 usable-rank distinction persist with a denser rank grid and different tolerance thresholds?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer nsyh (part i)

We sincerely thank the reviewer for recognizing the motivation behind our capacity analysis and the effectiveness of budget conditioning. We first address the reviewer's two concerns below.

Q1. Evaluation scope and attribution.

Table 1 adopts the consistent QA evaluation protocol for direct comparison with prior work. We independently identified and audited the overlap inherited from this protocol, disclosing it in Table 1 and Appendix A.6. Since such overlap can make QA results overly optimistic, we explicitly broaden the evaluation in Table 2 to more tasks, requiring memory to preserve precise quantities, relations, and constraints, as an orthogonal challenge beyond recovering answer-bearing facts from Wiki passages.

We do agree with the reviewer that the longer Wiki-based document aligns more directly with our motivation for context reuse. To strengthen this aspect, we conduct additional experiments by removing overlapping queries from the training set, following the matching rule in Appendix A.6, and retrain the specialization stage with the evaluation and other conditions unchanged. The results are reported below (the same table also in our response to Reviewer eo5Q).

MethodASQAHotpotQANQTriviaQAPopQAAvg.
Full context, Qwen3-8B755469936471
MemTower, original725366926269
PISCO, filtered674861886365
MemTower, filtered705264916168

MemTower retains its advantage under the new setting, reaching 68% against 65% for the baseline. Its gains thus extend across question-independent passage memory and reasoning-oriented compression. We will incorporate these results and clarify the complementary evaluation roles in the camera ready version.

Q2. Rank utilization and the role of regularization.

We appreciate the reviewer's distinction between spectral spread and downstream utility, which is central to our paper position. Proposition 1 establishes a necessary capacity constraint under the linear interface: exposing kk independent directions requires rank at least kk; however, their usefulness must also be learned. A broader spectrum can be regarded as a necessary condition, but not a sufficient condition towards better performance; effective rank utilization depends on the entire training design and memory interface. Therefore, we view rank as a measurement, as the two are closely intertwined and jointly shaped by the design choices. We will clarify this more clearly in the final version.

Our whole training recipe is shown to encourage such rank geometry and utility. Regularization increases spectral spread from 0.80 to 0.95 at x16. Meanwhile, truncation at k=8k=8 achieves 68% accuracy, surpassing the 59% without budget conditioning and 61% without regularization, suggesting that our method sustains better prediction from the same number of retained directions.

AuthorsOpenReview ↗

Author Response to Reviewer nsyh (part ii)

Regarding the reviewer's specific questions.

  1. For the training conditions, in particular the 8B→8B comparison in Table 2, our MemTower model and the reimplemented PISCO adopt the same Qwen3-8B encoder and decoder. Both fully train the encoder and use decoder LoRA, with the same pretraining and specialization data, teacher responses, hyperparameters, and training steps. The pretraining for PISCO is the naive LM objective with fixed rate, no budget announcement, no regularization; also it excludes the connector by its original design.

  2. For the CSQA and MMLU entries in Table 7, we use the officially released tau/commonsense_qa/train and cais/mmlu/auxiliary_train on Huggingface. For our training set, we randomly sample 50k distinct records from MMLU, and we take all 9741 distinct examples from CSQA and repeat twice to balance the two datasets, resulting in near 20k records.

    For the overlap in train-eval splits, both datasets themselves have no direct overlap. However, we do find 284 CSQA eval records in the general SFT training set Tulu 3. Removing these records from the CSQA eval set results in less than a 0.8% drop across all trained models, leaving our overall findings unchanged. We will explicitly mark this in the final version.

  3. The role of spectral spreading: as discussed above, spreading alone is not a sufficient guarantee; the gains arise from the interplay among our proposed training strategies. Specifically, the anti-collapse term discourages near-identical embeddings, while the spectral term discourages the concentration of representations into a small number of directions.

    To isolate the contribution of the spectral term, we extend the x16 ablation in Table 3 by removing the spectral term while keeping the anti-collapse coefficient, backbone, data, and all other conditions unchanged. After retraining the second stage, the model achieves 52% accuracy, compared with 56% under the full regularizer. This 4% gain over anti-collapse alone provides empirical evidence that the log-determinant spectral-spread term serves as an effective surrogate for promoting decorrelation.

  4. For assess the sensitivity of usable rank curve, we add k=12k=12 and 2424 to the original {2,4,8,16,32}\{2,4,8,16,32\} sweep for MemTower and both component ablations at ×4. We recompute usable rank at 1%, 2%, and 5% relative tolerances, and to compare the ranks needed for a common 70% accuracy target.

    AccuracyR^*_{1\\%}R^*_{2\\%}R^*_{5\\%}Min. tested rank for 70% accuracy
    MemTower79%24161212
    Without budget conditioning73%32241616
    Without rank regularization78%32241616

    At 2% tolerance, this outcome would refine the original comparison to 16 versus 24, retaining MemTower's advantage under the denser grid. The same ordering across tolerances and at a common accuracy target supports more efficient use of retained directions beyond a single threshold choice.

We hope these clarifications and additional comparisons could address the reviewer's concerns. We especially appreciate the thoughtful questions and will incorporate these new remarks into the final version. Please let us know if there are any remaining questions, and we thank the reviewer for the time and efforts again.

Reviewer_eo5QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this manuscript, the authors generally focus on the question of how to compress text into reusable memory embeddings without losing the information an LLM needs. Based on the observation that many memory embeddings contain redundant information, the authors argue that the memory matrix’s rank is an important constraint on what attention can read. The problem is formalized under an idealized linear attention interface. To tackle the problem, the authors proposed the so-called MemTower, which works roughly in three steps:

  1. An LLM encoder compresses source text into a small set of continuous memory embeddings;
  2. A connector maps these embeddings into a separate decoder LLM’s input space;
  3. The decoder uses the memory in place of the original text.

The authors validate MemTower through benchmark comparisons, component ablation, and controlled rank experiments. The experiment results generally justify the effectiveness of MemTower. However, some QA evaluation sets substantially overlap with training questions, weakening that evidence. The reasoning experiments provide complementary support, but the paper does not comprehensively validate practical speed, storage, or total computational savings.

Critical strengths
  • The paper distinguishes memory size from the independent, task-relevant information it provides. The problem that many memory embeddings contain redundant information is worth investigating.
  • It combines a rank bound with interventions that directly measure how removing memory directions affects accuracy.
  • The new method’s gain is obvious in some experimental settings. For example, at 4× compression, MemTower improves average accuracy from PISCO’s 72% to 79%.
Critical weaknesses
  • Some QA evaluation sets substantially overlap with training questions, weakening that evidence.
  • The reasoning experiments provide complementary support, but the paper does not comprehensively validate practical speed, storage, or total computational savings.
Questions for authors
  • Do the QA improvements remain after removing evaluation questions that overlap with training?
  • Could you please elaborate on the total training, encoding, storage, and inference costs compared with full-context LLMs and PISCO? How many times must a memory be reused before its encoding cost is recovered?

I would like to increase my score if the authors can properly address my concerns and questions.

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer eo5Q

We sincerely appreciate the reviewer's thoughtful feedback and for recognizing the value of our rank analysis and the gains achieved by MemTower. We clarify the two raised issues and present additional results as below.

Q1. Training-eval overlap in QA benchmarks.

Table 1 follows the evaluation protocol established by prior work to enable direct comparison with existing baselines. The resulting dataset overlap is thus inherited from this setup. We independently identified and audited this overlap, explicitly disclosing it in Table 1 and Appendix A.6. As a result, such overlap may lead to overly optimistic evaluation results, which further motivates us to conduct broader evaluation in Table 2 with additional benchmarks. These benchmarks require memory representations to preserve the quantities, relations, and constraints essential for downstream reasoning, providing a more comprehensive assessment beyond Wiki-based QA. MemTower achieves an accuracy of 79%, compared with 72% for the baseline, surpassing all four benchmarks, which demonstrates the effectiveness of MemTower beyond the Wiki-QA setting.

To address the reviewer's question for cleaner results on Wiki-QA, we additionally remove those overlapping queries from the training set, following the matching rule as described in Table 8, and re-train the specialization stage with other conditions unchanged.

MethodASQAHotpotQANQTriviaQAPopQAAvg.
Full context, Qwen3-8B755469936471
MemTower, original protocol725366926269
PISCO, filtered training674861886365
MemTower, filtered training705264916168

The results are presented in the above table. MemTower achieves 68% accuracy against 65% for PISCO. While both models show degradation, MemTower remains more robust attributed to our proposed training strategies.

Q2. Savings and cost on encoding, reuse, and overall.

Overall, the computational savings come from two complementary sources. First, memory reuse amortizes encoding across queries: a context is only encoded once for subsequent consumption. Second, decoupling representation from generation allows a smaller encoder, reducing the initial encoding cost itself.

Our table 2 already supports the smaller-encoder mechanism: 4B→8B achieves comparable performance as 8B→8B, with half the encoder size. We further extend this comparison with a 0.6B→8B model shown below.

For a concrete comparison, we fix 1024 source context tokens, 4× compression (256 memory embeddings), a 64-token query, a 128-token answer, and an 8B decoder. We estimate using approximately 2P2P FLOPs per processed token:

MethodReasoning accuracyEncodingFirst use, total10 uses, total
Full context, 8B85%—19.46 (100%)194.56 (100%)
PISCO, 8B→8B72%20.4827.65 (142.1%)92.16 (47.4%)
MemTower, 8B→8B79%20.4827.65 (142.1%)92.16 (47.4%)
MemTower, 4B→8B78%10.2417.41 (89.5%)81.92 (42.1%)
MemTower, 1.7B→8B76%4.3511.52 (59.2%)76.03 (39.1%)
MemTower, 0.6B→8B76%1.548.70 (44.7%)73.22 (37.6%)
  • Compute columns are TFLOPs; percentages are relative to full context = 100% in the same column. These estimates cover model weight multiplications during encoding, prefill, and generation. Accuracy comes from Table 2 and the additional 0.6B experiment.
  • At matched 8B sizes, MemTower improves on PISCO's accuracy at the same leading inference cost. Reducing the encoder then lowers the initial investment: 0.6B→8B uses 55.3% less computation on the first use and 62.4% less over ten uses than full context in this approximation. Each additional query costs 7.17 TFLOPs versus 19.46 TFLOPs, so the 8B encoder recovers encoding on the second use, and the smaller encoders recovers on the first.

Regarding embedding storage.

We store one memory matrix per context. With the same 1k-token context, 4x compression, Qwen3-8B embedding width of 4096, and BF16 precision, this is about 2MB per context. The same matrix is reused across queries: ten uses still require only one 2 MiB copy. Overall, this is a storage-compute trade-off: the saved embeddings occupy more space than raw text but enable cheaper repeated context processing.

We hope the above could address the reviewer's original concerns. Please let us know if there are any remaining questions or issues, and we would be happy to continue the discussion during the rebuttal period. We once again express our gratitude to the time and efforts in reviewing this work, which we believe will make a better version of the paper.

Comments on the paper

1 comment

We would like to thank all reviewers again for the constructive feedbacks, which has helped us better clarify the positioning and contributions of this work. As the rebuttal period comes to a close, we look forward to any further thoughts or questions regarding our clarifications and additional analyses.

Overall, our approach studies on a pluggable memory representation for LLM context, and we bring up a central insight that memory size alone does not determine how much information a decoder can effectively exploit. We establish rank as a structural bottleneck and further show that representational diversity alone is insufficient, unless the additional directions carry information useful for downstream prediction. Motivated by this, the design of our proposed MemTower trains the memory tower to better distribute and utilize the available representational capacity, translating it into stronger task performance.

Thank you again for your time and consideration, and we would be happy to address any remaining questions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.