- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduced MemToken, a query-agnostic compression method for LLMs. It constructs a new dataset for long-context LLM compression with subsets for pre-training and supervised fine-tuning. MemToken uses small attention module that compresses a chunk of token hidden states into one, which serves as the surrogate hidden state for LLM attention. The compression training contains multiple stages, which include a short context warmup, a mixed short and long contexts adaption, and a supervised fine-tuning training. The resultant method is validated on context length up to 128k tokens and outperform several baselines.
Critical strengths
- MemToken releases both its method and training data. It can future development and fair comparison.
- It explains its training design strategy with experiments, such as the choice of warmup training is based on poor performance with straightforward long context training.
- The results looks good with clear advantage over baseline methods.
- The paper is well written and easy to follow. The appendices are extensive.
Critical weaknesses
- Although it claims to support context length up to 128k tokens, it appears only at pretraining stage. The context length of the SFT stage does not exceed 64k tokens, and the context length of the evaluation benchmarks is not discussed in the paper. The authors should consider discussing the context length of the benchmarks and introducing tasks that can reach 128k tokens, such as RULER.
- The idea of compressing multiple hidden states into one state isn't new. For example, the commonly adopted multihead latent attention is similar, though not operating at the group level. The author should better differentiate their method from literature.
- Some of the design choices are confusing to me, see my question section.
- There are multiple citations in the reference section containing the wording "VERIFY citation." Is this a message left by AI tools?
Questions for authors
- In section 4.3, the authors said directly applying long context training makes the model to ignore long context, therefore a short-context warmup is necessary. Can you explain more on why model ignores long context, and what role does the warmup play?
- In section 4.1, MemToken compresses a group of hidden states into one by using a mini-attention. The softmax distribution is of shape , which is strange to me: Does this mean that the distribution is calculated over each individual dimension? If so, why is this perferred over a token-level or multi-head level softmax?
3 comments
Response to the reviewer's concerns (2/3)
R3. Why short-context warmup helps the frozen LLM use compressed memory (Question 1)
The optimization difficulty arises from teacher forcing combined with the initial mismatch between compressed representations and the frozen LLM's input space. During reconstruction, the LLM receives the compressed memory, a repetition instruction, and the gold reconstruction prefix. At initialization, the memory has not yet learned to convey source information in a form the LLM can use. The frozen LLM can nevertheless predict many subsequent tokens from the growing gold prefix using its existing language-modeling ability.
This alternative prediction path becomes particularly strong for long reconstruction targets. Later tokens have a substantial preceding text prefix, allowing the model to obtain a relatively low next-token loss while making limited use of compressed memory. The first few output positions depend more strongly on the memory because little source text has yet been exposed through teacher forcing. As the target grows, those positions account for a smaller fraction of the average token loss. This provides an optimization explanation for the observed behavior in Figure 1(b).
Figure 1(b) compares compression training with a plain autoregressive reference that receives no compressed memory. With 512-token sources, the compression loss falls clearly below this reference as training proceeds. With 65,536-token sources trained directly from initialization, the losses remain close, indicating that the compressed tokens provide little additional predictive benefit in that setting.
Short-context warmup first establishes a useful communication interface between the compressor and the frozen LLM. With shorter reconstruction targets, a larger share of the prediction task depends on information supplied by the compressed memory. Once the compressor learns representations that support reconstruction, the same interface can be trained on longer inputs. The answering LLM remains frozen throughout, so this alignment is learned by the compressor.
In our reported recipe, Stage 1 starts with 80,000 contexts of 128–255 tokens and then mixes the six eligible length bins below 8,192 tokens, using 1,500,000 contexts in total. Stage 2 extends this capability to longer contexts through question answering, positional lookup, in-context induction, and code understanding. We will clarify this sequence and explain that the mixed lengths in Stage 1 remain below 8K, with the larger length extension occurring in Stage 2.
R4. The normalization axis and rationale for channel-wise aggregation (Weakness 3 and Question 2)
Yes, each output channel has its own distribution over the positions in a group. The softmax is taken along the group-position axis. For , let and . The aggregation is
$ A_{j,i,k} =\frac{\exp(U_{j,i,k})}{\sum_{r=1}^{c}\exp(U_{j,r,k})}, \qquad \widetilde H_{j,k} =\sum_{i=1}^{c}A_{j,i,k}V_{j,i,k}. $
For every output channel , . Thus, contains distributions over the group positions. There is no normalization across the feature dimension or across all entries jointly.
The purpose is to preserve complementary features from different source positions when forming one memory state. A token-level pooling weight assigns the same positional mixture to every output channel. Head-level pooling allows different mixtures between heads but shares a positional distribution among channels within each head. Channel-wise aggregation allows each projected output channel to select its own mixture. For example, one channel can draw more strongly from an earlier position while another retains information from a later position, within the same compressed token.
The learned dense projections are also important. Each column of is computed from the full hidden vector at that position, and provides a learned value representation. Consequently, channel-wise weighting operates on learned features informed by all input dimensions. It does not assume that individual original hidden dimensions are independent or correspond to predefined semantic attributes. Tying the positional weights across output channels recovers the weight-sharing pattern of token-level pooling, while tying them within channel groups recovers the corresponding head-level pooling pattern.
We will describe this operator more explicitly as channel-wise gated pooling and add the indexed equation above. This makes its normalization and its role in feature aggregation clear, while distinguishing it from conventional query-key multi-head attention.
Response to the reviewer's concerns (3/3)
R5. Reference-verification annotations (Weakness 4)
The phrase “VERIFY citation” came from internal reference-checking reminders stored in the note fields of our BibTeX entries. The three original entries are shown below.
@misc{qwen35,
title = {Qwen3.5 Technical Report},
author = {{Qwen Team}},
year = {2026},
note = {VERIFY citation}
}
@misc{nemotron3nano,
title = {{NVIDIA Nemotron 3 Nano}},
author = {{NVIDIA}},
year = {2026},
note = {VERIFY citation}
}
@misc{deepseekv4,
title = {{DeepSeek-V4}},
author = {{DeepSeek-AI}},
year = {2026},
note = {VERIFY citation; judge model DeepSeek-V4-Flash-0731}
}
We overlooked that the bibliography style used by the ICLR template renders these note fields in the reference list. All cited references are genuine and were not generated by AI. We will remove these internal notes from the BibTeX entries in the revised manuscript.
References
[1] Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., ... & Xu, Z. (2024). Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434.
Response to the reviewer's concerns (1/3)
We thank the reviewer for the constructive feedback. We provide additional downstream evaluations on contexts approaching 128K, clarify how MemToken differs from Multi-head Latent Attention, and explain the optimization rationale for short-context warmup and channel-wise aggregation.
R1. Downstream evaluation on contexts approaching 128K (Weakness 1)
To directly assess compression quality near 128K, we have added evaluations on LongBench Pro restricted to 64K–128K contexts and InfiniteBench restricted to 100K–128K contexts, using Qwen3.5-9B. These evaluations also demonstrate length generalization beyond this backbone's 64K SFT cap.
MemToken exceeds the strongest evaluated baseline, KVzip, by 0.190 on the reported LongBench Pro metric. On InfiniteBench, we discard complete examples exceeding 128K and report total accuracy below.
MemToken achieves 80.53% and 71.01% accuracy at 4× and 8× compression, improving over KVzip by 20.56 and 17.90 percentage points, respectively. These results provide direct downstream evidence of effective compression on the evaluated contexts approaching 128K. They address the reviewer's request for tasks reaching this length regime through two additional benchmarks.
We will include these results and their length-filtering protocols in the revised paper. We will also expand the benchmark description to report source-context length statistics and the number of examples above 64K for each evaluated configuration, making the length coverage directly visible alongside the aggregate scores.
R2. Core contributions and relation to Multi-head Latent Attention (Weakness 2)
MemToken advances query-agnostic soft context compression through downstream domain generalization and effective compression of source contexts up to 128K tokens. Our training data and two-stage training strategy jointly address these challenges while keeping the answering LLM frozen. Broad reconstruction pretraining establishes information-preserving compressed representations, and multi-task SFT develops their usefulness across downstream tasks and extends their context-length coverage. The evaluations across LongBench, LooGLE, CCEval, and FRAMES assess transfer across different domains and task requirements. The additional evaluations in R1 directly demonstrate effective compression on contexts approaching 128K.
MLA [1] addresses a different compression axis within the language model. It forms a low-dimensional latent state for each token to represent its keys and values, reducing the KV-cache dimension per source position. MemToken instead reduces the number of source positions presented to the answering LLM, mapping n source tokens to m=⌈n/c⌉ input-level memory tokens. The answering LLM's weights and architecture remain unchanged. At 4× compression, approximately 128K source tokens occupy 32K input positions, releasing context-window capacity for additional material.
Within this framework, channel-wise aggregation combines contextualized source features into compact memory states. Prefix blocks propagate information across group boundaries before aggregation, and suffix blocks further integrate the compressed sequence. This supports information redistribution across a context with uneven information density. The operator's normalization and design rationale are explained in R4.
We will foreground downstream domain generalization and scaling compression to 128K as MemToken's central contributions, explain how our data and training strategy enable them, and clarify the role of attention-based aggregation within the overall framework.