acceptodds
Under review as a conference paper at ICLR 2027

MemToken: Scaling Query-Agnostic Context Compression across Tasks and Context Lengths

Abstract

Long contexts in real-world applications make the inference of Large Language Model (LLMs) expensive since attention prefill and key–value cache costs grow with sequence length, which motivates the research on context compression. Soft token compression aims to encode a long context into a short sequence of continuous vectors that can be directly inputed into the LLM and replace the original context. Existing query-agnostic soft compressors are typically trained on limited data distributions and relatively short contexts, leaving generalization to unseen data and longer sources unresolved. To address these limitations, we introduce MemToken, a query-agnostic soft token compressor that condenses groups of context representations into memory tokens consumed by a frozen LLM. To improve distributional generalization, we construct a broad-coverage training set that combines web-text reconstruction with synthetic supervised samples across diverse downstream tasks. To scale source length, we use a short-context reconstruction warmup, interleave short and long contexts, and fine-tune on longer task contexts. Across four unseen long-context benchmarks with different LLMs, MemToken scales the compressed context length to 128K and significantly outperforms the state-of-the-art methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Token compression, large language models, long-context inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_no2jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduced MemToken, a query-agnostic compression method for LLMs. It constructs a new dataset for long-context LLM compression with subsets for pre-training and supervised fine-tuning. MemToken uses small attention module that compresses a chunk of token hidden states into one, which serves as the surrogate hidden state for LLM attention. The compression training contains multiple stages, which include a short context warmup, a mixed short and long contexts adaption, and a supervised fine-tuning training. The resultant method is validated on context length up to 128k tokens and outperform several baselines.

Critical strengths
  1. MemToken releases both its method and training data. It can future development and fair comparison.
  2. It explains its training design strategy with experiments, such as the choice of warmup training is based on poor performance with straightforward long context training.
  3. The results looks good with clear advantage over baseline methods.
  4. The paper is well written and easy to follow. The appendices are extensive.
Critical weaknesses
  1. Although it claims to support context length up to 128k tokens, it appears only at pretraining stage. The context length of the SFT stage does not exceed 64k tokens, and the context length of the evaluation benchmarks is not discussed in the paper. The authors should consider discussing the context length of the benchmarks and introducing tasks that can reach 128k tokens, such as RULER.
  2. The idea of compressing multiple hidden states into one state isn't new. For example, the commonly adopted multihead latent attention is similar, though not operating at the group level. The author should better differentiate their method from literature.
  3. Some of the design choices are confusing to me, see my question section.
  4. There are multiple citations in the reference section containing the wording "VERIFY citation." Is this a message left by AI tools?
Questions for authors
  1. In section 4.3, the authors said directly applying long context training makes the model to ignore long context, therefore a short-context warmup is necessary. Can you explain more on why model ignores long context, and what role does the warmup play?
  2. In section 4.1, MemToken compresses a group of hidden states into one by using a mini-attention. The softmax distribution is of shape c×dc \times d, which is strange to me: Does this mean that the distribution is calculated over each individual dimension? If so, why is this perferred over a token-level or multi-head level softmax?
3 comments
AuthorsOpenReview ↗

Response to the reviewer's concerns (1/3)

We thank the reviewer for the constructive feedback. We provide additional downstream evaluations on contexts approaching 128K, clarify how MemToken differs from Multi-head Latent Attention, and explain the optimization rationale for short-context warmup and channel-wise aggregation.

R1. Downstream evaluation on contexts approaching 128K (Weakness 1)

To directly assess compression quality near 128K, we have added evaluations on LongBench Pro restricted to 64K–128K contexts and InfiniteBench restricted to 100K–128K contexts, using Qwen3.5-9B. These evaluations also demonstrate length generalization beyond this backbone's 64K SFT cap.

MethodLongBench Pro, 64K–128K, 4×
MemToken0.559
KVzip0.369
LLMLingua-20.350
Selective-Context0.319
ARC-Encoder0.130

MemToken exceeds the strongest evaluated baseline, KVzip, by 0.190 on the reported LongBench Pro metric. On InfiniteBench, we discard complete examples exceeding 128K and report total accuracy below.

CompressionMethodAccuracy (%)
—No context4.15
—Full text88.25
4×First+last28.16
4×LLMLingua-212.00
4×KVzip59.97
4×MemToken80.53
8×First+last17.10
8×LLMLingua-212.60
8×KVzip53.11
8×MemToken71.01

MemToken achieves 80.53% and 71.01% accuracy at 4× and 8× compression, improving over KVzip by 20.56 and 17.90 percentage points, respectively. These results provide direct downstream evidence of effective compression on the evaluated contexts approaching 128K. They address the reviewer's request for tasks reaching this length regime through two additional benchmarks.

We will include these results and their length-filtering protocols in the revised paper. We will also expand the benchmark description to report source-context length statistics and the number of examples above 64K for each evaluated configuration, making the length coverage directly visible alongside the aggregate scores.

R2. Core contributions and relation to Multi-head Latent Attention (Weakness 2)

MemToken advances query-agnostic soft context compression through downstream domain generalization and effective compression of source contexts up to 128K tokens. Our training data and two-stage training strategy jointly address these challenges while keeping the answering LLM frozen. Broad reconstruction pretraining establishes information-preserving compressed representations, and multi-task SFT develops their usefulness across downstream tasks and extends their context-length coverage. The evaluations across LongBench, LooGLE, CCEval, and FRAMES assess transfer across different domains and task requirements. The additional evaluations in R1 directly demonstrate effective compression on contexts approaching 128K.

MLA [1] addresses a different compression axis within the language model. It forms a low-dimensional latent state for each token to represent its keys and values, reducing the KV-cache dimension per source position. MemToken instead reduces the number of source positions presented to the answering LLM, mapping nn source tokens to m=⌈n/c⌉m=\lceil n/c\rceil input-level memory tokens. The answering LLM's weights and architecture remain unchanged. At 4× compression, approximately 128K source tokens occupy 32K input positions, releasing context-window capacity for additional material.

Within this framework, channel-wise aggregation combines contextualized source features into compact memory states. Prefix blocks propagate information across group boundaries before aggregation, and suffix blocks further integrate the compressed sequence. This supports information redistribution across a context with uneven information density. The operator's normalization and design rationale are explained in R4.

We will foreground downstream domain generalization and scaling compression to 128K as MemToken's central contributions, explain how our data and training strategy enable them, and clarify the role of attention-based aggregation within the overall framework.

AuthorsOpenReview ↗

Response to the reviewer's concerns (2/3)

R3. Why short-context warmup helps the frozen LLM use compressed memory (Question 1)

The optimization difficulty arises from teacher forcing combined with the initial mismatch between compressed representations and the frozen LLM's input space. During reconstruction, the LLM receives the compressed memory, a repetition instruction, and the gold reconstruction prefix. At initialization, the memory has not yet learned to convey source information in a form the LLM can use. The frozen LLM can nevertheless predict many subsequent tokens from the growing gold prefix using its existing language-modeling ability.

This alternative prediction path becomes particularly strong for long reconstruction targets. Later tokens have a substantial preceding text prefix, allowing the model to obtain a relatively low next-token loss while making limited use of compressed memory. The first few output positions depend more strongly on the memory because little source text has yet been exposed through teacher forcing. As the target grows, those positions account for a smaller fraction of the average token loss. This provides an optimization explanation for the observed behavior in Figure 1(b).

Figure 1(b) compares compression training with a plain autoregressive reference that receives no compressed memory. With 512-token sources, the compression loss falls clearly below this reference as training proceeds. With 65,536-token sources trained directly from initialization, the losses remain close, indicating that the compressed tokens provide little additional predictive benefit in that setting.

Short-context warmup first establishes a useful communication interface between the compressor and the frozen LLM. With shorter reconstruction targets, a larger share of the prediction task depends on information supplied by the compressed memory. Once the compressor learns representations that support reconstruction, the same interface can be trained on longer inputs. The answering LLM remains frozen throughout, so this alignment is learned by the compressor.

In our reported recipe, Stage 1 starts with 80,000 contexts of 128–255 tokens and then mixes the six eligible length bins below 8,192 tokens, using 1,500,000 contexts in total. Stage 2 extends this capability to longer contexts through question answering, positional lookup, in-context induction, and code understanding. We will clarify this sequence and explain that the mixed lengths in Stage 1 remain below 8K, with the larger length extension occurring in Stage 2.

R4. The normalization axis and rationale for channel-wise aggregation (Weakness 3 and Question 2)

Yes, each output channel has its own distribution over the cc positions in a group. The softmax is taken along the group-position axis. For Hj∈Rc×dH_j\in\mathbb{R}^{c\times d}, let Uj=HjW2+PU_j=H_jW_2+P and Vj=HjW1V_j=H_jW_1. The aggregation is

$ A_{j,i,k} =\frac{\exp(U_{j,i,k})}{\sum_{r=1}^{c}\exp(U_{j,r,k})}, \qquad \widetilde H_{j,k} =\sum_{i=1}^{c}A_{j,i,k}V_{j,i,k}. $

For every output channel kk, ∑i=1cAj,i,k=1\sum_{i=1}^{c}A_{j,i,k}=1. Thus, Aj∈Rc×dA_j\in\mathbb{R}^{c\times d} contains dd distributions over the cc group positions. There is no normalization across the feature dimension or across all cdcd entries jointly.

The purpose is to preserve complementary features from different source positions when forming one memory state. A token-level pooling weight assigns the same positional mixture to every output channel. Head-level pooling allows different mixtures between heads but shares a positional distribution among channels within each head. Channel-wise aggregation allows each projected output channel to select its own mixture. For example, one channel can draw more strongly from an earlier position while another retains information from a later position, within the same compressed token.

The learned dense projections are also important. Each column of HjW2H_jW_2 is computed from the full hidden vector at that position, and HjW1H_jW_1 provides a learned value representation. Consequently, channel-wise weighting operates on learned features informed by all input dimensions. It does not assume that individual original hidden dimensions are independent or correspond to predefined semantic attributes. Tying the positional weights across output channels recovers the weight-sharing pattern of token-level pooling, while tying them within channel groups recovers the corresponding head-level pooling pattern.

We will describe this operator more explicitly as channel-wise gated pooling and add the indexed equation above. This makes its normalization and its role in feature aggregation clear, while distinguishing it from conventional query-key multi-head attention.

AuthorsOpenReview ↗

Response to the reviewer's concerns (3/3)

R5. Reference-verification annotations (Weakness 4)

The phrase “VERIFY citation” came from internal reference-checking reminders stored in the note fields of our BibTeX entries. The three original entries are shown below.

@misc{qwen35,
  title  = {Qwen3.5 Technical Report},
  author = {{Qwen Team}},
  year   = {2026},
  note   = {VERIFY citation}
}

@misc{nemotron3nano,
  title  = {{NVIDIA Nemotron 3 Nano}},
  author = {{NVIDIA}},
  year   = {2026},
  note   = {VERIFY citation}
}

@misc{deepseekv4,
  title  = {{DeepSeek-V4}},
  author = {{DeepSeek-AI}},
  year   = {2026},
  note   = {VERIFY citation; judge model DeepSeek-V4-Flash-0731}
}

We overlooked that the bibliography style used by the ICLR template renders these note fields in the reference list. All cited references are genuine and were not generated by AI. We will remove these internal notes from the BibTeX entries in the revised manuscript.

References

[1] Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., ... & Xu, Z. (2024). Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434.

Reviewer_xo8AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MemToken, a query-agnostic soft context compression method that condenses long contexts into continuous memory tokens via a learnable token aggregation module, combined with reconstruction pretraining and multi-task SFT to improve the generalization of compressed representations across diverse tasks and context lengths. Experiments show that the proposed method achieves strong performance across multiple long-context benchmarks.

Critical strengths
  1. Practical significance: Query-agnostic context compression is of great importance for long-document understanding and multi-turn query scenarios.
  2. Strong empirical performance: The method achieves competitive results across multiple benchmarks, model scales, and compression ratios.
  3. Well-motivated training recipe: The two-stage training strategy is specifically designed to address optimization difficulties in long-context reconstruction training, and its effectiveness is supported by relevant ablation studies.
Critical weaknesses
  1. Insufficient comparison with existing Soft Context Compression methods. Most baselines in the main experiments are based on hard token compression or KV-cache pruning, while ARC-Encoder is the only directly comparable learned soft-compression baseline. Comparisons with other representative methods, such as PCC (ACL 2025), Activation Beacon (ICLR 2025), and SAC (ICLR 2026), are missing. Moreover, ARC-Encoder and MemToken are trained under different settings, while the retrained mean-pooling variant in Table 2 already achieves performance close to MemToken on LongBench. Therefore, it remains difficult to disentangle the contributions of the proposed compressor architecture from those of the training data and training strategy.
  2. Insufficient validation of 128K long-context compression capability. Although the paper emphasizes supporting 128K contexts, Stage 1 actually only trains on samples shorter than 8K, the proportion of samples exceeding 64K in Stage 2 is also low, and the SFT length cap for both Qwen3.5-9B and Nemotron is 64K. While Section 5.3 provides performance analyses across different context lengths, it does not separately report downstream task accuracy in the 64K–128K range. Therefore, current experiments are insufficient to fully verify information retention within this length regime.
Questions for authors
  1. On soft-compression baselines: Could the authors provide comparisons with other representative soft-compression methods under comparable experimental settings, where feasible, to better isolate the contribution of the proposed architecture? In particular, could the authors further discuss the differences from SAC, Activation Beacon, and PCC?
  2. On the autoencoding training objective: Although MemToken's Stage 2-only ablation performs poorly, works such as SAC suggest that autoencoding may not always be necessary under alternative compression architectures. Have the authors considered replacing reconstruction pretraining with a language modeling objective (e.g., predicting subsequent text conditioned on compressed representations) under a comparable training budget? Such an experiment could help clarify the role of the reconstruction objective relative to the two-stage training strategy.
  3. On long-context scalability: Could the authors further provide downstream task accuracy within the 64K–128K range and analyze the impact of different evidence positions on compression performance?
  4. On practical efficiency: Appendix B.10 (Table 17) shows that when the full context also uses KV-cache reuse, MemToken's latency advantage is relatively limited, whereas the reduction in cache storage overhead is more pronounced. Could the authors further clarify the primary target deployment scenarios for MemToken in the main text and discuss the Accuracy–Latency–Memory trade-off under different caching strategies?
4 comments
AuthorsOpenReview ↗

Response to the reviewer's concerns (1/3)

We thank the reviewer for the constructive feedback. We have expanded the learned-compression comparisons and added evaluations on contexts approaching 128K. Below, we clarify the contributions of the training recipe and aggregation operator, explain the role of reconstruction pretraining, and specify the deployment scenarios and caching trade-offs.

R1. Soft-compression comparisons and architectural contributions (Weakness 1 and Question 1)

To broaden the comparison, we added two learned soft-compression methods, LCIRC [1] and GMSA [2]. We retrained both on the Qwen3.5-9B backbone using their respective official implementations and training datasets, and evaluated them at 4× and 8× compression.

CompressionMethodLongBench (judge)LooGLECCEvalFRAMES
4×GMSA0.3860.3800.1040.290
4×LCIRC0.3670.3590.0970.273
4×MemToken0.5260.5820.1910.399
8×GMSA0.2970.2810.0690.214
8×LCIRC0.2810.2590.0620.195
8×MemToken0.4530.4820.1660.367

MemToken outperforms both additional baselines on all four benchmarks at both ratios, with absolute score gains of 0.087–0.202 over the stronger baseline. These results extend the evaluation beyond ARC-Encoder using a shared backbone and compression ratios. Each method retains its own training data and recipe.

PCC [3] shares MemToken's frozen-reader setting and represents the compressed context as input-level memory embeddings. It compresses independent segments into special memory tokens and uses a converter to align them with the downstream LLM, combining reconstruction and text-completion pretraining before domain-level fine-tuning. MemToken processes the full context globally and places aggregation inside the compressor stack, allowing contextualization across segment boundaries before and after compression.

Activation Beacon [4] progressively compresses chunks into beacon tokens' layer-wise KV activations. The original LLM parameters remain frozen, while additional beacon projections are trained through compression-based autoregression. Its compression and generation operations are integrated through the KV-memory interface. MemToken uses a separate compressor to produce reusable input embeddings for an answering LLM whose architecture remains unchanged.

SAC [5] derives compressed representations from selected context tokens, which are marked with learnable anchor embeddings and encoded with bidirectional attention. It retains the anchors' KV states and trains the compressor with language modeling followed by task fine-tuning, while freezing the target LLM. MemToken instead aggregates globally contextualized states into input embeddings and trains them through reconstruction followed by multi-task SFT.

The published results of these methods use different backbones, data, and training protocols, so controlled comparisons require a shared evaluation setting. For KV-based methods, retained positions and stored bytes are distinct budget measures. We will report both to specify the representation budget.

Table 2 includes a Mean pool variant that fixes the selector to the window mean from the start of training and retrains the compressor with the same training recipe. This provides a controlled comparison of the aggregation operator on Qwen3.5-2B at 4×:

VariantLongBench (official)LongBench-QA (judge)LooGLE (token-F1)LooGLE (judge)
MemToken0.3670.4060.2750.438
Mean pool0.3630.4100.2620.405

The strong LongBench results of both variants demonstrate that our training recipe is effective across aggregation operators. The learned selector provides additional gains on LooGLE, improving token-F1 by 0.013 and judge accuracy by 0.033. The selector-override analyses in Appendix B.5 further associate its contribution with positional fidelity on retrieval tasks. We will foreground these complementary contributions: the training recipe establishes broadly usable compressed memory, and the learned aggregation improves retention on tasks sensitive to the selected information.

AuthorsOpenReview ↗

Response to the reviewer's concerns (2/3)

R2. Reconstruction versus continuation language modeling (Question 2)

Reconstruction pretraining is designed to learn semantically lossless compression of the complete context. By using the entire source as the reconstruction target, it provides supervision for recovering information at every source position, including the beginning of the context. This matches the goal of query-agnostic memory, because future questions may require details from any part of the source.

The continuation objective predicts subsequent text from the compressed representations. Because much of this prediction can rely on recent context, it can favor compressed representations that mainly retain information from the source suffix. A low continuation loss therefore need not require effective preservation of earlier source information. In our long-context setting, this local prediction shortcut can bias the compressor toward retaining the semantics of the context suffix while losing information from the distant prefix. Full-context reconstruction directly supervises recovery across the source and encourages the compressor to retain both early and late information.

Following the reviewer's suggestion, we replaced the Stage-1 reconstruction loss with a language-modeling objective that predicts subsequent text conditioned on compressed representations. This variant is denoted as Auto-regression in the table, and its downstream performance is evaluated after Stage-2 SFT on Qwen3.5-9B.

CompressionStage-1 objectiveLongBenchLooGLECCEvalFRAMES
4×Reconstruction (MemToken)0.5260.5820.1910.399
4×Auto-regression0.2270.1860.0310.138
8×Reconstruction (MemToken)0.4530.4820.1660.367
8×Auto-regression0.2190.1640.0230.107

Reconstruction pretraining outperforms this continuation objective across all four benchmarks at both compression ratios, with absolute score gains of 0.143–0.396. The Auto-regression variant remains substantially weaker after Stage-2 SFT. These results indicate that continuation pretraining does not establish a sufficiently informative compressed representation for the frozen LLM to use effectively, and subsequent SFT does not recover comparable downstream performance.

Our curriculum first uses short-context reconstruction to establish information-preserving compression and recovery, then extends these representations to longer contexts through multi-task SFT. SAC [5] combines semantic anchors and bidirectional KV encoding with language-modeling pretraining. The new objective ablation evaluates continuation language modeling within MemToken's global input-embedding compressor and demonstrates the advantage of full-context reconstruction in this setting. We will include this comparison and its training protocol in the revised paper.

R3. Downstream performance near 128K and evidence positions (Weakness 2 and Question 3)

We add two evaluations using Qwen3.5-9B: LongBench Pro restricted to 64K–128K contexts and an InfiniteBench subset containing 100K–128K contexts. These evaluations directly test length generalization beyond the 64K SFT cap of Qwen3.5-9B. The reconstruction stage below 8K serves as an efficient initialization, followed by long-context task adaptation. The reported SFT cap is 128K for Qwen3.5-2B and 64K for Qwen3.5-9B and Nemotron-3-Nano-4B.

On the 64K–128K LongBench Pro subset at 4×, MemToken obtains 0.559, exceeding the strongest evaluated baseline, KVzip, by 0.190 on the reported metric.

MethodLongBench Pro, 64K–128K, 4×
MemToken0.559
KVzip0.369
LLMLingua-20.350
Selective-Context0.319
ARC-Encoder0.130

On the 100K–128K InfiniteBench subset, we exclude complete examples exceeding 128K and report total accuracy:

CompressionMethodAccuracy (%)
—No context4.15
—Full text88.25
4×First+last28.16
4×LLMLingua-212.00
4×KVzip59.97
4×MemToken80.53
8×First+last17.10
8×LLMLingua-212.60
8×KVzip53.11
8×MemToken71.01

MemToken achieves 80.53% and 71.01% at 4× and 8×, improving over KVzip by 20.56 and 17.90 percentage points, respectively. These results provide direct downstream evidence of useful information retention on the evaluated contexts approaching 128K.

We will add the length-specific results and filtering protocol to the revised paper.

AuthorsOpenReview ↗

Response to the reviewer's concerns (3/3)

R4. Context-window capacity and deployment trade-offs (Question 4)

MemToken's primary purpose is to reduce the number of positions that source context occupies in the answering LLM's limited context window. By replacing a long context with a shorter sequence of memory tokens, it leaves more room for additional documents, interaction history, new inputs, and generated responses. At 4× and 8× compression, a source of approximately 64K tokens occupies about 16K and 8K input positions, respectively. This increases the amount of source content that can be represented within a fixed context window.

This objective is particularly relevant to document-intensive assistants, repository-level assistance, and agents with persistent interaction histories. Their accumulated context competes with new inputs for a finite context budget. MemToken allows previously encountered material to remain available in compact memory form while preserving capacity for subsequent content. Full-text KV-cache reuse avoids repeated prefill, but the cached source still occupies its original context positions. MemToken reduces that positional footprint.

The primary trade-off is therefore between compression quality, the number of context positions released, and retained memory. We summarize task quality and KV-cache costs for Qwen3.5-9B at 64K below. Task quality is measured by MuSiQue F1. Bold marks the better compressed method at each ratio.

Method and storage policyMuSiQue F1Retained state per context (MB)Total latency, Q=1 (ms)Total latency, Q=16 (ms)
Full text, KV cache0.4472,105.36,0299,256
MemToken 4×, KV cache0.326546.46,3139,814
KVzip 4×, KV cache0.2512,105.317,25420,951
MemToken 8×, KV cache0.390286.65,4588,988
KVzip 8×, KV cache0.1002,105.317,28421,353

F1 scores are from Table 16, evaluated on 56 MuSiQue questions at 64K. Latency and retained state are from Table 17, measured over eight contexts per length. These summarize separate quality and cost evaluations. KVzip's compression ratios are defined on KV-cache bytes. Its retained allocation in Table 17 remains full-sized because the evaluated kvpress implementation masks evicted entries without freeing their storage.

At 4× and 8×, MemToken improves MuSiQue F1 over KVzip by 0.075 and 0.290, respectively. Its total latency for 16 queries is less than half that of KVzip at both ratios. The measured retained state is also smaller, reaching an 86.4% reduction at 8× relative to the full-text cache. These results highlight MemToken's quality and cost advantages over KVzip in the evaluated setting, alongside its primary benefit of releasing context-window capacity.

We will present task quality, released context capacity, and cache costs together under explicit evaluation and caching assumptions. Per-context state will also be distinguished from peak GPU memory, since the current Qwen3.5-9B implementation keeps a separate compressor copy resident, adding about 15 GB of model overhead.

We will foreground context-window utilization as the primary deployment objective in the main text. The efficiency discussion will separately report compression cost, answer-time cost, retained state, and peak GPU memory.

References

[1] An, S., Sung, J., Park, W., Park, C., & Seo, H. (2025, April). Lcirc: A recurrent compression approach for efficient long-form context and query dependent modeling in llms. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 10431-10442).

[2] Tang, J., Zhang, Z., Wu, S., Ye, J., Bai, L., Wang, Z., ... & Kim, H. G. (2026, July). Gmsa: Enhancing context compression via group merging and layer semantic alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 28690-28704).

[3] Dai, Y., Lian, J., Huang, Y., Zhang, W., Zhou, M., Wu, M., ... & Liao, H. (2025, July). Pretraining context compressor for large language models with embedding-based memory. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 28715-28732).

[4] Zhang, P., Liu, Z., Xiao, S., Shao, N., Ye, Q., & Dou, Z. (2025, May). Long context compression with activation beacon. In International Conference on Learning Representations (Vol. 2025, pp. 101932-101948).

[5] Liu, X., Zhao, R., Huang, P., Liu, X., Xiao, J., Xiao, T., ... & Zhu, J. (2026, April). Autoencoding-free context compression for llms via contextual semantic anchors. In International Conference on Learning Representations (Vol. 2026, pp. 138643-138665).

Thanks for the additional experiments. My concern about the fairness of the comparison remains: the architecture and the training recipe are still entangled, and none of the baselines are trained with your recipe. If the recipe is the main contribution, it should be shown to work across different architectures. Therefore, I am keeping my score.

Reviewer_4M2vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemToken, a query-agnostic soft token compression method that converts long contexts into compact continuous representations for frozen LLMs. It introduces a group-based compression architecture and a two-stage training strategy combining reconstruction pretraining with multi-task supervised fine-tuning to improve generalization across tasks and context lengths. Experiments on four long-context benchmarks demonstrate competitive performance at 4× and 8× compression ratios.

Critical strengths
  1. The experimental results demonstrates promising performance, suggesting the effectiveness of MemToken for query-agnostic context compression.

  2. The two-stage training strategy combines reconstruction pretraining with diverse supervised tasks, providing a systematic approach to improving generalization across different downstream tasks and context lengths.

  3. The paper provides extensive ablation studies and efficiency analyses, offering useful insights into the contributions of the proposed components and the computational benefits of context compression.

Critical weaknesses
  1. ARC-Encoder is the only soft token compression baseline in the main experiments, making it difficult to assess MemToken's architectural advantages over other soft token compression methods.

  2. The main results focus on Qwen3.5 models, while the additional Nemotron results show weaker performance on some benchmarks. Since many SFT answer targets are derived from Qwen3.5-2B, generalization across model families remains insufficiently established.

  3. Although the paper claims generalization to 128K-token contexts, the evaluated benchmarks do not provide sufficient evidence of compression performance at such lengths. Most evaluations focus on shorter contexts, and the longest context-length bin in Figure 4(a) groups all contexts above 32K together, without separately reporting performance in the 64K–128K range.

Questions for authors
  1. How does MemToken compare with other soft token compression methods, such as LCIRC [1] and GMSA [2], under comparable compression ratios and training settings?

  2. Could you provide further experimental evidence on a different LLM family and clarify how the Qwen3.5-2B-based SFT answer generation affects cross-model generalization, particularly given the weaker Nemotron results?

  3. Could you provide additional evaluations on benchmarks containing 100K+ token contexts, such as InfiniteBench or L-Eval, to demonstrate that MemToken maintains its compression quality at lengths approaching 128K?

  4. Section 4.2.1 describes a pretraining dataset covering contexts up to 128K, but Appendix A.4 states that only contexts shorter than 8K are used in Stage 1. The usage of the remaining data and the construction of the Stage 2 reconstruction replay set are unclear.

  5. Unlike segment-based approaches such as AutoCompressors [3], and LLoCO [4], MemToken compresses the entire context in a single global pass. What advantages does this design offer over compressing multiple segments separately?

[1] An et. al., LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs, NAACL 2025

[2] Tang et. al., GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment, ACL 2026

[3] Chevalier et. al., Adapting Language Models to Compress Contexts, EMNLP 2023

[4] Tan et. al., LLoCO: Learning Long Contexts Offline, EMNLP 2024

3 comments
AuthorsOpenReview ↗

Response to the reviewer's concerns (1/3)

We thank the reviewer for the constructive feedback. We add comparisons with LCIRC and GMSA, extend the baseline evaluation on Nemotron-3-Nano-4B across LongBench and ZeroSCROLLS, and provide additional long-context evaluations on LongBench Pro and InfiniteBench. We address each concern below.

R1. Additional soft token compression baselines (Weakness 1 and Question 1)

We thank the reviewer for suggesting these comparisons. GMSA fully fine-tunes the answering decoder for downstream tasks, which does not guarantee preservation of its original uncompressed performance. LCIRC freezes the pretrained LLM weights but introduces trainable gated cross-attention modules within the answering model. MemToken keeps both the weights and architecture of the answering LLM unchanged throughout training and optimizes only a separate compressor, preserving the original model for uncompressed inference.

To broaden the comparison, we retrained LCIRC and GMSA on the Qwen3.5-9B backbone using their respective official implementations and training datasets, and evaluated both methods at 4× and 8× compression.

CompressionMethodLongBenchLooGLECCEvalFRAMES
4×GMSA0.3860.3800.1040.290
4×LCIRC0.3670.3590.0970.273
4×MemToken0.5260.5820.1910.399
8×GMSA0.2970.2810.0690.214
8×LCIRC0.2810.2590.0620.195
8×MemToken0.4530.4820.1660.367

As shown in the above table, MemToken outperforms both additional baselines on all four benchmarks at both compression ratios, with absolute score gains of 0.087–0.202 over the stronger of the two baselines. We will include these results and clarify the method-specific training settings in the revised paper.

R2. Cross-family performance and Qwen-derived SFT targets (Weakness 2 and Question 2)

We thank the reviewer for highlighting this issue. The original manuscript reported MemToken’s results on Nemotron-3-Nano-4B but did not include comparisons with compression baselines on the same backbone. We have now added First+last, LLMLingua-2, Selective-Context, and KVzip on Nemotron-3-Nano-4B, evaluated on LongBench and ZeroSCROLLS at both 4× and 8× compression.

CompressionMethodLongBenchZeroSCROLLS
—No context0.1810.074
—Full context0.3760.304
4×First+last0.2320.200
4×LLMLingua-20.2020.190
4×Selective-Context0.1670.149
4×KVzip0.2230.196
4×MemToken0.3010.267
8×First+last0.2110.157
8×LLMLingua-20.1650.178
8×Selective-Context0.1580.125
8×KVzip0.1760.158
8×MemToken0.2460.203

As shown in the above table, MemToken outperforms all evaluated compression baselines on both benchmarks at both ratios. On LongBench, it achieves 0.301/0.246 at 4×/8×, compared with 0.232/0.211 for the strongest baseline. On ZeroSCROLLS, the corresponding scores are 0.267/0.203, compared with 0.200/0.178 for the strongest baseline. These comparisons provide additional evidence of MemToken’s effectiveness beyond the Qwen model family.

Our SFT dataset provides shared, task-grounded supervision across model families. Qwen3.5-2B generates candidate answers for the Wikipedia QA subset using the uncompressed context, and a candidate replaces the original target only when its token-F1 against the reference answer is at least 0.6. This reference-based quality control anchors the supervision to the task answers. Other tasks use passage-derived labels, sampled mappings, code-derived answers, or the source text itself. The same target set is used across all backbones, and the strong Nemotron-3-Nano-4B results obtained with this shared supervision further support the transferability of our SFT dataset beyond the Qwen family.

We will incorporate the expanded comparisons and document the shared SFT target construction in the revised paper.

AuthorsOpenReview ↗

Response to the reviewer's concerns (2/3)

R3. Compression quality at 64K–128K and 100K–128K (Weakness 3 and Question 3)

To evaluate compression quality near 128K, we add two evaluations using Qwen3.5-9B: LongBench Pro restricted to 64K–128K contexts, and an InfiniteBench subset containing 100K–128K contexts.

On the 64K–128K LongBench Pro subset at 4× compression, MemToken obtains 0.559, compared with 0.369 for KVzip, 0.350 for LLMLingua-2, 0.319 for Selective-Context, and 0.130 for ARC-Encoder. The improvement over the strongest evaluated baseline is 0.190 on the reported metric.

MethodLongBench Pro, 64K–128K, 4×
MemToken0.559
KVzip0.369
LLMLingua-20.350
Selective-Context0.319
ARC-Encoder0.130

We further evaluate Qwen3.5-9B on the 100K–128K InfiniteBench subset, excluding complete examples that exceed the 128K limit, and report total accuracy. MemToken achieves 80.53% accuracy at 4× compression and 71.01% at 8×, outperforming the strongest baseline, KVzip, by 20.56 and 17.90 percentage points, respectively. These results demonstrate strong overall accuracy on the evaluated contexts approaching 128K.

CompressionMethodAccuracy (%)
—No context4.15
—Full text88.25
4×First+last28.16
4×LLMLingua-212.00
4×KVzip59.97
4×MemToken80.53
8×First+last17.10
8×LLMLingua-212.60
8×KVzip53.11
8×MemToken71.01

Together, these evaluations extend the evidence beyond the original aggregated “>32K” bin and support MemToken’s effectiveness on the evaluated long-context subsets. We will include the results and filtering protocols in the revised paper.

R4. Efficient training and context-length expansion (Question 4)

Our reconstruction dataset covers contexts from 128 tokens to 128K. The reported training uses a compute-efficient curriculum: compression is first learned through reconstruction below 8K, and then extended to longer contexts through multi-task SFT. The dataset description in Section 4.2.1 gives the full corpus coverage, while Appendix A.4 specifies the subset used by this training recipe.

This design addresses the cost of long-context reconstruction. Stage 1 requires reproducing the entire source, so a 128K context entails approximately 128K supervised target tokens in addition to the compressed input. Even with teacher forcing, these long targets impose substantial decoder-side memory and computation costs. Task-oriented SFT supports long-context training with much shorter response targets, making it a more efficient way to extend compression to longer inputs.

The curriculum proceeds as follows. Stage 1 begins with 80,000 contexts of 128–255 tokens to establish alignment between compressed memory tokens and the frozen LLM's input space. It then mixes the six eligible length bins below 8,192 tokens, using 1,500,000 contexts in total. This warmup teaches the compressor to produce representations usable by the frozen LLM before training is extended to longer inputs.

Stage 2 extends the context range through question answering, positional lookup, in-context induction, and code understanding. Long inputs include prose augmented with shuffled distractor passages, synthetic demonstrations, and whole code files. For prose, evidence fractions of 10%, 5%, and 2% create longer contexts while retaining task-specific answer supervision. The SFT set covers contexts up to 128K: the reported training cap is 128K for Qwen3.5-2B and 64K for Qwen3.5-9B and Nemotron-3-Nano-4B. The new Qwen3.5-9B evaluations at 100K–128K further demonstrate generalization beyond its 64K SFT cap.

This curriculum establishes usable compression with short-context reconstruction and expands its length coverage through task supervision, avoiding reliance on costly 128K reconstruction pretraining. We will clarify the corpus coverage, training curriculum, and per-backbone length limits in Sections 4.2–4.3 and Appendix A.4.

AuthorsOpenReview ↗

Response to the reviewer's concerns (3/3)

R5. Global compression and information redistribution (Question 5)

Information is unevenly distributed across a long context: some segments contain dense semantic content, while others mainly provide background or redundant information. Compressing segments independently with uniform token budgets can leave capacity underused in information-sparse segments while constraining information-dense ones. Our global compression design aims to make better use of the limited memory-token budget by supporting information redistribution across representations.

MemToken processes the full context as one sequence. The prefix blocks contextualize source-token representations across segment boundaries before the intermediate compression layer, and the suffix blocks further integrate the compressed sequence. Consequently, local aggregation operates on contextualized states that can carry information from beyond their own groups. This allows cross-segment information to be propagated and consolidated before the final memory tokens are produced. The objective is to preserve more semantic information within the same overall token budget, particularly when information density varies substantially across the context.

We will clarify this motivation and the role of global contextualization in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.