- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies context compression for frozen language models from the perspective of information preservation and predictive mismatch. The authors argue that retaining information in compressed representations does not necessarily guarantee that the target model can effectively utilize it. They formalize this distinction through a decomposition of predictive log-loss and propose LatentWeave, a framework that combines a context compressor with a lightweight residual adapter to improve the compatibility between compressed memories and the frozen reader. The method uses KV and attention supervision to initialize the adapter, followed by joint compressor–adapter refinement through predictive feedback. Experiments on LongMemEval show improvements over existing compression methods across several Qwen models, with reduced inference latency and, in some settings, slightly better accuracy than the uncompressed baseline.
Critical strengths
-
The paper provides an interesting perspective on context compression by distinguishing information preservation from the ability of frozen language models to effectively utilize compressed representations. The theoretical analysis offers a useful explanation for why compression can sometimes improve predictive performance.
-
LatentWeave introduces a lightweight residual adapter that improves compressed memory representations without modifying the target LLM. The proposed framework is relatively simple and demonstrates promising effectiveness.
-
The experiments cover multiple Qwen model families and compression ratios, showing consistent improvements over LatentPress. The ablation studies further support the benefits of residual adaptation and joint refinement, while the end-to-end latency results demonstrate practical efficiency gains.
Critical weaknesses
-
The novelty of the main motivation is not sufficiently distinguished from prior work. Existing studies, such as [1], have already explored the mismatch between conventional compression objectives and downstream task performance. The paper would benefit from a clearer discussion of how its proposed perspective differs from and advances these existing efforts.
-
The baseline comparisons are not sufficiently comprehensive. Several relevant context compression methods, such as [2] and [3], are missing from the evaluation. Including these approaches, where comparable experimental settings can be established, would provide a more convincing assessment of the proposed method's effectiveness.
-
The paper does not sufficiently justify its decision to leave user turns uncompressed. While this may be reasonable for traditional conversations, modern agentic workflows often involve lengthy tool outputs, retrieved documents, and other observations that substantially increase the proportion of non-assistant tokens. The lack of experiments on agentic or tool-augmented tasks makes it difficult to assess the applicability of the proposed method to these increasingly common long-context scenarios.
-
The connection between the theoretical analysis and the proposed method is not sufficiently clear. Although the paper decomposes predictive loss into information loss and predictive mismatch, neither is directly measured in the experiments. It remains unclear whether the improvements actually come from better representation alignment or simply from additional task-specific optimization.
[1] Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors
[2] Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models
[3] 500xcompressor: Generalized prompt compression for large language models
Questions for authors
see weakness