acceptodds
Under review as a conference paper at ICLR 2027

LatentWeave: Rethinking Context Compression through Residual Memory Adaptation

Abstract

Long-context inference imposes growing demands on computation, memory capacity, and data movement. Soft-token compression reduces these costs, yet preserving relevant information does not guarantee its effective use by a frozen language model. We decompose excess predictive log-loss into compression-induced information loss and predictive mismatch, showing that lossy compression can improve prediction when the reduction in mismatch outweighs the information lost. Guided by this analysis, we introduce **LatentWeave**, a residual memory adaptation framework that jointly optimizes memory construction and representation adaptation. LatentWeave couples a compressor with a shared low-rank residual adapter that applies nonlinear, content-dependent corrections to the complete history-memory prefix, including embeddings from uncompressed segments. Target-derived key-value (KV) and attention supervision initialize the adapter, after which both components are jointly refined through the frozen target. On LongMemEval, LatentWeave achieves higher mean overall accuracy than the uncompressed Qwen3-8B target across all evaluated nominal assistant compression factors from to , while reducing end-to-end inference latency by 39.3–49.5%. Across Qwen2.5-14B, Qwen3-8B, Qwen3.5-4B, and Qwen3.8-27B, LatentWeave also consistently outperforms the leading soft-token compression method LatentPress in mean overall and user-fact accuracy, with relative gains of up to 27.5% in mean overall accuracy. Component ablations further show that joint compressor–adapter refinement achieves lower reference-answer log-loss than updating either component alone. LatentWeave brings memory construction and representation adaptation into a unified framework, allowing context compression to improve predictive quality while reducing inference cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
kv cache compression, Latent press, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sdhGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies context compression for frozen language models from the perspective of information preservation and predictive mismatch. The authors argue that retaining information in compressed representations does not necessarily guarantee that the target model can effectively utilize it. They formalize this distinction through a decomposition of predictive log-loss and propose LatentWeave, a framework that combines a context compressor with a lightweight residual adapter to improve the compatibility between compressed memories and the frozen reader. The method uses KV and attention supervision to initialize the adapter, followed by joint compressor–adapter refinement through predictive feedback. Experiments on LongMemEval show improvements over existing compression methods across several Qwen models, with reduced inference latency and, in some settings, slightly better accuracy than the uncompressed baseline.

Critical strengths
  1. The paper provides an interesting perspective on context compression by distinguishing information preservation from the ability of frozen language models to effectively utilize compressed representations. The theoretical analysis offers a useful explanation for why compression can sometimes improve predictive performance.

  2. LatentWeave introduces a lightweight residual adapter that improves compressed memory representations without modifying the target LLM. The proposed framework is relatively simple and demonstrates promising effectiveness.

  3. The experiments cover multiple Qwen model families and compression ratios, showing consistent improvements over LatentPress. The ablation studies further support the benefits of residual adaptation and joint refinement, while the end-to-end latency results demonstrate practical efficiency gains.

Critical weaknesses
  1. The novelty of the main motivation is not sufficiently distinguished from prior work. Existing studies, such as [1], have already explored the mismatch between conventional compression objectives and downstream task performance. The paper would benefit from a clearer discussion of how its proposed perspective differs from and advances these existing efforts.

  2. The baseline comparisons are not sufficiently comprehensive. Several relevant context compression methods, such as [2] and [3], are missing from the evaluation. Including these approaches, where comparable experimental settings can be established, would provide a more convincing assessment of the proposed method's effectiveness.

  3. The paper does not sufficiently justify its decision to leave user turns uncompressed. While this may be reasonable for traditional conversations, modern agentic workflows often involve lengthy tool outputs, retrieved documents, and other observations that substantially increase the proportion of non-assistant tokens. The lack of experiments on agentic or tool-augmented tasks makes it difficult to assess the applicability of the proposed method to these increasingly common long-context scenarios.

  4. The connection between the theoretical analysis and the proposed method is not sufficiently clear. Although the paper decomposes predictive loss into information loss and predictive mismatch, neither is directly measured in the experiments. It remains unclear whether the improvements actually come from better representation alignment or simply from additional task-specific optimization.

[1] Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors

[2] Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models

[3] 500xcompressor: Generalized prompt compression for large language models

Questions for authors

see weakness

Reviewer_aNLmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatentWeave, which is a context-compression framework for frozen language models that jointly learns memory construction and representation adaptation. It analyzes predictive loss as compression-induced information loss plus predictive mismatch, showing that lossy compression can outperform raw context when mismatch reduction outweighs lost information. The method builds a memory prefix from pooled compressed segments and uncompressed token embeddings, then applies a shared low-rank residual adapter refined via KV-guided initialization and joint predictive training.

Critical strengths
  1. The paper decomposes predictive log-loss into compression-induced information loss and predictive mismatch, yielding the precise condition under which lossy compression can outperform raw context.
Critical weaknesses
  1. Stage I KV guidance requires an auxiliary KV mapper "restored from previously trained KV-reconstruction weights" and a pretrained compressor (Section E.4), neither of which is produced within the described pipeline, so the method is not self-contained and its Stage-I behavior depends on unspecified upstream training.
  2. The compressor partitions the history into fixed-size groups and emits exactly one pooled memory vector per group, so the compression budget is allocated uniformly across the context regardless of where answer-relevant information actually sits, which is likely to impose a rigid bottleneck.
Questions for authors

Missed related work [1]. Please add a discussion of this to the related work section.

[1] InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing

Reviewer_9MdCOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

LatentWeave studies soft-token compression for a frozen language model, emphasizing that preserving information does not guarantee its effective use by the target. The paper decomposes excess predictive log-loss into compression-induced information loss and predictive mismatch, showing that compressed memory can yield lower predictive log-loss than uncompressed inputs when the reduction in predictive mismatch outweighs the information lost.

The proposed residual memory adaptation framework jointly optimizes memory construction and representation adaptation. A compressor uses a two-layer encoder, gated fusion, and attention pooling to construct a history-memory prefix. A shared low-rank residual adapter applies nonlinear, content-dependent corrections to the complete prefix, including embeddings from uncompressed segments. Stage I starts from a pretrained compressor and a pretrained auxiliary KV mapper and updates only the adapter through KV-guided initialization. Stage II jointly refines the compressor and adapter using answer supervision, output-distribution distillation, and reference regularization through the frozen target.

Trained on UltraChat and evaluated on LongMemEval using evidence-bearing sessions, LatentWeave reports modest gains in overall accuracy and larger gains in user-fact accuracy over Raw with Qwen3-8B at nominal assistant compression factors of 8×–256×. These correspond to realized history ratios of 4.56×–9.65× because user turns remain uncompressed. End-to-end inference latency decreases by 39.3–49.5%. LatentWeave also achieves higher mean overall and user-fact accuracy than LatentPress across four frozen Qwen target models with separately trained memory interfaces. Ablations support the contributions of residual adaptation and KV guidance, with joint compressor–adapter refinement achieving the lowest reference-answer log-loss.

Critical strengths
  1. Useful framing of predictive mismatch, supported by fixed-compressor ablations. The paper distinguishes information preservation from the frozen target’s ability to use compressed memory through a decomposition of excess predictive log-loss into compression-induced information loss and predictive mismatch (Theorem 1). Although based on a standard identity, this provides a useful perspective on latent-memory compression. The fixed-compressor ablation supports the importance of representation adaptation: adding and refining the residual adapter reduces reference-answer log-loss from 5.08 to 2.61 nats/token without changing the compressor output (Table 4). This combination of conceptual framing and controlled empirical evidence is one of the main points of my positive assessment.

  2. Consistent improvements over the closest baseline under matched compression budgets. LatentWeave achieves higher mean overall and user-fact accuracy than LatentPress in all fifteen reported configurations across four frozen Qwen models, at identical realized history ratios (Tables 1 and 3). Both methods are independently retrained and evaluated over five seeds. Overall accuracy improvements vary from 0.04 to 7.52 percentage points, so their magnitude depends on the configuration. This strengthens my assessment because the comparison demonstrates improvements over the closest baseline without increasing the memory budget, providing evidence that the proposed memory interface and training procedure offer practical value across multiple target models. End-to-end inference latency reductions of 39.3–49.5% relative to Raw with Qwen3-8B, including online compression and adaptation, further support the method’s practical value.

Critical weaknesses
  1. The source of the accuracy improvement over Raw is insufficiently isolated. LatentWeave’s memory interface receives answer supervision and output-distribution distillation, whereas Raw receives no corresponding adaptation. Theorem 1 establishes a condition under which compressed memory can improve predictive log-loss, but does not identify the mechanism responsible for the empirical gains. Table 4 reports that compressed memory without an adapter already reduces reference-answer log-loss from 6.445 for Raw to 5.08 nats/token. These observations do not establish whether the advantage depends on compression or could also be obtained through supervised adaptation of uncompressed input representations. An uncompressed-input adaptation baseline trained with comparable supervision would clarify this distinction. Furthermore, Raw comparisons are limited to Qwen3-8B, where mean overall accuracy gains are 0.28–1.44 percentage points without paired significance tests. Evaluation uses only evidence-bearing sessions, leaving effectiveness on complete histories containing irrelevant sessions untested. This is consequential because outperforming uncompressed inference is a headline result, yet the source, robustness, and broader applicability of the accuracy advantage remain insufficiently established.

  2. The contribution beyond LatentPress is not sufficiently isolated from additional optimization and pretrained dependencies. LatentWeave uses a soft-token interface closely related to LatentPress, together with a residual adapter, KV-guided initialization, and joint compressor–adapter refinement. Matching realized history ratios does not establish matched training compute or trainable parameter budgets. Although Table 4 includes adapter-only refinement, compressor-only refinement, and joint refinement without KV guidance, it lacks a control that gives the underlying compressor comparable additional predictive training with the adapter absent. The inherited pretrained compressor and auxiliary KV mapper also introduce dependencies whose pretraining procedures and costs are insufficiently documented. Stage I combines ten weighted loss components and Stage II four, without targeted ablations or sensitivity analysis establishing which are necessary. Moreover, the component ablations report reference-answer log-loss rather than generated-answer accuracy. This weakens my assessment because the proposed architecture and training procedure constitute the main methodological contribution, but their specific contribution to downstream accuracy and justification for the added complexity remain unclear.

Questions for authors

Clarifications and additional evidence for the current claims

  1. Is the gain over Raw specific to compressed memory? LatentWeave’s compressor and adapter receive answer supervision and output-distribution distillation, whereas Raw is not adapted. How does LatentWeave compare with an uncompressed-input baseline that applies the same residual adapter to raw history embeddings, keeps the target frozen, and uses the same supervision and data? Please report trainable parameter counts and training compute for both. This would help distinguish benefits of compression from benefits of learned input adaptation.

  2. Does the accuracy improvement extend beyond single-session user facts? Please report the number of questions contributing to overall and user-fact accuracy, clarify whether abstention questions are included in the user-fact category, and specify how overall accuracy is aggregated. If overall accuracy averages 500 questions and user-fact accuracy averages 64 questions, Table 1 implies that, at 16×, accuracy on the remaining questions decreases from approximately 47.80% for Raw to 46.83% for LatentWeave. Under these assumptions, aggregate accuracy on the remaining questions decreases at every reported compression factor. Is this interpretation correct? Please provide per-category results to clarify whether the overall improvement is concentrated in single-session user facts.

  3. What do the inherited components and additional training contribute? Appendix E.4 states that Stage I starts from a pretrained compressor and a pretrained auxiliary KV mapper. How were these trained, including their data, objectives, and computational cost, and is the compressor a LatentPress checkpoint? Table 4 lacks a control in which the compressor receives comparable additional predictive training with the residual adapter absent.

  4. What context lengths does the evaluation cover? Each question is paired only with its evidence-bearing sessions. What are the history-length distributions in this setting, and are results available on complete LongMemEval histories containing irrelevant sessions? Did any training histories exceed the target’s context window? If so, how were the uncompressed-target predictions required by Stage II obtained?

Questions 1–3 would most directly affect my assessment.

Suggestions for future work

  1. Extend evaluation to full LongMemEval histories and a non-conversational long-context benchmark, examining performance as context length increases.
  2. Investigate alternative pooling operations, adapter architectures, and encoder depths; ablate individual Stage I and Stage II loss terms and examine sensitivity to their weights, reporting generated-answer accuracy alongside reference-answer log-loss.
  3. Report component-wise latency, peak GPU memory, and inference latency against LatentPress at matched realized history ratios.
  4. Test whether a single trained interface transfers across compression factors without retraining.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.