acceptodds
Under review as a conference paper at ICLR 2027

LatentLM: Learning a Discrete Internal Language for Large Language Model

Abstract

In this paper, we study whether an off-the-shelf LLM can be adapted into a discrete, variable-length token compressor and decompressor for long-context processing. To this end, we design a self-expressive autoencoding framework that fine-tunes a pretrained LLM with lightweight LoRA adapters to map long texts into compact sequences of learned latent codes, termed Z-tokens, and to decode them back into natural language or task outputs. The resulting representation is content-adaptive: less predictable or more information-dense segments can receive more Z-tokens, while redundant regions can be represented more compactly through a budget-aware length regularizer. We evaluate our method on seven long-context datasets (e.g., RULER, HotpotQA, and QuALITY), demonstrating that, compared with several strong baselines, our approach achieves better reconstruction quality and downstream performance, especially in scenarios with multiple queries on the same input, while reducing effective context length and memory usage and improving throughput in the generation stage. This design supports direct decoding from compressed contexts, and we further explore promising paradigms for autoregressive generation in the Z-token space, providing a practical interface for efficient long-context reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long context compress, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_u3awOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LatentLM adapts a pretrained LLM with LoRA into a compressor, decompressor, and latent-space inference model. It autoregressively represents input text using a variable number of discrete “Z-tokens,” with regularizers intended to control length, maintain contextual consistency, and encourage reuse of latent codes. Experiments cover reconstruction, continuation, RULER, summarization, and long-context QA. The paper also analyzes whether the number of allocated Z-tokens correlates with information-density proxies and studies reuse of compressed contexts across multiple queries.

Critical strengths
  1. Learning discrete, variable-length latent codes inside an existing generative model is an interesting interface between context compression and language modeling. Unlike fixed soft-token compressors, the representation can in principle be stored, inspected, and reused.

  2. The paper evaluates several aspects beyond reconstruction, including continuation, long-context retrieval, QA, summarization, latent-space inference, allocation behavior, and multi-query amortization. The appendix explicitly reports compression overhead and break-even points, which is valuable.

Critical weaknesses
  1. The main efficiency presentation is misleading for ordinary inference. The headline latency and TTFT tables exclude the compression pass while describing the results as end-to-end improvements. The appendix shows that compression takes 25.1–74.6 seconds, compared with savings of only 0.27–1.14 seconds per query, requiring approximately 65–92 queries over the same document to break even. Thus, the proposed method substantially increases latency for single-query and moderately reused contexts, and its practical applicability is much narrower than the main presentation suggests.

  2. Reconstruction quality is difficult to interpret because teacher-forced and free-decoding BLEU are not consistently foregrounded. The ablation reports 99.41 teacher-forced BLEU but only 72.39 free-decoding BLEU, raising the possibility that the near-perfect reconstruction numbers in the principal tables mainly reflect teacher forcing. Faithful decompression should be evaluated primarily under autonomous decoding, with clear generation settings and uncertainty estimates.

Questions for authors
  1. Are the reconstruction results in Tables 1 and 3 teacher-forced or freely decoded? Please report both throughout, including exact-match or edit-distance statistics and failure rates under autonomous decompression.

  2. Can the main efficiency tables include compression, decompression, and downstream generation in one end-to-end measurement? What workloads realistically reuse exactly the same compressed representation for 65–92 or more queries?

  3. Does the compressed representation itself require first materializing the full-context KV cache, and what is the peak memory and latency of the compression phase?

  4. How sensitive are the downstream results to the autoregressive compression decisions? In particular, what is the variance across compression samples or decoding seeds?

Reviewer_WsZ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatentLM, a method that encodes context into "Z-token" space, a latent space with discrete and variable-length virtual tokens called "Z tokens". These tokens are used as a compressed representation of the original texts, and can be used to reconstruct the original texts or decode new texts. Unlike previous work that extracts soft or hard representation from existing hidden space, LatentLM uses autoregressive decoding to generate Z tokens. Experiments show that it can be used to speed up LM decoding and can outperform baseline methods on some benchmarks.

Critical strengths
  • The idea is novel and interesting. It is intuitive to use "abstractive language" to represent language, and the resultant Z tokens are shown to preserve most of the semantics of the original texts.
  • The experiment results are good and the method has potential to be used for efficient LM decoding.
  • The qualitative examples are interesing (e.g. figure 2 and section 3.4) and may be used to better understand how LLMs understand human language.
Critical weaknesses
  • The layout of this paper is too busy. The introduction is a single paragraph extending to the end of the second page. Many spaces around figures and tables are squeezed out (not sure if this violates ICLR policy). The authors should consider abridging the languages and moving some of the contents to appendices.
  • Using discrete tokens to compress contexts is an interesting idea but also a waste of representation space, as vector space has a clearly larger semantic space, and Z tokens need to be re-encoded into vector space anyway. It's not convincing that Z token is a better solution than other context compression methods in terms of efficiency and effectiveness.
  • The experiments remain in traditional tasks such as summarization and QAs, while the effectiveness of LLMs are better judged by multi-turn conversations or agentic scenarios.
Questions for authors

I don't have questions that can affect my assessment.

Reviewer_u61uOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work presents a framework for adapting a large language model (LLM) into an autoencoder with a variable-length latent "Z-tokens" space.

The framework supports autoregressive inference in data space using the decompressor conditioned on compressed tokens as well as autoregressive inference in the Z-token space before a final call to the decompressor to bring the output to the data space.

The method is trained with LoRA with regularization for length, codebook collapse, and embedding consistency and is compared to other methods which adapt off-the-shelf LLMs as autoencoders on reconstruction, summarization, question-answering (QA), and long-context.

Additionally, ablations are performed to justify design choices.

Critical strengths

Variable-length latent space is a novel contribution to the LLM autoencoder literature

Critical weaknesses

Autoregressive inference module in Z-space is introduced, but not extensively evaluated or discussed

Compression is only evaluated on low-target ratios

Questions for authors
  • How is the Z-Token codebook initialized? The same as VQ-VAE?
  • Why would one use the Z-Token inference method when it appears to provide no benefit at (presumably) increased cost?
  • One of the key points of 500xCompressor is its performance at high compression ratios. This method trends better than ICAE and 500xCompressor, but can you share details for the average latent space sizes in Table 3 for your method and performance at higher compression ratios?
  • Why include both pearson and spearman correlation for comparing Z-token count to semantic-density proxies? Can you share plots of the data used for these calculations?
  • How were the different models chosen for the different experiments? Why not stick to a single model across all experiments?
Reviewer_4LpjOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses an LLM trained with cross-entropy loss and some additional regularization terms to compress tokens in the base vocabulary into a Z-token space. Models with LoRA adapters are able to compress the sequence into latent space, reason in latent space, and decompress into human-readable text space. The models can be easily combined with sliding window compression for even more compressive gains on long context tasks. They report that their method outperforms fixed-length and learned compressors on a range of tasks, model sizes, and compression ratios. They show that their model improves efficiency compared to a baseline of full context processing.

Critical strengths
  1. The idea of using an LLM to compress language for long-context reasoning is promising and well motivated. The autoregressive generation allows for variable-length compression, which, according to some experiments, actually increased the effective compression rate. The contextual consistency of Z-tokens opens up promising avenues of research for efficient reasoning. This presents a new and useful idea with room for more interesting work.
  2. The paper included experiments on many tasks at multiple model scales with appropriate baselines.
Critical weaknesses
  1. Many details were not explained with sufficient detail
    • Overlap loss: when in training was this used, which experiments used models using this loss term. Z^overlap is undefined
    • Z-token codebook: unclear how these embeddings were trained, the algorithm in the appendix did not update them, only the LoRA parameters
    • Efficiency and serving: “KV caching enabled” doesn’t say whether the baseline can reuse a cached document prefix, which would undercut the repeat-query advantage
    • Contextual consistency: the functional definition is under-defined. Window definition, width, evaluation set, and similarity function should be included in the appendix. How a window is defined in the Z-space should also be clarified
    • The paper shows correlation between semantic density and length allocation but doesn't show that it helps: an ablation with a fixed-length version of LatentLM on the same model could show variable length improves accuracy
    • The temperature and schedule used for the Gumbel-Softmax is missing
  2. Lots of typos. The number of typos distracts from the content; please proofread carefully. Here is a list of just a few of the most obvious
    • "Semamtic” line 270
    • "500xCompresor” (Table 6)
    • Baseline section missing space between name and author in citations (eg. ICAEGe et al.)
    • "we use Qwen3-0.6B [to?] further conduct..." on line 366\
    • "Appendi B" line 402
Questions for authors
  1. Please add the relevant details for the sections mentioned in the weaknesses
  2. Please proofread the paper a few more times, perhaps with the help of an AI tool.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.