acceptodds
Under review as a conference paper at ICLR 2027

Lossy Compressive Text Autoencoders

Abstract

Our work explores learning a compressed latent representation of text, at the intersection of data compression and representation learning. We propose an autoencoder architecture that performs residual downscaling and upscaling of hidden representations along the time axis, with a residual low-dimension discrete bottleneck. We analyze our approach for different quantization methods, training objectives, and datasets. For different levels of compression, we evaluate the similarity between the original and reconstructed text both at the surface-level (BLEU) and at the semantic-level (LLM-based judge). Additionally, we evaluate our models on downstream question-answering and semantic text similarity benchmarks. Our approach results in compressed representations which are on par with lossless text compression algorithms at 2.24 bits per byte on web text data, while having good reconstruction and downstream task performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
text autoencoder, text compression, lossy compression, embedding model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EGv6OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies lossy text compression using Transformer autoencoders. The proposed architecture combines progressive sequence downsampling and upsampling, a residual bottleneck projection, and discrete quantization. The authors compare architectural choices, quantization methods, training objectives, and data variants, finding that CRVQ and encoder masking work well in their experiments. Evaluation covers text reconstruction, LLM-based semantic judgments, downstream question answering, and semantic textual similarity. The 4.3B model achieves high lexical reconstruction quality at an estimated 2.24 bits/byte. A separate experiment with a modified autoencoder finds that strong reconstruction does not necessarily translate into useful representations for semantic similarity.

Critical strengths
  1. The ablations are useful and cover a substantial range of design choices, including pooling, residual projections, quantization configurations, masking, and supervision. The results help identify which choices improve reconstruction under a limited bitrate budget. Reporting unsuccessful training and data variants also adds value.

  2. The evaluation considers both reconstructed text and latent representations. The finding that reconstruction quality and semantic similarity performance can diverge is informative, and the authors discuss the limitations of their representations openly.

Critical weaknesses
  1. The semantic evaluation needs stronger validation on actual model reconstructions. The synthetic transformations and summary experiments provide useful checks, including the Gemma evaluator in Appendix B. However, these checks do not establish how reliably the judges detect factual changes, omissions, or harmless paraphrases in autoencoder outputs. There is no systematic comparison with human judgments, particularly for the stronger models. The QA evaluation also uses the same Qwen model for both answer extraction and answer comparison. These limitations make it difficult to judge how much semantic information the reconstructions preserve.

  2. The practical benefit of allowing reconstruction errors remains unclear. In Table 5, the 4.3B model achieves an estimated 2.24 bits/byte, essentially the same rate as Zstandard level 22 at 2.22, while introducing errors. The neural lossless baselines achieve substantially lower estimated rates of 0.85–0.96 bits/byte. Thus, the reported results do not show a favorable rate–distortion trade-off relative to these baselines. An advantage in runtime, memory use, or downstream computation could still justify the approach, but none is demonstrated. The neural rates, including those of the lossless baselines, are also estimates rather than measured bitstream sizes. The paper would be stronger if it identified a concrete application and set of constraints under which this method offers an advantage.

Questions for authors
  1. Could you run the complete entropy coding pipeline on held-out documents and report actual file sizes, including metadata and coding overheads? Please report shared model and codebook storage separately, with the assumed amortization over documents.

  2. How well do the judge scores agree with blinded human judgments on reconstructions from the final 2B and 4.3B models? Could the Gemma evaluation be extended to these reconstructions to check whether the conclusions hold across evaluator families?

  3. Could you compare model sizes and quantization configurations at matched rates after entropy modeling? The current scaling experiments fix the nominal rate at 11 bits/token, but the estimated rates increase with model size, making it harder to separate the effects of model capacity and bitrate.

Reviewer_N2hiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a multi-resolution Transformer autoencoder for lossy text compression, combining residual sequence downscaling/upscaling, a residual bottleneck projection, and chunked residual vector quantization (CRVQ). A 4.3B-parameter model achieves a reported compression rate of 2.24 bits per byte, comparable to strong conventional lossless compression, while preserving much of the input text's content. The paper also studies reconstruction–representation trade-offs, downstream question answering, and an LLM-based semantic similarity judge.

Critical strengths
  • Jointly optimizing text compression and semantic preservation is a meaningful direction for efficient language-model context handling. Multi-resolution encoding, residual projection, and CRVQ provide a coherent design, with experiments comparing alternative quantization and supervision strategies.

  • The paper evaluates several model sizes, compression rates, reconstruction metrics, and downstream tasks. The corruption experiments demonstrate that surface-level metrics such as BLEU and ROUGE can remain high despite substantial semantic damage.

Critical weaknesses
  • The architectural components build on established ideas from autoencoding, multiscale networks, residual quantization, and context compression. The paper should more clearly isolate what is fundamentally new relative to the closest prior methods.

  • The proposed bitrate is estimated using a separately trained autoregressive entropy model over latent code indices, whereas conventional compressors operate directly on text. The practical accounting should clearly include all relevant coding overheads and explain the comparability of the reported rates.

  • The judge is tested for monotonicity across synthetic summaries and corruption examples, but this does not establish agreement with human judgments on realistic reconstruction errors. Independent human evaluation would strengthen the central semantic-preservation claim.

  • The question-answering experiment uses an LLM both to extract answers and to compare them with references. Results are close to the unmodified-passage upper bound, but broader tasks and independent metrics are needed to establish general utility.

  • The reported semantic-text-similarity results reveal a substantial gap relative to dedicated embedding models, including after similarity-matrix distillation. This limits the broader claim of learning compressed, general-purpose representations.

  • The paper does not sufficiently establish end-to-end benefits in runtime, memory, or energy, especially given the large autoencoder and additional entropy model. The fixed-depth results also suggest that flexible rate control through stochastic depth degrades reconstruction.

Questions for authors
  1. How does the proposed architecture compare with the closest prior text autoencoders under matched model size, training data, sequence length, and effective bitrate?

  2. Does the reported 2.24 bits per byte include all code-index, framing, and entropy-coding overheads? How sensitive is the rate to the entropy model and evaluation corpus?

  3. Have human annotators evaluated semantic equivalence and the preservation of critical details such as numbers, names, negation, and factual relations?

  4. Can the downstream QA results be replicated with task-specific metrics or human-verified answers, across multiple datasets and compression levels?

  5. What are the end-to-end encoding/decoding latency, memory requirements, and computational costs relative to conventional compression and competing neural approaches?

  6. Can the method achieve useful rate adaptation without the performance degradation observed with stochastic-depth training?

Reviewer_FRCxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Proposes a transformer text autoencoder with residual time-axis down- and upscaling, a residual low-dimensional bottleneck, chunked RVQ, and an autoregressive entropy model over codes. Compares quantizers, losses, masking and datasets. Introduces an MQM-style LLM judge. Evaluates on MS MARCO QA and STS.

Critical strengths

Clear question: compression and representation learning jointly. Systematic ablations of quantizers, losses, masking and synthetic targets. Residual bottleneck finding is simple and useful. Honest about the reconstruction–representation trade-off and copying behavior. Judge sanity checks (monotonicity, corrupt set) a reasonable start.

Critical weaknesses

Limited novelty: combination of known components (hourglass/funnel transformers, RVQ/FSQ, entropy coding). Closest architectural prior work (Funnel-Transformer, Hourglass Transformers) not cited. Reconstruction errors are entity and fact substitutions: worst case for a compressor. Copying behavior noted but not analyzed. The most interesting finding is left unexplored. Comparison to context-compression and text-AE work (ICAE, gist tokens, 500xCompressor) missing. Judge contribution incremental. Not validated against humans. Same model family generates data and judges.

Questions for authors

What is the key new idea beyond combining existing components? What does this enable that lossless compression doesn’t? Why does the entropy model barely reduce rate, and why does rate rise with model size? How does the judge agree with human ratings? How does it compare to existing context-compression methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.