- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies lossy text compression using Transformer autoencoders. The proposed architecture combines progressive sequence downsampling and upsampling, a residual bottleneck projection, and discrete quantization. The authors compare architectural choices, quantization methods, training objectives, and data variants, finding that CRVQ and encoder masking work well in their experiments. Evaluation covers text reconstruction, LLM-based semantic judgments, downstream question answering, and semantic textual similarity. The 4.3B model achieves high lexical reconstruction quality at an estimated 2.24 bits/byte. A separate experiment with a modified autoencoder finds that strong reconstruction does not necessarily translate into useful representations for semantic similarity.
Critical strengths
-
The ablations are useful and cover a substantial range of design choices, including pooling, residual projections, quantization configurations, masking, and supervision. The results help identify which choices improve reconstruction under a limited bitrate budget. Reporting unsuccessful training and data variants also adds value.
-
The evaluation considers both reconstructed text and latent representations. The finding that reconstruction quality and semantic similarity performance can diverge is informative, and the authors discuss the limitations of their representations openly.
Critical weaknesses
-
The semantic evaluation needs stronger validation on actual model reconstructions. The synthetic transformations and summary experiments provide useful checks, including the Gemma evaluator in Appendix B. However, these checks do not establish how reliably the judges detect factual changes, omissions, or harmless paraphrases in autoencoder outputs. There is no systematic comparison with human judgments, particularly for the stronger models. The QA evaluation also uses the same Qwen model for both answer extraction and answer comparison. These limitations make it difficult to judge how much semantic information the reconstructions preserve.
-
The practical benefit of allowing reconstruction errors remains unclear. In Table 5, the 4.3B model achieves an estimated 2.24 bits/byte, essentially the same rate as Zstandard level 22 at 2.22, while introducing errors. The neural lossless baselines achieve substantially lower estimated rates of 0.85–0.96 bits/byte. Thus, the reported results do not show a favorable rate–distortion trade-off relative to these baselines. An advantage in runtime, memory use, or downstream computation could still justify the approach, but none is demonstrated. The neural rates, including those of the lossless baselines, are also estimates rather than measured bitstream sizes. The paper would be stronger if it identified a concrete application and set of constraints under which this method offers an advantage.
Questions for authors
-
Could you run the complete entropy coding pipeline on held-out documents and report actual file sizes, including metadata and coding overheads? Please report shared model and codebook storage separately, with the assumed amortization over documents.
-
How well do the judge scores agree with blinded human judgments on reconstructions from the final 2B and 4.3B models? Could the Gemma evaluation be extended to these reconstructions to check whether the conclusions hold across evaluator families?
-
Could you compare model sizes and quantization configurations at matched rates after entropy modeling? The current scaling experiments fix the nominal rate at 11 bits/token, but the estimated rates increase with model size, making it harder to separate the effects of model capacity and bitrate.