- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LLMAE repurposes a decoder-only LLM as a continuous text autoencoder. Learned tokens gather the input into a fixed-length latent at an intermediate layer, and structured attention masks force reconstruction through this bottleneck. LoRA adapts the backbone, while latent regularization and an optional residual codec refine the representation. On Gemma-3-270M and Qwen2.5-0.5B, reconstruction substantially improves over the evaluated ICAE and COSMOS checkpoints. A separate image-conditioned diffusion model generates caption latents, which the frozen LLMAE decoder converts into text autoregressively.
Critical strengths
-
The idea is interesting, and the architecture is clean. A masked bottleneck within one pretrained stack offers a practical way to combine compression with a pretrained generative prior, without a separate full-sized encoder–decoder pair.
-
The reconstruction results are strong across three domains. Qwen reaches 97.4% whitespace-normalized exact match and 0.005% word edit distance on the up-to-1024-token C4 setting (Table 1).
Critical weaknesses
- The idea is interesting, but the motivation is not sufficiently clear. Why study text VAEs, and what makes design choices such as splitting an LLM reasonable and efficient? I acknowledge the importance of investigating NAR methods. However, some of the authors' arguments are not very convincing. For example, they reason that because image latent diffusion relies on high-quality autoencoders, text also needs a high-fidelity continuous representation. This logical connection seems somewhat tenuous.
More importantly, although LLMAE uses relatively few trainable parameters and achieves strong text reconstruction results, its downstream generation experiments do not yet demonstrate sufficiently convincing advantages. The full generation pipeline involves multiple training and inference components, but the paper does not adequately establish that this additional complexity yields corresponding gains in generation quality or end-to-end efficiency.
- The paper omits an important related work, LaDiC [1], which already explores splitting pretrained BERT into encoder and decoder components, performing image-conditioned diffusion over intermediate text latents, and applying this framework to caption generation. LLMAE does have its own distinctions, such as a decoder-only backbone and learned sequence compression. However, the core idea of using LLM hidden states as a latent space for downstream NAR generation is not new.
[1] LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation? NAACL 2024.
Questions for authors
See wkness
2 comments
Thank you for your detailed response, which has addressed most of my concerns. I will therefore raise my rating.
Thank you for the review and feedback. The revision now states the motivation directly: a fixed length latent for non-autoregressive (NAR) generation (W1a). On efficiency, we added new results: with Jacobi decoding, the LLMAE decoder needs about 3 passes instead of 195 for a caption (about 5 for a 430-token document), and 100 diffusion steps instead of 250 keep the caption quality close (W1b). We also added LaDiC to the related work (W2).
W1a: The motivation is not clear: why text autoencoders, and why is splitting an LLM reasonable and efficient? The motivation is a fixed length latent for NAR generation (C1 in the general response): NAR models work on fixed-shape tensors, and with a latent that does not lose the text, the autoencoder is not a source of errors. We use the image domain only as inspiration and for a proof of concept, and the revision now states this motivation directly (Sec. 1 and 2). Splitting one LLM is efficient: there is no separate encoder or decoder model, only 10.7M (Gemma) and 23.8M (Qwen) parameters are trained in about 22 hours. The split is inspired by the "entropy valley": initial layers specialize in encoding tokens and final layers in decoding tokens, and our depth ablation supports an intermediate layer on both backbones (Table 3).
W1b: The downstream experiments do not show convincing advantages for the added complexity. The only added component is the autoencoder. It is trained once on general text and then kept frozen, and we did not need to specialize it for diffusion: the diffusion model is taken from COSMOS with small changes. On quality, we do not beat autoregressive captioners (C2 in the general response); however, we show that LLMAE is a better latent for diffusion than COSMOS (VLM judge 3.44 vs 2.24). On efficiency, we added new experiments: with Jacobi decoding, the LLMAE decoder needs about 3 passes instead of 195 for a caption of about 190 tokens (new Sec. 4.5), and fewer diffusion steps keep the caption quality close:
W2: LaDiC is missing; using LLM hidden states as a latent for NAR generation is not new. Thank you, we added LaDiC to the related work (Sec. 2). LaDiC splits BERT into an encoder and a decoder and runs diffusion on one latent per caption token, for short captions (at most 24 tokens on COCO, and 60 for paragraph captions). LLMAE is different in three ways: it uses a decoder-only LLM with learned latent tokens that compress up to 1024 tokens into 256 latent vectors (4x shorter than the text), it reconstructs long text near-losslessly (100% exact match on Qwen), and it works for latent diffusion to generate long captions (about 160 words, or 190 tokens, on average).