acceptodds
Under review as a conference paper at ICLR 2027

Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders

Abstract

Next-token prediction has enabled highly fluent autoregressive language models, but it represents global structure only indirectly through sequential factorization. In contrast, high-fidelity autoencoders have become a standard primitive in image generation, enabling generative models to operate over continuous latent spaces; text lacks a comparably faithful continuous representation. We propose LLMAE, a method for repurposing a pretrained decoder-only language model as a continuous text autoencoder by exposing the activations of an intermediate layer as a fixed-length latent bottleneck. LLMAE achieves this interface with structured attention masks and LoRA adaptation, leveraging the generative prior of the original LLM. Across two backbones (270M Gemma 3 and 0.5B Qwen2.5), LLMAE reconstructs sequences up to 1024 tokens with high accuracy, reproducing up to 97% of documents verbatim. We demonstrate downstream utility by training a lightweight diffusion model that generates detailed image captions directly in the frozen LLMAE latent space. By mapping text into a fixed-length continuous latent space, our approach provides an effective substrate for downstream adaptation while benefiting from the fluency of the original LLM.

Primary area:
generative models
Keywords:
text autoencoder, text representations, non-autoregressive text generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_ADZ3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LLMAE repurposes a decoder-only LLM as a continuous text autoencoder. Learned tokens gather the input into a fixed-length latent at an intermediate layer, and structured attention masks force reconstruction through this bottleneck. LoRA adapts the backbone, while latent regularization and an optional residual codec refine the representation. On Gemma-3-270M and Qwen2.5-0.5B, reconstruction substantially improves over the evaluated ICAE and COSMOS checkpoints. A separate image-conditioned diffusion model generates caption latents, which the frozen LLMAE decoder converts into text autoregressively.

Critical strengths
  1. The idea is interesting, and the architecture is clean. A masked bottleneck within one pretrained stack offers a practical way to combine compression with a pretrained generative prior, without a separate full-sized encoder–decoder pair.

  2. The reconstruction results are strong across three domains. Qwen reaches 97.4% whitespace-normalized exact match and 0.005% word edit distance on the up-to-1024-token C4 setting (Table 1).

Critical weaknesses
  1. The idea is interesting, but the motivation is not sufficiently clear. Why study text VAEs, and what makes design choices such as splitting an LLM reasonable and efficient? I acknowledge the importance of investigating NAR methods. However, some of the authors' arguments are not very convincing. For example, they reason that because image latent diffusion relies on high-quality autoencoders, text also needs a high-fidelity continuous representation. This logical connection seems somewhat tenuous.

More importantly, although LLMAE uses relatively few trainable parameters and achieves strong text reconstruction results, its downstream generation experiments do not yet demonstrate sufficiently convincing advantages. The full generation pipeline involves multiple training and inference components, but the paper does not adequately establish that this additional complexity yields corresponding gains in generation quality or end-to-end efficiency.

  1. The paper omits an important related work, LaDiC [1], which already explores splitting pretrained BERT into encoder and decoder components, performing image-conditioned diffusion over intermediate text latents, and applying this framework to caption generation. LLMAE does have its own distinctions, such as a decoder-only backbone and learned sequence compression. However, the core idea of using LLM hidden states as a latent space for downstream NAR generation is not new.

[1] LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation? NAACL 2024.

Questions for authors

See wkness

2 comments

Thank you for the review and feedback. The revision now states the motivation directly: a fixed length latent for non-autoregressive (NAR) generation (W1a). On efficiency, we added new results: with Jacobi decoding, the LLMAE decoder needs about 3 passes instead of 195 for a caption (about 5 for a 430-token document), and 100 diffusion steps instead of 250 keep the caption quality close (W1b). We also added LaDiC to the related work (W2).

W1a: The motivation is not clear: why text autoencoders, and why is splitting an LLM reasonable and efficient? The motivation is a fixed length latent for NAR generation (C1 in the general response): NAR models work on fixed-shape tensors, and with a latent that does not lose the text, the autoencoder is not a source of errors. We use the image domain only as inspiration and for a proof of concept, and the revision now states this motivation directly (Sec. 1 and 2). Splitting one LLM is efficient: there is no separate encoder or decoder model, only 10.7M (Gemma) and 23.8M (Qwen) parameters are trained in about 22 hours. The split is inspired by the "entropy valley": initial layers specialize in encoding tokens and final layers in decoding tokens, and our depth ablation supports an intermediate layer on both backbones (Table 3).

W1b: The downstream experiments do not show convincing advantages for the added complexity. The only added component is the autoencoder. It is trained once on general text and then kept frozen, and we did not need to specialize it for diffusion: the diffusion model is taken from COSMOS with small changes. On quality, we do not beat autoregressive captioners (C2 in the general response); however, we show that LLMAE is a better latent for diffusion than COSMOS (VLM judge 3.44 vs 2.24). On efficiency, we added new experiments: with Jacobi decoding, the LLMAE decoder needs about 3 passes instead of 195 for a caption of about 190 tokens (new Sec. 4.5), and fewer diffusion steps keep the caption quality close:

Diffusion stepsDecodingDecoder passesRefCLIPPPL
250 (original submission)autoregressive1950.45025.5
250Jacobi3.30.45025.5
100Jacobi3.30.45029.3
50Jacobi3.50.45038.4

W2: LaDiC is missing; using LLM hidden states as a latent for NAR generation is not new. Thank you, we added LaDiC to the related work (Sec. 2). LaDiC splits BERT into an encoder and a decoder and runs diffusion on one latent per caption token, for short captions (at most 24 tokens on COCO, and 60 for paragraph captions). LLMAE is different in three ways: it uses a decoder-only LLM with learned latent tokens that compress up to 1024 tokens into 256 latent vectors (4x shorter than the text), it reconstructs long text near-losslessly (100% exact match on Qwen), and it works for latent diffusion to generate long captions (about 160 words, or 190 tokens, on average).

Thank you for your detailed response, which has addressed most of my concerns. I will therefore raise my rating.

Reviewer_3UFBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper discusses how to turn decoder only transformer models aka the vast majority of LLMs into valid autoencoders. The encoder-decoder transformer has lost popularity over the years and these models are now pre-trained and out there and can be easily used for various other purposes. Overall the paper is easy to follow and the core value proposition is clear, however i had several empirical qualms (see below)

Critical strengths

The fall-off of encoder-decoder transformers is indeed a thing to be fixed. Generally with the proliferation of strong OSS LLMs work that can quickly adapt them to different scenarios is going to enjoy strong tailwinds as opposed to a recipe of training a new encoder-decoder transformer LLM. Hence I like this critical design choice.

Critical weaknesses

My main concerns are as follows :

  • I did not understand why classical, "NLP-like" metrics were used to gauge the soundness of the auto encoding procedure. In fact reconstruction or BLEU etc. are very imperfect metrics as opposed to LLM-like "downstream" benchmarks such as terminal bench. I am aware that the models studied are very weak and get low scores on terminal bench, but the interesting question is that, can we scale up to Gemma 1B-class models, and show that the reconstruction does not break e.g. their agentic coding capabilities, or QA ? Even with the models studied, such metrics carry more meaning. (In fact the training took only 22h, so I think scaling up to 1B would've been not that much harder...)

  • What are the downstream effects of having these representations ? This seems to be not fully explored. For example, arguments can be made for faster drafter-like work / better speculative decoding. The example of a diffusion model is nice but I did not see it as the optimal fit.

Questions for authors

Please see weaknesses.

1 comment

Thank you for the review and feedback. We ran three experiments: a downstream QA check on reconstructed text (W1a); larger backbones, where Qwen2.5-1.5B reaches 99.6% exact match up to 1024 tokens (W1b); and faster decoding, which gave a new result: the LLMAE decoder needs about 5 Jacobi passes instead of about 430 autoregressive ones for a document, and about 3 instead of 195 for a caption (W2).

W1a: Why classical NLP metrics instead of downstream benchmarks? In this work we train LLMAE to serve only as an autoencoder, by leveraging LLMs' language pretraining. We do not try to preserve the reasoning or completion abilities of the backbone. We agree that an interesting future direction would be to encode a more semantic latent that is completed utilizing the LLM's abilities. For LLMAE, the question is whether the text comes back, and exact match is the strictest measure of that. As suggested, we ran a downstream check: a fixed QA reader gets the same score on reconstructed contexts as on the original contexts. This is expected since LLMAE reconstructs the context almost exactly:

SQuAD v1.1 dev (10,570 questions)EMF1Contexts reconstructed exactly
Original contexts87.2593.68-
Qwen LLMAE reconstructions87.2593.6899.8%
Gemma LLMAE reconstructions87.2593.6899.9%

W1b: Can it scale to 1B-class models? Yes. We trained LLMAE on Qwen2.5-1.5B and Gemma-3-1B, with the reconstruction loss only and a larger LoRA rank (new App. M, Table 23):

BackboneBLEUExact match (up to 1024 tokens)
Gemma-3-270M (Table 1)0.99890.0%
Qwen2.5-0.5B (Table 1)1.000100.0%
Qwen2.5-1.5B (new)1.00099.6%
Gemma-3-1B (new)0.99675.2%

Qwen2.5-1.5B reaches 99.6% exact match after two epochs, so the method scales up to a larger backbone. Gemma-3-1B is also close to lossless (BLEU 0.996).

W2: What are the downstream effects of these representations (drafting, speculative decoding)? Thank you for this suggestion. It led us to test Jacobi decoding (Santilli et al., 2023): instead of generating one token per decoder pass, each pass predicts all tokens at once from the previous pass, and decoding stops when a pass changes nothing, which gives exactly the normal (greedy) output. LLMAE needs only about 5 such passes for a document of up to 1024 tokens, where normal autoregressive decoding needs one pass per token. In LLMAE, the decoder layers are fine-tuned only for decoding the latent. ICAE, which decodes its latent with a frozen LLM, still needs about 430 passes, and our autoregressive captioner about 170 for 180-token captions, with the same procedure (new Sec. 4.5 and Table 6; an example is in App. L):

DecoderTokensAutoregressive passesJacobi passes
LLMAE Qwen, documents up to 1024 tokens4244254.9
LLMAE Gemma, documents up to 1024 tokens4324335.2
ICAE (Mistral-7B), documents up to 1024 tokens456457432
LLMAE Gemma, generated captions1941953.3
Autoregressive Qwen2.5-0.5B captioner180181173

For captioning this cuts the cost. A caption needs the diffusion steps plus the decoder passes. Jacobi decoding needs 3.3 to 4.6 passes per caption at every number of diffusion steps:

Diffusion stepsDecodingDecoder passesRefCLIPPPL
250 (paper)autoregressive1950.45025.5
250Jacobi3.30.45025.5
100Jacobi3.30.45029.3
50Jacobi3.50.45038.4
25Jacobi4.60.44663.2

So the decoder is no longer the slow part: it needs about 3 passes instead of 195. With 100 diffusion steps instead of 250, the captions keep the same RefCLIP with a small loss in fluency (PPL 29.3 vs 25.5).

Reviewer_8Y4sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper repurposes pretrained decoder-only LLMs as continuous text autoencoders. An intermediate-layer bottleneck compresses text into a fixed number of latent tokens, and the remaining layers reconstruct the input. The paper reports strong reconstruction accuracy and demonstrates image-conditioned latent diffusion for detailed captioning.

Critical strengths
  • The intermediate-layer construction is simple and effective. The reconstruction results, particularly for the Qwen backbone, are strong.

  • The paper includes useful ablations of bottleneck depth, latent length, and latent regularization.

Critical weaknesses
  • Reconstruction is not sufficient evidence of a good diffusion latent space. Most of the evidence concerns reconstruction from encoded latents. For generation, the latent distribution must also be learnable, its semantic structure useful, and the decoder tolerant of errors in generated latents, which I think lack disccusion in this paper.

  • Decoder robustness is tested under a limited error model. Appendix D evaluates independent Gaussian perturbations. The paper does not show how those errors affect decoded text, or whether reconstruction fidelity predicts downstream generation quality across its autoencoder variants.

  • The downstream evidence is narrow. The only generative application is image-conditioned captioning, and its results remain below stronger autoregressive captioners in Table 5. This makes the broader case for LLMAE as a language-generation interface less convincing. The comparison with COSMOS is useful, but a broader evaluation under matched generator capacity and training budgets would better establish the benefit of this particular latent space.

  • The distinction from recent text latent models needs sharper discussion. CoLa-DLM[1] also reports near-perfect clean-latent reconstruction for its Text VAE, although its architecture and reconstruction metric differ. TextLDM[2] explicitly finds that reconstruction fidelity alone is insufficient for downstream diffusion quality. These works make it especially important to establish what LLMAE contributes beyond high-fidelity reconstruction.

[1] Continuous Latent Diffusion Language Model

[2] TextLDM: Language Modeling with Continuous Latent Diffusion

Questions for authors
  1. Across the autoencoder variants in this paper, how well does reconstruction accuracy correlate with diffusion validation loss and final generation quality?

  2. Can the authors evaluate decoding from diffusion-predicted latents at different noise levels, and analyze the resulting semantic and textual errors?

  3. Can the authors provide stronger evidence that LLMAE latents are easier to model than relevant alternatives, using the same diffusion architecture, compute budget, and evaluation task? How does the method relate experimentally to CoLa-DLM and TextLDM, while accounting for their different latent lengths and training setups?

  4. Does the advantage extend to a text-generation task without image conditioning ?

2 comments

Thank you for the review and feedback. We ran three experiments you asked for: decoding diffusion-predicted latents at different noise levels (Q2), an autoregressive baseline on the same backbone and data (Q3), and generation without image conditioning (Q4).

Q1 / W1: How well does reconstruction accuracy correlate with diffusion validation loss and generation quality across the autoencoder variants?

One diffusion training run takes about 3 days on 8 GPUs, so we did not study the "diffusability" of each of the autoencoder variants in our ablations. Based on our runs, the Gemma and Qwen LLMAE models perform similarly, and both are better than COSMOS (Tables 1 and 5):

AutoencoderExact matchVLM judgeCapArena-Auto
COSMOS6.0% (512 tokens)2.24-88.3
LLMAE Gemma90.0% (1024 tokens)3.44-57.8
LLMAE Qwen (captioner at step 163K of 226K)100.0% (1024 tokens)3.42-61.0

We understand that good reconstruction does not imply that a latent is easy to model with diffusion, and the revision now says so in Limitations (Sec. 5). Reconstruction is sufficient (but not required) evidence that there is a latent to learn that decodes each text faithfully. We show that LLMAE can be modeled with diffusion, and we will release all LLMAE variants studied with different lengths and KL/codec components.

Q2 / W2: Can you decode diffusion-predicted latents at different noise levels and analyze the errors? We did this on 100 test images. We encode the reference caption into its latent z0z_0, add noise up to time tt with the training noise schedule, and run the diffusion model once at time tt, conditioned on the image (guidance 3.0). The model is trained to output the clean latent, so this single pass gives a prediction z^0\hat{z}_0, which we decode and compare with the reference caption (BLEU-4 and word edit distance, as in Table 1):

Noise level ttGemma: BLEU (W-ED)Qwen at step 163K: BLEU (W-ED)
0.10.992 (0.6%)0.991 (0.7%)
0.30.958 (2.1%)0.982 (1.1%)
0.50.788 (10.1%)0.853 (7.4%)
0.70.242 (55.7%)0.289 (51.3%)
0.90.012 (112.4%)0.012 (107.2%)

Noise removes details of the reference caption, which the model fills in with its own words. The errors come in this order: word substitutions first, then dropped spans, and repetition only at t=0.9. The content is fixed early: at the halfway step, the caption differs from the final one in only 1-3% of words. The decoder is tested here on the diffusion model's own errors, not on Gaussian noise (App. D): the text stays close to the reference up to t=0.3, 7-10% of its words differ at t=0.5 (BLEU-4 drops faster, as one changed word breaks up to four 4-word matches), and it breaks down only when the latent is mostly noise.

Q3 / W3: Stronger evidence that LLMAE latents are easier to model than alternatives, with matched capacity and budget. COSMOS is the comparison with the same diffusion architecture (their own), data, budget and judge. For matched capacity, we added an autoregressive baseline on the same backbone and data (Table 5):

Captioner (same data)VLM judgeCapArena-Auto
Diffusion on COSMOS latents2.24-88.3
Diffusion on LLMAE latents (Gemma / Qwen at step 163K of 226K)3.44 / 3.42-57.8 / -61.0
Autoregressive Qwen2.5-0.5B (new)5.26+1.5

LLMAE latents are better than COSMOS latents for diffusion. The autoregressive baseline is better than our diffusion model (C2 in the general response). A direct comparison with the diffusion models of CoLa-DLM and TextLDM is not possible, as they train on language modeling tasks (see W4). The revised related work now cites both (Sec. 2).

Q4: Does the advantage extend to text generation without image conditioning? We ran on ROCStories (unconditional short form story generation), scored with the COSMOS evaluation script:

ROCStoriesMAUVEPPL
GPT-20.78920.5
SEDD0.59870.8
LD4LG0.71630.6
TEncDM0.76229.1
COSMOS0.94026.3
Diffusion on LLMAE latents (ours)0.8047.1

Baseline numbers are from the COSMOS paper. So the latent also works without image conditioning. COSMOS is still better on this short-text task. We believe our high fidelity autoencoder is better suited for long text with strong conditioning, such as detailed captioning, where it gives better captions than COSMOS (Table 5).

W4: Distinction from CoLa-DLM and TextLDM. Both keep one latent per token, so the latent grows with the text (up to 1024 x 64 for TextLDM, 512 x 16 for CoLa-DLM), and train the text VAE from scratch. LLMAE maps up to 1024 tokens into a fixed 256 latent vectors with the pretrained LLM itself.

Reviewer_8Y4sOpenReview ↗

Response of Author's Rebuttal

Thank you for the thoughtful response and additional experiments. The reconstruction and Jacobi decoding results are interesting, but I remain unconvinced that this language autoencoder addresses a broad practical need comparable to the role of VAEs in image generation. I therefore maintain my original score.

Reviewer_Gm4HOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper turns a small AR LLM into a text autoencoder with a fixed-length continuous latent. Learnable tokens are appended to the input, and their activations at an intermediate layer serve as the latent. An attention mask forces the decoder to reconstruct the text from this latent only. With LoRA and some regularization, reconstruction of up to 1024 tokens is near-lossless, well above ICAE and COSMOS. The latent is then used for a latent diffusion captioner.

Critical strengths
  • The method is simple and cheap: 10-24M trainable parameters and about 22 hours of training. Reconstruction is strong, and the depth ablation is convincing: reading out at an intermediate layer works, while early layers or an input-level soft prompt fail.
Critical weaknesses
  • Reconstruction is not really the problem, and I don't see a clear advantage over ICAE. Near-lossless reconstruction at this compression ratio is already possible in the literature. ICAE (Ge et al., 2024) reports 99.3 BLEU on 512 tokens with 128 memory slots, and CALM (Shao et al., 2025) reaches over 99.9% accuracy at 4 tokens per vector, so the claim that text "lacks a comparably faithful continuous representation" is too strong. The design is basically ICAE with different implementation, except that the latent is read from a middle layer and the decoder is also tuned. With a fixed K = 256, it should also hit the same capacity limit on longer inputs: Gemma already drops from 92.6% to 70.6% exact match going from 512 to 1024 tokens.

  • Better reconstruction does not mean a better latent for diffusion. Work on image latent diffusion shows that high-dimensional, reconstruction-focused latents are often harder to generate in (Rombach et al., 2022; Yao et al., 2025; Skorokhodov et al., 2025; Zheng et al., 2025). The only downstream test here is strongly conditioned captioning, with no autoregressive baseline on the same backbone and data. So it is unclear whether going through the latent helps at all.

  1. Ge, T., Hu, J., Wang, L., Wang, X., Chen, S.-Q., and Wei, F. In-context Autoencoder for Context Compression in a Large Language Model. ICLR 2024.

  2. Shao, C., Li, D., Meng, F., and Zhou, J. Continuous Autoregressive Language Models. arXiv:2510.27688, 2025.

  3. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.

  4. Yao, J., Yang, B., and Wang, X. Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models. CVPR 2025.

  5. Skorokhodov, I., et al. Improving the Diffusability of Autoencoders. ICML 2025.

  6. Zheng, B., Ma, N., Tong, S., and Xie, S. Diffusion Transformers with Representation Autoencoders. ICLR 2026.

Questions for authors
  1. Why are the ICAE numbers in Table 1 so much lower than those in the ICAE paper?
  2. What does the design gain over ICAE beyond tuning the decoder?
  3. How does reconstruction hold up beyond 1024 tokens?
  4. Can you add an autoregressive captioning baseline with the same backbone and data?
1 comment

Thank you for the review and feedback. We address your two main concerns, then your four questions.

W1: Reconstruction is not really the problem; no clear advantage over ICAE. ICAE is the closest prior work. LLMAE's advantage is exact reconstruction of long text with a smaller model. On our fairly constructed, held-out test sets, the public ICAE checkpoint (7B) reproduces 24.2% of documents exactly up to 512 tokens and 13.4% up to 1024; LLMAE reproduces 99.6-100% and 90-100% with 0.27B / 0.5B models (Table 1). The ICAE paper reports that at 500 tokens the median document matches exactly for about 60% of its length (its Fig. 4). A second advantage is parallel decoding: about 5 passes for all tokens, against about 430 for ICAE (Table 6). CALM encodes short chunks independently and generates them autoregressively; a fixed shape suits NAR models better. We agree that claiming text "lacks a comparably faithful continuous representation" was too strong; the revision now says "fixed-length". On the capacity limit: after the bug fix, Gemma goes from 99.6% (512 tokens) to 90.0% (1024 tokens) exact match, not from 92.6% to 70.6%, and Qwen is 100% at both (Table 1). More latent tokens add capacity (Table 4).

W2: Better reconstruction does not mean a better latent for diffusion. Perfect reconstruction is sufficient evidence that a latent exists for each text, but we agree it does not tell how easy the latent is to model with diffusion. We do not fully study "diffusability", and say so explicitly in the revision (Sec. 5). We show that the latent can be modeled with diffusion, and better than COSMOS latents, which are trained for diffusion, using the COSMOS diffusion architecture for both, with the same data and budget (Table 5). You ask whether the latent helps at all. For caption quality, we do not beat an autoregressive baseline (Q4). What the latent gives is a decoder that need not be sequential: all tokens in about 5 Jacobi passes (new Sec. 4.5), while ICAE needs about 430 and the autoregressive captioner about 170.

Q1: Why are the ICAE numbers in Table 1 lower than in the ICAE paper? The ICAE paper reports BLEU 99.1-99.8 at 512 tokens (its Tables 3 and 6), on a test set that is not public. The authors released their final fine-tuned Llama model, but not the Llama autoencoding checkpoint behind these numbers (requested on the ICAE GitHub since April 2024, issue #11), so they cannot be reproduced or compared on the same documents. We used the public pretrained ICAE "v2" autoencoding checkpoint (Mistral-7B) with the authors' inference script, on our public test sets. We admit this was not clearly mentioned in the submission; the revision now states it (Sec. 4.1 & App. I). By document length (new Table 17, App. I):

BLEU by length (words)0-100100-200200-300300-400All
C40.9670.8970.7200.5420.832
Pile (ICAE pretraining)0.8700.8200.6650.5230.760

Q2: What does the design gain over ICAE beyond tuning the decoder? LLMAE is closely related to ICAE, with two major design changes:

  • The latent is established at an intermediate layer; ICAE reads it at the last layer and feeds it back as input.
  • A single LLM is split by depth: the lower layers encode and the upper layers decode; ICAE runs the full LLM twice, with LoRA to encode and frozen to decode.

This gives two gains:

  1. Near-perfect reconstruction with much smaller models: with the ICAE design, the same Gemma-270M backbone and data reach BLEU 0.393 at best (App. I), against 0.987 with ours.
  2. Parallel decoding in about 5 passes for up to 1024 tokens, against about 430 for ICAE (W1).

The depth ablation (Table 3) shows that using an intermediate layer is better than layers near the top or bottom. This is more strongly apparent for Gemma than Qwen, but the pattern holds for both.

Q3: How does reconstruction hold up beyond 1024 tokens? We used our model trained on up to 1024 tokens and it did not immediately work well on longer sequences (BLEU 0.47 with Qwen on held-out C4 documents of 1024 to 2048 tokens). We chose 1024 tokens for training because it is already long enough for many text generation tasks (our detailed captions are about 190 tokens), and is in fact twice the 512 tokens of COSMOS and of the ICAE paper's results. For tasks such as reasoning it might fall short; training on longer text is future work.

Q4: Autoregressive captioning baseline with the same backbone and data? We ran this experiment: Qwen2.5-0.5B finetuned on the same data, attending to the same SigLIP features (new row in Table 5).

CaptionerVLM judgeCapArena-Auto
Autoregressive Qwen2.5-0.5B5.26+1.5
Diffusion, LLMAE Qwen (step 163K of 226K)3.42-61.0
Diffusion, LLMAE Gemma3.44-57.8
Diffusion, COSMOS2.24-88.3

The autoregressive model is better; we claim only that LLMAE is an effective latent for diffusion (W2).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their time and valuable feedback. We ran new experiments that incorporate the feedback and strengthen the paper, and uploaded a revision PDF with the diff highlighted in blue font (we will change it back to black for the final version). We also note that this is the largest improvement we have made to a submission during the rebuttal period as addressing the reviewer questions led to identifying a bug and improving performance, and to a new result on parallel decoding. Thank you!

Bug fix: During our rebuttal experiments, we found a discrepancy in our inference batch encode function: training and inference build their sequences separately, and the inference call was missing the padding mask. Input padding tokens were masked out at training time but not at test time, which confused the encoder. Note, there was never any improper information leak. This is an evaluation-only bug: it changes the reported numbers, but not the method, the trained models or the baselines. We corrected it and updated all affected evaluations in the revision. The two captioning models were trained on latents from the old encode function, so we trained them again on corrected latents: the Gemma model has finished, and the Qwen model is at step 163K of 226K (in red in Table 5; we will update it tomorrow when training completes). The main implications are:

  1. Reconstruction metrics. After the correction, LLMAE reconstruction (Table 1) improved further, reaching 100% exact match on the Qwen KL+codec variant (C4, up to 1024 tokens); it was 97% before.
  2. Depth ablation. On Qwen, the bottleneck works at many depths (Table 3: 80–99% exact match at layers 3 to 19); new runs at the first and last layers are worse (47–73%), which supports an intermediate depth. Gemma keeps a stronger "entropy valley" pattern across mid-to-late layers.
  3. Regularization. On Gemma, adding KL and then the codec now improves exact match at each step (68.2% → 83.2% → 90.0%); the submission reported that KL reduces it.

Below, we address shared concerns about method, motivation and utility:

C1: Autoencoding with a fixed-length latent for variable length text
The key gap LLMAE fills is a fixed length latent space: the number of latent vectors (K) is fixed and independent of the input length (N). This is more suitable for non-autoregressive (NAR) generation with established modeling paradigms such as continuous diffusion. Several autoencoders mentioned in reviews (CALM, CoLa-DLM, TextLDM) compress tokens into a variable length latent. We should have been clearer about this, and have changed the revision. We use ICAE and COSMOS as two high quality baselines with a fixed length latent. LLMAE maps variable length text (up to 1024 tokens, ~ 760 words) into a single fixed-shape representation. We are not aware of prior work that performs near-lossless reconstruction at this length into a fixed-size latent. ICAE is the closest (see our response to Reviewer Gm4H): it showed that an LLM can reconstruct text from a fixed length latent, and LLMAE builds on this direction. ICAE encodes with a 7B LLM plus LoRA and decodes with the same LLM, frozen: 7.46B parameters, 0.22B of them trained. LLMAE splits a much smaller LLM by depth, the lower layers encoding and the upper layers decoding: 279M parameters with 10.7M trained (Gemma), or 494M with 23.8M trained (Qwen). On C4 up to 1024 tokens, LLMAE reproduces 90.0% (Gemma) and 100% (Qwen) of documents exactly, against 13.4% for ICAE (Table 1).

Parallel decoding (new Sec. 4.5). We find that LLMAE decodes in very few passes: on our stratified test set of up to 1024 tokens, it needs only about 5 Jacobi decoding passes, instead of about 430 autoregressive passes, with the same output as normal decoding. Reviewer 3UFB asked about drafting, which led us to test this. In LLMAE, the decoder layers are fine-tuned only for decoding the latent. ICAE, which decodes its latent with a frozen LLM, needs about 430 passes with the same procedure (Table 6). This further supports the modeling motivation and the utility of this work for non-autoregressive generation: with recent advances in one-step or few-step latent generation, decoding with LLMAE can simplify text generation even further.

C2: Downstream utility
Our core contribution is the autoencoder itself, which we believe is sufficiently motivated in the field: NAR models need a fixed length latent that does not lose the text. For downstream modeling we studied detailed image captioning (Table 5), chosen because traditional NAR methods have struggled with long text generation. While we do not beat autoregressive baselines (we added one with the same backbone and data), we show that the latent can be modeled with diffusion, and better than COSMOS latents, which are trained for diffusion. We use the COSMOS diffusion architecture (adapted to image conditioning) for both latents, under the same data and budget.

AuthorsOpenReview ↗

Update: final Qwen captioning model

The Qwen captioning model has finished retraining on corrected latents (226K steps, like the Gemma model). Its final scores are VLM Judge 3.63 and CapArena-Auto -50.7, which replace the step-163K numbers (3.42 and -61.0) in Table 5.

With either backbone, LLMAE latents are well ahead of COSMOS latents with the same diffusion architecture, data and budget:

Diffusion onVLM JudgeCapArena-Auto
LLMAE Qwen3.63-50.7
LLMAE Gemma3.44-57.8
COSMOS2.24-88.3

We have uploaded a new revision. The only changes since yesterday's revision are the Qwen row of Table 5 and the sentence about it in Sec. 4.4, which were in red yesterday.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.