acceptodds
Withdrawn from ICLR 2027

RoAR-VAE: Improving Latent Diffusability through Robust Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An effective visual tokenizer must preserve image fidelity while producing latents that downstream diffusion models can learn efficiently. Prior work improves latent diffusability through semantic alignment or joint tokenizer–diffusion training. Revisiting REPA-E, a representative approach, we find that removing its diffusion objective and dedicated generative blocks while retaining semantic prediction from corrupted latents yields comparable downstream generation performance under a matched setting. This observation motivates a general principle of *robust prediction*: generation-relevant information should remain recoverable from corrupted latents. We instantiate this principle with two complementary objectives. *RobustAlign* predicts pretrained semantic features from corrupted latents to preserve high-level structure, while *RobustRecon* reconstructs the clean image to retain fine-grained spatial and appearance information. Together, they form **Ro**bust **A**lignment and **R**econstruction VAE (**RoAR-VAE**). We attribute the resulting improvement in diffusability to robust prediction making useful information more accessible in high-variance latent directions, which diffusion models tend to learn earlier. Empirically, RoAR-VAE improves average generation benchmark performance by 2.19 points over the strongest baseline tokenizer under the same training budget, while improving small-text reconstruction accuracy by 16.93 percentage points over FLUX.1-VAE. These results demonstrate that RoAR-VAE achieves a favorable balance between reconstruction fidelity and downstream generative learnability.

Primary area:
generative models
Keywords:
Visual Tokenizer, Diffusability, Latent Diffusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_u9PoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RoAR-VAE, a modification to the REPA-E training pipeline that identifies that the joint training objective of REPA-E is generally unnecessary, and that the benefits of REPA-E style VAE training almost exclusively originate from the separate semantic loss as well as input noising. Following this, the authors detach the end-to-end VAE training from a VAE semantic loss. The authors additionally add an additional reconstruction loss. The authors then evaluate across several experiments, first claiming improvement over the REPA-E baseline, as well as strong reconstruction metrics such as PSNR and strong generation metrics such as GenEVAL scores.

Critical strengths

Identifying that the REPA-E VAE does not benefit from the downstream diffusion model training and demonstrating that splitting the component improves performance is a useful takeaway.

The performance on established benchmarks is good, provided that baselines are sufficient. The reconstruction quality, both quantitative and qualitative, is good.

The method is overall simple and well motivated.

Critical weaknesses

I am concerned about the novelty of the method. Prior work such as DeTok [1] perform seemingly every approach in this paper. DeTok contains additive noise to the latent embeddings, a separate (optional) DINO semantic alignment loss, and a MAE style reconstruction decoder. The only difference I see is that the MAE reconstruction in DeTok is weight-tied with the VAE, whereas the RobustReconstruction model here is separate. However, given the extremely minor gains from the RobustReconstruction model in Table 4, I do not believe this difference is sufficient for novelty.

Second, I am concerned about the baselines in this paper. The work is described as an improvement over REPA-E, however in Table 1 the results are marginal at best, losing or tying to REPA-E in half of the reported numbers. Following this, the reported baselines do not include ImageNet for generation. Although it may be argued that ImageNet is outdated or too small a benchmark, this is harmful to the paper because this is a primary setting the baseline is evaluated in. This is compounded by the how the GenEval scores appear inconsistent with those reported for E2E-FLUX-VAE (REPA-E-FLUX) in [2]. What does 35k optimization steps refer to, is this different from the 100k and 500k cited in the provided post? For marginal novelty work (earlier point), I would want to see more rigorous, replicable, and understandable baselines.

[1] Latent Denoising Makes Good Tokenizers, ICLR 2026

[2] https://end2end-diffusion.github.io/repa-e-t2i/

Questions for authors

How do the methods provided in this paper differ from an approach like DeTok other than weight tying. Are these differences sufficiently novel to teach the reader something new about tokenization that they would not get from prior semantic alignment works?

Could the authors provide a more consistent baseline showing improvements over REPA-E?

Reviewer_5meZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors revisit REPA-E and propose RoAR-VAE, showing that predicting semantic features from noise-corrupted latents can yield comparably diffusion-friendly VAE representations without jointly training a diffusion model. They further introduce a robust reconstruction objective to preserve fine-grained visual details. Experiments demonstrate improvements in both downstream generation and image reconstruction.

Critical strengths

Strengths

  1. The paper addresses an important distinction between VAE reconstruction fidelity and the ease with which downstream diffusion models learn the resulting latent space. This provides a clear motivation for improving latent diffusability without sacrificing visual detail.
  2. The controlled comparison with REPA-E usefully isolates the role of semantic prediction from corrupted latents, showing that comparable downstream performance can be achieved without jointly training a full diffusion model during tokenizer refinement.
  3. RobustAlign and RobustRecon target complementary aspects of the representation: high-level semantic structure and fine-grained visual details. The ablations show that both objectives contribute to downstream performance.
  4. The comparison between corrupted- and clean-latent prediction is particularly informative. It helps establish that robustness to latent corruption—not merely adding auxiliary prediction losses—is important to the method.
  5. The evaluation considers both downstream generation and reconstruction, including challenging small-text and face details. This makes the claimed reconstruction–generation trade-off more convincing.
  6. The auxiliary predictors are discarded after tokenizer training, so the proposed objectives do not add components to the final tokenizer’s inference path.
Critical weaknesses

Weaknesses and questions

  1. The downstream comparisons may be confounded by timestep sampling. RoAR-VAE uses shifted logit-normal sampling, whereas each baseline uses its official or recommended scheduler. Evaluating all tokenizers under a common scheduler, or conducting a cross-scheduler comparison, would better isolate the contribution of the tokenizer itself.
  2. The proposed mechanism is not directly validated. The authors argue that robust prediction makes useful information more accessible in high-variance latent directions. While the PCA visualizations are suggestive, they neither quantify this redistribution of information nor establish that it causes faster downstream learning. Quantitative spectral analysis or noisy-latent probing would strengthen this explanation.
  3. The contribution of RobustRecon to preserving fine-grained details needs a more targeted ablation. Table 4 shows that adding it to RobustAlign increases the average generation score from 62.49 to 63.86, but does not report component-wise small-text or face-reconstruction results. Such measurements would better substantiate the stated role of RobustRecon.
  4. The tokenizer is refined with corruption levels sampled uniformly from t∈[0,1]t\in[0,1]. At high tt, the latent approaches pure noise, making the clean semantic and image targets difficult to infer. An analysis of performance across noise levels, or an ablation of the timestep sampling distribution, would clarify whether these high-noise samples benefit or hinder VAE training.
Questions for authors

See the weakness

Reviewer_ER12OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper revisits REPA-E and argues that its tokenizer improvement comes from predicting semantic features from noise-corrupted latents rather than from joint diffusion training. Building on this, it proposes RoAR-VAE, which fine-tunes a VAE with two auxiliary objectives on corrupted latents: predicting DINOv2 features, and reconstructing the clean image with a separate decoder. The authors attribute the improved diffusability to useful information being concentrated in high-variance latent directions that diffusion models learn first. On text-to-image generation with a shared downstream model, RoAR-VAE outperforms several recent tokenizers while achieving strong reconstruction, including on small text and faces.

Critical strengths
  1. Strong reconstruction together with good downstream learnability. RoAR-VAE reaches 32.88 dB PSNR on ImageNet and clearly improves out-of-domain fine-grained reconstruction on TokBench (small-text accuracy 69.00 vs. 52.07 for FLUX.1-VAE, small-face similarity 0.65 vs. 0.53), while also achieving the best downstream text-to-image average (63.86). Most diffusability-oriented tokenizers sacrifice fine detail (e.g., VA-VAE reaches only 7.05 small-text accuracy), so a recipe that improves both is practically valuable.

  2. A clean downstream evaluation protocol with useful ablations. All tokenizers are evaluated with the same downstream MMDiT, data, and sequence length, with four sampling seeds and learning curves over training. The ablations in Tab. 4 cleanly show that both auxiliary objectives help on top of the same-architecture baseline (50.50 → 59.38 and 62.49) and that they are complementary (63.86).

Critical weaknesses
  1. Limited novelty. RobustAlign is essentially Send-VAE [R1], which already predicts DINOv2 features from noisy latents through a Transformer head and matches the E2E-VAE of REPA-E [R2] without diffusion training; it is not cited. RobustRecon follows l-DeTok [R3], which is cited but not compared. The only difference, a separate auxiliary decoder, adds 1.7 points (Tab. 5: 62.13 vs. 63.86).

  2. Key evidence and mechanism are weak. The REPA-E comparison (Tab. 1) gives no tokenizer training details and shows mixed results. The no-corruption ablation (Tab. 4, row 5) is confounded, since without noise RobustRecon just duplicates the main decoder. The high-variance-mode explanation [R4] is never measured beyond PCA visualizations.

[R1] J. Page, X. Niu, K. Wu, and K. Gai. Boosting Latent Diffusion Models via Semantic-Disentangled VAE. arXiv:2601.05823, 2026.

[R2] X. Leng, J. Singh, Y. Hou, Z. Xing, S. Xie, and L. Zheng. REPA-E: Unlocking VAE for End-to-End Tuning of Latent Diffusion Transformers. In ICCV, 2025.

[R3] J. Yang, T. Li, L. Fan, Y. Tian, and Y. Wang. Latent Denoising Makes Good Tokenizers. In ICLR, 2026.

[R4] B. Wang and C. Pehlevan. An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models. In NeurIPS, 2025.

Questions for authors

1 How does RobustAlign differ from Send-VAE, and how does RobustRecon differ from l-DeTok beyond using a separate decoder? If possible, please compare RobustRecon against l-DeTok-style noisy training of the main decoder, starting from the same warm-up checkpoint. 2. In Tab. 1, was the REPA-E tokenizer trained with the same architecture, data, and number of steps as RobustAlign-VAE? 3. Without corruption, what does RobustRecon reduce to, and how should the drop in row 5 of Tab. 4 be attributed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.