- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses the poor reconstruction fidelity of representation autoencoders that use frozen vision foundation model features as latents. It proposes LV-RAE, in which a shallow residual encoder adds low-level information to frozen DINOv3 features under a semantic alignment loss. It also fine-tunes the decoder with latent noise for robustness and injects noise at inference to improve generation. Experiments on ImageNet 256×256 show reconstruction PSNR above 32 dB with high semantic alignment, unguided gFID of 2.42 with the Base decoder, and 1.86 after switching to a larger XL decoder with the latents held fixed.
Critical strengths
The motivation and intuition are clear.
Treating the frozen VFM feature as a fixed semantic base and learning only the missing low-level residual on top of it is a natural idea. It also avoids the moving-target problem of directly finetuning the VFM, which the training curves in Fig. 8 illustrate well.
Critical weaknesses
Weakness 1. The reconstruction gain does not carry over to generation, which undermines the paper’s motivation.
Although App. H.1 states that LV-RAE does not aim for state-of-the-art generation, the introduction motivates it by reconstruction fidelity being a primary bottleneck for scaling LDMs. The main generation results (Table 4) use 0.1 latent noise, under which the Base decoder reconstructs at only 23.78 dB (Table 2). The reported 30.9 to 32.3 dB is therefore measured on clean latents rather than at the operating point used for generation. Table 8 also suggests that r contributes little to generation quality as measured by gFID. The DINO decoder, which is almost insensitive to r (Table 7), matches the best gFID among the tested decoders (1.86/1.41). Even with the better decoders, LV-RAE trails the RAE numbers listed in Table 4 for the same DiT^DH-XL model (1.86/1.42 vs. 1.60/1.28). Moreover, the cited RAE paper (Zheng et al., 2025; arXiv:2510.11690) reports 1.51/1.13 for this setting (DiT^DH-XL, 800 epochs, 256×256), not the 1.60/1.28 listed in Table 4, so the authors should clarify where these numbers come from. RAE also uses a ViT-XL decoder, comparable in scale to the XL decoder here, so this gap is not explained by decoder size. The encoder (DINOv3 vs. DINOv2) and the use of QK-Norm also differ, but the remaining gap suggests that the residual makes the latent harder to model. App. H.2 attributes the gap to the decoder, but since it only varies the decoder with the generated latents fixed, it cannot rule out this explanation.
Weakness 2. The scope is unclear, and each part of it lacks matching evidence.
If the goal is reconstruction, the use cases motivated in the introduction (low-level vision, precise editing) are never evaluated beyond 256×256 reconstruction metrics. If the goal is a better latent for generation, the premise that reconstruction fidelity bottlenecks the scaling of LDMs is never tested, and at the scale studied the higher-fidelity latent generates worse than RAE. If the goal is the decoder-side analysis, the claims are heavily hedged, and the evidence is either known or confounded. RAE already uses noise-augmented decoder training with a ViT-XL decoder, the Base and XL decoders differ in initialization and schedule, and the toy decoder amplifies off-manifold directions by construction. The remaining new component, inference-time noise injection, has a context-dependent optimal level that must be tuned empirically (App. E). Overall, the most critical point is that the gain is confined to reconstruction metrics on 256×256 images and is not shown to translate into better generation than RAE or into any downstream use.
Questions for authors
Please see weaknesses.
6 comments
W1 (4): The numbers reported for RAE
1.60 / 1.28 are taken from RAE's own appendix table (row "Latent Diffusion with RAE (Ours)", DiT^DH-XL, 800 epochs), which reports two sampling protocols:
- random class labels: 1.60 / 1.28;
- class-balanced sampling: 1.51 / 1.13, the numbers quoted in RAE's abstract.
We cite the random-label numbers because all of our results use random labels, as do the other methods in Tab. 4. Comparing our random-label results with RAE's class-balanced results would mix protocols. The caption of Tab. 4 now states the protocol.
W1 (5): The remaining gap and decoder size
We agree that RAE uses a ViT-XL decoder, but we do not claim that the gap is explained by decoder size. Tab. 8 shows the opposite: the DINO decoder uses the Base architecture and reaches the same 1.86 as the XL decoder on the same generated latents. What matters is decoder behavior, i.e., its response to errors in generated latents (§3.2), not its parameter count. The 0.26 gap should also be read against what it buys: on COCO, +13.7 dB PSNR (32.07 vs. 18.41) and rFDD 8.77 vs. 45.37 (Tab. 3).
W1 (6): Does the residual make the latent harder to model?
The review lists two differences between the systems that it does not quantify (encoder, QK-Norm), then attributes the remaining gap (0.26 gFID without guidance, 0.14 with) to a third, the residual. Both of the first two have been shown to raise gFID in RAE-style training:
- Encoder. Under the original RAE recipe, which our DiT pipeline follows (App. F.1), replacing DINOv2-B with DINOv3-B raises gFID from 3.75 to 4.25 (+0.50; RAEv2, Singh et al., 2026, Tab. 2, DiT^DH-XL, 20 epochs). The authors note that DINOv3-B performs "worse than DINOv2-B under the original RAE recipe". RAE's reported numbers use DINOv2-B; ours use DINOv3-B.
- QK-Norm. Chen et al. (2025) describe QK-Norm as "necessary to avoid loss spikes" in long runs. They report that enabling it raises LightningDiT's gFID from 2.17 to 2.50 without guidance and from 1.35 to 1.52 with guidance (800 epochs, their Tab. 4). As the reviewer notes, our pipeline uses QK-Norm and RAE's does not. We enable it for training stability (App. F.1), as is common in large-scale diffusion transformers such as SD3 (Esser et al., 2024).
Our setup therefore differs from RAE's in two factors, each shown to shift gFID by more than the gap in question. Those comparisons were run in other settings and their effects need not add, so we do not claim they account for the gap exactly. They do show that the gap cannot be read as evidence that r makes the latent harder to model.
Our own observations point the same way. Under the same decoder, diffusion training on u and on u + r leads to very similar generation performance, so the residual does not noticeably change the convergence of the generative model in our setting. This is also consistent with our inpainting experiment (W2), where an identically trained generator achieves a better FID on LV-RAE latents than on RAE latents.
References
- [RAE] B. Zheng, N. Ma, S. Tong, S. Xie. Diffusion Transformers with Representation Autoencoders. arXiv:2510.11690, 2025.
- [RAEv2] J. Singh, B. Zheng, Z. Wu, R. Zhang, E. Shechtman, S. Xie. Improved Baselines with Representation Autoencoders. arXiv:2605.18324, 2026.
- [Chen et al., 2025] B. Chen et al. Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models. arXiv:2509.25162, 2025.
- [Esser et al., 2024] P. Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML 2024.
W2: Scope
We respectfully disagree that the paper pursues three separate goals. It makes one connected argument, which the revision states at the start of §1:
- How to improve reconstruction without leaving the semantic space (§3.1). At the same latent size as RAE, LV-RAE improves PSNR by 13 dB, while CKNNA stays at 0.987 and a frozen linear probe keeps 80.8%.
- What problem this creates (§3.2). Once the latent carries low-level information, the decoder becomes highly sensitive to latent errors. Before noise fine-tuning, adding 0.1 noise drops our decoder from 32.32 to 17.72 dB (Tab. 2). By contrast, a decoder trained on the semantic feature alone barely changes when r is added (20.24 → 19.83 dB, Tab. 7). A decoder that extracts more information from the latent is also more sensitive to it. This is a decoder-side source of the reconstruction–generation trade-off, complementary to the latent-side explanation in prior work (Yao et al., 2025).
- A transparent study of that problem (§3.3, §4, App. H). We measure how far known remedies mitigate it and report what remains (the gap to RAE) and what is still open (App. E).
The reviewer observes that, at this scale, the higher-fidelity latent generates somewhat worse than RAE. This is not a counterexample to the paper; it is the phenomenon parts 2 and 3 study.
On the specific points:
- Downstream use. We report a controlled inpainting experiment, which is not in the submitted paper and will be included in the appendix of the revised paper (table below). On preserved content LV-RAE gains 6.7 dB PSNR over RAE, with better FID, so fidelity carries over to content-preserving generation.
| Tokenizer | FID↓ | PSNR on unmasked region↑ |
|---|---|---|
| RAE | 5.51 | 18.91 |
| LV-RAE | 5.28 | 25.63 |
DiT-XL with an identical training setting (80 epochs), only the tokenizer changed; 25% rectangular masks; first 5K ImageNet-val images; LV-RAE decoded with the Base decoder at σ̄ = 0.1.
- The scaling premise. This is motivation drawn from prior work (Labs et al., 2025; Team et al., 2025), not a claim we test. We do not claim better generation than RAE at the scale studied (App. H.1).
- Base/XL decoder comparison. The hedges are deliberate: they mark where our evidence does not isolate a cause. For the Base/XL comparison (§4.2), we state that the decoders differ in schedule as well as capacity. We have now separated the two. The Base decoder had converged at 100k steps, and training only the decoder (encoder frozen) to the XL schedule (150k) brings no gain (30.85 dB / gFID 2.50 vs. 30.91 / 2.42). The XL decoder's improvement (32.07 dB / 1.86) is therefore attributable to capacity; see our reply to tKCD. With the generated latents and the generator held fixed, a decoder-side change thus improves gFID by 0.56 and reconstruction by 1.2 dB at the same time (Tab. 7, Tab. 8).
- Novelty. The tool is known; the problem we study is not. RAE uses noise-augmented decoding so that the decoder of a semantic latent tolerates the gap between encoded and generated latents. Such a latent carries little fine-grained information, so noise costs little: in RAE, τ = 0.8 changes rFID only from 0.49 to 0.57. The problem we study appears only once reconstruction is improved (point 2 above). A decoder that must exploit fine-grained variations becomes sensitive to latent errors, a tension between fidelity and robustness that does not arise in RAE's setting. Our question is therefore how to gain robustness without losing that fidelity, and RAE's noise scale is far too large for it. Even a fixed σ = 0.1, one sixth of RAE's mean noise level, costs 7 dB of clean reconstruction, whereas U(0, 0.2) keeps 30.91 dB (Tab. 2). Likewise, RAE's ViT-XL decoder makes our XL decoder the scale-matched comparison, not a claimed component.
Response to Reviewer sEvG
We thank the reviewer for a detailed review. We address each step of the argument in the below.
W1 (1): Motivation (App. H.1 vs. the introduction)
These two statements are not in conflict. H.1 says that we do not target state-of-the-art gFID on ImageNet class-conditional generation. The introduction argues that reconstruction fidelity bounds what a latent diffusion model can produce: a generator cannot express detail that the decoder cannot render. Whether a given generator reaches that bound depends on its capacity. Our results are consistent with this: larger generators fit the information-rich latent more accurately and need less inference noise (Fig. 6, DiT^DH-XL vs. DiT-XL). A single ImageNet-256 gFID benchmark at one model scale therefore does not test a claim about scaling, which is the distinction H.1 draws.
W1 (2): Operating point of the reconstruction numbers
The 30.9–32.3 dB figures are measured on the reconstruction path (image → encoder → decoder), which never adds noise. σ̄ is applied only to latents produced by the diffusion model. An unconditionally generated sample has no ground-truth image, so "reconstruction PSNR at the generation operating point" is not defined for unconditional generation itself. The 23.78 dB in Tab. 2 is a robustness measurement (a real latent decoded after adding noise), not the fidelity of any pipeline in the paper. Note also that 32.3 dB is the decoder before noise fine-tuning; the decoder used for generation reconstructs at 30.9 dB on COCO.
More fundamentally, σ̄ is not a property of the tokenizer. It is a sampling-time setting that compensates for the generator's latent error, and how much it helps depends on how far the generator's samples deviate from the latent distribution. In our experiments we use a single value, σ̄ = 0.1, for all generators, chosen from the ablation in Fig. 6. Reconstruction fidelity is therefore the ceiling of the representation, reached as the generator approaches the latent distribution (σ̄ → 0). The operating point at σ̄ = 0.1 reflects the current generator rather than the representation. Our results follow this pattern: the advantage of the larger DiT^DH-XL over DiT-XL is largest at low σ̄ (Fig. 6), i.e., the stronger generator relies less on noise injection. Evaluating the tokenizer's fidelity at σ̄ = 0.1 thus conflates the representation with the generator trained on it.
Even at the reviewer's proposed operating point, the comparison with RAE is unchanged. With σ̄ = 0.1 added, our Base decoder reconstructs COCO at 23.78 dB / SSIM 0.735 / LPIPS 0.100 (Tab. 2). RAE decodes clean latents at generation time, and on the same data it reconstructs at 18.41 dB / 0.516 / 0.153 (Tab. 3). With each method at its own generation operating point, LV-RAE keeps a 5.4 dB advantage. The same holds in a generation task that does have a reference. Our inpainting experiment (see W2) uses exactly this operating point (Base decoder, σ̄ = 0.1), and LV-RAE preserves the unmasked region at 25.63 dB vs. 18.91 dB for RAE.
W1 (3): Whether r contributes to generation (Tab. 7, Tab. 8)
We agree with the observation but not with the conclusion drawn from it. The DINO decoder (about 20 dB) and the XL decoder (32 dB) reach the same gFID of 1.86 on the same generated latents. A metric that cannot separate a 12 dB difference in fidelity cannot measure what r contributes, so the observation describes gFID rather than r. Fig. 7 shows the same thing: decoded structure is shared across decoders, and the differences lie in fine-grained texture, which is exactly the information r carries.