- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes NA-VAE, a simple method using a spatial-aware alignment head and noise injection for improving VAE & representation-aligned variants' performance across diverse settings including ImageNet, T2I, T2A, and T2V.
Critical strengths
- The proposed method is simple and effective. Through small changes on the alignment head and the noise injection, the paper shows improvement with FID score across multiple different tasks.
- Extensive investigation. The paper supports the method with abundant ablations, including alignment-head architectures, noise schedules, generator sizes, and representation encoders, and provides extensive results over several different datasets.
Critical weaknesses
- My biggest concern is that, although demonstrating good improvement over the baseline VAE & representation-aligned VAE variants, the FID gain seems to diminish beyond the 80 epoch mark and become less competitive with baseline methods, especially when CFG is enabled. While improving convergence speed is desirable, I believe the terminal performance matters more.
- While the paper provides a good analysis on spatial inductive bias and low frequency's positive impact on latent diffusability, it remains unclear to me that whether the improvement with different alignment head comes from (1) improved expressiveness, or (2) simply more compute. It would be great if the author can supply more evidence with compute-controlled experiments.
- The authors should also include the terminal performance of the baselines in Table 2. for a more comprehensive understanding to the readers.
Questions for authors
- Have the authors tried this method on RAE (Zheng et al., 2026)? Is it as effective?
- It seems noise injection, while effective on SiT blocks, does not demonstrate a clear gain on MLP head. Do the authors have a clear intuition on this? Also it seems the clean Conv head stack is missing from Table 1.
- Do the authors have a clear intuition on why CFG appears less effective comparing to baselines? Have the authors tried other CFG methods for pushing the FID numbers?
We appreciate your thoughtful review and constructive suggestions. We respond to each of your concerns in detail.
Q1. Performance with extended training.
Notably, Tab.2 compares NA-VAE at 480 epochs with SD-VAE at 800 epochs. We resumed training during rebuttal and have now reached 600 epochs, obtaining gFID 1.26 / IS 312.4 with CFG, surpassing the baseline's 1.28 / 305.7. These results alleviate concerns about extended training. Moreover, (1) we have not extensively tuned CFG, whose effectiveness depends on latent properties (as discussed in REPA-E, RAE, RAEv2); (2) ImageNet may be insufficiently challenging for near-saturation comparisons, whereas Fig.7 shows NA-VAE substantially outperforming FLUX-VAE and Qwen-Image-VAE in diverse T2I settings with extended training budgets.
Q2. Compute-controlled alignment-head comparisons.
We summarize the results from Tabs.6–8 below.
\begin{array}{ccc} \textbf{SD-VAE} & \textbf{IN-VAE} & \textbf{VA-VAE} \\\\ \begin{array}[t]{c|cc} \text{Head} & \text{gFID}\downarrow & \text{IS}\uparrow \\\\ \hline \text{SiT-1} & 4.47 & 154.9 \\\\ \text{SiT-4} & 4.53 & 150.9 \\\\ \text{MLP} & 8.18 & 123.3 \\\\ \text{Conv} & 5.05 & 147.8 \end{array} & \begin{array}[t]{cc} \text{gFID}\downarrow & \text{IS}\uparrow \\\\ \hline 3.79 & 156.4 \\\\ 3.96 & 154.1 \\\\ 4.91 & 146.3 \\\\ 4.09 & 151.5 \end{array} & \begin{array}[t]{cc} \text{gFID}\downarrow & \text{IS}\uparrow \\\\ \hline 3.06 & 170.1 \\\\ 3.10 & 167.5 \\\\ 3.85 & 158.7 \\\\ 3.20 & 167.0 \end{array} \end{array}$$ SiT-1 outperforms SiT-4 with fewer FLOPs across all three VAE families, showing that more compute does not guarantee better generation. The Conv head also consistently outperforms MLP. **Q3. Terminal performance of baselines in Tab.2.** > *The authors should also include the terminal performance of the baselines in Table 2.* Please refer to Q1 for the intermediate results. We will add the 800-epoch results to Tab.2 once training is complete. **Q4. Applicability to RAE.** > *Have the authors tried this method on RAE (Zheng et al., 2026)? Is it as effective?* We have not evaluated NA-VAE on RAE or RAEv2. RAE keeps its pretrained encoder frozen to preserve semantic representations, whereas NA-VAE updates the encoder to learn diffusable latents. We view these as parallel research directions. **Q5. Noise injection with MLP and Conv heads.** > *It seems noise injection, while effective on SiT blocks, does not demonstrate a clear gain on MLP head. Do the authors have a clear intuition on this? Also it seems the clean Conv head stack is missing from Table 1.* We summarize the results from Tabs.6–8 below, including the clean Conv head. $$\scriptsize \begin{array}{ccc} \textbf{SD-VAE} & \textbf{IN-VAE} & \textbf{VA-VAE} \\\\ \begin{array}[t]{c|cc} \text{Head} & \text{gFID}\downarrow & \text{IS}\uparrow \\\\ \hline \text{MLP-}z & 7.02 & 133.6 \\\\ \text{MLP-}z_t & 8.18 & 123.3 \\\\ \text{Conv-}z & 4.81 & 147.1 \\\\ \text{Conv-}z_t & 5.05 & 147.8 \end{array} & \begin{array}[t]{cc} \text{gFID}\downarrow & \text{IS}\uparrow \\\\ \hline 4.74 & 147.6 \\\\ 4.91 & 146.3 \\\\ 4.60 & 145.3 \\\\ 4.09 & 151.5 \end{array} & \begin{array}[t]{cc} \text{gFID}\downarrow & \text{IS}\uparrow \\\\ \hline 4.15 & 157.8 \\\\ 3.85 & 158.7 \\\\ 3.60 & 158.8 \\\\ 3.20 & 167.0 \end{array} \end{array}$$ As discussed in our paper, we claim both noise and spatial inductive bias are essential to our approach, whereas the pointwise MLP has limited spatial modeling capacity. **Q6. CFG effectiveness** > *Do the authors have a clear intuition on why CFG appears less effective comparing to baselines?* As presented in Tab.2, carefully tuned CFG makes SD-VAE more competitive with E2E-VAE and NA-VAE in gFID, but trades diversity for precision: Recall decreases from 0.68 to 0.64, while Precision increases from 0.70 to 0.79. We interpret the smaller CFG gains of VA-VAE, REPA-E, and NA-VAE as reflecting near-saturated performance on ImageNet. Larger-scale T2I training can better reveal differences between these representations, as discussed in Q1.