- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes FS-VAE, a cross-resolution decoder fine-tuning method for recovering high-frequency details from low-resolution VAE latents. It uses a high-resolution reconstruction as a teacher and applies a steerable-pyramid frequency loss to the low-resolution decoder. Experiments on image and video VAEs report improvements in reconstruction metrics and face similarity.
Critical strengths
The method is easy to implement and preserves the existing latent space and encoder. The frequency-based analysis is clearly presented.
Critical weaknesses
-
The main frequency analysis is too obvious to constitute a meaningful contribution. In the band-swap experiment, the 256² reconstruction is first upsampled to 512² and then its frequency bands are replaced with those from the teacher. Since an upsampled 256² reconstruction necessarily lacks high-frequency information, inserting the teacher’s high-frequency bands will naturally improve the result. This experiment therefore only confirms an expected fact and provides limited scientific insight.
-
The practical value is not demonstrated. Under the same latent architecture, a 256-resolution representation has a fundamental quality ceiling below that of a 512-resolution representation, and the paper does not establish a compelling use case for recovering additional detail at 256 resolution.
-
The paper does not evaluate downstream diffusion generation quality using the fine-tuned VAE, which is essential for its claimed compatibility with latent diffusion models.
Questions for authors
Please see the weaknesses above.
4 comments
W1. Significance of the frequency analysis
We agree that the intuition behind our analysis is expected, as low-resolution VAEs lose fine detail, and upsampling a 256² reconstruction cannot recover information beyond its resolution. Beyond confirming this intuition, our analysis identifies the specific frequency bands in which identity information is lost, and shows how these bands vary with latent capacity.
The steerable pyramid splits each image into a high-pass band H, five frequency levels L0–L4 (centered at 128, 64, 32, 16, and 8 cycles per image), and a low-pass band Lo. As shown in Figure 3 in the main paper, the loss is not uniform across these bands. For 4-channel VAEs, the largest F-Sim gains come from L1 and L2, with H and L0 (the highest frequency bands) showing much less gain. This is the opposite of what upsampling alone would predict. H and L0 hold the frequencies that a 256² image cannot fully represent, so if the gains came only from the missing resolution, they would be largest there. Instead, most of the loss lies in L1 and L2, which 256² can represent. For 16-channel VAEs, the gains are smaller and shift to L0 and L1.
To confirm this directly, we repeat the experiment with I_G downsampled to 256² and upsampled back to 512² (GT256). GT256 contains only frequencies that 256² can represent, so any F-Sim gain from swapping in its bands is information the VAE lost, not information the resolution could not carry (Table R9).
Table R9: Band-swap F-Sim gains when swapping bands from GT256 and from the 512² teacher (ŷ_T).
| SD-VAE-ft-MSE | SDXL-VAE | |||
|---|---|---|---|---|
| band | from GT256 | from ŷ_T | from GT256 | from ŷ_T |
| H | −0.0020 | +0.0006 | −0.0021 | +0.0003 |
| L0 | +0.0290 | +0.0289 | +0.0384 | +0.0324 |
| L1 | +0.2011 | +0.1478 | +0.2027 | +0.1508 |
| L2 | +0.1383 | +0.1233 | +0.1260 | +0.1106 |
| L3 | +0.0344 | +0.0314 | +0.0301 | +0.0275 |
| L4 | +0.0039 | +0.0051 | +0.0031 | +0.0044 |
| Lo | −0.0016 | +0.0004 | −0.0016 | +0.0003 |
The largest gains again come from L1 and L2, while H gives none. The VAE loses identity information in bands that 256² can fully represent, so the loss is not an artifact of upsampling. This motivates the band-wise objective of FS-VAE.
W2. Practical value: fine detail at 256² with 4× cheaper generation, and generalization beyond 256²
We thank the reviewer for this comment. It is true that a 256² representation holds less information than 512², which is why fine-grained details are lost at lower resolutions.
Practical value at 256².
-
Cheaper generation. Lowering the resolution from 512² to 256² reduces the number of latent tokens by 4×, which makes generation about 4× cheaper for DiT. Low-resolution inputs are also common in practice, for example in satellite imagery, security, and edge devices (Section 1 in the main paper). FS-VAE improves reconstruction at 256² while keeping the cost low.
-
Zero-shot generalization to higher resolutions FS-VAE improves F-Sim on small faces at 256² by 8.7% and 10.2% (Table 2 in the main paper). The same decoder also improves small faces when evaluated directly at 512², without any training at 512² (Table R10).
Table R10: Evaluation of TokBench faces at 512², with FS-VAE trained only at 256².
| Model | s < 0.1 | 0.1–0.2 | 0.2–0.3 | s > 0.3 |
|---|---|---|---|---|
| SD-VAE-ft-MSE | 0.3794 | 0.5498 | 0.8222 | 0.9545 |
| FS-VAE, Stage 1 | 0.3989 (+5.1%) | 0.5654 (+2.8%) | 0.8255 (+0.4%) | 0.9549 (+0.0%) |
3.Generalization to other resolution pairs. We also train FS-VAE with a 1024² teacher and a 512² student on about 100K images from the same dataset used in the paper. It improves PSNR, SSIM, and F-Sim at 512² (Table R11). Our video VAE results also use a different resolution than 256² (Table 1 in the main paper). This shows training with a 1024² teacher and a 512² student also improves reconstruction at 512², thereby, making our method generalizable to other resolution pairs beyond 256².
Table R11: Reconstruction at 512² with FS-VAE trained with a 1024² teacher and a 512² student.
| Model (at 512²) | PSNR ↑ | SSIM ↑ | F-Sim ↑ |
|---|---|---|---|
| SD-VAE-ft-MSE | 29.83 | 0.8434 | 0.6059 |
| FS-VAE, Stage 1 (1024² → 512²) | 30.01 | 0.8504 | 0.6158 |
While our main motivation is to improve low-resolution latents for faster generation, these results show that FS-VAE generalizes across resolutions. We add these results to Appendix A.9 of the revised paper (Tables 14 and 15).
W3. Evaluation on downstream diffusion generation quality using the fine-tuned VAE
We thank the reviewer for this suggestion. We now evaluate FS-VAE inside pretrained diffusion models, keeping the generated latents fixed and replacing only the decoder. On ImageNet with DiT-XL/2, FS-VAE improves gFID (2.445 → 2.298), gsFID (5.16 → 4.42), and Inception Score (268.5 → 274.8). To measure fine detail in generated images, we run a face detector on images from SD1.5 and DiT-XL/2 on crowded scenes to analyze if the facial details are reconstructed well. With FS-VAE, up to 66.7% more of the smallest faces (10–16 px) are detected, meaning more small faces are generated with recognizable features. Full details are in our response to Reviewer BbWc, W5 (Tables R6 and R7). We add these results to Section 5.3 of the revised paper (Tables 4 and 5), with qualitative examples and video editing results in Appendix A.10 (Figure 14, Table 16).
We thank the reviewer for their feedback. We appreciate their recognition that our method is easy to implement and preserves the existing latent space and encoder, and that the frequency-based analysis is clearly presented.