acceptodds
Under review as a conference paper at ICLR 2027

Frequency-Steered VAEs: Restoring Fine Details in Low-Resolution Latents

Abstract

Latent diffusion models (LDM) rely on VAEs to compress visual inputs for computational efficiency, which is particularly consequential for very high resolution visual inputs. Through an in-depth interpretability analysis, we show that this compression disproportionately degrades high-frequency information at lower spatial resolutions, discarding fine details such as facial features or text patterns. Building on this insight, we propose a we call that uses a higher-resolution VAE as a teacher to supervise its lower-resolution counterpart, recovering lost frequency information via a band-wise . FS-VAE enables richer reconstruction from low-resolution latents without expanding channel dimensions, reference encoders, external models, or retraining downstream diffusion architectures. Furthermore, we analyze the impact of distinct loss objectives on reconstruction quality and demonstrate our approach's generalizability across diverse image and video VAEs. By preserving the original architecture and latent dimensionality, FS-VAE acts as an efficient, plug-and-play replacement for existing LDM pipelines.

Primary area:
generative models
Keywords:
Variational autoencoders, Low resolution image reconstruction, Self-distillation, Steerable pyramid

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_V6tBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FS-VAE, a cross-resolution decoder fine-tuning method for recovering high-frequency details from low-resolution VAE latents. It uses a high-resolution reconstruction as a teacher and applies a steerable-pyramid frequency loss to the low-resolution decoder. Experiments on image and video VAEs report improvements in reconstruction metrics and face similarity.

Critical strengths

The method is easy to implement and preserves the existing latent space and encoder. The frequency-based analysis is clearly presented.

Critical weaknesses
  1. The main frequency analysis is too obvious to constitute a meaningful contribution. In the band-swap experiment, the 256² reconstruction is first upsampled to 512² and then its frequency bands are replaced with those from the teacher. Since an upsampled 256² reconstruction necessarily lacks high-frequency information, inserting the teacher’s high-frequency bands will naturally improve the result. This experiment therefore only confirms an expected fact and provides limited scientific insight.

  2. The practical value is not demonstrated. Under the same latent architecture, a 256-resolution representation has a fundamental quality ceiling below that of a 512-resolution representation, and the paper does not establish a compelling use case for recovering additional detail at 256 resolution.

  3. The paper does not evaluate downstream diffusion generation quality using the fine-tuned VAE, which is essential for its claimed compatibility with latent diffusion models.

Questions for authors

Please see the weaknesses above.

4 comments

We thank the reviewer for their feedback. We appreciate their recognition that our method is easy to implement and preserves the existing latent space and encoder, and that the frequency-based analysis is clearly presented.

AuthorsOpenReview ↗

W1. Significance of the frequency analysis

We agree that the intuition behind our analysis is expected, as low-resolution VAEs lose fine detail, and upsampling a 256² reconstruction cannot recover information beyond its resolution. Beyond confirming this intuition, our analysis identifies the specific frequency bands in which identity information is lost, and shows how these bands vary with latent capacity.

The steerable pyramid splits each image into a high-pass band H, five frequency levels L0–L4 (centered at 128, 64, 32, 16, and 8 cycles per image), and a low-pass band Lo. As shown in Figure 3 in the main paper, the loss is not uniform across these bands. For 4-channel VAEs, the largest F-Sim gains come from L1 and L2, with H and L0 (the highest frequency bands) showing much less gain. This is the opposite of what upsampling alone would predict. H and L0 hold the frequencies that a 256² image cannot fully represent, so if the gains came only from the missing resolution, they would be largest there. Instead, most of the loss lies in L1 and L2, which 256² can represent. For 16-channel VAEs, the gains are smaller and shift to L0 and L1.

To confirm this directly, we repeat the experiment with I_G downsampled to 256² and upsampled back to 512² (GT256). GT256 contains only frequencies that 256² can represent, so any F-Sim gain from swapping in its bands is information the VAE lost, not information the resolution could not carry (Table R9).

Table R9: Band-swap F-Sim gains when swapping bands from GT256 and from the 512² teacher (ŷ_T).

SD-VAE-ft-MSESDXL-VAE
bandfrom GT256from ŷ_Tfrom GT256from ŷ_T
H−0.0020+0.0006−0.0021+0.0003
L0+0.0290+0.0289+0.0384+0.0324
L1+0.2011+0.1478+0.2027+0.1508
L2+0.1383+0.1233+0.1260+0.1106
L3+0.0344+0.0314+0.0301+0.0275
L4+0.0039+0.0051+0.0031+0.0044
Lo−0.0016+0.0004−0.0016+0.0003

The largest gains again come from L1 and L2, while H gives none. The VAE loses identity information in bands that 256² can fully represent, so the loss is not an artifact of upsampling. This motivates the band-wise objective of FS-VAE.

AuthorsOpenReview ↗

W2. Practical value: fine detail at 256² with 4× cheaper generation, and generalization beyond 256²

We thank the reviewer for this comment. It is true that a 256² representation holds less information than 512², which is why fine-grained details are lost at lower resolutions.

Practical value at 256².

  1. Cheaper generation. Lowering the resolution from 512² to 256² reduces the number of latent tokens by 4×, which makes generation about 4× cheaper for DiT. Low-resolution inputs are also common in practice, for example in satellite imagery, security, and edge devices (Section 1 in the main paper). FS-VAE improves reconstruction at 256² while keeping the cost low.

  2. Zero-shot generalization to higher resolutions FS-VAE improves F-Sim on small faces at 256² by 8.7% and 10.2% (Table 2 in the main paper). The same decoder also improves small faces when evaluated directly at 512², without any training at 512² (Table R10).

Table R10: Evaluation of TokBench faces at 512², with FS-VAE trained only at 256².

Models < 0.10.1–0.20.2–0.3s > 0.3
SD-VAE-ft-MSE0.37940.54980.82220.9545
FS-VAE, Stage 10.3989 (+5.1%)0.5654 (+2.8%)0.8255 (+0.4%)0.9549 (+0.0%)

3.Generalization to other resolution pairs. We also train FS-VAE with a 1024² teacher and a 512² student on about 100K images from the same dataset used in the paper. It improves PSNR, SSIM, and F-Sim at 512² (Table R11). Our video VAE results also use a different resolution than 256² (Table 1 in the main paper). This shows training with a 1024² teacher and a 512² student also improves reconstruction at 512², thereby, making our method generalizable to other resolution pairs beyond 256².

Table R11: Reconstruction at 512² with FS-VAE trained with a 1024² teacher and a 512² student.

Model (at 512²)PSNR ↑SSIM ↑F-Sim ↑
SD-VAE-ft-MSE29.830.84340.6059
FS-VAE, Stage 1 (1024² → 512²)30.010.85040.6158

While our main motivation is to improve low-resolution latents for faster generation, these results show that FS-VAE generalizes across resolutions. We add these results to Appendix A.9 of the revised paper (Tables 14 and 15).

AuthorsOpenReview ↗

W3. Evaluation on downstream diffusion generation quality using the fine-tuned VAE

We thank the reviewer for this suggestion. We now evaluate FS-VAE inside pretrained diffusion models, keeping the generated latents fixed and replacing only the decoder. On ImageNet with DiT-XL/2, FS-VAE improves gFID (2.445 → 2.298), gsFID (5.16 → 4.42), and Inception Score (268.5 → 274.8). To measure fine detail in generated images, we run a face detector on images from SD1.5 and DiT-XL/2 on crowded scenes to analyze if the facial details are reconstructed well. With FS-VAE, up to 66.7% more of the smallest faces (10–16 px) are detected, meaning more small faces are generated with recognizable features. Full details are in our response to Reviewer BbWc, W5 (Tables R6 and R7). We add these results to Section 5.3 of the revised paper (Tables 4 and 5), with qualitative examples and video editing results in Appendix A.10 (Figure 14, Table 16).

Reviewer_CyiTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces the Frequency-Steered VAE (FS-VAE), a framework designed to improve the reconstruction of fine-grained visual details from low-resolution VAE latents. The authors first investigate frequency-dependent reconstruction degradation through a steerable-pyramid band-swapping analysis, showing that replacing selected mid-to-high frequency bands significantly improves face reconstruction. In particular, the paper introduces a cross-resolution self-distillation objective that aligns the frequency-band magnitudes of low-resolution reconstructions with those obtained from a higher-resolution teacher. The method fine-tunes only the VAE decoder, preserving the original latent representation, and involves an additional adversarial post-tuning stage to mitigate reconstruction artifacts.

The method is evaluated across image and video VAEs using reconstruction quality and face similarity metrics, with particular attention to small regions. The authors also investigate different distillation targets and frequency-band training schedules, conduct a human preference study, and provide qualitative examples of integration with a pretrained diffusion model.

Critical strengths

The paper presents two main strengths, particularly regarding the practical applicability of the proposed approach and the insights provided by the preliminary analysis.

Decoder-only adaptation. The proposed framework improves reconstruction fidelity in several settings while preserving the pretrained encoder and latent representation. The reported gains on small face regions and the improvements obtained when integrating the frequency-aware objective into RefDecoder suggest that decoder-only adaptation can be a useful strategy for enhancing existing VAE architectures. Moreover, preserving the latent interface makes the approach potentially attractive for integration into pretrained generative pipelines without retraining the diffusion model.

Frequency-band analysis. The investigation presented in Section 3 is particularly valuable, as it goes beyond simply proposing a new reconstruction objective and attempts to provide a deeper understanding of the underlying problem. The steerable-pyramid band-swapping analysis offers an interesting and interpretable characterization of frequency-dependent reconstruction degradation in low-resolution VAEs. By identifying the frequency bands whose replacement most strongly affects facial similarity and highlighting differences across latent capacities, the study provides useful empirical insights and a meaningful motivation for frequency-aware reconstruction objectives.

Critical weaknesses

The paper presents several weaknesses that substantially affect the assessment of its scientific contribution and significance. The two most critical weaknesses concern the validation and the methodological novelty.

Insufficient validation of the benefits for latent diffusion models. The paper is strongly motivated by the limitations that VAE reconstruction imposes on latent diffusion models, presenting FS-VAE as a plug-and-play solution for improving generation without retraining the downstream diffusion model. However, the experimental evaluation focuses almost entirely on VAE reconstruction, leaving the central generative motivation insufficiently validated. Some qualitative DiT-XL/2 examples are only provided in Appendix A.5 suggesting that the proposed decoder can be integrated into a pretrained diffusion pipeline. However, even in the appendix such examples do not provide systematic quantitative evidence that improved reconstruction fidelity translates into better generative performance. This distinction is particularly important because a diffusion model may partially compensate for reconstruction limitations by learning suitable latent representations, and improvements in VAE reconstruction do not necessarily imply corresponding improvements in generated samples. Furthermore, the potential computational benefits of using more compact latents, explicitly discussed in the Introduction, remain unverified. A quantitative evaluation of generation quality using pretrained diffusion models, including an analysis of high-frequency details in generated images, would be essential to substantiate the paper's central motivation. Without such evidence, the demonstrated contribution remains primarily a VAE reconstruction improvement rather than a validated advancement for latent diffusion models.

Limited methodological novelty. Despite the interesting diagnostic analysis in Section 3, the proposed methodology largely combines established techniques, namely cross-resolution self-distillation, frequency-domain reconstruction losses, and adversarial post-tuning. While such a combination can be valuable, the experiments do not sufficiently establish the necessity or individual contribution of the main design choices. Most importantly, a direct comparison between conventional cross-resolution distillation and the proposed steerable-pyramid-based distillation is missing, making it difficult to determine whether the improvements arise from the frequency-aware objective or simply from the teacher-student setup. Furthermore, the choice of matching only the magnitudes of complex frequency coefficients is not adequately justified, particularly given the importance of phase information for spatial structure and the periodic artifacts introduced during Stage 1. The subsequent adversarial post-tuning mitigates these artifacts but also reduces some of the reconstruction gains, raising questions about the effectiveness of the two-stage design compared with a unified optimization objective. More targeted ablation studies are needed to establish the specific methodological contribution of FS-VAE beyond the combination of existing training strategies.

Additional concerns. The interpretation of the frequency-band analysis in Section 3 appears somewhat overstated, as band swapping identifies frequency components that affect facial similarity without establishing whether the corresponding information is lost during encoding or inadequately reconstructed by the decoder. Furthermore, the experimental evaluation remains predominantly focused on faces, limiting the assessment of the method's generalization to other fine-grained structures, such as text and textures. The human preference study is also relatively limited in scale and primarily evaluates perceptual realism rather than reconstruction fidelity. Finally, the choice of an orientation-selective steerable pyramid could be better motivated, given that the reported analysis aggregates results across orientations, and the Related Work would benefit from a broader discussion of frequency-aware approaches operating directly within diffusion models.

Questions for authors

The following questions aim to clarify aspects of the proposed methodology and experimental validation that could affect the assessment of the paper.

Q1. Impact on diffusion generation. Given that the paper is strongly motivated by improving latent diffusion models, can the authors provide convincing quantitative evidence that FS-VAE leads to significant improvements in the final generative performance of pretrained diffusion pipelines? Beyond demonstrating decoder compatibility, it is important to establish where and how the proposed approach improves the generation process, and to what extent the observed reconstruction gains translate into better generated samples, particularly in terms of high-frequency detail preservation. Currently, the only direct diffusion-generation experiment consists of a limited qualitative comparison with DiT-XL/2 in Appendix A.5. Considering the central role of diffusion models in motivating the work, such an evaluation would deserve a prominent place in the main paper, supported by systematic quantitative results under identical sampling conditions.

Q2. Contribution of frequency-domain distillation. How much of the observed improvement can be attributed specifically to the proposed steerable-pyramid distillation objective, rather than to cross-resolution teacher-student training itself? Could the authors provide a controlled comparison with conventional pixel-space or perceptual distillation using the same teacher, student architecture, and training budget?

Q3. Information recovery vs. detail synthesis. Since the encoder and latent representation remain unchanged, what evidence supports the claim that FS-VAE recovers information already present in the low-resolution latents, rather than synthesizing perceptually plausible high-frequency details? More specifically, how does the frequency band-swapping analysis distinguish information loss during encoding from limitations of the original decoder?

Q4. Magnitude-only matching and adversarial post-tuning. What motivates the choice of matching only the magnitudes of the complex steerable-pyramid coefficients, given the importance of phase information for spatial structure? Could the authors clarify whether the periodic artifacts observed after Stage 1 are related to this choice, and whether alternative frequency-domain objectives or joint optimization could avoid the need for a separate adversarial post-tuning stage?

8 comments

We thank the reviewer for their detailed and thoughtful feedback. We appreciate their recognition of the practical value of decoder-only adaptation, which preserves the pretrained encoder and latent representation and allows integration into pretrained generative pipelines without retraining the diffusion model. We are also glad that the reviewer finds our frequency-band analysis particularly valuable, as it provides a deeper understanding of the underlying problem and a meaningful motivation for frequency-aware reconstruction objectives.

AuthorsOpenReview ↗

W1. Insufficient validation of the benefits for latent diffusion models

We thank the reviewer for this suggestion. We now evaluate FS-VAE inside pretrained diffusion models, keeping the generated latents fixed and replacing only the decoder. On ImageNet with DiT-XL/2, FS-VAE improves gFID (2.445 → 2.298), gsFID (5.16 → 4.42), and Inception Score (268.5 → 274.8). To measure fine detail in generated images, we run a face detector on images from SD1.5 and DiT-XL/2 on crowded scenes to analyze if the facial details are reconstructed well. With FS-VAE, up to 66.7% more of the smallest faces (10–16 px) are detected, meaning more small faces are generated with recognizable features. Full details are in our response to Reviewer BbWc, W5 (Tables R6 and R7). Please refer to Section 5.3 (Tables 4, 5), Appendix A.10 in the revised paper

AuthorsOpenReview ↗

W2. Limited methodological novelty

We thank the reviewer for this comment. In our response to Reviewer BbWc, W1 (Tables R1 and R2), we ablate each component under the same setup. Pixel distillation (L1 + LPIPS) from a 512² target lowers F-Sim and worsens rFID (12.07 vs. 9.04), and the frequency loss with ground-truth targets lowers F-Sim or blurs the image. Only the combination of the frequency loss and the 512² teacher gives the highest F-Sim, which shows that both components are needed. For the choice of magnitude-only matching and the two-stage design, please see Q4 below.

Reviewer_BbWcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies how to improve the detail reconstructions capacity of VAE on small faces. It propose pyramid “band-swap” analysis and uses face-identity similarity (TokBench F-Sim) to analyze which bands are more important for preserving visual quality such as face details. the authors conclude that the loss is concentrated in mid-to-high-frequency bands (L1/L2 for 4-channel VAEs) and is more severe for lower-capacity latents. Then the paper proposed FS-VAE that finetues the VAE decoder. The author propose a steerable-pyramid self-distillation loss, which tries to match the the per-band steerable-pyramid magnitudes of the same VAE’s 512 reconstruction. Experiments cover SD-VAE-ft-MSE, SDXL-VAE, Wan 2.1 VAE, and RefDecoder. On TokBench faces at 256², F-Sim improves from 0.4542 to 0.4667.

Critical strengths
  • Practical, cleanly scoped problem with a plug-and-play design. Loss of identity-bearing detail at low resolution is a real limitation, and F-Sim captures it better than PSNR, SSIM, or LPIPS. Because only the decoder is changed, the latent space is preserved and existing diffusion models can use the new decoder directly. The method is shown on both image and video VAEs, including as an add-on to RefDecoder.
  • An intuitive diagnostic tool. The band-swap analysis is a simple, interpretable way to ask where in the frequency spectrum a reconstruction loses task-relevant information. The face-size breakdown, with gains of +8.7% and +10.2% for the smallest scale bins, is a useful complement to aggregate metrics.
Critical weaknesses
  • Novelty. Fine-tuning with frequency- and wavelet-domain reconstruction losses are established. The cross-resolution distillation is somewhat novel, but it needs a better analysis to separate where the gain comes from, e.g. is it from cross-resolution distillation, or using a 512px GT image as training target, or using the steerable-pyramid loss.
  • Marginal gains over the baselines:
    1. Against a decoder fine-tuned on the same data with L1, LPIPS, and GAN losses, the final model improves F-Sim by about 0.007 (0.4595 → 0.4667).
    2. On Wan 2.1, F-Sim is essentially unchanged (0.5215 → 0.5219).
    3. The RefDecoder gain is reported as +2.73% relative to the released checkpoint, but it is only 0.5644 → 0.5684 relative to the same-data fine-tune.
    4. rFID often gets worse: SDXL 7.10 → 7.58, ImageNet 0.70 → 1.18, Wan 9.13 → 9.54.
  • There are some inconsistent reporting in the paper:
    1. App. A.4 states that Stage 2 brings ImageNet rFID below the baseline, but Table 4 shows 0.70 → 1.18.
    2. The conclusion claims lower FVD across video VAEs, but Wan 2.1 rFVD on TokBench worsens from 8.30 to 8.80.
    3. In Fig. 11, three different faces show identical F-Sim values (0.168 / 0.208 / 0.214 / 0.237).
  • The evaluation only considers 256px resolution and the method is heavily optimized for reconstructing small face. It's unclear how it can generalize to other important small objects such as texts, small structures.
  • Can the proposed method work for diffusion-generated images? The paper need conduct experiment on real GenAI use case, otherwise it hurt the usability of the method.
Questions for authors
  • The author are encourage to conduct experiment to show the generalize of the method, e.g. can the proposed method generalize to small text or small structures other than faces. Also how will it work on the real T2I settings?
  • better analysis or ablation are needed to separate where the gain comes from, e.g. is it from cross-resolution distillation, or using a 512px GT image as training target, or using the steerable-pyramid loss.
7 comments

We thank the reviewer for their thoughtful evaluation and constructive feedback. We appreciate their recognition of the importance of preserving identity-bearing details, the plug-and-play nature of our approach, and the insights provided by our band-swapping analysis.

AuthorsOpenReview ↗

W1. Novelty: both higher-resolution supervision and the frequency loss are needed

We thank the reviewer for this comment. FS-VAE makes two contributions.

  1. Targeted frequency supervision. Prior frequency-domain methods (Björk et al., 2022; Lin et al., 2023; Sadat et al., 2024; Kiruluta, 2025; Li et al., 2025) apply frequency losses uniformly across the spectrum. We first use the band-swap analysis to find where identity is lost: it is concentrated in a few bands, and these bands change with the VAE's latent capacity. We then supervise these bands with the higher-resolution teacher.
  2. No architectural changes. Prior methods recover fine detail by expanding the latent channels or changing the architecture (Yao et al., 2025; Cai et al., 2026; Medi et al., 2026). FS-VAE fine-tunes only the decoder, so the encoder, latent space, and architecture stay the same, and FS-VAE works with pretrained diffusion models without retraining.

Where do the gains come from? FS-VAE combines a frequency-aware loss with a higher-resolution target. We ablate each one.

Higher-resolution target alone is not enough. We replace the steerable-pyramid loss with pixel losses (L1 + LPIPS) against a 512² target (Table R1). With either the teacher or the ground truth as the target, F-Sim is lower, rFID is much worse (12.07 vs. 9.04), and fine details are not well reconstructed, unlike FS-VAE Stage 1.

Table R1: Pixel distillation versus frequency-aware supervision.

MethodL1 + LPIPS targetSteerable targetPSNR ↑SSIM ↑LPIPS ↓rFID ↓F-Sim ↑
Pixel distillationTeacher @ 512²✗26.100.76920.172712.070.4662
Pixel distillationGT @ 512²✗25.940.76560.173911.930.4674
FS-VAE (Stage 1)GT @ 256²Teacher @ 512²26.360.78040.15259.040.4729

The frequency loss alone is not enough either. We keep the steerable-pyramid loss but change its target (Table R2). With the 256² ground truth, pixel metrics improve slightly but the image is visibly blurred. With the 512² ground truth, rFID improves but F-Sim drops. The 512² teacher gives the highest F-Sim.

Table R2: Effect of the steerable-pyramid target.

MethodL1 + LPIPS targetSteerable targetPSNR ↑SSIM ↑LPIPS ↓rFID ↓F-Sim ↑
Frequency supervisionGT @ 256²GT @ 256²26.570.78270.15159.130.4709
Cross-resolution frequency supervisionGT @ 256²GT @ 512²26.180.77660.15198.580.4669
FS-VAE (Stage 1)GT @ 256²Teacher @ 512²26.360.78040.15259.040.4729

The two components are complementary: only their combination gives the best identity preservation. We add these ablations to Section 5.3 of the revised paper (Table 3), with full results in Appendix A.5.1 and visual comparisons in Appendix A.6 (Figure 13).

AuthorsOpenReview ↗

W2. Marginal gains over the baselines

W2.1. Against a decoder fine-tuned on the same data with L1, LPIPS, and GAN losses, the final model improves F-Sim by about 0.007 (0.4595 → 0.4667)

We thank the reviewer for this observation. The average F-Sim gain over fine-tuning is modest, but FS-VAE is better in three other ways.

Higher pixel fidelity. Fine-tuning lowers PSNR and SSIM below the baseline. FS-VAE Stage 1 improves both (PSNR 26.36 vs. 25.86, SSIM 0.7804 vs. 0.7672; Table R3).

Table R3: Reconstruction quality on TokBench at 256² (from Table 1 in the main paper).

MethodAdversarial trainingPSNR ↑SSIM ↑LPIPS ↓rFID ↓F-Sim ↑
SD-VAE-ft-MSE–26.150.77030.15367.860.4542
+ fine-tuning (L1 + LPIPS + GAN)Full training25.860.76720.15157.380.4595
+ FS-VAE, Stage 1None26.360.78040.15259.040.4729
+ FS-VAE, Stage 1+25 epochs, 100K images26.150.77600.15037.760.4667

FS-VAE offers simpler training. A GAN loss needs a discriminator, a second optimizer, and an extra forward and backward pass. The steerable-pyramid loss has no learnable parameters and can be added to any loss.

FS-VAE offers faster training. Adding the steerable-pyramid loss to L1 + LPIPS increases time per batch by only 4.5%. Adding the GAN loss increases it by 49.7% (Table R4). FS-VAE uses the GAN loss only for a short Stage 2 (5 epochs on 100K images), while fine-tuning uses it for the full run.

Table R4: Training time per batch (batch size 48).

Lossms / batchvs. L1 + LPIPS
L1 + LPIPS733.9–
+ steerable pyramid (FS-VAE, Stage 1)766.7+4.5%
+ GAN (fine-tuning)1098.6+49.7%
+ both1126.2+53.4%

Stage 2 is also not always needed. For RefDecoder, Stage 1 alone, with no GAN loss, improves all metrics (see Table 1 in the main paper).

W2.3. The RefDecoder gain is reported as +2.73% relative to the released checkpoint, but it is only 0.5644 → 0.5684 relative to the same-data fine-tune

Thank you for pointing this out. The +2.73% gain is relative to the released RefDecoder. Relative to the same-data fine-tune, the F-Sim gain is +0.7% (0.5644 → 0.5684). We will correct this in the revised manuscript.

Against the same-data fine-tune, FS-VAE still improves all metrics (Table 1 in the main paper). The largest gain is in temporal consistency: rFVD drops by 16% (9.65 → 8.11), and rFID improves from 9.70 to 9.11.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.