- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes PSDR-AD, which jointly optimizes image-specific latents and a decoder without an encoder. It also refines encoder-initialized latents using frozen pretrained decoders, with experiments on reconstruction and downstream generation. Training from scratch adds Gaussian noise with a learnable scale regularized toward a target value. The FLUX-VAE experiment reports a reduction in ImageNet reconstruction FID from 0.176 to 0.005 with the decoder fixed.
Critical strengths
The problem is well motivated: improving reconstruction without enlarging the latent representation is useful. Table 3 reports improvements across 18 model–dataset pairs with fixed decoders, supporting the strong reconstruction results. The CIFAR-10 generation comparisons also improve over the corresponding AE/VQ-VAE baselines, making the evaluation broader than reconstruction alone.
Critical weaknesses
The distinction from earlier latent-optimization methods needs to be clearer. I would also like to understand the unusually strong reconstruction results more precisely: the appendix gives optimization settings, but FLUX's perceptual-loss network and its relationship to the rFID features remain unclear. These details matter when judging the reported gains.
The practical benefit also needs clarification. FLUX refinement costs 9.50 seconds per new image in batch-four throughput, which may matter for representation extraction despite the proposed offline use. Motivated by RAE and VAVAE, I am also concerned that independently optimized latents may be harder for a generative prior to learn. The positive CIFAR-10 results are encouraging, but leave this question open for the modern pretrained decoders.
Questions for authors
How much gain remains at shorter optimization budgets, and how does wall-clock cost compare with an encoder forward pass on the same hardware?
Which perceptual network and features are used for FLUX refinement, and how do they relate to the rFID features? Please specify any relevant differences in preprocessing.
Is there any existing evidence that refined pretrained-model latents are as easy to model as encoder outputs under a matched generative prior and training budget?
5 comments
Response to Reviewer FuNM — Part 2: Perceptual Loss and Generative Modeling
Q2: Inception-Based Perceptual Loss and Its Relationship to rFID
The original refinement experiments in Tables 3–4 combine pixel-space L2 loss with an LPIPS-style perceptual loss computed from the same Inception-v3 feature layer used for FID: the final average-pooling layer with 2,048-dimensional output. The perceptual term compares these features for each reconstruction and its corresponding target image. In contrast, rFID measures the Fréchet distance between the means and covariances of Inception features over the reconstructed and reference image sets. The loss and rFID therefore share the feature representation, but use different comparisons: paired-image feature distances versus distribution-level means and covariances. We explicitly acknowledge this dependence when interpreting the rFID improvement.
Using learned feature distances as optimization objectives has precedent. VIPaint uses a pretrained Inception network to compute a perceptual reconstruction loss for latent diffusion priors [1]. FastFID goes further by enabling direct optimization of FID as a training loss [2]. These precedents support the use of such objectives, while making clear the need to evaluate improvements beyond the shared feature space. The original Table 4 results also improve SSIM and VGG-based LPIPS, neither of which is a term in that Inception-based refinement objective: for FLUX-VAE, SSIM increases from 0.893 to 0.9144 and VGG-LPIPS decreases from 0.051 to 0.0286. These measurements provide complementary evidence of reconstruction improvement.
rFID uses pytorch-fid v0.2.1 with 2,048-dimensional Inception-v3 features. We will state the shared feature layer explicitly and document the checkpoint, normalization, and preprocessing used by the loss and rFID implementations in the revised manuscript.
[1] VIPaint, ICLR 2025 workshop paper, reconstruction-loss implementation and Figure 8.
[2] Alexander Mathiasen and Frederik Hvilshøj. Backpropagating through Fréchet Inception Distance, 2021 revision (FastFID).
Q3: Generative Modeling of Refined Representations
Table 5 provides direct downstream evidence under matched generative architectures and training budgets: PSDR-AD improves CIFAR-10 generation FID from 63.66 to 59.10 for autoregressive generation and from 81.96 to 68.02 for diffusion generation. These results support the generative utility of the learned representations in the evaluated setting. The reviewer's question about refined latents from modern pretrained decoders is a further evaluation: the CIFAR-10 results do not establish their relative modeling difficulty. We will make this scope explicit and retain the distinction between the demonstrated pretrained-decoder reconstruction benefits and the downstream generation evidence.
Follow-up after author response
My main concerns remain as follows.
-
The main demonstrated benefit is improved reconstruction, but obtaining these representations requires substantial per-image latent optimization. For downstream generative modeling, this would add significant preprocessing time when preparing a large latent training dataset. Offline extraction and caching allow this cost to be paid once per image, but the paper does not establish that the downstream benefits justify it. Another relevant alternative is to increase encoder or decoder capacity, potentially improving reconstruction and representation quality at a lower encoding cost than per-image latent optimization. Comparing against such alternatives would help clarify whether the proposed approach offers a worthwhile quality–cost trade-off.
-
The effect on representation quality remains unclear. Better reconstruction does not necessarily imply more useful features. For the pretrained FLUX-VAE experiments, the paper does not establish how refinement affects downstream representation-learning performance. ImageNet linear probing or another suitable evaluation could help clarify whether the optimized representations offer benefits beyond reconstruction.
-
The generative modeling experiments are limited in scale. The CIFAR-10 results do not establish whether the improvements persist on more complex datasets or with latents refined through modern pretrained decoders. The broader benefit for generative modeling therefore remains uncertain.
I am maintaining my weak-reject recommendation.
Response to Reviewer FuNM — Part 1: Contributions and Computational Cost
Thank you for recognizing the soundness of the work, the reconstruction improvements, and the positive generation results. This first part addresses the contributions and Q1 on computational cost; Part 2 addresses Q2 and Q3.
W1: Relationship to Earlier Latent Optimization Methods
As detailed in our general response, our contributions include the PSDR-Var formulation, noise-regularized auto-decoder training, and systematic evaluation of reconstruction improvements in modern pretrained tokenizers. Direct latent optimization is a capability of the framework that also enables its application to existing autoencoders. Tables 1–2 evaluate joint latent–decoder training, while Tables 3–4 establish the applicability and practical benefit of fixed-decoder optimization. We will make these contributions and their relationship to earlier methods more explicit.
Q1: Optimization Budget, Initialization, and Runtime
Our new experiments show that encoder initialization obtains most of the observed reconstruction improvement within a short optimization budget. We evaluate the frozen FLUX.1-dev VAE on 256 × 256 ImageNet validation images using an RTX 4080 SUPER, float32, batch size 1, and Adam with learning rate 0.01. These additional experiments use MSE + 0.1 VGG-LPIPS; their objective and timing configuration differ from the original Inception-based refinement experiments.
In a 32-image diagnostic, 100 updates capture approximately 96.5% of the loss reduction observed at 2,000 updates, improving PSNR from 32.47 to 34.15 dB. A further 1,900 updates add only 0.12 dB. Thus, the full optimization budget is unnecessary to obtain most of the improvement in this experiment.
Initialization also substantially affects quality at a given budget. The 256-image evaluation below averages metrics over three seeds per image and then over images. Gaussian initialization matches channel-wise encoder-latent statistics estimated from training images. At 1,000 updates, encoder initialization achieves substantially better reconstruction; even 10,000 Gaussian-initialized updates do not match it.
On the same hardware, a warmed-up encoder pass takes 10.83 ms, and encoder-only reconstruction (encode + decode) takes 30.59 ms per image. Encoder-initialized optimization costs 49.33 ms per update. Including initialization and final decoding, 100 updates are therefore estimated to cost approximately 4.96 seconds per image. This is a component-based estimate, not a directly measured end-to-end 100-update runtime. Separately measured complete jobs on eight images take 0.701 s/image for encoder-only reconstruction, 49.700 s/image for 1,000 encoder-initialized updates, and 492.282 s/image for 10,000 Gaussian-initialized updates, including startup, model loading, image I/O, and output saving.
These results quantify the quality–cost trade-off: encoder initialization permits substantial reconstruction gains at a shorter budget, while iterative refinement remains more expensive than a forward pass. The 32-image and 256-image results are separate evaluations; the new timings should also be distinguished from the original batch-four throughput measurement of 9.50 s/image.