acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Encoder: Latent Representation Learning via Parametric Soft Discrete Random Variables

Abstract

We propose the Parametric Soft Discrete Random Variable Auto-Decoder (PSDR-AD), a novel encoder-free framework for learning latent representations through directly optimized Parametric Soft Discrete Random Variables (PSDR-Vars). Our approach is motivated by the hypothesis that encoder-produced representations may not be optimal for image reconstruction because they are restricted by the learned encoder mapping. Specifically, each image is associated with a dedicated PSDR-Var, a directly optimizable discrete latent variable with a soft stochastic relaxation induced by noise injection. During training, the PSDR-Vars are fed into a shared decoder to reconstruct their corresponding images, while the PSDR-Vars and decoder are jointly optimized. During testing, the decoder parameters are fixed, while newly initialized PSDR-Vars are assigned to unseen images and optimized to minimize their reconstruction errors. Unlike conventional autoencoders, which treat latent extraction as a forward function evaluation that maps an input image to the output of an encoder, PSDR-AD treats it as an inverse optimization problem: given a target image as the desired output of the decoder, it optimizes the decoder input to reproduce that output. This removes the additional constraint imposed by the encoder mapping and can improve reconstruction quality even when PSDR-AD is applied to a decoder from a pre-trained autoencoder. Comprehensive experiments demonstrate the effectiveness of PSDR-AD for latent representation learning. Using the same pre-trained FLUX-VAE decoder, PSDR-AD reduces the rFID from 0.176 to 0.005 on ImageNet through direct latent optimization, achieving a lower rFID than existing continuous reconstruction methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
latent representation learning, auto-decoder, direct latent optimization, discrete representation learning, image reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FuNMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PSDR-AD, which jointly optimizes image-specific latents and a decoder without an encoder. It also refines encoder-initialized latents using frozen pretrained decoders, with experiments on reconstruction and downstream generation. Training from scratch adds Gaussian noise with a learnable scale regularized toward a target value. The FLUX-VAE experiment reports a reduction in ImageNet reconstruction FID from 0.176 to 0.005 with the decoder fixed.

Critical strengths

The problem is well motivated: improving reconstruction without enlarging the latent representation is useful. Table 3 reports improvements across 18 model–dataset pairs with fixed decoders, supporting the strong reconstruction results. The CIFAR-10 generation comparisons also improve over the corresponding AE/VQ-VAE baselines, making the evaluation broader than reconstruction alone.

Critical weaknesses

The distinction from earlier latent-optimization methods needs to be clearer. I would also like to understand the unusually strong reconstruction results more precisely: the appendix gives optimization settings, but FLUX's perceptual-loss network and its relationship to the rFID features remain unclear. These details matter when judging the reported gains.

The practical benefit also needs clarification. FLUX refinement costs 9.50 seconds per new image in batch-four throughput, which may matter for representation extraction despite the proposed offline use. Motivated by RAE and VAVAE, I am also concerned that independently optimized latents may be harder for a generative prior to learn. The positive CIFAR-10 results are encouraging, but leave this question open for the modern pretrained decoders.

Questions for authors

How much gain remains at shorter optimization budgets, and how does wall-clock cost compare with an encoder forward pass on the same hardware?

Which perceptual network and features are used for FLUX refinement, and how do they relate to the rFID features? Please specify any relevant differences in preprocessing.

Is there any existing evidence that refined pretrained-model latents are as easy to model as encoder outputs under a matched generative prior and training budget?

5 comments
AuthorsOpenReview ↗

Response to Reviewer FuNM — Part 1: Contributions and Computational Cost

Thank you for recognizing the soundness of the work, the reconstruction improvements, and the positive generation results. This first part addresses the contributions and Q1 on computational cost; Part 2 addresses Q2 and Q3.

 

W1: Relationship to Earlier Latent Optimization Methods

As detailed in our general response, our contributions include the PSDR-Var formulation, noise-regularized auto-decoder training, and systematic evaluation of reconstruction improvements in modern pretrained tokenizers. Direct latent optimization is a capability of the framework that also enables its application to existing autoencoders. Tables 1–2 evaluate joint latent–decoder training, while Tables 3–4 establish the applicability and practical benefit of fixed-decoder optimization. We will make these contributions and their relationship to earlier methods more explicit.

 

Q1: Optimization Budget, Initialization, and Runtime

Our new experiments show that encoder initialization obtains most of the observed reconstruction improvement within a short optimization budget. We evaluate the frozen FLUX.1-dev VAE on 256 × 256 ImageNet validation images using an RTX 4080 SUPER, float32, batch size 1, and Adam with learning rate 0.01. These additional experiments use MSE + 0.1 VGG-LPIPS; their objective and timing configuration differ from the original Inception-based refinement experiments.

In a 32-image diagnostic, 100 updates capture approximately 96.5% of the loss reduction observed at 2,000 updates, improving PSNR from 32.47 to 34.15 dB. A further 1,900 updates add only 0.12 dB. Thus, the full optimization budget is unnecessary to obtain most of the improvement in this experiment.

Encoder-initialized inference (32 images)UpdatesPSNR ↑Reconstruction loss ↓
Encoder-only032.470.008054
Short-budget refinement10034.150.004638
Extended refinement2,00034.270.004515

Initialization also substantially affects quality at a given budget. The 256-image evaluation below averages metrics over three seeds per image and then over images. Gaussian initialization matches channel-wise encoder-latent statistics estimated from training images. At 1,000 updates, encoder initialization achieves substantially better reconstruction; even 10,000 Gaussian-initialized updates do not match it.

Inference (256 images)UpdatesPSNR ↑SSIM ↑VGG-LPIPS ↓
Encoder-only031.000.88950.0516
Encoder initialization1,00032.910.91590.0274
Matched Gaussian initialization1,00026.450.71730.1634
Matched Gaussian initialization10,00030.800.86920.0532

On the same hardware, a warmed-up encoder pass takes 10.83 ms, and encoder-only reconstruction (encode + decode) takes 30.59 ms per image. Encoder-initialized optimization costs 49.33 ms per update. Including initialization and final decoding, 100 updates are therefore estimated to cost approximately 4.96 seconds per image. This is a component-based estimate, not a directly measured end-to-end 100-update runtime. Separately measured complete jobs on eight images take 0.701 s/image for encoder-only reconstruction, 49.700 s/image for 1,000 encoder-initialized updates, and 492.282 s/image for 10,000 Gaussian-initialized updates, including startup, model loading, image I/O, and output saving.

These results quantify the quality–cost trade-off: encoder initialization permits substantial reconstruction gains at a shorter budget, while iterative refinement remains more expensive than a forward pass. The 32-image and 256-image results are separate evaluations; the new timings should also be distinguished from the original batch-four throughput measurement of 9.50 s/image.

 

AuthorsOpenReview ↗

Response to Reviewer FuNM — Part 2: Perceptual Loss and Generative Modeling

Q2: Inception-Based Perceptual Loss and Its Relationship to rFID

The original refinement experiments in Tables 3–4 combine pixel-space L2 loss with an LPIPS-style perceptual loss computed from the same Inception-v3 feature layer used for FID: the final average-pooling layer with 2,048-dimensional output. The perceptual term compares these features for each reconstruction and its corresponding target image. In contrast, rFID measures the Fréchet distance between the means and covariances of Inception features over the reconstructed and reference image sets. The loss and rFID therefore share the feature representation, but use different comparisons: paired-image feature distances versus distribution-level means and covariances. We explicitly acknowledge this dependence when interpreting the rFID improvement.

Using learned feature distances as optimization objectives has precedent. VIPaint uses a pretrained Inception network to compute a perceptual reconstruction loss for latent diffusion priors [1]. FastFID goes further by enabling direct optimization of FID as a training loss [2]. These precedents support the use of such objectives, while making clear the need to evaluate improvements beyond the shared feature space. The original Table 4 results also improve SSIM and VGG-based LPIPS, neither of which is a term in that Inception-based refinement objective: for FLUX-VAE, SSIM increases from 0.893 to 0.9144 and VGG-LPIPS decreases from 0.051 to 0.0286. These measurements provide complementary evidence of reconstruction improvement.

rFID uses pytorch-fid v0.2.1 with 2,048-dimensional Inception-v3 features. We will state the shared feature layer explicitly and document the checkpoint, normalization, and preprocessing used by the loss and rFID implementations in the revised manuscript.

[1] VIPaint, ICLR 2025 workshop paper, reconstruction-loss implementation and Figure 8.

[2] Alexander Mathiasen and Frederik Hvilshøj. Backpropagating through Fréchet Inception Distance, 2021 revision (FastFID).

 

Q3: Generative Modeling of Refined Representations

Table 5 provides direct downstream evidence under matched generative architectures and training budgets: PSDR-AD improves CIFAR-10 generation FID from 63.66 to 59.10 for autoregressive generation and from 81.96 to 68.02 for diffusion generation. These results support the generative utility of the learned representations in the evaluated setting. The reviewer's question about refined latents from modern pretrained decoders is a further evaluation: the CIFAR-10 results do not establish their relative modeling difficulty. We will make this scope explicit and retain the distinction between the demonstrated pretrained-decoder reconstruction benefits and the downstream generation evidence.

Reviewer_FuNMOpenReview ↗

Follow-up after author response

My main concerns remain as follows.

  1. The main demonstrated benefit is improved reconstruction, but obtaining these representations requires substantial per-image latent optimization. For downstream generative modeling, this would add significant preprocessing time when preparing a large latent training dataset. Offline extraction and caching allow this cost to be paid once per image, but the paper does not establish that the downstream benefits justify it. Another relevant alternative is to increase encoder or decoder capacity, potentially improving reconstruction and representation quality at a lower encoding cost than per-image latent optimization. Comparing against such alternatives would help clarify whether the proposed approach offers a worthwhile quality–cost trade-off.

  2. The effect on representation quality remains unclear. Better reconstruction does not necessarily imply more useful features. For the pretrained FLUX-VAE experiments, the paper does not establish how refinement affects downstream representation-learning performance. ImageNet linear probing or another suitable evaluation could help clarify whether the optimized representations offer benefits beyond reconstruction.

  3. The generative modeling experiments are limited in scale. The CIFAR-10 results do not establish whether the improvements persist on more complex datasets or with latents refined through modern pretrained decoders. The broader benefit for generative modeling therefore remains uncertain.

I am maintaining my weak-reject recommendation.

Reviewer_PqB1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PSDR-AD, an encoder-free auto-decoder framework. Each training image is assigned an independent latent variable. The latent variable is passed through a controlling function ψ\psi, Gaussian noise with a learnable scale σ\sigma is added, and the result is fed into the decoder. The latent variables, the decoder, and σ\sigma are optimized jointly (σ\sigma is regularized toward a target value σ0\sigma_0). At test time, the decoder is fixed, and a latent variable is optimized for each unseen image (for both the continuous and VQ variants). The paper also gives a theoretical justification based on an inclusion relation (Proposition 3.3): for a fixed decoder, the set of latent representations that the encoder can output is a subset of the set of independently optimized latent variables, so optimizing over this larger set does not increase the reconstruction error. When the models are trained from scratch on CIFAR-10, ImageNet-1K, and COCO2017, PSDR-AD outperforms AE, VQ-VAE, and GLO. In addition, the paper shows that optimizing the latent inputs to pretrained decoders, starting from the encoder outputs, improves reconstruction in all evaluated settings, and that the rFID of FLUX-VAE on ImageNet-1K improves substantially.

Critical strengths
  1. The paper shows that current tokenizers still have clear room for improvement in reconstruction, at least in terms of PSNR. I consider this a strength. The decoder, images, and metric implementations are kept the same before and after optimization, and the evaluation covers 18 model-dataset pairs as well as LDM-f4 and FLUX-VAE on ImageNet-1K. These results are informative for tokenizer design.

  2. The noise ablation in Tab. 1 is useful. It suggests that, within the auto-decoder framework, perturbing the latent variables during training contributes to the improvement by itself, in addition to test-time optimization. The explanation in Sec. 5.1, that the perturbation encourages consistent reconstruction within local latent neighborhoods, is not verified in the paper, but I think it is a hypothesis worth testing.

Critical weaknesses
  1. It is not shown whether the improvement can be attributed to removing the encoder (i.e., the auto-decoder formulation). In Tab. 2, AE and VQ-VAE reconstruct images with forward passes only, whereas PSDR-AD performs test-time optimization for each image. As a result, the effects of (i) test-time optimization, (ii) latent noise during training, and (iii) removing the encoder are not separated. In Tab. 1, where test-time optimization is applied to both models, the auto-decoder trained with σ0=0\sigma_0=0 performs worse than AE+Opt., so the effect of (iii) by itself is not confirmed. The paper shows that noise is effective within the auto-decoder framework, but there is no control in which an AE trained with the same noise (σ0=0.2\sigma_0=0.2) receives the same test-time optimization. Therefore, it cannot be judged whether the improvement is specific to the proposed method. Also, the VQ variant uses σ0=0\sigma_0=0, so the contribution of the "soft" relaxation, from which the method takes its name, cannot be evaluated from the results of the VQ variant.

  2. The positioning of the contributions is not consistent with the prior work that the paper itself cites in its related work. The inclusion argument in Proposition 3.3 has already been stated in the GLO paper (Bojanowski et al., 2018), and it is also a deterministic analog of the amortization gap (Cremer et al., 2018). Refining latents starting from the encoder outputs has also been used in semi-amortized inference (Kim et al., 2018), GAN inversion (Zhu et al., 2020), and neural compression (Yang et al., 2020). Although the paper cites three of these works (GLO, Cremer et al., and Yang et al.), it presents Contribution 1 as a “new perspective”. In addition, the FLUX-VAE experiment, which is the main result, optimizes latents from the encoder outputs without noise injection. It therefore does not support the proposed training framework. The novelty I can recognize lies in auto-decoder training with noise regularization and in the systematic measurement of the remaining room for improvement in current tokenizers. The positioning of the paper should be revised accordingly.

  3. The paper does not rule out the possibility that the state-of-the-art reconstruction performance results from optimizing the evaluation metrics themselves. The network used for LpercL_{\text{perc}} in the refinement of Tabs. 3-4 is not specified, and it cannot be excluded that the refinement directly optimizes a network of the same type as those used for the evaluation metrics. In fact, for FLUX-VAE, the improvement in rFID is much larger than the improvements in the other metrics. Two additional results are needed: refinement with ℓ2\ell_2 only, and evaluation with metrics that are not used in the objective.

[Minor comment]

  • The inclusion argument in Prop. 3.3 requires σ=0\sigma=0 to exactly reproduce the encoder outputs, but Definition 3.1 defines the PSDR-Var with σ>0\sigma>0. It would be more appropriate to state the result for σ≥0\sigma\ge 0, or to present it as an observation, since, as the paper itself states, it follows directly from the inclusion of the feasible sets.

[References]

  • Y. Kim, S. Wiseman, A. C. Miller, D. Sontag, and A. M. Rush. Semi-Amortized Variational Autoencoders. In ICML, 2018.
  • J. Zhu, Y. Shen, D. Zhao, and B. Zhou. In-Domain GAN Inversion for Real Image Editing. In ECCV, 2020.
Questions for authors
  1. Does PSDR-AD outperform an AE that is trained with the same latent noise (σ0=0.2\sigma_0=0.2) and receives the same test-time optimization? Also, what are the results of AE+Opt. and VQ-VAE+Opt. in the ImageNet-1K and COCO2017 settings of Tab. 2? If PSDR-AD outperforms all of these baselines, I think the central claim of the paper would be much stronger.

  2. Which network is used to implement LpercL_{\text{perc}} in Tabs. 3-4? Are the improvements in Tabs. 3-4 maintained when the refinement uses only ℓ2\ell_2, or when the results are evaluated with metrics that are not used in the objective?

  3. What are the values of σ\sigma, the norm of the latent variables ∥z∥\|z\|, and the effective signal-to-noise ratio ∥z∥/(σd)\|z\|/(\sigma\sqrt{d}) at the end of training? Do you apply weight decay or any other constraint to the latent variables? I think these values would help confirm that the noise regularization actually works.

3 comments
AuthorsOpenReview ↗

## Response to Reviewer PqB1 — Part 1: Experimental Controls and Contributions

Thank you for recognizing the reconstruction results and noise ablation. This first part addresses W1/Q1 and W2; Part 2 addresses the remaining questions and minor comment.

 

W1 / Q1: AE Noise Ablation and Existing Optimization Controls

Our paper already includes AE+Opt. and VQ-AE+Opt. experiments. Table 1 provides a controlled CIFAR-10 comparison using the same decoder architecture and 5,000 test-time optimization epochs: PSDR-AD achieves 27.72 dB PSNR, 0.886 SSIM, and 0.171 LPIPS, compared with 27.02 dB, 0.870, and 0.184 for AE+Opt. The same table evaluates the training-noise contribution, improving PSNR from 26.13 dB without noise to 27.72 dB with fixed σ = 0.20. Thus, the evidence includes both a test-time optimization control and a noise ablation.

We additionally evaluate noise injection in the AE itself. The CIFAR-10 results below show little variation across the evaluated noise scales. At σ = 0.20, AE achieves 26.2117 dB PSNR, compared with 26.2478 dB without noise; LPIPS also changes from 0.2074 to 0.2089. The best PSNR across the sweep exceeds the zero-noise baseline by only 0.0371 dB. Thus, adding noise to the AE does not produce a comparable reconstruction improvement in this evaluation.

AE noise scalePSNR ↑SSIM ↑LPIPS ↓
0.0026.24780.85220.2074
0.0526.28490.85270.2080
0.1026.27540.85280.2074
0.1526.27080.85360.2069
0.2026.21170.85260.2089
0.2526.25020.85270.2084
0.3026.24030.85220.2092

We further perform latent optimization on the AE trained with σ = 0.20, directly evaluating the requested combination of training noise and test-time optimization. This noise-trained AE+Opt. achieves 26.9725 dB PSNR, 0.8690 SSIM, and 0.1874 LPIPS. PSDR-AD achieves 27.72 dB, 0.8861, and 0.1706, respectively, improving PSNR by approximately 0.75 dB, SSIM by 0.0171, and LPIPS by 0.0168. Thus, adding the same noise scale to AE training and subsequently optimizing its latents does not reproduce the PSDR-AD results in this experiment.

CIFAR-10 methodTraining noise σTest-time latent optimizationPSNR ↑SSIM ↑LPIPS ↓
AE0.20No26.21170.85260.2089
AE+Opt.0.20Yes26.97250.86900.1874
PSDR-AD0.20Yes27.720.88610.1706

Table 3 further evaluates AE and VQ-AE latent optimization on ImageNet-1K and COCO2017, alongside four additional datasets. LDM-AE+Opt. improves PSNR from 25.67 to 27.40 dB on ImageNet and from 25.43 to 27.66 dB on COCO. LDM-VQAE+Opt. improves the corresponding scores from 22.67 to 24.52 dB and from 22.58 to 24.46 dB, with SSIM and LPIPS also improving. These results establish the applicability of latent optimization to both continuous and quantized pretrained representations.

The Table 3 models are pretrained models, rather than the separately trained Table 2 baselines; the latter's ImageNet/COCO AE+Opt. and VQ-VAE+Opt. results are not reported. Our current evidence supports the complete noise-regularized PSDR-AD formulation, rather than attributing its full gain to encoder removal alone. The VQ experiments evaluate support for quantized representations, while Table 1 supplies the evidence for noise regularization. We will clarify these experimental roles and ask the reviewer to consider the existing optimization controls when assessing this concern.

 

W2: Contributions of the Framework and the Pretrained-Decoder Experiments

As detailed in our general response, our contributions comprise the PSDR-Var formulation, the noise-regularized auto-decoder training procedure, and the systematic empirical evaluation of modern pretrained decoders. The connections to GLO, amortization gaps, and latent refinement are discussed in Related Work. Proposition 3.3 expresses the feasible-set motivation for direct representation optimization; the contribution of the paper should be assessed together with its computational formulation and experimental findings.

Direct optimization with a fixed decoder is a capability of the framework, allowing it to be applied to existing autoencoders. The FLUX-VAE experiment evaluates this capability and its practical benefit, while Tables 1–2 evaluate joint latent–decoder training. The pretrained experiments establish the applicability and magnitude of the reconstruction improvements; the training experiments support the proposed learning procedure. We will make this distinction explicit and revise the “new perspective” wording to better reflect the relationship to prior work.

 

AuthorsOpenReview ↗

## Response to Reviewer PqB1 — Part 2: Loss Function, Noise Statistics, and Theory

W3 / Q2: Complementary Losses and Evaluation of Reconstruction Quality

Different reconstruction objectives favor different aspects of image quality. Pixel-space squared error directly favors the MSE criterion underlying PSNR, whereas a perceptual term emphasizes feature-space similarity. Our combined objective evaluates both aspects. An L2-only ablation would help isolate the perceptual term, while evaluating a different quality trade-off. We do not currently report this ablation for Tables 3–4. Our response to the concern about metric dependence is the explicit disclosure of the feature network and the complementary SSIM and VGG-LPIPS results under the original combined objective.

Tables 3–4 report improvements across PSNR, SSIM, and LPIPS, with Table 4 additionally reporting rFID. For FLUX-VAE, PSNR increases from 31.06 to 32.66 dB, SSIM from 0.893 to 0.9144, and LPIPS decreases from 0.051 to 0.0286, alongside the rFID improvement. The reported gains therefore extend across several measures of reconstruction quality. The perceptual term computes an LPIPS-style paired-image loss using the 2,048-dimensional features from the final average-pooling layer of Inception-v3, the same feature layer used for FID; reported LPIPS uses VGG. Thus, the optimization and rFID share the feature representation, while the paired perceptual objective differs from the distribution-level Fréchet statistic. Prior work has even investigated optimizing FID directly (Mathiasen and Hvilshøj, Backpropagating through Fréchet Inception Distance). We will explicitly document this feature overlap and the implementation details; the SSIM and VGG-LPIPS improvements provide complementary evidence beyond rFID.

 

Q3: Fixed Noise Scale, Latent Magnitude, and Effective SNR

The noise scale is fixed in our experiments: σ = 0.20 throughout training in the main continuous setting, so its final value is also 0.20. Table 1 varies this fixed scale across runs; the zero-noise and VQ configurations use σ = 0. The learnable, target-regularized σ in the equations describes an extension of the framework. We acknowledge that the current wording, particularly in Appendix A.4, does not clearly distinguish this extension from the implemented experiments. We will revise the method and experimental descriptions to make this distinction explicit. For the 50,000 CIFAR-10 training latents of shape 3 × 8 × 8 (d = 192), the global coordinate mean is 0.016965 and the population variance is 3.162217. These give a coordinate RMS of √(variance + mean²) = 1.7783, and an RMS per-image latent norm of √E[‖z‖₂²] = 24.6414. With σ = 0.20, the corresponding RMS effective SNR is √E[‖z‖₂²]/(σ√d) = 8.8917. These are aggregate RMS quantities computed from the full latent tensor, rather than the arithmetic mean of per-image norms or SNRs. The injected noise scale is approximately 11.25% of the latent coordinate RMS. Latent parameters are optimized using Adam with only the learning rate specified; all other optimizer settings remain at their defaults, including weight_decay = 0. The continuous configuration uses the identity controlling function and does not impose GLO-style latent norm projection. We will include these statistics and clarify the fixed-noise experimental configuration in the revised manuscript.

 

Minor Comment: The Zero-Noise Case in Proposition 3.3

We agree that the feasible-set argument must explicitly include σ = 0. We will distinguish the noisy case (σ > 0), for which the density in Definition 3.1 applies, from the deterministic case (σ = 0), where the latent distribution consists of point masses. Proposition 3.3 will explicitly allow the deterministic case so that encoder outputs are feasible decoder inputs. The observation concerns the unregularized reconstruction optimum and does not guarantee the same bound for training with a fixed positive noise scale.

 

Reviewer_PqB1OpenReview ↗

Thank you for the response; remaining concerns

Thank you for the detailed response and the additional experiments. The experiment with an AE trained with latent noise and test-time optimization directly answers the first part of Q1, and I found it useful. It shows that an AE trained with the same σ does not reproduce the PSDR-AD results on CIFAR-10 even with test-time optimization. Within this scope, I consider my main concern in W1 largely resolved for the continuous variant.

I also thank you for committing to revise the "new perspective" wording, to make the distinction between the FLUX-VAE experiments and the noise-regularized training explicit, to document the feature overlap between the optimization objective and rFID, to clarify that σ is fixed, and to include the σ = 0 case in Prop. 3.3.

My remaining concerns and suggestions are as follows.

  • Tab. 1 and the new results together show that only the auto-decoder trained with noise outperforms both AE+Opt. variants (with and without training noise) on CIFAR-10. Removing the encoder alone (σ = 0) gives lower performance than AE+Opt. (26.13 vs. 27.02 dB). Therefore, I think the results do not support the claim that removing the encoder during training is, by itself, the source of the improvement. Please revise the statements that attribute the improvement to encoder removal alone and describe it as the effect of encoder removal combined with noise regularization.

  • The FLUX-VAE results come from a known procedure: refining latents initialized from the encoder outputs. The prior work listed in my original review should also be cited. I still consider the systematic measurement a useful contribution. It would be more appropriate to attribute Prop. 3.3 to GLO and to present it as an observation.

  • The rFID of the +Opt. results is measured in the same feature space that the objective matches, so it is not independent evidence. The rFID-based claims in the abstract and Sec. 5.3.2 should be qualified accordingly. Since SSIM and VGG-LPIPS are not directly included in the objective, I think their improvements support the view that the gains are not due only to optimizing the evaluation metrics.

  • The response says that the perceptual loss uses Inception-v3 features, but Sec. 5.3.1 says that the latents are optimized "using the reconstruction and perceptual loss terms of the original autoencoder." All relevant parts should be revised to match the actual setting. Sec. A.4 currently states that σ remains learnable even when σ0 = 0. When revising the description of σ, please update Sec.2, Sec. A.4, Sec. A.5, and the text after Eq.(1) and (4).

I will keep my current rating for now. If the committed revisions and the four points above are reflected in the revised version, most of my concerns about soundness will be resolved. However, the controlled evidence for the novel part is still limited to CIFAR-10 because the ImageNet/COCO comparisons requested in Q1 are not available.

Reviewer_3NF7OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an encoder-free autoencoding framework for latent representation learning using freshly intialized random variable which will get optimized per image sample.

The papre proposes to jointly optimizes per-image latent parameters and a shared decoder (no encoder), with noise injection during training.

At inference, it optimizes the latent representation for each input while keeping the decoder fixed.

The motivation is that a shared encoder’s output may be suboptimal for reconstruction under a given decoder.

Indeed, experiments report improved reconstruction with continuous and quantized latents on CIFAR-10, ImageNet, and COCO.

The most exciting part (yet get least space) is the preliminary generation results on CIFAR-10 which shows improved generation (that's why it's exciting) but setting is too samll and too prelim to read reliably.

The  main empirical contribution is still improved reconstruction which I believe insufficient without showing benefits for generation or compression.

Critical strengths
  1. Promising downstream generation results. Table 5 provides the most ecxciting evidence: the CIFAR-10 experiments report generation improvements with both discrete and continuous representations (although the FID numbe ranges are suboptimal, should be ~20 FID, Yang et al. 2022, GenVIT). This suggests that the optimized representations may benefit a downstream task beyond reconstruction. Athough, the evidence remains very preliminary.

  2. Substantial reconstruction improvements. The experiments report gains for both continuous and quantized latents across CIFAR-10, ImageNet, and COCO. But I want to emphasize that these gains alone is not sufficient see Weaknessese.

Critical weaknesses
  1. The practical value of the learned representations is insufficiently established. This is the main reason for my rejection. Improved reconstruction alone does not establish that a representation is useful for generation or compression (or any other downstream task). Table 5 provides a positive signal, but the generation experiments are limited to CIFAR-10, and the authors acknowledge that fully converged performance has not been established. The reported improvement therefore leaves open whether the benefit persists with stronger, adequately trained baselines and on more challenging datasets. I would need more convincing downstream evidence to support acceptance (see Questions)

  2. The representation budget does not establish a compression advantage. Matching the number of latent scalar elements does not necessarily match the number of bits required to store them, particularly for continuous latents. Although the paper acknowledges this distinction, it provides no rate–distortion evaluation using actual coded bitrates. which couold have been an angle that ther paper could contirbute meanignfully given limited computation for generation tasks. (see Questions)

Questions for authors
  1. Do the generation gains persist with stronger, adequately converged baselines? Table 5 is the most relevant evidence for my assessment. A more competitive range of FID in the order of ~20 (Yang et al., 2022) would defintely help my assessment although, ideally, a reasonable scale of experiment should be at ImageNet. I do recognize the author's computation limitation. Alternatively,... 

  2. Do better recons have better compression? Can the authors demonstrate a rate–distortion advantage in the units of bits per byte? Matching the number of latent scalar elements alone does not establish an equivalent storage budget. Convincing compression evidence could address my concern independently of generation.

Without a strong rebuttal on either (not both) of the questions, I'm unlikely to change my assessment.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3NF7

Thank you for recognizing the substantial reconstruction improvements and the promising downstream generation results. We address the two central concerns below, together with their corresponding questions.

 

W1 / Q1: Existing Downstream Evidence and the Scope of the Generation Claim

As the reviewer recognizes, Table 5 provides positive evidence of downstream generation benefits on CIFAR-10. These improvements are obtained with matched generative architectures and training budgets within each comparison, supporting our claim under the evaluated conditions.

The scale of CIFAR-10 limits the scope of the evidence, but does not invalidate the effectiveness demonstrated on this benchmark. As the reviewer acknowledges, our computational resources are limited: all experiments use a single RTX 4080 SUPER with 16 GB of memory. Extending these controlled generation comparisons to ImageNet would require substantially greater computation. We respectfully believe that making such a scale of validation a prerequisite for recognizing the framework's demonstrated benefits would place a disproportionate burden on a study conducted with these limited academic resources. Stronger, fully converged baselines remain a valuable direction for further validation. We ask the reviewer to reconsider this weakness in light of the existing controlled results, the scope of our claims, and the acknowledged resource constraints.

 

W2 / Q2: Better Reconstruction at the Same Bit Budget

Reducing distortion at the same rate is also an improvement in the rate–distortion trade-off. Our discrete experiments evaluate this direction with matched token counts and codebook sizes. On CIFAR-10, both VQ-VAE and PSDR-AD(VQ) use 64 tokens and a 1,024-entry codebook, requiring 10 bits per token under fixed-length coding. Both therefore have an index payload of 640 bits per image, or 0.625 bpp, excluding the shared model and codebook. At this identical rate, PSDR-AD(VQ) improves PSNR from 26.13 to 26.75 dB, SSIM from 0.8678 to 0.8772, and LPIPS from 0.1860 to 0.1671. Table 2 also shows reconstruction improvements with matched token counts and codebook sizes on ImageNet-1K and COCO2017.

We agree that matching continuous latent scalar counts alone does not establish equivalent coded bitrates, as acknowledged in Section 6. The discrete comparison, however, permits the explicit fixed-length bit accounting above and demonstrates better reconstruction at the same representation rate. Entropy-coded rate–distortion evaluation would provide additional evidence, since token distributions can yield different entropy-coded rates. We therefore ask the reviewer to recognize these equal-bit reconstruction gains when assessing the compression-related value and practical contribution of our work.

&nbsp

For the first part, I do understand the compute limitaion of the authors. But I would not risk accepting a paper at this level of evidence, this is simply not good enough signal. I might be proven wrong, but then there needs to be a person to prove it.

On the compression point, I might have missed the VQ part, thanks for pointing it out. On a second thought, I do think that the compression gain results on Imagenet and COCO do look promising. Since the gains are from the controlled same decoder architecture, they are likely comparable, the difference is just the baseline has an encoder, the proposed one doesn't (and needs to optimize for the image latent; this is a computation-compression trade-off; but that's okay).

Dataset VQ-VAE LPIPS PSDR-AD(VQ) LPIPS Reduction CIFAR-10 0.1860 0.1671 10.2% ImageNet-1K 0.1668 0.0893 46.5% COCO2017 0.1494 0.0814 45.5%

I would raise by score a bit to reflect this interesting finding however I still do think that this paper experimentation is below what I expect from ICLR.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Novelty and Contributions of PSDR-Var and PSDR-AD

We thank the reviewers for recognizing the reconstruction improvements and the potential of the learned representations. Before addressing each reviewer individually, we would like to clarify the relationship between the PSDR-Var formulation, its application to existing autoencoders, and the methodological and practical contributions of our work.

 

1. PSDR-Var Formulation and the PSDR Auto-Decoder

Our methodological contribution is the formulation of Parametric Soft Discrete Random Variables (PSDR-Vars) and their use in an encoder-free representation-learning framework. Each image is associated with independently optimizable latent parameters, transformed by a controlling function and perturbed during training by Gaussian noise with a fixed scale in our experiments. The general formulation additionally allows a learnable, target-regularized noise scale. PSDR-AD jointly learns these representations and a shared decoder. This formulation supports both continuous and vector-quantized representations and provides a common framework for joint latent–decoder training and representation extraction through fixed decoders.

The relationship to earlier latent optimization methods is discussed in Related Work. A shared optimization operation alone does not subsume the formulation, training procedure, and empirical contributions of the complete framework. For example, our continuous identity-map configuration uses additive noise regularization in place of GLO's latent norm projection. Table 1 evaluates the contribution of this training mechanism: a fixed noise scale of 0.20 improves held-out PSNR from 26.13 to 27.72 dB relative to the zero-noise setting and exceeds the AE+Opt. reference of 27.02 dB, with corresponding improvements in SSIM and LPIPS. These results support the effectiveness of noise regularization within the proposed auto-decoder framework.

 

2. Application to Existing Autoencoders Is a Property of the Framework

Direct optimization through a fixed decoder is a natural capability of PSDR-Var. Once the decoder is fixed, its input representation can be optimized independently of an encoder. Consequently, the framework can also use a decoder learned by a conventional autoencoder, with its encoder output providing an initialization. This application makes the method directly usable with existing pretrained models.

The joint-training and pretrained-decoder experiments evaluate complementary aspects of this framework. Tables 1–2 examine the representations and decoders learned through PSDR-AD training. Tables 3–4 examine the improvements obtainable by directly optimizing representations through existing decoders. Noise is used in the continuous joint-training setting and disabled during fixed-decoder inference. The pretrained experiments therefore demonstrate the applicability and practical benefit of direct representation optimization, while the training experiments evaluate the proposed noise-regularized learning procedure.

 

3. Systematic Empirical Evidence and Practical Contribution

Our work also contributes a systematic measurement of the reconstruction capacity that modern pretrained tokenizers leave accessible through latent optimization. Table 3 reports improvements across all 18 model–dataset pairs. Table 4 extends this evaluation to strong pretrained decoders: on ImageNet, FLUX-VAE improves from 31.06 to 32.66 dB PSNR, from 0.893 to 0.9144 SSIM, from 0.051 to 0.0286 LPIPS, and from 0.176 to 0.005 rFID. Within each pair, the decoder parameters and latent dimensionality remain fixed.

These results establish the magnitude and consistency of the reconstruction improvements available through this procedure. Improving existing pretrained models without decoder retraining or additional latent dimensions is itself a practical contribution, with additional inference computation explicitly reported in Appendix A.1. Table 5 further evaluates the learned representations in downstream autoregressive and diffusion generation, showing improvements under the matched training budgets considered there.

We therefore ask that the contribution be assessed through the PSDR-Var formulation, the noise-regularized auto-decoder training procedure, and the systematic empirical and practical advances together. We will make these contributions and their relationship to prior latent optimization methods more explicit in the revised manuscript.

 

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.