acceptodds
Under review as a conference paper at ICLR 2027

Efficient Fréchet Distributional Decoder Alignment for Latent Generative Models

Abstract

Fréchet Distributional (FD) objectives provide an effective distribution-level signal for post-training generative models. For latent generative models, however, existing FD-based post-training methods optimize only the latent generator while keeping the decoder frozen: the decoder is trained on reconstruction latents obtained from real images but deployed on generator-produced latents at generation time. When these latent distributions differ, this decoder train-generation mismatch means that the decoder is deployed on inputs drawn from a distribution it was not directly trained on. We introduce Fréchet Distributional Decoder Alignment (FDDA), an efficient post-training method that freezes the latent generator and directly optimizes the decoder on generation-time latents using an FD objective. A single epoch of FDDA consistently improves gFID and gFDr across diverse flow-based and autoregressive latent generative models. Across all 11 evaluated models, FDDA reduces gFDr by 34.0-59.6% from the original pretrained models and provides a further 10.2-57.2% reduction after generator-side FD post-training. FDDA also reduces total wall-clock time by factors of - relative to generator-side FD post-training across these models. These results show that generator-side post-training alone does not exhaust the benefits of FD optimization, and that further gains can be obtained by adapting the decoder at substantially lower cost.

Primary area:
generative models
Keywords:
latent generative models, decoder alignment, Fréchet distributional objective, generative model post-training, autoregressive image generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gYLROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces a method for post-training of a latent decoder model to improve generation quality for latent generative models of image-type data, called FDDA. The key idea is to adapt recent ideas from Frechet Distribution losses as a training objective for generator model post-training into a method for latent decoder post-training. During FDDA, the latent generator is frozen and the decoder is unfrozen. The EMA version of FD post-training is used, and the FD loss only flows through the EMA FD statistics contributed by the current training batch. A suite of 6 different encoder models are used for FD loss in a balanced way to avoid overfitting to the FD distribution of a single encoder. Experiments show that the proposed method can consistently improve generation quality of a variety of pre-trained image generation models both in the case where the original generator is used and only the decoder is post-trained, and in the case where the generator first undergoes FD post-training followed by FD post-training of the decoder with FDDA.

Critical strengths
  • This method provides a well-scoped and effective method of improving the quality of latent generator models. The FD-loss methodology is naturally suitable for decoder as well as generator post-training, and the FDDA approach is simple and makes intuitive sense.
  • The experimental settings provide convincing evidence that FDDA provides generation benefits across a variety of pre-trained generative models.
Critical weaknesses
  • There is no theoretical guarantee that FD loss will lead to better perceptual generative quality or to a closer match of the data distribution. Overfitting to the FD distribution of the different encoders at the expense of visual quality remains a theoretical possibility, although in practice it looks like FD loss does lead to better perceptual quality. This weakness is shared by other works using FD loss for the generator model, etc.
  • The gains diminish in the more realistic setting where the generator is first FD post-trained, and the method itself is not especially surprising given the recent trend of FD loss for the generator model.
Questions for authors

My main request is for the visualization in Appendix D. In the cases where generator-side FD and FDDA are both used, could you display three rows: pretrained, generator FD only, and generator FD + FDDA? Without the middle row of generator FD only, it is difficult to assess the improvement of the FDDA component after generator FD.

3 comments
AuthorsOpenReview ↗

Response to Reviewer gYLR

We appreciate that the reviewer recognized FDDA as a well-scoped, simple, and intuitive method that naturally extends FD-loss post-training from the generator to the decoder. We also appreciated the recognition that our experiments provide convincing evidence of consistent generation benefits across a variety of pre-trained generative models. Thank you for taking the time to review our paper and for your thoughtful feedback. We address your questions and concerns below.


R 3.1 There is no theoretical guarantee that FD loss will lead to better perceptual generative quality or to a closer match of the data distribution. Overfitting to the FD distribution of the different encoders at the expense of visual quality remains a theoretical possibility, although in practice it looks like FD loss does lead to better perceptual quality. This weakness is shared by other works using FD loss for the generator model, etc.

We agree that FD loss has no such theoretical guarantee, and we note this in Appendix B. To check for overfitting, FDDA trains with only three representations (Inception, MAE, SigLIP2) and holds out DINOv2, CLIP and ConvNeXt for evaluation. The held-out DINOv2 and CLIP FDs decrease in all 22 configurations of Tables 1 and 2.

R 3.2 The gains diminish in the more realistic setting where the generator is first FD post-trained, and the method itself is not especially surprising given the recent trend of FD loss for the generator model.

We agree that the gains are smaller after generator-side FD post-training, but they remain consistent across all 11 configurations (Table 2). Starting from the same generator-side FD-post-trained checkpoint, we continue either generator-side or decoder-side FD post-training for the same additional wall-clock time on a single H100. The decoder-side time includes building the latent pool, which takes 2.68 h for GigaTok-S-S, so its first result is at 3 h.

Table R1. gFDr6\text{gFDr}^6 (↓) under the same additional wall-clock time, starting from generator-side FD-post-trained models. Percentages are relative to 0 h.

ModelScheme0 h1 h2 h3 h4 h5 h
iMF-B/2Generator FD (continued)5.575.565.475.415.485.47 (↓1.8%)
Decoder FD (FDDA)5.575.244.884.614.544.50 (↓19.2%)
iMF-L/2Generator FD (continued)2.722.702.692.682.632.66 (↓2.2%)
Decoder FD (FDDA)2.722.812.902.582.472.41 (↓11.4%)
iMF-XL/2Generator FD (continued)2.462.452.402.382.382.39 (↓2.8%)
Decoder FD (FDDA)2.462.872.712.482.322.21 (↓10.2%)
GigaTok-S-SGenerator FD (continued)5.344.975.105.445.964.98 (↓6.7%)
Decoder FD (FDDA)5.34––4.854.444.01 (↓24.9%)

After 5 h, continued generator-side post-training reduces gFDr6\text{gFDr}^6 by only 1.8-6.7%, while FDDA reduces it by 10.2-24.9%. This comparison is added to Section 5.4 (Table 6) of the revised manuscript.

R 3.3 My main request is for the visualization in Appendix D. In the cases where generator-side FD and FDDA are both used, could you display three rows: pretrained, generator FD only, and generator FD + FDDA? Without the middle row of generator FD only, it is difficult to assess the improvement of the FDDA component after generator FD.

Thank you for the suggestion. In Appendix I of the revised manuscript, we add three-row comparisons on TiTok-L-32 (Figures 18-26): the pretrained model, generator-side FD only, and generator-side FD + FDDA. The pretrained model uses a different generator, so its samples cannot share the same generated latents with the other two rows; all three rows use the same class labels and random seeds. The middle and bottom rows share the same generated latents and differ only in the decoder, so their difference shows the effect of FDDA after generator-side FD post-training.


Please let us know if you have any remaining questions.


Thanks for the response. I will stick with my score. The proposed FDDA method follows naturally (although not especially surprisingly) from generator-side FD learning and appears to be a useful part of the practitioner tool-kit of FD related techniques.

AuthorsOpenReview ↗

Thank you!

We're glad you find FDDA a useful addition to the toolkit of FD-related techniques. Thank you for your feedback and for supporting the paper being presented at ICLR!

Reviewer_CsoVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the mismatch between the latents used to train a decoder and those produced by a generator at inference. It proposes FDDA, which freezes the generator and fine-tunes the decoder on fully generated latents. Training uses multi-representation Fréchet distance losses without paired reconstruction targets. Experiments cover 11 autoregressive and one-step flow model configurations on ImageNet 256×256. One epoch reduces gFDr6 by 34.0%–59.6% on the original models. It provides a further 10.2%–57.2% reduction after generator-side FD post-training. The paper also reports training costs, a latent-source ablation, and changes in reconstruction quality.

Critical strengths
  • Consistent gains across models and after generator-side post-training. Tables 1 and 2 show lower gFDr6 across all evaluated configurations. For example, TiTok-L-32 improves from 2.75 to 1.34 gFID. After generator-side post-training, VAR-d24 improves further from 2.65 to 1.67 gFDr6. Held-out DINOv2 and CLIP metrics also improve. These results support the method’s practical value and complementarity.

  • A simple training scheme with a useful latent-source ablation. FDDA requires no paired targets and adds no inference steps. Appendix C shows that training on reconstruction latents worsens gFDr6 for all three post-trained iMF models. Training on generation-time latents still improves it. This supports the choice to adapt the decoder using its actual generation-time inputs.

Critical weaknesses

The benifits over existing decoder post-training methods is not directly established. RobusTok and GAR/Decoder Adaptation already adapt decoders using generation-related latents. The FD objective, EMA estimator, and multi-representation normalization come from FD-loss. FDDA’s main change is to remove paired reconstruction supervision and train directly on fully generated latents. This is a meaningful difference. However, the paper lacks controlled comparisons with prior decoder methods. Its advantages in quality, cost, and reconstruction trade-offs therefore remain unclear.

Questions for authors
  • How does FDDA compare with prior decoder post-training methods under matched checkpoints, sampling settings, and compute budgets? Comparisons on representative autoregressive and flow models would be useful.
  • Why does convNext FD worsen for most configurations in Table 2?
  • The largest reported cost ratio compares one decoder-training epoch with 100 generator-training epochs. How does generator-side post-training perform under the same total time budget?
10 comments
AuthorsOpenReview ↗

Response to Reviewer CsoV (1/4)

We appreciate that the reviewer recognized the consistent gains of FDDA across all evaluated configurations, including further improvements after generator-side FD post-training, which support its practical value and complementarity. We also appreciated the recognition of our simple training scheme without paired targets or extra inference steps, and of the latent-source ablation supporting the use of generation-time latents. Thank you for taking the time to review our paper and for your thoughtful feedback. We address your questions and concerns below.


R 2.1 The benifits over existing decoder post-training methods is not directly established. RobusTok and GAR/Decoder Adaptation already adapt decoders using generation-related latents. The FD objective, EMA estimator, and multi-representation normalization come from FD-loss. FDDA’s main change is to remove paired reconstruction supervision and train directly on fully generated latents. This is a meaningful difference. However, the paper lacks controlled comparisons with prior decoder methods. Its advantages in quality, cost, and reconstruction trade-offs therefore remain unclear.

Thank you for this comment. We add controlled comparisons with GAR and RobusTok in terms of generation quality, cost and reconstruction, under the same generator, initial decoder and budget (one epoch). For each method, we test both its original objective and the FD loss on its latents, which separates the effect of the training latents from that of the objective. Costs are total wall-clock time on a single H100. These comparisons are added to Section 5.3 (Table 5) and Appendices D-E (Tables 9-13) of the revised manuscript.

(i) Comparison with GAR. We use GAR's default setting (GAN loss with paired supervision, noise level sampled uniformly from [0.25, 0.45]) on iMF-B/2, iMF-L/2 and iMF-XL/2, before and after generator-side FD post-training.

Table R1a. Comparison with GAR, before generator-side FD post-training. Percentages are relative to "none". Best result among decoder post-training variants in bold.

ModelDecoder post-trainingCost (h)gFID (↓)gIS (↑)gFDconv\text{gFD}_\text{conv} (↓)gFDdino\text{gFD}_\text{dino} (↓)gFDmae\text{gFD}_\text{mae} (↓)gFDsig\text{gFD}_\text{sig} (↓)gFDclip\text{gFD}_\text{clip} (↓)gFDr6\text{gFDr}^6 (↓)
iMF-B/2none–3.49253.7188.08206.170.48719.65156.1415.40
GAR latents + GAR objective9.003.01261.6121.11180.050.42816.08121.5312.58 (↓18.3%)
GAR latents + FD loss4.383.01266.3187.80154.090.37612.46109.3810.90 (↓29.2%)
FDDA3.891.65268.1115.47144.390.2328.10105.068.54 (↓44.5%)
iMF-L/2none–1.88275.4105.53111.560.28211.30101.189.11
GAR latents + GAR objective9.901.70273.273.79104.050.2669.3985.047.83 (↓14.1%)
GAR latents + FD loss4.761.66284.3133.1489.270.1886.2773.336.30 (↓30.8%)
FDDA4.271.26281.075.9383.770.1375.1272.125.48 (↓39.8%)
iMF-XL/2none–1.81281.997.57103.010.26110.3794.588.44
GAR latents + GAR objective10.461.61279.267.6099.950.2568.6479.307.33 (↓13.2%)
GAR latents + FD loss5.021.56290.3126.1781.420.1735.7469.045.84 (↓30.8%)
FDDA4.531.14288.367.1277.300.1174.6767.745.03 (↓40.4%)
AuthorsOpenReview ↗

Response to Reviewer CsoV (2/4)

Table R1b. Comparison with GAR, after generator-side FD post-training. Same conventions as Table R1a.

ModelDecoder post-trainingCost (h)gFID (↓)gIS (↑)gFDconv\text{gFD}_\text{conv} (↓)gFDdino\text{gFD}_\text{dino} (↓)gFDmae\text{gFD}_\text{mae} (↓)gFDsig\text{gFD}_\text{sig} (↓)gFDclip\text{gFD}_\text{clip} (↓)gFDr6\text{gFDr}^6 (↓)
iMF-B/2none–0.86307.165.9789.050.1545.4670.355.57
GAR latents + GAR objective9.061.19301.262.5894.690.2256.9172.416.40 (↑14.9%)
GAR latents + FD loss4.391.13304.194.7684.740.1745.0963.885.41 (↓2.9%)
FDDA3.890.91304.074.6680.480.1003.9159.694.54 (↓18.5%)
iMF-L/2none–0.77303.443.6038.890.0502.4638.852.72
GAR latents + GAR objective9.981.06294.244.2450.500.1253.6244.613.68 (↑35.3%)
GAR latents + FD loss4.790.94300.153.9038.950.0902.5636.752.89 (↑6.3%)
FDDA4.270.80299.165.1336.920.0352.0533.722.43 (↓10.7%)
iMF-XL/2none–0.76306.740.3433.850.0432.2135.742.46
GAR latents + GAR objective10.541.08297.440.9246.770.1223.4738.493.38 (↑37.4%)
GAR latents + FD loss5.050.92305.391.6335.360.0863.0735.023.02 (↑22.8%)
FDDA4.530.78303.757.5332.510.0301.8231.612.21 (↓10.2%)

Before generator-side FD post-training, FDDA reduces gFDr6\text{gFDr}^6 by 39.8-44.5%, compared with 13.2-18.3% for GAR's original objective and 29.2-30.8% for the FD loss on GAR's latents. After generator-side FD post-training, GAR's original objective increases gFDr6\text{gFDr}^6 on all three models, and the FD loss on GAR's latents improves it only on iMF-B/2, while FDDA still reduces it by 10.2-18.5%. FDDA is also the cheapest variant: GAR's original objective takes about 2.3× as long, and the FD loss on GAR's latents takes 11-13% longer.

(ii) Comparison with RobusTok. We compare with RobusTok on GigaTok-S-S before generator-side FD post-training, using a preservation ratio of 0.8, with both its original objective and the FD loss on its latents.

Table R2. Comparison with RobusTok on GigaTok-S-S, before generator-side FD post-training. Percentages are relative to "none".

Decoder post-trainingCost (h)gFID (↓)gIS (↑)gFDconv\text{gFD}_\text{conv} (↓)gFDdino\text{gFD}_\text{dino} (↓)gFDmae\text{gFD}_\text{mae} (↓)gFDsig\text{gFD}_\text{sig} (↓)gFDclip\text{gFD}_\text{clip} (↓)gFDr6\text{gFDr}^6 (↓)
none–4.34268.5135.95182.660.43215.25112.2612.29
RobusTok latents + RobusTok objective8.874.58278.5141.19163.500.41814.68101.3811.56 (↓5.9%)
RobusTok latents + FD loss7.534.85304.0134.05125.180.2968.4977.978.22 (↓33.1%)
FDDA5.711.68281.699.96123.430.1696.3274.756.59 (↓46.4%)

FDDA reduces gFDr6\text{gFDr}^6 by 46.4% and gFID from 4.34 to 1.68, while both RobusTok variants increase gFID. FDDA is also faster: RobusTok's original objective takes 1.55× as long, and the FD loss on its latents 1.32×.

AuthorsOpenReview ↗

Response to Reviewer CsoV (3/4)

(iii) Reconstruction trade-off. Table 5 of the original submission, now Table 12 in Appendix E of the revised manuscript, reports reconstruction before and after FDDA. Table R3 adds the decoders adapted by GAR and RobusTok, which is included as Table 13. All these methods train the decoder on generated or generation-related latents rather than on reconstruction latents zrz_r, so reconstruction from zrz_r is expected to drop, and PSNR decreases for every adapted decoder. In practice, this trade-off can be avoided. FDDA does not change the encoder, so the original decoder can be kept for reconstruction and the adapted decoder used for generation.

Table R3. Reconstruction metrics of the adapted decoders

ModelDecoder post-trainingPSNR (↑)SSIM (↑)rFID (↓)LPIPS (↓)
iMF (B/L/XL)none27.6172.50.750.068
iMF-B/2GAR latents + GAR objective26.9170.60.830.075
GAR latents + FD loss26.2769.00.570.075
FDDA26.6569.70.920.081
iMF-L/2GAR latents + GAR objective26.9470.80.810.074
GAR latents + FD loss26.5469.40.530.071
FDDA27.0671.00.860.076
iMF-XL/2GAR latents + GAR objective27.0470.90.860.073
GAR latents + FD loss26.4569.70.550.072
FDDA27.1571.10.890.076
GigaTok-S-Snone22.4754.51.010.132
RobusTok latents + RobusTok objective22.2754.81.220.139
RobusTok latents + FD loss21.6852.21.090.154
FDDA21.2250.81.420.163

R 2.2 How does FDDA compare with prior decoder post-training methods under matched checkpoints, sampling settings, and compute budgets? Comparisons on representative autoregressive and flow models would be useful.

Tables R1a, R1b and R2 in R 2.1 provide this comparison. For each model, all variants start from the same generator checkpoint and the same pretrained decoder, use the same sampling settings for training latents and evaluation, and are trained for one epoch. We compare with GAR on a flow model (iMF-B/2, iMF-L/2, iMF-XL/2) and with RobusTok on an AR model (GigaTok-S-S). FDDA gives the lowest gFDr6\text{gFDr}^6 in all settings and also the lowest wall-clock cost: GAR's original objective takes about 2.3× as long, and RobusTok's original objective 1.55×. This comparison is summarized in Section 5.3 (Table 5) of the revised manuscript.

R 2.3 Why does ConvNext FD worsen for most configurations in Table 2?

In Tables 1 and 2, the held-out gFDdino\text{gFD} _\text{dino} and gFDclip\text{gFD} _\text{clip} decrease in all 22 configurations, but gFDconv\text{gFD} _\text{conv} does not. We think the feature spaces are not fully consistent, so lower FD in the training spaces does not imply lower ConvNeXt FD. To test this, we train the decoder of iMF-B/2 with ConvNeXt as the only FD feature extractor, before and after generator-side FD post-training.

Table R5. FDDA on iMF-B/2 with different FD feature extractors (one epoch). Best result among decoder post-training variants in bold, within each generator setting.

Generator FDDecoder FDFD feature extractorsgFID (↓)gIS (↑)gFDconv\text{gFD}_\text{conv} (↓)gFDdino\text{gFD}_\text{dino} (↓)gFDmae\text{gFD}_\text{mae} (↓)gFDsig\text{gFD}_\text{sig} (↓)gFDclip\text{gFD}_\text{clip} (↓)gFDr6\text{gFDr}^6 (↓)
✗✗–3.49253.7188.08206.170.48719.65156.1415.40
✗✓Inception + MAE + SigLIP21.65268.1115.47144.390.2328.10105.068.54
✗✓ConvNeXt only4.07243.767.51153.911.18724.98234.6821.10
✓✗–0.86307.165.9789.050.1545.4670.355.57
✓✓Inception + MAE + SigLIP20.91304.074.6680.480.1003.9159.694.54
✓✓ConvNeXt only1.72283.558.78102.030.42311.34127.8310.22

Before generator-side FD post-training, training with ConvNeXt alone lowers gFDconv\text{gFD} _\text{conv} from 188.08 to 67.51, but makes gFID and the MAE, SigLIP2 and CLIP FDs worse than those of the pretrained model, and raises gFDr6\text{gFDr}^6 from 15.40 to 21.10. After generator-side FD post-training, it lowers gFDconv\text{gFD} _\text{conv} from 65.97 to 58.78, but makes gFID, gIS and all the other FDs worse than the baseline, and raises gFDr6\text{gFDr}^6 from 5.57 to 10.22. Reducing ConvNeXt FD therefore conflicts with reducing FD in the other spaces. These results are added to Appendix G (Table 15).

Reviewer_xE5YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FDDA: it freezes the latent generator and updates the decoder on sampled and cached generation-time latents using an existing multi-representation Fréchet Distance objective. The authors demonstrate metric improvements from both original checkpoints and checkpoints that have already undergone generator-side FD post-training, and report lower post-training costs. Its main value is applying the existing FD post-training paradigm to the decoder and demonstrating its effectiveness.

Critical strengths
  1. The method is simple and practically useful. It updates only the existing decoder, requires neither backpropagation through generator sampling nor paired per-example supervision, and adds no inference modules or sampling steps.

  2. Improvements in the main metrics cover a useful range of configurations. Both gFID and gFDr6 improve from the original checkpoints across five model families and 11 configurations; after generator-side FD post-training, gFDr6 improves further in every configuration. The input-distribution ablation in Table 6 also supports a practical role for using generated latents.

Critical weaknesses
  1. The incremental benefit over direct prior work and the efficiency advantage are not sufficiently established. GAR and RobusTok already study decoder adaptation to generation distributions, and the FD objective is inherited from prior work. The ablations do not include GAR’s intermediate latents or original objective. Comparing generator-side and decoder-side schemes with different budgets also does not establish acceleration at matched quality or superiority at the same additional budget. The AR comparison additionally changes loss normalisation and the backpropagation path, so the gains cannot be attributed entirely to the updated module.

  2. Stability and the interpretation of quality gains need stronger evidence. Improvements across configurations do not replace independent training repetitions. gFDr6 includes representations used for optimisation, while Table 2 shows numerical degradation in ConvNeXt FD for 9/11 configurations and gFID for 4/11. Improvements in held-out DINO and CLIP representations are positive evidence but do not fully explain this divergence. Actual validation is also limited to ImageNet 256×256, with continuous generation tested only on one-step iMF; broader effectiveness remains unclear.

Overall assessment: I recognise the practical value and improvements in the main metrics, and lean toward weak acceptance. The technical increment is limited, and stronger direct comparisons, budget analysis, and independent quality validation are needed to establish the advantages over direct prior work and the stability of the gains more convincingly.

Questions for authors
  1. With the same generator, initial decoder, and a clearly specified budget, could you compare GAR’s original objective, FD on GAR’s intermediate latents, and FD on fully generated latents to isolate the effects of the added design choices?

  2. Could you report quality versus total training time for representative configurations, and compare FDDA against continuing generator training with the same additional budget? Can the AR objective aggregation and timing scope be matched?

  3. Could independent training repetitions for representative AR and iMF configurations, together with independent perceptual quality, class-consistency, or diversity evaluation, clarify small gains and disagreement among held-out metrics?

  4. Please specify baseline checkpoints, sampling settings, and metric implementations; explain differences from the original LlamaGen, GigaTok, and REG reports and the GigaTok reconstruction metrics; and clarify whether evaluation samples are independent of the training latent pool. These differences are currently questions requiring clarification, rather than established errors.

6 comments
AuthorsOpenReview ↗

Response to Reviewer xE5Y(1/6)

We appreciate that the reviewer recognized the simplicity and practicality of FDDA, which updates only the decoder without backpropagation through generator sampling or additional inference cost, as well as its consistent improvements across diverse configurations, including further gains on top of generator-side FD post-training. Thank you for taking the time to review our paper and for your thoughtful feedback. We address your questions and concerns below.


R 1.1 The incremental benefit over direct prior work and the efficiency advantage are not sufficiently established. GAR and RobusTok already study decoder adaptation to generation distributions, and the FD objective is inherited from prior work. The ablations do not include GAR’s intermediate latents or original objective. Comparing generator-side and decoder-side schemes with different budgets also does not establish acceleration at matched quality or superiority at the same additional budget. The AR comparison additionally changes loss normalisation and the backpropagation path, so the gains cannot be attributed entirely to the updated module.

Thank you for this comment. Below we (i) clarify how FDDA differs from GAR and RobusTok, (ii) isolate the effect of GAR's intermediate latents, (iii) compare against GAR's original objective, (iv) compare generator-side and decoder-side post-training under the same additional wall-clock budget, and (v) discuss the loss normalisation and backpropagation path in the AR comparison.

(i) Relation to GAR and RobusTok. As already discussed in Section 2 and Section 4.1, GAR and RobusTok rely on paired supervision, which restricts decoder adaptation to intermediate latents that still correspond to source images. FDDA uses a distribution-level objective and can therefore train the decoder on fully generated latents. The experiments below examine whether this difference accounts for the gains.

(ii) GAR's intermediate latents. On iMF-B/2 before generator-side FD post-training, we apply the FD loss without paired supervision to GAR's intermediate latents with noise levels from 0.1 to 1.0. A noise level of 1.0 corresponds to fully generated latents (FDDA).

Table R1. Noise-level sweep on iMF-B/2, before generator-side FD post-training (FD loss, one epoch).

Noise levelgFID (↓)gIS (↑)gFDconv\text{gFD}_\text{conv} (↓)gFDdino\text{gFD}_\text{dino} (↓)gFDmae\text{gFD}_\text{mae} (↓)gFDsig\text{gFD}_\text{sig} (↓)gFDclip\text{gFD}_\text{clip} (↓)gFDr6\text{gFDr}^6 (↓)
none3.49253.7188.08206.170.48719.65156.1415.40
0.13.15265.6187.57171.580.38115.92138.4212.97
0.23.07268.9199.57160.870.33713.45120.5511.48
0.33.03268.3163.36155.310.35912.84125.9311.38
0.43.04269.2170.03151.880.32611.53108.0210.33
0.52.44261.4130.42152.600.32010.32112.469.94
0.62.50240.2147.37154.750.2579.37109.239.41
0.72.32242.6140.11156.110.2498.97110.189.28
0.81.83256.7125.38148.420.2438.52102.658.72
0.91.78263.9127.82149.650.2478.40104.168.77
1.0 (FDDA)1.65268.1115.47144.390.2328.10105.068.54

gFDr6\text{gFDr}^6 decreases almost monotonically from 12.97 at 0.1 to 8.54 at 1.0, and 1.0 gives the best result on six of the eight metrics. Higher noise levels bring the training latents closer to generation-time latents. GAR is limited to lower noise levels because paired supervision requires correspondence with the source image; FDDA has no such constraint.

We then compare GAR's default noise range [0.25, 0.45] with fully generated latents on iMF-B/2, iMF-L/2 and iMF-XL/2, before and after generator-side FD post-training, using the same FD loss ("GAR latents + FD loss" vs. "FDDA" in Tables R2a and R2b).

AuthorsOpenReview ↗

Response to Reviewer xE5Y(2/6)

(iii) GAR's original objective. We also run GAR with its default objective (GAN loss with paired supervision, noise level sampled uniformly from [0.25, 0.45]). All variants are trained for one epoch.

Table R2a. Before generator-side FD post-training. Percentages are relative to "none".

ModelDecoder post-traininggFID (↓)gIS (↑)gFDconv\text{gFD}_\text{conv} (↓)gFDdino\text{gFD}_\text{dino} (↓)gFDmae\text{gFD}_\text{mae} (↓)gFDsig\text{gFD}_\text{sig} (↓)gFDclip\text{gFD}_\text{clip} (↓)gFDr6\text{gFDr}^6 (↓)
iMF-B/2none3.49253.7188.08206.170.48719.65156.1415.40
GAR latents + GAR objective3.01261.6121.11180.050.42816.08121.5312.58 (↓18.3%)
GAR latents + FD loss3.01266.3187.80154.090.37612.46109.3810.90 (↓29.2%)
FDDA1.65268.1115.47144.390.2328.10105.068.54 (↓44.5%)
iMF-L/2none1.88275.4105.53111.560.28211.30101.189.11
GAR latents + GAR objective1.70273.273.79104.050.2669.3985.047.83 (↓14.1%)
GAR latents + FD loss1.66284.3133.1489.270.1886.2773.336.30 (↓30.8%)
FDDA1.26281.075.9383.770.1375.1272.125.48 (↓39.8%)
iMF-XL/2none1.81281.997.57103.010.26110.3794.588.44
GAR latents + GAR objective1.61279.267.6099.950.2568.6479.307.33 (↓13.2%)
GAR latents + FD loss1.56290.3126.1781.420.1735.7469.045.84 (↓30.8%)
FDDA1.14288.367.1277.300.1174.6767.745.03 (↓40.4%)

Table R2b. After generator-side FD post-training. Percentages are relative to "none".

ModelDecoder post-traininggFID (↓)gIS (↑)gFDconv\text{gFD}_\text{conv} (↓)gFDdino\text{gFD}_\text{dino} (↓)gFDmae\text{gFD}_\text{mae} (↓)gFDsig\text{gFD}_\text{sig} (↓)gFDclip\text{gFD}_\text{clip} (↓)gFDr6\text{gFDr}^6 (↓)
iMF-B/2none0.86307.165.9789.050.1545.4670.355.57
GAR latents + GAR objective1.19301.262.5894.690.2256.9172.416.40 (↑14.9%)
GAR latents + FD loss1.13304.194.7684.740.1745.0963.885.41 (↓2.9%)
FDDA0.91304.074.6680.480.1003.9159.694.54 (↓18.5%)
iMF-L/2none0.77303.443.6038.890.0502.4638.852.72
GAR latents + GAR objective1.06294.244.2450.500.1253.6244.613.68 (↑35.3%)
GAR latents + FD loss0.94300.153.9038.950.0902.5636.752.89 (↑6.3%)
FDDA0.80299.165.1336.920.0352.0533.722.43 (↓10.7%)
iMF-XL/2none0.76306.740.3433.850.0432.2135.742.46
GAR latents + GAR objective1.08297.440.9246.770.1223.4738.493.38 (↑37.4%)
GAR latents + FD loss0.92305.391.6335.360.0863.0735.023.02 (↑22.8%)
FDDA0.78303.757.5332.510.0301.8231.612.21 (↓10.2%)

Before generator-side FD post-training, all three variants reduce gFDr6\text{gFDr}^6, with FDDA giving the largest reduction (39.8-44.5%, vs. 29.2-30.8% for GAR latents + FD loss and 13.2-18.3% for the GAR objective). After generator-side FD post-training, GAR latents + FD loss improve gFDr6\text{gFDr}^6 only on iMF-B/2 (2.9%) and degrade it on iMF-L/2 and iMF-XL/2, and the GAR objective degrades it on all three models (14.9-37.4%). FDDA still reduces gFDr6\text{gFDr}^6 by 10.2-18.5%. These comparisons are added to Section 5.3 (Table 5), with the noise-level sweep in Appendix C (Table 8) and all metrics in Appendix D (Tables 9-10) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer xE5Y(3/6)

(iv) Same additional budget. Starting from the same generator-side FD-post-trained checkpoint, we continue either generator-side or decoder-side FD post-training for the same additional wall-clock time on a single H100. The decoder-side time includes latent-pool construction, which takes 2.68 h for GigaTok-S-S, so no result is available before 3 h.

Table R3. gFDr6\text{gFDr}^6 under matched additional wall-clock time.

ModelScheme0 h1 h2 h3 h4 h5 h
iMF-B/2Generator FD (continued)5.575.565.475.415.485.47 (↓1.8%)
Decoder FD (FDDA)5.575.244.884.614.544.50 (↓19.2%)
iMF-L/2Generator FD (continued)2.722.702.692.682.632.66 (↓2.2%)
Decoder FD (FDDA)2.722.812.902.582.472.41 (↓11.4%)
iMF-XL/2Generator FD (continued)2.462.452.402.382.382.39 (↓2.8%)
Decoder FD (FDDA)2.462.872.712.482.322.21 (↓10.2%)
GigaTok-S-SGenerator FD (continued)5.344.975.105.445.964.98 (↓6.7%)
Decoder FD (FDDA)5.34––4.854.444.01 (↓24.9%)

After 5 h, continued generator-side post-training reduces gFDr6\text{gFDr}^6 by 1.8-6.7%, while decoder-side post-training reduces it by 10.2-24.9%. FDDA surpasses the best gFDr6\text{gFDr}^6 reached by continued generator-side post-training within 5 h after 1 h on iMF-B/2, 3 h on iMF-L/2 and GigaTok-S-S, and 4 h on iMF-XL/2. This comparison is added to Section 5.4 (Table 6) of the revised manuscript.

(v) Loss normalisation and backpropagation path. Both schemes use the same FD objective and loss normalisation. The backpropagation path differs only because a different module is updated: generator-side FD post-training must backpropagate into the AR generator via rollouts and straight-through estimation over discrete tokens, whereas FDDA updates only the decoder and needs neither. This is inherent to decoder-side post-training, not an additional design change, and partly explains FDDA's lower cost.

R 1.2 Stability and the interpretation of quality gains need stronger evidence. Improvements across configurations do not replace independent training repetitions. gFDr6 includes representations used for optimisation, while Table 2 shows numerical degradation in ConvNeXt FD for 9/11 configurations and gFID for 4/11. Improvements in held-out DINO and CLIP representations are positive evidence but do not fully explain this divergence. Actual validation is also limited to ImageNet 256×256, with continuous generation tested only on one-step iMF; broader effectiveness remains unclear.

Thank you for this comment. Below we (i) report independent training repetitions, (ii) discuss the gFID changes, (iii) analyze the disagreement between ConvNeXt FD and the other representations, and (iv) extend the evaluation to a multi-step latent diffusion model.

(i) Independent repetitions. We run FDDA three times independently on the four configurations in Table 2 where gFID is worse than the baseline (LlamaGen-L, iMF-B/2, iMF-L/2, iMF-XL/2), all after generator-side FD post-training.

Table R4. Three independent FDDA runs after generator-side FD post-training (mean ± std). The row without decoder FD is the generator-side FD-post-trained baseline.

ModelDecoder FDgFID (↓)gIS (↑)gFDconv\text{gFD}_\text{conv} (↓)gFDdino\text{gFD}_\text{dino} (↓)gFDmae\text{gFD}_\text{mae} (↓)gFDsig\text{gFD}_\text{sig} (↓)gFDclip\text{gFD}_\text{clip} (↓)gFDr6\text{gFDr}^6 (↓)
LlamaGen-L✗1.32315.974.5457.090.1043.1854.273.96
✓1.32±0.02313.2±0.687.13±2.8648.75±0.450.053±0.0032.39±0.0145.32±0.563.20±0.02 (↓19.2%)
iMF-B/2✗0.86307.165.9789.050.1545.4670.355.57
✓0.92±0.01305.0±0.774.62±2.3579.76±1.050.102±0.0023.94±0.0259.13±0.694.53±0.03 (↓18.7%)
iMF-L/2✗0.77303.443.6038.890.0502.4638.852.72
✓0.81±0.01300.8±2.371.80±9.7236.73±0.430.035±0.0022.05±0.0434.44±0.772.47±0.04 (↓9.2%)
iMF-XL/2✗0.76306.740.3433.850.0432.2135.742.46
✓0.79±0.01302.7±1.366.32±6.8232.34±0.400.029±0.0021.84±0.0231.81±0.732.24±0.04 (↓8.9%)

The standard deviation of gFDr6\text{gFDr}^6 is at most 0.04, while its reduction over the baseline ranges from 0.22 to 1.04. These results are added to Appendix F (Table 14).

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Responses to All Reviewers

Thank you to all the reviewers for their valuable feedback. We sincerely appreciate your constructive comments. We are grateful to reviewers xE5Y and CsoV for highlighting the simplicity and practical value of FDDA, which updates only the decoder without paired supervision or additional inference cost, as well as its consistent gains across model families and its complementarity to generator-side FD post-training. We also thank reviewer gYLR for recognizing FDDA as a well-scoped and effective method that naturally extends FD-loss post-training from the generator to the decoder. Finally, we appreciate all reviewers for noting the empirical validation and overall clarity of our work.

In response to the reviewers' feedback, we have revised the manuscript, with all changes highlighted in blue. The main additions are summarized as follows:

  • Controlled Comparisons with Prior Decoder Post-Training Methods: In Section 5.3 (Table 5), we compare FDDA with GAR on iMF and with RobusTok on GigaTok-S-S, using the same generator, initial decoder, and number of training epochs. For each method, we evaluate both its original objective and the FD loss on its latents, which separates the effect of the training latents from that of the objective. Appendix C (Table 8) further sweeps the noise level of GAR's intermediate latents, Appendix D (Tables 9-11) reports all metrics, and Appendix E (Tables 12-13) reports reconstruction results.
  • Comparison under the Same Wall-Clock Time: In Section 5.4 (Table 6), we compare generator-side and decoder-side FD post-training under the same wall-clock time, starting from both the pretrained and the generator-side FD-post-trained checkpoints.
  • Training Stability and Held-Out Metrics: In Appendix F (Table 14), we report three independent FDDA runs on LlamaGen-L, iMF-B, iMF-L, and iMF-XL. In Appendix G (Table 15), we train FDDA with ConvNeXt as the only FD feature extractor to analyze the disagreement between ConvNeXt FD and the other representations.
  • Multi-Step Latent Diffusion Model: In Appendix H (Table 16), we apply FDDA to RAE, a multi-step latent diffusion model.
  • Qualitative Comparisons: In Appendix I (Figures 18–26), we add three-row comparisons showing the pretrained model, generator-side FD only, and generator-side FD + FDDA.

Once again, we sincerely thank the reviewers for their thoughtful comments. We have carefully addressed each point in our individual responses to clarify the motivation, effectiveness, and generality of our proposed method.

AuthorsOpenReview ↗

Thank you!

We're glad our rebuttal addressed your questions and concerns, and, following your new two suggestions, we have revised Section 4.2 and the abstract accordingly. Thank you for your feedback and for supporting the paper being presented at ICLR!

AuthorsOpenReview ↗

Discussion Summary for the Area Chair

Dear Area Chair,

We thank the reviewers for a constructive discussion. All three reviewers recommend acceptance, with final ratings of 3/3/3: CsoV raised the score (2→3), and xE5Y and gYLR maintained weak acceptance. gYLR described FDDA as "a useful part of the practitioner tool-kit" of FD techniques.

Our paper shows that generator-side FD post-training does not exhaust the gains of distribution-level optimization. The decoder is trained on reconstruction latents but deployed on generated latents. FDDA freezes the generator and optimizes the decoder directly on fully generated latents with an FD objective, with no paired targets and no extra inference cost. One epoch reduces gFDr$^6$ by 34.0–59.6% across 11 models, and by a further 10.2–57.2% after generator-side FD post-training.

The rebuttal resolved the main concerns with new experiments:

  • Benefit over prior decoder post-training (xE5Y, CsoV). With the same generator, initial decoder, and epochs, FDDA reduces gFDr$^6$ by 40–46%, versus 13–18% for GAR and 6% for RobusTok, at lower cost. After generator-side FD post-training, only FDDA still improves all iMF models. Both reviewers said this clarifies what FDDA adds.
  • Efficiency (xE5Y, CsoV, gYLR). With the same additional wall-clock time after generator-side FD post-training, FDDA reduces gFDr$^6$ by 10–25%, while continued generator training gains only 2–7%.
  • Stability and generality (xE5Y, CsoV). Three independent runs give a gFDr$^6$ standard deviation of at most 0.04. FDDA also improves multi-step RAE and four text-to-image models on COCO, while preserving CLIP score.

We have incorporated both of xE5Y’s minor revision requests into the revised manuscript. On novelty (gYLR), the controlled comparisons show that the gains come from training on fully generated latents, which paired decoder adaptation methods cannot use.

We believe FDDA is a simple, low-cost, and well-validated complement to generator-side post-training.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.