acceptodds
Withdrawn from ICLR 2027

FuseReg: Regularizing Layer Fusion Closes the Reconstruction-Generation Gap in Representation Autoencoders

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Representation autoencoders (RAEs) reuse features from a pretrained visual encoder as reconstruction and diffusion latents, integrating strong visual representations into image generation. However, RAEs still need to decide which encoder layers form the shared latent space for the generator and pixel decoder. This choice involves a trade-off. Shallower layers tend to preserve fine pixel details better, while deeper layers tend to yield better generation metrics. A fixed heuristic layer fusion therefore couples two stages that benefit from different information. We introduce FuseReg, which replaces heuristic feature selection with training over random subsets of encoder layers. We theoretically analyze the underlying mechanism: subset sampling explicitly penalizes sensitivity to cross-layer disagreement. On ImageNet-256 with DINOv3-L, a single FuseReg decoder reconstructs from full, sparse, and single-layer fusions without retraining, achieving higher PSNR than decoders specialized to fixed fusions. This flexibility also benefits generation: decoder replacement alone reduces unguided gFID by 27% with an unchanged RAEv2 DiT-XL generator. Applying FuseReg to both stages also reduces unguided gFID by 29% on DiT-Base. The reconstruction and generation benefits also extend to other encoder families. FuseReg narrows the reconstruction-generation gap without additional training cost or architectural changes.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Autoencoder, Diffusion Model, Feature Fusion, Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ckXFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper replaces a fixed fusion during training (basically the RAEv2 method) with the normalized mean of a randomly sampled subset of layers. The decoder reconstructs images from randomized fusions. During training, the diffusion models predict full sets of selected layer representations. The results show that the decoder trained with randomized subsets is robust to be directly replaced with RAEv2. The gFID under certain configuration is improved.

Critical strengths
  1. Extensive ablations that separate different effects. The paper evaluates decoder and generator regularization rates separately and jointly, compares DiT-Base and DiT-XL, studies guided and unguided generation, and includes additional encoder families.

  2. Explicit mathematical analysis of the proposed perturbation. The mean-preservation and covariance results characterize what random subset fusion changes, and the linear squared-loss analysis connects it to penalties on sensitivity to layer disagreement. The distinction between exact sampling identities and the assumptions required for the downstream objective analysis is useful.

Critical weaknesses
  1. The practical gain over a well-tuned fixed-fusion RAEv2 pipeline remains insufficiently established. The main generation grids tune FuseReg rates with a 23-layer candidate pool and 40-epoch generator training. The native 23-layer unguided decoder-swap improvement from 3.01 to 2.21 is meaningful, but the comparison across fusion choices remains incomplete. In particular, RAEv2 reports gFID 1.06 at 80 epoch training with internal guidance. This is the optimal configuration and should be directly compared against. The authors should be able to finish 80-epoch training if they can finish 40 (and the diffusion with RAE training is fast). The most directly comparable baseline is RAEv2, so the comparison should be complete. RAEv2 is also the method that the paper is based on.

In the abstract, the author also emphasizes DiT-base cases. But what we really care about in foundational works (i.e. foundational diffusion training, representation, and VAEs) is better convergence at scale. DiT-Base can be a good preview, but DiT-XL at full convergence should be what we target. Though it is hard to scale to billions of parameters, comparisons at suboptimal config and small models are not sufficient for foundational AI research.

  1. The motivation is still unclear. I still feel the paper is a little incremental. (I am not saying incremental is meaningless, as most works are incremental but effective). It introduces random dropping based on RAEv2. However, it is more likely a training trick to improve decoder robustness instead of finding some overlooked features. Though there is some statistical analysis, the nature of augmenting decoder input does not change. In this case, I would like to see real improvements under optimal settings with baselines.
Questions for authors

I would like to see the optimal settings (XL, 80 epochs, maybe 7 layers) of this paper. RAEv2 reports a gFID of 1.06 in that setting. I will raise to WA if 1) the gFID at the optimal config improves significantly, or 2) the author someshow shows a better scaling/convergence behavior.

2 comments

Motivation and choice of setting. As clarified in our official comment, we investigate why better reconstruction does not necessarily translate into better diffusion generation. Our hypothesis is that the two objectives encourage different information shortcuts: cues that facilitate pixel reconstruction need not make the latent distribution easier to model. A frozen pretrained encoder provides a controlled hierarchy of features for studying this tension. FuseReg intervenes on layer selections, and our analysis connects this intervention to regularization along cross-layer disagreement directions.

We focus on k=23k=23 because it exposes this tension clearly: incorporating more layers improves reconstruction but worsens generation relative to k=7k=7. This setting allows us to investigate whether changing how downstream models use the available information can improve the trade-off while retaining the richer representation.

Why study both DiT-Base and DiT-XL? The comparison probes how generator capacity affects the usefulness of regularization. Generator-side regularization provides additional gains on DiT-Base, while DiT-XL achieves its best reported gFID with decoder-only regularization. This pattern is consistent with the interpretation that a smaller generator benefits more from regularizing layer-specific variation, whereas a larger generator can model that variation better. We will present this as a capacity-dependent interpretation and revise statements implying uniform generator-side gains.

New DiT-XL results through 80 epochs. Following the reviewer's request, we provide DiT-XL 80 epochs. The following comparison uses k=23k=23, pdit=0p_{\mathrm{dit}}=0, and the same guidance setting. At each checkpoint, both decoders render the same generated latents; only the decoder changes.

DiT-XL training epochs20406080
RAEv2 decoder: guided gFID ↓2.201.571.311.25
FuseReg decoder (pdec=0.95p_{\mathrm{dec}}=0.95): guided gFID ↓1.971.511.301.25

FuseReg improves the reported guided gFID at the earlier checkpoints and matches the 80-epoch baseline, without modifying or retraining the generator. These results show an improved image-quality–versus–generator-training-budget trade-off through decoder replacement.

We distinguish this matched k=23k=23 comparison from RAEv2's best reported guided gFID of 1.06 at k=7k=7. Our contribution is to identify and regularize information reliance within the shared representation, with benefits evaluated separately by fusion, model scale, and guidance setting.

Thanks the author for responding.

I still do not understand why this paper is important and what is the benefits? The proposed method does not either improve generation quality nor the convergence (in the DiT XL guided experiments, it does converge faster at earlier stage, but the slope of convergences becomes flat and match RAEv2-K23). The results for K7 at epoch 80 is not provided, where I view this as a negative signal, because this flattening becomes much earlier in K=7 setting, where the proposed method achieves a worse gFID when further training.

Yet the tradeoff between generation and reconstruction is worth of studying, and there exists many works on that [1-3] showing that latent spaces fitting to high-frequency will be better for reconstruction not generation and vice versa. I do not think this paper introduces new findings. So the paper should aim for solve this issue. The proposed solution is fusereg, but it based on a suboptimal configuration and shows no benefits when basing on optimal configuration. The reconstruction is not better significantly: PSNR and SSIM better than RAEv2 K23 but rFID worse, in K23 setting for both. Similarly, for K7 setting, there is no improvements significantly for reconstruction and better generation. The paper should be at least supported by one of: better reconstruction with similar generation at optimal setting, slightly better for both, or better generation with similar reconstruction.

Currently, at optimal setting, I only see a better reconstruction with worse generation. This behavior can even be achieved by change the KL weight for VAE etc.

Therefore, I stick with rejection

[1] Camuto, A., & Willetts, M. (2022). Variational autoencoders: A harmonic perspective. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), Proceedings of Machine Learning Research.

[2] Takida, Y., Liao, W.-H., Lai, C.-H., Uesaka, T., Takahashi, S., & Mitsufuji, Y. (2022). Preventing oversmoothing in VAE via generalized variance parameterization. Neurocomputing

[3] Xiang, X., Tian, X., Zhang, G., Chen, Y., Zhang, S., Wang, X., Tao, X., & Fan, Q. (2025). Denoising vision transformer autoencoder with spectral self-regularization. arXiv preprint arXiv:2511.12633.

Reviewer_6v9kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FuseReg, which is an extension of previous work RAEv2. RAEv2 uses multiple layers from the vision encoder through a fixed layer-fusion mechanism. However, it suffers from a trade-off between reconstruction and generation. Specifically, reconstruction prefers features from shallower layers, while generation benefit from deeper layers.

To address the trade-off, FuseReg trains both the decoder and the diffusion model on randomly sampled subsets of encoder layers instead of a fixed fusion Authors mathematically prove that it is a penalty on cross-layer disagreement. Authors did experiments on ImageNet-256 to verify the effectiveness of FuseReg across fusions methods and improves unguided generation.

Critical strengths
  • The paper’s motivation is clear: it tries to address the trade-off between reconstruction and generation in RAEs.
  • The paper writing for the method and mathematical proof parts is easy to follow.
  • The FuseReg method is simple. It can be used not only with FuseReg-trained generation models, but also with fixed-fusion-based RAEv2 generation models.
  • The effectiveness of FuseReg on:
    • apply method for reconstruction only and re-use RAEv2 generation model (DiT-XL), achieving 27% lower unguided FID.
    • apply method for both reconstruction and generation training, achieving 29% lower unguided FID on DiT-B.
Critical weaknesses
  • The paper writing of the experiment part is scattered and hard to follow. Each table uses its own baseline protocol. There is no unified, system-level comparison against fully trained RAEv2 and other generation methods.
  • The scalability of FuseReg seems limited in the DiT-XL experiments. Applying FuseReg to both stages isn’t better than reusing the generation model from RAEv2.
    • unguided performance (Table 2c): RAEv2 (2.91) -> reconstruction stage (2.38) -> both stages (2.39)
    • guided performance (Table D.2): RAEv2 (1.57) -> reconstruction stage (1.50) -> both stages (1.53)
  • The real application of FuseReg is limited. In terms of image generation, guided results matter more. However, FuseReg barely improves the guided FID performance for DiT-XL (1.25 -> 1.25 in Table D.1).
Questions for authors
  • Please refer to the weakness section above.
  • Please provide the detailed experimental setup for all tables and figures, e.g., how was the RAEv2 DiT in Fig. 3 / Table D.1 trained?
1 comment

Q2 / W1: Experimental Setup & Protocol Alignment

We agree that baseline provenance in Fig. 3 / Table D.1 and exact hyperparameters should be made explicit up front. §5 and Appendix B describe our shared setup (frozen DINOv3-L with K=23K=23 on ImageNet-256, 50k evaluation samples, 50 Euler steps). In the revision, we will add the two summary tables below to §5 and Appendix B.

1. Overview of Experimental Settings All decoders we trained for 16 epochs with an identical recipe. Generation experiments strictly compare models within each setting:

SettingTables / FigsGeneratorDecoders
1. ReconstructionTable 1, C.1NoneOfficial RAEv2 decoders vs. FuseReg (16 ep, same configs)
2. Decoder Swap (Zero-shot on 80-ep DiT)Fig. 3, Table D.180-ep RAEv2 DiT-XL (k=7k=7: public checkpoint; k=23k=23: reproduced)Our 16-ep pdecp_{\text{dec}} grid (all decoding the exact same 50k generated latents)
3. Joint Rate Grid (Controlled ablation)Table 2, D.2Our 40-ep DiT-B / DiT-XL trained from scratch across pditp_{\text{dit}} gridSame as above

2. Complete Hyperparameter Specification

HyperparameterPixel DecoderDiT-B GeneratorDiT-XL Generator
ArchitectureViT (28L, width 1152, 16H)DDT-head DiT: body 12×768, head 2×1024 (base @ L4)Body 28×1440, head 2×2048 (base @ L8)
Input & Sampling16×16×102416 \times 16 \times 1024 subset fusion (L1–23) + ℓ23\ell_{23} mean surrogate; each layer dropped w.p. pp (at least 1 kept)Same (patch size 1)Same (patch size 1)
ObjectiveMSE + LPIPS (1.0) + adaptive GAN (0.75, DINO disc. from ep 8)xx-pred of zfullz_{\text{full}}, ρ(t)=max⁡(1−t,0.05)−2\rho(t)=\max(1-t,0.05)^{-2}, logit-normal tt (shift 8), label drop 0.1Same
OptimizerAdamW (β=(0.9,0.95)\beta=(0.9, 0.95), wd 0)Muon (2D params) + AdamW (β=(0.9,0.95)\beta=(0.9, 0.95), wd 0)Same
LR Schedule2e-4 (2-ep warmup, cosine)2e-4 (6-ep warmup, constant)2e-4 (25-ep warmup →\to 2e-5 @ ep 50)
Batch / Epochs256 / 16 epochs2048 / 40 epochs1024 / 40 epochs
EMA / Grad Clip0.9995 / 1.00.9995 / 1.00.9995 / 1.0
Inference (50k)PSNR, SSIM, rFID on val50-step Euler (gFID, IS)50-step Euler; internal guidance 1.781.78

W2: Scaling to DiT-XL & Stage-Wise Roles

Your observation aligns directly with our analysis in §6 ("Why the two stages respond differently") and §D.2: decoder-side and generator-side regularization serve distinct functions across model scales.

  • Decoder-side regularization (p_dec > 0) transfers across all scales. Without touching the generator (p_dit = 0), replacing the decoder consistently improves unguided gFID on DiT-B (13.96 -> 12.96, Table 2a), and 80-epoch RAEv2 DiT-XL (3.01 -> 2.21, Table D.1).
  • Generator-side regularization (p_dit > 0) is capacity-dependent. Capacity-limited generators like DiT-B easily overfit to layer-shortcuts, so joint regularization yields our largest relative gain (13.96 -> 9.93, -29%), exceeding the sum of individual stage gains (§D.2). In contrast, high-capacity DiT-XL models already capture whole latent effectively; adding p_dit > 0 provides limited benefit (2.38 vs. 2.39 unguided; 1.50 vs. 1.53 guided).

Practically, this gives FuseReg complementary utility across compute regimes: when training smaller or compute-constrained diffusion models, joint regularization substantially improves gFID; at the XL scale, training only the decoder suffices, providing a no generator-side cost upgrade.

We will update the abstract and §5.3 to explicitly state that generator-side gains are capacity-dependent, and we will remove the inaccurate claim in §5.3 that joint regularization unlocks further gains on DiT-XL).


W3: Guided Generation & Practical Utility

Guidance is a sampling-time correction whereas FuseReg regularizes the training-time latent interface shared by the decoder and generator, our method provides two clear benefits alongside competitive guided scores as we discussed in limitation:

  1. One decoder for every fusion: A single FuseReg decoder reconstructs from k=23k=23, k=7k=7, and single-layer inputs (Table 1) and decodes the latents of both the k=23k=23 and k=7k=7 RAEv2 DiT-XL generators (Table D.1).
  2. A better unguided distribution without touching the generator: Guidance is a sampling-time correction that trades diversity for fidelity and must be tuned for each model. The unguided metric therefore measures most directly the part FuseReg changes: the distribution produced by the generator and decoder themselves. As in prior RAE work, we report both.

We will make the complementary roles of training-time interface regularization and sampling-time guidance explicit in the introduction and conclusion.

Reviewer_XsHZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to fuse multi-layer features from a pretrained vision foundation model in Representation Autoencoders (RAEs). Existing methods such as RAEv2 choose the layers by a fixed heuristic, for example summing the last k layers. The authors argue that this choice is not optimal for both reconstruction and generation.

Instead of searching for one best fusion, FuseReg randomly drops encoder layers during training, like dropout at the layer level. It then uses the normalized mean of the remaining layers as the latent for both the pixel decoder and the DiT. At inference, all layers are used. The goal is to make both models robust to many different layer combinations.

The paper also shows that the layers that help reconstruction do not always help generation. The method is simple and adds almost no extra cost. The experiments are extensive, and the paper includes a theoretical analysis.

Critical strengths
  1. The motivation is clear and the problem is well analyzed. Figure 1 (the learnable gate collapses onto a shallow layer) and Figure 4 (the decoder relies heavily on shallow layers) clearly show why a fixed or learned global fusion is a problem.
  2. The method is simple and adds almost no extra cost: no new parameters and no change to the architecture.
  3. The experiments are extensive and cover several settings, and the paper also gives a theoretical analysis.
Critical weaknesses

W1. The main claim, "narrowing the reconstruction–generation gap", is not directly shown, and the key comparison is missing

The introduction uses RAEv2 results to show that the gap exists. To show that the gap is narrowed, FuseReg should reach the reconstruction quality of k = 23 and the generation quality of k = 7 at the same time. The paper's numbers are:

ConfigurationrFIDUnguided gFIDGuided gFID
RAEv2, k = 70.301.651.06
RAEv2, k = 230.183.011.25
FuseReg decoder (p_dec = 0.95), k = 23 DiT0.422.211.25
FuseReg decoder (p_dec = 0.95), k = 7 DiT0.601.921.42

The rFID of each FuseReg row is measured on the matching fusion (Table 1). On rFID and gFID, the fixed k = 7 RAEv2 is better than both FuseReg configurations. FuseReg is better only on PSNR.

In addition, the baseline in the k = 7 column of Table D.1 uses a decoder trained on k = 23 to decode k = 7 latents ("The gray row uses the reproduced fixed-k=23 decoder"). The large drop from 27.73 to 1.92 therefore mainly reflects an input mismatch in the baseline. It is not evidence that generation quality improved.

The k = 23 baseline in Table D.1 (3.01 and 1.25) matches the introduction exactly. If the k = 7 results in the introduction (1.65 and 1.06) were obtained under the same protocol, a fair k = 7 baseline is likely better than FuseReg.

W2. Key baselines that explain why the method works are missing

  1. Latent noise augmentation. The original RAE paper already trains the decoder with noise augmentation, so that it can handle generated latents that are off the data manifold. The decoder-side gain of FuseReg may come from this general robustness to latent perturbations, not from the specific structure of "perturbing along layer-disagreement directions".
    • Tables C.1 and 2 support this concern. At p_dec = 0.05, full-fusion reconstruction is the best (PSNR 28.59), but generation is the worst (DiT-Base gFID 17.28). The generation gain appears only when the perturbation is strong enough.
    • Suggested comparisons:
      • isotropic Gaussian noise with the same total variance as the layer-disagreement perturbation;
      • perturbations of the same strength along random directions;
      • token-level or channel-level dropout.
    • Please also state whether the reproduced RAEv2 baseline decoder uses noise augmentation.
  2. Deterministic multi-fusion training. Train the decoder by switching between a few fixed fusions, for example k = 7, k = 23, and a few single layers. This would show whether covering all 2^K − 1 subsets is really needed.
  3. Learned fusion. Figure 1 only shows how the gate collapses; it gives no reconstruction or generation numbers for the learned gate. DRoRAE, which uses input-dependent routing and is the most direct competing method, is only mentioned in the related work and is not compared in the experiments.

W3. The gain from generator-side regularization (p_dit) is not stable, and the text overstates it

  • DiT-XL, unguided (Table 2c). The best gFID, 2.38, is at (p_dec = 0.95, p_dit = 0). In the p_dec = 0.95 column, every setting with p_dit > 0 (2.45, 2.42, 2.39) is worse than regularizing the decoder alone. Section 5.3 says "joint regularization with the decoder successfully unlocks further gFID gains". This does not match the table, and it contradicts Section 6, which says "DiT-XL favors … p_dit = 0".
  • DiT-XL, guided (Table D.2). For every p_dec, gFID with p_dit > 0 is worse than with p_dit = 0, and the best IS is at (0, 0).
  • DiT-Base. The "super-additive" gain (1.16 beyond the sum of the two single gains) comes from one point picked from the grid. Section D.2 itself says this is not a statistical test.
  • The abstract. says "Applying FuseReg to both stages also reduces unguided gFID by 29% on DiT-Base". It does not mention that, for the larger model and with guidance, generator-side regularization does not help and can even hurt.

W3. The abstract and the Table 1 caption describe the reconstruction results too positively

  • The abstract says FuseReg achieves "higher PSNR than decoders specialized to fixed fusions", and the Table 1 caption says it "matches or outperforms any deterministic fusion".
  • However, rFID is worse in both cases: 0.42 vs. 0.18 at k=23k = 23, and 0.60 vs.vs. 0.30 at k=7k = 7.
  • "Without additional training cost" holds only for a single training run. The two drop rates must be tuned by grid search for each model size, metric, and guidance setting (as Section 7 admits), so the real tuning cost is not small.
Questions for authors

Please see the weakness.

2 comments

W1. Clarifying the reconstruction–generation gap and its mitigation.

The gap concerns a mismatch in layer preference: reconstruction training encourages the decoder to exploit shallow, pixel-aligned shortcuts, whereas deeper-layer fusions yield better generation in the motivating RAEv2 comparison. The introduction already illustrates this conflict using paired reconstruction fidelity: increasing k from 7 to 23 improves PSNR 22.58 → 27.04 dB, while unguided gFID worsens 1.65 → 3.01. By “narrowing,” we mean reducing this shortcut dependence and improving compatibility between the decoder and generator while retaining reconstruction fidelity.

Evidence for mitigation. Figs. 4 and C1 show reduced dependence on shallow layers and stronger reconstruction from individual layers and subsets, supporting improved access to reconstruction information across depth. At k=23, FuseReg improves PSNR 27.04 → 27.52, SSIM 0.806 → 0.826, and LPIPS 0.0908 → 0.0884. In the controlled decoder-swap experiment, keeping the DiT and sampled latents fixed reduces unguided gFID 3.01 → 2.21 while preserving guided gFID at 1.25. These diagnostics and controlled results jointly support mitigation of the mismatch at the shared latent interface.

On rFID. rFID is a set-level distributional metric: it compares Inception feature means and covariances without using input–reconstruction pairing. Permuting reconstructions across inputs leaves rFID unchanged, even when every input–output pairing becomes incorrect. Thus, lower rFID alone cannot establish greater faithfulness to the input. It also need not imply better visual quality: FD-loss (Yang et al., NeurIPS 2026) optimizes a distribution-level objective without per-sample targets and shows that perceptually better samples can receive worse Inception FID. We therefore assess paired reconstruction fidelity using PSNR, SSIM, and LPIPS, which quantify pixel, structural, and perceptual agreement with the corresponding input. FuseReg’s rFID increases 0.18 → 0.42 at k=23; we report this result explicitly, while specifying that the reconstruction improvements concern paired fidelity.

On the k=7 comparison. We will relabel the gray row in Table D.1 as “k=23-trained decoder evaluated on k=7 latents.” Its 27.73 → 1.92 comparison demonstrates cross-fusion robustness. We will add the native k=7 specialist baseline with its protocol and provenance clearly identified, including the introduction-reported 1.65/1.06 alongside FuseReg’s 1.92/1.42. This distinguishes cross-fusion robustness from generation performance relative to a fusion-specific decoder.

We will revise the introduction and table captions to connect “narrowing” explicitly to reduced shallow-layer dependence, improved paired reconstruction fidelity, and the controlled generation improvement at k=23.

W2. Perturbation and fusion baselines.

Noise augmentation and rate dependence. RAEv2 decoder is already trained with additive Gaussian noise (σ∼U(0,0.8)\sigma\sim\mathcal U(0,0.8) per sample). FuseReg adds no latent noise; its perturbation comes only from random layer-subset fusion. That stronger regularization helps generation is discussed in §5.3 and §6: increasing pdecp_{\mathrm{dec}} amplifies regularization along layer-disagreement directions through cK(p)c_K(p), so clean reconstruction and decoding of imperfect generated latents need not favor the same rate. Rate dependence alone does not separate structured from generic perturbations, which motivates the controls below.

Energy-matched perturbations and deterministic multi-fusion training. All five decoders use DINOv3-L, 5 epochs, batch 256, MSE + LPIPS, and GAN from epoch 3. Gaussian noise and inverted token/channel dropout are calibrated to match E∥δF∥2\mathbb E\Vert \delta_F\Vert ^2, where δF=msubset−m\delta_F=m_{\mathrm{subset}}-m at pdec=0.95p_{\mathrm{dec}}=0.95, estimated on 4,096 training images. The Gaussian control (σ=0.667\sigma=0.667) also provides random perturbation directions at matched energy. The multi-fusion baseline switches among k=23k=23, k=7k=7, and ℓ11\ell_{11}. Evaluation uses 10K images without stochastic augmentation; each cell is PSNR↑ / SSIM↑ / rFID↓.

Decoderk=23k=23k=7k=7ℓ11\ell_{11}
Gaussian noise24.12 / 0.688 / 3.4713.18 / 0.397 / 15.1718.24 / 0.534 / 7.47
Token dropout16.79 / 0.603 / 8.7313.02 / 0.363 / 38.3315.54 / 0.464 / 12.92
Channel dropout14.02 / 0.499 / 9.9713.78 / 0.402 / 14.7514.15 / 0.433 / 8.07
Deterministic multi-fusion24.97 / 0.702 / 3.7422.68 / 0.583 / 5.8023.49 / 0.619 / 4.98
FuseReg (pdec=0.95p_{\mathrm{dec}}=0.95)26.12 / 0.776 / 2.4523.14 / 0.624 / 3.6824.13 / 0.672 / 2.98

FuseReg is best on all three fusions. Matching perturbation energy does not reproduce its accuracy or cross-fusion robustness, and FuseReg also outperforms deterministic multi-fusion training, which trains directly on all three evaluation fusions. This supports the value of layer-structured subset sampling.

Learned fusion. The global gate in Fig. 1 reaches PSNR 34.49 dB, SSIM 0.965, and rFID 0.36 on its learned fusion after five epochs, yet a DiT-Base trained on this latent for 40 epochs obtains gFID 57.32 (IS 48.6). Strong reconstruction alone does not ensure generation compatibility under this budget. This baseline is the global gate of Fig. 1, distinct from DRoRAE's input-dependent routing and staged training.


W3. Scope of the generation, reconstruction, and cost claims.

Decoder-side gains. Our central finding is that decoder-side FuseReg improves unguided generation with an unchanged generator: replacing only the decoder reduces gFID 3.01 → 2.21 with the DiT and generated latents fixed, isolating the benefit of decoder-side training. As clarified in W1, our reconstruction claims concern paired fidelity (PSNR, SSIM, LPIPS) and robustness across layer fusions.

Generator-side gains depend on model scale. On DiT-Base, joint regularization reaches 9.93 gFID vs. 12.96 (decoder-only) and 13.96 (baseline), and at fixed pdec=0.9p_{\mathrm{dec}}=0.9 all six nonzero generator rates improve over decoder-only regularization. We will describe the 1.16 excess over the additive prediction as a configuration-specific observation, not a statistically established interaction. On DiT-XL, the lowest gFID in Table 2(c), 2.38, occurs at pdit=0p_{\mathrm{dit}}=0, and guided gFID also favors pdit=0p_{\mathrm{dit}}=0. These findings establish a decoder-side generation benefit and additional generator-side gains on DiT-Base, with preferred rates depending on scale, metric, and guidance. We will make this explicit in Sec. 5.3 and the abstract.

Computational cost. FuseReg adds no per-run training cost: layer-subset sampling replaces the fixed fusion within the same recipe and step budget, with no extra modules or training steps. On the decoder side, a FuseReg decoder is trained once and swapped in for the original, improving unguided gFID without retraining the DiT and without adding inference-time modules or sampling steps. Our cost statement does not cover hyperparameter selection: the regularization rates were chosen from a grid, and the generator-side rate may need to be selected per DiT size. We will replace the broad “no additional computational cost” wording with “no additional per-run training cost” and state the selection cost explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.