- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper replaces a fixed fusion during training (basically the RAEv2 method) with the normalized mean of a randomly sampled subset of layers. The decoder reconstructs images from randomized fusions. During training, the diffusion models predict full sets of selected layer representations. The results show that the decoder trained with randomized subsets is robust to be directly replaced with RAEv2. The gFID under certain configuration is improved.
Critical strengths
-
Extensive ablations that separate different effects. The paper evaluates decoder and generator regularization rates separately and jointly, compares DiT-Base and DiT-XL, studies guided and unguided generation, and includes additional encoder families.
-
Explicit mathematical analysis of the proposed perturbation. The mean-preservation and covariance results characterize what random subset fusion changes, and the linear squared-loss analysis connects it to penalties on sensitivity to layer disagreement. The distinction between exact sampling identities and the assumptions required for the downstream objective analysis is useful.
Critical weaknesses
- The practical gain over a well-tuned fixed-fusion RAEv2 pipeline remains insufficiently established. The main generation grids tune FuseReg rates with a 23-layer candidate pool and 40-epoch generator training. The native 23-layer unguided decoder-swap improvement from 3.01 to 2.21 is meaningful, but the comparison across fusion choices remains incomplete. In particular, RAEv2 reports gFID 1.06 at 80 epoch training with internal guidance. This is the optimal configuration and should be directly compared against. The authors should be able to finish 80-epoch training if they can finish 40 (and the diffusion with RAE training is fast). The most directly comparable baseline is RAEv2, so the comparison should be complete. RAEv2 is also the method that the paper is based on.
In the abstract, the author also emphasizes DiT-base cases. But what we really care about in foundational works (i.e. foundational diffusion training, representation, and VAEs) is better convergence at scale. DiT-Base can be a good preview, but DiT-XL at full convergence should be what we target. Though it is hard to scale to billions of parameters, comparisons at suboptimal config and small models are not sufficient for foundational AI research.
- The motivation is still unclear. I still feel the paper is a little incremental. (I am not saying incremental is meaningless, as most works are incremental but effective). It introduces random dropping based on RAEv2. However, it is more likely a training trick to improve decoder robustness instead of finding some overlooked features. Though there is some statistical analysis, the nature of augmenting decoder input does not change. In this case, I would like to see real improvements under optimal settings with baselines.
Questions for authors
I would like to see the optimal settings (XL, 80 epochs, maybe 7 layers) of this paper. RAEv2 reports a gFID of 1.06 in that setting. I will raise to WA if 1) the gFID at the optimal config improves significantly, or 2) the author someshow shows a better scaling/convergence behavior.
2 comments
Thanks the author for responding.
I still do not understand why this paper is important and what is the benefits? The proposed method does not either improve generation quality nor the convergence (in the DiT XL guided experiments, it does converge faster at earlier stage, but the slope of convergences becomes flat and match RAEv2-K23). The results for K7 at epoch 80 is not provided, where I view this as a negative signal, because this flattening becomes much earlier in K=7 setting, where the proposed method achieves a worse gFID when further training.
Yet the tradeoff between generation and reconstruction is worth of studying, and there exists many works on that [1-3] showing that latent spaces fitting to high-frequency will be better for reconstruction not generation and vice versa. I do not think this paper introduces new findings. So the paper should aim for solve this issue. The proposed solution is fusereg, but it based on a suboptimal configuration and shows no benefits when basing on optimal configuration. The reconstruction is not better significantly: PSNR and SSIM better than RAEv2 K23 but rFID worse, in K23 setting for both. Similarly, for K7 setting, there is no improvements significantly for reconstruction and better generation. The paper should be at least supported by one of: better reconstruction with similar generation at optimal setting, slightly better for both, or better generation with similar reconstruction.
Currently, at optimal setting, I only see a better reconstruction with worse generation. This behavior can even be achieved by change the KL weight for VAE etc.
Therefore, I stick with rejection
[1] Camuto, A., & Willetts, M. (2022). Variational autoencoders: A harmonic perspective. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), Proceedings of Machine Learning Research.
[2] Takida, Y., Liao, W.-H., Lai, C.-H., Uesaka, T., Takahashi, S., & Mitsufuji, Y. (2022). Preventing oversmoothing in VAE via generalized variance parameterization. Neurocomputing
[3] Xiang, X., Tian, X., Zhang, G., Chen, Y., Zhang, S., Wang, X., Tao, X., & Fan, Q. (2025). Denoising vision transformer autoencoder with spectral self-regularization. arXiv preprint arXiv:2511.12633.
Motivation and choice of setting. As clarified in our official comment, we investigate why better reconstruction does not necessarily translate into better diffusion generation. Our hypothesis is that the two objectives encourage different information shortcuts: cues that facilitate pixel reconstruction need not make the latent distribution easier to model. A frozen pretrained encoder provides a controlled hierarchy of features for studying this tension. FuseReg intervenes on layer selections, and our analysis connects this intervention to regularization along cross-layer disagreement directions.
We focus on k=23 because it exposes this tension clearly: incorporating more layers improves reconstruction but worsens generation relative to k=7. This setting allows us to investigate whether changing how downstream models use the available information can improve the trade-off while retaining the richer representation.
Why study both DiT-Base and DiT-XL? The comparison probes how generator capacity affects the usefulness of regularization. Generator-side regularization provides additional gains on DiT-Base, while DiT-XL achieves its best reported gFID with decoder-only regularization. This pattern is consistent with the interpretation that a smaller generator benefits more from regularizing layer-specific variation, whereas a larger generator can model that variation better. We will present this as a capacity-dependent interpretation and revise statements implying uniform generator-side gains.
New DiT-XL results through 80 epochs. Following the reviewer's request, we provide DiT-XL 80 epochs. The following comparison uses k=23, pdit=0, and the same guidance setting. At each checkpoint, both decoders render the same generated latents; only the decoder changes.
FuseReg improves the reported guided gFID at the earlier checkpoints and matches the 80-epoch baseline, without modifying or retraining the generator. These results show an improved image-quality–versus–generator-training-budget trade-off through decoder replacement.
We distinguish this matched k=23 comparison from RAEv2's best reported guided gFID of 1.06 at k=7. Our contribution is to identify and regularize information reliance within the shared representation, with benefits evaluated separately by fusion, model scale, and guidance setting.