- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors propose a method of adding multi-scale consistency regularization to the training of a GAN architecture based on the architecture proposed from "Scalable GANs with Transformers" by Sangeek Hyun etal. The authors evaluate the performance of their method over Image-Net 256 and compare to select baselines on FID-50k. Reported results show an improvement of FID from the baseline architecture from 2.18 to 1.56 which is also an improvement over the best reported flow based model "Improved mean flows: On the challenges of fastforward generative models" by Zhengyang Geng etal.
Critical strengths
-
The proposed method of regularizing multi-scale (or hierarchical) training of a transformer based GAN models appears to offer notable performance gains across different training horizons with respect to the chosen evaluation metrics; notably, FID 50k.
-
The authors provide a detailed ablation study for each of the different elements used by the proposed method as well as providing alternative implementation ideas. This improves the clarity of the method and addresses potential questions around alternative baselines derived from the GAT model.
Critical weaknesses
In no particular order, here are some comments I have regarding both the presentation and distinction of the proposed work from existing methods.
-
Reading the paper, I found the motivation poorly presented. While the resulting improvements in FID are beneficial and the notion of consistency well established within diffusion model literature to accelerate sampling, it was unclear to me from reading the paper top to bottom why consistency is important for such a one-step sampling model. I would strongly suggest the authors rearrange the elements of the Introduction, Preliminary, and Observation contained within the Proposed Method sections. In particular, it seems to me the only motivating idea, outside of metrics, is the observaton in presented in Section.4.1 describing the change in for different stages of the generator and how this relates to the samples presented in Figure.3. I would suggest moving these comments and observation into either the motivation or directly following the Preliminary section in a dedicated section, removing other mentions from the introduction prior to this. The presentation of Fig.2 in the paper I found confusing and not particularly illustrative of the issues the authors are attempting to motivate and correct; it would be much better received if this figure actually illustrated a toy sampling processes rather than what I assume is a baseless illustration.
-
Throughout the paper, the lack of mathematical formalism in the statements of the methods and architecture is a concern; I found it necessary to read the paper "Scalable GANs with Transformers" by Sangeek Hyun etal. to understand the proposed method and notation within the paper. Considering the main contributions of this work are, to my understanding, some alterations to the transformer generator architecture via skip connections and a loss regularizer it is especially frustrating these contributions are not clearly presented and distinguished from related works. I would strongly encourage the authors to add considerably more detail around how the network is parameterized differently, if at all, from GAT, the latent space parameterization, and describe the missing information around how the model is trained as done in the aforementioned work.
-
Adding to my above point, the notation used in this paper is unclear. Beginning with eq(1) it would seem but is never used in the equation. The symbols and are not defined but I presume are both neural networks representing respectively the generator and discriminators. In eq(2) the resizing function does not make sense being applied to the latent code from a VAE, which I believe is the case here based on reading the aforementioned paper, as the latent code is not necessarily a coherent image. I assume the authors dropped the VAE decoder from this equation. In eq(4) I wonder if there is a stop-grad missing from one of the variables in the equation.
Questions for authors
See comments about critical weaknesses.
5 comments
(continued, 2/3)
2. W1: Rearrange the Introduction, Preliminary, and Observation, and move the observation of Sec. 4.1 into a dedicated section
As suggested, the observations of Sec. 4.1 now form a dedicated section right after the Preliminaries, and the introduction no longer previews them.
- Sec. 1, Introduction: the motivation of Point 1, with Fig. 2 (schematic and toy experiment).
- Sec. 3, Preliminaries: the formal setup, including the GAT generator and the scale-wise discriminator (Point 4).
- Sec. 4, Motivating Observation: the ImageNet diagnostics (discrepancy, rewriting, alignment) with the examples of the former Fig. 3 (now Fig. 4).
- Sec. 5, Method: and the full objective.
Revised in: Secs. 1, 3–5.
3. W1: Fig. 2 should actually illustrate a toy sampling process
Fig. 2 now pairs the schematic (a) with a toy experiment (b): every stage matches its target distribution, yet without the last stage moves about half of the samples or more to a different mode in 5 of 10 training runs (30% of all samples), whereas with this happens in none (0.08%).
- Setup. The data are scalars from two coarse clusters, each split into two fine modes. Since a scalar cannot be resized, we emulate coarser scales by adding Gaussian noise to the data, with more noise for earlier stages. As in CAT, the generator accumulates its stage outputs , and each stage has its own discriminator.
- Result. Fig. 2(b) traces samples through the stages. We count a sample as rewritten if its mode changes between stage 2, the first stage at which the data separate all four modes, and the final output. We trained each setting 10 times.
| Toy experiment, 10 training runs per setting | w/o | w/ |
|---|---|---|
| Runs in which the last stage rewrites 49–100% of the samples | 5 of 10 | 0 of 10 |
| Samples rewritten, all runs together | 30.1% | 0.08% |
| Distance between the final output and the data (1-Wasserstein, mean over runs) | 0.030 | 0.013 |
- Matching the data at every scale does not determine which sample each stage depicts. Both settings fit the data at every stage (Appendix A.5). Without , the last stage often swaps the two fine modes of a cluster, which leaves every stage's distribution unchanged and is therefore invisible to the adversarial loss.
- With , the final output also matches the data distribution more closely. Its distance to the data is 0.009–0.016 in the 10 runs, lower than in every run without (0.018–0.046).
- The appendix repeats the experiment on 8×8 images with actual downsampling, with the same result.
Details: Fig. 2, Appendix A.5.
4. W2: Lack of mathematical formalism in the statements of the methods and architecture
The network is parameterized exactly as in GAT, including the output skip connections. CAT changes only how the intermediate outputs are supervised and adds . Sec. 3 now states the model, the latent space, and the objective formally, so the paper can be read without the GAT paper.
- Network parameterization. We use the transformer generator of GAT unchanged. As in GAT, its output layers are linked by output skip connections: the output layer of stage predicts a residual that is added to the previous output, which gives the accumulated outputs (). These connections are therefore not a contribution of CAT. Our contributions are the scale-wise supervision of these outputs and .
- Latent space and the terms "image" and "latent". As in GAT, the model is trained in the SD-VAE latent space, and Sec. 3 now states this in a dedicated paragraph that fixes the two terms. denotes a pixel image (), and denotes its latent (). The generator outputs and the resized are all latents. The paper calls an "image" only for simplicity, because the latent keeps the spatial layout of the image. All operations, including the resizing , act on latents, and the decoder is applied only at inference (Point 5).
- Training. Sec. 3 defines the resized latent that supervises stage (the "scale- image" of the paper), the masked discriminator with one logit per scale, and the adversarial objective. The appendix lists all differences from GAT in a table (Table 10).
Revised in: Sec. 3, Appendix A.2.
(continued, 3/3)
5. W3: The notation used in this paper is unclear
We revised the notation as the reviewer pointed out. On Eq. (2) of the submission, no decoder was dropped: CAT is trained in the latent space, and the resizing acts on the latent itself.
- Eq. (1). The generator and the discriminator are now defined, and the loss is written with the per-scale logits of : with for a generated and a real sample, the generator minimizes .
- Eq. (2) of the submission (the definition of , now in Sec. 3): no decoder was dropped, and we apologize for the confusion. The submission called an "image" although it is a latent. As described in Point 4, resizes the latent itself, and the decoder is used only at inference.
- Resizing the latent is meaningful because the SD-VAE latent keeps the spatial layout of the image. Its encoder is convolutional and downsamples by 8, so each latent cell summarizes the patch at the same location [A]. A resized latent is therefore a coarser summary of the same image.
- The supervision does not require the resized latent to be decodable. Real latents are resized by the same operator, so the discriminator at scale compares with .
- Eq. (4) of the submission (Eq. (3) in the revision). Our implementation applies no stop-gradient. We additionally trained a variant with a stop-gradient on : both improve FID greatly at B/2 (from 5.43 to 4.06 without it and to 3.84 with it), but with it most H/2 runs diverged early, so we do not use it (Table 3).
Revised in: Secs. 3, 5, and 6, Table 3.
We hope these clarifications and new results address the reviewer's concerns, and we are happy to discuss any remaining questions.
References
[A] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.
We thank the reviewer for the detailed suggestions on presentation, which shaped the main changes of this revision. Our responses make three points:
1. W1: Why is consistency important for such a one-step sampling model?
Our intended motivation is that cross-scale consistency is the condition under which multi-scale supervision fully benefits the final sample. We now state this reasoning in the introduction:
Our view and evidence.
Revised in: Sec. 1, Fig. 2.
References
[A] Ho, Jonathan, et al. "Denoising diffusion probabilistic models." Advances in Neural Information Processing Systems (NeurIPS). 2020.
[B] Song, Yang, et al. "Consistency models." International Conference on Machine Learning (ICML). PMLR, 2023.
[C] Karnewar, Animesh, and Oliver Wang. "Msg-gan: Multi-scale gradients for generative adversarial networks." 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.
[D] Kang, Minguk, et al. "Scaling up gans for text-to-image synthesis." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.
[E] Hyun, Sangeek, et al. "Scalable GANs with transformers." International Conference on Machine Learning (ICML). 2026.