acceptodds
Under review as a conference paper at ICLR 2027

Cross-Scale Aligned Supervision for Training GANs

Abstract

Modern GANs often use multi-scale adversarial supervision to provide direct realism feedback to intermediate generator outputs. However, we identify an important limitation of standard scale-wise supervision: each intermediate output is independently optimized for realism at its own resolution, without explicitly coordinating outputs across generator stages. As a result, realistic intermediate outputs need not remain aligned with the final output from the same generated sample. This motivates treating per-scale realism and cross-scale consistency as complementary objectives for multi-scale adversarial generation. Based on this observation, we propose CAT, a Cross-scale Aligned Transformer, which preserves scale-wise adversarial supervision for direct per-scale realism feedback while introducing a simple generator-side consistency regularization to align intermediate outputs with the final output. We further show that jointly exposing multiple scales to the discriminator improves cross-scale consistency but substantially degrades generation quality, supporting our design of enforcing cross-scale consistency separately on the generator side. In a matched 40-epoch CAT-H/2 comparison on ImageNet-256, the proposed consistency regularization improves FID-50K from 4.35 to 1.76. CAT-H/2 achieves an FID-50K of 1.56 with a single generator evaluation after only 60 training epochs.

Primary area:
generative models
Keywords:
Image Generation, Generative Models, Generative Adversarial Networks

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_LHzxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a method of adding multi-scale consistency regularization to the training of a GAN architecture based on the architecture proposed from "Scalable GANs with Transformers" by Sangeek Hyun etal. The authors evaluate the performance of their method over Image-Net 256 and compare to select baselines on FID-50k. Reported results show an improvement of FID from the baseline architecture from 2.18 to 1.56 which is also an improvement over the best reported flow based model "Improved mean flows: On the challenges of fastforward generative models" by Zhengyang Geng etal.

Critical strengths
  • The proposed method of regularizing multi-scale (or hierarchical) training of a transformer based GAN models appears to offer notable performance gains across different training horizons with respect to the chosen evaluation metrics; notably, FID 50k.

  • The authors provide a detailed ablation study for each of the different elements used by the proposed method as well as providing alternative implementation ideas. This improves the clarity of the method and addresses potential questions around alternative baselines derived from the GAT model.

Critical weaknesses

In no particular order, here are some comments I have regarding both the presentation and distinction of the proposed work from existing methods.

  • Reading the paper, I found the motivation poorly presented. While the resulting improvements in FID are beneficial and the notion of consistency well established within diffusion model literature to accelerate sampling, it was unclear to me from reading the paper top to bottom why consistency is important for such a one-step sampling model. I would strongly suggest the authors rearrange the elements of the Introduction, Preliminary, and Observation contained within the Proposed Method sections. In particular, it seems to me the only motivating idea, outside of metrics, is the observaton in presented in Section.4.1 describing the change in δk,Rk,Ak\delta_k, R_k, A_k for different stages of the generator and how this relates to the samples presented in Figure.3. I would suggest moving these comments and observation into either the motivation or directly following the Preliminary section in a dedicated section, removing other mentions from the introduction prior to this. The presentation of Fig.2 in the paper I found confusing and not particularly illustrative of the issues the authors are attempting to motivate and correct; it would be much better received if this figure actually illustrated a toy sampling processes rather than what I assume is a baseless illustration.

  • Throughout the paper, the lack of mathematical formalism in the statements of the methods and architecture is a concern; I found it necessary to read the paper "Scalable GANs with Transformers" by Sangeek Hyun etal. to understand the proposed method and notation within the paper. Considering the main contributions of this work are, to my understanding, some alterations to the transformer generator architecture via skip connections and a loss regularizer it is especially frustrating these contributions are not clearly presented and distinguished from related works. I would strongly encourage the authors to add considerably more detail around how the network is parameterized differently, if at all, from GAT, the latent space parameterization, and describe the missing information around how the model is trained as done in the aforementioned work.

  • Adding to my above point, the notation used in this paper is unclear. Beginning with eq(1) it would seem dk=LGAN(⋅)d_k=L_{GAN}(\cdot) but dkd_k is never used in the equation. The symbols GG and DD are not defined but I presume are both neural networks representing respectively the generator and discriminators. In eq(2) the resizing function does not make sense being applied to the latent code from a VAE, which I believe is the case here based on reading the aforementioned paper, as the latent code is not necessarily a coherent image. I assume the authors dropped the VAE decoder from this equation. In eq(4) I wonder if there is a stop-grad missing from one of the variables in the equation.

Questions for authors

See comments about critical weaknesses.

5 comments

We thank the reviewer for the detailed suggestions on presentation, which shaped the main changes of this revision. Our responses make three points:

  1. Cross-scale consistency is the condition under which multi-scale supervision fully benefits the final sample, and we now state this reasoning explicitly (Point 1).
  2. Fig. 2 now pairs the schematic with a toy experiment, and the ImageNet observations follow the Preliminaries in a dedicated section (Points 2, 3).
  3. The paper now states the model, its latent space, and the training objective formally, and the notation is corrected (Points 4, 5).

1. W1: Why is consistency important for such a one-step sampling model?

Our intended motivation is that cross-scale consistency is the condition under which multi-scale supervision fully benefits the final sample. We now state this reasoning in the introduction:

  1. Image generation is naturally coarse-to-fine, and multi-scale supervision asks the generator to follow this order. An image is organized from global structure to fine detail, so a deep generator can build it the same way: early stages form the coarse structure, and later stages refine it. Supervising the intermediate outputs at coarser scales assigns each stage this role.
  2. This process works only if the intermediate outputs depict the final sample. In a coarse-to-fine process, each intermediate output is a coarse version of the final sample, and the later stages refine it. If the later stages redraw the sample instead, the coarse-to-fine process breaks down, and the feedback given to the intermediate outputs may not carry over to the final sample.
  3. Scale-wise supervision gives the most direct per-scale signal, but it does not ensure this. Each stage is judged only on whether it looks real at its own scale, so nothing prevents later stages from redrawing the sample.
  4. Lcons\mathcal L_{\text{cons}} closes this gap. It ties each intermediate output to the final output of the same forward pass, so the per-scale feedback acts on the final sample.

Our view and evidence.

  • We assume that a one-step generator follows this general coarse-to-fine process across its depth. In diffusion models, the same process is split across timesteps, from coarse structure at high noise to fine detail at low noise [A]. This is why our introduction opened with multi-step models. The consistency we study is therefore along the depth of one forward pass, not across sampling steps as in consistency models [B], which enforce it to reduce the number of steps.
  • Prior GANs have also observed that multi-scale supervision helps. Supervising intermediate generator outputs at multiple scales has been reported to stabilize training and improve sample quality [C, D, E].
  • Fig. 2(a) illustrates this failure. The scale-wise adversarial loss pushes each intermediate output only toward the manifold of real images at its own scale, and any point on that manifold satisfies it equally. The helpful direction is toward the coarse version of the final output (blue), but nothing in the loss prefers it: the output can just as well move toward another realistic coarse image (red), which is then not aligned with the final sample and must be redrawn by the later stages.
  • Following the reviewer's suggestion, we added a toy experiment (Fig. 2(b)), which shows that this redrawing happens even when every stage matches the data (Point 3).
  • Fig. 3 of the submission (Fig. 4 in the revision) and the consistency metrics show the same on ImageNet. Without Lcons\mathcal L_{\text{cons}}, the intermediate outputs change color and structure across stages. At H/2, Lcons\mathcal L_{\text{cons}} reduces the discrepancy from 1.157 to 0.627, together with lower rewriting and higher alignment.
  • Once the stages are consistent, the per-scale feedback pays off at every model size. Lcons\mathcal L_{\text{cons}} improves FID-50K from 5.43 to 4.06 at B/2 (20 epochs), from 2.34 to 1.93 at M/2 (40 epochs), and from 4.35 to 1.76 at H/2 (40 epochs).

Revised in: Sec. 1, Fig. 2.


References

[A] Ho, Jonathan, et al. "Denoising diffusion probabilistic models." Advances in Neural Information Processing Systems (NeurIPS). 2020.

[B] Song, Yang, et al. "Consistency models." International Conference on Machine Learning (ICML). PMLR, 2023.

[C] Karnewar, Animesh, and Oliver Wang. "Msg-gan: Multi-scale gradients for generative adversarial networks." 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.

[D] Kang, Minguk, et al. "Scaling up gans for text-to-image synthesis." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

[E] Hyun, Sangeek, et al. "Scalable GANs with transformers." International Conference on Machine Learning (ICML). 2026.

(continued, 2/3)


2. W1: Rearrange the Introduction, Preliminary, and Observation, and move the observation of Sec. 4.1 into a dedicated section

As suggested, the observations of Sec. 4.1 now form a dedicated section right after the Preliminaries, and the introduction no longer previews them.

  • Sec. 1, Introduction: the motivation of Point 1, with Fig. 2 (schematic and toy experiment).
  • Sec. 3, Preliminaries: the formal setup, including the GAT generator and the scale-wise discriminator (Point 4).
  • Sec. 4, Motivating Observation: the ImageNet diagnostics (discrepancy, rewriting, alignment) with the examples of the former Fig. 3 (now Fig. 4).
  • Sec. 5, Method: Lcons\mathcal L_{\text{cons}} and the full objective.

Revised in: Secs. 1, 3–5.


3. W1: Fig. 2 should actually illustrate a toy sampling process

Fig. 2 now pairs the schematic (a) with a toy experiment (b): every stage matches its target distribution, yet without Lcons\mathcal L_{\text{cons}} the last stage moves about half of the samples or more to a different mode in 5 of 10 training runs (30% of all samples), whereas with Lcons\mathcal L_{\text{cons}} this happens in none (0.08%).

  • Setup. The data are scalars from two coarse clusters, each split into two fine modes. Since a scalar cannot be resized, we emulate coarser scales by adding Gaussian noise to the data, with more noise for earlier stages. As in CAT, the generator accumulates its stage outputs h0,…,h3h_0,\dots,h_3, and each stage has its own discriminator.
  • Result. Fig. 2(b) traces samples through the stages. We count a sample as rewritten if its mode changes between stage 2, the first stage at which the data separate all four modes, and the final output. We trained each setting 10 times.
Toy experiment, 10 training runs per settingw/o Lcons\mathcal L_{\text{cons}}w/ Lcons\mathcal L_{\text{cons}}
Runs in which the last stage rewrites 49–100% of the samples5 of 100 of 10
Samples rewritten, all runs together30.1%0.08%
Distance between the final output and the data (1-Wasserstein, mean over runs)0.0300.013
  • Matching the data at every scale does not determine which sample each stage depicts. Both settings fit the data at every stage (Appendix A.5). Without Lcons\mathcal L_{\text{cons}}, the last stage often swaps the two fine modes of a cluster, which leaves every stage's distribution unchanged and is therefore invisible to the adversarial loss.
  • With Lcons\mathcal L_{\text{cons}}, the final output also matches the data distribution more closely. Its distance to the data is 0.009–0.016 in the 10 runs, lower than in every run without Lcons\mathcal L_{\text{cons}} (0.018–0.046).
  • The appendix repeats the experiment on 8×8 images with actual downsampling, with the same result.

Details: Fig. 2, Appendix A.5.


4. W2: Lack of mathematical formalism in the statements of the methods and architecture

The network is parameterized exactly as in GAT, including the output skip connections. CAT changes only how the intermediate outputs are supervised and adds Lcons\mathcal L_{\text{cons}}. Sec. 3 now states the model, the latent space, and the objective formally, so the paper can be read without the GAT paper.

  • Network parameterization. We use the transformer generator of GAT unchanged. As in GAT, its output layers are linked by output skip connections: the output layer of stage kk predicts a residual that is added to the previous output, which gives the accumulated outputs h0,…,hKh_0,\dots,h_K (K=3K=3). These connections are therefore not a contribution of CAT. Our contributions are the scale-wise supervision of these outputs and Lcons\mathcal L_{\text{cons}}.
  • Latent space and the terms "image" and "latent". As in GAT, the model is trained in the SD-VAE latent space, and Sec. 3 now states this in a dedicated paragraph that fixes the two terms. II denotes a pixel image (3×256×2563\times256\times256), and x=E(I)x=\mathcal{E}(I) denotes its latent (4×32×324\times32\times32). The generator outputs hkh_k and the resized xkx_k are all latents. The paper calls xx an "image" only for simplicity, because the latent keeps the spatial layout of the image. All operations, including the resizing rkr_k, act on latents, and the decoder is applied only at inference (Point 5).
  • Training. Sec. 3 defines the resized latent xk=rk(hk)x_k=r_k(h_k) that supervises stage kk (the "scale-kk image" of the paper), the masked discriminator with one logit dkd_k per scale, and the adversarial objective. The appendix lists all differences from GAT in a table (Table 10).

Revised in: Sec. 3, Appendix A.2.

(continued, 3/3)


5. W3: The notation used in this paper is unclear

We revised the notation as the reviewer pointed out. On Eq. (2) of the submission, no decoder was dropped: CAT is trained in the latent space, and the resizing acts on the latent itself.

  • Eq. (1). The generator GG and the discriminator DD are now defined, and the loss is written with the per-scale logits dkd_k of DD: with Δk=dk(xk)−dk(xkreal)\Delta_k=d_k(x_k)-d_k(x_k^{\text{real}}) for a generated and a real sample, the generator minimizes Ladv=1K+1∑kE[softplus(−Δk)]\mathcal L_{\text{adv}}=\frac{1}{K+1}\sum_k\mathbb{E}[\mathrm{softplus}(-\Delta_k)].
  • Eq. (2) of the submission (the definition of xkx_k, now in Sec. 3): no decoder was dropped, and we apologize for the confusion. The submission called xkx_k an "image" although it is a latent. As described in Point 4, rkr_k resizes the latent itself, and the decoder is used only at inference.
  • Resizing the latent is meaningful because the SD-VAE latent keeps the spatial layout of the image. Its encoder is convolutional and downsamples by 8, so each latent cell summarizes the 8×88\times8 patch at the same location [A]. A resized latent is therefore a coarser summary of the same image.
  • The supervision does not require the resized latent to be decodable. Real latents are resized by the same operator, so the discriminator at scale kk compares rk(hk)r_k(h_k) with rk(xreal)r_k(x^{\text{real}}).
  • Eq. (4) of the submission (Eq. (3) in the revision). Our implementation applies no stop-gradient. We additionally trained a variant with a stop-gradient on hKh_K: both improve FID greatly at B/2 (from 5.43 to 4.06 without it and to 3.84 with it), but with it most H/2 runs diverged early, so we do not use it (Table 3).

Revised in: Secs. 3, 5, and 6, Table 3.

We hope these clarifications and new results address the reviewer's concerns, and we are happy to discuss any remaining questions.


References

[A] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.

Reviewer_sd52OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the limitations of standard scale-wise adversarial supervision in multi-stage GANs, pointing out that per-scale realism does not guarantee cross-scale consistency. To address this, the authors propose CAT (Cross-scale Aligned Transformer). CAT keeps the scale-wise adversarial feedback in the discriminator (via a block-diagonal attention mask) but introduces a simple yet effective generator-side consistency regularization loss. This loss encourages intermediate latent outputs to align with the final output during generation. The method achieves an impressive FID-50K of 1.56 on ImageNet-256 with a single evaluation (1-NFE), demonstrating state-of-the-art performance and significantly lower training compute compared to strong baselines like GAT and iMF.

Critical strengths
  1. Clear Problem Identification and Elegant Solution: The paper correctly identifies a fundamental flaw in scale-wise GAN supervision: independent realism doesn't guarantee a coherent trajectory. The proposed solution (a weighted L2 consistency loss in the latent space) is remarkably simple yet highly effective.

  2. Strong Empirical Results and Efficiency: Achieving an FID of 1.56 on ImageNet-256 at 1-NFE is a strong result. The efficiency analysis (Table 5) is a major strength. CAT-H/2 requires ~2.2x less training compute than GAT-XL/2 and ~16.7x less than iMF-XL/2, making it highly practical.

  3. Well-Designed Diagnostic Analyses: The paper doesn't just present a new loss; it thoroughly analyzes why it works. Figure 7 and Table 13 (LPIPS/DINOv2 metrics) provide solid evidence that the regularization effectively enforces semantic and perceptual consistency across generator depths.

Critical weaknesses
  1. Limited Technical Novelty: While effective, the core contribution—adding an intermediate feature alignment loss to the final output—is conceptually similar to deep supervision or intermediate feature matching (e.g., in Distillation or standard GAN literature). The structural novelty is somewhat limited, as the main change from GAT is simply the addition of L_cons and the removal of cross-scale attention in the discriminator.

  2. Comparison Scope: The paper focuses heavily on latent-space SD-VAE generation. The pixel-space experiment (Appendix A.3) is only preliminary (40 epochs). It would be beneficial to see how CAT performs in raw pixel space compared to other pixel-space GANs.

  3. Unaddressed Training Dynamics: Forcing h_k o align with h_K early in training might slow down the learning of coarse structures if the final output is initially noisy. The paper mentions using weaker weights for coarser stages, but a deeper discussion on the training stability or gradient dynamics of this loss would strengthen the paper.

Questions for authors
  1. Choice of L2 Loss: You use an L2 distance in the latent space for L_cons. Did you experiment with perceptual losses (e.g., LPIPS or DINOv2 features) directly as the consistency loss during training? If so, how did they compare to the simple L2 latent loss?

  2. Alignment Strategy: Instead of forcing intermediate features h_k to match the final output h_K, did you consider cross-scale attention mechanisms within the generator (rather than just the discriminator) to share information across scales? Why is the explicit regularization approach preferred over architectural changes?

  3. Scale Weights: You set w_k to decrease toward lower resolutions (1/3, 1/2, 1). Is this heuristic robust across different datasets, or is it highly dependent on the ImageNet class distribution?

  4. Discriminator Masking: In Fig. 4, the block-diagonal mask completely prevents cross-scale interaction. Given that the generator now explicitly aligns scales, is there still a strict need to mask the discriminator, or could a partially masked/cross-scale discriminator now work better without the FID degradation observed in Fig. 8?

4 comments

We thank the reviewer for the constructive questions. The new results support our central claim:

  1. We identify a problem in multi-scale GAN training: per-scale realism does not make the stages depict the same sample, and fixing this with Lcons\mathcal L_{\text{cons}} improves FID at every model size (Point 1).
  2. Both parts of CAT are needed, and no architectural change replaces either (Points 5, 7).
  3. Lcons\mathcal L_{\text{cons}} does not slow early training, and its weights need no tuning (Points 3, 6).

1. W1: Is Lcons\mathcal L_{\text{cons}} conceptually deep supervision or feature matching?

We agree that, in form, Lcons\mathcal L_{\text{cons}} resembles deep supervision and intermediate feature matching. Our contribution is the problem it solves, which we identify in multi-scale GAN training, and a simple yet effective solution to it.

  • We identify that multi-scale adversarial supervision lets later stages redraw the sample. Each stage only has to look real at its own scale, so nothing ties the stages of one forward pass to the same sample. Without Lcons\mathcal L_{\text{cons}}, the intermediate outputs of H/2 are almost as far from their own final output as an unrelated real image is (discrepancy 1.157 vs. 1.188; response to Reviewer vbX5, Point 3).
  • Solving this problem with a simple loss gives large gains at every model size. We trained each size with and without Lcons\mathcal L_{\text{cons}}: FID 5.43→4.06 (B/2, 20 epochs), 2.34→1.93 (M/2, 40 epochs), and 4.35→1.76 (H/2, 40 epochs). Only with Lcons\mathcal L_{\text{cons}} does the larger H/2 outperform M/2, reaching 1.56 FID at 1 NFE after 60 epochs.
  • The simplicity is an advantage: Lcons\mathcal L_{\text{cons}} adds no parameters or sampling cost and leaves the architecture unchanged.

Details: Sec. 6, Table 4.


2. W2: The pixel-space experiment is preliminary. How does CAT compare with pixel-space GANs?

We agree that it is preliminary: it shows that CAT trains in pixel space and is competitive with pixel-space GANs at a much lower cost.

Model, ImageNet-256Generator paramsGFLOPsEpochsFID-50K ↓
pMF-B/16 [A]119M341603.53
CAT-B/16136M24403.54
BigGAN [B]112M59–6.95
GigaGAN [C]569M–4803.45
StyleGAN-XL [D]166M1574–2.30
CAT-H/2 (latent space)960M167 + 310601.56

GFLOPs of the generator per image. For CAT-H/2, the SD-VAE decoder adds 310.

  • CAT-B/16 is close to GigaGAN with a generator about four times smaller and 12 times fewer training epochs. It reaches 3.54 vs. 3.45 FID with 136M vs. 569M parameters and 40 vs. 480 epochs.
  • CAT-B/16 is far cheaper at inference than convolutional GANs. Convolutional GANs run many layers on high-resolution feature maps, whereas CAT-B/16 needs no decoder and uses 16×16 tokens in every layer. It needs 24 GFLOPs, about 65 times fewer than StyleGAN-XL (1574). StyleGAN-XL remains better in FID (2.30), but it relies on classifiers trained with ImageNet labels, both as discriminator features and for classifier guidance, which favors the ImageNet classes. The pretrained encoder in our pixel-space run is self-supervised and uses no labels.
  • Our main latent-space model, CAT-H/2, outperforms all of these models (1.56 FID). Even with the SD-VAE decoder, its 477 GFLOPs are about three times fewer than those of StyleGAN-XL. The models also differ in the pretrained networks they use, so this comparison is only indicative.

Details: Appendix A.7, Table 15.


References

[A] Lu, Yiyang, et al. "One-step latent-free image generation with pixel mean flows." International Conference on Machine Learning (ICML). 2026.

[B] Brock, Andrew, et al. "Large scale GAN training for high fidelity natural image synthesis." International Conference on Learning Representations (ICLR). 2019.

[C] Kang, Minguk, et al. "Scaling up gans for text-to-image synthesis." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

[D] Sauer, Axel, et al. "Stylegan-xl: Scaling stylegan to large diverse datasets." ACM SIGGRAPH 2022 Conference Proceedings. ACM, 2022.

(continued, 2/3)


3. W3: Does aligning to a still-noisy final output slow early training? Training dynamics.

No: Lcons\mathcal L_{\text{cons}} aligns the stages and the final output mutually, and this mutual form keeps training stable without slowing early training; only a one-sided pull toward the final output, the case the reviewer describes, becomes unstable.

  • A one-sided pull toward the still-noisy final output is indeed unstable. With a stop-gradient target, sg(hK)\mathrm{sg}(h_K), which pulls only the intermediate outputs, most H/2 runs diverged early, whereas the same form trains well at B/2.
  • Our mutual form is stable at every model size and even removes an instability at H/2. Lcons\mathcal L_{\text{cons}} is not detached, so its gradient reaches both hkh_k and hKh_K. Without Lcons\mathcal L_{\text{cons}}, H/2 repeatedly enters short ranges of iterations (9 iterations on median) in which the final-scale discriminator loss collapses from 0.59 to about 0.03 and the generator loss doubles; after 20K iterations, there are 292 such sharp drops, increasingly frequent (1.6% of iterations at 20K–30K, 3.5% at 80K–90K and 100K–110K). With Lcons\mathcal L_{\text{cons}}, there are none (training loss curves in Fig. 13).
  • Early training is not slowed: with Lcons\mathcal L_{\text{cons}}, FID matches or beats FID without it from the first evaluation. We trained both:
ModelLcons\mathcal L_{\text{cons}}FID-5K at 4 epochs8 epochs12 epochs20 epochs
B/2✗58.1224.4016.5912.55
B/2✓58.5021.4514.5011.17
H/2✗60.8627.5018.6813.41
H/2✓54.8613.7810.849.43

At B/2, the two match at 4 epochs and the gap opens afterwards; at H/2, Lcons\mathcal L_{\text{cons}} is better from the first evaluation.

Details: Sec. 6, Table 3; Appendix A.13, Fig. 13, Tables 22–23.


4. Q1: Did you try LPIPS or DINOv2 features as the consistency loss?

We did not, because CAT is trained in the SD-VAE latent space: LPIPS and DINOv2 would require decoding every stage at every iteration, and our simple latent L2 loss already improves consistency in both feature spaces.

  • LPIPS and DINOv2 cannot be applied to our outputs directly, because they are latents. Both networks [A, B] take RGB images as input, whereas every stage of CAT outputs an SD-VAE latent [C], and Lcons\mathcal L_{\text{cons}} compares these latents.
  • Using them would require decoding every stage's output at every iteration, which is prohibitively costly. Each iteration would need a forward and backward pass through the VAE decoder for all four stages. We measured a full CAT-B/2 training iteration on a 24 GB GPU: our latent loss trains with up to 320 samples per GPU, whereas an LPIPS or DINOv2 consistency loss fits only 2 (4 with gradient checkpointing of the decoder), i.e., an 80–160× smaller batch. Per sample, training is also 10–14× slower.
  • Our simple latent L2 loss already improves consistency in both feature spaces. With Lcons\mathcal L_{\text{cons}}, the discrepancy drops by 15–22% in LPIPS space and by 10–13% in DINOv2-L space at every model size, together with lower rewriting and higher alignment (Table 18 in Appendix A.10).

Details: Appendix A.10, Table 18.


References

[A] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2018.

[B] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." Transactions on Machine Learning Research (TMLR). 2024.

[C] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.

(continued, 3/3)


5. Q2: Why a regularizer rather than cross-scale attention in the generator?

The misalignment comes from the objective, not the architecture, so an architectural change (e.g., cross-scale attention in the generator, as the reviewer suggests) does not address it; Lcons\mathcal L_{\text{cons}} fixes the objective directly, with no extra parameters or sampling cost.

  • Adding cross-scale attention to the generator, as an example of such a change, leaves the stages as misaligned as without it. We added cross-attention from every block of stages 1–3 to the tokens at the end of all previous stages and trained it without Lcons\mathcal L_{\text{cons}}. Both the discrepancy and FID stay at the level of our model without Lcons\mathcal L_{\text{cons}}, far from our model with it:
Generator (CAT-B/2, 20 epochs)Lcons\mathcal L_{\text{cons}}Discrepancy δˉ\bar\delta ↓FID-50K ↓
Ours✗0.9135.43
Ours + cross-scale attention (+28% parameters)✗0.9295.45
Ours✓0.5604.06
  • This is expected, because the architecture already shares information across scales. All stages run on one token sequence, so later stages already see the full features of earlier ones; what the generator lacks is an objective that asks the stages to depict the same sample, which Lcons\mathcal L_{\text{cons}} provides.

Details: Appendix A.16, Table 26.


6. Q3: Are the stage weights robust across datasets?

The stage weights help slightly, but Lcons\mathcal L_{\text{cons}} works well without them, and among the variants we tested, our form works best:

Consistency penalty (CAT-B/2, 20 epochs)FID-50K ↓
None5.43
Same weight for every stage: ∑k∥hk−hK∥2\sum_k \lVert h_k-h_K\rVert^24.16
Adjacent stages only: ∑k∥hk+1−hk∥2\sum_k \lVert h_{k+1}-h_k\rVert^24.13
Resized outputs only: ∑kwk∥rk(hk)−rk(hK)∥2\sum_k w_k\lVert r_k(h_k)-r_k(h_K)\rVert^24.56
Ours: ∑kwk∥hk−hK∥2\sum_k w_k\lVert h_k-h_K\rVert^2, wk=1/(K−k)w_k=1/(K-k)4.06

rkr_k: resizing to the resolution of stage kk.

  • Without the stage weights, Lcons\mathcal L_{\text{cons}} still gives most of its gain (4.16 vs. 4.06, from 5.43).
  • Other forms of the penalty also improve FID, but ours works best. Tying only adjacent stages gives 4.13, and tying only the resized outputs gives 4.56.
  • The same weights work well on another dataset and in pixel space. We use the same wkw_k and λcons\lambda_{\text{cons}} as in our latent-space ImageNet experiments in two other settings: unconditional FFHQ-256 in latent space, a dataset without class structure, where CAT outperforms GAT [A] (7.76 vs. 9.74); and ImageNet-256 in pixel space, where CAT-B/16 reaches 3.54 FID after 40 epochs.

Details: Sec. 6, Table 3; Appendix A.7, Table 15; Appendix A.8, Table 16.


7. Q4: Is the discriminator mask still needed once Lcons\mathcal L_{\text{cons}} aligns the scales?

Yes: without the mask, FID collapses, and adding Lcons\mathcal L_{\text{cons}} does not recover it (33.80 vs. 7.61 with the mask), even though its stages are as consistent as ours (discrepancy 0.510 vs. 0.564). The unmasked and masked rows are in the submission (Appendix A.9 in the revision), and the partial mask is a new run:

Discriminator (CAT-B/2, 12 epochs)Lcons\mathcal L_{\text{cons}}Discrepancy, stages vs. final output ↓FID-50K ↓
Unmasked: each logit sees all scales
✗0.53747.25
✓0.51033.80
Partial mask: each logit sees its own and all coarser scales
✓0.44114.63
Masked (ours): each logit sees one scale
✗0.8839.60
✓0.5647.61
  • We tested the partial mask that the reviewer suggests, opened toward the final scale. Since the final output is the one used at inference, we relaxed the mask step by step toward it: the logit of each scale also attends to all coarser scales, so the coarsest scale still sees only itself and the final scale sees every scale.
  • The partial mask is better than no mask, but the full mask is still needed. With Lcons\mathcal L_{\text{cons}}, it improves FID over the unmasked discriminator (33.80 → 14.63) yet remains far from the full mask (7.61). We believe this is because it restores scale-specific feedback only partly: the coarsest logit again judges its own scale alone, whereas the finer logits, including the final one, still see the coarser scales and can rely on cross-scale compatibility.

Details: Sec. 6, Fig. 8; Appendix A.9, Table 17.

We hope these clarifications and new results address the reviewer's concerns, and we are happy to discuss any remaining questions.


References

[A] Hyun, Sangeek, et al. "Scalable GANs with transformers." International Conference on Machine Learning (ICML). 2026.

Dear Reviewer sd52,

Thank you again for your review. As only a short time remains in the discussion period, we would be grateful if you could take a look at our response. In short:

  • Novelty (W1). Our contribution is the problem we identify, that per-scale realism leaves the stages of one sample unaligned, and a simple solution that improves FID at every model size.
  • Pixel space (W2). CAT-B/16 is close to GigaGAN (3.54 vs. 3.45) with a four times smaller generator, and it needs about 65 times fewer GFLOPs than StyleGAN-XL (24 vs. 1574).
  • Training dynamics (W3). Lcons\mathcal L_{\text{cons}} does not slow early training, and it removes an instability at H/2.
  • Cross-scale attention (Q2). Adding it to the generator leaves the stages unaligned (FID 5.45 vs. 4.06 with Lcons\mathcal L_{\text{cons}}).
  • Discriminator mask (Q4). A partial mask is better than no mask (14.63 vs. 33.80) but far from the full mask (7.61).
  • Q1 and Q3 are answered with new measurements and ablations.

We would be happy to answer any remaining questions.

Reviewer_vbX5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GAN model can use multi-scale adversarial training to improve quality of generated images by provide intermediate feedback for generators. However, the paper recognised an important limitation that generators normally optimise the image on its own resolution. This lead to the intermediate output does not align with the final output from the same generated sample. This inconsistency make the later stage of resolution can just rewrite the images without using the input from intermediate layer. Motivated on that, the paper prose new GAN method called CAT (Cross-scale Aligned Transformer) for multi-scale GAN. The method CAT use a shared discriminator architecture and a consistency loss to regularise the consistency between final outputs and intermediate outputs built as weighted mean square error across scales. The method can improve quality of generated images with lower FID-50K scores on ImageNet-256 dataset with efficient computations compare with one step generation methods.

Critical strengths

The problem paper try to solve is clear, which is inconsistency between intermediate stages of the generator in GAN with the evidence on metric can be evaluated can compared on consistency in the Figure 5.

The method is simple with a regularisation loss yet effective.

The experimental results are convincing with supports from ablation studies.

Critical weaknesses

The Table 3 Show to support abstract that regularisation loss help CAT-H2 improved FID from 3.45 to 1.59. However, look closer, I recognise that G-H/2 has worse FID compared to smaller one G-M/2 and G-B/2 when they all do not use consistency loss. This can reveal that the model G-H/2 maybe are not optimised and consistency loss that make the unstable in the minimax game loss more stable rather than contributing by make consistency between intermediate layer better.

The Table 6 shows “the gain does not simply come from increasing adversarial model capacity”. However, the version of CAT without consistency loss seem not included. In addition, the GAT model with regularisation loss is not included also.

The metrics for consistency do not include reference value when the image is the same to see if the metrics are high in this case. In addition, the metric can be high with blurry images, then in this case, when resolution change, the image still stay stable since the number of pixels need to present images is low. In this case, the consistency loss can encourage GAN model generate blurry images, which can conflict with what the GAN should learn and generate.

The method CAT should compare with SOTA generative models to see position of CAT in the field instead of only comparing with 1-NFE setting. In this case, the paper should discuss trade-off of cost-utility of all methods. This can show when CAT should be used to save computational coset but keep a reasonable image quality in application.

The method CAT has overlap ideas on consistency between sub generator in paper [A], which can limit the novelty of the paper.

[A] Lin J, Zhang R, Ganz F, Han S, Zhu JY. Anycost gans for interactive image synthesis and editing. In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2021 Jun 20 (pp. 14981-14991). IEEE.

Questions for authors

The Table 3 and the gain in CAT-H/2 should be checked and verified to see if the contribution come from consistency improvements. Can the authors clarify that?

Can the authors provide comparisons on claim the gain does not simply come from increasing adversarial model capacity” in Table 6.

Can the paper provide comparisons with SOTA methods with trade off between cost and utility of them.

Can the method be encouraged to generate blurry images? Does the blur images have better value in consistency metrics provided by the paper?

Can the authors compare on abstract methodology design, results with the paper [A] and provide the key difference of CAT to the method in the GAN model in [A]?

7 comments

We thank the reviewer for the thoughtful comments. The new results directly support our central claim:

  1. Lcons\mathcal L_{\text{cons}} both improves consistency and stabilizes training. It lowers FID at B/2 and M/2, where training is already stable, and it removes an instability at H/2 (Point 1).
  2. The gain over GAT [A] comes from supervision, not capacity. With the same architecture, changing only how the intermediate outputs are supervised improves FID from 9.53 to 4.06 (Point 2).
  3. Lcons\mathcal L_{\text{cons}} does not favor blur. Its outputs keep the fine detail of real images, and blurring only worsens FID (Point 3).

1. W1 & Q1: Does the H/2 gain come from better consistency, or from stabilizing the minimax game?

We first note that the abstract reports 4.35 → 1.76 for the matched 40-epoch H/2 comparison (Table 3 of the submission, Table 4 in the revision), and 1.56 after 60 epochs.

Both effects are present: Lcons\mathcal L_{\text{cons}} stabilizes H/2, as the reviewer suggests, and it also improves consistency, which lowers FID even where training is already stable.

  • Stabilization at H/2. Without Lcons\mathcal L_{\text{cons}}, the adversarial balance at H/2 is repeatedly lost: for a few iterations, the final-scale discriminator separates real from generated samples almost perfectly (its loss falls from 0.59 to 0.03) and the generator loss doubles. This happens 292 times, increasingly often during training, and never with Lcons\mathcal L_{\text{cons}} (training loss curves in Fig. 13). Together with its worse FID than M/2 (4.35 vs. 2.34 at 40 epochs), this points to instability.
  • Consistency at every size. Without Lcons\mathcal L_{\text{cons}}, B/2 and M/2 train stably (the balance is lost only 2 and 0 times), so there is no instability to remove. Lcons\mathcal L_{\text{cons}} still improves their FID (5.43→4.06 for B/2 at 20 epochs; 2.34→1.93 for M/2 at 40 epochs) together with every consistency metric, so stabilization alone cannot explain the gain.

Details: Appendix A.13, Fig. 13, Tables 22–23.


2. W2 & Q2: Does the gain in Table 6 come from model capacity? CAT without Lcons\mathcal L_{\text{cons}} and GAT with Lcons\mathcal L_{\text{cons}} are missing.

Our claim compares CAT with GAT at the same or comparable size, where the two differ only in how the intermediate outputs are supervised, and this difference alone yields a large gap. We add both requested rows and state this comparison explicitly in Sec. 6 and Appendix A.15.

ModelEpochsLcons\mathcal L_{\text{cons}}FID-50K
Same size
GAT-B/220✗9.53
GAT-B/220✓7.76
CAT-B/220✗5.43
CAT-B/220✓4.06
Larger models
GAT-XL/240✗2.96
CAT-M/240✗2.34
CAT-M/240✓1.93
CAT-H/240✗4.35
CAT-H/240✓1.76

GAT values from Table 6 of the submission (Table 7 in the revision, 20 epochs) and the GAT paper (40 epochs, its best reported value).

  • Same size. GAT-B/2 and CAT-B/2 share the generator, the discriminator's transformer blocks and head, and the training recipe, apart from one minor hyperparameter (the gradient-penalty strength); the discriminators differ only in input and output. Scale-wise supervision alone improves FID from 9.53 to 5.43, and Lcons\mathcal L_{\text{cons}} improves it further to 4.06.
  • Larger models. CAT-H/2 outperforms GAT-XL/2, which has a comparable total (G+D) parameter count (1.76 vs. 2.96 at 40 epochs; 2.55 vs. 4.02 at 20). Both parts of the supervision are needed: without Lcons\mathcal L_{\text{cons}}, CAT-H/2 is worse than CAT-M/2 (4.35 vs. 2.34), so a larger model alone does not explain the gain.
  • GAT with Lcons\mathcal L_{\text{cons}}. Adding Lcons\mathcal L_{\text{cons}} to GAT-B/2, with all other settings unchanged, also improves alignment (discrepancy 0.583 → 0.544) and FID-50K (9.53 → 7.76). It still falls far short of CAT-B/2 (4.06), and even of CAT-B/2 without Lcons\mathcal L_{\text{cons}} (5.43). We attribute the remaining gap mainly to the weaker adversarial signal of GAT's discriminator, which judges all stages jointly instead of giving each stage its own scale-specific feedback.

Details: Sec. 6, Table 7; Appendix A.15, Table 25.


References

[A] Hyun, Sangeek, et al. "Scalable GANs with transformers." International Conference on Machine Learning (ICML). 2026.

(continued, 2/3)


3. W3 & Q4: Do the consistency metrics have a reference value, and can Lcons\mathcal L_{\text{cons}} favor blurry images?

Lcons\mathcal L_{\text{cons}} does not make the outputs blurry, and the reference values confirm the problem it solves.

(a) The FID gain of Lcons\mathcal L_{\text{cons}} is not a blur effect.

Model (epochs)Lcons\mathcal L_{\text{cons}}FIDFID after Gaussian blur (σ\sigma = 1 px)HF, latentHF, pixel
B/2 (20)✗ / ✓5.43 / 4.0610.93 / 9.270.480 / 0.4810.0410 / 0.0414
M/2 (40)✗ / ✓2.34 / 1.937.32 / 6.550.480 / 0.4840.0406 / 0.0406
H/2 (40)✗ / ✓4.35 / 1.769.67 / 6.600.481 / 0.4690.0404 / 0.0385
Real images (VAE reconstruction)0.4840.0411

HF: share of energy in the upper half of the frequencies, i.e., the fine detail that halving the resolution removes.

Blur worsens FID by about 5 for every model, whereas Lcons\mathcal L_{\text{cons}} improves it, and HF stays at the real-image level.

(b) Reference values for the discrepancy, and what they show. δˉ\bar\delta (δk\delta_k of the paper, mean over stages) measures how far intermediate outputs are from the final one:

Stages compared with the final outputδˉ\bar\delta
Identical to it0
A real image resized to each stage's scale0.714
An unrelated real image1.188
B/2, w/o → w/ Lcons\mathcal L_{\text{cons}}0.913 → 0.560
H/2, w/o → w/ Lcons\mathcal L_{\text{cons}}1.157 → 0.627
  • Without Lcons\mathcal L_{\text{cons}}, the stages drift away from the final sample. B/2 is farther than a resized real image, and H/2 is nearly as far as an unrelated image: later stages largely redraw the sample. Lcons\mathcal L_{\text{cons}} brings both below the real-image value. Blurring a real image also lowers its value (0.714 → 0.558 at σ\sigma = 2 px), which is why (a) checks blur directly.
  • Falling below the real-image value is expected and helpful. A resized real image holds only coarse content, but our intermediate outputs (hkh_k, on the full latent grid) can already contain finer detail of the same sample, unseen by the discriminator at that scale. Tying only what the discriminator sees, i.e., comparing the two outputs after resizing both to the scale of stage kk, gives worse FID (4.56 vs. 4.06; B/2, 20 epochs).

Details: Appendix A.6, Table 14; Appendix A.12, Tables 20–21.


4. W4 & Q3: How does CAT compare with multi-step state-of-the-art models in cost and quality?

CAT approaches strong multi-step diffusion models in FID with a single network evaluation, with 178–209× faster generator sampling per image and training for far fewer epochs. Multi-step models still reach a lower FID; CAT suits efficient one-step generation and GAN applications such as semantic editing.

ModelNFEEpochsFID-50KGenerator time / imageGFLOPs / image
DiT-XL/2 [A]250×214002.272.72 s59,300
SiT-XL/2 [B]250×214002.062.72 s59,300
SiT-XL/2 + REPA [C]250 (+172)8001.422.32 s50,000
SiT-XL/2 + REPA, 50 steps50 (+34)8001.940.46 s9,960
CAT-H/2 (ours)1601.560.013 s167

One TITAN RTX, batch 64, fp16; shared VAE decoding (0.019 s) excluded. (+n): extra guidance evaluations. GFLOPs: per-forward GFLOPs × evaluations. We sampled the 50-step REPA ourselves.

Details: Appendix A.4, Table 12.


References

[A] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

[B] Ma, Nanye, et al. "SiT: Exploring flow and diffusion-based generative models with scalable interpolant transformers." European Conference on Computer Vision (ECCV). Springer, 2024.

[C] Yu, Sihyun, et al. "Representation alignment for generation: Training diffusion transformers is easier than you think." International Conference on Learning Representations (ICLR). 2025.

(continued, 3/3)


5. W5 & Q5: How does CAT differ from AnyCost GAN?

AnyCost GAN [A] keeps cheaper sub-generators consistent with a pre-trained generator, whereas CAT learns a consistent generation trajectory from scratch to improve the final sample.

  • AnyCost GAN. It starts from a pre-trained StyleGAN2 [B] and distills its generators toward a pre-trained generator (Eq. 5; Appendix B.1), kept fixed in the official code. The stated purpose is consistent previews (Sec. 3.2), not image quality.
  • CAT. Without a teacher, it aligns each stage with the generator's own final output. We show that scale-wise realism leaves the stages of a one-step transformer untied, and that tying them with Lcons\mathcal L_{\text{cons}} improves the final sample (Points 1–3).
  • A different regime. We fine-tuned AnyCost GAN's multi-resolution generator (FFHQ-256, no distillation) with and without Lcons\mathcal L_{\text{cons}}. Lcons\mathcal L_{\text{cons}} helps slightly (FID 4.81→4.60 at 256 px), but the stages are already consistent without it (PSNR 23.2 and 30.7 dB at 64 and 128 px). We believe this is because each convolutional stage acts locally and adds detail at its own resolution, whereas every transformer block attends over the whole image and can rewrite it; our H/2 without Lcons\mathcal L_{\text{cons}} is nearly at the unrelated-image level (Point 3).

Details: Appendix A.17, Table 27.

We hope these clarifications and new results address the reviewer's concerns, and we are happy to discuss any remaining questions.


References

[A] Lin, Ji, et al. "Anycost gans for interactive image synthesis and editing." 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021.

[B] Karras, Tero, et al. "Analyzing and improving the image quality of stylegan." 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.

Reviewer_yFFtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work follows GAT, a one-step transformer GAN operating in the SD-VAE latent space. The generator emits four intermediate latents at different scales; each is scored separately by a shared discriminator. The authors note that these intermediates drift from the final output and that the drift increases over training. They mitigate this with a weighted L2 loss that pulls each intermediate toward the final latent. On ImageNet-256, H/2 improves FID from 4.35 to 1.76 at 40 epochs, and CAT-H/2 reaches 1.56 at 60 epochs with 1 NFE, beating GAT-XL/2 (2.18) and iMF-XL/2 (1.72, 800 epochs) with much less compute. Ablations include the loss weight, discriminator size, a scale-aggregated discriminator, FFHQ-256, pixel space, and recall.

Critical strengths
  1. Strong, cheap results. FID 1.56 at 1 NFE / 60 epochs beats GAT-XL/2 and the 800-epoch iMF-XL/2 with only a Base discriminator. Gains hold across B/M/H and are large for H/2; FFHQ and pixel space agree; code is out. If it reproduces, this is a real step for one-step generation.

  2. Clear diagnosis. The three consistency metrics make the problem concrete, it is confirmed in LPIPS/DINOv2 space, the scale-aggregated control is informative, and recall rules out a diversity trade-off.

Critical weaknesses
  1. Attribution. (a) The scale-wise baseline already improves GAT-B/2 from 9.53 (Table 6) to 5.43 (Table 3) at 20 epochs—more than L_cons—but replacing GAT's joint multi-scale discrimination with separate per-scale discrimination is neither claimed nor ablated. (b) Table 12: the scale-aggregated discriminator gives much better consistency (0.537 vs. 0.883) and 5× worse FID, so consistency isn't driving quality; un-regularized H/2 worse than M/2 suggests L_cons is a stabilizer. No control separates alignment from residual suppression (e.g., penalize only update size per stage, or L2 to a detached random target).

  2. Statistics and scope. Single-seed FIDs, with several small key gaps (3.27 vs. 3.00; 4.06 vs. 4.45). Only ImageNet-256 at scale; recall but no precision; unclear whether the final latent is detached in the consistency loss—this affects the mechanism and whether the loss can pull the final output toward coarser stages.

Questions for authors
  1. Is the final latent detached (stop-gradient) in the consistency loss? If not, what happens to FID and to high-frequency detail when it is?

  2. Can you add a matched control—penalize only the size of each stage-to-stage update, or regress intermediates toward a detached permuted/low-passed version of the final latent? If those work equally well, the story is stabilization, not alignment.

  3. Are the 5.43 (Table 3) and 9.53 (Table 6) B/2 numbers from identical 20-epoch settings? If so, separate per-scale discrimination should be claimed and analyzed as a contribution. Does L_cons also help on top of GAT's original supervision?

  4. All FIDs are single runs. Could you report mean ± std over ≥3 seeds for the B/2 and M/2 rows in Tables 3–4? Several of the gaps there (3.27 vs. 3.00; 4.06 vs. 4.45) are small enough to be within run-to-run noise.

  5. Why is un-regularized H/2 worse than M/2—instability or slow convergence? Does a bigger discriminator or lower lr fix the baseline?

All the questions are clarification requests.

4 comments

We thank the reviewer for the careful reading and precise suggestions. The new results directly support our central claim:

  1. Scale-wise supervision provides per-scale realism, but not cross-scale consistency. It improves over the joint supervision of GAT [A] by a wide margin (9.53→5.43), yet the intermediate outputs are not aligned with the final output, and later stages rewrite them (Points 1, 5).
  2. Lcons\mathcal L_{\text{cons}} adds the missing cross-scale consistency. On top of this much stronger baseline, it lowers FID at every model size (by 25% at B/2 and 60% at H/2), and a random-target control falls short of it, so the gain comes from aligning the outputs of the same sample (Points 4, 5).
  3. Consistency regularization also stabilizes generator scaling. Without Lcons\mathcal L_{\text{cons}}, H/2 is worse even than M/2; with it, the larger model is better and reaches 1.56 FID at 1 NFE (Points 2, 3).

1. W1(a) & Q3: Is the gain over GAT due to scale-wise supervision rather than Lcons\mathcal L_{\text{cons}}? Does Lcons\mathcal L_{\text{cons}} help GAT?

Scale-wise supervision alone already beats GAT at equal capacity (9.53→5.43), and Lcons\mathcal L_{\text{cons}} improves this much stronger baseline further (5.43→4.06).

  • Same setting. Both models share the generator, the discriminator's transformer blocks and head, and the training recipe; we only tuned the gradient-penalty strength. The discriminators differ only in input and output: GAT concatenates its noise-perturbed levels along channels and outputs one logit, whereas we concatenate the resized scales along the token axis and output one logit per scale (340 vs. 256 tokens).
  • Scale-wise judgment matters. Letting all scales attend to each other (scale-aggregated) gives much worse FID (47.25 vs. 9.60, 12 epochs).

We had not emphasized this because scale-wise supervision is established (AnyCost GAN [B], GigaGAN [C]), but the revision now states it in the text of Sec. 6 (with Table 7) and in Appendix A.15.

Lcons\mathcal L_{\text{cons}} on GAT. Adding Lcons\mathcal L_{\text{cons}} to GAT-B/2, with all else unchanged, also improves alignment (discrepancy 0.583 → 0.544) and FID-50K (9.53 → 7.76), so its benefit is not specific to our supervision. It still falls far short of CAT (4.06); we attribute the remaining gap mainly to the weaker adversarial signal of GAT's discriminator, which judges all stages jointly.

Details: Sec. 6, Table 7; Appendix A.15, Table 25.


2. W2 & Q1: Is the final latent detached, and is high-frequency detail lost?

The final latent is not detached: the gradient reaches both hkh_k and hKh_K (Sec. 6, Table 3). Both forms align the stages, but only the symmetric form also stabilizes training, which is why we use it.

FID-50KCAT-B/2 (20 ep)CAT-H/2 (20 ep)
No Lcons\mathcal L_{\text{cons}}5.436.86
Lcons\mathcal L_{\text{cons}}, no stop-gradient (ours)4.062.55
Lcons\mathcal L_{\text{cons}} with sg(hK)\mathrm{sg}(h_K)3.84diverges
  • Alignment. At B/2, where training is stable, the stop-gradient form gives a similar gain (3.84 vs. 4.06): alignment alone improves FID.
  • Stabilization. At H/2, most stop-gradient attempts diverged within the first few iterations, while the symmetric form is stable at every size. We believe the two-sided pull helps, since early on a one-sided pull drives every intermediate output toward a still-noisy final output.
  • The final output keeps its high-frequency detail. We measure the finest frequency band of the final latent: the component removed when the 32×32 latent is average-pooled to 16×16, i.e., detail that exists only at the output resolution. Its RMS is 0.495 with Lcons\mathcal L_{\text{cons}}, 0.493 with the stop-gradient, and 0.492 without Lcons\mathcal L_{\text{cons}}, vs. 0.496 for real latents (Base model, B/2, 20 epochs).

Details: Sec. 6, Table 3; Appendix A.12, Tables 20–21.


References

[A] Hyun, Sangeek, et al. "Scalable GANs with transformers." International Conference on Machine Learning (ICML). 2026.

[B] Lin, Ji, et al. "Anycost gans for interactive image synthesis and editing." 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021.

[C] Kang, Minguk, et al. "Scaling up gans for text-to-image synthesis." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

(continued, 2/3)


3. W1(b) & Q5: Is Lcons\mathcal L_{\text{cons}} a stabilizer? Why is the unregularized H/2 worse than M/2?

Alignment improves FID at every model size, including B/2 and M/2 where training is already stable. At H/2, Lcons\mathcal L_{\text{cons}} additionally removes an instability, and only with it does scaling the generator pay off (H/2 vs. M/2: 1.76 vs. 1.93 with Lcons\mathcal L_{\text{cons}}, 4.35 vs. 2.34 without).

  • Alignment at every size. Without Lcons\mathcal L_{\text{cons}}, B/2 and M/2 train stably (only 2 and 0 of the sharp drops described next). Lcons\mathcal L_{\text{cons}} still improves their FID and every consistency metric, so stabilization cannot explain the gain; it comes from same-sample correspondence (Point 4).
  • Additional stabilization at H/2. Without Lcons\mathcal L_{\text{cons}}, the adversarial balance at H/2 is repeatedly lost: for a few iterations, the final-scale discriminator separates real from generated samples almost perfectly (its loss falls from 0.59 to 0.03) and the generator loss doubles. This happens 292 times, increasingly often during training, and never with Lcons\mathcal L_{\text{cons}} (training loss curves in Fig. 13).
  • Q5. Without Lcons\mathcal L_{\text{cons}}, H/2 is much worse even than M/2 (4.35 vs. 2.34 at 40 epochs), which, with these increasingly frequent sharp drops, points to instability. With Lcons\mathcal L_{\text{cons}}, the same H/2 is the better model, with the Base discriminator and all hyperparameters unchanged. Lacking time and compute, we could not retrain the unregularized H/2 with a larger discriminator or lower learning rate.

Details: Appendix A.13, Fig. 13, Tables 22–23.


4. W1(b) & Q2: A control that separates alignment from residual suppression

The gain comes from tying the stages of the same sample: with a random target, the same penalty either diverges or gives a much weaker effect. This is the control the reviewer proposed. Penalties are summed over k<Kk<K with the normalization of Eq. (3); wk=1/(K−k)w_k=1/(K-k) are the stage weights, π\pi is a random permutation of the batch that pairs each sample with a different one, and rkr_k is the area resizing to the resolution of scale kk (Sec. 3).

Penalty (B/2, 20 epochs)FID-50KDiscrepancy ↓Rewrite ↓Alignment ↑
None5.430.9130.8490.477
Ours: ∑kwk∥hk−hK∥2\sum_k w_k\lVert h_k-h_K\rVert^24.060.5600.4920.690
Random target, detached: ∑kwk∥hk−sg(hKπ)∥2\sum_k w_k\lVert h_k-\mathrm{sg}(h_K^{\pi})\rVert^2diverges–––
Random target: ∑kwk∥hk−hKπ∥2\sum_k w_k\lVert h_k-h_K^{\pi}\rVert^24.780.7960.6600.556
Update size only: ∑k∥hk+1−hk∥2\sum_k \lVert h_{k+1}-h_k\rVert^24.13–––
Low-passed target: ∑kwk∥rk(hk)−rk(hK)∥2\sum_k w_k\lVert r_k(h_k)-r_k(h_K)\rVert^24.56–––

Consistency metrics as in the paper, averaged across scales.

  • Random target. With a detached random target, training diverges. Without detaching, training succeeds: pulling toward another sample still limits how far each trajectory drifts, so the stages align partly, but far less than with the sample's own target, and FID improves only about half as much (5.43 → 4.78, vs. 4.06 for ours).
  • Update size and low-passed target still tie the stages of the same sample, so they are variants of our method rather than controls without alignment. In our generator, hK−hkh_K-h_k is the sum of the later stage-to-stage updates, so penalizing each update also pulls all stages of the same sample together and performs like ours (4.13 vs. 4.06). The low-passed target ties only the coarse content of the same sample and helps less (4.56). Only the random target removes the correspondence.

Details: Sec. 6, Table 3.


5. W1(b): The scale-aggregated discriminator is more consistent but worse in FID. Does consistency drive quality?

We agree that consistency alone does not determine quality, and this is the central claim of the paper: per-scale realism and cross-scale consistency are complementary, and both are needed. The scale-aggregated discriminator is more consistent but judges cross-scale compatibility instead of each image's realism (Fig. 8, Appendix A.9), which costs far more in FID than consistency gains. With either discriminator, more consistency still improves FID: Lcons\mathcal L_{\text{cons}} lowers it from 47.25 to 33.80 with the scale-aggregated discriminator and from 9.60 to 7.61 with ours (12 epochs). CAT therefore keeps the scale-wise realism signal and adds consistency in the generator, improving FID at every model size (e.g., 5.43→4.06 at B/2, 4.35→1.76 at H/2).

Details: Sec. 6, Fig. 8; Appendix A.9, Table 17.

(continued, 3/3)


6. W2 & Q4: Single-seed FIDs with small gaps

The seed-to-seed variation of CAT-B/2 (3.98 ± 0.09 over three seeds) is far smaller than the gain from Lcons\mathcal L_{\text{cons}} (5.43→4.06). The M/2 gap at 20 epochs (0.27) is small, and we do not claim significance for it; the gap grows with training (0.41 at 40 epochs) and with model size (2.59 at H/2, 40 epochs).

Details: Appendix A.14, Table 24.


7. W2: Precision is not reported

Lcons\mathcal L_{\text{cons}} improves precision or recall at every model size without lowering the other, so its gain is not a fidelity–diversity trade-off.

Model (epochs)Precision w/o → w/Recall w/o → w/
B/2 (20)0.772 → 0.7890.500 → 0.501
M/2 (40)0.807 → 0.8080.524 → 0.545
H/2 (40)0.789 → 0.8080.474 → 0.557

Mean over three sets of 50K samples; the recall table in the appendix is updated accordingly.

Details: Appendix A.11, Table 19.


8. W2: Only ImageNet-256 at scale

CAT's advantage is not specific to ImageNet latents.

  • On FFHQ-256, CAT improves over GAT (7.76 vs. 9.74 FID).
  • In pixel space, CAT-B/16 matches pMF-B/16 [A] with 4× fewer epochs (3.54 at 40 epochs vs. 3.53 at 160).

We did not run higher resolutions within the rebuttal period.

Details: Appendix A.7, Table 15; Appendix A.8, Table 16.

We hope these clarifications and new results address the reviewer's concerns, and we are happy to discuss any remaining questions.


References

[A] Lu, Yiyang, et al. "One-step latent-free image generation with pixel mean flows." International Conference on Machine Learning (ICML). 2026.

Dear Reviewer yFFt,

Thank you again for your review. As only a short time remains in the discussion period, we would be grateful if you could take a look at our response. In short:

  • Attribution (W1a, Q3). Scale-wise supervision alone improves over GAT (9.53 → 5.43), and Lcons\mathcal L_{\text{cons}} improves it further (4.06). Lcons\mathcal L_{\text{cons}} also helps GAT (9.53 → 7.76).
  • Alignment or stabilization (W1b, Q2, Q5). The gain comes from aligning each stage with the final output of the same sample: our loss reaches 4.06, whereas the same penalty toward another sample's output reaches only 4.78, or diverges when detached. The gain also holds at B/2 and M/2, where training is already stable.
  • Stop-gradient (Q1). The final output is not detached, and it keeps its high-frequency detail.
  • Statistics (W2, Q4). Three seeds give 3.98 ± 0.09, and we added precision.

We would be happy to answer any remaining questions.

Comments on the paper

1 comment

We sincerely thank all reviewers and the area chair for their time and for the careful, constructive reviews. The comments helped us improve the paper substantially.

We have uploaded a revised version. Each response points to the parts of the revision related to it (the "Details" and "Revised in" lines), and all added or modified text in the revised PDF is shown in blue. The main additions are:

  • Motivation and presentation. A toy experiment in Fig. 2, a dedicated section for the ImageNet observations, and a formal statement of the model and objective (Secs. 1, 3, 4).
  • Variants of the consistency loss. Stop-gradient, random-target, adjacent-stage, and resized-output variants, which show that the gain comes from aligning the outputs of the same sample (Sec. 6, Table 3).
  • Alignment vs. stabilization. Training-stability analysis across model sizes and seed variability (Appendix A.13, A.14).
  • Blur and reference values. Sharpness analysis, precision and recall, and reference values for the discrepancy (Appendix A.6, A.11, A.12).
  • Comparisons. Lcons\mathcal L_{\text{cons}} on GAT, cross-scale attention in the generator, AnyCost GAN, pixel-space GANs, and multi-step models (Appendix A.4, A.7, A.15–A.17).

We would be glad to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.