Cross-Scale Aligned Supervision for Training GANs

Modern GANs often use multi-scale adversarial supervision to provide direct realism feedback to intermediate generator outputs. However, we identify an important limitation of standard scale-wise supervision: each intermediate output is independently optimized for realism at its own resolution, without explicitly coordinating outputs across generator stages. As a result, realistic intermediate outputs need not remain aligned with the final output from the same generated sample. This motivates treating per-scale realism and cross-scale consistency as complementary objectives for multi-scale adversarial generation. Based on this observation, we propose CAT, a Cross-scale Aligned Transformer, which preserves scale-wise adversarial supervision for direct per-scale realism feedback while introducing a simple generator-side consistency regularization to align intermediate outputs with the final output. We further show that jointly exposing multiple scales to the discriminator improves cross-scale consistency but substantially degrades generation quality, supporting our design of enforcing cross-scale consistency separately on the generator side. In a matched 40-epoch CAT-H/2 comparison on ImageNet-256, the proposed consistency regularization improves FID-50K from 4.35 to 1.76. CAT-H/2 achieves an FID-50K of 1.56 with a single generator evaluation after only 60 training epochs.