- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This is an interesting paper which studies the dynamics of latent space in drifting models. It is well-known that drifting works far better in latent space than in raw pixel space. This work studies the impact of training the encoder together with training the drifting directly. This is done by training a discriminative loss through Eq. (7), then alternating training of the generative and discriminative parts of the model. It's important to note that the drifting affects the dynamics of the encoder, as the discrimination is done between the generated samples and the true ones, so this is truly a novel framework. Some empirical studies are done showing the usefulness of the framework, in particular demonstrating that the method works considerably better than simply training directly in pixel space.
Critical strengths
This is a very reasonable thing to do, the results are broadly positive, and it is certainly important to obtain a detailed understanding of how the latent space interacts with drifting dynamics, which this work does.
Critical weaknesses
My primary concern is that it is not clear to me that the work here really demonstrates that this method is significantly better than using an existing pretrained encoder (which is standard in the drifting literature, and is thus the more natural baseline); it seems to me that really one should be comparing to that rather than comparing to direct pixel space comparisons. Some work towards this is done in the empirical section, but in my eyes this should be argued for slightly more forcefully and clearly than it currently is, which at the moment is really only done in the second experiment in Section 5.1. I would be happy to increase my rating if this could be strengthened.
Questions for authors
-
Suggestion for further development: It seems that Section 5.1 is the primary comparison between the new method and drifting with a fixed pretrained encoder. Here indeed it is shown in Figure 2 that the new method is slightly better on FID, and also exhibits lower discriminative loss (and correspondingly higher discriminative accuracy). I think it would be good to strengthen this part. Here I have a few concrete suggestions: 1a. It would be good to show visually some samples from drifting with a frozen encoder and drifting with the new method (i.e. add the frozen encoder samples to Figure 3); if these look very different to the eye then this would be strong evidence for the superiority of the new method, perhaps more convincing than the marginal gain on FID on CIFAR10 in Figure 2. 1b. It is not entirely clear to me how to interpret that the KDE accuracy in the right panel of Figure 2 is highest for new method; I understand that the generator and discriminator are playing somewhat adversarial roles, but one might have expected that if this worked perfectly, then eventually the discriminator would fail to tell apart the generated and data images, which is not what we see. Could the authors spend a bit of time explaining this? I.e. is this expected from experience with GANs? 1c. In general any other ablations etc. that could demonstrate the superiority of this method over a frozen encoder would be appreciated. E.g. perhaps one could do experiments in section 5.2 where you train the encoder and the drifting mode separately, and also train them together using the method of the paper while keeping the total compute fixed, and demonstrate that the new method is better? This would conclusively show the benefits arising from the new paradigm.
-
Clarification request: It's instructive to study a simple synthetic dataset as in Figure 1. Nevertheless, in Figure 1 (b), the "vanilla" raw space drifting baseline seems to really do very poorly to the extent that one might worry that it is actually mistuned -- could you elaborate on exactly how the reference field is constructed? (Appendix A.1 states how many samples are used but doesn't make quite clear what this reference field means) In particular, what should we interpret from the fact that the alignment gets smaller and smaller as the training proceeds? (Is the generator distribution getting farther and farther away from the target? Surely not -- but then is this alignment really the right measure?)
5 comments
Response to Reviewer u28d [2/3]
Q1b Explain why higher KDE discrimination accuracy accompanies better generation, given the expected indistinguishability at adversarial equilibrium.
We thank the reviewer for this insightful question about the interpretation of KDE discrimination accuracy. Higher KDE discrimination accuracy during training is consistent with better generation because it reflects the representation's ability to detect remaining discrepancies, rather than the indistinguishability expected only at equilibrium. The discriminator fails to tell apart generated and data images only at the global optimum, where the generated distribution equals the data distribution [2]. During training, the discriminator keeps identifying the remaining discrepancy between the two distributions, and this discrepancy provides the learning signal for the generator. BigGAN [3] reports that the discriminator separates real and generated training images almost perfectly throughout successful training. The authors regard this as in line with the role of the discriminator, which is to distill the training data and provide a useful learning signal for the generator.
The near-chance KDE accuracy of pixel-space drifting in Figure 2 reflects a weak ability of the pixel representation to detect the discrepancies that remain. Its FID stays poor, so the generated distribution remains far from the data distribution. Each KDE accuracy in Figure 2 is computed in the representation used by the corresponding method, so it depends on how well that representation detects the remaining discrepancy. In pixel space, the kernel cannot separate real and generated samples even though the two distributions differ substantially, and the accuracy stays close to chance.
Compared with pixel space, the persistent representation detects the discrepancies that remain for the current generator. This gives the drifting field a discriminative signal, and the FID improves substantially. We measure generation quality by FID and use KDE accuracy as a diagnostic of how well a representation detects the remaining discrepancy. We have added this explanation to the revised version.
Response to Reviewer u28d [3/3]
Q2 Clarify the synthetic reference field, the tuning of pixel-space Drifting Models, and the interpretation of decreasing field alignment during training.
We thank the reviewer for this clarification request. The reference field is constructed using the informative semantic coordinates, the raw-space baseline uses matched training settings, and decreasing alignment reflects inaccurate drift directions rather than increasing distance from the target distribution. We explain these points below.
Construction of the reference field. The reference field is the drift field that the same KDE estimator produces when its kernel uses only the two semantic coordinates. The eight Gaussian modes are defined in these coordinates, and the 30 nuisance coordinates carry no information about the target distribution. For each of the 1,024 generated evaluation samples , we compute
,
where extracts the two semantic coordinates, are 4,096 fresh samples from the target distribution, are 4,096 fresh samples from the current generator, and the weights are softmax normalized over with the training temperature . Both anchor sets are drawn independently of the evaluation samples. is therefore the drift field under an ideal representation that keeps the semantic coordinates and discards the nuisance coordinates, estimated from 4,096 real and 4,096 generated anchors. The estimated field uses the same formula on the 32-dimensional observations, with 1,024 real samples and the 1,024 generated samples themselves as anchors (each sample excluded from its own repulsion term). Its kernel distances are computed in the raw space for the raw-space baseline and in the learned representation for our method. We project this field onto the two semantic coordinates and report the cosine similarity with at each sample, averaged over the 1,024 samples.
Tuning of the raw-space baseline. The raw-space baseline and our method share the generator, optimizer, learning rate, batch size, number of steps, KDE temperature, and random seed, and differ only in the space where kernel distances are computed. The raw-space baseline computes them on the 32-dimensional observations, and our method computes them in the persistent representation, which is initialized as the identity map and updated during training.
Decreasing alignment and its interpretation. The decreasing alignment in raw space is caused by nuisance-dominated kernel distances, which prevent the estimated drift from accurately tracking the reference field. The alignment is the right measure for the question that Figure 1(b) addresses: whether the drift field computed in a representation points in the direction of the drift field under the ideal representation. Its decrease does not indicate deteriorating generation quality, as the distance in semantic space decreases throughout training. While the reference field's average magnitude decreases from 0.87 to 0.23, the raw-space drift's semantic projection remains large (1.33-1.70) due to nuisance dimensions, causing alignment to fall from 0.40 to 0.03. With the learned representation, the estimated drift magnitude decreases alongside the reference field (1.70 to 0.23), and alignment improves to 0.81. These results show that the learned representation produces drift directions that more accurately reflect the remaining discrepancy between the generated and target distributions. We have clarified this interpretation in the revised appendix.
References
[1] V. G. Turrisi da Costa, E. Fini, M. Nabi, N. Sebe, E. Ricci. solo-learn: A Library of Self-supervised Methods for Visual Representation Learning. JMLR 2022.
[2] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative Adversarial Nets. NeurIPS 2014.
[3] A. Brock, J. Donahue, K. Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. ICLR 2019.
Response to Reviewer u28d [1/3]
We sincerely thank Reviewer
u28dfor the thoughtful and constructive feedback. We are encouraged that the reviewer finds the study of representation dynamics interesting and the empirical results broadly positive. We particularly appreciate the concrete suggestions for strengthening the comparison with frozen pretrained encoders and clarifying the diagnostics. We address each point below.We agree that comparisons with frozen pretrained encoders are important and deserve greater prominence. Our compute-matched experiments show that persistent representation learning continues to improve generation quality after separate pretraining and frozen-encoder training plateau. Intuitively, a pretrained encoder is optimized for its pretraining objective rather than specifically for generation, so its fixed representation may not capture all the discrepancies that remain as the generator evolves. In contrast, persistent representation learning allows the feature geometry to adapt to the current generated distribution, potentially providing more informative drifting fields throughout training. Following the reviewer's suggestion, we conducted a compute-matched comparison of separate and joint encoder–generator training on CIFAR-10.
Setup. For separate training, we pretrained our residual CNN encoder on CIFAR-10 using MoCo-v2+, following the solo-learn recipe [1] adapted to our architecture with the flattened encoder output and the generator's input normalization. Pretraining for 1,000 epochs required 1.4 GPU-hours and achieved 80.8% linear-probe accuracy. We then froze the pretrained encoder and trained the generator using Drifting Models. For joint training, our method learns the same encoder architecture from identity initialization alongside the generator. Both settings use the same generator architecture, encoder architecture, batch size, kernel temperature, velocity clipping, and evaluation protocol, while differing in their encoder training strategies.
Results. The table reports the best FID over 50k generated samples achieved within each total compute budget on GH200 GPUs, including encoder pretraining for separate training.
Separate training initially performs better but plateaus at an FID of 18.42 after 19.5k generator steps, with no further improvement over the following 20k steps. Joint training continues to improve, overtakes separate training at approximately 35 GPU-hours, and ultimately reaches an FID of 12.81. This suggests that, beyond learning a discriminative representation through pretraining, persistently adapting the representation to the evolving generator provides additional benefits for drift-based generation.
Frozen ImageNet encoder. Section 5.1 provides complementary evidence with an ImageNet-pretrained MoCo-v2 encoder: adapting its final residual block improves FID from 8.90 to 6.23 under the same generator and training schedule (see Q1a).
We have included the compute-matched comparison and its pretraining details in the revised manuscript.
We agree that a visual comparison with the frozen encoder would complement the FID comparison. Our new visual comparison confirms the improvement from adapting the pretrained representation, consistent with the reduction in FID from 8.90 to 6.23. We added uncurated samples of Drifting Models with the frozen MoCo-v2 encoder and with our adapted encoder to Appendix E of the revised version (Figure 13). Both runs follow the CIFAR-10 setting of Figure 2 and use the same generator and training schedule for 10k steps, and both generators sample from the same 64 latent noise vectors. The FID improvement represents a 30% relative reduction over the frozen encoder. The samples also show a qualitative improvement, with more objects appearing in clear and recognizable shapes.