- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DriftXpress to reduce the training cost of Drifting Models. Since the training data distribution is fixed, the attraction component is computed with these fixed landmarks, and the summaries can be precomputed. The repulsion component is kept the same as the original because the generated distribution changes over time. While projecting to the landmark set can introduce error, it makes attraction evaluation faster and speeds up convergence thanks to the full positive support represented by the landmark set. There are two versions: shared and non-shared. Sharding is more memory-efficient because it computes the summaries per class. The method is tested on 4 datasets: CIFAR10, CIFAR100, SVHN, ImageNet. Its performance is on par with standard drifting models, with faster training time and convergence. With large batch sizes, the benefit is mainly computational efficiency, and with small batch sizes, it’s faster convergence.
Critical strengths
The method is sound and simple. Projecting only for the attraction component makes sense because the data distribution does not change during training.
Critical weaknesses
-
From the provided implementation, all models seem to be unconditional (the number of classes in Unet is set to 0, the label is dropped in the dataloader and is not used when calling
model()to generate data), but in Appendix C1, Imagenet is still mentioned as class-conditional. Unconditional generation means that the attraction needs to be compared against full training samples rather than samples from a single class. This also links to the complexity of the two methods. In Drifting Models, the ( in the original Drifting Models paper) is relatively small compared to the full training set size, so the running time may differ, depending on the number of positive samples for each class. If the ImageNet experiment is class-conditional generation, please specify the number of classes and positive samples. In the implementation, data is gathered across GPUs in standard Drifting Models, while it’s not the case for Driftxpress (drift_global_query_gather=True,nystrom_global_query_gather=False). How much does this contribute to the reported running time and to the performance? -
Figure 2 shows FID vs elapsed time. FID may decrease faster because each epoch is trained faster, so more epochs are completed, and this might not be because of a larger effective support set. To show the benefit of the larger effective support set and the smoother, more stable field estimate, Figures 1C–D, 3, and 5 are more convincing, since they compare performance at matched training steps rather than matched training time.
Questions for authors
- Please address my concern regarding the experimental setup and implementation.
- Optional: It would be nice to run an experiment on the class-conditional ImageNet dataset, where the standard Drifting Models use 64 positive samples per class
6 comments
We thank the reviewer for their continued engagement with our work.
The timing table in our response uses two H100 GPUs with a per-GPU batch size of 1,000, while Table 1 uses eight H100 GPUs with a per-GPU batch size of 1,500. These correspond to global batch sizes of 2,000 and 12,000, respectively. The speedup depends on the batch size, as also illustrated in Figure 3. In particular, larger batches increase the baseline’s computational cost, which is why these numbers differ.
To clarify, the ImageNet results in Table 1 were evaluated using 50,000 images generated without class conditioning, compared against the 50,000 real images in the ILSVRC2012 validation split. The reference split contains 50 real images per class across 1,000 classes. This class balance applies to the real reference set. We will revise the appendix to make this explicit.
For the new class-conditioned ImageNet experiment, we compute FID between 50,000 generated images, with 500 per class generated using the corresponding class labels, and 5,000 held-out ILSVRC2012 validation images, with 50 per class from the same 100 selected classes.
Gentle follow-up on class-conditioned generation and experimental setup
Dear Reviewer tcUt, thank you again for the constructive discussion and for engaging with our responses. We wanted to check in as the discussion period draws to a close to confirm whether our latest clarifications on class-conditioned generation and experimental setups have fully resolved your remaining concerns. If everything looks satisfactory, we hope you will consider raising your score. Please let us know if any further questions remain!
Thanks for the additional experiments.
1a. If “class-conditional” referred to landmarks selected per class, why is it mentioned in the FID evaluation together with “50 generated images per class”?
1b. Can you describe the setting used in the table? From the table, disabling global-query gathering reduces the mean step time by 9.4–12.5%, but DriftXpress is only about 1.5× faster per training step than Drifting Models on CIFAR10 and SVHN, while Table 1 in the main paper reports a speedup of at least 6×. Can you clarify this?