- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Geometrically Modified Outputs (GMOs) to improve generative model self-training. The method modifies generated samples using the top singular direction of the generator’s Jacobian, then uses these samples for Neon or SIMS. The authors show improvements on several image generation models and compare their method with Gaussian noise.
Critical strengths
-
The paper proposes a useful modification to self-training and shows some improvement over standard Neon on the tested models.
-
The comparison with Gaussian noise and the analysis of fine-tuning behavior are clear.
Critical weaknesses
-
The improvement seems small given the additional compute on some models. For AlphaFlow-XL, the total cost is about five times that of standard Neon, while FID improves from 2.64 to 2.59. For MeanFlow-L, the cost is about 1.7 times higher, while FID improves from 3.74 to 3.70.
-
Appendix C includes the cost of generating GMOs, and Figure 10 compares different fine-tuning budgets. But my main question is still: how does GMO compare with standard Neon under the same total compute budget? I would like to see this comparison, counting both sample generation and parameter updates. Standard Neon should be allowed to use more training samples or adjust its fine-tuning budget. Since Figure 10 shows that more updates do not always help, simply training Neon for longer would not fully answer this question.
-
The paper should compare different choices of singular directions. The comparison with Gaussian noise is useful, but it does not explain why modifying the top singular direction is the best choice. What happens if the method uses the top few directions, or other directions, with a similar perturbation strength? This would help show how much the improvement depends on this specific choice.
Questions for authors
-
How does GMO compare with standard Neon under the same total compute budget?
-
How does using the top few singular directions compare with using only the first?
6 comments
Response to reviewer CtKd — Part 2: Singular-direction ablation
Top-k and alternative singular directions
Reviewer (W3): The paper should compare different choices of singular directions. The comparison with Gaussian noise is useful, but it does not explain why modifying the top singular direction is the best choice. What happens if the method uses the top few directions, or other directions, with a similar perturbation strength? This would help show how much the improvement depends on this specific choice.
Reviewer (Q2): How does using the top few singular directions compare with using only the first?
You suggest comparing the original top-1 construction with multiple leading singular directions. In the submitted method, GMOs shrink the Jacobian spectrum and reallocate the removed spectral energy to the leading singular direction while preserving total spectral energy. We initially focused on top-1 for its simplicity: it avoids additional choices about the number of directions and how to distribute energy among them. Your suggestion motivated a simple top-k extension that yields further improvements.
We ran the requested ablation on IMM (ImageNet-256), using 50,000-sample synthetic pools and α = 0.2, the setting of the IMM row in Table 1. The ablation variants share the same generated points and Jacobian computations, with the finetuning budget and merge weight tuned separately, matching the original paper’s sweep. FID is evaluated on 50,000 generated images.
For the top-k variants, the energy αE is split evenly across the k leading singular directions: σ̃ᵢ = √((1 − α)σᵢ² + αE/k), for i ≤ k, with the remaining singular values scaled by √(1 − α). Here E is the Jacobian’s squared Frobenius norm.
We use equal energy allocation as a simple choice that preserves total spectral energy. A more comprehensive study would compare additional direction counts and alternative allocation rules, potentially adapting these choices to the model or individual sample. Exploring these possibilities is a promising direction for further improving GMOs, now discussed in Section 5.
We also evaluate a random-output-direction variant, which retains the common Jacobian shrinkage term and replaces the leading-direction correction with a scaled random unit vector in the output latent space.
| Synthetic training samples | FID | Optimal finetuning budget (images) |
|---|---|---|
| Standard outputs (Table 1) | 7.32 | 2.5M |
| Top 1 (Table 1) | 6.25 | 2.9M |
| Top 2 | 6.18 | 2.9M |
| Top 4 | 5.98 | 2.9M |
| Random direction | 6.59 | 2.5M |
Interestingly, the random-output-direction variant also improves FID over standard-output Neon, from 7.32 to 6.59. The leading singular-direction variants achieve substantially larger improvements, reaching 6.25 for the submitted top-1 result, 6.18 for top-2, and 5.98 for top-4.
Redistributing energy across several leading singular directions further improves GMOs: top-4 reaches FID 5.98, improving on the submitted top-1 result of 6.25 by approximately 0.27. Top-4 performs best among the tested choices for IMM.
The original top-1 construction provides a simple, effective choice, while this ablation identifies additional improvement within the GMO framework. We have added the top-k formulation and singular-direction ablation to Appendix A.2, Table 3, with a summary in Section 4.1.
We hope the additional experiments and clarifications address your concerns. We would be grateful if you would reconsider your assessment in light of these revisions, and we would be happy to answer any further questions.
[1] Alemohammad, Wang, and Baraniuk. Neon: Negative Extrapolation from Self-Training Improves Image Generation. ICLR 2026.
[2] Alemohammad et al. Self-Improving Diffusion Models with Synthetic Data. arXiv:2408.16333, 2024.
Thank you for the additional experiments and for the work you put into the rebuttal. I was glad to see that redistributing spectral energy across multiple leading singular directions further improves FID. I consider this an important finding because it shows how the choice of directions affects performance.
My concern about the FID gains relative to the additional compute remains. On AlphaFlow-XL, standard Neon reduces FID by 0.29 at a cost of 0.10% of pretraining compute. Top-1 GMOs reduce FID by a further 0.05 while raising the combined cost of sample generation and fine-tuning to 0.51% of pretraining compute. On MeanFlow-L, standard Neon reduces FID by 0.23 at a cost of 0.63% of pretraining compute. Top-1 GMOs reduce FID by a further 0.04 while raising the combined cost to 1.06% (Tables 1 and 5).
The top-4 result on IMM is encouraging. FID drops from 6.25 with top-1 to 5.98 with top-4 (Table 3). Could you report the combined cost of sample generation and fine-tuning for top-4 relative to both top-1 and standard Neon? I would also like to know whether top-4 offers a similar improvement over top-1 on MeanFlow-L and AlphaFlow-XL, where the gains from top-1 were modest.
I would consider raising my score if top-k GMOs consistently outperformed standard Neon across a broader range of models at a comparable or lower total compute cost. Given the current experimental evidence, I will maintain my original score.
Response to reviewer CtKd — Part 1: Practical gains and matched compute
We thank the reviewer for the careful review and constructive feedback. We are glad that you found the modification to self-training useful and the Gaussian-noise comparison and finetuning analysis clear. Your questions motivated the new experiments.
New results and revisions are highlighted in blue in the revised manuscript.
Practical gains and additional compute
You raise an important question about the practical significance of these gains relative to the additional compute. Recall that Neon [1] and SIMS [2] both use a model finetuned on synthetic samples as a negative guide: Neon extrapolates model parameters, while SIMS extrapolates predictions during sampling. GMOs improve both frameworks by replacing the standard synthetic samples used for finetuning. We agree that the absolute FID gains on MeanFlow-L and AlphaFlow-XL are modest, but they extend the improvement achieved by standard-output Neon by approximately 17% on both models. We therefore assess the contribution of GMOs through the additional improvement over standard samples within each framework.
On MeanFlow-L, standard Neon reduces FID by 0.23, and GMOs contribute an additional 0.04. On AlphaFlow-XL, Neon reduces FID by 0.29, and GMOs add another 0.05. Across the tested Neon settings, GMOs add 17–105% to the FID reduction obtained with standard samples (Section 4.4 and Table 9). For SIMS on IMM, replacing standard samples with GMOs increases the FID reduction from approximately 0.75 to 1.42, corresponding to approximately 89% more improvement (Section 4.5; Appendix D.3, Table 7).
These improvements retain the generator architecture and the finetuning objective of the underlying framework. The results across both guidance mechanisms support GMOs as a complementary way to improve self-training through the choice of synthetic samples.
Comparison at matched total compute
You ask whether standard Neon can recover the improvement with additional samples or a different finetuning budget. Standard outputs are unmodified samples from the pretrained generator. Both experiments below use IMM (ImageNet-256).
Increasing the synthetic training pool
For each standard-output pool size, we tune the finetuning budget and merge weight using the same sweep as the original paper and select the best FID. As you note, more finetuning updates do not necessarily improve FID. We select the best-performing budget and merge weight for each pool size rather than simply using the final checkpoint. This comparison appears in Appendix C.1, Table 6.
Increasing the standard-output pool tenfold does not improve FID. With 50K samples, GMOs outperform every tested standard-output pool, including the ten-times-larger 500K pool.
Matching generation and finetuning compute
We separately compare 620K standard outputs with 5K GMOs (α = 0.2). Under the forward-equivalent cost model in Appendix C, one GMO costs approximately as much as 124 standard outputs. The pools have equal generation cost under this model, and both runs use the same finetuning budget (Section 4.6, Table 2).
GMOs improve FID by 0.91 at matched total compute, using a 124× smaller synthetic pool. Together, these results show that GMOs provide a useful finetuning signal that is not reproduced by simply increasing the number of standard synthetic samples in the tested settings.