acceptodds
Under review as a conference paper at ICLR 2027

Improving Generative Model Self-Training with Geometrically Modified Outputs

Abstract

Self-training generative models – the continued improvement of a model using its own outputs – is becoming increasingly important as high-quality training data becomes scarce. However, naïvely finetuning on model-generated samples leads to degradation through model collapse and the model autophagy disorder. Negative-guidance self-training methods turn this degradation into a useful signal, using a model finetuned on its own outputs to guide the original model toward improved generation. Existing methods, however, take the negative signal in standard model outputs as given. We instead ask whether this signal can be explicitly strengthened. We introduce Geometrically Modified Outputs (GMOs), which reweight the singular values of the generator’s input-output Jacobian to increase the influence of its leading singular directions. This geometric modification amplifies the mode-seeking behavior and distortions of standard outputs, providing a stronger and more targeted negative signal for self-training. Across a range of one-step generative models, GMOs consistently improve the performance of negative-guidance methods, including Neon and SIMS, compared with using standard model outputs.

Primary area:
generative models
Keywords:
Synthetic Data, Generative Models, Self-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CtKdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Geometrically Modified Outputs (GMOs) to improve generative model self-training. The method modifies generated samples using the top singular direction of the generator’s Jacobian, then uses these samples for Neon or SIMS. The authors show improvements on several image generation models and compare their method with Gaussian noise.

Critical strengths
  1. The paper proposes a useful modification to self-training and shows some improvement over standard Neon on the tested models.

  2. The comparison with Gaussian noise and the analysis of fine-tuning behavior are clear.

Critical weaknesses
  1. The improvement seems small given the additional compute on some models. For AlphaFlow-XL, the total cost is about five times that of standard Neon, while FID improves from 2.64 to 2.59. For MeanFlow-L, the cost is about 1.7 times higher, while FID improves from 3.74 to 3.70.

  2. Appendix C includes the cost of generating GMOs, and Figure 10 compares different fine-tuning budgets. But my main question is still: how does GMO compare with standard Neon under the same total compute budget? I would like to see this comparison, counting both sample generation and parameter updates. Standard Neon should be allowed to use more training samples or adjust its fine-tuning budget. Since Figure 10 shows that more updates do not always help, simply training Neon for longer would not fully answer this question.

  3. The paper should compare different choices of singular directions. The comparison with Gaussian noise is useful, but it does not explain why modifying the top singular direction is the best choice. What happens if the method uses the top few directions, or other directions, with a similar perturbation strength? This would help show how much the improvement depends on this specific choice.

Questions for authors
  1. How does GMO compare with standard Neon under the same total compute budget?

  2. How does using the top few singular directions compare with using only the first?

6 comments
AuthorsOpenReview ↗

Response to reviewer CtKd — Part 1: Practical gains and matched compute

We thank the reviewer for the careful review and constructive feedback. We are glad that you found the modification to self-training useful and the Gaussian-noise comparison and finetuning analysis clear. Your questions motivated the new experiments.

New results and revisions are highlighted in blue in the revised manuscript.

Practical gains and additional compute

Reviewer (W1): The improvement seems small given the additional compute on some models. For AlphaFlow-XL, the total cost is about five times that of standard Neon, while FID improves from 2.64 to 2.59. For MeanFlow-L, the cost is about 1.7 times higher, while FID improves from 3.74 to 3.70.

You raise an important question about the practical significance of these gains relative to the additional compute. Recall that Neon [1] and SIMS [2] both use a model finetuned on synthetic samples as a negative guide: Neon extrapolates model parameters, while SIMS extrapolates predictions during sampling. GMOs improve both frameworks by replacing the standard synthetic samples used for finetuning. We agree that the absolute FID gains on MeanFlow-L and AlphaFlow-XL are modest, but they extend the improvement achieved by standard-output Neon by approximately 17% on both models. We therefore assess the contribution of GMOs through the additional improvement over standard samples within each framework.

On MeanFlow-L, standard Neon reduces FID by 0.23, and GMOs contribute an additional 0.04. On AlphaFlow-XL, Neon reduces FID by 0.29, and GMOs add another 0.05. Across the tested Neon settings, GMOs add 17–105% to the FID reduction obtained with standard samples (Section 4.4 and Table 9). For SIMS on IMM, replacing standard samples with GMOs increases the FID reduction from approximately 0.75 to 1.42, corresponding to approximately 89% more improvement (Section 4.5; Appendix D.3, Table 7).

These improvements retain the generator architecture and the finetuning objective of the underlying framework. The results across both guidance mechanisms support GMOs as a complementary way to improve self-training through the choice of synthetic samples.

Comparison at matched total compute

Reviewer (W2): Appendix C includes the cost of generating GMOs, and Figure 10 compares different fine-tuning budgets. But my main question is still: how does GMO compare with standard Neon under the same total compute budget? I would like to see this comparison, counting both sample generation and parameter updates. Standard Neon should be allowed to use more training samples or adjust its fine-tuning budget. Since Figure 10 shows that more updates do not always help, simply training Neon for longer would not fully answer this question.

Reviewer (Q1): How does GMO compare with standard Neon under the same total compute budget?

You ask whether standard Neon can recover the improvement with additional samples or a different finetuning budget. Standard outputs are unmodified samples from the pretrained generator. Both experiments below use IMM (ImageNet-256).

Increasing the synthetic training pool

For each standard-output pool size, we tune the finetuning budget and merge weight using the same sweep as the original paper and select the best FID. As you note, more finetuning updates do not necessarily improve FID. We select the best-performing budget and merge weight for each pool size rather than simply using the final checkpoint. This comparison appears in Appendix C.1, Table 6.

Synthetic training samplesPool sizeFID
Standard outputs50K7.27 ± 0.03
Standard outputs150K7.29 ± 0.02
Standard outputs300K7.34 ± 0.03
Standard outputs500K7.40 ± 0.04
GMOs (α = 0.2)50K6.29 ± 0.03

Increasing the standard-output pool tenfold does not improve FID. With 50K samples, GMOs outperform every tested standard-output pool, including the ten-times-larger 500K pool.

Matching generation and finetuning compute

We separately compare 620K standard outputs with 5K GMOs (α = 0.2). Under the forward-equivalent cost model in Appendix C, one GMO costs approximately as much as 124 standard outputs. The pools have equal generation cost under this model, and both runs use the same finetuning budget (Section 4.6, Table 2).

Synthetic training samplesPool sizeFID
Standard outputs620K7.34 ± 0.05
GMOs (α = 0.2)5K6.43 ± 0.04

GMOs improve FID by 0.91 at matched total compute, using a 124× smaller synthetic pool. Together, these results show that GMOs provide a useful finetuning signal that is not reproduced by simply increasing the number of standard synthetic samples in the tested settings.

AuthorsOpenReview ↗

Response to reviewer CtKd — Part 2: Singular-direction ablation

Top-k and alternative singular directions

Reviewer (W3): The paper should compare different choices of singular directions. The comparison with Gaussian noise is useful, but it does not explain why modifying the top singular direction is the best choice. What happens if the method uses the top few directions, or other directions, with a similar perturbation strength? This would help show how much the improvement depends on this specific choice.

Reviewer (Q2): How does using the top few singular directions compare with using only the first?

You suggest comparing the original top-1 construction with multiple leading singular directions. In the submitted method, GMOs shrink the Jacobian spectrum and reallocate the removed spectral energy to the leading singular direction while preserving total spectral energy. We initially focused on top-1 for its simplicity: it avoids additional choices about the number of directions and how to distribute energy among them. Your suggestion motivated a simple top-k extension that yields further improvements.

We ran the requested ablation on IMM (ImageNet-256), using 50,000-sample synthetic pools and α = 0.2, the setting of the IMM row in Table 1. The ablation variants share the same generated points and Jacobian computations, with the finetuning budget and merge weight tuned separately, matching the original paper’s sweep. FID is evaluated on 50,000 generated images.

For the top-k variants, the energy αE is split evenly across the k leading singular directions: σ̃ᵢ = √((1 − α)σᵢ² + αE/k), for i ≤ k, with the remaining singular values scaled by √(1 − α). Here E is the Jacobian’s squared Frobenius norm.

We use equal energy allocation as a simple choice that preserves total spectral energy. A more comprehensive study would compare additional direction counts and alternative allocation rules, potentially adapting these choices to the model or individual sample. Exploring these possibilities is a promising direction for further improving GMOs, now discussed in Section 5.

We also evaluate a random-output-direction variant, which retains the common Jacobian shrinkage term and replaces the leading-direction correction with a scaled random unit vector in the output latent space.

Synthetic training samplesFIDOptimal finetuning budget (images)
Standard outputs (Table 1)7.322.5M
Top 1 (Table 1)6.252.9M
Top 26.182.9M
Top 45.982.9M
Random direction6.592.5M

Interestingly, the random-output-direction variant also improves FID over standard-output Neon, from 7.32 to 6.59. The leading singular-direction variants achieve substantially larger improvements, reaching 6.25 for the submitted top-1 result, 6.18 for top-2, and 5.98 for top-4.

Redistributing energy across several leading singular directions further improves GMOs: top-4 reaches FID 5.98, improving on the submitted top-1 result of 6.25 by approximately 0.27. Top-4 performs best among the tested choices for IMM.

The original top-1 construction provides a simple, effective choice, while this ablation identifies additional improvement within the GMO framework. We have added the top-k formulation and singular-direction ablation to Appendix A.2, Table 3, with a summary in Section 4.1.

We hope the additional experiments and clarifications address your concerns. We would be grateful if you would reconsider your assessment in light of these revisions, and we would be happy to answer any further questions.

[1] Alemohammad, Wang, and Baraniuk. Neon: Negative Extrapolation from Self-Training Improves Image Generation. ICLR 2026.

[2] Alemohammad et al. Self-Improving Diffusion Models with Synthetic Data. arXiv:2408.16333, 2024.

Thank you for the additional experiments and for the work you put into the rebuttal. I was glad to see that redistributing spectral energy across multiple leading singular directions further improves FID. I consider this an important finding because it shows how the choice of directions affects performance.

My concern about the FID gains relative to the additional compute remains. On AlphaFlow-XL, standard Neon reduces FID by 0.29 at a cost of 0.10% of pretraining compute. Top-1 GMOs reduce FID by a further 0.05 while raising the combined cost of sample generation and fine-tuning to 0.51% of pretraining compute. On MeanFlow-L, standard Neon reduces FID by 0.23 at a cost of 0.63% of pretraining compute. Top-1 GMOs reduce FID by a further 0.04 while raising the combined cost to 1.06% (Tables 1 and 5).

The top-4 result on IMM is encouraging. FID drops from 6.25 with top-1 to 5.98 with top-4 (Table 3). Could you report the combined cost of sample generation and fine-tuning for top-4 relative to both top-1 and standard Neon? I would also like to know whether top-4 offers a similar improvement over top-1 on MeanFlow-L and AlphaFlow-XL, where the gains from top-1 were modest.

I would consider raising my score if top-k GMOs consistently outperformed standard Neon across a broader range of models at a comparable or lower total compute cost. Given the current experimental evidence, I will maintain my original score.

Reviewer_uKKxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Geometrically Modified Outputs (GMOs) modify generated samples by transferring spectral energy from small singular components of the generator's Jacobian to its leading component. A model is fine-tuned on these samples and then used as a negative guide to improve the generator. The method aims to strengthen the negative training signal while preserving spectral energy.

Critical strengths
  • Experiments on five ImageNet models show that GMOs improve Neon FID compared with standard outputs. An additional SIMS-style experiment on IMM also shows an improvement.

  • The experiments compare GMOs with Gaussian perturbations of the same magnitude and test transfer across model sizes and inference strategies. They also report image quality, diversity, and computational costs, including GMO generation.

  • The proposed formulation allows approximate GMO generation without storing the full Jacobian, using power iteration and trace estimation to approximate the required spectral values.

Critical weaknesses
  • The paper reports the additional cost of GMO generation, but does not compare performance with standard Neon under the same total computational budget.

  • The proposed mode-seeking explanation for GMOs is supported mainly by embedding visualizations. These plots do not establish concentration in high-density regions of the original model distribution.

Questions for authors
  • Under what conditions do GMOs increase concentration in high-density regions of the original model distribution?

  • Does the proposed intuition behind GMOs also apply to GAN generators, which also generate samples in one step but are trained adversarially? Could the authors clarify how GMO-based fine-tuning and negative guidance would be applied in this setting?

  • Does the total cost reported in Table~2 include hyperparameter search and evaluation? Could the authors compare standard Neon and Neon with GMOs at matched total computational cost?

  • Do the five seeds for the main results include independent synthetic-data generation and fine-tuning runs? For each setting in Figure5 and Table4, are all metrics evaluated using the same selected checkpoint and guidance configuration?

7 comments
AuthorsOpenReview ↗

Response to reviewer uKKx — Part 1: Matched compute and synthetic pool size

We thank the reviewer for the detailed review and thoughtful questions. We are glad that you highlighted the improvements with both Neon [1] and SIMS [2], the transfer experiments, and the ability to construct GMOs without storing the full Jacobian. Your questions helped us strengthen the compute comparisons and examine the mode-seeking intuition more directly.

New results and revisions are highlighted in blue in the revised manuscript.

Reviewer (W1): The paper reports the additional cost of GMO generation, but does not compare performance with standard Neon under the same total computational budget.

Reviewer (Q3, matched compute): Could the authors compare standard Neon and Neon with GMOs at matched total computational cost?

You ask whether GMOs retain their advantage when standard Neon receives the same total computational budget. Standard outputs are unmodified samples from the pretrained generator. We compare them with GMOs using the same Neon finetuning and weight extrapolation procedure. We added two experiments on IMM (ImageNet-256) to test whether additional standard samples can recover the improvement from GMOs.

1. Increasing the synthetic training pool

We compare 50K, 150K, 300K and 500K standard outputs, tuning the finetuning budget and merge weight for each pool size using the same sweep as the original paper and selecting the best FID. The pools use the same sampler and seeding convention. We have added this comparison to Appendix C.1, Table 6.

All results below report FID on 50,000 generated images, as mean ± sample standard deviation over five evaluation seeds.

Synthetic training samplesPool sizeFID
Standard outputs50K7.27 ± 0.03
Standard outputs150K7.29 ± 0.02
Standard outputs300K7.34 ± 0.03
Standard outputs500K7.40 ± 0.04
GMOs (α = 0.2)50K6.29 ± 0.03

Increasing the standard-output pool tenfold, from 50K to 500K, does not improve FID: it changes from 7.27 to 7.40, despite tuning the finetuning budget and merge weight for each pool size. With just 50K samples, GMOs outperform every tested standard-output pool, including the ten-times-larger 500K pool. These results support the interpretation that GMOs provide a more useful finetuning signal: simply increasing the number of standard outputs does not reproduce their benefit within the tested search grid.

2. Comparison at matched compute

We separately compare 620K standard outputs with 5K GMOs (α = 0.2), using the finetuning budget and merge weight selected for best FID in the preceding sweeps. We have added this comparison to Section 4.6, Table 2. Under the forward-equivalent cost model in Appendix C, one GMO costs approximately as much as 124 standard outputs, so these pools have equal estimated generation cost. Both models also use the same 2.46M-image finetuning budget, matching the estimated total cost of generation and parameter updates.

Synthetic training samplesPool sizeFID
Standard outputs620K7.34 ± 0.05
GMOs (α = 0.2)5K6.43 ± 0.04

With a 124× smaller synthetic pool, GMOs improve FID by 0.91 at matched estimated total compute, achieving 6.43 versus 7.34 for standard outputs.

Together, these results show that GMOs provide a useful finetuning signal that is not reproduced by simply increasing the number of standard synthetic samples in the tested settings.

AuthorsOpenReview ↗

Response to reviewer uKKx — Part 2: Mode-seeking and GAN applicability

Mode-seeking and original-model density

Reviewer (W2): The proposed mode-seeking explanation for GMOs is supported mainly by embedding visualizations. These plots do not establish concentration in high-density regions of the original model distribution.

Reviewer (Q1): Under what conditions do GMOs increase concentration in high-density regions of the original model distribution?

You ask whether the embedding visualizations establish concentration in high-density regions of the original model distribution. Here, mode-seeking refers to the proposed interpretation that GMOs move samples toward regions assigned higher density by the original generator. These visualizations are intended to provide intuition for this behavior. To examine it more directly in a simpler setting, we added a two-dimensional toy experiment in which the original model’s density can be evaluated exactly.

We train a small nonlinear generator on 100 samples from a Gaussian distribution in two dimensions. The generator samples one coordinate, then samples the second conditioned on the first. This lets us evaluate the trained model’s own output density exactly. We keep the model fixed and use this density to compare how many original samples and GMOs fall within the same high-density region.

We repeat the experiment over five training runs of the same small toy model with 20,000 paired samples per run. For each trained model, we define a fixed high-density region using the density threshold corresponding to the densest 25% of a separate set of original samples.

GMO strength αSamples in the high-density region
025.32%
0.126.04%
0.226.77%
0.428.72%
0.630.98%

As α increases from 0 to 0.6, the share of samples in the fixed high-density region rises from 25.32% to 30.98%, an increase of 5.66 percentage points.

Figure 8 visualizes this movement at α = 0.6: arrows connect original samples to their corresponding GMOs on the original model’s density surface and in a view from above. This makes the movement toward higher-density regions directly visible. Together, the figure and quantitative results illustrate the proposed mode-seeking intuition in this toy setting. We have added the experiment to Appendix B.1, with quantitative results in Table 4 and the visualization in Figure 8.

Applicability to GAN generators

Reviewer (Q2): Does the proposed intuition behind GMOs also apply to GAN generators, which also generate samples in one step but are trained adversarially? Could the authors clarify how GMO-based fine-tuning and negative guidance would be applied in this setting?

You ask how GMO-based finetuning and negative guidance could be applied to adversarially trained GAN generators. In Neon [1], a copy of the generator is finetuned on synthetic samples, and the original generator parameters are then extrapolated away from the finetuned parameters. We agree that GANs are a natural setting in which to investigate the broader applicability of GMOs. Indeed, A Geometric Perspective on Recursive Synthetic Training [3] observes that GAN generators exhibit declining Jacobian effective ranks and deteriorating sample quality under recursive synthetic training. These findings support the geometric motivation for applying GMOs to adversarially trained generators. The GMO construction itself requires only the generator’s input–output Jacobian and is independent of its training objective.

Applying the complete self-training procedure to GANs, however, requires a carefully designed finetuning protocol. Adversarial finetuning involves coupled generator and discriminator updates, making the relative learning rates, update schedules, and regularization consequential. Our recipe of retaining the original training objective and reducing the learning rate to one tenth of its pretraining value therefore cannot be assumed to transfer directly.

A possible extension would use GMOs as the reference dataset during adversarial finetuning, followed by negative extrapolation of the generator parameters. We hypothesize that GMOs could improve this procedure when finetuning produces a stable, mode-seeking degradation direction suitable for negative extrapolation. The prior GAN observations motivate this hypothesis, while establishing a reliable GAN-specific finetuning protocol remains an important direction for further investigation.

AuthorsOpenReview ↗

Response to reviewer uKKx — Part 3: Compute accounting and evaluation protocol

Compute accounting

Reviewer (Q3, accounting): Does the total cost reported in Table 2 include hyperparameter search and evaluation?

You ask whether the reported compute includes hyperparameter search and evaluation. Table 2 of the submitted manuscript (Table 5 in the revised manuscript) includes synthetic training-data generation, including GMO construction, and finetuning for the selected configuration, expressed as a percentage of the base model’s pretraining compute. It excludes the cumulative costs of hyperparameter search and evaluation. We have made this accounting explicit in Appendix C and the Table 5 caption.

This scope is consistent with the reported training budgets in Neon [1], SIMS [2], DDO [4], and Autoguidance [5], which account for training the evaluated configuration rather than the full search and evaluation campaign. For example, DDO specifies its per-round training duration separately from its hyperparameter sweeps, and Autoguidance reports guiding-model training overhead separately from the search and evaluation costs described in its Appendix B.1.

GMO construction adds generation cost relative to standard outputs. Our matched-compute experiment in Section 4.6, Table 2, also discussed in Part 1 of this response, accounts for this overhead by comparing 5K GMOs with 620K standard outputs under the forward-equivalent cost model in Appendix C, while keeping the finetuning budget equal.

Meaning of the five seeds

Reviewer (Q4, seeds): Do the five seeds for the main results include independent synthetic-data generation and fine-tuning runs?

You ask whether the five seeds include independent synthetic-data generation and finetuning runs. The five seeds correspond to five FID evaluations of each selected model, each using 50,000 generated images with a different sampling seed. The synthetic training dataset, finetuned checkpoint, and selected hyperparameters are held fixed across these evaluations.

Repeating synthetic-data generation and finetuning five times for every model and setting would have required substantially more computation. The reported standard deviations quantify variability from evaluation sampling, rather than independent training runs. Across the five tested model settings, spanning IMM, MeanFlow, and AlphaFlow, GMOs consistently improve mean FID over standard-output Neon. We have made the evaluation-seed interpretation explicit in the Table 1 caption.

Consistency across reported metrics

Reviewer (Q4, metrics): For each setting in Figure 5 and Table 4, are all metrics evaluated using the same selected checkpoint and guidance configuration?

You ask whether all reported metrics use the same selected checkpoint and guidance configuration. All metrics use the same FID-selected checkpoint and weight: FID, precision and recall [6], and density and coverage [7].

For Figure 5, we select the finetuning budget B and merge weight w that achieve the lowest FID among the tested combinations, separately for each α, including α = 0. All other metrics are evaluated at that same configuration. This allows each α to use its best tested budget and merge weight, since their preferred values can change with GMO strength. Table 4 of the submission (Table 8 in the revised manuscript) likewise reports all metrics at the FID-selected configuration for each model and method. We have clarified this protocol in Section 4.2 and the Figure 5 caption.

We hope the additional experiments and clarifications address your concerns. We would be grateful if you would reconsider your assessment in light of these revisions, and we would be happy to answer any further questions.

[1] Alemohammad, Wang, and Baraniuk. Neon: Negative Extrapolation from Self-Training Improves Image Generation. ICLR 2026.

[2] Alemohammad et al. Self-Improving Diffusion Models with Synthetic Data. arXiv:2408.16333, 2024.

[3] Batsell, Walker, and Baraniuk. A Geometric Perspective on Recursive Synthetic Training. DeLTa Workshop at ICLR 2026.

[4] Zheng et al. Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model Is Secretly a GAN Discriminator. ICML 2025.

[5] Karras et al. Guiding a Diffusion Model with a Bad Version of Itself. NeurIPS 2024.

[6] Kynkäänniemi et al. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS 2019.

[7] Naeem et al. Reliable Fidelity and Diversity Metrics for Generative Models. ICML 2020.

Reviewer_3uZnOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method to improve the self-training of one-step generative models. The authors argue that directly using a model's generated outputs for self-training may degrade generation quality and lead to mode collapse. To address this issue, they build on negative-guidance self-training, which uses generated samples to obtain a negative learning signal rather than directly adopting the model fine-tuned on these samples. The authors further hypothesise that modifying the generated outputs before using them for negative-guidance self-training can improve generation quality. To this end, they propose "Geometrically Modified Outputs (GMO)", which reweighs the singular values of the generator's input-output Jacobian matrix by concentrating spectral energy in the largest singular direction. The proposed method is evaluated using three generative model families: IMM, MeanFlow, and AlphaFlow. The authors demonstrate its effectiveness with two negative-guidance self-training algorithms,Neon and SIMS. The experiments are conducted primarily on the ImageNet-256 dataset, with additional experiments on CIFAR-10. Performance is evaluated mainly using FID, along with precision, recall, density, and coverage to assess generation quality and diversity. The authors also investigate the transferability of GMOs across models of different sizes and inference strategies. The results show consistent improvements in FID over the corresponding negative-guidance self-training baselines, although the magnitude of improvement varies across models.

Critical strengths
  1. The paper is well-written, clearly presented, and easy to follow.
  2. The paper addresses an interesting and important research direction of improving self-training for recently proposed one-step generative models.
  3. The paper proposes an approach that can be integrated with existing generative models to improve negative-guidance self-training without requiring changes to the underlying model architecture.
  4. The paper presents an interesting methodological contribution by introducing Jacobian-based geometric modifications to improve existing negative-guidance self-training methods. The approach can be integrated with generative models without requiring changes to their underlying architectures.
Critical weaknesses
  1. Limited exploration of alternative spectral redistribution strategies. The proposed method concentrates spectral energy in the largest singular direction. It would be interesting to investigate whether distributing the spectral energy across the top -k singular directions could lead to better results.

  2. Limited evaluation across datasets. Most experiments are conducted on ImageNet-256. Evaluating the proposed method on other benchmark datasets for generative models, such as CelebA-HQ and LSUN, would help establish its generalisability across different data distributions.

  3. Modest improvements on larger generative models. Although the proposed method consistently improves FID, the improvements on larger MeanFlow and AlphaFlow models are relatively small, raising questions about the practical significance of these gains.

Questions for authors

Please refer weaknesses section.

3 comments
AuthorsOpenReview ↗

Response to reviewer 3uZn - Part 1: Singular-direction ablation

We thank the reviewer for the careful reading and constructive suggestions. We are glad that you found the paper clear and recognized the value of improving negative-guidance self-training without changing the generator architecture. Your suggestion to explore multiple singular directions led to a stronger IMM result, which we discuss below.

New results and revisions are highlighted in blue in the revised manuscript.

Reviewer (W1): Limited exploration of alternative spectral redistribution strategies. The proposed method concentrates spectral energy in the largest singular direction. It would be interesting to investigate whether distributing the spectral energy across the top -k singular directions could lead to better results.

You suggest comparing the original top-1 construction with multiple leading singular directions. In the submitted method, GMOs shrink the Jacobian spectrum and reallocate the removed spectral energy to the leading singular direction while preserving total spectral energy. We initially focused on top-1 for its simplicity: it avoids additional choices about the number of directions and how to distribute energy among them. Your suggestion motivated a simple top-k extension that yields further improvements.

We ran the requested ablation on IMM (ImageNet-256), using 50,000-sample synthetic pools and α = 0.2, the setting of the IMM row in Table 1. The ablation variants share the same generated points and Jacobian computations, with the finetuning budget and merge weight tuned separately, matching the original paper’s sweep. FID is evaluated on 50,000 generated images.

For the top-k variants, the energy αE is split evenly across the k leading singular directions: σ̃ᵢ = √((1 − α)σᵢ² + αE/k), for i ≤ k, with the remaining singular values scaled by √(1 − α). Here E is the Jacobian’s squared Frobenius norm.

We use equal energy allocation as a simple choice that preserves total spectral energy. A more comprehensive study would compare additional direction counts and alternative allocation rules, potentially adapting these choices to the model or individual sample. Exploring these possibilities is a promising direction for further improving GMOs.

We also evaluate a random-output-direction variant, which retains the common Jacobian shrinkage term and replaces the leading-direction correction with a scaled random unit vector in the output latent space.

Synthetic training samplesFIDOptimal finetuning budget (images)
Standard outputs (Table 1)7.322.5M
Top 16.252.9M
Top 26.182.9M
Top 45.982.9M
Random direction6.592.5M

Interestingly, the random-output-direction variant also improves FID over standard-output Neon, from 7.32 to 6.59. The leading singular-direction variants achieve substantially larger improvements, reaching 6.25 for the submitted top-1 result, 6.18 for top-2, and 5.98 for top-4.

Redistributing energy across several leading singular directions further improves GMOs: top-4 reaches FID 5.98, improving on the submitted top-1 result of 6.25 by approximately 0.27. Top-4 performs best among the tested choices for IMM.

The original top-1 construction provides a simple, effective choice, while this ablation identifies additional improvement within the GMO framework. We have added the top-k formulation and singular-direction ablation to Appendix A.2, Table 3, with a summary in Section 4.1.

AuthorsOpenReview ↗

Response to reviewer 3uZn - Part 2: Dataset coverage and practical gains

W2 — Dataset coverage

Reviewer (W2): Limited evaluation across datasets. Most experiments are conducted on ImageNet-256. Evaluating the proposed method on other benchmark datasets for generative models, such as CelebA-HQ and LSUN, would help establish its generalisability across different data distributions.

You ask whether GMOs generalize to datasets beyond those evaluated in the paper. Our evaluation includes CIFAR-10 as well as ImageNet-256. On CIFAR-10, we study the effects of GMOs on finetuning budget and merge weight using a MeanFlow UNet (Appendix C, Figure 9), and evaluate the transfer of GMOs from a one-step IMM generator to a two-step IMM model (Figure 6 and Appendix D.1).

Our benchmark choices follow the original IMM [1], MeanFlow [2], and AlphaFlow [3] studies, which do not report evaluations on CelebA-HQ or LSUN. Extending our comparisons to these datasets would require suitable additional pretrained checkpoints or training and validating new base models, with the latter incurring substantial computation beyond the self-improvement procedure itself.

The GMO construction operates on the generator’s input–output Jacobian and does not require assumptions specific to a particular dataset. Its motivation comes from the geometry of the generator and the mode-seeking behavior of synthetic sampling underlying negative-guidance self-training. This motivates broader evaluation as suitable pretrained models become available. We now discuss this direction in Section 5, Limitations and Future Directions.

W3 — Practical significance of larger-model gains

Reviewer (W3): Modest improvements on larger generative models. Although the proposed method consistently improves FID, the improvements on larger MeanFlow and AlphaFlow models are relatively small, raising questions about the practical significance of these gains.

You raise an important question about the practical significance of the gains on larger models. Recall that Neon [4] and SIMS [5] both use a model finetuned on synthetic samples as a negative guide: Neon extrapolates model parameters, while SIMS extrapolates predictions during sampling. GMOs improve both frameworks by replacing the standard synthetic samples used for finetuning. We agree that the absolute FID gains on MeanFlow-L and AlphaFlow-XL are modest, but they extend the improvement achieved by standard-output Neon by approximately 17% on both models. We therefore assess the contribution of GMOs through the additional improvement over standard samples within each framework.

On MeanFlow-L, standard Neon reduces FID by 0.23, and GMOs contribute an additional 0.04. On AlphaFlow-XL, Neon reduces FID by 0.29, and GMOs add another 0.05. Across the tested Neon settings, GMOs add 17–105% to the FID reduction obtained with standard samples (Table 9). For SIMS on IMM, replacing standard samples with GMOs increases the FID reduction from approximately 0.75 to 1.42, corresponding to approximately 89% more improvement (Appendix D.3, Table 7).

These improvements retain the generator architecture and the finetuning objective of the underlying framework. The results across both guidance mechanisms support GMOs as a complementary way to improve self-training through the choice of synthetic samples.

We hope the additional experiments and clarifications address your concerns. We would be grateful if you would reconsider your assessment in light of these revisions, and we would be happy to answer any further questions.

[1] Zhou, Ermon, and Song. Inductive Moment Matching. ICML 2025.

[2] Geng et al. Mean Flows for One-step Generative Modeling. NeurIPS 2025.

[3] Zhang et al. AlphaFlow: Understanding and Improving MeanFlow Models. ICLR 2026.

[4] Alemohammad, Wang, and Baraniuk. Neon: Negative Extrapolation from Self-Training Improves Image Generation. ICLR 2026.

[5] Alemohammad et al. Self-Improving Diffusion Models with Synthetic Data. arXiv:2408.16333, 2024.

AuthorsOpenReview ↗

Thank You!

Dear Reviewer 3uZn,

Thank you for taking the time to read our responses carefully and for updating your score. We are glad that the clarifications on alternative spectral directions, dataset coverage, and the practical significance of the gains on larger models were helpful. Your questions led directly to the direction ablation and to a clearer discussion of scope, both of which are included in the revised manuscript and have made the paper stronger.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion for the Area Chair

Dear Area Chair,

For your convenience, we briefly summarize the outcome of the discussion below.

Reviewer status at the close of the discussion:

  • 3uZn raised their score by one point. They found our clarifications and experiments satisfactory.
  • uKKx maintained their positive score. They confirmed that our results and clarifications addressed all their concerns.
  • CtKd kept their score. They called the top-k result "an important finding" and asked for top-k comparisons on more models, which we provided (points 1 and 2). In their final comment, after seeing these results, they wrote that the top-k extension "could bring the paper above that threshold if broader comparisons confirm these gains".

The main outcomes of the discussion follow; each point opens with its takeaway in bold. New results and revisions are highlighted in blue in the revised manuscript.

1. At matched total compute, GMOs outperform standard Neon by 0.91 to 1.21 FID on IMM (CtKd, uKKx). The three pools below have matched generation plus finetuning cost (Section 4.6, Table 2; the top-4 row is from our follow-up):

Synthetic pool (IMM, ImageNet-256)SamplesFID
Standard outputs620K7.34 ± 0.05
Top-1 GMOs5K6.43 ± 0.04
Top-4 GMOs4K6.13 ± 0.03

More standard samples do not help Neon: pools of 50K to 500K standard outputs give FID between 7.27 and 7.40, against 6.29 for 50K GMOs (Appendix C.1, Table 6).

2. Using more leading directions improves GMOs further, and top-4 GMOs also outperform standard Neon on another architecture and dataset (3uZn, CtKd). On IMM at α = 0.2, standard outputs give FID 7.32, a random output direction 6.59, top-1 6.25, top-2 6.18 and top-4 5.98 (Appendix A.2, Table 3). A top-4 GMO takes 70 network passes per sample against 62 for top-1. Reviewer CtKd also asked for top-4 on MeanFlow-L and AlphaFlow-XL. A full top-4 study on these ImageNet-256 models was not feasible in the two days that remained, so we evaluated top-4 on a smaller model with a different architecture and dataset, a MeanFlow UNet on CIFAR-10 (base FID 3.42), to examine the generality of the result. Top-4 GMOs reach 2.39 ± 0.01 with 6K samples, against 2.51 ± 0.03 for standard Neon with 6K samples and 2.64 ± 0.01 with 780K samples, a comparable generation cost.

3. GMOs add 17 to 105% to the FID reduction of the self-training methods tested (3uZn, CtKd). With Neon, this holds on all five model settings, including about 17% on MeanFlow-L and AlphaFlow-XL (Section 4.4, Table 9). The smaller absolute gains on these two models are consistent with diminishing returns: a stronger model leaves less room for improvement, and standard Neon's absolute gains are also relatively small on these two models. For SIMS-style guidance on IMM, GMOs increase the FID reduction from about 0.75 to 1.42 (Appendix D.3, Table 7).

4. A toy experiment with an exactly computable density supports the mode-seeking intuition (uKKx). The share of samples in a fixed high-density region rises from 25.32% for standard outputs to 30.98% for GMOs at α = 0.6 (Appendix B.1, Table 4 and Figure 8).

5. The evaluation protocol and compute accounting are now explicit in the manuscript (3uZn, uKKx). The revised manuscript states that the five seeds are evaluation seeds at the optimal checkpoints and gives more detail on the compute accounting (Appendix C, Table 5). It also discusses broader dataset evaluation (Section 5).

We are grateful to the reviewers, whose questions led directly to these additions and made the paper stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.