acceptodds
Under review as a conference paper at ICLR 2027

SAGE: Subspace Alignment for Classifier-Free Guidance in Mixture-of-Experts Diffusion Models

Abstract

Diffusion Transformers with Mixture-of-Experts (MoE) routing are a leading recipe for scaling generative models. Classifier-Free Guidance (CFG) is essential for generation quality, yet excessively high guidance scales trigger collapse. We identify a previously unreported failure mode in their combination: the two CFG branches route independently, so their realized activations occupy different subspaces. The unconditional write then leaves the conditional subspace, and CFG amplifies that residual linearly in the guidance scale. We propose SAGE, a training-time regularizer that aligns unconditional MoE activations to the conditional subspace without restricting routing diversity, at zero inference cost. Toy experiments show that SAGE dramatically suppresses extreme drift by 9.2. When scaled to a 1B-parameter text-to-image model, SAGE significantly improves generation quality, delivering a 9.3% boost in peak DPG-Bench performance. Extensive experiments demonstrate that SAGE consistently outperforms the baseline.

Primary area:
generative models
Keywords:
Diffusion Transformer, Generative Models, Diffusion Models, Classifier-Free Guidance, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CDr9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the degradation in generation quality of MoE diffusion models under high CFG scales. The authors attribute this issue to subspace leakage caused by routing mismatch between the conditional and unconditional branches. To address this, they propose SAGE, which introduces a subspace alignment regularizer during training while preserving routing diversity. The effectiveness of SAGE is validated through a 1B-parameter text-to-image model.

Critical strengths
  1. This paper investigates the interaction between MoE routing and CFG, highlighting the performance degradation of MoE diffusion models under high CFG scales. This is an interesting and practically relevant problem that provides a valuable research direction for understanding MoE-based generative models.

  2. The proposed method aligns activation subspaces rather than enforcing routing agreement, improving generation quality while preserving expert routing diversity, without introducing additional inference overhead.

  3. The experiments on a 1B-parameter text-to-image model demonstrate clear improvements on DPG-Bench, GenEval, and FID, particularly showing better stability under high CFG scales. The paper also provides a comprehensive set of ablation studies.

Critical weaknesses
  1. The authors characterize subspace leakage as a MoE-specific issue. However, different expert routing decisions do not necessarily lead to different activation subspaces. Conversely, even when the conditional and unconditional branches select the same experts, their outputs may still lie in different subspaces due to differences in their input hidden states. Therefore, the causal relationship between routing mismatch and subspace leakage is not rigorously established. In my understanding, the linear growth of leakage with the CFG scale follows directly from the definition of the orthogonal projection residual, which does not necessarily demonstrate that routing mismatch is the primary cause of generation quality degradation at high CFG scales.

  2. The current experiments mainly demonstrate the effectiveness of SAGE but do not sufficiently establish whether its improvements come from addressing MoE-specific routing mismatch. For example, experiments applying the same SAGE loss to other architectures with comparable parameter counts are missing.

  3. SAGE requires additional paired conditional/unconditional forward passes and QR decomposition. However, the paper does not quantify the actual training time and memory overhead, nor does it provide comparisons under equal computational budgets.

Questions for authors
  1. The linear growth of leakage with the CFG scale appears to follow directly from the mathematical definition of the projection residual. Could the authors further demonstrate how this residual affects the final image quality?

  2. Could the authors provide controlled experiments to further validate the relationship between routing mismatch and subspace leakage?

  3. Can SAGE also be applied to dense diffusion models? If the same subspace alignment loss is introduced into a dense model, would it yield similar improvements?

  4. How much additional training time and memory overhead does SAGE introduce compared with standard CFG training? Does it still provide an advantage under equal training FLOPs or GPU-hours?

I would be happy to raise my score if the authors can adequately address these concerns.

Reviewer_ZTd7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors identify a new failure mode in MoE-based DiT models, where the use of CFG may lead to different routing decisions between the conditional and unconditional branches, resulting in performance degradation. To address this issue, the authors propose SAGE, a training-time regularizer that constrains the unconditional branch of CFG to lie within the subspace of the conditional branch, rather than explicitly enforcing consistency between the Expert Routing decisions of the two CFG branches. Experimental results demonstrate that this method can effectively improve the generation performance of MoE models under high CFG scales.

Critical strengths
  1. A valuable observation. The authors identify the problem that CFG in MoE diffusion models is prone to performance degradation at higher Guidance Scales, revealing a phenomenon worthy of attention.
  2. A simple and effective method. The proposed SAGE method has a simple design, requires no modifications to the model architecture, and introduces no additional inference overhead.
  3. Relatively comprehensive ablation experiments. Through ablation experiments and toy experiments, the authors largely validate the observed phenomenon and demonstrate the effectiveness of SAGE.
  4. Relatively comprehensive appendix analysis. The appendix provides detailed theoretical analysis and derivations, along with extensive additional experimental results, further supporting the main conclusions of the paper.
Critical weaknesses
  1. Insufficient comparative analysis with related methods. Although the authors provide an experimental comparison with ProMoE in Appendix B.3, given that ProMoE is closely related to the research problem addressed in this paper, I suggest including this comparison in the main text and further analyzing the differences between the two methods and their respective advantages.
  2. Subspace Leakage is central to this paper, but there is no direct experimental evidence demonstrating the relationship between Subspace Leakage and performance degradation in MoE models.
Questions for authors

For a paper that identifies such a clear phenomenon and provides an effective solution, I am very glad to have had the opportunity to review it. However, I still have some questions that I hope can be further analyzed in future work.

  1. Could SAGE compromise the capability of individual Experts? An individual Expert may have a greater representational capacity, but the proposed SAGE training constrains its capability within a subspace.
  2. Could the training of SAGE potentially be averaged out? For example, consider Expert 1, where the constraints imposed by Expert 2 and Expert 3 on Expert 1 may conflict with each other, causing the training of Expert 1 to be averaged out.
Reviewer_xTmjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that the instability (collapse) issue of CFG is caused by the subspace-misalignment between the activation of conditional route and the unconditional route of diffusion-based MOE. Specifically, the difference between conditional velocity and unconditional velocity that lies outside the subspace spanned by the conditional route will scale linearly with the guidance strength, therefore pushing the guided activations outside of the conditional subspace, leading to unwanted artifacts. To address this issue, they propose SAGE, a training-based algorithm that enforces the unconditional route to lie in the subspace spanned by the conditional route. Empirically, they show that SAGE leads to a better best-case model performance when CFG is enabled.

Critical strengths
  1. The algorithm is well-motivated and intuitive: The subspace perspective of CFG is quite intuitive and interesting.

  2. Empirical improvement seems substantial: The proposed algorithm leads to a decent improvement on both GenEval and DPG-Bench.

Critical weaknesses
  1. Lack of theoretical justification: The paper argues that restricting the subspace of unconditional route is beneficial, which lacks theoretical justification. It is possible that the unique subspace of the unconditional route contains important information which should not be simply discarded. Moreover, how the proposed algorithm changes the behavior of the standard CFG is not theoretically justified.

  2. Lack of empirical experiments on better models: The proposed algorithm SAGE is only tested on the models trained by the authors themselves. From the provided information, those base models seem not very good, as GenEval score of 54.77 indicates poor performance based on my experience. It is not clear whether SAGE can be beneficial when being applied to stronger base models, such as Stable diffusion or FLUX.

Questions for authors
  1. Why is it necessary to restrict the unconditional route to be in the same subspace as the conditional one? In theory, the unconditional score need not lie in the subspace of the conditional one, right?

  2. What is the theoretical consequence of constraining the unconditional route to be in the subspace of the conditional one? There are many works studying the theoretical properties of CFG [1-4]. How does SAGE change those theoretical results? Why is SAGE beneficial?

  3. Can you fine-tune sota diffusion models using SAGE? Can SAGE bring benefit in those strong models?

  4. SAGE can be applied to non-MOE diffusion models as well, can you test if SAGE is a universal method that can improve the model's performance whether it is a MOE or not?

[1] Yuchen Wu, Minshuo Chen, Zihao Li, Mengdi Wang, and Yuting Wei. Theoretical insights for diffusion guidance: A case study for gaussian mixture models. In Forty-first International Conference on Machine Learning, 2024.

[2] Muthu Chidambaram, Khashayar Gatmiry, Sitan Chen, Holden Lee, and Jianfeng Lu. What does guidance do? a fine-grained analysis in a simple setting. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.

[3] Li, Xiang, Rongrong Wang, and Qing Qu. "Towards understanding the mechanisms of classifier-free guidance." Advances in Neural Information Processing Systems 38 (2026): 62835-62889.

[4] Li, Gen, and Yuchen Jiao. "Provable efficiency of guidance in diffusion models for general data distribution." arXiv preprint arXiv:2505.01382 (2025).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.