- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the degradation in generation quality of MoE diffusion models under high CFG scales. The authors attribute this issue to subspace leakage caused by routing mismatch between the conditional and unconditional branches. To address this, they propose SAGE, which introduces a subspace alignment regularizer during training while preserving routing diversity. The effectiveness of SAGE is validated through a 1B-parameter text-to-image model.
Critical strengths
-
This paper investigates the interaction between MoE routing and CFG, highlighting the performance degradation of MoE diffusion models under high CFG scales. This is an interesting and practically relevant problem that provides a valuable research direction for understanding MoE-based generative models.
-
The proposed method aligns activation subspaces rather than enforcing routing agreement, improving generation quality while preserving expert routing diversity, without introducing additional inference overhead.
-
The experiments on a 1B-parameter text-to-image model demonstrate clear improvements on DPG-Bench, GenEval, and FID, particularly showing better stability under high CFG scales. The paper also provides a comprehensive set of ablation studies.
Critical weaknesses
-
The authors characterize subspace leakage as a MoE-specific issue. However, different expert routing decisions do not necessarily lead to different activation subspaces. Conversely, even when the conditional and unconditional branches select the same experts, their outputs may still lie in different subspaces due to differences in their input hidden states. Therefore, the causal relationship between routing mismatch and subspace leakage is not rigorously established. In my understanding, the linear growth of leakage with the CFG scale follows directly from the definition of the orthogonal projection residual, which does not necessarily demonstrate that routing mismatch is the primary cause of generation quality degradation at high CFG scales.
-
The current experiments mainly demonstrate the effectiveness of SAGE but do not sufficiently establish whether its improvements come from addressing MoE-specific routing mismatch. For example, experiments applying the same SAGE loss to other architectures with comparable parameter counts are missing.
-
SAGE requires additional paired conditional/unconditional forward passes and QR decomposition. However, the paper does not quantify the actual training time and memory overhead, nor does it provide comparisons under equal computational budgets.
Questions for authors
-
The linear growth of leakage with the CFG scale appears to follow directly from the mathematical definition of the projection residual. Could the authors further demonstrate how this residual affects the final image quality?
-
Could the authors provide controlled experiments to further validate the relationship between routing mismatch and subspace leakage?
-
Can SAGE also be applied to dense diffusion models? If the same subspace alignment loss is introduced into a dense model, would it yield similar improvements?
-
How much additional training time and memory overhead does SAGE introduce compared with standard CFG training? Does it still provide an advantage under equal training FLOPs or GPU-hours?
I would be happy to raise my score if the authors can adequately address these concerns.