acceptodds
Under review as a conference paper at ICLR 2027

Intrinsic Distribution Matching

20 views

Abstract

One- and few-step text-to-image generation must preserve fine-grained prompt fidelity with limited opportunities for refinement. Distribution matching distillation (DMD) guides a generator using the score difference between generated and reference distributions. Following prior work, we estimate this difference with internal velocity branches, avoiding a separate fake-score network. We ask whether an explicit moment constraint improves this learned score supervision. A decomposition of the reverse-KL field into a Gaussian moment component and a non-Gaussian residual identifies statistics that can be constrained explicitly. We introduce **Intrinsic Distribution Matching (IDM)**, which retains the full DMD correction and matches feature means and per-coordinate standard deviations using the generator's exponential-moving-average (EMA) copy, requiring no external feature encoder. Experiments use full-parameter and low-rank adaptation (LoRA) tuning on Z-Image-Turbo 6B and Qwen-Image 20B. At two network evaluations, IDM reaches 94.25% and 90.20% on Chinese and English LongText-Bench, the highest scores among the compared methods at this budget. Controlled ablations show gains from the added moment loss, while generator EMA features perform competitively with an external DINOv2 encoder across the five reported metrics.

Primary area:
generative models
Keywords:
Distribution Matching, Momentum Matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eV5mOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains 2 NFE text-to-image generators with an internal DMD, where the real and fake velocities are two condition branches of one network, Fθ(xt,t,c)F_\theta(x_t,t,c) and Fθ(xt,t,−c)F_\theta(x_t,t,-c), following APEX.

It adds Lpop\mathcal L_{\mathrm{pop}}, the squared W2W_2 distance between diagonal Gaussians fitted to mean-pooled features of the generator's EMA copy. This term is motivated by splitting the reverse-KL field into a Gaussian (mean/covariance) part and a residual. Validation: full and LoRA tuning of Z-Image-Turbo 6B and Qwen-Image 20B on about 5.6M mostly synthetic images, reaching 94.25 / 90.20 on LongText-CN/EN at 2 NFE, plus GenEval, DPG, OneIG and an ablation grid.

Critical strengths
  1. Practical at scale. Extensive experiments at 6B and 20B large-scale models are conducted and proved to be effective at two-stage generation.
  2. A detailed controlled ablation. Loss removal, regression weighting, and four feature maps (EMA, DINOv2, teacher, live) are ablated, so the advantage and disadvantage of each design in this paper can be read directly from Table 2.
Critical weaknesses
  1. The empirical contribution of proposed loss is marginal
  • The gain of the proposed pop loss is really marginal, according to the ablation studies in Table 2. Most of the gain even exists before the DMD and moment loss, and the base loss alone already is better than the other two-step baselines on the long text bench.
  • As the abstract and introduction emphasize a lot for the long-text benchmark, there is no clear explanation on what is the main contributing factor to this excellent long-text benchmark performance, and what is the role of the proposed loss?
  • Also the external comparisons are unfair considering the data (5M text-rendering images vs. APEX's 200K) and training steps (10K vs. 2K on Qwen-Image).
  1. The connection from motivation to theory to practice is not very clear and convincing.
  • The Gaussian-plus-residual split is in sample space and used to describe the score function, whereas Lpop\mathcal L_{\mathrm{pop}} is in EMA-feature space.
  • And in the end, the loss is almost the same as the FD-loss proposed in existing work. While the authors propose to change the feature extractor to an EMA version of the generator, there is no explanation on why this choice is better. Also, for the ema feature extractor, all the features extracted by a moving/changing feature extractor, and we calculate the mean and variance of these features, Which makes it hard to understand.
Questions for authors
  1. How do matched-data APEX-style and DMD2 baselines perform with and without Lpop\mathcal L_{\mathrm{pop}}?
  2. Do we use equation 8 somewhere in the paper?
  3. What's the difference between L-pop and L-pop hat?
Reviewer_2sntOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

For one/few-step text-to-image distillation, the paper introduces IDM, which combines with a feature-moment loss matching the mean and per-coordinate standard deviations of features from the generator's EMA copy between generated and reference samples, accumulated in 16K-entry detached queues. Experiments are done with Z-Image-Turbo 6B (full + LoRA) and Qwen-Image 20B (LoRA + full), evaluated at 1–2 NFE on a range of tests, with controlled ablations over losses, feature maps, and regression weights.

Critical strengths
  1. The method is clean and self-contained. Both the score difference and the features come from the generator and its EMA, without external encoder, discriminator, or other tricks. This is nice and parsimonious.
  2. The evaluation done on LongText-Bench is demanding. IDM performs well at 2 NFE on Z-Image (94.25 CN, 90.20 EN), ahead of Z-Image-Turbo++ and APEX.
  3. LoRA (r=32/64) on the 20B backbone retains most performance at a fraction of the parameters.
Critical weaknesses
  1. The novelty is not clear to me. The new loss matches pooled feature means and per-coordinate standard deviations. It would be nice to show mechanistically why first/second moments of a pooled feature distribution should improve fine-grained, per-prompt text rendering.
  2. The SOTA claim is confounded by training data. IDM trains on a text-focused collection (~5M text-rendering + 0.6M FLUX-Reason prompts, plus ShareGPT-4o/BLIP-3o), whereas the APEX reference used ~200K poster prompts. This is an unfair comparison. It would be nice to use the same training data. Otherwise, one cannot be certain of the algorithmic contribution.
  3. Moment matching could plausibly affect diversity. I’d like to see some kind of diversity metric for IDM.
Questions for authors

Please see weakness.

Reviewer_t1VFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Intrinsic Distribution Matching (IDM), which adds a feature-moment objective that matches feature means and pre-coordinate standard deviations extracted by an exponential-moving-average (EMA) copy of the generator. The paper demonstrates improvements for one- and few-step text-to-image generation, especially for the tasks of text rendering.

Critical strengths
  1. The authors propose and validate a feature-moment regularization method based on representations from an EMA copy of the generator to enhance existing internal DMD training.
  2. The design of Lpop\mathcal{L}_{\mathrm{pop}} is simple and straightforward, and the paper provides detailed theoretical proofs and clear design rationales.
  3. The paper provides detailed and informative ablation experiments, evaluating the removal of loss terms, different regression formulations, and alternative feature extractors under a shared training configuration.
  4. The method shows a notable improvement in text rendering, achieving 94.25 on LongText-CN compared with APEX’s 90.38.
Critical weaknesses
  1. Although the method achieves 94.25 on LongText-CN, it scores 86.82 on DPG-Bench compared with APEX’s 88.64, suggesting a potential trade-off in general text-to-image generation performance.
  2. Although the method achieves scores of 94.25 on LongText-CN and 90.20 on LongText-EN, the training data used in this paper contains a substantial amount of text-rendering data. The authors should provide sufficient evidence or additional experiments to demonstrate that the improvements on LongText-Bench are not primarily attributable to the training data rather than the proposed method. For example, the authors could train an APEX-style baseline using the same training data as IDM to enable a fair comparison.
  3. The paper does not sufficiently justify the choice of EMA features.
  4. Appendix D.5 defines the auxiliary fake-velocity loss. However, its update frequency remains unclear.
Questions for authors
  1. Please clarify the rationale for using features from an EMA copy of the generator rather than those extracted using the copy of generator itself.
  2. Please analyze the reasons for the lower DPG-Bench score. While the qualitative results suggest advantages in text rendering, please provide more side-by-side visual comparisons with other methods on both text rendering and general text-to-image generation tasks to better demonstrate the method’s strengths.
  3. Please provide sufficient evidence or additional experiments to demonstrate that the improvements on LongText-Bench are not primarily attributable to the training data rather than the proposed method. For example, the authors could train an APEX-style baseline using the same training data as IDM to enable a fair comparison.
  4. Please provide further details on the optimization of Lfake\mathcal{L}_{\mathrm{fake}}.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.