acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Momentum Teachers for Pixel-Space Generative Self-Alignment

Abstract

Generative self-alignment methods such as Self-Representation Alignment (SRA) and Self-Flow supervise a generator's intermediate features with targets from a momentum teacher, an exponential-moving-average (EMA) copy of the network. SimSiam showed that a stop-gradient and a predictor can replace this copy for representation learning; we ask whether it is needed for pixel-space image generation. We call the construction without it SiamFlow: both noisy views pass through one network, and the target path, which sees the cleaner view and supplies deeper features, receives no gradient. On ImageNet with JiT-L/16, SiamFlow obtains FID , compared with for the same recipe with an EMA teacher and without alignment. Taking the targets from the current network instead of the EMA copy also lowers FID in the SRA recipe adapted to JiT, and SiamFlow lowers FID at JiT-XL/16 and at . The preferred teacher depends on the recipe: with an MLP + LN projector and neither variance regularization nor token standardization, the online run's FID oscillates and the EMA teacher gives lower FID; adding variance regularization reverses the ranking. Cross-noise instance retrieval ranks the two teachers in the opposite order to FID across training, so this retrieval criterion does not predict the teacher choice for generation. Every model is still sampled with EMA weights.

Primary area:
generative models
Keywords:
Self-supervised Learning, Generative Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AAPGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces , a self-alignment training method for pixel-space diffusion models. It replaces the EMA targets of the prior latent-space works with online targets, while introducing additional variance regularization techniques to mitigate training difficulties. The experiments on ImageNet-256 show that its improvement on model performance is decent.

Critical strengths
  1. The motivation is easy to follow, and the overall presentation is clear.
  2. The performance of SiamFlow on small-size model is on par with PixelREPA, which relies on external models.
Critical weaknesses

My primary concern is that, given the current state of the paper, the main motivation appears to be simply attempting self-alignment training in pixel space and introducing several heuristics to improve the FID score. And the resulting performance gains seem marginal and decrease monotonically as the model parameter count increases (Table 3).

As pointed out by the authors, certain techniques from latent-space self-alignment methods do not directly transfer to pixel-space training. Specifically, the introduction mentions that prior latent-space works favor EMA targets over online targets. Given this, the paper shoul focus more on explaining the role of EMA targets and why they can be simplified using variance regularization. Additionally, does this regularization also benefit latent-space methods? What are the key takeaways for self-alignment training in pixel space, and how much further can these methods be simplified? I'm not asking for more experiments but providing potential directions for refining the paper.

Minors on writings and presentations

  1. To properly position the paper within the literature, I suggest providing a deeper discussion of the technical differences between the proposed method and relevant prior work, rather than merely introducing them sequentially.
  2. In line 159, I suggest using distinct symbols like ss and tt to denote different timesteps rather than using tt with subscripts. In line 180, the authors should define the meaning of hsh_s and hth_t. For completeness, it would also be helpful to explain the core intuition behind self-alignment and why it works. Finally, Section 6.3 is currently confusingly written and should be revised to eliminate ambiguity.
Questions for authors
  1. What does "Gap" signify in Table 1?
  2. Could you clarify what is meant by "cross-noise retrieval"?
  3. Compared to the JiT baseline, the performance gains decrease noticeably on larger models. Considering this, how much does SiamFlow improve over latent-space self-alignment approaches (such as SRA or Self-Flow) when scaled to larger models (e.g., 676M parameters)?
1 comment

I thank the authors for their detailed responses. After carefully reading the rebuttal, the additional experiments address several of my methodological questions. However, they do not substantially address my concerns.

  1. Feature alignment, token standardization, etc, are established ideas, and the contribution is mainly careful adaptation to pixel-space generative training based on JiT. The contribution is therefore primarily an empirical adaptation to pixel-space self-alignment. The gains are also modest. With evidence limited to JiT on ImageNet, the broader insights of the paper remains unclear.

  2. The current state of the paper reads more like an expanded rebuttal than a coherent study, making it difficult to identify the primary contribution and takeaway.

Therefore, I decide to maintain my current weak rejection assessment.

Reviewer_waBrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a method of training a flow-based generative model using an auxiliary feature alignment loss derived from an online version of the network, as opposed to EMA, that gives marginal gains in FID over select baselines on ImageNet 256. Based on provided ablation studies the proposed method necessitates the use of a variance regularizer from "Variance-invariance-covariance regularization for self-supervised learning" by Adrien Bardes etal. in order to achieve the reported results.

Critical strengths
  • The authors effectively show that online feature alignment can be performed for pixel space DiT architectures, they select the JiT methodology proposed in "Back to Basics: Let Denoising Generative Models Denoise" by Tiagnhong Li and Kaiming He, and offers marginal gains over related EMA based approaches. The design of the method is also supported by various ablations provided in the text that illustrates the effect of each design decision. This work offers practitioners yet another method for improving image quality of generative models.
Critical weaknesses

I have several minor critiques of the papers writing along with a few concerns surrounding the proposed methodology in the experiment section.

  • Beginning with some minor stylistic concerns: The formatting of the introduction and contributions section can be improved to avoid the page break. The authors make use of the term "branches" in several places referring to different parts of the computational graph given in the text. I would suggest the authors edit these occurrences to specifically reference, using proper notation, the exact terms and computations. The authors also state in the text "[...] so VV has zero upstream gradient in exact arithmetic when the numerical clamp is inactive." which I find confusing as the notion of an "upstream gradient" is meaningless. I suspect the authors meant here that VV is expected to have zero gradient norm provided the standardization used in eq(1) is accurate.

  • As the method proposed in the text is preliminary stated with respect to flow-matching generative models, at least based on eq(2), it is disappointing to find so little discussion (e.g,. background and related work) on flow matching itself. For example, the loss term L\mathcal{L}_{FM} in eq(2) is not defined within the text. It would benefit the presentation of the paper to include a brief background section on flow-matching with relevant citations.

  • The content of Section.3.1, while informative, seems unrelated to the method being proposed in the paper. While the optimal velocity field and feature maps are certainly related to how the optimization objective in eq(2), which is the one used to train the proposed model, it is not clear to me how this information was used to inform the choice of objective or how the authors mitigated the miss-alignment between the feature alignment minimizer and the flow matching loss minimizer. The authors should explain, in clear terms, in the text the purpose of including this information and its relevance to reported results. Perhaps the authors intended to show the gap between feature alignment using and EMA copy of the network is larger than that from the online copy of the network used in the paper? This is unclear.

  • In regards to the primary point of the paper, a method of using an online version of the model (as opposed to an EMA copy) for feature alignment and improving image generation performance, in order for the method to be applied by other practitioners it is necessary that the authors provide guidelines for selecting the different hyperparameters used in the method; namely, the student and teacher feature depths, feature alignment weight λ\lambda, the regularization weight λvar\lambda_{var}, etc. Preferential selection for some of these is addressed in the ablation section, for instance the importance of variance regularization and feature standardization, however, I found the discussion around selecting the relative depths uninformative. Selecting the relative depths by performing a hyperparamter sweep is quite costly and does not offer the reader any initial guess on which configuration to start with for dataset other ImageNet 256.

  • I have a concern about the EMA ablation performed in Section.6. As I understood the experimental setup, for different EMA values, the authors are using an EMA copy in place of the online network in their proposed method, as opposed to using an EMA teacher model of the baseline JiT architecture with unmodified self-representation alignment (SRA). If this is the case, then in my option the ablation does not address the question posed in this section. Namely, do the improvements in FID mainly result from the online network being used in place of an EMA copy or a combination of the tricks presented? In order to answer this question, the setup would need to be altered to the one I mentioned above and would need to show as the EMA value is lowered the baseline SRA FID result remains worse than the authors method.

Questions for authors

See comments about critical weaknesses.

Reviewer_YqmBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper revisits the role of momentum/EMA teachers in generative self-alignment for pixel-space generative transformers. It proposes SiamFlow, a SimSiam-style self-alignment scheme in which two differently noised views of the same image are processed by the same JiT backbone. A shallow student feature is aligned to a stop-gradient deeper target feature taken from a cleaner view. The teacher uses the current online backbone weights, not an EMA copy, while the projector is shared, token standardization is applied, and a student-only predictor is used. The training objective combines flow matching, cosine feature alignment, and a variance regularizer. The main empirical claims are on ImageNet with JiT-L/16. The paper also reports improvements over the JiT baseline at larger model sizes and higher image resolution. And the authors investigate several design choices for online self-alignment, including projector architecture, variance regularization, feature standardization, and student–target alignment depth. They further examine the effects of EMA decay on target discrepancies and gradient differences, and analyze the relationship between cross-noise retrieval performance and generation quality.

Critical strengths

(1) SiamFlow is conceptually simple: use the current generator as the target branch with stop-gradient, keep a shared projector and a student-only predictor, and add token standardization/variance control for stability. No external encoder and no extra modules are needed at sampling, which seems practically attractive.

(2) The paper shows that the choice of student and target layers has a substantial effect on generation quality. In particular, aligning block 4 to block 18 produces FID 2.13, whereas same-layer alignment at block 4 produces FID 2.91. Aligning to the final target block is also less effective than using an intermediate target layer. These results suggest that the usefulness of alignment targets depends on the hierarchical roles of intermediate generator representations, rather than simply increasing with network depth.

Critical weaknesses

(1) The paper's main claim is that online teachers outperform EMA teachers under matched alignment configurations. However, the reported results do not include variability across independent training seeds or confidence intervals. This concern becomes more important for smaller differences reported in the ablation studies, such as the 0.02 FID difference between the convolutional configurations with and without the variance penalty. Although Figure 4 shows that the online model outperforms its EMA counterpart at multiple checkpoints, these measurements come from correlated training trajectories and do not substitute for independent repetitions.

(2) The paper evaluates SiamFlow on multiple JiT settings, including different model sizes and image resolutions. While these experiments demonstrate useful scalability within JiT, they do not establish generality across pixel-space generative architectures. Furthermore, the larger-model and higher-resolution experiments compare SiamFlow with the corresponding JiT baseline, but do not include matched online-versus-EMA teacher comparisons. Therefore, the paper provides evidence that SiamFlow improves JiT across the tested configurations, but not that the online teacher advantage itself persists across model scales and resolutions.

(3) The core mechanism—stop-gradient target branch and a student-only predictor without a momentum encoder—is essentially the SimSiam idea, and the paper cites it. SRA and Self-Flow use EMA teachers; swapping to an online teacher is a natural and useful empirical idea, but it is not a fundamentally new principle. The novelty is therefore mainly in the application and ablation, not in the conceptual mechanism.

Questions for authors

(1) The authors need to provide multi-seed results for the key L/16 comparisons. At minimum, report mean and standard deviation over 2–3 seeds for baseline JiT, SiamFlow + EMA, and SiamFlow online. If multi-seed runs are too expensive, report the variance of FID evaluation and clarify which differences are within noise.

(2) Since both teacher variants use a shared online projector, to what extent might the observed difference reflect the interaction between the teacher backbone and projector dynamics? Would the conclusions differ if the EMA branch also used an EMA projector?

(3) Given that EMA outperforms online alignment in the unregularized MLP setting, what conditions determine when an online teacher is preferable? Is the main conclusion intended to establish the non-necessity of EMA, rather than the general superiority of online teachers?

(4) Why does aligning block 4 to block 18 outperform both same-layer alignment and alignment to the final block? Can the authors provide an interpretation based on the roles of intermediate features or existing activation statistics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.