acceptodds
Withdrawn from ICLR 2027

MaskAlign: Token-Subset Representation Alignment for Efficient Diffusion Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Representation alignment with pretrained vision models has recently shown strong potential for accelerating diffusion transformer training. By aligning intermediate diffusion features with representations of clean images from pretrained vision encoders, existing methods improve convergence and generation quality. Recent studies further suggest that the effectiveness of representation alignment is closely related to spatial structure. In this paper, we study representation alignment across different samples. We find that, when representation alignment is applied to all tokens, tokens with large alignment gradient norms exhibit a stable spatial preference across different samples, suggesting that the alignment objective does not affect all tokens uniformly and may encourage the model to fit spatial patterns associated with the encoder rather than visual features specific to each image, which may interfere with the learning of representations that reflect image content. To address this issue, we propose MaskAlign, a representation alignment method that applies alignment to randomly sampled token subsets during training. By exposing the model to different token subsets across iterations, MaskAlign reduces the repeated reinforcement of stable spatial preferences and encourages the model to learn features that are more closely related to individual images. To mitigate the information loss caused by directly dropping tokens, we further introduce a lightweight token mixing block that shares information across tokens before masking. Experiments on ImageNet show that MaskAlign consistently improves training convergence and generation quality. On SiT-XL/2, applying MaskAlign to REPA reduces FID from 7.9 to 5.5 at 400K iterations and from 6.4 to 4.9 at 1M iterations. When applied to REG, MaskAlign reduces FID from 3.4 to 2.8 at 400K iterations and from 2.7 to 2.4 at 1M iterations, while reducing training time per step by 11.7%. For REG, the improvement also holds when FID is computed using DINOv2-L, CLIP-L, and VGG16 features, both without and with classifier-free guidance.

Primary area:
generative models
Keywords:
Diffusion Models, Representation Alignment, Training Acceleration, Token Masking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X8i2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper observes that under full-token REPA/REG alignment, high-gradient patch tokens concentrate at the same, encoder-specific spatial positions across images. It proposes MaskAlign: two extra SiT blocks as a pre-mask token mixer, then random dropping of 25% of patch tokens during training, with both denoising and alignment losses computed on visible tokens. On ImageNet 256 (SiT-XL/2, 400K, w/o CFG), FID improves 7.9 → 5.5 (REPA) and 3.4 → 2.8 (REG), with 11.7% lower per-step time.

Critical strengths
  1. Consistent gains at lower cost. Improvements hold across REPA/REG, three alignment encoders, multiple budgets, and multiple FID feature spaces, while per-step time decreases.
  2. New diagnostic. The cross-sample analysis of alignment gradients is simple and novel.
Critical weaknesses
  1. Mechanism not isolated. The method masks the backbone input and the denoising loss, not just alignment, making it essentially MicroDiT (mixer + masking) on top of REG. In Table 4, masking alone (3.54) and mixing alone (3.20) are both worse than the baseline (3.01); only the combination helps (2.67). The key missing ablation is alignment-only masking with a full-token forward pass, plus comparisons to MicroDiT/TREAD.
  2. Weak causal evidence. The heatmap patterns (grid stripes, corner token) resemble known DINOv2 artifact/high-norm tokens, which is not discussed. Nothing links reducing the pattern to the FID gain. GrG_r (Eq. 9) is circular, since a mask-trained model is naturally robust to masking. There are no seeds or error bars (CFG gains like 1.59 → 1.56 may be noise), and no parameter-matched baseline despite ~8% more parameters.
Questions for authors
  1. Results with alignment-only masking (full-token forward pass and denoising loss)? (Key to my assessment.)
  2. Why does mixing without masking hurt (3.01 → 3.20)? Multi-seed results?
  3. Is Fig. 2b computed with all tokens? Please quantify concentration and relate it to DINOv2 artifact tokens.
  4. Were CFG settings identical to REG in Table 2? Why does IS drop (321.4 → 310.0)?
  5. Wall-clock time or FLOPs to reach the same FID?
Reviewer_KKerOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MaskAlign modifies representation-aligned diffusion training by adding two token-mixing blocks and randomly dropping patch tokens before the main backbone. Both the prediction and alignment losses use the retained tokens. The motivation is that full-token alignment produces persistent spatial concentration in its gradients. Experiments on ImageNet 256×256 show better generation quality with REPA and REG, together with a reduction in REG training time per step. Inference uses all tokens and retains the mixing blocks.

Critical strengths
  1. The unguided improvements are substantial.

  2. The controls go beyond a single encoder or metric.

Critical weaknesses
  1. The proposed explanation is not isolated by the experiments. Persistent high-gradient positions need not represent harmful encoder preferences; image layout could also contribute. More directly, the intervention changes the backbone's input tokens and prediction loss as well as alignment supervision (in sec 5.1). Table 4 supports the recipe, but it does not tell us how much of the gain is specific to representation alignment rather than masked diffusion training more generally.

  2. The small guided improvements are difficult to assess. Table 2 reports changes from 1.86 to 1.80 and from 1.59 to 1.56 for REG. The PDF does not give repeated-run uncertainty or explicitly describe the guidance scales and how they were selected. These details matter more for differences this small than for the larger unguided gains.

Questions for authors
  1. How much does the same mixing-and-masking intervention help when the alignment loss is disabled? A matched comparison would help establish whether the benefit depends on representation alignment.

  2. In Figure 2, are both trained models probed with the same full-token inputs? If masking is active during measurement, how are omitted positions handled in the top-10% statistic? This affects the interpretation of the flatter heatmap.

  3. What guidance settings and selection procedure were used in Table 2, and how do its small FID differences compare with training or evaluation variability?

Reviewer_f5r5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MaskAlign: Token-Subset Representation Alignment for Efficient Diffusion Training

Critical strengths

The reported unguided FID improvements are useful. At 400K steps, REPA improves from 7.9 to 5.5, and REG from 3.4 to 2.8. REG’s training time per step decreases by 11.7%.

Critical weaknesses
  1. The gains are limited on stronger baselines and under CFG. Without CFG, REG improves from 2.7 to 2.4 FID at 1M iterations, while the SiT-B setting improves only from 15.2 to 14.8. With CFG, REG improves from 1.86 to 1.80 at 80 epochs and from 1.59 to 1.56 at 160 epochs. These small guided gains are not supported by repeated runs or uncertainty estimates. Although the unguided REPA improvement is substantial, the evidence for a meaningful advantage in stronger, guided settings is much less convincing.

  2. The experiments do not establish an alignment-specific contribution. REG grows from 677M to 732M parameters, with two additional blocks retained at inference. Meanwhile, masking changes both backbone inputs and denoising supervision. The existing component ablation does not distinguish improvements from general masked training from those attributable to correcting alignment bias. Lower training time is useful, but does not establish the claimed mechanism or justify the additional inference complexity.

Questions for authors

Does masking only the alignment loss improve performance with the backbone and denoising loss unchanged? How does the method compare with matched deferred-masking baselines? What controlled experiment establishes that the observed spatial preferences are harmful encoder artifacts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.