acceptodds
Withdrawn from ICLR 2027

Taming Outlier Tokens in Diffusion Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study outlier tokens in Diffusion Transformers (DiTs) for image generation. Prior work has shown that Vision Transformers (ViTs) can produce a small number of high-norm tokens that attract disproportionate attention while carrying limited local information, but their role in generative models remains underexplored. We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines: pretrained ViT encoders can produce outlier representations, and DiTs themselves can develop internal outlier tokens, especially in intermediate layers. Moreover, simply masking high-norm tokens does not improve performance, indicating that the problem is not only caused by a few extreme values, but is more closely related to corrupted local patch semantics. To address this issue, we introduce Dual-Stage Registers (DSR), a register-based intervention for both components: trained registers when available, recursive test-time registers otherwise, and diffusion registers for the denoiser. Across ImageNet and large-scale text-to-image generation, these interventions consistently reduce outlier artifacts and improve generation quality. Our results highlight outlier-token control as an important ingredient in building stronger DiTs.

Primary area:
generative models
Keywords:
Diffusion Transformers, representation learning, outlier tokens, massive activation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DBBrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies outlier tokens in diffusion transformers and observes that they occur in both pretrained vision encoders and diffusion models. The authors hypothesize that these outliers reflect degraded patch-level semantics rather than merely abnormally large feature norms. Based on this hypothesis, they introduce register-based interventions for both the vision encoder and the diffusion transformer. Experiments show that these interventions improve training stability and generation quality across ImageNet generation and large-scale text-to-image generation.

Critical strengths
  1. The proposed register-based interventions are simple and effective, consistently improving training stability and generation quality across different model variants and generation settings.

  2. The method is evaluated on both ImageNet generation and large-scale text-to-image generation, demonstrating its applicability across different tasks and model configurations.

Critical weaknesses
  1. The motivation of the proposed method is not sufficiently justified. The authors hypothesize that outlier tokens reflect degraded patch-level semantics, but the evidence provided is not sufficient to establish this relationship. In particular, the authors use the improvements brought by their register-based interventions to support this hypothesis, while these improvements could potentially arise from other mechanisms. This makes the proposed insight less convincing, as the effectiveness of the method does not necessarily validate the underlying hypothesis that motivates it.

  2. With unclear motivation, the novelty appears limited. The proposed approach primarily adapts register-based interventions from prior work to diffusion models, while the underlying motivation remains insufficiently established. Consequently, it is unclear whether the proposed method offers a novel insight beyond applying an existing technique to a new setting.

Questions for authors

Could the authors provide more direct evidence that outlier tokens are responsible for the degradation of patch-level semantics? In particular, how can the authors distinguish the proposed explanation from alternative mechanisms through which register-based interventions may improve feature representations?

Reviewer_hop9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies outlier tokens in RAE encoders and corresponding DiTs and finds that, for SigLIP2-based RAEs, the outliers are present in both the encoder and the DiT. The authors propose Dual-Stage Registers (DSR) to combine test-time encoder registers with learnable diffusion registers. The results show improvements from DSR on ImageNet and text-to-image RAE setups.

Critical strengths
  1. The paper explores an interesting question about outlier emergence in RAE-DiTs. The SigLIP2 analysis is convincing, with layer-wise diagnostics, encoder/denoiser ablations, and patch-level probes providing useful evidence beyond generation metrics.

  2. The results and analysis highlight the need to mitigate encoder outliers if a DiT is to be trained in its representation space. This might be important because, even though ImageNet RAE used a DINO encoder w/ registers as its main encoder, ScaleRAE used SigLIP2, which still has outliers.

Critical weaknesses
  1. SigLIP2 representations already contain outliers, which diffusion simply seems to inherit. Diffusion outliers after SigLIP2 + TTR (Figs. 5 and 9) are significantly suppressed. Most empirical gains also come from encoder TTR, with little additional benefit from diffusion registers. For ScaleRAE, there is no ablation which registers provide most gains.

  2. For the strongest original DINOv2-based RAE setup (which already includes encoder registers), Figure 4 and concurrent work [1] suggest that DiTs largely avoid outliers in that case. Confirming that, Table 6 shows small FID improvements: 1.51 → 1.49 without guidance and 1.23 → 1.17 with guidance. Appendix F reports a larger FDr⁶ improvement, from 3.92 to 3.25. However, [2] reports 3.26 FDr⁶ and 1.16 FID for vanilla RAE-XL, without FD-loss post-training. It essentially matches DSR’s 3.25 FDr⁶ and 1.17 FID. Therefore, DSR does not show any gains in the main RAE ImageNet setup.

Overall, these results and analysis lead to a straightforward conclusion: avoid outliers for RAE encoders. In other words, better representations of the encoders w/ registers are also beneficial for DiT training.

Also, I find it important that the paper can lead to the conclusion that diffusion registers are needed for RAE-DiTs, which is not well supported by the current results. (That is the reason why I put a rating 1 instead of 2.)


[1] Registers Matter for Pixel-Space Diffusion Transformers, 2026

[2] Representation Fréchet Loss for Visual Generation, 2026

Questions for authors

Please address the concerns in Weaknesses.

In particular, I would be happy to see additional evidence on whether diffusion registers are needed when the base encoder does not exhibit outliers in the first place. If they are not, I would encourage the authors to revisit the DSR approach and its central claims, and/or discuss the specific cases in which both sets of registers might be needed.

In that case, I would also consider framing the motivation around the observation that an effective RAE encoder should avoid outliers in the first place. Otherwise, addressing them during DiT training may be non-trivial, as supported by the loss-masking analysis.

Reviewer_ARwNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles an interesting topic of outlier tokens in Diffusion Transformers. These tokens have unusally large feature norms. They appear in pretrained image encoders and inside diffusion models. The authors propose "Dual Stage Registers" (DSR). The method adds auxilliary tokens to both the encoder and the diffusion model. These tokens help reduceoutlier behavior. The experiments cover ImageNet generation and T2I generation. The results show that DSR reduces FID while improves the GenEval performance. The authors suggest that registers improve generation by preserving local image information.

Critical strengths
  1. DSRs improve tested models that operate on pixels and different latent representations. The method also shows an improvement in the T2I scenario. This is seen in ImageNet where the FID improves from 3.85 to 3.26 after 800 epochs.
  2. The improvements are also seen in the longer training scenario. In the SigLIP configuration there is a reduction in FID after training for 800 epochs.
  3. The evaluation also includes inmage distributions in 6 feature spaces. The full DSR config improves all six distances over the DINOv2 baseline with encoder registers.
Critical weaknesses
  1. The loss masking experiment does not test whether large activations affect generation. Removing selected tokens from the loss means that their prediction errors are no longer in the training objective. However these tokens still remain in the model. Other tokens can still attend to them. The results shown do not rule out the effect from these large activations.
  2. The Encoder registers reduce the FID score from 5.89 to 4.63 in the main configuration. Adding diffusion registers reduces this to 4.58 which is a very small gain. The authors should report variability across independent training seeds to establish whether the improvement is consistent.
  3. The authors show that the Diffusion registers increase the XL-model FLOPs by 10%. The baseline therefore uses less computation when both models train for the same number of steps. An ablation with the same total training compute would help establish the benefit of diffusion registers.
Questions for authors
  1. Table 11 shows that in-context conditioning performs better than diffusion registers for JiT-H. Does the JiT-H baseline already include in-context conditioning?
  2. Table 12 shows that retraining the decoder improves rFID but degrades generation FID. Were the same diffusion checkpoint and generated latents used for each decoder evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.