acceptodds
Under review as a conference paper at ICLR 2027

Registers Matter for Pixel-Space Diffusion Transformers

Abstract

Vision Transformers (ViTs) are known to exhibit high-norm patch-token outliers that degrade feature map quality, a problem effectively mitigated by register tokens. As diffusion models increasingly adopt transformer architectures and move toward pixel-space training, they become closer in form to ViTs, raising the question of whether register tokens are also useful for Diffusion Transformers (DiTs). In this work, we show that DiTs differ from ViTs in a key respect: they do not exhibit patch-token outliers but still benefit from registers. Interestingly, registers are more effective in pixel-space DiTs than in latent-space DiTs. By analyzing intermediate representations, we find that register tokens produce cleaner feature maps at high noise levels, which may contribute to their effectiveness in pixel-space generation. We further observe that recent pixel-space DiT architectures implicitly incorporate register-like mechanisms, which may partially account for their strong empirical performance. Motivated by these observations, we propose Register Guidance, a technique that amplifies the contribution of register tokens responsible for improving visual structure and coherence.

Primary area:
generative models
Keywords:
image generation, diffusion models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_21bBOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work analyzes register tokens for Vision Transformers, which have been characterized in the literature on discriminative vision models, in the context of (pixel-space) diffusion models. The work shows that, somewhat surprisingly, inclusion of register tokens improves the performance of pixel-space diffusion models even though the pathological high-norm outlier tokens do not appear in these models in the fist place. Further analysis shows that register tokens act both as high-norm attention sinks and global semantic aggregators; at the patch token level, inclusion of registers leads to smoother feature representations and lower activation norms. The work offers the interpretation that the training objective of diffusion models enforces strong local representations, therefore not allowing for outliers to emerge. Register tokens then alleviate the tension between local and global encoding. Finally, a new method is presented, in which non-register models are used in an AutoGuidance framework to further improve the generation performance of models with registers.

Critical strengths

This work is very insightful. The benefit of including register tokens despite the absence of outlier tokens is an interesting and novel finding. The experiments to support this are rigorous and convincing. The analysis of why this is the case in 2.2/2.3 is also strong and offers results that are not only applicable to pixel-space models, but the literature on register tokens in general. The connection to in-context conditioning that is drawn in 2.6 offers an interesting perspective on recent work that is well-supported by the experiments. Overall, the insights provided by the analysis section of the paper are strong and, in my opinion, form an important contribution to the field. The presentation of the findings is very clear, which supports the informative character of the work. The new method for generation based on register models seems interesting in so far it gives a clear candidate model for guidance. This may be particularly attractive because the differences between models with and without registers have been explored, yielding a somewhat interpretable pipeline.

Critical weaknesses

I do not have any major reservations about this work. The set of models tested in the analysis section is somewhat small; only DINOv2 is used for ViT models and the analysis (beyond performance findings) is limited to one model. The work relies quite heavily on references to the appendix. I think the camera-ready version could benefit from making use of the extra space to mitigate this as much as possible.

Questions for authors
  1. Could you clarify in the text (around l. 203) whether TV is sensitive to the overall activation norm? We already know here that register reduce the norm. If TV just captures the same effect, the additional insight might be limited. To make a claim that the activations are actually more/less noisy, I think a norm-invariant measure would be good.

  2. The title of section 2.3, as well as the very first paragraph of section 3 could benefit from a rewrite.

3 comments

We sincerely thank the reviewer for the positive evaluation and thoughtful feedback. We address each point below.


W1. The analysis (beyond performance findings) is limited to one model.

Due to space constraints, the main text presents the analysis for the B/16 model. The appendix extends this analysis to larger models, higher resolution, latent-space models, few-step models and text-to-image models. In the revision, we will use the extra space to move the key results of these analyses into the main text.


Q1. Could you clarify in the text (around l. 203) whether TV is sensitive to the overall activation norm?

Thank you for this important question. The TV in the paper is computed on raw patch tokens and is therefore sensitive to the overall activation norm. To remove this dependence, we additionally compute a norm-invariant TV, where each patch token is normalized to unit L2 norm before computing TV.

In addition, we compute other feature quality metrics, which are also invariant to the activation norm, to provide more evidence that registers lead to better features. Specifically, we consider

  • Correlation Decay Slope (CDS), the slope of the cosine similarity between patches as a function of their spatial distance. Lower values mean stronger spatial organization.
  • Local vs. Distant Similarity (LDS), the mean similarity of nearby patches minus the mean similarity of distant patches. Higher values mean stronger spatial organization.
  • Semantic-Region Self-Similarity (SRSS), the similarity between patches of the same object minus the similarity between object and background patches. Higher values mean that the features better separate the object from the background.
  • Spatial alignment with DINOv2, the Pearson correlation between the patch similarity matrix of the model and that of DINOv2 for the same image. Higher values mean that the spatial structure of the features is closer to that of DINOv2.

We present the results via anonymous link at [LINK] and find a pattern similar to that of the TV in the paper. At high noise levels, registers improve all of these metrics, making the features smoother and better spatially organized.

We also provide more PCA visualizations at [LINK1], [LINK2], [LINK3], where the features of the model with registers are visibly less noisy.

We will clarify this in the revision.


Q2. The title of section 2.3, as well as the very first paragraph of section 3 could benefit from a rewrite.

Thanks for the suggestion. We will improve clarity in the revision.


We believe these clarifications address the reviewer questions, and we are happy to incorporate them into the manuscript or provide any further details as needed.

AuthorsOpenReview ↗

Invitation to discussion

Dear Reviewer 21bB,

We look forward to the discussion. In our rebuttal, we have addressed your questions, including the norm-invariant TV and additional feature quality metrics, the rewritten Section 2.3 title and first paragraph of Section 3, and the analysis across more models. All of these changes are now included in the revised paper. We would be glad to know whether our response addresses your concerns, and we are happy to provide any further details.

Thank you for the response. I maintain my view that the work would be an interesting contribution to the conference.

Reviewer_4LDpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the role of register tokens in Diffusion Transformers (DiTs), particularly pixel-space DiTs. The authors observe that register tokens can improve generation quality even when pronounced high-norm patch-token outliers are absent. They further analyze how registers affect intermediate representations, compare their effectiveness across pixel-space and latent-space models, and propose Register Guidance (RG) to improve image generation.

Critical strengths

The paper provides an empirical study across different model architectures, sizes, and training spaces. The comparison between pixel-space and latent-space DiTs is particularly interesting. The effectiveness of register tokens is supported by quantitative results, along with ablation studies on the number of registers and their insertion layers. Register Guidance is straightforward to implement and demonstrates improvements in generation quality. Its compatibility with CFG is also useful.

Critical weaknesses

1.Register Guidance is quite similar to AutoGuidance. The main difference is using the model without registers as the weaker counterpart. I would like to see a more direct comparison with AutoGuidance and other guidance methods, especially considering the additional inference cost. The paper briefly discusses this, but the comparison could be more thorough.

2.Explanation of why registers help. The authors mainly use feature norms and Total Variation to argue that registers produce better intermediate representations. However, smoother features do not necessarily mean better representations. The connection between these observations and the actual improvement in generation quality is still not very clear.

3.Discussion of high-norm outliers. One of the main observations is that DiTs do not exhibit high-norm patch-token outliers, while Appendix C.2 shows that such outliers do appear in FLUX and PixArt-α. Although they disappear in later layers, this suggests that the behavior may be architecture-dependent. I think this deserves a clearer discussion in the main paper.

4.Inference cost. For larger models, the gains are sometimes fairly small. For example, the FID of JiT-H/16 improves from 1.86 to 1.80. It is not clear whether this improvement is worthwhile compared with simply using more sampling steps or other guidance strategies at a similar computational cost.

Questions for authors

seed weakness

5 comments
AuthorsOpenReview ↗

Reply [1/3]

We sincerely thank the reviewer for their support and thoughtful questions. We address each of them below.


W1. Register Guidance is quite similar to AutoGuidance. I would like to see a more direct comparison with AutoGuidance and other guidance methods, especially considering the additional inference cost.

Comparison with AutoGuidance. As discussed in the paper, we tried to apply AG to JiT but could not make it work. Other works support this observation. [1] report that AG does not improve over CFG for JiT, and [2] find that internal guidance even degrades JiT-B/16.

We therefore believe that AG and other AutoGuidance-like approaches are not straightforward to adapt to pixel-space DiTs, since they were originally developed and tested mainly on latent-space models. We hypothesize that pixel-space generation with DiTs is more sensitive to the choice of the weak model, which makes a suitable one harder to find. This motivated our design of RG, in which a single backbone provides both the positive and the negative directions, so that the two directions are better aligned with each other.

We report the results of AG with different variants of the weak model, using smaller models and earlier training checkpoints, and carefully tune the guidance scale for each of them. None of these setups outperforms CFG or our RG.

MethodFID
ReferenceH/16L/16B/16
CFG1.862.363.66
RG (ours)1.852.003.16
AG, guided modelWeak modelw = 1.25w = 1.5w = 2.0w = 2.5
H/16L/166.376.0412.7537.74
H/16B/165.214.269.0230.76
L/16B/1610.908.315.956.56
B/16B/16, 50 ep–59.7168.1985.45
B/16B/16, 100 ep–63.4867.8376.89
B/16B/16, 200 ep–65.2569.4175.64


Visualizations of AG samples are also available at [AG]. At small scales AG barely changes the samples, and at larger scales it introduces high-frequency stripe artifacts in fur, foliage and water, which explains the sharp increase in FID.

[1] A. Baade, E. R. Chan, K. Sargent, C. Chen, J. Johnson, E. Adeli, L. Fei-Fei. Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation.
[2] Z. Pan, M. Chen, C. Shi, T. Hu, Z. Yue, Z. Wang, Y. Wang, W. Li, S. Yang. Learning Representations to Guide Pixel Diffusion Transformers.


Comparison with training-free guidance methods. As with AG, we found that other AutoGuidance-like methods do not work well for JiT. We consider PAG [1] and SLG [2], which obtain the weak prediction by replacing the self-attention of one block with the identity or by skipping the block. Both methods perform far worse than CFG and our RG.

MethodFID-50K
CFG (w = 3.0)3.66
Ours, RG (w = 3.5)3.16
Ours, CFG (w = 2.0) + RG (w = 1.8)2.96
PAG (perturbed block)w = 1.5w = 2.0w = 2.5w = 3.0
PAG, block 321.7733.6743.6552.94
PAG, block 410.536.766.517.03
PAG, block 510.228.589.8111.44
PAG, block 610.9010.4211.9513.64
PAG, block 714.0518.7923.5526.94
SLG (skipped block)w = 1.5w = 2.0w = 2.5w = 3.0
SLG, block 333.2541.9047.5752.44
SLG, block 416.6821.0223.7725.24
SLG, block 515.0117.5822.6428.60
SLG, block 614.4617.8821.9125.32
SLG, block 711.1013.7218.2422.44


Additionally, we provide visualizations of PAG and SLG via anonymous links [PAG], [SLG]. As with AG, both methods produce clear artifacts. Several blocks lead to corrupted or heavily artifacted samples (0, 2, 6, 8, 10 for PAG; blocks 0, 8, 10 for SLG), while the others produce blurry results (block 4 for PAG; blocks 2, 4, 6 for SLG).

[1] Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim, SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attention guidance. In ECCV, 2024.
[2] Junha Hyung, Kinam Kim, Susung Hong, Min-Jung Kim, and Jaegul Choo. Spatiotemporal skip guidance for enhanced video diffusion sampling. In CVPR, 2025.

AuthorsOpenReview ↗

Reply [2/3]

W1. Register Guidance is quite similar to AutoGuidance.

Comparison with SSG. To our knowledge, the only AG-like approach shown to work for JiT is the concurrent SSG [1]. Similar to our method, SSG requires additional training to obtain the negative model. The positive and negative predictions in SSG also share the same backbone. This design supports our hypothesis that pixel-space generation is more sensitive to the choice of the negative model.

We provide the comparison below. First, without CFG, our approach significantly outperforms SSG for all model sizes and resolutions. Second, in combination with CFG, our approach outperforms CFG+SSG for the B and L models at both resolutions, while it underperforms for the H/16 model.

We also note that SSG trains an additional adapter on top of the frozen model and therefore uses more parameters, whereas RG introduces no additional parameters.

Without CFGWith CFG
ModelCFGSSGRG (ours)CFG + SSGCFG + RG (ours)
Forward passes per step21 + adapter22 + adapter3
JiT-B/16, 2563.669.473.163.292.96
JiT-L/16, 2562.364.472.002.232.00
JiT-H/16, 2561.862.261.851.701.80
JiT-B/32, 5124.0213.373.623.673.34
JiT-L/32, 5122.534.512.242.312.23
JiT-H/32, 5121.942.732.011.841.87

We address the inference cost concern raised in this weakness in our response to W4.

We will include everything mentioned above in the revision.

[1] Z. Fu, C. Wang, L. Guo, K. Zhou, J. Nie, B. Wen. A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples.


W2. Explanation of why registers help. The authors mainly use feature norms and Total Variation to argue that registers produce better intermediate representations. However, smoother features do not necessarily mean better representations.

Beyond showing that registers lead to smoother, less noisy representations, we evaluate additional feature quality metrics from iREPA [1] that measure spatial organization, and find that registers also improve this property at high noise levels.

Thus, registers lead to less noisy features with better spatial organization, and prior work associates both properties with better FID. REPA [2] shows less noisy feature maps of SiT-XL/2 at high noise levels together with better FID (Fig. 38). LSEP [3] reports the same effect without an external encoder (Fig. 8). In pixel space, [4] find noisy, high-frequency features in JiT-B and improve FID with more structured features (Fig. 7). iREPA [1] measures the spatial structure of 27 target representations and finds that it correlates with FID (Fig. 4).

More specifically, we additional consider the following metrics:

  • TV on unit-norm tokens. The same TV as in the paper, computed after normalizing each patch token, so it does not depend on the feature norm.
  • Correlation Decay Slope (CDS), the slope of the cosine similarity between patches as a function of their spatial distance. Lower values mean stronger spatial organization.
  • Local vs. Distant Similarity (LDS), the mean similarity of nearby patches minus the mean similarity of distant patches. Higher values mean stronger spatial organization.
  • Semantic-Region Self-Similarity (SRSS), the similarity between patches of the same object minus the similarity between object and background patches. Higher values mean that the features better separate the object from the background.
  • Spatial alignment with DINOv2, the Pearson correlation between the patch similarity matrix of the model and that of DINOv2 for the same image. Higher values mean that the spatial structure of the features is closer to that of DINOv2, which REPA [2] and iREPA [1] use as a target for better generation.

We report the ratio (with / without registers) for every block and noise level at [LINK]. All metrics show the same trend. At high noise levels (t ∈ [0, 0.2]) registers make the features smoother and improve their spatial organization

We also provide more PCA visualizations at [LINK1], [LINK2], [LINK3], where the features of the model with registers are visibly less noisy.

[1] J. Singh et al. What matters for Representation Alignment: Global Information or Spatial Structure?
[2] S. Yu et al. Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think.
[3] J. Yun, Y. U. Alcalar, M. Akcakaya. No Alignment Needed for Generation: Learning Linearly Separable Representations in Diffusion Models.
[4] Z. Pan et al. Learning Representations to Guide Pixel Diffusion Transformers.

AuthorsOpenReview ↗

Reply [3/3]

W3. Discussion of high-norm outliers. One of the main observations is that DiTs do not exhibit high-norm patch-token outliers, while Appendix C.2 shows that such outliers do appear in FLUX and PixArt-α. Although they disappear in later layers, this suggests that the behavior may be architecture-dependent.

We believe that the outliers observed in FLUX and PixArt-α do not contradict our observation. High-norm outliers among patch tokens appear only inconsistently and are much weaker than in ViTs, which is not enough to confirm that they exist systematically.

First, patch-token outliers appear in only two of the five text-to-image models we analyze, while SD3.5, Qwen-Image and Z-Image show uniform image-token norms. In ViTs, in contrast, such outliers appear consistently across different models. Second, only one or a few image tokens become outliers (a single token in PixArt-α), whereas in ViTs about 2% of patch tokens become outliers. Third, these outliers appear only in intermediate layers and disappear in later layers, and in FLUX their norms are much smaller than those of the text-token outliers. In ViTs, in contrast, outliers persist until the output.

Regarding architecture dependence, SD3.5 and Qwen-Image share the MMDiT design with FLUX but show no patch-token outliers, so the architecture alone does not explain them. We therefore consider these cases as exceptions rather than a systematic behavior.

We will add this discussion to the main paper.


W4. Inference cost. For larger models, the gains are sometimes fairly small (FID of JiT-H/16 improves from 1.86 to 1.80). It is not clear whether this improvement is worthwhile compared with simply using more sampling steps or other guidance strategies at a similar computational cost.

First, we would like to note that for H/16 our approach shows a considerably stronger improvement in terms of FDr6 (6.10 vs 7.66), a metric that captures more aspects of image quality than FID [1].

Indeed RG+CFG needs more network evaluations than CFG. For a comparison at similar or higher cost, we raise the number of CFG steps to 60, 75 and 90 (238, 298 and 358 NFE) and report FID-50K. We find that additional steps do not improve CFG for any checkpoint. The reported configurations already use the optimal number of steps, so the improvement of RG does not come from the additional inference cost.

ConfigurationStepsNFEFID-50K
B/16
CFG, w = 3.0501983.66
CFG, w = 3.0602383.74
CFG, w = 3.0752983.69
CFG, w = 3.0903583.70
RG+CFG, w = 1.8 / 2.0502972.96
H/16
CFG, w = 2.2501981.86
CFG, w = 2.2602381.93
CFG, w = 2.2752981.97
CFG, w = 2.2903582.00
RG+CFG, w = 1.1 / 1.4502971.80


In practice, a higher inference cost can be acceptable when it brings better quality, especially since diffusion models are often distilled afterwards. As shown in [2] guided sampling can be distilled into a single network that requires one forward pass per step. So, improving the base model is therefore the main goal, and the extra evaluations can be removed at the distillation stage.

To verify this, we apply the guidance distillation from [2] to the B/16 model and distill RG+CFG, which requires three forward passes, into a single conditional forward pass. The student is initialized from the teacher model and trained for 9.6 epochs on ImageNet. This reduces the sampling cost from 297 to 99 NFE, while FID changes only from 2.96 to 2.98.

GuidancePasses per stepNFEFID-50K
CFG21983.66
RG+CFG32972.96
RG+CFG, distilled1992.98

[1] Jiawei Yang, Zhengyang Geng, Xuan Ju, Yonglong Tian, and Yue Wang. Representation Fréchet loss for visual generation. arXiv preprint arXiv:2604.28190, 2026.
[2] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In CVPR, 2023.


We will add all the discussion in the revision. We believe these clarifications address the reviewer questions, and we are happy to incorporate them into the manuscript or provide any further details as needed.

Reviewer_J5AfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper investigates whether and why register tokens benefit diffusion transformers. The authors find that registers improve generation quality, especially in pixel space, despite the absence of patch-token outliers. Their analyses suggest improved spatial organization of intermediate features, while generation examples show more coherent structures with similar image content. Motivated by these observations, Register Guidance (RG) uses predictions without registers as a weaker reference to amplify the benefits of registers during sampling. RG improves ImageNet generation across model sizes and resolutions and complements classifier-free guidance, although combining the two requires additional model evaluations and increases the cost of image generation.

Critical strengths
  • The finding that registers help DiTs even without patch-token outliers is interesting, and the larger gain in pixel space holds across model sizes, resolutions and a shared backbone.

  • The comparison with JiT shows that most of the benefit of its in-context class tokens comes from having extra tokens rather than from the class information, which helps explain an existing design choice.

Critical weaknesses
  • PAG (Ahn et al., 2024) and Skip Layer Guidance (SD3.5) already improve generation by contrasting normal and internally degraded predictions, without additional training. RG's methodological contribution is the choice of register removal as the degradation, but neither alternative is compared. The benefit of this choice remains unclear, particularly given the register-drop adaptation required for pretrained models.

  • The analyses do not establish whether lower norms, smoother features, or global information in registers explain the FID gains. The proposed regularization explanation for the larger pixel-space gains is also untested.

  • The claim that registers help only in deeper layers rests on pDiT-B ablations up to 120 epochs. Table 18 also reports an improvement from early insertion with four registers (FID 11.1 to 10.4), contradicting this blanket claim.

Questions for authors
  • How does RG compare in generation quality with existing methods that use degraded predictions for guidance? What advantages does register removal offer over other forms of degradation, considering any differences in training and inference cost?

  • What evidence supports the proposed explanation for the FID gains, beyond observing that feature properties change when registers are added? Does it also explain why the gains are larger in pixel space?

  • How general is the advantage of introducing registers in later layers? Do the results support early insertion being ineffective, or only delayed insertion performing better in the tested settings?

5 comments
AuthorsOpenReview ↗

Reply [1/2]

We sincerely thank the reviewer for the important concerns raised. We address each of them below.


W1, Q1. PAG (Ahn et al., 2024) and Skip Layer Guidance (SD3.5) already improve generation without additional training. RG’s methodological contribution is the choice of register removal as the degradation, but neither alternative is compared.

We provide a direct comparison between RG and PAG, SLG. For both PAG and SLG, we carefully tune the perturbed layer and the guidance scale on the B/16 model. However, neither PAG nor SLG reaches the quality of CFG or RG in any of the considered configurations.

MethodFID-50K
CFG (w = 3.0)3.66
Ours, RG (w = 3.5)3.16
Ours, CFG (w = 2.0) + RG (w = 1.8)2.96
PAG (perturbed block)w = 1.5w = 2.0w = 2.5w = 3.0
PAG, block 321.7733.6743.6552.94
PAG, block 410.536.766.517.03
PAG, block 510.228.589.8111.44
PAG, block 610.9010.4211.9513.64
PAG, block 714.0518.7923.5526.94
SLG (skipped block)w = 1.5w = 2.0w = 2.5w = 3.0
SLG, block 333.2541.9047.5752.44
SLG, block 416.6821.0223.7725.24
SLG, block 515.0117.5822.6428.60
SLG, block 614.4617.8821.9125.32
SLG, block 711.1013.7218.2422.44


To our knowledge, these approaches were not tested on pixel-space DiTs. We find that it is not straightforward to apply them to these models. We hypothesize that the pixel space is more sensitive, and thus choosing the negative model can be harder than in latent space. For the same reason, RG requires additional training and use a shared backbone to obtain a suitable negative model.

Our observation is further supported by other works that apply AutoGuidance-like approaches to JiT. [1] report that AutoGuidance does not improve results compared to CFG for JiT, and [2] find that internal guidance even degrades JiT-B/16. SSG [3] obtains gains only after training an additional adapter on top of the frozen model.

Thus, a key difference and methodological contribution of RG is that it brings AutoGuidance-like approaches to pixel-space DiTs, where, unlike in latent-space models, applying them is not straightforward. We believe that our results can be interesting for future work on adapting training-free guidance approaches to pixel-space DiTs.

Additionally, we provide visualizations of PAG and SLG via anonymous links: [PAG], [SLG]. Several blocks lead to corrupted or heavily artifacted samples (0, 2, 6, 8, 10 for PAG; blocks 0, 8, 10 for SLG), while the others produce blurry results (block 4 for PAG; blocks 2, 4, 6 for SLG).

We will include PAG and SLG in the related work, as well as all the above discussions and results, in the revised version.

[1] A. Baade, E. R. Chan, K. Sargent, C. Chen, J. Johnson, E. Adeli, L. Fei-Fei. Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation.
[2] Z. Pan, M. Chen, C. Shi, T. Hu, Z. Yue, Z. Wang, Y. Wang, W. Li, S. Yang. Learning Representations to Guide Pixel Diffusion Transformers.
[3] Z. Fu, C. Wang, L. Guo, K. Zhou, J. Nie, B. Wen. A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples.

AuthorsOpenReview ↗

Reply [2/2]

W2, Q2. The analyses do not establish whether lower norms, smoother features, or global information in registers explain the FID gains. The proposed regularization explanation for the larger pixelspace gains is also untested.

We attribute the better FID to better, less noisy representations of the model due to the use of registers. The link between FID gains and less noisy features is well established in the literature.

  • REPA [1] visualizes the intermediate features of SiT-XL/2 and shows that REPA leads to less noisy feature maps at large noise (Fig. 38) and improves FID.
  • LSEP [2] reports the same effect without an external encoder, with clearer feature components in early layers and at high noise levels (Fig. 8).
  • In pixel space, [3] find that JiT-B "shows high-frequency patterns across layers, with unclear contours and noisy object features" (Fig. 7), and their model with more structured features achieves better FID.
  • Beyond visualizations, iREPA [4] measures the spatial structure of 27 target representations and finds that it correlates with FID (Fig. 4).

Additional evidence comes from latent spaces.

  • REPA-E [5] finds that SD-VAE latents "have high-frequency noise" and that end-to-end tuning reduces this noise while improving FID (Figs. 2, 6),
  • EQ-VAE [6] obtains smoother latents and 7x faster DiT convergence (Fig. 1),
  • [7] show that suppressing excessive high-frequency components in latents reduces FID by 19% (Figs. 2-6, Table 1).


Registers produce a similar effect in pixel-space DiTs. They lead to smoother feature maps, mainly at high noise levels (Fig. 2 of the paper), which is the regime where REPA and LSEP report the largest difference. We provide additional PCA visualizations of intermediate features with and without registers via anonymous links: [LINK1], [LINK2], [LINK3]

We also evaluate more feature quality metrics from iREPA [4] and find that, with registers, the features have better spatial structure at high noise levels [LINK]. This provides more evidence that registers lead to better features. We will include the above discussion and references in the revision.

[1] S. Yu et al. Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. ICLR 2025.
[2] J. Yun, Y. U. Alcalar, M. Akcakaya. No Alignment Needed for Generation: Learning Linearly Separable Representations in Diffusion Models. arXiv 2025.
[3] Z. Pan et al. Learning Representations to Guide Pixel Diffusion Transformers. arXiv 2026.
[4] J. Singh et al. What matters for Representation Alignment: Global Information or Spatial Structure? ICLR 2026.
[5] X. Leng et al. REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers. arXiv 2025.
[6] T. Kouzelis et al. EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling. ICML 2025.
[7] I. Skorokhodov et al. Improving the Diffusability of Autoencoders. ICML 2025.


W3, Q3. The claim that registers help only in deeper layers rests on pDiT-B ablations up to 120 epochs. Table 18 also reports an improvement from early insertion with four registers (FID 11.1 to 10.4), contradicting this blanket claim.

The difference between 11.1 and 10.4 is marginal and is likely not statistically significant at this scale. We continued this analysis until full convergence (600 epochs) and found that early insertion brings at most a marginal gain. With 4 registers inserted from the first block, FID improves only from 4.80 to 4.71, which is not enough to claim a clear improvement from early insertion.

In contrast, deeper insertion with 32 registers improves FID from 4.80 to 3.80, a gain more than ten times larger. We therefore keep the claim that registers are most effective when introduced from deeper layers.

Registers configurationFID at Epoch
SizeStartEnd4080120600
w/ reg.3241137.79.596.453.80
3201159.719.311.94.85
401154.816.210.44.71
w/o reg.–––60.618.411.14.80

We believe these clarifications address the reviewer questions, and we are happy to incorporate them into the manuscript or provide any further details as needed.

AuthorsOpenReview ↗

Invitation to discussion

Dear Reviewer J5Af,

We look forward to the discussion. In our rebuttal, we have addressed your concerns, including the comparison with PAG, SLG, the explanation of the FID gains, the effect of register depth at full convergence. All of these results are now included in the revised paper. We would be glad to know whether our response addresses your concerns, and we are happy to provide any further details.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global reply with revised paper [1/2]

We thank the reviewers for their thorough work. Their comments help us significantly improve the paper.


Revision

We have uploaded a revised version of the paper and highlighted the changes in red. Specifically,

(i) For the reviewers 4LDp\color{firebrick}{\textbf{4LDp}} and J5Af\color{firebrick}{\textbf{J5Af}}: we have added a comparison with guidance methods that rely on a weak model, AG and the concurrent SSG (please see Table 8 and lines 509–524). AG does not outperform CFG or RG for any choice of the weak model. RG outperforms SSG for all models without CFG, and for the B and L models with CFG. We also discuss our hypothesis on why AG is not effective for pixel-space DiTs and visualize its artifacts (Figure 29).

(ii) For the reviewers J5Af\color{firebrick}{\textbf{J5Af}} and 4LDp\color{firebrick}{\textbf{4LDp}}: we have added a comparison with the training-free guidance methods PAG and SLG (please see Table 21 and lines 1345–1373). Neither method reaches the quality of CFG or RG in any configuration.

(iii) For the reviewer 4LDp\color{firebrick}{\textbf{4LDp}}: we have extended the discussion of the patch-token outliers in FLUX and PixArt-α in the main text (lines 200–202).

(iv) For the reviewer J5Af\color{firebrick}{\textbf{J5Af}}: we have added a discussion of prior work linking less noisy and better spatially organized features to generation quality (please see line 248 and lines 924–932).

(v) For the reviewers J5Af\color{firebrick}{\textbf{J5Af}}, 4LDp\color{firebrick}{\textbf{4LDp}}, 21bB\color{firebrick}{\textbf{21bB}}: we have evaluated the internal representations with additional metrics from iREPA that measure the spatial structure of the features, together with a norm-invariant TV computed on unit-norm tokens, and clarified that TV on raw features depends on the feature norm (please see lines 248–250 and Figure 30). At high noise levels, registers improve these metrics, leading to smoother and better spatially organized features.

(vi) For the reviewer J5Af\color{firebrick}{\textbf{J5Af}}: we have extended the ablation of register configurations to 600 epochs (please see Table 19, line 1298). At convergence, early insertion of registers brings at most a marginal gain (4.80 vs. 4.71), while deeper insertion improves FID from 4.80 to 3.80.

(vii) For the reviewers 4LDp\color{firebrick}{\textbf{4LDp}} and J5Af\color{firebrick}{\textbf{J5Af}}: we have added a discussion of the inference cost (please see Table 20 and lines 1336–1344). With guidance distillation, CFG+RG is distilled into a single forward pass per step. At the same cost of 99 NFE, the distilled CFG+RG model achieves an FID of 2.98, compared with 3.60 for the distilled CFG model.

(viii) For the reviewer 21bB\color{firebrick}{\textbf{21bB}}: we have rewritten the title of Section 2.3 (line 253) and the first paragraph of Section 3 (lines 365–368).

(ix) For the reviewer 21bB\color{firebrick}{\textbf{21bB}}: we have moved the attention-map visualizations from the Appendix into the main text (Figure 2, lines 120–122). The figure compares several models, two ViTs (ViT and DINOv2) and three DiTs (RAE, SiT and pDiT), and shows that, unlike ViTs, DiTs do not exhibit attention-map outliers.

AuthorsOpenReview ↗

Global reply with revised paper [2/2]

Positive things

We appreciate the reviewers positive feedback on various aspects, including the novel findings, the insightful analysis, the simple and effective method, the rigorous experiments and the clear presentation. We summarize these points below.

(i). Findings:

  • Reviewer J5Af\color{firebrick}{\textbf{J5Af}}: The finding that registers help DiTs even without patch-token outliers is interesting, and the larger gain in pixel space holds across model sizes, resolutions and a shared backbone. The comparison with JiT shows that most of the benefit of its in-context class tokens comes from having extra tokens rather than from the class information, which helps explain an existing design choice.
  • Reviewer 4LDp\color{firebrick}{\textbf{4LDp}}: The comparison between pixel-space and latent-space DiTs is particularly interesting.
  • Reviewer 21bB\color{firebrick}{\textbf{21bB}}: This work is very insightful. The benefit of including register tokens despite the absence of outlier tokens is an interesting and novel finding. The analysis of why this is the case is also strong and offers results that are not only applicable to pixel-space models, but the literature on register tokens in general. The insights provided by the analysis section of the paper are strong and form an important contribution to the field.

(ii). Method:

  • Reviewer J5Af\color{firebrick}{\textbf{J5Af}}: RG improves ImageNet generation across model sizes and resolutions and complements classifier-free guidance.
  • Reviewer 4LDp\color{firebrick}{\textbf{4LDp}}: Register Guidance is straightforward to implement and demonstrates improvements in generation quality. Its compatibility with CFG is also useful.
  • Reviewer 21bB\color{firebrick}{\textbf{21bB}}: The new method gives a clear candidate model for guidance. This may be particularly attractive because the differences between models with and without registers have been explored, yielding a somewhat interpretable pipeline.

(iii). Experiments:

  • Reviewer 4LDp\color{firebrick}{\textbf{4LDp}}: The paper provides an empirical study across different model architectures, sizes, and training spaces. The effectiveness of register tokens is supported by quantitative results, along with ablation studies on the number of registers and their insertion layers.
  • Reviewer 21bB\color{firebrick}{\textbf{21bB}}: The experiments to support this are rigorous and convincing. The connection to in-context conditioning offers an interesting perspective on recent work that is well-supported by the experiments.

(iv). Paper writing:

  • Reviewer J5Af\color{firebrick}{\textbf{J5Af}}: Clear and easy to follow.
  • Reviewer 21bB\color{firebrick}{\textbf{21bB}}: The presentation of the findings is very clear, which supports the informative character of the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.