- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work analyzes register tokens for Vision Transformers, which have been characterized in the literature on discriminative vision models, in the context of (pixel-space) diffusion models. The work shows that, somewhat surprisingly, inclusion of register tokens improves the performance of pixel-space diffusion models even though the pathological high-norm outlier tokens do not appear in these models in the fist place. Further analysis shows that register tokens act both as high-norm attention sinks and global semantic aggregators; at the patch token level, inclusion of registers leads to smoother feature representations and lower activation norms. The work offers the interpretation that the training objective of diffusion models enforces strong local representations, therefore not allowing for outliers to emerge. Register tokens then alleviate the tension between local and global encoding. Finally, a new method is presented, in which non-register models are used in an AutoGuidance framework to further improve the generation performance of models with registers.
Critical strengths
This work is very insightful. The benefit of including register tokens despite the absence of outlier tokens is an interesting and novel finding. The experiments to support this are rigorous and convincing. The analysis of why this is the case in 2.2/2.3 is also strong and offers results that are not only applicable to pixel-space models, but the literature on register tokens in general. The connection to in-context conditioning that is drawn in 2.6 offers an interesting perspective on recent work that is well-supported by the experiments. Overall, the insights provided by the analysis section of the paper are strong and, in my opinion, form an important contribution to the field. The presentation of the findings is very clear, which supports the informative character of the work. The new method for generation based on register models seems interesting in so far it gives a clear candidate model for guidance. This may be particularly attractive because the differences between models with and without registers have been explored, yielding a somewhat interpretable pipeline.
Critical weaknesses
I do not have any major reservations about this work. The set of models tested in the analysis section is somewhat small; only DINOv2 is used for ViT models and the analysis (beyond performance findings) is limited to one model. The work relies quite heavily on references to the appendix. I think the camera-ready version could benefit from making use of the extra space to mitigate this as much as possible.
Questions for authors
-
Could you clarify in the text (around l. 203) whether TV is sensitive to the overall activation norm? We already know here that register reduce the norm. If TV just captures the same effect, the additional insight might be limited. To make a claim that the activations are actually more/less noisy, I think a norm-invariant measure would be good.
-
The title of section 2.3, as well as the very first paragraph of section 3 could benefit from a rewrite.
3 comments
Invitation to discussion
Dear Reviewer 21bB,
We look forward to the discussion. In our rebuttal, we have addressed your questions, including the norm-invariant TV and additional feature quality metrics, the rewritten Section 2.3 title and first paragraph of Section 3, and the analysis across more models. All of these changes are now included in the revised paper. We would be glad to know whether our response addresses your concerns, and we are happy to provide any further details.
Thank you for the response. I maintain my view that the work would be an interesting contribution to the conference.
Reply
We sincerely thank the reviewer for the positive evaluation and thoughtful feedback. We address each point below.
Due to space constraints, the main text presents the analysis for the B/16 model. The appendix extends this analysis to larger models, higher resolution, latent-space models, few-step models and text-to-image models. In the revision, we will use the extra space to move the key results of these analyses into the main text.
Thank you for this important question. The TV in the paper is computed on raw patch tokens and is therefore sensitive to the overall activation norm. To remove this dependence, we additionally compute a norm-invariant TV, where each patch token is normalized to unit L2 norm before computing TV.
In addition, we compute other feature quality metrics, which are also invariant to the activation norm, to provide more evidence that registers lead to better features. Specifically, we consider
We present the results via anonymous link at [LINK] and find a pattern similar to that of the TV in the paper. At high noise levels, registers improve all of these metrics, making the features smoother and better spatially organized.
We also provide more PCA visualizations at [LINK1], [LINK2], [LINK3], where the features of the model with registers are visibly less noisy.
We will clarify this in the revision.
Thanks for the suggestion. We will improve clarity in the revision.
We believe these clarifications address the reviewer questions, and we are happy to incorporate them into the manuscript or provide any further details as needed.