- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This manuscript provides an investigation of 'artifact tokens', which have been discovered and discussed in the computer vision literature in recent years, and are characterized by large norm values compared to the activations of non-outlier tokens. The work can easily be summarized, as it is presented in terms of five main findings:
- Artifacts arise from the model weights alone and are not image dependent.
- Artifacts emerge only in large models.
- Register tokens that are added to alleviate the artifact problem merely move the same information outside of the patch tokens.
- Models that are trained with text-image contrastive learning depend on outlier tokens for their performance, whereas self-supervised models do not.
- Finding 4. is reproduced with different evaluation protocols.
Here, self-supervised models refer to the DINO family, and text-contrastive models correspond to CLIP and SigLIP. Methodology is from the standard deep-learning evaluation tool kit; the main contribution of the paper is therefore the insight about artifact tokens in trained models that can be derived from the findings.
Critical strengths
The presentation of the work (apart from the 2nd half of the introduction and §8, which are somewhat cumbersome for the reader) is well-presented. I especially want to highlight the good structure of the text. The main findings 1-4 are clear and accessible. More broadly, I think this is a good presentation of what is known about artifact tokens and I would recommend this paper to a person that is interested in learning about this phenomenon. I enjoyed reading F1, F4 and F5 and took important information from the text. I also appreciated the detailed report of additional negative findings, which I believe are also of interest to the community. The methodology is sound and down-to-earth; the results are therefore easy to interpret and allow for simple take-away messages.
Critical weaknesses
While the methodology is sound, the scope in terms of model variety is somewhat limited and this is not always properly reflected by the claims that are made. Examples:
CONTRASTIVE MODELS DEPEND ON ARTIFACT TOKENS (l. 310)
The contrastive models are limited to CLIP and SigLIP, and non-contrastive are DINO-family models and DeiT-III.
In image–text contrastive models, [outlier tokens] carry image-discriminative content, and removing them degrades every evaluator we tested. In self-distillation models, they carry no such content but distort the representation. (l. 413f.)
The analysis is limited to DINO models and CLIP. Here, even SigLIP is missing from Fig. 4. This may lead the reader to wonder whether SigLIP properly displays the effect claimed for image-text contrastive models. Overall, these claims appear overly general given the experimental results. I think the work would be strengthened substantially by including more model families (iBOT, SigLIP2, DINOv3, MAE, ...).
F2 and F3 have been shown in [1] and should not be claimed as novel results. The tension between distance-based evaluation and learned-probe evaluation (l. 415f) has also been discussed there. The work is not cited here, so I assume the the authors were not aware of it. I would argue that this prior work would be important to discuss here, and novelty claims should be appropriately calibrated.
[1] Register and [CLS] tokens induce a decoupling of local and global features in large ViTs. Alexander Lappe, Martin A. Giese. NeurIPS 2025
Questions for authors
I have no further specific questions. I do think the paper has a lot of merit and am open to discussion. I think the concerns I raised above are actionable, and if they are addressed I would increase my score.
4 comments
Follow-up — prior work calibrated, model panel extended
Thank you again for the Lappe & Giese reference. We have read it, agree it covers F2 and F3, and rewritten both sections as corroboration by independent measurement rather than as novel findings, with a Related Work paragraph and the novelty claims removed. We state precisely what remains ours: matched random-ablation controls, a third scale, and the attention-mass decomposition.
On model variety, SigLIP is now in Figure 4 and the full layer sweep, and we added the supervised AugReg checkpoint, which does not fit the two-family framing and is the reason F5 is narrowed. We were not able to add iBOT, SigLIP2 or DINOv3 within the discussion window on a single consumer GPU, and have scoped the claims to the tested checkpoints rather than implying generality.
Thank you for the response and running additional experiments. I would like to point out that SigLIP still appears to be missing in Fig. 4 and and Tab. 5 (and so is the new AugReg). Based on your response it seems like that may be unintentional.
In any case, the findings appear to be complicated by the inclusion of AugReg, which narrows the claim about the clear split between contrastive text-image models and their complement. I still think that the manuscript is quite interesting; while it has been improved by better contextualization in terms of prior work, the narrowing of the claims hurts the potential impact. I would ask the authors to comment on the inclusion of new models in the tables/figures mentioned above before posting a final assessment.
Figure 4 corrected; on the narrowing
Thank you, you are right, and it was unintentional. We added the paired SigLIP and AugReg measurements as a new appendix table but did not regenerate Figure 4 from them, so the figure still carried the earlier four-model set. Table 5 in the submitted version was the pooling-operator table, which does include SigLIP; it moved to the appendix in the revision, so the number now points at the segmentation table instead, which likely explains the second half of what you saw.
Figure 4a is now rebuilt from the paired run, with SigLIP B/16, CLIP B/16, DINOv2 B/14 and AugReg B/16, all on identical images at each model's own peak-artifact layer, with five-seed random controls shown as markers. The revised PDF is uploaded.
| Model | pooled -NN | pooled probe | per-token -NN |
|---|---|---|---|
| SigLIP B/16 (contrastive) | (ctrl ) | ||
| CLIP B/16 (contrastive) | |||
| DINOv2 B/14 (self-dist.) | |||
| AugReg B/16 (supervised) |
On the narrowing and its effect on impact, we take the point and will not argue that the paper is as strong as submitted. One observation, though, on what AugReg does. It does not only complicate the split; it rules out the explanation we had offered for it. Before AugReg, the CLIP-versus-DINOv2 contrast was consistent with the training objective being the cause, and that is what we claimed. AugReg is supervised, carries artifacts (), and falls with the contrastive models under one readout and with DINOv2 under the other. A single-cause account is therefore excluded by measurement rather than left open, and the remaining statement, uniform sign under global readout, family-dependent sign under per-token readout, is narrower but is what the data support.
We would rather report that than keep a cleaner claim we can no longer defend, and the checkpoint that forced it was one added at a reviewer's request. F1 and F4 are unchanged, and the diagnostics, run outputs and nine documented negative results are released for reuse. We are grateful for a review that made the paper more accurate, whatever assessment follows.
Response to Reviewer Y4YJ
Thank you for the Lappe & Giese reference, which materially changed this revision, and for the constructive framing. We also posted a general response describing the two substantive corrections and the revised PDF.
On Lappe & Giese. We had not seen it. Having read it, we agree that F2 and F3 are substantially covered: their Figure 2a shows CLS-to-register attention growing with scale, and their Figure 3 shows by attention-zeroing and CKA that registers dominate the global output in large models. We have rewritten both sections as corroboration by independent measurement, added a Related Work paragraph, and removed the novelty claims. What we retain and now state precisely: matched random-ablation controls, a third scale, and the attention- mass decomposition. We also note that their Section 4 finding — that CLS acts as a register through the skip connection — supplies the mechanism for an observation of ours that was previously unexplained. We report one quantitative difference as an open question: they locate the transition between 86M and 300M measuring CLS-to-register attention, while we locate it between 22M and 87M measuring read score on high-norm patch tokens in models without registers.
On model variety. We agree the claims outran the panel. Two changes:
First, we added AugReg B/16 (supervised, artifact-bearing) to the F4 and F5 evaluations. It does not fit the two-family framing — pooled −4.4 like the contrastive models, per-token +5.6 like DINOv2 — and we report this against our own prior framing. It is the main reason F5 has been narrowed.
Second, we added SigLIP to Figure 4, which you correctly noted was missing, together with its full layer sweep across all three evaluators.
We were not able to add iBOT, SigLIP2 or DINOv3 within the discussion period; our compute is a single 12 GB consumer GPU and the evaluator sweep is the costlier part of the pipeline. We have scoped the claims to the tested checkpoints throughout rather than implying generality, and named these families as the natural next test in Limitations.
On presentation. We have rewritten the second half of the introduction and Section 8, both of which you found cumbersome. Section 8 in particular was carrying an overstated claim, and narrowing it also shortened it.