acceptodds
Under review as a conference paper at ICLR 2027

What Artifact Tokens in Vision Transformers Are, and What They Do

Abstract

Vision Transformers develop a handful of high-norm “artifact” tokens that occupy low-information image regions and absorb a disproportionate share of attention. Register tokens were introduced to eliminate them, and a subsequent audit reported that the phenomenon does not generalise uniformly across architectures. We present a measurement study across 15 pretrained checkpoints and 10 datasets spanning 9 visual domains. First, artifact formation is determined by model weights, not input statistics, as varying the fraction of low-information patches over a range changes artifact strength with a coefficient of variation of only –, while leaving the peak-artifact layer exactly invariant for every model on every dataset, including histopathology, chest radiography, and satellite imagery. Second, the behaviour emerges with scale, appearing between 22M and 87M parameters on matched DINOv2 checkpoints. Third, register tokens work by capturing the read target. Adding four registers collapses attention mass on high-norm patch tokens from to , renders those patches causally inert under ablation, and makes the registers load-bearing instead. Fourth, for image–text contrastive models, removing eight of roughly 200 tokens costs 15–21 accuracy points under every pooling operator tested, with matched random-ablation controls showing no effect. Fifth, the sign of the ablation effect depends on the training objective rather than the evaluation protocol. It remains invariant across a band of layers, three evaluators, and neighbourhood sizes spanning two orders of magnitude, while exceeding matched random controls by –. This resolves an apparent protocol dependence in which pooled and per-token evaluation yield opposite-signed conclusions about the same intervention.

Primary area:
interpretability and explainable AI
Keywords:
vision transformers, artifact tokens, interpretability, attention sinks, representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y4YJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript provides an investigation of 'artifact tokens', which have been discovered and discussed in the computer vision literature in recent years, and are characterized by large norm values compared to the activations of non-outlier tokens. The work can easily be summarized, as it is presented in terms of five main findings:

  1. Artifacts arise from the model weights alone and are not image dependent.
  2. Artifacts emerge only in large models.
  3. Register tokens that are added to alleviate the artifact problem merely move the same information outside of the patch tokens.
  4. Models that are trained with text-image contrastive learning depend on outlier tokens for their performance, whereas self-supervised models do not.
  5. Finding 4. is reproduced with different evaluation protocols.

Here, self-supervised models refer to the DINO family, and text-contrastive models correspond to CLIP and SigLIP. Methodology is from the standard deep-learning evaluation tool kit; the main contribution of the paper is therefore the insight about artifact tokens in trained models that can be derived from the findings.

Critical strengths

The presentation of the work (apart from the 2nd half of the introduction and §8, which are somewhat cumbersome for the reader) is well-presented. I especially want to highlight the good structure of the text. The main findings 1-4 are clear and accessible. More broadly, I think this is a good presentation of what is known about artifact tokens and I would recommend this paper to a person that is interested in learning about this phenomenon. I enjoyed reading F1, F4 and F5 and took important information from the text. I also appreciated the detailed report of additional negative findings, which I believe are also of interest to the community. The methodology is sound and down-to-earth; the results are therefore easy to interpret and allow for simple take-away messages.

Critical weaknesses

While the methodology is sound, the scope in terms of model variety is somewhat limited and this is not always properly reflected by the claims that are made. Examples:

CONTRASTIVE MODELS DEPEND ON ARTIFACT TOKENS (l. 310)

The contrastive models are limited to CLIP and SigLIP, and non-contrastive are DINO-family models and DeiT-III.

In image–text contrastive models, [outlier tokens] carry image-discriminative content, and removing them degrades every evaluator we tested. In self-distillation models, they carry no such content but distort the representation. (l. 413f.)

The analysis is limited to DINO models and CLIP. Here, even SigLIP is missing from Fig. 4. This may lead the reader to wonder whether SigLIP properly displays the effect claimed for image-text contrastive models. Overall, these claims appear overly general given the experimental results. I think the work would be strengthened substantially by including more model families (iBOT, SigLIP2, DINOv3, MAE, ...).

F2 and F3 have been shown in [1] and should not be claimed as novel results. The tension between distance-based evaluation and learned-probe evaluation (l. 415f) has also been discussed there. The work is not cited here, so I assume the the authors were not aware of it. I would argue that this prior work would be important to discuss here, and novelty claims should be appropriately calibrated.

[1] Register and [CLS] tokens induce a decoupling of local and global features in large ViTs. Alexander Lappe, Martin A. Giese. NeurIPS 2025

Questions for authors

I have no further specific questions. I do think the paper has a lot of merit and am open to discussion. I think the concerns I raised above are actionable, and if they are addressed I would increase my score.

4 comments

Response to Reviewer Y4YJ

Thank you for the Lappe & Giese reference, which materially changed this revision, and for the constructive framing. We also posted a general response describing the two substantive corrections and the revised PDF.

On Lappe & Giese. We had not seen it. Having read it, we agree that F2 and F3 are substantially covered: their Figure 2a shows CLS-to-register attention growing with scale, and their Figure 3 shows by attention-zeroing and CKA that registers dominate the global output in large models. We have rewritten both sections as corroboration by independent measurement, added a Related Work paragraph, and removed the novelty claims. What we retain and now state precisely: matched random-ablation controls, a third scale, and the attention- mass decomposition. We also note that their Section 4 finding — that CLS acts as a register through the skip connection — supplies the mechanism for an observation of ours that was previously unexplained. We report one quantitative difference as an open question: they locate the transition between 86M and 300M measuring CLS-to-register attention, while we locate it between 22M and 87M measuring read score on high-norm patch tokens in models without registers.

On model variety. We agree the claims outran the panel. Two changes:

First, we added AugReg B/16 (supervised, artifact-bearing) to the F4 and F5 evaluations. It does not fit the two-family framing — pooled −4.4-4.4 like the contrastive models, per-token +5.6+5.6 like DINOv2 — and we report this against our own prior framing. It is the main reason F5 has been narrowed.

Second, we added SigLIP to Figure 4, which you correctly noted was missing, together with its full layer sweep across all three evaluators.

We were not able to add iBOT, SigLIP2 or DINOv3 within the discussion period; our compute is a single 12 GB consumer GPU and the evaluator sweep is the costlier part of the pipeline. We have scoped the claims to the tested checkpoints throughout rather than implying generality, and named these families as the natural next test in Limitations.

On presentation. We have rewritten the second half of the introduction and Section 8, both of which you found cumbersome. Section 8 in particular was carrying an overstated claim, and narrowing it also shortened it.


AuthorsOpenReview ↗

Follow-up — prior work calibrated, model panel extended

Thank you again for the Lappe & Giese reference. We have read it, agree it covers F2 and F3, and rewritten both sections as corroboration by independent measurement rather than as novel findings, with a Related Work paragraph and the novelty claims removed. We state precisely what remains ours: matched random-ablation controls, a third scale, and the attention-mass decomposition.

On model variety, SigLIP is now in Figure 4 and the full layer sweep, and we added the supervised AugReg checkpoint, which does not fit the two-family framing and is the reason F5 is narrowed. We were not able to add iBOT, SigLIP2 or DINOv3 within the discussion window on a single consumer GPU, and have scoped the claims to the tested checkpoints rather than implying generality.

Thank you for the response and running additional experiments. I would like to point out that SigLIP still appears to be missing in Fig. 4 and and Tab. 5 (and so is the new AugReg). Based on your response it seems like that may be unintentional.

In any case, the findings appear to be complicated by the inclusion of AugReg, which narrows the claim about the clear split between contrastive text-image models and their complement. I still think that the manuscript is quite interesting; while it has been improved by better contextualization in terms of prior work, the narrowing of the claims hurts the potential impact. I would ask the authors to comment on the inclusion of new models in the tables/figures mentioned above before posting a final assessment.

AuthorsOpenReview ↗

Figure 4 corrected; on the narrowing

Thank you, you are right, and it was unintentional. We added the paired SigLIP and AugReg measurements as a new appendix table but did not regenerate Figure 4 from them, so the figure still carried the earlier four-model set. Table 5 in the submitted version was the pooling-operator table, which does include SigLIP; it moved to the appendix in the revision, so the number now points at the segmentation table instead, which likely explains the second half of what you saw.

Figure 4a is now rebuilt from the paired run, with SigLIP B/16, CLIP B/16, DINOv2 B/14 and AugReg B/16, all on identical images at each model's own peak-artifact layer, with five-seed random controls shown as markers. The revised PDF is uploaded.

Modelpooled kk-NNpooled probeper-token kk-NN
SigLIP B/16 (contrastive)−20.0-20.0−10.0-10.0−1.9-1.9 (ctrl −1.8-1.8)
CLIP B/16 (contrastive)−9.0-9.0−10.2-10.2−14.1-14.1
DINOv2 B/14 (self-dist.)−1.8-1.8−1.2-1.2+6.2+6.2
AugReg B/16 (supervised)−4.4-4.4−4.4-4.4+5.6+5.6

On the narrowing and its effect on impact, we take the point and will not argue that the paper is as strong as submitted. One observation, though, on what AugReg does. It does not only complicate the split; it rules out the explanation we had offered for it. Before AugReg, the CLIP-versus-DINOv2 contrast was consistent with the training objective being the cause, and that is what we claimed. AugReg is supervised, carries artifacts (ρart=9.2\rho_{\mathrm{art}} = 9.2), and falls with the contrastive models under one readout and with DINOv2 under the other. A single-cause account is therefore excluded by measurement rather than left open, and the remaining statement, uniform sign under global readout, family-dependent sign under per-token readout, is narrower but is what the data support.

We would rather report that than keep a cleaner claim we can no longer defend, and the checkpoint that forced it was one added at a reviewer's request. F1 and F4 are unchanged, and the diagnostics, run outputs and nine documented negative results are released for reuse. We are grateful for a review that made the paper more accurate, whatever assessment follows.

Reviewer_BM1sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how high-norm artifact tokens form in pretrained ViTs and what they do. On frozen public checkpoints, it characterizes outlier tokens with a robust z-score, a max/median norm ratio and a read score, and probes their role by zeroing them at an intermediate layer. It covers CLIP, SigLIP, DINOv2 and other families on 10 datasets, with functional tests mostly on Oxford-IIIT Pet plus one SigLIP ablation on EuroSAT. Five findings are stated about cross-domain stability, scale, registers, discriminative content and the training objective.

Critical strengths
  1. Extensive experiments. The same diagnostics are applied to many checkpoints and 10 datasets. I think this descriptive evidence is the main contribution worth keeping.
  2. A useful warning about over-reading ablations. The same intervention on the same model can flip sign between pooled readout, per-token readout and segmentation. Showing this systematically helps experimental design, even though the paper's explanation is not convincing yet.
  3. Transparent reporting. The appendix lists hypotheses that were tested and rejected, the limitations are stated openly, and part of the analysis code is released, which makes the work easier to build on.
Critical weaknesses
  1. F5 is not supported by the reported results. F5 says the sign of the ablation effect depends on the training objective, not the protocol. Yet for SigLIP, pooled k-NN drops 13.3 points in Table 4 while per-token k-NN rises 2.9 points in Table 13, and +6.0 at layer 7 is reported in the text. The samples are not strictly paired, so I can't say the protocol alone flips the sign, but the general claim no longer holds as stated. Sec. 8 also explains Fig. 1 as two protocols applied to different models, while Fig. 1 uses a single DINOv2 L/14. Since classification and segmentation also differ as tasks, the scope of the conclusion needs to be stated clearly.
  2. Zeroing in F4 does not identify "image-specific discriminative content". Zeroing high-norm tokens may cause a large perturbation that changes attention in later layers. The random control matches token count but not perturbation size, and changing the pooling operator in Table 5 cannot rule out redistributed attention. Sun et al. (2024) found, in a different setting, that zeroing massive activations hurts while mean replacement does not. I see this as an alternative to test, not as direct counter-evidence. Without mean replacement, norm rescaling that keeps direction, or magnitude-matched controls, the results only show that the intervention affects performance.
  3. Overlap with prior work and thin support for some findings. Registers taking over the high-norm role (Darcet et al., 2024; Sun et al., 2024), registers redirecting rather than removing attention sinks together with a CLS-to-patch handoff (Fesser et al., 2026, described only partially here), and input-independent artifact components (SINDER; Denoising ViT) are already known. Sun et al., SINDER and Denoising ViT are not cited. So F1 to F3 mostly re-measure known mechanisms at a broader scale, and I think the paper should say so. Some support is also thin: the narrow-band emergence claim rests on three DINOv2 scales, random controls use five seeds in Fig. 5 but one in Table 13, "statistically indistinguishable" comes without a test, and the abstract's "exactly invariant for every model" and the conclusion's "at most 10%" do not match the DeiT-III entries and SigLIP's 23% range.
Questions for authors
  • How do you explain SigLIP flipping sign between pooled and per-token evaluation, and what exactly is the scope of F5?
  • Paired results on the same images, layer and tokens would help, as would full SigLIP results for all three evaluators and all k values, since the scripts include SigLIP where some figures use DINO.
  • Does F4 still hold under mean replacement, direction-preserving norm rescaling or magnitude-matched random interventions?
  • Finally, which run outputs correspond to the final tables? Raw logs for the core ablations and controls, the image count for Table 1, and the seeds and error ranges for Tables 6 and 13 would make the results much easier to check.
3 comments

Response to Reviewer BM1s

Thank you for the soundness objections; the intervention concern in particular led to the most informative new experiment in this revision. We also posted a general response describing the two substantive corrections and the revised PDF.

On F5. Addressed in the general response. We agree the general claim does not hold as stated and have narrowed it. Your observation that the samples were not strictly paired was the right diagnosis: under paired conditions SigLIP's per-token value is −1.9-1.9 against a −1.6-1.6 control, not +2.9+2.9.

On zeroing versus other interventions. This was the most valuable comment and we ran all three alternatives you proposed (new Appendix L, N=1000N{=}1000, five-seed controls, paired images):

Modelzeromean replacementdirection-preserving rescale
SigLIP B/16−20.0-20.0−20.0-20.0−3.0-3.0
CLIP B/16−9.0-9.0−9.2-9.2−3.0-3.0
DINOv2 B/14−1.8-1.8−1.6-1.6−0.4-0.4
AugReg B/16−4.4-4.4−5.0-5.0−3.4-3.4

Mean replacement reproduces zeroing almost exactly, so the effect is not an artifact of injecting an out-of-distribution zero vector and the Sun et al. result does not transfer to this setting. You were also right that the random control matched count but not perturbation size: mean ∥δ∥\lVert\delta\rVert is 56.056.0 for top-kk zeroing versus 11.611.6 for the random control in SigLIP. Mean replacement controls for this directly, perturbing by a comparable 53.853.8 and producing the identical effect.

Direction-preserving rescaling is the more interesting result. Setting the ablated tokens to the layer-median norm while keeping their direction recovers most of the effect. We have therefore restated F4: these tokens contribute a large-magnitude component that pooled readout depends on in contrastive models, rather than simply "carrying image-discriminative content." We thank you for the correction.

On overlap with prior work and missing citations. We have added Sun et al. (2024), SINDER (Wang et al., 2024), Denoising ViT and Lappe & Giese (2025), and rewritten F2 and F3 as corroboration rather than novel findings, as set out in the general response and the finding-by-finding summary above.

On thin support. Corrected throughout:

  • The abstract's "exactly invariant for every model" now reads "invariant for the four models tested"; the conclusion's "at most 10%10\%" now reads "CV ≤0.102\le 0.102", which is the quantity we actually measure and is consistent with DeiT-III.
  • "Statistically indistinguishable" is replaced by the numerical comparison and the control interval; we do not claim a test we did not run.
  • Random controls now use five seeds everywhere, including the tables that previously used one.
  • The narrow-band emergence claim is explicitly scoped to the three DINOv2 checkpoints and flagged against the different threshold reported by Lappe & Giese. On run outputs. The supplementary material now contains runs/*.json with every number in every table, and the figure scripts read from those files, so figures and tables cannot disagree. Image counts, seeds and gallery sizes are stated per table.

Thank the authors for the thorough and candid rebuttal. The new controls and corrections address most of my soundness concerns. However, they do so largely by narrowing the claims: F5 is withdrawn in its original form, F2 and F3 are now corroboration of prior works, and F4 is reduced to a weaker statement. Though the revised paper is more rigorous, its remaining contribution is limited. I tend to keep my rating.

AuthorsOpenReview ↗

Response — soundness rating and what the narrowing did not change

We appreciate the engagement and accept the characterisation: the revision narrows claims rather than defends them. Three points in response, briefly.

Soundness rating. Your review rated soundness 1 ("significant concerns that do not appear easy to address"), and you now write that the new controls address most of them. If that reflects your current view, we would be grateful if the soundness score could be updated accordingly, independently of the overall rating.

What the narrowing did not touch. It affected F5, which over-claimed, and the novelty framing of F2 and F3. F1 and F4 are unchanged, and they are the two findings we claim as new. F1 is also the most broadly evidenced result in the paper: 10 datasets across 9 visual domains, a 9.1×9.1\times range in low-information content, and a synthetic control at 0.000%0.000\% flat patches where artifacts still form at 8080--95%95\% of panel-mean strength. F4 reports that artifact severity and functional importance are dissociated — DINOv2 B/14 at ρart=57.9\rho_{\mathrm{art}} = 57.9 is unaffected while CLIP at 23.223.2 collapses — and it gained content from your own suggestion: direction-preserving rescaling shows the dependence is on a large-magnitude component rather than on direction, which we could not have stated before. We agree the functional findings (F2--F5) rest on Oxford-IIIT Pet with one cross-domain check, as you noted, and this remains stated in Limitations.

The diagnostics. Artifact strength is scale-free and reference-free, and read score is computed over an outlier set rediscovered per image rather than a fixed token type. These are what make F1 measurable at all: CKA and attention partitioning, used in the closest prior work, require a reference representation or a predefined token type and cannot yield one comparable number per dataset or apply to models without registers. We release them with run outputs for every reported number, and we think they are reusable independently of which of our findings survive.

Your review identified the descriptive evidence as "the main contribution worth keeping." That evidence is unchanged and now carries matched five-seed controls, intervention alternatives, a saturation control, and nine documented negative results. We understand if you judge that insufficient for this venue, and we are grateful for review that made the paper more accurate.

Reviewer_JG5fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

An inference-only study of high-norm “artifact” tokens across 15 ViT checkpoints and 10 datasets. Using scale-free, per-image diagnostics (ρart\rho_{art}, sreads_{read}) and random-ablation controls, the paper reports five findings: artifact strength is stable across input distributions (F1); read behaviour emerges with scale (F2); registers relocate the sink function (F3); contrastive models depend on artifact tokens (F4); and the sign of the ablation effect is set by the training objective (F5).

Critical strengths
  1. Careful cross-domain measurement. ρart\rho_{art} varies with a coefficient of variation of only 0.030–0.102 across ten datasets, including medical and satellite imagery, with a constant peak layer per model. The synthetic no-flat-patch control (Table 8) argues convincingly against a saturation explanation. The diagnostic suite is reusable and may explain discrepancies across prior work.

  2. Severity and function are dissociated (F4). Under global readout, DINOv2 B/14 has the strongest artifacts (ρart\rho_{art} = 57.9) but changes far less under ablation than CLIP (ρart\rho_{art} = 23.2). The CLIP/SigLIP drop persists under magnitude-free pooling, while random controls stay within about 1 point (Table 5).

Critical weaknesses
  1. F5’s generality is contradicted by the paper’s own results. SigLIP, the second contrastive model, drops 13.3 points under global readout (Table 4) and 20–21 points under pooling (Table 5). Under per-token k-NN, however, it gains +2.9 (Table 13), against −4.5 for the random control, and +6.0 at layer 7 (Section 8). Figure 5 also appears to show its sign changing across layers. The invariance statements in Section 8 and in the Figure 5 caption cover only DINOv2, CLIP and DINO. In addition, attributing the difference to the training objective rests on CLIP versus DINOv2, which also differ in data, patch size and distillation; the artifact-bearing supervised augreg model (Table 11) is never evaluated. Since F5 drives the paper’s main practical recommendation, it should be narrowed to the tested checkpoints and settings.

  2. F3’s attention accounting mixes layers. Section 6 cites an outlier-mass drop from 0.382 to 0.007 and refers to Table 12 for where that mass goes. Table 12, however, is at layer 8, where outliers receive only 0.018. There, about 88% of the registers’ 0.304 comes from CLS (−0.267). The 0.382 figure corresponds to layer 10 (Table 7). Read-score values also differ between Table 7 and Tables 2/11 (0.394 vs. 0.382; 0.565 vs. 0.575). Table 3 supports the qualitative conclusion, but the quantitative argument is not internally aligned.

Questions for authors
  1. Please provide Table 12 at layer 10, using the same threshold and image set as the 0.382 / 0.007 figures, and explain the differences in read-score values across tables.
  2. Is F5 intended to cover SigLIP? Please report SigLIP under all evaluators of Figure 4a and across the full layer sweep, with random controls and uncertainty estimates.
  3. In per-token k-NN, what label is assigned to each token, and at which layer is it read out? Did SigLIP’s global readout in Table 4 use its native attention-pooling head?
  4. Please add augreg to the F4/F5 evaluations and report the random controls for Table 4.
2 comments

Response to Reviewer JG5f

Thank you for the two specific, verifiable objections. Both were correct. We also posted a general response describing the two substantive corrections and the revised PDF.

On the attention accounting mixing layers. You are right that Table 12 was computed at layer 8 while the 0.382/0.0070.382 / 0.007 figures come from layer 10, and that the two cannot be compared. We recomputed the full decomposition at layer 10 on the same images and threshold:

ModelCLSregistersoutlier patchesordinary patches
DINOv2 B/140.050—0.3870.563
DINOv2 B/14 +reg0.0200.3700.0010.608

Outlier patches lose 0.3860.386 and registers gain 0.3700.370: a direct transfer of 96%96\%. Ordinary patches absorb the remainder (0.563→0.6080.563 \to 0.608) and CLS is nearly unchanged. Our submitted claim that the mass came from CLS was an artifact of measuring at layer 8, where outliers hold only 0.0260.026 and dumping has not yet begun. Section 6 and Table 12 are corrected, and the accounting now closes.

On read-score differences across tables. Both models peak at layer 10. The discrepancies are sampling variation between a 32-image and a 256-image subset:

32 images256 images
CLIP B/160.5750.564
DINOv2 B/140.3820.387

All read-score values are now computed on 256 images and a single number is reported per model.

On F5 and SigLIP. Addressed in the general response. The paired re-run gives SigLIP −1.9-1.9 per-token against a −1.6-1.6 control at its peak layer, so the sign does not flip there; but AugReg, which you asked us to add, does break the objective-based framing, and F5 is narrowed accordingly.

On per-token k-NN labels and readout. Each patch token inherits its image's class label. Gallery and query are split by image, not by token, so patches from the same image cannot appear on both sides; we sample 32 patches per image with a seed held fixed across conditions. Tokens are read at the model's peak-artifact layer, the same layer at which ablation is applied. SigLIP has no CLS token, so its "global readout" in Table 4 is mean pooling over patch tokens, not its native attention-pooling head; this is now stated in the caption.

On random controls for Table 4. Added, five seeds, in the revised Table 4.


AuthorsOpenReview ↗

Follow-up — both weaknesses addressed with new measurements

Both points you raised are now resolved in the revised PDF. (1) The attention accounting was indeed mixing layers. Recomputed at layer 10 on the same images and threshold, outlier patches hold 0.3870.387 in the base model and 0.0010.001 with registers, while the registers receive 0.3700.370, a direct 96%96\% transfer, so the quantitative argument now closes. Our earlier claim that the mass came from CLS was an artifact of measuring at layer 8. (2) SigLIP is now reported under all three evaluators and across the full layer sweep with five-seed controls (Appendix K), and AugReg has been added to the F4/F5 evaluations. Your other questions, per-token labels and readout layer, and SigLIP's global readout, are answered in the revised Section 3.

We would be glad to run anything further that would help in the remaining days.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers, summary of changes and revised PDF

We thank all three reviewers for careful reading. Two criticisms were correct and have changed the paper, and we report both plainly before anything else. A revised PDF is uploaded, with all changes marked in blue. A companion comment, Relation to prior work, finding by finding, calibrates our novelty claims.

F5 as stated was wrong, and SigLIP was the evidence. All three reviewers identified that SigLIP drops under pooled readout but gains under per-token readout. The comparison was not paired: Table 4 used N=3000N{=}3000 and Table 13 a different subset with a single-seed control. We re-ran every evaluator on identical images at each model's peak-artifact layer with five-seed controls. At its peak layer SigLIP gives −20.0-20.0 (pooled k-NN), −10.0-10.0 (pooled probe) and −1.9-1.9 (per-token k-NN) against a per-token control of −1.6-1.6; the per-token value sits inside the control band rather than being positive. The +2.9+2.9 we reported was an artifact of the unpaired comparison.

The broader objection nonetheless stands. We added the supervised AugReg model as requested (JG5f). It carries artifacts (ρart=9.2\rho_{\mathrm{art}}=9.2) and behaves like DINOv2 under per-token readout (+5.6+5.6) while behaving like the contrastive models under pooled readout (−4.4-4.4). Training objective alone therefore does not determine the sign, and F5 is narrowed to what the data support: under global readout every artifact-bearing model loses accuracy; under per-token readout the sign separates contrastive models from self-distillation and supervised ones. We no longer claim protocol-invariance.

F2 and F3 substantially overlap with Lappe & Giese (2025), which we had not seen. We thank Y4YJ for the reference. Their Figure 2a establishes that CLS attention to registers grows with model size, and their Figure 3 establishes by attention-zeroing and CKA that registers dominate the global output in large models. These are our F2 and F3, reached by different measurements. We have rewritten both sections as independent corroboration rather than novel findings and added a Related Work paragraph. We retain them for the matched random-ablation controls (not reported there; their checklist item 7 states no error bars), a third model scale, and an attention-mass decomposition. Their Section 4 result that CLS acts as a register via the skip connection also supplies a mechanism we had observed but could not explain.

We have also added the mean-replacement and magnitude-matched controls requested by BM1s, corrected the layer mismatch in the attention accounting found by JG5f, and fixed the overstated numerical claims BM1s listed.

Changes in the revised PDF

  1. F5 narrowed (§8, abstract, conclusion): claim restricted to global readout; the per-token split described separately; protocol-invariance claim removed.
  2. AugReg added to the F4/F5 evaluations (new Appendix K).
  3. SigLIP reported under all three evaluators and across the full layer sweep, paired on identical images with five-seed controls (new Appendix K).
  4. Attention accounting corrected to the read-score peak, layer 10 (§6, Table 12): outlier patches 0.387→0.0010.387 \to 0.001, registers receive 0.3700.370, a 96%96\% transfer.
  5. Read-score sample sizes stated per table and reconciled (new Table 17): both models peak at layer 10, and the differences between tables are <0.012<0.012 sampling variation.
  6. New Appendix L: mean replacement, direction-preserving rescaling, and the perturbation magnitudes of treatment and control; F4 restated accordingly.
  7. Related Work paragraph on Lappe & Giese (2025), Sun et al. (2024), SINDER and Denoising ViT.
  8. Overstated claims corrected in the abstract, §4 and the conclusion; five-seed controls throughout; untested significance language removed.
  9. Three rows added to the rejected-hypotheses table (saturation, zeroing-artifact, objective-determines-sign).
  10. Supplementary now ships runs/ JSON files containing every reported number, with the figure scripts reading from them.

Reviewers BM1s and Y4YJ asked for calibrated novelty claims. The revised paper states the following; we summarise it here for convenience.

F1 (artifact formation is weight-driven) — novel. Darcet et al. observe that artifacts occupy low-information regions but do not vary the input distribution. Baxevanakis et al. report cross-architecture variation without identifying its source. Lappe & Giese use COCO and ImageNet only. No prior work measures artifact strength across a controlled range of low-information content. We span 6.4%6.4\% to 58.6%58.6\% flat patches over 10 datasets and 9 domains and find CV 0.0300.030–0.1020.102 with an invariant peak layer, plus a synthetic control at 0.000%0.000\% flat patches where artifacts still form at 8080–95%95\% of panel-mean strength. This analysis needs a scale-free statistic: CKA and attention partitioning both require a reference representation or a predefined token type and cannot yield one comparable number per dataset.

F2 (emergence with scale) — corroboration, not novel. Darcet et al. note artifacts appear only in larger models. Lappe & Giese quantify the transition for CLS-to-register attention between 86M and 300M. We measure a different quantity — read score on high-norm patch outliers in models that have no registers — and locate it between 22M and 87M. The two thresholds need not coincide; we now report the discrepancy as an open question rather than presenting ours as the threshold.

F3 (registers capture the read target) — corroboration with additions. Lappe & Giese establish register dominance by attention-zeroing and CKA at the last layer. We establish it by token-zeroing with matched random controls across three scale pairs, and add a full attention-mass decomposition. The decomposition is new: at layer 10, outlier patches hold 0.3870.387 in the base model and 0.0010.001 with registers while the registers receive 0.3700.370, a direct transfer of 96%96\% of the mass lost.

F4 (contrastive models depend on artifact tokens) — novel. No prior work compares the downstream effect of artifact ablation across training objectives. Lappe & Giese analyse CLIP only through the skip-connection lens and do not measure ablation effects on accuracy; Darcet et al. measure dense-task quality, not retrieval under ablation. The dissociation we report has not appeared: DINOv2 B/14 has the strongest artifacts in the panel (ρart=57.9\rho_{\mathrm{art}}=57.9) yet is unaffected, while CLIP at 23.223.2 collapses. The revised version adds that the effect survives mean replacement and largely disappears under direction-preserving rescaling, so what these tokens contribute is a large-magnitude component rather than a direction carrying the content on its own.

F5 — narrowed, as described in the general response. The 2×2 protocol decomposition and the layer sweep are new; the claim built on them is weaker than submitted.

Diagnostics — new and reusable. Artifact strength is scale-free and reference-free. Read score identifies the outlier set per image rather than partitioning against a fixed token type, so it applies to models without registers, where partitioning has nothing to partition against. Together these are what make F1 measurable at all.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.