- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper performs layer-wise activation patching on four frozen DINOv2 checkpoints (ViT-S/B/L/G, each with 4 register tokens). Clean/corrupted image pairs are built from 90 Pascal VOC images by corrupting only the background (Gaussian noise, black replacement, blur) while leaving foreground pixels unchanged, across three scene-complexity groups, for nine conditions in total. At each layer one of two interventions is applied and the network is run to the final layer: a single bottleneck token ([CLS] or one REG) is restored to its clean state, or all patch tokens are restored while the bottleneck tokens stay corrupted. The normalized cosine recovery of that token's final-layer representation toward its clean counterpart defines Self-Autonomy (A) and Patch Receptivity (R), respectively. The reported results are: A rises and R falls with depth (termed an Autonomy-Receptivity duality); the depth at which Δ = A − R crosses zero varies with model scale and register index (about 25% of trajectories cross before the final block in S/B, 92% to 100% in L/G); the dispersion of Δ across the four REGs increases monotonically with depth (36/36 model-condition combinations); and within a model, the late-depth ranking of REGs is consistent across the nine conditions (Kendall's W 0.86 to 1.00). As a functional validation, a linear probe is trained on full-register [CLS] features on CIFAR-100 and then fixed; at test time, register tokens are deleted after a given depth. Deleting them entirely drops ViT-B Top-1 from 90.7% to 18.9%, and keeping them through 90% of depth largely restores it. The paper concludes that registers evolve from "transient but influential interaction states" at shallow depth to "self-persistent and specialized information-routing streams" at deeper layers.
Critical strengths
The object of study is the right one, and an interesting one: how the representations of the register tokens and the [CLS] token change from layer to layer, and whether individual registers differentiate with depth, is worth investigating.
Critical weaknesses
1. The premise of the study is asserted rather than established, and the quantity it measures has no established significance
The framework rests on treating [CLS] and the four REG tokens as "bottleneck streams" (Sec. 3.1): tokens through which patch information is funneled and then shapes subsequent computation. The premise is the authors' own. It carries no citation and is never tested, and the established account is the opposite: registers are auxiliary buffers that absorb high-norm spatial signal (Darcet et al., 2024), as the paper's own Introduction states.
The final-layer REG representations are discarded at readout; only [CLS] (and patch tokens, for dense tasks) are used. A and R measure how well a discarded state can be reconstructed. The paper gives no consequence of that reconstruction being high or low, so the meaning of A and R, and of every statistic built on them (duality, crossover, dispersion, ranking), cannot be assessed. The premise is asserted, the measured object has no established role, and the curve shapes that remain follow from the construction of the measures (Weakness 3): I could not identify a result about registers in this paper.
2. Missing baseline; corruption design contradicts experimental observation and published literature
Everything the paper measures is the difference between the final-layer [CLS] and REG representations of the clean and the corrupted forward passes, and both A and R have 1 − S_corrupt as their denominator. S_corrupt itself, the raw cosine between the clean and corrupted final bottleneck representations, is never reported: no value, no distribution, no per-model or per-token comparison, and no statement of what the "damage" changes, from what state to what state, or what information is lost. It is the precondition for quantifying comparisons two and three at all. Without it, every curve in Sec. 4 is a ratio over an unknown quantity, and the most important data in the paper is skipped entirely.
The corruption design contradicts experimental observation and published literature. Only the background is corrupted and foreground pixels are left untouched; the implicit premise is that the [CLS] and REG representations are determined by the foreground object, so that background corruption is a controlled damage to them. That premise is wrong. Vision Transformers Need More Than Registers (CVPR 2026) shows that the [CLS] token of a ViT attends predominantly to background patches, that CLS–patch similarity concentrates in background regions (Fig. 1, Fig. 2a), and that the global representation is aggregated mainly from background tokens; on a supervised ViT, masking up to 70% of the highest-scoring (background-dominated) patches from the input leaves ImageNet accuracy unchanged (Fig. 2b). As a reviewer who has run experiments in this area, my observation of ViT attention agrees with that paper: in attention heatmaps, roughly twenty to thirty percent of the attention falls on the foreground object to be recognized, and the rest is spread across the whole image. For this paper, it means that the background corruption rewrites exactly the tokens the [CLS] token mainly reads from, while barely affecting the decision content of [CLS]. The paper's "damage" can then be only one of two things. Either it is small, 1 − S_corrupt is close to 0, A and R are ratios of small quantities, and the curve shapes are amplified noise; or it is not small, but what is damaged is the content of background shortcut tokens rather than object information, and it is then unclear what Sec. 4 is "recovering". In either case Sec. 4 cannot be interpreted, and the whole experimental setup rests on a premise that runs against the literature.
3. The outcomes of comparisons two and three are determined by construction
Self-Autonomy restores one bottleneck token to its clean state at layer l and reads out at the final layer; Patch Receptivity restores all patch tokens at layer l and reads out at the final layer. The paper itself states (Sec. 4.1) that at l = L, A ≈ 100% and R ≈ 0%. The two curves are interpolations between two pinned endpoints: the closer the injection point is to the readout, the larger the share of the token's own residual stream and the smaller the share re-read from the patches. Any network with residual connections and attention has this property; it has nothing to do with registers. Change the reference to another layer's representation and the curve shape changes with it; reverse the direction of the swap, injecting a corrupted token into the clean trajectory, and the result is the mirror image. The outcome can be written down before the experiment is run. The paper argues that this is not a shared depth effect because the partial correlation remains negative after controlling for normalized depth. That argument fails: the partial correlation removes only the linear part of the depth dependence, A and R do not vary linearly with depth, what remains after the removal still has one curve rising and the other falling, and a negative partial correlation is therefore guaranteed and rules out nothing. The duality, crossover, and specialization claims built on top of this all describe consequences of the construction itself.
From the side of A alone: the later a state is restored, the fewer layers remain to rewrite it, and at the final block A is about 100% by definition, so "A rises with depth" is close to a necessity. Accordingly, the A curves of [CLS] and all four REGs in Fig. 1 share the same trend; what the paper calls specialization is the spread of positions along this one trajectory. The paper also measures no A curve for any token that is not a register, neither an ordinary patch token in the same model nor the high-norm artifact token that plays the register role in the no-register DINOv2. If a patch token's A curve has the same shape, which the construction predicts, "registers progressively become self-persistent streams" is a property of every token in a residual Transformer. Question 2 asks for this control.
4. The required AI use statement is missing
The ICLR 2027 Author Guidelines list the AI use statement as Required, and the AI Policy for Authors requires authors to disclose AI use in a separate mandatory section of the paper that does not count toward the page limit. This submission contains an Abstract, Secs. 1–6, References, and Appendix A; the section is absent. This violates a hard submission requirement that had to be met in the submitted version; adding the section during the rebuttal does not change the fact that the submitted version is non-compliant.
5. The functional validation in Sec. 5 does not hold
Sec. 5 is the only place the measurements are tied to a model output, and it does not support its conclusion. A linear classifier is trained once on the final full-register [CLS] representation of a frozen DINOv2; backbone and classifier are then fixed, and register tokens are deleted after a prescribed fraction of blocks. ViT-B/14 at 224 × 224 has 256 patch tokens, one [CLS], and four registers; deleting the four registers removes 1.5% of the tokens, and CIFAR-100 Top-1 falls from 90.72% to 18.91% (ViT-G: 94.16% to 37.59%). A 72-point drop from removing 1.5% of the tokens is structural damage rather than loss of register content: the frozen blocks were trained with registers in every attention operation, deleting them changes the attention normalization of every remaining token at every later layer, and the classifier is never adapted. Two published facts point the same way. An untrained extra token inserted at test time, with the high-norm activations shifted onto it, reproduces the effect of trained registers (Jiang et al., 2025, Vision Transformers Don't Need Trained Registers, NeurIPS 2025, cited by the paper itself); a register provides a location rather than learned content. DINOv2 trained without registers performs well, with the same high-norm global computation placed in a few low-information patch tokens (Darcet et al., 2024). The early-exit curve thus shows that deleting the network's norm-sink slots breaks the forward pass; it is no evidence that shallow register states "exert substantial downstream influence", and the paper's final claim about "transient but influential interaction states" loses its only functional support.
Questions for authors
1. Report the raw clean-versus-corrupted data
Please report the unnormalized S_corrupt: the raw cosine between the final-layer representations of [CLS] and REG1 through REG4 in the clean and corrupted forward passes, broken down by the four models, the five tokens, and the nine conditions, with the distribution over the 90 images rather than only the mean. Please also provide the unnormalized S_auto(l) and S_rec(l) curves. This quantity is the denominator of every quantitative conclusion in Sec. 4; without it A and R cannot be interpreted. It also answers directly how much the background corruption changes the bottleneck representations and what it changes.
2. Measure the same quantities for tokens that are not registers
Please measure Self-Autonomy and Patch Receptivity, with the identical patching protocol, for two kinds of control tokens. (i) In the original DINOv2 checkpoints without registers, the high-norm artifact patch tokens that are known to take on the register role (Darcet et al., 2024): if their A curves match those of the registers, self-persistence is a property of the norm-sink function, whichever token happens to carry it, and not of registers. (ii) In the register-equipped models, ordinary patch tokens, foreground and background alike: the paper claims registers are "special information-routing streams" but never contrasts them with any non-register token, so none of the reported phenomena, the rise of A, the fall of R, the crossover, the growth of dispersion with depth, can currently be attributed to registers rather than to every token in the network. Please report both control sets alongside the [CLS] and REG curves in Fig. 1.