acceptodds
Withdrawn from ICLR 2027

Functional Specialization and Self-Persistent Routing of Native Tokens in ViTs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision Transformers (ViTs) increasingly employ auxiliary tokens such as and register () tokens, yet their computational roles remain poorly understood. We analyze register-token dynamics using dual-path activation patching and introduce two complementary measures: Self-Autonomy, capturing causal persistence of token states, and Patch Receptivity, capturing their recoverability from spatial patch information. Across four DINOv2 scales and nine controlled visual conditions, register streams progressively specialize with depth and exhibit a clear Autonomy-Receptivity duality: shallow states are strongly patch-driven, whereas deeper states become increasingly self-persistent. These specialization patterns remain consistent across visual conditions despite changes in routing strength. Test-time register early-exit interventions further show that shallow register states can exert substantial downstream influence despite low Self-Autonomy. Our results reveal register tokens as heterogeneous, depth-dependent information-routing streams whose functional roles emerge and specialize throughout the network.

Primary area:
interpretability and explainable AI
Keywords:
Vision Transformers, Register Tokens, Mechanistic Interpretability, Activation Patching, Information Routing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7AQ4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper performs layer-wise activation patching on four frozen DINOv2 checkpoints (ViT-S/B/L/G, each with 4 register tokens). Clean/corrupted image pairs are built from 90 Pascal VOC images by corrupting only the background (Gaussian noise, black replacement, blur) while leaving foreground pixels unchanged, across three scene-complexity groups, for nine conditions in total. At each layer one of two interventions is applied and the network is run to the final layer: a single bottleneck token ([CLS] or one REG) is restored to its clean state, or all patch tokens are restored while the bottleneck tokens stay corrupted. The normalized cosine recovery of that token's final-layer representation toward its clean counterpart defines Self-Autonomy (A) and Patch Receptivity (R), respectively. The reported results are: A rises and R falls with depth (termed an Autonomy-Receptivity duality); the depth at which Δ = A − R crosses zero varies with model scale and register index (about 25% of trajectories cross before the final block in S/B, 92% to 100% in L/G); the dispersion of Δ across the four REGs increases monotonically with depth (36/36 model-condition combinations); and within a model, the late-depth ranking of REGs is consistent across the nine conditions (Kendall's W 0.86 to 1.00). As a functional validation, a linear probe is trained on full-register [CLS] features on CIFAR-100 and then fixed; at test time, register tokens are deleted after a given depth. Deleting them entirely drops ViT-B Top-1 from 90.7% to 18.9%, and keeping them through 90% of depth largely restores it. The paper concludes that registers evolve from "transient but influential interaction states" at shallow depth to "self-persistent and specialized information-routing streams" at deeper layers.

Critical strengths

The object of study is the right one, and an interesting one: how the representations of the register tokens and the [CLS] token change from layer to layer, and whether individual registers differentiate with depth, is worth investigating.

Critical weaknesses

1. The premise of the study is asserted rather than established, and the quantity it measures has no established significance

The framework rests on treating [CLS] and the four REG tokens as "bottleneck streams" (Sec. 3.1): tokens through which patch information is funneled and then shapes subsequent computation. The premise is the authors' own. It carries no citation and is never tested, and the established account is the opposite: registers are auxiliary buffers that absorb high-norm spatial signal (Darcet et al., 2024), as the paper's own Introduction states.

The final-layer REG representations are discarded at readout; only [CLS] (and patch tokens, for dense tasks) are used. A and R measure how well a discarded state can be reconstructed. The paper gives no consequence of that reconstruction being high or low, so the meaning of A and R, and of every statistic built on them (duality, crossover, dispersion, ranking), cannot be assessed. The premise is asserted, the measured object has no established role, and the curve shapes that remain follow from the construction of the measures (Weakness 3): I could not identify a result about registers in this paper.

2. Missing baseline; corruption design contradicts experimental observation and published literature

Everything the paper measures is the difference between the final-layer [CLS] and REG representations of the clean and the corrupted forward passes, and both A and R have 1 − S_corrupt as their denominator. S_corrupt itself, the raw cosine between the clean and corrupted final bottleneck representations, is never reported: no value, no distribution, no per-model or per-token comparison, and no statement of what the "damage" changes, from what state to what state, or what information is lost. It is the precondition for quantifying comparisons two and three at all. Without it, every curve in Sec. 4 is a ratio over an unknown quantity, and the most important data in the paper is skipped entirely.

The corruption design contradicts experimental observation and published literature. Only the background is corrupted and foreground pixels are left untouched; the implicit premise is that the [CLS] and REG representations are determined by the foreground object, so that background corruption is a controlled damage to them. That premise is wrong. Vision Transformers Need More Than Registers (CVPR 2026) shows that the [CLS] token of a ViT attends predominantly to background patches, that CLS–patch similarity concentrates in background regions (Fig. 1, Fig. 2a), and that the global representation is aggregated mainly from background tokens; on a supervised ViT, masking up to 70% of the highest-scoring (background-dominated) patches from the input leaves ImageNet accuracy unchanged (Fig. 2b). As a reviewer who has run experiments in this area, my observation of ViT attention agrees with that paper: in attention heatmaps, roughly twenty to thirty percent of the attention falls on the foreground object to be recognized, and the rest is spread across the whole image. For this paper, it means that the background corruption rewrites exactly the tokens the [CLS] token mainly reads from, while barely affecting the decision content of [CLS]. The paper's "damage" can then be only one of two things. Either it is small, 1 − S_corrupt is close to 0, A and R are ratios of small quantities, and the curve shapes are amplified noise; or it is not small, but what is damaged is the content of background shortcut tokens rather than object information, and it is then unclear what Sec. 4 is "recovering". In either case Sec. 4 cannot be interpreted, and the whole experimental setup rests on a premise that runs against the literature.

3. The outcomes of comparisons two and three are determined by construction

Self-Autonomy restores one bottleneck token to its clean state at layer l and reads out at the final layer; Patch Receptivity restores all patch tokens at layer l and reads out at the final layer. The paper itself states (Sec. 4.1) that at l = L, A ≈ 100% and R ≈ 0%. The two curves are interpolations between two pinned endpoints: the closer the injection point is to the readout, the larger the share of the token's own residual stream and the smaller the share re-read from the patches. Any network with residual connections and attention has this property; it has nothing to do with registers. Change the reference to another layer's representation and the curve shape changes with it; reverse the direction of the swap, injecting a corrupted token into the clean trajectory, and the result is the mirror image. The outcome can be written down before the experiment is run. The paper argues that this is not a shared depth effect because the partial correlation remains negative after controlling for normalized depth. That argument fails: the partial correlation removes only the linear part of the depth dependence, A and R do not vary linearly with depth, what remains after the removal still has one curve rising and the other falling, and a negative partial correlation is therefore guaranteed and rules out nothing. The duality, crossover, and specialization claims built on top of this all describe consequences of the construction itself.

From the side of A alone: the later a state is restored, the fewer layers remain to rewrite it, and at the final block A is about 100% by definition, so "A rises with depth" is close to a necessity. Accordingly, the A curves of [CLS] and all four REGs in Fig. 1 share the same trend; what the paper calls specialization is the spread of positions along this one trajectory. The paper also measures no A curve for any token that is not a register, neither an ordinary patch token in the same model nor the high-norm artifact token that plays the register role in the no-register DINOv2. If a patch token's A curve has the same shape, which the construction predicts, "registers progressively become self-persistent streams" is a property of every token in a residual Transformer. Question 2 asks for this control.

4. The required AI use statement is missing

The ICLR 2027 Author Guidelines list the AI use statement as Required, and the AI Policy for Authors requires authors to disclose AI use in a separate mandatory section of the paper that does not count toward the page limit. This submission contains an Abstract, Secs. 1–6, References, and Appendix A; the section is absent. This violates a hard submission requirement that had to be met in the submitted version; adding the section during the rebuttal does not change the fact that the submitted version is non-compliant.

5. The functional validation in Sec. 5 does not hold

Sec. 5 is the only place the measurements are tied to a model output, and it does not support its conclusion. A linear classifier is trained once on the final full-register [CLS] representation of a frozen DINOv2; backbone and classifier are then fixed, and register tokens are deleted after a prescribed fraction of blocks. ViT-B/14 at 224 × 224 has 256 patch tokens, one [CLS], and four registers; deleting the four registers removes 1.5% of the tokens, and CIFAR-100 Top-1 falls from 90.72% to 18.91% (ViT-G: 94.16% to 37.59%). A 72-point drop from removing 1.5% of the tokens is structural damage rather than loss of register content: the frozen blocks were trained with registers in every attention operation, deleting them changes the attention normalization of every remaining token at every later layer, and the classifier is never adapted. Two published facts point the same way. An untrained extra token inserted at test time, with the high-norm activations shifted onto it, reproduces the effect of trained registers (Jiang et al., 2025, Vision Transformers Don't Need Trained Registers, NeurIPS 2025, cited by the paper itself); a register provides a location rather than learned content. DINOv2 trained without registers performs well, with the same high-norm global computation placed in a few low-information patch tokens (Darcet et al., 2024). The early-exit curve thus shows that deleting the network's norm-sink slots breaks the forward pass; it is no evidence that shallow register states "exert substantial downstream influence", and the paper's final claim about "transient but influential interaction states" loses its only functional support.

Questions for authors

1. Report the raw clean-versus-corrupted data

Please report the unnormalized S_corrupt: the raw cosine between the final-layer representations of [CLS] and REG1 through REG4 in the clean and corrupted forward passes, broken down by the four models, the five tokens, and the nine conditions, with the distribution over the 90 images rather than only the mean. Please also provide the unnormalized S_auto(l) and S_rec(l) curves. This quantity is the denominator of every quantitative conclusion in Sec. 4; without it A and R cannot be interpreted. It also answers directly how much the background corruption changes the bottleneck representations and what it changes.

2. Measure the same quantities for tokens that are not registers

Please measure Self-Autonomy and Patch Receptivity, with the identical patching protocol, for two kinds of control tokens. (i) In the original DINOv2 checkpoints without registers, the high-norm artifact patch tokens that are known to take on the register role (Darcet et al., 2024): if their A curves match those of the registers, self-persistence is a property of the norm-sink function, whichever token happens to carry it, and not of registers. (ii) In the register-equipped models, ordinary patch tokens, foreground and background alike: the paper claims registers are "special information-routing streams" but never contrasts them with any non-register token, so none of the reported phenomena, the rise of A, the fall of R, the crossover, the growth of dispersion with depth, can currently be attributed to registers rather than to every token in the network. Please report both control sets alongside the [CLS] and REG curves in Fig. 1.

Reviewer_2pdjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tries to understand register tokens and the way information propagates through them across layers. For this, the authors use pretrained DINOv2 models with registers (four sizes) and pairs of clean and background-corrupted images. They define Self-Autonomy and Patch Receptivity, computed by restoring clean states into the corrupted run at different layers: either a single register (Self-Autonomy) or all patch tokens (Patch Receptivity), and measuring how much of the register’s final representation is recovered. The experiments use 90 Pascal VOC images under nine scene and corruption conditions. In early layers, a register’s final state is mostly recovered by restoring the patches, while in later layers the register itself carries it, and the depth of this switch depends on model size. Registers also become more distinct from each other with depth, and their relative ordering stays the same across conditions. Finally, an early exit experiment on CIFAR-100 shows that removing registers partway through the network lowers accuracy.

Critical strengths
  1. I like the methodology of restoring clean registers or patches into the corrupted run at different layers to measure the different effects on the final representation. It keeps the comparisons clean.
  2. The results are consistent for different model sizes so the general trends hold and seem reasonable. (register differenciation increases with depth).
  3. The paper clearly explains what it measures and how and even has an interesting early-exit experiment showing how much the network relies on registers at different depths.
Critical weaknesses
  1. In early exit experiment the registers are removed but that might cause not just loss of information but the sinks the model depends on. Instead of removing them the authors can try to replace them with dataset average to validate that their conclusion is true without altering the overall presence of tokens. Otherwise its hard to know if its missing register information or the fact that its missing the sink.
  2. Restoring the register at later layer leaves fewer layers for corrupt patches to affect it, so is self-autonomy trend by design of the experiment? Similarly for patch receptivity fewer layers left for restored patches to correct the register. How do the authors account for this?
  3. The deepest measured point is 0.917 for S/B but 0.958–0.975 for L/G, and some L crossovers lie beyond 0.917 (Table 2). Crossover prevalence (Figure 2) should be compared at a common cutoff.
  4. Currently scope is limited to dinov2, authors should have a different set of models to show generality of their claims. Dinov3 would have been a more updated and good additional comparison to add.
Questions for authors

I hope the authors can address the questions and clarifications requested in the weaknesses, in particular regarding the experimental settings, the generality of the findings to other models such as DINOv3, and accounting for the different depth ranges across model sizes.

Reviewer_9TjQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how register tokens in DINOv2 evolve across network depth. The authors introduce two activation-patching measurements, Self-Autonomy and Patch Receptivity, which respectively measure the persistence of intermediate token states and the recoverability of their final representations through spatial patch information.

Across four DINOv2 model sizes, the authors find that register states become increasingly self-persistent with depth, while their dependence on restored spatial information decreases. Individual registers also exhibit different recovery profiles, with relatively stable rankings across the evaluated visual conditions. An additional early-exit experiment examines the importance of register participation for downstream classification.

Critical strengths
  • The paper addresses an interesting question about the internal function of register tokens, beyond their established role in mitigating attention artifacts.
  • The activation-patching framework provides a useful way to examine how intermediate representations contribute to their final states.
  • The experiments cover several model sizes and controlled visual conditions. The authors also make a useful distinction between the persistence of a token's own representation and its broader influence on the network.
Critical weaknesses

The main concern of this reviewer is the interpretation of the observed differences as functional specialization. The authors define specialization through differences in the proposed recovery measurements, and demonstrate that these differences increase with depth. This is an interesting observation, but does not necessarily establish that the register tokens develop distinct functional roles. The paper would benefit from a clearer distinction between heterogeneous recovery profiles and specialization in the underlying computation.

A second concern is that the reported Autonomy-Receptivity duality may partly follow from the intervention setup. Self-Autonomy restores a single register state, while Patch Receptivity restores all spatial patch states. As the intervention approaches the final layer, the former is increasingly expected to recover the target representation, while the latter has fewer opportunities to influence it. Although the authors control for depth when computing the correlations, it is less clear how much of the reported relationship represents a property of the learned register organization rather than the structure of the interventions. Some additional controls or discussion would help clarify this.

Finally, the early-exit experiments establish that register participation is important for downstream classification, but provide limited evidence for the proposed specialization. All registers are removed together, so the experiments cannot distinguish the functional contributions of individual register streams. This leaves the connection between the heterogeneous recovery profiles and the downstream function of the registers somewhat uncertain.

Questions for authors
  1. Functional specialization: The paper identifies differences in the recovery profiles of individual registers, but how do these relate to distinct functional roles? Could the authors provide an intervention that demonstrates different downstream contributions from individual registers?

  2. Autonomy-Receptivity duality: To what extent is the inverse relationship between the two measurements expected from the intervention design itself? Could the authors compare against a suitable control, for instance a randomly initialized network, to establish how much of this relationship emerges through learning?

  3. Intervention asymmetry: Self-Autonomy restores a single register, while Patch Receptivity restores all spatial tokens. How does this difference in the amount of restored information affect the interpretation of their relative magnitudes and the reported crossover depths?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.