acceptodds
Under review as a conference paper at ICLR 2027

Vision Transformers Need Cross Aggregation for Patch Semantics

Abstract

A Vision Transformer (ViT) is tasked to learn both a useful global representation through the CLS token and distinct local semantics through patch tokens. Yet a loss on the CLS token does not always induce learning of discriminative patch tokens. We theoretically observe that a CLS-based loss induces low rank patch gradients and in practice, most patches receive similar gradients. Adding patch-based losses may not compensate for the strong unidirectional CLS induced gradients. Conse- quently, even under global-local objectives, patch updates can remain insufficiently differentiated and spatially correlated. We therefore introduce cross-aggregation, which redefines the CLS token by aggregating patch-level concept heads in the expanded FFN space of every block. This parameter-free operation adds only 0.6% FLOPs to ViT-S and more directly couples global semantics to patch tokens. Combined with local depth-wise convolutions, it further improves the patch seman- tics, in a variety of pre-training methods. Together, these components form the Cross Aggregation and Local Mixing Transformer (CALiT). CALiT consistently improves semantic probes under both global-only and global-local pre-training objectives and vision-language finetuning. In particular, CALiT improves thin class semantics mean IoU by double digits in many pre-training/linear probe pairs and provides moderate gains in all other local semantics.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Transformers, Cross Aggregation, Depth-wise Convolutions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PUQmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that ViT patch features are weak partly because of how CLS-based losses train them. In the last block, patches influence the CLS loss only through CLS attention (the value and key routes), so the patch-gradient matrix has rank at most 2 × #heads. Empirically, this gradient is dominated by a single shared direction, so foreground and background patches receive similar updates. To address this, the authors propose CALiT, which modifies only the FFN of each block. Cross Aggregation (CAg) is a parameter-free attention in the post-GELU hidden space: the CLS hidden vector is split into slots that attend to all patch slots, and the result replaces the CLS hidden. Local Mixing (LMix) is a Dirac-initialized 3×3 depthwise convolution on the patch hidden, adapted from LocalViT. The total overhead is about +1.4% FLOPs at ViT-S.

The method is validated mainly with frozen-backbone linear probes on ADE20K, VOC, Cityscapes and COCO-Stuff, under supervised, DINO, iBOT and DINOv2 pre-training as well as LeVLJEPA fine-tuning. Thin-class and ACDC evaluations, single-variable ablations, and gradient statistics (effective rank and foreground/background gradient contrast) complement these results. Dense-probe gains are largest for supervised ViT-S (e.g., +24 mIoU on Cityscapes) and become smaller under SSL and at B scale, while ImageNet top-1 drops in 5 of the 7 settings.

Critical strengths

1.Simple, cheap, plug-in change with consistent dense gains. CAg is parameter-free, and the whole change costs about 1.4% FLOPs. Dense linear-probe mIoU improves across five training regimes, with large gains on thin classes (+10 to +28 mIoU on Cityscapes thin classes) and on ACDC. If these gains hold under controlled comparisons, this is an easy-to-adopt improvement for any CLS-trained ViT. This is the main reason the paper is worth considering.

2.A fresh, checkable angle on CLS–patch interaction. The last-block rank bound is simple and is confirmed empirically (e.g., effective rank about 20 versus a bound of 24 for ViT-B). The CAg-only model (CAT-S, no convolution) already gives large gains over DeiT-S in supervised ViT-S (+5.1 / +7.2 / +18.6 / +5.6 on ADE / VOC / Cityscapes / COCO-Stuff). This shows the idea has real substance at least in that setting.

Critical weaknesses

1.The proposed mechanism is not causally supported.

  • The claim that patch losses cannot compensate is untested: for iBOT, only the CLS-loss part of the gradient is measured. The paper’s own numbers suggest patch losses compensate a lot, since CALiT’s gains shrink from +7 / +11 / +24 (supervised S) to +1 / +3 / +5 (iBOT-B) on ADE / VOC / Cityscapes.
  • The gradient evidence is correlational. Effective rank rises almost automatically once the CLS loss passes through each patch’s own FFN nonlinearity, regardless of how the aggregation is done. Consistent with this, the uniform-mean CAg in Table 3, which sends the same hidden-space gradient to every patch, still retains a large part of the dense gain.

2.The experimental comparisons do not isolate the claimed contribution.

  • Compared with the baselines, CALiT adds two components at once: CAg and a 3×3 depthwise convolution (LMix). LMix is not a contribution of this paper, and it is unclear whether it is an empirical design choice or follows from the gradient analysis. Since both are added together, most reported gains reflect their combined effect. More CAg-only experiments (CAT without LMix), especially under SSL and at B scale, are needed to support the claim that the improvements come from cross aggregation.
  • Several comparisons do not cleanly isolate the architectural contribution. Table 1 mixes public checkpoints, results borrowed from another paper, and newly trained CALiT models, with different pre-training schedules and incompletely specified recipe matching. Beating a longer-trained baseline on dense probes is encouraging, but by itself does not establish an architectural gain or faster convergence; that would require matched learning curves or compute-controlled comparisons.
  • Comparisons with the most relevant alternatives are missing: registers (Darcet et al., 2024), the [CLS]/patch specialization of Marouani et al. (2026), which the authors call the most related work, and LaSt-ViT (Shi et al., CVPR 2026), which also aggregates patch features into the CLS token and is not cited. Marouani et al. report their method under the same probe protocol and on the same DeiT-III-B and DINOv2-B baselines borrowed here, so that comparison could be added at no training cost. Without it, it is unclear whether CALiT improves over existing solutions to the same problem.
  • The broader representation-quality and efficiency claims require stronger qualification. Dense linear-probe gains coexist with lower ImageNet classification accuracy in every self-supervised block of Table 1, including a 2.8-point drop for DINOv2-B. This suggests a meaningful global–local trade-off rather than an unqualified improvement in representation quality, which the abstract and introduction do not acknowledge. Moreover, the small FLOP overhead does not establish low practical cost: CAg breaks the per-token independence of the FFN and LMix requires a full 2D token grid, yet training throughput, inference latency, and peak memory are not reported.
Questions for authors
  • Fair comparison. Were the ViT baselines and CALiT trained under strictly matched settings (same codebase, recipe, schedule, and architectural details such as attention bias)? If not, could you report same-recipe, same-codebase ViT baselines probed with your own pipeline, at least for supervised ViT-S and iBOT ViT-S at 300 epochs?
  • Source of the gains beyond supervised ViT-S. Outside supervised ViT-S, all results use the full model (cross aggregation + depthwise convolution), so the two cannot be separated. In at least one self-supervised setting (e.g., iBOT ViT-S), could you report (a) cross aggregation only, without the convolution, and (b) a plain ViT with only the Dirac-initialized depthwise convolution, both trained with the same recipe as CALiT?
  • Inconsistent results. Why does CALiT-B (iBOT) obtain 59.66 Cityscapes mIoU in Table 1 but 58.46 in Table 2, although Table 2 states it uses “the Table 1 linear probe”? Cityscapes values differ for six of the seven models (all lower in Table 2), and this row carries an undefined “¶” marker.
2 comments
AuthorsOpenReview ↗

Gradients, Recipe and Inconsistent Results

We thank the reviewer for the valuable feedback. We discuss the majority of the concerns in the general comment and address the remaining ones below.

Patch MIM on LMix: We include the iBOT-S+LMix 300 epoch experiments. We will also include plots to show total gradient agreements with iBOT on object patches in the main paper within the rebuttal period. The fact that mean pooled aggregation works is not a weakness but validates the low rank gradients. Note that same pooling weights does not imply same gradients.

CAg only experiments: CAg is a constraint on CLS and not itself a compute of patches, therefore it provides improvement on patches through other modules. In particular, it improves the impact of LMix further. We additionally ablate CAg in iBOT-S CALiT and show that the performance of LMix is tied to CAg. Although, we are not the first ones to propose LMix, we are the first to show LMix can improve performance in segmentation when combined with CAg, e.g., for high resolution images with thin classes (e.g., AC-DC, Cityscapes), and not by small margins. Similarly we also add ViT-S + LMix (same recipe) which provides another reference. The CAg ablation shows that in the S scale, it is simply not possible to obtain comparable results as CALiT with supervised loss. Table 3 further provides additional configurations in the exact Deit-I recipe. We agree to add the results of Marouani et al’s table in Table 1. Also we agree to qualify the contributions of FLOPs with actual throughput measured, -9% for CAg in ViT-S, with no change in memory observed.

Matched recipe: We trained CALiT with a matched recipe of Deit-I in CALiT-S, though for compute efficiency, we doubled the batch size to 2048. In order to address the recipe issue, we additionally train iBOT-S+LMix and ViT+LMix, which shows a directionally similar impact of LMix. LocalViT is weaker here because of activation and initialization. We thank the reviewer for identifying this experiment which we believe has improved the work. In order to show the impact of CAg, we choose to ablate it in ViT-S+LMix, with comparison below. We will also provide full gradient agreement plots on the appendix.

Inconsistent Results : We thank the reviewer for finding the inconsistencies. On the first two pairs, the small inconsistencies are simply due to noise in re-probing, as earlier linear heads from Table 1 were lost. However, the last row with marker “¶” of Table 2 results were unfortunately added before convergence was achieved. We will include the final result in Table 2 (59.13) and average it with the probes obtained in Table 1 for lower noise. We agree to release the linear heads as well as the checkpoints for reproducibility.

Thank you for the clarifications and additional ablations. However, my main concerns remain: the comparisons still need strictly matched plain ViT baselines and a CAg-only ablation under self-supervised training. I therefore maintain my rating.

Reviewer_amunOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CALiT to improve patch representations in ViT. It shows that CLS-based losses provide limited gradients to patches in the final transformer block. CALiT addresses this through CAg to improve gradient flow from the CLS token to patches and depthwise convolution for local feature mixing. Results show improved dense prediction performance on different pre-training methods and datasets.

Critical strengths
  • Clear motivation: the paper identifies a limitation of CLS-based supervision via gradient analysis and motivate the CALiT architectural design.
  • Simple but effective: CALiT introduces a simple modification to the standard ViT architecture. It requires no additional learnable parameters and has minor computational overhead.
  • Comprehensive evaluations: Experiments include supervised learning, DINO, iBOT, DINOv2, and vision-language fine-tuning. CALiT improves dense linear-probe performance across multiple datasets.
Critical weaknesses
  • Performance loss: Table 1 shows that CALiT mostly reduces ImageNet classification accuracy compared with baselines. No sufficient discussions on the performance drop or possible ways to mitigate it.
  • Misaligned setups: Some results (e.g. ViT-B in table 1), are previous reported result, so it is unclear whether the experimental settings are fully aligned. Using fewer pre-training epochs also raises concerns about the fairness of the comparisons.
  • Generalizability: The evaluation mainly uses frozen-backbone linear probes, no results for end-to-end fine-tuning scene. CALiT improves patch representations but may reduce image classification accuracy. This trade-off raises concerns about its effect as general vision encoder.
Questions for authors
  1. (Important, need to address) No direct comparison with Revisiting [CLS] and Patch Token Interaction in Vision Transformers. This work also improves patch representations while maintaining classification performance, and some reported baseline results are from this work. How does CALiT compare with this method under same settings?
  2. CALiT improves patch semantics but reduces classification accuracy in several settings, particularly with DINOv2. Could the authors further investigate this trade-off and provide additional classification results on other datasets?
2 comments

We thank the reviewer for the feedback. We answer Q2 in the general section above with additional experimental results.

Q1. IN1k - linear probe loss. We agree to add discussion on probe loss. First the hard constraint of the CAg forces the CLS to additionally satisfy patch attention. In SSL, there is no semantic loss on the CLS similar to language alignment or classification, therefore the performance loss is higher. Two ways to mitigate this exist. First is to relax this CAg as shown in Table 3, so cross-aggregation does not happen on all heads. The second approach is to add a CLS based loss, e.g., in language alignment, so any performance loss is insignificant. We agree to provide additional results within the discussion period.

Q3. Generalizability. Full model fine-tuning. In semantic probes, a full model finetuning is no longer used as a good indicator, as was common in early 2020s. Therefore, models like DINOv2, SigLIP2 and the progressions use linear probes instead. We follow the same protocols.

Q4. Comparisons. We agree to include the results of Marouani et al. in table 1. In particular, our DINOv2 probes show significant gain in Cityscapes as is expected from having both CAg and LMix (CAg improves LMix).

Thank you for the additional experiments and explanations. The response address some of my concerns. However, direct comparisons (under same setup) with related methods are still missing, making it difficult to assess the contribution of CALiT.

Reviewer_b4u9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to improve patch feature quality in Vision Transformers from an architectural perspective. It observes that gradient flowing from losses over the CLS token to patches in the final blocks has a very small rank which makes them insufficiently differentiated and spatially correlated. To remedy this issue, the paper proposes adding a cross aggregation module inside every block’s FFN, which refines the CLS token by aggregating patch features into it. This module enriches gradients to patches by providing another gradient path through them. The paper additionally adds a Local Mixing module to further improves patch quality.

The paper shows improved performance on segmentation tasks compared to plain ViT and LocalViT in several training paradigms. It improves significantly the performance on thin class semantic segmentation. The paper also provides ablation studies demonstrating the contributions of different design choices.

Critical strengths

The paper’s analysis is thorough from both theoretical and empirical perspectives. It derives that the rank of gradients from losses on CLS to patches is bounded and small, and validated this with empirical results for ViT-S. The observation on the collapse of the gradient through the Value path is interesting.

The paper provides experimental results with different training paradigms (supervised, self-supervised, vision-language fine-tuning) to validate its proposed method. The results look good compared to plain ViT and LocalViT in segmentation tasks. Performance gain is particularly impressive in thin objects segmentation.

Critical weaknesses
  • Though interesting, the analysis on the rank of patch gradient only applies in the last block. Patches in the earlier blocks should receive gradients with higher ranks. It is not clear the issue in the last block justifies adding CAg in very block’s FFN. It is also not clear how an issue located only in the last block could have such large impact to the patch features, as shown in Figure. 1.

  • The specific implementation of CAg is not very well justified. It could be implemented anywhere in the blocks's FFN. It is not clear what motivates the paper's choice of post GELU activations. Also, it it not clear why CAg is necessary in every blocks. Claim "patches in the subsequent blocks still receive lower quality corrective signals without CAg in every block" in L.464-465 are not justified either experimentally or empirically. It would be also interesting to visualize patch gradients (something similar to Fig.1b) with CAg and without LMix to isolate the impact of CAg.

  • There are some confounding factors in the comparisons which make them not completely controlled:

    • The paper trains models for 300 epochs instead of 800 epochs for DINO and iBoT. The paper claims that better gradient allows shorter trainings but this claim is not justified experimentally. Indeed, the significant drop in IN1k seems to counter it. Also, previous works observe that patch quality could degrade when training longer, it would be important to verify if improved performance in segmentation comes from this confounding factor.
    • The paper uses numbers reported in Marouani et al. for ViT-B scale instead of reevaluating them. This adds a confounding factor in the results and leads to some N/A in Table 1.
  • The experimental section lacks some important comparisons to other methods that also seek to improve patch features through architectural modifications such as Marouani et al. (ViT + specialisation) and Darcet et al. (ViT + registers).

  • The paper does not report results in ViT-Large scale. Since ViT behaviours could change when scaling (Darcet et al.), it is important to validate CALiT in larger scale.

  • The paper only considers segmentation benchmarks. Related methods consider both depth estimation and segmentation. Adding depth estimation as benchmarks is necessary to have a fuller picture on the impact of the proposed method.

Questions for authors

Q: L.308 stated that the fact supervised learning benefits the most from CALiT is predicted by the rank deficiency in the gradient, what justifies this claim?

Suggestions: the paper would strengthens significantly if concerns in "Weaknesses" are properly addressed.

1 comment
AuthorsOpenReview ↗

Scaling and CAg location

We thank the reviewer for the feedback. We answer Q1, Q3--5 in the general section.

Q2. CAg Location: There are two valid choices, before and after the activation in FFN. Our motivation is clarified using the idea of privileged basis argument in Page 2. In order for a meaningful pooling of patch features, they need to occur in a semantic space, and therefore after the activation. This is discussed in the Introduction L74 –86.

Q4--5. Scaling and depth probes: CALiT for patch semantics when trained with IN1k is already saturated at scale S, therefore CALiT reaches the data limit much earlier than ViT. Note that linear probe in scale B uses twice the parameters (D X N_classes) as scale S and yet has similar performance. DINOv2 IN21k uses 8X data and therefore brings noticeable gain with B scale providing similar performance as DINOv2-L in Cityscapes. We show that CALiT scales with data and compute through the DINOv2-B and LeVLJEPA experiments, which use far more unique data than others. A true CALiT-L on an even larger amount of data (as is needed in CALiT) is currently not possible.

Depth probes in ViT do not provide accurate depth values well enough to be actually useful, compared to semantics. The convolutions in LMix further learn smoothing as well as edge filters, the former is largely beneficial only for semantics and somewhat detrimental to depth. Although CAg (the CAT model) shows improved performance vs ViT on the depth probe, we prefer to explicitly position our paper on semantics. For example, we have the following results in depth probes:

Supervised S/16 (IN-1k, 300 ep)SUN RGB-D RMSE ↓AbsRel ↓δ1 ↑KITTI RMSE ↓AbsRel ↓δ1 ↑
ViT-S0.60080.21460.66314.0960.12150.8494
CAT-S (CAg only)0.58350.21050.67334.0100.11350.8587
CALiT-S (CAg + LMix)0.60210.21220.66163.8790.11880.8521

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary and general replies

We thank the reviewers for their time and the feedback which have changed the paper positively.

In summary, the contributions restated are: i) Discovering patch gradients from the CLS-loss (supervised ViT, SigLiP) are rank limited on the final layer. ii) Correcting it via CAg a flexible (50% already recovers 90% gains in ViT-S) coupling, which specifically provides support to FFN convolutions (LMix). CAg especially improves ViT+LMix (short convolutions in FFN) beyond just ViT+CAg. Evaluations are performed using semantic segmentation with linear probes showing very large gains, specifically in thin classes and high resolutions (Cityscapes, ACDC). We reply to the major concerns here and address other concerns individually : i) Meaning of patch gradients on the last block ii) Additional controlled experiments iii) Additional Comparisons.

G1. Patch Gradients: We agree that patch gradients in earlier blocks can have a higher algebraic rank after passing through further layers. But under a CLS-only loss, every gradient reaching patches in earlier blocks is the back-propagation of the signal at the input of the last block. This being low rank implies fewer bases in the corrective signal at the source, for the patches. We measure the gradient ranks empirically, and measure the consequences of the designs in a thorough ablation, which justify the modification.

Next, the issue is not only in the last block. The loose coupling of CLS and patches exist until the first block, hence the same issue repeats down the blocks as gradient of CLS in block L-2 output is rank-limited on the input patches of block L-2. Another issue is that the Cross Aggregation is a hard constraint as is currently implemented and forces CLS to re-calibrate itself to the space of patches. Therefore, having CAg in earlier layers helps late blocks to follow the same constraint during the optimization. We agree to add discussions in the main text of the paper.

G2. Confounding factors: We provide additional evaluations and training to ascertain this is not the case. iBOT-S + LMix: We choose this configuration for two reasons - it contains LMix, an ablation of CAg with patch MIM improves the validation. Additionally, from the patch MIM, iBOT-S+LMix 300 epochs should also give an upper bound for iBOT-S 300 epochs. We report the following comparison on iBOT-S vs iBOT-S+LMix, which shows CAg is necessary in order to obtain gain from LMix even with patch MIM:

iBOT ViT-S/16 (IN-1k, 300 ep)ADE20KVOCCityscapesCOCO-Stuff
ViT + LMix (no CAg)31.5170.2853.5336.02
CALiT (CAg + LMix)32.9674.1256.1337.93
Δ (CAg)+1.45+3.84+2.60+1.91
Supervised ViT-S/16 (IN-1k, 300 ep)ADE20KVOCCityscapesCOCO-Stuff
ViT + LMix (no CAg)25.7758.7244.2529.09
CALiT (CAg + LMix)31.6671.4351.1536.68
Δ (CAg)+5.89+12.71+6.90+7.59
Note that the re-probing always results in slightly different numbers (single run).

G3. Additional Comparisons: Additionally we note that, DINOv2 600k is the largest scale training, which shows similar performance as ViT-L from Marouani et al. No checkpoints are released from the authors, therefore we leave the COCO results out.

Additionally, Darcet et al., which provides a different direction of changes in ViT, and corrects the high norm patch attention, but does not directly benefit linear probe-based segmentations. We observed no gain in ViT-S. We agree to include Marouani et al., in the final version for at least 3 models, though only modest changes can be expected here as is shown in the original paper. Similarly, we agree to add linear probes on ViT-B of LastViT. Currently, we have ADE and VOC probes for DeiT-B (26.59, 60.40) which is lower than our model. We will add the full evals in the main paper within the rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.