- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that ViT patch features are weak partly because of how CLS-based losses train them. In the last block, patches influence the CLS loss only through CLS attention (the value and key routes), so the patch-gradient matrix has rank at most 2 × #heads. Empirically, this gradient is dominated by a single shared direction, so foreground and background patches receive similar updates. To address this, the authors propose CALiT, which modifies only the FFN of each block. Cross Aggregation (CAg) is a parameter-free attention in the post-GELU hidden space: the CLS hidden vector is split into slots that attend to all patch slots, and the result replaces the CLS hidden. Local Mixing (LMix) is a Dirac-initialized 3×3 depthwise convolution on the patch hidden, adapted from LocalViT. The total overhead is about +1.4% FLOPs at ViT-S.
The method is validated mainly with frozen-backbone linear probes on ADE20K, VOC, Cityscapes and COCO-Stuff, under supervised, DINO, iBOT and DINOv2 pre-training as well as LeVLJEPA fine-tuning. Thin-class and ACDC evaluations, single-variable ablations, and gradient statistics (effective rank and foreground/background gradient contrast) complement these results. Dense-probe gains are largest for supervised ViT-S (e.g., +24 mIoU on Cityscapes) and become smaller under SSL and at B scale, while ImageNet top-1 drops in 5 of the 7 settings.
Critical strengths
1.Simple, cheap, plug-in change with consistent dense gains. CAg is parameter-free, and the whole change costs about 1.4% FLOPs. Dense linear-probe mIoU improves across five training regimes, with large gains on thin classes (+10 to +28 mIoU on Cityscapes thin classes) and on ACDC. If these gains hold under controlled comparisons, this is an easy-to-adopt improvement for any CLS-trained ViT. This is the main reason the paper is worth considering.
2.A fresh, checkable angle on CLS–patch interaction. The last-block rank bound is simple and is confirmed empirically (e.g., effective rank about 20 versus a bound of 24 for ViT-B). The CAg-only model (CAT-S, no convolution) already gives large gains over DeiT-S in supervised ViT-S (+5.1 / +7.2 / +18.6 / +5.6 on ADE / VOC / Cityscapes / COCO-Stuff). This shows the idea has real substance at least in that setting.
Critical weaknesses
1.The proposed mechanism is not causally supported.
- The claim that patch losses cannot compensate is untested: for iBOT, only the CLS-loss part of the gradient is measured. The paper’s own numbers suggest patch losses compensate a lot, since CALiT’s gains shrink from +7 / +11 / +24 (supervised S) to +1 / +3 / +5 (iBOT-B) on ADE / VOC / Cityscapes.
- The gradient evidence is correlational. Effective rank rises almost automatically once the CLS loss passes through each patch’s own FFN nonlinearity, regardless of how the aggregation is done. Consistent with this, the uniform-mean CAg in Table 3, which sends the same hidden-space gradient to every patch, still retains a large part of the dense gain.
2.The experimental comparisons do not isolate the claimed contribution.
- Compared with the baselines, CALiT adds two components at once: CAg and a 3×3 depthwise convolution (LMix). LMix is not a contribution of this paper, and it is unclear whether it is an empirical design choice or follows from the gradient analysis. Since both are added together, most reported gains reflect their combined effect. More CAg-only experiments (CAT without LMix), especially under SSL and at B scale, are needed to support the claim that the improvements come from cross aggregation.
- Several comparisons do not cleanly isolate the architectural contribution. Table 1 mixes public checkpoints, results borrowed from another paper, and newly trained CALiT models, with different pre-training schedules and incompletely specified recipe matching. Beating a longer-trained baseline on dense probes is encouraging, but by itself does not establish an architectural gain or faster convergence; that would require matched learning curves or compute-controlled comparisons.
- Comparisons with the most relevant alternatives are missing: registers (Darcet et al., 2024), the [CLS]/patch specialization of Marouani et al. (2026), which the authors call the most related work, and LaSt-ViT (Shi et al., CVPR 2026), which also aggregates patch features into the CLS token and is not cited. Marouani et al. report their method under the same probe protocol and on the same DeiT-III-B and DINOv2-B baselines borrowed here, so that comparison could be added at no training cost. Without it, it is unclear whether CALiT improves over existing solutions to the same problem.
- The broader representation-quality and efficiency claims require stronger qualification. Dense linear-probe gains coexist with lower ImageNet classification accuracy in every self-supervised block of Table 1, including a 2.8-point drop for DINOv2-B. This suggests a meaningful global–local trade-off rather than an unqualified improvement in representation quality, which the abstract and introduction do not acknowledge. Moreover, the small FLOP overhead does not establish low practical cost: CAg breaks the per-token independence of the FFN and LMix requires a full 2D token grid, yet training throughput, inference latency, and peak memory are not reported.
Questions for authors
- Fair comparison. Were the ViT baselines and CALiT trained under strictly matched settings (same codebase, recipe, schedule, and architectural details such as attention bias)? If not, could you report same-recipe, same-codebase ViT baselines probed with your own pipeline, at least for supervised ViT-S and iBOT ViT-S at 300 epochs?
- Source of the gains beyond supervised ViT-S. Outside supervised ViT-S, all results use the full model (cross aggregation + depthwise convolution), so the two cannot be separated. In at least one self-supervised setting (e.g., iBOT ViT-S), could you report (a) cross aggregation only, without the convolution, and (b) a plain ViT with only the Dirac-initialized depthwise convolution, both trained with the same recipe as CALiT?
- Inconsistent results. Why does CALiT-B (iBOT) obtain 59.66 Cityscapes mIoU in Table 1 but 58.46 in Table 2, although Table 2 states it uses “the Table 1 linear probe”? Cityscapes values differ for six of the seven models (all lower in Table 2), and this row carries an undefined “¶” marker.
2 comments
Thank you for the clarifications and additional ablations. However, my main concerns remain: the comparisons still need strictly matched plain ViT baselines and a CAg-only ablation under self-supervised training. I therefore maintain my rating.
Gradients, Recipe and Inconsistent Results
We thank the reviewer for the valuable feedback. We discuss the majority of the concerns in the general comment and address the remaining ones below.
Patch MIM on LMix: We include the iBOT-S+LMix 300 epoch experiments. We will also include plots to show total gradient agreements with iBOT on object patches in the main paper within the rebuttal period. The fact that mean pooled aggregation works is not a weakness but validates the low rank gradients. Note that same pooling weights does not imply same gradients.
CAg only experiments: CAg is a constraint on CLS and not itself a compute of patches, therefore it provides improvement on patches through other modules. In particular, it improves the impact of LMix further. We additionally ablate CAg in iBOT-S CALiT and show that the performance of LMix is tied to CAg. Although, we are not the first ones to propose LMix, we are the first to show LMix can improve performance in segmentation when combined with CAg, e.g., for high resolution images with thin classes (e.g., AC-DC, Cityscapes), and not by small margins. Similarly we also add ViT-S + LMix (same recipe) which provides another reference. The CAg ablation shows that in the S scale, it is simply not possible to obtain comparable results as CALiT with supervised loss. Table 3 further provides additional configurations in the exact Deit-I recipe. We agree to add the results of Marouani et al’s table in Table 1. Also we agree to qualify the contributions of FLOPs with actual throughput measured, -9% for CAg in ViT-S, with no change in memory observed.
Matched recipe: We trained CALiT with a matched recipe of Deit-I in CALiT-S, though for compute efficiency, we doubled the batch size to 2048. In order to address the recipe issue, we additionally train iBOT-S+LMix and ViT+LMix, which shows a directionally similar impact of LMix. LocalViT is weaker here because of activation and initialization. We thank the reviewer for identifying this experiment which we believe has improved the work. In order to show the impact of CAg, we choose to ablate it in ViT-S+LMix, with comparison below. We will also provide full gradient agreement plots on the appendix.
Inconsistent Results : We thank the reviewer for finding the inconsistencies. On the first two pairs, the small inconsistencies are simply due to noise in re-probing, as earlier linear heads from Table 1 were lost. However, the last row with marker “¶” of Table 2 results were unfortunately added before convergence was achieved. We will include the final result in Table 2 (59.13) and average it with the probes obtained in Table 1 for lower noise. We agree to release the linear heads as well as the checkpoints for reproducibility.