- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks why feature-level distillation from a self-supervised vision foundation model into CLIP does not transfer the VFM's robustness to domain shift, and motivates its answer with three observations comparing EVA-CLIP and DINOv2-reg: UMAP plots of clean and shifted patch features, layer-wise linear CKA between ACDC adverse and reference domains, and the Jaccard overlap of the top-10% "prediction-relevant" neurons per layer, all of which favour DINOv2-reg. The proposed ReoNa fine-tunes EVA02-CLIP-B/16 on COCO. GT region masks pool frozen DINOv2-reg features from an image and a photometrically shifted view into a cross-view consensus; at blocks 4, 8 and 12, a teacher router fed with this consensus and a student router fed with pooled CLIP features each output a softmax over the 2048 MLP hidden channels of that block, which Eq. 6 turns into a mean-one channel gate. Training combines KL distillation from the teacher to the student routing with a JS consistency between views, a swap loss that exchanges the gates of the two views and aligns the resulting class predictions, and a masked reconstruction of the VFM consensus under the teacher gates. At inference, a first unrouted pass builds soft regions from text-similarity maps and a second, gated pass produces the dense features. On the OVDG-SS benchmarks, ReoNa reaches 44.0 and 39.4 average zero-shot mIoU on Dv-19 and Dv-58, against 32.6 and 32.7 for DeCLIP, and improves by 0.7 to 2.2 mIoU over the best baseline when combined with the S²-Corr decoder.
Critical strengths
The empirical gains are large and consistent across settings. Making CLIP's dense features robust to domain shift is a relevant problem for open-vocabulary perception, and ReoNa improves zero-shot mIoU substantially and remains ahead with fine-tuned decoders, including CAT-Seg, which is not designed for domain generalisation. The appendix also shows that SAM masks reach 43.6 and 38.9 mIoU against 44.0 and 39.4 with GT regions, so the method does not depend on COCO annotations. Looking at which internal channels a model relies on across domains is an interesting angle. Asking what feature-level distillation fails to transfer, beyond the output features themselves, is a fresh way to frame robustness distillation.
Critical weaknesses
1. The central claim that teaching CLIP which channels to use is better than aligning features is supported neither by the motivating analysis nor by the ablations. The "neurons" of Observation 3 are, according to App. B.2, the 768 channels of each block's output, i.e. the residual stream, whereas the method routes and gates the 2048 hidden channels inside the MLP, and the teacher router is a learned map from VFM features to CLIP channels, so the VFM's own neuron structure is never transferred. The terminology also overstates what is measured: the importance score in Eq. 14–15 is the activation–gradient product of the squared norm of the spatially averaged final feature, which involves no prediction, the "final feature" is defined differently for the two models in App. B.1, and no random-selection baseline is given. On the experimental side, Table 4 starts from the published DeCLIP number and then adds a "routing" step that, according to its footnote, already includes the unmasked reconstruction loss, so it changes the data, the region pooling, the cross-view target, the feature reconstruction and the routing at once. The decisive control, the same pipeline with the same regions and cross-view consensus trained with region-level feature distillation only, without routers or gates, is missing.
2. The comparisons are not fair and omit the most relevant alternatives, so the size of the gains cannot be attributed to the method. ReoNa distills from DINOv2-reg, whereas its main baseline DeCLIP uses DINOv2-B by default, and the paper's own Table 6 shows that with DINOv2-B ReoNa drops from 44.0 and 39.4 to 38.5 and 36.1 mIoU. That means about half of its zero-shot advantage over DeCLIP may come from the teacher alone. The paper does not state whether the baseline rows were re-trained or copied, or whether their data, resolution and schedule match ReoNa's, and the zero-shot comparison gives ReoNa a two-pass, vocabulary-conditioned inference that recomputes most of the encoder, while all baselines use a single pass and the inference cost is not reported. Encoders that already combine language alignment with self-supervised objectives, such as SILC [1], TIPS [2] and SigLIP 2 [3], target the same weakness of CLIP's dense features but are neither discussed nor compared, so it is unclear whether the method still helps a stronger, current student.
[1] M. F. Naeem, Y. Xian, X. Zhai, L. Hoyer, L. Van Gool, F. Tombari. SILC: Improving Vision Language Pretraining with Self-Distillation. ECCV, 2024.
[2] K.-K. Maninis et al. TIPS: Text-Image Pretraining with Spatial Awareness. ICLR, 2025.
[3] M. Tschannen et al. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv:2502.14786, 2025.
Questions for authors
-
Please report the routing-free control: the same COCO data, regions, photometric view pairs and cross-view consensus, trained with region-level feature distillation only, without routers or gates. How much of the gain in Table 4 remains attributable to routing?
-
How does DeCLIP perform when re-trained with DINOv2-reg and the same data, resolution and schedule as ReoNa? Please state for every row of Tables 1–3 whether it was re-trained or copied.
-
What is the zero-shot performance of ReoNa with a single unrouted pass, and what are its inference latency and FLOPs compared with DeCLIP?
-
Does ReoNa still improve a student that is already trained with self-supervised objectives, such as SigLIP 2, and how does it compare with SILC and TIPS on these benchmarks?
9 comments
(2/3) Response to 8sMt-W1
(3) The importance score in Eq. 14-15 involves no predictor
In Appendix B.2, we use a class-agnostic objective to quantify the influence of an activation on feature energy , measured by . This class-agnostic objective enables us to compare how consistently the two models reuse channels to support dense representations across domains, without training an additional probe.
(i) How to compute ?
For a single image , we spatially average its final dense features and define their mean squared magnitude as the attribution target: where is the number of spatial tokens and is the feature dimension. The scalar measures the channel-normalized squared magnitude of the globally pooled dense representation, without an additional per-token normalization before computing the objective. Each encoder's readout is held fixed across domains, and channel-set overlap is computed within the same encoder, rather than by matching feature coordinates or absolute energies between the two models.
The absolute activation-gradient attribution is
(ii) Why use to measure the effect of a relative change?
We use this product because it measures the magnitude of the first-order energy change per unit fractional perturbation of an activation. Let denote the feature response in channel of patch token at the output of Transformer block for image . The gradient measures the energy sensitivity to an absolute change in that response.
To quantitatively evaluate the effect of on feature energy , consider a small relative attenuation produces . A first-order Taylor approximation therefore gives Hence, : for the same small attenuation fraction, a larger attribution predicts a larger local energy change. The activation supplies the perturbation scale, while the gradient supplies the energy sensitivity, which explains why their product measures influence under relative perturbations. We average these absolute contributions over spatial positions and sampled images to rank channels. The overlap of high-scoring channels then probes whether the same output-supporting components remain influential across domains.
(4)"Final feature" is defined differently
In Appendix B.1--B.2, the final feature is the dense visual representation at spatial position used to define the attribution objective. For EVA-CLIP, we use the dense encoder output. For DINOv2-reg, we use the spatial patch tokens after the final Transformer block and before the final LayerNorm, excluding the class and register tokens. These are encoder-specific dense readouts for dense prediction tasks like segmentation.
(3/3) Response to 8sMt-W1
(5) Random selection baseline
Thanks for your advice on comparing with randomly selected channels. To test the semantic relevance of the selected channels, we have added a random-channel attenuation control evaluated through segmentation.
For DINOv2-reg and EVA-CLIP, we freeze the image encoder and train a linear segmentation probe on ACDC training data, then evaluate on ACDC night. Following Appendix B.2, we select the top-10% block-output channels per layer using only the associated normal-condition reference images, and fix their indices for night-time evaluation. We compare attenuating these reference-selected channels with attenuating an equal number of random channels, using the same per-layer channel counts, intervention locations, and attenuation factor. The encoder weights and trained probe remain fixed during intervention evaluation.
| Intervention on ACDC night | DINOv2-reg mIoU | DINOv2-reg mIoU | EVA-CLIP mIoU | EVA-CLIP mIoU |
|---|---|---|---|---|
| No attenuation | 40.5 | NA | 18.4 | NA |
| Attenuate reference-selected top-10% channels | 24.7 | -15.8 | 17.4 | -1.0 |
| Attenuate an equal number of random channels | 39.8 | -0.7 | 18.2 | -0.2 |
For DINOv2-reg, attenuating the selected channels reduces performance by 15.8 mIoU, compared with only 0.7 mIoU for random channels. For EVA-CLIP, the corresponding drops are 1.0 and 0.2 mIoU. Since channel selection uses reference images while semantic evaluation uses night images with those indices held fixed, the results show that energy-selected neurons support semantic prediction across views, particularly for DINOv2-reg. Together with its higher shared-neuron similarity in Observation 3, the much stronger selected-versus-random contrast supports the interpretation that DINOv2-reg's circuit sharing reflects semantically useful, reusable internal support. EVA-CLIP exhibits a substantially weaker contrast under the same intervention protocol.
(6) Routing-free control (Corresponds to Q1)
Thank you for suggesting this control. We agree that the original cumulative ablation in Table 4 does not separately isolate the benefit of the cross-view consensus and reconstruction objectives from that of routing. We have therefore added two routing-free controls that retain the same COCO data, region masks, photometric view pairs, and VFM cross-view consensus, but remove the region-to-channel routers and channel-modulation gates. The first aligns region features to the consensus using feature-level supervision only. The second additionally reconstructs the same consensus from masked inputs, still without any routing or channel modulation. This progressively tests whether the gains can be explained by the augmented-view consensus and masked reconstruction alone. The DeCLIP reference uses its officially released weights.
| Method | CS-19 | ACDC-19 | BDD-19 | Mapi-19 | Avg. (Dv-19) | ACDC-41 | BDD-41 | Mapi-30 | RW-10 | Avg. (Dv-58) |
|---|---|---|---|---|---|---|---|---|---|---|
| Feature alignment (DeCLIP) | 32.5 | 30.0 | 30.9 | 37.1 | 32.6 | 32.4 | 39.6 | 33.1 | 25.7 | 32.7 |
| Feature alignment to cross-view consensus | 33.2 | 30.0 | 31.0 | 36.7 | 32.7 | 33.0 | 39.4 | 33.6 | 25.5 | 32.9 |
| Feature alignment + masked consensus reconstruction | 34.0 | 30.3 | 31.0 | 36.9 | 33.1 | 34.7 | 41.8 | 33.8 | 25.2 | 33.9 |
| ReoNa | 44.2 | 40.7 | 43.8 | 47.2 | 44.0 | 42.4 | 48.8 | 37.5 | 28.8 | 39.4 |
Both feature-level additions provide incremental benefits. Using the cross-view consensus for feature-level alignment increases average performance from 32.6 to 32.7 mIoU on Dv-19 and from 32.7 to 32.9 mIoU on Dv-58. Adding masked reconstruction of that consensus further improves performance to 33.1 and 33.9 mIoU, respectively, contributing another 0.4 and 1.0 mIoU. Thus, incorporating appearance-augmented teacher evidence and learning to reconstruct it from masked inputs are beneficial without channel modulation. Without routing, these two additions retain gains of only 0.5 mIoU on Dv-19 and 1.2 mIoU on Dv-58 over DeCLIP, compared with the full ReoNa gains of 11.4 and 6.7 mIoU. These results show that richer feature-level targets and masked reconstruction alone do not account for most of the improvement. The substantial remaining advantage supports explicitly learning and applying region-conditioned channel support across appearances, rather than only aligning the resulting region features.
We have updated Table 4 and corresponding content in Section 4 in our revised PDF to include this control.
(1/3) Response to 8sMt-W1
We thank the reviewer for their kind review and feedback on our work. Below, we provide point-to-point responses to each weakness raised by the reviewer.
For Weakness 1, we organize our response into six parts: (1) the connection between block-output analysis and MLP-channel routing; (2) discuss what is distilled; (3) the importance-score definition and rationale; (4) the final-feature definitions; (5) the random selection baseline; and (6) the routing-free control.
(1) Relation between block-output dimension and MLP channels
The 768 block-output channels and the 2048 MLP hidden channels are connected through the MLP output projection. In Observation 3, we analyze the 768-dimensional block outputs because they expose the feature representations on which dense prediction is built. Analysis at this level characterizes cross-domain reuse as it is expressed in the features directly for dense prediction.
During training, we gate the 2048 hidden channels inside the MLP because they are the computational components that generate the MLP's update to these representations. The output projection maps their contributions back into the 768-dimensional residual stream, where they are added to the residual state. Gating before this projection directly controls how each MLP component contributes to the block's features. Thus, the analysis examines the resulting feature representations, while training intervenes in the internal computation that shapes them.
(2) VFM's own neuron structure is not transferred.
The SSL-pretrained VFM and CLIP are trained with fundamentally different objectives and learn representations in distinct feature spaces. Their channels have no predefined semantic correspondence, so directly copying the identity or connectivity of particular teacher neurons into CLIP is not a well-defined transfer operation. Our goal is to learn reusable internal support across appearance changes. ReoNa learns this support in CLIP's own MLP-channel space, guided by domain-shared semantic evidence from the VFM.
Specifically, the teacher router takes the VFM cross-view consensus as input and predicts QT=softmax(RT(Tcons)), an assignment over CLIP's MLP channels, not an importance vector over VFM neurons. These assignments modulate the CLIP MLPs during consensus reconstruction. The reconstruction loss updates the teacher router according to how its assignments affect CLIP's ability to recover the shared VFM region semantics. Student routers then learn to predict the corresponding assignments from CLIP features through routing distillation. Cross-view assignment consistency and routing exchange further encourage this channel support to remain useful when the same region changes appearance.
Thus, what is distilled is a VFM-guided region-to-channel assignment rule that promotes reusable internal support within CLIP, rather than the VFM's channel indices, weights, or circuit topology. The term circuit-level describes the explicit supervision and modulation of the student's internal channel usage, as opposed to supervising only its output features. The VFM analysis motivates the desired reuse property, the learned routers realize that property using the student's own channels.