acceptodds
Under review as a conference paper at ICLR 2027

From Fish to Fishing: Teaching CLIP to Generalize through Circuit-Level Distillation

Abstract

CLIP aligns image content with text-defined categories, enabling dense image-text prediction. But generalizing its ability to unseen domains remains challenging. Feature-level distillation improves local semantics and spatial coherence, yet the resulting representations can still deteriorate under appearance shifts. We investigate this gap at both the representation and circuit levels, observing that the self-supervised vision foundation model (VFM) maintains more consistent dense features and greater overlap in prediction-relevant neurons than CLIP across the examined domain shifts. These observations motivate transferring not only region representations, but also the ability to reuse internal channel support across appearances. Based on that, we propose ReoNa, a novel framework that uses VFM-guided Region-to-channel distillation to teach CLIP to reuse the internal MLP channels supporting each semantic region across appearances. Extensive experiments across diverse open vocabulary domain generalized semantic segmentation (OVDG-SS) benchmarks demonstrate that ReoNa significantly improves CLIP dense representation despite domain shifts. Our code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-vocabulary segmentation, domain generalization, distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8sMtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks why feature-level distillation from a self-supervised vision foundation model into CLIP does not transfer the VFM's robustness to domain shift, and motivates its answer with three observations comparing EVA-CLIP and DINOv2-reg: UMAP plots of clean and shifted patch features, layer-wise linear CKA between ACDC adverse and reference domains, and the Jaccard overlap of the top-10% "prediction-relevant" neurons per layer, all of which favour DINOv2-reg. The proposed ReoNa fine-tunes EVA02-CLIP-B/16 on COCO. GT region masks pool frozen DINOv2-reg features from an image and a photometrically shifted view into a cross-view consensus; at blocks 4, 8 and 12, a teacher router fed with this consensus and a student router fed with pooled CLIP features each output a softmax over the 2048 MLP hidden channels of that block, which Eq. 6 turns into a mean-one channel gate. Training combines KL distillation from the teacher to the student routing with a JS consistency between views, a swap loss that exchanges the gates of the two views and aligns the resulting class predictions, and a masked reconstruction of the VFM consensus under the teacher gates. At inference, a first unrouted pass builds soft regions from text-similarity maps and a second, gated pass produces the dense features. On the OVDG-SS benchmarks, ReoNa reaches 44.0 and 39.4 average zero-shot mIoU on Dv-19 and Dv-58, against 32.6 and 32.7 for DeCLIP, and improves by 0.7 to 2.2 mIoU over the best baseline when combined with the S²-Corr decoder.

Critical strengths

The empirical gains are large and consistent across settings. Making CLIP's dense features robust to domain shift is a relevant problem for open-vocabulary perception, and ReoNa improves zero-shot mIoU substantially and remains ahead with fine-tuned decoders, including CAT-Seg, which is not designed for domain generalisation. The appendix also shows that SAM masks reach 43.6 and 38.9 mIoU against 44.0 and 39.4 with GT regions, so the method does not depend on COCO annotations. Looking at which internal channels a model relies on across domains is an interesting angle. Asking what feature-level distillation fails to transfer, beyond the output features themselves, is a fresh way to frame robustness distillation.

Critical weaknesses

1. The central claim that teaching CLIP which channels to use is better than aligning features is supported neither by the motivating analysis nor by the ablations. The "neurons" of Observation 3 are, according to App. B.2, the 768 channels of each block's output, i.e. the residual stream, whereas the method routes and gates the 2048 hidden channels inside the MLP, and the teacher router is a learned map from VFM features to CLIP channels, so the VFM's own neuron structure is never transferred. The terminology also overstates what is measured: the importance score in Eq. 14–15 is the activation–gradient product of the squared norm of the spatially averaged final feature, which involves no prediction, the "final feature" is defined differently for the two models in App. B.1, and no random-selection baseline is given. On the experimental side, Table 4 starts from the published DeCLIP number and then adds a "routing" step that, according to its footnote, already includes the unmasked reconstruction loss, so it changes the data, the region pooling, the cross-view target, the feature reconstruction and the routing at once. The decisive control, the same pipeline with the same regions and cross-view consensus trained with region-level feature distillation only, without routers or gates, is missing.

2. The comparisons are not fair and omit the most relevant alternatives, so the size of the gains cannot be attributed to the method. ReoNa distills from DINOv2-reg, whereas its main baseline DeCLIP uses DINOv2-B by default, and the paper's own Table 6 shows that with DINOv2-B ReoNa drops from 44.0 and 39.4 to 38.5 and 36.1 mIoU. That means about half of its zero-shot advantage over DeCLIP may come from the teacher alone. The paper does not state whether the baseline rows were re-trained or copied, or whether their data, resolution and schedule match ReoNa's, and the zero-shot comparison gives ReoNa a two-pass, vocabulary-conditioned inference that recomputes most of the encoder, while all baselines use a single pass and the inference cost is not reported. Encoders that already combine language alignment with self-supervised objectives, such as SILC [1], TIPS [2] and SigLIP 2 [3], target the same weakness of CLIP's dense features but are neither discussed nor compared, so it is unclear whether the method still helps a stronger, current student.

[1] M. F. Naeem, Y. Xian, X. Zhai, L. Hoyer, L. Van Gool, F. Tombari. SILC: Improving Vision Language Pretraining with Self-Distillation. ECCV, 2024.

[2] K.-K. Maninis et al. TIPS: Text-Image Pretraining with Spatial Awareness. ICLR, 2025.

[3] M. Tschannen et al. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv:2502.14786, 2025.

Questions for authors
  1. Please report the routing-free control: the same COCO data, regions, photometric view pairs and cross-view consensus, trained with region-level feature distillation only, without routers or gates. How much of the gain in Table 4 remains attributable to routing?

  2. How does DeCLIP perform when re-trained with DINOv2-reg and the same data, resolution and schedule as ReoNa? Please state for every row of Tables 1–3 whether it was re-trained or copied.

  3. What is the zero-shot performance of ReoNa with a single unrouted pass, and what are its inference latency and FLOPs compared with DeCLIP?

  4. Does ReoNa still improve a student that is already trained with self-supervised objectives, such as SigLIP 2, and how does it compare with SILC and TIPS on these benchmarks?

9 comments
AuthorsOpenReview ↗

(1/3) Response to 8sMt-W1

We thank the reviewer for their kind review and feedback on our work. Below, we provide point-to-point responses to each weakness raised by the reviewer.


W1. The central claim that teaching CLIP which channels to use is better than aligning features is supported neither by the motivating analysis nor by the ablations.

For Weakness 1, we organize our response into six parts: (1) the connection between block-output analysis and MLP-channel routing; (2) discuss what is distilled; (3) the importance-score definition and rationale; (4) the final-feature definitions; (5) the random selection baseline; and (6) the routing-free control.

(1) Relation between block-output dimension and MLP channels

The 768 block-output channels and the 2048 MLP hidden channels are connected through the MLP output projection. In Observation 3, we analyze the 768-dimensional block outputs because they expose the feature representations on which dense prediction is built. Analysis at this level characterizes cross-domain reuse as it is expressed in the features directly for dense prediction.

During training, we gate the 2048 hidden channels inside the MLP because they are the computational components that generate the MLP's update to these representations. The output projection maps their contributions back into the 768-dimensional residual stream, where they are added to the residual state. Gating before this projection directly controls how each MLP component contributes to the block's features. Thus, the analysis examines the resulting feature representations, while training intervenes in the internal computation that shapes them.


(2) VFM's own neuron structure is not transferred.

The SSL-pretrained VFM and CLIP are trained with fundamentally different objectives and learn representations in distinct feature spaces. Their channels have no predefined semantic correspondence, so directly copying the identity or connectivity of particular teacher neurons into CLIP is not a well-defined transfer operation. Our goal is to learn reusable internal support across appearance changes. ReoNa learns this support in CLIP's own MLP-channel space, guided by domain-shared semantic evidence from the VFM.

Specifically, the teacher router takes the VFM cross-view consensus as input and predicts QT=softmax⁡(RT(Tcons))Q^T=\operatorname{softmax}(\mathcal R_T(T^{\mathrm{cons}})), an assignment over CLIP's MLP channels, not an importance vector over VFM neurons. These assignments modulate the CLIP MLPs during consensus reconstruction. The reconstruction loss updates the teacher router according to how its assignments affect CLIP's ability to recover the shared VFM region semantics. Student routers then learn to predict the corresponding assignments from CLIP features through routing distillation. Cross-view assignment consistency and routing exchange further encourage this channel support to remain useful when the same region changes appearance.

Thus, what is distilled is a VFM-guided region-to-channel assignment rule that promotes reusable internal support within CLIP, rather than the VFM's channel indices, weights, or circuit topology. The term circuit-level describes the explicit supervision and modulation of the student's internal channel usage, as opposed to supervising only its output features. The VFM analysis motivates the desired reuse property, the learned routers realize that property using the student's own channels.

AuthorsOpenReview ↗

(2/3) Response to 8sMt-W1

(3) The importance score in Eq. 14-15 involves no predictor

In Appendix B.2, we use a class-agnostic objective to quantify the influence of an activation uu on feature energy e(x)e(x), measured by ∣u∂e(x)∂u∣\left|u\frac{\partial e(x)}{\partial u}\right|. This class-agnostic objective enables us to compare how consistently the two models reuse channels to support dense representations across domains, without training an additional probe.

(i) How to compute ∣u∂e∂u∣\left|u\frac{\partial e}{\partial u}\right|?

For a single image xx, we spatially average its final dense features and define their mean squared magnitude as the attribution target: zˉ(x)=1N∑pzp(x),e(x)=1D∥zˉ(x)∥22,\bar z(x)=\frac{1}{N}\sum_p z_p(x), \qquad e(x)=\frac{1}{D}\|\bar z(x)\|_2^2, where NN is the number of spatial tokens and DD is the feature dimension. The scalar e(x)e(x) measures the channel-normalized squared magnitude of the globally pooled dense representation, without an additional per-token ℓ2\ell_2 normalization before computing the objective. Each encoder's readout is held fixed across domains, and channel-set overlap is computed within the same encoder, rather than by matching feature coordinates or absolute energies between the two models.

The absolute activation-gradient attribution is ∣u∂e(x)∂u∣=2D∣u zˉ(x)⊤∂zˉ(x)∂u∣.\left|u\frac{\partial e(x)}{\partial u}\right| =\frac{2}{D}\left|u\,\bar z(x)^\top\frac{\partial\bar z(x)}{\partial u}\right|.

(ii) Why use ∣u∂e∂u∣\left|u\frac{\partial e}{\partial u}\right| to measure the effect of a relative change?

We use this product because it measures the magnitude of the first-order energy change per unit fractional perturbation of an activation. Let u=Up,cl(x)u=U_{p,c}^l(x) denote the feature response in channel cc of patch token pp at the output of Transformer block ll for image xx. The gradient ∂e(x)/∂u\partial e(x)/\partial u measures the energy sensitivity to an absolute change in that response.

To quantitatively evaluate the effect of uu on feature energy e(x)e(x), consider a small relative attenuation u′=(1−ϵ)uu'=(1-\epsilon)u produces Δu=−ϵu\Delta u=-\epsilon u. A first-order Taylor approximation therefore gives Δe(x)≈∂e(x)∂uΔu=−ϵu∂e(x)∂u.\Delta e(x)\approx\frac{\partial e(x)}{\partial u}\Delta u =-\epsilon u\frac{\partial e(x)}{\partial u}. Hence, ∣Δe(x)∣≈ϵ∣u∂e(x)∂u∣|\Delta e(x)|\approx\epsilon\left|u\frac{\partial e(x)}{\partial u}\right|: for the same small attenuation fraction, a larger attribution predicts a larger local energy change. The activation supplies the perturbation scale, while the gradient supplies the energy sensitivity, which explains why their product measures influence under relative perturbations. We average these absolute contributions over spatial positions and sampled images to rank channels. The overlap of high-scoring channels then probes whether the same output-supporting components remain influential across domains.


(4)"Final feature" is defined differently

In Appendix B.1--B.2, the final feature zp(x)z_p(x) is the dense visual representation at spatial position pp used to define the attribution objective. For EVA-CLIP, we use the dense encoder output. For DINOv2-reg, we use the spatial patch tokens after the final Transformer block and before the final LayerNorm, excluding the class and register tokens. These are encoder-specific dense readouts for dense prediction tasks like segmentation.

AuthorsOpenReview ↗

(3/3) Response to 8sMt-W1

(5) Random selection baseline

Thanks for your advice on comparing with randomly selected channels. To test the semantic relevance of the selected channels, we have added a random-channel attenuation control evaluated through segmentation.

For DINOv2-reg and EVA-CLIP, we freeze the image encoder and train a linear segmentation probe on ACDC training data, then evaluate on ACDC night. Following Appendix B.2, we select the top-10% block-output channels per layer using only the associated normal-condition reference images, and fix their indices for night-time evaluation. We compare attenuating these reference-selected channels with attenuating an equal number of random channels, using the same per-layer channel counts, intervention locations, and attenuation factor. The encoder weights and trained probe remain fixed during intervention evaluation.

Intervention on ACDC nightDINOv2-reg mIoUDINOv2-reg Δ\Delta mIoUEVA-CLIP mIoUEVA-CLIP Δ\Delta mIoU
No attenuation40.5NA18.4NA
Attenuate reference-selected top-10% channels24.7-15.817.4-1.0
Attenuate an equal number of random channels39.8-0.718.2-0.2

For DINOv2-reg, attenuating the selected channels reduces performance by 15.8 mIoU, compared with only 0.7 mIoU for random channels. For EVA-CLIP, the corresponding drops are 1.0 and 0.2 mIoU. Since channel selection uses reference images while semantic evaluation uses night images with those indices held fixed, the results show that energy-selected neurons support semantic prediction across views, particularly for DINOv2-reg. Together with its higher shared-neuron similarity in Observation 3, the much stronger selected-versus-random contrast supports the interpretation that DINOv2-reg's circuit sharing reflects semantically useful, reusable internal support. EVA-CLIP exhibits a substantially weaker contrast under the same intervention protocol.


(6) Routing-free control (Corresponds to Q1)

Thank you for suggesting this control. We agree that the original cumulative ablation in Table 4 does not separately isolate the benefit of the cross-view consensus and reconstruction objectives from that of routing. We have therefore added two routing-free controls that retain the same COCO data, region masks, photometric view pairs, and VFM cross-view consensus, but remove the region-to-channel routers and channel-modulation gates. The first aligns region features to the consensus using feature-level supervision only. The second additionally reconstructs the same consensus from masked inputs, still without any routing or channel modulation. This progressively tests whether the gains can be explained by the augmented-view consensus and masked reconstruction alone. The DeCLIP reference uses its officially released weights.

MethodCS-19ACDC-19BDD-19Mapi-19Avg. (Dv-19)ACDC-41BDD-41Mapi-30RW-10Avg. (Dv-58)
Feature alignment (DeCLIP)32.530.030.937.132.632.439.633.125.732.7
Feature alignment to cross-view consensus33.230.031.036.732.733.039.433.625.532.9
Feature alignment + masked consensus reconstruction34.030.331.036.933.134.741.833.825.233.9
ReoNa44.240.743.847.244.042.448.837.528.839.4

Both feature-level additions provide incremental benefits. Using the cross-view consensus for feature-level alignment increases average performance from 32.6 to 32.7 mIoU on Dv-19 and from 32.7 to 32.9 mIoU on Dv-58. Adding masked reconstruction of that consensus further improves performance to 33.1 and 33.9 mIoU, respectively, contributing another 0.4 and 1.0 mIoU. Thus, incorporating appearance-augmented teacher evidence and learning to reconstruct it from masked inputs are beneficial without channel modulation. Without routing, these two additions retain gains of only 0.5 mIoU on Dv-19 and 1.2 mIoU on Dv-58 over DeCLIP, compared with the full ReoNa gains of 11.4 and 6.7 mIoU. These results show that richer feature-level targets and masked reconstruction alone do not account for most of the improvement. The substantial remaining advantage supports explicitly learning and applying region-conditioned channel support across appearances, rather than only aligning the resulting region features.

We have updated Table 4 and corresponding content in Section 4 in our revised PDF to include this control.

Reviewer_FYdDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReoNa to transfer a self-supervised VFM’s cross-domain robustness to CLIP. Motivated by differences in representation stability and important-channel overlap, it uses cross-view regional consensus to guide routers that modulate CLIP’s MLP channels. Routing consistency, routing exchange, and masked reconstruction encourage stable channel use across appearance changes. The method improves performance on open-vocabulary cross-domain semantic segmentation benchmarks.

Critical strengths
  1. The paper identifies a useful direction beyond feature alignment. Its analysis of cross-domain representations and channel overlap motivates transferring more stable internal channel use from a VFM to CLIP.

  2. The method provides a straightforward way to guide CLIP’s internal computation with teacher knowledge. Region-specific routers directly reweight MLP activations, allowing different regions to use different channel weights while encouraging consistency across appearances.

  3. Performance gains extend across several evaluation settings. Improvements are particularly strong without an additional segmentation decoder and also hold with S²-Corr and CAT-Seg. Alternative region sources perform close to ground-truth regions, and inference does not require the teacher.

Critical weaknesses
  1. The neuron analysis does not directly establish shared support for semantic predictions. Observation 3 makes this interpretation, but Appendix B.2, Eq. (14), computes importance using the squared norm of spatially averaged features. Higher overlap under this objective does not by itself show that the same neurons support the same semantic predictions across domains.

  2. The contribution of routing is not isolated in Table 4. The routing variant already includes unmasked consensus reconstruction, so the reported gain cannot be attributed to routing alone. A baseline that reconstructs the same consensus without channel modulation, under matched training conditions, would clarify its added value.

  3. The benefit of averaging the two teacher views remains unclear. A shared target is a reasonable choice, but the VFM is already presented as relatively stable across appearances. Its original-view regional feature could also serve as the target for both views; this simpler alternative is not evaluated.

Questions for authors
  1. What evidence links the feature-norm attribution in Appendix B.2 to the semantic predictions discussed in Observation 3?

  2. How does ReoNa compare with reconstructing the same consensus without channel modulation, under matched training conditions?

  3. Does cross-view averaging outperform using the original-view VFM feature as the shared target for both views?

3 comments
AuthorsOpenReview ↗

(1/2) Response to FYdD-W1(Q1)

We thank the reviewer for their kind review and feedback on our work. Below, we provide point-to-point responses to each weakness raised by the reviewer.


W1(Q1). What evidence links the feature-norm attribution in Appendix B.2 to the semantic predictions discussed in Observation 3?

Thank you for raising this question. In Appendix B.2, we use a class-agnostic objective to quantify the influence of an activation uu on feature energy e(x)e(x), measured by ∣u∂e(x)∂u∣\left|u\frac{\partial e(x)}{\partial u}\right|.

(1) How to compute ∣u∂e∂u∣\left|u\frac{\partial e}{\partial u}\right|?

For clarity, consider a single image xx. We spatially average its final dense features and define their mean squared magnitude as the attribution target: zˉ(x)=1N∑pzp(x),e(x)=1D∥zˉ(x)∥22.\bar z(x)=\frac{1}{N}\sum_p z_p(x), \qquad e(x)=\frac{1}{D}\|\bar z(x)\|_2^2.

Here, zp(x)z_p(x) is the final dense feature at spatial position pp, NN is the number of spatial tokens, and DD is the feature dimension. The scalar e(x)e(x) measures the channel-normalized squared magnitude of the globally pooled visual representation. The absolute activation-gradient attribution is ∣u∂e(x)∂u∣=2D∣u zˉ(x)⊤∂zˉ(x)∂u∣.\left|u\frac{\partial e(x)}{\partial u}\right| =\frac{2}{D}\left|u\,\bar z(x)^\top\frac{\partial\bar z(x)}{\partial u}\right|.


(2) Why use ∣u∂e∂u∣\left|u\frac{\partial e}{\partial u}\right| to measure the effect of a relative change?

We use this product because it measures the magnitude of the first-order energy change per unit fractional perturbation of an activation. Let u=Up,cl(x)u=U_{p,c}^l(x) denote the feature response in channel cc of patch token pp at the output of Transformer block ll for image xx. The gradient ∂e(x)/∂u\partial e(x)/\partial u measures the energy sensitivity to an absolute change in this feature value.

To quantitively evaluate the effect of uu on feature energy e(x)e(x), consider a small relative attenuation u′=(1−ϵ)uu'=(1-\epsilon)u instead produces Δu=−ϵu\Delta u=-\epsilon u, so a first-order Taylor approximation gives Δe(x)≈∂e(x)∂uΔu=−ϵu∂e(x)∂u.\Delta e(x)\approx\frac{\partial e(x)}{\partial u}\Delta u =-\epsilon u\frac{\partial e(x)}{\partial u}. Therefore, ∣Δe(x)∣≈ϵ∣u∂e(x)∂u∣|\Delta e(x)|\approx\epsilon\left|u\frac{\partial e(x)}{\partial u}\right|. For the same small attenuation fraction ϵ\epsilon, a larger absolute activation-gradient product predicts a larger local change in the output-representation energy. The activation supplies the perturbation scale, while the gradient supplies the energy sensitivity, which is why we use their product to measure influence under relative perturbations. Averaging these absolute contributions over spatial positions and sampled images gives the domain-level channel rankings. Their cross-domain overlap probes whether the same output-supporting channels remain influential under appearance shifts.

AuthorsOpenReview ↗

(2/2) Response to FYdD-W1(Q1)

(3) Further evidence to support the effectiveness of our neuron selection strategy

We agree that this objective does not directly reflect semantic predictions: it contains neither class logits nor class-conditioned targets. It is a class-agnostic probe of internal representational support, applicable to both CLIP and the SSL-pretrained VFM without introducing an additional classifier or text-prediction head. Its connection to semantic circuits is indirect: the attribution traces internal components that influence the energy of the dense representations used for semantic prediction. We therefore supplement the overlap analysis with a direct functional test.

For DINOv2-reg and EVA-CLIP, we freeze the image encoder and train a linear segmentation probe on ACDC training data, then evaluate on ACDC night. Following Appendix B.2, we select the top-10% block-output channels per layer using only the associated normal-condition reference images, and hold their indices fixed for night-time evaluation. We compare attenuating these reference-selected channels with attenuating an equal number of random channels, using the same per-layer channel counts, intervention locations, and attenuation factor. The encoder weights and trained probe remain fixed during intervention evaluation. The semantic readout is segmentation mIoU, rather than feature-energy change.

The results are shown below.

Intervention on ACDC nightDINOv2-reg mIoUDINOv2-reg Δ\Delta mIoUEVA-CLIP mIoUEVA-CLIP Δ\Delta mIoU
No attenuation40.5NA18.4NA
Attenuate reference-selected top-10% channels24.7-15.817.4-1.0
Attenuate an equal number of random channels39.8-0.718.2-0.2

For DINOv2-reg, attenuating the selected channels causes a 15.8 mIoU drop, compared with only 0.7 mIoU for random channels. For EVA-CLIP, the corresponding drops are 1.0 and 0.2 mIoU. Because selection uses reference images and the indices are held fixed on night images, these results provide functional evidence that the neurons identified by energy-based attribution support semantic prediction across views, particularly for DINOv2-reg. Together with its higher shared-neuron similarity in Observation 3, this supports the interpretation that DINOv2-reg's circuit sharing reflects reusable internal support with a semantic function. The much smaller selected-versus-random contrast in EVA-CLIP is consistent with weaker cross-view reuse of this support.

We have updated this experiment in Appendix B.3 of our revised pdf.

AuthorsOpenReview ↗

(1/1) Response to FYdD-W2&W3 (Q2 and Q3)

W2(Q2). How does ReoNa compare with reconstructing the same consensus without channel modulation, under matched training conditions?

Thank you for suggesting this control. We have added an experiment that removes the region-to-channel routers and channel-modulation gates, and trains the CLIP image encoder using only reconstruction of the same cross-view consensus and the feature correlation alignment objective proposed by DeCLIP. The comparison is shown below, the DeCLIP reference uses its officially released weights.

MethodCS-19ACDC-19BDD-19Mapi-19Avg. (Dv-19)ACDC-41BDD-41Mapi-30RW-10Avg. (Dv-58)
Feature alignment (DeCLIP)32.530.030.937.132.632.439.633.125.732.7
Feature alignment + consensus reconstruction, without channel modulation34.030.331.036.933.134.741.833.825.233.9
ReoNa44.240.743.847.244.042.448.837.528.839.4

The routing-free setting improves the performance over feature alignment alone. However, these gains remain substantially smaller than those of ReoNa. Thus, adding reconstruction of the cross-view consensus to feature correlation alignment does not account for most of ReoNa's improvement. We have updated Table 4 and the corresponding content of Section 4 of the revised PDF to include this control.


W3(Q3). Does cross-view averaging outperform using the original-view VFM feature as the shared target for both views?

Thank you for suggesting this control. We have added an experiment that uses only the original-view VFM region feature as the shared teacher target for both student views, instead of the cross-view consensus. The appearance-augmented student view and the other training settings are retained. As shown below, this replacement reduces average performance from 44.0 to 41.4 mIoU on Dv-19 and from 39.4 to 37.6 mIoU on Dv-58, corresponding to decreases of 2.6 and 1.8 mIoU, respectively.

Shared teacher targetCS-19ACDC-19BDD-19Mapi-19Avg. (Dv-19)ACDC-41BDD-41Mapi-30RW-10Avg. (Dv-58)
Original-view only41.537.441.645.141.440.447.135.227.737.6
Cross-view consensus (ReoNa)44.240.743.847.244.042.448.837.528.839.4

Our interpretation is that appearance augmentation is beneficial for domain generalization because it exposes the model to the same semantic regions under diverse appearance characteristics. Although the VFM is comparatively robust to appearance shifts, its region features are not identical across views. Incorporating the augmented-view VFM feature into the shared target allows the supervision itself to reflect this appearance diversity. Averaging the two teacher representations can therefore emphasize semantic evidence shared across appearances. This provides a more suitable target for learning region representations and channel support that remain useful when appearance changes. We have updated this control in Appendix C.5 of our revised PDF.


We sincerely thank you for your time and effort in reviewing our paper and providing valuable feedback. We hope that our detailed responses and additional experiments can address your concerns. If any questions or concerns remain, we would greatly appreciate your feedback and would be happy to provide further clarification. Thank you again for your constructive comments and consideration.

Best regards, The Authors of Paper ID 39248

Reviewer_9bAVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ReoNa, a region-to-channel distillation framework to improve CLIP's generalization to unseen domains. The authors observe that self-supervised vision foundation models, such as DINOv2, have more stable dense features and share more important neurons across different domains than CLIP. Based on this observation, ReoNa aims to help CLIP reuse similar internal MLP channels for the same semantic regions under different appearances. Specifically, the method uses cross-view region features from a frozen vision foundation model to guide channel routing in CLIP. It also introduces cross-view routing exchange and masked reconstruction to encourage stable channel usage and consistent region features. Experiments on several open-vocabulary domain-generalized semantic segmentation benchmarks show that ReoNa improves CLIP's performance under domain shifts.

Critical strengths
  1. The paper provides an interesting motivation by analyzing the differences in feature and neuron consistency between CLIP and self-supervised vision models across domains.
  2. The proposed region-to-channel distillation framework offers a new way to improve CLIP's generalization by learning stable channel usage across domains.
  3. Extensive experiments on multiple benchmarks show consistent improvements over existing methods.
Critical weaknesses
  1. Insufficient evidence for semantic circuit consistency. Figure 4 aims to show that DINOv2 reuses more prediction-relevant neurons across domains than CLIP. However, Appendix B.2 uses the squared norm of globally averaged dense features for attribution, which may not directly reflect semantic predictions. Thus, higher shared-neuron similarity may not necessarily indicate more stable semantic circuits. It would be helpful to provide further evidence to support this claim.

  2. Unclear update mechanism for the Teacher Router. My understanding is that RTR_T is updated through the reconstruction loss in Eq. (8). However, in lines 292–293, the authors state: “Simultaneously, we update RTl\mathcal{R}_T^l using the JS divergence between FiF_i and F^i\hat{F}_i.” Since both features are extracted from the frozen DINOv2, I am not sure how this loss can update RTlR_T^l. Could the authors kindly clarify this point?

  3. Potential mismatch between routing decisions and routed features. ReoNa predicts routing weights from unrouted features in the first forward pass and applies them in the second pass. However, routing in earlier layers may change the features received by later layers. As a result, the predicted weights may not fully match the features they are applied to. It would be interesting to discuss whether this mismatch affects performance, especially across multiple routed layers.

  4. Clarity and presentation. Figure 5 uses similar grids to represent Dense Features, Region Consensus, and Routing Matrices, although they have different shapes and meanings. This may be a little confusing. Also, the many superscripts and subscripts make some parts of the method harder to follow. It would be helpful to clarify the tensor shapes in Figure 5 and simplify the notation where possible.

Overall, I appreciate the authors' work and would be happy to raise my score if these concerns are addressed.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

(1/2) Response to 9bAV-W1

We thank the reviewer for their kind review and feedback on our work. Below, we provide point-to-point responses to each weaknesses raised by the reviewer.

W1. Insufficient evidence for semantic circuit consistency.

Thank you for raising this point. In Appendix B.2, we use a class-agnostic objective to quantify the influence of an activation uu on feature energy e(x)e(x), measured by ∣u∂e(x)∂u∣\left|u\frac{\partial e(x)}{\partial u}\right|. We detail the analysis below:


(1) How to compute ∣u∂e∂u∣\left|u\frac{\partial e}{\partial u}\right|?

For clarity, we take one image xx for example. We first spatially average the final dense features of xx and define their mean squared magnitude as the attribution target e(x)e(x), given by:

\qquad e(x)=\frac{1}{D}\|\bar z(x)\|_2^2.$$ Here, $z_p(x)$ is the dense feature at position $p$, $N$ is the number of spatial tokens, and $D$ is the feature dimension. The resulting $e(x)$ is a scalar measuring the channel-normalized squared magnitude of the globally pooled visual representation. For an intermediate activation $u$, its absolute activation-gradient attribution is: $$\left|u\frac{\partial e}{\partial u}\right|=\frac{2}{D}\left|u\,\bar z^\top\frac{\partial\bar z}{\partial u}\right|.$$ --------- **(2) Why use $\left|u\frac{\partial e}{\partial u}\right|$ to measure the effect of a relative change?** We use $\left|u\frac{\partial e}{\partial u}\right|$ because it measures the magnitude of the first-order energy change per unit fractional perturbation of an activation. Let $u=U_{p,c}^l(x)$ denote the intermediate feature response in channel $c$ of patch token $p$ at the output of Transformer block $l$ for image $x$. The gradient $\partial e(x)/\partial u$ measures the sensitivity of the globally pooled dense-feature energy to an absolute change in this feature value. To quantitatively evaluate the effect of $u$ on feature energy $e(x)$, we consider a relative attenuation $u'=(1-\epsilon)u$ that changes the activation by $\Delta u=-\epsilon u$. Substituting this change into a first-order Taylor approximation gives $$\Delta e(x) \approx\frac{\partial e(x)}{\partial u}\Delta u =-\epsilon u\frac{\partial e(x)}{\partial u}.$$ Therefore, $|\Delta e(x)|\approx\epsilon\left|u\frac{\partial e(x)}{\partial u}\right|$. For the same small attenuation fraction $\epsilon$, a larger $\left|u\frac{\partial e(x)}{\partial u}\right|$ predicts a larger local change in the output-representation energy. **This is why we use the absolute activation-gradient product to measure an activation's influence under relative perturbations:** the activation supplies the perturbation scale, while the gradient supplies the energy sensitivity. We aggregate these magnitudes over spatial positions to rank channels, and compare the high-scoring channel sets across domains. The cross-domain overlap of high-scoring channels probes whether the same output-supporting components remain influential under appearance shifts.
AuthorsOpenReview ↗

(2/2) Response to 9bAV-W1

(3) Further evidence to support the effectiveness of our neuron selection strategy

We agree that this objective does not directly reflect semantic predictions (neither class logits nor class-conditioned targets). It's a class-agnostic probe of internal representational support, applied to both CLIP and the SSL-pretrained VFM without introducing the additional classifier or text-prediction head. Its connection to semantic circuits is indirect but meaningful, since the attribution traces internal components that influence the energy of those representations.

To directly test whether the neurons identified by energy-based attribution remain important for semantic prediction across views, we conduct a channel-attenuation experiment on ACDC night. For DINOv2-reg and EVA-CLIP, we freeze the image encoder and train a linear probing on ACDC training data and test it on ACDC night. Following Appendix B.2, we select the top-10% block-output channels per layer using only the normal-condition reference images, and fix these indices for evaluation on ACDC night. We then compare (i) attenuating these selected channels with (ii) attenuating an equal number of random channels, both using the same per-layer counts, locations, and attenuation factor:

The results are shown below.

Intervention on ACDC nightDINOv2-reg mIoUDINOv2-reg Δ\Delta mIoUEVA-CLIP mIoUEVA-CLIP Δ\Delta mIoU
No attenuation40.5NA18.4NA
Attenuate reference-selected top-10% channels24.7-15.817.4-1.0
Attenuate an equal number of random channels39.8-0.718.2-0.2

For DINOv2-reg, attenuating the selected channels reduces mIoU by 15.8, compared with only 0.7 for random channels. For EVA-CLIP, the corresponding drops are only 1.0 and 0.2 mIoU. Since the channels are selected on reference images and held fixed on evaluated images. Such a result shows that the the neurons identified by energy-based attribution support semantic prediction across views, particularly for DINOv2-reg. Together with DINOv2-reg's higher shared-neuron similarity in Observation 3, this supports the interpretation that its circuit sharing reflects reusable internal support with a semantic function, whereas the much weaker contrast in EVA-CLIP is consistent with weaker cross-view reuse.

We have clarified the descriptions in Appendix B.2 and updated this experiment in Appendix B.3 of our revised pdf.

AuthorsOpenReview ↗

(1/1) Response to 9bAV-W2&W3

W2. Unclear update mechanism for the Teacher Router.

Thank you for pointing out this imprecise description. The JS divergence is computed between the teacher router's clean-view and shifted-view routing distributions, QTQ^T and Q^T\hat Q^T for each router, rather than directly between the frozen VFM features FF and F^\hat F. Although VFM features are frozen, their routing distributions depend on the trainable parameters of RT\mathcal R_T. Gradients from this JS term therefore propagate through both teacher-router outputs and update RTl\mathbf{R}_T^{l}. We have corrected the wording in lines 293–294.

Specifically, within each selected block, the teacher router RT\mathcal R_T processes the region features from the two views and produces channel-assignment distributions via a row-wise softmax. We encourage their cross-view consistency through JS divergence between QTQ^T and Q^T\hat Q^T, which encourages the teacher router to assign consistent CLIP channel support to the same semantic region across appearance changes.


W3. Potential mismatch between routing decisions and routed features.

Thank you for raising this point. We agree that this mismatch can occur: routing at an earlier block changes the features received by later blocks, whereas their gates were predicted from the fully unrouted pass. To assess its practical impact, we compare the original two-stage inference with sequential gate refresh, using the same trained checkpoint without retraining.

Specifically, we refresh the gates in depth order, G4→G8→G12G_4 \rightarrow G_8 \rightarrow G_{12}. First, an initial pass to the the router at block 4 to predicts G4G_4. We then fix this router and apply G4G_4 to recompute the subsequent features up to block 8. The updated block-8 patch features are passed to router at block 8 to obtain a refreshed G8G_8. After applying the refreshed G8G_8, we similarly refresh G12G_{12} using features affected by both preceding gates, and apply it to obtain the final dense features. In this way, each later gate is recomputed from features that already incorporate the routing decisions at all preceding selected blocks. The results are shown below:

Inference methodCS-19ACDC-19BDD-19Mapi-19Avg. (Dv-19)ACDC-41BDD-41Mapi-30RW-10Avg. (Dv-58)Inference latencyFLOPs
Original two-stage inference44.240.743.847.244.042.448.837.528.839.418.0 ms/image85.4 GFLOPs
Sequential gate refresh43.441.644.845.743.943.050.637.227.539.618.9 ms/image88.4 GFLOPs

Sequential gate refresh changes the average performance from 44.0 to 43.9 mIoU on Dv-19 and from 39.4 to 39.6 mIoU on Dv-58, with mixed changes across individual datasets. These results indicate that refreshing later gates has only a small effect on benchmark averages and does not yield a consistent improvement, while increasing inference latency from 18.0 to 18.9 ms/image. Thus, although the potential mismatch exists, its impact on average performance appears limited under the evaluated conditions.

One possible explanation is that the two-stage inference follows the same forward paradigm as the training stage. During training, the student routers predict assignments from unrouted first-pass region features, and these assignments are subsequently used in routed feature extraction. At inference, the first pass identifies image regions and predicts their region-to-channel assignments, the second pass holds these assignments fixed while extracting dense features. The encoder and routers are therefore optimized for this two-stage pattern rather than for assignments that are refreshed after every routed block. This correspondence may help explain why refreshing gates only at evaluation provides little additional benefit.

The analysis of mismatch between the feature in the first forward and the modulated feature is provided in Appendix C.6

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary from Authors

Dear Area Chairs,

We sincerely thank you and all reviewers for the time and effort devoted to the review process, and for reviewer's valuable and constructive feedback. Overall, the reviewers have expressed positive assessments of our proposed method, as well as its motivations and contributions.

The main concern shared by three reviewers is whether the shared-neuron similarity can reflect stable semantic circuits across domains. We address this concern by clarifying the neuron-selection criterion and adding functional channel-attenuation experiments. The results connect the higher neuron overlap in DINOv2-reg and ReoNa to stronger semantic reliance on the same channels across appearance changes. Together with the fixed-channel visualizations, this evidence supports the functional relevance of shared-neuron similarity and strengthens the empirical grounding and rigor of our central argument. And we are pleased that majority of reviewers (Reviewer 9bAV and Reviewer FYdD) have regarded this main concern as fully resolved.

In addition, following the reviewers' suggestions, we have added more comprehensive ablation studies covering cross-view consensus, reconstruction without routing, and inference strategies. These additional experiments clarify the contributions of our design choices and provide a more complete empirical evaluation of ReoNa.

Regarding Reviewer 9bAV and Reviewer FYdD, we are pleased that our response have addressed their concerns. Specifically, reviewer 9bAV raised the score from weak reject (2) to weak accept (3), and reviewer FYdD keeps the score at weak accept (3). We sincerely appreciate their recognitions on our work and efforts during the rebuttal and discussion period.

Regarding Reviewer 8sMt, we appreciate that our rebuttal has resolved part of concerns and Reviewer 8sMt has increased rating from weak reject (2) to borderline. We further provide detailed clarifications and additional experimental reuslts to address their remaining concerns about the central contribution, shared-neuron analysis, and inference procedure. Although we politely follow up to see whether our new response can address the remaining concerns, we have not received further feedback from Reviewer 8sMt. As the rebuttal deadline is fast approaching, We respectfully hope that the ACs will take these additional results and clarifications into consideration in their final assessment, and we believe that our additional experiments and explanations have adequately addressed the remaining concerns.

Meanwhile, we have updated our manuscript following reviewer's suggestions (Tab. 4 in Sec. 4; Appendix Sec. B.3; Appendix Sec. C5 and Sec. C6). Finally, we sincerely thank the ACs and all reviewers once again for their time, effort, and valuable feedback throughout the review process.

Regards,

Authors of submission 39248

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.