acceptodds
Under review as a conference paper at ICLR 2027

On the Role of Mechanistic Interpretability for Vision-Language Prompt Learning

Abstract

While mechanistic interpretability tools such as sparse autoencoders (SAEs) have been proposed for extracting monosemantic, human-understandable features from vision-language models (VLMs) such as CLIP, existing work focuses on using these tools for post-hoc diagnostic analysis. We posit that SAE-based interpretability methods are more than passive probing tools, but can actively guide adaptation of VLMs to downstream tasks. We propose IPL (Interpretability-Guided Prompt Learning), a framework which leverages SAE decoders to extract interpretable concept directions, composes them into prompt tokens via a learnable attention selector, and injects the resulting tokens into both the vision and text encoder layers of CLIP for adaptation. A projection-preservation regularizer keeps prompt-token geometry aligned with the underlying concept directions. Across 15 benchmark datasets and experimental settings covering base-to-novel generalization, domain generalization, cross-dataset transfer, and few-shot learning, IPL consistently outperforms prior prompt-learning methods, with unified (multimodal) concept directions achieving the strongest results, compared to vision-only and text-only concept directions.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
prompt learning, adaptation, generalization, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_B2mWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Authors extract SAE directions as basis for CLIP prompt tokens. They initialise learnable queries that then attend over this basis and the resulting tokens are injected into vision and text layers. They test it on base-to-novel, domain generalization, cross-dataset transfer, and few-shot learning tasks using unified VL-SAE concepts.

Critical strengths

Authors perform quite a lot of experiments which is good.

The paper is well written.

Critical weaknesses

I am wondering if SAEs used to build E seen evaluation images and/or class names? This would be useful to check to find potential leakage.

The method is totally dependable by SAE quality. The datasets used by the authors are quite standard and most concepts will be recognised by SAEs, however it is not clear whether the SAE concepts would be good initialisation when SAEs are weak or a dataset is truly outside of SAEs concepts. I think EuroSAE is quite representative of this where the method doesn’t not perform well (Table 6). Additional experiments on OOD data such as medical or other remote-sensing datasets or with deliberately weaker SAEs (smaller dictionaries, less training data) would show whether the method is robust or only works where the SAE is already strong.

Can authors clarify how parameters k, N and and layers have been chosen? What split was used etc?

Authors claim that prompt tokens are interpretable because as they come from SAEs. But the attention and projection layers can mix and change those directions. Could authors provide a quantitative test showing the learned tokens are still interpretable? As for example readable prompts in CPL or IPO it is clear that they are interpretable.

Questions for authors

See weaknesses

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We sincerely thank reviewers for carefully reading our paper and raising meaningful question and concerns

W1:

The SAE’s are trained without class labels and VL-SAE specifically is trained on CLIP embeddings of CC3M image-caption pairs. The dictionary E is selected using base-class training data only, and prompts are trained on base classes only, novel images and names are used only during evaluation. Hence, IPL’s protocol ensures that there is no explicit data leakage. Also, it's important to note that any semantic overlap between web data and downstream categories might be present through CLIP’s own pretraining on web image-text pairs, which all the baseline use and that’s why the zero shot performance of CLIP is already 74.22 on novel classes. Many prompt learning methods including IPL does not aim to learn from data that the pretrained model has never seen, rather it adds context in the form of concept-composed prompt tokens that adapts an already pretrained model to downstream tasks which includes novelty and distribution shifts. The standard benchmarks therefore measure generalization beyond the data used for prompt learning, which is the setting in which IPL and all baselines are evaluated.

W2:

Following the reviewer’s suggestion, we evaluate on RESISC45, an aerial remote sensing dataset outside the standard benchmarks. We report the results in the table below and all methods use the same splits and seeds and other configurations.

Table 5

MethodBaseNovelHM
Zero-shot CLIP69.3463.3166.19
MMRL94.67 ± 0.4773.57 ± 6.0182.80
IPL94.83 ± 0.4976.40 ± 1.3984.62

We observe that IPL improves over MMRL by +2.83 on novel classes and +1.82 on HM with less variance in accuracy across seeds on novel classes. The SAE concepts therefore remain a useful basis on a domain outside the standard benchmarks. IPL's effectiveness does depend on the interpretability tool, and we explicitly acknowledge this as a limitation in Sec. 5. Since the main goal of our work is to investigate a conceptual feasibility question of whether interpretability tools can act as training guides during adaptation, this dependence is therefore inherent to the question. Hence, IPL is plug-and-play with respect to the interpretability tool and any new tool that yields concept directions and concept strengths can be used without changes to the method. We therefore expect IPL to benefit directly as these tools improve.

W3:

The concept dictionary size K=256, the number of prompt tokens N=5, the injection layers 6-12 and alpha=0.7 form a single setting shared by all datasets and all four tasks; none of them is tuned per dataset. They were chosen from the ablations in Fig. 3b (layers, on ImageNet), Fig. 3c (K, averaged over the 11 datasets) and Fig. 6 (N, on ImageNet), using the described split in our experiments section. Training settings (epochs, learning rate, batch size, optimizer) follow MMRL.

These choices are not the optimum ones. We choose those because performance saturates at N=5 (Fig. 6), insertion at layers 3-12 performs similarly to 6-12 (Fig. 3b), and our choice depended on similar rationales for K and lambda ablations.

W4:

Firstly, we would like to clarify that IPL’s prompt tokens are not natural language strings as in CPL or IPO. We define the prompt tokens’ interpretability at the level of concept directions derived from the interpretability tools like SAE’s. Taking reviewers suggestions into consideration, we have proposed a variant of attention selector such that the prompt tokens constructed lie in the span of original concept directions (for more details please refer to our response to #Lage, W1/Q1). We also note that the results of the convex selector variant matches the accuracy of the earlier variant (82.15 - convex selector vs. 82.01 - naive MHA average HM over four datasets) indicating that the results are mostly driven by SAE’s original concept directions E irrespective of the transformation it undergoes via attention selector.

Thank you for the response. The weaker-SAE experiment is still missing. For hyperparameters, k seems to have been chosen on the evaluation datasets so please clarify whether a separate validation split was used. The convex selector helps, but tokens still pass through the projections and no quantitative interpretability test was given.

AuthorsOpenReview ↗

Clarification regarding weaker SAE, K and quantitative interpretability test

We sincerely thank the reviewer for responding to our rebuttal. Below we clarify the issues:

Weaker SAEs. Our results already include IPL being driven by weaker interpretability signals. Vision-only and text-only VL-SAE dictionaries capture a single modality of the unified concept space. With these weaker signals, IPL performance actually degrades and we can see this evidently in Table 1. In our response to #WgfD, Q1, table 3, we also performed an experiment where used bottom 256 and random 256 directions from the same SAE and observed that using top 256 activated directions yields better results. Retraining a deliberately weakened SAE was not feasible within the discussion period and is slightly out of scope of our central research question (which is to to verify the conceptual feasibility of whether already existing SAE based interpretability tools can be used as guide for adaptation).

Validation split for kk. No separate validation split was used. k=256k = 256 was selected from Fig. 3c , i.e., on base-to-novel test accuracy averaged over the 11 datasets (we have discussed this in our ablation section too), and the injection layers and NN were selected on ImageNet (Figs. 3b and 6), following common practice in prompt-learning ablations. Each is a single value shared by all datasets and all four tasks, and the domain-generalization target domains were never used in any selection.

Quantitative interpretability test. We report this in our response to Reviewer #WgfD (Q2), measured on the actual prompt tokens, including after the projections. With the convex selector, every token's residual outside the span of the concept directions is 0.000, and each token equals its selected concept mixture (cosine 1.00) and we report other quantitative metrics like geometric distortion and concept alignment too. We will also include all these supporting experiments in our main draft.

We hope these clarifications address remaining concerns. If so, we would be really grateful if the reviewer would consider updating score.

Reviewer_WgfDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Interpretability-Guided Prompt Learning (IPL), a framework that uses sparse autoencoder (SAE) features to guide the adaptation of pretrained vision-language models. The method selects task-relevant SAE dictionary directions based on their average activations, constructs learnable prompt tokens through multi-head attention over these directions, and injects the resulting tokens into multiple layers of CLIP's vision and text encoders. A projection-preservation regularizer is introduced to encourage preservation of the original concept geometry. The authors evaluate IPL on base-to-novel generalization, domain generalization, cross-dataset transfer, and few-shot classification. The unified vision-language SAE variant achieves the strongest overall performance. The paper also provides dictionary replacement controls, loss ablations, and qualitative analyses of selected concepts. The central contribution is the proposal to use representations discovered by mechanistic interpretability tools as an active source of information for downstream adaptation, rather than solely for post-hoc analysis.

Critical strengths
  1. A relevant research question with a meaningful methodological connection. The paper explores whether SAE-based analysis of pretrained representations can actively improve downstream learning. This is a worthwhile direction because it investigates a potential practical benefit of mechanistic interpretability beyond post-hoc explanation. The proposed integration of SAE dictionaries with multimodal prompt learning is a concrete approach to this question.

  2. Broad evaluation and meaningful dictionary controls. The evaluation covers several established prompt-learning protocols, and the unified variant obtains competitive results against recent methods. Importantly, the authors go beyond comparisons against conventional prompt-learning baselines by replacing the SAE dictionary with Gaussian, orthogonal, and PCA-derived alternatives (Tables 4, 9, and 10). These controls support the conclusion that the choice of dictionary matters, rather than merely the addition of the attention-based prompt architecture.

Critical weaknesses
  1. The central interpretability motivation is not sufficiently distinguished from representation-based regularization. The main motivation is that interpretable SAE concepts provide particularly useful guidance for prompt learning. However, the experiments primarily establish that an SAE-derived dictionary outperforms several alternative dictionaries. This is not equivalent to showing that the semantic interpretability of its directions is responsible for the improvement. In particular, the control dictionaries in Tables 4, 9, and 10 differ from the SAE dictionary not only in interpretability but also in how their directions were obtained and in their representation statistics. Therefore, the observed performance differences could potentially be explained by properties of SAE representation learning, such as sparsity, feature specialization, or the quality of the learned representation basis, without requiring the directions to be human-interpretable. The qualitative results in Figure 4, Tables 11–12, and Figure 7 provide examples of semantically meaningful features and activation changes. These are useful supporting observations, but they do not quantitatively establish that semantic interpretability is the mechanism responsible for the classification improvements. Consequently, the paper convincingly supports the narrower claim that SAE-derived representations can improve prompt learning, while the stronger claim that mechanistic interpretability itself guides successful adaptation remains insufficiently isolated. This distinction matters because the latter is the primary conceptual motivation and claimed contribution of the work.

  2. The mathematical formulation does not establish the claimed preservation of interpretable concept directions throughout prompt construction. In Section 3.2.2, Equation (4) defines the prompt tokens.The text subsequently states that the resulting prompt tokens are, by construction, expressible in the same interpretable basis as the original concept dictionary. However, standard multi-head attention includes learned value and output projections. With unconstrained projections, an attention output is not necessarily an attention-weighted combination of the original, untransformed dictionary vectors in their original representation space. Additional assumptions or architectural constraints would be required for this property to hold. Furthermore, Equation (7) regularizes the projections of the dictionary (E) through (R2V) and (R2T). It does not directly constrain the learned transformations inside the attention selector or measure the geometry of the actual prompt tokens after their composition and projection. Thus, preserving geometric relationships among projected dictionary vectors does not, by itself, demonstrate that the final learned prompt tokens retain interpretable concept semantics. This is consequential because semantic grounding and geometry preservation are presented as central methodological advantages of IPL, rather than incidental properties.

Questions for authors

Q1. Separating SAE representation quality from interpretability. Can the authors provide a control that more directly separates semantic interpretability from other properties of SAE features? For example, comparing the original top-activated concepts against alternative subsets of directions from the same SAE, while controlling for dictionary size and relevant feature statistics, would help establish which properties actually explain the gains.

Q2. Mathematical guarantees and empirical preservation. Does the attention selector use the standard learnable value and output projections? If so, under what conditions do the resulting prompt tokens remain in the span of the original SAE concept directions? Could the authors measure the actual reconstruction error, concept alignment, or geometry distortion of the final learned prompt tokens, rather than relying only on the regularization objective?

Q3. Magnitude and reliability of improvements over strong baselines. In Tables 2 and 3, IPL (Unified) improves average domain-generalization accuracy from 61.16 to 61.50 over VaMP and average cross-dataset accuracy from 67.74 to 68.23. These improvements are relatively small compared with the stronger gains in other settings. Since Section 4.1 states that baseline results are taken from the original papers, could the authors clarify whether the closest baselines were evaluated under matched seeds and implementations, and provide paired uncertainty estimates for these comparisons?

Q4. Split-dependent inference. Section 3.2.4 uses fused classification probabilities for base classes but only the standard classification probability for novel classes. Appendix Table 13 acknowledges that a unified classification rule reduces the average harmonic mean from 83.67 to 83.21. Although this reduction is modest and does not eliminate the reported improvement, could the authors clarify the intended inference scenario and report the principal comparisons under a single scoring rule applicable without knowing whether a test instance belongs to the base or novel evaluation split?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for insightful comments. We note that the reviewer acknowledges our concrete approach to research question and the experimental rigor. We address the issues questions raised below:

W1:

Regarding IPL being not sufficiently distinguished from representation based regularization:

We would like to clarify that in representation-based regularization, prompt tokens remain free vectors and a loss pulls the model’s representations towards a target. In IPL, the prompt tokens are composed of the interpretability tool’s concept directions, with each token representing composition of concept directions and IPL training learns how to select them and map them into each injecting layer. Hence, IPL which uses SAE dictionary departs significantly from representation based regularization methods.

could potentially be explained by properties of SAE representation learning, such as sparsity, feature specialization, or the quality of the learned representation basis, without requiring the directions to be human-interpretable

These properties are not alternatives to interpretability in SAE. Feature specialization (monosemanticity) is how interpretability is operationalized in SAE literature, and sparse dictionary learning is the objective designed to achieve it. Huben et al. [Ref1] show that SAE features are substantially more interpretable than those of non-sparse decompositions of the same activations. We would like to clarify that IPL does not claim that a human’s ability to name a feature is a part of its training mechanism; perhaps human readable labels are never used in our training as highlighted in Table 11.

The qualitative results do not quantitatively establish the mechanism.

Our attention selector (described in sec 3.2.2) itself establishes the results quantitatively; where every prompt token is composed of concept directions chosen by the selector. In Fig.4 we also show attention weights and in Table 11 we show each token’s composition.

Consequently, the paper convincingly supports the narrower claim that SAE-derived representations can improve prompt learning, while the stronger claim that mechanistic interpretability itself guides successful adaptation remains insufficiently isolated. This distinction matters because the latter is the primary conceptual motivation and claimed contribution of the work.

In IPL, guidance by the SAE itself means guidance by mechanistic interpretability. SAE’s have emerged as the central tool of mechanistic interpretability, their decoder directions are the features the mech interp community studies and measure each feature’s presence in the input (Huben et al.). In IPL, we build prompts from these highly activated top SAE decoder directions (E). The mechanistic interpretability guidance argument therefore holds good because the core of IPL’s operational mechanism is driven by where SAE based prompt tokens enter and how it impacts adaptation. The central research question of our work is to verify the conceptual feasibility of whether interpretability tools like SAE’s can guide adaptation. We also have made the central scope of our work explicitly clear in our introduction (Section 1)

W2:

We request the reviewer to please refer to table 1 and our detailed response to Lage (W1/Q1) which addresses the similar concern. We provide a summary of the same response below:

We agree with the reviewer that our current attention selector weights with MHA’s value and output weights, the prompt tokens do not necessarily lie in the span of E and hence might not be inherently expressible in the same basis. To address this concern we designed a variant of an attention selector whose selected directions always span the basis of E. Our convex selector sets the MHA’s value and output weights to identity and the output bias to 0, so that I(m) = A(m) E lies in the span of E and now its concept compositions come explicitly from A(m) (attention weights). Hence, the deviation of every injected token under R2V and R2T is bounded by deviation on E. We also note that the results of the convex selector variant matches the accuracy of the earlier variant (82.15 - convex selector vs. 82.01 - naive MHA average HM over four datasets) indicating that the results are mostly driven by SAE’s original concept directions E irrespective of the transformation it undergoes via attention selector.

AuthorsOpenReview ↗

Rebuttal by Authors [continued]

Q1:

As requested by the reviewer, we have designed a control experiment by holding the SAE dictionary size (256), architecture, configuration and seed fixed and we compare IPL’s dictionary (the top 256 features by activation on the base classes) with two alternative subsets of the same SAE : 256 features sampled uniformly from the remaining features (random 256) and the 256 least activated features (bottom-256). In the table below, we report the accuracies for each control variant.

Table 3

VariantBase (B)Novel (N)HM
Real top-25683.38 ± 0.1180.70 ± 0.2882.02 ± 0.18
SAE random-25681.51 ± 1.0874.81 ± 0.1878.01 ± 0.53
SAE bottom-25682.46 ± 0.1374.48 ± 0.3578.27 ± 0.20

We can observe that selecting top-256 highly activated features clearly outperforms other control variants with the same held out SAE dictionary. This observation implies that the gains are not due to SAE feature statistics but the semantic meaning coming from added prompt tokens which provide context during adaptation.

Q2:

We request the reviewer to refer to our response to #Lage;w1/q1 where we construct a variant of attention selector convex selector whose selected directions are convex combinations of original concept directions and lie in their span.

As requested by the reviewer, we measured final learned prompt token’s reconstruction error, concept alignment, and geometric distortions below:

Table 4

MetricWhat it tells usMHA selectorConvex selectorReference
Residual outside span(E) ↓How much of the token cannot be expressed by concept directions (reconstruction error)0.540.0000.82 (random vectors)
cos(token, A(m)E) ↑Whether the token attends to original concepts (concept alignment)0.471.00—
Effective no. of concepts ↓How many concepts a token is composed of196.4128.1256 (uniform)
Top-10 attention mass ↑Share of a token built from its 10 strongest concepts0.060.440.04 (uniform)
(R2V(m)(i),i) ↑Whether injection into CLIP preserves the token's direction (geometry distortion)0.370.660.00
B2N HM ↑Accuracy82.0182.15—
Prompt-side GMAC ↓Compute2.201.13—

The experiments were performed on the trained prompt tokens averaged over four datasets (ImageNet, StanfordCars, SUN397, UCF101) with 3 seeds. We observe that the convex selector performs impressively better compared other variants in all measures by minimizing the geometric distortion and maximizing the concept alignment while cutting the prompt side compute to almost half. This indicates that prompts selected by convex selector lie in the span of the original concept directions.

Q3:

Regarding evaluation protocol:

Following the standard practice in prompt learning (CoOp through MMRL and VaMP), baseline results are taken from the original papers. All methods use the same CLIP ViT-B/16 backbone, the same 16-shot splits and the same evaluation protocol which we have detailedly mentioned in our experiments section. We also provide paired uncertainty estimates for each run in appendix table 7.

Clarification regarding scope of our claim:

The central scope of our work is to verify the conceptual feasibility of whether interpretability tools like SAE’s can guide adaptation, and as stated in our contributions, we observe that IPL outperforms prior interpretable prompt learning methods meanwhile showing competitive performance against blackbox methods.

Q4:

The base to novel benchmark evaluates base and novel classes as separate splits, so the split is already known. Our asymmetric rule follows MMRL where representation token head is trained on base classes only and is therefore applied only to them. We match this protocol in order to be fair and comparable with the MMRL baseline.

We agree with the reviewer that if a split is unknown at test time, a single inference rule is required. As acknowledged by the reviewer, we already report and analyze this in table 13, where we computed both asymmetric and unified rules offline from the IPL’s dumped logits. Note that the unified rules by construction can be used as a split agnostic inference protocol when split is unknown for a test sample.

AuthorsOpenReview ↗

Sincere reminder to acknowledge our rebuttal

Dear Reviewer WgfD, As the discussion period closes, we wanted to gently check whether our responses have addressed your concerns. We would be glad to clarify anything further.

Reviewer_LageOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2(+1)
Read review
Paper summary

This paper proposes IPL, which aims to shift mechanistic interpretability tools, such as SAEs, from post-hoc analysis to actively guiding the downstream adaptation of vision-language models. Specifically, IPL first computes the average SAE concept activations on downstream base-class data and selects the top-k decoder directions to form a concept dictionary. Learnable queries then compose these concepts into prompt tokens through layer-wise multi-head attention. These tokens are linearly projected and injected into the intermediate and later layers of the frozen CLIP vision and text encoders. A projection-preserving regularization term is further introduced to constrain the geometric structure of concepts, while training jointly optimizes the cross-entropy losses of the CLS and rep-token branches. Evaluations show that the unified concept dictionary integrating vision and language modalities generally outperforms existing interpretable prompt-learning methods and achieves comparable or superior performance to the strongest black-box methods.

Critical strengths
  1. The problem setting is meaningful and research question is well-motivated. Shifting interpretability tools such as SAEs from post-hoc analysis to actively guiding model adaptation is a worthwhile direction to explore, and the motivation is clearly stated.

  2. The evaluation is reasonably comprehensive, covering four standard protocols across a broad set of datasets, with IPL results averaged over three random seeds. Per-dataset results and computational overhead are provided, and the authors acknowledge the increased FLOPs.

  3. The framework is relatively agnostic to the choice of SAE and supports vision-only, text-only, and unified concept dictionaries, demonstrating a degree of generality.

Critical weaknesses
  1. The paper’s central claim of "interpretable prompt learning" is insufficiently supported. Standard MHA includes learnable WVW_V and WOW_O, and I(m)I^{(m)} in general does not lie in the span of EE. Therefore, the claim in Section 3.2.2 that prompt tokens "by construction are expressible in the same interpretable basis" is not justified. Moreover, Lproj\mathcal{L}_{\mathrm{proj}} only constrains the geometric consistency between R2V(m)(E)\mathrm{R2V}^{(m)}(E) and EE, and between G(R2T(m)(E))G(\mathrm{R2T}^{(m)}(E)) and G(E)G(E), without imposing corresponding constraints on the actual injected prompts I(m)I^{(m)} or the transformations induced by WVW_V and WOW_O. Thus, the paper lacks sufficient justification for the leap from projection-based geometric preservation to prompt interpretability.

  2. The paper attributes the gains to the SAE concept dictionary, but lacks key controls to establish this attribution. Table 10 shows that replacing EE with random directions still yields an average HM of 79.09, close to MMRL, suggesting much of the gain may come from the MMRL-like architecture, with the dictionary contributing only about 2.9 HM points in B2N. Moreover, the prompt-side FLOPs are roughly 96×96\times those of MMRL, leaving the contribution of added capacity unclear.

  3. The method exhibits a potential layer-space mismatch. VL-SAEs are trained on CLIP's encoder representations (the space the paper calls the residual-stream space), yet IPL injects EE into intermediate-layer residual streams. The core assumption that semantic directions meaningful at the final representation retain the same semantics at intermediate layers has not been validated.

  4. The paper does not discuss potential overlap between the training data of the interpretability tools and the downstream evaluation datasets. PatchSAE is trained on ImageNet yet evaluated on ImageNet and its variants, raising concerns about potential data leakage. VL-SAE is trained on CC3M, yet the paper does not analyze potential semantic or image-level overlap with the 15 downstream datasets, particularly their novel classes. Given that the paper’s central claim is that SAE concept directions improve novel-class generalization, if the concept dictionary has implicitly exposed the model to downstream categories, some of the observed gains may stem from SAE training-data contamination rather than the proposed concept-guidance mechanism.

Questions for authors

How does performance change when WVW_V and WOW_O are removed from MHA, restricting each prompt token to a convex combination of the rows of EE? Since standard MHA includes learnable WVW_V and WOW_O, the resulting tokens are not necessarily confined to span⁡(E)\operatorname{span}(E). This ablation would directly test whether the claimed interpretable basis is sufficient for the performance gains.

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for providing insightful feedback and we note that the reviewer acknowledges the importance of our research direction and well motivated research direction. We address all the concerns and questions raised by the reviewers below:

W1 & Q1:

We would like to clarify that at each injection layer mm, the standard MHA selector computes attention weights �2��2� for each head �3��3�, so each row of �4��4� is a distribution over the �5��5� concept directions for one prompt token. With learned value and output projections �6�and�7��6� and �7�, our Equation (4) evaluates to �8��8�, where �9�,�10��9�, �10� is the corresponding block of the output projection, and b(m)b^{(m)} is the combined bias. Note that concept selection here is driven by �12��12�, and �13��13� is a linear map shared across prompt tokens within each head. We acknowledge the reviewer’s argument regarding the statement in Section 3.2.2 where we had claimed the interpretability span property more strongly than the MHA selector actually guarantees.

We now have an unambiguous convex-selector variant formulation taking into consideration the suggestion given by the reviewer in Q1: We remove the learned value and output transformations and directly compute �14�,where�15��14�, where �15� has nonnegative entries and each row sums to 1, so that every token selected is exactly a convex combination of the original concept directions before projection. With this selector configuration, we repeated our experiments for Table 1 on four datasets and have reported the results in the table below.

Impact of convex selector on projection-preservation regularizer:

Now that each row of �16��16� sums to 1 and �17��17� is affine, �18��18�, so, provided �19��19� preserves dimensionality, for every prompt token nn, $

∥R2V(m)(I(m)[n,:])−I(m)[n,:]∥2≤max⁡j∥R2V(m)(E[j,:])−E[j,:]∥2.\left\|R2V^{(m)}(I^{(m)}[n,:])-I^{(m)}[n,:]\right\|_2 \leq \max_j\left\|R2V^{(m)}(E[j,:])-E[j,:]\right\|_2.

$ This bound shows that the maximum Euclidean deviation of the original SAE concept directions �21��21� under �22��22� caps the Euclidean deviation of every token obtained via the convex selector under the same projection.

Results:

We observe that the results for base-to-novel generalization using the convex selector (the selector where each token is a convex combination of concept directions) are similar to the results of our earlier formulation on the four evaluated datasets. This indicates that explicit composition of SAE’s original concept directions EE is sufficient to retain similar mean performance in these experiments without the unconstrained value and output transformations of the original selector.

Table 1:

DatasetIPL (MHA) HMIPL (convex) HMΔ
ImageNet76.8376.97+0.14
StanfordCars81.1481.41+0.27
SUN39782.5582.39−0.16
UCF10187.5287.84+0.32
Average82.0182.15+0.14
AuthorsOpenReview ↗

Rebuttal by Authors [continued]

W2

We respectfully disagree with the reviewer’s interpretation regarding our control experiment in table 10. The control experiments are not performed with “MMRL-like architecture”; they are performed with IPL architecture by replacing SAE dictionaries with random directions of identical shape that are not derived from an SAE. Every other component remains unchanged here. IPL’s architecture is also substantially different from MMRL’s in terms of initializations and prompt token projections. The control we proposed in Table 10 therefore isolates the contribution of the SAE concept directions for downstream task performance. Also, because our control experiments hold architecture and capacity fixed across the IPL experiment and yield in HM close to MMRL, it shows that IPL’s architecture and capacity alone do not improve over MMRL; rather the improvement is attributable to the SAE concept dictionary.

Clarification regarding the claim of our work: We would like to clarify that +2.49 HM accuracy in B2N over the previous baseline is not the central goal of our work. As stated in our introduction, our work primarily tests the conceptual feasibility of the idea that the interpretability tools can be used for actionable downstream adaptation, and not just for probing and interpreting model internal mechanisms.

Regarding FLOP’s and computational complexity clarification: In the limitation section (Appendix Section F) we acknowledge that most of IPL's prompt-side FLOPs come from the independent attention selector at each of the seven injection layers, each of which projects the 256 concept directions into keys and values. But, with a single attention selector shared across layers, this cost is incurred once, and the total forward cost becomes comparable to MMRL's and meanwhile performance remains similar to that of the independent per-layer selectors. The convex selector introduced earlier in our response to W1 also reduces FLOPs by further halving the prompt-side compute at the same accuracy. Finally, the selector takes no image input, so all prompt tokens are input-independent: they are computed once after training and cached. At inference, IPL's cost therefore equals that of the CLIP encoders with injected tokens, the same as MMRL's.

Table 2:

SelectorTrainable paramsPrompt-side GMAC (training)Total GMAC (inference)B2N HM
MMRL4.99M0.02320.9579.40
IPL, independent per-layer MHA23.8M2.20323.1382.01
IPL, shared MHA9.65M0.34421.2782.18
IPL, convex (response to W1)15.6M1.12621.9382.15

W3:

We would like to clarify that E is extracted at the layer the SAE was trained on. For instance, VL-SAE is trained on CLIP’s final embedding, and PatchSAE on the penultimate layer. Since our contribution is to leverage existing SAE’s rather than train new ones, we take E from the layer each SAE was designed for, which is also where well-formed, fine-grained concepts emerge in vision transformers.

We appreciate a valid question raised regarding how semantics of E are represented at different intermediate layers. We provide the clarification below: IPL does not insert the same E vector at each layer. Each injection layer in IPL has its own selector and its own projections, so each layer receives its own composition of concepts, mapped separately into that layer’s residual stream. Prompt tokens are appended as additional sequential positions rather than added to the residual stream. Each layer receives their own set of prompt tokens via a layer specific attention selector as a context and does not decode them as a direction in its residual. In Appendix section D.1 we can clearly observe how concepts are represented in each injected layer and in Section E we show how the strength of these concepts change before and after injection of IPL prompts.
Injection is restricted to mid to late layers in IPL. Our ablation in Fig 3b shows that insertion into early layers generalizes worse, while layers 6-12 perform better, and this empirical observation is one of the reasons why the final-layer concepts are most compatible with later residual streams.

AuthorsOpenReview ↗

Rebuttal by Authors [continued]

W4:

VL-SAE is trained without class labels, to reconstruct CLIP embeddings of CC3M image-text pairs. CLIP itself was pretrained on 400M web image-text pairs of the same kind, so any semantic overlap between web data and the downstream categories is already present in the frozen CLIP backbone that IPL and perhaps every other baseline method use. This is exactly the reason why B2N protocol defines novel classes as unseen during prompt learning rather than unseen during pretraining and zero shot CLIP performance already reaches 74.22. Hence, novel classes are novel for IPL in exactly the same sense as every other prompt learning baseline. Also, E is selected using only base class training images and prompts (we describe it in Sec 3.2.1) and training uses only base class data. Novel class images, labels and names are used ONLY at evaluation.

The similar argument made above holds at the level of entire datasets, In cross-dataset transfer, E is selected and the prompts are trained on ImageNet ONLY, the 10 target datasets are never used to build E or to train the prompts. IPL nonetheless achieves the highest average target accuracy as reported in Table 2. IPL’s advantage therefore does not depend on any access to the evaluation data, and holds on data whose categories played no role in constructing the concept dictionary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.