acceptodds
Under review as a conference paper at ICLR 2027

TransLP++: Transductive Multimodal Linear Probing for Few-Shot CLIP

Abstract

Vision-Language Models (VLMs) such as CLIP provide strong transferable representations for few-shot learning. Existing methods such as Linear Probe++ (LP++) efficiently adapt pretrained VLMs through multimodal linear probing, but remain inductive and do not exploit unlabeled query samples. We propose TransLP++, a transductive extension of multimodal linear probing that incorporates query-set information into LP++ while keeping the pretrained VLM frozen. Specifically, TransLP++ jointly optimizes the multimodal classifier and auxiliary query assignments by combining labeled support supervision, query-set mutual information, and CLIP-guided semantic regularization. We develop an alternating optimization procedure with block-wise Lipschitz analysis and data-driven initialization for efficient adaptation. The proposed method requires no modification to the pretrained VLM and retains the lightweight, black-box nature of classifier-based adaptation. Experiments on 11 few-shot classification benchmarks and multiple CLIP backbones demonstrate consistent improvements over strong inductive and transductive baselines, including LP++, TransCLIP, and TIM++. Our code is available at https://anonymous.4open.science/r/ICLR-8924-TransLP.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Few-shot Learning, Linear Probe, Transductive Learning, CLIP, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zGDoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TransLP++, a transductive extension of LP++ for few-shot CLIP adaptation. The classifier keeps the LP++ parameterization (a visual prototype wk\mathbf{w}_k plus a learnable scaled text embedding αktk\alpha_k \mathbf{t}_k) and is optimized with an objective that combines (i) label-smoothed cross-entropy on the support set, (ii) a mutual-information term over the unlabeled query set, and (iii) a KL regularizer toward CLIP's zero-shot predictions. Auxiliary assignment variables q\mathbf{q} are introduced to relax the objective, leading to an alternating scheme: an approximate closed-form update for q\mathbf{q} and block-coordinate updates of (W,α)(\mathbf{W}, \boldsymbol{\alpha}) with step sizes derived from block-wise Lipschitz constants; in practice a momentum variant with a fixed learning rate is used. Experiments on 11 datasets, 5 CLIP backbones and 1–16 shots show consistent improvements over LP++, TransCLIP and TIM++.

Critical strengths
  1. The motivation is clear (LP++ is inductive and discards query-set information, which matters most in the low-shot regime), and the method follows directly from it.
  2. The experimental coverage is broad: 11 datasets × 5 backbones × 5 shot settings × 3 seeds, with standard deviations, a sensitivity analysis (Table 7), an ablation of the two query-related terms (Table 3) and an optimizer comparison (Tables 5–6). Gains over TIM++ are consistent across all 25 backbone–shot combinations, and the largest gains appear at 1–2 shots, as the motivation predicts.
  3. A single hyperparameter configuration is used for all datasets, backbones and shots, which is good practice. Anonymous code is provided.
  4. The writing is clear and the derivations are given in full in the appendix.
Critical weaknesses
  1. Limited novelty with respect to TIM++ and LP++. Eq. (2) is the TIM/TIM++ objective (support CE + query mutual information + KL to zero-shot predictions) combined with the LP++ logit parameterization and label smoothing. The relaxation with auxiliary variables (Prop. 1), the KKT update (Prop. 2), the block Lipschitz constants (Prop. 3) and the centroid initialization (Prop. 4) all closely follow the TIM/TIM++ and LP++ derivations. The ablation in Table 3 attributes almost all of the gain to the query-MI term (+14.72 / +5.07 / +0.98), which is TIM++'s contribution, while the KL term alone is marginal or negative at 16 shots. The paper should state precisely what is new beyond "TIM++ objective + LP++ classifier", and add an ablation that isolates the αktk\alpha_k \mathbf{t}_k term (e.g., TransLP++ with α\boldsymbol{\alpha} fixed to 0 vs. TIM++ under identical optimization).

  2. The theory is not what is evaluated. The Lipschitz step sizes of Prop. 3 are presented as a main contribution, but every main result uses the momentum variant with a fixed lr = 1e-3 and a per-step ℓ2\ell_2 normalization of each wk\mathbf{w}_k, which the analysis does not cover. Convergence of the full alternating scheme (including the q\mathbf{q}-update) is not established; citing Theorem 2.2 of LP++ only covers the (W,α)(\mathbf{W}, \boldsymbol{\alpha}) sub-problem with q\mathbf{q} fixed. The approximation in Eq. (6), which treats the per-sample normalization constant CiC_i as sample-independent, is not analyzed or empirically validated (e.g., by reporting the objective value under the exact vs. approximate update).

  3. Computational cost relative to the closest competitor is underplayed. Table 4 shows TIM++ at ~2.28 s and TransLP++ at ~118 s on ImageNet-16shot (roughly 50×), for an average gain of 1.2–1.6 points. The text compares the runtime to LP++ (113 s) rather than to TIM++. Please explain why T=1000×20T = 1000 \times 20 inner iterations are necessary when TIM++ converges in a few seconds, and report accuracy-vs-time curves.

  4. Missing comparisons. LIMO [1] is cited as a related transductive few-shot CLIP method but is not compared. Standard few-shot CLIP baselines used in the LP++ paper (e.g., CoOp, Tip-Adapter-F, PromptSRC) are absent from Table 1. In addition, the query set is the full test set (∣Q∣=50,000|Q| = 50{,}000 on ImageNet); results with smaller or batched query sets and under class imbalance, as analyzed in TransCLIP, are needed to assess the robustness of the transductive setting.

  5. Hyperparameter selection and consistency. Section 4.1 states that hyperparameters were chosen on "validation performance" under the 4-shot setting; since the method is transductive on the test set, please confirm that a held-out validation split was used and that the query set at tuning time was not the test set. Table 7 also shows δ=0.5\delta = 0.5 outperforming the chosen δ=1.0\delta = 1.0 at all three shot settings (76.13 vs. 75.88, 79.04 vs. 78.73, 82.17 vs. 82.01) and larger μ\mu helping at 4/16 shots, which is inconsistent with the stated selection procedure.

  6. Per-dataset failures are not discussed. On Caltech101 all transductive methods, including TransLP++, fall below zero-shot CLIP at 1–2 shots (89.71 vs. 93.31) and below LP++ at every shot; on ImageNet TransLP++ trails LP++ and TIM++ at 1–8 shots. An analysis of when transduction hurts (e.g., when the zero-shot prior is already strong) would make the contribution more informative.

  7. Presentation issues. Zanella et al. 2024a and 2024b are the same paper (duplicate reference); "Appendix 1" in Section 3.3 should be Algorithm 1; a stray closing quotation mark ends Appendix C.2; "Notablly" in A.3; Lemma 2 defers its proof to LP++ without restating it; Eq. (12) says the initializers are normalized so that ρ1,ρ2\rho_1, \rho_2 cancel, yet Algorithm 1 (lines 4–5) still carries τ/ρ\tau/\rho factors; the choice π2=4\pi_2 = 4 (vs. the stated bound π2≥1\pi_2 \ge 1) is not justified.

Overall, this is a carefully executed and well-evaluated engineering combination of two existing methods. However, the technical contribution over TIM++/LP++ is thin, and the theoretical component is not used in the reported system. If the authors can (a) isolate the contribution of the LP++ parameterization against TIM++, (b) justify or reduce the 50× cost relative to TIM++, and (c) add LIMO and the standard few-shot CLIP baselines together with a query-set-size analysis, I would consider raising my score.

[1] Baklouti et al., Language-aware information maximization for transductive few-shot CLIP, TMLR 2026.

Questions for authors

See above

6 comments

We sincerely thank the reviewer for the positive assessment of our work, particularly for recognizing the clear motivation, comprehensive experimental evaluation, consistent improvements over TIM++, the unified hyperparameter configuration across all experiments, and the clarity of the manuscript and theoretical derivations. We also appreciate the reviewer’s constructive questions and suggestions, which help us further clarify and strengthen our work. We address these concerns one by one below.

Response 1: Contribution of the TransLP++ parameterization over TIM++

We thank the reviewer for this constructive feedback, which motivated us to further investigate the factors contributing to TransLP++'s performance gains over the state-of-the-art method TIM++. In addition to the ablation studies on different information sources in TransLP++ (Table 3), we conduct further experiments to examine two key differences between TransLP++ and TIM++: (1) the classification logits and (2) label smoothing. As shown in the following table, simply replacing TIM++'s distance-based logits with visual similarity logits fi⊤wk\boldsymbol{f}_i^\top\boldsymbol{w}_k does not consistently improve performance. It slightly degrades accuracy in low-shot settings while yielding gains in higher-shot settings. In contrast, incorporating the CLIP text prior through fi⊤(wk+αktk)\boldsymbol{f}_i^\top(\boldsymbol{w}_k+\alpha_k\boldsymbol{t}_k) consistently and substantially improves performance across all shot settings. Label smoothing provides additional but relatively modest gains. These results indicate that incorporating CLIP's text prior into the classifier logits is the primary contributor to TransLP++'s performance improvement over TIM++.

MethodClassification logitsLabel smoothing1-shot2-shot4-shot8-shot16-shot
TIM++−τ2∥fi−wk∥2-\frac{\tau}{2}\|\boldsymbol f_i-\boldsymbol w_k\|^2×\times74.3275.7077.4679.0280.85
TransLP++, w/o CLIP priorfi⊤wk\boldsymbol f_i^\top\boldsymbol w_k×\times73.8475.5477.6779.3981.33
TransLP++, with CLIP priorfi⊤(wk+αktk)\boldsymbol f_i^\top(\boldsymbol w_k+\alpha_k\boldsymbol t_k)×\times75.3376.6178.4980.0081.75
TransLP++, with CLIP priorfi⊤(wk+αktk)\boldsymbol f_i^\top(\boldsymbol w_k+\alpha_k\boldsymbol t_k)✓\checkmark75.8877.2578.7380.3682.01
TransLP++, w/o CLIP priorfi⊤wk\boldsymbol f_i^\top\boldsymbol w_k✓\checkmark74.7076.3878.1979.8881.70
TIM++−τ2∥fi−wk∥2-\frac{\tau}{2}\|\boldsymbol f_i-\boldsymbol w_k\|^2✓\checkmark74.2075.5477.2578.6780.52

A straightforward way to also incorporate the text prior into TIM++ would be to add a text-distance term to the classifier logits, namely, −τ2∣fi−wk∣2−β2∣fi−tk∣2-\frac{\tau}{2}|\boldsymbol f_i-\boldsymbol w_k|^2-\frac{\beta}{2}|\boldsymbol f_i-\boldsymbol t_k|^2. However, cross-modal semantic alignment in CLIP does not necessarily imply small Euclidean distances between visual and textual embeddings. Thus, directly introducing a text-distance term may not effectively capture their semantic correspondence. This suggests that incorporating CLIP's text prior into the prediction function of TIM++ is not as straightforward as simply extending its original distance-based formulation.

We thank the reviewer for this constructive feedback and will clarify these analysis in the revised manuscript.

Response 2: Justify the computation cost relative to TIM++

We thank the reviewer for this question. The computational overhead of TransLP++ compared with TIM++ is real, and we report this difference transparently. The gap mainly arises from their different classifier parameterizations and optimization procedures. Although TIM++ and TransLP++ minimize the same family of losses (support cross-entropy, query-set mutual information, and CLIP-guided KL regularization), their optimization subproblems differ substantially. TIM++ uses a negative squared-distance prototype classifier (−τ2∥fi−wk∥2-\frac{\tau}{2}\|\boldsymbol f_i-\boldsymbol w_k\|^2) and optimizes only the prototype matrix W\mathbf W. With the auxiliary-variable reformulation, both the qq-update and the W\mathbf W-update admit closed-form solutions.

In contrast, TransLP++ adopts the more expressive LP++ parameterization, fi⊤(wk+αktk)\boldsymbol f_i^\top(\boldsymbol w_k+\alpha_k\boldsymbol t_k), and jointly optimizes the classifier weights W\mathbf W and class-specific text coefficients α\boldsymbol\alpha. Due to the coupling introduced by the shared softmax, these subproblems do not admit closed-form solutions. We therefore employ block-coordinate Majorize-Minimize optimization, requiring multiple inner gradient updates with block-wise Lipschitz step sizes. Specifically, the Momentum variant uses 1000 outer iterations with 20 inner W\mathbf W updates per iteration, resulting in approximately 20000 effective update steps, whereas TIM++ uses 150 outer iterations with closed-form updates. This difference is the primary source of the runtime gap.

Overall, the additional computation reflects a trade-off between optimization efficiency and classifier expressiveness. TIM++ benefits from closed-form prototype updates, whereas TransLP++ adopts a more flexible multimodal classifier that explicitly incorporates CLIP's text prior, leading to consistent accuracy improvements over TIM++ across the evaluated backbone-shot settings. Our additional experiments in the following table show that TransLP++ with 200 iterations already outperforms TIM++ across all five shot settings, with a runtime of 27.70 s on ImageNet, while the default 1,000-iteration setting further improves accuracy to achieve the best results (118 s). These results demonstrate the flexibility to balance computational cost and performance by adjusting the number of iterations. We will incorporate this analysis into the Computational Efficiency subsection to clarify the sources of the overhead and the trade-off between computational cost and performance.

MethodOuter Iterations1-shot2-shot4-shot8-shot16-shotRuntimex(ImageNet, 16-shot)
TIM++150 steps74.32 ± 0.6175.70 ± 0.6477.46 ± 0.7279.02 ± 0.4880.85 ± 0.342.28s
TransLP++100 steps73.56 ± 0.7675.58 ± 0.7377.68 ± 0.6479.32 ± 0.4380.87 ± 0.2913.78s
TransLP++200 steps74.70 ± 0.8276.31 ± 0.7578.22 ± 0.5679.85 ± 0.4481.60 ± 0.2927.70s
TransLP++300 steps75.20 ± 0.8076.62 ± 0.7578.47 ± 0.5280.08 ± 0.4681.83 ± 0.3241.73s
TransLP++1000 steps75.88 ± 0.5977.25 ± 0.5378.73 ± 0.5580.36 ± 0.4482.01 ± 0.35~118s

Response 3: Additional Baselines

We thank the reviewer for this valuable suggestion. We agree that LIMO is a strong method. However, our work focuses on black-box adaptation, where the pretrained encoders remain frozen, and adaptation is performed using their extracted features without accessing or modifying the internal architecture or parameters of the pretrained VLMs. LIMO introduces trainable LoRA modules into the query, key, and value (QKV) projections of CLIP's vision and text encoders, thus requiring access to and modification of the pretrained VLM's internal architecture. In contrast, the proposed TransLP++ and the baselines considered in our paper perform adaptation exclusively at the feature and classifier levels, without modifying the pretrained CLIP encoders. This black-box setting offers broader applicability, particularly when access to the internal architecture or parameters of pretrained VLMs is restricted.

To make our comparison more comprehensive, we additionally include several representative few-shot black-box adaptation methods, including TF, BD-CSPN, LaplacianShot, PT-MAP, TIM, and Tip-Adapter-F. These methods complement the existing comparisons with LP, LP++, TransCLIP, and TIM++. As shown in the updated table, TransLP++ achieves the best performance across all five shot settings, consistently outperforming the other methods. We will incorporate these additional results and explicitly clarify the baseline selection criteria in the revised manuscript.

Method1-shot2-shot4-shot8-shot16-shot
TF (Arxiv'19)43.50 ± 2.1356.99 ± 1.6865.22 ± 0.8470.70 ± 0.7573.74 ± 0.57
BD-CSPN (ICCV'20)51.02 ± 1.8361.86 ± 1.6967.01 ± 1.2370.81 ± 0.8973.59 ± 0.65
LaplacianShot (ICML'20)51.53 ± 2.2162.63 ± 1.2867.45 ± 1.2470.50 ± 0.7172.94 ± 0.49
PT-MAP (ICANN'21)53.02 ± 2.2362.39 ± 1.6466.12 ± 0.5668.70 ± 0.4370.53 ± 0.45
TIM (NeurIPS'20)55.07 ± 2.8364.82 ± 1.8571.02 ± 1.7275.17 ± 1.3478.15 ± 1.22
LP (ICML'21)45.35 ± 1.4456.57 ± 1.0666.18 ± 1.4073.36 ± 0.4878.23 ± 0.46
Tip-Adapter-F (ECCV'22)71.38 ± 0.6473.80 ± 0.6676.13 ± 0.5078.55 ± 0.6481.32 ± 0.48
LP++ (CVPR'24)70.27 ± 0.4872.83 ± 0.8975.62 ± 0.8477.94 ± 0.6380.27 ± 0.38
TransCLIP (NeurIPS'24)73.01 ± 1.3974.32 ± 1.1375.96 ± 0.6877.30 ± 0.6578.32 ± 0.66
TIM++ (AAAI'26)74.32 ± 0.6175.70 ± 0.6477.46 ± 0.7279.02 ± 0.4880.85 ± 0.34
TransLP++ (Ours)75.88 ± 0.5977.25 ± 0.5378.73 ± 0.5580.36 ± 0.4482.01 ± 0.35

~~

Response 4: Effect of query-set size

We thank the reviewer for this constructive question. To investigate the effect of query-set size, we conduct experiments on all 11 benchmark datasets with the support set fixed. We consider 15, 100, 500, and 1,000 query samples, as well as the entire query set. For each limited query-set size, we randomly sample 100 times and report the mean accuracy and standard deviation across trials. Results using the entire query set are averaged over three random seeds. As shown in the following table, TransLP++ consistently outperforms TIM++ across all query-set sizes and shot settings. Its performance generally improves as more query samples become available, while the standard deviations tend to decrease. Notably, TransLP++ maintains relatively stable performance even with only 15 query samples, whereas TIM++ suffers substantial degradation in several settings. These results demonstrate the effectiveness and robustness of TransLP++ under varying query-set sizes.

Query SizeMethod1-shot2-shot4-shot8-shot16-shot
15TIM++59.47 ± 14.7561.07 ± 13.1342.95 ± 17.0225.38 ± 14.1122.51 ± 14.91
15TransLP++63.44 ± 11.3163.79 ± 11.3464.25 ± 11.2364.85 ± 11.3365.25 ± 11.30
100TIM++62.60 ± 7.0361.12 ± 8.4650.56 ± 6.5949.17 ± 5.8752.12 ± 8.03
100TransLP++65.68 ± 4.5766.85 ± 4.5367.91 ± 4.3568.92 ± 4.3869.66 ± 4.17
500TIM++68.99 ± 2.5765.74 ± 3.6666.76 ± 1.7968.54 ± 1.7370.17 ± 1.67
500TransLP++71.48 ± 2.3473.22 ± 2.0474.89 ± 1.8676.36 ± 1.7977.66 ± 1.73
1,000TIM++70.61 ± 1.7971.99 ± 1.5871.92 ± 3.7370.90 ± 2.5675.44 ± 3.51
1,000TransLP++73.14 ± 1.8574.98 ± 1.5776.69 ± 1.3678.25 ± 1.2679.73 ± 1.17
AllTIM++74.32 ± 0.6175.70 ± 0.6477.46 ± 0.7279.02 ± 0.4880.85 ± 0.34
AllTransLP++75.88 ± 0.5977.25 ± 0.5378.73 ± 0.5580.36 ± 0.4482.01 ± 0.35
Reviewer_VGehOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TransLP++, a transductive extension of the multimodal linear probing framework (LP++) for few-shot adaptation of frozen Vision-Language Models (CLIP). The authors augment the inductive support-set cross-entropy loss of LP++ with three components: label smoothing on the support set, an entropy-based mutual information (MI) objective over the unlabeled query set, and a KL-divergence regularization term anchoring query predictions to zero-shot CLIP textual probabilities. To optimize this non-convex objective, the authors apply an Alternating Direction Method (ADM) by introducing auxiliary query assignment variables qq, alternating between an approximate closed-form KKT update for qq and block-coordinate Majorize-Minimize (MM) updates (or momentum gradient descent) for the visual prototypes WW and text-blending coefficients α\alpha. Experiments across 11 standard classification benchmarks and 5 CLIP backbones show consistent accuracy improvements over LP++, TransCLIP, and TIM++.

Critical strengths
  • Consistent Empirical Performance Across Backbones and Shot Regimes: By unifying the multimodal parameterization of LP++ (wk+αktkw_k + \alpha_k t_k) with the transductive mutual information and CLIP-prior regularizers of TIM++/TransCLIP, the method achieves state-of-the-art accuracies across all 25 backbone-shot combinations evaluated (Tables 1 and 2), outperforming TIM++ by ~1.1–1.5% on average.
  • Transparent Formulation and Data-Driven Initialization: The mathematical presentation of the ADM splitting, the difference-of-convex initialization strategy (Proposition 4) that blends support and query centroids to initialize wk(0)w_k^{(0)} and αk(0)\alpha_k^{(0)}, and the block-Hessian spectral bounds are clearly written and easy to verify.
Critical weaknesses
  • Limited Conceptual Novelty and Disconnection Between Theory and Practice: Conceptually, TransLP++ is an incremental combination of LP++ (Huang et al., 2024) and TIM/TIM++ (Boudiaf et al., 2020; Li et al., 2026) / TransCLIP (Zanella et al., 2024). More importantly, the paper devotes substantial space to deriving block-coordinate Lipschitz constants γW=π1τ24λmax⁡(μ∣S∣∑i∈Sfifi⊤+δ+β−γ∣Q∣∑i∈Qfifi⊤)\gamma_W = \frac{\pi_1 \tau^2}{4}\lambda_{\max}\left(\frac{\mu}{|\mathcal{S}|}\sum_{i \in \mathcal{S}} f_i f_i^\top + \frac{\delta+\beta-\gamma}{|\mathcal{Q}|}\sum_{i \in \mathcal{Q}} f_i f_i^\top\right) and γα=π2τ24max⁡k(μ∣S∣∑i∈S(fi⊤tk)2+δ+β−γ∣Q∣∑i∈Q(fi⊤tk)2)\gamma_\alpha = \frac{\pi_2 \tau^2}{4}\max_k\left(\frac{\mu}{|\mathcal{S}|}\sum_{i \in \mathcal{S}} (f_i^\top t_k)^2 + \frac{\delta+\beta-\gamma}{|\mathcal{Q}|}\sum_{i \in \mathcal{Q}} (f_i^\top t_k)^2\right) (Proposition 3, Appendix A.3) to justify parameter-free MM step sizes 1/γW1/\gamma_W and 1/γα1/\gamma_\alpha, only to abandon them in the main experiments in favor of heuristic momentum gradient descent with a fixed learning rate of 10−310^{-3} and ad-hoc post-update ℓ2\ell_2 weight normalization (Algorithm 1, line 23) because the Lipschitz-guided GD is too slow (480s vs. 118s on ImageNet, and still ~50x slower than TIM++'s 2.28s). Furthermore, once wkw_k is projected onto the unit sphere (∥wk∥2=1\Vert{}w_k\Vert{}_2 = 1) after every inner gradient step, the theoretical convergence guarantees of the unconstrained quadratic majorizer (Proposition 3) no longer strictly apply to the actual algorithm used.
  • Unjustified Approximation in Auxiliary Update and Evaluation Protocol Ambiguity: In Proposition 2 (Appendix A.2, Eq. 24), to get a closed-form update for qikq_{ik}, the authors assume the per-sample partition function Ci=∑j=1Kpijδ+β−γδ⋅y^ijγδq^j1δC_i = \sum_{j=1}^K \frac{p_{ij}^{\frac{\delta+\beta-\gamma}{\delta}} \cdot \hat{y}_{ij}^{\frac{\gamma}{\delta}}}{\hat{q}_j^{\frac{1}{\delta}}} is constant (Ci≈CC_i \approx C) across all query samples i∈Qi \in \mathcal{Q}. Since CiC_i measures the alignment between sample ii's model prediction pip_i, its zero-shot prior y^i\hat{y}_i, and the class marginals q^\hat{q}, CiC_i varies wildly between confident in-distribution samples and ambiguous samples. Replacing an exact iterative Sinkhorn/fixed-point scaling of Eq. (23) with qik∝pikδ+β−γδ⋅y^ikγδ(∑i∈Qpikδ+β−γδ⋅y^ikγδ)1δ+1q_{ik} \propto \frac{p_{ik}^{\frac{\delta+\beta-\gamma}{\delta}} \cdot \hat{y}_{ik}^{\frac{\gamma}{\delta}}}{\left(\sum_{i \in \mathcal{Q}} p_{ik}^{\frac{\delta+\beta-\gamma}{\delta}} \cdot \hat{y}_{ik}^{\frac{\gamma}{\delta}}\right)^{\frac{1}{\delta+1}}} in Eq. (26) invalidates the KKT optimality claim of Proposition 2.
  • The paper does not specify the exact structure of the query set Q\mathcal{Q} used in the experiments (e.g., full test split transductive inference where ∣Q∣=50,000|\mathcal{Q}|=50,000 on ImageNet vs. small episodic sampling with uniform class balance). Because the −∑k=1Kp^klog⁡p^k-\sum_{k=1}^K \hat{p}_k \log \hat{p}_k marginal entropy term strongly forces uniform class predictions, performance under realistic, class-imbalanced, or small-batch query sets is left unexamined.
Questions for authors
  1. Clarification on Query Set Protocol and Class Balance: Can you clarify the exact transductive evaluation protocol used across the 11 datasets? Specifically, is the entire test set passed simultaneously as Q\mathcal{Q} (as suggested by ∣Q∣=50,000|\mathcal{Q}|=50,000 for ImageNet in Section 4.4), and how does TransLP++ perform when the query set is small (e.g., 15–100 samples) or exhibits a skewed/Dirichlet class distribution where maximizing marginal entropy p^k\hat{p}_k is detrimental?
  2. Clarification on Proposition 2 Approximation: In Eq. (23), finding q^k=1∣Q∣∑i∈Qqik\hat{q}_k = \frac{1}{|\mathcal{Q}|}\sum_{i \in \mathcal{Q}} q_{ik} only requires a simple 1D fixed-point iteration over a vector of size KK (which takes microseconds). Why introduce the mathematically inaccurate assumption that Ci≈CC_i \approx C across all samples ii in Eq. (24) instead of iterating Eq. (23) for 3–5 steps? What is the empirical difference between solving Eq. (23) vs. using the approximation in Eq. (26)?
  3. Clarification on ℓ2\ell_2 Normalization of wkw_k: In Algorithm 1, line 23, wkw_k is normalized to unit ℓ2\ell_2 norm at every inner iteration during Momentum optimization, but not during Lipschitz-guided GD (lines 18–19). Why is unit-norm projection applied to one optimizer and not the other, and doesn't fixing ∥wk∥2=1\|w_k\|_2 = 1 fundamentally change the optimization landscape analyzed in Proposition 3?
6 comments

We thank the reviewer for recognizing the consistent empirical improvements across different backbones and shot settings, as well as the clarity of our mathematical formulation. We also sincerely appreciate the reviewer’s thorough review and constructive feedback, which helps us to improve our paper. We address the comments and questions point by point below.

Response 1: Clarification on Query Set Protocol and Class Balance

1) Query-set protocal. We thank the reviewer for raising this important concern. In all 11 datasets, we use the entire official test split as the query set for transductive inference, rather than episodic sampling with artificially balanced classes. No additional subsampling or class rebalancing is performed. Therefore, our evaluation reflects realistic query sets with their original, naturally imbalanced class distributions. We summarize the number of classes and query set sizes for all 11 datasets below. These datasets cover diverse query set sizes and numbers of classes, allowing us to evaluate the effectiveness of TransLP++ under different realistic transductive evaluation settings. We will clarify this evaluation protocol in the revised manuscript.

MetricImageNetSUN397AircraftEuroSATStanfordCarsFood101Oxford_PetsFlower102Caltech101DTDUCF101
Number of Classes1000397100101961013710210147101
Query-set Size50000198503333810080413030036692463246516923783

2) Effect of query-set size. We thank the reviewer for this constructive question. To investigate the effect of query-set size, we conduct experiments on all 11 benchmark datasets with the support set fixed. We consider 15, 100, 500, and 1,000 query samples, as well as the entire query set. For each limited query-set size, we randomly sample 100 times and report the mean accuracy and standard deviation across trials. Results using the entire query set are averaged over three random seeds. As shown in the following table, TransLP++ consistently outperforms TIM++ across all query-set sizes and shot settings. Its performance generally improves as more query samples become available, while the standard deviations tend to decrease. Notably, TransLP++ maintains relatively stable performance even with only 15 query samples, whereas TIM++ suffers substantial degradation in several settings. These results demonstrate the effectiveness and robustness of TransLP++ under varying query-set sizes.

Query SizeMethod1-shot2-shot4-shot8-shot16-shot
15TIM++59.47 ± 14.7561.07 ± 13.1342.95 ± 17.0225.38 ± 14.1122.51 ± 14.91
15TransLP++63.44 ± 11.3163.79 ± 11.3464.25 ± 11.2364.85 ± 11.3365.25 ± 11.30
100TIM++62.60 ± 7.0361.12 ± 8.4650.56 ± 6.5949.17 ± 5.8752.12 ± 8.03
100TransLP++65.68 ± 4.5766.85 ± 4.5367.91 ± 4.3568.92 ± 4.3869.66 ± 4.17
500TIM++68.99 ± 2.5765.74 ± 3.6666.76 ± 1.7968.54 ± 1.7370.17 ± 1.67
500TransLP++71.48 ± 2.3473.22 ± 2.0474.89 ± 1.8676.36 ± 1.7977.66 ± 1.73
1,000TIM++70.61 ± 1.7971.99 ± 1.5871.92 ± 3.7370.90 ± 2.5675.44 ± 3.51
1,000TransLP++73.14 ± 1.8574.98 ± 1.5776.69 ± 1.3678.25 ± 1.2679.73 ± 1.17
AllTIM++74.32 ± 0.6175.70 ± 0.6477.46 ± 0.7279.02 ± 0.4880.85 ± 0.34
AllTransLP++75.88 ± 0.5977.25 ± 0.5378.73 ± 0.5580.36 ± 0.4482.01 ± 0.35

3) Effect of Query-Set Class Distribution. To further investigate the effect of query-set class distribution, we conduct experiments using Dirichlet-based sampling with a concentration parameter of 2. Specifically, we sample class proportions from a KK-dimensional Dirichlet distribution and allocate nqueryn_{\text{query}} samples accordingly. We consider query-set sizes of 15, 100, 500, and 1,000. Results are averaged over 11 datasets using the ViT-B/16 backbone. As shown in the following table, TransLP++ consistently outperforms TIM++ across all query-set sizes and shot settings. Compared with random sampling (the above table), Dirichlet-based sampling yields better performance for both methods with small query sets (15 and 100 samples), possibly because explicitly allocating samples across classes helps mitigate the potential lack of class coverage under random sampling. In contrast, random sampling generally performs better with larger query sets (500 and 1,000 samples). This observation is consistent with the objective formulation, as both methods incorporate a marginal entropy term that encourages uniform class predictions. Although deviations from a uniform class distribution may affect performance, TransLP++ maintains reasonable performance under Dirichlet-based sampling, indicating a certain degree of robustness to class imbalance.

Query SizeMethod1-shot2-shot4-shot8-shot16-shot
15TIM++62.48 ± 14.5663.69 ± 13.1344.86 ± 17.0926.95 ± 13.8924.40 ± 15.58
15TransLP++67.07 ± 10.6267.76 ± 10.5868.34 ± 10.6068.89 ± 10.6969.29 ± 10.58
100TIM++63.96 ± 6.8063.93 ± 7.9952.35 ± 6.3951.00 ± 5.2053.64 ± 6.98
100TransLP++68.70 ± 4.3969.97 ± 4.3771.06 ± 4.2972.11 ± 4.1472.94 ± 4.07
500TIM++66.47 ± 2.7862.29 ± 4.1363.31 ± 2.5164.96 ± 2.4866.46 ± 2.42
500TransLP++69.86 ± 2.6871.65 ± 2.5773.29 ± 2.5174.78 ± 2.4476.16 ± 2.31
1,000TIM++66.31 ± 2.5167.35 ± 2.4667.75 ± 4.4866.68 ± 4.7770.51 ± 4.68
1,000TransLP++70.06 ± 2.4771.80 ± 2.3673.52 ± 2.2475.24 ± 2.0876.81 ± 1.98
AllTIM++74.32 ± 0.6175.70 ± 0.6477.46 ± 0.7279.02 ± 0.4880.85 ± 0.34
AllTransLP++75.88 ± 0.5977.25 ± 0.5378.73 ± 0.5580.36 ± 0.4482.01 ± 0.35

Response 2: Clarification on Proposition 2 Approximation

We appreciate the reviewer’s careful examination of Proposition 2. We agree that the approximation (Ci≈CC_i\approx C) is an additional assumption introduced to obtain the simplified closed-form update in Eq. (26). Therefore, Eq. (26) should be interpreted as an approximate update rather than an exact closed-form solution to the original KKT conditions. We will clarify it in the revised manuscript.

1)Theoretical Analysis. To explain why we adopt Eq. (26) instead of directly iterating Eq. (23), we first rewrite the self-consistent KKT relation as follows. Given the exact update of qikq_{ik} in Eq. (23), taking the marginal over the query set yields

\frac{1}{|\mathcal Q|} \sum_{i\in\mathcal Q} \frac{a_{ik}\hat q_k^{-1/\delta}}{C_i}$$ where \$ a_{ik}=p_{ik}^{(\delta+\beta-\gamma)/\delta}\hat y_{ik}^{\gamma/\delta} \$ and \$ C_i=\sum_{j=1}^{K}a_{ij}\hat q_j^{-1/\delta} \$ .Thus, directly solving Eq. (23) requires iterating $\hat{q}_k^{(t+1)}=G_k(\hat{\mathbf q}^{(t)})$. Importantly, **the convergence of this iteration depends on the local Jacobian of the mapping and is not guaranteed in general.** To illustrate this, we examine the local Jacobian of the marginal mapping $G$. Consider the special case in which all query assignments are uniform, i.e., $q^*_{ik}=1/K$ for every query sample $i$ and class $k$. The Jacobian of the marginal mapping at this fixed point is $$J_{kl}= \frac{1}{\delta} \left( \frac{1}{K}\mathbf 1\mathbf 1^\top-I \right)_{kl}.$$ Here, $\mathbf{1}$ denotes the all-ones vector in $\mathbb{R}^{K}$, and $I$ is the $K\times K$ identity matrix, thus $\tfrac{1}{K}\mathbf{1}\mathbf{1}^{\top}-I$ has diagonal $-\tfrac{K-1}{K}$ and off-diagonal $\tfrac{1}{K}$. Its eigenvalues are $0$ and $-1/\delta$. Consequently, when $\delta<1$, namely $\lvert-1/\delta\rvert>1$, the fixed point is locally unstable. When $\delta=1$ (the setting in our experiments), the linearized mapping is non-contractive ($\lvert-1/\delta\rvert=1$), and local convergence is not guaranteed. In conclusion, for multiple query samples, the local stability of the fixed-point iteration depends on the prediction distributions across query samples, and convergence is not guaranteed in general. In contrast, under the approximation $C_i \approx C$, Eq. (23) can be simplified to yield the closed-form update in Eq. (26), which avoids potentially unstable fixed-point iterations. **2) Experimental Results.** To further evaluate this approximation, we compare Eq. (26) with 5 and 10 fixed-point iterations of Eq. (23), using both Momentum and Lipschitz-guided GD. As shown in the following table, the approximate update consistently achieves higher accuracy across all five shot settings. Moreover, increasing the number of fixed-point iterations from 5 to 10 yields nearly identical results, indicating that additional iterations provide negligible performance improvement. These results support the effectiveness of the approximate update in practice. It provides a more accurate empirical solution in our experiments while avoiding the additional computational overhead of repeated fixed-point iterations. We will add the theoretical clarification and experimental comparison to the revised manuscript. | Optimizer | q-Update Strategy | 1-shot | 2-shot | 4-shot | 8-shot | 16-shot | |:---|:---|---:|---:|---:|---:|---:| | Momentum | Fixed-point (Eq. (23), 5 iterations) | 74.71 ± 0.64 | 76.14 ± 0.67 | 77.67 ± 0.47 | 79.42 ± 0.36 | 81.02 ± 0.29 | | Momentum | Fixed-point (Eq. (23), 10 iterations) | 74.72 ± 0.63 | 76.14 ± 0.67 | 77.68 ± 0.47 | 79.42 ± 0.36 | 81.02 ± 0.29 | | Momentum | **Approximate (Eq. (26))** | **75.88 ± 0.59** | **77.25 ± 0.53** | **78.73 ± 0.55** | **80.36 ± 0.44** | **82.01 ± 0.35** | | Lipschitz-guided GD | Fixed-point (Eq. (23), 5 iterations) | 74.91 ± 0.58 | 76.13 ± 0.71 | 77.71 ± 0.48 | 79.38 ± 0.43 | 81.09 ± 0.32 | | Lipschitz-guided GD | Fixed-point (Eq. (23), 10 iterations) | 74.91 ± 0.57 | 76.12 ± 0.71 | 77.71 ± 0.49 | 79.38 ± 0.43 | 81.08 ± 0.31 | | Lipschitz-guided GD |**Approximate (Eq. (26))**| **75.91 ± 0.74** | **77.13 ± 0.63** | **78.83 ± 0.59** | **80.35 ± 0.52** | **82.03 ± 0.33** |
Reviewer_hiQGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TransLP++, a transductive multimodal linear probing framework that extends the inductive LP++ paradigm to explicitly exploit unlabeled query-set information. In addition, the authors develop an alternating optimization procedure with block-wise Lipschitz analysis, which provides theoretically motivated step sizes for classifier updates. Extensive experiments across 11 benchmark datasets and 5 different CLIP backbones demonstrate that TransLP++ consistently achieves state-of-the-art accuracy compared to both inductive and transductive baselines.

Critical strengths
  1. While the momentum variant is fast, the theoretically grounded Lipschitz-guided gradient descent is computationally expensive, requiring approximately 480 seconds on ImageNet 16-shot compared to just 2.28 seconds for the competing TIM++ method.

  2. The method inherently assumes access to a batch of query samples at once. It may not be applicable in strict online inference scenarios where query samples arrive sequentially one by one and require immediate predictions without access to a broader query distribution.

Critical weaknesses
  1. The introduction lacks a clear and detailed overview of the methodological pipeline. Its significant to explicitly map which specific modules or components of the proposed framework are designed to solve which corresponding challenges raised earlier in the text.

  2. The core components of the objective function have been heavily explored in prior works like TIM++ and TransCLIP, such as transductive information maximization and KL-divergence regularization towards zero-shot predictions. The primary contribution lies in applying these established techniques specifically to the LP++ classifier architecture.

  3. The method inherently assumes access to a batch of query samples at once. It may not be applicable in strict online inference scenarios where query samples arrive sequentially one by one, and require immediate predictions without access to a broader query distribution.

Questions for authors
  1. How sensitive is the performance of TransLP++ to the size of the query set? Does the accuracy degrade significantly if only a very small batch of unlabeled query samples is available during the transductive adaptation phase?

  2. How does TransLP++ perform under out-of-distribution (OOD) scenarios or domain shifts, where the underlying distribution of the unlabeled query set significantly diverges from the distribution of the labeled support set?

3 comments

Response to Reviewer hiQG

We sincerely thank the reviewer for the positive assessment of our empirical results and the constructive suggestions. We appreciate the reviewer’s recognition of the consistent performance improvements across 11 benchmarks and 5 CLIP backbones. We address the main concerns below.

Response 1: Effect of query-set size

We thank the reviewer for this constructive question. To investigate the effect of query-set size, we conduct experiments on all 11 benchmark datasets with the support set fixed. We consider 15, 100, 500, and 1,000 query samples, as well as the entire query set. For each limited query-set size, we randomly sample 100 times and report the mean accuracy and standard deviation across trials. Results using the entire query set are averaged over three random seeds.

As shown in the table, TransLP++ consistently outperforms TIM++ across all query-set sizes and shot settings. Its performance generally improves as more query samples become available, while the standard deviations tend to decrease. Notably, TransLP++ maintains relatively stable performance even with only 15 query samples, whereas TIM++ suffers substantial degradation in several settings. These results demonstrate the effectiveness and robustness of TransLP++ under varying query-set sizes.

Query SizeMethod1-shot2-shot4-shot8-shot16-shot
15TIM++59.47 ± 14.7561.07 ± 13.1342.95 ± 17.0225.38 ± 14.1122.51 ± 14.91
15TransLP++63.44 ± 11.3163.79 ± 11.3464.25 ± 11.2364.85 ± 11.3365.25 ± 11.30
100TIM++62.60 ± 7.0361.12 ± 8.4650.56 ± 6.5949.17 ± 5.8752.12 ± 8.03
100TransLP++65.68 ± 4.5766.85 ± 4.5367.91 ± 4.3568.92 ± 4.3869.66 ± 4.17
500TIM++68.99 ± 2.5765.74 ± 3.6666.76 ± 1.7968.54 ± 1.7370.17 ± 1.67
500TransLP++71.48 ± 2.3473.22 ± 2.0474.89 ± 1.8676.36 ± 1.7977.66 ± 1.73
1,000TIM++70.61 ± 1.7971.99 ± 1.5871.92 ± 3.7370.90 ± 2.5675.44 ± 3.51
1,000TransLP++73.14 ± 1.8574.98 ± 1.5776.69 ± 1.3678.25 ± 1.2679.73 ± 1.17
AllTIM++74.32 ± 0.6175.70 ± 0.6477.46 ± 0.7279.02 ± 0.4880.85 ± 0.34
AllTransLP++75.88 ± 0.5977.25 ± 0.5378.73 ± 0.5580.36 ± 0.4482.01 ± 0.35

Response 2: Robustness to distribution shifts

Thank you for suggesting this important evaluation. We have conducted additional experiments to investigate the behavior of TransLP++ under distribution shifts.

1)Effect of Query-Set Class Distribution. We sample class proportions from a KK-dimensional Dirichlet distribution and allocate nqueryn_{\text{query}} samples accordingly. We consider query-set sizes of 15, 100, 500, and 1,000. Results are averaged over 11 datasets using the ViT-B/16 backbone. As shown in the following table, TransLP++ consistently outperforms TIM++ across all query-set sizes and shot settings. Compared with random sampling (the above table), Dirichlet-based sampling yields better performance for both methods with small query sets (15 and 100 samples), possibly because explicitly allocating samples across classes helps mitigate the potential lack of class coverage under random sampling. In contrast, random sampling generally performs better with larger query sets (500 and 1,000 samples). This observation is consistent with the objective formulation, as both methods incorporate a marginal entropy term that encourages uniform class predictions. Although deviations from a uniform class distribution may affect performance, TransLP++ maintains reasonable performance under Dirichlet-based sampling, indicating a certain degree of robustness to class imbalance.

Query SizeMethod1-shot2-shot4-shot8-shot16-shot
15TIM++62.48 ± 14.5663.69 ± 13.1344.86 ± 17.0926.95 ± 13.8924.40 ± 15.58
15TransLP++67.07 ± 10.6267.76 ± 10.5868.34 ± 10.6068.89 ± 10.6969.29 ± 10.58
100TIM++63.96 ± 6.8063.93 ± 7.9952.35 ± 6.3951.00 ± 5.2053.64 ± 6.98
100TransLP++68.70 ± 4.3969.97 ± 4.3771.06 ± 4.2972.11 ± 4.1472.94 ± 4.07
500TIM++66.47 ± 2.7862.29 ± 4.1363.31 ± 2.5164.96 ± 2.4866.46 ± 2.42
500TransLP++69.86 ± 2.6871.65 ± 2.5773.29 ± 2.5174.78 ± 2.4476.16 ± 2.31
1,000TIM++66.31 ± 2.5167.35 ± 2.4667.75 ± 4.4866.68 ± 4.7770.51 ± 4.68
1,000TransLP++70.06 ± 2.4771.80 ± 2.3673.52 ± 2.2475.24 ± 2.0876.81 ± 1.98
AllTIM++74.32 ± 0.6175.70 ± 0.6477.46 ± 0.7279.02 ± 0.4880.85 ± 0.34
AllTransLP++75.88 ± 0.5977.25 ± 0.5378.73 ± 0.5580.36 ± 0.4482.01 ± 0.35

2)Robustness under domain shifts. We conduct additional experiments in which the labeled support set and unlabeled query set are drawn from different data distributions. Specifically, we use ImageNet samples as the labeled support set and evaluate TransLP++ on four distribution-shifted benchmarks: ImageNet-Adversarial, ImageNet-V2, ImageNet-Rendition, and ImageNet-Sketch. We also report the zero-shot CLIP performance as a reference. The results averaged on three random seeds are summarized.

As seen in the following table, TransLP++ consistently improves over zero-shot CLIP on ImageNet-V2, ImageNet-Rendition, and ImageNet-Sketch, with the average accuracy increasing from 60.32% to 61.95%--64.09% as the number of shots increases from 1 to 16. ImageNet-Adversarial presents a much more severe domain shift, so TransLP++ performs below zero-shot CLIP and decreases from 45.75% to 43.03% as the number of shots increases, suggesting possible negative transfer under extremely large distribution discrepancies.

Compared with TIM++, TransLP++ achieves substantially better performance across all shot settings, particularly on ImageNet-Adversarial, ImageNet-Rendition, and ImageNet-Sketch, where TIM++ suffers from severe performance degradation. These additional experiments demonstrate the stronger robustness of TransLP++ under diverse domain-shift settings, while also revealing its limitation under extremely severe shifts such as ImageNet-Adversarial.

ShotMethodImageNet-AdversarialImageNet-V2ImageNet-RenditionImageNet-SketchAverage on ImageNet-V2/Rendition/Sketch
0Zero-shot CLIP47.7660.8673.9946.1060.32
1TIM++25.5361.3970.740.1044.08
1TransLP++45.7561.8374.1849.8561.95
2TIM++13.3862.4371.030.1044.52
2TransLP++**44.4762.3274.5850.2062.37
4TIM++1.0163.2647.890.1037.08
4TransLP++43.5463.0674.5150.7662.77
8TIM++12.0064.1447.990.1037.41
8TransLP++43.5364.4774.9350.8963.43
16TIM++19.5464.8824.560.1029.85
16TransLP++43.0365.0575.4351.8064.09

Response 3: Inductive vs. transductive inference

We thank the reviewer for raising this important point. In the inductive setting, query samples are predicted independently one by one, whereas transductive methods jointly exploit a batch of unlabeled query samples and their collective distribution. TransLP++ is specifically designed for transductive few-shot adaptation. Therefore, it requires batch access to query samples and is not directly applicable to strict online inference where query samples arrive sequentially one by one. We will clarify this distinction and the scope of our method in the revised manuscript.

Response 4: Clarification of the contribution

We appreciate the reviewer’s observation regarding the relationship between TransLP++ and existing methods. We agree that mutual information maximization and KL regularization have been explored in prior works such as TIM++ and TransCLIP, and we do not claim these individual objectives as novel contributions. Instead, our contribution lies in extending the exsiting linear probe framework (LP, LP++) to the transductive setting, where support supervision, collective query-set information, and CLIP semantic priors are jointly incorporated into classifier-level adaptation. This extension requires a unified optimization framework to adapt the visual classifier weights and text-blending coefficients while accounting for unlabeled query assignments.

Furthermore, the additional experiments in the rebuttal process have allowed us to further evaluate the strengths of TransLP++. Beyond consistently outperforming the state-of-the-art baseline TIM++ across different backbones and shot settings, TransLP++ demonstrates better robustness under various challenging settings, including limited query-set sizes, different query-set class distributions, and domain shifts. We believe these additional results provide a more comprehensive understanding of the effectiveness and practical applicability of our proposed TransLP++.

We sincerely appreciate your constructive questions, which have motivated us to investigate the robustness of our method. We hope that this discussion adequately addresses your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.