acceptodds
Under review as a conference paper at ICLR 2027

REPurpose: From Prediction to Correction with Task-Specialized Representations

Abstract

Base-to-novel adaptation of CLIP aims to learn task-specific knowledge from base classes while preserving generalization to unseen classes. Prior work has shown that adaptation-specific representations learned through inserted tokens can become highly task-specialized and perform poorly when directly used for novel-class prediction, limiting their role in novel-class inference. We revisit these representations from a different perspective and introduce REPurpose, a training-free inference framework that repurposes REP from prediction to correction: instead of asking REP to predict novel classes directly, REPurpose uses its sample-specific information to propose constrained corrections to CLS decisions and verifies them with frozen CLIP semantics. Across 11 base-to-novel benchmarks, applying REPurpose to MMRL++ improves its average Novel accuracy from 78.32% to 79.25%, achieving state-of-the-art performance among methods without knowledge distillation. Beyond MMRL++, the same correction principle consistently improves multiple adaptation hosts and transfer settings, and further extends to the autoregressive M2PT inference pipeline without modifying decoding.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
vision-language models, CLIP, prompt learning, base-to-novel generalization, representation learning, training-free inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KEN5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that adaptation-specific representations in CLIP base-to-novel hosts are poor standalone novel-class predictors, but still encode complementary information. It proposed REP that is a training-free inference wrapper that repurposesMMRL/MMRL++, acting as a corrective evidence rather than a predictor. Evaluated on 11 base-to-novel benchmarks across four hosts (MaPLe, PromptSRC, MMRL and MMRL++) and three seets it achieve state-of-the-art performance.

Critical strengths
  1. Simple, well-motivated reframing from prediction to correction: a training-free plug-in that leaves the host and Base predictions untouched and is grounded in three diagnostics (linear probes with a shuffled-REP control, rank-locality of host errors with median ground-truth rank 3, and 15.9% frozen-CLIP recoverability of host errors) — consequential because each stage is motivated by evidence rather than asserted, and it applies to released checkpoints at 1.17–1.35× inference cost.

  2. Thorough, consistent empirical protocol: 11 datasets, 4 hosts, 3 seeds with paired seed CIs excluding zero for all hosts, CD/DG transfer, component ablation, shuffled-REP control (78.30 vs 77.40), fusion and reranking controls, crossing-order and semantic-resource factorials, and an honest EuroSAT failure analysis — consequential because the gains are small (~1 pp) and this breadth makes them credible.

Critical weaknesses
  1. The attribution of the gain to REP is weaker than claimed. Table 3/12 show that with standard class-name semantics REPURPOSE gives only +0.20 pp on MMRL (77.37 vs 77.17) and no gain on MMRL++ (78.30 vs 78.32 for the host), while frozen-CLIP reranking with CoPrompt descriptions alone already reaches 77.78 (MMRL). The improvement over the host therefore depends on LLM-generated CoPrompt descriptions, an external knowledge source most compared methods do not use; the fair baseline is description-based reranking/ensembling rather than the bare host — consequential because it weakens the headline narrative that repurposing REP drives the gain, and the fairness of the SOTA comparison.

  2. Several design choices look tuned to the benchmark: the energy threshold (ρ=0.95\rho = 0.95), the candidate rule, and especially the safeguard scale τver=0.07\tau_{ver}=0.07 (fixed for all datasets, activated only on MMRL++/EuroSAT). The paper mitigates this with sensitivity analyses, but the additive complexity is a barrier to practical adoption and raises the risk that gains do not transfer to less-curated settings.

Questions for authors
  1. Clarification. With class-name semantics, MMRL++ + REPURPOSE (78.30) does not exceed the host (78.32). How do you reconcile this with the claim that REP is the source of the corrective evidence, and how does REPURPOSE compare with simply giving the host the same CoPrompt descriptions (prompt ensembling or reranking)?

  2. Suggestion. Were the safeguard, τver,ρ\tau_{ver}, \rho, and the K rule chosen without access to test results? Please report Table 1 without the safeguard, and a version where all choices are made on a held-out split of the base classes.

Reviewer_bTB9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a training-free method to help the CLIP model can better adapt novel classes during inference. They find that the REP tokens can help provide assistant information for novel class prediction. The performances are all boosted for different baselines across multiple datasets.

Critical strengths

The authors reveal an interesting phenomenon: the REP tokens can also contribute to novel class prediction. Thus, they propose a training-free method to involve REP tokens to help provide novel information for prediction. The performance is also good and demonstrates the validation of the proposed method.

Critical weaknesses

The writing quality needs improvement to enhance clarity and readability. In particular, the mathematical notation used in the equations is not adequately explained. Moreover, the task definition is unclear. Based on the methodology section, I can only infer that the authors aim to address a task similar to open-vocabulary classification. However, the paper does not explicitly define the task or clearly explain its objectives.

Questions for authors
  1. The motivation presented in the Introduction is not sufficiently convincing. In Lines 51–53, the authors state that REP tends to weaken predictions for novel classes. However, they subsequently raise the possibility that REP could provide useful information for novel-class prediction, without offering a clear rationale for this shift in perspective. The connection between these two seemingly contradictory statements is not adequately explained. Moreover, the authors provide insufficient theoretical justification or empirical evidence to support this hypothesis, making the motivation behind the proposed method unconvincing.
  2. To support their motivation, the authors conduct experiments by training separate linear classifiers using CLS, REP, and REP+CLS representations. However, it remains unclear whether the superior performance of REP+CLS genuinely demonstrates the complementary benefits of combining these representations or simply results from differences in hyperparameter tuning. How do the authors ensure that the comparisons are fair and that the observed improvements are attributable to the combined representations rather than potentially more favorable hyperparameter settings? A more rigorous experimental analysis with consistent hyperparameter tuning protocols would help strengthen the motivation.
  3. I find the concepts of "rank-local" and "CLIP retains corrective potential" somewhat unclear. Regarding "rank-local," do the authors mean that the correct class may not always receive the highest predicted probability (Top-1) but could instead appear among the lower-ranked Top-K predictions? If so, this observation seems relatively straightforward and may not provide sufficient motivation for the proposed method. As for the claim that "CLIP retains corrective potential," I believe this is already well established in the literature, as numerous studies have demonstrated CLIP's strong zero-shot generalization ability, particularly for novel classes. Therefore, it remains unclear what new insights these observations provide or how they specifically motivate the proposed method. The authors should clarify these concepts and better distinguish their findings from existing knowledge.
  4. I find Eq. (3) difficult to understand due to the lack of clear explanations of the mathematical notations and their underlying meanings. First, what does (C) represent? Is it the number of classes present in the input image or the total number of classes in the dataset? Second, the authors state that the candidate-set size is determined by the effective support of (p_i). However, it is unclear how (p_i) is defined and incorporated into Eq. (3). The authors should explicitly explain the relationship between the probability distribution and the candidate-set size. Furthermore, assuming that the entropy is computed from a Softmax probability distribution over (C) classes, the exponential of the entropy is bounded between 1 and (C). If (C) represents the total number of classes, the exponential term cannot exceed (C). In this case, what is the purpose of taking the maximum with (C) in Eq. (3)? Would this operation not always return (C), making the adaptive candidate-set selection ineffective? The authors should clarify the mathematical formulation and justify the necessity of this operation.
  5. For Eq. (4), I just guess that B(x) has a size of 1xd, if d represents the channel size. Then, z_REP is also 1xd; b(x)^Tz_REP will become dxd, and then the final output is 1xd again. Is it correct? Please explain more about this part.
  6. Why does the step size required to reach its pairwise decision boundary along delta(x) follow Eq. (6)?
  7. It is a little confusing that z(gamma) is not used in the following equations. The original REP token and delta(x) are used, respectively, in Eq. (6). What is the role of z(gamma)?
  8. I find the proposed method difficult to follow without referring to the appendix. The main paper lacks sufficient explanations of several key components and mathematical formulations, making it challenging to understand the overall methodology. The authors should provide a more self-contained description of the proposed approach, including clearer explanations of the key operations and their underlying motivations, rather than relying heavily on the appendix for essential implementation details.
Reviewer_qoEVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REPurpose, a training-free inference framework that repurposes adaptation-specific representations (REP) to correct novel-class predictions in CLIP-based models. The method projects REP onto a candidate-specific semantic subspace to obtain a correction direction, then identifies the first reachable decision boundary under the original classifier. A frozen CLIP model is subsequently used to verify whether the proposed correction should be accepted. Experiments across 11 benchmarks demonstrate improvements over multiple CLIP adaptation methods, including an increase in MMRL++ novel-class accuracy from 78.32% to 79.25%, with additional evaluations on cross-dataset transfer, domain generalization, and autoregressive inference.

Critical strengths
  1. Practical and Training-Free Framework.
  2. Comprehensive Experimental Evaluation.
Critical weaknesses
  1. Limited Methodological Novelty. The proposed method primarily reuses existing adaptation-specific representations and combines standard operations, including semantic projection, linear decision-boundary analysis, and frozen CLIP verification. While repurposing these representations for inference-time correction is an interesting perspective, the technical contributions appear largely incremental rather than a substantial methodological advance.

  2. Insufficient Justification and Reliability of the Correction Mechanism. The paper provides limited justification for why projecting REP onto a candidate-defined semantic subspace should produce a meaningful correction direction. The linear-probe results demonstrate that REP contains novel-class information, but do not establish that the projected direction reliably leads toward the correct class. Furthermore, the first-cross rule does not guarantee a correct transition, and frozen CLIP verification can also accept incorrect corrections, as demonstrated by the EuroSAT failure case.

  3. Limited Generalizability Across VLM Architectures. The method assumes the availability of adaptation-specific representations, which are not universally present in VLM architectures. Moreover, MMRL's dedicated REP tokens and the visual prompt hidden states extracted from MaPLe and PromptSRC serve different original purposes, yet the paper provides limited justification for treating them as equivalent corrective signals. It remains unclear how the proposed framework would generalize to architectures without explicit adaptation-specific tokens or whether alternative hidden-state choices would be equally effective.

Questions for authors
  1. What justifies treating the projected REP representation as a meaningful correction direction? Since linear probing only demonstrates the presence of class-discriminative information, can the authors provide stronger evidence that the projected direction reliably leads toward the correct class, particularly when the first-cross rule and frozen CLIP verification may both introduce errors?

  2. Practical Applicability: Given the additional semantic resources, computation, and potential frozen CLIP forward pass, how practical is REPurpose for real-world inference? More importantly, since the safeguard relies on prediction statistics from the unlabeled test set, can the method operate reliably in online or single-sample inference settings where such dataset-level statistics are unavailable?

1 comment

Thank you to the authors for the detailed responses and additional experimental results. However, my main concerns regarding limited methodological novelty and insufficient justification of the correction mechanism remain largely unresolved. While the authors clarify the motivation and provide empirical support, the proposed framework still primarily combines existing representations with standard mathematical operations. Moreover, demonstrating that semantic projection provides the best approximation within a subspace does not sufficiently explain why the resulting direction should lead to reliable corrections. The generalizability beyond prompt-based architectures also remains unclear.

Overall, I appreciate the authors' efforts during the rebuttal, but I do not find the responses sufficient to substantially change my original assessment. I therefore intend to maintain my original rating.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your efforts so far. As we are halfway through the discussion period, I would like to kindly encourage you to review the authors' rebuttals and engage in further discussions, particularly if the authors have addressed your questions or concerns.

Thank you for your time and contributions!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.