REPurpose: From Prediction to Correction with Task-Specialized Representations

Base-to-novel adaptation of CLIP aims to learn task-specific knowledge from base classes while preserving generalization to unseen classes. Prior work has shown that adaptation-specific representations learned through inserted tokens can become highly task-specialized and perform poorly when directly used for novel-class prediction, limiting their role in novel-class inference. We revisit these representations from a different perspective and introduce REPurpose, a training-free inference framework that repurposes REP from prediction to correction: instead of asking REP to predict novel classes directly, REPurpose uses its sample-specific information to propose constrained corrections to CLS decisions and verifies them with frozen CLIP semantics. Across 11 base-to-novel benchmarks, applying REPurpose to MMRL++ improves its average Novel accuracy from 78.32% to 79.25%, achieving state-of-the-art performance among methods without knowledge distillation. Beyond MMRL++, the same correction principle consistently improves multiple adaptation hosts and transfer settings, and further extends to the autoregressive M2PT inference pipeline without modifying decoding.