Dynamic-IsoCLIP: Cross-Modal Prior Spectral Routing for Instance-Aware CLIP Adaptation

We propose Dynamic-IsoCLIP, a supervised and parameter-efficient framework for adapting frozen CLIP representations to image-to-image retrieval and base-to-new classification. Existing IsoCLIP-style methods improve intra-modal alignment through a discrete topological surgery, namely fixed hard truncation of the CLIP projector spectrum, but apply the same manually selected mask to every input. To address this limitation, Dynamic-IsoCLIP introduces Cross-Modal Prior Spectral Routing (CMPSR), which replaces the binary mask with a continuous, image-conditioned mask over a spectral basis constructed offline from the frozen image and text projectors. The text projector supplies an offline cross-modal geometric prior, while a lightweight MLP router selects spectral directions using only image features at inference, avoiding costly online cross-modal attention. Prototype supervision breaks the routing symmetry, and representation distillation and spectral sparsity act as semantic anchors and training stabilizers for the adaptation. Experiments on eleven benchmarks show that Dynamic-IsoCLIP achieves the highest image-to-image retrieval mAP among the evaluated supervised baselines. Base-to-new evaluation further shows improved performance over the original CLIP reference on five datasets under the specified text-classifier protocol, indicating that adaptive spectral routing can improve downstream discrimination while retaining unseen-class recognition within the evaluated setting.