- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Dynamic-IsoCLIP to replace the fixed hard spectral truncation in IsoCLIP with soft mask. The method constructs spectral basis from the frozen CLIP image and text projectors and trains a MLP to predict the contribution of each spectral direction, and only image features are required at inference time. The router is trained using class prototypes derived from frozen CLIP representations, together with a distillation loss and a mean-mask regularizer. The authors evaluate the approach on eleven image-to-image retrieval benchmarks, where test images serve as queries and training images form the gallery. The reported retrieval improvements are substantial. Additional experiments examine mask initialization, loss components, and base-to-new classification.
Critical strengths
- The motivation is reasonable. Soft mask is obviously better than fixed spectral basis. The proposed method is sample.
- The reported retrieval improvements are substantial.
Critical weaknesses
- Incremental contribution and insufficient isolation of dynamic routing. The contribution appears incremental, and the paper does not sufficiently establish that dynamic routing itself is the key source of the observed performance gains. There is no matched global learnable soft mask baseline using the same spectral basis and training objective.
- The baselines are too weak. The authors should train some method base on IsoCLIP, which is training-free. It is necessary to use supervision to train IsoCLIP as direct comparison. A fair comparison should include a supervised IsoCLIP baseline, where the fixed IsoCLIP mask is kept unchanged but the same prototype supervision, distillation loss, and sparsity regularization are used.
- Unclear contribution of the cross-modal prior. The additional value of the text projector in constructing the spectral basis is not sufficiently validated. Since the proposed spectral basis is derived from the frozen image and text projectors, the paper does not isolate how much the text information contributes beyond the image representation.
Questions for authors
None
3 comments
The rebuttal solved most of my concern, so I decide to raise my rating.
Dear Reviewer ZUjg,
Thank you very much for updating your score and for acknowledging our rebuttal efforts. We deeply appreciate the time and expertise you dedicated to evaluating our work.
Best regards.
Authors
W 1. The central contribution is image-conditioned spectral routing over a frozen cross-modal basis. A global learnable soft mask assigns the same spectral weights to every image, whereas Dynamic-IsoCLIP predicts an input-dependent mask, allowing individual images to emphasize different spectral directions. This distinction goes beyond replacing hard truncation with continuous weights.
To directly isolate this contribution, we added a global learnable soft-mask baseline using the same spectral basis, supervision, loss functions, teacher features, class prototypes, optimizer, and training schedule (no-prior initialization). The following results report retrieval mAP (%) as mean ± standard deviation:
Values report retrieval mAP. Dynamic-IsoCLIP values are the no-prior results reported in Table 2. Both methods optimize continuous spectral weights under the same objective, yet Dynamic-IsoCLIP achieves higher mAP by 40.89, 29.41, and 20.53 percentage points, respectively. Because a shared static vector rescales all images identically, it cannot adapt to sample-specific visual attributes and is easily over-constrained by the global sparsity penalty. These controls confirm the contribution of instance conditioning beyond global spectral-weight optimization.
W 2. A supervised fixed-mask baseline requires specifying which parameters remain trainable. In our architecture, the CLIP encoders, projection matrices, and cross-modal spectral basis are frozen, and the router is the only trainable representation component. Class prototypes are computed from frozen teacher features. Therefore, fixing the binary IsoCLIP mask leaves no trainable representation parameters: the task and distillation losses cannot change the representation, and the sparsity term is constant.
Under this architecture, label supervision yields two relevant controls:
Our central innovation is image-conditioned spectral routing over a frozen cross-modal basis. The global control learns one spectral-weight vector for the entire dataset, whereas Dynamic-IsoCLIP predicts input-dependent weights for each image. Thus, the comparison isolates the benefit of instance conditioning beyond label supervision or global optimization of spectral weights.
W 3. The text projector contributes structurally through the construction of the cross-modal spectral basis, rather than through online text inputs to the router. This structural prior is distinct from mask initialization: the no-prior variant removes the IsoCLIP truncation initialization but retains the cross-modal directions 𝐔.
To isolate how much the text projector contributes beyond image-only representations or arbitrary orthogonal coordinates, we compared 𝐔 with an image-only PCA basis and a random orthogonal basis under the exact same router and no-prior training protocol. The following results report retrieval mAP (%) as mean ± standard deviation:
The cross-modal basis exceeds image-only PCA by 4.40–6.42 percentage points and the random orthogonal basis by 2.56–3.43 points. Geometrically, 𝐟∈ℝ⁷⁶⁸ has a higher dimension than the projected embedding (d=512). Image-only PCA in ℝ⁷⁶⁸ prioritizes high-variance domain/background directions that lie outside the subspace spanned by the image projector 𝐖ᵢ. In contrast, 𝐔 , the image-side left singular directions of 𝐖ᵢᵀ 𝐖ₜ ,spans a cross-modal subspace within row(𝐖ᵢ) and aligns the coordinates with multimodal semantic axes, yielding a superior structural basis at zero additional inference cost.