- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks if per-class student accuracy can be predicted in data-free CLIP customization, before any real image of the target classes exists. The forecaster combines six cheap statistics, three from the class-name embeddings and three from the first synthetic inversions. It uses a linear fit with isotonic recalibration. The authors also argue that no label-free statistic can fully recover per-class accuracy. Experiments use a ViT-B/32 teacher and a ResNet-18 student on four vocabularies, under leave-one-vocabulary-out and a 70/30 class split. Global rank correlation is low and mostly below a labeled teacher oracle. However, the worst-30% shortlist beats the oracle on Flowers-102 and FGVC-Aircraft under class split.
Critical strengths
- Observation 1 shows that no label-free statistic can fully determine per-class accuracy, and the method is framed only as triage before labeling. This bounds what readers should expect, so the mixed results read as calibrated rather than overstated.
- Data-free customization ships a student with a mean accuracy but no per-class report and no test set. The forecaster reuses only by-products the pipeline already produces, the anchors and the first inversions, so it adds little cost.
Critical weaknesses
- Global ranking is weak. Spearman ranges from −0.06 to 0.46 leave-one-out, is near zero on Caltech-101 under both protocols, and trails the oracle on three of four vocabularies.
- One pipeline, one teacher–student pair, and four public vocabularies, so each leave-one-out fit uses only three vocabularies. It is unclear whether the rule compositional or fine-grained names transfers beyond this setup.
- The ablation is thin, with one seed, one split, and only Caltech and Flowers.
Questions for authors
- Were the trust conditions incompositional or fine-grained names fixed before seeing results on the four vocabularies?
- Were any settings chosen using the evaluation vocabularies? How sensitive are the leave-one-out results to them?
3 comments
Dear Reviewer E1Hn,
Thank you for your constructive feedback. As rebuttal closes soon, we would be grateful for any follow-up on our response.
We scoped global Spearman claims more carefully, separating class-split triage from leave one vocabulary out results; we added alternative student backbones as a pilot while keeping main tables on ResNet-18; and we extended feature ablations from one seed and two vocabularies to five seeds across all four datasets, with Section 4.6 aligned to that evidence.
We hope our rebuttal addresses your concerns. Please let us know if anything remains unclear.
Thank you for the reply, particularly the multi-seed evaluations and tests with alternative student backbones. These additions strengthen the empirical evidence, but the leave-one-vocabulary-out results remain mixed, and the new class-split experiments do not resolve my concerns about generalization to unseen vocabularies. It also remains unclear whether the proposed trust conditions and experimental settings were established independently of the evaluation results, and how sensitive the leave-one-vocabulary-out results are to these choices. Given these unresolved concerns, I will maintain my rating.
Respond to Reviewer E1Hn
We appreciate the constructive feedback on calibrated wording, an explicit pilot scope, and clean separation of class-split vs leave one vocabulary out protocols. Below we add five-seed class-split evidence and tighten ranking claims, report alt-student runs under the same auditing recipe, and extend feature ablations to all four vocabularies.
W1 — Global rank correlation is weak and usually trails a labeled teacher zero-shot oracle.
Reply:
The revision scopes claims to triage before labeling: φ recycles customization by-products; it does not promise full per-class recovery from names alone. Abstract, Tables 2–3, and Limitations will separate class-split vs leave one vocabulary out Spearman instead of a single universal ranking story.
Class-split aggregates (ResNet-18, held-out ρ after isotonic, mean±SD):
φ carries more rank signal on Flowers/CUB/FGVC than on Caltech; teacher per-class zero-shot dominates especially on CUB (+0.79). Under class split the paper already reports that worst-30% overlap for φ can exceed the teacher on Flowers and FGVC; new five-seed runs give φ overlap 58%±8% (Flowers) and 58%±11% (FGVC) vs teacher overlap reported in main tables. Those claims stay protocol- and vocabulary-specific—not global ranking superiority.
leave one vocabulary out Spearman in the submission remains mixed (roughly −0.06 to +0.46 across settings; near zero on Caltech under both protocols). We do not reinterpret leave one vocabulary out as strong forecasting in the rebuttal text. Sec. 4.8 and conclusion captions will separate leave one vocabulary out from class-split triage evidence.
W2 — A single teacher–student pipeline on four vocabularies leaves transfer unclear
Reply:
We widens evidence inside that stack—multi-seed runs on four vocabularies, alt student backbones on Caltech + Flowers, and five-seed feature ablations.
Transfer here means φ is student-agnostic. Features come from the frozen teacher + pool only; swapping the student does not re-fit φ. We held teacher, pool, prompts, KD, and auditing fixed and changed only the students on Caltech-101 and Flowers-102 (5 seeds × 3 alt backbones × 2 vocabularies, 30/30 complete). Held-out Spearman ρ (mean±SD):
On Flowers-102, φ stays moderately informative for every backbone (0.30–0.47); EfficientNet-B0 is on par with ResNet-18. On Caltech-101, φ stays near zero for all students including EfficientNet (+0.17±0.02) matching W1: coarse vocabulary under class split is hard, not a ResNet-18 quirk. Teacher per-class zero-shot on real images remains the stronger Spearman competitor on Flowers for every student (0.46–0.62). Main-text four-vocabulary φ and teacher numbers use the paper ResNet-18 student only; alt-backbone rows are an extra pilot check.
W3 — Feature importance and the value of all six statistics are not supported across seeds or vocabularies.
Reply:
The submission’s ablations used one seed and Caltech + Flowers only (Table 7). We reran the same 70/30 class split for five seeds × four vocabularies, comparing full six-feature φ, text-only (ρ, ℓ, η), and inversion-only (γ, σ, π). Held-out Spearman ρ (mean±SD):
Table 7 stays a single-seed illustration; multi-seed stability claims use this table only. On Flowers and CUB, inversion-only clearly exceeds text-only—the pool-geometry narrative in Sec. 4.6 is revised accordingly. On Caltech, inv-only (+0.25±0.10) can beat full φ (+0.11±0.04); we mention once in Limitations that the full six-feature mix can dilute signal on very coarse vocabularies, not in the headline claim.