acceptodds
Under review as a conference paper at ICLR 2027

Can Class Names Forecast Per-Class Accuracy for Data-Free CLIP Customization?

Abstract

Data-free customization trains a compact student from class names alone, but gives no per-class report when training finishes. We study a cheap way to estimate that gap without any real image of the target vocabulary. Our model forecast from name and inversion-based statistics targets student rather than teacher per-class accuracy on a data free customization system. We evaluate it under two protocols within and between classes. Scoring takes seconds once a synthetic pool exists; the estimate is coarse, Spearman trailing a labeled teacher oracle on global rank for three of four vocabularies under both protocols. But on compositional and fine-grained name lists it is even better than estimation from teacher with real labeled data: class-split worst precision beats the oracle on Flowers-102 and FGVC-Aircraft, and CUB-200 reaches leave-one-out global rank without target photographs. However, Our model cannot replace labeled evaluation when real images are provided, but it is a cheap per-class estimate that requires no real images.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
data-free distillation, vision-language models, open-vocabulary customization, per-class failure prediction, label-free forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_E1HnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks if per-class student accuracy can be predicted in data-free CLIP customization, before any real image of the target classes exists. The forecaster combines six cheap statistics, three from the class-name embeddings and three from the first synthetic inversions. It uses a linear fit with isotonic recalibration. The authors also argue that no label-free statistic can fully recover per-class accuracy. Experiments use a ViT-B/32 teacher and a ResNet-18 student on four vocabularies, under leave-one-vocabulary-out and a 70/30 class split. Global rank correlation is low and mostly below a labeled teacher oracle. However, the worst-30% shortlist beats the oracle on Flowers-102 and FGVC-Aircraft under class split.

Critical strengths
  1. Observation 1 shows that no label-free statistic can fully determine per-class accuracy, and the method is framed only as triage before labeling. This bounds what readers should expect, so the mixed results read as calibrated rather than overstated.
  2. Data-free customization ships a student with a mean accuracy but no per-class report and no test set. The forecaster reuses only by-products the pipeline already produces, the anchors and the first inversions, so it adds little cost.
Critical weaknesses
  1. Global ranking is weak. Spearman ranges from −0.06 to 0.46 leave-one-out, is near zero on Caltech-101 under both protocols, and trails the oracle on three of four vocabularies.
  2. One pipeline, one teacher–student pair, and four public vocabularies, so each leave-one-out fit uses only three vocabularies. It is unclear whether the rule compositional or fine-grained names transfers beyond this setup.
  3. The ablation is thin, with one seed, one split, and only Caltech and Flowers.
Questions for authors
  1. Were the trust conditions incompositional or fine-grained names fixed before seeing results on the four vocabularies?
  2. Were any settings chosen using the evaluation vocabularies? How sensitive are the leave-one-out results to them?
3 comments
AuthorsOpenReview ↗

Respond to Reviewer E1Hn

We appreciate the constructive feedback on calibrated wording, an explicit pilot scope, and clean separation of class-split vs leave one vocabulary out protocols. Below we add five-seed class-split evidence and tighten ranking claims, report alt-student runs under the same auditing recipe, and extend feature ablations to all four vocabularies.

W1 — Global rank correlation is weak and usually trails a labeled teacher zero-shot oracle.

Reply:

The revision scopes claims to triage before labeling: φ recycles customization by-products; it does not promise full per-class recovery from names alone. Abstract, Tables 2–3, and Limitations will separate class-split vs leave one vocabulary out Spearman instead of a single universal ranking story.

Class-split aggregates (ResNet-18, held-out ρ after isotonic, mean±SD):

VocabularyφTeacher ZS
Caltech-101+0.07±0.09+0.51±0.02
Flowers-102+0.42±0.11+0.62±0.05
CUB+0.52±0.05+0.79±0.04
FGVC+0.44±0.11+0.46±0.10

φ carries more rank signal on Flowers/CUB/FGVC than on Caltech; teacher per-class zero-shot dominates especially on CUB (+0.79). Under class split the paper already reports that worst-30% overlap for φ can exceed the teacher on Flowers and FGVC; new five-seed runs give φ overlap 58%±8% (Flowers) and 58%±11% (FGVC) vs teacher overlap reported in main tables. Those claims stay protocol- and vocabulary-specific—not global ranking superiority.

leave one vocabulary out Spearman in the submission remains mixed (roughly −0.06 to +0.46 across settings; near zero on Caltech under both protocols). We do not reinterpret leave one vocabulary out as strong forecasting in the rebuttal text. Sec. 4.8 and conclusion captions will separate leave one vocabulary out from class-split triage evidence.


W2 — A single teacher–student pipeline on four vocabularies leaves transfer unclear

Reply:

We widens evidence inside that stack—multi-seed runs on four vocabularies, alt student backbones on Caltech + Flowers, and five-seed feature ablations.

Transfer here means φ is student-agnostic. Features come from the frozen teacher + pool only; swapping the student does not re-fit φ. We held teacher, pool, prompts, KD, and auditing fixed and changed only the students on Caltech-101 and Flowers-102 (5 seeds × 3 alt backbones × 2 vocabularies, 30/30 complete). Held-out Spearman ρ (mean±SD):

StudentCaltech φFlowers φFlowers teacher ZS
ResNet-18 (paper)+0.07±0.09+0.42±0.11+0.62±0.05
ResNet-34+0.10±0.06+0.37±0.07+0.46±0.05
MobileNetV3-Small−0.01±0.06+0.30±0.04+0.55±0.06
EfficientNet-B0+0.17±0.02+0.47±0.07+0.61±0.09

On Flowers-102, φ stays moderately informative for every backbone (0.30–0.47); EfficientNet-B0 is on par with ResNet-18. On Caltech-101, φ stays near zero for all students including EfficientNet (+0.17±0.02) matching W1: coarse vocabulary under class split is hard, not a ResNet-18 quirk. Teacher per-class zero-shot on real images remains the stronger Spearman competitor on Flowers for every student (0.46–0.62). Main-text four-vocabulary φ and teacher numbers use the paper ResNet-18 student only; alt-backbone rows are an extra pilot check.


W3 — Feature importance and the value of all six statistics are not supported across seeds or vocabularies.

Reply:

The submission’s ablations used one seed and Caltech + Flowers only (Table 7). We reran the same 70/30 class split for five seeds × four vocabularies, comparing full six-feature φ, text-only (ρ, ℓ, η), and inversion-only (γ, σ, π). Held-out Spearman ρ (mean±SD):

Vocabularyfull φtext-only (ρ, ℓ, η)inv-only (γ, σ, π)
Caltech-101+0.11±0.04+0.11±0.05+0.25±0.10
Flowers-102+0.45±0.08+0.14±0.06+0.51±0.12
CUB+0.50±0.05+0.19±0.04+0.47±0.05
FGVC-Aircraft+0.49±0.07+0.56±0.05+0.48±0.09

Table 7 stays a single-seed illustration; multi-seed stability claims use this table only. On Flowers and CUB, inversion-only clearly exceeds text-only—the pool-geometry narrative in Sec. 4.6 is revised accordingly. On Caltech, inv-only (+0.25±0.10) can beat full φ (+0.11±0.04); we mention once in Limitations that the full six-feature mix can dilute signal on very coarse vocabularies, not in the headline claim.

Dear Reviewer E1Hn,

Thank you for your constructive feedback. As rebuttal closes soon, we would be grateful for any follow-up on our response.

We scoped global Spearman claims more carefully, separating class-split triage from leave one vocabulary out results; we added alternative student backbones as a pilot while keeping main tables on ResNet-18; and we extended feature ablations from one seed and two vocabularies to five seeds across all four datasets, with Section 4.6 aligned to that evidence.

We hope our rebuttal addresses your concerns. Please let us know if anything remains unclear.

Thank you for the reply, particularly the multi-seed evaluations and tests with alternative student backbones. These additions strengthen the empirical evidence, but the leave-one-vocabulary-out results remain mixed, and the new class-split experiments do not resolve my concerns about generalization to unseen vocabularies. It also remains unclear whether the proposed trust conditions and experimental settings were established independently of the evaluation results, and how sensitive the leave-one-vocabulary-out results are to these choices. Given these unresolved concerns, I will maintain my rating.

Reviewer_BXzNOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Summary: The paper studies whether class names and synthetic inversion statistics can forecast per-class student accuracy after data-free CLIP customization. The problem is practically relevant, but the current evidence does not sufficiently establish the contribution over existing approaches or the reliability of the proposed forecasts.

Critical strengths
  • The paper addresses an interesting problem: identifying potentially weak classes when real images of those classes are unavailable. Reusing synthetic images already produced during customization makes the proposed audit potentially inexpensive.

  • The authors explicitly acknowledge weak results and limitations in cross-vocabulary transfer, helping readers understand the method’s scope.

Critical weaknesses
  • The scope of ''data-free'' should be stated more consistently. Forecasting requires no real images or labels from the held-out target classes, but fitting the auditor (ϕ\phi) requires accuracy measurements obtained from labeled calibration data. Under class split, calibration uses 70% of the vocabulary’s classes; under LOVO, it uses classes from other vocabularies. Although Section 4 explains these requirements, the overall framing and Table 1 should clearly distinguish forecasting without real data from held-out classes from an entirely label-free auditing process.

  • Although (ϕ\phi) returns per-class scores and a ranking, worst-30% precision evaluates overlap between selected sets. This is appropriate for the stated shortlist objective, but it does not establish reliable ordering within that shortlist. The weaker results at tighter budgets in Table 6 further limit the evidence that the method can prioritize inspection when only a few classes can be examined. The demonstrated practical benefit is therefore primarily the recovery of a broadly weak subset.

  • The related-work distinction seems overstated. Robbins et al., Will It Zero-Shot?, already predicts per-class accuracy using text and generated images and describes leave-one-dataset-out calibration. The distinction here is forecasting the distilled student and reusing customization outputs. Establishing the value of this extension requires comparisons with adapted prior predictors and simple individual-feature baselines under comparable information budgets. The labeled teacher reference alone does not establish the proposed method’s advantage.

  • The feature interpretation is internally inconsistent. Section 4.6 attributes the strongest Flowers signal to compositional leakage and paraphrase rank, whereas Table 7 shows almost unchanged correlation when either is removed. Removing prototype alignment produces a substantial decrease, from 0.31 to 0.14. Moreover, these ablations cover only one seed/ split and two datasets, limiting broader conclusions about feature importance and the benefit of combining all six statistics.

Questions for authors
  • Can Table 1 distinguish calibration access under class split and LOVO explicitly? How sensitive is LOVO performance to the amount and diversity of labeled calibration data?

  • How reliable is the ordering within the predicted worst-30% subset? How are ties in isotonic predictions and empirical class accuracies handled?

  • Does the proposed predictor outperform adapted prior class-level predictors and individual inversion statistics when using the same synthetic pools and calibration labels?

  • Can the authors reconcile the feature-importance claims in Section 4.6 with Table 7 and assess whether the ablation conclusions hold across additional seeds and vocabularies?

4 comments
AuthorsOpenReview ↗

Respond to Reviewer BXzN (Part 1)

We appreciate the detailed comments on framing, worst-k overlap, adapted baselines, and ablation breadth. Below we revise Table 1 to separate features from calibration labels, report worst-30% overlap alongside Spearman, run 20 adapted Robbins and 20 multi-vocabulary ablations, and reconcile Sec. 4.6 with multi-seed.

W1 — The “data-free” scope is inconsistent: held-out classes lack real images/labels, but fitting the auditor requires labeled calibration data.

Reply:

The revision distinguishes held-out forecast inputs from calibration requirements. Table 1 will make the split explicit:

StageHeld-out target classesCalibration classes
Features φNames + frozen CLIP + inversion-pool stats only; no real photosSame feature rule
Labels for fitNo student labels from held-out classesYes realized per-class student accuracies (70% of classes, or LOVO vocabularies)
Teacher ZSNot an input to φUsed only as baseline, not trained into φ
Evaluation—Held-out student labels only to score forecasts

Under class split, 70% of classes in a vocabulary supply calibration labels; 30% are held out with frozen split. Under LOVO, calibration draws labels from the other three vocabularies in the pilot; the held-out vocabulary has no real images in φ. In both cases, held-out classes do not contribute labels to the ridge/isotonic fit.


W2 — Worst-30% overlap does not establish reliable ordering within the predicted weak shortlist.

Reply:

Worst-k overlap measures what fraction of the true worst-k classes appears in the predicted worst-k set. It does not certify correct ordering inside the shortlist. Secs. 4.7–4.8 will state that limitation explicitly.

Table 6 already shows gains shrink at tight inspection budgets. We will not extrapolate routing benefits beyond those curves. The practical claim is narrowed: φ helps recover a moderate fraction of weak classes when the goal is set-level triage, not fine-grained rank within the list.

Held-out worst-30% overlap (φ isotonic vs adapted Robbins, mean±SD):

VocabularyφRobbins adapt.
Caltech-10136%±8%47%±4%
Flowers-10258%±8%47%±13%
CUB51%±8%45%±3%
FGVC58%±11%29%±5%

φ wins on overlap on Flowers/CUB/FGVC; Robbins is higher on Caltech—so Spearman and overlap can disagree (Caltech: Robbins ρ +0.27 vs φ +0.07). We did not add within-shortlist Spearman in this window.

AuthorsOpenReview ↗

Respond to Reviewer BXzN (Part 2)

W3 — The contribution over Robbins-style predictors and simple feature baselines is not established under a comparable information budget.

Reply:

We reimplemented Robbins compound score (consistency + multimodal silhouette, α=4, λ=2.5) on the same CVLM datapool and prompts, same 70/30 isotonic calibration to student (not teacher) per-class accuracy 20/20 runs across four vocabularies. Spearman on held-out classes:

Vocabularyφ (E2)Robbins adapt.Teacher ZS
Caltech-101+0.07±0.09+0.27±0.04+0.51±0.02
Flowers-102+0.42±0.11+0.14±0.06+0.62±0.05
CUB (150)+0.52±0.05+0.44±0.04+0.79±0.04
FGVC+0.44±0.11+0.24±0.11+0.46±0.10

φ exceeds adapted Robbins on three of four vocabularies under this matched protocol—supporting value for pool-native inversion features beyond Robbins CLIP geometry on identical synthetic images. Teacher zero-shot remains the stronger global rank baseline on Caltech/CUB/Flowers when real-image teacher accuracies are available.

Besides Robbins and teacher, we compare simple feature blocks under the same CVLM pool, 70/30 split, and isotonic calibration as φ: text-only (stageA: ρ, ℓ, η), inversion-only (stageB: γ, σ, π), and full φ (paper6). This shows that text-side statistics alone are weak on fine-grained vocabularies while inversion-side pool geometry carries most of the rank signal. Held-out Spearman, five seeds (mean±SD):

Vocabularyfull φtext-onlyinv-only
Flowers-102+0.45±0.08+0.14±0.06+0.51±0.12
CUB+0.50±0.05+0.19±0.04+0.47±0.05

W4 — Ablation evidence is insufficient to support which features matter and whether combining all six statistics helps.

Reply:

Sec. 4.6 is rewritten so the feature narrative matches the multi-seed evidence below. Table 7 (one seed, Caltech + Flowers only) stays in the paper as a snapshot; any multi-seed stability claim uses the table here. Same 70/30 class split as W3; five seeds × four vocabularies. Held-out Spearman ρ (mean±SD):

Vocabularyfull φtext-only (ρ, ℓ, η)inv-only (γ, σ, π)
Caltech-101+0.11±0.04+0.11±0.05+0.25±0.10
Flowers-102+0.45±0.08+0.14±0.06+0.51±0.12
CUB (150)+0.50±0.05+0.19±0.04+0.47±0.05
FGVC-Aircraft+0.49±0.07+0.56±0.05+0.48±0.09

On Flowers and CUB, inversion-only clearly exceeds text-only; FGVC is text-strong in this pilot. The supplement adds leave-one-feature-out fits (each drops one of the six statistics) with held-out ρ and MAE; we do not report worst-30% overlap for ablation predictors (overlap stays on main φ class-split runs only).

Dear Reviewer BXzN,

Thank you for your detailed comments on scope, triage metrics, baselines, and ablations. With the discussion deadline approaching, we would value any further assessment of whether our rebuttal resolves your concerns.

We clarified the split between label-free inputs on held-out classes and calibration labels used only for fitting; we narrowed worst-k claims to set-level triage rather than ordering within the shortlist; we compared φ to an adapted Robbins predictor under the same synthetic pool, prompts, and 70/30 student calibration; and we added five-seed full, text-only, and inversion-only comparisons across four vocabularies, with multi-seed claims tied to those runs rather than the single-seed table alone.

We hope our response addresses your concerns. Please let us know if anything remains unclear.

I thank the authors for the detailed response. Further explanation partially solved my concern. The questions for the authors have not yet been answered. Therefore, I will rate my score as 2.

Reviewer_5Gy9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies forecasting per-class student accuracy in data-free CLIP customization without real images of held-out classes. The method combines six class-name and synthetic-inversion statistics through a linear predictor with isotonic recalibration. Experiments under cross-vocabulary and within-vocabulary protocols show ranking performance, with useful weakest-class identification on selected datasets.

Critical strengths
  • The paper addresses a practical problem. The proposed predictor is simple and efficient once the synthetic pool is available.

  • The evaluation considers different calibration protocols and inspection budgets.

Critical weaknesses
  • The generalization ability of student-specific forecasting remains unclear. All experiments use a CLIP ViT-B/32 teacher, and a ResNet-18 student. The authors are suggested to evaluate different student architectures and distillation settings.

  • The discussion of the features does not fully match the results. Sec. 4.6 says that two text-based features are the most important on Flowers-102. However, Tab.7 shows that removing the image–text similarity feature causes the largest drop in performance. The authors should explain this difference and provide more experiments to show the role of each feature.

  • The benefit of the calibration step is unclear. The experiments mainly measure how well classes are ranked, while the calibration step changes the score values without reversing their order. The authors should compare results with and without calibration and report the errors between predicted and actual accuracies.

Questions for authors

please see the weakness above.

3 comments
AuthorsOpenReview ↗

Respond to Reviewer 5Gy9

We appreciate the constructive feedback on student scope, the alignment between Section 4.6 and the ablations, and the role of isotonic calibration under rank-based metrics. Below we summarize new multi-seed runs and the revision items that address W1–W3.

W1 — Generalization across student architectures and distillation settings is not established.

Reply:

φ is unchanged across runs (frozen ViT-B/32 + CVLM pool features; no student checkpoint). We held teacher, pool, prompts, KD recipe, and class-split auditing fixed (split_seed=42, 70/30 ridge + isotonic) and added ResNet-34, MobileNetV3-Small, EfficientNet-B0 beside ResNet-18 on Caltech-101 and Flowers-102 (5 seeds × 3 backbones × 2 vocabularies). Held-out Spearman ρ (mean±SD):

StudentCaltech φFlowers φFlowers teacher ZS
ResNet-18 (paper)+0.07±0.09+0.42±0.11+0.62±0.05
ResNet-34+0.10±0.06+0.37±0.07+0.46±0.05
MobileNetV3-Small−0.01±0.06+0.30±0.04+0.55±0.06
EfficientNet-B0+0.17±0.02+0.47±0.07+0.61±0.09

On Flowers-102, φ stays moderately informative for every backbone (0.30–0.47). EfficientNet-B0 matches or slightly exceeds ResNet-18; MobileNet is lower but still positive. On Caltech-101, φ is near zero or weak for all students, including EfficientNet (+0.17±0.02 with very low seed variance). That pattern matches the main E2 table: coarse vocabulary + class split is hard for any student, not a ResNet-18 artifact.

Teacher per-class zero-shot accuracies on real images remain a stronger Spearman baseline on Flowers for every student (0.46–0.62). Changing the student shifts absolute student accuracies but does not remove the teacher rank competitor, so the revision will not claim “student agnostic forecasting beats the teacher.”


W2 — Text-side vs inversion-side feature importance on Flowers is stated inconsistently with the ablations.

Reply:

Section 4.6 is rewritten. It currently suggests that text-side features (compositional leakage ℓ, paraphrase rank η) dominate on Flowers; Table 7 shows the opposite for a single-seed Flowers split: removing prototype alignment π (mean image–text similarity in the pool) drops Spearman from about 0.31 to 0.14, the largest among one feature removals, while removing ℓ or η barely changes ρ.

To test stability beyond one seed and two datasets, we ran E8 ablations with the same 70/30 split: stageA (text-only: ρ, ℓ, η), stageB (inversion-only: γ, σ, π), and paper6 (full six-feature φ). Five seeds × four vocabularies . Held-out Spearman ρ (mean±SD):

Vocabularyfull φtext-only (A)inv-only (B)
Caltech-101+0.11±0.04+0.11±0.05+0.25±0.10
Flowers-102+0.45±0.08+0.14±0.06+0.51±0.12
CUB-200+0.50±0.05+0.19±0.04+0.47±0.05
FGVC-Aircraft+0.49±0.07+0.56±0.05+0.48±0.09

On Flowers and CUB, inversion-only stageB clearly exceeds text-only and is on par with or above full φ on some seeds—consistent with π and pool-geometry features driving rank signal when synthetic images carry fine-grained cues. On Caltech, all variants are weak; inv-only is highest but still modest. On FGVC, text-only is unusually strong in this pilot; we will report that as vocabulary-specific, not as “text always wins.”


W3 — The benefit of isotonic calibration is unclear when evaluation is dominated by rank metrics.

Reply:

The paper’s primary figures are Spearman ρ and worst-30% overlap both rank-based. The pipeline fits a linear ridge index on calibration classes, then isotonic regression maps that index to realized accuracies using only calibration labels. If the isotonic fit is strictly increasing on held-out scores, Spearman on held-out classes is unchanged vs the linear index (monotone transform preserves ranks). That is the case on Flowers/CUB/FGVC in our aggregates; Caltech shows larger seed variance on ρ.

Isotonic still matters for calibrated absolute predictions and MAE on held-out classes (practitioners who read predicted percentages, not only rank). New Table pairs ρ_linear, ρ_isotonic, and MAE. ResNet-18, five seeds, class split:

Vocabularyρ (isotonic)ρ (linear)MAE
Caltech-101+0.070±0.085+0.097±0.0500.251±0.010
Flowers-102+0.418±0.113+0.418±0.1130.175±0.017
CUB+0.516±0.048+0.512±0.0470.114±0.008
FGVC+0.444±0.108+0.433±0.1020.029±0.008

Ridge and isotonic are fit on 70% calibration classes only; held-out labels are used only to evaluate Spearman and MAE, never to refit φ or the isotonic map.

Dear Reviewer 5Gy9,

Thank you again for your thoughtful review. As the rebuttal period is nearing its end, we would greatly appreciate any further feedback on our response and planned revisions.

In response to your comments, we added multi-backbone class-split runs with φ fixed on the frozen teacher and inversion pool, ran five-seed ablations on all four vocabularies, revised Section 4.6 so the Flowers narrative matches the ablations (including the role of π and inversion-side features), and reported how isotonic calibration relates to Spearman versus calibrated accuracy and MAE.

We hope these additions address your concerns. We are glad to clarify anything still open.

Dear Reviewer 5Gy9,

We are writing again as the author–reviewer discussion closes in about 24 hours. We have not seen a reply to our rebuttal and would be grateful for any brief feedback while the window remains open.

Your main points were whether φ generalizes beyond a single student backbone, whether the Flowers feature narrative matches the ablations, and what isotonic calibration adds when evaluation is rank-based. In our response we addressed each of these: multi-backbone class-split runs with φ fixed on the frozen teacher and pool, a revised Section 4.6 together with five-seed text-only versus inversion-only ablations on four vocabularies, and a comparison of linear versus isotonic scores for Spearman and MAE. Relevant revisions and new tables appear in the updated manuscript and appendix (marked in blue where updated).

ICLR encourages reviewers to consider new information and revisions during the discussion phase. Our initial score may predate this material. If our rebuttal resolves your concerns, we would appreciate an updated assessment; if not, a short note on what still needs clarification would help us greatly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.