- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
- Objective: The manuscript addresses Few-Shot Open-Set Recognition using vision-language models by three main modules: 1) constructing useful representations for unknown (unseen) categories, 2) regularizing the prompt embedding space, and 3) making decisions beyond single-score response confidence.
- Methodology: Keeps both textual and visual CLIP encoders completely frozen, then introduces a 3-stage framework:
- SDSA (Syntax-Decoupled Semantic Anchor Synthesis): Factorizes 1,000 auxiliary label token embeddings into a sparse dictionary and composes synthetic unknown semantic anchors using learnable composition weights with Gaussian perturbation.
- USAR (Unknown-Aware Semantic Anchor Regularization): Trains known prompts and unknown anchors using joint unknown label smoothing, and intra-unknown dispersion.
- CEI (Conflict-Aware Evidential Inference): Maps similarities via Softplus function to in-distribution and out-of-distribution evidence, constructs Dempster-Shafer masses with prior ignorance, and derives a decision score.
- Key Findings: Evaluated across 8 standard FSOR benchmarks under the target 16-shot protocol (CIFAR-10, C10+10, C10+50, CIFAR-100, TinyImageNet, TinyImageNet-Hard, MiniImageNet, ImageNet-200) using CLIP ViT-B/16. Outperforms the baseline UTL across all 8 settings in open-set AUROC (+1.35 points average gain; max +3.58 on C10+10) while achieving top closed-set classification accuracy on 6 of 8 benchmarks with 13,192 trainable parameters.
Critical strengths
- The manuscript provides an end-to-end integration of structured unknown anchor synthesis, geometric prompt-space regularization, and evidential belief-plausibility score discounting under a fully frozen CLIP backbone. The mathematical derivation in Eq. 16 & 17 proves strict monotonicity, offering a principled alternative to heuristic single-confidence thresholding.
- The extensive results shown that the proposed framework consistently outperforms existing methods across 8 standard benchmarks with low parameter overhead (13,192 parameters). The qualitative t-SNE visualizations (Figure 3) and score density distributions (Figure 4) confirm that USAR module effectively disperses unknown anchors throughout open space while CEI separates known and unknown score distributions.
Critical weaknesses
- There is a problematic issue in Figure 1 (under the list of contributions); it appears without any accompanying figures.
- The paper provides a progressive ablation (Table 4) rather than a full factorial ablation, failing to isolate the individual contributions of each element forming the overall objective (loss function), making it hard to prove whether both hinge losses are equally necessary. Furthermore, Appendix B (Table 7) reveals that varying the number of synthesized unknown anchors from 1 to 100 changes C100 AUROC by a negligible 0.01% (87.84% vs 87.85%), raising concerns that generating 50 dictionary-composed anchors adds unnecessary architectural complexity over a simpler negative prompt vector.
- Main evaluations are limited to a single visual backbone (CLIP ViT-B/16). Additionally, benchmark results in Tables 1 and 2 represent single point estimates without reporting multi-seed standard deviations or statistical significance tests (e.g., paired t-tests or Wilcoxon signed-rank tests) across support set samplings. Additionally, it lacks of evidence for the contribution of CEI, while the paper explicitly concedes that CEI does not perform Dempster's combination rule, nor does it provide calibrated epistemic uncertainty.
Questions for authors
- Can you provide a full factorial ablation isolating SDSA, , , and CEI individually? Specifically, what is the AUROC impact of vs. independently, and how does CEI perform on an unregularized prompt space?
- Given that anchor achieves virtually identical AUROC to anchors on C100 (87.84% vs 87.85% in Table 7), what is the technical justification for fixing and building a 100-atom sparse dictionary?
- Does SAEC generalize to CNN-based CLIP backbones or larger Vision Transformers?
- Can you report mean AUROC/ACC and standard deviations across at least 5 independent random 16-shot support set splits?
2 comments
Thanks for your response. I decided to keep my rating.
Response to Reviewer 3wQY
We thank the reviewer for recognizing SAEC’s framework, parameter efficiency, and mathematical analysis. We have expanded the ablations, added multi-seed statistics, and clarified our conclusions.
Q1: Figure 1 presentation.
A1: We corrected the figure insertion and placement. Figure 1 is now fully displayed and explicitly referenced in the text.
Q2: Component contributions, factorial ablation, and CEI without geometric regularization.
A2: The revision includes progressive and targeted ablations:
Separation alone improves AUROC by 0.38/0.70 points on C10+10/T200; dispersion alone improves it by 0.21/0.86 points. Combining them gives gains of 0.87/1.28 points and the highest mean ACC. Their contributions differ across datasets; these results support using both without claiming equal necessity.
CEI: Table 7 compares CEI with Max-Known, Evidence-Margin, Evidence-Ratio, ID-Belief, and Pignistic without additional representation learning. CEI achieves the highest reported mean AUROC on all four settings, averaging 90.85%, versus 90.35% for ID-Belief. Alternative scores use four-run statistics and CEI uses five-run statistics, making this supplementary rather than fully seed-matched evidence.
These studies do not constitute a full factorial analysis across all components, and we do not claim to isolate every interaction. Base removes both geometric losses but retains unknown label smoothing and CEI. It therefore evaluates the framework without geometric regularization, but does not isolate CEI’s gain over alternative scores in that space. This specific matched comparison remains untested.
CEI uses a belief–plausibility product for open-set scoring; it neither applies Dempster’s combination rule nor claims calibrated epistemic uncertainty. Table 9 additionally shows higher mean residual ignorance mass for incorrect, low-confidence, and OOD samples relative to their respective comparison groups. This characterizes behavior rather than establishing calibration.
Q3: Why use 50 anchors and a 100-atom dictionary?
A3: We agree that Table 11 shows little AUROC benefit from increasing anchor count on C100: Nu=1,50,100 yield 87.84%, 87.85%, and 87.61%. However, 50 anchors give higher ACC than one (88.42% vs. 87.72%). We retain 50 as a shared configuration considering both metrics, without claiming universal superiority or an optimal count.
Dictionary size K=100 specifies the reusable basis, whereas Nu=50 specifies the synthesized prompts. Table 6 evaluates dictionary-based construction but does not establish the necessity of exactly 100 atoms. We keep K=100, α=1, and σ=0.1 fixed; dictionary-size sensitivity remains unevaluated.
Q4: Generalization to other CLIP backbones.
A4: SAEC operates through prompts, anchor embeddings, and image–text similarities while freezing both encoders. It is structurally compatible with CLIP architectures providing aligned embeddings and a compatible text-prompt interface. However, experiments only evaluate ViT-B/16. We distinguish this compatibility from demonstrated cross-backbone generalization, which remains future work.
Q5: Five independent runs and uncertainty reporting.
A5: Tables 1–2 now report mean AUROC/ACC and sample standard deviations across seeds {0,1,2,3,4}. Each seed controls class partitioning, 16-shot support sampling, and trainable-parameter initialization. SAEC and UTL use identical partitions and support sets within each seed.
SAEC improves mean AUROC on all eight settings, averaging 1.35 points, with a maximum gain of 3.58 points. All 40 seed-wise AUROC differences are positive. Mean ACC exceeds UTL on all eight settings and ranks highest among compared methods on six. These results support consistency across sampled conditions; we do not claim statistical significance without a paired test.
We thank the reviewer again for helping clarify the component evidence and evaluation scope.