- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Qualification-Guided Synthetic Class Expansion (Q-SCE) to assess whether novel classes trained solely on synthetic data can reliably generalize to real-world inputs without accessing real examples of those classes. The method leverages historical class replay to predict real-world recall using four descriptors and Ridge regression, combined with multiple gating mechanisms for class admission. Experiments across visual and text classification benchmarks demonstrate improved recall prediction over a historical-mean baseline, although the admission benefits over simpler baselines remain limited.
Critical strengths
-
The paper is easy to follow, with a clearly defined problem setting and a well-described methodology.
-
The experiments are relatively comprehensive, covering both visual and text classification tasks across multiple benchmarks. The authors also provide comparisons with several baselines and ablation studies to evaluate the proposed method.
Critical weaknesses
-
The method relies on pretrained features to represent novel classes, but these classes may exhibit large intra-class variation or poor feature separability. It remains unclear how the method distinguishes inherent class diversity from representation limitations, and how these factors affect recall prediction and admission decisions.
-
Large-scale pretrained backbones such as DINOv2 may have already learned representations of many candidate classes. Thus, classes unseen by the downstream classifier are not necessarily novel to the backbone. The lack of experiments on genuinely unseen categories raises concerns about the generalizability of the proposed method.
-
The method relies on several manually specified thresholds for admission decisions, but their robustness across different datasets and backbones remains unclear. If these thresholds require careful tuning for each setting, the practical applicability of the method would be limited. More systematic sensitivity analysis is needed.
Questions for authors
-
The paper mainly relies on DINOv2 for visual experiments, with limited evaluation using ResNet-50. How sensitive is Q-SCE to the choice of visual backbone, especially when using weaker pretrained representations or backbones with different pretraining distributions?
-
How does Q-SCE handle novel classes with large intra-class variation or multi-modal distributions? Can the proposed descriptors adequately capture such diversity and still provide reliable recall predictions and admission decisions?
-
How sensitive are the admission decisions to the manually chosen thresholds? Can these thresholds generalize across different datasets and backbones without requiring additional tuning?
Rebuttal Discussion
Dear Reviewers,
The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.
Thank you very much for your time and valuable contributions!