acceptodds
Under review as a conference paper at ICLR 2027

Qualification Before Expansion: Auditable Synthetic-Only Class Addition

Abstract

Synthetic data can make a new class trainable before any real example exists, but trainability does not show that the expanded classifier will recognize the class in deployment. We formulate pre-deployment qualification: deciding, from historical classes and candidate synthetic data alone, whether the evidence supports admitting a class. Qualification-Guided Synthetic Class Expansion (Q-SCE) replays each historical class as a pseudo-missing candidate under the same information budget, tests whether candidate-observable descriptors predict revealed real recall better than a historical-mean reference, and abstains when they do not. We froze one instantiation, a four-descriptor Ridge qualifier with task, support, and retention gates and an empirical lower-score tier, before opening any candidate target data on DomainNet Clipart→Real (45 candidates) and Stable-Diffusion→Caltech-256 (101). Candidate-recall MAE falls from .259 to .224 and from .138 to .111. Admission gains are small. On Caltech the policy admits 78 classes with one failure, a detector-p10≥ .50 screen does as well, and CLIP zero-shot already succeeds on 99 classes; on DomainNet the policy admits one class and abstains on the rest. The results support replay-based qualification with frozen abstention. They do not show a better ranker than simple scores or a guarantee on the failure rate among admitted classes.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
synthetic-only class expansion, pre-deployment qualification, synthetic-to-real transfer, transferability estimation, selective prediction, abstention, domain shift, deployment reliability, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_w8BXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Qualification-Guided Synthetic Class Expansion (Q-SCE) to assess whether novel classes trained solely on synthetic data can reliably generalize to real-world inputs without accessing real examples of those classes. The method leverages historical class replay to predict real-world recall using four descriptors and Ridge regression, combined with multiple gating mechanisms for class admission. Experiments across visual and text classification benchmarks demonstrate improved recall prediction over a historical-mean baseline, although the admission benefits over simpler baselines remain limited.

Critical strengths
  1. The paper is easy to follow, with a clearly defined problem setting and a well-described methodology.

  2. The experiments are relatively comprehensive, covering both visual and text classification tasks across multiple benchmarks. The authors also provide comparisons with several baselines and ablation studies to evaluate the proposed method.

Critical weaknesses
  1. The method relies on pretrained features to represent novel classes, but these classes may exhibit large intra-class variation or poor feature separability. It remains unclear how the method distinguishes inherent class diversity from representation limitations, and how these factors affect recall prediction and admission decisions.

  2. Large-scale pretrained backbones such as DINOv2 may have already learned representations of many candidate classes. Thus, classes unseen by the downstream classifier are not necessarily novel to the backbone. The lack of experiments on genuinely unseen categories raises concerns about the generalizability of the proposed method.

  3. The method relies on several manually specified thresholds for admission decisions, but their robustness across different datasets and backbones remains unclear. If these thresholds require careful tuning for each setting, the practical applicability of the method would be limited. More systematic sensitivity analysis is needed.

Questions for authors
  1. The paper mainly relies on DINOv2 for visual experiments, with limited evaluation using ResNet-50. How sensitive is Q-SCE to the choice of visual backbone, especially when using weaker pretrained representations or backbones with different pretraining distributions?

  2. How does Q-SCE handle novel classes with large intra-class variation or multi-modal distributions? Can the proposed descriptors adequately capture such diversity and still provide reliable recall predictions and admission decisions?

  3. How sensitive are the admission decisions to the manually chosen thresholds? Can these thresholds generalize across different datasets and backbones without requiring additional tuning?

Reviewer_KAVnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper formulates synthetic-only class addition as a pre-deployment qualification problem, where candidate real data are unavailable and historical classes are replayed as pseudo-missing trials. It proposes Q-SCE, a frozen Ridge-based recall qualifier with task/support/retention gates and an empirical lower-score tier for abstaining class admission. The method is evaluated prospectively on DomainNet and Caltech-256, with additional retrospective and protocol-isolated analyses.

Critical strengths

The experimental protocol is rigorous and the results are verifiable, i.e., matched information budget, pre-registration-style freezing, gains with statistical significance.

Critical weaknesses
  1. On DomainNet, the proposed L90 policy admits only 1 out of 45 candidate classes; on Caltech-256, it admits 78 classes, while a simple detector-p10 threshold admits 83 classes, with one failure each. Moreover, CLIP zero-shot already succeeds on 99 out of 101 classes. These outcomes raise doubts about whether the proposed framework offers meaningful benefits over much simpler approaches and severely limit its practical utility.

  2. The manuscript strongly resembles the output of an automated research-and-writing workflow rather than a focused, hypothesis-driven scientific investigation, although an AI Use Statement is provided. The paper contains numerous artifacts of iterative development, and several components were added or corrected after the reveal. The writing is unnecessarily verbose and defensive, repeatedly emphasizing limitations, exceptions, and qualifications of individual findings. The accumulation of loosely connected experiments and post-hoc explanations makes the paper difficult to follow and far from what a good paper should look like.

Questions for authors

At this stage, I have no specific questions whose answers could materially affect my assessment.

Reviewer_WxzmOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Synthetic data can enable learning new classes before real examples become available. However, synthetic trainability does not establish whether a class will satisfy deployment requirements without observing its real distribution. The paper proposes Q-SCE, which replays historical classes as pseudo-missing candidates and combines recall prediction, qualification gates, and empirical admission tiers under frozen policies. Prospective DomainNet and Caltech evaluations reduce recall-prediction MAE from .259 to .224 and .138 to .111, respectively, but demonstrate limited additional admission value.

Critical strengths
  1. The distinction between successful synthetic transfer and evidence supporting pre-deployment admission identifies a meaningful evaluation problem.
  2. Class-level replay and frozen pre-reveal policies provide a careful protocol that respects the absence of candidate-real observations.
  3. Extensive diagnostics transparently distinguish prediction, ranking, calibration, and admission outcomes while reporting negative results and retrospective analyses.
Critical weaknesses
  1. Added admission value remains weak: the diagnostic .50-screen admits 83 Caltech classes with one failure versus Q-SCE’s 78 with one, while DomainNet yields only one L90 admission.
  2. The central MAE improvement is measured against the historical mean without a historical-median baseline, although the median minimizes absolute-error risk among constant predictors.
  3. The task gate’s benefit is unproven: it prevents no observed failures, while removing it adds 11 successful admissions across the CLINC regeneration runs.
  4. Cross-class generalization remains insufficiently tested beyond random/hash visual splits, since there is no prospective semantic visual holdout and CLINC regeneration reuses previously revealed intents.
  5. Although explicitly acknowledged, the empirical tiers provide no selected-set risk control, limiting their operational interpretation as qualification decisions.
Questions for authors
  1. Do the prospective MAE gains and task-gate outcomes remain favorable against a historical-median reference using identical replay splits?
  2. What operational objective justifies the task gate’s lost successful admissions when its error-based criterion does not demonstrably improve admission decisions?
  3. Can screening-plus-audit label costs be compared with simpler policies at matched admission coverage and false-admission risk?
  4. How sensitive are the frozen policy’s prediction and admission results to systematically different candidate classes rather than randomly held-out classes?

Comments on the paper

2 comments
Area_Chair_UN1rOpenReview ↗

Rebuttal Discussion

Dear Reviewers,

The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.

Thank you very much for your time and valuable contributions!

Dear Area Chair UN1r,

After reading the rebuttal, I maintain my rating of 2. I appreciate the clarifications on the novelty setting and threshold reuse. However, the evidence does not directly address reliability under substantial intra-class variation, and the weaker-backbone results provide limited support for practical admission utility. These concerns remain sufficient to leave my overall assessment unchanged.

Best regards,

Reviewer w8BX

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.