acceptodds
Withdrawn from ICLR 2027

Prompt-SSLC: A Unified Framework for Dual Prompt augmented Semi-Supervised Sequential Leader Clustering in On-the-Fly Category Discovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-the-fly Category Discovery (OCD) enables intelligent systems to perform real-time predictions while adapting to emerging classes without full offline retraining. However, deploying OCD in dynamic environments faces three critical bottlenecks: prototype drift under continuous streams, static feature bias toward base classes, and severe latency caused by sample-by-sample clustering. To address those challenges, we propose **Prompt-SSLC**, a unified framework that tightly couples robust streaming clustering, adaptive prompting, and routing. First, **Semi-Supervised Sequential Leader Clustering (SSLC)** anchors labeled class priors and utilizes a Distance-Aware update rule to maintain explicit decision margins and prevent prototype overlap. Second, a **Dual Prompt** architecture decouples global task context from local feature alignment; a **Task Prompt** guides category discovery, while an **Instance Prompt** dynamically recalibrates representations using neighbor prototypes to prevent base-class collapse without parameter tuning. Third, an **Open-Set Aware (OSA) Router** leverages uncertainty estimation to instantly classify high-confidence known samples, drastically cutting latency by only routing novel or ambiguous samples downstream. This cohesive integration of streaming clustering, adaptive prompting, and routing establishes a robust framework that balances stability and adaptability. Extensive experiments across generic and fine-grained benchmarks show that Prompt-SSLC achieves state-of-the-art accuracy while maintaining the low computational overhead required for real-time OCD deployment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
On-the-Fly Category Discovery, Dual Prompt Learning, Semi-Supervised Clustering, Sequential Leader Clustering, Open-Set Recognition, Streaming Data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wzNuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Prompt-SSLC for on-the-fly category discovery (OCD). The approach combines Semi-Supervised Sequential Leader Clustering with a Distance-Aware prototype update, dual prompt learning to adapt vision foundation models, and an Open-Set-Aware classifier to route known and potentially novel samples. Experiments on coarse-grained and fine-grained benchmarks show improvements over existing OCD baselines on several metrics.

Critical strengths

1.OCD is a meaningful problem in visual recognition, as models cannot be expected to encounter all possible categories during training.

2.The paper evaluates the method on several coarse- and fine-grained benchmarks, with ablation studies and additional experiments under domain shifts.

3.Initializing known-class prototypes with labeled data and updating them as unlabeled samples arrive is a reasonable design for OCD, as it allows the model to retain information about known classes while discovering new ones online.

Critical weaknesses

1. The motivation behind the three challenges is not sufficiently justified. The authors identify three challenges: prototype drift, static feature bias, and computational latency. I agree that prototype drift is a genuine issue in online clustering. However, I am not fully convinced by the second challenge. DINOv2 is pretrained on large-scale data through self-supervised learning, while novel classes in OCD/GCD are mainly simulated through dataset splits and may have already appeared during pretraining. The authors should clarify whether the claimed base-class bias comes from pretraining or subsequent supervised training, and provide direct experimental evidence. Regarding computational latency, hash-based methods such as SMILE are already designed for efficient online category discovery. The authors should justify the need for prototype-based clustering and clarify whether the complete method actually offers an efficiency advantage.

2. The reliability of OSA is insufficiently validated. OSA relies solely on Maximum Softmax Probability (MSP) to distinguish known and unknown classes, but its reliability in OCD remains unclear. The paper does not clearly explain how the OSA classifier is trained or how its confidence is calibrated. If a novel-class sample is incorrectly assigned to a known class with high confidence, it bypasses SSLC and cannot be discovered. Although ablation studies show improvements, the paper lacks direct analyses of Old/New routing errors, MSP threshold sensitivity, and the impact of incorrect routing on novel category discovery.

3. The novelty of the Dual Prompt design needs further clarification. The Task Prompt and Instance Prompt encode general task information and input-dependent information, respectively, which is similar to the G-Prompt and E-Prompt in DualPrompt (ECCV 2022). Although the authors introduce different training objectives and dynamic prototype-based prompts for OCD, the core novelty compared with DualPrompt should be further clarified.

4. The contribution of partial label masking is not sufficiently isolated. The authors introduce partial label masking, self-supervised contrastive learning, and Task Prompt simultaneously, but the ablation studies do not sufficiently isolate their individual contributions. In particular, it remains unclear whether the improvements come from the modified training objective or Task Prompt itself. More controlled experiments are needed to establish the independent contribution of Task Prompt to novel-class generalization.

Questions for authors
  1. Static feature bias. From my perspective, Challenge 2 is not sufficiently justified. As mentioned in my review, given that DINO is pretrained through self-supervised learning on large-scale data, it is unclear why its features would exhibit a bias toward base classes. This claim requires more rigorous experimental evidence. Otherwise, I do not think the paper should make such a strong claim about static feature bias.

  2. Computational latency. The authors should provide a more rigorous discussion and comparison of the computational costs of prototype-based clustering and hash-based methods. In particular, how does the proposed method compare with hash-based approaches in terms of inference latency and scalability as the number of discovered categories grows?

  3. Reliability of OSA. How reliable is OSA in practice? Compared with the classifier proposed in DebGCD (ICLR 2025), what are the advantages of OSA? Its reliability needs to be supported by more direct experiments, rather than ablation studies alone. It would also be helpful to provide failure cases to illustrate when OSA incorrectly routes novel-class samples to known classes and how such errors affect subsequent category discovery.

  4. Partial Label Masking. I believe Partial Label Masking has potential, but I have serious concerns about the current design. The authors divide the training classes into pseudo-old and pseudo-new classes, removing class-level supervision for the latter and replacing it with self-supervised contrastive learning. Although this introduces a new training objective, it may also lead to a potentially substantial trade-off: fewer classes contribute to supervised contrastive learning, which could directly weaken the model's ability to learn discriminative representations across classes. In my experience, reducing the number of classes involved in supervised training can cause a much greater performance drop than simply reducing the number of training images, because the model loses explicit supervision signals for those classes. However, the paper does not clearly specify the class split ratio or splitting strategy, nor does it analyze how this design affects the discriminative ability of known-class representations. How do the authors justify this trade-off, and what evidence supports the effectiveness of this design?

Reviewer_A67DOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Prompt-SSLC, a unified framework for On-the-fly Category Discovery (OCD). It integrates Semi-Supervised Sequential Leader Clustering (SSLC) with distance-aware prototype updates, dual prompting for feature adaptation, and an Open-Set Aware (OSA) classifier for efficient sample routing. Experiments on several coarse-grained and fine-grained benchmarks demonstrate competitive performance.

Critical strengths

The paper addresses the practical and challenging problem of On-the-fly Category Discovery. The proposed framework effectively integrates streaming clustering, dual prompting, and open-set recognition. The overall design is intuitive and well-structured. Extensive experiments on both coarse-grained and fine-grained datasets demonstrate competitive performance. Ablation studies support the effectiveness of individual components.

Critical weaknesses
  1. The framework mainly combines existing techniques, including sequential clustering, prompt learning, and MSP-based open-set recognition. The fundamental novelty beyond integrating these components remains relatively limited.

  2. The proposed Distance-Aware update is intended to mitigate prototype drift and maintain inter-class separation. However, the paper lacks a rigorous analysis of its stability or convergence, particularly under long-term streaming and distribution shifts.

  3. Insufficient scalability analysis. As novel classes continuously emerge, the prototype pool grows. The paper lacks systematic experiments investigating how the number of classes affects clustering accuracy, computational overhead, and prototype stability.

  4. Compared with recent methods such as AGE and TALON, the improvements are relatively modest on several benchmarks, and the proposed method underperforms on some datasets. Moreover, the additional forward pass required by Instance Prompt raises concerns about the trade-off between accuracy and computational efficiency.

Questions for authors
  1. Although DA shows empirical improvements, can the authors provide theoretical justification for its ability to maintain prototype separation and stability?

  2. How does the performance change as the number of discovered classes continuously increases?

  3. Can the authors provide computational efficiency comparisons with recent methods such as AGE and TALON?

  4. Does the use of a shared distance threshold affect performance when different novel classes exhibit substantially different intra-class variances?

Reviewer_NuDYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses On-the-fly Category Discovery, classifying a streaming unlabeled sequence into known and emerging categories without batch collection or retraining. Prompt-SSLC combines a distance-aware sequential leader clustering scheme that limits prototype overlap, a dual prompt mechanism injecting task- and neighbor-conditioned prompts into a frozen backbone, and an open-set-aware classifier that routes confident known-class samples away from clustering. Across coarse- and fine-grained benchmarks, the method improves on several clustering baselines and is competitive with a test-time-adaptive baseline, backed by component ablations and supplementary robustness analyses.

Critical strengths

The supplementary analyses probe the method along several axes — backbone/dataset scale, domain shift, modality extension — beyond the headline numbers, giving more confidence the gains are not specific to one configuration. The open-set routing targets a real, specific failure mode (prototype contamination by confident known samples) and is validated by a corresponding ablation showing exactly the expected effect. The attempt to separate old- and new-class evaluation via a modified matching protocol is a worthwhile methodological contribution independent of whether the clustering method itself is adopted.

Critical weaknesses

The open-set routing relies on Maximum Softmax Probability to decide whether a sample is a confidently-known class, but MSP is well documented in the open-set recognition literature to be poorly calibrated specifically on novel-class inputs, which tend to receive overconfident softmax scores precisely because the classifier has never seen anything like them and defaults to its most familiar class. This is a structural concern for OSA's design, not a tuning detail: if a genuinely novel-category sample receives a high MSP score by virtue of being out-of-distribution rather than because it resembles a known class, OSA would route it away from clustering and into the known-class pool, contaminating exactly the known-class prototypes the mechanism is meant to protect — the opposite of its intended effect. The paper does not report how often this specific failure occurs (e.g., the rate at which OSA routes true new-class samples into the known-class pool), and a distance-based or energy-based OOD score, which does not share MSP's overconfidence pathology, would be a natural alternative to compare against.

The sequential leader clustering mechanism spawns a new prototype whenever an incoming sample's distance to existing prototypes exceeds a threshold, which means the method's behavior is fundamentally sensitive to how well a single global distance threshold matches the natural intra-class versus inter-class distance structure of the dataset. On fine-grained benchmarks, where intra-class variation can be large relative to inter-class separation, a threshold tuned to avoid over-fragmenting coarse-grained classes could just as easily under-split fine-grained ones, or vice versa. The paper does not report whether the same threshold value transfers across its coarse- and fine-grained benchmarks or whether it was separately tuned per dataset; if the latter, this raises a question about how a true streaming deployment — where the eventual class granularity is not known in advance — would set this threshold without the benchmark-specific tuning available in evaluation.

The Instance Prompt is conditioned on neighbor prototypes, which means its signal quality depends on the clustering state already having formed a reasonable set of prototypes. Early in any stream, before enough samples have arrived to populate the prototype set, this conditioning signal is necessarily sparse or unreliable, creating a cold-start regime where the dual-prompt mechanism has the least information to work with precisely when streaming methods are typically most fragile. The paper's own appendix shows sensitivity to stream ordering, but this is better understood as a direct, structural consequence of conditioning on an online-built state rather than only a statistical-variance concern — different orderings produce different early-stream prototype configurations, which the Instance Prompt then conditions on for the rest of the stream. An experiment specifically measuring accuracy as a function of position in the stream (early versus late) would clarify how much of the method's performance is a cold-start artifact versus a stable steady-state behavior.

The combined Push-and-Rotation update (Table 4, "Both schemes") underperforms Rotation alone, and the paper attributes this to "Push interfering with Rotation's goal," but the exact combined update rule is never written out, so it is not possible to check whether this is a genuine interaction between the two correction terms or simply an artifact of how they happen to be combined (e.g., summed without reweighting versus applied in sequence). Since this ablation is used to justify the final design choice (Rotation alone), the underlying mechanism for why combining them hurts should be demonstrable from the equations, not only observed empirically.

Questions for authors

Could the authors clarify why the two protocols produce such different rankings against the strongest baseline, and discuss whether the modified protocol systematically favors their method? Could the authors double-check the CIFAR10 Old/New columns in the appendix given the inconsistency with the main table? It would help to see timing for the omitted baselines added to the efficiency comparison. Please also specify the exact combined update rule for the relevant ablation row.

What is the relationship between this paper and PromptCCD?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Area Chair and Reviewers

We thank the Area Chair and Reviewers NuDY, A67D, and wzNu. We respectfully push back on key concerns resolved by our submitted results and stream diagnostics, and scope our next revision.

Addressing Key Concerns

  1. OCD Novelty, Bias, & Paradigms

    • OCD coupling: Unlike PromptCCD (buffered EM retraining), DualPrompt (task IDs), DebGCD (transductive Dl∪DuD^l \cup D^u co-training with KK), or TALON (test-time backprop; batch_size=64 EMA skipping ≤1\le 1-sample updates) which all relaxing strict OCD, Prompt-SSLC is a closed-loop single-instance (batch_size=1), buffer-free, forward-only system: 768-d unit-sphere CLS prototypes (§3.3, Eq. 5; clarifying §4.1/Fig. 2: 4096-d ℓ2\ell_2 MLP head is offline-only) supply parameter-free PinstP_{\text{inst}} alongside 27,648-param PtaskP_{\text{task}} (within App. A's 55,296 prompt-slot budget), with super-additive OSA + Rotation gains (Table 4).
    • Bias & PtaskP_{\text{task}}: Base-class bias (Abstract/§1 vs. §3.3) stems from closed-set DlD^l fine-tuning (§4.1; Fig. II), not DINOv2 pretraining. Under 50:50 masking (LsupconL_{\text{supcon}} + LInfoNCEL_{\text{InfoNCE}}, clarifying §3.3), PtaskP_{\text{task}} beats LoRA/Adapter by +1.5–2.6% New (Table 5), and ∣Cold∣|C_{\text{old}}| DlD^l anchors give #1 Old on all 6 benchmarks (Tables 1–2).
  2. OSA, Warm-Up, Shared τ\tau, & Scaling

    • OSA & OOD: Pass-1 OSA trains a linear head on detached Pass-1 features (90:10 DlD^l split, qosa=50q_{\text{osa}}=50th percentile, zero DuD^u; clarifying §3.4/§4.1), adding +4.3% New / +2.2% Old on IM-100 (Table 4) while bypassing Pass 2/SSLC so misrouted novel inputs never contaminate prototypes (trace-verified). On Scars (6,144 images, 50% held-out known acceptance, no retraining), MSP (0.872 AUROC, 6.0% novel→old FPR), Energy (T=1T=1: 0.875, 5.7%), and nearest-anchor distance (0.878, 5.8%) match—refuting MSP as a bottleneck.
    • Warm-up, τ\tau, & scaling: Seeding ∣Cold∣|C_{\text{old}}| anchors at t=0t=0 prevents PinstP_{\text{inst}} cold start; across Scars quartiles (Q1–Q4, 1,536 images each), slice New rises +7.4–11.2 pts (Q1→Q3/Q4) across constrained views (+3.8%/+3.5% under Dirichlet α=0.1\alpha=0.1, Table XIII; ~8% vs. >14% SLC reassignments, App. A). Instead of a fixed τ\tau, §4.3's 95th-percentile DlD^l rule varies ≤1.2%\le 1.2\% All across 90th–99th percentiles (Table 7; per-cluster τ\tau: +0.3% All in SSLC, Table VIII), and on 683-class Herbarium-19 (45.0% All, Table 2), post-pruning K^=654.4±6.9\hat{K} = 654.4 \pm 6.9 (647–664, 5 stream-order seeds).
  3. Spherical Separation, Protocol, & Latency

    • Sphere & Both: Formalizing §3.2 on Sd−1\mathbb{S}^{d-1} with ℓ2\ell_2-normalization and λ/(n(1)+1)\lambda/(n_{(1)}+1) (λ=0.05\lambda=0.05, §4.1), seeding obeys (τ−δ)(\tau-\delta)-packing and Rotation widens margins via local Robbins–Monro contraction (+16.4%/+13.6% New over SMILE on ImageNet-R/C, Table X), whereas Push + Rotation (Both, Table 4) causes collinear double repulsion (μ(2)\mu_{(2)} to xtx_t, +1/(n(2)+1)+1/(n_{(2)}+1)) and unincremented μ(2)\mu_{(2)} drift.
    • Protocol & Table I (§4.1, App. C): Table I's CIFAR-10 Old/New columns are not swapped: DuD^u is 1/3 Old (12.5k) and 2/3 New (25.0k, Table 3), so All≈13Old+23New\text{All} \approx \frac{1}{3}\text{Old} + \frac{2}{3}\text{New} as global matching shifts mixed clusters to 2×-larger New. Our protocol (DINOv2 constrained in Table 1 vs. DINO unconstrained in Table I) does not favor Prompt-SSLC: TALON leads CIFAR-10 All in Table 1 (55.2% vs. 53.3%), while Prompt-SSLC leads Table I (68.2% vs. 65.0%), beats AGE on all 6 benchmarks (Tables 1–2), and is #1 in All on all fine-grained datasets under both protocols (Tables 2, II).
    • Latency & hashing: 768-d prototypes avoid hash quantization (+17.2%/+13.0% All over SMILE/PHE on CUB, Table 2; +12.5% on ImageNet-1K, Table IV) at O(K^td)O(\hat{K}_t d) cost (~3.1 MB FP32, 10310^3 classes). In Table 9 (RTX 3090, batch_size=1), single-pass (29s/530s, CUB/IM-100) is more than 20% faster than SMILE (37s/700s), and dual-pass (41s/760s) adds <11% over SMILE without test-time gradients or buffers.

Decision to Withdraw

While these results resolve the raised concerns, formal proofs and unified ablations belong in the manuscript rather than discussion comments. We are withdrawing to add:

  • Formal spherical proofs on Sd−1\mathbb{S}^{d-1}: (τ−δ)(\tau-\delta)-packing bounds on K^t\hat{K}_t, Rotation margin expansion, projective Robbins–Monro contraction, and Both variance/drift analysis; and
  • Protocol-unified ablations: multi-seed factorials across DlD^l objectives, pseudo-class splits, Pass-1 qosaq_{\text{osa}} curves, and matched-hardware batch_size=1 vs. minibatch efficiency controls.

Thank you again for your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.