acceptodds
Under review as a conference paper at ICLR 2027

Similarity-Aware Prompt Routing for Data-Efficient Continual Learning

Abstract

Continual Learning (CL) enables models to adapt to new data after deployment. Recent approaches increasingly use representations from pretrained or foundation models to reduce catastrophic forgetting as new tasks arrive. Yet, recent approaches often assume abundant per-task data or strictly non-overlapping tasks. In this paper, we address a more realistic setting where tasks have limited data and may contain overlapping concepts. In this setting, knowledge transfer must be carefully controlled to overcome low-sample noise while mitigating interference between tasks. To address this, we propose Similarity-Aware Prompt Routing (SPR), an adaptive routing framework for prompt-based continual learning that progressively builds task-similarity awareness over a pool of prompts. SPR relies on two key components: incremental global pooling, which mitigates prompt association noise through gradual prompt introduction, and instance-wise prompt masking, which separates incoming samples into those aligning with current prompts and those requiring new ones. Experiments across varying data volumes and inter-task similarities demonstrate that our method actively prevents negative transfer, enhances sample efficiency, and is broadly applicable.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Prompting, Pre-Trained Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pbxaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses rehearsal-free prompt-based continual learning (CL) in settings where per-task data is scarce and concepts may overlap across tasks. The authors argue that task-specific prompts cannot leverage knowledge from related tasks, while global prompt pools introduce routing noise that leads to negative transfer. They propose Similarity-Aware Prompt Routing (SPR), which has two components: (1) incremental global pooling, where new prompts are added for each task while existing prompts remain available for both training and routing; and (2) instance-wise prompt masking, where each training sample is routed either to the existing prompts (in-distribution) or to the new prompts (out-of-distribution) based on a Relative Mahalanobis Distance estimated online in the frozen backbone's feature space.

Critical strengths

The problem setting is practical and the motivation is sound. I agree with the observation in Fig. 1 that similar or recurring concepts reappear over time (e.g., Task 1 and Task t+1t+1 sharing the same higher-level concept), and few-shot CL is itself a highly practical topic. The hypothesized tension between knowledge sharing and routing noise is also reasonable and provides a clear framing for the method.

Critical weaknesses
  1. The method shows almost no improvement over the state of the art, and the central claim that it performs better under limited data is not supported by the experiments.

    • In Table 4, SPR trails HiDe-Prompt on all three metrics for Split CIFAR-100, and on Split ImageNet-R the FAA and FFM are essentially unchanged; the only notable gain is in CAA (+1.6). Table 2 shows the same pattern.
    • The comparison in Fig. 4 is against the authors' own routing variants rather than against state-of-the-art methods, and the gain over global pooling appears roughly constant across data proportions. It is therefore unclear that the method is particularly effective when data is scarce.
    • Table 3 shows gains on the protocols with task overlap, but the main text never discusses Table 3, so the reader cannot tell what insight these results are meant to convey.
    • If the paper targets data scarcity, it should be framed and evaluated as few-shot CL, which it is not. The set of baselines is also too small: there is no experimental comparison with ConvPrompt (CVPR'24), LAE (ICCV'23), OVOR (ICLR'24), or other prompt-based CL methods beyond HiDe-Prompt (NeurIPS'23).
  2. The method is underspecified, and several of its key premises are asserted without evidence, which makes its novelty difficult to assess.

    • Masking is defined only for training; the paper does not explain how routing is performed at inference time.
    • The construction of the historical summary that drives the binary mask is only sketched in Sec. 5.3. The handling at t=0t=0 (when no Ht−1\mathcal{H}_{t-1} exists), how the pooled covariance Σ\Sigma is derived from the per-class Σc\Sigma_c, and the value of ϵ\epsilon are all unspecified.
    • Since all prompts remain routable and trainable, old prompts are apparently not frozen, but the rationale for this design choice (e.g., to exploit overlap) is not given.
    • The paper states that restricting the pool size forces reuse, but the relevant quantity is not the nominal pool size but the number of prompts that have actually received updates. Since the pool size is itself a hyperparameter, this argument does not hold as stated.
    • The claim that orthogonality diminishes as data accumulates is unsupported, and it conflicts with the paper's own premise that data remains scarce.
    • There is no evidence that routing noise causes negative transfer when a global pool is optimized end-to-end, as in CODA-Prompt. Table 1 measures the separability of learnable keys, task centroids, and a task classifier, but it does not cover end-to-end optimized pools such as CODA-Prompt's attention-based aggregation, nor does it connect low AUC to actual negative transfer.
    • There is no preliminary analysis showing that overlap actually arises under similarity-based prompt selection, and no evaluation showing that routing error is actually reduced by SPR.
    • The main results are built on HiDe-Prompt, but the ablations use S-Prompt++. For the ablations to inform the main results, they should be run on HiDe-Prompt as well. Based on the algorithm and implementation details, the method appears to be HiDe-Prompt with additional optimizations, and I could not identify a clearly novel mechanism.
Questions for authors
  1. How are prompts selected at inference time without a task ID? Is RMD also used at test time?
  2. How is the historical summary Ht−1\mathcal{H}_{t-1} handled at t=0t=0, how is the pooled covariance Σ\Sigma derived from the per-class Σc\Sigma_c, and what value of ϵ\epsilon is used?
  3. Why are old prompts not frozen? Is this an intentional choice to exploit overlap?
  4. Is there evidence that routing noise causes negative transfer in end-to-end optimized global pools such as CODA-Prompt? Table 1 does not include CODA-Prompt's attention-based routing. Can you provide a preliminary analysis showing that overlap actually occurs under similarity-based selection?
  5. Can you show, in the evaluation, that SPR actually reduces routing error, beyond the AUC reported in Table 1?
  6. Can you provide the data-proportion experiment of Fig. 4 with state-of-the-art baselines such as HiDe-Prompt? In the current results, the gain appears independent of the data proportion.
  7. Can you provide the ablation results on the HiDe-Prompt base?
  8. Prompts compress past information into parameters rather than storing raw data, so it is accurate to say the system uses little memory, but not that the amount of data is small. Is there any observation or data showing that adaptation works well specifically when the number of prompts is small?
  9. Please clarify the rationale for restricting the pool size. What is actually being controlled: the pool size or the number of prompts that receive updates?
  10. Which OOD threshold is used, q=0.99q=0.99 (main text) or q=0.95q=0.95 (Appendix)? Please also specify the number of prompts added per task, top-KK, the prompt length, and the layers into which prompts are inserted.
  11. Please add a discussion of Table 3 to the main text, including what explains the gains observed there.
  12. The index range in the problem formulation (Eq. 1) is inconsistent with the task indexing.
  13. The transpose is missing from the Mahalanobis distance in Definition 1.
  14. Split ImageNet-R is cited as the ImageNet-21k paper (Ridnik et al.). Please correct this and verify all references.
  15. Typo at L444: "HiDEe-prompt".
2 comments

W1. Improvement over SOTA and the low-data claim.

  • a. We agree that SPR does not improve over HiDe-Prompt on standard disjoint splits, and we have narrowed our claims to limited-data settings with possibly overlapping tasks (see our response to SxVu, W1). SPR nonetheless remains competitive in the full-data disjoint setting. We consider this important, since in practice the learner does not know in advance whether it will face task overlap, limited data, or long task sequences, and a method should perform well in each case.
  • b. While the absolute gain over global pooling in Fig. 4 is roughly constant across data fractions, the proportional gain is larger in the low-data regime. Our improvements on extended task streams also indicate that SPR can help more as the number of prompts grows and the data per task shrinks.
  • c. The setting of Table 3 is introduced in “Robustness on Mixed Data Streams,” but its results were not discussed, which we have now added. SPR obtains its largest gains in this mixed-dataset setting, indicating that it routes samples of recurring concepts to the existing prompt. Thank you for catching this oversight.
  • d. We do not target few-shot CL. Our tasks are subsampled (e.g., 10%) rather than limited to a few shots per class. SPR's gains are also larger on longer streams, where each task has less data (Table 2).
  • e. The focus of our work is in improving the popular prompt-pool setup. We have added SMoPE as a baseline (see our response to 3iqU, W2), and are working to include the full results in the camera-ready version if accepted. We will provide a discussion of these mentioned baselines in our related work, as notable alternatives to the standard prompt-pool.

W2. Method specification and novelty.

  • a. The SPR gate is applied only during training, to decide which prompts each training sample may update. At inference, prompts are selected by the base method (HiDe's task-identity head, or key-query search for S-Prompt++). We will clarify this point in Section 5.2.
  • b. There is no history for the first task, so all samples train the initial prompt set. Please see the updated Appendix for the online estimate. We add a diagonal with ϵ=1e−4\epsilon=1e-4 when estimated covariance matrices are singular.
  • c. Old prompts are intentionally not frozen, to allow backward transfer when concepts recur. They are only updated by samples the gate marks as in-distribution for earlier classes, which limits interference from unrelated data.
  • d. SPR controls the effective pool size for each sample, through incremental expansion and masking. This restriction encourages prompt reuse and supports backward transfer.
  • e. In Sec. 5.2, “orthogonal” refers to samples that existing prompts cannot explain. As more samples and tasks are observed, existing prompts cover more concepts, so fewer samples need to be routed to new prompts. This concerns data accumulated across the stream, not within a task, so it does not conflict with limited data per task.
  • f. Our claim on routing noise concerns discrete selection over prompt pools, where a misrouted sample updates the wrong prompt. CODA-Prompt instead composes each prompt as an attention-weighted sum of components, trained end-to-end. A similar effect may arise there if attention weight is allocated to unrelated components during training, but this falls outside the discrete-selection setting that Table 1 evaluates, and we have clarified this scope in the paper.
  • g. SPR keeps the base method's test-time selection but changes which prompts each sample trains (and therefore how prompt keys are trained), so test-time selection better matches training. Fig. 5 shows that prompt usage at test time is closer to training-time usage with SPR.
  • h. We chose S-Prompt++ to isolate the effect of prompt routing, as it is a simple task-expert method without additional algorithmic components. When applied to HiDe-Prompt, the gate acts as an additional training-time filter on top of HiDe's own routing. Since the same gate applies to both bases, SPR is not a modification of HiDe-Prompt but a training algorithm for prompt-pool methods, whose main mechanism is a per-sample OOD gate.

Questions.

  • Q1. See W2.a.
  • Q2. See W2.b.
  • Q3. See W2.c.
  • Q4. See W2.f and W2.g.
  • Q5. See W2.g.
  • Q6. See W1.b.
  • Q7. See W2.h.
  • Q8. By limited data, we mean few labeled samples per task, not limited memory. On longer task streams (Table 2), with more prompts and less data per task, SPR's larger gains suggest it more effectively routes limited samples within a larger prompt pool.
  • Q9. See W2.d.
  • Q10. We use q = 0.95 in all experiments and corrected the example value in the main text. All other hyperparameters are now provided directly in the appendix.
  • Q11. See W1.c.
  • Q12-15. Thank you for catching these errors, which we have corrected in our revision.
AuthorsOpenReview ↗

**Revised pooled covariance formulation and storage (W2).**

The original submission described storing a covariance matrix for each class and obtaining the pooled covariance by frequency-weighted averaging. While preparing our response on storage overhead to reviewer 3qiU, we realized that the same pooled covariance can be obtained without storing any per-class covariances, by instead maintaining an estimate of the pooled within-class scatter matrix. The revised Section 5.3 and Appendix B provide more details on the feature statistics estimation in this new form. Since the resulting covariance estimate is identical, all reported accuracy results of SPR are unaffected, only the storage overhead of our method has been updated.

Reviewer_3qiUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies rehearsal-free continual learning under limited per-task training data and potentially overlapping tasks. The authors propose Similarity-Aware Prompt Routing (SPR), which consists of two main components: (1) incremental global pooling, which gradually expands the prompt pool to reduce routing noise, and (2) instance-wise prompt masking, which uses OOD detection to distinguish samples that can reuse existing prompts from those requiring new prompts. Experimental results suggest that the proposed routing strategy can improve performance, particularly in mixed-task and extended-task-stream scenarios.

Critical strengths
  • The paper addresses an important and practical setting in continual learning, where training data are scarce and tasks may overlap.

  • The evaluation covers several benchmarks and investigates different training-data volumes and task-sequence lengths. The results on mixed-task benchmarks are encouraging.

Critical weaknesses
  • My primary concern is the novelty of SPR. The method combines incremental prompt pooling with OOD-based prompt routing, but both underlying ideas have been explored in prior work. For example, LW2G [1] investigates knowledge reuse and prompt expansion based on task differences. More importantly, CLOM [2] has previously explored the connection between OOD detection and task masking in continual learning. Although SPR combines these ideas differently, the paper does not sufficiently clarify what fundamentally new capability or insight it provides beyond existing approaches. In particular, the proposed RMD-based masking mechanism appears to be a relatively straightforward application of an established OOD detection technique to determine whether historical or newly introduced prompts should be used.
  • The experimental evaluation does not include comparisons with some recent state-of-the-art prompt-based continual learning methods, such as [3].
  • The paper does not report the associated memory requirements, computational overhead, or inference latency.

[1] LW2G: Learning Whether to Grow for Prompt-based Continual Learning, arxiv 2025.

[2] Continual Learning Based on OOD Detection and Task Masking, arxiv 2022.

[3] One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning, ICLR 2026.

Questions for authors

Please refer to the weaknesses discussed above.

2 comments

W1. Novelty relative to LW2G and CLOM.

Our main novelty is a prompt routing mechanism that uses OOD detection as a gate between past and new prompts, deciding for each training sample which prompts it may update. To our knowledge, this is new for prompt-pool based CL, where routing during training is typically decided per task. Since tasks often mix seen and novel patterns, sample-level routing lets the model refine existing prompts for familiar patterns and dedicate new ones to unfamiliar data.

Relative to the works you mention:

  • Beyond task vs. sample-level reuse decisions, LW2G and SPR differ in how they update reused prompts. When LW2G reuses a prompt set, it constrains updates to be orthogonal to the old tasks’ feature space, so the reused prompt can add new knowledge but cannot refine existing knowledge. This protection is useful when earlier tasks have enough data to learn reliable prompts. In our data-scarce setting, however, a prompt trained on a single task may capture shared patterns only imperfectly. Without refinement, a recurring pattern’s data is fragmented across prompts, which may amplify routing errors at inference. Instead, SPR lets overlapping samples from later tasks update that prompt, refining what was learned from few examples.

  • In short, CLOM is designed for non-overlapping tasks, training each task model to reject samples from other tasks and using per-task masks that let new tasks reuse earlier features but not refine them. SPR makes no non-overlap assumption, and uses an OOD score during training to decide, per sample, whether to refine an existing prompt or learn a new one. See our response to Reviewer SxVu (W3) for a full discussion.

We do not claim RMD itself as a contribution, but find it well suited to this role: it separates seen from unseen tasks better than routing signals common in prompt-based methods (Table 1). Below, we add baselines on CIFAR-100 and downsampled 5-Datasets and DomainNet. RMD, designed for near-OOD detection, outperforms standard Mahalanobis distance (MD) and all other detectors.

AUC for distinguishing samples of seen vs. unseen tasks.

Routing Signal5-Datasets (10%)CIFAR-100DomainNet, 10 tasks (10%)
Learnable key78.563.150.2
Task centroids84.864.050.5
Task classifier (linear probe)82.184.360.2
KNN [1], k = 1088.981.058.0
MD [2], full Σ91.975.356.2
MD, diagonal Σ90.972.854.4
RMD, full Σ96.485.167.9
RMD, diagonal Σ92.784.364.3

W2. No comparison with SMoPE (ICLR'26).

Thank you for the pointer. We adapted SMoPE's official code to our setting and ran it on 10-task ImageNet-R with varied downsampling rates (tables below). For fairness, all methods share one setup combining our optimization recipe (steps, learning-rate schedule) with SMoPE’s prompt structure (prompt layers, length). Absolute numbers therefore differ from those in the paper, but the comparison between methods is controlled. We report mean ± std FAA and CAA over 3 random data partitions. In FAA, SMoPE is stronger at 5% and 10% data, possibly because a single shared MoE prompt is more data-efficient than a growing prompt pool, while SPR is higher from 25% upward. CAA shows a similar crossover, with SMoPE still slightly ahead at 25% and SPR higher from 50% upward. We will add this comparison, complete with other baselines, in the final version.

FAA:

Method5%10%25%50%100%
SMoPE33.9 ± 0.242.8 ± 0.851.3 ± 0.755.0 ± 0.358.6 ± 0.2
S-Prompt++25.6 ± 0.936.4 ± 0.348.2 ± 1.355.3 ± 1.860.4 ± 1.4
+ SPR26.6 ± 1.140.2 ± 0.751.5 ± 0.657.3 ± 0.262.0 ± 0.2

CAA:

Method5%10%25%50%100%
SMoPE37.8 ± 2.245.6 ± 2.056.7 ± 1.862.2 ± 1.666.4 ± 1.3
S-Prompt++31.6 ± 2.442.0 ± 1.754.4 ± 1.661.7 ± 1.866.5 ± 2.3
+ SPR32.1 ± 2.543.9 ± 2.356.4 ± 2.063.3 ± 1.868.4 ± 1.1

W3. Memory, compute, and latency.

SPR has no inference overhead, as the gate runs only during training. There, it adds ~1% runtime per iteration (computing OOD scores, updating class statistics) and storage of ~1% of the base model’s (ViT-B) parameters: Nclasses+1N_{classes}+1 means and covariances (+1 for RMD’s background Gaussian) and the scalar OOD scores used to set the threshold. In the OOD evaluation above (W1), diagonal covariances cost only 0.8–3.7 AUC and still match or exceed all non-RMD signals, suggesting a way to further reduce costs while maintaining the benefits of SPR.

Overhead of SPR.

BenchmarkStored ParametersSize (fp32)Storage overhead (%)Training overhead (%)
ImageNet-R804,2423.22 MB+0.92%+0.8%
CIFAR-100693,2612.77 MB+0.78%+0.9%
DomainNet956,6663.83 MB+1.09%+1.0%
[1] Sun et al., ICML 2022
[2] Lee et al., NeurIPS 2018

Thank you for your response. Most of my concerns have been addressed. I decided to increase the score.

Reviewer_SxVuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses rehearsal-free, prompt-based continual learning with limited per-task data and potentially overlapping concepts. The proposed SPR uses an online Relative Mahalanobis Distance (RMD) score to route samples either to an existing prompt pool or to a newly added prompt. Feature statistics are maintained incrementally, and new prompts are initialized orthogonally to existing ones.

Critical strengths

1.The setting of limited data and unknown task overlap is practically relevant and relatively underexplored. 2.Table 1 provides convincing motivation: online RMD achieves a seen/unseen discrimination AUC of 0.877, compared with 0.602–0.749 for alternative routing mechanisms. 3.The Usage Gap metric offers a useful diagnostic of prompt underuse and routing collapse.

Critical weaknesses
  1. Limited gains under standard protocols. On Split CIFAR-100, SPR slightly underperforms its HiDe-Prompt base recipe across all three metrics. It also performs worse on 5-task DomainNet (78.65 vs. 78.92). The evidence therefore supports a narrower claim about the proposed low-data setting, rather than a general improvement over the underlying method.
  2. The claimed task-similarity awareness is not explicitly established. The method estimates sample novelty relative to historical class statistics, but does not directly estimate relationships between tasks. The authors should explain how this constitutes progressively learned task similarity, or revise the terminology and claims to reflect the mechanism more precisely.
  3. Insufficient differentiation from closely related work. The claimed contribution of connecting continual learning with OOD detection appears to overlap substantially with Kim et al. (CVPR 2022), which the paper itself cites. A direct comparison and a clearer account of the methodological differences are needed.
  4. The evaluation protocol needs greater transparency. The number of runs is unspecified despite the reported error bars.
Questions for authors

The presentation requires revision. Section 6.2 contains two consecutive versions of the same paragraph, while §4 devotes substantial space to standard ViT and prompt-retrieval operations.

1 comment

W1. Limited gains under standard protocols.

We agree that the gains on standard disjoint CIL are small. Our setting of interest is one where tasks may or may not share concepts, and the learner does not know in advance which is the case. We thus evaluate on both overlapping and non-overlapping settings, as a method should remain competitive regardless of the deployment environment. We will narrow the abstract and contributions to focus on tasks with limited data and possibly overlapping concepts (e.g., our mixed-dataset experiments, Table 3).

W2. Task-similarity awareness is not established.

In SPR, we compare each new training sample with Gaussian models of the features of classes seen in previous tasks. The resulting OOD score determines whether the sample resembles a previously seen feature distribution (collectively determined by class label and covariate distributions). Samples that resemble a seen known distribution refine an existing prompt, and novel samples train a new prompt. Similarity is thus measured per sample rather than per task, which makes routing more flexible. In particular, because our score is based on the relative Mahalanobis distance (RMD), which is sensitive to near-OOD shifts, even samples that share a known class's semantics but differ significantly in covariates can be delegated to a new prompt. We find this flexibility beneficial in our mixed-dataset experiments, where both recurring concepts and covariate-shifted samples occur. That said, we agree that "task-similarity awareness" is imprecise and instead say that SPR "progressively builds a distributional model of encountered classes and routes each new training sample according to its similarity to this history".

W3. Difference from Kim et al. (CVPR 2022).

The key differences come from the problem each method addresses. CLOM [1] is designed for tasks that do not overlap. In this setting, every sample from another task is out-of-distribution for the current one. CLOM therefore trains each task model to separate known from unknown classes (using augmentations of the task’s own data as surrogate OOD samples), so that it gives low scores to samples from other tasks. To prevent forgetting, CLOM learns a hard-attention mask per task over each layer’s units (neurons in a CNN) and blocks gradients to units used by earlier tasks. A new task can reuse these units but cannot update them, and because the mask is conditioned on the task, every sample within a task passes through the same mask.

We consider the setting where a class or concept may recur in a later task, and make no non-overlap assumption. Under such overlap, CLOM’s premise that samples from other tasks are out-of-distribution no longer holds, and its masks prevent earlier knowledge from being refined when a concept recurs. Our OOD score is used only during training, to gate knowledge sharing via prompts, by determining if new samples should refine existing prompts or train a new one. In contrast, CLOM’s masks let a new task reuse earlier features but not refine them, and sharing is fixed per task rather than decided per sample.

We will revise our contribution statement accordingly. We build on the line of work connecting continual learning and OOD detection [1], and our contribution is to use OOD detection as a per-sample, training-time gate for knowledge sharing. A direct empirical comparison with CLOM is not straightforward, as CLOM is not prompt-based and is designed for non-overlapping tasks, so the two would not be evaluated under comparable conditions.

W4. Evaluation Protocol.

We will expand our description of the experimental setup in the main text (in each table caption) and in the appendix. All current error bars report the mean and standard deviation over 3 seeds.

Presentation.

We appreciate your attention to detail. We removed the repetitive phrasing and moved the standard ViT/MSA details to the appendix. We will use this space to better explain our experimental setup and to position our method within related work.

[1] Continual Learning Based on OOD Detection and Task Masking (Kim et al., CVPR 2022)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.