- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses rehearsal-free prompt-based continual learning (CL) in settings where per-task data is scarce and concepts may overlap across tasks. The authors argue that task-specific prompts cannot leverage knowledge from related tasks, while global prompt pools introduce routing noise that leads to negative transfer. They propose Similarity-Aware Prompt Routing (SPR), which has two components: (1) incremental global pooling, where new prompts are added for each task while existing prompts remain available for both training and routing; and (2) instance-wise prompt masking, where each training sample is routed either to the existing prompts (in-distribution) or to the new prompts (out-of-distribution) based on a Relative Mahalanobis Distance estimated online in the frozen backbone's feature space.
Critical strengths
The problem setting is practical and the motivation is sound. I agree with the observation in Fig. 1 that similar or recurring concepts reappear over time (e.g., Task 1 and Task sharing the same higher-level concept), and few-shot CL is itself a highly practical topic. The hypothesized tension between knowledge sharing and routing noise is also reasonable and provides a clear framing for the method.
Critical weaknesses
-
The method shows almost no improvement over the state of the art, and the central claim that it performs better under limited data is not supported by the experiments.
- In Table 4, SPR trails HiDe-Prompt on all three metrics for Split CIFAR-100, and on Split ImageNet-R the FAA and FFM are essentially unchanged; the only notable gain is in CAA (+1.6). Table 2 shows the same pattern.
- The comparison in Fig. 4 is against the authors' own routing variants rather than against state-of-the-art methods, and the gain over global pooling appears roughly constant across data proportions. It is therefore unclear that the method is particularly effective when data is scarce.
- Table 3 shows gains on the protocols with task overlap, but the main text never discusses Table 3, so the reader cannot tell what insight these results are meant to convey.
- If the paper targets data scarcity, it should be framed and evaluated as few-shot CL, which it is not. The set of baselines is also too small: there is no experimental comparison with ConvPrompt (CVPR'24), LAE (ICCV'23), OVOR (ICLR'24), or other prompt-based CL methods beyond HiDe-Prompt (NeurIPS'23).
-
The method is underspecified, and several of its key premises are asserted without evidence, which makes its novelty difficult to assess.
- Masking is defined only for training; the paper does not explain how routing is performed at inference time.
- The construction of the historical summary that drives the binary mask is only sketched in Sec. 5.3. The handling at (when no exists), how the pooled covariance is derived from the per-class , and the value of are all unspecified.
- Since all prompts remain routable and trainable, old prompts are apparently not frozen, but the rationale for this design choice (e.g., to exploit overlap) is not given.
- The paper states that restricting the pool size forces reuse, but the relevant quantity is not the nominal pool size but the number of prompts that have actually received updates. Since the pool size is itself a hyperparameter, this argument does not hold as stated.
- The claim that orthogonality diminishes as data accumulates is unsupported, and it conflicts with the paper's own premise that data remains scarce.
- There is no evidence that routing noise causes negative transfer when a global pool is optimized end-to-end, as in CODA-Prompt. Table 1 measures the separability of learnable keys, task centroids, and a task classifier, but it does not cover end-to-end optimized pools such as CODA-Prompt's attention-based aggregation, nor does it connect low AUC to actual negative transfer.
- There is no preliminary analysis showing that overlap actually arises under similarity-based prompt selection, and no evaluation showing that routing error is actually reduced by SPR.
- The main results are built on HiDe-Prompt, but the ablations use S-Prompt++. For the ablations to inform the main results, they should be run on HiDe-Prompt as well. Based on the algorithm and implementation details, the method appears to be HiDe-Prompt with additional optimizations, and I could not identify a clearly novel mechanism.
Questions for authors
- How are prompts selected at inference time without a task ID? Is RMD also used at test time?
- How is the historical summary handled at , how is the pooled covariance derived from the per-class , and what value of is used?
- Why are old prompts not frozen? Is this an intentional choice to exploit overlap?
- Is there evidence that routing noise causes negative transfer in end-to-end optimized global pools such as CODA-Prompt? Table 1 does not include CODA-Prompt's attention-based routing. Can you provide a preliminary analysis showing that overlap actually occurs under similarity-based selection?
- Can you show, in the evaluation, that SPR actually reduces routing error, beyond the AUC reported in Table 1?
- Can you provide the data-proportion experiment of Fig. 4 with state-of-the-art baselines such as HiDe-Prompt? In the current results, the gain appears independent of the data proportion.
- Can you provide the ablation results on the HiDe-Prompt base?
- Prompts compress past information into parameters rather than storing raw data, so it is accurate to say the system uses little memory, but not that the amount of data is small. Is there any observation or data showing that adaptation works well specifically when the number of prompts is small?
- Please clarify the rationale for restricting the pool size. What is actually being controlled: the pool size or the number of prompts that receive updates?
- Which OOD threshold is used, (main text) or (Appendix)? Please also specify the number of prompts added per task, top-, the prompt length, and the layers into which prompts are inserted.
- Please add a discussion of Table 3 to the main text, including what explains the gains observed there.
- The index range in the problem formulation (Eq. 1) is inconsistent with the task indexing.
- The transpose is missing from the Mahalanobis distance in Definition 1.
- Split ImageNet-R is cited as the ImageNet-21k paper (Ridnik et al.). Please correct this and verify all references.
- Typo at L444: "HiDEe-prompt".
2 comments
**Revised pooled covariance formulation and storage (W2).**
The original submission described storing a covariance matrix for each class and obtaining the pooled covariance by frequency-weighted averaging. While preparing our response on storage overhead to reviewer 3qiU, we realized that the same pooled covariance can be obtained without storing any per-class covariances, by instead maintaining an estimate of the pooled within-class scatter matrix. The revised Section 5.3 and Appendix B provide more details on the feature statistics estimation in this new form. Since the resulting covariance estimate is identical, all reported accuracy results of SPR are unaffected, only the storage overhead of our method has been updated.
W1. Improvement over SOTA and the low-data claim.
W2. Method specification and novelty.
Questions.