acceptodds
Under review as a conference paper at ICLR 2027

Measurement-Driven Isometric Shared-Prompt Transport for General Continual Learning

Abstract

General continual learning (GCL) requires single-pass learning from non-stationary streams without explicit task boundaries. Prompt tuning enables parameter-efficient adaptation of pretrained models, but recurring classes and changing class mixtures demand both knowledge reuse and distribution-specific adjustment. Selecting or combining independently learned prompts does not explicitly separate shared adaptation structure from distribution-specific variation, potentially dispersing reusable knowledge across prompt states under limited supervision. Meanwhile, imbalanced class observations can bias classifiers toward recent data, further limiting the effective use of historical knowledge. We propose MIST (Measurement-conditioned Isometric Shared-prompt Transport), a replay-free framework that coordinates structured prompt adaptation and adaptive classification through continuous historical associations. MIST uses frozen-feature statistics to associate inputs with historical distributions and condition isometric transformations of shared latent prompts, preserving token geometry across conditions for a given shared latent representation. The same associations weight temporal classifier ensembles, while statistical novelty and confidence advantage regulate their fusion with an online plastic classifier to balance knowledge retention and adaptation. Experiments demonstrate absolute gains of up to 12.10%, 9.14%, and 24.25% in average anytime accuracy over the strongest evaluated baselines on CIFAR-100, ImageNet-R, and CUB-200, respectively.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
General continual learning, Prompt learning, Parameter-efficient adaptation, Replay-free learning, Isometric prompt transport

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TERLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a replay-free framework for general continual learning, which is named MIST. MIST maintains historical second-order statistics in a frozen feature space to associate current inputs with past distributions, and uses these associations to condition orthogonal transformations of a shared latent prompt. It further combines historical EMA classifier predictions with an online plastic classifier using statistical novelty and confidence-based fusion. Results show strong anytime and final accuracy and retention.

Critical strengths
  1. The paper comprehensively evaluates MIST on three benchmarks with six pretrained backbones.

  2. MIST consistently achieves competitive or superior accuracy and retention, especially under self-supervised pretraining.

  3. MIST achieves strong continual-learning performance with only a small number of trainable parameters.

Critical weaknesses
  1. The current cumulative ablation does not establish whether isometric transport itself is the main contributor.

  2. The gains compared to FlyPrompt are very large under some self-supervised settings, but the paper does not clearly state whether optimizer, learning rate, augmentation, local optimization steps, and tuning budgets are strictly matched across methods. In particular, MIST explicitly performs two optimization steps per incoming mini-batch, and their actual optimization budgets are different?

  3. Continuous Statistical Measurement (CSM) is always enabled, so its individual contribution is not evaluated, and the cumulative design does not reveal interactions among CIT, GC, and APF. The authors explicitly acknowledge that the current ablation does not isolate CSM.

  4. The necessity of the isometric constraint is not sufficiently demonstrated. The paper lacks a controlled comparison against simpler conditional prompt transformations using the same statistical associations and classifier-side components. Therefore, it remains unclear whether preserving latent prompt geometry is essential for the observed gains.

  5. The theoretical invariant does not directly guarantee preservation of the actual prompts or representations used by the backbone.

  6. The task-free setting still relies on manually scheduled internal intervals. This partially weakens the motivation of fully boundary-free GCL.

  7. Hyperparameter sensitivity is incomplete, including E_max, d_m, etc.

  8. The performance advantage is not universal. On ImageNet-R with the supervised Sup-21K backbone, FlyPrompt outperforms MIST in accuracy, and this trend persists across the evaluated blurry-ratio settings. This suggests that the advantage of MIST depends on the pretraining and dataset regime.

Questions for authors

See above.

Reviewer_ASbvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MIST, a replay-free framework for general continual learning that coordinates prompt adaptation and prediction through continuous historical associations computed from frozen-feature statistics. MIST represents adaptation using a shared latent prompt and input-dependent orthogonal transformations, thereby preserving latent prompt geometry across conditions for a fixed shared configuration. The same historical associations are further used to aggregate interval-specific temporal classifiers and adaptively fuse historical predictions with an online plastic classifier. The method is evaluated on CIFAR-100, ImageNet-R, and CUB-200 across six supervised and self-supervised pretrained ViT configurations, where it achieves particularly strong improvements under self-supervised pretraining.

Critical strengths

The paper addresses an important challenge in task-boundary-free continual learning: how to reuse adaptation knowledge under recurring classes and evolving class mixtures without storing past samples. The conditional isometric prompt transport is technically interesting and provides a clear structural separation between a shared latent representation and input-dependent variation. Unlike purely intuitive geometric arguments, the orthogonal construction provides an explicit invariance of latent token norms and pairwise inner products. The empirical evaluation is also extensive covering three benchmarks and six pretrained-model configurations. The improvements are substantial in several challenging self-supervised settings, particularly on CUB-200, suggesting that the method addresses a meaningful weakness of existing prompt-based GCL approaches.

Critical weaknesses
  1. The empirical analysis does not yet fully establish which properties of the relatively complex MIST design are responsible for its gains. This ablation process cannot fully clarify the independent contributions or interactions of the various components. For example, the improvement from CIT does not isolate the benefit of isometry itself from the additional capacity and input-dependent conditioning introduced by conditional transport, which is important for validating the paper's central geometry-preservation motivation.

  2. The measurement mechanism would benefit from direct empirical validation. The method interprets PGM-derived responses as historical associations and the isotropic response as statistical novelty, but the experiments mainly validate these quantities indirectly through downstream accuracy. It remains unclear how accurately the associations identify genuinely relevant historical intervals, or how well the proposed novelty score distinguishes recurring from genuinely novel observations.

  3. Several methodological details are insufficiently explained. In Sec. 3.2, Lines 234–244 introduce Ul(x)U^l(x), Zl(x)Z^l(x), effective angles, and givens rotations, but the practical construction is difficult to follow. In Sec. 3.3, the definitions of the classifier weights, novelty score, fusion quantities, and the motivation for the fixed 0.7/0.3 weighting also require clearer explanation. Given the sensitivity analyses, complexity of the framework, and computational-overhead analyses would improve the evidence for the proposed design.

Questions for authors
  1. The historical association re(x)r_e(x) is measured in a frozen, unprompted feature space but is subsequently used to control prompt adaptation and historical classifier aggregation. Could the authors directly evaluate whether these association weights identify the intervals that are actually relevant to the current input after adaptation?

  2. The proposed transport guarantees preservation of pairwise geometry among latent prompt tokens for a fixed shared configuration. Could the authors provide empirical evidence connecting preservation of this geometry to functional knowledge retention or reduced forgetting?

  3. Algorithm 1 shows that the proposed method requires a significant computational cost. Could the authors report practical training and inference overhead relative to the prompt-based GCL baselines? Meanwhile, there are several key mechanism-specific hyperparameters. Could the authors provide sensitivity analyses of these parameters?

1 comment
Reviewer_ASbvOpenReview ↗

Official Comment by Reviewer ASbv

Thank you for the author's response, but I still have some concerns. In Table R2.1, prototype supervision proves effective across multiple evaluation metrics. For example, its AUC value improves by 7.89 compared to the baseline model and by 7.36 to the w/both. However, MIST improves by only 5.36 compared to the baseline model with enabling prototype supervision, so it is puzzling that MIST’s performance boost is weaker than that of simple prototype supervision. I am unsure whether this trade-off between performance gains and computational cost is reasonable. The independent contributions or interactions of the various components seem to remain unclear.

Reviewer_KrLXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies replay-free general continual learning (GCL) under the Si-Blurry protocol, where data arrive once, classes recur, and task identities are unavailable. The proposed MIST framework extracts frozen-feature second-order statistics to associate an input with historical intervals and estimate novelty. These associations condition Givens rotations of a shared low-dimensional prompt and weight historical EMA classifiers, which are fused with a plastic classifier. The method also uses class-prototype supervision. Experiments on CIFAR-100, ImageNet-R, and CUB-200 with six pretrained ViT configurations report strong results, especially with self-supervised backbones.

Critical strengths

S1. Relevant setting and coherent motivation. The study addresses a realistic replay-free stream with recurring classes and changing class mixtures. The combination of shared adaptation and history-aware prediction is conceptually motivated. S2. Broad and promising results. The paper evaluates six ViT pretraining variants and reports mean and standard deviation across five seeds. For iBOT-21K/CUB-200, MIST improves average anytime accuracy from FlyPrompt's 28.86% to 50.01% (Table 1). S3. Transparent technical detail. The appendix supplies mathematical properties, a training algorithm, hyperparameters, and failure cases. The authors acknowledge that latent isometry does not imply invariance of the projected prompts.

Critical weaknesses

W1. Performance attribution is unclear In Table 2, CSM-only already obtains 44.14% average anytime accuracy on MoCo v3/CUB-200, compared with 27.92% for FlyPrompt in Table 1; full MIST reaches 49.50%. CSM-only retains the shared-prompt classifier, masking, and class-prototype supervision. The prototype term carries 50% of the classification-loss weight when available (Appendix E.2). Hence, it is unclear how much of the headline improvement comes from the new transport and fusion mechanisms. In the same ablation, final forgetting is 10.46 for CSM-only versus 13.08 for the full method; the full system does not uniformly improve every measure. W2. Important baseline and fairness questions. The closely related SinglePrompt approach (Park et al., 2026), which is cited in the related work, is not included in Table 1. Frozen-feature approaches such as SLDA, NCM/SimpleCIL, or RanPAC would also help contextualize the gains. In particular, please clarify whether self-supervised baseline configurations were re-tuned or matched for optimizer, augmentation, and local update steps (MIST uses AutoAugment and K=2). W3. Isometric transport has not been isolated. The guaranteed isometry applies to 32-dimensional latent prompts at a fixed model state. After the learned projection W and additive embeddings B, the actual prompts need not retain that geometry (Sec. 3.2; Appendix D.2). The existing CIT ablation compares conditioning with no conditioning, not with a parameter-matched non-isometric transformation. A simple matched linear/MLP control is essential. W4. Measurement and scheduling choices require controls. The PGM-based interval weights are not directly compared with uniform weighting, cosine similarity, or simpler statistical routing. Training also predominantly conditions on the active interval (beta=0.2), whereas inference uses historical associations alone. The fixed sample-count interval schedule assumes a known stream length; sensitivity to these choices is unclear. W5. Appendix comparisons and efficiency are underexplained. Appendix Fig. 4 suggests that COSO can match or surpass MIST in some self-supervised settings, notably ImageNet-R/iBOT-21K. Appendix Table 5 shows a substantially different COSO ranking under Sup-21K, and the task-boundary/protocol comparability of conventional CL baselines needs clarification. MIST also requires both unprompted and prompted backbone passes at inference (Appendix F), yet no measured latency comparison is provided.

Questions for authors

Q1. Can you report a factorial ablation that separately removes prototype supervision, EMA fusion, CSM, and isometric transport? How much of the self-supervised gain remains without prototype supervision? Q2. Can you include SinglePrompt (Park et al., 2026), preferably under the same stream, backbone, augmentation, and K-step budget? Were existing baselines re-tuned for each self-supervised initialization? Q3. How does Givens-based transport compare with parameter-matched non-isometric linear/MLP conditioning, and how do PGM associations compare with uniform or cosine-based weights? Q4. Can you clarify the experimental protocol for the conventional CL methods in Appendix Fig. 4/Table 5, especially task-boundary information and the differences in COSO results across backbone initializations? Q5. What are the actual training/inference latency and memory costs, and how sensitive are the results to beta, the number of internal intervals, and the measurement/gating hyperparameters? Are historical angle slices strictly frozen despite optimizer state?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.