acceptodds
Withdrawn from ICLR 2027

Stream-Aware Prompt Inference via Simplex Search for Class-Incremental Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Class-incremental learning (CIL) requires models to acquire new classes while retaining previously learned knowledge continually. With the emergence of powerful pre-trained models (PTMs), particularly Vision Transformers, CIL has increasingly shifted toward parameter-efficient adaptation, where frozen backbones are augmented with lightweight prompts or adapters. While these approaches have advanced rehearsal-free CIL, their inference mechanisms are predominantly designed for individual samples or task-specific routing, conventionally assuming an independent test stream. In practical deployments, however, incoming observations may exhibit local statistical coherence, sharing latent context such as task, domain, environment, or scene, while independent and coherent observations may also be interleaved. Our experiments reveal that such stream structure provides an additional source of information for CIL at inference time, yet it remains largely unexplored in prompt-based CIL. We introduce a stream-aware prompt inference paradigm that exploits local coherence when present while remaining effective for uncorrelated streams. We realize this through ARC (Anchor-Residual Composition), which decomposes each task prompt into a shared anchor and low-rank task residuals, yielding a linearly composable prompt space. Building on this representation, SPS (Simplex Prompt Search) performs entropy-driven optimization over convex combinations of task residuals, adapting inference to the incoming stream. For uncorrelated streams, SPS performs sample-wise optimization, while locally coherent streams enable batch-level optimization that exploits shared statistical structure. Across diverse distribution-shift CIL benchmarks, ARC consistently outperforms existing prompt- and adapter-based methods under both independent and locally coherent streams. Further analysis shows that the effectiveness of stream-aware prompt composition is governed by the geometry of the learned prompt space, highlighting stream structure and prompt geometry as complementary dimensions of inference in continual learning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Class-Incremental Learning, Continual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RgyxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to incorporate test-time inference into class-incremental learning. The authors introduce a test-stream setting and propose a prompt composition method to leverage relationships among test samples. The method demonstrates improvements under the constructed test streams. However, I am not fully convinced that the proposed setting reflects the real-world scenarios used to motivate this work. The design of the anchor also requires further clarification and justification. In addition, the experimental evaluation is relatively limited, making it difficult to fully assess anti-forgetting ability of the proposed method.

Critical strengths

The paper considers continual learning from the perspective of test-time inference rather than focusing only on the training process. I think exploiting correlations among test samples is an interesting direction and could be useful in some real-world applications, although the current experiments do not fully validate this motivation.

Critical weaknesses
  1. The paper motivates stream-aware inference using examples from surveillance and autonomous driving, where consecutive observations naturally share similar scenes or environmental context. However, in the experiments, the correlation is introduced by simply arranging test samples according to their incremental tasks. Since these tasks are defined by class partitions, samples from the same task do not necessarily share meaningful visual similarities. It is also unclear why test samples would arrive in this task-specific order in practice, especially for image classification datasets such as ImageNet-R and ImageNet-A. I am therefore not convinced that this experimental setting reflects the type of correlation used to motivate the method.

  2. The anchor is learned jointly with the residual on Task 0 and then frozen for all subsequent tasks. However, I am not sure why the first task is sufficient for learning an anchor that is supposed to capture shared information across all tasks. Would the learned anchor be sensitive to which classes are assigned to Task 0? I did not find an analysis of this issue. Another concern is that both the anchor and the Task 0 residual are trained using the same cross-entropy loss, without any explicit constraint to encourage them to learn different information. It is therefore unclear what ensures that the anchor captures shared knowledge while the residual captures task-specific knowledge.

  3. The main experiments only compare final accuracy and average incremental accuracy with prior methods, without reporting multi-seed results or forgetting rates. Although the authors provide these results in the appendix, they are not compared with existing methods. In continual learning, both the stability across random seeds and the ability to mitigate catastrophic forgetting are important for evaluating a method. Prior works such as DualPrompt and CODA-Prompt have reported results across multiple seeds. Without these comparisons, it is difficult to assess whether the proposed method consistently outperforms existing approaches and whether it is actually more effective at preventing forgetting.

Questions for authors

See weaknesses.

Reviewer_mAAeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper proposes ARC and SPS for prompt-based class-incremental learning. ARC learns a shared anchor and task-specific residual prompts. During inference, SPS optimizes the mixing weights of these residuals by minimizing the average prediction entropy over a test batch. The method aims to improve classification accuracy by exploiting correlations among test samples.

Critical strengths

The paper explores an interesting direction in class-incremental learning by considering the structure of test streams. SPS only optimizes a few mixing coefficients without updating the backbone or stored prompts. The ablation studies and inference cost analysis are also useful.

Critical weaknesses

I have a few questions. 1.According to Appendix J.1, the test sets are concatenated in task order without shuffling. How does SPS perform when samples from different tasks are mixed within the same batch? Were the baselines in Table 1 evaluated using the same test-stream protocol, and how was the additional inference cost of SPS accounted for?

2.Section 5.4 reports that 97.3% of batches on ImageNet-R converge to a dominant vertex. How is a dominant vertex defined? Would SPS still achieve similar accuracy if only the residual with the largest mixing weight were used for prediction, instead of the weighted combination?

3.The Oracle selects the residual corresponding to each sample's ground-truth task ID, which may not be the best-performing residual for that sample. How does outperforming this Oracle demonstrate that convex combinations are better than selecting individual residuals?

Questions for authors

See weaknesses.

Reviewer_rimiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper challenges the per-sample inference convention in prompt-based class-incremental learning, claiming that test streams often exhibit local coherence that existing methods discard. It proposes ARC, which decomposes each task prompt into a frozen shared anchor and a low-rank task residual so that convex combinations of residuals remain valid prompts, forming a searchable prompt manifold; inference then adapts to the stream by optimizing mixing weights over this manifold, per-sample when samples are independent and batch-level when locally coherent. A first-order analysis supports the design and predicts when manifold search helps from prompt geometry. Experiments on standard distribution-shift CIL benchmarks show competitive accuracy under independent streams and notable gains under task-correlated streams.

Critical strengths
  1. ARC's anchor plus low-rank residual design makes task prompts convexly combinable while training with pure cross-entropy only, yielding a competitive prompt-based CIL method even under the standard protocol, independent of the paper's more debatable stream assumption.
  2. The observed gain over oracle discrete selection, together with the prediction of gain from prompt-space cosine, gives the method a mechanism-level explanation rather than a purely empirical claim, which materially raises my confidence in the paper's core insight.
Critical weaknesses

Weakness 1. The paper's stream setting is claimed to be realistic, but is only tested on standard continual learning tasks.

The motivating examples involve scene-level coherence with mixed class distributions, whereas all experiments construct coherence by grouping same-task samples into a batch. Task-grouped evaluation and batch-level task inference are also common practice in continual learning, for example in MoE-Adapter [1], yet the paper neither cites such work nor clarifies how its contribution differs from it. Since the main gains and the conclusion of beating Oracle hold only under this protocol, the core claim rests on an assumption that is neither validated on realistic scenarios nor novel. The batch-aware baselines in Appendix B are variants of the authors' own DES that force a single task on the whole batch, so they are not convincing comparators.

Weakness 2. The comparison is unfair, since only the proposed method can exploit the information that defines the correlated protocol.

All baselines perform per-sample inference by default, so correlated batches cannot help them, and only this method obtains cross-sample information in Table 1. The reported gains therefore cannot be attributed to the architecture or the prompt parameterization. The natural comparators are methods that already use batch-level information, namely MoE-Adapter style approaches that infer one task per batch and route accordingly, and test-time adaptation methods that adapt the model by minimizing prediction entropy over a batch, which is exactly what SPS optimizes. Neither is compared, and test-time adaptation is not cited. Without these controls, the paper does not show that stream-aware prompt composition is better than applying an existing batch-level adaptation mechanism to a frozen backbone with a fixed prompt.

[1] Boosting continual learning of vision-language models via mixture-of-experts adapters

Questions for authors
  1. Can the authors provide comparisons with baselines that can also exploit the information that a batch belongs to one task, such as batch-routing mixture-of-experts methods and test-time adaptation methods?

  2. The paper motivates stream-aware inference with deployment scenarios where consecutive observations share scene or domain, yet the compared baselines are standard CIL methods that cannot use this structure. Could the authors add baselines designed for batch-level adaptation?

  3. The cited baselines are from 2025 and earlier. Could the authors include a recent 2026 state-of-the-art method to confirm the comparison is current?

  4. Proposition 1's error bound decreases monotonically with γ 2 , which implies smaller γ is always better, yet 0.75 is optimal and 0.25 and 0.5 are worse. How does the claim that γ < 1 is a theoretical requirement hold, and can the theory explain the optimal value?

  5. The last paragraph of Section 2, Stream-Aware Inference, contains no citations; please provide the references that support it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.