acceptodds
Under review as a conference paper at ICLR 2027

CAR: Contextual Adapter Routing for Rehearsal-Free Class-Incremental Learning

Abstract

Modern rehearsal-free class-incremental learning (CIL) increasingly builds on pretrained models augmented with task-specialized prompts, adapters, or other modular components. While substantial effort has focused on how such modules are learned and stored, inference is still typically performed independently for each test sample, discarding contextual evidence that may be available across nearby observations. We introduce Contextual Adapter Routing (CAR), a frozen context-conditioned inference strategy that exploits a small unlabeled coherent test context without replay, task labels, retraining, or test-time parameter updates. CAR observes that task-specialized modules contain weak but useful task-discriminative information in their predictive uncertainty: although this signal is noisy for individual samples, aggregating it across a coherent context yields substantially more reliable module preferences. CAR converts this contextual preference into a soft task prior while retaining image-specific all-class prediction, thereby improving routing without replacing the underlying continual learner. Across multiple CIL benchmarks and modular learners, CAR improves average continual and final-stage accuracy by up to and percentage points, respectively. Controlled analyses show that the gain depends on contextual coherence: distinct coherent observations provide substantially stronger evidence than same-image augmentations or unsupervised grouping of shuffled samples. We further study unreliable contexts and use routing confidence to fall back toward native sample-wise inference when contextual evidence is weak. Theoretically, when observations share a local shifted distribution and the appropriate module has a positive expected entropy margin, the probability of routing error decreases exponentially with context size. Together, these results establish frozen context-conditioned inference as a complementary design axis for modular continual learning: preserve native sample-wise inference when reliable context is unavailable, while exploiting coherent unlabeled context when it is informative.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, class-incremental learning, parameter-efficient adaptation, test-time inference, adapter routing, unsupervised context discovery, uncertainty estimation, test-time adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WkR5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper is addressing modern rehearsal-free class incremental learning. Specifically, the proposed method is built upon CL-LoRA and EASE. In this paper, the fundamental assumption is that the contextual evidence MAY be available across nearby observations and Contexture Adapter Routing (CAR) is introduced. A frozen context-conditioned inference strategy that exploits a small unlabeled coherent test context without replay, task labels, retraining, or test-time parameter updates. CAR observes that task-specialized modules obtain weak but useful task-discrminative information in their predictive uncertainty. Although this signal is noisy for individual samples, aggregating it across a coherent context yields substantially more reliable module preferences. CAR converts this contextual preference into a soft task prior while retaining image-specific all-class prediction, thereby improving routing without replacing the underlying continual learner.

Critical strengths
  • This paper has attention to an adapter routing method instead of solving the catastrophic forgetting problem in CIL. Furthermore, this paper formulates frozen context-conditioned inference as a distinct CIL inference settings that lies between independent frozen inference and parameter updating TTA.

  • The paper proposed a training-free contextual routing mechanism applicable to modular continual learners via frozen predictive responses like CL-LoRA and EASE.

  • They tried to show theoretically that, when observations share a local distribution and the appropriate module has a positive expected entropy margin, the probability of routing error decreases exponentially with context size. However, there is still some uncertainty.

  • They claimed the proposed method is superior over 10% to the previous method in the traditional CIL evaluation protocols. However, it is not a fair comparison because the proposed method does not follow the traditional protocols (e.g., multiple seeds)

Critical weaknesses
  • One of the major concerns for this paper is that it is not a fair comparison with the previous CIL methods. In previous CIL methods followed the standard CIL evaluation protocol, where each test sample is classified independently. However, the proposed method primarily evaluates its performance using a context of eight test images (e.g., M=8) belonging to the same class. The paper claims that the labels information of context images are not used but we can think the same bird images (in Fig. 2) are flocked when building the context set. In TTA evaluation protocol, the same context test images are not deployed in the same batch. Furthermore, in Table 1, The results of previous methods are directly taken from their original publications without re-evaluating them under the same contextual inference protocol. Consequently, the reported performance comparisons do not establish a fair or convincing superiority over existing CIL methods.

  • The proposed method assumes that semantically coherent test contexts are available (like video stream), but does not provide a sufficiently effective mechanism for constructing such contexts in image classification tasks. I just wonder why the paper does not address the video recognition problem, not the CIFAR-based image classification. To solve this concern, the paper investigated label-free context discovery methods, including feature kNN, DBSCAN, and pseudo-class grouping, but it is too simple and naive. Moreover, the substantial performance degradation observed with randomly constructed contexts further demonstrates the method's sensitivity to context quality. In this respect, the effectiveness of the proposed method under realistic test streams containing heterogeneous classes or tasks remains insufficiently validated.

  • On CIFAR-100, the reproduced CL-LoRA baseline shows 87.94% final accuracy, while the proposed method with M=1 achieves 97.95%. It indicates no improvement under single-sample inference by the proposed routing strategy. The paper already noted the single image is noisy but the unlabeled contexts also can be noisy.

  • The proposed method assumes that the adapter producing the lowest task-local prediction entropy is the most appropriate adapter for a given context. However, low prediction entropy does not necessarily indicate correct task identification, as an incorrect adapter may produce overconfident predictions.

  • The reproduced method and the proposed method comparisons are primarily based on a single random seed, limiting the assessment of robustness across different class orders, adapter initializations, and context sampling strategies.

Questions for authors
  • Code should be provided for checking the fair comparison concerns.
  • Detailed analyses of inference latency, FLOPs, memory consumption, and throughput should be provided.
4 comments

We thank the reviewer for the careful evaluation and constructive comments. We appreciate the recognition of CAR as a different inference-time approach to continual learning. We would like to clarify the intended comparison and present additional experiments addressing the concerns.

1. Comparison with existing CIL methods

We agree that conventional CIL evaluates each image independently, whereas CAR uses information from multiple observations. However, this difference is intentional and is the main motivation of our work.

CAR is not intended to replace existing CIL training methods. Instead, it improves inference using contextual information from an already trained model. In our experiments, we compare exactly the same frozen CL-LoRA or EASE model, with and without CAR. Therefore, the improvement comes from the additional contextual information rather than changes in training.

We agree that the original same-class groups were constructed using ground-truth labels. These experiments were intended to demonstrate CAR's potential under coherent contexts, not to claim a direct comparison under identical standard CIL inference conditions. We will make this distinction clearer in the paper.

2. Video evaluation and context construction

We agree that evaluating naturally occurring sequences is important. We therefore conducted an additional experiment on ImageNet-VID, where contexts are formed using consecutive video frames rather than class labels.

Native CL-LoRA achieves 86.93% accuracy, while CAR achieves 92.89%, 95.18%, and 96.56% for M=4,8,16M=4,8,16, respectively. At M=16M=16, the improvement is +9.63 percentage points over 436 frames, with a paired video-bootstrap 95% CI of [+4.17, +16.51] pp.

We also tested synthetic CIFAR-100 streams where images from different classes arrive consecutively within the same task. Using fixed windows without labels or task boundaries, CAR improves accuracy from 87.94% to 95.63% at M=4M=4 when the average task duration is 32 observations.

These results demonstrate that CAR can benefit from contextual information without requiring same-class grouping.

3. Single-image routing and incorrect adapter selection

We agree that low entropy does not always indicate the correct adapter. This is one of the motivations for CAR: entropy from individual images can be unreliable, while averaging across related observations can make adapter selection more reliable.

We would also like to clarify that CAR at M=1M=1 achieves approximately 87.95% on CIFAR-100, not 97.95%. The larger improvements come from aggregating multiple related observations.

Our additional experiments confirm that CAR can harm images from minority tasks when observations from different tasks appear in the same context. We also found that the original λ=1\lambda=1 effectively behaves like hard task selection.

Reducing λ\lambda to 0.05 improves accuracy from 88.47% to 95.11% at M=8M=8 on synthetic CIFAR-100 streams. These results suggest that giving less weight to the selected task can reduce errors in mixed contexts. We present this as a sensitivity analysis rather than a new validated parameter setting.

4. Multiple training seeds

We agree that testing multiple training seeds is important. We have now reproduced the complete TUNA framework using three independent training seeds with a fixed class order.

Native TUNA achieves 91.60 ± 0.06% final accuracy on CIFAR-100. Adding contextual adapter selection improves accuracy to 94.54 ± 0.23% at M=16M=16 under same-class grouping. On synthetic task-persistent streams, accuracy reaches 92.80–94.25%, depending on task duration.

These results show that contextual aggregation can also benefit another trained continual learning method across multiple training runs. The original CL-LoRA experiments remain single-seed.

5. Code and computational cost

We agree that the implementation and computational costs should be explained more clearly. We will clarify the experimental configurations, class orders, and implementation details.

We also measured inference time. With 10 adapters and batch size 16, native CL-LoRA takes 200.30 ms, compared with 200.77 ms for CAR at M=8M=8. CAR does not require additional backbone forward passes. Peak GPU memory was also unchanged at the measured precision, although contexts require buffering observations.

We thank the reviewer again. We hope these new experiments clarify that CAR is intended to complement existing continual learners and that its benefits extend beyond the original same-class evaluation.

The main concerns on this method is the fairness on the evaluation between the proposed one and related works. However, the rebuttal does not sufficient address this concern. Because this significant concern is not solved, it is natural to conclude that the rebuttals for other weakness are not proper now. In this respect, I will keep my original rate.

We thank the reviewer for the follow-up. We understand the concern regarding evaluation fairness and would like to clarify one point.

Our intention is not to claim that CAR outperforms existing CIL methods under the same standard sample-wise inference protocol. Rather, CAR studies whether additional contextual information, when available, can improve the predictions of an already trained continual learner. For this reason, our main comparison uses exactly the same frozen model and test images, with and without CAR.

We agree that the original same-class results represent a controlled setting and should not be interpreted as a direct comparison under the standard CIL protocol. This is also why we added the ImageNet-VID experiment, where contexts are formed naturally from consecutive video frames without using labels for grouping.

We appreciate the reviewer's concern and will make this distinction and the scope of our claims clearer in the paper.

Reviewer_WkR5OpenReview ↗

Comment from the Reviewer WkR5

Thank you for the clear explanaition. I now better understand the intended scope of CAR. However, my main concern remains that CAR relies on additional contextual information that is not available under the standard sample-wise CIL evaluation protocol. The additional ImageNet-VID experiment is useful, but it does not fundamentally resolve my main concern regarding the evaluation fairness now. So, I keep my rating.

Reviewer_gUznOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CAR, a frozen context-conditioned inference strategy for rehearsal-free class-incremental learning over modular learners such as CL-LoRA and EASE. For a small unlabeled batch assumed to form a coherent context, CAR computes each task adapter's task-local predictive entropy over the batch, selects the adapter with the lowest average entropy, converts that preference into an epsilon-smoothed task prior, and adds the log-prior to the native all-class logits, with an optional fallback when the routing margin is small. Nothing is trained, and every constant is pinned. A Hoeffding-style theorem bounds routing error as exponentially decreasing in context size under a positive expected entropy margin and i.i.d. contexts. Experiments on CIFAR-100, ImageNet-R, ImageNet-A, and VTAB report large gains under same-class contexts of size eight (CIFAR reaches 98.06 final against a 98.98 task-oracle ceiling), transfer to EASE, and diagnostics showing that no unsupervised grouping reconstructor beats native inference and that full context corruption collapses the method to 8.70. All new results are single-seed, and no code is released.

Critical strengths

The mechanism is simple, fully specified, and genuinely new within the CIL lineage. CAR averages a known per-sample signal across a coherent batch, and the paper pins every constant and gives an exact algorithm, so the method is reimplementable in about a hundred lines over any checkpoint. This is consequential because it makes the idea immediately usable and testable by others, and because the regime itself, frozen context-conditioned inference between per-sample CIL and test-time adaptation, is a real gap the paper names.

The diagnostics are unusually honest and are themselves a contribution. The paper reports that every unsupervised context reconstructor it tries fails (74 to 88 percent against 87.94 native), that full corruption collapses the method to 8.70 percent, and that same-image augmentations gain under one point while distinct coherent observations gain over ten. This is consequential because it defines when context helps and when it does not, which is the question the field needs answered before the method can be deployed.

Critical weaknesses

The headline depends on label-informed, near-oracle contexts. The Table 1 CAR row uses M = 8 same-class groups, CIFAR reaches 98.06 against a 98.98 task-oracle ceiling, and the caption does not state the grouping rule. Every deployment-plausible reconstructor the paper itself tests fails to beat native inference (74 to 88 versus 87.94), and the soft prior is arithmetically a hard gate at epsilon = 1e-3 and lambda = 1 (about a 9.1-nat inter-block bias against a native spread near 0.025). This is consequential because the paper sells a method while demonstrating an upper bound in an analysis setting, and the deployment claim is exactly what the diagnostics do not support.

The comparative evidence is single-seed and the strongest comparator is missing. Every new number comes from one training seed with no error bars, and the reproduced CL-LoRA baseline sits 3.75 to 5.18 points below its published values on ImageNet-R and ImageNet-A, so the headline gains shrink to about +7.4 and +8.5 against published numbers and reverse sign on ImageNet-A. TUNA, the conceded closest prior, is never run head-to-head even though CAR at context size 1 reduces exactly to its rule. This is consequential because the central claim is comparative, and the comparison is against a weak single-seed reproduction of the baseline rather than the baseline at its published strength.

Questions for authors

Clarification requests (these could change my recommendation):

1- What are CAR's gains against published CL-LoRA numbers, with variance over at least three training seeds?

2- Can you run TUNA head-to-head on the same frozen checkpoints and report how the CAR-minus-TUNA delta scales with context size?

3- How does accuracy change as lambda sweeps from 0 to hard selection, and does an intermediate setting soften the corruption collapse?

4- Can you add one naturally coherent, label-free stream (for example, video-adjacency or acquisition-order batches) and report the deployment-valid result?

5- Why does the reproduced baseline fall 3.75 to 5.18 points below published on ImageNet-R and ImageNet-A, and which comparator is fair?

6- What is the context size implied by Theorem 1 at your measured margin (the paper's own numbers suggest tens of thousands against the 8 used), and how do you interpret that gap?

7- How is the all-class logit vector computed for CL-LoRA and EASE, and can you release the code, class orders, and seeds?

Suggestions for future work (not needed for my assessment):

  • Report per-task routing confusion matrices and adapter-selection histograms.
  • Report wall-clock and memory cost at K = 40, and vary the adapter count to test the log-K behavior.
  • Study aggregation variants beyond the mean (median, trimmed mean).
1 comment

We thank the reviewer for the detailed and constructive comments and for recognizing the novelty of CAR. We appreciate the concerns regarding context construction, fair comparison, and robustness. We conducted several additional experiments to address these points.

1. Context construction and realistic evaluation (W1, Q4)

We agree that the original same-class grouping uses ground-truth labels and was intended to demonstrate CAR's potential when coherent contexts are available.

To address this concern, we evaluated CAR on ImageNet-VID, forming contexts using consecutive video frames rather than class labels. Native CL-LoRA achieves 86.93% accuracy, while CAR achieves 92.89%, 95.18%, and 96.56% for M=4,8,16M=4,8,16, respectively. At M=16M=16, the improvement is +9.63 percentage points over 436 frames (paired video-bootstrap 95% CI: [+4.17, +16.51] pp).

We also evaluated synthetic CIFAR-100 streams where images from different classes arrive in task-persistent sequences. Contexts are formed using fixed consecutive windows without knowing class labels or task boundaries. At M=4M=4, accuracy improves from 87.94% to 92.63% and 95.63% for average task durations of 16 and 32 observations, respectively.

These results show that CAR can improve accuracy without requiring same-class grouping.

2. Comparison with TUNA and multiple seeds (W2, Q1–Q2)

We reproduced the complete TUNA framework, including its task-specific and universal adapters, using three independent training seeds with a fixed class order.

Native TUNA achieves 91.60 ± 0.06% final accuracy on CIFAR-100. We then replaced its individual adapter selection with contextual entropy averaging, keeping the trained models and other prediction steps unchanged. At M=16M=16, accuracy increases to 94.54 ± 0.23% under same-class contexts (+2.94 pp). On synthetic task-persistent streams, accuracy reaches 92.80–94.25%, depending on task duration.

These results show that contextual aggregation can also benefit TUNA without changing its training. We clarify that CAR at M=1M=1 shares the minimum-entropy selection idea with TUNA but is not equivalent to the complete TUNA method. The three runs use different training seeds, not different class orders.

3. Effect of prior strength (Q3)

We agree that our original setting (λ=1\lambda=1) effectively behaves like hard task selection. We therefore evaluated different values of λ\lambda, from 0 to 2.

On synthetic CIFAR-100 streams, reducing λ\lambda to 0.05 improves accuracy from 88.47% to 95.11% at M=8M=8. At M=16M=16, using λ=0.01\lambda=0.01 improves accuracy from 81.02% to 92.83%.

This shows that a weaker preference can reduce errors when different tasks appear in the same context. These are additional sensitivity results, rather than independently validated parameter choices.

4. CL-LoRA comparison (Q1, Q5)

We would like to clarify that CAR is an inference-time addition to existing CIL methods, not a new training method. Our main comparisons use exactly the same trained CL-LoRA model and test images, with and without CAR. Therefore, the observed improvement comes from contextual inference rather than differences in training.

We acknowledge that our reproduced ImageNet-R/A baselines are 3.75/5.18 points below the published results. Our investigation identified differences in training updates and ImageNet-R evaluation coverage (19,024 versus 30,000 archived images). Their individual effects remain unquantified.

For completeness, the M=8M=8 CAR results are +7.39 pp on ImageNet-R and -1.34 pp on ImageNet-A relative to the published CL-LoRA averages, although these are not comparisons under identical experimental conditions. The original CL-LoRA results remain single-seed.

5. Theoretical bound and implementation (Q6–Q7)

We agree that Theorem 1 gives a conservative bound. It establishes that averaging entropy over related observations reduces routing errors under a positive expected entropy margin. It does not guarantee improved classification accuracy. The bound can require much larger context sizes than those observed experimentally; we do not claim it tightly predicts the required MM. We will clarify this distinction.

CAR takes the all-class scores from the frozen CL-LoRA or EASE model and adjusts the scores for each task using the contextual preference. No model parameters are updated. We will clarify the score construction and experimental settings to make the implementation easier to reproduce.

We thank the reviewer again for the suggestions. The new experiments provide additional evidence that contextual aggregation is useful beyond the original same-class setting.

Reviewer_gH2WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Contextual Adapter Routing (CAR), a method for rehearsal-free class-incremental learning that aims to improve the inference performance of continual learning models by exploiting contextual relationships among test samples. CAR aggregates the predictive entropy of task-specific adapters over a group of neighboring test samples and uses the resulting routing preferences to adjust the frozen model's all-class predictions without updating model parameters. The paper evaluates CAR on CL-LoRA and EASE and provides a theoretical bound showing improved routing reliability with increasing context size under the assumption of a positive expected entropy margin.

Critical strengths
  1. Existing modular continual learning methods mainly focus on preserving previously learned knowledge during training and preventing parameter interference caused by learning new tasks.In contrast, this paper investigates the problem from an inference-time perspective, exploring how contextual information among neighboring test samples can improve the accuracy of selecting existing task-specific adapters.

  2. The paper not only reports performance across different continual learning benchmarks but also further analyzes the effects of context size, contextual coherence, different grouping strategies, and unreliable contexts on routing performance. It clearly distinguishes between single-image routing, same-image augmentation, and multiple coherent images, and transparently reports the unsuccessful results of unsupervised grouping methods.

Critical weaknesses
  1. The strong results in Tables 1 and 2 rely on "analysis-only contexts" constructed using ground-truth class labels. Although the labels are not used in entropy computation, they still determine which images are grouped together. The paper's own unsupervised grouping experiments fail to reproduce these gains, with some methods even performing worse than the original model (Appendix Table 5). Therefore, the current results effectively demonstrate the potential of CAR "given coherent contexts," but do not yet establish its benefits in general unlabeled deployment scenarios.

  2. The theoretical guarantee depends on the assumption of a positive expected entropy margin and mainly establishes that CAR can more reliably select the adapter with the lowest expected predictive entropy. However, it does not prove that this adapter necessarily corresponds to the correct task, nor does it directly guarantee improvements in final classification accuracy.

Questions for authors
  1. Could the authors report end-to-end accuracy and grouping purity on test streams where grouping is determined solely by information available during real-world deployment, without exploiting class-based ordering?

  2. When reliable coherent contexts are unavailable, could the authors provide a confidence-gating threshold validated independently and report results under different levels of context mixing to clarify the applicability of the method?

1 comment

We thank the reviewer for the thoughtful comments and for recognizing the value of using contextual information at inference time. We conducted additional experiments to address the concerns regarding context construction and reliability.

1. Context construction (W1, Q1)

We evaluated CAR on ImageNet-VID, forming contexts from consecutive video frames without using ground-truth labels for grouping. Native CL-LoRA achieves 86.93% accuracy, while CAR achieves 92.89%, 95.18%, and 96.56% for M=4,8,16M=4,8,16, respectively. At M=16M=16, the improvement is +9.63 percentage points over 436 frames, with a paired video-bootstrap 95% CI of [+4.17, +16.51] pp.

We also evaluated CAR on synthetic CIFAR-100 streams where different classes arrive consecutively within the same task. Unlike the original same-class experiments, we form fixed windows without using labels or task boundaries. At M=4M=4, accuracy improves from 87.94% to 92.63% and 95.63% for average task durations of 16 and 32 observations, respectively.

We also examined context purity, including windows containing observations from multiple tasks. The results show that CAR can benefit from task-level coherence even when images belong to different classes. These experiments demonstrate that same-class grouping is not necessary for contextual routing to be effective.

2. Unreliable contexts and confidence (Q2)

We further analyzed what happens when a context contains images from different tasks. We found that CAR generally benefits images from the dominant task but can harm minority-task images because the same task preference is applied to the whole context.

We therefore tested a simple approach that detects changes between neighboring observations and splits the context accordingly. The detection thresholds were selected using 2,000 development images and kept fixed when evaluating 8,000 different images.

With this approach, accuracy improves from 88.06% (native) to 94.73% at M=8M=8 and 93.58% at M=16M=16, averaged across four task-persistence settings. Original CAR without splitting achieves 88.51% and 81.04%, respectively.

These results show that detecting changes within contexts can substantially reduce errors. We present this as an additional experiment rather than a change to the original CAR method. The original confidence fallback is more conservative and often rejects useful contextual information. The development and evaluation images are disjoint, although both come from previously studied CIFAR-100 data.

3. Theoretical guarantee (W2)

We agree with the reviewer that the theorem does not guarantee that the lowest-entropy adapter is always the correct one. It establishes that, under the stated assumptions, averaging entropy across related observations reduces the probability of selecting an adapter with higher expected entropy.

The theorem explains why contextual information can make adapter selection more reliable, but does not directly guarantee better classification accuracy. We will clarify this distinction in the paper.

We thank the reviewer again. We hope these additional experiments help clarify when CAR can improve inference and how context reliability affects its performance.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your efforts so far. As the discussion period is coming to an end, I would like to kindly encourage you to review the authors' rebuttals and engage in further discussions, particularly if the authors have addressed your questions or concerns.

Thank you for your time and contributions!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.