acceptodds
Under review as a conference paper at ICLR 2027

Projected Routing In Scaled-up Mappings for Continual Post-Training

Abstract

Continual post-training of pre-trained models offers a promising way to acquire new classes while retaining a shared representation and allocating adaptation capacity to individual tasks. However, this setting introduces a fundamental inference problem: the task associated with an input is unknown, yet selecting the correct task-specific adaptation is necessary for accurate classification. Existing approaches either concatenate features from all task-specific modules, which grows the representation dimensionality, or evaluate every module under a selection criterion such as distance or predictive entropy. The latter family is effective when the correct expert is selected, yielding high task-local classification accuracy. However, expert selection remains the bottleneck: batch-level criteria assume that test samples belong to the same task, while the per-sample alternative requires evaluating every stored module, making inference cost grow linearly with the number of tasks. We introduce PRISM (Projected Routing In Scaled-up Mappings) that explicitly separates task identification from classification. It learns a globally consistent router in a high-dimensional random feature space, enabling instance-wise selection of a single task-specific adaptation module without evaluating all adapters. This design provides constant-path inference while maintaining a shared routing function throughout the continual sequence. Across standard class-incremental benchmarks, PRISM outperforms existing methods, with particularly pronounced improvements under challenging distribution shifts. These results demonstrate that explicitly decoupling task identification from class prediction provides an effective and scalable mechanism in continual post-training.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Routing, Foundation Models, PEFT

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6KdaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on modular continual post-training in class-incremental learning. To avoid the linear inference cost of evaluating all task experts, the authors propose PRISM, which decouples task selection from class prediction. PRISM maps frozen features into a high-dimensional random space and fits a global ridge router in closed form to predict the task ID, activating only one task-specific module per test sample. Evaluated on six benchmarks, PRISM shows notable gains under distribution shifts like ImageNet-A/R while maintaining constant-time inference cost.

Critical strengths
  1. Clear motivation. The paper shows the weakness of batch-level expert selection. On ImageNet-A, TOSCA's accuracy drops from 65.37% to 9.45% with shuffled batches. This highlights a limitation of batch-level routing.

  2. Efficient inference. PRISM selects only one expert per sample, avoiding the cost of evaluating all experts. The experiments show a good balance between accuracy and inference speed.

Critical weaknesses
  1. Limited novelty. PRISM mainly combines RanPAC's random projection and ridge regression with TOSCA's existing experts. The task router is also closely related to aggregating RanPAC's class scores by task. The main technical contribution seems limited.

  2. Order dependence. The proposition in Sec. 4.3 assumes a fixed backbone, but the backbone is trained on Task 1. Changing the first task may therefore affect the whole system. The paper also refers to an order-invariance experiment in Sec. 5.3, but the results are missing.

  3. Missing baseline comparison. Per-sample TOSCA reaches 59.10% on ImageNet-A (Table 1), close to PRISM's 59.51% (Table 2). However, it is missing from the main comparison. Gains on CUB, OmniBenchmark, and VTAB are also within seed noise.

  4. Limited routing accuracy. PRISM achieves only 63.86% routing accuracy on ImageNet-A, with a 22.48-point gap in final accuracy compared to oracle routing. Task selection remains a major source of error.

Questions for authors
  1. How does PRISM's router differ from predicting task IDs by summing RanPAC's class scores for each task?

  2. How sensitive is PRISM to the first task? Have the authors tested different task orders?

  3. Can the authors compare PRISM with per-sample TOSCA on all six benchmarks, using the same backbone? Both accuracy and inference time would be helpful.

  4. Can the authors provide per-task routing accuracy or a confusion matrix to explain the remaining routing errors?

3 comments

Summary. Thank you for the detailed review. We ran all requested experiments: max routing beats the sum rule by up to 9.2 points (W1); changing the first task moves final accuracy by at most 1.2 points (W2); PRISM outperforms per-sample TOSCA on other benchmarks (W3); routing errors show no forgetting or recency bias (W4).

W1 & Q1: Novelty and summing class scores per task

PRISM routes in two steps: it builds a score for each task from the class-ridge scores, then selects the task with the highest score. The two rules differ only in the first step. PRISM takes the max class score within each task, t̂ = argmaxₜ max{s(c) : c ∈ Y(t)}; summing gives t̂ = argmaxₜ Σ{ s(c) : c ∈ Y(t) }. Since ridge is linear in its targets, Eq. 4 as written (ridge on task labels) equals the sum rule. All results in the paper and the code use the max rule, and we will correct Eq. 4. To answer your question directly, we also ran the sum rule as a new experiment. Routing by the sum lowers final accuracy on every benchmark, by 5.8 points on IN-R and 9.2 on IN-A:

A_tCIFARIN-RIN-A
Sum of class scores per task90.86 ± 0.3975.57 ± 0.2750.05 ± 1.25
Class ridge, max per task (PRISM)91.40 ± 0.3081.41 ± 0.2059.26 ± 0.98
Difference+0.5+5.8+9.2
Router accuracy, sum / max91.33 / 91.7977.68 / 83.4854.79 / 63.84

We agree the components come from RanPAC and TOSCA. The contribution is the factorisation: the readout only selects a task, and that task's expert classifies within it. On IN-A, routing is correct for 63.8% of inputs and the expert is correct on 92.8% of these, against 54.7 for RanPAC classifying directly. TOSCA reaches 59.1 only by evaluating every expert per sample (9.45 with batch selection). PRISM reaches 59.5 with one backbone pass and one expert, matching that accuracy at RanPAC's cost.

W2 & Q2: Order dependence and sensitivity to the first task

You are right that the Proposition holds for a fixed backbone. Since the backbone is adapted to the first task, the full system is not order-invariant by construction. We will state this scope explicitly in Section 4.3 and add the missing results to Section 5.3.

In practice the effect is small. First, the main results already vary the order: each of the 5 seeds uses a different random class order, so the first task differs across seeds and the standard deviations in Table 2 include this variation. Second, we ran a controlled test: we keep the task partition fixed and permute the task order three times, each with a different first task (original tasks 3, 5 and 6), over 5 seeds:

A_TOriginal orderPermutation 1Permutation 2Permutation 3Max. deviation
IN-R81.41 ± 0.2081.10 ± 0.6480.77 ± 0.3781.08 ± 0.310.64
IN-A59.26 ± 0.9858.43 ± 1.3860.43 ± 1.4358.10 ± 1.531.17

Final accuracy changes by at most 0.64 points on IN-R and 1.17 on IN-A, within the seed standard deviation on IN-A. Routing accuracy stays at 82.90–83.14 on IN-R (83.48 original) and 62.80–64.57 on IN-A (63.84 original). PRISM is therefore robust to the choice of first task on these benchmarks.

W3 & Q3: Per-sample TOSCA on all six benchmarks

We ran per-sample TOSCA on all six benchmarks with the same ViT-B/16-IN21K backbone, 5 seeds and the protocol of Section 5.1 (A_bar / A_t):

CIFARCUBIN-RIN-AOmniVTAB
Per-sample TOSCA86.82 / 81.4482.75 / 72.1365.49 / 55.5060.12 / 59.1070.46 / 58.9688.83 / 78.83
PRISM94.45 / 91.4293.62 / 90.0985.84 / 81.3467.27 / 59.5186.42 / 80.3695.87 / 92.78

PRISM improves A_t by 10.0 to 25.8 points on five benchmarks. ImageNet-A is the one case where per-sample TOSCA is close in A_t (59.10 vs 59.51, within noise), but PRISM is 7.2 points higher in A_bar. On inference cost, per-sample TOSCA evaluates every expert on every input (≈390 GFLOPs/sample, Section 5.2), while PRISM uses one backbone pass and one expert (≈15–20 GFLOPs, ≈3 ms/sample). We add this row to Table 2.

On CUB, OmniBenchmark and VTAB, we agree the gains over prior methods are within noise and claim parity there (Section 5.1). We will state this in the abstract and conclusion.

W4 & Q4: Per-task routing accuracy and remaining errors

Routing confusion matrices after the final task, over all seeds:

Routing accuracy by task position (%)Lowest taskHighest taskOverallErrors routed to an earlier / later task
CIFAR10089.7294.7291.7950.8 / 49.2
CUB (20 tasks)87.6894.2090.7051.4 / 48.6
IN-R (10 tasks)81.5585.7883.4853.5 / 46.5
IN-A (10 tasks)59.3765.9863.8451.1 / 48.9
Omni (10 tasks)78.3985.5482.3448.6 / 51.4
VTAB (5 tasks)93.0195.8994.4548.5 / 51.5
  • No router forgetting: accuracy shows no trend with task position, as the closed-form refit predicts.
  • No recency bias: misrouted samples go to earlier and later tasks about equally often, so errors reflect task overlap.
  • Hard samples: on the 36% of IN-A samples PRISM misroutes, even the correct expert reaches only ~62% (oracle: 81.68).

Full matrices will be added to the appendix.

We believe these results answer the points raised in the review and would appreciate it if you reconsidered your score in light of them. We are glad to answer further questions during the discussion period.

Reviewer_6KdaOpenReview ↗

Response to rebuttal

Thank you for the detailed rebuttal and additional experiments. Your responses largely address my concerns regarding task-order sensitivity (W2), per-sample TOSCA comparisons (W3), and routing-error analysis (W4).

However, my main concern on novelty (W1) remains, and the rebuttal reveals a significant mismatch between the paper and the implementation: the actual router simply uses RanPAC's top-1 class prediction to select a TOSCA expert, while the task-level ridge formulation written in Eq. 4 performs substantially worse (dropping 9.2 points on ImageNet-A). Given the incremental technical contribution, parity with prior methods on half of the benchmarks, and the need to revise the core method formulation, I maintain my current rating.

Reviewer_WE4aOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

the work has tried to work on the problem of class-agnostic, class-incremental learning for image classification. Or at least, I think this is what it has been trying to do.

Critical strengths

None

Critical weaknesses

I think the work overpromises in its title and presentation more than it actually delivers. It uses the concept of post-training, which has been largely used for large language models, VLMs, and, more recently, other large models for the task of continual learning in just image classification. I don't think that's an appropriate usage of the term, and it unnecessarily confuses the reader from the get-go. Further, the title is an inaccurate representation of the work.

The abstract is confusing. Continual post-training is not an established or concrete framework to be used as such in the abstract. It is quite unclear what is being referred to. What does "Continual post-training ... Allocating adaptation capacity to individual tasks." actually mean, which is the very first sentence of the abstract. The next sentence of the abstract is completely unintelligible without adequate context - "However, this setting introduces a fundamental inference problem: the task associated with an input is unknown, yet selecting the correct task-specific adaptation is necessary for accurate classification." It is unknown what fundamental inference problems are referred to here. Or what task or input is being referred to here? There is too much LLM usage in the abstract itself, and it is not easily human-readable. I would strongly recommend the authors give their paper to somebody who has already worked and published in this area, and have them read the abstract and induction. If the objective is to do class-agnostic or task-agnostic class-incremental continual learning for image classification, we can refer to it as such. I'm not sure why it is being presented in such a convoluted manner.

In the introduction, what are the modules that you're talking about? Are you referring to low-rank adapters or the different instances of the model themselves? Why aren't they clarified? There suddenly is a routing problem introduced in the second paragraph of the introduction, which probably corresponds to mixture-of-experts kind of models. What does "avoid coupling its decision to the composition of the test batch" even mean? Figure 1 introduces a ridge router and doesn't explain anything about it. The last paragraph of the introduction is incomprehensible.

There is no proper definition of components/modules. The key design principle is basically not key at all, given how confusing it is. It's also unclear what exactly the proposition and theory our contribution to him. What do you mean by routing space, and what is ridge readout? Why is this proposition even introduced? The theory does not seem to be for human readability and doesn't exactly seem to support much of anything in the paper.

Questions for authors

the paper currently requires complete rewriting, minimum usage of LLMs, and probably an appropriate review by a senior researcher. There's a chance that the authors can do this during the rebuttal period, but I'm not inclined towards re-reviewing the entire paper.

1 comment

Given the reviewer's stated unwillingness to engage with the rebuttal and several comments that go beyond constructive scientific criticism, we find it unfortunate that a constructive exchange may not be possible. We nevertheless provide our clarifications below for the Area Chair's consideration: we respectfully disagree with their characterizations in the review. We address each concern based on the definitions and explanations already provided in the manuscript.

Terminology and problem formulation. The term continual post-training deliberately describes our setting: incrementally adapting a pretrained model to downstream classification tasks without revisiting pretraining. We do not claim that post-training is a new paradigm or exclusive to language models. Rather, we study its application to continual adaptation of pretrained vision models. Focusing on image classification does not make this terminology inappropriate, nor does the title misrepresent our work. Section 3.1 explicitly formalizes the problem as class-incremental learning with disjoint task-specific classes, where predictions must cover all observed classes without task identity at inference or access to previous training samples. We investigate a specific inference problem within this established setting, not a new learning paradigm.

Abstract and introduction. Reviewer may find some terminology insufficiently explicit. However, the problem being addressed is well-defined: a model encounters inputs from incrementally arriving classes, without being provided with task identity at inference, while task-specific adaptation must be selected appropriately. The routing mechanism addresses precisely this challenge. The references to modules refer to task-specific adaptation components (prompts and adapters), while the ridge router refers to the ridge classifier that finds the relevant module. Section 3.2 provides three empirical observations motivating it: (i) batch-level expert selection fails when samples from different tasks are interleaved; (ii) the frozen [CLS] representation provides insufficient task separability; and (iii) gradient-based sequential training of the router itself suffers from catastrophic forgetting. These observations motivate the three corresponding design requirements and the proposed analytic routing mechanism. The phrase "avoid coupling its decision to the composition of the test batch" distinguishes batch-level from instance-wise routing. Batch-level selection can choose an inappropriate expert for mixed-task batches, whereas PRISM predicts the task independently for each input, as demonstrated in Table 1 and discussed in Section 3.2. Figure 1 illustrates the three alternatives: concatenating task-specific representations, entropy-based selection across adapters, and PRISM's projected-feature router selecting a single adapter.

Design principle, methodology, and theoretical contribution. The reviewer questions the meaning and motivation of the key design principle. This principle follows directly from the three observations in Section 3.2: components that must operate across all tasks need to retain information from previous tasks, whereas task-specific components can be trained on their respective tasks and frozen thereafter. The proposition establishes the order independence of the accumulated ridge solution. Because its sufficient statistics are sums of per-sample outer products, they are independent of the order in which tasks are observed. Consequently, the resulting router is exactly identical to the solution obtained with simultaneous access to all observed task data. This property provides the formal justification for using accumulated statistics without revisiting previous samples and explains why the router does not inherit the forgetting dynamics of a sequentially updated predictor. It establishes a specific property essential to our proposed estimation procedure.

Overall assessment and additional remarks. We can understand that some terminology and transitions could be improved. However, we strongly disagree that the paper requires complete rewriting or that its methodological and theoretical contributions are fundamentally unclear. The problem setting, component definitions, routing mechanism, and theoretical proposition are explicitly introduced and motivated.

We also note that the review's assertion regarding excessive LLM usage is speculative and does not identify a specific scientific or technical deficiency. We respectfully maintain that the scientific contribution should be evaluated based on the proposed methodology, its theoretical properties, and the experimental evidence presented, rather than broad characterizations of the manuscript. We hope these clarifications help the Area Chair assess the actual technical merits of the paper.

Reviewer_yY6GOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents three important findings and then constructs an efficient method to classify tasks based on these three findings. The three findings are -

  1. TOSCA performs well with sample-level entropy selection but performs poorly over mixed tasks in a batch.
  2. The task identity cannot be properly inferred from the frozen [CLS] representation.
  3. If we try to route a model incrementally, the model routing itself becomes a continual learning problem and is prone to forgetting. The paper then presents their method - PRISM, which aims to fix these three problems. First, a random projection of the frozen [CLS] representation into a high-dimensional space improves the separation of classes. It then predicts a sequential ridge regression for P(t∣x)P(t|x). The method outperforms most of the known methods by a considerable margin in the presented experiments. The ablations provide some interesting findings, such as what the dimension of the random projection should be, etc
Critical strengths

The paper is very well presented and principled. I believe the simplicity of the method is its main strength since it is based on well-known existing principles. The combination of random projections to separate [CLS] representations and then using a ridge regressor to classify tasks from data points, while outperforming most of the other methods by quite a margin, is actually a very nice result. The paper also takes care to compare latency, which is actually important for deployment or practical applications.

Critical weaknesses

The per-sample TOSCA described in Section 3.2 actually has an accuracy that is comparable to PRISM. Although I understand that per-sample TOSCA is more expensive because you would need to run all the adapters for each sample, it would be nice to see how per-sample TOSCA performs on the other datasets.

I understand that PRISM will be computationally much cheaper than per-sample TOSCA if batch inference is considered. Still, I believe the per-sample TOSCA comparison would provide an answer on whether PRISM is mainly a superior computational method or whether it also increases the accuracy significantly over per-sample TOSCA (which I believe can be considered SOTA). PRISM uses a relatively large random-feature dimension (D=15,000D=15{,}000) and refits the ridge router after each task.

Questions for authors

Could the authors report the per-task router fitting time and memory cost? From my understanding, the reported latency appears to cover inference only, so the computational overhead of the analytic update is currently unclear.

2 comments

Summary. Thank you for the positive review, and for noting the simplicity of the method and the latency comparison. We answer your questions below.

Adding per-sample TOSCA results. Thank you for the constructive feedback. We evaluated TOSCA with per-sample routing and found that it substantially improves performance, bringing it closer to several competing approaches. However, this comes at a considerable inference-time cost, while still falling short of PRISM across all datasets. In particular, per-sample TOSCA incurs 2.7–14.2× higher inference latency than PRISM. We will add these results to the comparison tables and included a discussion in the main results in the camera-ready version.

Table: Average Accuracy / Incremental Accuracy on each benchmark �0��0�

CIFARCUBIN-RIN-AOmniVTAB
Per-sample TOSCA86.82 ± 0.83 / 81.44 ± 0.4482.75 ± 1.89 / 72.13 ± 1.1665.49 ± 0.65 / 55.50 ± 0.2760.12 ± 1.72 / 59.10 ± 0.8470.46 ± 1.40 / 58.96 ± 0.6588.83 ± 2.89 / 78.83 ± 2.63
PRISM (ours)94.45 ± 0.65 / 91.42 ± 0.3193.62 ± 0.59 / 90.09 ± 0.2285.84 ± 0.36 / 81.34 ± 0.3167.27 ± 1.79 / 59.51 ± 0.7286.42 ± 0.93 / 80.36 ± 0.3995.87 ± 0.43 / 92.78 ± 0.19

Table: Inference time (ms/sample, A100)

CIFARCUBIN-RIN-AOmniVTAB
Per-sample TOSCA44.7744.9922.5322.8022.6111.38
PRISM (ours)3.173.473.854.183.423.00

Memory and Time Costs. All components except C\mathbf{C} and the task-specific TOSCA modulation EE have fixed memory requirements, independent of sequence length. The projection matrix R\mathcal{R} requires no persistent storage, as its entries are regenerated from a fixed seed. The Gram matrix G\mathbf{G} is preallocated at its final size and never resized. Memory growth therefore comes from two components: EE, which adds 3.16 MB per task, and C\mathbf{C}, which adds one MM-dimensional column per class (0.04 MB per class). We also report the additional per-task router fitting time, measured on an A100, which ranges from 0.40 to 0.81 seconds across datasets. For example, in CIFAR100 with 20 task split, PRISM spends ~9.4 seconds in total for fitting the router. These costs will be included in the camera-ready version.

Table: Memory decomposition for PRISM

ComponentShapeSizeGrowth
Gram matrix G\mathbf{G} (32-bit)D×DD \times D, upper triangle202.6 MB0
Correlation C\mathbf{C}D×∥Y∥D \times \\\|\mathcal{Y}\\\|0.04 MB/class+0.04 MB/class
Projection R\mathcal{R}Seed only0 MB0
TOSCA modulation EE4×d×r4 \times d \times r3.16 MB/task+3.16 MB/task

Table: Additional router fitting time per task (seconds, A100)

CIFAR100CUBIN-RIN-AOmniVTAB
Router fitting time0.470.450.810.430.620.40

We believe these results answer the points raised in the review and would appreciate it if you reconsidered your score in light of them. We are glad to answer further questions during the discussion period.

Reviewer_yY6GOpenReview ↗

Response to Authior Rebuttal

The comparison with per-sample TOSCA is a nice addition. However, I will keep my current score as the two main components of the paper - random projections and ridge regression are both well known.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.