- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on modular continual post-training in class-incremental learning. To avoid the linear inference cost of evaluating all task experts, the authors propose PRISM, which decouples task selection from class prediction. PRISM maps frozen features into a high-dimensional random space and fits a global ridge router in closed form to predict the task ID, activating only one task-specific module per test sample. Evaluated on six benchmarks, PRISM shows notable gains under distribution shifts like ImageNet-A/R while maintaining constant-time inference cost.
Critical strengths
-
Clear motivation. The paper shows the weakness of batch-level expert selection. On ImageNet-A, TOSCA's accuracy drops from 65.37% to 9.45% with shuffled batches. This highlights a limitation of batch-level routing.
-
Efficient inference. PRISM selects only one expert per sample, avoiding the cost of evaluating all experts. The experiments show a good balance between accuracy and inference speed.
Critical weaknesses
-
Limited novelty. PRISM mainly combines RanPAC's random projection and ridge regression with TOSCA's existing experts. The task router is also closely related to aggregating RanPAC's class scores by task. The main technical contribution seems limited.
-
Order dependence. The proposition in Sec. 4.3 assumes a fixed backbone, but the backbone is trained on Task 1. Changing the first task may therefore affect the whole system. The paper also refers to an order-invariance experiment in Sec. 5.3, but the results are missing.
-
Missing baseline comparison. Per-sample TOSCA reaches 59.10% on ImageNet-A (Table 1), close to PRISM's 59.51% (Table 2). However, it is missing from the main comparison. Gains on CUB, OmniBenchmark, and VTAB are also within seed noise.
-
Limited routing accuracy. PRISM achieves only 63.86% routing accuracy on ImageNet-A, with a 22.48-point gap in final accuracy compared to oracle routing. Task selection remains a major source of error.
Questions for authors
-
How does PRISM's router differ from predicting task IDs by summing RanPAC's class scores for each task?
-
How sensitive is PRISM to the first task? Have the authors tested different task orders?
-
Can the authors compare PRISM with per-sample TOSCA on all six benchmarks, using the same backbone? Both accuracy and inference time would be helpful.
-
Can the authors provide per-task routing accuracy or a confusion matrix to explain the remaining routing errors?
3 comments
W3 & Q3: Per-sample TOSCA on all six benchmarks
We ran per-sample TOSCA on all six benchmarks with the same ViT-B/16-IN21K backbone, 5 seeds and the protocol of Section 5.1 (A_bar / A_t):
| CIFAR | CUB | IN-R | IN-A | Omni | VTAB | |
|---|---|---|---|---|---|---|
| Per-sample TOSCA | 86.82 / 81.44 | 82.75 / 72.13 | 65.49 / 55.50 | 60.12 / 59.10 | 70.46 / 58.96 | 88.83 / 78.83 |
| PRISM | 94.45 / 91.42 | 93.62 / 90.09 | 85.84 / 81.34 | 67.27 / 59.51 | 86.42 / 80.36 | 95.87 / 92.78 |
PRISM improves A_t by 10.0 to 25.8 points on five benchmarks. ImageNet-A is the one case where per-sample TOSCA is close in A_t (59.10 vs 59.51, within noise), but PRISM is 7.2 points higher in A_bar. On inference cost, per-sample TOSCA evaluates every expert on every input (≈390 GFLOPs/sample, Section 5.2), while PRISM uses one backbone pass and one expert (≈15–20 GFLOPs, ≈3 ms/sample). We add this row to Table 2.
On CUB, OmniBenchmark and VTAB, we agree the gains over prior methods are within noise and claim parity there (Section 5.1). We will state this in the abstract and conclusion.
W4 & Q4: Per-task routing accuracy and remaining errors
Routing confusion matrices after the final task, over all seeds:
| Routing accuracy by task position (%) | Lowest task | Highest task | Overall | Errors routed to an earlier / later task |
|---|---|---|---|---|
| CIFAR100 | 89.72 | 94.72 | 91.79 | 50.8 / 49.2 |
| CUB (20 tasks) | 87.68 | 94.20 | 90.70 | 51.4 / 48.6 |
| IN-R (10 tasks) | 81.55 | 85.78 | 83.48 | 53.5 / 46.5 |
| IN-A (10 tasks) | 59.37 | 65.98 | 63.84 | 51.1 / 48.9 |
| Omni (10 tasks) | 78.39 | 85.54 | 82.34 | 48.6 / 51.4 |
| VTAB (5 tasks) | 93.01 | 95.89 | 94.45 | 48.5 / 51.5 |
- No router forgetting: accuracy shows no trend with task position, as the closed-form refit predicts.
- No recency bias: misrouted samples go to earlier and later tasks about equally often, so errors reflect task overlap.
- Hard samples: on the 36% of IN-A samples PRISM misroutes, even the correct expert reaches only ~62% (oracle: 81.68).
Full matrices will be added to the appendix.
We believe these results answer the points raised in the review and would appreciate it if you reconsidered your score in light of them. We are glad to answer further questions during the discussion period.
Response to rebuttal
Thank you for the detailed rebuttal and additional experiments. Your responses largely address my concerns regarding task-order sensitivity (W2), per-sample TOSCA comparisons (W3), and routing-error analysis (W4).
However, my main concern on novelty (W1) remains, and the rebuttal reveals a significant mismatch between the paper and the implementation: the actual router simply uses RanPAC's top-1 class prediction to select a TOSCA expert, while the task-level ridge formulation written in Eq. 4 performs substantially worse (dropping 9.2 points on ImageNet-A). Given the incremental technical contribution, parity with prior methods on half of the benchmarks, and the need to revise the core method formulation, I maintain my current rating.
Summary. Thank you for the detailed review. We ran all requested experiments: max routing beats the sum rule by up to 9.2 points (W1); changing the first task moves final accuracy by at most 1.2 points (W2); PRISM outperforms per-sample TOSCA on other benchmarks (W3); routing errors show no forgetting or recency bias (W4).
W1 & Q1: Novelty and summing class scores per task
PRISM routes in two steps: it builds a score for each task from the class-ridge scores, then selects the task with the highest score. The two rules differ only in the first step. PRISM takes the max class score within each task, t̂ = argmaxₜ max{s(c) : c ∈ Y(t)}; summing gives t̂ = argmaxₜ Σ{ s(c) : c ∈ Y(t) }. Since ridge is linear in its targets, Eq. 4 as written (ridge on task labels) equals the sum rule. All results in the paper and the code use the max rule, and we will correct Eq. 4. To answer your question directly, we also ran the sum rule as a new experiment. Routing by the sum lowers final accuracy on every benchmark, by 5.8 points on IN-R and 9.2 on IN-A:
We agree the components come from RanPAC and TOSCA. The contribution is the factorisation: the readout only selects a task, and that task's expert classifies within it. On IN-A, routing is correct for 63.8% of inputs and the expert is correct on 92.8% of these, against 54.7 for RanPAC classifying directly. TOSCA reaches 59.1 only by evaluating every expert per sample (9.45 with batch selection). PRISM reaches 59.5 with one backbone pass and one expert, matching that accuracy at RanPAC's cost.
W2 & Q2: Order dependence and sensitivity to the first task
You are right that the Proposition holds for a fixed backbone. Since the backbone is adapted to the first task, the full system is not order-invariant by construction. We will state this scope explicitly in Section 4.3 and add the missing results to Section 5.3.
In practice the effect is small. First, the main results already vary the order: each of the 5 seeds uses a different random class order, so the first task differs across seeds and the standard deviations in Table 2 include this variation. Second, we ran a controlled test: we keep the task partition fixed and permute the task order three times, each with a different first task (original tasks 3, 5 and 6), over 5 seeds:
Final accuracy changes by at most 0.64 points on IN-R and 1.17 on IN-A, within the seed standard deviation on IN-A. Routing accuracy stays at 82.90–83.14 on IN-R (83.48 original) and 62.80–64.57 on IN-A (63.84 original). PRISM is therefore robust to the choice of first task on these benchmarks.