- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper systematically investigates the impact of inference strategies in parameter-efficient fine-tuning (PEFT)-based class-incremental learning (CIL). By fixing task-specific LoRA experts and varying the backbone, classification head, routing signal, and expert combination strategy, the authors show that inference choices can lead to substantial performance differences, with accuracy varying by 19–38 percentage points across benchmarks. The proposed combination of a regularized least-squares classifier and Mahalanobis-weighted expert ensemble achieves an average accuracy of 78.12% across eight settings, outperforming RanPAC (76.88%). Further experiments demonstrate that the proposed inference strategies can improve existing PEFT-based CIL methods, highlighting the importance of inference design beyond expert training.
Critical strengths
- The paper highlights an underexplored aspect of PEFT-based continual learning: inference design. Its controlled analysis demonstrates that inference choices can substantially affect performance, providing useful insights for the community.
- The authors systematically investigate multiple inference components, including classification heads, routing signals, and expert combination strategies. The analysis of expert interference and routing reliability helps explain why different inference strategies perform differently.
- The proposed inference configuration improves multiple existing methods across a range of experimental settings. The paper also discusses the trade-off between training efficiency and inference cost, making the findings practically relevant.
Critical weaknesses
- The paper demonstrates that changing the classification head and inference rule can substantially affect accuracy when the LoRA experts are fixed. However, the main controlled analysis removes continual-learning-specific mechanisms, while the transfer study covers only three existing methods. The results also show that replacing the readout does not consistently improve performance across methods and datasets. This leaves an important question unresolved: whether the identified inference configurations are generally effective or depend strongly on the feature representations and inter-expert relationships induced by a particular training method. A more systematic cross-evaluation of multiple training methods and inference configurations, ideally with matched backbones and data splits, would help quantify this interaction and clarify the extent to which the findings generalize beyond the mechanism-free setting.
- The paper compares entropy- and Mahalanobis-based routing and includes a candidate-expert stress test, but the analysis is limited to ImageNet-R and ImageNet-A and primarily reports final classification accuracy. Final accuracy alone does not reveal whether a routing strategy selects the most appropriate expert, nor whether an incorrect selection is harmful because of the selected expert itself or because of how its predictions are combined with those of other experts. This distinction is important given the paper's observation that including irrelevant experts can dilute predictions. The analysis would be more informative if it separately measured expert-selection quality and downstream accuracy, for example by comparing the selected experts with oracle-selected experts and reporting the performance gap under different candidate sets. Such an analysis could better establish when each routing signal is effective and explain the mechanisms behind its observed performance.
- The proposed lstsq + maha_e configuration achieves an average accuracy of 78.12% across eight settings, compared with 76.88% for RanPAC. Although this result supports the competitiveness of the proposed configuration, the absolute difference is relatively modest, and the paper reports results averaged over only three runs. Furthermore, the proposed configuration does not dominate across benchmarks: it trails RanPAC on CUB-200 and performs differently across the other datasets. Without paired statistical tests or confidence intervals for the aggregate comparison, it is difficult to determine whether the overall advantage is robust to random seeds and data splits. Reporting paired per-seed differences and uncertainty estimates, alongside per-dataset results, would make the comparative claims more reliable.
Questions for authors
Please refer to the Weaknesses section.
2 comments
The response has addressed my concerns, and I therefore raise my score.
Response to Reviewer XEtW
We thank the reviewer for the constructive suggestions; they led to the main new experiments in this rebuttal.
W1. Do inference choices transfer beyond independently trained experts? Yes; see General Response 1(GR1), where the study now covers further published methods. On EASE's own adapters we evaluated all 18 head × rule configurations from our grid, with no retraining and no exemplars:
Replacing EASE's native inference with lstsq + maha_e improves 6 of 8 settings (+1.9 on average, +8.3 on ImageNet-A, T=20). The best readout can still depend on the representation, and we will say so.
W2. Is the bottleneck expert selection or prediction combination? We now measure the two separately on all 8 settings (lstsq + maha_e, 3 seeds):
Combination loss is the larger of the two in all 8 settings, accounting for 71–97% of the gap between oracle and ensemble accuracy. Even when the correct expert is selected, it receives only 41–58% of the ensemble weight.
Mahalanobis does improve selection; for example, top-1 retrieval on ImageNet-A (T=20) rises from 40.7% to 54.5%. Yet final accuracy rises by only 1.8 points.
We also ran a stress test with the head and combiner fixed. Mahalanobis top-k gives the best candidate set in all 8 settings (+3.3 to +6.8 over random-k). However, forcing the oracle expert into a set of three still leaves a gap of 3.2–9.4 points to oracle-only inference. The open problem is therefore combination, not only retrieval.
W3. How robust is the gain over RanPAC? The gain is modest and depends on the benchmark, and we will state this explicitly. Mean accuracy is 78.12% for ours and 76.88% for RanPAC.
RanPAC draws its class orders differently, so the runs are not paired, and we use unpaired Welch t-tests:
The strongest statistical evidence comes from comparisons inside our grid, where the experts, class orders and seeds are identical:
The higher-capacity ridge-10k head is reported in GR2.