Dissecting PEFT-Based Continual Learning: From Expert Training to Inference

Parameter-efficient fine-tuning (PEFT) has become a common strategy for class-incremental learning, with lightweight task-specific experts trained on a frozen pretrained backbone. However, existing methods often change several parts of the system simultaneously, making it difficult to determine whether reported gains arise from expert training or from choices such as the backbone, classification head, routing signal, and inference rule. We address this problem through a controlled component analysis of PEFT-based class-incremental learning. Holding expert training fixed, we vary these components across four benchmarks and two task granularities. Strong frozen-feature methods such as RanPAC provide a demanding reference point, yet a mechanism-free PEFT learner with a regularized least-squares readout and Mahalanobis-weighted expert ensembling achieves the highest mean across the eight settings. The value of an inference rule nevertheless depends on the head with which it is paired. Entropy remains a useful and simpler routing signal, particularly for probability ensembling, while Mahalanobis routing is more reliable when predictive confidence is poorly calibrated. Routing stress tests further show that retrieving the correct expert is not sufficient when unrelated experts dilute its prediction. Applying the same readout and combination choices to experts learned by established methods improves 19 of 24 settings and substantially changes their relative ranking, although the gains remain representation and dataset dependent. These findings show that the inference system surrounding expert learning is a major source of performance variation and should be evaluated separately from the learning mechanism itself.