Orthogonal Selection of Gradient-Free Task Posteriors for Class-Incremental Learning

In class-incremental learning (CIL), a prediction is correct only when both within-task prediction (WP) and task-id prediction (TP) succeed. Methods built on a frozen pre-trained model realize TP through prompt matching, expert routing, or task-identity heads and learn WP within the selected modules. Yet, their modules and classifier heads are trained with a cross-entropy loss restricted to the classes of the current task. This paper shows that this objective is invariant to every logit shift that is constant within each task, and its gradient has no component in the task-level subspace of the logits, which affects not WP but TP. The same subspace is exactly the set of additive corrections that leave WP unchanged for every logit vector. Building on these two results, we propose a method that supplies TP through orthogonal selection: a task posterior, computed in closed form from class-wise first- and second-order moments of frozen features with shared whitening and rank-4 class-specific corrections, is added to the logits without any gradient path. Across 15 methods, five datasets, 14 task splits, and five seeds (1,050 runs under a common training protocol), TP fails in 92.8–97.4% of the errors of every existing method, and the variation of final accuracy across methods is almost entirely a variation of TP. The proposed method raises TP by 16.70 points and final accuracy by 15.74 points while leaving WP unchanged, reaches 78.25% average final accuracy, and its gain increases monotonically with the number of tasks on all five datasets.