LOSS AND MARGIN POLICIES FOR CONTINUAL LEARNING

Models that are updated as new tasks arrive must learn new capabilities while preserving those already in use. The loss applied to new-task data is a rarely examined control over this trade-off: cross-entropy (CE) keeps pushing examples that are already classified correctly, whereas a hinge loss stops once a prescribed logit gap is reached. We compare CE, a multiclass hinge, and the high error margin (HEM) loss for task-incremental learning under matched update budgets and replay opportunities, with validation-based selection and separate measurement of new-task acquisition and old-task retention. On Split-CIFAR-100 the selected hinge–HEM ranking reverses with the training budget: HEM leads at 10 epochs per task and hinge at 40, because quadrupling the budget costs the selected CE and HEM policies 3.6–3.8 points of old-task accuracy while hinge gains 1.0; the size of the longer-budget advantage depends on the acquisition requirement used for selection. A 2×2 ablation with margin and learning rate fixed reproduces the reversal within one loss family: HEM-style active-sample averaging helps at the short budget (+0.55 points) and hurts at the long one (−3.02). On Split-TinyImageNet, whose test split was held out until the confirmation run, both margin losses beat CE with replay by about five points through retention rather than acquisition, but hinge and HEM are not distinguished (+0.13 [−0.34, 0.59]). In an all-hinge study, geometrically discounting the margin across tasks improves final accuracy over a fixed margin by +1.80 points (5% discount, 128 seeds) and +4.80 (20%, 32 seeds), because forgetting falls by more than acquisition does. An elementary gradient analysis explains why margin, normalization, and learning rate are distinct controls. The results give budget-dependent operating points for the new-task loss and quantify the acquisition cost of the retention they buy.