- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how the objective for new tasks controls acquisition and retention in task incremental learning. It compares cross entropy (CE), multiclass hinge, and HEM under matched update and replay budgets. The selected loss ranking reverses between 10 and 40 epochs on CIFAR100, while both margin losses improve retention over CE on TinyImageNet. A 2×2 ablation examines sample averaging, and a separate hinge study quantifies the acquisition cost of geometric margin discounting.
Critical strengths
-
The paper follows a clear structure from loss design to acquisition and retention analysis. Section 3 defines the measurements, and Section 6 uses Equation 6 to explain how reduced forgetting improves final accuracy despite weaker acquisition. This organization makes the benefits and costs of the proposed policies easy to assess.
-
The experimental design controls training resources and includes substantial replication. Studies A and B branch from a shared checkpoint and match update and replay budgets (Section 3). The 2×2 ablation examines aggregation and averaging separately (Section 5, Figure 1d), while the margin study confirms the 5% discount benefit across 128 paired seeds (Section 6, Table 2).
Critical weaknesses
-
The loss comparison lacks a common acquisition requirement for interpreting the preferred policy. Section 5 shows that removing the loss specific constraints shrinks the E40 validation gap between hinge and HEM from 3.7 to 1.2 points, although the ranking reversal persists. The preferred loss under a shared acquisition target therefore remains unclear.
-
The experimental validation of margin policies lacks tuned baseline comparisons. Studies A and B fix margins without tuning them (Section 3), while Study C compares discounting with a constant margin of 0.25 (Section 6, Table 2). These comparisons leave open whether tuning a smaller constant margin could produce similar benefits, limiting the evidence for the proposed schedule.
Questions for authors
-
Regarding Weakness 1: Under a common absolute acquisition floor, how do the E40 rankings change, and which loss would the practical guidance in Section 7 favor?
-
Regarding Weakness 2: How do the loss rankings and discounting gains change when margins in A/B and the constant margin baseline in C receive comparable validation tuning?
3 comments
Thank you for these additional experiments and for your detailed responses.
While responses to W1 and W2 clarify the experimental boundaries, they also shrink the paper's contribution considerably. Now that the margin-schedule recommendation is withdrawn, the Hinge-vs.-HEM ranking flips across settings without establishing any mechanism. Hence, the main result reduces to a tuning observation on CIFAR-100 and TinyImageNet, which does not hold on SVHN or FashionMNIST and still depends on grid-boundary selections in a restrictive task-incremental setup.
Given this, what do you see as the paper's core contribution?
Thank you for the direct question. We agree that, as of our last reply, the contribution had narrowed to a tuning observation. The revised PDF changes this, and the core contribution is now the following.
Core contribution: a two-axis view of the new-task loss, and a schedule along those axes that works. CE, hinge, and HEM are members of one loss family that differ along two axes (Section 4): class-update sparsity (how many wrong classes one update pushes down) and margin. CE and hinge are fixed points; HEM moves along both axes within each task. This view explains why the hinge-HEM ranking flips across settings: no fixed point is best. It also leads to a method. HCE (Section 7) moves smoothly within each task from sparse hinge updates with a margin to dense CE updates without one, as the learning rate anneals.
What this adds beyond a tuning observation:
- A method that beats tuned baselines, not a ranking between two of them. Without replay, HCE beats tuned HEM (+3.3 CIFAR-100, 16/16; +1.9 TinyImageNet, 8/8), tuned CE, EWC (+5.0 / +4.6), and experience replay (+8.5 / +12.2). With replay for every loss, it beats HEM with replay (+3.9 / +2.1) and ER (+9.9 / +12.8). Every baseline selects its own learning rate on validation seeds.
- Evidence about mechanism. Preregistered controls test the reading that sparse updates protect old tasks while steps are large, and dense updates finish the new task once steps are small. Reversing the schedule loses 24.5 points (forgetting rises to 49.7), an abrupt switch loses 6.4, and a constant within-task learning rate removes 6.1 points of the gain. Against tuned HEM, HCE acquires the same on CIFAR-100 and forgets less, so it does not win by learning less.
- The margin result, restated. You were right that our across-task margin discount did not survive a tuned learning rate. The schedule that adds to a tuned learning rate is the one within each task: HCE beats the hinge with a tuned learning-rate decay by +3.3 and adds +2.4 (31/32) on top of the best margin-discount policy.
Scope we state plainly: all results are task-incremental; HCE was not run on SVHN or FashionMNIST; among the tested settings with a learning-rate anneal, the one exception is the CE-prefix replay design of Section 5 (tuned HEM +1.6), where only the last three tasks use the new loss. Several baseline selections lie at grid edges; CE's best rate is at the top of a grid we extended twice; each doubling of the rate gained CE about 1.4 validation points, far less than its 15-point gap to HCE.
Response to Reviewer uTGk
Thank you for the two concrete comparisons. We ran both; one control narrows our schedule recommendation. All results below are post-submission, from new runs, and never pooled with the submitted tables. Differences are final test accuracy in percentage points; brackets give pointwise 95% paired-t intervals.
W1: a common acquisition requirement. At each threshold T, we select within each loss the policy with highest mean validation final accuracy subject to mean immediate acquisition >= T, then evaluate the frozen choice on test. This is a common minimum, not exact acquisition matching or a guarantee for every test seed.
On CIFAR-100 at 40 epochs/task with replay, hinge minus HEM is +1.05 [0.63,1.46] at tested floors through 78.82%, -1.50 [-2.21,-0.80] at 80.5%, and -1.76 [-2.19,-1.33] at 82%. Floors 79%, 80% and 81% remain unresolved. Hinge minus CE is positive throughout the tested grid (8/8 seeds at every floor).
At 10 epochs/task (same grid and rule), margin losses still exceed CE at moderate floors (hinge-CE +1.21 to +1.35 through 77%; HEM-CE +0.77 to +1.82 through 78%), but at floors of 80.5% and above CE is better than HEM (-1.53 [-2.37,-0.68] and -1.95 [-2.80,-1.10], 0/8 each), and hinge cannot meet floors from 79% on validation.
On TinyImageNet (40 epochs/task): 15 configurations on 8 paired seeds, with replay weight fixed per loss.
Both margin losses exceed CE at every tested floor through 64%; the advantage is about 3 points through 62% and smaller at 64%. No threshold establishes hinge superiority over HEM. These are descriptive, dependent frontier rows, not independent confirmations. Mean test acquisition meets the selected floors, but individual seeds can miss them (4/8 for hinge at T=64). Tiny's test split has now been used three times.
W2: comparable tuning. The common CIFAR margin/LR/replay search gives hinge-HEM +1.05 above, compared with the submitted +3.38 under different selection rules and seeds. Tiny's table uses the completed limited margin/LR retune; it is not full joint replay tuning. CIFAR used 48 candidates per loss/budget; Tiny used 12 per loss with replay weight fixed. Some selected rates or replay weights lie at grid boundaries, so equal search counts do not establish global optimization. The submitted loss-specific constraints and fixed margins will be distinguished from these controls.
A separate equal LR-schedule search gives hinge-CE +3.55 [2.70,4.41] and hinge-HEM -0.41 [-0.73,-0.09] on 8 seeds. The latter passes the paired-t/Holm analysis but not the sign test, and does not support hinge dominance.
For Study C, extending the initial-learning-rate grid to .000625 changes the answer. Both the constant family and the family allowing geometric schedules select the same constant policy, m=.25, with no task-wise LR decay. Their preregistered contrast is therefore NA by construction, not evidence of equivalence. The earlier +1.908 schedule advantage holds within the tested grid with LR >= .00125. In an explicitly post-hoc comparison on the same 32 seeds, the new constant exceeds that earlier geometric policy by +1.12 [0.53,1.71], with higher earlier-task accuracy (+2.34) and lower acquisition (-3.31). The new selection is again at a grid boundary; no further extension was made.
Thus, the submitted fixed-baseline gains remain valid, but an added benefit from geometric scheduling after LR tuning is not established in these searches. We will remove the general discount recommendation. This is an empirical result, not an equivalence between changing margins and changing learning rates.
Revised practical guidance: state the update/replay budget and acquisition requirement, then select a feasible policy on validation. On these grids, the preferred margin loss depends on the floor; at the highest Tiny floors a margin-loss advantage over CE is unresolved, and at 10 epochs with a strict floor plain CE is preferable to HEM. Tune the learning rate before adding a margin schedule.
Do these completed comparisons resolve W1 and W2, or is there a specific remaining concern we should address in the revision?