acceptodds
Under review as a conference paper at ICLR 2027

LOSS AND MARGIN POLICIES FOR CONTINUAL LEARNING

Abstract

Models that are updated as new tasks arrive must learn new capabilities while preserving those already in use. The loss applied to new-task data is a rarely examined control over this trade-off: cross-entropy (CE) keeps pushing examples that are already classified correctly, whereas a hinge loss stops once a prescribed logit gap is reached. We compare CE, a multiclass hinge, and the high error margin (HEM) loss for task-incremental learning under matched update budgets and replay opportunities, with validation-based selection and separate measurement of new-task acquisition and old-task retention. On Split-CIFAR-100 the selected hinge–HEM ranking reverses with the training budget: HEM leads at 10 epochs per task and hinge at 40, because quadrupling the budget costs the selected CE and HEM policies 3.6–3.8 points of old-task accuracy while hinge gains 1.0; the size of the longer-budget advantage depends on the acquisition requirement used for selection. A 2×2 ablation with margin and learning rate fixed reproduces the reversal within one loss family: HEM-style active-sample averaging helps at the short budget (+0.55 points) and hurts at the long one (−3.02). On Split-TinyImageNet, whose test split was held out until the confirmation run, both margin losses beat CE with replay by about five points through retention rather than acquisition, but hinge and HEM are not distinguished (+0.13 [−0.34, 0.59]). In an all-hinge study, geometrically discounting the margin across tasks improves final accuracy over a fixed margin by +1.80 points (5% discount, 128 seeds) and +4.80 (20%, 32 seeds), because forgetting falls by more than acquisition does. An elementary gradient analysis explains why margin, normalization, and learning rate are distinct controls. The results give budget-dependent operating points for the new-task loss and quantify the acquisition cost of the retention they buy.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, task-incremental learning, catastrophic forgetting, margin losses, stability-plasticity trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uTGkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the objective for new tasks controls acquisition and retention in task incremental learning. It compares cross entropy (CE), multiclass hinge, and HEM under matched update and replay budgets. The selected loss ranking reverses between 10 and 40 epochs on CIFAR100, while both margin losses improve retention over CE on TinyImageNet. A 2×2 ablation examines sample averaging, and a separate hinge study quantifies the acquisition cost of geometric margin discounting.

Critical strengths
  1. The paper follows a clear structure from loss design to acquisition and retention analysis. Section 3 defines the measurements, and Section 6 uses Equation 6 to explain how reduced forgetting improves final accuracy despite weaker acquisition. This organization makes the benefits and costs of the proposed policies easy to assess.

  2. The experimental design controls training resources and includes substantial replication. Studies A and B branch from a shared checkpoint and match update and replay budgets (Section 3). The 2×2 ablation examines aggregation and averaging separately (Section 5, Figure 1d), while the margin study confirms the 5% discount benefit across 128 paired seeds (Section 6, Table 2).

Critical weaknesses
  1. The loss comparison lacks a common acquisition requirement for interpreting the preferred policy. Section 5 shows that removing the loss specific constraints shrinks the E40 validation gap between hinge and HEM from 3.7 to 1.2 points, although the ranking reversal persists. The preferred loss under a shared acquisition target therefore remains unclear.

  2. The experimental validation of margin policies lacks tuned baseline comparisons. Studies A and B fix margins without tuning them (Section 3), while Study C compares discounting with a constant margin of 0.25 (Section 6, Table 2). These comparisons leave open whether tuning a smaller constant margin could produce similar benefits, limiting the evidence for the proposed schedule.

Questions for authors
  1. Regarding Weakness 1: Under a common absolute acquisition floor, how do the E40 rankings change, and which loss would the practical guidance in Section 7 favor?

  2. Regarding Weakness 2: How do the loss rankings and discounting gains change when margins in A/B and the constant margin baseline in C receive comparable validation tuning?

3 comments
AuthorsOpenReview ↗

Response to Reviewer uTGk

Thank you for the two concrete comparisons. We ran both; one control narrows our schedule recommendation. All results below are post-submission, from new runs, and never pooled with the submitted tables. Differences are final test accuracy in percentage points; brackets give pointwise 95% paired-t intervals.

W1: a common acquisition requirement. At each threshold T, we select within each loss the policy with highest mean validation final accuracy subject to mean immediate acquisition >= T, then evaluate the frozen choice on test. This is a common minimum, not exact acquisition matching or a guarantee for every test seed.

On CIFAR-100 at 40 epochs/task with replay, hinge minus HEM is +1.05 [0.63,1.46] at tested floors through 78.82%, -1.50 [-2.21,-0.80] at 80.5%, and -1.76 [-2.19,-1.33] at 82%. Floors 79%, 80% and 81% remain unresolved. Hinge minus CE is positive throughout the tested grid (8/8 seeds at every floor).

At 10 epochs/task (same grid and rule), margin losses still exceed CE at moderate floors (hinge-CE +1.21 to +1.35 through 77%; HEM-CE +0.77 to +1.82 through 78%), but at floors of 80.5% and above CE is better than HEM (-1.53 [-2.37,-0.68] and -1.95 [-2.80,-1.10], 0/8 each), and hinge cannot meet floors from 79% on validation.

On TinyImageNet (40 epochs/task): 15 configurations on 8 paired seeds, with replay weight fixed per loss.

T (%)Hinge-CEHEM-CEHinge-HEM
none, 50, 55+3.29 [2.45,4.13]+3.35 [2.70,4.00]-0.06 [-0.29,0.17]
58+3.40 [2.78,4.01]+3.46 [2.99,3.93]-0.06 [-0.29,0.17]
60+3.40 [2.78,4.01]+3.42 [2.76,4.08]-0.02 [-0.51,0.47]
62+2.86 [2.40,3.32]+3.16 [2.46,3.85]-0.30 [-0.66,0.07]
64+0.82 [0.54,1.11]+1.73 [1.09,2.37]-0.91 [-1.33,-0.48]
65-0.21 [-0.71,0.30]+0.09 [-0.70,0.89]-0.30 [-1.06,0.47]
66hinge infeasible-0.05 [-0.83,0.74]—
70all infeasible——

Both margin losses exceed CE at every tested floor through 64%; the advantage is about 3 points through 62% and smaller at 64%. No threshold establishes hinge superiority over HEM. These are descriptive, dependent frontier rows, not independent confirmations. Mean test acquisition meets the selected floors, but individual seeds can miss them (4/8 for hinge at T=64). Tiny's test split has now been used three times.

W2: comparable tuning. The common CIFAR margin/LR/replay search gives hinge-HEM +1.05 above, compared with the submitted +3.38 under different selection rules and seeds. Tiny's table uses the completed limited margin/LR retune; it is not full joint replay tuning. CIFAR used 48 candidates per loss/budget; Tiny used 12 per loss with replay weight fixed. Some selected rates or replay weights lie at grid boundaries, so equal search counts do not establish global optimization. The submitted loss-specific constraints and fixed margins will be distinguished from these controls.

A separate equal LR-schedule search gives hinge-CE +3.55 [2.70,4.41] and hinge-HEM -0.41 [-0.73,-0.09] on 8 seeds. The latter passes the paired-t/Holm analysis but not the sign test, and does not support hinge dominance.

For Study C, extending the initial-learning-rate grid to .000625 changes the answer. Both the constant family and the family allowing geometric schedules select the same constant policy, m=.25, with no task-wise LR decay. Their preregistered contrast is therefore NA by construction, not evidence of equivalence. The earlier +1.908 schedule advantage holds within the tested grid with LR >= .00125. In an explicitly post-hoc comparison on the same 32 seeds, the new constant exceeds that earlier geometric policy by +1.12 [0.53,1.71], with higher earlier-task accuracy (+2.34) and lower acquisition (-3.31). The new selection is again at a grid boundary; no further extension was made.

Thus, the submitted fixed-baseline gains remain valid, but an added benefit from geometric scheduling after LR tuning is not established in these searches. We will remove the general discount recommendation. This is an empirical result, not an equivalence between changing margins and changing learning rates.

Revised practical guidance: state the update/replay budget and acquisition requirement, then select a feasible policy on validation. On these grids, the preferred margin loss depends on the floor; at the highest Tiny floors a margin-loss advantage over CE is unresolved, and at 10 epochs with a strict floor plain CE is preferable to HEM. Tune the learning rate before adding a margin schedule.

Do these completed comparisons resolve W1 and W2, or is there a specific remaining concern we should address in the revision?

Thank you for these additional experiments and for your detailed responses.

While responses to W1 and W2 clarify the experimental boundaries, they also shrink the paper's contribution considerably. Now that the margin-schedule recommendation is withdrawn, the Hinge-vs.-HEM ranking flips across settings without establishing any mechanism. Hence, the main result reduces to a tuning observation on CIFAR-100 and TinyImageNet, which does not hold on SVHN or FashionMNIST and still depends on grid-boundary selections in a restrictive task-incremental setup.

Given this, what do you see as the paper's core contribution?

Thank you for the direct question. We agree that, as of our last reply, the contribution had narrowed to a tuning observation. The revised PDF changes this, and the core contribution is now the following.

Core contribution: a two-axis view of the new-task loss, and a schedule along those axes that works. CE, hinge, and HEM are members of one loss family that differ along two axes (Section 4): class-update sparsity (how many wrong classes one update pushes down) and margin. CE and hinge are fixed points; HEM moves along both axes within each task. This view explains why the hinge-HEM ranking flips across settings: no fixed point is best. It also leads to a method. HCE (Section 7) moves smoothly within each task from sparse hinge updates with a margin to dense CE updates without one, as the learning rate anneals.

What this adds beyond a tuning observation:

  • A method that beats tuned baselines, not a ranking between two of them. Without replay, HCE beats tuned HEM (+3.3 CIFAR-100, 16/16; +1.9 TinyImageNet, 8/8), tuned CE, EWC (+5.0 / +4.6), and experience replay (+8.5 / +12.2). With replay for every loss, it beats HEM with replay (+3.9 / +2.1) and ER (+9.9 / +12.8). Every baseline selects its own learning rate on validation seeds.
  • Evidence about mechanism. Preregistered controls test the reading that sparse updates protect old tasks while steps are large, and dense updates finish the new task once steps are small. Reversing the schedule loses 24.5 points (forgetting rises to 49.7), an abrupt switch loses 6.4, and a constant within-task learning rate removes 6.1 points of the gain. Against tuned HEM, HCE acquires the same on CIFAR-100 and forgets less, so it does not win by learning less.
  • The margin result, restated. You were right that our across-task margin discount did not survive a tuned learning rate. The schedule that adds to a tuned learning rate is the one within each task: HCE beats the hinge with a tuned learning-rate decay by +3.3 and adds +2.4 (31/32) on top of the best margin-discount policy.

Scope we state plainly: all results are task-incremental; HCE was not run on SVHN or FashionMNIST; among the tested settings with a learning-rate anneal, the one exception is the CE-prefix replay design of Section 5 (tuned HEM +1.6), where only the last three tasks use the new loss. Several baseline selections lie at grid edges; CE's best rate is at the top of a grid we extended twice; each doubling of the rate gained CE about 1.4 validation points, far less than its 15-point gap to HCE.

Reviewer_6dyJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the effect of loss functions in continual learning (CL) under different training budgets (epochs) and settings. The losses studied include CE loss, a multiclass hinge loss, and the high error margin (HEM) loss. The experiments on two task-incremental datasets show that hinge and HEM losses outperform CE under different training budgets. Specific settings of the loss, including the margin discount and normalization, contribute to old task retention.

Critical strengths
  1. The study of loss policies is interesting and may benefit CL. The paper considers different settings for different losses and conducts thorough experiments to show their effects.

  2. The paper provides both theoretical and experimental analysis of the studied problem.

Critical weaknesses
  1. The motivation of the work is unclear.

    • The original HEM paper shows improved performance in continual learning, while this work further studies HEM in CL with a budget constraint. It is unclear why only the number of epochs per task is considered in the budget (why not the batch size or replay buffer size?), and how significant this budget constraint is in CL.
  2. Some settings of experiments are not well justified.

    • The selection of losses to compare: why select hinge and HEM loss to compare with CE? Is there any reason that other losses are excluded from the analysis?

    • The CL evaluation is on two splits of CIFAR-100 and TinyImageNet, which is limited. A certain segmentation of tasks is used for evaluation without justification, e.g. 0-6 tasks as earlier tasks.

  3. The writing of the paper can be improved. Sentences like ‘The transferable result is a margin-loss advantage over CE, not hinge dominance.’ in line 74 are hard to interpret.

Questions for authors

Please see the weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6dyJ

Thank you. We will make the motivation explicit: after a common training history, which new-task loss policy gives the best final performance for a stated update/replay budget and a minimum acceptable acquisition? Acquisition is accuracy when a task is first learned; final averages alone can reward learning less of it. HEM already established that margin losses can help continual learning; our contribution is the controlled comparison of when their ranking and acquisition-retention trade-off change.

Why these losses and this design. CE is the standard reference; multiclass hinge provides a simple cutoff at a required logit gap; HEM is the closest existing margin-based continual-learning objective and changes both class aggregation and sample averaging. These are established losses, not a new algorithm or a comprehensive loss benchmark. Tasks 0-6 form a shared CE-trained history: restoring the same checkpoint before tasks 7-9 makes earlier-task accuracy differences attributable to subsequent changes, rather than different initial learning. The seven/three split is a design choice; we now also test shorter histories (below).

More backbones and datasets. All new results here are post-submission and never pooled with the submitted tables: 8 paired seeds, 40 epochs per later task, validation-only selection; differences are points of final test accuracy with pointwise 95% paired-t intervals.

SettingHinge-CEHinge-HEM
CIFAR-100, MobileNetV2+4.62 [3.50,5.73]-0.28 [-0.75,0.19]
CIFAR-100, ViT-Tiny+4.30 [2.95,5.65]+0.15 [-0.06,0.36]
SVHN, ResNet-18+2.66 [-0.67,5.98]+6.97 [3.15,10.79]
FashionMNIST, ResNet-18+2.38 [0.23,4.53]+5.71 [-2.00,13.43]

Three contrasts survive Holm adjustment across all eight: hinge-CE on both CIFAR backbones and hinge-HEM on SVHN, each positive in 8/8 seeds. FashionMNIST hinge-CE is not established: within-setting Holm p=.069; across-eight p=.173. On SVHN, HEM's mean is below CE's (78.41% versus 82.73%). So the margin-loss advantage is not dataset-independent.

(This reruns a summarized study whose run files were not kept; there, FashionMNIST's within-setting p was .021. SVHN and FashionMNIST test sets have now been used twice.)

What the budget comparison means. Our 10/40-epoch conditions fix batch size and replay events while varying new-task updates; they do not match elapsed time. With a common search on CIFAR, hinge-HEM changes from -0.48 [-0.68,-0.27] to +1.05 [0.63,1.46]. These are policies selected separately at each budget; with the same fixed recipes at both budgets, hinge-HEM is +3.37 [1.78,4.95] at 10 epochs and +0.49 [0.02,0.97] at 40. Since cosine horizon and replay density also change, we do not attribute the ranking to extra epochs alone.

Resources and scope. A five-images-per-class replay probe keeps a hinge-CE gap of +3.95 [3.41,4.48] (with more memory and replay exposure). Five task orders also give positive long-budget hinge-CE differences. Batch-size sensitivity remains untested.

Other losses (new). Under the same 12-point LR/replay search per loss, hinge exceeds label smoothing (eps=.1) by +3.08 [2.20,3.96] and focal loss (gamma=2) by +2.80 [1.86,3.74] (8/8 each; Holm over two). Both variants improve on CE only modestly (+0.75, +1.03), mainly through acquisition. Hinge's advantage comes from less forgetting (-4.41/-4.23), while its acquisition is slightly lower than with label smoothing (-0.99 [-1.64,-0.34]). HEM is again slightly ahead of hinge (-0.39 [-0.63,-0.15]). Only these two fixed settings were tested.

Shorter common history (new). With a CE history of 1 or 4 tasks instead of 7 (LR re-selected per loss), hinge exceeds CE by +3.71 [2.58,4.83] and +5.38 [4.51,6.25] (8/8 each; Holm over two). With one history task, HEM exceeds hinge (-1.68 [-2.46,-0.91], 0/8) and hinge's gain over CE comes mainly from acquisition, with more forgetting. Three of six selected rates, including CE's at one history task, lie at the grid's lower edge, so these comparisons are conditional on the grid. Task populations differ across lengths, so we claim no effect of history length.

Mechanism and writing. Gating CE to below-margin examples improves earlier-task accuracy by +1.32 [0.36,2.28], but changes acquisition by -3.79 [-4.23,-3.35]; its final-accuracy effect is unresolved. Matching CE's trunk-step norms to hinge's does not recover hinge's result either. These controls do not identify the cause of the gap. We will remove the stronger normalization-as-cause wording, replace the unclear "transferable result" sentence with setting-specific findings, and state that no added geometric-schedule benefit is established after LR tuning.

The revision will center the budget/acquisition question, the positive evidence on CIFAR and moderate Tiny floors, and the settings where that evidence does not extend. Does this clarification address the motivation and experimental-design concerns?

Thank you. Your main concerns were the motivation (what is new beyond HEM, and why these losses and epochs) and readability. The revision addresses them directly.

Motivation and why these losses. Section 4 now explains the losses along two axes: class-update sparsity (how many wrong classes one update pushes down) and margin. CE updates all wrong classes, hinge only the strongest, and HEM lies in between. The three are therefore representative points on one axis, not an arbitrary set. Analyzing HEM this way shows that HEM itself moves along both axes within a task. This led to the method of Section 7 (HCE), which beats a tuned HEM by +3.3 (CIFAR-100, 16/16 seeds) and +1.9 (TinyImageNet, 8/8). That is the new contribution beyond HEM.

Why the number of updates (Budget). The submitted results show that which fixed loss is best changes with the number of updates per task (HEM at 10 epochs, hinge at 40). In the new view, this means no fixed point on the axes is best, which motivates moving along them during training.

Datasets and baselines. Section 7 adds EWC, experience replay, and tuned CE/HEM on both CIFAR-100 and TinyImageNet, each with its own validation-tuned learning rate. HCE beats all of them without replay. As a check, our EWC implementation reproduces a published result (Appendix I).

Readability. We rewrote the abstract and introduction around one question: which loss to use for new-task data. The contributions are listed as a question-and-answer sequence, and changes are marked in blue.

Reviewer_BvP1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how the choice of loss affects the balance between learning new tasks and retaining earlier ones. It compares cross-entropy (CE), multiclass hinge, and high error margin (HEM) in task-incremental learning. The main experiments start each loss from the same CE-trained checkpoint and match the update and replay budgets.

The clearest result is that hinge outperforms CE at 40 epochs per task across the reported Split-CIFAR-100 checks. The ranking of hinge and HEM is more sensitive to the training budget and the minimum validation accuracy required on new tasks. A separate experiment finds that decreasing the hinge margin improves on a fixed baseline, but the reported searches establish no added benefit once the learning rate is tuned.

Critical strengths

Clear question and controlled comparison. Starting from the same checkpoint makes differences in old-task retention easier to interpret. Reporting how well each task is learned when it first arrives also helps distinguish stronger retention from weaker acquisition.

The loss ranking changes under practical constraints. Figure R1 is the most informative result. On Split-CIFAR-100, HEM leads hinge at 10 epochs under the common search, hinge leads at 40 epochs, and HEM leads again at 40 epochs when the required validation acquisition rises to 80.5%. The related work positions the study clearly, with the contribution coming from the controlled comparison of existing losses.

Critical weaknesses

What explains the retention gain from switching losses? Section 4 and Appendix E show how changing the hinge margin can change the direction of an update. Appendix O also includes a gradient-direction audit and shows that matching CE’s trunk update sizes to hinge’s does not reproduce hinge’s retention gain. These controls narrow the explanation. A direct comparison of CE, hinge, and HEM updates at the same checkpoint and on the same data would help establish whether the losses interfere differently with earlier tasks and whether that difference explains the observed retention and acquisition results.

The results give limited guidance on when switching is worth the added complexity. The common CE history is a useful control, but it leaves open whether the same choice would help after a margin-loss history or whether using hinge throughout would deliver similar gains under the same protocol. The paper compares three losses and runs many follow-up checks, yet its clearest guidance remains specific to the tested histories and budgets. The presentation is generally understandable. A more direct introduction and a clearer distinction between the primary results and the follow-up checks would make the main finding easier to follow.

Questions for authors

Can the existing diagnostics in Appendix O tell us whether the full updates from CE, hinge, and HEM differ in how much they interfere with earlier tasks? How does that relate to the observed changes in new-task accuracy?

Is there a directly comparable result for using hinge throughout versus starting with CE and switching to hinge? If the available experiments address different settings, what practical guidance about switching follows from the shared-prefix comparison?

POST REBUTTAL. Thank you for the detailed responses and revised manuscript. HCE makes the practical contribution clearer by providing a concrete schedule for changing the loss within each task. Its consistent gains over tuned HEM on both datasets, together with nearly identical acquisition on CIFAR-100 and the schedule controls, address my main concern about whether changing the loss offers a useful benefit. The explanation based on class-update sparsity remains partly unresolved, and HCE’s evaluation is limited to two datasets and one backbone. Nevertheless, the additional evidence strengthens the contribution enough that I will raise my rating to 3 (Weak Accept).

7 comments

Thank you for reviewing. The text appears to concern a different submission: it discusses fixed-rank subspace protection, ReDo, ImageNet-32 and Procgen, whereas #54880 studies CE, hinge and HEM loss policies on Split-CIFAR-100 and Split-TinyImageNet. Could the association with this submission please be checked? We would welcome comments on the actual manuscript. A summary of the evidence for the actual manuscript is in our general response.

Thank you for flagging the mix-up and for pointing me to your general response. I’ve posted the corrected review for submission #54880, Loss and Margin Policies for Continual Learning. I mistakenly posted a review for another submission here, and I recognize that this left you less time to respond. My main question in the corrected review is what switching losses changes in the optimization and when that helps retain earlier tasks while still learning the new task well.

I also want to apologize for posting the wrong review and for the time this cost you during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of new evidence

We thank the reviewers. Our question is: after a common training history, which new-task loss policy gives the best final performance for a stated update/replay budget and a minimum acceptable acquisition? Acquisition is accuracy when a task is first learned; final averages alone can reward learning less of it. The high error margin (HEM) paper already showed that margin losses can help continual learning; we add a controlled comparison of cross-entropy (CE), hinge and HEM: when their ranking changes, and at what acquisition cost. Setting: task-incremental; by default tasks 0-6 are trained with CE, then each loss trains tasks 7-9 from that checkpoint.

Findings from post-submission controls (kept separate from the submitted tables):

  1. Rankings are conditional. The submitted hinge-HEM lead at 40 epochs/task (+3.38) is +1.05 under a common search on new seeds and reverses at high acquisition floors. At 10 epochs/task HEM leads hinge without a floor, and under a strict floor CE beats HEM.
  2. At 40 epochs/task hinge exceeds CE in every CIFAR-100 comparison below (eight estimates, +3.55 to +5.38; +1.94 to +3.97 across tested floors). On TinyImageNet both margin losses exceed CE through a 64% floor. This is not established on SVHN or FashionMNIST (on SVHN, HEM's mean is below CE's) or at TinyImageNet floors from 65%.
  3. No added benefit of a margin schedule is established once the learning rate (LR) is tuned. The submitted gains over the fixed baseline stand; we withdraw the recommendation to discount the margin.
  4. Not claimed: hinge dominance over HEM, margin-LR interchangeability, or any mechanism; our gated-CE and step-norm controls do not explain the gap to CE.

Key numbers. X-Y denotes X minus Y in final test accuracy (points), with pointwise 95% paired-t intervals over 8 paired seeds unless stated. Default: CIFAR-100, ResNet-18, replay, 40 epochs per later task. Selection uses validation only, under frozen protocols. At floor T, each loss uses its best validation policy whose mean validation acquisition is at least T: a common minimum, not exact matching or a per-seed guarantee.

ComparisonResult
Common margin/LR/replay searchHinge-HEM +1.05 [0.63,1.46] through T=78.82%; -1.50 [-2.21,-0.80] at 80.5%; -1.76 [-2.19,-1.33] at 82%; unresolved at 79%, 80%, 81%. Hinge-CE +3.90 [3.30,4.49] with no floor, +1.94 [1.13,2.74] at 82%.
Same at 10 epochs/taskHinge-HEM -0.48 [-0.68,-0.27] with no floor. Hinge-CE +1.21 to +1.35 through 77%, unresolved at 78-78.82%; hinge cannot meet floors from 79%. HEM-CE -1.53 [-2.37,-0.68] at 80.5-81%, -1.95 [-2.80,-1.10] at 82%.
TinyImageNet, limited margin/LR retuneHinge-CE +3.29 [2.45,4.13] with no floor, +2.86 [2.40,3.32] at 62%, +0.82 [0.54,1.11] at 64%, unresolved at 65%. At 64%, HEM-CE +1.73 [1.09,2.37] and hinge-HEM -0.91 [-1.33,-0.48].
Equal LR-schedule searchHinge-CE +3.55 [2.70,4.41]; hinge-HEM -0.41 [-0.73,-0.09].
MobileNetV2 / ViT-TinyHinge-CE +4.62 [3.50,5.73] / +4.30 [2.95,5.65].
SVHN / FashionMNISTHinge-CE +2.66 [-0.67,5.98] / +2.38 [0.23,4.53]; neither survives Holm correction.
Five task orders, 4 seeds eachHinge-CE +4.98 [3.56,6.40] over the 5 order means; positive in all 5.
5 replay images/class instead of 1Hinge-CE +3.95 [3.41,4.48]; memory and replay exposure both grow.
Label smoothing (eps=.1), focal loss (gamma=2), common LR/replay searchHinge exceeds these two fixed settings by +3.08 [2.20,3.96] and +2.80 [1.86,3.74].
CE history of 1 / 4 tasks instead of 7Hinge-CE +3.71 [2.58,4.83] / +5.38 [4.51,6.25]. With 1 task, HEM exceeds hinge (-1.68 [-2.46,-0.91]) and CE's selected LR is at the grid edge.
Margin schedule: all hinge, no replay, 32 seedsWith one lower LR (.000625) added, the constant-margin family and the family allowing geometric schedules select the same constant margin (m=.25), leaving no contrast to test. Post hoc, it exceeds the previously selected geometric schedule by +1.12 [0.53,1.71].

Limits. The probes are separate and bounded, not fully crossed; some share seeds or frozen policies. Searches are finite; some selections sit at grid edges. Budget effects describe selected policies, not extra epochs alone. Test sets were reused (TinyImageNet three times; SVHN and FashionMNIST twice; CIFAR had earlier exposure). Batch size is untested.

Revision (target: 16 November). It keeps the title, Studies A-C and all submitted tables and figures; rewrites the abstract, contributions and guidance around findings 1-3; removes the normalization-as-cause wording and the discount recommendation; and adds a labeled post-submission section (frontier figure, table) and an appendix (protocols, grids, test-set exposure), with changes marked. Guidance: state the budget and required acquisition, pick a feasible policy on validation, and tune the LR before adding a margin schedule. The reviewer-specific replies give details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.