acceptodds
Under review as a conference paper at ICLR 2027

The Illusion of Global Calibration: Group-Aware Risk in Class-Incremental Learning

Abstract

Reliable class-incremental learning requires confidence estimates that remain informative after sequential training, because downstream decisions such as selective prediction and human referral depend on per-sample confidence rather than on accuracy alone. Calibration is typically summarized by aggregate scores, yet such averages can hide substantial errors within groups of classes introduced at different times during the training schedule. In this paper, we show that global temperature scaling can improve overall calibration while leaving large errors in specific schedule-age groups, and that the direction of these group-level changes differs across learners. Aggregate improvement therefore does not by itself certify uniformly reliable confidence. To make this discrepancy visible, we introduce a group-aware audit that evaluates aggregate and group-level calibration jointly, together with the direction of confidence bias, the source of calibration data, and the scale of the scores being calibrated. Across class-incremental benchmarks, the discrepancy persists even in high-accuracy models, although its magnitude depends on both the temperature search range and the data used for fitting. These findings support reporting group-level calibration alongside aggregate scores whenever continual learners are compared or deployed.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Class-Incremental Learning, Calibration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jbqAOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines whether aggregate calibration scores give an adequate picture of reliability in class-incremental learning. It proposes an audit that reports calibration within groups defined by class introduction time, alongside confidence quartiles, expected-age quantiles, and label-dependent confidence-gap groups. The experiments use a class-stratified retrospective calibration/evaluation split, separate comparisons of calibration-data sources, and several secondary protocols. The main observation is that global temperature scaling can substantially reduce pooled ECE while leaving large and uneven errors across age groups, with the direction of the worst-group change depending on the learner. Experiments with the higher-accuracy ACMap model find smaller residual discrepancies and show that respecting the native score scale is essential. Oracle age-conditional scaling, an inference-available expected-age proxy, calibration-source comparisons, signed confidence gaps, and selective-prediction measurements help interpret these results.

Critical strengths
  • I found the main point useful. An overall calibration score can look good even when the results for some groups are much worse. Table 1 gives a clear example of this problem and helps explain why reporting only one overall score may leave out important information.

  • The experiments cover several datasets and different models. This gives the main observation more support than a result from a single setting. I also appreciate that the authors include a model with much higher accuracy to check whether the problem still appears.

  • The authors are open about the limits of their results. For example, they explain that the size of the problem differs between models and is smaller for the more accurate model they study. This makes the discussion more balanced and helps readers understand the scope of the conclusions.

Critical weaknesses
  • The paper considers many experimental settings and ways of dividing the data into groups, which made some parts hard to follow. I was not always sure how the different experiments fit together or which ones most directly support the main conclusion. A simpler overview of the experiments would help readers who are less familiar with this area.
Questions for authors

N/A

Reviewer_rTyJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a group-aware calibration audit for CIL. Alongside aggregate ECE, NLL, and Brier score, it reports group-level metrics across schedule age, raw confidence, label-dependent confidence gaps, and expected-age quantiles (EAQ), which provide an inference-available proxy. The audit includes worst-group error, disparity, signed confidence gaps, temperature-boundary indicators, and score-scale metadata. The primary study audits four learners trained from scratch on SplitCIFAR-100 over five seeds, using a class-stratified 20% calibration / 80% evaluation split and ten exact-age groups. Additional experiments examine high-accuracy ACMap models, calibration-data sources, recurrence across datasets and learner families, and selective-prediction risk. Age-Conditional Temperature Scaling (ACTS) is explicitly an oracle diagnostic, rather than a deployable calibrator or a universal performance bound.

Critical strengths
  1. The primary experiment provides substantive evidence. After global temperature scaling, LwF has 3.58% overall ECE and 24.43% worst exact-age ECE; DER++ has 4.34% and 26.09%, respectively (Table 1). Table 28 reports per-seed results, supporting the observation within individual runs rather than only through cross-seed averaging.

  2. The distinction between calibration direction and operational risk is useful. The paper does not interpret every large ECE as overconfidence or equate the worst-calibrated group with the highest-risk retained group. Tables 19 and 30-31 show learner-dependent differences in bias direction and group identity.

  3. The score-scale and calibration-data analyses are practically informative. An unsuitable temperature range leaves ACMap's overall ECE near 84-85%, whereas scale-aware fitting reduces it to approximately 1-2% (Table 2). Table 3 varies fitting-data sources while holding each learner's final checkpoints and evaluation stream fixed, and appropriately distinguishes these comparisons from algorithm comparisons with matched data budgets.

  4. The study includes substantial supporting evidence and acknowledges its limits. It reports multiple seeds, optimizer comparisons, size-matched permutations, joint-training controls, and fitting-budget controls. The authors identify the low accuracy of the primary benchmark, boundary saturation, differences among audit protocols, and the associational nature of the age/replay analysis.

Critical weaknesses

4.1 Structural ECE ordering and the interpretation of the high-accuracy gap

For a partition of the same evaluation set into disjoint groups, all using the same confidence-bin boundaries, Appendix A.8 bounds the absolute pooled confidence-accuracy gap in each bin by the weighted sum of the within-group absolute gaps. Summing across bins gives

\le \sum_g \pi_g\\,\mathrm{ECE}_g \le \max_g \mathrm{ECE}_g, \qquad \pi_g = \frac{n_g}{n}.$$ Here, $\pi_g$ is the proportion of all evaluation samples belonging to group $g$, rather than the group weight within a particular bin. This relation uses the same predictions and bin boundaries for the pooled and group calculations. Thus, worst-group ECE being no smaller than aggregate ECE is not itself a new empirical finding. The contribution lies in the size and stability of the difference and the additional structure revealed by the predefined partition. The inequality neither guarantees a strict difference nor determines its magnitude. It therefore does not justify dismissing the large primary-experiment discrepancies as inevitable or as noise. The paper appropriately presents the underlying relation as elementary. The high-accuracy results are important to the claim that the discrepancy extends beyond poor predictive performance. Under scale-aware global fitting, ACMap has (1.94 +/- 0.21)% overall ECE and (6.55 +/- 1.55)% worst-age ECE on SplitCIFAR-100. The corresponding Tiny-ImageNet values are (1.25 +/- 0.21)% and (4.75 +/- 0.72)% (Table 2; mean and sample standard deviation over three seeds). These differences may be meaningful, but the ratio of worst-group to aggregate ECE alone does not establish their statistical interpretation. Group ECE is estimated from fewer samples, and selecting the maximum across groups adds a selection effect. The joint-training control illustrates the limitation of interpreting ratios alone. It reports approximately 1.4% overall ECE, 4.6% worst-age ECE, and 4.1% worst pseudo-age ECE (Table 12). Its model and audit protocol differ from ACMap's, so it cannot serve as a direct null model or refute the ACMap result. Nevertheless, it shows that a similar ratio can arise without the same sequential-training process. Existing controls deserve acknowledgment. The primary study randomizes class order across seeds, so its age groups are not tied to one fixed assignment of classes. Table 11 also shows much larger worst-group error for the actual age partition than for size-matched permutations under the secondary chronological protocol. These results address dependence on a fixed class assignment and on group size, respectively. They do not replace a corresponding analysis for the high-accuracy protocol. The manuscript should also specify whether its permutations operate on individual samples or intact classes, since these test different hypotheses about class composition. Table 22 supplies further high-accuracy evidence from ImageNet-100: L2P reaches 80.1% accuracy and CL-LoRA 78.0%. These experiments use the secondary chronological split and coarse age groups, however, and also lack a matched grouping reference for their high-accuracy setting. Using the existing ACMap predictions, I suggest comparing exact-age groups with: - random sample partitions matched in group count and sample counts; and - random partitions that keep each class intact and match the age groups' class counts and sample counts. All comparisons should use the same fitted calibrator and evaluation set. Sample-level partitions provide a reference for group size and maximization, while class-preserving partitions help assess the information associated with age beyond ordinary class composition. Neither is automatically a pure-noise null under perfect calibration, and a standard bootstrap alone does not remove ECE estimation bias. Please state the assumptions and interpretation of each reference, and report paired differences, uncertainty, group counts, and signed confidence gaps. The missing evidence concerns the reproducibility and magnitude of high-accuracy group heterogeneity. Schedule age need not be the strongest partition or exceed every random class grouping to remain a useful audit axis. **4.2 The significance of the additional empirical knowledge needs a more specific argument** The related-work section already acknowledges hidden stratification, conditional calibration, subgroup auditing, and continual calibration. Consequently, the general distinction between aggregate and subgroup calibration should not carry the novelty argument by itself. Prior continual-calibration work addresses validation-data retention, and T-CIL addresses the limitations of fitting calibration on training replay when old-task validation data are unavailable. The cited DATS manuscript also discusses task heterogeneity and worst-task calibration. I do not treat the absence of a matched DATS experiment or comparisons with contemporaneous or arXiv-only work as a reason for rejection. The authors already identify a more specific contribution: a systematic audit combining schedule age, learner-dependent bias direction, score scale, calibration-data provenance, and multiple diagnostic axes. This combination is potentially valuable. My concern is therefore about the significance of the incremental knowledge established by the experiments, rather than the absence of a contribution statement. The findings with the greatest potential value appear to be: - learner-dependent changes in worst-age ECE and signed confidence bias after global scaling; - differences between the groups identified by worst ECE and worst selective risk; - the interaction between score scale, temperature-search range, and apparent calibration failure; and - changes in calibration outcomes with fitting-data provenance at fixed checkpoints. These findings could support a useful evaluation contribution. The paper would be stronger if it identified one or two as its central new conclusions and explained which existing experiments establish their scope and implications. This would also distinguish systematic validation of a known principle from additional empirical knowledge. The high-accuracy analysis is particularly relevant because it affects the extent to which the findings generalize beyond low-accuracy failure cases. If the age-associated component is small, the study could still be valuable through its score-scale, data-provenance, and signed-bias analyses, but its contribution should be assessed and presented on that basis. I do not require a new algorithm, a difficult theorem, or state-of-the-art performance; the question is whether the supported findings provide sufficient additional value to the community. **4.3 Auxiliary controls limit some interpretations of ACTS and temperature boundaries** This concern affects the interpretation of particular results, not the validity of the primary descriptive audit. ACTS is explicitly an oracle diagnostic, so its extra fitting flexibility does not invalidate the observation that aggregate and age-group calibration can differ. The primary stratified split supplies calibration data for all ten exact-age groups. The secondary chronological protocol instead fits on samples from the first two experiences. Table 16 states that ACTS temperatures for ages 0, 1, and 2-3 remain at the default $t=1$ because those groups lack calibration samples. Comparisons with random-group fits under this protocol therefore also reflect differences in calibration support. Table 13 provides four-temperature budget-matched controls, but these do not match Table 1's ten-temperature exact-age ACTS configuration. In Table 1, adding age information and increasing fitting flexibility occur together, so the improvement does not isolate the contribution of schedule-age information. The evaluation partition in Table 13 also needs clarification: if each fit is evaluated on its own partition, the rows measure different worst-group targets; if all fits use one common age partition for evaluation, please state this explicitly. For Naive and EWC, every primary global fit reaches $t_{\max}=10$, and all older exact-age ACTS fits also reach this boundary (Tables 1 and 15). Table 7 provides a useful continuous L-BFGS comparison, with changes below one percentage point in overall and worst-age ECE. However, it uses a separate twelve-run experiment, different LwF/DER++ settings, a non-stratified random split, and coarse age groups. As the authors acknowledge, it does not establish optimizer invariance for the primary checkpoints. If the ACTS gap is intended to quantify the specific value of age information, an appropriate control would compare age-based and class-preserving random fits with the same ten-temperature budget and calibration support, evaluating both on the original exact-age groups. Otherwise, the gap should remain an oracle diagnostic of additional group-specific fitting. For boundary sensitivity, extending the range or applying continuous fitting to the original Naive/EWC checkpoints would directly test the affected results. These are limited checks or opportunities to narrow interpretation. They are not evidence that the entire discrepancy is a boundary artifact: LwF and DER++ select temperatures inside the primary grid and still show large overall/worst-age gaps.
Questions for authors

Q1. Under the same fitted ACMap calibrator and evaluation split, how does worst exact-age ECE compare with size-matched random sample partitions and class-preserving random partitions? Please state what each reference tests and report paired differences, uncertainty, group counts, and signed confidence gaps. How much of the observed high-accuracy gap is reproducible beyond these reference effects?

Q2. Which one or two empirical findings are the central new conclusions beyond the established aggregate/subgroup calibration distinction? The learner-dependent changes, score-scale and fitting-data effects, and calibration/selective-risk group mismatch seem promising. Which existing results establish their scope and implications for evaluation practice?

Q3. Is Table 13's "Worst-bucket ECE" evaluated on one common age partition or on each fitting method's own partition? How should Table 1's ACTS gap be interpreted given the increase from one to ten temperatures? Do the primary Naive/EWC results change materially when fitting is allowed beyond the temperature boundary? These questions concern interpretation and sensitivity, not a presumed failure of the core audit.

Q4. Optional clarification, not an independent reason for rejection: At 50%, 70%, and 90% overall coverage, please report retained sample counts and within-group coverage for each age group, including any empty or very small retained groups. These statistics would help interpret the selective-risk results.

2 comments

The class-preserving controls address my main concern about the primary benchmark. The worst-age calibration error exceeded this reference in all 20 primary runs, but in only one of the six high-accuracy ACMap runs. I had hoped the replay experiment would help explain that difference, but overall accuracy and the accuracy gap between older and newer classes change together as replay increases. Given prior work on worst-task calibration, I’m still looking for a clearer account of when grouping by age reveals more than ordinary class differences, so I’m keeping my score at 2.

Thank you for engaging with the new controls and for stating precisely what remains open. We agree that the replay sweep alone does not separate overall accuracy from the accuracy gap between older and newer classes. Below we summarize what the 86 runs do establish about when age grouping reveals more than ordinary class differences, and how this relates to worst-task calibration.

1. Low accuracy alone cannot make age informative. Class order is randomized for each seed. For any model whose training does not depend on that order, the exact-age partition is therefore itself a random class-preserving partition, so REF-C is its exact null at every accuracy level, and the test exceeds it with probability at most 5%. The two order-independent models we tested, SimpleCIL and the jointly trained linear head, exceed REF-C in 1/12 runs. An age-specific residual therefore requires that the training order leave a mark on calibration. What remains is how large that mark must be relative to ordinary class differences.

2. What the 86 runs show (new Table 48). We place all runs of Tables 40 and 45 in disjoint groups.

GroupRuns exceeding REF-C
Training does not depend on class order (SimpleCIL, joint linear head)1/12
Sequential, newest age group less than 20 points more accurate than older groups2/9
Sequential, gap 20 to 60 points, overall accuracy at least 50%0/12
Sequential, gap 20 to 60 points, overall accuracy at most 40%15/19
Sequential, gap at least 60 points34/34

Both ends are clear. In the intermediate band, the runs that exceed REF-C are the lower-accuracy ones. They also come from different learners: the 15/19 runs are LwF, DER++ and DGC, which train the backbone, and the 0/12 runs are frozen-feature heads with replay. This band therefore cannot tell whether accuracy, the learner, or how errors are distributed across classes decides the outcome. We state the result as this descriptive boundary and do not claim a single-variable mechanism.

3. Relation to worst-task calibration. A worst-task or worst-age calibration error is always at least the aggregate error, and the high-accuracy runs show that it can be no larger than ordinary class heterogeneity would produce. ACMap, SimpleCIL and the joint heads have worst-age ECE about 2 to 5 times their overall ECE, yet only 2/18 of these runs exceed REF-C. Under strong forgetting, the same reference shows an age effect far beyond class heterogeneity, with mean excess over REF-C of 12 to 62 pp on the primary benchmark. A worst-task number alone cannot distinguish these two situations. The class-preserving reference does so for each model, and we recommend reporting it next to any worst-task calibration error. This is the evaluation-practice consequence of the high-accuracy analysis.

4. Results that do not depend on this boundary. As your review noted, schedule age need not exceed every class grouping to remain a useful audit axis. The learner-dependent direction of worst-age change after global scaling, and its effect on learner selection, are established on the primary benchmark, where the age residual exceeds REF-C in 20/20 runs (Tables 1, 28, 40, 42). The order-of-magnitude effects of score scale and calibration data hold at fixed checkpoints, including the high-accuracy ACMap model (Tables 2 and 3). At high accuracy, even where the residual is comparable to class heterogeneity, a 5% overall risk target still leaves the worst age group of ACMap at 5.7 to 7.2% risk (Table 46).

We have added the order-independence argument and this boundary to Section 4.2 and to the evidence map at the start of Section 4, and the grouped counts as the new Table 48 in Appendix J of the updated PDF, shown in magenta. Section 4.2 now also recommends reporting the class-preserving reference next to any worst-group or worst-task calibration error. We hope this clarifies the scope of the age-specific claim, and we would be glad to discuss the boundary or its presentation further. If it addresses your remaining concern, we would be grateful if you would reconsider your assessment.

Reviewer_ZhezOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies calibration in class-incremental learning, where new classes are introduced sequentially. It examines how global temperature scaling can improve aggregate calibration while leaving substantial calibration errors within groups of classes introduced at different stages. The paper proposes a group-aware evaluation protocol and uses group-conditioned temperature scaling to investigate these errors. While this is an interesting and useful direction, I have the following concerns:

  1. The contribution is primarily empirical, so the conclusions depend heavily on the experimental design. For example, ECE is estimated using 15 confidence bins, and this choice can affect both aggregate and group-level estimates, particularly when groups contain fewer samples. The paper does include sensitivity analyses with 10, 15, and 20 bins. However, these analyses use the secondary chronological split and coarse age groups, rather than the primary stratified split and exact-age groups. Consequently, they do not directly establish the robustness of the primary results. Applying these controls to the same checkpoints and evaluation protocol as the main experiments would make the evidence more convincing.

  2. The practical significance of the high-accuracy results is not established. In Table 2, appropriate global temperature scaling already removes most of the initially large calibration error. On SplitCIFAR-100, oracle age-conditioned scaling changes overall ECE from 1.94% to 1.78%, while worst-age ECE remains essentially unchanged, moving from 6.55% to 6.56%. On Tiny-ImageNet, it reduces overall ECE from 1.25% to 1.15% and worst-age ECE from 4.75% to 3.93%. These additional improvements are modest, and the oracle requires true group information unavailable at inference. These results still support the observation that aggregate and group-level calibration can differ, but they leave the practical importance of that difference unclear. Paired uncertainty estimates and downstream decision experiments in these high-accuracy settings would help establish when the audit provides consequential information beyond aggregate metrics.

  3. The writing makes is hard to follwo this paper. I had to read several sections multiple times to understand precisely what was being evaluated and what conclusion each experiment supported. The paper moves between different group definitions, calibration-data sources, temperature search ranges, and experimental protocols. Although these distinctions need to be dicussed, their implications are difficult to track, especially when important supporting analyses use different settings from the main experiments. A clearer presentation of the question addressed by each experiment, the factors held fixed, and the specific conclusion supported would substantially improve the paper.

Given the primarily empirical nature of the contribution, I would need more convincing controlled experiments of robustness and practical relevance to support acceptance. The paper contains experimental material, but the connection between these results and its broader conclusions remains insufficiently clear. As it stands, I do not recommend acceptance.

Critical strengths

see above.

Critical weaknesses

see above.

Questions for authors

see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.