- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper examines whether aggregate calibration scores give an adequate picture of reliability in class-incremental learning. It proposes an audit that reports calibration within groups defined by class introduction time, alongside confidence quartiles, expected-age quantiles, and label-dependent confidence-gap groups. The experiments use a class-stratified retrospective calibration/evaluation split, separate comparisons of calibration-data sources, and several secondary protocols. The main observation is that global temperature scaling can substantially reduce pooled ECE while leaving large and uneven errors across age groups, with the direction of the worst-group change depending on the learner. Experiments with the higher-accuracy ACMap model find smaller residual discrepancies and show that respecting the native score scale is essential. Oracle age-conditional scaling, an inference-available expected-age proxy, calibration-source comparisons, signed confidence gaps, and selective-prediction measurements help interpret these results.
Critical strengths
-
I found the main point useful. An overall calibration score can look good even when the results for some groups are much worse. Table 1 gives a clear example of this problem and helps explain why reporting only one overall score may leave out important information.
-
The experiments cover several datasets and different models. This gives the main observation more support than a result from a single setting. I also appreciate that the authors include a model with much higher accuracy to check whether the problem still appears.
-
The authors are open about the limits of their results. For example, they explain that the size of the problem differs between models and is smaller for the more accurate model they study. This makes the discussion more balanced and helps readers understand the scope of the conclusions.
Critical weaknesses
- The paper considers many experimental settings and ways of dividing the data into groups, which made some parts hard to follow. I was not always sure how the different experiments fit together or which ones most directly support the main conclusion. A simpler overview of the experiments would help readers who are less familiar with this area.
Questions for authors
N/A
The class-preserving controls address my main concern about the primary benchmark. The worst-age calibration error exceeded this reference in all 20 primary runs, but in only one of the six high-accuracy ACMap runs. I had hoped the replay experiment would help explain that difference, but overall accuracy and the accuracy gap between older and newer classes change together as replay increases. Given prior work on worst-task calibration, I’m still looking for a clearer account of when grouping by age reveals more than ordinary class differences, so I’m keeping my score at 2.