- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper identifies inference interference induced by label-space expansion as an overlooked source of performance degradation in generative class-incremental learning (CIL). To address this, the authors propose UnL^2, which combines candidate likelihood scoring with online Sinkhorn-based calibration to mitigate persistent generative preferences among classes.
Critical strengths
S1. Insightful problem identification and analysis
The paper provides a clear analysis showing that old-class performance degradation can arise from label-space expansion rather than solely from catastrophic forgetting during parameter updates. The controlled experiments separating these two effects offer a valuable perspective on generative CIL.
S2. Effective method with strong empirical results
UnL² provides a relatively simple inference-time solution without modifying the training objective. The experiments demonstrate consistent improvements across different incremental settings, with particularly substantial gains in longer task sequences. The ablation studies and theoretical analysis further support the effectiveness of the proposed calibration.
Critical weaknesses
W1. The underlying assumption about generative preference is insufficiently justified
The proposed method assumes that persistent class-wise generative preferences represent undesirable biases that should be removed through calibration. However, such preferences may also reflect legitimate class priors, semantic similarities, or meaningful discriminative evidence. Although the theoretical analysis establishes robustness to shared additive preference shifts, it does not demonstrate that the estimated corrections correspond specifically to harmful inference interference. Consequently, it remains unclear whether the method can reliably distinguish unwanted generative bias from useful predictive information, particularly under distribution shifts or non-uniform class distributions.
Questions for authors
- Bias versus useful evidence (same as W1): How can the authors distinguish harmful generative preference from legitimate class-prior information or discriminative evidence? Could controlled experiments with injected class-wise biases help validate this assumption?
- Robustness to distribution shifts: Since Online Calibration matches predictions to a uniform target marginal, how does the method perform under severe class imbalance, changing class priors, or nonstationary test streams? Could this calibration inadvertently suppress useful class-frequency information?
- Dependence on test-time history: How much of the improvement depends on accessing previously processed test samples? Would comparable performance be achievable using a fixed calibration set or corrections estimated from training data?
- Scalability: How do inference latency and memory consumption scale with the number of classes?
5 comments
Response to Reviewer cHyV (2/2)
Q3 Dependence on test history and calibration evidence
Response Table R5 compares correction sources on the same 5,200-image held-out suffix at the final ImageNet-R 100-task checkpoint. For each of five test permutations, the first 800 images form a disjoint prefix from which one correction is estimated and then frozen during suffix inference. Continuous Online Calibration instead updates causally from preceding unlabeled evidence. Fixed References estimates a correction from 128 randomly sampled external images whose classes are disjoint from the entire incremental label space . Their labels serve only to select the class-disjoint reference pool and are excluded from Sinkhorn matching. Unlabeled latest-task training images and Diversity-Preserving Queue provide further evidence-source comparisons.
Response Table R5. Accuracy (%) on the 5,200-image held-out suffix at the final ImageNet-R 100-task checkpoint, summarized across five test permutations. Fixed References uses 128 randomly sampled, class-disjoint unlabeled images. Diversity-Preserving Queue uses capacity , and continuous Online Calibration uses FIFO capacity .
| Calibration evidence | Accuracy (%) |
|---|---|
| Candidate Scoring | 77.38 |
| 201 latest-task training images, labels unused | 81.20 |
| UnL² (Fixed References, 128 images) | 86.22 |
| UnL² (Diversity-Preserving Queue, capacity 128) | 86.49 |
| Fixed correction from 800-image test prefix | 86.59 |
| UnL² (Continuous Online Calibration) | 87.15 |
Fixed References reaches 86.22% without using the target-test stream, compared with 87.15% for continuous Online Calibration. A correction frozen after the 800-image prefix reaches 86.59%, only 0.56 points below continuous updates on the same suffix. Calibration from 201 latest-task training images is less effective because those images cover only the two newest classes among 200 candidates. Strong results with independent references and a fixed prefix show that persistent generative preference can be estimated without continuous updates. Fixed References provides order-independent target predictions using an external unlabeled set, while Online Calibration incorporates recent test evidence causally.
Q4 Computational complexity and practical inference cost
Candidate Scoring evaluates the autoregressive likelihoods of all valid class answers while reusing the cached image-question prefix across candidates. Manuscript Table 4 reports 27.42× and 33.84× speedups over sequential candidate scoring for Qwen2.5-VL-7B and LLaVA-OneVision-7B, respectively. End-to-end inference includes the additional cost of evaluating candidate likelihoods, requiring 1.475 seconds per image for UnL² compared with 0.520 seconds for GMM on LLaVA-OneVision-7B. MiniGPT-4 profiling separates candidate evaluation from Sinkhorn matching (Response Table R6). At 200 classes, Candidate Scoring takes 1,175.1 ms per image, whereas a Sinkhorn solve takes 18.1 ms. Peak GPU memory remains between 9.10 and 9.38 GiB across the tested candidate counts. Shared-prefix caching thus accelerates exhaustive likelihood evaluation relative to sequential scoring, while the bounded calibration queue keeps correction overhead small.
Response Table R6. MiniGPT-4 component measurements on one RTX 4090 using candidate chunks of 50 and an Online Calibration FIFO queue of capacity . Candidate Scoring timings average five images and three repetitions after warm-up. Sinkhorn timings average 20 CPU solves.
| Candidate classes | Candidate Scoring (ms/image) | Sinkhorn (ms/solve) | Peak GPU memory (GiB) |
|---|---|---|---|
| 20 | 494.7 | 4.2 | 9.10 |
| 50 | 490.6 | 8.8 | 9.24 |
| 100 | 731.4 | 9.9 | 9.38 |
| 200 | 1,175.1 | 18.1 | 9.38 |
Dear reviewer:
Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.
Sincerely,
Authors
Response to Reviewer cHyV (1/2)
We sincerely thank the reviewer for their careful reading of our manuscript and constructive feedback. We appreciate the opportunity to clarify these points and provide detailed responses below.
W1 Shared generative preference and useful predictive evidence
Manuscript Theorem 1 characterizes the effect of shared additive class-wise score perturbations without assigning every persistent score component the meaning of harmful bias. Adding the same class-wise vector d to the queued evidence and current sample changes the Sinkhorn objective by the constant mπ⊤d under its fixed column marginal. Optimal soft assignments remain unchanged, while the induced class-wise correction absorbs the shift up to a class-independent offset. Under the theorem's positive-marginal and dual-curvature assumptions, sample-specific residuals produce bounded changes in calibrated pairwise evidence. Manuscript Fig. 5 tests these perturbation properties. Recognition experiments then assess their relevance to naturally occurring scores. Candidate classes with stronger generative preference attract more erroneous shifts under fixed-parameter label-space expansion (manuscript Fig. 1(c)), and preference estimates remain highly consistent across disjoint image subsets (manuscript Fig. 4(b)). Manuscript Fig. 7 illustrates corrections to valid-candidate rankings, while Table 3 reports the resulting recognition gains. When test histories become nearly single-class, repeated genuine visual support can resemble a shared score component, making evidence selection crucial. Manuscript Figs. 9–10 analyze this regime and evaluate Fixed References and Diversity-Preserving Queue as alternative sources of cross-class calibration evidence.
Q2 Nonstationary distributions and concentrated streams
Online Calibration estimates class-wise corrections from recent unlabeled scores, making the cross-class diversity of queued evidence important to estimation reliability. Rank-decayed test streams retain substantial gains at imbalance ratio 100, improving from 72.95% to 82.53% (manuscript Fig. 4(c)). A more concentrated, nonstationary protocol tests three consecutive 2,000-position segments with freshly drawn Dirichlet class priors while the calibration queue persists across transitions (Response Table R4). At α=1.0, Online Calibration improves over Candidate Scoring by 10.98 points. At α=0.1, repeatedly observed dominant classes reduce cross-class variation, and FIFO calibration drops by 5.20 points. Fixed References and Diversity-Preserving Queue retain stronger results because they change the source of calibration evidence, as also motivated by the concentration and transition analyses in manuscript Figs. 9–10. Specifically, Theorem 1 and stable preference estimates across disjoint subsets (manuscript Fig. 4(b)) motivate Fixed References, which scores 128 randomly sampled external images from classes disjoint from the entire incremental label space. Diversity-Preserving Queue instead keeps one score vector per uncalibrated predicted class, refreshes its representative upon repeated predictions, and activates Sinkhorn after 128 distinct predicted classes are represented. Response Table R1 in the General Response reports complete-sequence final-task Last Accuracy for ImageNet-R 10-, 20-, 25-, 50-, and 100-task divisions. Both alternatives outperform GMM throughout. At 100 tasks, Fixed References and Diversity-Preserving Queue achieve 86.22% and 86.04%, compared with 77.48% for GMM and 87.05% for default Online Calibration.
Response Table R4. Accuracy (%) over 6,000 ImageNet-R stream positions with class priors redrawn after each 2,000-position segment, summarized across five runs. Online Calibration uses FIFO capacity M=800. Fixed References uses 128 randomly sampled, class-disjoint unlabeled images, and Diversity-Preserving Queue uses capacity M=128.