acceptodds
Under review as a conference paper at ICLR 2027

UnL: Unlocking Label Likelihoods in Multimodal Large Language Models for Class-Incremental Learning

Abstract

Multimodal large language models (MLLMs) have shown strong potential for class incremental learning (CIL), as flexible semantic generation supports the acquisition of new classes while retaining recognition of previous ones. In this paper, we reveal overlooked inference interference induced by label-space expansion, with persistent generative preference among candidate classes progressively biasing predictions toward newly introduced classes and degrading recognition of previous ones. To address this issue, we propose Unlocking Label Likelihoods (UnL), which mitigates inference interference by calibrating preference from generative evidence. Specifically, UnL derives generative evidence over the accumulated label space through Candidate Scoring and dynamically updates a calibration queue using the derived evidence, enabling Online Calibration to estimate a class-wise correction via Sinkhorn matching for subsequent predictions. Theoretical analysis establishes exact absorption of the shared additive class-wise preference component and bounded sensitivity of calibrated pairwise evidence to sample-specific deviations. Extensive experiments on multiple CIL benchmarks show consistent improvements over existing methods, with larger gains as the number of incremental tasks increases.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Class-Incremental Learning, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cHyVOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies inference interference induced by label-space expansion as an overlooked source of performance degradation in generative class-incremental learning (CIL). To address this, the authors propose UnL^2, which combines candidate likelihood scoring with online Sinkhorn-based calibration to mitigate persistent generative preferences among classes.

Critical strengths

S1. Insightful problem identification and analysis

The paper provides a clear analysis showing that old-class performance degradation can arise from label-space expansion rather than solely from catastrophic forgetting during parameter updates. The controlled experiments separating these two effects offer a valuable perspective on generative CIL.

S2. Effective method with strong empirical results

UnL² provides a relatively simple inference-time solution without modifying the training objective. The experiments demonstrate consistent improvements across different incremental settings, with particularly substantial gains in longer task sequences. The ablation studies and theoretical analysis further support the effectiveness of the proposed calibration.

Critical weaknesses

W1. The underlying assumption about generative preference is insufficiently justified

The proposed method assumes that persistent class-wise generative preferences represent undesirable biases that should be removed through calibration. However, such preferences may also reflect legitimate class priors, semantic similarities, or meaningful discriminative evidence. Although the theoretical analysis establishes robustness to shared additive preference shifts, it does not demonstrate that the estimated corrections correspond specifically to harmful inference interference. Consequently, it remains unclear whether the method can reliably distinguish unwanted generative bias from useful predictive information, particularly under distribution shifts or non-uniform class distributions.

Questions for authors
  1. Bias versus useful evidence (same as W1): How can the authors distinguish harmful generative preference from legitimate class-prior information or discriminative evidence? Could controlled experiments with injected class-wise biases help validate this assumption?
  2. Robustness to distribution shifts: Since Online Calibration matches predictions to a uniform target marginal, how does the method perform under severe class imbalance, changing class priors, or nonstationary test streams? Could this calibration inadvertently suppress useful class-frequency information?
  3. Dependence on test-time history: How much of the improvement depends on accessing previously processed test samples? Would comparable performance be achievable using a fixed calibration set or corrections estimated from training data?
  4. Scalability: How do inference latency and memory consumption scale with the number of classes?
5 comments
AuthorsOpenReview ↗

Response to Reviewer cHyV (1/2)

We sincerely thank the reviewer for their careful reading of our manuscript and constructive feedback. We appreciate the opportunity to clarify these points and provide detailed responses below.

W1 Shared generative preference and useful predictive evidence

Manuscript Theorem 1 characterizes the effect of shared additive class-wise score perturbations without assigning every persistent score component the meaning of harmful bias. Adding the same class-wise vector dd to the queued evidence and current sample changes the Sinkhorn objective by the constant mπ⊤dm\pi^\top d under its fixed column marginal. Optimal soft assignments remain unchanged, while the induced class-wise correction absorbs the shift up to a class-independent offset. Under the theorem's positive-marginal and dual-curvature assumptions, sample-specific residuals produce bounded changes in calibrated pairwise evidence. Manuscript Fig. 5 tests these perturbation properties. Recognition experiments then assess their relevance to naturally occurring scores. Candidate classes with stronger generative preference attract more erroneous shifts under fixed-parameter label-space expansion (manuscript Fig. 1(c)), and preference estimates remain highly consistent across disjoint image subsets (manuscript Fig. 4(b)). Manuscript Fig. 7 illustrates corrections to valid-candidate rankings, while Table 3 reports the resulting recognition gains. When test histories become nearly single-class, repeated genuine visual support can resemble a shared score component, making evidence selection crucial. Manuscript Figs. 9–10 analyze this regime and evaluate Fixed References and Diversity-Preserving Queue as alternative sources of cross-class calibration evidence.

Q2 Nonstationary distributions and concentrated streams

Online Calibration estimates class-wise corrections from recent unlabeled scores, making the cross-class diversity of queued evidence important to estimation reliability. Rank-decayed test streams retain substantial gains at imbalance ratio 100, improving from 72.95% to 82.53% (manuscript Fig. 4(c)). A more concentrated, nonstationary protocol tests three consecutive 2,000-position segments with freshly drawn Dirichlet class priors while the calibration queue persists across transitions (Response Table R4). At α=1.0\alpha=1.0, Online Calibration improves over Candidate Scoring by 10.98 points. At α=0.1\alpha=0.1, repeatedly observed dominant classes reduce cross-class variation, and FIFO calibration drops by 5.20 points. Fixed References and Diversity-Preserving Queue retain stronger results because they change the source of calibration evidence, as also motivated by the concentration and transition analyses in manuscript Figs. 9–10. Specifically, Theorem 1 and stable preference estimates across disjoint subsets (manuscript Fig. 4(b)) motivate Fixed References, which scores 128 randomly sampled external images from classes disjoint from the entire incremental label space. Diversity-Preserving Queue instead keeps one score vector per uncalibrated predicted class, refreshes its representative upon repeated predictions, and activates Sinkhorn after 128 distinct predicted classes are represented. Response Table R1 in the General Response reports complete-sequence final-task Last Accuracy for ImageNet-R 10-, 20-, 25-, 50-, and 100-task divisions. Both alternatives outperform GMM throughout. At 100 tasks, Fixed References and Diversity-Preserving Queue achieve 86.22% and 86.04%, compared with 77.48% for GMM and 87.05% for default Online Calibration.

Response Table R4. Accuracy (%) over 6,000 ImageNet-R stream positions with class priors redrawn after each 2,000-position segment, summarized across five runs. Online Calibration uses FIFO capacity M=800M=800. Fixed References uses 128 randomly sampled, class-disjoint unlabeled images, and Diversity-Preserving Queue uses capacity M=128M=128.

Dirichlet concentrationCandidate ScoringUnL² (Online Calibration)UnL² (Fixed References)UnL² (Diversity-Preserving Queue)
α=1.0\alpha=1.072.1783.1583.2782.23
α=0.1\alpha=0.176.9971.7984.5581.15
AuthorsOpenReview ↗

Response to Reviewer cHyV (2/2)

Q3 Dependence on test history and calibration evidence

Response Table R5 compares correction sources on the same 5,200-image held-out suffix at the final ImageNet-R 100-task checkpoint. For each of five test permutations, the first 800 images form a disjoint prefix from which one correction is estimated and then frozen during suffix inference. Continuous Online Calibration instead updates causally from preceding unlabeled evidence. Fixed References estimates a correction from 128 randomly sampled external images whose classes are disjoint from the entire incremental label space C1:T\mathcal{C}_{1:T}. Their labels serve only to select the class-disjoint reference pool and are excluded from Sinkhorn matching. Unlabeled latest-task training images and Diversity-Preserving Queue provide further evidence-source comparisons.

Response Table R5. Accuracy (%) on the 5,200-image held-out suffix at the final ImageNet-R 100-task checkpoint, summarized across five test permutations. Fixed References uses 128 randomly sampled, class-disjoint unlabeled images. Diversity-Preserving Queue uses capacity M=128M=128, and continuous Online Calibration uses FIFO capacity M=800M=800.

Calibration evidenceAccuracy (%)
Candidate Scoring77.38
201 latest-task training images, labels unused81.20
UnL² (Fixed References, 128 images)86.22
UnL² (Diversity-Preserving Queue, capacity 128)86.49
Fixed correction from 800-image test prefix86.59
UnL² (Continuous Online Calibration)87.15

Fixed References reaches 86.22% without using the target-test stream, compared with 87.15% for continuous Online Calibration. A correction frozen after the 800-image prefix reaches 86.59%, only 0.56 points below continuous updates on the same suffix. Calibration from 201 latest-task training images is less effective because those images cover only the two newest classes among 200 candidates. Strong results with independent references and a fixed prefix show that persistent generative preference can be estimated without continuous updates. Fixed References provides order-independent target predictions using an external unlabeled set, while Online Calibration incorporates recent test evidence causally.

Q4 Computational complexity and practical inference cost

Candidate Scoring evaluates the autoregressive likelihoods of all valid class answers while reusing the cached image-question prefix across candidates. Manuscript Table 4 reports 27.42× and 33.84× speedups over sequential candidate scoring for Qwen2.5-VL-7B and LLaVA-OneVision-7B, respectively. End-to-end inference includes the additional cost of evaluating candidate likelihoods, requiring 1.475 seconds per image for UnL² compared with 0.520 seconds for GMM on LLaVA-OneVision-7B. MiniGPT-4 profiling separates candidate evaluation from Sinkhorn matching (Response Table R6). At 200 classes, Candidate Scoring takes 1,175.1 ms per image, whereas a Sinkhorn solve takes 18.1 ms. Peak GPU memory remains between 9.10 and 9.38 GiB across the tested candidate counts. Shared-prefix caching thus accelerates exhaustive likelihood evaluation relative to sequential scoring, while the bounded calibration queue keeps correction overhead small.

Response Table R6. MiniGPT-4 component measurements on one RTX 4090 using candidate chunks of 50 and an Online Calibration FIFO queue of capacity M=800M=800. Candidate Scoring timings average five images and three repetitions after warm-up. Sinkhorn timings average 20 CPU solves.

Candidate classesCandidate Scoring (ms/image)Sinkhorn (ms/solve)Peak GPU memory (GiB)
20494.74.29.10
50490.68.89.24
100731.49.99.38
2001,175.118.19.38

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Reviewer_QCo6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on an overlooked source of forgetting in generative class-incremental learning: as the candidate label space grows, MLLM predictions drift toward newly added classes because of a persistent, class-dependent generative preference, independent of parameter updates. The authors propose UnL², which replaces free-form generation with length-normalized likelihood scoring over all accumulated candidate labels (Candidate Scoring), and estimates a class-wise additive correction from a bounded FIFO queue of previously scored test samples via Sinkhorn matching to a uniform target marginal (Online Calibration). Experiments on Tiny-ImageNet and ImageNet-R show consistent gains over baselines.

Critical strengths
  • Useful observation that the bias carries over to MLLMs. Classifier bias under a growing label space is well known in conventional CIL, but the paper shows convincingly that generative MLLMs suffer the same problem. The controlled comparison in Fig. 1(a) is the strong part: with the label space held fixed, successive checkpoints actually improve old-class accuracy, while with the model held fixed, growing the label space alone causes the drop.
Critical weaknesses

The paper identifies that label-space expansion degrades generative classification performance in MLLMs. However, the proposed solution relies on an artificial test-time assumption that fails in real-world deployment and replicates well-known CIL concepts under a new framing:

  • The uniform assumption breaks in real use. The method forces the test batch to look evenly spread across classes. If the real test data is unbalanced, or if many images of the same class arrive in a row, the method treats that as model bias and pushes the correct class down.

  • This is a fix at test time, not better learning. Nothing about the model gets better at handling more classes. It is a post-processing trick that works because the benchmarks happen to have equal numbers of samples per class.

  • The problem they find is already known. Old classes losing out as the label list grows, while the features stay fine, is the same classifier bias studied in Bias Correction/BiC [1] and Weight Aligning/WA [2]. Calling it generative preference does not make it new.

  • How it is not test-time adaptation Test-time adaptation adjusts the model to test data it has not seen. Here the target is fixed (uniform prior) and decided in advance, and the method just forces predictions to match it.

  • The paper is hard to follow, and key quantities are never defined. Fig. 1(a) takes real effort to parse: line 74 says accuracy is evaluated over "the same old-class label space", but never says whether that means all previously seen classes or only the first task. The figure also has no clear legend. Fig. 1(b): the y-axis is a percentage shift toward new classes but there is no mention of how is this even computed. Similarly, persistent generative preference, which the whole paper is built on, is never defined, yet Fig. 1(c) and Fig. 10 plot it as a standardized value with no formula given. The imbalance ratio in Fig. 4(c) is also never defined. Since the motivation rests entirely on these figures, this is not just a polish issue.

  1. Wu et. al., Large Scale Incremental Learning, CVPR 2019.
  2. Zhao et. al., Maintaining Discrimination and Fairness in Class Incremental Learning, CVPR 2020
Questions for authors
  1. Have you evaluated the method on an imbalanced test stream? Concretely, after task 8, suppose every incoming image belongs to a single class from task 2. The uniform target then forces the queue marginal away from the truth, so the correction should actively suppress the correct class. What accuracy does your method get in that setting?
7 comments
AuthorsOpenReview ↗

Rsponse to Reviewer QCo6 (1/3)

We sincerely thank the reviewer for their careful reading of our manuscript and constructive feedback. We appreciate the opportunity to clarify these points and provide detailed responses below.

Single-class goldfinch evaluation

Goldfinch-only arrivals expose the dependence of FIFO correction on cross-class evidence. Although high goldfinch likelihood provides valid visual support, repeated goldfinch images make that support appear as a persistent class-wise component in the unlabeled queue, causing Sinkhorn calibration to attenuate the correct score. At Task 8 of the ImageNet-R 10-task sequence, all 160 accumulated candidates remain available. Candidate Scoring recognizes 92.31% of the 39 distinct goldfinch images, while default FIFO Online Calibration falls to 7.18% (Response Table R3). The shared-component analysis in manuscript Theorem 1 and cross-subset stability in Fig. 4(b) motivate Fixed References, which estimates a correction from 128 randomly sampled images belonging to classes disjoint from the entire incremental label space. Evaluated against the same candidates, this independent evidence raises goldfinch accuracy to 94.36%. Manuscript Figs. 9–10 also motivate the Diversity-Preserving Queue by showing how concentrated FIFO histories lose class diversity. That queue retains one generative evidence vector per class predicted by uncalibrated Candidate Scoring and refreshes each representative upon repetition. Calibration begins only after 128 distinct predicted classes are represented. The 39 distinct goldfinch images cannot produce that many predicted classes under deterministic scoring, so this variant retains Candidate Scoring's 92.31% accuracy.

Response Table R3. Accuracy (%) at Task 8 of ImageNet-R with 160 candidate classes, evaluated on 39 distinct goldfinch images and two intervals of a 2,400-position replay over five runs. Online Calibration uses FIFO capacity M=800M=800. Fixed References uses 128 randomly sampled, class-disjoint unlabeled reference images. Diversity-Preserving Queue uses capacity M=128M=128 and falls back to Candidate Scoring until 128 predicted classes are represented.

Method39 distinct imagesPositions 1–800Positions 801–2,400
GMM87.1885.7086.71
Candidate Scoring92.3191.3892.08
UnL² (Online Calibration, FIFO)7.185.385.38
UnL² (Fixed References)94.3694.1594.34
UnL² (Diversity-Preserving Queue)92.3191.3892.08

W1 Uniform marginal under imbalanced and nonstationary streams

A uniform target in Sinkhorn matching supplies a class-symmetric reference for estimating persistent generative preference without using unknown test-label priors (manuscript Eq. (3)). Its constraint applies to the aggregate soft assignment of calibration evidence. Final predictions remain the argmax of corrected generative scores, although the estimated correction can change their rankings. Calibration reliability therefore depends on which classes are represented in the evidence, rather than on global label imbalance alone. At imbalance ratio 100, Online Calibration improves ImageNet-R accuracy from 72.95% to 82.53% and slightly exceeds correction toward the oracle realized marginal (82.30%, manuscript Fig. 4(c)). With class priors redrawn across three consecutive Dirichlet segments, Online Calibration raises accuracy from 72.17% to 83.15% at α=1.0\alpha=1.0. More severe concentration at α=0.1\alpha=0.1 reduces FIFO accuracy to 71.79%, while Fixed References and Diversity-Preserving Queue reach 84.55% and 81.15%. Manuscript Figs. 9–10 explain the contrast through the loss of cross-class variation in homogeneous histories. Response Table R1 in the General Response evaluates both alternatives over complete ImageNet-R 10-, 20-, 25-, 50-, and 100-task sequences. Their final-task Last Accuracy exceeds GMM in every division. At 100 tasks, Fixed References and Diversity-Preserving Queue achieve 86.22% and 86.04%, compared with 77.48% for GMM and 87.05% for default Online Calibration.

AuthorsOpenReview ↗

Response to Reviewer QCo6 (2/3)

W2 Scientific contribution beyond generic post-processing

UnL² identifies structured inference interference in generative CIL and develops a correction directly from the likelihood evidence that reveals it. Under the original 20-class candidate space, successive parameter updates improve old-class recognition. Expanding the candidate space to 200 classes under fixed final parameters instead reduces accuracy (manuscript Fig. 1(a)). Candidate competition accompanies any label-space expansion, but the observed generative prediction shifts are associated with persistent class-wise generative preference (manuscript Figs. 1(b–c) and 4(b)). Candidate Scoring makes length-normalized autoregressive likelihoods explicit for all accumulated classes (manuscript Eq. (2)), exposing a common score representation from which Online Calibration estimates class-wise corrections through Sinkhorn matching (manuscript Eq. (3)). Manuscript Theorem 1 characterizes the correction's exact invariance to shared additive perturbations and bounded sensitivity to sample-specific residuals under its assumptions. Recognition ablations establish the practical value of combining the two components. At 100 tasks, identical Online Calibration raises Candidate Scoring from 77.33% to 87.05% Last Accuracy, while GMM rises from 77.48% to 79.24% (manuscript Table 3). The diagnostic evidence, method design, and measured gains therefore address the same structured likelihood phenomenon.

W3 Relation to BiC and Weight Aligning

BiC and Weight Aligning (WA) address old/new-class bias associated with incremental training through adjustments to new-class logits and classifier weights, respectively. Their group-level operations preserve the ranking of classes within the old-class group. Generative MLLMs instead compare autoregressive likelihoods of candidate answer sequences, where class-specific preference can appear even among classes trained together. At the first checkpoint of the 100-task sequence, Candidate Scoring assigns all 45 test images to one of two jointly trained classes, before an old/new grouping arises. Manuscript Fig. 1(a) further separates parameter-update effects from candidate expansion, with Fig. 1(c) associating the resulting shifts with generative preference. A group-preserving oracle tests the importance of correction granularity by providing true old/new group membership and correct labels for new-class images while preserving the original ordering among old classes. On the 5,200-image held-out suffix at the final 100-task checkpoint, the oracle attains 77.78%. Class-wise calibration reaches 86.59% on the same suffix using a correction estimated from a disjoint unlabeled 800-image prefix. Remaining errors under oracle group information establish the importance of correcting rankings within the old-class group, which group-only reweighting leaves unchanged.

AuthorsOpenReview ↗

Response to Reviewer QCo6 (3/3)

Response to Reviewer QCo6 (3/3)

W4 Causal test-time calibration

Online Calibration updates a class-wise correction from unlabeled test history while keeping model parameters fixed (manuscript Sec. 4.2). Each incremental evaluation begins with an empty FIFO queue and a zero correction. Candidate Scoring first evaluates the incoming image, whose prediction uses a correction estimated solely from preceding score vectors. The current vector enters the queue only after prediction. Sinkhorn matching obtains the correction by aligning the queued soft-assignment marginal with the target marginal, while the final label is selected by the argmax of corrected generative scores (manuscript Eq. (3)). Calibration is therefore causal and label-free, with adaptation confined to the correction rather than model parameters.

W5 Definitions and controlled-comparison interpretation

Manuscript Fig. 1(a) compares two controlled evaluations on identical first-task test images. Checkpoint evolution varies θ1,…,θT\theta_1,\ldots,\theta_T with the original 20-class label space C1\mathcal{C}_1 held fixed. Label-space expansion instead fixes θT\theta_T and enlarges the candidate set from C1\mathcal{C}_1 to C1:t\mathcal{C}_{1:t}, reaching 200 classes. Under a fixed checkpoint, AtA_t contains previously correct old-class images before expansion, and Et⊆AtE_t\subseteq A_t contains those reassigned to newly added classes. Their ratio ∣Et∣/∣At∣|E_t|/|A_t| defines the shift rate (manuscript Fig. 1(b)). For a newly introduced class cc, generative preference gc=(sˉc−μ)/σg_c=(\bar s_c-\mu)/\sigma standardizes its mean Candidate Scoring value over old-class test images using the mean and standard deviation across classes entering at the same transition (manuscript Eq. (2), Fig. 1(c)). Rank-decayed sampling in manuscript Fig. 4(c) uses pc∝IR−rc/(K−1)p_c\propto\mathrm{IR}^{-r_c/(K-1)} with randomly permuted ranks, where IR=max⁡cpc/min⁡cpc\mathrm{IR}=\max_c p_c/\min_c p_c.

Reviewer_B67VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies generative class-incremental learning (CIL) with multimodal large language models (MLLMs). The authors identify an overlooked inference-interference problem: as the accumulated label space expands, the model exhibits a persistent generative preference toward newly introduced classes, systematically biasing old-class predictions toward new classes. Controlled experiments disentangle this effect from that of parameter updates, showing that prediction shifts concentrate disproportionately on new classes and correlate positively with class-wise generative preference; the phenomenon is replicated across multiple MLLM backbones. The authors propose UnL2 method, which scores candidates by normalized answer likelihoods and corrects class-wise preference via Sinkhorn matching on queued test evidence. Theory shows this correction exactly absorbs shared class-wise preference while remaining robust to sample-specific deviations. Experiments show consistent improvements over generative and discriminative baselines, with gains growing as the number of incremental tasks increases, along with backbone generalization and inference speedups.

Critical strengths
  1. This paper is Well-motivated. Figure 1(a)'s controlled decomposition of "parameter updates" versus "label-space expansion" is the most valuable part of the paper; the preference–shift correlation analysis in Figure 1(c), replicated on two heterogeneous backbones in Appendix Figure 10 (ρ=0.66/0.51), gives the analysis credibility independent of the proposed method.
  2. The authors conduct comprehensive ablations. The "GMM w/ OC" ablation rules out the alternative explanation of history access alone; comparisons with running mean / EMA / predictive frequency / content-free calibration (Figure 6) show Sinkhorn matching outperforms simple statistics and PMI-style calibration; sensitivity to 7 evaluation orders (Figure 8), queue-capacity sweeps (Figure 4a), and target-marginal comparisons (Figure 4c) cover the main concerns one would raise about an online calibration method.
Critical weaknesses
  1. Online Calibration uses test-time information, changing the evaluation protocol. The method calibrates the current prediction using "previously processed test samples," which constitutes test-time adaptation / transductive inference; all discriminative baselines in the main tables (Continual-CLIP, EASE, MACIL, etc.) are evaluated under a strictly inductive, independent-prediction protocol. Although the GMM w/ OC ablation partially addresses the fairness concern: (a) the paper should state this protocol difference earlier and more explicitly in the Method and Experiments sections, and ideally report "discriminative baseline + the same online calibration" comparisons; (b) Table 3 shows that at 100 tasks, UnL2 w/o OC achieves Last=77.33, on par with GMM's 77.48 — i.e., under the longest-sequence setting, all of the gain comes from test-time calibration. This is a substantive limitation for real streaming deployments (unknown test distribution and ordering, no advance shuffling of the full test set), and the main text does not discuss it at all; (c) performance depends on evaluation order: Figure 8 shows gaps of up to ~3.1 pp between orders (Tiny-ImageNet 5 tasks: reverse-tasks 78.70 vs reverse-loader 81.83), and the "random" order itself presupposes advance shuffling of the complete test set.

  2. The collapse of UnL2 w/o OC at 100 tasks (Avg=68.94) in Table 3 is left completely unexplained. At 10–50 tasks this variant clearly outperforms GMM on both Avg and Last (e.g., 50 tasks: 84.29/84.87 vs 83.98/83.35), but at 100 tasks its Avg drops abruptly by 15.35 pp to 68.94 — 9.25 pp below GMM — while its Last (77.33) remains on par with GMM. This non-smooth pattern implies a performance collapse at some intermediate sessions, could the authors explain this further ?

Questions for authors

See weaknesses above

4 comments
AuthorsOpenReview ↗

Response to Reviewer B67V (1/2)

We sincerely thank the reviewer for their careful reading of our manuscript and constructive feedback. We appreciate the opportunity to clarify these points and provide detailed responses below.

W1(a) Evaluation protocol and matched calibration of discriminative baselines

Standard discriminative CIL baselines classify each test image independently. Online Calibration instead estimates a correction causally from an initially empty FIFO queue, applying it to the current Candidate Scoring vector before that vector is enqueued (manuscript Sec. 4.2). Predictions therefore use preceding unlabeled evidence without test labels or future observations. Fixed-Reference Calibration provides a separate order-independent protocol in which one correction is estimated from 128 images randomly sampled from classes disjoint from the complete incremental label space C1:T\mathcal{C}_{1:T} (Response Table R1). Class labels serve only to construct the reference pool and are not used by Sinkhorn matching. To compare calibration effects under matched history access, we apply the same Online Calibration procedure to discriminative baselines and generative scores. MACIL gains modestly across all ImageNet-R task divisions (Response Table R2). At 100 tasks, frozen CLIP ViT-B/16 improves by 0.93 points, Continual-CLIP ViT-B/16 declines by 0.76 points, and GMM increases from 77.48% to 79.24%. Candidate Scoring improves from 77.33% to 87.05% under the identical calibration procedure (manuscript Table 3). Matched access to preceding unlabeled samples thus produces markedly different calibration benefits, highlighting the role of explicit candidate-level generative evidence.

Response Table R2. Avg / Last Accuracy (%) of MACIL with and without causal Online Calibration on complete ImageNet-R incremental sequences. Both calibrated and uncalibrated evaluations use the same prediction order. Online Calibration uses an initially empty FIFO queue with capacity M=800M=800.

TasksMACIL Avg / LastMACIL + Online Calibration Avg / Last
1085.76 / 81.8286.63 / 82.75
2084.25 / 79.4685.07 / 80.44
2581.90 / 76.1882.81 / 77.03
5077.47 / 70.1078.43 / 70.99
10063.04 / 55.0263.98 / 55.85

W1(b) Sources of improvement along long incremental sequences

UnL² combines two operations that address different sources of error in autoregressive recognition. Candidate Scoring evaluates only valid class answers, avoiding ambiguity from free-form generation and semantic matching. Online Calibration then corrects persistent generative preference among those valid candidates, as illustrated in manuscript Fig. 7. On ImageNet-R at 100 tasks, calibration increases Candidate Scoring's Last Accuracy from 77.33% to 87.05%, whereas applying the same calibration procedure to GMM raises its accuracy from 77.48% to 79.24% (manuscript Table 3). Nearly identical uncalibrated accuracies therefore lead to substantially different corrected results. A correction estimated once from an unlabeled 800-image test prefix achieves 86.59% on the disjoint 5,200-image suffix at the final checkpoint, compared with 87.15% for continuous causal updates on the same suffix. Persistent class-wise preference can consequently be estimated from a finite set of unlabeled observations, with online updates providing a additional gain.

AuthorsOpenReview ↗

Response to Reviewer B67V (2/2)

W1(c) Dependence on evaluation order

Evaluation order changes the composition of the bounded calibration queue, although each prediction still uses only previously observed evidence. Class-diverse histories help distinguish persistent generative preference from sample-dependent support. Extended single-class runs reduce that variation and may cause the correction to suppress genuine support for the dominant class. Across seven arrival orders and eight incremental settings, UnL² outperforms GMM, with greater variation under concentrated orders (manuscript Fig. 8). Gains under rank-decayed sampling at imbalance ratio 100 further distinguish global class-frequency imbalance from local queue homogeneity (manuscript Fig. 4(c)). Manuscript Figs. 9–10 isolate the latter effect using M=128M=128, distinct from the default FIFO capacity M=800M=800. Theorem 1 and the stable generative preference across image subsets in Fig. 4(b) motivate Fixed References, which estimates a correction from 128 class-disjoint external images and is independent of test arrival order. The observed loss of cross-class evidence motivates the Diversity-Preserving Queue, which retains one vector per class predicted by uncalibrated Candidate Scoring, updates repeated-class representatives, and enables Sinkhorn after 128 distinct predicted classes are represented. Response Table R1 in the General Response compares final-task Last Accuracy across complete ImageNet-R 10-, 20-, 25-, 50-, and 100-task sequences. Both alternatives outperform GMM in every division. At 100 tasks, Fixed References and Diversity-Preserving Queue reach 86.22% and 86.04%, respectively, against 77.48% for GMM and 87.05% for default Online Calibration.

W2 Early-checkpoint Avg Accuracy without Online Calibration

Manuscript Fig. 11(h) shows the per-checkpoint Last Accuracy trajectory, while supplementary statistics locate the low 100-task Candidate Scoring Avg Accuracy primarily in the earliest checkpoints. Accuracy averages 59.62% over the first ten evaluations and 77.92% over the final 25. Equal weighting across checkpoints therefore yields 68.94% Avg despite 77.33% Last. At the first checkpoint, Candidate Scoring assigns all 45 test images to one of the two classes trained together in that task, revealing a strongly concentrated ranking before any old/new-class imbalance exists. A matched comparison on the same 3,008 images with 100 accumulated classes further distinguishes checkpoint behavior from candidate-space size. The 100-task checkpoint predicts 83 classes with 65.43% accuracy, compared with 98 classes and 87.13% for the 50-task checkpoint. Stronger prediction concentration is thus associated with finer task increments even at the same candidate-space size. By correcting class-wise preference in the resulting generative evidence, Online Calibration raises 100-task Avg / Last Accuracy to 87.00% / 87.05% (manuscript Table 3).

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Dear Reviewer,

We sincerely thank you for your time and insightful comments, which have substantially strengthened our work. As the discussion phase draws to a close, we would greatly appreciate your feedback on our response. The gains of UnL² stem from its methodological design rather than from historical information access. Identical history-based calibration raises GMM only from 77.48% to 79.24% on ImageNet-R with 100 tasks, whereas UnL² reaches 87.05%. This gap shows that the comparable class-wise generative evidence from Candidate Scoring is essential for correcting persistent generative preference. Such evidence also supports estimating the correction without any test history: UnL² (fixed references) achieves 86.22%, independent of evaluation order, and outperforms GMM across all incremental settings. Without this correction, Candidate Scoring produces highly concentrated predictions at the earliest checkpoints, which accounts for its low 100-task Avg accuracy. Online Calibration restores the Avg / Last accuracy to 87.00% / 87.05%. We would be glad to address any remaining concerns.

Sincerely, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Controlled evaluations identify inference interference as a distinct source of recognition degradation in generative class-incremental learning (CIL). On the same first-task test images, successive parameter updates improve old-class accuracy by 15.8 percentage points when the original 20-class candidate space is held fixed. Expanding the candidate space to 200 classes under the fixed final checkpoint instead reduces accuracy by 13.0 points (manuscript Fig. 1(a)). Beyond the competition inherent to any expanding label space, prediction shifts in generative CIL exhibit a systematic class-wise pattern. Newly introduced classes with stronger generative preference receive more erroneous shifts (manuscript Fig. 1(c), ρ=0.47\rho=0.47, p<10−10p<10^{-10}). Preference estimates remain highly consistent across disjoint image subsets (manuscript Fig. 4(b), ρ=0.998±0.001\rho=0.998\pm0.001), with corresponding trends observed across MLLM backbones (manuscript Fig. 12). Together, the controlled comparisons and preference measurements identify a structured inference effect that cannot be attributed solely to parameter updating.

UnL² addresses this inference interference through complementary Candidate Scoring and Online Calibration. Candidate Scoring replaces free-form response generation with length-normalized autoregressive likelihoods for every valid candidate answer, producing explicit generative evidence over the accumulated label space (manuscript Eq. (2)). Online Calibration estimates a class-wise correction from preceding unlabeled evidence through Sinkhorn matching, using a uniform target as a class-symmetric calibration reference rather than an assumed test-label distribution (manuscript Eq. (3)). Manuscript Theorem 1 establishes exact absorption of shared additive class-wise perturbations and bounded sensitivity of calibrated pairwise evidence to sample-specific residuals under its stated assumptions. Controlled perturbations in manuscript Fig. 5 examine these properties, and manuscript Fig. 7 illustrates how the two components address different recognition errors.

Long-sequence ablations demonstrate the value of this coupling. On ImageNet-R at 100 tasks, Online Calibration increases Candidate Scoring's Last Accuracy from 77.33% to 87.05%, compared with 77.48% to 79.24% when the same calibration procedure is applied to GMM (manuscript Table 3). Improvements across datasets and incremental checkpoints appear in manuscript Tables 1–2 and Fig. 3. Calibration quality also depends on the diversity of evidence used to estimate persistent preference. Online Calibration improves accuracy from 72.95% to 82.53% under rank-decayed sampling at imbalance ratio 100 (manuscript Fig. 4(c)), while prolonged near-single-class arrivals can make genuine visual support resemble shared preference in a FIFO queue (manuscript Figs. 9–10). Theorem 1 and the cross-subset stability in Fig. 4(b) motivate Fixed References, which estimates a correction from independent images evaluated against the same candidates. The concentration analysis motivates the Diversity-Preserving Queue, which maintains one generative evidence vector per class predicted by uncalibrated Candidate Scoring and applies Sinkhorn only after 128 distinct predicted classes are represented. Both approaches change the evidence used to estimate the same class-wise correction. Across five complete ImageNet-R incremental sequences, they preserve substantial gains over GMM, while default Online Calibration achieves the highest Last Accuracy under standard evaluation (Response Table R1).

Response Table R1. Last Accuracy (%) after the final task of each complete ImageNet-R incremental sequence. Online Calibration uses an initially empty FIFO queue with capacity M=800M=800. Fixed References uses 128 unlabeled images randomly sampled from classes disjoint from the entire incremental label space C1:T\mathcal{C}_{1:T}. Diversity-Preserving Queue uses capacity M=128M=128.

Method10 tasks20 tasks25 tasks50 tasks100 tasks
GMM84.3784.1384.0083.3577.48
UnL² (Online Calibration)89.4389.0788.8788.6887.05
UnL² (Fixed References)89.0588.8288.5488.2986.22
UnL² (Diversity-Preserving Queue)88.9688.6388.3988.1886.04

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.