acceptodds
Withdrawn from ICLR 2027

Integrating Label-Guided Decision Modeling with Cross-Modal Statistical Alignment for Medical Multimodal Class-Incremental Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical multimodal incremental learning aims to identify emerging diseases or subtypes while retaining knowledge of previously learned conditions to meet evolving diagnostic needs. Existing methods typically map multimodal information into a shared representation space and select classes based on similarity relationships among the extracted features. However, merely maintaining cross-modal alignment fails to model the discriminative space required for downstream classification tasks, making it difficult to effectively distinguish between visually similar categories. To address this, we propose a continual statistical learning framework based on frozen visual-language representations. We first introduce dimensional uncertainty modeling (), which utilizes class-conditional statistical information to dynamically calibrate the reliability of visual features, providing a more reliable basis for subsequent cross-modal statistical organization. Building on this calibrated organization, we then address the more fundamental alignment-classification mismatch through a Label-Guided Decision Module, which integrates statistics linking features to class labels to recover missing category information and transforms organized representations into discriminative decision structures. By leveraging accumulated sufficient statistics, the framework achieves replay-free continual adaptation without requiring access to historical samples or further optimization of the pre-trained encoder. Experiments across six medical continual learning protocols demonstrate the dynamic evolution of feature reliability and sustained improvements in classification performance. Notably, decision geometry improvements far exceed gains in mean class recall (MCR), which averages recall equally across classes. This further underscores the necessity of employing label-conditioned decision modeling alongside cross-modal alignment.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Medical Image Classification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G8yAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a replay-free continual-learning framework for medical multimodal class-incremental learning. It uses dimension-wise uncertainty modeling to calibrate visual feature reliability and introduces a label-guided decision module that incorporates feature-label statistics to transform aligned representations into class-discriminative decision structures. The method is evaluated on six medical class-incremental datasets and reports improvements over existing baselines, with additional analyses of feature reliability, decision geometry, robustness and component contributions.

Critical strengths
  1. The motivation and the structural information gap are clearly identified and convincing. The paper clearly identifies the objective mismatch between feature representation alignment method and downsream classification task, which makes aligning visual and textual representations not necessarily provide the decision structure needed to separate visually similar classes.
  2. The experimental analysis is thorough. The evaluation includes not only standard classification metrics but also decision-geometry measures that quantify class separation, margins, and boundary violations. These analyses are particularly informative for classification, where the quality of the learned decision boundaries is central. The geometry results provide a relatively clear picture of how uncertainty calibration and label-guided decision modeling affect both predictive performance and the underlying decision structure.
Critical weaknesses
  1. The novelty of the Label-Guided Decision Module relative to existing analytic continual-learning methods is unclear. The formulation of the module appears closely related to RanPAC’s decorrelated ridge-regression classifier. In particular, when the textual feature term is removed, the proposed ridge regression objective reduces to the same form, while the preceding random feature expansion also follows RanPAC. The main extension therefore appears to be incorporating aligned textual feature-label statistics into this existing analytic formulation. The paper should clarify the methodological novelty beyond adding the text branch and provide a direct experimental comparison with RanPAC under the same backbone and experimental setting.
  2. The experimental comparison does not sufficiently isolate where the gains come from. It is unclear whether all baselines in Table 1 use the same OpenCLIP ViT-B/16 backbone. Moreover, the current ablations do not cleanly separate the benefit of textual prompts from uncertainty calibration and label-guided decision modeling. A matched vision-only variant and experiments using a vision-only pretrained encoder, e.g., an ImageNet-pretrained ViT without a text branch, would help establish whether the gains come from the proposed components rather than the underlying multimodal representation or textual information.

Reference: Mark D. McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. RanPAC: Random Projections and Pre-Trained Models for Continual Learning. Advances in Neural Information Processing Systems (NeurIPS), 36:12022–12053, 2023.

Questions for authors
  1. Please clarify the implementation of the zero-shot baseline in Section 4.2.1 and Table 2. Since OpenCLIP zero-shot classification matches image features with textual class prompts, does the reported 12.92% use both image and text features? If so, the distinction between the zero-shot baseline and the visual-only NCC readout should be stated more clearly.
  2. Figure 3(b) needs more explanation and a clearer takeaway. Section 4.2.2 reports the evolution of one component on COVID, but it is not immediately clear what conclusion the reader should draw from it or how this observation motivates dynamic dimension-wise reliability calibration.
  3. Why does the full method report a Mean Last MCR of 80.27% in Table 1 but 80.31% in Table 3?
  4. Sections 4.3 and 4.4 could be integrated more naturally into the main experimental narrative, avoiding unnecessary fragmented paragraphs. I suggest reorganizing the experimental section around a few clear research questions, each followed by the relevant evidence and a concise takeaway, while moving some secondary tables and figures to the Appendix.
Reviewer_MMz5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses replay-free class-incremental learning for medical image classification with frozen vision-language models (OpenCLIP). It identifies two issues: visual feature reliability varies across dimensions and tasks, and cross-modal alignment alone does not determine class-discriminative decisions. To address them, the authors propose a cumulative statistical learning framework with two modules: (1) a Visual Uncertainty-Calibrated Alignment Module, which estimates a dimension-wise uncertainty matrix from within-class variance and uses it to regularize a cross-modal transformation built from accumulated second-order statistics; and (2) a Label-Guided Decision Module, which learns a shared ridge-regression classifier from label-conditioned statistics of visual and mapped textual features. All statistics are updated recursively without storing historical samples or tuning the pretrained encoders. Experiments on six medical datasets show improved Last and Avg MCR over baselines, and a decision-geometry analysis (center separation, safety margin, sample margin, boundary violation rate) suggests that label guidance reshapes class boundaries beyond what MCR alone captures.

Critical strengths
  1. The method is fully analytic, requires no replay, and avoids backpropagation through the pretrained model. The direct (no-SVD) form of the alignment mapping yields a 36.7× speedup with essentially identical accuracy.

  2. The method achieves the best Last MCR on all six datasets and the best mean Last/Avg among fully evaluated methods. The ablations show that each component contributes, and the decision-geometry analysis provides evidence that label guidance reshapes class boundaries.

  3. Six medical datasets, multiple incremental protocols, robustness to random anchors, order/partition invariance, and capacity analyses make the empirical section comprehensive.

Critical weaknesses
  1. The method combines frozen CLIP features, accumulated second-order statistics, and a ridge-regression classifier—all of which exist in prior work (RanPAC, CCA-CL, ACIL). The new elements are a diagonal variance weight in the visual covariance and a label-conditioned ridge term. The paper does not isolate which of these drives the gains, nor does it compare against a simple “frozen features + ridge regression” baseline that would test whether the added machinery is necessary.
  2. U^t is simply the diagonal of the average within-class variance, used as a regularization weight; there is no probabilistic model or posterior. The paper assumes that high within-class variance implies low feature reliability, but never validates this. A dimension with high within-class variance could still be highly discriminative. A simple correlation between per-dimension variance and per-dimension discriminative power would settle this, and it is absent.
  3. Theorem C.1 shows that perfect alignment does not determine Bayes-optimal MCR—a true but unsurprising statement. Appendix D is an algebraic re-derivation of the ridge solution. Neither result predicts when the label-guided module helps, bounds its error, or explains why the specific statistics chosen are the right ones. The theoretical sections read as post-hoc justification rather than as the basis for the design.
  4. CCA-CL and the proposed method are reported as point estimates without standard deviations on Skin8, Skin40, and Path16 (Table 1), while other methods report means and standard deviations. It is not stated which baseline numbers are reproduced under the paper’s protocol and which are copied from prior work. The gains on Path16 (+1.94) and Blood (+3.82) are small enough that they could be within run-to-run variation. No sensitivity analysis is provided for the key regularization coefficients \lambda and \beta, which are fixed to 1 without justification.
  5. Figure 4 shows large increases in M, C, and B after adding the label module, but these are measured after fitting a supervised linear classifier—of course margins increase when the classifier is trained on labels.
Questions for authors

see weaknesses

Reviewer_mGFROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

(1) The method applies fixed random feature expansion to frozen image and text embeddings, then maintains cross-modal products and class statistics as new categories arrive. (2) Within-class variances determine visual regularization. A cross-modal mapping and a supervised ridge classifier are computed from accumulated statistics. Evaluation covers six medical class-incremental settings.

Critical strengths

(1) The cumulative statistical formulation permits adaptation without retaining historical examples or repeatedly updating the encoder. This is a useful practical setting. (2) The paper reports gains across six datasets and provides analyses of feature expansion and statistical updates. These results make the approach worth investigating further.

Critical weaknesses

(1) The scientific contribution is insufficiently distinguished from established supervised multiview and statistical classifiers. Label-informed cross-view discrimination predates this work (Sharma et al., 2012), as does continual statistical discrimination on deep features (Hayes & Kanan, 2020). Appendix C varies labels while holding the image-text distribution fixed, whereas the actual text prompts and uncertainty estimates already depend on labels. The connection between this construction and the claimed information gap is not established.

(2) The experiments do not establish the necessity of the proposed components. RanPAC (McDonnell et al., 2023) and ACIL (Zhuang et al., 2022) are discussed but absent from Table 1, and a pure visual ridge classifier with identical expanded features is missing. Table 3 only adds components cumulatively. Replacing identity regularization with U also changes its overall scale. Finally, Appendix G.1 lists dataset abbreviations and task counts without the sources, sample counts, splits, or class orders needed to reproduce the results.

References:

Hayes, T. L., & Kanan, C. (2020). Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 220–221). IEEE. https://openaccess.thecvf.com/content_CVPRW_2020/html/w15/Hayes_Lifelong_Machine_Learning_With_Deep_Streaming_Linear_Discriminant_Analysis_CVPRW_2020_paper.html

McDonnell, M. D., Gong, D., Parvaneh, A., Abbasnejad, E., & van den Hengel, A. (2023). RanPAC: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36, 12022–12053. https://doi.org/10.52202/075280-0526

Sharma, A., Kumar, A., Daumé, H., III, & Jacobs, D. W. (2012). Generalized multiview analysis: A discriminative latent space. In 2012 IEEE Conference on Computer Vision and Pattern Recognition (pp. 2160–2167). IEEE. https://doi.org/10.1109/CVPR.2012.6247923

Zhuang, H., Weng, Z., Wei, H., Xie, R., Toh, K.-A., & Lin, Z. (2022). ACIL: Analytic class-incremental learning with absolute memorization and privacy protection. Advances in Neural Information Processing Systems, 35, 11602–11614. https://doi.org/10.52202/068431-0843

Questions for authors

(1) Please specify each dataset's source, class set, sample counts, splits, and class arrival order, and clarify which settings are shared by the baselines. (2) What are the results for pure visual ridge, RanPAC, and ACIL using the same frozen features, random expansion, and model-selection protocol? (3) Does the benefit of U remain against isotropic regularization with matched scale and separately tuned strength? (4) How does Appendix C apply when text features are constructed from class names and the alignment stage already uses class-dependent statistics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.