acceptodds
Under review as a conference paper at ICLR 2027

CGCP: Clinically-Guided Contrastive Pretraining with Mixed-Type Clinical Signals for 3D Brain MRI

Abstract

Neuroimaging cohorts often pair brain MRI with structured clinical assessments, yet representation pretraining either ignores these signals or incorporates them without establishing whether they provide information beyond the downstream diagnostic target. We introduce Clinically-Guided Contrastive Pretraining (CGCP), a representation-learning framework for 3D brain MRI that integrates heterogeneous clinical supervision through continuous cognitive-severity learning, discrete stage-aware contrastive learning with a label-tracking momentum memory bank, and variance regularisation. A Bayesian optimisation procedure balances these objectives while retaining a minimum contribution from clinically meaningful staging supervision. Beyond proposing a pretraining strategy, we use CGCP as an analytical probe to determine when mixed-type clinical supervision provides genuinely complementary information. We evaluate longitudinal-to-cross-sectional transfer against label-free self-supervised, supervised, and label-aware alternatives under subject-level evaluation, complemented by targeted loss ablations, frozen and nonlinear probing, layer-wise analysis, calibration, data-efficiency experiments, and sensitivity studies. The resulting evidence shows that clinically guided representations can support strong downstream classification, but their benefit is not uniform: correlated clinical instruments can act as label proxies, measurement ceilings can restrict continuous supervision, representation rankings can change substantially after fine-tuning, and memory-constrained optimisation can complicate multi-objective balancing. We consolidate these observations into a validity framework for evaluating clinically guided representation learning before improvements are attributed to additional clinical supervision.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Clinically-Guided Pretraining, Contrastive Representation Learning, 3D Brain MRI, Neuroimaging, Clinical Metadata, Cognitive Severity, Medical Representation Learning, Bayesian Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PYNVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

CGCP pretrains a 3D ResNet-18 on OASIS-2 using three objectives: MMSE regression, CDR-supervised contrastive loss and variance regularization, with loss weights selected through Bayesian optimization. Finally, CGCP is evaluated on OASIS-1 through finetuning and linear probing for AD/CN classification and MMSE regression.

Critical strengths

The paper presents CGCP both as a pretraining method and as an analytical framework for investigating clinical supervision.

Critical weaknesses
  1. The paper, in general, is not easy to follow, making it confusing and difficult to assess the main contribution in its current form. The paper alternates between presenting CGCP as a pretraining method and as an empirical investigation of clinical supervision. However, the supplied analysis does not sufficiently establish either a methodological advantage or general findings about when mixed clinical supervision helps. For example:
    • “MMSE alone predicts CDR-based diagnosis at AUC= 0.875 (5-fold CV logistic regression), exceeding every imaging pipeline evaluated, including fine-tuned CGCP (0.780)... Removing Lmmse has no statistically distinguishable effect on any metric.” while the paper proposes Lmmse in the pretraining objective. Also, “The resulting evidence shows that clinically guided representations can support strong downstream classification, but their benefit is not uniform..”, “CGCP-BO is not statistically distinguishable from its strongest competitors. CGCP is thus a strong but not dominant classification method under rigorous evaluation“, so what exactly is the contribution here?
    • If we look at a strict baseline comparison with matching variables: y-aware (with MMSE and no CDR, table 3) has 76.1% AUC and 65.2% BA whereas the same ablated CGCP (with MMSE and no CDR, table 4) shows 75.9% AUC and 65.6% BA, likewise, SupCon (with CDR and no MMSE, table 3) has 75% AUC whereas the ablated CGCP (with CDR and no MMSE, table 4) shows 74.5% AUC. These are currently in different tables, and show no clear advantage for CGCP’s single-signal variants. What if both the signals were supplied to these prior works? Would CGCP still outperform? This is not clear. Furthermore, Table 6 shows that SimCLR outperforms CGCP under frozen linear probing. The authors acknowledge it, but this limits any broad interpretation that CGCP learns better representations.
  2. The experimental setting is too limited for the broader conclusions. Evaluation on only one dataset,i.e., OASIS, two tasks, and tested with a single backbone. Currently, I am unable to evaluate the framework’s generalizability, given that this domain comes with inherent challenges of cohort shifts and confounders. “possible participant overlap between the anonymized OASIS releases”, if this could be a limitation, then isn’t it better to opt for a different dataset for evaluation? Also, there is no sufficiently strong technical or empirical contribution. The method combines existing components: y-Aware-style proxy supervision, a VICReg variance term, BO weight tuning, without new mechanism.
  3. Missing Interpretability analysis: In the clinical setting, evaluations are reliable with both performance and interpretability analysis (not just UMAPs). The study does not adequately assess whether the learned information truly reflects cognitive impairment rather than demographic or acquisition-related correlates or other factors.
  4. The paper lacks comparisons against several prior works, such as supervised models(Jang et al's M3T [CVPR 2022], Jiang et al's AAGN [MICCAI 2024]) and recent pre-training based frameworks (Rui et al's BrainMVP [CVPR 2025], Lin et al's ADLIP [Alzheimer’s Dement. 2025])
Questions for authors

Please address the points raised in weakness.

Other questions:

  1. MMSE and CDR scores are noisy in practice as they depend on the subject, clinical history, examiner, and many other factors. So the assumption that “these annotations encode exactly the pathological structure” is not entirely true. Was there any analysis performed for this? Say for samples with scores not consistent with the label? If all were correlated, then how is it different from directly using the diagnostic labels during pretraining?
  2. Fig 1 is confusing. What does passing info from one loss to another mean?
  3. What to infer from the UMAPs in fig 2? The inference is sec 4.6 is difficult to follow. It just states, w/o mmse there are distinct clusters, but is that an expected behaviour? Clustered by what? Or what is an expected behaviour? etc.. these are unclear.
  4. What does linear probing on supervised-only in table 6 mean? The supervised-only run is for the task-specific training right? What was probed?
Reviewer_TcANOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors address the problem of leveraging structured clinical assessments (e.g., MMSE and CDR scores) during the pre-training of self-supervised models for neuroimaging, and investigate their effect on the resulting latent space. To this end, they propose CGCP, a clinically guided pre-training objective that combines continuous and discrete clinical labels with variance-based regularisation. The method builds on existing contrastive learning approaches, notably MoCo, and draws on the idea that discrete labels can organise examples into semantically meaningful clusters in the latent space, while continuous labels can preserve within-cluster ordering. This idea is implemented through a three-term loss function, which constitutes the main contribution of the work: An MMSE regression term to account for the continuous clinical signal. A MoCo-like InfoNCE term to model the discrete CDR signal. A variance-based regularisation term, inspired by VICReg, to discourage representation collapse. The weights of these terms are selected through Bayesian optimisation. The authors also investigate two variants: one that replaces Bayesian optimisation with GradNorm for loss balancing, and another that replaces the MMSE regression term with a Rank-N-Contrast objective. The authors pre-train a 3D ResNet-18 on OASIS-2 and evaluate the resulting representations on AD vs. CN classification and MMSE regression tasks on OASIS-1, using both fine-tuning and linear probing. They compare CGCP against label-free and label-aware pre-training objectives, and perform several ablation and sensitivity analyses. The results do not establish a consistent performance advantage for CGCP over the evaluated alternatives. The authors investigate four potential confounds that may explain the limited gains: label-proxy redundancy, instrument ceiling effects, evaluation-protocol sensitivity, and small-batch architectural constraints. These analyses provide insights into the challenges of clinically guided pre-training, although the contribution of the proposed objective remains unclear.

Critical strengths
  • This work addresses an important problem in AI for neuroimaging, where limited sample sizes make it valuable to exploit complementary sources of information, such as clinical assessments.
  • The idea of combining continuous and discrete clinical labels to guide self-supervised pre-training is interesting and potentially useful.
  • The paper investigates several potential reasons why clinical supervision may fail to improve downstream performance, including label-proxy redundancy, instrument ceiling effects, and evaluation-protocol sensitivity. These analyses could provide useful insights for future work on clinically guided pre-training.
  • The experimental protocol is carefully designed to compare different pre-training objectives under the same backbone and downstream evaluation settings. The authors also perform ablations and statistical tests to investigate the contribution of the proposed components.
  • The paper evaluates the learned representations using both fine-tuning and linear probing, as well as UMAP visualisations and quantitative analyses of the latent space.
  • The authors use subject-level cross-validation and discuss the limitations imposed by the small sample size. The supplementary analyses also provide a more detailed investigation of the results and their sensitivity to different evaluation choices.
  • The paper provides pseudocode, detailed preprocessing and data augmentation procedures, and uses open-access datasets, which facilitate reproducibility.
Critical weaknesses
  • The comparison with existing methods is incomplete. The authors compare CGCP with several label-free and label-aware objectives, but do not compare it with recent neuroimaging foundation models, such as NeuroJEPA or NeuroVFM. It would be interesting to see how the proposed method compares with these approaches, although I realise that a fair comparison may not be straightforward.

  • The motivation for using clinical assessments could be better explained. Why should combining MMSE and CDR scores improve the representations learned from MRI, and under which conditions should this be beneficial? The paper discusses the possible redundancy between these signals, but I would have liked a clearer motivation for the proposed approach and a better explanation of what it is expected to bring beyond existing label-aware objectives.

  • The loss formulation raises some questions. In particular, the MMSE regression loss includes a Pearson correlation term, but the training batch size is only two. With two samples, the correlation is essentially either -1 or +1 when the predictions differ, or zero when they coincide. I am not convinced that this provides a meaningful signal for learning continuous relationships. The authors should explain why this term is appropriate in this setting and whether a more conventional regression loss would work as well. I also find the justification for the variance regularisation term insufficient. It is inspired by VICReg, but does not use the full VICReg objective, and its behaviour with such a small batch size deserves more discussion. Finally, the loss weights are selected using Bayesian optimisation based on frozen embeddings after only five epochs, whereas the relative performance of the methods changes between linear probing and fine-tuning. This makes me question whether the optimisation proxy is appropriate for the final evaluation.

  • The results do not show a clear advantage over simpler objectives. CGCP achieves the best balanced accuracy and F1 scores, but the statistical tests do not establish significant differences between methods across the six downstream metrics. Moreover, removing the MMSE regression objective does not lead to a statistically distinguishable change in performance. The experiments therefore do not demonstrate that the proposed combination of losses provides a reliable improvement. The limited number of folds also makes it difficult to draw strong conclusions from the statistical tests.

  • It is unclear whether MMSE and CDR provide complementary information. The paper shows that MMSE alone predicts the CDR-based diagnostic label with an AUC of 0.875, which is higher than the performance of the evaluated imaging models. MMSE and CDR are also strongly correlated. Given that removing the MMSE objective does not significantly change downstream performance, I am not convinced that the proposed method effectively exploits complementary information from the two clinical signals. The paper investigates this redundancy, but does not show that combining the signals brings a clear benefit.

  • The evaluation is limited in terms of generalisation. The method is pre-trained on OASIS-2 and evaluated on OASIS-1. Although using separate datasets is a positive aspect, both come from the same research programme, and participant independence cannot be guaranteed because the anonymised identifiers do not reliably establish identity across the two releases. This does not mean that participant overlap occurred, but it makes the independence of the evaluation less certain. Testing on an independent cohort, such as ADNI, would make the results more convincing. The evaluation also relies on a single backbone, a single seed, and five-fold cross-validation, so it remains unclear how stable the results are across different experimental settings.

  • The presentation could be improved. I found parts of the introduction and discussion too concise. The motivation for using clinical assessments during pre-training needs to be clearer, and the discussion could do more to interpret the results. The analysis of the four potential confounds is interesting, but some of the explanations remain hypotheses rather than conclusions directly supported by the experiments. I would have liked a clearer distinction between the two.

Questions for authors

Overall, the paper addresses a relevant problem with a novel idea. The authors propose a robust experimental setup, and the analysis of potential confounds provides useful insights into the limitations of clinically guided pre-training.

However, the paper does not establish a clear advantage over existing methods, and the contribution of the individual loss components remains unclear. In particular, the use of a Pearson-correlation term with a batch size of two and the selection of loss weights based on frozen embeddings raise questions about the proposed methodology. The limited evaluation on small cohorts also restricts the conclusions that can be drawn about generalisation, and lacks comparison with recent, state-of-the-art foundation models for neuroimaging.

Overall, while the paper addresses a relevant problem and presents some interesting analyses, I find the methodological contribution insufficiently substantiated for a broad machine learning audience in its current form.

Reviewer_fhWrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce a pretraining framework for clinical brain MRI. They make the observation that longitudinal neuroimaging cohorts tend to record metadata, and correctly note that this valuable source of information is commonly discarded for pretraining. They therefore combine regression on a cognitive score (MMSE) and a dementia class-based contrastive loss (MoCo momentum encoder) with a collapse-prevention term (VICReg variance). A 3D Resnet is trained on OASIS-2 and then evaluated on OASIS-1, which have small sample sizes for pretrianing. This means that downstream performance is evaluated on a single (small) dataset, using 2 tasks but which appear to measure a highly correlated construct (cognitive score / dementia). Gains over older baselines tend to be small and, due to small sample sizes, generally partially drowned by noise.

Critical strengths

None

Critical weaknesses
  1. The methodology does not appear convincing. While the rational to use existing metadata for pretraining is sound and perhaps deserves more attention in the field, the paper doesn't manage to translate this strategy into convincing gains. More specifically:
  • gains do not clearly exceed noise,
  • there is potential for leakage (potentially shared subjects due to the shared team between OASIS-1 and 2, this seems it would impact the partially-supervised method much more before it directly uses the labels),
  • only older baselines are included,
  • no primary comparison versus existing larger pretrained foundation models (if pretraining with metadata is to be useful, it should beat models which were pretrained without labels on much larger samples) Overall, more diverse datasets or larger samples sizes for evaluation would significantly strenghten to paper; currently, it is just not really possible to tell whether the additional complexity of the introduced methodology is worthwhile.
Questions for authors
  1. Why are there no comparisons to models trained on larger, unlabeled pretraining corpuses? (To be clear, no training would be required by the authors themselves.) Meta-data won't be available for a lot of the samples in datasets such as FOMO, so would the authors agree that such comparisons are important to understand the utility of pretraining with meta-data?

  2. Why do the authors think that the method does not win in terms of data efficiency? (tables 16/17) I would have expected an improvement especially here, due to the labels/metadata.

  3. I appreciate that the authors provide some information about how their method was tuned, but this appears somewhat incomplete: on what data were parameters such as bank size chosen, and based on what kind of evaluations? For the supervised baseline, how was its recipe tuned exactly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.