acceptodds
Withdrawn from ICLR 2027

DGTune: Discriminative Geometry Tuning for Noise-Affected Medical Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pre-training on large-scale datasets followed by fine-tuning on downstream tasks has become a popular paradigm for medical vision–language models. However, medical descriptions and answers provide imperfect upstream supervision, raising a fundamental question: does clean downstream learning overcome the noise inherited from pre-training? We investigate this problem through a controlled family of medical vision–language models trained with different levels of supervision noise. Our experiments reveal persistent downstream performance deficits, including under distribution shift, despite clean adaptation labels. Through Fisher analysis, we link these persistent deficits to weakened class-discriminative structure in the adapted representations. Crucially, strengthening discrimination on observed samples alone does not ensure that the same discriminative relations generalize to new samples. Motivated by this distinction, we propose Discriminative Geometry Tuning (DGTune), a lightweight adaptation method that jointly strengthens weak Fisher directions and promotes their cross-sample generalization. Its two complementary regularizers enhance weak class separation and preserve the corresponding directions and class-contrast patterns across balanced, disjoint sample groups. Experiments on controlled noisy and public medical models show improved in-distribution and out-of-distribution generalization with DGTune.

Primary area:
foundation or frontier models, including LLMs
Keywords:
catastrophic inheritance, vision-language model, noise model learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dTNWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how supervision noise in medical VLM pre-training affects downstream adaptation, even with clean labels. Through controlled experiments, the authors identify persistent performance degradation and weakened Fisher discriminative structure. To address this, they propose DGTune, which combines weak Fisher-spectrum strengthening with cross-sample Fisher transfer to improve class discrimination and generalization. Experiments across different noise levels, medical VLMs, and other pretrained backbones demonstrate consistent improvements on ID and OOD tasks.

Critical strengths
  1. The paper addresses a meaningful problem: the persistent impact of noisy pre-training supervision on downstream adaptation, particularly in medical VLMs.
  2. The proposed method is well motivated by the Fisher discriminative analysis.
  3. The experimental results are strong and extensive, showing consistent improvements across different noise levels, backbones, and ID/OOD settings.
Critical weaknesses
  1. NMTune has already investigated the impact of noisy pre-training on downstream tasks, while weak Fisher-spectrum strengthening is closely related to DeepLDA. The main novelty appears to lie in the cross-sample Fisher transfer component.

  2. DGTune and NMTune use different training budgets and adaptation architectures. While the authors follow the official NMTune configuration, a comparison under matched training budgets would help better isolate the contribution of the proposed regularizers.

  3. The main figure is not clear for me.

Questions for authors

The component ablation studies are currently limited to Kvasir-v2. I wonder whether the contributions of the two regularizers are similarly consistent across other ID datasets and OOD settings, particularly given that cross-sample generalization is a key motivation of the proposed method. Additional evidence on this would help strengthen the paper. I would be happy to consider raising my score if the authors can address my concerns.

Reviewer_JApdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether supervision noise during medical vision and language pretraining continues to affect downstream performance after adaptation with clean labels. The authors construct six 7B medical models with different levels of image and text mismatch noise. They report that high upstream noise can reduce both in-distribution and out-of-distribution performance. Through Fisher analysis, they associate this degradation with weak class discriminative directions in the adapted representations.

Based on this observation, the paper proposes Discriminative Geometry Tuning, or DGTune. The method freezes the pretrained backbone and trains an MLP classifier with two regularizers. The first strengthens weak Fisher directions, while the second encourages the same directions and class relationships to remain discriminative across disjoint sample groups. Experiments on medical models, general visual encoders, and biomedical language models show consistent improvements over MLP tuning and NMTune.

However, the central motivation is not fully convincing. Since the backbone is frozen, it is expected that information damaged during pretraining may not be recovered by a small downstream MLP. The experiments therefore show persistence under frozen feature adaptation rather than demonstrating that clean downstream learning generally cannot overcome upstream noise. Moreover, the paper does not establish that weak Fisher geometry causes the performance degradation. The strong gains of DGTune even at zero injected noise further suggest that it may be a general discriminative regularizer rather than a method specifically addressing inherited noise.

Critical strengths
  1. Controlled Study of Upstream Noise

The paper provides a carefully controlled study using six 7B medical models trained with different noise levels. This design separates upstream supervision noise from downstream label quality and allows the authors to examine its effect under a fixed adaptation protocol. The evaluation includes multiple medical datasets and both in distribution and out of distribution settings. This is valuable because most noisy label research focuses on corrupted downstream labels rather than noise already encoded in a pretrained representation.

  1. Effective Discriminative Regularization

DGTune addresses an important limitation of spectrum based regularization. Increasing Fisher eigenvalues alone does not ensure that different sample groups use the same discriminative directions or preserve the same class relationships. The cross sample objective provides a reasonable way to encourage such consistency. The two components are supported by ablation results, and the method performs well across medical, general visual, and biomedical language models. It also adds no geometric computation during inference.

Critical weaknesses
  1. The Proposed Mechanism Is Not Fully Established

The paper does not convincingly demonstrate that weak Fisher geometry is the cause of the observed performance degradation. It separately shows that upstream noise can reduce accuracy, that Fisher strength can become weaker, and that DGTune improves performance. However, it does not establish a systematic relationship between Fisher strength and accuracy across models, tasks, and noise levels.

Moreover, the downstream experiments freeze the backbone and train only a small MLP. Under this restriction, it is expected that information lost during pretraining may not be recoverable. The results therefore demonstrate persistence under frozen feature adaptation, but they do not show that clean downstream learning more generally cannot overcome upstream noise. The strong performance gains at zero injected noise also suggest that DGTune may be a general adaptation method rather than a method specifically designed to correct inherited noise.

  1. The Baseline Comparison is Limited

DGTune combines balanced sampling, discriminative regularization, and consistency across sample groups. Similar benefits might be obtained from supervised contrastive learning, center loss, DeepLDA, or simpler class centroid consistency objectives. Comparisons with these methods are needed to show that the Fisher based formulation is necessary, especially since DGTune requires substantially more training time than standard MLP tuning.

Questions for authors

See weaknesses.

Reviewer_gVBNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether noisy medical vision–language supervision leaves a measurable deficit after downstream adaptation with clean labels. Starting from Qwen2.5-VL-7B, the authors construct six medical models with different rates of text-target substitution and analyze the weakest Fisher discriminants in their adapted representations. They then propose DGTune, combining weak-spectrum strengthening with cross-sample Fisher transfer in a trainable MLP head. The reported improvements are substantial on several medical tasks. My main reservation is that the experiments do not yet separate a noise-specific explanation from the benefits of stronger supervised representation learning.

Critical strengths
  1. The paper studies a worthwhile transfer-learning question. Errors inherited from upstream image–text training are distinct from noisy labels in the downstream dataset, and it is useful to test whether clean adaptation can undo their effects.

  2. I like the distinction between making discriminants stronger on the observed samples and asking whether the same directions remain useful on disjoint samples. The cross-sample component is more specific than simply increasing a spectral separation score.

  3. The authors invest in a controlled series of upstream models rather than inferring noise effects only from unrelated public checkpoints. The evaluation spans several noise rates, medical classification settings, additional backbones, and component ablations, giving the paper a solid empirical starting point.

Critical weaknesses
  1. The synthetic noise does not cover the most relevant clinical errors. Section 2.1 corrupts supervision by replacing an image/prompt pair's text target with a target from another example. That is a clean test of correspondence mismatch, but realistic clinical noise may preserve the diagnosis while misstating severity, omitting findings, or giving a plausible wrong differential. The same noise rate is also used in both medical training stages. Please separate alignment-stage from instruction-stage corruption and include at least one semantically constrained noise condition.

  2. The Fisher analysis is suggestive rather than explanatory. Figure 3 does not show a simple monotonic deterioration in weakest Fisher strength: moderate corruption improves that measure for some task and transfer settings. The paper therefore has not shown that a weaker Fisher direction explains the corresponding loss in accuracy. A within-task analysis relating held-out accuracy to Fisher strength and cross-sample consistency across noise levels and seeds would be more informative than the present relative-change plot.

  3. The method's benefits are not confined to noisy upstream models. On Kvasir-v2, Table 1 reports 89.62% accuracy for DGTune versus 78.75% for the MLP even at 0% injected noise. This is a large gain in the nominal clean-reference condition, so DGTune may be a generally useful discriminative regularizer rather than a targeted remedy for inherited noise. Please quantify noise robustness separately from absolute accuracy gain, for example by comparing each method's degradation from its own 0% reference and against standard metric-learning objectives.

  4. The comparison with NMTune is not adequately controlled. Appendix B.1 uses NMTune's official settings, including a different residual head and 30 training epochs, whereas the MLP and DGTune runs use 50 or 100 epochs. Reporting the official baseline is legitimate, but the performance difference also reflects head capacity and training budget. A common-head comparison or at least an equal-epoch/equal-compute run is necessary before attributing the full gap to the proposed loss.

  5. The central two-component result needs replication beyond Kvasir-v2. Appendix Table 6 shows an unusually large benefit from combining the two regularizers: at 30% noise, the full method reaches 88.12%, compared with 77.45% for spectrum-only and 76.62% for cross-sample-only. This synergy is arguably the main algorithmic finding, yet the component study is restricted to one dataset. Please repeat it on a retinal dataset and a cross-center setting, and compare against a simple supervised contrastive or discriminant-analysis regularizer under the same features.

  6. The Fisher estimates may become unstable for rare classes. The method relies on balanced, disjoint sample groups and a shrinkage-regularized within-class covariance estimate. In small or imbalanced medical datasets, the available independent samples per class may be limited, while near-equal weak eigenvalues can make individual directions unstable. The paper would benefit from group-size and shrinkage sensitivity tests, together with the actual sample counts per class and the handling of degenerate estimates.

Questions for authors
  1. What changes when noise is injected only during image–text alignment or only during instruction tuning? Does one stage account for most of the inherited deficit?

  2. Can the authors show that DGTune specifically reduces sensitivity to increasing upstream noise, rather than mainly improving performance at every noise level including the 0% reference?

  3. Does the large gain from combining the two regularizers persist on retinal and OOD tasks under matched training budgets and multiple seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.