acceptodds
Withdrawn from ICLR 2027

ClinKD: Cross-Modal Distillation for Fine-Grained Medical VQA

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained medical visual question answering requires a model to identify findings, ground them spatially, and generate clinically relevant text. We investigate two potential weaknesses in adapting a general-purpose multimodal model to this setting: contiguous position indices do not distinguish image-text boundaries, and uniform distillation can overemphasize uncertain teacher targets. ClinKD addresses these issues with Modality-aware Cross-modal Gap Rotary Position Embedding (MCG-RoPE) and confidence-margin curriculum distillation. MCG-RoPE inserts explicit boundary gaps while preserving the native spatial grid, whereas the distillation objective weights teacher-forced token distributions by confidence and top-two probability margin and gradually relaxes its admission threshold. On Med-GRIT-Test30k, the full pipeline obtains 67.51, 82.35, 70.56, and 65.69 on visual grounding, referring object classification, referring captioning, and medical image analysis, respectively, averaging 71.53. This is 14.87 points above the reported BiRD baseline, which uses a different backbone and is therefore not architecture-matched; on LLaVA-Med-ga0.2k, mBMR increases from 21.06 to 23.54. Single-component experiments show gains from MCG-RoPE and distillation separately, although the archived results do not isolate every interaction in the full pipeline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical VQA, Multimodal Learning, Cross-Modal Distillation, Knowledge Distillation, Rotary Position Embedding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wmGkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes ClinKD for fine-grained medical visual question answering. It introduces Modality-aware Cross-modal Gap Rotary Position Embedding (MCG-RoPE) to distinguish image-text boundaries and confidence-margin curriculum distillation to reduce unreliable teacher supervision. Experiments on Med-GRIT and external medical VQA datasets demonstrate improvements over existing baselines.

Critical strengths
  1. Interesting combination of modality-aware encoding and distillation. The paper combines a lightweight modification to positional encoding with confidence-guided knowledge distillation. Both components address relevant challenges in fine-grained medical image understanding.

  2. Encouraging multi-task experimental results. ClinKD achieves substantial improvements across four Med-GRIT tasks and eight medical imaging modalities. Additional evaluations on external medical VQA datasets suggest potential generalization benefits.

Critical weaknesses
  1. Lack of fair and architecture-matched comparisons. The main performance improvements are reported against BiRD, which uses a different backbone. Although a Qwen2-VL SFT baseline is provided, the comparisons do not adequately control for backbone and training differences.

  2. Insufficient ablation and reproducibility. The experiments do not fully isolate the contributions of MCG-RoPE, curriculum distillation, reflective correction, and semantic reranking. Furthermore, key implementation details, including the teacher checkpoint and several training configurations, are missing.

Questions for authors
  1. Can the authors provide architecture-matched comparisons using identical backbones, training data, and computational budgets?

  2. Can the authors provide a complete ablation separating MCG-RoPE, curriculum distillation, reflective correction, and semantic reranking?

  3. How sensitive is MCG-RoPE to the boundary gap size, given that only (g=2) is evaluated?

  4. Can the authors provide the teacher checkpoint and evaluate confidence weighting, margin weighting, and curriculum scheduling separately against standard knowledge distillation?

  5. Can you enlarge the font size of the text in your figures?

Reviewer_YdUzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ClinKD, a cross-modal distillation framework for fine-grained medical visual question answering (Med-VQA), aiming to address limitations in multimodal positional encoding and uncertainty in teacher supervision. The authors evaluate ClinKD on four Med-GRIT tasks across eight medical imaging modalities, including visual grounding, referring object classification, referring captioning, and medical image analysis.

Critical strengths
  1. The paper addresses two meaningful challenges in fine-grained medical VQA: modeling image–text modality boundaries and handling uncertainty in teacher supervision.
  2. The reported results show numerical improvements over the BiRD baseline across four Med-GRIT tasks and several medical imaging modalities.
Critical weaknesses
  1. This paper remains unclear whether the original positional encoding causes measurable grounding errors and whether teacher confidence and top-two probability margins reliably indicate supervision quality. Furthermore, the main empirical comparison relies heavily on BiRD (2024), with limited evaluation against recent representative methods. Related studies such as Circle-RoPE (ICML 2026) and Beta-KD (CVPR 2026) are also not sufficiently discussed, making the incremental novelty and competitiveness of the proposed approach difficult to assess.
  2. While Table 3 demonstrates individual improvements from MCG-RoPE and Pseudo-KD, it lacks a joint ablation of these two components without auxiliary stages, and reflective text correction is not independently evaluated.
  3. More comprehensive ablations under matched backbone and training settings are needed to better substantiate the effectiveness of the proposed framework.
Questions for authors
  1. Figure 3 suggests that MCG-RoPE produces more concentrated attention in clinically relevant regions. Were the attention maps generated using identical backbones, training settings, attention extraction procedures, and normalization methods? Is there quantitative evidence that the observed attention differences are associated with improved grounding accuracy? Additionally, how do the authors verify that teacher confidence and top-two probability margins reliably reflect the correctness of medical supervision?
  2. Could the authors clarify the contributions of reflective text correction and SASG, and report the performance of MCG-RoPE + Pseudo-KD without these auxiliary components?
  3. The authors acknowledge in Sections 4.1 and 4.8 that the main comparison with BiRD is not architecture-matched. Although Table 3 includes a Qwen2-VL SFT baseline, it does not provide a complete set of controlled comparisons on the same backbone. Could the authors clarify how much of the reported improvement can be attributed to the proposed components? In particular, could they provide results for MCG-RoPE and Pseudo-KD using the same Qwen2-VL initialization, training data, and computational budget as the SFT baseline?
Reviewer_7nn4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ClinKD addresses fine-grained medical VQA through modality-boundary gaps in rotary position indices and confidence-margin curriculum distillation. The full pipeline additionally includes reference-guided text correction and inference-time semantic reranking. Results show numerical improvements on Med-GRIT and an external medical VQA dataset.

Critical strengths
  1. The position-index modification is simple and introduces no additional trainable parameters. Filtering uncertain teacher supervision is also reasonably motivated.

  2. The reported improvements cover multiple tasks and imaging modalities. The paper clearly acknowledges several limitations of its evaluation.

Critical weaknesses
  1. The main comparison is confounded by different backbones. The component experiments are not architecture-matched Qwen2-VL retrainings, and the full pipeline combines additional training and inference stages. Consequently, the contribution of the proposed components remains unclear.

  2. The teacher checkpoint is unspecified, which is a fundamental reproducibility gap for a distillation method. Other missing settings include image resolution, effective batch size, and reranking configuration.

  3. The methodological contribution appears modest and insufficiently validated. Only one boundary gap is evaluated, and the distillation experiments do not isolate confidence weighting, probability margin, and curriculum scheduling. Repeated-run uncertainty is also unavailable.

Questions for authors
  1. Are candidate-validation cases separate from the experiences used to propose edits? What metric determines acceptance, and what observation cutoff is used for mortality prediction?

  2. Could the authors report paired confidence intervals for the main improvements and clarify the two-indicator input setting used in the cross-dataset transfer experiment?

  3. The related-work discussion would benefit from citing and discussing EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis. Its EHR-specific reasoning enhancement provides relevant context for distinguishing parameter-based reasoning improvements from OntoSkill’s non-parametric skill evolution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.