acceptodds
Withdrawn from ICLR 2027

MedSkill: Internalizing Clinically Grounded Visual Skills for Medical Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical multimodal large language models (MLLMs) have shown strong performance in medical visual question answering, yet their visual competence is still largely acquired from image–text correlations and task-specific supervision. In clinical practice, however, image interpretation relies on reusable visual procedures for locating anatomy, recognizing findings, assessing spatial relations, and characterizing visual appearance and morphology. These procedures are explicitly reflected in standardized reporting systems and assessment guidelines, but remain largely implicit in current medical MLLMs.We introduce MedSkill, a framework for internalizing clinically grounded visual skills into medical MLLMs. MedSkill derives reusable visual skills from authoritative clinical standards and grounds them in diverse medical images. Through structured supervision, these skills are internalized as persistent visual capabilities, enabling the model to apply clinical visual expertise directly during reasoning.Experiments on multiple medical VQA benchmarks show that MedSkill improves overall question-answering performance and strengthens the corresponding visual skills measured through targeted capability evaluation. Further analyses demonstrate that effective transfer depends on how skills are represented, favoring concise, operation-oriented supervision over generic image descriptions or explicit reasoning traces. These results support visual skill internalization as a complementary direction for medical multimodal learning, in which clinically structured visual expertise is transformed from external knowledge into reusable model capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Medical Multimodal Learning, Multimodal Large Language Models, Medical Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_owcEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MedSkill derives five visual skill families from clinical standards, uses Qwen3.8-Max to convert medical captions into 18,888 skill QA instances, and trains Qwen3-VL-8B through a five-stage pipeline, reporting gains on VQA-RAD, SLAKE, and PathVQA along with higher judge-scored skill ratings.

Critical strengths
  1. The representation study offers practical guidance: concise, operation-focused skill QA transfers to VQA more effectively than raw captions, which degrade performance relative to direct VQA fine-tuning.

  2. The method introduces no additional inference cost: skills are learned during training, so the model answers directly from image and question without retrieval, tools, or prompts.

Critical weaknesses
  1. In the Table 2 experiment, the training data, the judge model, and the reference answers all come from Qwen3.8-Max, which risks data overlap and evaluation bias. The reliability of the judge is never assessed.

  2. In Table 1, no original Qwen3-VL baseline is included, LLaVA-Med lacks equivalent fine-tuning, and several baselines report incomplete results, rendering the comparison unfair.

  3. The five-stage training pipeline lacks stage-wise ablations, and no single-stage joint-training baseline is provided to show that this complexity is necessary.

Questions for authors
  1. In the Table 2 experiment, the training data, the judge model, and the reference answers of the evaluation set all originate from Qwen3.8-Max, which may introduce data overlap and evaluation bias. Can the results reliably reflect actual performance? I recommend that the authors conduct evaluation using alternative models. Additionally, how is the reliability of Qwen3.8-Max’s scoring assessed?

  2. The comparison in Table 1 is unfair. The authors do not include results from the original Qwen3-VL, which would help clarify the actual contribution of MedSkill. Moreover, the authors further fine-tune on the training data of the three benchmarks (VQA-RAD, SLAKE, and PathVQA), whereas the compared model LLaVA-Med is evaluated without equivalent fine-tuning. In addition, results for several methods in Table 1 are incomplete. These issues undermine the fairness of the comparison between MedSkill and existing methods.

  3. The authors introduce a complex five-stage training pipeline. Has the contribution of each stage been sufficiently validated? If the external data, the target VQA data, and the skill corpus data were combined for single-stage joint training, could comparable or superior results be achieved?

Reviewer_QEwiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MedSkill, a framework for internalizing clinically grounded visual skills into medical multimodal large language models. It derives five skill families from clinical reporting standards and constructs 18,888 skill-oriented supervision instances from three external medical image collections. The framework combines medical-domain adaptation with progressive training on visual skills and downstream medical VQA, while retaining direct image–question–answer inference. Experiments on VQA-RAD, SLAKE, and PathVQA show improvements in medical VQA performance and judge-assessed visual-skill execution. The paper also investigates alternative supervision representations, finding that concise, operation-oriented supervision transfers more effectively than raw captions.

Critical strengths

The paper addresses a practically relevant question: whether medical MLLMs can acquire reusable visual capabilities beyond learning image–answer associations. Deriving skill supervision from clinical guidelines and standardized reporting systems is a well-motivated approach.

MedSkill achieves the highest reported overall score among the compared methods across three medical VQA benchmarks, although it does not outperform all baselines on every individual metric.

The method incorporates visual-skill supervision during training while preserving direct image–question–answer inference, without requiring retrieval, external tools, or intermediate skill traces at inference time. This is a practical advantage because it avoids additional inference-time components.

Critical weaknesses

The quality of the generated skill supervision lacks independent validation. Qwen3.8-Max generates the focused questions, answers, and supporting annotation spans used to construct the skill corpus. Although the paper describes filtering criteria and reports rejection counts, it provides no independent human evaluation of the retained annotations. A small, stratified evaluation by medically qualified annotators would help establish whether the questions are visually answerable, the answers are correct and image-grounded, and the assigned skill categories are appropriate.

The visual-skill acquisition evaluation relies on the same model used to generate the supervision. Qwen3.8-Max both constructs the skill annotations and evaluates model responses using the resulting reference states. This introduces a risk of correlated annotation and evaluation errors and may favor responses consistent with the teacher’s interpretation. Anonymizing responses is useful but does not address this dependence. Evaluation with an independent LLM judge, together with expert validation on a subset, would strengthen confidence in the reported improvements.

The ROI proposal methods in Figure 2(c) are insufficiently specified. Image-KG ROI is mentioned in the accompanying text without an adequate definition or explanation, while Question-KG ROI and Direct ROI appear in the figure without being introduced in the text. The figure also uses different numbers of candidate regions: k = 3 for Image-KG ROI and Question-KG ROI, and k = 4 for Direct ROI. These differences are not explained and could affect the comparison. The authors should describe how each method generates regions, define how ROI utility is measured, and justify the different k values or provide a comparison using a shared candidate budget.

Some evaluation metrics are not directly comparable, and differences between reported results require clarification. Table 1 reports open-ended recall for generative methods but accuracy for discriminative methods, so their aggregated overall scores should not be interpreted as directly equivalent. In addition, the “Direct VQA” result in Table 4 differs from the QA-only configurations in Table 3 without a clear explanation of the differences in training data, initialization, or training schedule. Clarifying these experimental settings would make the comparisons easier to interpret.

Questions for authors

Could the authors clarify the training configuration and aggregation procedure underlying “Direct VQA” in Table 4 (89.03 closed-ended / 59.61 open-ended)? How does this configuration differ from the QA-only configurations in Table 3 and the direct-answering backbone evaluated in Figure 2(a)? Please explain whether the differences arise from training data, initialization, optimization budget, evaluation metrics, or dataset weighting.

Could the authors define Image-KG ROI, Direct ROI, and Question-KG ROI and explain how each generates candidate regions?

Reviewer_LXDjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MedSkill, a framework for teaching medical multimodal models clinically motivated visual skills through training-time supervision. It organizes visual operations into five families: anatomical localization, finding recognition, spatial relations, appearance characterization, and morphology description. Using a teacher model and source annotations, the authors construct 18,888 skill instances from PathCap, ROCOv2, and MultiCaRe. Training combines medical domain adaptation, skill supervision, and downstream VQA alignment, while retaining direct image–question–answer inference. Evaluation covers VQA-RAD, SLAKE, and PathVQA, with results on Qwen3-VL and LLaVA-Med. The paper reports an aggregate score of 78.24 versus 77.53 for the strongest listed baseline, improvements in model-judged skill execution, and ablations suggesting that concise, operation-specific supervision transfers better than raw captions.

Critical strengths
  1. A practical, structured approach to constructing visual supervision. Decomposing observations into a target, a visual operation, and an observable state provides a concrete way to turn heterogeneous medical captions into focused learning examples. Retaining supporting annotation spans and applying explicit filtering criteria improves the auditability of the resulting corpus. This is a useful contribution because it offers a training-based route to incorporating clinical visual knowledge while preserving direct inference without retrieval or external tools.

  2. Promising evidence that supervision granularity matters. Table 4 reports an aggregate score of 78.24 for coarse-plus-atomic supervision, compared with 74.80 for coarse skills and 72.90 for raw captions. Together with competitive results across radiology and pathology benchmarks, these findings suggest that the organization of external supervision deserves attention beyond simply increasing its volume. Although stronger experimental controls are needed, this representation-level finding is consequential and potentially useful to subsequent work.

Critical weaknesses
  1. The skill evaluation does not establish generalization to unseen examples. Appendix B.10 states that the 2,000 candidate evaluation questions are sampled from the external skill corpus and reuse existing questions and reference states without modification. Sections 4.2–4.3 describe training on that corpus, but no held-out partition or exclusion procedure is specified. Consequently, Table 2 may measure performance on previously trained image–question–state combinations rather than transferable visual competence. This is an unresolved overlap concern, not proof of leakage, but it directly affects the paper’s central conclusion. Moreover, Qwen3.8-Max both generates the supervision and judges the resulting capabilities, without reported independent calibration. Anonymizing model responses does not resolve that dependence.

  2. The contribution of the proposed skill-learning procedure is insufficiently isolated and specified. Section 4.3 introduces adaptive sampling, skill weighting, optimizer restarts, answer-concept alignment, and VQA-only consolidation, but omits the rules and budgets needed to reproduce these stages. It is therefore unclear whether comparisons match initialization, data exposure, and optimization effort. For example, Table 4’s Direct VQA baseline scores 74.32, whereas Table 3’s Answer SFT baseline scores 70.82, without explaining their relationship. The uniform-sampling control mentioned in Section 5.1 is also not reported. These omissions prevent a clear attribution of the gains to structured skills and adaptive training rather than additional supervision or training choices.

  3. Figure 4 also raises a concrete concern about visual grounding. In the inference panel, the highlighted ROI in the full CT does not correspond to the enlarged inset, which appears to depict a different region of the image. Nevertheless, the accompanying response confidently identifies a HCC and describes its visual characteristics. This inconsistent correspondence undermines the example’s credibility, particularly given the paper’s emphasis on clinically grounded visual skills. The manuscript should clarify whether this is an actual model prediction or a schematic illustration and establish the provenance and clinical correctness of the example.

Questions for authors
  1. Were the skill-evaluation questions and their corresponding images excluded from every training stage, including initial domain adaptation? Please specify the partitions and overlap counts. If they were not excluded, what evidence supports skill generalization on a disjoint evaluation set?

  2. Are transfer utility, sampling probabilities, and skill weights computed and updated? Please explain the initialization, training budgets, and model-selection criteria for Tables 3–4, including the difference between the 70.82 Answer SFT and 74.32 Direct VQA baselines. What is the result of the stated uniform-sampling control under matched conditions?

  3. Do the skill improvements persist under an independent radiologist or a blinded clinician assessment of a representative subset? This would help distinguish improved visual correctness from agreement with the model that generated the supervision.

  4. Where are the numerical results supporting the claim that explicit state–answer traces degrade performance? Appendix B specifies these formats but does not report their outcomes or training conditions. These results matter to the paper’s conclusion about which representations transfer effectively.

  5. Is the inference example in Figure 4 an actual MedSkill output or a manually constructed illustration? Please explain the mismatch between the highlighted ROI and the enlarged inset, provide the source image and reference diagnosis, and clarify how the accompanying clinical interpretation was verified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.