acceptodds
Under review as a conference paper at ICLR 2027

Context-Space Adaptation of Frozen Multimodal Large Language Models for Medical Image Diagnosis

Abstract

Multimodal large language models (MLLMs) offer a promising paradigm for medical image diagnosis, yet their performance can degrade substantially when applied to medical datasets with heterogeneous visual distributions and diagnostic tasks. Fine-tuning can improve domain adaptation, but requires repeated model updates for each target dataset. In this work, we investigate gradient-free adaptation of frozen MLLMs through the construction of dataset-specific context states. We propose Clinician Mimetic Workflow, a context-space adaptation framework that adapts frozen MLLMs by jointly constructing complementary visual and experiential context states. The framework comprises two coordinated context construction processes. Discriminative Exemplar Coreset Selection (DECS) organizes representative diagnostic cases in a frozen DINOv2 feature space and refines their adaptive retrieval keys to construct the visual context state. Self-Refined Experience Summarization (SRES) distills reusable diagnostic experience from diverse reasoning trajectories and maintains it through adaptive experience-state updates to construct the experiential context state. During inference, the visual exemplars provided by DECS and the diagnostic experience accumulated by SRES are jointly incorporated into the multimodal context, enabling the frozen MLLM to perform dataset-specific diagnostic reasoning without updating pretrained model parameters. We systematically evaluate the proposed framework on a subset of the public MedMNIST 2D datasets and a private nasopharyngeal carcinoma MRI dataset. Experimental results show that our method consistently improves over representative gradient-free in-context learning methods and achieves competitive diagnostic performance on real-world clinical data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, in-context learning, medical image diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mD4cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies gradient-free adaptation of frozen multimodal large language models to target medical image classification datasets. The proposed clinician mimetic workflow constructs two dataset-specific context states without updating the parameters of Qwen3-VL-8B.

Critical strengths
  1. The paper distinguishes query-specific visual evidence from reusable cross-case diagnostic heuristics. This is a sensible design because a small number of retrieved images and a compact experience bank can encode complementary forms of target-dataset information without maintaining a separate fine-tuned MLLM checkpoint for every dataset.
Critical weaknesses
  1. The DECS retrieval key can become inconsistent with the actual evidence presented to the MLLM. The paper explicitly states that the refined key is not intended to reproduce its associated exemplar. This weakens the interpretation that DECS retrieves representative diagnostic evidence. The paper should quantify key-image mismatch and compare against consistent alternatives. This concern is central because DECS is used both for test-time visual context and for selecting the cases from which SRES constructs experience.

  2. The main comparison is unfair. More specifically, it does not match in supervision, context, or computation. If I understand correctly, the method uses substantially more labeled information, model calls, generated tokens, and inference-time context than the reported retrieval baselines.

Questions for authors
  1. MedMNIST was originally designed as a lightweight, MNIST-like benchmark for rapid prototyping, and several of its tasks rely on heavily standardized images. There are so many large-scale professional clinical datasets (such as MIMIC, CheXpert, CT-Rate), why is the 2D MedMNIST an adequate primary benchmark for the paper's claim of adapting MLLMs to heterogeneous, real-world medical-image distributions?

  2. What is the final Experience Bank length in entries and tokens for every dataset? Can the authors compare against an equal-token static class summary or equal-token randomly generated memory?

  3. How many Qwen forward passes, generated tokens, GPU hours, and wall-clock hours are required to construct each Experience Bank?How does this compare with training a linear probe or LoRA adapter on the same target split?

Reviewer_YCmaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Clinician Mimetic Workflow, a gradient-free adaptation framework for medical image classification using frozen MLLMs. Instead of updating pretrained model parameters, the method constructs two complementary dataset-specific context states. The first component, DECS builds a class-balanced exemplar bank using a frozen DINOv2 encoder and refines retrieval keys to incorporate complementary intra-class information. The second component, SRES generates multiple diagnostic reasoning trajectories, evaluates their quality, and distills reusable diagnostic heuristics into an Experience Bank through Add, Delete, Keep, and Modify operations. During inference, retrieved visual exemplars and the constructed Experience Bank jointly condition the frozen MLLM. However, several methodological choices and experimental limitations require further justification before the claimed advantages can be fully established.

Critical strengths

Strength 1: The paper proposes a context-space adaptation framework for frozen MLLMs without updating model parameters. DECS constructs a class-balanced exemplar bank and refines retrieval keys using complementary intra-class information, enabling reusable dataset-specific visual context for medical image diagnosis.

Strength 2: SRES constructs reusable diagnostic experience by evaluating multiple reasoning trajectories and maintaining an Experience Bank through dynamic state updates. This design integrates dataset-level diagnostic knowledge with query-specific visual exemplars, allowing the frozen MLLM to leverage complementary information during inference.

Strength 3: The experiments demonstrate consistent improvements over the evaluated gradient-free baselines across 11 MedMNIST datasets and a private NPC MRI dataset. The ablation studies further support the complementary contributions of DECS and SRES.

Critical weaknesses

Weakness 1: DECS refines retrieval keys by assigning training samples to their least-similar same-class keys. However, the associated exemplar images remain unchanged. Consequently, the refined retrieval keys may no longer closely align with their paired images. It is unclear whether this discrepancy could reduce the visual relevance of retrieved demonstrations or how the proposed refinement compares with established coreset construction strategies.

Weakness 2: Although the ablation studies demonstrate the benefits of experience-state updating, they do not fully distinguish the effects of experience quality and context length from those of providing additional task-relevant textual information. More controlled comparisons are needed to substantiate the proposed mechanism and clarify its novelty relative to existing memory-based approaches.

Weakness 3: Although the framework avoids gradient-based updates, SRES introduces additional computational costs through repeated reasoning generation, evaluation, and experience-state updates. The absence of comparisons with lightweight fine-tuning methods makes it difficult to assess whether the proposed approach offers practical advantages in terms of both computational efficiency and diagnostic performance.

Questions for authors

Q1. In Equations (3)–(4), each training sample is assigned to the least-similar same-class retrieval key, while the real exemplar image paired with that key remains fixed. Could the authors further justify how this procedure improves query-exemplar relevance despite the potential discrepancy between refined keys and their associated images? It would be useful to analyze how this discrepancy affects the relevance of retrieved exemplars. Comparisons against alternative assignment strategies would help establish whether the proposed refinement is genuinely beneficial.

Q2. Although Table 4 demonstrates the benefits of experience-state updating, it remains unclear whether the improvements come from the proposed refinement mechanism or differences in the amount and content of textual context. Could the authors compare SRES with simpler alternatives, while controlling for the amount of textual context?

Q3. Could the authors compare the proposed method with lightweight parameter-efficient fine-tuning approaches, using the same backbone and target-dataset training split? In particular, reporting both classification performance and computational costs would help clarify the practical advantages of the proposed gradient-free adaptation framework.

Reviewer_xTF3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adapts a frozen MLLM (Qwen3-VL-8B) to a medical image classification dataset by building context instead of updating weights. The method has two parts. DECS selects a class-balanced exemplar bank in DINOv2 feature space and refines a retrieval key for each exemplar. SRES samples several reasoning rollouts on those exemplars, has the same MLLM score them against the label, and turns the contrast between good and bad rollouts into short textual heuristics. These heuristics are kept in an Experience Bank that is edited with Add/Delete/Modify/Keep actions. At test time, the retrieved exemplars and the Experience Bank together form the prompt. The method is evaluated on 11 MedMNIST 2D datasets and a private nasopharyngeal carcinoma (NPC) MRI staging dataset, where it outperforms retrieval-based in-context learning baselines.

Critical strengths
  1. The setting is practical. Adapting a frozen MLLM to a new medical dataset without any gradient updates is attractive for hospitals and labs with limited compute or restrictions on fine-tuning, and the method is simple enough to reimplement from Algorithms 1–2.
  2. The comparison is controlled and the gains are consistent. All methods share the same backbone, image encoder, number of exemplars and data splits, and the method improves over the baselines on every MedMNIST dataset rather than only a few.
  3. The ablations are informative. The step-by-step build-up (random ICL, kNN retrieval, DECS, DECS+SRES) shows what each component adds. The finding that an edited Experience Bank works much better than an append-only one, while staying small, is useful on its own for anyone building experience memories.
Critical weaknesses
  1. The work largely applies existing training-free skill/experience learning to a medical subdomain. SRES closely mirrors Training-Free GRPO (arXiv 2510.08191): group rollouts scored by a reward, contrastive extraction of textual experience, and an experience library updated with Add/Delete/Modify/Keep. Combining retrieved exemplars with distilled insights in the prompt of a frozen model follows ExpeL (arXiv 2308.10144), and MedExpMem (arXiv 2605.22872) already brought experience memory to medical image diagnosis with VLMs. None of these is cited or compared. What remains new is mostly the DECS exemplar selection and the application to MedMNIST, which makes the contribution incremental.
  2. The evaluation uses too few, and too narrow, datasets. All experiments are single-label classification: 11 MedMNIST 2D subsets (Sec. 4.1) plus one private binary NPC staging set (Appendix B). MedMNIST consists of small, standardized benchmark images, and the only real clinical data is private and from one site. There is no other public clinical dataset, no VQA or report-generation task, and no external validation, so it is unclear whether the method holds up on realistic diagnostic data.
  3. There is no uncertainty reporting. Tables 1–4 give five-run means with no standard deviations, confidence intervals or test-set sizes. Several conclusions rest on gains of a few points (for example, DECS over kNN), which may be within sampling noise at temperature 0.8.
Questions for authors
  1. How does SRES differ from Training-Free GRPO? Could you compare DECS+SRES with Training-Free GRPO and an ExpeL-style exemplar-plus-insight baseline using the same Qwen3-VL-8B backbone and splits?
  2. Can you report results on public clinical datasets beyond MedMNIST (for example, full-resolution chest X-ray, dermoscopy or fundus datasets) and on an external NPC cohort from another site?
  3. How does the method compare with LoRA fine-tuning of Qwen3-VL-8B and with a kNN or linear classifier on the same DINOv2 features? How many MLLM calls and GPU hours does building the context take per dataset?
  4. Could you add mean ± std over the five runs and the test-set size for each dataset, and state whether the full MedMNIST test sets were used?

Comments on the paper

1 comment

Thank you for the detailed response. However, several concerns remain unaddressed.

W1: My question was not about the rationale of refining keys, which the response restates. My concerns are about quantifying the mismatch between refined keys and their paired images and comparing against consistent retrieval alternatives. Table 3 shows an aggregate gain but does not isolate this effect.

Q2: The response only promises to add equal-token static class summaries and randomly generated memory. No results are provided. So the question is still open on whether the gains come from the diagnostic content of the ExperienceBank, ather than from extra context tokens alone.

Q1: There is no direct response to my Q1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.