acceptodds
Withdrawn from ICLR 2027

MedVESS: From Seeing to Understanding Medical Images with Training-Free Latent Steering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

General Vision-Language Models (VLMs) offer a promising foundation for medical image understanding without requiring task-specific model design. However, reliable medical image understanding remains challenging, with insufficient reliance on medical visual evidence and limited medical semantic interpretation of the visual content. To address these limitations, we propose MedVESS, a training-free latent steering framework that constructs two complementary, modality-specific steering directions. Visual Evidence Steering (VES) contrasts original medical images with fully masked counterparts to strengthen reliance on medical visual evidence, while Clinical Semantic Steering (CSS) contrasts complete medical reports with semantic-depleted counterparts under the same visual input to reinforce associations between medical visual patterns and corresponding clinical concepts. The two steering vectors are constructed offline and jointly injected into a selected intermediate layer at inference time without updating model parameters. Experiments on four medical VQA benchmarks spanning radiology, fundus photography, and pathology show consistent improvements across two general VLMs, achieving competitive performance against several specialized medical VLMs without medical-domain fine-tuning. These results highlight the potential of training-free latent steering as a lightweight approach to improving reliable medical image understanding in general VLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Vision-Language Models, Medical Image Understanding, Activation Steering, Training-Free Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p3PyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MedVESS, which changes a VLM’s internal activations without updating its weights. It builds a Visual Evidence Steering direction from original versus black images and a Clinical Semantic Steering direction from full reports versus reports with medical findings removed. Separate directions are built for each image type and added at an offline-selected decoder layer. Both Qwen models improve on four medical VQA datasets.

Critical strengths

The method adds little computation at inference. Appendix B clearly explains vector construction, sign selection, scaling, and where the edit is applied. The experiments cover radiology, fundus, and pathology, with component tests and analyses of layer choice and edit strength. The class-level results and always-Yes baseline help explain the large gain on the imbalanced fundus dataset.

Critical weaknesses
  • The main comparison lacks prior activation-steering baselines on the same backbone. VTI and VISTA are relevant candidates, but applying them to the Qwen models may require changes to vector construction and intervention sites. Comparisons with suitable adaptations, or a clear explanation of why these methods are not applicable, would help establish the added value of MedVESS.

  • CSS provides most of the gain, but its effect on image understanding is unclear. Both inputs contain the same image, while report editing changes length, medical terms, and positive or negative statements. Please compare CSS built without images, mismatched image-report pairs, and edits that keep text length similar. These tests would help show whether CSS improves use of the image or mainly changes answer preferences.

Questions for authors

See the weaknesses above.

Reviewer_tD1qOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This submission introduces MedVESS, a training-free activation-steering method for general VLMs on medical VQA. It builds two steering vectors for each modality, both as PC1 of paired activation differences at the last token. The two vectors are summed and injected at one layer, chosen offline by the geometric mean of a magnitude score and a direction-change score. The paper reports gains on MIMIC-CXR, IU-Xray, PMC-OA and Harvard-FairVLMed, competitive with medical VLMs.

Critical strengths
  1. The method, designed along the steering vector track, is simple and cheap.
  2. The Visual Correction Rate (VCR) is a useful diagnostic, and the natural-vs-medical reliance gap is a clean motivating observation.
  3. The ablations are reasonable: VES/CSS components, shared vs per-modality vectors, a full 36-layer sweep, an α sweep, and a class-wise analysis on FairVLMed.
Critical weaknesses

1.Overall, the 2 steering designs are not fully convincing and reasonable.

  • (1) CSS might be shifting the answer or language prior rather than improving “semantic interpretation.” For example, in CSS, the method construct the contrastive pair by appending a correct or a wrong report with key findings removed, but inference has no report. CSS therefore are probably encoding a prior such as “a findings is in context.” Thus, injecting it plausibly pushes the model toward affirming findings. Interestingly, the results in Appendix (Table 7) additionally support this by showing the increasing Yes-bias after applying the steering. (2) The visual part design is actually weak according to the ablation results, for example, CSS alone gives most of the improvement (FairVL 32.8→65.5, versus +1 from VES). This makes the “visual evidence” framing hard to defend.
  • The modality level construction is too general and not flexible enough for adapting to each image in each case. The construction of visual vectors: the method simply takes the top PCA dimensions and apply that at modality-level. However, this only injects modality-shared information, which is not adaptive enough for each image's specific condition and findings in the modality, resulting in limited improvement. Same issue with the text semantic vectors, you are basically injecting a quite general semantic steering vector, which is not at a fine-grained level, resulting in more like a "language-prior" injection.
  • And there is no substancial technical contribution of the steering design, because previous multimodal works such as VTI already apply similar steering on both text and visual sides.

2.Continue on the previous weakness, some important ablations are missing. (1) One decisive experiment for visual steering is missing: apply MedVESS under the no-image or black-image condition. If the gains persist, the method is not increasing reliance on the image. (2) Other missing controls: a random or Yes-bias (language prior bias) direction of matched norm; prediction-distribution shifts on MIMIC, IU and PMC.

3.Baselines are not fully comparable and incomplete. (1) There is no comparison to steering or decoding methods on the same backbones: VTI, VISTA, AFTER, MEDA, VCD, ARCD. These are the actual competitors, and MEDA is the closest prior work. (2) Table 1 mixes backbones. RULE and MMed-RAG are built on LLaVA-Med, and the baseline numbers appear to be imported from other papers. “Competitive with medical VLMs” therefore conflates backbone strength with the method’s contribution. Ideally, you should be able to apply your steering method to any backbone model, including medical backbones, and the comparison with baselines should also be conducted on various backbones to fully validate the effectiveness.

4.Like I mentioned in 3, the experiments are not complete. More experiments applying the designed steering method on more backbones such as medical ones are highly expected.

5.Evaluation scope. The evaluation covers closed-ended VQA only. There is no open-ended VQA, no report generation.

6.The layer-selection and modality-specific claims are weaker than presented. (1) Qwen3 selects L21 for all four datasets, so the criterion does not appear to be modality-adaptive in practice. Table 5 compares only five hand-picked candidates. (2) The t-SNE separation of modalities is expected and does not imply that the steering directions differ. Cross-modality cosines of the vectors should be reported. (3) Per-modality vectors require knowing the modality at test time. Routing, and the effect of misrouting, are not addressed.

Questions for authors
  1. What does MedVESS do when applied with a black image or no image? And why black, do you try different masking strategies, like blurring, zeroing or just removing the image input
  2. What are the label-prediction distributions before and after steering on MIMIC, IU and PMC?
  3. How was α selected? Was a held-out validation split used?
Reviewer_eDeMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MedVESS aims to improve medical image understanding in general vision-language models without updating their weights. It builds two steering vectors for each imaging modality. Visual Evidence Steering (VES) compares an original image with a black image while keeping the question and report unchanged. Clinical Semantic Steering (CSS) compares a full report with a version that has its clinical content removed, using the same image. The vectors are built offline with PCA. They are then added to the current token's hidden state at one selected decoder layer.

The paper evaluates Qwen2.5-VL-7B and Qwen3-VL-8B on four medical VQA benchmarks across radiology, fundus imaging, and pathology. Both accuracy and F1 improve in all eight model-dataset settings. The paper also studies the two components, modality-specific construction, layer selection, steering strength, and construction-set size. The results support the method's practical value. The stronger claim about better visual and clinical understanding needs more evidence.

Critical strengths
  1. The method gives useful gains without updating model weights. Table 1 reports higher accuracy and F1 for both backbones on all four benchmarks. For Qwen3-VL-8B, MIMIC-CXR accuracy rises from 72.46% to 79.00%, and PMC-OA accuracy rises from 61.84% to 68.05%. The radiology vectors built from MIMIC-CXR are also used on IU-Xray. This gives some evidence that the approach can transfer beyond the construction dataset. A simple intervention with these gains could be useful when full fine-tuning is costly.

  2. The ablations and implementation details are fairly complete. Tables 2 and 3 support the combined design and modality-specific construction. Appendix D goes beyond the few layers shown in the main text. It tests all 36 layers on MIMIC-CXR. The selected L21 ranks second and is close to the best layer, L22. The sample-size study also shows useful results with 50-500 construction examples. These analyses make the method more convincing as a practical approach.

Critical weaknesses
  1. The main mechanism is not yet separated from simpler explanations. In Section 4.4, CSS keeps the image fixed and changes the report. This isolates a report-related difference, but does not show that the difference captures a link between image evidence and clinical concepts. Removing clinical text can also change report length, style, information content, and disease-word frequency. Some negative reports become a short neutral placeholder. A vector built this way may mainly change medical language or answer preferences.

    This issue matters because CSS accounts for most of the gains in Table 2. On MIMIC-CXR, CSS alone reaches 78.46% accuracy, compared with 79.00% for the full method. On Harvard-FairVLMed, it reaches 65.54%, compared with 67.53% for the full method.

    Table 7 makes this concern more concrete. Harvard-FairVLMed has 86.43% Yes labels. For Qwen3, accuracy improves from 32.79% to 67.53%, and Macro-F1 improves from 32.50% to 53.78%. But No-class F1 barely changes, from 28.06% to 28.57%. The share of Yes predictions rises from 20.16% to 68.12%. If all 4,266 predictions are counted as Yes or No, the reported totals imply balanced accuracy of about 59.66% before steering and 59.23% after steering. These are my calculations from the summary numbers, not values reported by the authors. The Macro-F1 result is better than always predicting Yes. Still, it does not rule out a large role for answer-bias correction.

    The VCR increase in Figure 3 is useful supporting evidence. However, it is unclear whether the comparison uses the same set of black-image errors. If that set changes after steering, the two conditional rates use different groups. The t-SNE plots also show modality differences, but do not by themselves establish a need for separate steering directions. Table 3 is stronger evidence for that design choice.

    The most useful next experiment is a controlled CSS comparison using correctly paired images versus shuffled or omitted images. Keep the reports, sample count, and tuning budget the same. This would test whether correct image-report pairing is important to the gain. Further useful checks include black-image construction, a length-matched report control, and evaluation with mismatched or black images. A simple answer-bias calibration baseline would help explain the FairVLMed result. Please also report balanced accuracy and per-class recall alongside Macro-F1. A short mechanism discussion should explain why the CSS difference is expected to capture image-text interaction rather than text content alone.

  2. The closest method comparisons and some evaluation details are missing. Section 2.2 discusses VTI, VISTA, AFTER, and MEDA, but Table 1 includes no direct comparison with an existing activation-steering method. The medical fine-tuning and retrieval baselines are useful. They do not establish the value of the new VES/CSS construction over prior steering designs. VES-only is an internal ablation and cannot replace those comparisons. Please compare with the closest compatible medical steering method, such as MEDA, and a relevant general steering method on the same Qwen backbone. Keep the data split, construction budget, decoding settings, and tuning budget as similar as possible. Report any unavoidable differences. Without this, the paper shows an improvement over the base model, but its added contribution remains uncertain.

    Appendix E reports a steering-strength sweep for each model-dataset setting. It does not clearly say how the values used in Table 1 were selected, or which independent validation set was used. This needs clarification; it is not evidence that test-set tuning occurred. Also, the IU-Xray accuracy gains are only 0.46 and 0.71 percentage points. Confidence intervals or repeated construction-set samples would help establish their stability. The lightweight design is appealing, but a measured cost comparison would make that claim more useful. Please include inference latency and the offline cost of vector construction and report processing.

Questions for authors
  1. How was the steering strength selected for each result in Table 1? Was it chosen on an independent validation set? Please state the selection rule and whether evaluation labels were used in this choice.

  2. In Figure 3, is VCR measured on one fixed set of black-image errors for both systems, or on a separate error set for each system? Can you report the results on a fixed set, including cases where restoring the image changes a correct answer into a wrong one?

  3. Does CSS still provide a similar gain when its construction images are shuffled or omitted? A controlled comparison with the same reports and budget would directly affect my view of the claimed visual-semantic mechanism. Optional length-matched report and answer-bias controls could help explain any remaining gain.

  4. How does MedVESS compare with a compatible medical steering method, such as MEDA, and a relevant general steering method under matched settings? Please also provide uncertainty estimates for the small gains and measured construction and inference costs. These checks would help establish the method's added value and practical benefit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.