- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MedVESS, which changes a VLM’s internal activations without updating its weights. It builds a Visual Evidence Steering direction from original versus black images and a Clinical Semantic Steering direction from full reports versus reports with medical findings removed. Separate directions are built for each image type and added at an offline-selected decoder layer. Both Qwen models improve on four medical VQA datasets.
Critical strengths
The method adds little computation at inference. Appendix B clearly explains vector construction, sign selection, scaling, and where the edit is applied. The experiments cover radiology, fundus, and pathology, with component tests and analyses of layer choice and edit strength. The class-level results and always-Yes baseline help explain the large gain on the imbalanced fundus dataset.
Critical weaknesses
-
The main comparison lacks prior activation-steering baselines on the same backbone. VTI and VISTA are relevant candidates, but applying them to the Qwen models may require changes to vector construction and intervention sites. Comparisons with suitable adaptations, or a clear explanation of why these methods are not applicable, would help establish the added value of MedVESS.
-
CSS provides most of the gain, but its effect on image understanding is unclear. Both inputs contain the same image, while report editing changes length, medical terms, and positive or negative statements. Please compare CSS built without images, mismatched image-report pairs, and edits that keep text length similar. These tests would help show whether CSS improves use of the image or mainly changes answer preferences.
Questions for authors
See the weaknesses above.