- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CheXpercept, a chest X-ray benchmark designed to assess visual perception beyond disease-presence classification. Its 12,000 question–answer items, derived from 2,400 images and reviewed by six medical experts, cover lesion detection, contour evaluation and revision, and attribute extraction for eight pulmonary and cardiac lesions. An evaluation of 18 general and medical vision-language models finds that performance declines substantially beyond coarse-level detection. Recent general models outperform the evaluated medical models overall, with backbone capability appearing to have a greater effect than domain adaptation.
Critical strengths
This paper is well written and easy to follow. The motivation is quite interenting and important.
Critical weaknesses
In this work, the authors introduced a medical benchmark, named CheXpercept. They firstly collected a related dataset and evaluated 18 general and medical VLMs. Yet the above contributions might be insufficient for an acceptance of top-tier conferences with the following concerns. The first one is the gap or mismatch between motivation and their finished work. The motivation of expert-level lesion perception is quite interesting and important, yet their collected dataset aims to find the regions of lesion step by step, which seems like an active segmentation mask annotation tool or framework. Even the generated questions mainly about the locations of the lesions, in other words, just like the segmentation masks. The indeed clinical impact mainly focuses on the diagnosis as well as their interpretability. If the generated QA pairs are almost related to the locations of lesions, why not just using SAM-like segmentation models?
Further, if the authors could step forward, adding more usages of lesions, such as the improvement for Chain-of-Thought of MLLMs in Medical Image Understanding, it would be much more important and promising.
The claims or findings about the experimental comparisons might be unfair, such as ‘Notably, recent general models outperform recent medical VLMs overall.’ In abstract. The general models are usually 27B, 31B, even 122B, while the medical models are range from 4B, 7B 32B. The authors need to consider the model size in the final claims.
Questions for authors
see weakness
If we have misunderstood the intent of your feedback or questions in any way, please let us know. We are more than happy to engage in further discussion promptly.
W: Weakness, Q: Question
[W1] Limited human evaluation
Recognizing the practical burden of asking individual medical experts to manually evaluate a large volume of questions, we limited the number of evaluation items per physician. However, we carefully constructed the questions so that all lesion types and evaluation paths were uniformly included. Furthermore, to enhance the reliability of the human baseline, we expanded the pool of evaluators from 3 to 5 by recruiting two additional experts (who evaluated 24 questions each). The updated results, including the 95% confidence intervals, are presented in the table below:
[W2] Reliability of ground-truth masks and potential selection bias
Reliability of the ground truth
Given the practical limitations of manually annotating every lesion mask for a large-scale dataset, we adopted a model-assisted approach where physicians selected the highest-quality mask from model-generated candidates. During this process, physicians implicitly envision an ideal contour for the lesion and choose only the candidate that best aligns with this standard. Therefore, the selected masks are of comparable quality to those drawn entirely by hand, providing sufficient reliability to serve as ground truth for evaluation.
Potential Selection Bias
To address concerns that certain types of cases (e.g., difficult-to-judge lesions) might have been systematically excluded during the selection process, we compared our dataset (CheXpercept) with the source population (MIMIC-ILS). We used Qwen3.8-27B to structure the MIMIC-CXR radiology reports for both groups, extracting information on lesion severity and extent to compare their distributions. The analysis revealed no significant differences between the two groups. Notably, "mild" or "small" lesions, which are often harder to assess due to ambiguous boundaries, still account for a substantial proportion in CheXpercept. Pneumothorax cases were excluded from this specific comparison because their radiology reports were unavailable. The detailed results (including the macro average for each set and the detailed distribution across CheXpert lesions) are presented in the table below: