- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Perception-Bench and RadSight to address fine-grained visual perception deficiencies in medical MLLMs. Perception-Bench comprises 1.13 million samples across six tasks involving 2D X-rays and 3D CT scans, exposing substantial perceptual limitations in existing models. Building on these findings, RadSight employs native 2D/3D visual encoders and four-stage progressive training on 8.37 million samples to improve perception, diagnosis, and report generation. Experiments demonstrate consistent gains across Perception-Bench and multiple public medical benchmarks.
Critical strengths
- The paper addresses an important limitation of medical MLLMs by highlighting the role of fine-grained visual perception in reliable clinical diagnosis and report generation.
- Perception-Bench provides a comprehensive evaluation framework covering six tasks across 2D X-rays and 3D CT scans, enabling systematic assessment of perceptual and clinical understanding capabilities.
- RadSight is trained on 8.37 million samples and demonstrates consistent improvements across Perception-Bench and multiple public medical benchmarks. However, the accessibility of its training samples to the research community remains unclear.
Critical weaknesses
- RadSight primarily integrates existing 2D/3D visual encoders, multi-task supervision, and curriculum learning, without introducing novel visual perception mechanisms. Its contributions lie mainly in large-scale data construction and the integration of existing training strategies.
- RadSight combines specialized visual encoders, 8.37 million training samples, and progressive training. Although the authors provide some ablation studies, they do not isolate the individual contributions of the 2D and 3D encoders or conduct rigorous comparisons under matched data scales, pretraining resources, and computational budgets.
- Perception-Bench contains 1.13 million evaluation instances, potentially leading to substantial evaluation costs and sample redundancy. The paper lacks analyses of model ranking stability and discriminative power across different evaluation scales, leaving the necessity of such a large benchmark insufficiently justified.
Overall, the work relies heavily on large-scale data and computational resources, while offering limited methodological novelty and insufficient analysis of whether these additional resources are necessary to achieve the reported improvements.
If the authors adequately address these concerns, I would consider raising my score.
Questions for authors
Please refer to Weaknesses.
5 comments
Response to Reviewer ho4V (Part 2 of 2)
[continued] W3. Benchmark scale, redundancy, and ranking stability.
(3) Smaller subsets increase score variability and affect model rankings. To preserve evaluation dimensions, we further random sampled within each dimension over 200 runs per scale, reporting mean ± standard deviation. Smaller subsets yield greater variability, and relative rankings change across scales, indicating sensitivity to evaluation size.
| Task and dimension | Model | 10% | 20% | 50% | 100% |
|---|---|---|---|---|---|
| Task 1: Enhancement attribute | Qwen3-VL-8B | 63.89 ± 12.47 | 61.49 ± 8.51 | 62.52 ± 4.03 | 62.28 |
| InternVL-3.5-8B | 61.75 ± 12.50 | 61.65 ± 7.39 | 61.30 ± 3.98 | 61.40 | |
| Task 3: Liver segment VII identification | InternVL-3.5-8B | 59.51 ± 4.87 | 59.04 ± 3.36 | 59.23 ± 1.58 | 59.29 |
| M3D-Phi-3-4B | 59.39 ± 4.84 | 60.05 ± 3.36 | 59.64 ± 1.53 | 59.73 | |
| Task 4: Hydronephrosis-related disease prediction | Qwen3-VL-8B | 22.17 ± 3.98 | 22.35 ± 2.78 | 22.26 ± 1.36 | 22.22 |
| MedGemma-1.5-4B | 21.75 ± 3.48 | 22.59 ± 2.48 | 22.56 ± 1.09 | 22.60 | |
| Task 5: Hydronephrosis anomaly detection | M3D-Phi-3-4B | 54.94 ± 4.30 | 54.15 ± 2.61 | 54.51 ± 1.23 | 54.56 |
| InternVL-3.5-8B | 54.76 ± 4.06 | 54.91 ± 2.51 | 54.65 ± 1.45 | 54.62 |
(4) The benchmark must support broader model evaluation. The evaluated models represent only a subset of medical MLLMs. Benchmark scale should therefore support fine-grained comparisons of future models, rather than depend solely on current aggregate rankings. Million-instance evaluation sets also exist in general multimodal benchmarks; for example, ANetQA (CVPR 2023) contains approximately 1.50 million test instances [Ref1].
Taken together, these analyses support retaining the full Perception-Bench for fine-grained coverage and stable evaluation.
[Ref1] ANetQA: A Large-Scale Benchmark for Fine-Grained Compositional Reasoning over Untrimmed Videos. CVPR 2023.
I thank the authors for their detailed response. However, I believe several concerns remain insufficiently addressed.
-
Regarding W1: The authors state that "Our contributions center on problem identification, data curation, and effective training." However, the training approach appears to rely primarily on standard techniques. Given that the main contributions lie in problem identification and dataset construction rather than methodological innovation, I believe this work would be better suited to the Evaluation and Datasets Track.
-
Regarding W3: I am not convinced by the authors' argument that "Benchmark scale should therefore support fine-grained comparisons of future models, rather than depend solely on current aggregate rankings." The authors should provide concrete experimental evidence demonstrating that the 1.13 million evaluation instances enable meaningful fine-grained comparisons that cannot be achieved with smaller benchmarks.
Finally, I do not understand the relevance of the authors' claim that "ANetQA (CVPR 2023) contains approximately 1.5 million test instances." ANetQA does not provide sufficient evidence to justify the necessity of Perception-Bench's scale.
In light of the authors' response, I have decided to maintain my score.
Response to Reviewer ho4V (Part 1 of 2)
We thank the reviewer for recognizing the importance of fine-grained visual perception, Perception-Bench’s comprehensive evaluation, and RadSight’s consistent performance gains. We will fully open-source our data, code, and model weights. Below, we address the comments point by point.
W1. Clarification of the contributions.
R1. Thank you for considering our contributions. Our contribution is not a new architecture for visual perception, but (1) the perception-failure framing and Perception-bench, recognized by reviewer 2X9D, identifies visual perception deficiencies in medical MLLMs; and (2) a perception-oriented training corpus provides explicit fine-grained supervision. Native 3D encoding and the perception-first curriculum support this data-driven approach by preserving spatial information and progressively developing perceptual capabilities. RadSight emerging naturally from this approach.
Our contributions center on problem identification, data curation, and effective training. We view high-quality data and targeted training as key ways to addressing such limitations in the era of LLM.
Open-sourcing is an independent contribution: We fully recognize the value of open data to the research community. Before submission, we had planned to release Perception-Bench, the training corpus, code, and model weights, consistent with our previous releases on Hugging Face. The code was included with the submission. We apologize for not clearly stating our release plan. With the revision, we will publicly release all redistributable data without access restrictions, the code, and model weights. For datasets requiring credentials, users must obtain access independently and use our LLM preprocessing code to reconstruct the benchmark and training corpus, respecting the source datasets’ access restrictions and licenses. Our anonymous project website (https://anonymous.4open.science/r/radsight-anonymous-8900/) now provides training code, data-processing code, and reproduction instructions (the checkpoints exceeded the upload size limit). We will document these release details in the revision.
W2. Ablation of native 3D encoding.
R2. We thank the reviewer for this suggestion. We added a randomly initialized 3D encoder ablation (Table R1). Despite completing less than one epoch due to time and resource constraints, it outperforms the slice-based baseline on Tasks 1–4, suggesting benefits beyond pretraining. Both settings use the same data and curriculum; total optimizer steps will be matched upon completion. We will update the full one-epoch results and clarify sampling settings and token budgets in the revision.
Table R1. Effects of native 3D encoding. Results without 3D encoder pretraining are preliminary (<1 epoch) due to time and resource constraints and will be updated after one full epoch.
W3. Benchmark scale, redundancy, and ranking stability.
R3. Thank you for raising concerns about evaluation scale, cost, and potential redundancy. We respond in four parts.
(1) Scale reflects broad coverage rather than duplicated questions. Perception-Bench uses the official test splits of multiple 2D chest X-ray and 3D CT datasets, covering six tasks across modalities, organs, diseases, and attributes. Its 1.13 million instances represent distinct evaluation dimensions, rather than repeated questions with different prompt templates.
(2) Random subsampling reduces fine-grained coverage. For Task 1, we random sampled 10%, 20%, and 50% of instances over 200 runs per scale. Coverage of disease–attribute combinations fell to 68.65%, 79.17%, and 91.66%, respectively. No run retained all combinations at any reduced scale.