acceptodds
Under review as a conference paper at ICLR 2027

RadSight: Towards Perceptually Reliable Multimodal Radiology Image Understanding

Abstract

Medical multimodal large language models (MLLMs) are increasingly expected to perform complex image understanding tasks, yet their reliability is often compromised by frequent errors in visual interpretation. To systematically trace these failures, we traverse the hierarchy from high-level clinical tasks down to fundamental visual perception. We therefore introduce Perception-Bench, a large-scale benchmark that assesses medical MLLMs across six dimensions: attribute judgment, spatial grounding, spatial understanding, disease prediction, anomaly detection, and report generation, spanning both 2D and 3D radiology images. Our analysis on Perception-Bench reveals that existing MLLMs lack the ability to capture even the most basic lesion attributes, such as location, size, and density. This failure to ground clinical predictions in visual evidence reveals a critical yet overlooked source of diagnostic unreliability: deficient low-level visual perception. Motivated by this, we propose RadSight, a perception-driven MLLM with dedicated 2D and 3D visual encoders that preserve native imaging spatial structures. RadSight is trained on an 8.37-million-sample perception-oriented corpus through a four-stage curriculum: visual-language alignment, fine-grained visual perception, clinical diagnosis, and diagnostic interpretation. This curriculum explicitly develops perceptual capabilities before progressing to diagnosis and report generation. RadSight outperforms existing MLLMs across all six dimensions of Perception-Bench, with particularly strong gains in spatial grounding and clinical diagnosis. It also achieves consistent improvements on public 2D and 3D medical benchmarks, further demonstrating that robust low-level visual perception is a critical foundation for reliable clinical understanding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Medical Multimodal Large Language Models, Medical Image Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ho4VOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Perception-Bench and RadSight to address fine-grained visual perception deficiencies in medical MLLMs. Perception-Bench comprises 1.13 million samples across six tasks involving 2D X-rays and 3D CT scans, exposing substantial perceptual limitations in existing models. Building on these findings, RadSight employs native 2D/3D visual encoders and four-stage progressive training on 8.37 million samples to improve perception, diagnosis, and report generation. Experiments demonstrate consistent gains across Perception-Bench and multiple public medical benchmarks.

Critical strengths
  1. The paper addresses an important limitation of medical MLLMs by highlighting the role of fine-grained visual perception in reliable clinical diagnosis and report generation.
  2. Perception-Bench provides a comprehensive evaluation framework covering six tasks across 2D X-rays and 3D CT scans, enabling systematic assessment of perceptual and clinical understanding capabilities.
  3. RadSight is trained on 8.37 million samples and demonstrates consistent improvements across Perception-Bench and multiple public medical benchmarks. However, the accessibility of its training samples to the research community remains unclear.
Critical weaknesses
  1. RadSight primarily integrates existing 2D/3D visual encoders, multi-task supervision, and curriculum learning, without introducing novel visual perception mechanisms. Its contributions lie mainly in large-scale data construction and the integration of existing training strategies.
  2. RadSight combines specialized visual encoders, 8.37 million training samples, and progressive training. Although the authors provide some ablation studies, they do not isolate the individual contributions of the 2D and 3D encoders or conduct rigorous comparisons under matched data scales, pretraining resources, and computational budgets.
  3. Perception-Bench contains 1.13 million evaluation instances, potentially leading to substantial evaluation costs and sample redundancy. The paper lacks analyses of model ranking stability and discriminative power across different evaluation scales, leaving the necessity of such a large benchmark insufficiently justified.

Overall, the work relies heavily on large-scale data and computational resources, while offering limited methodological novelty and insufficient analysis of whether these additional resources are necessary to achieve the reported improvements.

If the authors adequately address these concerns, I would consider raising my score.

Questions for authors

Please refer to Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ho4V (Part 1 of 2)

We thank the reviewer for recognizing the importance of fine-grained visual perception, Perception-Bench’s comprehensive evaluation, and RadSight’s consistent performance gains. We will fully open-source our data, code, and model weights. Below, we address the comments point by point.

W1. Clarification of the contributions.

R1. Thank you for considering our contributions. Our contribution is not a new architecture for visual perception, but (1) the perception-failure framing and Perception-bench, recognized by reviewer 2X9D, identifies visual perception deficiencies in medical MLLMs; and (2) a perception-oriented training corpus provides explicit fine-grained supervision. Native 3D encoding and the perception-first curriculum support this data-driven approach by preserving spatial information and progressively developing perceptual capabilities. RadSight emerging naturally from this approach.

Our contributions center on problem identification, data curation, and effective training. We view high-quality data and targeted training as key ways to addressing such limitations in the era of LLM.

Open-sourcing is an independent contribution: We fully recognize the value of open data to the research community. Before submission, we had planned to release Perception-Bench, the training corpus, code, and model weights, consistent with our previous releases on Hugging Face. The code was included with the submission. We apologize for not clearly stating our release plan. With the revision, we will publicly release all redistributable data without access restrictions, the code, and model weights. For datasets requiring credentials, users must obtain access independently and use our LLM preprocessing code to reconstruct the benchmark and training corpus, respecting the source datasets’ access restrictions and licenses. Our anonymous project website (https://anonymous.4open.science/r/radsight-anonymous-8900/) now provides training code, data-processing code, and reproduction instructions (the checkpoints exceeded the upload size limit). We will document these release details in the revision.

W2. Ablation of native 3D encoding.

R2. We thank the reviewer for this suggestion. We added a randomly initialized 3D encoder ablation (Table R1). Despite completing less than one epoch due to time and resource constraints, it outperforms the slice-based baseline on Tasks 1–4, suggesting benefits beyond pretraining. Both settings use the same data and curriculum; total optimizer steps will be matched upon completion. We will update the full one-epoch results and clarify sampling settings and token budgets in the revision.

Table R1. Effects of native 3D encoding. Results without 3D encoder pretraining are preliminary (<1 epoch) due to time and resource constraints and will be updated after one full epoch.

Modeltask1task2task3task4task5task6Average
Qwen3-VL-finetune51.8346.3569.8253.1476.2132.8055.03
RadSight w/o 3D encoder pretraining52.0347.4072.2455.0274.2830.8255.30
RadSight52.4549.4572.4456.1681.1036.4558.01

W3. Benchmark scale, redundancy, and ranking stability.

R3. Thank you for raising concerns about evaluation scale, cost, and potential redundancy. We respond in four parts.

(1) Scale reflects broad coverage rather than duplicated questions. Perception-Bench uses the official test splits of multiple 2D chest X-ray and 3D CT datasets, covering six tasks across modalities, organs, diseases, and attributes. Its 1.13 million instances represent distinct evaluation dimensions, rather than repeated questions with different prompt templates.

(2) Random subsampling reduces fine-grained coverage. For Task 1, we random sampled 10%, 20%, and 50% of instances over 200 runs per scale. Coverage of disease–attribute combinations fell to 68.65%, 79.17%, and 91.66%, respectively. No run retained all combinations at any reduced scale.

Task 1 sampling proportionDisease–attribute coverageRuns retaining all combinations
10%68.65%0.0%
20%79.17%0.0%
50%91.66%0.0%
100%100.00%Full-set reference
AuthorsOpenReview ↗

Response to Reviewer ho4V (Part 2 of 2)

[continued] W3. Benchmark scale, redundancy, and ranking stability.

(3) Smaller subsets increase score variability and affect model rankings. To preserve evaluation dimensions, we further random sampled within each dimension over 200 runs per scale, reporting mean ± standard deviation. Smaller subsets yield greater variability, and relative rankings change across scales, indicating sensitivity to evaluation size.

Task and dimensionModel10%20%50%100%
Task 1: Enhancement attributeQwen3-VL-8B63.89 ± 12.4761.49 ± 8.5162.52 ± 4.0362.28
InternVL-3.5-8B61.75 ± 12.5061.65 ± 7.3961.30 ± 3.9861.40
Task 3: Liver segment VII identificationInternVL-3.5-8B59.51 ± 4.8759.04 ± 3.3659.23 ± 1.5859.29
M3D-Phi-3-4B59.39 ± 4.8460.05 ± 3.3659.64 ± 1.5359.73
Task 4: Hydronephrosis-related disease predictionQwen3-VL-8B22.17 ± 3.9822.35 ± 2.7822.26 ± 1.3622.22
MedGemma-1.5-4B21.75 ± 3.4822.59 ± 2.4822.56 ± 1.0922.60
Task 5: Hydronephrosis anomaly detectionM3D-Phi-3-4B54.94 ± 4.3054.15 ± 2.6154.51 ± 1.2354.56
InternVL-3.5-8B54.76 ± 4.0654.91 ± 2.5154.65 ± 1.4554.62

(4) The benchmark must support broader model evaluation. The evaluated models represent only a subset of medical MLLMs. Benchmark scale should therefore support fine-grained comparisons of future models, rather than depend solely on current aggregate rankings. Million-instance evaluation sets also exist in general multimodal benchmarks; for example, ANetQA (CVPR 2023) contains approximately 1.50 million test instances [Ref1].

Taken together, these analyses support retaining the full Perception-Bench for fine-grained coverage and stable evaluation.

[Ref1] ANetQA: A Large-Scale Benchmark for Fine-Grained Compositional Reasoning over Untrimmed Videos. CVPR 2023.

I thank the authors for their detailed response. However, I believe several concerns remain insufficiently addressed.

  1. Regarding W1: The authors state that "Our contributions center on problem identification, data curation, and effective training." However, the training approach appears to rely primarily on standard techniques. Given that the main contributions lie in problem identification and dataset construction rather than methodological innovation, I believe this work would be better suited to the Evaluation and Datasets Track.

  2. Regarding W3: I am not convinced by the authors' argument that "Benchmark scale should therefore support fine-grained comparisons of future models, rather than depend solely on current aggregate rankings." The authors should provide concrete experimental evidence demonstrating that the 1.13 million evaluation instances enable meaningful fine-grained comparisons that cannot be achieved with smaller benchmarks.

Finally, I do not understand the relevance of the authors' claim that "ANetQA (CVPR 2023) contains approximately 1.5 million test instances." ANetQA does not provide sufficient evidence to justify the necessity of Perception-Bench's scale.

In light of the authors' response, I have decided to maintain my score.

Reviewer_2X9DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that medical MLLMs fail at diagnosis and report generation because they first fail at low-level visual perception, and proposes a benchmark and a model built on that premise. As contributions, I list: (1) the perception-failure framing, motivated by progressively simplifying report generation to diagnosis to attribute QA; (2) Perception-Bench; (3) RadSight native 2D/3D encoding plus the perception-first curriculum.

Critical strengths

Overall, I find the manuscript well-organized. The figures are generally nice and informative, although the font sizes are too small. Some parts of fig 1 are difficult to read without 200% of zoom (a printed version would be impossible to read). The use of bold combined with small font sizes makes parts of Figs 1 and 3 even more difficult to read.

A large, unified 2D+3D perception-oriented dataset with a documented construction pipeline is valuable. The 8.37M-sample training corpus and 1.13M-instance benchmark span six tasks, including bidirectional concept--region supervision in native 3D, across 12 public sources, and the manuscript documents the two-stage extraction prompts, the attribute schema, and the CT preprocessing (code). This is consequential because it is the contribution that survives my methodological concerns: in the authors' own ablation, simply fine-tuning a generalist Qwen3-VL on the corpus, with CT handled as slices, closes most of the gap to RadSight. Roughly +17 points attribute judgment, +19 AUC, +23 CT report CE-F1 relative to the zero-shot Qwen3-VL-8B in Table 1 (the fine-tuned model's size is unstated). Whatever one concludes about the architecture, the data works, and if released, it is a resource the community would use

The released training code implements the loss, stage-wise freezing, replay, and CT preprocessing exactly as described. The 3D comparisons explicitly mark training-split use, and RadSight-4B outperforms dedicated 3D models under matched adaptation (DeepTumorVQA: 73.40 vs. 59.80 for the best fine-tuned baseline).

The authors adopt macro-AUC with a convincing demonstration of why accuracy can mislead under class imbalance, use clinical-efficacy metrics rather than n-gram overlap, and report results that run counter to their own narrative: very low lesion grounding and diagnostic supervision contribute more than perceptual supervision to CXR report generation. This matters because it indicates the sound measurements can be trusted, and a 4B model that natively handles both radiographs and CT volumes is a practical contribution

Critical weaknesses

The benchmark's validity as a measure of visual perception is not established, and the headline perception results are the most exposed. The ground truth for attribute judgment, disease prediction, and anomaly detection is the Qwen3-Max-parsed report text. The quality check is inter-grader agreement among four LLMs on 500 pairs, with no clinician-validated subset and no reported accuracy rate.

Temporal attributes that cannot be perceived from a single image are among the most frequent entities in the Stage-2 attribute data (fig 9), and the benchmark is built with the same pipeline. The 3D spatial tasks, which supply the advertised "particularly strong gains in spatial grounding," use largely TotalSegmentator-derived organ boxes on anatomy-cropped, fixed-size volumes, so coordinates are stereotyped. 2D/slice-based baselines that cannot localize in a volume still score 18-37% on 3D spatial understanding (tab 1), against roughly 2% for uniform guessing and likely under 10% for a majority-class baseline, and RadSight reaches 96.6%, while no blind/text-only baseline is reported.

Lesion grounding, the capability the thesis actually concerns itself with, achieves only 2.6-8.3 mIoU. This is critical because Perception-Bench is the first contribution, and every downstream claim is measured against it, and I believe that if its perception tasks largely reward anatomical priors and agreement with the labeling pipeline, the paper's central finding is not demonstrated.

Native 3D encoding and the perception-first curriculum are claims not supported by the comparisons presented. Table 1 evaluates baselines zero-shot on a benchmark drawn from RadSight's own training distribution and answer formats. The authors' own Qwen3-VL-Finetune ablation implies that fine-tuning a generalist on the corpus accounts for roughly 80-97% of RadSight's margin over zero-shot Qwen3-VL, leaving 0.6-4.9 points for the architecture. This is a decomposition that the paper never discusses.

The curriculum vs. joint comparison is confounded. Stage 4 upsamples the report data roughly 3x and trains for two epochs, but the total optimizer steps are not reported to match. Moreover, the released 4B scripts give Stage 2 about 8x the optimization reported in Table 9.

Zero-shot transfer is in-distribution transfer on held-out patients. Table 2 apparently compares benchmark-fine-tuned RadSight against as-released baselines (C.3 confirms the former); leakage is unaudited for the RADAR encoder's pretraining data, AMOS-22/BTCV versus the AMOS-MM/AbdomenAtlas training sources, and the 99.4-99.5% RadFig-VQA result.

No seeds or confidence intervals. This is consequential because the paper is framed and titled as a method paper, making a causal claim (perception failures cause clinical failures, and perception-first training fixes them), whereas what the evidence supports is a data contribution. Without a blind baseline, compute-matched ablations, and a leakage audit, the main contributions remain assertions.

Questions for authors

What do a text-only LLM (question and coordinates, no image) and a per-category mean-box prior score on 3D spatial grounding and 3D spatial understanding? If they approach RadSight's 64.7 mIoU / 96.6% accuracy, these tasks are not measuring perception.

The 500-pair audit reports only the grader agreement. What fraction of QA pairs was judged factually accurate? How many benchmark attribute items are temporal (unchanged, stable, prior)? Is any subset clinician-validated?

For each model group: the exact prompts; how each baseline's native coordinate convention (absolute pixels, 0–1000, or [0,1]) and 3D axis order were converted to the benchmark format; how yes/no confidence was extracted when the first generated token is not yes/no; how many slices 2D models received per CT volume; and whether 3D-only models were scored on 2D items. Do all Table 1 columns share the same denominator across model groups?

Fig 4a implies that fine-tuning a generalist model on the corpus accounts for most of RadSight's margin over zero-shot Qwen3-VL, leaving 0.6-4.9 points for the architecture. Do you agree with this reading? What size was the Qwen3-VL-Finetune model, and was it trained with the same curriculum and step count?

What data was the RADAR 3D encoder pretrained on, and does it overlap with any evaluation patients (AbdomenAtlas/DeepTumorVQA, Merlin, AMOS-22, BTCV)? Were AMOS-22 and BTCV volumes excluded from the AMOS-MM and AbdomenAtlas training subsets? How is 99.4-99.5% on RadFig-VQA explained?

C.3 states RadSight was fine-tuned on the public benchmarks' training splits. Were the Table 2 baselines fine-tuned on the same splits? If not, please add RadSight without benchmark fine-tuning or mark the asymmetry

Was joint training matched in total optimizer steps and in report-generation sampling weight (Stage 4 upsamples reports ~3x and runs two epochs)? Which Stage 1-2 configuration produced the reported 4B results, Table 9 (BS 1024, 1 epoch) or the released scripts (BS 512 and 256, 2 epochs)?

Will Perception-Bench, the training corpus, and model weights be released, and under what terms given the PhysioNet, Stanford AIMI, and TCIA licenses? Please state how the data-use agreements were satisfied when report text from credentialed datasets (MIMIC-CXR, MS-CXR, REFLACX) was processed by hosted LLM services

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2X9D (Part 1 of 4)

We thank the reviewer for the careful assessment and constructive comments. We appreciate the recognition of our well-organized presentation, informative figures, unified 2D/3D dataset as a contribution in its own right, publicly released code and faithful implementation, rigorous evaluation, and the practical value of RadSight-4B. We will increase the font sizes in the figures to improve readability. Below, we address the comments point by point.

W1. Full release of data, code, and model weights

R1. Thank you for the question. We agree that the data offers value independent of the methodology through community access. Before submission, we had planned to release Perception-Bench, the training corpus, code, and model weights, consistent with our previous releases on Hugging Face. The code was included with the submission. We apologize for not clearly stating our release plan. With the revision, we will publicly release all redistributable data without access restrictions, the code, and model weights. For datasets requiring credentials, users must obtain access independently and use our LLM preprocessing code to reconstruct the benchmark and training corpus, respecting the source datasets’ access restrictions and licenses. Reports were processed using a locally deployed Qwen3-Max series (Qwen3-235B-A22B) model. Our anonymous project website (https://anonymous.4open.science/r/radsight-anonymous-8900/) now provides training code, data-processing code, and reproduction instructions (the checkpoints exceeded the upload size limit). We will document these release details in the revision.

W2. Expert validation of report-derived questions

R2. Thank you for this valuable suggestion. We randomly sampled 300 QA–report pairs, 100 per task, covering attribute judgment, disease prediction, and anomaly detection. A radiologist assessed faithfulness (support from the source report) and coverage (preservation of all question-relevant information). 96.33% satisfied both criteria (Table R1), supporting the accuracy and completeness of report-derived QA pairs.

Table R1. Radiologist assessment of faithfulness and coverage in report-derived QA pairs.

CriterionPassFailPass rate (%)
Faithfulness2901096.67
Coverage298299.33
Both criteria2891196.33

W3. Temporal attributes.

R3. Thank you for this valuable suggestion. We agree that questions involving “stable,” “progression,” or other attributes requiring prior examinations should be removed. Figure 9 shows entity frequencies, not question-type proportions: terms such as post-operative changes and degenerative changes describe current findings without requiring prior images, so word counts can overestimate temporal questions. Reviewing all 181,744 attribute-judgment QA pairs, we identified 9,877 (5.43%) requiring temporal comparisons. We will remove these pairs and update the evaluation results in the revision.

W4. Blind/text-only baseline for spatial understanding and spatial grounding tasks.

R4. Thank you for this insightful suggestion, which we found highly instructive and valuable for strengthening our analysis.

For spatial understanding, we added coordinate-only, image-mismatch, and per-class mean-box prior experiments, with organ and lesion results reported separately in Table R2. All three perform substantially below RadSight, demonstrating that coordinate priors alone cannot explain its accuracy.

  • Coordinate-only: We trained Qwen-VL using only questions and coordinates for four epochs, versus one epoch for RadSight, yet accuracy remained much lower (36.08 vs. 96.63). To investigate potential optimization issues, we inspected training: the loss decreased from 10.4258 to 0.1493 before plateauing with fluctuations. This suggests ambiguity in mapping coordinates to semantic categories, motivating the mean-box prior experiment.
  • Per-class mean-box prior: Accuracy reaches 65.09% for organs and 14.97% for lesions, indicating that anatomical priors are informative but insufficient, particularly for lesions. This limitation is consistent with substantial anatomy/lesion variation in position, shape, and size across individuals, a widely recognized challenge in medical image segmentation.
  • Image-mismatch: Mismatching image content within queried boxes while preserving questions and coordinates reduces accuracy to 33.16%, confirming RadSight's reliance on visual evidence.

Together, these results show that organ recognition, although relatively easier, still requires visual evidence, while lesion recognition remains more challenging. We will separate these results in the revision and clarify the contributions of coordinate priors and visual evidence.

AuthorsOpenReview ↗

Response to Reviewer 2X9D (Part 2 of 4)

[continued] W4. Blind/text-only baseline for spatial understanding and spatial grounding tasks.

For spatial grounding, the mean-box prior achieves an organ mIoU of 37.44, well below RadSight's 64.73, and a negligible lesion mIoU of 0.014 (Table R3). We will add these results and report organ and lesion performance separately in the revision.

Table R2. 3D spatial understanding performance with visual evidence and coordinate priors.

CategoryCoordinate-onlyImage-mismatchMean-box priorRadSight
Organ36.1233.2165.0996.70
Lesion27.1025.6814.9777.29
Overall36.0833.1664.8996.63

Table R3. Spatial grounding performance.

Grounding targetMean-box priorRadSight-4B
Organ37.4464.73
LesionPancreatic lesion0.0248.30
Kidney lesion0.0017.60
Liver lesion0.0242.60
Lesion average0.0146.17
Overall37.3864.68

W5. Low lesion grounding results.

R5. 3D lesion grounding remains challenging due to diverse lesion morphology and size, limited data and annotations. We did not exclude difficult cases to simplify Perception-Bench, so the low absolute mIoU also reflects its difficulty. Nevertheless, RadSight achieves 6.17 lesion mIoU, substantially exceeding M3D (1.68) and 3D-Rad (1.06), which both trained on CT lesion grounding (Table R4). The mean-box prior scores only 0.014, indicating that anatomical priors alone cannot explain these gains. Our goal is to expose and alleviate fine-grained visual perception limitations in medical MLLMs, rather than claim that lesion grounding is solved. Thank you for prompting this clarification. Perception-Bench measured these limitations, motivating explicit supervision through our perception training corpus. The results demonstrate progress, while lesion grounding remains far from solved. We will clarify this scope and separate organ and lesion results in the revision.

Table R4. Spatial grounding performance comparison.

Grounding targetM3D3D-RadMean-box priorRadSight-4B
Organ27.395.1637.4464.73
LesionPancreatic lesion1.930.850.0248.30
Kidney lesion1.430.250.0017.60
Liver lesion1.672.080.0242.60
Lesion average1.681.060.0146.17
Overall27.365.1537.3864.68

W6. Decomposition of native 3D encoding and the perception-first curriculum.

R6. Thank you for the suggestion. We added a randomly initialized 3D encoder ablation (Table R5). Despite completing less than one epoch due to time and resource constraints, it outperforms the slice-based baseline on Tasks 1–4, suggesting benefits beyond 2D or pretraining. We will update the full one-epoch results in the revision. Figure 4b and Table 5 support the perception-first curriculum. Fine-tuning Qwen3-VL on our corpus accounts for 80%–97% of RadSight’s gains, underscoring our data’s independent value. Qwen3-VL-Finetune uses Qwen3-VL-4B-Instruct and is compared with RadSight-4B under the same staged training curriculum, training data volume (including upsampled examples), and total optimizer update steps. We acknowledge that distribution and answer-format adaptation may explain some gains. However, Tables 3 and 4 show that RadSight outperforms all baselines on external benchmarks under matched fine-tuning or zero-shot settings, indicating benefits beyond such adaptation.

Table R5. Effects of native 3D encoding. Results without 3D encoder pretraining are preliminary (<1 epoch) due to time and resource constraints and will be updated after one full epoch.

ModelTask1Task2Task3Task4Task5Task6Average
Qwen3-VL-finetune51.8346.3569.8253.1476.2132.8055.03
RadSight w/o 3D encoder pretraining52.0347.4072.2455.0274.2830.8255.30
RadSight52.4549.4572.4456.1681.1036.4558.01

W7. Training budget matching and configuration consistency.

R7. Thank you for highlighting these issues. Joint and curriculum training use identical total optimizer update steps and sample exposures, including upsampling and repeated training in Stages 3–4. Report upsampling also applies to joint training, so curriculum training receives no additional budget. We will clarify this in the revision. The reported results follow Table 9; some released scripts inadvertently retained exploratory configurations. We apologize for this oversight and will correct the scripts to match Table 9.

AuthorsOpenReview ↗

Response to Reviewer 2X9D (Part 3 of 4)

W8. Evaluation settings and potential data leakage?

R8. Thank you for your comments. Table 2 is not a zero-shot evaluation and was not presented as such, we will clarify its training and evaluation settings. RADAR was pretrained on private data from a single Chinese hospital, with no leakage involving our evaluation data. We confirmed shared data sources between AMOS-22/BTCV and the AMOS-MM/AbdomenAtlas training sets. This affects only Appendix Table 12, which we will remove.

The cited 99.4%–99.5% likely refers to OmniMedVQA’s 99.39%/99.51% in Table 2. These are internal test results after training on the corresponding training set, not zero-shot results. Recent work reports 99.5% on this testset (MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants, NeurIPS2025), suggesting relatively low difficulty. We believe these high scores reflect task difficulty rather than data leakage.

W9. Seeds and confidence intervals.

R9. Thank you for the suggestion. We report 95% confidence intervals for Table 1 using 10,000 bootstrap resamples (Table R6). All experiments used fixed seeds. Given current time and compute constraints, we will add multi-seed experiments and 95% confidence intervals for all evaluation results in the revision.

Table R6. Performance on Perception-Bench with 95% bootstrap confidence intervals.

TaskTask1Task2-2DTask2-3DTask3-2DTask3-3DTask4Task5Task6
Result52.4534.1964.6848.2596.6356.1681.1036.45
95% CI52.22–52.6833.34–35.0764.50–64.8546.63–49.8796.46–96.7855.86–56.4680.63–81.5734.91–37.91

W10. Baseline implementation details and evaluation comparability

R10. Thank you for highlighting these details. We follow each model's original configuration where available; otherwise, we maximize input information within GPU memory limits while maintaining fair comparisons. Experiments use H20 GPUs with 96 GB of memory, with selected 2D models receiving 80 slices per CT volume. We clarify the settings below and will document them in the revised manuscript.

(1) Prompts

All models receive identical benchmark questions through their official chat templates, visual placeholders, and processors.

(2) Model-specific input processing

ModelCT preprocessingSlice count / volume organization
Qwen3-VL-8BRemove all-black slices at both volume ends and use the official visual processor.80 slices per volume.
InternVL3.5-8BRemove all-black slices at both volume ends and use the official preprocessing pipeline and input template.80 slices per volume.
Lingshu-7BUse the official Qwen2.5-VL processor and visual input interface.80 slices per volume.
HuluMed-7BUse the official NIfTI pathway with type="3d"; the processor handles slice extraction and preprocessing.The official example uses nii_axis=2 and nii_num_slices=180.
MedGemma-1.5-4BFollow the official CT pipeline: HU processing, three-window RGB encoding, and indexed image sequences.The official example sets MAX_SLICE=85, uniformly sampling longer volumes and retaining all slices otherwise.
OmniCT-7BFollow the official volume-loading and slice-organization pipeline.Sample 10 positions, combining the preceding, current, and following slices into an RGB image at each position.
M3D-Phi-3-4B / 3D-Rad-Phi-3-4BUse volumetric tensors following the official 3D preprocessing pipeline.M3D uses C×D×H×W = 1×32×256×256; 3D-Rad follows the corresponding M3D pipeline.

For MedGemma, each slice's HU windows are encoded as R: [-1024,1024], G: [-135,215], and B: [0,80]. Channels are clipped, mapped to [0,255], and converted to uint8. Each image is followed by a SLICE i marker, and the task question follows the image sequence. Further processing uses the official processor.

(3) Coordinate formats and axis order

RadSight uses coordinates normalized to [0,1]: [x_min, y_min, x_max, y_max] for 2D boxes and [x_min, y_min, z_min, x_max, y_max, z_max] for 3D boxes, with x, y, and z corresponding to width, height, and depth.

For 2D localization, prompts request only normalized boxes in this format, separated by commas when multiple regions exist. MedGemma's native [y_min, x_min, y_max, x_max] output is converted to the benchmark's axis order and coordinate range. M3D and 3D-Rad outputs are similarly converted from their training conventions to the benchmark's 3D format. Localization metrics for all models are computed after aligning coordinate ranges and axis order to avoid bias from differing native conventions.

Reviewer_E9LXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Perception-Bench for evaluating medical MLLMs on six tasks involving 2D radiographs and 3D CT scans. It also proposes RadSight, which combines separate 2D/3D encoders with a four-stage training pipeline: alignment, perception, diagnosis, and report generation. The model is trained on 8.37M samples and evaluated on both Perception-Bench and public benchmarks.

Critical strengths

The dataset construction and coverage of both 2D and 3D tasks represent a substantial effort. I also appreciate that the authors include comparisons against same-corpus Qwen3-VL fine-tuning and joint mixed training, rather than relying only on comparisons with off-the-shelf models. The ablations show that adding perceptual supervision helps several downstream tasks, particularly CT report generation.

Critical weaknesses
  1. I am not convinced that the results establish the paper’s main claim about fine-grained visual perception. Showing that a model fails at report generation, diagnosis, and attribute questions does not by itself show that the first two failures are caused by the third. These tasks could share other limitations, such as inadequate task-specific training or weak vision-language alignment. The training results suggest that perceptual supervision is useful, but the stronger explanation of why it helps remains insufficiently tested. I am particularly concerned about the spatial-understanding task. The question includes the normalized bounding-box coordinates, which may already provide considerable information about the organ through its location and shape. How much of the 96.63% accuracy requires looking at the image? A coordinate-only baseline and an image-mismatch test would help answer this. Separating organ recognition from lesion recognition would also make this result easier to interpret. The lesion-grounding results are much less encouraging than the aggregate score: Table 10 reports mIoU of 8.30, 7.60, and 2.60 for pancreatic, kidney, and liver lesions, compared with 64.68 for overall 3D grounding. The authors acknowledge this distinction, but it substantially limits what the overall result tells us about lesion perception. On zero-shot DeepTumorVQA, RadSight also falls short of the best baseline on recognition and measurement; its strongest advantage is in medical reasoning. I therefore find the evidence for improved downstream performance stronger than the evidence for the claimed improvement in fine-grained lesion perception.

  2. The contribution beyond additional supervision and existing pretrained components is still unclear. The architecture and training objective are fairly standard, so much of the methodological contribution rests on the curriculum. However, removing the perception stage also removes 4.56M training examples. The joint-training comparison is helpful, but I could not determine whether it matches the curriculum in total updates, task exposure, replay, and report-generation upsampling. Without these details, it is difficult to tell whether the gain comes from the proposed ordering or from differences in training allocation. An alternative stage order under the same budget would be a more direct test. The encoder comparison has a similar issue. RadSight uses a segmentation-pretrained 3D encoder, whereas Qwen3-VL-Finetune processes slices. This changes both the input representation and the pretrained knowledge, making it difficult to attribute the improvement specifically to native 3D encoding. There is also some ambiguity in the public-benchmark comparisons. Appendix C.3 describes joint fine-tuning on the benchmark training sets, but Table 2 does not clearly state the adaptation setting for each baseline. I would like to see the main comparisons made under matched supervision before drawing conclusions about the advantage of the proposed design.

Questions for authors

How well does a coordinate-only baseline perform on spatial understanding? Does performance drop when the image is replaced with a case containing different content in the queried region?

Are the curriculum and joint-training baselines matched in total updates and per-task exposure, including replay and upsampling? Have you tested a different stage order under the same training budget?

Which models in Table 2 were fine-tuned on the corresponding benchmark training sets? Were they given the same adaptation data as RadSight?

Have experts checked whether the report-derived attribute questions are unambiguously answerable from the actual input images, particularly for small or multiple lesions? Text-only judge agreement does not resolve this concern.

How much of the improvement over Qwen3-VL-Finetune comes from the segmentation-pretrained encoder rather than the 3D representation itself? Please also clarify the slice-sampling and visual-token budgets used in this comparison.

Can you report 3D spatial-understanding results separately for organs and lesions? Given the low lesion-grounding scores in Table 10, this breakdown would help clarify what the aggregate 96.63% accuracy tells us about lesion perception.

5 comments
AuthorsOpenReview ↗

Response to Reviewer E9LX (Part 1 of 3)

We thank the reviewer for the careful assessment and constructive comments. We appreciate the recognition of our dataset construction and 2D/3D coverage, same-corpus and joint-training comparisons, and the downstream benefits of perceptual supervision. Below, we provide point-by-point responses to the concerns raised.

W1.1. Evidence for perceptual deficiencies and why perceptual supervision helps.

R1.1. Thank you for this comment. Our intended interpretation is not that attribute-QA failures cause diagnosis and report-generation failures. Rather, task simplification reveals a shared perceptual deficiency: models struggle with basic lesion attributes even when reasoning and generation demands are reduced. This suggests that inadequate fine-grained visual perception may underlie failures across all three tasks. Fine-grained perception also supports reliable vision–language alignment, as relevant visual features must be captured before they can be effectively aligned with language. We will revise Figure 1 to illustrate this point more clearly. Perceptual supervision turns report-derived attributes, such as location, size, and density, along with spatial correspondences, into explicit learning targets, strengthening alignment between low-level visual evidence and clinical semantics. Table 5 supports this explanation: after Stage 1 alignment, report-generation training alone yields 15.90 CT-RATE F1, while adding perceptual supervision increases it to 26.21 (+10.31). Thus, our argument draws on both failure analysis and training interventions demonstrating benefits beyond task-specific supervision.

W1.2. Spatial understanding task.

R1.2. We thank the reviewer for this insightful suggestion, which we found highly instructive and valuable for strengthening our analysis. We added coordinate-only, image-mismatch, and per-class mean-box prior experiments, with organ and lesion results reported separately in Table R1. All three perform substantially below RadSight, demonstrating that coordinate priors alone cannot explain its accuracy.

  • Coordinate-only: We trained Qwen3-VL using only questions and coordinates for four epochs, versus one epoch for RadSight, yet accuracy remained much lower (36.08 vs. 96.63). To investigate potential optimization issues, we inspected training: the loss decreased from 10.4258 to 0.1493 before plateauing with fluctuations. This suggests ambiguity in mapping coordinates to semantic categories, motivating the mean-box prior experiment.
  • Per-class mean-box prior: Accuracy reaches 65.09% for organs and 14.97% for lesions, indicating that anatomical priors are informative but insufficient, particularly for lesions. This limitation is consistent with substantial anatomy/lesion variation in position, shape, and size across individuals, a widely recognized challenge in medical image segmentation.
  • Image-mismatch: Mismatching image content within queried boxes while preserving questions and coordinates reduces accuracy to 33.16%, confirming RadSight's reliance on visual evidence.

Together, these results show that organ recognition, although relatively easier, still requires visual evidence, while lesion recognition remains more challenging. We will separate these results in the revision and clarify the contributions of coordinate priors and visual evidence.

Table R1. 3D spatial understanding performance with visual evidence and coordinate priors.

CategoryCoordinate-onlyImage-mismatchMean-box priorRadSight
Organ36.1233.2165.0996.70
Lesion27.1025.6814.9777.29
Overall36.0833.1664.8996.63

W1.3. Lesion grounding and zero-shot results.

R1.3. 3D lesion grounding remains challenging due to diverse lesion morphology and size, limited data and annotations. We did not exclude difficult cases to simplify Perception-Bench, so the low absolute mIoU also reflects its difficulty. Nevertheless, RadSight achieves 6.17 lesion mIoU, substantially exceeding M3D (1.68) and 3D-Rad (1.06), which both trained on CT lesion grounding (Table R2). Our goal is to expose and alleviate fine-grained visual perception limitations in medical MLLMs, rather than claim that lesion grounding is solved. The results demonstrate progress, while lesion grounding remains far from closed. We will clarify this scope and separate organ and lesion results in the revision.

Table R2. Spatial grounding performance.

Grounding targetM3D3D-RadRadSight-4B
Organ27.395.1664.73
LesionPancreatic lesion1.930.858.30
Kidney lesion1.430.257.60
Liver lesion1.672.082.60
Lesion average1.681.066.17
Overall27.365.1564.68
AuthorsOpenReview ↗

Response to Reviewer E9LX (Part 2 of 3)

[continued] W1.3. Lesion grounding and zero-shot results.

Our further analysis of zero-shot DeepTumorVQA shows:

  • Recognition: RadSight achieves substantially higher positive-class accuracy with slightly lower negative-class accuracy, indicating greater sensitivity to visual abnormalities (Table R3). Because negative samples dominate the test set, overall accuracy underrepresents this advantage. We therefore additionally report Balanced Accuracy and AUC, on both of which RadSight significantly outperforms the baselines.
  • Measurement: RadSight trails Lingshu but outperforms all other compared methods.
  • Reasoning: Unlike text-only reasoning, these tasks rely on recognizing abnormalities and measuring their attributes in images. RadSight’s reasoning gains therefore provide complementary evidence of its fine-grained visual perception capabilities.

Together, these results demonstrate improvements in perceptual tasks, including lesion localization and abnormality recognition, alongside visual perception-based reasoning.

Table R3. Recognition performance on DeepTumorVQA.

ModelOverall accuracyYes-class accuracyNo-class accuracyBalanced accuracyAUC
Lingshu58.4831.6473.6352.6459.68
HuLuMed60.0631.5176.1853.8460.86
RadSight-4B59.3249.5964.8157.2088.86

W2.1. Curriculum effectiveness under matched training budgets.

R2.1. We thank the reviewer for this comment. Table 5 evaluates the benefits of adding perceptual supervision, while Figure 17 tests the curriculum against joint training. The joint baseline randomly mixes all task samples, matching the curriculum in total updates, cumulative per-task exposure, replay, and report-generation upsampling. Thus, the comparison controls data volume and training budget while varying task ordering. The consistent gains in Figure 17 support the effectiveness of progressing from perception to diagnosis and report generation over random mixed training. We will clarify these in the revision.

W2.2. Ablation of native 3D encoding.

R2.2. We added a randomly initialized 3D encoder ablation (Table R4). Despite completing less than one epoch due to time and resource constraints, it outperforms the slice-based baseline on Tasks 1–4, suggesting benefits beyond 2D or pretraining. We will update the full one-epoch results in the revision.

Table R4. Effects of native 3D encoding. Results without 3D encoder pretraining are preliminary (<1 epoch) due to time and resource constraints and will be updated after one full epoch.

ModelTask1Task2Task3Task4Task5Task6Average
Qwen3-VL-finetune51.8346.3569.8253.1476.2132.8055.03
RadSight w/o 3D encoder pretraining52.0347.4072.2455.0274.2830.8255.30
RadSight52.4549.4572.4456.1681.1036.4558.01

W2.3. Supervision settings and fairness of public-benchmark comparisons in Table 2.

R2.3. We thank the reviewer for this comment. We now specifies each model’s supervision data in Table R5, all baselines use publicly released checkpoints. We found that target-benchmark training exposure alone does not determine performance: Lingshu and HuluMed outperform OmniCT on OmniMedVQA and RadFig-VQA without training on either. Among the listed benchmarks, RadSight uses only PMC-VQA additionally compared with OmniCT, yet RadSight leads across all subtasks. We will clarify these supervision differences in the revision.

Table R5. Supervision settings of public-benchmark comparisons.

ModelSLAKEPMC-VQAOmniMedVQARadFig-VQA
RadFM-14B✓✓——
Lingshu-7B✓✓——
Hulu-Med-7B✓✓——
MedGemma-1.5-4B✓———
OmniCT-7B✓—✓✓
RadSight✓✓✓✓
AuthorsOpenReview ↗

Response to Reviewer E9LX (Part 3 of 3)

Q. Expert validation of report-derived questions.

R. Thank you for this valuable suggestion. Source reports are grounded in imaging examinations and undergo two-stage physician review before release, providing a clinical basis for QA annotation. We randomly sampled 300 QA–report pairs, 100 per task, covering attribute judgment, disease prediction, and anomaly detection, including small and multiple lesions. A radiologist assessed faithfulness (support from the source report) and coverage (preservation of all question-relevant information). 96.33% satisfied both criteria (Table R6), supporting the accuracy and completeness of report-derived QA pairs.

Table R6. Radiologist assessment of faithfulness and coverage in report-derived QA pairs.

CriterionPassFailPass rate (%)
Faithfulness2901096.67
Coverage298299.33
Both criteria2891196.33

Comments on the paper

1 comment

Dear Reviewers,

Please take a moment to read the authors’ rebuttal and participate in the discussion.

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.