acceptodds
Under review as a conference paper at ICLR 2027

Unifying Medical Vision-Language Learning Across Imaging Modalities in Vietnamese: A Multimodal Dataset and Benchmark

Abstract

Medical report generation has advanced rapidly with Vision-Language Models (VLMs), yet progress remains constrained by the scarcity of high-quality paired medical image-report data, particularly for low-resource languages. Existing datasets are largely modality-specific, covering common modalities such as X-ray, CT, or MRI individually rather than jointly, while PET/CT remains underrepresented. Moreover, many datasets are assembled from figure screenshots or re-captured images rather than the native DICOM studies acquired in routine clinical practice, limiting their clinical fidelity and practical value. To address these limitations, we introduce **ViMed-Rad**, a large-scale multimodal radiology dataset jointly covering four major imaging modalities: X-ray, CT, MRI, and PET/CT. ViMed-Rad comprises 93,781 clinical studies paired with Vietnamese reports authored by radiologists, including 62,148 X-ray, 13,247 CT, 12,010 MRI, and 6,376 whole-body PET/CT studies. All studies are native DICOM acquisitions exported from hospital PACS archives, retaining acquisition metadata and spanning diverse anatomical regions and imaging protocols. Building on ViMed-Rad, we establish **ViMed-Bench**, a unified benchmark that couples clinical report generation and report-grounded VQA tasks with a multimodal framework integrating modality-specific visual representations into a shared language generation space, so that a single model serves all four modalities. Benchmarking existing medical VLMs reveals substantial performance gaps across imaging types, particularly for underrepresented modalities, together with a consistent degradation when reports are generated in Vietnamese. Experiments on ViMed-Rad show that the proposed framework accommodates heterogeneous planar, volumetric, and functional-anatomical imaging within one model, outperforming baselines across all four modalities while retaining most of the accuracy of modality-specific counterparts, and providing a scalable foundation for multi-modality medical VLMs rather than separate models for each imaging domain.

Primary area:
datasets and benchmarks
Keywords:
Medical Vision-Language Models, Medical Report Generation, Multimodal Medical Imaging, Vietnamese Medical Dataset, Medical Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uiPPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ViMed-Rad, a Vietnamese-language medical imaging dataset comprising 93,781 clinical studies across X-ray, CT, MRI, and whole-body PET/CT, paired with radiology reports. Building on this dataset, the authors develop ViMed-Bench, which combines report generation and report-grounded visual question answering (VQA), and propose a unified vision-language framework with modality-specific visual encoders and a shared language model. The study also evaluates existing medical vision-language models (VLMs) in Vietnamese and investigates the effect of joint multimodal training and additional translated public data.

Critical strengths

The work addresses an important problem: the scarcity of native-language medical image-report datasets for low-resource languages, particularly for volumetric and functional-anatomical imaging. The inclusion of native clinical imaging and radiologist-authored reports across four modalities is potentially valuable. The collection of native DICOM studies paired with Vietnamese radiology reports addresses a meaningful gap in medical vision-language research. Including CT, MRI, and PET/CT alongside X-ray broadens the scope beyond conventional chest radiography datasets.

Critical weaknesses

Major concern.

  1. The dataset is a central contribution of this work. However, the reproducibility statement explicitly indicates an intention to release the implementation and reproduction instructions upon publication, while the manuscript does not provide a comparably clear, verifiable commitment regarding public access to ViMed-Rad itself. Whether ViMed-Rad will be publicly released, including the native imaging data, corresponding reports, and derived VQA annotations, or whether controlled access, a data-use agreement, or an evaluation server will be provided. This distinction is particularly important because the dataset and benchmark appear to be the primary contributions. If the core dataset remains inaccessible and no practical alternative enables independent evaluation, the reproducibility and broader community impact of the work would be substantially reduced.

  2. Table 1 compares datasets across modalities, formats, sample counts, text supervision, language, and data sources. However, the current presentation makes it difficult to compare the actual scale and dimensionality of the imaging data, particularly for volumetric modalities. I recommend adding the following information for each relevant dataset and modality: Number of studies and unique patients. Number of 3D volumes, where applicable. Total number of slices and average slices per volume, where applicable.

For example, a dataset containing thousands of 3D CT volumes is not directly comparable to a dataset containing a similar number of selected 2D slices. The authors should also clearly distinguish the size of the full ViMed-Rad collection from the smaller modality-specific subsets actually used in the experiments. This is particularly important for PET/CT, where the full dataset contains 6,376 scans but the experimental subset and region-level report counts differ.

The related-work discussion covers several established resources. However, it appears to omit several highly relevant works that should be discussed to better establish the novelty and positioning of ViMed-Rad as follows.

[1] Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench. [2] MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT [3] PETWB-REP: A Multi-Cancer Whole-Body FDG PET/CT Dataset with Corresponding Radiology Reports [4] Development of a large-scale grounded vision language dataset for chest CT analysis

  1. The manuscript describes CT, MRI, and PET/CT as volumetric data and uses modality-specific encoders. However, the evaluation includes baselines with different capabilities and input representations. Some models directly encode 3D volumes, whereas others receive selected axial slices or grids of slices. The authors should report, for every evaluated model and modality: Whether the model receives a native 3D volume, individual 2D slices, a 2D slice grid, or a combination of these representations. The number of slices used, the slice-selection strategy, and the anatomical coverage. For PET/CT, whether the input contains PET, CT, fused PET/CT, or additional projections such as maximum-intensity projections.

  2. The paper demonstrates that existing medical VLMs can perform substantially worse when evaluated against Vietnamese references than against English references. However, the evaluation does not appear to include the important alternative of generating an English report first and then translating it into Vietnamese using a strong translation system or LLM.

I recommend adding the following comparisons on the same test images:English report generation followed by machine translation into Vietnamese. English report generation followed by translation using a representative LLM, such as GPT. This experiment would help determine whether the observed language gap primarily reflects limitations in visual understanding, target-language generation, or the translation process. It would also clarify the practical benefit of training directly on native Vietnamese clinical reports.

  1. The manuscript acknowledges that the data originate from a limited number of centres and that PET/CT comes from a different source than the other modalities. This raises a potential confounding issue: observed modality-specific differences may also reflect differences in institutions, scanners, acquisition protocols, patient populations, or reporting conventions.

The authors should clarify the provenance of each modality and report whether the data splits prevent patient overlap across training, validation, and test sets. They should also provide more detailed information on acquisition protocols and, where possible, evaluate performance across distinct centres or acquisition settings. For PET/CT in particular, generalization across tracers, disease types, and acquisition protocols would be important. If external validation is not available, the manuscript should explicitly distinguish internal test performance from evidence of cross-centre generalization.

Minor Comments

Please use consistent terminology to distinguish a study, patient, volume, image, slice, and region-level report throughout the manuscript.

Please report the number of unique patients alongside the number of studies for each modality, and clearly explain the PET/CT conversion from whole-body scans to region-level report-generation and VQA examples.

Please ensure that all reported comparisons use clearly specified validation and test splits, and provide confidence intervals or statistical significance analyses where appropriate.

Please clarify which public datasets contribute to each training configuration and whether translated reports are used for training, validation, testing, or multiple purposes.

Please provide a clear data and code availability statement, including repository links, licenses, access restrictions, and the intended release timeline.

Questions for authors

I appreciate the importance of addressing the Vietnamese-language gap in medical vision-language research, and I agree that this work has the potential to make a valuable contribution by providing a multimodal dataset with native clinical reports and a benchmark spanning multiple imaging modalities, including volumetric PET/CT. However, several important questions remain regarding the accessibility and reproducibility of the dataset, its positioning relative to closely related recent resources, the fairness and comparability of the 2D/3D evaluation protocols, and the evidence supporting the benefits of the proposed unified framework. I encourage the authors to address these concerns and clarify the corresponding methodological details and experimental evidence. My current assessment is Weak Accept, reflecting the potential value of the work, but this score is tentative. I will reconsider my rating after evaluating the authors’ responses, depending on how satisfactorily these concerns are resolved.

7 comments
AuthorsOpenReview ↗

Response to Reviewer uiPP (Part 1)

We thank the Reviewer for the careful and constructive review, and for recognizing the value of native Vietnamese image–report data across four modalities, including volumetric PET/CT. We address each point below; the corresponding changes are included in the revised manuscript.


W1. The dataset is a central contribution of this work. However, the reproducibility statement explicitly indicates an intention to release the implementation and reproduction instructions upon publication, while the manuscript does not provide a comparably clear, verifiable commitment regarding public access to ViMed-Rad itself. Whether ViMed-Rad will be publicly released, including the native imaging data, corresponding reports, and derived VQA annotations, or whether controlled access, a data-use agreement, or an evaluation server will be provided. This distinction is particularly important because the dataset and benchmark appear to be the primary contributions. If the core dataset remains inaccessible and no practical alternative enables independent evaluation, the reproducibility and broader community impact of the work would be substantially reduced.

We thank the Reviewer for raising this important point. The complete dataset will be released upon acceptance, including all de-identified native DICOM studies, the radiologist-authored Vietnamese reports, the VQA annotations, the data splits, and the preprocessing, training and evaluation code. It will be hosted on Hugging Face as a gated repository under the CC BY-NC-SA 4.0 licence with a data use agreement: users submit a short access form and agree to non-commercial research use, no re-identification and no redistribution. During the discussion period, a de-identified sample is already available at https://anonymous.4open.science/r/anonymous_project-42BD/README.md: all 37,369 chest X-ray studies of the benchmark subset, and 140 studies each of CT, MRI and PET/CT (100 training, 20 validation, 20 test), with their reports, VQA items and splits.

We have revised the Reproducibility Statement accordingly: it now states the content of the release, the licence, the access procedure, the timeline and the link to the sample.

AuthorsOpenReview ↗

Response to Reviewer uiPP (Part 2)

W2. Table 1 compares datasets across modalities, formats, sample counts, text supervision, language, and data sources. However, the current presentation makes it difficult to compare the actual scale and dimensionality of the imaging data, particularly for volumetric modalities. I recommend adding the following information for each relevant dataset and modality: Number of studies and unique patients. Number of 3D volumes, where applicable. Total number of slices and average slices per volume, where applicable.

For example, a dataset containing thousands of 3D CT volumes is not directly comparable to a dataset containing a similar number of selected 2D slices. The authors should also clearly distinguish the size of the full ViMed-Rad collection from the smaller modality-specific subsets actually used in the experiments. This is particularly important for PET/CT, where the full dataset contains 6,376 scans but the experimental subset and region-level report counts differ.

Thank you for this suggestion. In the revised manuscript, Table 1 reports the number of samples, unique patients and slices of each dataset, and Table 3 gives, for each ViMed-Rad modality of the full cohort, the studies, unique patients, images or volumes per study, total slices and slices per volume. Figure 4 and Table A1 separate the full collection from the subset used in the experiments: they trace each modality from the full cohort to the benchmark splits and report unique patients, studies and report-generation targets per split. For PET/CT, they distinguish the 6,376 scans of the full cohort from the 2,757 scans of the benchmark subset, which carry 8,271 region-level reports. For the Reviewer's convenience, the table below consolidates these numbers and adds the number of volumes and slices of the benchmark subset.

Dataset scale (consolidated from Tables 1, 3 and A1 of the revised manuscript). Comparison of medical imaging-text datasets. 2D/3D: planar/volumetric imaging; Studies: imaging examinations (scans for CT-RATE and ViMed-PET, cases for RadGenome-Brain MRI, captioned cases for M3D-Cap); Patients: unique patients; Images / vol.: 2D images, or 3D volumes for 3D modalities (DICOM series for ViMed-Rad MRI); Slices: total 2D slices over all volumes, with the mean per volume in parentheses. n.r.: not reported by the original publication (PMC-OA: 1.65M subfigures cropped from 381K figures of PubMed Central articles; OmniMedVQA: 118K images pooled from 73 public datasets; MedMD: 2D and 3D scans pooled from web and public sources). For ViMed-Rad, the full cohort and the benchmark subset used in all experiments are listed separately; patients are counted within each modality.

DatasetX-rayCTMRIPET/CTFormatStudiesPatientsImages / vol.SlicesText TypeLang.Source
MIMIC-CXR2D–––DICOM227,83565,379377,110–ReportENPhysioNet
CT-RATE–3D––NIfTI25,69221,30450,188>14.3MReportENCT-RATE
RadGenome-Brain MRI––3D–NIfTI1,007n.r.3,408n.r.ReportENPublic MRI
PMC-OA2D2D2D2DJPEGn.r.n.r.1.65M–CaptionENPMC OA
OmniMedVQA2D2D2D–PNGn.r.n.r.118K–VQAENWebsites
M3D-Data–3D––Numpy42,496n.r.120Kn.r.Text & VQAENWebsites
MedMD2D3D3D–Numpyn.r.n.r.16Mn.r.Text & VQAENWebsites
ViX-Ray2D–––PNGn.r.n.r.5,400–ReportVIHospital
ViMed-PET–––3DDICOM2,7572,1825,5141.57M (284)Report & VQAVIHospital
ViMed-Rad (Ours), full cohort
X-ray2D–––DICOM62,14854,07789,514–ReportVIHospital
CT–3D––DICOM13,24711,83068,58119.59M (286)ReportVIHospital
MRI––3D–DICOM12,01011,410155,5675.59M (36)ReportVIHospital
PET/CT–––3DDICOM6,3766,30912,7523.58M (280)ReportVIHospital
Total93,781
ViMed-Rad (Ours), experimental subset
X-ray2D–––DICOM37,36934,57040,242–Report & VQAVIHospital
CT‡–3D––DICOM5,4285,02326,2678.02M (305)Report & VQAVIHospital
MRI––3D–DICOM3,6493,64862,4072.68M (43)Report & VQAVIHospital
PET/CT–––3DDICOM2,7572,1825,5141.57M (284)Report & VQAVIHospital
AuthorsOpenReview ↗

Response to Reviewer uiPP (Part 3)

W3. The related-work discussion covers several established resources. However, it appears to omit several highly relevant works that should be discussed to better establish the novelty and positioning of ViMed-Rad as follows.

[1] Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench. [2] MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT [3] PETWB-REP: A Multi-Cancer Whole-Body FDG PET/CT Dataset with Corresponding Radiology Reports [4] Development of a large-scale grounded vision language dataset for chest CT analysis

Thank you for this valuable suggestion. We have revised the Related Work section to include and discuss all four references: PET-Bench, MetaStructAtlas, PETWB-REP and RadGenome-Chest CT. These resources add spatial or metabolic supervision, but each covers a single modality in English: RadGenome-Chest CT grounds chest CT reports in segmented regions, MetaStructAtlas grounds whole-body PET/CT findings in organs, PET-Bench probes PET perception through hierarchical VQA, and PETWB-REP pairs 490 whole-body PET/CT scans with reports. In contrast, ViMed-Rad pairs native Vietnamese reports across four modalities with report-grounded VQA tasks of increasing difficulty. This discussion is now available in the Related Work section (Section 2) of the revised manuscript.


W4. The manuscript describes CT, MRI, and PET/CT as volumetric data and uses modality-specific encoders. However, the evaluation includes baselines with different capabilities and input representations. Some models directly encode 3D volumes, whereas others receive selected axial slices or grids of slices. The authors should report, for every evaluated model and modality: Whether the model receives a native 3D volume, individual 2D slices, a 2D slice grid, or a combination of these representations. The number of slices used, the slice-selection strategy, and the anatomical coverage. For PET/CT, whether the input contains PET, CT, fused PET/CT, or additional projections such as maximum-intensity projections.

We thank the Reviewer for this suggestion. We have added Table A4 (Appendix D) to the revised manuscript, which summarizes the native image input of each zero-shot baseline as specified in its original publication:

Table A4. Native image input of the zero-shot baselines, as specified in their original publications.

ModelNative inputX-rayCTMRI
RadFM2D and 3D; 3D ViT512×512, depth 4256×256×D, D ≤ 64256×256×D, D ≤ 64
LLaVA-MedSingle 2D image; CLIP ViTSingle image––
MedVersa2D (Swin-B) and 3D (3D U-Net); 224×224Multi-view, 224×224Seg. only†Seg. only†
M3D3D CT; 3D ViT–32×256×256–
MedGemma 1.52D, 896×896; volumes as 2D slicesFrontal (+ prior)≤ 85 axial slices≤ 85 axial slices

–: not supported. †3D encoder trained for segmentation only; no CT/MRI report generation.

Building on this table, Appendix D details the input actually given to each model. RadFM and M3D encode the CT and MRI volume directly. Models without native volumetric support receive evenly spaced axial slices: LLaVA-Med as a 4×4 grid of 16 slices, MedVersa as 16 separate slices, and GPT-4o and DeepSeek-v4.1 as 4×4 grids, one per available MRI sequence. For the few-shot baselines, CT and MRI (FLAIR) are tiled into one 5×5 grid of 25 slices. None of the baselines supports PET/CT natively; they receive PET slices as eight 5×5 grids (200 slices per region) in the few-shot setting, and 4×4 grids plus a coronal maximum-intensity projection in the zero-shot setting (GPT-4o and DeepSeek-v4.1). X-ray images are provided as acquired.

Reviewer_HcbHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces ViMed-Rad, a Vietnamese image–report dataset covering X-ray, CT, MRI, and PET/CT, alongside benchmarks for report generation and question answering.

Critical strengths

The work addresses medical imaging understanding in a low-resource language and provides a valuable addition to Vietnamese clinical data resources.

Critical weaknesses
  1. Unclear data availability. Whether the dataset will be publicly released remains unclear, limiting assessment of its accessibility and reproducibility.
  2. Concerns about comparison fairness. The proposed model is fine-tuned on local data, whereas major baselines use zero-shot or few-shot settings. Differences in imaging inputs further limit attribution of the performance gains to the method itself.
  3. Limited benefits of unification. The joint models achieve lower clinical F1 than modality-specific models on X-ray, CT, and MRI, while additional public data do not yield consistent improvements.
Questions for authors
  1. Will the complete DICOM data, reports, and data splits be released? Please clarify access procedures, licensing conditions, and the release timeline.
  2. Given its primary contribution, might this work be better suited to a dataset track?
8 comments
AuthorsOpenReview ↗

Response to Reviewer HcbH (Part 1)

We thank the Reviewer for recognizing the value of ViMed-Rad as a Vietnamese clinical data resource. We address each concern below.


W1. Unclear data availability. Whether the dataset will be publicly released remains unclear, limiting assessment of its accessibility and reproducibility.

The complete dataset will be released upon acceptance and made publicly available before the ICLR 2027 main conference (April 26–28, 2027). The release includes:

  • all de-identified native DICOM studies of the four modalities (93,781 studies), including those not used in our experiments;
  • the radiologist-authored Vietnamese reports, with the normalized CT reports;
  • the VQA-Select, VQA-Respond and VQA-Compose annotations;
  • the train/validation/test splits of every task, the list of selected series per study, and the preprocessing, training and evaluation code with the exact configurations used in the paper.

Access and licence. The dataset will be hosted on Hugging Face as a gated repository under the CC BY-NC-SA 4.0 licence, together with a data use agreement. To obtain access, users submit a short form (name, affiliation, intended research use) and agree to use the data for non-commercial research only, not to attempt re-identification of any patient, and not to redistribute the original data; requests are reviewed by the authors.

Sample for review. To allow Reviewers to inspect the data during the discussion period, we have already released a de-identified sample with its reports, VQA items and splits: all 37,369 chest X-ray studies of the benchmark subset, and 140 studies each of CT, MRI and PET/CT (100 training, 20 validation, 20 test). It is available at the following anonymous link:

https://anonymous.4open.science/r/anonymous_project-42BD/README.md

The samples follow the same format, de-identification and licence as the full release. We have revised the Reproducibility Statement accordingly: it now states the content of the release, the licence, the access procedure, the timeline and the link to the sample.

AuthorsOpenReview ↗

Response to Reviewer HcbH (Part 2)

W2. Concerns about comparison fairness. The proposed model is fine-tuned on local data, whereas major baselines use zero-shot or few-shot settings. Differences in imaging inputs further limit attribution of the performance gains to the method itself.

We thank the Reviewer for raising this concern. We clarify the purpose of each comparison, point to the stronger baselines we include, and make the input representations explicit.

(1) Purpose of the comparisons. Our comparisons with existing medical VLMs are not intended to establish architectural superiority, as these models are evaluated without local fine-tuning. They measure how well available models generate Vietnamese clinical reports in the target setting, which is the practical alternative to collecting local data. Our primary controlled comparison is among Single, Joint-I and Joint-F, which share the same splits, encoders, language model and evaluation protocol: Single and Joint-I use the same in-house supervision to measure the trade-off between modality-specific and joint learning, and Joint-F adds public supervision to measure the effect of data scaling. We will revise Section 5 to state this explicitly and to avoid attributing the gains over external baselines to the method itself.

(2) Stronger baselines. Beyond the medical VLMs, we evaluate two strong general-purpose models, GPT-4o and DeepSeek-v4.1, with the radiologist-designed clinical F1 (Table 4) and GREEN (Table A11). Our best model remains ahead on all four modalities, although by very different margins: 48.82 vs 43.65 clinical F1 on X-ray, 48.86 vs 47.83 on CT, 48.67 vs 43.84 on MRI, and 58.95 vs 20.16 on PET/CT. We also add a translation baseline on the public benchmarks (Table A12, Appendix G.4), in which each medical VLM generates an English report that is then translated into Vietnamese, which removes the disadvantage of generating directly in Vietnamese. On the same translated test sets, Joint-F remains ahead on CT-RATE (ROUGE-L 34.73 vs 14.73) and MR-RATE (27.96 vs 11.81) and is comparable on MIMIC-CXR (24.45 vs 21.52) (Table 5 and Table A12).

(3) Differences in imaging inputs. We acknowledge that the baselines receive different image representations (Appendix D). This follows from their architectures rather than from our choice: most existing VLMs, including LLaVA-Med, MedGemma, GPT-4o and DeepSeek-v4.1, accept only 2D images and cannot encode CT, MRI or PET/CT volumes natively. To evaluate them on volumetric data at all, we convert each volume into evenly spaced axial slices, tiled into grids for most models, a common practice for applying 2D models to 3D imaging. Models that support volumetric input, RadFM and M3D, receive the full 3D volume. The input representation alone does not explain the gap: even with native 3D input, RadFM and M3D reach ROUGE-L of 5.83 and 33.05 and BERTScore of 56.73 and 66.98 on CT, against 63.02 and 84.89 for our single-modality model (Table 4). The gap therefore reflects the lack of Vietnamese clinical supervision rather than the loss of volumetric information. Table A4 summarizes the native input of each baseline and Appendix D details the input given to each model; we will extend Table A4 to report, for every evaluated model and modality, the input representation, the number and selection of slices, the intensity window, the MRI sequences and the PET/CT channels.

AuthorsOpenReview ↗

Response to Reviewer HcbH (Part 3)

W3. Limited benefits of unification. The joint models achieve lower clinical F1 than modality-specific models on X-ray, CT, and MRI, while additional public data do not yield consistent improvements.

Thank you for this comment. Our additional experiments show that joint training does not necessarily sacrifice accuracy for unification.

We evaluated two joint training configurations (X-ray+CT and X-ray+CT+PET/CT) against their corresponding single-modality baselines, using an identical training recipe and per-modality training schedule (Table A10, Appendix G.2). Joint training substantially improves CT performance: when trained with X-ray, CT ROUGE-L increases from 53.4 to 62.0, BERTScore from 80.6 to 84.7, and GREEN from 32.9 to 44.7 (paired Δ +11.8, 95% CI [+9.9, +13.6]). The three-modality model further improves X-ray GREEN by 1.6 points (95% CI [+0.8, +2.4]) and reduces false findings per report from 0.55 to 0.39.

The benefits are modality-dependent: while CT and X-ray improve, PET/CT performance remains largely comparable in ROUGE-L (70.7 vs. 69.6). We acknowledge that, in our main four-modality configuration, training across all modalities introduces modest trade-offs compared with modality-specific models: Joint-F retains 98.6%, 97.6%, 92.4% and 98.2% of their clinical F1 on X-ray, CT, MRI and PET/CT (Table 4). Under GREEN (Table A11), however, the joint models match or exceed the single-modality models on X-ray, CT and PET/CT (51.0 vs 49.7, 40.6 vs 37.5, 62.0 vs 58.6), with MRI the only modality where Single remains clearly better.

On public data. The effect of adding translated public data is likewise modality-dependent and follows the amount of public data available for each modality. From Joint-I to Joint-F, clinical F1 increases by 1.71 points on X-ray (36,000 public training reports) and 0.95 on CT (7,597), is unchanged on MRI (+0.03, 800 reports), and decreases by 1.36 on PET/CT, for which no public data exist (Tables 4 and A8). Translated public reports therefore help only where they are plentiful and match the modality, and they do not replace native Vietnamese supervision such as ViMed-Rad.

Nevertheless, we believe that a unified framework provides a valuable foundation for future cross-modal learning, enabling knowledge sharing and transfer across imaging modalities. These findings highlight the potential of joint training beyond deployment convenience. These results are reported in Table A10 (Appendix G.2) of the revised manuscript; we will add clinical F1 scores for the new configurations, and revise Section 5 to describe the effect of joint training and public data as modality-dependent.

Table A10. Joint versus single-modality training under an identical recipe and per-modality training schedule, on the in-house test sets (×100). Single denotes recipe-matched single-modality models trained for this comparison; they differ from the Single models of Table 4. R-L: ROUGE-L; BERT: BERTScore; GREEN: translated GREEN (higher is better). ΔGREEN: paired difference to Single with 95% patient-level bootstrap intervals (B = 2,000). False findings: GREEN's clinically significant false findings per report (lower is better). Bold: best per column. –: not computed.

Table A10. Joint vs. single-modality training with an identical recipe (×100). Single: recipe-matched single-modality models, distinct from those in Table 4. ΔGREEN: paired difference to Single [95% patient-level bootstrap CI]. False findings: per report, lower is better. Bold: best per column.

Training modalitiesX-ray R-LX-ray BERTX-ray GREENCT R-LCT BERTCT GREENPET/CT R-LPET/CT BERTPET/CT GREEN
Single40.6978.7357.1753.3580.5732.8670.7490.0262.02
X-ray + CT41.4079.1157.2861.9884.6544.67–––
X-ray + CT + PET/CT41.2379.3458.7561.9984.5945.3269.6289.8161.08
ΔGREEN vs. Single [95% CI]
X-ray + CT+0.1 [−0.8, +1.0]+11.8 [+9.9, +13.6]–
X-ray + CT + PET/CT+1.6 [+0.8, +2.4]+12.5 [+10.8, +14.1]−0.9 [−1.6, −0.3]
False findings per report
Single0.551.840.19
X-ray + CT0.490.74–
X-ray + CT + PET/CT0.390.720.18

Reviewer_hE3jOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a dataset (ViMed-Rad) and a benchmark (ViMed-Bench). The dataset contains 62K X-rays, 13K CTs, 12K MRIs, and 6K PET/CTs paired with radiologist reports written in Vietnamese. ViMed-Bench is a benchmark for both report generation as well as VQA at multiple levels of difficulty.

For evaluation on the benchmark, the authors propose a 4-stage training procedure (stage 1 and 2 are contrastive alignment, 3 updates a projector that feeds into a LM, and stage 4 updates both the projector and applies LoRA to the LM). The benchmark has 4 tasks: report generation, VQA-Select (multiple-choice), VQA-Respond (open-ended), VQA-Compose (utilizes structured findings to create questions, varying difficulty).

Evaluation is done using both standard NLP evaluation metrics (BLEU, ROUGE, BERTScore) as well as clinical evaluation where an LLM extracts attributes from both AI-generated and GT reports and compares overlap (precision, recall, and F1). The proposed methods perform better than both specialized models and generalist models across all metrics.

Critical strengths
  1. Illustrates the need for non-English report-image datasets. This provides data and reports that reflect a much different population than most pre-existing medical vision-language datasets.
  2. Dataset scale. While most of the samples are X-rays, the dataset includes ~30K 3D volumes with reports, which greatly adds to the growing collection of public datasets.
Critical weaknesses
  1. The chosen metrics leave some findings unclear. BLEU and ROUGE offer very little insight into report quality (and say much more about language gaps). BERTScore is slightly better, but not as insightful as a clinically motivated metric like GREEN. The clinical evaluation section is also insightful, but the comparison models do quite well and the scores are generally very close. Expanding to one or two more metrics would likely strengthen the results.
  2. Much of the paper stresses the importance of multi-modality dataset/models, but there is not a lot of strong reasoning/evidence of why that is the case.
Questions for authors
  1. Do you see there being any trade-off between releasing DICOM versus NIFTI?
  2. There is emphasis on a joint-modality model even though the individual modality model generally outperforms. Are there real clinical scenarios where a single joint model would be preferred over multiple more accurate models?
  3. Figure 3 is hard to interpret. In general it seems that including more training tasks maintains/slightly degrades performance?
3 comments
AuthorsOpenReview ↗

Response to Reviewer hE3j (Part 1)

We thank the Reviewer for recognizing the value of non-English image-report data and the scale of the volumetric studies in ViMed-Rad. We address each point below.


W1. The chosen metrics leave some findings unclear. BLEU and ROUGE offer very little insight into report quality (and say much more about language gaps). BERTScore is slightly better, but not as insightful as a clinically motivated metric like GREEN. The clinical evaluation section is also insightful, but the comparison models do quite well and the scores are generally very close. Expanding to one or two more metrics would likely strengthen the results.

We appreciate the Reviewer's suggestion on the evaluation metrics. We agree that BLEU, ROUGE and BERTScore measure textual and semantic similarity rather than clinical correctness; we report them for comparability with prior work. Our primary clinical evaluation is the finding-level P/R/F1 of Section 4 and Appendix F: radiologists designed a schema per modality covering findings, location, negation and certainty, and verified 100 extracted records per modality (6% corrected).

Following the Reviewer's suggestion, we additionally report GREEN alongside the clinical P/R/F1 (Table A11, Appendix G.3). Since GREEN was developed for English chest radiographs, we translated both the reference and the generated report into English independently with DeepSeek before scoring with GREEN-RadLlama2-7b; the same translation is applied to all systems and to the references. We evaluate GREEN on the test studies with abnormal findings.

Table A11. Clinical metrics on the ViMed-Rad test set (×100). P, R and F1 are macro-averaged over finding labels; GREEN is computed on studies with abnormal findings, with 95% patient-level bootstrap intervals. Bold: best; italic: second best, per modality and metric.

ModalityModelPRF1GREENGREEN 95% CI
X-rayGPT-4o44.8145.1743.6530.6[25.8, 35.9]
DeepSeek-v4.146.0342.4140.8147.9[42.8, 53.4]
Single48.6649.7448.8249.7[45.3, 54.6]
Joint-I46.0446.8346.4251.0[46.5, 55.7]
Joint-F48.0548.4548.1349.8[45.1, 54.6]
CTGPT-4o45.9549.2746.778.6[7.3, 9.9]
DeepSeek-v4.148.2149.9547.8323.2[21.3, 25.3]
Single51.0350.7948.8637.5[35.6, 39.5]
Joint-I46.3149.8646.7439.1[37.1, 40.9]
Joint-F46.6550.0147.6940.6[38.7, 42.5]
MRIGPT-4o43.5543.9742.7416.9[15.1, 18.9]
DeepSeek-v4.145.3644.8543.8429.8[27.4, 32.2]
Single50.7249.5548.6757.3[54.6, 60.0]
Joint-I46.0946.9244.9552.4[49.8, 55.0]
Joint-F48.8747.4444.9851.8[49.3, 54.5]
PET/CT†GPT-4o10.085.906.978.7[7.9, 9.5]
DeepSeek-v4.126.8318.2020.166.5[5.7, 7.3]
Single68.8056.2458.6358.6[57.0, 60.1]
Joint-I79.4752.2058.9562.0[60.3, 63.6]
Joint-F67.9955.4657.5958.4[56.7, 60.1]

†PET/CT P, R and F1 are computed per case over matched lesions and averaged over cases.

As shown, our models achieve the best performance in terms of the GREEN metric on all four modalities, with the largest margins on CT, MRI and PET/CT (e.g., 40.6 vs 23.2 for DeepSeek on CT, 57.3 vs 29.8 on MRI, and 62.0 vs 8.7 for GPT-4o on PET/CT). Both joint models match or exceed the single-modality models on X-ray, CT and PET/CT, with MRI the only modality where Single remains clearly better (57.3 vs 52.4). We will add the GREEN protocol to the revised paper.


AuthorsOpenReview ↗

Response to Reviewer hE3j (Part 2)

W2. Much of the paper stresses the importance of multi-modality dataset/models, but there is not a lot of strong reasoning/evidence of why that is the case.

We thank the Reviewer for this insightful question. Below, we provide further clarification on the importance of multimodal datasets and models.

Multi-modality data.

  • Organization of the imaging. Public X-ray, CT and MRI resources are fragmented: each covers one modality, comes from a different institution and population, and is distributed in its own format, so they cannot be combined into one consistent cohort. ViMed-Rad instead organizes every examination by patient and study, as native DICOM linked to its signed report through the request identifier of the hospital information systems (Section 3.1). X-ray, CT and MRI come from the same institution and reporting convention, so differences between them reflect the modality rather than the data source.
  • PET/CT. PET/CT image–report data are especially scarce, with 6,376 whole-body scans paired with reports, ViMed-Rad is, to our knowledge, the largest PET/CT image–report collection.
  • Several modalities per patient. In clinical practice, a patient is often examined with more than one modality, for example a chest radiograph followed by a chest CT, and each modality captures a different aspect of the same condition: planar anatomy, volumetric structure, soft-tissue contrast or metabolic activity. Because ViMed-Rad organizes all studies by patient under one institution and reporting convention, it provides the richer, complementary information needed to learn across examinations of the same patient, which single-modality datasets cannot offer.
  • Native reports. Translated English reports are not a substitute for reports written by Vietnamese radiologists. Native reports follow local conventions, such as templated impressions and hedging expressions for "suspected" (theo dõi, literally "to be monitored"), "suggestive of" (nghĩ nhiều) and "cannot be excluded" (chưa loại trừ) (Appendix A.2), which translation does not reproduce. Empirically, adding 36,000 translated MIMIC-CXR reports, about as many as our 35,246 in-house X-ray training reports, improves X-ray clinical F1 by only 1.71 points (Joint-I vs Joint-F, Table 4), and translated data bring no gain where they are scarce (MRI, +0.03).

Multi-modality model. In new controlled experiments with an identical training recipe (Table A10, Appendix G.2), training CT jointly with X-ray improves CT GREEN from 32.9 to 44.7 (paired Δ +11.8, 95% CI [+9.9, +13.6]), and adding PET/CT improves X-ray GREEN by 1.6 points (95% CI [+0.8, +2.4]) and lowers X-ray false findings per report from 0.55 to 0.39. X-ray and CT therefore benefit from each other through the shared language model, most clearly CT, the smaller of the two. PET/CT is largely preserved, with a small decrease in GREEN (62.0 vs 61.1, Δ −0.9, 95% CI [−1.6, −0.3]). Our model processes one study at a time; it shares clinical language and reporting knowledge across modalities rather than fusing several examinations of the same patient.

We will revise the Introduction and Section 5 accordingly.

Table A10. Joint vs. single-modality training with an identical recipe (×100). Single: recipe-matched single-modality models, distinct from those in Table 4. ΔGREEN: paired difference to Single [95% patient-level bootstrap CI]. False findings: per report, lower is better. Bold: best per column.

Training modalitiesX-ray R-LX-ray BERTX-ray GREENCT R-LCT BERTCT GREENPET/CT R-LPET/CT BERTPET/CT GREEN
Single40.6978.7357.1753.3580.5732.8670.7490.0262.02
X-ray + CT41.4079.1157.2861.9884.6544.67–––
X-ray + CT + PET/CT41.2379.3458.7561.9984.5945.3269.6289.8161.08
ΔGREEN vs. Single [95% CI]
X-ray + CT+0.1 [−0.8, +1.0]+11.8 [+9.9, +13.6]–
X-ray + CT + PET/CT+1.6 [+0.8, +2.4]+12.5 [+10.8, +14.1]−0.9 [−1.6, −0.3]
False findings per report
Single0.551.840.19
X-ray + CT0.490.74–
X-ray + CT + PET/CT0.390.720.18

AuthorsOpenReview ↗

Response to Reviewer hE3j (Part 3)

Q1. Do you see there being any trade-off between releasing DICOM versus NIFTI?

We appreciate this practical question on the data format. We release the data in DICOM because it is the standard format for medical imaging: it is what radiologists read in PACS, and it retains the full acquisition metadata, such as the original bit depth of radiographs and the patient weight, injected dose and injection time needed to compute SUV on PET/CT. NIfTI, by contrast, is a preprocessing step for AI models: converting to it involves task-specific choices (series selection, windowing, resampling) and discards part of this metadata. We therefore release DICOM as the source data, together with our preprocessing code and configurations, so that users can reproduce the exact NIfTI volumes and images used in our experiments or adapt the preprocessing to their own tasks. We will clarify this in the revised paper.


Q2. There is emphasis on a joint-modality model even though the individual modality model generally outperforms. Are there real clinical scenarios where a single joint model would be preferred over multiple more accurate models?

We thank the Reviewer for raising the question of clinical applicability. A joint model is preferable when a case requires integrating information from several modalities. From our clinical experts' experience, a typical example is the lung cancer pathway: a suspicious nodule is first detected on a chest radiograph, then characterized on chest CT (size, margins, invasion of adjacent structures), and finally staged on whole-body PET/CT (metabolic activity, nodal involvement and distant metastases). The radiologist interprets each examination in light of the others, and the same findings, terminology and anatomical locations recur across the modalities. A single joint model trained on all the modalities of this pathway can leverage this cross-modal information, since knowledge learned from one modality transfers to the others, whereas separate single-modality models learn each modality in isolation.

Our experiments support this cross-modal benefit. We evaluated two joint training configurations (X-ray+CT and X-ray+CT+PET/CT) against their corresponding single-modality baselines, using an identical training recipe and per-modality training schedule (Table A10, Appendix G.2). When trained jointly with X-ray, CT ROUGE-L increases from 53.4 to 62.0, BERTScore from 80.6 to 84.7, and GREEN from 32.9 to 44.7 (paired Δ +11.8, 95% CI [+9.9, +13.6]). Adding PET/CT, the final step of the pathway, further improves X-ray GREEN by 1.6 points (95% CI [+0.8, +2.4]) and reduces false findings per X-ray report from 0.55 to 0.39, while PET/CT itself remains comparable (ROUGE-L 70.7 vs 69.6). Each modality of the chest imaging pathway therefore benefits from being learned together with the others, which separate single-modality models cannot exploit. These results are reported in Table A10 (Appendix G.2) of the revised manuscript, and we will add clinical F1 scores for the new configurations.


Q3. Figure 3 is hard to interpret. In general it seems that including more training tasks maintains/slightly degrades performance?

We apologize that the current presentation of Figure 3 makes it hard to interpret. We will revise the caption and the main text to point out more clearly the two main insights of this figure.

(1) Adding tasks does not noticeably degrade earlier tasks. Figure 3 shows the BERTScore averaged over the four modalities. Each line connects the scores of one task across the cumulative configurations (R-S, R-S-P, R-S-P-C), and the lines remain nearly flat: as new tasks are added, the performance on a given task drops only slightly. For example, report BERTScore changes by −0.14 from R-S to R-S-P-C, and VQA-Select accuracy by −1.17 (Table A13).

(2) The VQA tasks increase in difficulty as designed. The height of the points of each task on the vertical axis follows the intended order of difficulty: VQA-Select scores highest (99.07 in R-S-P-C), followed by VQA-Respond (89.39) and VQA-Compose (86.75). Report generation (86.07) is close to VQA-Compose, the most difficult VQA task. This is expected, since a report must describe all findings of a study together with their location, attributes, negation and certainty, which is the same kind of compositional content that VQA-Compose questions combine.

The per-task changes are summarized in Table A13, and the full per-modality results for all metrics, together with the zero-shot DeepSeek-v4.1 baseline, are presented in Table A14 (Appendix G.5).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Concerns Across Reviewers

We thank all Reviewers (hE3j, HcbH, uiPP) for their careful reading and helpful comments. The Reviewers point out the need for native non-English image–report data (hE3j, HcbH, uiPP), the scale of ViMed-Rad, including its volumetric and whole-body PET/CT studies (hE3j, uiPP), and its value as a Vietnamese clinical resource beyond chest radiography (HcbH, uiPP). We have uploaded a revised manuscript with all changes marked in blue. We have replied to each Reviewer in detail in the individual responses, and summarize the main points as follows.

1. Data availability (HcbH, uiPP). We will release the full dataset upon acceptance: 93,781 de-identified DICOM studies, the radiologist-written reports, the VQA annotations, the splits and the code. It will be hosted on Hugging Face as a gated repository under CC BY-NC-SA 4.0 with a data use agreement. A de-identified sample is already online for the review period: all 37,369 X-ray studies of the benchmark subset and 140 studies each of CT, MRI and PET/CT. (Reproducibility Statement)

2. Clinical evaluation (hE3j, HcbH). We added GREEN next to our radiologist-designed P/R/F1, with 95% patient-level bootstrap intervals. Our models score best on GREEN in all four modalities, with the clearest gains on CT (40.6 vs 23.2), MRI (57.3 vs 29.8) and PET/CT (62.0 vs 8.7). (Tables A9 and A11, Appendix G.1 and G.3)

3. Joint training across modalities (hE3j, HcbH). We ran a new controlled experiment with the same training recipe for joint and single-modality models. Training CT together with X-ray raises CT GREEN from 32.9 to 44.7 (Δ +11.8, 95% CI [+9.9, +13.6]) and cuts false findings per CT report from 1.84 to 0.74. Adding PET/CT improves X-ray GREEN by 1.6 points (95% CI [+0.8, +2.4]) at a small cost on PET/CT (Δ −0.9). On GREEN, the joint models match or beat the single-modality models on X-ray, CT and PET/CT. On clinical F1, the four-modality Joint-F keeps 92.4–98.6% of the single-modality scores. (Introduction, Section 5, Table A10, Appendix G.2)

4. Baseline comparisons (HcbH, uiPP). In our responses, we explained that the comparison with external VLMs measures the practical alternative to collecting local data, while the controlled comparison is between Single, Joint-I and Joint-F. In the revision, we report the native and actual image input of every baseline (3D volume or slice grid, number of slices, PET/CT inputs). We also added translate-then-evaluate baselines, translating English outputs with an LLM (DeepSeek) and with NLLB-200. Translation closes most of the language gap, so most of the gap comes from generating in Vietnamese. Joint-F still leads on CT-RATE (ROUGE-L 34.73 vs 14.73) and MR-RATE (27.96 vs 11.81), and is comparable on MIMIC-CXR. (Section 2, Table A4, Appendix D, Table A12, Appendix G.4)

5. Dataset scale, positioning and provenance (uiPP). Tables 1 and 3 now give unique patients and slices. A "Counting units" paragraph (Section 3.2), Figure 4 and Table A1 separate the full cohort from the benchmark subset for each split; for PET/CT, the full cohort has 6,376 scans and the benchmark 2,757, giving 8,271 region-level reports. Table A2 lists acquisition statistics from the DICOM headers, and all splits are patient-disjoint. Related Work now covers RadGenome-Chest CT, MetaStructAtlas, PET-Bench and PETWB-REP.

6. Multi-task training (hE3j). Tables A13 and A14 (Appendix G.5) show that adding VQA tasks changes earlier tasks only slightly (on average −0.14 report BERTScore and −1.17 VQA-Select accuracy), and that scores follow the intended order of task difficulty.

To our knowledge, ViMed-Rad is the first resource to bring X-ray, CT, MRI and whole-body PET/CT together as native DICOM studies with radiologist-written reports in a low-resource language, and it includes the largest PET/CT image–report collection. With fixed patient-level splits and radiologist-designed clinical metrics, ViMed-Bench shows three things: current medical VLMs struggle to write reports in Vietnamese, native Vietnamese supervision brings gains that translation does not, and one model can cover four imaging modalities at a small, measurable cost while helping smaller modalities such as CT. We hope our responses and the revised manuscript answer the Reviewers' concerns, and we are happy to discuss any remaining questions.

Area_Chair_Y2nJOpenReview ↗

Dear HcbH and HcbH, thanks for your timely feedback. Dear hE3j, could you please let us know whether the authors’ rebuttal changed your rating? Please submit your final rating and reasoning. Thank you!

Dear ALL,

Dear HcbH and HcbH, thanks for your timely feedback. Dear hE3j, could you please let us know whether the authors’ rebuttal changed your rating? Please submit your final rating and reasoning. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.