acceptodds
Desk rejected at ICLR 2027

MedSensus: Benchmarking Medical Reasoning in Omni-Modal Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Clinical reasoning rarely relies on a single source of evidence. Clinicians combine complementary observations to understand the same problem from different perspectives. Although modern models can now process heterogeneous inputs together, publicly accessible medical datasets with reliable clinical annotation remain scarce. Most existing benchmarks still focus on individual modalities or narrower pairings. We introduce MedSensus, a medical benchmark built from publicly available resources and reviewed by clinical experts. MedSensus comprises 9,539 evaluation instances organized into 1,865 evidence groups that share visual and acoustic evidence. Questions may have one or multiple correct answers. This structure supports question accuracy and group accuracy, with the latter requiring all related questions to be answered correctly. The benchmark also provides a factorial evaluation suite that selectively withholds the image, audio, or question stem, enabling estimation of the conditional ablation effect of each input source. Across fifteen representative models, the best model under direct prompting reaches 63.84% question accuracy and 17.04% group accuracy, compared with the clinician reference of 91.12% and 72.97%. These results show that accuracy on individual questions does not imply consistent reasoning across clinical evidence. By combining expert validation with controlled diagnostic evaluation, MedSensus provides a basis for studying how medical models integrate heterogeneous evidence.

Primary area:
datasets and benchmarks
Keywords:
medical multimodal learning, omni-modal models, clinical reasoning, medical audio

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fPV6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces MedSensus, a medical benchmark that pairs one image and one audio recording with multiple-choice questions that can have one or more correct answers. It contains 499 images, 1,865 recordings (clinical speech, respiratory sounds, and ECG sonifications), and 2,639 questions, forming 9,539 instances in 1,865 evidence groups. The paper reports question accuracy and group accuracy, which requires every question in a group to be correct. It also includes ablations that remove the image, audio, or question stem, comparisons of raw audio with text versions, and tests where the audio agrees or conflicts with the image. Across 15 omni-modal models, the best reaches 63.84% question accuracy and 17.04% group accuracy, compared with 91.12% and 72.97% for clinicians.

Critical strengths
  1. Evaluating medical images and clinical audio together within the same case is relatively understudied.

  2. The ablation design is clean. Every subset of image, audio, and question is tested, an options-only control is included, and removed inputs are not replaced with placeholders. The paper also reports many extra analyses, including option order, open-ended answers, and subgroup results, strengthening the robustness.

Critical weaknesses
  1. The necessity of omni-modal models and the extent of actual multimodal understanding are unclear. It is unclear why the benchmark needs omni-modal models at all. The audio can be turned into formats that ordinary models already handle. Speech can become a transcript, and physiological signals can become a written description, a waveform image, or a time series. An ECG is normally read as a trace, yet the benchmark turns it into sound. The speech is also read from pre-written scripts, so it carries little beyond its text. Removing the audio lowers accuracy, but this only shows that the information in the recordings is useful, not that it has to come as audio. When the audio is replaced with text, results are about the same, and for physiological signals the text works better. Giving both works best, and the paper attributes this to cues that text loses. However, it never tests which cues matter, and the gain may simply come from giving the same content twice. The omni requirement also leaves out strong models like Claude and GPT, which could handle the converted inputs. Moreover, models can score well from the answer options alone (potential leakage). Given only the options, with no image, audio, or question, every model gets about 33%, far above the 4% random baseline. For the weaker models, this is most of their full score. So potentially a large part of the accuracy comes from patterns in the options rather than from understanding the case.

  2. The reliability of the benchmark and its reported results is questionable. The whole benchmark relies on expert-written questions, but the paper doesn't say how many experts there were, what fields they work in, or how disagreements were settled. It also says in one place that experts wrote all the questions and in another that they "created or revised" them, without saying what was revised. The three clinicians used as the human baseline are on the research team. If they also wrote or reviewed the questions, their high scores are also questionable. The images are also likely already in the models' training data. They seem to come from existing public benchmarks such as GMAI-MMBench and OmniMedVQA, which have been out since 2024. Models may have seen these images and their labels during training, and the paper doesn't check for this. It also doesn't list the source datasets or their licenses, so it's unclear whether the benchmark can be shared. Furthermore, several numbers in the paper don't add up. Many percentages in Table 5 are impossible with 1,200 test items. In Table 6, under the paper's own description of "answer consistency," some models' average accuracy is higher than the other columns allow. In Figure 5b, the gap between the two model groups is exactly the same in every cell, which is very unlikely for real results. The headline 17.04% group accuracy can't be produced with 1,865 groups. In the appendix examples, the second-best model gets every question wrong, while a mid-ranked model gets almost all of them right. Together, these issues make it hard to trust the reported results.

Questions for authors
  1. What does the benchmark test that a non-omni setup can't, such as images plus transcripts or signal plots? Which audio cues make the combined input work better, and how would strong models like Claude or GPT do with the converted inputs?
  2. Who wrote and reviewed the questions, how many people were involved, and in what fields? Which questions were revised, and what were they based on? Did the three reference clinicians take part?
  3. What patterns in the options let models reach about 33% without seeing the case, and can they be removed?
  4. Can models recall the source labels from the images alone? Do the source licenses allow the benchmark to be redistributed?
  5. Can you explain the numbers in Tables 5 and 6, Figure 5b, and the 17.04% result, and share the prediction records so they can be checked? How were the appendix examples chosen?
10 comments
AuthorsOpenReview ↗

Response to Reviewer fPV6 (part 1)

W1-1: Why omni-modal evaluation is needed in medicine

Omni-modal modeling has been a central research direction in recent years. Starting with GPT-4o [1] in 2024, followed by Qwen2.5-Omni [2] and Qwen3-Omni [3] in 2025 and Qwen3.8-Omni [4] in 2026, native processing of audio, images, and text has become a basic capability of mainstream models, and general-domain benchmarks such as OmniBench [5] evaluate it directly. As models move in this direction, evaluation has to follow.

Medicine is one of the domains that needs this capability most, because clinical diagnosis relies on sound. The patient's history is given orally and contributes most to the final diagnosis [6]; heart and lung sounds are obtained by auscultation; and cardiac rhythm is monitored by the tone of a bedside monitor. In practice, an image proposes candidate diagnoses and the spoken account or the auscultated sound decides among them. This is exactly the structure of a MedSensus evidence group: one image and one recording with several related questions.

Yet no benchmark evaluates this setting in medicine. Existing medical audio work, including CaReAQA [7] and MedMosaic [8], covers audio only, while medical visual benchmarks such as OmniMedVQA [9] cover images and text. To our knowledge, joint evaluation of images and raw audio has not been done before. MedSensus fills this gap: it evaluates whether a model can integrate an image and an acoustic recording as evidence for the same case and stay consistent across the related questions about it.

A natural next question is whether converting audio to text would make the audio unnecessary. It would not, and the reason starts with how clinicians work. Auscultation is itself the diagnostic act: no clinician has someone else write the sounds down and diagnoses from the note. The expert descriptions in Table 2 were written by physicians, so they hand the judgment under evaluation to a physician beforehand. Spoken histories also carry clinical information in hesitations, pauses, and self-corrections. In real-time consultation, telemedicine, and bedside monitoring, transcription or manual description further adds latency, compounding errors, and labor, so the model has to receive the raw evidence itself. The data agree: in Table 2, audio plus text beats the better of the two single inputs by +1.48 to +3.81 pp on clinical speech and +1.18 to +4.71 pp on physiological sound, positive for all 15 models.

The following compact summary is derived from Table 2 of the paper. It reports the waveform-minus-transcript difference for clinical speech and the gain of audio plus text over the better single input for each audio type. All values are percentage-point differences; the full per-condition accuracies are in the paper.

ModelSpeech: waveform − transcriptSpeech: gain over better single inputPhysiological sound: gain over better single input
Qwen3.8-Omni-Flash+0.91+2.37+2.43
Qwen3.5-Omni-Plus+1.74+2.54+2.45
Gemini 3.8 Flash+1.47+2.81+2.69
Gemini 3.1 Pro+1.01+2.55+4.48
Gemini 3 Pro+0.51+3.81+4.24
Gemini 3 Flash+1.96+1.58+3.19
Gemini 3.5 Flash-0.14+3.48+3.80
Doubao-Seed-2.0-Lite+0.79+1.78+1.18
Qwen3-Omni-30B+0.83+2.15+3.12
MiniCPM-o 4.5-0.05+3.05+3.76
Qwen2.5-Omni-7B+0.68+3.49+4.71
OmniVinci-9B+1.78+3.01+3.83
OLA-7B-0.10+2.64+2.50
Qwen2.5-Omni-3B+1.22+2.78+2.50
Baichuan-Omni-1.5+1.37+1.48+3.80

References [1] OpenAI et al., GPT-4o System Card, 2024. [2] Xu et al., Qwen2.5-Omni Technical Report, 2025. [3] Xu et al., Qwen3-Omni Technical Report, 2025. [4] Qwen Team, Qwen3.8-Omni: Towards Native Omni-Modal Agents, 2026. [5] Li et al., OmniBench: Towards The Future of Universal Omni-Language Models, NeurIPS 2025. [6] Hampton et al., Relative contributions of history-taking, physical examination, and laboratory investigation to diagnosis and management of medical outpatients, BMJ 1975. [7] Wang et al., CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoning, CHIL 2025. [8] Rajgarhia et al., MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio, ICML 2026. [9] Hu et al., OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM, CVPR 2024.

AuthorsOpenReview ↗

Response to Reviewer fPV6 (part 2)

W1-2: Why ECG is represented as sonified audio

Converting ECG to sound is well grounded. Kather et al. [1] used polyphonic sonification of ECG to diagnose cardiac pathologies in Scientific Reports, and a 2025 review in npj Digital Medicine [2] surveys this line of work. In clinical settings, ECG sonification has also been developed for the assessment, diagnosis, and monitoring of ST-elevation myocardial infarction in emergency care [3]. Sonified ECG in MedSensus has a known source signal and diagnosis, so it can be paired with the image in four relations: concordant, noninformative, confusable, and discordant. This allows us to test whether a model can recognize the rhythm from sound. The results show that models use this signal: in the ECG panel of Fig. 5b, accuracy decreases monotonically as the evidence moves from concordant to discordant.

References [1] Kather et al., Polyphonic sonification of electrocardiography signals for diagnosis of cardiac pathologies, Sci Rep 2017. [2] Elgendi et al., ECG sonification methods for robust and generalizable clinical decision support, npj Digit Med 2025. [3] Hermann et al., ST Elevation Sonification of a 12-Lead ECG for the Assessment, Diagnosis, and Monitoring of ST Elevation Myocardial Infarction, Sensors 2025.

W1-3: Clinical information in scripted speech

Recording from a script is an established practice in clinical assessment and teaching. Standardized patients, who perform a fixed history from a script, have long been used so that the clinical content is identical in every assessment [1]. Scripted recording is also the usual way speech resources are built. In medicine, simulated physician-patient interviews are acted out from prepared cases and recorded, as in the datasets of Fareez et al. [2] and PriMock57 [3]; and benchmarks of voice assistants prepare the text first and synthesize much of the spoken input from it, as in VoiceBench [4]. MedSensus follows the same idea. As described in Section 2.2, semantic content is verified first, and delivery characteristics such as pauses, hesitation, and tempo are introduced afterwards; human and synthesized recordings follow the same content, and their provenance is recorded. The verified transcript therefore becomes a clean control: any difference between a recording and its transcript reflects how something is said, not what is said. Table 2 in Section 3.4 of the paper, shows that the audio carries information beyond the text. On clinical speech, the raw waveform outperforms the transcript for 12 of 15 models, by 0.93 pp on average. Adding the waveform to the transcript improves every model, with Acc_q gains of +2.57 to +4.79 pp. Reading from a script thus does not reduce speech to its text; delivery still provides measurable information.

References [1] Barrows, An overview of the uses of standardized patients for teaching and evaluating clinical skills, Acad Med 1993. [2] Fareez et al., A dataset of simulated patient-physician medical interviews with a focus on respiratory cases, Scientific Data 2022. [3] Papadopoulos Korfiatis et al., PriMock57: A Dataset Of Primary Care Mock Consultations, ACL 2022. [4] Chen et al., VoiceBench: Benchmarking LLM-Based Voice Assistants, TACL 2026.

AuthorsOpenReview ↗

Response to Reviewer fPV6 (part 3)

W1-4: Whether text can replace raw audio

In Table 2 in Section 3.4 of the paper, we compared raw audio with its text counterparts, the verified transcript for speech and the expert description for physiological sound. On clinical speech, the raw waveform outperforms the transcript for 12 of 15 models, so the results are not "about the same". On physiological sound, the expert description does score higher than the raw signal. This description is a manual summary written by physicians, and no such text exists ready-made in a real workflow, so the comparison shows what a physician's reading adds, not that the audio is unnecessary. Adding the raw signal to the description still improves all 15 models by +1.18 to +4.71 pp, which means the description discards information that the signal contains. The lower score of the raw signal alone reflects how limited current models still are at perceiving acoustic patterns directly, which is what MedSensus is built to measure.

W1-5: Why audio plus text outperforms either input alone

In Table 2 in Section 3.4 of the paper, we compared raw audio, text, and audio plus text on the same set of questions. Audio plus text beats the better of the two single inputs by +1.48 to +3.81 pp on clinical speech and +1.18 to +4.71 pp on physiological sound, positive for all 15 models. On clinical speech, the raw waveform alone also outperforms the transcript for 12 of 15 models. The paper already states this in Section 3.4: "a transcript preserves lexical content but removes delivery cues carried by the waveform", and "combining the recording with its textual representation performs better than either representation alone". This means the audio contributes information that the text does not contain: on speech, the pauses, hesitation, and tempo that Section 2.2 deliberately controls; on physiological sound, the acoustic detail beyond an expert's summary. It is not a repetition of the same content.

W1-6: Evaluation of Claude and GPT on converted inputs

GPT-5.6 Sol and Claude Opus 5 are not omni-modal models. Their official model pages list only text and image as inputs and audio as unsupported [1–3], so we cannot feed them the recordings MedSensus evaluates. Due to budget constraints, we did not run experiments in which the audio is converted into transcripts, descriptions, or waveform images and given to these models. We believe that our experiments on fifteen omni-modal models, together with the comparison of raw audio against its text counterparts in Table 2, are sufficient to support our conclusions. If the reviewer considers it necessary, we are glad to add this experiment.

References [1] OpenAI API documentation, GPT-5.6 Sol, Input: text, image; Output: text. [2] Amazon Bedrock model card, GPT-5.6 Sol. [3] Amazon Bedrock model card, Claude Opus 5.

Reviewer_f3JPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MedSensus is an omni-modal medical benchmark that tests how well AI models integrate visual and acoustic clinical evidence while maintaining reasoning consistency across related questions. Using Group Accuracy metric, it reveals a massive performance gap between current models and human clinicians in synthesizing multi-modal clinical contexts.

Critical strengths
  • Innovative Consistency Metric: The "evidence-group" design and "Group Accuracy" metric effectively evaluate a model's ability to maintain a unified clinical interpretation across multiple related questions, closely mimicking real-world case analysis.

  • Deep Diagnostic Ablations: The factorial evaluation suite (e.g., withholding image/audio) and evidence-conflict tests successfully isolate whether models genuinely fuse multi-modal data or merely exploit language shortcuts.

Critical weaknesses
  • MCQ Limitations: Despite penalties for incomplete answers, the reliance on multiple-choice questions still allows models to guess via option structures and text priors, failing to fully capture open-ended diagnostic generation.

  • Unrealistic Audio Modalities: The use of "physiological audio" (like ECG sonification) lacks real-world clinical authenticity, as doctors process ECGs and vitals as visual waveforms or structured data, not sound.

Questions for authors

See the critical weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer f3JP (part 1)

Weakness 1: Guessing from options and open-ended diagnostic generation

We thank the reviewer for raising this. We agree with both concerns: when options are kept, a model can partly guess from their wording, and multiple-choice questions cannot fully capture a model's ability to write a diagnosis itself. The paper contains three checks that bear on this, and the numbers are below. Weakness 2 is answered in the next comment.

Options-only accuracy and group-level consistency

Values are from Table 1 under direct prompting and are reported in percent.

ModelComplete Acc_qOptions-only Acc_qComplete Acc_gOptions-only Acc_g
Gemini 3.8 Flash63.8433.1417.000.91
Gemini 3 Flash56.7333.6810.671.02
Doubao-Seed-2.0-Lite57.9533.4311.580.97
Qwen3-Omni-30B49.3632.966.170.91
Qwen2.5-Omni-7B41.1232.743.000.91
Baichuan-Omni-1.534.9232.381.550.86
Clinicians91.12n/a72.97n/a

The uniform-subset chance baseline is 4.22%. Section 3.4 already notes that options-only scores are well above this baseline, which reveals language and answer-structure priors, so complete-input scores should be read together with these controls. The table shows how large the prior is: for all 15 models, options-only Acc_q lies between 32.38% and 33.68%, almost the same for the strongest and the weakest model, so it comes from the option text itself and does not vary with model strength. At the group level, however, guessing almost never works: options-only Acc_g never exceeds 1.02%, while the best model reaches 17.00% with complete input. This is why we also report group accuracy. Scoring is exact match, as described in Section 3.2, so both omitted and unsupported options count as wrong. We revise Gemini 3.8 Flash's complete-input Acc_g in Table 1 from 17.04% to 17.00%, using all 1,865 groups; the submitted calculation omitted five groups with API timeouts or connection resets.

Open-ended generation without answer options

Values are from Appendix A.4, Table 5. Experts scored 1,200 instances against written rubrics; values are reported in percent.

ModelAcc_q with optionsCorrect without optionsDecrease (pp)
Gemini 3.8 Flash63.7455.598.15
Gemini 3 Flash57.1345.6411.49
Doubao-Seed-2.0-Lite58.4047.3811.02
Qwen3-Omni-30B49.4134.5714.84
Qwen2.5-Omni-7B41.3932.119.27
Baichuan-Omni-1.534.4921.1213.37

"Correct" counts only fully correct answers. All 15 models perform worse without options, so options do help. The stronger models nevertheless stay stronger, and the ranking is largely preserved (only two pairs of adjacent models swap places), so the differences between models are not produced by multiple-choice cues. This check covers 1,200 instances and does not replace a full open-ended diagnosis benchmark: MedSensus is designed to test whether a model integrates evidence from the image, the audio, and the question, not to grade free-text diagnoses.

Robustness to shuffled option orders

Values are from Appendix A.5, Table 6, covering all 9,539 instances and reported in percent.

ModelOriginal-order Acc_qShuffled-order mean Acc_qChange (pp)Correct under all orders
Gemini 3.8 Flash63.8462.27−1.5756.33
Gemini 3 Flash56.7354.90−1.8348.52
Doubao-Seed-2.0-Lite57.9556.33−1.6248.64
Qwen3-Omni-30B49.3646.56−2.8036.57
Qwen2.5-Omni-7B41.1238.90−2.2229.73
Baichuan-Omni-1.534.9231.56−3.3624.49

Shuffling lowers accuracy slightly for all 15 models (by 1.57 to 3.99 pp), and the ranking is identical to that under the original order. The share of questions answered correctly under every order is clearly lower; as Section 3.5 states, performance depends on both option semantics and answer presentation, and we report this as a limitation.

In short, guessing from options is real and we report its size. It does not change the comparison between models, and it does not make answering a whole evidence group easy. Thank you again for the comment.

AuthorsOpenReview ↗

Response to Reviewer f3JP (part 2)

Weakness 2: Clinical authenticity of sonified physiological audio

We thank the reviewer for the comment. Doctors usually read an ECG as a trace, and turning it into sound is indeed uncommon, but there is support for it. Kather et al. [1] used polyphonic sonification of ECG to diagnose cardiac pathologies, and a 2025 review [2] surveys this line of work; in emergency care, ECG sonification has been developed for the assessment, diagnosis, and monitoring of ST-elevation myocardial infarction [3]. The sonified ECG in MedSensus also has documented source signals or derivation procedures (Section 2.2).

In the paper we use it to test whether a model makes use of an extra piece of audio evidence, or is disturbed by it. This requires control over how the audio relates to the image. Because the source and diagnosis of a sonified ECG are known, we can pair it with the image in four relations: concordant (C), noninformative (N), confusable (F), and discordant (D). Every question is still given the image, the audio, and the question together; the sound is an additional source of evidence and does not replace the image.

The corresponding experiment is Fig. 5b in Section 3.5 (full curves in Figure 16, Appendix B.8). The table below gives the ECG panel (%; the first four columns are Acc_q, G is Acc_g; Prop. = proprietary models, Open = open-weight models, All = all 15 models; I = questions whose answer should not change with the audio, S = questions whose answer should change with the audio, Section 2.4):

RowCNFDG
Prop.-I62.758.950.645.79.7
Prop.-S62.158.349.346.910.1
All-I54.850.942.938.16.1
All-S54.250.441.739.06.5
Open-I45.841.834.029.52.0
Open-S45.241.432.930.12.3

All six rows decrease monotonically from concordant to discordant, and the respiratory panel shows the same trend. Take the I questions: their answer should not depend on the audio, so a model that ignored the sound would score about the same under the four relations, yet accuracy falls steadily. The models therefore do not ignore the sound; as Appendix B.8 notes, they let a convincing recording override evidence that should remain more important. This is what the experiment is designed to expose: current models cannot yet tell when a sound should be trusted and when it should be disregarded.

ECG and respiratory audio make up a small part of the dataset: 30 case groups (146 questions) and 11 case groups (55 questions), 41 of the 523 case groups in total (7.8%). The main results are computed on all 9,539 instances. Thank you again for the comment.

References

[1] Kather et al., Polyphonic sonification of electrocardiography signals for diagnosis of cardiac pathologies, Sci Rep 2017. [2] Elgendi et al., ECG sonification methods for robust and generalizable clinical decision support, npj Digit Med 2025. [3] Hermann et al., ST Elevation Sonification of a 12-Lead ECG for the Assessment, Diagnosis, and Monitoring of ST Elevation Myocardial Infarction, Sensors 2025.

Reviewer_kbBgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents MedSensus, a medical benchmark consisting of medical questions with aligned visual and audio evidence. Different questions are associated to different groups, allowing for a group accuracy metric to be computed as well, where models are evaluated on their ability to answers all questions in the same group correctly.

The work reveals that there are still significant differences between clinicians and SOTA omni-modal models, especially in group accuracy, which requires all questions associated with the same image-audio pair to be answered correctly (~17% for the best model vs. ~73% for clinicians).

Critical strengths

[S1] Medical diagnosis inherently involves consolidating information from multiple modalities, and yet there are relatively few benchmarks evaluating joint reasoning over visual and raw audio evidence with multiple related clinical questions. This work fills an important gap.

[S2] I like the analysis in Figure 4 and Table 2 combined, which highlights the contributions of different modalities to medical QA performance using LLMs.

Critical weaknesses

[W1] No detailed information on the source datasets, and generally little information on the reproducibility of MedSensus. A detailed list of source datasets and their contributions would be useful.

[W2] The writing is generally convoluted, and I had trouble understanding some of the parts, like the following:

  • Line 46: "Source labels may disagree" - In what way? What do these labels contain? Examples of such disagreements would be helpful.
  • Line 161: "Agreement across sources counts… provide additional support." - How is it determined whether annotations were independently derived?
  • Line 163: "Conflicting interpretations… automatic majority vote." - How exactly are conflicting interpretations resolved by the experts? Is there a systematic adjudication protocol?
  • Line 185: "Alternative physiological recordings … associated with the case." - Where are these alternatives sourced from, and how is their validity affirmed? What percentage of the data points have these alternative recordings?
  • Lines 200-206: No idea what the takeaway of this paragraph is.
  • Line 231: What does it mean for a control to have just the options? What's the instruction in this case for the model to do? And why does it perform better in Table 1? For example, Baichuan performs better with only the options than in the audio-ablated, image-ablated, and question-ablated conditions.
  • Line 241: What are these rubrics mentioned?
  • Line 266: What's this supplementary log? Didn’t find it in the appendix or the attached supplementary zip.
  • Line 276: “Without audio, each of the 2,639 questions is answered once and replicated over matched recording identifiers only for scoring” - What does this mean?
  • Line 295: “replicated predictions preserve matched weighting” - What's being weighted here, and why?

[W3] I would appreciate it if there were a more detailed version with the exact numbers or proportions for Figure 3. It's quite vague in its current form.

[W4] Line 269: “Each clinician evaluates every combination of image, recording, and question as a separate instance, including repeated questions paired with different recordings.” This looks like a potentially unfair comparison for the LLMs, as the clinicians could have prior questions from the same case in memory, but not the LLMs. Were the instances randomized, and were clinicians allowed to see previous questions? I would like to see a comparison where the LLMs also have the previous questions from the same group in context. Additionally, since the clinicians are members of the research team, were they involved in constructing or annotating the benchmark before performing the evaluation?

[W5] Line 304: Do the ‘nonempty answer subsets’ come from the answering models or the datasets themselves? Also, why is this appropriate as a random baseline?

[W6] Table 1 would be easier to read with the deltas than the absolute values in the Factorial controls.

Minor: Figure 4 size differences can be made more contrastive, as they signify performance figures. Currently, they look very similar.

Questions for authors

See Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer kbBg (part 1)

W1: Source datasets and reproducibility

We have added the name, modality, and license of each source dataset, together with a fixed snapshot of the dataset and the evaluation settings.

Source datasetModalityLicense
GMAI-MMBenchMedical imagingApache-2.0; constituent-dataset licenses also apply
OmniMedVQAMedical imagingPer-source licenses; no single license
PTB-XL 1.0.3ECGCC BY 4.0
OmniBrainBenchBrain imagingCC BY-SA 4.0
UNICORN few-shotsPathologyCC BY-NC-SA 2.0
MedATLAS-BenchMedical imagingCC BY-NC 4.0
VQA-RADRadiologyCC0 1.0
SLAKE, ECGBench, OpenMM-MedicalImaging / ECGAs stated on each official page
ICBHIRespiratory sound recordingsAs stated on the official page
Clinical speechRecorded and synthesized by usOurs

Data. The fixed snapshot contains 523 case groups, 499 images, 1,865 recordings, 2,639 questions, and 9,539 evaluation instances; every object has a stable identifier and a checksum, and can be traced back to its original source. Evaluation settings. Complete input uses a direct and a reasoning prompt, and the other seven ablation conditions use the direct prompt, with a missing input simply removed and no placeholder left; all conditions share one response instruction that asks for the letters of all correct options, separated by commas, and there is no system prompt. Decoding uses temperature 0 where the model supports it, and the other parameters are each model's defaults at the time of the calls.

W2-1: (L46) Meaning and examples of source-label disagreements

A source label is the annotation released with an image by its original dataset, usually a diagnosis or finding category, and sometimes a location or grade. For the same kind of lesion, labels can disagree in three ways: different diagnoses, different granularity, and different names. For example, two Barrett's esophagus endoscopy images in GMAI-MMBench carry the categories "proper barrett's esophagus" and "barrett's esophagus". Both point to the same lesion, but the granularity and the naming differ. This is why Section 2.2 states that “source labels are treated as annotations to be examined, not as ground truth,” and has experts verify them.

W2-2: (L161) How annotation independence is determined

We judge independence from the provenance of the data. Two labels are treated as not independent when any of the following holds: one dataset documents that it reuses the images or annotations of another, as when aggregated benchmarks such as GMAI-MMBench and OmniMedVQA integrate earlier datasets; the label text and metadata are identical and trace back to the same original release; or the files are exact or near duplicates from the same upstream source. Inherited labels are collapsed into one source before we check whether the sources agree. This is the rule stated in Section 2.2: “Agreement across sources counts as corroboration only when the annotations were independently derived; inherited labels and duplicated records do not provide additional support.”

W2-3: (L163) Protocol for resolving conflicting expert interpretations

Yes, conflicts follow a fixed protocol described in Section 2.2. Two experts, at least one from the relevant specialty, read the image, all source labels, and the source context separately, without seeing each other's judgment. If they agree, their conclusion becomes the verified anchor for the image. If they disagree, a third senior expert decides after discussion. Cases that remain undetermined are excluded from the benchmark. How often a label appears across sources is only a reference and never decides the outcome, which is why Section 2.2 states that “conflicting interpretations are resolved against the image and source context rather than by automatic majority vote.”

AuthorsOpenReview ↗

Response to Reviewer kbBg (part 2)

W2-4: (L185) Sources, validity, and proportion of alternative recordings

The alternative recordings comprise respiratory sounds from ICBHI and ECG from PTB-XL. In the controlled evidence-relation experiment, each recording is paired with the case image and labeled as concordant, noninformative, confusable, or discordant with the visual case, as described in Section 2.2. Their validity rests on two things: the signal carries the diagnosis of its source dataset, and experts review the relation label. As Section 2.2 puts it, “alternative physiological recordings remain separate interventions rather than replacements for the recording associated with the case.” As shown in Fig. 5b, this part contains 11 respiratory case groups with 55 questions and 30 ECG case groups with 146 questions. The total is 41 case groups, or 7.8% of the 523 case groups described in Section 2.3.

Alternative recordingsSource datasetCase groupsQuestions
Respiratory soundsICBHI1155
ECGPTB-XL30146
Total41, or 7.8% of 523201

W2-5: (L200–206) Data traceability and controlled assignments

This paragraph covers two points: how every data object can be traced, and how the controlled experiments avoid changing what has been verified, as described in Section 2.2. The frozen manifest assigns a stable identifier to every case, image, recording, question, and evaluation instance. A separate provenance crosswalk keeps the original identifiers and file locations, and the analysis metadata records which task and which evidence-relation label each question belongs to. Controlled assignments, which pair the case with an alternative noninformative, confusable, or discordant recording, are interventions added on top of the verified finding and the original recording relation, and they never overwrite either. The paper states this in Section 2.2: “Controlled assignments never overwrite the verified finding or the original recording relation.”

W2-6: (L231) Instruction and performance under the options-only control

With options only, the model receives just the options and the response instruction shared by all conditions: "One or more options may be correct. Respond with the letters of all correct options only, separated by commas (for example: A, C). Do not add any other text." It sees no image, audio, or question stem, so this condition measures how much can be scored from the option text alone when there is no case evidence. For Baichuan, Table 1 in Section 3.3 shows 32.38 with options only, which is indeed higher than the audio-ablated, image-ablated, and question-ablated conditions, whose scores are 30.88, 31.83, and 31.42. Image plus options is also below options only for all 7 open models. The weaker open models score below the options-only level in several incomplete-evidence conditions, which shows that they often fail to make effective use of the available clinical evidence. The strong models show the opposite pattern: Gemini 3.8 Flash scores 33.14 with options only and 39.14 to 59.46 in the other conditions. The relevant part of Table 1 is reproduced below for the two strongest models and all seven open models. Values are Acc_q in percent under direct prompting.

ModelOptions only (O)Audio ablatedImage ablatedQuestion ablatedImage + options
Qwen3.8-Omni-Flash32.8856.9158.9244.3838.92
Gemini 3.8 Flash33.1458.3959.4645.3739.14
Qwen3-Omni-30B32.9644.3646.1234.9231.10
MiniCPM-o 4.533.2740.9142.1834.3130.02
Qwen2.5-Omni-7B32.7436.9838.0433.2228.76
OmniVinci-9B33.0935.7336.5132.9828.12
OLA-7B32.6134.0834.9232.4227.16
Qwen2.5-Omni-3B33.3431.9232.7831.8825.83
Baichuan-Omni-1.532.3830.8831.8331.4225.16
AuthorsOpenReview ↗

Response to Reviewer kbBg (part 3)

W2-7: (L241) Expert-written rubrics for open-ended validation

These are the scoring rubrics of the open-ended validation, in which the options are removed, as described in Section 2.4 with results in appendix Table 5. For each question in the 1,200-instance open-ended subset, the authoring expert wrote a rubric with three parts: the key points a correct answer must cover, corresponding to the correct option set; the points that make an answer only partly correct; and the statements that make an answer wrong or unsupported, corresponding to the distractors. Two experts who did not know which model produced the response scored it separately as correct, partial, incorrect or unsupported, or unscorable, and a third expert resolved any disagreement. Section 2.4 states the same: “A separate task removes answer options and scores free responses with rubrics written by experts.”

W2-8: (L266) Contents of the supplementary evaluation log

The log records, for each evaluated model, the exact checkpoint or API version, the access date, and the decoding parameters described in Section 3.1. The calls were made in September 2026; temperature was set to 0 where the model supports it, and all other parameters were each model's defaults.

W2-9: (L276) Why predictions are replicated in the no-audio condition

As described in Section 2.3, a case with K recordings and M questions yields K×M evaluation instances. Once the audio is removed, the K instances of one question receive identical input: the same image, the same question stem, and the same options. We therefore query the model once per question, making 2,639 calls instead of 9,539, and copy that prediction to the question's K instances, following the procedure in Section 3.1. The model never receives any audio. The copy only ensures that the no-audio condition is scored on the same 9,539 instances and 1,865 evidence groups as the complete-input condition. This is what the sentence in Section 3.1 means: “Without audio, each of the 2,639 questions is answered once and replicated over matched recording identifiers only for scoring.”

W2-10: (L295) Matched weighting of replicated predictions

The weight is the number of times a question is counted under the scoring rule in Section 3.2. Under complete input, a question in a case with K recordings is counted once in each of K evidence groups. Copying the no-audio prediction keeps the same number of counts, so both conditions share the same denominator and the difference Δ_A compares the same set of samples. Without the copy, questions from cases with many recordings would be down-weighted in the no-audio condition alone.

W3: Exact image counts and proportions in Figure 3

We give the exact numbers for Figure 3. The 499 images fall into five visual families and 15 modalities. The two ophthalmic microscopy images are included in the pathology and microscopy family rather than the ophthalmic imaging family:

Visual familyImagesShare
Radiology and ultrasound24549.1%
Endoscopy and surgery13126.3%
Pathology and microscopy10621.2%
Ophthalmic imaging112.2%
Physiological signals61.2%
ModalityImagesShare
Histopathology / IHC8517.0%
MRI7615.2%
Laparoscopic / robotic surgery7314.6%
CT6713.4%
GI endoscopy5410.8%
Musculoskeletal X-ray408.0%
Ultrasound316.2%
Chest X-ray275.4%
Hematology / cytology193.8%
ECG61.2%
Fundus photography61.2%
Ophthalmic OCT51.0%
Mammography40.8%
ENT endoscopy40.8%
Ophthalmic microscopy20.4%

Radiology and ultrasound, endoscopy and surgery, and pathology and microscopy account for most images, while ophthalmic imaging and physiological signals are less represented. This distribution reflects the composition of the source datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.