- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces MedSensus, a medical benchmark that pairs one image and one audio recording with multiple-choice questions that can have one or more correct answers. It contains 499 images, 1,865 recordings (clinical speech, respiratory sounds, and ECG sonifications), and 2,639 questions, forming 9,539 instances in 1,865 evidence groups. The paper reports question accuracy and group accuracy, which requires every question in a group to be correct. It also includes ablations that remove the image, audio, or question stem, comparisons of raw audio with text versions, and tests where the audio agrees or conflicts with the image. Across 15 omni-modal models, the best reaches 63.84% question accuracy and 17.04% group accuracy, compared with 91.12% and 72.97% for clinicians.
Critical strengths
-
Evaluating medical images and clinical audio together within the same case is relatively understudied.
-
The ablation design is clean. Every subset of image, audio, and question is tested, an options-only control is included, and removed inputs are not replaced with placeholders. The paper also reports many extra analyses, including option order, open-ended answers, and subgroup results, strengthening the robustness.
Critical weaknesses
-
The necessity of omni-modal models and the extent of actual multimodal understanding are unclear. It is unclear why the benchmark needs omni-modal models at all. The audio can be turned into formats that ordinary models already handle. Speech can become a transcript, and physiological signals can become a written description, a waveform image, or a time series. An ECG is normally read as a trace, yet the benchmark turns it into sound. The speech is also read from pre-written scripts, so it carries little beyond its text. Removing the audio lowers accuracy, but this only shows that the information in the recordings is useful, not that it has to come as audio. When the audio is replaced with text, results are about the same, and for physiological signals the text works better. Giving both works best, and the paper attributes this to cues that text loses. However, it never tests which cues matter, and the gain may simply come from giving the same content twice. The omni requirement also leaves out strong models like Claude and GPT, which could handle the converted inputs. Moreover, models can score well from the answer options alone (potential leakage). Given only the options, with no image, audio, or question, every model gets about 33%, far above the 4% random baseline. For the weaker models, this is most of their full score. So potentially a large part of the accuracy comes from patterns in the options rather than from understanding the case.
-
The reliability of the benchmark and its reported results is questionable. The whole benchmark relies on expert-written questions, but the paper doesn't say how many experts there were, what fields they work in, or how disagreements were settled. It also says in one place that experts wrote all the questions and in another that they "created or revised" them, without saying what was revised. The three clinicians used as the human baseline are on the research team. If they also wrote or reviewed the questions, their high scores are also questionable. The images are also likely already in the models' training data. They seem to come from existing public benchmarks such as GMAI-MMBench and OmniMedVQA, which have been out since 2024. Models may have seen these images and their labels during training, and the paper doesn't check for this. It also doesn't list the source datasets or their licenses, so it's unclear whether the benchmark can be shared. Furthermore, several numbers in the paper don't add up. Many percentages in Table 5 are impossible with 1,200 test items. In Table 6, under the paper's own description of "answer consistency," some models' average accuracy is higher than the other columns allow. In Figure 5b, the gap between the two model groups is exactly the same in every cell, which is very unlikely for real results. The headline 17.04% group accuracy can't be produced with 1,865 groups. In the appendix examples, the second-best model gets every question wrong, while a mid-ranked model gets almost all of them right. Together, these issues make it hard to trust the reported results.
Questions for authors
- What does the benchmark test that a non-omni setup can't, such as images plus transcripts or signal plots? Which audio cues make the combined input work better, and how would strong models like Claude or GPT do with the converted inputs?
- Who wrote and reviewed the questions, how many people were involved, and in what fields? Which questions were revised, and what were they based on? Did the three reference clinicians take part?
- What patterns in the options let models reach about 33% without seeing the case, and can they be removed?
- Can models recall the source labels from the images alone? Do the source licenses allow the benchmark to be redistributed?
- Can you explain the numbers in Tables 5 and 6, Figure 5b, and the 17.04% result, and share the prediction records so they can be checked? How were the appendix examples chosen?
10 comments
Response to Reviewer fPV6 (part 2)
W1-2: Why ECG is represented as sonified audio
Converting ECG to sound is well grounded. Kather et al. [1] used polyphonic sonification of ECG to diagnose cardiac pathologies in Scientific Reports, and a 2025 review in npj Digital Medicine [2] surveys this line of work. In clinical settings, ECG sonification has also been developed for the assessment, diagnosis, and monitoring of ST-elevation myocardial infarction in emergency care [3]. Sonified ECG in MedSensus has a known source signal and diagnosis, so it can be paired with the image in four relations: concordant, noninformative, confusable, and discordant. This allows us to test whether a model can recognize the rhythm from sound. The results show that models use this signal: in the ECG panel of Fig. 5b, accuracy decreases monotonically as the evidence moves from concordant to discordant.
References [1] Kather et al., Polyphonic sonification of electrocardiography signals for diagnosis of cardiac pathologies, Sci Rep 2017. [2] Elgendi et al., ECG sonification methods for robust and generalizable clinical decision support, npj Digit Med 2025. [3] Hermann et al., ST Elevation Sonification of a 12-Lead ECG for the Assessment, Diagnosis, and Monitoring of ST Elevation Myocardial Infarction, Sensors 2025.
W1-3: Clinical information in scripted speech
Recording from a script is an established practice in clinical assessment and teaching. Standardized patients, who perform a fixed history from a script, have long been used so that the clinical content is identical in every assessment [1]. Scripted recording is also the usual way speech resources are built. In medicine, simulated physician-patient interviews are acted out from prepared cases and recorded, as in the datasets of Fareez et al. [2] and PriMock57 [3]; and benchmarks of voice assistants prepare the text first and synthesize much of the spoken input from it, as in VoiceBench [4]. MedSensus follows the same idea. As described in Section 2.2, semantic content is verified first, and delivery characteristics such as pauses, hesitation, and tempo are introduced afterwards; human and synthesized recordings follow the same content, and their provenance is recorded. The verified transcript therefore becomes a clean control: any difference between a recording and its transcript reflects how something is said, not what is said. Table 2 in Section 3.4 of the paper, shows that the audio carries information beyond the text. On clinical speech, the raw waveform outperforms the transcript for 12 of 15 models, by 0.93 pp on average. Adding the waveform to the transcript improves every model, with Acc_q gains of +2.57 to +4.79 pp. Reading from a script thus does not reduce speech to its text; delivery still provides measurable information.
References [1] Barrows, An overview of the uses of standardized patients for teaching and evaluating clinical skills, Acad Med 1993. [2] Fareez et al., A dataset of simulated patient-physician medical interviews with a focus on respiratory cases, Scientific Data 2022. [3] Papadopoulos Korfiatis et al., PriMock57: A Dataset Of Primary Care Mock Consultations, ACL 2022. [4] Chen et al., VoiceBench: Benchmarking LLM-Based Voice Assistants, TACL 2026.
Response to Reviewer fPV6 (part 3)
W1-4: Whether text can replace raw audio
In Table 2 in Section 3.4 of the paper, we compared raw audio with its text counterparts, the verified transcript for speech and the expert description for physiological sound. On clinical speech, the raw waveform outperforms the transcript for 12 of 15 models, so the results are not "about the same". On physiological sound, the expert description does score higher than the raw signal. This description is a manual summary written by physicians, and no such text exists ready-made in a real workflow, so the comparison shows what a physician's reading adds, not that the audio is unnecessary. Adding the raw signal to the description still improves all 15 models by +1.18 to +4.71 pp, which means the description discards information that the signal contains. The lower score of the raw signal alone reflects how limited current models still are at perceiving acoustic patterns directly, which is what MedSensus is built to measure.
W1-5: Why audio plus text outperforms either input alone
In Table 2 in Section 3.4 of the paper, we compared raw audio, text, and audio plus text on the same set of questions. Audio plus text beats the better of the two single inputs by +1.48 to +3.81 pp on clinical speech and +1.18 to +4.71 pp on physiological sound, positive for all 15 models. On clinical speech, the raw waveform alone also outperforms the transcript for 12 of 15 models. The paper already states this in Section 3.4: "a transcript preserves lexical content but removes delivery cues carried by the waveform", and "combining the recording with its textual representation performs better than either representation alone". This means the audio contributes information that the text does not contain: on speech, the pauses, hesitation, and tempo that Section 2.2 deliberately controls; on physiological sound, the acoustic detail beyond an expert's summary. It is not a repetition of the same content.
W1-6: Evaluation of Claude and GPT on converted inputs
GPT-5.6 Sol and Claude Opus 5 are not omni-modal models. Their official model pages list only text and image as inputs and audio as unsupported [1–3], so we cannot feed them the recordings MedSensus evaluates. Due to budget constraints, we did not run experiments in which the audio is converted into transcripts, descriptions, or waveform images and given to these models. We believe that our experiments on fifteen omni-modal models, together with the comparison of raw audio against its text counterparts in Table 2, are sufficient to support our conclusions. If the reviewer considers it necessary, we are glad to add this experiment.
References [1] OpenAI API documentation, GPT-5.6 Sol, Input: text, image; Output: text. [2] Amazon Bedrock model card, GPT-5.6 Sol. [3] Amazon Bedrock model card, Claude Opus 5.
Response to Reviewer fPV6 (part 1)
W1-1: Why omni-modal evaluation is needed in medicine
Omni-modal modeling has been a central research direction in recent years. Starting with GPT-4o [1] in 2024, followed by Qwen2.5-Omni [2] and Qwen3-Omni [3] in 2025 and Qwen3.8-Omni [4] in 2026, native processing of audio, images, and text has become a basic capability of mainstream models, and general-domain benchmarks such as OmniBench [5] evaluate it directly. As models move in this direction, evaluation has to follow.
Medicine is one of the domains that needs this capability most, because clinical diagnosis relies on sound. The patient's history is given orally and contributes most to the final diagnosis [6]; heart and lung sounds are obtained by auscultation; and cardiac rhythm is monitored by the tone of a bedside monitor. In practice, an image proposes candidate diagnoses and the spoken account or the auscultated sound decides among them. This is exactly the structure of a MedSensus evidence group: one image and one recording with several related questions.
Yet no benchmark evaluates this setting in medicine. Existing medical audio work, including CaReAQA [7] and MedMosaic [8], covers audio only, while medical visual benchmarks such as OmniMedVQA [9] cover images and text. To our knowledge, joint evaluation of images and raw audio has not been done before. MedSensus fills this gap: it evaluates whether a model can integrate an image and an acoustic recording as evidence for the same case and stay consistent across the related questions about it.
A natural next question is whether converting audio to text would make the audio unnecessary. It would not, and the reason starts with how clinicians work. Auscultation is itself the diagnostic act: no clinician has someone else write the sounds down and diagnoses from the note. The expert descriptions in Table 2 were written by physicians, so they hand the judgment under evaluation to a physician beforehand. Spoken histories also carry clinical information in hesitations, pauses, and self-corrections. In real-time consultation, telemedicine, and bedside monitoring, transcription or manual description further adds latency, compounding errors, and labor, so the model has to receive the raw evidence itself. The data agree: in Table 2, audio plus text beats the better of the two single inputs by +1.48 to +3.81 pp on clinical speech and +1.18 to +4.71 pp on physiological sound, positive for all 15 models.
The following compact summary is derived from Table 2 of the paper. It reports the waveform-minus-transcript difference for clinical speech and the gain of audio plus text over the better single input for each audio type. All values are percentage-point differences; the full per-condition accuracies are in the paper.
References [1] OpenAI et al., GPT-4o System Card, 2024. [2] Xu et al., Qwen2.5-Omni Technical Report, 2025. [3] Xu et al., Qwen3-Omni Technical Report, 2025. [4] Qwen Team, Qwen3.8-Omni: Towards Native Omni-Modal Agents, 2026. [5] Li et al., OmniBench: Towards The Future of Universal Omni-Language Models, NeurIPS 2025. [6] Hampton et al., Relative contributions of history-taking, physical examination, and laboratory investigation to diagnosis and management of medical outpatients, BMJ 1975. [7] Wang et al., CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoning, CHIL 2025. [8] Rajgarhia et al., MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio, ICML 2026. [9] Hu et al., OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM, CVPR 2024.