- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EverydayMedQA, a novel medical benchmark consisting 28,000 items covering six input modalities (text, images, resp audio, speech, video, and documents). This benchmark allows its users to evaluate models on lay-phrased questions that are paired with consumer-grade media rather than clinical-jargon inputs.
The data used to construct the dataset was collected from 20 different sources to constitute the 6 modalities. Each question item in the benchmark gets both a multiple-choice and an open-ended variant, allowing for classification and generation task testing. In the MCQ portion, the image and audio items use 8-option single-select questions, while other modalities use select-all-that-apply items. An AI auditor is used to flag inconsistencies in the items for human review. On the other hand, open-ended answers are scored by Gemma-4-31B as a judge, wherein image and audio items use a single diagnosis-correctness rubric, while the other modalities use a weighted rubric (clinical correctness 60%, safety/triage 25%, question fulfillment 15%).
The authors used EverydayMedQA to evaluate 23 proprietary and open-source models. In the results, Gemini-3.1-Pro leads overall, averaging 61.2 out of 100 on open-ended answers and 65.7% accuracy on multiple-choice across the six modalities. The gap between it and the best open-source models is small on text but large on video and documents. From these results, the authors conclude that a model’s performance in one modality does not reliably carry over to others, and that the open-ended and multiple-choice formats measure different abilities. To check the judge, they compare its rankings with those of a second, smaller Gemma model, finding strong agreement (rank correlation of 0.94). They also have one physician rank responses on 30 items, which agrees with the judge in 74% of pairwise comparisons, a moderate level of agreement (Cohen’s κ = 0.48).
Critical strengths
-
The benchmark targets an under-measured evaluation gap: given the increasing use of consumer-facing health assistants, the paper contributes new, consent-documented material: 500 human speech recordings across 12 first languages, 116 physician-contributed prescriptions, and 550 physician-reviewed video items. It also places all of this under one protocol and evaluates 23 models.
-
The paper's evaluation design is thoughtful and transparent, with the authors taking steps to avoid common benchmark pitfalls. For example, when the correct answer is a long description, a model can often pick it out in a standard multiple-choice question just because it looks more detailed than the other options. To prevent this, the authors split each reference answer into separate statements and require the model to select exactly the right set. The judge is also instructed not to use its own medical knowledge to fill gaps in the reference answer.
Critical weaknesses
-
The claim that performance does not carry over across modalities is not supported by the current analysis. This is becuase the six modalities differ from each other in many ways besides the input type, such as: (i) coming from different source datasets; (ii) asking different kinds of questions [Image and audio items ask only for a diagnosis, while the others ask for broader medical information]; (iii) using different scoring rubrics; and (iv) using different multiple choice formats. Those differences show us that differences in scores between modalities cannot be attributed just to the modality. The paper’s own results show this problem. The speech questions are a subset of the text questions, so a model should do about as well or worse on speech. Instead, the average open-ended score is higher on speech (51.5) than on text (43.3). One model, Nemotron-Nano-Omni, scores 23.8 on text but 60.3 on speech. This suggests the speech subset is easier than the full text set, rather than reflecting anything about speech as an input. A direct comparison of the same 1,000 questions presented as text and as speech would resolve this.
-
It is unclear whether the refence answers are valid for image and audio items. Those items make up half the benchmark (13,519 items). In these items, a generative model wrote the question around an existing diagnosis label. About 80% of the respiratory audio items ask the model to determine COVID status from a cough recording, and most of the rest ask about tuberculosis. Neither condition can be diagnosed from a cough recording, even by a physician. The scoring rubric makes this worse. It gives full credit only for the exact diagnosis and ignores disclaimers. As a result, a safe and appropriate answer, such as “this cannot be diagnosed from a recording, please get tested,” would receive close to zero. The safety and triage criterion is also not applied to image or audio items at all.
-
The video set (550 items) and document set (335 items) are small, yet they count equally toward the overall score, and no confidence intervals or significance tests are reported.
Questions for authors
-
Which model generated the incorrect answer options and split the reference answers into statements, and which model served as the auditor? Did a clinician check that the statements marked as incorrect are actually false?
-
Did you check for data contamination? Some source datasets are publicly available and are kept word for word, so models may have seen them during training.
-
What share of all items received physician review?
-
Can you add a judge from a different model family, or have humans score a sample from each modality, to rule out a judge that favors certain models? This is especially pertinent for a Gemma judge evaluating Gemini models.
-
For the select-all-that-apply questions, how many options and how many correct options does a typical item have, and what would random guessing score? Dataset statistics would be nice to see.