acceptodds
Withdrawn from ICLR 2027

EverydayMedQA: A Consumer-Oriented Multimodal Benchmark for Medical Question Answering in Everyday Settings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Conversational AI assistants are increasingly used for healthcare guidance, both before and after consulting a clinician, to prepare for appointments, clarify medical advice, and seek additional information. Recent large multimodal models (LMMs), including omni-modal systems that process text, image, audio, and video, can now respond to medical queries expressed through many forms of input. Yet most medical AI benchmarks remain clinician-oriented: they use expert terminology, rely on clinical-grade inputs, cover narrow medical domains, and typically evaluate a single modality. As a result, how these models handle the informal, consumer-generated queries common in real-world telemedicine remains poorly understood. We introduce EverydayMedQA, a consumer-oriented, multi-modal medical question-answering (QA) benchmark of 28,025 question-answer pairs across six input types: text, image, audio, speech, video, and document. Every question is phrased as a layperson query and paired with consumer-grade media rather than clinical instrumentation, spanning a broad range of medical conditions, specialties, anatomical regions, geographic contexts, and consumer question intents. Certified physicians contributed to benchmark construction through data provision and review of cases requiring clinical judgment. Each item is available in both multiple-choice and open-ended formats, evaluated using a reference-based protocol. Evaluating a comprehensive set of 23 leading models spanning omni-modal models, LLM, VLM, medical VLM, and audio LLMs, we observe that medical question answering performance does not transfer uniformly across input modalities. An omni-modal model that performs well on text may still struggle with patient questions presented in speech, images, audio, video, or documents. We also find that multiple-choice and open-ended questions probe different abilities: a model may select the right options but still give an incomplete free-form medical answer for the same sample. In many cases, models give fluent and relevant-sounding answers that remain clinically incomplete or incorrect. These results suggest that clinician-curated, single-modality benchmarks may overestimate model readiness for consumer-facing telemedicine. To our knowledge, EverydayMedQA is the first consumer-oriented medical QA benchmark to span these six input types under a unified evaluation protocol. Our benchmark and evaluation script will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
"medical question answering", "multimodal benchmark", "consumer health", "omni-modal evaluation"

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_o9wiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EverydayMedQA, a novel medical benchmark consisting 28,000 items covering six input modalities (text, images, resp audio, speech, video, and documents). This benchmark allows its users to evaluate models on lay-phrased questions that are paired with consumer-grade media rather than clinical-jargon inputs.

The data used to construct the dataset was collected from 20 different sources to constitute the 6 modalities. Each question item in the benchmark gets both a multiple-choice and an open-ended variant, allowing for classification and generation task testing. In the MCQ portion, the image and audio items use 8-option single-select questions, while other modalities use select-all-that-apply items. An AI auditor is used to flag inconsistencies in the items for human review. On the other hand, open-ended answers are scored by Gemma-4-31B as a judge, wherein image and audio items use a single diagnosis-correctness rubric, while the other modalities use a weighted rubric (clinical correctness 60%, safety/triage 25%, question fulfillment 15%).

The authors used EverydayMedQA to evaluate 23 proprietary and open-source models. In the results, Gemini-3.1-Pro leads overall, averaging 61.2 out of 100 on open-ended answers and 65.7% accuracy on multiple-choice across the six modalities. The gap between it and the best open-source models is small on text but large on video and documents. From these results, the authors conclude that a model’s performance in one modality does not reliably carry over to others, and that the open-ended and multiple-choice formats measure different abilities. To check the judge, they compare its rankings with those of a second, smaller Gemma model, finding strong agreement (rank correlation of 0.94). They also have one physician rank responses on 30 items, which agrees with the judge in 74% of pairwise comparisons, a moderate level of agreement (Cohen’s κ = 0.48).

Critical strengths
  1. The benchmark targets an under-measured evaluation gap: given the increasing use of consumer-facing health assistants, the paper contributes new, consent-documented material: 500 human speech recordings across 12 first languages, 116 physician-contributed prescriptions, and 550 physician-reviewed video items. It also places all of this under one protocol and evaluates 23 models.

  2. The paper's evaluation design is thoughtful and transparent, with the authors taking steps to avoid common benchmark pitfalls. For example, when the correct answer is a long description, a model can often pick it out in a standard multiple-choice question just because it looks more detailed than the other options. To prevent this, the authors split each reference answer into separate statements and require the model to select exactly the right set. The judge is also instructed not to use its own medical knowledge to fill gaps in the reference answer.

Critical weaknesses
  1. The claim that performance does not carry over across modalities is not supported by the current analysis. This is becuase the six modalities differ from each other in many ways besides the input type, such as: (i) coming from different source datasets; (ii) asking different kinds of questions [Image and audio items ask only for a diagnosis, while the others ask for broader medical information]; (iii) using different scoring rubrics; and (iv) using different multiple choice formats. Those differences show us that differences in scores between modalities cannot be attributed just to the modality. The paper’s own results show this problem. The speech questions are a subset of the text questions, so a model should do about as well or worse on speech. Instead, the average open-ended score is higher on speech (51.5) than on text (43.3). One model, Nemotron-Nano-Omni, scores 23.8 on text but 60.3 on speech. This suggests the speech subset is easier than the full text set, rather than reflecting anything about speech as an input. A direct comparison of the same 1,000 questions presented as text and as speech would resolve this.

  2. It is unclear whether the refence answers are valid for image and audio items. Those items make up half the benchmark (13,519 items). In these items, a generative model wrote the question around an existing diagnosis label. About 80% of the respiratory audio items ask the model to determine COVID status from a cough recording, and most of the rest ask about tuberculosis. Neither condition can be diagnosed from a cough recording, even by a physician. The scoring rubric makes this worse. It gives full credit only for the exact diagnosis and ignores disclaimers. As a result, a safe and appropriate answer, such as “this cannot be diagnosed from a recording, please get tested,” would receive close to zero. The safety and triage criterion is also not applied to image or audio items at all.

  3. The video set (550 items) and document set (335 items) are small, yet they count equally toward the overall score, and no confidence intervals or significance tests are reported.

Questions for authors
  1. Which model generated the incorrect answer options and split the reference answers into statements, and which model served as the auditor? Did a clinician check that the statements marked as incorrect are actually false?

  2. Did you check for data contamination? Some source datasets are publicly available and are kept word for word, so models may have seen them during training.

  3. What share of all items received physician review?

  4. Can you add a judge from a different model family, or have humans score a sample from each modality, to rule out a judge that favors certain models? This is especially pertinent for a Gemma judge evaluating Gemini models.

  5. For the select-all-that-apply questions, how many options and how many correct options does a typical item have, and what would random guessing score? Dataset statistics would be nice to see.

Reviewer_CeQBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

People increasingly ask AI assistants about their health, often with a photo, a voice message, a short video or a prescription. Most medical benchmarks, however, use expert questions, clinical-grade images and a single type of input. The paper presents EverydayMedQA, a benchmark of 28,025 medical questions phrased as a layperson would ask them. It covers six types of input: text, spoken questions, images, recordings of coughs or breathing, videos and medical documents. Most items come from existing public datasets, and they are built in different ways: text questions and physician answers are taken unchanged from online medical consultation services and forums; questions and answers for images and breathing sounds are written by GPT-5.5 from the diagnostic label of each sample; video and document items are written by Gemini 3.1 Flash-Lite from the source material. The authors add new data: 500 spoken questions recorded by 43 volunteers, 500 synthetic spoken questions, 550 videos from TikTok and YouTube, and 116 prescriptions contributed by 22 physicians. An AI model checks the generated items, and humans or physicians review the flagged ones; five physicians check all videos. Each item has an open-ended version, scored by Gemma-4-31B against the reference answer, and a multiple-choice version. The authors evaluate 23 models. Gemini 3.1 Pro obtains the best overall scores. The gap with open-source models is small on text but large on video and documents. Breathing sounds are the hardest input, and the ranking of models changes between the open-ended and multiple-choice formats. As a check of the judge, a single physician ranked the responses for 30 items, and the judge agreed with this physician on 74% of the pairwise comparisons.

Critical strengths

The benchmark covers six types of consumer input under one protocol, including inputs that are rare in medical benchmarks: spoken questions, phone recordings of coughs, self-recorded videos and photographed prescriptions. Every item comes in both an open-ended and a multiple-choice version. This makes it possible to see that a model strong on one type of input can be weak on another, which benchmarks limited to a single input type cannot show.

Part of the data is new and hard to collect. The spoken questions were recorded by 43 consenting volunteers with 12 different first languages, who could choose voice masking. The prescriptions were contributed by 22 physicians, and all videos were checked by five physicians. The full judge prompts are given in the appendix, and the authors plan to release the scoring scripts.

Critical weaknesses
  • The reference answers for videos and documents were written by Gemini 3.1 Flash-Lite, and the judge, Gemma-4-31B, is also a Google model. On text, the references were written by physicians. There Gemini 3.1 Pro leads the best open-source model by only 2.4 points, and a text-only model, Granite 4.2 30B, scores higher than Gemini (55.1 against 52.7). On videos and documents, the references were written by Gemini. There its lead grows to 14.7 and 16.9 points, and on the document multiple-choice questions it reaches 82.7 against 29.2. Part of this advantage may come from the match between Gemini's answers and references written by a Gemini model. The paper does not test this, for example by having the video and document references written by another model or by physicians.
  • The judge is checked against a single physician, on 30 items (five per input type), with 74.0% agreement on pairwise comparisons and a Cohen's kappa of 0.48. This is moderate agreement, from one person, on a very small sample. The check covers only how responses are ranked, not whether the scores themselves are right. The open-ended results, which are the core of the paper, therefore rest on a judge with limited validation.
  • The judge prompt tells the judge to treat the reference as correct even when it is "limited, incomplete, boilerplate". It also caps clinical correctness at 4 when the reference gives no clear conclusion. The text references are physician replies taken from online consultation services and forums such as r/AskDocs. A response that is more complete or more accurate than such a reply is therefore scored as worse. The open-ended score measures agreement with one physician's reply rather than the quality of the answer.
  • About 26,000 of the 28,025 items come from existing public datasets. The text questions and answers are reused unchanged, and the 500 synthetic spoken questions are the same texts read by a speech synthesizer. The questions on images and breathing sounds, 13,519 items, are written by GPT-5.5 from the diagnostic label of each sample, so they are not questions asked by real users. For about half of the benchmark, the claim that every item is a consumer query therefore holds only in form.
  • The paper states that breathing sounds are recorded with phone or web microphones, excluding stethoscopes and clinical devices. Yet it includes 233 recordings from ICBHI 2017. According to the database's official page, ICBHI 2017 was recorded with Littmann and Meditron stethoscopes and an AKG microphone.
  • On breathing sounds, the two judges agree poorly (rank correlation 0.58), and the open-ended and multiple-choice rankings point in opposite directions (−0.14). In the multiple-choice version, which has eight options per question, the smallest model, Qwen2.5-Omni-3B, scores best: 46.9, above Gemini at 43.4. The paper does not report how the diagnoses are distributed, for example how many recordings are labeled healthy. It is therefore unclear whether a model can score well simply by always giving the most common answer.
  • Of the human recordings flagged by the automatic check, 13 were discarded because the speech recognizer, Whisper, failed to understand the speaker's pronunciation or accent. These are exactly the realistic, hard cases that a consumer benchmark should keep. The paper also does not report results separately for human and synthetic voices, so the effect of real speech cannot be seen.
  • Physicians review all videos, but only the 79 flagged documents. For images and audio, the paper describes only checks by an AI model and a human against the source labels, and does not say how many items, if any, were reviewed by physicians. Two image sources have public labels with no medical validation. The AI model used as auditor is not named. No agreement is reported among the five video reviewers or the six document reviewers. The prompts used to write the questions, answers and multiple-choice options are not given either: Appendix A.2 shows only the judge prompts.
  • Almost all the content is already public. The text questions and answers are reused unchanged, and the spoken questions read the same texts. The videos are public TikTok and YouTube posts, and the images and audio come from public datasets. Only the 116 prescriptions are new documents. The evaluated models may have seen much of this material during training. The paper neither checks for contamination nor compares results on new and reused items, so it cannot show whether the scores are inflated.
  • The 226 YouTube videos all come from a single channel, and each of the 108 TikTok videos yields three questions, so these items are not independent. The videos are released only as links to TikTok and YouTube (Ethics Statement). They can be removed at any time, so the benchmark may not be reproducible later. The Reproducibility Statement, in contrast, promises the release of the "media files" without excluding the videos.
  • The TikTok clips show patients with visible medical conditions who may be identifiable, posted on a public platform. The paper lists their basis only as "Public; creator-posted", and, unlike for the YouTube videos, no patient consent is documented. Using health content of private individuals in a benchmark raises privacy concerns, even if only links are released.
  • The paper does not describe how the inputs are processed or how the models are run. It gives no image resolution, no number of video frames or frame rate, no video length and no audio sampling settings. It also omits the prompt given to the models, the decoding settings, and how answers containing the model's internal reasoning are handled. The planned release covers the scoring scripts and the scripts that trim the videos, but not the code used to run the models (Reproducibility Statement). This affects the results. Nemotron-Nano-Omni scores 0 on one video because it prints its reasoning instead of an answer. It also obtains 23.8 on text against 60.3 on spoken questions drawn from the same sources. Such gaps seem to reflect the output format rather than medical ability. No tests with constrained decoding.
  • The multiple-choice options are written by a generative model that is not named. The paper does not say whether the order of the options is shuffled, or whether the correct options are spread evenly across positions. If they are not, a model's preference for certain positions can raise or lower its score.
  • Video clips keep their sound when it is medically relevant, such as coughing or abnormal breathing. Yet models without audio input are also evaluated on videos: GPT-4o and GPT-4o mini, and the vision-language models in Table 3. The paper does not say how many video items need the sound, or how these models are handled on them. Their video scores may therefore reflect missing input rather than weaker understanding.
  • No confidence interval or statistical test is reported, and the paper does not say whether models were run more than once. Some sets are small, such as 335 documents and 550 videos, and many differences between models are a few points: for example, 50.3 against 49.8 in the overall open-ended score. It is therefore unclear which differences are real.
  • In the introduction, one sentence is garbled: "the benchmark contains Samples map to over 25 geographic countries , with the some drawn from global or web-based sources". The best model, Gemini 3.1 Pro Preview, is a preview version, which makes its results hard to reproduce over time.
Questions for authors
  • How do Gemini and the other models score when the video and document references are written by another model or by physicians?
  • How well does the judge agree with several physicians, on a larger sample and on the scores themselves rather than only on rankings?
  • What are the results on the few new items compared with the reused ones?
  • Why are the ICBHI recordings, made with stethoscopes, included, and how are the diagnoses distributed in the breathing-sound set?
  • Are the multiple-choice options shuffled, and which model wrote them?
  • What are the results for human and synthetic voices separately?
  • How many videos need their sound, and how are models without audio input handled on them?
  • What image resolution, number of video frames, prompts and decoding settings are used, and how are answers containing internal reasoning handled?
Reviewer_XhvgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EverydayMedQA is a consumer-oriented medical question-answering benchmark containing 28,025 QA pairs across text, spoken questions, images, respiratory audio, video, and medical documents. It combines existing sources with newly collected recordings and prescriptions, using source-grounded QA generation, automated auditing, and varying levels of human and physician review. The benchmark supports open-ended evaluation with reference-based LLM judges and deterministic multiple-choice evaluation. Experiments with 23 models reveal uneven performance across input types and differences between answer generation and answer selection.

Critical strengths
  1. A useful consumer-facing evaluation resource. The benchmark brings several practically relevant health-information inputs into one evaluation suite. Its contribution is the breadth and organization of the resource, including spoken questions, respiratory recordings, short videos, and photographed documents. This is a worthwhile direction for evaluating assistants used by non-experts.
  2. Substantial curation and human involvement. The work includes 500 human speech recordings from 43 volunteers, prescriptions contributed by 22 physicians, and physician review of video QA. The source-by-source description appropriately distinguishes inherited annotations from newly reviewed material. These efforts add value beyond simply collecting existing datasets.
  3. Complementary evaluation formats and broad model coverage. Open-ended and multiple-choice versions expose different model behaviors. The evaluation spans general, medical, and modality-specific models, and includes both judge comparisons and a physician ranking check. These analyses make the benchmark a useful basis for further investigation.
Critical weaknesses
  1. Input type is confounded with task and scoring format. Image and respiratory-audio items primarily assess diagnosis, whereas the other subsets contain broader clinical-information questions. The open-ended rubrics differ, and multiple-choice scoring mixes eight-option single-select with strict multi-select exact match. Consequently, cross-subset score differences cannot be attributed solely to modality understanding. Paired analyses on the shared text and speech questions would strengthen the interpretation.
  2. Media dependence and answerability need stronger validation. Many QA pairs are generated using source labels and metadata. Question-only and shuffled-media controls would establish whether models need the associated media rather than textual cues. Separately, a clinically verified source label does not establish that the diagnosis is inferable from the restricted input shown to the model. Expert review should assess input sufficiency and appropriate uncertainty, especially when exact diagnosis matching receives the highest score.
  3. Judge validation is encouraging but limited. The physician check covers 370 responses to only 30 distinct questions, with one physician and five questions per input type. The reported 74.0% pairwise agreement and kappa of 0.481 provide useful initial support, but broader multi-physician assessment would strengthen confidence in clinical and safety scoring. Agreement between two judges from the same model family does not exclude shared biases. Reference-based safety scores should also be distinguished from comprehensive clinical safety assessment.
  4. Consumer-input eligibility needs clarification. The paper states that respiratory inputs exclude stethoscope and clinical-grade recordings, while Table 1 includes 233 ICBHI 2017 samples. The original database includes recordings acquired using stethoscopes and specialized microphones. The authors should explain which recordings were selected and how they satisfy the stated inclusion criteria. This issue concerns the scope of the consumer-facing claim rather than invalidating the entire resource.
Questions for authors
  1. How much performance remains with the media removed or shuffled? For the paired text and speech items, how do scores differ under matched questions and reference answers?
  2. Were experts asked whether each target conclusion is inferable from the model-visible input, rather than merely consistent with the source label? How are appropriate uncertainty and alternative clinically valid answers handled?
  3. Can you expand the physician validation across more independent questions and multiple raters, reporting agreement by input type and criterion?
  4. What acquisition-device filtering was applied to ICBHI 2017? Please reconcile its inclusion with the stated exclusion of clinical recording equipment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.