acceptodds
Under review as a conference paper at ICLR 2027

CARE-Med: A Capability-Resolved Benchmark for Medical Perception across Photographic and Physiological-Signal Images

Abstract

Medical multimodal large language models (MLLMs) are still compared mostly by a single aggregate accuracy, which reveals that a model errs but not which competency it lacks. Existing benchmarks also concentrate on sensor-reconstructed imaging (radiology, pathology, ultrasound, and microscopy), leaving camera-captured images and physiological-signal visualizations under-tested. We present CARE-Med, a 2,023-question Chinese-language benchmark over five under-tested domains: body fluids, in-vitro diagnostic (IVD) reagents, endoscopy and surgical fields, ex-vivo organs and tissues, and physiological-signal visualizations. Three of them (macroscopic body fluids, IVD-reagent readouts, and gross ex-vivo organs and tissues) have, to our knowledge, no dedicated vision–language evaluation, and a self-built four-level image taxonomy quantifies this coverage bias across seven prior benchmarks. Every question is mapped to leaves of a 34-leaf capability tree whose top level takes three knowledge types (factual, conceptual, procedural) from Bloom’s revised taxonomy, and every question, answer, and label is verified by clinicians. Evaluating 14 frontier and medical MLLMs, we find that a single score, even when split by knowledge type, is misleading: within each frontier model the three knowledge types differ by only 1–3 points, yet its capability leaves span more than 30 (GPT-5.5 ranges from 53% to 89% and reaches only 71% overall). The three frontier models fail on largely the same capability leaves (ρ≈0.8), a shared weakness in concrete visual determination alongside strong explanation, and medical-specialized models redistribute competence across the tree rather than closing this gap. CARE-Med turns an opaque score into a diagnosis of where medical perception breaks.

Primary area:
datasets and benchmarks
Keywords:
benchmark, evaluation, multimodal large language models, vision-language models, medical imaging, medical visual question answering, fine-grained evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oVmQOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents CARE-Med, a Chinese-language benchmark of 2,023 questions (1,060 multiple-select and 963 fill-in-the-blank) built on 1,430 images from five domains that the authors argue are under-tested by existing medical MLLM benchmarks: macroscopic body fluids, in-vitro diagnostic (IVD) reagent readouts, endoscopy and surgical fields, gross ex-vivo organs and tissues, and physiological-signal visualizations. A self-built 70-leaf image taxonomy is used to quantify the coverage of seven prior benchmarks and to motivate the choice of domains. Each question is mapped to leaves of a 34-leaf capability tree whose top level follows the factual/conceptual/procedural knowledge dimension of Bloom's revised taxonomy. Questions are generated by LLMs from a ground-truth description consolidated from three models' image-only captions, in two phases: one curated by clinicians throughout, and one largely automated with difficulty-targeted selection. Every final item is reported as clinician-verified. Evaluating 14 MLLMs, the authors report that the best model reaches 71.1%, that knowledge types barely separate difficulty while capability leaves span more than 30 points, that the three flagship models fail on largely the same capabilities (ρ≈0.8), and that medical-specialized models do not close the gap to general-purpose ones.

Critical strengths
  1. The paper targets image domains that are clinically relevant, increasingly encountered by consumer-facing models, and largely absent from existing benchmarks. The taxonomy-based analysis of seven prior benchmarks, audited by two medical annotators with high agreement, makes concrete how strongly prior benchmarks concentrate on radiology and microscopy. The authors are also appropriately careful not to claim domain novelty for endoscopy and ECG, where prior work exists.

  2. The motivation of moving from a single aggregate score toward capability-level diagnosis is well-founded, and the paper takes concrete steps in that direction: every item is mapped to explicit capability labels, and multiple-select and fill-in-the-blank formats are combined to reduce guessing.

Critical weaknesses
  1. It's hard to judge the quality of this benchmark, because the construction process leaves out key details. The paper keeps saying clinicians reviewed the data, but never says how many there were, what their qualifications are, or how the review was done. The image sources are also vague: we don't know which websites or books the images came from, or whether they are real clinical images. The biggest issue is the ground truth. It is built from captions written by three MLLMs, not from the original captions or figure legends, even though those were available and usually state what the specimen or diagnosis actually is. So the answer keys come from the same kind of models being tested, and the paper itself shows that these models tend to fail in the same places, which means their agreement doesn't guarantee correctness. Human checking is limited too: in Phase 1, content that all three models agreed on was never reviewed, and in Phase 2 every step before the final review was done by LLMs. The screening model, filtering rates and selection details are also not reported.

  2. The paper doesn't show that these domains are important, or that the images actually contain enough information to answer the questions. For body fluids and gross specimens, doctors usually reach conclusions through lab tests or pathology, not by looking at a photo. Yet many questions ask the model to identify a specimen, its cause, or the next clinical step from the image alone, and the answers were generated by models with the same limited information. For example, in Figure 11 it is unclear how two unlabeled tubes can be identified as blood and chylous fluid rather than bloody urine and chyluria. Without checking this, or comparing against human performance, we can't tell whether a model got a question wrong or the question itself has no clear answer.

Questions for authors
  1. How many clinicians participated in construction, and what were their specialties and years of experience? How were items assigned to them, was each item reviewed by more than one clinician, and was the answer key visible during review? Please report inter-reviewer agreement and, for each phase, the number of items, answer keys and capability labels that were corrected or removed.
  2. Which web collections, books and literature were the images drawn from, and what is the per-domain breakdown by source? what licenses can they be redistributed?
  3. Why were the source captions and figure legends not used to construct the ground truth? On a random sample of items, how often do the model-derived answer keys agree with the information stated in the source legend?
  4. In Phase 1, in what fraction of cases did all three captions agree, and what fraction of the final ground truth consists of unanimous content that was never reviewed by a clinician? Has the accuracy of the consolidated ground truth been audited against an independent reference?
  5. Which LLMs performed Phase-1 question generation and Phase-2 relevance scoring, claim verification and question screening? What were the removal rates at each stage, the weights of the selection score, and the target share of probe-incorrect items? Can the 14 models' results be reported separately for Phase 1 and Phase 2?
  6. What evidence supports the clinical importance of the five domains, for example their frequency in clinical workflows or in user queries to medical AI systems?
  7. For what fraction of items did clinicians judge that the image alone contains sufficient information to determine the answer? Could you report a blind clinician baseline per domain, in which clinicians answer without seeing the key, to separate model errors from underdetermined or mis-keyed items?
  8. For the item in Figure 11, what information in the image distinguishes blood and chylous effusion from haematuria and chyluria, and what supports the association between a turbid red appearance and intravascular haemolysis?

If the authors can address major parts of above questions, I would be willing to raise my score.

3 comments
AuthorsOpenReview ↗

Response to oVmQ (1/3): W1 - Clinical review, sources and GT

Thank you for your careful review. We address W1/W2 and Q1–Q8 in three comments. References are to the uploaded revised manuscript.

W1: Construction transparency and GT reliability (Q1–Q4).

Q1. Clinicians and review. Three licensed physicians participated. The two initial reviewers had laboratory-medicine and surgical backgrounds, each with at least five years of clinical experience. The third had a neurology background and at least ten years of experience.

Every QA entering final clinical review in either phase was assigned to the first two physicians. They independently answered the same version and assessed validity with the candidate key hidden. After submitting their judgments, they checked the stem, options, key and capability labels. The third physician checked evidence and helped adjudicate disagreements about retention, correction or removal. The original two confirmed substantive revisions.

Agreement below uses first independent judgments, before discussion, modification or adjudication. It measures inter-reviewer agreement, not accuracy or agreement with the candidate key.

MeasurePhase 1Phase 2
QA items entering final review9711,057
Answer agreement899/971 (92.6%)985/1,057 (93.2%)
Exact label-set agreement869/971 (89.5%)952/1,057 (90.1%)
QA items retained9671,056
QA items with stem/option corrections156
QA items with key corrections180128
QA items with label corrections112111
QA items removed, with their keys41

These are final-review counts, not all generated candidates. Correction categories overlap and cannot be added as distinct corrected items. Label corrections count QA items, not individual labels: 112/967 (11.6%) and 111/1,056 (10.5%) of retained items. The summaries do not specify the fill-in answer-equivalence rule, deduplicated corrections across categories, or label-level additions/removals, including labels removed with discarded items. The 4/1 removed keys accompany the same discarded QAs.

Q2. Sources and redistribution. Images were collected at scale from the Internet. Specific websites, collections, book and literature references cannot currently be recovered, so we cannot provide a verifiable per-domain source breakdown or verify each image's clinical provenance from those records. Image-associated alt text remains available, without restoring source references.

Original licences and image-specific redistribution permissions remain unverified. The full collection is unreleased. Our plan is controlled access: requesters state their purpose and accept terms specifying non-commercial research/education use and that we do not own the image copyrights. This notice is not rights-holder permission to redistribute.

Q3. Source text versus GT. Crawled text included uninformative, unrelated or insufficiently detailed descriptions. It was used for image–text relevance filtering, rather than adopted directly as GT; GT construction consolidated three image-only model captions.

On 100 randomly sampled final-benchmark images, one physician involved in construction compared the clinician-reviewed GT descriptions with retained alt text: 43/100 agreed. This is a GT-description comparison, not a direct per-QA answer-key audit. Reasons for the other 57 disagreements were not classified, so they cannot all be attributed to faulty source text.

Q4. Consensus and review coverage. Model agreement alone does not establish correctness. In Phase 1, clinicians reviewed the entire consolidated description, including shared content, against the image using medical expertise, focusing particularly on disagreements. Thus, unanimous content exempted from clinical review accounts for 0% of final GT; this is a review-coverage statement, not an accuracy estimate. The original wording did not make full-description review explicit.

For 150 randomly sampled Phase-1 images, Seed-1.8 automatically compared the original caption triplets:

Caption comparisonImages
All substantive content semantically equivalent0/150 (0.0%)
Only additions, omissions or compatible differences11/150 (7.3%)
At least one incompatible substantive judgment139/150 (92.7%)

One incompatible pair sufficed; all three pairs need not conflict. The absence of wholly equivalent triplets does not mean there was no shared content. Seed-1.8 was itself a caption generator, so this automated comparison is not independent verification. We have no separate consolidated-GT accuracy audit against an independent reference to report.

The revised manuscript clarifies sourcing and full-GT review in Section 3.4 and Appendix A/Table 2, adds the clinical protocol and statistics in A.1/Tables 4–5, and updates the Ethics statement.

AuthorsOpenReview ↗

Response to oVmQ (2/3): W1/Q5 - Construction and phase results

1. Models. GPT-5.4 generated the Phase-1 questions. Seed-1.8 performed Phase-2 relevance scoring, claim verification and question screening.

2. Removal rates. Phase-1 automated and clinician image screening excluded 10.88% and 18.1%, respectively (absolute counts unavailable in the summary). Rows 1–5 below concern Phase 2:

StageInputExcluded / not retainedRate
Image-text relevance, recorded batch6,662,565 records2,371,65735.60%
QA format/completeness174,533 output records3580.205%
LLM question screening174,175 QAs1,118 rejected; 42 processing errors0.642%; 0.0241%
Strong image-dependence requirement173,015 QAs200.0116%
Domain-scope restriction172,995 QAs6,5733.80%
Final clinical review, Phase 1971 QAs40.412%
Final clinical review, Phase 21,057 QAs10.0946%

Relevance filtering is batch-specific; format failures include failed/empty outputs. Processing errors and scope exclusions are not medical rejections. Screening left 173,015 usable QAs (0.666% excluded), including error-free records marked for manual review; that status does not mean clinical review was completed. Final selection contains 1,056 of 166,422 in-scope QAs; quota non-selection does not imply invalidity. These summaries are not an end-to-end flow: the final 1,056-item file is not the 1,057-item clinical-entry snapshot.

3. Selection scores and targets. Seed-1.6 was the difficulty probe. Scores were additive and domain-specific. IVD used coverage, support, then quota filling. Leaf terms for coverage (uncovered leaves), support (under-supported leaves) and fill (all leaves) were 1200+240/sqrt(f), 700+120*d+100/sqrt(f), and 35/[sqrt(f)*(1+c)], where f is pool frequency, d the support deficit, and c the count selected in that call. For k relevant leaves, coverage additionally awarded 40*max(k-1,0) and support 25*max(k-1,0). Minimum support was 2 where available. The probe term was +24/-10 for incorrect/correct candidates below target, changing to -6/+8 at target; other terms balanced format, difficulty, screening status, and repeated answers.

Other domains used these main coefficients:

TermBody fluidOrgans/tissuesEndoscopy/surgical field
Incorrect, below target72+2.5*min(g,10)72+2.5*min(g,10)70+2.5*min(g,10)
Incorrect, target reached-12-12-10
Correct, below/at target-16 / +10-16 / +10-14 / +8
Leaf base coefficient b141612
Tail-leaf extra coefficient t10118
Screening keep / review+10 / -6+10 / -6+9 / -4

Here g is the remaining incorrect-item count target; each leaf receives b/sqrt(c+1), plus t/sqrt(c+1) for designated tail leaves, with c including retained Phase-1 items and already selected additions. Multi-select difficulty contributes +14/+9/-10 for hard/medium/easy items. Additional terms reward underrepresented leaves and domain subgroups and discourage repeated answers; image/source and answer-pattern constraints also restrict selection. These coefficients are not a normalized global weight vector.

Probe-incorrect targets and achieved shares:

DomainFill-in targetFill-in actualMulti-select targetMulti-select actual
Body fluid37.7%66/175 (37.71%)28%45/161 (27.95%)
IVD75%97/129 (75.19%)75%55/81 (67.90%)
Organs/tissues70%99/142 (69.72%)60%59/98 (60.20%)
Endoscopy/surgical field70%106/151 (70.20%)60%71/119 (59.66%)

Targets were not guarantees: IVD multi-select achieved 67.90%, below 75%.

Appendix A (p.14) now distinguishes IVD's three-stage selector from the other rules. We also clarified the scope of the construction description in Section 3.4 (p.6) and the Reproducibility statement (p.10).

4. Results by phase. Accuracies reuse the existing per-item judgments:

ModelPhase 1, n=967 (%)Phase 2, n=1,056 (%)
GPT-5.572.7069.70
Seed-2.172.4964.58
GPT-5.462.7764.77
Gemini-3.1-Pro65.5661.17
Gemini-2.5-Pro65.2558.52
Seed-1.864.2252.65
Lingshu-32B37.6452.56
Qwen3-VL-8B30.6144.41
InternVL3.5-8B34.0240.81
GLM-4.6V-Flash27.2041.19
MedGemma-27B23.2735.23
MiniCPM-V-4.518.1026.89
Citrus-V-8B14.9926.80
HuatuoGPT-V-34B14.7919.70

Phase 1 includes 544 photographic and 423 physiological-signal questions; Phase 2 has 1,056 photographic questions. Domain/format compositions differ, so these descriptive differences do not isolate the causal effect of probe selection.

AuthorsOpenReview ↗

Response to oVmQ (3/3): W2 - Clinical relevance and validation

W2: Clinical relevance and answerability (Q6–Q8).

Q6. Clinical relevance. These domains contain visual tasks within clinical workflows; they complement laboratory and pathological investigations, rather than replace them:

  • Body fluids: Purulent pleural fluid indicates empyema and the need for drainage (BTS).
  • IVD: Rapid-test interpretation requires recognizing control/test lines, faint reactions and invalid results (WHO).
  • Endoscopy/surgical fields: The critical view of safety requires identifying the cystic duct and artery (guideline). A national census reported 44.5 million digestive endoscopies in China in 2019 (Jiao et al.).
  • Gross specimens: Gross assessment guides sampling of heterogeneous, necrotic and viable tissue (CAP, p.14).
  • Physiological signals: ECG interpretation informs identification and triage of suspected acute coronary syndromes (ANZCOR).

These support clinical relevance, not frequencies of image queries to medical AI. The revised Introduction replaces "ubiquitous in practice" with this workflow-based rationale.

Q7. Independent blind clinician baseline. Both physicians judged 285/300 sampled items (95.0%) sufficient using the image, question, applicable options and medical knowledge. We randomly sampled 60 retained QAs per domain. Two physicians uninvolved in construction independently answered with the same inputs as the models, without keys, source captions, model outputs or each other's answers. Answers and sufficiency judgments were locked before follow-up review.

Scoring used the original keys and benchmark rules: exact set match for multiple-select and Seed-1.8 equivalence grading for fill-in, with respondent identity withheld.

Domain (60 QAs each)Both judged sufficientPhysician A correctPhysician B correct
Body fluids58 (96.7%)55 (91.7%)54 (90.0%)
IVD56 (93.3%)53 (88.3%)54 (90.0%)
Endoscopy/surgical fields57 (95.0%)54 (90.0%)55 (91.7%)
Gross specimens56 (93.3%)54 (90.0%)53 (88.3%)
Physiological signals58 (96.7%)56 (93.3%)55 (91.7%)

Overall, physicians A/B scored 272/300 (90.7%)/271/300 (90.3%), versus GPT-5.5's 212/300 (70.7%) from existing formal evaluation records. Senior-physician follow-up identified seven insufficient/ambiguous items and three incorrect/incomplete keys. Excluding these ten from the comparison, A/B scored 268/290 (92.4%)/267/290 (92.1%), versus 211/290 (72.8%) for GPT-5.5; both physicians answered 72 of its 79 errors correctly. The identified item problems do not account for the remaining human–model gap in this sample. These equally allocated domain samples do not give a precise full-benchmark rate. Appendix A.2/Table 6 reports the assessment; Section 6 notes its limited coverage.

For controlled inspection, we have uploaded CARE-Med_25_reviewer_samples.zip with 25 selected complete image–question–reference-answer examples and independent clinician responses. This illustrative package is separate from the random 300-item assessment.

Q8. Figure 11. The original text-only summary omitted relevant appearance: the left liquid is relatively uniform red and turbid; the right is milky-white and turbid. These features do not establish specimen origin or exclude haematuria/chyluria. The question asks about appearance and possible clinical significance.

A's "may have occurred" expresses a possible association: free haemoglobin can discolour plasma pink/red in intravascular haemolysis (case report). This explanation is conditional on plasma; turbidity is nonspecific. The photograph neither establishes plasma nor distinguishes intravascular from post-collection haemolysis, which requires additional evidence.

B's "can be seen in" links milky turbidity to chylous effusions without excluding chyluria. G instead asserts that both specimens are urine, which the image does not establish. AB is the clinician-reviewed reference key on the intended basis of possible associations, not confirmed specimen identity or diagnosis. We have expanded Figure 11's description (Appendix G, p.34) with the colours and turbidity.

Reviewer_nHs2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CARE-Med, a medical benchmark for VLMs, curated to evaluate the underrepresented categories and domains in the existing medical benchmarks. The authors use a human AI collaboration pipeline to gather, generate, and filter the data. The dataset consists of multiple-choice and fill-in-the-blank questions on 70 sub-categories. The authors evaluate a wide range of proprietary and open-source VLMs on the dataset and compare and discuss their performances in each category. They further show that the distribution of performance per catgory is different between general VLMs and medical ones.

Critical strengths
  1. The dataset is well-motivated, and it focuses on underrepresented types/categories of medical questions. The dataset is designed to address the limitations of current benchmarks and provide better coverage of medical tasks, which is valuable for the community.

  2. The use of human verification in the generation pipeline is an important addition that increases the quality of samples. Human verification is usually time-consuming and costly, but results in better filtering.

Critical weaknesses
  1. The writing is a bit rough in some parts (e.g., lines 22 and 248), with either overly complicated sentences or vague explanations, which makes it harder for the reader to follow. Another example is Figure 5, where the bottom right contains a small correlation matrix that is not clearly described in the caption, making it difficult for the reader to interpret. Moreover, the generation pipeline is missing some important details. For example, the paper does not provide the prompts used for dataset generation, even in the appendix, which reduces reproducibility. Also, in my opinion, the paper should discuss the samples rejected by the experts and provide at least the rejection rate and, where applicable, the types of failures that the clinicians detected. Overall, the presentation issues and missing details make it hard to recommend the paper for acceptance.

  2. The ablation in which the image is not passed to the model is interesting, but it also reveals a potential weakness in the dataset. Since GPT-5.5 achieves around 50% accuracy without seeing the images, this raises concerns about sample difficulty and the extent to which the benchmark evaluates the models’ multimodal capabilities rather than their ability to answer using textual cues and prior knowledge. These concerns about the samples’ effectiveness in identifying multimodal failure modes are further amplified by the lack of access to the dataset or even a subset of samples (a few samples are present in the appendix, but without their corresponding images).

Questions for authors
  1. What are the clinician rejection rates at the different stages of Phase 1 and during the final verification of Phase 2? Are there any systematic failure modes in the automated pipeline (e.g., difficulty generating valid samples from a certain category)?

  2. What is the expected performance of a random predictor? This helps me understand whether the accuracy of GPT 5.5 without images is due to randomness or whether some questions can be answered without images.

  3. Did the clinicians filter samples based on their difficulty? If so, what criteria did they use?

2 comments
AuthorsOpenReview ↗

Response to nHs2 (1/2): W1/Q1/Q3 - Construction and revisions

Thank you for the constructive review. We group overlapping weaknesses and questions into two comments. Manuscript references below use the uploaded revised manuscript.

W1/Q1. Clinical filtering and recurring pipeline failures.

Clinical review stageRemoved / reviewedRejection rate
Phase 1: image screeningAbsolute counts unavailable in the summary18.1%
Phase 1: final QA review4 / 9710.412%
Phase 2: final QA review1 / 1,0570.0946%

The QA denominators count items entering final clinical review, which included corrections as well as removals, not all generated candidates. Clinicians also reviewed the complete consolidated Phase-1 GT descriptions; a separate GT-review rejection rate is unavailable in the summaries. Clinician-detected problems included inferences beyond the image evidence and ambiguous questions, without counts by failure type. Appendix A.1 now states these rates and qualitative findings alongside the correction/removal counts in Table 5. Image and QA rejection rates have different denominators and should not be added.

Across seven Phase-2 screening batches (174,175 candidate QAs), Seed-1.8 rejected 1,118 (0.642%); 42 processing errors were counted separately. The main automatic flags were incorrect or non-unique answer keys (941), answers exceeding image/GT evidence (157), and questions answerable without the image (128). These overlapping flag counts are automated judgments, not clinician-adjudicated error counts. The screen excluded candidates flagged for hard failures; selected items then underwent final clinical review and correction or removal. Screening rejection rates ranged from 0.529% to 0.857% across these candidate pools; these are screening-stage rates, not end-to-end generation failure rates for the five domains.

Q3. Did clinicians select for difficulty?

During Phase-1 question-validity review, clinicians did not deliberately filter by difficulty; they assessed validity and checked the stem, options, key and capability labels. Challenge was addressed through question design. Early experience with single-choice questions suggested insufficient challenge and too much scope for guessing, motivating multiple-select and short-answer fill-in questions. The generation prompt also used distractors involving visually similar entities, overinterpretation of salient features and confusion about specimen state, while requiring image dependence and grounding in clinician-reviewed GT. These were design choices, not a quantified guarantee of increased difficulty.

Phase 2 used algorithmic difficulty-targeted selection: Seed-1.6 probe performance and automated difficulty labels were combined with capability coverage, format and balancing constraints. Both probe-incorrect and probe-correct items were selected. This preceded the final clinical validity review and should be distinguished from clinicians judging difficulty.

W1. Writing, Figure 5 and generation prompts.

  • Writing: We split the sentence around original line 22 into statements about the three under-tested domains and taxonomy-based coverage. Around original line 248, we separated taxonomy coverage from the clinician check of classifier assignments, retaining the statistics.
  • Figure 5: The small inset values are now displayed in a larger numerical matrix below the heatmap, with full model names. The caption explains that cells give pairwise Spearman correlations between GPT-5.5, Gemini-3.1-Pro and Seed-2.1's accuracy rankings across all 34 capability leaves, including marked small-sample leaves. This measures relative profile similarity, not agreement on individual answers. Values are unchanged.
  • QA-generation prompt: Appendix H (pp. 36–41) contains the full English translation of the Chinese prompt used for QA generation, including capability definitions, the output schema and demonstrations. We have also uploaded CARE-Med_QA_generation_prompt.zip, containing the original Chinese prompt and its English translation. Generation used Chinese; the English version documents that prompt.
AuthorsOpenReview ↗

Response to nHs2 (2/2): W2/Q2 - Random baselines and image contribution

W2/Q2. Random baselines and what the no-image results imply.

GPT-5.5's no-image accuracy is well above the defined random baselines below. It indicates that non-image information can support correct answers, particularly on multiple-select items; it should not be attributed to these guessing strategies alone.

We computed format-specific baselines on all 2,023 items:

  • Multiple-select: uniformly sample a non-empty subset of the K available options. Exact-set accuracy is 1/(2^K - 1), averaged over actual option counts; the predictor is not told the number of correct options.
  • Fill-in: uniformly sample another fill-in QA record from the same benchmark domain, excluding all records sharing the target image, and output its reference answer. This preserves answer frequencies without using the target question's content. It is an empirical answer-prior baseline, not a universal chance level for free-form answers. We used ten seeds (0–9) and the original Seed-1.8 medical-equivalence scoring prompt and settings.
FormatNRandom (%)GPT-5.5, no image (%)GPT-5.5, with image (%)
Multiple-select1,0600.78368.4077.36
Fill-in9630.405 ± 0.14228.9764.28
Overall2,0230.603 ± 0.06849.6371.13

Multiple-select random accuracy is an exact expectation; fill-in reports mean ± sample SD across seeds, not a confidence interval. Overall results are weighted by item count, with the multiple-select expectation fixed. GPT-5.5 scores reuse the original evaluations on the same items.

Providing images raises accuracy by 21.50 percentage points overall, with a larger gain for fill-in (+35.31) than multiple-select (+8.96). Thus, images contribute substantially, while text and medical prior knowledge also contribute. The comparison does not separate those non-image sources or imply that every item requires an image (Section 5.3; Table 12).

We also examined the 822 items that none of GPT-5.5, Gemini-3.1-Pro and Seed-2.1 answered correctly in their no-image runs. GPT-5.5 reaches 48.3% with images on this subset, so it remains challenging even with visual input. Across all 14 models, full-benchmark and subset rankings correlate at Spearman rho = 0.965 (Appendix F, Table 20). This supports robustness of the overall model comparison to this restriction. The subset is defined by observed model behavior; it does not establish that all capability-level patterns are unchanged or that individual errors have purely visual causes.

W2. Complete samples for direct inspection.

We have uploaded CARE-Med_25_reviewer_samples.zip, with complete images, original Chinese questions, applicable options and reference answers for 25 examples, five per domain. Two clinicians uninvolved in construction independently answered the same image-question-option inputs, locking initial answers before seeing the keys or each other's responses. The package includes their initial answers and available individual answerability/disagreement records, with supplementary judgments separated from initial responses. These purposefully selected examples enable inspection of concrete items; they are not a random quality audit or a guarantee that all items are unambiguous.

Reviewer_kg4NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CARE-Med, a Chinese multimodal evaluation benchmark for medical photographic images and physiological-signal visualizations. The benchmark contains 1,430 different images and 2,023 questions, including 1,060 multiple-select questions and 963 fill-in-the-blank questions, covering five domains: macroscopic appearance of body fluids, in vitro diagnostic reagents, endoscopic and surgical fields, ex vivo organs and tissues, and physiological signal visualization. The authors also evaluated 14 general and medical speciality multimodal models. The paper further provides results for no-image comparisons, results by question type and image domain, paired significance tests, permutation tests, and analysis on subsets of questions where all three flagship models answered incorrectly in the absence of an image.

Critical strengths
  1. Specific data coverage contribution, good human quality control. CARE-Med jointly evaluated five categories of medical image objects, among which the application contexts of macroscopic body fluids, in vitro diagnostic reagent readings, and ex vivo specimens are clearly defined. The authors not only clearly explained the reasons for choosing these domains but also analyzed seven existing benchmarks using a unified image classification system. Two medical annotators independently verified the classification results of 480 stratified sampled images. Data construction did not rely entirely on automatic generation. The paper explicitly reports that all final questions, reference answers, and capability labels in both stages were reviewed and revised by clinical staff, and discloses the roles of the generation model, difficulty probe, and scoring model.

  2. The experiment went beyond a single leaderboard, with multiple complementary checks for key descriptive findings. The paper evaluated 14 models and reported results per image domain, capability leaf, and question type. The no-image control showed that the overall scores of the three flagship models decreased by approximately 21.5–23.3 percentage points, indicating that image input substantially contributed to the results. The authors also re-evaluated all models on a subset where all three flagship models answered incorrectly without images, reporting a high correlation between the overall ranking and the complete benchmark. Furthermore, the statistical analysis was comprehensive. While providing the overall confidence intervals, the authors also conducted paired tests and multiple comparison corrections on model differences, performed permutation tests to preserve sample size and label co-occurrence structure for the score ranges of the capability leaves, and additionally performed permutations within question types.

Critical weaknesses
  1. The validity of capability labels and differences in question composition still limit the interpretation of "capability diagnosis." The multi-label design itself is reasonable, but errors in the same question can simultaneously contribute to the scores of multiple leaves: if a question involves identification, localization, and decision-making, the final incorrect answer alone cannot determine which ability is problematic. Different capability leaves may also have different image domains, question types, and construction stages. The two stages contributed 967 and 1,056 questions respectively, with the second stage favoring questions answered incorrectly by the difficulty probe Seed-1.6. The authors addressed some of these issues through permutations that preserve label co-occurrence and permutations within question types. These controls are indeed reasonable, but the manuscript does not report whether the main ability differences remain stable after further controls on the image domain and construction stage. Furthermore, the manuscript does not report the results of handling independent label consistency or discrepancies in capability labels. The double-checking of 480 images per person was for the classification of existing benchmark images and cannot replace the verification of CARE-Med capability labels.

  2. Lack of Independent Calibration in LLM Scoring. All 963 fill-in-the-blank questions were scored by Seed-1.8; while the multiple-select section primarily used deterministic option set matching, out of 14,840 model–item pairs across 14 models, 1,502 (approximately 10.1%) used LLM fallback scoring. The authors did disclose relatively strict medical equivalence scoring rules, setting the scoring temperature to 0 and blinding the identity of the answering model; the reference answers were also verified by clinical personnel. However, verifying the correctness of questions and reference answers is different from verifying whether a model's answer should be judged as correct. The manuscript does not report an independent audit of the model output scoring by humans, therefore it remains difficult to determine whether synonyms, omissions of key parts, unit conversions, negations, or incomplete steps were consistently handled.

Questions for authors
  1. Regarding scoring calibration: Can you provide a set of human scoring audits covering different models, image domains, and capability leaves? Please report the consistency between human scoring and Seed-1.8, the main types of misjudgments, and explain whether the ranking of similar models and the main ability conclusions changed after human correction or replacement of judges.

  2. Regarding capability labels and main explanations: Were the capability labels reviewed by independent experts? Are there records of consistency and disagreement handling? After controlling for image domains, question types, and the construction phase, does the pattern of "relatively weaker performance in specific visual judgments and relatively stronger performance in explanation-based questions" still exist?

  3. Regarding multiple questions on the same image and statistical stability: 2,023 questions come from 1,430 different images. After resampling by image, are the key model differences, capability leaf score ranges, and correlations between model leaf rankings stable?

  4. Regarding independent verification of question quality: The manuscript acknowledges the lack of a human problem-solving baseline, and the example page does not show real images due to authorization restrictions. Is it possible to provide reviewers with a small batch of complete images-questions-reference answers in a controlled manner, along with the results of independent clinical personnel answering the questions under the same information conditions, or a record of any ambiguities or disagreements? This would help distinguish between genuine modeling difficulties and questions with answerability issues.

3 comments
AuthorsOpenReview ↗

Response to kg4N (1/3): W1/Q2 - Capability labels and composition

Thank you for the thoughtful review. We address the overlapping weaknesses and questions in three comments. Manuscript references use the uploaded revised manuscript.

W1/Q2. What capability diagnosis means.

A final incorrect answer on a multi-label item cannot identify which constituent capability caused the error. Our leaf scores summarize accuracy on questions involving each capability, describing performance across task groups rather than assigning an individual error uniquely to one label. Section 3.3 now makes this scope explicit, with corresponding wording aligned in the abstract, introduction and conclusion.

Independent label judgments and disagreement handling. Candidate labels were primarily proposed by the question-generation model and judged separately by two clinicians involved in construction. On the same item version, exact agreement on the complete label set was 869/971 (89.5%) in Phase 1 and 952/1,057 (90.1%) in Phase 2, before discussion, modification or adjudication. A third clinician checked evidence and participated in resolving disagreements; the original two confirmed substantive revisions. Label corrections affected 112 and 111 QA items, respectively, not that many individual labels. This is independent judgment between construction clinicians, not external blind reannotation, and is separate from the 480-image taxonomy audit of existing benchmarks (Appendix A.1).

After controlling composition, the aggregate contrast remains positive. We compared the manuscript-named visual-judgment leaves V={2,6,10,19,24} with explanation-related leaves E={14,20,21,22}. These sets follow the existing findings and were fixed before computing the adjusted results; they do not partition all 34 leaves. We jointly stratified by image domain, question format and construction phase. Each QA counts once per group; dual-group items enter both means. Within-stratum E-minus-V differences use common weights for both groups and all models, proportional to each stratum's unique QA union.

ModelMain: 14 strata (pp)Coverage check: 17 strata (pp)
GPT-5.5+12.24+12.62
Gemini-3.1-Pro+14.29+15.03
Seed-2.1+16.60+16.24

The main analysis requires at least five distinct images per group per stratum: 1,073 QA items on 943 images (V=974, E=228, overlap=129). The subsequent coverage check includes all 17 strata with both groups present: 1,172 QA items on 1,014 images, covering 97.7% of the 1,200 items carrying these labels. Adding 99 sparse-stratum items changes each contrast by less than 0.74 points. The remaining 28 are Phase-1 endoscopy fill-in items in a stratum with no E items, so no within-stratum contrast is available.

On the same nine eligible strata with identical weights, retaining versus excluding dual-group items gives +14.20/+16.72, +16.01/+19.18 and +14.35/+13.55 points for GPT-5.5, Gemini-3.1-Pro and Seed-2.1. The direction remains positive, but these are aggregate descriptive contrasts: not every stratum has the same direction. The table reports point estimates without a significance claim. These controls remain conditional on the constructed benchmark and do not remove all within-phase effects of probe-based selection.

Appendix F.1 and Table 22 (p. 31) now report the composition and sensitivity analyses, with a short pointer in Section 5.2.

AuthorsOpenReview ↗

Response to kg4N (2/3): W2/Q1 - Human scoring calibration

W2/Q1. Human calibration of model-output scoring.

Reference-answer verification and calibration of model-output scoring are distinct. We report audits of both fill-in scoring and multiple-select LLM fallback scoring.

Fill-in audit. We sampled 20 questions uniformly without replacement per image domain: 100 questions and 300 existing responses from GPT-5.5, Seed-2.1 and Gemini-3.1-Pro. Sampling did not use capability labels, model answers or scores. Two clinically trained experts independently graded responses under the medical-equivalence rubric, seeing the question, reference answer and anonymized response, but not model identity, Seed scores or capability groups. Initial expert agreement was 289/300 (96.3%); a third senior expert reviewed all 11 disagreements. Human judgments were locked before comparison with Seed-1.8, followed by review of human–judge disagreements.

Human–Seed agreement was 286/300 (95.3%). The seven false negatives involved equivalent terminology/phrasing (4) or numerical/unit expressions (3). The seven false positives involved missing necessary qualifiers or procedural steps (5), or conflicting statements alongside correct content (2). Agreement by domain was 58/60 for body fluids, 57/60 for IVD, 56/60 for endoscopy/surgical fields, 58/60 for ex vivo specimens and 57/60 for physiological signals.

Multiple-select fallback audit. We randomly sampled 150 model–item response records from the 1,502 fallback-scored responses. Two experts scored them independently, with review by a third. Agreement with final human judgments was 149/150 (99.3%). The sole error was a false negative for MedGemma-27B (human 1, Seed-1.8 0). Applying only this known correction would change its full-benchmark accuracy from 597/2,023 (29.51%) to 598/2,023 (29.56%), leaving its rank of 11th unchanged; this does not estimate unobserved errors.

Effect on rankings and capability comparisons. For the fill-in audit, arrows denote Seed-1.8 scoring → human scoring; values are percentages.

ModelAccuracy (n=100)Human–Seed agreementV accuracy (n=63)E accuracy (n=24)
GPT-5.565 → 649557.1 → 58.779.2 → 83.3
Seed-2.162 → 629654.0 → 55.675.0 → 70.8
Gemini-3.1-Pro58 → 599550.8 → 52.470.8 → 70.8

The sample ordering is unchanged. We reuse V={2,6,10,19,24} and E={14,20,21,22}; all nine leaves are represented. Each question counts once within a group, with 12 dual-group questions in both. E remains higher than V for each flagship under human scoring. Excluding overlap preserves the pooled direction: V-only accuracy changes from 75/153 (49.0%) to 77/153 (50.3%), and E-only from 27/36 (75.0%) to 26/36 (72.2%). These denominators represent three models answering 51 and 12 questions, respectively.

These are descriptive checks on the audited responses, not full-benchmark human rescoring or evidence that nearby models are statistically separated. The two audits have different sampling designs and are not combined into one agreement estimate. Appendix F.2 and Table 23 (p. 32) report the audits and sensitivity results in the revised manuscript.

AuthorsOpenReview ↗

Response to kg4N (3/3): Q3/Q4 - Image-cluster stability and clinical verification

Q3. Stability after resampling by image.

Using the original judgments on all 2,023 questions, we performed 10,000 unstratified image-cluster bootstrap resamples (seed 0), each drawing 1,430 images with replacement. All questions, labels and model outcomes for an image move together. Accuracy remains question-weighted, preserving the original scoring convention and pairing between models.

Model differences. For the same 15 comparisons reported in the manuscript, simultaneous 95% intervals exclude zero for 10 pairs, matching the set identified by the original Holm-adjusted tests. GPT-5.5 exceeds GPT-5.4 by 7.32 percentage points [4.40, 10.23], and Seed-2.1 exceeds Seed-1.8 by 10.18 [7.18, 13.19]. Gemini-3.1-Pro versus Gemini-2.5-Pro remains unresolved: 1.53 [−1.75, 4.82]. GPT-5.5 versus Seed-2.1 is 2.77 [0.04, 5.49], with the lower bound close to zero. Five comparisons remain statistically unresolved.

Leaf score ranges. We retained the original 31 leaves with at least 30 questions. Joint 95% bands over the three flagships' 93 leaf scores give conservative maximum-minus-minimum accuracy bounds:

ModelObserved spread (pp)Conservative spread bounds
GPT-5.536.00[9.76, 58.66]
Gemini-3.1-Pro33.99[4.76, 67.58]
Seed-2.130.94[4.54, 60.29]

Positive lower bounds support within-model differences across leaves, not a guarantee that the underlying spreads exceed 30 points.

Leaf-rank correlations. Across all 34 leaves, Spearman correlations and marginal 95% image-bootstrap intervals are 0.847 [0.458, 0.869] for GPT-5.5/Gemini-3.1-Pro, 0.813 [0.499, 0.882] for GPT-5.5/Seed-2.1, and 0.787 [0.412, 0.843] for Gemini-3.1-Pro/Seed-2.1. All remain positive on the fixed 31-leaf subset (lower bounds 0.555, 0.565, 0.492). The 34-leaf intervals use 9,995 valid resamples; five lacked leaf 28 and were left undefined without dropping a leaf or drawing replacements. The 31-leaf analysis uses all 10,000 draws. These support shared profiles, not fixed ranks for every leaf. Appendix F.3 and Tables 24–25 (pp. 33–34) now report these methods and results.

Q4. Complete samples and independent clinical responses.

We have uploaded CARE-Med_25_reviewer_samples.zip for controlled reviewer inspection: 25 complete examples, five per domain, with 11 multiple-select and 14 fill-in questions. It includes original images, Chinese questions, applicable options, benchmark keys, readable question/clinician-response sheets and JSONL. These are purposefully selected examples, not a random quality audit.

Two clinicians uninvolved in construction independently answered the same image-question-option inputs as the models, locking initial answers before seeing keys or each other's responses. The package preserves both initial answers and available individual answerability judgments. B's supplementary judgments are separated from locked responses; the records are not a joint adjudication or evidence that all 25 items are unambiguous.

Separately, the blind clinician baseline randomly sampled 60 retained items per domain (300 total). Two physicians uninvolved in construction answered under the same input and blinding conditions; both judged 285/300 (95.0%) sufficient. Senior-physician follow-up identified seven insufficient or ambiguous items and three incorrect or incomplete keys. On the remaining 290 items, physician accuracies were 92.4% and 92.1%, versus GPT-5.5's 72.8%, using exact-set scoring for multiple-select and Seed-1.8 equivalence grading for fill-in. Of GPT-5.5's 79 errors, 72 were answered correctly by both physicians. Thus, the identified item problems do not account for the remaining human–model gap in this sample. The revised manuscript reports the 300-item assessment in Appendix A.2 (Table 6) and its limited coverage in Section 6. These analytical exclusions do not mean the benchmark or its keys have been changed.

Comments on the paper

3 comments

Dear AC,

I cannot see the authors’ rebuttal. Could you please check whether it is visible to reviewers?

AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank the reviewers for their careful reading and constructive feedback. We have uploaded the revised manuscript, incorporating the clarifications and additional evidence summarized below. Our reviewer-specific responses address the individual concerns in detail.

  • Clinical review and independent baseline. Appendix A.1 and Tables 4–5 now document the construction clinicians' qualifications, review procedures, answer/label agreement, and corrections/removals. Separately, two physicians uninvolved in construction completed a blind assessment of 300 randomly sampled questions, 60 per domain, using the same inputs as the models. Both initially judged 285/300 (95.0%) sufficient. Appendix A.2 and Table 6 report the per-domain baseline and follow-up findings on answerability and reference keys.

  • Human scoring calibration. We audited both LLM-scoring paths. Agreement with final human judgments was 286/300 (95.3%) for responses from three flagship models to 100 sampled fill-in questions, and 149/150 (99.3%) in a separate random sample of multiple-select fallback responses. In the fill-in audit, human scoring preserved the sample model ordering and the explanation-over-visual accuracy pattern for the named capability groups. Appendix F.2 and Table 23 report the audits, error types and sensitivity results, with their sample-specific scope.

  • Capability interpretation and robustness. Section 3.3 and the corresponding abstract, introduction and conclusion now clarify that leaf scores describe performance across capability-related task groups, rather than uniquely identifying the cause of an individual error. For the named visual-judgment and explanation-related groups, the aggregate accuracy contrast remains positive for all three flagships after jointly controlling domain, question format and construction phase (Appendix F.1, Table 22; Section 5.2). Image-cluster resampling supports within-model leaf differences and shared leaf rankings, while retaining uncertainty for close model comparisons (Appendix F.3, Tables 24–25).

  • Presentation and reproducibility. We clarified the clinical rationale, construction procedures, sourcing limitations and controlled-access plans. We simplified the passages identified by the reviewer, displayed Figure 5's correlation values in a larger numerical matrix below the heatmap with an explanatory caption, and expanded Figure 11's appearance description. Appendix H provides the full English translation of the Chinese QA-generation prompt; generation used Chinese.

For controlled reviewer inspection, we have uploaded CARE-Med_25_reviewer_samples.zip, containing 25 complete image–question–reference-answer examples and independent clinician responses and individual judgments. These selected examples are separate from the random 300-item assessment. The reviewer-specific replies also provide format-specific random baselines alongside the no-image results, phase-separated results for all 14 models, and construction-stage filtering and selection details.

We appreciate your patience while we completed the additional clinician assessments, and apologize for posting our responses later in the discussion period.

AuthorsOpenReview ↗

General Response and Summary of Revisions (Updated)

We have further revised the manuscript to clarify its contributions, construction procedures and supporting evidence. This update focuses on the manuscript changes beyond our earlier General Response.

  • Clearer contribution structure. The introduction now distinguishes four contributions: capability-resolved evaluation, taxonomy-based coverage analysis, the clinician-verified five-domain benchmark, and findings from 14 MLLMs. Sections 3.2–3.3 clarify the different roles of the image taxonomy and capability tree: measuring image coverage and describing performance on capability-related tasks.

  • Clearer construction and evaluation procedures. Section 3.4 separates consolidated image descriptions, generated questions and reference keys, and final clinical review. Table 2 names the confirmed models for the relevant generation and screening stages. Section 4 specifies the answering inputs—image, question and applicable options—and explains that the no-image comparison measures the contribution of images beyond question text and medical priors.

  • More visible validation evidence. Section 3.4 now summarizes the independent clinician baseline, and Section 4 reports human scoring-audit agreement. Appendix A.2 includes the matched clinician–GPT-5.5 comparison, including results after excluding identified problematic items. Section 5.3 connects the main performance findings to the image-cluster checks.

The clinical validation, scoring audits, composition controls and resampling results were reported in our earlier responses; their full protocols, findings and scope remain available in Appendices A.1–A.2 and F.1–F.3.

For reviewer inspection, CARE-Med_rebuttal_supplements.zip contains CARE-Med_25_reviewer_samples.zip (25 complete image–question–reference-answer examples with independent clinician responses and individual judgments) and CARE-Med_QA_generation_prompt.zip (the Chinese prompt used for generation and its English translation). The selected examples are separate from the random 300-question clinical assessment. The English prompt is also provided in Appendix H.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.