- Rating
- 2(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents CARE-Med, a Chinese-language benchmark of 2,023 questions (1,060 multiple-select and 963 fill-in-the-blank) built on 1,430 images from five domains that the authors argue are under-tested by existing medical MLLM benchmarks: macroscopic body fluids, in-vitro diagnostic (IVD) reagent readouts, endoscopy and surgical fields, gross ex-vivo organs and tissues, and physiological-signal visualizations. A self-built 70-leaf image taxonomy is used to quantify the coverage of seven prior benchmarks and to motivate the choice of domains. Each question is mapped to leaves of a 34-leaf capability tree whose top level follows the factual/conceptual/procedural knowledge dimension of Bloom's revised taxonomy. Questions are generated by LLMs from a ground-truth description consolidated from three models' image-only captions, in two phases: one curated by clinicians throughout, and one largely automated with difficulty-targeted selection. Every final item is reported as clinician-verified. Evaluating 14 MLLMs, the authors report that the best model reaches 71.1%, that knowledge types barely separate difficulty while capability leaves span more than 30 points, that the three flagship models fail on largely the same capabilities (ρ≈0.8), and that medical-specialized models do not close the gap to general-purpose ones.
Critical strengths
-
The paper targets image domains that are clinically relevant, increasingly encountered by consumer-facing models, and largely absent from existing benchmarks. The taxonomy-based analysis of seven prior benchmarks, audited by two medical annotators with high agreement, makes concrete how strongly prior benchmarks concentrate on radiology and microscopy. The authors are also appropriately careful not to claim domain novelty for endoscopy and ECG, where prior work exists.
-
The motivation of moving from a single aggregate score toward capability-level diagnosis is well-founded, and the paper takes concrete steps in that direction: every item is mapped to explicit capability labels, and multiple-select and fill-in-the-blank formats are combined to reduce guessing.
Critical weaknesses
-
It's hard to judge the quality of this benchmark, because the construction process leaves out key details. The paper keeps saying clinicians reviewed the data, but never says how many there were, what their qualifications are, or how the review was done. The image sources are also vague: we don't know which websites or books the images came from, or whether they are real clinical images. The biggest issue is the ground truth. It is built from captions written by three MLLMs, not from the original captions or figure legends, even though those were available and usually state what the specimen or diagnosis actually is. So the answer keys come from the same kind of models being tested, and the paper itself shows that these models tend to fail in the same places, which means their agreement doesn't guarantee correctness. Human checking is limited too: in Phase 1, content that all three models agreed on was never reviewed, and in Phase 2 every step before the final review was done by LLMs. The screening model, filtering rates and selection details are also not reported.
-
The paper doesn't show that these domains are important, or that the images actually contain enough information to answer the questions. For body fluids and gross specimens, doctors usually reach conclusions through lab tests or pathology, not by looking at a photo. Yet many questions ask the model to identify a specimen, its cause, or the next clinical step from the image alone, and the answers were generated by models with the same limited information. For example, in Figure 11 it is unclear how two unlabeled tubes can be identified as blood and chylous fluid rather than bloody urine and chyluria. Without checking this, or comparing against human performance, we can't tell whether a model got a question wrong or the question itself has no clear answer.
Questions for authors
- How many clinicians participated in construction, and what were their specialties and years of experience? How were items assigned to them, was each item reviewed by more than one clinician, and was the answer key visible during review? Please report inter-reviewer agreement and, for each phase, the number of items, answer keys and capability labels that were corrected or removed.
- Which web collections, books and literature were the images drawn from, and what is the per-domain breakdown by source? what licenses can they be redistributed?
- Why were the source captions and figure legends not used to construct the ground truth? On a random sample of items, how often do the model-derived answer keys agree with the information stated in the source legend?
- In Phase 1, in what fraction of cases did all three captions agree, and what fraction of the final ground truth consists of unanimous content that was never reviewed by a clinician? Has the accuracy of the consolidated ground truth been audited against an independent reference?
- Which LLMs performed Phase-1 question generation and Phase-2 relevance scoring, claim verification and question screening? What were the removal rates at each stage, the weights of the selection score, and the target share of probe-incorrect items? Can the 14 models' results be reported separately for Phase 1 and Phase 2?
- What evidence supports the clinical importance of the five domains, for example their frequency in clinical workflows or in user queries to medical AI systems?
- For what fraction of items did clinicians judge that the image alone contains sufficient information to determine the answer? Could you report a blind clinician baseline per domain, in which clinicians answer without seeing the key, to separate model errors from underdetermined or mis-keyed items?
- For the item in Figure 11, what information in the image distinguishes blood and chylous effusion from haematuria and chyluria, and what supports the association between a turbid red appearance and intravascular haemolysis?
If the authors can address major parts of above questions, I would be willing to raise my score.
3 comments
Response to oVmQ (2/3): W1/Q5 - Construction and phase results
1. Models. GPT-5.4 generated the Phase-1 questions. Seed-1.8 performed Phase-2 relevance scoring, claim verification and question screening.
2. Removal rates. Phase-1 automated and clinician image screening excluded 10.88% and 18.1%, respectively (absolute counts unavailable in the summary). Rows 1–5 below concern Phase 2:
| Stage | Input | Excluded / not retained | Rate |
|---|---|---|---|
| Image-text relevance, recorded batch | 6,662,565 records | 2,371,657 | 35.60% |
| QA format/completeness | 174,533 output records | 358 | 0.205% |
| LLM question screening | 174,175 QAs | 1,118 rejected; 42 processing errors | 0.642%; 0.0241% |
| Strong image-dependence requirement | 173,015 QAs | 20 | 0.0116% |
| Domain-scope restriction | 172,995 QAs | 6,573 | 3.80% |
| Final clinical review, Phase 1 | 971 QAs | 4 | 0.412% |
| Final clinical review, Phase 2 | 1,057 QAs | 1 | 0.0946% |
Relevance filtering is batch-specific; format failures include failed/empty outputs. Processing errors and scope exclusions are not medical rejections. Screening left 173,015 usable QAs (0.666% excluded), including error-free records marked for manual review; that status does not mean clinical review was completed. Final selection contains 1,056 of 166,422 in-scope QAs; quota non-selection does not imply invalidity. These summaries are not an end-to-end flow: the final 1,056-item file is not the 1,057-item clinical-entry snapshot.
3. Selection scores and targets. Seed-1.6 was the difficulty probe. Scores were additive and domain-specific. IVD used coverage, support, then quota filling. Leaf terms for coverage (uncovered leaves), support (under-supported leaves) and fill (all leaves) were 1200+240/sqrt(f), 700+120*d+100/sqrt(f), and 35/[sqrt(f)*(1+c)], where f is pool frequency, d the support deficit, and c the count selected in that call. For k relevant leaves, coverage additionally awarded 40*max(k-1,0) and support 25*max(k-1,0). Minimum support was 2 where available. The probe term was +24/-10 for incorrect/correct candidates below target, changing to -6/+8 at target; other terms balanced format, difficulty, screening status, and repeated answers.
Other domains used these main coefficients:
| Term | Body fluid | Organs/tissues | Endoscopy/surgical field |
|---|---|---|---|
| Incorrect, below target | 72+2.5*min(g,10) | 72+2.5*min(g,10) | 70+2.5*min(g,10) |
| Incorrect, target reached | -12 | -12 | -10 |
| Correct, below/at target | -16 / +10 | -16 / +10 | -14 / +8 |
| Leaf base coefficient b | 14 | 16 | 12 |
| Tail-leaf extra coefficient t | 10 | 11 | 8 |
| Screening keep / review | +10 / -6 | +10 / -6 | +9 / -4 |
Here g is the remaining incorrect-item count target; each leaf receives b/sqrt(c+1), plus t/sqrt(c+1) for designated tail leaves, with c including retained Phase-1 items and already selected additions. Multi-select difficulty contributes +14/+9/-10 for hard/medium/easy items. Additional terms reward underrepresented leaves and domain subgroups and discourage repeated answers; image/source and answer-pattern constraints also restrict selection. These coefficients are not a normalized global weight vector.
Probe-incorrect targets and achieved shares:
| Domain | Fill-in target | Fill-in actual | Multi-select target | Multi-select actual |
|---|---|---|---|---|
| Body fluid | 37.7% | 66/175 (37.71%) | 28% | 45/161 (27.95%) |
| IVD | 75% | 97/129 (75.19%) | 75% | 55/81 (67.90%) |
| Organs/tissues | 70% | 99/142 (69.72%) | 60% | 59/98 (60.20%) |
| Endoscopy/surgical field | 70% | 106/151 (70.20%) | 60% | 71/119 (59.66%) |
Targets were not guarantees: IVD multi-select achieved 67.90%, below 75%.
Appendix A (p.14) now distinguishes IVD's three-stage selector from the other rules. We also clarified the scope of the construction description in Section 3.4 (p.6) and the Reproducibility statement (p.10).
4. Results by phase. Accuracies reuse the existing per-item judgments:
| Model | Phase 1, n=967 (%) | Phase 2, n=1,056 (%) |
|---|---|---|
| GPT-5.5 | 72.70 | 69.70 |
| Seed-2.1 | 72.49 | 64.58 |
| GPT-5.4 | 62.77 | 64.77 |
| Gemini-3.1-Pro | 65.56 | 61.17 |
| Gemini-2.5-Pro | 65.25 | 58.52 |
| Seed-1.8 | 64.22 | 52.65 |
| Lingshu-32B | 37.64 | 52.56 |
| Qwen3-VL-8B | 30.61 | 44.41 |
| InternVL3.5-8B | 34.02 | 40.81 |
| GLM-4.6V-Flash | 27.20 | 41.19 |
| MedGemma-27B | 23.27 | 35.23 |
| MiniCPM-V-4.5 | 18.10 | 26.89 |
| Citrus-V-8B | 14.99 | 26.80 |
| HuatuoGPT-V-34B | 14.79 | 19.70 |
Phase 1 includes 544 photographic and 423 physiological-signal questions; Phase 2 has 1,056 photographic questions. Domain/format compositions differ, so these descriptive differences do not isolate the causal effect of probe selection.
Response to oVmQ (3/3): W2 - Clinical relevance and validation
W2: Clinical relevance and answerability (Q6–Q8).
Q6. Clinical relevance. These domains contain visual tasks within clinical workflows; they complement laboratory and pathological investigations, rather than replace them:
- Body fluids: Purulent pleural fluid indicates empyema and the need for drainage (BTS).
- IVD: Rapid-test interpretation requires recognizing control/test lines, faint reactions and invalid results (WHO).
- Endoscopy/surgical fields: The critical view of safety requires identifying the cystic duct and artery (guideline). A national census reported 44.5 million digestive endoscopies in China in 2019 (Jiao et al.).
- Gross specimens: Gross assessment guides sampling of heterogeneous, necrotic and viable tissue (CAP, p.14).
- Physiological signals: ECG interpretation informs identification and triage of suspected acute coronary syndromes (ANZCOR).
These support clinical relevance, not frequencies of image queries to medical AI. The revised Introduction replaces "ubiquitous in practice" with this workflow-based rationale.
Q7. Independent blind clinician baseline. Both physicians judged 285/300 sampled items (95.0%) sufficient using the image, question, applicable options and medical knowledge. We randomly sampled 60 retained QAs per domain. Two physicians uninvolved in construction independently answered with the same inputs as the models, without keys, source captions, model outputs or each other's answers. Answers and sufficiency judgments were locked before follow-up review.
Scoring used the original keys and benchmark rules: exact set match for multiple-select and Seed-1.8 equivalence grading for fill-in, with respondent identity withheld.
| Domain (60 QAs each) | Both judged sufficient | Physician A correct | Physician B correct |
|---|---|---|---|
| Body fluids | 58 (96.7%) | 55 (91.7%) | 54 (90.0%) |
| IVD | 56 (93.3%) | 53 (88.3%) | 54 (90.0%) |
| Endoscopy/surgical fields | 57 (95.0%) | 54 (90.0%) | 55 (91.7%) |
| Gross specimens | 56 (93.3%) | 54 (90.0%) | 53 (88.3%) |
| Physiological signals | 58 (96.7%) | 56 (93.3%) | 55 (91.7%) |
Overall, physicians A/B scored 272/300 (90.7%)/271/300 (90.3%), versus GPT-5.5's 212/300 (70.7%) from existing formal evaluation records. Senior-physician follow-up identified seven insufficient/ambiguous items and three incorrect/incomplete keys. Excluding these ten from the comparison, A/B scored 268/290 (92.4%)/267/290 (92.1%), versus 211/290 (72.8%) for GPT-5.5; both physicians answered 72 of its 79 errors correctly. The identified item problems do not account for the remaining human–model gap in this sample. These equally allocated domain samples do not give a precise full-benchmark rate. Appendix A.2/Table 6 reports the assessment; Section 6 notes its limited coverage.
For controlled inspection, we have uploaded CARE-Med_25_reviewer_samples.zip with 25 selected complete image–question–reference-answer examples and independent clinician responses. This illustrative package is separate from the random 300-item assessment.
Q8. Figure 11. The original text-only summary omitted relevant appearance: the left liquid is relatively uniform red and turbid; the right is milky-white and turbid. These features do not establish specimen origin or exclude haematuria/chyluria. The question asks about appearance and possible clinical significance.
A's "may have occurred" expresses a possible association: free haemoglobin can discolour plasma pink/red in intravascular haemolysis (case report). This explanation is conditional on plasma; turbidity is nonspecific. The photograph neither establishes plasma nor distinguishes intravascular from post-collection haemolysis, which requires additional evidence.
B's "can be seen in" links milky turbidity to chylous effusions without excluding chyluria. G instead asserts that both specimens are urine, which the image does not establish. AB is the clinician-reviewed reference key on the intended basis of possible associations, not confirmed specimen identity or diagnosis. We have expanded Figure 11's description (Appendix G, p.34) with the colours and turbidity.
Response to oVmQ (1/3): W1 - Clinical review, sources and GT
Thank you for your careful review. We address W1/W2 and Q1–Q8 in three comments. References are to the uploaded revised manuscript.
W1: Construction transparency and GT reliability (Q1–Q4).
Q1. Clinicians and review. Three licensed physicians participated. The two initial reviewers had laboratory-medicine and surgical backgrounds, each with at least five years of clinical experience. The third had a neurology background and at least ten years of experience.
Every QA entering final clinical review in either phase was assigned to the first two physicians. They independently answered the same version and assessed validity with the candidate key hidden. After submitting their judgments, they checked the stem, options, key and capability labels. The third physician checked evidence and helped adjudicate disagreements about retention, correction or removal. The original two confirmed substantive revisions.
Agreement below uses first independent judgments, before discussion, modification or adjudication. It measures inter-reviewer agreement, not accuracy or agreement with the candidate key.
These are final-review counts, not all generated candidates. Correction categories overlap and cannot be added as distinct corrected items. Label corrections count QA items, not individual labels: 112/967 (11.6%) and 111/1,056 (10.5%) of retained items. The summaries do not specify the fill-in answer-equivalence rule, deduplicated corrections across categories, or label-level additions/removals, including labels removed with discarded items. The 4/1 removed keys accompany the same discarded QAs.
Q2. Sources and redistribution. Images were collected at scale from the Internet. Specific websites, collections, book and literature references cannot currently be recovered, so we cannot provide a verifiable per-domain source breakdown or verify each image's clinical provenance from those records. Image-associated alt text remains available, without restoring source references.
Original licences and image-specific redistribution permissions remain unverified. The full collection is unreleased. Our plan is controlled access: requesters state their purpose and accept terms specifying non-commercial research/education use and that we do not own the image copyrights. This notice is not rights-holder permission to redistribute.
Q3. Source text versus GT. Crawled text included uninformative, unrelated or insufficiently detailed descriptions. It was used for image–text relevance filtering, rather than adopted directly as GT; GT construction consolidated three image-only model captions.
On 100 randomly sampled final-benchmark images, one physician involved in construction compared the clinician-reviewed GT descriptions with retained alt text: 43/100 agreed. This is a GT-description comparison, not a direct per-QA answer-key audit. Reasons for the other 57 disagreements were not classified, so they cannot all be attributed to faulty source text.
Q4. Consensus and review coverage. Model agreement alone does not establish correctness. In Phase 1, clinicians reviewed the entire consolidated description, including shared content, against the image using medical expertise, focusing particularly on disagreements. Thus, unanimous content exempted from clinical review accounts for 0% of final GT; this is a review-coverage statement, not an accuracy estimate. The original wording did not make full-description review explicit.
For 150 randomly sampled Phase-1 images, Seed-1.8 automatically compared the original caption triplets:
One incompatible pair sufficed; all three pairs need not conflict. The absence of wholly equivalent triplets does not mean there was no shared content. Seed-1.8 was itself a caption generator, so this automated comparison is not independent verification. We have no separate consolidated-GT accuracy audit against an independent reference to report.
The revised manuscript clarifies sourcing and full-GT review in Section 3.4 and Appendix A/Table 2, adds the clinical protocol and statistics in A.1/Tables 4–5, and updates the Ethics statement.