- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper builds MRareBench, a benchmark for diagnosing rare diseases from clinical text plus medical images. The cases come from PMC case reports. There are two tracks. T1 asks the model to list a top-10 differential diagnosis (300 cases). T2 asks the model to describe the image evidence for a disease (608 items). The main idea is to turn inputs on and off: with or without images, with or without a text description of the image findings (FD/LC/TO), and with or without the disease name (Grd/NoDx/Rec). The authors test 37 MLLMs and report three things. Images help somewhat. Models describe the evidence much better once they are told the disease, indicating being good at describing evidence does not mean the model can find the right disease on its own.
Critical strengths
- The on/off input design is simple and clean, and it gives useful finding. Adding a single sentence that describes the image findings raises closed-source Recall@1 from 29.3 to 42.5. This is a good warning for other medical VQA benchmarks that put image findings into the question text.
- The evaluation is large (37 models) and the appendix is detailed. The filtering rules are listed one by one (Table A2), two judges are used, and there is an attempt at a memorization check by publication year.
Critical weaknesses
-
It is not clear what the multimodal part adds, since the images may be replaced by text. The benchmark is meant to test multimodal rare-disease diagnosis, but every T1 case comes with an expert-written findings text describing its images (cases without one were removed). The paper never tests context + findings text without images (TO also removes the findings text), so we can't tell whether the image adds anything beyond its text description. The paper's own numbers suggest it may not: in every cohort, adding the findings text beyond image even helps more than adding the images (closed-source +13.2 vs +11.5, open-source +9.7 vs +5.8, medical +7.8 vs +5.0). For many models the image gain is only 1–3 points on 300 cases (e.g., GPT-4o +2.3, Qwen2.5-VL-3B −0.3), with no significance test. If the text is enough, the benchmark splits into image reading, which general medical VQA benchmarks cover, and text-based rare-disease diagnosis, which RareBench and RareArena cover. Also, as far as I know, many rare diseases are confirmed by genetic testing rather than imaging; the Orphanet analysis the paper cites finds roughly 70% are genetic. Since the benchmark only keeps case reports with figures and leans toward rare tumors and pathology, "images matter" is partly true by construction and may not hold for rare diseases in general.
-
Many models are tested, but some of the most important baselines are missing. The newest model is Gemini-3.5-Flash (May 2026), a Flash-tier model, and the OpenAI and Anthropic models stop at GPT-5.2 and Opus 4.6, even though the authors used GPT-5.4 to build and judge the benchmark. Models have improved a lot in 2026, and this Flash model already reaches 82.0% R@10 and 80.5 on T2 Grd. So it's hard to tell whether the benchmark is close to saturated, or how much of the PMC material newer models have already seen in training. That makes the long-term value of the benchmark unclear. Further, there is no reference point from human experts or existing rare-disease tools.The clinicians in the paper only check item quality and never take the test, and there is no comparison with standard tools such as HPO-based ranking tools (e.g., Phenomizer, LIRICAL), Without these, it's hard to judge whether 54.7% R@1 is good or bad.
Questions for authors
- Could you add a view with context + findings text but no images, and report how close it gets to FD for each model?
- Is the main goal of the benchmark to test image reading or rare-disease reasoning? If it is image reading, how does T1 measure it, since T1 only scores the final diagnosis and the paper itself concludes that the bottleneck is hypothesis formation rather than reading?
- Could you break down the 227 diseases by type (e.g., rare tumors, genetic syndromes, immune/inflammatory) and by how they are usually confirmed (imaging, pathology, genetic testing)? Do the results differ across these groups?
- Could you add a few of the strongest current models, even on a subset? Since the best flash model already reaches 82.0% R@10 and 80.5 on T2 Grd, how much headroom is left?
- For newer models, could you run a contamination check that matches each model's training cutoff, or a simple probe (show the image, ask for the paper title, caption, or PMCID)? Which MedCaseReasoning split do the cases come from?
- Could you add a human baseline on a subset (e.g., general doctors and specialists, under both LC and TO), and compare against standard rare-disease tools such as HPO-based ranking tools or the agentic system you cite (Zhao et al., 2026)?
- In Table A3, 97 of 120 sampled items are marked "revise" and 68 of 120 have "fixable" leakage. Were these fixes applied to the released set?
2 comments
Q7: Status of the fixes in Table A3
Yes. Table A3 validates the construction pipeline and is not a list of outstanding edits. Clinical reviewers re-assigned the pipeline's labels on a stratified sample, and their labels agreed with the pipeline's on 39–40 of 40 items. "Revise" and "fixable" mark items that needed correction, and these corrections are applied in the released benchmark. We are sorry for the confusion and will make this clearer.
Other issues
We thank the reviewer for the suggestions on a findings-text view without images, a breakdown by disease category and confirmation route, and the evaluation of newer models, and will take them into account in the next update.
GPT-5.4 was the newest model available when the benchmark was built. It was used in benchmark construction, and GPT-5.4-mini served as the primary judge, so we left it out of the evaluated models to avoid self-evaluation.
We are grateful for the reviewer's comments and insightful suggestions. Once again, thank you!
References
[1] Zuo, Y., Qu, S., Li, Y., et al. MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding. In Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 80961–80990, 2025.
[2] Chen, H., Zhao, Z., Zhou, S., et al. RareArena: a comprehensive benchmark dataset unveiling the potential of large language models in rare disease diagnosis. The Lancet Digital Health, 8(2), 2026.
We thank the reviewer for the careful reading and detailed comments, and for the chance to clarify several points below.
The goal is rare-disease diagnosis from clinical context and images. The forward-diagnosis task, T1, asks only for a ranked list of ten diagnoses, so it scores the diagnostic outcome. The evidence-verification task, T2, asks the model to describe the image findings that support a disease, with the disease name either given or withheld. Because 402 of the 608 T2 questions come from the same case reports as 289 T1 cases, we can ask whether a model that reports the right evidence for a case can also name the disease on its own.
Separate image-reading and text-based rare-disease benchmarks cannot answer this question, because they do not share cases. Section 4.4 shows that among model–case pairs with strong evidence reporting, only 26.3% also get the top-ranked diagnosis right. When the diagnosis is missed, the correct disease is absent from the ten candidates in about 73% of cases. We present difficulty in forming the right diagnostic hypothesis as an interpretation of these outcomes, not as a separately measured reasoning step.
We agree with the reviewer's observation. MRareBench includes only case reports with usable images and removes cases that the text alone can answer, so the resulting cohort is enriched for diseases with characteristic radiological or pathological patterns. The image gain should therefore be read within this setting rather than generalized to all rare diseases, and we will state this scope and the resulting disease composition explicitly. Within this setting, the benchmark measures how well models use visual evidence that is diagnostically important, and open-source and medical models gain only about five to six points from the images on average. The image gain is positive for 36 of the 37 models and is largest for the strongest ones: Recall@1 rises from 26.0 to 54.7 for Gemini-3.5-Flash and from 27.7 to 48.7 for GPT-5 when the images are added. A genetic cause and a visual presentation are not in conflict: proposing a diagnosis from the clinical picture and the images precedes confirmatory testing, and this is the step we evaluate.
We agree that room for improvement is an important question, and that the strongest model scores highly on some metrics. However, these scores depend on how leniently success is defined. Recall@10 counts a case as solved if the correct disease appears anywhere among ten candidates; we report it for completeness, as a lenient measure. Under stricter definitions the picture changes: Gemini-3.5-Flash reaches 82.0% Recall@10 but 54.7% top-1 accuracy, and the closed-source average top-1 accuracy is 29.3%. Similarly, its T2 score of 80.5 is obtained with the disease name given and falls to 63.6 when the name is withheld. High scores under a lenient criterion or a supplied diagnosis do not mean that independent diagnosis is close to solved.
Existing medical benchmarks handle contamination in different ways, for example by rewriting questions to reduce leakage risk [1]. Exposure to public case reports nonetheless remains difficult to rule out for models whose training corpora are undisclosed. RareArena [2], a prior rare-disease benchmark built from PMC case reports, compares performance on cases published before and after the knowledge cutoff for four models with reported cutoffs. Our audit in Appendix E uses publication year instead of model-specific cutoffs and covers 27 general-purpose models. It examines three publication-year-based indicators of potential memorization. The correlation between publication year and correctness is not significant for any model at p < 0.05, newer-report gains are not significantly correlated with model strength, and model rankings stay stable across publication periods, with Spearman correlations of at least 0.83 with the overall ranking. These indicators provide indirect evidence; a comparison aligned with each model's knowledge cutoff or a source-recall probe, as the reviewer suggests, would complement this audit.
HPO-based tools such as Phenomizer and LIRICAL rank diseases from structured phenotype terms and do not take images, so they are not directly comparable in the image-grounded setting that is our primary view. We agree that a clinician reference would be informative, and we will take this suggestion into account in the next version.