acceptodds
Withdrawn from ICLR 2027

MRareBench: A Multimodal Rare-Disease Benchmark for Evidence-Diagnosis Correspondence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) now approach specialist-level performance on chest radiograph interpretation, dermatology and pathology image classification, and medical visual question answering. These gains, however, come from common diseases, where the clinical narrative alone often narrows the differential. Rare diseases reverse these conditions. Each disease is individually uncommon and sparsely documented, prevalence gives little guidance across thousands of candidates, and the decisive findings are frequently visual and distributed across several images and modalities. Rare disease therefore offers a stringent test of whether a MLLM reaches a diagnosis from evidence or from prior familiarity. Current rare disease benchmarks describe patients in text and omit medical images, while multimodal medical benchmarks supply images but concentrate on common conditions. To address this limitation, we introduce MRareBench, the first benchmark for multimodal rare disease diagnosis. MRareBench comprises 908 evaluation items spanning multiple medical images and diverse imaging modalities, and pairs forward diagnosis with evidence verification under controlled disclosure of the diagnosis. Evaluation of 37 closed-source, open-source, and medical MLLMs shows that visual evidence improves rare disease diagnosis. A gap remains between evidence recognition and open diagnosis. MLLMs often identify relevant visual evidence after receiving the rare disease diagnosis but are less reliable in using multimodal evidence to identify the disease independently. The results indicate a difficulty in inferring rare disease diagnoses from multimodal evidence. MRareBench supplies the measurement needed to advance multimodal diagnosis in rare-disease settings.

Primary area:
datasets and benchmarks
Keywords:
multimodal large language models, rare-disease diagnosis, medical benchmark, visual grounding, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sCDkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper builds MRareBench, a benchmark for diagnosing rare diseases from clinical text plus medical images. The cases come from PMC case reports. There are two tracks. T1 asks the model to list a top-10 differential diagnosis (300 cases). T2 asks the model to describe the image evidence for a disease (608 items). The main idea is to turn inputs on and off: with or without images, with or without a text description of the image findings (FD/LC/TO), and with or without the disease name (Grd/NoDx/Rec). The authors test 37 MLLMs and report three things. Images help somewhat. Models describe the evidence much better once they are told the disease, indicating being good at describing evidence does not mean the model can find the right disease on its own.

Critical strengths
  1. The on/off input design is simple and clean, and it gives useful finding. Adding a single sentence that describes the image findings raises closed-source Recall@1 from 29.3 to 42.5. This is a good warning for other medical VQA benchmarks that put image findings into the question text.
  2. The evaluation is large (37 models) and the appendix is detailed. The filtering rules are listed one by one (Table A2), two judges are used, and there is an attempt at a memorization check by publication year.
Critical weaknesses
  1. It is not clear what the multimodal part adds, since the images may be replaced by text. The benchmark is meant to test multimodal rare-disease diagnosis, but every T1 case comes with an expert-written findings text describing its images (cases without one were removed). The paper never tests context + findings text without images (TO also removes the findings text), so we can't tell whether the image adds anything beyond its text description. The paper's own numbers suggest it may not: in every cohort, adding the findings text beyond image even helps more than adding the images (closed-source +13.2 vs +11.5, open-source +9.7 vs +5.8, medical +7.8 vs +5.0). For many models the image gain is only 1–3 points on 300 cases (e.g., GPT-4o +2.3, Qwen2.5-VL-3B −0.3), with no significance test. If the text is enough, the benchmark splits into image reading, which general medical VQA benchmarks cover, and text-based rare-disease diagnosis, which RareBench and RareArena cover. Also, as far as I know, many rare diseases are confirmed by genetic testing rather than imaging; the Orphanet analysis the paper cites finds roughly 70% are genetic. Since the benchmark only keeps case reports with figures and leans toward rare tumors and pathology, "images matter" is partly true by construction and may not hold for rare diseases in general.

  2. Many models are tested, but some of the most important baselines are missing. The newest model is Gemini-3.5-Flash (May 2026), a Flash-tier model, and the OpenAI and Anthropic models stop at GPT-5.2 and Opus 4.6, even though the authors used GPT-5.4 to build and judge the benchmark. Models have improved a lot in 2026, and this Flash model already reaches 82.0% R@10 and 80.5 on T2 Grd. So it's hard to tell whether the benchmark is close to saturated, or how much of the PMC material newer models have already seen in training. That makes the long-term value of the benchmark unclear. Further, there is no reference point from human experts or existing rare-disease tools.The clinicians in the paper only check item quality and never take the test, and there is no comparison with standard tools such as HPO-based ranking tools (e.g., Phenomizer, LIRICAL), Without these, it's hard to judge whether 54.7% R@1 is good or bad.

Questions for authors
  1. Could you add a view with context + findings text but no images, and report how close it gets to FD for each model?
  2. Is the main goal of the benchmark to test image reading or rare-disease reasoning? If it is image reading, how does T1 measure it, since T1 only scores the final diagnosis and the paper itself concludes that the bottleneck is hypothesis formation rather than reading?
  3. Could you break down the 227 diseases by type (e.g., rare tumors, genetic syndromes, immune/inflammatory) and by how they are usually confirmed (imaging, pathology, genetic testing)? Do the results differ across these groups?
  4. Could you add a few of the strongest current models, even on a subset? Since the best flash model already reaches 82.0% R@10 and 80.5 on T2 Grd, how much headroom is left?
  5. For newer models, could you run a contamination check that matches each model's training cutoff, or a simple probe (show the image, ask for the paper title, caption, or PMCID)? Which MedCaseReasoning split do the cases come from?
  6. Could you add a human baseline on a subset (e.g., general doctors and specialists, under both LC and TO), and compare against standard rare-disease tools such as HPO-based ranking tools or the agentic system you cite (Zhao et al., 2026)?
  7. In Table A3, 97 of 120 sampled items are marked "revise" and 68 of 120 have "fixable" leakage. Were these fixes applied to the released set?
2 comments

We thank the reviewer for the careful reading and detailed comments, and for the chance to clarify several points below.

Q2, W1: Image reading versus rare-disease diagnosis

The goal is rare-disease diagnosis from clinical context and images. The forward-diagnosis task, T1, asks only for a ranked list of ten diagnoses, so it scores the diagnostic outcome. The evidence-verification task, T2, asks the model to describe the image findings that support a disease, with the disease name either given or withheld. Because 402 of the 608 T2 questions come from the same case reports as 289 T1 cases, we can ask whether a model that reports the right evidence for a case can also name the disease on its own.

Separate image-reading and text-based rare-disease benchmarks cannot answer this question, because they do not share cases. Section 4.4 shows that among model–case pairs with strong evidence reporting, only 26.3% also get the top-ranked diagnosis right. When the diagnosis is missed, the correct disease is absent from the ten candidates in about 73% of cases. We present difficulty in forming the right diagnostic hypothesis as an interpretation of these outcomes, not as a separately measured reasoning step.

Q3, W1: Disease types and clinical scope

We agree with the reviewer's observation. MRareBench includes only case reports with usable images and removes cases that the text alone can answer, so the resulting cohort is enriched for diseases with characteristic radiological or pathological patterns. The image gain should therefore be read within this setting rather than generalized to all rare diseases, and we will state this scope and the resulting disease composition explicitly. Within this setting, the benchmark measures how well models use visual evidence that is diagnostically important, and open-source and medical models gain only about five to six points from the images on average. The image gain is positive for 36 of the 37 models and is largest for the strongest ones: Recall@1 rises from 26.0 to 54.7 for Gemini-3.5-Flash and from 27.7 to 48.7 for GPT-5 when the images are added. A genetic cause and a visual presentation are not in conflict: proposing a diagnosis from the clinical picture and the images precedes confirmatory testing, and this is the step we evaluate.

Q4, W2: Benchmark saturation

We agree that room for improvement is an important question, and that the strongest model scores highly on some metrics. However, these scores depend on how leniently success is defined. Recall@10 counts a case as solved if the correct disease appears anywhere among ten candidates; we report it for completeness, as a lenient measure. Under stricter definitions the picture changes: Gemini-3.5-Flash reaches 82.0% Recall@10 but 54.7% top-1 accuracy, and the closed-source average top-1 accuracy is 29.3%. Similarly, its T2 score of 80.5 is obtained with the disease name given and falls to 63.6 when the name is withheld. High scores under a lenient criterion or a supplied diagnosis do not mean that independent diagnosis is close to solved.

Q5, W2: Training-data contamination

Existing medical benchmarks handle contamination in different ways, for example by rewriting questions to reduce leakage risk [1]. Exposure to public case reports nonetheless remains difficult to rule out for models whose training corpora are undisclosed. RareArena [2], a prior rare-disease benchmark built from PMC case reports, compares performance on cases published before and after the knowledge cutoff for four models with reported cutoffs. Our audit in Appendix E uses publication year instead of model-specific cutoffs and covers 27 general-purpose models. It examines three publication-year-based indicators of potential memorization. The correlation between publication year and correctness is not significant for any model at p < 0.05, newer-report gains are not significantly correlated with model strength, and model rankings stay stable across publication periods, with Spearman correlations of at least 0.83 with the overall ranking. These indicators provide indirect evidence; a comparison aligned with each model's knowledge cutoff or a source-recall probe, as the reviewer suggests, would complement this audit.

Q6, W2: Human and tool reference points

HPO-based tools such as Phenomizer and LIRICAL rank diseases from structured phenotype terms and do not take images, so they are not directly comparable in the image-grounded setting that is our primary view. We agree that a clinician reference would be informative, and we will take this suggestion into account in the next version.

Q7: Status of the fixes in Table A3

Yes. Table A3 validates the construction pipeline and is not a list of outstanding edits. Clinical reviewers re-assigned the pipeline's labels on a stratified sample, and their labels agreed with the pipeline's on 39–40 of 40 items. "Revise" and "fixable" mark items that needed correction, and these corrections are applied in the released benchmark. We are sorry for the confusion and will make this clearer.

Other issues

We thank the reviewer for the suggestions on a findings-text view without images, a breakdown by disease category and confirmation route, and the evaluation of newer models, and will take them into account in the next update.

GPT-5.4 was the newest model available when the benchmark was built. It was used in benchmark construction, and GPT-5.4-mini served as the primary judge, so we left it out of the evaluated models to avoid self-evaluation.

We are grateful for the reviewer's comments and insightful suggestions. Once again, thank you!

References

[1] Zuo, Y., Qu, S., Li, Y., et al. MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding. In Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 80961–80990, 2025.

[2] Chen, H., Zhao, Z., Zhou, S., et al. RareArena: a comprehensive benchmark dataset unveiling the potential of large language models in rare disease diagnosis. The Lancet Digital Health, 8(2), 2026.

Reviewer_yxVAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed MRareBench, a multimodal benchmark for rare disease diagnosis. The data to curate the dataset is from PMC case reports, which include 300 open-ended diagnosis cases and 608 evidence verification items, including 227 rare diseases and 21 imaging modalities. Extensive experiments demonstrate the effectiveness of the incremental value of the visual infromation and evidence-diagnosis correspondence across 37 models.

Critical strengths
  1. The research question has clinical value.

This paper focuses on multi-image information integration and covers multiple diseases and imaging modalities, which I believe provide valuable resources for evaluating models on complex cases.

  1. The ablation studies for testing the effectiveness of visual information shows the importance of imaging information in rare disease diagnosis.
Critical weaknesses
  1. The relationship among existing multimodal rare-disease resources is under-discussed, which reduces the novelty of this work.

The manuscript claims that existing works focus on text-based rare-disease evaluation and common-disease multimodal medical evaluation, and emphasizes that this work is the first to use multimodal information. However, it ignores existing rare disease datasets that have already used multimodal information.

For example, GestaltMatcher [1] has already used facial imaging information to conduct phenotype matching for patients with rare diseases. They used the GestaltMatcher Database (GMDB) to provide rare disease images. This is similar with the method using visual phenotypes to help with rare-disease diagnosis that is the objective of this manuscript.

GestaltMML [2] further combines facial images, demographic information, and clinical phenotype text to conduct rare disease diagnosis. It is similar to combining clinical context and imaging information to produce a differential, which is proposed in this paper. It shows that the combination of text and imaging for rare disease diagnosis has its precedent.

ZebraMap [3] curated clinical context, structured case information, and related medical images from case reports. Then, it used Orphanet to organize disease information. This prior work shows a similar data curation process as provided in this paper.

The authors are highly recommended to provide a detailed comparison between these existing works and the provided work and then discuss the real research gap that this work is going to bridge rather than say that this is the first multimodal rare disease benchmark, which I believe is overclaiming.

  1. The report of model coverage is insufficient. The paper evaluated 37 models, but did not include models like GPT 5.6 sol, GPT 6 astra, Claude Fable 5, etc., which are the latest and strongest LLMs for vision and language tasks. It is highly recommended to include these models to see their performance so that we can know to what extent current cutting-edge models can handle rare-disease diagnosis.

References

[1] Hsieh, T.-C., Bar-Haim, A., Moosa, S., et al. GestaltMatcher facilitates rare disease matching using facial phenotype descriptors. Nature Genetics, 54, 349–357, 2022.

[2] Wu, D., Wang, Z., Chen, H., et al. GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Text. arXiv preprint arXiv:2312.15320, 2023; revised 2026.

[3] Islam, M. S., Jamal, A., and Alkhathlan, A. ZebraMap: A Multimodal Rare Disease Knowledge Map with Automated Data Aggregation & LLM-Enriched Information Extraction Pipeline. Diagnostics, 16(1), 107, 2026. First published online December 29, 2025.

Questions for authors

None

2 comments

We appreciate the careful and constructive review and are encouraged that the reviewer finds our research question clinically relevant. The pointers to related work are particularly helpful, and we address each comment below.

W1: Positioning relative to existing multimodal rare-disease resources

We will discuss these works in Related Work and add them to Table 1. Below we outline how MRareBench relates to each work.

GestaltMatcher [1]. GestaltMatcher develops a dedicated method for rare-disease matching from facial photographs and shows that visual phenotypes carry diagnostic signal for rare diseases. MRareBench instead evaluates existing MLLMs, without benchmark-specific training, on rare-disease cases that combine clinical context with multiple medical images per case across 21 imaging modalities, including radiology and pathology. Our question is therefore how existing MLLMs use heterogeneous medical images to form an open-ended differential diagnosis for a rare disease, rather than whether a dedicated model can exploit facial photographs for rare-disease matching.

GestaltMML [2]. GestaltMML trains a multimodal model that combines facial photographs with clinical text to predict rare genetic disorders. It is the closest precedent for combining images with clinical text in rare-disease diagnosis, so multimodal input alone does not distinguish our work. GestaltMML predicts among a predefined set of rare genetic disorders. MRareBench asks models to generate an open-ended rare-disease differential with no candidate list and additionally scores free-text reporting of the visual evidence for the rare disease against source-grounded rubrics. Our main addition is to compare this evidence reporting with the rare-disease diagnosis supplied and withheld, and to link these scores to diagnostic outcomes on the same rare-disease cases, a question that a disease-prediction model is not designed to address.

ZebraMap [3]. ZebraMap organizes rare-disease case-report narratives, clinical information, and images into a multimodal knowledge resource using Orphanet. Our construction pipeline shares some of these ingredients, including rare-disease case reports and Orphanet normalization, and we will acknowledge this explicitly. The outputs differ: ZebraMap produces a rare-disease knowledge resource, whereas MRareBench produces controlled evaluation items for rare-disease diagnosis. These items separate the clinical context, narrated findings, and images so that each can be withheld, mask answer-revealing text, attach source-grounded evidence rubrics, and link diagnosis and evidence items derived from the same rare-disease case report. These design choices enable the controlled comparisons described next.

W1: The research gap MRareBench addresses

MRareBench asks whether a model that can report the evidence for a rare disease can also identify that disease when the name is not given. Answering this question requires measuring diagnosis and evidence reporting on the same cases under controlled inputs:

  • A diagnosis task asks models to generate an open-ended differential for each rare-disease case. Each case is presented with the clinical context alone, with the medical images added, and with the imaging findings additionally described in text. This separates what the images contribute from what a textual description of the findings contributes.
  • An evidence-reporting task asks models to describe the visual evidence supporting the rare-disease diagnosis, with the disease name either supplied or withheld while the images and instructions stay the same. Responses are scored against evidence rubrics drawn from the source case report.
  • Because both tasks are built from the same case reports, a model's evidence reporting on a case can be compared with whether it diagnosed that case correctly.

This design reveals a pattern that diagnostic accuracy alone cannot show. Supplying the disease name raises recall of visible findings for all 37 models, by 24.7 points on average, but strong evidence reporting does not reliably translate into naming the disease when it is withheld: among model–case pairs with strong evidence reporting, only 26.3% also name the disease correctly. Among the diagnoses that miss at the top rank, 73% omit the correct disease from the ten candidates altogether. This points to a limitation in forming the right rare-disease hypothesis from multimodal evidence, rather than in ranking the candidates, as shown in sections 4.3 and 4.4.

We agree that "the first benchmark for multimodal rare disease diagnosis" is broader than this gap, since facial images and clinical text have been combined for rare genetic disorders before. We will therefore state this claim more precisely in the abstract and conclusion.

W2: Coverage of recent models

This is a helpful suggestion, and we agree that results on the newest models would make the benchmark more informative. The current evaluation covers 37 models across closed-source general, open-source general, and medical-specialized cohorts, including recent closed-source models such as GPT-5.2, Gemini-3.5-Flash, and Claude-Opus-4.6, as listed in Table A4. We plan to include them in a future update.

Once again, we thank the reviewer for the thoughtful and expert review. The comments reflect a clear understanding of our work and certainly have helped us articulate more precisely where our work fits within this line of work.

References

[1] Hsieh, T.-C., Bar-Haim, A., Moosa, S., et al. GestaltMatcher facilitates rare disease matching using facial phenotype descriptors. Nature Genetics, 54(3), 349–357, 2022.

[2] Wu, D., Yang, J., Liu, C., et al. GestaltMML: enhancing rare genetic disease diagnosis through multimodal machine learning combining facial images and clinical texts. arXiv, 2024.

[3] Islam, M. S., Jamal, A., and Alkhathlan, A. Zebramap: a multimodal rare disease knowledge map with automated data aggregation & LLM-enriched information extraction pipeline. Diagnostics, 16(1), 107, 2025.

Reviewer_wYKvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents MRareBench, a benchmark dataset for multimodal rare disease diagnosis, which consists of 908 cases of 227 diseases. The authors evaluated 37 MLLMs and showed that visual evidence improves rare disease diagnosis while it remains challenging to infer rare disease diagnoses from multimodal evidence.

Critical strengths
  1. According to the authors' claim, this is the first benchmark for multimodal rare disease diagnosis, although it may be overclaimed.
  2. A comprehensive suite of 37 MLLMs is evaluated on this dataset.
Critical weaknesses
  1. The most critical concern is that the constructed dataset is too small. There are only 908 cases covering 227 diseases. The authors did not specify the train-test split in either manuscript or supplementary materials. Anyway, evaluation on such a dataset does not lead to any solid conclusion.
  2. Furthermore, most of the key findings observed on this dataset are commonly observable on other benchmark datasets, offering no new insights. This greatly reduces the values and significance of the constructed dataset, especially it is much smaller compared to other common multimodal medical diagnosis datasets.
  3. The authors did not sufficiently identify the unique values of this rare disease dataset that differs significantly from other much larger common disease datasets. The authors should better position the constructed dataset in regards to other common datasets, rather than other rare disease datasets.
  4. While the authors listed three contributions, only the MRareBench dataset constitutes to a recognizable contribution. There are no novel designs or novel findings in other two claimed contributions.
  5. Fig. 2 shows the nine stages in dataset construction. The authors broke the pipeline into too many small stages, which prevents the readers from better comprehending the key procedures or novel designs in this pipeline. In addition, the description in Sec. 3 is disassociated from the nine stages in Fig. 2.
  6. The methodology section does not sufficiently focus on the novel designs. The technical details in this section is not well organized around the key novel designs or contributions. Many less important details are populated in the methodology section, e.g., definitions of evaluation metrics.
  7. The experimental analysis remains on the surface discussion of the results. There lacks in-depth analysis. In particular, there are even no sufficient discussions on the main results in Tab. 2!
Questions for authors
  1. What is the evaluation protocol? What is the train-test split? What are the evaluation metrics?
  2. What are the unique values of this relatively small rare diseases in comparison to other much larger but common datasets? What are the unique findings that are not observable from other datasets?
  3. Besides the dataset itself, can the authors better articulate novel designs in dataset construction or novel findings that are not observable on other datasets?
2 comments

We thank the reviewer for the careful reading and the questions.

Below we clarify the evaluation protocol, the motivation, and the contribution.

Q1, W1: Evaluation protocol, split, and metrics.

MRareBench is an evaluation-only benchmark. All 908 evaluation items are used only for testing, and there is no training split by design. The benchmark measures how existing MLLMs diagnose rare diseases from multimodal evidence. It has two tracks.

  • T1, forward diagnosis. The model reads the clinical description and the images, then names the disease as a ranked list of 10 candidates. Each case is run three ways: images plus a text description of the imaging findings, images without that description, and clinical text alone. The primary metric is Recall@1, whether the top candidate is correct. Recall@10, MRR, and mean rank are reported in Tables A16–A18.
  • T2, evidence verification. The model describes what the images show that supports a diagnosis. Each question is run with the diagnosis given, with the diagnosis withheld, and with the images withheld. Answers are scored against evidence points taken from the source report. Credit for relating findings across images, or for linking them to the disease, counts only after the basic visible findings are reported.
  • 402 T2 questions come from the same case reports as 289 T1 cases. This lets us check whether a model that can describe the evidence can also name the disease on its own.

These definitions are given in section 3.1 and Appendix G. We agree that the evaluation-only protocol should be stated more explicitly.

Q3: Benchmark construction.

The source case reports are publicly available on PMC. Our contribution lies in turning them into controlled test cases and questions, which required substantial effort on each one. Each was checked individually against its source report. This involved aligning clinical text with the correct images, separating narrated imaging findings from other clinical context, removing answer-revealing cues, and normalizing diagnoses to Orphanet. These steps are described in section 3.2.


Q2, W1, W3: Data size and value relative to common-disease datasets.

Our work targets rare disease as a distinct setting, rather than as a subset of common-disease evaluation. For common conditions, prevalence and the clinical narrative often narrow the differential before the images are read. Strong models already score highly on many such benchmarks, so it is difficult to tell whether a diagnosis comes from the evidence or from familiar priors. Rare disease weakens these priors and places more weight on the visual evidence.

Regarding data size, our conclusions do not depend on per-disease estimates. They compare the same cases and questions across three settings, and the main effects point in the same direction across the 37 models. Supplying the diagnosis raises visible-finding recall for every model, by 24.7 points on average, and withholding the images lowers diagnostic accuracy for all but one small model. A larger set would mainly support finer per-disease and per-modality analysis, which we plan to add in the next version.

General benchmarks such as GMAI-MMBench and MedXpertQA do include rare-disease questions. These questions are part of a broad coverage and are posed in a multiple-choice format. As Table 1 shows, none of these benchmarks studies open multi-image rare-disease diagnosis while controlling which evidence the model receives and whether the diagnosis is given.

Q2, Q3, W2: Findings beyond existing benchmarks.

The reviewer states that our key findings are commonly observed on other benchmarks. We would appreciate specific references, and we are happy to discuss them further.

Our findings come from a setting that differs in three ways. First, every case is a rare disease, so prevalence and familiar symptom patterns give little help in narrowing the diagnosis. Second, each case includes at least two images, which may come from different imaging modalities, and the diagnosis often depends on combining findings across them. Third, in the main setting, the written description of the imaging findings is removed from the input, so the model must read the evidence from the images rather than from the text.

Under these conditions, our paired design asks a specific question. Because T1 and T2 share cases from the same reports, we can test whether a model that reports the correct evidence can also identify the disease on its own. Section 4.4 shows that it often cannot. Among model–case pairs with strong evidence reporting, only 26.3% also name the disease correctly. Among the diagnoses that miss at the top rank, 73% omit the correct disease from the ten candidates altogether. This gap has not been exactly measured in multimodal rare-disease diagnosis.

W4: Contributions.

We thank the reviewer for raising this point, and we would like to clarify the contributions of our work beyond the dataset itself. MRareBench makes three contributions.

  • A benchmark dedicated to multimodal rare-disease diagnosis. Built from published case reports, each case pairs the clinical presentation with multiple images across imaging modalities and requires an open differential diagnosis rather than a choice among options.
  • A controlled evaluation of evidence-based diagnosis. Where pretest probability offers little guidance, we separate what the images contribute from what the text or the diagnosis supplies. The reported imaging findings are withheld from the input, an image-withheld condition measures what the images add, and the diagnosis is either disclosed or withheld, with differential diagnosis and evidence reporting paired at the case level.
  • A gap between evidence recognition and diagnostic reasoning in the rare-disease long tail. When the diagnosis is disclosed, recall of visible findings increases in all 37 models. When it is withheld, the correct rare disease is absent from the top-10 differential in roughly three quarters of missed cases. This points to a limitation in diagnostic hypothesis generation rather than in perceiving the findings.

We will state these contributions and our motivation more explicitly in the next version.

Other issues.

We thank the reviewer for the suggestions on presentation, including the discussion of Table 2 and the organization of section 3, and will take them into account in the next version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.