- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces CLIMB, a benchmark for testing whether LLMs can identify multiple co-occurring diseases through multi-turn clinical conversations. The authors build synthetic multimorbid patients from ePOCT+ and DDXPlus, and evaluate several models under both full-information and interactive settings. The paper also includes controlled analyses of information gathering, anchoring, and diagnostic recovery.
Critical strengths
The paper studies an important and relatively underexplored problem: whether LLMs can identify multiple diseases during interactive diagnosis rather than only predict one main diagnosis. The experimental analysis is also fairly thorough, with controls such as oracle disease count, matched cohorts, opening manipulation, and forced continuation. The data generation and evaluation pipeline are described in good detail.
Critical weaknesses
- The opening presentation is biased toward a single disease component. In DDXPlus, multiple single-disease records are merged into one multimorbid patient, but the opening complaint comes only from the highest-severity component (Sec. 3.2; App. B.4). Figure 1 shows a patient with Pneumonia, Scabies, and Anemia, but the opening is only “My son has a cough.” It is natural for the model to focus on respiratory diseases in this situation. Scabies and Anemia are not clearly related to the initial complaint, so missing them when the patient does not mention rash or pallor does not by itself show a failure of multi-hypothesis reasoning.
- Pairwise disease association does not guarantee a plausible higher-order combination. For ePOCT+, P(A) and P(A,B) come from cohort statistics, but cases with three or four diseases are still built from pairwise associations (Sec. 3.1; App. B.3). Even if A+B, A+C, and B+C are each plausible, A+B+C is not necessarily a realistic disease combination. This issue is stronger in DDXPlus, where the original data contain only single-disease records and the co-occurrence multipliers are elicited from an LLM rather than observed from real concurrent diagnoses (Sec. 3.2; App. B.4).
- “A finding of its own” is not necessarily a clinically informative clue. The benchmark requires each diagnosis to keep at least one finding contributed only by that disease component (Sec. 3.1–3.2; App. B.2–B.3). However, this only makes the finding component-specific, not disease-specific. For example, Figure 9 includes cough under the Common Cold component, but cough is also common in pneumonia, influenza, and bronchitis. A finding coming from only one source component does not mean that it is a clear diagnostic clue for that disease.
- The clinical validity of the merged multimorbid cases is not sufficiently checked. The benchmark largely constructs multimorbid patients by merging findings from separate single-disease components (Sec. 3.1–3.2). In real patients, co-occurring diseases can change each other’s presentation, so the combined phenotype is not always a simple union of the two single-disease phenotypes. The paper itself uses HIV + tuberculosis as an example in the co-occurrence prompt (App. B.8). Clinically, HIV can change how tuberculosis presents: advanced HIV is more often associated with extrapulmonary disease, lower-lobe infiltrates, or hilar adenopathy, while classic upper-lobe cavitary disease may be less common. This is not captured by simply adding an HIV record to a TB record. In addition, Sec. 3.3 randomly adds 1–2 “unrelated background complaints.” These symptoms may still be clinically meaningful. For example, Figure 9 adds night-time urination, which can itself suggest diabetes, BPH, or other urinary conditions. Calling such symptoms “background” does not make them diagnostically irrelevant.
Questions for authors
-
Opening presentation and patient simulator behavior: Since the opening complaint comes from only a single component (e.g., the highest-severity condition in DDXPlus), how often do the evaluated models ask broad screening or review-of-systems questions (such as "Do you have any other symptoms?"), and does the patient simulator reliably volunteer complaints from the other unrelated conditions when asked such open-ended questions?
-
Higher-order disease combinations (): Given that sets of 3 and 4 diseases are sampled sequentially using only pairwise co-occurrence multipliers, did clinicians review the resulting higher-order combinations for clinical plausibility? How do the authors ensure that or tuples represent realistic multimorbid profiles rather than mutually plausible pairs that are implausible together?
-
Diagnostic specificity of "a finding of its own": Requiring each component to contribute at least one unique finding to the merged record only ensures component-level uniqueness, not clinical specificity to that target disease. For a human clinician—not just a logistic regression reader trained on the generator's distribution—are all target diseases actually identifiable from the retained findings, or are some conditions clinically indistinguishable from broader differentials?
-
Phenotype interaction and background complaints: Merging single-disease records by simple union ignores how co-occurring diseases alter each other's clinical presentation, and adding 1–2 random background symptoms can introduce clinically meaningful findings (e.g., night-time urination) that point to other candidate diagnoses. Did clinicians audit a sample of the final merged cases () to verify that the combined phenotype remains clinically valid and that the injected background complaints do not legitimately support "false positive" diagnoses?
-
Reliance on the Appendix: Many details essential for evaluating the benchmark's validity—such as the exact opening-complaint selection rules, the LLM prompts for co-occurrence estimation, and the construction of the controlled cohorts in Section 6—are currently deferred to a 38-page appendix. How do the authors plan to make the main text self-contained within the page limit?
4 comments
New comments
Some of my concerns are only partially resolved, and some of the new results still do not support the benchmark's clinical validity.
-
Q1: Table R8 shows that open screening questions occur in only 5.4% to 6.3% of consultations, and even when the patient reveals a non-opening finding (81.5%), recovery of non-opening diagnoses increases by only +3.0 points [-2.5, 8.5]. Is this statistically significant?
-
Q3: High accuracy from a logistic regression or decision tree trained on 10,000–80,000 cases from the same generator only shows that the generator's rule boundaries are learnable, not that the retained findings are clinically specific.
-
Q4: Even at with full information, models achieve ~89% recall but low exact-set recovery because they output ~2.1 diagnoses on average, which is confounded by prompting them to actively look for co-occurring diseases while injecting 1–2 extra background complaints. Also, why is night-time urination injected into pediatric ePOCT+ cases (<15 years, Fig. 9)? Can you explain?
I can raise the score if the authors if these are fully addressed.
Response to Q1 and Q3 (1/2)
Thank you for reading our response in detail and for the follow up with important points.
The clinician panel ratings are now posted (see our comment "Clinician panel ratings"). The clinicians find 87% of the diagnosis sets realistic or possible.
For clinical validity, we manage the process carefully together with our clinician co-authors, with statistical and rule-based constraints. Before our answers, we briefly want to reiterate the key points that support our vignette construction choices:
- Real patients motivate the need to study multimorbidity. We ran the models on 160 real ePOCT+ consultations and observed the same drop in accuracy as k increases (general response, part 1, Table R1).
- The generated ePOCT+ cases mimic the real cohort (general response, part 1).
- Each new diagnosis has to pass statistical and rule based checks validated by clinicians.
- Sampling the diseases independently follows classical multi-disease systems such as QMR-DT (Shwe et al., 1991; Middleton et al., 1991).
We hope that this resolves your concerns. Now we would like to answer the three points you raised:
Q1. No, it is not statistically significant, the interval [−2.5, 8.5] includes zero. This is also how we read the result and what we would expect. The patient does not hide the other disease and when the doctor asks an open question, the patient reveals a finding of a non-opening diagnosis. The doctor hears this finding but we do not observe that it names the diagnosis more often. Therefore, our conclusion is that missing diagnoses are not explained by missing findings. We think this is because the doctor does not follow multiple hypotheses.
Q3. We agree with your reading and we did not want to oppose you on that point. We will make it more clear in the paper. Our purpose in using them was to show that the information needed for the full set is there and can be found. We raised some other arguments about clinical validity above. Now, we would like to explain again why we use them. In Section 5 we observe the failure and in Section 6 we want to understand why it happens. There can be two reasons: either the LLM is bad at collecting the evidence or it is bad at reasoning over it even if it collects the evidence well.
For each of the two we wanted the best solution we can get from the data, so that we can see where the LLM loses the diagnosis. DDXPlus was also introduced with trained agents that ask questions and diagnose (Fansi Tchango et al., 2022). Training a model on cases that are generated from a knowledge base is an established approach in this line (Ravuri et al., 2018).
For the decision tree, we wanted a ceiling on question selection. We understand your concern about the clinical validity of its questions. At each step it asks the most informative question in the data. Learning a decision tree from data is an established way to build clinical decision rules (Podgorelec et al., 2002). Our assumption is that doctors should ask the most discriminative question in the space. As we wrote in the previous response, when LLM is given these questions its exact-set increases (Table 12).
For the logistic reader, we wanted a ceiling on the decision. It shows that the full set is in the record and the task is not impossible to solve. The same argument holds here, fitting a logistic regression on data is the standard way to build clinical prediction models (Christodoulou et al., 2019), (Trottet et al., 2023).
Response to Q4 (2/2)
Q4. You are right while you are saying that naming too many diagnoses explains the low exact-set at k = 1. Showing this is actually part of our goal (Section 5.2) and we will make it more clear in the paper.
Our approach is this: Multimorbidity is very common, in the ePOCT+ data a consultation ends with 2.3 diagnoses on average (Poitiers et al., 2025). Until now LLMs are asked for a diagnosis and a differential and then the answer is scored against a single correct label (top-1 or top-k). This is not how it works in practice. A real doctor has to solve the problem without knowing if k is 1, 2 or 3. Even at k = 1 the doctor should not stop at the first diagnosis (satisfaction of search). In this sense k = 1 and k = 4 are the same problem: the doctor returns a set of diagnoses and we compare it with the true set D, and k = |D| is never given.
Naming too many diagnoses hurts k = 1 the most and this comes from the prompt. When we ask models for a single diagnosis the score increases (Table 17). But k = 1 still has the highest score. So it does not explain why the score falls with k.
We also ran an ablation for this and it is in the paper. When we give the model the true number of diagnoses it cannot name more. Here the score at k = 1 increases as expected but the collapse continues, the score still falls as k grows (Appendix E.3). We also report Jaccard, which gives partial credit for each correct diagnosis, and we see the drop there too (Table 15).
For the background symptoms, we wrote about them in the general response and it was not clear enough in our individual response, so we want to make it more clear here. A patient generated only from disease records reports nothing but the findings of the diseases. Real patients are not like that. Most people carry a few everyday symptoms that have nothing to do with why they came (Elnegaard et al., 2015). We added them for realism, so that the patient is not made of pure disease findings and the doctor has some noise to set aside. We think this is a valid intention. Our experiments show that they do not affect the result, so you would have no statistically significant change from what we have now (Table R3).
To explain the process for background sampling again: we sample the background symptoms independently of the diseases with some constraints (Appendix B.6). This is where your example on night-time urination comes from. We understand your concern, our aim was only to add general symptoms and not a new diagnosis. We expect doctor to ask additional questions not to be biased by the background symptoms.
We hope this answers your remaining questions. Thank you again for reading our work with interest and for your questions.
Response to Reviewer Hmjg
Thank you for the careful review, and for finding the problem important and the pipeline well documented. Most points are shared with other reviewers, so our general response covers higher-order combinations and the LLM multipliers (W2, Q2), the clinician review and clinical validity (Q2, Q4) and the background complaints (W4, Q4) in parts 1 and 2, Tables R2 and R3. There we also show that the drop with k appears in real cases. Here we answer the points specific to your review.
On the opening complaint (W1, Q1), we agree that it steers which diagnoses are found, but this is what CLIMB tests. A real consultation also starts with a complaint, and the important question here is if the doctor keeps looking after the first diagnosis (satisfaction of search, Berbaum et al., 1990). We would like to note two points. First, the problem also occurs with full information, where every finding is shown (Table 2). Second, in DDXPlus cases the opening complaint comes from the most severe diagnosis so that one fixed rule applies to every case (Section 3.2), and the opening effect remains when the two diagnoses are equally severe (Appendix E.7).
In Q1 you ask about screening questions. We labelled the 10,579 candidate screening questions with an LLM judge (GPT-5.6) as ‘open’, ‘listing’ or ‘neither’. We observe that open questions such as "Do you have any other symptoms?" are in 5.4% of DDXPlus and 6.3% of ePOCT+ consultations (Table R8). When such a question is asked in a DDXPlus case with k ≥ 2, the patient reveals a finding of a non-opening diagnosis in 81.5% [75.6, 87.5] of cases, yet these consultations recover the non-opening diagnoses only 3.0 points more often [−2.5, 8.5].
On your points about a finding of its own (W3, Q3), we state our arguments for the clinician review and the clinical validity of the cases in the general response, and we hope it answers your concern. We would like to add some discussion on your specific points. Symptom overlap is a central difficulty in multimorbidity. We analysed the overlap between the diagnoses of one patient in Section 6.2.2 and Appendix E.8. For overlap with diagnoses outside the set, we ran an experiment following a concern of another reviewer. When predictions in the DDXPlus differential (the plausible alternatives you mention) are not counted as errors, the score still falls with k (Table R6 in our reply to Reviewer pwn3). For the decision tree and the reader, our goal was to show that the diagnoses can be told apart from the findings. In a controlled experiment with the same menu and budget, a decision tree recovers most of the sets, but an AI doctor does not (Appendix D.6, Table 12). When the AI doctor reads the questions and answers of the tree, its score increases, so these findings carry diagnostic information that an LLM can use.
We agree that co-occurring diseases can change each other's presentation (W4), and we will state this as a limitation. However, seminal work in the line, QMR-DT, also combines the findings of co-occurring diseases without interaction (Shwe et al., 1991). Merging them should keep each disease’s typical findings and so it should make the cases easier. For ePOCT+, the merge is more than a plain union and accepts only sets whose paths are consistent under the algorithm's own rules (Appendix B.3), and in clinical practice a consultation still ends with more than 2 diagnoses on average after clinician review (Poitiers et al., 2025). Most importantly, the drop with k also appears in real patients (Table R1).
The motivation for background complaints is given in the general response (part 2). For your point specifically, the night-time urination example is reported by 48.7% of respondents within four weeks in the population survey we sample background complaints from, independently of the diagnoses (Elnegaard et al., 2015). On DDXPlus, adding such complaints shows no consistent effect on exact-set recovery (Table R3).
On the presentation (Q5), we are sorry that the paper was hard to follow. The opening rule and the LLM elicitation are stated in Section 3.2, and the cohorts in Section 4.2; the appendix holds the prompts and full algorithms for reproducibility. We will use the extra page for a table of the generation rules and the new analyses. We would also be grateful for pointers to any other passages that were hard to read.
We hope this answers your questions, and we would be glad to discuss any remaining concerns.