acceptodds
Under review as a conference paper at ICLR 2027

CLIMB: A Clinical Multimorbidity Benchmark for Diagnosing Co-occurring Conditions through Multiturn Conversations

Abstract

Patients often have several co-occurring clinical conditions, and the findings needed to identify and disambiguate them emerge over the course of a consultation. Evaluating clinical reasoning in this setting requires both multi-turn interaction and multi-label diagnosis. We introduce CLIMB, a benchmark in which a doctor model interviews a simulated patient to recover a ground truth set of co-occurring clinical conditions. Cases are synthesized from clinical decision algorithms and diagnostic datasets, grounding multimorbid presentations in structured clinical knowledge. Across six frontier and open models, none recovers the exact set of conditions in more than 10% of interactive cases. Diagnostic performance declines when conditions co-occur, even when models receive the full clinical record and the true number of conditions. Interaction reduces performance further. In controlled experiments, models behave like single-hypothesis trackers: they anchor on the diagnosis suggested by the opening findings, keep questioning around it, and recover a second condition mainly when a finding in view points to it. Questioning them further does not complete the set but adds mostly wrong diagnoses. We formalise this pattern with a theoretical reference model of single-hypothesis tracking. The benchmark, generator, and evaluation code are available at https://anonymous.4open.science/r/CLIMB-8340.

Primary area:
datasets and benchmarks
Keywords:
large language models, clinical diagnosis, multi-turn interaction, multi-label classification, benchmark, patient simulator, comorbidity, hypothesis tracking, medical question answering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HmjgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces CLIMB, a benchmark for testing whether LLMs can identify multiple co-occurring diseases through multi-turn clinical conversations. The authors build synthetic multimorbid patients from ePOCT+ and DDXPlus, and evaluate several models under both full-information and interactive settings. The paper also includes controlled analyses of information gathering, anchoring, and diagnostic recovery.

Critical strengths

The paper studies an important and relatively underexplored problem: whether LLMs can identify multiple diseases during interactive diagnosis rather than only predict one main diagnosis. The experimental analysis is also fairly thorough, with controls such as oracle disease count, matched cohorts, opening manipulation, and forced continuation. The data generation and evaluation pipeline are described in good detail.

Critical weaknesses
  1. The opening presentation is biased toward a single disease component. In DDXPlus, multiple single-disease records are merged into one multimorbid patient, but the opening complaint comes only from the highest-severity component (Sec. 3.2; App. B.4). Figure 1 shows a patient with Pneumonia, Scabies, and Anemia, but the opening is only “My son has a cough.” It is natural for the model to focus on respiratory diseases in this situation. Scabies and Anemia are not clearly related to the initial complaint, so missing them when the patient does not mention rash or pallor does not by itself show a failure of multi-hypothesis reasoning.
  2. Pairwise disease association does not guarantee a plausible higher-order combination. For ePOCT+, P(A) and P(A,B) come from cohort statistics, but cases with three or four diseases are still built from pairwise associations (Sec. 3.1; App. B.3). Even if A+B, A+C, and B+C are each plausible, A+B+C is not necessarily a realistic disease combination. This issue is stronger in DDXPlus, where the original data contain only single-disease records and the co-occurrence multipliers are elicited from an LLM rather than observed from real concurrent diagnoses (Sec. 3.2; App. B.4).
  3. “A finding of its own” is not necessarily a clinically informative clue. The benchmark requires each diagnosis to keep at least one finding contributed only by that disease component (Sec. 3.1–3.2; App. B.2–B.3). However, this only makes the finding component-specific, not disease-specific. For example, Figure 9 includes cough under the Common Cold component, but cough is also common in pneumonia, influenza, and bronchitis. A finding coming from only one source component does not mean that it is a clear diagnostic clue for that disease.
  4. The clinical validity of the merged multimorbid cases is not sufficiently checked. The benchmark largely constructs multimorbid patients by merging findings from separate single-disease components (Sec. 3.1–3.2). In real patients, co-occurring diseases can change each other’s presentation, so the combined phenotype is not always a simple union of the two single-disease phenotypes. The paper itself uses HIV + tuberculosis as an example in the co-occurrence prompt (App. B.8). Clinically, HIV can change how tuberculosis presents: advanced HIV is more often associated with extrapulmonary disease, lower-lobe infiltrates, or hilar adenopathy, while classic upper-lobe cavitary disease may be less common. This is not captured by simply adding an HIV record to a TB record. In addition, Sec. 3.3 randomly adds 1–2 “unrelated background complaints.” These symptoms may still be clinically meaningful. For example, Figure 9 adds night-time urination, which can itself suggest diabetes, BPH, or other urinary conditions. Calling such symptoms “background” does not make them diagnostically irrelevant.
Questions for authors
  1. Opening presentation and patient simulator behavior: Since the opening complaint comes from only a single component (e.g., the highest-severity condition in DDXPlus), how often do the evaluated models ask broad screening or review-of-systems questions (such as "Do you have any other symptoms?"), and does the patient simulator reliably volunteer complaints from the other unrelated conditions when asked such open-ended questions?

  2. Higher-order disease combinations (k≥3k \ge 3): Given that sets of 3 and 4 diseases are sampled sequentially using only pairwise co-occurrence multipliers, did clinicians review the resulting higher-order combinations for clinical plausibility? How do the authors ensure that k=3k=3 or k=4k=4 tuples represent realistic multimorbid profiles rather than mutually plausible pairs that are implausible together?

  3. Diagnostic specificity of "a finding of its own": Requiring each component to contribute at least one unique finding to the merged record only ensures component-level uniqueness, not clinical specificity to that target disease. For a human clinician—not just a logistic regression reader trained on the generator's distribution—are all target diseases actually identifiable from the retained findings, or are some conditions clinically indistinguishable from broader differentials?

  4. Phenotype interaction and background complaints: Merging single-disease records by simple union ignores how co-occurring diseases alter each other's clinical presentation, and adding 1–2 random background symptoms can introduce clinically meaningful findings (e.g., night-time urination) that point to other candidate diagnoses. Did clinicians audit a sample of the final merged cases (S∪BS \cup B) to verify that the combined phenotype remains clinically valid and that the injected background complaints do not legitimately support "false positive" diagnoses?

  5. Reliance on the Appendix: Many details essential for evaluating the benchmark's validity—such as the exact opening-complaint selection rules, the LLM prompts for co-occurrence estimation, and the construction of the controlled cohorts in Section 6—are currently deferred to a 38-page appendix. How do the authors plan to make the main text self-contained within the page limit?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Hmjg

Thank you for the careful review, and for finding the problem important and the pipeline well documented. Most points are shared with other reviewers, so our general response covers higher-order combinations and the LLM multipliers (W2, Q2), the clinician review and clinical validity (Q2, Q4) and the background complaints (W4, Q4) in parts 1 and 2, Tables R2 and R3. There we also show that the drop with k appears in real cases. Here we answer the points specific to your review.

On the opening complaint (W1, Q1), we agree that it steers which diagnoses are found, but this is what CLIMB tests. A real consultation also starts with a complaint, and the important question here is if the doctor keeps looking after the first diagnosis (satisfaction of search, Berbaum et al., 1990). We would like to note two points. First, the problem also occurs with full information, where every finding is shown (Table 2). Second, in DDXPlus cases the opening complaint comes from the most severe diagnosis so that one fixed rule applies to every case (Section 3.2), and the opening effect remains when the two diagnoses are equally severe (Appendix E.7).

In Q1 you ask about screening questions. We labelled the 10,579 candidate screening questions with an LLM judge (GPT-5.6) as ‘open’, ‘listing’ or ‘neither’. We observe that open questions such as "Do you have any other symptoms?" are in 5.4% of DDXPlus and 6.3% of ePOCT+ consultations (Table R8). When such a question is asked in a DDXPlus case with k ≥ 2, the patient reveals a finding of a non-opening diagnosis in 81.5% [75.6, 87.5] of cases, yet these consultations recover the non-opening diagnoses only 3.0 points more often [−2.5, 8.5].

R8. Consultations with an open screening question, six models (%)k = 1k = 2k = 3k = 4All
DDXPlus7.44.35.14.65.4
ePOCT+8.87.25.73.76.3

On your points about a finding of its own (W3, Q3), we state our arguments for the clinician review and the clinical validity of the cases in the general response, and we hope it answers your concern. We would like to add some discussion on your specific points. Symptom overlap is a central difficulty in multimorbidity. We analysed the overlap between the diagnoses of one patient in Section 6.2.2 and Appendix E.8. For overlap with diagnoses outside the set, we ran an experiment following a concern of another reviewer. When predictions in the DDXPlus differential (the plausible alternatives you mention) are not counted as errors, the score still falls with k (Table R6 in our reply to Reviewer pwn3). For the decision tree and the reader, our goal was to show that the diagnoses can be told apart from the findings. In a controlled experiment with the same menu and budget, a decision tree recovers most of the sets, but an AI doctor does not (Appendix D.6, Table 12). When the AI doctor reads the questions and answers of the tree, its score increases, so these findings carry diagnostic information that an LLM can use.

We agree that co-occurring diseases can change each other's presentation (W4), and we will state this as a limitation. However, seminal work in the line, QMR-DT, also combines the findings of co-occurring diseases without interaction (Shwe et al., 1991). Merging them should keep each disease’s typical findings and so it should make the cases easier. For ePOCT+, the merge is more than a plain union and accepts only sets whose paths are consistent under the algorithm's own rules (Appendix B.3), and in clinical practice a consultation still ends with more than 2 diagnoses on average after clinician review (Poitiers et al., 2025). Most importantly, the drop with k also appears in real patients (Table R1).

The motivation for background complaints is given in the general response (part 2). For your point specifically, the night-time urination example is reported by 48.7% of respondents within four weeks in the population survey we sample background complaints from, independently of the diagnoses (Elnegaard et al., 2015). On DDXPlus, adding such complaints shows no consistent effect on exact-set recovery (Table R3).

On the presentation (Q5), we are sorry that the paper was hard to follow. The opening rule and the LLM elicitation are stated in Section 3.2, and the cohorts in Section 4.2; the appendix holds the prompts and full algorithms for reproducibility. We will use the extra page for a table of the generation rules and the new analyses. We would also be grateful for pointers to any other passages that were hard to read.

We hope this answers your questions, and we would be glad to discuss any remaining concerns.

Reviewer_HmjgOpenReview ↗

New comments

Some of my concerns are only partially resolved, and some of the new results still do not support the benchmark's clinical validity.

  • Q1: Table R8 shows that open screening questions occur in only 5.4% to 6.3% of consultations, and even when the patient reveals a non-opening finding (81.5%), recovery of non-opening diagnoses increases by only +3.0 points [-2.5, 8.5]. Is this statistically significant?

  • Q3: High accuracy from a logistic regression or decision tree trained on 10,000–80,000 cases from the same generator only shows that the generator's rule boundaries are learnable, not that the retained findings are clinically specific.

  • Q4: Even at k=1k=1 with full information, models achieve ~89% recall but low exact-set recovery because they output ~2.1 diagnoses on average, which is confounded by prompting them to actively look for co-occurring diseases while injecting 1–2 extra background complaints. Also, why is night-time urination injected into pediatric ePOCT+ cases (<15 years, Fig. 9)? Can you explain?

I can raise the score if the authors if these are fully addressed.

AuthorsOpenReview ↗

Response to Q1 and Q3 (1/2)

Thank you for reading our response in detail and for the follow up with important points.

The clinician panel ratings are now posted (see our comment "Clinician panel ratings"). The clinicians find 87% of the diagnosis sets realistic or possible.

For clinical validity, we manage the process carefully together with our clinician co-authors, with statistical and rule-based constraints. Before our answers, we briefly want to reiterate the key points that support our vignette construction choices:

  • Real patients motivate the need to study multimorbidity. We ran the models on 160 real ePOCT+ consultations and observed the same drop in accuracy as k increases (general response, part 1, Table R1).
  • The generated ePOCT+ cases mimic the real cohort (general response, part 1).
  • Each new diagnosis has to pass statistical and rule based checks validated by clinicians.
  • Sampling the diseases independently follows classical multi-disease systems such as QMR-DT (Shwe et al., 1991; Middleton et al., 1991).

We hope that this resolves your concerns. Now we would like to answer the three points you raised:

Q1. No, it is not statistically significant, the interval [−2.5, 8.5] includes zero. This is also how we read the result and what we would expect. The patient does not hide the other disease and when the doctor asks an open question, the patient reveals a finding of a non-opening diagnosis. The doctor hears this finding but we do not observe that it names the diagnosis more often. Therefore, our conclusion is that missing diagnoses are not explained by missing findings. We think this is because the doctor does not follow multiple hypotheses.

Q3. We agree with your reading and we did not want to oppose you on that point. We will make it more clear in the paper. Our purpose in using them was to show that the information needed for the full set is there and can be found. We raised some other arguments about clinical validity above. Now, we would like to explain again why we use them. In Section 5 we observe the failure and in Section 6 we want to understand why it happens. There can be two reasons: either the LLM is bad at collecting the evidence or it is bad at reasoning over it even if it collects the evidence well.

For each of the two we wanted the best solution we can get from the data, so that we can see where the LLM loses the diagnosis. DDXPlus was also introduced with trained agents that ask questions and diagnose (Fansi Tchango et al., 2022). Training a model on cases that are generated from a knowledge base is an established approach in this line (Ravuri et al., 2018).

For the decision tree, we wanted a ceiling on question selection. We understand your concern about the clinical validity of its questions. At each step it asks the most informative question in the data. Learning a decision tree from data is an established way to build clinical decision rules (Podgorelec et al., 2002). Our assumption is that doctors should ask the most discriminative question in the space. As we wrote in the previous response, when LLM is given these questions its exact-set increases (Table 12).

For the logistic reader, we wanted a ceiling on the decision. It shows that the full set is in the record and the task is not impossible to solve. The same argument holds here, fitting a logistic regression on data is the standard way to build clinical prediction models (Christodoulou et al., 2019), (Trottet et al., 2023).

AuthorsOpenReview ↗

Response to Q4 (2/2)

Q4. You are right while you are saying that naming too many diagnoses explains the low exact-set at k = 1. Showing this is actually part of our goal (Section 5.2) and we will make it more clear in the paper.

Our approach is this: Multimorbidity is very common, in the ePOCT+ data a consultation ends with 2.3 diagnoses on average (Poitiers et al., 2025). Until now LLMs are asked for a diagnosis and a differential and then the answer is scored against a single correct label (top-1 or top-k). This is not how it works in practice. A real doctor has to solve the problem without knowing if k is 1, 2 or 3. Even at k = 1 the doctor should not stop at the first diagnosis (satisfaction of search). In this sense k = 1 and k = 4 are the same problem: the doctor returns a set of diagnoses and we compare it with the true set D, and k = |D| is never given.

Naming too many diagnoses hurts k = 1 the most and this comes from the prompt. When we ask models for a single diagnosis the score increases (Table 17). But k = 1 still has the highest score. So it does not explain why the score falls with k.

We also ran an ablation for this and it is in the paper. When we give the model the true number of diagnoses it cannot name more. Here the score at k = 1 increases as expected but the collapse continues, the score still falls as k grows (Appendix E.3). We also report Jaccard, which gives partial credit for each correct diagnosis, and we see the drop there too (Table 15).

For the background symptoms, we wrote about them in the general response and it was not clear enough in our individual response, so we want to make it more clear here. A patient generated only from disease records reports nothing but the findings of the diseases. Real patients are not like that. Most people carry a few everyday symptoms that have nothing to do with why they came (Elnegaard et al., 2015). We added them for realism, so that the patient is not made of pure disease findings and the doctor has some noise to set aside. We think this is a valid intention. Our experiments show that they do not affect the result, so you would have no statistically significant change from what we have now (Table R3).

To explain the process for background sampling again: we sample the background symptoms independently of the diseases with some constraints (Appendix B.6). This is where your example on night-time urination comes from. We understand your concern, our aim was only to add general symptoms and not a new diagnosis. We expect doctor to ask additional questions not to be biased by the background symptoms.

We hope this answers your remaining questions. Thank you again for reading our work with interest and for your questions.

Reviewer_hjP8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces CLIMB, a benchmark for multi-turn, multi-label clinical diagnosis. It constructs synthetic multimorbid cases from ePOCT+ and DDXPlus and evaluates six LLMs under full-information and interactive settings. The authors report poor exact-set recovery and investigate potential failure modes, including ineffective evidence acquisition, diagnostic anchoring, and premature stopping. A theoretical reference model is introduced to characterize single-hypothesis tracking.

Critical strengths
  1. Meaningful problem setting. The combination of multi-turn interaction and multi-label diagnosis addresses an important limitation of conventional clinical benchmarks.
  2. Useful diagnostic analyses. The comparisons between full-information and interactive settings, together with controlled experiments on evidence acquisition and anchoring, offer potentially useful insights into model behavior.
Critical weaknesses
  1. Questionable benchmark validity. The benchmark constructs multimorbid patients by merging single-disease records, while DDXPlus co-occurrence statistics are estimated by an LLM rather than derived from observed multimorbid cases. Consistency checks and pairwise clinical screening do not guarantee clinically valid or complete ground-truth diagnosis sets. The strong performance of a classifier trained on the same generator demonstrates learnability, not clinical validity. Consequently, the reported exact-set failures may partly reflect artifacts of synthetic data construction.
  2. Insufficient evidence for the claimed failure mechanism. The observed anchoring and missing diagnoses do not uniquely establish single-hypothesis tracking; alternative explanations such as evidence salience and poor multi-label calibration remain plausible. Moreover, the interactive evaluation relies on an imperfect LLM patient simulator, while doctor models are evaluated with reasoning disabled or minimized. These limitations substantially weaken the generality and mechanistic interpretation of the findings.
Questions for authors
  1. Can the authors provide independent clinician-level validation of the generated cases, particularly regarding ground-truth completeness and clinically plausible alternative diagnoses?
  2. How robust are the interactive results to patient-simulator errors and different reasoning configurations? Would a deterministic patient interface yield similar conclusions?
  3. What direct evidence distinguishes single-hypothesis tracking from alternative explanations? Can an explicit multi-hypothesis reasoning baseline help validate this claim?
1 comment
AuthorsOpenReview ↗

Response to Reviewer hjP8

Thank you for the review, and for finding the problem setting meaningful and the diagnostic analyses useful. Your concerns about the validity of the cases (W1, Q1) and about the simulator and reasoning (W2, Q2) are shared with the other reviewers and are answered in our general response (parts 1 to 3, Tables R1 to R4). We hope the general response addresses most of your concerns, and here we would like to share the points specific to your review.

You ask about other alternative explanations to the model’s behavior and named calibration and salience for testing.

If the problem were calibration, the diagnosis that model missed would sit just below the cut off. Then, naming more labels should recover them. But, in some setup, models already name too many labels (Appendix E.2). Also, when we push a model to name exactly k labels, the collapse with k remains, which means giving them the true k does not help (Appendix E.3), and a correct count rarely comes with the correct set (Appendix E.2).

For salience, it actually plays a role, we do not deny it and show it in our anchoring experiment (Appendix E.7). But if salience were the whole story, adding a second disease would not change how often the model names the first, because the evidence for the first stays the same. But, in our experiment, we observed that the second disease does not just go unnoticed, and it also makes the first one less likely to be diagnosed (Appendix E.6, Table 18).

Lastly, as you suggest, we added one prompting baseline for multi hypothesis reasoning. In this setup, the doctor model is prompted to list every diagnosis with a probability, inspired from MAI-DxO (Nori et al., Science 2026), and ask a question that tests the open ones. Here, we observe that it does not recover more sets (Table R7).

R7. Interactive DDXPlus, 50 cases per k, paper prompt to multi-hypothesis promptk = 1k = 2k = 3k = 4All k [95% CI of change]
GPT-5.6, Jaccard0.47 to 0.420.32 to 0.230.24 to 0.230.22 to 0.120.31 to 0.25 [−0.11, −0.02]
Qwen3.8-Flash, Jaccard0.28 to 0.430.25 to 0.260.22 to 0.250.22 to 0.220.24 to 0.29 [0.01, 0.09]
GPT-5.6, exact-set0.28 to 0.240.10 to 0.000.00 to 0.020.00 to 0.000.10 to 0.07 [−0.07, 0.01]
Qwen3.8-Flash, exact-set0.08 to 0.140.06 to 0.040.02 to 0.000.00 to 0.000.04 to 0.04 [−0.03, 0.04]

We hope this answers your questions, and we would be glad to discuss any remaining concerns.

Reviewer_pwn3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CLIMB introduces a benchmark for multi-turn, multi-label clinical diagnosis. It constructs 800 synthetic cases from each of ePOCT+ and DDXPlus (200 per diagnosis count k = 1, 2, 3, 4), and evaluates six LLM doctors under full-record and interactive consultation settings using exact-set recovery and Jaccard similarity. Interactive exact-set recovery is at most 10% across the reported models. Matched cohorts, a decision-tree questioner, and fixed diagnostic readers are used to investigate evidence acquisition, evidence integration, and anchoring. The contribution is principally a benchmark and failure analysis rather than a new diagnostic method.

Critical strengths
  1. The benchmark targets a meaningful limitation of single-diagnosis medical evaluations: correctly naming one condition is not equivalent to recovering the complete set of concurrent conditions. The controlled variation in k, shared candidate vocabulary, and paired full-information/interactive conditions make this question measurable. The paper also documents sampling rules, exclusion masks, algorithms, prompts, and confidence intervals in substantial detail (Sections 3–5 and Appendices B, D, and E).

  2. The analysis goes beyond reporting low exact-match scores. The matched respiratory cohort separates evidence collection from evidence interpretation by comparing GPT-5.6 with a learned decision tree and a fixed reader (Figure 5; Appendix D.6). The opening-order interventions and symptom-sharing controls further probe anchoring (Section 6.2; Appendix E.7–E.8). These experiments strengthen the behavioral findings, even though they do not identify the model’s internal reasoning mechanism.

Critical weaknesses
  1. The validity and completeness of the synthetic ground truth are not adequately established. DDXPlus disease co-occurrence weights are elicited from an LLM rather than measured from multimorbid patients; clinicians verified exclusion masks, not a blinded sample of complete generated cases (Appendices B.4–B.5). Background complaints are added without a subsequent label-preservation check (Appendix B.6). Figure 2 shows a possible inconsistency: Pneumonia and Scabies are selected in the sampling step, but Anaemia appears in the final ground-truth set after palmar pallor is introduced as a background complaint. This may be a figure error, but it raises the question of whether plausible diagnoses outside the constructed label set are counted as false positives. High recovery by a reader trained on the generator shows recoverability of the generated labels, not their clinical completeness.

  2. The evaluation does not sufficiently separate diagnostic limitations from simulator and prompting effects. Free-text interaction uses an LLM patient whose screening accuracy for absent findings is only 0.83 (Appendix D.1, Table 8). Doctor reasoning is disabled or minimized for every model, and the paper reports substantial changes in open-model stopping behavior when the question counter is moved between prompt locations (Appendix E.10). More importantly, the main evaluation reports one response per case and no independent repeated runs of the same model-condition on the same cases. The bootstrap intervals resample cases; they do not capture variation across complete stochastic model–patient interactions. The decision tree is trained on the same synthetic generator, while the single-hypothesis tracker is an idealized behavioral model rather than evidence of an internal LLM mechanism.

Questions for authors
  1. Can the authors provide an independent, blinded clinician audit of a representative sample of complete generated cases, reporting clinical plausibility, adequacy of evidence for each target label, and the frequency of additional clinically justified but unlabelled diagnoses? In particular, please test whether background complaints introduce new plausible diagnoses, quantify this by source and k, and reconcile the disease-sampling and ground-truth labels in Figure 2. Please also clarify how sensitive DDXPlus case composition is to the LLM-elicited pairwise association factors.

  2. How robust are the central interactive results to the patient and doctor protocols? Please repeat the complete doctor–patient interaction several times on a fixed, representative subset, and report per-run scores and mean ± standard deviation separately from case-bootstrap confidence intervals. A controlled subset using deterministic record-grounded answers or an independently validated patient simulator, together with a stronger reasoning configuration, would help determine whether the observed failures reflect multimorbidity reasoning rather than interaction noise or a particular prompting setup.

1 comment
AuthorsOpenReview ↗

Response to Reviewer pwn3

Thank you for a careful and constructive review. We write a general response since most of the points are shared with the other reviews. You can check the points about: the clinical validity (parts 1 and 2), LLM factors and background complaints (part 2, Tables R2 and R3), the simulator reliability and reasoning (part 3, Table R4), and repeated runs with per-run scores (part 4, Table R5). Here we would like to share the points specific to your review.

For Figure 2, you are right, Anaemia was never sampled. We will redraw the figure from a stored case. The question behind it is still important: are plausible diagnoses outside the label set counted as false positives? On DDXPlus, each source record comes with a differential diagnosis. When predictions in that list are not counted as errors, mean Jaccard still falls from 0.47 at k = 1 to 0.27 at k = 4 in interaction (Table R6). Note that this list grows with k, which favours larger k, yet Jaccard still falls.

R6. DDXPlus Jaccard, six models, differential diagnoses not counted as errorsk = 1k = 2k = 3k = 4
Interactive0.470.350.290.27
Full information0.690.550.500.48

For the sensitivity of sampling with LLM, we do a simple check. When we set every factor to 1, which means independence, the share of strongly associated pairs (factor ≥ 3) falls from 41% to 16%. So, as expected, factors change which pairs are sampled, but as you will see in the general response, scores at k = 2 do not differ between weakly and strongly associated pairs (Table R2).

For the 0.83 in Appendix D.1, we would like to clarify that it comes from a regex based screen, which we used to choose the patient model at a low cost. You can check our audit of the deployed patient in part 3 of the general response.

For the decision tree and the tracker, we agree that both are reference models and they are not claims about the model's internal behaviors (Appendices D.6 and F).

We hope this answers your questions, and we would be glad to discuss any remaining concerns.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General response (1/4): Overview and clinical validity

Dear AC, dear reviewers,

We would like to thank the reviewers for a set of careful and constructive reviews. Across the reviews there are two topics of criticism and questions:

  • How do you know that your benchmark is clinically valid? Or in other words: Since you are measuring behaviour on synthetic data, how do you know that it represents clinically relevant real-world scenarios?
  • How do you know that your measurements are robust? There are several sources of noise in your benchmarking setup, did you account for them correctly?

We will answer both questions here, and then write short individual replies to each reviewer to discuss additional points.

For the first question, clinical validity, we can now present additional data that supports both the relevance as well as the faithfulness to real-world scenarios.

For the second question, robustness and reliability, we would like to push back against the reviewers’ concerns. We hope that our arguments here will convince them of the soundness and adequate statistical power of our measurements.

Clinical Validity

We agree that the fundamental sanity check for a domain specific benchmark must be: Is the data realistic?

The key motivation for our benchmark comes from an analysis of the ePOCT+ clinical decision support algorithm. We have access to a large-scale dataset of consultations driven by ePOCT+, and we noted that in these real-world cases, multimorbidity is very high. On average, a consultation ended with 2.3 diagnoses after clinician review in Tanzania and Rwanda (Poitiers et al., 2025).

This setting is surprisingly understudied in the literature on AI-driven decision support. While the field has investigated multimorbidity in single-turn scenarios, there is no established benchmark that measures multimorbidity in multiturn consultations. Our benchmark aims to study a very real and very relevant gap.

But this doesn’t yet answer the question whether our samples are representative of real-world cases. Unfortunately, patient privacy and partner agreements prevent us from publishing the raw ePOCT+ consultation data.

However, after submitting the manuscript, we have received approval to share aggregate statistics that were used internally to validate our sample construction.

For the stratum of k=2 multimorbidities, our synthetic patient vignettes match real patients almost perfectly. We counted how often each disease appears in our 200 generated cases and in the real cohort, and the two frequencies correlate at r = 0.985. A random sample of 200 real patients reaches r = 0.988.

Additional to that, we ran the models on 160 real consultations (40 per k) in the same interactive pipeline. These patients were not produced by our generator, but they are coming from consultations of physicians that used ePOCT+. In these cases, the pattern is the same as on our generated cases:

R1. Real ePOCT+ patients, interactive, exact-setk = 1k = 2k = 3k = 4
GPT-5.60.600.280.050.03
Qwen3.8-Flash0.620.420.030.00

Here, the same drop with k appears, even though the models are explicitly told that the patient may have more than one disease and that they should keep investigating after the first likely diagnosis (Appendix D.9). Therefore, our conclusion is that this is not about how we build the cases, because real cases also create the same failure.

AuthorsOpenReview ↗

General response (2/4): Clinical validity, continued

For higher multimorbidity counts we are facing the curse of dimensionality: We can no longer reliably model a joint distribution, because the possible sets of three or four diseases far outnumber the records we have. Instead, we build larger sets from pairs: we look at how often two diseases occur together, and chain these pairs into sets of three or four. Longer disease patterns have also been built from pairwise associations before (Jensen et al., Nat Commun 2014). Each new diagnosis also has to pass clinically verified exclusion masks and consistency rules (Appendices B.3 to B.5). However, while we do not know the correct probability, we do know the support of the real cases. We can assure the reviewer: The synthetic samples are plausible.

On the ePOCT+ data, our findings are alarming: While LLMs show some capability to navigate multiturn consultation scenarios, their capability collapses drastically when multimorbidity is present. This raises the question: Is this shortcoming specifically limited to ePOCT+?

We have decided to extend our benchmark construction to the seminal medical vignette dataset: DDXPlus. Here we must choose a different approach to synthesize multimorbid patients, because every record has a single disease. Our medical co-authors have advised this process carefully and our decisions are grounded in literature. Prevalences come from the data itself. Without further information, the natural choice would be to treat diseases as independent. This is actually what the classical multi-disease systems such as QMR-DT did, and it was evaluated on real clinical cases (Shwe et al., 1991; Middleton et al., 1991).

Here, we add one step on this independence assumption. For each pair, an LLM judges whether the two conditions occur together more often than chance. Eliciting such association strengths from LLMs is established practice (AutoElicit, Capstick et al., ICML 2025). The factor only would push the sampling towards the associations which are common and should make the task easier. We observe no change in the score across factors (Table R2). .

R2. DDXPlus k = 2, by LLM factor of the pair, six modelsUp to 11 to 3Above 3Above 3 minus up to 1 [95% CI]
Full information, exact-set0.170.130.12−0.05 [−0.19, 0.06]
Full information, Jaccard0.470.450.43−0.04 [−0.15, 0.08]
Interactive, exact-set0.080.040.05−0.04 [−0.09, 0.02]
Interactive, Jaccard0.320.260.26−0.06 [−0.14, 0.03]

The reviewers also asked about the background complaints. Why add them at all? We believe that a patient who is generated only from disease records would report nothing but the finding of their diseases. However, real patients are not like that. At any time, most people may carry a few everyday symptoms and these have nothing to do with why they came, which is also known as the symptom iceberg (Elnegaard et al., BMC Public Health 2015). We therefore add such complaints which are drawn from the symptom space of Elnegaard et al. We believe that an LLM doctor has to set them aside, as how clinician also would do it.

However, based on the reviewers’ concerns about background symptoms, we want to check two things about them: do they change the output accuracy of the models? and are they clinically proper to integrate to the vignettes? For the first question, we observe two things: adding these symptoms on full vignette on DDXPlus cases changes the exact score by at most 0.01 (95% CI within ±0.02). We also do a controlled experiment, by changing the number of background symptoms, and we see no visible effect (Table R3). This showed us that background symptoms do not change the clinical accuracy, so our results are not an artifact of it. For the second question, we run an LLM as judge (GPT-5.6) passes on dynamically generated 300 cases and LLM found only 3/300 as harder to identify with the background symptoms.

R3. Exact-set by number of background complaints, 50 DDXPlus cases (k = 1 and 2)01234
GPT-5.6, full information0.180.160.180.160.16
Qwen3.8-Flash, full information0.320.280.360.300.30
GPT-5.6, interactive0.100.060.100.060.10
Qwen3.8-Flash, interactive0.200.160.100.100.18

Finally, all three reviewers asked for a clinician check. We hold a clinician panel who are now rating generated cases on two things: whether the merge is clinically reasonable in terms of disease and symptom space, and whether the background makes the diagnoses harder to identify. In these same cases, LLM as judge found that the merge was clinically reasonable in 95%, 93%, 78% of cases at k=2, 3, 4. We plan to post panel ratings before the end of discussion period. You can sample dynamic cases from our generator here: https://climb-case-review.vercel.app/

AuthorsOpenReview ↗

General response (3/4): Reliability

Reliability

Several reviewers discuss reliability and soundness of our benchmarking methodology. We believe one key concern is: Our scenario involves multiple sources of stochasticity. Diagnostic uncertainty, as well as noisy replies from a patient simulator, can contribute to the observed failures. What implications does that have for the robustness of our observations?

We thank the reviewers for bringing up this important topic. Indeed, research on LLM behaviour often suffers from arbitrary choices around prompt design and is subject to noise and reproducibility issues.

A multiturn medical consultation is essentially an active learning task. A set of latent diseases gives rise to the symptoms recorded in a vignette. The (AI) doctor must infer these diseases by asking about symptoms, with answers provided by a noisy oracle, the (LLM) patient. It faces uncertainty both in how diseases manifest and in how symptoms are reported. The two sources of uncertainty are intimately connected, our benchmark deliberately models the complete interaction.

The field has already established how to “talk to a vignette” (AgentClinic, Schmidgall et al., npj Digit Med 2026; MediQ, Li et al., NeurIPS 2024; CRAFT-MD, Johri et al., Nat Med 2025; PatientSim, Kyung et al., NeurIPS 2025; VivaBench, Chiu et al., NeurIPS 2025; SDBench, Nori et al., 2025): an LLM patient answers the doctor’s questions using the structured information of a clinical record. We build on the AgentClinic protocol (Schmidgall et al., npj Digit Med 2026), as documented in Appendix D.8. Our contribution is to introduce vignettes with controlled multimorbidity.

So, that is why we reported our main numbers with this interactive patient evaluation. More importantly, we should note that the failure does not actually need an interactive patient at all. Two other setups, both already in the paper, show the same drop with k: (i) the full information setup which does not involve any simulator (Tables 2 and 6) and (ii) the menu channel which is an interactive deterministic patient simulator, as the reviewers asked for (Table 12).

The reviewers on the other hand rightfully questioned if our setup gives the doctor models a fair chance. We see two common points here: (1) how would it behave if we make reasoning on, (2) do our models reliably simulate the patient?

(1) As we stated in the paper, reasoning was at minimal effort by design (Appendix D.1), as reasoning models "deteriorate in similar ways to non-reasoning models" in multi-turn conversation (Laban et al., ICLR 2026). Still, based on the concerns, we reran two doctors on the 200 DDxPlus cases (50 per k) with reasoning on and we observe that it does not fix the failure (Table R4)

R4. Exact-set, reasoning off to on, 200 DDXPlus cases (GPT-5.6 at medium effort, Qwen3.8-Flash with an 8k-token budget)k = 1k = 2k = 3k = 4All k
GPT-5.6, full information0.32 to 0.500.08 to 0.200.02 to 0.080.00 to 0.040.10 to 0.20
Qwen3.8-Flash, full information0.46 to 0.340.20 to 0.040.02 to 0.040.00 to 0.000.17 to 0.10
GPT-5.6, interactive0.28 to 0.100.10 to 0.000.00 to 0.000.00 to 0.000.10 to 0.03
Qwen3.8-Flash, interactive0.08 to 0.140.06 to 0.020.02 to 0.020.00 to 0.040.04 to 0.06

(2) As we stated above, we followed the line of related work and integrate an established protocol. To address the concerns about the patient, we ran LLM as judge on 300 cases, and based on the output, the model never denied a recorded finding and volunteered an unrecorded evidence in about 5% answers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.