DIAGRARE-BENCH: SEPARATING DISEASE PRIORS FROM PATIENT EVIDENCE IN LANGUAGE MODELS FOR CLINICAL DIAGNOSIS

Diagnostic benchmarks tell us whether a language model selected the correct disease, but they do not identify what drove that decision. A diagnosis can rank highly because the findings in the case support it, because the disease is common, or because both signals agree. We introduce DIAGRARE-BENCH to separate these effects at the level of the model's diagnostic ranking. Given a ranked differential, we fit a Plackett-Luce choice model with two interpretable features: a disease-prior feature and a case-specific evidence feature. The fitted coefficient ${\gamma }_{\mathrm{evidence}}$ measures how strongly pairwise diagnostic log odds change with an evidence contrast when the prior contrast is held fixed. The estimator is model agnostic and uses ranked outputs only. We evaluate 25 open-weight checkpoints, spanning both general-purpose models and models specialized for medicine. The primary evaluation contains 984 diagnostic cases covering 82 diseases and nine organ systems; 21 checkpoints yield identifiable baseline coefficients. Evidence responsiveness is associated with top-1 accuracy ($r=0.843$), but our main tests ask whether the coefficient predicts behavior when the evidence is changed independently of the rankings used to estimate it. The cross-model ordering transfers to all 3,562 CUPCase reports ($r=0.662$) and all 22,901 RareArena cases ($r=0.790$) under a different evidence representation. In a randomized $3\times 3$ experiment that varies evidence strength and stated prevalence independently, baseline ${\gamma }_{\mathrm{evidence}}$ predicts the change in target choice caused by stronger evidence ($r=0.689$, $p=5.5\times 10^{-4}$). In a sequential diagnosis task, it predicts whether a model revises an initial competing diagnosis after target-specific evidence arrives ($r=0.860$, $p=9.8\times 10^{-6}$). QLoRA adaptation, self-consistency, and test-time reasoning experiments further show that diagnostic accuracy and evidence responsiveness can change by different amounts. These results support evidence responsiveness as a distinct and measurable property of diagnostic decision behavior.