GC-NCE: Graph-Constrained Nuisance-Aware Clinical Evidence Bottleneck for Medical Image Classification under Domain Shift

Concept bottleneck models (CBMs) predict medical image diagnoses through human-interpretable concepts, but they can remain unreliable under distribution shift: an unconstrained diagnostic head can weight concepts by source-domain correlations rather than their clinical roles, and predicted concept scores can encode nuisance factors that classifiers exploit as shortcuts. We introduce GC-NCE, a framework that constructs clinical evidence graphs from retrieved medical literature and uses them to constrain diagnostic inference. The graphs assign typed roles to concepts: direct findings and target mechanisms provide diagnostic support, differential findings provide inhibition, anatomy modulates evidence visibility, and nuisance influence edges restrict where acquisition effects may be corrected. Four LLM agents build each graph through compiler-gated proposals and cross-agent review, and an observability-aware loop refines concepts that cannot be reliably predicted from images. GC-NCE translates these roles into probabilistic inference over latent disease mechanisms, with adaptive nuisance correction enabled per dataset only when source validation confirms its benefit. On ten domain-shift benchmarks, GC-NCE achieves the highest mean out-of-distribution AUROC in both chest X-ray (78.5%) and skin lesion (84.5%) modalities.