CLAIR-Med: Selective Neural–Symbolic Reasoning with Weakly Supervised Concepts for Medical Image Classification

Rule-based concept models expose an explicit decision process but are often less accurate than neural predictors. However, always falling back to the neural model renders the symbolic explanation irrelevant to the final decision. We investigate a narrower question: how often can an explicit symbolic branch make the prediction while retaining competitive task utility? CLAIR-Med constructs medical concept vocabularies using a language model, derives pseudo concept targets from vision–language similarity, and learns an image-to-concept interface without manual concept annotations. The predicted concept state is consumed by two structurally different branches: a concept-augmented predictive branch and a Deep Concept Reasoner (DCR) branch that executes fuzzy rules on concept truth values. We distill class-level predictive information from the predictive branch into the DCR branch and use the predictive entropy of DCR's output distribution to dynamically route each input between the two branches at inference. Experiments on five medical image datasets and CUB show that distillation narrows the predictive–DCR utility gap across all six benchmarks without degrading predictive-branch accuracy. Entropy-based routing enables the DCR branch to supply the final prediction on a subset of inputs while retaining near-predictive task utility. We further evaluate whether the displayed rules reproduce DCR's decision behavior, while blinded medical-domain ratings and concept-deletion analysis assess concept quality and prediction sensitivity, respectively. Together, these results demonstrate the feasibility of selective symbolic prediction through an explicit concept-rule pathway while retaining competitive task utility.