Language-Guided Graph Reasoning for Concept Bottleneck Models

We present MoE-SGT, a concept bottleneck model that combines editable concept scores with offline question–answer (QA) knowledge. Cached answer–question graphs encode class semantics; answer–concept graphs combine sample-dependent concept evidence with semantic and co-occurrence priors. Present and absent concept embeddings let corrected scores update predictions without retraining. A structure-aware router selects two experts alongside a shared expert. MoE-SGT achieves the highest Top-1 accuracy among the evaluated CBMs on all four single-label datasets, including CUB-200, and the highest ROC-AUC and F1 on both medical datasets. On CUB-200, graph reasoning raises accuracy from 75.15% with the linear head to 80.31%. With ten concept corrections, accuracy reaches 86.39% under graph-informed selection and 89.92% under oracle selection.