acceptodds
Withdrawn from ICLR 2027

Language-Guided Graph Reasoning for Concept Bottleneck Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present MoE-SGT, a concept bottleneck model that combines editable concept scores with offline question–answer (QA) knowledge. Cached answer–question graphs encode class semantics; answer–concept graphs combine sample-dependent concept evidence with semantic and co-occurrence priors. Present and absent concept embeddings let corrected scores update predictions without retraining. A structure-aware router selects two experts alongside a shared expert. MoE-SGT achieves the highest Top-1 accuracy among the evaluated CBMs on all four single-label datasets, including CUB-200, and the highest ROC-AUC and F1 on both medical datasets. On CUB-200, graph reasoning raises accuracy from 75.15% with the linear head to 80.31%. With ten concept corrections, accuracy reaches 86.39% under graph-informed selection and 89.92% under oracle selection.

Primary area:
interpretability and explainable AI
Keywords:
Concept Bottleneck Models, Concept Intervention, Language-Guided Graph Reasoning, Interpretable Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BaYtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MoE-SGT, a language-guided graph reasoning framework for concept bottleneck models (CBMs). The method integrates editable concept predictions with offline-generated question–answer knowledge through answer–concept and answer–question graphs. It employs present/absent concept embeddings, graph attention with semantic and co-occurrence priors, and a structure-aware mixture-of-experts module with shared and sparsely routed experts.

The model is evaluated on four single-label classification datasets (CUB-200, ImageNet, CIFAR-10, and CIFAR-100) and two medical multi-label datasets (MIMIC-CXR and CheXpert). The reported results outperform the evaluated CBM baselines across these datasets. Additional experiments investigate concept prediction, test-time interventions, module ablations, and computational efficiency. On CUB-200, the method achieves 80.31% Top-1 accuracy, increasing to 86.39% with ten graph-informed concept corrections and 89.92% under oracle correction.

Critical strengths

1. Meaningful integration of language knowledge and editable concept reasoning.

The paper presents a coherent framework for combining offline class-level QA semantics with sample-dependent concept evidence. The use of present/absent concept embeddings and answer-conditioned graphs provides a reasonable mechanism for incorporating both positive and negative concept evidence. Importantly, concept corrections can update candidate retrieval and graph predictions without retraining, which is valuable for interpretable and interactive classification.

2. Extensive empirical evaluation and encouraging intervention results.

The evaluation covers six datasets, including natural-image and medical-image classification. The proposed method consistently outperforms the evaluated CBM baselines. The concept intervention experiments are particularly encouraging: on CUB-200, ten graph-informed corrections improve accuracy from 80.31% to 86.39%, outperforming uncertainty-only and random selection. The ablation studies also provide evidence that graph reasoning, language knowledge, and sparse experts contribute to predictive performance.

Critical weaknesses

1. Limited methodological novelty and insufficient isolation of the main contributions.

The proposed framework combines several established techniques, including concept embeddings, graph-based reasoning, language-guided representations, and sparse mixture-of-experts routing. Although their integration is reasonable, the paper does not clearly establish which components constitute a substantive methodological advance beyond prior graph-based CBMs and related architectures. In particular, the distinction from GraphCBMs and existing graph Transformer approaches deserves a more rigorous analysis. The current ablations demonstrate incremental performance gains but do not fully isolate the necessity of answer-conditioned QA reasoning or structure-aware expert routing.

2. Insufficient experimental rigor to establish robust and fairly controlled improvements.

All main results are reported using a single training seed, without standard deviations or confidence intervals. This is particularly concerning on CUB-200, where the improvement over Sparse-CBM is only 0.34 percentage points. Furthermore, some baselines use different concept acquisition and supervision procedures, making it difficult to attribute performance differences exclusively to the proposed reasoning architecture. The frozen-feature probe also outperforms the proposed model on all six datasets, and the paper does not sufficiently quantify the resulting accuracy–interpretability trade-off. Stronger controlled comparisons and multi-seed evaluations are needed to substantiate the claims.

Questions for authors
  1. Novelty and architectural necessity: How does MoE-SGT differ fundamentally from GraphCBMs and existing graph Transformer-based reasoning approaches? Can the authors provide a controlled comparison using the same concept features and supervision, replacing the proposed answer-conditioned graph architecture with a simpler concept graph Transformer?

  2. Statistical reliability: Can the authors report results over at least three independent training seeds, particularly on CUB-200, where the improvement over Sparse-CBM is only 0.34 percentage points? Are the reported gains consistent across runs?

  3. Fairness of baseline comparisons: Since the baselines use different concept acquisition and supervision protocols on several datasets, can the authors provide comparisons under a unified concept vocabulary and supervision setting? This would help isolate the benefit of the proposed reasoning module.

  4. Accuracy–interpretability trade-off and expert routing: Given that the frozen-feature probe outperforms MoE-SGT on all six datasets, how do the authors justify the additional architectural complexity? Can they provide matched-parameter and matched-compute comparisons together with intervention performance, especially against a dense graph Transformer and a feature-only MoE router?

Reviewer_sWhdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoE-SGT, a concept bottleneck model that integrates language-derived knowledge with editable concept representations through answer-conditioned graph reasoning. The framework combines answer–concept and answer–question graphs with a structure-aware mixture-of-experts module to improve classification and support concept interventions. Experiments on six benchmark datasets demonstrate competitive improvements over existing CBMs, while ablation studies and intervention evaluations further validate the effectiveness of the proposed components.

Critical strengths

(1) The integration of language-guided graph reasoning with editable concept representations provides a meaningful approach to improving both predictive performance and concept-level intervention capabilities. (2) The method achieves strong results across six datasets, consistently outperforming the evaluated CBM baselines and demonstrating applicability to both general image classification and medical imaging tasks. (3) The paper provides comprehensive ablation studies covering graph reasoning, structural priors, expert routing, and concept interventions, helping clarify the contributions of individual components.

Critical weaknesses

(1) The framework combines several existing techniques, including graph Transformers, concept embeddings, and mixture-of-experts, making the methodological novelty somewhat incremental. (2) Despite outperforming other CBMs, the proposed model consistently underperforms the frozen-feature probe across all six datasets, raising concerns about the accuracy–interpretability trade-off and the practical value of its additional complexity. (3) All experiments use a single training seed, and the paper lacks sufficient analysis of robustness to variations in LLM-generated concepts and QA knowledge, making the stability of the reported improvements unclear.

Questions for authors

How sensitive is MoE-SGT's performance to the quality and consistency of LLM-generated concepts and QA knowledge, particularly when using different language models or alternative prompts?

Reviewer_EtH7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoE-SGT, a language-guided graph reasoning framework for Concept Bottleneck Models (CBMs). The method leverages large language models to generate class-level question–answer (QA) knowledge and constructs answer–question graphs and answer–concept graphs to integrate class-level semantic knowledge with sample-dependent concept evidence. It further introduces editable concept representations and a structure-aware Mixture-of-Experts (MoE) Graph Transformer to support prediction updates through concept interventions. Experiments on multiple visual classification and medical multi-label classification datasets show that MoE-SGT achieves competitive classification performance compared with the evaluated CBM baselines and demonstrate that concept corrections can improve prediction outcomes.

Critical strengths
  1. The research problem is meaningful. The paper investigates concept relation modeling and concept intervention in CBMs, exploring how language knowledge and graph reasoning can improve model interpretability and intervention capabilities.
  2. The framework is systematically designed. The paper integrates class-level QA knowledge, editable concept representations, dual-graph reasoning, and a structure-aware MoE mechanism into a unified pipeline, covering knowledge construction, classification, and concept correction.
  3. The experimental evaluation is relatively comprehensive. The paper evaluates the proposed method on multiple visual classification and medical multi-label classification datasets, covering classification performance, concept prediction, and concept intervention.
  4. The paper includes extensive ablation studies. By comparing different knowledge inputs, graph reasoning modules, structural priors, and expert routing strategies, the authors investigate the contributions of individual components to the overall performance.
Critical weaknesses
  1. The core research question is insufficiently articulated, and methodological novelty is limited. The paper primarily integrates existing techniques, including language-guided concept learning, graph reasoning, Graph Transformers, and MoE, into a CBM framework. However, the central scientific question that the proposed method aims to address is not clearly defined, and the method does not sufficiently demonstrate substantive technical innovations that distinguish it from existing work.
  2. The motivation and necessity of the dual-graph architecture are insufficiently justified. The paper constructs separate answer–concept graphs and answer–question graphs to model sample-dependent concept evidence and class-level language knowledge, respectively. However, it remains unclear why these two types of information need to be modeled using two separate graphs rather than a more unified representation and reasoning mechanism. The rationale for this architectural choice requires further justification.
  3. The source of the performance gains from the MoE mechanism remains unclear. The paper introduces a structure-aware routing mechanism intended to achieve expert specialization by leveraging node features, node types, and structural context. However, the causal relationship between the observed performance improvements and the proposed routing mechanism has not been sufficiently established, making it difficult to assess the distinctive value of this design.
  4. The independent contribution of the concept intervention mechanism is insufficiently demonstrated. Concept correction is already a fundamental capability of existing CBMs. Although the paper combines class-conditioned graph reasoning with graph-informed concept selection, the substantive innovations of these designs relative to existing concept intervention methods remain unclear.
  5. The writing and visual presentation could be improved. The paper contains numerous equations and complex mathematical notations, and some technical details are presented in an unnecessarily elaborate manner, making the method difficult to understand. In addition, the overall framework diagram contains numerous modules and information flows, resulting in a visually cluttered presentation that reduces the paper's readability.
Questions for authors
  1. What is the necessity of the class-conditioned dual-graph architecture?

The paper constructs separate answer–concept graphs and answer–question graphs to model sample-dependent concept evidence and class-level language knowledge, respectively. Compared with a shared global concept graph or class-conditioned feature fusion through class embeddings and attention mechanisms, what irreplaceable advantages does the proposed answer-conditioned graph construction offer? In particular, what benefits does class-conditioned graph construction provide beyond the class-semantic information itself? The current experiments do not sufficiently establish the necessity of the dual-graph architecture or demonstrate its advantages over simpler alternatives.

  1. Is explicit graph structure necessary for the proposed reasoning mechanism?

The paper constructs graph structural priors using concept semantic similarity, concept co-occurrence relationships, and node types, and employs a Graph Transformer to model relationships among nodes. However, a standard Transformer or self-attention mechanism can also facilitate interactions between concept evidence and class-level semantics. How much of the observed performance improvement comes from the explicit graph structure and its relational priors, and how much could be achieved through conventional attention-based fusion? The current experiments do not sufficiently demonstrate the independent advantages of explicit graph structure over attention-based fusion without graph structure.

  1. How can the contributions of the proposed method, external language knowledge, and additional model capacity be disentangled?

The paper introduces LLM-generated QA knowledge and combines it with graph reasoning and MoE modules to construct the overall model, but its primary comparisons are against existing CBM methods. How can the authors establish that the performance improvements primarily result from the proposed reasoning mechanism rather than the additional class-level semantic knowledge or model capacity? Furthermore, LLM-generated QA knowledge may vary in accuracy, redundancy, and class-discriminative information, and the effects of these factors on the final performance remain unclear. Are the current experiments sufficient to disentangle the contributions of external knowledge from those of the proposed architecture and to characterize the method's dependence on QA knowledge quality?

  1. What mechanism actually accounts for the performance gains of MoE?

The paper introduces structure-aware routing, which selects experts based on node features, node types, and structural context. Ablation studies demonstrate performance improvements over a dense FFN and feature-only routing. However, these results do not sufficiently explain the underlying source of the gains. Do different experts actually learn distinct transformations of node representations or specialize in different structural roles? Specifically, how does structural context influence expert selection and improve graph reasoning? Do the current experiments provide sufficient evidence that the benefits of structure-aware routing arise from effective expert specialization rather than additional model capacity or conditional computation alone?

  1. What additional benefits does the proposed concept intervention mechanism offer over existing editable CBMs?

Concept correction is already a fundamental capability supported by existing CBMs. The paper further uses graph reasoning to update concept evidence and selects concepts for correction based on concept uncertainty and graph attention importance. Compared with existing editable CBMs, what independent technical benefits do the proposed graph-based concept propagation and graph-informed selection mechanisms provide, respectively? In particular, to what extent do the reported results reflect the contribution of graph reasoning itself, rather than the general benefits of concept correction as an existing capability? Furthermore, the oracle strategy relies on ground-truth concept labels for selecting concepts that have not yet been queried. How should the performance gap between the oracle and practical graph-informed strategies be interpreted, and to what extent do these results support the effectiveness of the proposed method in real-world interactive settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.