acceptodds
Withdrawn from ICLR 2027

BrainIPAgent: Multi-Agent Interpretable Structural Priors for Multimodal Brain Network Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The effective integration of structural connectivity (SC) and functional connectivity (FC) is crucial for brain disorder diagnosis. However, existing methods remain predominantly data-driven and fail to effectively incorporate interpretable priors, leading to limited circuit-level interpretability. Static multimodal fusion strategies further limit the adaptive reliance on structural priors. In this paper, we propose BrainIPAgent, a multimodal brain network learning framework with multi-agent interpretable structural priors and RL-guided alignment. BrainIPAgent maps structural patterns into an attention bias in the logit space, enabling soft control over information flow in functional representation learning. To preserve structural consistency across hierarchies, we introduce a mask propagation operator that projects structural constraints through soft pooling. Multimodal fusion is formulated as a bilevel optimization problem, where contrastive alignment models the distributional discrepancy between textual embeddings derived from iteratively refined clinical reports and continuous functional representations. An RL-guided controller then adaptively regulates the strengths of structural guidance and contrastive alignment throughout training. Experiments on ADNI and real-world datasets demonstrate that BrainIPAgent achieves superior classification performance while providing interpretable insights. The code is available at https://anonymous.4open.science/r/BrainIPAgent.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain network learning, multi-agent, multimodal fusion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gm1rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BrainIPAgent, a framework for integrating structural connectivity and functional connectivity for brain disorder classification. It extracts structural patterns, generates clinical reports, and uses them to guide functional encoding and classification. Experiments on ADNI and two clinical cohorts demonstrate improvements over reported baselines.

Critical strengths
  1. The framework uses SC through both explicit attention biases and textual representations, providing complementary pathways for integrating structural knowledge into functional representation learning.
  2. The framework also offers a basis for studying how multi-agent systems as tools are involved in the brain model.
Critical weaknesses
  1. The interpretability claims exceed the current evidence. The generated reports are SC-derived input priors whose embeddings enter the agents, so they describe the input and its presumed clinical relevance but do not establish faithfulness to the proposed model’s decisions. In other words, they are from the LLM's knowledge and the value of dMRI. In particular, prompts encourage disease-relevant patterns and later utilize the disease-related paper to explain these findings.
  2. The current ablation of the RL controller does not separate policy learning from repeated candidate evaluation and validation-based selection. Could you please compare against a matched non-learning best-of-four sampler and a tuned (α\alpha, λ\lambda) schedule. The claim in line 62 that structural constraints help early training but limit later plasticity also requires stage-specific analyses or parameter trajectories.
  3. The hierarchical pooling formulation needs to be clarified. The dimensions of attention and hierarchical mask are unclear, particularly between Eqs. 6–7. If only pooling SC mask, their dimension seems to be mismatched with each other. Further, could you please clarify whether cluster centers are trainable and how cluster centers are determined?
  4. The contribution of SC-guidance versus direct prior knowledge from LLMs remains unclear. The gains may be primarily from generated textual information related to the corresponding brain disease.
Questions for authors
  1. Although the authors report some computational costs, no exact model complexity is reported. Since the RL-controller add 5 minibatches per epoch and LLM's inference, their cost may be substantial relative to standard training. Could you please provide matched runtime or FLOP/MAC comparisons with the baselines, and report both cached-prior and end-to-end inference latency, including prior generation and text encoding.
  2. Could a type of agent finish extracting the pairwise/triplet patterns and generate the corresponding clinical reports? Extracting the top value from SC might be an easy task.
Reviewer_zjALOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BrainIPAgent, a multimodal brain network learning framework that integrates structural connectivity (SC) and functional connectivity (FC) for diagnosing brain disorders. The framework employs a multi-agent LLM system to extract pairwise and higher-order structural patterns and generate clinically oriented textual priors. These priors guide functional representation learning through structure-aware attention and hierarchical mask propagation, while a reinforcement learning controller dynamically adjusts structural guidance and contrastive alignment strengths during training. Experiments on ADNI and two clinical cohorts, including a combined multi-site evaluation, report improvements over existing single-modal and multimodal baselines. Ablation studies, analyses of multi-agent dialogue, and interpretability visualisations are presented to investigate the effectiveness of the proposed components.

Critical strengths

-The paper addresses an important challenge in multimodal brain network learning by incorporating structural connectivity patterns into functional representation learning. Structure-aware attention and hierarchical mask propagation provide a meaningful way to guide functional interactions using anatomical information.

-The reinforcement learning controller dynamically adjusts the strengths of structural guidance and contrastive alignment during training, addressing the limitations of fixed multimodal fusion strategies. This adaptive mechanism could be a useful contribution for handling heterogeneous brain connectivity modalities.

-The proposed method achieves the highest reported classification accuracy across all four evaluation settings. The ablation studies also show that textual alignment, structural guidance, and RL-based adaptation improve overall performance, supporting the proposed framework.

Critical weaknesses

-Although the paper emphasises collaborative clinical reasoning as a central contribution, it does not sufficiently demonstrate that multiple specialised LLM agents provide meaningful advantages over simpler alternatives, such as a single LLM, direct structural feature encoding, or template-based report generation. The reported dialogue-turn analysis does not isolate the benefits of multi-agent collaboration, making it difficult to justify the additional architectural complexity.

  • The claim that the generated structural priors and clinical reports provide biologically grounded interpretations is primarily supported by qualitative visualisations and representative examples. The paper lacks systematic quantitative evaluation of anatomical correctness, clinical reliability, and consistency of the generated explanations. Given the possibility of LLM hallucinations, stronger validation is necessary to support the claimed clinical interpretability.

  • The proposed controller introduces additional training complexity, but the paper does not adequately establish why reinforcement learning is necessary compared with simpler adaptive weighting strategies or predetermined schedules. Although removing RL reduces accuracy, this ablation does not isolate RL's advantage over other dynamic optimisation methods. Furthermore, the reward design and controller stability require more detailed analysis.

-The reported improvements are encouraging, but the paper does not provide statistical significance tests directly comparing the proposed method against the strongest baselines. The evaluation also lacks comprehensive analysis of computational overhead, inference cost, and scalability associated with multi-agent interactions and RL-guided optimisation. These omissions make it difficult to assess whether the performance gains justify the added complexity.

Questions for authors

-What specific benefits does the proposed multi-agent framework provide over a single LLM or directly encoding structural connectivity patterns without generating clinical reports? Can the authors provide controlled comparisons showing that iterative agent collaboration improves classification performance and interpretability?

  • Why is reinforcement learning preferable to simpler adaptive weighting mechanisms, such as learnable parameters, validation-based scheduling, or heuristic annealing? Can the authors provide comparative experiments and analyse the stability of the learned structural guidance and alignment weights throughout training?

  • How is the clinical correctness of the LLM-generated reports and structural priors evaluated? Can the authors provide quantitative evidence, ideally involving clinical experts, that the generated explanations are anatomically accurate, reproducible across LLM backbones, and more informative than explanations produced by existing interpretability methods?

Reviewer_3j5mOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

BrainIPAgent proposes a multimodal brain network learning framework integrating structural connectivity (SC), functional connectivity (FC), LLM-generated clinical textual priors, and reinforcement-learning-based adaptive alignment. Multiple agents extract pairwise and triplet-based structural patterns, generate and refine clinical descriptions, and use these outputs to guide attention and cross-modal contrastive learning. The framework is technically ambitious and obtains competitive performance on ADNI and two clinical cohorts.

Critical strengths
  1. The research entry point is valuable. Using structural connectivity (SC) as a structural prior for modeling functional connectivity (FC) is more consistent with neuroscientific principles than simply concatenating the two modalities.

  2. The method for incorporating the structural prior is well-conceived. By injecting a structural mask into the attention logits—rather than imposing hard constraints that would overwrite the FC—the model retains a degree of functional plasticity.

Critical weaknesses
  1. The authors employ LLM agents to select significant regions based on structural connectivity and generate clinical reports, which are subsequently refined by an "Expert Commentator Agent." However, the fact that one LLM assigns a high score to a neuroscience-based interpretation generated by another LLM does not establish the clinical validity of that interpretation; this significantly undermines both the motivation and the methodological innovation of the paper.

  2. So-called high-order structural patterns are actually just three pairwise connections in a triplet. Eq. (2) represents higher-order patterns as combinations of triplets, but the combination of three pairwise edges is not automatically equivalent to a true higher-order interaction. The authors need to prove that the triplet prior provides information beyond three individual pairwise connections; otherwise, the authenticity of the circuit-level higher-order reasoning is questionable.

  3. The methodology incorporates multiple LLM agents, multi-turn dialogues, text encoding, hierarchical graph encoding, and epoch-by-epoch reinforcement learning (RL) trials. Notably, as described in Appendix B, multiple candidate actions are tested during each epoch, which incurs additional training overhead. Although the authors discuss this overhead, a comparison against strong baselines—given an equivalent computational budget—is necessary to justify the complexity introduced by the multi-agent and RL components.

Questions for authors

see the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.