- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BrainAgent, an inference-time pipeline for classifying functional brain graphs and producing textual explanations. It converts graph features into structured descriptions, invokes tools, retrieves neuroscience knowledge and labeled cases, and uses language-model reflection. Experiments span four resting-state fMRI datasets and several language-model backbones, with component ablations and an explanation-quality study.
Critical strengths
- The application of language-model tools to subject-level brain graphs is timely. The system separates graph description, retrieval, case evidence, and reflection into identifiable stages, making its workflow relatively easy to inspect.
- The use of multiple datasets and language-model backbones, together with component ablations, is helpful for examining whether the pipeline is broadly applicable rather than tuned to a single model.
Critical weaknesses
- The pass@3 results should not be interpreted as the accuracy of a deployable single-label classifier. Appendix D.3, Eq. (8), counts a sample as successful when any of three outputs matches the true label. This is a valid oracle-style evaluation of repeated sampling, but it does not specify how one prediction would be selected without the test label. The stated aggregation of pass@k predictions for precision and recall also requires a clearer, label-independent definition.
- The source of the gains is not sufficiently isolated. Section 2.4 builds case memory from training examples with known labels, which is a legitimate supervised support resource and is not evidence of test-label leakage. However, this evidence and the additional retrieval/tool/reflection budget differ from direct prompting. The paper needs comparisons that give simpler methods the same labeled support information and inference resources before attributing gains to agentic reasoning.
Questions for authors
- What are the repeated pass@1 results and the results of a prespecified label-independent aggregation rule (e.g., majority vote), with uncertainty intervals?
- How does BrainAgent compare with k-nearest-neighbor classification on the same graph representations and a single-pass LLM given the same retrieved labeled cases and inference budget?
- Do the gains persist under repeated subject-level or site-held-out splits, and can explanation faithfulness be tested using graph perturbations or blinded experts?
Thank you for your constructive comments.
W1.
We would like to clarify that pass@3 is intentionally designed as an oracle-style evaluation metric, as explicitly defined in Appendix D.3, Eq. (8). Its purpose is to measure whether repeated LLM inference can generate the correct prediction among multiple attempts, rather than represent the performance of a deployable single-label classifier. Importantly, we also report pass@1, which directly reflects single-run prediction performance without access to the ground-truth label during prediction. For Recall@3 and Precision@3, we follow the same oracle-based aggregation protocol to ensure consistency with the pass@3 evaluation setting. We will make this distinction more explicit in the revised manuscript to avoid interpreting these oracle-based metrics as practical single-label classification results.
W2.
We would like to clarify that our goal is not to attribute all performance improvements exclusively to multi-round agentic reasoning. Instead, BrainAgent is designed to explore how general-purpose LLMs can leverage brain-specific tools, external neuroscience knowledge, and historical cases to perform brain network analysis without task-specific LLM fine-tuning. We acknowledge that CARD introduces labeled training cases as an external support resource. This is an intentional part of our knowledge-enhanced inference framework. Unlike supervised classifiers, BrainAgent does not update the LLM parameters using these labels; instead, it retrieves relevant cases as contextual evidence during inference. Moreover, our ablation studies in Section 3.5 evaluate the contributions of topology-aware understanding, HARK, CARD, and reflection separately. The results show that removing these components generally reduces performance, supporting the effectiveness of their integration. Therefore, our contribution is not limited to multi-round reasoning alone, but lies in exploring a unified agentic framework for knowledge-grounded brain network analysis.
Q1.
As mentioned above, pass@1 and oracle pass@3 serve different evaluation purposes. We use pass@1 to evaluate single-run prediction performance, while pass@3 measures the potential of repeated inference. We agree that majority voting provides an additional perspective on practical multi-run classification. Since our experiments already involve three independent inference runs, the existing prediction records can be used to evaluate a label-independent majority-voting strategy. We will further report repeated pass@1 statistics and majority-voting results, together with their variability, to provide a clearer distinction between single-run performance, practical multi-run aggregation, and oracle-style evaluation.
Q2.
We appreciate this suggestion, as these comparisons can help distinguish the contribution of retrieved cases from that of the agentic reasoning process. Our CARD module is not simply a nearest-neighbor classifier. It retrieves relevant cases using both graph representations and textual reports, and the retrieved information is subsequently integrated with graph analysis and neuroscience knowledge through LLM reasoning. The resulting prediction is generated from multiple sources of evidence rather than directly adopting the labels of the nearest neighbors. Nevertheless, comparing BrainAgent with kNN using the same graph representations and a single-pass LLM provided with the same retrieved labeled cases would be useful for further isolating these effects. We will consider these controlled comparisons, including matched inference budgets, to better characterize the contribution of the overall agentic workflow.
Q3.
Regarding generalization, we would like to highlight that BrainAgent has been evaluated on four different brain network datasets, covering ASD, ADHD, MDD, and HCP gender classification. The same overall framework can be applied to these tasks without task-specific fine-tuning of the backbone LLM. This demonstrates its applicability beyond a single disease or classification scenario. Furthermore, several datasets used in our experiments, particularly ABIDE and Rest-meta-MDD, contain neuroimaging data collected from multiple sites. Their heterogeneous acquisition settings and populations provide a meaningful evaluation context for brain network analysis. Although site-held-out evaluation specifically examines generalization to unseen acquisition sites, our current results already demonstrate the effectiveness of BrainAgent across multiple tasks and heterogeneous datasets. Regarding explanation faithfulness, BrainAgent differs from unconstrained LLM inference by explicitly grounding its reasoning in computed graph properties, retrieved neuroscience knowledge, and similar historical cases. The region-level, subgraph-level, and graph-level measurements provide concrete structural evidence that can be checked against the input brain network.