acceptodds
Under review as a conference paper at ICLR 2027

Braintree: An agentic framework for in-silico vision neuroscience

Abstract

Vision neuroscience has many of the ingredients needed to conduct experiments entirely in-silico. We need AI systems that can reason over scientific knowledge, generative methods for creating precisely controlled stimuli, image-computable models that can predict neural responses with increasing accuracy, and tools that can analyze and interpret data. Here we take the first step toward making a vision neuroscience experiment end-to-end computable. We introduce Braintree, an agentic framework that can autonomously run multi-experiment studies directly with in-silico models of human visual cortex. Given a hypothesis, Braintree can reason over the current literature, design an experiment, build a stimulus set from algorithmically manipulated or synthesized stimuli, run the experiment on candidate brain encoding models, analyze the responses and then use the results to design even stronger tests. The entire loop runs autonomously and can accept human input at any stage. We evaluated Braintree on three increasingly open-ended scientific problems. First, Braintree reproduced published findings using their original stimuli and then asked whether the same effects could be replicated with entirely new stimulus sets that it constructed. Second, it designed new experiments to find stimulus dimensions that could drive two brain regions differently, even when those regions otherwise had similar response profiles. Third, it searched for experiments that could tell competing models of the same brain region apart. Experiments discovered using just two models also exposed differences across a much broader set of models, suggesting that Braintree had found more general dimensions along which brain models disagree. Braintree can augment the capabilities of a vision neuroscientist as a verifier for existing hypotheses, a test-bed for new ideas, and/or a tool for generating stimuli that can discriminate between competing hypotheses or models of the brain. Braintree shows how AI agents and brain models can work together to make in-silico vision neuroscience experiment computable, allowing vision scientists to explore many more experiment conditions and identify the most informative ones to test on the human brain.

Primary area:
applications to neuroscience & cognitive science
Keywords:
agentic systems, in-silico experiments, fMRI encoding models, computational neuroscience, autonomous research

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3AZzOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Braintree, a multi-agent framework for conducting in-silico experiments in vision neuroscience. The system connects literature grounding, experimental design, stimulus construction, evaluation on neural encoding models, and interpretation, using results to guide subsequent experiments. The demonstrations include recovering published effects, testing those effects with newly constructed stimuli, and finding manipulations that distinguish brain regions or competing encoding models. The paper also evaluates whether disagreements discovered using one model pair extend to ten additional backbones.

Critical strengths
  1. The framework addresses a practical research need. Connecting scientific questions to stimulus construction and model evaluation involves considerable manual work. Braintree brings these steps into an executable workflow and produces concrete experimental contrasts that could inform subsequent neuroscience studies.

  2. Several design choices support experimental reliability. The framework separates stimulus design from auditing, records planned comparisons before execution, and uses fixed code for quantitative analysis. These are sensible safeguards when language models are involved in designing and interpreting experiments.

  3. The demonstrations cover several meaningful use cases. The evaluation includes both reproduction with original stimuli and testing with new stimuli, alongside region and model dissociation. The additional analysis on ten backbones is particularly useful, as it provides evidence that the discovered disagreements extend beyond the model pair used during search.

Critical weaknesses
  1. What does the multi-agent structure and iterative feedback contribute beyond simpler alternatives? The experiments show that the complete system can produce useful results, but there is no controlled comparison that isolates the benefit of its main components. Could the authors compare Braintree with a single agent using the same tools and a non-adaptive set of proposed experiments under matched budgets? Ablations of feedback and auditing would also help establish whether these components improve experimental validity, discovery yield, or efficiency.

  2. What exactly counts as a successful replication, and how should the reported 23/36 success rate be interpreted? This outcome depends on correctly reconstructing the original experiment, generating appropriate stimuli, and the encoding model capturing the relevant effect. A failed test cannot readily distinguish among these factors. Could the authors provide explicit success criteria and results for all tested findings, including those involving small or absent effects? Independent expert assessment of the reconstructed experiments and an analysis of failure cases would make this evaluation considerably more informative.

  3. How were discovery and confirmation separated in the reported searches? Appendix A.1 states that evaluation uses separate model configurations or held-out stimuli, but the manuscript does not clearly specify how this was implemented for each experiment. Could the authors report the relevant splits, stimulus counts, stopping budgets, and which results come from independent confirmation after the manipulation and its direction were fixed? Chronological search records and a clear account of multiple-testing control would help readers assess the reliability of results obtained through adaptive experiment selection.

Questions for authors

My main questions are included in the three concerns under Critical Weaknesses. I would be open to raising my score if the authors can satisfactorily address most of these concerns with clearer methodological details and additional evidence.

4 comments
AuthorsOpenReview ↗

Response 1 to Reviewer to 3AZz

We thank Reviewer 3AZz for their review and for providing very actionable feedback on the paper. We’re also grateful for their openness to reconsider their score. We have made substantial changes to the paper now to address all of the concerns raised. We summarize these changes and address each of the reviewer's concern in turn below.

Changes related to W1. Braintree against a single-agent baseline and against feedback and auditor ablations. This is a very important suggestion! We ran all 3 requested comparisons. (1) single-agent baseline, (2) non-adaptive (fixed-plan) ablation, and (3) an auditor ablation.The results are now included the paper as two new sections, “Braintree outperforms single-agent baselines” and “Both experimental feedback and auditor are critical for Braintree performance”, with three new tables (Tables 1-3).

The experiment setup. We tested the models across both brain- (FFA vs OFA) and model-dissociation (RN50 vs ViT-B/32) experiments. Each system received the exact same prompt, had access to the same stimulus generators, identical brain encoding models, and were given the same budget of 50 experiments/run. We also repeated each experiment 3 times to see the variability across runs, efficiency, and yield (number of successful dissociations). There was no human intervention. We counted a dissociation only if it was significant and generalized to new NSD subjects and brain models.

Single agent vs Braintree. We tested 4 general-purpose coding agents (Claude Fable 5.1, Opus 5.5, Sonnet 5.5, and Codex GPT-5.6). Braintree found 12.7 generalizing region dissociations per run. The best single agent (Fable 5.1) found 2.7. On the model task, Braintree found 10.3 and the best single agent found 5.7. Braintree used the same or fewer tokens per experiment than three of the four baselines (58k vs 58k–335k on the region task). Braintree did take longer per experiment (9.9 min vs 2.5-5.4 min on the region task).

Feedback ablation. The fixed-plan variant committed to 50 experiments in advance without ever seeing the outcome. When those experiments were run, the system found only 1.3 generalizing region dissociations per run, compared with 12.7 for the baseline Braintree. On the model task it found 8 successful dissociations, compared with 10.3 for the baseline Braintree. The fixed-plan variant used fewer tokens per experiment (was more efficient). We now report this tradeoff and the detailed table in the paper.

Auditor ablation. In this experiment we found that removing the auditor did not reduce the number of dissociations (13.7 vs 12.7 for regions; 11.5 vs 10.3 for models). This was initially puzzling to us. But then when we inspected the images post-hoc we found that many of the stimulus sets were haphazard and did not in fact correctly implement the manipulation needed to test the hypothesis. When corrected for these errors, the number of valid dissociations fell to 9 for regions and 6.5 for models (much fewer than the baseline Braintree). We have also added a Figure which shows the examples of the failed stimulus sets. This experiment shows that auditor is very important to make sure the experiments from Braintree are in fact valid hypotheses that should be taken seriously.

To summarize, Braintree substantially outperforms general-purpose single-agent baselines (including those using higher-tier models. Ablation tests further show that both feedback and auditing contribute to the efficiency and reliability of the results obtained from Braintree.

AuthorsOpenReview ↗

Response 2 to Reviewer to 3AZz

Changes related to W2. Replication logic, criteria, and interpretation Thanks for this question. We realize that the original submission did not explain (1) the rationale of this section and the (2) success criterion, and (3) the interpretation clearly enough. We’ve revised the paper on all 3 counts.

Rationale: We paste our new updated text below which explains this section better

“Evaluation Logic: We evaluate Braintree in a staged manner so that we can progressively evaluate its capabilities. We begin with a simple question: Can in-silico models of the brain reproduce findings from vision neuroscience? In this first experiment (paper reproduction), Braintree has access to the original stimuli and experimental conditions from published studies. It only tests whether the encoding models recover the reported effects. This experiment does not require scientific reasoning or new stimulus design. The purpose of this experiment is to establish a baseline reproduction success rate for Braintree because the ground truth is known.

In the next stage (paper replication), we remove the original stimuli and ask Braintree to design new images to test the same scientific claims. This experiment tests Braintree’s ability to design new stimuli, run the experiment, and evaluate the outcomes. We first consider findings that the encoding models successfully reproduced in the first study. This way, if an experiment fails, we can better isolate the issue to the new experimental design rather than the encoding model’s inability to capture the original effect (although we do assume that the original findings generalize to new stimuli). We then expand to others for which no reproduction step was run. Failures in the broader set are harder to attribute but we report both success rate and directional effects.”

2. The replication success criterion. Based on the reviewer's feedback we now include an explicit (Main text) Method section that clearly explains the criterion. We paste the text below:

Replication criterion: Published findings typically compare two conditions (A and B) with a predicted outcome (A > B, A < B, or no difference). Braintree evaluates these findings using CLIP-RN50-based encoding models trained on all NSD participants. For findings that predict a larger response to one condition than another (e.g., A > B), we consider the result replicated if the difference (1) is in the predicted direction (A > B rather than B > A) and (2) is statistically significant (one-sided t-test across images, p < 0.05). For findings that predict no difference between conditions, replication requires the difference to fall within predefined equivalence bounds (two one-sided tests, p < 0.05). All other outcomes are considered failed replications.

3. The interpretation: We now report the findings in 2 groups. In the first group (10 findings), the original stimuli were available. So we could confirm that the models could reproduce the published effects. Braintree then designed new stimuli to test the same findings and replicated 8/10. Because models could reproduce the original effects, the interpretation here is that the two failures can be considered limitations of Braintree’s ability to design stimuli based on previously published papers.

In the second group (all 36 findings). Braintree reconstructed experiments autonomously. We used this experiment to demonstrate Braintree’s ability to go from a paper to result. Braintree read each study, reconstructed the experiment design, generated stimuli from scratch, and tested the published effect on new images. Here we reported that Braintree could autonomously replicate 23/36 findings.

Based on the reviewer’s feedback, we went back and inspected the failures closely. Of the 13 failures that did not meet our pre-defined criterion, 11 showed an effect in the right direction (but did not reach our statistical significance criterion). This proportion is significantly higher than expected by chance (11/13, two-tailed binomial test against 50% random chance, p = 0.022). Only 2 had effects in the opposite directions (but also not significant). We agree with the reviewer that failures in this setting are harder to interpret. They could be because of the limits of the encoding model, the reconstructed images. Also note that effect sizes are generally lower for replications than when reproducing from original stimuli. We nonetheless counted them all failures and kept the reported rate at 23/36 because our criterion was fixed before the experiment. Relaxing it after seeing the results did not seem right. As requested by the reviewer, we also include a finding-by-finding table that reports the effect direction, observed effect size, statistical significance, and replication outcome for all 36 findings (including failures).

We hope these comprehensive changes and clarification address all of the concerns raised by the reviewer related to W2.

AuthorsOpenReview ↗

Response 3 to Reviewer to 3AZz

Changes related to W3. Discovery and confirmation, stimulus runs, variabiiity etc.

Thanks for this question. We made every effort to separate the discovery from confirmation, Here are some of the salient points.

Discovery and confirmation were independent. The results reported in the paper came from a single search run. However, discovery and confirmation were separated for every run. In the discovery phase, Braintree only had access to encoding models trained on NSD subjects 1 and 2. It iteratively designed experiments, generated stimuli, and evaluated predicted responses using only these models. Once Braintree identified a manipulation, the experimental conditions and predicted directions were fixed before confirmation. The manipulation was then tested on held-out models trained on NSD subjects 5 and 7. For brain-region dissociations, we additionally also tested generalization on a different encoder backbone (ImageNet-trained ResNet-50). Importantly, Braintree never accesses these held-out models or their results during discovery. Braintree maintains a chronological record of the all experiment specifications and runs. We now have an additional section in the Methods that clearly explains this for our readers.

Stimulus counts. Each experiment compared two conditions with 40 images per condition (80 images total). In all baseline and ablation comparisons, every system receives the same budget of 50 in-silico experiments per run. Although Braintree supports resuming a search from its saved state, this functionality does not change the budgets used in the reported comparisons.

Search budget, multiple run variability: This is a fantastic suggestion. (To address W1), we ran many additional experiments to compare Braintree with baseline (general-purpose and single-agent) models. We also tested the effect of auditor and feedback ablations. We test the effects clearly under a fixed budget of 50 in-silico experiments per run. We also repeated each experiment multiple times to assess the variability and reproducibility of the results. All these results are now included in the paper within additional sections.

We hope that these revisions comprehensively address all of Reviewer 3AZz's concerns. The paper is now considerably strengthened by the new evidence and updates. We sincerely thank the Reviewer 3AZz for their constructive feedback. We hope to engage further and welcome any further suggestions.

AuthorsOpenReview ↗

Summary of changes: Reviewer 3AZz

Reviewer 3AZz: We want to share a short summary of changes we have made to the paper based on your feedback.

Changes based on W1

1. Multi-agent versus single agent baselines: These comparisons are now included in the Results (including higher-tier single-agent baseline models, Section 3.4). All systems were given access to the same tools, brain models, and had the same experiment budget. Braintree consistently outperformed these baselines and discovered more significant dissociations that generalized to held-out subjects and models.

2. Role of iterative feedback: Braintree comparisons with non-adaptive baseline (under shared budget, as suggested by the reviewer) are now included in the Results section (Section 3.4). These results show the importance of experiment feedback to guide new experiments (rather than executing a fixed set of experiments).

3. Auditor ablations: We’ve conducted these analyses and they’re now included in the main Results section (also Section 3.4). We also compared discovery yield and efficiency as suggested by the reviewer. These analyses show that auditing is in fact very important for Braintree's reliability (we found a number of false positives otherwise).

Changes related to W2

4. Replication criterion: Now explicitly stated in the Methods section and highlighted in figures. his makes it easier to distinguish statistically significant replications from experiments that recover effects in the predicted direction but do not meet our significance threshold.

5. Analysis of failures: We have now analyzed the failed replications individually, including whether the observed effects were in the predicted direction (see updated Figure 2). We found that several experiments recovered the expected direction of the effect despite not meeting our (stringent) statistical criterion. These analyses are now included in the results text (Section 3.1), Figure 2, and Appendix. We have also clarified the interpretation of failures in the text.

6. Staged comparisons: Are failures because of the encoder or the designer? We now explain how expt reproduction v replication addresses this question and how one can diagnose issues much better. We have updated Figure 1 to show the staged evaluation process. We have explained the rationale for these tests better as well.

Changes related to W3

7. Discovery v confirmation: We have moved the relevant methodological details into the main text and clarified how experiments are discovered and subsequently evaluated on held-out subjects and encoding models.

8. Details: Experiment counts, splits, stopping budget etc. have now been highlighted clearly. We have also changed the figures to show the summary of all 50 experiments that were run. We are also including run records and multiple testing variability analyses (standard errors in Section 3.4) for the readers.

We’ve now uploaded the current version of the revised paper (changes highlighted in red). We’re sharing this (in progress) draft so that we can get early feedback. Please let us know if there any outstanding issues for us to address. We will continue to clean up and organize the Appendix over the next few days.

We thank the Reviewer again for their constructive feedback. We’ve updated the paper with extensive changes to address all the concerns raised by the reviewer. We feel that the paper is significantly stronger and better based on the feedback and that all of the reviewer’s concerns have been comprehensively addressed. We respectfully request them to reconsider their scores in light of all the new data and evidence supporting the central claims of the paper and novelty of this approach in vision neuroscience.

Reviewer_ytjdOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Braintree, a multi-agent system that runs vision neuroscience experiments on image-computable encoding models of human visual cortex. An orchestrator coordinates planner, designer, experimenter, and analyzer agents, with auditors checking each stage; all agents are separate Claude instances. The encoding models are voxelwise linear readouts from standard embedding models such as CLIP-RN50 or DINOv2, fit to the Natural Scenes Dataset. Given a paper or a hypothesis, Braintree builds a stimulus set from retrieved, edited, or generated images, runs it through the encoding model, and applies fixed statistical code to the predicted responses.

The authors present three demonstrations: reproduction of 10 published findings with the original stimuli (10/10 successful); replication with newly generated stimuli (8/10, and 23/36 on a larger set); and a search for stimulus manipulations that dissociate brain areas or reveal disagreement between models. In my reading, the main novelty of the paper is the agentic system, but the system is neither compared against baselines nor validated through ablations, and the underlying approach of using encoding models as in silico stand-ins for the brain is established. The neuroscience findings are predictions of encoding models that are not tested on brain data. In its current form, I do not think the paper is a sufficient contribution for ICLR. It would fit a neuroscience venue as an application paper, or it could become an ML paper if the focus moved to validating the agentic design with baselines and ablations.

The validation consists of replication rates for published findings. There are no baselines, ablations, or measures of run-to-run variability, runtime and cost are reported only as “a single day”, and the 13 failed replications are not analyzed.

Critical strengths

Clear and transparent evaluation: The work is technically sound and the paper is clearly written. The 36-study replication test is a reasonable use of in silico models, and the paper reports failures alongside successes.

Useful model-dissociation experiment: The model-dissociation experiment follows the controversial stimuli approach of Golan et al. (2020), and the check on ten additional backbones adds robustness. This shows a concrete use case in which automated stimulus search could support model comparison. In the current manuscript, however, neither strength outweighs the weaknesses below.

Critical weaknesses

Unclear novelty and no baselines: Agentic systems for science already exist, and several are cited in the submission (Lu et al., 2024; Gottweis et al., 2025; Ghareeb et al., 2026). Agentic systems built specifically for neuroscience also exist (BrainPilot, SeekBrain). None of them is compared against. Braintree is a set of Claude Code subagents combined with an encoding model, image datasets, and tools for image editing and generation. I expect that a current general-purpose agent such as Claude Code or Codex, given the same tools and instructions, would perform about as well, and the paper does not test this. There is no baseline at all, not even a single-agent version of the same pipeline. The underlying method is also established: using encoding models as stand-ins for the brain to screen stimuli or compare models has been done before (Ratan Murty et al., 2021; Gifford et al., 2025b; Golan et al., 2020). What the agent adds is automating the choice of stimuli around that method. This is consequential because the agentic system is the paper’s main contribution, and without baselines it remains unclear whether its design adds anything over a simpler system.

Neither the agentic system nor its findings are validated: The prompts, agent instructions, tool definitions, skills, and audit criteria are not provided, and the code is promised but not available. There are no ablations, such as removing the auditors or changing the model tier for a role, and no report of run-to-run variability, although the agents are stochastic. The 13 failed replications are not analyzed, so it is unclear whether the encoding model or the designer’s stimulus set was at fault. On the neuroscience side, the findings are predictions of encoding models, whose generalization is limited, and none of the discovered dissociations is tested on brain data. This is consequential because a reader cannot tell which parts of the system’s design matter, whether it is reproducible, or whether its discoveries hold in the brain. I would not yet call the discovered dissociations results.

Questions for authors

How does Braintree compare to a plain Claude Code session (one agent, same tools, same prompt), and to an existing framework such as AI Scientist or BrainPilot, on the same 36 replication tasks?

Can the authors provide the prompts and skill definitions for each agent, and the pass/fail criteria used by the auditors?

How many times was each study run, and how often did the outcome (replicated or not) change across runs?

What caused each of the 13 failed replications? How often did an auditor reject a stimulus set, and what happened next?

What is the wall-clock time and token cost per study?

Have any of the discovered region or model dissociations been tested on held-out NSD data or on new fMRI data?

1 comment
AuthorsOpenReview ↗

Response 1 to Reviewer ytjd: Clarifications, planned experiments, and invitation to engage

We'd like to thank Reviewer ytjd for their review. We're reaching out within hours into the discussion period to engage early. We're particularly encouraged by the reviewer’s initial assessment that concerns about the paper soundness are minor and fixable. We are confident that we can address the Reviewer’s concerns thoroughly.

We want to first respectfully disagree with the assessment of Braintree’s novelty relative to existing domain-general approaches (like SeekBrain or BrainPilot). There are several differences in (1) the scope and (2) the capabilities, of these systems.

Braintree is not a general-purpose agentic system for neuroscience applications. It is a framework designed specifically for vision neuroscience. And because it is domain specific, it has specialized tools and harnesses that integrate hypothesis testing, stimulus generation, algorithmic editing, experiment execution (on NeuroAI models), and scientific inference in an interactive and close-loop.

This is a substantive distinction. Neither of these (concurrent.. these papers are not published and are under-review) papers show this integrated workflow on in-silico NeuroAI models. SeekBrain covers multimodal analyses of already recorded neuroscience data. Brainpilot shows workflow with mouse physiology and human fMRI (connectivity) analyses and EEG model development. These studies are valuable contributions. But they are not equivalent to Braintree. They cannot natively run any of the experiments we show in the paper or make inferences from them. We hope the reviewer agrees with this assessment. One cannot do the same science or run any of the experiments that Braintree enables on these other platforms.

But we agree that a careful comparison is valuable. We will add a systematic capability-level comparison that will make the distinctions explicit.

Many of the other concerns can be addressed with new analyses and clearer reporting. For example, we already have precise measurements of run-to-run walltimes, costs and other details which we will provide.

Over the next few days, we will focus our revisions on three fronts:

1. Comparison with existing systems: We will precisely document which capabilities existing neuroscience agent systems have with respect to Braintree. We will also show that there is in fact no existing agentic system that can reason, audit, generate stimuli, run the experiment and reason in a loop in a domain-specific application to vision neuroscience.

2. Controlled comparisons and ablations: We will evaluate the performance of a general-purpose coding-agent baseline and conduct ablations in two ways. First we will perform agentic ablations so that their overall necessity. We will also perform directional prompting (as additional tests).

3. Additional validation tests: We share the reviewer’s concerns about the importance of validations. We had included several validations in our paper already which we will first highlight. Based on the reviewer’s feedback we will also evaluate new, held-out fMRI data and add further tests on new models.

We’re excited about Braintree because it is a truly new approach that integrates NeuroAI models and stimulus generation and reasoning (see also Reviewer Cpxb’s assessment on novelty). Braintree has the potential to meaningfully accelerate neuroscience discovery in vision cognitive neuroscience.

We are starting this discussion early so that the Reviewer can engage with us early even as we run these experiments and make changes to the paper. We request reviewer ytjd to let us know if we should change our plan in any specific way, or if there are any specific controls or comparisons that would be informative. We sincerely hope to resolve the substantive scientific or workflow disagreements and reach a fair assessment of Braintree’s contributions.

Reviewer_CpxbOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present an agentic framework for vision science research that enables open-ended exploration, replication of existing studies, and the design of new experiments with the help of agents. The framework is autonomous and has many implications, especially in the NeuroAI field.

Overall, this is a strong and novel use of agentic AI to advance science at a faster pace.

However, there are some questions that weaken the claims in the paper. They are mentioned below in the weaknesses section.

Critical strengths
  1. The paper addresses an important and timely need. Tools that automate hypothesis testing could substantially speed up scientific discovery.
  2. The three experiments are interesting and make a promising proof of concept for the framework.
  3. The authors have tested this framework with various experiments, each showing the true potential of this framework.
Critical weaknesses
  1. Novelty relative to prior agentic frameworks. The paper needs to position itself more clearly against existing multi-agent and tool-augmented LLM frameworks, such as CAMEL (Li et al., 2023) and Toolformer (Schick et al., 2023). What are the key contributions that set this framework apart from these approaches? A direct comparison, either conceptual (e.g., a summary table) or empirical, would strengthen the novelty claims.

  2. Role of human oversight. It is unclear where, if anywhere, humans take part in the loop. Could the authors say which stages are fully automated and which need human input, such as experimental design, checking outputs, or interpreting results? A discussion of whether human oversight is needed for reliable results would help readers judge how the framework could be used in practice.

  3. Scope of hypothesis testing. The three experiments are a good starting point, but the paper would benefit from a discussion of the framework’s broader scope. What kinds of hypotheses can it test, and where are its limits? For example, how does it handle more complex hypotheses, or ones that require new stimuli or new data collection?

  4. Restriction to vision science. Why do the authors present the framework as limited to vision science experiments? Is this limitation inherent to the design, or does it only reflect the choice of evaluation? If the framework is domain-agnostic, a discussion or small demonstration of how it carries over to other fields would broaden the paper’s impact.

  5. Hypothesis specification and potential bias. Is the hypothesis given to the agent in advance? If so, how do the authors ensure that the hypothesis prompt does not leak information that biases the agent toward a particular outcome? Because prompt design matters so much here, the full hypothesis prompts should be included in the paper. An ablation on prompt phrasing (e.g., neutral versus directional wording) would also be informative. Relatedly, can the agent generate its own hypotheses, and if so, how do they compare with those supplied by the authors?

  6. Failure modes. The paper would benefit from a systematic analysis of failure modes. Many agentic frameworks have been shown to work, but knowing where and why they break down matters just as much for designing more robust experiments and prompts. Concrete examples of failures, along with how often they occur and what causes them, would be valuable to the community.

  7. Criteria for selecting experiments. Apart from the availability of stimuli, what criteria guided the choice of the three experiments?

Questions for authors

Please see the weaknesses section.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Summary of changes: Reviewer ytjd P1

Reviewer yjtd: We want to share new updates on the changes we have made to the paper based on your feedback. We summarize these changes below.

1. Novelty relative to existing frameworks: We now motivate the paper better in the Intro and have a new Appendix. We want to emphasize again that Braintree is not a general-purpose agent applied to neuroscience. It is a domain-specific framework ofr in-silico vision neuroscience. It integrates scientific reasoning, experiment design, stimulus generation, execution on predictive brain models, and statistical inference in a closed experimental loop.

2. Comparison against baselines: These comparisons are now included in the Results. We also included comparison higher-tier general-purpose coding agents (with access to the same tools, brain models, and experiment budgets). We find that Braintree outperforms all baselines. It finds more significant dissociations that generalize to held-out subjects and models. Braintree is also more efficient (uses fewer tokens).

We want to add that we agree with the reviewer that using encoding models as stand-ins for the brain has precedent in neuroscience. Our work would not exist without these foundational studies (Ratan Murty et al., 2021; Gifford et al., 2025b; Golan et al., 2020) and we are very familiar with this line of work. These baseline comparisons were indeed crucial for showing clearly what Braintree adds beyond these early approaches.

We thank the reviewer for pushing us to conduct these experiments. We hope that the new results and revisions address the reviewer’s concerns and clarify the novelty and contributions of Braintree and how none of the existing agentic frameworks in neuroscience tackle this problem.

3. Validation of the agentic system. We have now performed controlled ablations of the auditing and experimental feedback components of Braintree (Section 3.4). We find that both are important and contribute to Braintree’s abilities.

4. Full prompts, agent instructions, tool and skill definitions, and audit criteria. We are providing the full prompts in the Appendix. All other files will be uploaded as a zip supplemental file. With Braintree, we want to make in-silico experiments accessible to the broader vision neuroscience community. We commit to making the code open-access and as widely available as possible.

5. Run-to-run variability: We now show the variability explicitly by reporting them in Section 3.4 (where we compare models against general-purpose agents and perform ablation tests). Readers can now see and assess the variability between runs.

6. Analysis of failed experiments: We have now analyzed the failed replications individually and clarified what these failures can and cannot tell us. Our statistical criterion for successful replication is quite stringent, and we found that many experiments in fact show the published effects in the predicted direction even when they did not reach statistical significance. We have made changes to Figure 2, described the stage-wise evaluation in Figure 1, explained the results better in Section 3.1 and included an Appendix with the failure cases. We also clarify that, when the original stimuli are unavailable, we cannot determine whether a failure arises from the encoding model or the experiment designer (Section 3.2). We hope you find these changes to be a more complete and appropriately nuanced interpretation of the replication results.

7. Validation in human data: We want to emphasize that Braintree is a framework for in-silico neuroscience. We feel we have been appropriately careful to highlight this fact (including the title). We have many internal conditions to guard against overfitting. We require that dissociations generalize to models from held-out subjects and to new encoding models that were not used during the experiment search. We have further highlighted and clarified these validation criteria in the manuscript. We do not think that collecting new human brain data is necessary to establish the central contribution of this paper about automated hypothesis design, stimulus design, and in-silico tests and analysis. Particularly, the model-dissociation experiments already give us stimuli for which models make different predictions. So this experiment is almost guaranteed to ‘succeed’.

We of course agree that these in-silico tests need to be validated against actual data from the human brain. We further emphasize this in the Discussion. We do not claim that the discovered dissociations are ‘discoveries’. Our claim is that Braintree can reason and generate concrete, testable (and falsifiable) hypotheses and the stimuli needed to evaluate them experimentally.

AuthorsOpenReview ↗

Summary of changes: Reviewer ytjd P2

We’ve now uploaded a draft version of the paper (changes highlighted in red). We’re sharing this (in progress) draft so that we can get early feedback. So please let us know if there are outstanding issues that we have not addressed or clarified. We will continue to clean up and organize the Appendix over the next few days.

We thank Reviewer ytjd again for their constructive feedback and for engaging with us. We've reinforced the paper with many new experiments and analyses now. We feel that the paper is significantly stronger and better based on the feedback and that all of the reviewer’s concerns have been comprehensively addressed. We respectfully request them to reconsider their scores in light of all the new data and evidence supporting the central claims of the paper and novelty of this approach in vision neuroscience.

AuthorsOpenReview ↗

Response to Reviewer ytjd

We thank the Reviewer ytjd for recognizing the improvements to the paper and for raising their score.

Based on their response it looks like the only remain question/concern regards the scope of the paper? We offer 3 considerations and have one question for the reviewer.

First, please consider that the scope of the paper is in-silico vision (cognitive computational) neuroscience (even though see the push-back from reviewer 1 who has encouraged us to scope it more broadly). Our claim is not that Braintree is an agentic system for computational neuroscience. We have been very careful in defining this scope.

Within this domain (vision), Braintree is also not restricted only to the questions demonstrated in the paper. It can investigate selectivity, discover invariances, characterize response properties of previously unexplored brain regions, identify dissociations between regions, and design experiments that distinguish competing computational models. These are different scientific questions, all of which can be addressed within the current framework. We also want to clarify that Braintree is not restricted to binary or categorical neural responses (though those are easiest to interpret here). It can already run on continuous neural response predictions, and experiments can investigate a wide range of response properties and statistical relationships. We only used the binary outcomes is shows to clearly test if the statistical criteria to test a hypothesis is supported or not. This is not a limitation on the scientific questions the system can investigate.

Second, please consider the distinction between the capabilities of a framework and the number of applications feasible to demonstrate in a paper. It is not practical to exhaustively demonstrate every scientific question that Braintree can enable within a 10-page paper. We chose representative problems based on well-studied regions that allow us to test its scientific reasoning, experimental design, and discovery capabilities, including generalization to held-out subjects and computational models. We can already run models for monkey neuron data, human single-neuron data, and also using whole-brain fMRI recordings. We’re happy to run/show you a few extra capabilities. Please let us know if you have anything specific in mind!

Third, please consider the baseline here (core consideration for submissions to ICLR). There is currently no system in cognitive neuroscience that can fully execute this full workflow from hypothesis generation, stimulus design, tests on in-silico model, data analysis in a loop. The appropriate comparison is against what existing systems can accomplish (they're slow and human-led), and not against a hypothetical system capable of addressing every problem in computational neuroscience.

One question for the reviewer: Is there a specific class of scientific question within in-silico vision neuroscience that, if we can demonstrate, would address your remaining concern about scope/abilities of the paper?

We would be happy to attempt the experiment on Braintree in the remainder of the discussion period and provide new data and evidence. And if we cannot, we would be happy to acknowledge and discuss that limitation explicitly.

Thanks again for your time and engagement. Your feedback has indeed made the paper stronger. We really hope we can have one more round of discussion to clarify the scope before the AC discussions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.