- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Braintree, a multi-agent framework for conducting in-silico experiments in vision neuroscience. The system connects literature grounding, experimental design, stimulus construction, evaluation on neural encoding models, and interpretation, using results to guide subsequent experiments. The demonstrations include recovering published effects, testing those effects with newly constructed stimuli, and finding manipulations that distinguish brain regions or competing encoding models. The paper also evaluates whether disagreements discovered using one model pair extend to ten additional backbones.
Critical strengths
-
The framework addresses a practical research need. Connecting scientific questions to stimulus construction and model evaluation involves considerable manual work. Braintree brings these steps into an executable workflow and produces concrete experimental contrasts that could inform subsequent neuroscience studies.
-
Several design choices support experimental reliability. The framework separates stimulus design from auditing, records planned comparisons before execution, and uses fixed code for quantitative analysis. These are sensible safeguards when language models are involved in designing and interpreting experiments.
-
The demonstrations cover several meaningful use cases. The evaluation includes both reproduction with original stimuli and testing with new stimuli, alongside region and model dissociation. The additional analysis on ten backbones is particularly useful, as it provides evidence that the discovered disagreements extend beyond the model pair used during search.
Critical weaknesses
-
What does the multi-agent structure and iterative feedback contribute beyond simpler alternatives? The experiments show that the complete system can produce useful results, but there is no controlled comparison that isolates the benefit of its main components. Could the authors compare Braintree with a single agent using the same tools and a non-adaptive set of proposed experiments under matched budgets? Ablations of feedback and auditing would also help establish whether these components improve experimental validity, discovery yield, or efficiency.
-
What exactly counts as a successful replication, and how should the reported 23/36 success rate be interpreted? This outcome depends on correctly reconstructing the original experiment, generating appropriate stimuli, and the encoding model capturing the relevant effect. A failed test cannot readily distinguish among these factors. Could the authors provide explicit success criteria and results for all tested findings, including those involving small or absent effects? Independent expert assessment of the reconstructed experiments and an analysis of failure cases would make this evaluation considerably more informative.
-
How were discovery and confirmation separated in the reported searches? Appendix A.1 states that evaluation uses separate model configurations or held-out stimuli, but the manuscript does not clearly specify how this was implemented for each experiment. Could the authors report the relevant splits, stimulus counts, stopping budgets, and which results come from independent confirmation after the manipulation and its direction were fixed? Chronological search records and a clear account of multiple-testing control would help readers assess the reliability of results obtained through adaptive experiment selection.
Questions for authors
My main questions are included in the three concerns under Critical Weaknesses. I would be open to raising my score if the authors can satisfactorily address most of these concerns with clearer methodological details and additional evidence.
4 comments
Response 2 to Reviewer to 3AZz
Changes related to W2. Replication logic, criteria, and interpretation Thanks for this question. We realize that the original submission did not explain (1) the rationale of this section and the (2) success criterion, and (3) the interpretation clearly enough. We’ve revised the paper on all 3 counts.
Rationale: We paste our new updated text below which explains this section better
“Evaluation Logic: We evaluate Braintree in a staged manner so that we can progressively evaluate its capabilities. We begin with a simple question: Can in-silico models of the brain reproduce findings from vision neuroscience? In this first experiment (paper reproduction), Braintree has access to the original stimuli and experimental conditions from published studies. It only tests whether the encoding models recover the reported effects. This experiment does not require scientific reasoning or new stimulus design. The purpose of this experiment is to establish a baseline reproduction success rate for Braintree because the ground truth is known.
In the next stage (paper replication), we remove the original stimuli and ask Braintree to design new images to test the same scientific claims. This experiment tests Braintree’s ability to design new stimuli, run the experiment, and evaluate the outcomes. We first consider findings that the encoding models successfully reproduced in the first study. This way, if an experiment fails, we can better isolate the issue to the new experimental design rather than the encoding model’s inability to capture the original effect (although we do assume that the original findings generalize to new stimuli). We then expand to others for which no reproduction step was run. Failures in the broader set are harder to attribute but we report both success rate and directional effects.”
2. The replication success criterion. Based on the reviewer's feedback we now include an explicit (Main text) Method section that clearly explains the criterion. We paste the text below:
Replication criterion: Published findings typically compare two conditions (A and B) with a predicted outcome (A > B, A < B, or no difference). Braintree evaluates these findings using CLIP-RN50-based encoding models trained on all NSD participants. For findings that predict a larger response to one condition than another (e.g., A > B), we consider the result replicated if the difference (1) is in the predicted direction (A > B rather than B > A) and (2) is statistically significant (one-sided t-test across images, p < 0.05). For findings that predict no difference between conditions, replication requires the difference to fall within predefined equivalence bounds (two one-sided tests, p < 0.05). All other outcomes are considered failed replications.
3. The interpretation: We now report the findings in 2 groups. In the first group (10 findings), the original stimuli were available. So we could confirm that the models could reproduce the published effects. Braintree then designed new stimuli to test the same findings and replicated 8/10. Because models could reproduce the original effects, the interpretation here is that the two failures can be considered limitations of Braintree’s ability to design stimuli based on previously published papers.
In the second group (all 36 findings). Braintree reconstructed experiments autonomously. We used this experiment to demonstrate Braintree’s ability to go from a paper to result. Braintree read each study, reconstructed the experiment design, generated stimuli from scratch, and tested the published effect on new images. Here we reported that Braintree could autonomously replicate 23/36 findings.
Based on the reviewer’s feedback, we went back and inspected the failures closely. Of the 13 failures that did not meet our pre-defined criterion, 11 showed an effect in the right direction (but did not reach our statistical significance criterion). This proportion is significantly higher than expected by chance (11/13, two-tailed binomial test against 50% random chance, p = 0.022). Only 2 had effects in the opposite directions (but also not significant). We agree with the reviewer that failures in this setting are harder to interpret. They could be because of the limits of the encoding model, the reconstructed images. Also note that effect sizes are generally lower for replications than when reproducing from original stimuli. We nonetheless counted them all failures and kept the reported rate at 23/36 because our criterion was fixed before the experiment. Relaxing it after seeing the results did not seem right. As requested by the reviewer, we also include a finding-by-finding table that reports the effect direction, observed effect size, statistical significance, and replication outcome for all 36 findings (including failures).
We hope these comprehensive changes and clarification address all of the concerns raised by the reviewer related to W2.
Response 3 to Reviewer to 3AZz
Changes related to W3. Discovery and confirmation, stimulus runs, variabiiity etc.
Thanks for this question. We made every effort to separate the discovery from confirmation, Here are some of the salient points.
Discovery and confirmation were independent. The results reported in the paper came from a single search run. However, discovery and confirmation were separated for every run. In the discovery phase, Braintree only had access to encoding models trained on NSD subjects 1 and 2. It iteratively designed experiments, generated stimuli, and evaluated predicted responses using only these models. Once Braintree identified a manipulation, the experimental conditions and predicted directions were fixed before confirmation. The manipulation was then tested on held-out models trained on NSD subjects 5 and 7. For brain-region dissociations, we additionally also tested generalization on a different encoder backbone (ImageNet-trained ResNet-50). Importantly, Braintree never accesses these held-out models or their results during discovery. Braintree maintains a chronological record of the all experiment specifications and runs. We now have an additional section in the Methods that clearly explains this for our readers.
Stimulus counts. Each experiment compared two conditions with 40 images per condition (80 images total). In all baseline and ablation comparisons, every system receives the same budget of 50 in-silico experiments per run. Although Braintree supports resuming a search from its saved state, this functionality does not change the budgets used in the reported comparisons.
Search budget, multiple run variability: This is a fantastic suggestion. (To address W1), we ran many additional experiments to compare Braintree with baseline (general-purpose and single-agent) models. We also tested the effect of auditor and feedback ablations. We test the effects clearly under a fixed budget of 50 in-silico experiments per run. We also repeated each experiment multiple times to assess the variability and reproducibility of the results. All these results are now included in the paper within additional sections.
We hope that these revisions comprehensively address all of Reviewer 3AZz's concerns. The paper is now considerably strengthened by the new evidence and updates. We sincerely thank the Reviewer 3AZz for their constructive feedback. We hope to engage further and welcome any further suggestions.
Summary of changes: Reviewer 3AZz
Reviewer 3AZz: We want to share a short summary of changes we have made to the paper based on your feedback.
Changes based on W1
1. Multi-agent versus single agent baselines: These comparisons are now included in the Results (including higher-tier single-agent baseline models, Section 3.4). All systems were given access to the same tools, brain models, and had the same experiment budget. Braintree consistently outperformed these baselines and discovered more significant dissociations that generalized to held-out subjects and models.
2. Role of iterative feedback: Braintree comparisons with non-adaptive baseline (under shared budget, as suggested by the reviewer) are now included in the Results section (Section 3.4). These results show the importance of experiment feedback to guide new experiments (rather than executing a fixed set of experiments).
3. Auditor ablations: We’ve conducted these analyses and they’re now included in the main Results section (also Section 3.4). We also compared discovery yield and efficiency as suggested by the reviewer. These analyses show that auditing is in fact very important for Braintree's reliability (we found a number of false positives otherwise).
Changes related to W2
4. Replication criterion: Now explicitly stated in the Methods section and highlighted in figures. his makes it easier to distinguish statistically significant replications from experiments that recover effects in the predicted direction but do not meet our significance threshold.
5. Analysis of failures: We have now analyzed the failed replications individually, including whether the observed effects were in the predicted direction (see updated Figure 2). We found that several experiments recovered the expected direction of the effect despite not meeting our (stringent) statistical criterion. These analyses are now included in the results text (Section 3.1), Figure 2, and Appendix. We have also clarified the interpretation of failures in the text.
6. Staged comparisons: Are failures because of the encoder or the designer? We now explain how expt reproduction v replication addresses this question and how one can diagnose issues much better. We have updated Figure 1 to show the staged evaluation process. We have explained the rationale for these tests better as well.
Changes related to W3
7. Discovery v confirmation: We have moved the relevant methodological details into the main text and clarified how experiments are discovered and subsequently evaluated on held-out subjects and encoding models.
8. Details: Experiment counts, splits, stopping budget etc. have now been highlighted clearly. We have also changed the figures to show the summary of all 50 experiments that were run. We are also including run records and multiple testing variability analyses (standard errors in Section 3.4) for the readers.
We’ve now uploaded the current version of the revised paper (changes highlighted in red). We’re sharing this (in progress) draft so that we can get early feedback. Please let us know if there any outstanding issues for us to address. We will continue to clean up and organize the Appendix over the next few days.
We thank the Reviewer again for their constructive feedback. We’ve updated the paper with extensive changes to address all the concerns raised by the reviewer. We feel that the paper is significantly stronger and better based on the feedback and that all of the reviewer’s concerns have been comprehensively addressed. We respectfully request them to reconsider their scores in light of all the new data and evidence supporting the central claims of the paper and novelty of this approach in vision neuroscience.
Response 1 to Reviewer to 3AZz
We thank Reviewer 3AZz for their review and for providing very actionable feedback on the paper. We’re also grateful for their openness to reconsider their score. We have made substantial changes to the paper now to address all of the concerns raised. We summarize these changes and address each of the reviewer's concern in turn below.
Changes related to W1. Braintree against a single-agent baseline and against feedback and auditor ablations. This is a very important suggestion! We ran all 3 requested comparisons. (1) single-agent baseline, (2) non-adaptive (fixed-plan) ablation, and (3) an auditor ablation.The results are now included the paper as two new sections, “Braintree outperforms single-agent baselines” and “Both experimental feedback and auditor are critical for Braintree performance”, with three new tables (Tables 1-3).
The experiment setup. We tested the models across both brain- (FFA vs OFA) and model-dissociation (RN50 vs ViT-B/32) experiments. Each system received the exact same prompt, had access to the same stimulus generators, identical brain encoding models, and were given the same budget of 50 experiments/run. We also repeated each experiment 3 times to see the variability across runs, efficiency, and yield (number of successful dissociations). There was no human intervention. We counted a dissociation only if it was significant and generalized to new NSD subjects and brain models.
Single agent vs Braintree. We tested 4 general-purpose coding agents (Claude Fable 5.1, Opus 5.5, Sonnet 5.5, and Codex GPT-5.6). Braintree found 12.7 generalizing region dissociations per run. The best single agent (Fable 5.1) found 2.7. On the model task, Braintree found 10.3 and the best single agent found 5.7. Braintree used the same or fewer tokens per experiment than three of the four baselines (58k vs 58k–335k on the region task). Braintree did take longer per experiment (9.9 min vs 2.5-5.4 min on the region task).
Feedback ablation. The fixed-plan variant committed to 50 experiments in advance without ever seeing the outcome. When those experiments were run, the system found only 1.3 generalizing region dissociations per run, compared with 12.7 for the baseline Braintree. On the model task it found 8 successful dissociations, compared with 10.3 for the baseline Braintree. The fixed-plan variant used fewer tokens per experiment (was more efficient). We now report this tradeoff and the detailed table in the paper.
Auditor ablation. In this experiment we found that removing the auditor did not reduce the number of dissociations (13.7 vs 12.7 for regions; 11.5 vs 10.3 for models). This was initially puzzling to us. But then when we inspected the images post-hoc we found that many of the stimulus sets were haphazard and did not in fact correctly implement the manipulation needed to test the hypothesis. When corrected for these errors, the number of valid dissociations fell to 9 for regions and 6.5 for models (much fewer than the baseline Braintree). We have also added a Figure which shows the examples of the failed stimulus sets. This experiment shows that auditor is very important to make sure the experiments from Braintree are in fact valid hypotheses that should be taken seriously.
To summarize, Braintree substantially outperforms general-purpose single-agent baselines (including those using higher-tier models. Ablation tests further show that both feedback and auditing contribute to the efficiency and reliability of the results obtained from Braintree.