acceptodds
Under review as a conference paper at ICLR 2027

Which Experiment Would Change Your Mind? Bayesian Optimal Experimental Design for Connectome-Constrained Models

Abstract

Connectome-constrained models combine maps of synaptic wiring with task-driven learning to predict neural activity. These networks can solve the same task while making incompatible predictions about neural activity. However, it’s imperative to ask the question: what recording would distinguish them? We formulate the joint choice of cell type and stimulus as Bayesian experimental design over trained networks, accounting for what calcium imaging can measure. The framework integrates out recording-specific gain and offset analytically, marginalizes uncertain voltage-to-calcium transforms at inference, and searches 38,870 candidate recordings subject to genetic targeting constraints. In a benchmark of 91 networks calibrated using 313 published calcium recordings, one selected recording identifies the generating network in 63.7% of single-trial and 80.8% of five trial measurements. A conventional T5b motion experiment reaches 50.8% and 63.9%, and the first recording of a standard battery reaches 37.1% and 55.4%. For single-trial measurements, stimulus optimization on the selected target adds 12.3 percentage points over a generic edge, demonstrating the value of joint design. A protocol fixed on the original ensemble reaches 90% identification within 65 seconds of scheduled recording in two retrained ensembles. Retrospective tests on published recordings improve held-out prediction over random conditions. These results turn disagreement among connectome-constrained models into concrete recording choices, with identification evaluated within an explicit finite hypothesis set.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Bayesian experimental design, connectome-constrained neural networks, computational neuroscience, active learning, model discrimination, model misspecification, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Jg8tOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

From my understanding, I believed this is what the author has done in a simplified explanation (in my own words): there are five elements to the system:

  1. visual input: sequences of images (input)
  2. motion prediction: model's estimate of optical flow (output)
  3. neural connections are fixed.
  4. model parameters: to be trained to adjust parameters such as synaptic gains, neural time constants
  5. calcium image recording: these are external evidence for comparing's model's "internal" (like the neural activation) predictions with responses measured in a fly.

The paper has two main steps:

Firstly, several connectome constrained networks re trained on the same optic-flow prediction tasks. The connectivity if fixed, but the model parameter is learned.

Secondly, the authors chooses which experiment would best distinguish those networks by jointly select which cell type to record and which visual stems to present. The resulting calcium recording can be compared with each model's predicted recording to update which models are better supported by the evidence.

Critical strengths

I believe the author has a clear motivation, the connectom constrained networks with similar task performance can make incompatible predictions about the neural activity. Simply put, the same input/same output may have very different "internal" representation. So the paper propose method to see which recording would distinguish them.

The paper was written generally quite clear. Although the area of focus is different from a typical submissions into ICLR, however, in my view, this is good thing for ICLR to have a diverse topics.

The paper also shows a detailed evaluation and practical constraints, for example the 91 trained network.

Critical weaknesses

My main concern is that the paper provides a potentially valuable contribution to neuroscience. However, the techniques used in this paper, such as expected information gain (EIG), “Bayesian experimental design” and “measurement-aware Bayesian experimental design" are established techniques, and the methodological advance primarily lies in combining measurement modeling, joint target–stimulus selection, and experimental constraints for connectome-constrained networks.

Also, I believe “Bayesian experimental design” is an established term. However, the full phrase “measurement-aware Bayesian experimental design” appears to be the authors’ own proposed terminology. What is the need to invent such a term?

I found that the methodologies from this paper, has no broader machine learning insights. The applicability beyond this setting are not clear to me. I am not sure if this paper is a good fit for ICLR, but instead, can make a good paper in neuroscience literature.

Questions for authors

n/a

Reviewer_gijYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a measurement-aware Bayesian experimental design framework for connectome-constrained neural networks. The method jointly selects cell types and visual stimuli to maximize expected information gain (EIG) over an ensemble of trained networks, while accounting for calcium imaging dynamics, measurement uncertainty, and experimental constraints.

Experiments on 91 trained networks demonstrate improved model identification compared with conventional experimental choices. The authors further evaluate protocol transfer across independently retrained ensembles and conduct retrospective evaluations using published biological recordings.

Critical strengths
  1. Interesting and practically motivated problem.

The paper addresses an important question: how to efficiently select biological experiments to distinguish competing neural network models. Incorporating calcium imaging characteristics and experimental constraints makes the framework more practically relevant than approaches based solely on simulated neural activity.

  1. Extensive experimental evaluation.

The authors provide comprehensive comparisons with conventional and model-based baselines, robustness analyses, protocol-transfer experiments, and retrospective evaluations on published biological recordings. These evaluations help clarify both the strengths and limitations of the proposed approach.

Critical weaknesses
  1. Substantial gap between simulated model differences and biological variability.

The framework selects experiments based on differences among networks trained on a single connectome. However, real neural responses can vary across individual neurons of the same cell type and across different animals due to differences in connectivity, intrinsic dynamics, and biological conditions.

Although the paper accounts for measurement noise and calibration uncertainty, it remains unclear whether the differences among trained models are sufficiently distinguishable relative to these sources of biological variability. The protocol-transfer experiments use independently retrained networks sharing the same connectome, which does not directly address this concern.

Consequently, there is a substantial gap between identifying simulated networks and reliably distinguishing biological explanations in real neural systems. The scientific objective appears considerably broader than what the current experiments establish.

  1. Unclear biological significance of identifying individual trained networks.

The main benchmark evaluates whether a selected recording can identify the generating network within a finite hypothesis set. However, most candidate networks share the same connectome and training objective, differing primarily in their learned parameters. It is unclear what meaningful biological insights can be obtained by identifying one particular network.

More importantly, successful identification does not necessarily imply that the selected model provides a more accurate description of biological neural dynamics. The out-of-ensemble experiments in Appendix J highlight this limitation, showing that EIG-based selection does not consistently improve prediction of unseen neural responses compared with standard or random acquisition.

The retrospective biological evaluations also show limited improvements over well-designed conventional protocols. Therefore, the connection between improved model identifiability and meaningful biological model validation remains insufficiently established.

Questions for authors
  1. Biological variability: How do the response differences among trained networks compare with the variability observed across neurons of the same cell type and across individual flies? Can the authors provide an analysis demonstrating that the selected experiments remain informative under such biological variability, beyond the measurement noise considered in the current framework?

  2. Scientific significance: What biological conclusions can be drawn from identifying a particular member of an ensemble trained on the same connectome? Given the out-of-ensemble results in Appendix J, how should improved identification accuracy be interpreted in terms of biological validity or predictive performance on previously unobserved neural responses?

Reviewer_MG2COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats each network of the flyvis connectome-constrained ensemble (Lappalainen et al., 2024; extended with two retrained ensembles to 91 networks) as a hypothesis. It chooses the (cell type, stimulus) recording with maximal expected information gain (BALD over a finite set), using a calcium observation model:

  • Rectification, GCaMP6f kinetics and frame averaging.
  • Gain and offset integrated out analytically, giving a rank-two-updated Gaussian with O(T) likelihood.
  • A five-component mixture over voltage-to-calcium transforms.
  • Noise calibrated to 313 published recordings.

A pairwise-distance screen shortlists designs among 38,870, and lazy greedy selection with driver-line and time constraints builds protocols. In leave-one-out identification, one EIG recording identifies the generating network in 63.7% (1 trial) and 80.8% (5 trials) of cases, versus 50.8% and 63.9% for a T5b edge. A frozen protocol transfers to retrained ensembles. Retrospective tests on published data show gains over random conditions but not over standard batteries.

Critical strengths
  1. A well-engineered, measurement-aware design pipeline for a real and timely problem. Task-trained connectome models are degenerate: equally good networks disagree on response sign for half the cell types. Turning that into concrete, genetically feasible recording choices, with an exact nuisance-marginalized likelihood, data-calibrated noise and explicit time budgets, is practically valuable for experimentalists.
  2. Unusually thorough and candid evaluation. Strong baselines (max-min discrepancy, PageRank target, the field-default T5b experiment, a stimulus-only control), Holm-corrected paired tests, power analysis, and many robustness checks (kinetics, saturation, amplitude-dependent noise, out-of-ensemble flies) are reported, including those that go against the method.
Critical weaknesses
  1. The headline numbers come from a closed world, and the paper's own open-world test shows the method fails there. In every headline result, the generating network is one of the hypotheses and the observation model matches the simulator. A real fly is never an ensemble member. App. J tests this case: 41 networks retrained with the same connectome and recipe are scored against the 50 published members.

    • One EIG recording puts 0.95 of the posterior on a member whose fit to the generating network ranks 22nd–26th of 50, where chance is 25.5.
    • It improves prediction of unseen responses less than a standard battery or a random recording (0.046 vs. 0.077/0.075, Table 10).
    • It makes flash-sign predictions for other cell types worse than the unweighted ensemble (−4.8 points, the largest drop of any rule).

    "Identifying" a member therefore says nothing about the fly, even for networks trained with the same recipe. On published data, EIG beats only random conditions: it is within 0.007 R² of a standard contrast battery, and the calcium-vs-voltage ordering reverses between T5 and Tm9. The abstract's claim that the method "turns disagreement … into concrete recording choices" is not supported where it matters.

  2. The Bayesian machinery is not shown to be necessary. EIG over a finite ensemble with Gaussian nuisance marginalization is standard BALD / Box–Hill model discrimination, and a simple heuristic does as well:

    • Under realistic uncertainty, max-min discrepancy is statistically tied with EIG (Table 1: 61.1 vs. 63.7, CI [−3, 8]; 78.8 vs. 80.8). It is numerically ahead in the 50-network version (70.9 vs. 67.5 and 81.6 vs. 78.3, Table 6).
    • The authors measure signal-dependent noise in the real ROIs (1.47× larger above the median amplitude). Under that noise, max-min beats EIG significantly in the 50-network benchmark (by 9.5 and 5.5 points). At 91 networks the two are level, and EIG's lead over the field-default experiment is not significant even after re-deriving the objective (App. D).
    • EIG also loses to max-min under slower indicator decay and wide gain priors.
    • EIG favours large responses (median peak 8.5 typical ranges), which is the regime indicator saturation degrades most (67.5 → 38.0% at the measured 90th percentile).

    The main text is also very dense, and most decisive results and caveats appear only in the long appendices.

Questions for authors
  1. (Could change my assessment.) Can you design for an objective aligned with predicting the real fly, for example class-level or prediction-oriented EIG with a model-discrepancy term? Does it beat a standard battery on out-of-ensemble networks and held-out published responses?
  2. (Could change my assessment.) Given that max-min discrepancy matches or beats EIG under the noise actually measured in the data, which method should an experimentalist use, and what does the full Bayesian treatment add?
  3. (Clarification.) What is 91-network recovery when indicator saturation is set at the measured 90th percentile, for both EIG and max-min?
  4. (Suggestion.) Is a prospective recording of the frozen TmY4/Tm5b/Mi10 protocol available or planned? Even a small pilot would substantially strengthen the claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.