- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how a fixed experimental budget should be organized in sequential scientific discovery: as one pooled epistemic state, or as M isolated states that never share updates. Discovery is formalized as adaptive search over an expanding hypothesis graph, where resolving a hypothesis exposes its neighbours for later testing. The multi-state organization is called ISLAND. On a K-bridge world, the authors show that (i) any state needs at least (K-1)n_min tests to reach the target, so too much splitting gives zero success; (ii) for symmetric states, splitting helps iff M*Psi(B/M) > Psi(B), where Psi is the cumulative discovery hazard, and splitting cannot help when pooled and isolated states share the same IFR discovery-time law; (iii) when initial commitments are heterogeneous, every fixed M >= 2 eventually overtakes M = 1; (iv) independent priors are never worse than a shared prior. Simulations, SciGym reaction-network recovery and CausalBench GRN recovery show pooled search winning at small budgets and distributed search at larger budgets. On K562, M = 16 reaches W1 0.209 and BioPrec 0.550, compared with 0.182 and 0.424 for M = 1.
Critical strengths
- The question is important and timely. As AI-scientist and lab-in-the-loop systems are deployed, how many parallel lines of inquiry to keep is a real design choice, and the paper makes it explicit.
- The theory is clean. The IFR condition gives a useful rule of thumb: if continued search becomes more promising, pool the budget; splitting only pays off when some commitments are good and some are bad.
- The evidence covers several levels: theory, controlled simulations and two biological benchmarks. The CausalBench controls (global-from-start, random subspaces) and the seed x prior factorial are good designs.
Critical weaknesses
W1. The pooled baseline is weak by construction, so the paper does not isolate the value of isolation.
(a) The pooled state uses a single prior draw w0 ~ nu, i.e., one overconfident and possibly misdirected prior. A pooled agent with the mixture prior over w would already hedge internally. The paper itself notes that an unrestricted controller could emulate every isolated state (Sec. 2). So what is shown is closer to "with myopic acquisition rules and an overconfident prior, forcing diverse commitments helps". This is the known idea behind Thompson sampling and bootstrapped / randomized-prior exploration (Osband et al., 2016; 2018).
(b) Baselines that pool evidence but keep diversity are missing, e.g., a pooled state with a mixture prior, a pooled state with diverse batch acquisition (BatchBALD, which is cited), a pooled state with restarts, and partially communicating states. Without them, the claim that isolation is a design variable is not supported. A sweep over communication levels would also connect directly to Zollman (2010), which the paper lists as future work.
W2. The theoretical novelty is limited, and the theory gives no practical way to choose M.
(a) Lemma B.1 shows that isolated states are equivalent in law to sequential restarts with private state. The main allocation result (the best local budget maximizes Psi(t)/t, Prop. B.25) is therefore a fixed-budget version of Luby et al. (1993), as the authors acknowledge.
(b) The crossover in Prop. 3.3 follows almost directly from the surrogate: an unfavourable state never finds the target, so the single state saturates at alpha by assumption. Prop. 3.4 is a one-line Jensen argument, and Prop. 3.1 is a counting argument on the bridge graph. Please clarify which results depend on the expanding frontier and would not hold in a standard restart / portfolio setting.
(c) The theory says what determines M* but cannot estimate it, and the plug-in rule needs pilot hazard estimates that the authors admit are hard to obtain (App. B.7). Luby's universal schedule needs no knowledge of the distribution, and successive halving is another option. Both are cited, but neither is tested.
W3. The benchmark experiments do not test the mechanism in the theory, and the gains are small or non-monotone.
(a) SciGym: In smaller systems the difference is -0.01 points at B = 20 and +1.03 at B = 24 (Table 4), and the budget only goes up to 24. The pooled beam starts higher (0.417 vs 0.398 at B = 0) and barely changes after B = 12 (0.443-0.446). This looks like the well-known diversity collapse of beam search, so the comparison may be closer to diverse beam search vs. beam search than to the isolation mechanism.
(b) CausalBench: The "budget" is the number of bootstrap resamples of the same observational matrix, not new experiments. The gain is not monotone: at 640 updates, M = 2 and M = 4 are below M = 1 on both metrics, and M = 8 is lower on BioPrec (Table 7). The authors explain this by the cap on frontier growth (at most 640/M genes per state). This suggests that the gain comes from keeping each state local, i.e., estimating partial correlations in small subspaces and then averaging them, rather than from epistemic diversity. Also, the random-subspace control uses 16 genes per update while ISLAND states see at least 64 genes (Eq. 8), so the control is not size-matched.
(c) Intervention selection (Table 10): M = 4 improves over M = 1 by only 0.007 in W1 and 0.014 in BioPrec. Most of the gain over the baselines comes from the base scoring method, not from having multiple states.
(d) The CausalBench comparison only includes GRNBoost and two DCDI variants. Chevalley et al. (2025) report a wider set of methods. Please include more of them or explain the selection.
Questions for authors
See weaknesses.
5 comments
W2(a,b). Theoretical novelty
"Lemma B.1 shows that isolated states are equivalent in law to sequential restarts with private state. The main allocation result ... is therefore a fixed-budget version of Luby et al. (1993)"
We built the theory on existing frameworks deliberately. When pooled and isolated states have the same discovery-time law, our results reduce to known restart results (Lemma B.1, Prop. B.25), so readers can check the connection directly. Our setting is more general. In an expanding-frontier problem each state's history decides which hypotheses it can test, so the way the search is organised changes the discovery-time law, and the restart results no longer decide the allocation. In the bridge world with 10 noise variables, for example, 6 of the 48 candidates next to a seed lead towards the target, but only 1 of the 63 candidates one step before the target does (Cor. B.6), and the pooled state has to search the union of all seed neighbourhoods.
"Please clarify which results depend on the expanding frontier and would not hold in a standard restart / portfolio setting."
| Result | Uses the expanding frontier? | Role |
|---|---|---|
| Prop. 3.1, Lemmas B.3 to B.5, Cor. B.6 | Yes, depth and competition come from exposure along the graph | Fragmentation penalty, compounding with depth |
| Lemmas B.9 to B.13 | Yes, newly exposed candidates compete for tests | Larger frontiers slow progress on prerequisites |
| Prop. B.20 | Motivated by it, since pooled and isolated frontiers differ | Crossover without a "never succeeds" assumption |
| Props. 3.2, B.18, B.25 | No (restart / portfolio) | Allocation identity, IFR rule, Luby analogue |
| Props. 3.3, B.22, 3.4 | No (surrogate / prior family) | Commitment hedging and diversity gain |
The crossover does not depend on the assumption in Prop. 3.3 that unfavourable commitments never succeed. Prop. B.22 allows poor commitments to advance slowly (the advantage at a fixed budget holds when they are slow enough), App. B.6 allows the pooled state to be better directed than each isolated state (), and Prop. B.20 derives a crossover from a difference in discovery-time laws alone. Prop. 3.4 has a short proof, and we use it to separate the diversity gain from the cost of splitting the budget (Cor. B.23), which Fig. 3C then measures. We will add this table to Sec. 3.
W2(c). Choosing M in practice
"The theory says what determines M* but cannot estimate it ... Luby's universal schedule needs no knowledge of the distribution, and successive halving is another option. Both are cited, but neither is tested."
Our main aim is to explain why discovery systems should keep separate epistemic states, with local frontiers and persistent trajectories, and when they should not, namely when each state's budget is too small to clear the prerequisites. Estimating for a new problem requires its discovery hazards, and these are hard to observe when discoveries cannot be verified directly, as discussed in our limitations section.
Luby's schedule needs no knowledge of the discovery-time law, but it guarantees expected time to success only up to a logarithmic factor, and at a fixed budget that factor is lost discovery. It also spends roughly equal total budget on each run length (1, 2, 4, ...), and every run too short to clear the prerequisite chain fails with certainty (Prop. 3.1(i)). Successive halving drops states based on an intermediate score, and expanding-frontier discovery gives only a weak signal of this kind. In the bridge world, a state on the right path makes progress by rejecting false intermediate hypotheses, so it may not look promising until late. We tested both schedules in an exploratory experiment (LUCB, 10 paired worlds).
| Method | Bridge, target found (B = 2k / 8k / 16k) | SBM24, factors found (B = 2k / 8k / 16k) |
|---|---|---|
| Fixed M=1 | 1 / 2 / 2 | 6.2 / 11.4 / 13.6 |
| Fixed M=4 | 0 / 4 / 6 | 0.0 / 15.4 / 19.2 |
| Fixed M=16 | 0 / 0 / 1 | 0.0 / 0.0 / 1.6 |
| Luby schedule | 0 / 1 / 4 | 0.0 / 6.3 / 10.1 |
| Successive halving (from M=16) | 0 / 0 / 3 | 0.0 / 8.8 / 14.3 |
Neither schedule matched the best fixed allocation, which changed from at the lowest budget to at higher budgets. We will state the practical rule that follows from Prop. 3.1. At small budgets , and a system should add parallel states only when each can afford to clear the prerequisite chain.
W3(a). SciGym
"This looks like the well-known diversity collapse of beam search, so the comparison may be closer to diverse beam search vs. beam search than to the isolation mechanism."
In our SciGym setup, the pooled search keeps one beam of 16 candidate reaction networks, and ISLAND keeps four beams of 4. If the pooled beam had collapsed, its 16 members would have become near-copies of one network, and we measured whether this happens. For each structural component of a candidate network (a reactant or product assigned to a reaction), we computed the fraction of beam members that contain it. From these fractions, gives the expected number of components on which two randomly drawn members differ. is zero when all members are the same network, so a collapsing beam would show falling towards zero as the budget grows. In our runs stays flat, at 2.94 at and 2.95 at (95% CI [2.87, 3.03], bootstrapping over systems), so two members still differ on about three components on average. Because an average could hide a subset of collapsed beams, we also checked every run individually. At , the least diverse of the 900 pooled runs (90 systems, 10 seeds each) still has , and 90% of runs have above 2.39. No pooled beam collapses, and the pooled search plateaus in F1 while its beam remains diverse. Its members all score experiments against one shared history, while ISLAND's four beams each update on their own history.
"the budget only goes up to 24"
We stopped at because both curves had plateaued and the crossover had already happened, and the crossover is what this experiment is designed to show. We have now extended the budget to (all 90 systems, 10 paired seeds). N.B. This is a fresh run with new seeds, so values differ from Table 4.
| B | Pooled M=1 | Isolated M=4 | Difference (F1 points) |
|---|---|---|---|
| 0 | 0.372 | 0.341 | −3.08 |
| 20 | 0.397 | 0.395 | −0.23 |
| 24 | 0.397 | 0.402 | +0.43 |
| 28 | 0.397 | 0.405 | +0.79 |
| 32 | 0.398 | 0.409 | +1.13 |
Between and , M=1 gains 0.04 F1 points (95% CI [−0.17, 0.25]) and M=4 gains 0.73 points ([0.19, 1.31]). The pooled search has stopped improving, the isolated search keeps improving, and M=4 stays ahead after the crossover.
Thank you for the detailed review, and for noting that the question is important, the theory is clean, and our experiments and controls are well designed. We ran the experiments you suggested and respond to each point below.
W1(a). Strength of the pooled baseline
We ask how a discovery system with a fixed experimental budget and a fixed acquisition rule should organise its search, as one search process or as several isolated ones. Each process keeps its own epistemic state, made up of its prior, its history and the hypotheses it has exposed. As Sec. 2 notes, an unrestricted controller could emulate every isolated state, but a real system, such as a lab-in-the-loop system with several agents, still has to choose one organisation, and how to divide effort under a fixed budget is the question this paper investigates.
We ran the stronger pooled agents you suggested, including a pooled agent with a mixture prior (LUCB, B=16,384, 100 paired worlds).
ISLAND finds the bridge target 37 percentage points more often than the mixture-prior agent (paired 95% CI [27, 47]). On SBM24 its margin over the strongest pooled agents is smaller but consistent, with 0.83 more factors than full evidence sharing (CI [0.51, 1.15]) and the same number of false acceptances. At a small budget (B=2,048) the mixture-prior agent wins instead (bridge 9/100 vs 1/100; SBM 6.9 vs 0.1 factors), so the stronger pooled agent shows the same budget-dependent crossover as the paper.
The mixture couples all hypotheses through its component weights, but it still searches a single frontier, and our ablation in the reply to Reviewer a4q4 suggests that in the bridge world, separate beliefs help only when each state also keeps its own frontier.
Thompson sampling and randomised priors randomise beliefs within one search process. ISLAND separates the search processes themselves, so each has its own history and its own set of testable hypotheses. Thompson sampling is one of our acquisition rules, and an isolated advantage still appears, in an intermediate budget band, when every state uses it (App. E.3, Fig. 7), so posterior sampling within one state does not reproduce the benefit of separate states. These methods also assume a fixed action space, whereas in our setting the search history decides which hypotheses can be tested at all, and none can be tested before its prerequisites are resolved. We will add this discussion and cite Osband et al. (2016).
W1(b). Baselines that pool evidence but keep diversity
The table above includes the mixture prior, restarts and full evidence sharing. For diverse batch acquisition, BatchBALD selects the same batch as greedy BALD in our belief model. Beliefs factorise across hypotheses and each test informs only the tested hypothesis, so for distinct hypotheses the joint mutual information is additive, I(yh1,…,yhb;μ)=∑kI(yhk;μhk), and the top-b hypotheses by BALD maximise it. BatchBALD's diversity comes from correlated parameters, which appear here only under the mixture prior. The closest pooled-evidence, diverse-acquisition baseline is the four-state system with full evidence sharing, where states with different priors choose tests against the same evidence, and it still trails ISLAND (19/100 vs 55/100 on the bridge).
For partial communication, we varied the probability p with which states share information (full results in our reply to Reviewer a4q4). Sharing posteriors drops SBM24 performance from 20.4 factors at p=0 to 15.0 at p=1, because each state counts its own evidence twice when it returns through the others' posteriors. Sharing raw observations changes SBM24 little, but cuts bridge success from 55/100 to 19/100. As Zollman (2010) found for scientific communities, how isolated the states are matters as much as how many there are. We leave a full study of communication protocols to future work.