acceptodds
Under review as a conference paper at ICLR 2027

How Many Scientists? Epistemic Diversity in Expanding-Frontier Discovery

Abstract

Scientific discovery is inherently sequential: each experiment updates beliefs and changes which hypotheses are worth testing next. As budget-limited discovery systems such as laboratory-in-the-loop agents are deployed at scale, this raises a basic organizational question: should discovery be driven by a single shared belief state, or distributed across several persistent searches? We formalize this question as an adaptive search process over an expanding hypothesis graph, and introduce (Independent Search over Local Adaptive Neighborhoods for Discovery), which allocates a fixed experimental budget across persistent epistemic states with locally expanding frontiers. Multiple states can preserve distinct search directions and local coverage, but they fragment the budget each trajectory needs to resolve prerequisite hypotheses. Theoretically, we show that fragmentation limits what each isolated trajectory can reach, while heterogeneous commitments and prior diversity can offset this loss, yielding a budget-dependent crossover from concentrated to distributed search. Controlled simulations recover this regime structure, and on the SciGym systems-biology benchmark, distributed search likewise overtakes pooled search as the experimental budget grows. Applying these principles to gene regulatory network recovery on CausalBench, we find that persistent local search states improve over a matched single-state variant and existing baselines on K562. Together, these results identify the number and isolation of persistent search states as design variables for sequential scientific discovery.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific Discovery, Epistemic diversity, Adaptive experimental design, Expanding-frontier search, Multi-agent search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1keNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how a fixed experimental budget should be organized in sequential scientific discovery: as one pooled epistemic state, or as M isolated states that never share updates. Discovery is formalized as adaptive search over an expanding hypothesis graph, where resolving a hypothesis exposes its neighbours for later testing. The multi-state organization is called ISLAND. On a K-bridge world, the authors show that (i) any state needs at least (K-1)n_min tests to reach the target, so too much splitting gives zero success; (ii) for symmetric states, splitting helps iff M*Psi(B/M) > Psi(B), where Psi is the cumulative discovery hazard, and splitting cannot help when pooled and isolated states share the same IFR discovery-time law; (iii) when initial commitments are heterogeneous, every fixed M >= 2 eventually overtakes M = 1; (iv) independent priors are never worse than a shared prior. Simulations, SciGym reaction-network recovery and CausalBench GRN recovery show pooled search winning at small budgets and distributed search at larger budgets. On K562, M = 16 reaches W1 0.209 and BioPrec 0.550, compared with 0.182 and 0.424 for M = 1.

Critical strengths
  1. The question is important and timely. As AI-scientist and lab-in-the-loop systems are deployed, how many parallel lines of inquiry to keep is a real design choice, and the paper makes it explicit.
  2. The theory is clean. The IFR condition gives a useful rule of thumb: if continued search becomes more promising, pool the budget; splitting only pays off when some commitments are good and some are bad.
  3. The evidence covers several levels: theory, controlled simulations and two biological benchmarks. The CausalBench controls (global-from-start, random subspaces) and the seed x prior factorial are good designs.
Critical weaknesses

W1. The pooled baseline is weak by construction, so the paper does not isolate the value of isolation.

(a) The pooled state uses a single prior draw w0 ~ nu, i.e., one overconfident and possibly misdirected prior. A pooled agent with the mixture prior over w would already hedge internally. The paper itself notes that an unrestricted controller could emulate every isolated state (Sec. 2). So what is shown is closer to "with myopic acquisition rules and an overconfident prior, forcing diverse commitments helps". This is the known idea behind Thompson sampling and bootstrapped / randomized-prior exploration (Osband et al., 2016; 2018).

(b) Baselines that pool evidence but keep diversity are missing, e.g., a pooled state with a mixture prior, a pooled state with diverse batch acquisition (BatchBALD, which is cited), a pooled state with restarts, and partially communicating states. Without them, the claim that isolation is a design variable is not supported. A sweep over communication levels would also connect directly to Zollman (2010), which the paper lists as future work.


W2. The theoretical novelty is limited, and the theory gives no practical way to choose M.

(a) Lemma B.1 shows that isolated states are equivalent in law to sequential restarts with private state. The main allocation result (the best local budget maximizes Psi(t)/t, Prop. B.25) is therefore a fixed-budget version of Luby et al. (1993), as the authors acknowledge.

(b) The crossover in Prop. 3.3 follows almost directly from the surrogate: an unfavourable state never finds the target, so the single state saturates at alpha by assumption. Prop. 3.4 is a one-line Jensen argument, and Prop. 3.1 is a counting argument on the bridge graph. Please clarify which results depend on the expanding frontier and would not hold in a standard restart / portfolio setting.

(c) The theory says what determines M* but cannot estimate it, and the plug-in rule needs pilot hazard estimates that the authors admit are hard to obtain (App. B.7). Luby's universal schedule needs no knowledge of the distribution, and successive halving is another option. Both are cited, but neither is tested.


W3. The benchmark experiments do not test the mechanism in the theory, and the gains are small or non-monotone.

(a) SciGym: In smaller systems the difference is -0.01 points at B = 20 and +1.03 at B = 24 (Table 4), and the budget only goes up to 24. The pooled beam starts higher (0.417 vs 0.398 at B = 0) and barely changes after B = 12 (0.443-0.446). This looks like the well-known diversity collapse of beam search, so the comparison may be closer to diverse beam search vs. beam search than to the isolation mechanism.

(b) CausalBench: The "budget" is the number of bootstrap resamples of the same observational matrix, not new experiments. The gain is not monotone: at 640 updates, M = 2 and M = 4 are below M = 1 on both metrics, and M = 8 is lower on BioPrec (Table 7). The authors explain this by the cap on frontier growth (at most 640/M genes per state). This suggests that the gain comes from keeping each state local, i.e., estimating partial correlations in small subspaces and then averaging them, rather than from epistemic diversity. Also, the random-subspace control uses 16 genes per update while ISLAND states see at least 64 genes (Eq. 8), so the control is not size-matched.

(c) Intervention selection (Table 10): M = 4 improves over M = 1 by only 0.007 in W1 and 0.014 in BioPrec. Most of the gain over the baselines comes from the base scoring method, not from having multiple states.

(d) The CausalBench comparison only includes GRNBoost and two DCDI variants. Chevalley et al. (2025) report a wider set of methods. Please include more of them or explain the selection.

Questions for authors

See weaknesses.

5 comments

Thank you for the detailed review, and for noting that the question is important, the theory is clean, and our experiments and controls are well designed. We ran the experiments you suggested and respond to each point below.

W1(a). Strength of the pooled baseline

"The pooled state uses a single prior draw w0 ~ nu, i.e., one overconfident and possibly misdirected prior. A pooled agent with the mixture prior over w would already hedge internally."

We ask how a discovery system with a fixed experimental budget and a fixed acquisition rule should organise its search, as one search process or as several isolated ones. Each process keeps its own epistemic state, made up of its prior, its history and the hypotheses it has exposed. As Sec. 2 notes, an unrestricted controller could emulate every isolated state, but a real system, such as a lab-in-the-loop system with several agents, still has to choose one organisation, and how to divide effort under a fixed budget is the question this paper investigates.

We ran the stronger pooled agents you suggested, including a pooled agent with a mixture prior (LUCB, B=16,384B=16{,}384, 100 paired worlds).

MethodBridge, target foundSBM24, factors found (of 24)
Pooled, random prior draw (paper's baseline)18/10013.9
Pooled, averaged prior21/10015.2
Pooled, max-of-draws prior20/10019.6
Pooled, evidence-reweighted mixture prior18/10014.8
Pooled with evidence-retaining restarts19/10019.6
Four states with full raw-evidence sharing19/10019.6
ISLAND, M=455/10020.4

ISLAND finds the bridge target 37 percentage points more often than the mixture-prior agent (paired 95% CI [27, 47]). On SBM24 its margin over the strongest pooled agents is smaller but consistent, with 0.83 more factors than full evidence sharing (CI [0.51, 1.15]) and the same number of false acceptances. At a small budget (B=2,048B=2{,}048) the mixture-prior agent wins instead (bridge 9/100 vs 1/100; SBM 6.9 vs 0.1 factors), so the stronger pooled agent shows the same budget-dependent crossover as the paper.

The mixture couples all hypotheses through its component weights, but it still searches a single frontier, and our ablation in the reply to Reviewer a4q4 suggests that in the bridge world, separate beliefs help only when each state also keeps its own frontier.

"This is the known idea behind Thompson sampling and bootstrapped / randomized-prior exploration (Osband et al., 2016; 2018)."

Thompson sampling and randomised priors randomise beliefs within one search process. ISLAND separates the search processes themselves, so each has its own history and its own set of testable hypotheses. Thompson sampling is one of our acquisition rules, and an isolated advantage still appears, in an intermediate budget band, when every state uses it (App. E.3, Fig. 7), so posterior sampling within one state does not reproduce the benefit of separate states. These methods also assume a fixed action space, whereas in our setting the search history decides which hypotheses can be tested at all, and none can be tested before its prerequisites are resolved. We will add this discussion and cite Osband et al. (2016).

W1(b). Baselines that pool evidence but keep diversity

"Baselines that pool evidence but keep diversity are missing, e.g., a pooled state with a mixture prior, ... a pooled state with restarts, and partially communicating states."

The table above includes the mixture prior, restarts and full evidence sharing. For diverse batch acquisition, BatchBALD selects the same batch as greedy BALD in our belief model. Beliefs factorise across hypotheses and each test informs only the tested hypothesis, so for distinct hypotheses the joint mutual information is additive, I(yh1,…,yhb;μ)=∑kI(yhk;μhk)I(y_{h_1},\dots,y_{h_b};\mu)=\sum_{k} I(y_{h_k};\mu_{h_k}), and the top-bb hypotheses by BALD maximise it. BatchBALD's diversity comes from correlated parameters, which appear here only under the mixture prior. The closest pooled-evidence, diverse-acquisition baseline is the four-state system with full evidence sharing, where states with different priors choose tests against the same evidence, and it still trails ISLAND (19/100 vs 55/100 on the bridge).

For partial communication, we varied the probability pp with which states share information (full results in our reply to Reviewer a4q4). Sharing posteriors drops SBM24 performance from 20.4 factors at p=0p=0 to 15.0 at p=1p=1, because each state counts its own evidence twice when it returns through the others' posteriors. Sharing raw observations changes SBM24 little, but cuts bridge success from 55/100 to 19/100. As Zollman (2010) found for scientific communities, how isolated the states are matters as much as how many there are. We leave a full study of communication protocols to future work.

W2(a,b). Theoretical novelty

"Lemma B.1 shows that isolated states are equivalent in law to sequential restarts with private state. The main allocation result ... is therefore a fixed-budget version of Luby et al. (1993)"

We built the theory on existing frameworks deliberately. When pooled and isolated states have the same discovery-time law, our results reduce to known restart results (Lemma B.1, Prop. B.25), so readers can check the connection directly. Our setting is more general. In an expanding-frontier problem each state's history decides which hypotheses it can test, so the way the search is organised changes the discovery-time law, and the restart results no longer decide the allocation. In the bridge world with 10 noise variables, for example, 6 of the 48 candidates next to a seed lead towards the target, but only 1 of the 63 candidates one step before the target does (Cor. B.6), and the pooled state has to search the union of all seed neighbourhoods.

"Please clarify which results depend on the expanding frontier and would not hold in a standard restart / portfolio setting."

ResultUses the expanding frontier?Role
Prop. 3.1, Lemmas B.3 to B.5, Cor. B.6Yes, depth and competition come from exposure along the graphFragmentation penalty, compounding with depth
Lemmas B.9 to B.13Yes, newly exposed candidates compete for testsLarger frontiers slow progress on prerequisites
Prop. B.20Motivated by it, since pooled and isolated frontiers differCrossover without a "never succeeds" assumption
Props. 3.2, B.18, B.25No (restart / portfolio)Allocation identity, IFR rule, Luby analogue
Props. 3.3, B.22, 3.4No (surrogate / prior family)Commitment hedging and diversity gain

The crossover does not depend on the assumption in Prop. 3.3 that unfavourable commitments never succeed. Prop. B.22 allows poor commitments to advance slowly (the advantage at a fixed budget holds when they are slow enough), App. B.6 allows the pooled state to be better directed than each isolated state (α0≥α\alpha_0 \geq \alpha), and Prop. B.20 derives a crossover from a difference in discovery-time laws alone. Prop. 3.4 has a short proof, and we use it to separate the diversity gain from the cost of splitting the budget (Cor. B.23), which Fig. 3C then measures. We will add this table to Sec. 3.

W2(c). Choosing M in practice

"The theory says what determines M* but cannot estimate it ... Luby's universal schedule needs no knowledge of the distribution, and successive halving is another option. Both are cited, but neither is tested."

Our main aim is to explain why discovery systems should keep separate epistemic states, with local frontiers and persistent trajectories, and when they should not, namely when each state's budget is too small to clear the prerequisites. Estimating M⋆M^\star for a new problem requires its discovery hazards, and these are hard to observe when discoveries cannot be verified directly, as discussed in our limitations section.

Luby's schedule needs no knowledge of the discovery-time law, but it guarantees expected time to success only up to a logarithmic factor, and at a fixed budget that factor is lost discovery. It also spends roughly equal total budget on each run length (1, 2, 4, ...), and every run too short to clear the prerequisite chain fails with certainty (Prop. 3.1(i)). Successive halving drops states based on an intermediate score, and expanding-frontier discovery gives only a weak signal of this kind. In the bridge world, a state on the right path makes progress by rejecting false intermediate hypotheses, so it may not look promising until late. We tested both schedules in an exploratory experiment (LUCB, 10 paired worlds).

MethodBridge, target found (B = 2k / 8k / 16k)SBM24, factors found (B = 2k / 8k / 16k)
Fixed M=11 / 2 / 26.2 / 11.4 / 13.6
Fixed M=40 / 4 / 60.0 / 15.4 / 19.2
Fixed M=160 / 0 / 10.0 / 0.0 / 1.6
Luby schedule0 / 1 / 40.0 / 6.3 / 10.1
Successive halving (from M=16)0 / 0 / 30.0 / 8.8 / 14.3

Neither schedule matched the best fixed allocation, which changed from M=1M=1 at the lowest budget to M=4M=4 at higher budgets. We will state the practical rule that follows from Prop. 3.1. At small budgets M⋆=1M^\star = 1, and a system should add parallel states only when each can afford to clear the prerequisite chain.

W3(a). SciGym

"This looks like the well-known diversity collapse of beam search, so the comparison may be closer to diverse beam search vs. beam search than to the isolation mechanism."

In our SciGym setup, the pooled search keeps one beam of 16 candidate reaction networks, and ISLAND keeps four beams of 4. If the pooled beam had collapsed, its 16 members would have become near-copies of one network, and we measured whether this happens. For each structural component cc of a candidate network (a reactant or product assigned to a reaction), we computed the fraction p(c)p(c) of beam members that contain it. From these fractions, D=2∑cp(c)(1−p(c))D = 2\sum_c p(c)(1-p(c)) gives the expected number of components on which two randomly drawn members differ. DD is zero when all members are the same network, so a collapsing beam would show DD falling towards zero as the budget grows. In our runs DD stays flat, at 2.94 at B=0B=0 and 2.95 at B=32B=32 (95% CI [2.87, 3.03], bootstrapping over systems), so two members still differ on about three components on average. Because an average could hide a subset of collapsed beams, we also checked every run individually. At B=32B=32, the least diverse of the 900 pooled runs (90 systems, 10 seeds each) still has D=2.16D = 2.16, and 90% of runs have DD above 2.39. No pooled beam collapses, and the pooled search plateaus in F1 while its beam remains diverse. Its members all score experiments against one shared history, while ISLAND's four beams each update on their own history.

"the budget only goes up to 24"

We stopped at B=24B=24 because both curves had plateaued and the crossover had already happened, and the crossover is what this experiment is designed to show. We have now extended the budget to B=32B=32 (all 90 systems, 10 paired seeds). N.B. This is a fresh run with new seeds, so values differ from Table 4.

BPooled M=1Isolated M=4Difference (F1 points)
00.3720.341−3.08
200.3970.395−0.23
240.3970.402+0.43
280.3970.405+0.79
320.3980.409+1.13

Between B=24B=24 and B=32B=32, M=1 gains 0.04 F1 points (95% CI [−0.17, 0.25]) and M=4 gains 0.73 points ([0.19, 1.31]). The pooled search has stopped improving, the isolated search keeps improving, and M=4 stays ahead after the crossover.

Reviewer_a4q4OpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Based on the assumption of isolation in distributed cases, the authors provide a theoretical view about ``how many scientist or discovery branch should be given under the limited resources'' and a clear pattern: in small budget, single scientist is preferred, while in large budget, many scientists give board exploration and yields better overall performance. The authors evaluate their framework in simulations and benchmarks like SciGym, and the demonstration of CausalBench indeed confirms their framework's hypothesis.

Critical strengths
  • Modeling the problem of adaptive search under a limited budget on an expanding hypothesis graph is a sound approach. This could potentially foster further expansion within the community.
  • Experiments conducted in controlled simulated environments, SciGym and CausalBench, support this framework, i.e., a finding that, to some extent, aligns with earlier theoretical propositions.
Critical weaknesses
  • The assumptions required for the theoretical proofs are relatively stringent, for instance, those concerning prerequisite depth, the advance rate of a well-directed search, and the proportion of well-directed initial commitments. Furthermore, there is a lack of analysis and reflection regarding the relationship between the theoretical framework and real-world scenarios; additionally, at the experimental level, the reasons underlying the advantages of various states are not sufficiently clearly articulated.
Questions for authors

Could the authors consider the other situation that many parallel searching states CAN communicate with each other, perhaps discuss some theoretical analysis and a few demonstrations even in well-structured simulations?

4 comments

Thank you for your review, and for recognising our problem motivation and the potential impact of the framework. We address each point in your review below.

The assumptions required for the theoretical proofs are relatively stringent, for instance, those concerning prerequisite depth, the advance rate of a well-directed search, and the proportion of well-directed initial commitments.

We would like to clarify that these are not restrictive assumptions, but parameters of our theoretical framework. They mostly decide where the crossover happens, not whether our conclusions apply. Concretely:

  • Prop. 3.1(i) (fragmentation) holds under any acquisition rule: no state can reach the target with a budget below the prerequisite depth.
  • Prop. 3.2 (allocation identity) makes no assumption on depth, advance rate or commitment quality, and holds for any discovery-time laws. Prop. B.18 (the IFR result) adds only that pooled and isolated states share one discovery-time law with an increasing failure rate.
  • Prop. 3.3 uses a surrogate in which unfavourable commitments never succeed. Prop. B.22 relaxes this (poor commitments advance slowly instead of failing), and App. B.6 shows the crossover survives when the pooled state is better directed than each isolated state (α0≥α\alpha_0 \geq \alpha).
  • Prop. B.20 gives a crossover with no "unfavourable commitment" assumption at all, only that isolated states move faster through their slowest stage.
  • Prop. 3.4 (diversity gain) holds for any prior family.

In more detail, these are the assumptions for each of our key results.

ResultWhat it assumesUses the expanding frontier?
Prop. 3.1 (fragmentation)Bridge world; part (i) holds under any acquisition rule; part (ii) uses uniform selection (extended to softmax, App. B.4)Yes
Lemma B.9, Cor. B.11, Lemma B.12 (frontier competition)Gaussian posteriors; Thompson or LUCB-style ruleYes
Prop. 3.2 (allocation identity)Independence across states onlyNo
Prop. B.18 (IFR, no splitting gain)Same discovery-time law for pooled and isolated states, and that law is IFRNo
Prop. 3.3 (crossover)Surrogate with α,Q,p\alpha, Q, p; poor commitments never succeedNo
Prop. B.22Poor commitments advance slowly (ε>0\varepsilon > 0)No
Prop. B.20 (crossover from different laws)Isolated states faster through their slowest stageMotivated by it
Prop. 3.4 (diversity gain)Matched seeds; any prior familyNo

Furthermore, there is a lack of analysis and reflection regarding the relationship between the theoretical framework and real-world scenarios

We agree that the main paper should make this clearer. Informally, imagine a lab of autonomous scientific agents, each with access to some experimental toolkits and existing literature. Its "epistemic state" is what it believes to be true, and its frontier is the set of experiments it can currently do, which is limited by its knowledge (e.g. if it does not know a pathway exists, it cannot test it) and by its access (e.g. if it cannot run a given assay, it cannot test a hypothesis that needs that assay). The frontier expands as the agent learns. For example, once it finds evidence or a paper suggesting the pathway exists, experiments on that pathway become possible.

This also relates to the three parameters you mentioned above. In this example, prerequisite depth is the number of intermediate findings needed before a hypothesis becomes testable (e.g. establishing a pathway before testing a downstream regulator); the advance rate is the hit rate of a well-directed programme; and the fraction of well-directed commitments is how often a starting hypothesis or prior points in a productive direction. Medicinal chemistry is another real-life example. A confirmed hit opens its analogue series (compounds one substitution away, matching our one-edit graph), and industry already runs parallel "backup series" from different starting points, which can be considered an informal version of isolated states.

Both of our benchmarks emulate this expanding frontier. In SciGym, each search state can only test reaction networks one edit away from its current candidates, so which networks become reachable depends on what it has already found. In CausalBench, the hypotheses a state can test depend on which genes it has promoted into its candidate set, i.e. which genes it "knows" to be relevant. One limitation is that any fixed benchmark dataset is, by construction, not truly open-ended, so we have to create unknowns that are hidden from the agent but known to us.

at the experimental level, the reasons underlying the advantages of various states are not sufficiently clearly articulated.

Our theory identifies two mechanisms: separate beliefs (priors and evidence), which hedge against a poor commitment, and separate frontiers, which stop each state being diluted by distractors exposed elsewhere. Our experiments show that both contribute, with their relative weight depending on the problem.

Seed × prior factorial (Fig. 3C, App. E.5; bridge K4/X18, LUCB, M=4M=4, B=4,096B=4{,}096, 100 replicates):

Starting seedsShared priorDiverse priors
Shared26%36%
Distributed35%46%

Diverse priors and distributed starting points each add about 10 points, and the gains are roughly additive.

(new experiment) belief-sharing × frontier-sharing ablation. As a directional exploration, we further separated these two mechanisms. In each world, the pooled state (shared beliefs, shared frontier) is M=1M=1, the other cells use M=4M=4, and all cells have the same total budget.

Bridge world (K4/X15, LUCB, B=16,384B=16{,}384; target found in 16 paired replicates):

Shared frontierPrivate frontiers
Shared beliefs5/16 (pooled)6/16
Private beliefs0/167/16 (ISLAND)

SBM world with 24 unknown true factors (same budget; mean true factors found, 16 paired worlds):

Shared frontierPrivate frontiers
Shared prior13.13 (pooled)18.44
Private priors20.6920.94 (ISLAND)

The two worlds stress the mechanisms differently, in the way the theory predicts:

  • In the bridge world (one target, many distractor variables), the two mechanisms interact. Splitting beliefs and budget while keeping one shared frontier is the worst option (0/16), because each state has a quarter of the budget but faces the full, diluted frontier (Prop. 3.1, Lemma B.12). Private frontiers with shared beliefs perform about as well as pooled search (6/16 vs 5/16), so private beliefs pay off only when each state also keeps its own frontier (7/16). With 16 replicates we read these counts as directional.
  • In the SBM world (24 targets spread across blocks), useful candidates are less diluted, so the frontier matters less and diverse starting directions matter more. Private priors alone add 7.6 factors over pooled, private frontiers alone add 5.3, and with both, private priors add 2.5 on top of private frontiers (better in 13/16 worlds, tied in 3, worse in none).

In both worlds, performance was highest when both frontiers and beliefs (priors) were private.

Could the authors consider the other situation that many parallel searching states CAN communicate with each other, perhaps discuss some theoretical analysis and a few demonstrations even in well-structured simulations?

We originally left this as future work, since designing a proper communication protocol between search states is probably a paper on its own. We have now added a short analysis below, with exploratory experiments in the next comment.

What do states share? Within our model, states can share raw evidence (observations) or beliefs (posteriors). We hypothesise the following.

  • Sharing beliefs causes recursive collapse. If a state adopts others' posteriors, then in the next round those posteriors already contain its own evidence, which is passed back and counted again. Repeated every turn, this creates an echo chamber in which beliefs converge regardless of the evidence. In the simplest form, blending each state towards the consensus, bi←(1−ϵ) bi+ϵ bˉb_i \leftarrow (1-\epsilon)\, b_i + \epsilon\, \bar b, leaves the mean unchanged but shrinks every state's deviation from it by (1−ϵ)(1-\epsilon) per round, i.e. to (1−ϵ)t(1-\epsilon)^t after tt rounds. With ϵ=0.25\epsilon = 0.25, only 0.755≈24%0.75^5 \approx 24\% of the original diversity is left after five rounds.
  • Collapse removes the hedging term. The isolated advantage comes from the product form of Prop. 3.2, 1−J=∏iSi(Bi)1 - J = \prod_i S_i(B_i), which needs states to fail independently. Converging beliefs make failures positively correlated, so joint failure is at least the product of individual failures (by the same association argument as Prop. B.24, which we have not proved for communicating states). This offsets the diversity gain of Prop. 3.4, and at full sharing all states act as one pooled state.
  • Sharing raw evidence could be effective. Fresh observations are not double-counted, so sharing them pools information without the echo-chamber effect. It still pulls commitments together, but more slowly.

Exploratory experiments for communication between parallel states

(a) Posteriors vs raw observations (SBM, 24 unknown factors, M=4M=4, LUCB, B=16,384B=16{,}384, 100 paired worlds; mean true factors found):

Information exchangedp=0 (isolated)p=0.01p=0.1p=1
Raw observations20.4020.0420.0619.57
Posterior summaries20.4019.6718.3315.00

As predicted, sharing posteriors collapses performance as pp grows (−5.4 factors at p=1p=1), while sharing raw observations costs little.

(b) Raw-observation broadcast in the bridge world (K4/X15, LUCB, M=4M=4, B=16,384B=16{,}384, 100 paired worlds; target found):

p0 (isolated)0.010.11
Target found55/10038/1000/10019/100

In this single-target world with many distractor variables, even raw sharing hurts, and it hurts most at intermediate pp. Our reading is that partial sharing pays both costs. It pulls states towards the same commitment, while each state still searches with only a quarter of the budget. At p=1p=1 the states pool all evidence and partly recover, but still trail isolation.

(c) Forced consensus on CausalBench (K562, intervention phase): after each intervention step, every state is blended towards the consensus with coefficient ϵ\epsilon. This is the recursive-collapse rule above, applied deliberately as a stress test. Values are mean W1.

MIsolatedϵ=0.25\epsilon=0.25ϵ=0.5\epsilon=0.5ϵ=0.75\epsilon=0.75
40.5880.4750.4550.441
80.5880.4610.4430.437
160.5900.4400.4260.414

W1 falls monotonically with ϵ\epsilon at every MM. Isolated states are also combined, but only once, at evaluation. The loss therefore comes from merging states during search, and the timing of combination matters as much as the number of states.

(d) Occasional synchronisation can help (bridge K4/X4, softmax, M=4M=4, B=4,096B=4{,}096, 32 paired runs; full sync every kk queries):

ProtocolNo sharingEvery 256Every 64Every 16
Target found7/3219/3218/3218/32

In this small world with few distractor variables, pooling evidence helps, but the gain plateaus almost immediately. Syncing every 256 queries performs as well as syncing every 16, so states that share rarely capture nearly all of the benefit, and sharing 16 times more often adds nothing. The same sharing hurts in the bridge world of (b), which has many more distractors, where raw broadcast cut success from 55/100 to 19/100.

Summary. Sharing beliefs collapses diversity, as the analysis predicts. Sharing raw evidence costs little in some worlds and a great deal in deep worlds with many distractors, and where it helps, rare synchronisation captures almost all of the gain. These simulations also treat communication as free, but in real discovery systems every exchange has a cost. For example, autonomous agents must pass long experimental histories into each other's context, and labs would need to standardise protocols and data formats before their results can be pooled. Full evidence sharing at every step is therefore rarely feasible, and the plateau in (d) suggests it is rarely needed. Together with work on transient diversity in scientific communities (Zollman, 2010), these results support the main message of the paper, that the number and isolation of search states are design variables. Full isolation is a strong, cheap default, and occasional synchronisation is a natural extension when a problem is small and evidence is easy to share.

I thank the authors for the additional experiments and detailed clarifications. The ablations and discussion on communication have improved my understanding of the paper's internal mechanics.

However, my core concerns regarding the paper's practical validity and scientific impact remain unresolved:

  1. In real-world scientific discovery, researchers do not blindly synchronize unverified drafts at every step or average out conflicting beliefs; rather, they prefer to share validated milestones or high-value evidence. Concluding that full isolation is a strong default based solely on the failure of naive broadcasting/averaging does not convince me that this insight transfers to realistic collaborative discovery.

  2. Regarding how to determine the optimal number of states M∗M^* in practice, the authors suggest using M=1M=1 at small budgets and only adding parallel states when each can afford to clear the prerequisite chain. In real discovery problems, the prerequisite depth and hazard rates are unknown a priori, which is exactly why search is needed. Thus, this rule of thumb is circular and offers limited actionable guidance.

Overall, while the mathematical formulation is self-consistent within its assumptions, the gap between these idealized mechanisms and scientific workflows remains too large. In the current AI4Science landscape, I do not believe these findings provide reliable or impactful guidance to warrant acceptance. I appreciate the authors' hard work during the rebuttal, and I will maintain my rating while increasing my confidence.

Reviewer_aLA6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

As budget-constrained discovery systems such as laboratory-in-the-loop agents are deployed at scale, a question arises: should discovery be driven by a single shared belief state, or distributed across several persistent searches? Here, the authors formalize this question as an adaptive search process over an expanding hypothesis graph, and introduce ISLAND (Independent Search over Local Adaptive Neighborhoods for Discovery), which allocates a fixed experimental budget across persistent epistemic states with locally expanding frontiers. The experimental results indicate that distributed search overtakes pooled search as the experimental budget grows. Moreover, their results identify the number and isolation of persistent search states as design variables for sequential scientific discovery.

Critical strengths

The paper is well written. It addresses an important problem. The approach is practical and conceptually sound. The strong empirical results across diverse settings support the proposed approach.

Critical weaknesses

Although the empirical evaluation covers SciGym and CausalBench, additional studies are required to determine whether these principles extend to broader scientific discovery domains, including materials design and chemical synthesis.

The proposed approach introduces additional computational and coordination overhead compared to a single shared belief state for maintaining multiple persistent epistemic states with locally expanding frontiers, which could limit scaling to extremely large-scale hypothesis spaces.

Questions for authors

The distributed search is capable of preserving diverse directions and escaping local traps. However, it fragments the budget required to resolve prerequisite hypotheses. How to determine this crossover point? Tuning this might be hard in practice because it might be very sensitive to specific downstream domain constraints.

Could you explain more about whether adaptive budget splitting can overcome the fragmentation penalty in severely constrained budget settings?

1 comment

Thank you for your positive review, and for finding the paper well written, the problem important, and the approach practical and sound.

"additional studies are required to determine whether these principles extend to broader scientific discovery domains, including materials design and chemical synthesis."

Materials design and chemical synthesis are natural next domains, and they share the structure our framework models. We chose SciGym and CausalBench because they test this structure differently. SciGym asks a search to design sequential experiments that recover a reaction network, and CausalBench asks it to recover a gene network over hundreds of genes. In both, what a search has already found decides which hypotheses it can test next. In chemistry, once a medicinal chemist confirms a hit, its analogues (compounds one substitution away) become worth testing, which matches the one-edit hypothesis graph in our model. Pharmaceutical teams also run parallel "backup series" from different chemical starting points, which behave much like isolated search states. We will add this mapping to the Discussion and plan a closed-loop chemistry or materials study as future work.

"The proposed approach introduces additional computational and coordination overhead compared to a single shared belief state ... which could limit scaling to extremely large-scale hypothesis spaces."

ISLAND adds no coordination cost, because states do not communicate during search and we combine their results only once, at evaluation. Our budget counts experiments (Sec. 2, App. B.1), since in a lab-in-the-loop system an experiment costs far more than maintaining a belief state.

Isolated states can also need less computation than a single state. On CausalBench K562, each of the 16 states works with at most about 75 genes (Eq. 8, since at M=16M=16 each state can promote at most 40 genes), while a single state grows towards all 622 genes. Each update inverts a covariance matrix over these genes, and the cost of inversion grows roughly with the cube of their number, so an update over 75 genes costs several hundred times less than an update over 622 genes. Isolation therefore makes large hypothesis spaces easier to handle.

"How to determine this crossover point? Tuning this might be hard in practice because it might be very sensitive to specific downstream domain constraints."

The exact crossover for a new domain is hard to estimate, but the theory shows that it moves with the prerequisite depth, the rate at which a well-directed search advances, and the fraction of starting commitments that are well directed (Props. 3.3 and B.20). Deeper prerequisite chains and larger hypothesis spaces move the crossover to larger budgets (Fig. 2B), and a smaller fraction of good starting points moves it to smaller budgets. A practitioner can reason about these quantities from domain knowledge without estimating them precisely.

We also tested two schedules that need no knowledge of the crossover, Luby's universal restart schedule and successive halving, in an exploratory experiment (LUCB, 10 paired worlds).

MethodBridge, target found (B = 2k / 8k / 16k)SBM24, factors found (B = 2k / 8k / 16k)
Fixed M=11 / 2 / 26.2 / 11.4 / 13.6
Fixed M=40 / 4 / 60.0 / 15.4 / 19.2
Fixed M=160 / 0 / 10.0 / 0.0 / 1.6
Luby schedule0 / 1 / 40.0 / 6.3 / 10.1
Successive halving (from M=16)0 / 0 / 30.0 / 8.8 / 14.3

In both worlds the best fixed choice moves from M=1 to M=4 as the budget grows, as the theory predicts. Both schedules fall behind a fixed M=4. Luby's schedule spends much of its budget on runs too short to clear the prerequisite chain, and successive halving depends on early signs of progress, which discovery rarely gives. In the bridge world, a state on the right path advances by rejecting false intermediate hypotheses and may look unpromising until late. In our settings, a moderate number of states, each with enough budget to clear the expected prerequisite depth, worked best.

"Could you explain more about whether adaptive budget splitting can overcome the fragmentation penalty in severely constrained budget settings?"

When each state's budget is smaller than the prerequisite depth, no isolated state can reach the target under any schedule (Prop. 3.1(i), Prop. 3.3(i)), so a single search process is better. In the experiment above, at B=2,048B=2{,}048 the single state performs best in both worlds, while Luby's schedule and successive halving find nothing. Adaptive splitting starts to help only once the budget lets each state clear the prerequisites. For multi-agent scientific systems, this means a small budget is best spent on one well-resourced line of inquiry, and parallel agents pay off only when each can afford to search deeply.

We hope the new results strengthen your confidence in the paper, and thank you again for your support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.