acceptodds
Under review as a conference paper at ICLR 2027

Scientific Search Depends on Its Substrate: Constructing and Auditing Exploration Representations

Abstract

Scientific search depends on decisions made before the first experiment: which alternatives remain distinguishable and which transitions a policy can take. We formalize this organization as a scientific exploration substrate and introduce CASES (Constructing and Auditing Scientific Exploration Substrates), a protocol for constructing, auditing, and activating substrates. A task policy records scientific, semantic, structural, and executable requirements; the audit authorizes the states and operations those requirements support. Controlled comparisons show that changing the substrate can alter search outcomes even when the policy remains fixed. Across scientific settings, the effects vary with task structure and consumer capability, shaping both which solutions can be reached and how evidence can be reused. Together, these results establish scientific search organization as an explicit experimental variable, separable from the policy that explores it.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for Scientific Discovery, Scientific Agents, Hypothesis Search, Search Space Representation, Scientific Exploration, Sequential Experimental Design, Auditable AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NycsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CASES, a framework for constructing and auditing scientific search representations. It introduces a formal substrate model and evaluates its effects on search performance across six scientific settings.

Critical strengths

The paper includes controlled comparisons across several scientific benchmarks. However, these experiments do not establish a sufficiently novel or generalizable contribution.

Critical weaknesses
  1. The novelty is limited. The dependence of search performance on representation and neighborhood structure is already well established. CASES largely reformulates existing concepts without introducing a clearly distinct search mechanism.

  2. The theoretical contributions are weak. The main propositions follow directly from standard mathematical properties, while Theorem 1 relies on a simple triangle-inequality argument. Their relevance to the implemented method and empirical improvements remains unclear.

  3. The GB1 experiment does not convincingly demonstrate the value of CASES. Hill climbing achieves 8/30 optimum hits using the same Hamming-1 neighborhood, compared with 9/30 for CASES. The improvement appears primarily attributable to the manually specified neighborhood rather than the proposed framework.

  4. The empirical gains are small and inconsistent. ResearchBench improves only from 3.23 to 3.29 over ResearchAgent, MADE shows marginal improvements, and CASES underperforms the summary baseline on two PiEvo tasks. The evidence is insufficient to support broad effectiveness claims.

  5. Practicality remains questionable. Only 4 of 121 audited candidates were activated, and the paper does not adequately evaluate construction overhead or end-to-end efficiency. More importantly, CASES does not demonstrate how effective representations can be automatically discovered.

Questions for authors
  1. What concrete capability does CASES provide beyond existing representation-aware search methods?

  2. How do the theoretical guarantees translate into measurable empirical improvements?

  3. Given the comparable GB1 results from simple hill climbing, what improvement can be attributed specifically to CASES?

  4. How is the substantial construction and auditing overhead justified by the relatively modest performance gains?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Nycs (1/2)

Thank you for your detailed review and constructive suggestions. We have revised the manuscript to clarify the contribution of CASES and added controlled experiments on scientific fidelity, attribution, and search. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.

Q1: What concrete capability does CASES provide beyond existing representation-aware search methods?

A1: CASES checks whether an executable state or operation faithfully realizes a registered scientific requirement, then uses that check to authorize the search space. This complements methods that learn a search geometry, generate executable programs or verify formal statements.

Table R6. Components explicitly reported in representative methods. ✓ = present; ❌ = not reported as the defined component. The last column refers specifically to authorization of scientific identities and operations before exploration.

MethodGraph/kernel/latent geometryLLM guidanceProgram/formal checksSource-meaning checksScientific pre-search gate
ALDE (Nat. Commun. 2025) [1]✓❌❌❌❌
BOPRO (ICLR 2025) [2]❌✓✓❌❌
AlphaEvolve (arXiv 2025) [3]❌✓✓❌❌
LGBO (ICLR 2026) [4]✓✓❌❌❌
LABO (ICML 2026) [5]✓✓❌❌❌
ReForm (ICLR 2026) [6]❌✓✓✓❌
Kernel Discovery (arXiv 2026) [7]✓✓✓❌❌
CASES (Ours)✓✓✓✓✓

References: [1] ALDE; [2] BOPRO; [3] AlphaEvolve; [4] LGBO; [5] LABO; [6] ReForm; [7] Kernel Discovery.

ReForm explicitly tests whether formal statements retain the meaning of their natural-language source; AlphaEvolve tests the behavior of generated programs. CASES instead checks whether scientific candidates remain distinguishable and whether proposed operations preserve their declared meaning. In a controlled GB1 fault test, schema validation accepts 50/80 corrupted proposals, while the scientific audit rejects 80/80 and accepts 10/10 intact controls. The distinction is illustrated in Section 2 and Appendix D's method comparison.

Q2: How do the theoretical guarantees translate into measurable empirical improvements?

A2: Theorem 1 guarantees preservation of registered distinctions under its conditions. The practical quantity is therefore the number of discoveries that can be attributed to unambiguous scientific identities.

Table R7. GB1 search with 10% identity merges. Fifty paired starts, 480 queries.

PolicySearchable statesTotal new hitsUnambiguous hits
Intact reference5,000207.64207.64
Selective audit4,500198.98198.98
Bypass identity check4,750204.54190.90

Merging 500 of the 5,000 identities into 250 pairs leaves 4,750 computational states, including 250 ambiguous ones. The selective policy excludes them and retains 4,500 unambiguous states. It obtains 8.08 more attributable hits than bypassing the audit, despite the latter's slightly higher nominal hit count. The full-coverage rule rejects this space; selective authorization is a separate tested extension (Appendix C).

Q3: Given the comparable GB1 results from simple hill climbing, what improvement can be attributed specifically to CASES?

A3: We intervene on each component separately: first change the scientific relations while keeping selection fixed, then compare selectors using the same graph and full 480-query budget.

Table R8. GB1 attribution on 149,361 variants. One hundred paired starts, 96 initial observations.

ComparisonCASESControl
Best fitness, true vs degree-shuffled graph7.834.59
Best fitness, full-budget restart hill climbing7.837.39
Starts reaching fitness 7.091/10071/100
Query index to fitness 7.0*275.3322.2

Runs not reaching the threshold are counted at query 480.

Shuffling the edges reduces mean best fitness by 3.24 without changing the selector. On the true graph, CASES improves by 0.44 over full-budget restart hill climbing and reaches the fitness threshold in twenty more starts. These controls separate the contribution of scientific relations from the policy that uses them (Section 3.3; Appendix D). The full-pool GP-EI comparison is discussed in Nycs Q6 (Table R11).

AuthorsOpenReview ↗

Response to Reviewer Nycs (2/2)

Q4: The empirical gains are small and inconsistent. ResearchBench improves only from 3.23 to 3.29 over ResearchAgent, MADE shows marginal improvements, and CASES underperforms the summary baseline on two PiEvo tasks. The evidence is insufficient to support broad effectiveness claims.

A4: The clearest gains occur in early reaction discovery.

Table R9. Discovery and transfer. BH: 20 paired starts; query-80 counts exclude initialization. PiEvo replay: one seed.

Setting / endpointCASESComparator
BH AdBrettPhos: target recovery by query 8035.9%GP probability: 12.5%
BH BTMG: target recovery by query 8028.2%GP probability: 8.2%
ResearchBench: original overall score3.294ResearchAgent: 3.229
ResearchBench: without Audit revision3.2942.957
MADE: mean mSUN0.372Summary: 0.354
PiEvo original MBO / NHO scores4.001 / 1.213Summary: 4.850 / 1.497
Fixed-pool PiEvo, MBO mean at query 32.553 (matched GP)Original order: 1.863
Fixed-pool PiEvo, NHO mean at query 61.388 (matched GP)Original order: 1.046

By query 80, CASES finds 2.9× / 3.4× as many qualifying reactions in the two BH tasks. Expanded ResearchAgent performs better under similar generation tokens, while removing audit-guided revision lowers the original CASES score by 0.337. MADE and PiEvo results vary by task; the single-seed offline PiEvo replay shows how a task-matched policy can improve use of the same substrate. The revised claim focuses on audited substrate construction and structured discovery, rather than uniform gains across all downstream methods.

Q5: Practicality remains questionable. Only 4 of 121 audited candidates were activated, and the paper does not adequately evaluate construction overhead or end-to-end efficiency. More importantly, CASES does not demonstrate how effective representations can be automatically discovered.

A5: The agent uses task requirements and outcome-blind pool statistics to write and audit an executable relation before search, directly testing the originally prescribed construction step.

Table R10. Evidence for model construction.

Construction settingVerified resultSearch effect
Model-written GB1 relations, 16 pools16/16 accepted42.36% vs 38.54% hand-written density rule
Model-selected relation families64/64 accepted+15.92–17.23 pp vs Hamming-1
Original open-form archive4/120 activationsFull failure record: Table R4

The agent selects radius two for twelve sparse pools and radius one for four dense pools. This pool-conditioned choice produces auditable graphs from registered operators and improves recovery over fixed Hamming-1 and the hand-written density rule. Historical open-form failures are reported under yoZv Q2 (Appendices C–D).

Q6: How is the substantial construction and auditing overhead justified by the relatively modest performance gains?

A6: The answer depends on what is expensive: objective evaluations in the laboratory, or computation when the objective is a cheap lookup.

Table R11. Different resource regimes. BH: five timed starts with fresh construction (distinct from the 20-start confirmation in R9); GB1: fifty-run reuse with preselected query schedules.

SettingCASESGP baseline
BH AdBrettPhos13.21 s/run; Q50 89.41.17 s; Q50 166.8
BH BTMG13.45 s/run; Q50 106.41.11 s; Q50 213.8
GB1, 50-reuse amortized compute3,936 queries; 4.41 s; 50/50 optimaGP-EI: 480 queries; 4.89 s; 25/50 optima

Fresh BH construction adds roughly 12 s/run and reduces queries to half-recovery by 46–50%; physical-assay costs were not measured. With one GB1 substrate reused fifty times, sparse-neighborhood selection completes 8.2× as many cheap evaluations in similar measured compute time. These are preselected schedules, not hard real-time cutoffs. At equal 480 queries, GP-EI obtains higher fitness (8.05 vs 7.58). The trade-off is objective-evaluation cost versus search computation (Appendix D).

Thank you for the detailed response. My concerns have been addressed, and I have raised my rating to 3 (Weak Accept).

AuthorsOpenReview ↗

Thank you for your feedback and updated rating

Thank you very much for taking the time to review our response and for raising your rating. We truly appreciate your rigorous questions and constructive suggestions, which meaningfully strengthened the clarity and rigor of our paper.

Reviewer_yoZvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how the way a scientific search space is organized affects discovery, separately from the search policy. It introduces CASES, a procedure for constructing and auditing these representations, and evaluates its effects on hypothesis generation, search, and evidence reuse across six settings.

Critical strengths

The GB1 intervention is the clearest result: changing the representation while keeping the selector and query budget fixed increases optimum hits from 0/30 to 9/30. I also appreciate the shared-consumer controls and the authors’ reporting of mixed results on MADE and PiEvo. These help clarify where the approach works and where its limitations lie.

Critical weaknesses
  • The paper makes a stronger case that representation matters than that CASES is an effective way to construct it. The useful GB1 neighborhood is specified in advance, and simple hill-climbing reaches the optimum in 8/30 runs, close to CASES’ 9/30.

  • It is still hard to judge how reliable the full workflow is in practice. Only four of 121 auditable construction candidates activate. This is a candidate-level statistic, but task-level success rates and the cost of failed attempts are needed to put it in context.

  • The link between the theory and experiments needs more work. Appendix C.7 checks distinction coverage, but not the separation condition required by Theorem 1. This leaves the main guarantee unverified in practice.

Questions for authors
  1. What is the overall task success rate, and how much does construction cost when failed proposals and revisions are included?

  2. Can the separation condition in Theorem 1 be measured on at least one experimental task?

5 comments
AuthorsOpenReview ↗

Response to Reviewer yoZv

Thank you for your valuable comments and suggestions. Below, we address your concerns point by point and we’ve revised our paper according to your suggestions. We would appreciate it if you could let us know whether your concerns are addressed by our response.

Q1: The paper makes a stronger case that representation matters than that CASES is an effective way to construct it. The useful GB1 neighborhood is specified in advance, and simple hill-climbing reaches the optimum in 8/30 runs, close to CASES’ 9/30.

A1: The agent now writes and audits an executable relation before exploration. Its input consists of the task requirements and candidate-pool statistics, without unqueried fitness values. We fix the search policy to measure the effect of the constructed relation.

Table R3. Construction and full-budget search controls. GB1: 16 held-out pools, five search starts per pool, 480 queries; gains use unrounded means.

ComparisonCASESControlGain
Model-written relation: recovery42.36%Hand-written density: 38.54%+3.82 pp
Same relation vs original Hamming-142.36%23.43%+18.93 pp
GLM / Qwen family proposals: recovery39.35% / 40.67%Hamming-1: 23.43%+15.92 / +17.23 pp
Full-budget restart hill climbing: best fitness7.837.39+0.44

All 16/16 written relations pass audit. The model uses the registered two-substitution operator on twelve sparse pools and retains single substitutions on four dense pools. The choice follows pool structure without access to unqueried fitness. It improves recovery over both the original fixed neighborhood and the hand-written density rule. Appendix D includes the deterministic radius-two control, which reproduces the resulting graphs under the same degree cap. The separate family-and-parameter experiment also obtains 64/64 accepted proposals.

The original optimum-hit counts (9/30 for CASES, 8/30 for hill climbing) do not capture performance on difficult starts. We therefore added 100 paired starts with the same graph and the full 480-query budget for restart hill climbing. CASES improves mean best fitness by 0.44 and reaches fitness 7.0 in 91 vs 71 of 100 starts. The gain is clearer in the difficult starts than in the small original optimum-hit count.

Q2: What is the overall task success rate, and how much does construction cost when failed proposals and revisions are included?

A2: We report the original open-form attempts and the newer task-registered experiments separately, including failed attempts and revisions.

Table R4. Construction outcomes and costs.

StudyOutcomeFailure-inclusive record
Original 14-task archive4/120 unique runs activated139,563 s over 128 logged rows
Corrected-checker follow-up1/51 passed146 attempts; 49,160 s
Registered GB1/BH construction156/160 pass finalized re-audit196 calls; 364,754 tokens
Three specified analysis tasks: staged / joint9/9 vs 7/9 completeStaged semantic audit: 5.67 calls, 13,759 tokens/run, including synthesis

The review's 4/121 counts non-error log rows; 4/120 counts distinct task–seed–method runs. The 156/160 figure is a later re-audit of saved proposals, not the original pre-search activation rate. It measures reliability on registered GB1/BH tasks; the 9/9 result measures implementation of explicitly specified analyses. These findings identify where registration and staged construction work well, while the early archive documents the difficulty of broader open-form requests. Human preparation time was not recorded (Appendix C).

Q3: Can the separation condition in Theorem 1 be measured on at least one experimental task?

A3: Yes. We enumerate registered scientific identities in GB1 and BH. Here γ is the minimum metric separation between required scientific identities, and ε is the certified bound on realization error.

Table R5. Exact finite checks of Theorem 1. Margin = γ − 2ε.

CaseγεMarginCoverageIdentity collisions
Intact GB1 (six pools)101100%0
Intact BH (287 / 800 identities)101100%0
GB1, declared identity merges11−1100%250 detected
GB1, concealed merges11−195%250 missed
GB1, equality boundary0.250.1250100%250

Intact realizations satisfy γ > 2ε and preserve every registered distinction. Deliberate mergers show how the margin fails; concealed mergers reveal the importance of complete realization coverage. Appendix C provides the class sets, exact distances, and injected mappings.

Thank you for the detailed reply. Most of my concerns have been addressed. Could you provide more clarification about the practical reliability of this full workflow?

AuthorsOpenReview ↗

Response to Reviewer yoZv (Follow-up): Practical Reliability

Thank you for the follow-up and for confirming that our previous response addressed most of your concerns. Below, we clarify the practical reliability of CASES across prospective execution, pre-search auditing, and runtime enforcement.

Prospective end-to-end reliability. In a prospective BH study, we evaluated 20 new construction proposals across two 800-reaction tasks (AdBrettPhos and BTMG), using five starts per task under each of two prompt configurations. All 20 proposals passed the as-run audit on the first attempt without retries, and all 20 completed the full 240-query search budget. Both prompt configurations produced the same supported one-component relation and matched the search metrics of the deterministic reference interface. Including construction, the compact workflow averaged 13.3 seconds per run (versus 42.8 seconds for the full prompt). Under the same 240-query budget, mean target recovery was 90.9% for CASES compared with 76.0% for GP target-probability selection (90.9% vs. 84.8% on AdBrettPhos; 90.9% vs. 67.3% on BTMG). Note that this comparison matches query budgets rather than wall-clock time, where GP setup is amortized across five starts.

Audit and authorization reliability. In a controlled audit challenge, the full audit rejected all 80 deliberately corrupted proposals while accepting all 10 unaltered controls and 32 equivalent terminology variants—whereas schema-only validation accepted 50/80 corrupted proposals. In a separate recovery study, CASES recovered all 40 eligible cases and none of the 8 prohibited cases, whereas Greedy, HS-tree, and an ungoverned control each recovered all 8 prohibited cases.

Runtime stability and transfer. Once activated, the substrate remains fixed throughout each exploration episode. Runtime checks continuously monitor its representation type, scientific identities, realization mapping, and interpretation; no prohibited changes were recorded in any logged episode (Appendix C.6). In a separate transfer study, a relation frozen on development data and applied unchanged across eight confirmation sets achieved 54.61% recovery versus 47.67% for development-selected fixed controls (+6.94 pp), winning on six sets and tying on two.

Operational scope. Because these diagnostics test distinct workflow stages, we report them separately rather than pooling them into a single aggregate rate. Together, they demonstrate reliable end-to-end operation when task requirements and supported relation families are explicitly registered—whereas unconstrained open-form construction across arbitrary scientific interfaces remains challenging (as reflected in our earlier 4/120 cross-task archive and 1/51 corrected-checker follow-up). For the same reason, we keep the retrospective 156/160 re-audit distinct from prospective end-to-end execution rates.

We have consolidated these diagnostics in Appendices C.2–C.6 and D.5–D.6 of the revised manuscript. Please let us know if this resolves your remaining question!

Reviewer_mTX9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that scientific-search performance depends not only on the selection policy but also on the scientific exploration substrate: the representation that determines which hypotheses/states are distinguishable, how they are interpreted, and what transitions a downstream consumer can execute. The paper introduces CASES, a Construct–Audit–Activate protocol. A task policy is registered before construction; candidate substrates are audited for scientific admissibility, semantic fidelity, structural validity, and executability; only a supported subset of states and operations is activated. The paper gives formal claims about non-weakening of task requirements, representation-family-neutral activation, and conditional preservation of registered distinctions.

Critical strengths
  1. The paper identifies an important and often under-isolated source of variation in scientific agents: the representation and action structure made available to a policy. The distinction between “what the policy chooses” and “what it can distinguish or reach” is conceptually useful.

  2. The proposed interface is broader than a standard embedding or graph: it explicitly combines scientific content, computational form, realization, and executable operations. This is a promising framing for scientific-agent evaluation.

Critical weaknesses

The paper compares against strong named baselines, but it is not always obvious whether they receive equivalent candidate pools, representations, prompt budgets, oracle access, and tuning.

Questions for authors

What is the cost of construction and audit (LLM calls, wall-clock time, human specification effort), and how does this compare with the baselines under equal total budget?

2 comments
AuthorsOpenReview ↗

Response to Reviewer mTX9

Thank you for your careful review and constructive suggestions. We have revised the manuscript to clarify the experimental controls and construction costs. We address your questions below and please kindly let us know if our response addresses your concerns.

Q1: The paper compares against strong named baselines, but it is not always obvious whether they receive equivalent candidate pools, representations, prompt budgets, oracle access, and tuning.

A1: In the query-controlled GB1 and BH comparisons listed below, candidate pools, initial observations, objective oracles, and query allowances are shared within each comparison. Representation and selection differ by method. ResearchBench and the transfer tasks instead report shared inputs and realized model budgets; offline tuning is documented where available.

Table R1. Comparison conditions. Initial observations are included in total queries.

StudyCandidates; initial/total queriesComparison
GB1, relation intervention (100 starts)149,361; 96 initial / 480 totalTrue vs degree-shuffled graph; same selector
GB1, GP comparison (50 starts)149,361; 96 initial / 480 totalCASES vs one-hot RBF GP
BH, full pool (50 starts)4,312; 40 initial / 240 totalCASES vs fitted GP / DKL
BH, fresh tasks (20 starts/task)800/task; 40 initial / 240 totalCASES vs GP probability / EI
ResearchBench160 common difficult tasksShared inputs, model and judge; actual tokens recorded
MADE / PiEvoSame task inputs and hostCommon guidance caps; actual tokens may differ

On the full 149,361-variant GB1 pool, GP-EI reaches a higher mean best fitness at 480 queries (8.05 vs 7.58 for CASES); the separate compute-budget comparison is reported in Nycs Q6 (Table R11). Under the matched BH query budget, CASES recovers 90.5% vs 83.4% of qualifying AdBrettPhos reactions and 89.1% vs 65.9% on BTMG. These are independent 20-start confirmations.

We also rechecked the original full-BH GP fitting path. The archived implementation could enter an unintended fallback, so that estimate has been replaced by a fully fitted RBF-GP comparison (550/550 successful fits): 112.04 qualifying reactions for CASES versus 86.00 for GP over the same 50 starts. The corrected results appear in Table 5, Figure 2(b), and Appendix D.2.1 of the revised PDF. Historical offline tuning effort was not logged; we describe the recorded configurations without claiming equal tuning histories.

Q2: What is the cost of construction and audit (LLM calls, wall-clock time, human specification effort), and how does this compare with the baselines under equal total budget?

A2: We include unsuccessful proposals and revisions, and distinguish model generation from computation spent on search. The BH timed study charges CASES for a fresh construction on every run.

Table R2. Construction costs and resource-controlled outcomes. BH: five timed starts/task, including fresh construction; ResearchBench: 160 paired tasks.

SettingCASESComparator / result
Registered GB1/BH construction196 calls; 364,754 tokens; 6,070 s batch elapsed156/160 later pass finalized audit
Compact BH construction1 call; 514 tokens; 13.3 sFull prompt: 1 call; 2,075 tokens; 42.8 s; both 10/10 pass
BH AdBrettPhos, construction + search13.21 s/run; Q50 89.4GP: 1.17 s; Q50 166.8
BH BTMG, construction + search13.45 s/run; Q50 106.4GP: 1.11 s; Q50 213.8
ResearchBench, similar generation tokens4 calls; 8,130 tokens; 3.350/5Expanded RA: 5 calls; 7,982 tokens; 3.600/5

The 160-proposal ledger includes 20,215 s of summed model latency and 3.25 s of deterministic audit time. The acceptance result is a later audit of saved proposals. In the construction-inclusive BH runs, CASES uses 46.4% and 50.2% fewer queries to recover half the qualifying candidates, at a higher compute cost of approximately 12 s per run. The ResearchBench comparison matches generation-token use closely, not all resource costs; expanded ResearchAgent obtains a higher score.

The human time spent writing task specifications was not recorded. Each specification can be reused across runs; its preparation time and physical-assay costs therefore remain outside the measured totals (Appendices C, D, and F).

AuthorsOpenReview ↗

Kind Follow-up on Baseline Controls and Construction Costs

Dear Reviewer mTX9,

Thank you again for your careful review and constructive questions on baseline parity and construction overhead. To make our main updates easy to locate, we briefly highlight the key results addressing both points:

  • Controlled comparisons and corrected BH GP (Table R1; Section 4.2, Appendix F): We standardized candidate pools, initial observations, objective oracles, and query budgets across paired comparisons, and replaced the fallback-affected full-BH GP estimate with a fully fitted RBF-GP baseline (112.04 vs. 86.00 qualifying reactions across 550/550 fits; Table 5, Figure 2(b), Appendix D.2.1).
  • Failure-inclusive construction costs (Tables R2 & R11; Appendices C–D): Including all attempts and revisions, fresh BH construction adds ~12 s per run over GP while reducing queries to 50% recovery by 46.4% (AdBrettPhos) and 50.2% (BTMG) at the 240-query budget. Table R11 separately reports the 50-run amortized compute regime on GB1.

Please let us know if any details remain unclear—we would be very happy to clarify anything further regarding the experimental setups or cost accounting.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Major Contributions, Strengths, and Responses to Concerns

Dear Area Chair, Senior Area Chair, and Program Chairs,

We sincerely thank you for your dedicated service and thoughtful oversight of the review process. Following the rebuttal and discussion period, all concerns raised across the reviews have been addressed, and our submission has received unanimous positive ratings (3, 3, 3). Initially, we received one positive rating from Reviewer mTX9 (3) and two negative ratings from Reviewers yoZv (2) and Nycs (2). After our detailed responses and revised manuscript on Nov 16, Reviewer yoZv (following a reliability follow-up) and Reviewer Nycs both kindly confirmed that their concerns were resolved and raised their ratings from 2 to 3 on Nov 17. We also posted a brief follow-up to Reviewer mTX9 on Nov 18; while we have not received a reply so far, their rating remains 3.

To assist with your final assessment and hopefully reduce your review burden, we concisely summarize our contributions, acknowledged strengths, and rebuttal outcomes below.

1. Core Contributions

  • Scientific Exploration Substrate: Formalizes pre-search organization (identities, distinctions, permitted operations) as an explicit variable separable from the search policy.
  • CASES Protocol: Translates natural-language task requirements into executable substrates and audits scientific, semantic, structural, and executable fidelity prior to search.
  • Verifiable Guarantees & Controlled Validation: Establishes finite-checkable separation bounds (γ−2ε>0\gamma - 2\varepsilon > 0), isolates substrate vs. selector effects on GB1/BH, demonstrates pool-adaptive construction, and candidly evaluates transfer on ResearchBench, MADE, and PiEvo.

2. Acknowledged Strengths

  • Conceptual Value: Reviewer mTX9 noted that CASES "identifies an important and often under-isolated source of variation in scientific agents," praising the separation between "what the policy chooses" and "what it can distinguish or reach" as "conceptually useful" and "a promising framing."
  • Rigorous Controls & Transparency: Reviewer yoZv called the GB1 intervention "the clearest result" and appreciated our "shared-consumer controls" and candid reporting of mixed results on MADE and PiEvo. Reviewer Nycs also acknowledged our "controlled comparisons across several scientific benchmarks."

3. Summary of Rebuttal and Reviewer Feedback

Both initially negative reviewers (yoZv: 2 → 3; Nycs: 2 → 3) confirmed their concerns were addressed. For Reviewer mTX9 (rating 3, no reply yet), we systematically resolved both concerns via controlled baselines, a corrected GP fit, and failure-inclusive cost accounting:

ReviewerConcern / RequestOur Response & New Evidence
mTX9 (3, no reply)Baseline parity & BH GPStandardized pools, inits, oracles, and query caps (Table R1, App. F); replaced fallback-affected BH GP with fully fitted RBF-GP (112.04 vs. 86.00, 550/550 fits; Table 5, Fig. 2(b)).
Construction & audit costFailure-inclusive ledgers (Tables R2, R11): fresh BH construction adds ~12 s/run while cutting queries to 50% recovery by 46.4%–50.2%; 50-run GB1 reuse gives 8.2× throughput under similar compute.
yoZv (2 → 3)Active construction & HCModel writes/audits relations on 16 held-out GB1 pools (+3.82 pp over manual density rule; 16/16 pass); +0.44 fitness over full-budget restart HC across 100 starts (Table R3).
End-to-end reliabilityKept open-form failure records (Table R4); prospective BH runs achieve 20/20 first-pass activation and completion (90.9% vs. 76.0% recovery), 0 runtime violations, and +6.94 pp frozen transfer gain (8 sets).
Theorem 1 checkExact GB1/BH enumeration verifies γ−2ε=1\gamma - 2\varepsilon = 1 (0 collisions) on intact substrates and detects all 250 declared merges (Table R5).
Nycs (2 → 3)Novelty & theory valueDifferentiated pre-search gating from 7 baselines (Table R6); audit rejects 80/80 corrupted proposals; selective audit yields +8.08 more attributable hits under 10% merges (Table R7).
Attribution & scopeEdge shuffling lowers GB1 fitness by 3.24 (Table R8); BH early recovery reaches 2.9×–3.4× GP by query 80 (Table R9); claims scoped to structured, evaluation-expensive discovery (Table R11).

4. Concluding Remarks

All newly added experiments and analyses are highlighted in blue in the revised PDF. We are deeply grateful to the reviewers for helping strengthen our paper and to the chairs for your time and thoughtful consideration.

Respectfully yours, The Authors of CASES

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.