- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CASES, a framework for constructing and auditing scientific search representations. It introduces a formal substrate model and evaluates its effects on search performance across six scientific settings.
Critical strengths
The paper includes controlled comparisons across several scientific benchmarks. However, these experiments do not establish a sufficiently novel or generalizable contribution.
Critical weaknesses
-
The novelty is limited. The dependence of search performance on representation and neighborhood structure is already well established. CASES largely reformulates existing concepts without introducing a clearly distinct search mechanism.
-
The theoretical contributions are weak. The main propositions follow directly from standard mathematical properties, while Theorem 1 relies on a simple triangle-inequality argument. Their relevance to the implemented method and empirical improvements remains unclear.
-
The GB1 experiment does not convincingly demonstrate the value of CASES. Hill climbing achieves 8/30 optimum hits using the same Hamming-1 neighborhood, compared with 9/30 for CASES. The improvement appears primarily attributable to the manually specified neighborhood rather than the proposed framework.
-
The empirical gains are small and inconsistent. ResearchBench improves only from 3.23 to 3.29 over ResearchAgent, MADE shows marginal improvements, and CASES underperforms the summary baseline on two PiEvo tasks. The evidence is insufficient to support broad effectiveness claims.
-
Practicality remains questionable. Only 4 of 121 audited candidates were activated, and the paper does not adequately evaluate construction overhead or end-to-end efficiency. More importantly, CASES does not demonstrate how effective representations can be automatically discovered.
Questions for authors
-
What concrete capability does CASES provide beyond existing representation-aware search methods?
-
How do the theoretical guarantees translate into measurable empirical improvements?
-
Given the comparable GB1 results from simple hill climbing, what improvement can be attributed specifically to CASES?
-
How is the substantial construction and auditing overhead justified by the relatively modest performance gains?
4 comments
Response to Reviewer Nycs (2/2)
Q4: The empirical gains are small and inconsistent. ResearchBench improves only from 3.23 to 3.29 over ResearchAgent, MADE shows marginal improvements, and CASES underperforms the summary baseline on two PiEvo tasks. The evidence is insufficient to support broad effectiveness claims.
A4: The clearest gains occur in early reaction discovery.
Table R9. Discovery and transfer. BH: 20 paired starts; query-80 counts exclude initialization. PiEvo replay: one seed.
| Setting / endpoint | CASES | Comparator |
|---|---|---|
| BH AdBrettPhos: target recovery by query 80 | 35.9% | GP probability: 12.5% |
| BH BTMG: target recovery by query 80 | 28.2% | GP probability: 8.2% |
| ResearchBench: original overall score | 3.294 | ResearchAgent: 3.229 |
| ResearchBench: without Audit revision | 3.294 | 2.957 |
| MADE: mean mSUN | 0.372 | Summary: 0.354 |
| PiEvo original MBO / NHO scores | 4.001 / 1.213 | Summary: 4.850 / 1.497 |
| Fixed-pool PiEvo, MBO mean at query 3 | 2.553 (matched GP) | Original order: 1.863 |
| Fixed-pool PiEvo, NHO mean at query 6 | 1.388 (matched GP) | Original order: 1.046 |
By query 80, CASES finds 2.9× / 3.4× as many qualifying reactions in the two BH tasks. Expanded ResearchAgent performs better under similar generation tokens, while removing audit-guided revision lowers the original CASES score by 0.337. MADE and PiEvo results vary by task; the single-seed offline PiEvo replay shows how a task-matched policy can improve use of the same substrate. The revised claim focuses on audited substrate construction and structured discovery, rather than uniform gains across all downstream methods.
Q5: Practicality remains questionable. Only 4 of 121 audited candidates were activated, and the paper does not adequately evaluate construction overhead or end-to-end efficiency. More importantly, CASES does not demonstrate how effective representations can be automatically discovered.
A5: The agent uses task requirements and outcome-blind pool statistics to write and audit an executable relation before search, directly testing the originally prescribed construction step.
Table R10. Evidence for model construction.
| Construction setting | Verified result | Search effect |
|---|---|---|
| Model-written GB1 relations, 16 pools | 16/16 accepted | 42.36% vs 38.54% hand-written density rule |
| Model-selected relation families | 64/64 accepted | +15.92–17.23 pp vs Hamming-1 |
| Original open-form archive | 4/120 activations | Full failure record: Table R4 |
The agent selects radius two for twelve sparse pools and radius one for four dense pools. This pool-conditioned choice produces auditable graphs from registered operators and improves recovery over fixed Hamming-1 and the hand-written density rule. Historical open-form failures are reported under yoZv Q2 (Appendices C–D).
Q6: How is the substantial construction and auditing overhead justified by the relatively modest performance gains?
A6: The answer depends on what is expensive: objective evaluations in the laboratory, or computation when the objective is a cheap lookup.
Table R11. Different resource regimes. BH: five timed starts with fresh construction (distinct from the 20-start confirmation in R9); GB1: fifty-run reuse with preselected query schedules.
| Setting | CASES | GP baseline |
|---|---|---|
| BH AdBrettPhos | 13.21 s/run; Q50 89.4 | 1.17 s; Q50 166.8 |
| BH BTMG | 13.45 s/run; Q50 106.4 | 1.11 s; Q50 213.8 |
| GB1, 50-reuse amortized compute | 3,936 queries; 4.41 s; 50/50 optima | GP-EI: 480 queries; 4.89 s; 25/50 optima |
Fresh BH construction adds roughly 12 s/run and reduces queries to half-recovery by 46–50%; physical-assay costs were not measured. With one GB1 substrate reused fifty times, sparse-neighborhood selection completes 8.2× as many cheap evaluations in similar measured compute time. These are preselected schedules, not hard real-time cutoffs. At equal 480 queries, GP-EI obtains higher fitness (8.05 vs 7.58). The trade-off is objective-evaluation cost versus search computation (Appendix D).
Thank you for the detailed response. My concerns have been addressed, and I have raised my rating to 3 (Weak Accept).
Thank you for your feedback and updated rating
Thank you very much for taking the time to review our response and for raising your rating. We truly appreciate your rigorous questions and constructive suggestions, which meaningfully strengthened the clarity and rigor of our paper.
Response to Reviewer Nycs (1/2)
Thank you for your detailed review and constructive suggestions. We have revised the manuscript to clarify the contribution of CASES and added controlled experiments on scientific fidelity, attribution, and search. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.
A1: CASES checks whether an executable state or operation faithfully realizes a registered scientific requirement, then uses that check to authorize the search space. This complements methods that learn a search geometry, generate executable programs or verify formal statements.
Table R6. Components explicitly reported in representative methods. ✓ = present; ❌ = not reported as the defined component. The last column refers specifically to authorization of scientific identities and operations before exploration.
References: [1] ALDE; [2] BOPRO; [3] AlphaEvolve; [4] LGBO; [5] LABO; [6] ReForm; [7] Kernel Discovery.
ReForm explicitly tests whether formal statements retain the meaning of their natural-language source; AlphaEvolve tests the behavior of generated programs. CASES instead checks whether scientific candidates remain distinguishable and whether proposed operations preserve their declared meaning. In a controlled GB1 fault test, schema validation accepts 50/80 corrupted proposals, while the scientific audit rejects 80/80 and accepts 10/10 intact controls. The distinction is illustrated in Section 2 and Appendix D's method comparison.
A2: Theorem 1 guarantees preservation of registered distinctions under its conditions. The practical quantity is therefore the number of discoveries that can be attributed to unambiguous scientific identities.
Table R7. GB1 search with 10% identity merges. Fifty paired starts, 480 queries.
Merging 500 of the 5,000 identities into 250 pairs leaves 4,750 computational states, including 250 ambiguous ones. The selective policy excludes them and retains 4,500 unambiguous states. It obtains 8.08 more attributable hits than bypassing the audit, despite the latter's slightly higher nominal hit count. The full-coverage rule rejects this space; selective authorization is a separate tested extension (Appendix C).
A3: We intervene on each component separately: first change the scientific relations while keeping selection fixed, then compare selectors using the same graph and full 480-query budget.
Table R8. GB1 attribution on 149,361 variants. One hundred paired starts, 96 initial observations.
Runs not reaching the threshold are counted at query 480.
Shuffling the edges reduces mean best fitness by 3.24 without changing the selector. On the true graph, CASES improves by 0.44 over full-budget restart hill climbing and reaches the fitness threshold in twenty more starts. These controls separate the contribution of scientific relations from the policy that uses them (Section 3.3; Appendix D). The full-pool GP-EI comparison is discussed in Nycs Q6 (Table R11).