- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces EVALBENCH, a benchmark for evaluating reference-free scoring methods in PDE program search. It separates target identifiability, static ranking quality (M1), and active search performance (M2). Through experiments across different PDE problems and scoring methods, the authors find that strong static ranking does not necessarily lead to better search outcomes. The paper also introduces SAAS, an operator-aware score aggregation method. Overall, the work highlights the importance of evaluating scoring methods through their impact on actual search performance rather than static ranking alone.
Critical strengths
-
The paper makes a meaningful distinction between target identifiability, static ranking quality (M1), and active search performance (M2). This provides a useful perspective for evaluating scoring methods beyond their performance on fixed candidate sets.
-
The authors investigate multiple PDE operators, scoring methods, and search configurations, including parameter and proposer shifts. These experiments provide useful observations about the limitations of different scoring methods, even though some conclusions require stronger supporting evidence.
Critical weaknesses
-
Limited discriminative power of the benchmark. The results in Appendix B.2 raise concerns about whether EVALBENCH can effectively distinguish different scoring methods. Most evaluated methods achieve identical 8/8 success rates on nearly all operators, while challenging cases such as gCLM remain largely unsolved. Table 4 also shows that most selectors have identical overall solved counts. The benchmark appears to provide limited discrimination in terms of task success. This raises a fundamental question about its value as an evaluation benchmark and whether its task difficulty is appropriately designed to reveal meaningful differences between scoring methods.
-
Limited generalizability to realistic agentic search. The paper motivates EVALBENCH through recent advances in LLM-based scientific program search, but its experiments rely on a restricted library of predefined mathematical expressions and non-LLM proposers. Although several grammar and proposer shifts are evaluated, these remain within the same constrained search framework. It is therefore unclear whether the observed findings would generalize to more realistic LLM-driven program search systems, limiting the practical significance of the benchmark.
Questions for authors
-
Benchmark Discriminative Power: In Appendix B.2, most methods achieve identical 8/8 success rates on nearly all operators, while challenging cases such as gCLM remain largely unsolved. How do the authors justify the discriminative power of EVALBENCH? Can the benchmark reliably distinguish different scoring methods beyond small differences in continuous error?
-
Generalizability to Agentic Search: The paper motivates the benchmark through LLM-based scientific program search, but the experiments use a predefined library of mathematical expressions and non-LLM proposers. Why was this setting chosen, and how well do the authors expect the findings to generalize to realistic LLM-driven program search?
-
Relationship Between M1 and M2: The paper shows that high static ranking quality does not guarantee better active search performance. However, the main M2 comparisons involve scoring methods with similarly high M1 values. Could the authors clarify what the experiments reveal about the relationship between M1 and M2, particularly when ranking quality differs substantially across scoring methods?
-
Effect of the Fitting Objective: In the core experiments, all candidates are fitted using the baseline four-component loss, while only the outer selection score changes. Could this setup limit the potential benefit of alternative scoring methods? How should the results of the fitting factorial in Appendix B.1 be interpreted?
-
Reliability of Local Improvements: Some substantial improvements, such as SAAS-no-holdout on Keller–Segel, are reported using only eight development seeds. Given the large number of operator–selector combinations evaluated, have these improvements been validated on independent test seeds?
4 comments
Thank you for clarifying these points and for running the additional experiments. My main concern is still the benchmark's ability to separate selectors, since the new results only show differences on generic Burgers at the lowest budget, and even there SAAS does no better than the baseline (7/24 vs. 12/24 at six rounds). I have raised my soundness score accordingly but will keep my overall rating.
Official Comment by Authors
Thank you for the further careful reading and for recognizing the soundness clarifications. We appreciate that your central concern is the benchmark's ability to separate selectors. We have added independent evidence beyond the single Burgers setting and state clearly what it does and does not resolve.
1. Independent headroom beyond generic Burgers.
Using baseline-only development screening on eight original-registry operators, we fixed six target groups, two budgets per operator, both full and common-generic libraries, and five selectors before testing 64 new shared seed blocks. There are 3,520 actual formal searches. The 7,680 budget/library observations reuse exact prefixes and identical Heat-library aliases, so they are not independent replications. This study is a post-review confirmation and is not relabelled as the original registration (Table 48, p. 42).
In the generic library, Allen-Cahn baseline solves 23/64 and 54/64 at three and six rounds, with SAAS 26/64 and 57/64. Burgers baseline solves 14/64 and 39/64 at six and twelve rounds, with SAAS 16/64 and 35/64. NLS is nearer a ceiling, at baseline 55/64 and 63/64. These results extend partial success beyond Burgers while preserving Heat's ceiling and the generic Graveleau/Poisson floors (Table 49, p. 43). We also retain the earlier Burgers result of SAAS 7/24 versus baseline 12/24.
2. Observed differences and statistical limits.
The planned full-library diagnostic supplies an additional selector-sensitive case: higher-order solves only 12/64 and 27/64 on Graveleau at six and twelve rounds, while baseline, held-out and SAAS each solve 64/64 at both budgets. Thus the evidence includes an operator-specific harmful-score effect beyond generic Burgers. These full-library controls are reported separately from the primary generic comparisons (Table 50, p. 44).
We agree that this does not establish broad discrimination among the competitive strong selectors. None of the 39 primary contrasts passes the Holm-adjusted 0.05 threshold. SAAS has small changes of both signs, including a decline on twelve-round Burgers, and its aggregate success change is +0.65 percentage points. We make no claim of general SAAS superiority or equivalence (Table 51, p. 44).
3. Diagnosis and revised claims.
Of baseline's 392 unsuccessful generic budget observations, 387 have no logged solved candidate, four are final selection misses and one loses an earlier solved candidate. Even on Allen-Cahn, Burgers and NLS alone, 131 of 136 failed observations have no logged solved candidate. This localizes a candidate-supply bottleneck in these trajectories, without proving that the grammar cannot represent a better solution. Same-pool selection and cutoff diagnostics are also provided (Tables 53 and 54, p. 45).
We revised the abstract opening and the introduction, discussion and conclusion to emphasize joint evaluation and regime-specific diagnosis. Reliable fine separation among strong selectors remains a stated limitation. The restricted grammar and absence of an LLM proposer also remain explicit. All reported old and new outcomes, including unfavorable ones, are available in the single anonymous supplementary package (Abstract, Secs. 1, 4.4, 5 and 7, Appendix G, pp. 42–45).
Thank you again for the constructive suggestions. They helped us provide a more informative test while keeping the conclusions within its evidence.
Thank you for the additional experiments and for clearly marking the revisions. I think this research direction is valuable, and I appreciate that all outcomes are reported openly. However, my main concern remains. The new results still do not separate the strong selectors, as the authors also acknowledge. In addition, the revisions are now quite extensive, and the main claims have changed considerably. Some parts of the original text still follow the earlier claims, such as the “ranking-to-search gap” in Section 4.
For a future version, I suggest the authors reconsider whether a benchmark is the right framing for the main contribution. The work seems more convincing as a diagnostic protocol and an analysis of where search fails. The diagnostics also rely on analytic reference solutions, so it would help to discuss how they could apply to real search settings without a known solution. I think the paper would benefit from a full revision around these points before resubmission. For this reason, I will keep my overall rating.
Official Comment by Authors
Thank you for your careful review. Your questions helped us clarify the scope, fitting evidence and validation.
W1: Discriminative scope.
Thank you for highlighting the ceiling/floor problem. The original ten-operator success table alone cannot reliably separate strong selectors. We add intermediate generic-budget outcomes and distinguish unavailable candidates from selection failures, reporting success rates and continuous errors together. Unfavorable SAAS results remain. Discrimination is setting-dependent, and small differences remain uncertain (Secs. 4.3–4.4 and 7, Table 23, p. 29, Table 5, p. 8).
W2: Generalization beyond grammar search.
Thank you for emphasizing this limitation. The libraries encode structural prior knowledge, sometimes including exact target families. All tested grammar/proposer shifts remain within a non-LLM framework. We restrict the empirical conclusions accordingly. No external LLM proposer was evaluated, so transfer to LLM-driven search remains unverified (Secs. 3 and 7, Appendix E.2, p. 32).
Q1: Success-level discrimination.
Thank you for requesting evidence beyond precision differences. On generic Burgers, baseline solves 12/24, 21/24 and 22/24 at 6/12/24 rounds. Held-out solves 9/24, 22/24 and 22/24, and SAAS 7/24, 21/24 and 21/24. NLS ceiling and Keller–Segel floor outcomes remain included. Outside the shared gCLM floor, higher-order has 11 terminal selection failures and one earlier solved candidate absent at termination. These reveal headroom and failure modes, but not every difference is statistically resolved (Table 23, p. 29, Table 5, p. 8).
Q2: Choice of controlled proposers.
Thank you for asking about this choice. Fixing the proposer, grammar and fitter isolates outer selection within a restricted hypothesis class. The protocol can be used with another proposer, but these experiments do not establish how LLM proposal distributions, feedback or costs affect M1–M2 behavior. We distinguish the protocol's possible application from unverified empirical generalization (Secs. 3, 5 and 7).
Q3: M1–M2 comparisons with unequal ranking quality.
Thank you for drawing attention to the baseline distinction. Historical held-out M1 0.813 versus baseline 0.817 does not establish improved ranking over that baseline. In the finite-domain generic supplement, raw residual has ΔM1=−0.6247, G=+1.2735, and 0/96 solved versus baseline 55/96. Held-out has ΔM1=+0.0181 (95% interval [0.0064,0.0372]), G=+0.0105 [−0.0130,0.0571], and 51/96 solved. Here G<0 favors the alternative. The M1 contrast subtracts the baseline median from the selector median across fixed operators. Intervals resample 24 shared seed blocks across four fixed operators. Thus worse ranking can accompany worse search, while a modest M1 improvement has no clear active benefit here. This does not establish equivalence or universal M1 irrelevance (Table 4, p. 8, Table 33, p. 33, and Appendix D.2, p. 24).
Q4: Fitting objectives and factorial evidence.
Thank you for raising this concern. Fixing the inner loss can restrict joint gains. The historical 2×4 factorial changes sampling but retains the same loss weights. We added baseline/resampled/higher-order fitters crossed with baseline/held-out/higher-order selectors on generic Burgers and Allen–Cahn: 16 shared seeds, six rounds, 288 searches, with six engineering pilots separate. The two operators share 16 seed blocks, not 32 independent seeds per aggregate cell. Higher-order fitting changes the loss itself. At fixed higher-order selection, higher-order versus baseline fitting gives G=−0.0015 [−0.0172,0.0930], 23/32 versus 20/32 solved, and a ratio of median prediction-point costs of 1.55. None of the 12 primary success comparisons passes Holm-adjusted 0.05. Error intervals are descriptive. Zero median intervals can reflect identical returns, not equivalence. All cells and interactions are retained. This does not cover all objectives or end-to-end SAAS fitting (Table 26, p. 30, Tables 43–45, pp. 40–41).
Q5: Independent Keller–Segel confirmation.
Thank you for requesting independent validation. The eight development seeds were exploratory. A separately fixed comparison uses 32 disjoint seeds 21000–21031. Frozen sources, specification, grammar, fitter/selector implementations, budgets and the 4,096-point error rule match. All 80 saved return errors reproduce exactly. Runtime identity is not assumed. SAAS-no-holdout wins 31 pairs and loses one, with paired median G=−1.2925, 95% interval [−1.4205,−1.1413], and two-sided sign p=1.54×10−8. Both methods solve 32/32 with zero catastrophes. This confirms a local precision effect, not a success-rate or global SAAS gain. The single comparison was fixed before its new-seed outcomes and does not validate every screened combination (Tables 46–47, p. 41).
Thank you again for your careful review and helpful suggestions.