acceptodds
Under review as a conference paper at ICLR 2027

EVALBENCH: Benchmarking Reference-Free PDE Scores from Ranking to Search

Abstract

PDE program search relies on reference-free scores, yet accurate ranking of a fixed candidate bank need not translate into accurate solutions when a score guides search. We introduce EVALBENCH, a benchmark that separates target identifiability, static score–error rank agreement (M1), and the error of the program returned by active search (M2). Core comparisons hold the proposer and fitter fixed; parameter, grammar, and proposer shifts test robustness. Across seven identifiable one-dimensional targets, five combined or resampled scores achieve median Spearman correlations of 0.813–0.817, compared with 0.417 for raw residual. Nevertheless, held-out scoring gives a median paired log10 error ratio of -0.0036 against baseline (95% interval [-0.032, 0.003]), showing no clear active-search gain. A Keller–Segel scale-family control demonstrates why ranking cannot resolve an incomplete public specification. SAAS, an operator-aware rank aggregator, fails its prespecified success criterion; a separate ten-operator development study localizes gains and persistent failures. We provide conditional transfer bounds under distribution coverage and simultaneous risk certificates for fixed selectors using independent calibration seeds from the deployment distribution. EVALBENCH makes specification ambiguity, ranking failure, and search limitations separately measurable, while reporting per-operator errors, catastrophic outcomes, coverage, and evaluation cost.

Primary area:
datasets and benchmarks
Keywords:
PDE benchmarks, reference-free evaluation, program search, model selection, risk calibration, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CNP1OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces EVALBENCH, a benchmark for evaluating reference-free scoring methods in PDE program search. It separates target identifiability, static ranking quality (M1), and active search performance (M2). Through experiments across different PDE problems and scoring methods, the authors find that strong static ranking does not necessarily lead to better search outcomes. The paper also introduces SAAS, an operator-aware score aggregation method. Overall, the work highlights the importance of evaluating scoring methods through their impact on actual search performance rather than static ranking alone.

Critical strengths
  • The paper makes a meaningful distinction between target identifiability, static ranking quality (M1), and active search performance (M2). This provides a useful perspective for evaluating scoring methods beyond their performance on fixed candidate sets.

  • The authors investigate multiple PDE operators, scoring methods, and search configurations, including parameter and proposer shifts. These experiments provide useful observations about the limitations of different scoring methods, even though some conclusions require stronger supporting evidence.

Critical weaknesses
  1. Limited discriminative power of the benchmark. The results in Appendix B.2 raise concerns about whether EVALBENCH can effectively distinguish different scoring methods. Most evaluated methods achieve identical 8/8 success rates on nearly all operators, while challenging cases such as gCLM remain largely unsolved. Table 4 also shows that most selectors have identical overall solved counts. The benchmark appears to provide limited discrimination in terms of task success. This raises a fundamental question about its value as an evaluation benchmark and whether its task difficulty is appropriately designed to reveal meaningful differences between scoring methods.

  2. Limited generalizability to realistic agentic search. The paper motivates EVALBENCH through recent advances in LLM-based scientific program search, but its experiments rely on a restricted library of predefined mathematical expressions and non-LLM proposers. Although several grammar and proposer shifts are evaluated, these remain within the same constrained search framework. It is therefore unclear whether the observed findings would generalize to more realistic LLM-driven program search systems, limiting the practical significance of the benchmark.

Questions for authors
  1. Benchmark Discriminative Power: In Appendix B.2, most methods achieve identical 8/8 success rates on nearly all operators, while challenging cases such as gCLM remain largely unsolved. How do the authors justify the discriminative power of EVALBENCH? Can the benchmark reliably distinguish different scoring methods beyond small differences in continuous error?

  2. Generalizability to Agentic Search: The paper motivates the benchmark through LLM-based scientific program search, but the experiments use a predefined library of mathematical expressions and non-LLM proposers. Why was this setting chosen, and how well do the authors expect the findings to generalize to realistic LLM-driven program search?

  3. Relationship Between M1 and M2: The paper shows that high static ranking quality does not guarantee better active search performance. However, the main M2 comparisons involve scoring methods with similarly high M1 values. Could the authors clarify what the experiments reveal about the relationship between M1 and M2, particularly when ranking quality differs substantially across scoring methods?

  4. Effect of the Fitting Objective: In the core experiments, all candidates are fitted using the baseline four-component loss, while only the outer selection score changes. Could this setup limit the potential benefit of alternative scoring methods? How should the results of the fitting factorial in Appendix B.1 be interpreted?

  5. Reliability of Local Improvements: Some substantial improvements, such as SAAS-no-holdout on Keller–Segel, are reported using only eight development seeds. Given the large number of operator–selector combinations evaluated, have these improvements been validated on independent test seeds?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your careful review. Your questions helped us clarify the scope, fitting evidence and validation.

W1: Discriminative scope.

Thank you for highlighting the ceiling/floor problem. The original ten-operator success table alone cannot reliably separate strong selectors. We add intermediate generic-budget outcomes and distinguish unavailable candidates from selection failures, reporting success rates and continuous errors together. Unfavorable SAAS results remain. Discrimination is setting-dependent, and small differences remain uncertain (Secs. 4.3–4.4 and 7, Table 23, p. 29, Table 5, p. 8).

W2: Generalization beyond grammar search.

Thank you for emphasizing this limitation. The libraries encode structural prior knowledge, sometimes including exact target families. All tested grammar/proposer shifts remain within a non-LLM framework. We restrict the empirical conclusions accordingly. No external LLM proposer was evaluated, so transfer to LLM-driven search remains unverified (Secs. 3 and 7, Appendix E.2, p. 32).

Q1: Success-level discrimination.

Thank you for requesting evidence beyond precision differences. On generic Burgers, baseline solves 12/24, 21/24 and 22/24 at 6/12/24 rounds. Held-out solves 9/24, 22/24 and 22/24, and SAAS 7/24, 21/24 and 21/24. NLS ceiling and Keller–Segel floor outcomes remain included. Outside the shared gCLM floor, higher-order has 11 terminal selection failures and one earlier solved candidate absent at termination. These reveal headroom and failure modes, but not every difference is statistically resolved (Table 23, p. 29, Table 5, p. 8).

Q2: Choice of controlled proposers.

Thank you for asking about this choice. Fixing the proposer, grammar and fitter isolates outer selection within a restricted hypothesis class. The protocol can be used with another proposer, but these experiments do not establish how LLM proposal distributions, feedback or costs affect M1–M2 behavior. We distinguish the protocol's possible application from unverified empirical generalization (Secs. 3, 5 and 7).

Q3: M1–M2 comparisons with unequal ranking quality.

Thank you for drawing attention to the baseline distinction. Historical held-out M1 0.8130.813 versus baseline 0.8170.817 does not establish improved ranking over that baseline. In the finite-domain generic supplement, raw residual has ΔM1=−0.6247\Delta\mathrm{M1}=-0.6247, G=+1.2735G=+1.2735, and 0/96 solved versus baseline 55/96. Held-out has ΔM1=+0.0181\Delta\mathrm{M1}=+0.0181 (95% interval [0.0064,0.0372][0.0064,0.0372]), G=+0.0105G=+0.0105 [−0.0130,0.0571][-0.0130,0.0571], and 51/96 solved. Here G<0G<0 favors the alternative. The M1 contrast subtracts the baseline median from the selector median across fixed operators. Intervals resample 24 shared seed blocks across four fixed operators. Thus worse ranking can accompany worse search, while a modest M1 improvement has no clear active benefit here. This does not establish equivalence or universal M1 irrelevance (Table 4, p. 8, Table 33, p. 33, and Appendix D.2, p. 24).

Q4: Fitting objectives and factorial evidence.

Thank you for raising this concern. Fixing the inner loss can restrict joint gains. The historical 2×42\times4 factorial changes sampling but retains the same loss weights. We added baseline/resampled/higher-order fitters crossed with baseline/held-out/higher-order selectors on generic Burgers and Allen–Cahn: 16 shared seeds, six rounds, 288 searches, with six engineering pilots separate. The two operators share 16 seed blocks, not 32 independent seeds per aggregate cell. Higher-order fitting changes the loss itself. At fixed higher-order selection, higher-order versus baseline fitting gives G=−0.0015G=-0.0015 [−0.0172,0.0930][-0.0172,0.0930], 23/32 versus 20/32 solved, and a ratio of median prediction-point costs of 1.55. None of the 12 primary success comparisons passes Holm-adjusted 0.05. Error intervals are descriptive. Zero median intervals can reflect identical returns, not equivalence. All cells and interactions are retained. This does not cover all objectives or end-to-end SAAS fitting (Table 26, p. 30, Tables 43–45, pp. 40–41).

Q5: Independent Keller–Segel confirmation.

Thank you for requesting independent validation. The eight development seeds were exploratory. A separately fixed comparison uses 32 disjoint seeds 21000–21031. Frozen sources, specification, grammar, fitter/selector implementations, budgets and the 4,096-point error rule match. All 80 saved return errors reproduce exactly. Runtime identity is not assumed. SAAS-no-holdout wins 31 pairs and loses one, with paired median G=−1.2925G=-1.2925, 95% interval [−1.4205,−1.1413][-1.4205,-1.1413], and two-sided sign p=1.54×10−8p=1.54\times10^{-8}. Both methods solve 32/32 with zero catastrophes. This confirms a local precision effect, not a success-rate or global SAAS gain. The single comparison was fixed before its new-seed outcomes and does not validate every screened combination (Tables 46–47, p. 41).

Thank you again for your careful review and helpful suggestions.

Thank you for clarifying these points and for running the additional experiments. My main concern is still the benchmark's ability to separate selectors, since the new results only show differences on generic Burgers at the lowest budget, and even there SAAS does no better than the baseline (7/24 vs. 12/24 at six rounds). I have raised my soundness score accordingly but will keep my overall rating.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the further careful reading and for recognizing the soundness clarifications. We appreciate that your central concern is the benchmark's ability to separate selectors. We have added independent evidence beyond the single Burgers setting and state clearly what it does and does not resolve.

1. Independent headroom beyond generic Burgers.

Using baseline-only development screening on eight original-registry operators, we fixed six target groups, two budgets per operator, both full and common-generic libraries, and five selectors before testing 64 new shared seed blocks. There are 3,520 actual formal searches. The 7,680 budget/library observations reuse exact prefixes and identical Heat-library aliases, so they are not independent replications. This study is a post-review confirmation and is not relabelled as the original registration (Table 48, p. 42).

In the generic library, Allen-Cahn baseline solves 23/64 and 54/64 at three and six rounds, with SAAS 26/64 and 57/64. Burgers baseline solves 14/64 and 39/64 at six and twelve rounds, with SAAS 16/64 and 35/64. NLS is nearer a ceiling, at baseline 55/64 and 63/64. These results extend partial success beyond Burgers while preserving Heat's ceiling and the generic Graveleau/Poisson floors (Table 49, p. 43). We also retain the earlier Burgers result of SAAS 7/24 versus baseline 12/24.

2. Observed differences and statistical limits.

The planned full-library diagnostic supplies an additional selector-sensitive case: higher-order solves only 12/64 and 27/64 on Graveleau at six and twelve rounds, while baseline, held-out and SAAS each solve 64/64 at both budgets. Thus the evidence includes an operator-specific harmful-score effect beyond generic Burgers. These full-library controls are reported separately from the primary generic comparisons (Table 50, p. 44).

We agree that this does not establish broad discrimination among the competitive strong selectors. None of the 39 primary contrasts passes the Holm-adjusted 0.05 threshold. SAAS has small changes of both signs, including a decline on twelve-round Burgers, and its aggregate success change is +0.65 percentage points. We make no claim of general SAAS superiority or equivalence (Table 51, p. 44).

3. Diagnosis and revised claims.

Of baseline's 392 unsuccessful generic budget observations, 387 have no logged solved candidate, four are final selection misses and one loses an earlier solved candidate. Even on Allen-Cahn, Burgers and NLS alone, 131 of 136 failed observations have no logged solved candidate. This localizes a candidate-supply bottleneck in these trajectories, without proving that the grammar cannot represent a better solution. Same-pool selection and cutoff diagnostics are also provided (Tables 53 and 54, p. 45).

We revised the abstract opening and the introduction, discussion and conclusion to emphasize joint evaluation and regime-specific diagnosis. Reliable fine separation among strong selectors remains a stated limitation. The restricted grammar and absence of an LLM proposer also remain explicit. All reported old and new outcomes, including unfavorable ones, are available in the single anonymous supplementary package (Abstract, Secs. 1, 4.4, 5 and 7, Appendix G, pp. 42–45).

Thank you again for the constructive suggestions. They helped us provide a more informative test while keeping the conclusions within its evidence.

Thank you for the additional experiments and for clearly marking the revisions. I think this research direction is valuable, and I appreciate that all outcomes are reported openly. However, my main concern remains. The new results still do not separate the strong selectors, as the authors also acknowledge. In addition, the revisions are now quite extensive, and the main claims have changed considerably. Some parts of the original text still follow the earlier claims, such as the “ranking-to-search gap” in Section 4.

For a future version, I suggest the authors reconsider whether a benchmark is the right framing for the main contribution. The work seems more convincing as a diagnostic protocol and an analysis of where search fails. The diagnostics also rely on analytic reference solutions, so it would help to discuss how they could apply to real search settings without a known solution. I think the paper would benefit from a full revision around these points before resubmission. For this reason, I will keep my overall rating.

Reviewer_LSafOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

This work introduces EVALBENCH, a benchmarking framework designed to evaluate reference-free score functions used for selecting candidate partial differential equation (PDE) solutions during automated program search. The evaluation decouples three distinct questions: whether public specifications identify the target (Stage 0); whether a score's ranking of a fixed candidate bank agrees with held-out error (M1); and the final solution error produced by score-guided search (M2). The authors find that while several score functions achieve high M1 rank correlation (0.813–0.817, compared to 0.417 for the raw residual) and claim this does not reliably translate into clear gains during active search.

Critical strengths
  • The three-stage decomposition framework is thoughtful and methodologically useful.
  • Experimental practices are rigorous, featuring paired same-seed comparisons, per-operator performance metrics, and transparent cost reporting.
  • Negative results are reported openly and thoroughly.
Critical weaknesses
  1. Flawed central comparison: The evaluation of search error is primarily benchmarked against a four-component baseline whose median M1 (0.817, Table 3) matches that of held-out, ensemble, multi-slice, and variation-weighted variants. Demonstrating performance parity among selectors with identical M1 does not prove that a superior M1 fails to yield better M2 outcomes. Furthermore, the raw residual—the low-M1 baseline highlighted in the abstract—appears omitted from the active selector comparisons (Fig. 4, Table 4). Crucially, in instances where M1 drops significantly below baseline (such as higher-order variants on Burgers and porous medium equations), search error degrades correspondingly.
  2. Limited headroom: Across the ten-operator diagnostic suite, nearly all selectors successfully solve 8 out of 8 runs on nine of the operators, while every selector completely fails (0/8) on the gCLM task (Table 7). This severe ceiling and floor effect leaves virtually no dynamic range for differentiability between selectors.
  3. Narrow experimental scope: Although the paper is framed broadly around agentic program search, all empirical tests rely exclusively on grammar-based proposers operating under a single, modest budget (288 candidates per search).
  4. Presentation and organizational issues: The manuscript is dense and difficult to follow. Multiple studies utilize differing numerical specifications for operators sharing the same name, making it hard to trace results back to specific setups. The generic-only grammar is never formally defined, the registered study's 5,200 searches cannot be reconstructed from Section 3, and only a single cell of the 2×42 \times 4 fitter-by-selector factorial is reported. Furthermore, development seeds (24–31) overlap with the registered study's range (24–47), and terminology shifts inconsistently (e.g., "smoothness" versus "curvature," and using "objective" interchangeably with "selector" in Section 4.3).
Questions for authors
  1. Was the raw residual or another selector with a distinctively different M1 score run in active search? If so, how does the per-case search error ratio relate to the per-case M1 difference from the baseline?
  2. Do the atomic families include the exact functional forms of the target equations? If the generic-only grammar excludes them, could you report absolute errors and solved rates under that constraint?
  3. In the diagnostic experiments, the operator router's median performance matches the baseline on nine operators and the higher-order selector on the Keller-Segel system (Tables 6 and 8). Does this accurately reflect its intended routing behavior?
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your detailed review. Your suggestions helped us correct the main comparison and clarify the settings.

W1: Active baseline and the headline comparison.

Thank you for identifying this important gap. Historical held-out M1 0.8130.813 versus active baseline 0.8170.817 does not establish improved ranking over that baseline. We corrected the interpretation with minimal abstract edits and revised Sec. 4.3. Worse ranking can accompany worse search: on porous medium, higher-order has ΔM1=−1.4700\Delta\mathrm{M1}=-1.4700, G=+4.4675G=+4.4675, and 4/16 solved versus baseline 16/16. On Burgers, ΔM1=−0.3563\Delta\mathrm{M1}=-0.3563, G=+2.2028G=+2.2028, although both solve 16/16. The supported claim is limited to the tested same-baseline comparisons (Table 4, p. 8, Table 33, p. 33).

W2: Ceiling/floor effects and headroom.

We agree that the original development success rates provide limited headroom. Generic Burgers baseline success is 12/24, 21/24 and 22/24 at 6/12/24 rounds. At six rounds, held-out and SAAS solve 9/24 and 7/24. These settings provide intermediate outcomes, while NLS remains at a ceiling and Keller–Segel largely at a floor. We report success and continuous-error diagnostics, retain unfavorable outcomes, and do not claim that every task reliably separates strong selectors (Table 23, p. 29, Table 5, p. 8).

W3: Proposer scope and computational budget.

Thank you for emphasizing this limitation. Fixing the proposer and fitter isolates outer selection within a restricted hypothesis class. The supplement covers 6/12/24 search rounds and 25/100/200 inner-iteration limits. The round budgets log 144/288/576 pool events, not that many unique programs. Actual costs are reported separately. No external LLM proposer was evaluated. Secs. 3 and 7 restrict empirical conclusions to the tested grammar-based search (Table 25, p. 30).

W4: Organization and reproducibility.

Thank you for these concrete suggestions. Table 29 maps each study's specification, domain, constants and seeds. The 5,200 registered searches comprise 2,880 main, 560 each no-decoy/restart, 400 generic-only and 800 parameter-shift searches. Development seeds 24–31 overlap main seeds 24–47 and are not independent confirmation. All eight historical fitter–selector cells are reported, and the generic grammar is formalized. Fitting loss denotes inner optimization and selection score denotes outer ranking. Curvature is a scalar component. Smoothness combines baseline and curvature ranks. Their incorrect same-score M1–M2 pairing was removed, and both were audited on 44 banks (Table 29, p. 32, Table 26, p. 30, Appendix E.2, p. 32, Table 42, p. 39).

Q1: Raw residual and direct M1–M2 pairing.

Thank you for requesting this comparison. We added raw-residual paired active comparisons. In the separately specified finite-domain generic supplement, its ΔM1=−0.6247\Delta\mathrm{M1}=-0.6247 and paired median log-error ratio G=+1.2735G=+1.2735, with 0/96 solved versus baseline 55/96. Held-out instead has ΔM1=+0.0181\Delta\mathrm{M1}=+0.0181 (95% interval [0.0064,0.0372][0.0064,0.0372]), G=+0.0105G=+0.0105 [−0.0130,0.0571][-0.0130,0.0571], and 51/96 solved. Here G<0G<0 favors the alternative. The aggregate M1 contrast is the difference between the selector and baseline medians across the fixed operators. Intervals resample 24 shared seed blocks across four fixed operators, not 96 independent seeds. All 424 case-wise cells and both bank origins are retained. The origins reuse M2 (Table 4, p. 8, Table 33, p. 33, Tables 34–36, pp. 34–35).

Q2: Target families and generic-only errors.

Thank you for asking us to make this prior explicit. Many full libraries contain exact reference-family expressions. We provide 31 case records covering the historical and follow-up specifications. Generic programs sum 1–4 sinusoid/gauss/poly/hf/spike atoms, with initialization ranges distinguished from optimization bounds. The four supplementary generic tasks exclude their front/sech/bump witnesses. This does not prove non-approximability. We report all 50 historical generic cells and all 72 supplementary cells. In six-round supplementary Burgers, baseline has median relative error E=0.0483E=0.0483, absolute RMS A=0.0379A=0.0379, and 12/24 solved. Sampled Euclidean L2=64AL_2=64A on 4,096 points. It is not a continuous-domain certificate. Historical and supplementary settings remain separate (Tables 30–32, pp. 32–33, Table 37, p. 36, Tables 38–40, pp. 37–38).

Q3: Intended operator routing.

Thank you for the careful observation. The agreement is intended: higher-order is used only when all three public tags radial, log_diffusion and scale_invariant are present. Otherwise, baseline is used. This routes Keller–Segel to higher-order and the other nine tasks to baseline. All 160 paired returns have identical program hashes and official errors. This is fixed metadata routing, not learned adaptation or additional independent evidence (Table 41, p. 38).

Thank you again for your careful review and constructive suggestions.

Reviewer_EhBLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EVALBENCH separates whether a PDE specification identifies its target, whether a public score ranks a fixed candidate bank correctly, and whether that score produces better solutions during search. The experiments compare scores under shared search machinery and several shifts, while testing an operator-aware rank aggregator, SAAS. The main result is that strong static correlations coexist with little aggregate improvement over the baseline in active search.

Critical strengths

Overall really liked this paper! Readable. And the distinction between specification ambiguity, scoring errors, and search limitations is valuable, and the Keller-Segel scale control makes the first issue concrete. I appreciate evaluating the actual returned program, reporting failures and evaluation costs, and retaining unfavorable SAAS results. The fitting intervention and reference-based refitting diagnostic also help investigate whether changing the selector can plausibly fix the observed failures.

Critical weaknesses

The ranking-to-search abstract/headline result needs a little love. Table 3 shows held-out scoring at median M1 0.813 versus 0.817 for the four-component baseline used in M2. The improvement over raw residual therefore does not demonstrate that improved ranking over the active baseline failed to improve search. Scope is also limited by the restricted grammar and easy-or-unsolved pattern in the ten-operator diagnostic: baseline solves every non-gCLM run, while no selector solves gCLM. This leaves limited evidence about cases where selection can change success, and failure of finite-budget reference fitting does not establish an approximation-capacity limit. Finally, the conditional theory does not by itself verify the coverage or PDE stability assumptions for the implemented scores. All fixable and mostly fixable just by wording/presentation alone.

Questions for authors
  1. Can you directly pair each selector's change in M1 relative to the four-component baseline with its M2 outcome, including behavior near the selection cutoff?
  2. Can you report best-pool error and terminal selection gaps for the completed runs, separating unavailable good candidates from good candidates that were discarded?
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for recognizing the diagnostic framework and transparent reporting. Your careful comments helped us correct the comparison and clarify the limits of the evidence.

W1: Ranking-to-search inference.

Thank you for identifying the baseline mismatch. Historical held-out M1 is 0.8130.813, versus 0.8170.817 for the active four-component baseline, so this comparison does not establish improved ranking over that baseline. We clarified the baseline with minimal abstract edits and corrected Sec. 4.3. Matched supplementary comparisons do not establish a clear aggregate gain from held-out scoring here, equivalence, or a general failure of M1 (Table 4, p. 8).

W2: Headroom and restricted grammar.

Thank you for highlighting this limitation. The generic grammar is explicitly sinusoid/gauss/poly/hf/spike, tested at 6/12/24 rounds. Burgers baseline success is 12/24, 21/24 and 22/24. The NLS ceiling and largely unsolved Keller–Segel remain visible. We retain every operator and selector and describe discrimination as setting-dependent. Secs. 3 and 7 limit the empirical conclusions to the tested grammar-based search. No external LLM proposer was evaluated (Table 23, p. 29).

W3: Finite-budget fitting and approximation capacity.

We appreciate this correction. Failure on 64 tested structure–budget pairs does not establish an approximation-capacity limit. Secs. 4.4 and 7 now restrict the interpretation to the tested structures, optimizers and budgets. Reranking a fixed pool cannot create an absent solved candidate, while changing earlier selection can change later proposals. Better optimization or other structures remain possible (Appendix B.1, p. 15).

W4: Conditional theory and implemented scores.

Thank you for requesting this distinction. Coverage/density-ratio and calibration conditions, bounded fresh validation losses and PDE stability estimates have not been verified for every implemented score. We now state these requirements explicitly. SAAS fixed-pool aggregation stability does not establish correct hidden-error rankings, and the denser-grid audit is a sensitivity check, not a coverage, convergence or stability proof (Secs. 2 and 7, Appendix A.4, p. 15).

Q1: Paired selectors and selection-cutoff behavior.

Thank you for requesting direct pairing. In the separately specified finite-domain supplement, four generic tasks at six rounds give held-out ΔM1=+0.0181\Delta\mathrm{M1}=+0.0181 (95% interval [0.0064,0.0372][0.0064,0.0372]), paired median log-error ratio G=+0.0105G=+0.0105 [−0.0130,0.0571][-0.0130,0.0571], and 51/96 solved versus baseline 55/96. Here G<0G<0 favors the selector. The aggregate M1 contrast is the difference between the selector and baseline medians across the fixed operators. Intervals resample 24 shared seed blocks across fixed operators. All selectors, budgets and 424 case-wise cells are retained. Identical 24-candidate pools are reranked with eight retained. On final pools from baseline-source trajectories at six rounds, held-out's cutoff-correlation gain is +0.0069+0.0069 [−0.1208,0.1387][-0.1208,0.1387], with 95/96 defined pairs. Replay is observational and does not replace active M2 (Table 4, p. 8, Tables 13–16, pp. 25–26, and Table 19, p. 27).

Q2: Best-pool errors and discarded candidates.

Thank you for this diagnostic suggestion. The 5,512-search matrix now has complete history labels, with official returns unchanged. We compute EpoolE_{\rm pool}, EhistE_{\rm hist} and Eret−EpoolE_{\rm ret}-E_{\rm pool} per run before aggregation. Across 160 replication runs per selector, baseline median pool error is 2.93×10−72.93\times10^{-7} and median terminal gap is 5.39×10−85.39\times10^{-8}. Baseline, held-out and SAAS each solve 144/160. Their failures share the gCLM unavailable-candidate floor. Higher-order additionally has 11 terminal selection failures and one earlier solved candidate absent at termination. Four disjoint categories distinguish solved returns, unsolved returns with a solved final-pool candidate, earlier solutions absent at termination, and no recorded solution, using E<0.05E<0.05. Earlier absence includes immediate rejection (Table 5, p. 8, Tables 21–22, p. 28, Table 24, p. 29).

Thank you again for your careful review and constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.