acceptodds
Under review as a conference paper at ICLR 2027

RIGOR: Representation-Instrumented Generation with Outcome-Gated Refinement for Budgeted Discovery

Abstract

AI scientist systems built on large language models (LLMs) can generate hypotheses and design experiments, but evaluating a candidate can be costly in time, computation, or materials, so the choice of the next experiment matters. We present Representation-Instrumented Generation with Outcome-Gated Refinement (RIGOR), a framework for budgeted discovery that describes candidate designs with an explicit rubric of interpretable dimensions, which record how designs differ rather than how good they are. Measuring a candidate on these dimensions gives its rubric-based representation, which hypothesis generation and outcome prediction share. An LLM outcome predictor estimates each candidate's outcome and uncertainty from its complete design, its rubric-based representation, and previous outcomes, and an upper confidence bound (UCB) rule selects the next experiment. As evidence accumulates, RIGOR may propose new rubric dimensions, but it activates a dimension only if it reduces prediction error by at least 5% on the next five outcomes, with every forecast recorded before the outcome is observed. Across four scientific-design tasks and four MLS-Bench algorithm-design tasks, with Opus 4.7 and GPT-5.5, a budget of 24 evaluations, and three runs per setting, RIGOR outperforms existing baselines: it achieves the highest mean final best score in 11 of 16 model–task settings, the highest mean evaluated outcome in all 16, and the lowest within-run variation over the second half of the budget in 15. Its prospective gate admits only 16 of 86 proposed dimensions, each of which improves prediction of subsequent outcomes.

Primary area:
optimization
Keywords:
AI scientist, AI for discovery, Autoresearch

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_amhFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RIGOR, a framework for budgeted scientific discovery that integrates rubric-guided hypothesis generation, LLM-based outcome prediction, and uncertainty-aware experiment selection. A prospective refinement mechanism introduces new representation dimensions only when they improve prediction accuracy on subsequent outcomes. The authors evaluate RIGOR on four scientific-design tasks and four algorithm-design tasks using two foundation models. Results show improvements over PiEvo and ReElicit in final discovery performance, average evaluated outcome, and search stability.

Critical strengths
  1. The prospective refinement mechanism is a well-motivated contribution. By validating proposed representation dimensions against future outcomes rather than relying solely on historical fit, the framework introduces a useful safeguard against overfitting during iterative discovery. The separation between dimension proposal and activation is particularly well designed.

  2. The experimental evaluation is relatively thorough, covering eight tasks, two foundation models, and multiple performance metrics. The improvements in mean evaluated outcomes and search stability are encouraging, and the additional analyses of prediction accuracy, uncertainty calibration, and refinement decisions strengthen the empirical results.

Critical weaknesses
  1. The contributions of individual components are not sufficiently isolated. RIGOR combines rubric-guided generation, LLM-based outcome prediction, UCB selection, and prospective refinement, but comparisons against PiEvo and ReElicit do not establish which components drive the improvements. In particular, the prospective gate is validated through prediction MAE rather than downstream discovery performance. It remains unclear whether activating new dimensions actually improves search outcomes or whether most gains come from the initial rubric and selection mechanism. This limits the evidence supporting the central methodological contribution.

  2. The evaluation provides limited insight into hypothesis-space exploration. While RIGOR achieves strong average outcomes, this does not necessarily imply better exploration or coverage of the design space. Related work such as HypoSpace: Evaluating LLM Creativity as Set-Valued Hypothesis Generators under Underdetermination (ICML 2026) highlights the importance of validity, uniqueness, and recovery in evaluating hypothesis generation, but is not discussed in the paper. A more explicit analysis of candidate diversity, coverage, and exploration–exploitation trade-offs would help clarify whether RIGOR improves exploration or primarily concentrates on promising design families.

  3. The statistical robustness and computational efficiency require further validation. The experiments use only three runs per setting and a fixed budget of 24 evaluations, with relatively small performance differences in several tasks. Moreover, RIGOR introduces additional LLM inference costs through repeated rubric measurements, outcome predictions, and prospective validation. Although the evaluation budgets are matched, the total computational costs are not systematically compared. This makes it difficult to assess whether the improvements remain significant and cost-effective under larger evaluation budgets.

Questions for authors
  1. Could the authors provide component-level ablations, particularly demonstrating whether prospective refinement improves downstream discovery performance beyond prediction MAE?

  2. Does RIGOR improve hypothesis diversity and exploration coverage, or primarily exploit previously successful design families?

  3. How robust are the improvements across additional seeds and longer evaluation budgets, and how does RIGOR compare with baselines in total computational cost?

I am open to increasing my rating if these concerns are adequately addressed.

9 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer amhF (1/7)

W1. The contributions of individual components are not sufficiently isolated. RIGOR combines rubric-guided generation, LLM-based outcome prediction, UCB selection, and prospective refinement, but comparisons against PiEvo and ReElicit do not establish which components drive the improvements. In particular, the prospective gate is validated through prediction MAE rather than downstream discovery performance. It remains unclear whether activating new dimensions actually improves search outcomes or whether most gains come from the initial rubric and selection mechanism. This limits the evidence supporting the central methodological contribution.

Re:

(1) "comparisons against PiEvo and ReElicit do not establish which components drive the improvements."

UCB selection and the rubric drive the improvements. Following the reviewer's suggestion, we tested each component with a controlled comparison. We ran the ablation studies with GPT-5.5 and compared each new run with one RIGOR run from the paper.

  • UCB selection picks better candidates than a random pick. Specifically, we scored all 3,081 candidates that RIGOR proposed in the 24 scientific-design runs, including the candidates that RIGOR did not select. The scientific-design tasks score each design with a pretrained evaluator, so we can score the unselected candidates after the runs. The results show that the candidates that UCB selects score 3.9% higher than a random pick from the same round (95% CI 2.1% to 5.6%), and score higher in each of the 24 runs.
  • The predictor forecasts outcomes better than the median of the earlier outcomes. The predictor has a lower MAE than this median in 46 of 48 runs (Section 4.4).
  • Both kinds of candidates improve the best design so far. Of the 375 improvements to the best design so far, 202 come from single-change children, which change one decision of an evaluated design, and 173 come from independently specified candidates (Section 4.4).
  • The rubric plays an important role in RIGOR's search performance. Specifically, we conducted an ablation study that removes the rubric from RIGOR: no rubric judge, no rubric in generation or prediction, and no refinement. The results show that, on scaling-law modeling and online bandits, RIGOR without the rubric has a lower final best score in all 6 runs, by 0.4% to 6.7%.

Means over three runs:

TaskFinal best score: RIGORFinal best score: without the rubricMean evaluated outcome QQ: RIGORMean evaluated outcome QQ: without the rubric
Scaling law0.60530.57690.56120.5358
Online bandit0.97510.95460.94340.9137
AuthorsOpenReview ↗

Authors' Response to Reviewer amhF (2/7)

W1. The contributions of individual components are not sufficiently isolated. RIGOR combines rubric-guided generation, LLM-based outcome prediction, UCB selection, and prospective refinement, but comparisons against PiEvo and ReElicit do not establish which components drive the improvements. In particular, the prospective gate is validated through prediction MAE rather than downstream discovery performance. It remains unclear whether activating new dimensions actually improves search outcomes or whether most gains come from the initial rubric and selection mechanism. This limits the evidence supporting the central methodological contribution.

Re:

(2) "the prospective gate is validated through prediction MAE rather than downstream discovery performance. It remains unclear whether activating new dimensions actually improves search outcomes or whether most gains come from the initial rubric and selection mechanism."

Activating new dimensions through the gate makes the search more stable and more efficient, and it keeps the rubric interpretable. Specifically, we conducted two ablations on TMC and SuperCon, the two tasks on which RIGOR adopts new dimensions in 5 of its 6 runs. RIGOR with a fixed rubric keeps the initial rubric for the whole run. RIGOR without the gate adopts every proposed dimension immediately. Each new run starts from the initial rubric of one RIGOR run and is compared with that RIGOR run. The results show that:

  • Without the gate, the search is less stable and less efficient. The outcomes of evaluations 13–24 vary more without the gate on both tasks. Their standard deviation is 17.04 on TMC and 2.67 on SuperCon without the gate, against 16.21 and 0.59 with the gate. Without the gate, the mean evaluated outcome is lower than RIGOR's in 5 of the 6 runs. On TMC, two runs of RIGOR without the gate find the best design at evaluations 17 and 23, against evaluations 9 and 12 for two runs with the gate.
  • Compared with a fixed rubric, RIGOR evaluates better designs and is more stable. RIGOR has a higher mean evaluated outcome than RIGOR with a fixed rubric in 4 of the 6 runs, and RIGOR's outcomes of evaluations 13–24 vary less in 4 of the 6 runs.

Means over three runs. QQ is the mean evaluated outcome, and SlateS_{\rm late} is the standard deviation of the outcomes of evaluations 13–24 (Section 4.3):

StatisticRIGORFixed rubricNo gate
TMC: mean evaluated outcome QQ451.87457.70449.39
TMC: late within-run variation SlateS_{\rm late}16.2116.2017.04
SuperCon: mean evaluated outcome QQ109.42107.62107.70
SuperCon: late within-run variation SlateS_{\rm late}0.590.802.67

The gate keeps the rubric interpretable. The gate adopts a proposed dimension only after the dimension improves forecasts of later outcomes, and the gate needs no additional experiments (Section 3.5):

  • In the paper's runs, the gate adopts 16 of 86 proposed dimensions. Each adopted dimension lowers the MAE on the five outcomes after its proposal by 5.38% to 41.80% (Section 4.5).
  • Of the 70 rejected dimensions, 52 increase prediction error. A gate that decided after two outcomes would adopt 14 dimensions whose gain does not persist over five outcomes (Section 4.5).
  • On TMC and SuperCon, RIGOR without the gate adds 12 dimensions to the rubric, and RIGOR adopts the 5 dimensions that pass the gate.
  • The adopted dimensions record which design decisions matter and separate better designs from worse ones. On SuperCon, Hg-1223 compositions average 109.9 K and Hg-1234 compositions average 104.2 K. On TMC, complexes with two BIFZEX ligands average 484.4, and complexes with two GUQBUQ ligands average 440.4 (BIFZEX and GUQBUQ are two of the ligands that a TMC design can use).

The revised paper includes all of these ablations in Section 4.6 and Appendix H (Tables 14 and 15).

AuthorsOpenReview ↗

Authors' Response to Reviewer amhF (3/7)

W2. The evaluation provides limited insight into hypothesis-space exploration. While RIGOR achieves strong average outcomes, this does not necessarily imply better exploration or coverage of the design space. Related work such as HypoSpace: Evaluating LLM Creativity as Set-Valued Hypothesis Generators under Underdetermination (ICML 2026) highlights the importance of validity, uniqueness, and recovery in evaluating hypothesis generation, but is not discussed in the paper. A more explicit analysis of candidate diversity, coverage, and exploration–exploitation trade-offs would help clarify whether RIGOR improves exploration or primarily concentrates on promising design families.

Re:

"A more explicit analysis of candidate diversity, coverage, and exploration–exploitation trade-offs would help clarify whether RIGOR improves exploration or primarily concentrates on promising design families."

RIGOR does both: it explores early and concentrates later experiments on the most promising design families (Sections 4.3 and 4.4). RIGOR's exploration weight βt=β0(1−t/T)\beta_t=\beta_0(1-t/T) sets this order.

  • Exploration. We say that uncertainty changes a selection when UCB selects a candidate whose point forecast is not the highest. Uncertainty changes 32.0% of the selections in evaluations 1–8, 26.6% in evaluations 9–16, and 6.0% in evaluations 17–24. Independently specified candidates produce 173 of the 375 improvements to the best design so far.
  • Concentration. In 44 of the 48 RIGOR runs, the outcomes in the second half of the budget vary less than the outcomes in the first half. Of RIGOR's 1,152 evaluated designs, 686 are single-change children of evaluated designs.

We thank the reviewer for pointing us to HypoSpace. Under HypoSpace's criteria, RIGOR concentrates more than PiEvo, never repeats an exact design, and recovers the largest share of near-optimal families on TMC. In all 8 scientific-design settings, RIGOR also has the highest mean evaluated outcome. For this analysis, we adapted HypoSpace's uniqueness and recovery criteria to the designs evaluated in all 72 scientific-design runs of the three methods.

  • A family groups designs of the same kind: a cell of the grid that divides each of the four parameter ranges into three equal bins (Nanohelix), a Bemis–Murcko scaffold (ChEMBL), an element set (SuperCon), and the donor-atom pairs of the neutral and anionic ligands (TMC).
  • A near-optimal family is a family in which any of the nine runs that the three methods have in a setting evaluates a design within 5% of the best outcome observed in that setting. Recovery is the share of near-optimal families that a run evaluates at least once.

The results show that:

CriterionResult
ConcentrationRIGOR evaluates fewer distinct families than PiEvo in 8 of 8 settings. In evaluations 9–24, RIGOR places a higher share of its evaluations in the family of the best design so far than PiEvo does, in 8 of 8 settings. Against ReElicit: fewer families in 5 of 8 settings, and a higher share in 6 of 8
UniquenessRIGOR never repeats an exact design. In one ChEMBL run of ReElicit, 13 of the 24 evaluations repeat an earlier design exactly
Recovery of near-optimal familiesOn TMC, RIGOR's recovery is 0.58 with both models, against 0.08–0.33 for PiEvo and 0.17–0.25 for ReElicit

The revised paper discusses HypoSpace in Section 2, reports the HypoSpace analysis in Appendix K, and states in Section 5 how RIGOR balances exploration and concentration over the budget.

Reviewer_w7yQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes RIGOR, an LLM-guided framework for budgeted scientific and algorithmic discovery that integrates rubric-based design representation, outcome prediction, uncertainty-aware candidate selection, and prospective rubric refinement. RIGOR uses a rubric judge to characterize candidate designs, an outcome predictor to estimate their potential performance and uncertainty, and an upper confidence bound (UCB) strategy to guide candidate selection. It also introduces a prospective refinement mechanism that adopts new rubric dimensions when they improve prediction accuracy on a predefined window of future outcomes. The authors evaluate RIGOR on eight scientific and algorithmic tasks using two LLMs, comparing it with PiEvo and ReElicit under a budget of 24 external evaluations per run. Results show that RIGOR achieves the highest mean final best score in 11 of 16 model-task settings, the highest mean evaluated outcome in all 16 settings, and the lowest late-stage within-run variation in 15 settings. While these findings suggest that RIGOR can improve outcome quality and consistency under limited evaluation budgets, the individual contributions of its components and its overall computational overhead remain insufficiently established.

Critical strengths
  1. Systematic framework design. The paper integrates rubric-based representation, outcome prediction, uncertainty-aware candidate selection, and prospective rubric refinement into a unified iterative discovery framework, establishing a coherent pipeline for candidate generation, evaluation, and selection. This design offers a promising approach to LLM-driven scientific and algorithmic discovery under limited evaluation budgets.
  2. Empirical evaluation across diverse tasks and models. The paper evaluates RIGOR on eight scientific design and algorithmic optimization tasks using two LLMs, with comparisons against PiEvo and ReElicit. The results show that RIGOR achieves favorable mean evaluation outcomes across multiple task settings and exhibits lower late-stage variation in most settings. These findings provide preliminary empirical evidence of the framework’s applicability across different tasks.
Critical weaknesses
  1. Insufficient evidence for the contributions of individual components. RIGOR combines rubric-based representation, outcome prediction, UCB-based selection, and prospective rubric refinement, but the paper does not provide sufficient ablation studies to isolate the contribution of each component. In particular, the outcome predictor receives both the full design text and the rubric representation, making it unclear whether the rubric representation provides substantial predictive value beyond the textual input. Consequently, the empirical results do not adequately establish the necessity or effectiveness of the proposed design choices.
  2. Incomplete accounting of computational costs. The evaluation uses the same number of external evaluations for each method, but this does not ensure a fair comparison of overall computational costs. RIGOR involves additional LLM calls for rubric evaluation, outcome prediction, and rubric refinement, yet the paper does not provide a comprehensive comparison of LLM call counts, token consumption, or monetary costs across methods. Without such an analysis, it is difficult to determine whether the reported performance gains justify the additional computational overhead.
  3. Limited statistical evidence for the reported performance differences. Each model-task-method setting is evaluated in only three independent runs, and the comparisons rely primarily on mean performance and rankings across settings. This limited number of repetitions makes the results sensitive to run-to-run variability and provides insufficient evidence to determine whether the observed differences between methods are statistically reliable. More extensive repeated experiments or appropriate statistical analyses are needed to support the paper’s comparative claims.
Questions for authors
  1. Contribution of Individual Components: Could the authors provide additional ablation results to clarify the individual contributions of rubric-based representation, outcome prediction, UCB-based selection, and prospective rubric refinement? In particular, how much predictive value does the rubric representation provide beyond the full design text when both are used as inputs to the outcome predictor? This clarification would help establish whether the proposed components are individually necessary and whether the observed improvements can be attributed to the specific design choices.
  2. Computational Overhead: Could the authors provide a more comprehensive comparison of computational costs across methods, including the number of LLM calls, token consumption, and estimated monetary costs? Since RIGOR introduces additional LLM calls for rubric evaluation, outcome prediction, and rubric refinement, such a comparison would help determine whether its performance improvements justify the additional computational overhead.
  3. Statistical Reliability of the Results: Given that each model-task-method setting is evaluated in only three independent runs, could the authors provide additional repeated experiments or statistical analyses to establish the reliability of the reported performance differences? In particular, are the observed improvements statistically significant, or could they be explained by run-to-run variability? This clarification is important for assessing the strength of the paper’s comparative claims.
  4. Availability of the Final Rubrics: I could not find the final versions of the evolved rubrics in either the main paper or the appendix. Could the authors share these rubrics, ideally including the initial rubrics and their final evolved versions? Access to these artifacts would help readers understand how the rubrics evolve during the process and better assess the practical reproducibility of the proposed method.
8 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer w7yQ (1/7)

W1. Insufficient evidence for the contributions of individual components. RIGOR combines rubric-based representation, outcome prediction, UCB-based selection, and prospective rubric refinement, but the paper does not provide sufficient ablation studies to isolate the contribution of each component. In particular, the outcome predictor receives both the full design text and the rubric representation, making it unclear whether the rubric representation provides substantial predictive value beyond the textual input. Consequently, the empirical results do not adequately establish the necessity or effectiveness of the proposed design choices.

Re:

(1) "the paper does not provide sufficient ablation studies to isolate the contribution of each component."

Each component contributes. Specifically, we followed the reviewer's suggestion and isolated each component with a controlled comparison (the full table is in our answer to Question 1). The results show that:

  • UCB selection picks better candidates than a random pick from the same round. The selected candidates score 3.9% higher than a random pick.
  • The rubric plays an important role in RIGOR's search performance. On scaling-law modeling and online bandits, RIGOR without the rubric has a lower final best score in all 6 runs.
  • The gate makes the search more stable and keeps the rubric interpretable. The gate tests every proposed dimension before RIGOR adopts it. Without the gate, the outcomes of evaluations 13–24 vary more on both TMC and SuperCon. The standard deviation of these outcomes is 17.04 on TMC and 2.67 on SuperCon without the gate, against 16.21 and 0.59 with the gate. Without the gate, the mean evaluated outcome is lower in 5 of the 6 runs. Without the gate, 12 dimensions enter the rubric, against 5 with the gate.

(2) "the outcome predictor receives both the full design text and the rubric representation, making it unclear whether the rubric representation provides substantial predictive value beyond the textual input."

The rubric provides value beyond the design text: without the rubric, RIGOR's final best score is lower. Specifically, we conducted an ablation study that removes the rubric from RIGOR. In RIGOR without the rubric, the predictor reads only the complete design text and the history. The results show that, on scaling-law modeling and online bandits, RIGOR without the rubric has a lower final best score in all 6 runs, by 0.4% to 6.7%.

AuthorsOpenReview ↗

Authors' Response to Reviewer w7yQ (2/7)

W2. Incomplete accounting of computational costs. [...] Without such an analysis, it is difficult to determine whether the reported performance gains justify the additional computational overhead.

Re:

"The evaluation uses the same number of external evaluations for each method, but this does not ensure a fair comparison of overall computational costs. [...] the paper does not provide a comprehensive comparison of LLM call counts, token consumption, or monetary costs across methods. Without such an analysis, it is difficult to determine whether the reported performance gains justify the additional computational overhead."

RIGOR is designed for discovery in which the experiment, not the model call, is the dominant cost. This is why the paper compares all methods at the same number of experiments. RIGOR's model calls cost USD 2.76 to 12.09 per experiment, and they save experiments. In 11 of 16 settings, RIGOR reaches PiEvo's mean final best score with fewer experiments than PiEvo, by a median of 15 experiments.

The paper states throughout that each evaluation is expensive and that all methods share the same evaluation budget:

  • evaluating a candidate "can be costly in time, computation, or materials, so the choice of the next experiment matters" (abstract);
  • the goal is to maximize the best observed outcome within an evaluation budget TT (Section 3.1), and all methods share the budget of 24 evaluations (Section 4.1);
  • RIGOR "targets settings in which each evaluation is expensive; it uses LLM calls on measurement, prediction, and prospective testing before every experiment to improve efficiency and reduce the cost of running real experiments" (Section 5).

We thank the reviewer for the suggestion to report model usage. The table below gives the average model calls, tokens, and cost per run for each method. The averages count every model call of each method. We compute the cost with the same per-token prices for all methods. On MLS-Bench, the rows of RIGOR and PiEvo include the model calls of the Executor and the fidelity audit, which implement and check each selected design.

BenchmarkModelMethodModel callsInput tokens (millions)Output tokens (millions)Cost (USD)
Scientific designOpus 4.7RIGOR8077.611.4080.47
Scientific designOpus 4.7PiEvo941.30†0.13†10.85†
Scientific designOpus 4.7ReElicit4821.600.2916.73
Scientific designGPT-5.5RIGOR7485.431.9683.07
Scientific designGPT-5.5PiEvo870.64†0.07†5.82†
Scientific designGPT-5.5ReElicit3770.820.7928.80
MLS-BenchOpus 4.7RIGOR95217.461.53138.15
MLS-BenchOpus 4.7PiEvo39612.230.9593.31
MLS-BenchOpus 4.7ReElicit‡3818.020.8166.33
MLS-BenchGPT-5.5RIGOR93514.855.87241.64
MLS-BenchGPT-5.5PiEvo45811.531.5677.24
MLS-BenchGPT-5.5ReElicit‡3886.241.2365.75

† On the scientific-design tasks, PiEvo's call counts cover all runs. PiEvo's tokens and cost are means over 4 runs with Opus 4.7 and 1 run with GPT-5.5.

‡ On MLS-Bench, ReElicit's rows count only ReElicit's own calls. ReElicit's rows do not include ReElicit's Executor and audit calls (176 per run with Opus 4.7 and 236 with GPT-5.5), so ReElicit's rows are lower bounds on ReElicit's usage.

  • RIGOR's model calls cost between USD 2.76 (Nanohelix, Opus 4.7) and USD 12.09 (scaling-law modeling, GPT-5.5) per experiment. The experiments that RIGOR targets, such as synthesis and characterization or long simulations, cost far more than these model calls.
  • RIGOR's model calls save experiments. We count how many experiments a method needs before its best-so-far score, averaged over its three runs, reaches PiEvo's mean final best score. In 11 of 16 settings, RIGOR reaches PiEvo's mean final best score with fewer experiments than PiEvo, by a median of 15 experiments. In 10 of these 11 settings, the experiments saved are worth more than RIGOR's entire model cost for all 24 experiments if one experiment costs more than USD 37.29. This holds even if PiEvo's model calls are counted as free. In the eleventh setting, scaling-law modeling with Opus 4.7, RIGOR's model cost is lower than PiEvo's (USD 140.26 per run against USD 188.21 per run). RIGOR also reaches ReElicit's mean final best score with fewer experiments than ReElicit in 13 of 16 settings, by a median of 12 experiments.

The revised paper includes this table and the analysis of the experiments saved in Appendix J.

AuthorsOpenReview ↗

Authors' Response to Reviewer w7yQ (3/7)

W3. Limited statistical evidence for the reported performance differences. Each model-task-method setting is evaluated in only three independent runs, and the comparisons rely primarily on mean performance and rankings across settings. This limited number of repetitions makes the results sensitive to run-to-run variability and provides insufficient evidence to determine whether the observed differences between methods are statistically reliable. More extensive repeated experiments or appropriate statistical analyses are needed to support the paper's comparative claims.

Re:

"provides insufficient evidence to determine whether the observed differences between methods are statistically reliable."

The differences are statistically reliable. Specifically, we conducted an exact permutation test over the 16 settings. In the test, each run is one observation. The results show that RIGOR's mean evaluated outcome is significantly higher than PiEvo's (Holm-corrected p=3.8×10−18p=3.8\times10^{-18}) and ReElicit's (p=6.2×10−14p=6.2\times10^{-14}). Its final best score is also significantly higher than PiEvo's (p=0.042p=0.042) and ReElicit's (p=2.7×10−4p=2.7\times10^{-4}).

In each of the 16 settings, RIGOR and a baseline have three runs each. The test assigns the method labels to these six runs at random and checks how often the RIGOR runs get at least as many wins as the RIGOR runs actually have:

  • Statistic. In each setting, UU counts the pairs of a RIGOR run and a baseline run in which the RIGOR run is better (ties count 1/21/2, so 0≤U≤90\le U\le 9). UtotU_{\rm tot} is the sum of UU over the 16 settings.
  • Null distribution. Under the null hypothesis, method labels are exchangeable within each setting, so each setting has 20 equally likely label assignments. The exact distribution of UtotU_{\rm tot} is the convolution of the per-setting distributions of UU.
  • Reported values. We report one-sided pp-values with Holm correction over the two baselines. PS is the probability that a RIGOR run beats a baseline run of the same setting, PS=Utot/(16×9)\mathrm{PS}=U_{\rm tot}/(16\times9).
Metricvs. PiEvovs. ReElicit
Final best scorePS 0.61, p=0.042p=0.042; RIGOR has the higher mean in 12 of 16 settingsPS 0.73, p=2.7×10−4p=2.7\times10^{-4}; RIGOR has the higher mean in 13 of 16 settings
Mean evaluated outcome over all 24 evaluationsPS 0.98, p=3.8×10−18p=3.8\times10^{-18}; RIGOR has the higher mean in 16 of 16 settingsPS 0.93, p=6.2×10−14p=6.2\times10^{-14}; RIGOR has the higher mean in 16 of 16 settings

The revised paper includes the test and this table in Section 4.2, with the details in Appendix I.

Reviewer_nijqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to build on the PiEvo body of work for using AI to accelerate hypothesis-generation. In particular, it proposes a framework that uses a rubric-based representation for both hypothesis generation and outcome prediction, together with a variety of other supporting mechanisms, such as rubric augmentation and an upper confidence bound rule for selecting the next experiment.

Critical strengths

Building on the PiEvo body of work and benchmark by using a unified format for outcome prediction and hypothesis generation within a closed experimental loop is a useful direction.

Critical weaknesses

Overall, the construction of RIGOR seems quite ad-hoc, which is a significant weakness both for evaluating the quality of the results as well as understanding how the findings can be used in future studies. a) The presentation of the RIGOR framework emphasizes multiple components/ideas that each have different philosophical justifications (e.g. rubric-based representation shared by hypothesis generation and outcome prediction, UCB selection rule, rubric augmentation, budgeted discovery). However, the global research takeaway from the union of these components is unclear. b) It does not seem surprising that hypothesis generation and outcome prediction could have the same representation because they are deeply connected processes within the scientific method. The paper does not explain how the rubric structure that embeds this unified representation is novel. c) There are no ablation studies for any of the components in this framework. Without these studies, it is impossible to judge which aspects of RIGOR are important for hypothesis generation and in what manner. For example, several mechanisms within RIGOR, such as the "single-change children" proposed by the hypothesis agent and the outputs of the executor are treated as details, but without the ablation, it's not clear if they are more or less relevant than the main emphasis of the rubric-based representation. d) It is not clear what the next steps for this framework are--what other benchmarks it may be relevant to and how the overall principles can be integrated into advancing AI4science.

The relationship between the current study and the cited papers used to structure the evaluation is not explained. a) The paper seems to target an improvement on the PiEvo body of work and benchmarks. However, it adds several algorithm-design tasks from MLS-Bench without explaining why they are relevant for inclusion and how they add to our understanding of the framework's strengths. b) The relevance of the ReElicit framework as a comparison is not explained. The ReElicit paper broadly aimed to study how system prompts can be tuned in LLM's; it is not clear how it is relevant specifically to AI4science frameworks. There is some discussion related to semantic feature learning and Bayesian optimization in the intro, but how this relates to the construction of RIGOR is not explained. As such, it is hard to evaluate how important it is that RIGOR improves over ReElicit.

Questions for authors

What is the philosophical basis /motivation for each of the components of RIGOR both individually and when combined into one framework?

How do PiEvo and MLS-Bench benchmarks complement each other and what are the next benchmarks that this framework can be tested on?

How can ideas from RIGOR be adapted to advance AI4science as a field?

7 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer nijq (1/6)

W1. Overall, the construction of RIGOR seems quite ad-hoc, [...] d) It is not clear what the next steps for this framework are--what other benchmarks it may be relevant to and how the overall principles can be integrated into advancing AI4science.

Re:

"Overall, the construction of RIGOR seems quite ad-hoc"

We believe RIGOR's design follows one principle: forecasts decide which experiment to run next. Specifically, RIGOR is one loop, and each component answers one question in this loop. Point (a) below states the loop, and our answer to Question 1 gives the motivation for each component.

(a) "the global research takeaway from the union of these components is unclear."

The takeaway is that, when experiments are expensive and the budget is small, LLM-driven discovery should forecast the outcome of every candidate and let the forecasts choose the experiments. The loop has four steps:

  1. The hypothesis agent writes several complete candidates from the earlier designs and outcomes.
  2. The outcome predictor forecasts the outcome of each candidate, with a prediction interval.
  3. UCB selection picks the candidate for the next experiment.
  4. A rubric of named design dimensions describes the candidates. The rubric changes only when a new dimension improves forecasts of later outcomes.

The global response gives the evidence for each component.

(b) "It does not seem surprising that hypothesis generation and outcome prediction could have the same representation because they are deeply connected processes within the scientific method. The paper does not explain how the rubric structure that embeds this unified representation is novel."

Because hypothesis generation and outcome prediction are deeply connected, RIGOR gives them one explicit representation. RIGOR is new in two ways (Section 1): the rubric judge builds the representation without compressing the design, and the gate tests any new dimension on future outcomes.

  • How the rubric is built. The rubric judge is an LLM that never sees outcomes. The rubric judge measures every candidate on named stance, formula and categorical dimensions, and each measurement quotes the design. Compositions and executable algorithms thus get the same kind of named measurements. The predictor reads the complete design together with these measurements, so the design is not compressed into a feature vector.
  • How the rubric changes. When a new dimension is proposed, the gate freezes its definition. Before each of the next five outcomes, RIGOR records one forecast with the proposed dimension and one forecast without it. The gate adopts the dimension only if the dimension lowers the MAE on those five outcomes by at least 5%. In the paper's runs, the gate adopts 16 of 86 proposed dimensions, and 52 of the 70 rejected dimensions increase prediction error.
  • How the rubric differs from related methods (Section 2). PiEvo evolves free-text principles. ReElicit learns semantic features from evaluated candidates. CAAFE and LLM-FE select generated features by their predictive value. In RIGOR, a new dimension must improve forecasts of outcomes that were not yet observed when the dimension was proposed.
AuthorsOpenReview ↗

Authors' Response to Reviewer nijq (2/6)

W1. Overall, the construction of RIGOR seems quite ad-hoc, [...] d) It is not clear what the next steps for this framework are--what other benchmarks it may be relevant to and how the overall principles can be integrated into advancing AI4science.

Re:

(c) "There are no ablation studies for any of the components in this framework. [...] several mechanisms within RIGOR, such as the 'single-change children' proposed by the hypothesis agent and the outputs of the executor are treated as details, but without the ablation, it's not clear if they are more or less relevant than the main emphasis of the rubric-based representation."

UCB selection and the rubric both matter to RIGOR's search performance. Of the two mechanisms that the reviewer names, single-change children produce 202 of the 375 improvements to the best design so far (Section 4.4), and the Executor is the same for all three methods. Following the reviewer's suggestion, we tested each component:

  • UCB selection picks better candidates than a random pick. Specifically, we scored all 3,081 candidates that RIGOR proposed in the 24 scientific-design runs, including the candidates that UCB did not select. On the scientific-design tasks, a pretrained evaluator scores each design, so we can score the unselected candidates after the runs. The results show that the selected candidates score 3.9% higher than a random pick from the same round (95% CI 2.1% to 5.6%), and higher in each of the 24 runs.
  • The rubric plays an important role in RIGOR's search performance. Specifically, we conducted an ablation study that removes the rubric from RIGOR. We ran this ablation study with GPT-5.5 on scaling-law modeling and online bandits, three runs each. RIGOR without the rubric has no rubric judge, no rubric in generation or prediction, and no refinement. The results show that RIGOR without the rubric has a lower final best score than RIGOR in all 6 runs, by 0.4% to 6.7%. The table is in our response to Reviewer amhF, Weakness 1.
  • The gate makes the search more stable and keeps the rubric interpretable. Specifically, we conducted an ablation study on TMC and SuperCon. RIGOR without the gate adopts every proposed dimension immediately. The results show that, without the gate, the outcomes of evaluations 13–24 vary more on both tasks. Their standard deviation is 17.04 on TMC and 2.67 on SuperCon without the gate, against 16.21 and 0.59 with the gate. Without the gate, the mean evaluated outcome is lower than with the gate in 5 of the 6 runs. Without the gate, 12 dimensions enter the rubric, against 5 with the gate (our response to Reviewer E6dc). In the paper's runs, the gate adopts 16 of 86 proposed dimensions, and 52 of the 70 rejected dimensions increase prediction error (Section 4.5).
  • Single-change children and independently specified candidates both improve the best design so far. A single-change child changes one decision of an evaluated design. Of the 375 improvements to the best design so far, 202 come from single-change children and 173 come from independently specified candidates (Section 4.4).
  • All three methods use the same Executor. On MLS-Bench, PiEvo and ReElicit use the same Executor, Verifier and fidelity audit as RIGOR. The scientific-design tasks have no Executor, and RIGOR still has the highest mean evaluated outcome in all 8 scientific-design settings.

(d) "It is not clear what the next steps for this framework are--what other benchmarks it may be relevant to and how the overall principles can be integrated into advancing AI4science."

The next step is to connect RIGOR to real experimental platforms, where every experiment is expensive, and to test RIGOR on further design benchmarks.

  1. Self-driving laboratories and simulation pipelines. On our benchmarks, RIGOR already saves experiments. In 11 of 16 settings, RIGOR reaches PiEvo's mean final best score with fewer experiments than PiEvo, by a median of 15 experiments (our response to Reviewer w7yQ, Weakness 2).
  2. Further design tasks. The same RIGOR configuration can run on additional MLS-Bench tasks and additional scientific-design tasks like those of PiEvo.

Our answer to Question 3 explains how these principles (forecasting before each experiment, recording design decisions in an explicit rubric, and testing new dimensions on future outcomes) transfer to AI4science more broadly.

AuthorsOpenReview ↗

Authors' Response to Reviewer nijq (3/6)

*W2. The relationship between the current study and the cited papers used to structure the evaluation is not explained. a) The paper seems to target an improvement on the PiEvo body of work and benchmarks. [...] but how this relates to the construction of RIGOR is not explained. As such, it is hard to evaluate how important it is that RIGOR improves over ReElicit.

Re:

(a) "it adds several algorithm-design tasks from MLS-Bench without explaining why they are relevant for inclusion and how they add to our understanding of the framework's strengths."

MLS-Bench tests RIGOR on designs that are executable programs. RIGOR leads on executable programs as well: it has the highest mean evaluated outcome in all 8 MLS-Bench settings and the highest mean final best score in 6 of the 8 settings.

  • Why the MLS-Bench tasks are included. In the scientific-design tasks from PiEvo, a design is a helix geometry, a molecule, a composition, or a ligand set, and a pretrained evaluator scores the design. In the MLS-Bench tasks (clustering, online bandits, scaling-law modeling, and multi-objective evolutionary algorithms), every candidate must be implemented as code that the official Verifier runs. A program has no fixed parameterization and therefore no feature vector for classical Bayesian optimization.
  • What the MLS-Bench tasks add. One RIGOR configuration serves both scientific design and algorithm design. One rubric format, with stance, formula, and categorical dimensions, records both parameter choices and algorithmic mechanisms. The rubric plays an important role in RIGOR's search performance: RIGOR without the rubric has a lower final best score than RIGOR in all 6 runs on scaling-law modeling and online bandits (our response to Reviewer amhF, Weakness 1).

The revised paper states the motivation for the MLS-Bench tasks in Section 4.1.

(b) "The relevance of the ReElicit framework as a comparison is not explained."

ReElicit addresses the same problem as RIGOR: Bayesian optimization over candidates written as free text, with a representation derived by an LLM. ReElicit is therefore the natural baseline for testing how RIGOR represents a design. ReElicit was introduced to optimize prompts, but its optimizer is general. ReElicit learns semantic features from evaluated candidates, fits a Gaussian process on the learned features, and generates new candidates toward the feature values that its acquisition function selects. Whereas ReElicit compresses each candidate into a learned feature vector for a Gaussian process, RIGOR's predictor reads the complete design together with the design's rubric measurements. Comparing RIGOR with ReElicit therefore tests both ways of bringing Bayesian optimization to LLM-generated hypotheses. Both methods use the same task descriptions, evaluators, feasibility constraints, and evaluation budget, and on MLS-Bench both methods use the same Executor.

RIGOR outperforms our implementation of ReElicit. On the mean evaluated outcome, a RIGOR run beats a ReElicit run of the same setting with probability 0.93 (Holm-corrected p=6.2×10−14p=6.2\times10^{-14}). On the final best score, a RIGOR run beats a ReElicit run of the same setting with probability 0.73 (p=2.7×10−4p=2.7\times10^{-4}). Our response to Reviewer w7yQ, Weakness 3, describes the test. The revised paper states the motivation for ReElicit as a baseline in Section 2.

Reviewer_E6dcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses LLM-guided scientific and algorithmic discovery under limited evaluation budgets. The authors propose RIGOR (Representation-Instrumented Generation with Outcome-Gated Refinement), which uses shared, interpretable rubric dimensions to represent candidate designs. An LLM-based predictor estimates outcomes and uncertainty, while a UCB strategy guides experiment selection. A key contribution is the Prospective Gate, which adopts new rubric dimensions only when they reduce prediction MAE by at least 5% over five subsequent successful evaluations, using forecasts recorded before outcomes are observed. This aims to prevent representation updates based solely on historical fit. The framework is evaluated on eight scientific and algorithmic design tasks using Opus 4.7 and GPT-5.5. Under a budget of 24 evaluations per run, RIGOR achieves the highest mean final best score in 11 of 16 model–task settings and the highest mean evaluated outcome in all 16.

Critical strengths
  1. Well motivated prospective refinement mechanism One of the main strengths of this paper is the prospective gate, which allows new rubric dimensions to be adopted based on their predictive value on subsequent outcomes, rather than relying solely on historical observations. This helps reduce the risk of introducing features that only explain past results and provides a more evidence-driven and interpretable approach to updating design representations. I consider this a meaningful contribution because it addresses an important challenge in adaptive LLM guided discovery: how to refine representations using experimental feedback without simply overfitting to previous observations.

  2. Encouraging empirical performance across diverse tasks The paper evaluates RIGOR on eight tasks spanning scientific discovery and algorithm design, using two foundation models under a consistent evaluation budget. RIGOR achieves the highest mean final best score in 11 of 16 settings and the highest mean evaluated outcome in all 16 settings. These results are encouraging because they suggest that the framework can improve both the quality of the best discovered solutions and the overall quality of evaluated candidates. The relatively broad experimental coverage also strengthens the practical relevance of the approach, particularly in settings where external evaluations are expensive

Critical weaknesses

Limited evidence isolating the contribution of prospective refinement While RIGOR demonstrates encouraging overall performance, the paper lacks controlled ablations that isolate the contributions of its main components, particularly dynamic rubric refinement. Although the prospective gate is shown to improve prediction accuracy and the paper provides illustrative examples, its direct contribution to subsequent search performance remains unclear. Comparisons with fixed-rubric or no-prospective-gate variants would strengthen the evidence for the proposed central mechanism.

Questions for authors
  1. Contribution of dynamic refinement Could the authors clarify how much of RIGOR's performance improvement comes from dynamic rubric refinement, compared with using a fixed rubric or removing the prospective gate?

  2. Downstream search performance Beyond the reported improvements in prediction accuracy and illustrative case studies, is there evidence that adopting new rubric dimensions consistently improves subsequent search performance?

2 comments
AuthorsOpenReview ↗

Authors' Response to Reviewer E6dc (1/2)

W. Limited evidence isolating the contribution [...]. Comparisons with fixed-rubric or no-prospective-gate variants would strengthen the evidence for the proposed central mechanism.

Re:

"the paper lacks controlled ablations that isolate the contributions of its main components, particularly dynamic rubric refinement. [...] its direct contribution to subsequent search performance remains unclear. Comparisons with fixed-rubric or no-prospective-gate variants would strengthen the evidence for the proposed central mechanism."

The gate makes the search more stable and more efficient, and it keeps the rubric interpretable. Compared with a fixed rubric, RIGOR evaluates better designs and is more stable. Specifically, we followed the reviewer's suggestion and conducted both ablations on TMC and SuperCon with GPT-5.5, three runs each. TMC and SuperCon are the two tasks on which RIGOR adopts new dimensions in 5 of its 6 runs. RIGOR with a fixed rubric keeps the initial rubric for the whole run. RIGOR without the gate adopts every proposed dimension immediately. Each new run starts from the initial rubric of one RIGOR run and is compared with that run.

(1) Without the gate, the search is less stable and less efficient. The results show that:

  • Without the gate, the outcomes of evaluations 13–24 vary more on both TMC and SuperCon. The standard deviation of these outcomes is 17.04 on TMC and 2.67 on SuperCon without the gate, against 16.21 on TMC and 0.59 on SuperCon with the gate.
  • On SuperCon, without the gate, one outcome of evaluations 13–24 is as low as 83.4 K, whereas RIGOR's lowest outcome of evaluations 13–24 is 105.9 K.
  • RIGOR without the gate has a lower mean evaluated outcome than RIGOR in 5 of the 6 runs.
  • On TMC, two runs of RIGOR without the gate find the best design at evaluations 17 and 23, against evaluations 9 and 12 for two runs with the gate.

(2) Compared with a fixed rubric, RIGOR evaluates better designs and is more stable. The results show that RIGOR has a higher mean evaluated outcome than RIGOR with a fixed rubric in 4 of the 6 runs, including all three SuperCon runs. In evaluations 13–24, RIGOR's outcomes also vary less than those of RIGOR with a fixed rubric in 4 of the 6 runs.

Means over three runs. QQ is the mean evaluated outcome, and SlateS_{\rm late} is the standard deviation of the outcomes of evaluations 13–24 (Section 4.3):

StatisticRIGORFixed rubricNo gate
TMC: mean evaluated outcome QQ451.87457.70449.39
TMC: late within-run variation SlateS_{\rm late}16.2116.2017.04
SuperCon: mean evaluated outcome QQ109.42107.62107.70
SuperCon: late within-run variation SlateS_{\rm late}0.590.802.67

(3) The gate keeps the rubric interpretable. The gate adopts a proposed dimension only after the dimension improves forecasts of later outcomes, and the gate needs no additional experiments (Section 3.5).

  • The gate is selective. In the paper's runs, the gate adopts 16 of 86 proposed dimensions. Each of the 16 adopted dimensions lowers the MAE on the five outcomes after its proposal by 5.38% to 41.80%, and 11 of the 16 adopted dimensions would also pass a 10% MAE-reduction threshold (Section 4.5).
  • The gate keeps out dimensions that make prediction worse. Of the 70 rejected dimensions, 52 increase prediction error. A gate that decided after two outcomes would adopt 14 dimensions whose gain does not persist over five outcomes (Section 4.5). On TMC and SuperCon, RIGOR without the gate adds 12 dimensions to the rubric, whereas RIGOR adopts 5 dimensions.
  • The adopted dimensions record which design decisions matter. A fixed rubric cannot record these design decisions, which separate better designs from worse ones. On SuperCon, Hg-1223 compositions average 109.9 K, whereas Hg-1234 compositions average 104.2 K. Compositions with bismuth in the charge reservoir average 111.7 K, whereas compositions without bismuth average 104.5 K. On TMC, complexes with two BIFZEX ligands average 484.4, whereas complexes with two GUQBUQ ligands average 440.4 (BIFZEX and GUQBUQ are two of the ligands that a TMC design can use).

The revised paper includes both ablations in Section 4.6 and Appendix H (Table 15). The global response gives the evidence for UCB selection and for the rubric.

AuthorsOpenReview ↗

Authors' Response to Reviewer E6dc (2/2)

Q1. Contribution of dynamic refinement Could the authors clarify how much of RIGOR's performance improvement comes from dynamic rubric refinement, compared with using a fixed rubric or removing the prospective gate?

Re:

"Could the authors clarify how much of RIGOR's performance improvement comes from dynamic rubric refinement, compared with using a fixed rubric or removing the prospective gate?"

Dynamic rubric refinement through the gate makes the search more stable and more efficient. Specifically, we conducted three ablation studies: RIGOR without the rubric, RIGOR with a fixed rubric, and RIGOR without the gate. The results show that:

  • Without the rubric, the final best score is lower than RIGOR's in all 6 runs on scaling-law modeling and online bandits.
  • Without the gate, the outcomes of evaluations 13–24 vary more on both TMC and SuperCon, and the mean evaluated outcome is lower than RIGOR's in 5 of the 6 runs.
  • Compared with a fixed rubric, RIGOR has the higher mean evaluated outcome in 4 of the 6 runs on TMC and SuperCon, and RIGOR's outcomes of evaluations 13–24 vary less in 4 of the 6 runs.

Dynamic rubric refinement works through the gate, which decides which dimensions enter the rubric and needs no additional experiments (Section 3.5). In the paper's runs, 52 of the 70 dimensions that the gate rejects increase prediction error (Section 4.5). The table is in our response to the weakness above, and the global response gives the evidence for each component.


Q2. Downstream search performance Beyond the reported improvements in prediction accuracy and illustrative case studies, is there evidence that adopting new rubric dimensions consistently improves subsequent search performance?

Re:

"Beyond the reported improvements in prediction accuracy and illustrative case studies, is there evidence that adopting new rubric dimensions consistently improves subsequent search performance?"

By adopting new dimensions, RIGOR evaluates better designs and is more stable than RIGOR with a fixed rubric. Specifically, we conducted an ablation study on TMC and SuperCon that compares RIGOR against RIGOR with a fixed rubric. RIGOR with a fixed rubric keeps the initial rubric and never adopts a dimension. The results show that RIGOR has the higher mean evaluated outcome in 4 of the 6 runs, and RIGOR's outcomes of evaluations 13–24 vary less in 4 of the 6 runs. The adopted dimensions also separate better designs from worse ones. On SuperCon, for example, Hg-1223 compositions average 109.9 K and Hg-1234 compositions average 104.2 K. The table is in our response to the weakness above.

In the paper's runs, each of the 16 adopted dimensions lowers the MAE by 5.38% to 41.80% on the five outcomes after its proposal, and 11 of the 16 adopted dimensions would also pass a 10% MAE-reduction threshold (Section 4.5).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Authors' Global Response (1/2)

We thank all reviewers for their careful and constructive reviews. We provide a global response here that summarizes the strengths that the reviewers recognized, the new evidence, our answers to the questions that several reviewers raised, and what the revised paper adds. We answer each reviewer's other questions in our response to that reviewer.

Strengths recognized by the reviewers

  • The prospective gate. "One of the main strengths of this paper is the prospective gate", which "addresses an important challenge in adaptive LLM guided discovery: how to refine representations using experimental feedback without simply overfitting to previous observations" (E6dc). The prospective refinement mechanism is "a well-motivated contribution", and "the separation between dimension proposal and activation is particularly well designed" (amhF).
  • One closed loop for generation, prediction and selection. "[U]sing a unified format for outcome prediction and hypothesis generation within a closed experimental loop is a useful direction" (nijq). RIGOR establishes "a coherent pipeline for candidate generation, evaluation, and selection" and "offers a promising approach to LLM-driven scientific and algorithmic discovery under limited evaluation budgets" (w7yQ).
  • A broad and thorough evaluation. The results "suggest that the framework can improve both the quality of the best discovered solutions and the overall quality of evaluated candidates", and "the relatively broad experimental coverage also strengthens the practical relevance of the approach, particularly in settings where external evaluations are expensive" (E6dc). "The experimental evaluation is relatively thorough", and "the additional analyses of prediction accuracy, uncertainty calibration, and refinement decisions strengthen the empirical results" (amhF).

New evidence

The new evidence confirms the paper's findings. Specifically, we followed the reviewers' suggestions and conducted new runs and new analyses of the paper's runs. The new runs cover three ablation studies: RIGOR without the rubric, RIGOR with a fixed rubric, and RIGOR without the gate.

New evidenceResult
RIGOR without the rubricThe final best score is lower in all 6 runs on scaling-law modeling and online bandits
RIGOR without the gateThe search is less stable on both tasks tested, and the mean evaluated outcome is lower in 5 of the 6 runs
Scores of all 3,081 candidates that RIGOR proposed in the 24 scientific-design runs, selected or notThe candidates that UCB selects score 3.9% higher than a random pick from the same round, and higher in each of the 24 runs
Permutation tests over all 144 runsRIGOR's mean evaluated outcome is significantly higher than PiEvo's and ReElicit's (p≤6.2×10−14p\le6.2\times10^{-14})
Model calls, tokens and cost of all three methodsRIGOR's model calls cost USD 2.76 to 12.09 per experiment. In 11 of 16 settings, RIGOR reaches PiEvo's mean final best score with fewer experiments than PiEvo
Initial and final rubrics of all 48 RIGOR runsIn the supplementary material
AuthorsOpenReview ↗

Authors' Global Response (2/2)

Questions raised by several reviewers

Which components drive RIGOR's gains? UCB selection and the rubric drive the gains.

  • UCB selection picks better candidates than a random pick. Specifically, we scored all 3,081 candidates that RIGOR proposed in the 24 scientific-design runs, including the candidates that UCB did not select. The results show that the selected candidates score 3.9% higher than a random pick from the same round (95% CI 2.1% to 5.6%), and higher in each of the 24 runs.
  • The predictor forecasts outcomes better than the median of the earlier outcomes. The predictor has a lower MAE than the median in 46 of the 48 RIGOR runs (Section 4.4).
  • Both kinds of candidates improve the best design so far. Of the 375 improvements to the best design so far, 202 come from single-change children, which change one decision of an evaluated design, and 173 come from independently specified candidates (Section 4.4).
  • The rubric plays an important role in RIGOR's search performance. Specifically, we conducted an ablation study that removes the rubric from RIGOR. The results show that, on scaling-law modeling and online bandits, RIGOR without the rubric has a lower final best score in all 6 runs, by 0.4% to 6.7%.
  • The gate makes the search more stable and more efficient, and it keeps the rubric interpretable. Specifically, we conducted an ablation study on TMC and SuperCon that compares RIGOR against RIGOR without the gate. RIGOR without the gate adopts every proposed dimension immediately. The results show that, without the gate, the outcomes of evaluations 13–24 vary more on both tasks. Their standard deviation is 17.04 on TMC and 2.67 on SuperCon without the gate, against 16.21 and 0.59 with the gate. Without the gate, the mean evaluated outcome is lower than with the gate in 5 of the 6 runs. On TMC, two runs of RIGOR without the gate find the best design at evaluations 17 and 23, against evaluations 9 and 12 for two runs with the gate. Without the gate, 12 dimensions enter the rubric, against 5 with the gate. In the paper's runs, the gate adopts 16 of 86 proposed dimensions, and 52 of the 70 rejected dimensions increase prediction error (Section 4.5). The gate needs no additional experiments (Section 3.5).

Are the differences reliable with three runs per setting? Yes. Specifically, we conducted an exact permutation test in which each of the 144 runs is one observation. The results show that RIGOR's mean evaluated outcome is significantly higher than PiEvo's (p=3.8×10−18p=3.8\times10^{-18}) and ReElicit's (p=6.2×10−14p=6.2\times10^{-14}). Its final best score is also significantly higher than PiEvo's (p=0.042p=0.042) and ReElicit's (p=2.7×10−4p=2.7\times10^{-4}).

Is the additional model cost justified? Yes. RIGOR is designed for discovery in which the experiment, not the model call, is the dominant cost. This is why the paper compares all methods at the same number of experiments (Sections 3.1 and 4.1). RIGOR's model calls cost USD 2.76 to 12.09 per experiment. In 11 of 16 settings, RIGOR reaches PiEvo's mean final best score with fewer experiments than PiEvo, by a median of 15 experiments. In 10 of these 11 settings, the experiments saved are worth more than RIGOR's entire model cost if one experiment costs more than USD 37.29. This holds even if PiEvo's model calls are counted as free. In the eleventh setting, RIGOR's model cost is lower than PiEvo's.

What the revised paper adds

  • Section 4.6 and Appendix H: the three ablation studies.
  • Section 4.4 and Appendix I: UCB selection against a random pick.
  • Section 4.2 and Appendix I: the permutation tests.
  • Appendix J: model calls, tokens and cost of all three methods, and the experiments saved.
  • Section 2 and Appendix K: a discussion of HypoSpace, and an analysis of exploration and concentration across design families with HypoSpace's criteria.
  • Appendix L and the supplementary material: the complete rubric of one run, and the initial and final rubrics of all 48 RIGOR runs.
  • Sections 2 and 4.1: the motivation for ReElicit as a baseline and for the MLS-Bench tasks.
  • Section 5: how RIGOR balances exploration and concentration over the budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.