- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents RIGOR, a framework for budgeted scientific discovery that integrates rubric-guided hypothesis generation, LLM-based outcome prediction, and uncertainty-aware experiment selection. A prospective refinement mechanism introduces new representation dimensions only when they improve prediction accuracy on subsequent outcomes. The authors evaluate RIGOR on four scientific-design tasks and four algorithm-design tasks using two foundation models. Results show improvements over PiEvo and ReElicit in final discovery performance, average evaluated outcome, and search stability.
Critical strengths
-
The prospective refinement mechanism is a well-motivated contribution. By validating proposed representation dimensions against future outcomes rather than relying solely on historical fit, the framework introduces a useful safeguard against overfitting during iterative discovery. The separation between dimension proposal and activation is particularly well designed.
-
The experimental evaluation is relatively thorough, covering eight tasks, two foundation models, and multiple performance metrics. The improvements in mean evaluated outcomes and search stability are encouraging, and the additional analyses of prediction accuracy, uncertainty calibration, and refinement decisions strengthen the empirical results.
Critical weaknesses
-
The contributions of individual components are not sufficiently isolated. RIGOR combines rubric-guided generation, LLM-based outcome prediction, UCB selection, and prospective refinement, but comparisons against PiEvo and ReElicit do not establish which components drive the improvements. In particular, the prospective gate is validated through prediction MAE rather than downstream discovery performance. It remains unclear whether activating new dimensions actually improves search outcomes or whether most gains come from the initial rubric and selection mechanism. This limits the evidence supporting the central methodological contribution.
-
The evaluation provides limited insight into hypothesis-space exploration. While RIGOR achieves strong average outcomes, this does not necessarily imply better exploration or coverage of the design space. Related work such as HypoSpace: Evaluating LLM Creativity as Set-Valued Hypothesis Generators under Underdetermination (ICML 2026) highlights the importance of validity, uniqueness, and recovery in evaluating hypothesis generation, but is not discussed in the paper. A more explicit analysis of candidate diversity, coverage, and exploration–exploitation trade-offs would help clarify whether RIGOR improves exploration or primarily concentrates on promising design families.
-
The statistical robustness and computational efficiency require further validation. The experiments use only three runs per setting and a fixed budget of 24 evaluations, with relatively small performance differences in several tasks. Moreover, RIGOR introduces additional LLM inference costs through repeated rubric measurements, outcome predictions, and prospective validation. Although the evaluation budgets are matched, the total computational costs are not systematically compared. This makes it difficult to assess whether the improvements remain significant and cost-effective under larger evaluation budgets.
Questions for authors
-
Could the authors provide component-level ablations, particularly demonstrating whether prospective refinement improves downstream discovery performance beyond prediction MAE?
-
Does RIGOR improve hypothesis diversity and exploration coverage, or primarily exploit previously successful design families?
-
How robust are the improvements across additional seeds and longer evaluation budgets, and how does RIGOR compare with baselines in total computational cost?
I am open to increasing my rating if these concerns are adequately addressed.
9 comments
Authors' Response to Reviewer amhF (2/7)
W1. The contributions of individual components are not sufficiently isolated. RIGOR combines rubric-guided generation, LLM-based outcome prediction, UCB selection, and prospective refinement, but comparisons against PiEvo and ReElicit do not establish which components drive the improvements. In particular, the prospective gate is validated through prediction MAE rather than downstream discovery performance. It remains unclear whether activating new dimensions actually improves search outcomes or whether most gains come from the initial rubric and selection mechanism. This limits the evidence supporting the central methodological contribution.
Re:
(2) "the prospective gate is validated through prediction MAE rather than downstream discovery performance. It remains unclear whether activating new dimensions actually improves search outcomes or whether most gains come from the initial rubric and selection mechanism."
Activating new dimensions through the gate makes the search more stable and more efficient, and it keeps the rubric interpretable. Specifically, we conducted two ablations on TMC and SuperCon, the two tasks on which RIGOR adopts new dimensions in 5 of its 6 runs. RIGOR with a fixed rubric keeps the initial rubric for the whole run. RIGOR without the gate adopts every proposed dimension immediately. Each new run starts from the initial rubric of one RIGOR run and is compared with that RIGOR run. The results show that:
- Without the gate, the search is less stable and less efficient. The outcomes of evaluations 13–24 vary more without the gate on both tasks. Their standard deviation is 17.04 on TMC and 2.67 on SuperCon without the gate, against 16.21 and 0.59 with the gate. Without the gate, the mean evaluated outcome is lower than RIGOR's in 5 of the 6 runs. On TMC, two runs of RIGOR without the gate find the best design at evaluations 17 and 23, against evaluations 9 and 12 for two runs with the gate.
- Compared with a fixed rubric, RIGOR evaluates better designs and is more stable. RIGOR has a higher mean evaluated outcome than RIGOR with a fixed rubric in 4 of the 6 runs, and RIGOR's outcomes of evaluations 13–24 vary less in 4 of the 6 runs.
Means over three runs. is the mean evaluated outcome, and is the standard deviation of the outcomes of evaluations 13–24 (Section 4.3):
| Statistic | RIGOR | Fixed rubric | No gate |
|---|---|---|---|
| TMC: mean evaluated outcome | 451.87 | 457.70 | 449.39 |
| TMC: late within-run variation | 16.21 | 16.20 | 17.04 |
| SuperCon: mean evaluated outcome | 109.42 | 107.62 | 107.70 |
| SuperCon: late within-run variation | 0.59 | 0.80 | 2.67 |
The gate keeps the rubric interpretable. The gate adopts a proposed dimension only after the dimension improves forecasts of later outcomes, and the gate needs no additional experiments (Section 3.5):
- In the paper's runs, the gate adopts 16 of 86 proposed dimensions. Each adopted dimension lowers the MAE on the five outcomes after its proposal by 5.38% to 41.80% (Section 4.5).
- Of the 70 rejected dimensions, 52 increase prediction error. A gate that decided after two outcomes would adopt 14 dimensions whose gain does not persist over five outcomes (Section 4.5).
- On TMC and SuperCon, RIGOR without the gate adds 12 dimensions to the rubric, and RIGOR adopts the 5 dimensions that pass the gate.
- The adopted dimensions record which design decisions matter and separate better designs from worse ones. On SuperCon, Hg-1223 compositions average 109.9 K and Hg-1234 compositions average 104.2 K. On TMC, complexes with two BIFZEX ligands average 484.4, and complexes with two GUQBUQ ligands average 440.4 (BIFZEX and GUQBUQ are two of the ligands that a TMC design can use).
The revised paper includes all of these ablations in Section 4.6 and Appendix H (Tables 14 and 15).
Authors' Response to Reviewer amhF (3/7)
W2. The evaluation provides limited insight into hypothesis-space exploration. While RIGOR achieves strong average outcomes, this does not necessarily imply better exploration or coverage of the design space. Related work such as HypoSpace: Evaluating LLM Creativity as Set-Valued Hypothesis Generators under Underdetermination (ICML 2026) highlights the importance of validity, uniqueness, and recovery in evaluating hypothesis generation, but is not discussed in the paper. A more explicit analysis of candidate diversity, coverage, and exploration–exploitation trade-offs would help clarify whether RIGOR improves exploration or primarily concentrates on promising design families.
Re:
"A more explicit analysis of candidate diversity, coverage, and exploration–exploitation trade-offs would help clarify whether RIGOR improves exploration or primarily concentrates on promising design families."
RIGOR does both: it explores early and concentrates later experiments on the most promising design families (Sections 4.3 and 4.4). RIGOR's exploration weight sets this order.
- Exploration. We say that uncertainty changes a selection when UCB selects a candidate whose point forecast is not the highest. Uncertainty changes 32.0% of the selections in evaluations 1–8, 26.6% in evaluations 9–16, and 6.0% in evaluations 17–24. Independently specified candidates produce 173 of the 375 improvements to the best design so far.
- Concentration. In 44 of the 48 RIGOR runs, the outcomes in the second half of the budget vary less than the outcomes in the first half. Of RIGOR's 1,152 evaluated designs, 686 are single-change children of evaluated designs.
We thank the reviewer for pointing us to HypoSpace. Under HypoSpace's criteria, RIGOR concentrates more than PiEvo, never repeats an exact design, and recovers the largest share of near-optimal families on TMC. In all 8 scientific-design settings, RIGOR also has the highest mean evaluated outcome. For this analysis, we adapted HypoSpace's uniqueness and recovery criteria to the designs evaluated in all 72 scientific-design runs of the three methods.
- A family groups designs of the same kind: a cell of the grid that divides each of the four parameter ranges into three equal bins (Nanohelix), a Bemis–Murcko scaffold (ChEMBL), an element set (SuperCon), and the donor-atom pairs of the neutral and anionic ligands (TMC).
- A near-optimal family is a family in which any of the nine runs that the three methods have in a setting evaluates a design within 5% of the best outcome observed in that setting. Recovery is the share of near-optimal families that a run evaluates at least once.
The results show that:
| Criterion | Result |
|---|---|
| Concentration | RIGOR evaluates fewer distinct families than PiEvo in 8 of 8 settings. In evaluations 9–24, RIGOR places a higher share of its evaluations in the family of the best design so far than PiEvo does, in 8 of 8 settings. Against ReElicit: fewer families in 5 of 8 settings, and a higher share in 6 of 8 |
| Uniqueness | RIGOR never repeats an exact design. In one ChEMBL run of ReElicit, 13 of the 24 evaluations repeat an earlier design exactly |
| Recovery of near-optimal families | On TMC, RIGOR's recovery is 0.58 with both models, against 0.08–0.33 for PiEvo and 0.17–0.25 for ReElicit |
The revised paper discusses HypoSpace in Section 2, reports the HypoSpace analysis in Appendix K, and states in Section 5 how RIGOR balances exploration and concentration over the budget.
Authors' Response to Reviewer amhF (1/7)
W1. The contributions of individual components are not sufficiently isolated. RIGOR combines rubric-guided generation, LLM-based outcome prediction, UCB selection, and prospective refinement, but comparisons against PiEvo and ReElicit do not establish which components drive the improvements. In particular, the prospective gate is validated through prediction MAE rather than downstream discovery performance. It remains unclear whether activating new dimensions actually improves search outcomes or whether most gains come from the initial rubric and selection mechanism. This limits the evidence supporting the central methodological contribution.
Re:
UCB selection and the rubric drive the improvements. Following the reviewer's suggestion, we tested each component with a controlled comparison. We ran the ablation studies with GPT-5.5 and compared each new run with one RIGOR run from the paper.
Means over three runs: