acceptodds
Under review as a conference paper at ICLR 2027

Scientific Discovery under Validation Congestion via Multi-Fidelity Pairwise Rankings

Abstract

Modern computational methods can now propose candidate molecules, materials, and other scientific designs at an unprecedented scale, creating a *validation congestion* where candidates are abundant, but experimental capacity to physically evaluate them remains scarce. Discovering novel scientific designs has therefore become increasingly dependent on *curation*: selecting a small set of promising designs for slow and costly experiments. Existing curation methods typically rely on data-driven regression models that predict absolute scores, but training these models requires substantial experimental data to begin with. Yet, useful curation signals do not have to take the form of absolute measurements, as scientific design discovery is often comparative in nature. Here, we propose that curation can instead be primarily driven by expert pairwise rankings, which are substantially easier to gather. The expertise can come from computational tools or human input of multiple levels of fidelity, ranging from empirical rules of thumb to agentic workflows and experienced scientists. We introduce PRISMS, a framework that uses pairwise rankings from one or more experts, potentially spanning multiple levels of expertise, to identify the most promising candidates without relying on data-hungry regressors. When experts differ in fidelity and cost, PRISMS escalates pairwise queries from lower- to higher-fidelity rankers based on a Fisher-information criterion. In iterative screening that selects designs from fixed drug discovery libraries, PRISMS achieves 50% top-10 discovery recall in ∼42% fewer rounds than regression-only active learning, and in ∼15% fewer rounds than the ranking-based method with no selective escalation. In optimization that generates new designs without restriction to a predefined library, PRISMS achieves ∼18.8% higher hypervolume than the Bayesian optimization baseline. These results show that multi-fidelity expert pairwise rankings can provide an efficient alternative to regression-driven curation in scientific design discovery.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
pairwise ranking, scientific discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zEQNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PRISMS, a framework for scientific design selection using multi-fidelity pairwise rankings instead of relying primarily on regression models trained with experimental measurements. It combines rankings from different sources through a regularized Bradley–Terry model and selectively queries higher-fidelity rankers using a Fisher information-based criterion. Experiments on drug discovery and engineering datasets show improved screening efficiency and ranking performance over existing baselines. The method is also evaluated on multi-objective molecular optimization, with additional ablation and sensitivity studies.

Critical strengths
  1. The motivation is interesting and well justified, as pairwise comparisons can be easier to obtain than accurate absolute property predictions, especially when experimental data is scarce.
  2. The proposed framework provides a reasonable way to combine pairwise rankings from different sources with available experimental measurements. The information-based selection of higher-fidelity queries is also well motivated.
  3. The experiments cover 14 drug discovery datasets, three engineering datasets, and a molecular optimization task. PRISMS achieves consistent improvements over several regression-based and ranking-based baselines.
  4. The ablation and sensitivity analyses are fairly comprehensive, showing the effects of graph connectivity, query selection, ranker accuracy, and query budgets.
Critical weaknesses
  1. Most experiments rely on a simulated high-fidelity ranker with 75% accuracy, constructed by randomly flipping ground-truth comparisons. While useful for controlled evaluation, this setting does not fully reflect the reliability and biases of real scientific experts or computational models, limiting the evidence for practical applicability.
  2. The evaluation with a real LLM ranker is limited to three tasks and three random seeds, with considerably lower performance than the simulated oracle. Moreover, the lack of comparisons with baselines using the same LLM ranker makes it difficult to assess the effectiveness of PRISMS in realistic settings.
  3. The method assumes a fixed ordering of ranker reliability and uses fixed weights for each ranker. However, the LLM analysis shows that ranking accuracy varies substantially with the property difference between candidates. Although different noise settings are evaluated, the query selection criterion does not explicitly account for such variations in ranker reliability.
Questions for authors
  1. Could the authors clarify how the two independently predicted objectives are used to select candidates in the multi-objective optimization setting?
2 comments

We are glad that the reviewer found the work interesting and well justified, with extensive experiments, ablations, and sensitivity analysis. We address the reviewer’s concerns below.

Expanding Experiments with Non-Simulated / Realistic Ranker

In a new experiment, we replaced the 75%-accurate simulated ranker with TabPFN 2.5, a tabular foundation model. TabPFN is chosen here over proprietary LLMs (such as Claude Opus 5.5 in our previous experiments) as it is open-source and can be run on a local GPU, making it more accessible to a larger audience. As TabPFN allows “in-context learning” during inference, its pairwise ranking accuracy increases with the number of labels. Table C.1. below shows the average pairwise ranking accuracy of TabPFN 2.5 across 12-round iterative screening campaigns on 14 datasets .

Table C.1. Pairwise ranking accuracy of TabPFN 2.5 across varying number of labels revealed (context)

# Labels Revealed102030405060708090100110120
Accuracy63.20%65.81%66.14%67.86%67.71%67.98%69.30%69.79%69.57%69.82%70.35%71.06%

We use the TabPFN ranker for screening campaigns with similar setup as in Section 4.1 (14 datasets, 10 random seeds, 12 rounds). In this new experiment, PRISMS achieves 0.323 discovery-recall AUC, compared to 0.285 for RankRefine++, the second-best method we evaluated. The results, shown in Table C.2. below, demonstrate that PRISMS retains clear advantage when using realistic, dynamic, higher-fidelity rankers.

Table C.2. Experimental results for iterative screening in drug discovery datasets using TabPFN 2.5 as the higher fidelity ranker, across 10 random seeds.

MethodDiscovery recallNDCGTop-1% F1Top-5% F1
Random sampling0.076 (0.006)0.334 (0.020)0.120 (0.010)0.134 (0.006)
GP-EI0.139 (0.011)0.401 (0.020)0.190 (0.014)0.205 (0.009)
GP-EI Pairwise0.221 (0.014)0.403 (0.022)0.306 (0.018)0.252 (0.010)
HodgeRank0.285 (0.018)0.473 (0.020)0.349 (0.020)0.327 (0.013)
Active top-k0.283 (0.018)0.476 (0.020)0.239 (0.013)0.308 (0.011)
Spectral MLE0.271 (0.018)0.469 (0.020)0.340 (0.020)0.304 (0.012)
RankRefine++0.285 (0.019)0.480 (0.019)0.341 (0.021)0.325 (0.014)
PRISMS0.323 (0.018)0.507 (0.020)0.352 (0.018)0.354 (0.013)

Fixed ordering of ranker reliability and fixed weights for each ranker

In principle, the weights in Equation 2 can be adjusted online based on prior knowledge of each ranker's reliability in the target domain. For example, if ranker g2g_2 is known to be less reliable than g1g_1 when comparing pairs (x1,x2)(x_1, x_2) from subgroup XaX_a, we can assign it a lower weight: λ1(2,a)<λ1(1,a)\lambda_1^{(2,a)} < \lambda_1^{(1,a)}.

However, such adaptation requires knowledge about the domain and the ranker. For example, comparing similar molecules may be easier for a chemist because their structural differences are easier to isolate, whereas comparing similar airfoils may be harder for an engineer because their lift differences are less apparent. Thus, we leave online adaptation as potential future work that is out of scope for our current submission.

That being said, the new results with TabPFN as a realistic ranker show that PRISMS still outperform the baselines on more realistic ranker regimes.

How are candidates selected in multi-objective optimization?

On round tt, a method estimates nn objectives independently, resulting in each candidate represented by nn predicted scores. The promising candidates are then selected from the nn-dimensional Pareto front of size mm. For a validation budget of bb, bb candidates are sampled uniformly from the Pareto front if b<mb < m. If b>mb > m, the full Pareto front candidates are selected and then augmented with (b−m)(b-m) non-Pareto front candidates with the highest mean normalized scores.

Thank you for the detailed response and additional experiments, which have adequately addressed my concerns. I will maintain my positive rating and wish the authors the best of luck!

Reviewer_54DYOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses design curation under validation congestion, arguing that curation need not rely on label-hungry absolute regression. PRISMS maintains a graph of pairwise rankings, infers property values via a regularized Bradley-Terry objective anchored on measured values and a weak prior, and allocates ranker queries using Fisher information; RankRefine++ is a special case. Across 14 drug-discovery and 3 engineering tasks, PRISMS reaches 50% top-10 recall in 11.1 rounds versus 19.3 for GP-EI and 13.1 for RankRefine++, with higher terminal hypervolume than GP-BO.

Critical strengths
  1. The paper includes ablations, a sensitivity analysis across three noise models, per-dataset paired differences and a candid limitations section. These additions strengthen the empirical and methodological assessment.

  2. The shift from predicting absolute values to comparing candidates is substantive and well argued, since pairwise judgments are genuinely easier to elicit than calibrated measurements.

Critical weaknesses
  1. The “42% fewer rounds” claim compares PRISMS with GP-EI, but GP-EI receives 10 labels per round, whereas PRISMS receives those same labels plus 100 comparisons from an oracle derived from the ground-truth labels. The fair comparison is against the ranking-based baselines.

  2. The abstract says the gain comes from removing selective escalation. But the main baseline, RankRefine, also has no selective escalation, so this comparison cannot show that selective escalation is the cause. In addition, Table 2 shows that random escalation changes the result by 7.7%, which is only about one third of the claimed effect.

  3. Random Sampling reaches 23.3 against a horizon of 24, which means that nearly all campaigns never attain 50% recall and that the reported statistic is effectively driven by the censoring bound. Without the fraction of campaigns that reach the threshold, “19.3 rounds” cannot be interpreted as a time-to-event, and the conversion into “8–24 months” is not supported.

  4. Equation 7 scores each pair independently and then selects the top-b pairs, without accounting for overlap among them. Because the log-determinant gain is submodular, sequential lazy-greedy selection is the natural approach, yet the paper neither discusses nor compares against it.

Questions for authors
  1. The runtimes in Appendix B.1 scale by factors of 22 and 100 across successive decades of N, implying near-quadratic growth. Extrapolating places a single round at roughly 90 hours for N = 10⁶. Which component dominates, and do you see a sparse or amortized variant that would make 10⁶ and beyond tractable?

  2. Do HodgeRank, Spectral MLE, and Active top-k receive hard anchoring on measured values and the regressor prior θᵢ? If they do not, the reported gaps may reflect anchoring rather than information-guided escalation.

  3. How does PRISMS perform with a criterion targeting the top-k decision boundary, in the spirit of Mohajer et al. (2017), which you cite but do not compare against? D-optimality minimizes global parameter uncertainty, whereas the task is top-k identification.

  4. What is the aggregate pairwise accuracy of the LLM ranker pooled across Figure 8? If it falls below 60%, the entire sensitivity grid in Section 4.5 lies outside the empirically observed operating regime.

  5. What fraction of campaigns reach 50% top-10 recall by round 24 for each method? With Random Sampling at RMST 23.3 against a horizon of 24, the statistic appears censoring-dominated, which would undermine the translation into 8–24 months of saved cycle time.

2 comments

We appreciate that the reviewer found that our proposal to shift curation from predicting absolute values to comparing candidates substantive and well argued, with strong empirical and methodological assessment. We address the reviewer’s concerns below.

Which component dominates the runtime and how to scale to libraries with one million candidates?

Graph construction accounts for approximately 97% of the total runtime in our analysis. During this step, PRISMS connects each query node to its nearest neighbors in the property space. To reduce this overhead, we consider two alternatives: (1) pre-sorting the estimated scores once per round to avoid repeated nearest-neighbor searches, and (2) randomly connecting query nodes instead of selecting nearest neighbors.

In a new experiment on synthetic data, when the candidate size grows from 10410^4 to 10510^5 PRISMS runtime increases from 15.415.4 s to 2,0452,045 s. Pre-sorting the estimates for all query nodes reduces the runtime to 8.18.1 s and 858858 s, respectively, but the scaling remains roughly quadratic. Random assignment achieves the best runtime performance, scaling nearly linearly from 1.061.06 s to 14.4514.45 s. Even at a candidate size of 10610^6, random assignment completes in 152152 s per round.

However, random assignment leads to a slight performance drop, as shown in Table 2 in the Appendix. Users should consider this tradeoff between runtime and performance.

Do the ranking-based baselines receive hard anchoring on measured values?

Yes. Similar to PRISMS, the ranking-based methods (HodgeRank, Spectral MLE, and Active top-k) keep the measured nodes’ values fixed once their labels are revealed.

How does PRISMS perform with a criterion targeting the top-k decision boundary, in the spirit of Mohajer et al. (2017)

We show, in the paper, the results from Mohajer et al. (2017) method under the label “Active top-k”. We omitted the method from Figure 3 (left) and Figure 10 (left) for clarity as it performs similarly to HodgeRank. The results from Mohajer et al. (2017) method are still available in Figure 3 (right), Table 1, and Figure 10 (right). Overall, PRISMS consistently outperforms the method.

What is the aggregate pairwise accuracy of the LLM ranker pooled across Figure 8? Claude Opus 5.5 High Effort achieves 57.1% over 10,700 comparisons pooled across three datasets and three seeds, which the reviewer correctly recognizes is below the lowest setting in the sensitivity grid (60%).

In a new, more extensive experiment, we use TabPFN 2.5 as the realistic higher-fidelity ranker. As TabPFN allows “in-context learning” during inference, its pairwise ranking accuracy increases with the number of labels, spanning 63.20% to 71.06% which are inside the hyperparameter grid of the sensitivity analysis. Detailed accuracies are listed in Table B.1.

Table B.1. Pairwise ranking accuracy of TabPFN 2.5 across varying number of labels revealed (context)

# Labels Revealed102030405060708090100110120
Accuracy63.20%65.81%66.14%67.86%67.71%67.98%69.30%69.79%69.57%69.82%70.35%71.06%

We use the TabPFN ranker for screening campaigns with similar setup as in Section 4.1 (14 datasets, 10 random seeds, 12 rounds). In this new experiment, PRISMS achieves 0.323 discovery-recall AUC, compared to 0.285 for RankRefine++, the second-best method we evaluated. The results, shown in Table B.2. below, demonstrate that PRISMS retains clear advantage when using realistic, dynamic, higher-fidelity rankers.

Table B.2. Experimental results for iterative screening in drug discovery datasets using TabPFN 2.5 as the higher fidelity ranker, across 10 random seeds.

MethodDiscovery recallNDCGTop-1% F1Top-5% F1
Random sampling0.076 (0.006)0.334 (0.020)0.120 (0.010)0.134 (0.006)
GP-EI0.139 (0.011)0.401 (0.020)0.190 (0.014)0.205 (0.009)
GP-EI Pairwise0.221 (0.014)0.403 (0.022)0.306 (0.018)0.252 (0.010)
HodgeRank0.285 (0.018)0.473 (0.020)0.349 (0.020)0.327 (0.013)
Active top-k0.283 (0.018)0.476 (0.020)0.239 (0.013)0.308 (0.011)
Spectral MLE0.271 (0.018)0.469 (0.020)0.340 (0.020)0.304 (0.012)
RankRefine++0.285 (0.019)0.480 (0.019)0.341 (0.021)0.325 (0.014)
PRISMS0.323 (0.018)0.507 (0.020)0.352 (0.018)0.354 (0.013)

What fraction of campaigns reach 50% top-10 recall by round 24 for each method? If results are dominated by censoring (i.e., many campaigns have not reached 50% recall by round 24 but are assigned a round-to-50%-recall = 24 anyway), the translation into 8-24 months of saved cycle time would be undermined.

As shown in Table B.3., PRISMS and RankRefine++ are not dominated by the censoring, while GP-EI barely passes the 50% threshold. Therefore, the translation into 8-24 months of saved cycle time over conventional Bayesian approach can be seen as a likely lower bound even under the censoring (since significantly more GP-EI campaigns are censored compared to PRISMS). We will mention this in the paper.

Table B.3. RMST to 50% Recall and fraction of campaigns reaching 50% recall for selected methods.

Method24-round RMST to 50% RecallFraction of campaigns reaching 50% recall by round 24
PRISMS11.187.1%
RankRefine++13.184.0%
GP-EI19.352.9%
Random sampling23.312.6%
Reviewer_t3JROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Scientific discovery can generate a huge amount of candidate designs, but experimental evaluations remain slow and expensive. Instead of training regressors on scarce experimental data, this paper proses PRISMS, a curation framework that aggregates pairwise rankings from multiple sources and selectively escalates comparisons from low to high fidelity rankers. In iterative screening that selects designs from fixed drug discovery libraries, PRISMS achieves 50% top-10 discovery recall in ~42% fewer rounds than regression-only active learning, and in ~15% fewer rounds than the ranking-based method with no selective escalation.

Critical strengths

Strengths:

  • Well motivated problem and clear framing.
  • The framework is principled and coherent, tightly integrates pairwise ranking, multi-fidelity supervision, and information-guided escalation.
  • Extensive empirical evaluation across drug-discovery and engineering datasets with multiple seeds, including ablations and sensitivity analyses.
Critical weaknesses

Weaknesses:

  • Limited methodological novelty. The main methodological contribution lies in integrating existing techniques into a coherent framework.
  • Optimistic high fidelity ranker. The main results highly rely on simulated high fidelity ranker that is uniformly 75% accurate across all pairs. This is more idealized than realistic settings, and the LLM ranker evaluated in the paper doesn’t reach that level. The gain over RankRefine++ becomes modest in low-accuracy or structured-noise settings. I suggest reporting the main results under a more realistic noise model.
  • Limited validation with realistic rankers. The only real ranker experiment (an LLM ranker) is small (3 tasks, 3 seeds) and only reported in the appendix.
Questions for authors

Questions

  • Does PRISMS retain clear advantage over RankRefine++ with more realistic higher fidelity rankers? Could the authors expand the LLM-ranker experiment (more tasks and seeds, different LLMs), with baselines using the same LLM ranker as controls?
  • In practice, ranker reliability may vary and the fidelity order is not always fixed (i.e., g2 may be less accurate than g1 on some pairs). How robust is PRISMS to this case? Could the ranker weights be adapted online?
1 comment

We appreciate the reviewer’s recognition of PRISMS’s clear framing, principled design, and extensive evaluation. We address the reviewer’s concerns below.

Methodological Contribution

We agree with the reviewer that PRISMS’ methodological strengths lie in bringing together established yet fragmented techniques into one principled and coherent framework to solve a specific problem: how to combine scalar measurements with multifidelity pairwise rankings to curate computational scientific designs. To the best of our knowledge, our work is the first that proposes such an algorithm. We also introduce a new position: as the generation of candidate designs outpaces the capacity for physical validation, the scientific design discovery bottleneck shifts from generation to curation. In this data-scarce regime, we advocate that scientific discovery should be driven by pairwise rankings rather than regressions.

Performance with realistic rankers

In a new experiment, we replaced the 75%-accurate simulated ranker with TabPFN 2.5, a tabular foundation model. TabPFN is chosen here over proprietary LLMs (such as Claude Opus 5.5 in our previous experiments) as it is open-source and can be run on a local GPU, making it more accessible to a larger audience. As TabPFN allows “in-context learning” during inference, its pairwise ranking accuracy increases with the number of labels. Table A.1. below shows the average pairwise ranking accuracy of TabPFN 2.5 across 12-round iterative screening campaigns on 14 datasets.

Table A.1. Pairwise ranking accuracy of TabPFN 2.5 across varying number of labels revealed (context)

# Labels Revealed102030405060708090100110120
Accuracy63.20%65.81%66.14%67.86%67.71%67.98%69.30%69.79%69.57%69.82%70.35%71.06%

We use the TabPFN ranker for screening campaigns with similar setup as in Section 4.1 (14 datasets, 10 random seeds, 12 rounds). In this new experiment, PRISMS achieves 0.323 discovery-recall AUC, compared to 0.285 for RankRefine++, the second-best method we evaluated. The results, shown in Table A.2. below, demonstrate that PRISMS retains clear advantage when using realistic, dynamic, higher-fidelity rankers.

Table A.2. Experimental results for iterative screening in drug discovery datasets using TabPFN 2.5 as the higher fidelity ranker, across 10 random seeds.

MethodDiscovery recallNDCGTop-1% F1Top-5% F1
Random sampling0.076 (0.006)0.334 (0.020)0.120 (0.010)0.134 (0.006)
GP-EI0.139 (0.011)0.401 (0.020)0.190 (0.014)0.205 (0.009)
GP-EI Pairwise0.221 (0.014)0.403 (0.022)0.306 (0.018)0.252 (0.010)
HodgeRank0.285 (0.018)0.473 (0.020)0.349 (0.020)0.327 (0.013)
Active top-k0.283 (0.018)0.476 (0.020)0.239 (0.013)0.308 (0.011)
Spectral MLE0.271 (0.018)0.469 (0.020)0.340 (0.020)0.304 (0.012)
RankRefine++0.285 (0.019)0.480 (0.019)0.341 (0.021)0.325 (0.014)
PRISMS0.323 (0.018)0.507 (0.020)0.352 (0.018)0.354 (0.013)

Dynamic Ranker Accuracy / Fidelity.

In principle, the weights in Equation 2 can be adjusted online based on prior knowledge of each ranker's reliability in the target domain. For example, if ranker g2g_2 is known to be less reliable than g1g_1 when comparing pairs (x1,x2)(x_1, x_2) from subgroup XaX_a, we can assign it a lower weight: λ1(2,a)<λ1(1,a)\lambda_1^{(2,a)} < \lambda_1^{(1,a)}.

However, such adaptation requires knowledge about the domain and the ranker. For example, comparing similar molecules may be easier for a chemist because their structural differences are easier to isolate, whereas comparing similar airfoils may be harder for an engineer because their lift differences are less apparent. Thus, we leave online adaptation as potential future work that is out of scope for our current submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.