- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes PRISMS, a framework for scientific design selection using multi-fidelity pairwise rankings instead of relying primarily on regression models trained with experimental measurements. It combines rankings from different sources through a regularized Bradley–Terry model and selectively queries higher-fidelity rankers using a Fisher information-based criterion. Experiments on drug discovery and engineering datasets show improved screening efficiency and ranking performance over existing baselines. The method is also evaluated on multi-objective molecular optimization, with additional ablation and sensitivity studies.
Critical strengths
- The motivation is interesting and well justified, as pairwise comparisons can be easier to obtain than accurate absolute property predictions, especially when experimental data is scarce.
- The proposed framework provides a reasonable way to combine pairwise rankings from different sources with available experimental measurements. The information-based selection of higher-fidelity queries is also well motivated.
- The experiments cover 14 drug discovery datasets, three engineering datasets, and a molecular optimization task. PRISMS achieves consistent improvements over several regression-based and ranking-based baselines.
- The ablation and sensitivity analyses are fairly comprehensive, showing the effects of graph connectivity, query selection, ranker accuracy, and query budgets.
Critical weaknesses
- Most experiments rely on a simulated high-fidelity ranker with 75% accuracy, constructed by randomly flipping ground-truth comparisons. While useful for controlled evaluation, this setting does not fully reflect the reliability and biases of real scientific experts or computational models, limiting the evidence for practical applicability.
- The evaluation with a real LLM ranker is limited to three tasks and three random seeds, with considerably lower performance than the simulated oracle. Moreover, the lack of comparisons with baselines using the same LLM ranker makes it difficult to assess the effectiveness of PRISMS in realistic settings.
- The method assumes a fixed ordering of ranker reliability and uses fixed weights for each ranker. However, the LLM analysis shows that ranking accuracy varies substantially with the property difference between candidates. Although different noise settings are evaluated, the query selection criterion does not explicitly account for such variations in ranker reliability.
Questions for authors
- Could the authors clarify how the two independently predicted objectives are used to select candidates in the multi-objective optimization setting?
2 comments
Thank you for the detailed response and additional experiments, which have adequately addressed my concerns. I will maintain my positive rating and wish the authors the best of luck!
We are glad that the reviewer found the work interesting and well justified, with extensive experiments, ablations, and sensitivity analysis. We address the reviewer’s concerns below.
Expanding Experiments with Non-Simulated / Realistic Ranker
In a new experiment, we replaced the 75%-accurate simulated ranker with TabPFN 2.5, a tabular foundation model. TabPFN is chosen here over proprietary LLMs (such as Claude Opus 5.5 in our previous experiments) as it is open-source and can be run on a local GPU, making it more accessible to a larger audience. As TabPFN allows “in-context learning” during inference, its pairwise ranking accuracy increases with the number of labels. Table C.1. below shows the average pairwise ranking accuracy of TabPFN 2.5 across 12-round iterative screening campaigns on 14 datasets .
Table C.1. Pairwise ranking accuracy of TabPFN 2.5 across varying number of labels revealed (context)
We use the TabPFN ranker for screening campaigns with similar setup as in Section 4.1 (14 datasets, 10 random seeds, 12 rounds). In this new experiment, PRISMS achieves 0.323 discovery-recall AUC, compared to 0.285 for RankRefine++, the second-best method we evaluated. The results, shown in Table C.2. below, demonstrate that PRISMS retains clear advantage when using realistic, dynamic, higher-fidelity rankers.
Table C.2. Experimental results for iterative screening in drug discovery datasets using TabPFN 2.5 as the higher fidelity ranker, across 10 random seeds.
Fixed ordering of ranker reliability and fixed weights for each ranker
In principle, the weights in Equation 2 can be adjusted online based on prior knowledge of each ranker's reliability in the target domain. For example, if ranker g2 is known to be less reliable than g1 when comparing pairs (x1,x2) from subgroup Xa, we can assign it a lower weight: λ1(2,a)<λ1(1,a).
However, such adaptation requires knowledge about the domain and the ranker. For example, comparing similar molecules may be easier for a chemist because their structural differences are easier to isolate, whereas comparing similar airfoils may be harder for an engineer because their lift differences are less apparent. Thus, we leave online adaptation as potential future work that is out of scope for our current submission.
That being said, the new results with TabPFN as a realistic ranker show that PRISMS still outperform the baselines on more realistic ranker regimes.
How are candidates selected in multi-objective optimization?
On round t, a method estimates n objectives independently, resulting in each candidate represented by n predicted scores. The promising candidates are then selected from the n-dimensional Pareto front of size m. For a validation budget of b, b candidates are sampled uniformly from the Pareto front if b<m. If b>m, the full Pareto front candidates are selected and then augmented with (b−m) non-Pareto front candidates with the highest mean normalized scores.