acceptodds
Withdrawn from ICLR 2027

Proving the Proxy Wrong: Witness Rank for Auditing Scientific Design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Proxy-guided scientific design can rank far more candidates than trusted experiments can evaluate, yet standard metrics do not quantify the evidence for the single candidate ultimately selected. We ask how many trusted evaluations are needed to overturn that nomination. On a declared finite pool, the margin-aware *witness rank* is the position, under an ordering fixed without trusted labels, of the first candidate that improves on the nomination by more than . Across 230 audits on four fully measured protein and DNA landscapes, including two published offline optimizers, conditional median witness ranks are three to five despite the largest pool containing over 150,000 candidates. At sixteen trusted evaluations, proxy ordering achieves 1.68 to 8.60 times the exact random-order recall. Controlled rankings with Spearman correlation near 0.95 among the top candidates by measured performance still yield conditional median witness ranks of two to four, showing that high ranking quality alone does not determine audit cost. The same audit measurements identify replacements that cannot reduce measured performance, while independent random probes bound pool witness prevalence when no witness is found. To complement these retrospective biological audits, we demonstrate the auditing and replacement protocol in a prospective laboratory study of ionic conductivity in battery electrolytes. Together, these results make post-selection reliability measurable in experimental-budget units and guide whether to retain a nomination or replace it with a measured improvement.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for science, offline model-based optimization, surrogate models, model auditing, experimental design

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_vCP7OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes witness rank to measure how many trusted evaluations are needed to overturn a proxy nomination. The authors audit four landscapes at a reasonably large scale and find a median rank of only 3–5; top-tail correlation does not predict audit cost. In my view, the main value is making audit cost explicit and providing systematic empirical evidence, not proposing a new method. The audit itself does not improve the proxy or optimizer, so the technical increment is limited.

Critical strengths
  1. The empirical study is fairly systematic. Multiple landscapes, multiple proxy families, matched-budget campaigns, and reproductions of published optimizers are uncommon in audit-style work.
  2. Proposition 2 is a clean and citable separation result: top-tail correlation of 1 can coexist with arbitrarily large witness rank.
Critical weaknesses
  1. Technical novelty is limited. The paper does not propose a new optimizer, proxy architecture, or learning algorithm; it combines known ingredients—uniform error bounds, tail-correlation limitations, hypergeometric exact bounds, best-observed delivery—into an audit protocol.
  2. The paper lacks comparison with per-candidate certification. The authors abandon split conformal due to calibration-size arithmetic and turn to a pool-level random-probe bound, but do not discuss the directly relevant line on conformal candidate certification.
Questions for authors
  1. If the trusted score is unreliable on the deployment distribution, what exactly does certification guarantee? Should the protocol include an oracle-validity check?
  2. Why is a pool-level witness-prevalence bound more appropriate than a per-candidate conformal certificate?
Reviewer_RVg5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the setting of finite pool multi fidelity or surorgate guided optimization.

For a finite subset X∈Pfin(X),X⊂XX \in \mathcal{P}_{fin}(\mathcal{X}), X\subset \mathcal{X}  of a domain X\mathcal{X} with  $N=| X | $ candidates, they consider an (expensive) reference score f: X→Rf: \mathcal{X} \to \mathbb{R}   and a (cheap) proxy f^:X→R\hat{f}:\mathcal{X} \to \mathbb{R}  derived from a disjoint training dataset D⊂X∧D∩X=∅\mathbb{D}\subset \mathcal{X} \land \mathbb{D} \cap X = \empty .

(my notation for the search algorithm)

A search algorithm AA turns these scorers into a nomination of xA/x^A∈Xx_A/\hat{x}_A \in X using an evaluation budget Bsearch∈NB_{search} \in \mathbb{N} , i.e. AA takes the scorer, a search budget and a finite set and returns an ordering on a BsearchB_{search} subset ΠBsearch(X):X→[1,…,Bsearch]∪∞\Pi_{B_{search}}(X): X \to [1,\dots,B_{search}] \cup {\infty} assigning ranks 1 to BsearchB_{search} to all candidates encountered during the search and a “not ranked” value of ∞\infty  to all elements of the finite subset.

  A:((X→R)× N× Pfin(X))→ΠB(X)A: \left( \left(\mathcal{X} \to \mathbb{R}) \times \mathbb{N} \times  \mathcal{P}_{fin }(\mathcal{X}\right)\right) \to \Pi_{B_{}}(X) .

The paper then introduces the  notion of a Δ\Delta witness rank RΔR_\Delta defined as the number of evaluations order until one can find a proxy error RΔ≔min⁡{r:f(xr)≥f(x^sel+Δ)}R_\Delta \coloneqq \min \lbrace r:f(x_r) \geq f(\hat{x}_{sel}+\Delta) \rbrace when following the proxy ranking ΠBsearch(X)\Pi_{B_{search}}(X), subject to a certification budget of BcertB_{cert} that bounds the size of the auditing set SBS_B (ins some places of the paper SBS_B need not follow the proxy ranking ΠBsearch\Pi_{B_{search}} or be ordered )  .

The paper then claims sharp worst case bounds on audit costs (theorem 1), considers top tail correlations relationship with with audit cost (prop 2) discusses the difficulty of giving confidence bounds ( line 209ff) and empirically studies RΔR_\Delta on  constructed datasets based of GB1, TrpB TF-Bind-8 and avGFP binding landscape datasets using four proxy families (pairwise/additive regression, extratree ensemble on token features, pairwise ranking and an objective surrogate) with 2 ranking models (RaM-ListNet, COMs)   at different certification budgets.   Additionally, they claim a real-world validation on prospective electrolyte selection.

They also study how much using a found Δ\Delta witness that improves over x^sel\hat{x}_{sel}  improves the total regret compared to following the original, audited selection.

Critical strengths

None.

Critical weaknesses

The paper is illegible in the current form due to AI slop, and does not contribute meaningful novel ideas.  I did my best to parse it with extensive AI help but am sorry to come to this conclusion.

  1. Presentation: some examples of AI slop

    1. no citations except in introduction, prior art and when referencing datasets or baselines, despite extensive re-invention of existing methods (see point 2)

    2. no motivation or clear throughline of why we should care or what implication the papers “analysis” implies. Only the chatgpt flavoured non-consequtior chaining of confidence-ish statements of “Noun (statement). Noun (statement)”.  with weak hedging and no grounding

      1. Examples: line 33ff “Scientific design pipelines (…). A trusted score (…). A poor result (..). An optimizer (…). Bettetr search (…). … Average prediction error (…)”

      2. lines 75ff and 87ff similarly chain the referencs without throughline

      3. The notation is messy and overloaded and obfuscatd more than it explains (which I tried to rectify in my notation above). The space of which the papers pool and the disjoint training dataset are drawn is never cleanly stated,  whether or not SBS_B is ordered changes without clear motivation or acknowledgement throughout the paper. new notions are overloaded onto the existing notation as section string along more concepts (e.g. xsafe(B)x_{safe}(B) in line 202)

      4. lots of space is spent explaining what the paper does not prove (e.g. the vacuous  confidence bounds in 212ff, the explicit mention of a not selected dataset i the main body in 240ff

    3. on a higher level: what is the point of the papers analysis? i can’t tell after carefully trying to make sense of it, you can construct arbitrary counter examples for which the witness rank becomes vacuous and no certificates are given nor is it motivated why we should care about the empriical evaluation

  2. The paper doesn’t engage with existing lineages that quite obviously provided the shape of the results,e.g.

    1. information retrieval https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings17/pdf/evia/04-EVIA2023-EVIA-DiazF.pdf  (

    2. optimal auditing (the papers computed probability bound is literally this wikipedia example https://en.wikipedia.org/wiki/Hypergeometric_distribution#Application_to_auditing_elections  as best I can tell)

    3. multi fidelity optimization e.g. bandits https://arxiv.org/abs/2601.20989 and kriging (the https://arxiv.org/abs/2601.20989 reference contains moreor less the setup of the paper up to reformulation on page 4, under probabilistic assessment)

    4. rank verification https://arxiv.org/pdf/1610.03944 which is a decades old field trying to give certificate bounds on exatly the question the paper studies (quoting from my reference,

      > This article concerns the problem of rank verifica-tion — post hoc significance tests of whether the orderings discovered in the data reflect the population ranks.

Questions for authors

None.

Reviewer_UvzQOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

I don’t have sufficient background in scientific design, so my review may sound a bit ill-informed or naive. Please keep in mind that this is not my domain of expertise.

From what I understood, the paper proposes a method to find out whether a nominated candidate can be outperformed and, if so, whether this can be shown with a small number of trusted evaluations or experiments. The authors nominate a candidate through proxy-guided optimization, then rank the selection list of candidates for testing, often based on their proxy scores. They test the top (B) ranked candidates to find whether any outperform the nominated candidate, either by any amount or by a specified margin. The position of the first candidate that does so is called its witness rank. Witness rank can be used to compare audit outcomes for different nominations and testing orders, although a higher witness rank alone may not necessarily indicate a better optimizer or model.

Critical strengths

These are the main paper strengths in my opinion :

  1. The experiments cover four different datasets, so the method is evaluated across a variety of test cases.

  2. All propositions are accompanied by proofs in the appendix, providing theoretical support for the proposed method.

  3. Prioritizing candidates for trusted evaluation using their proxy rankings is an intuitive and practical choice when experimental evaluations are limited.

Critical weaknesses

Main Weaknesses in my opinion:

  1. The paper was difficult for me to read and understand. This may partly reflect my limited background in scientific design, but I found it hard to develop an intuition for the method and identify the paper’s main contribution from the current presentation.

  2. The related work section mentions papers around lines 78 and 82 that also use proxy rankings, possibly to nominate candidates. As I understand it, this paper’s contribution is to evaluate a fixed, ranked set of candidates to find a measured replacement for the nomination and, in doing so, show that the nomination is suboptimal. This feels somewhat limited in novelty to me. Is the proposed audit closely related to top-(k) evaluation, which has been used extensively in other areas such as computer vision like clustering, classification ? If so, it would help to explain more clearly what is new about applying this approach to scientific design.

  3. The main results appear to compare the proposed audit primarily with random probing. Although the paper also evaluates other acquisition strategies, including Thompson sampling and batch Bayesian optimization, it is difficult to tell how they compare on the central question: how many trusted evaluations are needed to find a candidate that disproves a nomination. Could the authors compare these strategies under the same nomination, candidate pool, margin, and evaluation budget, or explain why such a comparison is not appropriate?

Questions for authors

Questions for Authors:

  1. Could the authors clarify how distribution-free inference and best-arm identification relate to the proposed audit? As I understand it, this paper tests whether a nominated candidate can be outperformed; it does not propose a method for finding the best candidate in the pool. I found it difficult to understand the connection when reading this part of the related work section.

  2. Do the authors provide a theoretical bound, or a probabilistic guarantee, on how close the replacement candidate’s trusted score is to the pool’s optimal trusted score? I understand that the paper reports empirical results around line 222, but I could not find a theoretical guarantee for the quality of the replacement relative to the optimum.

Reviewer_49ZKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines how many trusted evaluations are needed to overturn a proxy-guided recommendation in scientific design. The authors introduce margin-aware witness rank, the position of the first candidate whose trusted score exceeds that of the recommendation by a specified margin, under an ordering fixed without access to unobserved trusted scores. They derive worst-case audit-cost bounds and connect auditing to a simple replacement rule that returns the best evaluated candidate. Experiments on measured protein and DNA landscapes show that qualifying improvements can often be found with few trusted evaluations, with proxy ordering outperforming random probing. A prospective electrolyte study further illustrates the procedure by identifying measured improvements over several original recommendations.

Critical strengths
  1. Clear error decomposition between search and selection failures. The paper separates errors due to incomplete search from those caused by proxy-based selection, clarifying why further optimization of the proxy may not improve the recommended design.
  2. Diverse experimental evaluation. The experiments cover measured protein and DNA landscapes, controlled mixture problems, and an electrolyte study, complemented by comparisons across proxy models and auditing strategies.
Critical weaknesses
  1. Insufficient positioning relative to established results and metrics. The relationship to established tools deserves clearer discussion. The witness-prevalence bound uses standard hypergeometric tail inversion [1], while the uniform-error regret bound follows the same approximation argument as the ERM guarantee in Lemma 4.2 of [2]. Witness rank is also closely related to fixed-target runtime evaluation [3]. These connections should be discussed to clarify which contributions lie in the audit framing, the witness-rank bounds, and the empirical findings.

For a fixed ordering and improvement threshold, Equation (5) identifies the distribution of witness rank with the corresponding success-versus-budget curve, and witness rank can be recovered from the complete per-run best-of-budget trajectory. The nomination-relative interpretation is useful, but its additional explanatory or decision-making value beyond these established evaluations remains insufficiently justified. Dense, repetitive presentation in both the main text and parts of the appendix further obscures this distinction; clearer organization and transitions would help readers identify the central contributions.

  1. The prospective study provides limited evidence for improvement under measurement uncertainty. Appendix O reports one measurement per formulation and computes witness ranks directly from the observed conductivities, without uncertainty bounds on the gains. The additional analysis using a 0.50 mS/cm improvement margin is post hoc. Although the study illustrates the auditing procedure, it does not establish that the replacements improve underlying performance with statistical confidence. Repeated measurements or validated measurement-error estimates, together with an appropriate noisy certificate as discussed in Appendix A.1, would strengthen the practical evidence.

References

[1] Jay Bartroff, Gary Lorden, and Lijia Wang. Optimal and Fast Confidence Intervals for Hypergeometric Successes. The American Statistician, 77(2):151–159, 2023. See Section 5, Equation (20).

[2] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014. See Lemma 4.2.

[3] Nikolaus Hansen, Anne Auger, Dimo Brockhoff, Dejan Tušar, and Tea Tušar. COCO: Performance Assessment. arXiv:1605.03560, 2016. See Sections 2.1–2.2.

Questions for authors
  1. How sensitive is the high-quality ranking result to the independent Gaussian perturbations used in Appendix E.2? A small controlled comparison with sequence-correlated or subgroup-biased proxy errors, at similar achieved ranking quality, would help assess whether the observed low audit costs depend on this construction.

  2. For the actual COMs outputs in the constrained-pool experiments (§5.5; Appendix G), could the authors report witness existence and certification recall conditional on witness existence, alongside the unconditional top-16 rates of 0–0.2? This would distinguish absent witnesses from late-ranked witnesses and clarify how the low-cost auditing conclusions extend beyond exact proxy optima.

Comments on the paper

1 comment

We thank the reviewers and Area Chair for the time and effort devoted to reviewing our submission.

We have decided to withdraw the submission. We appreciate the feedback and suggestions, which will help inform future revisions and the continued development of this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.