- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes witness rank to measure how many trusted evaluations are needed to overturn a proxy nomination. The authors audit four landscapes at a reasonably large scale and find a median rank of only 3–5; top-tail correlation does not predict audit cost. In my view, the main value is making audit cost explicit and providing systematic empirical evidence, not proposing a new method. The audit itself does not improve the proxy or optimizer, so the technical increment is limited.
Critical strengths
- The empirical study is fairly systematic. Multiple landscapes, multiple proxy families, matched-budget campaigns, and reproductions of published optimizers are uncommon in audit-style work.
- Proposition 2 is a clean and citable separation result: top-tail correlation of 1 can coexist with arbitrarily large witness rank.
Critical weaknesses
- Technical novelty is limited. The paper does not propose a new optimizer, proxy architecture, or learning algorithm; it combines known ingredients—uniform error bounds, tail-correlation limitations, hypergeometric exact bounds, best-observed delivery—into an audit protocol.
- The paper lacks comparison with per-candidate certification. The authors abandon split conformal due to calibration-size arithmetic and turn to a pool-level random-probe bound, but do not discuss the directly relevant line on conformal candidate certification.
Questions for authors
- If the trusted score is unreliable on the deployment distribution, what exactly does certification guarantee? Should the protocol include an oracle-validity check?
- Why is a pool-level witness-prevalence bound more appropriate than a per-candidate conformal certificate?
Everyone
We thank the reviewers and Area Chair for the time and effort devoted to reviewing our submission.
We have decided to withdraw the submission. We appreciate the feedback and suggestions, which will help inform future revisions and the continued development of this work.