acceptodds
Withdrawn from ICLR 2027

Winners Need Witnesses: Certified Replay Selection and Reporting with PAIRS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selecting a good configuration and estimating its performance are distinct statistical tasks. We formulate joint selection and reporting: under a compute budget, a procedure returns an ε-optimal configuration and an η-accurate performance estimate with a joint certificate, or abstains. Shared randomness can make comparisons nearly noiseless while absolute-performance estimation remains costly. We establish a finite-pool minimax characterization and, for two candidates with known Gaussian covariance and complete-pair queries, instance-dependent bounds matching up to logarithmic factors. PAIRS combines paired sequential selection with protected reporting. A report-aware controller has a variance- and gap-dependent completion bound; paid variance learning can change the cheapest acceptable winner. Under the stated information-use assumptions, reports are selection-conditionally unbiased and incorrect joint certification has probability at most δ. Four replay benchmarks yield reference-relative Pε = 93.2%, versus 81.1% for uniform allocation at equal mean selection cost. In a 20-run Split-CIFAR-10 task-order study, fresh same-winner reporting reduces referencerelative macro absolute signed bias by 92%. A 1,000-pair application evaluation gives a 2.30 ratio between the smallest evaluated budgets meeting prespecified completion and correct-joint-output targets, with reporting and audit charged and abstentions counted as failures. A heterogeneous-variance study shows that additional comparisons can reduce total cost by shifting selection toward an easierto-report acceptable candidate. PAIRS makes reporting precision and completion explicit objectives of replay selection.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
replay selection, winner’s curse, post-selection inference, confidence sequences, uncertainty quantification, shared randomness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZKeBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how to select an acceptable configuration and estimate its performance accurately under a limited compute budget. The procedure provides a statistical guarantee for both selection and estimation, or abstains. PAIRS uses comparisons under shared randomness and reports performance using independent new observations, or by splitting Gaussian observations into independent selection and reporting data. It also considers further comparisons to reduce reporting costs, provides Gaussian and finite-pool complexity results, and evaluates the method in replay experiments and simulations.

Critical strengths
  1. The problem is clearly defined. The paper separates selection quality, reporting accuracy, and completion, keeping abstentions in the denominator. This evaluates all runs rather than only successful returns. The distinction is useful in practice: selecting a good configuration does not justify reporting its winning score as its true performance.

  2. The theory explains why comparisons and performance estimation have different costs. Theorems 2–3 use the same paired observation scheme and cost calculation for their lower and achievable upper bounds. Even when comparison noise is small, estimating a candidate's mean performance can require many observations. This explains both the benefit and the limits of shared randomness.

  3. The authors specify which data selection and reporting may use. For Gaussian observations with known covariance, Proposition 1 and Appendix B show that the selection and reporting data are independent when the stated rules for data use are followed. This supports conditionally unbiased estimation after selection and stopping. The paper also distinguishes this method from reporting with independent new observations.

  4. The controlled comparisons show when including reporting costs in selection helps. Appendix Z.9 compares four policies, separating pairing, choosing a candidate with a lower reporting cost among certified candidates, and further comparison before reporting. With only one acceptable candidate in Table 75, further comparisons provide no additional benefit. These results support the claim that more comparisons can sometimes reduce total cost and clarify the conditions needed.

Critical weaknesses
  1. The certification rule in the main replay experiment is not sufficiently specified. Table 1 reports 177 certified outputs among 190 runs, with approximately one to two selection observations per candidate in several settings. Appendices B and I explicitly use different confidence boundaries or sampling schedules and do not directly explain these runs. I could not find the actual rule's complete formula, assumptions, and error allocation, so I cannot yet check whether these certificates satisfy Eq. (15). The counts alone do not establish invalidity; another valid boundary or additional information could explain them.

  2. The additional benefit of including reporting costs in selection remains unclear in real replay experiments. Appendix Z.9 already compares the policies in a Gaussian model, but Appendices V and W primarily compare the methods as a whole. W changes whether observations are paired, the confidence boundaries, and the rules for evaluating candidates. It therefore does not separately measure the benefit of further comparisons. The authors should identify which parts of PAIRS were used in each experiment and whether existing records establish an additional benefit. This does not invalidate the comparison between the methods as a whole.

  3. Independent training runs need a clearer link to the records of computational cost. Appendix V states that each observation used for reporting comes from an independent complete training run, while Table 76 converts counts using an average runtime. The aggregate tables supplied here are insufficient to check independence, failure handling, and the reporting sample's mean and variance. Linking observations to individual training records would help establish whether the assumptions for independent reporting hold and whether the cost results can be reproduced.

  4. Repetition and missing transitions make the paper harder to follow. Sections 3, 5, and 6 move directly from headings into subsections without explaining how the parts connect. Appendix J.1 introduces a study again after summarizing it, and the context for Section 6.3's two ratios of 2.30 is scattered across appendices. Some passages read as heavily AI-assisted prose that has not been carefully edited. I recommend adding section introductions, removing repeated openings, and explaining the experimental units and cost measures in one place.

Questions for authors
  1. Which confidence sequence was used for Table 1? Please state its assumptions, error allocation, and any independent pilot or prior information and associated costs. Please provide a certified run's full record, including the candidates evaluated, paired observations available at each step, intervals, eliminations, and stopping decisions.

  2. Which parts of PAIRS were used in Appendices V and W? W's 2.30 budget ratio compares the methods as a whole and does not separately measure further comparisons to reduce reporting cost. Are controls that use paired comparisons for selection with the same independent reporting method, or records of the reporting budget reserved for each candidate, available to clarify the additional benefit in replay experiments?

  3. Which observation and training records establish that Appendix V's aggregate counts come from independent complete training runs? These records should allow the reporting sample's mean, variance, and failure handling to be checked. Table 76's estimate of GPU-hours from average runtime does not establish those links.

Reviewer_zZWfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the problem of how to identify and estimate a good configuration with limited budget, a problem that is close to bandit problem with goal of optimizing the selection bias and the cost. The authors called their method as 'guarded continuation and protected reporting'. A theoretical result is provided that bounds the error probability where either the identification or the estimation task fail.

Critical strengths

none.

Critical weaknesses

The main concern about this paper is a significant amount of writing that is unclear and unconventional that makes reading very hard, this includes

  1. PAIRS is never defined what it shorts for and used directly from abstract.
  2. i believe the problem itself can be framed with just a little bit modification to established bandits problem instead of using the words of selecting, reporting etc that is too conversational sometime with less rigor in many places.
  3. there are also large concerns about reference. the related work section is purely listing some papers without any clear discussion on how they connect to the paper. in the main body of the paper, i also see improper missing cite. for example, line 185, median elimination suddenly jumps out, without it being cited / explained on what it is, while it is indeed a bandit algorithm, till later on, it appears again in the appendix.
  4. the main algorithm alg1 has readability issue. many of the very conversational sentence such as 'if no complete admissible query is affordable' appear without clear mathematical meaning, also alg2 is quoted in alg1, but there is not any discussion on what alg2 is and why necessary here. later it appears in appendix, but another hard reading one.
Questions for authors

In addition to the concerns in weakness above. How does the result in theorem 4 compared / connected to many of the classic results on multi-armed bandit problem, especially best arm identification problem, or more practical version of best arm identification problem that has been studied such as hyperband, which is included in reference as well, but not discussed.

Reviewer_myU7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript proposes PAIRS which combines paired sequential selection with protected reporting.

Critical strengths

None

Critical weaknesses

W1: It’s obvious that the figure is AI generated, which is hard to understand what is being expressed. W2: This manuscript lacks logical coherence, there is no introduction, and the connections between chapters are unclear. The chapter titles are disorganized, which is a clear sign for the generation by AI.

Questions for authors

The AI tools contribute to nearly every stage of the research process. Could the authors clarify which methodological ideas and technical contributions were independently developed by the research team, and how the AI outputs were validated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.