- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how to select an acceptable configuration and estimate its performance accurately under a limited compute budget. The procedure provides a statistical guarantee for both selection and estimation, or abstains. PAIRS uses comparisons under shared randomness and reports performance using independent new observations, or by splitting Gaussian observations into independent selection and reporting data. It also considers further comparisons to reduce reporting costs, provides Gaussian and finite-pool complexity results, and evaluates the method in replay experiments and simulations.
Critical strengths
-
The problem is clearly defined. The paper separates selection quality, reporting accuracy, and completion, keeping abstentions in the denominator. This evaluates all runs rather than only successful returns. The distinction is useful in practice: selecting a good configuration does not justify reporting its winning score as its true performance.
-
The theory explains why comparisons and performance estimation have different costs. Theorems 2–3 use the same paired observation scheme and cost calculation for their lower and achievable upper bounds. Even when comparison noise is small, estimating a candidate's mean performance can require many observations. This explains both the benefit and the limits of shared randomness.
-
The authors specify which data selection and reporting may use. For Gaussian observations with known covariance, Proposition 1 and Appendix B show that the selection and reporting data are independent when the stated rules for data use are followed. This supports conditionally unbiased estimation after selection and stopping. The paper also distinguishes this method from reporting with independent new observations.
-
The controlled comparisons show when including reporting costs in selection helps. Appendix Z.9 compares four policies, separating pairing, choosing a candidate with a lower reporting cost among certified candidates, and further comparison before reporting. With only one acceptable candidate in Table 75, further comparisons provide no additional benefit. These results support the claim that more comparisons can sometimes reduce total cost and clarify the conditions needed.
Critical weaknesses
-
The certification rule in the main replay experiment is not sufficiently specified. Table 1 reports 177 certified outputs among 190 runs, with approximately one to two selection observations per candidate in several settings. Appendices B and I explicitly use different confidence boundaries or sampling schedules and do not directly explain these runs. I could not find the actual rule's complete formula, assumptions, and error allocation, so I cannot yet check whether these certificates satisfy Eq. (15). The counts alone do not establish invalidity; another valid boundary or additional information could explain them.
-
The additional benefit of including reporting costs in selection remains unclear in real replay experiments. Appendix Z.9 already compares the policies in a Gaussian model, but Appendices V and W primarily compare the methods as a whole. W changes whether observations are paired, the confidence boundaries, and the rules for evaluating candidates. It therefore does not separately measure the benefit of further comparisons. The authors should identify which parts of PAIRS were used in each experiment and whether existing records establish an additional benefit. This does not invalidate the comparison between the methods as a whole.
-
Independent training runs need a clearer link to the records of computational cost. Appendix V states that each observation used for reporting comes from an independent complete training run, while Table 76 converts counts using an average runtime. The aggregate tables supplied here are insufficient to check independence, failure handling, and the reporting sample's mean and variance. Linking observations to individual training records would help establish whether the assumptions for independent reporting hold and whether the cost results can be reproduced.
-
Repetition and missing transitions make the paper harder to follow. Sections 3, 5, and 6 move directly from headings into subsections without explaining how the parts connect. Appendix J.1 introduces a study again after summarizing it, and the context for Section 6.3's two ratios of 2.30 is scattered across appendices. Some passages read as heavily AI-assisted prose that has not been carefully edited. I recommend adding section introductions, removing repeated openings, and explaining the experimental units and cost measures in one place.
Questions for authors
-
Which confidence sequence was used for Table 1? Please state its assumptions, error allocation, and any independent pilot or prior information and associated costs. Please provide a certified run's full record, including the candidates evaluated, paired observations available at each step, intervals, eliminations, and stopping decisions.
-
Which parts of PAIRS were used in Appendices V and W? W's 2.30 budget ratio compares the methods as a whole and does not separately measure further comparisons to reduce reporting cost. Are controls that use paired comparisons for selection with the same independent reporting method, or records of the reporting budget reserved for each candidate, available to clarify the additional benefit in replay experiments?
-
Which observation and training records establish that Appendix V's aggregate counts come from independent complete training runs? These records should allow the reporting sample's mean, variance, and failure handling to be checked. Table 76's estimate of GPU-hours from average runtime does not establish those links.