- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies whether inexpensive evaluations preserve the best candidate in a finite library after its continuation is optimized. It decomposes final scores into an aligned component and an orthogonal residual, derives a sharp pairwise ranking condition and top-k retention certificates, and extends the decomposition to multiple stages. It also evaluates regression-based screening (SCREEN) and budget-based promotion (ENV) on simulated control tasks and public multi-fidelity benchmarks.
Critical strengths
The finite-library formulation makes the distinction between preserving the full ranking and retaining the eventual optimum explicit. The pairwise boundary is sharp within the stated residual model. The experiments report favorable cost–accuracy trade-offs and include matched-schedule comparisons, which help separate the effects of promotion rules from those of schedule selection.
Critical weaknesses
The theoretical novelty is limited. Applying the classical three-correlation inequality to the two score vectors and the vector ((e_i-e_j)/\sqrt{2}), where (e_i) and (e_j) are coordinate basis vectors, yields Theorem 6, including its sharpness. [1] Theorem 9 follows directly from linearity of the least-squares projection. Neither requires an argument specific to optimizing continuations.
The algorithms provide limited additional novelty. SCREEN combines a standard regression prediction bound with a stopping rule using the current best value; its guarantee assumes simultaneous validity of the unread candidates’ bounds, which is not established for the sequential procedure. ENV changes promotion counts and schedules within the successive-halving framework. [5] Neither algorithm derives its decisions or guarantees from Theorem 6 or Theorem 9.
The empirical gains show that these particular heuristics can be useful, but do not resolve the central novelty concern. Overall, I do not find the theoretical or algorithmic contribution sufficient for acceptance.
Questions for authors
What mathematical contribution remains beyond the classical three-correlation inequality and linearity of projection? Please identify a precise statement or proof step that is not recovered by these reductions.
What is the independent algorithmic contribution of SCREEN and ENV beyond regression screening and successive-halving-style allocation? Please identify the substantive change and explain which comparisons isolate its benefit from schedule selection and budget accounting.