- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces the QC-PHAST Router, which formulates algorithm selection in scientific search as a sequential replacement problem relative to an executable default scheme, and utilizes survival prediction and group conformal calibration to make decisions between selecting specialized algorithms, paying for observations, and falling back. The method provides conditional finite-stage protection at the 90th percentile under the condition of bounded discovery loss. This method calibrates the risk differences relative to the default approach and the additional observed auxiliary utility, respectively, to determine whether to replace the algorithm or continue with the validation. The paper’s evaluation results primarily consist of an analysis of why the implementation failed to perform as intended and do not demonstrate its advantages.
Critical strengths
The article provides a very broad framework for discussion, covering applications across multiple fields. It lays out a comprehensive background and, in fact, discusses many specific scenarios.
Critical weaknesses
1.This paper is written in a very unclear manner; the problem background, model, and contributions are all described in plain text, many concepts are not clearly defined, and many conceptual terms lack corresponding references, making it very difficult to read. 2.This paper does indeed present a rule for selecting sequential algorithms. However, I can only see a list of concepts, distinctions between different concepts, and the direct application of existing methods. The contributions listed by the author also do not reflect any algorithmic or theoretical contributions.
Questions for authors
1.At first, I didn’t understand what the abbreviation “p90” stood for or why it was important, or why this specific quantile was chosen as the core focus of the study. 2.What is the significance of the experimental design and the presentation of results? Why can the value of “sqaure” exceed 1? 3.Why does the article provide so much background information on the application, yet the main text lacks empirical analysis? 4.Why is D11 in the appendix blank?