acceptodds
Under review as a conference paper at ICLR 2027

When to Search, Continue, or Fall Back: Risk-Controlled Routing for Rare-Regime Discovery

Abstract

Scientific rare-regime discovery requires choosing a search method under limited information, uncertain performance, and costly evaluations. We formulate this choice as a sequential replacement decision: when should a learned selector replace an executable fallback, pay for another observation, or fall back? QC-PHAST Router combines a permutation-invariant history encoder, route-conditioned survival predictor, and source-bank-group calibration to assess fallback-relative p90 loss. Information access determines admissible solvers, observations are charged, and proposed hits require independent verification. Finite-stage protection assumes within-track exchangeability; latent-risk protection additionally requires joint inner-interval coverage. Neither statement guarantees lower uncapped cost or protection under family shift. A frozen external evaluation records 1,152 configurations and 147,456 paired trials across 6 information tracks. The recorded procedure passes 0 of 3 primary replacement tests and makes 0 specialist commitments. A post hoc diagnosis finds that calibration corrections exceed the largest expressible prediction gap in 3 tracks. A subsequent audit identifies inherited-history errors in continuation evidence and related deployment paths, limiting interpretation of the recorded cost comparisons. We retain these records without claiming corrected routing efficacy. The study identifies two prerequisites for useful protection: enough decision resolution to authorize a route, and faithful accounting of the observations on which that decision depends.

Primary area:
optimization
Keywords:
algorithm selection, scientific machine learning, sequential decision-making, conformal calibration, rare-event discovery, survival analysis

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_CoheOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces the QC-PHAST Router, which formulates algorithm selection in scientific search as a sequential replacement problem relative to an executable default scheme, and utilizes survival prediction and group conformal calibration to make decisions between selecting specialized algorithms, paying for observations, and falling back. The method provides conditional finite-stage protection at the 90th percentile under the condition of bounded discovery loss. This method calibrates the risk differences relative to the default approach and the additional observed auxiliary utility, respectively, to determine whether to replace the algorithm or continue with the validation. The paper’s evaluation results primarily consist of an analysis of why the implementation failed to perform as intended and do not demonstrate its advantages.

Critical strengths

The article provides a very broad framework for discussion, covering applications across multiple fields. It lays out a comprehensive background and, in fact, discusses many specific scenarios.

Critical weaknesses

1.This paper is written in a very unclear manner; the problem background, model, and contributions are all described in plain text, many concepts are not clearly defined, and many conceptual terms lack corresponding references, making it very difficult to read. 2.This paper does indeed present a rule for selecting sequential algorithms. However, I can only see a list of concepts, distinctions between different concepts, and the direct application of existing methods. The contributions listed by the author also do not reflect any algorithmic or theoretical contributions.

Questions for authors

1.At first, I didn’t understand what the abbreviation “p90” stood for or why it was important, or why this specific quantile was chosen as the core focus of the study. 2.What is the significance of the experimental design and the presentation of results? Why can the value of “sqaure” exceed 1? 3.Why does the article provide so much background information on the application, yet the main text lacks empirical analysis? 4.Why is D11 in the appendix blank?

Reviewer_dyvrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Paper attempts to address the issue of how to choose a search method for rare scientific-regime discovery when evaluations are expensive, information access varies, and verification is required. Solution is formulated as a sequential replacement decision: replace executable fallback vs. pay for another observation vs. fall back.

The main Router algorithm is: predict fallback-relative p90, add a group-conformal margin, replace only if the whole interval beats fallback; otherwise buy one more π-chosen observation only if its lower-bound value is positive; otherwise fall back.

Validation results are negative, the algorithm fails to pass validation tests or reduce costs.

Critical strengths

Paper is honest about its negative results.

Critical weaknesses

Paper very hard to read. It seems written by AI with incoherent sentences and paragraphs. No clear unifying narrative. Jumps into concepts without properly introducing them. Examples: Fallback-relative p90, protected-route, source-bank-group calibration, policy-matched acquisition utility, inherited-history audit... Equation 2 on its own suddenly introduces 8 new parameters/concepts that are not explained. Paper benefits from proper natural language narrative and introduction.

In addition, there are two different failures conflated in the flow. Prediction-range obstruction and inherited-history error are independent, but authors interleave them. A reader had to disentangle decision infeasibility from invalid cost evidence.

A second major weakness, is that the paper is a negative result. The proposed method makes 0 specialist commitments and fails all three tests. The negative result is partially attributed to an inherited-history error.

"The study did not establish a cost-reducing router. It nevertheless distinguishes several failure mechanisms that should be checked separately when building a protected selector." But the lessons of this failure are all in the appendix. Authors may want to rewrite the paper as a lessons learned paper?

Questions for authors

N/A

Reviewer_NoojOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a router that decides whether to commit to a learned search route, buy another observation, or fall back, using conformally calibrated fallback-relative p90 loss. In a preregistered held-out evaluation it makes zero specialist commitments, and an audit finds history errors that prevent interpreting the recorded costs as efficacy evidence.

Critical strengths
  • The authors preregister a family-held-out evaluation, report zero commitments and failed tests without retuning, and disclose an implementation defect they found themselves.
  • The paper distinguishes bounded calibration loss from uncapped cost, the difference of p90s from the p90 of differences, and route protection from scientific verification.
Critical weaknesses
  • The empirical results do not validate the proposed method. The router makes zero specialist commitments, and even taken at face value, Table 9 reports unfavorable costs across all tracks. More importantly, an unquantified history-replay defect affects training, calibration, and deployment. The implementation also lacks the joint inner coverage required for latent-risk protection. Thus, neither efficacy nor the intended guarantee is established.

  • The protection is overly conservative, with limited theoretical novelty. The maximum-based calibration (Eq. 6), combined with 16-bin predictions, makes specialist routing impossible in three tracks. Table 8 already shows zero route actions during development, so this failure was detectable before the external evaluation. The resulting feasibility conditions are elementary, and the propositions largely apply standard split-conformal arguments.

Questions for authors

See above weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.