- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper separates identifying a training-feasibility boundary from finding high-reward hyperparameters. It fits a finite grammar of threshold models, selects short-run experiments using template disagreement, and inverts the fitted boundary to obtain a learning-rate scale. The analysis explains why reward-maximizing queries can leave a threshold unidentified. Synthetic experiments and CIFAR-10 studies evaluate boundary recovery and the downstream performance of compiled schedules separately.
Critical strengths
-
Evaluating identification and collected-run reward separately is sensible. The matched-budget synthetic comparison makes their different objectives easy to see.
-
The scale-by-shape control and actual-schedule BO comparison are useful. They help locate the downstream benefit in the learning-rate scale rather than attributing every gain to a new schedule shape.
Critical weaknesses
-
Figure 1 is not prepared to an acceptable standard for a conference submission. The Collector panel contains malformed and repeated binary-outcome text, and the Compiler panel contains the broken phrase “law cut the law translated translated.” The arrow labelled “next short run” points from identification scoring toward compiled-controller scoring, rather than clearly returning to the Collector. Other connectors also do not clearly show the discovery loop described in the caption. This is the main overview figure, so these unchecked errors directly obstruct understanding of the method. It needs to be redrawn and carefully checked, rather than inserted in its current state.
-
The methodological increment is limited. Propositions 1–3 largely formalize familiar facts about threshold disagreement, boundary-avoiding observations, and binary search, while Theorem 1 gives monotone inversion when the fitted model already equals the plant. The implemented acquisition is a weighted heuristic, and there is no recovery guarantee for that procedure. The paper acknowledges the connection to active level-set estimation, but the additional scientific contribution beyond fitting an invertible template bank is not convincing enough.
-
The main neural-network target provides weak evidence for a training-feasibility boundary. In the default CIFAR campaigns, no explosion is recorded, so the label is effectively whether 15-epoch accuracy exceeds 0.25 (Section 5). This identifies an early-accuracy threshold. The separate sharpness study is a useful addition, but its Active and GP-straddle results are very close and variable. Although the paper distinguishes these labels, its main empirical support for discovering training feasibility remains limited.
-
The strongest controls do not establish a clear benefit of the full proposed pipeline. On the product plant, near-boundary-only sampling matches or slightly improves on the full acquisition (Appendix E.4). For deployment, actual-schedule BO reaches 0.869–0.872 versus about 0.868 for the compiled controller (Appendix E.3). An explicit fitted boundary may still be useful, but the paper has not shown enough additional value to justify the complexity and the strength of its contribution claim.
Questions for authors
-
Can you clarify the intended directions and meanings of the Figure 1 connectors, particularly “next short run” and “after budget exhausted”? Which connections represent the actual implemented loop?
-
What specific methodological result goes beyond standard active threshold/level-set identification followed by inversion? Is there a guarantee for the implemented template-selection and acquisition procedure, rather than for a known or correctly recovered feature?
-
In the existing neural-network experiments, what downstream decision benefits specifically from the recovered boundary that cannot be obtained by the matched-budget actual-schedule BO baseline?
Thank you for your thorough, constructive review and for carefully examining both the main text and the appendices. Your feedback is very helpful for our upcoming revision. Below, we address your comments and questions point by point.
Figure 1 and implemented loop (W1, Q1)
You are right about Figure 1. The malformed repeated binary-outcome text, broken compiler phrase, and ambiguous arrows should not have appeared in the main overview. The implemented loop is: Collector runs a short experiment and records (ui,yi,ri); Discoverer refits the finite grammar; Experimentalist scores candidate controls using Eq. (4); the selected "next short run" returns to the Collector. After the short-run budget is exhausted, the fitted law is passed once to the Compiler, which inverts the boundary and evaluates a long-horizon schedule. BO is a reward-search fork, not part of the boundary-discovery loop.
Methodological increment beyond standard level-set estimation (W2, Q2)
We agree that the paper builds on active threshold and level-set ideas. The increment is not a new general LSE theorem. It is the combination of a finite, invertible threshold grammar with a protocol that evaluates three different outputs: boundary identification, reward collected during discovery, and compiled long-horizon control. This is why the paper compares against GP straddle/cUCB, passive sampling, reward BO, near-boundary sampling, and actual-schedule BO, and why it reports cases where the full acquisition is not uniquely responsible for the gain.
There is no recovery guarantee for the implemented weighted acquisition in Eq. (4). The formal results are deliberately narrower: Propositions 1 to 3 explain why boundary-avoiding or reward-maximizing queries can leave cuts ambiguous, and Theorem 1 justifies monotone inversion once the fitted model matches the plant. Empirical recovery of the full acquisition is tested in Tables 2, 3, 6, and 9.
Neural-network label and feasibility interpretation (W3)
We agree the default CIFAR proxy is mainly an early-accuracy threshold, since the 180 default discovery runs recorded no explosions. The paper states this in Section 5 and Discussion, and separates it from the wide-η sharpness label. The intended interpretation is therefore not "we prove dynamical stability on CIFAR". It is: under a specified proxy label, the method learns a predictive boundary that can be compiled into a useful LR scale; a separate sharpness probe shows the same machinery can target a more dynamical label, though the Active and GP-straddle means there are close and variable (0.752 vs. 0.748).
Benefit over actual-schedule BO (W4, Q3)
Actual-schedule BO is a strong downstream controller baseline, and we agree it reaches comparable or slightly higher Env 3 accuracy (0.869 to 0.872 versus about 0.868 for Law-single-κ). The claimed benefit of the full pipeline is not higher final accuracy against every schedule search. It is that the discovery campaign returns an explicit boundary model in addition to a controller. That model can be inspected, evaluated on held-out/interventional controls, and inverted at new target noise levels without reinterpreting a set of schedule-search trials. If the sole goal is Env 3 accuracy under the same schedule family, Table 7 shows schedule BO is an effective alternative.
Thank you again for your detailed critique, especially for catching the Figure 1 rendering defects and helping us sharpen the framing of our contributions.