acceptodds
Withdrawn from ICLR 2027

IDENTIFICATION IS NOT REWARD: DISCOVERING TRAINING FEASIBILITY BOUNDARIES

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A successful hyperparameter search finds configurations that train well, but may reveal little about where training ceases to be feasible. We study how to recover this boundary from a fixed budget of short runs and turn it into a usable learning-rate scale. Our approach fits a finite grammar of threshold models, selects experiments through template disagreement, and compiles the fitted boundary into a schedule. The protocol evaluates boundary identification and collected-run reward separately. A threshold-identification analysis explains why reward-optimal sampling can leave the boundary unidentified, while monotone inversion connects an identified cut to a control scale. On synthetic tasks, active discovery recovers planted cuts to the reported precision while reward-seeking Bayesian optimization achieves higher sample reward. On CIFAR-10, the same protocol improves identification in harder proxy-label settings and supports a separate sharpness-based feasibility probe. Under label-noise stress, compiled scales improve over hand-set cosine schedules; a scale-by-shape comparison attributes the improvement primarily to the scale. These results establish a practical route from short-run experiments to explicit, actionable descriptions of training feasibility.

Primary area:
optimization
Keywords:
hyperparameter optimization, active learning, training dynamics, learning rate schedules, feasibility boundary identification

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_gPrJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper separates identifying a training-feasibility boundary from finding high-reward hyperparameters. It fits a finite grammar of threshold models, selects short-run experiments using template disagreement, and inverts the fitted boundary to obtain a learning-rate scale. The analysis explains why reward-maximizing queries can leave a threshold unidentified. Synthetic experiments and CIFAR-10 studies evaluate boundary recovery and the downstream performance of compiled schedules separately.

Critical strengths
  1. Evaluating identification and collected-run reward separately is sensible. The matched-budget synthetic comparison makes their different objectives easy to see.

  2. The scale-by-shape control and actual-schedule BO comparison are useful. They help locate the downstream benefit in the learning-rate scale rather than attributing every gain to a new schedule shape.

Critical weaknesses
  1. Figure 1 is not prepared to an acceptable standard for a conference submission. The Collector panel contains malformed and repeated binary-outcome text, and the Compiler panel contains the broken phrase “law cut the law translated translated.” The arrow labelled “next short run” points from identification scoring toward compiled-controller scoring, rather than clearly returning to the Collector. Other connectors also do not clearly show the discovery loop described in the caption. This is the main overview figure, so these unchecked errors directly obstruct understanding of the method. It needs to be redrawn and carefully checked, rather than inserted in its current state.

  2. The methodological increment is limited. Propositions 1–3 largely formalize familiar facts about threshold disagreement, boundary-avoiding observations, and binary search, while Theorem 1 gives monotone inversion when the fitted model already equals the plant. The implemented acquisition is a weighted heuristic, and there is no recovery guarantee for that procedure. The paper acknowledges the connection to active level-set estimation, but the additional scientific contribution beyond fitting an invertible template bank is not convincing enough.

  3. The main neural-network target provides weak evidence for a training-feasibility boundary. In the default CIFAR campaigns, no explosion is recorded, so the label is effectively whether 15-epoch accuracy exceeds 0.25 (Section 5). This identifies an early-accuracy threshold. The separate sharpness study is a useful addition, but its Active and GP-straddle results are very close and variable. Although the paper distinguishes these labels, its main empirical support for discovering training feasibility remains limited.

  4. The strongest controls do not establish a clear benefit of the full proposed pipeline. On the product plant, near-boundary-only sampling matches or slightly improves on the full acquisition (Appendix E.4). For deployment, actual-schedule BO reaches 0.869–0.872 versus about 0.868 for the compiled controller (Appendix E.3). An explicit fitted boundary may still be useful, but the paper has not shown enough additional value to justify the complexity and the strength of its contribution claim.

Questions for authors
  1. Can you clarify the intended directions and meanings of the Figure 1 connectors, particularly “next short run” and “after budget exhausted”? Which connections represent the actual implemented loop?

  2. What specific methodological result goes beyond standard active threshold/level-set identification followed by inversion? Is there a guarantee for the implemented template-selection and acquisition procedure, rather than for a known or correctly recovered feature?

  3. In the existing neural-network experiments, what downstream decision benefits specifically from the recovered boundary that cannot be obtained by the matched-budget actual-schedule BO baseline?

1 comment

Thank you for your thorough, constructive review and for carefully examining both the main text and the appendices. Your feedback is very helpful for our upcoming revision. Below, we address your comments and questions point by point.

Figure 1 and implemented loop (W1, Q1)

You are right about Figure 1. The malformed repeated binary-outcome text, broken compiler phrase, and ambiguous arrows should not have appeared in the main overview. The implemented loop is: Collector runs a short experiment and records (ui,yi,ri)(u_i,y_i,r_i); Discoverer refits the finite grammar; Experimentalist scores candidate controls using Eq. (4); the selected "next short run" returns to the Collector. After the short-run budget is exhausted, the fitted law is passed once to the Compiler, which inverts the boundary and evaluates a long-horizon schedule. BO is a reward-search fork, not part of the boundary-discovery loop.

Methodological increment beyond standard level-set estimation (W2, Q2)

We agree that the paper builds on active threshold and level-set ideas. The increment is not a new general LSE theorem. It is the combination of a finite, invertible threshold grammar with a protocol that evaluates three different outputs: boundary identification, reward collected during discovery, and compiled long-horizon control. This is why the paper compares against GP straddle/cUCB, passive sampling, reward BO, near-boundary sampling, and actual-schedule BO, and why it reports cases where the full acquisition is not uniquely responsible for the gain.

There is no recovery guarantee for the implemented weighted acquisition in Eq. (4). The formal results are deliberately narrower: Propositions 1 to 3 explain why boundary-avoiding or reward-maximizing queries can leave cuts ambiguous, and Theorem 1 justifies monotone inversion once the fitted model matches the plant. Empirical recovery of the full acquisition is tested in Tables 2, 3, 6, and 9.

Neural-network label and feasibility interpretation (W3)

We agree the default CIFAR proxy is mainly an early-accuracy threshold, since the 180 default discovery runs recorded no explosions. The paper states this in Section 5 and Discussion, and separates it from the wide-η\eta sharpness label. The intended interpretation is therefore not "we prove dynamical stability on CIFAR". It is: under a specified proxy label, the method learns a predictive boundary that can be compiled into a useful LR scale; a separate sharpness probe shows the same machinery can target a more dynamical label, though the Active and GP-straddle means there are close and variable (0.752 vs. 0.748).

Benefit over actual-schedule BO (W4, Q3)

Actual-schedule BO is a strong downstream controller baseline, and we agree it reaches comparable or slightly higher Env 3 accuracy (0.869 to 0.872 versus about 0.868 for Law-single-κ\kappa). The claimed benefit of the full pipeline is not higher final accuracy against every schedule search. It is that the discovery campaign returns an explicit boundary model in addition to a controller. That model can be inspected, evaluated on held-out/interventional controls, and inverted at new target noise levels without reinterpreting a set of schedule-search trials. If the sole goal is Env 3 accuracy under the same schedule family, Table 7 shows schedule BO is an effective alternative.

Thank you again for your detailed critique, especially for catching the Figure 1 rendering defects and helping us sharpen the framing of our contributions.

Reviewer_UmBaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents a method for identifying feasible learning rate scales. It uses "templates" of relations between learning rate (η\eta) and TT. I think TT is the target noise level, but this is not very clearly stated in the paper. Several short runs are done with these template relations, checking accuracy and possible "explosions" in the learning. Results are presented which indicate that the method is better than Bayesian Optimization for identifying learning feasibility boundaries.

Critical strengths

None

Critical weaknesses

I think the paper is very dense, to the point of being hard to decipher in places. Lots of notations are introduced and it takes considerable investigations (including the appendices) to work out what some of them mean, symbols are not explained when they appear the first time. (I was on the verge of stating "insufficient expertise" for this paper because I found it so hard to read, but this material can be made more accessible, the basic ideas are not that difficult.)

Questions for authors

I strongly advice to write with a broader readership in mind. The paper is in part so dense that (I think) it requires a person doing active research in this specifc field to fully appreciate the message.

Tables 2 and 3 indicate that passive sampling of templates is doing very well compared to the proposed active method. Is it really worth it to do the active part?

1 comment

Dear Reviewer UmBa,

Thank you very much for your time, thoughtful review, and constructive advice on improving the accessibility of our manuscript.

We completely agree that the notation density and presentation should be simplified with a broader readership in mind—including clearly defining symbols such as TT upon their first appearance and better highlighting the empirical advantages of active sampling over passive baselines. We will carefully incorporate your suggestions as we thoroughly revise the paper for future submission. Thank you again for your helpful insights.

Reviewer_fUXdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is very hard to read from the first few sentences of the introduction. Why is "finding a configuration that trains well" one part of a "broader" question "how does the feasible training range change with the environment?" What is a "point solution"? What is "an explicit boundary in learning rate and noise"?

I could not understand the logic of the introduction at all and had to feed the PDF into an AI to let it summarize the overall logic flow of the paper for me. From my understanding, the paper wants to identify the boundary between "feasible" and "infeasible" training configurations for some reasons that I do not fully understand. The boundary is like a curve in a 2D coordinate system, where the x-axis is some factor (e.g., label noise), and the y-axis is the maximum LR such that the validation accuracy is higher than some threshold. Then they fit some formulas for the boundary using active learning. Finally, an application of this is to choose the LR by multiplying the fitted maximum feasible LR by a factor such as 0.7.

Critical strengths

N/A

Critical weaknesses
  • The presentation of the paper is not very bad. Several parts of the paper are difficult to follow, and the explanations are often unclear.
  • I cannot understand the logic flow at all. Why is it interesting to identify the LR boundary? Why can't we just fit a formula for the optimal LR? Why is it easier to fit a formula for the LR boundary than to fit the optimal LR? How do the authors justify that the optimal LR is linearly related to the LR boundary?
Questions for authors

N/A

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

After careful consideration of the feedback received during the review and discussion process, we have decided to withdraw Submission #40040 from ICLR 2027.

First of all, we would like to express our sincere gratitude to the Area Chair for their conscientiousness, fairness, and dedication in facilitating a constructive and professional discussion. Second, we thank all the reviewers for their time, effort, and participation in evaluating our work. In particular, we thank Reviewer fUXd for the acknowledgement and the revised review of raising valuable and scientifically meaningful questions regarding our motivation and scaling formulation, as well as Reviewer UmBa and Reviewer gPrJ for their helpful insights on clarity, presentation, and technical positioning.

We recognize that there are still many aspects of the manuscript that can be improved. We will carefully incorporate all of your constructive suggestions as we thoroughly revise and strengthen the paper for a future venue.

Thank you again for your time and support.

Sincerely,

Authors of Submission #40040

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.