acceptodds
Under review as a conference paper at ICLR 2027

What Early Stopping Hides: Cost-Aware Auditing in Post-Training

15 views

Abstract

Early stopping saves computation when selecting post-training updates but leaves the outcomes of rejected runs unknown. We introduce ReSight, which uses predictions from intermediate training results and remaining costs to sample stopped runs for completion and evaluation. Each completed run is compared with the original choice using constraints and costs fixed at screening. We correct for unequal sampling probabilities and count unresolved comparisons as possible errors to estimate the mean number of missed improvements per decision. Under stated sampling and evaluation assumptions, we derive upper bounds on the true mean over a fixed set of decisions. On the Berkeley Function Calling Leaderboard (BFCL) static benchmark with Qwen2.5-1.5B-Instruct, ReSight reduces the root mean squared error by 28.8% for decisions that retained the original model, compared with selecting runs based only on their remaining costs, for the same expected expenditure. In a prospective audit of 120 model selection decisions, estimates are close to reference counts, while measured cost, including setup, is 24.0% lower than the cost of completing and evaluating all runs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, post-training, audit, early stopping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zt1NOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RESIGHT, a method for auditing early stopping decisions in LLM post-training. Given a portfolio of partially complete (stopped) training runs, RESIGHT estimates the number of these runs that would have produced a better model than the run that was chosen, called “rejection errors.” The relevant estimator uses inverse probability weighting based on both acquisition costs (i.e. the cost of completing a selected training run) as well as predictions of the probability that each run was stopped in error, coming from a ML model trained on historical screening decisions and outcomes. Notably, the primary goal of the method is post-hoc auditing of a fixed early-stopping policy. Across several experimental settings, RESIGHT estimates rejection error counts more accurately and surfaces more improved candidates during sampling compared to baseline audit methods at the same budget.

Critical strengths

Across several benchmark post-training datasets, RESIGHT identifies beneficial alternatives in more portfolios (i.e. “repairs”) than baseline audit methods at comparable budgets. Because these alternatives are completed candidate training runs that could replace the model that was originally selected, the reported value gains provide evidence that the sampling method underlying RESIGHT’s estimation procedure is effective for identifying promising candidate training runs.

Critical weaknesses

For a fixed reference model and early-stopping screening policy, RESIGHT estimates how many better candidate training runs were not fully explored during post-training. My main concern is that this post-hoc estimate is of limited practical utility. First, RESIGHT’s target does not necessarily convey the extent to which early-stopping sacrificed model quality—it is simple to imagine post-training settings in which there were many unexplored, marginally-better alternatives (high estimate, low impact) and others with few substantially better candidates (low estimate, high impact). While the experiments do report average value gain from sampled completions, there is no notion of missed value across all rejected runs. Second, excluding information gleaned from completing sampled training runs, RESIGHT’s estimate of “rejection errors” does not give insight into which modifications to the screening policy would have avoided these errors, limiting practical utility.

Questions for authors

• Do the reported cost comparisons (e.g. between RESIGHT and EVALUATE-ALL) include the cost of acquiring historical outcomes used to train the sampling predictor?

• Suppose a practitioner runs RESIGHT on their post-training experiments and obtains an estimate of rejection errors. How should they interpret this estimate, and what would it allow them to do differently beyond switching to better candidates discovered during auditing?

• In practice, what is a realistic audit budget as a fraction of the cost to run all completions? For both single-turn and multi-turn settings, RESIGHT is effective at medium-to-high cost but is outperformed by baselines in the low-cost regime (see Figure 3).

9 comments
AuthorsOpenReview ↗

Response to Reviewer zt1N (1)

We are grateful for your detailed feedback. Below, we address each of your comments and will reflect the corresponding changes in the revised manuscript.

1. Practical value and recovered quality

At the same original expected continuation budget on the fully observed 120-portfolio BFCL static bank, ReSight's mean target-score gain is 8.03 percentage points (pp), or 94.1% of the 8.54 pp available through full completion; Cost-Only achieves 5.45 pp, or 63.8%. The remaining mean score gap is 0.51 pp for ReSight versus 3.09 pp for Cost-Only. Mean gains are computed over all 120 portfolios relative to screening alone, with zero gain for unchanged decisions and at most one replacement per portfolio; percentages use unrounded values.

The rejection-error count measures how many qualifying alternatives screening missed per decision; the score gain measures the effect on the selected model. Many small improvements can contribute more errors than one large improvement. Section 4.1's value gains concern acquired alternatives. Appendix D's portfolio 021 illustrates three confirmed errors producing one portfolio-level repair. The audit characterizes recorded screening outcomes. Action-specific audit estimates can guide follow-up at the update/no-update group level.

For the same bank, operational execution costs 76.0% of Evaluate-All and one-time preparation adds 21.3%, giving a first-batch total of 97.3%.

AuthorsOpenReview ↗

Response to Reviewer zt1N (2)

2. Historical preparation costs

Historical bank construction, including the cost of collecting the final evaluation results used to train the sampling predictor, is recorded separately in the complete-study ledger (Appendix B.4). For the 120-portfolio comparison in Figure 5, ReSight cost 76.0% of Evaluate-All, a 24.0% operational saving. This total includes operational setup and execution, with fitting, calibration, and shared costs attributed to this batch counted once (Appendix H.2).

One-time preparation adds 21.3% of the Evaluate-All cost. Including this charge, the first batch costs 76.0% + 21.3% = 97.3% of Evaluate-All, saving 2.7% overall and recovering the preparation costs within that batch.

AuthorsOpenReview ↗

Response to Reviewer zt1N (3)

3. Interpreting the estimates and prioritizing follow-up

ReSight reports action-specific audit estimates for update and no-update groups. Practitioners can prioritize the group with the larger estimated conservative count for rechecking or testing a revised screening rule. Section 4.5 compares these audit outputs with full-completion reference outcomes for validation.

Each action-group estimate is the supported conservative error count per decision (Eq. (6)), not an error probability or lost-value measure. Several errors per decision are possible, and ambiguous comparisons may remain positive. Section 3.3 separately provides a confidence upper bound: Uk≤ϵU_k\le\epsilon certifies a prespecified count tolerance under the assumptions, while Uk>ϵU_k > \epsilon is inconclusive. The reported MB bound 0.832 cannot certify 0.1.

Reviewer_YUj9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReSight, a protocol for auditing post-training decisions made with early stopping. It estimates how many stopped candidates would have been feasible improvements over the selected update or the reference model. A regularized logistic classifier fitted on historical records predicts conservative positive labels; these scores and acquisition costs determine which runs are sampled for completion. Horvitz-Thompson weighting corrects unequal acquisition probabilities, and a fixed-horizon bound accounts for evaluation ambiguity, failures, and unsupported candidates. Tool-use experiments report lower estimation RMSE and more repairs on the evaluated banks. A 120-portfolio prospective audit reports a setup-inclusive cost 24.0% below full completion and evaluation.

Critical strengths
  1. The paper raises an important auditing question and clearly defines what it aims to study. By keeping the original decisions and continuation costs fixed, it makes the comparisons fair and meaningful. The evaluation also considers practical cases, such as incomplete evaluations, failed runs, and candidates that have no chance of being selected.
  2. The controlled allocation comparisons provide evidence that checkpoint information can improve auditing efficiency on the evaluated workloads. The experiment results suggest that selectively continuing evaluations is a practical way to audit a fixed screening policy.
Critical weaknesses
  1. The methodological contribution and its advantage over related allocation methods are not sufficiently established. Logistic regression, HT estimation, the square-root cost allocation, and the concentration argument are established components, as the paper partly acknowledges. Using a simple classifier is reasonable, but the contribution therefore rests on the audit target and its integration.
    Classifier-driven candidate selection is already well studied, for example in BORE [1]. More directly, UQ-Guided HPO [2] already addresses early elimination of promising iterative learners through uncertainty-aware selection and budget allocation; Cost-Sensitive Freeze-thaw BO [3] is also relevant. Auditing a recorded policy is meaningfully different from optimizing that policy, so these works do not automatically subsume ReSight. However, their omission leaves the relationship and incremental value insufficiently established. Table 3's p(1-p) ablation reuses the historical predictor and is not a comparison with a complete online UQ-guided method. The current evidence does not establish an advantage over competitive UQ-derived audit priorities at a matched end-to-end cost. [1] Tiao et al. BORE: Bayesian Optimization by Density-Ratio Estimation. ICML 2021. https://proceedings.mlr.press/v139/tiao21a.html [2] Liu et al. UQ-Guided Hyperparameter Optimization for Iterative Learners. NeurIPS 2024. https://papers.nips.cc/paper_files/paper/2024/hash/010c5ba0cafc743fece8be02e7adb8dd-Abstract-Conference.html [3] Lee et al. Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning. NeurIPS 2025. https://arxiv.org/abs/2510.21379
  2. The evidence does not adequately resolve the predictor's data requirements or demonstrate broad practical benefits. The allocation classifier is fitted on historical portfolios and frozen during auditing (Sections 3.2-3.3). Informative labels require endpoint evidence, yet the provenance and selection of historical completed candidates are insufficiently detailed. Historical prefixes alone cannot establish which stopped runs would later improve; selectively completed histories may also be unrepresentative. This concerns allocation efficiency, not the validity of the HT audit under its assumptions. The Qwen-to-Phi experiment provides some transfer evidence, but does not establish reliable cold-start or cross-task performance, or when the costs of obtaining historical labels are recovered. The evaluation is also narrow: the headline BFCL static NU comparison has 14 no-update portfolios, while multi-turn uses four API-family portfolios per model with shared scenario indices. Repeated acquisition draws characterize these fixed banks, rather than new training problems. The motivating prospective demonstration uses four target items, zero continuation-cost penalty, and the reference model for protected requests. Finally, useful certification remains unproven: the prospective MB upper bound is 0.832 against a reference count of 0.0361, and Cost-Only has tighter mean bounds for both actions in Table 6. These limitations weaken the evidence that the additional prediction and audit machinery is worthwhile in realistic constrained HPO.
Questions for authors
  1. How many fitting candidates have observed endpoint evaluations, and how were they selected for completion? Please distinguish labels supported by observed endpoints from conservative positives caused by missing evaluations or failures, and explain how the fitting population represents newly rejected candidates when historical completion is selective.

  2. Is ReSight mainly intended for repeated audits of similar tasks where historical evaluation results are already available? Please clarify how the reported 24.0% cost saving is calculated. Does it include the cost of collecting historical data, fitting the predictor, and calibrating new candidates? If these costs are excluded, how many audits would be needed to recover them? Also, is there evidence that ReSight can achieve similar efficiency gains on a new model or task without collecting additional endpoint labels?

  3. Please explain the additional contribution relative to [2] and [3], separating audit estimation from improvements in final model selection. Does the Table 3 uncertainty ablation support only a comparison of priorities for a fixed HT estimator and historical classifier? Please also report measured baseline expenditures at the nominal 0.40 setting, given the different count and cost budget definitions.

  4. Appendix B.3 specifies census targets and a Hoeffding construction. The reported upper bounds seem quite loose compared with the actual error counts. For example, the MB error count is only 0.0361, while its upper bound is 0.832. What practical decisions can users make based on such bounds? Appendix C.3 also shows that Cost-Only produces tighter bounds than ReSight. Please discuss this trade-off and clarify the practical value of the proposed bounds.

13 comments
AuthorsOpenReview ↗

Response to Reviewer YUj9 (1)

We appreciate your careful reading of the manuscript. We respond to your comments in order below and will update the relevant sections of the revised manuscript.

1. Methodological contribution

ReSight estimates the mean number of missed feasible improvements per decision in a fixed screening batch, including rejected runs not acquired by the audit. Finding one better final model does not answer that population question. Section 2 fixes the original actions, comparators, feasibility constraints, and continuation costs; Section 3 estimates the supported conservative count and separately bounds the true count. With fixed potential labels and known positive inclusion probabilities, Eq. (6) is unbiased for that conservative count, independently of predictor accuracy. Appendices F/H specify ambiguity, failures, unsupported runs, and faithful continuation. Repairs leave the original audit target unchanged.

Table 3 isolates the value of checkpoint information for this target: BFCL NU RMSE is 0.2148 with full scores, 0.2673 with recipe-only scores, and 0.3094 with permuted scores at the same expected acquisition cost.

AuthorsOpenReview ↗

Response to Reviewer YUj9 (2)

2. Related HPO methods and active evaluation

BORE constructs an expected-improvement acquisition rule using classification; UQ-Guided HPO uses uncertainty for candidate selection and budget allocation; cost-sensitive freeze-thaw chooses continuations by predicted cost–performance utility [1, 2, 3]. ReSight also uses predictive selection, but holds the recorded screening decisions fixed and reports a supported conservative-count estimate and a true-count upper bound for the batch's rejected candidates.

The uncertainty ablation in Table 3 compares priorities with the historical classifier and HT estimator fixed: full allocation reduces BFCL static NU RMSE from 0.2317 to 0.2148 at matched expected acquisition cost. Appendix B.2 implements residual-based ASI, Robust ASI, and Active Testing/ASE with five-member ensembles updated after each acquisition; each baseline retains its own estimator. Figure 3 compares measured cost–RMSE: ASE is best at the smallest BFCL multi-turn MB budget, while ReSight has lower RMSE in the middle and upper ranges; xLAM–Hammer curves are closer.

References

[1] L. C. Tiao et al. BORE: Bayesian Optimization by Density-Ratio Estimation. ICML, 2021. https://proceedings.mlr.press/v139/tiao21a.html.

[2] J. Liu et al. UQ-Guided Hyperparameter Optimization for Iterative Learners. NeurIPS, 2024. https://papers.nips.cc/paper_files/paper/2024/hash/010c5ba0cafc743fece8be02e7adb8dd-Abstract-Conference.html.

[3] D. B. Lee et al. Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning. NeurIPS, 2025. https://arxiv.org/abs/2510.21379.

AuthorsOpenReview ↗

Response to Reviewer YUj9 (3)

3. Data requirements and practical value

All 288 candidates in the 96 fitting portfolios completed their planned training and had valid endpoint evaluations, irrespective of halfway scores. One-time preparation for the 120-portfolio operational workload cost 21.3% of Evaluate-All, below the first batch's 24.0% operational saving.

In Table 5, 12 fitting portfolios yield NU RMSE 0.2729 at normalized cost 0.403, while 96 yield 0.2148 at 0.404; Cost-Only yields 0.3016 at 0.401. Table 4 transfers the frozen Qwen scorer to Phi without refitting on Phi outcomes, improving MB/NU RMSE from 0.0949/0.2699 to 0.0874/0.2629. Across three additional training portfolios, ReSight improves MB RMSE, repair probability, and mean score gain over Cost-Only and ASE at matched completion cost (Table Y1).

The original banks contain 14 static NU decisions and four shared-scenario multi-turn portfolios per model (Appendices B.1/B.3). The prospective example uses four target items, λ=0\lambda=0, and reference routing; multi-turn uses larger panels and 0.02 regression limits. Table 6 favors Cost-Only for mean bound tightness.

 

Table Y1. Matched-cost results on three additional BFCL multi-turn training portfolios.

PortfolioMethodCost (%)MB RMSERepair prob. (%)Score gain (pp)
1Cost-Only40.80.7136.72.30
ASE40.80.4941.02.68
ReSight40.80.2748.33.28
2Cost-Only39.70.6135.61.83
ASE39.70.5040.02.24
ReSight39.70.3047.43.06
3Cost-Only41.30.8237.61.88
ASE41.30.5341.82.29
ReSight41.30.3250.23.24

 

Qwen2.5-1.5B-Instruct uses halfway screening. Mean measured completion cost is matched within each portfolio and normalized by completing and evaluating all its rejected candidates. Mean target-score gain is relative to screening alone and includes zero gains for unchanged decisions; pp denotes percentage points. The portfolios use different training-data samples and random seeds, were excluded from fitting and tuning, and use fixed method settings and evaluation panels. Table Y1 reports the same three portfolios as Table E2 in our comment titled “Response to Reviewer EAnC (2)”.

Reviewer_EAnCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper addresses a key blind spot of using early stopping in post-training model selection: because runs are terminated early to save compute, it is impossible to know how they would have ultimately performed. Specifically, the authors investigate how often a fixed screening policy makes two types of mistakes: prematurely discarding a superior run (missed-better, or MB), or retaining the base model when a beneficial update was actually available (false-no-update, or NU). To quantify these errors, the authors propose ReSight, an auditing method that completes a strategic random sample of stopped runs. ReSight assigns higher sampling probabilities to runs that are predicted to hide an error and lower probabilities to those that are expensive to finish. Using Horvitz–Thompson weighting, the method then estimates the average error count per decision and provides a finite-sample upper bound. Empirical evaluations on the BFCL benchmark show that ReSight reduces NU estimation error (RMSE) by 28.8% compared to cost-only sampling, and a prospective audit of 120 decisions demonstrates that it reduces total compute costs by 24% compared to completing every run.

Critical strengths

S1\textbf{S1} The paper poses a novel and practical question that, to my knowledge, has not been previously explored: "How many better runs did an early-stopping policy reject?" Whereas existing work uses learning-curve predictions to decide whether to pause or resume runs (Swersky et al., 2014), this work focuses on auditing the mistakes left behind. Furthermore, the resulting error bound holds in finite samples and does not depend on the accuracy of the underlying predictor. This offers a distinct advantage over active statistical inference frameworks (Zrnic & Candès, 2024), which rely on asymptotic confidence intervals.

S2\textbf{S2} The experimental evaluation is notably rigorous and transparent. In the prospective study, the authors froze both the screening policy and the audit rule prior to execution, and their cost ledger properly accounts for setup overhead. Additionally, the screening gate is carefully calibrated on nominees generated by the exact same selection mechanism using split conformal prediction (Lei et al., 2018). Finally, it is commendable that the authors transparently report unfavorable results, such as noting instances where the Cost-Only baseline yields tighter bounds than ReSight.

Reference:

Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018). Distribution-free predictive inference for regression. JASA, 113(523), 1094--1111.

Swersky, K., Snoek, J., & Adams, R. P. (2014). Freeze-thaw Bayesian optimization. arXiv:1406.3896.

Zrnic, T., & Candès, E. J. (2024). Active statistical inference. ICML 2024, PMLR 235.

Critical weaknesses

W1\textbf{W1} The audit spends much of the compute saved by early stopping without giving a tight statistical guarantee in return. With only a few candidates and a single screening cut, early stopping saves relatively little to begin with, and the audit eats up a large share of those savings. While auditing occasionally fixes bad decisions, the upper bound on missed-better errors far exceeds the reference count and is looser than with cost-only sampling. This is because ReSight chooses sampling probabilities to reduce estimation variance, whereas the bound's width depends on a worst-case term that cost-only sampling minimizes. Because the bound treats every run as a potential error, tightening it requires sampling more runs, which erodes the savings early stopping was meant to provide.

W2\textbf{W2} The experiments are too small and simplified to show that the method works in realistic post-training. Each portfolio only tests a few short LoRA runs that are screened once halfway through training. Standard pruning methods cut runs much earlier and across multiple rounds (Jamieson & Talwalkar, 2016; Li et al., 2018), where finishing audited runs would likely be more expensive and predicting their final performance harder. Even in this simple setup, the evidence is thin. Gains are measured on just a few target items, so a hidden error often comes down to a single extra correct answer, and the results rely on very few positive runs.

References:

Jamieson, K., & Talwalkar, A. (2016). Non-stochastic best arm identification and hyperparameter optimization. AISTATS 2016.

Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2018). Hyperband: A novel bandit-based approach to hyperparameter optimization. JMLR, 18(185), 1--52.

Questions for authors

Q1\textbf{Q1} What is the audit cost per repair in absolute compute, compared with not early-stopping at all? Could periodic or one-off audits justify the cost?

Q2\textbf{Q2} Why screen only once at 50% when 10% and 25% checkpoints were saved [App. B.1]? How do savings, audit cost and ReSight’s advantage change under multi-round pruning?

5 comments
AuthorsOpenReview ↗

Response to Reviewer EAnC (1)

Thank you for your thoughtful assessment of our work. We address your comments point by point below and will incorporate the corresponding updates into the revised manuscript.

1. Cost per repair and practical audit budgets

Additional candidate compute per repaired decision is 21.5 GPU-minutes at the original budget and 16.8 at half budget, compared with 46.4 when completing every stopped run (Table E1). Across 120 BFCL static portfolios with fixed screening decisions, the half-budget audit saved 3.73 GPU-hours, repaired 13.6 decisions on average versus 21.1 at the original budget, and retained 62% of the original target-score gain.

 

Table E1. Cost and model-selection gains on the same 120 BFCL portfolios.

MetricScreening onlyHalf budgetOriginal budgetComplete all
Reported operational cost (% of complete all)58.9%67.5%76.0%100%
Candidate GPU-hours, total23.1626.9730.7040.94
Additional candidate GPU-hours over screening only03.817.5417.78
Mean number of decisions repaired013.5721.0923
Mean target-score gain across all 120 portfolios (percentage points)0+5.01+8.03+8.54
Additional GPU-minutes per repaired decision—16.821.546.4

 

A portfolio counts as repaired if a completed candidate scores more than one percentage point higher than the original choice.

At the original budget, operational execution costs 76.0% of Evaluate-All, a 24.0% operational saving. One-time preparation adds 21.3%, giving a first-batch total of 97.3%, or a 2.7% saving. For repeated related batches, reusable preparation is charged once.

Cost-Only minimizes the worst-case sampling allowance (Appendix G.2), which explains the tighter bounds in Table 6. ReSight allocates more compute to candidates likely to improve the original decision. In the separate matched-cost comparison in Table 1, this allocation increased mean repairs from 4.77 to 6.41 and reduced NU estimation error by 28.8%.

In a follow-up replay on the same 120 BFCL portfolios at the original continuation budget, an allocation variant reduced the mean upper bound on the batch-wide total of MB and NU missed improvements from 124.5 to 121. The variant set each candidate's continuation probability to the average of the probabilities assigned by ReSight and Cost-Only, weighting each method equally. Cost-Only's mean upper bound was 118.8. The variant repaired 20.2 decisions on average, compared with 21.1 for the original ReSight allocation and 15.0 for Cost-Only, retaining about 96% of the original repair count.

AuthorsOpenReview ↗

Response to Reviewer EAnC (2)

2. Experimental scale and additional portfolios

The BFCL multi-turn evaluation uses 80 target and 240 retained cases per portfolio, versus four target items in the prospective illustration (Appendix B.1). Training budgets are 620, 620, and 720 steps. Each model has four API-family portfolios with shared scenario indices. In the paper's BFCL multi-turn comparison, ReSight reduced MB and NU RMSE and increased mean repairs relative to Cost-Only for both Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. A three-round BFCL static replay at 10%, 25%, and 50% used 11.73 GPU-hours with ReSight, versus 15.34 for a single 50% cut. Across three additional BFCL multi-turn training portfolios, ReSight reproduced its advantage at matched mean measured completion cost, with lower MB RMSE, higher repair probability, and greater mean target-score gains than both Cost-Only and ASE in all three groups (Table E2).

Appendix B.3 reports how often screening missed qualifying candidates. The BFCL static Qwen2.5-1.5B bank has 14 confirmed errors plus one unresolved comparison across 14 no-update decisions. In the separate 120-portfolio bank, 29 of the 111 stopped candidates in no-update portfolios were confirmed errors. Counting one unresolved comparison gives 30 conservative positives. These counts establish the frequency of missed alternatives in the evaluated banks. Repeated audit draws quantify sampling variability within those banks.

 

Table E2. Matched-cost comparison on three additional BFCL multi-turn training portfolios.

Training portfolioMethodCompletion cost (%) ↓MB RMSE ↓Repair probability (%) ↑Mean score gain (pp) ↑
New group 1Cost-Only40.80.7136.72.30
ASE40.80.4941.02.68
ReSight40.80.2748.33.28
New group 2Cost-Only39.70.6135.61.83
ASE39.70.5040.02.24
ReSight39.70.3047.43.06
New group 3Cost-Only41.30.8237.61.88
ASE41.30.5341.82.29
ReSight41.30.3250.23.24

 

Qwen2.5-1.5B-Instruct with halfway screening. Mean measured completion cost is matched within each portfolio and expressed relative to completing and evaluating all rejected candidates in that portfolio. Mean target-score gains are relative to screening alone, including zero gains for unchanged decisions; pp denotes percentage points. The three additional training portfolios used different training-data samples and random seeds, were excluded from ReSight fitting and tuning, and were evaluated using the three methods at matched mean measured completion cost under the manuscript's fixed method settings, halfway screening rule, and evaluation panels.

AuthorsOpenReview ↗

Response to Reviewer EAnC (3)

3. Screening timing and multi-round pruning

The 50% cut is the common reference point for the main allocation comparisons: every method starts from the same screening decisions and rejected candidates. Table E3 compares that setting with single cuts at 10% and 25% and a three-round schedule at 10% → 25% → 50% on the same BFCL static Qwen2.5-1.5B-Instruct trajectories.

Across screening schedules, ReSight used 23.5% less GPU time including continuation with three rounds than with a single 50% cut (11.73 vs. 15.34 GPU-hours). Within the three-round schedule, ReSight repaired 40.6 decisions on average versus 33.8 for Cost-Only, 6.8 more repairs. In that same comparison, ReSight had lower NU RMSE (0.151 vs. 0.176) and higher MB RMSE (0.218 vs. 0.074).

 

Table E3. Screening timing, compute, and audit outcomes.

Screening scheduleMethodScreening GPU-hTotal GPU-hMB RMSENU RMSEMean repairs
Single cut: 10%Cost-Only2.119.61N/A0.18237.4
ReSight2.119.84N/A0.16843.2
Single cut: 25%Cost-Only5.1111.35N/A0.17439.1
ReSight5.1111.59N/A0.17941.6
Single cut: 50%Cost-Only11.3815.060.00000.30164.770
ReSight11.3815.340.00000.21486.410
Multi-round: 10% → 25% → 50%Cost-Only5.6711.910.0740.17633.8
ReSight5.6711.730.2180.15140.6

 

The screening GPU-hours shown for each schedule average the two timing records. Total GPU-hours are from the replay. The 50% RMSE and repair counts reproduce the manuscript's nominal 0.40 acquisition results. N/A indicates unavailable MB RMSE. Mean repairs count distinct repaired portfolios per audit draw. This retrospective bank is separate from Table E1's 120-portfolio execution.

At each screening round, the replay uses only checkpoint information available at that round, and only candidates retained in that round advance to the next checkpoint. Both audit allocations start from the same recorded screening decisions and rejected candidates.

Earlier rejection leaves more continuation work for a fixed trajectory. Table E3 includes that continuation cost in the comparison.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Revised Manuscript Uploaded

Dear Area Chair and Reviewers,

Thank you for your time and feedback. We have uploaded the revised manuscript to OpenReview. The changes are marked in blue.

Best regards,
The Authors

AuthorsOpenReview ↗

Summary for the Area Chair, Senior Area Chair, and Program Chairs (2/2) — Reviewer Feedback and Follow-up

Dear Area Chair, Senior Area Chair, and Program Chairs,

Following our summary of the technical responses in Part 1, we would like to briefly summarize the discussion with each reviewer.

  • EAnC. The reviewer wrote that the additional experiments resolved most of their concerns and raised their score from 2 to 3.
  • YUj9. We posted point-by-point replies and a separate summary, and followed up several times to invite the reviewer to read our responses and the revised manuscript. As of this note, we have yet to receive further feedback.
  • zt1N. In their follow-up, the reviewer maintained their original score and identified the need for direct evidence that ReSight’s estimates could guide subsequent compute allocation and improve the models ultimately selected. We then posted the matched-cost, two-batch comparison summarized in Part 1 as a direct reply in the same thread. This comparison provides the evidence the reviewer requested.

We sincerely thank all three reviewers for their constructive feedback and for helping us strengthen the paper.

We have responded to every question and weakness raised by the reviewers, with additional explanations, experiments, and mathematical derivations where appropriate. We believe these additions address the concerns raised in the reviews.

This paper grew out of a problem we encountered in our daily work in this field. We use early stopping to save compute, and after screening we want to know how many feasible improvements we missed and where to spend our next round of compute. That practical question motivated ReSight.

ReSight estimates how many feasible improvements a fixed screening policy misses and provides finite-sample upper bounds under stated assumptions. On BFCL static, ReSight reduced estimation RMSE for no-update decisions by 28.8% at the same expected acquisition cost. In the two-batch offline replay, using the first batch’s estimates to guide follow-up allocation improved the mean target score of the models selected in the second batch by 0.76 percentage points over Cost-Only at the same total expected cost. We believe the revised paper clears the bar for acceptance because it makes missed improvements measurable and shows how practitioners can use those estimates to select better models within the same compute budget.

We understand this is a busy time, and we respectfully ask that you consider our summary, point-by-point replies, supplementary evidence, and revised manuscript when evaluating the paper. We sincerely appreciate your time and consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary for the Area Chair, Senior Area Chair, and Program Chairs (1/2) — Reviewer Concerns, Responses, and Evidence

Dear Area Chair, Senior Area Chair, and Program Chairs,

Below we summarize the concerns raised by all three reviewers and the evidence we provided in response.

Reviewer EAnC

  1. What is the compute cost per repair, and is a smaller audit worthwhile? On the same 120 BFCL static portfolios, the additional compute per repaired decision was 21.5 GPU minutes at the original budget, 16.8 at half budget, and 46.4 under full completion. Halving the budget saved 3.73 GPU hours, still repaired 13.6 decisions on average, and retained 62% of the original target-score gain (Table E1).

  2. Why screen at 50%, and what happens with earlier or repeated screening? The 50% cut gives the main allocation comparisons a common set of screening decisions and rejected candidates. We compared single cuts at 10%, 25%, and 50% with three-round screening. Including continuation, ReSight used 23.5% less GPU time with three rounds than with the single 50% cut. Within the three-round setup, ReSight repaired 40.6 decisions on average versus 33.8 for Cost-Only (Table E3).

  3. How much evidence is there beyond the small evaluation panels? The BFCL multi-turn evaluation uses 80 target and 240 retained cases per portfolio. We added three training portfolios held out from fitting and tuning. At matched measured completion cost, ReSight achieved lower MB RMSE, higher repair probability, and greater mean score gain than Cost-Only and ASE in all three (Table E2).

Reviewer YUj9

  1. Where did the historical labels come from? All 288 candidates across 96 fitting portfolios completed their planned training and had valid final evaluations. Only halfway scores informed screening. Among the 198 candidates rejected at halfway, final evaluations confirmed feasible improvements for 131, were inconclusive for 6, and ruled out improvement for 61. The conservative labeling rule counts the confirmed and inconclusive cases as positives.

  2. What does the 24.0% saving include, when does preparation pay off, and does the scorer transfer? Operational execution, including setup, cost 76.0% of Evaluate-All. One-time preparation added 21.3%, bringing the first-batch total to 97.3% and recovering the preparation cost within that batch. The frozen Qwen scorer also lowered MB and NU RMSE on Phi without refitting on Phi outcomes.

  3. What does ReSight add relative to prior HPO methods, and how do the baselines compare at measured cost? UQ-Guided HPO and Cost-Sensitive Freeze-Thaw BO choose which configurations to continue during search. ReSight takes recorded screening decisions as fixed and estimates how many feasible improvements were missed across the batch, with a statistical upper bound. The Table 3 ablation holds the historical classifier and HT estimator fixed and changes the sampling priority. We also reported measured baseline expenditures. At the nominal 0.40 setting on BFCL multi-turn, ReSight and ASE had MB RMSEs of 0.2745 and 0.3103 at measured completion costs of 40.6% and 47.6%, respectively.

  4. Why is the upper bound wide, and what is gained when Cost-Only gives a tighter bound? The MB point estimate of 0.0358 was close to the full-completion reference of 0.0361. We provided the derivation showing how the bound also accounts for sampling uncertainty and why Cost-Only minimizes the worst-case sampling correction. The lowest tested budget that brought the mean MB upper bound across repeated draws to 0.1 or below ranged from 98.4% to 98.8% of full continuation training tokens. At the original 40% expected continuation budget, ReSight repaired 21.1 decisions on average versus 15.0 for Cost-Only.

Reviewer zt1N

  1. How much model quality can be recovered? Full completion recovered an average of 8.54 percentage points over screening alone across the 120 portfolios. At the same expected continuation budget, ReSight recovered 8.03 points (94.1%) and Cost-Only recovered 5.45 points (63.8%).

  2. Are preparation costs counted, and what can a smaller budget buy? The first-batch total of 97.3% includes one-time preparation. The half-budget audit used 21.4% of the additional candidate GPU time needed to complete all rejected runs, repaired 13.6 decisions on average, and preserved 62% of the original target-score gain.

  3. Can the estimates guide subsequent compute and improve the models ultimately selected? We used the first 60 portfolios to identify decisions that retained the original model as the priority for follow-up, then tested that allocation on the remaining 60. At the same total expected cost of 6.33 GPU hours, the estimate-guided allocation repaired 0.86 more decisions on average and raised the mean target score of the selected models by 0.76 percentage points relative to Cost-Only.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.