- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RESIGHT, a method for auditing early stopping decisions in LLM post-training. Given a portfolio of partially complete (stopped) training runs, RESIGHT estimates the number of these runs that would have produced a better model than the run that was chosen, called “rejection errors.” The relevant estimator uses inverse probability weighting based on both acquisition costs (i.e. the cost of completing a selected training run) as well as predictions of the probability that each run was stopped in error, coming from a ML model trained on historical screening decisions and outcomes. Notably, the primary goal of the method is post-hoc auditing of a fixed early-stopping policy. Across several experimental settings, RESIGHT estimates rejection error counts more accurately and surfaces more improved candidates during sampling compared to baseline audit methods at the same budget.
Critical strengths
Across several benchmark post-training datasets, RESIGHT identifies beneficial alternatives in more portfolios (i.e. “repairs”) than baseline audit methods at comparable budgets. Because these alternatives are completed candidate training runs that could replace the model that was originally selected, the reported value gains provide evidence that the sampling method underlying RESIGHT’s estimation procedure is effective for identifying promising candidate training runs.
Critical weaknesses
For a fixed reference model and early-stopping screening policy, RESIGHT estimates how many better candidate training runs were not fully explored during post-training. My main concern is that this post-hoc estimate is of limited practical utility. First, RESIGHT’s target does not necessarily convey the extent to which early-stopping sacrificed model quality—it is simple to imagine post-training settings in which there were many unexplored, marginally-better alternatives (high estimate, low impact) and others with few substantially better candidates (low estimate, high impact). While the experiments do report average value gain from sampled completions, there is no notion of missed value across all rejected runs. Second, excluding information gleaned from completing sampled training runs, RESIGHT’s estimate of “rejection errors” does not give insight into which modifications to the screening policy would have avoided these errors, limiting practical utility.
Questions for authors
• Do the reported cost comparisons (e.g. between RESIGHT and EVALUATE-ALL) include the cost of acquiring historical outcomes used to train the sampling predictor?
• Suppose a practitioner runs RESIGHT on their post-training experiments and obtains an estimate of rejection errors. How should they interpret this estimate, and what would it allow them to do differently beyond switching to better candidates discovered during auditing?
• In practice, what is a realistic audit budget as a fraction of the cost to run all completions? For both single-turn and multi-turn settings, RESIGHT is effective at medium-to-high cost but is outperformed by baselines in the low-cost regime (see Figure 3).
9 comments
Response to Reviewer zt1N (2)
2. Historical preparation costs
Historical bank construction, including the cost of collecting the final evaluation results used to train the sampling predictor, is recorded separately in the complete-study ledger (Appendix B.4). For the 120-portfolio comparison in Figure 5, ReSight cost 76.0% of Evaluate-All, a 24.0% operational saving. This total includes operational setup and execution, with fitting, calibration, and shared costs attributed to this batch counted once (Appendix H.2).
One-time preparation adds 21.3% of the Evaluate-All cost. Including this charge, the first batch costs 76.0% + 21.3% = 97.3% of Evaluate-All, saving 2.7% overall and recovering the preparation costs within that batch.
Response to Reviewer zt1N (3)
3. Interpreting the estimates and prioritizing follow-up
ReSight reports action-specific audit estimates for update and no-update groups. Practitioners can prioritize the group with the larger estimated conservative count for rechecking or testing a revised screening rule. Section 4.5 compares these audit outputs with full-completion reference outcomes for validation.
Each action-group estimate is the supported conservative error count per decision (Eq. (6)), not an error probability or lost-value measure. Several errors per decision are possible, and ambiguous comparisons may remain positive. Section 3.3 separately provides a confidence upper bound: certifies a prespecified count tolerance under the assumptions, while is inconclusive. The reported MB bound 0.832 cannot certify 0.1.
Response to Reviewer zt1N (1)
We are grateful for your detailed feedback. Below, we address each of your comments and will reflect the corresponding changes in the revised manuscript.
1. Practical value and recovered quality
At the same original expected continuation budget on the fully observed 120-portfolio BFCL static bank, ReSight's mean target-score gain is 8.03 percentage points (pp), or 94.1% of the 8.54 pp available through full completion; Cost-Only achieves 5.45 pp, or 63.8%. The remaining mean score gap is 0.51 pp for ReSight versus 3.09 pp for Cost-Only. Mean gains are computed over all 120 portfolios relative to screening alone, with zero gain for unchanged decisions and at most one replacement per portfolio; percentages use unrounded values.
The rejection-error count measures how many qualifying alternatives screening missed per decision; the score gain measures the effect on the selected model. Many small improvements can contribute more errors than one large improvement. Section 4.1's value gains concern acquired alternatives. Appendix D's portfolio 021 illustrates three confirmed errors producing one portfolio-level repair. The audit characterizes recorded screening outcomes. Action-specific audit estimates can guide follow-up at the update/no-update group level.
For the same bank, operational execution costs 76.0% of Evaluate-All and one-time preparation adds 21.3%, giving a first-batch total of 97.3%.