acceptodds
Under review as a conference paper at ICLR 2027

When Fixed Thresholds Fail: Trace-Grounded Scheduling for Multi-Tenant LLM Fine-Tuning under SLOs

Abstract

Multi-tenant platforms can schedule LLM fine-tuning by watching each job's evaluation curve, terminating runs whose score declines and handing the freed GPUs to another tenant. A recently published rule of this kind reports a false-positive rate. We replay it on real per-checkpoint trajectories from public pretraining runs. Every run is healthy, so every termination is an error, and the rule fires on of them. Neither obvious explanation holds up. The synthetic concave curve used for calibration predicts an even higher rate, and real evaluation noise is milder than assumed rather than harsher. What actually carries the published number is a quantity it never states, the per-interval signal-to-noise ratio. Real curves sit at a median of , whereas at demands . Correct that single parameter and the original generator reproduces our measurement to within four points. Raising does not rescue the rule, because real declines cluster, and a classical quickest-detection bound puts the delay needed for a false-alarm rate at three times the checkpoints a job has, so by that bound no causal detector reaches the published operating point on these curves. Standard reporting would not have caught this. Replaying a -job production trace, we find that how SLO attainment is aggregated moves the reported number further than which scheduler produced it, by as much as points, and reorders the policies outright. Early termination then corrupts the usual metrics in one direction. On a workload with no declining jobs at all, the published rule “improves” mean job completion time by while destroying of useful compute. At the trace's actual prevalence of decline, nothing we tested pays for itself, including a learned policy that never beats a one-parameter rule normalising drawdown by an online noise estimate. We recommend net goodput reported with a false-positive rate quoted at its checkpoint cadence, the one pairing that survived every probe we ran.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM fine-tuning, Multi Tenant, GPU cluster scheduling, Early stopping, Quickest change detection, Detectability limits, Service-level objective, Trace-driven simulation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_vybWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a multi-tenant tuning scheduling for LLM jobs based on a concept of "curve-based early termination" rules (such as terminating jobs after kk consecutive evaluation score declines) to reallocate GPU resources. The authors replay a heuristic against real trajectories and a cluster trace. The authors showed that fixed-threshold stopping rules suffer from high false-positive rates and claimed that adaptive heuristics cannot meet the SLA required by realistic workloads.

Critical strengths

The authors quantify how standard scheduling metrics e.g., mean job completion time can be corrupted and manipulated by early termination behaviors. This is an interesting insight into the scheduling problem and associated metrics.

Critical weaknesses

The premise of modeling LLMs simply as a set of GPU jobs with (static) resource demands in GPU is extremely simplistic and fragile. Such modeling cannot reflect current datacenter complexities of modern LLM workload. Moreover, the methodology relies on synthetic data, as it seems that the authors artificially inject some trajectories into performance curves. This leaves the entire evaluation dependent on unrealistic assumptions and weakens the results and the insights that it could have offered.

Questions for authors

Some important questions that the authors need to address. The paper models LLM workloads as set of GPU jobs with some GPU demands. How does this model account for real-world production complexities in modern LLM workloads like dynamic prompt/generation lengths, batching overheads, KV-cache memory pressure, and multi-node pipeline parallelism? Would the proposed formulation remain tractable or hold up under these dynamic conditions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer vybW: Scope, Real (Non-Synthetic) Evidence, Device Sharing

Thank you for the review, and for noting the result on how early termination corrupts mean job completion time. Numbers refer to the revised PDF; Table G1 is in the general response.

Scope. The paper does not propose a scheduler or a stopping rule: it tests a published stopping rule (Zhang et al., 2026) and seven existing schedulers. It studies fine-tuning, that is training jobs, not inference serving. The revised introduction says both (Sec. 1).

W. Static GPU-job model; reliance on synthetic data.

We agree that execution speed is not static when training jobs share a device, and the revision measures and tests this. On one L40S, two co-located fine-tuning jobs each run 1.19x slower (a lower bound; Sec. 6.3; App. H.5). We added a slowdown model for device sharing and multi-device communication and swept 15 settings whose range includes this value (Sec. 6.3; App. H.4). The ordering of stop rules by net benefit is unchanged in all 15 and their sizes move by at most 0.4 points; the finding that the published rule "improves" mean JCT while destroying useful compute holds in all 135 cells. These runs use 3 trace windows and are indicative.

The headline result uses no synthetic data, and the revision adds real fine-tuning runs and real collapses. The published k = 2 rule fires on 35 of 78 real, healthy pretraining curves (44.9% [34.3, 55.9]) and on 9 of 15 healthy LoRA fine-tuning runs (60.0% [35.7, 80.2]; Sec. 4.5; Table 2), against the published 1.5%. The probe behind the JCT result has no declining jobs at all. Injected declines are still what the simulator and the recall figures use. To check them against real ones we ran 6 fine-tuning runs that really collapse (10x learning rate): k = 2 detects 4 of 6 while stopping 9 of 15 healthy runs (Table G1; App. H.5). Limits: one 9B model, about 5 independent task units, abrupt collapses only, not RLHF.

Q. Prompt/generation lengths, batching overheads, KV-cache pressure, multi-node parallelism; tractability.

Prompt and generation lengths, batching overheads and KV-cache pressure are properties of inference serving; they do not arise for whole fine-tuning jobs measured in GPU-hours and checkpoints, and the paper makes no claim about serving. Multi-node parallelism does apply to training; the communication term of the slowdown model addresses it, as a swept parameter rather than a measurement. On tractability: Eq. 2 (App. D) defines the objective against which policies are scored; the paper proposes no solver, and all results come from replaying a trace.

AuthorsOpenReview ↗

Follow-up: New Experiments for Each Dynamic Condition You Raised

We also ran the experiments your question asks for, translated to fine-tuning, the setting the paper studies. Hardware: Qwen3.5-9B + LoRA (bf16) on one L40S (44.4 GiB usable). All quantities were defined in writing before the runs. A figure of these results will be added to the revised PDF.

1. Dynamic prompt lengths: speed fluctuates per step, but not over a checkpoint interval (measured)

training stream (real data)tokens per itemper-step time CVthroughput CV, 60 s windows (measured)600 s windows (derived)
5 tasks pooled, natural lengths, batch 431-24910%0.8%0.2%
concatenated prompts, batch 1128-102425%1.7%0.6%
same, token-budget batching128-102417%6.9%2.1-2.4%

The simulator works at the 600 s checkpoint interval, where the measured variability is at most 2.4%.

2. Batching overhead: padding, throughput and memory (measured on an idle host)

batchpadding, unbucketed / bucketedtokens/s, unbucketed / bucketedpeak memory, unbucketed
219.7% / 1.0%554 / 54123.6 GiB
438.1% / 1.6%1,021 / 1,06830.0 GiB
850.2% / 2.2%1,117 / 1,87143.1 GiB (1.3% of steps OOM)
1658.5% / 2.0%does not fit / 2,253 (20% of steps OOM)does not fit

3. Memory pressure / KV cache: the binding constraint is jobs per GPU (measured)

settingpeak memoryjobs per 44.4 GiB card
training, 256 padded tokens per batch20.8 GiB2 (derived); 2 fit together (measured)
training, 512-1024 tokens24.4-31.8 GiB1 (derived)
training, >= 2048 tokensout of memory0
generation, 1024 new tokens, batch 1 / 8 / 32memory above weights (KV cache) 0.14 / 0.89 / 3.43 GiB; 62 / 65 / 73 ms per decode step-
two training jobs co-located, variable lengths (31-249 tokens)20.6 GiB eachboth fit; slowdown 1.03x median (p10-p90 1.01-1.04; batch 1 only, similar in size to solo run-to-run variation)

Peak training memory is 17.1 GiB + 14.3 GiB per 1,000 padded tokens (fit over 4 cells, residual < 0.05 GiB). With gradient checkpointing it is 17.0 GiB + 0.91 GiB per 1,000 tokens, so two jobs fit up to about 5,100 tokens, at 1.7x step time.

4. Does the formulation hold under these conditions? Simulator re-run (6 trace windows x 3 seeds, 95% CIs over windows)

conditionk=2 net benefitdrawdown net benefitk=2 JCT change, no declining jobss per cell
as submitted (static)-57.3 [-59.7, -55.5]-24.9 [-27.3, -22.9]-16.7%17
measured costs (restart, snapshot, co-location)-57.3-24.9-17.2%26
+ speed variability, CV 0.50 (above every measured value)-57.3 [-59.6, -55.5]-24.9 [-27.4, -22.9]-17.2%24
+ memory cap, 2 jobs per GPU-57.3-24.9-17.5%23
+ multi-device overhead 0.30 (swept, not measured)-57.3-24.9-23.0%87
all three combined-57.3 [-59.6, -55.5]-24.9 [-27.3, -22.8]-23.1%94

Net benefit is % of never-stop useful compute; every rule stays within 0.3 points in all 10 conditions, and false-positive rates and recall do not change. This is partly structural: rules read scores indexed by training progress, so timing changes alter costs, not what a rule sees. The formulation stays tractable: about 185k events per cell in every condition, and the harshest setting is 3.6x slower than the measured-cost baseline. One pre-specified scheduler result changes: under multi-device overhead, FIFO's negative per-job minus GPU-hour gap (-24.8 points) disappears (+2.5 [+0.7, +4.7] at 0.30). This is mostly because deadlines are unchanged while jobs slow; with deadlines extended by each job's own overhead, it stays negative (-7.8 [-11.9, -3.7]).

Reviewer_NK6rOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores metrics reported by early stopping rules for multi-tenant fine-tuning under SLOs by comparing reported job completion against jobs that have been stopped prematurely. For example, this paper finds that under a published rule for jobs that are by construction still progressing, there was a reported 16.7% improvement of completion time, at the cost of 44% of the jobs being stopped prematurely. This paper finds that SLO attainment rests more on aggregation than actual scheduling policy.

The main contributions are:

  • evaluation of a fixed decline threshold on real evaluation curves, along with failure attribution (signal to noise, which changes with model scale)
  • measurements of failures
  • findings of a detectability floor
  • a trace-driven simulator
Critical strengths

This is a really interesting paper with implications on how multi-tenant fine tuning runs should be approached under SLOs. This paper presents pretty convincing evidence that new policies that have been published often kill healthy runs and present that as a "win" for reducing job completion time. A good chunk of this is due to different properties of real, large fine tuning runs vs. smaller, synthetic runs used to create policies (higher signal to noise for the real runs). This paper also showed that a single threshold stopping rule is likely problematic for real world settings (both large and small models being hosted on the same server, with different, model-dependent SNRs), and needs an adaptive rule with higher levels of evidence required for stopping.

Critical weaknesses
  1. The traces in this paper are from pre-training rather than fine tuning, but the results are applied to fine tuning.
  2. There is little comparison in settings with runs where there is collapse, and runs should be stopped.
Questions for authors
  1. How do your results hold up when there actually are runs that need to be stopped (eg Table 1)? Does the aggregation result still hold? Is there still significant damage?
2 comments
AuthorsOpenReview ↗

Response to Reviewer NK6r: Real Fine-Tuning Runs and Real Collapses

Thank you for the careful reading; your two points led to the two largest additions to the paper. Numbers refer to the revised PDF; Tables G1-G2 are in the general response.

W1. The curves are from pre-training, not fine-tuning.

We ran fine-tuning, and the result holds and is stronger. On 15 healthy LoRA fine-tuning runs (one 9B model, 5 multiple-choice tasks x 3 seeds; Sec. 4.5; Table 2) the published k = 2 rule stops 9/15 = 60.0% [35.7, 80.2], against 44.9% on pretraining curves and the published 1.5%. Of those 9 runs, 8 finish with higher accuracy than they had when stopped. The explanation carries over: fine-tuning sits at a lower SNR than pretraining (median 0.13 [0.11, 0.22], max 0.39, against the 1.17 the published rate needs), and the paper's SNR curve predicts 76.5% for these runs, inside the interval. The submitted Sec. 4.5 hedged the other way and is corrected.

One result does not carry over. The noise-normalised drawdown rule (Eq. 3) is clean on the 62 pretraining curves that improve well above their noise (0/62), but it stops 7/15 = 46.7% [24.8, 69.9] of healthy fine-tuning runs (Table G1). Limits: one model, about 5 independent task units, supervised fine-tuning, not RLHF.

W2 / Q. Runs that need to be stopped; aggregation; damage.

We now have real collapses as well as injected declines. The damage to healthy jobs does not shrink, the aggregation gap persists and widens, and one conclusion changed.

Real collapses (App. H.5). Six runs at 10x learning rate all collapse to chance. Fixed k = 2 detects 4 of 6 while stopping 9/15 healthy runs; k = 3 detects 1 of 6 and stops 4/15; Eq. 3 detects 6 of 6 and stops 7/15. No rule from the paper combines detection with few false stops. What separates the runs is the size of the decline: healthy runs fall 1.6-9.9 points below their running best, collapses 19.8-63.9. A rule that requires a drop of at least 10 points stops 0/15 healthy runs [0.0, 20.4] and all 6 collapses within two checkpoints. This is post hoc and in-sample (the value was read off these runs), and the collapses are abrupt: 4 of 6 are at chance by the first of 14 checkpoints, so counts include the zero-shot point. Slow degradation is untested.

Simulator, injected declines, pretraining curves. Table 1 (right) already had 20% of evaluable jobs declining; we extended the sweep to 80%, replaced a goodput metric that charged a perfect oracle -19.2% by net benefit (waste avoided minus useful compute destroyed), and added an oracle row.

Table N1. Net benefit, % of never-stop useful compute, against the share of declining jobs. SJF-Est, 6 windows x 3 seeds, CIs over windows. Accounting and grid fixed before the runs; the 62-curve columns (six larger Pythia scales only) are post hoc.

declining shareOracleDrawdown, all 78Drawdown, 62 (post hoc)k=2, all 78k=2, 62 (post hoc)
0.2+8.9 [+8.5, +9.3]-24.9 [-27.3, -22.9]+5.4 [+5.1, +5.7]-57.3 [-59.7, -55.5]-37.0 [-38.6, -35.4]
0.4+20.3 [+19.9, +20.8]-13.2 [-15.7, -11.5]+12.8 [+12.6, +13.2]-41.3 [-43.1, -39.8]-22.2 [-23.7, -20.8]
0.8+53.0 [+51.7, +54.3]+19.4 [+17.0, +21.7]+34.1 [+33.1, +35.2]+3.4 [+1.5, +5.6]+18.2 [+15.8, +19.8]

What changed: the submission said no rule pays. On all 78 curves that holds up to a declining share of 0.58 (drawdown) and 0.77 (k = 2). On the 62 curves the drawdown rule pays from 0.01; k = 2 only from 0.64 (Tables 18-19, 25). Given W1, we do not extend that gain to fine-tuning.

Damage. False stops do not depend on how many other jobs are declining, because each decision reads only the job's own curve: k = 2 stops 36.3% [35.5, 37.3] of healthy evaluable jobs at 20% declining and 36.3% [35.4, 37.4] at 40%.

Aggregation. Indicative only (3 windows, one seed; mean [min, max]). With a stop rule active the per-job minus GPU-hour gap widens: for SJF-Est from +10.6 [+5.0, +15.8] points with no stop rule to +35.9 [+30.3, +41.3] under k = 2. Among healthy jobs only it is +11.0 and +30.7, so most of the growth is false stops of large jobs. Under k = 2 the schedulers become nearly indistinguishable (spread 3.1 points per job, 0.4 by GPU-hours), so "aggregation moves the number further than the scheduler" holds more strongly and "aggregation reorders the schedulers" only weakly. FIFO's negative gap (-30.2) does not survive an active stop rule (Sec. 6.1; Tables 27-29).

Your reading that stopping needs "higher levels of evidence" is supported: real curves sit below the SNR the published calibration needs (median 0.55 in pretraining and 0.13 in fine-tuning, against 1.17). The fine-tuning runs add that the bar should be on the size of the decline: a statistically significant decline from the running best occurs in 11 of 15 healthy runs at 28 checkpoints.

AuthorsOpenReview ↗

Follow-up: Real Fine-Tuning Declines, DPO Runs, and Aggregation at Full Power

Follow-up: real fine-tuning declines, DPO runs, and the aggregation question at full power

Since our response we ran the three experiments that address your points directly. Same model and hardware as Sec. 4.5 (Qwen3.5-9B + LoRA, one L40S); decline criteria and rules were written down before the runs.

1. W2, "little comparison with runs where there is collapse": real gradual declines. We induced degradation with a raised learning rate (3x, no decay) on a 500-item training subset for 24 epochs; the setting was chosen by a pre-stated pilot rule. All 6 runs (3 tasks x 2 seeds) qualify as real declines (peak not at the end, peak > final, McNemar p < 0.001), and in all 6 the fall is spread over several checkpoints (the largest single interval is 38-69% of the drop). Two (Winogrande) first rise significantly above zero-shot; four decline without rising first. All end at chance. Detection on the 14-checkpoint grid, against the 15 healthy runs:

rulereal declines detectedmedian delay (ckpts)healthy runs stopped
k=2 (published default)6/619/15
k=34/624/15
drawdown, Eq. 36/627/15
drop >= 10 points below running best6/62 (0-3)0/15 [Wilson 0, 20.4]
drop >= 5 points6/602/15

The 10-point threshold was fixed before these runs (it was read from the healthy runs and the abrupt collapses), so its 6/6 detection is out of sample; its 0/15 false stops are not.

2. W1, pretraining vs. fine-tuning, and the RLHF/DPO setting of the prior work: real DPO runs. LoRA DPO on public preference data (Anthropic hh-rlhf, helpful-base), 3 seeds, 1,000 held-out pairs. Held-out preference accuracy improves in 3/3 (44.5% -> 47.4 / 48.3 / 48.1%, p < 0.001; standard error 1.58%). Per-interval SNR is 0.23 / 0.34 / 0.25, far below the 1.17 that the published 1.5% FPR at k=2 requires, consistent with your reading that "higher levels of evidence" are needed. On these healthy runs k=2 stops 2/3 and Eq. 3 1/3; k=3-5 and both drop rules stop 0/3. With n = 3 these are counts, not rates.

3. Your question: does the aggregation result hold with a stop rule active? Now 6 trace windows x 3 seeds (was 3 windows, one seed), measured hardware costs, 20% declining jobs. Per-job minus GPU-hour attainment gap, points:

schedulerno stopk=2drawdown
EDF-SLO+15.6 [+11.3, +19.1]+34.4 [+31.8, +36.4]+24.8 [+21.2, +27.8]
SJF-Est+13.8 [+9.3, +18.2]+34.2 [+31.7, +36.0]+24.6 [+21.1, +27.6]
DRF-fair+10.6 [+7.5, +13.7]+34.1 [+31.6, +36.0]+24.0 [+20.8, +26.6]
FIFO-24.8 [-32.3, -17.4]+25.8 [+21.9, +29.2]+0.8 [-2.5, +4.4]

All four statements we had made from the 3-window runs hold under tests fixed beforehand. k=2 widens the gap for 7/7 schedulers and stops 36.3% of healthy evaluable jobs under every scheduler. It also nearly erases the scheduler comparison by GPU-hours (spread across the 7 schedulers 25.3 -> 1.7 points). New at full power: FIFO's gap changes sign under k=2, because stopping jobs relieves its head-of-line blocking.

Limits. One model; declines from one mechanism (learning-rate mis-setting; overfitting at the standard learning rate did not produce a decline in the one pilot run that tried it, so slow degradation of an otherwise healthy configuration remains untested on real data); DPO on short dialogues, not PPO-based RLHF; the healthy negatives are the same 15 runs from which the 10-point threshold was read.

Reviewer_xheDOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates early-stopping rules using selected Pythia trajectories and Alibaba trace-driven scheduling simulations (Section 4.1; Section 5). Its main contribution is an empirical analysis of stopping-rule calibration and the effects of evaluation metrics on system-level conclusions. The declining trajectories are synthetically injected (Section 5).

Critical strengths
  1. Useful empirical diagnosis. Under the authors’ healthy-run definition, the default k=2 rule terminates 35 of 78 selected trajectories, yielding a reported FPR of 44.9%. The subsequent SNR and checkpoint-budget analyses help explain calibration sensitivity.

  2. Informative evaluation of scheduling metrics. The paper demonstrates that different SLO aggregations change scheduler rankings, providing a concrete reason to report multiple perspectives on system performance.

Critical weaknesses
  1. The economic conclusion needs clarification. The relationship between completed-job goodput and the separate compute-saving analysis is insufficiently explicit (Figure 5(d)). Moreover, Figure 5(d)’s title places GBM’s break-even prevalence at approximately 0.08, whereas its caption states that every rule remains below break-even at π=0.198. This inconsistency affects a central conclusion.

  2. The detection claim appears broader than the supporting analysis. Section 4.4 states that “no causal detector attains it,” while immediately qualifying the calculation as an order-of-magnitude argument rather than a theorem. The categorical wording should be aligned with this qualification and the assumptions acknowledged in Appendix B.

Questions for authors
  1. Which break-even statement in Figure 5(d) is correct? Please also clarify how its compute-saving measure relates to completed-job goodput in Table 1 and which calculation supports the conclusion that no tested policy pays for itself.

  2. Given the explicitly stated limitations of the detection calculation, would the authors narrow the “no causal detector” claim, or provide further justification for its scope?

2 comments
AuthorsOpenReview ↗

Response to Reviewer xheD: Figure 5(d) Resolved, Detection Claim Withdrawn

Thank you: both points were right, and each changed a claim in the paper. Numbers refer to the revised PDF; Tables G1-G2 are in the general response.

Q1 / W1. Which Figure 5(d) statement is correct, how the two measures relate, and which calculation supports "no policy pays".

The in-figure title is correct; the caption, the Sec. 6.2 sentence citing it and the abstract were wrong. The title's break-evens reproduce (0.076 / 0.371 / 0.367 / 0.427 for GBM, drawdown, linear, k = 2). At prevalence 0.198 the GBM saves +0.0085 [-0.0071, +0.0241] run-fractions per job, positive but not distinguishable from zero over 4 folds (Fig. 5(d); Table 26).

The two measures. Figure 5(d) is an offline calculation on curves alone: a declining run never stopped counts fully as waste, and a healthy run stopped by mistake is charged only its sunk compute. Table 1's "net goodput" is the GPU-hours of jobs that ran to completion, so it removes every stopped job, including a declining job stopped correctly. The first is lenient to false stops; the second penalises correct stops.

Which calculation. Neither, as submitted: under the Table 1 metric a perfect oracle scores -19.2%. We replaced it, with definitions fixed before any new run: net benefit = waste avoided minus useful compute destroyed, as % of never-stop useful compute, next to an oracle row (Table 4; Sec. 3). On 6 windows x 3 seeds the oracle gains +8.9% [+8.5, +9.3], and on all 78 curves every tested rule is negative in 18 of 18 cells (Table G2). Four of five accounting variants agree; under last-checkpoint credit k = 3 is +1.5% [+1.2, +1.7] (App. H.3).

Two pools (post hoc). Eq. 3 fires only on the 16 curves from the two smallest Pythia scales, 11 of which do not improve. In the all-78 runs its FPR is 0.015 on jobs replaying the 62 curves and 0.843 on jobs replaying the 16, and 99.0% of the healthy compute it destroys is on the 16. For k = 2 the FPRs are 0.236 and 0.853 and the share is 46.5%: its damage is in both pools. Restricted to the 62 curves, the drawdown rule pays +5.4% [+5.1, +5.7] and k = 2 stays at -37.0% [-38.6, -35.4] (Sec. 6.2; Table 17). So on all 78 pretraining curves no tested rule pays at 20% declining; on the 62 the drawdown rule does. That gain does not extend to fine-tuning (last section).

Q2 / W2. Would the authors narrow the "no causal detector" claim?

Yes: it is withdrawn, not reworded. We measured what causal detectors achieve (Sec. 4.4; Table 1).

Table X1. Pre-specified. Best of 86 causal detector configurations at a 1.5% false-alarm budget (realised 1.2-2.5%); thresholds from dependence-preserving resamples of other model scales; held-out scales; CIs over 8 or 6 scales. g = slope removed, in units of the healthy slope (g=1 is a stall).

curveschangerecall %same thresholds on the actual curves: false alarms; recall %
all 78g=14.3 [2.6, 5.6]1/78; 10.7 [3.8, 16.3]
all 78g=215.6 [8.7, 21.0]1/78; 34.6 [19.6, 46.3]
all 78g=446.5 [28.4, 61.5]1/78; 50.4 [28.6, 69.0]
the 62g=117.4 [14.7, 20.2]0/62; 16.7 [9.5, 25.2]
the 62g=266.5 [63.2, 70.0]0/62; 95.2 [89.9, 99.0]
the 62g=497.8 [96.7, 98.8]0/62; 100.0 [100.0, 100.0]

What was wrong. Appendix B said dependence lengthens the delay; measured, i.i.d. resampling is the pessimistic case (recall 24.8% against 66.5% at g=2 on the 62 curves), and "45 checkpoints" matches only that case (12.6 [12.0, 13.3] with dependence kept; App. B). And on the 62 improving curves Eq. 3 at the submitted c = 1.5 fires on 0/62 with 97.8% [95.1, 99.5] stall recall (in-sample; a zero count bounds the rate at 4.7%, not 1.5%).

What remains. (1) Fixed-k cannot reach the published point for stalls or declines up to g=2 at any k: with no false alarm on the 62 curves its best recall is 44.6% [28.3, 62.3] and 76.3% [58.7, 91.4] (post hoc). (2) No detector we tried works once the 16 low-signal curves are included (Table X1, top rows). (3) Under the pre-specified analysis a stall is not detected at a 1.5% budget in either population.

Your summary: "the declining trajectories are synthetically injected".

That remains true of the simulator and of every recall figure above; we have now added real runs (Sec. 4.5; Table 2, App. H.5): 15 healthy LoRA fine-tuning runs of one 9B model and 6 real collapses (10x learning rate). Fixed k = 2 stops 9/15 = 60.0% [35.7, 80.2] of the healthy runs and detects 4 of 6 collapses. Eq. 3 detects 6 of 6 but stops 7/15 healthy runs: its clean result on the 62 pretraining curves does not transfer to fine-tuning, where SNR is far lower (median 0.13). Post hoc, a drop of at least 10 points below the running best separates all 21 runs (Table G1). Limits: one model, about 5 independent task units, abrupt collapses only (4 of 6 are at chance by the first of 14 checkpoints, so counts include the zero-shot point), not RLHF.

AuthorsOpenReview ↗

Follow-up: Real (Non-Injected) Declines

Follow-up: real (non-injected) declines, as your summary noted that all declining trajectories were synthetic

You noted that the declining trajectories in the paper are synthetically injected. We now have real ones. On Qwen3.5-9B + LoRA (one L40S), a raised learning rate (3x, no decay) on a 500-item subset for 24 epochs produced 6 real declines (3 tasks x 2 seeds). Every run qualifies under a criterion fixed before the runs (peak not at the end, peak > final, McNemar p < 0.001). The fall is gradual in all 6 (no single interval holds more than 69% of the drop). Two runs rise above zero-shot before falling; four decline from the start. These complement the 6 abrupt collapses (10x learning rate) already in Sec. 4.5.

Detection on real declines vs. false stops on 15 healthy fine-tuning runs (14-checkpoint grid):

rulereal gradual declines detectedhealthy runs stopped
k=2 (published default)6/69/15 = 60% [35.7, 80.2]
k=34/64/15
drawdown, Eq. 3 (c = 1.5)6/67/15
drop >= 10 points below running best6/6 (median delay 2 ckpts)0/15 [0, 20.4]
prefix-improvement gate (t >= 1.645)0/62/15

How this bears on your two questions:

  • The detection claim (Q2). On real declines the picture matches the narrowed claim in the revision. Counting consecutive declines cannot separate decline from health on these runs at any k: k=2 catches everything but stops 60% of healthy runs, and raising k loses real declines before it removes false stops. Eq. 3 is not adequate in the fine-tuning regime either (7/15 false stops), because healthy fine-tuning curves are low-SNR (median 0.13). What separates the two populations here is the size of the decline: a 10-point drop below the running best catches 6/6 gradual declines (and the 6 abrupt collapses of Sec. 4.5 when the zero-shot point is included) and stops 0/15 healthy runs. The threshold was fixed before the gradual-decline runs, so their detection is out of sample. The 0/15 is in sample, and its upper bound of 20.4% cannot certify a 1.5% false-positive rate. We present this as a direction, not as a calibrated detector.
  • The gate we proposed offline fails on real data, and we say so: it never opens on a declining run (prefix t from -1.80 to -0.21), so it detects none of them.

DPO (the prior work's setting). On 3 healthy LoRA DPO runs (public hh-rlhf preference data), held-out preference accuracy improves (p < 0.001) with per-interval SNR 0.23-0.34. k=2 stops 2 of 3; the 10-point rule stops none.

Limits. One model; declines come from one mechanism (learning-rate mis-setting; overfitting at the standard learning rate did not decline in a pilot run); healthy negatives are the 15 runs the 10-point threshold was read from; DPO is not PPO-based RLHF.

Reviewer_1xHJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines an early-stopping rule that terminates a training job after its evaluation score decreases for kk consecutive intervals, freeing its GPUs for other jobs. The authors first test for false terminations using evaluation trajectories from ordinary Pythia pretraining runs, then simulate the scheduling effects using the Alibaba PAI 2020 cluster trace. They attribute the false terminations mainly to low SNR: training progress is small relative to score fluctuations, making the fixed rule mistake normal fluctuations for degradation. The released resources allow other jobs to start sooner, reducing mean job completion time (Mean JCT), which includes both waiting and training time. A lower mean JCT therefore does not necessarily indicate better system performance: the evaluation must also account for jobs that should have continued training but were stopped.

Critical strengths
  • The manuscript is well organized, with a coherent flow from problem formulation through design and experimental validation.
  • The zero-decline control experiment is convincing. It clearly shows how mean completion time can obscure incorrect termination decisions.
  • The analysis of SNR and evaluation intervals helps explain why the false-positive rate of a fixed rule varies across conditions.
Critical weaknesses

W1. The GPU execution model needs validation

My main concern is how the simulator treats execution time. L129–130 states:

“We adopt a non-preemptive, no-slowdown execution model, as the predecessor’s simulator does, so SjS_j is schedule-invariant.”

In shared-GPU or multi-GPU training, execution time can depend on contention for compute resources, memory bandwidth, and PCIe/NVLink communication. The paper needs to explain why these effects can be neglected in the intended deployment and whether this simplification changes the reported scheduling benefits. Hardware measurements or model calibration are currently missing.

L452 also describes restart overhead and snapshot interval as “choices, not measurements.” The authors sweep these parameters, but measurements are needed to establish which ranges correspond to actual deployments. A production trace provides workload information; it does not by itself validate execution costs under a changed policy. I therefore see the current results as evidence about the simulated model, with insufficient validation for conclusions about benefits in real GPU fine-tuning platforms.

W2. The writing is too compressed to communicate the contribution and experimental findings clearly

I spent considerable effort unpacking sentences and still struggled to identify the main point. For example, L066–069 summarizes the contributions as follows:

“This paper contributes that refutation and its attribution (Section 4), the measurement of objective fragility (Sections 3 and 6), a break-even accounting and the detectability floor behind the negative learning result (Section 6), and an open trace-driven simulator with a mechanically enforced information boundary (Section 5).”

This sentence combines a refutation of prior work, causal attribution, evaluation metrics, detection limits, and a simulator under labels such as “objective fragility,” “detectability floor,” and “mechanically enforced information boundary.” A contribution paragraph should help readers understand what was done and what was learned. Here, readers must first unpack these labels and work out how the claims relate to one another.

Another example appears at L414–416:

“Separately, exposing slo class hands a policy a coarsened duration oracle, the classes being derived from duration antimodes, and hiding it degrades EDF-SLO to byte-identical equality with its FIFO-backfill fallback at a cost of 5.56pp of per-job attainment.”

The technical point is straightforward, but the sentence is difficult to follow. The inserted explanation of how the classes are derived interrupts the account of the experiment, and “degrades ... to byte-identical equality” is an awkward way to describe the fallback behavior. The change in attainment is then added to the same sentence. Readers have to unpack all of this to recover a simple experimental finding. Please split it into two or three sentences stating the intervention, the scheduler's behavior, and the change in attainment.

I recommend a careful revision by the authors, paragraph by paragraph. The motivation and main contribution should be stated plainly, and sentences like these should be broken into explanations that readers can follow. The current presentation makes understanding and evaluating the work unnecessarily difficult and is not ready for acceptance.

Questions for authors
  • Which specific problem in deployed fine-tuning platforms motivates this work? Is there operational data or a deployment example showing its importance?
  • Section 4.3 explains the difference from prior results by adjusting SNR. Beyond this finding, what do the authors consider the main new insight of the paper?
  • What concrete guidance can system designers take from these results? For example, can the authors identify conditions under which a stopping rule should or should not be used, and explain how to assess those conditions in practice?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1xHJ: Hardware Measurements, Rewritten Contributions, Designer Guidance

Thank you for the detailed review; both main concerns were fair. Numbers refer to the revised PDF.

W1. The GPU execution model needs validation.

We agree. We have now measured execution costs on real hardware and added a slowdown model; we still do not claim end-to-end validation of the simulator.

Measured (one L40S, 9B model, LoRA; Sec. 6.3; App. H.5): Snapshot of adapter and optimizer: 2.66 s; of full weights: 145.8 s. Restart to first training step: 8.6 s warm, 155.9 s cold (n=3). Both lie inside the 0-600 s the submission swept. Two jobs fit on the card only at batch 4, where each runs 1.19x slower (a lower bound), so the no-slowdown assumption does not hold when jobs share a GPU. Evaluating 28 checkpoints costs 1.18x the training time (0.59x at 14), a cost the simulator does not charge.

Slowdown model (Sec. 6.3; App. H.4). A job runs at speed 1/((1+αsharen)(1+αcommm))1/((1+\alpha_{share}n)(1+\alpha_{comm}m)), with nn co-tenants on its shared device and m=1m=1 if it spans several devices; the measured 1.19x is αshare≈0.19\alpha_{share}\approx 0.19. We swept 15 settings up to αshare=0.4\alpha_{share}=0.4, αcomm=0.15\alpha_{comm}=0.15 (3 trace windows, indicative):

  1. Stop-rule net benefit keeps its ordering in 15/15 settings and moves by at most 0.4 points.
  2. With no declining jobs, k = 2 still "improves" mean JCT in 135/135 cells while destroying 42.8-42.9% of completed work.
  3. The aggregation gap keeps its sign for 5 of 6 schedulers; its size depends on the deadline definition.

Sec. 4 never passes through the simulator. Restart and snapshot costs (L452) act only on preempted jobs; all stop-rule results use SJF-Est, which does not preempt. A simulator re-run at the measured values will be posted here.

W2. The writing is too compressed.

Agreed; the paper was edited paragraph by paragraph. The passages you quoted now read:

Contributions (Sec. 1):

  1. A test of a published stop rule on real curves. The rule stops a job after two consecutive drops in its evaluation score and reports a 1.5% false-positive rate. It fires on 44.9% of 78 healthy pretraining curves and on 60.0% of 15 healthy fine-tuning runs.
  2. An explanation. The published rate needs a run to improve by more than its noise at every checkpoint. Real curves do not, fine-tuning runs least of all. 3.-4. Two measurement problems and an accounting for whether stopping pays (waste avoided minus useful compute destroyed, next to an oracle).

Sec. 6.3, Sensitivity:

SLO classes are derived from job duration, so a policy that reads a job's class learns roughly how long the job will run. We hid the class from EDF-SLO. It then makes the same decisions as FIFO-backfill, and its per-job attainment falls by 5.56 points.

Q1. Which deployed-platform problem motivates this?

Stopping jobs on their evaluation curves is a published and deployed practice, it acts on almost all of a cluster's compute, and its damage is measurable. Curve-based stopping is deployed in tuning services (Vizier's median stopping rule; Sec. 2), and Zhang et al. (2026) propose it at the scheduler for multi-tenant RLHF. In the production trace, the long-running bulk jobs such a rule can act on hold 96.05% of GPU-hours. We use no operational data from a deployed platform.

Q2. Main insight beyond the SNR finding?

Whether early termination pays is decided by the population of curves, and the standard metrics hide the damage. The same drawdown rule scores -24.9% on all 78 pretraining curves and +5.4% on the 62 that improve well above their noise (post hoc; Table G2, general response), and it stops 7/15 healthy fine-tuning runs. Mean JCT improves by 16.7% while k = 2 destroys 42.7% of useful compute; completed-job goodput charges a perfect oracle -19.2%. Net benefit against an oracle exposes it.

Q3. Guidance for designers (Sec. 7; App. H.5)

  1. Measure on your own healthy runs, at your cadence: per-interval SNR, and how far a healthy run falls below its running best.
  2. Do not count consecutive declines: k = 2 stops 30.6% of clearly improving pretraining curves and 60.0% of healthy fine-tuning runs (9/15); k = 5 stops none but detects none of 6 real collapses.
  3. Do not stop on statistical evidence of a decline alone. Eq. 3 fires on 0/62 improving pretraining curves but on 7/15 healthy fine-tuning runs. An improvement gate that looked good offline on pretraining curves (false alarms 16/78 to 2/78) fails on fine-tuning: it stops 2/15 healthy runs and detects 0 of 6 collapses. We do not recommend it.
  4. Require a decline larger than what the tenant considers harmful. Post hoc on our runs, a drop of at least 10 points below the running best stops 0/15 healthy runs [0.0, 20.4] and all 6 collapses within two checkpoints. The value was read off these runs (healthy drops 1.6-9.9 points, collapses 19.8-63.9); one model, abrupt collapses only.
  5. Report net benefit against never stopping and an oracle, with the FPR at its cadence; not mean JCT.
AuthorsOpenReview ↗

Follow-up: simulator re-run at the measured hardware costs (W1)

As promised, we re-ran the simulator with its execution parameters set from the L40S measurements instead of the submitted choices. All parameter values were written down before the runs. Same trace windows, seeds and rules as Table 4: SJF-Est, 20% declining jobs, 6 windows x 3 seeds, CIs over windows.

parametersubmittedmeasured (re-run)
restart after preemption60 s9.59 s (warm restart, process total)
snapshot costfree2.66 s per 600 s (adapter + optimizer save)
slowdown per co-tenant on a shared GPU00.185 (two co-located jobs, 1.19x)

Stop rules: no conclusion changes. Net benefit, % of never-stop useful compute (submitted -> measured):

ruleall 78 curves62 improving curves (post hoc)
Oracle+8.9 -> +8.9 [+8.6, +9.3]+8.9 -> +8.9
Drawdown (Eq. 3)-24.9 -> -24.9 [-27.1, -23.0]+5.4 -> +5.4 [+5.1, +5.6]
Learned-9.5 -> -9.5+0.33 -> +0.35
Fixed k=3-23.9 -> -23.8-5.6 -> -5.6
Fixed k=2 (published)-57.3 -> -57.3 [-59.4, -55.6]-37.0 -> -37.1

No sign and no ordering changes; false-positive rates and recall change by at most 0.002. This is partly structural: stop decisions are taken at checkpoints counted in service time, so a slowdown rescales the cost of stopping and of not stopping alike. With no declining jobs, k=2 still "improves" mean JCT (-17.2% [-17.4, -17.0], was -16.7%) while destroying 42.7% [41.6, 43.9] of completed work.

Schedulers: levels move, conclusions hold. Jobs run slower against unchanged deadlines (mean stretch 1.23), so SLO attainment falls by 9-21 points for every scheduler (e.g. SJF-Est per job 0.984 -> 0.896). The per-job minus GPU-hour gap keeps its sign for all 7 schedulers and widens for the 6 positive ones (SJF-Est +10.9 -> +13.8 points; FIFO stays negative, -25.6 -> -24.8). The leaders are unchanged (EDF-SLO per job, DRF-fair by GPU-hours); some mid-table ranks swap.

Limits. The 0.185 was measured for two jobs at batch 4; three or more did not fit in memory. The simulator applies it linearly per co-tenant, and in one audited cell about 38% of running job-time (5% of GPU-time) had two or more co-tenants, so the attainment levels rest partly on that extrapolation. The stop-rule results do not. The re-run is one model's costs on one GPU; it calibrates parameters and is not an end-to-end validation.

AuthorsOpenReview ↗

Follow-up 2: Full-Power Slowdown Sweep and Online Regime Test (Q3)

Follow-up 2: the slowdown sweep at full power, and an answer to Q3 (how a practitioner can tell, online, whether a stop rule is usable)

1. Interference / slowdown sweep (W1), now 6 trace windows x 3 seeds (was 3 windows). SJF-Est, 20% declining jobs; per-co-tenant slowdown alpha_share in {0, 0.05, 0.1, 0.2, 0.4} x multi-device overhead alpha_comm in {0, 0.05, 0.15} (15 settings, 18 cells each). Net benefit of stopping, % of never-stop useful compute:

rulealpha = 0 [95% CI]range over 15 settingslargest change vs alpha = 0
oracle+8.9 [+8.5, +9.3]+8.8 to +8.90.11 pp
k=3-23.9 [-25.6, -22.5]-23.9 to -23.60.27 pp
drawdown (Eq. 3)-24.9 [-27.3, -22.9]-24.9 to -24.80.11 pp
k=2 (published)-57.3 [-59.7, -55.5]-57.4 to -57.30.05 pp

All four statements we had made from the 3-window runs, with tests fixed before this run, hold in 15/15 settings: same ordering, same signs, every rule within 1 pp of alpha = 0, and with no declining jobs k=2 still "improves" mean JCT (-16.7% to -20.7%) while destroying 42.5-42.7% of completed work. As in our previous follow-up, this insensitivity is largely structural (rules read scores indexed by training progress), so we report it as robustness of the conclusions, not as independent confirmation. Only alpha_share = 0.185 is measured; the grid is a sweep.

2. Q3: can a practitioner tell online whether a run is in the regime where a decline rule is usable? Rule written down before computing: after the first m eval increments, call the run "measurably improving" if the one-sided t statistic of their mean is >= 1.645; compare with the same test on the full run.

corpusrunsimproving (full run)agreement at m = 4m = 5m = 8accuracy >= 0.9 from m =
Pythia pretraining, 14 checkpoints785083.3%88.5%94.9%7
fine-tuning (9B LoRA), 14 checkpoints15086.7%93.3%100%5
fine-tuning, 28 checkpoints15093.3%100%100%4

So the regime can be read off the first 5-8 checkpoints of the job itself. None of our 15 healthy fine-tuning runs is measurably improving at its cadence, which is exactly why decline counting misfires there. Practical guidance, as in the revised paper: measure this first; if the increments are not significant, do not auto-terminate on eval declines; if they are, use a noise-normalised drawdown, never a count of consecutive declines; report net benefit against an oracle with the FPR at its cadence.

Limits: the label is computed from the same curve, so agreement at large m is optimistic; no healthy fine-tuning run is improving, so sensitivity cannot be estimated there; one model; designed during the discussion period.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of Revisions, New Real Fine-Tuning Evidence, and 2 Corrected Claims

We thank the four reviewers. Two of the points raised were correct and changed what the paper claims. New real fine-tuning runs confirm the paper's main result and confine one positive result of the revision to pretraining curves. Numbers refer to the revised PDF.

Real fine-tuning runs (new) Sec. 4.5; Table 2. Scope: one model (Qwen3.5-9B, LoRA, supervised multiple-choice), 15 healthy runs (5 tasks x 3 seeds, about 5 independent units), 6 real collapses (10x learning rate), one L40S; not RLHF.

  • Confirmed. The published k = 2 rule stops 9/15 = 60.0% [35.7, 80.2] of healthy fine-tuning runs. Fine-tuning has lower SNR than pretraining (median 0.13 [0.11, 0.22], max 0.39; no run reaches the 1.17 the published rate needs), so the hedge in the submitted Sec. 4.5 pointed the wrong way.
  • Not confirmed. The noise-normalised drawdown rule (Eq. 3), which fires on 0 of 62 improving pretraining curves, stops 7/15 = 46.7% [24.8, 69.9] of healthy fine-tuning runs. Its positive result does not transfer.
  • New. In fine-tuning, harmful and harmless runs differ in the size of the decline, not in the statistical evidence for one (Table G1; post hoc; App. H.5).

Corrections to the submission

(a) "No causal detector attains the published operating point" is withdrawn (xheD). Of 86 causal detector configurations at a 1.5% false-alarm budget, recall on the 62 larger-scale Pythia curves is 17.4% [14.7, 20.2] for stalls but 66.5% [63.2, 70.0] and 97.8% [96.7, 98.8] for declines two and four times the healthy slope. Appendix B's dependence argument had the wrong sign and is removed.

(b) "Nothing we tested pays for itself" is narrowed (xheD, NK6r). The Figure 5(d) caption contradicted the figure, and the Table 1 metric charged a perfect oracle -19.2%. With net benefit (waste avoided minus useful compute destroyed) and an oracle row, fixed before any new run, every tested rule is negative on all 78 Pythia curves, while on the 62 the drawdown rule gains +5.4% [+5.1, +5.7] (Table G2; split post hoc).

(c) That gain belongs to curves that improve well above their noise. Fine-tuning runs do not, and we have no net-benefit figure for them.

What stands

  1. Counting consecutive declines fails on real pretraining and fine-tuning curves at every useful k (Table G1).
  2. With no declining jobs, k = 2 "improves" mean JCT by 16.7% [16.4, 17.0] while destroying 42.7% [41.6, 43.9] of useful compute.
  3. k = 2 does not pay under any of five accounting variants, in either Pythia pool.
  4. Whether stopping pays must be reported as net benefit against an oracle, with the FPR quoted at its cadence.

Table G1. Healthy real curves stopped by each rule, and real collapses detected. Pythia: 78 pretraining curves split by model scale (post hoc). Healthy counts use 14 checkpoints. Collapses are counted with the zero-shot point; on the 14-checkpoint grid alone only 2 of 6 are observable, because 4 are at chance by the first checkpoint. The gate was designed on the Pythia curves; the 10-point value was read off the fine-tuning runs.

rulePythia, 2 smallest scalesPythia, 6 larger scalesfine-tuning, healthy runs stopped [Wilson 95%]real collapses detected
Fixed k=2 (published)16/1619/629/15 = 60.0 [35.7, 80.2]4/6
Fixed k=310/164/624/15 = 26.7 [10.9, 52.0]1/6
Fixed k=51/160/620/15 = 0.0 [0.0, 20.4]0/6
Drawdown, Eq. 3 (c=1.5)16/160/627/15 = 46.7 [24.8, 69.9]6/6
Eq. 3 + improvement gate (t >= 1.645)2/160/622/15 = 13.3 [3.7, 37.9]0/6
Drop >= 10 points below running best (post hoc)not testednot tested0/15 = 0.0 [0.0, 20.4]6/6

Table G2. Simulator, Pythia curves: net benefit (% of never-stop useful compute) and break-even share of declining jobs. SJF-Est, 20% declining, 6 windows x 3 seeds, CIs over windows.

stop rulenet benefit %, all 78net benefit %, 62 (post hoc)break-even, all 78break-even, 62
Oracle+8.9 [+8.5, +9.3]+8.9 [+8.5, +9.3]0.000.00
Fixed k=3-23.9 [-25.6, -22.5]-5.6 [-6.8, -4.7]0.650.35
Drawdown (Eq. 3)-24.9 [-27.3, -22.9]+5.4 [+5.1, +5.7]0.580.01
Fixed k=2 (published)-57.3 [-59.7, -55.5]-37.0 [-38.6, -35.4]0.770.64

Where. Detector frontier: Sec. 4.4, App. B. Accounting: Table 4, Sec. 6.2. Slowdown sweep: Sec. 6.3, App. H.4. Fine-tuning runs: Sec. 4.5; Table 2. Real collapses and the size rule: App. H.5. Hardware measurements: Sec. 6.3; App. H.5. All changes: App. H.1.

Provisional: the slowdown sweep and the scheduler runs with a stop rule active use 3 trace windows; Eq. 3's 0/62 is in-sample (upper bound 4.7%); Pythia recall is against synthetic slope changes; the size rule is post hoc and in-sample. A simulator re-run calibrated with the measured hardware costs is pending.

AuthorsOpenReview ↗

Revised PDF: Second-Round Experiments (Appendix H.6)

Thank for all reviewers for their insightful reviews. We uploaded a revision that adds every experiment from our follow-ups as a new appendix; changes are in blue, following suggestions.

whatwhereasked by
6 real gradual declines and 3 DPO runs (9B LoRA), stop rules evaluated on themApp. H.6.1 (Fig. 15, Tables 41-42)xheD, NK6r
Hardware under dynamic conditions: variable lengths, batching, memory/KV cache, co-locationApp. H.6.2 (Fig. 16, Tables 43-45)vybW
Simulator under 11 dynamic conditions (variable speed, memory caps, multi-device overhead)App. H.6.3 (Fig. 17, Tables 46-47)vybW
Aggregation gap with a stop rule active, and slowdown sweep, at 6 windows x 3 seedsApp. H.6.4 (Fig. 18, Tables 48-50)NK6r, 1xHJ
Simulator re-run at measured hardware costsApp. H.6.5 (Table 51)1xHJ
Recognising the regime online from the first checkpointsApp. H.6.6 (Table 52)1xHJ

Main-text updates that follow from these results: Sec. 6.1 and Sec. 6.3 now quote the 6-window x 3-seed runs at measured costs instead of the earlier 3-window runs, and Sec. 7 (limitations and guidance) reflects the new measurements. One change narrows a statement: the main text no longer says that DPO curves are unmeasured; it points to the three DPO runs and says that RLHF remains unmeasured. The summary of changes is in App. H.1 (items 21-26).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.