When Fixed Thresholds Fail: Trace-Grounded Scheduling for Multi-Tenant LLM Fine-Tuning under SLOs

Multi-tenant platforms can schedule LLM fine-tuning by watching each job's evaluation curve, terminating runs whose score declines and handing the freed GPUs to another tenant. A recently published rule of this kind reports a $1.5\%$ false-positive rate. We replay it on $78$ real per-checkpoint trajectories from public pretraining runs. Every run is healthy, so every termination is an error, and the rule fires on $44.9\%$ of them. Neither obvious explanation holds up. The synthetic concave curve used for calibration predicts an even higher rate, and real evaluation noise is milder than assumed rather than harsher. What actually carries the published number is a quantity it never states, the per-interval signal-to-noise ratio. Real curves sit at a median of $0.55$, whereas $1.5\%$ at $k=2$ demands $1.17$. Correct that single parameter and the original generator reproduces our measurement to within four points. Raising $k$ does not rescue the rule, because real declines cluster, and a classical quickest-detection bound puts the delay needed for a $1.5\%$ false-alarm rate at three times the checkpoints a job has, so by that bound no causal detector reaches the published operating point on these curves. Standard reporting would not have caught this. Replaying a $714,903$-job production trace, we find that how SLO attainment is aggregated moves the reported number further than which scheduler produced it, by as much as $17.7$ points, and reorders the policies outright. Early termination then corrupts the usual metrics in one direction. On a workload with no declining jobs at all, the published rule “improves” mean job completion time by $16.7\%$ while destroying $44\%$ of useful compute. At the trace's actual prevalence of decline, nothing we tested pays for itself, including a learned policy that never beats a one-parameter rule normalising drawdown by an online noise estimate. We recommend net goodput reported with a false-positive rate quoted at its checkpoint cadence, the one pairing that survived every probe we ran.