- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a multi-tenant tuning scheduling for LLM jobs based on a concept of "curve-based early termination" rules (such as terminating jobs after consecutive evaluation score declines) to reallocate GPU resources. The authors replay a heuristic against real trajectories and a cluster trace. The authors showed that fixed-threshold stopping rules suffer from high false-positive rates and claimed that adaptive heuristics cannot meet the SLA required by realistic workloads.
Critical strengths
The authors quantify how standard scheduling metrics e.g., mean job completion time can be corrupted and manipulated by early termination behaviors. This is an interesting insight into the scheduling problem and associated metrics.
Critical weaknesses
The premise of modeling LLMs simply as a set of GPU jobs with (static) resource demands in GPU is extremely simplistic and fragile. Such modeling cannot reflect current datacenter complexities of modern LLM workload. Moreover, the methodology relies on synthetic data, as it seems that the authors artificially inject some trajectories into performance curves. This leaves the entire evaluation dependent on unrealistic assumptions and weakens the results and the insights that it could have offered.
Questions for authors
Some important questions that the authors need to address. The paper models LLM workloads as set of GPU jobs with some GPU demands. How does this model account for real-world production complexities in modern LLM workloads like dynamic prompt/generation lengths, batching overheads, KV-cache memory pressure, and multi-node pipeline parallelism? Would the proposed formulation remain tractable or hold up under these dynamic conditions?
2 comments
Follow-up: New Experiments for Each Dynamic Condition You Raised
We also ran the experiments your question asks for, translated to fine-tuning, the setting the paper studies. Hardware: Qwen3.5-9B + LoRA (bf16) on one L40S (44.4 GiB usable). All quantities were defined in writing before the runs. A figure of these results will be added to the revised PDF.
1. Dynamic prompt lengths: speed fluctuates per step, but not over a checkpoint interval (measured)
| training stream (real data) | tokens per item | per-step time CV | throughput CV, 60 s windows (measured) | 600 s windows (derived) |
|---|---|---|---|---|
| 5 tasks pooled, natural lengths, batch 4 | 31-249 | 10% | 0.8% | 0.2% |
| concatenated prompts, batch 1 | 128-1024 | 25% | 1.7% | 0.6% |
| same, token-budget batching | 128-1024 | 17% | 6.9% | 2.1-2.4% |
The simulator works at the 600 s checkpoint interval, where the measured variability is at most 2.4%.
2. Batching overhead: padding, throughput and memory (measured on an idle host)
| batch | padding, unbucketed / bucketed | tokens/s, unbucketed / bucketed | peak memory, unbucketed |
|---|---|---|---|
| 2 | 19.7% / 1.0% | 554 / 541 | 23.6 GiB |
| 4 | 38.1% / 1.6% | 1,021 / 1,068 | 30.0 GiB |
| 8 | 50.2% / 2.2% | 1,117 / 1,871 | 43.1 GiB (1.3% of steps OOM) |
| 16 | 58.5% / 2.0% | does not fit / 2,253 (20% of steps OOM) | does not fit |
3. Memory pressure / KV cache: the binding constraint is jobs per GPU (measured)
| setting | peak memory | jobs per 44.4 GiB card |
|---|---|---|
| training, 256 padded tokens per batch | 20.8 GiB | 2 (derived); 2 fit together (measured) |
| training, 512-1024 tokens | 24.4-31.8 GiB | 1 (derived) |
| training, >= 2048 tokens | out of memory | 0 |
| generation, 1024 new tokens, batch 1 / 8 / 32 | memory above weights (KV cache) 0.14 / 0.89 / 3.43 GiB; 62 / 65 / 73 ms per decode step | - |
| two training jobs co-located, variable lengths (31-249 tokens) | 20.6 GiB each | both fit; slowdown 1.03x median (p10-p90 1.01-1.04; batch 1 only, similar in size to solo run-to-run variation) |
Peak training memory is 17.1 GiB + 14.3 GiB per 1,000 padded tokens (fit over 4 cells, residual < 0.05 GiB). With gradient checkpointing it is 17.0 GiB + 0.91 GiB per 1,000 tokens, so two jobs fit up to about 5,100 tokens, at 1.7x step time.
4. Does the formulation hold under these conditions? Simulator re-run (6 trace windows x 3 seeds, 95% CIs over windows)
| condition | k=2 net benefit | drawdown net benefit | k=2 JCT change, no declining jobs | s per cell |
|---|---|---|---|---|
| as submitted (static) | -57.3 [-59.7, -55.5] | -24.9 [-27.3, -22.9] | -16.7% | 17 |
| measured costs (restart, snapshot, co-location) | -57.3 | -24.9 | -17.2% | 26 |
| + speed variability, CV 0.50 (above every measured value) | -57.3 [-59.6, -55.5] | -24.9 [-27.4, -22.9] | -17.2% | 24 |
| + memory cap, 2 jobs per GPU | -57.3 | -24.9 | -17.5% | 23 |
| + multi-device overhead 0.30 (swept, not measured) | -57.3 | -24.9 | -23.0% | 87 |
| all three combined | -57.3 [-59.6, -55.5] | -24.9 [-27.3, -22.8] | -23.1% | 94 |
Net benefit is % of never-stop useful compute; every rule stays within 0.3 points in all 10 conditions, and false-positive rates and recall do not change. This is partly structural: rules read scores indexed by training progress, so timing changes alter costs, not what a rule sees. The formulation stays tractable: about 185k events per cell in every condition, and the harshest setting is 3.6x slower than the measured-cost baseline. One pre-specified scheduler result changes: under multi-device overhead, FIFO's negative per-job minus GPU-hour gap (-24.8 points) disappears (+2.5 [+0.7, +4.7] at 0.30). This is mostly because deadlines are unchanged while jobs slow; with deadlines extended by each job's own overhead, it stays negative (-7.8 [-11.9, -3.7]).
Response to Reviewer vybW: Scope, Real (Non-Synthetic) Evidence, Device Sharing
Thank you for the review, and for noting the result on how early termination corrupts mean job completion time. Numbers refer to the revised PDF; Table G1 is in the general response.
Scope. The paper does not propose a scheduler or a stopping rule: it tests a published stopping rule (Zhang et al., 2026) and seven existing schedulers. It studies fine-tuning, that is training jobs, not inference serving. The revised introduction says both (Sec. 1).
W. Static GPU-job model; reliance on synthetic data.
We agree that execution speed is not static when training jobs share a device, and the revision measures and tests this. On one L40S, two co-located fine-tuning jobs each run 1.19x slower (a lower bound; Sec. 6.3; App. H.5). We added a slowdown model for device sharing and multi-device communication and swept 15 settings whose range includes this value (Sec. 6.3; App. H.4). The ordering of stop rules by net benefit is unchanged in all 15 and their sizes move by at most 0.4 points; the finding that the published rule "improves" mean JCT while destroying useful compute holds in all 135 cells. These runs use 3 trace windows and are indicative.
The headline result uses no synthetic data, and the revision adds real fine-tuning runs and real collapses. The published k = 2 rule fires on 35 of 78 real, healthy pretraining curves (44.9% [34.3, 55.9]) and on 9 of 15 healthy LoRA fine-tuning runs (60.0% [35.7, 80.2]; Sec. 4.5; Table 2), against the published 1.5%. The probe behind the JCT result has no declining jobs at all. Injected declines are still what the simulator and the recall figures use. To check them against real ones we ran 6 fine-tuning runs that really collapse (10x learning rate): k = 2 detects 4 of 6 while stopping 9 of 15 healthy runs (Table G1; App. H.5). Limits: one 9B model, about 5 independent task units, abrupt collapses only, not RLHF.
Q. Prompt/generation lengths, batching overheads, KV-cache pressure, multi-node parallelism; tractability.
Prompt and generation lengths, batching overheads and KV-cache pressure are properties of inference serving; they do not arise for whole fine-tuning jobs measured in GPU-hours and checkpoints, and the paper makes no claim about serving. Multi-node parallelism does apply to training; the communication term of the slowdown model addresses it, as a swept parameter rather than a measurement. On tractability: Eq. 2 (App. D) defines the objective against which policies are scored; the paper proposes no solver, and all results come from replaying a trace.