- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In long-context SFT, hard length truncation discards terminal supervision, while full-sequence supervision spends GPU hours on completion tails that may carry little gradient signal. The paper asks: under an explicit compute budget, when is stochastic tail truncation justified? TailBudget keeps each tail with probability and reweights it by , so the gradient is unbiased. A periodic gate allows truncation only when the throughput gain outweighs the added variance, after charging the cost of probing, and otherwise falls back to full supervision.
Critical strengths
- Most token/sequence-dropping work either ignores bias or ignores the cost of deciding what to drop. TailBudget keeps the estimator unbiased and charges diagnostics.
- Across nine 0.6B–7.6B models trained with LoRA for one epoch (2,048-token sequences), the authors report about 14% wall-clock savings at quality parity on math and dialogue, while on code, the gate mostly falls back to full supervision.
Critical weaknesses
- Long-CoT reasoning SFT typically uses several epochs. Repeating a smaller dataset for more epochs beats training once on more data, see for example arxiv:2602.11149. Several passes over the same data allow learning tails efficiently. The real problem is that the paper never shows training has converged. There is no validation-loss curve showing a plateau. The cosine schedule anneals to zero, so the endpoint is “converged for this schedule,” not “converged on this data.” The tail is learned in expectation, but the cost is extra gradient noise due to TailBudget. Late in training, the true gradient signal is small relative to noise, so extra variance hurts more, and the tail is important. Hence, one epoch run of the paper, leading to 14% saving, comes mostly from early training. A 2–3-epoch training would shrink saving, and it is not clear if, in terms of the final validation loss, TailBudget will be competitive against full data training to convergence.
- With a maximum length of 2048 and short-answer data (GSM8K, CodeAlpaca, SGD dialogues), the “tail” is mostly the final answer and formatting. Models learn those quickly. Long-CoT data (10k–30k tokens), where tails carry long-range reasoning, and the method’s motivation is strongest, is never tested.
- Eq.1 is well-known Horvitz-Thompson method, Lemma A.1 is a one-line conditional-variance computation, Lemma B.1 is the standard nested variance decomposition. There is no convergence result from the theoretical side: the gate minimizes one-step gradient MSE per unit budget.
Questions for authors
See weaknesses.