acceptodds
Under review as a conference paper at ICLR 2027

TailBudget: When Is Prefix Truncation Worth Its Cost?

Abstract

Randomly retaining an answer prefix can reduce computation without changing the expected supervised gradient. The harder question is whether this is preferable to keeping complete answers and tuning the batch size. We study this question through TailBudget, a finite-budget decision rule for frozen-gradient estimation. Starting from classical inverse-probability weighting, we charge the cost of learning whether a tail is inexpensive to omit. The resulting acceptance condition requires gross variance–cost savings to exceed the diagnostic budget fraction. A four-corner certificate handles uncertain gradient moments and execution costs; an integer allocation rule makes the comparison executable under an expected-cost budget. The guarantee applies to accepted policies, not to a cost-free fallback after diagnostics have been paid. Exact finite-vector checks and source-informed synthetic scenarios expose overhead reversals, batch-baseline confounding, and the limits of length-only decisions. Scenario scores are authored rather than checkpoint measurements or fitted estimates; previously reported CPU runs were unavailable for verification. The result is a precise account of when truncation earns its additional noise, rather than a claim of faster language-model training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
supervised fine-tuning, prefix sampling, gradient estimation, computation budgets

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cNMdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In long-context SFT, hard length truncation discards terminal supervision, while full-sequence supervision spends GPU hours on completion tails that may carry little gradient signal. The paper asks: under an explicit compute budget, when is stochastic tail truncation justified? TailBudget keeps each tail with probability qq and reweights it by 1/q1/q, so the gradient is unbiased. A periodic gate allows truncation only when the throughput gain outweighs the added variance, after charging the cost of probing, and otherwise falls back to full supervision.

Critical strengths
  • Most token/sequence-dropping work either ignores bias or ignores the cost of deciding what to drop. TailBudget keeps the estimator unbiased and charges diagnostics.
  • Across nine 0.6B–7.6B models trained with LoRA for one epoch (2,048-token sequences), the authors report about 14% wall-clock savings at quality parity on math and dialogue, while on code, the gate mostly falls back to full supervision.
Critical weaknesses
  • Long-CoT reasoning SFT typically uses several epochs. Repeating a smaller dataset for more epochs beats training once on more data, see for example arxiv:2602.11149. Several passes over the same data allow learning tails efficiently. The real problem is that the paper never shows training has converged. There is no validation-loss curve showing a plateau. The cosine schedule anneals to zero, so the endpoint is “converged for this schedule,” not “converged on this data.” The tail is learned in expectation, but the cost is extra gradient noise due to TailBudget. Late in training, the true gradient signal is small relative to noise, so extra variance hurts more, and the tail is important. Hence, one epoch run of the paper, leading to 14% saving, comes mostly from early training. A 2–3-epoch training would shrink saving, and it is not clear if, in terms of the final validation loss, TailBudget will be competitive against full data training to convergence.
  • With a maximum length of 2048 and short-answer data (GSM8K, CodeAlpaca, SGD dialogues), the “tail” is mostly the final answer and formatting. Models learn those quickly. Long-CoT data (10k–30k tokens), where tails carry long-range reasoning, and the method’s motivation is strongest, is never tested.
  • Eq.1 is well-known Horvitz-Thompson method, Lemma A.1 is a one-line conditional-variance computation, Lemma B.1 is the standard nested variance decomposition. There is no convergence result from the theoretical side: the gate minimizes one-step gradient MSE per unit budget.
Questions for authors

See weaknesses.

Reviewer_89aUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author observed that you can basically use a Horvitz-Thompson estimation procedure to reduce how much compute you spend during SFT by truncating the completions while preserving unbiasedness.

Critical strengths

It's a principled method that provides a pretty good compute efficiency boost. And I think the empirical evaluation is quite strong, especially because it covers many data sets and models. I like the idea of cleverly trading off some small amount of gradient variance in exchange for wall-clock savings.

Critical weaknesses

It's not clear how much this is a real practical use case because SFT is usually a fairly not a huge use of compute throughout the post-training stack.

Questions for authors

I guess the interaction between how well you estimate the true gradient vs the optimization dynamics is still a relatively open question?

Reviewer_audMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TailBudget randomly retains completion tails and reweights their losses to preserve expected pre-clipping gradients. A diagnostic gate compares added gradient variance against compute savings, accounting for probing overhead and discrete sample counts. The paper reports approximately 14% training-time savings across several model families and tasks.

Critical strengths
  1. The inverse-probability estimator is sound under the stated causal-execution and normalization assumptions. It provides a principled alternative to permanently discarding terminal supervision.

  2. Accounting for diagnostic costs and aggregate tail-gradient energy addresses practical limitations of simpler truncation heuristics.

Critical weaknesses
  1. The experimental configuration contains a major inconsistency. Appendix E.1 lists deepseek-ai/DeepSeek-Math-V2 as a 3.2B dense model, while the official repository identifies a 685B model. Its reported memory and runtime therefore need reconciliation before the scaling results can be trusted. Figure 3(a) and Table 7 also show substantially different throughput values without explaining the measurement scope.

  2. The statistical certification is incomplete. Algorithm 1 labels an expression containing the variance lower bound (V_-) as an MSE upper bound, which is not justified. The construction of jointly valid bounds across candidate selection and repeated probing is also unspecified. The variance identities alone do not establish the claimed finite-budget certification.

Questions for authors
  1. Which exact checkpoint and revision were used for DeepSeek-Math-V2? Please reconcile its parameter count, memory footprint, and training duration.

  2. How are the joint confidence bounds constructed, and why does the claimed MSE upper bound use (V_-)?

  3. Please reconcile Figure 3(a) with Table 7 and provide the probing implementation, including how prefix and full gradients are extracted and timed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.