acceptodds
Under review as a conference paper at ICLR 2027

Dynamic LLM Decoding Length Prediction: Learning from Branching Futures

Abstract

Length prediction during large language model (LLM) decoding provides an important signal for allocating limited computational resources across concurrent requests. Existing decode-time predictors typically learn from sampled trajectories, each of which reveals only one realized future at every visited prefix. However, a prefix admits multiple possible futures, and its remaining-length distribution evolves as more response content is observed. The challenge is therefore to obtain informative conditional supervision without exhaustively exploring this branching future space. In this paper, we propose *BRANching-supervised Distribution prediction* (BranD), an approach for learning prefix-conditioned remaining-length distributions under a limited continuation budget. BranD combines current and historical target-model representations to update its predictions as decoding proceeds. For training, it samples independent continuations from selected prefixes to refine conditional distribution targets, while reusing their paths to expand prefix coverage. An adaptive allocation strategy combines estimated conditional variability with predictor sensitivity to determine how many continuations to sample at each prefix. We provide a fixed-feature risk analysis that connects conditional sampling error to shared prediction error and justifies the allocation strategy. Experiments across multiple models and tasks validate the effectiveness of our proposed BranD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM decoding length prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Uyd5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BranD, a framework for predicting prefix-conditioned remaining-length distributions during LLM decoding. It combines three components: (1) a history-aware distributional predictor built on frozen target-model representations; (2) branching supervision that samples multiple continuations from selected prefixes to refine conditional length distributions and expand training coverage; and (3) adaptive continuation allocation based on conditional length variability and predictor sensitivity.

The paper provides a theoretical risk decomposition under fixed-feature CDF regression, motivating its greedy allocation strategy. Experiments on Qwen2.5-7B and Llama-3-8B across mathematical reasoning, coding, and chat tasks demonstrate improvements in MAE and CRPS over existing methods. Additional experiments show potential benefits for LLM serving under KV-cache pressure.

Critical strengths
  1. Well-motivated problem formulation and principled supervision strategy.

The paper identifies an important limitation of trajectory-based length prediction: a single sampled continuation cannot adequately characterize uncertainty over possible futures from a given prefix. The proposed branching supervision directly addresses this issue. The fixed-feature risk analysis also provides a meaningful theoretical motivation for allocating continuation samples based on both conditional variability and predictor sensitivity.

  1. Consistent empirical improvements and practical relevance.

BranD achieves the best reported MAE and CRPS across all six model–task settings in Table 1. The experiments also cover different decoding stages and include a practical serving evaluation, demonstrating the potential utility of improved length predictions for resource management.

Critical weaknesses

A central contribution is the predictor-aware continuation allocation strategy, but the paper does not provide direct comparisons against uniform sampling, random sampling, or variability-only allocation under matched budgets. Consequently, it is difficult to determine whether the proposed allocation strategy offers meaningful benefits beyond collecting additional training data.

Furthermore, Table 2 indicates that most improvements arise from the distributional predictor rather than branching supervision. For example, branching reduces CRPS from 86.38 to 86.20 on Qwen Chat and from 23.12 to 23.06 on Llama Coding. These marginal improvements raise questions about the practical significance of the branching component. Since the external baselines do not receive additional continuation samples, the overall comparisons do not fully isolate the contribution of the proposed supervision strategy.

Questions for authors
  1. Effectiveness of adaptive allocation.

Can the authors compare the proposed predictor-aware allocation strategy against uniform, random, and variability-only allocation, using the same distributional predictor and matched continuation budgets? Ideally, please include performance curves against both the number of continuation calls and the total generated-token or GPU-time budget. This would help establish whether the adaptive allocation mechanism provides meaningful benefits beyond additional supervision.

  1. Practical efficiency and robustness.

What are the training-time acquisition costs and inference-time overhead of BranD? Can the authors report serving results under less restrictive KV-cache budgets and varying request loads, while controlling for differences in scheduling policies? These results would help clarify the generality and practical significance of the reported serving improvements.

Reviewer_r1zkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates "remaining-length prediction" for autoregressive LLMs during the generation process. The authors note that a single prefix can lead to future sequences of varying lengths, whereas a standard generation trajectory provides only one observed outcome for each prefix encountered. To address this, the proposed method, BranD, samples multiple continuations from a given prefix to learn a conditional length distribution, while reusing the prefixes from these newly generated paths as additional training data. The approach also allocates a sampling budget based on the degree of length variation and the sensitivity of the predictor. Experiments cover two 7B/8B models across three types of tasks and demonstrate the method's benefits in serving scenarios where KV cache resources are constrained.

Critical strengths
  1. The problem is well-defined, and the method is grounded in clear theoretical motivation. The paper defines the objective as a prefix-conditioned remaining-length distribution and uses the Continuous Ranked Probability Score (CRPS) to quantify prediction error. By collecting multiple conditional samples from the same prefix, the "branching supervision" strategy directly overcomes the limitations inherent in observing only a single future trajectory.
  2. The experiment is comprehensive. BranD achieves the lowest MAE and CRPS across all six model–task configurations; the serving experiments further demonstrate the potential practical value of length prediction.
Critical weaknesses
  1. The gain from "branching supervision" appears to be small. In Table 2, the improvement in CRPS after adding branching on top of a distribution predictor is generally limited (e.g., for Qwen Math, it decreases from 26.60 to 24.99; for Llama Math, from 13.50 to 13.48). Consequently, the experiments do not yet fully demonstrate the practical contribution of this core component.
  2. The serving results show promise, but reproducibility and the scope of applicability remain limited. The paper does not sufficiently clarify how predicted lengths determine KV cache admission and request priority. Furthermore, the serving evaluation is restricted to a single model, one type of chat workload, and a relatively tight KV cache budget. Since the controller primarily utilizes the median of the predicted distribution, it remains unclear whether a lower Continuous Ranked Probability Score (CRPS) yields system-level gains that are truly independent of the accuracy of individual point-estimate length predictions.
Questions for authors
  1. Could you provide more details about the serving controller? e.g., the conversion of KV cache budgets, priority rules, and the handling of cases where the length is underestimated?
  2. In scenarios where the Mean Absolute Error (MAE) of the median prediction is similar, can a more accurate length distribution still lead to improvements in serving performance?
Reviewer_fYBYOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BranD, which models remaining-length prediction as a prefix-conditioned distribution and designs (1) a dynamic distribution predictor that integrates current and historical target-model states, (2) branching-future supervision based on multiple independent continuations, which constructs empirical conditional CDFs and reuses intermediate prefixes along continuation paths to expand training coverage, and (3) a pilot-estimated, predictor-aware continuation-budget allocation rule that combines conditional dispersion with predictor sensitivity.

Critical strengths
  1. The paper advances decode-time length prediction toward a prefix-conditioned remaining-length distribution and explicitly addresses branching future spaces under a limited continuation budget. The authors further evaluate BranD in LLM serving under KV pressure, reporting P99 latency, throughput, TPOT, and KV preemptions.

  2. The paper provides broad experimental coverage with consistent main results. Across six model–task settings (Qwen2.5-7B and Llama-3-8B on Math, Coding, and Chat), BranD achieves the lowest MAE and CRPS in all settings; its average CRPS is reduced by about 27.0% and 28.6% on Qwen and Llama, respectively, relative to the strongest baseline (OUTLETS).

Critical weaknesses
  1. The Introduction states that ProD ignores how the prefix-conditioned distribution evolves with the response and does not address budget allocation over branching futures, whereas Related Work and Experiments acknowledge that ProD learns prompt-conditioned length distributions from repeated generations and updates them via survival conditioning during decoding. The authors do not clarify the substantive difference in conditioning variables and supervision construction, nor BranD’s core novelty boundary; the paper can thus be summarized as “a prefix-level version of ProD plus a greedy allocation rule.” Moreover, BranD uses additional branching continuations while baselines use only ordinary trajectories; the text does not clarify whether comparisons use the same continuation budget, backbone, labels, and compute, and the ablations are insufficient to separate the three components’ contributions.

  2. Theorem 1 and Lemma 1 cover only fixed features, independent direct futures, and fixed planning features, but not the nonlinear refitting and path coverage actually used by BranD. The authors do not explain this gap, nor adequately validate whether the empirical gains arise from the proved sampling-value mechanism rather than from more labels, path coverage, or stronger fitting. Pilot stability, allocation-variant comparisons, and separate ablations of path coverage versus conditional-target refinement are insufficient, and the scope of the theory is unclear, weakening soundness.

Questions for authors
  1. The Introduction states that ProD ignores how F_v evolves with the response and does not address budget allocation over branching futures, whereas Related Work and Experiments acknowledge that ProD learns prompt-conditioned length distributions from repeated generations and updates them via survival conditioning during decoding. Please clarify the substantive difference between the two in terms of conditioning variables and supervision construction, state the core novelty boundary of BranD, and explain how BranD avoids being summarized as “a prefix-level version of ProD plus a greedy allocation rule.”
  2. Theorem 1 and Lemma 1 cover only fixed features, independent direct futures, and fixed planning features, but not the nonlinear refitting and path coverage actually used by BranD. Please explain why this gap exists, whether the empirical gains can be attributed to the proved sampling-value mechanism, and to what extent the theoretical results can be validated and what their scope of applicability is.
  3. Please clarify whether BranD and the baselines are compared under the same continuation budget, the same predictor backbone, and the same number of labels/training compute; whether the current ablations are sufficient to separate the contributions of prefix conditioning, branching-future supervision, and predictor-aware allocation; and whether the experimental evidence genuinely supports the paper’s core claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.