- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BranD, a framework for predicting prefix-conditioned remaining-length distributions during LLM decoding. It combines three components: (1) a history-aware distributional predictor built on frozen target-model representations; (2) branching supervision that samples multiple continuations from selected prefixes to refine conditional length distributions and expand training coverage; and (3) adaptive continuation allocation based on conditional length variability and predictor sensitivity.
The paper provides a theoretical risk decomposition under fixed-feature CDF regression, motivating its greedy allocation strategy. Experiments on Qwen2.5-7B and Llama-3-8B across mathematical reasoning, coding, and chat tasks demonstrate improvements in MAE and CRPS over existing methods. Additional experiments show potential benefits for LLM serving under KV-cache pressure.
Critical strengths
- Well-motivated problem formulation and principled supervision strategy.
The paper identifies an important limitation of trajectory-based length prediction: a single sampled continuation cannot adequately characterize uncertainty over possible futures from a given prefix. The proposed branching supervision directly addresses this issue. The fixed-feature risk analysis also provides a meaningful theoretical motivation for allocating continuation samples based on both conditional variability and predictor sensitivity.
- Consistent empirical improvements and practical relevance.
BranD achieves the best reported MAE and CRPS across all six model–task settings in Table 1. The experiments also cover different decoding stages and include a practical serving evaluation, demonstrating the potential utility of improved length predictions for resource management.
Critical weaknesses
A central contribution is the predictor-aware continuation allocation strategy, but the paper does not provide direct comparisons against uniform sampling, random sampling, or variability-only allocation under matched budgets. Consequently, it is difficult to determine whether the proposed allocation strategy offers meaningful benefits beyond collecting additional training data.
Furthermore, Table 2 indicates that most improvements arise from the distributional predictor rather than branching supervision. For example, branching reduces CRPS from 86.38 to 86.20 on Qwen Chat and from 23.12 to 23.06 on Llama Coding. These marginal improvements raise questions about the practical significance of the branching component. Since the external baselines do not receive additional continuation samples, the overall comparisons do not fully isolate the contribution of the proposed supervision strategy.
Questions for authors
- Effectiveness of adaptive allocation.
Can the authors compare the proposed predictor-aware allocation strategy against uniform, random, and variability-only allocation, using the same distributional predictor and matched continuation budgets? Ideally, please include performance curves against both the number of continuation calls and the total generated-token or GPU-time budget. This would help establish whether the adaptive allocation mechanism provides meaningful benefits beyond additional supervision.
- Practical efficiency and robustness.
What are the training-time acquisition costs and inference-time overhead of BranD? Can the authors report serving results under less restrictive KV-cache budgets and varying request loads, while controlling for differences in scheduling policies? These results would help clarify the generality and practical significance of the reported serving improvements.