acceptodds
Under review as a conference paper at ICLR 2027

Progressive Conformal Prediction for Remaining-Length Estimation in Large Language Models

Abstract

Autoregressive decoding creates an “unknown horizon”: although serving engines allocate KV pages on demand, admission and preemption decisions are made before a request's future demand is known. We introduce Progressive Mondrian Conformal Prediction, a training-free method that converts stopping probability, predictive entropy, and decoding progress into online upper bounds on remaining length. We distinguish an empirical pooled estimator from prompt-exchangeable fixed-step and two-fold request-level constructions; the latter provide finite-sample step or whole-request coverage without assuming independence among states of one request. Across models from 3.8B to 70B, the pooled method reduces mean width by 41–49% relative to global calibration at the same nominal level. The request construction records 4.8% empirical request violations at a 10% budget. A vLLM prototype confirms that the bound can drive admission metadata, while the statistical theorem is explicitly separated from empirical multi-request OOM and preemption outcomes. The bound adds 0.22% per-state overhead.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, uncertainty quantification, LLM serving, generation length prediction, autoregressive decoding, dynamic inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_S4DBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a progressive conformal prediction framework for estimating the remaining generation length of LLM requests. The method combines a training-free geometric estimator based on stopping probability with Mondrian conformal calibration using predictive entropy and decoding progress. It further introduces fixed-step and request-level calibration schemes to provide finite-sample coverage guarantees.

Critical strengths
  1. The paper addresses an important problem in LLM serving: estimating the remaining generation length to support memory-aware admission control and resource management.

  2. The proposed approach leverages stopping probability, predictive entropy, and decoding progress without requiring an auxiliary prediction model, introducing only limited inference overhead.

Critical weaknesses
  1. The proposed approach mainly combines existing techniques, including geometric length estimation, Mondrian conformal prediction, and trajectory-level calibration. Recent studies have also begun exploring the use of conformal prediction for LLM output-length estimation, with MAPS (ICML 2026) being one example. The authors are encouraged to cite and discuss these related works to better contextualize their contributions.

  2. Mismatch Between Theoretical Guarantees and Main Experimental Results. The strongest efficiency results are obtained using FULL-STEP, which lacks formal finite-sample coverage guarantees. The theoretically guaranteed FULL-REQUEST variant produces substantially wider bounds, and its advantages over comparable request-level conformal baselines remain unclear.

  3. Limited Real-World Serving Validation. The system-level results rely primarily on simulation, without a direct comparison against unmodified vLLM. Important serving metrics, including throughput, TTFT, TPOT, and tail latency, are not reported, making the practical benefits difficult to assess.

  4. The primary experiments use a relatively small number of calibration and test prompts. The method is not sufficiently evaluated on long reasoning traces, agentic workloads, or substantial workload distribution shifts, where the stopping behavior and calibration effectiveness may differ considerably.

Questions for authors
  1. Since FULL-STEP lacks formal coverage guarantees, how much of the reported efficiency improvement remains when using FULL-REQUEST? Could the authors compare different methods under the same request-level coverage requirements?

  2. Could the authors apply the proposed Mondrian and trajectory-level calibration schemes to existing predictors, such as AUX-MLP and HAZARD-CP, and provide comparisons at matched empirical coverage levels?

  3. Could the authors provide end-to-end experiments against unmodified vLLM under realistic concurrent workloads, reporting throughput, TTFT, TPOT, tail latency, and OOM rates to substantiate the claimed serving benefits?

Reviewer_D4nYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes predicting upper bounds for LLM remaining-generation length. It uses stopping probability as a lightweight anchor and calibrates the bound with stopping probability. It further provides fixed-step and request-level constructions to handle dependence across decoding steps.

Critical strengths
  1. Implementation is lightweight. The method requires no auxiliary prediction model and reports only about 0.22% per-step overhead, making it practical as a serving-side signal.
Critical weaknesses
  1. The system motivation is not convincingly validated. The method may reduce preemption by reserving future KV headroom, but this can also reduce batchsize or concurrency. The paper reports preemption and memory waste, but no valid throughput/goodput or latency result, so it is unclear whether the system actually improves serving performance.

  2. The evaluated generation length is too limited. The main systems experiment uses a 512-token maximum output length, while long-reasoning and agent workloads may generate more tokens.

Questions for authors
  1. Can the authors compare against prediction-free admission policies using recomputation, swapping, or migration, and report throughput/goodput, TTFT, TPOT latency? This is important to establish that reducing preemption actually improves serving performance.

  2. Can the authors evaluate long-generation workloads, e.g. several thousand to tens of thousands of output tokens, where remaining-length uncertainty is more significant?

Reviewer_sjT2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this work, authors have proposed progressive remianing-length estimation and mondarial conformal prediction. The method in this work is training-free and can convert stopping probability, predictive entropy, and decoding progress into online upper bounds of the remaining token length. The experiments are performed up to 70B LLMs, and authors claims great gain with minor overhead.

Critical strengths

None

Critical weaknesses
  1. In figure 1, it shows token pruning at step-t, while the method is progressive remaining estimation flow. The context does not explain the token pruning at all. Moreover, the writing quality of the overall article needs significant improvement. For example, the description of proposed method in abstract and introduction is not consistent. The

  2. The LLM models used in the experiments (e.g., Phi3.5, Qwen2.5 and Llama3.3) are outdated, those models are released two years ago, while it is unclear the proposed method still achieve promising results on SOTA models.

  3. The article use the multiple baseline for comparison, for example the CUMUL-PROB, however, their setups and references are not properly given. Moreover, it lacks the comparison with SOTA competing design to demonstrate the effectiveness and advantage of proposed method.

  4. The organization of the manuscript should be improved. For example, symbols used in figure 3 (V and W) are not explained in figure caption which lowers the readbility of the paper.

  5. It is unclear the end-to-end performance gain of the LLM serving integrated with remaining length prediction. Nowadays, KV offloading to host memory is supported by vLLM, OOM problem will not appear. Moreover, OOM appears normally appear on long-context inference and batching case, the chosen of batch size and benchmark should include large batch and long-context/agentic benchmarks.

Questions for authors

check the weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.