- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a progressive conformal prediction framework for estimating the remaining generation length of LLM requests. The method combines a training-free geometric estimator based on stopping probability with Mondrian conformal calibration using predictive entropy and decoding progress. It further introduces fixed-step and request-level calibration schemes to provide finite-sample coverage guarantees.
Critical strengths
-
The paper addresses an important problem in LLM serving: estimating the remaining generation length to support memory-aware admission control and resource management.
-
The proposed approach leverages stopping probability, predictive entropy, and decoding progress without requiring an auxiliary prediction model, introducing only limited inference overhead.
Critical weaknesses
-
The proposed approach mainly combines existing techniques, including geometric length estimation, Mondrian conformal prediction, and trajectory-level calibration. Recent studies have also begun exploring the use of conformal prediction for LLM output-length estimation, with MAPS (ICML 2026) being one example. The authors are encouraged to cite and discuss these related works to better contextualize their contributions.
-
Mismatch Between Theoretical Guarantees and Main Experimental Results. The strongest efficiency results are obtained using FULL-STEP, which lacks formal finite-sample coverage guarantees. The theoretically guaranteed FULL-REQUEST variant produces substantially wider bounds, and its advantages over comparable request-level conformal baselines remain unclear.
-
Limited Real-World Serving Validation. The system-level results rely primarily on simulation, without a direct comparison against unmodified vLLM. Important serving metrics, including throughput, TTFT, TPOT, and tail latency, are not reported, making the practical benefits difficult to assess.
-
The primary experiments use a relatively small number of calibration and test prompts. The method is not sufficiently evaluated on long reasoning traces, agentic workloads, or substantial workload distribution shifts, where the stopping behavior and calibration effectiveness may differ considerably.
Questions for authors
-
Since FULL-STEP lacks formal coverage guarantees, how much of the reported efficiency improvement remains when using FULL-REQUEST? Could the authors compare different methods under the same request-level coverage requirements?
-
Could the authors apply the proposed Mondrian and trajectory-level calibration schemes to existing predictors, such as AUX-MLP and HAZARD-CP, and provide comparisons at matched empirical coverage levels?
-
Could the authors provide end-to-end experiments against unmodified vLLM under realistic concurrent workloads, reporting throughput, TTFT, TPOT, tail latency, and OOM rates to substantiate the claimed serving benefits?