- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the scheduler for shared-prefix decoding with unknown output lengths. The authors provide a coverage-GSA method and achieve a 76/3 competitive ratio under specific assumptions. The authors gain a makespan improvement in TGI by making prefixes available immediately.
Critical strengths
The idea is clear and strong through prefix sharing. The authors provide theoretical proof of the competitive ratio.
Critical weaknesses
The results are too limited and strongly depend on arrivals, known prefix, and serving method.
Questions for authors
Is it possible that the method can be used on other inference software such as vLLM? Can you show related results?
Thank you for suggesting the vLLM comparison. We implemented the original staggered schedule in vLLM and compared it with native scheduling under a common KV pool and public output cap. In the eager long-first batch, the original schedule reduces mean completion time from 25.50 to 24.22 s, with lower throughput than native. Our reply to etLJ Q2 gives the full comparison.
Starting at slice 32 and handing the unfinished tail to native vLLM gives the following results on short-heavy batches.
Sharing is identified from the prompts and cache metadata. The vLLM tests vary request order within simultaneous batches. To examine arrival spacing, Figure 2(b) spreads the same 16 TGI requests over 0, 2, 5 and 20 seconds. The earlier-publication makespan gain tapers from 1.93× to 1.17×, 1.10× and 1.03× as overlap falls.