- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
BridgeSTOP connects streaming tokenization with prefill through asynchronous workers, a token pool, and speculative replenishment. Speculative tokens are verified and mismatched suffixes recomputed before decoding. Experiments measure serving capacity across five workloads on Qwen3-8B.
Critical strengths
-
The paper evaluates end-to-end serving rather than tokenizer speed alone. Table 1 includes useful comparisons within a fixed tokenizer configuration.
-
The verification barrier prevents decoding from unchecked speculative tokens. Appendix B also appropriately limits the CPU scheduling argument rather than extending it directly to GPU execution.
Critical weaknesses
-
I am not yet convinced that the Stream2LLM [S26] comparison isolates the contribution. Appendix C.2 implements fixed-byte-chunk overlap, whereas [S26] also includes streaming-aware scheduling and preemption. Please clarify which mechanisms are retained and compare equally asynchronous pipelines with the same tokenizer and prefill budget. The positioning should distinguish token delivery from chunked-prefill scheduling [SS24], memory-aware admission [W25], and KV-growth-induced congestion and eviction [CG26]. Faster token delivery alone does not establish stable end-to-end throughput.
-
The evaluation needs clearer correctness and component checks. The retrained zip2zip model and vocabulary adapters complicate cross-tokenizer comparisons: recovering the same text does not establish equal model behavior. Table 3 shows 71.5 versus 70.0 QPS with and without speculation on MMLU, where only 5% of requests trigger it; its benefit elsewhere and run-to-run variability remain unclear.
The problem is useful, but the additional benefit over a matched streaming baseline is not yet clear. Clarifying this comparison and checking model behavior could change my assessment.
Questions for authors
-
Which [S26] mechanisms are implemented, and can both systems use the same asynchronous tokenizer? Relative to [SS24, W25, CG26], please distinguish token-pool control from scheduling and admission, and clarify how KV-memory pressure, eviction, and recomputation enter the capacity measurements.
-
Have final prompt-token equality and generation quality been checked against the corresponding non-speculative pipeline? Please provide existing no-speculation results beyond MMLU and repeated-run variability.
[S26] Bachkaniwala, R., Luo, C., So, R., Mahajan, D., and Rong, K. (2026). Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT). arXiv:2604.16395. https://arxiv.org/abs/2604.16395.
[SS24] Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B., Tumanov, A., and Ramjee, R. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation, 117–134. https://www.usenix.org/conference/osdi24/presentation/agrawal.
[W25] Ao, R., Luo, G., Simchi-Levi, D., and Wang, X. (2025). Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints. arXiv:2504.11320, revised June 2026. https://arxiv.org/abs/2504.11320.
[CG26] Ao, R., Dong, J., Luo, G., and Simchi-Levi, D. (2026). Service-Induced Congestion in Memory-Constrained LLM Serving. arXiv:2606.15555. https://arxiv.org/abs/2606.15555.
5 comments
Response to Weakness 2:
Response to W2:
1. Cross-tokenizer comparisons: To evaluate cross-tokenizer effects, we report token counts, prefill and decode times, downstream quality, and serving capacity across tokenizers. 4-shot GSM8K results are presented in Table D below. Full results and analysis including 5-shot MMLU are provided in Appendix D.7 (page 26) of the updated PDF.
1.1. Token count correlates with times but not capacity: Different tokenizers produce different token counts, thus affecting prefill and decode times. On GSM8K, token count correlates strongly with prefill and decode times (all Pearson Corr > 0.96). But, token count correlates weakly with capacity (Pearson Corr 0.099, Spearman Corr 0.051). Hence in our experiments, token count only has limited effects on our assessment of serving capacity values and rankings, thus not affecting advantages of BridgeSTOP in capacity.
1.2. Retrained zip2zip model: Model weights can affect output quality and decode workload. On GSM8K, zip2zip achieves 83.4% accuracy versus 89.5% for native Qwen3 BPE, whose decrease is acceptable. zip2zip's average decode time is 30.2 ms over 31.5 ms from BPE, thanks to fewer token outputs. Given zip2zip’s shorter decode time, IncreBPE + BridgeSTOP still achieves higher serving capacity than the zip2zip baselines, supporting our reported advantage.
1.3. Model behavior and downstream quality: Different token IDs can affect model behavior, i.e. downstream quality. Some tokenizers even trade quality for efficiency like zip2zip. But, the quality variations 83.1%-89.5% remain acceptable for our evaluation and thus do not change our assessment of serving capacity and latency.
Table D: Cross-tokenizer comparisons of token counts, latency, capacity, and output quality on 4-shot GSM8K. Prefill and decode times use batch size 1.
| Metric | LoPT | zip2zip | MAGNET | IncreBPE | SPEAK |
|---|---|---|---|---|---|
| Total tokens | 126665 | 119625 | 122422 | 126665 | 120294 |
| Relative to BPE | 100.0% | 94.4% | 96.7% | 100.0% | 95.0% |
| Avg. prefill (ms) | 31.5 | 30.2 | 31.1 | 31.5 | 30.4 |
| Avg. decode (ms) | 6658.3 | 6533.8 | 6606.4 | 6658.3 | 6582.4 |
| Capacity (QPS) #rank | 8.5 #4 | 9.5 #3 | 7.5 #5 | 15.0 #1 | 14.0 #2 |
| Output quality (%) | 89.5 | 83.4 | 83.1 | 89.5 | 86.9 |
2. Vocabulary adapters: MAGNET and SPEAK use vocabulary adapters with the unchanged Qwen3 model. Although they do not exactly reproduce Qwen3 BPE, only OOV tokens require adaptation, which is reported as 0.1% tokens in Appendix C.2. Hence, such a negligible fraction is unlikely to alter tokenization quality or efficiency, thus imposing limited effects on serving capacity.
3. Speculated requests are fewer but longer: Across five workloads, 5.2–12.0% of requests trigger speculation, but their mean input lengths are 1.16×–1.82× over the dataset averages, indicating these requests are longer and engage more tokens.
Full statistics and analysis are provided in Appendix D.4 (page 24) of the updated PDF.
4. Speculation offers larger benefits under memory constraints: To examine its benefit elsewhere, we evaluate speculation on MMLU under tighter memory constraints. Longest improves capacity over Waiting by 10.9% (23->25.5) under the 20GB memory constraint, higher than the original 2.1% (70->71.5) under the 32GB memory constraint. The speculated request ratio increases from 5.2% to 12.0%. These results demonstrate larger serving benefits under memory constraints.
Full results and analysis are provided in Appendix D.5 (page 25) of the updated PDF.
5. Run-to-run variability: We performed 3 repeated runs with different seeds on MMLU. Waiting consistently satisfied the SLOs at 70.0 QPS and failed at 70.5 QPS, while Longest satisfied them at 71.5 QPS and failed at 72.0 QPS. These results confirm the reported capacity values.
Responses to Question 1-2
Response to Q1:
1. Retained Stream2LLM mechanisms: See response to W1, point 1.
2. Asynchronous comparison: See response to W1, point 2.
3. Distinguish from others: See response to W1, point 3.
4. KV-memory pressure, eviction, and recomputation: Capacity is measured end to end under the configured GPU-memory budget and SLO constraints. Any waiting, eviction, and recomputation caused by KV-memory pressure are reflected in measured serving latencies and therefore in the maximum sustainable request rate. Verification and rollback recomputation costs are also included.
Response to Q2:
1. Final prompt-token equality and generation quality: Speculation does not change the final token outputs. Speculative tokens are verified against exact tokenization outputs, and mismatches trigger rollback and re-prefill from the first mismatch onward. This preserves the exact prompt tokens and corresponding KV states. Generation therefore uses the same verified prompt-token input, model, and decoding settings as the non-speculative pipeline. Hence, we do not need this check.
2. No-speculation results and run-to-run variability: Table E below compares Waiting, i.e. the non-speculative pipeline, with our Longest-match speculation across all five workloads. Longest consistently outperforms Waiting. All reported capacity points satisfied the SLOs in 3 repeated runs, while the next tested request rate for each configuration violated the SLOs in all 3 runs.
Table E: Serving capacity of IncreBPE + BridgeSTOP with longest-match speculation and waiting for exact tokens across five workloads. Relative gain = (Longest / Waiting − 1) × 100%.
| Capacity (QPS) | ShareGPT4 | MMLU | GSM8K | LongBench | CLongEval |
|---|---|---|---|---|---|
| Longest | 1.25 | 71.5 | 19.5 | 0.1525 | 0.2025 |
| Waiting | 1.20 | 70.0 | 18.5 | 0.1375 | 0.1925 |
| Relative gain | 4.17% | 2.14% | 5.41% | 10.91% | 5.19% |
Response to Weakness 1
We thank the reviewer for the helpful comments. We will add additional details and experiments discussed during the rebuttal to the revised paper.
Response to W1:
1. Retained Stream2LLM mechanisms: We reproduce Stream2LLM on vLLM, retaining its original mechanisms for incremental prefill, streaming-aware scheduling, and preemptive KV-cache management under FCFS. We adapt our static inputs to Stream2LLM’s append-mode interface, so that Stream2LLM's original mechanisms can be reproduced unchanged. Appendix C.2 (page 19) explains details of the workload differences and how this input adaptation enables reproduction.
2. Asynchronous comparison: BAPT improves tokenization efficiency while changing how upstream token outputs are released, requiring compatible downstream handling through BridgeSTOP’s token pool. Sarathi and Stream2LLM lack this interface and cannot directly consume BAPT’s token outputs, although Stream2LLM supports streaming. Hence, BAPT’s benefits cannot translate into their serving gains, so we only ablate BAPT from BridgeSTOP.
We set the prefill token budget to 512 for Stream2LLM, Sarathi, and BridgeSTOP to ensure a fair comparison.
3. Distinguish from SS24 W25 CG26: SS24 improves prefill–decode coordination through chunked prefill and batching, while BridgeSTOP improves tokenization–prefill efficiency transfer through token-pool control. Their target stages, objectives, and mechanisms differ. W25, CG26, and BridgeSTOP all optimize scheduling. However, W25 and CG26 address memory pressure caused by KV-cache growth during decoding, while BridgeSTOP addresses upstream efficiency translation to downstream prefill. Thus, despite optimizing scheduling, they target different serving stages and solve different core problems.
4. That tokenization cannot guarantee throughput is what we address: Faster tokenization alone cannot guarantee stable end-to-end throughput because downstream prefill may not effectively translate such upstream efficiency. This is precisely the main problem our paper aim to address. For example, IncreBPE (ICML 2026 Spotlight) proposes an efficient streaming tokenization method, but its evaluation focuses on tokenization, without end-to-end serving evaluation because it lacks effective bridging to downstream prefill. These upstream benefits require compatible downstream mechanisms to translate into serving gains. We therefore propose BridgeSTOP to preserve and propagate upstream tokenization efficiency.