- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes StaleSpec, which uses self-speculative verification to run the exact suffix refresh with proposal generation in parallel, thereby improving the system throughput for agentic LLM serving.
Critical strengths
- The overall paper structure is relatively clear, with reasonable presentation.
- The idea of self-speculative verification is novel and somewhat interesting.
- The evaluation results are good, with somewhat proper analysis.
Critical weaknesses
-
The paper writing needs significant improvement, with incoherent sentences and out-of-context terms throughout the paper. While AI-assisted tools are allowed for paper writing, the readability of this paper is very low for scientific readers. The fundamental purpose of scientific writing is not merely to convert the data into sentences and paragraphs, but to have the audience perceive what the authors have in mind.
-
Figure 1 is a bit too complex to understand. Corresponding algorithms or separate workflows would significantly improve the readability.
-
The evaluation setups are unclear to me. The authors mention 1:1 PD pools, but do not explicitly mention how the prefill and decode instances are served. Is it single-GPU or multi-GPU? Similarly, the subsequent GLM-4.7-Flash and Qwe3-235B settings are also not directly mentioned. Later in Table 4 and Figure 2, there is also Llama3.1-8B with unknown settings. Overall, the evaluation section is very unorganized and hard to follow the purpose of each paragraph.
-
There are also many small typos throughout the paper. For instance, in line 042, the authors use 'Large language model', but in line 029, 'LLM' is used. There is no labelling for all the equations.
Questions for authors
Please see the weaknesses.
2 comments
Response to Reviewer bB2o (2/2)
W1–W4 are addressed through concrete manuscript revisions, not new correctness claims. The revised text provides a shorter mechanism explanation, verified protocol/placement disclosures, and defined metrics/equation labels. The Figure 1 simplification is planned for a subsequent revision.
| Requested fix | Revision location | What the revision documents |
|---|---|---|
| Clear scientific story | Abstract; Secs. 1–4 | No unexplained switches among proposal, commit, and throughput |
| Workflow explanation | Algorithm 1; method text | Proposal-only stale view and fresh-verification boundary made explicit in prose |
| Reproducible setup | Sec. 4; Appendices B.5, D.5, E.1/E.5 | Verified runtime, hardware, protocol, and repeats; no invented placements |
| Terminology and typesetting | Equations; tables; captions | Correct units, labels, names, and all reference targets |
Revision incorporated — Abstract; Secs. 1–4; Algorithm 1; Appendices A.1, B.5, D.2/D.4/D.5, and E.1/E.3/E.5. The 10-page main paper presents the concise mechanism and key numerical summaries, with new controls and protocol-specific tables in the appendices. The proposed Figure 1 redraw remains planned, not incorporated.
Response to Reviewer bB2o (1/2)
W1. Scientific writing and evaluation structure
Response. We agree that the manuscript's flow obscures its main invariant. We have placed four concise statements at the start of the revised explanation. An edit invalidates the suffix. Stale KV is proposal-only. Fresh verification determines every committed token. A cost gate bypasses unprofitable turns. The revised evaluation uses the following order: exactness, incremental latency, end-to-end serving, gate/generalization, and RoR capacity.
Revision incorporated — Abstract; Secs. 1–4; Appendix D.2 and E.3–E.5. The 10-page body introduces the four-step invariant and separates first proposal, fresh commitment, measured serving, shifted gate, and RoR capacity. Detailed resource, gate-transfer, and RoR evidence is in the cited appendices; the abstract explicitly limits generalization.
W2. Figure 1 is difficult to follow
Response. We agree that the original Figure 1 is visually dense. We will simplify it in a subsequent revision by separating the proposal-only stale KV view from exact fresh verification and by making the first-commitment boundary explicit. We will also move low-level page mappings and RoR details out of the main illustration.
Planned figure improvement — Fig. 1. We will revise the schematic for readability.
W3. 1:1 PD placement and missing per-model hardware setup
Response. The reviewer is right: 1:1 is an engine ratio, not a GPU count. It means one prefill/refresh engine and one decode/proposal engine. We distinguish engine ratios from known per-pool GPU allocations and parallelism in the revised protocol. The original 110K Qwen3-30B-A3B experiment has two B200 GPUs total,
G=64, andC∈{1,4,6,8}; its PD-style timing is composed rather than a live cross-process connector.G, concurrency, request/turn counts, repeatsInterpretation. Qwen3-235B-A22B is labeled TP4 in the original manuscript, and the revision states only verified hardware and parallelism information. It does not invent unspecified accelerator placements from missing launch manifests.
Revision incorporated — Sec. 4; Appendices B.5, D.5, E.1, and E.5. The 1:1 notation is defined as an engine ratio, not a GPU count. Appendix E.1 states verified configurations, Appendix D.5 gives replay conditions, Appendix B.5 describes composed PD timing, and Appendix E.5 specifies two B200s total for the 110K C=8 repeat. Undocumented placement/EP flags are not inferred.
W4. Typos, equation labels, and metric consistency
Response. We agree. We have standardized model names and terminology, labeled the core equations, and removed the ambiguous use of "gain." In particular, Table 2 reports latency reduction, not a speedup factor; these are mathematically different.
1 − T_SS/T_B1T_B1/T_SSR_SS/R_B1 − 1Revision incorporated — Abstract; Secs. 1–4; Appendix D.2. The acronym LLM is introduced at first use; latency reduction and speedup factor are distinguished in the experiments; acceptance, residual, gate, and commitment-timing equations are labeled in Secs. 2–3. Appendix D.2 reports user-visible committed-TTFT rather than proposal availability.