- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Xtra-MTP, focusing on how to efficiently implement MTP-style speculative decoding for hybrid language models. Xtra-MTP first augments the native MTP chain with a one token alternative to mitigate the early mismatch problem in speculative decoding, and, consequently improve request-p95 TPOT. It further extends its idea to hybrid architecture, it intergrates a deferred reccurent states materilaztion and a specialized parrellel solver for spine-plus-leaf recurrent states calculation. Experiments on three Qwen models shows improvement towards both throughput and p95-TPOT.
Critical strengths
- The paper addresses a critical bottleneck in hybrid model serving. Hybrid models face different challenges like branch-dependent recurrent states, expensive state duplication and so on.
- The paper systematically implements the proposed method into serving framework. It provides integration with SGlang and also CUDA graph compatibility.
- The paper provides extensive experiments for the effectiveness of the proposed optimizations
Critical weaknesses
- The main contribution is not sufficiently focused. The paper introduces three different components, C1 one token branching, C2 deferred materializaton for recurrent states, C3 parellel computation for branching. C3 aims to mitigates the time cost brought by C1, while C2 addresses a different bottleneck. A clearer positioning of the primary contribution would strengthen the paper.
- Component 1 provides relatively mariginal improvement for end-to-end serving, it is only selected once under 18 workloads with latency constraints in appendix B.3.
- Although the paper provides detailed analysis, it remains unclear how much of the final serving improvements comes from each components, a detailed ablation study is expected.
Questions for authors
Q1: Can the authors clarify the relationship among these components, the central contribution would be contributions of MTP optimization or hybrid-model serving optimization?
Q2: Can the authors provide a detailed end-to-end ablation to each component? How does each component affects throughput and p95 TPOT?
3 comments
Reply to Reviewer fwRj (2/2)
Q1. How do the contributions relate? The central contribution is a hybrid-model serving optimization: a verification backend that keeps the recurrent (GDN) state of the selected continuation recoverable during native-MTP verification. MTP drafting itself is unchanged.
- C2 decides what is stored and when the selected state is rebuilt: one FP32 checkpoint plus a journal of raw records, folded inside a verification that needs it when the journal fills, so the checkpoint is written on 32-46% of rounds at depth 7 (simulated from measured serving traces, Sec. 3.1).
- C3 computes candidate outputs from that representation: the journal and the new candidates form one chain from the checkpoint, solved path-first and tiled across requests, heads and value dimensions, with BF16x3 products.
C2 sets the memory traffic and state size, and C3 makes the per-round computation cheap. The ablation (W3) shows both matter: removing the journal costs 3.5%, and replacing tensor-core products with IEEE FP32 costs 10.9%. The submission's C1 is not part of the final design.
Q2. Which component produces the throughput and request-p95 gains? At serving level the journal and tensor-core arithmetic (W3 table); folding every round, eager recovery and the solver choice are within noise at concurrency 16 but measurable at layer level (App. D.1, Table 15). Request-p95:
| Request-p95 TPOT, Xtra vs | B200 | H100 |
|---|---|---|
| Released ReplaySSM | -1.1% | -1.2% |
| Our port (exact FP32) | -8.7% | - |
Takeaway: tail latency follows throughput. Higher concurrencies and 27B were not ablated.
Effect on task outputs (App. D.3, Table 17). On identical prompts (160 prompts, three repeats), Xtra differs from its reference by -1.5 to +3.1 points (GSM8K and MATH-500 against released ReplaySSM, HumanEval against SGLang), and every 95% paired interval includes 0, although outputs are not token-identical; released ReplaySSM and our port behave alike. The full table is in the general response (item 4).
Budget sweep. The latency- and memory-budget results are re-measured with the final design (Figs. 6-8, Tables 6 and 9; table in the general response, item 6). H1 selection stays in App. D.10, measured with the submitted implementation.
I do appreciate the authors’ effort in revising the manuscript and adding new experiments. However, the revised version substantially changes the paper’s main contribution. C1, the one-token branching mechanism presented as a core contribution in the original submission, has been removed from the final design. As a result, both “Xtra” and “MTP” in the title have lost their connection to the paper's main contribution. Given the substantially shifting of the contribution, I will maintain my original score.
Reply to Reviewer fwRj (1/2)
Thank you for the review. We number your weaknesses W1-W3 in the order they appear. Your points on focus and on C1 are correct, and the revision is restructured around them. Numbers refer to the revised PDF; the general response summarizes all changes.
W1. The contributions are unfocused. The paper now has one question and one design.
End to end, greedy decoding, three repeats per configuration in rotated order; all arms in one SGLang stack (Sec. 5.2, Tables 2, 19, 20):
Takeaway: clearly faster than SGLang; at parity with released ReplaySSM.
Same runs, Xtra relative to released ReplaySSM:
Takeaway: same throughput and tail latency, with half of ReplaySSM's speculative state.
At kernel level Xtra's output error is about 32x lower and its committed-state error about 1,400x lower than released ReplaySSM's (general response, item 2).
W2. C1 contributes little and is rarely selected. We agree, and the final design does not include it: Xtra verifies chains. The C1 evidence, measured with the submitted implementation, is in App. C.4 and D.10:
Submitted implementation, submitted H100 sweep (App. D.10, Tables 21-22); throughput relative to the per-workload oracle:
Takeaway: a profile from other workloads always picks chain, and the final design verifies chains, so no chain/H1 profiling remains.
Repeated low-concurrency runs showed a small p95 gain at concurrency 4 and regressions at 16.
W3. No end-to-end ablation. We added a serving-level single-variable ablation:
Serving ablation, B200, 4B and 35B-A3B on six workloads, depth 7, concurrency 16, three repeats; run-to-run noise about 2% (App. D.2, Table 16):
Takeaway: the journal (every configuration slower without it) and tensor-core arithmetic (every configuration and repeat slower with IEEE dots) explain Xtra's advantage over exact FP32 replay; folding every round, eager recovery and the solver are within noise at this load (at layer level eager recovery costs 3.6% / 24.5% of lifecycle time for the two head geometries and the Neumann solver 6.2-8.1%). Higher concurrencies and 27B were not ablated.