acceptodds
Under review as a conference paper at ICLR 2027

Xtra-MTP: State-Efficient Parallel Verification for Hybrid Language Models

Abstract

Native multi-token prediction (MTP) uses a model’s own prediction module to draft future tokens for joint target verification. In hybrid language models, which combine full-attention and recurrent layers, this opportunity faces three coupled constraints: uneven numbers of accepted tokens, duplicated candidate states, and dependencies that limit verification parallelism. We present Xtra-MTP, which jointly designs candidate coverage, recoverable state, and Gated DeltaNet execution. One-token alternatives extend one recursively drafted path without recursively expanding additional branches. A shared checkpoint and compact update records preserve candidate continuations; the next verifier recovers only the selected path, sharing state access with verification. A topology-aware solver separates updates that propagate along the path from terminal readouts, while value tiling exposes parallel execution. Across three Qwen models and six workloads, with the 95th percentile of request-average time per output token (TPOT) constrained to 30 ms, measured configurations improve six-workload geometric-mean throughput by 29.4–39.2% over the best tested SGLang configurations and 5.9–9.2% over our ReplaySSM integration.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
multi-token prediction, speculative decoding, hybrid language models, Gated DeltaNet, LLM inference, parallel verification, state-efficient inference, LLM serving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fwRjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Xtra-MTP, focusing on how to efficiently implement MTP-style speculative decoding for hybrid language models. Xtra-MTP first augments the native MTP chain with a one token alternative to mitigate the early mismatch problem in speculative decoding, and, consequently improve request-p95 TPOT. It further extends its idea to hybrid architecture, it intergrates a deferred reccurent states materilaztion and a specialized parrellel solver for spine-plus-leaf recurrent states calculation. Experiments on three Qwen models shows improvement towards both throughput and p95-TPOT.

Critical strengths
  1. The paper addresses a critical bottleneck in hybrid model serving. Hybrid models face different challenges like branch-dependent recurrent states, expensive state duplication and so on.
  2. The paper systematically implements the proposed method into serving framework. It provides integration with SGlang and also CUDA graph compatibility.
  3. The paper provides extensive experiments for the effectiveness of the proposed optimizations
Critical weaknesses
  1. The main contribution is not sufficiently focused. The paper introduces three different components, C1 one token branching, C2 deferred materializaton for recurrent states, C3 parellel computation for branching. C3 aims to mitigates the time cost brought by C1, while C2 addresses a different bottleneck. A clearer positioning of the primary contribution would strengthen the paper.
  2. Component 1 provides relatively mariginal improvement for end-to-end serving, it is only selected once under 18 workloads with latency constraints in appendix B.3.
  3. Although the paper provides detailed analysis, it remains unclear how much of the final serving improvements comes from each components, a detailed ablation study is expected.
Questions for authors

Q1: Can the authors clarify the relationship among these components, the central contribution would be contributions of MTP optimization or hybrid-model serving optimization?

Q2: Can the authors provide a detailed end-to-end ablation to each component? How does each component affects throughput and p95 TPOT?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer fwRj (1/2)

Thank you for the review. We number your weaknesses W1-W3 in the order they appear. Your points on focus and on C1 are correct, and the revision is restructured around them. Numbers refer to the revised PDF; the general response summarizes all changes.

W1. The contributions are unfocused. The paper now has one question and one design.

  • The question. During native-MTP verification in a hybrid model, what must be stored and computed so that the selected continuation's recurrent state stays recoverable, and at what cost in latency, speculative state and numerical precision?
  • The design. Xtra keeps an FP32 checkpoint and a journal of up to 16 raw BF16 records, folded into the checkpoint inside a later verification when it fills (C2, Sec. 4.2). Candidate outputs come from a path-first, value-tiled solve in BF16x3 (C3).
  • The result (Sec. 5.2, Table 2):

End to end, greedy decoding, three repeats per configuration in rotated order; all arms in one SGLang stack (Sec. 5.2, Tables 2, 19, 20):

Throughput, Xtra vs4B35B-A3B27BAll configurations
SGLang built-in verification, B200+15.6%+10.7%+24.0%+16.7% (18 of 18 faster)
SGLang built-in verification, H100+34.1%+16.1%not run+29.4% (8 configs that fit in memory)
Released ReplaySSM, B200+0.9%+1.2%+0.6%+0.9% [per config -2.2, +6.4]
Released ReplaySSM, H100+1.8%+1.0%not run+1.4% [per config -0.5, +4.6]
Our port (exact FP32), B200+8.7%+6.7%+11.7%+9.0% (18 of 18 faster)
Our port (exact FP32), H100+15.6%+9.1%not run+12.3%

Takeaway: clearly faster than SGLang; at parity with released ReplaySSM.

Same runs, Xtra relative to released ReplaySSM:

MetricB200 (18 configs)H100 (12 configs)
Request-p95 TPOT-1.1%-1.2%
Throughput, same-cohort baseline+1.5%-
Speculative GDN state0.51x0.50-0.51x
Speculative GDN state of SGLang, relative to Xtra42-94x-

Takeaway: same throughput and tail latency, with half of ReplaySSM's speculative state.

At kernel level Xtra's output error is about 32x lower and its committed-state error about 1,400x lower than released ReplaySSM's (general response, item 2).

W2. C1 contributes little and is rarely selected. We agree, and the final design does not include it: Xtra verifies chains. The C1 evidence, measured with the submitted implementation, is in App. C.4 and D.10:

Submitted implementation, submitted H100 sweep (App. D.10, Tables 21-22); throughput relative to the per-workload oracle:

Policy or quantityResult
H1 selected under 20/30/40 ms budgets2 of 54 cells (one met 30 ms by 0.018 ms in a single run)
Chain/H1 per (D, C) chosen from the other five workloadsidentical to always-chain; within 0-3.7% of the oracle
Always H1-4.0 to -15.6%
One fixed (D, C, topology) taken from five workloads0 to -22.3%; over the 20 ms budget on 1 held-out workload
Profiling cost of the sweep (estimate, 201 timed runs)about 150 H100 GPU-hours for 1,830 runs; H1 runs about 22

Takeaway: a profile from other workloads always picks chain, and the final design verifies chains, so no chain/H1 profiling remains.

Repeated low-concurrency runs showed a small p95 gain at concurrency 4 and regressions at 16.

W3. No end-to-end ablation. We added a serving-level single-variable ablation:

Serving ablation, B200, 4B and 35B-A3B on six workloads, depth 7, concurrency 16, three repeats; run-to-run noise about 2% (App. D.2, Table 16):

ChangeRelative toThroughputRequest-p95
Fold every round instead of when the journal fillsXtra-0.8%+0.4%
No journal (checkpoint rewritten every round)Xtra-3.5%+3.4%
...plus eager recovery at commitno journal-0.6%+1.8%
...plus Neumann solver instead of path-firstno journal-0.3%+0.2%
...plus IEEE FP32 dots instead of BF16x3no journal-10.9%+12.7%
Released ReplaySSM / our portXtra-1.2% / -5.7%+1.9% / +6.6%

Takeaway: the journal (every configuration slower without it) and tensor-core arithmetic (every configuration and repeat slower with IEEE dots) explain Xtra's advantage over exact FP32 replay; folding every round, eager recovery and the solver are within noise at this load (at layer level eager recovery costs 3.6% / 24.5% of lifecycle time for the two head geometries and the Neumann solver 6.2-8.1%). Higher concurrencies and 27B were not ablated.

AuthorsOpenReview ↗

Reply to Reviewer fwRj (2/2)

Q1. How do the contributions relate? The central contribution is a hybrid-model serving optimization: a verification backend that keeps the recurrent (GDN) state of the selected continuation recoverable during native-MTP verification. MTP drafting itself is unchanged.

  • C2 decides what is stored and when the selected state is rebuilt: one FP32 checkpoint plus a journal of raw records, folded inside a verification that needs it when the journal fills, so the checkpoint is written on 32-46% of rounds at depth 7 (simulated from measured serving traces, Sec. 3.1).
  • C3 computes candidate outputs from that representation: the journal and the new candidates form one chain from the checkpoint, solved path-first and tiled across requests, heads and value dimensions, with BF16x3 products.

C2 sets the memory traffic and state size, and C3 makes the per-round computation cheap. The ablation (W3) shows both matter: removing the journal costs 3.5%, and replacing tensor-core products with IEEE FP32 costs 10.9%. The submission's C1 is not part of the final design.

Q2. Which component produces the throughput and request-p95 gains? At serving level the journal and tensor-core arithmetic (W3 table); folding every round, eager recovery and the solver choice are within noise at concurrency 16 but measurable at layer level (App. D.1, Table 15). Request-p95:

Request-p95 TPOT, Xtra vsB200H100
Released ReplaySSM-1.1%-1.2%
Our port (exact FP32)-8.7%-

Takeaway: tail latency follows throughput. Higher concurrencies and 27B were not ablated.

Effect on task outputs (App. D.3, Table 17). On identical prompts (160 prompts, three repeats), Xtra differs from its reference by -1.5 to +3.1 points (GSM8K and MATH-500 against released ReplaySSM, HumanEval against SGLang), and every 95% paired interval includes 0, although outputs are not token-identical; released ReplaySSM and our port behave alike. The full table is in the general response (item 4).

Budget sweep. The latency- and memory-budget results are re-measured with the final design (Figs. 6-8, Tables 6 and 9; table in the general response, item 6). H1 selection stays in App. D.10, measured with the submitted implementation.

I do appreciate the authors’ effort in revising the manuscript and adding new experiments. However, the revised version substantially changes the paper’s main contribution. C1, the one-token branching mechanism presented as a core contribution in the original submission, has been removed from the final design. As a result, both “Xtra” and “MTP” in the title have lost their connection to the paper's main contribution. Given the substantially shifting of the contribution, I will maintain my original score.

Reviewer_K7UtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper studies speculative decoding with native multi-token prediction in hybrid models that mix full attention with Gated DeltaNet (GDN) layers, using the Qwen3.5/3.6/3.8 family. The authors point to three bottlenecks: uneven numbers of accepted tokens, duplicated candidate states, and dependencies that limit verification parallelism. The authors propose Xtra-MTP, which jointly designs candidate coverage, recoverable state, and Gated DeltaNet execution. The system is built on SGLang and evaluated on three Qwen checkpoints and six workloads.

Critical strengths
  • Timely problem, and having three bottlenecks makes the paper structured.
  • Substantial systems work with full SGLang integration.
Critical weaknesses
  • The paper is hard to follow: terms such as spine, H1 and beam2 are used before they are defined, which makes the contributions difficult to pin down.
  • There is no ablation study, so it is unclear how much each of the three contributions accounts for the end-to-end gain.
  • Mixed precisions across arms (FP8, BF16, FP32) make it unclear whether the gains come from the proposed methods or from Xtra running at lower precision than the baselines.
  • The gain over the strongest baseline, ReplaySSM, is marginal, and the paper does not explain clearly how Xtra differs from it or why the gain is small.
  • The experimental scope is narrow (one model family, one GPU type, one hybrid design, one latency budget), which is narrower than the title suggests.
Questions for authors

Here is my feedback on this paper below.

First, the paper is not easy to follow its content and contributions clearly. The authors often use some terminology without defining it clearly in the first place; to name a few, spine, H1, beam2. I do not think that these are standard terms that all reviewers and readers would understand without any confusion (please correct me if I'm wrong).

So I would like to suggest that the authors clearly provide a definition of every new term before referring to them in the paper.

Second, it is unclear which part of the design, out of the three contributions, produces the end-to-end gain. According to the paper, each contribution is proposed to resolve the corresponding three challenges. However, the impact of each proposed solution is missing.

I would like to suggest that the authors conduct a comprehensive ablation study to quantify the impact of each contribution. Without this information, it's not possible to gauge the effectiveness of each proposed method.

Third, in the experimental setup and in the Appendix, the authors mentioned that different precisions such as FP8, BF16, and FP32 are used in the evaluation. This point makes the evaluation confusing because it's not clear whether the improved throughputs are due to the proposed methods or to the fact that Xtra uses lower precision and baselines use higher precision.

Fourth, as shown in Figures 6, 7, and 8 in Section 5, the performance difference between Xtra and its strong baseline, ReplaySSM, is marginal. I believe the paper needs more description of how Xtra is meaningfully different from its strong baseline, what the novelty of Xtra is, and reasons why the performance gain is small.

Last but not least, the experimental evidence is narrower than the title suggests in that the paper covers only a narrow scenario with one model family, one GPU type, one hybrid architecture design, and one request budget setup (30ms request-p95 budget).

I would like to encourage the authors to add different hybrid model architectures, different GPUs, different hybrid designs (e.g., Mamba-2), and different request budget setups.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer K7Ut (1/2)

Thank you for the review. We number your weaknesses W1-W5 in the order they appear (your five questions follow the same order). Your points on presentation and precision led to the largest changes. Numbers refer to the revised PDF; the general response summarizes all changes.

W1. Hard to read; coined terms undefined. We agree. The revision adds a terminology and notation table (Table 1, Sec. 2), describes "consumer-coupled state recovery" as recovery deferred to, and fused into, later verifications, uses "checkpoint" only for the GDN state checkpoint, always qualifies "ReplaySSM" as released or port, presents one design (Sec. 4.2), and closes Sec. 5 with a Limitations paragraph. Key terms:

TermMeaning
Spinethe recursively drafted path of D tokens
Checkpointthe FP32 GDN state after the root
Journalup to 16 raw records (BF16 key/value, FP32 gates) per slot and layer
Foldwriting the journal into the checkpoint inside a verification, only when it fills (32-46% of rounds at depth 7, simulated from measured traces)
Consumerthe next verification
Lifecycle timeGDN verification, selection, record maintenance and final reconstruction
BF16x3a dot product as three BF16 products with FP32 accumulation
B, C, P / D, E, Nactive batch, client concurrency, request slots / spine length, leaves, verified nodes

W2. No ablation of which component produces the gain. We added a serving-level single-variable ablation:

Serving ablation, B200, 4B and 35B-A3B on six workloads, depth 7, concurrency 16, three repeats; run-to-run noise about 2% (App. D.2, Table 16):

ChangeRelative toThroughputRequest-p95
Fold every round instead of when the journal fillsXtra-0.8%+0.4%
No journal (checkpoint rewritten every round)Xtra-3.5%+3.4%
...plus eager recovery at commitno journal-0.6%+1.8%
...plus Neumann solver instead of path-firstno journal-0.3%+0.2%
...plus IEEE FP32 dots instead of BF16x3no journal-10.9%+12.7%
Released ReplaySSM / our portXtra-1.2% / -5.7%+1.9% / +6.6%

Takeaway: the journal (every configuration slower without it) and tensor-core arithmetic (every configuration and repeat slower with IEEE dots) explain Xtra's advantage over exact FP32 replay; folding every round, eager recovery and the solver are within noise at this load (at layer level eager recovery costs 3.6% / 24.5% of lifecycle time for the two head geometries and the Neumann solver 6.2-8.1%). Higher concurrencies and 27B were not ablated.

W3. FP8, BF16 and FP32 are mixed; is the gain a precision artefact? Not relative to released ReplaySSM. Relative to our exact FP32 port, Xtra does rely on tensor-core arithmetic, and we concede that point with numbers.

Arithmetic per arm, read from the kernel sources (Table 4); FP8 in a model name is the precision of the released weights, which every arm shares:

ArmCommitted stateDot products
SGLang, our portFP32FP32 element-wise on CUDA cores, no tensor cores
Released ReplaySSMBF16-rounded checkpointone BF16 tensor-core product, FP32 accumulation
XtraFP32 checkpointBF16x3 (three BF16 products, FP32 accumulation)

Takeaway: Xtra uses tensor cores like ReplaySSM but keeps an FP32 committed state.

Layer-level effect of the arithmetic (single GDN layer, H100, depth 7, no-journal kernel; App. D.1, Table 15):

Arithmetic change in the no-journal kernelEffect on lifecycle time
IEEE FP32 dots instead of BF16x31.6-1.8x slower than our port
Single BF16 products instead of BF16x3only 8-13% faster

Takeaway: tensor cores are needed to beat exact FP32 replay, but precision within tensor-core modes matters little.

Single GDN layer, H100, depth 7, 32 rounds, three fresh processes; error on 12 matched fixtures over 2,048 rounds against an FP32 reference (App. D.1, Table 15):

ArmLatency vs ReplaySSMAux. state (MiB)Output error (rel. L2)Committed-state error (rel. Linf)
Released ReplaySSM124.254.41e-36.71e-3
Xtra1.06-1.11x12.251.36e-44.9e-6

Takeaway: about 32x lower output error and 1,400x lower committed-state error, with no drift over 2,048 rounds; the 6-11% kernel gap comes from fold rows re-reading the checkpoint.

End to end Xtra matches released ReplaySSM's throughput (+0.9% B200, +1.4% H100; W4), so it does not buy speed by giving up precision relative to ReplaySSM.

AuthorsOpenReview ↗

Reply to Reviewer K7Ut (2/2)

W3 (continued). Effect on outputs.

Task accuracy on identical prompts, B200 greedy outputs, depth 7, concurrency 16, 160 prompts x 3 repeats; difference from the reference in points with 95% paired bootstrap interval (App. D.3, Table 17):

Task, modelReference score (%)XtraReleased ReplaySSMOur port
GSM8K, 4BReplaySSM 80.0-1.0 [-3.8, +1.2]ref.-1.2 [-3.1, 0.0]
GSM8K, 35B-A3BReplaySSM 83.3+1.7 [-1.0, +4.6]ref.+1.5 [-1.0, +4.2]
MATH-500, 4BReplaySSM 25.0+3.1 [-0.6, +7.5]ref.+2.3 [-1.5, +6.3]
MATH-500, 35B-A3BReplaySSM 23.3+2.7 [-2.1, +7.5]ref.+3.5 [-1.7, +8.8]
HumanEval, 4BSGLang 78.8-1.5 [-3.3, 0.0]0.0 [-2.5, +2.5]0.0 [-2.3, +2.3]
HumanEval, 35B-A3BSGLang 90.6-1.3 [-3.8, +1.3]0.0 [-3.1, +3.1]-0.6 [-1.9, 0.0]
HumanEval, 27BSGLang 88.8-0.2 [-2.3, +1.7]+0.6 [-1.3, +3.1]+0.6 [-1.9, +3.8]

Takeaway: no backend differs from its reference by more than about 3.5 points and every interval includes 0, although only 0-56% of outputs are token-identical. GSM8K and MATH-500 have no SGLang run, so released ReplaySSM is their reference; MATH-500 scores are low because of the 512-token cap and a fallback answer extraction; MT-Bench, Alpaca and LiveCodeBench were not scored.

Token agreement with SGLang's chain output, 4B, depth 3, concurrency 1, exact matches out of 160 prompts (App. D.6):

ArmHumanEval, B200MT-Bench, B200HumanEval, H100MT-Bench, H100
Xtra852785-
Released ReplaySSM85248334
Our port (exact FP32)7928160160

Takeaway: only our port is token-identical, and only on H100; Xtra and released ReplaySSM agree with SGLang at the level of SGLang's own beam2 against its chain.

W4. Single-digit gain over ReplaySSM; difference unclear. We now compare with released ReplaySSM directly, in the same stack.

End to end, greedy decoding, three repeats per configuration in rotated order; all arms in one SGLang stack (Sec. 5.2, Tables 2, 19, 20):

Throughput, Xtra vs4B35B-A3B27BAll configurations
SGLang built-in verification, B200+15.6%+10.7%+24.0%+16.7% (18 of 18 faster)
SGLang built-in verification, H100+34.1%+16.1%not run+29.4% (8 configs that fit in memory)
Released ReplaySSM, B200+0.9%+1.2%+0.6%+0.9% [per config -2.2, +6.4]
Released ReplaySSM, H100+1.8%+1.0%not run+1.4% [per config -0.5, +4.6]
Our port (exact FP32), B200+8.7%+6.7%+11.7%+9.0% (18 of 18 faster)
Our port (exact FP32), H100+15.6%+9.1%not run+12.3%

Takeaway: clearly faster than SGLang; at parity with released ReplaySSM.

Same runs, Xtra relative to released ReplaySSM:

MetricB200 (18 configs)H100 (12 configs)
Request-p95 TPOT-1.1%-1.2%
Throughput, same-cohort baseline+1.5%-
Speculative GDN state0.51x0.50-0.51x
Speculative GDN state of SGLang, relative to Xtra42-94x-

Takeaway: same throughput and tail latency, with half of ReplaySSM's speculative state.

  • The claim. Against released ReplaySSM we claim parity, not a speedup, measured at concurrency 16-128; at batch 1-16 Xtra's layer kernel is slower except at batch 1 with a small-batch launch setting (App. D.1).
  • The difference is state and precision. Xtra keeps an FP32 checkpoint and raw records instead of BF16 history and a BF16-rounded checkpoint; it holds half of ReplaySSM's speculative state, and its committed-state error is about 1,400x lower (W3).
  • Already in ReplaySSM. Compact records, deferred write-back and a triangular solve already exist there, and the introduction says so.

W5. Narrow scope. End-to-end results now cover NVIDIA B200 (18 configurations, three models) and H100 (12 configurations), with three repeats each. The abstract and Limitations scope every claim to Gated DeltaNet hybrids. Not done: other hybrid designs such as Mamba-2, and other model families; sampling decoding; 27B on H100 against released ReplaySSM; budget sweeps with released ReplaySSM. The submission already had 20/30/40 ms budgets, process-memory budgets and fitted curves; the revision re-measures them with the final design (Figs. 6-8, Tables 6 and 9; table in the general response, item 6).

AuthorsOpenReview ↗

Follow-up on W5 (one model family, one hybrid design).

The updated PDF adds App. D.12 and Table 23: Xtra's design ported to Kimi Delta Attention (KDA), the recurrence of Kimi Linear. KDA has one decay gate per key channel instead of GDN's one per head. This is layer level only.

What carries over: the FP32 checkpoint, the raw journal folded inside a later verification, the path-first solve and BF16x3 products.

What had to change:

  • Records. Journal records carry the per-channel gate, so a record is 1,028 bytes per head and token (516 for GDN).
  • Exponents. They are kept non-positive by relative factoring, because naive cumulative scaling would overflow or underflow FP32 with the released gates.
  • Fold schedule. It adapts to batch: fold every round at small batch and at depth 15, the 16-record journal otherwise.

Setting: one KDA layer with Kimi-Linear-48B-A3B's released shapes and gate parameters, synthetic activations, depth 7, microseconds per round, three fresh processes per cell.

GPU, batchXtra-KDAExact FP32 replaySGLang KDA verifyXtra faster than replay / SGLang
B200, batch 114.818.122.61.23x / 1.53x
B200, batch 128307.5632.6872.12.06x / 2.84x
H100, batch 119.920.924.31.05x / 1.22x
H100, batch 128457.8828.01750.11.81x / 3.82x

Key numbers across depths 3, 7 and 15:

  • Against SGLang's KDA verification: faster at batch 16 and above, by 1.4-3.0x on B200 and 1.7-4.0x on H100.
  • Against the exact replay: faster at batch 16 and above on B200 and at batch 32 and above on H100.
  • Error: against FP64 over 2,048 rounds, output error is at most 3.4e-4 and committed-state error at most 1.9e-5, with no drift.
  • State: 15-43x less speculative state than SGLang, but 1.5-4.3x more than the exact replay.

Where it is not better:

  • Small batch. It loses to SGLang in a few small-batch cells: B200 depth 3 at batch 4 and 8, and H100 depth 3 at batch 2.
  • Exact replay. It is slower than the exact replay on B200 at depth 3, batch 4, and on H100 at depth 3, batch 16, and depth 15, batch 1-2.
  • Batch 1 on H100. It is roughly at parity at depths 3 and 7.

Limits:

  • No serving result. Kimi-Linear-48B-A3B has no native MTP head, so there is no MTP model to serve.
  • No ReplaySSM comparison. Released ReplaySSM has no KDA kernel.
  • Layer study only. It is a single layer with synthetic activations.
  • Still not covered. Other hybrid designs such as Mamba-2.

Takeaway: the design's state and arithmetic choices carry over to a second gated linear-attention family at layer level. Our serving claims remain limited to the Qwen GDN hybrids.

Reviewer_2hzzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies parallel verification of native multi-token prediction (MTP) drafts in hybrid models that interleave full attention with Gated DeltaNet (GDN) layers. It identifies three serving bottlenecks: tail latency that does not improve with median latency, memory consumed by per-candidate recurrent states, and limited parallelism in the recurrent dependency solve. Xtra-MTP addresses them with one-token terminal leaves placed along a single drafted spine, a post-root checkpoint plus raw operand journal whose recovery is deferred into the next verifier, and a spine-first GDN solve with value tiling. Validation is in SGLang on three Qwen hybrid models and six workloads. Under a request-p95 latency budget, Xtra-MTP improves throughput by roughly a third over the best SGLang configuration and by 6 to 9% over the ReplaySSM integration.

Critical strengths

The method is exact in real arithmetic and gives a consistent end-to-end throughput gain over every baseline in every model-workload combination, measured over a broad grid of models, workloads, draft depths and concurrency levels. This supports acceptance because hybrid models with native MTP heads are now being deployed, and a lossless serving improvement that holds across this range is directly usable.

The kernel-level evidence is careful and explains the end-to-end results. Figure 4d isolates the spine-first solver and value tiling within the same implementation, and Figure 21a shows that GDN accounts for only part of each round's time, which is why a large kernel speedup becomes a modest throughput gain. This matters because the gain can be traced to specific components rather than only to the system as a whole.

Critical weaknesses

The improvement specific to Xtra-MTP is modest, and the first of the three contributions adds little to it. ReplaySSM is the fastest baseline throughout, and Xtra-MTP's margin over it is in the single digits (Table 3). Against ReplaySSM, the memory peaks are about equal, so the large memory-budget gains are relative to SGLang's per-candidate matrices, which compact-state designs already remove. Budgeted terminal coverage is rarely selected: the chosen Xtra configuration is almost always the plain chain, and adding the terminal leaf lowers throughput at higher concurrency (Figure 21b). The abstract and introduction present the three components as jointly producing the throughput gain. This is consequential because, over existing compact-state designs, what is new reduces to the recovery and solver changes, and their benefit is small.

The small margin is measured against the authors' own port, and the paper is hard to read. Appendix A.1 states that the ReplaySSM port differs from the published formulation, so the margin depends on how faithful the port is. Separately, the coined vocabulary (spine, Global H1, lifecycle time, consumer, journal, cursor), together with dense shorthand and many numbers per sentence, makes the method difficult to reconstruct without the appendix. This weighs on the recommendation because the only comparison that isolates the contribution is against a reimplementation, and the presentation makes that comparison hard to check.

Questions for authors
  • What fraction of the gain over ReplaySSM remains if Xtra is restricted to the chain topology only? This would separate terminal coverage from the recovery and solver changes.
  • How does the ReplaySSM port compare with the published implementation where both can run, and which differences in time and memory would be expected between the two formulations?
  • How costly is the offline profiling that selects chain or H1 per model, workload, depth, and concurrency, and how does a single fixed profile perform on an unseen workload?
  • All runs use greedy decoding on one GPU type. Scoping the claims to this setting, or adding one result with sampling, would help readers judge how far the results generalize.
  • A short glossary in the main text, or fewer coined terms, would make the design much easier to follow.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer 2hzz (1/2)

Thank you for the review. We number your weaknesses W1-W2 and questions Q1-Q5 in the order they appear, and answer Q2 first, because it changes how the rest should be read. Numbers refer to the revised PDF; the general response summarizes all changes.

Q2. Comparison with the released ReplaySSM. We integrated the authors' released kernel (base16, unchanged) into our SGLang stack. It is bitwise-qualified against its own API (480/480 layer outputs, 140/140 flushes on real serving state), and every arm uses the same launch commands. Parity below is measured at concurrency 16-128; at batch 1-16 Xtra's layer kernel is slower, except at batch 1 with a small-batch launch setting (App. D.1).

End to end, greedy decoding, three repeats per configuration in rotated order; all arms in one SGLang stack (Sec. 5.2, Tables 2, 19, 20):

Throughput, Xtra vs4B35B-A3B27BAll configurations
SGLang built-in verification, B200+15.6%+10.7%+24.0%+16.7% (18 of 18 faster)
SGLang built-in verification, H100+34.1%+16.1%not run+29.4% (8 configs that fit in memory)
Released ReplaySSM, B200+0.9%+1.2%+0.6%+0.9% [per config -2.2, +6.4]
Released ReplaySSM, H100+1.8%+1.0%not run+1.4% [per config -0.5, +4.6]
Our port (exact FP32), B200+8.7%+6.7%+11.7%+9.0% (18 of 18 faster)
Our port (exact FP32), H100+15.6%+9.1%not run+12.3%

Takeaway: clearly faster than SGLang; at parity with released ReplaySSM.

Same runs, Xtra relative to released ReplaySSM:

MetricB200 (18 configs)H100 (12 configs)
Request-p95 TPOT-1.1%-1.2%
Throughput, same-cohort baseline+1.5%-
Speculative GDN state0.51x0.50-0.51x
Speculative GDN state of SGLang, relative to Xtra42-94x-

Takeaway: same throughput and tail latency, with half of ReplaySSM's speculative state.

We read this as parity with released ReplaySSM: the margins are within about one repeat-spread in several configurations.

Single GDN layer, H100, depth 7, 32 rounds, three fresh processes; error on 12 matched fixtures over 2,048 rounds against an FP32 reference (App. D.1, Table 15):

ArmLatency vs ReplaySSMAux. state (MiB)Output error (rel. L2)Committed-state error (rel. Linf)
Released ReplaySSM124.254.41e-36.71e-3
Xtra1.06-1.11x12.251.36e-44.9e-6

Takeaway: about 32x lower output error and 1,400x lower committed-state error, with no drift over 2,048 rounds; the 6-11% kernel gap comes from fold rows re-reading the checkpoint.

Caveats: ReplaySSM gets a zero-copy packed-QKV view that slightly favours it. SGLang JIT compiles inside the timed window occur in every arm; in the baseline cohort, excluding those runs moved the means by at most 0.7 points.

W1. C1 contributes little; contributions unfocused; single-digit gain.

  • C1. We agree. The final design verifies chains and does not include the one-token terminal alternatives. Their evidence, measured with the submitted implementation, moved to App. C.4 and D.10 (table under Q3).
  • Focus. The paper now has one question and one design. What must be stored and computed to keep the selected continuation's recurrent state recoverable? Xtra's answer is an FP32 checkpoint plus a 16-record raw journal folded inside a later verification (C2), with a path-first, value-tiled solve in BF16x3 (C3).
  • Size of the gain. Against SGLang's built-in verification, +16.7% on B200 and +29.4% on H100 with 42-94x less speculative state; against released ReplaySSM, parity at half the state and far lower error, not a speedup (tables under Q2).
  • Memory. We agree that against ReplaySSM the process peaks are close. Same end-to-end runs as Q2:
Memory, Xtra vs released ReplaySSMB200H100
Process peak (NVML)within +0.1 / -1.2 GiBabout 1% lower
Speculative GDN state0.51x0.50-0.51x

Takeaway: the large memory-budget gains are against SGLang's per-candidate states, which compact-state designs, ReplaySSM included, also avoid; Xtra's own difference is half of ReplaySSM's speculative state.

W2. The baseline is our own port; readability.

  • The port's role. Released ReplaySSM is now the main ReplaySSM comparison. Our port remains only as the exact-arithmetic baseline: it reproduces SGLang token for token on H100. On B200, released ReplaySSM is 8.1% faster than the port and Xtra 9.0% faster.
  • Readability. See Q5.
AuthorsOpenReview ↗

Reply to Reviewer 2hzz (2/2)

Q1. Chain-only gain, and which component produces it. The final design verifies chains only, so every result in the revision is chain-only: parity with released ReplaySSM (+0.9% B200, +1.4% H100) at half its state. In the submitted sweep (submitted implementation, App. D.10, 30 ms budget):

ModelShare of the gain over our port kept with chains only
4B100%
27B100%
35B-A3B56.6%

To separate the remaining components we ran a serving-level single-variable ablation:

Serving ablation, B200, 4B and 35B-A3B on six workloads, depth 7, concurrency 16, three repeats; run-to-run noise about 2% (App. D.2, Table 16):

ChangeRelative toThroughputRequest-p95
Fold every round instead of when the journal fillsXtra-0.8%+0.4%
No journal (checkpoint rewritten every round)Xtra-3.5%+3.4%
...plus eager recovery at commitno journal-0.6%+1.8%
...plus Neumann solver instead of path-firstno journal-0.3%+0.2%
...plus IEEE FP32 dots instead of BF16x3no journal-10.9%+12.7%
Released ReplaySSM / our portXtra-1.2% / -5.7%+1.9% / +6.6%

Takeaway: the journal (every configuration slower without it) and tensor-core arithmetic (every configuration and repeat slower with IEEE dots) explain Xtra's advantage over exact FP32 replay; folding every round, eager recovery and the solver are within noise at this load (at layer level eager recovery costs 3.6% / 24.5% of lifecycle time for the two head geometries and the Neumann solver 6.2-8.1%). Higher concurrencies and 27B were not ablated.

Q3. Profiling cost; a fixed profile on a new workload. The final design verifies chains, so there is no chain-versus-H1 decision to profile; choosing depth and concurrency under a budget is the same search for every method. For the submission's chain/H1 choice:

Submitted implementation, submitted H100 sweep (App. D.10, Tables 21-22); throughput relative to the per-workload oracle:

Policy or quantityResult
H1 selected under 20/30/40 ms budgets2 of 54 cells (one met 30 ms by 0.018 ms in a single run)
Chain/H1 per (D, C) chosen from the other five workloadsidentical to always-chain; within 0-3.7% of the oracle
Always H1-4.0 to -15.6%
One fixed (D, C, topology) taken from five workloads0 to -22.3%; over the 20 ms budget on 1 held-out workload
Profiling cost of the sweep (estimate, 201 timed runs)about 150 H100 GPU-hours for 1,830 runs; H1 runs about 22

Takeaway: a profile from other workloads always picks chain, and the final design verifies chains, so no chain/H1 profiling remains.

Q4. Scope.

  • Covered. End-to-end results now cover two GPU types: B200 (18 configurations, three models) and H100 (12 configurations).
  • Not covered. All evidence is for GDN hybrids of one model family under greedy decoding. Sampling was not evaluated, and 27B was not run on H100 against released ReplaySSM.
  • Latency budgets. These were not single in the submission either, and the revision re-measures all of them with the final design: 20/30/40 ms budgets in Fig. 7 and Table 6, process-memory budgets in Fig. 8 and Table 9, fitted curves in Fig. 6. The table is in the general response (item 6).

Q5. Undefined terms. The revision adds a terminology table at the start of Sec. 2 (Table 1):

TermMeaning
Spinethe recursively drafted path of D tokens
Checkpointthe FP32 GDN state after the root
Journalup to 16 raw records (BF16 key/value, FP32 gates) per slot and layer
Foldwriting the journal into the checkpoint inside a verification when it fills
Consumerthe next verification
Lifecycle timeGDN verification, selection, record maintenance and final reconstruction
B, C, P / D, E, Nactive batch, client concurrency, request slots / spine length, leaves, verified nodes

"Consumer-coupled state recovery" is now described as recovery deferred to, and fused into, later verifications. "ReplaySSM" is always qualified as released or port.

Thank you for the response. Against the released ReplaySSM, throughput, tail latency, and memory peaks are at parity, so the large gains are only relative to SGLang, and what remains is a smaller speculative state and lower numerical error without a measurable accuracy benefit. The revision also replaces the submitted design, removing terminal coverage, one of the three contributions. The evidence remains limited to one model family under greedy decoding. I will keep my score.

AuthorsOpenReview ↗

Response to Reviewer 2hzz

Thank you for the follow-up and for confirming the measured facts. We would like to clarify two points for the record.

1. Evidence beyond one model family. The revision already evaluates a second model family with a different recurrent design: Kimi Linear, whose Kimi Delta Attention (KDA) uses one decay gate per key channel instead of GDN's one per head (App. D.12, Table 23; our follow-up comment of Nov. 15).

  • What carries over. The same design: FP32 checkpoint, raw journal folded inside a later verification, path-first BF16x3 solve. The only changes are per-channel gates stored in the journal and exponents kept non-positive.
  • Speed. Against SGLang's own KDA verification, Xtra-KDA is faster in every measured cell except three small-batch cells. At batch 16 and above it is 1.4–3.0× faster on B200 and 1.7–4.0× on H100.
  • State. 15–43× less speculative state than SGLang's KDA verification.
  • Error. Output error at most 3.4e-4 and committed-state error at most 1.9e-5 against an FP64 reference, with no drift over 2,048 rounds.
  • Why layer level only. Kimi-Linear-48B-A3B ships no native MTP head, so an end-to-end native-MTP serving run is not possible on that family. Released ReplaySSM has no KDA kernel. So the scope limit comes from the available checkpoints, not from the method.
  • End to end within GDN. We cover three Qwen hybrids (4B, 27B, 35B-A3B), two GPU types (H100 and B200), and the full 276-configuration budget sweep.

2. Greedy decoding. The acceptance rule decides which prefix is accepted. The verification backend computes and recovers the recurrent state for whatever prefix is accepted, through the same journal and fold path.

  • Sampling changes acceptance lengths, and hence the throughput gain, but not the mechanism or its correctness.
  • We did not measure sampling and do not claim results for it. The limitations section says so.

On the remaining points.

  • Parity with released ReplaySSM. We agree, and the paper claims parity, not a speedup. Xtra matches its throughput and tail latency with half of its speculative state and an FP32-accurate committed state. We do not claim a task-accuracy benefit from the lower error.
  • Gains relative to SGLang. These are against the verification path that SGLang deploys by default (+16.7% on B200 in all 18 configurations; +38–75% under latency budgets).
  • The design change. Terminal coverage was removed because, as you noted in W1, it contributed little. The two remaining components, deferred state recovery and the path-first solve, are those of the submission. Their fold schedule changed, as disclosed in the paper.

Thank you again for the detailed reviews and the discussion.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1/4)

Thank you for the reviews. The revised PDF marks changed text in blue. Section, table and appendix numbers below refer to the revision.

Implementation. The revised implementation defers checkpoint folding through a 16-record raw journal, and all main-text results use it except the Sec. 3.2-3.3 controls (Fig. 4); those controls, App. B outside Tables 6 and 9, App. C and App. D.10 use the submitted per-round checkpoint rewrite.

What Xtra-MTP is. Xtra keeps an FP32 checkpoint and a journal of up to 16 raw BF16 update records per slot and layer. Each verification reads the journal and the new candidates as one chain from the checkpoint. The journal is folded into the checkpoint inside a verification only when it fills, which happens on 32-46% of rounds at depth 7 when measured serving traces are replayed through the fold rule (Sec. 3.1, Fig. 2; 28% on the kernel benchmark's synthetic trace). Dot products sum three BF16 products with FP32 accumulation (BF16x3).

1. Main comparison. We integrated the authors' released ReplaySSM kernel (base16, unchanged, bitwise-qualified against its own API) into the same SGLang stack as every other arm.

End to end, greedy decoding, three repeats per configuration in rotated order; all arms in one SGLang stack (Sec. 5.2, Tables 2, 19, 20):

Throughput, Xtra vs4B35B-A3B27BAll configurations
SGLang built-in verification, B200+15.6%+10.7%+24.0%+16.7% (18 of 18 faster)
SGLang built-in verification, H100+34.1%+16.1%not run+29.4% (8 configs that fit in memory)
Released ReplaySSM, B200+0.9%+1.2%+0.6%+0.9% [per config -2.2, +6.4]
Released ReplaySSM, H100+1.8%+1.0%not run+1.4% [per config -0.5, +4.6]
Our port (exact FP32), B200+8.7%+6.7%+11.7%+9.0% (18 of 18 faster)
Our port (exact FP32), H100+15.6%+9.1%not run+12.3%

Takeaway: clearly faster than SGLang; at parity with released ReplaySSM.

Same runs, Xtra relative to released ReplaySSM:

MetricB200 (18 configs)H100 (12 configs)
Request-p95 TPOT-1.1%-1.2%
Throughput, same-cohort baseline+1.5%-
Speculative GDN state0.51x0.50-0.51x
Speculative GDN state of SGLang, relative to Xtra42-94x-

Takeaway: same throughput and tail latency, with half of ReplaySSM's speculative state.

Against released ReplaySSM the claim is parity, not a speedup: the differences are within about one repeat-spread in several configurations.

2. Kernel-level latency, state and error.

Single GDN layer, H100, depth 7, 32 rounds, three fresh processes; error on 12 matched fixtures over 2,048 rounds against an FP32 reference (App. D.1, Table 15):

ArmLatency vs ReplaySSMAux. state (MiB)Output error (rel. L2)Committed-state error (rel. Linf)
Released ReplaySSM124.254.41e-36.71e-3
Xtra1.06-1.11x12.251.36e-44.9e-6

Takeaway: about 32x lower output error and 1,400x lower committed-state error, with no drift over 2,048 rounds; the 6-11% kernel gap comes from fold rows re-reading the checkpoint.

3. Serving-level ablation.

Serving ablation, B200, 4B and 35B-A3B on six workloads, depth 7, concurrency 16, three repeats; run-to-run noise about 2% (App. D.2, Table 16):

ChangeRelative toThroughputRequest-p95
Fold every round instead of when the journal fillsXtra-0.8%+0.4%
No journal (checkpoint rewritten every round)Xtra-3.5%+3.4%
...plus eager recovery at commitno journal-0.6%+1.8%
...plus Neumann solver instead of path-firstno journal-0.3%+0.2%
...plus IEEE FP32 dots instead of BF16x3no journal-10.9%+12.7%
Released ReplaySSM / our portXtra-1.2% / -5.7%+1.9% / +6.6%

Takeaway: the journal (every configuration slower without it) and tensor-core arithmetic (every configuration and repeat slower with IEEE dots) explain Xtra's advantage over exact FP32 replay; folding every round, eager recovery and the solver are within noise at this load (at layer level eager recovery costs 3.6% / 24.5% of lifecycle time for the two head geometries and the Neumann solver 6.2-8.1%). Higher concurrencies and 27B were not ablated.

AuthorsOpenReview ↗

General response (2/4)

4. Task accuracy.

Task accuracy on identical prompts, B200 greedy outputs, depth 7, concurrency 16, 160 prompts x 3 repeats; difference from the reference in points with 95% paired bootstrap interval (App. D.3, Table 17):

Task, modelReference score (%)XtraReleased ReplaySSMOur port
GSM8K, 4BReplaySSM 80.0-1.0 [-3.8, +1.2]ref.-1.2 [-3.1, 0.0]
GSM8K, 35B-A3BReplaySSM 83.3+1.7 [-1.0, +4.6]ref.+1.5 [-1.0, +4.2]
MATH-500, 4BReplaySSM 25.0+3.1 [-0.6, +7.5]ref.+2.3 [-1.5, +6.3]
MATH-500, 35B-A3BReplaySSM 23.3+2.7 [-2.1, +7.5]ref.+3.5 [-1.7, +8.8]
HumanEval, 4BSGLang 78.8-1.5 [-3.3, 0.0]0.0 [-2.5, +2.5]0.0 [-2.3, +2.3]
HumanEval, 35B-A3BSGLang 90.6-1.3 [-3.8, +1.3]0.0 [-3.1, +3.1]-0.6 [-1.9, 0.0]
HumanEval, 27BSGLang 88.8-0.2 [-2.3, +1.7]+0.6 [-1.3, +3.1]+0.6 [-1.9, +3.8]

Takeaway: no backend differs from its reference by more than about 3.5 points and every interval includes 0, although only 0-56% of outputs are token-identical. GSM8K and MATH-500 have no SGLang run, so released ReplaySSM is their reference; MATH-500 scores are low because of the 512-token cap and a fallback answer extraction; MT-Bench, Alpaca and LiveCodeBench were not scored.

5. Arithmetic and token agreement.

Arithmetic per arm, read from the kernel sources (Table 4); FP8 in a model name is the precision of the released weights, which every arm shares:

ArmCommitted stateDot products
SGLang, our portFP32FP32 element-wise on CUDA cores, no tensor cores
Released ReplaySSMBF16-rounded checkpointone BF16 tensor-core product, FP32 accumulation
XtraFP32 checkpointBF16x3 (three BF16 products, FP32 accumulation)

Takeaway: Xtra uses tensor cores like ReplaySSM but keeps an FP32 committed state.

Token agreement with SGLang's chain output, 4B, depth 3, concurrency 1, exact matches out of 160 prompts (App. D.6):

ArmHumanEval, B200MT-Bench, B200HumanEval, H100MT-Bench, H100
Xtra852785-
Released ReplaySSM85248334
Our port (exact FP32)7928160160

Takeaway: only our port is token-identical, and only on H100; Xtra and released ReplaySSM agree with SGLang at the level of SGLang's own beam2 against its chain.

AuthorsOpenReview ↗

General response (3/4)

6. Terminal coverage (C1) and the budget sweep. The final design verifies chains. The submission's one-token terminal alternatives (C1, H1 trees) are not part of it; that evidence is in App. C.4 and D.10. Sec. 3.1 now motivates the final design instead (Figs. 2-3: fold frequency from measured acceptance traces, state error and size).

Submitted implementation, submitted H100 sweep (App. D.10, Tables 21-22); throughput relative to the per-workload oracle:

Policy or quantityResult
H1 selected under 20/30/40 ms budgets2 of 54 cells (one met 30 ms by 0.018 ms in a single run)
Chain/H1 per (D, C) chosen from the other five workloadsidentical to always-chain; within 0-3.7% of the oracle
Always H1-4.0 to -15.6%
One fixed (D, C, topology) taken from five workloads0 to -22.3%; over the 20 ms budget on 1 held-out workload
Profiling cost of the sweep (estimate, 201 timed runs)about 150 H100 GPU-hours for 1,830 runs; H1 runs about 22

Takeaway: a profile from other workloads always picks chain, and the final design verifies chains, so no chain/H1 profiling remains.

Budget sweep, re-measured with the final design (Sec. 5.2-5.3, Figs. 6-8, Tables 6 and 9). Setting: the submission's 276 H100 configurations (three models, six workloads, depth 3/7, concurrency 1-128) run again with the final design; six-workload geometric mean of each method's best run under the budget, in token/s. Baselines are the submitted runs; for 4B (a changed budget winner) and 35B-A3B (drift +2.2%, changed budget winners), the latency budgets use the same-host re-run of each submitted winner configuration, kept even if it now misses the budget, which can only favour the baseline. The memory budgets use re-run winners too.

Latency budgetModelXtraSGLangOur portXtra vs SGLang / port
20 ms4B9,1785,9127,297+55.2% / +25.8%
20 ms27B1,8611,1341,486+64.2% / +25.2%
20 ms35B-A3B3,2192,3302,696+38.2% / +19.4%
30 ms4B9,4726,5187,934+45.3% / +19.4%
30 ms27B2,5391,5682,046+61.9% / +24.1%
30 ms35B-A3B4,7173,0193,845+56.2% / +22.7%
40 ms4B9,4726,5187,934+45.3% / +19.4%
40 ms27B2,6561,5682,302+69.4% / +15.4%
40 ms35B-A3B5,2873,0194,665+75.2% / +13.3%
Memory budgetModelXtraSGLangOur portXtra vs SGLang / port
16 GiB4B5,0372,7614,577+82.4% / +10.1%
24 GiB4B8,1014,2596,968+90.2% / +16.3%
32 GiB4B9,4725,3217,934+78.0% / +19.4%
40 GiB27B1,3334211,255+216.3% / +6.2%
48 GiB27B2,2081,0451,956+111.3% / +12.9%
64 GiB27B2,6561,5682,302+69.4% / +15.4%
40 GiB35B-A3B1,9348341,844+132.1% / +4.9%
48 GiB35B-A3B3,7931,9023,484+99.5% / +8.9%
64 GiB35B-A3B5,2873,0194,665+75.2% / +13.3%

Matched configurations (same model, task, depth and concurrency, same baselines): Xtra is 1.065x / 1.060x / 1.049x our port and 1.167x / 1.105x / 1.099x SGLang for 4B / 27B / 35B-A3B.

Takeaway: under the latency budgets Xtra is +38.2% to +75.2% faster than SGLang and +13.3% to +25.8% faster than our port; under the memory budgets +69.4% to +216.3% and +4.9% to +19.4%. Released ReplaySSM is not in these sweeps; end to end it is at parity with Xtra (item 1).

H1 selection stays in App. D.10, measured with the submitted implementation.

7. Scope and what was not measured (App. D.11).

  • All evidence is for GDN hybrids of one model family at tensor-parallel degree one, under greedy decoding. Sampling was not evaluated.
  • The comparison with released ReplaySSM covers 18 B200 and 12 H100 configurations at concurrency 16-128; 27B was not run on H100.
  • The budget sweeps do not include released ReplaySSM.
  • The ablation covers concurrency 16 on 4B and 35B-A3B only.

8. Material in the submission that may be easy to miss.

  • Latency budgets of 20, 30 and 40 ms: submission App. B.3, now Fig. 7 and Table 6 (final design), with the submitted implementation's bars in Figs. 13-15.
  • The process-memory-budget study: Sec. 5.3 as in the submission, Fig. 8 and Table 9 (final design).
  • Fitted throughput-latency curves: Fig. 6 (final design) and Fig. 9 (submitted implementation, other TPOT statistics), method in App. A.3.
AuthorsOpenReview ↗

General response (4/4)

Update to the revised PDF: we added two layer-level results. Nothing else changed, and all earlier section, table and appendix numbers are unchanged.

1. Porting the design to Kimi Delta Attention (new App. D.12, Table 23). KDA, the recurrence of Kimi Linear, has one decay gate per key channel instead of GDN's one per head.

  • Scope. Kimi-Linear-48B-A3B has no native MTP head, so this is a single-layer study, not a serving result.
  • Setup. One KDA layer with the released shapes and gate parameters, synthetic activations, three fresh processes per cell, on B200 and H100.
  • Baselines. Released ReplaySSM has no KDA kernel. The baselines are an exact FP32 replay and SGLang's own KDA verification, which keeps a full FP32 state per candidate.

What changed in the design:

  • Journal records carry the per-channel gate: 1,028 bytes per head and token, half of it the gate.
  • All exponents stay non-positive through relative factoring. Naive cumulative scaling would overflow or underflow FP32 in 6.6-55% of 32-token channel spans with the released gates.
  • The fold schedule adapts to batch. At batch 8 or less, and at depth 15, the journal is folded every round; otherwise the 16-record journal is used.

Setting: one KDA layer, depth 7, microseconds per round (depths 3 and 15 are in Table 23).

GPU, batchXtra-KDAExact FP32 replaySGLang KDA verifyXtra faster than replay / SGLang
B200, batch 114.818.122.61.23x / 1.53x
B200, batch 1642.876.898.41.79x / 2.30x
B200, batch 128307.5632.6872.12.06x / 2.84x
H100, batch 119.920.924.31.05x / 1.22x
H100, batch 1687.2108.6219.51.24x / 2.52x
H100, batch 128457.8828.01750.11.81x / 3.82x

Over all depths and batches:

  • Against SGLang's KDA verification, Xtra-KDA is faster in every cell except B200 depth 3 at batch 4 and 8, and H100 depth 3 at batch 2. At batch 16 and above it is 1.4-3.0x faster on B200 and 1.7-4.0x on H100.
  • Against the exact replay, it is faster in every cell at batch 16 and above on B200 (1.13-2.11x) and at batch 32 and above on H100 (1.31-1.81x). It is slower on B200 at depth 3, batch 4, and on H100 at depth 3, batch 16, and depth 15, batch 1-2.
  • Batch 1 is a clear win on B200 (1.19-1.23x over the replay). On H100 it is close to parity at depths 3 and 7 (1.04-1.05x), and 1.45x slower than the replay at depth 15.
  • State. Speculative state is 0.19-0.75 MiB per slot and layer: 1.5-4.3x more than the exact replay and 15-43x less than SGLang.
  • Error. Against FP64 over 2,048 rounds, output error is at most 3.4e-4 and committed-state error at most 1.9e-5, with no drift. Under a gate-stress test all values stay finite.

2. Small-batch check of the GDN kernel on H100 (App. D.1).

  • Unchanged kernel. At batch 1-16, Xtra's GDN layer kernel is 1.07-1.28x slower than released ReplaySSM on H100 (1.04-1.44x on B200).
  • Small-batch launch setting. It makes the kernel faster at batch 1 on both GPUs (1.09-1.34x on H100, 1.13-1.37x on B200), but not in most other cells. The serving build does not use this setting.
  • Scope of the parity claim. End-to-end parity with released ReplaySSM (Sec. 5.2) is measured at concurrency 16-128 only.

Takeaway: at layer level the design carries over to a per-channel-gated recurrence, with clear gains over SGLang's verification and the exact replay at batch 16 and above. At small batch the advantage shrinks or disappears, and none of this is a serving result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.