- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SpecRelay, an edge–cloud speculative decoding system for high and variable network RTTs. It combines asynchronous verification, continuation hedging, and lineage-preserving rebuilding to improve draft reuse while avoiding stale cloud requests. An RTT-conditioned router further selects between local inference modes and cloud speculation based on estimated task difficulty, output length, and hardware latency profiles. Experiments with Qwen3 and additional model pairs show up to 1.5× throughput improvement over the reproduced SpecEdge baseline, rising to 2.2× speedup with routing at matched aggregate target-model accuracy under high added RTT.
Critical strengths
-
The problem is critical and well motivated. Existing edge–cloud speculative decoding methods are much less attractive once network RTT reaches tens or hundreds of milliseconds. The paper directly addresses this practical limitation rather than treating communication as a small constant.
-
The ablation study is unusually thorough. The paper separates edge-kernel optimizations, continuation hedging, and asynchronous scheduling through S1, S3, A1, and A3. It additionally studies draft depth, tree budget, anchor count, latency jitter, a second edge GPU, other model pairs, sampling, and server-side cost.
-
The design is technically sensible. Prefix-gated verification, early stale-request rejection, continuation hedging, and lineage-preserving rebuilding form a coherent design. In-order processing and non-mutating rejection also provide a convincing argument that asynchronous execution preserves the target model’s greedy output.
Critical weaknesses
-
The evaluation lacks a competitive asynchronous edge–cloud baseline. The primary comparison is against SpecEdge, which uses a synchronous schedule. This establishes that asynchronous runahead can outperform a synchronous system, but it does not establish that SpecRelay is superior to the closest asynchronous or pipelined alternatives. The related-work section discusses PicoSpec, PipeSpec, FlowSpec and other runahead systems, yet none is evaluated empirically or adapted into a comparable baseline. Differences in tree versus chain drafting do not by themselves make these comparisons unnecessary. For example, a strong chain-based asynchronous baseline with matched edge compute, verification budget, and server utilization would help determine how much benefit specifically comes from multi-anchor and lineage preservation. Consequently, the empirical novelty relative to prior asynchronous speculation remains somewhat unclear.
-
The claimed “full-system” routing results are not measured end to end. The router results use per-query latencies generated from calibrated profiles and the output produced by each selected mode, rather than executing the routed workload end to end. The routed system’s server time is estimated by scaling A3’s per-token server time by the routed cloud-token share. It may overlooks several effects, mode switching, request setup, queuing, CPU processing, network-stack overhead may not compose linearly, contention between asynchronously arriving routed requests.
Questions for authors
-
Is implementing at least two competitive asynchronous chain-based baselines possible? Moreover, if possible, a tree-runahead baseline?
-
Is the router as a real end-to-end system and report measured latency distributions—not only averages—including p50, p95, and p99 latency, router overhead, queueing time, CPU processing, reconciliation, and network serialization available?
-
RTX 4090/3090 results alone do not fully substantiate the edge-device setting. Are metrics for lower-power edge hardware available, along with corresponding energy and memory measurements?
-
I believe all the figures in the paper should be vector graphics. It would be better if the figures could be replaced with their vector versions.
2 comments
W2 / Q2: End-to-end measurement of the routed system.
End-to-end measurement. We ran the routed system live on the paper's testbed (RTX 4090 edge, A100 server): the router decides each query at run time from the measured RTT, and the query runs in the chosen mode (Table 2).
- What the live run checks. The R column of the paper's Table 2 sums, per query, the time of the chosen mode: its measured speed at the query's actual output length, over the 6,637 SPROUT test queries. The live run measures the same quantities on 64 queries per RTT: the router chose the same mode as offline for 252 of the 256 queries, the edge modes ran at their profiled speed (median of measured over composed time: 0.90–0.99), and the measured speedup is within 7 % of the one composed for the same queries at their output lengths (Table 2).
- Latency of a query. Over the test set the router lowers the median latency at every RTT (by 15–47 %), p95 from 300 ms (by 10–20 %) and p99 at 500 ms (by 19 %); below these RTTs the tails differ by at most 6 %.
- Sample size. The edge reasoning modes sample their output, up to 16k tokens, and a few long answers set the total time of a few dozen queries: over sets of 64 test queries the composed speedup spans the range in Table 2, and the 64 live queries are one such set.
- Overheads. Queueing at the server, reconciliation and serialization each take at most 3.2 % of a cloud query's time.
Table 2. The routed system end to end (RTX 4090 edge, A100 server, SPROUT). Composed: as the R column of the paper's Table 2, for the live queries at their output lengths. Live speedup: 64 queries per RTT, full-length answers.
| added RTT (ms) | 0 | 100 | 300 | 500 |
|---|---|---|---|---|
| latency of a query (s), p50 / p95 / p99 (6,637 test queries, composed) | ||||
| R | 8.2 / 30.0 / 67.1 | 10.2 / 41.5 / 99.0 | 14.8 / 61.5 / 161.6 | 17.6 / 76.2 / 184.9 |
| every query on Spec-Cloud | 9.7 / 28.2 / 67.0 | 14.4 / 41.6 / 98.8 | 23.7 / 68.2 / 161.9 | 33.3 / 95.7 / 227.0 |
| total time, every query on Spec-Cloud ÷ R | ||||
| 6,637 test queries (the paper's Table 2) | 1.05× | 1.14× | 1.33× | 1.52× |
| sets of 64 queries, lowest–highest | 0.75–1.29× | 0.89–1.54× | 1.03–1.71× | 1.23–1.98× |
| 64 live queries, composed | 1.07× | 1.14× | 1.12× | 1.30× |
| 64 live queries, measured | 1.14× | 1.19× | 1.14× | 1.27× |
| per verification request of R (ms), p50 / p95 / p99 | ||||
| queue wait at the server | 0.24 / 0.32 / 2.7 | 0.23 / 2.0 / 8.8 | 0.22 / 0.52 / 6.2 | 0.23 / 0.52 / 2.5 |
| reconciliation on the edge | 2.1 / 10.0 / 11.2 | 1.9 / 10.7 / 12.5 | 1.8 / 11.4 / 13.3 | 2.0 / 12.1 / 15.1 |
| serialization | 0.16 / 0.90 / 2.1 | 0.17 / 0.50 / 1.9 | 0.17 / 0.54 / 2.5 | 0.17 / 0.56 / 2.8 |
| share of a cloud query's time (%) | ||||
| server queue / reconciliation / serialization | 0.42 / 3.2 / 0.31 | 0.45 / 0.99 / 0.20 | 0.30 / 0.41 / 0.15 | 0.12 / 0.20 / 0.10 |
Router overhead. The 0.6B routing model runs on the edge while the edge prefills the query, so most of its time is hidden (Table 3): it adds 2.9 ms before generation at the median, 0.02 % of the end-to-end time.
Table 3. Cost of the 0.6B routing model on the edge (RTX 4090 edge, A100 server; 128 routed SPROUT queries at 0 / 100 / 300 / 500 ms added RTT).
| per query | p50 | p95 | p99 |
|---|---|---|---|
| router time, host (ms) | 1.6 | 3.6 | 4.9 |
| router CPU time (ms) | 1.5 | 3.4 | 4.7 |
| router GPU time, alongside the edge prefill (ms) | 10.5 | 23.0 | 37.1 |
| added before generation, routing minus the edge prefill (ms) | 2.9 | 4.6 | 6.2 |
| value | |
|---|---|
| added time, share of the end-to-end time | 0.02 % |
| energy per decision, the router alone (J) | 2.1 |
| edge GPU memory vs. always-cloud, allocated / peak reserved (GB) | +1.3 / +1.8 |
| one-time load at client start | |
| ↳ time (s) | 3.4–3.9 |
| ↳ energy (kJ) | 0.36–0.41 |
| ↳ host memory (GB) | +1.5 |
Q3: Lower-power edge hardware, energy and memory.
We extended the evaluation to a third, slower edge GPU, an NVIDIA A30, with energy and memory measurements (Table 4). On this slower GPU the gain of A3 over Ref is as large as on the RTX 4090 (1.28–1.51× against 1.18–1.48× at 0, 50, 100 and 300 ms added RTT). Most of the energy and memory difference to Ref comes with the edge draft optimizations of S1. For the camera-ready version we will add a Jetson-class edge device.
Table 4. Edge GPU energy and memory (one client, A30 edge, SPROUT).
| Ref | S1 | A3 | |
|---|---|---|---|
| energy per committed token (J), 50 ms added RTT | 3.96 | 2.73 | 2.72 |
| energy per committed token (J), 300 ms added RTT | 7.32 | 5.30 | 5.07 |
| peak GPU memory (MiB) | 4,614 | 4,869 | 4,927 |
Q4: Vector figures.
All figures are vector graphics in the revised paper.
We thank the reviewer for the insightful and constructive review. The additional experiments and the updated content described below are in the revised paper, marked in blue.
Changes to the system in the revision. The submitted system was evaluated with one client per server on a dedicated edge GPU. We extended it to support a multi-client server and a shared edge GPU:
With one client these leave the paper's results as they are (the paper's Table 2 on SPROUT, re-measured at five RTTs: A3 −1.0 to +1.7 %; Ref, S1, A1 −4.5 to +0.7 %).
The paper's Table 2 lists its configurations one mechanism apart to show what each contributes: A1 is the tree-runahead baseline (an asynchronous tree draft under one anchor), and A3 adds multi-anchor hedging with the lineage-preserving rebuild. To these we added the two chain-draft asynchronous baselines that the review asks for (AC^fix, AC^ad).
Table 1. Baselines and A3 inside our system (one NVIDIA A30 per client, SPROUT).
All outputs match the target model token by token. Verified requests per committed token, one / eight clients: AC^fix 0.29 / 0.29; AC^ad 0.23–0.24 / 0.18–0.20; A1 0.18–0.20 / 0.14–0.15; A3 0.18–0.19 / 0.14–0.15.