acceptodds
Desk rejected at ICLR 2027

SpecRelay: Amortize or Avoid the Round Trip in Edge–Cloud Speculative Decoding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Edge–cloud speculative decoding moves autoregressive drafting to an edge device while a larger cloud model verifies draft tokens in parallel, so that edge drafting and cloud verification overlap under a synchronous draft–verify schedule. Beyond low-latency local networks, however, the next verification request of that schedule can be sent only after the previous reply arrives, so every round still waits out a full round trip, and throughput falls as the network RTT grows. To recover this loss, we present SpecRelay, an edge–cloud LLM inference system that addresses this cost at two levels: it amortizes round trips when cloud verification is necessary and avoids them when local reasoning suffices. For cloud-bound queries, an asynchronous verification pipeline keeps fixed-budget draft trees in flight while earlier replies are still outstanding, and the cloud rejects stale requests before the target model forward pass (the Async pipeline). At the query level, an RTT-conditioned router (the Latency-Aware Router) estimates quality and completion latency for three on-device modes (direct generation, Chain-of-Draft, and full reasoning) and cloud speculative decoding, selecting among them under a configurable accuracy– latency objective without adding latency during inference, since the router runs concurrently with the draft-model prefill. The Async pipeline alone improves decoding throughput by up to 50% over a reproduced synchronous baseline across multiple benchmarks. At matched target-model accuracy and 250 ms of added round-trip latency, the full system with the router delivers 1.95× and 1.55× the throughput of the same baseline on SPROUT and OlympiadBench, respectively, while sending only 47% and 39% of the queries to the cloud.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, LLM Inference and Serving, Large Language Models, Efficient ML, Edge–Cloud Computing, Scalability of ML Systems, Reasoning & Test-Time Compute.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZrCNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SpecRelay, an edge–cloud speculative decoding system for high and variable network RTTs. It combines asynchronous verification, continuation hedging, and lineage-preserving rebuilding to improve draft reuse while avoiding stale cloud requests. An RTT-conditioned router further selects between local inference modes and cloud speculation based on estimated task difficulty, output length, and hardware latency profiles. Experiments with Qwen3 and additional model pairs show up to 1.5× throughput improvement over the reproduced SpecEdge baseline, rising to 2.2× speedup with routing at matched aggregate target-model accuracy under high added RTT.

Critical strengths
  1. The problem is critical and well motivated. Existing edge–cloud speculative decoding methods are much less attractive once network RTT reaches tens or hundreds of milliseconds. The paper directly addresses this practical limitation rather than treating communication as a small constant.

  2. The ablation study is unusually thorough. The paper separates edge-kernel optimizations, continuation hedging, and asynchronous scheduling through S1, S3, A1, and A3. It additionally studies draft depth, tree budget, anchor count, latency jitter, a second edge GPU, other model pairs, sampling, and server-side cost.

  3. The design is technically sensible. Prefix-gated verification, early stale-request rejection, continuation hedging, and lineage-preserving rebuilding form a coherent design. In-order processing and non-mutating rejection also provide a convincing argument that asynchronous execution preserves the target model’s greedy output.

Critical weaknesses
  1. The evaluation lacks a competitive asynchronous edge–cloud baseline. The primary comparison is against SpecEdge, which uses a synchronous schedule. This establishes that asynchronous runahead can outperform a synchronous system, but it does not establish that SpecRelay is superior to the closest asynchronous or pipelined alternatives. The related-work section discusses PicoSpec, PipeSpec, FlowSpec and other runahead systems, yet none is evaluated empirically or adapted into a comparable baseline. Differences in tree versus chain drafting do not by themselves make these comparisons unnecessary. For example, a strong chain-based asynchronous baseline with matched edge compute, verification budget, and server utilization would help determine how much benefit specifically comes from multi-anchor and lineage preservation. Consequently, the empirical novelty relative to prior asynchronous speculation remains somewhat unclear.

  2. The claimed “full-system” routing results are not measured end to end. The router results use per-query latencies generated from calibrated profiles and the output produced by each selected mode, rather than executing the routed workload end to end. The routed system’s server time is estimated by scaling A3’s per-token server time by the routed cloud-token share. It may overlooks several effects, mode switching, request setup, queuing, CPU processing, network-stack overhead may not compose linearly, contention between asynchronously arriving routed requests.

Questions for authors
  1. Is implementing at least two competitive asynchronous chain-based baselines possible? Moreover, if possible, a tree-runahead baseline?

  2. Is the router as a real end-to-end system and report measured latency distributions—not only averages—including p50, p95, and p99 latency, router overhead, queueing time, CPU processing, reconciliation, and network serialization available?

  3. RTX 4090/3090 results alone do not fully substantiate the edge-device setting. Are metrics for lower-power edge hardware available, along with corresponding energy and memory measurements?

  4. I believe all the figures in the paper should be vector graphics. It would be better if the figures could be replaced with their vector versions.

2 comments

We thank the reviewer for the insightful and constructive review. The additional experiments and the updated content described below are in the revised paper, marked in blue.

Changes to the system in the revision. The submitted system was evaluated with one client per server on a dedicated edge GPU. We extended it to support a multi-client server and a shared edge GPU:

  • Batched verification: the server verifies the requests of several clients in one forward pass.
  • Load-adaptive draft depth (A1, A3; 5 or 7 in the submission): the edge sets the depth from the measured acceptance of its drafts and adds the levels it can draft during the server's queue wait, which the server reports with each verdict (formula: revised appendix).
  • Router inputs: the response-time estimate includes the server's queue wait, and, in an offline evaluation, the edge's measured speed in place of its profile.

With one client these leave the paper's results as they are (the paper's Table 2 on SPROUT, re-measured at five RTTs: A3 −1.0 to +1.7 %; Ref, S1, A1 −4.5 to +0.7 %).


W1 / Q1: Competitive asynchronous baselines.

The paper's Table 2 lists its configurations one mechanism apart to show what each contributes: A1 is the tree-runahead baseline (an asynchronous tree draft under one anchor), and A3 adds multi-anchor hedging with the lineage-preserving rebuild. To these we added the two chain-draft asynchronous baselines that the review asks for (AC^fix, AC^ad).

  • Chain baselines. A chain sends a width-1 draft of gg tokens per request ahead of the verdict, with gg the best of a one-client sweep per RTT (AC^fix) or the load-adaptive draft depth of A1 and A3 (AC^ad).
  • Why inside our system. None of the systems the review names can be run across our round trip as released: PipeSpec keeps its models on one host with shared memory (its code, run with our models, reaches at best 1.10× the speed of its own target model alone), PicoSpec has no public code, and FlowSpec partitions the target model itself. We therefore implemented their chain schedule in our edge–cloud system, where the edge and the server share no KV cache: every row of Table 1 runs across the same round trip, on one engine, with the same models, edge GPU and server, so that the difference between two rows (chain draft, tree draft, hedging with the rebuild) is the effect of that mechanism.

Table 1. Baselines and A3 inside our system (one NVIDIA A30 per client, SPROUT).

tokens/s per client, by added RTT (ms)0 (one client)50 (one client)100 (one client)300 (one client)50 (eight clients)100 (eight clients)300 (eight clients)
Ref (synchronous)39.931.024.915.627.323.815.1
AC^fix (chain, fixed length)43.833.426.815.213.612.610.6
AC^ad (chain, adaptive length)45.633.627.015.522.920.614.1
A1 (tree run-ahead, one anchor)48.739.533.320.628.325.918.4
A3 (tree, K=3, lineage rebuild)51.043.637.423.629.227.720.3
A3 relative to A1+4.7 %+10.4 %+12.3 %+14.6 %+3.2 %+6.9 %+10.3 %
A3 relative to the better of AC^fix, AC^ad+11.8 %+29.8 %+38.5 %+52.3 %+27.5 %+34.5 %+44.0 %

All outputs match the target model token by token. Verified requests per committed token, one / eight clients: AC^fix 0.29 / 0.29; AC^ad 0.23–0.24 / 0.18–0.20; A1 0.18–0.20 / 0.14–0.15; A3 0.18–0.19 / 0.14–0.15.

  • Chain against tree. With one client AC^fix and AC^ad gain 8–14 % over the synchronous Ref up to 100 ms and nothing at 300 ms, and they need more verifications per committed token; with eight clients sharing one server that cost makes them slower than Ref. A chain request carries one continuation, so a verification commits fewer tokens, and the first mismatch discards every chain in flight.
  • Tree draft and hedging. The tree draft makes the asynchronous pipeline faster (A1), and hedging with the lineage-preserving rebuild (A3) adds to it at about the same number of verifications, because it keeps the work drafted during the round trip: 63–67 % of the verdicts land on an already drafted continuation, against 52–54 % with one anchor (RTX 4090 edge). Keeping that work across a network, where the KV caches of the edge and the server are never at the same point, is what the gate, anchor-relative coordinates and the lineage-preserving rebuild (§4) provide; a chain has no such state to keep.

W2 / Q2: End-to-end measurement of the routed system.

End-to-end measurement. We ran the routed system live on the paper's testbed (RTX 4090 edge, A100 server): the router decides each query at run time from the measured RTT, and the query runs in the chosen mode (Table 2).

  • What the live run checks. The R column of the paper's Table 2 sums, per query, the time of the chosen mode: its measured speed at the query's actual output length, over the 6,637 SPROUT test queries. The live run measures the same quantities on 64 queries per RTT: the router chose the same mode as offline for 252 of the 256 queries, the edge modes ran at their profiled speed (median of measured over composed time: 0.90–0.99), and the measured speedup is within 7 % of the one composed for the same queries at their output lengths (Table 2).
  • Latency of a query. Over the test set the router lowers the median latency at every RTT (by 15–47 %), p95 from 300 ms (by 10–20 %) and p99 at 500 ms (by 19 %); below these RTTs the tails differ by at most 6 %.
  • Sample size. The edge reasoning modes sample their output, up to 16k tokens, and a few long answers set the total time of a few dozen queries: over sets of 64 test queries the composed speedup spans the range in Table 2, and the 64 live queries are one such set.
  • Overheads. Queueing at the server, reconciliation and serialization each take at most 3.2 % of a cloud query's time.

Table 2. The routed system end to end (RTX 4090 edge, A100 server, SPROUT). Composed: as the R column of the paper's Table 2, for the live queries at their output lengths. Live speedup: 64 queries per RTT, full-length answers.

added RTT (ms)0100300500
latency of a query (s), p50 / p95 / p99 (6,637 test queries, composed)
R8.2 / 30.0 / 67.110.2 / 41.5 / 99.014.8 / 61.5 / 161.617.6 / 76.2 / 184.9
every query on Spec-Cloud9.7 / 28.2 / 67.014.4 / 41.6 / 98.823.7 / 68.2 / 161.933.3 / 95.7 / 227.0
total time, every query on Spec-Cloud ÷ R
6,637 test queries (the paper's Table 2)1.05×1.14×1.33×1.52×
sets of 64 queries, lowest–highest0.75–1.29×0.89–1.54×1.03–1.71×1.23–1.98×
64 live queries, composed1.07×1.14×1.12×1.30×
64 live queries, measured1.14×1.19×1.14×1.27×
per verification request of R (ms), p50 / p95 / p99
queue wait at the server0.24 / 0.32 / 2.70.23 / 2.0 / 8.80.22 / 0.52 / 6.20.23 / 0.52 / 2.5
reconciliation on the edge2.1 / 10.0 / 11.21.9 / 10.7 / 12.51.8 / 11.4 / 13.32.0 / 12.1 / 15.1
serialization0.16 / 0.90 / 2.10.17 / 0.50 / 1.90.17 / 0.54 / 2.50.17 / 0.56 / 2.8
share of a cloud query's time (%)
server queue / reconciliation / serialization0.42 / 3.2 / 0.310.45 / 0.99 / 0.200.30 / 0.41 / 0.150.12 / 0.20 / 0.10

Router overhead. The 0.6B routing model runs on the edge while the edge prefills the query, so most of its time is hidden (Table 3): it adds 2.9 ms before generation at the median, 0.02 % of the end-to-end time.

Table 3. Cost of the 0.6B routing model on the edge (RTX 4090 edge, A100 server; 128 routed SPROUT queries at 0 / 100 / 300 / 500 ms added RTT).

per queryp50p95p99
router time, host (ms)1.63.64.9
router CPU time (ms)1.53.44.7
router GPU time, alongside the edge prefill (ms)10.523.037.1
added before generation, routing minus the edge prefill (ms)2.94.66.2
value
added time, share of the end-to-end time0.02 %
energy per decision, the router alone (J)2.1
edge GPU memory vs. always-cloud, allocated / peak reserved (GB)+1.3 / +1.8
one-time load at client start
↳ time (s)3.4–3.9
↳ energy (kJ)0.36–0.41
↳ host memory (GB)+1.5

Q3: Lower-power edge hardware, energy and memory.

We extended the evaluation to a third, slower edge GPU, an NVIDIA A30, with energy and memory measurements (Table 4). On this slower GPU the gain of A3 over Ref is as large as on the RTX 4090 (1.28–1.51× against 1.18–1.48× at 0, 50, 100 and 300 ms added RTT). Most of the energy and memory difference to Ref comes with the edge draft optimizations of S1. For the camera-ready version we will add a Jetson-class edge device.

Table 4. Edge GPU energy and memory (one client, A30 edge, SPROUT).

RefS1A3
energy per committed token (J), 50 ms added RTT3.962.732.72
energy per committed token (J), 300 ms added RTT7.325.305.07
peak GPU memory (MiB)4,6144,8694,927

Q4: Vector figures.

All figures are vector graphics in the revised paper.

Reviewer_BnATOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper combines asynchronous verification, multi-anchor hedging, and RTT-aware routing for edge-cloud speculative decoding. The considered problem is relevant but I have concerns about scalability and computation overhead of the proposed solution, as well as fairness of the baseline comparisons.

Critical strengths

-The paper addresses RTT-induced performance degradation in edge–cloud speculative decoding, an important challenge for distributed LLM inference over high-latency networks.

-In controlled simulation setups the proposed method demonstrate considerable throughput improvements.

Critical weaknesses

-The framework primarily combines existing asynchronous pipelining, speculative hedging, and LLM routing techniques, with insufficient fundamental algorithmic innovation.

-Reliance on artificially injected RTTs, without comprehensive real-world wireless traces, packet loss, or congestion experiments, limits practical applicability.

-Experiments predominantly involve a single edge client, leaving server contention, queueing delays, and resource overhead under large-scale concurrent deployments unexplored.

-The proposed asynchronous verification protocol lacks rigorous correctness guarantees under stochastic decoding.

Questions for authors

1-How does SpecRelay guarantee correctness and liveness under stochastic decoding, particularly given the execution stalls reported at temperature 0.7?

2-How does the asynchronous pipeline perform with hundreds of concurrent edge clients? How will server queueing and contention eliminate the reported speedups? Scalability should be studied or simulated.

3-How much of the reported 2.2× speedup comes from the proposed asynchronous protocol versus differences in reasoning capabilities between local and cloud models? Would the gains persist if both used equivalent reasoning policies?

4-How sensitive are routing decisions to inaccurate or outdated RTT estimates, particularly when RTT changes significantly during query execution? How is the current RTT estimated in practice? Is it based on instantaneous measurements, an exponentially weighted moving average, or another prediction method?

5-What is the peak KV-cache and CPU overhead of maintaining multiple in-flight trees plus lineage-preserving rebuild on a memory-constrained edge device relative to a synchronous client?

6-How do time-varying RTT and jitter affect P95/P99 query completion latency, rather than only average throughput?

7-The paper attributes gains to td, α, and o, but never reports measured acceptance, overlap, or draft time per variant and RTT. Eq. (1) also ignores async’s extra verification work, so how can we verify the claim “o is lowered”?

5 comments

We thank the reviewer for the insightful and constructive review. The additional experiments and the updated content described below are in the revised paper, marked in blue.


W1: Novelty beyond existing asynchronous pipelining, hedging and routing.

Asynchronous verification, hedging and routing each have precedents, and the paper builds on them. Used directly across a network, however, the first two gain little: on SPROUT, averaged over 50–300 ms added RTT, hedging alone adds 1 % (S3 over S1) and asynchronous verification alone 6 % (A1 over S1), against 26 % for the two under our protocol (A3 over S1; the paper's Table 2). The contribution therefore lies not in a single component but in what it takes to make them work across a network round trip, which raises two problems: (i) to hide the round trip the edge keeps drafting, so a verdict arrives only after several further requests have been drafted on bets about it; (ii) the round trip is a cost that some queries are better off not paying. We solve them with a protocol that keeps those requests exact and usable (§4) and a router that weighs the round trip against on-device reasoning (§5). (i) and (ii) below describe the two; the comparison with asynchronous chain baselines follows in the next comment.

(i) Lineage-preserving asynchronous tree drafting across a network with no shared KV state (§4). In PEARL, PipeSpec and SwiftSpec the draft and target models share memory on one host. Over a network, sharing it would make decoding bandwidth-bound as well as latency-bound: the KV state of one token position is about 0.1 MB on the edge against a few bytes for its token ID. Only token IDs and positions are therefore exchanged (§4.1), and the two KV caches are never at the same point: the edge's cache holds drafts the server has not verified yet, and the server's committed prefix reaches the edge one round trip later. A verdict arrives while later requests are already in flight, each drafted under anchors that it may not confirm (Table 1). To our knowledge no prior system runs tree drafts asynchronously in this setting (PicoSpec drafts a short chain; FlowSpec partitions the target model itself).

Three mechanisms handle this:

  • The gate: the server verifies only from its own committed prefix, so the output stays exact, and rejects stale requests before any forward: no server waste.
  • Anchor-relative coordinates: a verdict hits when the accepted path ends at an anchor and the bonus token is among the children drafted under it (§4.2); the trees drafted under that anchor are kept and no draft forward is repeated (Fig. 3c of the paper).
  • The lineage-preserving rebuild: when the accepted path ends before an anchor, the edge resumes from the drafts that continue it instead of restarting from the verified prefix (Fig. 3d).

With one anchor (A1) 52–54 % of the verdicts are hits, with three (A3) 63–67 % (Table 1).

Table 1. Requests in flight and hits under A3 (RTX 4090 edge, SPROUT, one client).

A3, by added RTT (ms)0100300500
requests in flight at a verdict, mean / max1.5 / 32.8 / 65.7 / 138.5 / 17
verdicts that are hits62.9 %65.2 %67.0 %66.6 %
hits through the second or third anchor14.7 %17.2 %18.4 %18.0 %

(ii) A router that decides per query from the measured response time of the cloud path and the query itself (§5). It measures that response time online (the network round trip and, in the revision, the server's queue wait) and predicts, for each mode, the correctness and the output length on that query, so it chooses jointly where a query runs and how much the edge reasons. Network-aware routers exist, but they choose which model serves a query; we are not aware of one that weighs the on-device reasoning effort against the round trip to the cloud.

The speedup of the full system over Ref has four steps (Table 7 under Q3):

  • Edge draft optimizations (Ref → S1) shorten every draft forward; engineering that we do not claim.
  • Asynchronous verification alone (S1 → A1) gains mostly at long round trips (4–13 % from 100 ms, level with S1 up to 50 ms): the trees drafted during a round trip follow one predicted continuation and are discarded when the verdict leaves it, about half the time.
  • Continuation hedging with the lineage-preserving rebuild (A1 → A3) is the largest protocol step because a verdict then redirects the drafting instead of resetting it: the edge carries on from the drafted continuation that the verdict confirms. The gain grows with the RTT, as more requests are in flight.
  • Routing (A3 → R) grows with the RTT because every query answered on the edge saves its cloud round trips (Q3).

The revision states the protocol as an algorithm (the server's gate-and-verify step and the edge's rebuild on a verdict) and proves its exactness and progress from it (W4).

W1 (continued): Existing asynchronous pipelining in our setting.

To see what asynchronous pipelining alone provides in our setting, we implemented the chain schedule of PipeSpec and PicoSpec in our system with a fixed (AC^fix) and a load-adaptive (AC^ad) chain length, so that only the schedule differs (Table 2; the A30 results use the revised system, see W3). With one client AC^fix and AC^ad gain 8–14 % over the synchronous Ref up to 100 ms and nothing at 300 ms; with eight clients they are slower than Ref (Table 4: AC^ad): a chain request carries a single continuation, so one verification commits fewer tokens than with a tree draft, and the first mismatch discards every chain in flight.

Table 2. Chain-draft baselines in our system: tokens/s (one client, A30 edge, SPROUT); all rows but Ref are asynchronous.

variant, by added RTT (ms)050100300
Ref (synchronous, tree)39.931.024.915.6
AC^fix (chain, fixed length)43.833.426.815.2
AC^ad (chain, load-adaptive length)45.633.627.015.5
A1 (tree, one anchor)48.739.533.320.6
A3 (tree, K=3, lineage rebuild)51.043.637.423.6
R (A3 with the router)53.647.843.331.5

AC^fix: best length of a sweep per RTT. Verifications per committed token: 0.23–0.29 (AC^fix, AC^ad), 0.18–0.20 (A1, A3). R: composed as the paper's R column, A3's tokens/s times the router's speedup over A3 at matched accuracy.


W2 / Q6: Real traces, packet loss, congestion; P95 / P99.

To test A3 under real network conditions, we replaced the constant added delay by RTT traces of commercial mobile networks (Table 3): 5G, stationary and driving (Raca et al., MMSys 2020), and 5G from a car moving between strong and weak coverage (CICV5G, Scientific Data 2026). A3 stays ahead of Ref on every trace; with eight clients the server's queue takes part of the gain, as at a constant added RTT (W3). Measured live on the same traces, the routed system R is about as fast as A3 with one client (0.95–1.03×) and 1.09–1.10× faster with eight clients, where it reads the server's queue as a longer round trip and answers more queries on the edge.

  • Packet loss. The last 11 minutes of the driving recording contain 12 outages in which no ping was answered: the link is down a quarter of the time, for up to 17 s. We replayed this part with its outages (a message sent during an outage arrives when the link answers again, as over TCP): A3 stays exact, and its lead over Ref is larger than on the other traces (row “weak coverage”).

  • Congestion on the link appears in these traces as delay spikes (up to 2.5 s on the car trace), and congestion at the server is the eight-client experiment of W3.

  • P95 / P99. The completion latency of a query is lower for A3 than for Ref at p50, p95 and p99, with eight clients and through the outages (lower block). An outage stalls a query whichever variant runs it: nothing can be committed without a verdict, and A3 sends a few further requests and then waits as Ref does, so through the outages p99 roughly doubles for both (rows with eight clients). A3 stays lower because it commits tokens faster while the link answers, so a query is exposed to the link for less time. Under the paper's jitter models, the p95 per-query slowdown of A3 is below that of the synchronous S1 in 35 of 36 settings.

Table 3. RTT traces replayed in place of a constant added RTT (one NVIDIA A30 per client, SPROUT).

trace (clients)RTT p50 / p95 / p99 (ms)RefA3RA3 vs. Ref
tokens/s per client
CICV5G: car, strong / weak coverage (1)19 / 33 / 28933.346.446.8+39.3 %
Raca: stationary (1)73 / 75 / 7526.840.541.7+51.1 %
Raca: driving (1)73 / 105 / 33026.439.240.1+48.5 %
Raca: driving (8)79 / 105 / 33024.328.030.6+15.2 %
Raca: driving, weak coverage (1)135 / 377 / 57214.524.222.9+66.9 %
Raca: driving, weak coverage (8)135 / 377 / 57214.118.019.8+27.7 %
completion latency of a query (start to last token), p50 / p95 / p99 (s)
Raca: driving (8)79 / 105 / 33023.5 / 45.4 / 53.418.8 / 45.0 / 47.5
Raca: driving, weak coverage (1)135 / 377 / 57231.4 / 89.6 / 102.824.9 / 56.4 / 58.2
Raca: driving, weak coverage (8)135 / 377 / 57234.3 / 92.6 / 115.627.5 / 78.1 / 93.1

RTT: over the part of each recording that the runs replayed (its first 9–13 minutes; weak coverage: the last 11 minutes of the driving recording in a loop, answered pings only). Ref uses the paper's depth rule at the median RTT of each trace; A3 has no RTT-dependent setting. R: A3's tokens/s times the speedup of the routed system over the same clients sending every query to Spec-Cloud, measured on the same trace (32 queries per client, full-length answers).

W3 / Q2: Shared server: contention, queueing, resource overhead; hundreds of clients.

Changes to the system in the revision. The submitted system was evaluated with one client per server on a dedicated edge GPU. We extended it to support a multi-client server and a shared edge GPU:

  • Batched verification: the server verifies the requests of several clients in one forward pass.
  • Load-adaptive draft depth (A1, A3; 5 or 7 in the submission): the edge sets the depth from the measured acceptance of its drafts and adds the levels it can draft during the server's queue wait, which the server reports with each verdict (formula: revised appendix).
  • Router inputs: the response-time estimate includes the server's queue wait, and, in an offline evaluation, the edge's measured speed in place of its profile.

With one client these leave the paper's results as they are (the paper's Table 2 on SPROUT, re-measured at five RTTs: A3 −1.0 to +1.7 %; Ref, S1, A1 −4.5 to +0.7 %).

With the first two we ran eight clients, extending the paper's two-client pilot (Table 4).

  • A3 keeps its lead at both ends of the load: with an idle server (one client, Table 2) and with eight clients, whose queue adds 47–68 ms to every verdict, it stays above Ref at every added RTT. Relative to S3^until (the paper's synchronous variant that drafts until the reply arrives), A3 gains as the round trip grows (last row of the table): a single proactive tree cannot use a long wait, whereas A3 uses it for further trees.

  • Hundreds of clients. Servers are independent replicas, so the load per server decides: at these per-client rates 200 A3 clients need 25 such servers (arithmetic on the measured rates).

  • Resource overhead. With eight clients A3 uses no more of the server than Ref (Table 5): the gate rejects its stale requests before any forward, and the verified ones share forward passes, so it needs fewer passes and less target-model GPU time per committed token; the server's peak memory is the same. On the edge it keeps the GPU as busy as Ref (84 % against 83 % at 50 ms).

Table 4. Eight clients on one A100 server: tokens/s summed over the clients (one A30 each, SPROUT).

variant, by added RTT (ms)50100300
Ref (the paper's draft depth)216.7186.9118.3
AC^ad (chain baseline of W1)178.6162.8111.0
A1222.3202.7145.7
S3^until244.6212.5133.3
A3231.4218.2158.7
A3 vs. Ref+6.8 %+16.7 %+34.2 %
A3 vs. S3^until−5.4 %+2.7 %+19.1 %

A3 at 50 ms: mean of two runs (234.3 and 228.5).

Table 5. The server with eight clients, Ref / A3 (per query unless noted; SPROUT).

added RTT (ms)50300
requests arriving80.1 / 128.362.7 / 223.6
↳ stale, rejected before a forward0 / 47.40 / 138.1
forward passes of the target model31.6 / 23.053.3 / 42.8
requests verified per forward pass2.54 / 3.521.18 / 2.00
target GPU time per committed token (ms)3.20 / 2.714.11 / 3.89
server peak memory (GB)36.6 / 36.636.6 / 36.6

Per query: the server's totals divided by the number of queries of all clients; one forward pass verifies the requests of several clients.


Q5: KV-cache and CPU overhead relative to a synchronous client.

  • Memory. A3's allocated peak GPU memory on the edge is 0.3 GiB above Ref's, and the difference comes with the edge draft optimizations of S1, not with the trees in flight: S1 itself peaks at 4,869 MiB against 4,927 MiB for A3. The peak does not grow with the requests in flight, since the edge preallocates one KV buffer per query and all trees live inside it.

  • CPU. A3 uses the same CPU pinning as S1. Its CPU overhead is the bookkeeping, applying verdicts and building requests: 2.3–2.9 ms of CPU time per committed token (7–10 % of the 22–42 ms that A3 takes per token), where the synchronous clients spend at most 0.5–0.8 ms on the same work (their wall time for it; column “verdicts and requests” of Table 6, which splits the edge's wall time). In return A3 spends less time on rejected drafts than Ref and less time waiting than S1 (same draft engine), which is idle once its proactive tree is complete. No draft forward is repeated.

Table 6. Edge wall time per committed token, in ms (one client, A30 edge, SPROUT).

added RTT (ms)variantaccepted draftsrejected draftsverdicts and requestswaiting, othertotal
50Ref16.215.00.50.031.6
S110.29.50.63.724.0
S3^until10.711.20.60.222.7
A19.511.91.81.624.8
A39.58.92.41.722.4
300Ref16.139.00.57.463.0
S19.924.50.619.254.3
S3^until10.427.30.815.754.2
A19.331.82.93.947.9
A310.024.13.83.941.7
Reviewer_5sPwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an asynchronous cloud verification pipeline with anchor point hedging: the paper introduces a prefix gated asynchronous verification protocol that allows multiple fixed budget draft trees to continuously fly in parallel in the network (In flight). By discarding old requests on the server before the target model is propagated forward, the limitation of waiting for the complete RTT in each round in synchronous scheduling is eliminated. Design a reconstruction mechanism for continuous hedging and maintaining blood relationships: At the junction of trees, the edge crosses K most likely continuation anchor points for hedging growth, and after receiving the server's judgment, continues the draft directly from the surviving blood relationships through "line preserving rebuild", thereby improving the acceptance probability without increasing cloud verification costs. Test time Compute routing mechanism for RTT perception: A lightweight per query router is proposed, which combines frozen embedded encoders and non parametric regressors with current RTT estimates to jointly select between three end-to-end inference modes (direct generation, speculative draft CoD, and complete inference) and cloud based speculative decoding. The forward computation of the encoder can overlap with the pre filling of the draft model.

Critical strengths

The paper first proposes an asynchronous cloud verification pipeline with anchor point hedging, and introduces a prefix gated asynchronous verification protocol that allows multiple fixed budget draft trees to continuously fly in parallel in the network (In flight), discarding stale requests before the target model is propagated forward by the server. At the same time, a reconstruction mechanism is designed for continuous hedging and maintaining blood relationships. At the junction of trees, the edge end crosses K most likely continuation anchor points for hedging growth, and after receiving the server's judgment, continues the draft directly from the surviving blood relationship through "line preserving rebuild", thereby improving the acceptance probability without increasing cloud verification costs

Critical weaknesses

1、Unrealistic Edge Resource Assumptions: The evaluation assumes that the edge device (RTX 4090/3090) is exclusively dedicated to the inference task. However, in real-world edge or mobile environments, users concurrently run graphical user interfaces, background services, or other AI applications. Severe contention for GPU memory bandwidth, cache, and compute resources between the edge models (e.g., the 1.7B draft model and 0.6B router) and resident background processes will invalidate the static hardware latency profiles assumed in Eq. (4), introducing unpredictable latency spikes and degrading routing reliability.

2、Lack of Evaluation Under Real-World Cloud Multi-Tenancy: Although Appendix B.10 provides a rudimentary two-client sharing probe, the vast majority of throughput and server overhead analyses in the main text rely on an idealized "single-client, static batch size = 1" setting. In real-world cloud environments characterized by multi-tenancy, high concurrency, and dynamic batching, the scattered, out-of-order request streams generated by the asynchronous verification pipeline will severely disrupt the continuous batching efficiency and KV cache management (e.g., PagedAttention) of the cloud target model (A100), potentially overestimating the system's practical throughput gains.

Questions for authors

Quantify the Wasted Computation Ratio in Asynchronous Pipeline: Please calculate the percentage of "stale requests" that are directly discarded by the server due to branch prediction errors or failed predecessor requests at different RTT and jitter intensities, and calculate the resulting floating point operations (FLOPs) and GPU cycles wasted at the edge and cloud

3 comments

We thank the reviewer for the insightful and constructive review. The additional experiments and the updated content described below are in the revised paper, marked in blue.

Changes to the system in the revision. The submitted system was evaluated with one client per server on a dedicated edge GPU. We extended it to support a multi-client server and a shared edge GPU:

  • Batched verification: the server verifies the requests of several clients in one forward pass.
  • Load-adaptive draft depth (A1, A3; 5 or 7 in the submission): the edge sets the depth from the measured acceptance of its drafts and adds the levels it can draft during the server's queue wait, which the server reports with each verdict (formula: revised appendix).
  • Router inputs: the response-time estimate includes the server's queue wait, and, in an offline evaluation, the edge's measured speed in place of its profile.

With one client these leave the paper's results as they are (the paper's Table 2 on SPROUT, re-measured at five RTTs: A3 −1.0 to +1.7 %; Ref, S1, A1 −4.5 to +0.7 %).


W1: A shared edge GPU.

A second job on the edge GPU slows the edge models, so the latency profile of Eq. (4) no longer matches the device. We examined how the router behaves under this condition (Table 1) and how much of the edge GPU the cloud path uses (Table 2).

  • Router with a stale profile (as submitted). We re-ran the router's selection on the SPROUT test set at 100 ms added RTT with the edge 10 / 30 / 50 % slower than its profile. The router keeps its mode shares and its accuracy, and its latency grows with the slowdown, as does that of A3 and Ref, which draft on the same edge: it stays faster than Ref throughout (1.15× at 50 %) and than A3 up to a 30 % slowdown (1.05×); at 50 % A3 alone is faster (0.95×).
  • Router with the measured edge speed (new router input, offline). The step time of the 1.7B draft model, which runs in every mode, gives the edge's current speed. In the same re-run with it in place of the profile, the router moves queries to Spec-Cloud (56 → 76 % at a 50 % slowdown) and stays 1.11–1.13× faster than A3 (1.34–1.54× over Ref), within 0.2 points of the target model's accuracy and without retraining.

Table 1. The router when the edge runs slower than its profile (SPROUT test set, 100 ms added RTT, RTX 4090 profile; the router's selection re-run offline). A3 and Ref draft on the same edge and slow down with it, scaled from their measured speeds on the RTX 4090 and A30 edges. A3 alone sends every query to Spec-Cloud (accuracy 79.6 %).

edge speedrouter decides withEdge-Direct / Spec-Cloud / Edge-Think / Edge-CoD (%)accuracy (%)mean latency (s) R / A3 aloneR / A3R / Ref
as profiledthe profile20 / 56 / 7 / 1679.615.6 / 17.91.151.59
−10 %the profile20 / 56 / 7 / 1679.616.7 / 18.71.121.52
the measured speed20 / 59 / 6 / 1479.416.5 / 18.71.131.54
−30 %the profile20 / 56 / 7 / 1679.619.9 / 20.91.051.35
the measured speed20 / 67 / 3 / 1079.518.6 / 20.91.121.45
−50 %the profile20 / 56 / 7 / 1679.625.4 / 24.20.951.15
the measured speed18 / 76 / 1 / 479.521.7 / 24.21.111.34
  • Edge GPU time. With one client A3 holds the edge GPU for less time per committed token than Ref and keeps it less busy. Part of this gap is S1, the paper's edge draft optimizations, which we do not claim; against S1 itself A3 needs about the same edge GPU time (time × utilization) and takes 6–23 % less time per committed token. S1 also drafts during the wait, but its draft depth is fixed by the RTT and a single proactive tree cannot use a long wait: its deeper levels are rarely on the path the server commits, so S1 stops at its depth limit and the edge idles (utilization 60 % at 300 ms), whereas A3 uses the wait for further trees.
  • Shared server. With eight clients the server answers more slowly, and S1 cannot use the queue wait either: it idles (utilization 52 % at 50 ms), while A3 deepens its drafts by the queue wait and takes 5 / 2 / 15 % less time per committed token at 50 / 100 / 300 ms.
  • Memory. The edge's peak is 4.8 GB of 24 GB, as for S1 (Ref: 4.5 GB).

Table 2. Use of the edge GPU (A30 edge, SPROUT).

ms per committed token / GPU utilization / power (W)
clients, variant, by added RTT (ms)50100300
1, Ref32.1 / 95 % / 12340.0 / 95 % / 12263.9 / 84 % / 115
1, S124.4 / 77 % / 11230.9 / 75 % / 11154.9 / 60 % / 97
1, A322.9 / 81 % / 11926.6 / 80 % / 11942.3 / 80 % / 120
8, Ref36.2 / 83 % / 11441.9 / 89 % / 11865.8 / 81 % / 112
8, S135.7 / 52 % / 9336.4 / 63 % / 10156.9 / 56 % / 97
8, A334.0 / 84 % / 11935.8 / 83 % / 11948.4 / 84 % / 120

W1 (continued): A second job on the edge GPU, measured.

With a job that keeps the edge GPU busy half of the time, a draft level takes 16 instead of 12 ms; Ref runs at 22.6 / 12.7 and A3 at 33.9 / 20.7 tokens/s at 50 / 300 ms added RTT (without the job: 31.0 / 15.6 and 43.6 / 23.6), so A3's lead grows from 41 / 51 % to 50 / 63 %.


W2: Evaluation under cloud multi-tenancy.

To evaluate this setting, we ran eight clients against one A100 with batched verification and the load-adaptive draft depth (Table 3); every output matches the target model token by token. The gain over Ref shrinks under load (40–51 % with one client, Table 2) but remains. The same experiment on SpecBench and OlympiadBench gives the same picture (the gain grows with the RTT); the revised paper reports all three benchmarks.

Table 3. Eight A30 clients sharing one A100 (SPROUT): tokens/s summed over the clients (A3 at 50 ms: mean of two runs).

variant, by added RTT (ms)50100300
Ref217187118
A3231218159
  • Routing under load. With eight clients at 50 ms, R reads the server's queue as a longer round trip and answers 40 % of the queries on the edge; against the same clients sending every query to Spec-Cloud, the server's queue wait halves and the queries it still verifies run 18 % faster (Table 4).

  • Batching and KV cache. Stale requests (those whose continuation an earlier verdict did not confirm) are never forwarded: the gate rejects them before the target model runs. A batch therefore holds only requests that advance a query, and the batches of A3 are fuller than those of Ref (Table 5). The server keeps one preallocated KV region per client, which only a verified request extends, from the committed prefix; a rejected request never reaches it. Requests that arrive out of order therefore leave the cache untouched, and the server's peak memory is the same for both variants.

Table 4. The router with eight clients sharing one A100 at 50 ms added RTT, against the same clients sending every query to Spec-Cloud (A30 edges, SPROUT, 32 queries per client, one run each).

Revery query on Spec-Cloud
queries answered on the edge40 %0 %
RTT the router decides with, median (ms)112–
server queue wait, mean (ms)3669
requests verified per forward pass1.73.2
time per forward pass of the target model, median (ms)4463
speed of the queries sent to Spec-Cloud (same queries)1.18×1.00×

Table 5. Verification on the server per query (eight clients, added RTT 50 ms, SPROUT).

per queryRefA3
requests arriving80.1128.3
↳ stale, rejected by the gate047.4
↳ verified80.180.9
forward passes of the target model31.623.0
requests verified per forward pass2.543.52
committed tokens per forward pass18.325.1

Per query: the server's totals divided by the number of queries of all clients; one forward pass verifies the requests of several clients.

Q: Wasted computation of the asynchronous pipeline.

A3 shifts work toward the side that has slack (Table 6).

  • Where the work goes. With one client the server is idle, so A3 keeps its trees shallow and asks for verification more often: it spends more target-model GPU time than Ref and gains throughput. With eight clients the server is loaded, so the edge drafts deeper during the queue wait and asks less often; its requests share forward passes, and A3 spends less target-model GPU time than Ref; its edge GPU time per token rises from 18.5 to 28.4 ms (at 50 ms).
  • Cloud. Stale requests waste no FLOPs in the cloud: the gate rejects them before the target model runs, by comparing token ids and positions (0.3 ms each).
  • Edge GPU time, energy and FLOPs. On the edge, the costs that count per committed token, GPU time and energy, are lower for A3 than for Ref in every setting of Table 6 (by 2–42 %) and, with one client, about those of S1 (with eight clients S1 idles through the queue wait, W1). The GPU time that goes into drafts that are not committed is lower as well, except with eight clients at 50 ms, where A3 deepens its drafts by the queue wait. Its count of wasted draft FLOPs is higher because draft inference at this batch size is latency-bound: a forward of a few tree nodes leaves most of the GPU's capacity unused, and its time is set by the depth of the model, not by the number of rows. A3 fills that capacity with the nodes of three anchors: its draft forwards carry 2.4–4.8 times the rows of Ref's, yet a tree level takes about as long as in S1 (11.7–12.3 against 11.2–11.8 ms). The FLOP count therefore shows that A3 uses the edge GPU more fully, not that it costs more.
  • Jitter. Under the paper's jitter models (one client, RTX 4090 edge, 0 ms added RTT) the stale share rises from 18 % to 30–33 %, with under 2 % change in target-model forward passes.

Table 6. Stale requests and work per committed token, Ref / A3 (A30 edges, SPROUT).

Ref / A3 per committed token, by added RTT (ms)50 (one client)300 (one client)5G trace (one client)50 (eight clients)300 (eight clients)
stale requests of A3 (share of those sent)26 %60 %28 %37 %62 %
edge: wasted draft FLOPs (GFLOP)84 / 17695 / 37189 / 21184 / 21195 / 357
edge: GPU time on drafts not committed (ms)15.0 / 8.939.0 / 24.120.2 / 10.814.9 / 19.939.2 / 32.3
edge: GPU time (ms)30.4 / 18.553.6 / 33.635.4 / 20.430.1 / 28.453.2 / 40.5
edge: GPU energy (J)3.96 / 2.727.32 / 5.074.57 / 3.014.13 / 4.047.33 / 5.79
cloud: target GPU time (ms)5.96 / 8.054.61 / 8.165.39 / 8.003.20 / 2.714.11 / 3.89

Wasted draft FLOPs: all draft forwards minus the committed draft tokens (3–4 GFLOP); they include the drafts sent in stale requests. 5G trace: a driving trace of a commercial 5G network (RTT p50 / p95 / p99 of the replayed part: 73 / 105 / 330 ms).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.