acceptodds
Under review as a conference paper at ICLR 2027

SODA: Subspace Online Drafter Adaptation for Speculative Decoding

Abstract

Speculative decoding accelerates large language model inference by verifying tokens proposed by a lightweight drafter, but a drafter trained offline on a fixed corpus may not match the local request stream encountered during serving, thereby reducing accepted tokens. Online drafter adaptation can specialize the drafter to this stream, yet optimizing the full drafter introduces computation and communication overhead that can offset the resulting decoding gains or require substantial additional resources. We introduce SODA, which restricts stream-local drafter correction to an offline-learned low-dimensional subspace, thereby largely reducing the adaption and communication cost. Ablation on Qwen3-8B target model shows that SODA updates only 65.5K parameters, compared with up to 1.05B under full drafter fine-tuning, while reaching 97.1%–99.7% of its absolute mean accepted tokens. These results demonstrate that compact stateful adaptation can improve speculative decoding over within-domain request streams with limited additional serving resources.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large language model, speculative decoding, test-time training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ejGUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SODA accelerates speculative decoding by specializing a lightweight drafter to the live request stream without the cost of full online fine-tuning. Offline, it learns a low-rank LoRA correction on DFlash’s target-feature projection layer and freezes the shared basis. Only the compact coefficients are updated from target verification signals already produced during decoding, via an asynchronous worker that stays off the critical path. On math and code benchmarks with Qwen3-8B and Gemma-4-12B-IT, SODA raises mean accepted tokens by about 10–39% and cuts end-to-end latency by about 7–22% versus fixed drafters, approaching full-drafter fine-tuning with orders-of-magnitude fewer online parameters.

Critical strengths

Practical problem framing. Online drafter–target mismatch under distribution shift is a real serving bottleneck; using verification logits as free supervision is well motivated.

Strong efficiency trade-off. Updating only B yields most of full fine-tuning’s acceptance gains while sharply reducing gradients, optimizer state, and parameter transfer.

Serving-aware design. Asynchronous updates, cross-domain ordering tests, and latency reporting show the authors care about end-to-end cost, not only MAT.

Critical weaknesses

Limited algorithmic novelty. The method is essentially offline LoRA with a frozen basis and online B-only updates—an incremental systems recipe rather than a new learning principle; “subspace adaptation” largely reframes standard LoRA factorization.

Ad-hoc layer choice and brittle assumptions. Adapters are hand-placed on the target-feature projection with no selection principle. Appendix failures (e.g., FC rank-16 B-only no gain until q/v or higher rank) show dependence on placement and subspace capacity, weakening generality claims.

Narrow evaluation. Results cover two targets and four math/code sets; ordering and hardware analyses are mostly Qwen3-8B-only, with limited evidence for long-term drift, arbitrary domain mixtures, or other drafter architectures.

Core claim under-tested. Online LoRA (A+B) nearly matches SODA on MAT, so gains are mainly state size and async overhead—not accuracy. Stronger low-cost baselines and deeper shared-basis analyses would better isolate what the subspace design uniquely contributes.

Questions for authors

See weakness above.

Reviewer_uGTBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a lightweight way to adapt a draft model during inference. The main idea is to first learn two low-rank factors A0A_0 and B0B_0 offline, then freeze A0A_0 during online adaptation to reduce the number of parameters to be updated.

Critical strengths
  1. Updating "B-only" achieves close performance in terms of MAT compared to updating "A+B". This is useful for keeping speculative decoding fast while reducing the resource consumption of online optimizer.
  2. Updating only the target-feature projection has similar performance as updating the full-drafter, demonstrating that a small interface update can provide effective specialization.
Critical weaknesses
  1. It is unclear that if a lower-rank (smaller than 16) ordinary LoRA would achieve the same or better performance. To elaborate, Table 3 shows that SODA updates 65.5K parameters at rank 16, what would the performance be for Online LoRA (FC, A+B) with lower rank that also updates approximately 65.5K parameters?
Questions for authors
  1. Does the "Fixed drafter" mean the original DFlash drafter (WW) or the drafter that has been updated during the Offline Subspace Learning stage (W+A0B0W + A_0 B_0)?
Reviewer_Sdn8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

SODA is an online adaptation method for speculative decoding drafters. It first trains a rank-16 LoRA offline on the target-feature projection (FC) layer of a DFlash drafter, then freezes one factor at serving time and updates only the other (65.5K parameters on Qwen3-8B) using verified target tokens as hard labels. To avoid blocking decoding, updates happen asynchronously on a separate worker every 64 verification rounds. Across GSM8K, Math500, MBPP, and LiveCodeBench (with Qwen3-8B and Gemma-4-12B-IT), SODA increases mean accepted tokens by 9.9–38.8% and lowers end-to-end latency by 6.5–21.6% over a static drafter, achieving 97.1–99.7% of full fine-tuning MAT on Qwen3-8B. The paper also includes ablations on random frozen bases, joint A+B updates, sync vs. async execution, and boundary-free cross-domain streams.

Critical strengths

Table 3's comparison between learned and random bases makes a clear point: at the same 65.5K online parameter budget, a random frozen factor lifts MAT by just 2–4%, whereas the offline-learned basis delivers 10–17%, with joint A+B or full-drafter tuning adding very little beyond that. For engineers designing online drafter adaptation, this confirms that per-stream state can stay tiny without sacrificing much acceptance. The Table 4 async ablation is also thoughtfully set up—the no-apply control isolates the 2.0–5.6% training-path overhead, and the sync run proves that updates must stay off the critical path.

Critical weaknesses
  1. Unmeasured systems trade-offs. While the paper motivates SODA by the overhead of full drafter updates, Table 3 only reports MAT for online LoRA and full fine-tuning without latency or throughput—leaving the core systems trade-off unverified. Moreover, because the adapter sits at the drafter's input FC projection, backprop still traverses every drafter layer. Updating only B saves weight gradients, optimizer states, and transfer overhead, which is far narrower than the "four orders of magnitude" parameter ratio implies. The E2E numbers also omit the update worker's accelerator share (0.05–0.07 per inference process) and cold-start cost, and all experiments assume single-stream sequential serving on homogeneous benchmarks.2. Missing baselines and overstated novelty. There is no static (A0,B0)(A_0, B_0) baseline (offline LoRA applied without online updates) to separate genuine online adaptation from extra offline training on held-out data. The evaluation also omits prior online drafters (OSD, OnlineSpec, Test-Time Speculation) and nonparametric baselines like prompt lookup or suffix decoding; since updates trigger every ~300–400 tokens on responses up to 1024 tokens, within-request n-gram reuse could explain part of the gain. Finally, freezing one LoRA factor while updating the other is the LoRA-FA method (Zhang et al., 2023), which goes uncited—making the random-basis control essentially online LoRA-FA.
Questions for authors

Clarifications:Which LoRA factor is actually updated online? As written, B∈Rr×dinB \in \mathbb{R}^{r \times d_{\text{in}}}, which for DFlash's FC layer should be 16×(k×4096)16 \times (k \times 4096). However, the reported parameter counts—65.5K=16×409665.5\text{K} = 16 \times 4096 and 61.4K=16×384061.4\text{K} = 16 \times 3840—match the hidden-size side, and 328K−65.5K328\text{K} - 65.5\text{K} leaves 16×1638416 \times 16384 for the other factor. This suggests the input projection is frozen while the output projection (PEFT's lora_B) is updated online, which would make the fixed subspace the row space rather than the column space described around Eq. (6). Could you clarify?Baseline and initialization details: Does the fixed-drafter baseline include A0B0A_0 B_0? The no-apply variant in Table 4 matches it to two decimal places, which fits either interpretation. Also, in the random-basis control, is BB initialized from zero or from B0B_0?Async system overhead for baselines: Could you report E2E latency—or throughput per accelerator accounting for the update worker—for online LoRA (A+BA+B) and full fine-tuning under the exact same async pipeline? This comparison is the most likely to change my rating.Supervision after the first rejection: Your loss includes token positions after the first rejected draft token, where target labels are conditioned on an incorrect prefix. How does MAT change if the loss is truncated at the first rejection?Hyperparameter selection and variance: Appendix E shows the default setup yields no gain on OpenSeeker. How should adapter placement and rank be selected for a new workload without prior tuning? Additionally, since all numbers appear to be single-run results, reporting variance across seeds or stream shuffles would be helpful.Suggestion (non-blocking): The introduction motivates online adaptation via drifting target distributions (citing ReSpec and DAS). Evaluating SODA in a setting where the target model actively shifts—such as during RL training—would make a much stronger case than fixed-target benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.