acceptodds
Under review as a conference paper at ICLR 2027

Recovering Off-Policy Supervision for Speculative Decoding

Abstract

Block drafters for speculative decoding are commonly trained on corpora written by external models, where a single off-policy token invalidates supervision for all subsequent slots in a block. Existing approaches discard these divergent slots, resulting in severe supervision loss. To resolve this problem while preserving the training corpus, we propose a rollout-based training framework that recovers full supervision through two complementary components. The first component, Anchor-Label Relabelling (ALR), replaces corpus labels with distributions from greedy target rollouts, restoring valid supervision across all predicted slots. The second component, In-Rollout Anchors (IRA), places draft blocks directly inside these rollouts to expose the drafter to target-generated context, reusing precomputed rollout features at no additional target cost. Across fixed vision-language and text corpora, our framework increases greedy accepted length by up to 36.5% over DFlash and consistently outperforms erasing baselines. Notably, a single epoch of our method surpasses the best multi-epoch erase schedules and matches the acceptance length of training on target-regenerated responses. These results show that our framework provides an effective and compute-efficient approach for training speculative drafters on fixed corpora without modifying the original text.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, draft model distillation, off-policy supervision, vision-language models, inference efficiency, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9xH9OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a training framework for speculative decoding on fixed off-policy corpora. It introduces two components: Anchor-Label Relabelling (ALR), which replaces corpus-conditioned labels with target-generated rollout distributions, and In-Rollout Anchors (IRA), which exposes the drafter to target-generated contexts. Experiments on DFlash demonstrate improvements in acceptance length and decoding speedup over conventional training and erasing-based baselines.

Critical strengths
  1. The paper clearly identifies the supervision mismatch caused by off-policy training data for parallel drafters.
  2. The proposed methods are technically straightforward and supported by extensive experiments and ablation studies.
  3. The improvements over DFlash and erasing-based baselines are consistent across multiple models and datasets.
Critical weaknesses
  1. The motivation is insufficiently justified. The central premise is that the original training corpus cannot be modified. However, regenerating responses using the target model is already a standard practice in speculative decoding training, supported by widely used frameworks such as SpecForge and Speculators. Modern inference engines such as vLLM and SGLang make this process highly efficient. More importantly, the paper does not convincingly explain why replacing the original responses is unacceptable while replacing their supervision labels and introducing target-generated contexts through ALR and IRA is permissible.

  2. The practical advantage over response regeneration is unclear. Table 4 shows that directly regenerating responses achieves essentially the same acceptance length as ALR + IRA (3.833/3.856 vs. 3.841). Therefore, the proposed framework does not demonstrate a performance advantage over the standard regeneration approach.

  3. The computational efficiency claim lacks an appropriate comparison. Response regeneration only needs to be performed once, and the resulting data can be reused across training epochs. In contrast, ALR + IRA repeatedly performs target rollouts during training, increasing per-epoch computational costs. The paper does not compare the total cost of regeneration + standard training against ALR + IRA training. Without this comparison, it is unclear whether the proposed approach offers any practical computational advantage.

Questions for authors
  1. What realistic scenarios require preserving the original training responses while allowing their supervision labels and contexts to be replaced with target-generated content? How common are these constraints in practical speculative decoding training?
  2. Can the authors provide an end-to-end computational cost comparison against target response regeneration followed by standard drafter training, particularly over multiple training epochs?
  3. Given that response regeneration achieves nearly identical acceptance length, what concrete advantage does ALR + IRA offer over simply regenerating the training corpus?
18 comments
AuthorsOpenReview ↗

Response to Reviewer 9xH9 (1/11)

We thank the reviewer for the careful reading.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

Our answers in brief.

  1. Motivation for a fixed corpus (W1, Q1). The paper studies drafter training on stored responses that must remain the training text, without assuming that regeneration is unavailable or uncommon. Regeneration replaces that text, so its availability does not meet this constraint (revised Section 1, Lines 53 to 57). The paper cites regeneration when responses may be replaced (Section 1, Lines 50 to 53). Training on the stored responses is the route used by EAGLE, HASS and Falcon. Every drafter we train, including the DFlash and erase baselines, already reads target signals inside the training step. ALR + IRA adds at most 14 continuation tokens from each stored prefix, computed in that step and never stored, so the constraint fixes the text rather than the teacher signals.
  2. Concrete advantages over regeneration (W2, Q3). The regeneration comparison tests recovery of regeneration's accepted length while keeping stored responses, without requiring a higher accepted length. ALR + IRA exceeds both regeneration references after the human-written first sentence of a caption on COCO and three more caption datasets, and trains on every multi-turn conversation as stored. Preserving the training text therefore gives an acceptance advantage after human-written text and keeps the context of later stored turns (Table R5, Table R11, Table R10 under Q1). An LLM judge flags the next stored user turn as no longer fitting a regenerated first response in 15.7% of conversations, against 5.5% with the stored one. After three epochs on ALLaVA, ALR + IRA also reaches the accepted length of both length-matched regeneration references at T=0 and T=1 without replacing any stored response, the result stated in Section 1, Lines 84 to 86 (Table 4a, Table R3).
  3. Complete cost and target reuse (W3, Q2). The paper's compute claim compares ALR + IRA with erase, the other method that keeps the stored responses, and is not a cost comparison with regeneration, which replaces them. Table R1 gives the complete cost of both routes, including generation, at one, three and six epochs for the methods that keep the corpus. One epoch of ALR + IRA exceeds every erase schedule up to six epochs in 41% less training time than the best one. A regenerated corpus has to be generated again for each new target. A corpus regenerated by Qwen2.5-VL-7B-Instruct, the previous Qwen-VL generation, and reused for Qwen3-VL-8B trains a drafter below erase (Table R12 under W3). Table 4 supplies the three-epoch accepted lengths.
AuthorsOpenReview ↗

Response to Reviewer 9xH9 (2/11)

W1. Training on stored responses is established

"The central premise is that the original training corpus cannot be modified. However, regenerating responses using the target model is already a standard practice in speculative decoding training, supported by widely used frameworks such as SpecForge and Speculators."

Answer. The paper studies training a drafter to reach regeneration's accepted length when stored responses must remain the training text. It does not argue that regeneration is unavailable or uncommon. Regeneration replaces that text and must be repeated for each new target.

To compare the stored-response route with regeneration, the table below collects the accepted lengths of Table 4a and the results of Tables R5, R10, R11 and R12.

ComparisonALR + IRARegeneration
Accepted length, Table 4a3.8413.833 truncated, 3.856 loss window
Next user turn flagged, Table R105.5% with stored response15.7% with regenerated response
Human caption prefix, COCO, T=0 and T=1, Table R53.628 and 3.206 accepted length3.534 and 3.097 truncated, 3.562 and 3.116 loss window
Human caption prefix, three datasets pooled, T=0, Table R11+3.60% over truncated, +3.08% over loss windowTruncated and loss-window references
Reuse after a target update, T=0, Table R123.841 accepted length3.501 with the previous target's corpus

An audited or licensed release clears only its stored text, and Q1 and W3 measure three more cases for keeping the stored responses. In multi-turn data, a regenerated first response leaves the next stored user turn flagged as no longer fitting in 15.7% of the conversations, against 5.5% with the stored one (Table R10). After human-written text at the start of a caption response, ALR + IRA exceeds both regeneration references (Tables R5 and R11). A regenerated corpus reused after a target update trains a drafter below erase (Table R12). Keeping the stored responses preserves later-turn context and lets every target train on the same text. Table R3 under W2 lists the accepted lengths of the regeneration comparison.

The paper states regeneration as the route for corpora whose responses may be replaced. It cites regeneration for that case (Section 1, Lines 50 to 53, Section 2, Lines 111 to 117). The second route trains on the stored responses with target signals computed on them, and it is the route of widely used drafters. EAGLE (Li et al., 2024a), cited in Section 2, states the choice in its own words. It writes that "EAGLE uses a fixed dataset for training" (EAGLE, Section 4.3.3). HASS (Zhang et al., 2024) and Falcon (Gao et al., 2025) train on the same ShareGPT conversations as EAGLE. Section 2 cites both.

The frameworks named in the review support this route as well. SpecForge and Speculators both offer online training, in which the target's hidden states are computed inside the training run from the stored dataset. Both run regeneration as a separate generation pass over the corpus before training. Citing EAGLE, the SpecForge data-preparation guide notes that "the performance is still good even if you use the original dataset". It presents regeneration as a way to "improve the acceptance rate of the draft model". After three epochs on ALLaVA, ALR + IRA reaches the accepted length of length-matched regeneration without that pass or any replacement of a stored response (Table 4a).

The open question of this route is how much of the benefit of regeneration a drafter can recover while the stored responses stay the training text. EAGLE compared the two kinds of training data in its own ablation (EAGLE, Table 6). HASS reports that "HASS trained on the fixed dataset even achieves better performance than EAGLE-2 trained on the model-generated data consistently" (HASS, Appendix A.4). The erase baseline applies the confidence-adaptive weights of PARD-2 (An et al., 2026). These weights reduce supervision after corpus tokens the target is unlikely to produce (Section 2, Lines 93 to 97).

The paper studies recovering regeneration's accepted length while the stored responses remain the training text. This matters because reviewed releases and stored conversations require that text, as Section 1 of the revised PDF explains in Lines 53 to 57. After three epochs on ALLaVA, ALR + IRA reaches both length-matched regeneration references within 0.4% without replacing any stored response, and erase stays below all three (Table 4a).

AuthorsOpenReview ↗

Response to Reviewer 9xH9 (3/11)

W1, continued. Transient distillation signals preserve the corpus

"More importantly, the paper does not convincingly explain why replacing the original responses is unacceptable while replacing their supervision labels and introducing target-generated contexts through ALR and IRA is permissible."

The paper does not claim that replacing responses is unacceptable in general. Its constraint is on the training text, not on the use of target signals, and ALR + IRA never replaces that text.

Table 4a compares the same two kinds of training data as EAGLE and HASS, the stored responses and responses written by the target. A drafter of the EAGLE line learns from target hidden states computed on the stored corpus. In our comparison, every drafter, the DFlash baseline included, reads target hidden states computed inside the step. KD and erase also train on the target's next-token distributions along the corpus (Section 3.1, Equations 1 and 2). None of these signals is stored, and training on them does not modify the corpus. The concern in the review therefore applies to a method that replaces the stored responses, and ALR + IRA does not replace them.

ALR reads the same distributions along the target's own greedy continuation from each anchor in the corpus, at most 14 tokens long. IRA anchors half of the blocks inside these continuations and reuses their states (Section 3.2). This means that every block conditions on a stored prefix. A block anchored inside a continuation adds at most 14 target-written tokens to that prefix, its anchor token included. Section 3.2 states this at Lines 218 to 219 and 225 to 228. Nothing the target writes is stored or written back (Section 1, Lines 63 to 64).

Replacing the labels with target distributions is knowledge distillation, which Equations 1 and 2 already apply along the stored corpus for KD and erase. Replacing each stored response with a complete target response is sequence-level distillation (Kim and Rush, 2016, cited in Section 2). It produces a second corpus and changes the conversation that later turns were written for.

The paper studies how close a drafter can come to regeneration while every stored response stays the training text. This matters when only an audited or licensed release may be used for training, as Section 1 of the revised PDF explains in Lines 53 to 57, and in the multi-turn and human-written cases above. ALR + IRA keeps that text, as KD and erase do, and adds target text only as labels and as at most 14 tokens after a stored prefix.

The new Appendix K of the revised PDF expands this definition.

Reviewer_3gjXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses a supervision mismatch in training parallel drafters on a fixed off-policy corpus: once the target model's continuation diverges from the stored response, the supervision at later slots no longer reflects the prefixes the drafter will see at deployment, and existing survival-gate methods simply down-weight or discard it. The authors propose Anchor-Label Relabelling (ALR), which replaces corpus labels with distributions from greedy target rollouts, and In-Rollout Anchors (IRA), which relocates half of the draft blocks into these rollouts so that the drafter is also trained on target-generated context, reusing the rollout's target states without additional target forward passes.

Experiments on Qwen3-VL-8B/4B (ALLaVA and ShareGPT4V training; COCO, TextVQA, DocVQA evaluation) and Qwen3-4B text models (UltraChat training; seven benchmarks) at T=0 and T=1 report accepted-length gains of up to 36.5% over DFlash trained on identical data, and smaller but consistent gains over a tuned soft-gated erase baseline (e.g., 2.82x vs. 2.62x on ALLaVA and a 2.80x vs. 2.67x seven-task text average at T=0).

Critical strengths

The paper separates two distinct sources of training mismatch, inconsistent continuation labels and the absence of target-generated context, and addresses each with a targeted mechanism (ALR and IRA, respectively). The label-source versus survival-weight comparison separates changing the supervision from merely changing loss weights, the slot-matched control rules out explanations based on the number of supervised blocks, weights, or offsets, and length-matched regeneration provides an upper reference.

The erase baseline uses the same architecture, initialization, anchors, and optimization settings, and is itself tuned (the soft gate is shown to outperform hard and ungated distillation). The paper reports training replicates, prompt resampling, and repeated timing, and its cost accounting includes rollout computation while making clear that "no additional target forward" refers only to IRA's reuse, not to the total cost relative to erase. These results support the central claims within the evaluated scope.

Critical weaknesses

The evidence for generality is restricted. All targets belong to the Qwen3 family and all drafters use the DFlash architecture, generation is capped at 256 new tokens, the vision-language evaluation scores 99 prompts per domain, and the speed measurements are mainly at batch size 1, with some timings excluding the first draft construction. The conclusions are therefore well supported for this regime, but not yet for other drafter architectures, long generations, or serving workloads with continuous batching.

Questions for authors
  1. At T=1, how does ALR with sampled target rollouts compare with greedy-rollout ALR at the same training budget?
  2. What fraction of outputs reaches the 256-token limit, and how do the gains vary with generated length? How do end-to-end speedups change when the first draft construction is included and at larger batch sizes?
14 comments
AuthorsOpenReview ↗

Response to Reviewer 3gjX (1/11)

We thank the reviewer for the detailed and balanced review.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

Our answers in brief.

  1. Generality beyond Qwen3 and DFlash (W1). ALR + IRA retains its gains with a Llama-family target and Medusa-style heads, so the gains extend beyond both evaluated architecture choices. With Llama-3.1-8B-Instruct, it exceeds erase by 2.02% at T=0 and 2.01% at T=1 in aggregate accepted length across the seven text benchmarks, and also exceeds DFlash (Table R8). With Medusa-style heads replacing the DFlash block drafter on Qwen3-4B, it has the longest accepted length of the four label rules on all seven text benchmarks at both temperatures (Table R9, Table 26). Its aggregate gains over erase are 2.35% and 2.39% at the two temperatures. Every aggregate gain in Tables R8 and R9 has a 95% interval above zero.
  2. Length cap and longer generations (W2, Q2). ALR + IRA keeps its gain on outputs reaching the 256-token limit and with longer limits, so short outputs do not explain the gain. Most outputs reach that limit on six of the seven text benchmarks and on COCO, and their gain over erase is in line with the overall gain (Table R2). With up to 2,048 new tokens at T=0, ALR + IRA stays above erase on all four text benchmarks rerun and in every range of output positions (Table R4). On COCO captions with up to 1,024 new tokens, it is 7.57% above erase at T=0, and nearly all captions end before the limit (W2).
  3. Vision-language sample size (W3). ALR + IRA exceeds erase on all three domains with 499 scored prompts in each domain, so the gain also holds on the larger set. With Qwen3-VL-8B trained on ALLaVA at T=0, it exceeds erase on COCO, TextVQA and DocVQA, with an aggregate accepted-length gain of 8.64%, close to the gain on Table 8's prompts (Table R7). The aggregate intervals from resampling prompts or whole images lie above zero for the larger set and, in Appendix E, on ALLaVA, ShareGPT4V and with the smaller target. The text results use every prompt of the seven benchmark sets of Appendix D.
  4. First-draft timing and serving load (W4, Q2). The text speedups already include the target prefill and first draft construction, and ALR + IRA retains its speed advantage when timed over whole requests. On Qwen3-4B with continuous batching in SGLang on an A100, it has the highest throughput of the three drafters on both tasks at every load served (Table R6). Its accepted length stays above erase at every load, linking the acceptance gain to serving throughput. Timed on an A100 from the start of generation, the three vision-language domains also give ALR + IRA a 9.48% higher speedup than erase, computed from unrounded speedups (Appendix H).
  5. Sampled versus greedy training rollouts (Q1). Drawing every rollout token from the target at T=1 leaves ALR within 1% of greedy-rollout ALR in Table 8 at both temperatures. Greedy training rollouts therefore also train a drafter for sampled decoding (Appendix J).

The table below collects the aggregate gains over erase in answer 1.

Aggregate gain of ALR + IRA over erase, seven text benchmarksT=0T=1
Llama-3.1-8B-Instruct target (Table R8)+2.02%+2.01%
Medusa-style heads on Qwen3-4B (Table R9)+2.35%+2.39%
AuthorsOpenReview ↗

Response to Reviewer 3gjX (2/11)

W1. Gains with a Llama-family target

"All targets belong to the Qwen3 family and all drafters use the DFlash architecture"

Answer. ALR + IRA exceeds erase and DFlash with Llama-3.1-8B-Instruct as the target, and with Medusa-style heads as the drafter it has the longest accepted length of the four label rules.

To test whether the gains hold on a target from another model family, we trained the three drafters of Table R8 from the public DFlash drafter for this target. Each was trained on the same 26,196 UltraChat conversations for three epochs with the hyperparameters of Appendix D. We scored every prompt of the seven text benchmarks with the decoder of Table 1.

Table R8. Accepted lengths with Llama-3.1-8B-Instruct as the target are means over the training replicates of DFlash, erase and ALR + IRA trained and scored as above (Appendix O). Each aggregate averages the within-replicate geometric means over the seven text benchmarks of Table 1, with all 2,103 prompts and at most 256 new tokens. At T=0 these trained drafters give the same output text on every prompt. Gains use the unrounded accepted lengths. The intervals come from 10,000 resamples of prompts within each benchmark, with one draw shared by every method and training replicate. The intervals condition on these trained checkpoints.

MethodT=0T=1
DFlash3.0752.546
Erase3.4092.705
ALR + IRA3.4782.760
Gain over erase+2.02%+2.01%
Prompt resampling 95% interval+1.65 to +2.40+0.54 to +3.49
Gain over DFlash+13.11%+8.38%
Prompt resampling 95% interval+12.62 to +13.63+6.91 to +9.82

These results show that ALR + IRA has the highest aggregate accepted length of the three drafters at both temperatures. Every paired prompt-resampling interval of its gains over erase and DFlash lies above zero. Every training replicate of ALR + IRA is also above every erase replicate in the aggregate at both temperatures (Appendix O). At T=0 it exceeds erase on every one of the seven benchmarks (Table 25). The gain reaches 4.67% on Alpaca and is 3.29% on MT-Bench. It exceeds DFlash on every benchmark at both temperatures, and Table 25 gives every benchmark.

This target has its own tokenizer, chat template and hidden states. Its public drafter reads layers 1, 8, 15, 22 and 29 of the target. It uses the target's output head, which Llama-3.1-8B-Instruct keeps separate from its input embeddings (Appendix O). So the gains also hold with a second public drafter configuration.

We also measured how far the stored responses are from this target's own predictions. On 1,000 training conversations drawn at random, we compared the target's greedy prediction after the stored prefix with the stored token. The two differ at 31.5% of the assistant tokens in the 1,024-token training window for this target. For Qwen3-4B they differ at 33.1%. So the stored responses are off-policy for both targets (Appendix O).

So ALR + IRA keeps its gains on a target outside the Qwen3 family, with its own tokenizer, chat template and hidden states.

Table 25 in Appendix O breaks the aggregate down by benchmark.

AuthorsOpenReview ↗

Response to Reviewer 3gjX (3/11)

W1, continued. Gains with Medusa-style heads

"all drafters use the DFlash architecture"

With Medusa-style heads on Qwen3-4B, ALR + IRA has the longest accepted length on every benchmark at both temperatures. This comparison replaces the DFlash block drafter and tests all four label rules on the seven text benchmarks.

To test whether ALR and IRA help a drafter architecture other than DFlash, we ran a new experiment. For Table R9 we replaced the DFlash block drafter with Medusa-style heads, a parallel drafter that Section 2 cites (Lines 103 to 104). Each of seven heads maps the target's final hidden state at a position to one later token through the target's frozen output head. So this drafter reads no intermediate target layer and drafts each position independently from that one state, with one residual layer for each head and no attention. Every head is still supervised along the stored continuation after the anchor, the setting that ALR and IRA address. We trained the heads with corpus KD, erase, ALR and ALR + IRA labels on the UltraChat corpus. Training used the 128 anchors, learning rate, effective batch size 24 and three epochs of Appendix D. We scored every prompt of the seven benchmarks with chain verification and at most 256 new tokens, at both temperatures.

Table R9. Accepted lengths of Medusa-style heads on Qwen3-4B are means over the training replicates of each label rule, trained and scored as above (Appendix P). Each aggregate averages the within-replicate geometric means over the seven text benchmarks of Table 1, with all 2,103 prompts. Gains use the unrounded accepted lengths. The intervals come from 10,000 resamples of prompts within each benchmark, with one draw shared by every label rule and training replicate. The intervals condition on these trained heads. At T=0 all trained heads give the same output text on every prompt, so the comparison counts verification rounds for the same text.

Label ruleT=0T=1
Corpus KD1.3811.366
Erase1.3911.373
ALR1.4061.391
ALR + IRA1.4231.406
Gain over erase+2.35%+2.39%
Prompt resampling 95% interval+2.15 to +2.54+2.15 to +2.63
Gain over corpus KD+3.05%+2.92%
Prompt resampling 95% interval+2.84 to +3.27+2.65 to +3.18
Gain over ALR+1.23%+1.06%
Prompt resampling 95% interval+1.05 to +1.40+0.83 to +1.30

These results show that ALR + IRA has the highest aggregate accepted length of the four label rules at both temperatures. In aggregate it exceeds erase, corpus KD and ALR alone, and the prompt-resampling intervals of all these gains lie above zero. On every one of the seven benchmarks at both temperatures, the four label rules rank in the same order, corpus KD, erase, ALR and ALR + IRA. So ALR and IRA each add accepted length in all 14 cells of Table 26 and in the aggregate. On each benchmark, the interval of the gain of ALR + IRA over erase and over corpus KD lies above zero. In aggregate, every ALR + IRA training replicate exceeds every replicate of the other three label rules at both temperatures. On every benchmark, it exceeds every replicate of erase and corpus KD at both temperatures.

Accepted length counts the one token that the target adds in every round (Section 4). We also compare accepted length above that token.

Comparison above the target-added token, T=0 aggregateAccepted lengthRelative gain of ALR + IRA
ALR + IRA against erase0.423 against 0.3918%
ALR + IRA against corpus KD0.423 against 0.38111%

These results show that ALR + IRA also leads on accepted length above the target-added token.

So ALR and IRA each add accepted length with a second drafter architecture, one that reads no intermediate target layer and has no attention.

Table 26 in Appendix P breaks the aggregate down by benchmark.

Reviewer_u4YvOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies speculative-drafter training on fixed corpora whose responses were produced by external models and are therefore off-policy with respect to the target model. For a parallel block drafter, an early corpus token that differs from the target model’s continuation makes the supervision for all subsequent slots conditioned on a prefix that the target would not visit at inference. Existing survival-weighting approaches address this issue by down-weighting or erasing the affected slots, which can discard substantial supervision. The paper proposes two complementary components. Anchor-Label Relabelling (ALR) performs short greedy target rollouts from corpus anchors and replaces corpus-conditioned labels with target distributions along these rollouts. In-Rollout Anchors (IRA) moves a subset of draft blocks into the target rollouts, exposing the drafter to target-generated recent context while reusing already computed target features and distributions. ALR addresses label mismatch, whereas IRA addresses the remaining context-distribution mismatch. Experiments on Qwen3-VL-8B, Qwen3-VL-4B, and Qwen3-4B show consistent improvements over DFlash and a PARD-2-style erase baseline across vision-language and text benchmarks, under both greedy and sampled decoding. The paper also reports training cost, wall-clock decoding speedup, and comparisons with target-regenerated responses.

Critical strengths
  1. The problem formulation and method decomposition are clear. The paper convincingly explains why a single off-policy corpus token invalidates later-slot supervision for a parallel drafter. The separation between label mismatch, addressed by ALR, and recent-context mismatch, addressed by IRA, is intuitive and supported by the positional analyses.

  2. The paper includes strong erase baselines, ungated and hard-gated controls, rollout-depth sweeps, a slot-matched IRA control, multiple training horizons, target-regeneration references, chain verification, greedy and sampled decoding, and both acceptance-length and wall-clock speed measurements. The gains also transfer from vision-language targets to a text-only target.

  3. The improvements are practically meaningful and reasonably cost-effective. On Qwen3-VL-8B trained with ALLaVA, ALR+IRA improves acceptance length from 3.54 to 3.84 over erase and speedup from 2.62× to 2.82×. A single epoch of ALR+IRA outperforms the best multi-epoch erase schedule while costing less than two erase epochs. Reusing rollout features for IRA is an effective implementation choice.

Critical weaknesses
  1. The practical advantage over one-time target-response regeneration is not fully established. Existing drafter pipelines such as DFlash commonly train on target-generated responses. The paper’s own comparison shows that ALR+IRA and target regeneration have nearly identical acceptance length: 3.841 versus 3.833/3.856, while truncated regeneration gives a slightly higher measured speedup, 2.89× versus 2.82×. ALR+IRA recomputes short target rollouts inside each training epoch, whereas regenerated responses can be produced once and reused. The paper reports the per-epoch cost of erase, ALR, and ALR+IRA, but not the complete generation-plus-training cost of the regeneration baseline over different training horizons.

  2. The motivating “fixed corpus” constraint needs a more precise definition. ALR+IRA does not rewrite the stored responses, but it still trains on target-generated rollout tokens, hidden states, and distributions. It is therefore unclear under what concrete auditing or licensing policy transient generated contexts would be allowed while stored regenerated responses would not. Without such a constraint, direct target regeneration appears simpler and at least as effective.

  3. The evidence for cross-architecture generalization is limited. Although the experiments include Qwen3-VL-8B, Qwen3-VL-4B, and Qwen3-4B, all evaluated targets belong to the Qwen3 family and all drafter models use the DFlash architecture. The consistency across model sizes and modalities is encouraging, but it does not establish that the proposed off-policy supervision problem or the benefits of ALR and IRA generalize to substantially different target architectures. In particular, differences in hidden representations, tokenization, decoding behavior, and target-policy divergence may affect both the usefulness of rollout relabelling and the magnitude of context mismatch. Evaluation on a non-Qwen target, such as a Llama-family model, would substantially strengthen the generality claim.

Questions for authors
  1. Can the authors report the complete end-to-end cost of target regeneration, including one-time response generation, storage, and subsequent training epochs, and compare it with ALR+IRA at one, three, and more epochs in total GPU-hours?

  2. In what concrete practical or policy setting is it permissible to train on transient target-generated rollout tokens and hidden states, but impermissible to store and train on complete target-regenerated responses?

  3. Is the primary motivation reduced compute, preservation of a versioned corpus, avoidance of synthetic-response storage, or adaptation to a changing target checkpoint? Clarifying this would better define when ALR+IRA should be preferred over standard regeneration.

  4. How sensitive is IRA to the fraction of primary versus secondary blocks and to the distribution used to sample the in-rollout offset? The current half-block allocation and uniform offset appear heuristic.

  5. Can the authors evaluate ALR and IRA on a target model from a different architecture family, such as Llama, rather than only additional Qwen3 sizes?

16 comments
AuthorsOpenReview ↗

Response to Reviewer u4Yv (1/12)

We thank the reviewer for the careful and constructive review.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

Our answers in brief.

  1. Practical advantage and complete cost (W1, Q1). The regeneration comparison tests whether ALR + IRA recovers regeneration's accepted length while keeping the stored responses, not whether it exceeds regeneration. ALR + IRA has the highest aggregate accepted length among methods trained on the same fixed corpus in every setting of Table 1 at both temperatures. It therefore improves drafting while preserving the training text. After three epochs on ALLaVA, it also reaches both length-matched regeneration references at both temperatures without replacing any response (Table 4a, Table R3). Table R1 gives the complete cost of both routes over one, three and six epochs. Over the three epochs of Table 4, preserving every response costs 3.96 GPU-hours against 2.61 for regeneration. Within the fixed-corpus setting, one epoch of ALR + IRA exceeds every erase schedule up to six epochs in 41% less training time than the best one.
  2. Fixed-text constraint and motivation (W2, Q2, Q3). The fixed-corpus constraint keeps the stored training text without prohibiting transient target signals, which ALR + IRA computes and discards inside each step. Every block conditions on a stored prefix plus at most 14 target-written tokens, its anchor included, so transient signals do not become a replacement corpus. An audited release clears only its stored text, and later user turns in multi-turn chat reply to the stored responses. Regeneration changes those contexts. An LLM judge flags the next stored user turn as not fitting a regenerated first response in 15.7% of conversations, against 5.5% with the stored one (Table R10). When a COCO caption response begins with a human-written sentence, ALR + IRA exceeds both length-matched regeneration references, giving an acceptance advantage after text the target did not write (Table R5).
  3. Generality beyond Qwen3 and DFlash (W3, Q5). ALR + IRA improves accepted length with both a Llama-family target and Medusa-style heads. Its gains therefore extend to a different target family and drafter architecture. With Llama-3.1-8B-Instruct, the aggregate gains over erase across the seven text benchmarks are 2.02% and 2.01% at the two temperatures, and it also exceeds DFlash. All 95% intervals of these gains lie above zero (Table R8). With Medusa-style heads replacing the DFlash block drafter on Qwen3-4B, ALR + IRA has the longest accepted length of the four label rules on all seven text benchmarks at both temperatures. Its aggregate gains over erase are 2.35% and 2.39%, with these 95% intervals also above zero (Table R9, Table 26).
  4. IRA allocation sensitivity (Q4). Changing the primary share or offset law moves the one-epoch accepted length by at most 0.49%. All four allocations stay above ALR alone and erase, with every gain interval above zero, so the gain survives each tested change to the allocation (Appendix G).

The table below collects the aggregate gains over erase in answer 3.

Aggregate gain of ALR + IRA over erase, seven text benchmarksT=0T=1
Llama-3.1-8B-Instruct target (Table R8)+2.02%+2.01%
Medusa-style heads on Qwen3-4B (Table R9)+2.35%+2.39%
AuthorsOpenReview ↗

Response to Reviewer u4Yv (2/12)

W1. Fixed-corpus gains and parity with regeneration

"Existing drafter pipelines such as DFlash commonly train on target-generated responses. The paper’s own comparison shows that ALR+IRA and target regeneration have nearly identical acceptance length: 3.841 versus 3.833/3.856, while truncated regeneration gives a slightly higher measured speedup, 2.89× versus 2.82×."

Answer. Table 4a tests whether ALR + IRA reaches regeneration's accepted length without replacing the stored training text. It is not a test of higher accepted length than regeneration, and the paper's compute claim is against erase. ALR + IRA has the highest accepted length among methods that keep those responses as training text. Those responses must remain when only an audited or licensed release may be used for training, since the release clears only its stored text. They also matter in multi-turn data, where later user turns reply to them (Table R10 under Q2). After human-written text at the start of a response, ALR + IRA exceeds both regeneration references at both temperatures, with every paired 95% interval above zero (Table R5 under Q2).

To compare the fixed-corpus gains and regeneration references, the table below collects the values of Tables 1 and 4a.

ComparisonALR + IRAReference
Table 1, greedy accepted-length gainUp to 36.5%DFlash
Table 1, greedy accepted-length gainUp to 8.54%Erase
Table 4a, accepted length3.8413.833 truncated, 3.856 loss window

Every method in Table 1 trains on the same fixed corpus, and ALR + IRA has the highest aggregate accepted length in every setting at both temperatures. Section 1 states the greedy gains in Lines 77 to 78. EAGLE states that it "uses a fixed dataset for training" (EAGLE, Section 4.3.3). HASS and Falcon train on the same stored ShareGPT conversations, using target signals computed on them. The text results of Table 1 come from multi-turn data, the UltraChat corpus under Q2. There, 49.8% of the assistant tokens in the training window follow the first response.

Where the stored responses may be replaced, ALR + IRA matches both length-matched regeneration references after three epochs on ALLaVA. It is within 0.4% of each (Table 4a), the parity that Section 1 states in Lines 84 to 86.

We respectfully clarify the speedups in the quote. Each speedup in Table 16 divides by the autoregressive time measured in its own run. Appendix H states this in Lines 1082 to 1083. The quoted speedup of truncated regeneration comes from a timing that Table 16 does not repeat. The quoted speedup of ALR + IRA averages repeated timings (Table 16, Runs column). In the repeated timing session, autoregressive timing had a coefficient of variation of 3.4% to 5.6% across the three domains. The paper therefore bases close comparisons on accepted length rather than on small speed differences (Appendix H, Lines 1112 to 1114). By that measure, Table 16 shows parity with truncated regeneration.

To test whether the parity also holds under sampled decoding, we evaluated both regeneration references at T=1 with the protocol of Table 8.

Table R3. Mean acceptance length after three epochs, Qwen3-VL-8B trained on ALLaVA, geometric mean of COCO, TextVQA and DocVQA. Fixed-corpus values at both temperatures are from Table 8, and the regeneration values at T=0 are from Table 4a. The regeneration values at T=1 are new and use the protocol of Table 8.

Training textMethodT=0T=1
Fixed corpusErase3.5393.386
Fixed corpusALR3.6963.495
Fixed corpusALR + IRA3.8413.647
Regenerated responsesKD, truncated3.8333.601
Regenerated responsesKD, loss window3.8563.645

These results show that ALR + IRA matches both regeneration references at both temperatures. At T=1 it is within 0.1% of the loss-window reference and 1.3% above the truncated one. The loss-window reference is 4.3% above ALR alone at both temperatures. With IRA, this gap shrinks at T=0 to the parity stated above, and at T=1 it closes.

The paper studies drafting while the stored responses remain the training text. This matters because regeneration would replace the text that a release clears and that later user turns reply to, as Section 1 of the revised PDF, Lines 53 to 57, and its Appendix K explain. ALR + IRA leads among fixed-corpus methods without replacing those responses.

AuthorsOpenReview ↗

Response to Reviewer u4Yv (3/12)

W1, continued. Complete generation-plus-training cost

"The paper reports the per-epoch cost of erase, ALR, and ALR+IRA, but not the complete generation-plus-training cost of the regeneration baseline over different training horizons."

The paper's compute comparison is with erase, where one epoch of ALR + IRA exceeds every erase schedule in 41% less training time than the best one. Table R1 includes generation and training. Over three epochs, ALR + IRA costs 3.96 GPU-hours against 2.61 for regeneration, reaching the same accepted length while keeping every stored response.

To compare the complete generation-plus-training cost of regeneration with ALR + IRA, we measured both stages of regeneration. Table R1 gives the complete cost of both routes on the same four H100 GPUs. Regeneration adds a one-time vLLM pass that writes greedy responses for the 26,196 ALLaVA prompts. Its cost in Table R1 includes model loading. Each training epoch on the regenerated responses takes the same measured time as an erase epoch.

Table R1. Complete training cost in GPU-hours on four H100 GPUs, ALLaVA with Qwen3-VL-8B, with the accepted length printed in the paper at T=0. Erase and ALR + IRA epoch costs are from Appendix A, and the regeneration row is measured on the same GPUs.

Training textMethodEpochsGenerationTrainingTotalMAT at T=0
Fixed corpusErase100.750.753.533 (Table 2)
Fixed corpusErase, best schedule C302.252.253.539 (Table 2)
Fixed corpusErase604.504.503.472 (Table 12)
Fixed corpusALR + IRA101.321.323.728 (Table 2)
Fixed corpusALR + IRA303.963.963.841 (Table 4a)
Regenerated responsesKD, truncated30.362.252.613.833 (Table 4a)

These results show that among the methods that keep the corpus, one epoch of ALR + IRA exceeds every erase schedule up to six epochs, in 41% less training time than the best one. This is the measured compute result of the paper (Table 2, Figure 3). Against regeneration, the comparison runs over the three epochs of Table 4. There, ALR + IRA matches the accepted length of regeneration for 1.35 GPU-hours more, the difference between the two totals in Table R1.

To compare ALR + IRA with regeneration after human-written text and after a target update, the table below collects the values of Tables R5 and R12.

Comparison with regenerationALR + IRARegeneration
COCO with a human caption prefix, T=0 (Table R5)3.6283.534 truncated, 3.562 loss window
COCO with a human caption prefix, T=1 (Table R5)3.2063.097 truncated, 3.116 loss window
Reuse after a target update, T=0 (Table R12)3.8413.501 with the previous target's corpus

The paper studies training while stored responses remain the training text. This matters because a licensed or audited release covers only the stored text, and every later turn of a multi-turn conversation was written in reply to it, as Section 1 of the revised PDF, Lines 53 to 57, and its Appendix K explain. A cheaper regeneration run therefore does not serve a corpus that must stay as stored. One epoch of ALR + IRA exceeds every erase schedule in 41% less training time than the best one. The extra 1.35 GPU-hours over three epochs keeps every stored response while reaching regeneration's accepted length, exceeding both references after human-written text (Table R5), and using the same corpus for each new target, for which regeneration has to be repeated (Table R12).

The revised abstract and conclusion state the compute comparison with erase. The revised PDF adds Table R1 to Appendix K as Table 18.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response (1/6)

We thank the reviewers for their careful reading. All three reviews find that the paper identifies a real supervision mismatch for parallel drafters and that the experiments support its gains. Reviewers u4Yv and 3gjX add that ALR and IRA address the two sources of the mismatch separately, and that the erase baseline and the controls are strong. The reviews raise three shared questions. The first is what a fixed corpus fixes and when ALR + IRA should be used instead of regeneration. The second is what the complete cost is, and the third is whether the evidence extends beyond the evaluated regime. We answer them here and in each reply.

Our answers in brief.

  1. Fixed-corpus motivation. The paper studies recovering regeneration's accepted length while preserving the stored training text, without prohibiting transient target signals. Keeping that text preserves reviewed releases and the context of later turns. After three epochs on ALLaVA, the submission reports parity within 0.4% (Table 4a, Section 1, Lines 84 to 86). ALR + IRA exceeds both regeneration references when a COCO caption response begins with human-written text (Table R5). A regenerated first response leaves the stored next user turn flagged by an LLM judge as not fitting it in 15.7% of multi-turn conversations, against 5.5% with the stored one (Table R10). These are concrete reasons to train on the stored responses.
  2. Complete training cost. One epoch of ALR + IRA exceeds every erase schedule up to six epochs in 41% less training time than the best one. The compute advantage is over erase within the fixed-corpus setting. Against regeneration, preserving every response costs 3.96 GPU-hours over three epochs, against 2.61, quantifying the cost of keeping the training text (Table R1, the three epochs of Table 4).
  3. Generality beyond the evaluated models. ALR + IRA exceeds erase with a Llama target and with Medusa-style heads. The gains therefore extend to another target family and another drafter architecture (Tables R8 and R9).
AuthorsOpenReview ↗

General Response (2/6)

What the fixed corpus preserves

Answer. The paper studies drafter training when stored responses must remain the training text. This constraint does not prohibit transient target signals, and ALR + IRA never writes to the corpus.

Every drafter we train, the DFlash baseline included, already learns from what the target computes inside the training step. The drafter reads target hidden states. KD and erase also train on the target's next-token distributions along the corpus (Section 3.1, Equations 1 and 2).

ALR reads the same distributions along the target's own greedy continuation from each anchor in the corpus, at most 14 tokens long. IRA anchors half of the blocks inside these continuations and reuses their states (Section 3.2). Every block therefore conditions on a stored prefix. A block anchored inside a continuation adds at most 14 target-written tokens, its anchor token included. Section 3.2 states this at Lines 218 to 219 and 225 to 228. These continuations exist only inside the training step (Section 1, Lines 63 to 64).

The stored responses, which are what one reviews, versions and releases, stay unchanged. Regeneration replaces them with a second corpus written by the target. This is sequence-level distillation (Kim and Rush, 2016, cited in Section 2). Training on the stored corpus is the route of widely used drafters. EAGLE states that it "uses a fixed dataset for training" (EAGLE, Section 4.3.3). HASS and Falcon train on the same ShareGPT conversations.

In two kinds of data, the stored corpus is the text to train on. One is an audited or licensed release (Section 1, Lines 52 to 54). The other is multi-turn data. The text results in Table 1 come from our UltraChat corpus, whose statistics are reported in Appendix K.

UltraChat corpus statisticValue
Conversations with two or more assistant turns26,195 of 26,196
Conversations with a later assistant turn inside the 1,024-token training window23,861
Assistant tokens inside that window that follow the first response49.8%

These results show that replacing a response also changes the context of later turns for about half the assistant-token supervision.

When responses may be replaced, ALR + IRA matches both length-matched regeneration references after three epochs on ALLaVA. It keeps every stored response unchanged (Table 4a).

Method in the regeneration comparisonAccepted length
ALR + IRA3.841
Truncated regeneration3.833
Loss-window regeneration3.856

These results show that ALR + IRA lies between the two regeneration references. When a COCO caption response begins with a human-written sentence, text the target did not write, it exceeds both references (Table R5).

The paper studies recovering regeneration's accepted length without replacing stored responses. This matters because reviewed releases and multi-turn conversations require their stored text, as Section 1, Lines 52 to 54 and the revised Appendix K explain. ALR + IRA reaches the length-matched regeneration references after three epochs on ALLaVA while keeping that text.

AuthorsOpenReview ↗

General Response (3/6)

New results on the shared questions

Answer. ALR + IRA exceeds regeneration after a human-written caption prefix on four datasets and exceeds erase with a Llama target and Medusa-style heads.

Four new results answer the shared questions. When a caption response begins with text the target did not write, ALR + IRA exceeds both length-matched regeneration references at both temperatures, on COCO (Table R5) and on Flickr30k, NoCaps and TextCaps (Table R11). Every training replicate of ALR + IRA is above every replicate of both references at both temperatures. In multi-turn data, we replace the first response with the target's own greedy regeneration. An LLM judge then flags the stored next user turn as not fitting more often with this regeneration than with the stored response, also when the regeneration is cut to the length of the stored response (Table R10, Appendix K). ALR + IRA instead trains on each conversation as stored. With a Llama target, ALR + IRA exceeds both erase and DFlash (Table R8). With Medusa-style heads in place of the block drafter, it gives the longest aggregate accepted length of the four label rules (Table R9). Among the methods that keep the stored responses, one epoch of ALR + IRA exceeds every erase schedule up to six epochs, in 41% less training time than the best one (Table R1, Table 2). All intervals in this response are 95% intervals. Preserving stored responses therefore gives an acceptance advantage after human-written text and keeps the conversation that later user turns were written for.

To test whether the two routes differ after text the target did not write, we ran a new experiment on caption responses that begin with human-written text. Table R5 starts each COCO caption response with the first sentence of a human caption of the same image. We also score one fixed text after the prefix, the target's own greedy continuation. Table R11 repeats the prefix test on Flickr30k, NoCaps and TextCaps and pools the three datasets.

Caption continuation comparisonOver truncated regenerationOver loss-window regeneration
COCO, T=0, human caption prefix2.64%1.84%
COCO, T=1, human caption prefix3.53%2.87%
COCO, fixed greedy continuation after the prefix2.62%2.07%
Three more datasets pooled, T=0, human caption prefix3.60%3.08%
Three more datasets pooled, T=1, human caption prefix3.12%2.88%

These results show that ALR + IRA exceeds both references after human-written text. Every gain in this table has an interval above zero. On COCO without the prefix, ALR + IRA is within 1% of both (Appendix K).

To see whether a regenerated first response still fits the stored next user turn, we ran a new measurement on multi-turn conversations. Table R10 samples 1,000 multi-turn UltraChat conversations of the text corpus. They are drawn from conversations where greedy regeneration of the first response, stopped at the stored response's token length, ends inside the training window. We then replace the first stored assistant response with a greedy regeneration of Qwen3-4B with thinking off. It runs without that limit until it ends, with at most 4,096 new tokens. The LLM judge is Qwen3-32B, with one prompt for both versions in separate requests. It flags a stored next user turn that is not a coherent follow-up or refers to content the response does not contain.

Multi-turn comparisonRegenerated first responseStored first responseDifference or interval
Either judge flag15.7%5.5%+10.2 points, paired interval +8.0 to +12.4 points
Not a coherent follow-up (Appendix K)7.6%0.5%Paired interval +5.5 to +8.8 points
Next user turn starts beyond the training window4205Appendix K
Either flag in the other 580 conversations14.0%7.8%Paired interval +3.8 to +8.8 points
Either flag with regeneration cut to stored-response token length18.1%5.5%Appendix K

These results show that a regenerated first response fits the next stored user turn less often. A turn that starts beyond the training window, and every later turn, leaves the training text.

Regeneration therefore falls behind ALR + IRA after a human-written start of a caption on four datasets, and a regenerated response fits the stored next user turn less often than the stored response does.

The revised Appendices K and Q report the caption and multi-turn evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.