- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a training framework for speculative decoding on fixed off-policy corpora. It introduces two components: Anchor-Label Relabelling (ALR), which replaces corpus-conditioned labels with target-generated rollout distributions, and In-Rollout Anchors (IRA), which exposes the drafter to target-generated contexts. Experiments on DFlash demonstrate improvements in acceptance length and decoding speedup over conventional training and erasing-based baselines.
Critical strengths
- The paper clearly identifies the supervision mismatch caused by off-policy training data for parallel drafters.
- The proposed methods are technically straightforward and supported by extensive experiments and ablation studies.
- The improvements over DFlash and erasing-based baselines are consistent across multiple models and datasets.
Critical weaknesses
-
The motivation is insufficiently justified. The central premise is that the original training corpus cannot be modified. However, regenerating responses using the target model is already a standard practice in speculative decoding training, supported by widely used frameworks such as SpecForge and Speculators. Modern inference engines such as vLLM and SGLang make this process highly efficient. More importantly, the paper does not convincingly explain why replacing the original responses is unacceptable while replacing their supervision labels and introducing target-generated contexts through ALR and IRA is permissible.
-
The practical advantage over response regeneration is unclear. Table 4 shows that directly regenerating responses achieves essentially the same acceptance length as ALR + IRA (3.833/3.856 vs. 3.841). Therefore, the proposed framework does not demonstrate a performance advantage over the standard regeneration approach.
-
The computational efficiency claim lacks an appropriate comparison. Response regeneration only needs to be performed once, and the resulting data can be reused across training epochs. In contrast, ALR + IRA repeatedly performs target rollouts during training, increasing per-epoch computational costs. The paper does not compare the total cost of regeneration + standard training against ALR + IRA training. Without this comparison, it is unclear whether the proposed approach offers any practical computational advantage.
Questions for authors
- What realistic scenarios require preserving the original training responses while allowing their supervision labels and contexts to be replaced with target-generated content? How common are these constraints in practical speculative decoding training?
- Can the authors provide an end-to-end computational cost comparison against target response regeneration followed by standard drafter training, particularly over multiple training epochs?
- Given that response regeneration achieves nearly identical acceptance length, what concrete advantage does ALR + IRA offer over simply regenerating the training corpus?
18 comments
Response to Reviewer 9xH9 (2/11)
W1. Training on stored responses is established
"The central premise is that the original training corpus cannot be modified. However, regenerating responses using the target model is already a standard practice in speculative decoding training, supported by widely used frameworks such as SpecForge and Speculators."
Answer. The paper studies training a drafter to reach regeneration's accepted length when stored responses must remain the training text. It does not argue that regeneration is unavailable or uncommon. Regeneration replaces that text and must be repeated for each new target.
To compare the stored-response route with regeneration, the table below collects the accepted lengths of Table 4a and the results of Tables R5, R10, R11 and R12.
| Comparison | ALR + IRA | Regeneration |
|---|---|---|
| Accepted length, Table 4a | 3.841 | 3.833 truncated, 3.856 loss window |
| Next user turn flagged, Table R10 | 5.5% with stored response | 15.7% with regenerated response |
| Human caption prefix, COCO, T=0 and T=1, Table R5 | 3.628 and 3.206 accepted length | 3.534 and 3.097 truncated, 3.562 and 3.116 loss window |
| Human caption prefix, three datasets pooled, T=0, Table R11 | +3.60% over truncated, +3.08% over loss window | Truncated and loss-window references |
| Reuse after a target update, T=0, Table R12 | 3.841 accepted length | 3.501 with the previous target's corpus |
An audited or licensed release clears only its stored text, and Q1 and W3 measure three more cases for keeping the stored responses. In multi-turn data, a regenerated first response leaves the next stored user turn flagged as no longer fitting in 15.7% of the conversations, against 5.5% with the stored one (Table R10). After human-written text at the start of a caption response, ALR + IRA exceeds both regeneration references (Tables R5 and R11). A regenerated corpus reused after a target update trains a drafter below erase (Table R12). Keeping the stored responses preserves later-turn context and lets every target train on the same text. Table R3 under W2 lists the accepted lengths of the regeneration comparison.
The paper states regeneration as the route for corpora whose responses may be replaced. It cites regeneration for that case (Section 1, Lines 50 to 53, Section 2, Lines 111 to 117). The second route trains on the stored responses with target signals computed on them, and it is the route of widely used drafters. EAGLE (Li et al., 2024a), cited in Section 2, states the choice in its own words. It writes that "EAGLE uses a fixed dataset for training" (EAGLE, Section 4.3.3). HASS (Zhang et al., 2024) and Falcon (Gao et al., 2025) train on the same ShareGPT conversations as EAGLE. Section 2 cites both.
The frameworks named in the review support this route as well. SpecForge and Speculators both offer online training, in which the target's hidden states are computed inside the training run from the stored dataset. Both run regeneration as a separate generation pass over the corpus before training. Citing EAGLE, the SpecForge data-preparation guide notes that "the performance is still good even if you use the original dataset". It presents regeneration as a way to "improve the acceptance rate of the draft model". After three epochs on ALLaVA, ALR + IRA reaches the accepted length of length-matched regeneration without that pass or any replacement of a stored response (Table 4a).
The open question of this route is how much of the benefit of regeneration a drafter can recover while the stored responses stay the training text. EAGLE compared the two kinds of training data in its own ablation (EAGLE, Table 6). HASS reports that "HASS trained on the fixed dataset even achieves better performance than EAGLE-2 trained on the model-generated data consistently" (HASS, Appendix A.4). The erase baseline applies the confidence-adaptive weights of PARD-2 (An et al., 2026). These weights reduce supervision after corpus tokens the target is unlikely to produce (Section 2, Lines 93 to 97).
The paper studies recovering regeneration's accepted length while the stored responses remain the training text. This matters because reviewed releases and stored conversations require that text, as Section 1 of the revised PDF explains in Lines 53 to 57. After three epochs on ALLaVA, ALR + IRA reaches both length-matched regeneration references within 0.4% without replacing any stored response, and erase stays below all three (Table 4a).
Response to Reviewer 9xH9 (3/11)
W1, continued. Transient distillation signals preserve the corpus
"More importantly, the paper does not convincingly explain why replacing the original responses is unacceptable while replacing their supervision labels and introducing target-generated contexts through ALR and IRA is permissible."
The paper does not claim that replacing responses is unacceptable in general. Its constraint is on the training text, not on the use of target signals, and ALR + IRA never replaces that text.
Table 4a compares the same two kinds of training data as EAGLE and HASS, the stored responses and responses written by the target. A drafter of the EAGLE line learns from target hidden states computed on the stored corpus. In our comparison, every drafter, the DFlash baseline included, reads target hidden states computed inside the step. KD and erase also train on the target's next-token distributions along the corpus (Section 3.1, Equations 1 and 2). None of these signals is stored, and training on them does not modify the corpus. The concern in the review therefore applies to a method that replaces the stored responses, and ALR + IRA does not replace them.
ALR reads the same distributions along the target's own greedy continuation from each anchor in the corpus, at most 14 tokens long. IRA anchors half of the blocks inside these continuations and reuses their states (Section 3.2). This means that every block conditions on a stored prefix. A block anchored inside a continuation adds at most 14 target-written tokens to that prefix, its anchor token included. Section 3.2 states this at Lines 218 to 219 and 225 to 228. Nothing the target writes is stored or written back (Section 1, Lines 63 to 64).
Replacing the labels with target distributions is knowledge distillation, which Equations 1 and 2 already apply along the stored corpus for KD and erase. Replacing each stored response with a complete target response is sequence-level distillation (Kim and Rush, 2016, cited in Section 2). It produces a second corpus and changes the conversation that later turns were written for.
The paper studies how close a drafter can come to regeneration while every stored response stays the training text. This matters when only an audited or licensed release may be used for training, as Section 1 of the revised PDF explains in Lines 53 to 57, and in the multi-turn and human-written cases above. ALR + IRA keeps that text, as KD and erase do, and adds target text only as labels and as at most 14 tokens after a stored prefix.
The new Appendix K of the revised PDF expands this definition.
Response to Reviewer 9xH9 (1/11)
We thank the reviewer for the careful reading.
Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.
Our answers in brief.