- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes REST, a single-stage distillation framework that trains a few-step, CFG-free student alongside a forward-process diffusion RL teacher such as RAM or DiffusionNFT.
During training, the student imitates -step segment velocities, , taken from the teacher's reward-scored, CFG-guided rollouts. The teacher continues to improve through RL, while the student learns a few-step, CFG-free generator from the evolving teacher. Because the rollouts are reused, no additional sampling, reward calls or distillation data are required.
The paper also introduces an Advantage-Modulated Distillation (AMD) objective, which weights each trajectory's loss by , with and . Since this weight can be negative, low-reward rollouts push the student away. An EMA regularizer is added, and an auxiliary DMD loss complements AMD in the 4-step setting.
Evaluation uses SD3.5-M post-trained on GenEval, OCR and PickScore. The 8-step student roughly matches its 40-step teacher (GenEval 0.94 vs. 0.95, OCR 0.96 vs. 0.96, DrawBench PickScore 23.96 vs. 23.95; Tab. 1). The 4-step student outperforms RTDMD (23.97 vs. 23.71) with fewer training iterations (Tab. 2).
Critical strengths
-
A simple recipe. REST reuses the rollouts and rewards already produced during the teacher's RL training. It requires no extra sampling, reward calls or distillation dataset, and it works with two different teacher algorithms (RAM and DiffusionNFT).
-
Strong results for the compute spent. REST-NFT and REST-RAM incur only 32.8% and 26.1% training-time overhead over standard DiffusionNFT and RAM training, respectively. In return, they yield a 4/8-step, CFG-free student that matches the 40-step CFG teacher. The 4-step student also outperforms RTDMD with substantially fewer training iterations (1,000 vs. roughly 2,500–3,000).
Critical weaknesses
-
The signed AMD weighting may be sensitive to the shift .
- With , AMD weights lie in . Low-reward trajectories therefore receive negative loss weights, which could destabilize training.
- The sensitivity is visible in the paper's own ablation: sharply degrades performance (PickScore 21.48; App. E, Table 5), while , where all weights are non-negative, performs similarly to uniform weighting (23.76).
- The paper does not report what fraction of trajectories receives negative weights.
-
The value of single-stage co-training is not isolated from AMD.
- In Tab. 3, co-training without AMD (23.74) scores below RL-then-distill (23.83). All of REST's gain over the two-stage pipeline therefore comes from AMD (23.96).
- The paper presents co-training's benefit mainly as efficiency (it "retains most of the gains" of RL-then-distill "with far fewer training iterations"), yet contribution 3 claims "substantially better alignment performance" than two-stage pipelines.
- AMD only requires stored, scored rollouts, so it could equally be applied in the two-stage pipeline.
- Until "RL then distill + AMD" is compared, it is unclear whether the paper's central framing, RL-native single-stage distillation, drives the gains, or whether the contribution is better described as advantage-weighted distillation.
Questions for authors
- In the 8-step PickScore setting of Tab. 3, how does "RL then distill + AMD" on stored, scored rollouts perform, and at what iteration cost relative to REST? This would show whether REST's advantage over two-stage pipelines lies in quality, efficiency, or both. Please also report the iteration count for each row of Tab. 3.
- With , what fraction of trajectories receives a negative weight during training, and what keeps the negatively weighted loss from diverging? Does a non-negative variant, e.g. , retain the gain?
Response part 3/4
4. Relation to RG-LCD, Hyper-SD, and RDM, and the role of auxiliary DMD
Reviewer items: Q4.
4.1. Comparison with RG-LCD, Hyper-SD, and RDM
We appreciate your careful attention to the relationship with earlier joint methods. This comparison is important for distinguishing single-stage training itself from the particular co-training design and performance-retention goal of REST.
As discussed in Section 1 of the common response, we aim to preserve the performance of strong full-step RL methods while obtaining a few-step generator in a single training stage. Our emphasis goes beyond using reward optimization to partly compensate for quality loss caused by distillation, as in works that take a distillation perspective, such as RG-LCD, Hyper-SD and RDM (Table R2.2).
At the methodological level, however, some of these methods, such as RDM, also combine reward optimization and distillation in a single stage. We therefore acknowledge that our statement, “the first single-stage framework for RL-distillation collaboration,” is not sufficiently precise. We will revise it to “REST is a single-stage framework that accomplishes both post-training tasks—RL and few-step distillation—while preserving the full performance gains of the original RL method.” We will also expand the comparison with and discussion of these methods in the revised manuscript to clarify the differences in training design and performance goals.
The performance goal is discussed in Section 1 of the common response. The table below distinguishes whether a full-step RL comparison is reported and what it shows. An original pretrained multi-step teacher is not counted as a full-step RL reference.
Table R2.2. Reported full-step RL comparisons in related work.
Here, "No" for methods without a full-step RL comparison means that retention has not been demonstrated. The remaining entries describe reported scores, rather than statistical equivalence. DMDR does not fully specify matched training conditions for its RL references. RTDMD's Flow-GRPO results appear to be taken from the original paper, and sufficient GenEval experimental details are not provided to assess whether the comparison is fair.
Our student-teacher comparisons with RAM and DiffusionNFT use the same training data and reward models under a common evaluation protocol, demonstrating that the RL performance retention milestone can be reached with few-step, CFG-free inference (Table 1).