acceptodds
Under review as a conference paper at ICLR 2027

RL-Native Distillation: Exploiting Scored Trajectories for Few-Step Image Generation

Abstract

Efficient text-to-image generation requires both reinforcement-learning (RL)-based reward alignment and few-step distillation, yet these procedures are typically performed sequentially, increasing training cost and risking the loss of reward gains during compression. We take an RL-native perspective: diffusion RL already generates reward-scored finite-step trajectories, whose intermediate states provide distillation supervision. Based on this insight, we propose REST (Reward-Enhanced Scored-Trajectory Distillation), a single-stage co-training framework in which a decoupled student learns from the evolving RL teacher's trajectories without changing teacher optimization. Advantage-Modulated Distillation (AMD) transforms rollout advantages into signed weights, strengthening imitation of preferred trajectories and aligning distillation priorities with task value. The resulting framework is general and lightweight, requires no extra image rollouts, no separate distillation dataset, and no adversarial training. Experiments on compositional generation, visual text rendering, and human-preference alignment demonstrate competitive few-step, CFG-free generation with RAM or DiffusionNFT teachers. With only four sampling steps, REST-RAM achieves a DrawBench PickScore of 23.97, outperforming both the 40-step RAM teacher (23.95) and RTDMD (23.71).

Primary area:
generative models
Keywords:
DiT, Diffusion distillation, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n2kaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes REST, a single-stage distillation framework that trains a few-step, CFG-free student alongside a forward-process diffusion RL teacher such as RAM or DiffusionNFT.

During training, the student imitates KK-step segment velocities, ak=(xtqk+1−xtqk)/(tqk+1−tqk)a_k = (x_{t_{q_{k+1}}} - x_{t_{q_k}})/(t_{q_{k+1}} - t_{q_k}), taken from the teacher's reward-scored, CFG-guided rollouts. The teacher continues to improve through RL, while the student learns a few-step, CFG-free generator from the evolving teacher. Because the rollouts are reused, no additional sampling, reward calls or distillation data are required.

The paper also introduces an Advantage-Modulated Distillation (AMD) objective, which weights each trajectory's loss by λ(Amix+b)\lambda(A_{\text{mix}} + b), with λ=1\lambda = 1 and b=0.5b = 0.5. Since this weight can be negative, low-reward rollouts push the student away. An EMA regularizer is added, and an auxiliary DMD loss complements AMD in the 4-step setting.

Evaluation uses SD3.5-M post-trained on GenEval, OCR and PickScore. The 8-step student roughly matches its 40-step teacher (GenEval 0.94 vs. 0.95, OCR 0.96 vs. 0.96, DrawBench PickScore 23.96 vs. 23.95; Tab. 1). The 4-step student outperforms RTDMD (23.97 vs. 23.71) with fewer training iterations (Tab. 2).

Critical strengths
  1. A simple recipe. REST reuses the rollouts and rewards already produced during the teacher's RL training. It requires no extra sampling, reward calls or distillation dataset, and it works with two different teacher algorithms (RAM and DiffusionNFT).

  2. Strong results for the compute spent. REST-NFT and REST-RAM incur only 32.8% and 26.1% training-time overhead over standard DiffusionNFT and RAM training, respectively. In return, they yield a 4/8-step, CFG-free student that matches the 40-step CFG teacher. The 4-step student also outperforms RTDMD with substantially fewer training iterations (1,000 vs. roughly 2,500–3,000).

Critical weaknesses
  1. The signed AMD weighting may be sensitive to the shift bb.

    • With b=0.5b = 0.5, AMD weights lie in [−0.5,1.5][-0.5, 1.5]. Low-reward trajectories therefore receive negative loss weights, which could destabilize training.
    • The sensitivity is visible in the paper's own ablation: b=0b = 0 sharply degrades performance (PickScore 21.48; App. E, Table 5), while b=1b = 1, where all weights are non-negative, performs similarly to uniform weighting (23.76).
    • The paper does not report what fraction of trajectories receives negative weights.
  2. The value of single-stage co-training is not isolated from AMD.

    • In Tab. 3, co-training without AMD (23.74) scores below RL-then-distill (23.83). All of REST's gain over the two-stage pipeline therefore comes from AMD (23.96).
    • The paper presents co-training's benefit mainly as efficiency (it "retains most of the gains" of RL-then-distill "with far fewer training iterations"), yet contribution 3 claims "substantially better alignment performance" than two-stage pipelines.
    • AMD only requires stored, scored rollouts, so it could equally be applied in the two-stage pipeline.
    • Until "RL then distill + AMD" is compared, it is unclear whether the paper's central framing, RL-native single-stage distillation, drives the gains, or whether the contribution is better described as advantage-weighted distillation.
Questions for authors
  1. In the 8-step PickScore setting of Tab. 3, how does "RL then distill + AMD" on stored, scored rollouts perform, and at what iteration cost relative to REST? This would show whether REST's advantage over two-stage pipelines lies in quality, efficiency, or both. Please also report the iteration count for each row of Tab. 3.
  2. With b=0.5b = 0.5, what fraction of trajectories receives a negative weight during training, and what keeps the negatively weighted loss from diverging? Does a non-negative variant, e.g. max⁡(Amix+b,0)\max(A_{\text{mix}} + b, 0), retain the gain?
Reviewer_kjx6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REST, a single-stage framework that trains a few-step, CFG-free student alongside a diffusion RL teacher. The teacher is optimized with an existing forward-process RL algorithm (RAM or DiffusionNFT) on SD3.5-Medium with LoRA. The student reuses the teacher's CFG rollouts (MM steps, e.g. M=20M=20): it regresses the piecewise velocity of K-step subsets of each trajectory (K = 4 or 8; Eq. 6), so no extra rollouts are sampled. Advantage-Modulated Distillation (AMD, Eq. 12) weights each rollout's segment losses by λ(Amix+b)\lambda(A_{mix}+b), with clipped group-normalized advantages Amix∈[−1,1]A_{mix}\in[-1,1], λ=1\lambda=1 and b=0.5b=0.5, so low-advantage rollouts can receive negative weight. The student also has an EMA self-regularizer (Eq. 13), and the 4-step model adds an auxiliary DMD loss with a learned fake-score model. Table 1 reports 8-step REST-RAM on GenEval, OCR and PickScore, with 4-step REST-RAM and 4/8-step REST-NFT additionally evaluated on PickScore. In the evaluated settings, the students are reported to be comparable to their 40-step teachers. For example, 4-step REST-RAM reaches a DrawBench PickScore of 23.97, compared with 23.95 for 40-step RAM and 23.71 for a reproduced RTDMD (Table 2). The reported training-time overhead is 26–33% over the RL run alone. Reported ablations (Tables 3–5) cover sequential pipelines, co-training without AMD, AMD/DMD variants, the shift bb and the EMA term. The appendix also presents a PCM-based variant qualitatively (App. B).

Critical strengths

1. A simple, low-overhead way to obtain a few-step model from an RL run. REST reuses the rollouts and rewards that the RL loop already computes. It adds a student branch at a reported 26.1% (RAM) / 32.8% (DiffusionNFT) training-time overhead (p. 9), with no separate distillation dataset or stage. The resulting 4-step CFG-free student has similar reported scores to its 40-step CFG teacher on the preference task (Table 1: training PickScore 23.72 vs. 23.70; DrawBench ImageReward 1.38 vs. 1.35). Running the RL model directly at 8 steps loses noticeably more (training PickScore 23.30, DrawBench PickScore 23.55). Obtaining a few-step student within the RL run has clear practical value.

2. The student is compared against its own full-step RL teacher under a common protocol. The paper notes (p. 8) that prior RL+distillation work does not measure how much of the RL policy's gain survives distillation. Table 1 makes this comparison for RAM on three tasks and includes DiffusionNFT-based students for preference alignment, reporting out-of-domain DrawBench metrics alongside the training reward. This directly evaluates the retention of the teacher's gains, although uncertainty estimates would make the comparison more informative.

Critical weaknesses

1. Uncertainty estimates and clearer baseline details would strengthen the claims about teacher comparisons and individual components. The abstract states that 4-step REST-RAM outperforms the 40-step teacher, based on a DrawBench PickScore of 23.97 vs. 23.95 (Table 1). Several component comparisons also have small reported gaps: AMD-only vs. full, 23.93 vs. 23.97 (Table 4); without vs. with the EMA term, 23.91 vs. 23.96 (Table 5). The reported tables provide point estimates without seed counts, uncertainty estimates or the evaluation sample sizes needed to judge these gaps, and PickScore is both the training reward and the headline metric. The substantial improvement over direct few-step sampling is practically valuable. Uncertainty estimates would help determine whether the smaller differences support conclusions about superiority over the teacher and the contribution of individual components. The PCM evidence reported in Fig. 5 (App. B) consists of six qualitative image pairs rather than a quantitative comparison. More detail on the sequential baselines in Table 3 (distillation objective, data and tuning) would also help readers assess the comparison. Fig. 1 compares iteration counts without per-iteration costs, making its comparison of sequential-pipeline efficiency difficult to assess despite the separately reported overhead relative to RL alone.

2. The interpretation of signed weighting and its relationship to earlier reward-guided distillation could be explained more clearly. Sec. 3.3 presents AMD through an AWR interpretation, while the actual coefficients can be negative. With b=1b=1 (weights in [0,2][0,2]), the reported 8-step DrawBench PickScore is 23.76 (Table 5), close to the 23.74 of uniform imitation (RL & Distill, Table 3); without uncertainty estimates, this does not establish whether non-negative weighting offers a consistent improvement. At b=0.5b=0.5, the reported score is 23.96 and rollouts with Amix<−0.5A_{mix}<-0.5 receive negative weights, producing repulsive rather than imitative updates. With b=0b=0, the reported score falls to 21.48 (ImageReward 0.09). These results are consistent with a benefit from signed weighting. Since changing bb also changes positive weights and overall loss scaling, further analysis would help distinguish the effect of repulsion from these other changes. A clearer connection to conventional non-negative AWR weighting and an explanation of stability under the complete regularized objective would strengthen the method. On novelty, the paper claims to be "the first single-stage framework for RL-distillation collaboration" (p. 2). It would be helpful to distinguish the evolving-teacher co-training setting more explicitly from earlier reward-guided few-step distillation, such as RG-LCD and Hyper-SD, and related work such as RDM. The combination of reusing an evolving RL teacher's rollouts and signed weighting is potentially useful; a clearer statement of the specific contribution would help readers appreciate how it relates to these methods.

Questions for authors
  1. [Clarification] Could you provide the number of DrawBench prompts and images per prompt, the seed protocol, and any available uncertainty estimates for the key student–teacher and component comparisons (Tables 1, 3–5)? [Suggestion] Repeated sampling evaluations or additional training seeds would help assess these gaps. If the student–teacher difference is not statistically resolved, describing the result as "matching" rather than "outperforming" in the abstract would better reflect the available evidence. Clarifying this uncertainty would be particularly helpful for my assessment.
  2. [Clarification] For "RL then Distill" and "Distill then RL" (Table 3, Fig. 1), could you describe the distillation loss, data, CFG setting and tuning protocol? It would also help to clarify the compute accounting behind the efficiency comparison, including the 4-step DMD branch and fake-score model; GPU-hours, if available, would make the iteration-based comparison easier to interpret.
  3. [Clarification] Could you report the fraction of rollouts receiving negative weight at b=0.5b=0.5 and the αi\alpha_i used in Eq. 11? An explanation of the connection to non-negative AWR weighting and the role of regularization in stabilizing repulsive updates would also be useful. [Suggestion] A comparison that clips negative weights while matching positive weighting and loss scale would help isolate the role of repulsion; broader sensitivity results would also be useful if available.
  4. [Clarification] Could you clarify how the evolving-teacher co-training setting differs from reward-guided distillation such as RG-LCD and Hyper-SD, and from RDM, highlighting the specific contribution of the single-stage formulation? It would also be helpful to explain how the 4-step fake-score model relates to the discussion of multi-network designs in DMDR/RTDMD.
  5. [Suggestion] Quantitative evaluation of the PCM variant would strengthen the claim of generality across distillation losses. Results for 4-step students on GenEval/OCR would be a useful further extension, since the reported 4-step results are limited to PickScore training.
5 comments
AuthorsOpenReview ↗

Response part 3/4

4. Relation to RG-LCD, Hyper-SD, and RDM, and the role of auxiliary DMD

Reviewer items: Q4.

4.1. Comparison with RG-LCD, Hyper-SD, and RDM

We appreciate your careful attention to the relationship with earlier joint methods. This comparison is important for distinguishing single-stage training itself from the particular co-training design and performance-retention goal of REST.

As discussed in Section 1 of the common response, we aim to preserve the performance of strong full-step RL methods while obtaining a few-step generator in a single training stage. Our emphasis goes beyond using reward optimization to partly compensate for quality loss caused by distillation, as in works that take a distillation perspective, such as RG-LCD, Hyper-SD and RDM (Table R2.2).

At the methodological level, however, some of these methods, such as RDM, also combine reward optimization and distillation in a single stage. We therefore acknowledge that our statement, “the first single-stage framework for RL-distillation collaboration,” is not sufficiently precise. We will revise it to “REST is a single-stage framework that accomplishes both post-training tasks—RL and few-step distillation—while preserving the full performance gains of the original RL method.” We will also expand the comparison with and discussion of these methods in the revised manuscript to clarify the differences in training design and performance goals.

The performance goal is discussed in Section 1 of the common response. The table below distinguishes whether a full-step RL comparison is reported and what it shows. An original pretrained multi-step teacher is not counted as a full-step RL reference.

Table R2.2. Reported full-step RL comparisons in related work.

MethodFull-step RL comparison?Does it match full-step RL?
RG-LCDNo; the multi-step reference is the pretrained SD2.1 teacher.No.
Hyper-SDNo; comparisons use pretrained base models and other few-step methods, rather than a full-step ReFL model.No.
RDM / GNDMRYes; its own reproduced 50-step Flow-GRPO (Table 1).SD3.5-M: lower HPS/PickScore, 31.25/23.15 vs. 31.81/23.24. SD3-M: close, mixed results.
DMDRYes, with partial coverage: for the default ReFL branch (Tables 1–2); no corresponding full-step reference for its GRPO variant.Mixed: SD3-M 4-step HPS/PickScore exceed ReFL; SDXL 4-step values are lower. GRPO variant: No.
RTDMDYes, with partial coverage: full-step Flow-GRPO, GenEval only (Appendix L); no corresponding HPS/PickScore/OCR comparison.Near on GenEval: 0.94 vs. 0.95. Retention on other tasks is not demonstrated.

Here, "No" for methods without a full-step RL comparison means that retention has not been demonstrated. The remaining entries describe reported scores, rather than statistical equivalence. DMDR does not fully specify matched training conditions for its RL references. RTDMD's Flow-GRPO results appear to be taken from the original paper, and sufficient GenEval experimental details are not provided to assess whether the comparison is fair.

Our student-teacher comparisons with RAM and DiffusionNFT use the same training data and reward models under a common evaluation protocol, demonstrating that the RL performance retention milestone can be reached with few-step, CFG-free inference (Table 1).

AuthorsOpenReview ↗

Response part 4/4

4.2. The role of auxiliary DMD in REST compared with DMDR/RTDMD

Our concerns about DMDR/RTDMD include their use of DMD warm-up followed by joint RL+DMD optimization, resulting in multi-stage training. Their few-step generation capability is obtained through DMD, so they rely on an additional fake-score model. REST learns this capability through trajectory distillation and uses DMD only as an optional auxiliary student loss to compensate for the slight loss of visual quality in four-step trajectory distillation; DMD is not needed at eight steps. We believe the community will continue to develop trajectory distillation methods that improve on plain trajectory distillation, such as Parallel Decoding Distillation (PDD), making it possible to remove auxiliary DMD, as discussed below. Their DMD real model is a frozen, RL-agnostic base model, whereas ours is the evolving RL teacher. The optional four-step DMD extension in REST does add a fake-score model, whose training cost is included in Table C3.

REST does not rely on DMD

REST’s core mechanism is independent of DMD: the teacher retains its original RL training, while a separate student reuses the teacher’s trajectories and rewards with different distillation losses. Plain trajectory distillation and PCM both work well with AMD at eight steps (Table 1 and Table R2.3, respectively). At four steps, we observed slightly lower visual quality, although plain trajectory distillation with AMD already achieves a strong PickScore of 23.93 without DMD (Table 4). We therefore added DMD only as an auxiliary loss to improve visual quality. This reflects a limitation of the distillation objectives used in our current four-step implementation; the REST framework can accommodate improved objectives. Recent work has introduced promising trajectory-based distillation methods, such as Parallel Decoding Distillation (PDD) and Trajectory-Regularized Distribution Distillation (TRD). In future work, we will explore incorporating these methods into REST to eliminate the need for auxiliary DMD in the four-step setting.

An evolving RL teacher vs. a frozen, RL-agnostic base model

Our real model is the evolving RL teacher, whereas DMDR/RTDMD use a frozen, RL-agnostic base model as their DMD real model. Matching a frozen, RL-agnostic real model may pull against reward optimization.

Auxiliary DMD vs. auxiliary RL

REST uses DMD only as an auxiliary loss in the student branch and adapts it to reuse the teacher’s existing trajectories. The teacher’s original RL training remains unchanged. By contrast, DMDR’s ReFL branch adds a reward-maximization loss to the DMD training pipeline to improve the distilled generator. This exemplifies the distillation perspective discussed in Section 1 of the common response. From our RL-native perspective, the explicit performance goal of compression is to preserve the gains attainable by the original strong RL method.

5. Quantitative PCM and four-step OCR/GenEval

Reviewer items: Q5.

Thank you for requesting quantitative PCM+AMD results and the four-step task extensions. These are valuable tests of whether the benefit extends across distillation methods and task settings. We report the quantitative PCM+AMD comparison below.

Table R2.3. PCM+AMD quantitative evaluation.

ModelStepsPickScore ↑HPSv2 ↑Aesthetic ↑ImageReward ↑DeQA ↑
PCM823.780.326.221.324.13
PCM + AMD823.930.336.291.364.18

The four-step OCR and GenEval results are presented in Section 6 and Tables C4–C5 of the common response. Each newly added run uses 300 updates, with the remaining hyperparameters matching the four-step PickScore experiments.

Thank you again for the care and breadth of your review. Your comments have helped us strengthen statistical reporting, clarify AMD's interpretation and its relationship to prior work, and extend the experimental coverage. These improvements make the evidence for REST's contributions easier to assess.


Dear Reviewer,

Thank you again for your valuable feedback. We have provided a detailed response to your concerns and would greatly appreciate it if you could take a look when you have a chance.

We would be grateful to know whether our clarifications have adequately addressed your main concerns, or if there are any remaining questions that we could further clarify during the discussion period.

Thank you for your time and consideration.

Reviewer_ckdLOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes REST, a framework for joint RL alignment and few-step distillation. A separate student reuses the RL teacher’s reward-scored trajectories and learns to match segment velocities. AMD uses normalized reward advantages to assign signed weights to the distillation loss. An EMA penalty stabilizes training. The four-step version also uses an auxiliary DMD loss. Experiments use SD3.5 Medium with RAM and DiffusionNFT teachers. The 4-step and 8-step student largely preserves competitive performance.

Critical strengths
  • The reuse of RL trajectories has practical value. The student learns from trajectories and reward scores already produced by teacher training. This avoids separate student rollouts for distillation. The teacher’s RL objective remains unchanged, which makes the framework compatible with existing RL pipelines.

  • The eight-step results show good retention of teacher capabilities. REST-RAM achieves a GenEval score of 0.94, compared with 0.95 for its teacher. Both reach an OCR score of 0.96. Generic image-quality metrics also remain broadly comparable. These results support the framework’s ability to reduce sampling steps while preserving much of the teacher’s performance.

Critical weaknesses
  • The novelty of AMD and the benefit of negative weights is unclear. AWM already uses signed advantage weights for score/flow-matching losses. AMD applies this idea to teacher-trajectory distillation and adds a positive offset. This is a useful adaptation, but its methodological contribution needs clearer explanation. Table 3 shows that AMD improves over uniform distillation. However, it does not establish whether negative weights help beyond giving more weight to high-reward trajectories. Comparisons with nonnegative weighting are needed to assess this benefit and its effect on stability.

  • The advantages of online co-training over sequential training are not clearly isolated. Reusing RL rollouts is a sensible design. However, in Table 3, co-training without AMD scores 23.74, compared with 23.83 for RL followed by distillation. Full REST reaches 23.96, but also adds AMD. This comparison therefore mixes the effects of the training schedule and reward weighting. Moreover, fewer iterations do not necessarily imply lower compute cost. A sequential baseline using the same AMD objective and a matched compute budget would clarify the practical gains.

  • The compute and memory cost is actually high. Though under lora training, there needs to maintain multiple models: RL model, distilled model, fake score. And fake score needs to keep a updating frequency to distilled model. The whole combination makes the system heavy.

  • Distill then RL is not a very suitable baseline: distilled model cannot give correct scores, directly apply NFT objective on it could bring additional adaptation cost for learning scores and instability. As table3 shows, its benchmark scores are generally weaker than others.

Questions for authors
  • REST is a system that is loosely combined, which the RL loss and DMD loss is optimized actually separated. The question is what does REST want to get? (or the motivation)

    • REST wants to get a reward-optimized distilled student.
    • REST wants to get a reward-optimized RL model with a byproduct of distilled student. In the 1st case, directly bringing the whole RL process is too heavy. In the 2nd case, the novelty is very limited: only using a very straight combination of RL and distill process which is not very practical in production level training.
  • Under the same teacher rollouts, EMA settings, and training budget, how does AMD compare with clipped nonnegative weights, w=max⁡(A+0.5,0)w=\max(A+0.5,0), and exponential advantage weights? With comparable gradient scales, do negative weights provide a consistent benefit, and how do they affect stability?

  • With the same AMD objective, how does online co-training compare with distilling the final RL teacher?

  • Could you report total GPU hours for online and sequential training, including rollout generation,and teacher and student updates? Does online co-training retain an advantage under an equal compute budget?

10 comments

Thank you for the additional experiments. The nonnegative controls address my concern regarding negative weights in this setting, and the sequential baseline under the same AMD objective confirms that both schedules achieve comparable quality.

My remaining concern is the computational cost of co-training. While reusing rollouts is efficient when the full-step RL teacher is trained anyway, the full RL run becomes an inherent overhead if the end goal is a few-step student. Although the image experiments offer a useful reference, applying this approach to video, where both teacher rollouts and student updates are substantially more expensive, which is burdensome

AuthorsOpenReview ↗

Response 1/2

Thank you for acknowledging that the nonnegative controls address your concern about negative weights and that online and sequential training with the same AMD objective achieve comparable quality. We appreciate your remaining concern about computational cost and would like to clarify the intended comparison and the practical motivation for our framework.

1. Clarifying the efficiency comparison

We understand your emphasis on the cost of obtaining a few-step student. Our goal is not simply few-step distillation, as your phrase “if the end goal is a few-step student” seems to suggest. We would like to clarify again that our goal is joint RL and distillation: obtaining a few-step student whose performance matches its strong full-step RL teacher. This performance target goes beyond that of RTDMD and DMDR, as discussed in Section 1 of the common response and our earlier reply on motivation.

For this goal, we believe the most informative comparisons are with an RL-then-distill pipeline or other joint methods, since these workflows address both reward alignment and few-step generation. Distillation alone addresses a different training objective. Our contribution is therefore a framework for joint post-training, rather than a general-purpose distillation method.

2. The role of RL in industrial post-training

We recognize that RL adds computation compared with distillation alone. If the concern is that joint RL and distillation itself is too heavy compared with distillation alone, as the phrase “if the end goal is a few-step student” might suggest, acutally, RL is indeed already part of post-training in industry, whether it is combined with distillation or performed separately.

This is explicitly documented in several image-generation technical reports:

  • Qwen-Image, Section 4.2.2, includes RL after SFT.
  • Qwen-Image-2.0, Section 4.2, refines the base diffusion model through RLHF using an adapted GRPO framework.
  • Seedream 3.0, Section 2.3, and Seedream 4.0, Section 2.2, include RLHF in their post-training pipelines.
  • HunyuanImage 3.0, Section 4.2, uses MixGRPO and further reward-based refinement after SFT and DPO.
  • Z-Image, Sections 4.5.2 and 4.6, incorporates RL into distillation and also uses GRPO for online refinement.
  • Krea 2 uses multi-reward GRPO and explicitly states: “After the RL stage, we include an optional timestep-distillation stage”.

REST does not introduce a heavy RL stage out of nowhere; it aims to combine two existing stages into one.

These examples motivate REST’s design: combining two existing post-training tasks within one training stage. We therefore hope its computational cost can be assessed in the context of workflows that seek both reward alignment and efficient generation. The comparison should reflect the actual production workflow.

3. Where does REST stand among joint methods?

Compared with an RL-then-distill pipeline, our results indicate that REST reduces computational cost and simplifies the workflow by reusing the teacher’s trajectories and rewards, a benefit also recognized by several reviewers.

Our computational savings already include the full teacher RL run. We would also clarify that teacher rollouts during RL training use 10 or 20 denoising steps, depending on the RL algorithm, rather than the 40-step configuration used for evaluation. Table C3 accounts for teacher and student training, rollout generation, reward evaluation, and fake-score updates where applicable. Based on timing measurements on eight H20 GPUs, eight-step REST reduces the estimated total training cost by approximately 23.6% compared with RL followed by trajectory distillation. In the four-step setting with auxiliary DMD, the corresponding reduction is approximately 26.7% relative to the sequential workflow with auxiliary DMD. These savings arise from reusing teacher rollouts and reward evaluations within a single training stage, without requiring a trained RL teacher to be available beforehand.

We would welcome clarification of any particular absolute computational budget or relative cost criterion underlying your remaining concern. This would help us address it more directly while considering both computational cost and whether strong full-step RL performance is retained.

For context, DMDR also introduces a framework for collaboration between DMD distillation and RL training. It has been accepted at ECCV 2026 and has received over 40 citations in less than a year, while involving more hyperparameters and more complex optimization strategies. We view this as further interest in the broader research direction of combining these two post-training tasks.

AuthorsOpenReview ↗

Response 2/2

4. The focus of our paper

Our paper focuses on image generation: our title states “Image Generation”, and our experiments evaluate this task. We appreciate the video example as a useful consideration for extending the framework, while our present contribution is established in the image domain.

In the broader lifecycle of image and video generation systems, pretraining and continue training typically account for major computational costs. RL and few-step distillation operate within post-training, sharing a reletively small part of computational cost.

5. Considering efficiency alongside other contributions

We believe that REST offers efficiency advantages and a more streamlined optimization scheme. Regardless of whether REST is considered computationally heavy, we also hope computational efficiency can be considered alongside the performance target, methodological contribution, and practical applicability of the framework.

Additional training computation is a trade-off in related methods. For example, DMD trains an additional fake-score model beyond student distillation. Similarly, the original DETR was trained for 500 epochs, substantially more than the approximately 36 epochs commonly used by CNN-based detectors in 2020. Such costs are relevant to evaluation, alongside the capabilities and methodological advances these approaches provide.

As discussed in Section 2 of the common response, we systematically show that the milestone of retaining strong full-step RL performance with few-step generation can be reached. Our decoupled design and AMD establish a useful baseline for reaching this milestone. REST also offers practical value: it supports different RL methods and distillation losses, allowing it to adapt to different application settings and reducing the effort needed for adaptation and manual hyperparameter tuning.

We hope these contributions can be considered together with the computational comparison. Our intention is for REST to provide a useful baseline and encourage the development of more efficient methods that reach the same performance-retention milestone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.