acceptodds
Under review as a conference paper at ICLR 2027

ReDiF: Resource-Efficient Few-Step Diffusion Distillation via Reinforcement Learning

Abstract

Step distillation accelerates diffusion sampling by training a few-step student to imitate a many-step teacher, but distillation itself remains expensive. Typically, this requires thousands of GPU-hours and a large pre-generated trajectory dataset. We introduce ReDiF, which casts step distillation as terminal-reward policy optimization rather than step-wise regression. The student is optimized against a reward computed on the terminal sample, measuring alignment with the teacher's output, instead of matching the teacher's intermediate trajectory under a reconstruction or consistency loss. Because the reward need not be differentiable or trajectory-aligned, ReDiF admits non-differentiable objectives, multi-objective combinations, and preferences the teacher does not express, while exploration lets the student find sampling paths matched to its own step schedule. ReDiF converges in 400 policy updates with 3200 rollouts on a single A100 GPU with 1,000 noise-class pairs and no paired dataset: about 4 GPU-hours, against roughly 336 A100-hours reported for DMD2 on the same EDM teacher. At 8 steps on ImageNet-64, it achieves an FID at least 3.64 points better than prior state-of-the-art distillation methods in the low training regime under the same single-GPU budget. The formulation is also orthogonal to existing distillation objectives: added to the DMD2 loss, it further improves DMD2's FID by 4.4.

Primary area:
generative models
Keywords:
Step distillation, Reinforcement learning, Policy optimization, Efficient training, Few-step sampling, Diffusion models, PPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cfzQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a paper with a clear line of reasoning and attractive results achieved on a low budget; however, the experimental protocol and its positioning in terms of novelty are not yet sufficient to support strong conclusions.

I believe that ReDiF is indeed effective —‘single GPU, LoRA, very few training samples, and 5–8-step students’.However,it cannot yet be demonstrated that it is universally superior to existing distillation methods, nor can it be conclusively proven that its advantages stem from RL rather than experimentation budget, early stopping, parameterisation, ect.

Critical strengths

1.The paper's strongest idea is simple. ReDiF does not match the teacher at every denoising step. It optimizes only the final output.

2.The paper studies a practical setting. Training uses one GPU, a few hours, small batches, and few conditioning signals. The goal is to produce a usable student as quickly as possible. This setting matters for researchers with limited compute.

3.The paper is generally clear. The experiments are also reasonably thorough.

Critical weaknesses

1.The implementation details are inconsistent. Sec. 4.1, Alg. 1, and App. H say that teacher references are generated during training. They also say that teacher outputs are not stored. Tab. 1 and Apps. C and P say something different. They describe a teacher-sample pool that is generated once, saved to disk, and reused by all methods.

2.The baseline comparison has a narrow scope. It mainly measures adaptation speed under a low training budget. It does not measure general distillation performance. The results show that ReDiF learns quickly under the authors' protocol. They do not show that ReDiF is always more efficient or better than existing methods.

3.The paper also claims that PPO is better than direct reward backpropagation. However, the two settings differ in more than the optimizer. Several factors change at the same time. The experiment therefore does not isolate the effect of RL.

Questions for authors

1.Please clarify how teacher references are generated. Are they generated online for every rollout, or precomputed and stored before training? And also report the exact number of teacher forward passes.

2.How was the training budget selected? Was it chosen using ReDiF's held-out FID? If so, could this stopping point favor ReDiF over slower-converging baselines?

3.The PPO and direct-backpropagation settings differ in more than the optimizer. Can the authors provide a comparison with the same sampler, trajectory length, and compute budget?

1 comment

We warmly thank the reviewer for the careful and detailed review, and in particular for finding the inconsistency in how we described the teacher samples. Fixing it has made the paper clearer.

W1 and Q1. How teacher samples are generated, and the number of teacher forward passes. The reviewer is right that the text described two procedures. The method itself does not need stored teacher samples; they can be generated during training. In our experiments we generated them once and cached them, because every baseline had to be trained from scratch with the same teacher, and the same cache is shared by all methods. The teacher sampler is deterministic, so caching does not change the results, and a run with no cache at all gives the same FID within one standard deviation. We would kindly ask the reviewer to see the Data-free training and the cached teacher samples section of our general comment "Data-free and ReDiF vs. direct backprop clarifications", which gives the exact teacher forward-pass counts and the results without a cache. We rewrote Sections 3.3 and 4.1, Algorithm 1 and Appendices C, E and H of the revised paper so that they describe one procedure.

W2 and Q2. How the training budget was chosen. We set the budget from the compute we target: a few GPU-hours on a single A100. Held-out FID was not used for this choice. Within that target, the number of epochs was set slightly past the point where ReDiF's training reward converges, so that it represents the converged regime rather than one particular point: the reward converges around epoch 90 on ImageNet-64 and around epoch 14 on COCO, and we trained to 100 and 20 epochs. The same number of epochs was then used for every method, and every reported number comes from the checkpoint with the highest training reward within that budget.

The reviewer is right that this could favour ReDiF over slower baselines, and we appreciate this point. This is why we trained all methods on ImageNet-64 for twice the budget (200 epochs): ReDiF has a lower FID than every baseline at every evaluated epoch, and no baseline reaches it in this range (Appendix D.1 of the revised paper). Two baselines first get worse (FID above 35 around epochs 80–100) and then improve, so at our budget they are still recovering. We agree that our results show strength in the low-training regime and not general superiority; this is the claim we make (we would kindly ask the reviewer to see the Scope section of our general comment "Novelty and scope of the work").

W3 and Q3. Same sampler, trajectory length and compute budget. We would be very grateful if the reviewer could read the ReDiF vs. direct backpropagation section of our general comment "Data-free and ReDiF vs. direct backprop clarifications", where we answer this question in detail with new experiments. Direct backpropagation and ReDiF share the stochastic sampler (η = 1), the number of steps, the reward, the initialization, the teacher cache, the batch and the epoch budget. On EDM direct backpropagation goes through all 8 steps, and truncating it changes its FID by about 0.1. On SD1.5 direct backpropagation must be truncated to the last step because the full chain does not fit on one GPU, while ReDiF trains on the full chain without truncation; this memory advantage is one of the main strengths of our method.

Reviewer_xqp9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies resource-efficient few-step diffusion distillation. The key challenge is the high training cost of existing distillation methods. The authors propose ReDiF, a reinforcement learning-based framework that optimizes the student using terminal rewards. Experiments on ImageNet-64 and COCO demonstrate improved generation quality under limited computational budgets.

Critical strengths

S1. ReDiF achieves better generation quality than several retrained distillation baselines under the evaluated low-budget setting, demonstrating its practical potential for resource-constrained few-step diffusion distillation. S2. The terminal-reward objective eliminates the need for intermediate trajectory supervision and supports scalar rewards, including non-differentiable ones, within a unified optimization framework.

Critical weaknesses

W1. The paper lacks sufficient comparisons with recent RL-based diffusion distillation methods and resource-efficient few-step distillation methods. This makes it difficult to establish the unique advantages of the proposed terminal-reward RL formulation over existing RL-based approaches and its actual computational efficiency relative to other low-cost distillation methods. Some cited related works are not compared.

W2. The reduced FID evaluation protocols prevent direct numerical comparison with published results, and the choice of evaluation scale is insufficiently justified by the training-resource constraints. Moreover, comparisons primarily at fixed training epochs do not establish the compute required to reach comparable generation quality.

W3. The comparison with direct backpropagation changes rollout stochasticity and uses truncated gradients for the backpropagation baseline, making it difficult to attribute improvements specifically to PPO. In addition, the non-differentiable preference experiments are mainly qualitative, and the text-to-image experiments use only SD1.5, leaving reward flexibility and generalizability to newer architectures insufficiently validated.

Questions for authors

Q1 Could the authors evaluate ReDiF under standard ImageNet-50k and COCO-30k protocols and provide FID-versus-GPU-hours or FID-versus-FLOPs curves, including the training cost required to reach comparable quality levels?

Q2. Sections 3 and Appendix E describe teacher references generated during training, whereas Section 4.2 and Appendix C describe a pre-generated teacher-sample pool. Which procedure underlies the reported results, and how were reference generation and caching accounted for in GPU-hours and FLOPs?

4 comments

We thank the reviewer for the detailed and thoughtful review. The questions on the evaluation protocol and on the teacher samples were especially helpful, and we have revised the paper accordingly.

W1. Comparison with RL-based and low-cost methods. We compare against the main distillation families (progressive and consistency distillation, DMD, DMD2, SiD, SiDA), all retrained under the same budget, and against direct backpropagation of the same reward. DDPO and RLCM are alignment methods and do not distill; RG-LCD and the recent RL + DMD methods (GDMD, RTDMD) use their reward for alignment on top of a distillation loss (we kindly refer the reviewer to the Novelty section of our general comment "Novelty and scope of the work" for a fuller discussion). The experiment closest to that design is the loss mode of Table 2 of the revised paper: DMD2 keeps its loss and we add our reward, which lowers its FID from 25.67 to 21.32. For CTM and TFD, we could not find settings under our single-GPU protocol that gave a usable student, and reporting a poorly tuned number would be unfair to them. If the reviewer has another method in mind, we would be very glad to add it.

W2 and Q1. Evaluation protocol, and FID against compute. Low-budget distillation has not been studied before, so none of the baselines report results in this regime, and we had to train and evaluate every baseline from scratch, three seeds each. Counting all baselines and seeds, the reward ablations, the step-count sweep and the budget curves, the paper has more than 100 evaluated checkpoints on ImageNet alone. Evaluating all of them with the 50K and 30K protocols was not feasible within our single-GPU budget, so we used 10K samples (ImageNet) and 5K prompts (COCO) for all methods. Reduced protocols are also used in other distillation papers, for example InstaFlow (COCO2017-5K) and DMD2 (10K COCO2014 prompts for its SDXL comparison).

To check this choice, we also evaluated with larger protocols: 50K samples on ImageNet-64 and 20K prompts on COCO. ReDiF reaches an FID of 12.36 ± 0.63 on ImageNet-64 (50K), where the EDM teacher reaches 8.48, and about 24 on COCO (20K), where the SD1.5 teacher with our guidance scale of 5.0 reaches about 21 (20K). Published teacher numbers, such as 8.59 in DMD2, use guidance 3 and 30K prompts. For reference, published results with the EDM teacher on ImageNet-64 reach 1.1–1.3 (e.g., 1.28 for DMD2) with hundreds of GPU-hours, while our student uses 8 steps and about 4 GPU-hours. The order of the methods is the same under both protocols. A t-test over seeds also shows that the main gaps we report under the 10K protocol, between ReDiF and the strongest baseline on each dataset, are statistically significant, so the smaller protocol is enough to compare the methods.

For FID against compute, we added curves with GPU-hours on the x-axis (Appendix D.2 of the revised paper). The cost of one epoch is constant for a given method, so each curve is rescaled by that method's own measured GPU-hours per epoch. The order of the methods does not change. Up to twice our budget, no baseline reaches ReDiF's FID, so we cannot measure the compute a baseline needs to reach the same quality within this range, and we do not extrapolate.

Q2. Teacher samples: generated during training or cached? Thank you for catching this; both descriptions were in the paper, and we agree they read as two procedures. In the revised paper, Section 3 and Appendix E describe the method: the teacher sample can be produced during training, and no stored dataset is needed. Section 4.2 and Appendix C describe what we ran: a cache made once and shared by all methods. The teacher is deterministic, so a cached sample is the same image as one made during training, and a run with no cache at all gives the same FID within one standard deviation (we would kindly ask the reviewer to see the Data-free training and the cached teacher samples section of our general comment "Data-free and ReDiF vs. direct backprop clarifications", which gives the full results and the exact teacher forward-pass counts). The cache cost is in the PFLOPs column of every row; GPU-hours measure training only.

W3. Direct backpropagation, preference rewards, and SD1.5. For direct backpropagation, we would be grateful if the reviewer could read the ReDiF vs. direct backpropagation section of our general comment "Data-free and ReDiF vs. direct backprop clarifications", where we address this point in detail. For the preference rewards, we added training curves (Appendix K of the revised paper): over training, the JPEG size drops from 106 to 42 kB and the number of sharp edge pixels grows about 10 times, with no change to the method. For the choice of SD1.5, we kindly refer the reviewer to point 3 of our reply to Reviewer W2gZ, where we explain it in detail.

We thank the reviewer again for this helpful question. The 50K evaluation is now finished. We evaluated the direct-backpropagation and ReDiF students of Table 1 (EDM, ImageNet-64, 8 steps, last checkpoint, which on EDM is also the highest-training-reward one) with 50K generated samples against 50K real ImageNet-64 images:

MethodFID (50K) ↓
EDM teacher (256-step Heun)8.48
Direct backpropagation14.65 ± 0.58
ReDiF (ours)12.36 ± 0.63

ReDiF keeps a lower FID under the full protocol, and a Welch t-test over seeds gives p ≈ 0.005, so the gap is statistically significant and the order of the two methods is the same as under our 10K protocol. On SD1.5, with its higher input resolution (512×512), the gap between the two methods is larger, as Table 1 of the paper shows (26.36 ± 0.78 against 29.56 ± 0.57). We hope these results address the reviewer's concern, and we would be glad to provide any further details.

Reviewer_xqp9OpenReview ↗

After rebuttal

I read the rebuttal from the the authors and I am grateful for their response.

Nevertheless, core issues remain partially unresolved:

  • The PPO vs. direct backprop comparison mixes multiple confounding variables.
  • Preference reward experiments lack quantitative fidelity trade-off analysis.
  • Comparisons with RL-based distillation and low-resource methods are still insufficient.

I maintain my current rating

AuthorsOpenReview ↗

Response to post-rebuttal comments

We thank the reviewer for the follow-up and for the time spent on our paper.

  1. PPO vs. direct backpropagation. We would be grateful if the reviewer could tell us which parameter they believe differs between the two settings, so that we can check it and control for it. To our understanding the comparison is fully controlled. On ImageNet-64 (EDM) the two settings are identical in every component except the gradient estimator: the stochastic sampler (η = 1), the number of steps, the reward, the KL term and its coefficient, the initialization, the teacher cache, the batch size and the epoch budget, with the gradient passing through all 8 steps. On SD1.5 the only additional difference is truncation, which is needed because the full chain does not fit on one GPU; we measured its effect on EDM, where truncating to t = 4, 2 or 1 changes FID by only about 0.1 (Appendix P.1). We therefore believe the comparison is fair.

  2. Preference rewards. Preference rewards are a secondary capability of our formulation: they are introduced briefly in Section 3.5 and detailed in Appendix K, and are not the main focus of the paper. In the revision we added training curves of both the distillation reward and the preference reward (Appendix K of the revised paper), so that the change in each can be followed during training. We agree that a fuller study of this trade-off is possible, and we see it as a direction for future work.

  3. RL-based and low-resource methods. We reviewed the literature carefully and selected the compared methods with care, and the revision adds two recent works (GDMD and RTDMD). We would be very grateful if the reviewer could name the specific methods they would like us to compare with, so that we can add them and further improve the paper.

Reviewer_W2gZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method, namely ReDiF, for diffusion distillation with the application of reinforcement learning in matching the teacher model's outputs and student model's outputs. Matching such intermediate outputs is performed via terminal-reward policy optimisation rather than optimisation of classical reconstruction/consistency loss functions. The proposed method was evaluated on ImageNet. The experimental results show that ReDiF requires only 4-hour use of a single A100 GPU, opposed few hundred hours as reported by DMD2 using the same teacher model, while improving FID.

Critical strengths
  • The paper studies an important and practical research problem
  • The application of reinforcement learning in matching the teacher model's outputs and student model's outputs is simple but reasonable, and the method design sounds technical.
  • Results and analysis are clear and detailed.
Critical weaknesses
  • The novelty is limited as the proposed method is built upon existing techniques, e.g., DDPO, PPO surrogate. The use of a reward method appears the only contribution, which is minor given the existence of RG-LCD, RLCM.

  • The paper argues that ReDiF uses non-differentiable objectives but there lack a comparison with direct back-propagation (i.e., calculate the difference between the student's and teacher's outputs using LPIPS and perform back-propagation through all steps).

  • Experiments are not comprehensive. The method was evaluated with only low-res data and with an outdated diffusion model. More modern models should be experimented.

Questions for authors

Please address issues raised in the weaknesses section.

1 comment

We thank the reviewer for the kind words about the importance of the problem, the clarity of the presentation and the detail of the analysis, and for the helpful suggestions, which led to a new experiment.

1. Novelty. Thank you for raising this; we have made the distinction clearer in the paper. We would kindly ask the reviewer to read the Novelty section of our general comment "Novelty and scope of the work", where we discuss this in detail. In short: DDPO and RLCM are alignment methods that fine-tune an already trained or already distilled model toward an external reward, and do not distill. RG-LCD distills with a consistency loss and adds a backpropagated reward for alignment, and recent RL + DMD methods (GDMD, RTDMD) likewise use external preference rewards. In ReDiF the reward is the distillation objective itself. This removes the need for intermediate targets and for keeping the sampling graph in memory, and it allows non-differentiable rewards.

2. Comparison with direct backpropagation, and with LPIPS. Table 1 of the revised paper compares RL with direct backpropagation on both datasets, with our two main rewards R1 and R2; R2 includes LPIPS. On EDM these runs backpropagate through all 8 steps, and truncation does not change the result (we would be grateful if the reviewer could look at the ReDiF vs. direct backpropagation section of our general comment "Data-free and ReDiF vs. direct backprop clarifications", which includes the new truncation results).

Following the reviewer's suggestion, we also ran exactly the experiment described in the review: the LPIPS distance between the student's and the teacher's outputs as the only reward, backpropagated through all 8 sampling steps (EDM, ImageNet-64, 3 seeds; Appendix P.1 of the revised paper):

OptimizerRewardFID ↓
Direct backprop (all 8 steps)LPIPS27.10 ± 1.14
ReDiF (ours)LPIPS21.11 ± 1.55

With the same reward, ReDiF has an FID about 6 points lower. Our reward ablations (Tables 5 and 6 of the revised paper) also show that LPIPS alone is a weaker reward than the full combination R2 (18.60 FID). We are grateful for this suggestion, as it makes the comparison much more direct.

3. Choice of models. We understand this concern. Our setting is a single A100 with 80 GB, and a fair comparison requires retraining six baselines, three seeds each, plus generating the teacher samples. For SDXL this whole protocol does not fit our budget, so we used SD1.5 at 512×512, which is also the text-to-image setting of the baselines we compare against. EDM on ImageNet-64 adds a second architecture (pixel-space, class-conditional) and a second resolution. We fully agree that newer models are important, and we state this in the Limitations section of the revised paper as a direction for future work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Novelty and scope of the work

Novelty

Our main contribution is to formulate diffusion step distillation as RL policy optimization, where a reward on the final sample is the distillation objective itself.

DDPO and RLCM also use RL with diffusion models, but they are alignment methods, not distillation methods: they fine-tune a model that is already trained (DDPO) or already distilled (RLCM) toward an external reward such as aesthetics or compressibility. RG-LCD is a distillation method, but the distillation is done by its consistency loss; it adds a reward, optimized by direct backpropagation, to improve alignment with human preference. Recent works such as GDMD (arXiv 2604.19009) and RTDMD (arXiv 2605.26108) add policy-gradient training on top of distribution matching, again with external preference rewards, while teacher matching stays with the DMD term. In all of these, the reward is used for alignment. In ReDiF the reward is the distillation objective: it measures how close the student's final sample is to the teacher's. We now discuss these works in the Related Work section of the revised paper.

This formulation gives four things:

  1. The student needs no intermediate targets, so the teacher's and student's step grids do not have to match, and the student can find its own sampling path.
  2. We never backpropagate through the sampling chain, so its graph is never kept in memory. This makes single-GPU distillation of SD1.5 possible.
  3. The reward does not need to be differentiable, so non-differentiable preferences can be added without changing the method.
  4. It can be added to existing objectives: added to the DMD2 loss, it improves DMD2's FID by 4.4.

The method converges fast, which is the basis of our resource-efficiency claim: about 4 GPU-hours on one A100, against about 336 A100-hours reported for DMD2 on the same EDM teacher.

Scope: the low-training regime

We do not claim better results at convergence; our claim is that ReDiF works well in the low-training regime. We set the training budget from the compute we target: a few GPU-hours on a single A100. Within that target, the number of epochs was set slightly past the point where ReDiF's training reward converges, so that it represents the converged regime rather than one particular point: the reward converges around epoch 90 on ImageNet-64 and around epoch 14 on COCO, and we trained to 100 and 20 epochs (about 4 GPU-hours and under one GPU-hour). The same number of epochs was used for every method, and all reported numbers come from the checkpoint with the highest training reward within that budget.

Because this choice came from our own method's curve, we checked that it does not favour ReDiF: on ImageNet-64 we trained all methods for twice the budget (200 epochs). ReDiF has a lower FID than every baseline at every evaluated epoch, and no baseline reaches it (Appendix D.1 of the revised paper). The same curves with GPU-hours on the x-axis are in Appendix D.2 of the revised paper.

On SD1.5 we did not run this longer sweep. At the matched budget the baselines stay close to the 5-step teacher in FID (31.60–32.04, against 31.45), while ReDiF reaches 26.36. We added a side-by-side comparison of samples at this budget (same prompt and noise for every method, no selection) in Appendix D.4 of the revised paper.

AuthorsOpenReview ↗

Data-free and ReDiF vs. direct backprop clarifications

We sincerely thank all three reviewers for their time and for their careful, constructive reviews. Their comments helped us find several unclear parts of the paper, and we believe the revised version is clearer and stronger because of them. We have uploaded a revised PDF in which all changed text is in blue. This comment answers two points raised by more than one reviewer.

Data-free training and the cached teacher samples

Our method is data-free: it needs no real images and no stored dataset of teacher samples, since the teacher sample for a (noise, class) pair can be produced during training. In our experiments we generated the teacher samples for our 1,000 (noise, class) pairs once, before training, and cached them. We did this to save time: we trained every method and every baseline from scratch with several seeds, and generating the teacher samples once and reusing them across all of these runs saved a large amount of compute. The same cache is used by every method in Table 1 of the revised paper, so it does not favour ReDiF.

Caching does not change the results. The EDM teacher sampler is deterministic (Heun, no added noise), so a cached sample and a sample generated during training are the same image. To confirm this in practice, we also trained ReDiF with no cache at all: the teacher runs inside every training batch, on new noise each time (EDM, ImageNet-64, R2, mean ± sd over 3 seeds):

ReDiF, EDM, R2FID ↓PrecisionRecallDensityCoverage
Cached teacher (Table 1 of the revised paper)18.60 ± 1.630.743 ± 0.0080.622 ± 0.0160.799 ± 0.0100.802 ± 0.013
Online teacher, no cache19.45 ± 1.450.726 ± 0.0050.566 ± 0.0890.607 ± 0.0630.787 ± 0.022

The two agree in FID within one standard deviation, even though the online version also sees new noise in every batch instead of a fixed pool.

Teacher forward passes on EDM: 1,000 pairs × 511 = 511,000 for the cache, plus 25,600 for the KL term (3,200 rollouts × 8 steps). The teacher mean of the KL term is computed once per step at rollout time and reused in all 4 inner epochs; we corrected the formula in Appendix C of the revised paper to match. On COCO: 160 prompts × 50 steps × 2 (classifier-free guidance) = 16,000. While checking these counts, we found that the PFLOPs column had counted the cache for 10,000 pairs instead of the 1,000 we use. Table 1 of the revised paper uses the correct 1,000-pair cache (112.2 PFLOPs instead of 1122.0). This lowers every EDM row by the same amount and changes no comparison. We also rewrote Sections 3.3 and 4.1 and Appendices C, E and H of the revised paper so that they describe one procedure.

ReDiF vs. direct backpropagation

Direct backpropagation (DB) and ReDiF use the same reward, initialization, sampler, teacher cache, batch and epoch budget, and the same stochastic sampler (η = 1); for DB the noise is reparameterized so the gradient can pass through it. The only remaining difference is how far back the gradient can go:

  • EDM (ImageNet-64, 8 steps): the full chain fits on one 80 GB A100 with gradient checkpointing, so the DB rows of Table 1 of the revised paper use all 8 steps (t = 8).
  • SD1.5 (COCO, 5 steps): the full chain does not fit on one 80 GB A100, so DB can only run with the gradient truncated to the last step (t = 1).

This is one of the main advantages of our method. Policy optimization never keeps the graph of the sampling chain, so ReDiF needs the GPU memory of only one step at a time, far less than DB. This is why ReDiF trains on the full 5-step SD1.5 chain on a single GPU without any truncation.

Table 1 of the revised paper now shows t for every DB row. To measure the effect of truncation, we reran DB on EDM with t = 8, 4, 2 and 1 (reward R2, 3 seeds, new Appendix P.1 of the revised paper):

EDM, ImageNet-64, R2tFID ↓
Direct backprop8 (full)20.05 ± 0.30
Direct backprop419.97 ± 0.52
Direct backprop219.94 ± 0.38
Direct backprop120.05 ± 0.37
ReDiF—18.60 ± 1.63

Truncation changes FID by about 0.1, so it does not explain the difference between the two optimizers. On EDM that difference is modest: ReDiF's mean FID is about 1.4 lower, which is smaller than the variation across seeds (standard deviation 1.63). The difference is clear at higher resolution: on SD1.5 with 512×512 images, ReDiF reaches 26.36 ± 0.78 against 29.56 ± 0.57. It is also clear on EDM with an LPIPS-only reward, where ReDiF reaches 21.11 ± 1.55 against 27.10 ± 1.14 (Appendix P.1 of the revised paper). And only ReDiF can optimize the full SD1.5 chain on one GPU.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.