acceptodds
Under review as a conference paper at ICLR 2027

LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL

Abstract

Diffusion reinforcement learning (RL) has recently achieved significant success in post-training image and video generative models. However, most diffusion RL methods, including DanceGRPO and FlowGRPO, recompute selected timesteps with gradient tracking after rollout. Under on-policy training with the same backend for rollout and update, this recomputation is mathematically redundant. Intuitively, the rollout and policy update steps can reuse the same feed-forward backbone to avoid redundant computation, but doing so can incur a large memory overhead during rollout. To address the issue, we present LeanGRPO by restructuring the data-parallel layout and introducing two recompute-free training schedules for trajectory-logprob diffusion RL: (1) LeanGRPO-Retain enables gradient tracking during rollout and directly reuses the resulting computation graphs and saved activations for backward during update, requiring no recomputation; and (2) LeanGRPO-Reweight also enables gradients during rollout, but immediately backpropagates each selected step using a provisional advantage and delays gradient synchronization, then corrects the provisional gradients with the true advantage after the trajectory is completed. These schedules target different model scales and input sizes. Across FlowGRPO/DanceGRPO with FLUX.1-dev and Wan, LeanGRPO achieves up to 1.83x end-to-end speedup while preserving the original optimization objective.

Primary area:
reinforcement learning
Keywords:
Diffusion RL, Training accelerating, High performance computing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bT4qOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies update-stage forward recomputation in trajectory-based diffusion reinforcement learning. Under an unchanged policy snapshot and a common rollout/update execution path, it proposes two schedules that reuse rollout computation: Retain stores selected transition computation graphs until advantages become available, whereas Reweight immediately accumulates unweighted per-trajectory score-function gradients and applies the trajectory advantage afterward. A shared-prompt data-parallel layout distributes the memory burden, and Reweight delays gradient reduction until after advantage weighting. The paper derives equivalence to the native on-policy policy-loss gradient in exact arithmetic and evaluates latency, memory, numerical alignment, and reward convergence across several diffusion RL methods and backbones. It reports up to 1.83× end-to-end speedup. The main contribution is a systems execution strategy rather than a new policy-gradient estimator.

Critical strengths
  1. A practically relevant optimization with complementary execution choices. Retain and Reweight target different limiting resources: saved activations across selected timesteps versus full per-trajectory gradients. The paper evaluates multiple algorithms, model sizes, fine-tuning modes, precisions, and GPU counts, and reports both successful and out-of-memory configurations. This makes the proposal more informative than a single-setting speedup and provides evidence that avoiding update-stage transition re-forwards can be practically useful.

  2. Substantive numerical diagnostics beyond reward curves. Appendix F compares gradients, parameter-update vectors, and AdamW states under controlled transition data and real reward execution. It demonstrates exact agreement for Retain in the controlled setting and explicitly quantifies Reweight’s finite-precision differences rather than claiming bitwise identity. These diagnostics support the narrow mathematical argument and are a meaningful strength of the submission.

Critical weaknesses
  1. Unclear whether LeanGRPO preserves the full training procedure. Eqs. (5)–(9) establish gradient equivalence at a fixed policy, but this does not necessarily guarantee equivalence over the entire training process. In particular, when multiple optimizer updates are performed on the same rollout, retained graphs are computed under the rollout policy and cannot be reused once an optimizer step changes the parameters, as Appendix H acknowledges. However, it is unclear whether the experiments preserve the same optimizer-step schedules as the original baselines. Additionally, the paper does not discuss KL regularization, and describes only within-group advantage normalization, whereas some canonical Flow-GRPO configurations use a batch-wide standard deviation. It is unclear how either would be handled under LeanGRPO’s prompt-by-prompt schedule, particularly for Reweight. I think these implementation details need further clarification, since differences in training procedures could affect both the reported speedup and convergence comparisons.

  2. Unclear memory costs. I have concerns about the memory analysis. Table 1 and Eqs. (7) and (11) appear inconsistent in their memory scaling, and the paper does not sufficiently explain how Reweight obtains per-sample gradients when the local batch size is larger than one. I think these details are important since memory efficiency is a central motivation of the proposed methods.

Questions for authors
  1. Exact training recipes. For each timing and convergence configuration, including those based on Flow-GRPO and DanceGRPO, please specify the number of optimizer updates per rollout, gradient-accumulation boundaries, auxiliary losses and KL coefficient, and advantage-normalization rules. Which implementation versions or commits are used, and which settings, if any, differ from the corresponding canonical implementations? Are any such changes applied identically to Native and LeanGRPO? If nonzero KL terms or batch-wide reward normalization are used, how are they handled under LeanGRPO’s prompt-by-prompt execution schedule while preserving the original update?

  2. Autograd and memory semantics. Please give the exact definition of the gradient-tracked denoising step (Denoiseθgrad\mathrm{Denoise}^{\mathrm {grad}}_ {\theta} in Algorithm 1): which tensors carry gradients, and whether the input ztz_t and the sampled zt−1z_{t-1} are detached when computing ℓi,tgrad-roll\ell^{\text{grad-roll}}_{i,t}. How are separate per-trajectory gradients maintained when multiple samples are processed on the same rank (b>1b>1)? Under what assumptions do Eqs. (7) and (11) omit ⌈M/R⌉\lceil M/R\rceil? Please reconcile these expressions with Table 1 and account for the simultaneously retained buffers when M>RM>R and b>1b>1, including gradient shards accumulated from previous prompt groups.

  3. Measurement protocol. Why do the LoRA Retain/Reweight times in Table 2 (901.99/880.68 s) differ from Section 4.3/Fig. 4 (960.69/956.36 s), despite referring to the same nominal configuration? How are Fig. 8’s projected times computed, which overheads are included, and how are time-to-target thresholds selected across seeds? Are latency and peak-memory measurements taken after warmup and optimizer-state initialization, and what parameter, gradient, optimizer-state precision and offloading settings are used?

  4. Minor Issue: Notation Clarity. The main text introduces θupd\theta_ {\mathrm{upd}} only as the “update policy” and then sets θupd=θroll\theta_{\mathrm{upd}}=\theta_ {\mathrm{roll}} by assumption. Defining θupd\theta_ {\mathrm{upd}} explicitly as the parameters at which the update-stage term is evaluated would clarify its relationship to optimizer-step boundaries, which currently appears only in Appendix H. Similarly, ℓi,tgrad−roll\ell_{i,t}^ {\mathrm{grad-roll}} in Section 3.1 should be formally defined and distinguished from ℓi,troll\ell_{i,t}^{\mathrm{roll}} and ℓi,tupd\ell_{i,t}^{\mathrm{upd}}, including their gradient-tracking and stop-gradient semantics. Moving or adapting the consolidated notation table from page 16 of the Appendix into the main text would improve readability and make the assumptions behind the gradient-equivalence derivations easier to follow.

9 comments

First of all, we thank you for your time and detailed review. Your questions are insightful and meaningful, and we believe they will help us improve the completeness and clarity of our work. We have carefully considered every question and welcome further discussion if you have additional questions after reading our responses.

Q1-1: For each timing and convergence configuration, including those based on Flow-GRPO and DanceGRPO, please specify the number of optimizer updates per rollout, gradient-accumulation boundaries, auxiliary losses and KL coefficient, and advantage-normalization rules.

A1-1. LeanGRPO targets redundant recomputation in the on-policy setting. In our experiments, each rollout batch is used for exactly one optimizer update. For example, the FLUX.1-dev convergence configuration in Fig. 6(b) accumulates gradients over 24 prompt groups, with eight samples per prompt and five selected timesteps per sample, before performing one optimizer step. Gradients from each timestep are accumulated until optimizer.step() is invoked, and policy parameters remain unchanged throughout this interval.

Some training configurations perform multiple optimizer updates on the same rollout batch. After the first update, the current policy differs from the rollout policy, so subsequent updates fall outside the on-policy setting considered in this work. Further discussion is provided in Appendices H and L.

Auxiliary losses, KL coefficients, and advantage-normalization rules are discussed in A1-3.

Q1-2: Which implementation versions or commits are used, and which settings, if any, differ from the corresponding canonical implementations?

A1-2. LeanGRPO is developed from the official open-source implementations of FlowGRPO, DanceGRPO, and MixGRPO. Our modifications aim to accelerate training while preserving the native optimization objective, without changing the core algorithms. The upstream implementation versions and base commits are listed below; the detailed settings for each experimental configuration are documented in the newly added Appendix L.

Table 1. Official upstream repositories and base commits used to develop our integrations.

Official upstream repositoryBase commit
DanceGRPO15cc71d53cc2e6e18a68ee607d5fb6ba9a99e344
FlowGRPO / FlowGRPO-Fast9745241cf1101407e9b518dd4f10498092150522
MixGRPO / MixGRPO-Flashe7f299d40e3309b18880ec7717e7024c3aa714bb

Q1-3: If nonzero KL terms or batch-wide reward normalization are used, how are they handled under LeanGRPO's prompt-by-prompt execution schedule while preserving the original update?

A1-3-1: KL Regularization.

We thank the reviewer for pointing out this omission. One reason is that some algorithms, such as DanceGRPO and BranchGRPO, do not include explicit KL regularization in the implementations we evaluated. Another reason is that even algorithms that support KL regularization, such as Flow-GRPO, disable it in some released training configurations. For DanceGRPO, our experiments therefore omit the KL term, consistent with its native implementation. For FlowGRPO and MixGRPO, we set the corresponding KL coefficients to zero consistently across Native, Retain, and Reweight to ensure a fair comparison.

However, both LeanGRPO schedules can accommodate an additive fixed-reference KL regularizer. Using exactly the same KL definition and loss normalization as Native, the total gradient is

Gtotal=∑iAiGitmp⏟policy gradient+βGKL⏟KL gradient,GKL=∇θLKL.G_{\mathrm{total}} = \underbrace{\sum_i A_iG_i^{\mathrm{tmp}}}_{\text{policy gradient}} + \underbrace{\beta G_{\mathrm{KL}}}_{\text{KL gradient}}, \qquad G_{\mathrm{KL}}=\nabla_\theta\mathcal L_{\mathrm{KL}}.

The reference predictions are computed without gradient tracking, while gradients propagate through the current-policy predictions.

For Retain, enabling KL only requires an additional reference-model forward pass at each selected timestep. The policy loss and KL regularizer can be combined and backpropagated together once the advantages are available.

For Reweight, the policy and KL gradient contributions must remain separate until advantage correction is applied to the policy gradients. The KL contribution is not multiplied by the advantage, so it can be reduced, sharded, and accumulated independently. The corrected policy gradients are subsequently reduced and added to the accumulated KL gradients. Consequently, this implementation need not retain a second full, unsharded KL-gradient buffer throughout rollout. However, Reweight requires an additional backward pass to compute and accumulate GKLG_{\mathrm{KL}} separately. The reference model's forward pass also reduces the fraction of end-to-end training time attributable to recomputation, thereby limiting LeanGRPO's speedup.

Table 2. SD3.5-Medium training latency with reference KL regularization (β=0.04\beta=0.04). Times are mean ±\pm standard deviation in seconds over three updates after warm-up.

fself_{\mathrm{sel}}T′T^{\prime}Native time (s)Retain time (s)Retain speedupReweight time (s)Reweight speedup
0.44161.18±0.19161.18\pm0.19146.80±0.39146.80\pm0.391.098×1.098\times165.22±0.45165.22\pm0.450.976×0.976\times
0.66213.88±0.13213.88\pm0.13189.26±0.76189.26\pm0.761.130×1.130\times215.99±0.19215.99\pm0.190.990×0.990\times
0.88268.28±0.15268.28\pm0.15241.09±0.65241.09\pm0.651.113×1.113\times267.42±0.74267.42\pm0.741.003×1.003\times
1.010321.75±0.51321.75\pm0.51278.81±0.45278.81\pm0.451.154×1.154\times319.38±0.12319.38\pm0.121.007×1.007\times

Table 2 shows a comparison using the same configuration as Fig. 3(i), but with KL regularization enabled. The results show that Retain achieves a smaller speedup than in the KL-disabled setting because the additional reference-model forward pass reduces the fraction of end-to-end training time attributable to recomputation. Reweight does not degrade the results but achieves almost no speedup, primarily because computing GKLG_{\mathrm{KL}} separately requires an additional backward pass, offsetting the gains from eliminating recomputation. Nevertheless, Reweight remains valuable in most KL-disabled settings and achieves runtimes close to Native in the few KL-enabled settings, while Retain achieves favorable speedups both with and without KL regularization.

A1-3-2: Batch-Wide Reward Normalization.

LeanGRPO supports three normalization rules, but the main text focuses on the first. We have therefore added Appendix M to explain how LeanGRPO supports the other two normalization rules. Let GitmpG_i^{\mathrm{tmp}} denote the unweighted policy gradient of trajectory ii, summed over its selected timesteps. Here, "group-wise" refers to computations within a single prompt group.

1. Group-wise mean and standard deviation.

This is the setting primarily described in the main text, where advantages become available after each prompt group is completed. The computation therefore follows the procedure described in the main text.

2. Group-wise mean and batch-wide standard deviation.

This rule was already used in our FLUX convergence experiments but was not explicitly described in the main text. Specifically,

Ai=aidbatch,ai=ri−rˉgroup(i),dbatch=sbatch+ϵ.A_i=\frac{a_i}{d_{\mathrm{batch}}}, \qquad a_i=r_i-\bar r_{\mathrm{group}(i)}, \qquad d_{\mathrm{batch}}=s_{\mathrm{batch}}+\epsilon.

The policy gradient therefore satisfies

Gpolicy=1dbatch∑iaiGitmp.G_{\mathrm{policy}} =\frac{1}{d_{\mathrm{batch}}}\sum_i a_iG_i^{\mathrm{tmp}}.

Once a prompt group's rewards are available, its aia_i values are known. Reweight applies these values to the temporary gradients before aggregation and reduce-scatter. Retain instead backpropagates the aia_i-weighted losses through the retained graphs. Both schedules accumulate gradient shards without the batch-wide denominator. Once all batch rewards are available, they divide the accumulated policy-gradient shards by dbatchd_{\mathrm{batch}}, recovering the same gradient without an additional backward pass or retaining the entire batch's graphs.

Therefore, this normalization introduces no additional memory or computational overhead.

3. Batch-wide mean and standard deviation.

A_i=\frac{r_i-\bar r_{\mathrm{batch}}}{d_{\mathrm{batch}}}. \tag{1}

Neither batch statistic is generally known when an individual prompt group finishes. However, the policy gradient admits the decomposition

\begin{aligned} P^{\mathrm{norm}}&=\sum_{i\in\mathcal{B}}r_iG_i^{\mathrm{tmp}}, &Q^{\mathrm{norm}}&=\sum_{i\in\mathcal{B}}G_i^{\mathrm{tmp}},\\\\ G_{\mathrm{policy}} &=\frac{P^{\mathrm{norm}}-\bar r_{\mathrm{batch}}Q^{\mathrm{norm}}} {d_{\mathrm{batch}}}. \end{aligned} \tag{2}

Reweight. During rollout, Reweight backpropagates each selected timestep's unweighted policy loss before reward computation. With one sample per rank, these contributions form the local GitmpG_i^{\mathrm{tmp}} without reduce-scatter. Once the prompt-group reward rir_i is available, Reweight shallow-copies the list of gradient-tensor references, not the tensor data, so that the same GitmpG_i^{\mathrm{tmp}} can be reused. Two sequential FSDP2 reduce-scatter passes then form separate shards: the unweighted pass accumulates into QnormQ^{\mathrm{norm}}, and the reward-weighted pass accumulates into PnormP^{\mathrm{norm}}. Once all rewards are available, we weight QnormQ^{\mathrm{norm}} by rˉbatch\bar r_{\mathrm{batch}}, subtract it from PnormP^{\mathrm{norm}}, and divide the result by dbatchd_{\mathrm{batch}}. Only the two sharded accumulators persist across prompt groups; forming them requires neither another backward pass nor two full-model gradient copies.

Retain. After a prompt group's rewards become available, Retain can construct an unweighted loss for each trajectory and backpropagate it through the retained graphs to obtain GitmpG_i^{\mathrm{tmp}}. It then separately accumulates riGitmpr_iG_i^{\mathrm{tmp}} and GitmpG_i^{\mathrm{tmp}} into the PnormP^{\mathrm{norm}} and QnormQ^{\mathrm{norm}} shards, respectively. Once rollout for the entire batch is complete and rˉbatch\bar r_{\mathrm{batch}} is available, the QnormQ^{\mathrm{norm}} shard is scaled by −rˉbatch-\bar r_{\mathrm{batch}} and added to the PnormP^{\mathrm{norm}} shard. The result is then divided by dbatchd_{\mathrm{batch}}.

However, our survey found very few open-source training scripts that use batch-wide mean and standard deviation. We therefore think that this configuration may not be suitable for practical training.

Reviewer_EfejOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In DanceGRPO/FlowGRPO-style training, the rollout runs without gradients and then the selected timesteps are run again with gradients for the update. The paper points out that this second forward is redundant under on-policy training, and removes it in two ways: Retain keeps the rollout graphs and does backward later, and Reweight does backward right away and scales the gradient by the advantage once the reward is known. It reports up to 1.83x speedup on FLUX, SD3.5-M and Wan.

Critical strengths
  1. The observation is correct and practical. And the method to alleviate this problem is reasonable.
  2. Since the method doesn't change the objective, it doesn't need to prove it trains better, only that it gives the same gradient faster, and the experiments demonstrate this.
Critical weaknesses
  1. The method only applies in a narrow regime. It requires strictly on-policy training with one update per rollout, the same backend for rollout and training, and (for Reweight) the same advantage for every timestep of a trajectory.
  2. The speedup is smaller than claimed. In Table 2, Native with only delayed gradient sync (a standard FSDP trick) already gets 1.33x, so against that baseline reweight is about 1.36x rather than 1.81x, and retain is slower.
Questions for authors

N/A

1 comment

We thank the reviewer for recognizing the practical value of our observation, the rationale behind our methods, and the experimental evidence that LeanGRPO reduces training time while preserving the original update. We also appreciate your comments on its applicability and baseline selection.

Regarding applicability, LeanGRPO primarily targets on-policy training. We believe this regime can benefit training quality and represents a practically valuable setting for our methods.

Regarding execution backends, many trajectory-based diffusion RL implementations we examined use a single FSDP-based backend for both rollout and update. Other frameworks support separate backends, such as vLLM-Omni for rollout and FSDP for training in veRL-Omni. We believe both designs have advantages and tradeoffs.

A single-backend design allows rollout and update to share the same model weights. Although an unoptimized FSDP implementation may have lower rollout throughput, our tests show that disabling resharding during rollout and applying optimizations comparable to those in vLLM-Omni, including QKV fusion, optimized RoPE and RMSNorm kernels, and FlashAttention, can achieve comparable performance in the tested configuration, as shown in Table 2.

Specifically, we benchmarked FLUX.1-dev in BF16 under rollout-only execution, using eight A6000 GPUs to generate 128 images at 720×720720\times720 resolution with 16 sampling steps. After warm-up, we measured three complete rollout rounds within a single run.

Table 2. Rollout-only performance comparison on FLUX.1-dev in BF16 using eight A6000 GPUs. Each round generates 128 images at 720×720720\times720 resolution with 16 sampling steps. Both backends compile 57 blocks. Times are means ±\pm standard deviations over three complete rollout rounds after warm-up within a single run.

Rollout BackendCompiled BlocksTime, Mean ±\pm Std. (s)
FSDP2 + operator optimizations57107.69±0.70107.69\pm0.70
vLLM-Omni57103.01±1.33103.01\pm1.33

A dual-backend design provides out-of-the-box inference acceleration through vLLM-Omni. However, maintaining separate rollout and training model-weight copies may require offloading or other memory-management measures to fit within the same GPU memory budget. We therefore consider FSDP-based single-backend execution an important option for diffusion RL.

We also agree that delayed synchronization is an established optimization. In the BF16 full fine-tuning configuration, Reweight achieves an additional 1.36×1.36\times speedup over Native + Shared + Delayed, whereas the reported 1.81×1.81\times is relative to the original Native baseline. Retain does not outperform the optimized baseline in this configuration. We will clarify this distinction in future revisions of the manuscript.

Finally, we thank the reviewer again for recognizing our work. We will carefully incorporate your feedback to clarify its scope and improve the presentation.

Reviewer_4dbTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LeanGRPO, a system-level optimization framework designed to eliminate redundant forward recomputation in diffusion reinforcement learning (RL). The authors observe that existing methods, such as DanceGRPO and FlowGRPO, typically perform rollouts without gradient tracking and subsequently recompute selected timesteps during policy updates to construct computation graphs for backpropagation. Under an on-policy setting where the rollout and update use the same execution backend and the model parameters remain unchanged, such recomputation is mathematically redundant. To address this issue, the authors propose two complementary training schedules. LeanGRPO-Retain retains the computation graphs and activations of selected timesteps during rollout and directly reuses them for backpropagation once the advantages become available. It also reorganizes the data-parallel layout to reduce per-GPU memory overhead. LeanGRPO-Reweight, on the other hand, performs backward passes immediately at selected timesteps during rollout, accumulates temporary gradients without advantage weighting, and subsequently corrects these gradients using the corresponding advantages after trajectory completion. Additionally, the authors delay gradient synchronization to prevent gradients from different samples from being mixed before advantage weighting.

Critical strengths
  1. This paper addresses an important efficiency bottleneck in diffusion RL training. Existing methods perform redundant DiT forward computations across the rollout and policy update stages, introducing substantial computational overhead. The authors identify that such recomputation can be eliminated under specific on-policy conditions and show that it can account for up to approximately 32% of the end-to-end training time.

  2. The proposed Retain and Reweight schedules provide reasonable solutions to different GPU memory bottlenecks. Retain avoids redundant forward computation by reusing computation graphs retained during rollout, while Reweight exploits the fact that all selected timesteps within a trajectory share the same advantage, allowing backward passes to be performed during rollout and gradients to be corrected afterward.

  3. The paper evaluates LeanGRPO across multiple diffusion RL algorithms, model scales, numerical precisions, and input resolutions. The experiments include end-to-end training latency, runtime breakdowns, GPU memory usage, ablation studies, and training convergence analysis.

Critical weaknesses

1.The core idea of LeanGRPO is to retain computation graphs during rollout or compute and store temporary gradients in advance, thereby avoiding redundant forward computation during policy updates. Previous studies have explored related ideas from the perspectives of gradient caching, computation graph management, and training scheduling. For example, Gradient Cache methods reduce training memory overhead by decoupling gradient computation and caching intermediate gradients, while DigenRL investigates data parallelism and execution scheduling across generation and training stages. The authors need to give further discussion clarifying the specific novelty of LeanGRPO in computation graph management, gradient reuse, and distributed training scheduling. 2.While LeanGRPO reduces training time by eliminating redundant forward computation, it also introduces additional GPU memory overhead. Specifically, Retain needs to preserve computation graphs and activations for selected timesteps, whereas Reweight needs to store full temporary gradients before advantage weighting. Consequently, under limited GPU memory budgets, these methods may require smaller micro-batch sizes than the native implementation. When maintaining the same effective batch size, reducing the micro-batch size requires processing more micro-batches, which may decrease GPU utilization and introduce additional execution overhead. As a result, some of the speedup gained from eliminating redundant forward computation could potentially be offset. 3.LeanGRPO-Reweight introduces two major optimizations: eliminating redundant forward recomputation during policy updates and delaying gradient synchronization until after advantage weighting.The authors have already conducted relevant ablation studies in Table 2, showing that the contributions of these two optimizations vary across training configurations. For example, under BF16 full fine-tuning, delayed gradient synchronization alone provides substantial speedups, whereas its benefit is relatively limited in the LoRA setting.This observation suggests that the overall acceleration of Reweight does not come solely from eliminating redundant forward computation, but also depends on the gradient communication overhead of specific training configurations.

Questions for authors
  1. The authors need to give further discussion clarifying the specific novelty of LeanGRPO in computation graph management, gradient reuse, and distributed training scheduling. 2.It remains unclear whether the reported speedups of LeanGRPO are reliable and practically meaningful when the effective batch size is kept the same across different methods. The additional memory overhead introduced by Retain and Reweight may require smaller micro-batch sizes, potentially increasing the number of gradient accumulation steps and offsetting some of the computational savings. Although the paper provides some relevant experiments, it does not sufficiently demonstrate whether the reported speedups consistently hold under the same effective batch size and GPU memory constraints. The authors should clarify this issue and provide a more systematic comparison of end-to-end training efficiency under these conditions.
3 comments

Q1: Clarification of novelty.

A1. We thank the reviewer for recognizing the strengths of our work and for raising this important question about novelty. We acknowledge that retaining computation graphs, caching gradients, and processing data in smaller batches are established techniques. LeanGRPO's contribution is not to introduce these techniques themselves, but to design execution schedules that eliminate redundant rollout-update computation in diffusion RL under a fixed policy, while preserving the original update and addressing the associated memory and distributed gradient-management challenges.

First, LeanGRPO goes beyond enabling gradient tracking during rollout. Directly retaining rollout graphs causes activations to accumulate across samples and training timesteps, while advantages become available only after generation and reward computation. LeanGRPO-Retain combines a shared-prompt data-parallel layout with gradient tracking only at selected timesteps to reduce the states retained on each rank. We do not claim the DP layout itself as a standalone contribution. Instead, we emphasize its role in controlling graph retention and enabling training without repeated transition forward passes.

Second, Reweight addresses the activation accumulation that can still limit Retain for large inputs and long trajectories. Since all selected timesteps of a trajectory share the same advantage, Reweight immediately performs backward at each selected timestep, before the advantage is available. It accumulates unweighted per-trajectory parameter gradients and releases the corresponding activations. Once advantages become available, it weights and aggregates these temporary gradients. In distributed execution, gradients from different samples must remain separate until advantage weighting, after which they can be synchronized. Reweight therefore replaces activation storage across timesteps with temporary parameter-gradient storage, offering a different computation-memory tradeoff from Retain.

Compared with GradCache, the cached quantities, execution order, and computational costs differ. GradCache caches representation gradients and reruns encoder forward/backward passes to support large-batch contrastive learning under limited memory. Reweight instead uses the rollout forward pass to compute parameter gradients before the final advantage is available, then applies advantage weighting later. Its goal is to avoid repeating the same transition forward during the update. Thus, LeanGRPO is not a direct application of GradCache's sub-batch processing to diffusion RL.

DigenRL also targets a different system bottleneck. It improves throughput through generation-training pipelining, parallelism, and resource scheduling, while retaining trainer-side recomputation of transition log-probabilities. LeanGRPO instead eliminates this repeated computation across rollout and update in the applicable fixed-policy settings.

In summary, LeanGRPO's novelty lies in two execution schedules designed around delayed advantages in diffusion RL, their memory-management and gradient-correction mechanisms, and a distributed implementation that preserves the original update, rather than simply enabling gradients or splitting batches.

Q2: Concerns about micro-batch sizes.

A2. Thank you for highlighting this important computation-memory tradeoff. We agree that additional state storage can limit the micro-batch size and offset some of the computational savings. A fair comparison should therefore fix the effective batch size and GPU memory budget while allowing each method to use a different feasible micro-batch size.

For the video configuration in Fig. 5(b), increasing Native's micro-batch size from one to two provides only a 1.03×1.03\times speedup, and Native remains slower than Reweight with a micro-batch size of one. Thus, in this configuration, a larger micro-batch does not compensate for the cost of repeated forward computation.

For image generation, we supplement this comparison with FLUX BF16 full-parameter fine-tuning at an input size of 720×720×1720\times720\times1, with six sampling steps and two selected training timesteps (fsel=0.4f_{\mathrm{sel}}=0.4). Table 1 compares complete-update times at the same effective batch size on the same hardware, with gradient accumulation adjusted for the micro-batch size. Increasing Native's micro-batch size from eight to sixteen reduces its time from 135.51 s to 127.62 s. Retain takes only 113.36 s at a micro-batch size of eight, achieving a 1.13×1.13\times speedup over Native at sixteen. Both configurations fit within the same GPU memory budget, with similar peak reserved memory of 46.94 and 46.82 GiB, respectively.

Table 1. Supplementary micro-batch comparison for FLUX BF16 full-parameter fine-tuning at 720×720×1720\times720\times1, with T=6T=6, fsel=0.4f_{\mathrm{sel}}=0.4, and T′=2T^{\prime}=2. Micro-batch sizes are per rank. Times are for a complete optimizer update at a fixed effective batch size. Speedup is the corresponding Native baseline time divided by the method's time.

MethodMicro-batch per rankTime (s)Peak allocated (GiB)Peak reserved (GiB)Speedup vs. Native b=8b=8Speedup vs. Native b=16b=16
Native8135.5128.3935.381.00×1.00\times0.94×0.94\times
Native16127.6238.9746.821.06×1.06\times1.00×1.00\times
Retain8113.3642.8746.941.20×1.20\times1.13×1.13\times

We do not claim that these gains hold for every configuration. When more activations must be retained under a tight memory budget, a smaller micro-batch can indeed reduce the benefit. As future work, we plan to incorporate asynchronous offloading of timestep activations to reduce GPU memory pressure in long-trajectory training. These experiments show that LeanGRPO can still achieve end-to-end speedups in the tested configurations even when Native is allowed to use a larger micro-batch.

Thanks for the rebuttal, the authors have addressed most of my concerns. I will keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.