- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies update-stage forward recomputation in trajectory-based diffusion reinforcement learning. Under an unchanged policy snapshot and a common rollout/update execution path, it proposes two schedules that reuse rollout computation: Retain stores selected transition computation graphs until advantages become available, whereas Reweight immediately accumulates unweighted per-trajectory score-function gradients and applies the trajectory advantage afterward. A shared-prompt data-parallel layout distributes the memory burden, and Reweight delays gradient reduction until after advantage weighting. The paper derives equivalence to the native on-policy policy-loss gradient in exact arithmetic and evaluates latency, memory, numerical alignment, and reward convergence across several diffusion RL methods and backbones. It reports up to 1.83× end-to-end speedup. The main contribution is a systems execution strategy rather than a new policy-gradient estimator.
Critical strengths
-
A practically relevant optimization with complementary execution choices. Retain and Reweight target different limiting resources: saved activations across selected timesteps versus full per-trajectory gradients. The paper evaluates multiple algorithms, model sizes, fine-tuning modes, precisions, and GPU counts, and reports both successful and out-of-memory configurations. This makes the proposal more informative than a single-setting speedup and provides evidence that avoiding update-stage transition re-forwards can be practically useful.
-
Substantive numerical diagnostics beyond reward curves. Appendix F compares gradients, parameter-update vectors, and AdamW states under controlled transition data and real reward execution. It demonstrates exact agreement for Retain in the controlled setting and explicitly quantifies Reweight’s finite-precision differences rather than claiming bitwise identity. These diagnostics support the narrow mathematical argument and are a meaningful strength of the submission.
Critical weaknesses
-
Unclear whether LeanGRPO preserves the full training procedure. Eqs. (5)–(9) establish gradient equivalence at a fixed policy, but this does not necessarily guarantee equivalence over the entire training process. In particular, when multiple optimizer updates are performed on the same rollout, retained graphs are computed under the rollout policy and cannot be reused once an optimizer step changes the parameters, as Appendix H acknowledges. However, it is unclear whether the experiments preserve the same optimizer-step schedules as the original baselines. Additionally, the paper does not discuss KL regularization, and describes only within-group advantage normalization, whereas some canonical Flow-GRPO configurations use a batch-wide standard deviation. It is unclear how either would be handled under LeanGRPO’s prompt-by-prompt schedule, particularly for Reweight. I think these implementation details need further clarification, since differences in training procedures could affect both the reported speedup and convergence comparisons.
-
Unclear memory costs. I have concerns about the memory analysis. Table 1 and Eqs. (7) and (11) appear inconsistent in their memory scaling, and the paper does not sufficiently explain how Reweight obtains per-sample gradients when the local batch size is larger than one. I think these details are important since memory efficiency is a central motivation of the proposed methods.
Questions for authors
-
Exact training recipes. For each timing and convergence configuration, including those based on Flow-GRPO and DanceGRPO, please specify the number of optimizer updates per rollout, gradient-accumulation boundaries, auxiliary losses and KL coefficient, and advantage-normalization rules. Which implementation versions or commits are used, and which settings, if any, differ from the corresponding canonical implementations? Are any such changes applied identically to Native and LeanGRPO? If nonzero KL terms or batch-wide reward normalization are used, how are they handled under LeanGRPO’s prompt-by-prompt execution schedule while preserving the original update?
-
Autograd and memory semantics. Please give the exact definition of the gradient-tracked denoising step ( in Algorithm 1): which tensors carry gradients, and whether the input and the sampled are detached when computing . How are separate per-trajectory gradients maintained when multiple samples are processed on the same rank ()? Under what assumptions do Eqs. (7) and (11) omit ? Please reconcile these expressions with Table 1 and account for the simultaneously retained buffers when and , including gradient shards accumulated from previous prompt groups.
-
Measurement protocol. Why do the LoRA Retain/Reweight times in Table 2 (901.99/880.68 s) differ from Section 4.3/Fig. 4 (960.69/956.36 s), despite referring to the same nominal configuration? How are Fig. 8’s projected times computed, which overheads are included, and how are time-to-target thresholds selected across seeds? Are latency and peak-memory measurements taken after warmup and optimizer-state initialization, and what parameter, gradient, optimizer-state precision and offloading settings are used?
-
Minor Issue: Notation Clarity. The main text introduces only as the “update policy” and then sets by assumption. Defining explicitly as the parameters at which the update-stage term is evaluated would clarify its relationship to optimizer-step boundaries, which currently appears only in Appendix H. Similarly, in Section 3.1 should be formally defined and distinguished from and , including their gradient-tracking and stop-gradient semantics. Moving or adapting the consolidated notation table from page 16 of the Appendix into the main text would improve readability and make the assumptions behind the gradient-equivalence derivations easier to follow.
9 comments
Q1-3: If nonzero KL terms or batch-wide reward normalization are used, how are they handled under LeanGRPO's prompt-by-prompt execution schedule while preserving the original update?
A1-3-1: KL Regularization.
We thank the reviewer for pointing out this omission. One reason is that some algorithms, such as DanceGRPO and BranchGRPO, do not include explicit KL regularization in the implementations we evaluated. Another reason is that even algorithms that support KL regularization, such as Flow-GRPO, disable it in some released training configurations. For DanceGRPO, our experiments therefore omit the KL term, consistent with its native implementation. For FlowGRPO and MixGRPO, we set the corresponding KL coefficients to zero consistently across Native, Retain, and Reweight to ensure a fair comparison.
However, both LeanGRPO schedules can accommodate an additive fixed-reference KL regularizer. Using exactly the same KL definition and loss normalization as Native, the total gradient is
The reference predictions are computed without gradient tracking, while gradients propagate through the current-policy predictions.
For Retain, enabling KL only requires an additional reference-model forward pass at each selected timestep. The policy loss and KL regularizer can be combined and backpropagated together once the advantages are available.
For Reweight, the policy and KL gradient contributions must remain separate until advantage correction is applied to the policy gradients. The KL contribution is not multiplied by the advantage, so it can be reduced, sharded, and accumulated independently. The corrected policy gradients are subsequently reduced and added to the accumulated KL gradients. Consequently, this implementation need not retain a second full, unsharded KL-gradient buffer throughout rollout. However, Reweight requires an additional backward pass to compute and accumulate separately. The reference model's forward pass also reduces the fraction of end-to-end training time attributable to recomputation, thereby limiting LeanGRPO's speedup.
Table 2. SD3.5-Medium training latency with reference KL regularization (). Times are mean standard deviation in seconds over three updates after warm-up.
| Native time (s) | Retain time (s) | Retain speedup | Reweight time (s) | Reweight speedup | ||
|---|---|---|---|---|---|---|
| 0.4 | 4 | |||||
| 0.6 | 6 | |||||
| 0.8 | 8 | |||||
| 1.0 | 10 |
Table 2 shows a comparison using the same configuration as Fig. 3(i), but with KL regularization enabled. The results show that Retain achieves a smaller speedup than in the KL-disabled setting because the additional reference-model forward pass reduces the fraction of end-to-end training time attributable to recomputation. Reweight does not degrade the results but achieves almost no speedup, primarily because computing separately requires an additional backward pass, offsetting the gains from eliminating recomputation. Nevertheless, Reweight remains valuable in most KL-disabled settings and achieves runtimes close to Native in the few KL-enabled settings, while Retain achieves favorable speedups both with and without KL regularization.
A1-3-2: Batch-Wide Reward Normalization.
LeanGRPO supports three normalization rules, but the main text focuses on the first. We have therefore added Appendix M to explain how LeanGRPO supports the other two normalization rules. Let denote the unweighted policy gradient of trajectory , summed over its selected timesteps. Here, "group-wise" refers to computations within a single prompt group.
1. Group-wise mean and standard deviation.
This is the setting primarily described in the main text, where advantages become available after each prompt group is completed. The computation therefore follows the procedure described in the main text.
2. Group-wise mean and batch-wide standard deviation.
This rule was already used in our FLUX convergence experiments but was not explicitly described in the main text. Specifically,
The policy gradient therefore satisfies
Once a prompt group's rewards are available, its values are known. Reweight applies these values to the temporary gradients before aggregation and reduce-scatter. Retain instead backpropagates the -weighted losses through the retained graphs. Both schedules accumulate gradient shards without the batch-wide denominator. Once all batch rewards are available, they divide the accumulated policy-gradient shards by , recovering the same gradient without an additional backward pass or retaining the entire batch's graphs.
Therefore, this normalization introduces no additional memory or computational overhead.
3. Batch-wide mean and standard deviation.
A_i=\frac{r_i-\bar r_{\mathrm{batch}}}{d_{\mathrm{batch}}}. \tag{1}
Neither batch statistic is generally known when an individual prompt group finishes. However, the policy gradient admits the decomposition
\begin{aligned} P^{\mathrm{norm}}&=\sum_{i\in\mathcal{B}}r_iG_i^{\mathrm{tmp}}, &Q^{\mathrm{norm}}&=\sum_{i\in\mathcal{B}}G_i^{\mathrm{tmp}},\\\\ G_{\mathrm{policy}} &=\frac{P^{\mathrm{norm}}-\bar r_{\mathrm{batch}}Q^{\mathrm{norm}}} {d_{\mathrm{batch}}}. \end{aligned} \tag{2}
Reweight. During rollout, Reweight backpropagates each selected timestep's unweighted policy loss before reward computation. With one sample per rank, these contributions form the local without reduce-scatter. Once the prompt-group reward is available, Reweight shallow-copies the list of gradient-tensor references, not the tensor data, so that the same can be reused. Two sequential FSDP2 reduce-scatter passes then form separate shards: the unweighted pass accumulates into , and the reward-weighted pass accumulates into . Once all rewards are available, we weight by , subtract it from , and divide the result by . Only the two sharded accumulators persist across prompt groups; forming them requires neither another backward pass nor two full-model gradient copies.
Retain. After a prompt group's rewards become available, Retain can construct an unweighted loss for each trajectory and backpropagate it through the retained graphs to obtain . It then separately accumulates and into the and shards, respectively. Once rollout for the entire batch is complete and is available, the shard is scaled by and added to the shard. The result is then divided by .
However, our survey found very few open-source training scripts that use batch-wide mean and standard deviation. We therefore think that this configuration may not be suitable for practical training.
First of all, we thank you for your time and detailed review. Your questions are insightful and meaningful, and we believe they will help us improve the completeness and clarity of our work. We have carefully considered every question and welcome further discussion if you have additional questions after reading our responses.
Q1-1: For each timing and convergence configuration, including those based on Flow-GRPO and DanceGRPO, please specify the number of optimizer updates per rollout, gradient-accumulation boundaries, auxiliary losses and KL coefficient, and advantage-normalization rules.
A1-1. LeanGRPO targets redundant recomputation in the on-policy setting. In our experiments, each rollout batch is used for exactly one optimizer update. For example, the FLUX.1-dev convergence configuration in Fig. 6(b) accumulates gradients over 24 prompt groups, with eight samples per prompt and five selected timesteps per sample, before performing one optimizer step. Gradients from each timestep are accumulated until
optimizer.step()is invoked, and policy parameters remain unchanged throughout this interval.Some training configurations perform multiple optimizer updates on the same rollout batch. After the first update, the current policy differs from the rollout policy, so subsequent updates fall outside the on-policy setting considered in this work. Further discussion is provided in Appendices H and L.
Auxiliary losses, KL coefficients, and advantage-normalization rules are discussed in A1-3.
Q1-2: Which implementation versions or commits are used, and which settings, if any, differ from the corresponding canonical implementations?
A1-2. LeanGRPO is developed from the official open-source implementations of FlowGRPO, DanceGRPO, and MixGRPO. Our modifications aim to accelerate training while preserving the native optimization objective, without changing the core algorithms. The upstream implementation versions and base commits are listed below; the detailed settings for each experimental configuration are documented in the newly added Appendix L.
Table 1. Official upstream repositories and base commits used to develop our integrations.
15cc71d53cc2e6e18a68ee607d5fb6ba9a99e3449745241cf1101407e9b518dd4f10498092150522e7f299d40e3309b18880ec7717e7024c3aa714bb