- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In GRPO-based diffusion RL methods such as Flow-GRPO and DanceGRPO, clipping of importance ratios is often used to prevent the optimized policy from drifting too far from the old policy. This paper shows that the form of clipping done in these methods fails to bound the KL divergence between the optimized and old policy. Analogously to GRPO-Guard, the authors present a formula for the stepwise KL between the old and optimized policy. This is then used to derive F-Clip, a method similar to GRPO-Guard; Z-Clip, which attempts to clip the same proportion of samples at each time step; and K-Hold, an adaptive weight multiplier that targets a given per-step KL divergence. These methods are then used on top of the existing Flow-GRPO and DanceGRPO baselines to post-train SD3.5-M and FLUX.2-Klein, and K-Hold is shown to increase held-out rewards at the final checkpoint.
Critical strengths
- Empirical performance of K-Hold. The performance of the K-Hold method is shown to exceed that of GRPO-Guard on SD3.5-M training. K-Hold is ablated against Z-Clip with rescaled gradients, and shown to outperform it. Other important ablations are performed: Figure 1 tests whether F-Clip and GRPO-Guard perform similarly, and shows that Z-Clip clips an equal proportion of samples at each time step.
Critical weaknesses
-
Contributions and presentation. Within section 4, the novelty, significance, and role of each of the theorems is unclear. For example, Theorem 1 is also derived in GRPO-Guard as Equation (7), and Proposition 2 is essentially rearranging definitions, which detracts from the novelty of the F-Clip method. Z-Clip seems to just be an implementation of quantile clipping based on Gaussian laws. There are several technical terms which are introduced in the main text that are not properly explained (e.g. profiles, actuators, Euclidean scales, Fisher metrics), and the writing is very dense, particularly in Section 4.4. Overall, I think the exposition given in section 4 could be greatly improved and condensed so the reasoning behind each method is transparent.
-
Insufficient empirical results. Due to the adaptive multiplier of K-Hold, which is shown to drift below 1 in Figure 2, training both GRPO-Guard and K-Hold for the same number of epochs can result in under-optimization of K-Hold compared to GRPO-Guard, which can account for some of the improvement in held-out rewards. While final checkpoint rewards are appreciated, main results should be under comparable proxy scores, as in GRPO-Guard, to account for this. Indeed, in Figure 2(d), K-Hold and GRPO-Guard appear to traverse the same reward-quality frontier. Moreover, only one task, OCR, is tested. Another should be chosen to validate this method. Since K-Hold reduces the effective learning rate, the time taken to reach a specified reward may be longer, hence the time taken for each method should be reported.
Questions for authors
- Could the authors explain the novelty of each of the results that appear in section 4, and if applicable, how it relates to GRPO-Guard?
- Could the authors provide results on at least one other task (e.g. a preference task), reporting matched reward results and times taken? Sampled images from non-cherry-picked prompts, further plots like Figure 2 (c, d), and errors from multiple runs would be appreciated, including for the existing OCR experiments.
- Could the abstract and section 4 be simplified? The motivations behind each method seem relatively clear, but this is hidden behind the dense terminology that could be removed from the main text and placed in the appendix.
5 comments
Response (2/3)
W1 / Q1 / Q3 (ii). The novelty of Thm. 1. Thm. 1 starts from the Gaussian log-ratio expansion used in GRPO-Guard (Eq. 7). We identify its shift as the per-step KL and express the decomposition as , with properties (i)–(iii) following directly. While the underlying identity is shared with GRPO-Guard, its interpretation in terms of KL provides the basis for our subsequent analysis. Because sets both the mean and the spread of the log-ratio, Thm. 1 connects clipping behavior to policy displacement throughout Sec. 4.
W1 / Q1 / Q3 (iii). The novelty of Prop. 2. Prop. 2 expresses the three candidate clipped variables as the same Gaussian coordinate multiplied by different scale factors. This shows what a fixed threshold constrains in each case: on the KL scale for the debiased log-ratio, in the raw latent coordinates for the normalized log-ratio of GRPO-Guard, and alone for the standardized form. The clip-fraction results (Prop. 3, Cor. 3) follow from this characterization, and the standardized form provides the basis for Z-Clip.
W1 / Q1 / Q3 (iv). Relation to GRPO-Guard. GRPO-Guard centers and rescales the per-step log-ratio to reduce uneven clipping across denoising steps. Our analysis starts from the same Gaussian log-ratio identity (GRPO-Guard's Eq. 7), but focuses on what the resulting clipping rules constrain. We show that the centered and rescaled log-ratios use different measures of policy displacement for clipping (Prop. 2). We further derive how rescaling affects the gradients of unclipped samples (Thm. 2), leading to F-Clip. Beyond F-Clip, Z-Clip calibrates the clipping fraction, while K-Hold controls the mean per-step KL through learning-rate feedback. This distinction between clipping calibration and KL control is central to our analysis. We have added this comparison to Sec. 2.
W1 / Q1 / Q3 (v). Terminology. The Fisher metric measures the displacement of the policy means in units of the policy's standard deviation, . The Euclidean scale measures it in the raw latent coordinates, . As the reviewer suggests, we removed both terms from the main text. The per-step KL profile is the sequence of per-step KL values , described by its overall level and the ratios between steps. An actuator is a mechanism that adjusts this overall level.
W1 / Q1 / Q3 (vi). Presentation. In the revised manuscript, we have simplified the abstract and improved the readability of Sec. 4, where the revisions are highlighted in blue. We hope these changes clarify the role and novelty of each result, and we kindly ask the reviewer to reassess our contributions in light of these clarifications.
Response (3/3)
W2 / Q2 (i). Comparison at matched OCR performance. We compare K-Hold and GRPO-Guard at matched OCR, as in GRPO-Guard, on both backbones.
| Backbone | Method | OCR 0.65 | OCR 0.75 | OCR 0.85 | Final |
|---|---|---|---|---|---|
| SD3.5-M | GRPO-Guard | +0.67 (29) | +0.65 (37) | +0.55 (46) | -0.38 |
| K-Hold | +0.81 (44) | +0.85 (58) | +0.85 (92) | +0.47 | |
| FLUX.2-klein | GRPO-Guard | +0.78 (21) | +0.80 (29) | +0.67 (56) | -0.04 |
| K-Hold | +0.62 (39) | +0.74 (54) | +0.76 (101) | +0.52 |
On FLUX.2-klein, Fig. 2(d) supports the reviewer's observation that GRPO-Guard performs slightly better at moderate OCR levels. However, its performance declines beyond OCR 0.85, whereas K-Hold maintains its performance and finishes above the base model.
W2 / Q2 (ii). A second task. We trained SD3.5-M using PickScore as the training objective under the main protocol. The Average is computed from the other three held-out rewards. We report the final scores in our response to Reviewer gCwv (W1). At matched PickScore, we compare the two methods as follows:
| Method | PickScore 22 | PickScore 22.5 | PickScore 23 | Final (PickScore / Average) |
|---|---|---|---|---|
| GRPO-Guard | +1.51 (29) | +1.77 (55) | +1.88 (104) | 23.52 / +1.92 |
| K-Hold | +1.50 (34) | +1.72 (58) | +1.92 (118) | 23.36 / +1.99 |
The two methods perform similarly at moderate PickScore levels. At PickScore 23, K-Hold achieves a higher Average (+1.92 vs. +1.88). It also performs better at the final checkpoint (+1.99 vs. +1.92). This follows the same pattern as the OCR task, but with smaller margins. PickScore is itself a human preference model that agrees with the held-out rewards, accounting for the smaller margins. The corresponding plot of Average against PickScore, similar to Fig. 2(c), is provided in App. F of the revised manuscript.
W2 / Q2 (iii). Error bars and uncurated samples. Since standard deviation estimates based on only two independent runs are less reliable, we conducted an additional run for both GRPO-Guard and K-Hold. For reference, the original Flow-GRPO and GRPO-Guard papers report results from a single run.
| Method | Avg.@OCR 0.65 | Avg.@OCR 0.75 | Avg.@OCR 0.85 | Final OCR | Final Avg. |
|---|---|---|---|---|---|
| GRPO-Guard | +0.72 ± 0.11 | +0.71 ± 0.15 | +0.65 ± 0.19 | 0.975 ± 0.005 | -0.50 ± 0.34 |
| K-Hold | +0.76 ± 0.17 | +0.80 ± 0.16 | +0.78 ± 0.19 | 0.956 ± 0.021 | +0.47 ± 0.18 |
The final gap is well beyond the spread of either method. At matched OCR, K-Hold has a higher mean at all three levels, with differences within one standard deviation. We have added uncurated samples in App. K, which shows the first eight test prompts of each task in file order. The base model, GRPO-Guard, and K-Hold use the same prompts and sampling seeds at the final checkpoint.
Response (1/3)
We thank the reviewer for the careful reading and constructive comments. Our responses to the concerns are provided below.
W1 / Q1 / Q3 (i). Main contributions. We begin with an intuitive explanation of our work. Our work asks a simple question: can clipping alone keep the per-step KL near a chosen target? The clipping fraction counts how many samples are clipped, while the per-step KL measures policy displacement relative to the policy noise scale. We explain why controlling the first does not control the second, and show how to control them separately.
We start from the Gaussian log-ratio identity that also used in GRPO-Guard, which relates the mean and variance of the log-ratio to the per-step KL. The clipping variable of GRPO-Guard relies on both the per-step KL and the policy noise scale. Hence, two denoising steps with the same KL can have different clipping fractions. Our gradient analysis further shows that the noise-scale factor only reweights the gradients of unclipped samples across steps.
F-Clip removes this factor from the clipping variable and puts clipping on a common KL scale. Two problems remain. First, the clipping fraction still changes across denoising steps and during training, causing uneven clipping of sample gradients. Second, although F-Clip clips more when KL is larger, the KL level at which clipping balances the update depends on the advantages, sampling settings, and denoising step.
A target KL therefore cannot be set through clipping alone. Z-Clip addresses the first problem by scaling the clipping range with the batch spread. This makes the clipping fraction independent of the overall KL level, up to variation in KL across states in a batch. With a stable clipping fraction, however, clipping no longer increases when the KL grows, so the second problem remains.
K-Hold keeps the Z-Clip loss and adjusts the learning rate based on the measured mean KL. It lowers the learning rate when the KL is above the target and raises it when the KL is below. Our analysis gives the size of this adjustment and the condition under which the mean KL converges to the target under fixed gradient statistics. The clipping fraction and the KL target can thus be chosen separately.
We also summarize the technical contribution of each subsection in Sec. 4 below.
In the revised manuscript, we have added an explicit comparison with GRPO-Guard in Sec. 2 and organized Sec. 4 around the clipped variable, the clipping range, and the update.