acceptodds
Under review as a conference paper at ICLR 2027

Holding the Per-Step KL: Why Clipping Cannot, but Control Can

Abstract

GRPO-based reinforcement learning (RL) aligns flow-matching models with task rewards, but ratio clipping does not directly control the KL divergence of policy updates. We analyze this gap through an exact Gaussian decomposition of the per-step log-ratio, in which the per-step KL sets both its mean and variance. This reveals a distinction between calibrating the clipping frequency and controlling policy displacement: for the centered threshold rules considered, standardization removes the first-order dependence of the clipping gate on the overall displacement scale. Building on this distinction, we develop the method in three layers. F-Clip clips the centered log-ratio in KL-scaled coordinates; Z-Clip adapts the clipping range to the batch spread, so that its threshold corresponds to a target clipping fraction; and K-Hold uses learning-rate feedback to keep the floor-corrected mean KL over selected denoising steps near a prescribed target. Our analysis provides a unified characterization of ratio clipping in flow-matching RL, connecting the per-step log-ratio to the KL quantity that must ultimately be controlled. Experiments on OCR-based text rendering with SD3.5-M and FLUX.2-klein validate the predicted clipping and KL behavior. K-Hold improves held-out reward-model scores over the baselines, while preserving the gain under matched OCR performance.

Primary area:
generative models
Keywords:
Flow Matching, Policy Optimization, KL Control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3Q2iOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In GRPO-based diffusion RL methods such as Flow-GRPO and DanceGRPO, clipping of importance ratios is often used to prevent the optimized policy from drifting too far from the old policy. This paper shows that the form of clipping done in these methods fails to bound the KL divergence between the optimized and old policy. Analogously to GRPO-Guard, the authors present a formula for the stepwise KL between the old and optimized policy. This is then used to derive F-Clip, a method similar to GRPO-Guard; Z-Clip, which attempts to clip the same proportion of samples at each time step; and K-Hold, an adaptive weight multiplier that targets a given per-step KL divergence. These methods are then used on top of the existing Flow-GRPO and DanceGRPO baselines to post-train SD3.5-M and FLUX.2-Klein, and K-Hold is shown to increase held-out rewards at the final checkpoint.

Critical strengths
  1. Empirical performance of K-Hold. The performance of the K-Hold method is shown to exceed that of GRPO-Guard on SD3.5-M training. K-Hold is ablated against Z-Clip with rescaled gradients, and shown to outperform it. Other important ablations are performed: Figure 1 tests whether F-Clip and GRPO-Guard perform similarly, and shows that Z-Clip clips an equal proportion of samples at each time step.
Critical weaknesses
  1. Contributions and presentation. Within section 4, the novelty, significance, and role of each of the theorems is unclear. For example, Theorem 1 is also derived in GRPO-Guard as Equation (7), and Proposition 2 is essentially rearranging definitions, which detracts from the novelty of the F-Clip method. Z-Clip seems to just be an implementation of quantile clipping based on Gaussian laws. There are several technical terms which are introduced in the main text that are not properly explained (e.g. profiles, actuators, Euclidean scales, Fisher metrics), and the writing is very dense, particularly in Section 4.4. Overall, I think the exposition given in section 4 could be greatly improved and condensed so the reasoning behind each method is transparent.

  2. Insufficient empirical results. Due to the adaptive multiplier of K-Hold, which is shown to drift below 1 in Figure 2, training both GRPO-Guard and K-Hold for the same number of epochs can result in under-optimization of K-Hold compared to GRPO-Guard, which can account for some of the improvement in held-out rewards. While final checkpoint rewards are appreciated, main results should be under comparable proxy scores, as in GRPO-Guard, to account for this. Indeed, in Figure 2(d), K-Hold and GRPO-Guard appear to traverse the same reward-quality frontier. Moreover, only one task, OCR, is tested. Another should be chosen to validate this method. Since K-Hold reduces the effective learning rate, the time taken to reach a specified reward may be longer, hence the time taken for each method should be reported.

Questions for authors
  1. Could the authors explain the novelty of each of the results that appear in section 4, and if applicable, how it relates to GRPO-Guard?
  2. Could the authors provide results on at least one other task (e.g. a preference task), reporting matched reward results and times taken? Sampled images from non-cherry-picked prompts, further plots like Figure 2 (c, d), and errors from multiple runs would be appreciated, including for the existing OCR experiments.
  3. Could the abstract and section 4 be simplified? The motivations behind each method seem relatively clear, but this is hidden behind the dense terminology that could be removed from the main text and placed in the appendix.
5 comments
AuthorsOpenReview ↗

Response (1/3)

We thank the reviewer for the careful reading and constructive comments. Our responses to the concerns are provided below.

W1 / Q1 / Q3 (i). Main contributions. We begin with an intuitive explanation of our work. Our work asks a simple question: can clipping alone keep the per-step KL near a chosen target? The clipping fraction counts how many samples are clipped, while the per-step KL measures policy displacement relative to the policy noise scale. We explain why controlling the first does not control the second, and show how to control them separately.

We start from the Gaussian log-ratio identity that also used in GRPO-Guard, which relates the mean and variance of the log-ratio to the per-step KL. The clipping variable of GRPO-Guard relies on both the per-step KL and the policy noise scale. Hence, two denoising steps with the same KL can have different clipping fractions. Our gradient analysis further shows that the noise-scale factor only reweights the gradients of unclipped samples across steps.

F-Clip removes this factor from the clipping variable and puts clipping on a common KL scale. Two problems remain. First, the clipping fraction still changes across denoising steps and during training, causing uneven clipping of sample gradients. Second, although F-Clip clips more when KL is larger, the KL level at which clipping balances the update depends on the advantages, sampling settings, and denoising step.

A target KL therefore cannot be set through clipping alone. Z-Clip addresses the first problem by scaling the clipping range with the batch spread. This makes the clipping fraction independent of the overall KL level, up to variation in KL across states in a batch. With a stable clipping fraction, however, clipping no longer increases when the KL grows, so the second problem remains.

K-Hold keeps the Z-Clip loss and adjusts the learning rate based on the measured mean KL. It lowers the learning rate when the KL is above the target and raises it when the KL is below. Our analysis gives the size of this adjustment and the condition under which the mean KL converges to the target under fixed gradient statistics. The clipping fraction and the KL target can thus be chosen separately.

We also summarize the technical contribution of each subsection in Sec. 4 below.

  • Primitives (Sec. 4.1). Thm. 1 expresses the per-step log-ratio in terms of the per-step KL κt\kappa_t, following the Gaussian identity also derived in Eq. (7) of GRPO-Guard. The log-ratio has mean −κt-\kappa_t and variance 2κt2\kappa_t. This identity provides the basis for analyzing clipping geometry, clipping fraction, and KL control in the subsequent sections.
  • F-Clip (Sec. 4.2). Prop. 2 distinguishes the clipping geometries of the centered and normalized log-ratios. The contribution lies in the gradient analysis: Lem. 1 shows that centering cancels the displacement term in the gradient, while Thm. 2 establishes how ratio rescaling interacts with gradient weighting. These results provide the basis for removing the scale factor in GRPO-Guard, yielding F-Clip with a simpler clipped variable and a closely related gradient.
  • Z-Clip (Sec. 4.3). Prop. 3 shows that a fixed clipping threshold produces a clipping fraction that varies with policy displacement. While Gaussian quantile clipping is standard, Prop. 4 establishes the scale invariance of the batch-standardized threshold and characterizes the deviation caused by within-batch KL variation. This analysis leads to Z-Clip, which uses a target clipping fraction instead of a fixed absolute threshold.
  • K-Hold (Sec. 4.4). Prop. 5 establishes the central limitation: under the considered threshold rules, a uniform clipping fraction removes the first-order gradient response to displacement magnitude. Clipping calibration therefore cannot independently enforce a target KL. Lem. 4 establishes the quadratic dependence of KL on update scale, while Prop. 6 characterizes the KL profile under the stated coupling assumptions. These results motivate K-Hold, which adjusts the learning rate to control the mean per-step KL. Lem. 7 characterizes the controller's convergence under fixed gradient statistics.

In the revised manuscript, we have added an explicit comparison with GRPO-Guard in Sec. 2 and organized Sec. 4 around the clipped variable, the clipping range, and the update.

AuthorsOpenReview ↗

Response (2/3)

W1 / Q1 / Q3 (ii). The novelty of Thm. 1. Thm. 1 starts from the Gaussian log-ratio expansion used in GRPO-Guard (Eq. 7). We identify its shift as the per-step KL κt\kappa_t and express the decomposition as Lt=−κt−2κt,zL_t=-\kappa_t-\sqrt{2\kappa_t}\\,z, with properties (i)–(iii) following directly. While the underlying identity is shared with GRPO-Guard, its interpretation in terms of KL provides the basis for our subsequent analysis. Because κt\kappa_t sets both the mean and the spread of the log-ratio, Thm. 1 connects clipping behavior to policy displacement throughout Sec. 4.

W1 / Q1 / Q3 (iii). The novelty of Prop. 2. Prop. 2 expresses the three candidate clipped variables as the same Gaussian coordinate zz multiplied by different scale factors. This shows what a fixed threshold constrains in each case: κtz2\kappa_t z^2 on the KL scale for the debiased log-ratio, ∥Δμt∥,∣z∣\lVert\Delta\mu_t\rVert\\,\lvert z\rvert in the raw latent coordinates for the normalized log-ratio of GRPO-Guard, and ∣z∣\lvert z\rvert alone for the standardized form. The clip-fraction results (Prop. 3, Cor. 3) follow from this characterization, and the standardized form provides the basis for Z-Clip.

W1 / Q1 / Q3 (iv). Relation to GRPO-Guard. GRPO-Guard centers and rescales the per-step log-ratio to reduce uneven clipping across denoising steps. Our analysis starts from the same Gaussian log-ratio identity (GRPO-Guard's Eq. 7), but focuses on what the resulting clipping rules constrain. We show that the centered and rescaled log-ratios use different measures of policy displacement for clipping (Prop. 2). We further derive how rescaling affects the gradients of unclipped samples (Thm. 2), leading to F-Clip. Beyond F-Clip, Z-Clip calibrates the clipping fraction, while K-Hold controls the mean per-step KL through learning-rate feedback. This distinction between clipping calibration and KL control is central to our analysis. We have added this comparison to Sec. 2.

W1 / Q1 / Q3 (v). Terminology. The Fisher metric measures the displacement of the policy means in units of the policy's standard deviation, ∥Δμt∥/st\lVert\Delta\mu_t\rVert/s_t. The Euclidean scale measures it in the raw latent coordinates, ∥Δμt∥\lVert\Delta\mu_t\rVert. As the reviewer suggests, we removed both terms from the main text. The per-step KL profile is the sequence of per-step KL values (κ0,…,κT−1)(\kappa_0,\dots,\kappa_{T-1}), described by its overall level and the ratios κt/κt′\kappa_t/\kappa_{t'} between steps. An actuator is a mechanism that adjusts this overall level.

W1 / Q1 / Q3 (vi). Presentation. In the revised manuscript, we have simplified the abstract and improved the readability of Sec. 4, where the revisions are highlighted in blue. We hope these changes clarify the role and novelty of each result, and we kindly ask the reviewer to reassess our contributions in light of these clarifications.

AuthorsOpenReview ↗

Response (3/3)

W2 / Q2 (i). Comparison at matched OCR performance. We compare K-Hold and GRPO-Guard at matched OCR, as in GRPO-Guard, on both backbones.

BackboneMethodOCR 0.65OCR 0.75OCR 0.85Final
SD3.5-MGRPO-Guard+0.67 (29)+0.65 (37)+0.55 (46)-0.38
K-Hold+0.81 (44)+0.85 (58)+0.85 (92)+0.47
FLUX.2-kleinGRPO-Guard+0.78 (21)+0.80 (29)+0.67 (56)-0.04
K-Hold+0.62 (39)+0.74 (54)+0.76 (101)+0.52

On FLUX.2-klein, Fig. 2(d) supports the reviewer's observation that GRPO-Guard performs slightly better at moderate OCR levels. However, its performance declines beyond OCR 0.85, whereas K-Hold maintains its performance and finishes above the base model.

W2 / Q2 (ii). A second task. We trained SD3.5-M using PickScore as the training objective under the main protocol. The Average is computed from the other three held-out rewards. We report the final scores in our response to Reviewer gCwv (W1). At matched PickScore, we compare the two methods as follows:

MethodPickScore 22PickScore 22.5PickScore 23Final (PickScore / Average)
GRPO-Guard+1.51 (29)+1.77 (55)+1.88 (104)23.52 / +1.92
K-Hold+1.50 (34)+1.72 (58)+1.92 (118)23.36 / +1.99

The two methods perform similarly at moderate PickScore levels. At PickScore 23, K-Hold achieves a higher Average (+1.92 vs. +1.88). It also performs better at the final checkpoint (+1.99 vs. +1.92). This follows the same pattern as the OCR task, but with smaller margins. PickScore is itself a human preference model that agrees with the held-out rewards, accounting for the smaller margins. The corresponding plot of Average against PickScore, similar to Fig. 2(c), is provided in App. F of the revised manuscript.

W2 / Q2 (iii). Error bars and uncurated samples. Since standard deviation estimates based on only two independent runs are less reliable, we conducted an additional run for both GRPO-Guard and K-Hold. For reference, the original Flow-GRPO and GRPO-Guard papers report results from a single run.

MethodAvg.@OCR 0.65Avg.@OCR 0.75Avg.@OCR 0.85Final OCRFinal Avg.
GRPO-Guard+0.72 ± 0.11+0.71 ± 0.15+0.65 ± 0.190.975 ± 0.005-0.50 ± 0.34
K-Hold+0.76 ± 0.17+0.80 ± 0.16+0.78 ± 0.190.956 ± 0.021+0.47 ± 0.18

The final gap is well beyond the spread of either method. At matched OCR, K-Hold has a higher mean at all three levels, with differences within one standard deviation. We have added uncurated samples in App. K, which shows the first eight test prompts of each task in file order. The base model, GRPO-Guard, and K-Hold use the same prompts and sampling seeds at the final checkpoint.

Reviewer_gCwvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies that standard ratio clipping does not directly control the per-step KL divergence of policy updates. Through an exact Gaussian decomposition, the authors reveal that the per-step KL determines both the mean and variance of the log-ratio, exposing a structural trade-off where no fixed threshold rule can simultaneously maintain uniform clipping across denoising steps and respond to policy displacement. Building on this theoretical analysis, the method is developed in three progressive layers F-Clip, Z-Clip and K-Hold. Experiments on OCR-based text rendering with SD3.5-M and FLUX.2-klein validate the predicted clipping and KL behavior.

Critical strengths

The paper provides a mathematically exact Gaussian decomposition of the per-step log-ratio in flow-matching RL. By proving that the per-step KL divergence simultaneously determines both the mean shift and variance of the log-ratio, the authors establish a clear theoretical framework that explains why standard clipping mechanisms fail to directly control policy displacement.

The paper proposes solutions structured in three progressive layers: F-Clip, Z-Clip, and K-Hold. This layered approach offers improved practical usability compared to fixed-threshold baselines like GRPO-Guard.

The experiments are conducted on two distinct flow-matching models (SD3.5-M and FLUX.2-klein) using LoRA fine-tuning. The results comprehensively validate the theoretical predictions: F-Clip maintains the expected KL profile, Z-Clip successfully stabilizes the clip fraction across denoising steps, and K-Hold actively regulates the measured per-epoch KL near its target.

The paper includes detailed appendices containing full proofs, explicit justification of all mathematical assumptions via empirical measurements, and complete experimental configurations. This comprehensive documentation significantly facilitates reproducibility.

Critical weaknesses
  1. The experimental validation is focused on a single downstream task: visual text rendering optimized via OCR accuracy. While the authors evaluate general image quality using held-out reward models, the absence of experiments on other alignment tasks—such as diverse human preference optimization, prompt adherence, or video generation—limits the demonstrated generalizability of the proposed methods.
  2. K-Hold introduces a closed-loop feedback mechanism that requires measuring the floor-corrected mean KL after the final update of every epoch to adjust the learning rate multiplier. This adds computational and implementation overhead compared to standard open-loop PPO/GRPO clipping, though the exact latency impact is not quantified in the text.
  3. While K-Hold improves held-out reward-model scores compared to baselines, information is missing on whether the proposed explicit per-step KL control fundamentally prevents reward hacking more effectively than standard KL penalty terms.
Questions for authors
  1. K-Hold explicitly controls the per-step KL divergence to restrict the magnitude of policy displacement, but does this theoretically or empirically prevent reward hacking (i.e., the model exploiting imperfections in the reward function)? Does the method merely delay over-optimization, or does it fundamentally block the model from discovering adversarial shortcuts? Compared to standard KL penalty terms, does K-Hold offer a significant advantage in defending against reward hacking?
  2. K-Hold requires measuring the mean KL over selected denoising steps after each epoch to update the learning-rate multiplier. How much additional computational latency does this closed-loop feedback mechanism introduce? Is this overhead acceptable in large-scale production environments?
1 comment

We thank the reviewer for the careful reading and constructive comments. Our responses to the concerns are provided below.

W1. Evaluation on a single task. To evaluate K-Hold beyond the OCR task, we added human preference optimization using PickScore as the training reward for SD3.5-M, while keeping the original training configuration unchanged except for the reward function. The Average is the mean standardized gain over the base model across the three held-out reward models, evaluated on 256 test prompts as in Tab. 1. Note that all scores are computed on the PickScore test prompts, so they are not comparable to those in Tab. 1.

MethodPickScoreHPSv2ImageRewardUnifiedRewardAverage
Base (SD3.5-M)19.920.210-0.4342.406+0.00
+Flow-GRPO22.890.3221.2963.656+1.82
+GRPO-Guard23.520.3321.3643.690+1.92
+K-Hold (Ours)23.360.3381.3693.729+1.99

As shown in the table, K-Hold achieves the highest scores on all three held-out reward models and the highest Average. Its PickScore is close to that of GRPO-Guard, at 23.36 vs. 23.52. In this experiment, the training reward and held-out rewards improve together, with no observed degradation during training. In contrast, OCR optimization exhibits a gap between increasing training reward and declining held-out quality.

W2 / Q2. Overhead of the feedback loop. After the final update of each epoch, K-Hold measures the KL on a fixed probe batch of 8 samples at the control steps and updates the learning-rate multiplier mm. This requires additional forward passes but no backward pass. We measured the training time over 240 epochs on SD3.5-M. We also include GRPO-Guard with a standard KL penalty for comparison with W3 / Q1. The penalty is implemented as in Flow-GRPO by adding β,∥μθ−μref∥2/(2σt2)\beta\\,\lVert\mu_\theta-\mu_{\mathrm{ref}}\rVert^2/(2\sigma_t^2) to the GRPO-Guard loss at every trained step, with β=0.04\beta=0.04 and μref\mu_{\mathrm{ref}} denoting the mean of the base model.

MethodSeconds per epochTraining time (240 epochs)
Flow-GRPO363.324.22 h
GRPO-Guard361.624.11 h
Z-Clip363.924.26 h
K-Hold (Ours)376.625.11 h
GRPO-Guard + KL penalty454.030.27 h

As shown in the table, K-Hold incurs only a 3.5% overhead compared with Z-Clip. Since the probe batch is fixed, this additional cost does not grow with the number of training samples per epoch.

W3 / Q1. Reward hacking and the KL penalty. K-Hold does not prevent reward hacking [1]. It regulates the mean per-step KL in each epoch, but policy drift can accumulate across epochs. In our OCR experiments, the held-out score still decreases from +0.86 at OCR 0.8 to +0.74 at OCR 0.9, and then to +0.47 at the final OCR 0.959. The key question is whether K-Hold improves the trade-off between training reward and held-out quality or merely slows optimization. To explore this, we added a standard KL penalty to GRPO-Guard with β=0.04\beta=0.04.

MethodAvg.@OCR 0.8Avg.@OCR 0.9Final (OCR / Avg.)
GRPO-Guard+0.60+0.460.974 / -0.38
GRPO-Guard + KL penalty+0.53+0.430.977 / -0.09
K-Hold+0.86+0.740.959 / +0.47

At matched OCR, the KL penalty does not improve held-out scores over GRPO-Guard. In contrast, K-Hold achieves higher scores at both OCR levels, exceeding the KL-penalty baseline by 0.33 and 0.31, respectively. The KL penalty also keeps the model closer to the base model, with 377 versus 1144 nats at OCR 0.8. These results show that K-Hold improves the measured reward-quality trade-off, rather than merely slowing optimization. However, it does not guarantee protection against reward hacking. Please refer to App. G of the revised manuscript for additional experimental results.

[1] Gao et al., "Scaling laws for reward model overoptimization", 2023, ICML

Reviewer_TXsnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes Gaussian per-step policies in flow-model GRPO and separates clipping calibration from KL control. It introduces F-Clip, batch-standardized Z-Clip, and K-Hold, which adjusts the learning rate to maintain a target mean KL. OCR experiments on two backbones show improved held-out reward scores.

Critical strengths
  • The exact Gaussian log-ratio decomposition gives a useful account of why centered, standardized clipping need not control policy displacement.
  • Experiments combine two backbones, four held-out reward models, matched-OCR comparisons, fixed-learning-rate controls, and a second sampler, providing substantive evidence against simple training slowdown.
Critical weaknesses
  • The analyzed and implemented corrections differ. Appendix C states that the code retains a batch-mean KL correction, whereas Lemma 1 and Proposition 4 use per-state corrections. With heterogeneous states, the stated per-sample gradient cancellation and centered-noise calibration no longer follow; the discrepancy needs quantitative assessment.
  • Lemma 7 ignores the log-KL smoothing used in Eq. (13). Its claimed error contraction by 1-g and one-epoch convergence at g=1 hold for the unsmoothed, unsaturated controller, not generally for the implemented eta=0.35. The actual stability and convergence analysis needs correction.
  • Attribution remains incomplete: Table 2 compares fixed learning rates, but does not apply the same KL-feedback controller directly to GRPO-Guard or compare against a matched time-varying learning-rate schedule. Since F-Clip and Z-Clip provide little downstream improvement, these controls matter for establishing the value of the proposed stack.
Questions for authors
  1. Can you compare per-state versus batch-mean correction, reporting gradient discrepancies, clipping calibration, and downstream scores?
  2. Does K-Hold retain its advantage over GRPO-Guard with the identical controller and over a replayed learning-rate schedule, at matched OCR performance?
3 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for the careful reading and constructive comments. Our responses to the concerns are provided below.

W1 / Q1. Batch-mean vs. per-state correction. We agree that our implementation uses the batch-mean correction kˉt\bar k_t, while Lem. 1 and Prop. 4 analyze the per-state correction kt,xk_{t,x}. This difference affects two theoretical properties. First, the displacement term in Lem. 1 no longer cancels for each sample, leaving a residual (Δμt,x⊤Jt,x−EB[Δμt⊤Jt])/(d,st2)(\Delta\mu_{t,x}^\top J_{t,x}-\mathbb{E}_B[\Delta\mu_t^\top J_t])/(d\\,s_t^2). Second, the corrected log-ratio is no longer centered at each state. Its remaining mean offset, measured in units of the state-specific standard deviation, is ρx=(kˉt−kt,x)/2kt,x/d\rho_x=(\bar k_t-k_{t,x})/\sqrt{2k_{t,x}/d}. We quantified these differences using K-Hold checkpoints at epochs 20, 100, and 200, across all trained denoising steps. At each checkpoint, we evaluated both corrections on the same batch of 48 samples:

MetricValue
Median ∣ρ∣\lvert\rho\rvert0.09–0.15
Gradient cosine, batch-mean vs. per-state0.89–0.96
   reference: two batches, same constant0.02–0.05
clipping fraction, batch-mean / per-state (nominal 0.674)0.647 / 0.650
Clip decisions that change8.9%

As shown in the table, the gradients under the two corrections remain closely aligned, with cosine similarities of 0.89–0.96. This is substantially higher than the 0.02–0.05 observed between different batches under the same correction. At the sample level, the correction changes 8.9% of clipping decisions, close to the 7–11% estimated from the Gaussian first-order approximation 2ϕ(z⋆)E∣ρ∣≈0.73E∣ρ∣2\phi(z^\star)\mathbb{E}|\rho| \approx 0.73\mathbb{E}|\rho|. These changes largely balance out across the batch, leaving nearly identical clipping fractions of 0.647 and 0.650. Both remain below the nominal target of 0.674, as predicted by Prop. 4(ii) for across-state KL heterogeneity.

To test whether this difference affects downstream performance, we retrained K-Hold using the per-state correction under the same experimental protocol:

MethodAvg.@OCR 0.8Avg.@OCR 0.9Final (OCR / Avg.)
GRPO-Guard+0.60+0.460.974 / -0.38
K-Hold, batch-mean (paper)+0.86+0.740.959 / +0.47
K-Hold, per-state+0.81+0.680.968 / +0.21

With per-state correction, K-Hold still achieves higher held-out rewards than GRPO-Guard at matched OCR performance. Its final held-out score also remains above the base model. These results show that K-Hold's advantage over GRPO-Guard is preserved under both correction choices.

W2. Lem. 7 and the log-KL smoothing. We agree that Lem. 7 does not account for log-KL smoothing. The log-gain ∂log⁡κˉ/∂log⁡m=2\partial\log\bar\kappa/\partial\log m=2 remains valid, but the claimed contraction rate 1−g1-g holds only when η=1\eta=1.

To characterize the implemented controller, we incorporate smoothing into the error dynamics. Let en=log⁡(κˉn/κ⋆)e_n=\log(\bar\kappa_n/\kappa^\star) denote the log-error at epoch nn, and e^n\hat e_n its smoothed value. Under fixed gradient statistics and without learning-rate saturation, Eq. (13) gives

\hat e_{n+1}=(1-\eta)\\,\hat e_n+\eta\\,e_{n+1}.$$ These coupled updates yield the characteristic equation $$\lambda^2-(2-\eta-g\eta)\lambda+(1-\eta)=0.$$ The smoothed controller is asymptotically stable if and only if $g\eta<4-2\eta$ for $\eta\in(0,1]$ and $g>0$. Our implemented settings $g=0.75$ and $\eta=0.35$ satisfy this condition. The characteristic roots are complex with modulus $\sqrt{1-\eta}\approx0.81$. Consequently, the log-error exhibits damped oscillations with a period of approximately 12 epochs. Its oscillation envelope decays by a factor of 0.81 per epoch, instead of the originally claimed 0.25. When $\eta=1$, the roots reduce to $0$ and $1-g$, recovering the original contraction rate. Thus, smoothing changes the convergence behavior but preserves stability under the implemented settings and the stated assumptions. The steady-state offset $\nu/g$ under open-loop drift in Rem. 16 and the KL profile invariance remain unchanged. The stability result applies to the unsaturated controller under Ass. 5 (A2). We have revised Lem. 7, its proof, and Rem. 17 accordingly.
AuthorsOpenReview ↗

Response (2/2)

W3 / Q2. Attribution of the gain. To test whether K-Hold maintains its advantage under the same learning-rate feedback controller and replayed learning-rate schedules, we conducted three experiments on SD3.5-M under the main protocol. First, we applied Actuator C to GRPO-Guard using the same controller settings as K-Hold. Second, we repeated both methods without feedback, replaying their recorded learning-rate multiplier trajectories m1m_1 and m2m_2, respectively. Third, we trained Z-Clip with an exponential schedule fitted to m2m_2.

MethodAvg.@OCR 0.8Avg.@OCR 0.9Final (OCR / Avg.)
GRPO-Guard+0.60+0.460.974 / -0.38
GRPO-Guard + Actuator C+0.73+0.510.965 / +0.02
GRPO-Guard + replayed m1m_1+0.71+0.520.963 / +0.04
Z-Clip + Actuator C (K-Hold)+0.86+0.740.959 / +0.47
Z-Clip + replayed m2m_2+0.81+0.760.956 / +0.50
Z-Clip + exponential fit of m2m_2+0.84+0.740.966 / +0.32

With the same Actuator C, K-Hold achieves higher held-out rewards than GRPO-Guard at matched OCR, indicating that the improvement is not due to the feedback controller alone. Replaying the recorded learning-rate schedules yields comparable downstream scores for both methods. Similarly, the exponential fit of m2m_2 achieves similar scores at matched OCR. These results indicate that much of the downstream gain can be reproduced by the learning-rate schedules. However, as established in Prop. 5, uniform clipping alone cannot enforce a target per-step KL. Actuator C addresses this limitation by adjusting the learning rate based on the measured KL.

Thank you for the additional experiments, which address my main concerns. I remain inclined toward acceptance and will keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.