acceptodds
Under review as a conference paper at ICLR 2027

Beyond the KL Brake: Reward-Level Adaptive Anchoring for Diffusion Policy Optimization

Abstract

Diffusion models are difficult to directly align with specific preferences. While Reinforcement Learning (RL) improves alignment, it suffers from sparse rewards. A common method is to assign the only final reward to all timesteps, but this weakens credit assignment. When rewards are identical across steps, action-level differences are obscured, making it hard to correct erroneous denoising behaviors and increasing the risk of distributional shift. Kullback–Leibler (KL) divergence is widely used to prevent distributional collapse, yet the KL-as-loss paradigm has limitations. First, it only indirectly regularizes parameters and cannot directly suppress harmful actions. Local drift caused by such actions can accumulate through diffusion’s iterative process, a problem observed in UNet-based RL and amplified in Flow RL with SDE noise. Second, under KL-as-loss, the model must continually overcome the optimization resistance introduced by a static regularization term while pursuing higher rewards. As a result, more time is required to reach the same level of performance as training without KL. Third, a fixed KL weight creates a trade-off between preserving the pretrained distribution (exploitation) and efficient alignment (exploration): small weights cause rapid distribution drift, while large weights over-anchor the model to the pretrained distribution, weakening preference optimization. To address the above issues, we propose \ourmethod, which converts KL-as-loss into fine-grained step rewards, making distributional deviation an explicit per-step feedback signal. This suppresses drift accumulation and mitigates reward sparsity. By jointly modeling exploration intensity and reward trends, \ourmethod adaptively adjusts anchoring strength, preserving pretrained advantages while improving alignment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Kullback–Leibler (KL) divergence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X2DkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes StepKL for RL fine-tuning of text-to-image diffusion and flow-matching models. Instead of adding a KL penalty to the loss, it defines a per-denoising-step "anchoring reward" equal to the negative squared (or L1) discrepancy between the reference and trainable models' noise/velocity predictions, and adds it to the (backfilled) task reward. The anchoring reward is gated by an adaptive weight g_k that is nonzero only when the current step's deviation exceeds a trajectory-history threshold (mean + std of past deviations) and the step's quality gain, measured by scoring the predicted clean image x0_hat with a quality model Q, falls below a history threshold (mean - std of past gains). Experiments on SD3.5 and FLUX.1-dev (with MixGRPO/FlowGRPO/DiffusionNFT/FlashGRPO as baselines) and on SD1.4/1.5/2.1 (DDPO/DPOK/D3PO) report higher target rewards at matched sample budgets, better cross-reward transfer, a 12-metric table where StepKL ranks first on 11 metrics, reduced reward hacking (t-SNE, OCR case study, VLM pairwise judge), stability under high SDE noise where KL-as-loss collapses, and lower compute to reach a target GenEval score: 6.11 versus 14.91 GPU-hours for FlowGRPO in Figure 5.

Critical strengths
  1. The idea of turning the reference constraint into a step-level, trajectory-adaptive reward is natural for diffusion RL, where the final-reward backfilling makes credit assignment weak, and the breadth of settings (two parameterizations, five backbones, four prompt sets, several rewards, plug-in to two GRPO variants) is a plus. The high-noise experiment in Figure 9, where KL-as-loss collapses and StepKL keeps training, is the most convincing single result.
Critical weaknesses
  1. Important implementation details remain unclear. Appendix B.5.1 compares GenEval and PickScore for intermediate scoring, but the scorer used in each main experiment, the anchoring coefficient, the gating warm-up length, and the reduction or normalization of prediction discrepancies are not fully specified. The paper should also explain how the shaped rewards enter timestep-wise advantages and whether the anchoring reward and gates are detached during optimization. These details are needed to reproduce the method and understand its comparison with KL-as-loss.

  2. The robustness of the comparisons is not sufficiently documented. The number of independent training runs and variability across training seeds are not clearly reported. The adaptive KL-as-loss and shuffled-gate controls are useful, but a direct quantitative comparison with StepKL under documented tuning budgets and multiple seeds is still needed. Evaluation details are also missing, including the FID reference set, metric sample counts, and VLM judge configuration. These gaps make the broad superiority claims difficult to assess and are my main reason for leaning toward rejection.

Questions for authors
  1. Which intermediate scoring model is used in each main experiment? Which evaluation metrics, if any, are also used by the gate during training?

  2. Please provide the anchoring coefficient, gating warm-up length, discrepancy reduction/normalization, and exact advantage construction. Are the anchoring reward and gates detached during optimization?

  3. Can you extend the adaptive KL-as-loss control in Figure 6 to a direct quantitative comparison with StepKL, using documented coefficient tuning and multiple training seeds?

  4. Please report the training-seed counts and uncertainty for the main results and gate ablations, together with metric sample counts, the FID reference set, and the VLM judge model and evaluation protocol.

  5. Section 3.6 excludes startup and evaluation time. Does the training compute in Figure 5 include all reference-model forward passes, intermediate decoding, and reward-model calls used by the adaptive gate? Please provide the timing breakdown and training/evaluation configurations.

11 comments
AuthorsOpenReview ↗

Our Sincere Thanks to Reviewer X2Dk for Your Time

Dear Reviewer X2Dk,

First, we sincerely appreciate the considerable time and effort you have devoted to reviewing our work and providing detailed feedback. We have made every effort to carefully address all your questions and concerns, providing corresponding clarifications and supporting evidence in our rebuttal. We are pleased to have had the opportunity to resolve the issues you raised. If you find our responses satisfactory and believe that your concerns have been adequately addressed, we would sincerely appreciate your consideration of raising your score. Should you have any further questions or concerns, please do not hesitate to let us know. We will respond promptly with detailed explanations. Thank you again for your time and thoughtful evaluation.

AuthorsOpenReview ↗

### W1. Implementation details and reproducibility

Reviewer comment (excerpt):

Appendix B.5.1 compares GenEval and PickScore for intermediate scoring, but the scorer used in each main experiment, the anchoring coefficient, the gating warm-up length, and the reduction or normalization of prediction discrepancies are not fully specified. The paper should also explain how the shaped rewards enter timestep-wise advantages and whether the anchoring reward and gates are detached during optimization.

We thank the reviewer for identifying these implementation details. We have completed the specification in Table 5, Appendix C, and Appendix D.3. Q1–Q3 explain each item below.

  1. Scorers and hyperparameters. The intermediate scorer used in each experiment is specified in Q1, point 1. The anchoring coefficient and gate warm-up length are given in Q2, point 1; discrepancy reduction and normalization are given in Q2, point 2.
  2. Timestep-wise advantages and gradients. The complete construction of the advantages incorporating the shaped rewards is given in Q2, point 3 and Appendix C. Detaching the anchoring reward and gates is explained in Q2, point 4.
  3. Reproduction and KL-as-loss comparison. The reproduction parameters are given in Q2, point 1. The distinction from KL-as-loss is explained in Q2, point 4 and Q3.

Revision location: Appendix B.1, Table 5 (hyperparameters), PDF p. 16; Appendix C, Algorithm 1, and Listing 1 (implementation), PDF pp. 24–25; Eqs. (14)–(15) (policy objective), PDF p. 25; Appendix D.3 (scoring protocol), PDF p. 30.

AuthorsOpenReview ↗

### W2. Multi-seed robustness and evaluation protocol

Reviewer comment (excerpt):

The adaptive KL-as-loss and shuffled-gate controls are useful, but a direct quantitative comparison with StepKL under documented tuning budgets and multiple seeds is still needed. Evaluation details are also missing, including the FID reference set, metric sample counts, and VLM judge configuration.

We thank the reviewer for this concern. We provide multi-seed results, coefficient-tuning details, and the evaluation protocol, with the correspondence below.

  1. Seed counts and variability. Table 10 reports main-result statistics over five seeds. Table 3 and Figure 25 report the controls over two training seeds; see Q4, point 1.
  2. Direct comparisons and tuning. The direct comparison between StepKL and KL-as-loss, together with coefficient tuning, is given in Q3. Seed counts for the gate ablations are reported in Q4, point 1.
  3. Evaluation details. Metric sample counts, the FID reference set, and the VLM judge configuration are given in Q4, points 2(1), 2(2), and 2(3), respectively.

Revision location: Section 3.7, Table 3 (two-seed comparison), PDF pp. 9–10; Appendix D.1, Tables 8–10 (tuning and seeds), PDF pp. 26–28; Figure 25 (training controls), PDF p. 27; Appendix D.3 (evaluation protocol), PDF p. 30; Table 13 (sample counts), PDF p. 31.

Reviewer_oD5MOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces StepKL, a method for reinforcement learning fine-tuning of diffusion models that replaces conventional KL-as-loss regularization with fine-grained, step-wise anchoring rewards. These rewards penalize deviations from a pretrained reference model and are adaptively weighted based on prediction deviations and intermediate image-quality improvements.

The method is evaluated across several diffusion and flow-matching models, reward functions, and benchmarks. The experiments demonstrate improvements in reward optimization, sample efficiency, and robustness to reward hacking compared to existing RL approaches.

Critical strengths

The paper provides extensive experiments across multiple diffusion architectures, benchmarks, and reward functions. The results consistently demonstrate improvements over existing baselines, providing convincing empirical evidence of the method's effectiveness.

The cross-reward evaluation in Figure 3 and multi-metric evaluation in Table 1 suggest that the improvements are not limited to optimizing the target reward. This is particularly important given the risk of reward hacking in diffusion RL.

Critical weaknesses

Motivation for the adaptive mechanism. While the Anchoring Reward introduced in Section 2.2 is straightforward and intuitive, the additional mechanisms in Section 2.3 involve several heuristic design choices, including trajectory-level statistical thresholds, sigmoid-based gating, and their multiplicative combination. Although the paper provides ablation studies demonstrating their effectiveness, further justification for these particular choices would help clarify the underlying principles and generalizability of the proposed approach.

Questions for authors

Could the authors provide further justification for the particular design choices in Section 2.3? In particular, why were the proposed statistical thresholds and multiplicative gating selected? How sensitive is performance to these choices?

Could the authors clarify why incorporating the reference constraint into the reward provides fundamentally different learning signals from step-wise KL regularization in the loss? Given that adaptive KL-as-loss also improves performance in Figure 6, which advantages are specifically attributable to the reward-level formulation?

7 comments
AuthorsOpenReview ↗

Our Sincere Appreciation to Reviewer oD5M for Your Support

Dear Reviewer oD5M,

First, we sincerely thank you for the time and effort you have devoted to reviewing our paper and for your support and positive evaluation of our work. We greatly appreciate your recognition of our contributions and the extensive experimental results. We have made every effort to carefully address all your questions and concerns in our rebuttal. If our responses have adequately resolved your concerns, we would be truly grateful if you could consider raising your score. Should you have any further questions, please do not hesitate to let us know. We will respond promptly and provide any additional clarification needed. Thank you again for your valuable support and consideration.

AuthorsOpenReview ↗

### W1. Motivation for the adaptive mechanism

Reviewer comment (excerpt):

Although the paper provides ablation studies demonstrating their effectiveness, further justification for these particular choices would help clarify the underlying principles and generalizability of the proposed approach.

We thank the reviewer for this comment. The three designs respectively address scale adaptation, smooth control, and joint condition checking:

  • Trajectory-level statistical thresholds: During training, the scales of the deviation and the quality gain drift substantially, and different rewards have very different value ranges, so a fixed threshold does not transfer across settings. We therefore use the historical mean and standard deviation of each trajectory as the reference, and the threshold does not need to be set separately for each reward or backbone.

  • Sigmoid gating: The gate maps the degree to which the threshold is exceeded to [0,1)[0,1). It is 0 when the threshold is not exceeded, and the constraint becomes stronger the further the threshold is exceeded, eventually saturating. This avoids the abrupt change that a hard threshold would cause at the boundary.

  • Multiplicative combination: The product acts as a logical AND. The constraint is strengthened only when the deviation is abnormally large and the quality gain is abnormally small, so beneficial exploration and normal denoising are not penalized.

We have added the scale-drift statistics and gate-component comparisons to Appendix D.2. The SD3.5 fixed-timestep control is in Section 3.7 and Figure 11, with the FLUX extension in Figure 26. Our response to Q1 explains how these experiments support the design choices.

Revision location: Section 2.3 (adaptive gating), PDF pp. 4–5; Section 3.7 and Figure 11 (SD3.5 control), PDF pp. 9–10; Appendix D.2, Tables 11–12 (scales and components), PDF pp. 28–30.

AuthorsOpenReview ↗

### Q1_part1. Statistical thresholds, multiplicative gating, and sensitivity

Reviewer comment (excerpt):

In particular, why were the proposed statistical thresholds and multiplicative gating selected? How sensitive is performance to these choices?

We thank the reviewer for this question. It gives us the opportunity to explain the reasoning behind the design choices in Section 2.3 in more detail.

(1) Why adaptive gating is needed

Applying the same penalty to all deviations also suppresses exploration that genuinely improves generation quality. The gate strengthens the constraint only when the deviation is abnormally large but the quality improvement is insufficient. Empirically, adding the gate improves held-out GenEval from 0.694 (V1, without the gate) to 0.737 (Ours) (see Table 2 in the paper).

(2) Why statistical thresholds

(1) Signal scales drift during training. The scales of deviation and quality gain keep drifting, so fixed thresholds break down. The table below shows how these quantities change during SD3.5 training on GenEval:

Table 11: Scale drift of deviation and quality-gain signals during training.

QuantityEarly training mean (epochs 1–10)Late training mean (last 10 epochs)
Deviation dd0.00520.0838
Quality gain ∣Δq∣\lvert\Delta q\rvert0.01280.0451
Deviation history std sds^d0.00190.3689
Quality-gain history std sqs^q0.1130.206

Score scales also differ greatly across reward models: GenEval lies in 0–1, PickScore is around 20–23, and HPSv2 is around 0.28. Fixed thresholds would have to be tuned separately for each reward and backbone.

(2) Trajectories differ within the same training stage. Deviation scales also vary greatly across trajectories. The coefficient of variation (std / mean) of dd across samples is as high as 1.84, so a single global threshold can hardly fit all trajectories at once. We therefore use each trajectory’s own history as the reference. To test this, we added two controls, each removing a different part of the gate:

  • Shuffled correspondence (V2): We keep the distribution of gate values unchanged and only shuffle the correspondence between gates and trajectories. Held-out GenEval drops from 0.737 to 0.543, which is even below V1 without the gate (0.694) (Table 12).

  • Fixed timestep gating: We estimate the average gate gˉk\bar g_k at each denoising step from 2,880 independent trajectories, sampled using checkpoints from different epochs of previous training runs. During training, the control uses −gˉkdk-\bar g_k d_k instead of the trajectory-dependent −gi,kdi,k-g_{i,k}d_{i,k}, with the same global weight λ\lambda, sampling configuration, and other hyperparameters. This variant preserves the average timestep profile while removing the differences between trajectories within the same step. Under a shared budget of approximately 25.6k training samples, Figures 11 and 26 show higher training rewards for StepKL on both SD3.5 and FLUX with GenEval and PickScore. The comparison therefore supports the importance of the current trajectory’s deviation–quality correspondence beyond timestep weighting.

(3) Dimensionless self-calibration. Thresholds based on the historical mean and standard deviation, combined with standardization of the excess above the threshold, provide a dimensionless, self-calibrating criterion. The thresholds use only the history before the current step. Together with the normalization by sks_k in Eq. (8), the gate input becomes the deviation measured in units of the historical standard deviation. This quantity can be compared directly across timesteps, reward models, and training stages. Apart from Kmin⁡K_{\min}, the design introduces no additional threshold hyperparameters.

Reviewer_ypm5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes StepKL for reinforcement learning fine-tuning of diffusion models. It constructs a per-step anchoring penalty from the prediction discrepancy between the trainable model and a frozen reference model. A trajectory-dependent gate weights this penalty using historical prediction deviations and quality increments measured on intermediate clean-image predictions, aiming to suppress harmful drift while retaining productive exploration.

The evaluation covers U-Net diffusion models and flow-matching models, multiple prompt sets, and different reward objectives. The paper reports improvements in target rewards, cross-reward performance, image quality, diversity, and training compute required to reach selected reward thresholds. Supporting experiments include reward-level versus loss-level anchoring, different exploration-noise levels, gating ablations, and qualitative reward-hacking comparisons.

Critical strengths
  1. The paper provides useful empirical evidence about the interaction between reference anchoring and diffusion RL optimization. Figure 4 compares reward-level anchoring with loss-level regularization across MixGRPO and FlowGRPO, while Figure 9 examines different exploration-noise levels. Table 2 and Figure 6 investigate the anchoring reward and adaptive gate, and Figure 13 examines the two gating components. Together, these experiments provide meaningful support for the practical usefulness of the proposed design.
  2. The evaluation extends beyond the optimized reward. Figure 3 examines target- and cross-reward performance, Table 1 reports quality and diversity metrics, and Figure 8 presents matched-reward OCR examples illustrating differences in scene preservation. Appendix B.2 extends the evaluation to U-Net backbones. The paper also explicitly discusses additional reference-model and intermediate-scoring costs in Appendix B.3 and reports computational profiling in Figure 17.
Critical weaknesses
  1. The central reward-level update is insufficiently specified, leaving an important gap between the reward definition and the credit-assignment claim. Sections 2.2–2.3 define the prediction discrepancy, gate, and combined reward, but do not specify how the step-dependent penalties enter the advantage estimator and final policy objective. Appendix B.1 provides training hyperparameters, but does not resolve temporal accumulation, gradient treatment of the discrepancy and gate, or normalization of step-dependent rewards. These choices materially affect the update. In Eqs. (4)–(5), the discrepancy compares model predictions at the current state rather than directly comparing sampled actions from that state; the complete update is therefore necessary to understand how the signal provides the claimed action-level credit assignment. There is also a concrete ambiguity in the discrepancy definition: Eq. (5) uses an L1 velocity difference, whereas Appendix B.5.2 and Figure 19 describe L2-distance and cosine-similarity variants. These issues impede reproduction and assessment of the method’s distinction from direct prediction regularization.
  2. The adaptive-gating evidence supports effectiveness more clearly than the proposed interpretation of productive versus harmful exploration. Equations (6)–(7) construct thresholds from deviations and quality increments at preceding denoising steps. These quantities may vary systematically with noise level, and successive quality increments can reflect ordinary denoising progress as well as the consequences of policy deviation. The shuffled-gate experiment, component ablations, and adaptive loss-level KL comparison are useful controls, but do not directly separate trajectory-specific adaptation from timestep-dependent allocation of regularization. A fixed timestep schedule matched to the average gate strength would help distinguish these explanations. Appendix B.5.1 additionally shows that the intermediate scoring model can substantially affect performance, reinforcing the importance of establishing what the quality-gain signal measures. This limits the support for a central mechanistic claim, although it does not invalidate the reported empirical improvements.

Overall, I lean toward weak rejection. The empirical breadth and practical improvements are encouraging, but the incomplete specification of the core update and the unresolved interpretation of the adaptive mechanism prevent me from recommending acceptance at present. Clear algorithmic clarification and targeted supporting evidence could change my assessment.

Questions for authors
  1. Please provide the complete policy objective and advantage estimator incorporating Eq. (10). Are step penalties accumulated through return-to-go or used only at the corresponding timestep? Which quantities are detached, and how are the step-dependent rewards normalized?
  2. Which velocity discrepancy is used in the main experiments: the L1 difference in Eq. (5) or the L2 distance discussed in Appendix B.5.2? In reward-level versus loss-level comparisons, are the discrepancy measure, timestep/noise scaling, and adaptive weights otherwise identical?
  3. Does trajectory-specific gating outperform a fixed timestep schedule with comparable average regularization strength, estimated on independent trajectories? This would directly test whether the observed benefit requires trajectory-specific deviation–quality correspondence rather than temporal weighting.
  4. Appendix B.5.1 shows that changing the intermediate scoring model affects performance substantially. Which scoring model is used for each principal experiment, and is there evidence that its intermediate quality increments predict final generation benefits beyond ordinary denoising progress? This would help substantiate the productive-versus-harmful exploration interpretation.
13 comments
AuthorsOpenReview ↗

Sincere Appreciation to Reviewer ypm5

Dear Reviewer ypm5,

First, we sincerely appreciate the time, effort, and thoughtful consideration you have devoted to reviewing our paper. We have made every effort to carefully address all the questions and concerns you raised in our rebuttal.

We are particularly encouraged by your willingness to reconsider your assessment if our clarifications and additional evidence address your concerns. If you find that our responses have adequately resolved the remaining issues, we would be truly grateful if you could consider raising your score.

Should you have any further questions or concerns, please do not hesitate to let us know. We will be happy to address them promptly and provide any additional clarification needed.

Thank you again for your valuable time and constructive feedback.

AuthorsOpenReview ↗

### W1. Complete policy update and stepwise credit assignment

Reviewer comment (excerpt):

The central reward-level update is insufficiently specified, leaving an important gap between the reward definition and the credit-assignment claim.

Thank you for the detailed comments. The stepwise anchoring reward acts on the policy gradient of the corresponding action, and all reward-side quantities are detached. We have completed the implementation specification in Appendix C and added the targeted controls in Section 3.7 and Appendix D. We address each point below.

  1. Complete update and reproduction details. Algorithm 1 and Listing 1 in Appendix C provide the pseudocode; we have supplemented the latent-dimension reduction, detached rollout quantities, local update, and clipped objective in Eqs. (14)–(15). Q1 (1) provides the advantage estimator and policy objective; Q1 (2)–(4) specify temporal accumulation, gradient treatment, and per-step normalization. Q2 (1) clarifies the discrepancy definition. Q1 (3) and Q2 (2) explain the distinction from direct prediction regularization.

  2. Connection between prediction discrepancy and action distributions. The model prediction at the current state determines the action distribution. The step-kk policy is a Gaussian transition πθ(atk∣stk)\pi_\theta(a_{t_k}\mid s_{t_k}) whose mean is determined by the velocity prediction. The probability of any action therefore depends on that prediction. The prediction discrepancy measures the deviation between the training and reference policies' action distributions at this step:

d_k=\frac\{1\}\{D\}\left\lVert v^\{\mathrm\{ref\}\}_\{t_k\}-v^\{\mathrm\{pre\}\}_\{t_k\} \right\rVert_2^2.$$ As detailed in Q2 (2), although Eqs. (4)–(5) do not directly compare sampled actions, they measure the single-step decision that produces the action. 3. **Local rewards provide stepwise credit assignment.** $R^k_{\mathrm{an}}$ is not accumulated along the return. It enters only the step-$k$ advantage $\hat A_{i,k}$ and acts only on the probability ratio $\rho_{i,k}$ of that step's action (Q1 (1)–(2)). Thus, all steps no longer share a single signal as in Eq. (3). 4. **Quality increments distinguish actions from the same state.** The gate uses the local quality increment, expressed as the quality score after the action minus that before the action, $\Delta q=q_{\mathrm{after}}-q_{\mathrm{before}}$. Consequently, $g_k$ varies with the sampled action: from the same state and with the same deviation, an action yielding less quality improvement receives a larger penalty. With the stepwise KL reward, each intermediate denoising step receives **distinct action-dependent feedback**. Compared with the mainstream practice of backfilling the final reward equally to all denoising steps, which assigns a single reward to every intermediate step, this mechanism clearly improves action–reward attribution. **Revision location:** Appendix C, Algorithm 1, and Listing 1 (implementation), PDF pp. 24–25; Eqs. (14)–(15) (policy objective), PDF p. 25; Eq. (5) (corrected discrepancy), PDF p. 4.
AuthorsOpenReview ↗

### W2. Trajectory adaptation and quality-gain signals

Reviewer comment (excerpt):

The adaptive-gating evidence supports effectiveness more clearly than the proposed interpretation of productive versus harmful exploration.

The fixed-timestep control and same-state branching experiment test the explanations based on temporal weighting and action-level gains, respectively. We address each concern below.

  1. Evidence for productive versus harmful exploration. The same-state branching experiment directly tests this interpretation (Q4 (2)): from the same state, actions with larger quality increments Δqk\Delta q_k also lead to better final images. The low-gain gate BkB_k lets these deviations pass and strengthens the constraint on deviations with insufficient gains. This directly supports the gate's distinction between productive and harmful exploration.

  2. Trajectory adaptation versus noise level. The gate is not determined by noise level alone. The fixed-timestep control retains the component that varies with timestep and removes only within-step differences across trajectories, yet performs worse than StepKL (Q3 (1)). At the same denoising steps, gate trends differ across rewards, and gates also vary noticeably across trajectories within a timestep (Q3 (2)).

  3. Controlling for ordinary denoising progress and deviation magnitude. The same-state branching experiment rules out both explanations (Q4 (2)). Branches within a group share the same state, denoising progress, and deviation dkd_k, so differences in Δqk\Delta q_k arise only from the action at that step. Its predictive power therefore comes neither from ordinary denoising progress nor from policy deviation.

  4. Fixed schedules with matched average strength. Q3 provides the fixed-timestep control estimated on independent trajectories, together with per-step gate decomposition, directly separating trajectory-specific adaptation from timestep-dependent regularization allocation.

  5. Intermediate scorers and signal interpretation. Q4 (1) specifies the scorer used in each principal experiment, and Q4 (2) reports evidence that intermediate quality increments predict final outcomes.

Revision location: Section 3.7, Figure 11 (fixed-timestep control), and Table 4 (same-state branching), PDF pp. 9–10; Appendix D.2, Figures 26–27 (extended gate controls), PDF p. 29.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.