acceptodds
Under review as a conference paper at ICLR 2027

ReCAST: Reward Credit Assignment across Timesteps for Online Diffusion Reinforcement

Abstract

Training diffusion models with multiple rewards requires distinguishing user preference from reward informativeness. User preference determines how much each reward should contribute to the overall objective; reward informativeness determines when its feedback is useful during denoising. Some rewards can meaningfully evaluate a sample as soon as global structure emerges, but others become informative only when the sample is nearly clean. To address both questions jointly, we propose ReCAST (Reward Credit ASsignment across T}imesteps), the first method, to our knowledge, for per-reward, timestep-dependent credit assignment in diffusion reward fine-tuning. ReCAST separates user preferences from temporal allocation through a reward-by-timestep weight matrix , whose row sums match the user-specified reward budgets , while its column sums are equal, assigning the same total weight to each denoising step. Under these marginal constraints, ReCAST allocates weight according to each reward's informativeness, quantified by its Rényi discriminability gain at each step. These gains telescope to the total discriminability between the reward-induced positive policy and the current policy, providing a basis for temporal credit assignment. We evaluate ReCAST by training SD3.5-Medium under two distinct four-reward settings, each across five reward budgets . ReCAST improves the training rewards in one setting and matches them in the other, improves every held-out judge in both, and is preferred by an independent LLM-as-a-Judge. Together, these results show that ReCAST yields improvements that generalize beyond the training rewards and support its core principle: assigning each reward greater weight at the denoising timesteps where its feedback is most informative.

Primary area:
generative models
Keywords:
diffusion, flow matching, credit assignment, finetuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BAjgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates reward weights across denoising steps in multi-reward fine-tuning of diffusion models. The authors propose ReCAST, which uses a reward-by-timestep weight matrix to adjust each reward’s weight across denoising steps while preserving its total weight budget. The allocation is guided by a proxy for per-step Rényi discriminability gains. A Sinkhorn projection then enforces the specified per-reward budgets and a uniform total weight per timestep. Experiments are conducted on SD3.5-Medium in two multi-reward settings incorporating OCR and GenEval, respectively. The authors report improvements in the OCR setting's aggregate objective and comparable performance in the GenEval setting, along with higher average scores from held-out reward models in both settings.

Critical strengths

Clear Problem Formulation: The paper separates how much each reward matters overall from when it should have greater influence during denoising. This distinction is well motivated, and the weight matrix provides a clear mathematical formulation that preserves per-reward budgets while allowing reward-specific temporal allocation.

Evaluation Beyond Training Rewards: The experiments cover five reward budgets and three training seeds and include multiple held-out reward models. In the GenEval setting, ReCAST achieves higher average scores from all four held-out models despite comparable training-reward scores to the static baseline. This provides evidence that the benefits extend beyond the rewards directly optimized during training.

Critical weaknesses

Gap Between the Theoretical Divergence and the Implemented Surrogate: The temporal allocation is motivated by Rényi divergence gains, but the implemented estimator uses a common external generator in place of the reward-specific positive policies. Although Section 3.3 acknowledges this approximation and Appendix A.3 provides an error bound under a small-KL-mismatch assumption, the paper does not establish that this assumption holds in the experimental setting. Nevertheless, Appendix B.6 interprets the reduction in the surrogate quantity as a reduction in the corresponding Rényi divergence and invokes the Rényi-to-KL inequality to claim that the KL gap is nearly closed. This conclusion is not justified for the surrogate without additional assumptions. Moreover, clipping negative estimated increments breaks the telescoping identity in Eq. (7). While mean-normalization remains well defined when the total clipped gain is positive, its denominator can no longer generally be identified with the data-level divergence. These issues weaken the claimed theoretical interpretation of the implemented allocation.

Insufficient Controls for the Temporal Profiles and Their Computational Cost: The main comparison is against static reward weighting, while the no-Sinkhorn ablation tests the column constraint rather than the importance of the estimated temporal profiles. Comparisons with simple reward-specific schedules and shuffled or reversed profiles would help establish whether the gains depend on the proposed reward-specific temporal structure. This distinction matters because constructing the profiles requires substantial additional computation: Appendix E.2 implies approximately 90 GPU-hours for a 25-step curve, compared with approximately 37 GPU-hours for an OCR training-stage run. Although this one-time cost can be amortized across runs, a compute-matched comparison and an explicit account of the reuse assumptions are needed to assess the practical benefit.

Questions for authors

Interpretation of the surrogate and clipping: What assumptions justify applying the Rényi-to-KL inequality to the external-generator surrogate in Appendix B.6, and what evidence supports these assumptions in the experimental setting? Please also clarify how clipping negative estimated increments affects the telescoping identity in Eq. (7), the interpretation of the normalization denominator in Eq. (13), and the resulting allocation weights.

Contribution of the estimated gain curves: How does ReCAST compare with simple reward-specific temporal schedules or with shuffled/reversed gain curves under the same marginal constraints? Such controls would help determine whether the improvements depend on the estimated reward-specific temporal structure.

Computational cost: When gain-curve construction is included in the total computational budget, how does ReCAST compare with the static baseline given an equivalent budget? Please clarify how many downstream runs must reuse a curve for its construction cost to be amortized.

3 comments

Thank you for your feedback. We address each weakness and question below.

Weakness 1 — Gap between the theoretical divergence and implemented surrogate. Common response 3 defines the reward-moment correction and includes the surrogate correction and endpoint error in the KL bound. Common response 4 separates the signed telescoping sum from the clipped sum and quantifies the effect of clipping on the allocation weights.

Weakness 2 — Temporal-profile controls and computational cost. Common response 1 reports reversed and time-shuffled controls, including their training rewards and held-out evaluations. Common response 2 reports construction and projection costs, amortization across budgets and seeds.

Question 1 — Interpretation of the surrogate and clipping. Common response 3 gives the corrected KL bound and empirical correction estimates. For Eq. (7), the signed increments telescope, while clipping adds the negative mass Mi−M_i^-. The denominator in Eq. (13) is the mean clipped gain, equal to the signed mean plus Mi−/TM_i^-/T. Common response 4 reports that removing clipping changes the applied profiles by mean TV 0.0050.005–0.0090.009, compared with finite-sample variability of 0.0590.059.

Question 2 — Contribution of the estimated gain curves. Common response 1 provides the requested reversed and time-shuffled comparisons. The controls preserve both applied marginals. ReCAST has the highest mean aggregate and individual reward. The same response discusses why a prespecified schedule requires reward-specific timing choices that are not known in advance.

Question 3 — Computational cost and reuse. Common response 2 reports construction costs of approximately 22 GPU-hours for a 1010-step curve and 1212 GPU-hours for a 2525-step curve. Reusing each curve across 55 reward budgets and 33 training seeds gives 1515 downstream runs, amortizing these costs to approximately 0.130.13 and 0.80.8 GPU-hours per run.

Comparison with a larger training budget for static. On the held-out OCR test set at the balanced reward budget λ=(1,1,1,1)\boldsymbol\lambda=(1,1,1,1), we compare static at step 180180 (6060 OCR-stage updates) with ReCAST at step 170170 (5050 OCR-stage updates), giving static ten additional training updates. Across three matched seeds, ReCAST achieves a higher mean aggregate reward of 2.443±0.0482.443_{\pm0.048} versus 2.387±0.0582.387_{\pm0.058} for static (mean ±\pm SD). Under the 1515-run curve-reuse setting above, static's ten extra updates exceed the one-to-two-update equivalent of ReCAST's amortized curve-construction cost. ReCAST therefore retains the higher mean aggregate reward with a lower estimated total GPU compute budget in this comparison.

Thank you for the detailed rebuttal. Since your additional experiments, theoretical corrections, and amortization analysis have addressed my concerns, I will raise my score to 3.

Thank you for your constructive feedback, which helped us strengthen the experiments and clarify the theoretical analysis and computational costs. We are glad that the revisions have addressed your concerns.

Reviewer_fzB7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReCAST, which changes how much weight each reward gets at different diffusion timesteps while keeping the prescribed reward and timestep budgets. It chooses these weights using estimated Renyi-divergence gain profiles from an external generator. The experiments use SD3.5-Medium on OCR and GenEval, with five budgets and three seeds. Gains are clearer on OCR and held-out judges and the GenEval training-reward means change little.

Critical strengths

I like that the formulation lets us specify an overall budget for each reward and then choose where to place that weight in time. The OCR results are encouraging, with evaluation across five budgets and three seeds. The held-out judges also check whether the gains extend beyond the training rewards.

Critical weaknesses
  1. My main question is whether the estimated profiles are responsible for the gains. Beating static weighting shows that changing weights over time can help, but does not show that these particular profiles work better than a simpler schedule. The no-Sinkhorn ablation changes the marginal constraints, so it does not answer this question. A profile shuffle or a smooth schedule with the same marginals would be a useful check.

  2. I think the KL claim in Appendix B.6 goes too far. For current policy pp, bounded reward r≥0r \ge 0 with Z=Ep[r]>0Z=\mathbb{E}_p[r]>0, and proposal qq, the true divergence and the surrogate at clean data are D2(p+∥p)=log⁡Ep[r2]Z2,S(q,p,r)=log⁡Eq[r]ZD_2(p^+\|p)=\log\frac{\mathbb E_p[r^2]}{Z^2}, S(q,p,r)=\log\frac{\mathbb{E}_q[r]}{Z}. Here p+=(r/Z)pp^+=(r/Z)p, and SS has its pure-noise value subtracted. When q=pq=p, S=0S=0, while D2>0D_2>0 if the reward is nonconstant under pp. A small surrogate value alone therefore does not justify the stated KL bound. The approximation error also needs to be controlled.

Questions for authors
  1. Have you compared the estimated profiles with another schedule that satisfies the same marginal constraints?

  2. How is the surrogate's approximation error controlled when drawing the KL conclusion in Appendix B.6?

4 comments

Thank you for your feedback. We address each weakness and question below.

Weakness 1 — Contribution of the estimated profiles. Common response 1 reports reversed and time-shuffled controls. Both controls preserve both applied marginals, directly testing the contribution of reward-specific timing. ReCAST has the highest mean aggregate reward; the tables report the individual rewards and held-out judges.

Weakness 2 — KL interpretation of the surrogate. Your counterexample is valid: when π+=πold\pi^+=\pi^{\mathrm{old}}, the surrogate vanishes even if the true divergence is positive. Taking bi=D2(πi+∣πold)b_i=D_2(\pi_i^+\\|\pi^{\mathrm{old}}) accounts for the remaining divergence. Common response 3 gives the corrected bound and estimates both the surrogate correction and endpoint error.

Question 1 — Comparison with another schedule under the same marginals. Yes. The time-shuffled control in Common response 1 applies the same random column permutation to every reward, preserving both applied marginals. The same response explains why curve estimation is needed to choose a reward-specific schedule.

Question 2 — Control of the surrogate approximation error. Common response 3 defines bib_i and δi\delta_i and reports their empirical estimates at the paper's sampling budget. Including both corrections gives mean checkpoint-to-base estimated bound ratios of 0.100.10–0.220.22 across rewards.

Thanks for the additional experiments and the KL correction. They address some of my concerns. However, I still have concerns about the size of the contribution. Giving each reward its own timing is useful, but Rényi gain does not directly show where more training weight will help most. The new experiments give some support for this choice, but the gains in training rewards are mainly on OCR. Therefore, I maintain my initial score.

Thank you for reviewing the additional experiments and KL correction. We may not have explained the additional results clearly enough, so we would like to clarify further.

We believe the shuffle and reverse controls in Common Response 1 address your concern about where training weight is useful: they change the training weight on timing. Both perform worse than ReCAST, supporting the usefulness of the timing identified by Rényi gain.

Regarding your concerns in the training reward gains, we believe Common Response 1 shows improvements over static weighting or alternative weighting on all four training rewards: CLIPScore, HPSv2, PickScore, and OCR. Beyond the OCR setting, Table 2 reports higher mean scores on all four held-out judges for GenEval. These held-out judges' results provide evidence of improvement beyond the optimized objectives and should also reflect our method's contribution.

We hope this clarifies the scope of the improvements.

AuthorsOpenReview ↗

Intuitive explanations for our method

We realize that our intuitive explanations are scattered throughout the paper and not organized together, which may make the reviewers difficult to fully understand. We would therefore like to explain why we use per-step Rényi gain to guide reward allocation.

For reward ii, the discriminability Di,t=Dα(πi,t+∣πtold)D_{i,t}=D_\alpha\bigl(\pi_{i,t}^+|\pi_t^{\mathrm{old}}\bigr) measures the difference between the current policy and its reward-tilted policy at timestep tt. At the pure-noise end, both distributions are noise, so Di,T=0D_{i,T}=0. Under Theorem 3.1, discriminability is nondecreasing as noise is removed, reaching its full clean-image value at t=0t=0.

Our question is how Di,tD_{i,t} evolves as tt decreases from TT (noise end) to 00 (clean image end). On the noise end, both πi,T+\pi_{i,T}^+ and πTold\pi_T^{\mathrm{old}} are pure noise, so the divergence is 0 and there is no discriminability. On the clean end, π0old\pi_0^{\mathrm{old}} is the current policy that generates the clean image, and πi,0+\pi_{i,0}^+ is the current policy tilted by reward rir_i, the target policy we want to achieve, so the divergence is not zero and is the maximal (proven in Theorem 3.1).

In the denoising from TT to 00, Di,tD_{i,t} will be non-decreasing (Theorem 3.1), but which step will add the most discriminability? In other words, which step tt will have the largest ΔDi,t\Delta D_{i,t}? If that step exists, the ΔDi,t\Delta D_{i,t} curve is not almost flat. Before this step, the current and tilted policies are hard to distinguish, so they generated almost the same partially denoised images. After this step, the discriminability is large, and the current and tilted policies are easy to distinguish, they generate very different partially denoised images. So this is the critical step in the denoising, and we should put more training weights on that step.

We indeed observe different ΔDi,t\Delta D_{i,t} curve shapes for different rewards. OCR and Geneval have a peaky curve. This is within expectation. OCR checks the texts and GenEval uses object-detection to check whether a generated image matches prompts at an instance level. So once a global structure emerges at denoising step tt, these rewards can judge (this is for better understanding, in practice we do not judge partially denoised images, see Algorithm 1). This is the critical step, where the current and tilted policies depart from each other. On the other hands, other training rewards (ClipScore, HPSv2, PickScore) have flat ΔDi,t\Delta D_{i,t} curves. We explain it that they use holistic metrics and do not relate to structural emergence in denoising.

Overall, this is how we use the Renyi divergence in showing where more training weight will help most. We believe the shuffle and reverse experiments in Common Response 1 can further empirically support our computed Rényi gain as a useful timing signal. We will add this explanation in the updated paper to make readers understand our intuitions better.

Reviewer_W2UmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReCAST (Reward Credit ASsignment across Timesteps) optimizes multi-reward fine-tuning for diffusion models by applying feedback precisely when it is most useful during the denoising process.

The Core Problem: Different rewards provide valuable feedback at different stages of image generation (e.g., global structure vs. fine details). Models need a way to balance a user's overall preference for a specific reward against when that reward's feedback is actually informative.

The Mechanism: ReCAST introduces a reward-by-timestep weight matrix. It respects the user's overall reward budgets (the matrix row sums) while dynamically allocating weight to specific timesteps based on each reward's informativeness. This informativeness is quantified using "Rényi discriminability gain."

The Results: When evaluated on SD3.5-Medium, ReCAST matched or improved training rewards, consistently improved unseen (held-out) judges, and was preferred by LLM evaluators.

By separating user preference from temporal credit assignment, ReCAST proves that applying reward weights at their most informative timesteps leads to better generalization and stronger overall model performance.

Critical strengths

Temporal Precision: Instead of applying reward feedback uniformly, it targets the exact denoising timesteps where a specific reward is most informative (e.g., applying structural rewards early and detail-oriented rewards late).

Decouples Preference from Timing: It allows developers to set an overall "budget" for how much a reward matters (user preference) while the algorithm independently calculates when that reward should be applied.

Critical weaknesses

Algorithmic Complexity: To ensure the system respects both user preferences and denoising stability, ReCAST requires building a kernel from the discriminability gains and projecting it onto a constrained feasible set (to satisfy row and column marginals). This adds complexity to the training pipeline.

Total Dependency on Reward Quality: ReCAST optimizes exactly when to apply a reward based on the discriminability signals that the reward model itself provides. If a specific reward model is noisy, poorly calibrated, or yields spurious signals at certain timesteps, ReCAST may incorrectly allocate credit, amplifying the reward model's flaws.

Questions for authors

na

1 comment

Thank you for your feedback. We address your two weaknesses in order.

Weakness 1 — Algorithmic complexity. Common response 2 explains the curve-construction cost, implementation improvements, and reuse across reward budgets and training seeds. The curve is estimated once and reused across downstream runs, amortizing its construction cost. Sinkhorn takes fewer than 100100 iterations on a small matrix (m≤4m\le4, T≤25T\le25) and runs once per reward budget before training (Appendix C.1).

Weakness 2 — Dependency on reward quality. We use standard reward models from image post-training [1–4] to keep our results comparable with prior work. Studies of policy-gradient RL also report effective learning under reward noise [5,6]. Our sensitivity analysis evaluates random noise and multiplicative reward calibration.

We recomputed reward-noise sensitivity on the same 10001000 fixed-prompt subsets as the clipping analysis in Common response 4. Both use N=8,K=16N=8,K=16 and share the finite-sample TV reference 0.059±0.0130.059_{\pm0.013}. Appendix B.9 gives the perturbation and sampling details. Perturbation rows report ranges of mean TV across rewards; the reference row averages rewards within each draw.

PerturbationApplied-profile TV
Gaussian noise, 0.25σ0.25\sigma0.0120.012–0.0290.029
Gaussian noise, 0.5σ0.5\sigma0.0250.025–0.0580.058
Rescaling, r→0.5rr\to0.5r or 2r2r00 (exact)
Finite-sample reference0.059±0.0130.059_{\pm0.013}

Random noise and scaling. Mean profile changes under Gaussian noise up to 0.5σ0.5\sigma are below the shared finite-sample reference; uniform reward rescaling leaves the weights unchanged.

References

[1] Liu, J., Liu, G., Liang, J., Li, Y., Liu, J., Wang, X., ... & Ouyang, W. (2026). Flow-GRPO: Training Flow Matching Models via Online RL. Advances in neural information processing systems, 38, 40783-40818.

[2] Zheng, K., Chen, H., Ye, H., Wang, H., Zhang, Q., Jiang, K., ... & Liu, M. Y. (2026, April). DiffusionNFT: Online Diffusion Reinforcement with Forward Process. In International Conference on Learning Representations (Vol. 2026, pp. 134129-134150).

[3] Xue, Z., Wu, J., Gao, Y., Kong, F., Zhu, L., Chen, M., ... & Luo, P. (2025). DanceGRPO: Unleashing GRPO on Visual Generation. arXiv preprint arXiv:2505.07818.

[4] Fang, Z., Huang, W., Zeng, Y., Zhao, Y., Chen, S., Feng, K., ... & Zhao, F. (2026). Flow-OPD: On-Policy Distillation for Flow Matching Models. arXiv preprint arXiv:2605.08063.

[5] Plesner, A., Guzmán, F., & Athalye, A. (2026). An imperfect verifier is good enough: Learning with noisy rewards. arXiv preprint arXiv:2604.07666.

[6] Rad, A., Filom, K., Keivan, D., Mohajerin Esfahani, P. & Kamalinejad, E. (2026). Rate or Fate? RLV$^\varepsilon$R: Reinforcement Learning with Verifiable Noisy Rewards. Proceedings of the 43rd International Conference on Machine Learning.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Common responses (1/2)

1. Shuffled and reversed gain curves — fzB7 and BAjg

To test the contribution of the estimated timing, we evaluated two temporal controls constructed from ReCAST's weight matrix.

  • Reversed: reverse these columns for every reward, W:,trev=W:,T+1−tW^{\mathrm{rev}}_{:,t}=W_{:,T+1-t}. This preserves the matrix's row and column sums and each reward's set of weights.
  • Time-shuffled: use a random permutation σ\sigma, W:,trand=W:,σ(t)W^{\mathrm{rand}}_{:,t}=W_{:,\sigma(t)}. The random permutations are fixed across runs.

Both controls change when each reward receives weight while preserving both column and row marginals. We evaluate the final checkpoint at step 180180 with λ=(1,1,1,1)\boldsymbol\lambda=(1,1,1,1). Tables report means with standard deviations across runs as subscripts; the best mean in each column is bold. These comparisons support the contribution of the estimated temporal profiles.

Training rewards on the held-out OCR test set (Aggregate is the sum of the four rewards):

MethodAggregateCLIPScoreHPSv2PickScoreOCR
ReCAST2.435±0.058\textbf{2.435}_{\pm 0.058}0.321±0.001\textbf{0.321}_{\pm 0.001}0.306±0.006\textbf{0.306}_{\pm 0.006}0.900±0.004\textbf{0.900}_{\pm 0.004}0.908±0.055\textbf{0.908}_{\pm 0.055}
Static2.387±0.0582.387_{\pm 0.058}0.315±0.0030.315_{\pm 0.003}0.288±0.0070.288_{\pm 0.007}0.890±0.0020.890_{\pm 0.002}0.894±0.0510.894_{\pm 0.051}
Reversed2.382±0.0052.382_{\pm 0.005}0.315±0.0050.315_{\pm 0.005}0.276±0.0250.276_{\pm 0.025}0.888±0.0130.888_{\pm 0.013}0.903±0.0400.903_{\pm 0.040}
Time-shuffled2.386±0.0492.386_{\pm 0.049}0.305±0.0010.305_{\pm 0.001}0.286±0.0180.286_{\pm 0.018}0.890±0.0090.890_{\pm 0.009}0.905±0.023{0.905}_{\pm 0.023}

Held-out judges that are never optimized (10001000 OCR-panel prompts):

MethodAestheticImageRewardHPSv3UnifiedReward-2.0
ReCAST5.574±0.063\textbf{5.574}_{\pm 0.063}1.476±0.027\textbf{1.476}_{\pm 0.027}13.16±0.50\textbf{13.16}_{\pm 0.50}3.154±0.0303.154_{\pm 0.030}
Static5.493±0.0965.493_{\pm 0.096}1.433±0.0561.433_{\pm 0.056}12.86±0.3312.86_{\pm 0.33}3.179±0.071\textbf{3.179}_{\pm 0.071}
Reversed5.377±0.2365.377_{\pm 0.236}1.271±0.3231.271_{\pm 0.323}11.94±2.3911.94_{\pm 2.39}3.086±0.1253.086_{\pm 0.125}
Time-shuffled5.431±0.1065.431_{\pm 0.106}1.339±0.0311.339_{\pm 0.031}12.44±0.3212.44_{\pm 0.32}3.119±0.0293.119_{\pm 0.029}

Curve estimation determines whether a reward needs a flat or concentrated schedule and where its high-gain region lies. These properties vary across rewards and are not known in advance. A prespecified smooth schedule requires these choices in advance. Estimating the curves provides the reward-specific information needed to make them.

2. How much does curve construction cost, and when can it be reused? — W2Um and BAjg

The cost of computing a curve. The original submission reported a cost of 90 GPU-hours for a 25-step curve. We found that this cost was largely due to inefficiencies in our implementation, which we had not optimized much because the curve only needs to be computed once. We have since made two key improvements:

  1. Parallelizing each prompt across all GPUs. Previously, each GPU processed a separate prompt, causing some GPUs to wait for others to finish. We now use all GPUs to process one prompt at a time, reducing idle time.
  2. Increasing the inference batch size. We use larger batches during inference to improve GPU utilization.

After these improvements, construction takes approximately 22 GPU-hours for a 1010-step curve and 1212 GPU-hours for a 2525-step curve, excluding external-generator API time, as reported in Appendix E.2.

Curve reuse. For a fixed estimation setup, each reward's curve is estimated once and reused across reward budgets λ\boldsymbol\lambda, training seeds, and reward combinations. Sharing each curve across the paper's 55 budgets and 33 seeds (1515 runs) amortizes construction to approximately 0.130.13 GPU-hours per run for the 1010-step curve and 0.80.8 GPU-hours per run for the 2525-step curve. The latter corresponds to about one or two OCR-stage training updates.

Sinkhorn cost. The projection takes fewer than 100100 iterations on a small matrix (m≤4m\le4, T≤25T\le25) and is computed once per reward budget before training. Its cost is negligible compared with a single training update (Appendix C.1).

AuthorsOpenReview ↗

Common responses (2/2)

3. What do the corrected KL-bound estimates imply? — fzB7 and BAjg

Appendix B.7 now defines an estimable surrogate correction and reports the corrected KL-bound estimates.

We define the correction bi≥0b_i\geq 0 that measures the external generator's mean-reward shortfall relative to the reward-specific positive policy on a logarithmic scale:

bi:=[log⁡Eπi+[ri]Eπ+[ri]]+,b_i := \left[\log\frac{\mathbb{E}_{\pi_i^+}[r_i]}{\mathbb{E}_{\pi^+}[r_i]}\right]_+,

which can be estimated from the existing policy and external-generator samples. The signed population surrogate total is G~i=log⁡(Eπ+[ri]/Eπold[ri])\widetilde G_i=\log(\mathbb E_{\pi^+}[r_i]/\mathbb E_{\pi^{\mathrm{old}}}[r_i]), giving

\le D_2(\pi_i^+\\|\pi^{\mathrm{old}}) \le \widetilde G_i+b_i.$$ For an estimate covering both the clean and pure-noise endpoints, define $\widehat{\widetilde G}_i =\widehat{\widetilde D}_{i,0}-\widehat{\widetilde D}_{i,T}, \delta_i=[\widetilde G_i-\widehat{\widetilde G}_i]_+$. The signed increments telescope to $\widehat{\widetilde G}_i$. Clipping negative increments can only increase their sum, so $$\mathrm{KL}(\pi_i^+\\|\pi^{\mathrm{old}}) \le \sum_{t=1}^{T}\widehat{\widetilde{\Delta D}_{i,t}} +b_i+\delta_i.$$ Appendix B.7 gives the derivation and estimation details. The table reports $1000$ paired subsets at $N=8,K=16$ from the expanded $N=32,K=64$ set, retaining all $72$ prompts. The estimated bound adds the clipped gain (including the clean endpoint), the weighted correction $\widehat b_i$, and an empirical $95$th-percentile endpoint allowance $\widehat\delta_i^{95}$ relative to the expanded $N=32,K=64$ set. Subscripts give standard deviations across subsets. | Reward | Checkpoint | Clipped gain | $\widehat b_i$ | $\widehat\delta_i^{95}$ | Estimated bound | Checkpoint-to-base bound ratio | |---|---|---|---|---|---|---| | CLIPScore | $0$ | $0.255_{\pm0.002}$ | $0.0023_{\pm0.0004}$ | $0.004$ | $0.261_{\pm0.002}$ | — | | CLIPScore | $180$ | $0.037_{\pm0.002}$ | $0.017_{\pm0.001}$ | $0.002$ | $0.057_{\pm0.002}$ | $0.216_{\pm0.006}$ | | HPSv2 | $0$ | $0.501_{\pm0.002}$ | $0$ | $0.003$ | $0.504_{\pm0.002}$ | — | | HPSv2 | $180$ | $0.024_{\pm0.001}$ | $0.026_{\pm0.001}$ | $0.001$ | $0.051_{\pm0.001}$ | $0.101_{\pm0.002}$ | | OCR | $0$ | $2.271_{\pm0.019}$ | $0.068_{\pm0.089}$ | $0.029$ | $2.368_{\pm0.089}$ | — | | OCR | $180$ | $0.028_{\pm0.008}$ | $0.233_{\pm0.145}$ | $0.014$ | $0.276_{\pm0.143}$ | $0.115_{\pm0.055}$ | | PickScore | $0$ | $0.209_{\pm0.001}$ | $0$ | $0.001$ | $0.210_{\pm0.001}$ | — | | PickScore | $180$ | $0.024_{\pm0.001}$ | $0.0044_{\pm0.0003}$ | $0.001$ | $0.029_{\pm0.001}$ | $0.138_{\pm0.003}$ | **With all three terms included, the mean checkpoint-to-base estimated bound ratio is $0.10$–$0.22$ across rewards**, corresponding to a $78$–$90\\%$ reduction in the estimated upper bound. ### 4. What does clipping change? — fzB7 and BAjg We estimate the curve once at training step $0$ and reuse it throughout training. The following analysis concerns that initial OCR-stage curve and uses the same $1000$ subsets at $N=8,K=16$ described above. Details are in Appendix B. **Telescoping (Eq. 7).** The exact population Rényi gains are nonnegative. Our estimator uses an external-generator surrogate, so negative estimated increments can reflect sampling error. Let $\widehat d_{i,t}$ be a signed estimated increment and $M_i^-=\sum_t[-\widehat d_{i,t}]_+$ the clipped negative mass. Then $$\sum_t[\widehat d_{i,t}]_+=\sum_t\widehat d_{i,t}+M_i^-.$$ The signed sum telescopes to the endpoint difference. Clipping adds $M_i^-$. Appendix B.8 gives the derivation. The following values are means with standard deviations in subscripts: | Reward | Clipped increments $\sum_t\mathbf{1}\\{\widehat d_{i,t} < 0\\}$ (of $24$) | Signed sum $\sum_t\widehat d_{i,t}$ | Negative mass $M_i^-$ | $\operatorname{TV}_i$ | |---|---|---|---|---| | CLIPScore | $5.5_{\pm0.6}$ | $0.246_{\pm0.002}$ | $0.009_{\pm0.001}$ | $0.005_{\pm0.001}$ | | HPSv2 | $0$ | $0.501_{\pm0.002}$ | $0$ | $0.006_{\pm0.003}$ | | PickScore | $0$ | $0.209_{\pm0.001}$ | $0$ | $0.006_{\pm0.003}$ | | OCR | $1.2_{\pm0.8}$ | $2.250_{\pm0.019}$ | $0.021_{\pm0.020}$ | $0.009_{\pm0.006}$ | Clipping increases the signed total by 3.7% for CLIPScore and 0.9% for OCR on average. Removing clipping changes each reward's applied temporal profile by mean TV $0.005$–$0.009$. The finite-sample reference, **$0.059_{\pm0.013}$** averaged over rewards, compares the standard clipped weights from each $N=8,K=16$ subset with those from the expanded $N=32,K=64$ set. This provides a scale for interpreting the clipping effect relative to the variability inherent in estimating the weights from limited samples. Clipping changes the profiles less, on average, than this sampling variability.
AuthorsOpenReview ↗

Summary of changes

We thank the reviewers for appreciating ReCAST's clear problem formulation, its separation of reward budgets from timing, and its encouraging empirical results. Their main questions concern diagnosing the source of these gains, clarifying the surrogate's theoretical interpretation, and assessing robustness and computational cost. Our revisions address these questions through additional controls, corrected KL-bound estimates, sensitivity analyses, and a clearer account of computation and reuse.

Reviewer concernRebuttal responseChanges in the paper
Contribution of the estimated temporal profiles (fzB7, BAjg)Common response 1: reversed and time-shuffled controls, with training-reward and held-out-judge results.Appendix B.6 adds the controls and results; Section 4.5 links to the additional ablations.
Curve-construction cost and reuse (W2Um, BAjg)Common response 2: implementation improvements, updated timings, amortization across budgets and seeds.Appendix E.2 updates construction costs and clarifies reuse assumptions and per-run overhead; Appendix C.1 describes the Sinkhorn cost.
Surrogate approximation and KL interpretation (fzB7, BAjg)Common response 3: corrected KL bound with empirical surrogate and endpoint corrections.Appendix B.7 defines the correction terms and reports estimated bounds.
Clipping and normalization (fzB7, BAjg)Common response 4: effect on telescoping, clipped gain totals, and allocation weights; BAjg Question 1 clarifies normalization.Appendix B.8 explains the mean clipped-gain denominator and quantifies changes in the temporal profiles.
Reward noise and calibration (W2Um)Individual response to W2Um, Weakness 2: noise and rescaling sensitivity relative to sampling variability.Appendix B.9 compares perturbation effects with the finite-sample TV reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.