acceptodds
Under review as a conference paper at ICLR 2027

Chance-Constrained Flow Optimization

Abstract

Adapting pre-trained diffusion and flow models to optimize downstream utilities is central to their real-world deployment. Existing steering methods typically optimize average rewards or impose constraints on average properties. However, downstream requirements often concern the probability of generating samples that meet particular criteria, which cannot be represented by average rewards or constraints. To address this gap, we propose Chance-Constrained Flow Optimization (CCFO), a distributional steering framework that maximizes expected reward while explicitly controlling the probability of constraint satisfaction. For such a constrained optimization problem, we theoretically derive a closed-form expression for the optimal dual variable, which allows to directly characterize the optimal distribution. This characterization decomposes CCFO into a scalable two-stage procedure: a lightweight dual variable estimation and a single entropy-regularized fine-tuning step via a specific pseudo-reward. By decoupling dual estimation from model fine-tuning, CCFO achieves a computational cost comparable to that of standard expected-reward fine-tuning methods. Experiments on 2D illustrative examples, high-dimensional text-to-image generation, and protein sequence design establish that CCFO achieves better reward–feasibility trade-offs than state-of-the-art baseline methods.

Primary area:
optimization
Keywords:
Diffusion and flow models, fine-tuning, chance constraints optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rv2fOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method(CCFO) for fine-tuning diffusion and flow models to improve rewards satisfying a given constraint. The main contribution is that they simplifies the constrained optimization problem into estimating a single parameter from theoretical perspective and do experiments on 2D examples, image generation, and protein design to show better reward–feasibility trade-offs than existing methods.

Critical strengths

The paper addresses a practical problem, as controlling the probability to satisfy a constraint is often more useful than simply optimizing average performance. The proposed method is relatively simple and can work with existing fine-tuning approaches. The experiments also cover different applications and generally support the effectiveness of the method.

Critical weaknesses

I'm a little bit doubtful about the gap between the theoretical guarantees and practical results. The theory do proves constraint satisfaction for the ideal optimized distribution, but the actual fine-tuned models do not always meet the desired probability. So I'm pretty not sure how reliably the proposed method can enforce constraints in practice since some of the experiments results also don't fully meet the expectaion.

Questions for authors
  1. How reliably does CCFO achieve the target feasibility across different constraint levels and random seeds? Could the authors provide a more detailed analysis of cases where the target is not met?
  2. In the protein experiments, how much does performance change without the additional multiplier updates and reward smoothing? Can the original two-stage method achieve similar results on its own?
3 comments

We thank the reviewer for the positive feedback on the studied problem and experiments supporting the effectiveness. In the following, we address several points mentioned within the review.

Weakness and Q1: Reliability of feasibility control

Thanks for raising this point. We agree that there might exist a gap between theoretical guarantee and practical result on the feasibility. To illustrate the reliability of our method, we have tested our methods across different target feasibility levels, in Tables 5 and 6 in Appendix of the paper, for text-to-image and protein generation, respectively. These results show that our method achieves the desired target feasibility at most of the settings. To fully address your concern, we analyze the failure cases, refine the algorithm, and added new experimental results over more seeds.

In text-to-image generation, the additional results are presented below. We observe that CCFO with fixed multiplier meets the target feasibility in all three seeds at the tested targets from 70% to 93%. At the target of 58%, however, it does not consistently meet the requirement. In this setting, we observe that the estimated multiplier is close to zero, providing little additional feasibility incentive. To improve feasibility tracking, we introduce periodic multiplier updates, as used in the protein experiments. With this refinement, the 58% target is met in all three seeds.

Table 3.1 Additional results of CCFO in text-to-image generation.

Target feasibilityMean OCR ↑Achieved feasibility (%) ↑ImageReward ↑Successful seeds
93%0.34997.360.2053/3
85%0.62793.390.1893/3
82%0.77591.520.4213/3
75%0.82988.150.2473/3
70%0.90282.210.0753/3
58% (old)0.91155.79-0.1451/3
58% (with feedback)0.91062.380.0263/3

In protein generation, fine-tuning with fixed multiplier cannot reliably achieve the target feasibility, because the fine-tuning solver cannot realize the ideal distribution under limited sampling and training budgets. We therefore use multiplier feedback and constraint-indicator smoothing to improve feasibility tracking. Moreover, at the stricter target of 0.9, we observe that feasibility oscillates around the target, so the final checkpoint may fall slightly below the requirement. To accommodate these fluctuations, we introduce a small feasibility margin by setting the internal training target 0.92, slightly above the required level. With these practical refinements, empirical feasibility meets the required target in all five seeds at each tested level. These results together illustrate the relibility of our method.

Table 3.2 Additional results of CCFO in protein generation.

Target feasibilityMean fitness (all) ↑Achieved feasibility (%) ↑Distinct novel and hit designs ↑Successful seeds
90%0.29491.2690.25/5
80%0.39582.27145.25/5
70%0.40771.83133.65/5
60%0.41266.75124.95/5

Q2: CCFO without feedback and reward smoothing.

Thank you for raising this point. As discussed above, multiplier updates and constraint smoothing are practical refinements intended to improve feasibility tracking when fine-tuning only approximately realizes the ideal target distribution. To investigate their effects, we have added controlled ablations at target feasibility q=0.8q=0.8, as reported below. All three variants start from the closed-form multiplier estimate. Holding this estimate fixed and retaining the hard indicator yields a mean fitness of 0.410 and feasibility of 63.46%. Adding multiplier feedback increases feasibility to 75.01%, and subsequently introducing constraint smoothing raises it to 82.31%, achieving the target feasibility. These improvements in feasibility are accompanied by a decrease in mean fitness. The feedback-only variant yields the largest mean number of distinct feasible hits, while the smoothed variant achieves the desired feasibility and greater diversity among feasible samples than the feedback-only variant.

Importantly, comparison with the CFO results in Table 6 of the paper shows that CCFO still achieves higher mean fitness at comparable realized feasibility, regardless of whether multiplier feedback or indicator smoothing is used. This again illustrates that CCFO achieves better reward-feasibility trade-offs than CFO.

Table 3.3 Ablation of feedback and reward smoothing for TrpB generation at target feasibility q=0.8q=0.8.

MethodMean fitness (all)Feasibility (%)Diversity (given feasible)Distinct designs (feasible & hit)
No feedback, hard indicator0.41063.461.720119.0
Feedback, hard indicator0.40175.011.666178.0
Feedback, smooth indicator0.38982.311.708146.7

Dear Reviewer,

Thank you again for your constructive feedback. As the discussion period is coming to an end, we wanted to kindly follow up on our earlier response and would appreciate hearing whether our revisions have addressed your concerns. If so, we would be grateful if you could consider updating your assessment. Thank you for taking the time to consider our follow-up.

Thank you for all the additional experiments. I appreciate your efforts. While these additions provide useful insights, I still have concerns over the gap between theoretical formulation and its practical feasibility, especially the reliance on additional multiplier updates and constraint smoothing to achieve the target feasibility. Therefore, I acknowledge the improvements, but my remaining concerns lead me to maintain my original assessment.

Reviewer_1P7EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Chance-Constrained Flow Optimization (CCFO), a fine-tuning algorithm for diffusion/ flow models which does reward maximization while ensuring that a constraint is satisfied with a pre-defined probability. The algorithm proposed is a two-stage process: dual variable estimation and entropy regularized fine-tuning. Theoretically, the optimal fine-tuned distribution and the optimal dual multiplier are characterized in closed form. Further, a theoretical sensitivity analysis is also given, for when the dual multiplier and constraint evaluation are only approximate. Main experimental results are on text-to-image generation and protein design.

Critical strengths

(1) The paper proposes a principled algorithm for reward maximization subject to constraint satisfaction. The developed theoretical framework, despite using standard machinery, is clean and insightful. The reduction to the two-stage setup, with stage 1 estimating the dual multiplier and stage 2 performing reward maximization (with adjusted rewards) seems practical.

(2) The experimental results show that the algorithm indeed reaches high rewards, while satisfying the constraint with high probability. By adjusting the target probability β\beta, a fine-grained control can be achieved between constraint satisfaction and reward maximization.

Critical weaknesses

My primary concerns are with respect to the experiments:

  • In text-to-image generation, CCFO and CFO are within error bars in Table 1. The CFO baseline also seems more coarsely tuned. Its weight λ\lambda is swept over five values (Table 5), but with no runs between 89.26% and 99.71% feasibility. I am wondering if a finer sweep of λ\lambda might close the gap.

  • From my understanding, in protein design, CCFO does not use the closed-form multiplier alone. The multiplier is updated periodically based on the current model’s feasibility, whereas the CFO weights are kept fixed throughout training. Could the authors report CCFO without the feedback, or CFO with a comparable feedback mechanism?

Questions for authors

Any clarifications/ experiments addressing the weaknesses would be helpful.

1 comment

We sincerely thank the reviewer for the positive feedback on the theoretical framework and experimental results. In the following, we address several points mentioned within the review.

W1: A finer sweep for CFO.

Thanks for raising this point. We have performed a thorough sweep for CFO. The results are shown below. As one can expect, increasing λ\lambda would lead to larger feasibility, with the cost of reward performance. Importantly, the trade-off curves achieved by CFO is still worse than that of our method. We will include these additional results in the revised manuscript.

Table 2.1 Additional results of CFO on text-to-image generation.

MethodMean OCR ↑ (all)Feasibility (%) ↑Mean OCR ↑ImageReward ↑HPS ↑CLIPScore ↑
CFO (λ=0.20\lambda=0.20)0.781789.260.78970.41780.24170.2846
CFO (λ=0.24\lambda=0.24)0.606393.500.60700.20650.24020.2673
CFO (λ=0.28\lambda=0.28)0.340897.460.34180.11430.23890.2641
CFO (λ=0.32\lambda=0.32)0.257499.210.2587-0.12560.22360.2235
CFO (λ=0.35\lambda=0.35)0.244699.710.2445-0.30880.21850.2214

W2: CCFO without feedback.

Thank you for raising this point. We have conducted additional experiments for CCFO without the feedback in the protein generation task. The results are presented below. We observe that CCFO with a fixed multiplier cannot achieve the feasibility requirement for both targets 0.8 and 0.9. This is because the fine-tuning solver cannot realize the ideal target distribution under finite sampling and training budgets. By adding the multiplier update, in both settings CCFO achieves target feasibility, accompanied by a decrease in mean fitness. Therefore, the multiplier update is a useful practical refinement used to improve feasibility tracking when fine-tuning only approximately realizes the ideal target distribution. Moreover, we would like to note that, at comparable realized feasibility, CCFO with fixed multiplier still achieves higher mean fitness than CFO as in Table 6 of the paper. This again illustrates that CCFO achieves better reward-feasibility trade-offs than CFO.

Table 2.2 CCFO with and without multiplier feedback on protein design.

MethodMean fitness (all)Feasibility (%)Diversity (given feasible)Distinct designs (feasible & hit)
No feedback (q=0.8q=0.8)0.41063.461.720119.0
No feedback (q=0.9q=0.9)0.37972.391.729125.4
Feedback (q=0.8q=0.8)0.38982.311.708146.7
Feedback (q=0.9q=0.9)0.29990.011.71790.8
Reviewer_UyqfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to fine-tune a pretrained generative model to improve a reward while requiring a specified fraction of generated samples to satisfy a constraint. Chance-Constrained Flow Optimization (CCFO) combines expected-reward maximization with KL regularization and a lower bound on feasibility probability.

For a single binary feasibility indicator, the authors derive a closed-form optimal multiplier. This suggests a two-stage procedure, ie estimate the multiplier from pretrained samples, then fine-tune with the original reward plus a feasibility bonus. The paper characterizes the optimal distribution and its sensitivity to multiplier and constraint-evaluation errors. Experiments cover 2D examples, text-to-image generation balancing OCR accuracy and aesthetics, and protein sequence generation balancing predicted fitness and mutation distance.

Critical strengths

Strengths

  1. The method has a simple, useful interpretation. Proposition 1 shows that the feasibility bonus changes the probability assigned to feasible and infeasible samples while preserving the reward-tilted conditional distributions within those regions. This makes the effect of the constraint easy to understand. The sensitivity results provide further insight into estimation errors, and the 2D examples clearly illustrate why controlling an average property need not control the frequency of violations.
  2. The application results are encouraging. Table 5 reports an OCR score of 0.8984 for CCFO at 83.79% feasibility, compared with 0.8479 for CFO at 83.30% feasibility. For proteins, Table 2 reports 142±16.3142 \pm 16.3 distinct feasible designs with predicted fitness above wild type, compared with 94±10.294 \pm 10.2 for the strongest listed baseline, with means and standard deviations over three seeds. These results suggest useful improvements across different settings, although further comparisons are needed to establish what drives those gains.
Critical weaknesses
  1. I had limited time to work through all the mathematical details, so apologies if I have misunderstood part of the setup. When reading the multiplier derivation alongside the text-to-image experiment, I was unsure how the conditioning on prompts is handled. My understanding is that, when the prompt distribution is fixed, the image distribution should normalize separately for each prompt. I would therefore expect the overall feasibility to be obtained by first normalizing within each prompt and then averaging over prompts. I could not quite see how this is accounted for when estimating the multiplier from pooled values of AA and BB. The distinction seems important because the pooled calculation can increase overall feasibility partly by assigning more probability to prompts for which feasibility is easier to achieve. In the image experiment, however, the prompt frequencies seem to remain fixed. Could the authors clarify how these two formulations connect, and whether the implementation accounts for the separate normalizers? I may be missing a step here. If this does require a different calibration, it seems that a single scalar solve using prompt-specific quantities could preserve much of the method’s computational appeal. Understanding this would help me assess how directly the theoretical result supports the conditional-generation experiment.

  2. When thinking about the connection to CGM-reward, I wondered whether the main distributional result could be obtained by applying its binary calibration formula to the reward-tilted reference μ∝ppreer/α\mu\propto p_{\mathrm{pre}}e^{r/\alpha}, with clipping to account for the inequality constraint. Smith et al. already separate multiplier estimation from fine-tuning and derive the binary log-odds formula. I would therefore find it helpful to understand more precisely what CCFO adds to that construction. The sensitivity analysis and applications may be useful contributions in their own right. Since a reward-augmented CGM-reward baseline is mentioned, seeing its results under matched settings would also help clarify this comparison. For the protein experiment, I was similarly unsure how much of the improvement comes from the initial closed-form estimate and how much comes from subsequently updating the multiplier and smoothing the constraint. These seem like reasonable practical choices, but they make the role of the initialization harder to assess. How well does the method work with the estimated multiplier held fixed? Conversely, how well does the same feedback procedure work from a simple initialization? A comparison accounting for the additional feedback samples would help establish the practical benefit of the analytic estimate.

Questions for authors
  • Is the prompt distribution held fixed during calibration? If so, how does the implementation account for separate normalization for each prompt? Please specify the calibration value of α\alpha and its relationship to the velocity-MSE coefficient κ\kappa.
  • Could you report reward-augmented CGM-reward results, with matched target feasibility, reward, and regularization? What distinguishes its multiplier estimation from CCFO’s?
  • How much of the protein improvement remains with the analytically initialized multiplier held fixed and the hard indicator retained? How does the same feedback procedure perform from a generic initialization, with total sampling budgets matched?
4 comments

We sincerely thank the reviewer for the positive feedback on our theoretical results, illustrative 2D example, and encouraging experimental results. Below, we address the points and questions raised in the review.

W1. How is conditioning on prompts handled in conditional-generation experiments?

Thanks for raising this point. We clarify that the problem we intend to solve holds prompt distribution fixed, and the feasibility is averaged over this fixed prompt distribution. Our image implementation solves this problem by using prompt-specific normalization and then averaging over prompts. Please see the details below.

(1) The conditional problem we intend to solve

Let ρ(c)\rho(c) be the fixed prompt distribution, ppre(x∣c)p_{\mathrm{pre}}(x\mid c) the pretrained generator, and s(x,c)=1a(x)≥5.3s(x,c)=\mathbf{1}\\{a(x)\geq5.3\\} the feasibility indicator. Writing q=1−βq=1-\beta, the problem is

max⁡pEc∼ρ![Ex∼p(⋅∣c)r(x,c)−αDKL!(p(⋅∣c)∥ppre(⋅∣c))]s.t.Ec∼ρEx∼p(⋅∣c)s(x,c)≥q.\begin{aligned}\max_{p}\quad &\mathbb{E}_{c\sim\rho}\\!\left[\mathbb{E}_{x\sim p(\cdot\mid c)}r(x,c)-\alpha D_{\mathrm{KL}}\\!\left(p(\cdot\mid c)\Vert p_{\mathrm{pre}}(\cdot\mid c)\right)\right] \\\\ \text{s.t.}\quad &\mathbb{E}_{c\sim\rho}\mathbb{E}_{x\sim p(\cdot\mid c)}s(x,c)\geq q.\end{aligned}

Every conditional distribution is normalized separately. The chance constraint averages over prompts.

(2) Optimal conditional distribution and multiplier.

Using the same infeasible/feasible convention as Theorem 1, define Ac=Eppre(⋅∣c)[er(X,c)/α(1−s(X,c))]A_c=\mathbb{E}_{p_{\mathrm{pre}}(\cdot\mid c)}[e^{r(X,c)/\alpha}(1-s(X,c))] and Bc=Eppre(⋅∣c)[er(X,c)/αs(X,c)]B_c=\mathbb{E}_{p_{\mathrm{pre}}(\cdot\mid c)}[e^{r(X,c)/\alpha}s(X,c)]. For a fixed λ\lambda, maximizing the Lagrangian separately for each prompt gives the optimal conditional distribution

pλ(x∣c)=ppre(x∣c)e(r(x,c)+λs(x,c))/αZc(λ),Zc(λ)=Ac+eλ/αBc.p_\lambda(x\mid c)=\frac{p_{\mathrm{pre}}(x\mid c)e^{(r(x,c)+\lambda s(x,c))/\alpha}}{Z_c(\lambda)}, Z_c(\lambda)=A_c+e^{\lambda/\alpha}B_c.

Consequently, feasibility under the fixed prompt distribution is Qfixed(λ)=Ec∼ρ[eλ/αBcAc+eλ/αBc].Q_{\mathrm{fixed}}(\lambda)=\mathbb{E}_{c\sim\rho}\left[\frac{e^{\lambda/\alpha}B_c}{A_c+e^{\lambda/\alpha}B_c}\right]. The scalar convex dual is

min⁡λ≥0Gcond(λ),\min_{\lambda\geq0}G_{\mathrm{cond}}(\lambda), where Gcond(λ)=αEc∼ρlog⁡Zc(λ)−λq.G_{\mathrm{cond}}(\lambda)=\alpha\mathbb{E}_{c\sim\rho}\log Z_c(\lambda)-\lambda q. The constrained optimum is p⋆(⋅∣c)=pλ⋆(⋅∣c)p^\star(\cdot\mid c)=p_{\lambda^\star}(\cdot\mid c).

(3) Why pooling is a different calculation.

Let Aˉ=EρAc\bar A=\mathbb{E}_\rho A_c and Bˉ=EρBc\bar B=\mathbb{E}_\rho B_c. Applying the unconditional formula to these pooled masses calibrates Qpool(λ)=eλ/αBˉAˉ+eλ/αBˉ,Q_{\mathrm{pool}}(\lambda)=\frac{e^{\lambda/\alpha}\bar B}{\bar A+e^{\lambda/\alpha}\bar B}, with λpool=α[log⁡qAˉ(1−q)Bˉ]+.\lambda_{\mathrm{pool}}=\alpha\left[\log\frac{q\bar A}{(1-q)\bar B}\right]_+. This pooled feasibility generally differs from the average of prompt-specific ratios in the expression for Qfixed(λ)Q_{\mathrm{fixed}}(\lambda) above. Specifically, the pooled quantity is the feasibility of the joint tilt P~λ(c,x)=ρ(c)ppre(x∣c)e(r(x,c)+λs(x,c))/αEρZc(λ).\widetilde P_\lambda(c,x)=\frac{\rho(c)p_{\mathrm{pre}}(x\mid c)e^{(r(x,c)+\lambda s(x,c))/\alpha}}{\mathbb{E}_\rho Z_c(\lambda)}.

Here, all prompt–image pairs share one normalization constant. Integrating over images gives ρ~λ(c)=ρ(c)Zc(λ)EρZc(λ).\widetilde\rho_\lambda(c)=\frac{\rho(c)Z_c(\lambda)}{\mathbb{E}_\rho Z_c(\lambda)}. Therefore, pooled calibration can count gains from reallocating probability across prompts. Our intended calibration uses QfixedQ_{\mathrm{fixed}}, preserving the original prompt weights.

(4) Our image implementation.

For a candidate multiplier λ\lambda, Qfixed(λ)Q_{\mathrm{fixed}}(\lambda) denotes the feasibility of the corresponding ideal conditional distribution. We estimate this quantity using a fixed pool of M=64M=64 training prompts and K=16K=16 pretrained images per prompt, as reported in the appendix. To obtain the optimal λ\lambda, we use bisection to find λ\lambda such that Qfixed(λ)=1−βQ_{\mathrm{fixed}}(\lambda)=1-\beta. The selected λ\lambda is then held fixed during the fine-tuning stage.

Based on the points above, our method is applicable to conditional-generation experiments with a fixed prompt distribution. We will provide these conditional-generation details and clarification in the revised manuscript. Thank you for pointing out this!

W2.1. Connection to CGM-reward.

Thank you for pointing out this connection. After a careful review of CGM-reward by Smith et al., we find that the ideal distributional characterization in Theorem 1 can be obtained by applying the binary calibration formula to the reward-tilted reference, with clipping to account for the inequality constraint. We will provide the intermediate derivation and revise the presentation accordingly in the revised manuscript.

We would also like to clarify the scope of our work relative to Smith et al. Their formulation seeks to match prescribed distributional statistics while minimizing KL divergence from the pretrained distribution, without reward maximization. Our formulation explicitly incorporates reward maximization and imposes a minimum feasibility requirement, which leads to reward–feasibility trade-offs. Our experiments mainly focus on these reward–feasibility trade-offs, whereas their focus is on matching prescribed distributional statistics. Our inequality formulation also differs from their equality formulation in both mathematical specification and practical motivation. Furthermore, we analyze the mixture structure of the optimal distribution and quantify its sensitivity to multiplier estimation errors.

W2.2. Improvements from the initial closed-form estimate, multiplier updates, and constraint smoothing.**

Thank you for raising this point. Multiplier updates and constraint smoothing are practical refinements intended to improve feasibility tracking when fine-tuning only approximately realizes the ideal target distribution. In our 2D and text-to-image experiments, we use a fixed multiplier and the original hard indicator and obtain the desired performance. In the protein experiments, we use these refinements to address the feasibility shortfall observed with the fixed-multiplier implementation.

We have added controlled ablations at target feasibility q=0.8q=0.8, as reported in Table 1.1 below. All three variants start from the closed-form multiplier estimate. Holding this estimate fixed and retaining the hard indicator yields a mean fitness of 0.410 and feasibility of 63.46%. Adding multiplier feedback increases feasibility to 75.01%, and subsequently introducing constraint smoothing raises it to 82.31%, exceeding the target on average. These improvements in feasibility are accompanied by a decrease in mean fitness. The feedback-only variant yields the largest mean number of distinct feasible hits, while the smoothed variant achieves the desired feasibility and greater diversity among feasible samples than the feedback-only variant.

Importantly, comparison with the CFO results in Table 6 of the paper shows that CCFO still achieves higher mean fitness at comparable realized feasibility, regardless of whether multiplier feedback or indicator smoothing is used. We will incorporate these results and clarify the contribution of each refinement in the revised manuscript.

Table 1.1 Ablation of feedback and reward smoothing for TrpB generation at target feasibility q=0.8q=0.8.

MethodMean fitness (all)Feasibility (%)Diversity (given feasible)Distinct designs (feasible & hit)
No feedback, hard indicator0.41063.461.720119.0
Feedback, hard indicator0.40175.011.666178.0
Feedback, smooth indicator0.38982.311.708146.7

The benefit of the analytic multiplier estimate. Thank you for raising this point. As suggested by the reviewer, we explore the practical benefit of analytic initialization while accounting for the additional feedback samples. As shown in Table 1.2 below, on TrpB with target feasibility q=0.8q=0.8, zero initialization with 24 online calibration rounds (6,144 samples) fails to achieve the target feasibility. In contrast, analytic initialization with 10 online calibration rounds (2,560 samples) achieves the desired target feasibility, using approximately 58% fewer online samples. Moreover, without any calibration rounds, analytic initialization still improves feasibility to approximately 63%, while zero initialization decreases feasibility to 25.73%. These results support the analytic multiplier as an informative warm start under limited sampling budgets.

Table 1.2 Effect of multiplier initialization on TrpB generation at target feasibility q=0.8q=0.8.

Multiplier initializationCalibration roundsOnline samplesFeasibility (%)Mean fitness
Zero0025.730.498
Analytic0063.460.410
Zero102,56075.690.401
Analytic102,56081.260.393
Zero246,14478.230.397
Analytic246,14482.310.389

Q1.

The prompt distribution is fixed; please see our response to W1 for further details.

Regarding regularization, α\alpha and κ\kappa weight different regularization terms: α\alpha weights the distribution-level KL penalty, and κ\kappa weights the velocity-MSE reference penalty in the fine-tuning solver. Both discourage deviation from the pretrained model. In our text-to-image experiments, we use DiffusionNFT, an RL fine-tuning method whose implementation does not directly evaluate the terminal-distribution KL penalty. We therefore use its velocity-MSE reference penalty as a practical surrogate. We acknowledge that an exact correspondence between these two coefficients has not been established for our implementation, which is left as our future work. Alternatively, one can use suitable fine-tuning solvers, such as Adjoint Matching, that are compatible with KL regularization. Given the limited rebuttal period, we leave implementing and evaluating this alternative to future work. Finally, all methods use the same regularizer, so these results still provide experimental evidence for the effectiveness of our method.

Q2. Reward-augmented CGM-reward results

As discussed above, applying binary calibration to the reward-tilted reference yields the same multiplier and ideal target distribution as CCFO when the constraint is active. Therefore, this "Augmented CGM-reward" coincides with CCFO without multiplier updates and indicator smoothing. Its performance is reported in the No feedback, hard indicator row of Table 1.1. A detailed discussion of this connection and the scope of our contributions is provided in our reply to W2.1.

Q3.

Please see our reply to W2.2.

We would like to thank again the reviewer for constructive comments allowing us to improve our work. We will add the clarifications and additional theoretical and experimental results in the revised manuscript.

Dear Reviewer,

Thank you again for your constructive feedback. As the discussion period is coming to an end, we wanted to kindly follow up on our earlier response and would appreciate hearing whether our revisions have addressed your concerns.

If so, we would be grateful if you could consider updating your assessment. We are also happy to clarify any remaining questions.

Thank you for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.