- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method(CCFO) for fine-tuning diffusion and flow models to improve rewards satisfying a given constraint. The main contribution is that they simplifies the constrained optimization problem into estimating a single parameter from theoretical perspective and do experiments on 2D examples, image generation, and protein design to show better reward–feasibility trade-offs than existing methods.
Critical strengths
The paper addresses a practical problem, as controlling the probability to satisfy a constraint is often more useful than simply optimizing average performance. The proposed method is relatively simple and can work with existing fine-tuning approaches. The experiments also cover different applications and generally support the effectiveness of the method.
Critical weaknesses
I'm a little bit doubtful about the gap between the theoretical guarantees and practical results. The theory do proves constraint satisfaction for the ideal optimized distribution, but the actual fine-tuned models do not always meet the desired probability. So I'm pretty not sure how reliably the proposed method can enforce constraints in practice since some of the experiments results also don't fully meet the expectaion.
Questions for authors
- How reliably does CCFO achieve the target feasibility across different constraint levels and random seeds? Could the authors provide a more detailed analysis of cases where the target is not met?
- In the protein experiments, how much does performance change without the additional multiplier updates and reward smoothing? Can the original two-stage method achieve similar results on its own?
3 comments
Dear Reviewer,
Thank you again for your constructive feedback. As the discussion period is coming to an end, we wanted to kindly follow up on our earlier response and would appreciate hearing whether our revisions have addressed your concerns. If so, we would be grateful if you could consider updating your assessment. Thank you for taking the time to consider our follow-up.
Thank you for all the additional experiments. I appreciate your efforts. While these additions provide useful insights, I still have concerns over the gap between theoretical formulation and its practical feasibility, especially the reliance on additional multiplier updates and constraint smoothing to achieve the target feasibility. Therefore, I acknowledge the improvements, but my remaining concerns lead me to maintain my original assessment.
We thank the reviewer for the positive feedback on the studied problem and experiments supporting the effectiveness. In the following, we address several points mentioned within the review.
Thanks for raising this point. We agree that there might exist a gap between theoretical guarantee and practical result on the feasibility. To illustrate the reliability of our method, we have tested our methods across different target feasibility levels, in Tables 5 and 6 in Appendix of the paper, for text-to-image and protein generation, respectively. These results show that our method achieves the desired target feasibility at most of the settings. To fully address your concern, we analyze the failure cases, refine the algorithm, and added new experimental results over more seeds.
In text-to-image generation, the additional results are presented below. We observe that CCFO with fixed multiplier meets the target feasibility in all three seeds at the tested targets from 70% to 93%. At the target of 58%, however, it does not consistently meet the requirement. In this setting, we observe that the estimated multiplier is close to zero, providing little additional feasibility incentive. To improve feasibility tracking, we introduce periodic multiplier updates, as used in the protein experiments. With this refinement, the 58% target is met in all three seeds.
Table 3.1 Additional results of CCFO in text-to-image generation.
In protein generation, fine-tuning with fixed multiplier cannot reliably achieve the target feasibility, because the fine-tuning solver cannot realize the ideal distribution under limited sampling and training budgets. We therefore use multiplier feedback and constraint-indicator smoothing to improve feasibility tracking. Moreover, at the stricter target of 0.9, we observe that feasibility oscillates around the target, so the final checkpoint may fall slightly below the requirement. To accommodate these fluctuations, we introduce a small feasibility margin by setting the internal training target 0.92, slightly above the required level. With these practical refinements, empirical feasibility meets the required target in all five seeds at each tested level. These results together illustrate the relibility of our method.
Table 3.2 Additional results of CCFO in protein generation.
Thank you for raising this point. As discussed above, multiplier updates and constraint smoothing are practical refinements intended to improve feasibility tracking when fine-tuning only approximately realizes the ideal target distribution. To investigate their effects, we have added controlled ablations at target feasibility q=0.8, as reported below. All three variants start from the closed-form multiplier estimate. Holding this estimate fixed and retaining the hard indicator yields a mean fitness of 0.410 and feasibility of 63.46%. Adding multiplier feedback increases feasibility to 75.01%, and subsequently introducing constraint smoothing raises it to 82.31%, achieving the target feasibility. These improvements in feasibility are accompanied by a decrease in mean fitness. The feedback-only variant yields the largest mean number of distinct feasible hits, while the smoothed variant achieves the desired feasibility and greater diversity among feasible samples than the feedback-only variant.
Importantly, comparison with the CFO results in Table 6 of the paper shows that CCFO still achieves higher mean fitness at comparable realized feasibility, regardless of whether multiplier feedback or indicator smoothing is used. This again illustrates that CCFO achieves better reward-feasibility trade-offs than CFO.
Table 3.3 Ablation of feedback and reward smoothing for TrpB generation at target feasibility q=0.8.