- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new algorithm, Fenchel Tilt Flow Control (FTFC), for optimizing a pre-trained diffusion/ flow model given a general utility function defined on its output distribution regularized by an f-divergence. FTFC is a two-stage algorithm: (1) weight optimization from a sample bank (2) weighted regression with the weights computed in stage (1). Theoretically, it is shown that this two-stage decomposition for the original problem is justified. The key advantages of the proposed algorithm seem to be that: (1) it works with general utilities and f-divergences (2) it is efficient. Empirical results are on molecule and image generation
Critical strengths
(1) FTFC is theoretically well-motivated; the two stage decomposition, under reasonable assumptions, is justified and is effective for non-linear utilities. Although the theoretical tools used already exist as separate pieces, bringing them together in the form of this specific algorithm is novel as far as I know.
(2) FTFC is potentially more efficient since it does not rely on backpropagating through the sampling chain and can handle non-differentiable rewards. FTFC also preserves diversity (Table 2) and, on GEOM, validity (Table 3, Table 8) better than the baselines , though it is unclear whether this reflects the method or differences in effective regularization (see weaknesses).
Critical weaknesses
(1) The experimental evidence is rather limited:
-
While the derived theory is general, all main experiments use CVaR with KL-regularization. This setup reduces to the one considered already in TFFT. This raises the question of whether the extended theoretical model is useful in practice; experiments where more general utility functions and regularizers are useful need to be studied to establish the advantage of having a more general framework.
-
The quality/ performance gains over baselines, when they exist, seem marginal. So the main advantage of FTFC seems to be efficiency. In this case, a more controlled comparison with the baselines in terms of compute and wall clock time is necessary. In particular, as already shown in appendix B.4, accounting for sample bank generation time diminishes the gains. An ablation on number of samples needed in stage 1 would be necessary to establish efficiency gain (see weakness (2))
-
There are confounding factors within the baseline comparisons. In particular, on QM9 the regularization strength is not matched across methods (Appendix B.2), and the expected-reward baselines optimize a different objective altogether. The better quality metrics (e.g., topology reward on QM9, Table 1) may therefore reflect a more conservative trade-off rather than an advantage of the method itself. Sweeping the regularization strength and reporting reward-divergence curves would make the comparisons more informative. Further, for image generation FTFC trains an adapter, while baselines update the full U-Net. This could potentially lead to more conservative updates. Also, from Appendix B.4, the FTFC checkpoint for image generation is selected based on lower-tail CVaR on the evaluation prompts, whereas the baselines use their final checkpoints; this could potentially inflate the reported gains.
(2) Given that the algorithm’s performance depends on how effectively optimal weights can be estimated in Stage 1, a discussion is necessary regarding how the number of samples in stage 1 affects the performance. Even if a theoretical analysis is difficult, some ablation/ heuristic is needed to implement the algorithm in practice.
Questions for authors
(1) Any clarification/ additional experiments answering the points raised in weaknesses would be helpful.
(2) Additional ablations would help contextualize the need for stage 1 more clearly. For example, a direct reward-weighted fine-tuning using the sample bank is missing.
4 comments
3. Image sample-count ablation and checkpoint selection.
To eliminate potential evaluation-set selection bias on Stable Diffusion v1.5 (where Appendix B.4 noted checkpoint selection via lower-tail CVaR) and assess Stage-1 sample sensitivity, we conducted two additional controlled experiments using final checkpoints only:
- Unselected final checkpoints on held-out prompts: We evaluated all five final acquisition/controller fits on 84 held-out prompts without any post-hoc checkpoint or fit selection.
- Stage-1 prompt/sample scaling: We fitted controllers using 64, 128, and 256 training prompts from each acquisition recipe and evaluated the final checkpoints on a separate disjoint set of 64 new prompts alongside two frozen-generator controls (DDIM50 and SDE32).
| Method | ImageReward ↑ | Lower CVaR(.2) ↑ | DreamSim marginal diversity ↑ | |
|---|---|---|---|---|
| Frozen DDIM50 | 0.21 | −1.23 | 0.30 | |
| Frozen SDE32 | 0.19 | −1.22 | 0.34 | |
| FTFC (5-fit mean, final checkpoints, 84 held-out prompts) | 0.55 | −0.68 | 0.42 | |
| FTFC (64 training prompts, 64 new eval prompts) | 0.44 | -0.86 | 0.39 | |
| FTFC (128 training prompts, 64 new eval prompts) | 0.51 | -0.74 | 0.41 | |
| FTFC (256 training prompts, 64 new eval prompts) | 0.54 | -0.69 | 0.41 |
Even when strictly evaluating final checkpoints on unseen prompts, FTFC substantially improves mean ImageReward and lower-tail CVaR (+0.54 to +0.55) over the frozen baselines while increasing marginal diversity. Furthermore, varying the Stage-1 training bank across 64, 128, and 256 prompts shows that FTFC achieves strong tail and mean improvements even at 64 prompts (0.44 ImageReward, -0.86 L-CVaR), with performance saturating by 128–256 prompts (0.54 ImageReward, -0.69 L-CVaR). This confirms that Stage 1 requires only a modest sample bank to accurately calibrate the target distribution, preserving FTFC's wall-clock efficiency advantage even when accounting for bank generation time.
5. Regularization Sweeps, Confounding Factors, and Objective Differences
To directly test FTFC's sensitivity to regularization strength , update budget, and fitting proposals without confounding baseline differences, we conducted a controlled sweep on QM9 over Ha, update budgets , and two fitting proposals (3 banks 2 fitting seeds per setting; 18 settings, 109 evaluated laws, and 111,616 total attempts):
| Regularization (Ha) | Upper-CVaR(0.9) Range (Ha) ↑ | Sampling Time Range (min / 16 attempts) ↓ |
|---|---|---|
| Frozen reference | 33.93 | 0.32 |
| 33.93–33.95 | 0.34–0.34 | |
| 33.93–33.97 | 0.34–0.34 | |
| 33.92–33.94 | 0.34–0.34 |
Across this sweep, Upper-CVaR(0.9) remains stable within a tight band (33.92–33.97 Ha, with paired intervals against the frozen reference overlapping zero) and sampling overhead is consistent (6.8–7.3% slower than the frozen reference due to the adapter). We will add these sensitivity results to the camera-ready revision
6. Direct Reward-Weighted Fine-Tuning and the Role of Stage 1
Stage 1's mathematical role is to compute the effective marginal reward and normalized density-ratio weights for nonlinear distributional utilities and general -divergences.
-
For linear expected reward () under KL regularization, Stage 1 reduces to standard exponential tilting, , where the marginal reward is simply the static reward .
-
By contrast, for nonlinear utilities (such as CVaR, moment matching, or coverage entropy), the marginal reward depends on the unknown optimal target distribution itself. Moreover, non-KL -divergences require solving for the normalization multiplier inside the nonlinear conjugate subgradient , rather than normalizing the weights by dividing by their sum post hoc.
In our 2D rings benchmark (Figure 1, Appendix C.1), we compared FTFC against Implicit Tilt Matching (ITM), which applies direct exponential tilting without Stage-1 distribution calibration. The results demonstrate that Stage 1 is critical for preventing mode/arc collapse under nonlinear utilities, with an distance of 0.032 for FTFC versus 0.205 for ITM on coverage entropy.
We thank again for your review. As the discussion period is coming to an end, we wanted to check whether our response and the revised paper address your concerns. We are willing to answer to remaining questions or provide additional clarifications.
Dear Reviewer TAdG, could you check if the responses from the authors have addressed your concerns?
We thank the reviewer for the constructive feedback and for highlighting the novelty and theoretical justification of FTFC's two-stage decomposition. Below, we address each concern with new experiments on general distribution-dependent utilities, non-KL f divergences, Stage-1 sample-bank size ablations, and checkpoint-selection controls.
1. Practical value beyond KL/CVaR.
We agree that KL/CVaR experiments alone do not fully demonstrate the practical benefit of our general framework. While Appendix C.1 demonstrates coverage entropy on 2D rings, we performed new high-dimensional experiments on GEOM-Drugs (FlowMol3) using three distribution-dependent utilities under fixed KL regularization:
Because these objectives depend nonlinearly on distributional moments and bin occupancies, they cannot be reduced to a fixed per-sample reward or single-threshold CVaR. Evaluating across three independently acquired sample banks and three fitting seeds per objective yields consistent gains over the pretrained reference:
These results confirm that the structured Fenchel dual (Eq. 9) effectively optimizes diverse, distribution-dependent design criteria on molecular benchmarks.
2. Non-KL regularizers and molecular Stage-1 Bank Size ablation count.
To test non-KL divergences and quantify sensitivity to Stage-1 sample bank size N, we evaluated KL, half-Pearson χ2, and reverse KL on GEOM-Drugs using the exact same backbone, rank-16 adapter, fitting budget (120 updates), and sampler (β=0.9, α=1/6 in raw Hartree units). We ran 3 independent banks × 2 fitting seeds across four nested bank sizes for KL (N∈{256,1024,4096,10000}) and at N=10,000 for the non-KL divergences:
Non-KL regularizers: Half-Pearson χ2 (which permits exact zero weights via w=(1+u)+) achieves the highest upper-tail energy (95.62 Ha) while maintaining QED, demonstrating that alternative f-divergences offer practical advantages over standard exponential KL tilting.
Efficiency: While larger banks improve support coverage of the pretrained distribution, downstream generated performance saturates around N=1,024–4,096 and is slightly non-monotonic at N=10,000 due to finite-capacity adapter fitting. Crucially, even at N=1,024 (which cuts bank acquisition time by ∼10× compared to N=10,000), FTFC achieves 94.63 Ha and 0.69 QED. Thus, even when charging upfront bank generation time (Appendix B.3.3, Fig. 5), a moderate bank of N=1,024 reduces total wall-clock cost from 3.48 GPU-hours to <0.4 GPU-hours, making FTFC faster than online baselines without sacrificing quality.