acceptodds
Under review as a conference paper at ICLR 2027

Fenchel Tilting: Weighted Correction for Efficient Finetuning of Generative Models

Abstract

Adapting a pretrained generative model to an arbitrary preference expressed as a utility function underlies reward alignment, guided design, and constraint satisfaction, enabling diverse applications. Existing fine-tuning methods trade off generality against computational cost: they either restrict the family class of supported preferences to keep optimization simple or preserve generality at the expense of efficiency. We introduce Fenchel Tilt Flow Control (FTFC), which decouples utility optimization from generative-model fitting. FTFC first optimizes for a target distribution by jointly fitting an effective reward and density-ratio weights on pretrained samples. Method combines the utility’s variational structure with Fenchel duality, supporting general -divergence penalties that determine how rewards are transformed into an distribution-correction weights. These weights are then frozen and used to modify a diffusion or flow model in a single stage of importance-weighted denoising or flow matching, without differentiating through sampling trajectories. We establish exact duality for concave utilities under suitable conditions and show that weighted fitting reproduces the optimal target distribution for a given utility. Across image and molecule generation benchmarks, FTFC improves over baselines on diverse preference functions, while also being up to more efficient. Proposed method enables adaptation beyond expected-reward maximization without complex optimization, while preserving robustness for more general class of the utility functions compared to baselines.

Primary area:
generative models
Keywords:
diffusion models, flow matching, generative optimization, fine-tuning, stochastic optimal control, steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TAdGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new algorithm, Fenchel Tilt Flow Control (FTFC), for optimizing a pre-trained diffusion/ flow model given a general utility function defined on its output distribution regularized by an f-divergence. FTFC is a two-stage algorithm: (1) weight optimization from a sample bank (2) weighted regression with the weights computed in stage (1). Theoretically, it is shown that this two-stage decomposition for the original problem is justified. The key advantages of the proposed algorithm seem to be that: (1) it works with general utilities and f-divergences (2) it is efficient. Empirical results are on molecule and image generation

Critical strengths

(1) FTFC is theoretically well-motivated; the two stage decomposition, under reasonable assumptions, is justified and is effective for non-linear utilities. Although the theoretical tools used already exist as separate pieces, bringing them together in the form of this specific algorithm is novel as far as I know.

(2) FTFC is potentially more efficient since it does not rely on backpropagating through the sampling chain and can handle non-differentiable rewards. FTFC also preserves diversity (Table 2) and, on GEOM, validity (Table 3, Table 8) better than the baselines , though it is unclear whether this reflects the method or differences in effective regularization (see weaknesses).

Critical weaknesses

(1) The experimental evidence is rather limited:

  • While the derived theory is general, all main experiments use CVaR with KL-regularization. This setup reduces to the one considered already in TFFT. This raises the question of whether the extended theoretical model is useful in practice; experiments where more general utility functions and regularizers are useful need to be studied to establish the advantage of having a more general framework.

  • The quality/ performance gains over baselines, when they exist, seem marginal. So the main advantage of FTFC seems to be efficiency. In this case, a more controlled comparison with the baselines in terms of compute and wall clock time is necessary. In particular, as already shown in appendix B.4, accounting for sample bank generation time diminishes the gains. An ablation on number of samples needed in stage 1 would be necessary to establish efficiency gain (see weakness (2))

  • There are confounding factors within the baseline comparisons. In particular, on QM9 the regularization strength is not matched across methods (Appendix B.2), and the expected-reward baselines optimize a different objective altogether. The better quality metrics (e.g., topology reward on QM9, Table 1) may therefore reflect a more conservative trade-off rather than an advantage of the method itself. Sweeping the regularization strength and reporting reward-divergence curves would make the comparisons more informative. Further, for image generation FTFC trains an adapter, while baselines update the full U-Net. This could potentially lead to more conservative updates. Also, from Appendix B.4, the FTFC checkpoint for image generation is selected based on lower-tail CVaR on the evaluation prompts, whereas the baselines use their final checkpoints; this could potentially inflate the reported gains.

(2) Given that the algorithm’s performance depends on how effectively optimal weights can be estimated in Stage 1, a discussion is necessary regarding how the number of samples in stage 1 affects the performance. Even if a theoretical analysis is difficult, some ablation/ heuristic is needed to implement the algorithm in practice.

Questions for authors

(1) Any clarification/ additional experiments answering the points raised in weaknesses would be helpful.

(2) Additional ablations would help contextualize the need for stage 1 more clearly. For example, a direct reward-weighted fine-tuning using the sample bank is missing.

4 comments

We thank the reviewer for the constructive feedback and for highlighting the novelty and theoretical justification of FTFC's two-stage decomposition. Below, we address each concern with new experiments on general distribution-dependent utilities, non-KL ff divergences, Stage-1 sample-bank size ablations, and checkpoint-selection controls.

1. Practical value beyond KL/CVaR.

We agree that KL/CVaR experiments alone do not fully demonstrate the practical benefit of our general framework. While Appendix C.1 demonstrates coverage entropy on 2D rings, we performed new high-dimensional experiments on GEOM-Drugs (FlowMol3) using three distribution-dependent utilities under fixed KL regularization:

  • QED with log⁡P\log P moment matching: F(p)=Ep[QED]−(Ep[z])2\mathcal{F}(p) = \mathbb{E}_p[\mathrm{QED}] - (\mathbb{E}_p[z])^2, where zz is standardized log⁡P\log P
  • QED with coverage entropy: F(p)=Ep[QED]+0.2H(h)\mathcal{F}(p) = \mathbb{E}_p[\mathrm{QED}] + 0.2 H(h), where h=Ep[ϕ]h = \mathbb{E}_p[\phi] measures occupancy across four fixed log⁡P\log P bins
  • Moment matching with elemental/charge composition constraints.

Because these objectives depend nonlinearly on distributional moments and bin occupancies, they cannot be reduced to a fixed per-sample reward or single-threshold CVaR. Evaluating across three independently acquired sample banks and three fitting seeds per objective yields consistent gains over the pretrained reference:

ObjectiveFrozen utilityFTFC generated utility
Moment matching0.620.69
Coverage entropy0.940.98
Moment + composition constraints0.630.68

These results confirm that the structured Fenchel dual (Eq. 9) effectively optimizes diverse, distribution-dependent design criteria on molecular benchmarks.

2. Non-KL regularizers and molecular Stage-1 Bank Size ablation count.

To test non-KL divergences and quantify sensitivity to Stage-1 sample bank size NN, we evaluated KL, half-Pearson χ2\chi^2, and reverse KL on GEOM-Drugs using the exact same backbone, rank-16 adapter, fitting budget (120 updates), and sampler (β=0.9\beta = 0.9, α=1/6\alpha = 1/6 in raw Hartree units). We ran 3 independent banks ×\times 2 fitting seeds across four nested bank sizes for KL (N∈{256,1024,4096,10000}N \in \{256, 1024, 4096, 10000\}) and at N=10,000N = 10,000 for the non-KL divergences:

Method / divergenceRaw bank sizeQED ↑Upper-10% CVaR, Ha ↑Sampling, min/256 attempts ↓
Frozen reference—0.5790.22.75
FTFC / KL2560.6794.22.81
FTFC / KL1,0240.6994.632.81
FTFC / KL4,0960.6895.002.81
FTFC / KL10,0000.6794.712.82
FTFC / half-Pearson χ²10,0000.6895.622.82
FTFC / reverse KL10,0000.6894.552.82

Non-KL regularizers: Half-Pearson χ2\chi^2 (which permits exact zero weights via w=(1+u)+w = (1+u)_+) achieves the highest upper-tail energy (95.62 Ha) while maintaining QED, demonstrating that alternative ff-divergences offer practical advantages over standard exponential KL tilting.

Efficiency: While larger banks improve support coverage of the pretrained distribution, downstream generated performance saturates around N=1,024–4,096N = 1,024\text{--}4,096 and is slightly non-monotonic at N=10,000N = 10,000 due to finite-capacity adapter fitting. Crucially, even at N=1,024N = 1,024 (which cuts bank acquisition time by ∼10×\sim 10\times compared to N=10,000N = 10,000), FTFC achieves 94.63 Ha and 0.69 QED. Thus, even when charging upfront bank generation time (Appendix B.3.3, Fig. 5), a moderate bank of N=1,024N = 1,024 reduces total wall-clock cost from 3.48 GPU-hours to <0.4< 0.4 GPU-hours, making FTFC faster than online baselines without sacrificing quality.

3. Image sample-count ablation and checkpoint selection.

To eliminate potential evaluation-set selection bias on Stable Diffusion v1.5 (where Appendix B.4 noted checkpoint selection via lower-tail CVaR) and assess Stage-1 sample sensitivity, we conducted two additional controlled experiments using final checkpoints only:

  • Unselected final checkpoints on held-out prompts: We evaluated all five final acquisition/controller fits on 84 held-out prompts without any post-hoc checkpoint or fit selection.
  • Stage-1 prompt/sample scaling: We fitted controllers using 64, 128, and 256 training prompts from each acquisition recipe and evaluated the final checkpoints on a separate disjoint set of 64 new prompts alongside two frozen-generator controls (DDIM50 and SDE32).
MethodImageReward ↑Lower CVaR(.2) ↑DreamSim marginal diversity ↑
Frozen DDIM500.21−1.230.30
Frozen SDE320.19−1.220.34
FTFC (5-fit mean, final checkpoints, 84 held-out prompts)0.55−0.680.42
FTFC (64 training prompts, 64 new eval prompts)0.44-0.860.39
FTFC (128 training prompts, 64 new eval prompts)0.51-0.740.41
FTFC (256 training prompts, 64 new eval prompts)0.54-0.690.41

Even when strictly evaluating final checkpoints on unseen prompts, FTFC substantially improves mean ImageReward and lower-tail CVaR (+0.54 to +0.55) over the frozen baselines while increasing marginal diversity. Furthermore, varying the Stage-1 training bank across 64, 128, and 256 prompts shows that FTFC achieves strong tail and mean improvements even at 64 prompts (0.44 ImageReward, -0.86 L-CVaR), with performance saturating by 128–256 prompts (0.54 ImageReward, -0.69 L-CVaR). This confirms that Stage 1 requires only a modest sample bank to accurately calibrate the target distribution, preserving FTFC's wall-clock efficiency advantage even when accounting for bank generation time.

5. Regularization Sweeps, Confounding Factors, and Objective Differences

To directly test FTFC's sensitivity to regularization strength α\alpha, update budget, and fitting proposals without confounding baseline differences, we conducted a controlled sweep on QM9 over α∈{0.1,1,5}\alpha \in \{0.1, 1, 5\} Ha, update budgets ∈{50,200,600}\in \{50, 200, 600\}, and two fitting proposals (3 banks ×\times 2 fitting seeds per setting; 18 settings, 109 evaluated laws, and 111,616 total attempts):

Regularization α\alpha (Ha)Upper-CVaR(0.9) Range (Ha) ↑Sampling Time Range (min / 16 attempts) ↓
Frozen reference33.930.32
α=0.1\alpha = 0.133.93–33.950.34–0.34
α=1.0\alpha = 1.033.93–33.970.34–0.34
α=5.0\alpha = 5.033.92–33.940.34–0.34

Across this sweep, Upper-CVaR(0.9) remains stable within a tight band (33.92–33.97 Ha, with paired intervals against the frozen reference overlapping zero) and sampling overhead is consistent (6.8–7.3% slower than the frozen reference due to the adapter). We will add these sensitivity results to the camera-ready revision

6. Direct Reward-Weighted Fine-Tuning and the Role of Stage 1

Stage 1's mathematical role is to compute the effective marginal reward g∗g^{\ast} and normalized density-ratio weights w∗w^{\ast} for nonlinear distributional utilities F(p)\mathcal{F}(p) and general ff-divergences.

  • For linear expected reward (F(p)=Ep[r]\mathcal{F}(p) = \mathbb{E}_p[r]) under KL regularization, Stage 1 reduces to standard exponential tilting, pi∝qiexp⁡(ri/α)p_i \propto q_i \exp(r_i/\alpha), where the marginal reward is simply the static reward r(x)r(x).

  • By contrast, for nonlinear utilities (such as CVaR, moment matching, or coverage entropy), the marginal reward gp∗(x)=δFδp(p∗)(x)g_{p^{\ast}}(x) = \frac{\delta \mathcal{F}}{\delta p}(p^{\ast})(x) depends on the unknown optimal target distribution p∗p^{\ast} itself. Moreover, non-KL ff-divergences require solving for the normalization multiplier ν∗\nu^{\ast} inside the nonlinear conjugate subgradient wi∈∂f+∗((gi∗−ν∗)/α)w_i \in \partial f_+^{\ast}((g_i^{\ast} - \nu^{\ast})/\alpha), rather than normalizing the weights by dividing by their sum post hoc.

In our 2D rings benchmark (Figure 1, Appendix C.1), we compared FTFC against Implicit Tilt Matching (ITM), which applies direct exponential tilting without Stage-1 distribution calibration. The results demonstrate that Stage 1 is critical for preventing mode/arc collapse under nonlinear utilities, with an SW1\mathrm{SW}_1 distance of 0.032 for FTFC versus 0.205 for ITM on coverage entropy.

We thank again for your review. As the discussion period is coming to an end, we wanted to check whether our response and the revised paper address your concerns. We are willing to answer to remaining questions or provide additional clarifications.

Dear Reviewer TAdG, could you check if the responses from the authors have addressed your concerns?

Reviewer_Bt9ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Fenchel Tilt Flow Control (FTFC): first optimize a distribution over cached pretrained samples using utility-dependent density-ratio weights, then fit a generator to that distribution through weighted denoising or flow matching. Its principal theoretical contribution is a unified calibration framework for concave utilities. The framework is mathematically coherent and practically appealing. However, the experimental evidence does not currently establish the advertised breadth or comparative efficiency.

Critical strengths
  • The structured utility dual in Eq. (9) provides a useful way to jointly determine marginal rewards and normalized weights before generator fitting. The treatment of non-KL normalization is careful: the paper correctly explains why simply rescaling conjugate-response weights generally fails outside KL. Proposition 3 also provides a useful exact finite-bank search for lower-CVaR. -Promising results in text-to-image generation and molecule design
Critical weaknesses
  • Exact optimality concerns the relaxed target problem and idealized generator realization. It does not establish that the small cached banks used experimentally approximate the population optimum. This limitation is particularly important for extreme-tail objectives.
  • Misalignment. The GEOM discussion says FDC obtains higher mean reward, while Table 3 reports FTFC at 75.3 and FDC at 69.2.
  • Table 2 does not provide the replication its caption claims. The caption says “standard error across five seeds,” while Appendix B.4, lines 1759–1772, describes one evaluation per method.
  • Table 2 reports ImageReward metrics to only one decimal place, obscuring differences between methods. This precision is insufficient to assess the magnitude and uncertainty of the reported improvements.- Missing related work or comparisons. ITM appears only in the toy experiments, and no realistic matched comparison isolates FTFC from existing weighted fitting methods or TFFT calibration followed by the same regression backend.
Questions for authors
  • Can you provide a corrected, auditable set of Tables 2–3? Please reconcile the claimed five seeds, the uncertainty definitions.
  • Can you validate the claimed generality beyond KL/CVaR?
  • How sensitive are the results to bank coverage?

Typo:

  • line 147 : Figures ??
5 comments

Thank you for the careful review. We have revised the manuscript with corrections and additions in blue. Moreover, below we address each concern.

Q1: Exact optimality concerns the relaxed target problem and idealized generator realization

FTFC’s exactness result characterizes the optimal target of the stated relaxed problem. On a cached bank, the optimization certificate applies to the corresponding empirical problem. Exact generator realization additionally requires sufficient capacity and exact fitting. We have clarified these assumptions and separated target optimization, statistical approximation and generator realization explicitly.

To assess finite-bank sensitivity, we added experiments with 256,1024,4096256, 1024, 4096 and 1000010000 samples, using three independently acquired banks and two fitting seeds per setting. We report generated outcomes alongside eligibility, missing support groups, effective sample size and target–generator gaps. Performance varies nonmonotonically with bank size, so these experiments provide practical sensitivity evidence without asserting that a small bank recovers the population optimum.

FTFC’s contribution remains an exact, general procedure for calibrating the specified target and fitting a generator to it. The revised presentation makes the additional statistical and realization requirements explicit, allowing the theoretical guarantee and empirical evidence to be interpreted at their appropriate scope.

Bank-size sensitivity on GEOM-Drugs (KL regularization). Each FTFC row averages three independent acquisition banks × two fitting seeds, with 256 generated attempts per fit. Bank prefixes within each acquisition are nested.

MethodRaw bank sizeEligible bank samplesMissing size groups*Generated QED ↑Upper-10% CVaR, Ha ↑
Frozen reference———0.67363195.969392
FTFC25625644–520.67583794.155461
FTFC1,0241,021–1,02437–410.68674994.635957
FTFC4,0964,090–4,09319–260.68141095.050004
FTFC10,0009,983–9,9890–10.67405294.715301

*Ranges are across acquisition banks. Missing size groups are measured relative to each full bank, not the population. All paired QED/CVaR uncertainty intervals against the frozen reference include zero.

Independent extreme-tail evaluation on QM9. Every law receives 65536 evaluation attempts: one frozen reference and all six preselected FTFC fits, totaling 458,752 attempts. FTFC entries average law-specific metrics.

MethodEvaluated lawsMean reward, Ha ↑Upper-CVaR(.998), Ha ↑Connected validity ↑
Frozen reference128.63064138.6770410.967911
FTFC628.62154038.9272610.950734

Q2: Corrected Tables 2-3

Thank you for identifying this inconsistency. We have corrected Tables 2–3 and their uncertainty definitions. Table 2 contains one retained evaluation per method, using 100 prompts × 10 images; we withdraw the unsupported five-training-seed SE claim. ImageReward uncertainty is the sample SD of the 100 prompt means divided by √100, conditional on the retained model and noise panel. It does not measure training or checkpoint-selection uncertainty. Table 3 reports mean ± sample SD across three seeds, with 2000 attempts per seed. For training time, we report GPU-days, where the bank acquisition is included in time comparisons for FTFC:

Corrected Table 2: Stable Diffusion.

MethodImageReward ↑Lower CVaR(0.2) ↑CLIP ↑HPS ↑DreamSim within ↑Training + acquisition, GPU-days ↓
Pretrained0.292 ± 0.070−1.0400.2760.2550.3510.000
EXP-FT0.884 ± 0.064−0.3010.2800.2780.2983.428
FDC0.678 ± 0.065−0.4860.2770.2700.3173.434
L-TFFT0.822 ± 0.063−0.3060.2770.2720.2953.574
FTFC0.910 ± 0.070−0.2530.2890.3100.3451.990

Corrected Table 3: GEOM-Drugs.

MethodMean reward, Ha ↑Upper CVaR(0.9), Ha ↑Validity, % ↑Heavy-atom SA ↓Topology ↑Training + acquisition, GPU-hours ↓
Pretrained68.905 ± 0.14394.955 ± 0.21299.817 ± 0.0292.534 ± 0.0120.952 ± 0.0020.000
EXP-FT67.900 ± 0.75494.676 ± 1.50699.767 ± 0.1892.651 ± 0.0560.929 ± 0.0052.157 ± 0.015
FDC71.223 ± 2.38297.943 ± 1.87799.633 ± 0.1892.527 ± 0.1360.928 ± 0.01616.823 ± 0.217
R-TFFT65.598 ± 0.80890.778 ± 0.92899.750 ± 0.1802.649 ± 0.0580.926 ± 0.0175.541 ± 0.006
FTFC69.000 ± 0.20795.036 ± 0.28699.917 ± 0.1042.550 ± 0.0110.953 ± 0.0013.483 ± 0.001

Q3: How sensitive are the results to bank coverage?

We added a controlled GEOM bank-size study using three independently acquired banks, of 256, 1024, 4096 and 10,000 draws, and two fitting seeds per bank/prefix. The backbone, KL regularization, fitting budget and evaluation protocol are shared. Each FTFC row averages all six bank/fit combinations.

Method / bank sizeMissing size groups*Generated QED ↑Generated upper-10% CVaR, Ha ↑
Frozen reference—0.67363195.969392
FTFC / 25644–520.67583794.155461
FTFC / 102437–410.68674994.635957
FTFC / 409619–260.68141095.050004
FTFC / 100000–10.67405294.715301

Missing requested molecule-size groups are measured against the full-bank reference; ranges span three banks, rather than confidence intervals. QED uses all generated attempts; energy CVaR uses valid, oracle-accepted samples.

Larger banks improve empirical coverage, but generated performance does not improve monotonically. All descriptive paired 95% intervals for QED/CVaR differences between smaller prefixes and the 10,000-draw condition include zero.

Extreme tails require particular caution. At ββ = 0.998, a 256256-draw bank provides only 0.512 nominal tail observations. Under the tested empirical size-group constraints, all 27 target configurations at each of 256 and 1,024 draws request no tilt, despite ESS fractions near one. High ESS therefore does not establish rare-event coverage.

We now explicitly distinguish finite-bank optimization accuracy from population coverage and generator realization, and qualify the extreme-tail claims accordingly (Appendices A.1–A.2).

Q4: Generalization beyond KL/CVaR

We added two complementary experiments: non-KL regularization with CVaR, and structured utilities beyond CVaR. We also refer reviewer to our response to the reviewer w82p in Q1.

1. Non-KL divergences. On GEOM, we compared KL, half-Pearson χ² and reverse KL using three independently acquired 10,00010,000-draw banks and two fitting seeds per bank/divergence, with a common backbone, fitting budget and sampler.

Method / divergenceQED ↑Mean energy reward, Ha ↑Upper-10% energy CVaR, Ha ↑Validity ↑Geometry score ↑
Frozen reference0.6464.393.00.970.90
FTFC / forward KL0.6768.694.71.00.95
FTFC / half-Pearson χ²0.6868.695.61.00.96
FTFC / reverse KL0.6768.194.50.990.95

2. Structured utilities beyond CVaR. With KL regularization, we tested QED combined with logP moment matching, coverage entropy, and moment matching with elemental/charge constraints.

ObjectiveFrozen utilityFTFC utility
Moment matching0.620.69
Coverage entropy0.940.98
Moment + composition constraints0.630.68

All three objectives improve their intended generated utility and QED relative to the matched frozen control. The intervals retain bank, fit and native-batch variation. Generated utilities remain below their optimized finite-bank targets, and finite composition constraints do not automatically transfer to the generator.

These experiments support applicability beyond KL/CVaR in the tested settings. We explicitly retain the remaining limits: the structured-utility experiments use KL, and a full divergence–utility cross-product remains untested (Appendix A.1).

Reviewer_w82pOpenReview ↗
Rating
3(+1)
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Fenchel Tilt Flow Control (FTFC)]. The method uses Fenchel duality to estimate an optimal target distribution through normalized density-ratio weights, then fits a flow model using weighted flow matching without differentiating through sampling trajectories.

Critical strengths
  1. The proposed FTFC method provides a unified way to optimize nonlinear distributional utilities with different f-divergence penalties. And the idea of separating target selection from generator fitting is very clean and may be extended to other fine-tuning settings.

  2. The method is more efficient by reusing cached pre-trained samples and frozen importance weights. In particular, they show that there are quality-efficiency trade-offs in the image and molecular experiments.

Critical weaknesses
  1. I think the experiments are not enough to prove its validity. The paper claims that the method works for a wide range of settings, but the experiments mainly focus on CVaR with KL regularization. It would be helpful to see whether the method also works with other divergences and utility functions.

  2. Some comparison in experiments may not be fair enough. For example, in the comparison with TFFT, the baseline seems to receive limited useful gradients in some experiments. There are also some inconsistencies in how uncertainty is reported in the image experiments. In addition, the author may also clarify whether the reported speedups depend on the cost of generating samples or not.

Questions for authors
  1. Could the authors provide more experiments using non-KL divergences or structured utilities beyond CVaR? These would help substantiate the main generality claim.
  2. Could the authors clarify the image evaluation's seed and uncertainty reporting, and provide comparisons without evaluation-set checkpoint selection? Additionally, can the main efficiency comparisons consistently include sample acquisition and address the inactive-gradient limitations of the reconstructed TFFT baselines?
4 comments

We thank reviewer for additional questions and requests for clarification. We address everything below.

Q1: Expanding experiments to support claims of FTFC to work with general class of divergencies and utility functions. We conducted additional experiments along both requested directions: non-KL divergences and structured, distribution-dependent utilities beyond CVaR. We evaluated the resulting generated samples, including their utility, validity and deviations from the optimized finite-bank targets.

1. Non-KL divergences on GEOM-Drugs We compared forward KL, half-Pearson χ² and reverse KL while holding the acquisition banks, backbone, adapter, native fitting loss, update budget and evaluation protocol fixed. The corresponding convex generators are f(t)=tlog⁡t−t+1f(t)=t\log t-t+1, f(t)=(t−1)2/2f(t)=(t-1)^2/2 and f(t)=−log⁡t+t−1f(t)=-\log t+t-1. Thus, the comparison changes the regularizer used to construct the target weights within the same FTFC fitting framework. The matched comparison uses 10,000 raw samples per bank, three independently acquired banks and two fit seeds per bank, with CVaR parameter β=0.9\beta=0.9 and regularization α=1/6\alpha=1/6 on the raw reward scale.

Method / divergenceQED ↑Mean energy reward, Ha ↑Upper-10% energy CVaR, Ha ↑Validity ↑Geometry score ↑
Frozen reference0.6464.393.00.970.90
FTFC / forward KL0.6768.694.71.00.95
FTFC / half-Pearson χ²0.6868.695.61.00.96
FTFC / reverse KL0.6768.194.50.990.95

Both non-KL variants complete the same fitting and generated-evaluation pipeline. Half-Pearson has the highest full-bank FTFC QED and tail-reward point estimates

2. Structured utilities beyond CVaR on GEOM-Drugs

We additionally evaluated three objectives using forward-KL regularization with α=0.1\alpha=0.1:

  • Descriptor moment matching: Umoment(q)=Eq[QED]−(Eq[z])2U_{\mathrm{moment}}(q)=\mathbb{E}_q[\mathrm{QED}]-(\mathbb{E}_q[z])^2, where zz is standardized logP. This balances drug-likeness against deviation of the mean descriptor from its reference value.
  • Descriptor coverage: Ucoverage(q)=Eq[QED]+0.2H(pq)U_{\mathrm{coverage}}(q)=\mathbb{E}_q[\mathrm{QED}]+0.2H(p_q), where pqp_q is the distribution over four fixed logP bins. This balances drug-likeness and descriptor coverage.
  • Moment matching with composition constraints: the moment utility above, with 11 additional finite-target equalities preserving expected counts of ten elements and total formal charge relative to the bank reference. | Training objective | QED ↑ | Moment utility ↑ | Coverage utility ↑ | Validity ↑ | Geometry score ↑ | |---|---:|---:|---:|---:|---:| | Frozen reference | 0.66 | 0.62 | 0.93 | 1.0| 0.94 | | FTFC / moment matching | 0.70 | 0.68| 0.97 | 0.97 | 0.94 | | FTFC / coverage entropy | 0.71 | 0.70 | 0.98 | 1.0 | 0.95 | | FTFC / moment + composition constraints | 0.68 | 0.68 | 0.96 | 0.99 | 0.95 |

The table reports average across 55 different running seeds. The choice of utility indeed can drastically improve performance and this experiment showcases robustness of the FTFC under diverse utility objectives.

Q2: Fairness of the TFFT comparison, image seeds and uncertainty, checkpoint selection, and acquisition-inclusive efficiency

1. Learning signal in the reconstructed TFFT baseline. We audited this concern and verified the thresholds and reward scaling. TFFT’s original right-tail objective uses the hinge pseudo-reward r~(x)=[r(x)−t∗]+/(1−β)\widetilde r(x)=[r(x)-t^*]_+/(1-\beta), whose reward gradient is zero below the threshold by construction (TFFT, Section 5).

In our runs, the optimized threshold leaves only one of 9,988 eligible offline rewards active. None of the 952 requested training samples per run exceeded its scheduled threshold, explaining the observed zero tail-reward gradients. The zero-gradient problem is innate to the TFFT algorithm. Crucially, FTFC avoids this particular activation gate: it constructs target weights from the offline bank and trains through a weighted native flow/denoising loss. Its parameter gradients come from prediction errors, while tail preferences enter through the weights. Training therefore does not require fresh online samples to exceed a reward threshold before receiving a learning signal. This removes the specific hinge-gradient bottleneck observed in TFFT

Q3. Does the efficiency claim include sample acquisition?

We distinguish fitting with a cached bank from acquisition-inclusive adaptation cost. Charging FTFC for generating and scoring its sample bank gives the following comparison:

Benchmark / hardwareFDC (hours)TFFT (hours)FTFC incl. acquisition (hours)FTFC speedup vs FDCFTFC speedup vs TFFT
QM935.2315.000.9736.20×15.42×
GEOM-Drugs16.825.543.484.83×1.59×
Stable Diffusion82.4185.7747.761.73×1.80×

Even with acquisition stage, FTFC still faster.

Q3: Image evaluation seeds and selection

We completed five acquisition/value-fit replications, each using ten independently acquired banks. These fit conditional-value controllers over a frozen generator. The recipe was locked before fresh acquisition labels or confirmation results were inspected. We retained all five final fits, with no confirmation-based checkpoint, recipe or seed selection. Training, validation, confirmation and development contain 256/100/84/16256/100/84/16 distinct prompt strings, respectively. Generation streams are shared across methods for paired comparisons.

Moreover, we distinguish sample SD across the five fits, SE = SD/5\sqrt{5}, and prompt-only, fit-only and crossed paired intervals. The latter use 1010k bootstrap replicates, resampling whole fits and prompts while retaining dependent outputs together and counting each shared control once. CVaR is computed within each fitted model before averaging. The completed primary confirmation panel uses the 84 held-out prompts, three four-output blocks per prompt. For this new experiment setting, we obtained the following scores:

MethodImageReward ↑Lower CVaR(.2) ↑DreamSim marginal diversity ↑
Frozen DDIM500.21−1.230.30
Frozen SDE320.19−1.220.34
FTFC (5-fit mean, final checkpoints, 84 held-out prompts)0.55−0.680.42
FTFC (64 training prompts, 64 new eval prompts)0.44-0.860.39
FTFC (128 training prompts, 64 new eval prompts)0.51-0.740.41
FTFC (256 training prompts, 64 new eval prompts)0.54-0.690.41

We also kindly refer the reviewer to our response to the reviewer TAdG (question 3. Image sample-count ablation and checkpoint selection.)

Thank you for your detailed response to my questions and they really help me to better understand the contribution of your method, especially the new evaluations with non-KL divergences and structured utilities. However, my remaining concern is the fairness of the TFFT comparison, since the inactive-gradient issue may be specific to the reconstructed implementation rather than a general limitation of TFFT. Moreover, the new image experiments do not fully establish superiority over the stronger baselines under the revised evaluation protocol.

We thank reviewer for providing feedback and positively evaluating our detailed responses.

For image experiments, we agree that revised evaluation protocol does not establish stronger results on ImageReward metric. However, across tested benchmarks (GEOM Drugs and QM9), the image generation finetuning task is the only one on which FTFC underperforms (but the margin is not large). Despite this, FTFC greatly outperforms in terms of speed (training time), even with bank collection:

Historical method/componentRecorded time, hours
EXP-FT: accumulated training-loop time82.278
FDC: accumulated training-loop time82.406
L-TFFT: accumulated training-loop time85.769
FTFC: fitting along the selected checkpoint lineage4.045
FTFC: reported bank generation and scoring43.711
FTFC: sum of those fitting and bank components47.756

However, we strongly believe that quality for image generation experiments could be greatly improved. There are no obstacles from theoretical viewpoint which could prevent FTFC from outperforming baselines. Additional sweep across hyperparameters should be performed in order to establish critical / sensitive parameter for high-dimensional image generation. We will investigate this.

Comments on the paper

2 comments
Area_Chair_RvXpOpenReview ↗

Reviewer-author discussion kickoff

Thank you for your review. The authors have now provided their rebuttal. We would appreciate it if you could take some time to read their responses and engage in the discussion, particularly on whether your concerns have been adequately addressed.

Thank you for your time and contributions to the review process!

AuthorsOpenReview ↗

Response to the main concerns raised during review

We thank the Reviewers (w82p, Bt9R, TAdG) for their constructive feedback and for recognizing the novelty, mathematical clarity, and efficiency of our framework. Below is a summary of our core contributions and the new experiments added in the revision to address all shared concerns:

1. Novel Dual Formulation for General Generative Fine-Tuning Existing methods either restrict fine-tuning to specific preferences (e.g expected reward or KL-regularized CVaR in TFFT) or require expensive alternating outer-loop linearization and trajectory backpropagation (like FDC). FTFC resolves this trade-off via a novel two-stage Fenchel dual formulation (Theorem 1, Eq. 7–9). This is achieved by the decomposition to the following stages:

  • Joint Reward & Weight Calibration (Stage 1): For nonlinear utilities F(p)\mathcal{F}(p), the marginal reward depends on the unknown optimal target distribution p∗p^\ast. Using Fenchel duality, FTFC jointly recovers the exact supporting marginal reward g∗g^\ast and normalized density-ratio weights w∗=dp∗/dq∈∂f+∗((g∗−ν∗)/α)w^\ast = dp^\ast/dq \in \partial f_+^\ast ((g^\ast - \nu^\ast)/\alpha) directly on a cached sample bank, properly enforcing normalization inside the nonlinear conjugate response for general ff-divergences.

  • Single-Stage Native Fitting (Stage 2): Freezing w∗w^\ast reduces generator adaptation to a single stage of importance-weighted flow matching or denoising. This requires no reward gradients, no trajectory backpropagation, and no outer mirror-descent loops. Moreover, because parameter gradients come from native prediction errors weighted by w∗w^\ast, FTFC avoids the zero-gradient threshold starvation that occurs in hinge-based methods like TFFT.

2. Substantial Speedups (including bank acquisition) (w82p, Bt9R, TAdG) Decoupling utility optimization from generator training yields major computational speedups even when charging upfront sample-bank generation and scoring to FTFC (Table 7):

  • QM9: 0.97 GPU-hours for FTFC vs. 15.00 (TFFT) and 35.23 (FDC) — 15.4×15.4\times to 36.2×36.2\times faster while achieving the highest mean (29.14 Ha) and tail rewards (39.40 Ha).
  • GEOM-Drugs: 3.48 GPU-hours for FTFC vs. 5.54 (R-TFFT) and 16.82 (FDC) — 1.6×1.6\times to 4.8×4.8\times faster. At FTFC's 3.5 GPU-hour budget, FDC cannot even complete a single outer stage (~5.6 GPU-hours/stage), and our bank-size ablation shows that a N=1,024N=1,024 bank cuts total FTFC time to <0.4<0.4 GPU-hours without loss in QED (0.69).
  • Stable Diffusion: 47.76 GPU-hours (only 4.05 hours of actual model fitting + 43.71 hours of bank collection) vs. 82.41-85.77 GPU-hours for baselines (~1.8×1.8\times faster end-to-end, and >20×>20\times faster in fitting time).

3. Generality Validated Beyond KL and CVaR (w82p, Bt9R, TAdG) To support our generality claims on varying utility components, we added two controlled studies on GEOM-Drugs (Section 5.5, Appendix A.1):

  • Non-KL ff-Divergences: Comparing Forward KL, Half-Pearson χ2\chi^2, and Reverse KL under identical banks and budgets shows that all three improve QED (0.64→0.67–0.680.64 \rightarrow 0.67\text{–}0.68), validity (0.97→0.99–1.000.97 \rightarrow 0.99\text{–}1.00), and upper-tail energy (93.0→94.5–95.693.0 \rightarrow 94.5\text{–}95.6 Ha) over the frozen model, with Half-Pearson χ2\chi^2 (which allows exact zero weights) achieving the highest tail energy (95.6 Ha).

  • Distribution-Dependent Utilities: Optimizing QED combined with logP moment matching, four-bin logP coverage entropy, and 11 elemental/charge composition constraints consistently improves each target utility over the frozen reference (0.62→0.690.62 \rightarrow 0.69, 0.93→0.980.93 \rightarrow 0.98, and 0.63→0.680.63 \rightarrow 0.68).

4. Bank Sensitivity, and Audited Evaluations (w82p, Bt9R, TAdG) We separated finite-bank dual optimality, statistical bank coverage, and generator realization (Section 4, Appendices A.2 & C.3), supported by a nested bank-size ablation (N∈{256,1024,4096,10000}N \in \{256, 1024, 4096, 10000\}) on GEOM-Drugs and a 458,752-sample extreme-tail (β=0.998\beta=0.998) evaluation on QM9.

We hope these clarifications and additional experiments fully address all remaining questions, and we thank the reviewers for constructive suggestions, which have significantly strengthened our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.