- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies concept dominance in compositional diffusion generation, which it attributes to mode overlap between the joint distribution and the single-concept distributions . The authors argue that corrector-based methods such as CO3 take this ratio at intermediate timesteps, so they do not actually target the pure-mode distribution defined at .
The proposed method, TILT, instead defines pure-mode sampling as a reward at , , and solves a KL-regularized reward alignment problem. The reward is estimated from diffusion classifier energies, so no external reward model is needed. Sampling uses DPS-style guidance on the Tweedie estimate , applied only in the early denoising steps.
Experiments on SDXL (GenEval, T2I-CompBench) and AudioLDM (AudioCaps) show gains over CFG and prior compositional methods. Plugged into FKS or SCORE in place of CLIP or CLAP, the reward is competitive with these external rewards.
Critical strengths
-
The paper identifies a real problem. Corrector-based methods such as CO3 take the ratio of noised distributions at each intermediate step, while the mode overlap they aim to fix actually lives at . Noising and then dividing is not the same as dividing and then noising, so these methods do not actually sample from the pure-mode distribution they target. The observation is simple, but I think it is correct, and it is the most valuable part of the paper.
-
The reward and the sampling derivation are clear. The authors first define the target in data space, then use the KL-regularized formulation and posterior sampling to derive guidance at intermediate steps. Because the reward is a likelihood ratio, the intractable terms cancel under the ELBO approximation, and no external reward model is needed. The argument is easy to follow, and it is clear where each approximation enters.
Critical weaknesses
-
The step from the target distribution to the actual algorithm relies on two approximations, and I don't think their assumptions hold here. Eq. 17 follows DPS and replaces the posterior expectation in with a point estimate at . This works reasonably in DPS when the likelihood is smooth and the posterior is concentrated, but here the likelihood is with built from UNet denoising errors, and there is no reason to expect it to be smooth. Eq. 21 then follows MPGD and drops the Tweedie Jacobian, updating directly. MPGD's argument depends on the gradient lying in the tangent space of the data manifold, and even then it only says samples stay on the manifold, not that they follow the target distribution. The gradient here comes from a UNet-based energy, and the authors never check whether it satisfies this. What makes it worse is that guidance is only applied in the first 5 steps, at high noise, where is basically a blurry mean image and both approximations are at their weakest. The central argument of the paper is that CO3 fails to sample from the pure-mode distribution while TILT does it properly. After these approximations, though, it is just as unclear what distribution TILT ends up sampling from, so I don't find this contrast convincing.
-
The image experiments use only SDXL, which is fairly old at this point, and a lot of its compositional failures come from its CLIP text encoders. Newer models like FLUX or Qwen-Image use much stronger text encoders and handle composition noticeably better, so I'm not sure the problem is as severe there, or that TILT would still help. These models are also mostly flow-matching based, and some, such as FLUX.1-dev, are guidance-distilled, so it's not obvious the -based energy carries over directly. Without results on at least one recent model, it's hard to judge how much the method matters in practice today.
Questions for authors
-
What happens if you keep the Jacobian, i.e., use , or precondition with some approximation of the posterior covariance, instead of the direct update in Eq. 21? Since computing already requires backpropagating through the UNet, one extra VJP shouldn't add much cost. If the results are similar, the approximation is probably harmless. If they differ noticeably, the gap between the derivation and what is actually run is larger than the paper suggests. (Experiment request)
-
Why is guidance applied only in the first 5 steps? Did you try guiding at later steps, and what happened? Is this choice motivated by the derivation, or is it purely empirical? (Clarification)
-
Does the final algorithm actually approximate ? A low-dimensional example like the one in Fig. 1, where can be computed exactly, would make this easy to check: compare samples from TILT and from the CO3 corrector against the true . I think this is the most direct evidence for the paper's main claim that TILT targets the right distribution while correctors do not. (Experiment request)
-
When computing , do you backpropagate through the UNet? If not, and you use an SDS-style approximation instead, the update reduces to roughly , which is essentially guidance with the sub-concepts used as negative prompts. Could you discuss how TILT relates to this kind of negative-prompt guidance, and ideally add a simple baseline that uses the sub-concepts as negative prompts directly? That would show whether TILT gives anything beyond this simpler approach. (Clarification and experiment request)
Overall, I like the idea behind this paper, and my current rating of weak reject mainly reflects the practical issues above rather than doubts about the approach itself. If the authors can clearly address the questions and the weaknesses I raised, in particular the two approximations and the lack of results on more recent models, and the results support the method, I would be happy to raise my score to weak accept or higher.
6 comments
W2: The image experiments use only SDXL. it's not obvious the -based energy carries over to rectified flow & guidance distilled models directly.
In any Rectified flow model,
we can represent the noise as . Since this relation is linear, we get . So the current diffusion energy expression in TILT can be converted to a velocity-based energy and an equivalent energy based reward can be used for Rectified-flow-based models.
In this short time, we rapidly implemented a small-scale comparison to see the ballpark results. What we have is not conclusive and significant. Nonetheless, we see TILT outperforms the base model in both metrics. We feel there is room to extract out more performance. So if the paper gets accepted, we will make an earnest effort to build out on these preliminary experiments (including additional comparisons with baselines on more complex prompts).
Table: Quantitative results on GenEval prompts using SD3 as the backbone.
| Method | T2V | HPSv3 |
|---|---|---|
| CFG | 0.778 ± 0.03 | 7.21 ± 0.121 |
| TILT | 0.793 ± 0.02 | 7.51 ± 0.12 |
Q1: What happens if you keep the Jacobian?
Keeping the Jacobian gives comparable results on GenEval and T2I-CompBench (see the tables in our response to W1/Q3 above): the Jacobian-free formulation remains competitive or slightly better, indicating that the approximation does not degrade performance, while preserving the inference speed-up by avoiding the costly JVP operation for extra gradient propagation.
Q2: Why is guidance applied only in the first 5 steps?
This is an empirical observation from our evaluation results. The table below shows TILT's performance over different correction-steps and performance drops beyond across most categories. We observe that gradient from the reward becomes stronger and sharper as diffusion denoising moves toward the clean sample, which leads to over-correction. We suspect this stems from the increasingly ill-conditioned behavior of the denoiser gradient at lower time steps, but we are not sure. We adopt an early stopping mechanism to preserve image quality.
Table: TILT's performance over different correction steps.
| # Correction Steps | Color T2V | Color HPSv3 | Shape T2V | Shape HPSv3 | Texture T2V | Texture HPSv3 | Complex T2V | Complex HPSv3 |
|---|---|---|---|---|---|---|---|---|
| 0 (CFG) | 0.5183 ± 0.015 | 5.1216 ± 0.10 | 0.3565 ± 0.012 | 6.1377 ± 0.09 | 0.5529 ± 0.014 | 6.5021 ± 0.08 | 0.6238 ± 0.015 | 7.5702 ± 0.10 |
| 5 | 0.5688 ± 0.015 | 5.8002 ± 0.10 | 0.4019 ± 0.013 | 7.0523 ± 0.08 | 0.5932 ± 0.014 | 7.2136 ± 0.08 | 0.6350 ± 0.015 | 8.2156 ± 0.10 |
| 10 | 0.5598 ± 0.014 | 5.7102 ± 0.102 | 0.3834 ± 0.011 | 6.9086 ± 0.07 | 0.5861 ± 0.014 | 7.1448 ± 0.079 | 0.6368 ± 0.015 | 8.0945 ± 0.10 |
It's true that the Tweedie and the Jacobian-free updates introduce approximations in the proposed algorithm. But we also want to highlight the following: while DPS's likelihood is smooth, it has a data consistency term which is much stricter than the soft likelihood based reward used in TILT. TILT only needs to encourage the intermediate latents to move towards higher likelihood regions under the target distribution, which is much weaker than DPS's data consistency term, thus less sensitive towards approximation in Tweedie.
As for the Jacobian-free update, we observe empirical gains as presented in the new ablations in the tables below. The with-Jacobian and without-Jacobian updates approaches are comparable, so dropping it saves some compute cost. We will include this ablation in the revised version of the paper.
Table: TILT with and without Jacobian update on GenEval prompts.
Table: TILT with and without Jacobian update on T2I-CompBench prompts.
The toy experiment is a remarkable idea; thank you. We setup a 2-D mixtute of 6 Gaussians to model p(x∣C), p(x∣c1), and p(x∣c2). Each distribution is of the form p(x∣c)∝∑kπkP(c∣k)N(x;μk,σk2I) and C= "c1 and c2" collides with each single concept. In this case, all scores can be computed analytically. We use CFG++ as a base sampler and compare TILT against CO3. Each method draws 8000 samples per seed over 10 seeds. We draw exact samples of pβ∗∝qeβRw by importance resampling 400k CFG++ samples. We then calculate the KL divergence DKL(p^∥pβ∗) between the estimated distribution p^ and the target pβ∗.
The table below reports the results: for both β=1 and β=2, TILT achieves the lowest KL divergence, while maitaining a competitive expected reward with CO3. Higher KL divergence and higher reward for CO3 indicate that CO3 sacrifices diversity by over-optimizing the reward (which is a form of reward hacking). This imbalance is likely to get far more pronounced in higher dimensional spaces. TILT better balances reward optimization with target distribution-fidelity compared to CO3. We are adding these additional figures, implementation details, and results in the Appendix (Sec A.7.1, page 26).
Table: Distance to the exact target and expected reward attained by the samplers.