- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes semantic noising: condition a generator on a Gaussian-corrupted semantic embedding . By Bayes' rule, the posterior is an exponential tilt by a linear-quadratic reward in semantic space. So one model can serve a whole reward family, with the reward chosen after training. The paper gives two ways to realize this:
- Direct conditional generation. A generator is trained on .
- Analytic tilting with semantic diffusion. The paper uses the known natural-parameter shift . It applies this shift either (a) to a joint image+embedding diffusion model trained with independent noise levels, or (b) to a factorized prior+decoder model (unCLIP/Kandinsky), with no retraining.
The validation has three parts:
- MolGPT on CLAMP embeddings. Post-hoc linear property predictors control six properties individually and jointly. The method gives a better property–KL trade-off than separately trained property-conditioned MolGPTs.
- REG-style SiT-XL/2 on ImageNet 256 with an extra CLIP token. FID is 2.319 vs. 2.2 for REG. The model supports prompt-based editing via PF-ODE inversion and aesthetic-score tilting.
- Kandinsky 2.2. A qualitative tilt of the prior toward a reference image is shown.
Critical strengths
-
A clean, unifying, amortized view of reward tilting. Reading semantic conditioning through the lens of an exponential family is elegant and practically useful. The choice of defines the reward family, and indexes a member of it.
- It covers autoregressive, joint-diffusion and factorized models in one framework.
- It needs no reward gradients and no extra denoiser evaluations.
- The overhead is small: +0.5% parameters and +0.78% inference cost (App. D.3).
This is the main reason the paper is worth publishing. It turns "one fine-tune per reward" into a single trained model.
-
Convincing molecular evidence. The generator never sees property labels. Even so, independently fitted linear predictors give monotone control over six properties (Fig. 2, 9) with about 97% validity, and composable multi-property control works (Fig. 4). Fig. 3 uses exact autoregressive likelihoods for KL, so the reward–KL comparison is quantitative rather than anecdotal.
Critical weaknesses
-
The paper never checks that samples actually follow , and the learned model is clearly not faithful off-distribution.
- The identities in Eq. 3, 7 and 8 hold for the exact posterior or denoiser. Tilting, however, evaluates the network at shifted inputs , which may have low density under the training marginal of .
- Fig. 11 shows this directly. For normalized CLIP embeddings, define the same target distribution, yet they produce qualitatively different images; at the image even has a different subject.
- appears to be chosen by inspection, so the method has a hidden, sensitive hyperparameter with no selection rule.
- Nowhere does the paper measure the gap between the sampled distribution and the true tilt. In the molecular setting this is cheap to do: self-normalized importance sampling of the base model with weights gives reference values for and .
Without this check, "tilting" is shown only as "moves the property in the right direction", which is a much weaker claim than the title makes.
-
Weak baselines and limited novelty on the diffusion side.
- Novelty. The denoiser-shift identity is attributed to prior work (Liu 2024; Moitra et al. 2026; McDonald 2026; Holderrieth et al. 2026). The "small change" of sampling independent per-coordinate noise levels is essentially UniDiffuser's training scheme (Bao et al., 2023). The incremental contribution here is therefore mainly the choice of coordinate (a CLIP token added to REG).
- Image experiments have no baselines. There is no comparison with gradient-based guidance using the same linear aesthetic predictor, DPS/universal guidance, Best-of-, SMC/FK correctors, or SDEdit-style editing.
- Costs are not measured. The paper reports no realism or diversity cost as a function of (e.g., FID or precision/recall vs. ). For editing, it reports no source-fidelity measure (e.g., LPIPS) alongside CLIP similarity.
- Fig. 3 compares unlike objectives. Target-value conditioning is not designed to be KL-optimal for a given expected reward, so beating it is partly expected. Best-of- (with its known KL bound) or importance-weighted base sampling would be the natural reference.
- The Kandinsky result is a single qualitative example compared against interpolation baselines.
Questions for authors
- Fidelity of the tilt (MolGPT). Can you compare model samples at several values of against a SNIS estimate of from a large base-model sample? Please report the property mean and , ideally as a function of , to show where the amortization breaks down.
- Coverage of inference queries. How do the queries compare to the training marginal of , in both norm and direction? Table 1 reports only norms.
- Choosing and . For normalized embeddings, and are equivalent. Is there a principled rule for picking the representative, for example the one maximizing the training density of the shifted input? How was selected, and does the choice transfer across rewards and ?
- Classifier-free guidance in Kandinsky. The Kandinsky prior is typically sampled with CFG, whose combined output is not a conditional mean. Was the shift applied to each branch separately, and was CFG used in Fig. 8?
- Relation to UniDiffuser. Beyond the choice of coordinate, what distinguishes the joint-diffusion training from UniDiffuser's independent per-modality timesteps?
Suggestions (not required for acceptance):
- Add guidance and Best-of- baselines plus FID/diversity-vs- curves for aesthetic tilting, and a fidelity–alignment frontier for editing.
- Make the naming consistent: the aesthetic predictor is called "LAION" in Sec. 4.2 and "AVA" in App. D.
Reviewer-author discussion kickoff
Dear Reviewers,
Thank you for your review. The authors have now provided their rebuttal. We would appreciate it if you could take some time to read their responses and engage in the discussion, particularly on whether your concerns have been adequately addressed.
Thank you for your time and contributions to the review process!