acceptodds
Under review as a conference paper at ICLR 2027

Reward Tilting via Semantic Noising

Abstract

Steering a generative model toward a desired property typically requires separate fine-tuning for each reward or additional computation during sampling. We propose *semantic noising* which conditions generation on a noisy semantic representation of the data. The resulting posteriors correspond to exponential tilts by linear-quadratic rewards in semantic space, allowing a single model to support objectives and strengths chosen after training. A conditional generator can be trained to approximate these posteriors directly, while models that generate semantic representations by diffusion provide analytic tilted denoisers for the same family of rewards. We illustrate semantic noising in three settings. First, we train MolGPT, an autoregressive molecular generator, to condition on noisy semantic representations without using molecular property values, and use independently fitted linear predictors to control properties. Our model achieves a better reward–KL trade-off than separately trained property-conditioned models. Second, our joint diffusion model of images and their CLIP embeddings achieves FID comparable to REG on ImageNet while enabling prompt-based editing and adjustable aesthetic-score tilting. Finally, we show that pretrained factorized models such as Kandinsky support analytic tilting without retraining or additional denoiser evaluations.

Primary area:
generative models
Keywords:
diffusion models, reward tilting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_idhTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes semantic noising: condition a generator on a Gaussian-corrupted semantic embedding z~=τzϕ(x)+τz ε\tilde{\mathbf z} = \tau_z \phi(\mathbf x) + \sqrt{\tau_z}\,\varepsilon. By Bayes' rule, the posterior q(x∣z~=a,τz=γ)∝q(x)exp⁡(a⋅ϕ(x)−γ2∥ϕ(x)∥2)q(\mathbf x \mid \tilde{\mathbf z}=\mathbf a, \tau_z=\gamma) \propto q(\mathbf x)\exp(\mathbf a\cdot\phi(\mathbf x) - \tfrac{\gamma}{2}\|\phi(\mathbf x)\|^2) is an exponential tilt by a linear-quadratic reward in semantic space. So one model can serve a whole reward family, with the reward chosen after training. The paper gives two ways to realize this:

  1. Direct conditional generation. A generator is trained on (z~,τz)→x(\tilde{\mathbf z}, \tau_z) \to \mathbf x.
  2. Analytic tilting with semantic diffusion. The paper uses the known natural-parameter shift Dr(x~,τ)=D(x~+a,τ+γ)D^r(\tilde{\mathbf x},\tau) = D(\tilde{\mathbf x}+\mathbf a, \tau+\gamma). It applies this shift either (a) to a joint image+embedding diffusion model trained with independent noise levels, or (b) to a factorized prior+decoder model (unCLIP/Kandinsky), with no retraining.

The validation has three parts:

  • MolGPT on CLAMP embeddings. Post-hoc linear property predictors control six properties individually and jointly. The method gives a better property–KL trade-off than separately trained property-conditioned MolGPTs.
  • REG-style SiT-XL/2 on ImageNet 256 with an extra CLIP token. FID is 2.319 vs. 2.2 for REG. The model supports prompt-based editing via PF-ODE inversion and aesthetic-score tilting.
  • Kandinsky 2.2. A qualitative tilt of the prior toward a reference image is shown.
Critical strengths
  1. A clean, unifying, amortized view of reward tilting. Reading semantic conditioning through the lens of an exponential family is elegant and practically useful. The choice of ϕ\phi defines the reward family, and (z~,τz)(\tilde{\mathbf z}, \tau_z) indexes a member of it.

    • It covers autoregressive, joint-diffusion and factorized models in one framework.
    • It needs no reward gradients and no extra denoiser evaluations.
    • The overhead is small: +0.5% parameters and +0.78% inference cost (App. D.3).

    This is the main reason the paper is worth publishing. It turns "one fine-tune per reward" into a single trained model.

  2. Convincing molecular evidence. The generator never sees property labels. Even so, independently fitted linear predictors give monotone control over six properties (Fig. 2, 9) with about 97% validity, and composable multi-property control works (Fig. 4). Fig. 3 uses exact autoregressive likelihoods for KL, so the reward–KL comparison is quantitative rather than anecdotal.

Critical weaknesses
  1. The paper never checks that samples actually follow qrq^r, and the learned model is clearly not faithful off-distribution.

    • The identities in Eq. 3, 7 and 8 hold for the exact posterior or denoiser. Tilting, however, evaluates the network at shifted inputs (z~+a,τz+γ)(\tilde{\mathbf z}+\mathbf a, \tau_z+\gamma), which may have low density under the training marginal of (z~,τz)(\tilde{\mathbf z}, \tau_z).
    • Fig. 11 shows this directly. For normalized CLIP embeddings, γ∈{0,0.4,1}\gamma \in \{0, 0.4, 1\} define the same target distribution, yet they produce qualitatively different images; at γ=1\gamma=1 the image even has a different subject.
    • γ=0.01\gamma=0.01 appears to be chosen by inspection, so the method has a hidden, sensitive hyperparameter with no selection rule.
    • Nowhere does the paper measure the gap between the sampled distribution and the true tilt. In the molecular setting this is cheap to do: self-normalized importance sampling of the base model with weights exp⁡r\exp r gives reference values for Eqr[R^]\mathbb E_{q^r}[\hat R] and DKL(qr∥q)D_{\mathrm{KL}}(q^r\|q).

    Without this check, "tilting" is shown only as "moves the property in the right direction", which is a much weaker claim than the title makes.

  2. Weak baselines and limited novelty on the diffusion side.

    • Novelty. The denoiser-shift identity is attributed to prior work (Liu 2024; Moitra et al. 2026; McDonald 2026; Holderrieth et al. 2026). The "small change" of sampling independent per-coordinate noise levels is essentially UniDiffuser's training scheme (Bao et al., 2023). The incremental contribution here is therefore mainly the choice of coordinate (a CLIP token added to REG).
    • Image experiments have no baselines. There is no comparison with gradient-based guidance using the same linear aesthetic predictor, DPS/universal guidance, Best-of-NN, SMC/FK correctors, or SDEdit-style editing.
    • Costs are not measured. The paper reports no realism or diversity cost as a function of λ\lambda (e.g., FID or precision/recall vs. λ\lambda). For editing, it reports no source-fidelity measure (e.g., LPIPS) alongside CLIP similarity.
    • Fig. 3 compares unlike objectives. Target-value conditioning is not designed to be KL-optimal for a given expected reward, so beating it is partly expected. Best-of-NN (with its known KL bound) or importance-weighted base sampling would be the natural reference.
    • The Kandinsky result is a single qualitative example compared against interpolation baselines.
Questions for authors
  1. Fidelity of the tilt (MolGPT). Can you compare model samples at several values of cc against a SNIS estimate of qrq^r from a large base-model sample? Please report the property mean and DKLD_{\mathrm{KL}}, ideally as a function of ∥a∥\|\mathbf a\|, to show where the amortization breaks down.
  2. Coverage of inference queries. How do the queries z~=cw/sR^\tilde{\mathbf z} = c\mathbf w/s_{\hat R} compare to the training marginal of z~\tilde{\mathbf z}, in both norm and direction? Table 1 reports only norms.
  3. Choosing γ\gamma and τz\tau_z. For normalized embeddings, (z~,τ)(\tilde{\mathbf z}, \tau) and (z~,τ+δ)(\tilde{\mathbf z}, \tau+\delta) are equivalent. Is there a principled rule for picking the representative, for example the one maximizing the training density of the shifted input? How was γ=0.01\gamma=0.01 selected, and does the choice transfer across rewards and λ\lambda?
  4. Classifier-free guidance in Kandinsky. The Kandinsky prior is typically sampled with CFG, whose combined output is not a conditional mean. Was the shift applied to each branch separately, and was CFG used in Fig. 8?
  5. Relation to UniDiffuser. Beyond the choice of coordinate, what distinguishes the joint-diffusion training from UniDiffuser's independent per-modality timesteps?

Suggestions (not required for acceptance):

  • Add guidance and Best-of-NN baselines plus FID/diversity-vs-λ\lambda curves for aesthetic tilting, and a fidelity–alignment frontier for editing.
  • Make the naming consistent: the aesthetic predictor is called "LAION" in Sec. 4.2 and "AVA" in App. D.
Reviewer_a8LBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes semantic noising to steer generative models by "conditioning" the right noisy version of the semantic embedding + right time of the semantic conditioning. That is the gen. model needs to also learn the probability of the semantic embedding. 

It connects the literature which shows that conditioning on noisy embeddings is exponential reward tilting (i.e., guidance) , allowing reward objectives and strengths to be chosen after training without separate fine-tuning for each reward, as long as the reward can be expressed as a linear-qudratic function of the semantic embedding / input to the gen model.

The experiments cover molecular generation with MolGPT, joint image–semantic diffusion on ImageNet, and steering the semantic prior of pretrained Kandinsky 2.2. The paper reports improved reward–KL trade-offs over direct molecular-property conditioning. On ImageNet, the paper demonstrates prompt-based editing, aesthetic-score control, and reference-image guidance. 

I find it interesting that this work adopts a theoretical finding and makes it work in practice with convincing results, especially in the aesthetic-score guidance and MolGPT experiments, although I lack the expertise to judge the latter. I find these to be appropriate use cases for this method. However, I have reservations about other applications, such as text-based image editing, for which more straightforward methods (and potentially better) exist.

Critical strengths
  • Nice connection between theory and practical reward control. The paper translates a theoretical connection between noising and exponential reward tilting into a simple way to choose reward objectives and strengths after training. Of course, this is limited to the linear–quadratic reward family and requires compatible diffusion models.

  • Aesthetic-score control and molecular generation seem well suited to this method. These applications seem to have well-defined linear reward functions, and both seem to benefit from post-hoc guidance rather than conditional training from the start. The molecular results also look promising, with reported improvements in reward–KL trade-offs, although I lack the domain expertise to judge their practical significance. Comparisons with alternative guidance methods remain necessary to establish the relative performance of aesthetic-score control (see Weaknesses).

Critical weaknesses
  • Lack of comparative evaluation for image reward control. The aesthetic-score and prompt-based editing results look visually convincing, but they lack a favorable comparison against alternative guidance methods. A comparison using the same model and reward, with universal guidance or a DPS-style adaptation (where guidance is applied in image space), would help me get a sense of the gains over similar but older post-hoc methods. This is my reservation and might tip my recommendation toward borderline or weak rejection.

  • The method’s usefulness depends on how well the desired objective can be expressed as a linear reward on semantic embeddings. Aesthetic-score control is a convincing use case, and molecular generation seems promising, although I lack the expertise to judge its practical significance. I see the strongest potential in properties that are well captured by such rewards but difficult to specify through text. For text-based generation and editing, I am less convinced of the advantage over direct text conditioning, particularly when complex instructions must be captured through a single CLIP embedding. A clearer account of where this approach offers useful control beyond existing conditioning methods would strengthen the paper.

Questions for authors
  1. Comparison with inference-time guidance. Could you compare semantic tilting with gradient-based guidance, such as universal guidance or a DPS-style adaptation, using the same image model and reward? For aesthetic control, I would like to see the trade-off between reward, image quality, and sampling time; for editing, between prompt alignment and source-image preservation.

  2. Practical scope and advantage over direct conditioning. Could you clarify which objectives are well captured by linear rewards on semantic embeddings and benefit from semantic tilting beyond existing conditioning methods? I'm looking for concrete examples, or properties that are difficult to specify through text. I think a strong use-cases will strengthen the papers (and the paper will naturally benefit from more experiments along that line). The image-editing, to me, is not one of them.

Reviewer_yadyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Semantic Noising for exponential reward-tilting, which takes a semantic representation of the data (e.g., via a pretrained encoder or a fixed feature map) and adds Gaussian noise to corrupt it. The family of rewards that semantic noising supports are linear-quadratic functions of a fixed representation. When the representation's norm is constant, the quadratic term is also constant. The authors implement the framework through (1) direct conditional generation (2) joint diffusion of data and representations and (3) a modification of a pretrained semantic diffusion prior. Experiments apply semantic noising to molecular and image domains: post-training molecular property control on MolGPT, joint image-CLIP diffusion for ImageNet modeling (FID comparable to REG) while enabling image editing and aesthetic score control, and analytic tilting with Kandinsky without retraining.

Critical strengths
  1. Reusability. Semantic noising enables reusable reward control without needing to specify the downstream rewards during generator training. This allows a single model to support objectives and strengths chosen after training through independently fitted linear predictors, so labels for rewards are not required at training time.
  2. Empirical breadth. The paper demonstrates semantic noising on real-world autoregressive(MolGPT) and diffusion(image) models. Hence, the framework is applicable beyond a single architecture or generative framework.
Critical weaknesses
  1. My main concern is that the novelty/contribution of semantic noising relative to established prior methods is insufficiently separated. Many components of the framework, such as conditional training, pretrained representations, joint (image, feature) diffusion are standard. Also, several predecessors are cited, but I am not fully convinced that the paper establishes a sufficiently novel methodological contribution beyond applying these components.
  2. Comparisons against existing steering methods are not sufficiently evaluated. It is not established when the proposed semantic noising approach is preferable to existing inference-time steering methods that reuse a frozen generator. For example, the molecular reward KL only compares against one baseline type (direct property conditioning). A comparison against an appropriate inference-time steering method would show practicality of semantic noising.
Questions for authors
  1. Could the authors distinguish in more detail the novelty and contribution of the framework relative to existing joint representation diffusion models or analytic posterior denoiser transformations?
  2. Could the authors provide reward attainment/cost comparisons not just against direct property conditioning, but against appropriate inference-time steering baselines?
  3. What are the computational costs for the total generator adaptation plus predictor fitting? How many rewards or samples would make the proposed method more efficient than inference-time steering?

Comments on the paper

1 comment
Area_Chair_zwbJOpenReview ↗

Reviewer-author discussion kickoff

Dear Reviewers,

Thank you for your review. The authors have now provided their rebuttal. We would appreciate it if you could take some time to read their responses and engage in the discussion, particularly on whether your concerns have been adequately addressed.

Thank you for your time and contributions to the review process!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.