acceptodds
Under review as a conference paper at ICLR 2027

Direct Trajectory Optimization for Inference-Time Alignment of Text-to-Image Diffusion Models

Abstract

Alignment of text-to-image diffusion models with certain reward functions is a key challenge in controllable generation, yet existing approaches often rely on optimizing model inputs or fine-tuning denoising networks, which requires backpropagation through large denoising models and can be computationally expensive at inference time. In this work, we propose Direct Trajectory Optimization (DTO), a training-free framework that performs alignment by directly optimizing the sampling trajectory of diffusion models, treating intermediate denoising scores and sampled noise as optimization target variables without requiring gradients through the denoising network. To mitigate out-of-distribution problem, we introduce Reward-Optimized Score Guidance, which interpolates between optimized scores and pretrained model predictions to stabilize sampling while theoretically provably preserving reward improvement generally. Empirically, DTO improves alignment performance across three diffusion models on two benchmarks, outperforming existing inference-time alignment methods while maintaining high efficiency. The results suggest that trajectory-level optimization is a promising direction for efficient and effective diffusion model alignment.

Primary area:
generative models
Keywords:
Text-to-Image, Alignment, Training-Free, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5xLqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Direct Trajectory Optimization (DTO), a training-free method for aligning text-to-image diffusion and flow models with a differentiable reward at inference time. For an ODE sampler, the final sample is a deterministic function of the initial noise xTx_T and the per-step score/velocity predictions {ϵt}\{\epsilon_t\}. DTO treats all of these as free variables and runs gradient ascent on the reward. Gradients are propagated only through the scheduler and the VAE decoder, not through the denoising network. To limit drift away from the pretrained model, each update is followed by a re-sampling pass, Reward-Optimized Score Guidance (ROSG), that interpolates the optimized scores with fresh denoiser predictions, ϵ^t=ϵt′+wt(ϵt−ϵt′)\hat\epsilon_t = \epsilon'_t + w_t(\epsilon_t - \epsilon'_t). The best trajectory by reward is kept. Theorem 1 gives a first-order lower bound on the reward after ROSG under a Lipschitz assumption.

Experiments use HPSv2 as the reward on the HPDv2 test set (400 prompts) and GenAIBench (1,600 prompts), with SD1.5, SDXL and SD3. The appendix adds SD3.5-Large-Turbo, FLUX.1-dev, an SDE variant, and CLIPScore/PickScore as alternative rewards. DTO reports much higher HPSv2 than DNO, DATE, ReNeg and Best-of-N, gains on held-out metrics (PickScore, CLIPScore, VQAScore, ImageReward), about 10× lower latency than DNO, paired t-tests, a wall-clock-matched Best-of-N comparison, ablations of the optimized variables and of the wtw_t schedule, and small human evaluations.

Critical strengths
  1. A strong efficiency–performance trade-off across many backbones. Avoiding backpropagation through the denoiser makes DTO about 25× faster than DNO on SD1.5 (16.7 s vs. 427 s per image) with lower memory (11 GB vs. 18 GB). It applies unchanged to UNet and DiT models, including few-step and flow-matching samplers (SD1.5, SDXL, SD3, SD3.5-Turbo, FLUX). Under a wall-clock-matched comparison on SD1.5 (App. Table 11), DTO also exceeds Best-of-N with N=20 on all held-out metrics. This makes it a practical option for reward-guided generation.

  2. A simple method with an informative ablation and released code. The method takes only a few lines on top of a standard sampler. The ablations (App. Tables 12, 14, 16) make the trade-off explicit: removing ROSG increases the optimized reward but degrades the held-out metrics. They also show which trajectory variables drive the gains.

Critical weaknesses

1. DTO is closer to surrogate-gradient noise/latent optimization than to "trajectory optimization," and the paper does not compare against that simpler interpretation.

For the samplers used (DDIM, flow-matching Euler), each update is affine in (xt,ϵt)(x_t, \epsilon_t), so x0x_0 is an affine function of (xT,ϵT,…,ϵ1)(x_T, \epsilon_T, \dots, \epsilon_1) with constant coefficients (App. Eq. 20). For flow Euler, ∂x0/∂xT=I\partial x_0/\partial x_T = I and ∂x0/∂ϵt=ΔσtI\partial x_0/\partial \epsilon_t = \Delta\sigma_t I. Every trajectory variable therefore receives the same image-space gradient ∂R/∂x0\partial R/\partial x_0, up to a scalar. DTO effectively ascends the latent with a "denoiser Jacobian = 0" surrogate gradient and then re-samples through ROSG.

The authors' own ablation supports this reading. Optimizing xTx_T alone recovers most of the gain (37.49 vs. 40.75 HPSv2 on SD1.5, App. Table 16), while optimizing only the second half of the trajectory barely helps (24.25 vs. 23.48 for the base model). The "richer optimization space" offered as the explanation for beating the direct x0x_0 ablation (Table 12: 34.06) is therefore not established. The difference may instead come from step size, from the optimizer (the code uses per-tensor Adam with gradient clipping and a decaying learning rate, not the plain ascent of Algorithm 1), or from where ROSG re-injects the denoiser.

The paper also lacks comparisons with other inexpensive alternatives that exploit the same structure: straight-through or one-step-surrogate noise optimization (e.g., ReNO-style), gradient guidance (DPS / universal guidance), and DNO on SD3.

2. The evidence for genuine alignment, as opposed to reward hacking, is weak.

  • The headline metric is the optimization target. HPSv2 is both the optimized reward and the primary metric, and the best trajectory is also selected by that reward. Jumps from about 30 to 40–44 HPSv2, far beyond any sampling-based method, are what one would expect from exploiting the reward model.
  • Clear reward hacking when CLIPScore is the reward (App. Table 5). CLIPScore rises from about 29.5 to 44.5–45.6, while held-out ImageReward falls below Best-of-N (SD3: 1.128 vs. 1.216; SD1.5: 0.241 vs. 0.351), and PickScore does not exceed the base model on SD3 (22.32 vs. 22.33).
  • Small or negative held-out gains over Best-of-N with HPSv2 as the reward. On GenAIBench, DTO is below BoN on VQAScore for SD3 (76.69 vs. 77.03) and on CLIPScore and VQAScore for SDXL (29.26 vs. 29.31; 73.64 vs. 73.98). On HPDv2, several held-out differences are not significant (App. Table 4, e.g., SDXL CLIPScore p = 0.57 vs. BoN).
  • Weak distributional evidence. The FID results (App. Table 13) use about 400 single-seed images from HPDv2 prompts compared against MS-COCO training images (base FID about 135). At this sample size and with mismatched prompt and reference distributions, FID cannot establish that ROSG keeps samples in distribution.
  • The theory does not address the concern. Theorem 1 is a first-order Taylor expansion plus a Lipschitz bound. As the authors acknowledge (App. A.3), it does not show that ROSG keeps samples in distribution.
  • Limited evaluation protocol and baselines. Each prompt uses a single image (seed 42). ReNeg is evaluated with pretrained negative embeddings rather than as an inference-time method, and DATE uses a non-default configuration (App. B.1).
Questions for authors
  1. ROSG schedule: paper vs. code. In Algorithm 1, tt runs from TT (noise) to 1, so Eq. 14 sets wt=1w_t = 1 (fully optimized scores) for the low-noise half of the steps. In the released code, the optimized prediction fully replaces the denoiser output in the high-noise half, and the interpolation is applied in the low-noise half. Which schedule produced the reported results? If it is the code's, please correct Eq. 14 and the related discussion.
  2. Relation to surrogate-gradient noise optimization. Since x0x_0 is affine in the trajectory variables with constant Jacobians, and optimizing xTx_T alone recovers most of the gain (App. Table 16), how does DTO compare with directly optimizing xTx_T (or an intermediate latent) using the identity-Jacobian surrogate, under the same optimizer, learning-rate schedule, ROSG re-sampling and number of reward evaluations? What optimizer and learning rate did the "w/o TO" ablation use?
  3. Optimizer and implementation details. Algorithm 1 states plain gradient ascent, but the code uses per-tensor Adam, gradient-norm clipping and a linearly decaying learning rate, and computes a norm regularizer that is never added to the loss. Which version produced the reported numbers?
  4. Held-out evaluation. Can you evaluate with reward models that were not used for optimization or selection (e.g., optimize HPSv2 and report ImageReward/PickScore as primary metrics, or GenEval), using several seeds per prompt? Please also report FID/KID on a COCO-caption prompt set against matching reference images, with enough samples (≥5k) for the statistic to be meaningful.
  5. Metric identity. Is the HPSv2 metric reported in the tables the same checkpoint as the reward used for optimization (the code loads HPS v2.1)? If so, please make this explicit and treat it as a training metric rather than an evaluation metric.
  6. Additional baselines. Please add gradient guidance (DPS / universal guidance) and a ReNO-style one-step-surrogate noise optimization, as well as DNO on SD3, or explain why they are infeasible.
Reviewer_15e4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Paper proposes Direct Trajectory Optimization (DTO) that is a training free framework that align diffusion models by optimizing the sampling trajectory while not requiring gradient through the denoising network. The paper also proposes Reward-Optimized Score Guidance (ROSG) which stabilize sampling by interpolating between the optimized scores and pretrained method.

Critical strengths

1- The method is a inference time alignment so it needs less computation resources than the backpropagation or fine-tuning methods.

2- The method shows improvement over baselines and BoN while needing less memory than BoN.

Critical weaknesses

1- In the table 2 & 3 method is usually compared against the BoN and baseline, which is not very convincing, especially that the motivation of paper is that it resolves the back propagating methods. However if there isn't one to compare against, this claim might not be convincing. Since it is not clear if the costs vs benefits are justified.

2- Some of the methods like DATE has SD3, and SDXL in their paper however here the comparison only happened on SD1.5 which is an old and simple baseline. This might show that the method is more suitable for simpler baselines.

3- Paper claims that there are substantial gains, however table 2&3 in paper and table 5 in appendix shows that only on the reward model measure there is improvement, in other measures method doesn't improve much and this suggests that there are some reward hacking.

Questions for authors

1- In Algorithm 1 line 6, where nn has been defined?

Suggestions:

1- Figure 1, 2 and 4 have misaligned figures.

2- The theorem 1 is not mathematically impactful enough to be called a theorem, maybe a less claim like proposition or lemma would be more mathematically meaningful.

Reviewer_DW2uOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Direct Trajectory Optimization (DTO), an inference-time alignment method that treats the initial noise and intermediate denoising predictions as free optimization variables. Reward gradients propagate through the sampler and decoder without differentiating through the denoising network. After each optimization step, Reward-Optimized Score Guidance (ROSG) interpolates the optimized predictions with newly evaluated model predictions.

Critical strengths

The method has a clear practical motivation: avoiding backpropagation through the denoising network can reduce the computational cost of inference-time alignment. The reported improvements in the optimized reward are substantial, and the experiments explore several backbones, reward functions, and algorithmic variants.

Critical weaknesses
  1. The novelty appears limited and is insufficiently isolated. Several existing methods, including DNO, DATE, and ReNeg, already perform reward-driven optimization during inference. DTO’s main distinction is to optimize intermediate predictions directly and blend them with model predictions. Although component ablations are provided, they do not convincingly separate the benefit of this parameterization from changes in effective step size and denoiser-based correction. For the linear sampler analyzed in the paper, unconstrained trajectory gradient ascent is equivalent to a rescaled final-latent gradient step. A controlled comparison with final-latent optimization using matched effective updates and a comparable correction procedure is therefore essential.

  2. The main evaluation concentrates on older backbones. Most comparisons and detailed analyses use SD1.5, SDXL, and SD3. The appendix includes SD3.5-Large-Turbo and FLUX.1-dev, but compares them only against the base model and Best-of-N. This leaves the method’s competitiveness against stronger inference-time alignment methods on newer generators insufficiently established.

  3. The automatic metrics do not establish equally substantial improvements in perceptual quality. Evaluation relies on longstanding metrics such as HPSv2, PickScore, CLIPScore, and ImageReward, with HPSv2 also serving as the optimization objective. Large gains in this target reward transfer unevenly to other metrics. For example, on GenAIBench with SD3, DTO substantially improves HPSv2 over Best-of-N but slightly decreases VQAScore and barely changes ImageReward. Stronger independent evaluation is needed to support the claims of improved image quality and resistance to reward hacking.

  4. The human evaluation is too limited to provide convincing confirmation. The SD1.5 study and additional SD3 study each use only 50 samples and two annotators. They report individual 1–5 ratings rather than direct side-by-side preference comparisons, without uncertainty estimates for the differences between methods. In particular, the SD3 improvement over Best-of-N is only 4.14 to 4.26, making its reliability difficult to assess.

  5. The qualitative evidence is sparse. The paper presents only a few selected prompts, including the additional appendix examples. Broader side-by-side comparisons across prompts and backbones, together with more failure cases, are needed to judge whether the reward gains correspond to consistent visible improvements.

Questions for authors
  1. Can you compare DTO against final-latent optimization with matched effective update magnitudes and a comparable denoiser-based correction procedure?

  2. Can you extend the SD3.5-Large-Turbo and FLUX.1-dev experiments to include stronger inference-time alignment baselines under matched computational budgets?

  3. Can you provide a larger, blinded side-by-side human preference study, with confidence intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.