- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Direct Trajectory Optimization (DTO), a training-free method for aligning text-to-image diffusion and flow models with a differentiable reward at inference time. For an ODE sampler, the final sample is a deterministic function of the initial noise and the per-step score/velocity predictions . DTO treats all of these as free variables and runs gradient ascent on the reward. Gradients are propagated only through the scheduler and the VAE decoder, not through the denoising network. To limit drift away from the pretrained model, each update is followed by a re-sampling pass, Reward-Optimized Score Guidance (ROSG), that interpolates the optimized scores with fresh denoiser predictions, . The best trajectory by reward is kept. Theorem 1 gives a first-order lower bound on the reward after ROSG under a Lipschitz assumption.
Experiments use HPSv2 as the reward on the HPDv2 test set (400 prompts) and GenAIBench (1,600 prompts), with SD1.5, SDXL and SD3. The appendix adds SD3.5-Large-Turbo, FLUX.1-dev, an SDE variant, and CLIPScore/PickScore as alternative rewards. DTO reports much higher HPSv2 than DNO, DATE, ReNeg and Best-of-N, gains on held-out metrics (PickScore, CLIPScore, VQAScore, ImageReward), about 10× lower latency than DNO, paired t-tests, a wall-clock-matched Best-of-N comparison, ablations of the optimized variables and of the schedule, and small human evaluations.
Critical strengths
-
A strong efficiency–performance trade-off across many backbones. Avoiding backpropagation through the denoiser makes DTO about 25× faster than DNO on SD1.5 (16.7 s vs. 427 s per image) with lower memory (11 GB vs. 18 GB). It applies unchanged to UNet and DiT models, including few-step and flow-matching samplers (SD1.5, SDXL, SD3, SD3.5-Turbo, FLUX). Under a wall-clock-matched comparison on SD1.5 (App. Table 11), DTO also exceeds Best-of-N with N=20 on all held-out metrics. This makes it a practical option for reward-guided generation.
-
A simple method with an informative ablation and released code. The method takes only a few lines on top of a standard sampler. The ablations (App. Tables 12, 14, 16) make the trade-off explicit: removing ROSG increases the optimized reward but degrades the held-out metrics. They also show which trajectory variables drive the gains.
Critical weaknesses
1. DTO is closer to surrogate-gradient noise/latent optimization than to "trajectory optimization," and the paper does not compare against that simpler interpretation.
For the samplers used (DDIM, flow-matching Euler), each update is affine in , so is an affine function of with constant coefficients (App. Eq. 20). For flow Euler, and . Every trajectory variable therefore receives the same image-space gradient , up to a scalar. DTO effectively ascends the latent with a "denoiser Jacobian = 0" surrogate gradient and then re-samples through ROSG.
The authors' own ablation supports this reading. Optimizing alone recovers most of the gain (37.49 vs. 40.75 HPSv2 on SD1.5, App. Table 16), while optimizing only the second half of the trajectory barely helps (24.25 vs. 23.48 for the base model). The "richer optimization space" offered as the explanation for beating the direct ablation (Table 12: 34.06) is therefore not established. The difference may instead come from step size, from the optimizer (the code uses per-tensor Adam with gradient clipping and a decaying learning rate, not the plain ascent of Algorithm 1), or from where ROSG re-injects the denoiser.
The paper also lacks comparisons with other inexpensive alternatives that exploit the same structure: straight-through or one-step-surrogate noise optimization (e.g., ReNO-style), gradient guidance (DPS / universal guidance), and DNO on SD3.
2. The evidence for genuine alignment, as opposed to reward hacking, is weak.
- The headline metric is the optimization target. HPSv2 is both the optimized reward and the primary metric, and the best trajectory is also selected by that reward. Jumps from about 30 to 40–44 HPSv2, far beyond any sampling-based method, are what one would expect from exploiting the reward model.
- Clear reward hacking when CLIPScore is the reward (App. Table 5). CLIPScore rises from about 29.5 to 44.5–45.6, while held-out ImageReward falls below Best-of-N (SD3: 1.128 vs. 1.216; SD1.5: 0.241 vs. 0.351), and PickScore does not exceed the base model on SD3 (22.32 vs. 22.33).
- Small or negative held-out gains over Best-of-N with HPSv2 as the reward. On GenAIBench, DTO is below BoN on VQAScore for SD3 (76.69 vs. 77.03) and on CLIPScore and VQAScore for SDXL (29.26 vs. 29.31; 73.64 vs. 73.98). On HPDv2, several held-out differences are not significant (App. Table 4, e.g., SDXL CLIPScore p = 0.57 vs. BoN).
- Weak distributional evidence. The FID results (App. Table 13) use about 400 single-seed images from HPDv2 prompts compared against MS-COCO training images (base FID about 135). At this sample size and with mismatched prompt and reference distributions, FID cannot establish that ROSG keeps samples in distribution.
- The theory does not address the concern. Theorem 1 is a first-order Taylor expansion plus a Lipschitz bound. As the authors acknowledge (App. A.3), it does not show that ROSG keeps samples in distribution.
- Limited evaluation protocol and baselines. Each prompt uses a single image (seed 42). ReNeg is evaluated with pretrained negative embeddings rather than as an inference-time method, and DATE uses a non-default configuration (App. B.1).
Questions for authors
- ROSG schedule: paper vs. code. In Algorithm 1, runs from (noise) to 1, so Eq. 14 sets (fully optimized scores) for the low-noise half of the steps. In the released code, the optimized prediction fully replaces the denoiser output in the high-noise half, and the interpolation is applied in the low-noise half. Which schedule produced the reported results? If it is the code's, please correct Eq. 14 and the related discussion.
- Relation to surrogate-gradient noise optimization. Since is affine in the trajectory variables with constant Jacobians, and optimizing alone recovers most of the gain (App. Table 16), how does DTO compare with directly optimizing (or an intermediate latent) using the identity-Jacobian surrogate, under the same optimizer, learning-rate schedule, ROSG re-sampling and number of reward evaluations? What optimizer and learning rate did the "w/o TO" ablation use?
- Optimizer and implementation details. Algorithm 1 states plain gradient ascent, but the code uses per-tensor Adam, gradient-norm clipping and a linearly decaying learning rate, and computes a norm regularizer that is never added to the loss. Which version produced the reported numbers?
- Held-out evaluation. Can you evaluate with reward models that were not used for optimization or selection (e.g., optimize HPSv2 and report ImageReward/PickScore as primary metrics, or GenEval), using several seeds per prompt? Please also report FID/KID on a COCO-caption prompt set against matching reference images, with enough samples (≥5k) for the statistic to be meaningful.
- Metric identity. Is the HPSv2 metric reported in the tables the same checkpoint as the reward used for optimization (the code loads HPS v2.1)? If so, please make this explicit and treat it as a training metric rather than an evaluation metric.
- Additional baselines. Please add gradient guidance (DPS / universal guidance) and a ReNO-style one-step-surrogate noise optimization, as well as DNO on SD3, or explain why they are infeasible.