- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper treats KL-regularized reward fine-tuning of diffusion models as stochastic optimal control. It names "Late Horizon Steering" (LHS): methods with an auxiliary value network (∇-GFlowNet, VGGFlow) are said to match the optimal control only near the clean endpoint, which the authors conjecture limits compositional alignment.
- Theory. The optimal control is written through the score residual h = s_θ − s. Differentiating the HJB equation gives a residual PDE (Prop. 3.1); Itô's lemma turns it into a coupled state–residual SDE (Thm. 3.2), discretized for the VP SDE (Cor. 3.3).
- Method. Residual Consistency (RC; Alg. 1) trains h with a one-step bootstrapped loss on on-policy trajectories, anchored by λ⁻¹∇r at the last step, with no auxiliary network or backward adjoint pass.
- Diagnostic. A cat→dog reward with an analytic optimal control (App. F) measures per-timestep control error (Fig. 1).
Experiments: GenEval-style composition on SD1.5 and SDXL scored by Qwen3-VL-2B (Table 1), three preference rewards on SD1.5 (Table 2), and concept unlearning on SD1.4 (Table 3).
Critical strengths
-
A correct derivation that yields a simple, cheap objective.
- The chain from (HJB) to (ResPDE), (SR-SDE) and (VP-Res) checks out (Prop. 3.1, App. B; Thm. 3.2, App. C; Cor. 3.3, App. E).
- Replacing a PDE that contains a Laplacian with pathwise, first-order Jacobian products makes the target trainable with one VJP and one JVP per sampled pair (Sec. 3.4).
- Alg. 1 needs no auxiliary network and no backward adjoint sweep. Training cost is in line with the baselines (Table 5): 16.7 vs 16–23 GPU-hours on SDXL, and 13.3 vs 12–16.6 on SD1.5.
- The noise term √β_k J_hθ z_k reuses the sampling noise and has zero mean given X_k. It therefore appears to act as a control variate that leaves the fixed point unchanged.
This matters because a bootstrapped alternative to adjoint-based SOC fine-tuning at baseline cost would be useful in practice.
-
A reusable ground-truth diagnostic.
- The reward is the log-density ratio between two prompts of the same frozen model, which gives the exact optimal control at every noise level (App. F; the derivation checks out for λ = 1).
- On this task, RC tracks the oracle far better at intermediate times. Its J is ≈ 0.25 for t ∈ [0.5, 0.8], against ≈ 0.8–1.0 for ∇-GFlowNet and VGGFlow (Fig. 1 left; approximate readings of the plot).
This matters because per-timestep control error is rarely measurable in this literature. The task turns a qualitative story about when alignment acts into a quantity that others can reuse.
Critical weaknesses
-
"Late Horizon Steering" and its causal role in composition are not established.
- The definition does not separate late steering from no steering. Def. 3.1 holds whenever J ≥ 0.75 for all t ≤ 0.8 (p. 4). The zero control gives J ≡ 1, and Fig. 1 plots the untouched Base at 1.0, so by this definition the base model itself exhibits LHS.
- The curves appear equally consistent with baselines that barely steer at any time.
- In Fig. 1 neither baseline approaches the oracle, even at t = 1: J ≈ 0.65 for ∇-GFlowNet and ≈ 0.80 for VGGFlow. Near t = 0, VGGFlow is at ≈ 1.1, worse than no control.
- Over most of the trajectory their control carries only ≈ 1–14% of the optimal squared norm (Fig. 5 right). All of these are approximate readings from the plots.
- Neither baseline produces dogs (Fig. 1 right).
- This is not what Sec. 1 describes: "the learned control aligns with the optimal control only near the clean endpoint".
- Settings that could produce the same curves are not reported.
- Sec. 4.2 does not say at which training step J was measured, how many seeds were used, or which temperatures the baselines used. Table 4 has one row per experiment, not per method.
- VGGFlow (NeurIPS 2025) optimizes a "relaxed objective" with a "deterministic control". Its own Limitations say that the result approximates the KL-regularized target well only for small λ, so its optimum appears not to coincide in general with the SDE oracle u* that Fig. 1 measures against.
- VGGFlow's value-gradient model also weights its reward-gradient term by η_t (its Eq. 16). The default, η_t = t², is small at the noise end in VGGFlow's own time convention. If that default was kept, weak early control would be partly built into this baseline, independently of "a neural network's inability to derive meaningful information from these states" (Sec. 3.1). The paper does not say which η_t was used, and App. D.1 switches to a convention in which t = 1 is noise.
- The Sec. 3.1 hypothesis would also apply to RC's own residual network, which reads the same noisy states.
- No experiment links the timing of control to composition.
- The title asserts "Steer Early, Compose Better", while Sec. 1 calls the link a conjecture.
- LHS is measured only on cat→dog; the compositional gains come from a different reward and metric.
- Adjoint Matching (AM, ICLR 2025) also uses no auxiliary network, which makes it the natural control for the claim that avoiding auxiliary networks avoids LHS. It is absent from the oracle test.
This matters because contribution 1, the title and the motivation for the method all rest on this diagnosis.
-
The compositional gains cannot yet be attributed to Residual Consistency.
- The evaluator is the training reward.
- Table 1 thresholds the same Qwen3-VL-2B "yes" probability that serves as the training reward (Sec. 4.3; Table 1 caption).
- It uses 100 images per task, with no seeds or intervals; the word "seed" does not appear in the paper.
- Held-out prompts test generalization across prompts, not whether a model has learned to exploit its judge.
- The judge also appears lenient. It scores Base at 0.30 (SD1.5) and 0.57 (SDXL) on position, whereas GenEval's own detector pipeline reports 0.04 and 0.15 for these base models (GenEval, Table 2). Settings differ, so this comparison is indicative only.
- Margins over the strongest baseline are mostly within noise. The figures below come from unpaired two-proportion tests with n = 100 per arm.
- Over the next-best method, the position gains are +0.08 (SDXL, p ≈ 0.25) and +0.09 (SD1.5, p ≈ 0.20).
- The two-object gains are +0.08 (p ≈ 0.03) and +0.12 (p ≈ 0.01); neither survives a Bonferroni correction over the 8 cells.
- On count and color, RC is within 0.01 of RAM. RAM is contemporaneous, so parity with it is not itself a weakness.
- The large, significant gains are over ∇-GFlowNet and VGGFlow (e.g., SD1.5 position +0.20 and +0.25). Their settings are undocumented, and on SDXL they barely move from Base.
- The closest prior method is neither related nor compared.
- The residual row of (SR-SDE) has drift −J_bᵀh, which is AM's lean-adjoint ODE (AM Eq. 38 with zero running cost), up to sign and scale.
- AM already parameterizes the control as fine-tuned minus base (AM Eqs. 26–27), and describes its basic objective as "a consistency loss" (AM Sec. 5.2).
- RC therefore appears to be a one-step bootstrapped estimator of AM's regression target (AM Eq. 35).
- The paper discusses AM only as an expensive alternative (Sec. 1, Sec. 2.1, App. A). It runs AM in no experiment and never measures that cost, although VGGFlow ran AM as a baseline on SD3.
This matters because the compositional result is the paper's headline empirical claim ("up to 12%"). RC's significance depends on whether one-step bootstrapping beats AM's full backward target in accuracy or in cost.
- The evaluator is the training reward.
Questions for authors
-
(Clarification.) These questions concern Figs. 1 and 5.
- At what training step and GPU-hours were the curves measured, and with how many seeds and evaluation states?
- Table 4 lists γ = 1 for this experiment. Did each baseline use the equivalent temperature, as the oracle assumes λ = 1 (App. F)?
- Which η_t (VGGFlow, and in which time convention) and γ_t (∇-GFlowNet) were used, and which auxiliary architectures and initializations?
- Does any baseline checkpoint generate dogs?
A baseline that reaches RC's terminal reward yet keeps J ≥ 0.75 for t ≤ 0.8 would directly support LHS, especially if reported with cosine and norm ratio separately and with AM alongside.
-
(Suggestion.) Two checks on the saved Table 1 checkpoints need no retraining.
- (a) Re-score the images with GenEval's detector pipeline (or a larger, held-out VLM), with Wilson or bootstrap intervals.
- (b) A time-window swap: sample with RC's residual only for t ≤ τ (base score afterwards), and the reverse, for a few values of τ. Do the same for one baseline.
If the early window carries most of RC's gain, the title's claim is supported.
-
(Clarification.) Do you agree that the residual row of (SR-SDE) is AM's lean-adjoint ODE and that RC's fixed point is AM's target (AM Eq. 35), so that RC is a one-step bootstrapped estimator of it? How do RC and AM compare on the cat→dog oracle, in control error and in GPU-hours?
-
(Clarification.)
- Which sampler, step count and CFG scale do Base and each method use at evaluation: the SDE of Alg. 2, or the probability-flow ODE?
- How are s and h_θ defined under CFG (Table 4 uses scales 1–5), and is the KL taken against the guided base?
- Reading the fine-tuned model as a sampler of the reward-tilted distribution requires a memoryless base process (AM, Theorem 1). Please state this assumption.
What would move me to 3 (weak accept): all three of the following.
- LHS is shown to be distinct from under-training: baseline J curves at matched terminal reward or trained to a plateau, with their settings stated, and AM on the oracle task.
- Table 1, re-scored with GenEval's detector pipeline or an independent VLM and reported with intervals, still shows RC's gains over the explicit-value baselines.
- Either the time-window swap supports early steering, or the title and claims are recalibrated. In both cases, Sec. 3 should be positioned against AM's lean adjoint.
Minor (no effect on my rating):
- The 12% claim. Sec. 4.3 attributes "up to 12%" to position tasks, but Table 1's position gains are +0.08 (SDXL) and +0.09 (SD1.5). The +0.12 gain is SD1.5 two-object (0.93 vs 0.81).
- Overclaiming.
- The conclusion's "state-of-the-art performance on a broad range of tasks" overstates Tables 1–3 and 6. RAM matches RC on count and color (within 0.01). DRaFT reaches higher reward on Aesthetic and HPSv2 (Table 6: 8.60 and 3.67, vs 7.78 and 3.41). RC has the worst Aesthetic FID in Table 2 (74.14).
- Table 2's checkpoint-selection rule and the meaning of "±" are not stated.
- The abstract's premise ("Existing diffusion-alignment methods struggle with compositional tasks") is broader than the evidence. Flow-GRPO (NeurIPS 2025, cited) reports GenEval accuracy rising from 63% to 95% on SD3.5-M.
- Unlearning.
- SDD's Church FID (194.2) and UCE's Harm FID (108.5) look like failed runs; please explain or re-tune them.
- RC has the lowest detection rate on Harm only (0.24 vs 0.43). It ties on Van Gogh and Church and trails SDD on Nudity (0.08 vs 0.01).
- No alignment baseline is run on unlearning, although the abstract lists unlearning among the tasks where alignment methods struggle.
- RC's endpoint behaviour. RC's error rises from ≈0.27 at t = 0.9 to ≈0.97 at t = 1 (Fig. 1), and its norm ratio reaches ≈1.7 (Fig. 5 right); both are approximate readings. Sec. 4.2 does not discuss this. Is ∇r evaluated at the noisy final iterate of Alg. 2? AM evaluates its terminal adjoint after a noiseless final step for exactly this reason (AM App. G.1).
- Theory statements.
- Thm. 3.2 states only X*_0 ∼ N(0, I). Please add the terminal condition h*(X*_1, 1) = λ⁻¹∇r(X*_1); without it, h ≡ 0 solves (SR-SDE).
- App. A's description of AM is inaccurate. In AM, the lean adjoint defines the regression target, and the memoryless schedule is what removes the initial-value bias.
- The Limitations say these SOC principles are "not directly applicable to flow-based models", yet AM applies SOC fine-tuning to flow models through the memoryless schedule. A sentence on whether RC could do the same would help.
- Relating Thm. 3.2 to the stochastic maximum principle, and to deep backward BSDE schemes (e.g., Huré, Pham & Warin, arXiv 1902.01599), would help.
- Details.
- The time convention flips between Sec. 2 (t = 1 is data) and App. D.1 (t = 1 is noise).
- Sec. 3.4 should say which product, JVP or VJP, is computed for J_hθ z_k.
- Please report the training steps, the size of the sampled pair subset, the number of held-out prompts per task, and what was matched in "matched training budgets" (Sec. 4.1).
3 comments
Author Response Part 2
We continue our response from Author Response Part 1.
Response to "Questions to Authors"
Q1. (Clarification) These questions concern Figs. 1 and 5 … with cosine and norm ratio separately and with AM alongside.
We have elaborated on our experimental setup and hyperparameters in Appendix C.1. Additionally, we report the cosine similarity and norm ratios between the learned and optimal controls, along with AM, in Figure 6. We find that both AM and RC have higher cosine similarity to the optimal control in the early time steps; however, AM's norm ratio is lower than RC's.
Q2. (Suggestion) Two checks … larger, held-out VLM … A time-window swap
We have incorporated these suggestions in our updated manuscript. We rescore the results using Gemma4 12B and perform the time-window swap experiment to validate late-horizon steering, and we kindly refer the reviewer to responses R2 and R3 above for additional details. We believe that these experiments further validate our core claims of early steering.
Q3. (Clarification.) Do you agree that…
We believe this statement merits further investigation beyond the current scope of this work, and we cannot agree or disagree with it at this time without a separate, thorough study comparing the SR-SDE and the Adjoint ODE.
How do RC and AM compare on the cat→dog oracle…?
We compare RC and AM in Figure 1 of the updated manuscript. We see that AM does not exhibit Late Horizon Steering but is less aligned with optimal control compared to RC. We refer the reviewer to response R1 above for a detailed discussion.
Q4. (Clarification.) …choice of sampler, step count, sampling…
We provide these additional clarifications in Appendix D.
We would like to bring to the reviewer's attention that we have added additional results and made changes to the manuscript (highlighted in green) in the OpenReview submission based on the review. We have also reorganized our appendix to include several new experimental results.
The points that the reviewer has raised have strengthened our work. Given ICLR's discussion format, we believe we have clarified our understanding of the reviewer’s most pressing concerns and will continue to engage throughout the discussion period. If the reviewer’s main concerns have been sufficiently addressed, we would be pleased if they could kindly increase their appraisal of our work.
Request for engagement. Discussion period closing.
Dear Reviewer 3Ycn,
As the author-reviewer discussion period approaches its deadline (Nov 18 AoE), we would like to inform you that we have addressed your concerns regarding converged training for the oracle task with results for AM, an independent VLM with confidence intervals, and a time-window swap experiment in our official comments. Please let us know if you have any remaining questions before the discussion window closes. We would be very happy to clarify further.
Author Response Part 1
We would like to thank the reviewer for their thorough and detailed review of our work. We are pleased to read that the reviewer’s summary has captured the core contributions of our work. We are also pleased that the reviewer finds that the presentation of our work was "clear and easy to follow". We are eager to engage with the reviewer during the discussion period to strengthen our work, and we appreciate the clarity with which the reviewer has stated their conditions for increasing their score. We will first address these conditions, followed by their questions.
Response to "What would move me to 3…"
R1. LHS is shown to be distinct from under-training … AM on the oracle task.
We show that our baselines are trained to convergence and include AM on the oracle task, which does not exhibit late-horizon steering but shows poorer similarity to the optimal control than RC. In Figure 5, we show that all baselines have been trained to convergence. We also report Adjoint Matching (AM) on the oracle task in Figure 1 of the updated manuscript, and observe that AM does not exhibit late steering. However, we also see that the control norm from our proposed method is closer to optimal. We choose to select the checkpoints at convergence and not at equal reward because of the reward, which is the model’s conditional likelihood, cannot be directly computed, unlike the reward gradient, which is the conditional model score, that can be computed for training. We provide all experimental parameters for each baseline in Appendix C.1. In GPU-hours, AM is trained for 3 GPU-hours on our reported hardware, whereas our proposed method is trained for 2 GPU-hours. Training time details for the remaining baselines are provided in Table 7.
R2. Table 1, re-scored with … independent VLM … with intervals, still shows RC's gains over the explicit-value baselines.
We rescore using Gemma4 12B, along with confidence intervals, and observe similar performance improvements over the baselines. In Appendix C.7, we re-score the compositional tasks using Gemma4 12B and provide bootstrapped 95% confidence intervals for 60 prompts per task, with 10 seeds per prompt, for a total of 2400 samples. We continue to observe that our proposed method outperforms our strongest baselines across most settings, achieving up to 8.2% higher accuracy compared to our strongest baseline. We additionally provide qualitative comparisons in Appendix C.11, showing that our proposed algorithm produces higher-quality images while adhering to the specified fine-tuning task, compared with the baselines.
R3. time-window swap supports early steering…
We perform the time-window swap experiment and observe that our proposed method enables earlier intervention, thereby validating late-horizon steering. In Appendix C.8, we provide the results of the time-step ablation requested by the reviewer. We use the controlled generation provided by our algorithm for the first $t$ fraction of the timesteps, then resume generation with the base process. We observe that our proposed method achieves most of the compositional accuracy gains by $t=0.4$: 12% on counting and 25% on position tasks, out of final gains of 14% and 27% on the respective tasks, as shown in Table 11. In Figure 9, we qualitatively observe that our proposed method achieves a performance gain at early time steps. This experiment has further shown the effect of Late Horizon Steering and the importance of early intervention in compositional tasks.
We continue our response in the next official comment, Author Response Part 2.