acceptodds
Under review as a conference paper at ICLR 2027

Steer Early, Compose Better: Avoiding Late Horizon Steering by Optimal Residual Control for Diffusion Alignment

23 views

Abstract

Existing diffusion-alignment methods struggle with compositional tasks such as object-count constraints, object-attribute matching, and concept unlearning. We conjecture that this limitation arises from inaccurate estimates of intermediate-state values by the auxiliary networks that propagate rewards from clean samples to intermediate states. We demonstrate that such parameterizations exhibit Late Horizon Steering, a phenomenon in which steering is observed only near the clean endpoint, limiting their effectiveness on tasks that require intervention during the early stages of generation. To mitigate this problem, we build on a stochastic optimal-control formulation by exploiting the relation between the optimal control and the score residual: the difference between the controlled score field and the pretrained score field. Using the Hamilton-Jacobi-Bellman dynamics of the value function, we derive a coupled SDE governing the evolution of the diffusion state and its corresponding optimal score residual during the generation process, which we call the state-residual SDE. Based on this characterization, we propose Residual Consistency, a learning objective that enforces consistency with a discretization of the state-residual SDE, enabling alignment without an auxiliary network. We demonstrate that our proposed method achieves a superior trade-off across broad classes of post-training objectives, including human-preference, compositional, and safety alignment, achieving up to 12% improvement over baselines on compositional tasks and up to 20% lower detection rate on unlearnt concepts.

Primary area:
generative models
Keywords:
Diffusion Alignment, Stochastic Optimal Control, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3YcnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper treats KL-regularized reward fine-tuning of diffusion models as stochastic optimal control. It names "Late Horizon Steering" (LHS): methods with an auxiliary value network (∇-GFlowNet, VGGFlow) are said to match the optimal control only near the clean endpoint, which the authors conjecture limits compositional alignment.

  • Theory. The optimal control is written through the score residual h = s_θ − s. Differentiating the HJB equation gives a residual PDE (Prop. 3.1); Itô's lemma turns it into a coupled state–residual SDE (Thm. 3.2), discretized for the VP SDE (Cor. 3.3).
  • Method. Residual Consistency (RC; Alg. 1) trains h with a one-step bootstrapped loss on on-policy trajectories, anchored by λ⁻¹∇r at the last step, with no auxiliary network or backward adjoint pass.
  • Diagnostic. A cat→dog reward with an analytic optimal control (App. F) measures per-timestep control error (Fig. 1).

Experiments: GenEval-style composition on SD1.5 and SDXL scored by Qwen3-VL-2B (Table 1), three preference rewards on SD1.5 (Table 2), and concept unlearning on SD1.4 (Table 3).

Critical strengths
  1. A correct derivation that yields a simple, cheap objective.

    • The chain from (HJB) to (ResPDE), (SR-SDE) and (VP-Res) checks out (Prop. 3.1, App. B; Thm. 3.2, App. C; Cor. 3.3, App. E).
    • Replacing a PDE that contains a Laplacian with pathwise, first-order Jacobian products makes the target trainable with one VJP and one JVP per sampled pair (Sec. 3.4).
    • Alg. 1 needs no auxiliary network and no backward adjoint sweep. Training cost is in line with the baselines (Table 5): 16.7 vs 16–23 GPU-hours on SDXL, and 13.3 vs 12–16.6 on SD1.5.
    • The noise term √β_k J_hθ z_k reuses the sampling noise and has zero mean given X_k. It therefore appears to act as a control variate that leaves the fixed point unchanged.

    This matters because a bootstrapped alternative to adjoint-based SOC fine-tuning at baseline cost would be useful in practice.

  2. A reusable ground-truth diagnostic.

    • The reward is the log-density ratio between two prompts of the same frozen model, which gives the exact optimal control at every noise level (App. F; the derivation checks out for λ = 1).
    • On this task, RC tracks the oracle far better at intermediate times. Its J is ≈ 0.25 for t ∈ [0.5, 0.8], against ≈ 0.8–1.0 for ∇-GFlowNet and VGGFlow (Fig. 1 left; approximate readings of the plot).

    This matters because per-timestep control error is rarely measurable in this literature. The task turns a qualitative story about when alignment acts into a quantity that others can reuse.

Critical weaknesses
  1. "Late Horizon Steering" and its causal role in composition are not established.

    • The definition does not separate late steering from no steering. Def. 3.1 holds whenever J ≥ 0.75 for all t ≤ 0.8 (p. 4). The zero control gives J ≡ 1, and Fig. 1 plots the untouched Base at 1.0, so by this definition the base model itself exhibits LHS.
    • The curves appear equally consistent with baselines that barely steer at any time.
      • In Fig. 1 neither baseline approaches the oracle, even at t = 1: J ≈ 0.65 for ∇-GFlowNet and ≈ 0.80 for VGGFlow. Near t = 0, VGGFlow is at ≈ 1.1, worse than no control.
      • Over most of the trajectory their control carries only ≈ 1–14% of the optimal squared norm (Fig. 5 right). All of these are approximate readings from the plots.
      • Neither baseline produces dogs (Fig. 1 right).
      • This is not what Sec. 1 describes: "the learned control aligns with the optimal control only near the clean endpoint".
    • Settings that could produce the same curves are not reported.
      • Sec. 4.2 does not say at which training step J was measured, how many seeds were used, or which temperatures the baselines used. Table 4 has one row per experiment, not per method.
      • VGGFlow (NeurIPS 2025) optimizes a "relaxed objective" with a "deterministic control". Its own Limitations say that the result approximates the KL-regularized target well only for small λ, so its optimum appears not to coincide in general with the SDE oracle u* that Fig. 1 measures against.
      • VGGFlow's value-gradient model also weights its reward-gradient term by η_t (its Eq. 16). The default, η_t = t², is small at the noise end in VGGFlow's own time convention. If that default was kept, weak early control would be partly built into this baseline, independently of "a neural network's inability to derive meaningful information from these states" (Sec. 3.1). The paper does not say which η_t was used, and App. D.1 switches to a convention in which t = 1 is noise.
      • The Sec. 3.1 hypothesis would also apply to RC's own residual network, which reads the same noisy states.
    • No experiment links the timing of control to composition.
      • The title asserts "Steer Early, Compose Better", while Sec. 1 calls the link a conjecture.
      • LHS is measured only on cat→dog; the compositional gains come from a different reward and metric.
      • Adjoint Matching (AM, ICLR 2025) also uses no auxiliary network, which makes it the natural control for the claim that avoiding auxiliary networks avoids LHS. It is absent from the oracle test.

    This matters because contribution 1, the title and the motivation for the method all rest on this diagnosis.

  2. The compositional gains cannot yet be attributed to Residual Consistency.

    • The evaluator is the training reward.
      • Table 1 thresholds the same Qwen3-VL-2B "yes" probability that serves as the training reward (Sec. 4.3; Table 1 caption).
      • It uses 100 images per task, with no seeds or intervals; the word "seed" does not appear in the paper.
      • Held-out prompts test generalization across prompts, not whether a model has learned to exploit its judge.
      • The judge also appears lenient. It scores Base at 0.30 (SD1.5) and 0.57 (SDXL) on position, whereas GenEval's own detector pipeline reports 0.04 and 0.15 for these base models (GenEval, Table 2). Settings differ, so this comparison is indicative only.
    • Margins over the strongest baseline are mostly within noise. The figures below come from unpaired two-proportion tests with n = 100 per arm.
      • Over the next-best method, the position gains are +0.08 (SDXL, p ≈ 0.25) and +0.09 (SD1.5, p ≈ 0.20).
      • The two-object gains are +0.08 (p ≈ 0.03) and +0.12 (p ≈ 0.01); neither survives a Bonferroni correction over the 8 cells.
      • On count and color, RC is within 0.01 of RAM. RAM is contemporaneous, so parity with it is not itself a weakness.
      • The large, significant gains are over ∇-GFlowNet and VGGFlow (e.g., SD1.5 position +0.20 and +0.25). Their settings are undocumented, and on SDXL they barely move from Base.
    • The closest prior method is neither related nor compared.
      • The residual row of (SR-SDE) has drift −J_bᵀh, which is AM's lean-adjoint ODE (AM Eq. 38 with zero running cost), up to sign and scale.
      • AM already parameterizes the control as fine-tuned minus base (AM Eqs. 26–27), and describes its basic objective as "a consistency loss" (AM Sec. 5.2).
      • RC therefore appears to be a one-step bootstrapped estimator of AM's regression target (AM Eq. 35).
      • The paper discusses AM only as an expensive alternative (Sec. 1, Sec. 2.1, App. A). It runs AM in no experiment and never measures that cost, although VGGFlow ran AM as a baseline on SD3.

    This matters because the compositional result is the paper's headline empirical claim ("up to 12%"). RC's significance depends on whether one-step bootstrapping beats AM's full backward target in accuracy or in cost.

Questions for authors
  1. (Clarification.) These questions concern Figs. 1 and 5.

    • At what training step and GPU-hours were the curves measured, and with how many seeds and evaluation states?
    • Table 4 lists γ = 1 for this experiment. Did each baseline use the equivalent temperature, as the oracle assumes λ = 1 (App. F)?
    • Which η_t (VGGFlow, and in which time convention) and γ_t (∇-GFlowNet) were used, and which auxiliary architectures and initializations?
    • Does any baseline checkpoint generate dogs?

    A baseline that reaches RC's terminal reward yet keeps J ≥ 0.75 for t ≤ 0.8 would directly support LHS, especially if reported with cosine and norm ratio separately and with AM alongside.

  2. (Suggestion.) Two checks on the saved Table 1 checkpoints need no retraining.

    • (a) Re-score the images with GenEval's detector pipeline (or a larger, held-out VLM), with Wilson or bootstrap intervals.
    • (b) A time-window swap: sample with RC's residual only for t ≤ τ (base score afterwards), and the reverse, for a few values of τ. Do the same for one baseline.

    If the early window carries most of RC's gain, the title's claim is supported.

  3. (Clarification.) Do you agree that the residual row of (SR-SDE) is AM's lean-adjoint ODE and that RC's fixed point is AM's target (AM Eq. 35), so that RC is a one-step bootstrapped estimator of it? How do RC and AM compare on the cat→dog oracle, in control error and in GPU-hours?

  4. (Clarification.)

    • Which sampler, step count and CFG scale do Base and each method use at evaluation: the SDE of Alg. 2, or the probability-flow ODE?
    • How are s and h_θ defined under CFG (Table 4 uses scales 1–5), and is the KL taken against the guided base?
    • Reading the fine-tuned model as a sampler of the reward-tilted distribution requires a memoryless base process (AM, Theorem 1). Please state this assumption.

What would move me to 3 (weak accept): all three of the following.

  • LHS is shown to be distinct from under-training: baseline J curves at matched terminal reward or trained to a plateau, with their settings stated, and AM on the oracle task.
  • Table 1, re-scored with GenEval's detector pipeline or an independent VLM and reported with intervals, still shows RC's gains over the explicit-value baselines.
  • Either the time-window swap supports early steering, or the title and claims are recalibrated. In both cases, Sec. 3 should be positioned against AM's lean adjoint.

Minor (no effect on my rating):

  • The 12% claim. Sec. 4.3 attributes "up to 12%" to position tasks, but Table 1's position gains are +0.08 (SDXL) and +0.09 (SD1.5). The +0.12 gain is SD1.5 two-object (0.93 vs 0.81).
  • Overclaiming.
    • The conclusion's "state-of-the-art performance on a broad range of tasks" overstates Tables 1–3 and 6. RAM matches RC on count and color (within 0.01). DRaFT reaches higher reward on Aesthetic and HPSv2 (Table 6: 8.60 and 3.67, vs 7.78 and 3.41). RC has the worst Aesthetic FID in Table 2 (74.14).
    • Table 2's checkpoint-selection rule and the meaning of "±" are not stated.
    • The abstract's premise ("Existing diffusion-alignment methods struggle with compositional tasks") is broader than the evidence. Flow-GRPO (NeurIPS 2025, cited) reports GenEval accuracy rising from 63% to 95% on SD3.5-M.
  • Unlearning.
    • SDD's Church FID (194.2) and UCE's Harm FID (108.5) look like failed runs; please explain or re-tune them.
    • RC has the lowest detection rate on Harm only (0.24 vs 0.43). It ties on Van Gogh and Church and trails SDD on Nudity (0.08 vs 0.01).
    • No alignment baseline is run on unlearning, although the abstract lists unlearning among the tasks where alignment methods struggle.
  • RC's endpoint behaviour. RC's error rises from ≈0.27 at t = 0.9 to ≈0.97 at t = 1 (Fig. 1), and its norm ratio reaches ≈1.7 (Fig. 5 right); both are approximate readings. Sec. 4.2 does not discuss this. Is ∇r evaluated at the noisy final iterate of Alg. 2? AM evaluates its terminal adjoint after a noiseless final step for exactly this reason (AM App. G.1).
  • Theory statements.
    • Thm. 3.2 states only X*_0 ∼ N(0, I). Please add the terminal condition h*(X*_1, 1) = λ⁻¹∇r(X*_1); without it, h ≡ 0 solves (SR-SDE).
    • App. A's description of AM is inaccurate. In AM, the lean adjoint defines the regression target, and the memoryless schedule is what removes the initial-value bias.
    • The Limitations say these SOC principles are "not directly applicable to flow-based models", yet AM applies SOC fine-tuning to flow models through the memoryless schedule. A sentence on whether RC could do the same would help.
    • Relating Thm. 3.2 to the stochastic maximum principle, and to deep backward BSDE schemes (e.g., Huré, Pham & Warin, arXiv 1902.01599), would help.
  • Details.
    • The time convention flips between Sec. 2 (t = 1 is data) and App. D.1 (t = 1 is noise).
    • Sec. 3.4 should say which product, JVP or VJP, is computed for J_hθ z_k.
    • Please report the training steps, the size of the sampled pair subset, the number of held-out prompts per task, and what was matched in "matched training budgets" (Sec. 4.1).
3 comments
AuthorsOpenReview ↗

Author Response Part 1

We would like to thank the reviewer for their thorough and detailed review of our work. We are pleased to read that the reviewer’s summary has captured the core contributions of our work. We are also pleased that the reviewer finds that the presentation of our work was "clear and easy to follow". We are eager to engage with the reviewer during the discussion period to strengthen our work, and we appreciate the clarity with which the reviewer has stated their conditions for increasing their score. We will first address these conditions, followed by their questions.

Response to "What would move me to 3…"

R1. LHS is shown to be distinct from under-training … AM on the oracle task.

We show that our baselines are trained to convergence and include AM on the oracle task, which does not exhibit late-horizon steering but shows poorer similarity to the optimal control than RC. In Figure 5, we show that all baselines have been trained to convergence. We also report Adjoint Matching (AM) on the oracle task in Figure 1 of the updated manuscript, and observe that AM does not exhibit late steering. However, we also see that the control norm from our proposed method is closer to optimal. We choose to select the checkpoints at convergence and not at equal reward because of the reward, which is the model’s conditional likelihood, cannot be directly computed, unlike the reward gradient, which is the conditional model score, that can be computed for training. We provide all experimental parameters for each baseline in Appendix C.1. In GPU-hours, AM is trained for 3 GPU-hours on our reported hardware, whereas our proposed method is trained for 2 GPU-hours. Training time details for the remaining baselines are provided in Table 7.

R2. Table 1, re-scored with … independent VLM … with intervals, still shows RC's gains over the explicit-value baselines.

We rescore using Gemma4 12B, along with confidence intervals, and observe similar performance improvements over the baselines. In Appendix C.7, we re-score the compositional tasks using Gemma4 12B and provide bootstrapped 95% confidence intervals for 60 prompts per task, with 10 seeds per prompt, for a total of 2400 samples. We continue to observe that our proposed method outperforms our strongest baselines across most settings, achieving up to 8.2% higher accuracy compared to our strongest baseline. We additionally provide qualitative comparisons in Appendix C.11, showing that our proposed algorithm produces higher-quality images while adhering to the specified fine-tuning task, compared with the baselines.

R3. time-window swap supports early steering…

We perform the time-window swap experiment and observe that our proposed method enables earlier intervention, thereby validating late-horizon steering. In Appendix C.8, we provide the results of the time-step ablation requested by the reviewer. We use the controlled generation provided by our algorithm for the first $t$ fraction of the timesteps, then resume generation with the base process. We observe that our proposed method achieves most of the compositional accuracy gains by $t=0.4$: 12% on counting and 25% on position tasks, out of final gains of 14% and 27% on the respective tasks, as shown in Table 11. In Figure 9, we qualitatively observe that our proposed method achieves a performance gain at early time steps. This experiment has further shown the effect of Late Horizon Steering and the importance of early intervention in compositional tasks.


We continue our response in the next official comment, Author Response Part 2.

AuthorsOpenReview ↗

Author Response Part 2

We continue our response from Author Response Part 1.

Response to "Questions to Authors"

Q1. (Clarification) These questions concern Figs. 1 and 5 … with cosine and norm ratio separately and with AM alongside.

We have elaborated on our experimental setup and hyperparameters in Appendix C.1. Additionally, we report the cosine similarity and norm ratios between the learned and optimal controls, along with AM, in Figure 6. We find that both AM and RC have higher cosine similarity to the optimal control in the early time steps; however, AM's norm ratio is lower than RC's.

Q2. (Suggestion) Two checks … larger, held-out VLM … A time-window swap

We have incorporated these suggestions in our updated manuscript. We rescore the results using Gemma4 12B and perform the time-window swap experiment to validate late-horizon steering, and we kindly refer the reviewer to responses R2 and R3 above for additional details. We believe that these experiments further validate our core claims of early steering.

Q3. (Clarification.) Do you agree that…

We believe this statement merits further investigation beyond the current scope of this work, and we cannot agree or disagree with it at this time without a separate, thorough study comparing the SR-SDE and the Adjoint ODE.

How do RC and AM compare on the cat→dog oracle…?

We compare RC and AM in Figure 1 of the updated manuscript. We see that AM does not exhibit Late Horizon Steering but is less aligned with optimal control compared to RC. We refer the reviewer to response R1 above for a detailed discussion.

Q4. (Clarification.) …choice of sampler, step count, sampling…

We provide these additional clarifications in Appendix D.


We would like to bring to the reviewer's attention that we have added additional results and made changes to the manuscript (highlighted in green) in the OpenReview submission based on the review. We have also reorganized our appendix to include several new experimental results.

The points that the reviewer has raised have strengthened our work. Given ICLR's discussion format, we believe we have clarified our understanding of the reviewer’s most pressing concerns and will continue to engage throughout the discussion period. If the reviewer’s main concerns have been sufficiently addressed, we would be pleased if they could kindly increase their appraisal of our work.

AuthorsOpenReview ↗

Request for engagement. Discussion period closing.

Dear Reviewer 3Ycn,

As the author-reviewer discussion period approaches its deadline (Nov 18 AoE), we would like to inform you that we have addressed your concerns regarding converged training for the oracle task with results for AM, an independent VLM with confidence intervals, and a time-window swap experiment in our official comments. Please let us know if you have any remaining questions before the discussion window closes. We would be very happy to clarify further.

Reviewer_18WcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies “Late Horizon Steering” issues existing in prior diffusion alignment methods. Such issue aligns learned control with the optimal control only near the clean endpoint, which limits the compositional performance. To address the intermediate rewards problem, this work proposes an alignment framework that has no need for auxiliary networks called Residual Consistency. It learns an optimal score residual to realize reward alignment. Experiments across compositional alignment, human preference alignment, and concept unlearning validate its effectiveness.

Critical strengths
  1. The proposed Residual Consistency method does not rely on an auxiliary network, which may induce “Late Horizon Steering” issues or require expensive backward propagation.
  2. The theoretical justification seems solid. I agree with the claims that to realize better-aligned generation results, the optimization should be performed at the earlier denoising stage since it decides the structure of the whole image.
  3. The experiment evaluation compared with recent works seems convincing and broad.
Critical weaknesses
  1. Qualitative comparisons are limited. Only figure 1 and 2 show some qualitative results compared with previous methods. More qualitative results are encouraged to further improve the robustness of the proposed method.
  2. SD1.5 and SDXL are relatively outdated. Using a stronger base model as baseline is encouraged (e.g. FLUX) to show the effectiveness and generalizability of the proposed method.
  3. The paper writing can be improved. The contributions listed in the introduction are long and redundant, which makes it hard to capture the highlights of this paper.
Questions for authors

Can the authors show timestep-wise results for counting and positioning tasks to verify that Residual Consistency constructs the intended structure earlier? Currently, Figure 1 provides a simple concept transformation case.

2 comments

We thank the reviewer for their effort in reviewing our work. We are eager to engage with them during the ICLR discussion period to strengthen our work and address any questions that may prevent you from accepting it. We first address the reviewer’s summary of our work, then answer the questions raised by the reviewer and the critical weaknesses.

We would also like to flag the following points of our work that the reviewer has not raised in their summary. We derive a novel coupled state-residual SDE that governs the stochastic dynamics of the optimal control for the entropy-regularized reward maximization problem (P1) in terms of the state residual. We state this result formally in Theorem 3.2 and specialize it to VP processes in Corollary 3.3, thereby allowing us to design Algorithm 1. We believe this result was key to improving compositional alignment without the need for auxiliary networks or full rollouts.

Q1. Can the authors show timestep-wise results for counting and positioning tasks to verify that Residual Consistency constructs the intended structure earlier?

We include these results in the updated manuscript, where we observe that the intended structure is constructed at an earlier timestep. We have added qualitative examples and quantitative results of timestep-wise results in Appendix C.8, Figure 9, and Table 11 of the updated manuscript, along with experimental details. We apply controlled generation (using Residual Consistency) up to a particular timestep, then resume the base model’s generation after that timestep. We observe that our proposed method achieves most of the compositional accuracy gains by $t=0.4$: 12% on counting and 25% on position tasks, out of final gains of 14% and 27% on the respective tasks.

W1. Qualitative comparisons are limited…

In Appendix C.11, we provide additional qualitative results for the tasks we experimented with in our work. We observe that our proposed algorithm produces higher-quality images while adhering to the specified fine-tuning task, compared with the baselines. We initially followed our strongest baseline, RAM [1], to determine the number of qualitative images to include in the main body of our work.

W2. SD1.5 and SDXL are relatively outdated. Using a stronger base model as baseline is encouraged (e.g. FLUX) to show the effectiveness and generalizability of the proposed method.

We include results for SD3.5 Medium and observe similar performance improvements over the baselines. In Tables 1 and 10, we present results for SD3.5 Medium, the most performant model used by our baselines (RAM [1]), across a set of compositional tasks and provide a detailed experimental setup. We observe that our proposed method continues to deliver state-of-the-art performance on this model, achieving up to a 4.2% improvement over our strongest baseline, underscoring the algorithm’s effectiveness and generalizability. To further showcase the effectiveness and generalizability of our proposed method, we include scores from an additional VLM judge, Gemma4 12B, for all models considered in our work. We include these results in Appendix C.7 of the updated manuscript, where we continue to achieve state-of-the-art performance, with up to a 4.3% improvement over our strongest baseline.

W3. The paper writing can be improved. The contributions listed in the introduction are long and redundant, which makes it hard to capture the highlights of this paper.

In our initial submission, we attempted to keep our contributions precise and self-contained, but we have shortened them to emphasize the results of our work. We would like to gently point out that both reviewers n7q7 and 3Ycn rated the presentation of our work as "clear and easy to follow". We hope that this rewrite addresses the reviewer’s concern.


We would like to bring to the reviewer's attention that we have added additional results and made changes to the manuscript (highlighted in green) in the OpenReview submission based on the questions and weaknesses raised in this review. We have also reorganized our appendix to include several new experimental results.

We believe that we have addressed the core concerns the reviewer had, which prevented them from accepting our work. The points that they have raised have strengthened our work, and we are keen to further discuss their concerns and any new ones that may have arisen from the changes in the manuscript or from our clarifications here. If the reviewer’s concerns have been sufficiently addressed, we would be pleased if they could increase their appraisal of our work.


References

[1] Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow Models, Bergmeister et al., NeurIPS 2026.

AuthorsOpenReview ↗

Request for engagement. Discussion period closing.

Dear Reviewer 18Wc,

As the author-reviewer discussion period approaches its deadline (Nov 18 AoE), we would like to inform you that we have addressed your concerns about timestep-wise accuracy for the position and counting tasks, qualitative comparisons, results on a stronger base model, and writing in our official comments. Please let us know if you have any remaining questions before the discussion window closes. We would be very happy to clarify further.

Reviewer_n7q7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Residual Consistency for reward-aligning diffusion models. It defines the difference between aligned and base score fields as an implicit value-gradient/optimal-control representation, derives a coupled state-residual SDE from the HJB dynamics, and trains the score residual to satisfy a discretized bootstrapped consistency condition. The method avoids a separately parameterized intermediate value network. Experiments report better analytic-control recovery and improvements over ∇-GFlowNet, VGGFlow, and RAM on compositional tasks, preference-reward trade-offs, and concept unlearning.

Critical strengths

S1: The paper has a strong conceptual core. Representing optimal control as a score residual and deriving its trajectory-level dynamics is an elegant way to avoid an explicit value-gradient model.

S2: The analytic cat-to-dog control-recovery experiment is valuable. It measures the proposed intermediate-time mechanism against a known target rather than only relying on endpoint reward scores.

S3: The empirical scope is broad: two diffusion backbones, composition tasks, three preference objectives, and four unlearning concepts. The authors also report training cost and explicitly describe their adaptation of competing flow-based methods.

Critical weaknesses

W1: The theoretical-to-practical gap is substantial. The loss replaces the residual SDE's Jacobian terms with a JVP/VJP substitution, treats Jacobian computations as constants in differentiation, uses a bootstrapped target, and relies on an Euler discretization. The authors show the desired fixed point is preserved under smoothness assumptions, but offer no finite-step bias/convergence analysis or empirical ablation showing which approximation is necessary and stable.

W2: The causal claim that Late Horizon Steering explains compositional failure is stronger than the evidence. The phenomenon is established in one deliberately constructed likelihood-ratio prompt-transfer task, but the paper does not show that its magnitude predicts composition accuracy across tasks/methods or rule out reward/judge quality, LoRA capacity, sampling, or baseline implementation as alternative causes.

W3: The compositional evaluation is fragile. Table 1 uses Qwen3-VL-2B-Instruct with a hard 0.5 “yes” threshold on only 100 images per task, with no confidence intervals, alternative judge, human audit, or verification that the judge accurately assesses counts/relations under the changed image distribution. A 12% improvement can be within meaningful evaluation uncertainty at this scale.

Questions for authors

Q1: Please ablate the bootstrap target, JVP/VJP replacement, stop-gradient through Jacobians, timestep-pair sampling, and discretization resolution. How sensitive are results and control recovery to each approximation?

Q2: Can Table 1 be repeated with more samples, confidence intervals, an independent VLM/human audit, and an explicit calibration analysis of Qwen3-VL's binary judgment?

Q3: Can the authors validate the adapted RAM/VGGFlow implementations on their native or a shared reference setting, and compare with a native diffusion-alignment baseline where available?

2 comments

We thank the reviewer for their detailed review and for the positive appraisal of our work. We are pleased to see that the reviewer’s summary captures our core contributions. We address the questions raised by the reviewer below.

Q1: Please ablate the ... How sensitive are results and control recovery to each approximation?

We conduct and report these ablation studies in the updated manuscript. In Appendix C.5, we provide an ablation study on the number of bootstrapping timesteps for the proposed formulation. We consider k=1, 3, 5 timesteps of simulating the SR-SDE and matching the scores at the k-th step. We observe that choosing k=1, as we do in the manuscript, yields similar results while requiring almost half as many GPU-hours and a lower maximum memory utilization compared to k=5 (Table 8). We present a training ablation comparing the VJP and JVP in Appendix C.6, where we observe no dramatic change in performance with the JVP; however, training time increases slightly from 4.3 hours to 5 hours on our reported hardware setup. We note that our results are not sensitive to these particular approximations. Regarding ablations with stopgrad through Jacobians, we first note that this exact second-order term increases the memory utilization beyond feasibility. We also refer the reviewer to NablaGFN [1, Appendix G], where they state that "2nd-order gradients hurt the trade-off between reward convergence, diversity preservation, and prior preservation." Based on this, we do not compute second-order terms arising from the Jacobians. Additionally, we use uniform sampling over timesteps to avoid biasing the training dynamics toward any particular timestep range, as this would conflict with our main claim of late-horizon steering and would also prevent matching our baselines NablaGFN [1] and VGGFlow [2], which also sample timesteps uniformly. We also refer the reviewer to our baselines: NablaGFN [1], RAM [2], and VGGFlow [3], all of which utilize Euler/Euler-Maruyama Discretization where applicable.

Q2: Can Table 1 be repeated with more samples, confidence intervals, an independent VLM/human audit, and an explicit calibration analysis of Qwen3-VL's binary judgment?

We have reproduced our results with confidence intervals using 6 times as many samples and an independent VLM, and observed similar performance. We update Table 1 with 60 prompts for each of the 4 tasks, each with 10 seeds, for a total of 2400 generations. We have also updated the results in Table 1 to include bootstrapped confidence intervals and details of the experimental setup. We also report results using a separate VLM judge (Gemma4 12B) in Appendix C.7 and provide qualitative results in Appendix C.11. We observe that we continue to achieve up to a 10% improvement over our strongest baseline on the position task with SD1.5. We have also updated our manuscript to reflect these values.

Q3: Can the authors validate the adapted RAM/VGGFlow implementations on their native or a shared reference setting, and compare with a native diffusion-alignment baseline where available?

We include results with SD3.5 Medium, where we observe RC obtaining superior performance. We provide results for the SD3.5 Medium model, which is native to RAM [3]. We observe that our proposed method achieves up to 4.2% over our strongest baseline with equal training time, without an auxiliary network or advantage estimation via expensive trajectory rollouts.


We would like to bring to the reviewer’s attention that we have added all additional results and made the changes to the manuscript (highlighted in green) in the OpenReview submission. We have also reorganized our appendix to include several new experimental results.

In this response, we have attempted to address the questions raised by the reviewer and believe we have also addressed the critical weaknesses they identified. This discussion has already strengthened our manuscript, and we are eager to continue engaging throughout the discussion period and address any remaining concerns that may prevent the reviewer from championing this work for acceptance.


References:

[1] Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets, Liu et al. ICLR 2025
[2] Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow Models, Bergmeister et al., NeurIPS 2026.
[3] Value Gradient Guidance for Flow Matching Alignment, Liu et al., NeurIPS 2025.

AuthorsOpenReview ↗

Request for engagement. Discussion period closing.

Dear Reviewer n7q7,

As the author-reviewer discussion period approaches its deadline (Nov 18 AoE), we would like to inform you that we have addressed your concerns regarding the missing ablation studies, larger evaluation samples with confidence intervals, an independent judge, and native baseline comparison in our official comments. Please let us know if you have any remaining questions before the discussion window closes. We would be very happy to clarify further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request for reviewer engagement (5 days left in the discussion period)

Dear reviewers,

We would like to inform you that we have responded to your concerns in our official comments. We are eager to engage with you through the remainder of the discussion period. We wish to bring this to your attention in case the recent email outage prevented you from receiving our updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.