acceptodds
Under review as a conference paper at ICLR 2027

CoSCoDiff: Consensus Splitting for Coordinating Objectives and Constraints in Diffusion Models

Abstract

Controllable diffusion models are used for task-specific generation, which often optimize objectives while satisfying feasibility constraints. Existing methods commonly optimize these requirements jointly, where conflicting updates can degrade sample quality and weaken constraint satisfaction. To address this issue, we explicitly separate constraints from objectives and formulate controllable diffusion sampling as a constrained stochastic optimal control problem amenable to variable splitting. Building on this formulation, we propose CoSCoDiff, a training-free framework that coordinates the two components through terminal-state consensus splitting. To approximate the intractable trajectory-level optimization, CoSCoDiff combines sequential rollout with Tweedie prediction to construct tractable local control updates. These updates propagate objective and consensus signals to intermediate noisy states, while a constraint operator enforces feasibility on the clean sample. The framework accommodates existing objective-guided methods and supports both primal-only and primal–dual coordination, allowing historical dual feedback to be retained or omitted in the presence of approximation errors. Comprehensive experiments demonstrate that CoSCoDiff effectively coordinates objective optimization and hard constraint satisfaction, and achieves the best overall performance among the baselines across heterogeneous control requirements.

Primary area:
generative models
Keywords:
Controllable Diffusion Models, Optimization, ADMM Method

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xQsvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoSCoDiff, a training-free framework for diffusion-based generation that jointly addresses task objectives and feasibility constraints through consensus splitting. The authors formulate controllable diffusion as a constrained stochastic optimal control problem and introduce an auxiliary terminal-state variable to separate objective-guided denoising from feasibility correction. They develop a practical approximation using sequential rollout and Tweedie prediction to propagate consensus information to intermediate noisy states. The framework supports primal-only and primal–dual updates, together with SDEdit-style restarts for iterative refinement. Experiments on a constrained toy problem, trajectory planning, physics-informed video generation, and text-conditioned human motion demonstrate improvements in objective optimization and constraint satisfaction compared with several baselines.

Critical strengths

(1) A flexible framework for separating objective optimization from constraint enforcement. The proposed consensus-splitting formulation provides a practical way to coordinate differentiable objectives with potentially non-differentiable feasibility operators. Unlike methods that incorporate both requirements into a single guidance update, CoSCoDiff allows existing objective-guidance methods and task-specific constraint operators to be combined without retraining the diffusion model. This modularity is particularly useful for applications where feasibility constraints are difficult to express as differentiable losses.

(2) Promising empirical results across heterogeneous constrained-generation tasks. The controlled toy experiment demonstrates that consensus-based coordination can improve optimization while maintaining feasibility, particularly when the unconstrained optimum conflicts with the feasible region. The experiments on planning, video generation, and human motion further illustrate the applicability of the framework to different types of objectives and constraints. These results support the practical relevance of the proposed approach beyond a single application.

Critical weaknesses

(1) The contribution of consensus splitting is not sufficiently isolated from iterative refinement. Although CoSCoDiff performs favorably against several conventional baselines, its improvements over the closely matched Naive Iterate baseline are relatively limited in the more complex tasks. In trajectory planning and text-conditioned motion generation, CoSCoDiff-P achieves only modest improvements, while CoSCoDiff-PD sometimes performs worse. Since Naive Iterate already incorporates repeated SDEdit restarts and feasibility corrections, these results make it difficult to determine how much of the improvement is specifically attributable to consensus splitting. Furthermore, some comparisons use different semantic guidance objectives, making it difficult to attribute the larger performance differences to the proposed coordination mechanism rather than the underlying optimization objective.

(2) The connection between the stochastic optimal control formulation and the practical guidance update requires stronger mathematical justification. The proposed method is motivated as an approximation to a constrained stochastic optimal control problem, but several steps in the derivation are insufficiently justified. In particular, the path-space KL regularizer obtained through Girsanov's theorem appears inconsistent with the definition of the control as an additive drift term in the reverse SDE. Moreover, the proposed Tweedie response coefficient is introduced without establishing its relationship to the terminal effect of the local control. These issues are important because the optimal-control formulation is presented as a central theoretical motivation for the method, rather than merely an interpretation of a heuristic guidance procedure.

Questions for authors

(1) Isolating the contribution of consensus splitting. Since Naive Iterate already performs repeated objective-guided rollouts, SDEdit restarts, and feasibility corrections, could the authors provide a more systematic comparison isolating the effect of the consensus penalty and dual-variable updates? In particular, paired comparisons under identical computational budgets and uncertainty estimates for the improvements would help establish whether consensus splitting provides a meaningful advantage beyond iterative refinement.

(2) Fairness of the objective-guidance comparisons. In the text-conditioned motion experiment, CoSCoDiff explicitly optimizes an external text-motion embedding objective that is also used for evaluation, whereas some competing methods rely primarily on their native guidance mechanisms. Could the authors clarify how the comparison changes when the baselines are equipped with the same semantic objective and feasibility operator? An independent measure of generation quality would also help distinguish genuine improvements from direct optimization of the evaluation metric.

(3) Justification of the optimal-control approximation. Could the authors clarify the definition and scaling of the control variable in the stochastic optimal control formulation? When utu_t enters the reverse SDE as an additive drift control with diffusion coefficient g(t)g(t), Girsanov's theorem gives a quadratic path-space KL regularizer proportional to ∫∣∣ut∣∣2/g(t)2dt\int ||u_t ||^2/g(t)^2 dt, rather than ∫g(t)2∣∣ut∣∣2dt\int g(t)^2 || u_t || ^2 dt. Is utu_t instead intended to represent a score-space perturbation? Relatedly, how is the proposed Tweedie response coefficient κt\kappa_t derived from the effect of a local control on the terminal clean-state prediction?

11 comments

Response to Reviewer xQsv

We thank the reviewer for the detailed feedback. Below we address (1) component contributions under controlled budgets, (2) objective-guidance fairness and independent new metric evaluation, and (3) the theoretical basis of the guidance update and conditional approximation-error bounds.

(Answer 1: Weakness 1 and Question 1) Contributions beyond iterative refinement

To determine whether coordination adds value beyond repeated refinement, we combine (1) consensus-penalty and dual controls, (2) matched score-call budget comparisons, and (3) SDEdit on/off ablations. These comparisons separate component contributions from additional computation and test their interaction with warm restarts.

1. Consensus-penalty and dual controls

The objective, restart/correction schedule, and final-return rule are fixed within each task. Naive Iterate already includes iterative refinement; CoSCoDiff-P adds the penalty, and CoSCoDiff-PD adds dual in guidance and correction. CoSCoDiff-PD (ρ=0\rho=0) retains correction dual but removes consensus guidance.

MethodConsensus penaltyHistorical dualToy dopt↓d_{\mathrm{opt}}\downarrowSimple2D: Feasibility (%) / length ↓\downarrowHuman motion T2M ↓\downarrow
Naive IterateNoNo0.4979 ± 0.016898.67 ± 1.83 / 1.7945 ± 0.02191.7271 ± 0.6254
CoSCoDiff-PYesNo0.4928 ± 0.016699.33 ± 1.49 / 1.6676 ± 0.01581.7216 ± 0.6183
CoSCoDiff-PD (ρ=0\rho=0)NoCorrection only0.1805 ± 0.008798.67 ± 1.83 / 1.7957 ± 0.02231.9403 ± 0.9932
CoSCoDiff-PDYesGuidance + correction0.2130 ± 0.0093100.00 ± 0.00 / 1.6736 ± 0.01241.9724 ± 1.0611

Table R1: Component ablation. Mean ± sample STD across seeds (human motion reports result across cases). Planning length uses feasible trajectories only. All included Toy/motion outputs are feasible.

Table R1 shows that the consensus penalty and dual feedback contribute on different tasks. On Simple2D, the penalty enhance the feasibility and shortens feasible paths relative to Naive Iterate by 0.1269, while dual feedback alone gives no gain. On Toy, the large gain persists without the consensus penalty, supporting a contribution from correction dual feedback, since CoSCoDiff-PD (ρ=0\rho=0) reaches 0.1805 while CoSCoDiff-P stays at 0.4928. On human motion, CoSCoDiff-P has a slightly lower mean native T2M than Naive, while historical dual feedback may be harmful. Hence, the contribution of each component is task-dependent.

Return-rule correction. Submitted Simple2D Naive/P runs used best-iterate selection; these entries are replaced by final-iterate reruns. Archived CoSCoDiff-PD and the new ρ=0\rho=0 control return the final iterate. This is a reporting correction, not an algorithmic gain.

2. Matched score-call budgets

To test whether coordination gains persist under the same score-evaluation budget, we compare Naive, CoSCoDiff-P, and CoSCoDiff-PD at 300, 500, and 700 score calls on Toy (Table R2). At each budget, all methods share the same seeds, SDEdit and correction schedules, and final-iterate return rule.

Score calls per methodCoSCoDiff-P improvement over NaiveCoSCoDiff-PD improvement over Naive
3000.0033 [0.0031, 0.0035]0.2624 [0.2561, 0.2693]
5000.0048 [0.0045, 0.0051]0.2815 [0.2746, 0.2891]
7000.0051 [0.0046, 0.0057]0.2861 [0.2785, 0.2936]

Table R2: Toy score-call-budget comparison. Paired improvement is dopt(Naive)−dopt(P/PD)d_{\mathrm{opt}}(\mathrm{Naive})-d_{\mathrm{opt}}(\mathrm{P/PD}), mean [95% seed-bootstrap CI] across seeds. Budgets of 300/500/700 calls correspond to 6/11/16 completed outer rounds; each uses the final iteration output.

CoSCoDiff-PD retains a substantial Toy advantage at every tested score-call budget, whereas CoSCoDiff-P gives smaller gains. These improvements do not require additional score evaluations. The intervals are pointwise without multiplicity correction; matching score calls does not equalize all computational overhead.

3. SDEdit on/off ablations

We toggle SDEdit separately for Naive, CoSCoDiff-P, and CoSCoDiff-PD at fixed KK. Turning it off restarts each round from high noise while retaining coordination state and the same correction and return rules.

MethodToy: SDEdit onToy: SDEdit offSimple2D: SDEdit onSimple2D: SDEdit off
Naive Iterate0.4979 ± 0.01680.4957 ± 0.012398.67 ± 1.83 / 1.7945 ± 0.021999.33 ± 1.49 / 1.7971 ± 0.0236
CoSCoDiff-P0.4928 ± 0.01660.4364 ± 0.020799.33 ± 1.49 / 1.6676 ± 0.015899.33 ± 1.49 / 1.6683 ± 0.0118
CoSCoDiff-PD0.2130 ± 0.00930.4451 ± 0.0216100.00 ± 0.00 / 1.6736 ± 0.0124100.00 ± 0.00 / 1.6805 ± 0.0142

Table R3: SDEdit ablation at fixed KK. Mean ± sample SD; Toy: dopt↓d_{\mathrm{opt}}\downarrow; Simple2D: feasibility (%) ↑\uparrow / feasible path length ↓\downarrow. Toy feasibility is 100% throughout.

Without SDEdit, CoSCoDiff-P still reduces Simple2D feasible path length over Naive by 0.1246, supporting a consensus contribution beyond warm restarts. Table R3 shows that SDEdit benefits CoSCoDiff-PD but worsens CoSCoDiff-P on Toy. This interaction persists at 700 score calls: the paired SDEdit benefit is 0.2135 for CoSCoDiff-PD and −0.0743 for CoSCoDiff-P. Coordination and SDEdit thus have interacting effects rather than one component uniformly explaining the gains.

(Answer 2: Weakness 1 and Question 2) Objective-guidance fairness and independent new metric evaluation

To address objective-choice and evaluator-reuse concerns, we combine (1) baselines equipped with the same semantic objective and common Post-hoc correction, and (2) independent TMR evaluation of saved samples. The first tests whether objective matching closes the performance gap; the second checks whether gains extend beyond the optimized evaluation interface.

1. Same-objective controls

Naive/P/PD share the semantic objective and feasibility operator. We introduce ADMMDiff + T2M guidance and LGD-MC + Joint Objective, which add the same frozen T2M embedding objective to trajectory guidance. Their Post-hoc variants use the common path correction and motion re-encoding. Table R4 reports these objective-aligned comparisons. Their inference procedures differ, so Naive/P/PD remain the direct controls for isolating coordination effects.

2. Independent metric evaluation

We evaluate saved samples with independently pretrained TMR (Petrovich et al., 2023), unused in guidance or parameter/sample selection. TMR reports target cosine similarity. Native T2M is related to the optimized objective, but uses a skeleton round-trip rather than the direct feature-space loss.

MethodFeasibility (%)Native T2M ↓\downarrowIndependent TMR ↑\uparrow
ADMMDiff + T2M guidance0.005.1779±1.56800.4596±0.0562
ADMMDiff + T2M guidance + Post-hoc100.005.3765±1.68490.4643±0.0539
LGD-MC + Joint Objective0.005.1183±2.25200.5158±0.1261
LGD-MC + Joint Objective + Post-hoc100.005.3933±1.54930.4929±0.0761
Naive Iterate100.001.7271±0.62540.5169±0.0762
CoSCoDiff-P100.001.7216±0.61830.6091±0.0848
CoSCoDiff-PD100.001.9724±1.06110.6100±0.0835

Table R4: Same-semantic-objective comparisons on the same 100-case benchmark. Metrics are sample mean ± STD across cases; feasibility uses the common 2 cm path criterion. Naive/P/PD use K=5K=5.

Table R4 shows that matching the objective does not close the gap. CoSCoDiff-P and CoSCoDiff-PD are fully feasible, reach 1.72 and 1.97 native T2M against 5.178 and 5.118 for the T2M-guided baselines we add, and score 0.609 and 0.610 on TMR against at most 0.517 for the best feasible baseline. TMR provides an independent evaluation of text–motion alignment, rather than a comprehensive measure of motion quality.

(Answer 3: Weakness 2 and Question 3) Control scaling and the two approximations

Score-space versus drift-space control. The reviewer is correct for an additive drift control. In Eq. (6), however, utu_t is intended as a score-space perturbation:

+g(t)d\bar W_t.$$ The drift change is $-g^2(t)u_t$, not $u_t$. Under the same initial law, matched nondegenerate diffusion coefficients, and a progressively measurable state-feedback control satisfying the usual Girsanov absolute-continuity and integrability conditions ([Karatzas and Shreve](https://link.springer.com/book/10.1007/978-1-4612-0949-2)), $$D_{\mathrm{KL}}(P^u\\|P^0) =\frac12\mathbb E_{P^u}\int_0^T \left\\|\frac{-g^2(t)u_t}{g(t)}\right\\|_2^2dt =\frac12\mathbb E_{P^u}\int_0^T g^2(t)\\|u_t\\|_2^2dt.$$ Since the score-space perturbation enters the reverse drift as $-g^2(t)u_t$, its diffusion-normalized magnitude is $g(t)\\|u_t\\|$, regardless of the reverse-time sign convention. We assume $g(t)>0$ almost everywhere. If $u_t$ instead denotes an additive drift increment, the energy is $\frac12\mathbb E\int\\|u_t\\|^2/g^2(t)\\,dt$, as the reviewer states. We will explicitly distinguish these conventions; the identity concerns continuous path laws, not the finite-step sampler's exact KL. **Tweedie substitution is not a terminal-response theorem.** For the Gaussian forward kernel with $\alpha_t>0$, define the deterministic estimate $$\bar x_0:=\frac{x_t+\sigma_t^2[s_\theta(x_t,t)+u_t]}{\alpha_t} =\hat x_{0,\theta}(x_t,t)+\kappa_tu_t, \qquad\kappa_t=\frac{\sigma_t^2}{\alpha_t}.$$ VE uses $\alpha_t=1$. [Tweedie's identity (Efron, 2011)](https://pmc.ncbi.nlm.nih.gov/articles/PMC3325056/) justifies the uncontrolled posterior mean with an exact score; substituting $s_\theta+u_t$ is only an algebraic surrogate. A shared-noise reverse Euler step changes the next state by $g^2(t)\Delta t\\,u_t$, whose terminal effect additionally depends on nominal continuation. It need not equal $\kappa_tu_t$. We will define $\bar x_0$ as this deterministic estimate, not the actual controlled terminal mean, and provide the corresponding Euler-based response expansion and mismatch analysis in the revised appendix.
Reviewer_biAJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines stochastic optimal control (SOC) with the alternating direction method of multipliers (ADMM). By approximating the cost-to-go, the authors turn the original stochastic optimal control problem into a tractable local control problem, and then construct the guidance signal from it. They then generate samples step by step and apply a single ADMM update to each sample to obtain a new feasible solution. In this way, CoSCoDiff obtains an iterative scheme that tries to improve the objective value while preserving the prior provided by the diffusion model.

The paper also studies three variants , Naive Iterate, CoSCoDiff-P, and CoSCoDiff-PD, which respectively consider three settings: without terminal consensus, without dual feedback, and with both considered, in order to explore the algorithm's performance. Finally, the authors conduct experiments on four tasks: Toy Ring, Trajectory Planning, Physics-Informed Video Generation, and Text-Conditioned Human Motion Generation.

Critical strengths
  1. The algorithm is relatively simple, training-free, and can reuse existing guiders and constraint operators.
  2. The experiments cover a variety of tasks.
Critical weaknesses
  1. The overall theory of the algorithm is largely heuristic and lacks theoretical guarantees. The experiments on the number of outer iterations KK in Figure 6 of Appendix D.3 also show that, for the current algorithm, simply increasing KK does not consistently improve performance. However, the paper gives no good suggestion for how to choose KK. Moreover, the closed-form solution for ut\boldsymbol{u}_t in Equation (13) involves two approximations:
\to \mathbb{E}\left[L_{\mathrm{con}}^k \mid \boldsymbol{x}_t, \boldsymbol{u}_t\right] \to L_{\mathrm{con}}^k(\bar{\boldsymbol{x}}_0).$$ The paper gives little theoretical explanation of why these approximations are reasonable. Although they lead to a computable closed-form solution, there is no guarantee that the approximations themselves are sound. 2. The baselines used across tasks are inconsistent. On Toy Ring, the paper uses PDM, Naive Iterate, and CoSCoDiff, where the latter two are both algorithms proposed in this paper; the baselines for Trajectory Planning are relatively complete, while the Physics-Informed Video experiment only adds DPS as an additional baseline.
Questions for authors
  1. Regarding Weakness 1, could the authors provide some theoretical explanation, at least showing that these approximations are theoretically justified in certain cases?

  2. Regarding Weakness 2, the paper already compares with ADMMDiff on Text-Conditioned Human Motion Generation. Since CoSCoDiff and ADMMDiff both use ADMM-style variable splitting, could the authors add ADMMDiff experiments on other applicable tasks, or explain why it is difficult to make a fair comparison with ADMMDiff on those tasks?

  3. The performance gain of CoSCoDiff over Naive Iterate is not stable. For example, CoSCoDiff and Naive Iterate perform quite closely in Table 2, and the same holds in Table 4. Does the main improvement come from the SDEdit-style repeated generation and projection, or from the consensus splitting mechanism proposed in the paper?

If the authors are willing to address my questions, I would consider raising my score.

9 comments

Response to Reviewer biAJ

We thank the reviewer for the detailed feedback. Below we (1) analyze the two approximations behind Eq. (13) under explicit assumptions and discuss choosing KK, (2) add ADMMDiff comparisons on further tasks, and (3) separate the contribution of consensus coordination from that of SDEdit refinement.

(Answer 1: Weakness 1 and Question 1) Approximation-error analysis and choice of K

We agree that Eqs. (12)–(13) require a clearer distinction between the exact control problem and its two approximations. Below, we provide conditional approximation-error bounds using standard variational and estimation identities, with the key assumptions stated. At fixed outer iteration kk, we retain the manuscript's terminal loss

\mathcal L_{\mathrm{con}}^k(y):=h(y)+\frac\rho2 \\|y-\tilde x_0^k-z_0^k\\|_2^2,$$ where $h:\mathbb R^d\to\mathbb R$ is measurable and $\rho\ge0$. Throughout each analysis, $h,\rho,\tilde x_0^k,z_0^k$ are fixed. We give **conditional value/loss approximation bounds** in two parts below. ### 1. Nominal-continuation error in Eq. (12) Define the scaled terminal-loss function $$\mathcal L_{\mathrm{con},\lambda}^k:\mathbb R^d\to\mathbb R,\qquad \mathcal L_{\mathrm{con},\lambda}^k(y):= \lambda\mathcal L_{\mathrm{con}}^k(y),\quad\lambda\ge0.$$ Here $\lambda$ is an analysis parameter scaling the entire loss; $\lambda=1$ recovers the manuscript's loss. This is not a change of $\rho$ alone. At a fixed state $x_t=x$, let $\mathbb E_{\pi_0}^{t,x}$ denote expectation under the nominal reverse process initialized at $(t,x)$, and let $X_0$ be its random terminal sample. Denote the corresponding optimal control value by $V_\lambda^k(t,x)$ and the zero-control value by $V_\lambda^{k,\pi_0}(t,x)$. Assume that (i) admissible controls act in the diffusion's noise directions and include the zero-control policy; (ii) the change of measure is valid, with the control energy equal to the conditional path-space KL divergence; and (iii) the terminal loss has finite exponential moments in a neighborhood of zero. By Girsanov's theorem and the Gibbs variational principle ([Boué and Dupuis, 1998](https://mgubi.github.io/store/e0809/dupuis-boue.pdf)), the optimal control value is bounded below by the log-partition value, $$V_\lambda^k(t,x)\ge -\log\mathbb E_{\pi_0}^{t,x} \exp[-\lambda\mathcal L_{\mathrm{con}}^k(X_0)],$$ while the admissible zero-control policy gives $$V_\lambda^k(t,x)\le V_\lambda^{k,\pi_0}(t,x) =\lambda\mathbb E_{\pi_0}^{t,x} [\mathcal L_{\mathrm{con}}^k(X_0)].$$ Combining these inequalities and expanding the log moment-generating function yields $$\boxed{ 0\le{}V_\lambda^{k,\pi_0}-V_\lambda^k \le{} \frac{\lambda^2}{2} \operatorname{Var}_{\pi_0}^{t,x} [\mathcal L_{\mathrm{con}}^k(X_0)] +O(\lambda^3), \qquad \lambda\downarrow0. }$$ If the optimal exponential path tilt is realizable by an admissible control, the variational lower bound is attained, and the corresponding expansion becomes an equality. Furthermore, if $a\le\mathcal L_{\mathrm{con}}^k(X_0)\le b$ almost surely, Hoeffding's lemma ([Hoeffding, 1963](https://doi.org/10.1080/01621459.1963.10500830)) gives the nonasymptotic bound $$\boxed{ 0\le V_\lambda^{k,\pi_0}-V_\lambda^k \le\frac{\lambda^2(b-a)^2}{8}. }$$ Thus, nominal continuation has a second-order value-error bound in the weak-terminal-loss regime, without requiring the optimal exponential tilt to be realizable. These bounds are pointwise in $(t,x)$. For Eq. (12), the pointwise inequalities can also be averaged over the post-control state when the corresponding bounds are integrable; a uniform asymptotic rate additionally requires suitable control of the remainder terms. They do not imply uniformly accurate guidance gradients or small approximation error at the experimental setting $\lambda=1$.

2. Deterministic terminal-loss surrogate error in Eq. (13)

This part uses the unscaled loss Lconk\mathcal L_{\mathrm{con}}^k and does not require the exponential-moment assumption above. Consider the Gaussian forward kernel

\mathcal N(\alpha_t y,\sigma_t^2I_d), \qquad \alpha_t>0,\quad\sigma_t>0.$$ At fixed $(t,x,u_t)$, define $$\begin{aligned} \hat x_{0,\theta}(x,t) &:=\frac{x+\sigma_t^2s_\theta(x,t)}{\alpha_t},\\\\ \bar x_0(x,t,u_t) &:=\frac{x+\sigma_t^2[s_\theta(x,t)+u_t]}{\alpha_t}\\\\ &=\hat x_{0,\theta}(x,t)+\kappa_tu_t,\\\\ \kappa_t&:=\frac{\sigma_t^2}{\alpha_t}. \end{aligned}$$ This retains the manuscript's $\bar x_0,\kappa_t$ notation (VE uses $\alpha_t=1$), but **redefines $\bar x_0$ explicitly as a deterministic score-substituted estimate**. Tweedie's identity ([Efron, 2011](https://pmc.ncbi.nlm.nih.gov/articles/PMC3325056/)) gives a posterior mean when using the true marginal score. It does not establish that adding $u_t$ changes the actual terminal mean by $\kappa_tu_t$. Let $Y_u$ be the random terminal sample obtained from the specified controlled step followed by nominal continuation. All expectations below are under this sampling law at fixed $(t,x,u_t)$. Assume $\mathbb E\\|Y_u\\|_2^2<\infty$, and define only the moments and mismatch needed for the bound: $$\mu_u:=\mathbb E Y_u,\qquad \Sigma_u:=\mathbb E[(Y_u-\mu_u)(Y_u-\mu_u)^\top],\qquad e_u:=\mu_u-\bar x_0.$$ If $h$ is differentiable with $L_h$-Lipschitz gradient on an open convex set containing $\bar x_0,\mu_u$ and $Y_u$ almost surely, the standard Taylor remainder, applied around $\mu_u$ and then between $\mu_u$ and $\bar x_0$, gives $$\left|\mathbb E\mathcal L_{\mathrm{con}}^k(Y_u) -\mathcal L_{\mathrm{con}}^k(\bar x_0)\right| \le\underbrace{\frac{L_h+\rho}{2}\operatorname{tr}\Sigma_u}_{\text{terminal uncertainty}} +\underbrace{\\|\nabla\mathcal L_{\mathrm{con}}^k(\bar x_0)\\|_2\\|e_u\\|_2 +\frac{L_h+\rho}{2}\\|e_u\\|_2^2}_{\text{terminal-mean mismatch}}.$$ The bound follows from the vanishing centered first-order term in the Taylor expansion. For the quadratic consensus penalty, the uncertainty contribution is exactly $\frac{\rho}{2}\operatorname{tr}\Sigma_u$ by the bias–variance identity. **Exactness in a quadratic special case.** Beyond the general error bound, consider a quadratic terminal objective $$h(y)=\frac12(y-a)^\top Q(y-a), \qquad Q=Q^\top\succeq0,$$ and an affine stochastic terminal response $$Y_u=m+Bu+\xi, \qquad \mathbb E\xi=0,\quad \operatorname{Cov}(\xi)=\Sigma,$$ where $\Sigma$ is independent of $u$. If the deterministic surrogate coincides with the true terminal mean for every admissible control, i.e., $\bar x_0(u)=\mu_u=m+Bu,\ \forall u\in\mathcal U$, then $$\boxed{ \mathbb E\mathcal L_{\mathrm{con}}^k(Y_u)= \mathcal L_{\mathrm{con}}^k(\bar x_0) + \frac12\operatorname{tr}[(Q+\rho I)\Sigma]. }$$ Since the additional term is independent of $u$, the expected-loss and mean-state local control problems have identical minimizers when equipped with the same control-energy regularizer. Thus, mean-state evaluation is exact at the level of the local optimizer in this special case, even when the terminal covariance is nonzero. This requires the surrogate mean to match the true terminal response for all admissible controls and is not assumed to hold in general diffusion sampling.

Control-response mismatch. The remaining error eue_u reflects the discrepancy between the actual terminal mean and the deterministic Tweedie surrogate. Let μ0\mu_0 denote the terminal mean under zero control. We decompose

e_u= \underbrace{\mu_0-\hat x_{0,\theta}}_{\text{nominal-estimate mismatch}} + \underbrace{ (\mu_u-\mu_0)-\kappa_tu_t }_{\text{control-response mismatch}}. }$$ The first term captures the discrepancy between nominal sampling and the learned Tweedie estimate. Let $p_t^\star$ denote the marginal distribution of the ideal forward diffusion, with $m_t^\star(x):=\mathbb E_{p^\star}[X_0\mid X_t=x]$ and $s_t^\star(x):=\nabla_x\log p_t^\star(x)$. Tweedie's identity gives $$\mu_0-\hat x_{0,\theta}(x,t) =(\mu_0-m_t^\star(x)) +\kappa_t(s^\star(x)-s_\theta(x,t)).$$ Consequently, $$\boxed{ \\|\mu_0-\hat x_{0,\theta}(x,t)\\|_2 \le \\|\mu_0-m_t^\star(x)\\|_2 +\kappa_t\\|s^\star(x)-s_\theta(x,t)\\|_2.}$$ This separates nominal-sampler discrepancy from score-estimation error. Both terms vanish under exact nominal sampling and an exact score. The coefficient $\kappa_t=\sigma_t^2/\alpha_t$ is exact for score substitution in Tweedie's estimator, but it does not generally equal the response of the true terminal mean to a one-step drift intervention. The second term may remain nonzero even in this ideal setting. To characterize it, consider a shared-noise Euler reverse step of duration $\delta>0$, followed by nominal continuation. All expectations in this local analysis are conditional on the fixed current state $X_t=x$, and $u_t$ is held fixed during the Euler step. Both $\mu_u$ and $\mu_0$ use the same Euler step and Markov nominal continuation, differing only in the local control. Let $M_s(z)=\mathbb E_{\pi_0}[X_0\mid X_s=z]$ denote the nominal terminal-mean function, $Z=X_{t-\delta}^{0}$ the uncontrolled next state, and $a=g^2(t)\delta u_t$. Since $X_{t-\delta}^{u}=Z+a,$ $$\mu_u-\mu_0 =\mathbb E[M_{t-\delta}(Z+a)-M_{t-\delta}(Z)].$$ Assume $\mathbb E\\|JM_{t-\delta}(Z)\\|_{\mathrm{op}}<\infty$. The Hessian bound below is understood as $\\|D^2M_{t-\delta}(z)[v,w]\\|_2\le H\\|v\\|_2\\|w\\|_2$ for all $v,w$, along the segments $z=Z+\theta a$, $\theta\in[0,1]$, almost surely. If $M_{t-\delta}$ is twice differentiable with uniformly bounded Hessian norm $H$, Taylor's theorem gives $$\mu_u-\mu_0 =g^2(t)\delta\\,\mathbb E[JM_{t-\delta}(Z)]u_t +R_{t,\delta},$$ where $\\|R_{t,\delta}\\|\le \frac H2g^4(t)\delta^2\\|u_t\\|^2$. Consequently, $$\boxed{ \\|(\mu_u-\mu_0)-\kappa_tu_t\\|\le\\|[g^2(t)\delta\\,\mathbb E[JM_{t-\delta}(Z)]-\kappa_tI]u_t\\| +\frac H2g^4(t)\delta^2\\|u_t\\|^2.}$$ This separates the response-operator mismatch from the local Taylor remainder. Importantly, the response-operator mismatch need not be small, even when the local Taylor remainder is small. We therefore interpret $\kappa_tu_t$ as an exact score-substitution shift defining the deterministic Tweedie surrogate, rather than a generally exact approximation of the one-step terminal-mean response. Together with the preceding loss bound, this analysis identifies the terminal uncertainty and response mismatch as distinct sources of surrogate error. It provides conditional justification and an exact quadratic special case without claiming that the practical approximation is uniformly accurate. For the nonsmooth Toy objective, use its $L_0$-Lipschitz property ($L_0=1$ for Euclidean distance) instead of assuming a Lipschitz gradient. Cauchy–Schwarz and the same quadratic identity give $$\left|\mathbb E\mathcal L_{\mathrm{con}}^k(Y_u) -\mathcal L_{\mathrm{con}}^k(\bar x_0)\right| \le{}L_0\sqrt{\operatorname{tr}\Sigma_u} +\frac\rho2\operatorname{tr}\Sigma_u +\left(L_0+\rho\\|\bar x_0-\tilde x_0^k-z_0^k\\|_2\right)\\|e_u\\|_2 +\frac\rho2\\|e_u\\|_2^2.$$ These results provide conditional approximation guarantees without implying general convergence. We will revise Eq. (12) to use the random terminal sample and clarify Eq. (13) as a deterministic Tweedie-based local surrogate. In particular, we will explicitly define $\kappa_t$ as the exact coefficient of score substitution in Tweedie's estimator, rather than the true terminal-mean response to a local control intervention. We will incorporate these clarifications and theoretical results into the revised manuscript and upload an updated PDF during the rebuttal. **Choice of $K$.** We agree that larger $K$ does not guarantee better performance. We regard $K$ as a finite refinement budget rather than a convergence parameter. A practical selection protocol is to evaluate a small set of candidate budgets on held-out validation cases, prioritize constraint satisfaction, and then compare objective quality and computational cost. The chosen $K$ should be fixed before test evaluation. Our existing sensitivity study demonstrates non-monotonic behavior, so we do not claim a convergence rate or a universally optimal stopping rule.
Reviewer_MBN8OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CoSCoDiff, a training-free framework for controllable diffusion sampling under feasibility constraints. The key idea is to decouple objective optimization from feasibility enforcement through consensus splitting. Specifically, the method introduces an auxiliary variable representing a feasible solution and alternates between objective-guided diffusion sampling and projection onto the feasible set.

Critical strengths
  1. Principled formulation. The consensus-splitting formulation provides a clear optimization perspective for combining objective-guided diffusion with feasibility enforcement. Separating the objective and feasibility components is intuitive and allows the method to accommodate different constraint sets through task-specific projection operators.

  2. Training-free and modular design. The method can be applied to pretrained diffusion models without additional training. The primal-dual formulation also provides a natural mechanism for incorporating feedback from previous feasibility violations into subsequent sampling iterations.

Critical weaknesses
  1. The proposed algorithm relies heavily on the availability of an efficient projection operator, which may not be practical for complex constraints. Although the authors discuss iterative projection methods for more general constraints in the appendix, it remains unclear how the additional optimization overhead affects sampling efficiency and scalability, particularly when multiple projection steps are required across outer iterations.
Questions for authors
  1. How does the computational cost of the feasibility projection operator affect the overall sampling efficiency, particularly for complex constraints that require iterative corrections? Could the authors provide an end-to-end runtime comparison that includes both diffusion rollouts and feasibility corrections?

  2. The primal-dual variant outperforms the primal-only variant on some tasks but performs worse on others, such as trajectory planning and human motion generation. Could the authors provide more insight into when historical dual feedback is beneficial and when it may be detrimental? Is there a practical criterion for choosing between the two variants?

6 comments

Response to Reviewer MBN8

We thank the reviewer for the constructive feedback. Below we (1) quantify feasibility correction overhead through runtime separation across four tasks, and (2) investigate how historical dual feedback takes effects through controlled experiments, providing empirical insights into choosing between CoSCoDiff-P and CoSCoDiff-PD.

(Answer 1: Weakness 1 and Question 1) Feasibility and runtime of projection operators

We would like to note that CoSCoDiff accepts a task-provided feasibility operator, implemented as an exact projection, iterative solver, or domain-specific correction. In the evaluated implementations, feasibility correction is not the dominant runtime cost. Our contribution is to coordinate this operator with objective-guided generation, and designing a universal constraint solver is outside the framework's scope. In fact, projection-based constraint enforcement is a well-stuied approach in constrained diffusion, including PDM (Christopher et al., 2024) and post-hoc correction baselines.

To test how the feasibility operator affects overall efficiency, we measure the component-wise runtime of Naive Iterate and CoSCoDiff-P/PD under matched rollout and correction schedules across all four tasks. In Tables R1–R4, Total covers all outer iterations: for Toy, Trajectory Planning, and Human Motion, it measures end-to-end sampling, excluding model loading, warm-up, external evaluation, and serialization; for Video Frame Generation, it measures active refinement, excluding initial generation. Separate components consist of the cost of Guided rollout (score evaluations, sampling dynamics, and objective/consensus guidance), Correction (all intermediate and terminal feasibility updates, including associated detection), and Restart/Other (re-noising, coordination, and remaining overhead).

MethodTotal (s)Guided rollout (%)Correction (%)Restart/Other (%)
Naive Iterate0.2517±0.00420.2517\pm0.004285.400 ± 0.0300.189 ± 0.00214.411 ± 0.030
CoSCoDiff-P0.3019±0.00690.3019\pm0.006985.741 ± 0.1390.164 ± 0.00214.095 ± 0.138
CoSCoDiff-PD0.3017±0.00600.3017\pm0.006085.750 ± 0.1110.163 ± 0.00214.087 ± 0.109

Table R1: Toy runtime.

MethodEnvSimple2D Total (s)Guided rollout (%)Correction (%)Restart/Other (%)EnvDense2D Total (s)Guided rollout (%)Correction (%)Restart/Other (%)
Naive Iterate122.96±0.55122.96\pm0.5598.773 ± 0.0300.734 ± 0.0300.4927 ± 0.000394.05±1.8394.05\pm1.8397.689 ± 0.0711.818 ± 0.0750.493 ± 0.004
CoSCoDiff-P122.96±0.86122.96\pm0.8698.781 ± 0.0100.723 ± 0.0120.496 ± 0.00293.78±0.7893.78\pm0.7897.951 ± 0.4631.558 ± 0.4660.491 ± 0.003
CoSCoDiff-PD122.85±0.43122.85\pm0.4398.783 ± 0.0080.716 ± 0.0060.501 ± 0.00296.62±1.1596.62\pm1.1597.812 ± 0.4671.703 ± 0.4700.485 ± 0.003

Table R2: Trajectory planning runtime.

MethodTotal active time (s)Guided rollout (%)Correction (%)Restart/Other (%)
Naive Iterate49.8676±0.051749.8676\pm0.051799.773 ± 0.0080.136 ± 0.0040.091 ± 0.005
CoSCoDiff-P50.1781±0.053350.1781\pm0.053399.607 ± 0.2800.250 ± 0.2020.144 ± 0.078
CoSCoDiff-PD50.5002±0.095150.5002\pm0.095199.728 ± 0.0320.133 ± 0.0020.138 ± 0.030

Table R3: Video frame generation runtime.

MethodTotal (s)Guided rollout (%)Correction (%)Restart/Other (%)
Naive Iterate99.3141±2.066599.3141\pm2.066599.785±0.01599.785\pm0.0150.113±0.0120.113\pm0.0120.102±0.0060.102\pm0.006
CoSCoDiff-P99.9595±2.649699.9595\pm2.649699.780±0.01999.780\pm0.0190.113±0.0100.113\pm0.0100.107±0.0090.107\pm0.009
CoSCoDiff-PD100.7411±4.5661100.7411\pm4.566199.783±0.01599.783\pm0.0150.112±0.0080.112\pm0.0080.105±0.0080.105\pm0.008

Table R4: Human motion runtime.

Across Tables R1–R4, correction accounts for only 0.16–0.19% on Toy, 0.72–0.73% on Simple2D, 1.56–1.82% on Dense2D, 0.13–0.25% on Video, and 0.11% on Human Motion. We can therefore see that feasibility correction by the projection operator is a minor overhead in these implementations, while Guided rollout dominates the measured cost.

(Answer 2: Question 2) Insights into historical dual feedback

To clarify how historical dual feedback takes effect, we examine three complementary settings: (1) changing the Toy objective target, (2) inaccurate feasibility correction, and (3) isolating dual feedback in guidance and correction across tasks. These experiments show task- and location-dependent effects. They provide empirical insights into P/PD behavior, rather than a universal criterion for selecting between them. Each insight is provided next to the corresponding table.

All tables report mean ± sample STD with bold marking the better mean. The statistics are computed across seeds (human motion report result across cases) for all tested tasks using the same experimental settings from manuscript. Feasibility is the percentage of samples satisfying the evaluated constraints. For Toy, doptd_{\mathrm{opt}} is the mean Euclidean distance to the nearest constrained optimum on the unit circle for the given objective target, including all samples; lower is better. For the consensus-penalty weight ρ\rho and the number of outer iterations KK, we retain the same hyperparameter setting from the main experiment.

1. Effect of Objective–Constraint Conflict on P/PD Performance

This experiment tests whether the benefit of dual feedback depends on how strongly the objective conflicts with the constraint. In the Toy problem, the prior is the unit circle and the constraint is y≥0.2y\ge 0.2 (a half space). The objective target is the point (1−(y⋆)2, y⋆)(\sqrt{1-(y^\star)^2},\,y^\star) on the circle, and we test the six values of y⋆y^\star shown below. With the exact projection fixed, we move the target from outside the feasible region (y⋆=−1y^\star=-1), through the boundary (y⋆=0.2y^\star=0.2), to the interior (y⋆=1y^\star=1). Feasibility and doptd_{\mathrm{opt}} results for CoSCoDiff-P and CoSCoDiff-PD are reported in Table R5.

Method / y⋆y^\star−1.0-1.0−0.6-0.6−0.2-0.20.20.20.60.61.01.0
CoSCoDiff-P100.00 ± 0.00 / 0.4928 ± 0.0166100.00 ± 0.00 / 0.7055 ± 0.0271100.00 ± 0.00 / 0.6464 ± 0.0300100.00 ± 0.00 / 0.6602 ± 0.0275100.00 ± 0.00 / 0.5716 ± 0.0169100.00 ± 0.00 / 0.5998 ± 0.0160
CoSCoDiff-PD100.00 ± 0.00 / 0.2130 ± 0.0093100.00 ± 0.00 / 0.5552 ± 0.0273100.00 ± 0.00 / 0.5180 ± 0.0320100.00 ± 0.00 / 0.5674 ± 0.0290100.00 ± 0.00 / 0.6132 ± 0.0203100.00 ± 0.00 / 0.6722 ± 0.0171

Table R5: CoSCoDiff-P vs. CoSCoDiff-PD as the objective target moves from outside the feasible region to its interior. Feasibility (%) ↑\uparrow / dopt↓d_{\mathrm{opt}}\downarrow.

Both variants remain fully feasible. CoSCoDiff-PD achieves lower doptd_{\mathrm{opt}} for y⋆=−1,−0.6,−0.2,0.2y^\star=-1,-0.6,-0.2,0.2, whereas CoSCoDiff-P does so for 0.6,10.6,1. Historical feedback is beneficial here when the objective target violates or lies on the constraint boundary, but detrimental for targets farther inside the feasible half-space. This pattern is consistent with a role for objective–constraint conflict, but does not separate conflict from other changes in target geometry.

2. Inaccurate-correction stress test

To test whether accumulated dual feedback compensates for inaccurate feasibility correction, we replace the projection operator by $\mathcal P_{\mathcal C}^{(\eta)}(v) = v+\eta\bigl(\mathcal P_{\mathcal C}(v)-v\bigr),$ where 0<η≤10<\eta\leq1; v=x0k+1−z0kv=x_0^{k+1}-z_0^k for CoSCoDiff-PD and v=x0k+1v=x_0^{k+1} for CoSCoDiff-P. Thus, η=1\eta=1 recovers exact projection, whereas smaller values leave part of the constraint violation uncorrected.

Method / η\eta1.01.00.750.750.50.50.250.25
P100.00 ± 0.00 / 0.4928 ± 0.016678.28 ± 2.00 / 0.3765 ± 0.015570.28 ± 2.43 / 0.3327 ± 0.013847.80 ± 2.03 / 0.3225 ± 0.0114
PD100.00 ± 0.00 / 0.2130 ± 0.009322.99 ± 1.79 / 0.2775 ± 0.008618.40 ± 1.81 / 0.6082 ± 0.019916.47 ± 1.56 / 0.8653 ± 0.0251

Table R6: Toy inaccurate correction. Feasibility (%) ↑\uparrow / dopt↓d_{\mathrm{opt}}\downarrow; distance includes infeasible samples.

CoSCoDiff-P retains higher feasibility at every η<1\eta<1; at η=0.75\eta=0.75, CoSCoDiff-PD has lower distance but far lower feasibility. Historical feedback does not necessarily compensate for incomplete correction. However, a separate feasibility-preserving tangential-bias experiment still favors CoSCoDiff-PD, so this observation does not extend to every approximate operator.

3. Isolating Dual Feedback in Guidance and Correction

Where does historical dual feedback enter the algorithm? Full CoSCoDiff-PD uses it at two locations: (i) the consensus guidance inside the guided rollout, and (ii) the input shift to the constraint operator during correction. CoSCoDiff-P uses dual feedback at neither location, but retains the consensus penalty. To understand why dual feedback helps on some tasks and hurts on others, we perform an internal ablation of these two injection locations: retain only (i), retain only (ii), retain both (i.e., PD), or retain neither (i.e., P). The positive consensus penalty and remaining settings are fixed within each task. Thus these are location ablations within the coordination mechanism, not comparisons against Naive Iterate or tests of removing consensus splitting. The single-location variants diagnose how the two uses of the same dual history contribute and interact.

Toy. We use the original target (0,−1)(0,-1) and retain SDEdit in all four variants. The norm ratio measures consensus-gradient magnitude relative to objective-gradient magnitude; gradient cosine measures their directional alignment (negative values indicate opposition).

Dual-feedback locationFeasibility (%) / dopt↓d_{\mathrm{opt}}\downarrowConsensus/objective norm ratioGradient cosine
None (CoSCoDiff-P)100.00 ± 0.00 / 0.4928 ± 0.01660.730 ± 0.016-0.035 ± 0.034
Guidance only100.00 ± 0.00 / 0.7673 ± 0.01814.663 ± 0.091-0.841 ± 0.013
Correction only100.00 ± 0.00 / 0.1764 ± 0.00880.739 ± 0.0150.021 ± 0.033
Both (CoSCoDiff-PD)100.00 ± 0.00 / 0.2130 ± 0.00931.911 ± 0.060-0.425 ± 0.026

Table R7: Toy dual-path interventions. Gradient statistics average the final rollout; cosine compares consensus and objective gradients.

In Toy, retaining dual feedback only at correction gives the lowest distance. Using it only in the guided rollout worsens distance relative to CoSCoDiff-P and produces a stronger, opposing consensus gradient. The location matters: correction feedback helps here, while adding rollout feedback reduces that benefit. Full CoSCoDiff-PD nevertheless substantially improves over CoSCoDiff-P (0.21300.2130 versus 0.49280.4928), even though correction-only is better. These interventions alter subsequent states, so gradient opposition alone is not a causal explanation.

Human motion. We evaluate all four routes on 30 matched cases from from the previous 100-case benchmark. T2M distance measures text–motion mismatch in the native evaluator's embedding space; lower values indicate better alignment. All four variants are feasible.

Dual-feedback locationT2M distance ↓\downarrow
None (CoSCoDiff-P)1.6239 ± 0.5641
Guidance only1.6414 ± 0.5568
Correction only1.8375 ± 0.8469
Both (CoSCoDiff-PD)1.8327 ± 0.8349

Table R8: Human-motion dual-path ablation. Mean ± STD across cases.

In human motion, CoSCoDiff-P has the lowest native T2M distance; retaining dual feedback in the guided rollout is close to CoSCoDiff-P, whereas retaining it at correction is worse and close to full CoSCoDiff-PD. A same-raw-input intervention also increases native T2M by 0.2168 ± 0.4836 when correction dual is added. Unlike in Toy, the correction location is counterproductive for this metric in this setting. This identifies a task-dependent effect of the dual feedback location, not a universal degradation of motion quality.

Trajectory planning. We retain the consensus penalty and compare the same four dual routes, with matched hyperparameters and each environment's SDF settings, and all methods return the final iteration.

Dual-feedback locationSimple2D: Feasibility (%) / lengthDense2D: Feasibility (%) / length
None (CoSCoDiff-P)99.33 ± 1.49 / 1.6676 ± 0.015884.00 ± 5.48 / 2.0319 ± 0.1028
Guidance only100.00 ± 0.00 / 1.6674 ± 0.001290.00 ± 10.00 / 1.9911 ± 0.0236
Correction only100.00 ± 0.00 / 1.6727 ± 0.007470.00 ± 17.32 / 2.0636 ± 0.0254
Both (CoSCoDiff-PD)100.00 ± 0.00 / 1.6736 ± 0.012476.00 ± 11.40 / 2.2183 ± 0.1312

Table R9: Trajectory dual-path ablation. Mean ± sample STD across seeds; length uses feasible trajectories within each run.

On Simple2D, all variants achieve near-full feasibility with similar conditional path lengths. On Dense2D, guidance-only achieves the best feasibility and length, whereas correction-only has the worst feasibility and CoSCoDiff-PD has the worst length. Dual feedback has task-dependent effects at different injection locations; activating both does not necessarily combine their benefits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.