- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoSCoDiff, a training-free framework for diffusion-based generation that jointly addresses task objectives and feasibility constraints through consensus splitting. The authors formulate controllable diffusion as a constrained stochastic optimal control problem and introduce an auxiliary terminal-state variable to separate objective-guided denoising from feasibility correction. They develop a practical approximation using sequential rollout and Tweedie prediction to propagate consensus information to intermediate noisy states. The framework supports primal-only and primal–dual updates, together with SDEdit-style restarts for iterative refinement. Experiments on a constrained toy problem, trajectory planning, physics-informed video generation, and text-conditioned human motion demonstrate improvements in objective optimization and constraint satisfaction compared with several baselines.
Critical strengths
(1) A flexible framework for separating objective optimization from constraint enforcement. The proposed consensus-splitting formulation provides a practical way to coordinate differentiable objectives with potentially non-differentiable feasibility operators. Unlike methods that incorporate both requirements into a single guidance update, CoSCoDiff allows existing objective-guidance methods and task-specific constraint operators to be combined without retraining the diffusion model. This modularity is particularly useful for applications where feasibility constraints are difficult to express as differentiable losses.
(2) Promising empirical results across heterogeneous constrained-generation tasks. The controlled toy experiment demonstrates that consensus-based coordination can improve optimization while maintaining feasibility, particularly when the unconstrained optimum conflicts with the feasible region. The experiments on planning, video generation, and human motion further illustrate the applicability of the framework to different types of objectives and constraints. These results support the practical relevance of the proposed approach beyond a single application.
Critical weaknesses
(1) The contribution of consensus splitting is not sufficiently isolated from iterative refinement. Although CoSCoDiff performs favorably against several conventional baselines, its improvements over the closely matched Naive Iterate baseline are relatively limited in the more complex tasks. In trajectory planning and text-conditioned motion generation, CoSCoDiff-P achieves only modest improvements, while CoSCoDiff-PD sometimes performs worse. Since Naive Iterate already incorporates repeated SDEdit restarts and feasibility corrections, these results make it difficult to determine how much of the improvement is specifically attributable to consensus splitting. Furthermore, some comparisons use different semantic guidance objectives, making it difficult to attribute the larger performance differences to the proposed coordination mechanism rather than the underlying optimization objective.
(2) The connection between the stochastic optimal control formulation and the practical guidance update requires stronger mathematical justification. The proposed method is motivated as an approximation to a constrained stochastic optimal control problem, but several steps in the derivation are insufficiently justified. In particular, the path-space KL regularizer obtained through Girsanov's theorem appears inconsistent with the definition of the control as an additive drift term in the reverse SDE. Moreover, the proposed Tweedie response coefficient is introduced without establishing its relationship to the terminal effect of the local control. These issues are important because the optimal-control formulation is presented as a central theoretical motivation for the method, rather than merely an interpretation of a heuristic guidance procedure.
Questions for authors
(1) Isolating the contribution of consensus splitting. Since Naive Iterate already performs repeated objective-guided rollouts, SDEdit restarts, and feasibility corrections, could the authors provide a more systematic comparison isolating the effect of the consensus penalty and dual-variable updates? In particular, paired comparisons under identical computational budgets and uncertainty estimates for the improvements would help establish whether consensus splitting provides a meaningful advantage beyond iterative refinement.
(2) Fairness of the objective-guidance comparisons. In the text-conditioned motion experiment, CoSCoDiff explicitly optimizes an external text-motion embedding objective that is also used for evaluation, whereas some competing methods rely primarily on their native guidance mechanisms. Could the authors clarify how the comparison changes when the baselines are equipped with the same semantic objective and feasibility operator? An independent measure of generation quality would also help distinguish genuine improvements from direct optimization of the evaluation metric.
(3) Justification of the optimal-control approximation. Could the authors clarify the definition and scaling of the control variable in the stochastic optimal control formulation? When enters the reverse SDE as an additive drift control with diffusion coefficient , Girsanov's theorem gives a quadratic path-space KL regularizer proportional to , rather than . Is instead intended to represent a score-space perturbation? Relatedly, how is the proposed Tweedie response coefficient derived from the effect of a local control on the terminal clean-state prediction?
11 comments
3. SDEdit on/off ablations
We toggle SDEdit separately for Naive, CoSCoDiff-P, and CoSCoDiff-PD at fixed . Turning it off restarts each round from high noise while retaining coordination state and the same correction and return rules.
| Method | Toy: SDEdit on | Toy: SDEdit off | Simple2D: SDEdit on | Simple2D: SDEdit off |
|---|---|---|---|---|
| Naive Iterate | 0.4979 ± 0.0168 | 0.4957 ± 0.0123 | 98.67 ± 1.83 / 1.7945 ± 0.0219 | 99.33 ± 1.49 / 1.7971 ± 0.0236 |
| CoSCoDiff-P | 0.4928 ± 0.0166 | 0.4364 ± 0.0207 | 99.33 ± 1.49 / 1.6676 ± 0.0158 | 99.33 ± 1.49 / 1.6683 ± 0.0118 |
| CoSCoDiff-PD | 0.2130 ± 0.0093 | 0.4451 ± 0.0216 | 100.00 ± 0.00 / 1.6736 ± 0.0124 | 100.00 ± 0.00 / 1.6805 ± 0.0142 |
Table R3: SDEdit ablation at fixed . Mean ± sample SD; Toy: ; Simple2D: feasibility (%) / feasible path length . Toy feasibility is 100% throughout.
Without SDEdit, CoSCoDiff-P still reduces Simple2D feasible path length over Naive by 0.1246, supporting a consensus contribution beyond warm restarts. Table R3 shows that SDEdit benefits CoSCoDiff-PD but worsens CoSCoDiff-P on Toy. This interaction persists at 700 score calls: the paired SDEdit benefit is 0.2135 for CoSCoDiff-PD and −0.0743 for CoSCoDiff-P. Coordination and SDEdit thus have interacting effects rather than one component uniformly explaining the gains.
(Answer 2: Weakness 1 and Question 2) Objective-guidance fairness and independent new metric evaluation
To address objective-choice and evaluator-reuse concerns, we combine (1) baselines equipped with the same semantic objective and common Post-hoc correction, and (2) independent TMR evaluation of saved samples. The first tests whether objective matching closes the performance gap; the second checks whether gains extend beyond the optimized evaluation interface.
1. Same-objective controls
Naive/P/PD share the semantic objective and feasibility operator. We introduce ADMMDiff + T2M guidance and LGD-MC + Joint Objective, which add the same frozen T2M embedding objective to trajectory guidance. Their Post-hoc variants use the common path correction and motion re-encoding. Table R4 reports these objective-aligned comparisons. Their inference procedures differ, so Naive/P/PD remain the direct controls for isolating coordination effects.
2. Independent metric evaluation
We evaluate saved samples with independently pretrained TMR (Petrovich et al., 2023), unused in guidance or parameter/sample selection. TMR reports target cosine similarity. Native T2M is related to the optimized objective, but uses a skeleton round-trip rather than the direct feature-space loss.
| Method | Feasibility (%) | Native T2M | Independent TMR |
|---|---|---|---|
| ADMMDiff + T2M guidance | 0.00 | 5.1779±1.5680 | 0.4596±0.0562 |
| ADMMDiff + T2M guidance + Post-hoc | 100.00 | 5.3765±1.6849 | 0.4643±0.0539 |
| LGD-MC + Joint Objective | 0.00 | 5.1183±2.2520 | 0.5158±0.1261 |
| LGD-MC + Joint Objective + Post-hoc | 100.00 | 5.3933±1.5493 | 0.4929±0.0761 |
| Naive Iterate | 100.00 | 1.7271±0.6254 | 0.5169±0.0762 |
| CoSCoDiff-P | 100.00 | 1.7216±0.6183 | 0.6091±0.0848 |
| CoSCoDiff-PD | 100.00 | 1.9724±1.0611 | 0.6100±0.0835 |
Table R4: Same-semantic-objective comparisons on the same 100-case benchmark. Metrics are sample mean ± STD across cases; feasibility uses the common 2 cm path criterion. Naive/P/PD use .
Table R4 shows that matching the objective does not close the gap. CoSCoDiff-P and CoSCoDiff-PD are fully feasible, reach 1.72 and 1.97 native T2M against 5.178 and 5.118 for the T2M-guided baselines we add, and score 0.609 and 0.610 on TMR against at most 0.517 for the best feasible baseline. TMR provides an independent evaluation of text–motion alignment, rather than a comprehensive measure of motion quality.
(Answer 3: Weakness 2 and Question 3) Control scaling and the two approximations
Score-space versus drift-space control. The reviewer is correct for an additive drift control. In Eq. (6), however, is intended as a score-space perturbation:
+g(t)d\bar W_t.$$ The drift change is $-g^2(t)u_t$, not $u_t$. Under the same initial law, matched nondegenerate diffusion coefficients, and a progressively measurable state-feedback control satisfying the usual Girsanov absolute-continuity and integrability conditions ([Karatzas and Shreve](https://link.springer.com/book/10.1007/978-1-4612-0949-2)), $$D_{\mathrm{KL}}(P^u\\|P^0) =\frac12\mathbb E_{P^u}\int_0^T \left\\|\frac{-g^2(t)u_t}{g(t)}\right\\|_2^2dt =\frac12\mathbb E_{P^u}\int_0^T g^2(t)\\|u_t\\|_2^2dt.$$ Since the score-space perturbation enters the reverse drift as $-g^2(t)u_t$, its diffusion-normalized magnitude is $g(t)\\|u_t\\|$, regardless of the reverse-time sign convention. We assume $g(t)>0$ almost everywhere. If $u_t$ instead denotes an additive drift increment, the energy is $\frac12\mathbb E\int\\|u_t\\|^2/g^2(t)\\,dt$, as the reviewer states. We will explicitly distinguish these conventions; the identity concerns continuous path laws, not the finite-step sampler's exact KL. **Tweedie substitution is not a terminal-response theorem.** For the Gaussian forward kernel with $\alpha_t>0$, define the deterministic estimate $$\bar x_0:=\frac{x_t+\sigma_t^2[s_\theta(x_t,t)+u_t]}{\alpha_t} =\hat x_{0,\theta}(x_t,t)+\kappa_tu_t, \qquad\kappa_t=\frac{\sigma_t^2}{\alpha_t}.$$ VE uses $\alpha_t=1$. [Tweedie's identity (Efron, 2011)](https://pmc.ncbi.nlm.nih.gov/articles/PMC3325056/) justifies the uncontrolled posterior mean with an exact score; substituting $s_\theta+u_t$ is only an algebraic surrogate. A shared-noise reverse Euler step changes the next state by $g^2(t)\Delta t\\,u_t$, whose terminal effect additionally depends on nominal continuation. It need not equal $\kappa_tu_t$. We will define $\bar x_0$ as this deterministic estimate, not the actual controlled terminal mean, and provide the corresponding Euler-based response expansion and mismatch analysis in the revised appendix.
Response to Reviewer xQsv
We thank the reviewer for the detailed feedback. Below we address (1) component contributions under controlled budgets, (2) objective-guidance fairness and independent new metric evaluation, and (3) the theoretical basis of the guidance update and conditional approximation-error bounds.
(Answer 1: Weakness 1 and Question 1) Contributions beyond iterative refinement
To determine whether coordination adds value beyond repeated refinement, we combine (1) consensus-penalty and dual controls, (2) matched score-call budget comparisons, and (3) SDEdit on/off ablations. These comparisons separate component contributions from additional computation and test their interaction with warm restarts.
1. Consensus-penalty and dual controls
The objective, restart/correction schedule, and final-return rule are fixed within each task. Naive Iterate already includes iterative refinement; CoSCoDiff-P adds the penalty, and CoSCoDiff-PD adds dual in guidance and correction. CoSCoDiff-PD (ρ=0) retains correction dual but removes consensus guidance.
Table R1: Component ablation. Mean ± sample STD across seeds (human motion reports result across cases). Planning length uses feasible trajectories only. All included Toy/motion outputs are feasible.
Table R1 shows that the consensus penalty and dual feedback contribute on different tasks. On Simple2D, the penalty enhance the feasibility and shortens feasible paths relative to Naive Iterate by 0.1269, while dual feedback alone gives no gain. On Toy, the large gain persists without the consensus penalty, supporting a contribution from correction dual feedback, since CoSCoDiff-PD (ρ=0) reaches 0.1805 while CoSCoDiff-P stays at 0.4928. On human motion, CoSCoDiff-P has a slightly lower mean native T2M than Naive, while historical dual feedback may be harmful. Hence, the contribution of each component is task-dependent.
Return-rule correction. Submitted Simple2D Naive/P runs used best-iterate selection; these entries are replaced by final-iterate reruns. Archived CoSCoDiff-PD and the new ρ=0 control return the final iterate. This is a reporting correction, not an algorithmic gain.
2. Matched score-call budgets
To test whether coordination gains persist under the same score-evaluation budget, we compare Naive, CoSCoDiff-P, and CoSCoDiff-PD at 300, 500, and 700 score calls on Toy (Table R2). At each budget, all methods share the same seeds, SDEdit and correction schedules, and final-iterate return rule.
Table R2: Toy score-call-budget comparison. Paired improvement is dopt(Naive)−dopt(P/PD), mean [95% seed-bootstrap CI] across seeds. Budgets of 300/500/700 calls correspond to 6/11/16 completed outer rounds; each uses the final iteration output.
CoSCoDiff-PD retains a substantial Toy advantage at every tested score-call budget, whereas CoSCoDiff-P gives smaller gains. These improvements do not require additional score evaluations. The intervals are pointwise without multiplicity correction; matching score calls does not equalize all computational overhead.