- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Lagrangian Dual Flows, a method for enforcing nonlinear constraints on pretrained flow matching models at inference time. The main idea is to introduce a dual variable that evolves alongside the generated samples, allowing constraints to be enforced without expensive projection or optimization at every sampling step. The authors also provide theoretical convergence guarantees under certain assumptions. Experiments on synthetic distributions, MNIST inpainting, obstacle avoidance, and robot locomotion show that LDF achieves low constraint violations while being significantly faster than several existing constrained generation methods.
Critical strengths
- I like that the method only requires augmenting the original flow ODE with dual dynamics, without solving optimization problems at every step such as in PCFM. The runtime improvements over projection-based and CBF-based methods are quite convincing.
- The paper provides convergence analysis for both equality and inequality constraints and demonstrates the method across several different applications. The results suggest that LDF could be useful for constrained generation, especially in robotics and planning.
Critical weaknesses
- The convergence proofs require a full-rank constraint Jacobian and a small-variation assumption on its Gram matrix. It is unclear how restrictive these assumptions are for general nonlinear constraints.
- While the constraint satisfaction results are good, most experiments focus on violation and runtime rather than the quality of the resulting distribution or downstream performance.
Questions for authors
- How restrictive is the small-variation assumption in practice? Can the authors verify whether it holds for the nonlinear constraints used in the experiments, particularly obstacle avoidance?
- Since the experiments use the reduced dynamics in Eq. (7), is there any theoretical or empirical evidence that they retain the convergence properties of Eq. (6)?
- How much does the optional final projection contribute to the results? Table 1 shows that LDF without projection sometimes has noticeably larger violations. Could the authors compare the methods under matched constraint tolerances and clarify when projection is necessary?
3 comments
Response to Reviewer pXoY (2/2)
Q3: Contribution of the final projection. Table 1 now reports projection variants of all applicable algorithms. CBF and PCFM already project at each substep of the denoising process, so an extra projection at the end would not change their results. We reproduce the equality-constrained portion of the updated table below. The full table, including the inequality-constrained examples, is in the revised manuscript.
| Method | Star: Time | Star: Viol. | MNIST: Time | MNIST: Viol. |
|---|---|---|---|---|
| Penalty | 6.4 | 102.1 | ||
| + proj. | 6.6 | 103.8 | ||
| TV penalty | 6.0 | failed | 100.9 | failed |
| + proj. | 5.3 | failed | 102.4 | failed |
| PCFM | 25.2 | 147.5 | ||
| ΠGDM | 10.4 | 327.1 | ||
| + proj. | 10.7 | 329.0 | ||
| OPC | 20.1 | 337.8 | ||
| + proj. | 20.1 | 339.4 | ||
| DPS | 6.1 | 200.0 | ||
| + proj. | 6.2 | 202.0 | ||
| LDF (ours) | 6.4 | 102.4 | ||
| + proj. (ours) | 6.5 | 103.7 |
Mean per-sample generation time (ms) and constraint violation across 20 samples. "+ proj." rows apply a final Gauss-Newton projection to the samples of the method above. Bold times are within 20% of the lowest time for that problem; bold violations are below (near machine precision for single-precision float). failed: every sample produced a NaN error; such methods are excluded from the bold time comparison.
The "+ proj." rows also provide the matched-tolerance comparison. On the equality-constrained examples, LDF, the penalty baseline, ΠGDM, and OPC all reach very tight constraint satisfaction () after projection. Among these, LDF + proj. is the fastest or within 20% of the fastest on every example.
Projection is not always sufficient on its own, however. The inpainting-style methods (ΠGDM, OPC, DPS) can achieve tight constraint satisfaction with the final projection on these examples, but if the unconstrained sample is too far from the constraint set, projection can struggle or fail to enforce the constraint. We see this with DPS on the star example, where the violation after projection remains at . Projection can also degrade sample quality even when the constraints are technically satisfied. The revised appendix includes ImageNet inpainting examples with and without projection that illustrate this effect.
In summary, LDF without projection already drives the violation to small values, and the final projection serves to reach machine precision at negligible additional cost. Because LDF samples are already close to the constraint set, the projection is reliable and does not noticeably change the samples.
Thanks for the response and the extra experiments. The ImageNet results help address my concern about sample quality, and the updated comparisons clarify how much the final projection contributes. I’ll keep my rating at weak accept as the comparison with baselines shows a similar performance result and the robotics examples leaves open on whether the generated trajectories are physically feasible and perform well when executed and I’m still unsure how broadly the theoretical conditions apply to nonlinear constraints.
Response to Reviewer pXoY (1/2)
Thank you for your careful reading of our paper. We are glad you found the paper clear and easy to follow! Our revised paper has its changes shown in magenta. Below are our responses to your weaknesses and comments:
W1: Assumptions on the constraint Jacobian. We thank the reviewer for raising this point. Both are fair questions and we have clarified them in the revision.
Assumption (A2) is the continuous-time analog of the linear independence constraint qualification (LICQ) used in nonlinear programming. When the rows of Jg become linearly dependent, the multiplier enforcing g(x)=0 is not unique and may be unbounded, and the feasible set {x∣g(x)=0} need not be a smooth surface. We therefore view (A2) as a natural requirement.
We agree that the small-variation assumption (A4) deserves more discussion. It is a sufficient condition for constraint satisfaction, not a necessary one. Its role in the proof of Theorem 1 is to ensure that a single quadratic Lyapunov function decreases along trajectories. In the revised version, we have weakened the small-variation condition (A4), and its inequality-constrained counterpart (B4), so that it need only hold on a final interval [t1,1) rather than the full integration period [0,1). The constraint Jacobian may therefore vary arbitrarily during the early part of generation. While this is still only a sufficient condition, we would like to stress that it captures the important case of affine constraints. A more comprehensive analysis of alternative sufficient conditions is an interesting future direction.
W2: Distribution quality and downstream performance. We thank the reviewer for this suggestion. To quantify the quality of the generated distribution, we have added FID and KID metrics for the new large-scale ImageNet inpainting example (Table 2 of the revised paper, reproduced below). These metrics are computed both over the full image and over the inpainted region alone.
Image inpainting consistency metrics. FID and KID compare outputs with the reference images over the whole image and over the missing region (hole). "Viol." is mean absolute error on observed pixels.
LDF achieves the best full-image FID and KID and the best hole KID among all methods, and is competitive in hole FID, while also achieving low constraint violation. Several of these baselines are explicitly designed to sample from the conditional distribution.
We emphasize that this is an empirical observation: LDF is designed for constraint satisfaction, and we do not provide theoretical guarantees on the resulting distribution. Nonetheless, these results suggest that LDF preserves the quality of the pretrained model well in practice, and we have added a discussion of this point to the revised manuscript.
Q1: Restrictiveness of the small-variation assumption. We have relaxed the small-variation assumption so that it need only hold on a final interval [t1,1) rather than on the full integration interval. For the obstacle avoidance experiment, we found that the original sufficient condition was not satisfied, but that the new one is satisfied if we let t1 be sufficiently close to 1. However, in some cases the condition was only barely met, and verifying it numerically required a very small step size and 64-bit precision; LDF itself was run with the same settings as in Table 1. We provide more empirical analysis on the small-variation assumption in Appendix B.
Again, we would like to stress that this small-variation assumption is merely a sufficient condition for constraint satisfaction, and there may be alternative conditions that establish constraint satisfaction for certain structured classes of problems. The purpose of these nonlinear experiments is primarily to test if we see constraint satisfaction even when these sufficient conditions are not (immediately) satisfied.
Q2: Convergence of the reduced dynamics. Thank you for your suggestion! After we submitted our first version, we were working on proving convergence for these dynamics and we now have a proof. We have added the theorem to the main text; it is Theorem 4 there and the proof is in Appendix C. It uses the same assumptions as Theorem 3.