- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies collaborative generation with frozen pretrained diffusion models, producing a wide image from text, a texture for a supplied mesh, or a single image supporting two prompted meanings under a specified transformation. SyncVC formulates a reward-tilted posterior over the participating patches or views and balances consistency against deviation from the pretrained priors. Its sampler combines predecessor-conditioned transitions with additive controls optimized at each denoising timestep using Tweedie estimates and a DDIM surrogate.
Critical strengths
Individually plausible outputs need not form a coherent panorama or texture. Separating the consistency objective from the choice of trajectory dependencies is a clear way to organize this problem. The method reuses frozen models and accommodates different differentiable constraints.
The paper also instantiates the method for mesh texturing and optical illusions instead of demonstrating only one application.
Critical weaknesses
-
The theoretical justification does not yet match the practical sampler. The derivation assumes a joint initialization and transition model that are not fully consistent with the shared-noise initialization and DDIM surrogate used in practice. In particular, the implemented regularizer changes the weighting of the theoretical objective, and a finite mean-discrepancy penalty does not retain the same KL interpretation for deterministic transitions. These choices may be empirically effective, but their relationship to the claimed variational objective needs to be clarified to establish what theoretical justification actually carries over to the sampling algorithm.
-
The evaluation does not yet establish robust task-level gains. Appearance similarity and distributional quality do not directly measure seam continuity, cross-view consistency, or recognition of both intended meanings, and the reliability of the reported advantages remains uncertain.
-
The technical advance over DTM/NDTM and SyncSDE appears limited. DTM/NDTM already optimizes additive controls during sampling using a terminal reward, a penalty for deviation from the pretrained prior, and Tweedie estimates to avoid full rollouts. SyncSDE already conditions a trajectory on preceding trajectories through a task-dependent conditional-score construction. SyncVC combines these ingredients by optimizing controls within predecessor-conditioned transitions. Moreover, treating all patches as one joint state and defining an overlap-consistency reward already gives a direct route to a multi-patch variational-control objective; writing this objective does not by itself introduce a new inference principle. The distinctive choices are therefore the dependency structure and the greedy optimization across patches, but the paper does not clearly establish what substantive technical difficulty these choices resolve beyond adapting the existing control framework to synchronization.
Questions for authors
See weakness
4 comments
Response to Reviewer oRYS (2/2)
W3. The technical advance over DTM/NDTM and SyncSDE appears limited. [...]
Response: We respectfully disagree that our contributions are limited to adapting an existing control framework. We address each point below.
-
Core contributions. We agree that the per-step objective has the same structure as DTM/NDTM for a single trajectory and that the coupling in Eq. (7) follows SyncSDE. We do not claim these as contributions. To the best of our knowledge, SyncVC is the first framework to formulate diffusion synchronization as variational inference over coupled trajectories, whereas prior synchronization methods rely on heuristic or task-specific constructions. The resulting ELBO specifies what synchronized trajectories should optimize and does not depend on how they are coupled. Modeling the transition as in SyncSDE is a design choice. The objective in Eq. (5) does not depend on how the transition is parameterized, so our framework is more general and admits other designs.
-
SyncVC vs SyncSDE. The core idea behind SyncVC is to optimize controls across trajectories to maximize the overall reward over patches. To assess the importance of this idea, we set , which keeps the SyncSDE-style coupling in Eq. (7) and removes only the control optimization. As shown in Table C1, this degrades all five metrics to a level comparable to SyncSDE: the -histogram distance rises from to and the Intra-Style-Loss from to , against and for SyncSDE in Table 1 of the main paper. The gains over SyncSDE therefore come from control optimization rather than the shared coupling.
-
Greedy vs direct joint optimization. We also evaluated the direct route the reviewer describes: as shown in Table C2, optimizing the controls of all patches jointly (i.e., updating the controls of all patches simultaneously at each timestep) degrades KID from to and the -histogram distance from to . Full-rollout joint optimization also exceeds GPU memory at the default and is worse on four of the five metrics at (Table 5 of the main paper).
-
What the distinctive choices resolve. The autoregressive factorization and greedy optimization let each patch align with an already-synchronized earlier patch, which makes the objective tractable and stable. The dependency structure matters as well: a bidirectional factorization is worse on wide images (Table C2) and on meshes (Table 3b of the main paper).
| Setting | Intra-LPIPS | Intra-Style-Loss | -Histogram dist. | Histogram Intersect. | KID |
|---|---|---|---|---|---|
| SyncVC | 0.59 | 44.34 | 0.75 | 0.67 | 52.07 |
| 0.67 | 65.26 | 1.38 | 0.51 | 55.72 |
Table C1. Quantitative analysis on the effect of control optimization for wide image generation. KID values are scaled by .
| Method | Intra-LPIPS | Intra-Style-Loss | -Histogram dist. | Histogram Intersect. | KID |
|---|---|---|---|---|---|
| SyncVC (autoregressive, greedy) | 0.59 | 44.34 | 0.75 | 0.67 | 52.07 |
| Joint optimization | 0.59 | 70.49 | 1.16 | 0.57 | 112.01 |
| Bidirectional factorization | 0.62 | 58.56 | 0.88 | 0.63 | 52.17 |
Table C2. Ablation on control optimization and model factorization for wide image generation (Table 3a of the main paper). KID values are scaled by .
The rebuttal clarifies the limitations of the variational interpretation, and the additional ablations strengthen the evidence that control optimization and regularization improve the reported results. However, the existing metrics still provide only indirect evidence of seam continuity, cross-view consistency, and successful recognition of both intended meanings. Large sample counts alone do not establish the reliability of the reported differences. The novelty concern also remains: showing that simultaneous control updates perform worse than greedy updates does not establish a substantial advance beyond adapting DTM/NDTM with the coupling used in SyncSDE. Hence, I maintain my rating of 2 (Weak rejection).
Thank you for reading our rebuttal closely and for explaining your remaining concerns. We appreciate your recognition that the revision clarifies the scope of the variational interpretation and that the new ablations strengthen the evidence for control optimization and regularization.
Before the author-reviewer discussion phase closes, we would like to emphasize about the contribution again.
- To the best of our knowledge, SyncVC is the first framework that formulates synchronization as variational inference over coupled diffusion trajectories, with a joint consistency reward that can be specified independently of how the trajectories are coupled.
- The autoregressive sampler makes this formulation practical by letting each patch optimize against an already synchronized predecessor. This addresses a concrete difficulty in applying trajectory control to multiple patches: full-rollout joint optimization exceeded GPU memory at our default number of denoising steps, and simultaneous per-step updates performed worse on both KID and color consistency (Table C2).
We hope this distinction between the joint formulation and the practical role of the sampler is helpful as you consider the contribution. Thank you again for the thoughtful response.
Response to Reviewer oRYS (1/2)
Response: We agree and have made the following three deviations from the theory more explicit in the revised main text:
Deterministic transitions. The KL terms in Eq. (5) are finite only for stochastic transitions. For deterministic samplers such as DDIM (η=0), the implemented mean-discrepancy penalty is therefore a surrogate motivated by the ELBO rather than the ELBO itself, although it works well in practice.
Weighting of the regularizer. For stochastic transitions, each KL term equals the mean discrepancy scaled by 1/(2σt2). The implemented objective (Eq. (20)) absorbs this factor into a single constant λ and uses an upper bound on the penalty (Eq. (19)), so it reweights the per-step terms of Eq. (5) rather than matching them exactly. However, this reweighting works well in practice.
Initial noise. The derivation assumes independent initial noise per patch, whereas our experiments crop all patches from one noise canvas, following the baselines. However, each patch is still marginally standard Gaussian and the initial-noise term does not depend on the controls, so the optimized objective is unchanged. The derivation then carries over when the reference process uses the same shared initialization.
With these caveats, what carries over from the theory is the form of the objective: we optimize a terminal reward together with a per-step penalty on deviations from the pretrained prior.
Response: We respectfully disagree that the evaluation does not establish task-level gains, and address each point below.
Reliability of the reported gains. We note that all results are averaged over large sample sets for all the tasks considered in this paper: 750 wide images per method (15 prompts, 50 images each), 350 mesh--prompt pairs, and 500 illusion images. The reported averages are therefore stable.
Seam continuity (wide images). Intra-LPIPS, Intra-Style-Loss and the two histogram metrics compare non-overlapping crops of the same image, so they penalize the appearance shifts that make seams visible. SyncVC is best on all of these metrics (Table 1 of the main paper). The small-overlap setting (Figure 4 of the main paper), where seams are most likely, shows the same trend qualitatively.
Cross-view consistency (meshes). We compute FID and KID on renderings from 10 viewpoints that differ from the eight views used for generation, so inconsistencies between generated views appear as artifacts and are penalized (Table 2 of the main paper).
Both intended meanings (illusions). We compute FID and KID for each view against reference images of that view's prompt and then average, so an image that depicts only one meaning is penalized (Table 4 of the main paper).
We agree that metrics measuring these properties directly would be valuable, and we have noted this as future work in the conclusion of the revised manuscript.