acceptodds
Under review as a conference paper at ICLR 2027

Variational Test-time Optimization for Diffusion Synchronization

Abstract

Collaborative generation, which coordinates multiple diffusion trajectories to extend the capabilities of pretrained priors, has emerged as a powerful paradigm for extending the applicability of diffusion models. Among existing approaches, diffusion synchronization provides a scenario-agnostic solution by introducing general guidance mechanisms. However, existing approaches rely heavily on heuristics and still require task-specific tailoring, which limits their generalizability and performance. In this work, we mathematically derive a synchronization framework based on optimal control, providing a principled explanation of diffusion synchronization. During sampling, we optimize control variables to guide multiple trajectories toward coherent solutions while remaining close to the underlying diffusion prior. Our method operates entirely at test time without additional training, thereby enabling broad applicability across diverse scenarios. Experimental results show that our method consistently improves generation quality. For wide image generation, our method reduces color and style inconsistency by 42.7% and 26.7% relative to the strongest baseline. It also achieves the best distributional alignment among the baselines for 3D mesh texturing. These results establish a principled framework for leveraging pretrained generative models for collaborative generation.

Primary area:
generative models
Keywords:
diffusion, synchronization, image generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oRYSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies collaborative generation with frozen pretrained diffusion models, producing a wide image from text, a texture for a supplied mesh, or a single image supporting two prompted meanings under a specified transformation. SyncVC formulates a reward-tilted posterior over the participating patches or views and balances consistency against deviation from the pretrained priors. Its sampler combines predecessor-conditioned transitions with additive controls optimized at each denoising timestep using Tweedie estimates and a DDIM surrogate.

Critical strengths

Individually plausible outputs need not form a coherent panorama or texture. Separating the consistency objective from the choice of trajectory dependencies is a clear way to organize this problem. The method reuses frozen models and accommodates different differentiable constraints.

The paper also instantiates the method for mesh texturing and optical illusions instead of demonstrating only one application.

Critical weaknesses
  1. The theoretical justification does not yet match the practical sampler. The derivation assumes a joint initialization and transition model that are not fully consistent with the shared-noise initialization and DDIM surrogate used in practice. In particular, the implemented regularizer changes the weighting of the theoretical objective, and a finite mean-discrepancy penalty does not retain the same KL interpretation for deterministic transitions. These choices may be empirically effective, but their relationship to the claimed variational objective needs to be clarified to establish what theoretical justification actually carries over to the sampling algorithm.

  2. The evaluation does not yet establish robust task-level gains. Appearance similarity and distributional quality do not directly measure seam continuity, cross-view consistency, or recognition of both intended meanings, and the reliability of the reported advantages remains uncertain.

  3. The technical advance over DTM/NDTM and SyncSDE appears limited. DTM/NDTM already optimizes additive controls during sampling using a terminal reward, a penalty for deviation from the pretrained prior, and Tweedie estimates to avoid full rollouts. SyncSDE already conditions a trajectory on preceding trajectories through a task-dependent conditional-score construction. SyncVC combines these ingredients by optimizing controls within predecessor-conditioned transitions. Moreover, treating all patches as one joint state and defining an overlap-consistency reward already gives a direct route to a multi-patch variational-control objective; writing this objective does not by itself introduce a new inference principle. The distinctive choices are therefore the dependency structure and the greedy optimization across patches, but the paper does not clearly establish what substantive technical difficulty these choices resolve beyond adapting the existing control framework to synchronization.

Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer oRYS (1/2)

W1. The theoretical justification does not yet match the practical sampler. [...]

Response: We agree and have made the following three deviations from the theory more explicit in the revised main text:

  1. Deterministic transitions. The KL terms in Eq. (5) are finite only for stochastic transitions. For deterministic samplers such as DDIM (η=0\eta=0), the implemented mean-discrepancy penalty is therefore a surrogate motivated by the ELBO rather than the ELBO itself, although it works well in practice.

  2. Weighting of the regularizer. For stochastic transitions, each KL term equals the mean discrepancy scaled by 1/(2σt2)1/(2\sigma_t^2). The implemented objective (Eq. (20)) absorbs this factor into a single constant λ\lambda and uses an upper bound on the penalty (Eq. (19)), so it reweights the per-step terms of Eq. (5) rather than matching them exactly. However, this reweighting works well in practice.

  3. Initial noise. The derivation assumes independent initial noise per patch, whereas our experiments crop all patches from one noise canvas, following the baselines. However, each patch is still marginally standard Gaussian and the initial-noise term does not depend on the controls, so the optimized objective is unchanged. The derivation then carries over when the reference process uses the same shared initialization.

With these caveats, what carries over from the theory is the form of the objective: we optimize a terminal reward together with a per-step penalty on deviations from the pretrained prior.

W2. The evaluation does not yet establish robust task-level gains. [...]

Response: We respectfully disagree that the evaluation does not establish task-level gains, and address each point below.

  • Reliability of the reported gains. We note that all results are averaged over large sample sets for all the tasks considered in this paper: 750 wide images per method (15 prompts, 50 images each), 350 mesh--prompt pairs, and 500 illusion images. The reported averages are therefore stable.

  • Seam continuity (wide images). Intra-LPIPS, Intra-Style-Loss and the two histogram metrics compare non-overlapping crops of the same image, so they penalize the appearance shifts that make seams visible. SyncVC is best on all of these metrics (Table 1 of the main paper). The small-overlap setting (Figure 4 of the main paper), where seams are most likely, shows the same trend qualitatively.

  • Cross-view consistency (meshes). We compute FID and KID on renderings from 10 viewpoints that differ from the eight views used for generation, so inconsistencies between generated views appear as artifacts and are penalized (Table 2 of the main paper).

  • Both intended meanings (illusions). We compute FID and KID for each view against reference images of that view's prompt and then average, so an image that depicts only one meaning is penalized (Table 4 of the main paper).

We agree that metrics measuring these properties directly would be valuable, and we have noted this as future work in the conclusion of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer oRYS (2/2)

W3. The technical advance over DTM/NDTM and SyncSDE appears limited. [...]

Response: We respectfully disagree that our contributions are limited to adapting an existing control framework. We address each point below.

  • Core contributions. We agree that the per-step objective has the same structure as DTM/NDTM for a single trajectory and that the coupling in Eq. (7) follows SyncSDE. We do not claim these as contributions. To the best of our knowledge, SyncVC is the first framework to formulate diffusion synchronization as variational inference over coupled trajectories, whereas prior synchronization methods rely on heuristic or task-specific constructions. The resulting ELBO specifies what synchronized trajectories should optimize and does not depend on how they are coupled. Modeling the transition as in SyncSDE is a design choice. The objective in Eq. (5) does not depend on how the transition is parameterized, so our framework is more general and admits other designs.

  • SyncVC vs SyncSDE. The core idea behind SyncVC is to optimize controls across trajectories to maximize the overall reward over patches. To assess the importance of this idea, we set u=0u=0, which keeps the SyncSDE-style coupling in Eq. (7) and removes only the control optimization. As shown in Table C1, this degrades all five metrics to a level comparable to SyncSDE: the χ2\chi^2-histogram distance rises from 0.750.75 to 1.381.38 and the Intra-Style-Loss from 44.3444.34 to 65.2665.26, against 1.311.31 and 63.9863.98 for SyncSDE in Table 1 of the main paper. The gains over SyncSDE therefore come from control optimization rather than the shared coupling.

  • Greedy vs direct joint optimization. We also evaluated the direct route the reviewer describes: as shown in Table C2, optimizing the controls of all patches jointly (i.e., updating the controls of all patches simultaneously at each timestep) degrades KID from 52.0752.07 to 112.01112.01 and the χ2\chi^2-histogram distance from 0.750.75 to 1.161.16. Full-rollout joint optimization also exceeds GPU memory at the default T=50T=50 and is worse on four of the five metrics at T=8T=8 (Table 5 of the main paper).

  • What the distinctive choices resolve. The autoregressive factorization and greedy optimization let each patch align with an already-synchronized earlier patch, which makes the objective tractable and stable. The dependency structure matters as well: a bidirectional factorization is worse on wide images (Table C2) and on meshes (Table 3b of the main paper).

SettingIntra-LPIPS ↓\downarrowIntra-Style-Loss ↓\downarrowχ2\chi^2-Histogram dist. ↓\downarrowHistogram Intersect. ↑\uparrowKID ↓\downarrow
SyncVC0.5944.340.750.6752.07
u=0u=00.6765.261.380.5155.72

Table C1. Quantitative analysis on the effect of control optimization for wide image generation. KID values are scaled by 10310^3.

MethodIntra-LPIPS ↓\downarrowIntra-Style-Loss ↓\downarrowχ2\chi^2-Histogram dist. ↓\downarrowHistogram Intersect. ↑\uparrowKID ↓\downarrow
SyncVC (autoregressive, greedy)0.5944.340.750.6752.07
Joint optimization0.5970.491.160.57112.01
Bidirectional factorization0.6258.560.880.6352.17

Table C2. Ablation on control optimization and model factorization for wide image generation (Table 3a of the main paper). KID values are scaled by 10310^3.

The rebuttal clarifies the limitations of the variational interpretation, and the additional ablations strengthen the evidence that control optimization and regularization improve the reported results. However, the existing metrics still provide only indirect evidence of seam continuity, cross-view consistency, and successful recognition of both intended meanings. Large sample counts alone do not establish the reliability of the reported differences. The novelty concern also remains: showing that simultaneous control updates perform worse than greedy updates does not establish a substantial advance beyond adapting DTM/NDTM with the coupling used in SyncSDE. Hence, I maintain my rating of 2 (Weak rejection).

Thank you for reading our rebuttal closely and for explaining your remaining concerns. We appreciate your recognition that the revision clarifies the scope of the variational interpretation and that the new ablations strengthen the evidence for control optimization and regularization.

Before the author-reviewer discussion phase closes, we would like to emphasize about the contribution again.

  • To the best of our knowledge, SyncVC is the first framework that formulates synchronization as variational inference over coupled diffusion trajectories, with a joint consistency reward that can be specified independently of how the trajectories are coupled.
  • The autoregressive sampler makes this formulation practical by letting each patch optimize against an already synchronized predecessor. This addresses a concrete difficulty in applying trajectory control to multiple patches: full-rollout joint optimization exceeded GPU memory at our default number of denoising steps, and simultaneous per-step updates performed worse on both KID and color consistency (Table C2).

We hope this distinction between the joint formulation and the practical role of the sampler is helpful as you consider the contribution. Thank you again for the thoughtful response.

Reviewer_wu93OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SyncVC, which treats diffusion synchronization (wide images, mesh texturing, optical illusions) as test-time optimization. Patches are generated sequentially with a SyncSDE-style coupling, and at each step a control on the input is optimized to increase a consistency reward on Tweedie estimates, with a penalty for deviating from the pretrained transition. The penalty is motivated by an ELBO over the joint trajectories. The wide-image results are clearly better than those of the baselines in the paper.

Critical strengths
  • The consistency gains on wide images are large (e.g., χ2\chi^2 histogram distance 0.75 vs. 1.21 for MultiDiffusion, Table 1). The metrics are computed between non-overlapping crops, which reflect global consistency in color and style and not just the seams.

  • The runtime trade-off is reported in detail. The default setting is about 3.4×3.4\times slower than MultiDiffusion, but with fewer steps (T=18T=18, K=2K=2) it is faster than MultiDiffusion and still more consistent, at a slightly worse KID (Tables 6–7).

Critical weaknesses
  • My main concern is what the variational part adds. The coupling in Eq. (7) follows SyncSDE, and Eq. (20) has the same structure as the DDIM objective of NDTM [2]: a control on the input, a penalty on the deviation from the pretrained prediction, and greedy per-step optimization. Bringing these together for synchronization can be a useful contribution in itself, but since the penalty is what the variational view adds, it would be important to show that it matters. However, there is no run with λ=0\lambda = 0 and everything else fixed: Table 3 changes the factorization and the optimization scheme, and the λ\lambda sweep in Figure 17 covers only 0.1–0.5 on illusions. One possibility is that the small optimization budget (five Adam steps from u=0u = 0) already keeps the controls small, in which case the penalty would matter little.

  • SyncDiffusion [1] is not compared, although I think it is the most direct baseline: it also guides each window with the gradient of a consistency loss on Tweedie predictions, and the Intra-LPIPS and Intra-Style-Loss metrics used here come from it. The paper describes it as task-specific, but since task-specific baselines are already included for meshes and illusions, a comparison on wide images seems feasible.

  • It would help to state more clearly how far the ELBO derivation covers the implemented sampler. The experiments use deterministic DDIM (η=0\eta = 0), for which the KL terms in Eq. (5) are infinite whenever the controlled and pretrained means differ. Appendix B mention about this and keeps a mean-discrepancy penalty, which is reasonable, but the final objective is then an upper-bound surrogate with tuned weights rather than the ELBO itself. The derivation also assumes independent initial noise per patch, while the experiments crop all patches from one noise canvas. Making this distinction explicit in the main text would make the theoretical claims more precise.

[1] Lee et al., “SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions”, NeurIPS 2023.

[2] Pandey et al., “Variational Control for Guidance in Diffusion Models”, ICML 2025.

Questions for authors
  • Could the authors report a λ=0\lambda = 0 run on wide images with everything else fixed? A u=0u = 0 run in the same setup would also separate the effect of the controls from that of the coupling.

  • Could the authors compare with SyncDiffusion [1] on a subset of the wide-image prompts (same backbone, resolution and initialization), including runtime? These two experiments would most affect my assessment.

  • SyncTweedies [3] shows theoretically and empirically that MultiDiffusion and SyncTweedies are equivalent under matched conditions (the same crop layout, initial noise, denoiser and deterministic DDIM sampling), and their images in Figure 3 are indeed very similar, yet the two differ clearly in Table 1 (Intra-Style-Loss 58.46 vs. 78.05). Could the authors clarify which of these conditions differ in their setup?

  • Figures 5 and 14 show each textured mesh from a single viewpoint, and I could not find the project website mentioned in the appendix (there is no supplementary material). Could the authors render the same textured meshes shown in these figures from several viewpoints, e.g., the 10 views used for evaluation? This would help to assess coherence and the texture quality on the unseen sides.

  • As a minor point, could the authors clarify which baseline the 42.7% and 26.7% reductions in the abstract refer to? From Table 1, I get about 38% and 24% relative to MultiDiffusion.

[3] Kim et al., “SyncTweedies: A General Generative Framework Based on Synchronized Diffusions”, NeurIPS 2024.

5 comments
AuthorsOpenReview ↗

Response to Reviewer wu93 (1/2)

W1. My main concern is what the variational part adds. [...]

Q1. Could the authors report a λ=0\lambda = 0 run on wide images with everything else fixed? [...]

Response: [W1, Q1] On the relation to NDTM and SyncSDE: We agree that the coupling follows SyncSDE and that the per-step objective has the same structure as NDTM for a single trajectory. What the variational view adds is the joint objective over coupled trajectories, with a prior-deviation penalty and optimized controls. We show below that both matter on wide image generation.

To isolate the benefits of the variational formulation, we present the following ablations on wide image generation, keeping all other settings fixed at their defaults (also see Figure 19 of the revised manuscript).

Effect of λ\lambda: We sweep λ∈{0,0.5,1.0,1.5,2.0}\lambda \in \{0, 0.5, 1.0, 1.5, 2.0\}, where λ=2.0\lambda=2.0 is our default. As shown in Table B1, KID degrades monotonically as λ\lambda decreases, and removing the explicit penalty by setting λ=0\lambda=0 raises KID from 52.0752.07 to 65.1365.13 (a 25%25\% increase). Therefore, the five-step optimization budget from λ=0\lambda=0 does not by itself keep the controlled process close to the pretrained prior, and the explicit prior-deviation penalty is needed for distributional alignment.

λ\lambda00.51.01.52.0 (default)
KID ↓\downarrow65.1358.8158.6858.2152.07

Table B1. Effect of λ\lambda on KID for wide image generation. KID values are scaled by 10310^3.

Effect of control optimization (u=0u=0): Similarly, setting u=0u=0 keeps the SyncSDE-style coupling in Eq. (7) of the main paper and removes only the control optimization. As shown in Table B2, this degrades all five reported metrics to a level comparable to SyncSDE (included for reference from Table 1 of the main paper), so the gains come from control optimization rather than the coupling.

MethodIntra-LPIPS ↓\downarrowIntra-Style-Loss ↓\downarrowχ2\chi^2-Histogram dist. ↓\downarrowHistogram Intersect. ↑\uparrowKID ↓\downarrow
SyncVC0.5944.340.750.6752.07
u=0u=00.6765.261.380.5155.72
SyncSDE0.6563.981.310.5357.08

Table B2. Ablation of the control optimization (u=0u=0) on wide image generation. SyncSDE results are taken from Table 1 of the main paper. KID values are scaled by 10310^3.

W2. SyncDiffusion is not compared, although I think it is the most direct baseline [...]

Q2. Could the authors compare with SyncDiffusion on a subset of the wide-image prompts [...]

Response: [W2, Q2] Based on the reviewers' feedback, we have added a comparison with SyncDiffusion. For fair comparison, we use an LPIPS-augmented reward as a drop-in replacement for the default reward in SyncVC to account for SyncDiffusion's explicit optimization of perceptual consistency. Specifically, the adjacent-patch overlap reward in SyncVC is extended as follows:

$ \tilde{r}_n(y,x_0^{(1:n)}) = -\frac{\gamma}{2}|M^{(n)}\odot(f(x_0^{(n-1)},y,n)-x_0^{(n)})|_2^2 $

$\qquad \qquad \qquad -\lambda_{\mathrm{LPIPS}}\cdot\mathrm{LPIPS}(\mathcal{D}(x_0^{(n)}),\mathcal{D}(x_0^{(1)})),\quad n\geq2, $

where D\mathcal{D} denotes the Stable Diffusion VAE decoder and λLPIPS\lambda_{\mathrm{LPIPS}} weights the perceptual term. The rest of the SyncVC method remains unchanged. While SyncDiffusion uses the middle patch as its anchor, SyncVC uses the first patch to respect its autoregressive formulation. We use K=2K=2 optimization steps per denoising step and set λLPIPS=104\lambda_{\mathrm{LPIPS}}=10^4 for SyncVC and compare both methods in Table B3 below. We use the same 15 prompts and generate 50 images per prompt at a resolution of 2048×5122048\times512. Both methods use the pretrained Stable Diffusion v2.1-base model and the same noise initialization scheme.

MethodIntra-LPIPS ↓\downarrowIntra-Style-Loss ↓\downarrowχ2\chi^2-Hist. dist. ↓\downarrowHist. Intersect. ↑\uparrowKID ↓\downarrowRuntime (s) ↓\downarrow
SyncVC (Ours)0.5633.030.660.6958.04171.82
SyncDiffusion0.5836.210.880.6364.14110.80

Table B3. Quantitative comparison with SyncDiffusion.

As shown in Table B3, SyncVC substantially outperforms SyncDiffusion in generation quality across all five quality metrics.

Therefore, SyncVC can outperform a task-specific baseline like SyncDiffusion and serves as a general framework for collaborative generation.

AuthorsOpenReview ↗

Response to Reviewer wu93 (2/2)

W3. It would help to state more clearly how far the ELBO derivation covers the implemented sampler. The experiments use deterministic DDIM ($\eta = 0$), for which the KL terms in Eq. (5) are infinite whenever the controlled and pretrained means differ. Appendix B mention about this and keeps a mean-discrepancy penalty, which is reasonable, but the final objective is then an upper-bound surrogate with tuned weights rather than the ELBO itself. The derivation also assumes independent initial noise per patch, while the experiments crop all patches from one noise canvas. Making this distinction explicit in the main text would make the theoretical claims more precise.

Response: We thank the reviewer for these precise suggestions. We have made the following three deviations from the theory more explicit in the revised main text:

  1. Tweedie-based reward. The reward is evaluated on Tweedie estimates instead of rolling out the full reverse chain. While motivated purely from a computational tractability standpoint, empirically, this outperforms full-rollout optimization on four of the five quality metrics at a lower runtime (Table 5 of the main paper, at T=8T=8 and N=2N=2).

  2. Deterministic samplers. The KL terms in Eq. (5) are finite only for stochastic transitions. For deterministic samplers such as DDIM (η=0\eta=0), the implemented mean-discrepancy penalty is therefore a surrogate motivated by the ELBO rather than the ELBO itself, although it works well in practice.

  3. Initial noise. The derivation assumes independent initial noise per patch, whereas our experiments crop all patches from one noise canvas, following the baselines. However, each patch is still marginally standard Gaussian and the initial-noise term does not depend on the controls, so the optimized objective is unchanged. The derivation then carries over when the reference process uses the same shared initialization.

Q3. SyncTweedies shows theoretically and empirically that MultiDiffusion and SyncTweedies are equivalent under matched conditions [...]

Response: We evaluated both methods using their official implementations from github repository, with the same crop layout, noise initialization scheme, and deterministic DDIM sampling. However, SyncTweedies introduces cross-patch self-attention, whereas MultiDiffusion denoises patches independently, so their denoisers differ despite sharing the same pretrained checkpoint. Their decoding also differs: SyncTweedies decodes patches separately and averages their RGB outputs, whereas MultiDiffusion decodes the assembled latent canvas. These pipeline differences explain why the reported results need not coincide, without contradicting equivalence under fully matched conditions.

Q4. Figures 5 and 14 show each textured mesh from a single viewpoint [...]

Response: Thanks for pointing this out. We apologize for the missing project website and have updated the supplementary material. We have also added multi-view renderings in Figure 20 of the revised manuscript. As shown, SyncVC produces high-quality textures with coherent appearance across viewpoints. Since the original textured meshes were unavailable during the rebuttal period, we reran the texturing experiment on a different machine. The regenerated textures may therefore differ slightly from those in Figures 5 and 14 of the main paper, but still retain high visual quality and continue to show qualitative advantages over the baselines.

Q5. As a minor point, could the authors clarify which baseline the 42.7% and 26.7% reductions in the abstract refer to? From Table 1, I get about 38% and 24% relative to MultiDiffusion.

Response: We thank the reviewer for catching this. From Table 1 of the main paper, the strongest baseline for both metrics is MultiDiffusion, and SyncVC reduces the χ2\chi^2-histogram distance by 38.0% (1.21→0.751.21 \rightarrow 0.75) and the Intra-Style-Loss by 24.2% (58.46→44.3458.46 \rightarrow 44.34). This does not change the results of SyncVC, and we have corrected these values in the abstract.

I thank the authors for the additional experiments and clarifications. The u=0u = 0 and λ=0\lambda = 0 ablations are important for showing what the control optimization and the penalty, the main components of the proposed method, each contribute. The revised paper includes the λ\lambda sweep, but I could not find the u=0u = 0 results. Will they also be included? For the λ\lambda sweep, could the authors also report the other four metrics?

For SyncDiffusion, the setup is described in the response, but the table might be slightly misleading since the row labeled SyncVC uses an LPIPS-augmented reward. With the default reward in Table 1, SyncVC does not outperform SyncDiffusion on Intra-LPIPS and Intra-Style-Loss. I recognize that this is expected since SyncDiffusion explicitly optimizes perceptual consistency, but reporting the default SyncVC and the LPIPS-augmented variant as separate rows would avoid confusion and be more transparent. I think this comparison should also be added to the revision.

For the meshes, it is unfortunate that Figure 20 shows regenerated textures, as I cannot check the unseen sides of the results in Figures 5 and 14. Since there is enough space on that page, multi-view renderings for all meshes in these figures would be helpful.

As a minor point, again, Section 3.1 (L315) still states 42.7% and 26.7%.

Reviewer_G9krOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper tackles the problem of collaborative generation using a frozen pretrained diffusion model to produce several outputs that must agree where they overlap. To address the shortcomings of prior methods which largely depend on heuristics, this paper explores what the synchronized trajectories can be optimizing. It casts the problem as sampling from a reward-tilted posterior and derives an ELBO that trades expected overlap-consistency reward against a KL penalty keeping each patch’s controlled transition close to the pretrained prior. To make this practical, it factorizes the patches as a chain, with each patch conditioned on its predecessor plus a learned control, and then optimizes that control greedily at test time using the Tweedie estimate and a DDIM closed-form update, adding a SyncSDE-style regularizer toward the previous patch.

Critical strengths

This paper’s main strength is that it gives a unified formulation for a problem that has mostly been handled with heuristics. A few ideas are particularly interesting:

  • Conditioning each patch’s transition on its predecessor, rather than denoising all patches in parallel and reconciling afterwards, makes the coupling explicit and keeps the problem tractable.
  • The practical objective is cheap and well-motivated. Using the Tweedie estimate to evaluate the reward, optimizing the control greedily per patch turns an intractable trajectory-level problem into a per-step update that needs only gradients through the reward. That makes the method training-free and usable on top of an off-the-shelf pretrained model.
  • The scope of the evaluation is reasonable, covering both a standard 2D setting and a harder one (mesh texturing), and the paper positions itself against a sensible set of baselines, apart from the gaps noted earlier.
Critical weaknesses
  1. Why is SyncDiffusion not included as a baseline? For wide images it is the most similar gradient-based method, and it directly optimizes LPIPS, the same family as the paper’s Intra-LPIPS and style metrics. The reason it’s left out should be stated.

  2. When a simpler component, such as the SyncSDE-style regularizer term or a cheaper update, accounts for much of the gain, the optimization machinery looks less necessary than the framing implies.

  3. SyncSDE already uses the same chain structure and the same Gaussian-conditional regularizer, with one hyperparameter and no test-time optimization. The real delta is adding per-step control optimization on top of it. The contribution seems incremental and it costs more compute and more hyperparameters.

Questions for authors
  1. Is there way you can ablate what fraction of the improvement comes from the variational objective itself and what fraction is inherited from SyncSDE?
  2. The paper motivates everything through the ELBO but implements a greedy, per-patch optimization. Does the ELBO actually go up during sampling, and does a better bound value track better consistency?
4 comments
AuthorsOpenReview ↗

Response to Reviewer G9kr (1/3)

W1. Why is SyncDiffusion not included as a baseline? For wide images it is the most similar gradient-based method, and it directly optimizes LPIPS, the same family as the paper's Intra-LPIPS and style metrics. The reason it's left out should be stated.

Response: Based on the reviewers' feedback, we have added a comparison with SyncDiffusion. For fair comparison, we use an LPIPS-augmented reward as a drop-in replacement for the default reward in SyncVC to account for SyncDiffusion's explicit optimization of perceptual consistency. Specifically, the adjacent-patch overlap reward in SyncVC is extended as follows:

$ \tilde{r}_n(y,x_0^{(1:n)}) = -\frac{\gamma}{2}|M^{(n)}\odot(f(x_0^{(n-1)},y,n)-x_0^{(n)})|_2^2 $

$\qquad \qquad \qquad -\lambda_{\mathrm{LPIPS}}\cdot\mathrm{LPIPS}(\mathcal{D}(x_0^{(n)}),\mathcal{D}(x_0^{(1)})),\quad n\geq2, $

where D\mathcal{D} denotes the Stable Diffusion VAE decoder and λLPIPS\lambda_{\mathrm{LPIPS}} weights the perceptual term. The rest of the SyncVC method remains unchanged. While SyncDiffusion uses the middle patch as its anchor, SyncVC uses the first patch to respect its autoregressive formulation. We use K=2K=2 optimization steps per denoising step and set λLPIPS=104\lambda_{\mathrm{LPIPS}}=10^4 for SyncVC and compare both methods in Table A1 below.

MethodIntra-LPIPS ↓\downarrowIntra-Style-Loss ↓\downarrowχ2\chi^2-Hist. dist. ↓\downarrowHist. Intersect. ↑\uparrowKID ↓\downarrow
SyncVC (Ours)0.5633.030.660.6958.04
SyncDiffusion0.5836.210.880.6364.14

Table A1. Quantitative comparison with SyncDiffusion.

As shown in Table A1, SyncVC substantially outperforms SyncDiffusion in generation quality across all metrics. Therefore, SyncVC can outperform a task-specific baseline like SyncDiffusion and serves as a general framework for collaborative generation.

W2. When a simpler component, such as the SyncSDE-style regularizer term or a cheaper update, accounts for much of the gain, the optimization machinery looks less necessary than the framing implies.

Q1. Is there way you can ablate what fraction of the improvement comes from the variational objective itself and what fraction is inherited from SyncSDE?

Response: [W2, Q1] To isolate the contribution of control optimization, we compare SyncVC with a variant that sets u=0u=0 on wide image generation. This variant keeps the SyncSDE-style trajectory coupling in Eq. (7) of the main paper, and removes only the control optimization, leaving all other settings unchanged. As shown in Table A2, SyncVC improves all five reported metrics, most notably reducing the histogram distance from 1.381.38 to 0.750.75, and the style loss from 65.2665.26 to 44.3444.34. Moreover, the u=0u=0 variant performs comparably to SyncSDE, whose results from Table 1 of the main paper are included for reference (e.g., χ2\chi^2-histogram distance of 1.381.38 vs. 1.311.31, and style loss of 65.2665.26 vs. 63.9863.98). Therefore the gains in SyncVC come from control optimization and not merely the trajectory coupling term in Eq. (7) of the main paper.

SettingIntra-LPIPS ↓\downarrowIntra-Style-Loss ↓\downarrowχ2\chi^2-Histogram dist. ↓\downarrowHistogram Intersect. ↑\uparrowKID ↓\downarrow
SyncVC0.5944.340.750.6752.07
u=0u=00.6765.261.380.5155.72
SyncSDE0.6563.981.310.5357.08

Table A2. Quantitative analysis on the effect of control optimization for wide image generation. SyncSDE results are taken from Table 1 of the main paper. KID values are scaled by 10310^3.

Regarding a cheaper update, the optimization budget in SyncVC is tunable through the number of optimization steps KK, as shown in Figure 6 of the main paper. Even a single step (K=1K=1) reduces the χ2\chi^2-histogram distance from 1.381.38 (u=0u=0) to 0.910.91, and consistency improves further as KK increases. With T=18T=18 denoising steps and K=2K=2, SyncVC is approximately 5.6×5.6\times faster than the default configuration while retaining better color consistency than both the u=0u=0 variant and SyncSDE (1.001.00 vs. 1.381.38 and 1.311.31). Therefore, control optimization is beneficial even at a small budget.

AuthorsOpenReview ↗

Response to Reviewer G9kr (2/3)

W3. SyncSDE already uses the same chain structure and the same Gaussian-conditional regularizer, with one hyperparameter and no test-time optimization. The real delta is adding per-step control optimization on top of it. The contribution seems incremental and it costs more compute and more hyperparameters.

Response: We respectfully disagree that the contribution is incremental.

Re. Core contributions. SyncVC shares the chain structure and the coupling term in Eq. (7) of the main paper with SyncSDE, and we do not claim these as contributions. Modeling the transition as in SyncSDE is a design choice. The objective in Eq. (5) does not depend on how the transition is parameterized, so our framework is more general and admits other designs. Our contribution is the variational control formulation: SyncSDE constructs a conditional score from Gaussian conditional models, whereas SyncVC casts collaborative generation as variational inference with controls as variational parameters (Eqs. (1)--(5)), which to our knowledge is the first such formulation for this task. This matters empirically: with the same coupling but u=0u=0, all five metrics fall to the level of SyncSDE (Table A2 in our response to W2). It also lets new objectives enter directly through the reward, as with the LPIPS-augmented reward in our response to W1.

Re. Compute and hyperparameters. Test-time optimization adds cost, but the budget is tunable: with T=18T=18 and K=2K=2, SyncVC still improves color consistency over SyncSDE (χ2\chi^2-histogram distance of 1.001.00 vs. 1.311.31; Table 7 of the main paper). SyncVC has three coefficients with distinct roles (γ\gamma for the reward, λ\lambda for regularization toward the prior, and β\beta for the control strength), which are analyzed in Appendix E.4.4.

AuthorsOpenReview ↗

Response to Reviewer G9kr (3/3)

Q2. The paper motivates everything through the ELBO but implements a greedy, per-patch optimization. Does the ELBO actually go up during sampling, and does a better bound value track better consistency?

Response:

  • Why greedy optimization. Optimizing the ELBO in Eq. (5) of the main paper directly would require rolling out the full reverse chain for every control update, so SyncVC optimizes a tractable per-step surrogate (Eq. (8)) on Tweedie estimates.

  • Does the bound go up during sampling? We evaluate the surrogate bound at different sampling timesteps. As shown in Table A3, the bound increases as sampling progresses.

tt9818016014012011
Surrogate bound ↑\uparrow-3191.85-243.04-106.85-70.84-39.43-10.61

Table A3. Surrogate bound during sampling for wide image generation.

  • Does a better bound track better consistency? Increasing KK allows more optimization steps to improve the surrogate bound. We therefore examine how consistency changes with KK, using the χ2\chi^2-histogram distance. As shown in Table A4, the distance decreases from 1.381.38 at K=0K=0 (i.e., u=0u=0) to 0.750.75 at K=5K=5, showing that better bound improves color consistency.
KK01235
χ2\chi^2-Histogram dist. ↓\downarrow1.380.910.810.780.75

Table A4. Color consistency for different numbers of optimization steps KK on wide image generation (T=50T=50).

  • Greedy vs. direct optimization. Empirically, full-rollout joint optimization, which evaluates the reward after rolling out the full chain, is worse on four of the five metrics and slower at T=8T=8 and N=2N=2 (Table 5 of the main paper), so the greedy scheme does not sacrifice quality.

Dear Reviewer G9kr,

Thank you for your insightful review and constructive feedback. In response to your suggestions, we added a comparison with SyncDiffusion using an LPIPS-augmented reward and an ablation that disables control optimization while retaining the SyncSDE-style coupling. The ablation shows that optimizing the controls improves all five reported metrics. We also tracked the surrogate bound during sampling and found that it increased across the timesteps.

We additionally clarified how SyncVC differs from SyncSDE. SyncVC formulates collaborative generation as variational inference over coupled diffusion trajectories, with controls optimized to balance cross-patch consistency against deviation from the pretrained diffusion prior. This formulation also lets us incorporate new objectives directly into the reward, as illustrated by the LPIPS-augmented comparison.

As the discussion period ending soon, we would appreciate hearing whether our responses address your questions. If any concerns remain, we would be happy to clarify them.

Best regards,
SyncVC Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear ACs and SACs,

We summarize below how the rebuttal and revised manuscript address each reviewer’s main concerns.

  • G9kr. We added a SyncDiffusion comparison and isolated the effect of control optimization by setting u=0u=0 while retaining the SyncSDE-style coupling. Optimizing the controls improved all five metrics. We also reported how the surrogate bound changes during sampling and how color consistency improves with additional optimization steps (Tables A1-A4).
  • wu93. We added the u=0u=0 ablation to the main paper and reported all five metrics for the λ\lambda sweep in our follow-up, clarifying the trade-off between consistency and image fidelity. The revised manuscript reports default and LPIPS-augmented SyncVC separately in the SyncDiffusion comparison, includes runtimes and multi-view mesh renderings, and corrects the improvement percentages.
  • oRYS. We clarified how the deterministic DDIM surrogate, regularizer weighting, and shared initialization relate to the ELBO. We also explained what SyncVC adds beyond DTM/NDTM and SyncSDE. A joint variational objective over coupled trajectories whose consistency reward can be specified independently of the coupling. Our comparisons with simultaneous and full-rollout updates show the practical role of the autoregressive sampler. The reviewer acknowledged the clarifications and ablations.

Thank you very much!

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.