acceptodds
Withdrawn from ICLR 2027

Patch-Sheaf Residual Matching for Local-to-global Coherence in Latent Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern image generators produce locally plausible content while distorting structures that extend across spatial regions, such as roads, lanes, and object boundaries. We formulate this failure as a local-to-global compatibility problem over predicted-clean latent patch states. We construct a cellular sheaf with neighbouring-patch, parent-region, and sliding square-cell relations, whose residuals quantify incompatibility between local states. Clean latents can exhibit meaningful nonzero relation residuals at boundaries and transitions, motivating the use of clean relation energies as matching targets. We therefore match predicted-clean relation energies, i.e., the squared norms of the residuals, to clean-latent targets, calibrate per-patch incident-energy distributions, and weakly preserve their spatial organization. These auxiliary losses operate only within an intermediate signal-to-noise ratio window; the (+)tail extension adds timestep-conditioned tail calibration and localization, while the generative objective remains active at all timesteps. Using a DiT-XL/2 backbone, we evaluate on DeepGlobe Roads, LSUN-Church, and BDD100K against no-sheaf controls matched for training duration and learning-rate schedule. Across five independent phase-2 training seeds, (+)tail reduces mean FID relative to staged-LR controls by (11.8%), (12.9%), and (21.4%), respectively, while consistently improving KID, CMMD, recall, and coverage.

Primary area:
generative models
Keywords:
Latent diffusion models, cellular sheaves, residual-energy matching, local-to-global consistency, structural coherence, geometric deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pBwwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes patch-sheaf residual matching to improve structural consistency in latent diffusion models in terms of local-to-global coherence. The work constructs a cellular-sheaf representation to model cross-path compatibility among predicted patches in the latent space in order to improve local-to-global coherence. The residuals quantify the local incompatibility. Hence, it matches differences between predicted clean latent regions to those in clean training data, helping preserve meaningful boundaries and transitions. Additional losses guide the distribution and spatial arrangement of patches. Experiments on satellite, church, and driving images show improvements in generation quality and crosspatch structure over matched training controls. Ablation and sensitivity studies examine the contribution of the losses and the effect of different settings.

Critical strengths
  1. Improvements supported by controlled comparisons. Comparisons with standard continuation and the phase-2 learning-rate schedule, repeated across five seeds, provide useful experimental support. Improvements across three datasets suggest gains in image quality and diversity beyond those obtained from additional training or scheduling changes.
  2. A well-motivated approach to preserving image structure. Matching clean relation energies helps preserve meaningful boundaries and transitions. The zero-residual comparison supports better residual preservation, while the boundary-orientation and multi-hop analyses provide additional evidence of structural improvements.
Critical weaknesses
  1. The contribution of the adaptive sheaf construction is not clearly demonstrated. The benefit of adaptive sheaves remains unclear, as Blocks II and III account for much of the improvement. A focused ablation is needed to isolate their contribution from the other components.
  2. There is a clear disconnect between the experimental motivation and the theoretical formulations that feel heuristically motivated, yet the underlying heuristics are not made apparent. An effort needs to be made to better interpret the terms and losses and the intuition behind them.
  3. Insufficient details about the experimental setup and evaluation procedure. Important experimental details, including dataset splits, reference-set sizes, the latent encoder, and sampling settings, are unclear. The checkpoint selection rule and its separation from final evaluation also need clarification. These omissions limit reproducibility and make it difficult to assess the reliability of the reported improvements.
Questions for authors

Can the authors better connect the heuristics and the intuition behind selecting them? This would make the motivations for the work clearer.

Reviewer_2p9COpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose patch-sheaf residual matching to improve the structural coherence of image diffusion models, such as roads and lanes that break across regions. The predicted-clean latent is treated as a cellular sheaf over patches, and its residuals are matched to those of the clean latent through auxiliary losses. The authors run experiments on DeepGlobe Roads, LSUN-Church, and BDD100K with DiT-XL/2, and show that continued training with these losses improves over pre-trained phase-1 checkpoints

Critical strengths
  • Sensible design choice: Matching residuals to clean targets, rather than suppressing them, respects real boundaries in images, and the zero-residual ablation supports this choice.

  • Careful controlled comparison: All branches start from the same checkpoint, are matched in training steps and learning-rate schedule, and are averaged over five seeds.

Critical weaknesses
  • Questionable experimental setting: All models are trained from scratch, and phase 2 starts from a checkpoint that is likely far from converged (e.g., only 50k steps on DeepGlobe). The no-sheaf controls also behave unstably: on DeepGlobe, ordinary continuation worsens FID from 97.5 to 106.9. It therefore remains unclear whether the method is effective on a well-converged model.

  • Evaluation results could be hacked: Many reported metrics (e.g., residual W1, mass ratio, QCE, TailIoU) measure exactly the quantities optimized by the proposed losses, and are also used for checkpoint selection. Improvements on them do not prove much. We suggest evaluating on a standard benchmark such as ImageNet and reporting the results there.

  • No comparison with other methods: The method is compared only with its own no-sheaf continuations. Related methods discussed in the paper, such as TopoDiffusionNet (Gupta et al., 2025) and Glob-Diffusion (Kang et al., 2026), are not compared.

Questions for authors
  1. How could this be applied to text-to-image diffusion models such as Stable Diffusion or Flux?
Reviewer_PX2bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses the problem that the diffusion-generated image is locally plausible but globally distorted. Building on cellular sheaf theory from algebraic topology, it partitions the latent feature map into patches and defines 3 types of relations among them, using sheaf residuals to quantify the incompatibility between neighbouring patches. During a second-stage fine-tuning of a DiT-XL/2 latent diffusion model, the method introduces 3 auxiliary losses within an intermediate SNR window — relation-level energy matching, patch-level Wasserstein calibration of the residual distribution, and sliced-Wasserstein alignment incorporating spatial coordinates — while also calibrating and localizing high-residual regions conditioned on the timestep. On 3 datasets, compared with controls matched for training duration and learning rate, the method reduces mean FID across the board, consistently improves KID, CMMD, recall, and coverage, lowers the residual calibration error in boundary regions, and measurably improves edge coherence across patches, which gains that exceed what additional training or learning-rate scheduling alone can deliver.

Critical strengths
  • The core design is well motivated. The paper observes that clean latents naturally exhibit nonzero relation residuals at genuine boundaries, so the training objective should align predicted relation energies with clean target values rather than suppress them to zero. An ablation in the appendix supports this: at every cumulative depth, clean-target matching outperforms zero-residual minimization, which instead leads to residual collapse.

  • The ablation setting is reasonably rigorous. The authors include two no-sheaf controls (a duration-matched continuation and a staged-LR continuation matched for the learning-rate schedule); all branches share the same phase-1 checkpoint, main results are averaged over five independent seeds with standard deviations, and cumulative ablations plus sensitivity analyses are provided. Under this protocol, FID drops by 11.8%–21.4% with consistent improvements in KID, CMMD, recall, and coverage — gains that are difficult to attribute to extra training or schedule differences.

  • The evaluation is comprehensive. Beyond standard metrics such as FID, the paper reports residual calibration diagnostics, boundary-conditioned analysis based on Sobel edges, and cross-patch geometric coherence metrics. The consistent direction of evidence across these levels indicates that the method genuinely improves the local-to-global coherence it targets, rather than merely benefiting distributional metrics.

Critical weaknesses
  • The paper does not compare against simpler structural regularizers under the same phase-2 protocol. Without baselines such as latent gradient matching, pairwise residual matching without the sheaf construction, or decoded-image edge and perceptual losses, it is unclear whether the gains specifically require the proposed sheaf formulation.

  • The evidence for local-to-global coherence is not fully independent of the training objectives. Several residual metrics directly measure quantities targeted by the proposed losses, while BAE and IPGC mainly evaluate adjacent patch boundaries. The multi-hop analysis is limited to DeepGlobe and two-to-three-patch ranges, so scene-level structural coherence remains insufficiently validated.

  • The mathematical definition of the cellular-sheaf construction is incomplete. The paper does not fully specify the underlying cell poset, stalks, and restriction maps for all relation families. It also remains unclear in what precise sense the sample-dependent operator �0��0� is a cellular-sheaf Laplacian.

  • The main five-seed evaluation uses only one DiT-XL/2 backbone on three structured datasets. Appendix 5.4 provides a limited FFHQ experiment, but generalization to other architectures and pretrained conditional or text-to-image diffusion models remains untested.

  • Although the main branches are evaluated at common training steps, these steps are selected using both FID and residual diagnostics directly targeted by the proposed losses. The paper does not clearly establish whether the conclusions remain unchanged under a pre-specified budget or a common FID-only checkpoint-selection rule.

Questions for authors
  • Have you compared the method with simpler structural losses under the same phase-2 training schedule and computational budget?

  • Can you provide more independent evidence of global structural coherence, such as road-connectivity, object-boundary continuity, scene-layout metrics, or human evaluation? Can the multi-hop analysis also be extended to the other datasets and longer spatial ranges?

  • Can you explicitly define the cell poset, stalks, and restriction maps for the edge, parent-region, and sliding-square relations? How should the sample-dependent operator �0��0� be interpreted as a cellular-sheaf Laplacian?

  • Can you evaluate the method on at least one different backbone or pretrained conditional diffusion model to test whether the improvements transfer beyond the current setting?

  • Were the common evaluation steps selected using an independent validation set? Do the conclusions remain unchanged under a pre-specified training budget or a common FID-only checkpoint-selection rule?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.