acceptodds
Under review as a conference paper at ICLR 2027

What Should a Diffusion U-Net Preserve? Predictive Sufficiency at Multiscale Interfaces

Abstract

Downsampling in diffusion U-Nets is typically treated as a geometric compression operation. We instead view each resolution transition as a target-specific predictive information interface: a representation is useful only insofar as it preserves the information needed for its assigned denoising target. We formalize this idea through a predictive sufficiency gap and derive exact conditions for linear Gaussian interfaces. For rank-matched Gaussian measurements, sufficient geometry generically varies with diffusion SNR, yielding a conditioned-versus-static separation. A tractable Haar model then shows that a constrained local three-tap pre-projection transform can exactly realize this SNR-dependent geometry at fixed retained dimension and yields a strict pre-versus-post separation. In a controlled CIFAR-10 diffusion U-Net interface, correct time conditioning and access to fine-space directions improve held-out prediction, and the learned retained measurement geometry varies with diffusion time. A covariance-defined Gaussian-linear diagnostic further shows that, over the first 15 of 20 evaluation timesteps, the learned Time Pre measurement is closer than Fixed to the surrogate sufficient geometry under both surrogate PSG and subspace distance; the ordering reverses in the five lowest-SNR timesteps as that surrogate geometry approaches the fixed Haar subspace. In an end-to-end diffusion U-Net, matched fine-space access produces a small denoising advantage concentrated in the lowest-noise regime, while a DDPM generation evaluation does not preserve the matched ordering in FID. These results support predictive sufficiency as a principled criterion for analyzing and constraining multiscale diffusion-denoiser interfaces, while showing how interface-level information preservation contributes within the broader dynamics of the generative process.

Primary area:
generative models
Keywords:
Diffusion Model, U-Nets, Multiscale Representations, Predictive Sufficiency, Representation Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eXkKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a Predictive Sufficiency Gap (PSG) to reframe U-Net downsampling in diffusion models as a target-specific predictive information interface. It derives exact sufficiency conditions under Gaussian linear assumptions, proves that optimal retained subspaces generically vary with diffusion SNR, and validates these findings using a tractable Haar model and controlled CIFAR-10 experiments.

Critical strengths

The paper studies a clear and important research question, namely what and how to separate the important details during compression with UNets. The paper also included FID evaluations.

Critical weaknesses

From my point of view, the analysis is very limited and the results have little significance for current diffusion models due to studying only a single 32-16 downsampling step on CIFAR10. Furthermore, the gains are below 1% and do not preserve when measuring FID.

Questions for authors

Can the authors explain their exerimental setup choice and examine practical impact considerations?

1 comment

Response to Reviewer eXkK

Theoretical contribution and setup choice. We study what a target-specific multiscale interface must preserve, rather than proposing a new diffusion sampler. For Gaussian linear observations, Sec. 3.2 gives an exact predictive-sufficiency criterion. Under rank matching, the sufficient measurement subspace changes between distinct diffusion SNRs precisely when the target subspace is not invariant under the data covariance; in that case, time-conditioned measurements attain zero aggregate PSG while every static rank-matched subspace has a strictly positive minimum. Section 4 separately gives an explicit 8×8 periodic Gaussian Haar construction: a local, SNR-dependent three-tap preprojection realizes sufficiency at fixed retained dimension and yields a strict conditioned/static and pre/post separation under its stated assumptions. These results are specific to their stated assumptions.

The CIFAR-10 32→16 transition was selected as a controlled test of this interface question. It fixes an RGB noisy observation, a coarse ε target, normalized Haar projection, and matched PRE/COARSE parameterizations, while a Null-Time control isolates correct timestep conditioning. It also supports learned-subspace and Gaussian-linear surrogate diagnostics (Sec. 5.1; Figures 3–4). The revision also clarifies the complete-skip Bayes boundary (Sec. 3.3) and adds finite feature/Post-D readouts and a skip-access interaction study (Secs. 5.3–5.4).

Practical impact. Although the matched end-to-end MSE gain is only 0.0260% and PRE has worse FID than matched COARSE (16.1744 vs. 15.4174; Sec. 5.2), the contribution is a diagnostic framework for target-relevant information and its finite use in skip-connected computation, not demonstrated gains in generation quality or efficiency, or transfer beyond the tested setting.

Reviewer_dSkgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reframes downsampling steps in diffusion U-Nets as “target-specific predictive information interfaces”, and therefore the coarse representation only needs to preserve the information required to predict that level’s denoising target at that scale. The authors introduce a squared loss, predictive sufficiency gap (PSG) and show under a Gaussian-linear model that fixed pooling generally loses useful information and that the optimal retained subspace changes with diffusion SNR. An exactly solvable 8×8 Haar model further demonstrates that an SNR-dependent three-tap filter applied before downsampling can eliminate this loss at a fixed retained dimension.

Empirically, the authors support these findings through controlled CIFAR-10 experiments, showing approximately 0.8 – 0.9% improvements in denoising MSE and diffusion-time-dependent changes in the learned representation. However, in an end-to-end DDPM U-Net, the matched gain shrinks to 0.026%, with the coarse-only control outperforming the proposed adapter in generation quality (FID).

Critical strengths

1. Principled theoretical framework. The paper provides a mathematically grounded characterization of information preservation at diffusion U-Net downsampling interfaces, establishing the SNR dependence of sufficient representations and motivating time-conditioned pre-projection transformations.

2. Well-designed experiments. The authors use matched controls to isolate the effects of diffusion-time conditioning and access to fine-resolution information, supported by measurements of learned subspace geometry.

3. Honest reporting and detailed experimental analysis. The paper openly reports the small end-to-end gain and the MSE/FID reversal, and does not claim improved generation. The authors also provide detailed timestep-wise results and analyses of how the learned representations change with diffusion time, offering useful insights into the adapter's behavior.

Critical weaknesses

1. Limited empirical generalization. Experiments are only on CIFAR-10, with the intervention tested at one resolution transition, and one end-to-end U-Net architecture. Therefore, it is unclear whether the findings generalize to larger images, different diffusion settings or other architectures.

2. Marginal end-to-end improvements. The matched denoising improvement is only 0.026%, concentrated almost entirely in the lowest-noise timestep bin, raising questions about the practical significance of the proposed mechanism.

3. Limited generation-quality benefit. Although the proposed adapter improves FID over Fixed, the matched coarse-only control achieves better FID (15.4 vs. 16.2). This suggests that access to additional fine-scale information before downsampling does not provide a clear generation-quality advantage, despite its small denoising benefit.

4. Limited training-run robustness. Only one training run per condition is reported, and the confidence intervals do not account for variability across independent training runs. Given the small end-to-end improvements, it is unclear whether the reported differences would remain consistent across random seeds.

Questions for authors

1. Generalization. Can the authors evaluate the proposed approach on a higher-resolution dataset or another diffusion architecture to establish whether the observed improvements extend beyond the current experimental setup?

2. Timestep-dependent performance. Why is the end-to-end denoising improvement concentrated almost entirely in the lowest-noise regime? Can the authors investigate whether the mechanism provides meaningful benefits at other timesteps or resolution transitions?

3. Robustness. Can the authors report results over multiple training seeds (e.g., 3–5) with between-seed variance?

1 comment

Response to Reviewer dSkg

The revision strengthens the evidence for a time-dependent predictive-interface mechanism while keeping it distinct from practical gains in a jointly trained U-Net. We address the generalization, timestep, generation-quality, and robustness concerns separately.

Q1 — Experimental scope and controlled evidence. We agree that one CIFAR-10 32→16 transition and one architecture do not establish cross-setting generalization. This setting was chosen to compare preprojection fine access and diffusion-time conditioning under matched prediction architectures and targets. In Sec. 5.1, PRE achieves 0.0778408 MSE versus 0.0785282 for COARSE (a 0.8753% reduction; favorable at 17/20 times). Correct-time PRE improves over the equal-capacity Null-Time control by 0.783% (favorable at 20/20 times). Learned retained-subspace geometry varies with time (maximum projector distance 0.138645, versus 0.000699 in the Null-Time sweep); covariance-defined Gaussian-linear surrogate comparisons favor PRE over Fixed at 15/20 times and reverse at five low-SNR times (Figures 3–4). These results establish bounded controlled and surrogate observations, not generalization to untested architectures.

Q2 — Small end-to-end benefit and its time profile. The original jointly trained matched PRE–COARSE gain remains only 0.0260405%, favorable in 4/20 bins; the lowest-noise bin accounts for 100.4986% of the signed aggregate gain, with other bins slightly offsetting it (Sec. 5.2). The revision adds a separate skip-access experiment (S2), refitting a downstream suffix around one frozen encoder. All 60 suffix-seed × timestep mean access-by-skip interactions are positive. Excluding the first timestep still gives interaction 0.0094620 MSE (paired suffix-seed 95% interval [0.0090838, 0.0098402]); the first four times account for 75.24% of its total (Sec. 5.4; Figure 5). Thus pathway sensitivity extends beyond the first time bin, despite substantial low-noise concentration. Because S2 changes skip access, dynamic GroupNorm statistics, and optimization, it does not identify the cause of the original end-to-end time profile.

Generation quality. The original FIDs remain Fixed 18.6373, PRE 16.1744, COARSE 15.4174 (lower is better; Table 2). PRE does not outperform matched COARSE in generation quality. We therefore present the contribution as a predictive-interface analysis, not a demonstrated sampling-quality or efficiency improvement.

Q3 — Independent training robustness. There is one independently trained full U-Net per arm. Image-cluster bootstrap intervals and FID jackknife estimates condition on those checkpoints; they do not capture between-training-run variation. The three paired S2 suffix seeds and three Post-D readout seeds reuse one pretrained encoder and are not independent full-network replications. We retain these distinctions in Sec. 6 and acknowledge that broader-scale evaluation and independent end-to-end robustness remain unmeasured.

Reviewer_Cvw7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper derives which information the downsampling in a diffusion U-Net should keep to predict the denoising target. Under a Gaussian setting, the authors prove that this information generally changes with the noise level, so a fixed downsampling will lose some of this information at some noise level (Theorem 2). In a Gaussian example, the authors show that a small noise-dependent filter before downsampling can recover all of this information without keeping more values (Theorem 3).

Critical strengths

S1: The question is clean, and the derivation is correct in the authors' setting. Moreover, the 8×8 example gives an explicit closed-form solution, showing exactly how a small noise-dependent filter before downsampling can recover all the needed information. However, the empirical results on larger diffusion unet appears to be very weak (0.026% improvement).

Critical weaknesses

W1: Fundamentally, the paper's theory does not analyze the actual U-Net: it only derives the information loss of the coarse sub-U-Net, which is just an information bottleneck of spatial resolution. The actual U-Net has skip connections to ensure that the decoder receives the full-resolution features. Even if the main branch discards some information, that information is still available to the network, and this is the design principle of the U-Net.

The work, however, ignores this fundamental principle. The authors mention the skip connection (Figure 1, Section 5.2), but the framework does not model it, so a nonzero PSG does not imply that the U-Net loses anything. This is also consistent with the very weak experimental result on the end-to-end diffusion U-Net (a 0.026% gain).

Questions for authors

Q1 The theorems showing that a fixed downsampling must lose information assume that the downsampled features are limited to the dimension of their target. The controlled experiment imposes this limit (3 channels at 16×16), but the end-to-end U-Net does not (64 channels at 16×16). Could the authors estimate how much information the Fixed downsampling actually loses in the end-to-end U-Net?

1 comment

Response to Reviewer Cvw7

W1 — Scope of the coarse-interface theory. We agree that a positive local predictive sufficiency gap (PSG) does not imply information loss for the complete skip-connected U-Net. The revision makes this precise in Sec. 3.3: if the coarse representation Z = C(S, t) is determined by the complete accessible skip collection S and common conditioning, then R*(Y | S, Z, t) = R*(Y | S, t). Thus the relevant full-network question is how finite computation and training use available information, not whether adding a deterministic coarse feature supplies new Bayes information. We also clarify the rank issue in Q1: Gaussian sufficiency permits arbitrary measurement rank q, whereas the uniqueness/SNR-separation theorem assumes q = m. Neither the number of actual U-Net channels nor the controlled experiment's target dimension establishes the real network's sufficiency.

Q1 — Direct measurement at the original Fixed downsampling. We now freeze the original Fixed checkpoint at 130,000 updates, including its original learned remap D, and compare H = h₃₂, Z_P = PH, and Z_D = D(PH). Their mean full-RGB-ε MSEs under separate, nominally matched CNN readouts are 0.174312, 0.379546, and 0.374748, respectively. The post-D minus fine-feature difference is L(Z_D) − L(H) = +0.200436, with a paired readout-seed 95% t interval [0.186905, 0.213967], and is positive at all 20 evaluated timestep means (Sec. 5.3; Appendix D.2.1, Table 10). This uses three paired readout fits per observation, a common 1,024-update budget, and lossless spatial alignment. Active input dimensions differ despite matched nominal parameters, and optimization was still improving. Hence the result directly quantifies finite-readout predictive utility after the actual Fixed remap, not unrestricted Bayes information loss or the original U-Net's output degradation.

How this relates to skips. A separate experiment (Sec. 5.4) refits the downstream suffix, including a new D, around one frozen encoder. PRE improves over COARSE by 0.0609% with full skips and 19.45% with restricted skips, relative to each condition's own COARSE baseline. The paired access-by-skip interaction is 0.0135775 MSE (95% suffix-seed interval [0.0133115, 0.0138434]). This establishes an attained-risk interaction under the specified training protocol, while dynamic GroupNorm and incomplete optimization prevent pure information-mediation attribution. The original 0.0260% matched end-to-end gain and reversed matched FID remain unchanged. Together, G1, the original post-D diagnostic, and the skip-access experiment distinguish available information, finite predictive utility, and realized full-network performance without conflating them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.