acceptodds
Under review as a conference paper at ICLR 2027

Atlases Are Already Inside: Recovering Population Templates from Pretrained Diffusion Models

Abstract

We present a new inference-time sampler for diffusion models that gives a pretrained model a capability it was never trained for: constructing the atlas of the population it synthesizes. The sampler converges from every random seed to the population's central anatomy, which we call the intrinsic atlas. The advantage is threefold. (1) It requires no retraining. A diffusion model that has already learned a coherent population, including the released ones, yields its atlas in a single inference pass without involving deformable registration. (2) It applies to multiple domains, such as brain MRI, chest X-ray, faces, and 3D shapes. (3) It extends to subpopulations without new data. One age-conditioned model gives an atlas at any age in its training range, and the resulting family reproduces the CSF expansion of healthy aging. Evaluated as a registration target, the intrinsic atlas is best or second-best on every dataset against classical and learned templates, and the most central template on held-out brain MRI cohorts. Atlas construction can be reframed as a byproduct of generative modeling: a diffusion model is a learned representation of population structure, and the atlas is what it already contains.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
diffusion models, inference-time sampling, posterior-mean iteration, atlas construction, deformable templates, image registration, emergent capabilities, conditional generation, medical imaging

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zzvFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an inference-time sampling strategy for pre-trained Denoising Diffusion Probabilistic Models (DDPMs) that extracts a canonical population template, termed the intrinsic atlas, from a generator trained solely for individual synthesis. Motivated by Grenander's pattern theory, the authors posit that while injected noise during the reverse diffusion process drives individual variation, the learned denoiser isolates the shared population structure. Thus, replacing each stochastic reverse transition with its conditional expectation (xt−1←μθ(xt,t,y)x_{t-1} \leftarrow \mu_\theta(x_t, t, y)) causes arbitrary random Gaussian initializations xT∼N(0,I)x_T \sim \mathcal{N}(0, I) to contract onto a single sharp consensus image (cross-seed SSIM≥0.99\text{SSIM} \ge 0.99), a phenomenon termed canonical convergence. The authors validate this approach across diverse 3D and 2D medical and non-medical domains (brain MRI, chest X-ray, retinal images, faces, and 3D shapes), demonstrating that the intrinsic atlas achieves top-tier registration accuracy and superior geometric centrality under standardized ANTs registration protocols, naturally extends to covariate-conditioned atlas families (reproducing healthy aging CSF expansion on brain MRI), and diagnoses multi-modal or unaligned populations.

Critical strengths
  1. The paper presents a simple, reproducible, and interesting finding. Dropping injected noise from the DDPM reverse transition collapses arbitrary initializations onto a single sharp image (cross-seed SSIM ≥ 0.99, correlation 0.9999) across 3D brain MRI, chest X-ray, faces, and 3D shapes, with no atlas-specific objective. The effect is easy to verify, and the mechanism (Tweedie / posterior-mean iteration) is sound as far as it is stated.

  2. The presented atlas is useful and biologically coherent. Under a fixed ANTs protocol it is best or second on Dice and the most central and regular template on three held-out brain cohorts. The age-conditioned family reproduces the accelerating CSF-expansion trend of healthy aging, and a distant-age control shows conditioning carries

Critical weaknesses
  1. The paper has some missing baselines that would isolate what is new. There are no same-generator controls (DDIM/PF-ODE from x_T = 0, or the mean of N stochastic samples), no conditional deformable-template baselines (Dalca 2019, Atlas-GAN, AtlasMorph) against the age-conditioned family, and no 2D congealing baselines (GANgealing, Neural Congealing) on CelebA despite Section 2.1 positioning against them. Table 1 reports no variance, and several decisive margins are 0.001–0.004. Without these, it is unclear how much credit belongs to the sampler versus the underlying generator.

  2. The 2D Gaussian example does not explain why multi-step iteration yields a sharp mode rather than the blurry voxel mean, and no contraction conditions are given beyond the linear case. The pattern-theory analogy obscures that the method requires pre-aligned, background-normalized populations (the FFHQ "remedy" masks already landmark-aligned faces). Ambiguity about IXI/OASIS-1 train/test overlap (899 fine-tuning scans vs. 123 evaluation subjects; OASIS-TRT-20 inside Mindboggle101) must be resolved.

Questions for authors
  1. Were the 78 IXI and 45 OASIS-1 evaluation subjects, and the OASIS-TRT-20 subjects in Mindboggle101, strictly excluded from the 899 fine-tuning scans?
  2. What varies across the "at least 3 repeated runs" in Table 1, and what are the standard deviations?
  3. What does "per-seed r = +0.88" correlate, and how do the 1–4 pp error bars in Figure 7a reconcile with the "<0.2 pp spread" in the text and the s.e.m./s.d. mismatch?
  4. For the Khader et al. 3D model, is the iteration run in VQ-GAN latent space? What representation is used for ModelNet shapes, and how is 3D SSIM computed?
  5. Which statistical test produced the p-values in Section 5?
1 comment

W1. Missing baselines and variance.

We added all requested baselines except AtlasMorph, which has no public code or models.

  • GANgealing and Neural Congealing are reported in the revised Table 1. The results with standard deviations over repeated runs are provided in Table 5 in the supplementary due to the width limit.
  • GANgealing is reported on CelebA only, since it requires a StyleGAN2 trained on the target domain and none is available for chest X-ray.
  • Neural Congealing is reported on CelebA and Montgomery; it does not scale to full datasets in our experiments. We optimized on 25 subsampled training images for Neural Congealing.
  • The conditional templates of Dalca et al. (2019) and Atlas-GAN (Dey et al., 2021) are added to the age-conditioned comparison in Fig. 7b.
  • DDIM initialized at zero is provided in Table 5 in the supplementary.

The new results give additional support to the previous conclusions in the paper, as our method is still the best or second-best on every dataset when considering these additional experiments. The controlled DDIM experiment shows similar performance on Brain MRI datasets but worse on other datasets.

W2. Why does the method converge?

Sharp template vs. voxel mean, and contraction conditions. The revised Sec. 3 answers both through the deformable-template model. Each image is the template under a random deformation, I(x)=Itemp(x−u(x))I(x) = I_{\text{temp}}(x - u(x)), expanded in Eq. 3. When the higher-order terms of Eq. 3 are negligible, the population is Gaussian with center ItempI_{\text{temp}}, and the iteration contracts every initialization to this center. This is the contraction condition: the population shares one template and the higher-order terms of Eq. 3 are negligible.

Pre-alignment. We agree these are conditions of the method, and the revised Sec. 3 now states them through pattern theory rather than leaving them implicit. The deformable-template model (Eq. 3) assumes that every member is a deformation of one shared template. Pre-alignment keeps the deformations within the regime where the iteration provably converges to the template, and background normalization removes content that no deformation of the template can produce. The FFHQ remedy illustrates the second condition, not the first: the faces are already landmark-aligned, and cropping to the face oval removes the unshared background, hair and clothing, so the population satisfies Eq. 3 and our method outputs a sharp template (Sec. 6).

Overlap. All evaluation subjects in the revision are excluded from training: Table 1's cohorts are disjoint from the fine-tuning scans (Mindboggle101 excludes OASIS-TRT-20), and Sec. 5 uses models trained without its evaluation subjects.

Questions.

Q1. Yes. For the age-utility experiment, we excluded the 126 evaluation subjects before fine-tuning and trained on the remaining 773 of 899 scans. Mindboggle101 is evaluated without its OASIS-TRT-20 subjects. We have updated the paper accordingly.

Q2. The repeated runs differ in ANTs' stochastic sampling during registration, with the template and subjects fixed.

Q3. We thank the reviewer for the careful reading. We have updated the paper accordingly. rr is the Pearson correlation between atlas age and CSF fraction, computed per seed and averaged over 5 seeds. The "0.2 pp" in the text was an error from an earlier model; the seed s.d. of the model in Fig. 7a is 1.2–6.9 pp. The band is ±1 s.e.m., which the caption now states, and the red bars are the ±1 s.d. of the reference cohort.

Q4. The pretrained 3D model is 3D MedDiffusion (Wang et al., 2024); we corrected the citation. Yes, the iteration runs in the 8-channel latent space of the 4×4\times VQ autoencoder, and the endpoint is decoded through the frozen decoder. ModelNet shapes are 32332^3 soft occupancy grids built from 8,192 surface points, modeled by a voxel-space DDPM. 3D SSIM is computed with the scikit-image implementation (737^3 window) on the decoded volumes within the bounding box of the foreground. We included these computation details in Section B.2.

Q5. We use Wilcoxon signed-rank tests on per-subject values, pairing each subject's registrations to the two atlases compared. Sec. 5 now names the test.

Reviewer_7j5eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a novel inference-time sampling strategy that extracts population-level anatomical templates from pretrained diffusion models without additional training or image registration. The authors demonstrate consistent template recovery across multiple imaging domains and the ability to generate age-specific brain atlases through conditional diffusion models.

Critical strengths
  1. There is strong empirical evidence showing that removing stochasticity from a pretrained diffusion model can produce a consistent population template.

  2. It's conceptually novel yet with extremely simple implementation. It has great applicability as there is no re-training involved.

  3. The simple framework can be used for practically all image domains (as long as the concept of "template" makes sense)

  4. It also unifies with the task of conditional template generation.

Critical weaknesses

I feel the authors use multiple concepts interchangeably. Throughout the paper I'm not sure how the population template or population consensus are mathematically defined. Are they population mean? mode? or the most probable image? These are different concepts that should not be mixed.

My understanding is that the proposed posterior-mean iteration is not guaranteed to converge to the mode or the mean. The results show 'seed-independent' outputs, but the discussion should be bounded by theoretical justification.

Accordingly, the discussion around the blurred image in Fig 8c is muddy. The concept “consensus” is underspecified. Could it be pointing to the 'mean' (but low probability) between several modes? or high uncertainty? low model capacity?

Questions for authors

My questions are raised in the above section. If the authors work further on the theoretical rigor and clearly separate mathematical constructs from "algorithmic phenomenon", I will consider further raising my score.

1 comment

We thank the reviewer for the precise questions. We agree that the submission used "template" and "consensus" interchangeably. The revision fixes the terminology issue, separates the mathematical construct from the algorithmic phenomenon, and adds the theory that bounds our claims (Sec. 3, with the derivation in the supplement).

W1. Definitions.

Template (mathematical construct). Following the pattern theory, we model a population as one template under random deformations, I(x)=Itemp(x−u(x))I(x) = I_{\text{temp}}(x - u(x)), where uu is a zero-mean Gaussian random field. For small deformations, the first-order expansion I≈Itemp−∇Itemp⊤uI \approx I_{\text{temp}} - \nabla I_{\text{temp}}^\top u makes the population Gaussian, N(Itemp,C)\mathcal N(I_{\text{temp}}, C). In this regime, the mean, the mode, and the most probable image coincide, and all equal ItempI_{\text{temp}}.

Consensus (algorithmic phenomenon). The posterior-mean iteration defines the endpoint Φθ(xT;y)\Phi_\theta(x_T; y). "Consensus" names the observation that this endpoint is the same for almost every initialization xTx_T. It describes the outcome, which is not guaranteed to be a meaningful "template". We acknowledge the misuse in our manuscript. We edited the text so that "template" is used wherever the meaningful outcome is meant.

W2. Convergence.

Our claim concerns populations that share a template, not arbitrary data. We revised Sec. 3 to make this scope formal. For a population following the deformable-template model with negligible higher-order terms (Eq. 3), the population is Gaussian with mean ItempI_{\text{temp}}, the posterior mean is linear in xtx_t, and every initialization provably converges to the template along each direction of population variance λ\lambda:

x0=Itemp+(∏t=1Tρt)(xT/αˉT−Itemp),∏t=1Tρt=αˉTλαˉTλ+1−αˉT.x_0 = I_{\text{temp}} + \Big(\prod_{t=1}^{T} \rho_t\Big)\big(x_T/\sqrt{\bar\alpha_T} - I_{\text{temp}}\big),\qquad \prod_{t=1}^{T} \rho_t = \frac{\bar\alpha_T\lambda}{\bar\alpha_T\lambda + 1 - \bar\alpha_T}.

Since αˉT≈0\bar\alpha_T \approx 0, the endpoint is independent of xTx_T, and it is both the mean and the mode of the population.

W3. The blurred image in Fig. 8c.

For uncropped FFHQ, we observe a blurred endpoint that is approximately consistent across seeds. In theory, the blurred image is still the center of a population, but shares no meaningful template. By Eq. 4, the iteration converges to the population's center, which equals the template when the population consists of deformations of one template (Eq. 3). For uncropped faces, the deformation field uu must account not only for the face structure but also for the varying hair, background, and clothing. These variations are large and are not deformations of a shared structure, so the higher-order terms of Eq. 3 are no longer negligible, and the center is an average over this content rather than a template, hence blurred.

Reviewer_SPHwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes extracting population atlases from pretrained diffusion models by removing the random noise added during reverse sampling and retaining only conditional-mean updates. For populations with consistent shared structure, different random initialisations converge to nearly identical images, termed “intrinsic atlases.” Evaluation across brain MRI, chest X-rays, faces and 3D shapes shows good performance.

Critical strengths
  1. The idea of using a diffusion model to construct atlases is novel. More specifically, a simple modification to the diffusion sampling process enables the extraction of a useful population template from a model trained for image generation. This extends the capabilities of an existing model without requiring a dedicated atlas-training objective.

  2. There are experiments from multiple domains including brain MR, X-rays, faces and 3D objects.

Critical weaknesses

1: The computational advantage is unclear. For a new dataset without a suitable pretrained diffusion model, the method requires training a new generator, which can be computationally expensive. Therefore, the overall computational savings over conventional optimisation-based atlas construction are not established.

2: The method does not produce deformation fields. Conventional methods such as ANTs and many learning-based approaches jointly provide a template and transformations relating individual subjects to it. These deformation fields support downstream applications, including segmentation propagation and statistical shape modelling. Producing only a reference image limits the proposed method’s utility as a complete atlas-construction framework.

3: Registration-based evaluation introduces additional computational costs. To compute Dice, the authors register each subject to the atlas using an optimisation-based pairwise registration method and compare the warped anatomical labels in atlas space. This requires additional computation, whereas some competing frameworks provide the necessary transformations as part of atlas construction.

4: Age-conditioned atlas construction is not novel. Dalca et al. previously demonstrated conditional template learning in Learning Conditional Deformable Templates with Convolutional Networks, published at NeurIPS 2019. The authors should clarify their methodological contribution relative to this work. The bibliographic entry cited at line 524 also appears to be wrong.

Questions for authors

Consistently producing the same image during the reverse diffusion process does not establish that this image represents the population’s anatomical centre. What exactly does the endpoint estimate, and under what assumptions? Does it represent mean anatomy, typical anatomy, or a reference determined by the learned distribution and sampling schedule? If it is interpreted as a mean, is it an arithmetic mean, a geometric mean, or a Fréchet mean under a specified anatomical distance?

The role of the training objective also requires clarification. Under ideal conditions, MSE-based noise prediction estimates a conditional expectation, but repeatedly applying conditional-mean updates does not generally recover the unconditional population mean. Would alternative training objectives, such as SSIM- or NCC-based losses, change the recovered reference? Such objectives would alter the estimator and potentially its probabilistic interpretation, so their endpoints could not automatically be interpreted as alternative “means.”

A convincing validation would use anatomically realistic synthetic data generated through known deformations of a known template, with an explicitly defined population centre. By varying subgroup proportions and anatomical differences, the authors could test whether the recovered reference (atlas) changes.

The dependence on spatial prealignment also requires investigation. The examples presented appear to contain objects that are approximately centred and consistently positioned. If objects were randomly translated across the image grid, would the method still converge to a unique, anatomically meaningful template, or would it produce a blurred image or an endpoint that depends on the initialisation? In this case, registration-based methods will be able to reconstruct consistent atlases. Controlled experiments varying translation and rotation would help establish whether the recovered atlas reflects intrinsic anatomical structure or depends on the spatial normalisation of the training data.

5 comments

Thank you for identifying the need to clarify our contribution and its scope. Our main finding is that a diffusion model trained for synthesis can also provide a useful population template without a template-specific training objective. Our claim is about pretrained diffusion models. A generator trained only to synthesize a population already contains that population's template, and the posterior-mean iteration recovers it from any seed, without retraining or a deformation model. Registration enters only at evaluation, to measure whether the recovered image is a good template. Sec. 3 now formally explains why the template exists. Under the deformable-template model of computational anatomy, a population with small deformations is approximately Gaussian around ItempI_{\text{temp}}, and the iteration contracts every initialization to ItempI_{\text{temp}}. We address each point below, including new experiments the reviewer requested.

W1. Computational advantage. Our computational advantage is clearest when a suitable generator already exists: extracting its template requires only inference, with no atlas-specific training. When a generator must first be trained or adapted, that cost also matters. In our experiments, brain MRI training took about 7 hours, and CelebA training took about 4.9 hours on one V100. The corresponding ANTs estimates were 30 and 32 hours on eight CPU cores. These timings describe the tested configurations; they are not a comparison on equal hardware.

W2 and W3. Deformation fields and registration cost. We agree with the reviewer, and we discussed this in Sec. 7. Our method is a template-construction step. In the evaluation, we register every template with the same method (Klein et al., 2009), which takes about 35 s per brain volume on 8 CPU cores. We will update the limitations section to state this explicitly.

W4. Relation to Dalca et al. (2019). Thanks for pointing this out. We do not claim to introduce conditional templates, and the paper cites Dalca et al. as prior work in Sec. 2. Our emphasis is different. Conditional atlases come for free from conditional diffusion models through our sampler, with no template objective and no template-dedicated training. The age family demonstrates this feature. The bibliographic entry is fixed (NeurIPS 2019).

Q1. What does the endpoint estimate?

We agree that producing the same image does not by itself establish that it is the population's center. The revised Sec. 3 now derives which image the iteration converges to under the assumption of a pattern-theory model. The pattern-theory model of the population is the following: every member is a deformation of one template, I(x)=Itemp(x−u(x))I(x) = I_{\text{temp}}(x - u(x)) with uu a zero-mean Gaussian random field, and the higher-order terms of its Taylor expansion (Eq. 3) are negligible. The endpoint is the template ItempI_{\text{temp}} of the deformable-template model, i.e. the population at zero mean deformation. Each image is then I≈Itemp−∇Itemp⊤uI \approx I_{\text{temp}} - \nabla I_{\text{temp}}^\top u, a linear function of uu, and the population center is ItempI_{\text{temp}}. The rest follows from the diffusion model itself: Eq. 4 shows that every initialization converges to ItempI_{\text{temp}}, which is both the mean and the most typical image of the population, and corresponds to the Fréchet mean under a deformation metric rather than the arithmetic mean of the images. The sampling schedule affects only how completely the iteration contracts, not where it converges.

Q2. Training objective. We thank the reviewer for the question. The convergence of our iteration to the template relies on Tweedie's formula, E[x0∣xt]=(xt+(1−αˉt)∇log⁡pt(xt))/αˉt\mathbb{E}[x_0 \mid x_t] = \big(x_t + (1 - \bar\alpha_t)\nabla \log p_t(x_t)\big)/\sqrt{\bar\alpha_t}, which holds for the score learned by an L2 (MSE) objective. SSIM- or NCC-based objectives do not estimate this score. Empirically, NCC fails as a training objective without producing meaningful results. Models trained with SSIM show better performance on the face dataset ([email protected]/@0.05 of 0.6800.680 and 0.4790.479, vs 0.6590.659 and 0.4380.438 for ours) but worse on the X-ray dataset ([email protected]/@0.05 of 0.5950.595 and 0.3330.333, vs 0.6850.685 and 0.4290.429 for ours). The recovered reference therefore depends on the objective. Our analysis characterizes the MSE case through Tweedie's formula. However, for SSIM or NCC, no analogous identity links the denoiser's output to the population distribution, so their convergence would require a separate derivation, which we list as future work in Sec. 7.

Q3. Synthetic Training. We added a controlled experiment using 500 deformations of the known template T. After fine-tuning on this population, the recovered image has NCC 0.964 to T, compared with 0.949 for the voxel mean and 0.656 for the original generator's endpoint. This supports recovery of the generating template and shows that the endpoint responds to a change in the modeled population. This experiment uses one template; it does not yet test changes in the proportions of distinct anatomical subgroups.

The real training scans are from UK Biobank, and a synthetic population is generated using the ANTs-OASIS brain template TT. For the latter, we placed TT on the grid of our training scans and generated 500 members, each TT warped by its own known displacement field (ANTs, diffeomorphic, mean 2.52.5 mm), so that the center of this population is TT by construction.

NCC to TT ↑
Ours, real population0.656
Ours, synthetic population0.964
Voxel mean of the 500 synthetic members0.949

Q4. Pre-alignment.

Pre-alignment helps keep anatomical variation within the small-deformation regime of our analysis. We tested sensitivity by fine-tuning the generator on rigidly perturbed scans (1923192^3 voxels, 1 mm isotropic, 1 mm = 1 voxel). NCC to the aligned reference was 0.908 for the smallest perturbations, 0.879 for the intermediate level, and 0.791 for the largest. Thus, recovery degrades gradually as alignment worsens rather than failing immediately.

MisalignmentNCC to the aligned atlas
±15∘\pm 15^\circ, ±10\pm 10 mm0.908
±40∘\pm 40^\circ, ±20\pm 20 mm0.879
±180∘\pm 180^\circ, ±20\pm 20 mm0.791

We will add this experiment to the supplementary material.

I do not think the authors’ response adequately addresses my concerns.

Regarding W1, fast optimization-based methods for atlas estimation already exist, including GPU-accelerated implementations. Learning-based methods such as AtlasMorph and MultiMorph can also jointly estimate atlases and deformation fields very efficiently.

Regarding W2 and W3, the authors acknowledge the limitations I raised. Estimating deformation fields is an important part of atlas construction because these fields describe how individual images align with the atlas. There is no such information in the proposed method.

Regarding W4, the authors appear to agree that conditional atlas estimation has been explored in prior work. The response leaves unclear what the proposed method contributes beyond existing approaches.

In addition, the authors’ replies leave my questions about means and pre-alignment unresolved. Their claims would benefit from a theoretical justification or illustrative experiments that show how pre-alignment affects the estimated means.

Overall, my main concerns remain unresolved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.