acceptodds
Under review as a conference paper at ICLR 2027

Multiscale Generative Modeling by Recursive Refinement

263 views

Abstract

Physical systems exhibit structure across spatial scales, from large-scale organization to fine details. Coarse representations describe these systems with fewer degrees of freedom but do not uniquely determine their fine-scale structure. Learning the distribution of these unresolved structures allows generation to proceed through smaller refinement problems, each conditioned on the coarse structure already established. Here, we introduce a discrete multiscale generative framework that constructs samples through recursive refinement directly on the native grid, without a learned encoder or decoder. A single model is shared across scales and learns the conditional distribution of fine-scale configurations, with an explicit spatial correspondence between coarse and fine grids. The framework supports different stochastic refinement processes, including spin dynamics for binary structures and mass-conserving dynamics for particle allocation. The conservative processes preserve particle counts exactly, ensuring that every generated refinement aggregates to its prescribed coarse state. We demonstrate the framework on natural images, Julia-sets, and three-dimensional porous media. For Julia-sets, recursive refinement extends to scales unseen during training. For porous media, the model generates volumes of voxels with exactly prescribed porosity (void fraction), with no fixed architectural limit on volume size. We show that a single refinement model can generate structure across scales while honoring coarse-scale constraints.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific data, generative modeling, multiscale, 3D

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wU8dOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a generative method via recursive refinement. The method generates spatial fields through repeated coarse-to-fine refinement using one network shared across scales. Experiments cover Julia sets, natural images, and porous media.

Critical strengths
  • The recursive refinement formulation can be useful intuitively and enables fundamental applications.
  • The method is interesting and gives encouraging initial results.
  • This paper is technically sound.
Critical weaknesses
  • The quantitative metrics are weak.
  • The evaluation task narrows the potential application of the method. And the experiments on the Julia set may rely on the prior assumptions of the method.
Questions for authors
  • The method formulates the multi-scale training and sampling borrowing the lessons from diffusion models. It is interesting to discuss the scale generalization and non-Markovian sampling.
  • Why was a single Julia set chosen as the main quantitative benchmark for scale extrapolation? Given its recurring structure across scales, could you also evaluate on other Julia-set parameters and natural-image super-resolution tasks to test whether the learned refinement rule generalizes beyond this particularly favorable setting?
1 comment
AuthorsOpenReview ↗

Response to Reviewer wU8d

Thank you for recognizing the technical soundness and potential of the formulation. We have strengthened the quantitative evaluation and tested the limits of the learned refinement rule.

[W1] We trained a 2563256^3 Berea sandstone model and evaluated 150 generated realizations. Values are means ± standard deviations:

MetricReal cropsGenerated
Porosity (%)21.69 ± 2.4821.69 ± 2.48
Surface density0.0977 ± 0.00760.1002 ± 0.0428
Mean pore radius (voxels)2.68 ± 0.132.52 ± 0.06
Pore fraction connected along xx (%)97.93 ± 1.4796.39 ± 2.28
Connected porosity (%)21.26 ± 2.6520.97 ± 2.07
Permeability (mD)508 ± 119444 ± 67

We also computed two-point correlations on a separate subset ([W1 of rev-e78C]). These metrics assess learned geometry and transport beyond exact count conservation.

The Julia evaluation compares 16 held-out trajectories against recursive nearest-neighbor refinement and a deterministic transformer with the same architecture. At scale 10, the second unseen scale, relative pixel error is 5.65 ± 2.79% for our model, 18.06 ± 5.70% for the deterministic model, and 42.64 ± 15.71% for nearest neighbor. A new Appendix additionally reports CIFAR-10 ablations on sampling budgets, conditioning, and shared versus independent networks. The new conditional-image results are given in [Q2].

[W2] Julia sets provide fine-scale ground truth at successive zoom levels, including scales excluded from training. We agree that their recurring structure makes them a favorable setting for a controlled test of recursive error accumulation.

The network learns from images using spin refinement and shared weights, without scale conditioning. The Julia equation supplies the reference images, it is not used as the refinement rule. The assumption is that learned refinement statistics remain useful at finer scales. Weight sharing makes this possible; it does not guarantee it. Our experiments test that assumption and expose its limits.

The three applications deliberately test different capabilities of the same framework: Julia sets test extrapolation with ground truth, natural images test conservative conditional generation, and porous media test large-volume generation with exact count control and physical evaluation. The new parameter-transfer experiment below tests whether the learned rule remains useful beyond the training fractal.

[Q1] Within a given refinement stage, the process is Markovian: the fine configuration, reservoir counts or flip indicators, time, and fixed coarse condition. Across scales, each refinement conditions on the preceding coarse representation.

In a completed conservative hierarchy, earlier coarse grids are determined by aggregating the current grid. Supplying them again can change the parameterization but adds no observations. A history-dependent within-stage sampler would require discrete dynamics preserving the intended distribution and constraints. We have not evaluated this extension. Could you clarify whether you mean history across resolutions or an alternative sampler within each resolution?

[Q2] We now test the frozen model on 14 additional Julia parameters, perturbing the real or imaginary component by magnitudes from 10−410^{-4} to 10−110^{-1}. Two independent trajectories per parameter give 28 trajectories. Each starts from its true scale-0 image and then uses its own predictions recursively, without retraining. Scale-10 errors range from 4.9% to 30.3% at magnitude 10−410^{-4}. These results show useful transfer of the frozen model.

We also extend one held-out trajectory to scale 18, ten scales beyond training on 1–8. This example uses eight test-time augmentations with majority voting and shows eventual error accumulation; it is separate from the 16-trajectory comparison above. The final relative error at scale 18 is 12%.

For natural images, we added conditional refinement from held-out CIFAR-10 images. All methods use the same coarse inputs, produce 32×3232\times32 outputs, and use the same FID evaluation:

Coarse inputNearest neighborBilinearOurs
4×44\times4361.84255.1819.14
8×88\times8290.05160.2012.10
16×1616\times16140.9967.695.70

FID uses 10,000 outputs against the test set; learned refinement uses 512 reverse steps per remaining scale. They measure conditional generation within the training resolution range. The Appendix separately shows 32→64→12832\to64\to128 refinement without fine-tuning. That extrapolation is qualitative since CIFAR-10 has no corresponding high-resolution targets.

Reviewer_e78COpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This study introduces a multiscale generative framework that recursively constructs samples from coarse to fine. A single model is shared across scales, adding detail directly on the native grid without a learned encoder or decoder. Conservative refinement processes preserve particle counts exactly, ensuring that fine-scale configurations aggregate to the prescribed coarse state. Experiments cover natural images, Julia sets, and three-dimensional porous media, demonstrating refinement at unseen scales for Julia sets and exact porosity control for porous media.

Critical strengths
  1. The method generates diverse fine-scale structures while strictly satisfying coarse-scale constraints. It clearly separates constraint enforcement, learning, and computation, offering practical computational benefits.
  2. The experiments cover a broad range of settings: Julia sets, natural images, and three-dimensional porous media test scale extrapolation, conditional generation, and large-volume generation with exact conservation, respectively.
Critical weaknesses
  1. The porous-media experiments demonstrate 102431024^3-voxel generation and exact porosity control, but provide limited evidence of structural quality. Porosity is guaranteed by the transition rules, so even unrealistic structures can satisfy this constraint. Pore morphology should be validated through pore-size distributions, two-point correlations, and connectivity. If fluid transport applications are emphasized, permeability or effective transport properties should also be evaluated.
  2. The 102431024^3 experiment does not demonstrate generalization beyond the training size, since the training data already consist of full 100031000^3 volumes padded to 102431024^3. It shows that the method can train and generate at this scale while satisfying exact constraints. By itself, it does not establish that a model trained at coarser resolutions can generate statistically correct finer-scale structures.
Questions for authors
  1. How does computational cost scale with generation size, and how does quality change beyond the training size? How long does it take to generate one 102431024^3 sample? Have larger generation domains been tested, or has generalization from smaller training regions to larger domains been evaluated?
  2. How does the sampling procedure ensure both conservation and valid voxel values? Equation (4) conserves the total number of particles in the reservoir and the fine grid, but how is the reservoir guaranteed to empty within a finite number of sampling steps? For binary outputs, how is each voxel restricted to 00 or 11, preventing particle allocations from exceeding its capacity?
3 comments
AuthorsOpenReview ↗

Response to Reviewer e78C

Thank you for the detailed feedback and for recognizing the separation between constraint enforcement, learning, and computation. We have substantially strengthened the quantitative evaluation and clarified the scope of the generalization claims.

[W1] We agree, exact porosity alone does not establish realistic pore structure. We have trained a 2563256^3 Berea sandstone model and evaluated 150 generated realizations. Values are means ± standard deviations:

MetricReal cropsGenerated
Porosity (%)21.69 ± 2.4821.69 ± 2.48
Surface density0.0977 ± 0.00760.1002 ± 0.0428
Mean pore radius (voxels)2.68 ± 0.132.52 ± 0.06
Pore fraction connected along xx (%)97.93 ± 1.4796.39 ± 2.28
Connected porosity (%)21.26 ± 2.6520.97 ± 2.07
Permeability (mD)508 ± 119444 ± 67

These measurements directly assess morphology, connectivity, and transport beyond the imposed volume fraction. The generated volumes show similarly high connectivity. We also measured the two-point pore correlation S2(r)S_2(r). Selected separations are shown below as 100S2(r)100S_2(r), in percent, with mean ± standard deviation:

Separation (voxels)RealGenerated
021.43 ± 2.2621.43 ± 2.26
119.36 ± 2.1318.94 ± 1.65
415.30 ± 1.8914.68 ± 1.27
811.24 ± 1.6210.77 ± 1.08
167.24 ± 1.287.01 ± 1.00
324.88 ± 0.994.91 ± 0.97
484.70 ± 0.964.64 ± 0.97

The correlation of the generated samples follows the one of the original Berea crops, lower values at short separations and close agreement at larger separations.

[W2] The reviewer is correct. The porous-media experiment trains and generates over the same scale range: 100031000^3 training volumes are padded to 102431024^3. This is the intended scope: large-volume generation with exact local count consistency. Coarse observations alone do not determine unresolved fine structure without additional information or assumptions. We test scale extrapolation separately on Julia sets, with ground truth at unseen scales.

The Julia model trains on scales 1–8 and rolls out autoregressively through scales 9–10 without scale conditioning. Table 2 reports relative pixel errors across 16 held-out trajectories (mean ± standard deviation):

ScaleNearest neighborDeterministic transformerOur model
935.12 ± 10.95%11.89 ± 3.22%3.80 ± 1.90%
1042.64 ± 15.71%18.06 ± 5.70%5.65 ± 2.79%

The revision also examines a longer trajectory through scale 18 and transfer to 14 new Julia-set parameters. These results support extrapolation in the tested settings. The autoreggresive error at scale 18 is 12%, starting from scale 0.

The model learns refinement structure from training data; sharing weights does not guarantee statistically correct structure at arbitrary unseen scales.

[Q1] Each evaluation uses an 838^3 central window plus a one-cell halo: 1,000 parent tokens, with only the center updated. Dense coverage requires ∑sKsWs\sum_sK_sW_s evaluations (KsK_s reverse steps, WsW_s windows per step). With fixed windows and step budgets, work scales linearly with output voxels. Fully determined regions can be skipped. Network memory per window is bounded; field storage grows with volume. Full 102431024^3 timing and peak-memory measurements, with hardware and sampling settings, will follow. Generation beyond 102431024^3 and statistical generalization from smaller training regions to larger porous domains are out of scope for this work.

[Q2] Algorithm 2 explicitly enforces both finite reservoir release and valid voxel values. Every accepted allocation transfers one particle from its parent reservoir to a child, preserving Ri+∑jNj=ciR_i+\sum_jN_j=c_i. Proposals are capped by each child's remaining capacity and, if necessary, reduced to the available reservoir count.

At the final reverse step, all remaining particles are allocated, sampling only children with available capacity. For a feasible parent count cic_i and child capacities CjC_j,

∑j(Cj−Nj)=∑jCj−ci+Ri≥Ri\sum_j(C_j-N_j)=\sum_jC_j-c_i+R_i\geq R_i.

Thus there is always sufficient capacity, and the terminal loop ends after exactly RiR_i allocations. Children become ineligible as they fill. At the finest level, Cj=1C_j=1; integer counts therefore lie in 0,1{0,1}. Intermediate cells hold larger counts because they aggregate multiple final voxels. These guarantees hold independently of the learned rates. The network learns where particles go, the sampler ensures that every completed refinement exactly reproduces every prescribed count.

Thank you for your rebuttal, which addressed part of my concerns. However, I still feel that the paper does not meet the standards of ICLR. So I will raise my score to 2.

Thank you for reconsidering your assessment. Your feedback has helped us strengthen the paper, particularly its quantitative evaluation. We believe our work represents a substantial methodological advance over DSD (NeurIPS 2025 Spotlight). Could you clarify which specific concerns remain, and why you consider the revised contribution below ICLR’s standards? This would help us address the remaining issues directly.

Reviewer_26cUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a discrete multiscale generative framework that synthesizes the physical systems via iterative refinement directly in the canonical space. The generation progresses from coarser to finer, and each iterative generation step is conditioned upon the coarse structure established thus far. A single shared network with an explicit spatial correspondence between coarse and fine grids operates at all the scales. The proposed framework uses various stochastic refinements at each scale that codifies certain physical processes such as spin dynamics and mass conservation dynamics. Authors demonstrate the effectiveness of their framework on Julia-sets, 3D porous media, and natural images.

Critical strengths
  • The problem of generating physically consistent media is relevant and important interdisciplinary problem, and the high-level approach of iterative refinement is reasonable and valid.
  • Sharing a refinement model across spatial resolutions allows it to leverage multi-scale self-similarity, and can potentially help it achieve generalization for the resolutions unseen during training e.g. as showcased in the Julia-sets example.
Critical weaknesses
  1. The paper presents the framework as a “general paradigm” for enforcing physical constraints across scales, but it provides only four basic discrete transition processes as examples. These processes are limited to binary variables or scalar count conservations. In reality, scientific applications often involve continuous vectors/fields, and the constraints are often ‘differential’ in nature. It is unclear how such transitions could be used within this framework. In my understanding, in the current framework, each process needs to satisfy the requirements of having tractable forward marginals and precise reverse-time sampling defined. Authors should show more sophisticated physical structure scenarios such as predicting fluid transport within the 3D porous media that may involve continuous or differential physical constraints.

  2. No quantitative evaluation was provided to verify the physical consistency (claim of “exact control of porosity”) of the generated 3D porous media. Fig. 5c shows qualitative examples along with “prescribed porosity” values, which I believe are used as conditional input during the generation, but did not indicate the precise porosity value of the resulting structures. Ideally the model should be verified both qualitatively and quantitatively over a larger and more diverse eval set, and the physical consistency should be provided using error between input condition and measured value in the generated media.

  3. Experimental setup is quite limited in the case of natural images: showing results on only CIFAR-10 and FFHQ is inadequate: both datasets lack diversity, and more importantly, the results are shown at the terminal resolution of 32 x 32 and 64 x 64 which is unreasonably small for the paper claiming to produce “finer details” through iterative refinement. Moreover, the FID scores reported are significantly worse compared to existing general approaches such as vanilla flow matching.

  4. No comparisons are provided against existing 3D porous media baselines (e.g. Controlled Latent Diffusion Models for 3D Porous Media Reconstruction, Naiff et al.). Some comparisons, even at smaller resolutions, are necessary to determine if the physical fidelity matches the current baselines.

  5. The methods section is unclear and lacks several critical implementation details due to which some of my questions remain for the authors (see questions section). Compute and run time information is also not provided (crucial for porous media generation task at the volume of 1024^3).

Questions for authors
  1. How is x(0) sampled / initialized for unconditional generation in the case of CIFAR-10/FFHQ results?

  2. Were adjacent cubed windows and their 1-cell halos synchronized after every reverse step?

  3. Provide the exact mathematical/algorithmic specifications for spin flips, axis-wise allocation, and local redistribution.

  4. The shared model's architectural hyperparameters (number of layers / number of attention heads, MLPs, positional encoding, or how process time and scale index are embedded and combined) are not described.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 26cU

[W1] The reviewer is correct. Reservoir allocation has tractable forward sampling: Nt,j∣N0,j∼Binomial⁡(N0,j,α(t))N_{t,j}\mid N_{0,j}\sim\operatorname{Binomial}(N_{0,j},\alpha(t)), where α(t)\alpha(t) is the survival probability. The network learns reverse allocation rates; see Equation (2) and Appendix A.

We respectfully disagree that predicting a continuous fluid velocity field is necessary to establish our contribution: generative refinement between representations with different numbers of degrees of freedom. Even enforcing ∇⋅u=0\nabla\cdot u=0 requires a field representation, differential operator, and boundary conditions. Grid methods use discrete operators; finite spectral methods retain selected modes. How should a physical constraint be preserved when the computational state space itself changes? The operators and refinement maps must be compatible, and the model must learn the unresolved structure within those constraints.

A generally applicable solution across physical systems would be a huge advance in scientific generative modeling. That ambition motivates this work. Here, every completed refinement reproduces every coarse cell's prescribed count exactly, recursively across scales, independently of the learned rates. This is a hierarchy of local physical constraints, substantially more than matching global porosity. Permeability [W4] directly assesses transport in the generated geometries. Julia images at successive zooms are not count aggregates, so imposing conservation would enforce a relationship absent from the data. Spin refinement preserves binary values while allowing occupancy to change. The constraint must match the representation.

[W2] Porosity is imposed through a integer pore count MM: ϕ=M/V\phi=M/V for VV voxels. The prescribed count error is exactly zero. Algorithm 2 enforces capacities and empties the reservoir at the final step.

[W3] Natural images test coherent conservative refinement and variability across scales; very high-resolution quality remains untested. Our main application is scientific spatial generation. We also added conditional refinement on held-out CIFAR-10 images. For 16×1616\times16 to 32×3232\times32, FID is 5.70 versus 67.69 for bilinear interpolation with the same inputs. This evaluates conditional generation within the trained resolution range.

On 10,000 generated CIFAR-10 images our FID is ~20. DSD, the only other method that preserves total intensity, reports 46.3. Our 37.85M-parameter model was trained on an HP ZGX nano without optimizer or learning-rate sweeps. A 512×512512\times512 AFHQ experiment is underway; results will follow soon.

[W4] We have trained a 2563256^3 Berea model and evaluated 150 generated realizations. Values are means ± standard deviations:

MetricReal cropsGenerated
Porosity (%)21.69 ± 2.4821.69 ± 2.48
Surface density0.0977 ± 0.00760.1002 ± 0.0428
Mean pore radius (voxels)2.68 ± 0.132.52 ± 0.06
Pore fraction connected along xx (%)97.93 ± 1.4796.39 ± 2.28
Toal connected porosity (%)21.26 ± 2.6520.97 ± 2.07
Permeability (mD)508 ± 119444 ± 67

These measurements provide evidence beyond matching porosity. The generated volumes show similarly high pore connectivity to the real crops.

[W5] Each evaluation uses an 838^3 central window plus a one-cell halo: 1,000 parent tokens, with only the center updated. Dense coverage requires ∑sKsWs\sum_sK_sW_s evaluations (KsK_s reverse steps, WsW_s windows per step). With fixed windows and step budgets, work scales linearly with output voxels. Fully determined regions can be skipped. Network memory per window is bounded; field storage grows with volume. Full 102431024^3 timing and peak-memory measurements, with hardware and sampling settings, will follow.

[Q1] The initial mass is a 1×11\times1 RGB vector of channel totals from 10,000 randomly selected training images. This distribution could also be learned. We will state this initialization explicitly in the revised manuscript.

[Q2] Could you clarify what you mean by synchronization? During training, we randomly select coarse–fine image pairs and update the model.

[Q3] We agree that the individual processes need precise specifications and will include them in the revision due to the space constraints here.

[Q4] The 2D RGB model uses 12 transformer layers, width 512, eight 64-dimensional heads, pre-layer normalization, global attention within the context, and two-layer GELU feed-forward blocks of width 2,048. State features, normalized cell-center coordinates, time in 0,10,1, and scale ℓ/4\ell/4 are concatenated and projected to width 512. There are no learned or sinusoidal embeddings. The output has 12 values (four children per RGB channel), made positive by a Softplus function. Weights are shared across scales; Julia refinement omits scale conditioning.

[Q5] Compute and run-time information

For a 102431024^3 realization using the following schedule per scale: (256,192,128,96,64,64,64,64,64,64), the sampler performs

$ \sum_l T_l\max\left[1,\left(\frac{2^l}{8}\right)^3\right] =19{,}174{,}560 $

window evaluations. Therefore, we can perform batched inference using:

$ C_B=\sum_l T_l\left\lceil \frac{\max[1,(2^l/8)^3]}{B} \right\rceil $

network evaluations. The original implementation with B=32 requires 599,904 forward calls of the model. These evaluate the complete spatial lattice, and do not assume that we might be skipping evaluations in fully resolved regions (either solid or void).

On a single GPU using a heavyly optimized inference code with B=1024, BF16 precision, skipping regions, batched spatial parallelism, and a compiled model we can reduce the number of network evaluations to 19,552. The full 102431024^3 realization requires 40.6 s for sampling. Peak GPU memory was 9.14 GiB allocated and 11.67 GiB reserved.

[Q1-bis] Additionally, we have trained a small MLP to parameterize the distribution of the initial 1×1×31\times1\times3 RGB scale, which contains the images' channel totals. Sampling from this learned distribution enables fully unconditional CIFAR-10 generation, and our model achieved an FID of 22.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.