acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Normalizing Flows for Interpretable Scale-wise Extrapolation

Abstract

Generative models of multiscale physical systems usually lack the inductive bias needed to generalize to larger systems than they were trained on. We study scale-wise extrapolation with a hierarchical generative model whose levels are the maps between adjacent scales. Because each level adds one scale, a model of a larger system can be constructed by adding a new level, often a copy of an existing one, rather than by retraining. This extrapolation is possible when the way the target distribution changes across scales has low-dimensional structure. The simplest case is self-similarity, which the renormalization group (RG) formalizes as a fixed point of a coarse-graining map. Our proposed model is a hierarchical normalizing flow whose levels are local, symmetry-equivariant continuous normalizing flows, trained with regularizers and a normalization that fix the reparameterization freedom, making the levels identifiable and interpretable. We characterize this freedom and give conditions under which the regularizers leave at most one optimal solution. We then test the method on two-dimensional lattice models, which are exactly self-similar at a phase transition and can be checked against exact results and Monte Carlo simulations. For the 2D Ising model, independently trained models learn the same levels, and across temperatures the learned levels trace a low-dimensional manifold reflecting the RG flow. At the phase transitions of the Ising and XY models, we show that a model trained at one lattice size extends to larger lattices by repeating one level, without new training. In both cases, the extrapolated models match models trained directly on larger systems, whereas a parameter-matched flow-matching model extrapolates poorly.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
normalizing flows, hierarchical architecture, multiscale generative modeling, interpretability, renormalization group, Ising model, physics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Z5H3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors put forward a hierarchical generative model which is claimed to learn almost unique (up to local monotonous stretching) latent representations. For simple self-similar data from physics, it shows a better ability to extrapolate to larger scales.

Their pipeline is as follows: They have continuous degrees of freedom on sites. They first use translation invariance of the data and apply a local/site-level non-linear transformation to make site-marginals Gaussian. In the manuscript, they commit to 2x2 site blocks. They then transformer each configuration using a two-step coarse-graining (each one realized by an MLP modelling the velocity of a normalizing flow): One shifted block transformation, producing "external magnetization" fields coming from nearby blocks, and another transformation, acting within the blocks, which again outputs 4 variables -- one computed as the mean over the block is interpreted as the collective degree of freedom to be pushed to later steps, and 3 other variables (z's) which describe the strictly local information at that scale. Their generative model assumes the z's are decoupled Gaussians. Their final loss, further encourages Gaussian behaviour of each site value of xlx_l. Matching the log-likelihood, together with the latter loss, encourages the transformations to push $ x$'s (conditioned on the previous layer) and $ z$'s into their appropriate Gaussian behaviour. Finally, a Wasserstein-2 penalty based on the velocity field is added to the loss to support uniqueness.

The authors further claim to prove that for realizable distribution, uniqueness can be proven up to the aforementioned stretching.

On the numerical side the authors demonstrate that their approach beats flow matching using a U-Net (with suitable U-Net extrapolation), and looks stable across O(10) extrapolation (at least when tying the layers, namely in the repeated mode). Direct comparisons between between MC and their approach show some discrepancies but not major ones, and clearly much better than the baseline. Finally, tracking internal representations in PCA space, they find similar structures to RG flows.

Critical strengths

The paper studies an interesting qualitative problem (generative models for self-similar physics that reflect RG) and offers a valuable addition to prior work, promoting uniqueness via regulators and establishing some related mathematical results.

The method shows promise for studying critical systems and might help extract RG flows of more complex theories.

Critical weaknesses

I found the work quite hard to read and decipher. For instance, the paragraph "Hierarchical block flow" is extremely dense and often uses Jargon and references when it can simply state what is being done. It anthropomorphises objects (e.g. a decimator, which decides what each cell passes to the next scale), instead of indicating how they are defined and what pushes them to the right "decision" (it the loss objective and that's not stated clearly). It uses dramatization and narrative suspense (e.g. "Objective. This architecture realizes a density in many ways, since an invertible map applied at each site of a hidden field (hℓ or xℓ, the interfaces between layers) and undone by the next layer leaves qθ unchanged"-- how does this amount to an objective? what selects "among them" later on).

For a physics readership, perhaps the key readership of such a work, a more detailed study of the critical properties would be nice -- how far are critical exponents computed from this ansatz? How well can you estimate the critical temperature?

For an ML readership, the applications and benefits to real-world data (image, language) are not obvious. Also, this problem lacks clear benchmarks, and extrapolating the U-Net might not be the cutting-edge technology here. Images most likely contain a mix of different patch sizes as far as coarse-graining goes, and one committed patch size may cut off important features.

Questions for authors

Can you use the Sandik method (https://cpl.iphy.ac.cn/EN/abstract/article_105571.shtml) to estimate several critical exponents from your ansatz?

1 comment

Readability. Both paragraphs are rewritten. "Hierarchical block flow" now defines the disentangler and the decimator as block maps applied to every cell, and states that neither is prescribed: both are learned, and the objective determines them, including which function of the four cell values the decimator keeps. "Objective" now opens with the loss, then says why the likelihood alone does not determine the block maps and what each regularizer selects.

Exponents and β_c. Our method is not designed to replace finite-size scaling or Monte Carlo for measuring exponents. Its purpose is to use the structure of the RG to interpret the learned block maps and to extrapolate without training data at the larger size, when low-dimensional structure such as self-similarity exists.

However, we can use the exponents as a further check of the extrapolation, with power-law fits at β_c over the extended models at L=128 to 1024 (Table B.11). Again, this is done without additional data at larger system sizes (no data or training beyond L=64).

ExponentNRG repeatFM naiveExact
β/ν0.115 ± 0.0010.505 ± 0.1140.125
γ/ν1.768 ± 0.0031.403 ± 0.2731.75

The models are trained at the exact β_c, so the extrapolation assumes β_c and does not estimate it. Estimating it requires sweeping a fine grid of βs. Extrapolating generative models, like ours, might be a good way to accelerate MCMC sampling to generate the data required for this kind of estimate, but this is outside of the scope of this work.

Real-world data. Our method is targeted at multiscale physical systems, where the ideas from the renormalization group give a reason to expect low-dimensional structure across scales. As a first test on more complex data, Appendix B.8 applies the model to a passive scalar in forced 2D turbulence (TURB-Scalar, Calascibetta et al. 2025), whose scaling is intermittent and multifractal. The model successfully identifies the low-dimensional structure: at fine scales the learned decimators are nearly self-similar and evolve slowly with scale (Figure B.11). An L=128 model extended to L=256 without training there outperforms naive extrapolation of a flow-matching model (Table B.13, Figure B.12).

Model (128→256)SpectrumS2S4
NRG interpolate0.2050.1040.280
FM naive1.2210.3250.451
FM upsampled2.2140.2510.635

Errors are mean absolute log ratios to held-out data for the power spectrum and the second- and fourth-order structure functions. FM upsampled is the flow-matching model's L=128 samples interpolated to L=256.

In this experiment, the larger non-universal scales are captured less well, and the self-similar extension does not fully account for the intermittency. We do, however, see evidence of imperfect but non-trivial extrapolation (Table B.13, Figure B.12) and similarity/low-dimensional structure between scales (Figure B.11). Future work should consider combining our interpretable hierarchical model with more flexible generative models for non-universal behavior, and study extrapolation in the presence of intermittency.

Benchmarks. Because we are in the unusual problem setting of attempting to do interpretable extrapolation, not many other approaches exist. Super-resolution methods, for example, require data at the higher resolution to successfully generalize to larger sizes. If we assume there are no longer-range correlations beyond the current system size (finite correlation length), our flow-matching baseline is a simple and natural choice when trying to extrapolate convolutional architectures without additional high-resolution data. This, of course, breaks down at or near non-trivial fixed points (like the critical point), which is where our method has an advantage.

Patch size. The locality built into our model is both an advantage for interpretability and a limitation in expressivity, especially when there are highly non-local interactions. We are, in principle, able to progressively increase the receptive field by adding more disentangler layers, which our theory allows for, so this may be one way to improve expressivity in a controlled way in more complex systems.

Reviewer_bpfMOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper constructs a normalizing flow to sample field configurations for physical systems on a lattice. The flow consists of continuous-normalizing-flow blocks that split of a fraction of the latent variables and sample them from the latent distribution, resulting in a hierarchical structure similar to e.g. RealNVP. On top of the usual NLL-term, the paper proposes two additional regularization terms in the loss which make the blocks identifiable, as the authors show. The hierarchical structure of the flow means that, at the critical (self-similar) point of the physical system, a model for sampling states of a larger system can be created by repeating the blocks in a flow trained for a smaller system size. The identifiability means that blocks trained at different points in phase space of the system can be compared and a PCA analysis reveals the structure of the phase space. Empirical results are presented for the 2D Ising model and the XY model.

Critical strengths
  1. The core idea of using the hierarchical structure of a normalizing flow to construct a model suitable for a different lattice size is novel and interesting. The empirical results support the feasibility of this idea.
  2. Constraining the layers of the flow with additional regularization terms to make them identifiable is novel as well. The resulting empirical analysis of the phase space of the 2D Ising model is an interesting application.
Critical weaknesses
  1. As the authors mention in the limitations, the proposed extrapolation to unseen system sizes only works at the critical (self-similar) point. This is a substantial limitation since often physical systems need to be studied away from criticality. The authors mention possible interpolations of the blocks which could lead to a generalization, but do not provide any empirical evidence.
  2. The paper is hard to understand at times. In particular Section 3 is very dense and introduces a lot of new terminology. It contains very few equations but instead comparatively vague descriptions that are more difficult to parse than explicit mathematical expressions. The implications and significance of the various assumptions are not sufficiently discussed, aside from a short comment in the limitations.
Questions for authors
  1. How much of your setup is specific to the two systems you study? What would be required to generalize the setup to other systems?
  2. What is the impact of the additional regularization terms on the performance of the model? Did you compare the performance with and without these terms? Do you have an intuition for why the flow matching baseline outperforms the NRG baseline in Table 1?
  3. Why do you select the penultimate level for repetition in order to increase the system size that the model can handle? Did you arrive at this choice via an empirical ablation or did you have other reasons?
4 comments

W1, criticality only. Away from a transition the block maps change with scale, so a new level cannot be a copy. However, for Ising, because we expect and see that the learned block maps collapse onto a curve, we can graft blocks from a model trained at a shorter effective correlation length (at the same system size) to fill in the missing pieces. This allows us to extrapolate to larger sizes (L=128 to 512) even off-criticality by using two models trained on data at the smaller size (L=64).

In new experiments, we test this at ξ=32 on both sides of the transition: L=64 models are extended with the coarsest levels of L=64 models trained at ξ=16, 8, 4 (for extrapolating to L=128, 256, 512, respectively), with no training beyond L=64. The extended model beats the naive extrapolation in most cases, especially with increasing extrapolated system size. The naive flow-matching transfer remains reasonable on the disordered example, where the flow approaches the disordered fixed point and the block maps approach the identity (this is the best-case scenario for naive extrapolation).

Ordered side (Table B.8):

ModelLTV32|Δ⟨|m|⟩|mean |C/C_ref − 1||ΔU4|
NRG graft64→1280.075 ± 0.0140.0095 ± 0.00160.028 ± 0.0040.0011 ± 0.0000
FM naive64→1280.237 ± 0.0050.0598 ± 0.00040.131 ± 0.0030.0276 ± 0.0011
NRG graft64→2560.103 ± 0.0350.0072 ± 0.00300.023 ± 0.0080.0001 ± 0.0001
FM naive64→2560.570 ± 0.0120.1288 ± 0.00060.291 ± 0.0020.0498 ± 0.0026
NRG graft64→5120.119 ± 0.0500.0070 ± 0.00360.021 ± 0.0100.0004 ± 0.0001
FM naive64→5120.927 ± 0.0070.2127 ± 0.00370.471 ± 0.0040.0774 ± 0.0058

Disordered side (Table B.9):

ModelLTV32|Δ⟨|m|⟩|mean |C/C_ref − 1||ΔU4|
NRG graft64→1280.028 ± 0.0000.0008 ± 0.00010.088 ± 0.0010.0292 ± 0.0001
FM naive64→1280.029 ± 0.0020.0044 ± 0.00370.170 ± 0.0070.0159 ± 0.0053
NRG graft64→2560.013 ± 0.0000.0005 ± 0.00000.114 ± 0.0020.0257 ± 0.0003
FM naive64→2560.028 ± 0.0080.0049 ± 0.00220.442 ± 0.0010.0114 ± 0.0028
NRG graft64→5120.012 ± 0.0000.0012 ± 0.00010.127 ± 0.0010.0382 ± 0.0025
FM naive64→5120.023 ± 0.0020.0019 ± 0.00090.567 ± 0.0110.0187 ± 0.0001

W2, Section 3. Section 2 now defines every object Section 3 uses. In Section 3 the setup defines a hierarchy, the coarse value and the latent variables before the first theorem, Theorem 1 lists the three families of reparameterizations separately, and Corollary 4 states the no-tie hypothesis in full. Following your suggestion, we replaced verbal descriptions with explicit expressions: the action of a reparameterization (Equation 4), the conditional independence in Theorem 2, the bound of the kinetic energy by the mean squared displacement (Equation 5) and the two inequalities of the displacement condition (Equation 6). On the significance of the assumptions, the section now closes with a paragraph that says which hypotheses are assumed and not derived, that a trained model fits only approximately, and that the agreement of independently trained models is the empirical test. Appendix E gives the full list.

Q1, generality. In its current form, we expect to be able to train our model on any system with local interactions and translation symmetry while incorporating any additional symmetries into the equivariant block design. For example, the same model used on Ising can be applied to φ⁴, which recovers block maps that agree with the Ising ones beyond the finest non-universal levels (Appendix B.7). To achieve extrapolation, however, the system must exhibit some form of low-dimensional structure in scaling behavior, such as self-similarity. To address this in the paper, we include a new paragraph in Section 5.

Q2, regularizers. The likelihood alone does not determine the block maps. Following our theoretical work, the kinetic energy makes the maps unique but not comparable across levels, and the marginal term makes them comparable even across different models. This combination is what enables the interpretable extrapolation that we perform.

In a new set of experiments, we build the objective up one term at a time at L=64 and extend each model by repeating level 4 (Table B.10).

ExtensionObjectiveTV32|Δ⟨|m|⟩|mean |C/C_ref − 1||U4 − U4*|
64→128Likelihood0.286 ± 0.0120.0616 ± 0.00730.202 ± 0.0180.0067 ± 0.0033
64→128+ kinetic0.038 ± 0.0040.0088 ± 0.00020.030 ± 0.0000.0014 ± 0.0008
64→128+ kinetic + marginal0.050 ± 0.0060.0055 ± 0.00270.031 ± 0.0070.0012 ± 0.0001
64→256Likelihood0.499 ± 0.0440.1025 ± 0.01090.347 ± 0.0290.0073 ± 0.0034
64→256+ kinetic0.129 ± 0.0100.0224 ± 0.00130.079 ± 0.0070.0022 ± 0.0008
64→256+ kinetic + marginal0.049 ± 0.0030.0024 ± 0.00220.020 ± 0.0030.0012 ± 0.0006
64→512Likelihood0.641 ± 0.0360.1286 ± 0.01330.465 ± 0.0370.0064 ± 0.0034
64→512+ kinetic0.220 ± 0.0170.0391 ± 0.00280.154 ± 0.0140.0033 ± 0.0006
64→512+ kinetic + marginal0.041 ± 0.0030.0050 ± 0.00170.024 ± 0.0030.0015 ± 0.0011

Q2, flow matching performance. At a fixed size, the flow-matching model has a global receptive field and no constraints, so we expect it to perform best in this scenario. Our approach instead focuses on interpretability and its ability to enable extrapolation. Future work may find ways to augment our approach (e.g., larger patches or additional layers per level) to improve performance even at fixed size.

Q3, which level to repeat. The later levels should, in principle, be closer to the fixed point so would be better choices for repeating. In practice, we find repeating any of the similar-looking levels tends to perform reasonably with the penultimate level performing equivalently or better on most metrics (Table B.7).

Repeated level (64→128)TV32|Δ⟨|m|⟩|mean |C/C_ref − 1||U4 − U4*|
Level 4 (Table 1)0.051 ± 0.0040.0056 ± 0.00260.031 ± 0.0060.0014 ± 0.0000
Level 30.047 ± 0.0070.0085 ± 0.00300.037 ± 0.0070.0026 ± 0.0015
Level 20.045 ± 0.0070.0091 ± 0.00310.037 ± 0.0070.0036 ± 0.0017

Thank you for your rebuttal comments and the revised version of the manuscript.

The additional results you added concerning extrapolations away from the self-similar point are a very interesting extension of your original work. However, they seem to be specific to the Ising model and the fundamental restriction to self-similarity remains, as you mention in your answer to Q1 and the new paragraph on generality.

The rewritten sections are easier to read now and the new paragraph on generality clarifies what is specific to the systems studied in this paper. The new ablations also add to the clarity of the paper.

Overall, my concerns were mostly addressed and I will increase my score.

Thank you again for your careful feedback and for helping us make the paper clearer and more impactful!

Reviewer_iCqFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The authors present a method to learn normalizing flows that sample configurations from lattice systems in two-dimensional space. The method addresses the problem: hierarchical flows for lattice systems are non-identifiable and reparameterizations at a given layer are undone by later layers which leaves the likelihood unchanged. As a result, the coarse-graining maps are not unique and cannot be compared or reused. The authors instead, propose a hierarchical normalizing flow method which makes the solution interpretable and reusable across multiple system sizes.

Key components of the model pipeline include a sitewise Gaussianization step followed, at each level, by a “disentangler” acting on offset 2×2 cells and a “decimator” acting on aligned cells (both building on NeuralRG/RG-Flow). The coarse graining is enabled by the fixed Haar split which keeps the cell average as the coarse value and sends information to the latent space. The continuous normalising flow levels are made identifiable by two regularisation terms.

The theorems in section 3 characterise which reparameterizations leave the likelihood unchanged, demonstrate that the coarse value is sufficient and unique up to some monotone map and highlight, under the assumed conditions, there is at most one optimum.

The authors demonstrate the scalability of the approach by training on a 64x64 lattice and then extrapolating for a 128x128 lattice by repeating one level and demonstrate further scalability to 1024x1024 systems. Additionally, they compare the learned levels across a temperature ladder in two regimes: ordered and disordered. Extrapolation by repeating a level relies on self-similarity and is shown only at the critical point.

Critical strengths

The central idea, using identifiable levels and training a model at one scale that generalizes to other scales by adding a layer, is interesting and well motivated.

The suitability of the method for this task is well supported by experiments using the 2D Ising model across multiple scales.

The paper is clearly placed within the literature, building on NeuralRG and RG-Flow.

Critical weaknesses
  1. The paper is difficult to follow, particularly section 3. The theorems are wordy, and the amount of new terminology makes it hard to keep track of earlier definitions. The text would be easier to follow if it gave the key equations (brought forward from Appendix D) and some short comments that link the theorems.

  2. The temperature ladder runs for the high-temperature and low-temperature systems do not have sample quality numbers. The visual analysis gives some indication but a table of the same metrics given in Table 1 would be beneficial. There is no ablation of the regularizers. It is unclear how much the regularizers contribute to the agreement between seeds and the extrapolation.

Questions for authors
  1. How does this method generalize to other physical systems? Are there properties of the 2D Ising model that make this method favourable but limit it’s application elsewhere (e.g. systems off criticality, systems that are not self-similar, three dimensional systems)? Are there systems where you would expect the method to fail and why? (useful to include in the limitations section).

  2. The paper states: "trained with regularizers and a normalization that fix the reparameterization freedom". What happens to the results and to extrapolation when the regularizers are removed? How far are the canonical normalizations from the identity i.e. how much normalization is imposed after training versus through the penalty M (noting comment in Appendix E)?

  3. The baseline is a flow-matching model trained at a single lattice size. Are there baselines other than FM naïve that may relate better to the work?

4 comments

Thank you for the careful review! Several of your points overlap with those of the other reviewers and are addressed by our most recent revision. We are happy to incorporate any additional discussion here into the final draft.

W1, Section 3. Section 2 now defines every object Section 3 uses. In Section 3 the setup defines a hierarchy, the coarse value and the latent variables before the first theorem, and each theorem is followed by a short comment on what it settles. Following your suggestion, we replaced verbal descriptions with explicit expressions: the action of a reparameterization (Equation 4), the conditional independence in Theorem 2, the bound of the kinetic energy by the mean squared displacement (Equation 5) and the two inequalities of the displacement condition (Equation 6). The section now closes with a paragraph that says which hypotheses are assumed and not derived, that a trained model fits only approximately, and that the agreement of independently trained models is the empirical test. Appendix E summarizes the results and their limitations.

W2, sample quality on the temperature ladder. The table below gives the Table 1 metrics at the training size for the critical run and the eight ladder runs at L=128 (two seeds each, against held-out MCMC data at the same coupling).

Run (L=128)TV32|Δ⟨|m|⟩|mean |C/C_ref − 1||ΔU4|
ξ=1, disordered0.013 ± 0.0010.0000 ± 0.00000.138 ± 0.000 †0.0462 ± 0.0022 ‡
ξ=8, disordered0.025 ± 0.0000.0014 ± 0.00010.175 ± 0.002 †0.0524 ± 0.0010 ‡
ξ=32, disordered0.027 ± 0.0000.0044 ± 0.00030.089 ± 0.0000.0149 ± 0.0005
ξ=128, disordered0.054 ± 0.0030.0098 ± 0.00260.054 ± 0.0070.0021 ± 0.0013
β_c (Table 1)0.057 ± 0.0010.0046 ± 0.00100.032 ± 0.0020.0021 ± 0.0007
ξ=128, ordered0.079 ± 0.0060.0111 ± 0.00110.043 ± 0.0030.0004 ± 0.0002
ξ=32, ordered0.076 ± 0.0010.0097 ± 0.00010.029 ± 0.0000.0009 ± 0.0001
ξ=8, ordered0.078 ± 0.0060.0062 ± 0.00060.015 ± 0.0010.0006 ± 0.0001
ξ=1, ordered0.014 ± 0.0010.0038 ± 0.00040.006 ± 0.0000.0008 ± 0.0001

The β_c row is the model trained at the critical point at L=128 from Table 1, where the Binder cumulant is compared with its exact critical value. Sample quality on the ladder is at a similar level. The marked entries on the disordered side at ξ=1 and 8 need care. (†) The correlation function decays to the noise level well before half the lattice size, so the correlation error is taken only over the radii where the reference is resolved (10 and 56 sites). (‡) The magnetization is nearly Gaussian there, so U4 is close to zero and hard to estimate. With 4,000 held-out configurations the reference value has a standard error of about 0.03, so the differences of about 0.05 are about two standard errors.

In addition, new experiments test extrapolation away from criticality at ξ=32 on both sides of the transition (Tables B.8 and B.9, also shown in our response to Reviewer bpfM, W1).

W2 and Q2, regularizers. The likelihood alone does not determine the block maps. Following our theoretical work, the kinetic energy makes the maps unique but not comparable across levels, and the marginal term makes them comparable even across different models. This combination is what enables the interpretable extrapolation that we perform.

In a new set of experiments, we build the objective up one term at a time at L=64 and extend each model by repeating level 4 (Table B.10, shown in our response to Reviewer bpfM, Q2). With the likelihood alone the extended model fails at L=128, with the kinetic energy added it holds at L=128 but not at L=256, and with the marginal term it holds through L=512. Figure B.4 shows how well two independently trained seeds agree under each objective. At each level we measure the mean norm of the difference between the two seeds' decimators, as a percentage of their mean displacement norm. Across the levels this difference is 4 to 19% with the likelihood alone, 3 to 16% with the kinetic energy added, and 2 to 4% with the full objective.

Q2, how much the canonical normalization does. Within one model the normalization of a field cancels between the block map that produces the field and the block map that consumes it, so it changes no sample of a trained model. It enters only when block maps are compared or when blocks are recombined in an extended model. To quantify how far the normalization applied after training is from the identity, we measure, for each field of hidden variables, the root-mean-square of Q(x) − x over the marginal of that field, where Q is the normalization. This equals the 2-Wasserstein distance between the marginal before normalization and the standard normal. The table lists the fields from fine to coarse for the three objectives of Table B.10 (L=64, β_c, two seeds), where h_ℓ is the field between the disentangler and the decimator of level ℓ, and x_ℓ is the coarse field entering level ℓ.

FieldLikelihood+ kinetic+ kinetic + marginal
h_00.156 ± 0.0090.152 ± 0.0110.127 ± 0.006
x_10.130 ± 0.0020.129 ± 0.0010.102 ± 0.000
h_10.114 ± 0.0200.109 ± 0.0170.071 ± 0.005
x_20.142 ± 0.0040.138 ± 0.0030.092 ± 0.003
h_20.096 ± 0.0000.099 ± 0.0010.045 ± 0.002
x_30.166 ± 0.0040.150 ± 0.0040.075 ± 0.002
h_30.095 ± 0.0010.098 ± 0.0020.029 ± 0.000
x_40.194 ± 0.0170.146 ± 0.0050.051 ± 0.001
h_40.103 ± 0.0000.097 ± 0.0020.019 ± 0.001
x_50.394 ± 0.0080.127 ± 0.0030.031 ± 0.001

With the marginal term, the normalization is close to the identity beyond the finest levels (e.g., h_0, x_1), so most of the normalization is imposed during training. Without the marginal term, a much larger correction is needed. The largest correction is at the finest level, which likely has to do with the particular choice of continuous relaxation in this example.

Q1, other systems. In its current form, we expect to be able to train our model on any system with local interactions and translation symmetry while incorporating any additional symmetries into the equivariant block design. For example, the same model used on Ising can be applied to φ⁴, which recovers block maps that agree with the Ising ones beyond the finest non-universal levels (Appendix B.7). To achieve extrapolation, however, the system must exhibit some form of low-dimensional structure in scaling behavior, such as self-similarity, and we expect extrapolation to fail without it.

Off criticality, the block maps change with scale, so a new level cannot be a copy. However, for Ising, the learned block maps collapse onto a curve, so we can graft blocks from a model trained at a shorter effective correlation length to fill in the missing pieces (Tables B.8 and B.9, shown in our response to Reviewer bpfM, W1). For a system that is not exactly self-similar, Appendix B.8 applies the model to a passive scalar in forced 2D turbulence, where we see imperfect but non-trivial extrapolation (Table B.13, shown in our response to Reviewer Z5H3). We have not yet tested three-dimensional systems because of the computational expense and the lack of exact benchmark systems with known solutions, but this is, in theory, a straightforward generalization of our current work. The locality built into our model is both an advantage for interpretability and a limitation in expressivity, especially when there are highly non-local interactions. To address these points in the paper, we include a new "Generality" paragraph and an expanded "Limitations" paragraph in Section 5.

Q3, baselines. Because we are in the unusual problem setting of attempting to do interpretable extrapolation, not many other approaches exist. Super-resolution methods, for example, require data at the higher resolution to successfully generalize to larger sizes. If we assume there are no longer-range correlations beyond the current system size (finite correlation length), our flow-matching baseline is a simple and natural choice when trying to extrapolate convolutional architectures without additional high-resolution data. It also matches our setting: our model is trained at a single lattice size as well, so both models see data only at the training size. The naive extrapolation, of course, breaks down at or near non-trivial fixed points (like the critical point), which is where our method has an advantage.

Thank you for the comments and the revised paper.

The new ablation, adding the regularizers one at a time, directly addresses my W2 concern. The sample-quality table for the temperature ladder and the measurement of how far the canonical normalizations are from the identity also answer my questions, and the new Generality and Limitations paragraphs clarify which parts of the method depend on self-similarity.

The clarity of Section 3 has slightly improved, but the section is still dense and I have kept the presentation score.

My concerns about soundness have been addressed, and I have increased the soundness score.

Thank you for your feedback and for the updated assessment! We agree that Section 3 could use a bit more work, so we have revised its language and flow again while leaving the theorem statements, equations, and definitions unchanged.

Specifically:

  • The opening paragraph now poses the question (are the learned block maps uniquely identifiable after training?) and answers it in three steps that follow the theorems in order.
  • The text around Theorem 2 now says why the coarse value is treated first, and what the theorem means in plainer terms.
  • Before Theorem 3, we now specify what each regularizer does, and the displacement condition is summarized in one plain sentence before Equation 6.

Theory sections are always a careful balance between technical accuracy, plain-language interpretation, and length. We welcome any further suggestions on improving the clarity of this section.

Comments on the paper

1 comment

We thank both reviewers for their helpful comments! We are excited that the reviewers appreciate the novelty in our approach, and we made changes throughout the paper to improve clarity and consistency. In particular:

  1. Section 2: the disentangler and decimator are defined as block maps, the text states that both are learned and determined by the objective, and the objective paragraph states why two regularizers are needed and what each selects.
  2. Section 3: every term is defined before the theorem that uses it. The reparameterizations, the bound on the kinetic energy and the displacement condition are now written as explicit expressions (Equations 4 to 6), and Theorem 2 states its conditional independence as a formula. A closing paragraph says which assumptions are not derived and what they mean for a trained model. Appendix E summarizes the results and their limitations.
  3. Section 5: new "Generality" paragraph and expanded "Limitations".
  4. New experiments in Appendix B.5 and B.8: testing the choice of the repeated level (Table B.7), extrapolation away from criticality (Tables B.8 and B.9), the objective built up one term at a time (Table B.10), scaling exponents derived from extrapolated samples (Table B.11), and a preview of the method on a complex dataset for scalar advection in turbulence (Table B.13).
  5. We have also revised the theoretical proofs in the appendix for clarity.

We welcome any additional suggestions, especially regarding presentation if more clarification is needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.