acceptodds
Under review as a conference paper at ICLR 2027

Architecture-Dependent Information Floors for Learned Boltzmann Sampling

Abstract

Learned Boltzmann samplers often fit a tractable proposal and correct the remaining mismatch to a target by importance reweighting. This raises a basic question: before training, what sampling efficiency is attainable by a given model class? We study the architecture-dependent information floor for classes specified by conditional-independence structure. For factorised, finite-context autoregressive, and graph-Markov families, admits exact target-only expressions in terms of total correlation, conditional mutual information, and the entropy gap to an -projection. Combining these identities with asymptotic importance-sampling thresholds yields class-level lower bounds on the number of proposal draws required for reweighting. Under a uniform interface-information condition, the floor is extensive for bounded-size partitions and therefore induces exponential scaling with system size. We verify the identities against numerical optimisation and transfer-matrix calculations, compare the two-dimensional Ising measurements with Onsager's exact solution, and evaluate molecular examples in regimes where the floors are small. Finally, a pair-marginal certificate provides a data-driven lower bound without fitting a proposal. The resulting quantities separate optimisation error from structural mismatch and can be evaluated before training.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Boltzmann sampling, importance sampling, information theory, structured generative models, normalizing flows, total correlation, molecular simulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kmLdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the question what forward KL is achievable by different model classes (factorised, autoregressive and graph-Markov classes). The authors answer this question by using conducting a theoretical analysis. The authors then proceed to connect the results to the number of samples required for Boltzmann generators to reweight to the target density based on a result presented by Chatterjee and Diaconis.

Critical strengths

I enjoyed reading the beginning of the paper. Moreover, the paper is nicely structured until Section 3 and includes short proofs for their theorems.

Critical weaknesses
  • Section 3 opens with a theorem without any context which suddenly mentions Potts chains which are neither explained nor introduced. Moreover, total correlation was only introduced in the context of fully factorized classes. Its not clear what the author intends to show with it neither does the subsequent paragraph on Ising chains (which is again neither introduced nor explained).
  • Section 4 is just a theoretical result with an additional sentence. It is unclear to me what exactly it says and what the implications are. A bit more context would be helpful to make the paper more readable. Q: Does the result say that using annealed importance sampling requires less samples to do reweighting over the whole path compared to directly reweighting. If so, would this mean that the forward KL summed over the path is less then the forward KL from the initial proposal to the target?
  • The theorems in this work are textbook identities presented as new contributions. None of the prior work is cited properly.
  • The experiments are very small scale. It would be good for the authors to demonstrate the usefulness of their theory on some real-world applications.

After section 2, the paper appears to me as a collection of theorems without much context. Its difficult to read and related works do not seem to be credited appropriately. The paper needs significant work to be accepted at a venue such as ICLR.

Questions for authors
  • Which modern Boltzmann-generator papers lie within the model classes discussed in this work?

See weaknesses for further questions

1 comment
AuthorsOpenReview ↗

Response to Reviewer kmLd

Thank you for the detailed and constructive reading. We have revised the manuscript to address each of your points.

1. Context and definitions in Section 3 (Potts/Ising chains and total correlation). Total correlation TCS(μ)\mathrm{TC}_{\mathcal{S}}(\mu) is now formally defined for any partition in Proposition 2(a) before Section 3 uses it. Section 3 opens with the shared mechanism across all three conditional-independence classes (Equation 1: the information each site carries about past coordinates omitted from its parent set), followed by one extensivity theorem for all classes (Theorem 4). When introducing the 1D mm-state Potts chain (whose m=2m=2 case is the zero-field Ising chain), we write out its distribution explicitly and explain why Theorem 4 is an equality there: single-site marginals are uniform and the chain is Markov, so TC(μL)=(L−1)Iμ(x1;x2)\mathrm{TC}(\mu_L) = (L-1)I_\mu(x_1; x_2), the number of bonds times the per-bond mutual information.

2. Section 4 and your question on Annealed Importance Sampling (AIS). Yes, your intuition is exact, and Section 4 (Proposition 8) now spells out this trade-off. Along a KK-stage geometric path with exact stage-wise draws, the summed forward KL across stages is J(μ,q)/(2K)+O(K−2)J(\mu, q)/(2K) + O(K^{-2}), which can be made far smaller than the one-shot D(μ∥q)D(\mu \Vert q) by adding stages; accordingly, AIS avoids the exponential sample-size growth of direct reweighting. The work does not disappear, however: it moves into the stage transitions. Proposition 8 shows that total transition work W=KnW = Kn satisfies W≥J(μ,q)/ϵ2≥ΦQ(μ)/ϵ2W \ge J(\mu, q)/\epsilon^2 \ge \Phi_{\mathcal{Q}}(\mu)/\epsilon^2 and approaches J/ϵ2J/\epsilon^2 once K≫JK \gg J. Thus an extensive floor ΦQ=Ω(N)\Phi_{\mathcal{Q}} = \Omega(N) sets one-shot reweighting at eΩ(N)e^{\Omega(N)} draws while making annealed work linear in ΦQ\Phi_{\mathcal{Q}}. On an N=512N = 512 Ising chain, one-shot reweighting from the product proposal needs eΦ≈1072.7e^\Phi \approx 10^{72.7} draws, whereas AIS needs 1.9×1051.9 \times 10^5 transitions (Figure 6).

3. Credit to prior work. We agree and have revised Sections 2 and 7 accordingly. Proposition 2 presents the factorised, autoregressive, and graph-Markov identities as classical (Watanabe, 1960; Csiszár, 1975; Chow & Liu, 1968; Lauritzen, 1996). Section 7 credits Schneidman et al. (2003) for connected information, Erb & Ay (2004) and Lau & Grassberger (2013) for Ising multi-information, Neal (2001), Crooks (2007), and Grosse et al. (2013) for AIS variance and thermodynamic length, and Chatterjee & Diaconis (2018) and Sanz-Alonso (2018) for importance-sampling complexity, alongside lattice-scaling studies. Our new results (Theorem 5, Corollary 6, Proposition 7, and Proposition 10 on local receptive fields and diffusion samplers) are listed separately in Section 1.

4. Larger scale and real-world applications. The revision expands the experiments across three domains:

  • 2D lattice fields and volume transfer: Trained local coupling flows evaluated without retraining up to 64×6464 \times 64 (4,0964{,}096 sites) on massive free fields and 32×3232 \times 32 on strongly non-Gaussian double-well ϕ4\phi^4 fields (Figure 2(c), Tables 3–4);
  • Critical 2D Ising model: Validated via Wolff cluster sampling and thermodynamic integration against Kaufman's exact finite-torus solution and tracked to 512×512=262,144512 \times 512 = 262{,}144 spins against Onsager's limit (Figure 3, Tables 8–9);
  • Molecular benchmarks: Equilibrium replica-exchange simulations of alanine peptides Ace-(Ala)n_n-NMe (n=1,3,5n = 1, 3, 5; 1.3 μs1.3\ \mu\text{s} at 300 K300\text{ K}) with trained continuous von Mises mixture-density networks of every class (Figure 4(b,c), Tables 5–6), plus an nn-alkane chain (Figure 7).

5. Which modern Boltzmann generators lie in these classes (Table 1). Particle-wise maps (Gabrié et al., 2022) lie in Qprod\mathcal Q_{\mathrm{prod}}; VANs with reach shorter than the lattice (Wu et al., 2019) and hierarchical AR nets (Białas et al., 2022) lie in QAR/QG\mathcal Q_{\mathrm{AR}}/\mathcal Q_G; convolutional couplings (Albergo et al., 2021; Kanwar et al., 2020), fixed-neighbour materials GNNs (Schebek et al., 2026), and convolutional lattice diffusion samplers (Wang et al., 2024; Tan et al., 2026) lie in Qloc(R)\mathcal Q_{\mathrm{loc}}(R) (R=KrR = Kr) and obey Theorem 5; while full-context flow-matching/transformer BGs (TBG, SBG, Prose; Klein & Noé, 2024; Tan et al., 2025a,b) and global-score diffusion samplers (PIS, DDS, iDEM) are universal, with zero structural floor.

We hope these revisions address all of your questions and are happy to discuss further.

Reviewer_SH7XOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the mismatch that remains unavoidable when a proposal distribution is restricted to a particular dependence structure. The authors characterize the minimum forward KL divergence for block-factorized, finite-context autoregressive, and graph-Markov models, and connect these information floors to the number of samples needed for importance sampling. Under certain assumptions, they show that bounded-size blocks incur a floor that grows linearly with system size. They also discuss annealed reweighting and lower bounds based on pair marginals. Experiments on Gaussian targets, Ising models, and molecular torsions examine the predicted floors and how they change with model structure and system size.

Critical strengths
  1. The paper distinguishes mismatch caused by restrictions on the dependencies a model can represent from mismatch caused by imperfect training, and quantifies the former for three model classes.

  2. The Ising experiments illustrate how allowing additional dependencies reduces the floor.

Critical weaknesses
  1. I do not see what these results tell us about current Boltzmann samplers. The authors claim that their analysis complements recent Boltzmann generators (lines: 396-399), but do not show that the cited methods impose the conditional-independence restrictions considered here. Which existing samplers satisfy these restrictions, and when would these restrictions be a useful modeling choice?

  2. The paper has limited novelty. I think much of the theory here comes from putting together results that are already known. The authors connect the KL identities to sampling requirements using [3]. For contribution 4, [1] already relates AIS log-weight bias to consecutive KL divergences under perfect transitions and derives the Fisher-information limit. The connection to thermodynamic length and constant-speed scheduling is also known from [4]. I have the same concern about the pair-marginal bound: [2] already shows that the information captured by pairwise marginals lower-bounds total correlation. I do not see what choosing these marginals through a graph adds to that result. I understand the motivation for bringing these results together, but I do not think the paper shows that this amounts to a substantial new theoretical contribution.

  3. Poor presentation of the methods. Sections 2-5 read as a sequence of results with little explanation of how they fit together. Section 2 introduces three model classes, but Section 3 studies scaling only for block-factorized proposals (lines: 149-154). Remark 7 acknowledges this restriction without explaining its consequences for the other classes. Section 4 then moves to annealing without explaining how it relates to these structural restrictions.

[1] Roger B. Grosse, Chris J. Maddison, and Ruslan Salakhutdinov. Annealing between distributions by averaging moments. Advances in Neural Information Processing Systems 26, 2013.

[2] Elad Schneidman, Susanne Still, Michael J. Berry II, and William Bialek. Network information and connected correlations. 2003.

[3] Sourav Chatterjee and Persi Diaconis. The sample size required in importance sampling. 2018 (preprint 2015).

[4] Gavin E. Crooks. Measuring thermodynamic length. Physical Review Letters 99, 100602, 2007. See Equations 4-9.

Questions for authors
  1. What is the polynomial cost in Theorem 8 measuring exactly, and what does it guarantee (lines: 188-193; proof: 560-565)? Does the application of [3] assume independent samples from each intermediate target are already available, or is the cost of obtaining them included? Is it for each intermediate calculation or for the final estimate?

  2. In Figure 3 (left), the floor drops to numerical zero at context length c=1 for every correlation length, including ξ=4. Could the authors clarify how this supports the caption’s statement that the floor decreases once the context reaches the correlation scale (lines: 230–233)? If the targets use covariance Σi,j=e−∣i−j∣/ξ\Sigma_{i,j}=e^{-|i-j|/\xi}, wouldn’t they already be first-order Markov, making c=1 sufficient regardless of ξ?

  3. How sensitive are the conclusions about small molecular floors to the chosen binning resolution (Figure 3, right; lines: 230–244)? Can the authors distinguish the additional dependence revealed by finer binning from finite-sample estimation bias, particularly for aspirin, whose estimated floor continues to increase?

5 comments
AuthorsOpenReview ↗

Response to Reviewer SH7X (Part 1/2: Weaknesses)

Thank you for the thorough critique, which guided several major additions in our revision.

W1. Implications for modern Boltzmann samplers and when structural restrictions are useful. Table 1 now maps current samplers onto our model classes. In large-scale lattice and materials applications, the dominant structural restriction is locality: convolutional and fixed-neighbour message-passing coupling layers restrict each output to depend on latent noise within a finite receptive field RR. Our new Theorem 5 bounds this class, Qloc(R)\mathcal Q_{\mathrm{loc}}(R): D(μ∥q)D(\mu \Vert q) is at least the total correlation of blocks separated by more than 2R2R. This floor is extensive, tight in the exponent on Gaussian chains (Proposition 7), and gives explicit scaling laws (R≳(ξ/4)log⁡NR \gtrsim (\xi/4)\log N for finite correlation length ξ\xi, and R≳L4/5R \gtrsim L^{4/5} up to logarithms at 2D Ising criticality; Corollary 6).

The same framework covers diffusion samplers with convolutional score networks, which lie in Qloc(Kr)\mathcal Q_{\mathrm{loc}}(Kr) after KK integration steps of a radius-rr network (Lemma 9). On Gaussian chains this KrKr receptive-field count is exact (new Proposition 10: an optimal KK-step radius-rr linear-score sampler achieves the exact floor of a single convolutional map of radius KrKr), quantifying the structural penalty of step distillation. By contrast, full-context transformers and global flow-matching or diffusion samplers (TBG, SBG, Prose, PIS, DDS, iDEM) impose neither restriction; we state explicitly that their structural floors are zero. On equilibrium alanine peptides, trained continuous neural density models across classes obey these bounds on unseen runs (Section 6, Table 6). Finally, regarding why practitioners use local or factorised classes despite these floors: locality is what lets a sampler trained on a small system be applied to a larger volume without retraining at O(N)O(N) evaluation cost (Abbott et al., 2023; Schebek et al., 2026). Theorem 5 and Corollary 6 quantify the divergence cost of that choice and how fast RR (or KrKr) must grow with NN to avoid exponential ESS collapse.

W2. Novelty and attribution of classical results. We agree that the conditional-independence identities in Section 2 are classical, and we now present them as Proposition 2 (classical floors) crediting Watanabe (1960), Csiszár (1975), Chow & Liu (1968), Lauritzen (1996), and Minka (2005). Our specific contributions are separated clearly:

  1. Finite receptive-field floor (Theorem 5), with exponential tightness (Proposition 7), receptive-field scaling laws (Corollary 6), and exact step-radius equivalence for local diffusion samplers (Proposition 10), turning heuristic volume-scaling arguments in lattice and materials sampling into rigorous bounds;
  2. Unified extensivity theorem (Theorem 4) covering factorised, finite-context AR, and bounded-range graph-Markov classes via one parent-set chain-rule argument;
  3. Architecture-to-annealing work bound (Proposition 8) via J(μ,q)≥ΦQ(μ)J(\mu, q) \ge \Phi_{\mathcal{Q}}(\mu);
  4. Autocorrelation-controlled marginal certificates and experiments across lattice ϕ4\phi^4, critical Ising up to 512×512512 \times 512, and replica-exchange alanine peptides.

On your four references: [1] The AIS bias identity and Fisher/Jeffreys limit are credited to Grosse et al. (2013); our addition is linking total work WW to the architectural floor ΦQ(μ)\Phi_{\mathcal{Q}}(\mu). [4] The constant-speed schedule bound (Λ2≤J\Lambda^2 \le J) is credited to Crooks (2007). [2] We connect our spanning-tree bound to Schneidman et al. (2003), who decompose multi-information into non-negative connected terms; our tree sum lower-bounds their second-order connected information while remaining computable from O(N2)O(N^2) 2D histograms and extending to (2R+1)(2R+1)-separated sublattices for local flows. [3] Chatterjee & Diaconis (2018, Thm. 1.1) is used alongside the non-asymptotic bound ESS/n≤e−ΦQ(μ)\mathrm{ESS}/n \le e^{-\Phi_{\mathcal{Q}}(\mu)}.

W3. Presentation and cohesion. Sections 2–4 now read as one continuous argument: Theorem 4 proves extensivity for all conditional-independence classes (replacing Remark 7), Theorem 5 to Proposition 10 treat locality and diffusion steps, and Section 4 connects ΦQ\Phi_{\mathcal{Q}} to annealed sampling work, stating the practical implications after each result.

AuthorsOpenReview ↗

Response to Reviewer SH7X (Part 2/2: Questions)

Q1. What the cost WW in the annealing analysis measures. Proposition 8 now defines WW explicitly: W=KnW = Kn counts the total number of transitions (target-density evaluations) summed over all KK stages required to estimate ZZ to relative accuracy ϵ\epsilon, rather than a per-stage cost. Following Neal (2001) and Grosse et al. (2013), parts (i)–(ii) assume exact independent draws from each intermediate distribution πλℓ−1\pi_{\lambda_{\ell-1}} ("perfect transitions"). MCMC mixing cost is not hidden: as stated in part (iii), if each exact draw is replaced by τ\tau steps of an MCMC kernel, total target evaluations scale as τW\tau W. Under exact transitions, Var(log⁡w)=J/K+O(K−2)\mathrm{Var}(\log w) = J/K + O(K^{-2}) and W=K(eJ/K−1)/ϵ2≥J(μ,q)/ϵ2≥ΦQ(μ)/ϵ2W = K(e^{J/K}-1)/\epsilon^2 \ge J(\mu, q)/\epsilon^2 \ge \Phi_{\mathcal{Q}}(\mu)/\epsilon^2, approaching J/ϵ2J/\epsilon^2 once K≫JK \gg J. We verified this on Ising chains where every intermediate distribution can be sampled exactly: at N=128,K=16N=128, K=16, measured Var(log⁡w)\mathrm{Var}(\log w) is 6.206.20 vs. 6.196.19 predicted, and minimal work across K≤1024K \le 1024 lies within 1.006–1.22×J/ϵ21.006\text{–}1.22 \times J/\epsilon^2 for N=16–512N = 16\text{–}512 (Figure 6).

Q2. Original Figure 3 (left), now Figure 4(a). Thank you for catching this, you are right. Because a 1D Gaussian process with exponential covariance e−∣i−j∣/ξe^{-|i-j|/\xi} is first-order Markov, its exact autoregressive floor is zero for all c≥1c \ge 1 regardless of ξ\xi. We have replaced this panel with lattice restrictions of Matérn-3/23/2 Gaussian targets, which are not finite-order Markov. In Figure 4(a), the AR(cc) floor now decays geometrically in cc at a rate set by ξ\xi, requiring context c=2c = 2 at ξ=0.5\xi = 0.5 up to c=5c = 5 at ξ=4\xi = 4 to drop below 10−310^{-3} nats.

Q3. Binning resolution, autocorrelation, and finite-sample estimator bias. Your question led us to audit finite-sample bias in molecular trajectories directly. In short ab initio MD17 trajectories (such as aspirin), slow torsions have integrated autocorrelation times of 1.4×104–3.2×1041.4\times 10^4\text{–}3.2\times 10^4 frames, leaving effective sample sizes of ∼10\sim 10 where physical coupling cannot be separated cleanly from plug-in bias at fine binnings. To separate the two, we replaced the molecular benchmark as follows:

  1. Equilibrium replica-exchange data: We replaced MD17 with standard Boltzmann-generator benchmarks, Ace-(Ala)n_n-NMe (n=1,3,5n = 1, 3, 5 at 300 K300\text{ K}, amber14 + OBC2), sampled via temperature replica exchange across independent runs (1.3 μs1.3\ \mu\text{s} total at 300 K300\text{ K}), cutting the slowest backbone autocorrelation time from 29 ns29\text{ ns} (unbiased MD) to 72 ps72\text{ ps}.
  2. Circular-shift null subtraction: Every reported estimate subtracts the mean of a shift null: the same estimator evaluated after circularly shifting each torsion's time series by an independent random lag, preserving marginal distributions and autocorrelation while removing cross-coordinate dependence.
  3. Convergence and resolution checks: At b=8b = 8 bins, the shift null is ≤0.5\le 0.5% of the raw certificate; estimates change by ≤4.2\le 4.2% when simulated time is halved (Figure 8); and the pair-marginal certificate rises monotonically across nested refinements b∈{2,4,8,16,32}b \in \lbrace 2, 4, 8, 16, 32\rbrace with block-jackknife standard errors (Figure 4(b), Table 5).
  4. Out-of-sample neural validation: On disjoint test runs, fitted tabular ML models match the certified class differences to within 0.0100.010 nats (Figure 4(c)), while continuous von Mises mixture-density networks lose 1.646±0.0121.646 \pm 0.012 nats (torsion-factorised) and 0.755±0.0040.755 \pm 0.004 nats (residue-factorised), above the certified floors (1.0421.042 and 0.2970.297 nats; Table 6), with the trained product models within 0.0040.004 nats of the exact class optimum.

We hope these theoretical and empirical additions address your concerns, and we welcome any further questions during the discussion period.

Thank you for the detailed response and the substantial improvements to the paper. Several of my original concerns have been addressed. I have a couple more questions and concerns:

  1. Proposition 8 relates the work to J, but only lower-bounds J by the architectural floor. I do not see how this establishes that annealing makes the work linear in the floor, as stated in the abstract and Section 4. Is an additional assumption needed for J to grow at most linearly with the architectural floor?

  2. The revision introduces a new model class and several substantial theoretical results on locality and diffusion. I appreciate that these changes respond to my concerns. However, the revised abstract now presents the new locality result as the paper’s main result (lines: 14–25). This result and the analysis were not in the original submission. These are substantial additions beyond improving readability and reads like a new paper.

  3. Lastly, I am also still not entirely convinced about the strength and practical significance of the revised contribution. The locality results and experiments on lattices help me see the relevance to lattice sampling. However, I am less convinced about protein sampling. The peptide experiments show the cost of using factorized or limited-context torsion models, but I still do not see when these would be useful choices for a practical protein Boltzmann generator.

Though given these improvements, I will bump my score to a weak reject.

Reviewer_6XhNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors derive information floors for several Boltzmann Generator architectures to characterize their sampling efficiency. This enables derivation of lower bounds on the number of proposal draws typically required for reweighting with each architecture. A key result is that this number scales exponentially with system size. The authors then validate their theoretical results through numerical experiments on a Gaussian distribution, the Ising model, and molecular systems.

Critical strengths

The paper provides novel theoretical insights into the design of Boltzmann Generators and the architectural choices that affect their resampling efficiency. I believe these insights are a valuable contribution to the field of sampling and can help guide future architectural design.

Critical weaknesses

While the paper is technically sound, I believe it would benefit from more detail on the technical and implementation aspects of the experiments, as well as more complete proofs for some of the theorems.

Questions for authors

I ask the authors to provide expanded proofs for Theorems 3, 4, and 8 (polynomial scaling).

I also ask that they provide more detailed descriptions of the experimental setup and release the accompanying code.

Finally, I need some more clarity on section 5 - how is the corollary applied on simulation data and what are these pair marginals?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6XhN

Thank you for the positive assessment and concrete suggestions, all of which have been incorporated into the revised manuscript.

1. Complete proofs (original Theorems 3, 4, and 8). Appendix A now provides full, self-contained proofs for every theoretical statement:

  • Autoregressive floor (now Proposition 2(b), Appendix A.2): proved via the coordinate-wise chain rule, decomposing D(μ∥q)D(\mu \Vert q) into the conditional mutual information sum plus a non-negative expected conditional KL term (identifying the truncated conditionals as the exact minimiser), establishing monotonicity in cc via the chain rule for conditional mutual information, and proving that the floor vanishes iff μ\mu is cc-th order Markov.
  • Graph-Markov floor (now Proposition 2(c), Appendix A.2): proved for all decomposable graphs using junction-tree factorisation over cliques and separators and the Pythagorean identity D(μ∥q)=D(μ∥μG)+D(μG∥q)D(\mu \Vert q) = D(\mu \Vert \mu_G) + D(\mu_G \Vert q) for q∈QGq \in \mathcal Q_G, with Csiszár's (1975) II-projection theorem cited for general graphs.
  • Annealing cost (now Proposition 8, Appendix A.7): derives Var(log⁡w)=J(μ,q)/K+O(K−2)\mathrm{Var}(\log w) = J(\mu, q)/K + O(K^{-2}) and the log-weight bias J/(2K)+O(K−2)J/(2K) + O(K^{-2}) under independent stage draws, leading to the total transition bound W=Kn≥J(μ,q)/ϵ2≥ΦQ(μ)/ϵ2W = Kn \ge J(\mu, q)/\epsilon^2 \ge \Phi_{\mathcal{Q}}(\mu)/\epsilon^2, with all assumptions stated explicitly.

2. Experimental details and reproducibility code. Appendix B now documents every experiment in full, specifying targets, model parameterisations, network architectures, optimiser schedules, batch sizes, evaluation protocols, and simulation parameters (OpenMM amber14/OBC2 setup, Langevin integrator, replica-exchange temperature ladders, swap acceptance rates, autocorrelation times, and block-jackknife error estimation). The supplementary archive contains all scripts and a README mapping each script directly to the corresponding tables and figures.

3. Clarifying Section 5 (pair marginals and applying the certificate to trajectory data). Section 5 has been rewritten to spell out the practical procedure step by step. Given a reference trajectory over NN coordinates, a discretised pair marginal of (xi,xj)(x_i, x_j) is simply their 2D joint histogram on a b×bb \times b grid. From these O(N2)O(N^2) two-dimensional histograms, which require no trained model, we compute plug-in pairwise mutual informations I(xi;xj)I(x_i; x_j), find the maximum-weight spanning tree (Chow–Liu tree) over the NN coordinates, and sum its edge weights. By data processing (monotonicity under binning), coarsening coordinates cannot increase mutual information, so any binned spanning-tree sum gives a certified lower bound on the continuum total correlation TC(μ)\mathrm{TC}(\mu) and thus, via Corollary 3, a pre-training lower bound on KL loss and importance-reweighting cost. Section 5 also details our circular-shift null correction, which removes finite-sample upward bias under trajectory autocorrelation. On the solvable 9-torsion alkane chain, this pair-marginal certificate recovers 97.397.3% of the exact floor (Figure 7(c)).

We hope the expanded proofs, simulation protocols, and revised Section 5 fully address your points, and we are happy to answer any further questions.

I thank the authors for their thorough rebuttal and substantial work during the review process. I find all my concerns in the review addressed by the rebuttal. However, in my opinion, the new receptive field and diffusion results, and the experiments that accompany them change the main contributions of the paper. Given how much has changed, I’m most comfortable basing my assessment on the original contribution and the responses above, so I’ll keep my score at weak accept. I’d be glad for the AC and other reviewers to weigh the new results as they see fit, and I hope the authors keep building on this direction.

AuthorsOpenReview ↗

Response to Reviewer 6XhN's follow-up

Thank you for the careful follow-up, for confirming that all concerns from your review are addressed, and for maintaining your positive assessment.

We agree that the original contribution should stand on its own, and the revision is built so that it does. The expanded proofs (Appendix A), the experimental protocols and code (Appendix B and the supplementary archive), the step-by-step account of how the pair-marginal certificate is applied to simulation data (Section 5), and the equilibrium molecular data with trained models all concern the original floors and certificates. The receptive-field results apply the original block-factorised floor (original Theorem 2) to local architectures. We added them because the other reviews asked which current samplers the classes cover, and none of the conclusions about the original classes depends on them.

Thank you again for your support and for the encouragement to keep building on this direction.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 1/2): Summary of Revision, Theory and Positioning

We thank all reviewers for their careful and constructive feedback. Three main requests emerged across the reviews: (i) connect the information floors to the architectures used in modern scalable samplers, (ii) credit classical identities explicitly and state clearly what is new, and (iii) unify Sections 2–4 into a single coherent argument. We have revised the paper to address all three points and added the experiments summarised in Part 2.

1. A new main result for local architectures (Section 3, Theorem 5). To scale to large lattices and materials, coupling flows use convolutional or message-passing conditioners with a fixed number of layers, so each output depends on the latent noise only within a receptive field of radius RR (Albergo et al., 2021; Kanwar et al., 2020; Abbott et al., 2023; Schebek et al., 2026). We prove that for any such RR-local model, D(μ∥q)D(\mu \Vert q) is at least the total correlation of target blocks separated by distance greater than 2R2R (Figure 1(a)). This floor is extensive whenever the target is correlated beyond 2R2R, tight in the exponent on Gaussian chains (Proposition 7: closed-form two-sided bounds both decaying as e−4R/ξe^{-4R/\xi}, matching up to a factor 2R+12R+1), and yields explicit receptive-field scaling laws: R≳(ξ/4)log⁡NR \gtrsim (\xi/4)\log N away from criticality and R≳L4/5R \gtrsim L^{4/5} (up to logarithms) for the critical 2D Ising model (Corollary 6). While the exponential collapse of effective sample size (ESS) with volume for fixed local flows was previously observed empirically (Finkenrath, 2022; Abbott et al., 2023; Schebek et al., 2026, where ESS drops from 41.741.7% on 216 particles to 0.020.02% on 1,728) and argued heuristically, Theorem 5 gives the first proof together with the exponential rate.

The same bound covers diffusion and flow samplers with local score or velocity networks: KK integration steps of a radius-rr network give a model in Qloc(Kr)\mathcal Q_{\mathrm{loc}}(Kr) (Lemma 9), requiring KrKr to grow at least as (ξ/4)log⁡N(\xi/4)\log N. This count is exact: on a Gaussian chain, the best KK-step sampler with a radius-rr linear score attains the exact optimum of a single convolutional map of radius KrKr (new Proposition 10, verified numerically to 3×10−113\times 10^{-11} for all Kr≤12Kr \le 12), so distilling a local diffusion sampler to fewer steps raises its floor just as shrinking its network radius would.

2. Which samplers lie in which class (Table 1). Table 1 places recent samplers in our classes: particle-wise product maps (Gabrié et al., 2022) are factorised; masked-convolution VANs with reach shorter than the lattice (Wu et al., 2019) and hierarchical AR networks (Białas et al., 2022) lie in the autoregressive/graph-Markov classes; convolutional couplings, fixed-neighbour GNN couplings, and convolutional lattice diffusion samplers (Wang et al., 2024; Tan et al., 2026) are RR-local; and full-context transformers and flow-matching models (TBG, SBG, Prose, TarFlow) as well as global-network diffusion samplers (PIS, DDS, iDEM) are distributionally universal, so their structural floors vanish and their mismatch comes from finite capacity and training error alone.

3. One continuous argument from Section 2 to Section 4. Section 3 now proves extensivity for all three conditional-independence classes at once via a parent-set form of the chain rule (Theorem 4) before treating locality. Section 4 links the structural floor to annealed importance sampling (Proposition 8): along a geometric path with exact stage-wise draws, the total work for relative accuracy ϵ\epsilon satisfies W≥J(μ,q)/ϵ2≥ΦQ(μ)/ϵ2W \ge J(\mu, q)/\epsilon^2 \ge \Phi_{\mathcal{Q}}(\mu)/\epsilon^2 and approaches J/ϵ2J/\epsilon^2 for K≫JK \gg J, turning the one-shot cost eΦe^{\Phi} into work linear in the floor.

4. Credit to prior work and reproducibility. Section 2 presents the conditional-independence identities as classical (Proposition 2: Watanabe, 1960; Csiszár, 1975; Chow & Liu, 1968; Lauritzen, 1996). We credit Schneidman et al. (2003) for connected information, Erb & Ay (2004) for the Ising multi-information density, Grosse et al. (2013), Neal (2001), and Crooks (2007) for annealing variances and thermodynamic length, and Sanz-Alonso (2018) alongside lattice-scaling studies in Section 7. Appendix A proves every result, Appendix B gives full experimental protocols, and the supplementary archive provides scripts reproducing every number and figure.

AuthorsOpenReview ↗

General Response (Part 2/2): Summary of New and Strengthened Experiments

5. New and strengthened experiments (Section 6 & Appendices B–E).

  • Exact tightness of the receptive-field floor (Figure 2(a)): On Gaussian chains with ξ=1,2,4\xi = 1, 2, 4, fitted decay rates in RR are 4.01,2.01,1.024.01, 2.01, 1.02 for our block-TC bound and 4.00,2.01,1.074.00, 2.01, 1.07 for the best RR-local flow, matching the predicted rate 4/ξ4/\xi (Proposition 7).
  • Local diffusion samplers and step-radius equivalence (Table 7): On the ξ=4\xi = 4 chain, optimising a KK-step radius-rr linear-score diffusion sampler from random starts equals the exact optimum of a single radius-KrKr convolutional map to within 3×10−113 \times 10^{-11} relative error across all (K,r)(K, r) with Kr≤12Kr \le 12, decaying at rate 1.021.02 per unit KrKr (predicted: 4/ξ=14/\xi = 1).
  • Transfer of trained local coupling flows across sizes (Figure 2(b,c), Tables 2–4): Trained KK-layer affine coupling flows (R=KR = K) evaluated without retraining up to N=512N = 512 (1D) and 64×64=4,09664 \times 64 = 4{,}096 sites (2D massive free field) show a constant per-site divergence across system sizes, always above their receptive-field floors. At K=2K = 2, the trained flow comes within 0.30.3% of the exact optimum of its linearised parameterisation, confirming that the architecture rather than training sets the limit. The same size-independent per-site divergence holds on strongly non-Gaussian double-well lattice ϕ4\phi^4 up to 32×3232 \times 32 (0.097,0.032,0.0240.097, 0.032, 0.024 nats/site for R=2,4,6R = 2, 4, 6, using HMC reference samples and thermodynamic integration for log⁡Z\log Z; Table 4).
  • Critical 2D Ising model up to 512×512512 \times 512 spins (Figure 3, Tables 8–9): Floors estimated from Wolff cluster samples by thermodynamic integration agree with Kaufman's exact finite-torus solution within 0.70.7 standard errors at every size (6×66 \times 6 to 20×2020 \times 20). The exact per-site floor converges as 1/L1/L to Onsager's infinite-lattice value (0.386840.38684 nats/site, within 0.120.12% at 512×512512 \times 512), where any site-factorised proposal needs ≈1044,094\approx 10^{44{,}094} draws. Chow–Liu tree proposals remove a median of 7777% of the product floor but leave a residual floor growing linearly at 0.0950.095 nats/site (Table 8).
  • Annealed sampling work (Figure 6): On Ising chains with exact stage-wise draws, measured Var(log⁡w)\mathrm{Var}(\log w) matches J/KJ/K (6.206.20 vs. 6.196.19 at N=128,K=16N=128, K=16), and minimal work lies within 1.006–1.22×J/ϵ21.006\text{–}1.22 \times J/\epsilon^2 for N=16–512N = 16\text{–}512 (reducing 1072.710^{72.7} one-shot draws at N=512N=512 to 1.9×1051.9 \times 10^5 transitions).
  • Equilibrium alanine peptides and trained neural proposals (Figure 4(b,c), Figure 8, Tables 5–6): Replacing short MD17 trajectories, we sample Ace-(Ala)n_n-NMe (n=1,3,5n = 1, 3, 5; amber14 + OBC2 at 300 K300\text{ K}) to equilibrium via temperature replica exchange (1.3 μs1.3\ \mu\text{s} total at 300 K300\text{ K} across independent runs, cutting the slowest torsion autocorrelation time from 29 ns29\text{ ns} to 72 ps72\text{ ps}). Every estimate subtracts a circular-shift null (≤0.5\le 0.5% of raw at b=8b=8), changes by ≤4.2\le 4.2% when simulated time is halved, and rises monotonically with bin resolution. For Ace-(Ala)5_5-NMe, the certified continuous product floor is 1.042±0.0091.042 \pm 0.009 nats (ESS≤36\mathrm{ESS} \le 36%). On disjoint held-out runs, tabular ML models lose the certified class gaps to within 0.0100.010 nats, and continuous von Mises mixture-density networks lose 1.646±0.0121.646 \pm 0.012 nats (torsion-factorised) and 0.755±0.0040.755 \pm 0.004 nats (residue-factorised), above their pre-training certificates (1.0421.042 and 0.2970.297 nats), while the trained product models sit within 0.0040.004 nats of the exact class optimum. Reweighting the torsion-factorised neural model gives 0.280.28% ESS, within the 3636% ceiling.
  • Non-Markov autoregressive benchmark (Figure 4(a)): Replaced the first-order Markov exponential covariance with Matérn-3/23/2 targets, showing geometric decay in context length cc at a rate set by ξ\xi.

All revised text, proofs, figures, and tables are in the updated manuscript PDF: new theory in Table 1, Sections 3–4, and Appendix A (Lemma 9, Proposition 10), estimators in Section 5, and experiments in Section 6 and Appendices B–E.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.