acceptodds
Under review as a conference paper at ICLR 2027

When Do Local Score Models Extrapolate Across Size? A Diagnostic Theory and Benchmark

Abstract

Scientific generative modeling often requires size transfer, where models trained on small systems are evaluated on larger ones. While translation-invariant architectures enable this evaluation, we show that architectural locality alone does not guarantee stable size extrapolation. Instead, stable extrapolation depends on how far the Gaussian-smoothed score responds to a perturbation. Even when clean interactions are local, posterior correlations can make the smoothed score respond to distant perturbations. We formalize when this response remains sufficiently local for fixed-receptive-field models to transfer across system sizes, and prove a system-size-independent error bound for generated distributions on fixed local regions. We also introduce Finite-Depth Local Flow (FDLF), a white-box diagnostic benchmark with exact clean scores, densities, and controllable response ranges. Across 2D and 3D FDLF experiments, models transfer stably when the teacher response lies within their receptive fields. A related size-dependent pattern emerges near criticality in the standard 2D Ising model. In a 2D Ising test, the center-score change error of a fixed local model grows with system size near criticality but not away from it.

Primary area:
generative models
Keywords:
Score-based diffusion models, Size extrapolation, Smoothed-score locality

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PGg1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when fixed-receptive-field score models trained on small spatial systems can extrapolate to larger ones. Its central claim is that architectural locality and local clean interactions are insufficient: the relevant quantity is the spatial response of the Gaussian-smoothed score, which is governed by noisy-posterior covariance. The paper introduces a response-tail diagnostic and proves a system-size-independent bound on fixed-patch distribution error under dynamic quasi-locality and influence-weighted drift-error assumptions, together with regularity and initialization conditions. It also introduces Finite-Depth Local Flow, a synthetic benchmark with exact clean densities and scores and controllable clean-score response structure. Experiments on 2D and 3D FDLF systems examine size transfer and receptive-field dependence; an additional Ising experiment demonstrates increasing missed far-field response near criticality under a controlled perturbation test.

Critical strengths

1. The distinction between architectural locality and smoothed-score locality is important and well motivated. The posterior-covariance identity in Eq. (9), although standard, clarifies why local clean interactions need not imply a local denoising rule. The paper uses this identity and the response tail in Eq. (5) to formulate a concrete diagnostic for local score extrapolation, rather than treating the ability to run a convolutional model on larger inputs as evidence of valid transfer.

2. The white-box FDLF benchmark enables direct and interpretable evaluation of local size extrapolation. FDLF exposes exact clean scores and allows controlled variation of teacher response structure, supporting score-error and near/far-response measurements that are more directly interpretable than sample-quality metrics alone. The experiments include short-range controls, receptive-field sweeps, 2D and 3D evaluations, and multiple training seeds. These features make the benchmark potentially useful for distinguishing stable transfer from response-mismatch failures.

Critical weaknesses

1. The theoretical guarantee does not establish extrapolation from training-size error. Assumption 5 requires a system-size-independent bound on influence-weighted drift error along learned reverse trajectories. The paper does not derive this bound from DSM error at the training size or from the response tail beyond the model's receptive field. The theorem therefore provides a conditional stability guarantee for generated patch distributions, but does not establish that a model trained on a small system satisfies the required error bound on larger systems.

2. The empirical evidence does not fully validate or isolate the proposed mechanism. FDLF compares student score predictions on clean samples at σmin⁡=0.005\sigma_{\min}=0.005 with exact clean scores, rather than evaluating smoothed-score accuracy at typical noisy observations or the accuracy of generated patch distributions. The exact Gaussian reverse-process analysis in Appendix A.3 does not close this gap. The Ising size-transfer test in Figure 4 starts from y=0y=0; its perturbation lies entirely outside the receptive field, so the CNN's predicted center-score change is identically zero regardless of training. This demonstrates a representational limitation at the tested configurations, but not its prevalence under typical noisy observations or its impact on generation. Finally, the unnormalized long-range FDLF kernel allows total coupling to increase with size, leaving receptive-field limitations insufficiently disentangled from potential shifts in local marginals and field scale.

Questions for authors

1. Theoretical clarification: What additional assumptions would allow ηm\eta_m in Assumption 5 to be bounded in terms of population excess DSM risk at the training size and the response tail beyond radius RR? In particular, what conditions are needed to control cross-size changes in local noisy distributions and the distribution shift from noisy training data to learned reverse trajectories?

2. Evidence connecting the diagnostic to generation: Does the response-tail diagnostic predict score error at typical observations Y∼pL,σY\sim p_{L,\sigma} across several non-negligible noise levels, and does this relationship extend to fixed-patch distribution accuracy after learned reverse sampling? In particular, does the Ising size-dependent mismatch persist under typical noisy observations, rather than only in tests starting from y=0y=0?

3. Controls for interpreting the failure mechanism: Does long-range FDLF degradation persist in an additional control that approximately holds total coupling or field scale fixed across sizes? At each evaluation size, how does the transferred model compare with the same architecture trained directly at that size under comparable optimization conditions?

Reviewer_MdUHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when a local score model trained on a small lattice remains accurate on larger lattices. It relates the spatial response of the smoothed score to correlations in the noisy posterior and gives a conditional error bound for generated distributions on fixed patches. It also introduces FDLF, a benchmark with exact densities and clean scores, and studies transfer using FDLF and Ising experiments.

Critical strengths
  1. The question is relevant. Being able to run the same CNN on a larger lattice does not mean that its predictions remain correct, and the paper explains this distinction clearly. Eq. 9 gives a simple reason why local physics does not guarantee a local smoothed score, and the checklist in the conclusion is something I would use when reading size-transfer claims.

  2. I find the benchmark valuable. Exact densities and clean scores at every size make it possible to examine errors directly and to compare teachers of different range under one protocol, in 2D and 3D. The medium-range experiment in Fig. 2 shows a clear improvement with a larger receptive field: the score RMSE at L=64L=64 drops from 0.338 to 0.256 as ReffR_{\mathrm{eff}} goes from 4 to 14. The focus on fixed-patch distributions also makes sense for the scientific applications considered here, and deriving the influence weights from a drift-Jacobian condition (Prop. 3) is a nice step. I went through Props. 1 to 4 and Thm. 2 and did not find errors.

Critical weaknesses
  1. My main concern is that the experiments do not clearly separate a failure to transfer from a limitation of the local architecture. The CNN is the same family of local functions at every L>2R+1L > 2R+1, but the best local predictor can still change with LL. For the long-range FDLF teacher, a model trained directly on the larger lattice might also have substantial error. Without that comparison, it is difficult to tell how much of the degradation comes from training at the smaller size. Training-size errors for the depth sweep would also help, since the Reff=14R_{\mathrm{eff}}=14 window (29×2929\times29) covers most of the 32×3232\times32 training lattice. There is a related gap between the proposed explanation and the experiments. The main claim is about the smoothed score at σ>0\sigma > 0, but Eq. 13 compares the student at σmin⁡\sigma_{\min} to the clean score sL,0s_{L,0}, and the long-range teacher already has non-local clean dependence. The kernel e−r/60e^{-r/60} also has a decay length longer than every tested lattice, so the total coupling grows over the tested sizes (App. B.1), and range and coupling strength change together. The Ising experiment does show a long-range smoothed-score response near βc\beta_c, and I find that part convincing. In Fig. 4, however, the CNN response is zero by construction because the perturbation is outside its receptive field, and at y=0y=0 the posterior equals the zero-field prior for every σ\sigma. Fig. 4 therefore shows a real limit shared by every radius-5 model. It does not tell us how much that limit costs a trained model on the typical noisy inputs it sees during generation.

  2. I would like the scope of the theoretical result to be stated more carefully. Theorem 2 gives a sound sampling guarantee once the influence and drift-error bounds hold uniformly across sizes (Assumptions 4 and 5). However, Assumption 5 already requires control of the learned model's error along its own trajectories at the larger sizes, and the theorem does not establish when training on a small lattice provides that control. My concern is therefore with the conclusion drawn from the result: receptive-field coverage seems to be presented as sufficient for transfer, while the analysis also needs assumptions about the accuracy of the learned rule. No experiment evaluates generated samples, so Thm. 2 is also not tested directly.

Questions for authors
  1. Do you have results for the same architecture trained directly at the larger evaluation size? Comparing them with the transferred model would help clarify the first concern. For Fig. 2 this includes the error at L=32L=32.

  2. Could the proposed explanation be tested at positive noise levels using the Gaussian field in App. A.3? It has a local clean energy, an exact smoothed score at every σ\sigma, and a correlation length set by the mass, so it would connect response range with the actual transfer error. A long-range FDLF teacher with fixed total coupling would also help.

  3. Have you looked at patch statistics of generated samples at L>LtrainL > L_{\text{train}}, for FDLF or Ising? Such results would test Thm. 2 directly.

Clarifications:

  1. What additional conditions would connect small-lattice training to Assumption 5? If this connection is outside the paper's scope, could the conclusion make that limitation clearer?

  2. Gottwald et al. [1] use the same covariance identity to show that the smoothed score stays approximately local under correlation decay, and they bound the error of the best localized score. The introduction separates the present work from static marginal bounds such as Cui et al. [2]. Beyond that static vs. dynamic distinction, what does Thm. 2 add?

I see value in the question and the benchmark, but I am not yet convinced that the results establish when the proposed criterion predicts transfer of a learned model. A direct comparison with training at the larger size and a controlled positive-noise experiment would make the case stronger. I would be open to revising my assessment if the authors can address these points with additional evidence or clarification.

References

[1] G. A. Gottwald, S. Liu, Y. Marzouk, S. Reich, and X. T. Tong. Localized diffusion models. arXiv:2505.04417, 2025.

[2] T. Cui, S. Liu, and X. T. Tong. Stein's method for marginals on large graphical models. arXiv:2410.11771, 2024.

Reviewer_7DZqOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposed Finite-Depth Local Flow (FDLF) as a testbed diagnostic benchmark for local size extrapolation, which investigates the whether a lcoal score-based model trained on low resolution dataset can be transferred to high resolution dataset. The paper use spatial derivative of the score function to diagnose the receptive field. The experiments provides some evidence connecting to score locality, receptive field, and extrapolation performance.

Critical strengths
  1. Novel problem definition
Critical weaknesses
  1. The becnhmark dataset is too small, could you evaluate on higher resolution dataset like 256 x 256?
  2. Figure 2 only shows the results on 3 kinds of effective radius, can you try more different effective radius? like 32, 64, 128...,
  3. It's not easy to understand the motivation. The introduction doesn't define the problem clearly. It's weird to introduce score function in section 2 LOCAL SCORE EXTRAPOLATION, is that a standard definition for extrapolation? Or just an introduction for the score function?
  4. Convolution layers can capture local pattern, it should be a common sense in ML and CV field. However, I didn't see reference that mentioned this. Would you mind to cite more related work for the analysis of CNN?
Questions for authors

Same as Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.