acceptodds
Under review as a conference paper at ICLR 2027

SLED: Sobol-Leverage Designs for the Collocation Batch of Quasi-Newton Physics-Informed Neural Network Training

Abstract

A *two-phase optimization schedule* for physics-informed neural network training, a first-order *warmup* followed by a quasi-Newton optimizer, achieves more accurate results than single-phase approaches in a growing number of benchmarks. The quasi-Newton phase minimizes the objective on a collocation batch that is usually held fixed, and no pipeline selects that batch from the tangent features of the residual after the warmup. For the network linearized there, every gradient and Broyden-class method on a fixed batch returns the minimum-norm least-squares fit of that batch. The network leaves this linearization, but on all six benchmarks we study the linearized fit still orders the frozen batches by their loss off the batch, and a batch is judged by that loss. We build the batch once, after the warmup. Specifically, the Sobol-leverage design (Sled combines a scrambled-Sobol set with points drawn from the ridge leverage of the residual's tangent features, in a share computed on the candidate pool. The ridge-leverage sampling bound, stated for these features, gives a sufficient batch size for a leverage batch, and the two benchmarks on which leverage points alone fail are the two on which the batch lies below it. On six benchmarks, Sled reaches a median error at or below that of three established frozen batches, the uniform default among them, and of three refresh protocols, up to and below the default, at of the cost of the phase. It reaches the default's error at points with fewer points on four of them. Each component alone trails it somewhere: leverage points fail on two benchmarks, and the Sobol set is less accurate on two others.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
physics-informed neural networks, collocation point sampling, quasi-Newton optimization, neural tangent kernel, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mkXsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

SLED designs a frozen collocation batch for quasi-Newton refinement after first-order PINN warmup. It combines residual-tangent ridge-leverage sampling with scrambled Sobol points. A rank-correlation statistic sets their proportions. The paper analyzes linearized training and applies existing spectral-embedding results. Experiments cover six PDE benchmarks, frozen and refreshed baselines, batch-size scaling, and off-batch diagnostics.

Critical strengths

The post-warmup collocation design problem is practically relevant. Shared warmup checkpoints and ten-seed comparisons improve experimental control. Off-batch residual diagnostics also help distinguish fitting the selected points from generalization across the domain.

Critical weaknesses

1. The central allocation rule lacks sufficient justification and specification. Section 4.2 claims that agreement between leverage and kernel-diagonal rankings implies an absence of localized directions. This is false in general. For a nonconstant, strongly localized scalar tangent feature ϕ(x)\phi(x),

\ell_\lambda(x)=\frac{\kappa(x)}{\mathbb{E}_\rho[\phi^2]+\lambda}.$$ The Spearman correlation is 1 for every $\lambda>0$, despite localization. Stable rankings do not resolve this counterexample. Proposition 4.4 establishes monotonicity under adding points, not superiority over equal-budget batches. Corollary 4.3 bounds interpolation within $H_K$, not the full inconsistent least-squares or nonlinear training problem. These results do not derive $\mu_{\mathrm{sob}}=\hat\rho$. Section 4.2 also requires major rewriting. Nyström and Hutchinson alone do not uniquely specify $\hat\ell_\lambda$, $\hat\kappa$, or $\hat\lambda_{\max}$. Estimator definitions, algorithmic steps, heuristic choices, and guarantees should be clearly separated. Unexplained line references and the distinct meanings of $\rho$ and $\hat\rho$ add avoidable ambiguity. **2. The experiments do not isolate the contribution beyond existing methods [1–3].** LEV contains a 20% uniform mixture. It is not SLED with $\mu_{\mathrm{sob}}=0$. KD-W does not preserve SLED's Sobol component while changing only the score. These are not controlled ablations of allocation, coverage, and spectral normalization. The PINNACLE comparison uses only a one-shot, frozen K variant. It omits PINNACLE-S and adaptive reselection. Describing PINNACLE as limited to a "first-order phase" conflates its Adam-based experiments with a restriction on its selection rule. Whole-phase freezing is also not a general requirement of quasi-Newton optimization [4].
Questions for authors

1. The reproducibility statement indicates that code, checkpoints, and configurations will be released. I could not find them in the review materials. Please attach the anonymized code and experimental scripts. Please specify Nyström landmark selection, spectral normalization, Hutchinson probes, and multi-residual aggregation so Algorithm 1 can be reproduced.

2. Can the localization claim be justified under explicit assumptions? Please test the allocation rule against pure leverage and fixed Sobol shares. Vary the coverage distribution and sampling score separately to isolate their effects.

3. Why were PINNACLE-S and periodic or eNTK-triggered reselection [1] excluded? Since the refresh baselines retain quasi-Newton memory across collocation updates (Appendix C.3), could this compromise curvature estimation and bias the comparison? Please clarify how curvature pairs are handled during refresh and whether resetting the optimizer memory affects the results.

4. Which IC/BC features and loss weights affect the sampling scores? Does the method remain effective when IC/BC sampling or relative loss weights change? Please also discuss CoPINN [5] in Related Work. Its sample-difficulty-based scheduling is relevant to the paper's broad definition of domain importance adaptation.


References

[1] Lau et al., "PINNACLE: PINN Adaptive ColLocation and Experimental points selection", ICLR 2024.

[2] Cardenas et al., "CS4ML: A general framework for active learning with arbitrary data based on Christoffel functions", NeurIPS 2023.

[3] El Alaoui et al., "Fast Randomized Kernel Ridge Regression with Statistical Guarantees", NeurIPS 2015.

[4] Berahas et al., "A Multi-Batch L-BFGS Method for Machine Learning", NeurIPS 2016.

[5] Duan et al., "CoPINN: Cognitive Physics-Informed Neural Networks", ICML 2025.

Reviewer_aiYvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which collocation batch should be frozen during the quasi-Newton phase of two-stage PINN training. SLED combines scrambled Sobol points with approximate ridge-leverage samples of the residual's tangent features at the warmup checkpoint. Their mixture share is determined by a pool-based rank correlation. Classical linearized least-squares and sampling results motivate the design, and experiments compare ten sampling/refresh variants on six PDE benchmarks from shared per-seed warmup checkpoints. The paper also evaluates off-batch residual loss, batch-size scaling, and construction overhead.

Critical strengths

The paper presents an interesting idea of using post-warmup tangent features to design a fixed collocation batch for quasi-Newton PINN training, supported by a thorough and well-controlled experimental evaluation.

All sampling methods are compared using shared per-seed warmup checkpoints, with the same network architecture, loss balancing, and second-phase optimizer. The six PDE benchmarks, ten-seed evaluations, paired comparisons, batch-size scaling studies, and off-batch diagnostics provide convincing empirical evidence of the effectiveness of the proposed approach. The authors also report failure cases and computational overhead, making the experimental findings more informative and credible.

Critical weaknesses

Insufficient validation of the adaptive mixing rule: While the experiments demonstrate the complementary behavior of Sobol and leverage sampling, they do not fully establish the benefit of the proposed automatic mixing rule. In particular, the Sobol fraction is directly set to the nonnegative part of the Spearman rank correlation between the leverage scores and kernel-diagonal scores, but the paper does not compare this choice against simple fixed mixture ratios. Such an ablation would help distinguish the benefit of combining the two sampling strategies from the benefit of the proposed adaptive rule. Additionally, the LEV baseline includes a 20% uniform component and is therefore not the exact zero-Sobol endpoint of SLED, as the paper acknowledges. A matched endpoint comparison would make the component analysis more convincing.

Questions for authors
  1. How does the proposed automatic mixing rule compare with a small set of fixed Sobol/leverage mixture ratios on the existing benchmarks? Does the component analysis remain consistent when the zero-Sobol endpoint uses the same leverage sampling and weighting as SLED, without LEV's additional uniform component?
  2. The scope paragraph at L251–257 relates Corollary 4.3 to Lemma 3.1(ii), whereas Section 5.2 and Appendix D report that the experimental linearized systems fall under the inconsistent case in Lemma 3.1(iii). Does the leverage-sampling analysis in Section 4 yield an upper bound on the off-batch residual loss of that inconsistent weighted least-squares solution, beyond the kernel interpolation error bound? If additional assumptions or control of the out-of-range residual are needed, please clarify them and delimit what the current results guarantee.
  3. Minor presentation comments (no response required). The presentation could be improved, particularly in explaining the connections between the theoretical analysis and the proposed method, which sometimes require back-and-forth reading. Two minor points: “no pipeline” in the abstract could more explicitly refer to prior two-phase training pipelines, and RAD should be defined at its first occurrence (L63), rather than later at L94.
Reviewer_T1aSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies two-phase PINN training, where a first-order warmup is followed by a quasi-Newton phase on a fixed set of collocation points, and asks which points this set should contain. Authors propose SLED, where part of the points is sampled according to the ridge leverage scores of NTK features, and the rest is a scrambled Sobol set. The share of Sobol points is set by the rank correlation between the leverage scores and the NTK diagonal on a candidate pool. The design is motivated by an analysis of the network linearized at the end of the warmup and by a known sample-size bound for leverage sampling. Authors compare SLED with competitors on six PDE benchmarks, and provide ablations of the methodology.

Critical strengths
  1. The numerical study is well designed. Methods start from shared checkpoints after the warmup, 10 seeds are used, and paired ratios with bootstrap intervals are reported for the main pairs.
  2. The paper motivates the study of adaptive localization mechanisms by demonstrating how the batch fails on problems with localized structures (the coherence of the tangent features in Table 1 and the visualizations in Appendix C.8).
  3. Building the batch in SLED is cheap. It takes at most 4.2%4.2\% of the second training phase on the current benchmark suite.
Critical weaknesses
  1. My main concern is limited novelty. The proposed methodology is a combination of known techniques. Its claimed novelty is that leverage scores are computed for the tangent features of the residual, the leverage points are mixed with Sobol points, and the mixing share is set by an adaptive rule. However, NTK-based point selection for PINNs was already proposed in [1], and Sobol collocation was studied in [2]. Thus, the novelty of the work reduces to mixing leverage scores and Sobol with a switch designed by the authors. In my view, this is an incremental methodological contribution. The theoretical results in the paper are also classical and amount to repeating known arguments.
  2. The theory in the submission seems to be inapplicable. The second and third columns of Table 13 show that the final residual is determined by the nonlinear part. Therefore, the small errors of SLED cannot be explained by the theorems of the paper, which are stated for linear models. The fact that on KS the NTK kernel changes by 450%450\% during the second phase also indicates that the linear theory does not fit this work.
  3. Figure 1 shows that the gap between SLED and its competitors, even uniform one, shrinks as the number of collocation points in the batch grows. This raises questions about the scalability of the methodology to high-resolution grids in applied problems, where much larger batches are used. Therefore, the paper needs a study of how SLED and its competitors scale with nn to establish the upper limits of applicability of the methodology.

[1] Lau, Gregory Kang Ruey, et al. "PINNACLE: PINN adaptive collocation and experimental points selection." International Conference on Learning Representations. Vol. 2024. 2024.

[2] Mishra, Siddhartha, and T. Konstantin Rusch. "Enhancing accuracy of deep learning algorithms by training with low-discrepancy sequences." SIAM Journal on Numerical Analysis 59.3 (2021): 1811-1834.

Questions for authors

How does SLED compare with its competitors for nn larger than 16,38416,384?

Comments on the paper

1 comment

Dear reviewers,

Thank you for your reviews. The author-reviewer discussion closes on November 18, so please respond to the authors' rebuttal before then. Please say whether it addressed your concerns, and update your score if your view has changed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.