acceptodds
Under review as a conference paper at ICLR 2027

Random Resampling of Collocation Points Can Amplify Spectral Bias in PINNs

Abstract

Random resampling of collocation points is standard practice in physics-informed neural network (PINN) training, and is widely expected to be especially beneficial for multi-scale problems where resolving fine-scale structure seems to demand sufficient coverage of the domain. We show that this expectation is misleading. Across a range of multi-scale PINN benchmarks, random resampling rapidly recovers the macro-scale solution but substantially delays high-frequency recovery. In contrast, fixed sampling recovers high-frequency components earlier, but introduces global distortion through collocation bias. This delay coincides with the regime where the variance of the residual gradient under resampling exceeds its mean magnitude. Guided by this diagnosis, we introduce spatial and temporal interpolations between fixed and random resampling that consistently reduce the delay in fine-scale recovery. These findings suggest that spectral bias in PINNs is not only a consequence of sparse collocation, but also a phenomenon shaped by how collocation points are resampled during training.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Spectral Bias, Physics-Informed Neural Networks, Multi-Scale PDEs, Collocation Point Resampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wX1rOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper concerns PINN training methods for multiscale physics problems. It analyses how collocation point sampling strategies, fixed sampling and random resampling, affect the recovery of low- and high-frequency components in the solution, primarily through a set of experiments. It is shown that fixed sampling identifies high-frequency components faster than random resampling, but at the cost of bias in low-frequency components. Under random resampling, the variance of the residual gradient and the time-to-identification of a modal component are both positively correlated with the mode frequency and negatively correlated with the sample size. Further, the signal-to-noise ratio (SNR) of the residual gradient is negatively correlated with both the H1 seminorm error and the modal error. Based on these findings, the paper suggests using a mix of fixed sampling and random resampling, where part of the collocation sample is fixed during training and part of it is resampled at each iteration. The approach is experimentally validated on a set of benchmark problems and compared to related methods.

Critical strengths

The motivation and analysis developed through the series of experiments is logically sound and comprehensive, starting with why PINNs behave differently to regression when it comes to mode recovery under fixed and random resampling, subsequently analysing the relationship between delayed mode recovery and residual gradient variance, and finally analysing the relationship between SNR and mode recovery.

In light of these experiments, the idea of using a mix of a fixed and randomly resampled set of collocation points is logical. The method achieves strong experimental results on a set of standard benchmark problems in PINNs as compared to fixed sampling, random resampling, and a benchmark given by importance sampling. The method can be combined with other sampling methods (LAS and R3), achieving improvements in performance.

Critical weaknesses

While the paper provides ample experimental support, I am missing a theoretical analysis of the proposed method. For the spatial, temporal, and spatio-temporal interpolation strategies between fixed sampling and random resampling, could you provide theoretical insight into how they change the bias and variance of the residual gradient? Could you provide insight into how to choose \rho and p based on these results, perhaps connecting this to the adaptive interpolation scheme proposed in Appendix G.7?

When I look at the experiments in Appendix G.6 illustrating the sensitivity of the method to the choice of resampling fraction \rho or resampling probability p, it seems like low values of these give the best balance between low bias in the low-frequency component and decent time-to-identification in the high-frequency component (\rho=0.1 in Fig 18, p=0.01 in Fig 19). I would appreciate an analysis of why this is the case, especially considering the result in Figure 1, where a fixed collocation sample is associated with strong visible bias, while random resampling gives a much better L2 reconstruction error (while visibly missing high-frequency content). It would be great if you could give theoretical insight into this observation.

Questions for authors

In Fig 3, you observe a bias only for the lowest number of collocation points. Can you discuss this in relation to Fig 1, where all solutions have strong visible bias for the fixed sampling method? What would happen in Fig 1 if you increased the sample size?

Since the bias caused by a fixed collocation sample is related to overfitting, and the culprit in random resampling is poor SNR of the residual gradient, I am wondering how your method compares to dropout [Ref 1]. This method uses a Bayesian neural network parametrization to overcome overfitting, and a local reparameterization trick to provably reduce the variance of the loss gradient. Is it applicable, and how would the results compare to your method on this problem?

[1] Kingma, D. P., Salimans, T., & Welling, M. (2015). Variational dropout and the local reparameterization trick. Advances in Neural Information Processing Systems, 28.

3 comments

We sincerely thank the reviewer for the careful review of our paper. Please find our responses to each comment below.

W1. Theoretical analysis of interpolation methods

We thank the reviewer for this excellent suggestion. Following it, we added a bias-variance analysis of the interpolation schemes in Section 5.2 (Table 2), with derivations in Appendix G. The analysis shows that, at a fixed θk\theta_k, interpolation does not change the total gradient error but redistributes it between conditional bias and conditional variance. Spatial interpolation introduces a bias that persists throughout training, whereas temporal interpolation replaces the reused set after 1/p1/p iterations on average.

Regarding the interesting question about how to choose ρ\rho and pp, this analysis alone does not determine their optimal values, because the optimal split depends on the training dynamics. Nevertheless, it provides a rationale for the adaptive scheduler, which reduces ρ\rho (or pp) only when the residual gradient SNR falls below 1 and thus trades variance for bias only in the noise-dominated regime.

W2. Why small ρ\rho and pp can be effective

Thank you for this insightful observation.

Suppose the network can represent the exact solution, that is, there exists θ⋆\theta^\star with Lr(θ⋆)=0L_r(\theta^\star)=0. Under fixed sampling, a parameter satisfying L^r(θ;Xr)=0\widehat{L}_r(\theta;\mathcal{X}_r)=0 does not necessarily minimize the population loss Lr(θ)L_r(\theta), since the PDE may be satisfied only at the selected collocation points. This is consistent with the global distortion in Figure 1.

In contrast, spatial interpolation minimizes the expected objective (1−ρ)L^r(θ;XF)+ρLr(θ)(1-\rho)\widehat{L}_r(\theta;\mathcal{X}_F)+\rho L_r(\theta) given XF\mathcal{X}_F. Since both terms are nonnegative and vanish at θ⋆\theta^\star, any global minimizer with ρ>0\rho>0 must also satisfy Lr(θ)=0L_r(\theta)=0. Hence, if this expected objective is minimized well, the collocation bias of fixed sampling can be removed. Since this holds for any ρ>0\rho>0, it provides theoretical intuition for why even a small ρ\rho can be effective. Temporal interpolation follows a similar intuition, as repeated refreshes introduce new collocation points and allow residual errors outside previously sampled points to be corrected.

Q1. Large sample size with fixed sampling

Thank you for this question. To answer it, we repeated the experiments in Figure 1 with fixed sampling using 10 times more collocation points and added the results to Appendix H.1 of the revised manuscript. With the sufficiently larger collocation set, fixed sampling recovers the solutions without visible distortion. This indicates that the bias in Figure 1 arises from the limited collocation budget, consistent with Figure 3, where fixed sampling exhibits bias only for the small NrN_r.

Q2. Comparison with variational dropout

We thank the reviewer for this interesting suggestion. The analogy is natural, since collocation bias resembles overfitting to a fixed training set and our diagnosis concerns gradient variance. We would like to kindly note, however, that the variance reduced by the local reparameterization trick differs from the variance we study. The trick reduces the variance induced by sampling the stochastic weights, by making this weight noise independent across data points. The resampling variance in our setting instead arises from sampling the collocation points themselves and is present even for a deterministic network.

To examine this empirically, we applied Gaussian dropout with the local reparameterization trick following [1], with the dropout rate kept constant during training. We trained on the Poisson problem under fixed sampling and random resampling for several dropout rates pequivp_{\mathrm{equiv}}.

pequivp_{\mathrm{equiv}}Fixed eH1e_{H^1}Fixed eL2e_{L^2}Random eH1e_{H^1}Random eL2e_{L^2}
000.498±0.1760.498_{\pm 0.176}0.586±0.4070.586_{\pm 0.407}0.733±0.0180.733_{\pm 0.018}0.377±0.1090.377_{\pm 0.109}
10−410^{-4}0.512±0.0480.512_{\pm 0.048}0.591±0.1490.591_{\pm 0.149}0.719±0.0030.719_{\pm 0.003}0.259±0.0320.259_{\pm 0.032}
10−310^{-3}0.483±0.0620.483_{\pm 0.062}0.612±0.2450.612_{\pm 0.245}0.719±0.0040.719_{\pm 0.004}0.246±0.0410.246_{\pm 0.041}
10−210^{-2}0.544±0.0420.544_{\pm 0.042}0.594±0.1410.594_{\pm 0.141}0.719±0.0030.719_{\pm 0.003}0.244±0.0250.244_{\pm 0.025}
10−110^{-1}0.668±0.0220.668_{\pm 0.022}0.644±0.0910.644_{\pm 0.091}0.767±0.0620.767_{\pm 0.062}0.408±0.2340.408_{\pm 0.234}

In our experiments, we did not observe that dropout mitigates the collocation bias of fixed sampling or the high-frequency delay under random resampling.


[1] Kingma, D. P., Salimans, T., & Welling, M. (2015). Variational dropout and the local reparameterization trick. Advances in Neural Information Processing Systems.

W1. Theoretical analysis of interpolation methods

Thank you for the extended theoretical analysis. The conditional bias-variance decomposition clarifies your insights and provides a valuable addition to the work, in my view.

I checked the added computations in Appendix G, and I believe they are correct. For improved clarity, maybe you can modify “Since XF\mathcal{X}_F contains (1−ρ)Nr(1−\rho)N_r,…” to instead say how the variance scales relative to Σr\Sigma_r, as you do in the spatio-temporal computation, but it is not critical. Just watch out for typos “Eq.eq.” between eqs. 37 and 38 as well as 43 and 44.

W2. Why small pp and ρ\rho can be effective

Thank you for the clarification – I understand the point.

Q1. Large sample size with fixed sampling

Thank you for providing new figures. Since the limited collocation budget exaggerates the issue here, I suggest that you add a small comment in the main text discussion on Figure 1.

Q2. Comparison with variational dropout

Thank you for your comment and added experiments, and I agree with your point.

In light of the authors’ revisions and responses, I raise my score to 4. Clear acceptance.

We appreciate the reviewer's positive assessment. We have corrected the noted typos in Appendix G and added a clarification regarding the limited collocation budget in Section 3.2. Thank you again for your thoughtful engagement.

Reviewer_ispeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the effect of stochastic resampling of interior collocation points on the spectral learning dynamics of PINNs. The authors find that, on several multiscale problems, per-step resampling recovers the macroscopic structure well but markedly delays the recovery of high-frequency components; fixed collocation points recover high frequencies earlier but can introduce global distortion.

The authors analyze this phenomenon through residual gradient covariance and global and per-mode signal-to-noise ratios, and provide interventional evidence via noise-injection experiments. Building on this analysis, the paper proposes spatial, temporal, and combined collocation-reuse strategies, supplemented by an adaptive schedule based on gradient SNR. Experiments cover six manufactured-solution problems, reporting function-value errors, derivative errors, and partial runtime costs.

Critical strengths

The research question is clear and practically relevant. The paper draws attention to a common yet easily overlooked training choice: whether collocation points should be refreshed at every step. The authors examine the effects of frequency, number of collocation points, learning rate, Fourier features, and the collocation-reuse ratio. The appendix includes both isotropic and covariance-matched noise-injection experiments, which strengthens the plausibility of the claim that gradient noise affects the order in which frequencies are recovered. Partially retaining points and reducing the refresh frequency require no modification to the PDE or the network architecture. The fixed-hyperparameter version has per-step overhead close to that of conventional sampling, and the appendix further demonstrates results from combining the strategy with existing point-update rules.

Critical weaknesses

The gradient SNR analysis does not fully correspond to the actual Adam updates or to the boundary and initial-condition losses, so it cannot yet completely explain the stagnation of high-frequency learning. Collocation reuse simultaneously alters the gradient noise, the cross-step correlation, and the empirical objective, and the sources of the observed gains remain unseparated. The main tables rely on the exact solution for hyperparameter selection; the practically deployable adaptive scheme and its computational cost require more thorough validation. In addition, the variance-reduction baselines are not comprehensive, and the phenomenon weakens markedly under larger Fourier bandwidths, indicating that the conclusions have a limited scope of applicability.

Questions for authors

see weakness

1 comment

We sincerely thank the reviewer for the careful review of our paper. Please find our responses to each comment below.

W1. Adam updates and boundary/initial losses "The gradient SNR analysis does not fully correspond to the actual Adam updates or to the boundary and initial-condition losses ..."

We agree that the residual gradient SNR is defined for the interior residual gradient and does not account for the adaptive preconditioning of Adam or for the boundary and initial losses. We state this in the limitations of the revised manuscript, and extending the analysis to these settings is an important direction for future work.

W2. Analysis of the effects of interpolation "Collocation reuse simultaneously alters the gradient noise, the cross-step correlation, and the empirical objective ..."

We thank the reviewer for this insightful comment. To better understand the effects of collocation reuse, we have added a theoretical analysis in Appendix G of the revised manuscript, deriving the conditional bias, conditional variance, and correlation across iterations for each interpolation strategy. The analysis shows that increasing collocation reuse reduces the conditional gradient variance while introducing a persistent sampling bias, which reflects the change in the empirical objective, and stronger correlation across iterations, whereas the marginal gradient covariance remains unchanged at fixed network parameters. In addition, as the reviewer kindly acknowledged, our noise injection experiments in Appendix H.3 examine the effect of gradient noise while keeping the collocation points fixed.

W3. Hyperparameter selection "The main tables rely on the exact solution for hyperparameter selection ..."

We thank the reviewer for pointing this out. In the revised manuscript, Table 1 reports the results for all configurations in the hyperparameter grid rather than the best configuration for each benchmark.

W4. Computational cost of adaptive scheduling "... the practically deployable adaptive scheme and its computational cost require more thorough validation."

We thank the reviewer for this suggestion. We added adaptive interpolation to the throughput comparison in Appendix D. Since the residual gradient SNR is estimated every K=200K=200 iterations, we average the throughput over 10001000 iterations so that it includes the cost of this estimation. Adaptive interpolation achieves throughput comparable to random resampling on all six benchmarks, indicating negligible computational overhead.

W5. Variance reduction baselines "In addition, the variance-reduction baselines are not comprehensive ..."

Thank you for your comment. In our work, importance sampling is included in Table 1 as a representative variance reduction baseline across all six benchmarks, and SVRG is examined in Appendix E, where it becomes unstable because its large batch reference gradient becomes stale as the PDE residual field changes during training. We would be happy to include additional variance reduction methods for PINNs that the reviewer may suggest.

W6. Fourier feature embeddings with large bandwidths "... the phenomenon weakens markedly under larger Fourier bandwidths ..."

We thank the reviewer for this comment. As the reviewer noted, the amplification becomes less apparent under large Fourier bandwidths. To understand this behavior, we analyzed the residual gradient SNR during training for Fourier feature embeddings with both bandwidths on the Helmholtz equation (Figure 11 in Appendix F). For σ=2\sigma=2, where spectral bias remains, the H1H^1 error stays nearly flat while the SNR is low, exhibiting the same noise-dominated regime as vanilla PINNs. For σ=6\sigma=6, where the embedding already mitigates spectral bias, the SNR is relatively high from early in training and the error decreases from the beginning. We also added SIREN as another frequency-aware architecture and observed the same behavior as for σ=6\sigma=6. This suggests that the weaker amplification is associated with the residual gradient leaving the noise-dominated regime early when the architecture mitigates spectral bias. We therefore view this observation as consistent with our conclusions rather than limiting them.

Reviewer_88bFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how random resampling of collocation points affects the spectral learning dynamics of Physics-Informed Neural Networks (PINNs) for multiscale PDEs. The authors observe that random resampling improves the recovery of large-scale solution structures but can substantially delay high-frequency learning. In contrast, fixed sampling captures high-frequency components earlier but may introduce persistent collocation bias and distort the overall solution. To explain this trade-off, the authors analyze the covariance of stochastic residual gradients and introduce Global Gradient Signal-to-Noise Ratio (SNR) and Mode-wise SNR to characterize the relationship between sampling noise and frequency-dependent learning delays. Based on this analysis, they propose Spatial, Temporal, and Spatio-Temporal Interpolation strategies that partially fix or periodically reuse collocation points to balance sampling bias and gradient variance. They further introduce an Adaptive Interpolation Scheduler guided by Gradient SNR. The proposed methods are evaluated on six multiscale PDE benchmarks, including Poisson, Heat, Helmholtz, Darcy, Klein–Gordon, and Convection equations. The evaluation considers frequency-wise errors, capture time, relative L2L^2 and H1H^1 errors, noise injection experiments, and sampling strategy ablations. Results show that interpolation strategies generally reduce high-frequency-sensitive errors across most benchmarks, while static interpolation introduces limited computational overhead.

Critical strengths
  1. The paper identifies an interesting and practically relevant limitation of random collocation resampling in multi-scale PINNs.
  2. The proposed modal-gradient diagnostic and collocation reuse strategies provide useful empirical insights.
Critical weaknesses
  1. The experiments primarily use analytically constructed multiscale solutions and relatively small tanh MLPs. While these settings are suitable for studying spectral learning dynamics, they provide limited evidence of generalization to more challenging PDEs, including high-dimensional problems, complex boundaries, strong nonlinearities, discontinuities, and turbulent flows. Modern frequency-aware architectures and neural operators may also exhibit different gradient noise characteristics. Notably, the authors observe that the amplification of spectral bias caused by random resampling becomes less apparent with Fourier feature embeddings at σ=6\sigma=6. This suggests that the proposed method may be sensitive to the underlying architecture and its frequency representation capabilities. Furthermore, some interpolation hyperparameters in Table 1 are selected using evaluation errors against the ground-truth solutions. Although the Adaptive Scheduler partially addresses this concern, its reliability for practical PDE problems with unknown solutions requires further validation.

  2. The paper lacks a discussion of “AC-PKAN: Attention-Enhanced and Chebyshev Polynomial-Based Physics-Informed Kolmogorov-Arnold Networks (TMLR 2026”. This work investigates PINN-based PDE solving, complex oscillatory structures, gradient stability, and adaptive loss weighting. In particular, its Residual Gradient Attention (RGA) mechanism is relevant to the proposed analysis of stochastic residual gradients. The authors should discuss the relationship between RGA and resampling-induced gradient noise. Experimental comparisons would also help clarify whether the proposed interpolation strategies provide complementary benefits to gradient-aware training methods.

  3. The frequency-dependent gradient noise analysis may be affected by an important confounding factor. In the two-dimensional Poisson experiment, the target solution is u∗=sin⁡(c1x)sin⁡(c1y)+Csin⁡(c2x)sin⁡(c2y),u^*=\sin(c_1x)\sin(c_1y)+C\sin(c_2x)\sin(c_2y), where c1=4c_1=4, c2=40c_2=40, and C=0.1C=0.1. Applying the Laplacian gives f=2c12ϕc1+2Cc22ϕc2=32ϕ4+320ϕ40.f=2c_1^2\phi_{c_1}+2Cc_2^2\phi_{c_2}=32\phi_4+320\phi_{40}. Although the high-frequency component has only one-tenth of the low-frequency amplitude in the target solution, its forcing amplitude is ten times larger. A similar effect occurs in the one-dimensional Poisson experiment, where the high-frequency forcing amplitude is 40 times larger. Consequently, increasing c2c_2also increases the high-frequency forcing magnitude according to ∥fhigh∥∝Cc22.\|f_{\mathrm{high}}\|\propto Cc_2^2. Since the residual gradient satisfies ∇θℓr=2rθ∇θrθ,\nabla_\theta\ell_r=2r_\theta\nabla_\theta r_\theta, the increased forcing magnitude can contribute to larger stochastic gradient fluctuations, particularly before the network learns the high-frequency component. Therefore, the increase in gradient covariance reported in Figure 4 cannot be attributed exclusively to frequency-dependent learning difficulties. The current experiments do not adequately separate frequency effects from PDE operator scaling. A controlled experiment could adjust the high-frequency amplitude as C(c2)=C0(c1c2)2,C(c_2)=C_0\left(\frac{c_1}{c_2}\right)^2, which maintains a constant high-frequency forcing amplitude as c2c_2 varies. Spectral preconditioning or appropriate residual normalization could provide additional evidence. Comparisons across PDE operators of different orders would further strengthen the analysis.

  4. The paper's generality claims are not fully supported by its architecture ablations. Appendix F shows that random resampling still delays high-frequency learning with Fourier features at σ=2\sigma=2, while this effect becomes substantially less pronounced at σ=6\sigma=6. This raises an important question: To what extent is the observed spectral bias amplification caused by random resampling itself, and to what extent does it reflect the limited high-frequency learning capability of the underlying tanh MLP? The authors should conduct a systematic architecture ablation using vanilla PINNs, Fourier features with different bandwidths, SIREN, gradient-reweighted PINNs, and frequency-enhanced PINN/KAN architectures. Each architecture should be evaluated under fixed sampling, random resampling, and the proposed interpolation strategies. Beyond final prediction errors, the analysis should examine the relationship between Mode-wise SNR and high-frequency Capture Time across architectures. If stronger frequency-aware representations largely eliminate the resampling-induced delay, the practical benefits of interpolation may be concentrated in models with substantial spectral learning bottlenecks.

  5. The conclusions in Appendix G.8 appear inconsistent with several results in Table 7. First, the authors state that Spatial and Spatio-Temporal Interpolation improve both H1H^1 and L2L^2 errors for LAS across all six benchmarks. However, for Klein–Gordon, LAS + Spatial reduces the H1H^1 error from 0.799 to 0.689 but increases the L2L^2 error from 2.160 to 2.633. This contradicts the stated improvement across all benchmarks. Second, interpolation does not consistently improve R3 on the Convection benchmark. The baseline R3 achieves H1=1.025H^1=1.025 and L2=0.509L^2=0.509. In comparison:

  • R3 + Spatial: H1=1.627H^1=1.627, L2=0.733L^2=0.733.
  • R3 + Temporal: H1=1.714H^1=1.714, L2=0.669L^2=0.669.
  • R3 + Spatio-Temporal: H1=0.928H^1=0.928, L2=0.591L^2=0.591.

All three interpolation variants increase the L2L^2 error, while Spatial and Temporal Interpolation also substantially worsen the H1H^1 error. These results suggest that interpolation is not universally beneficial when combined with existing adaptive sampling methods. The authors should revise the corresponding claims and explicitly discuss the settings in which interpolation introduces performance degradation.

Questions for authors

see weaknesses above

7 comments

We sincerely thank the reviewer for the careful review of our paper. Please find our responses to each comment below.

W1. Hyperparameter selection

We thank the reviewer for pointing this out. In the revised manuscript, Table 1 reports the results for all configurations in the hyperparameter grid rather than the best configuration for each benchmark. Both spatial interpolation with ρ=0.3\rho=0.3 and temporal interpolation with p=0.01p=0.01 improve eH1e_{H^1} over random resampling on all six benchmarks.

W2. Gradient manipulating methods including RGA

We thank the reviewer for bringing AC-PKAN [1] to our attention. In response, we clarify the distinction between Residual Gradient Attention (RGA) in AC-PKAN and our analysis, and conduct additional experiments combining RGA and HARMONIC [2] with our interpolation strategies.

RGA is designed to mitigate loss imbalance, rather than stochastic gradient noise. Although RGA and our work both involve the residual gradient, they address distinct aspects of PINN optimization. Specifically, RGA combines residual-based attention, which assigns pointwise weights according to residual magnitudes, and gradient-related attention, which adjusts loss weights according to gradient norms. As described in AC-PKAN, this mechanism primarily aims to stabilize optimization, particularly in the presence of large gradients arising from the Chebyshev polynomial basis.

Our analysis instead concerns the resampling variance of the residual gradient. We characterize this variance across independently sampled collocation sets through the covariance Σr(θ)\Sigma_r(\theta), and its effect on spectral learning through the mode-wise SNR. RGA does not explicitly estimate or control this variance.

To further address this comment, we conducted additional experiments with gradient manipulating methods. On the Poisson problem, we applied RGA [1] and HARMONIC [2] under random resampling and combined each with spatial (ρ=0.5\rho=0.5) and temporal (p=0.01p=0.01) interpolation. Results are reported as the mean and standard deviation over five random seeds.

MethodSamplingeH1e_{H^1}eL2e_{L^2}
HARMONICRandom0.715±0.0020.715_{\pm 0.002}0.194±0.0250.194_{\pm 0.025}
+ Spatial0.552±0.0890.552_{\pm 0.089}0.698±0.2840.698_{\pm 0.284}
+ Temporal0.326±0.0760.326_{\pm 0.076}0.332±0.1340.332_{\pm 0.134}
RGARandom0.736±0.0090.736_{\pm 0.009}0.417±0.1070.417_{\pm 0.107}
+ Spatial0.511±0.1470.511_{\pm 0.147}0.607±0.2810.607_{\pm 0.281}
+ Temporal0.595±0.0910.595_{\pm 0.091}0.865±0.2120.865_{\pm 0.212}

Under random resampling, both methods yield eH1e_{H^1} close to that of the standard PINN, indicating that loss balancing or gradient manipulation alone does not resolve the high-frequency delay. For completeness, we cite AC-PKAN in the revised manuscript.


[1] Zhang et al., “AC-PKAN: Attention-Enhanced and Chebyshev Polynomial-Based Physics-Informed Kolmogorov–Arnold Networks,” Transactions on Machine Learning Research (TMLR), 2026.

[2] Bu et al., “Harmonized Cone for Feasible and Non-conflict Directions in Training Physics-Informed Neural Networks,” International Conference on Learning Representations (ICLR), 2026.

W3. Forcing amplitude and frequency

We thank the reviewer for the careful derivation and the valuable suggestion. As the reviewer points out, the forcing amplitude of the high-frequency mode grows as 2Cc222Cc_2^2. This growth follows from the Laplacian acting on a higher-frequency solution mode, so the solution frequency and the forcing amplitude are naturally coupled in the original setting. We agree that disentangling the two is valuable, and we conducted the suggested controlled experiment that keeps the forcing amplitude of the high-frequency mode fixed as c2c_2 varies.

With C0=0.1C_0 = 0.1, the suggested scaling C(c2)=C0(c1/c2)2C(c_2)=C_0(c_1/c_2)^2 makes the high-frequency amplitude very small, for example 10−310^{-3} at c2=40c_2 = 40. To remain aligned with the original experiment, we instead choose C0C_0 so that the case c2=40c_2 = 40 recovers the original setting C=0.1C = 0.1, which gives

C(c2)=0.1(40c2)2,2 C(c2) c22=320for all c2.C(c_2) = 0.1\left(\frac{40}{c_2}\right)^2, \qquad 2\,C(c_2)\,c_2^2 = 320 \quad \text{for all } c_2.

The table reports the gap between the low- and high-frequency capture times, with the mean and standard deviation over five random seeds.

c2c_2C(c2)C(c_2)Gap (iterations)
20200.4000.4001433±5011433_{\pm 501}
30300.1780.17810790±4,21410790_{\pm 4{,}214}
40400.1000.10039279±10,33839279_{\pm 10{,}338}
50500.0640.06446969±9,20346969_{\pm 9{,}203}
60600.0440.044122301±19,900122301_{\pm 19{,}900}

The gap still grows with c2c_2 when the high-frequency forcing amplitude is held fixed. Thus, the increase in the gap cannot be explained solely by the increase in forcing amplitude.

W4. Advanced PINN architectures

We thank the reviewer for the opportunity to clarify the scope of our work. We conducted additional experiments with SIREN and analyzed the residual gradient SNR for Fourier feature embeddings and SIREN.

Our goal is to investigate how random resampling affects spectral bias in vanilla PINNs, where spectral bias is well documented. As noted in the limitations of our original manuscript, extending the analysis to more challenging PDEs and architectures remains future work.

That said, we agree that it is important to examine whether our findings extend to advanced PINN architectures, in particular frequency-aware models designed to mitigate spectral bias. We therefore conducted the architecture study in Appendix F with additional experiments on SIREN, and analyzed the residual gradient SNR during training for Fourier feature embeddings with two bandwidths and for SIREN on the Helmholtz equation (Figure 11 in Appendix F). For σ=2\sigma=2, where spectral bias remains, the H1H^1 error stays nearly flat while the SNR is low, exhibiting the same noise-dominated regime as vanilla PINNs. For σ=6\sigma=6 and SIREN, where the architecture already mitigates spectral bias, the SNR is relatively high from early in training and the error decreases from the beginning.

These results show that the relation between the residual gradient SNR and error reduction also holds for the additional architectures. The delay under random resampling appears mainly when the architecture still exhibits spectral bias. When the architecture already mitigates spectral bias, the delay becomes weaker, and interpolation has less room to improve high-frequency recovery. We also note that our work focuses on analyzing how random resampling affects spectral bias, rather than proposing interpolation as a method that applies to all architectures. The interpolation strategies serve to show that this effect can be mitigated by simple changes to collocation sampling.

W5. Combination with existing sampling methods

Thank you very much for your careful comments. We agree that the conclusions in Appendix G.8 in original manuscript overstated the results, and we have revised them in Appendix H.8 of the revised version. Since R3 and LAS already retain part of the collocation set across iterations, combining them with interpolation can make the collocation set overly static in some settings. We added this discussion to Appendix H.8.

Thank you for the authors' response. For W3, the central concern is whether random resampling exacerbates spectral bias beyond the network's intrinsic difficulty in learning high-frequency modes. The increasing capture-time gap under random sampling does not establish this, as fixed sampling may exhibit the same trend. The relevant quantity is ΔG(k)=Grandom(k)−Gfixed(k)\Delta G(k)=G_{\mathrm{random}}(k)-G_{\mathrm{fixed}}(k), where Gs(k)=τhs(k)−τls(k)G_s(k)=\tau_h^s(k)-\tau_l^s(k). Moreover, gradient covariance and mode-wise SNR under constant forcing remain unexamined, leaving the covariance attribution unresolved. Since Ah∝k−2A_h\propto k^{-2} also reduces high-frequency amplitudes, absolute capture thresholds introduce potential confounding. I encourage the authors to compare Fixed, Random, and Interpolation sampling under constant forcing, report ΔG(k)\Delta G(k) using amplitude-normalized capture criteria, and examine gradient covariance and mode-wise SNR at matched model states. These controls are essential to distinguish intrinsic spectral bias, resampling-induced delays and PDE operator scaling. For W4, the correlation between SNR and error reduction does not establish interpolation's effectiveness across architectures. The additional experiments mainly characterize training dynamics without directly demonstrating improvements over random resampling in frequency-aware models. SIREN and high-bandwidth Fourier Features may already mitigate spectral bias, potentially limiting or even reversing interpolation's marginal benefit. This architectural dependence raises concerns about general applicability. I encourage controlled comparisons of all three sampling strategies on SIREN and high-bandwidth Fourier Features, reporting final errors, mode-wise SNR and capture times to establish whether interpolation provides consistent benefits beyond vanilla PINNs.

Comments on the paper

1 comment

We thank all the reviewers and the Area Chair for the time and care taken with our submission. We have addressed each weakness and question raised by the reviewers and ensured these changes are included in the revised manuscript.

Our main contributions are as follows.

  • Discovery. We show that random resampling of collocation points can amplify spectral bias in PINN training.
  • Understanding. A residual gradient SNR analysis identifies a noise-dominated regime that delays high-frequency learning.
  • Mitigation. Guided by this analysis, we show that a simple interpolation between fixed and random resampling can mitigate this amplification.

We summarize the key updates made during the rebuttal process below.

Hyperparameter selection. (Reviewer 88bF, ispe)

We have revised the main results table (Table 1) to report all evaluated hyperparameter configurations instead of only the best configuration. The reported results therefore no longer depend on the exact solution for model selection.

Frequency-aware architectures. (Reviewers 88bF, ispe)

We added SIREN to the architecture study and analyzed the residual gradient SNR for Fourier feature embeddings with two bandwidths and for SIREN (Appendix F). With a moderate bandwidth, where spectral bias remains, a similar noise-dominated regime appears as in vanilla PINNs. With a large bandwidth and SIREN, which already mitigate spectral bias, the SNR is relatively high from early in training. These results are consistent with our analysis rather than limiting it.

Loss balancing and gradient manipulation methods (including RGA). (Reviewer 88bF)

We clarified that Residual Gradient Attention (RGA) proposed in AC-PKAN, which the reviewer mentioned, addresses a different aspect of PINN training from our stochastic analysis. RGA reweights loss terms and collocation points according to gradient norms and residual magnitudes to alleviate loss instability and imbalance, and can also be applied under fixed sampling, where there is no stochastic gradient noise. We further conducted experiments with gradient manipulation methods including RGA, in which these methods alone do not resolve the high-frequency delay under random resampling, while combining them with interpolation reduces eH1e_{H^1}.

Theoretical analysis of bias and variance. (Reviewer ispe, wX1r)

We have derived explicit expressions for the gradient bias and variance under spatial interpolation and characterized the temporal correlations induced by collocation reuse in Appendix G. These results provide further theoretical insight into the bias-variance trade-off underlying our proposed strategies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.