- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper concerns PINN training methods for multiscale physics problems. It analyses how collocation point sampling strategies, fixed sampling and random resampling, affect the recovery of low- and high-frequency components in the solution, primarily through a set of experiments. It is shown that fixed sampling identifies high-frequency components faster than random resampling, but at the cost of bias in low-frequency components. Under random resampling, the variance of the residual gradient and the time-to-identification of a modal component are both positively correlated with the mode frequency and negatively correlated with the sample size. Further, the signal-to-noise ratio (SNR) of the residual gradient is negatively correlated with both the H1 seminorm error and the modal error. Based on these findings, the paper suggests using a mix of fixed sampling and random resampling, where part of the collocation sample is fixed during training and part of it is resampled at each iteration. The approach is experimentally validated on a set of benchmark problems and compared to related methods.
Critical strengths
The motivation and analysis developed through the series of experiments is logically sound and comprehensive, starting with why PINNs behave differently to regression when it comes to mode recovery under fixed and random resampling, subsequently analysing the relationship between delayed mode recovery and residual gradient variance, and finally analysing the relationship between SNR and mode recovery.
In light of these experiments, the idea of using a mix of a fixed and randomly resampled set of collocation points is logical. The method achieves strong experimental results on a set of standard benchmark problems in PINNs as compared to fixed sampling, random resampling, and a benchmark given by importance sampling. The method can be combined with other sampling methods (LAS and R3), achieving improvements in performance.
Critical weaknesses
While the paper provides ample experimental support, I am missing a theoretical analysis of the proposed method. For the spatial, temporal, and spatio-temporal interpolation strategies between fixed sampling and random resampling, could you provide theoretical insight into how they change the bias and variance of the residual gradient? Could you provide insight into how to choose \rho and p based on these results, perhaps connecting this to the adaptive interpolation scheme proposed in Appendix G.7?
When I look at the experiments in Appendix G.6 illustrating the sensitivity of the method to the choice of resampling fraction \rho or resampling probability p, it seems like low values of these give the best balance between low bias in the low-frequency component and decent time-to-identification in the high-frequency component (\rho=0.1 in Fig 18, p=0.01 in Fig 19). I would appreciate an analysis of why this is the case, especially considering the result in Figure 1, where a fixed collocation sample is associated with strong visible bias, while random resampling gives a much better L2 reconstruction error (while visibly missing high-frequency content). It would be great if you could give theoretical insight into this observation.
Questions for authors
In Fig 3, you observe a bias only for the lowest number of collocation points. Can you discuss this in relation to Fig 1, where all solutions have strong visible bias for the fixed sampling method? What would happen in Fig 1 if you increased the sample size?
Since the bias caused by a fixed collocation sample is related to overfitting, and the culprit in random resampling is poor SNR of the residual gradient, I am wondering how your method compares to dropout [Ref 1]. This method uses a Bayesian neural network parametrization to overcome overfitting, and a local reparameterization trick to provably reduce the variance of the loss gradient. Is it applicable, and how would the results compare to your method on this problem?
[1] Kingma, D. P., Salimans, T., & Welling, M. (2015). Variational dropout and the local reparameterization trick. Advances in Neural Information Processing Systems, 28.
3 comments
W1. Theoretical analysis of interpolation methods
Thank you for the extended theoretical analysis. The conditional bias-variance decomposition clarifies your insights and provides a valuable addition to the work, in my view.
I checked the added computations in Appendix G, and I believe they are correct. For improved clarity, maybe you can modify “Since contains ,…” to instead say how the variance scales relative to , as you do in the spatio-temporal computation, but it is not critical. Just watch out for typos “Eq.eq.” between eqs. 37 and 38 as well as 43 and 44.
W2. Why small and can be effective
Thank you for the clarification – I understand the point.
Q1. Large sample size with fixed sampling
Thank you for providing new figures. Since the limited collocation budget exaggerates the issue here, I suggest that you add a small comment in the main text discussion on Figure 1.
Q2. Comparison with variational dropout
Thank you for your comment and added experiments, and I agree with your point.
In light of the authors’ revisions and responses, I raise my score to 4. Clear acceptance.
We appreciate the reviewer's positive assessment. We have corrected the noted typos in Appendix G and added a clarification regarding the limited collocation budget in Section 3.2. Thank you again for your thoughtful engagement.
We sincerely thank the reviewer for the careful review of our paper. Please find our responses to each comment below.
We thank the reviewer for this excellent suggestion. Following it, we added a bias-variance analysis of the interpolation schemes in Section 5.2 (Table 2), with derivations in Appendix G. The analysis shows that, at a fixed θk, interpolation does not change the total gradient error but redistributes it between conditional bias and conditional variance. Spatial interpolation introduces a bias that persists throughout training, whereas temporal interpolation replaces the reused set after 1/p iterations on average.
Regarding the interesting question about how to choose ρ and p, this analysis alone does not determine their optimal values, because the optimal split depends on the training dynamics. Nevertheless, it provides a rationale for the adaptive scheduler, which reduces ρ (or p) only when the residual gradient SNR falls below 1 and thus trades variance for bias only in the noise-dominated regime.
Thank you for this insightful observation.
Suppose the network can represent the exact solution, that is, there exists θ⋆ with Lr(θ⋆)=0. Under fixed sampling, a parameter satisfying Lr(θ;Xr)=0 does not necessarily minimize the population loss Lr(θ), since the PDE may be satisfied only at the selected collocation points. This is consistent with the global distortion in Figure 1.
In contrast, spatial interpolation minimizes the expected objective (1−ρ)Lr(θ;XF)+ρLr(θ) given XF. Since both terms are nonnegative and vanish at θ⋆, any global minimizer with ρ>0 must also satisfy Lr(θ)=0. Hence, if this expected objective is minimized well, the collocation bias of fixed sampling can be removed. Since this holds for any ρ>0, it provides theoretical intuition for why even a small ρ can be effective. Temporal interpolation follows a similar intuition, as repeated refreshes introduce new collocation points and allow residual errors outside previously sampled points to be corrected.
Thank you for this question. To answer it, we repeated the experiments in Figure 1 with fixed sampling using 10 times more collocation points and added the results to Appendix H.1 of the revised manuscript. With the sufficiently larger collocation set, fixed sampling recovers the solutions without visible distortion. This indicates that the bias in Figure 1 arises from the limited collocation budget, consistent with Figure 3, where fixed sampling exhibits bias only for the small Nr.
We thank the reviewer for this interesting suggestion. The analogy is natural, since collocation bias resembles overfitting to a fixed training set and our diagnosis concerns gradient variance. We would like to kindly note, however, that the variance reduced by the local reparameterization trick differs from the variance we study. The trick reduces the variance induced by sampling the stochastic weights, by making this weight noise independent across data points. The resampling variance in our setting instead arises from sampling the collocation points themselves and is present even for a deterministic network.
To examine this empirically, we applied Gaussian dropout with the local reparameterization trick following [1], with the dropout rate kept constant during training. We trained on the Poisson problem under fixed sampling and random resampling for several dropout rates pequiv.
In our experiments, we did not observe that dropout mitigates the collocation bias of fixed sampling or the high-frequency delay under random resampling.
[1] Kingma, D. P., Salimans, T., & Welling, M. (2015). Variational dropout and the local reparameterization trick. Advances in Neural Information Processing Systems.