acceptodds
Under review as a conference paper at ICLR 2027

RAMPing Up PINN Training Signals via Parameter-Sensitivity-Guided Collocation Refinement

Abstract

Collocation point selection plays a crucial role in training physics-informed neural networks (PINNs), as physics constraints are enforced only at sampled locations. Most existing adaptive sampling methods prioritize points with large constraint violations, typically measured by PDE residual magnitude, thereby implicitly treating residual magnitude as a sufficient pointwise indicator of training utility. We revisit adaptive collocation from a parameter-space perspective and introduce *pointwise parameter sensitivity*, defined as the norm of the gradient of the pointwise PDE loss with respect to the trainable parameters. We show that sampling in proportion to this quantity directly amplifies the induced parameter-space training signal, up to a directional coherence factor. Building on this criterion, we propose **R**adius-bounded **A**scent for **M**aximizing **P**arameter-gradient norm (**RAMP**), a parameter-sensitivity-guided collocation refinement strategy that adapts sampled points to the current parameter state. Given candidates drawn from a prescribed base sampler, RAMP independently refines each point by projected unit-gradient ascent on pointwise parameter sensitivity within a bounded radius. This local refinement preserves the spatial coverage induced by the base sampler while steering collocation points toward regions that provide stronger parameter-space signals. We further analyze magnitude-based sampling criteria and show that they can be viewed as residual-only approximations of parameter sensitivity, which neglect the local parameter responsiveness of the PDE residual. Across benchmark PDEs and multiple PINN optimizers, RAMP achieves consistently larger aggregate gains over the base sampler than recent adaptive sampling methods, while retaining parallelizable point updates.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Physics-Informed Neural Network, Adaptive sampling, Training signals, Pointwise parameter sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hGveOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new collocation sampling method, called Radius-bounded Ascent for Maximizing Parameter-gradient norm (RAMP). The main idea of RAMP is to use the norm of the pointwise PDE residual loss gradient with respect to the model parameters as a signal for resampling collocation points, rather than relying solely on the residual loss magnitude. The proposed method is evaluated on a range of benchmark equations and consistently outperforms several baseline sampling strategies. The authors further demonstrate that RAMP can be effectively combined with different optimization algorithms and loss balancing methods, suggesting that the proposed sampling strategy is broadly applicable across different PINN training settings.

Critical strengths

None

Critical weaknesses

Unfair comparison. Using the residual gradient norm as a sampling signal is reasonable, but it introduces additional computational cost due to pointwise gradient evaluations. The current manuscript does not analyze this overhead, making the comparison with simpler baselines potentially unfair.

The authors should report both memory usage and wall-clock time. For example, under a matched memory budget, random resampling could use more collocation points instead of computing pointwise gradient norms. If such a baseline achieves comparable or better performance, the practical advantage of RAMP would be less clear.

This issue is particularly important because the main contribution of the paper is largely empirical rather than theoretical.

Presentation. The manuscript also requires careful revision for clarity. The terminology and mathematical notation are not always used consistently, and several parts are unnecessarily difficult to follow.

Related works. The paper should discuss a broader range of literature, particularly prior work on importance sampling. This would help better position RAMP with respect to existing approaches.

Questions for authors

Can the authors provide a fair comparison with the baselines in terms of computational cost? In particular, it would be helpful to report memory usage and wall-clock time, and to evaluate whether RAMP still outperforms random resampling under a matched memory budget.

8 comments

Weakness 3: Related works

We sincerely thank the reviewer for highlighting the importance of positioning RAMP more clearly with respect to existing importance sampling methods. We agree that our original presentation did not sufficiently distinguish the two approaches, particularly because both prioritize samples that induce strong parameter-space training signals. We greatly appreciate this observation, which motivated us to clarify their fundamental differences and further strengthen our theoretical analysis.

To address this concern, we have added Appendix B.1. Full-Domain Training Dynamics, which explicitly analyzes RAMP from a bias–variance perspective and compares it with gradient-norm importance sampling.

Both approaches emphasize pointwise parameter sensitivity, but their underlying mechanisms and objectives differ. Conventional importance sampling uses inverse-probability weighting to preserve an unbiased estimate of the reference gradient while reducing its variance. In contrast, RAMP transports uniformly sampled points through radius-bounded sensitivity ascent and aggregates their gradients without importance weighting, thereby introducing a sampling-induced bias.

Our new analysis demonstrates how this bias can contribute positively to full-domain optimization. Its parallel component strengthens the descent signal and can yield a strictly lower expected one-step full-domain loss under explicit conditions. Its orthogonal component can further improve subsequent parameter sensitivity and two-step loss through favorable curvature alignment.

These distinctions are also supported by the experiments in Appendix B.2. At ε=0.002\varepsilon=0.002, RAMP increases gradient-estimation variance by 0.3%0.3\%–1.8%1.8\%, whereas ideal sensitivity-proportional importance sampling reduces variance by factors of 1.401.40–1.691.69. Nevertheless, RAMP achieves improved one-step full-domain loss through its sampling-induced bias.

We are particularly grateful for the reviewer's suggestion, as it helped us identify an important omission in our original presentation and establish a clearer distinction between RAMP and existing sampling approaches. We believe the additional analysis substantially improves the positioning and theoretical understanding of RAMP, and we sincerely hope it addresses the reviewer's concern regarding related works.

Weakness 2: Presentation Unclarity

We sincerely thank the reviewer for this valuable comment. After carefully revisiting the manuscript in light of your feedback, we identified several instances of inconsistent terminology and mathematical notation, as well as expressions that could have been presented more clearly.

Revision in the manuscript

We have carefully revised these issues throughout the manuscript, with all changes highlighted in blue in the revised version. We would greatly appreciate it if the reviewer could take another look at these revisions.

We apologize for any confusion caused by the original presentation and sincerely thank the reviewer once again for this constructive feedback, which has helped us improve the clarity and consistency of our manuscript.

Weakness 1 & Question 1: Computational Cost

We thank the reviewer for raising this important concern. We agree that evaluating RAMP's practical effectiveness requires fair comparisons that account for both runtime and memory usage.

Wall-Clock-Matched Performance Comparison

For each PDE and optimizer, we evaluated performance under an estimated wall-clock time budget equivalent to that required to train RAMP for 20,000 epochs. We summarize the results using the same metrics as Table 1 of the manuscript: mean rank (MR) and aggregate relative improvement (Δ%\Delta\%).

Under the same wall-clock time budget, RAMP performs 1.38–7.04 times fewer parameter updates under Adam, 1.25–3.51 times fewer under LRA, and 1.21–4.11 times fewer under HARMONIC compared with the other methods across the 12 PDEs. The detailed numbers of training epochs are provided in Table 10 of Appendix D.8.

As shown in the table below, RAMP achieves competitive performance despite performing substantially fewer parameter updates than the other methods. These results provide further empirical support for our central hypothesis that locally maximizing pointwise-parameter-sensitivity enables RAMP to identify collocation points that induce stronger parameter-space training signals, thereby facilitating more effective error reduction even with substantially fewer parameter updates.

However, the number of parameter updates differs considerably across methods, by up to a factor of approximately 7 under Adam and nearly 4 under LRA and HARMONIC. To reduce this discrepancy, we introduce a computationally efficient variant of RAMP, denoted RAMP-P. It retains parameter-sensitivity-guided collocation refinement while reducing the frequency of refinement-direction computations.

Unlike the original RAMP, which computes the refinement direction di(k,s)d_i^{(k,s)} at every training iteration kk, RAMP-P computes di(k,s)d_i^{(k,s)} once every PP iterations. The resulting direction is then reused to progressively move each collocation point in equal increments over the subsequent PP iterations. Since a key advantage of RAMP is its ability to adapt collocation points to the current training state at every iteration, choosing a large PP could reduce its responsiveness to changes in the training state. We therefore set P=5P=5 and conducted additional experiments.

MethodAdam MR ↓Adam Δ%\Delta\% ↑LRA MR ↓LRA Δ%\Delta\% ↑HARMONIC MR ↓HARMONIC Δ%\Delta\% ↑
R34.42 (1.68)-37.03 (18.19)3.83 (1.34)+19.89 (7.12)5.17 (0.94)-78.42 (18.43)
RAD3.75 (1.36)-25.31 (14.38)3.25 (1.71)+27.19 (6.44)3.50 (1.68)+18.97 (14.20)
LAS3.92 (1.56)-26.89 (9.76)2.75 (1.76)+40.00 (1.04)3.00 (1.76)+17.78 (12.04)
PINNACLE3.08 (1.68)-7.69 (9.48)4.08 (1.62)+24.47 (9.49)3.00 (2.04)+18.78 (6.63)
RAMP3.33 (2.31)+6.32 (1.17)4.00 (2.22)+2.81 (1.37)3.50 (1.17)+19.57 (2.13)
RAMP-P2.50 (1.17)+8.12 (5.35)3.08 (1.44)+40.85 (3.79)2.83 (1.64)+35.98 (4.25)

As shown in the table, RAMP-P achieves the best performance in both MR and Δ%\Delta\% under Adam and HARMONIC, while achieving the highest Δ%\Delta\% under LRA. Under LRA, RAMP-P also achieves the second-lowest MR.

Revision in the manuscript

To address the reviewer's concern, we have expanded Appendix D.8 with the following results:

  • Table 9 (p.39): Computational cost per 100 epochs, including 95% confidence intervals, with the corresponding profiling procedures and numbers of repeated measurements.
  • Table 10 (p.40): PDE- and optimizer-wise training epoch allocations under computational budgets equivalent to 20,000 epochs of the RAMP.
  • Table 11 (p.41): PDE-wise relative L2L_2 errors under the matched computational budgets, including 95% confidence intervals where available.
  • Table 12 (p.42): Aggregate performance comparison in terms of MR and Δ%\Delta\%, corresponding to the summary table presented above.
Reviewer_GJeDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to study how the collocation points for PINN training can be better selected to improve the training convergence of PINNs. This is done so through the lens of parameter sensitivity, or trying to quantify the magnitude of potential parameter update given the training collocation points. The paper shows that this quantity is related to the norm of the loss gradient, which in turn will depend on the coherence of the individual gradient wrt each collocation point and also the magnitude of those individual gradient. This give rise to a weighted sampling method for collocation points, which can be used with radius-bounded ascent to make sure the sampled points do not drift away from the current distribution of training points too much. In the results, RAMP shows improvements over other methods that sample points based on the magnitude of residual or based on some NTK-related quantities, across a variety of PINN training problems.

Critical strengths
  1. The paper attempts to tackle a real problem of collocation point selection. It provides a good viewpoint that the sensitivity of the PINN parameter can be used to perform collocation points selection, and provides a clean implementation of such. It also attempts to verify it on a good, wide set of PDEs.

  2. The paper is well-written, and structured well. I find the theoretical result to be stated quite clearly, and the setup to be easy to understand and to follow for the reader. I would assume it is easy enough to follow even for people who are not super familiar with PINNs already (however cannot verify on this fact).

Critical weaknesses
  1. I have some small issues regarding the theoretical results in Section 3.

    First, while I think the results in Section 3.1 and 3.2 is interesting insight as to how the PINN training is, I want more discussion about how it compares with some of the existing work. Thm 1 seems quite a standard result for gradient-based training (if I understand correctly), not necessarily specific to PINNs. Eq 7 considers the inner product of loss gradient which seems to be closely related to NTK-based analysis for PINN training from prior works (which the author also cited in the paper). Parameter sensitivity seems to be a common viewpoint possibly in NN training dynamics. A bigger highlight as to why the theoretical results (and to an extent the proposed method) shown in the paper here is different from existing ones would highlight the contributions more clearly.

    Additionally, the extent of how much the theoretical results hold is unclear. More discussion can be made for Thm 1 as to whether PDEs may give loss landscapes which are beta-smooth, or what the magnitude of directional coherence in practice may be (this seems partially discussed in Appx A), or how much would radius-bounded ascent affect the results in some of the theorems. I understand that there could be a drift between theoretical results and the empirical findings, but it might be a good discussion to make regardless.

  2. I also have some concerns regarding reporting the loss as a function of number of training iterations directly, rather than as a function of some computation cost (i.e., training time). I see that in Table 5 there is a report on how long each algorithm takes to run (which is good), but it does also show that RAMP does in general take longer time per training step than some of the other algorithms. This may mean that in actual computation time, other algorithms may be able to achieve as good or better results when given equal training wall clock time (i.e., the cheaper algorithms can run more training steps to make it comparable). The result in Table 1 obfuscates this analysis since it only shows the training after 20k training steps, but doesn't discuss how long to perform that training, or how it would be if the training wasn't performed that long (since the graphs for the convergence also is not shown).

    For this part, I guess the authors should discuss more about either why we could solely consider the empirical performance wrt training step alone as a practical measure of performance, or present a more complete finding for Table 1 which discusses the effect when considering the actual computational time as well.

Questions for authors
  1. How does RAMP perform compared to other algorithms when given equal training time as other algorithms?

  2. How related or different are the theoretical analysis in the paper compared to the other existing works in PINN training convergence (or to NN training convergence in general)?

4 comments

Weakness 1 & Question 2: Comparison with other existing work

We sincerely thank the reviewer for the exceptionally thorough and thoughtful review. We greatly appreciate the careful examination of our theoretical results and their connections to existing literature. We agree that our original manuscript did not sufficiently clarify these connections or examine the practical validity of the theoretical assumptions. Your feedback has helped us substantially strengthen both aspects in the revised manuscript.

1. Relationship to existing theoretical analyses

We agree that Theorem 1 follows from the standard smoothness-based descent bound. Our intention is to use this established result as the starting point for interpreting parameter sensitivity as a measure of the training state. We have revised Section 3.1 to make this progression clearer.

We also appreciate the reviewer's observation regarding the connection between Eq. (7) and NTK-based analyses. Both involve parameter-gradient interactions, but they differ in their computational objectives. NTK-based approaches, such as PINNACLE, characterize training dynamics through pairwise kernel interactions and spectral information. These computations can be expensive in both memory and time, motivating approximations and periodic kernel updates. In contrast, RAMP seeks a pointwise criterion that can be evaluated independently and in parallel for each candidate, allowing the sampling decisions to reflect the current training state at every iteration without constructing an empirical NTK.

We have strengthened the discussion of these connections in Section 3.1 to clarify the motivation for our pointwise formulation.

2. Additional theoretical analysis of RAMP

More importantly, we acknowledge that the original theoretical results primarily motivated our sampling criterion without directly characterizing the proposed refinement algorithm.

To address this, we have added Appendix B.1. Full-Domain Training Dynamics, which analyzes the sampling distribution induced by RAMP and its effect on the original full-domain PDE loss. This analysis establishes a sufficient condition under which RAMP achieves strictly lower expected post-update full-domain loss than uniform sampling. It further decomposes RAMP's sampling-induced gradient bias into two components.

  • Parallel bias strengthens the descent signal along the full-domain gradient, improving one-step loss reduction under explicit conditions.

  • Orthogonal bias changes the parameter-update direction while preserving the first-order loss decrease. We derive curvature-based conditions under which this displacement increases subsequent parameter sensitivity and improves two-step loss.

These results also clarify RAMP's relationship to other parameter-sensitivity-based approaches. Unlike conventional importance sampling, which emphasizes unbiased gradient estimation and variance reduction, RAMP can exploit a beneficial sampling-induced bias. The orthogonal component further exhibits a connection to parameter teleportation through its effect on subsequent training dynamics.

3. Empirical validity of the theoretical conditions

We have also added Appendix B.2. Empirical Verification of Full-Domain Dynamics, to examine the assumptions and conditions highlighted by the reviewer.

Using five Burgers1D training states, we measure the local smoothness constant, directional coherence, and the effect of the refinement radius.

  • The estimated local smoothness constants satisfy βloc=6.69\beta_{\mathrm{loc}}=6.69–7.997.99, giving ηβloc=0.00669\eta\beta_{\mathrm{loc}}=0.00669–0.007990.00799 at η=10−3\eta=10^{-3}.

  • The measured directional coherence remains substantial (κZ=0.440\kappa_{\mathbf{Z}}=0.440–0.6610.661) and is nearly unchanged under small-radius refinement.

  • The sufficient condition for expected full-domain improvement is satisfied at all five states for ε≤0.02\varepsilon\leq0.02 and at four states for ε=0.2\varepsilon=0.2.

  • Direct evaluations show positive one-step full-domain improvements across all 25 tested combinations of training states and refinement radius.

We additionally verify the curvature conditions governing the orthogonal component, finding favorable alignment and corresponding two-step improvements at four of the five states.

We are particularly grateful that the reviewer's detailed observations encouraged us to clarify the distinction between RAMP and existing parameter-sensitivity approaches while developing a more complete theoretical analysis of the proposed algorithm itself. We believe these additions significantly strengthen the theoretical foundation and presentation of our work, and we sincerely hope they address the reviewer's concerns.

Weakness 2 & Question 1: Computational Cost

We thank the reviewer for raising this important concern. We agree that comparing methods solely after a fixed number of training iterations does not fully account for differences in computational cost.

Wall-Clock-Matched Performance Comparison

For each PDE and optimizer, we evaluated performance under an estimated wall-clock time budget equivalent to that required to train RAMP for 20,000 epochs. We summarize the results using the same metrics as Table 1 of the manuscript: mean rank (MR) and aggregate relative improvement (Δ%\Delta\%).

Under the same wall-clock time budget, RAMP performs 1.38–7.04 times fewer parameter updates under Adam, 1.25–3.51 times fewer under LRA, and 1.21–4.11 times fewer under HARMONIC compared with the other methods across the 12 PDEs. The detailed numbers of training epochs are provided in Table 10 of Appendix D.8.

As shown in the table below, RAMP achieves competitive performance despite performing substantially fewer parameter updates than the other methods. These results provide further empirical support for our central hypothesis that locally maximizing pointwise-parameter-sensitivity enables RAMP to identify collocation points that induce stronger parameter-space training signals, thereby facilitating more effective error reduction even with substantially fewer parameter updates.

However, the number of parameter updates differs considerably across methods, by up to a factor of approximately 7 under Adam and nearly 4 under LRA and HARMONIC. To reduce this discrepancy, we introduce a computationally efficient variant of RAMP, denoted RAMP-P. It retains parameter-sensitivity-guided collocation refinement while reducing the frequency of refinement-direction computations.

Unlike the original RAMP, which computes the refinement direction di(k,s)d_i^{(k, s)} at every training iteration kk, RAMP-P computes di(k,s)d_i^{(k, s)} once every PP iterations. The resulting direction is then reused to progressively move each collocation point in equal increments over the subsequent PP iterations. Since a key advantage of RAMP is its ability to adapt collocation points to the current training state at every iteration, choosing a large PP could reduce its responsiveness to changes in the training state. We therefore set P=5P=5 and conducted additional experiments.

MethodAdam MR ↓Adam Δ%\Delta\% ↑LRA MR ↓LRA Δ%\Delta\% ↑HARMONIC MR ↓HARMONIC Δ%\Delta\% ↑
R34.42 (1.68)-37.03 (18.19)3.83 (1.34)+19.89 (7.12)5.17 (0.94)-78.42 (18.43)
RAD3.75 (1.36)-25.31 (14.38)3.25 (1.71)+27.19 (6.44)3.50 (1.68)+18.97 (14.20)
LAS3.92 (1.56)-26.89 (9.76)2.75 (1.76)+40.00 (1.04)3.00 (1.76)+17.78 (12.04)
PINNACLE3.08 (1.68)-7.69 (9.48)4.08 (1.62)+24.47 (9.49)3.00 (2.04)+18.78 (6.63)
RAMP3.33 (2.31)+6.32 (1.17)4.00 (2.22)+2.81 (1.37)3.50 (1.17)+19.57 (2.13)
RAMP-P2.50 (1.17)+8.12 (5.35)3.08 (1.44)+40.85 (3.79)2.83 (1.64)+35.98 (4.25)

As shown in the table, RAMP-P achieves the best performance in both MR and Δ%\Delta\% under Adam and HARMONIC, while achieving the highest Δ%\Delta\% under LRA. Under LRA, RAMP-P also achieves the second-lowest MR.

Revision in the manuscript

To address the reviewer's concern, we have expanded Appendix D.8 with the following results:

  • Table 9 (p.39): Computational cost per 100 epochs, including 95% confidence intervals, with the corresponding profiling procedures and numbers of repeated measurements.
  • Table 10 (p.40): PDE- and optimizer-wise training epoch allocations under computational budgets equivalent to 20,000 epochs of the RAMP.
  • Table 11 (p.41): PDE-wise relative L2L_2 errors under the matched computational budgets, including 95% confidence intervals where available.
  • Table 12 (p.42): Aggregate performance comparison in terms of MR and Δ%\Delta\%, corresponding to the summary table presented above.

We appreciate the reviewer's constructive suggestion, which motivated a more comprehensive evaluation of RAMP's computational efficiency.

I thank the author for the additional results. I do see that in Table 11 that the results seem to be less favourable towards RAMP, which suggests that there is still a gap in the empirical performance of the method in practical use cases. While I appreciate some clarifications on the theoretical results, I am unsure if they distinguish enough from the existing work for it to be moved to an acceptance based on it alone. Therefore I will keep my score as is.

Thank you again for carefully considering our additional results and for clarifying your remaining concerns. We agree that practical performance should account for the cost of collocation refinement. We would like to clarify the benchmark-wide interpretation of Table 11 and its aggregate summary in Table 12, which together provide evidence of practical benefits beyond the fixed-iteration comparison.


Performance under the matched computational budget. Table 12 summarizes all 12 PDEs under computational budgets matched to the estimated wall-clock time of 20,000 default-RAMP iterations. The competing methods—R3, RAD, LAS, and PINNACLE—are themselves adaptive sampling methods developed for PINNs. The reported aggregate relative improvements are summarized below.

OptimizerDefault RAMP Δ%\Delta\% ↑Best existing sampler Δ%\Delta\% ↑RAMP-P Δ%\Delta\% ↑
Adam+6.32-7.69 (PINNACLE)+8.12
LRA+2.81+40.00 (LAS)+40.85
HARMONIC+19.57+18.97 (RAD)+35.98

Even default RAMP retains a higher aggregate Δ%\Delta\% than all four existing samplers under Adam and HARMONIC. RAMP-P achieves the highest reported Δ%\Delta\% under all three optimizers. It also achieves the lowest mean rank under Adam and HARMONIC, and the second-lowest under LRA.

Using the common reference in the definition of Δ%\Delta\%, RAMP-P's results correspond to approximately 14.7%, 1.4%, and 21.0% lower geometric-mean errors than the strongest existing sampler under Adam, LRA, and HARMONIC, respectively. Thus, it is close to LAS under LRA and shows larger aggregate gains under Adam and HARMONIC. These comparisons use the entire benchmark and already account for the additional refinement cost.

The sampling criterion and its refinement schedule. We acknowledge that the stronger fixed-iteration results do not translate unchanged into the matched-time comparison. However, this reveals a trade-off between the effectiveness of each parameter update and the cost of refreshing the refinement direction. At equal iteration counts, default RAMP achieves the lowest MR and highest Δ%\Delta\% under all three optimizers in Table 1. RAMP-P retains the same parameter-sensitivity criterion while computing the refinement direction once every five iterations and reusing it to move points progressively. With the same P=5P=5 across the evaluated problems, this schedule produces the matched-budget results above.

Taken together, these results support both the effectiveness of the sensitivity-based criterion and the practical value of amortizing its computation. We therefore believe the additional experiments establish more than improved accuracy at the expense of greater computation. They also demonstrate competitive or improved benchmark-wide performance when computation is explicitly accounted for.

Distinction from existing theoretical analyses. Appendix B complements this evidence by analyzing the actual RAMP-induced gradient estimator against the uniform full-domain objective. Under the stated conditions, its sampling-induced bias can improve one-step descent through its parallel component and subsequent optimization dynamics through its curvature-dependent orthogonal component. This directly analyzes RAMP's unweighted refinement, beyond the standard descent bound used for motivation, and distinguishes its mechanism from gradient-norm importance sampling aimed at variance reduction with an unbiased estimator.


We sincerely appreciate the effort you have devoted to this discussion. Your feedback prompted a more informative evaluation of the relationship between sampling effectiveness and computational cost. We hope that considering the benchmark-wide matched-budget results alongside the revised analysis supports a reconsideration of RAMP's practical and methodological contribution.

Reviewer_kJg6OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The selection of collocation points is crucial in the training of PINNs. Among existing methods, adaptive approaches to sampling points based on residuals has garnered attention. However, even when attempting to update the solution values at points with large residuals, it is not always possible to successfully adjust the neural network parameters so that the residuals are certainly reduced. In this paper, the authors propose incorporating the sensitivity of the solution at each point with respect to the network parameters into the selection of collocation points. They also propose a computationally efficient method based on this approach.

Critical strengths
  • As far as I know, incorporating the above sensitivity analysis into the selection of collocation points for PINNs is new.
  • Some theoretical analysis has been conducted, and the method is motivated by the results.
  • The paper is easy to read.
Critical weaknesses

The theory underlying this paper appears to be inappropriate.

  • Although a theory background is provided, it merely serves to motivate the method; no theoretical analysis of the proposed algorithm itself has been conducted. Consequently, the proposed algorithm remains purely heuristic and does not have any theoretically guaranteed properties.
  • Regarding the theory that motivates the method, Theorem 2 evaluates SPψS_{P_\psi}; in this sense, it analyzes the behavior of the solution when a loss function weighted by PψP_\psi is used. It is necessary to guarantee that this yields a better solution than the loss function defined using QQ.
  • Furthermore, Theorem 1 is a well-known fact, and it is not appropriate to regard it as a theoretical contribution (See, e.g., A. Beck, Introduction to Nonlinear Optimization: Theory, Algorithms, and Applications with MATLAB, SIAM, 2014).
Questions for authors

See Weakness.

4 comments

We sincerely thank the reviewer for the careful assessment and insightful criticism of our theoretical justification. We agree that the original submission did not sufficiently analyze RAMP itself. The theoretical results primarily motivated the sampling criterion, leaving the connection between the proposed algorithm and full-domain training improvement unclear. Your feedback prompted us to substantially strengthen this aspect of the paper. We have therefore added Appendix B. Additional Theoretical Analysis of RAMP, to the revised manuscript.

We address the three concerns as follows.

1. Theorem 1 and its contribution. We agree that the descent inequality in Theorem 1 is a well-established result. Our intention was to use this standard result as the starting point for interpreting parameter sensitivity as a measure of the training state. We have clarified this role in the revised manuscript.

2. Theorem 2 and full-domain improvement. We appreciate the reviewer's important distinction between increasing sensitivity under a modified sampling distribution and improving the original full-domain objective. To address this directly, Appendix B.1 defines the reference PDE loss LQ\mathcal{L}_Q over the entire domain and analyzes the gradient estimator induced by RAMP. Crucially, we establish a sufficient condition under which RAMP achieves strictly lower expected post-update LQ\mathcal{L}_Q than uniform sampling.

3. Theoretical analysis of RAMP itself. Appendix B.1 further characterizes RAMP as a sampling-induced modification of the gradient estimator and decomposes its bias into two components:

  • Parallel component: Strengthens the descent signal along the reference full-domain gradient, yielding improved one-step loss reduction under explicit smoothness and step-size conditions.

  • Orthogonal component: Changes the parameter-update direction without affecting the first-order loss decrease. We derive curvature-based conditions under which this displacement increases subsequent parameter sensitivity and improves the two-step full-domain loss.

This analysis further distinguishes RAMP from conventional importance sampling, which seeks to reduce variance while maintaining an unbiased estimator, whereas RAMP exploits a beneficial sampling-induced bias.

Finally, Appendix B.2 empirically examines these theoretical conditions at five Burgers1D training states across five refinement radius. The measured sufficient condition holds at all five states for ε≤0.02\varepsilon\leq0.02, with positive one-step full-domain improvements across all 25 tested combinations. The orthogonal curvature conditions are also satisfied at four of the five states, accompanied by the predicted improvements in subsequent sensitivity and loss.

We are particularly grateful for this criticism because it identified a genuine gap in our original presentation and motivated us to extend the work from a theoretical motivation for sensitivity-based sampling to a direct analysis of the proposed RAMP algorithm. We believe this addition substantially strengthens the paper's theoretical foundation, and we sincerely hope that the revised analysis addresses the reviewer's concerns regarding soundness.

Dear Reviewer kJg6,

Thank you again for your careful review and insightful criticism of the theoretical foundation of our work. We have taken your concerns seriously and substantially strengthened the manuscript in response. In particular, we have clarified the role of Theorem 1 and added Appendix B, which directly analyzes RAMP under the original full-domain objective, establishes sufficient conditions for improved training dynamics, and empirically verifies these theoretical conditions. All additional analyses and corresponding revisions have been incorporated into the updated manuscript, with changes highlighted in blue.

With the author–reviewer discussion period approaching its November 18 (AoE) deadline, we sincerely hope you can revisit our detailed response and the updated manuscript, and reconsider your evaluation in light of these new results. We would be grateful for any further feedback and would be happy to address any remaining questions or concerns before the deadline.

Thank you again for your time and consideration!

Thank you for your response. I have confirmed that detailed discussions have been added to the appendix. I believe this has strengthened the paper’s contributions; however, in order for the paper to be accepted, I believe the main text should also be significantly rewritten based on the added discussion. Considering the above, I will raise the score to 2.

Thank you very much for revisiting our response and providing your thoughtful follow-up.
We are genuinely pleased that the additional theoretical analysis has helped address your concerns and that you recognize its contribution to strengthening the paper.

We fully agree that these theoretical results should be more prominently presented in the main text.
Specifically, we plan to revise the end of Section 3.3 to better connect the proposed algorithm to full-domain training dynamics and introduce a new Section 3.4, Full-Domain Training Dynamics of RAMP.
This subsection would incorporate the RAMP-induced gradient bias and its decomposition (Appendix B.1.1-B.1.3), the sufficient condition for improved expected full-domain loss (Appendix B.1.3), and the curvature-based analysis of subsequent training dynamics (Appendix B.1.4).
We would also bring the key empirical verification from Appendix B.2 into the main text, while retaining detailed proofs and additional results in the appendix.

Given the limited time remaining in this discussion, we regret that we may not be able to present this restructuring for your consideration at this stage. Nevertheless, we are fully committed to incorporating these specific revisions to ensure that the main text clearly reflects the theoretical contributions developed in the appendix.

We sincerely appreciate your constructive feedback, careful reconsideration, and the opportunity to engage in this discussion, which have significantly helped us improve our work. Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.