- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a new collocation sampling method, called Radius-bounded Ascent for Maximizing Parameter-gradient norm (RAMP). The main idea of RAMP is to use the norm of the pointwise PDE residual loss gradient with respect to the model parameters as a signal for resampling collocation points, rather than relying solely on the residual loss magnitude. The proposed method is evaluated on a range of benchmark equations and consistently outperforms several baseline sampling strategies. The authors further demonstrate that RAMP can be effectively combined with different optimization algorithms and loss balancing methods, suggesting that the proposed sampling strategy is broadly applicable across different PINN training settings.
Critical strengths
None
Critical weaknesses
Unfair comparison. Using the residual gradient norm as a sampling signal is reasonable, but it introduces additional computational cost due to pointwise gradient evaluations. The current manuscript does not analyze this overhead, making the comparison with simpler baselines potentially unfair.
The authors should report both memory usage and wall-clock time. For example, under a matched memory budget, random resampling could use more collocation points instead of computing pointwise gradient norms. If such a baseline achieves comparable or better performance, the practical advantage of RAMP would be less clear.
This issue is particularly important because the main contribution of the paper is largely empirical rather than theoretical.
Presentation. The manuscript also requires careful revision for clarity. The terminology and mathematical notation are not always used consistently, and several parts are unnecessarily difficult to follow.
Related works. The paper should discuss a broader range of literature, particularly prior work on importance sampling. This would help better position RAMP with respect to existing approaches.
Questions for authors
Can the authors provide a fair comparison with the baselines in terms of computational cost? In particular, it would be helpful to report memory usage and wall-clock time, and to evaluate whether RAMP still outperforms random resampling under a matched memory budget.
8 comments
Weakness 2: Presentation Unclarity
We sincerely thank the reviewer for this valuable comment. After carefully revisiting the manuscript in light of your feedback, we identified several instances of inconsistent terminology and mathematical notation, as well as expressions that could have been presented more clearly.
Revision in the manuscript
We have carefully revised these issues throughout the manuscript, with all changes highlighted in blue in the revised version. We would greatly appreciate it if the reviewer could take another look at these revisions.
We apologize for any confusion caused by the original presentation and sincerely thank the reviewer once again for this constructive feedback, which has helped us improve the clarity and consistency of our manuscript.
Weakness 1 & Question 1: Computational Cost
We thank the reviewer for raising this important concern. We agree that evaluating RAMP's practical effectiveness requires fair comparisons that account for both runtime and memory usage.
Wall-Clock-Matched Performance Comparison
For each PDE and optimizer, we evaluated performance under an estimated wall-clock time budget equivalent to that required to train RAMP for 20,000 epochs. We summarize the results using the same metrics as Table 1 of the manuscript: mean rank (MR) and aggregate relative improvement ().
Under the same wall-clock time budget, RAMP performs 1.38–7.04 times fewer parameter updates under Adam, 1.25–3.51 times fewer under LRA, and 1.21–4.11 times fewer under HARMONIC compared with the other methods across the 12 PDEs. The detailed numbers of training epochs are provided in Table 10 of Appendix D.8.
As shown in the table below, RAMP achieves competitive performance despite performing substantially fewer parameter updates than the other methods. These results provide further empirical support for our central hypothesis that locally maximizing pointwise-parameter-sensitivity enables RAMP to identify collocation points that induce stronger parameter-space training signals, thereby facilitating more effective error reduction even with substantially fewer parameter updates.
However, the number of parameter updates differs considerably across methods, by up to a factor of approximately 7 under Adam and nearly 4 under LRA and HARMONIC. To reduce this discrepancy, we introduce a computationally efficient variant of RAMP, denoted RAMP-P. It retains parameter-sensitivity-guided collocation refinement while reducing the frequency of refinement-direction computations.
Unlike the original RAMP, which computes the refinement direction at every training iteration , RAMP-P computes once every iterations. The resulting direction is then reused to progressively move each collocation point in equal increments over the subsequent iterations. Since a key advantage of RAMP is its ability to adapt collocation points to the current training state at every iteration, choosing a large could reduce its responsiveness to changes in the training state. We therefore set and conducted additional experiments.
| Method | Adam MR ↓ | Adam ↑ | LRA MR ↓ | LRA ↑ | HARMONIC MR ↓ | HARMONIC ↑ |
|---|---|---|---|---|---|---|
| R3 | 4.42 (1.68) | -37.03 (18.19) | 3.83 (1.34) | +19.89 (7.12) | 5.17 (0.94) | -78.42 (18.43) |
| RAD | 3.75 (1.36) | -25.31 (14.38) | 3.25 (1.71) | +27.19 (6.44) | 3.50 (1.68) | +18.97 (14.20) |
| LAS | 3.92 (1.56) | -26.89 (9.76) | 2.75 (1.76) | +40.00 (1.04) | 3.00 (1.76) | +17.78 (12.04) |
| PINNACLE | 3.08 (1.68) | -7.69 (9.48) | 4.08 (1.62) | +24.47 (9.49) | 3.00 (2.04) | +18.78 (6.63) |
| RAMP | 3.33 (2.31) | +6.32 (1.17) | 4.00 (2.22) | +2.81 (1.37) | 3.50 (1.17) | +19.57 (2.13) |
| RAMP-P | 2.50 (1.17) | +8.12 (5.35) | 3.08 (1.44) | +40.85 (3.79) | 2.83 (1.64) | +35.98 (4.25) |
As shown in the table, RAMP-P achieves the best performance in both MR and under Adam and HARMONIC, while achieving the highest under LRA. Under LRA, RAMP-P also achieves the second-lowest MR.
Revision in the manuscript
To address the reviewer's concern, we have expanded Appendix D.8 with the following results:
- Table 9 (p.39): Computational cost per 100 epochs, including 95% confidence intervals, with the corresponding profiling procedures and numbers of repeated measurements.
- Table 10 (p.40): PDE- and optimizer-wise training epoch allocations under computational budgets equivalent to 20,000 epochs of the RAMP.
- Table 11 (p.41): PDE-wise relative errors under the matched computational budgets, including 95% confidence intervals where available.
- Table 12 (p.42): Aggregate performance comparison in terms of MR and , corresponding to the summary table presented above.
Weakness 3: Related works
We sincerely thank the reviewer for highlighting the importance of positioning RAMP more clearly with respect to existing importance sampling methods. We agree that our original presentation did not sufficiently distinguish the two approaches, particularly because both prioritize samples that induce strong parameter-space training signals. We greatly appreciate this observation, which motivated us to clarify their fundamental differences and further strengthen our theoretical analysis.
To address this concern, we have added Appendix B.1. Full-Domain Training Dynamics, which explicitly analyzes RAMP from a bias–variance perspective and compares it with gradient-norm importance sampling.
Both approaches emphasize pointwise parameter sensitivity, but their underlying mechanisms and objectives differ. Conventional importance sampling uses inverse-probability weighting to preserve an unbiased estimate of the reference gradient while reducing its variance. In contrast, RAMP transports uniformly sampled points through radius-bounded sensitivity ascent and aggregates their gradients without importance weighting, thereby introducing a sampling-induced bias.
Our new analysis demonstrates how this bias can contribute positively to full-domain optimization. Its parallel component strengthens the descent signal and can yield a strictly lower expected one-step full-domain loss under explicit conditions. Its orthogonal component can further improve subsequent parameter sensitivity and two-step loss through favorable curvature alignment.
These distinctions are also supported by the experiments in Appendix B.2. At ε=0.002, RAMP increases gradient-estimation variance by 0.3%–1.8%, whereas ideal sensitivity-proportional importance sampling reduces variance by factors of 1.40–1.69. Nevertheless, RAMP achieves improved one-step full-domain loss through its sampling-induced bias.
We are particularly grateful for the reviewer's suggestion, as it helped us identify an important omission in our original presentation and establish a clearer distinction between RAMP and existing sampling approaches. We believe the additional analysis substantially improves the positioning and theoretical understanding of RAMP, and we sincerely hope it addresses the reviewer's concern regarding related works.