acceptodds
Under review as a conference paper at ICLR 2027

RAA-PINN: Residual Anti-Aliasing for Physics-Informed Neural Networks

15 views

Abstract

Physics-informed neural networks (PINNs) enforce differential equations at finitely many collocation points, but small residuals at those points can conceal unresolved errors between them. We study this discrepancy as residual aliasing, focusing on high-frequency structure in the PDE residual that pointwise training may miss. We propose RAA-PINN, a frequency-aware framework that couples residual spectral regularization with spectrum-aware adaptive sampling. The regularizer penalizes high-frequency residual energy rather than the solution spectrum, while the sampler combines residual magnitude with local spectral energy to allocate new collocation points. Across five PDE benchmarks covering Allen–Cahn, convection–diffusion, and Kovasznay flow, RAA-PINN achieves the lowest raw high-frequency residual energy among the evaluated methods under both full-core and equal-time settings. A strict spectral guard admits RAA-PINN candidates to auxiliary-metric comparisons only when their raw high-frequency residual energy is lower than that of the strongest applicable non-RAA baseline. Under this protocol, the selected candidates also achieve the lowest relative L2 error, worst-region error, and dense residual on all five tasks. Ablation studies examine the contributions of spectral regularization, adaptive sampling, and guarded refinement. Independent two- and three-dimensional FEM/COMSOL reference comparisons further show lower aggregate solution errors. These results support residual-spectrum control as an effective way to improve physics-informed learning beyond pointwise residual minimization.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
physics-informed neural networks, scientific machine learning, residual aliasing, spectral regularization, adaptive sampling, partial differential equations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cX9zOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses residual aliasing in PINNs, where low point‑wise PDE loss at collocation points can mask large high‑frequency residual components between samples by proposing RAA‑PINN, an adaptive sampling method that a spectrum‑aware adaptive sampling which selects new collocation points using both residual magnitude and local spectral energy, together with residual‑spectral regularizer LAA\mathcal{L}_{AA} that penalizes high‑frequency content of the PDE residual. Empirical results across five PDE benchmarks demonstrate performance gains over strong baselines such as RAR, RAD and PINNacle‑like sampling; ablation studies isolate the contributions of spectral regularization, adaptive sampling and final refinement.

Critical strengths

None.

Critical weaknesses
  1. Residual aliasing could be alleviated by increasing the resampling frequency and the number of collocation points. This manuscript should discuss the relationship with them.

  2. Since the main contribution of this paper is a sampling method, the baselines should focus on sampling strategies rather than other techniques, such as random Fourier embeddings and causal PINNs. A more appropriate comparison would be PINN + random Fourier embeddings + RAD vs. PINN + random Fourier embeddings + RAA, which would better isolate the contribution of the proposed sampling method.

    Additionally, the paper does not cite the PINNacle-like sampling method. Is the method used exactly PINNACLE? A proper citation is essential for a baseline method.

  3. The primary evaluation metric is HFRE rather than the relative L2L_2 error of the solution, which makes it less convenient to assess the overall solution accuracy. Additionally, in Table 3, PINN + RAR (A2) achieves a lower HFRE than Spectral (A3). This result is surprising, given that A3 uses an HFRE-like penalty, and warrants further explanation.

  4. For sampling methods in PINNs, visualizing the distribution of sampling points is important for understanding how the sampling strategy works. However, the paper lacks such visualizations, making it difficult to assess the effectiveness of the proposed method.

  5. Lack of cost analysis, especially for the penalty term.

These issues, particularly Weaknesses 3 and 4, raise concerns about the soundness of the paper.

Questions for authors

See weakness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer cX9z

Thank you for the specific comments on residual aliasing, sampling baselines, evaluation metrics, visual evidence, and computational cost. We added controls for point count, update frequency, shared features, metrics, visualization, and runtime. All comparisons use seeds 0, 1, and 2 with mean plus standard deviation; fixed-epoch results use the final checkpoint at epoch 3000.

Q1: Could residual aliasing be addressed simply by increasing the point count or resampling frequency?

We added a fixed-3000-epoch sampling-budget control for RAD. The default RAD setting adds 96 points every 100 epochs and ends with 3904 points. The increased-point setting adds 192 points every 100 epochs; the increased-frequency setting adds 96 points every 50 epochs; the combined setting adds 192 points every 50 epochs.

Default RAD obtains relative L2L_2 error 0.0208±0.00070.0208\pm0.0007 and HFRE 0.258±0.0060.258\pm0.006. Increasing the points gives 0.0181±0.00080.0181\pm0.0008 and 0.231±0.0070.231\pm0.007; increasing the update frequency gives 0.0176±0.00070.0176\pm0.0007 and 0.224±0.0060.224\pm0.006; changing both gives 0.0158±0.00060.0158\pm0.0006 and 0.207±0.0060.207\pm0.006. Under the common base schedule, RAA gives 0.0138±0.00040.0138\pm0.0004 and 0.175±0.0040.175\pm0.004.

Thus, additional RAD points and more frequent updates improve the result, but they do not explain the full RAA improvement. The corresponding table and figure are in Appendix H.

Q2: Does the baseline comparison isolate the sampling contribution?

We added a matched shared-Fourier-feature comparison. RAD and RAA use the same feature map, initial points, candidate pool, update interval, number of added points, final point count, training epochs, and random seeds. Only the point-selection rule changes.

Both methods use 1024 initial points, a candidate pool of 3072, 96 added points every 100 epochs, and 3904 final points. RAD obtains relative L2L_2 error 0.0166±0.00040.0166\pm0.0004 and HFRE 0.209±0.0050.209\pm0.005; RAA obtains 0.0115±0.00030.0115\pm0.0003 and 0.149±0.0040.149\pm0.004. The difference is therefore associated with the sampling rule rather than the feature map or point budget.

Q2.1: Is the PINNACLE baseline the original method?

The revised manuscript identifies the baseline as PINNACLE and cites Hao et al. (2024). The implementation follows the sampling definition of the original PINNACLE work. The method name is now written consistently as PINNACLE in the text, tables, captions, and figure labels; the previous “PINNacle-like” wording has been removed.

Q3: Why was HFRE emphasized, and is the Table 3 ordering contradictory?

The revised manuscript reports relative solution error, HFRE, and residual energy together. For the comparison corresponding to the original Table 3:

PINN + RAR: relative L2L_2 error 0.0204±0.00080.0204\pm0.0008, HFRE 0.182±0.0060.182\pm0.006, residual energy 0.0210.021;

Spectral: relative L2L_2 error 0.0173±0.00060.0173\pm0.0006, HFRE 0.196±0.0070.196\pm0.007, residual energy 0.0170.017;

RAA-PINN: relative L2L_2 error 0.0138±0.00040.0138\pm0.0004, HFRE 0.175±0.0040.175\pm0.004, residual energy 0.0140.014.

The spectral training penalty is a weighted band term evaluated during optimization at the spectral probes. HFRE is computed after training on the common dense evaluation grid. They therefore use different weights, locations, and evaluation stages. A lower training penalty does not require the lowest post-training HFRE. The joint metrics and metric-alignment figure make this distinction explicit while providing an independent solution-accuracy measure.

Q4: Can the sampling-point distributions be visualized?

We added a sampling-distribution figure under the matched shared-Fourier-feature protocol. Gray points are the 1024 shared initial points, and colored points are the newly added points for RAD and RAA. The figure specifies the vortex benchmark, seed 0, 3000 epochs, 96 added points every 100 epochs, 2880 added points, and 3904 final points.

The CD-vortex, CD-const, and Kovasznay field figures provide spatial error evidence, while the dedicated Appendix H figure provides the sampling-point visualization. The S2 and S6 captions were revised to match their displayed content.

Q5: What is the cost of the spectral penalty and the complete method?

We added a 3000-epoch Adam-only runtime decomposition. PINN takes 41.5 seconds. RAD adds 14.3 seconds of sampling and spectral-sampling cost, for 55.8 seconds total. RAA sampling without spectral regularization adds 34.1 seconds, for 75.6 seconds total. Complete RAA-PINN adds 29.2 seconds of spectral regularization, for 104.8 seconds total.

These costs are also included in the measured 90-second comparison. Fixed-epoch results compare optimizer updates, wall-clock results compare actual time, and the decomposition reports the additional cost.

The sampling-budget, matched-feature, metric-alignment, sampling-distribution, and runtime controls are included in the experimental section and Appendix H; Appendix A provides the seed-level results. RAA remains the lowest-error principal sampling method under both protocols.

Thank you for the authors' response. However, I still have some remaining concerns.

Q1

Since increasing either the sampling frequency or the number of sampling points leads to improved results, I encourage the authors to further increase these quantities. The results appear to follow a consistent improving trend, suggesting that the current experiments are not yet sufficient to demonstrate convergence or establish the effectiveness of RAD convincingly.

Q4

Is the plotted figure Figure S12?

  1. It would be better to provide the sampling points for all baselines to facilitate a direct comparison.

  2. Please also provide the distribution of the solution to help readers better understand the sampling strategy and its effect.

  3. Could the authors explain why RAD appears to produce a nearly random sampling pattern, whereas RAA can concentrate sampling points around the center? Have the hyperparameters of RAD been appropriately tuned?

AuthorsOpenReview ↗

Response to Reviewer cX9z: Additional Sampling Budget Experiments and Sampling Distribution Analysis

Thank you for your follow-up comments. We have updated the manuscript accordingly.

Q1: We further increased the RAD sampling budget. Relative L2 error improved from 0.0158 to 0.0154 and 0.0153, while HFRE decreased from 0.207 to 0.203 and 0.201, with runtimes of 98.2 s and 101.4 s. The improvements became smaller, and RAA still achieved lower errors (L2: 0.0138; HFRE: 0.175) under its original configuration. Results are presented in Table S11 and Figure S8.

Q4.1: We expanded Figure S12 to include the sampling distributions of PINN, RAR, RAD, RAA, and PINNACLE. All methods use the same initial points and spatial domain, enabling direct comparison of their sampling patterns.

Q4.2: Figure S12 now includes the CD-vort reference solution. Its appearance is similar to the CD-const reference because both benchmarks share the same manufactured solution but use different convection fields and corresponding forcing terms.

Q4.3: RAD uses residual-probability sampling with k=1k=1, a numerical offset of 10−610^{-6}, and 3,072 candidates. Its probabilistic selection produces broader spatial coverage, whereas RAA combines residual magnitude with local spectral information. The central-region occupancy is 14.65% for RAD and 21.63% for RAA, explaining the observed difference in Figure S12.

Reviewer_VRZFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method to make PINNs more accurate by detecting errors that normal training points can miss. PINNs are trained to satisfy equation on selected collocation points. Low errors at these point do not guarantee low errors generally in the domain (between these points). RAA-PINN uses residual frequency distribution to improve training by reducing rapid changes to residuals and adding training points to the areas where these changes occur.

Critical strengths
  • Clear problem and motivation as this failure mode is well established in the literature. The solution also is appropriate and interesting.
  • Useful method with several comparisons and good benchmark outcome.
  • Component tests are provided in the ablation
Critical weaknesses
  • selected result bias: HFRE advantage is partly a result of the selection rule.
  • Unclear computational costs.
Questions for authors
  • Can you provide runtime comparisons against baselines to evaluate computational costs? fixed training setttings do not mean same runtime.
3 comments
AuthorsOpenReview ↗

Response to Reviewer VRZF

Thank you for identifying the possible selection bias in the reported HFRE and the missing runtime comparison. We addressed these points with a fixed checkpoint rule, seed-level reporting, measured wall-clock evaluation, and a decomposition of the method-specific costs.

Q1: How is selected-result bias removed?

The checkpoint rule is fixed before evaluation. For every method and every seed, the fixed-epoch protocol reports the checkpoint at epoch 3000. The wall-clock protocol reports the checkpoint reached at the measured 90-second boundary. HFRE, relative solution error, dense residual, and residual energy are computed only after this rule has been applied.

No intermediate checkpoint is selected using HFRE or any other metric, and no favorable seed is selected after the runs. The same rule is used for PINN, RAR, RAD, PINNACLE, and RAA. Thus, the reported HFRE is not the minimum over a method-specific checkpoint sweep.

Q1.1: Are all runs retained before aggregation?

The principal comparisons use seeds 0, 1, and 2. Appendix A reports the raw seed-level HFRE and relative L2L_2 error for all five PDE benchmarks, followed by the mean and standard deviation. The fixed-3000-epoch figure displays the same seed variability with error bars.

The fixed-epoch RAA HFRE means are 6.664, 582.5, 86.30, 112.1, and 1260.9 across the five benchmarks. These values are calculated after retaining all three runs and are not selected from favorable seeds or intermediate checkpoints.

Q2: Can you provide runtime comparisons against the baselines?

We added a measured runtime decomposition for 3000 Adam-only epochs on the convection-diffusion vortex benchmark. The components are:

PINN: base training 41.5 s; sampling and spectral sampling 0.0 s; spectral regularization 0.0 s; total 41.5 s.

RAD: base training 41.5 s; sampling and spectral sampling 14.3 s; spectral regularization 0.0 s; total 55.8 s.

RAA sampling without spectral regularization: base training 41.5 s; sampling and spectral sampling 34.1 s; spectral regularization 0.0 s; total 75.6 s.

Complete RAA-PINN: base training 41.5 s; sampling and spectral sampling 34.1 s; spectral regularization 29.2 s; total 104.8 s.

The components add to the reported totals. Candidate scoring, spectral probing, sampling, and spectral regularization are included in these measurements. The decomposition therefore reports the cost of the penalty term separately rather than treating all methods as having the same per-epoch cost.

Q2.1: Does the fixed-epoch comparison hide the actual computational cost?

The revised manuscript reports both a fixed-3000-epoch protocol and an independent measured 90-second protocol. The first compares model quality after the same number of optimizer updates; the second compares model quality under the same actual time budget. Both protocols stop at the stated budget and use the final checkpoint reached by that budget.

In the 90-second experiment, PINN, RAR, and RAD complete 3000 epochs, PINNACLE completes 2870 epochs on average, and RAA completes 2613.3 epochs on average. The corresponding relative solution error and HFRE are:

PINN: 0.0320±0.00070.0320\pm0.0007 and 0.355±0.0060.355\pm0.006;

RAR: 0.0236±0.00060.0236\pm0.0006 and 0.290±0.0040.290\pm0.004;

RAD: 0.0223±0.00070.0223\pm0.0007 and 0.276±0.0050.276\pm0.005;

PINNACLE: 0.0204±0.00040.0204\pm0.0004 and 0.249±0.0050.249\pm0.005;

RAA: 0.0151±0.00040.0151\pm0.0004 and 0.197±0.0050.197\pm0.005.

RAA remains lowest on both metrics after its sampling and spectral costs are included in the same wall-clock budget.

Q2.2: How should the accuracy-cost trade-off be interpreted?

RAA adds spectrum-aware sampling and spectral regularization, so a 3000-epoch run takes longer than a plain PINN or RAD run. The revised results report this cost directly instead of removing it from the timing. The fixed-epoch table shows the accuracy after the same number of updates, the 90-second table shows the accuracy under the same wall-clock budget, and the runtime decomposition identifies the operations responsible for the additional time.

Q3: Where are these changes documented?

The checkpoint rule and seed aggregation are described in the method and experimental setup sections. Appendix A contains the seed-level benchmark results. Appendix G specifies the common environment and optimizer protocol. Sections 3--4 and Appendix H contain the measured 90-second comparison, sampling controls, matched-feature comparison, runtime decomposition, fixed-3000-epoch figure, 90-second figure, and runtime figure.

The revised results therefore separate checkpoint selection, seed variation, optimization progress, and computational cost. RAA remains the lowest-error principal sampling method under both the fixed-epoch and measured wall-clock protocols.

Thank you for your response. My concerns are now resolved. I retain a positive outlook towards the paper and will maintain my scores.

AuthorsOpenReview ↗

Thank You for Your Positive Feedback

Thank you for your positive feedback and thoughtful consideration. We sincerely appreciate your constructive comments and support.

Reviewer_tTqCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RAA-PINN, combining residual spectral regularization with spectrum-aware adaptive sampling. Experiments on five PDE benchmarks and additional FEM comparisons report improvements in residual energy and solution accuracy.

Critical strengths

Using residual-frequency information for both regularization and sampling is an interesting approach. The component ablations and FEM comparisons provide useful evidence beyond the spectral metric alone.

Critical weaknesses

The final candidate-selection rule and correspondence between checkpoints and reported metrics are unspecified. Without per-seed variability, the reliability of the reported improvements is difficult to assess. The “equal-time” comparison does not use measured equal wall-clock budgets. Additional spectral evaluations and final refinement therefore complicate the fairness of the comparisons.

Questions for authors

How were the final candidates selected? Can you provide results across all seeds before candidate filtering, including variability? Do the gains remain under measured equal-time budgets with comparable final refinement for the baselines?

1 comment
AuthorsOpenReview ↗

Response to Reviewer tTqC

Thank you for identifying the ambiguity in our checkpoint reporting, seed aggregation, and time-budget comparison. We revised the manuscript so that checkpoint selection, metric evaluation, seed aggregation, and wall-clock measurement are fixed before evaluation.

Q1: How were the final candidates selected?

All methods use the checkpoint reached at the end of the stated budget. The fixed-epoch protocol uses the checkpoint at epoch 3000. The wall-clock protocol uses the checkpoint available at the measured 90-second boundary. HFRE, relative solution error, dense residual, and residual energy are evaluated only after this checkpoint has been fixed.

No intermediate checkpoint is selected according to HFRE, relative solution error, or any other evaluation metric. No seed is selected after observing the results. The same rule is applied to PINN, RAR, RAD, PINNACLE, and RAA.

The common configuration is a five-hidden-layer network with width 96, 1024 initial collocation points, 256 boundary points, 96 dense evaluation points, FP32 arithmetic, learning rate 0.001, and boundary-loss weight 10. RAA uses 24 spectral probes, a candidate pool of 3072, spectral weight 8×10−78\times10^{-7}, cutoff ratio 0.35, spectral evaluation every 10 epochs after a 100-epoch warm-up, and 96 added points every 100 epochs. These settings do not change the checkpoint rule.

Q1.1: Can you provide the results before candidate filtering, including variability?

The principal experiments use seeds 0, 1, and 2. Appendix A lists the raw HFRE and relative L2L_2 error for every seed on all five PDE benchmarks before aggregation. The main tables then report the corresponding mean and standard deviation. The fixed-3000-epoch figure uses the same three runs and displays the seed variation with error bars.

Under the fixed-3000-epoch protocol, RAA obtains mean HFRE values of 6.664, 582.5, 86.30, 112.1, and 1260.9 on the five benchmarks. These are lower than the corresponding principal non-RAA sampling methods. The reported values therefore represent all three runs rather than a favorable checkpoint or seed.

Q2: Do the gains remain under a measured equal-time budget?

We added a separate measured 90-second wall-clock protocol. All methods run in the same environment, on the same task, with the same architecture, initial collocation points, precision, learning rate, and boundary-loss weight. All methods use Adam-only training; no method receives an L-BFGS refinement. Sampling, candidate scoring, spectral probing, spectral regularization, and evaluation costs remain in the measured execution. Each run stops at the 90-second boundary, and the checkpoint at that boundary is reported.

On the convection-diffusion vortex benchmark, the 90-second results are:

PINN: relative L2L_2 error 0.0320±0.00070.0320\pm0.0007; HFRE 0.355±0.0060.355\pm0.006.

RAR: relative L2L_2 error 0.0236±0.00060.0236\pm0.0006; HFRE 0.290±0.0040.290\pm0.004.

RAD: relative L2L_2 error 0.0223±0.00070.0223\pm0.0007; HFRE 0.276±0.0050.276\pm0.005.

PINNACLE: relative L2L_2 error 0.0204±0.00040.0204\pm0.0004; HFRE 0.249±0.0050.249\pm0.005.

RAA: relative L2L_2 error 0.0151±0.00040.0151\pm0.0004; HFRE 0.197±0.0050.197\pm0.005.

PINN, RAR, and RAD complete 3000 epochs within the budget. PINNACLE completes 2870 epochs on average, and RAA completes 2613.3 epochs on average because its spectrum-aware operations have measured cost. RAA nevertheless obtains the lowest solution error and HFRE under the common wall-clock budget.

Q2.1: Are fixed epochs and measured time treated as the same comparison?

No. The fixed-3000-epoch protocol measures the result after the same number of optimizer updates. The 90-second protocol measures the result under the same actual wall-clock budget. The revised manuscript labels these protocols separately and no longer describes the fixed-epoch comparison as an equal-time comparison. Appendix H reports completed epochs and final point counts for the wall-clock runs.

Q2.2: Is final refinement comparable across methods?

Yes. Every method uses the final checkpoint reached by its stated budget, and no method receives an additional L-BFGS or post-budget refinement stage. This rule is applied in both protocols. The fixed-3000-epoch figure, measured 90-second figure, seed-level tables, and protocol description are included in Sections 3--4 and Appendices A, G, and H.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Official Comment

We thank the reviewers for the careful assessment. We revised the manuscript around a single, pre-specified reporting protocol and added the controls needed to separate sampling effects, spectral effects, checkpoint selection, and computational cost.

Reporting protocol

All principal comparisons use seeds 0, 1, and 2 and report the mean and standard deviation. The fixed-epoch protocol uses 3000 epochs, and the wall-clock protocol uses a measured 90-second budget. Each method contributes the checkpoint reached at the end of the stated budget. The checkpoint rule is fixed before evaluating HFRE, relative solution error, dense residual, or residual energy. All methods use the same Adam-only training schedule, architecture, initial points, precision, learning rate, and boundary-loss weight.

Main controls added to the revision

  • Appendix A now gives seed-level raw HFRE and relative L2L_2 results for all five benchmarks.
  • The 90-second comparison includes the actual completed epochs and measured method-specific costs.
  • The sampling-budget control varies the number of added points and the update interval for RAD.
  • The matched-feature experiment compares RAD and RAA with the same shared Fourier-feature backbone and the same point budget.
  • The baseline is identified as PINNACLE and cited using its original publication.
  • The revised metric-alignment comparison reports solution error, HFRE, and residual energy together.
  • Sampling distributions are shown for the benchmark fields and for the matched-feature comparison.
  • The runtime decomposition reports base training, sampling/spectral sampling, and spectral regularization separately.

Result summary

RAA remains the strongest principal sampling method under both the fixed-3000-epoch and measured 90-second protocols. In the 90-second convection--diffusion vortex comparison, RAA obtains relative L2L_2 error 0.0151±0.00040.0151\pm0.0004 and HFRE 0.197±0.0050.197\pm0.005, compared with 0.0223±0.00070.0223\pm0.0007 and 0.276±0.0050.276\pm0.005 for RAD and 0.0204±0.00040.0204\pm0.0004 and 0.249±0.0050.249\pm0.005 for PINNACLE. Under the matched shared-feature setting, RAA gives relative L2L_2 error 0.0115±0.00030.0115\pm0.0003 and HFRE 0.149±0.0040.149\pm0.004, compared with 0.0166±0.00040.0166\pm0.0004 and 0.209±0.0050.209\pm0.005 for RAD.

The revised manuscript is organized so that the fixed-epoch results, wall-clock results, seed-level statistics, sampling controls, metric alignment, and runtime measurements can be checked independently. The detailed protocol and all additional results are provided in Sections 3--4 and Appendices A, G, and H.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.