- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Core idea: Adaptive collocation methods for physics-informed neural networks (PINNs) pick B training points by scoring V candidates and keeping the best. Keeping the best B of V is truncation selection at quantile 1 − B/V, so V controls how hard the criterion is optimised. Yet V is almost never reported, and V/B varies across papers by about four orders of magnitude.
Main experiments: At a fixed budget of B = 40, the authors sweep V/B from 1.25 to 30 over 64 shared seeds on 1D tasks (a square-wave fit, “crenel”, and a Poisson problem with a sharp layer, “poisson1d”). They compare against a uniform control re-drawn on the rules’ own schedule.
Key findings:
- Pointwise rules degrade as V grows. RAR-G (the add-largest-residuals rule from DeepXDE), dynamicAL, PINNACLE, BADGE and RAD get worse. Two rules with unrelated scores are level with the control at V/B = 1.25 and fail every seed by V/B = 20, so the collapse is attributed to the selection form, not the score.
- Growth removes the harm. If the point set is allowed to grow, as in the shipped libraries, the same rule takes no harm. The damage is from searching hard at a fixed budget.
- Growth removes the harm. If the point set is allowed to grow, as in the shipped libraries, the same rule takes no harm. The damage is from searching hard at a fixed budget.
Critical strengths
- The candidate-set size is rarely reported, and the RAR-G snippet that scores 100k candidates to add one point makes the point well.
- It uses 64–100 seeds, failure rates rather than means over 3–10 seeds, shared seeds, and paired tests. It also has a multiverse analysis, pre-registered predictions, and failures reported openly, such as the failed 2,000-point gate and the failed replication on three tasks.
- The held-versus-re-drawn gap (35/64 vs 1/64 on poisson1d) is large, and the point that it changes which rules “cross” is valuable for benchmarking.
- The growing-set versus fixed-budget contrast is clean and directly relevant to how the cited libraries actually run.
Critical weaknesses
- The re-drawn control and the growing-versus-fixed-budget contrast are the most useful and defensible contributions. However, the evidence supports a much narrower claim than the title and abstract suggest. The harm is shown mainly at B = 40 on one- and two-dimensional problems that are either unsolved (burgers1d) or built to be irreducible (crenel), and it fades or becomes impractical by B ≈ 80 to 400.
- Everything is at B = 40, with 1–2D tasks, a 2×64 tanh network, and 4000 Adam steps. The field uses thousands of points. The authors’ own evidence shows how fragile the finding is outside this regime: Selection stops changing failure rates by B ≈ 80. At B = 400, PINNACLE’s V effect is “no longer large enough to act on.” A registered B = 2,000 cell missed its accuracy gate. On burgers1d, no cell reaches even a time-independent predictor.
Questions for authors
- Can you show the effect at a budget where uniform still has headroom and a published rule is used as published, on a PDE that is actually solved (for example Burgers with a hard constraint and L-BFGS)?
- Can you survey V/B across published PINN adaptive-sampling papers and codebases in a table?
- Which prediction of your theory could have failed and did not? So far the theory seems to accommodate both sign directions (capacity, GRAD-MATCH at B ≥ 80, crenel one-step decrease).
- Can you pin the reference and landmark counts at every rung to remove the estimator confound?
- Why is the headline not stated as “fixed-budget top-B selection on irreducible-residual tasks”?