- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces Critical-Point-Bench (CPB), a transition-aware benchmarking and model-selection framework for neural operators. The main motivation is that aggregate prediction errors may not adequately reflect a model's ability to capture important physical transitions, such as shock formation, phase selection, and pattern onset. CPB defines task-specific transition coordinates to evaluate prediction errors near critical regions, together with binary and continuous measures of event fidelity. Models must first satisfy predefined event-fidelity thresholds before being ranked by their critical-region prediction error. The framework abstains when no candidate meets the requirements. The authors evaluate CPB across several PDE systems and neural operator architectures (FNO, DeepONet, CNO, Transformer, and LinearNO). The results show that CPB can lead to different model-selection decisions, selecting FNO over LinearNO on Allen–Cahn with substantially lower critical-region error, while abstaining on an external PDEBench evaluation where no candidate meets the event requirements. Additional experiments investigate transition-specific failure patterns, model capacity, and targeted training interventions.
Critical strengths
Comprehensive and physically meaningful evaluation. The paper highlights an important limitation of aggregate prediction-error metrics in evaluating neural operators near physical transitions. By combining transition-local prediction errors with event recovery and continuous measures of physical structure, the proposed benchmark provides a more informative assessment of model performance. The experimental evaluation is extensive, covering multiple PDE systems and neural operator architectures, with independent validation-to-holdout comparisons, multiple training seeds, matched controls, and sensitivity analyses.
Critical weaknesses
Limited evidence for the added value of event-based model selection. Although the paper demonstrates that aggregate and transition-local errors can lead to different model rankings, the experiments do not clearly establish the additional benefit of the proposed event-fidelity criteria. In the Allen–Cahn experiment, where CPB selects FNO instead of LinearNO, both models already satisfy the event requirements, and the selection difference appears to arise entirely from their critical-region prediction errors. A simpler baseline that selects models solely by critical-region error would therefore likely produce the same result. On PDEBench, CPB abstains because no candidate satisfies the event requirements, but this does not demonstrate a case where event-based filtering leads to a better model selection. Including a critical-error-only selection baseline and an experiment where event fidelity meaningfully changes the selected model would help establish the benefit of the complete CPB framework.
Questions for authors
Questions:
-
Choice of event-fidelity thresholds. How are the task-specific thresholds for event recovery and continuous event discrepancy determined? Their choice could substantially affect model eligibility and potentially the selected architecture. For example, in the Allen–Cahn experiment, LinearNO has a lower continuous event discrepancy than FNO, so a stricter threshold could potentially reverse the reported selection. Could the authors clarify the rationale behind the chosen thresholds and investigate how sensitive the selection results are to them?
-
Extension to more complex transition structures. CPB is evaluated primarily on systems characterized by a single transition coordinate and a relatively simple event definition. How would the framework extend to physical systems with multiple phase transitions or several interacting fronts? In particular, how would the transition coordinates and event-fidelity criteria be defined in these settings?
Comments and Suggestions:
- Clarity of the introduction. The introduction could be more streamlined to explain the problem setup and motivation more clearly. In particular, the different physical systems used to evaluate CPB could be introduced more clearly, and the paragraphs could be better connected to provide a more coherent progression from the problem motivation to the proposed benchmark.