- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates when derivative supervision improves neural network predictions, focusing on derivative-label construction, loss balancing, and noise sensitivity. The authors compare different derivative estimators and weighting strategies, develop a finite-design risk decomposition to explain the effects of approximation bias and noise, and propose a last-layer-based rule to predict the usefulness of derivative supervision. Extensive experiments are conducted across synthetic functions and scientific applications, including financial models, PDEs, molecular modeling, and atmospheric reconstruction.
Critical strengths
Understanding when and why derivative supervision improves prediction is an important problem in scientific machine learning. While derivative-based supervision has been widely used in applications such as PDE learning, differential machine learning, and molecular modeling, its effectiveness is not always guaranteed. This paper addresses a meaningful question by investigating the conditions under which derivative information improves or degrades prediction performance.
Critical weaknesses
- The PDE experiments do not sufficiently distinguish additional derivative information from the effect of a modified training objective.
In the Burgers FNO experiment, derivative labels are constructed by applying finite-difference operators to the reference numerical solution. The same operators are applied to the predicted field. Consequently, the derivative loss can be expressed as
[ \mathcal L_g =|D_h\hat U-D_hU|^2 =|D_h(\hat U-U)|^2. ]
Therefore, when the solution is accurately predicted at every grid point, the corresponding finite-difference derivatives are automatically recovered. The constructed derivative labels do not provide independent information beyond the complete discrete solution field. Instead, they introduce an additional structured penalty on the prediction error, which can be interpreted as a discrete Sobolev-type loss or a frequency-dependent reweighting of the error.
Although the authors include a matched-observation experiment comparing derivative supervision with additional neighboring value observations, this does not fully distinguish the benefit of derivative information from the optimization effects induced by the modified loss.
As a result, the PDE experiments primarily demonstrate the effectiveness of a particular derivative-weighted training objective, rather than establishing the value of independently acquired derivative information. This distinction should be made more explicit, particularly given the paper's emphasis on derivative-label construction.
- Insufficient consideration of spatial resolution, observation sparsity, and PDE solution regularity.
The reliability of finite-difference derivative labels depends strongly on the underlying grid resolution, observation noise, and regularity of the target solution. Even for smooth solutions, finite-difference approximations introduce discretization errors, while numerical differentiation can amplify observation noise.
However, the PDE experiments primarily rely on numerical solution fields from PDEBench, where neighboring grid values are available for derivative construction. This assumption may not hold in practical settings with sparse or irregularly sampled observations, where the required finite-difference stencils cannot be directly constructed.
Furthermore, the Burgers experiments consider a viscous equation, for which solutions possess relatively favorable regularity under suitable conditions. The situation becomes substantially different for inviscid Burgers equations, where shock discontinuities may develop and classical pointwise derivatives may cease to exist. In such cases, derivative supervision based on squared pointwise gradient errors may not be an appropriate formulation.
These issues raise questions about the broader applicability of the PDE results. The paper would benefit from a systematic investigation of resolution dependence, sparse observations, and the regularity assumptions underlying derivative supervision.
- Limited connection between the theoretical analysis and practical derivative-label construction.
The theoretical risk decomposition is informative under its stated assumptions, but the exact analysis primarily concerns fixed feature representations and specified noise models. In contrast, finite-difference derivative labels in PDE applications are constructed from the same numerical observations as the value labels, potentially introducing correlated errors, discretization bias, and resolution-dependent uncertainty.
Although the paper discusses general label covariance, it does not establish a sufficiently direct connection between these practical sources of derivative-label error and the proposed predictive framework.
Moreover, the last-layer prediction rule requires an empirically measured clean-label benefit, obtained by training a derivative-supervised model with clean labels. Thus, the proposed rule evaluates how an already measured benefit deteriorates under additional noise, rather than providing a general criterion for determining whether an arbitrary derivative-label construction is reliable or useful in the first place.
This limits the practical scope of the claimed predictive capability.
- The manuscript suffers from excessive experimental accumulation, unclear organization, and presentation issues.
The paper contains a large number of experiments spanning different tasks, architectures, derivative constructions, weighting strategies, and evaluation metrics. However, the overall narrative is difficult to follow, and the logical connections between different experimental sections are not always sufficiently clear.
The presentation often relies on extensive numerical comparisons, error ratios, and rankings, without enough intuitive visualization of the underlying learning phenomena. For instance, the PDE experiments would benefit from direct visual comparisons of reference solutions, model predictions, and spatial error distributions, allowing readers to understand which structures are improved by derivative supervision.
Some figures also contain noticeable formatting issues. In Figure 3, the subfigure title "a Burgers FNO" is repeated twice in the left panel. Such errors suggest that the figures and overall manuscript require more careful proofreading and refinement.
Additionally, Table 1 aggregates a wide variety of heterogeneous experiments and evaluation metrics into a single dense table, making it difficult to extract the main conclusions. The extensive reporting of numerical results, combined with verbose descriptions, further obscures the central scientific contributions.
I recommend substantially streamlining the experimental presentation, improving figure quality, and establishing a clearer progression from the central research questions to the theoretical analysis and experimental evidence.
Questions for authors
-
In the Burgers FNO experiments, derivative labels are constructed directly from numerical solutions using finite differences. How do the authors distinguish the benefit of derivative supervision from that of a discrete Sobolev-type or frequency-weighted loss?
-
How sensitive are the PDE results to grid resolution and observation sparsity? Would the conclusions still hold when only sparse or irregularly sampled observations are available?
-
How does the approach generalize to PDE solutions with limited regularity, such as inviscid Burgers equations with shocks, where classical pointwise derivatives may not exist?
-
Since the proposed prediction rule requires measuring the clean-label benefit beforehand, how can it be applied when only noisy or biased derivative estimates are available?
-
Could the authors improve the organization and presentation of the experiments, particularly by providing more intuitive visualizations of the predicted solutions and the effects of derivative supervision?
response to rebuttal
Thank you for the clarifications and additional experiments. The comparison using identical raw value labels and validation-selected weights directly addresses my main concern, and the revised manuscript makes the clean-reference requirement of the noise-tolerance analysis clear. I have decided to maintain my original rating of 3 (Weak Accept).