- The proposed switching criterion lacks a convincing connection to the optimization mechanism it is intended to detect.
The main methodological contribution of the paper is the adaptive switching criterion based on the cosine similarity of successive prediction displacements. However, this quantity does not directly characterize whether the semi-gradient remains a descent direction for the original residual loss, nor whether switching to the full gradient would improve convergence.
As the authors correctly observe in Equation (7), the semi-gradient is a descent direction for the original objective only when
⟨gF,gS⟩>0.
However, the proposed criterion instead monitors the alignment of successive changes in the predicted solution, which is a different geometric quantity. In particular, a negative prediction-displacement cosine may occur along an otherwise convergent trajectory, while a positive cosine may persist even when the semi-gradient is not a descent direction.
A simple example illustrates this limitation. Consider the scalar equation u=0, parameterized by uθ=θ, with residual rθ=−θ. The full gradient is gF=2θ, whereas the semi-gradient is gS=−2θ. Consequently, semi-gradient descent with a positive constant learning rate diverges geometrically, although successive prediction displacements remain positively aligned. In this example, the PASS criterion would never trigger a switch.
This observation is closely related to the residual-sign dependence acknowledged by the authors in Section 3.2: replacing rθ by −rθ leaves the least-squares objective unchanged but reverses the semi-gradient PDE contribution.
Although the authors explicitly recognize that their criterion is heuristic, the paper does not sufficiently establish why prediction-displacement alignment should reliably indicate when semi-gradient optimization becomes inappropriate. Since this criterion constitutes the main methodological novelty, I consider this an important limitation.
- The experiments do not sufficiently isolate the benefit of the adaptive switching rule from alternative optimization strategies.
The comparison with fixed switching schedules is useful, but the tested schedules are relatively coarse. In the Burgers experiment, the authors consider fixed switches at 10%, 25%, 50%, and 75% of training, with only the last remaining numerically stable. However, the paper does not report the distribution of actual PASS switching times.
A particularly informative comparison would involve fixed switching schedules near the times selected by PASS, including a fixed switch at the median PASS switching step. This would help determine whether the improvement comes specifically from the proposed prediction-based criterion or more generally from a suitable semi-gradient warm-up.
Furthermore, the paper motivates semi-gradients partly through the computational cost of differentiating the PDE residual. However, it does not compare against alternatives based on freezing or periodically refreshing the residual Jacobian. For example, one could consider an update direction of the form
Jr(θˉ)⊤rθ,
where θˉ is a reference parameter value, fixed initially or updated periodically. Such approaches could also reduce the cost of repeatedly differentiating the PDE residual, although their memory requirements and Jacobian-refresh costs would need to be accounted for.
Comparisons against these alternatives, as well as adaptive switching criteria based on residual stagnation or prediction-update magnitudes, would help isolate the specific contribution of PASS.
Finally, the experiments use only three random seeds per configuration, and some improvements over stabilized gradient-residual training are relatively small compared with the reported variability. Since the two backward paths can also have substantially different gradient magnitudes, additional comparisons with appropriately stabilized or gradient-normalized full-gradient optimization would strengthen the conclusions.
Additional minor remark: Consistency of the semi-gradient formulation for PDE systems.
The proposed semi-gradient gS=Ju⊤rθ implicitly identifies the space of PDE residuals with the space of network outputs. While this is consistent for the scalar equations considered in the paper, it is not necessarily well-defined for more general PDE systems.
In particular, when uθ:Ω→Rm and rθ:Ω→Rq with m=q, the matrix product Ju⊤rθ is dimensionally inconsistent.
A more general formulation would require specifying a mapping between the residual and solution spaces, potentially changing the resulting optimization direction and its conditioning properties. Although the authors explicitly restrict their analysis to scalar PDEs, this raises a question about the consistency and applicability of the proposed semi-gradient construction to more general physics-informed learning problems.