- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper focuses on the Gradient Update Mismatch (GUM) issue that occurs in almost all gradient surgery methods but has barely been discussed before. The GUM issue comes from the gradient manipulation of modern optimizers, which may destroy the conflict-free nature of the input gradient provided by the gradient surgery methods. The authors carefully analyze the GUM in PINNs training and propose a gradient update alignment to solve the GUM. The analysis is intensive, the experiment is solid, and the result is quite promising.
Critical strengths
- The manuscript comes with a detailed mathematical analysis. The reviewer particularly enjoyed Sec 3.3, where the authors carefully discuss which types of optimizer manipulations will break the conflict-freeness of the input gradient.
- The experimental setup is solid, with many ablations that consider different perspectives on the problem. The reviewer particularly liked Section 4.5. Although adding additional ablations would be helpful (e.g., test more values of instead of only 0 and 1), the current discussion already provides a good analysis of the state alignment method.
Critical weaknesses
The specific design of the state alignment is artificial and less motivated:
- The core of GUM is actually a trade-off between conflict-freeness and nice features provided by the optimizer. If we want to fully respect the conflict-freeness of the gradient surgery method, we can use the vanilla SGD optimizer. If we want to utilize the nice feature provided by other modern optimizers, e.g., Adam, we have to sacrifice the conflict-freeness to allow the optimizer to modify the gradient via the optimizer states.
- From this point of view, the state alignment method is actually the core of the proposed GUA method, as it aims to maintain the nice feature of the optimizers as much as possible while maintaining the conflict-freeness. If this part does not work well, the overall method will be close to the vanilla SGD optimizer+gradient surgery method.
- Unfortunately, the current design of the state alignment method is not "elegant", i.e., it is not based on any mathematical motivation but explicit manual correction. Moreover, the coefficients used for optimizer-state alignment are hyperparameters that need to be finetuned for different experiments (Table 5). If we need to fine-tune hyperparameters to make the optimization work, we can simply add weights to each loss term and find the best weights. This is much cheaper since we don't need to use additional backpropagation to get loss-specific gradients for gradient surgery methods.
Questions for authors
- Can the author further explain the in Eq.3?
- It seems that the authors want to measure the conflicts between the loss-specific gradients, but it is not clear why is required. Shouldn't it be ?
- The reviewer will argue that does not make too much sense. Even , it does not mean that there is conflict in the optimization. This is because we will never use a single or to update the network. For example, if but , the update direction without any gradient surgery method, i.e., is conflict-free to both and . If the author wants a value to represent the "difficult/conflict" of a multi-loss optimization problem, is a better value to monitor.
- Very recently, a concurrent preprint has also discussed the impact of the optimizer on conflict-free methods: Cautious Conflict-Free Inverse Gradient Methods (https://arxiv.org/abs/2610.31855). Could the author give some comments on the differences between GUM and this Cautious ConFIG?
- Other minor issues/questions:
- The last line on page 1 (Liu et al., 2025) is cited twice for the same sentence.
- In line 209, do the authors mean "does not hold" instead of "need not hold"?
- In Eq.6, the definition of $\widetilde{m} $ and $\widetilde{v} $ never occurs in the main content.