- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a Taylor-approximated weighted score matching to efficiently replace nested reverse-mode automatic differentiation for training point-process models with weighted score matching. The authors propose a plug and play method to implement the same. The proposed method shares the common subset of computations across multiple methods, so that those intermediate results can be reused and we can observe a lower cost.
Critical strengths
A> The proposed method is dynamic - it takes into consideration the intermediate steps and then takes a decision
B> The implementation is clear and neat - it will be easy to plug and play on multiple downstream tasks.
Critical weaknesses
A> Taylor-mode automatic differentiation has been explored in prior work (e.g., [1]). The contribution of this paper is its application to WSM, together with a program-level cost analysis and empirical evaluation. However, the evaluation would be more convincing with comparisons against alternative higher-order differentiation strategies under the same experimental setup, including runtime, memory consumption, and end-to-end training efficiency. This would better establish the practical benefits of the proposed approach.
B> The evaluation is limited to JAX on a specific GPU platform and excludes compilation time. Since Taylor-mode differentiation may increase computation graph complexity, the authors should assess whether compilation overhead affects end-to-end training time and the practical speedups.
Furthermore, the pipeline might demonstrate better benefits where the workload has opportunities to cache or reuse intermediate computations (eg, video object detection). Evaluating these factors would strengthen the claims regarding the method's practical utility, ease of deployment and generalizability.
[1] Bettencourt, Jesse, Matthew J. Johnson, and David Duvenaud. "Taylor-mode automatic differentiation for higher-order derivatives in JAX." Program transformations for ML workshop at NeurIPS 2019. 2019.
Questions for authors
The paper is an interesting read. Please quantify your acceleration, preferably on other devices, and in non-JAX settings.
3 comments
Response to Question 1
Question 1. The paper is an interesting read. Please quantify your acceleration, preferably on other devices, and in non-JAX settings.
Response. Thank you for this constructive suggestion. We agree that evaluating the acceleration on different hardware and AD frameworks is important for understanding its practical applicability. To address the hardware aspect, we conducted additional experiments on an NVIDIA V100 GPU, beyond the RTX 5090 used in the original paper. We compared Taylor-WSM and Nested-WSM on four datasets using the same Vanilla deep STPP model, training configurations, and timing procedures, with JAX/JAXLIB 0.6.2 and float64 precision. The following table reports the forward and full training-step speedups.
Table 4. NVIDIA V100 evaluation of Taylor-WSM vs. Nested-WSM on four Vanilla deep STPP datasets (JAX, float64). Forward and full-step speedups are Nested/Taylor runtime ratios; values favor Taylor-mode. On the V100, Taylor-mode achieves 3.00–8.31$\times$ forward and 2.98–7.49$\times$ full-step speedups.
| Dataset | Forward Speedup | Full-Step Speedup |
|---|---|---|
| SC1 | 8.31$\times$ | 7.49$\times$ |
| AutoInt-2 | 4.61$\times$ | 3.75$\times$ |
| AutoInt-3 | 3.87$\times$ | 3.18$\times$ |
| Earthquake | 3.00$\times$ | 2.98$\times$ |
The results show that Taylor-mode remains faster on this additional GPU, achieving 3.00–8.31$\times$ forward speedups and 2.98–7.49$\times$ full training-step speedups across the four datasets.
Regarding non-JAX settings, we have not yet conducted experiments in other frameworks. This would require re-implementing Taylor-mode on the target framework's AD and execution stack and optimizing the corresponding kernels, which is substantial infrastructure work beyond the scope of the current study. Nevertheless, we believe that cross-framework evaluation is a valuable direction for future research on the computational efficiency of higher-order AD.
Response to Critical Weakness 2
Critical Weakness 2. The evaluation is limited to JAX on a specific GPU platform and excludes compilation time. Since Taylor-mode differentiation may increase computation graph complexity, the authors should assess whether compilation overhead affects end-to-end training time and the practical speedups.
Furthermore, the pipeline might demonstrate better benefits where the workload has opportunities to cache or reuse intermediate computations (eg, video object detection). Evaluating these factors would strengthen the claims regarding the method's practical utility, ease of deployment and generalizability.
Response. Thank you for your thoughtful comments on the practical efficiency and broader applicability of our approach. Regarding the evaluation on other GPU devices and non-JAX frameworks, we address these concerns in our response to Question 1, where we report additional GPU experiments and discuss cross-framework evaluation.
Regarding compilation overhead, our original experiments report steady-state training-step runtimes, excluding initial compilation overhead. Specifically, we record the runtime of each training step and begin measurement only after the runtimes stabilize, with the warm-up period determined separately for each device and configuration.
To further assess whether compilation overhead affects the observed speedups, we additionally measured the compilation times of Taylor-mode and Nested AD on four datasets using the same Vanilla deep STPP model. All measurements were conducted on an NVIDIA A800-SXM4-80GB GPU with JAX/JAXLIB 0.6.2 and float64 precision.
Table 3: Compilation times (seconds) of Nested AD and Taylor-mode on four deep STPP datasets. Reduction (%) is the percentage reduction relative to Nested AD.
| Dataset | Nested AD | Taylor-mode | Reduction |
|---|---|---|---|
| SC1 | 33.500 | 19.472 | 41.88% |
| AutoInt-2 | 32.809 | 19.308 | 41.15% |
| AutoInt-3 | 33.521 | 19.980 | 40.40% |
| Earthquake | 32.801 | 19.300 | 41.16% |
As shown in the table, Taylor-mode has 40.40%–41.88% lower compilation times than Nested AD across the four datasets. Since Taylor-mode also achieves faster steady-state training steps, including compilation overhead would not reverse its efficiency advantage under the tested configurations.
We also appreciate your suggestion to explore caching and intermediate-computation reuse. Such techniques are valuable in applications like video object detection, where intermediate features can be reused across frames. However, standard video object detection does not typically require the higher-order coordinate derivatives considered in our WSM setting. In addition, caching may benefit both AD strategies, so its effect on their relative efficiency requires further investigation. We believe that studying computation reuse in other higher-order AD applications would be a valuable direction for future research.
Thank you for your thorough and constructive review, as well as your insightful comments and suggestions. We address your Critical Weaknesses and Questions for Authors in the following responses. For closely related concerns, we provide a joint response to avoid repetition while ensuring that each point is addressed.
Response to Critical Weakness 1
Response. Thank you for this constructive suggestion. We agree that comparing against alternative higher-order AD strategies, in terms of runtime, memory usage, and training efficiency, would strengthen the evidence for our approach. We therefore added a comparison between Taylor-mode and Forward-over-Reverse (FoR) AD in Appendix G. Both methods are evaluated on four datasets using the same Vanilla deep STPP model and experimental settings; implementation details and results are given in Appendix G.
Since FoR and Taylor-mode are two exact AD implementations of the same WSM objective, matched training conditions give the same training trajectory and thus the same final loss; the only relevant difference is runtime. We therefore compare the two methods in terms of (i) forward and full training-step runtimes and (ii) peak GPU memory during the full training step. Table 1 reports the runtimes, and Table 2 reports the memory usage.
Table 1. Forward-over-Reverse (FoR) vs. Taylor-mode AD on four Vanilla deep STPP datasets. Forward and full training-step runtimes are in seconds; speedup is FoR/Taylor, so values >1 favor Taylor-mode. Taylor-mode achieves 2.51–6.83$\times$ forward and 2.39–4.51$\times$ full-step speedups.
Table 2. Peak GPU memory (MiB) during a complete training step for FoR and Taylor-mode AD on four Vanilla deep STPP datasets. Lower memory is better. Memory Reduction is the percentage saved by Taylor-mode relative to FoR; positive values favor Taylor-mode. Taylor-mode reduces peak GPU memory by 9.3%–11.3% across all four datasets.
Tables 1 and~2 show that Taylor-mode is faster and uses less GPU memory than FoR. It achieves 2.51–6.83$\times$ forward speedups and 2.39–4.51$\times$ full training-step speedups, and reduces peak GPU memory by 9.3%–11.3% on all four datasets. These results show that Taylor-mode is more efficient in both training time and memory. This provides stronger empirical evidence of its practical benefits over another exact higher-order AD strategy, as suggested by the reviewer.