acceptodds
Under review as a conference paper at ICLR 2027

Accelerating Weighted Score Matching for Point Processes

Abstract

Weighted score matching (WSM) avoids intractable intensity integration in the likelihood for point processes, but nested reverse-mode automatic differentiation (AD) for event-coordinate derivatives introduces a training bottleneck. To solve this issue, we propose Taylor-mode WSM, a plug-and-play method that replaces the nested derivative evaluator with a Taylor-mode evaluator while retaining reverse-mode AD for parameter updates. To explain and quantify the acceleration, we develop a program-level cost framework that derives the training-step speedup ratio, characterizes when the speedup is most pronounced, and establishes why the Taylor-mode evaluator is faster. Through experiments across statistical point processes, deep temporal point processes, and deep spatio-temporal point processes (STPPs), our framework explains the varying speedups across the three model families and further reveals that outer backpropagation is the dominant source of acceleration, accounting for 90.4%-99.6% of the total time reduction. Additional studies show that the speedup becomes more pronounced at higher derivative orders, while the advantage persists with increasing event-coordinate dimension. Overall, Taylor-mode WSM achieves an average speedup of and up to on deep STPPs, without sacrificing model performance.

Primary area:
learning on time series and dynamical systems
Keywords:
point processes, score matching, Taylor-mode differentiation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xi4ROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a Taylor-approximated weighted score matching to efficiently replace nested reverse-mode automatic differentiation for training point-process models with weighted score matching. The authors propose a plug and play method to implement the same. The proposed method shares the common subset of computations across multiple methods, so that those intermediate results can be reused and we can observe a lower cost.

Critical strengths

A> The proposed method is dynamic - it takes into consideration the intermediate steps and then takes a decision

B> The implementation is clear and neat - it will be easy to plug and play on multiple downstream tasks.

Critical weaknesses

A> Taylor-mode automatic differentiation has been explored in prior work (e.g., [1]). The contribution of this paper is its application to WSM, together with a program-level cost analysis and empirical evaluation. However, the evaluation would be more convincing with comparisons against alternative higher-order differentiation strategies under the same experimental setup, including runtime, memory consumption, and end-to-end training efficiency. This would better establish the practical benefits of the proposed approach.

B> The evaluation is limited to JAX on a specific GPU platform and excludes compilation time. Since Taylor-mode differentiation may increase computation graph complexity, the authors should assess whether compilation overhead affects end-to-end training time and the practical speedups.

Furthermore, the pipeline might demonstrate better benefits where the workload has opportunities to cache or reuse intermediate computations (eg, video object detection). Evaluating these factors would strengthen the claims regarding the method's practical utility, ease of deployment and generalizability.

[1] Bettencourt, Jesse, Matthew J. Johnson, and David Duvenaud. "Taylor-mode automatic differentiation for higher-order derivatives in JAX." Program transformations for ML workshop at NeurIPS 2019. 2019.

Questions for authors

The paper is an interesting read. Please quantify your acceleration, preferably on other devices, and in non-JAX settings.

3 comments

Thank you for your thorough and constructive review, as well as your insightful comments and suggestions. We address your Critical Weaknesses and Questions for Authors in the following responses. For closely related concerns, we provide a joint response to avoid repetition while ensuring that each point is addressed.

Response to Critical Weakness 1

Critical Weakness 1. Taylor-mode automatic differentiation has been explored in prior work. The contribution of this paper is its application to WSM, together with a program-level cost analysis and empirical evaluation. However, the evaluation would be more convincing with comparisons against alternative higher-order differentiation strategies under the same experimental setup, including runtime, memory consumption, and end-to-end training efficiency. This would better establish the practical benefits of the proposed approach.

Response. Thank you for this constructive suggestion. We agree that comparing against alternative higher-order AD strategies, in terms of runtime, memory usage, and training efficiency, would strengthen the evidence for our approach. We therefore added a comparison between Taylor-mode and Forward-over-Reverse (FoR) AD in Appendix G. Both methods are evaluated on four datasets using the same Vanilla deep STPP model and experimental settings; implementation details and results are given in Appendix G.

Since FoR and Taylor-mode are two exact AD implementations of the same WSM objective, matched training conditions give the same training trajectory and thus the same final loss; the only relevant difference is runtime. We therefore compare the two methods in terms of (i) forward and full training-step runtimes and (ii) peak GPU memory during the full training step. Table 1 reports the runtimes, and Table 2 reports the memory usage.

Table 1. Forward-over-Reverse (FoR) vs. Taylor-mode AD on four Vanilla deep STPP datasets. Forward and full training-step runtimes are in seconds; speedup is FoR/Taylor, so values >1>1 favor Taylor-mode. Taylor-mode achieves 2.51–6.83$\times$ forward and 2.39–4.51$\times$ full-step speedups.

DatasetFoR Fwd.Taylor Fwd.Fwd. SpeedupFoR StepTaylor StepStep Speedup
SC12.6680.4046.60$\times$7.0931.5814.49$\times$
AutoInt-215.3362.2466.83$\times$34.5507.6564.51$\times$
AutoInt-318.1035.4013.35$\times$37.38413.1322.85$\times$
Earthquake22.3438.8912.51$\times$39.41516.5102.39$\times$

Table 2. Peak GPU memory (MiB) during a complete training step for FoR and Taylor-mode AD on four Vanilla deep STPP datasets. Lower memory is better. Memory Reduction is the percentage saved by Taylor-mode relative to FoR; positive values favor Taylor-mode. Taylor-mode reduces peak GPU memory by 9.3%–11.3% across all four datasets.

DatasetFoRTaylorMemory Reduction
SC11883.91671.311.3%
AutoInt-21783.11592.710.7%
AutoInt-31682.51520.79.6%
Earthquake961.4871.69.3%

Tables 1 and~2 show that Taylor-mode is faster and uses less GPU memory than FoR. It achieves 2.51–6.83$\times$ forward speedups and 2.39–4.51$\times$ full training-step speedups, and reduces peak GPU memory by 9.3%–11.3% on all four datasets. These results show that Taylor-mode is more efficient in both training time and memory. This provides stronger empirical evidence of its practical benefits over another exact higher-order AD strategy, as suggested by the reviewer.

Response to Question 1

Question 1. The paper is an interesting read. Please quantify your acceleration, preferably on other devices, and in non-JAX settings.

Response. Thank you for this constructive suggestion. We agree that evaluating the acceleration on different hardware and AD frameworks is important for understanding its practical applicability. To address the hardware aspect, we conducted additional experiments on an NVIDIA V100 GPU, beyond the RTX 5090 used in the original paper. We compared Taylor-WSM and Nested-WSM on four datasets using the same Vanilla deep STPP model, training configurations, and timing procedures, with JAX/JAXLIB 0.6.2 and float64 precision. The following table reports the forward and full training-step speedups.

Table 4. NVIDIA V100 evaluation of Taylor-WSM vs. Nested-WSM on four Vanilla deep STPP datasets (JAX, float64). Forward and full-step speedups are Nested/Taylor runtime ratios; values >1>1 favor Taylor-mode. On the V100, Taylor-mode achieves 3.00–8.31$\times$ forward and 2.98–7.49$\times$ full-step speedups.

DatasetForward SpeedupFull-Step Speedup
SC18.31$\times$7.49$\times$
AutoInt-24.61$\times$3.75$\times$
AutoInt-33.87$\times$3.18$\times$
Earthquake3.00$\times$2.98$\times$

The results show that Taylor-mode remains faster on this additional GPU, achieving 3.00–8.31$\times$ forward speedups and 2.98–7.49$\times$ full training-step speedups across the four datasets.

Regarding non-JAX settings, we have not yet conducted experiments in other frameworks. This would require re-implementing Taylor-mode on the target framework's AD and execution stack and optimizing the corresponding kernels, which is substantial infrastructure work beyond the scope of the current study. Nevertheless, we believe that cross-framework evaluation is a valuable direction for future research on the computational efficiency of higher-order AD.

Response to Critical Weakness 2

Critical Weakness 2. The evaluation is limited to JAX on a specific GPU platform and excludes compilation time. Since Taylor-mode differentiation may increase computation graph complexity, the authors should assess whether compilation overhead affects end-to-end training time and the practical speedups.

Furthermore, the pipeline might demonstrate better benefits where the workload has opportunities to cache or reuse intermediate computations (eg, video object detection). Evaluating these factors would strengthen the claims regarding the method's practical utility, ease of deployment and generalizability.

Response. Thank you for your thoughtful comments on the practical efficiency and broader applicability of our approach. Regarding the evaluation on other GPU devices and non-JAX frameworks, we address these concerns in our response to Question 1, where we report additional GPU experiments and discuss cross-framework evaluation.

Regarding compilation overhead, our original experiments report steady-state training-step runtimes, excluding initial compilation overhead. Specifically, we record the runtime of each training step and begin measurement only after the runtimes stabilize, with the warm-up period determined separately for each device and configuration.

To further assess whether compilation overhead affects the observed speedups, we additionally measured the compilation times of Taylor-mode and Nested AD on four datasets using the same Vanilla deep STPP model. All measurements were conducted on an NVIDIA A800-SXM4-80GB GPU with JAX/JAXLIB 0.6.2 and float64 precision.

Table 3: Compilation times (seconds) of Nested AD and Taylor-mode on four deep STPP datasets. Reduction (%) is the percentage reduction relative to Nested AD.

DatasetNested ADTaylor-modeReduction
SC133.50019.47241.88%
AutoInt-232.80919.30841.15%
AutoInt-333.52119.98040.40%
Earthquake32.80119.30041.16%

As shown in the table, Taylor-mode has 40.40%–41.88% lower compilation times than Nested AD across the four datasets. Since Taylor-mode also achieves faster steady-state training steps, including compilation overhead would not reverse its efficiency advantage under the tested configurations.

We also appreciate your suggestion to explore caching and intermediate-computation reuse. Such techniques are valuable in applications like video object detection, where intermediate features can be reused across frames. However, standard video object detection does not typically require the higher-order coordinate derivatives considered in our WSM setting. In addition, caching may benefit both AD strategies, so its effect on their relative efficiency requires further investigation. We believe that studying computation reuse in other higher-order AD applications would be a valuable direction for future research.

Reviewer_FTa5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WSM avoids the intractable intensity integral in point-process likelihood training but needs first- and second-order derivatives w.r.t. event coordinates, normally obtained by nested reverse-mode AD, which becomes the bottleneck. The paper replaces that inner evaluator with Taylor-mode AD (propagating truncated Taylor coefficients along each coordinate direction, yielding the score and diagonal Hessian in one sweep) while keeping reverse-mode AD for the parameter gradient. The second contribution is a program-level cost framework decomposing a training step into shared cost, derivative-program forward cost and outer-backpropagation cost, giving multipliers τ_tay = dα₂ and τ_nest = (1+ρ⁽¹⁾)(1+dρ⁽²⁾), two sufficient conditions for speedup, and a "shared-cost dilution" effect. Experiments cover statistical Hawkes, deep TPPs (SAHP, THP) and deep STPPs (SMASH, two parameterizations), with numerical-equivalence checks, a DSM-vs-WSM derivative-order ablation and a dimension sweep.

Critical strengths

The framework locates the mechanism, and the answer is not the obvious one. One would expect the win from cheaper forward second-order evaluation; the measurements show R_forward ≈ 1.0–1.6, while 90.4–99.6% of the training-step saving comes from the outer backpropagation traversing a smaller derivative graph (Table 2). That diagnosis — that nested reverse mode is expensive because of the graph the parameter gradient runs through, not the per-coordinate passes — is useful and transfers beyond this specific substitution. It explains the cross-family pattern (shared-cost ratio 97.8% → 66.0%, R_step 1.32× → 3.85×) without appealing to implementation luck. The paper is also notably disciplined in Appendix E about distinguishing wall-clock ratios from the unit-cost inequalities, which makes the framework more credible rather than less.

Critical weaknesses

The methodological contribution is a known substitution, and only its accounting is new. Taylor-mode AD for higher-order derivatives is standard (Griewank et al. 2000), already available in JAX as jacfwd, and "use Taylor mode for coordinate derivatives, reverse mode for parameters" is the first thing one would try; calling the method a "drop-in replacement" concedes this while the contribution claims are phrased otherwise. What is genuinely new is the cost model, and it is validated on exactly one substitution in one framework. The framework would earn its place if it predicted a speedup for an untested variant (forward-over-reverse Hessians, vectorising over events, vmap-based baselines), and that is not attempted.

The performance claim rests only on numerical equivalence. Section 5.2 confirms the two evaluators agree to ~1e-14, which is a correctness check, not a quality result. There is no downstream evaluation on the real datasets (StackOverflow, Taobao, Japan Earthquake) — no test log-likelihood, no event-time error — and no end-to-end comparison against MLE, which is the comparison WSM exists to win. Relatedly, the Table 8 timings invite scepticism: SAHP on conditional-logistic spends 1.21 s forward and 8.76 s outer, Vanilla SMASH on Earthquake 0.47 s forward and 38.2 s outer, for models with d_model = 16 and one to four layers. Those ratios suggest per-event Python control flow rather than anything intrinsic to nested AD, in which case part of R_step reflects how poorly the baseline was compiled.

Questions for authors
  1. Can you compare against MLE (Monte Carlo or quadrature) on the same deep STPP models? Is Taylor-WSM now faster end-to-end than MLE, or only faster than nested-WSM?
  2. Can you report a downstream metric on the real datasets (test log-likelihood under the fitted intensity, event-time/space prediction error) for TAYLOR, NESTED and MLE? Numerical equivalence is a correctness check, not a performance result.
  3. A 10 s deep-TPP step with d_model = 16 and one layer, and a 38 s deep-STPP step, look dominated by overhead. Does the speedup survive when the baseline is vectorised across events rather than looped?
  4. Does the framework predict anything for an untested substitution, e.g. forward-over-reverse for the diagonal Hessian? A confirmed prediction would establish the framework rather than the implementation as the contribution.
  5. The dimension ablation shows R_forward < 1 from d_s = 6 onward. At d_s = 32 or 64, where the dα₂ forward term dominates, is there a crossover beyond which Taylor mode loses overall?
5 comments

Thank you for your thorough and constructive review, as well as your insightful comments and suggestions. We address your Critical Weaknesses and Questions for Authors in the following responses. For closely related concerns, we provide a joint response to avoid repetition while ensuring that each point is addressed.

Response to Critical Weakness 1 and Question 4

We address Critical Weakness 1 and Question 4 together, as both ask whether the proposed cost analysis generalizes beyond the original comparison between Taylor-mode and Nested AD.

Critical Weakness 1. The methodological contribution is a known substitution, and only its accounting is new. Taylor-mode AD for higher-order derivatives is standard (Griewank et al. 2000), already available in JAX as jacfwd, and "use Taylor mode for coordinate derivatives, reverse mode for parameters" is the first thing one would try; calling the method a "drop-in replacement" concedes this while the contribution claims are phrased otherwise. What is genuinely new is the cost model, and it is validated on exactly one substitution in one framework. The framework would earn its place if it predicted a speedup for an untested variant (forward-over-reverse Hessians, vectorising over events, vmap-based baselines), and that is not attempted.

Question 4. Does the framework predict anything for an untested substitution, e.g. forward-over-reverse for the diagonal Hessian? A confirmed prediction would establish the framework rather than the implementation as the contribution.

Response. Thank you for your thoughtful comments on the contribution of our work and the applicability of our cost analysis to other AD strategies. We agree that Taylor-mode AD itself is an established technique. Our main contribution is a program-level cost analysis that explains when and why Taylor-mode can accelerate WSM training. Importantly, we show that the acceleration comes not only from more efficient forward derivative computation, but also, and more significantly, from savings in outer backpropagation. In our Deep STPP experiments, outer backpropagation accounts for 90.4%–99.6% of the total training-step time savings, as Taylor-mode avoids the nested reverse-mode derivative program traversed during outer backpropagation.

To further examine whether our cost analysis applies beyond the original Nested AD comparison, we have added a new experiment with Forward-over-Reverse (FoR) AD in Appendix G. Following the cost mechanism discussed in Section 4, FoR constructs its derivative program using reverse-mode gradients followed by JVPs, while Taylor-mode directly propagates Taylor coefficients. We therefore expect FoR to exhibit a larger outer-backpropagation cost amplification, and investigate whether this difference is also reflected in actual training efficiency.

We evaluate both methods using the Vanilla deep STPP model on four datasets, comparing runtime, FLOP amplification, and peak GPU memory. The results show that FoR has a larger outer-backpropagation FLOP amplification factor (2.029–2.089) than Taylor-mode (1.417–1.438), consistent with our cost analysis. Moreover, Taylor-mode achieves 2.39–4.51× full training-step speedups, 2.51–6.83× forward speedups, and 9.3%–11.3% lower peak GPU memory across the four datasets.

Response to Critical Weakness 2 and Question 3

We address Critical Weakness 2 and Question 3 together, as both question whether the reported efficiency improvements rely on inadequately optimized baselines.

Critical Weakness 2. The performance claim rests only on numerical equivalence. Section 5.2 confirms the two evaluators agree to ~1e-14, which is a correctness check, not a quality result. There is no downstream evaluation on the real datasets (StackOverflow, Taobao, Japan Earthquake) — no test log-likelihood, no event-time error — and no end-to-end comparison against MLE, which is the comparison WSM exists to win. Relatedly, the Table 8 timings invite scepticism: [...] Those ratios suggest per-event Python control flow rather than anything intrinsic to nested AD, in which case part of R_step reflects how poorly the baseline was compiled.

Question 3. A 10 s deep-TPP step with d_model = 16 and one layer, and a 38 s deep-STPP step, look dominated by overhead. Does the speedup survive when the baseline is vectorised across events rather than looped?

Response. Thank you for your detailed and constructive comments. We agree that numerical consistency alone is insufficient to establish downstream model quality, and we appreciate your concerns about the efficiency of the original baselines. To address these concerns, we have added downstream evaluations of Taylor-WSM and Nested-WSM on three real-world datasets in Section 5.2, as well as comparisons of Taylor-WSM and MLE in both training efficiency and downstream performance in Appendix H. All newly added results are highlighted in red in the revised manuscript.

Specifically, Section 5.2 compares the downstream performance of Taylor-WSM and Nested-WSM on StackOverflow, Taobao, and Earthquake, covering both temporal and spatio-temporal tasks. Appendix H further compares Taylor-WSM with MLE on StackOverflow and Taobao. The table below summarizes the test log-likelihood per event and next-event time prediction errors, together with spatial prediction error for Earthquake. MLE results use K=64K=64 Gauss–Legendre quadrature points (other budgets: Appendix H).

Table 1. Downstream test performance of Taylor-WSM, Nested-WSM, and MLE on StackOverflow, Taobao, and Earthquake. Higher LL/event and lower time MAE, time RMSE, and spatial RMSE are better; a dash indicates an inapplicable metric. Taylor-WSM and Nested-WSM match on StackOverflow/Taobao and differ by less than 1% on Earthquake; MLE achieves higher test LL/event on the two TPP datasets.

DatasetMethodTest LL/event ↑\uparrowTime MAE ↓\downarrowTime RMSE ↓\downarrowSpatial RMSE ↓\downarrow
StackOverflowTaylor-WSM-2.6570.8111.172–
Nested-WSM-2.6570.8111.172–
MLE (K=64K=64)-2.3460.7121.040–
TaobaoTaylor-WSM0.5690.4530.487–
Nested-WSM0.5690.4530.487–
MLE (K=64K=64)0.6460.5270.705–
EarthquakeTaylor-WSM-5.60520.29770.409110.4243
Nested-WSM-5.61570.29830.411310.4285

The results show that Taylor-WSM and Nested-WSM achieve identical downstream metrics to the reported precision on StackOverflow and Taobao. On Earthquake, their differences across all four metrics are below 1%, indicating closely matched temporal and spatial predictive performance. Although the table shows that Taylor-WSM and MLE differ in training performance, this does not contradict their consistency theory: the two estimators converge to the same underlying model as the sample size grows to infinity. Therefore, in our finite-sample setting, MLE may perform slightly better in our experiments due to its lower asymptotic variance. The full training-step runtime comparison between Taylor-WSM and MC-MLE under different sampling budgets is provided in Appendix H and discussed in our response to Question 1.

Regarding the concern about baseline efficiency, we carefully checked the original implementations and confirmed that they already use event-level vectorization. Specifically, Deep TPP computes derivatives on batched event-time tensors, while Deep STPP uses jax.vmap across events. Therefore, the reported speedups are not caused by replacing per-event Python loops with vectorized computation.

As for the relatively high outer-backpropagation times in Table 8 of the manuscript, this is consistent with the cost mechanism analyzed in Section 4. Nested AD constructs a derivative program involving nested differentiation, and the outer parameter backward pass must differentiate through this program again. This can make outer backpropagation considerably more expensive than forward derivative computation. In contrast, Taylor-mode avoids constructing the nested reverse-mode derivative program, thereby reducing the overhead of outer parameter backpropagation.

Response to Question 1

Question 1. Can you compare against MLE (Monte Carlo or quadrature) on the same deep STPP models? Is Taylor-WSM now faster end-to-end than MLE, or only faster than nested-WSM?

Response. Thank you for this valuable suggestion. To compare Taylor-WSM with MLE, we have added an MC-MLE baseline in Appendix H of the revised manuscript. We compare their full training-step runtimes on four datasets (SC1, AutoInt-2, AutoInt-3, and Earthquake) using the same Vanilla deep STPP model and experimental settings. For MC-MLE, we test four sampling budgets, K∈{64,256,1024,2048}K\in\{64,256,1024,2048\}, where KK is the number of Monte Carlo integration samples. The following table reports the mean full training-step runtimes after warm-up.

Table 2. Full training-step runtime (seconds) of Taylor-WSM and Monte Carlo MLE (MC-MLE) on four Vanilla deep STPP datasets. KK is the number of Monte Carlo integration samples. MC-MLE is faster for K≤1024K\leq1024; Taylor-WSM is faster on AutoInt-3 at K=2048K=2048.

DatasetTaylor-WSMMC-MLE K=64K{=}64K=256K{=}256K=1024K{=}1024K=2048K{=}2048
SC11.3901.0651.0841.1621.363
AutoInt-21.3961.0711.0781.1151.311
AutoInt-31.4021.0561.0841.2471.559
Earthquake1.3701.0721.0531.1051.285

The results show that MC-MLE is faster per training step on all four datasets for K≤1024K \leq 1024. Its runtime advantage generally narrows as the Monte Carlo sample size increases. At K=2048K = 2048, Taylor-WSM is faster on AutoInt-3, taking 1.4021.402 seconds per step compared with 1.5591.559 for MC-MLE. Our original experiments show that Taylor-mode differentiation substantially accelerates WSM relative to nested automatic differentiation. The comparison with MC-MLE further shows that the relative per-step cost depends on the Monte Carlo sample size.

In practice, the choice of KK involves a trade-off between runtime and integration accuracy. With too few samples, the Monte Carlo estimate of the integrated intensity can be noisy, which can degrade MC-MLE training. When a larger KK is needed for accurate integration, Taylor-WSM can be competitive or faster per step, as observed on AutoInt-3.

Reviewer_yVgdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a way to rethink how the derivatives in weighted score matching are evaluated. It uses second-order automatic differentiation for event times and spatial coordinates, then reverse-mode differentiation for the parameter update. The loss and model keep unchanged. The experiments compare this implementation with nested reverse-mode differentiation on statistical Hawkes models, temporal point processes, and spatio-temporal point processes. The largest gains are in the last group, where the reported training-step speedups range from 2.09 to 5.59. A cost decomposition separates the work shared by both implementations from derivative evaluation and backpropagation through that evaluation.

Critical strengths

(1) The improvement preserves the original objective and model. This makes it easier to use in an existing WSM implementation, and avoids a tradeoff between cheaper training and a different estimator. The appendix section also explains why the required derivatives agree in exact arithmetic.

(2) The complete-step gains on deep STPPs are substantial across both model parameterizations and all four datasets, rather than being limited to a single example. Some table also reports the forward slowdowns at larger spatial dimensions instead of reporting only the favorable complete step numbers.

Critical weaknesses

(1) The explanation of the speedup is a littile bit weak. Section 4 gives a useful accounting decomposition, but leaves the key cost multipliers unspecified.

Questions for authors

(1) How does Taylor-mode compare with a forward-over-reverse implementation that computes the exact required Hessian diagonal? Please give complete training-step timings and describe the AD transformations, vectorization, and compilation used for both implementations.

(2) Do you use any estimator for Hessian digonal?

5 comments

Thank you for your thorough and constructive review, as well as your insightful comments and suggestions. We address your Critical Weaknesses and Questions for Authors in the following responses. For closely related concerns, we provide a joint response to avoid repetition while ensuring that each point is addressed.

Response to Critical Weakness 1

Critical Weakness 1. The explanation of the speedup is a littile bit weak. Section 4 gives a useful accounting decomposition, but leaves the key cost multipliers unspecified.

Response. Thank you for highlighting the need to clarify the cost multipliers. Section 4 defines these cost multipliers as ratios between the costs of computational programs, but their values are not fixed across models. This is because models differ in their operators and computation graphs, which in turn affect the derivative programs generated by AD transformations. Consequently, the operation counts of the primal and derivative programs need not scale proportionally, making these multipliers dependent on the specific implementation.

Nevertheless, we can still compare the computational costs of different AD methods through the relationships between these multipliers. Specifically, Section 4.4 provides two sufficient conditions for Taylor-mode to achieve a lower full training-step cost:

\qquad \rho_{\mathrm{tay}}\leq\rho_{\mathrm{nest}}.$$ Section 5.3 provides empirical support for these inequalities: Table 1 of the manuscript reports lower forward runtimes for Taylor-mode than Nested, while Table 2 of the manuscript shows smaller outer-backpropagation FLOP amplification factors.

Response to Question 1

Question 1. How does Taylor-mode compare with a forward-over-reverse implementation that computes the exact required Hessian diagonal? Please give complete training-step timings and describe the AD transformations, vectorization, and compilation used for both implementations.

Response. Thank you for your valuable suggestion. In response, we have added a comparison between Taylor-mode and Forward-over-Reverse (FoR) AD in Appendix G of the revised manuscript. FoR computes the required Hessian diagonal using reverse-mode gradients followed by forward-mode JVPs, while Taylor-mode directly propagates second-order Taylor coefficients.

We evaluate both methods on four datasets (SC1, AutoInt-2, AutoInt-3, and Earthquake) using the same Vanilla deep STPP model, training settings, event-level jax.vmap vectorization, and JAX compilation and timing procedures. Detailed implementations and experimental settings are provided in Appendix G. To quantify their efficiency differences, we report the forward and full training-step runtimes below, where speedup is defined as the FoR runtime divided by the Taylor-mode runtime.

Table 1. FoR versus Taylor-mode AD on four Vanilla deep STPP datasets. Both forward and full training-step runtimes are in seconds; speedup is FoR/Taylor, so values above 11 favor Taylor-mode. Taylor-mode is 2.51–6.83$\times$ faster in the forward phase and 2.39–4.51$\times$ faster per full training step.

DatasetFoR Fwd. (s)Taylor Fwd. (s)Fwd. SpeedupFoR Step (s)Taylor Step (s)Step Speedup
SC12.6680.4046.60$\times$7.0931.5814.49$\times$
AutoInt-215.3362.2466.83$\times$34.5507.6564.51$\times$
AutoInt-318.1035.4013.35$\times$37.38413.1322.85$\times$
Earthquake22.3438.8912.51$\times$39.41516.5102.39$\times$

As shown in the table, Taylor-mode consistently achieves lower runtimes across all four datasets, with forward speedups of 2.51–6.83$\times$ and full training-step speedups of 2.39–4.51$\times$. Notably, this efficiency advantage remains even when FoR computes only the required Hessian diagonal without explicitly constructing the full Hessian.

Response to Question 2

Question 2. Do you use any estimator for Hessian digonal?

Response. Thank you for your question. We do not use any Hessian-diagonal estimator. Instead, Taylor-mode AD directly computes the required diagonal entries. As described in Section 3, for each spatial coordinate jj, we propagate second-order Taylor coefficients along the unit direction ej\mathbf{e}_j:

Fj(ϵ)=log⁡qθ(s+ϵej∣t,H).F_j(\epsilon) =\log q_\theta(\mathbf{s}+\epsilon\mathbf{e}_j\mid t,\mathcal{H}).

Specifically, for each spatial coordinate jj, we propagate the second-order Taylor coefficients of Fj(ϵ)=log⁡qθ(s+ϵej∣t,H)F_j(\epsilon)=\log q_\theta(\mathbf{s}+\epsilon\mathbf{e}_j \mid t,\mathcal{H}). Multiplying the coefficient of ϵ2\epsilon^2 by 2!2! gives the exact Hessian diagonal entry. Summing these entries yields the spatial score divergence, and temporal derivatives are computed analogously. Further details and the correctness proof are provided in Section 3 and Appendix A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.