- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies model criticism for temporal point processes when only binned event counts are observed. Instead of imputing one exact within-bin event-time sequence and applying standard time-rescaling diagnostics, the paper evaluates the distribution induced on the observed counts while treating the missing event history as latent uncertainty.
Critical strengths
- The central idea is simple and convincing: test the observed record, not an arbitrary imputation.
- The Monte Carlo calibration gives the procedure strong statistical grounding even when particle forecasts are approximate.
Critical weaknesses
- My main concern is scalability. The particle-filter implementation works well in the evaluated settings, but it is unclear how it behaves for long histories, high-dimensional latent states, or marked/multivariate TPPs where particle degeneracy may become more severe.
Questions for authors
- The authors needs to provide a more comprehensive study on the scalability/efficiency of the proposed methods, especially when dimensionality becomes high, sampling SMC can be burdensome.
- Has the author studied the accumlation of error over long horizons of SMC? How does that affect the power and applicability of the developed test.
- The current particle formulation marginalizes uncertainty over latent event histories while treating the fitted TPP parameters as fixed. Could the framework be extended to propagate parameter uncertainty as well, yielding a Bayesian posterior-predictive critic with uncertainty estimates on the discrepancy? How would such an approach relate to the current finite-sample Monte Carlo calibration, and would the additional nested inference remain computationally feasible?
4 comments
Response to Reviewer WByn (2/2): parameter uncertainty
Q3. The current particle formulation marginalizes uncertainty over latent event histories while treating the fitted TPP parameters as fixed. Could the framework be extended to propagate parameter uncertainty as well, yielding a Bayesian posterior-predictive critic with uncertainty estimates on the discrepancy? How would such an approach relate to the current finite-sample Monte Carlo calibration, and would the additional nested inference remain computationally feasible?
Yes, and it fits the existing calibration, keeps the finite-sample guarantee, and costs about as much as the current test. Each null group is simulated from one draw of the parameters, and every record is scored with particles that each carry a parameter label, so the forecast mixes over parameters. Under the posterior predictive null, the observed group is distributed like the simulated groups, and all of them are scored in the same way. The statistics therefore remain exchangeable, and the Monte Carlo p-value stays valid in finite samples.
We tried this on an exponential Hawkes process fitted by maximum likelihood, with 8 bootstrap fits as the parameter draws (Appendix A.6). The nominal level is 4%.
| Fixed fit | Refit in every null group | Posterior predictive | |
|---|---|---|---|
| False rejection, 30 s of training data | 11.3% | 6.3% | 5.0% |
| False rejection, 100 or 1,000 s | 2–5% | 2–5% | 2–5% |
| Time per test | about 45 s | 43–163 s | 43–68 s |
Against a renewal alternative, the three versions have similar power.
Two limits remain. The parameter labels degenerate over long records (3 to 4 of the 8 survive after 40 s). For neural generators each refit takes minutes, and we have not tested that case. The Limitations now note that, when parameters are estimated from a short record, a rejection can also reflect estimation error.
I like the originality of the work and the fact that the paper is addressing an important question in the field. I will raise my score
Thank you for reading our responses and for raising your score. We are glad that you find the question important. Your comments led to the new experiments on long records, larger hidden states, multivariate processes and parameter uncertainty, and helped us improve the paper.
Response to Reviewer WByn (1/2): scaling and accumulation of SMC error
Thank you for the positive assessment. Your three questions go to the practical limits of the method, and the new Appendix A.6 answers them. We reply in two parts, with your comments quoted in italics.
W1. My main concern is scalability. The particle-filter implementation works well in the evaluated settings, but it is unclear how it behaves for long histories, high-dimensional latent states, or marked/multivariate TPPs where particle degeneracy may become more severe.
Q1. The authors needs to provide a more comprehensive study on the scalability/efficiency of the proposed methods, especially when dimensionality becomes high, sampling SMC can be burdensome.
On both points, the time to score a record grows linearly with its length, and particle degeneracy stays moderate as the latent dimension, the event rate and the number of event types grow (Appendix A.6).
Here the ESS ratio is the effective sample size divided by the number of particles. Calibration multiplies the cost by the number of null groups, which run independently and in parallel.
We have not tested continuous marks, which would need a matching proposal. Appendix A.1 says so.
Q2. Has the author studied the accumlation of error over long horizons of SMC? How does that affect the power and applicability of the developed test.
The error does not accumulate, and power is nearly the same with particle forecasts and with exact forecasts. We checked this on a Markov-modulated Poisson process, for which the count forecasts can be computed exactly.
Whatever the particle error, the Monte Carlo calibration covers the test as it is run. Particle error can cost power, but it cannot raise the false rejection rate. Path degeneracy does occur. It is harmless here because this model forgets its history, and we make no such claim for models with very long memory.