- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper considers how to represent the next-event time distribution at the output of a neural network. It predicts probability masses on fixed intervals, converts these to a piecewise-constant density, and uses an exponential tail for the last, unbounded interval. The interval boundaries are chosen before network training by dynamic programming on an empirical likelihood objective. The experiments compare output heads on 14 real-world datasets with three backbones, with further analyses of boundary selection and the effect of timestamp precision on likelihood scores.
Critical strengths
- I like the decision to keep the backbone fixed when comparing output heads. This makes it much easier to see what the distributional representation contributes.
- There is quite a lot of experimental work here: multiple datasets and backbones, repeated runs, confidence intervals, and ablations on interval placement and count. The density-versus-hazard comparison and the experiments with perturbed timestamps and continuous synthetic data are also useful for understanding the results.
- The timestamp-precision analysis is, for me, one of the more interesting parts of the paper. Narrow peaks in recorded event times can dominate density-based scores, and the authors do examine this rather than just reporting the gains. The cases where log-normal mixtures do better are useful too.
Critical weaknesses
- I am less convinced by the methodological novelty. Kvamme and Borgan [1] already predict a temporal PMF with a neural network and obtain constant-density interpolation by dividing interval mass by interval width; likelihood-based irregular histograms constructed by dynamic programming also have precedents [2], which Section 4 acknowledges. As I understand it, the new part is mainly the particular boundary-selection procedure and its use for history-conditioned prediction, backed by the empirical study. That seems a narrower contribution than the overall construction might suggest, and I would like the paper to be clearer about it.
- My main concern is still the size of the likelihood gains. Section 3.1 allows bins as narrow as the local spacing of float32 values. With density equal to mass divided by width, placing these bins around repeated timestamps can make pointwise likelihood much better. On MOOC, for example, NLL goes from 1.02 to -6.23 but MAE only changes from 290.5 to 290.4 (Table 2). Of course, MAE need not improve when a predictive distribution improves. Still, I am not sure what to take from such a large NLL difference here. The jitter experiments help, but I would really like to see an evaluation using interval probabilities, with the same meaningful temporal resolution for every model.
- Another question is the use of marginal data to choose boundaries for a conditional prediction problem. The appendix examples with conditional peaks are helpful, but what happens when different histories shift the peaks to different places? I did not get a clear sense of the cost of using one global partition. A fairly simple baseline would help here: an unconditional histogram with the same optimized boundaries. It would tell us how much is gained from fitting the marginal distribution before the history encoder contributes anything.
- Appendix C.5 explains the random 6:2:2 splits and the merging of the original Taobao and Amazon splits. The authors also acknowledge the resulting limit on comparison with published results. My remaining question is the splitting unit: are whole sequences assigned to partitions, and can overlapping windows or shared histories appear across them? This needs to be explicit.
Questions for authors
- If all models are scored by interval probabilities at a shared temporal resolution, how much of the likelihood advantage remains?
- What performance does an unconditional histogram achieve with the same optimized boundaries?
- How sensitive is the global partition to history-dependent changes in peak location or distribution shape?
- What exactly is assigned to each train, validation, or test split, and how is overlap between windows or histories handled?
4 comments
Thanks again for the feedback so far; it has been helpful in improving the paper! We are eager and ready to answer any further questions you may have before the rebuttal period ends.
Thanks for running these comparisons. Appendix C.5 answers my question about how the data were split. The no-history result is also useful. On MOOC, removing event history changes the pointwise NLL from −6.23 to −6.14, so most of the gap over the log-normal mixture remains even without history.My main concern was how much weight to put on that large NLL gap. Figure 9 helps answer this. On MOOC, the two heads score nearly the same at interval widths of six seconds and one minute. The piecewise-constant head pulls ahead at finer widths. Bae et al. (ICLR 2023, §5.1) already noted the issue with pointwise NLL, but I found it useful to see how much the gap can change on real data. I am still unsure which widths are meaningful for these prediction tasks and how much of the advantage remains at those widths.The revision makes the empirical contribution clearer. I still see the edge-selection procedure as a fairly modest extension of existing ideas, though, and the new results do not change my overall rating.
Thank you for the continued engagement.
I still see the edge-selection procedure as a fairly modest extension of existing ideas
I don't think we are in strong disagreement here. However, we argue that there is more to this paper than simple score improvements. An achievement of the paper is demonstrating that the ability to concentrate density in narrow intervals is a characteristic that explains the performance differences between existing models (Section 8.2 and Appendix E). We show that the concerns you have also apply to other commonly used heads, especially the logmix head. There are many papers tabulating differences between such models, but a scarcity of explanations for score differences, and we argue that the hypotheses and evidence will be of interest to all people working in the setting of neural TPPs.
We thank the reviewer for the valuable feedback provided! We hope to address the points raised.
We definitely do not intend to overemphasize novelty claims. Actually, our main motivation for writing this paper was to use the proposed method to raise awareness of what we see as important and under-appreciated properties of model-data interaction and issues with how models are being evaluated. The investigation allowed us to present evidence-supported hypotheses for why models score differently (Appendix E). We propose to add the following paragraph to the introduction, which hopefully makes the contribution scope clear and upfront:
Additionally, please have a look at the paragraph added in response to Reviewer vn7Z comment, which extends Section 3.1.
Added, as Ablation 5 (Appendix B.5).
This is a nice control, and we have added it as Ablation 1 (Appendix B.1).
We have modified the first paragraph of Appendix C.5 (Dataset splits) to clarify this. Including it here for convenience:
Does the added ablation (ablation 5, Appendix B.5) help answer your question? The experiment's results show a similar pattern to the jitter experiment.
Does the added ablation (ablation 1, Appendix B.1) help answer your question? The ablation shows that models with and without history encoders have similar performance differences for both the logmix and pc heads, and significant NLL improvements can be attributed to the unconditional distribution with the optimized edges.
Does the above added paragraph (for Appendix C.5) help answer your question?