- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies whether augmented recurrent updates with attention-based retrieval can solve the problem of jointly predicting the next event's time and type for irregularly timed events with categorical types. It proposes Moreva by adding causal-attention branches in parallel with selected layers of a pretrained Mamba-3 backbone and predicts the next event's time and type through separate output heads. Experiments are conducted on both synthetic and real-world benchmarks. The paper also provides a theoretical analysis of the potential benefit of retrieving historical information beyond a recurrent representation.
Critical strengths
- The idea is interesting and reasonable to introduce the attention mechanism, directly retrieving past events to reduce the computational cost in recurrent models.
- The ablation variants, Mamba-only and Transformer-only, using a shared event representation and similar training procedures, make the evaluation results convincing.
- Experiment settings are diverse and valuable.
Critical weaknesses
- The synthetic experiments do not establish that real-world event streams commonly contain the same kind of dependency or that attention improves forecasting under realistic long-range patterns.
- The multi-event comparison combines architecture differences with additional horizon-specific training for Moreva. A more informative comparison should train Mamba-only and Transformer-only models with the same multi-scale horizon supervision used for Moreva.
- The results in Table2 show that the improvements compared with Mamba-only and Transformer-only are quite subtle, and the analysis of when recurrence can maintain a sufficient state and when exact retrieval requires memory that grows with dependency length should be better clarified according to real-world benchmarks.
Questions for authors
- Given that Moreva and Mamba-only perform similarly on EasyTPP, what evidence shows that the added attention branches are responsible for the improvement?
- How do Mamba-only and Transformer-only perform when given the same multi-scale horizon fine-tuning as Moreva?
4 comments
W2 / Q2: Identical horizon fine-tuning for all three backbones
We now bring all three backbones together under identical five-source horizon training in revised Table 2. This extends the original Transformer-only horizon results in Appendix D.1 (Figure 5 in the original submission) and the three-source Mamba-only control in Appendix F to a common cohort and recipe, directly answering your request for a unified comparison.
All three backbones receive 128 updates with the same windows, learning-rate groups, seeds, and selection procedure. At H/D = 10%, five-source, three-seed macros are:
| Model after horizon FT | Type MAE ↓ | Count MAE ↓ | W1 ↓ | F1 ↑ |
|---|---|---|---|---|
| Moreva | 3.537 | 1.862 | 0.201 | 0.297 |
| Mamba-only | 3.520 | 1.858 | 0.200 | 0.297 |
| Transformer-only | 3.571 | 1.901 | 0.202 | 0.289 |
These follow the paper's retrospective development-set horizon protocol, conditional on completed rollouts. Appendix J.7 reports paired sequence-bootstrap uncertainty and results at all evaluated durations.
Horizon adaptation improves Type MAE by 0.92 / 0.97 / 0.71 relative to the respective Moreva / Mamba-only / Transformer-only parent checkpoints. Revised Table 2 makes the benefit of this shared training objective explicit, with Moreva and Mamba-only giving comparable forecasting scores. We interpret Type MAE jointly with Count MAE, W1, and F1, and retain the empty-prediction reference (Type MAE 3.298) to distinguish type-count error from overall forecasting quality.
Manuscript changes (orange): Sec. 4.4, revised Table 2; Appendix J.7, Table 23 (all durations, shared training, and uncertainty).
W3: Explicit conditions for recurrent sufficiency and growing memory
We have made the process conditions and their memory requirements explicit, extending Proposition 2 to multiscale excitation and adding power-law approximation in Appendix B.4.1. The fixed-checkpoint interventions then connect these representational cases to the mechanisms used by trained models. The resulting distinction is:
- Stacked exponential excitation: K types and M time scales admit KM real-valued excitation statistics. If represented exactly, these leave zero optimal-risk gain for extra history access.
- Power-law excitation: for at most N historical events, fixed process parameters, a positive baseline, and nonnegative excitation, exponential-sum approximation gives state dimension O(K log(1/ε) log(N/ε)) and residual optimal-risk gain O(ε).
- Exact lag-L retrieval: a deterministic recurrent predictor correct on every sequence over K types needs at least L log₂ K bits of persistent input-dependent state (Proposition 2).
Appendix B.4.2 adds a history-derived query that selects a past mark and contributes to the next-event intensity, with its predictive value described by the same next-event density and Theorem 1. It gives an explicit two-type example where retrieving a mark unresolved by the specified recurrent state improves type prediction while leaving timing unchanged; type-dependent retrieval strengths can also change the next-gap law. This is a theoretical construction conditional on the information retained in that state. The empirical tests comprise the two Hawkes processes for recurrent excitation, and LDP and sensor-network streams for fixed-lag retrieval.
Theorem 1 characterizes optimal squared-error and log-loss risks through the conditional mean gap and conditional type information. The exact-retrieval bound concerns worst-case memory, and the Hawkes constructions specify sufficient statistics. We assess learning, top-1 accuracy, and the pathways used by trained Mamba models empirically through the Hawkes and LDP interventions. This combination provides both explicit process-level conditions and direct evidence of recurrent history use and distant attention access in the corresponding trained predictors.
Manuscript changes (orange): Sec. 3.4; Appendix B.2, Proposition 2 (single- and multiscale sufficiency); Appendix B.4.1, Proposition 4 (power-law approximation); Appendix B.4.2 (conditional retrieval construction).
Thank you again for the constructive suggestions on fair horizon comparisons and the connection between theory and practice. They helped us separate recurrent use of history from attention-dependent retrieval and clarify the scope of each empirical result.
Thank you for the responses. I have read the rebuttal and have no further questions.
Thank you for your valuable feedback and suggestions. If your concerns have been addressed, we would greatly appreciate your consideration of a higher score.
Thank you for your thoughtful review and for recognizing the value of the shared-interface comparisons and the range of experimental settings. We appreciate your emphasis on comparable horizon training and on connecting the theoretical distinction between recurrence and retrieval to empirical evidence. We address these points below, distinguishing the controls in the original appendix from the additional experiments.
W1 / Q1: Complex processes distinguish history use from attention use
We have expanded the evidence around the two mechanisms identified by the theory: recurrent summarization of historical excitation and attention-based access to specific past events. Two-time-scale and power-law Hawkes processes test the first mechanism; LDP access interventions and fixed-order streams from real sensor measurements test the second.
The new Hawkes experiments show that trained recurrent models effectively use distant history for timing, supporting the recurrent-sufficiency side of the analysis. The history-truncation and attention-knockout results are consolidated in Appendix Table 4. Appendix B.4.3 gives the protocol and full-test results, alongside the theory and context-query construction in B.4.1–B.4.2. Keeping checkpoints fixed, we truncated inputs to 64 events on the same 20,000 test targets per process:
Each subsample is drawn uniformly from targets with more than 16 preceding events and shared across models and seeds; histories of at most 64 events remain unchanged. Entries average three seeds without retraining. Full history improves timing prediction for both processes, especially two-time-scale excitation. This comparison measures the contribution of history within the same trained predictors.
The known generating law provides the following numerical reference on the complete test sets:
Accuracy is in percent and RMSE in generated gap units. Both processes have eight types and irregular stochastic event times: decay rates 1 and 0.002 for the two-scale kernel, and (τ + 0.2)^−1.3 for the power-law kernel. Each uses 600/100/100 independent train/development/test sequences of 4,096 events and three training seeds under one fixed process parameterization. Full-test scores cover 409,500 targets per seed, separately from the truncation subsamples. The oracle is an approximate numerical reference computed by integration, including an exponential-sum approximation for the power-law kernel.
The recurrent path preserves this near-reference performance with attention disabled: mean accuracy stays within 0.002 points and RMSE within 0.00001 of full Moreva on both complete test sets. Together with the history-truncation results, this supports recurrent processing as the source of the measured history benefit.
On LDP, varying only Moreva's attention access while retaining the full recurrent prefix gives:
Restoring access to the relevant lag recovers much of Moreva's accuracy, directly supporting the trained model's use of distant retrieval. The same mechanism appears in fixed-order streams serialized from real PeMS Traffic, Electricity, and PEMS-BAY measurements: with 256 sensors, each sensor's previous reading lies exactly 256 events back. Moreva exceeds Mamba-only by 7.4 / 9.9 / 10.7 accuracy points, and fixed-checkpoint attention probes identify the contribution of retrieval (response to ZvvL, W1 / Q1). These experiments establish attention-dependent type prediction on real measurements under the specified serialization, artificial gaps, and development-selected training settings.
Manuscript changes (orange): Secs. 4.2 and 4.5; Appendix B.4.3, Tables 3–4 (numerical reference and history truncation); Appendix J.4.1, Table 17 (user-interaction streams); Appendix J.4.2, Tables 18–19 (sensor streams).