acceptodds
Under review as a conference paper at ICLR 2027

Moreva: When Does Attention Help Mamba Forecast Event Streams?

Abstract

Event streams often comprise irregularly timed events with categorical types, motivating joint prediction of the next event's time and type. State-space models such as Mamba efficiently summarize event histories in a fixed-size recurrent state, but may lose information about specific distant events. This limitation motivates hybrids that augment recurrent updates with attention-based retrieval. Although such hybrids have shown promise across sequence-modeling tasks such as Large Language Model and Time Series Forecasting, when attention improves temporal event forecasting and whether standard benchmarks contain dependencies that require retrieval beyond a recurrent summary remains unclear. We investigate these questions with Moreva, a shared event-history model that adds parallel causal-attention branches to selected layers of a pretrained Mamba backbone. On a synthetic process with known dependency length, attention lets Moreva recover type information placed hundreds to thousands of events back, which the Mamba-only model and the baselines miss. On EasyTPP benchmark datasets, Moreva achieves the state-of-the-art compared to time-point-process, regular time series and language-model-based baselines.

Primary area:
learning on time series and dynamical systems
Keywords:
time point process, event stream, mamba

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LcfuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether augmented recurrent updates with attention-based retrieval can solve the problem of jointly predicting the next event's time and type for irregularly timed events with categorical types. It proposes Moreva by adding causal-attention branches in parallel with selected layers of a pretrained Mamba-3 backbone and predicts the next event's time and type through separate output heads. Experiments are conducted on both synthetic and real-world benchmarks. The paper also provides a theoretical analysis of the potential benefit of retrieving historical information beyond a recurrent representation.

Critical strengths
  1. The idea is interesting and reasonable to introduce the attention mechanism, directly retrieving past events to reduce the computational cost in recurrent models.
  2. The ablation variants, Mamba-only and Transformer-only, using a shared event representation and similar training procedures, make the evaluation results convincing.
  3. Experiment settings are diverse and valuable.
Critical weaknesses
  1. The synthetic experiments do not establish that real-world event streams commonly contain the same kind of dependency or that attention improves forecasting under realistic long-range patterns.
  2. The multi-event comparison combines architecture differences with additional horizon-specific training for Moreva. A more informative comparison should train Mamba-only and Transformer-only models with the same multi-scale horizon supervision used for Moreva.
  3. The results in Table2 show that the improvements compared with Mamba-only and Transformer-only are quite subtle, and the analysis of when recurrence can maintain a sufficient state and when exact retrieval requires memory that grows with dependency length should be better clarified according to real-world benchmarks.
Questions for authors
  1. Given that Moreva and Mamba-only perform similarly on EasyTPP, what evidence shows that the added attention branches are responsible for the improvement?
  2. How do Mamba-only and Transformer-only perform when given the same multi-scale horizon fine-tuning as Moreva?
4 comments

Thank you for your thoughtful review and for recognizing the value of the shared-interface comparisons and the range of experimental settings. We appreciate your emphasis on comparable horizon training and on connecting the theoretical distinction between recurrence and retrieval to empirical evidence. We address these points below, distinguishing the controls in the original appendix from the additional experiments.

W1 / Q1: Complex processes distinguish history use from attention use

We have expanded the evidence around the two mechanisms identified by the theory: recurrent summarization of historical excitation and attention-based access to specific past events. Two-time-scale and power-law Hawkes processes test the first mechanism; LDP access interventions and fixed-order streams from real sensor measurements test the second.

The new Hawkes experiments show that trained recurrent models effectively use distant history for timing, supporting the recurrent-sufficiency side of the analysis. The history-truncation and attention-knockout results are consolidated in Appendix Table 4. Appendix B.4.3 gives the protocol and full-test results, alongside the theory and context-query construction in B.4.1–B.4.2. Keeping checkpoints fixed, we truncated inputs to 64 events on the same 20,000 test targets per process:

ProcessMamba-only RMSE: full → last 64 ↓Moreva RMSE: full → last 64 ↓
Two-time-scale Hawkes1.5409 → 2.17161.5407 → 2.1656
Power-law Hawkes1.7215 → 1.74081.7213 → 1.7408

Each subsample is drawn uniformly from targets with more than 16 preceding events and shared across models and seeds; histories of at most 64 events remain unchanged. Entries average three seeds without retraining. Full history improves timing prediction for both processes, especially two-time-scale excitation. This comparison measures the contribution of history within the same trained predictors.

The known generating law provides the following numerical reference on the complete test sets:

ProcessMamba-only accuracy ↑ / RMSE ↓Moreva accuracy ↑ / RMSE ↓Numerical oracle accuracy ↑ / RMSE ↓
Two-time-scale Hawkes25.48 / 1.575225.50 / 1.575325.57 / 1.5739
Power-law Hawkes36.23 / 1.673036.23 / 1.672936.32 / 1.6714

Accuracy is in percent and RMSE in generated gap units. Both processes have eight types and irregular stochastic event times: decay rates 1 and 0.002 for the two-scale kernel, and (τ + 0.2)^−1.3 for the power-law kernel. Each uses 600/100/100 independent train/development/test sequences of 4,096 events and three training seeds under one fixed process parameterization. Full-test scores cover 409,500 targets per seed, separately from the truncation subsamples. The oracle is an approximate numerical reference computed by integration, including an exponential-sum approximation for the power-law kernel.

The recurrent path preserves this near-reference performance with attention disabled: mean accuracy stays within 0.002 points and RMSE within 0.00001 of full Moreva on both complete test sets. Together with the history-truncation results, this supports recurrent processing as the source of the measured history benefit.

On LDP, varying only Moreva's attention access while retaining the full recurrent prefix gives:

Type accuracy (%) ↑Lag 256Lag 1,024
Separately trained Mamba-only32.4237.76
Moreva, full attention54.6254.72
Moreva, half-lag window22.6628.42
Moreva, lag-sized window52.5451.28

Restoring access to the relevant lag recovers much of Moreva's accuracy, directly supporting the trained model's use of distant retrieval. The same mechanism appears in fixed-order streams serialized from real PeMS Traffic, Electricity, and PEMS-BAY measurements: with 256 sensors, each sensor's previous reading lies exactly 256 events back. Moreva exceeds Mamba-only by 7.4 / 9.9 / 10.7 accuracy points, and fixed-checkpoint attention probes identify the contribution of retrieval (response to ZvvL, W1 / Q1). These experiments establish attention-dependent type prediction on real measurements under the specified serialization, artificial gaps, and development-selected training settings.

Manuscript changes (orange): Secs. 4.2 and 4.5; Appendix B.4.3, Tables 3–4 (numerical reference and history truncation); Appendix J.4.1, Table 17 (user-interaction streams); Appendix J.4.2, Tables 18–19 (sensor streams).

W2 / Q2: Identical horizon fine-tuning for all three backbones

We now bring all three backbones together under identical five-source horizon training in revised Table 2. This extends the original Transformer-only horizon results in Appendix D.1 (Figure 5 in the original submission) and the three-source Mamba-only control in Appendix F to a common cohort and recipe, directly answering your request for a unified comparison.

All three backbones receive 128 updates with the same windows, learning-rate groups, seeds, and selection procedure. At H/D = 10%, five-source, three-seed macros are:

Model after horizon FTType MAE ↓Count MAE ↓W1 ↓F1 ↑
Moreva3.5371.8620.2010.297
Mamba-only3.5201.8580.2000.297
Transformer-only3.5711.9010.2020.289

These follow the paper's retrospective development-set horizon protocol, conditional on completed rollouts. Appendix J.7 reports paired sequence-bootstrap uncertainty and results at all evaluated durations.

Horizon adaptation improves Type MAE by 0.92 / 0.97 / 0.71 relative to the respective Moreva / Mamba-only / Transformer-only parent checkpoints. Revised Table 2 makes the benefit of this shared training objective explicit, with Moreva and Mamba-only giving comparable forecasting scores. We interpret Type MAE jointly with Count MAE, W1, and F1, and retain the empty-prediction reference (Type MAE 3.298) to distinguish type-count error from overall forecasting quality.

Manuscript changes (orange): Sec. 4.4, revised Table 2; Appendix J.7, Table 23 (all durations, shared training, and uncertainty).

W3: Explicit conditions for recurrent sufficiency and growing memory

We have made the process conditions and their memory requirements explicit, extending Proposition 2 to multiscale excitation and adding power-law approximation in Appendix B.4.1. The fixed-checkpoint interventions then connect these representational cases to the mechanisms used by trained models. The resulting distinction is:

  • Stacked exponential excitation: K types and M time scales admit KM real-valued excitation statistics. If represented exactly, these leave zero optimal-risk gain for extra history access.
  • Power-law excitation: for at most N historical events, fixed process parameters, a positive baseline, and nonnegative excitation, exponential-sum approximation gives state dimension O(K log(1/ε) log(N/ε)) and residual optimal-risk gain O(ε).
  • Exact lag-L retrieval: a deterministic recurrent predictor correct on every sequence over K types needs at least L log₂ K bits of persistent input-dependent state (Proposition 2).

Appendix B.4.2 adds a history-derived query that selects a past mark and contributes to the next-event intensity, with its predictive value described by the same next-event density and Theorem 1. It gives an explicit two-type example where retrieving a mark unresolved by the specified recurrent state improves type prediction while leaving timing unchanged; type-dependent retrieval strengths can also change the next-gap law. This is a theoretical construction conditional on the information retained in that state. The empirical tests comprise the two Hawkes processes for recurrent excitation, and LDP and sensor-network streams for fixed-lag retrieval.

Theorem 1 characterizes optimal squared-error and log-loss risks through the conditional mean gap and conditional type information. The exact-retrieval bound concerns worst-case memory, and the Hawkes constructions specify sufficient statistics. We assess learning, top-1 accuracy, and the pathways used by trained Mamba models empirically through the Hawkes and LDP interventions. This combination provides both explicit process-level conditions and direct evidence of recurrent history use and distant attention access in the corresponding trained predictors.

Manuscript changes (orange): Sec. 3.4; Appendix B.2, Proposition 2 (single- and multiscale sufficiency); Appendix B.4.1, Proposition 4 (power-law approximation); Appendix B.4.2 (conditional retrieval construction).

Thank you again for the constructive suggestions on fair horizon comparisons and the connection between theory and practice. They helped us separate recurrent use of history from attention-dependent retrieval and clarify the scope of each empirical result.

Thank you for the responses. I have read the rebuttal and have no further questions.

Thank you for your valuable feedback and suggestions. If your concerns have been addressed, we would greatly appreciate your consideration of a higher score.

Reviewer_agGnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PC verification needed regarding desk rejection

There is no one-line spacing between any Table and its caption in this paper, and the paper thus does not comply with the ICLR format; verification by the PC appears necessary. Separately from this issue, however, I provide my comments and evaluation of the research content below.


This study proposes Moreva, a hybrid model that uses pretrained Mamba-3 for event prediction in temporal point processes (TPPs) and adds attention and FFNs to some layers. It converts the time intervals and event types of different datasets into a common input format and fine-tunes the entire model to predict the occurrence time and type of the next event.

To my knowledge, there are few cases of directly using pretrained Mamba for time series tasks, so I find this study very interesting. Furthermore, I find it interesting that the study examines how much Mamba and attention should be combined to obtain an appropriate trade-off between performance and efficiency. However, I find it somewhat disappointing that these experiments do not lead beyond demonstrating the effectiveness of general Mamba–attention hybrids to motivating the proposed Moreva architecture. I think this impression is reinforced by the lack of comparisons with the most closely related SSM-based TPP studies.

Overall, I consider this a study that includes meaningful experiments and analyses. However, I believe that the model's distinctive contribution and reproducibility need clearer support, and therefore recommend borderline reject.

Critical strengths

[S1] A design for using pretrained Mamba for TPP

  • This study retains the architecture of pretrained Mamba while designing input representations and prediction heads so that datasets with irregular time intervals and different event types can be trained on jointly.
  • It aims to handle multiple datasets with a single backbone through preprocessing such as time-interval normalization. I find it impressive that attention and FFNs are added to only some of the existing Mamba layers to incorporate the role of attention while adding as little computation as possible.
  • Furthermore, I think it is meaningful that the study goes beyond next-event prediction and extends to multi-event forecasting through additional training on future windows of various lengths. In particular, I find it interesting that the training includes not only events within the window but also the time interval of the first event beyond that window, thereby providing a supervision signal even when there are no events within the window.

[S2] Separate analyses of real and synthetic data, and comparisons of performance and efficiency

  • I appreciate that the study does not hide the finding from the real-benchmark ablation that Moreva, Mamba-only, and Transformer-only perform almost similarly, and separately analyzes the conditions under which differences between architectures emerge. The results in Figure 3 also show what advantages attention offers.
  • Figure 2 compares this predictive performance together with resource costs. It demonstrates efficiency by showing that Moreva maintains a smaller streaming state while achieving long-range prediction performance close to that of Transformer-only.
Critical weaknesses

[W1] Insufficient comparisons with closely related baselines and insufficient experimental reproducibility

  • MHP/MHP-E, S2P2, and FlexTPP are mentioned in Related Work but are not included among the actual baselines. In particular, I consider studies that model events using SSMs, such as MHP/MHP-E and S2P2, to be the most closely related comparisons for this study. Since MHP-E combines Mamba and Transformer, I particularly request that it be added as a comparison to establish what advantages Moreva's hybrid design offers over existing approaches.
  • The current Mamba-only and Transformer-only comparisons are useful for understanding the characteristics of each architecture, but they do not show the differences between existing hybrids and Moreva. It is necessary to distinguish the effect of simply adding some attention from the effect of Moreva's chosen placement and parallel fusion scheme. In its current form, the issue stated in the abstract, "when attention improves temporal event forecasting and whether standard benchmarks contain dependencies that require retrieval beyond a recurrent summary remains unclear", does not seem to be sufficiently resolved.
  • I also have some concerns regarding reproducibility. I understand that the released code is a kind of reference implementation and therefore may not be identical to the paper. Even so, I believe that experimental settings and artifacts corresponding to the paper need to be provided. I think configurations, checkpoints, and result files that allow the results in the current manuscript to be checked are needed.

[W2] Insufficient evidence for the contribution of the specific model design and the interpretation of the experimental results

  • I think the motivation of this study is valid. However, examining whether attention can use past information that a recurrent state alone cannot sufficiently exploit is a question that also applies to general recurrence–attention hybrids. The justification for why this question should lead to Moreva's specific design still seems insufficient.
  • Furthermore, when comparing Tables 1 and 2, I consider the differences in next-event prediction performance among the three models (Moreva, Mamba-only, and Transformer-only) to be relatively small, and their rankings also vary across metrics in multi-event forecasting. The authors interpret this as an effect of the common event interface and training procedure, but I think a comparison that directly examines the contribution of the proposed input representation is needed. Ablations of the architecture and horizon training exist, but this part appears to be missing and needs to be supplemented. Also, to interpret the use of pretrained Mamba as an advantage, a performance comparison against the same architecture with random initialization would be helpful.
  • Furthermore, the finding that attention offers little additional benefit on real benchmarks alone is insufficient to conclude that these benchmarks rarely require long-range retrieval. Dependencies that are needed only in some datasets or samples may not have appeared in the averaged results. Experiments that limit the length of the historical context or mask the distant past, for example, could help establish what information is actually used for prediction and make the study more complete.
Questions for authors

The questions below are based on the weaknesses above.

[Q1] Could you add comparisons with the most closely related SSM and hybrid TPP models?

  • I would like to know why MHP/MHP-E, S2P2, FlexTPP, and other methods mentioned in Related Work were not included in the actual comparisons. In particular, since both MHP-E and Moreva combine Mamba and attention, I think a direct comparison is necessary.
  • If possible, it would be helpful to compare against existing methods under similar training conditions and additionally compare sequential fusion with Moreva's parallel fusion while matching the number of attention layers and the resource budget. Such results would help distinguish the effect of the hybrid itself from that of the proposed fusion scheme.

[Q2] Could you provide code and artifacts corresponding to the experimental conditions in the current manuscript?

  • It would be helpful to provide configurations, actual checkpoints, and evaluation result files that match the results used in the paper. In particular, since evaluation metrics for TPPs sometimes differ across papers, it would be even better to also provide commonly used metrics.
  • Furthermore, it would be helpful to also provide some missing parts concerning SurF and multi-event forecasting that are missing from the released package.

[Q3] Could you provide ablation results for the proposed event input representation and the use of pretraining?

  • As mentioned above, I suspect that the input representation may have played a fairly important role. Whether my view is right or wrong can be established through experiments, so I find the absence of a related ablation somewhat disappointing. I would like to see the results of ablating the proposed six temporal input features.
  • Applying this to Mamba-only and Moreva separately would probably allow the effects of the input representation and the hybrid architecture to be distinguished more clearly.
  • Also, do you have results comparing initialization with pretrained weights against random initialization for the same architecture? Since applying pretrained Mamba to TPPs is an interesting aspect of this study, I would like to know how much of an advantage it actually provides. Of course, full training requires substantial time and memory, so it would be helpful to examine this within the available resources. It is also fine if these results are not available. What I am curious about is how much you believe the pretraining contributes, from your empirical perspective.

[Q4] Could you more directly support the interpretation that real benchmarks rarely require long-range retrieval?

  • Examining how performance changes when the historical context is progressively shortened or information from the distant past is masked would allow a more direct assessment of how much long-range information is needed on real benchmarks.
  • In particular, I am curious whether this tendency varies by dataset or sequence length. It would also be helpful if you could explain which conditions in real data are connected to the effect of long-range dependencies observed in the synthetic experiments.
8 comments

Thank you for your detailed and constructive review. We appreciate your positive assessment of the shared event interface, horizon supervision, and joint analysis of predictive quality and resource costs. Your requests for nearby baselines, component controls, and reproducible artifacts helped us strengthen the evidence for each part of the contribution.

W1 / Q1: Added nearby baselines and fusion/placement comparisons

We have expanded the comparisons to cover the nearby SSM and hybrid models you identified, together with direct fusion and placement controls. These additions build on the original Appendix F component ablations and Appendix G attention-layer sweep.

We have therefore added all four requested baselines, including the closely related hybrid MHP-E. Results use the same five EasyTPP datasets and splits, three seeds, and development-selected checkpoints; macros average seeds within each dataset, then weight datasets equally.

ModelTraining regimeMacro time RMSE ↓Macro type accuracy (%) ↑
S2P2One-source specialist3.64055.98
FlexTPPOne-source specialist3.64956.13
MHPOne-source specialist3.65258.42
MHP-EOne-source specialist3.61258.55
Mamba-onlyJoint → Fine-tune3.51458.80
Transformer-onlyJoint → Fine-tune3.51758.75
MorevaJoint → Fine-tune3.51358.84

Moreva has the strongest displayed macro point estimates for both metrics. This table compares complete training configurations: the rows differ in training exposure, initialization, parameter budgets, and optimization. MHP/MHP-E use official code, S2P2 the pinned EasyTPP port, and FlexTPP the paper-linked unofficial implementation with its native flow-median prediction. On LDP, Moreva reaches 54.62% / 54.72% at lags 256 / 1,024, compared with S2P2's 30.5% / 37.3%. We examine the roles of fusion and placement separately below.

We also compared fusion and placement variants, each with two attention modules and the same training procedure.

VariantEasyTPP FT accuracy (%) ↑LDP, lag 256 (%) ↑LDP, lag 1,024 (%) ↑
Parallel, layers 4 and 8 (Moreva)58.8454.6254.72
Sequential, layers 4 and 858.8554.5954.72
Parallel, layers 1 and 258.8154.6054.70
Parallel, layers 11 and 1258.8354.5754.72
Attention replaces mixers at layers 4 and 858.8632.5337.76
Mamba-only58.8032.4237.76

All four additive variants retain the long-lag retrieval gain, showing consistent performance across parallel and sequential fusion and the tested layer placements. Moreva retains every pretrained Mamba block; zeroing the added branches recovers the Mamba-only computation with the same retained weights.

The table also makes the component budgets explicit. Additive variants retain 12 Mamba mixers, use 12 heads per attention module, and add FFNs; replacement retains 10 mixers, uses 19 heads, and retains native FFNs without those additions. Five-source EasyTPP parameter counts are 103,457,929 versus 88,536,613. We therefore interpret the additive-versus-replacement rows as comparisons of these complete configurations.

The fixed-checkpoint access intervention in W2 tests the trained model's reliance on retrieval separately from these architectural differences.

Manuscript changes (orange): Sec. 4.3; Appendices J.1–J.2, Tables 11–12 (baselines, fusion/placement, and capacity differences).

W2: What Moreva contributes and why the design follows from the analysis

Moreva connects an analysis of predictive information to a concrete design that preserves recurrent processing and adds selective access to past events. The revised discussion explains this contribution through four connected elements:

  • Scope. Theorem 1 (Sec. 3.4; proof in Appendix B.1) characterizes the predictive information retrieval adds beyond a recurrent summary. We test this analysis through fixed-checkpoint interventions and quantify its state cost in Moreva. Fusion and placement are empirical design choices, with Appendix J.2 demonstrating retrieval gains across several additive configurations.
  • Theory. Proposition 2 constructs a sufficient recurrent state for single- and multiscale exponential Hawkes excitation, and Appendix B.4.1 extends it to power-law approximation. Theorem 1 decomposes the optimal retrieval gain into timing and type-information terms; B.4.2 gives a context-selected historical mark that supplies this information. These statements characterize representational capacity and optimal risks for the specified information sets; the interventions evaluate what the trained models learn.
  • Interventions. On two-time-scale Hawkes, full-history Moreva achieves RMSE 1.5407 versus 2.1656 with 64 events, while attention knockout changes full-test RMSE by less than 0.00001, supporting history use through recurrence. On LDP at lag 256, full attention achieves 54.62% accuracy versus 22.66% with a half-lag attention window, with the full recurrent prefix kept in both cases. Together, these probes identify how each pathway contributes to prediction.
  • Design. Moreva keeps all 12 Mamba blocks and adds attention and FFN branches at two layers, so zeroing the branches recovers the Mamba-only computation with the same weights. At lag 1,024 it nearly matches Transformer-only (54.72% vs. 54.80%) with about one-ninth of its counted streaming state at a separately measured 32,768-event context (201.1 vs. 1,824.0 MiB). This state comparison counts recurrent allocations and KV caches; throughput and the measurement protocol are reported in the response to ZvvL, W2, and Appendix D.1.

Manuscript changes (orange): Secs. 3.4 and 4.2; Appendices B.2 and B.4 (process structure and retrieval), B.4.3 (Hawkes experiments), and J.2 (design interpretation). Figure 2 reports the resource trade-off.

W2 / Q3: Six-feature and initialization ablations

Following your suggestion, we now quantify the contributions of the input representation and initialization separately for Mamba-only and Moreva. These controls complement the six-feature definitions and normalization in Sec. 3.2 and Appendix A.

For both architectures, we trained 12 masks in Joint and Fine-tune regimes with three seeds: six individual removals, five grouped masks, and removal of all time features. Individual/grouped removals retain FT accuracy of 58.79–58.86% for Mamba-only and 58.80–58.86% for Moreva, with RMSE 3.512–3.527. Without temporal input, accuracy is 58.73% / 58.75% and RMSE is approximately 3.876 for both, about 10% above full-input RMSE. Temporal input thus supports timing prediction, while the individual and grouped masks show redundancy among these cues.

To test whether the representation also helps the nearby baselines you identified, we added the same six causal time features to S2P2, MHP, and MHP-E through a time MLP added to each model's event embedding. Each model retains its native time encoding, decoder, training recipe, and development-only checkpoint selection. All three remain one-source specialists. Values below are five-source macro means across the same three seeds; each seed's macro weights datasets equally. Within each row, bold compares native and feature-augmented inputs for each metric.

ModelNative RMSE ↓+6 features RMSE ↓Native accuracy (%) ↑+6 features accuracy (%) ↑
S2P23.6403.68755.9855.13
MHP3.6523.60058.4258.44
MHP-E3.6123.55458.5558.62

The temporal features improve RMSE for both MHP and MHP-E and also improve their type accuracy; S2P2's native input gives its stronger result under the unchanged recipe. These controls show where the representation transfers successfully and quantify its contribution for each model. As in the baseline table, comparisons with Moreva concern the stated training regimes, initializations, and parameter budgets.

Under the same training budget, Moreva's pretrained/random initialization gives RMSE 3.513 / 3.518 and accuracy 58.84% / 58.67%; Mamba-only gives 3.514 / 3.515 and 58.80% / 58.69%. Pretrained initialization improves both displayed metrics for both architectures within this budget. Together with the feature masks, these results quantify the benefits of temporal input and initialization under the evaluated recipe.

Manuscript changes (orange): Appendix J.3, Tables 13–16 (all feature masks in both regimes, feature-augmented baselines, and initialization controls).

Reviewer_ZvvLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Moreva, a hybrid Mamba-Attention model for joint event-time and event-type prediction in irregular event streams. The model combines Mamba’s ability to summarize long histories efficiently with attention layers that can directly access specific events from the distant past. One strength of the paper is that it clearly characterizes the setting where this hybrid design is useful, and the synthetic experiments support this point. In particular, Moreva shows clear gains over standard Mamba when prediction depends on events hundreds of steps back. The empirical results are solid overall. Moreva performs well on the EasyTPP datasets for both time and type prediction, while also retaining some of Mamba’s practical benefits, such as fixed-size state and lower streaming memory use than a full Transformer.

Critical strengths
  1. Well-designed synthetic experiments. The synthetic setup does a good job of isolating long-range retrieval. The lagged-dependency process is constructed in a clean and controlled way, and the use of both a short-history baseline and a Bayes-optimal reference makes the results easy to interpret. The findings clearly separate the models: standard Mamba works well when the dependency is short, but degrades at longer lags, while Moreva stays much closer to the upper bound.

  2. Theory and experiments are well aligned. The theoretical analysis gives a clear account of when adding attention should help beyond a recurrent model. In short, attention matters when the needed predictive signal is not preserved in the recurrent state and must be retrieved from a specific distant event. The synthetic experiments match this story well: Moreva and Mamba-only behave similarly at short lags, while Moreva shows clear gains when the relevant information is far back in the sequence.

Critical weaknesses
  1. The experiments do not clearly demonstrate the effectiveness of the Attention mechanism. On the five EasyTPP datasets, Moreva, Mamba-only, and Transformer-only achieve very similar results. The improvement over the strongest baseline is also relatively limited, making it unclear how much the Attention branch contributes on real-world datasets.
  2. Attention introduces additional computational and memory costs. Moreva’s training throughput is approximately 12%–14% lower than that of Mamba-only, while streaming inference requires a KV cache that grows with the context length.
Questions for authors

Could the authors also evaluate Moreva on real-world datasets with longer histories, such as KuaiRand, the raw Taobao UserBehavior dataset, or Last.fm-1K? Right now, the long-range retrieval advantage is shown mainly in the synthetic setup, while the EasyTPP datasets seem relatively short. It would be useful to see whether the same benefits carry over to more natural long-sequence settings where Mamba and attention can be compared under stronger long-range dependencies.

3 comments

Thank you for your careful review and for recognizing the controlled retrieval experiments and their connection to the theoretical analysis. We appreciate your emphasis on evidence from real event streams and on the computational and memory costs of attention. Your suggestions helped us sharpen the empirical scope and the quality–memory comparison below.

W1 / Q1: Results on the requested long-history sources and on real streams with a fixed distant dependency

We agree that the small EasyTPP differences alone do not establish a practical benefit from attention on real event logs. The original LDP experiment in Sec. 4.2 and Appendix I isolates a known dependency length; extending that evidence to natural histories is a separate empirical question. We therefore followed your suggestion to evaluate the three source families below.

We evaluated KuaiRand-1K, Taobao UserBehavior, and Last.fm-1K with up to 4,096 events per user. KuaiRand predicts first video tags on clicks; Taobao uses the available processed 4% user subset, first seven days, pv events, and 16 training categories plus other; Last.fm uses JODIE's 1,000-song release. The Taobao run does not cover the full raw CSV setting. The table gives mean test type accuracy (%) over three seeds; higher is better. Settings are documented in Appendix J.4.1.

DatasetTypesMedian historyMamba-onlyMorevaTransformer-onlyMoreva, attention off
KuaiRand-1K553,90922.3822.2021.8822.20
Taobao UserBehavior1725369.2869.3069.0669.28
Last.fm-1K1,00071421.8122.4222.7122.42

These runs do not establish a consistent attention-attributable advantage: knockout changes the means by at most 0.02 points. Additional hyperparameter results are retained in Appendix J.4.1.

On EasyTPP, the five-source, three-seed Fine-tune checkpoints have full-prefix accuracy of 58.84% / 58.80% for Moreva / Mamba-only. Keeping only 32 events changes accuracy on affected targets by −0.08 / +0.00 points; keeping one changes it by −3.82 / −3.90. Disabling Moreva's attention costs 0.03 points overall. The tested models rely mainly on recent context and recurrent processing; these probes cannot rule out distant information that another model or training procedure could use.

We also built fixed-order sensor streams from three public recordings (PeMS Traffic, Electricity, PEMS-BAY): 256 sensors report in a fixed order, so each sensor's previous reading lies exactly 256 events back. Test type accuracy with development-selected hyperparameters (three seeds):

DatasetMamba-onlyMorevaTransformer-onlyMoreva, attention off
Traffic26.2833.7036.9910.40
Electricity18.3028.2328.348.17
PEMS-BAY18.4829.1829.295.92

Appendix J.4.2 documents the hyperparameter candidates and development-based selection.

With parameters and the recurrent prefix fixed, restricting the displayed Moreva checkpoints to a 64-event attention window removes most of the gain; 256 entries recover much of it, and 1,024 recover more. These interventions isolate the trained model's reliance on attention access. The streams use artificial exponential gaps in place of the original timestamps, so the result concerns type prediction under this serialization. Appendix J.4.2 gives construction details, controls, and the reporting cohort.

Manuscript changes (orange): Sec. 4.5; Appendix J.4.2, Tables 18–19 (sensor streams, hyperparameter selection, controls, and probes); Appendix J.4.1, Table 17 (requested event logs); Appendix J.5, Tables 20–21 (history and per-source probes).

W2: The retrieval benefit and its memory cost

You are right that attention adds both computation and memory. To clarify the scope of the memory claim, only the recurrent component has a state independent of history length; Moreva's full-context attention cache grows with the retained history, as reported in the original Figure 2 and Appendix D.1. We agree that the small EasyTPP gap alone does not justify this additional state. On LDP, Moreva approaches Transformer-only's retrieval accuracy with a much smaller cache:

ModelLDP accuracy, lag 1,024 (%) ↑Training at 32,768 events (k events/s) ↑State at 32,768 events (MiB) ↓
Mamba-only37.76789.1
Moreva54.7268201.1
Transformer-only54.80731,824.0

Across the measured sequence lengths, Moreva's training throughput is 12–14% below Mamba-only's (Appendix D.1). Accuracy and resources are separate evaluations. Throughput is a synchronized fixed-shape measurement on one RTX PRO 6000 Blackwell GPU; state counts recurrent allocations and bfloat16 KV-cache shapes, not end-to-end serving memory or latency.

To reduce cache size without assuming the dependency length, we selected the smallest window within 0.1 accuracy points of full context on development data, falling back to full context if necessary. EasyTPP additionally required RMSE within 0.5%.

SettingSelected windowsTest accuracy: full → selected (%) ↑
EasyTPP FT, 15 source/seed cells1 in 11 cells; 2–8 otherwise58.84 → 58.84
LDP, lag 256, three seeds256, 512, 2,04854.62 → 54.60
LDP, lag 1,024, three seeds1,024, full, full54.72 → 54.71

EasyTPP test RMSE rises by 0.15% (3.513 → 3.518). This is a retrospective development-based analysis. Fixed windows bound attention storage independently of stream length, but the lag-1,024 row shows that preserving accuracy does not always permit a cache capped at the lag. The useful cache size depends on the measured retrieval gain and the accuracy loss allowed.

Manuscript changes (orange): Sec. 4.5; Appendix J.6, Table 22 (development-selected cache windows).

Thank you again for encouraging us to test longer natural histories and to account for the cost of attention. These suggestions helped us state more precisely where retrieval gains are observed and what memory they require.

Thank you for your valuable feedback and suggestions. If your concerns have been addressed, we would greatly appreciate that you can raise the score. If you have any further concerns, we would be happy to address them before the rebuttal deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.