- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MoTIF assigns predefined temporal semantics to eleven attention experts: five calendar-cycle experts and six elapsed-range experts spanning minute-to-year scales. Analytic kernels supervise attention specialization. A detached teacher uses kernel-based predictive errors and temporal structure-function fits to supervise routing without ground-truth pattern labels. In the explicit formulation, expert queries are mixed before softmax attention (Appendix C.1, Equations 22-23, p. 17).
The model is pretrained for next-value prediction and evaluated on a controlled synthetic dataset and a public credit-card transaction dataset. The latter is also simulator-generated, as disclosed in the ethics statement and Appendix B.2. Synthetic routing recovery is 0.829 for cyclical patterns and 0.389 for range patterns, compared with 1/11, approximately 0.091, for uniform random expert assignment (Table 1, p. 7). Downstream fraud evaluation freezes the backbone and trains a DCN head or an embedding-only GBT probe (Appendix C.2, p. 18).
Critical strengths
The main strengths of this paper are listed as follows:
- Specific temporal semantics. Named calendar and elapsed-range patterns make expert behavior inspectable and connect model structure to understandable temporal concepts.
- Evidence beyond attention visualizations. Known generating patterns permit routing-recovery tests. The paper also evaluates specialization, routing diversity, and agreement with held-out kernel-based temporal statistics, including control signals (pp. 7-9 and Appendix C.4).
- Competitive prediction with an interpretation objective. Amount-prediction results improve over FAME on several metrics, and fraud-detection performance is broadly comparable. Maintaining predictive performance while adding structured diagnostics is a legitimate positive result; uniformly outperforming all baselines is not necessary for this contribution.
- Explicit interpretability and evaluation qualifications. The manuscript describes routing as a structural diagnostic rather than a causal explanation (p. 3). It discloses simulator provenance, three-seed reporting, frozen-backbone evaluation, and amount-specific routing supervision.
Critical weaknesses
I found some critical weaknesses in the methodology and the numerical evaluations, listed as follows:
- Component attribution is not sufficiently isolated. The comparison with unspecialized FAME tests the complete package, but does not separately establish the contributions of attention alignment, the routing teacher, and their interaction. A direct analytic-kernel model with a learned router would also help establish why learned query experts are needed. These are more actionable concerns than treating comparable fraud accuracy as a defect.
- The generality of temporal recovery remains uncertain. The range-based synthetic data use Ornstein-Uhlenbeck dynamics, and the teacher uses the corresponding structure function. This is a valid controlled recovery experiment under matching assumptions, not an invalid or label-leaking evaluation. However, it provides limited evidence about mixed patterns, changing correlation scales, or signals outside the expert library. The simulator-generated transaction dataset does not resolve this external-validity gap (Appendices B.1-B.2, pp. 15-17).
- Structural routing should not be equated with additive predictive attribution. Query mixing followed by softmax generally differs from mixing standalone expert attention outputs. Routing can therefore be meaningful as a pattern signature without giving proportional contributions to the prediction. The authors acknowledge the noncausal scope; intervention or perturbation tests would strengthen any stronger language about predictive decomposition. Agreement with the prescribed training kernels alone does not fully establish that stronger interpretation.
- The fraud protocol needs more precise generalization claims. Appendix B.2 balances positive and negative target transactions and gives transaction split proportions, but does not clearly establish chronological or account-disjoint separation for the fraud task. This is a clarification request, not a finding of leakage. AUPRC and operating-point metrics at the sampled prevalence do not directly establish performance under natural fraud prevalence. The text also needs to align its transaction-time versus next-transaction description of the GBT probe with Table 8's caption (pp. 17 and 23).
- Task trade-offs and computational cost should be contextualized. Table 6 reports slightly weaker category prediction than FAME, consistent with the amount-based teacher. Table 7 shows very similar DCN fraud results. The approximately 157 GB peak GPU memory is reported for the authors' training configuration, not as a demonstrated minimum deployment requirement (Appendix C.3, p. 19). Efficiency comparisons would clarify the cost of the interpretation mechanism.
Questions for authors
- Can attention-alignment-only, routing-supervision-only, and direct analytic-kernel baselines isolate the contribution of each component while preserving the other training settings?
- Does recovery remain meaningful for multiple simultaneous patterns, continuously changing scales, or processes that violate the teacher's structure-function assumptions?
- Under controlled changes to timestamps or the corresponding value patterns, do expert routing and predictions change as expected? Such tests would validate structural sensitivity without requiring a causal interpretation.
- Please specify the fraud split unit and chronology, the representation window used by each probe, and whether the flagged transaction is observed. How are transaction-time detection and next-transaction prediction distinguished in Table 8?
- Can evaluation retain the natural fraud prevalence, and can training and inference costs be compared with FAME at matched sequence length and batch size?
9 comments
Question 1: Contribution of each model component
We thank the reviewer for their helpful question. We have run ablation experiments to test the performance of our model after ablating attention-alignment, ablating routing-supervision, and replacing the learned expert attention queries with their direct analytic-kernels. Results are shown in Table 14 below.
Table 14: MoTIF ablations and the analytic-kernel baseline (mean over 6 runs). Differences are relative to MoTIF.
| Accuracy | Macro-F1 | Range recovery | Cyclical recovery | Alignment gap | |
|---|---|---|---|---|---|
| MoTIF | 0.735 | 0.660 | 0.36 | 0.79 | 2.85 |
| w/o alignment | 0.731 | 0.632 | 0.28 | 0.38 | 0.05 |
| Difference | -0.004 | -0.028 | -0.08 | -0.41 | -2.80 |
| w/o routing supervision | 0.738 | 0.649 | 0.02 | 0.34 | 4.40 |
| Difference | +0.003 | -0.011 | -0.35 | -0.44 | +1.54 |
| Analytic kernels + routing supervision | 0.737 | 0.641 | 0.32 | 0.00 | — |
| Difference | +0.002 | -0.019 | -0.04 | -0.79 | — |
| Analytic kernels, w/o routing supervision | 0.737 | 0.654 | 0.06 | 0.60 | — |
| Difference | +0.002 | -0.005 | -0.30 | -0.19 | — |
Attention-alignment ablation. We retrained MoTIF on the synthetic benchmark with the temporal-alignment loss weight set to 0 and all other settings unchanged, and compared it with the full model over 6 independent training runs. Routing recovery is the fraction of test sequences whose top-routed expert is the expert that generated them, reported separately for range and cyclical patterns (chance: 0.09). The alignment gap measures how much more closely each expert's attention matches its own target kernel than the other experts' kernels (higher is better).
Removing the alignment loss degrades both prediction and routing. Macro-F1 drops from 0.660 to 0.632, and cyclical-expert recovery falls from 0.79 to 0.38. Range-expert recovery decreases less (0.36 to 0.28), since it is mainly driven by the routing-supervision loss (see below). The alignment gap collapses from 2.85 to 0.05: without the alignment loss, the experts' attention no longer matches their intended temporal kernels, so the experts lose their distinct temporal specializations. The alignment loss is therefore necessary both for interpretable cyclical routing and for the model's macro-F1.
Routing-supervision ablation. Without routing supervision, range-expert recovery collapses from 0.36 to 0.02 and cyclical-expert recovery falls from 0.79 to 0.34. The experts themselves remain well specialized (the alignment gap is even higher, 4.40, since the alignment loss is now the only expert-specific objective), but the router no longer sends sequences to the expert that matches their generating pattern. Macro-F1 decreases from 0.660 to 0.649, while accuracy is essentially unchanged (0.738 vs.\ 0.735). Together with the previous ablation, this shows that the two losses play complementary roles: the alignment loss makes the experts specialize to their temporal kernels, and the routing-supervision loss makes the router use them correctly. Neither alone reproduces the full model's routing.
Direct analytic-kernel baseline. We built a mixture-of-experts in which each expert is a fixed attention matrix given by its target kernel (the same cyclical and range kernels used by MoTIF), with no learned query or key projections. The router assigns weights to the experts, and the attention used in each layer is the router-weighted sum of the fixed attention matrices. The router, value and output projections, feed-forward layers and load balancing are unchanged. This model has 11.4M parameters, versus 17.7M for MoTIF; the difference is exactly the learned expert projections. We trained it with and without the routing-supervision loss (6 independent runs each). The alignment loss is not applicable, because the experts' attention equals the target kernels by construction.
The analytic-kernel model reaches similar predictive performance (macro-F1 0.641--0.654 vs.\ 0.660 for MoTIF), but its routing is not interpretable. With routing supervision, cyclical-expert recovery collapses to 0.00: every cyclical sequence is routed to a range expert. Without routing supervision, cyclical recovery is 0.60 but range-expert recovery is at chance (0.06). Neither variant recovers both expert families, whereas MoTIF does (0.36 and 0.79).
We have added these results to the revised PDF, in Section D.4.
Question 2: Routing recovery under multiple simultaneous patterns, continuously changing scales, violation of structure-function assumptions
We appreciate the reviewer's helpful suggestion to further evaluate our model's behavior under additional settings. We have performed experiments for testing multiple simultaneous patterns, as well as an OU structure-function ablation experiment to help assess the utility of this assumption within our model.
Multiple simultaneous patterns. We test whether expert specialization generalizes to sequences driven by two temporal patterns at once. We add twelve additive mixed patterns spanning three family combinations (cyclical+cyclical, cyclical+range, and range+range) with four adjacent-scale pairs each (minute+hour, hour+day, day+week, week+month). Each superimposes its two unit-scale signals via a weighted sum, so the two corresponding experts have an independent signal each to detect. These sequences are generated for validation/test only ( per pattern) and never seen in training, so the previous MoTIF and FAME checkpoints (trained on single pattern sequences only) are evaluated without retraining. Since each has generating experts, we report coverage@ (the top- router weights vs.\ the generating experts ; recovers single simultaneous patterns) and \emph{both-in-top-2}, the fraction with coverage ; chance is and , respectively. Results are in Table 15. MoTIF recovers at least one generating expert well above chance for every combination and both experts for the cyclical+cyclical and range+range cases, while cyclical+range is hardest: the dominant cyclical expert is found but the single range expert is rarely co-selected. FAME stays at chance throughout. We have added these results to the revised PDF in Appendix Section D.5.
Table 15: Multiple simultaneous patterns. Routing on mixed sequences driven by two temporal patterns at once (additive), using the checkpoints trained on single-pattern sequences without retraining. Routing is split by the families of the two generating experts. coverage@2 is the fraction of a sequence's two generating experts in the top-2 routing weights (both-in-top-2 = both recovered); chance 0.18 and 0.018. Mean ± std over seeds {42,43,44}.
| Metric | MoTIF (ours) | FAME (no specialization) |
|---|---|---|
| coverage@2 — cyclical+cyclical (↑) | 0.786 ± 0.044 | 0.120 ± 0.045 |
| coverage@2 — cyclical+range (↑) | 0.393 ± 0.007 | 0.147 ± 0.025 |
| coverage@2 — range+range (↑) | 0.529 ± 0.008 | 0.180 ± 0.008 |
| both-in-top-2 — cyclical+cyclical (↑) | 0.638 ± 0.060 | 0.004 ± 0.006 |
| both-in-top-2 — cyclical+range (↑) | 0.014 ± 0.012 | 0.012 ± 0.003 |
| both-in-top-2 — range+range (↑) | 0.333 ± 0.015 | 0.006 ± 0.007 |
Structure-function assumptions. We note that our approach specifically defines the types of patterns it is designed to detect, namely, the cyclical and range-based patterns. Specifically, it defines ``range-based" patterns as sequences that contain autocorrelative patterns within a specific scale (e.g. minute, hour, day, etc.). The way we generate sequences with such pattern is through the OU structure function. Rather than attempt to generate these autocorrelative patterns via an alternative generating function, we ablated the OU structure function assumption from the model and observed the impact on the prediction performance. To do this, we replace OU structure function assumption within the routing teacher with the forecast-error signal used within the cyclical family, leaving everything else unchanged, and retrain the model (6 independent runs; Table 8 in above response to Reviewer Qa9Q Question 3 - https://openreview.net/forum?id=5knLkeq7P6¬eId=Ultcb8wc8W).
Replacing the OU structure function with the error signal led to notable reductions in routing recovery and the alignment gap (-0.14 for both range and cyclical recovery; -0.18 for alignment gap). This suggests that the OU structure-function assumption is beneficial to performance when the range-based synthetic sequences were generated from the OU structure function, as would be expected. Although it still relies on the assumption that the range-based sequences were generated from an OU structure function, the result is still useful, as it demonstrates that error signal is not sufficient to detect this pattern. It is possible that range-based autoregressive patterns in real data do not follow the OU structure function, but this is beyond the scope of our current model - perhaps future work could explore incorporating additional range-based experts that model alternative autoregressive generating functions, or that could more flexibly learn the generating function from the data. These results have been added to Appendix Section D.3.
We sincerely thank the reviewer for the time and attention they put into carefully reading and providing insightful question and comments on our paper, especially concerning isolating the contributions of individual model components, testing routing recovery under more complex temporal patterns, validation of sensitivity to alterations in input sequences, and clarification of our fraud experiments. These additional experiments and clarifications have helped to improve our work. We have responded to each of the reviewer questions below, and have uploaded a revised PDF with updates in blue.