acceptodds
Under review as a conference paper at ICLR 2027

Decomposing Irregular Time Series into Interpretable Temporal Patterns with Mixture-of-Experts

Abstract

Irregularly-spaced sequential data, such as transaction logs, encode multi-scale temporal structure that is critical for prediction but difficult to recover explicitly with standard sequence models, which often trade interpretability for performance. We introduce MoTIF, an interpretable framework using mixture-of-experts (MoE) attention to decompose irregular time series into temporal patterns spanning minute-to-year scales. Each expert is trained via temporal alignment that induces attention specialization to a designated cyclical or within-range pattern, with label-free routing supervision that learns to activate the correct expert from patterns present in the data. On synthetic data with known temporal structure, MoTIF recovers correct routing well above chance (0.389/0.829 for range/cyclical patterns vs. 0.091 random) and outperforms all baselines, including a standard transformer and an unspecialized MoE, on next-value prediction. On a public credit card transaction dataset, MoTIF achieves higher macro-F1 than all baselines for next-amount prediction. Its learned routing reveals structure a black-box model cannot: transactions are governed primarily by intraday and short-interval rhythms, while fraudulent transactions show bursty, calendar-agnostic timing versus the steadier periodicity of legitimate ones. On downstream fraud detection, MoTIF matches standard transformer and MoE baselines and clearly outperforms recent sequential recommendation models used as embedding baselines. These results show that decomposing temporal structure into interpretable, expert-specific patterns need not sacrifice predictive accuracy, offering a practical route to sequence models that are both accurate and auditable in high-stakes settings. Code is available at https://anonymous.4open.science/r/MoTIF-82E4.

Primary area:
learning on time series and dynamical systems
Keywords:
mixture-of-experts, interpretable machine learning, irregularly-spaced time series, multi-scale modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RuAHOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MoTIF assigns predefined temporal semantics to eleven attention experts: five calendar-cycle experts and six elapsed-range experts spanning minute-to-year scales. Analytic kernels supervise attention specialization. A detached teacher uses kernel-based predictive errors and temporal structure-function fits to supervise routing without ground-truth pattern labels. In the explicit formulation, expert queries are mixed before softmax attention (Appendix C.1, Equations 22-23, p. 17).

The model is pretrained for next-value prediction and evaluated on a controlled synthetic dataset and a public credit-card transaction dataset. The latter is also simulator-generated, as disclosed in the ethics statement and Appendix B.2. Synthetic routing recovery is 0.829 for cyclical patterns and 0.389 for range patterns, compared with 1/11, approximately 0.091, for uniform random expert assignment (Table 1, p. 7). Downstream fraud evaluation freezes the backbone and trains a DCN head or an embedding-only GBT probe (Appendix C.2, p. 18).

Critical strengths

The main strengths of this paper are listed as follows:

  1. Specific temporal semantics. Named calendar and elapsed-range patterns make expert behavior inspectable and connect model structure to understandable temporal concepts.
  2. Evidence beyond attention visualizations. Known generating patterns permit routing-recovery tests. The paper also evaluates specialization, routing diversity, and agreement with held-out kernel-based temporal statistics, including control signals (pp. 7-9 and Appendix C.4).
  3. Competitive prediction with an interpretation objective. Amount-prediction results improve over FAME on several metrics, and fraud-detection performance is broadly comparable. Maintaining predictive performance while adding structured diagnostics is a legitimate positive result; uniformly outperforming all baselines is not necessary for this contribution.
  4. Explicit interpretability and evaluation qualifications. The manuscript describes routing as a structural diagnostic rather than a causal explanation (p. 3). It discloses simulator provenance, three-seed reporting, frozen-backbone evaluation, and amount-specific routing supervision.
Critical weaknesses

I found some critical weaknesses in the methodology and the numerical evaluations, listed as follows:

  1. Component attribution is not sufficiently isolated. The comparison with unspecialized FAME tests the complete package, but does not separately establish the contributions of attention alignment, the routing teacher, and their interaction. A direct analytic-kernel model with a learned router would also help establish why learned query experts are needed. These are more actionable concerns than treating comparable fraud accuracy as a defect.
  2. The generality of temporal recovery remains uncertain. The range-based synthetic data use Ornstein-Uhlenbeck dynamics, and the teacher uses the corresponding structure function. This is a valid controlled recovery experiment under matching assumptions, not an invalid or label-leaking evaluation. However, it provides limited evidence about mixed patterns, changing correlation scales, or signals outside the expert library. The simulator-generated transaction dataset does not resolve this external-validity gap (Appendices B.1-B.2, pp. 15-17).
  3. Structural routing should not be equated with additive predictive attribution. Query mixing followed by softmax generally differs from mixing standalone expert attention outputs. Routing can therefore be meaningful as a pattern signature without giving proportional contributions to the prediction. The authors acknowledge the noncausal scope; intervention or perturbation tests would strengthen any stronger language about predictive decomposition. Agreement with the prescribed training kernels alone does not fully establish that stronger interpretation.
  4. The fraud protocol needs more precise generalization claims. Appendix B.2 balances positive and negative target transactions and gives transaction split proportions, but does not clearly establish chronological or account-disjoint separation for the fraud task. This is a clarification request, not a finding of leakage. AUPRC and operating-point metrics at the sampled prevalence do not directly establish performance under natural fraud prevalence. The text also needs to align its transaction-time versus next-transaction description of the GBT probe with Table 8's caption (pp. 17 and 23).
  5. Task trade-offs and computational cost should be contextualized. Table 6 reports slightly weaker category prediction than FAME, consistent with the amount-based teacher. Table 7 shows very similar DCN fraud results. The approximately 157 GB peak GPU memory is reported for the authors' training configuration, not as a demonstrated minimum deployment requirement (Appendix C.3, p. 19). Efficiency comparisons would clarify the cost of the interpretation mechanism.
Questions for authors
  1. Can attention-alignment-only, routing-supervision-only, and direct analytic-kernel baselines isolate the contribution of each component while preserving the other training settings?
  2. Does recovery remain meaningful for multiple simultaneous patterns, continuously changing scales, or processes that violate the teacher's structure-function assumptions?
  3. Under controlled changes to timestamps or the corresponding value patterns, do expert routing and predictions change as expected? Such tests would validate structural sensitivity without requiring a causal interpretation.
  4. Please specify the fraud split unit and chronology, the representation window used by each probe, and whether the flagged transaction is observed. How are transaction-time detection and next-transaction prediction distinguished in Table 8?
  5. Can evaluation retain the natural fraud prevalence, and can training and inference costs be compared with FAME at matched sequence length and batch size?
9 comments

We sincerely thank the reviewer for the time and attention they put into carefully reading and providing insightful question and comments on our paper, especially concerning isolating the contributions of individual model components, testing routing recovery under more complex temporal patterns, validation of sensitivity to alterations in input sequences, and clarification of our fraud experiments. These additional experiments and clarifications have helped to improve our work. We have responded to each of the reviewer questions below, and have uploaded a revised PDF with updates in blue.

AuthorsOpenReview ↗

Question 1: Contribution of each model component

We thank the reviewer for their helpful question. We have run ablation experiments to test the performance of our model after ablating attention-alignment, ablating routing-supervision, and replacing the learned expert attention queries with their direct analytic-kernels. Results are shown in Table 14 below.

Table 14: MoTIF ablations and the analytic-kernel baseline (mean over 6 runs). Differences are relative to MoTIF.

AccuracyMacro-F1Range recoveryCyclical recoveryAlignment gap
MoTIF0.7350.6600.360.792.85
w/o alignment0.7310.6320.280.380.05
Difference-0.004-0.028-0.08-0.41-2.80
w/o routing supervision0.7380.6490.020.344.40
Difference+0.003-0.011-0.35-0.44+1.54
Analytic kernels + routing supervision0.7370.6410.320.00—
Difference+0.002-0.019-0.04-0.79—
Analytic kernels, w/o routing supervision0.7370.6540.060.60—
Difference+0.002-0.005-0.30-0.19—

Attention-alignment ablation. We retrained MoTIF on the synthetic benchmark with the temporal-alignment loss weight set to 0 and all other settings unchanged, and compared it with the full model over 6 independent training runs. Routing recovery is the fraction of test sequences whose top-routed expert is the expert that generated them, reported separately for range and cyclical patterns (chance: 0.09). The alignment gap measures how much more closely each expert's attention matches its own target kernel than the other experts' kernels (higher is better).

Removing the alignment loss degrades both prediction and routing. Macro-F1 drops from 0.660 to 0.632, and cyclical-expert recovery falls from 0.79 to 0.38. Range-expert recovery decreases less (0.36 to 0.28), since it is mainly driven by the routing-supervision loss (see below). The alignment gap collapses from 2.85 to 0.05: without the alignment loss, the experts' attention no longer matches their intended temporal kernels, so the experts lose their distinct temporal specializations. The alignment loss is therefore necessary both for interpretable cyclical routing and for the model's macro-F1.

Routing-supervision ablation. Without routing supervision, range-expert recovery collapses from 0.36 to 0.02 and cyclical-expert recovery falls from 0.79 to 0.34. The experts themselves remain well specialized (the alignment gap is even higher, 4.40, since the alignment loss is now the only expert-specific objective), but the router no longer sends sequences to the expert that matches their generating pattern. Macro-F1 decreases from 0.660 to 0.649, while accuracy is essentially unchanged (0.738 vs.\ 0.735). Together with the previous ablation, this shows that the two losses play complementary roles: the alignment loss makes the experts specialize to their temporal kernels, and the routing-supervision loss makes the router use them correctly. Neither alone reproduces the full model's routing.

Direct analytic-kernel baseline. We built a mixture-of-experts in which each expert is a fixed attention matrix given by its target kernel (the same cyclical and range kernels used by MoTIF), with no learned query or key projections. The router assigns weights to the experts, and the attention used in each layer is the router-weighted sum of the fixed attention matrices. The router, value and output projections, feed-forward layers and load balancing are unchanged. This model has 11.4M parameters, versus 17.7M for MoTIF; the difference is exactly the learned expert projections. We trained it with and without the routing-supervision loss (6 independent runs each). The alignment loss is not applicable, because the experts' attention equals the target kernels by construction.

The analytic-kernel model reaches similar predictive performance (macro-F1 0.641--0.654 vs.\ 0.660 for MoTIF), but its routing is not interpretable. With routing supervision, cyclical-expert recovery collapses to 0.00: every cyclical sequence is routed to a range expert. Without routing supervision, cyclical recovery is 0.60 but range-expert recovery is at chance (0.06). Neither variant recovers both expert families, whereas MoTIF does (0.36 and 0.79).

We have added these results to the revised PDF, in Section D.4.

AuthorsOpenReview ↗

Question 2: Routing recovery under multiple simultaneous patterns, continuously changing scales, violation of structure-function assumptions

We appreciate the reviewer's helpful suggestion to further evaluate our model's behavior under additional settings. We have performed experiments for testing multiple simultaneous patterns, as well as an OU structure-function ablation experiment to help assess the utility of this assumption within our model.

Multiple simultaneous patterns. We test whether expert specialization generalizes to sequences driven by two temporal patterns at once. We add twelve additive mixed patterns spanning three family combinations (cyclical+cyclical, cyclical+range, and range+range) with four adjacent-scale pairs each (minute+hour, hour+day, day+week, week+month). Each superimposes its two unit-scale signals via a weighted sum, so the two corresponding experts have an independent signal each to detect. These sequences are generated for validation/test only (6464 per pattern) and never seen in training, so the previous MoTIF and FAME checkpoints (trained on single pattern sequences only) are evaluated without retraining. Since each has K=2K{=}2 generating experts, we report coverage@K=∣topK∩Tp∣/KK = |\mathrm{top}K \cap T_p| / K (the top-KK router weights vs.\ the generating experts TpT_p; K=1K{=}1 recovers single simultaneous patterns) and \emph{both-in-top-2}, the fraction with coverage =1=1; chance is 0.180.18 and 0.0180.018, respectively. Results are in Table 15. MoTIF recovers at least one generating expert well above chance for every combination and both experts for the cyclical+cyclical and range+range cases, while cyclical+range is hardest: the dominant cyclical expert is found but the single range expert is rarely co-selected. FAME stays at chance throughout. We have added these results to the revised PDF in Appendix Section D.5.

Table 15: Multiple simultaneous patterns. Routing on mixed sequences driven by two temporal patterns at once (additive), using the checkpoints trained on single-pattern sequences without retraining. Routing is split by the families of the two generating experts. coverage@2 is the fraction of a sequence's two generating experts in the top-2 routing weights (both-in-top-2 = both recovered); chance 0.18 and 0.018. Mean ± std over seeds {42,43,44}.

MetricMoTIF (ours)FAME (no specialization)
coverage@2 — cyclical+cyclical (↑)0.786 ± 0.0440.120 ± 0.045
coverage@2 — cyclical+range (↑)0.393 ± 0.0070.147 ± 0.025
coverage@2 — range+range (↑)0.529 ± 0.0080.180 ± 0.008
both-in-top-2 — cyclical+cyclical (↑)0.638 ± 0.0600.004 ± 0.006
both-in-top-2 — cyclical+range (↑)0.014 ± 0.0120.012 ± 0.003
both-in-top-2 — range+range (↑)0.333 ± 0.0150.006 ± 0.007

Structure-function assumptions. We note that our approach specifically defines the types of patterns it is designed to detect, namely, the cyclical and range-based patterns. Specifically, it defines ``range-based" patterns as sequences that contain autocorrelative patterns within a specific scale (e.g. minute, hour, day, etc.). The way we generate sequences with such pattern is through the OU structure function. Rather than attempt to generate these autocorrelative patterns via an alternative generating function, we ablated the OU structure function assumption from the model and observed the impact on the prediction performance. To do this, we replace OU structure function assumption within the routing teacher with the forecast-error signal used within the cyclical family, leaving everything else unchanged, and retrain the model (6 independent runs; Table 8 in above response to Reviewer Qa9Q Question 3 - https://openreview.net/forum?id=5knLkeq7P6&noteId=Ultcb8wc8W).

Replacing the OU structure function with the error signal led to notable reductions in routing recovery and the alignment gap (-0.14 for both range and cyclical recovery; -0.18 for alignment gap). This suggests that the OU structure-function assumption is beneficial to performance when the range-based synthetic sequences were generated from the OU structure function, as would be expected. Although it still relies on the assumption that the range-based sequences were generated from an OU structure function, the result is still useful, as it demonstrates that error signal is not sufficient to detect this pattern. It is possible that range-based autoregressive patterns in real data do not follow the OU structure function, but this is beyond the scope of our current model - perhaps future work could explore incorporating additional range-based experts that model alternative autoregressive generating functions, or that could more flexibly learn the generating function from the data. These results have been added to Appendix Section D.3.

Reviewer_LGL6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MoTIF, a model that aims to identify meaningful temporal patterns in irregular time series while maintaining predictive performance. It uses 11 predefined experts covering calendar cycles and dependencies over different elapsed-time scales. A router assigns weights to these experts at each time step.

The authors use analytic temporal kernels to guide each expert’s attention toward historical events that match its assigned pattern. They also construct routing supervision from prediction errors and autocorrelation statistics. This avoids the need for manually annotated pattern labels, although the expert types and supervision rules are predefined.

Critical strengths

1 The meaning of each expert is clearly defined, and the authors back this up with quantitative checks. Each expert is tied to a specific period or time scale; the authors constrain its attention with a temporal kernel and then test, on synthetic data with known generative patterns, whether the routing behaves as expected. These experiments give the experts’ semantics some grounding and make the model’s behavior easier to inspect.

2 The experiments also look at both interpretability and predictive performance. The authors compare against a FAME variant without temporal-pattern specialization, and add a GBT experiment that uses only the embeddings to see whether the representation itself carries fraud-detection signal. The results show that, even with these temporal-pattern constraints, the model remains competitively predictive overall.

Critical weaknesses
  1. The temporal decomposition claims need stronger support. Equations (22)–(23) mix expert queries before softmax, so routing weights do not directly measure each pattern’s contribution to the prediction. Although the authors describe routing as a structural diagnostic, some claims imply more. The validation also closely follows the model’s assumptions: raw-data checks reuse the teacher’s temporal kernels, and synthetic sequences contain single patterns with teacher-aligned dynamics. With the credit card data also simulated, it remains unclear whether the model can separate coexisting patterns or handle temporal structure beyond the predefined families.

  2. The experiments do not clearly isolate what the proposed design adds. Appendix C.3 reports common settings for hidden dimension, attention heads, and depth, but these do not guarantee comparable parameter counts across architectures. MoTIF introduces 11 experts and a router, yet the paper does not report model sizes or provide a parameter-matched Transformer comparison. FAME is a more relevant control for the additional capacity, but it is unclear whether its expert configuration and parameter count match MoTIF’s. Without component ablations or a baseline that uses the teacher’s routing directly, it is also difficult to tell which parts of the training objective are necessary and what the learned router adds beyond the handcrafted teacher.

3.The PANTHER comparison needs further clarification as well. Its CCT fraud-detection F1 is 0.8199 in this paper, compared with 0.911 in the original report. The paper should explain these differences and identify which PANTHER components and prediction signals were retained. This matters when interpreting the reported advantage over existing methods.

Questions for authors
  1. How should we interpret the routing weights given that expert queries are combined before softmax? Are they intended to indicate consistency with predefined temporal patterns, or each pattern’s contribution to the prediction? What existing evidence supports this interpretation?

  2. What are the parameter counts of MoTIF and the baselines? In particular, does FAME use the same number of experts and the same expert and router architectures as MoTIF? This would help separate the effect of temporal specialization from additional model capacity.

  3. PANTHER’s CCT fraud-detection F1 is 0.8199 here, compared with 0.911 in the original paper. Which differences in data processing, training, or downstream inputs could explain this gap? Which components of the original PANTHER pipeline were retained?

  4. Are any component ablations or teacher-only routing results already available? These would help clarify what the learned router adds beyond the handcrafted teacher and which training objectives account for the reported gains.

6 comments

We greatly appreciate the time and care the reviewer took in reading our paper and providing insightful comments and questions which helped improve our work, particularly with regard to interpreting the routing weights, differences from the PANTHER baseline's original pipeline, and clarification of the contributions of individual model components. We have addressed each of the reviewer's questions below. Corresponding updates within the uploaded revised PDF are in blue.

AuthorsOpenReview ↗

Question 1: Interpreting the routing weights

We thank the reviewer for their question. The routing weights (π\pi in equation 22) control how much each expert contributes to the final query before the attention computation. Our interpretation focuses on these routing weights to assess each pattern's contribution to the resulting attention, and thus also the impact on the downstream prediction. We acknowledge the reviewer's concern regarding the fact that the expert queries are combined before softmax within the attention computation, but because the interpretability framework looks at the router weights rather than the attention weights, we can still get a measure of the relative contribution of each pattern to the model's output. We have added this clarification to the revised PDF in section C.1 where the expert query attention is described.

For existing evidence supporting this interpretation, we point the reviewer to the synthetic data experiment results, especially for routing recovery (Section 4.1, Table 1, and Appendix Section D.1, Figure 3B). These results show that, for synthetic sequences generated with known temporal patterns, MoTIF model tends to assign higher routing weights to the experts specialized for correct known temporal patterns. This demonstrates that the routing patterns read out from the model can accurately recover the true underlying generative pattern for these sequences. On the other hand, the matched-capacity baseline with unspecialized attention experts (FAME) showed random routing, with a more even spread across all experts.

Additionally, in response to other reviewer questions, we conducted more experiments to further evaluate routing recovery under alternative settings. For additional evidence, you may be interested in viewing our responses to Reviewer RuAH Question 2 (routing under multiple simultaneous patterns shows recovery of at least one generating expert well above chance for every tested combination - https://openreview.net/forum?id=5knLkeq7P6&noteId=TaR2TNnTfy) and Reviewer RuAH Question 3 (effect of timestamp shuffling and pattern swapping within the input sequences on routing patterns - https://openreview.net/forum?id=5knLkeq7P6&noteId=OXipaddKlH).

AuthorsOpenReview ↗

Question 2: Model capacity

Please see the below table for the parameter counts of MoTIF and all baselines based on the configurations used for the synthetic and CCT dataset experiments. The largest models are MoTIF and FAME, followed by Transformer. Notably, FAME uses the same number of experts (11), and the expert and router architecture is identical to MoTIF, so the capacity is matched; we just add the expert specialization and routing supervision components, which are additional loss terms. Thus, any added benefits we see from MoTIF over FAME can be attributable to its training strategy for expert specialization and routing supervision rather than additional model capacity. We have added the below table to the revised PDF within Appendix Section C.3.

Table 13: Model parameter counts from synthetic and CCT dataset configurations.

Model / checkpointParameter Count (synthetic)Parameter Count (CCT)
MoTIF17.72M20.73M
FAME17.72M20.73M
Transformer12.47M15.48M
PANTHER0.29M1.09M
SASRec0.20M1.00M
HSTU0.16M0.96M
MHSA0.14M0.94M
Reviewer_Qa9QOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MoTIF addresses interpretable prediction over irregularly-sampled event sequences (e.g., transaction logs). The transformer backbone's per-head attention query is produced by a mixture-of-experts layer whose 11 experts are specialized to named temporal patterns: 5 cyclical experts aligned to von-Mises kernels over calendar phases (minute-of-hour through month-of-year) and 6 range experts aligned to causal Gaussian kernels over elapsed time at minute-to-year scales. Specialization is induced by a cross-entropy temporal alignment loss between each expert's attention and its analytic target kernel. Because true pattern labels are unavailable, a label-free routing supervision scheme builds a detached teacher distribution: per-expert autoregressive prediction error weighted by the target kernels decides among cyclical experts, a "cyclical-first" gate splits the pattern families, and within-range assignment comes from fitting an Ornstein-Uhlenbeck structure function to empirical squared increments. The router is trained toward this teacher with three decoupled KL terms plus a load-balancing loss. Phase 1 pretrains on next-token prediction; Phase 2 fine-tunes a DCN head for fraud detection. On synthetic data with known patterns, MoTIF recovers routing well above chance (0.389 range / 0.829 cyclical vs. 0.091 random) and beats all baselines on next-value prediction. On the IBM CCT credit-card dataset it achieves the best macro-F1, MAE, and RMSE for next-amount prediction (accuracy is second to FAME), matches the transformer and MoE baselines on fraud detection, and its routing reveals cohort-level structure: intraday and short-range experts dominate overall, fraud shifts toward short-range, calendar-agnostic experts, and routing alone yields AUROC 0.80 for fraud.

Critical strengths
  1. A concrete, validated mechanism for assigning fixed semantics to attention experts. The combination of kernel-alignment loss and label-free routing supervision, verified on synthetic data with known patterns (CE gap 2.850 vs. 0.026 for the unspecialized FAME baseline; routing recovery far above chance), shows that expert specialization can be engineered rather than hoped for post hoc. This is a useful methodological template beyond this application.
  2. A well-executed interpretability case study on transaction data. The routing-based analysis (expert prevalence, validation against independently computed data statistics, fraud-cohort contrast, and a routing-only fraud probe at AUROC 0.80) produces readable, checkable structure rather than post-hoc rationalization.
Critical weaknesses
  1. The real-data evidence is too narrow to support the paper's claims. There is one real dataset, and it is itself the output of a simulation (the IBM-generated CCT corpus). Predictive differences from the unspecialized FAME baseline are marginal (macro-F1 0.7374 vs. 0.7320, and accuracy is slightly worse), and no second domain (healthcare, clickstream, industrial logs) is tested. The central selling point, an interpretable decomposition that does not cost accuracy, is therefore demonstrated only in the most favorable setting.
  2. Circularity and faithfulness gaps in the interpretability argument. The teacher that supervises routing is built from the same kernel family used to define the experts, so recovering those patterns on synthetic data is partially guaranteed by construction. No causal or faithfulness test shows that the model's predictions actually rely on the routed expert's pattern (e.g., expert ablation or kernel shuffling at test time). Without this, routing remains a structured diagnostic, as the authors themselves concede, which weakens the "auditable in high-stakes settings" framing.
Questions for authors
  1. Can you add a faithfulness experiment: ablate or permute a dominant expert (e.g., hour-of-day) at test time and show that predictions change in the pattern-consistent direction and magnitude? This would substantially strengthen the interpretability claim.
  2. Is there a second real dataset (even one without fraud labels) where the routing decomposition recovers sensible structure? Generalization of the main finding is currently unsupported.
  3. How sensitive is routing recovery to the teacher's design choices: the cyclical-first gate threshold, the OU structure-function assumption, kappa = 4, and the three KL weights? Which stages are necessary?
  4. On CCT, FAME matches or slightly exceeds MoTIF on accuracy (0.6655 vs. 0.6615) and fraud detection. What exactly is the quantitative price/benefit of specialization on this dataset?
  5. Please fix the "ICLR 2025" page header, the Table 6 vs. Table 3 mismatch (the next-amount results are in main-text Table 3; Table 6 is next item category prediction), and the "PATHER" typo in Table 2.
11 comments
AuthorsOpenReview ↗

Question 1: faithfulness experiment - hour-of-day expert ablation

Thank you for your suggestion. We have performed faithfulness experiments, first by suppressing the hour-of-day expert at test time and observing both the resulting routing patterns and prediction changes. Then, we repeated the experiment for each of the 11 experts.

Hour-of-day expert ablation - Routing Results: Using the three trained MoTIF checkpoints reported in the paper (three independent training runs), frozen with no retraining, we suppressed the hour-of-day expert in every attention layer and renormalized the remaining routing weights. We then examined how the test sequences generated by the hour-of-day pattern are re-routed. To make the substitution explicit, Table 1 reports hard top-1 routing in the final layer, averaged over the three checkpoints.

Table 1: Final-layer top-1 routing of hour-of-day sequences before and after suppressing the hour-of-day expert (mean over 3 checkpoints).

ExpertShare beforeShare afterChange (pts)
hour-of-day (cyclical, suppressed)21.2%0.0%-21.2
minute-of-hour (cyclical)9.4%13.5%+4.1
day-of-week (cyclical)10.2%13.3%+3.1
within-month (range)5.6%8.5%+2.9
month-of-year (cyclical)13.9%16.7%+2.8
week-of-month (cyclical)10.7%13.1%+2.4
within-minute (range)5.3%7.6%+2.3
within-hour (range)3.8%5.3%+1.5
within-week (range)2.9%4.0%+1.1
within-day (range)2.4%3.3%+0.9
within-year (range)14.6%14.8%+0.2

The rerouting is consistent with the temporal structure of the experts. The largest gains go to the adjacent cyclical experts, minute-of-hour (+4.1+4.1) and day-of-week (+3.1+3.1), followed by within-month (+2.9+2.9) and the remaining cyclical experts, month-of-year (+2.8+2.8) and week-of-month (+2.4+2.4). The other range experts gain less (within-minute +2.3+2.3, within-hour +1.5+1.5, and +2.2+2.2 combined for within-day, within-week and within-year). Under the model's native soft routing, hour-of-day is the top-routed expert for these sequences, but by a small margin: it receives a routing weight of 0.105, versus 0.089--0.092 for each other expert. When it is suppressed, its weight is redistributed almost uniformly, with each remaining expert gaining about 0.010. The most frequent new top expert per sequence is the within-hour range expert in two of the three checkpoints (30% and 27% of sequences) and within-month in the third (33%).

Hour-of-day expert ablation - Prediction Results: For each token of the hour-of-day sequences, we compared the change in the predicted next amount (expected bucket) with the forecast of the expert that replaces hour-of-day.

As shown in Table 2, predictions move in the direction of the new top expert's forecast in most cases (63.5% under soft routing), but the size of the change is only weakly related to the size of the forecast shift (Spearman 0.12--0.18). The direction is therefore consistent with the replacement expert, and the magnitude is only partly so. The change is specific to hour-of-day sequences: under top-1 routing, 12.0% of their predictions change, compared with 2.5--4.4% for sequences of other patterns.

Table 2: Prediction changes on hour-of-day sequences after suppressing the hour-of-day expert (mean over 3 models).

Soft routingTop-1 routing
Predictions changed1.0%12.0%
Accuracy before0.7050.664
Accuracy after0.7030.652
Direction agrees with new top expert's forecast63.5%54.6%
Magnitude correlation with forecast shift (Spearman)0.120.18
AuthorsOpenReview ↗

Question 1 (cont.): faithfulness experiment - ablations of each of the 11 experts

Individually ablating each of the 11 experts: We repeated the top-1 intervention for each of the 11 experts, suppressing one expert at a time in the same three frozen checkpoints. Table 3 reports, for each suppressed expert, accuracy and the share of changed predictions on the sequences generated by its own pattern, accuracy over all sequences, and the share of changed predictions on sequences of the other patterns. We see that predictions more frequently changed for sequences generated from the pattern corresponding to the suppressed expert ("own pattern"), compared to sequences generated from "other patterns". Additionally, for most experts (all cyclical and 3/6 range-based), we see reductions in prediction accuracy for sequences generated by the ablated expert's "own pattern". Furthermore, these reductions in accuracy tend to be larger than for sequences generated from "other patterns".

Table 3: Suppressing one expert at a time (top-1 routing, mean over 3 models). "Own" = sequences generated by the suppressed expert's pattern; "All" = All sequences; "Other" = Other patterns.

Suppressed expertOwn: Acc beforeOwn: Acc afterOwn: ChangedAll: Acc beforeAll: Acc afterOther: Changed
minute-of-hour0.5940.59112.7%0.6790.6813.3%
hour-of-day0.6640.65212.0%0.6790.6793.4%
day-of-week0.7290.71410.4%0.6790.6783.5%
week-of-month0.7810.76110.7%0.6790.6783.7%
month-of-year0.8310.8138.6%0.6790.6793.8%
within-minute0.4340.43510.2%0.6790.6784.1%
within-hour0.5280.5274.9%0.6790.6782.6%
within-day0.6490.6456.1%0.6790.6774.3%
within-week0.7290.7376.7%0.6790.6835.8%
within-month0.7910.8025.2%0.6790.6855.2%
within-year0.8430.8568.9%0.6790.68410.5%

We have added all of the above Question 1 results to Appendix Section D.2.

AuthorsOpenReview ↗

Question 2: additional real dataset experiments

Thank you for the suggestion to help demonstrate the generalizability of our approach. We have conducted additional experiments applying MoTIF and the baselines to multivariate time series ICU data from PhysioNet 2012 [1,2]. We added dataset details to the Appendix, Section B.4. We selected the 17 most frequently measured clinical signals, including heart rate, blood pressure, urine output, temperature, Glasgow Coma Score, respiratory measures, pH, blood carbon dioxide and oxygen, and hematocrit. Since the time series are limited to the first 48 hours after ICU admission, we utilized only the four temporal experts applicable to this time range: cyclical hour-of-day and minute-of-hour, and range within-minute and within-hour. We keep a backbone with hidden dimension 512, two attention heads, and two layers, and use maximum sequence length of 256. We split the data into train, validation, and test, with results reported as mean ±\pm standard deviation over three independent runs (with seeds {42, 43, 44}).

We evaluated the temporal alignment and routing decomposition to assess expert specialization and identify any dominant temporal patterns detected by our model. Additionally, we evaluated performance against the baselines on next-value prediction (phase 1) and downstream in-hospital mortality prediction (phase2) to check the generalizability of the prediction performance.

Explainability: temporal alignment and routing diversity. We compare MoTIF (4 temporal pattern-specific experts: cyclical hour-of-day and minute-of-hour, and range within-minute and within-hour) with the FAME control (matched capacity, with 4 unspecialized experts). Alignment metrics compare each expert's attention with its assigned temporal kernel. Routing metrics summarize argmax dispatch over the four experts. Entropy ratio is normalized Shannon entropy (H) of expert usage (1 indicates uniform usage and 0 indicates use of one expert); effective number of experts is exp(H). Results are shown in Table 4.

Table 4: Expert specialization and routing diversity for the four-expert backbone. The random own-kernel rank is 1.5. Higher entropy ratios and effective expert counts indicate more diverse routing.

MetricMoTIF/FAME (ours)FAME (no spec.)
Expert specialization
Alignment diagonal CE (own kernel) ↓2.105 ± 0.0145.357 ± 0.161
Alignment off-diagonal CE4.508 ± 0.0065.378 ± 0.101
Alignment gap (off-diagonal - diagonal) ↑2.403 ± 0.0190.020 ± 0.061
Experts whose own kernel is best (/4)1.000 ± 0.0001.333 ± 0.471
Mean own-kernel rank (0 = best, 1.5 = random) ↓1.167 ± 0.1181.417 ± 0.118
Routing diversity
Token-dispatch entropy ratio ↑0.505 ± 0.0040.985 ± 0.015
Token-dispatch effective experts (/4)2.000 ± 0.0003.900 ± 0.082
Across-sequence dominant-expert entropy ratio ↑0.387 ± 0.1130.705 ± 0.180
Across-sequence effective dominant experts (/4)1.733 ± 0.2492.767 ± 0.634
Across-sequence top-expert share0.711 ± 0.1600.620 ± 0.131
Experts ever dominant (/4)2.000 ± 0.0004.000 ± 0.000

For the specialization assessment, we see that the cross-entropy between each expert's learned attention matrix and its own target kernel is lower than that with the target kernels of other experts, and that this gap. is notably larger than for the unspecialized FAME baseline (2.403 versus 0.020), consistent with the results on the synthetic and CCT datasets. Additionally, the mean own-kernel rank differed meaningfully from random, whereas that for FAME did not (1.167±0.1181.167 \pm 0.118 for MoTIF versus 1.417±0.1181.417 \pm 0.118 for FAME, with 1.5 indicating random).

[1] Tom Pollard, Benjamin E. Moody, Li-wei H. Lehman, Brian J. Gow, Chrystynne Fernandes, Chen Xie, Alistair Johnson, Roger G. Mark, and Thomas Heldt. PhysioNet as a global platform for biomedical research. Na- ture Health, 1(8):792–795, August 2026. ISSN 3005-0693. doi: 10.1038/s44360-026-00096-z. URL https: //doi.org/10.1038/s44360-026-00096-z.

[2] Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 computing in cardiology, pp. 245–248. IEEE, 2012.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.