acceptodds
Under review as a conference paper at ICLR 2027

Heterogeneous Pattern Representation and Interaction for General Irregular Multivariate Time Series Analysis

Abstract

Time series pattern machines (TSPMs) have been extensively studied for their ability to capture universal patterns and support diverse downstream tasks. However, existing TSPMs primarily target regularly sampled time series. In real-world scenarios, missing observations and differing sampling schedules often give rise to irregular multivariate time series (IMTS), characterized by uneven sampling intervals and asynchronous observations across variates. Developing TSPMs for IMTS is therefore more challenging because it requires capturing heterogeneous patterns, including temporal variations arising from intra-series irregularity and cross-variate dependencies arising from inter-series asynchrony. This calls for complementary representations of these patterns and effective interactions among them. To address these challenges, we propose **HeteroNet**, a TSPM designed for IMTS through **hetero**geneous pattern representation and interaction. It combines (1) Observation Encoding, which preserves the temporal context of individual observations, with (2) Sparse Pattern Tokenization, which explicitly represents recent states, changes over elapsed intervals, and variate histories. Then, (3) Pattern Interaction models dependencies among these heterogeneous patterns within and across asynchronous variates. Finally, (4) Kernel Fusion integrates the resulting representations with temporal kernel summaries of all observations. Extensive experiments across five IMTS analysis tasks show that HeteroNet achieves state-of-the-art performance, outperforming both general-purpose and task-specific baselines.

Primary area:
learning on time series and dynamical systems
Keywords:
Irregular multivariate time series, Heterogeneous pattern interaction, Time series pattern machines

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ptCgOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In order to build a general "pattern machine" for IMTS data, HeteroNet is proposed, which introduce a set of modules focusing on addressing irregularity within series. Extensive experiements domenstrate the competitive performance of the proposed method.

Critical strengths
  1. Extensive experiments: although existing papers mostly focus on classification and forecasting tasks on IMTS, while the true meaning of the rest tasks remain less verified in real-world scenarios, it is undeniable that the experiment is very comprehensive.
Critical weaknesses
  1. It is questionable whether "pattern machine" is reasonable in terms of IMTS. The word rarely appears in time series researches, and the most closely related method would be TimeMixer++. However, the assumptions in TimeMixer++ hold due to extensive studies for regularly sampled time series, but whether tasks like imputation, temporal point process, and generation have correspoding real-world applications are questionable. For example, for the most widely studied IMTS dataset MIMIC, each IMTS sample represents the clinical records from the patient's ICU stay. In this case, what's the point of generating a fake IMTS series or imputing it? For generation, no real patient actual correspond to that fake series. For imputation, irregularity itself carry sampling density information reflecting the patient's disease progression.
  2. The contributin of this paper lies in extensive experiment, rather than any technical novelty. Designing a "general pattern machine" in IMTS is not as simple as accounting for irregularity within a dataset, but also irregularity among different datasets that may be caused by different reasons. Furthermore, the method in this paper is mostly a stacking of existing components, and it's unclear why these modules are chosen other than the rest.
Questions for authors

See weaknesses.

Reviewer_VdgBOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HeteroNet, a time series pattern machine for general analysis of irregular multivariate time series (IMTS). Each observation is embedded with its value, timestamp, elapsed interval, and variate identity. Three token types are then built: the most recent observations, transitions between adjacent observations with the largest intervals, and per-variate profile statistics. A Transformer processes these tokens jointly, and the outputs are pooled into sequence- and variate-level representations fused with Gaussian-kernel summaries of all observations. A separate model with a task-specific head is trained for each of five tasks (classification, interpolation, extrapolation, temporal point process modeling, generation). Across 15 datasets and 40 baselines, it reports the best average performance on all five tasks.

Critical strengths
  • The evaluation is broad. 15 datasets, five tasks, and 40 baselines, with standard deviations reported for most tasks.
  • Gains on interpolation are consistent across all four datasets.
Critical weaknesses
  • There is no evidence on where task-specific IMTS models fall short, or why a single backbone benefits IMTS (e.g., shared representations, transfer).
  • A separate model with a task-specific head is trained per task, Kernel Fusion is removed for TPP, and generation relies on a separately trained latent diffusion model. This is a unified architecture. In other words, a reused encoder design, rather than a unified model, so the claimed benefit of reducing separate model development is weak.
  • The claim that general IMTS models are lacking may not hold. Latent ODE already handles interpolation, extrapolation, and classification; mTAN handles interpolation and classification; ContiFormer handles interpolation, classification, and event prediction.
  • The stated challenges (intra-series irregularity and inter-series asynchrony) are what existing IMTS methods already target. Nothing in them becomes harder because the analysis is general.
  • Embedding timestamps and elapsed intervals is already standard (e.g., mTAN, Warpformer). The transition token is the only component directly tied to Challenge 1, but it is unclear why it is better than existing time embeddings.
  • Asynchrony can be handled by applying self-attention to an unaligned token set (e.g., SeFT). Nothing explains why splitting tokens into observation, transition, and variate types is needed for this.
  • “Heterogeneous pattern interaction” amounts to concatenating the three token types and feeding them to a standard Transformer. There is no interaction mechanism specific to heterogeneous tokens.
  • The design is close to feature engineering. Transition and variate tokens are built directly from raw values, not from the Observation Encoding outputs. The variate tokens and Kernel Fusion are hand-crafted statistics (mean, std, min/max, kernel-weighted means). Kernel Fusion is nearly identical to the RBF kernel interpolation with an intensity channel in IPN (Shukla & Marlin, 2019).
  • Time is normalized per sample to [τmin⁡,τmax⁡][\tau_{\min}, \tau_{\max}], so the same physical interval maps to different values across samples. This conflicts with the motivation that elapsed time is essential, and may hurt extrapolation and TPP, where absolute time scale matters.
  • The five tasks require very different output structures: classification, query-conditioned decoding for interpolation/extrapolation, event-time modeling for TPP, and a generative model for generation. Nothing explains why a single sequence/variate representation can support all of them.
  • TPP uses marked event sequences with xi=1x_i = 1 (no values), so it is hard to regard as an IMTS task.
  • For generation, HeteroNet’s contribution is not separated from that of the latent diffusion model, and results are reported without standard deviations.
  • The main-text comparison for classification, interpolation, and extrapolation relies mostly on six forecasting-oriented baselines. Existing general IMTS models (Latent ODE, mTAN, ContiFormer) and classification-specific models (Raindrop, Warpformer) are missing.
  • The ablation removes only whole modules. There is no ablation by token type (obs/tr/pro) or by transition selection criterion (largest interval vs. random or shortest), and it is unclear what replaces SPT in “w/o SPT”.
Questions for authors

See details in the weaknesses.

Overall, the novelty over existing IMTS components is limited, and no part of the design supports the “general” claim. What do you think about the general meaning?

1 comment

Thank you for the detailed response and the new experiments. Some concerns remain.

  • Benefit of one backbone (W1/W4). Inconsistent baseline rankings show that no single baseline wins everywhere, but not that a shared backbone helps. Table R1 covers only interpolation and extrapolation, which are very close tasks, and frozen transfer is worse than training from scratch in all four settings.
  • Novelty (W7/W8). The interaction is a standard Transformer, and Kernel Fusion is IPN-style. The contribution is mainly the token design, and most gains in Table R3 are small (about 1–3%).
  • w/o SPT and the Observation Transformer. Attention over all observations is 35–39% worse (and up to 64.6% for the Observation Transformer), while removing one token type costs only 1–3%, and replacing transitions with extra observation tokens costs only 1–4%. Why does using all observations hurt so much? Were these variants tuned in the same way (sequence length, truncation, hyperparameters)?

Other concerns are resolved.

Reviewer_Xx5MOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HeteroNet for irregular multivariate time series. It represents observed states, changes over time intervals, and historical profiles of individual variates as observation, transition, and variate-profile tokens, models their relationships through attention, and incorporates temporal kernel summaries of the full observation history. Separate models are trained with task-specific output heads and evaluated on five tasks: classification, interpolation, extrapolation, temporal point process modeling, and generation.

Critical strengths
  1. The research problem has practical relevance. Irregular sampling and asynchronous observations across variates are common challenges in real-world data, and the design is clearly motivated by these issues.
  2. The representation design is clear. Explicitly distinguishing observed states, interval-level changes, and historical profiles of individual variates, while modeling them jointly, provides an intuitive representation scheme for irregular time series.
  3. The experiments support the effectiveness of the full backbone. The evaluation covers multiple tasks, and encoder comparisons with identical output heads and training objectives provide controlled evidence for the full backbone's effectiveness.
Critical weaknesses
  1. The incremental value of the heterogeneous representations and multiple branches is not yet sufficiently established. The main contribution is to jointly model observations, transitions, and variate profiles, supplemented by a kernel branch. However, under the matched readout configuration in Table 6C, removing individual token types causes only small performance changes and even improves some metrics. With the prediction head's direct kernel estimate disabled, the two-branch backbone also does not consistently outperform the single-branch variants on interpolation and extrapolation. The appendix acknowledges these findings, but the ablations report only point estimates, making it difficult to assess whether the small differences are stable. Although encoder comparisons with identical heads and training objectives support the full backbone's effectiveness, these controls do not yet clearly establish the stable gains from retaining all three token types and both branches over the corresponding simplified variants.

  2. The relationship between some ablation settings needs clarification. The generation KL is 2.46 for "w/o Kernel Fusion" in Table 6A, versus 1.91 for "Token branch only, head kernel off" in Table 6C, both in the tables' scaled units. The paper and submitted implementation indicate that generation does not use the prediction head's direct kernel estimate, so disabling that estimate alone cannot explain the difference. Other differences in readout or training settings between these experiments remain unclear, limiting the assessment of the backbone kernel branch's contribution.

Questions for authors
  1. Why retain all three representation types: observations, transitions, and variate profiles? The paper offers reasonable motivations for their respective roles, but removing individual token types in Table 6C produces only small performance changes. What advantages come from explicitly representing interval-level changes and variate profiles rather than learning these relationships from observation representations? Do the results support complementary roles, or suggest that some of the information is interchangeable? Please clarify the incremental value of the full representation design using comparisons with simplified variants under matched readouts and results from multiple runs.
  2. How do the token and kernel branches complement each other? Both use observed values and timestamps, with the paper assigning them to pattern interaction and local summarization, respectively. What information does each provide in the evaluated tasks? Do the controls show stable gains from combining them over using either branch alone? How does the lack of consistent gains from the two-branch backbone when the prediction head's direct kernel estimate is disabled affect the interpretation of their complementarity? Please distinguish the contributions of the backbone kernel summaries and the head's direct kernel estimate, and clarify what advantages of the combined design are supported by the evidence.
  3. How do the generation ablations in Tables 6A and 6C relate to each other? Both involve removing the backbone kernel branch, and generation itself does not use the head's direct kernel estimate. Please clarify any differences in architecture, readout, or training settings, and explain what conclusions the KL values of 2.46 and 1.91 support, so that the kernel branch's contribution can be assessed under consistent conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.