acceptodds
Withdrawn from ICLR 2027

A Hybrid Attention Model Learning Unified Time-aware Patch Representation for Irregular Multivariate Time Series Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Time series foundation models (TSFMs) have recently delivered impressive zero-shot performance across diverse forecasting tasks. However, real-world decision-making frequently relies on irregular multivariate time series (IMTS), where inconsistent inter-observation intervals and asynchronous sampling across variables coexist with informative missingness. In this paper, we propose a hybrid attention model that learns a unified time-aware patch representation for IMTS forecasting. We first design a time-aware patch encoder that maps a variable number of intra-patch timestamps into a fixed-size embedding, producing a uniform format for irregular patches without resorting to imputation. We then introduce a time bias attention mechanism that calibrates inter-patch temporal misalignment and asynchronous cross-channel dependencies as an auxiliary attention offset. Finally, on top of a decoder-only Transformer backbone, we adopt a hybrid causal mask that preserves a bidirectional full view over the historical context while keeping the forecast horizon strictly autoregressive. To support large-scale pretraining under irregular settings, we also curate VersaTSA, an expanded archive of over B observations that retains the native sampling sparsity of its sources. Experiments on three IMTS benchmarks and a standard regular-MTS benchmark show that our model achieves state-of-the-art zero-shot performance on IMTS and remains competitive when transferred to regular forecasting.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Foundation model, Irregular Multivariate Time Series Forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GBntOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniTIMS, a pretrained Transformer for zero-shot irregular multivariate time series forecasting. The model combines time-aware patch representations, an additive temporal attention bias, and a hybrid attention mask that allows bidirectional attention over the observed context while retaining causal attention over the forecast sequence. The authors also introduce VersaTSA for pretraining and evaluate the model on irregular forecasting benchmarks, regular multivariate forecasting datasets, and fev-bench. The reported results suggest promising transfer to irregular forecasting, although inconsistencies in the main comparison table and the method description limit the conclusions that can currently be drawn.

Critical strengths
  1. The paper addresses a practically relevant limitation of time series foundation models: transferring to observations with irregular intervals and asynchronous variables. Representing actual timestamps at the patch level and using their relationships to bias attention is a reasonable design for this setting. The combination with multivariate context provides a potentially useful approach to zero-shot irregular forecasting.

  2. The evaluation covers both irregular and regular forecasting, multiple model sizes, and several forecast horizons. In particular, the UniTIMS averages in Table 2 agree with the detailed results in Table 9 at the reported precision, and the reported ECL and ETTm2 results are competitive. Section 4.4.2 also explicitly specifies the exclusion of the three irregular and six regular evaluation datasets from pretraining, which is an appropriate precaution for assessing zero-shot transfer.

Critical weaknesses
  1. Several baseline results in Table 1 are assigned to the wrong datasets, affecting the central performance comparison. In Table 1 of the cited Hi-Patch paper (Luo et al., 2025), Hi-Patch's MSE is 2.57 +/- 0.02 on Human Activity and 4.86 +/- 0.03 on PhysioNet, using the displayed 10^-3 scale. The present submission reverses these assignments. The same issue occurs for several other methods, including iTransformer and PatchTST, whereas GRU-D is assigned correctly; a global exchange of the two column headers would therefore not resolve the table. If UniTIMS's own dataset labels are correct, its Base model's Human Activity MSE of 3.41 is worse than Hi-Patch's 2.57, while the Large model's 2.44 remains better but by a much smaller margin than the current table suggests. These errors do not establish that UniTIMS is ineffective, but they prevent the current table from supporting the stated comparison with specialized full-shot models. A corrected table and confirmation of the evaluation configurations are necessary to assess which conclusions remain supported.

  2. The description and validation of the core temporal representation are inconsistent. Section 4.1.2, Equation (3), defines patch-level temporal aggregation through masked attention pooling with a shared learnable query. However, Section 5.4 describes the full model as concatenating timestamp embeddings and applying a linear projection, with averaging used as the ablated alternative. These are different architectures, making it unclear which method produced the reported results and what the ablation establishes. Separately, the comparison in Section 5.5 changes both the attention mask and the available context: the causal setting uses channel-independent inputs, whereas the hybrid setting uses concatenated multivariate context. Consequently, this comparison does not isolate the contribution of the hybrid mask. Together, these issues weaken the evidence connecting the proposed mechanisms to the reported improvements.

Questions for authors
  1. Could you provide a corrected Table 1, confirm the dataset assignment of every result, and identify which rankings and conclusions change? Please also confirm whether the dataset splits, observation/forecast windows, preprocessing, and metric aggregation used for UniTIMS and the other foundation models match the cited full-shot baseline protocol, and identify any differences.

  2. Which temporal aggregation was actually used for the main experiments: Equation (3)'s attention pooling or the concatenation-and-projection operation described in Section 5.4? Were the ablations evaluated against that same implementation? For the Section 5.5 mask comparison, are results available with identical multivariate inputs and training settings, changing only the mask?

  3. Could you provide the expanded VersaTSA source manifest and observation-count definition underlying the 231B figure? The LOTSA rows listed in Table 3 sum to approximately 27.65B observations, so the additional expansion is not accounted for by that table. Please also identify the fev-bench version and clarify its exclusion protocol: sources such as M5, Favorita, and KDD Cup 2022 appear in both the pretraining inventory and fev-bench. Section 4.4.2 explicitly addresses the nine other evaluation datasets, but it is unclear how the fev-bench tasks were excluded. A source-, series-, and time-range-level account would help distinguish legitimate disjoint subsets from evaluation overlap; shared source names alone do not establish leakage.

Reviewer_XABPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniTIMS extends patch-based foundation forecasting to irregular multivariate observations using continuous-time patch embeddings, time-aware attention and a hybrid historical/target mask. It is pretrained on VersaTSA and evaluated on three irregular datasets, six regular datasets and fev-bench.

Critical strengths
  1. The problem is important, and the design addresses the right information: observation times, variable identity and multivariate historical context. A shared forecasting interface for regular and asynchronous data would be useful.

  2. The reported irregular-data results are encouraging. Base and Large take the best two positions across Table 1, with additional history/horizon settings in Tables 10–11. This supports further investigation, although larger models are not uniformly better and Figure 2 does not show leading general-purpose forecasting performance.

Critical weaknesses
  1. I cannot reconcile the stated method with the released experiment code. Equation (3) uses masked attention pooling, while Section 5.4 and the supplied time encoder use concatenation and projection; padded timestamps are not excluded from that encoder. The active attention path also lacks the stated learned temporal scaling parameter. In the released default RevIN training path, query values are overwritten before the loss reads its labels from the same batch. This is a source-level defect, not proof that the reported checkpoints used that path, but it makes identifying the actual experiment version essential. The HCM evidence has a separate control problem: Section 5.5 changes both the mask and access to multivariate context, so it does not isolate historical bidirectionality.

  2. The pretraining and evaluation records do not establish the scope of the data and zero-shot claims. Tables 3–5 list about 27.745B observations, whereas the main text claims over 231B and the released archive document describes 29.22B. These could be different snapshots, but their links to the evaluated checkpoints are missing. The stated benchmark exclusions are welcome; a post-filter source inventory is still needed, including clarification of the Weather entry and fev-bench overlap. The regular-grid TSFM baselines also need an explicit mapping from irregular observations to their inputs and from their outputs to query timestamps. Without these records, the reported advantage is difficult to assess.

My overall recommendation is clear rejection in the present version. The individual discrepancies may be fixable, which is the basis for my soundness subscore, but their combined effect prevents me from connecting the stated method and zero-shot claims to the evaluated implementation and checkpoints. The encoder mismatch, the source-level training-label issue, and the missing links between corpus snapshots and checkpoints affect the evidence for the central contribution rather than presentation alone. This does not establish that the reported checkpoints used the defective code path or that the results are invalid. Identifying the actual experiment version, showing how its labels were preserved, and documenting its data and baseline interfaces could materially change my assessment.

Questions for authors
  1. Which source/configuration produced the results and ablations? Please reconcile the time encoder, padding and temporal scaling, show how training labels were preserved, and compare HCM with causal attention while holding multivariate context fixed.

  2. Which corpus snapshot trained each checkpoint, how do its counts and exclusions relate to Tables 3–5, and how were regular-grid baseline predictions aligned to the irregular query set? Please provide the adapter and source-exclusion specifications.

Reviewer_uQbfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniTIMS, a time-series foundation model for irregular multivariate forecasting. It combines time-aware patch encoding, time-bias attention, and a hybrid causal mask to model irregular temporal structures and cross-variable dependencies. The model is pretrained on the large-scale VersaTSA archive. Experiments demonstrate promising zero-shot forecasting performance on irregular benchmarks and competitive results on regular time series.

Critical strengths
  1. The paper addresses an important practical problem by developing a foundation model that handles irregular sampling and asynchronous multivariate observations without relying on imputation.

  2. The integration of time-aware patch representations, temporal attention bias, and hybrid causal masking provides an interesting approach to modeling irregular temporal structures within a patch-based Transformer.

  3. The experimental results demonstrate promising zero-shot forecasting performance across multiple irregular time-series benchmarks, outperforming several specialized forecasting methods and pretrained foundation models.

Critical weaknesses
  1. The methodological novelty could be further highlighted. The proposed framework combines several established techniques, including Time2Vec-style temporal encoding, attention-based pooling, and hybrid causal masking. While their integration for irregular multivariate forecasting is meaningful, the authors could better emphasize the distinctive technical innovations of the proposed framework.

  2. The intra-patch aggregation ablation requires clarification. Section 4.1.2 (Eq. 3) defines attention-based pooling for aggregating timestamp embeddings, whereas Section 5.4 describes the w/o Patch-Aware variant as replacing concatenation followed by linear projection with average pooling. The authors should clarify the actual aggregation mechanism used in the full model and the corresponding ablation setting.

  3. The robustness to different degrees of temporal irregularity is not sufficiently evaluated. Although the paper evaluates three IMTS benchmarks, their sampling irregularity levels are not quantitatively characterized. Also, the experiments do not systematically examine how forecasting performance varies with sampling sparsity or irregularity. Such an analysis would better establish the robustness of the proposed temporal representations.

Questions for authors
  1. The methodological novelty could be further highlighted. The proposed framework combines several established techniques, including Time2Vec-style temporal encoding, attention-based pooling, and hybrid causal masking. While their integration for irregular multivariate forecasting is meaningful, the authors could better emphasize the distinctive technical innovations of the proposed framework.

  2. The intra-patch aggregation ablation requires clarification. Section 4.1.2 (Eq. 3) defines attention-based pooling for aggregating timestamp embeddings, whereas Section 5.4 describes the w/o Patch-Aware variant as replacing concatenation followed by linear projection with average pooling. The authors should clarify the actual aggregation mechanism used in the full model and the corresponding ablation setting.

  3. The robustness to different degrees of temporal irregularity is not sufficiently evaluated. Although the paper evaluates three IMTS benchmarks, their sampling irregularity levels are not quantitatively characterized. Also, the experiments do not systematically examine how forecasting performance varies with sampling sparsity or irregularity. Such an analysis would better establish the robustness of the proposed temporal representations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.