acceptodds
Under review as a conference paper at ICLR 2027

Long-Horizon Prediction from Multi-Modal and Irregular Behavior Data

Abstract

In this paper, we study long-horizon prediction from years of multimodal behavioral data and propose an adaptive multi-scale learning framework for effective cross-modal fusion under irregular temporal observations. Discriminative features are first extracted from modality-specific encoders and fused through scale-wise cross attention. To exploit the complementary information across modalities, we introduce a distillation mechanism based on unimodal teachers that learns shared representations through cross-modal contrastive alignment while retaining modality-specific information through correlation minimization. An error-aware adaptive scale router identifies the best-suited temporal resolution for final prediction. In the internal evaluation of an insurance company, with real data collected from users for critical-illness claim prediction, our method outperforms existing multimodal time-series methods. To validate its generality, we also conduct evaluation in the multi-model temporal prediction from two financial benchmarks and achieve state-of-the-art performance.

Primary area:
learning on time series and dynamical systems
Keywords:
Multimodal learning, Long-horizon prediction, Behavior data, Multi-scale learning, Knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T4nYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studied long-horizon prediction from multimodal behavioral data and proposed an adaptive multi-scale learning framework for cross-modal fusion under irregular temporal observations. Discriminative features were first extracted. Distillation was used to learn shared representations through cross-modal contrastive alignment. An error-aware adaptive scale router identified the best-suited temporal resolution for final prediction. A clinical dataset and two financial datasets were used to compare the proposed method with baselines.

Critical strengths
  1. The article is clearly written.
  2. Although lacking novelty, the methods were well explained, including proper formulation.
Critical weaknesses
  1. The identified problems are long-standing and have been studied by many groups, generating numerous publications.
  2. The proposed solutions lack technical innovation, it more of a combination of well-studied existing methods.
  3. Sections 2.2 and 2.3 are mostly about articles published in 2023 and 2024, lacking newer findings and benchmarks in the field.
  4. The source of the 300K user dataset is unclear.
  5. The results need statistical significance reporting. The value-wise improvements over the baselines seem marginal, such as these for the insurance and S$P 500 datasets.
Questions for authors
  1. The scale selection seems arbitrary. How was the selection made? What were the criteria? Are these the same methods across datasets?
  2. Since the insurance data is confidential, an explanation/comparison of the three datasets' common and unique characteristics should at least be disclosed to interpret the results.
3 comments
AuthorsOpenReview ↗

Response to Reviewer T4nY (Part I)

We thank the reviewer for the constructive feedback and positive assessment of the paper's presentation. We address the reviewer's comments and questions below.

Q1. Temporal Scale Selection

We adopt progressively coarser temporal scales {1, 2, 4}, where 1× preserves fine-grained variations, while 2× and 4× provide intermediate and coarser temporal information. This geometric progression offers a compact set of complementary resolutions. As shown in Fig. 2 of the manuscript, 1× performs best among single-scale settings (ROC-AUC 0.6642), while combining {1, 2, 4} improves ROC-AUC to 0.6651, suggesting that coarser resolutions provide complementary information.

We follow the same scale selection principle and use the same predefined scale factors {1, 2, 4} across the insurance and financial datasets, rather than selecting different scales for each dataset.

Q2. Dataset Characteristics and Comparison

All three datasets combine tabular features with textual and numerical temporal sequences, requiring the integration of heterogeneous information over time. The insurance dataset comprises real-world business records of approximately 300,000 policyholders from an insurance company, including tabular attributes and sparse, irregular behavioral sequences derived from expenditure activities. The task is binary prediction of critical-illness claims.

In contrast, the public S&P 500 and HS300 datasets represent the US and Chinese stock markets, respectively, with more regular trading-day observations and continuous-valued forecasting tasks. Thus, the datasets share the challenge of multimodal temporal modeling but differ in observation regularity, application domains, and prediction objectives.

AuthorsOpenReview ↗

Response to Reviewer T4nY (Part II)

Recent Works

We appreciate the reviewer's comment on the coverage of recent studies and benchmarks in Sections 2.2 and 2.3.

For modality imbalance, IPRM (IJCAI 2025) probes modality strengths and rebalances their learning, while Data Remixing (ICML 2025) reorganizes training samples to alleviate optimization interference. DGL (ICCV 2025) decouples the optimization of modality encoders and fusion modules, and RGM (AAAI 2026) jointly addresses modality imbalance and gradient conflicts. Unlike these optimization-oriented approaches, our DUMT focuses on separating modality-common and modality-specific representations through teacher-guided learning.

For multi-scale temporal modeling, AMD (AAAI 2025) introduces adaptive multi-scale decomposition and predictor fusion, while AdaMixT (IJCAI 2025) dynamically combines multi-scale experts through a gating network. ReIMTS (ICLR 2026) further explores recursive multi-scale representations for irregular time series while preserving their original timestamps. In contrast, our approach integrates multimodal information at different temporal resolutions and uses scale-wise prediction errors to supervise adaptive routing. Recent datasets and benchmarks, including TIME-IMM (NeurIPS 2025) and FinMultiTime (2025), also highlight the importance of multimodal time-series data in real-world applications. FinMultiTime is already used in our financial experiments.

These works provide additional context for positioning our approach within recent developments in multimodal learning and temporal modeling.

AuthorsOpenReview ↗

Response to Reviewer T4nY (Part III)

Multi-seed Results

We additionally conducted experiments with three random seeds under the 104-week input setting on the insurance dataset to evaluate the stability of the results. The mean ROC-AUC and standard deviation are reported below.

MethodROC-AUC ↑
SpecTF0.6426 ± 0.0004
TaTS0.6126 ± 0.0028
Time-MMD0.6533 ± 0.0009
FTS-text-MoE0.6373 ± 0.0003
Pathformer0.6223 ± 0.0078
TimeMixer0.6387 ± 0.0016
Ours0.6647 ± 0.0003

Our method consistently achieves the best performance across the three runs, outperforming the strongest baseline, Time-MMD, by 0.0114 ROC-AUC on average.

Reviewer_q6BJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates long-horizon prediction from multimodal behavioral data characterized by heterogeneous modalities and irregular temporal observations. The authors identify two main challenges: imbalanced learning across modalities and the difficulty of exploiting complementary information at different temporal resolutions. To address these challenges, they propose an adaptive multi-scale learning framework incorporating Decomposed Unimodal Teacher (DUMT), which combines unimodal teacher distillation with shared/private representation learning, and an error-aware adaptive scale router that integrates predictions from different temporal resolutions. The framework is evaluated on a large-scale real-world insurance dataset and two public financial forecasting benchmarks, achieving competitive performance against existing multimodal forecasting methods.

Critical strengths
  1. The paper addresses long-horizon prediction from heterogeneous behavioral records and evaluates the proposed framework on a large-scale insurance dataset involving approximately 300,000 users.

  2. The proposed framework achieves improvements over the reported baselines on insurance risk prediction and two public financial forecasting benchmarks.

Critical weaknesses
  1. The central difference between DUMT and conventional UMT is the decomposition of modality representations into shared and private components. However, the main ablation results show that introducing decomposition alone does not improve over UMT

  2. The shared/private probe results demonstrate predictive utility but do not directly verify the intended information decomposition. Stronger evidence is necessary to establish the contribution of this component beyond existing teacher distillation and representation regularization.

  3. The insurance ablation shows that the full model improves ROC-AUC by only 0.0009 over single-scale modeling and by approximately 0.0006 over ordinary multi-scale modeling. These marginal gains do not convincingly establish the importance of the proposed routing mechanism.

  4. The unimodal probe experiments suggest improved representation quality, but they do not establish that shared/private decomposition specifically alleviates modality imbalance.

  5. Irregular and partially observed behavioral records are central to the motivation of the paper. However, the proposed method mainly aggregates behavioral events into regular temporal intervals and constructs multi-scale representations through temporal average downsampling and positional embeddings.

  6. Important experimental details remain unclear, including the label prediction horizon in the insurance task.

  7. The training objective introduces seven loss-weighting hyperparameters. Such a complex multi-objective training formulation may introduce substantial tuning overhead

Questions for authors
  1. Can the authors compare DUMT against UMT with contrastive learning but without representation decomposition, as well as other established modality-balancing methods under the same architecture?

  2. How do the authors verify that the shared and private representations actually capture modality-common and modality-specific information, respectively?

  3. What makes the proposed error-aware routing strategy fundamentally different from existing adaptive multi-scale or mixture-of-experts methods?

4 comments
AuthorsOpenReview ↗

Response to Reviewer q6BJ (Part I)

We thank the reviewer for the constructive feedback and valuable suggestions regarding modality balancing, representation decomposition, and error-aware routing. We address these concerns below with additional experiments and clarifications.

Q1. Comparison with UMT+CL and Existing Modality-Balancing Methods

We first compare DUMT with UMT using contrastive learning but without representation decomposition (UMT+CL), to assess the benefit of DUMT's shared/private representation decomposition design. Both methods use the same prediction backbone and are evaluated across four historical input lengths (26, 52, 78, and 104 weeks).

ROC-AUC comparison between UMT+CL and DUMT on the insurance dataset.

Method26 weeks52 weeks78 weeks104 weeks
UMT+CL (without decomposition)0.65820.65790.66140.6620
DUMT (Ours)0.66020.66100.66320.6651

As shown in the table above, DUMT consistently achieves higher ROC-AUC than UMT+CL across all four input lengths. This suggests that explicitly separating shared and modality-specific representations provides additional benefits beyond cross-modal contrastive alignment.

In UMT+CL, unimodal teacher distillation preserves knowledge learned independently by each modality, while contrastive learning encourages cross-modal alignment. These objectives may impose competing constraints on the student representations, potentially leading to optimization conflicts. DUMT addresses this potential conflict by decomposing modality representations into shared and private components, applying cross-modal alignment to shared representations while preserving modality-specific information through private representations. A decorrelation constraint further encourages the separation of these components.

To examine whether such conflicts occur in UMT+CL, we calculate the cosine similarity between the gradients of the distillation and contrastive learning losses for each training batch and report the results at selected epochs.

Gradient cosine similarity between distillation and contrastive losses during UMT+CL training.

EpochMean Cosine Similarity
10.1760
5-0.1299
10-0.1999
15-0.2340
20-0.2800
Overall Mean (1–20)-0.1679

The mean cosine similarity is positive during the first two epochs but becomes negative from the third epoch onward, with an average of -0.1969 over epochs 3–20. Despite minor fluctuations, it generally becomes more negative as training progresses. These results provide evidence of local gradient conflicts between the two objectives in UMT+CL, suggesting potential interference between cross-modal alignment and unimodal teacher distillation.

We further compare DUMT with two established modality-balancing methods: OGM-GE, which balances modality optimization through gradient modulation, and PMR, which mitigates modality imbalance through prototype-based learning. Both methods are evaluated under the same prediction backbone and input settings.

ROC-AUC comparison with other modality-balancing methods on the insurance dataset.

Method26 weeks52 weeks78 weeks104 weeks
OGM-GE0.65410.65650.65790.6559
PMR0.65340.65650.65810.6580
DUMT (Ours)0.66020.66100.66320.6651

As shown in the table above, DUMT outperforms both OGM-GE and PMR across all four input lengths, demonstrating favorable performance relative to the evaluated gradient-based and prototype-based modality-balancing methods.

AuthorsOpenReview ↗

Response to Reviewer q6BJ (Part II)

Q2. Verification of Shared/Private Representations

To examine whether the shared and private representations capture modality-common and modality-specific information, respectively, we conduct a modality-level clustering analysis on representations extracted from the test set. Specifically, we form two sets of representations (shared and private), each containing representations from all three modalities (tabular, text, and numerical), and apply K-means clustering (K=3) independently to each set. We compare the resulting cluster assignments with the ground-truth modality labels using three metrics, all of which evaluate the agreement between the two partitions from different perspectives: Adjusted Rand Index (ARI) measures chance-adjusted pairwise agreement, Fowlkes-Mallows Index (FMI) evaluates pairwise precision and recall, and Adjusted Mutual Information (AMI) measures chance-adjusted mutual information.

Clustering results of shared and private representations.

RepresentationARIFMIAMI
Shared0.00420.43040.0228
Private0.89930.93300.8920

The private representations achieve substantially higher scores across all three metrics, indicating strong agreement between their cluster assignments and the ground-truth modality labels. In contrast, the shared representations exhibit much weaker agreement with modality labels, as reflected by their near-zero ARI and AMI and substantially lower FMI. These results are consistent with the intended distinction between shared and modality-specific representations in DUMT.

AuthorsOpenReview ↗

Response to Reviewer q6BJ (Part III)

Q3. Distinction of Error-Aware Routing

We thank the reviewer for this question. Existing methods route different objects in different ways. Pathformer (ICLR 2024) uses input-derived trend and seasonal features to select top-K patch-size pathways, then aggregates their intermediate features within each block. TimeMixer (ICLR 2024) mixes trend and seasonal information across downsampled scales, uses an independent predictor for each scale, and simply sums their predictions without sample-dependent weighting. FTS-Text-MoE routes each time-step token to selected FFN experts for sparse computation; its multi-resolution prediction heads correspond to different forecast horizons rather than different input scales.

Our approach instead routes independently supervised multimodal predictions from different temporal scales. Each scale receives a direct task loss (Lscale\mathcal L_{\mathrm{scale}}), preventing low-weight scales from being under-trained. We then convert the per-scale prediction errors for each sample into a soft target distribution and train the router to match it through Lrouter\mathcal L_{\mathrm{router}}. Thus, the router receives direct supervision about how well each scale predicts the current sample, rather than relying only on input-dependent gating and the final task loss.

The financial results support the effectiveness of this design. With a 144-day input, our method reduces MAE/MSE relative to the stronger of Pathformer and TimeMixer by 14.2%/18.3% on S&P 500 and 13.5%/26.2% on HS300. Notably, these relative gains increase as the input history grows from 72 to 144 days. The HS300 ablation further confirms the contributions of both supervision objectives: removing Lrouter\mathcal L_{\mathrm{router}} increases MAE/MSE by 2.2%/3.8%, while removing Lscale\mathcal L_{\mathrm{scale}} increases them by 2.3%/3.4%.

Thank you for the authors' response. However, there are still some core issues that remain unresolved.

The comparison with UMT+CL shows that DUMT achieves only marginal performance improvements.

The near-zero ARI of shared representations could also result from representation collapse or loss of modality information. Therefore, the clustering results do not sufficiently demonstrate that the proposed decomposition alleviates modality imbalance.

In addition, the author did not address the issues raised in the original opinion, such as the selection of weights for multiple loss items and the cost of parameter adjustment, did not explain whether aggregating irregular data would result in the loss of key events, and did not clearly state the prediction length of the insurance dataset.

Therefore, I maintain weak reject.

Reviewer_XbAYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a time series analysis framework to deal with long horizon prediction with irregular multi model data. Correspondingly, to deal with complex setting, this draft uses a series of neural network modules to handle them including encoder, fusion, router, etc. Experimental results show its superiority over baselines.

Critical strengths
  1. This paper goes with a practical context, time series prediction with long horizon requirement. Other settings as multi-modal and irregular behavior are also practically valuable.
  2. Modules designed to handle these settings are reasonable.
  3. Experimental results show its effectiveness, compared with recently published papers.
Critical weaknesses

This draft is more like a merge of a bunch of existing techniques under a specific setting, which better fits a engineering based project for some real world challenges. Therefore, I think this draft has limited novelty as a research paper. For example, cross attention, teacher model distillation and multi-modal encoder, these stuff are well known ones.

Questions for authors

Please check the section above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer XbAY

We thank the reviewer for recognizing the practical relevance of our problem setting and the effectiveness of our approach.

Regarding the concern that our approach combines existing techniques, our methodological contributions lie in two task-specific learning mechanisms. First, DUMT extends unimodal teacher distillation by explicitly distinguishing cross-modal commonality from modality-specific information. It decomposes the student representations into shared and private components, applies cross-modal contrastive alignment to the shared representations, and introduces a decorrelation constraint to encourage their separation. This provides distinct representation pathways for learning cross-modal commonality and preserving modality-specific information under unimodal teacher supervision, helping mitigate modality imbalance. Second, rather than relying solely on input-dependent scale selection or aggregating scale-specific predictions, our error-aware routing mechanism explicitly uses scale-wise prediction errors to supervise adaptive scale integration, connecting scale weighting to predictive performance.

These designs address the two challenges of modality imbalance and complementary temporal resolutions, respectively. Their individual contributions are further examined through our ablation studies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.