acceptodds
Under review as a conference paper at ICLR 2027

ATEMPO: Amortized Transfer Entropy Model Per-Lag Over-Time

Abstract

When one time series informs another, the delay at which that information arrives determines if models can utilize this relationship effectively. A source may be strongly predictive at one delay but uninformative at neighboring lags. Therefore, fixed input windows may obscure this structure by bundling informative and irrelevant history together. Transfer entropy (TE) can recover these delays by measuring how much a source's past tells us about a target beyond the target's own history. However, existing TE estimators require separate fits across source and conditioning depths, with further estimation on local windows to track delays over time. We introduce ATEMPO, a diffusion-based transfer entropy estimator amortized over lags and time. ATEMPO samples source and conditioning depths during training and conditions explicitly on temporal position, allowing a single score network to estimate the full TE surface without refitting. Differencing this surface recovers when information arrives. Across synthetic datasets with ground-truth TE, ATEMPO achieves the lowest mean relative error among baselines, and on datasets with known lags, ATEMPO achieves lag-detection accuracy. ATEMPO requires a single fit across source depths and target depths , whereas existing estimators require 18 separate fits to evaluate the same grid. On three systems with time-varying lag, ATEMPO is the only estimator to maintain perfect lag-detection accuracy across all evaluated window lengths. These recovered delays improve downstream forecasting: ATEMPO-selected inputs reduce MSE on WEATHER-5K by 7.25%, while 120 selected inputs outperform the benchmark's 240-feature default by 4.6%.

Primary area:
learning on time series and dynamical systems
Keywords:
information theory, transfer entropy, diffusion, mutual information, time series, lag, forecasting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GbYGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Estimate how much the past X helps to the prediction of Y and find out the delay steps. Previously, for every delay, at each time step, one needs to train a model. Now ATEMPO just trains one diffusion model, and takes time also as input. Once it's train, it can check at any delay at any time.

Critical strengths

The model covers all lags and times. Smoother lag profile since errors at neighbors are correlated.

Critical weaknesses

The baseline does not have time as input, so the comparison might not be fair. On Weather-5K, CCMI seems to be on par or better.

Questions for authors
  1. How does this method perform on gradual lag drift?
  2. Can you get baselines also time as an input?
  3. Does the 7% hold over more seeds?
Reviewer_9moLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ATEMPO to learn an amortized transfer entropy (TE) model conditioned on source and target depths as well as time position. ATEMPO is able to estimate TE on multiple source and target depths using one fit and capture correct time lag even when the lag is time-varying. The proposed methods are evaluated on real and synthetic datasets and shown improved performance over other baselines to estimate TE.

Critical strengths
  • Amortization over history depth and time position largely reduces the number of fits needed to estimate the lag.
  • Performs well in both synthetic and real datasets in lag estimate and time-series forecasting over other baselines on TE estimate, and captures a reasonable time lag on weather5K real data.
  • Ablation study demonstrates the effect of both features introduced by the proposed methods.
Critical weaknesses
  • Methods share large similarities with existing works (I thank the author for the honesty on comparison from existing TENDE method)
  • The real dataset evaluation is mainly conducted on one dataset: Weather5K (Results on MMTT are similar between proposed methods and baselines), on which the comparison is also not extended to typical time-series forecasting methods despite using forecasting performance as key metric.
  • It is unclear which dataset the ablation study is conducted on, and overall it would be an important aspect to report those results on real datasets, and more comprehensively. This is mainly to understand the data sample trade-off of the proposed methods, given that the proposed methods look to underperform at data scarcity scenarios, this would be an important axis to compare with other baselines.
Questions for authors
  1. Aside from taking source / target depths and time position as conditions, are there any main differences between ATEMPO and TENDE? Like model architecture. For example, I am not quite clear on the significance of the masking m(k,l), say when fitted per window, would TENDE achieve the same performance if it is given a dataloader that can sample over different source and target depths? How does TENDE handle different time positions?
  2. Since one focus here is actually the forecasting performance (not only TE estimate), how would ATEMPO compare to forecasting methods that also leverage TE, like TEB?
  3. I don’t quite understand equation (6), so can m_y be {-1, 0, 1} as well? What are the values of K and L, are they hyperparameters, or dataset specifics?
  4. Are the two score functions needed to learn in equation (4) are learned by ATEMPO at the same time (with different inputs) with loss function in equation (7)?
  5. I wonder how good the reported results on Weather5K are, compared to generic time-series forecasting methods. Could ATEMPO be evaluated on other widely used public benchmarks, for example those investigated in PatchTST (just an example)? Alternatively can the authors provide results of generic time-series forecasting on the same tasks of Weather5K?
  6. Could the author clarify on which datasets are used for generating Table 3? It seems that ATEMPO is particularly data-hungering, while methods like NPEET are stable and generally very competitive. While I acknowledge that amortization over many different conditions is likely the root cause, it would be great to know at which data scale ATEMPO starts to be competitive: for example at which samples per window between 500 and 1000 does the metric be crossing.

[TEB] Kalajdzievski et al. Transfer Entropy Bottleneck: Learning Sequence to Sequence Information Transfer. TMLR 2023

[PatchTST] Nie et al. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. ICLR 2023

Reviewer_tiVSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ATEMPO is a transfer entropy estimator that shares one diffusion model across source-history depths, target-history depths, and temporal positions. The estimated information gains are then used to select useful lags for forecasting. I find the motivation practical and the results promising, although some claims need stronger support. I also need clarification on the major benefit of ATEMPO compared to the vanilla attention mechanism.

Critical strengths
  1. The shared estimation approach is interesting. Treating history configurations as model inputs reduces repeated fitting and makes exploring different lags more practical.

  2. The time-conditioning ablation helps explain the method’s effectiveness. The results across different evaluation-window sizes provide useful evidence for handling changing dependencies.

  3. The forecasting experiments demonstrate practical value beyond estimating transfer entropy. Learning lag selections from 300 stations and applying them across WEATHER-5K is a useful evaluation.

Critical weaknesses
  1. The methodological improvement over TENDE seems to be history masking and temporal conditioning. Authors could more clearly distinguish its novelty from existing conditional diffusion estimators.

  2. The synthetic evaluation has some diversity limitations... The authors acknowledge approximately Gaussian systems with only one informative dimension. Performance with multiple interacting sources and more complex distributions remains unclear or they haven't expressed logically.

  3. Interpretation of the delay profile is too strong. because each lag is evaluated after conditioning on more recent source observations, correlated lags can affect attribution. so the largest increment need not identify the underlying physical delay.

  4. The time-varying experiments fit ATEMPO on the complete recording. They support retrospective lag recovery, but do not establish performance on unseen temporal changes. The weather experiment also uses fixed pooled lag selections.

Questions for authors
  1. Can you please provide a clarification on the major benefit of ATEMPO compared to the vanilla attention mechanism?

  2. How robust is the recovered delay profile when source observations are strongly autocorrelated or several lags jointly influence the target?

  3. Can the authors evaluate lag recovery on a held-out continuation where the timing or pattern of changes differs from the training recording?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.