acceptodds
Under review as a conference paper at ICLR 2027

CoLaDe: Competitive Latent Compression and Lag-Aware Decoding for Time Series Forecasting

Abstract

Multivariate time series forecasting relies on compact representations that preserve both temporal and inter-variable dependencies. Latent-bottleneck encoder–decoder models achieve this by compressing patch tokens from all variables into a small set of latent tokens and decoding future patches with learnable target queries. However, standard cross-attention does not regulate how latents allocate their attention across the input, which can lead to redundant compression. To address this limitation, this paper proposes Competitive Latent Compression and Lag-Aware Decoding (CoLaDe), which augments latent cross-attention with a signed competition term that reweights attention to tokens according to their relative preference across latents. The competition strength is learned per attention head, which allows latents either to specialize in distinct input regions or to share them. To retrieve historical information, a relative-lag prior explicitly models the temporal distance between each target query and observed patch. A lightweight low-rank correction then adapts this prior for each variable, accommodating heterogeneous temporal dependencies. Extensive evaluations on several real-world benchmark datasets show that CoLaDe achieves superior forecasting performance compared with state-of-the-art baselines.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting, Latent Bottleneck, Competitive Attention, Relative Position Bias

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wqEsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoLaDe, a framework for multivariate time-series forecasting that builds on an existing encoder-decoder architecture and generalized formulation [1] to improve information compression and historical information retrieval. Specifically, CoLaDe improves the encoder via (i) Competitive Latent Compression, which regulates how latent tokens specialize in or share input information, and (ii) Lag-Aware Decoding, which incorporates variable-adaptive relative-lag priors. Experiments on 7 benchmark datasets demonstrate competitive or superior forecasting performance.

Critical strengths
  • The paper takes a meaningful direction toward improving an existing end-to-end forecasting framework [1], rather than focusing solely on architectural modifications.

  • The competitive latent compression mechanism is an interesting design that allows the model to adaptively balance specialization and information sharing across latent tokens.

Critical weaknesses

Overall, the experimental details are critically insufficient to support to paper's main claims.

  • First, the experiments use drop_last=True for the test dataloader. This is a critical evaluation issue that has repeatedly arisen in the time-series forecasting community [2], as it excludes the final incomplete batch from evaluation. All experimental results should be recomputed with drop_last=False to ensure evaluation on the complete test set.

  • Second, the effectiveness of the latent design is not sufficiently validated. As shown in Table 2, removing the latent pathway improves forecasting performance in three of the four prediction horizons, raising concerns about its necessity. Furthermore, competitive latent compression shares similarities with existing mechanisms such as Slot Attention [3]. Comparisons with alternative latent and attention designs are needed to establish the effectiveness of the proposed mechanism.

Questions for authors
  • Given that CoLaDe builds on TimePerceiver [1], why is TimePerceiver not included as a baseline in the experiments?

  • How does the proposed competitive latent compression differ from existing mechanisms such as Slot Attention [3]? Could the authors provide comparisons to clarify the benefits of the learnable signed coefficient?

  • Could the authors compare Lag-Aware Decoding with standard learnable relative position biases and evaluate the contribution of the variable-adaptive correction? I think this would clarify whether the improvements come from the proposed design rather than simply incorporating relative positional information.

[1] Lee et al., "TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting", NeurIPS 2025
[2] Qiu et al., "TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods", PVLDB 2024
[3] Locatello, "Object-Centric Learning with Slot Attention", NeurIPS 2020

Reviewer_vUCMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Existing latent-bottleneck models for multivariate time-series forecasting compress input information into a small set of latent tokens. However, multiple latent tokens may attend to the same input regions, potentially leading to redundant information compression. In addition, when future queries retrieve historical information, these models do not explicitly account for relative temporal distances or variable-specific lag dependencies.

To address these limitations, the paper proposes CoLaDe, which modifies the attention scores within the existing latent-bottleneck architecture rather than introducing additional forecasting branches. During compression, CoLaDe jointly compresses patches from all variables into a shared set of latent tokens. It introduces a competition mechanism that measures how strongly each input token is preferred by a particular latent relative to other latents. A learnable signed coefficient for each attention head adjusts the attention scores: positive values encourage specialization, negative values allow information sharing, and zero recovers standard cross-attention.

During decoding, CoLaDe incorporates relative-lag attention biases between future queries and historical patches by combining a smooth lag function with a lag-specific lookup bias. A low-rank correction further adapts these biases to capture variable-specific temporal dependencies.

Critical strengths
  1. The paper proposes a simple attention mechanism that aims to improve both information compression and historical information retrieval in multivariate time-series forecasting.
  2. The proposed method demonstrates competitive forecasting performance across multiple datasets and prediction horizons. The paper also evaluates computational efficiency and stability across different random seeds.
Critical weaknesses
  1. Insufficient analysis and justification of the identified limitations. The paper identifies two limitations of existing latent-bottleneck architectures: redundant information compression and the lack of explicit relative-lag modeling. However, these limitations are primarily presented as assumptions rather than being supported by concrete empirical evidence. In particular, the paper does not provide actual examples demonstrating that redundant compression occurs in existing models, nor does it sufficiently analyze how these limitations negatively affect forecasting performance. A more rigorous analysis is needed to establish whether these issues are indeed significant and whether addressing them leads to meaningful improvements.
  2. Critical issue in the evaluation implementation. The released code sets drop_last=True in the test dataloader, which causes the final incomplete batch to be excluded from evaluation. As a result, some test samples are not evaluated, raising serious concerns about the validity and reliability of the reported results. Since this issue potentially affects all experimental results, the experiments should be rerun with drop_last=False. However, given the scope of this issue and the possibility of substantial changes to the reported results, simply correcting and rerunning the experiments during the rebuttal period would not be sufficient to increase my evaluation score.
  3. Insufficient ablation studies and direct evidence of the proposed mechanisms. The current ablation studies mainly examine performance changes when individual components are removed. However, such comparisons alone do not establish whether the proposed mechanisms actually achieve their intended objectives. For competitive latent compression, the authors should provide direct evidence that the mechanism reduces redundant information across latent tokens. Similarly, for relative-lag decoding, the authors should analyze how explicitly incorporating relative temporal distances improves the quality of forecasting compared with approaches that do not use such information. More targeted analyses are needed to demonstrate that each component addresses the specific limitations identified in the paper.
Questions for authors

N/A

Reviewer_RETSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoLaDe, a latent-bottleneck framework for multivariate time series forecasting. It introduces competitive latent compression to regulate information allocation across latent tokens through learnable signed competition coefficients. Also, lag-aware decoding incorporates relative temporal distances and variable-adaptive low-rank corrections to capture heterogeneous temporal dependencies. Experiments on seven real-world datasets demonstrate competitive forecasting performance against existing methods.

Critical strengths
  1. The combination of competitive latent compression and lag-aware decoding is an interesting design, allowing the model to regulate information aggregation across variables while explicitly considering temporal dependencies during forecasting.

  2. The signed competition mechanism is an interesting design, allowing different attention heads to adaptively encourage latent specialization or information sharing through learnable competition coefficients.

  3. The experiments are relatively comprehensive, covering 7 datasets and multiple forecasting horizons, with component ablations and computational efficiency comparisons.

Critical weaknesses
  1. The benefit of the latent pathway is not consistently demonstrated. As shown in Table 2, removing the entire latent pathway improves both MSE and MAE on three of the four prediction horizons. This raises questions about when latent compression is beneficial for forecasting. Further analysis would help clarify its contribution.

  2. The learned competition behavior is not sufficiently analyzed. Although the competition coefficients are learned per attention head, the paper does not report their learned values. Visualizing the distribution and signs of these coefficients would help clarify whether different attention heads favor latent specialization or information sharing.

  3. The sensitivity to key architectural hyperparameters is not evaluated. In particular, the number of latent tokens and the rank of the variable-adaptive correction may affect the compression capacity and the modeling of heterogeneous temporal dependencies, respectively. A sensitivity analysis would help assess the robustness of these design choices.

Questions for authors
  1. The benefit of the latent pathway is not consistently demonstrated. As shown in Table 2, removing the entire latent pathway improves both MSE and MAE on three of the four prediction horizons. This raises questions about when latent compression is beneficial for forecasting. Further analysis would help clarify its contribution.

  2. The learned competition behavior is not sufficiently analyzed. Although the competition coefficients are learned per attention head, the paper does not report their learned values. Visualizing the distribution and signs of these coefficients would help clarify whether different attention heads favor latent specialization or information sharing.

  3. The sensitivity to key architectural hyperparameters is not evaluated. In particular, the number of latent tokens and the rank of the variable-adaptive correction may affect the compression capacity and the modeling of heterogeneous temporal dependencies, respectively. A sensitivity analysis would help assess the robustness of these design choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.