acceptodds
Under review as a conference paper at ICLR 2027

LONG-MEMORY GAUSSIAN PROCESSES FOR STATE- SPACE INFERENCE AND REPRESENTATION LEARNING

Abstract

The Mat´ern covariance kernel is a popular choice for Gaussian process (GP) based modeling of temporal dependencies in time-series data. For half-integer smoothness, the Mat´ern kernel admits a finite-dimensional state-space representation, reducing GP inference from cubic to linear complexity in the time series length. This has made Mat´ern attractive both for direct sequential modeling and for representing structured latent priors in variational autoencoders. However, Mat´ern kernels decay exponentially with lag and therefore cannot represent persistent, polynomially decaying long-range dependencies. To address this, we introduce the GAMMA covariance kernel family, obtained by gamma-mixing of half-integer Mat´ern components. The resulting kernels separate local regularity from long-range memory, allowing near-origin smoothness and asymptotic dependence to be controlled independently. For scalable inference, we approximate the continuous mixture using generalized Gauss–Laguerre quadrature, yielding a finite weighted sum of Markovian components and linear inference complexity in time series length. Building on this construction, we further propose GGPVAE, which places GAMMA priors over latent trajectories. We also theoretically prove the convergence of the proposed approximations. Our experimental results show that GAMMA and GGPVAE capture long-range structure, improve prediction and reconstruction across long missing intervals, and retain scalable inference.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Gaussian processes, Matérn kernels, long-range dependence, state-space models, variational autoencoders, time-series modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eC5rOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduced GAMMA covariance for modeling long-range memory and studied its finite-dimensional Markov approximation based on generalized Gauss–Laguerre approximation. Theoretical results show that this surrogate preserves relevant low-order moments and local covariance structure of the exact GAMMA kernel, providing a principled finite-state approximation.

For experiments on kernel evaluation, the paper compares the exact GAMMA kernel and its approximation GAMMA–GL against matched Mat´ern baselines to assess both predictive performance and the fidelity of the scalable state-space approximation.

For experiments on representation learning, the paper evaluates GAMMA–GL as a temporal prior for variational representation learning by comparing GGPVAE with MGPVAE on long-memory Rotating MNIST and ERA5.

Critical strengths

None

Critical weaknesses

However, state-of-the-art methods for the evaluated tasks should also be compared. The current comparisons are too narrow, which just focuses on GP related methods.

Questions for authors

Could you provide a comparison on computational costs of different methods? That would be interesting to readers.

In addition, it would be much better to provide some visualization in page 2 to show the characteristics of different kernels.

1 comment
AuthorsOpenReview ↗

Expanded Baseline Comparisons, Long-Horizon Forecasting Results, Computational Efficiency Analysis, and Kernel Visualizations

We thank the reviewer for their evaluation and constructive suggestions. In response, we have substantially expanded the experiments across both kernel modeling and representation learning. We introduce broader GP and non-GP baselines, synthetic experiments generated directly from exact GAMMA, approximation-order sensitivity analyses, longer forecasting horizons, and computational-cost comparisons. We also introduce a new Rotating MNIST benchmark with independent Gaussian perturbations to evaluate different temporal priors and representation quality beyond GAMMA-generated data. Additionally, we provide new kernel visualizations illustrating smoothness and long-range dependence. These extensive revisions offer a more comprehensive evaluation of our contributions. We address each concern below.

Compare against state-of-the-art methods for the evaluated task:

To address the concern regarding limited baseline comparisons, we have substantially expanded the real-world experiments to include both GP-based and non-GP forecasting methods, namely the rational-quadratic kernel, ARFIMA, and DLinear, alongside freely parameterized Matérn mixtures and Gamma interval-bin approximations. Using 50 percent of each dataset for training, we evaluate forecasting windows of 25, 100, 250, and 500 time steps. As shown in Table 1 of the revised manuscript, GAMMA–GL achieves the lowest RMSE and NLPD among all evaluated methods on DTWEXBGS and PEP–KO for windows of 100, 250, and 500, while remaining competitive on UCI. At the longest horizon (w = 500), GAMMA–GL achieves RMSE values of 0.0646 and 0.1203 on DTWEXBGS and PEP–KO, compared with 0.0826 and 0.1633 for rational quadratic, 0.0935 and 0.2084 for ARFIMA, and 0.1341 and 0.1461 for freely parameterized Matérn mixtures, respectively. Moreover, GAMMA–GL offers a favorable accuracy–efficiency trade-off, requiring only 5.35 seconds on DTWEXBGS compared with 37.76 seconds for exact GAMMA and 72.02 seconds for the free Matérn mixture. These results demonstrate the practical benefits of GAMMA–GL beyond comparisons with single-component Matérn kernels, particularly for long-horizon forecasting, where its structured multi-scale covariance provides competitive predictive accuracy at substantially lower computational cost than several alternative GP models.

Comparison of the computational costs:

To address the concern regarding computational efficiency, we have added empirical runtime comparisons alongside predictive performance (RMSE and NLPD) across forecasting windows of 25, 100, 250, and 500 time steps (Table 1). The results show that GAMMA–GL achieves the lowest RMSE and NLPD on DTWEXBGS and PEP–KO for all windows bigger than 25, while remaining competitive on UCI. Importantly, GAMMA–GL offers a favorable accuracy–efficiency trade-off compared with alternative GP models. At the longest forecasting window (w = 500), GAMMA–GL requires only 5.35 seconds on DTWEXBGS, compared with 37.76 seconds for exact GAMMA, 72.02 seconds for freely parameterized Matérn mixtures, and 45.06 seconds for rational quadratic. Similarly, on PEP–KO, GAMMA–GL requires 18.89 seconds, compared with 259.16, 202.77, and 193.36 seconds, respectively. These correspond to approximately 7–14-fold reductions in computational time relative to these alternatives, while GAMMA–GL also achieves superior RMSE and NLPD on both datasets. Although simpler methods such as ARFIMA remain faster, these results demonstrate that GAMMA–GL provides a favorable balance between predictive accuracy and computational efficiency, particularly for long-horizon setting.

Kernel visualization :

To address the reviewer's suggestion, we have added a new kernel visualization (Figure 5 in the revised version, Appendix E.1.), comparing GAMMA with Matérn for p = 0 and p = 1, shape parameters α ∈ {0.3, 0.5, 1}, and effective ranges of 200 and 500. All kernels are calibrated to the same effective range for a fair comparison. The figure illustrates how smaller α values produce heavier polynomial tails and slower asymptotic decay, even when correlations decrease faster at shorter lags, while Matérn exhibits exponential decay. It also demonstrates how the smoothness order controls near-origin behavior and the effective range determines the temporal scale, highlighting GAMMA's ability to independently control local smoothness and long-range dependence. Due to space constraints, the figure is currently included in the appendix; we plan to incorporate it into the main text in the final version, if accepted.

We hope that the substantial experimental extensions, comparisons, and new results address the reviewer's concerns and provide stronger evidence of the paper's theoretical and practical contributions. We also hope that these revisions further strengthen the overall quality and significance of the paper. We will upload the revised manuscript before the deadline and sincerely appreciate the reviewer's time, effort, and constructive feedback.

Reviewer_UEeEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GAMMA, a covariance family constructed by mixing half-integer Matérn kernels to capture long-range temporal dependence. The authors develop a Gauss–Laguerre-based finite-dimensional state-space approximation with convergence guarantees and incorporate it into a Gaussian Process Variational Autoencoder (GGPVAE) for sequential representation learning.

Critical strengths
  1. Meaningful covariance construction for long-memory modeling. The proposed GAMMA covariance provides a principled way to model long-range temporal dependence while independently controlling local smoothness. Its construction using half-integer Matern components is particularly appealing because each component admits an exact finite-dimensional state-space representation, enabling efficient Kalman-based inference. This offers a potentially useful approach to combining flexible temporal dependence with computationally tractable Gaussian process modeling.

  2. Theoretical analysis supported by promising empirical results. The paper establishes functional convergence and moment-preservation properties for the proposed Gauss–Laguerre approximation, providing theoretical justification for the finite-dimensional surrogate. The experiments also demonstrate promising improvements in long-range prediction, particularly in the OCO-2 extrapolation task and ERA5 forecasting. These results suggest that the proposed covariance structure can provide practical benefits in applications involving persistent temporal dependencies.

Critical weaknesses
  1. Limited novelty relative to existing long-memory covariance constructions. Although the proposed GAMMA covariance is mathematically well motivated, its fundamental ingredients are closely related to existing approaches, including Gamma-mixed Ornstein–Uhlenbeck processes and Matern scale-mixture covariance models. Furthermore, the finite-dimensional GAMMA–GL approximation is ultimately a structured mixture of Matern kernels and exhibits exponential rather than polynomial covariance decay at sufficiently large lags. Consequently, the distinctive advantages of the proposed construction over existing long-memory covariance models and more general Matérn mixtures remain insufficiently established.

  2. Insufficient experimental comparisons to establish the specific advantages of GAMMA. The experimental evaluation primarily compares GAMMA and GAMMA-GL against single-component Matern kernels. However, these comparisons do not establish whether the improvements arise from the proposed Gamma-mixing structure or simply from using multiple temporal scales. In particular, comparisons against freely parameterized Matern mixtures with matched state dimensions and existing long-memory covariance approximations, such as rational-quadratic-based methods, are missing. Moreover, some synthetic and representation-learning experiments use data generated from GAMMA processes, which may favor the proposed model. Although the real-world results are promising, stronger comparisons are needed to establish the practical significance of the proposed covariance construction.

Questions for authors
  1. Could the authors compare GAMMA-GL with freely parameterized Matern mixtures of the same state dimension and existing long-memory kernel approximations to demonstrate the specific benefits of the proposed structured mixture?

  2. Since GAMMA-GL exhibits exponential decay for finite J, how does the approximation order affect the preservation of long-range dependence and forecasting performance over increasingly long horizons?

1 comment
AuthorsOpenReview ↗

Clarifying GAMMA's Novelty and Theoretical Contributions, with Expanded Comparisons Against Matérn Mixtures and Long-Memory Methods, Approximation-Order Analysis, Longer Forecasting Horizons, and Computational Efficiency Results

We thank the reviewer for their constructive feedback. We address each concern in detail below. Novelty to existing covariance constructions:

Our novelty lies in combining a tractable long-memory covariance with a theoretically justified Markov approximation. Building on Gamma-mixed OU processes (p = 0) and existing Matérn mixtures (Iglói and Terdik, 1999; Barndorff-Nielsen, 2001), Gamma mixing of half-integer Matérn kernels of arbitrary order p yields an explicit closed-form covariance involving finite sums of Gamma-function coefficients and rational powers. Unlike the confluent hypergeometric (CH) covariance of Ma et al. (2023), which requires potentially costly evaluations of Tricomi's function even for half-integer smoothness, our construction avoids special-function evaluation and enables direct, scalable state-space inference. Although Gauss–Laguerre quadrature is established (Solin and Särkkä, 2014), we additionally prove preservation of the first 2J mixing moments for GAMMA–GL. Thus, our contribution combines a closed-form long-memory covariance, efficient Markov approximation, and rigorous theoretical guarantees.

Exponential decay of the approximation:

We acknowledge that GAMMA–GL exhibits exponential rather than polynomial covariance decay at sufficiently large lags for any finite J. However, it aims to approximate exact GAMMA's persistent dependence over finite horizons while enabling efficient state-space inference. We therefore extended real-world forecasting to windows of 25, 100, 250, and 500 time steps. Table 1 shows that GAMMA–GL achieves the lowest RMSE and NLPD on DTWEXBGS and PEP–KO for windows of 100, 250, and 500, while remaining competitive on UCI. It also slightly outperforms exact GAMMA in several settings, suggesting that exact asymptotic covariance behavior is not essential for strong finite-horizon predictions, which also depend on the data and method.

Gamma mixing vs general Matérn mixtures:

To assess if the improvements arise from Gamma mixing rather than simply combining temporal scales, we added freely parameterized Matérn mixtures and Gamma interval-bin approximations (Appendix H.1). In the synthetic experiments, we generate 5,000 observations from the exact GAMMA covariance, using 2,000 test observations and a 500-step forecasting window, across J = 2, 4, 8, 16, and 32. Figure 1 shows that GAMMA–GL approaches exact GAMMA's predictive performance as J increases, while outperforming free Matérn mixtures in accuracy and computational efficiency, particularly at higher J. On real-world data (Table 1), GAMMA–GL achieves the lowest RMSE and NLPD on DTWEXBGS and PEP–KO for forecasting windows of 100, 250, and 500. At w = 500, its RMSE values are 0.0646 and 0.1203, versus 0.1341 and 0.1461 for free Matérn mixtures, respectively, with substantially lower computational times (5.35 vs. 72.02 seconds and 18.89 vs. 202.77 seconds).

Comparisons with other long-memory methods and potential synthetic-data bias:

We have broadened the evaluation to include rational quadratic, ARFIMA, DLinear, and alternative Matérn mixtures. Table 1 shows that GAMMA–GL achieves the lowest RMSE and NLPD on DTWEXBGS and PEP–KO for forecasting windows of 100, 250, and 500 time steps, while remaining competitive on UCI, demonstrating its benefits beyond single-component Matérn comparisons. To address potential GAMMA-specific data-generation bias, we introduce a new Rotating MNIST benchmark where angular velocities are driven by independent Gaussian perturbations instead of GAMMA-correlated noise, with rotation angles obtained by accumulating these velocities. We compare GAMMA–GL and freely parameterized Matérn mixtures for J = 4 and 8. Averaged over four seeds (Figure 8, Appendix H.3.2), GAMMA–GL with J = 8 achieves the highest content accuracy (74.5%) and lowest rotation-invariant RMSE (0.127), while Free Matérn with J = 4 achieves the best pixel-level RMSE (0.227) and NLPD (−0.825). These findings suggest that GAMMA–GL's structured temporal scales favor coherent latent content representations, whereas Free Matérn's flexibility improves pixel-level predictions.

Effect of approximation order J on long-horizon:

To assess the effect of approximation order on long-horizon forecasting, we extended the synthetic experiments to J ∈ {2, 4, 8, 16, 32} for both p = 0 and p = 1, using a 500-step forecasting window. Figure 1 shows that increasing J generally reduces the mean RMSE and NLPD differences between GAMMA–GL and exact GAMMA, while GAMMA–GL remains substantially more computationally efficient than freely parameterized Matérn mixtures, particularly at larger J, and achieves better predictive accuracy than Gamma interval-bin approximations in most settings. Figures 5 and 6 in Appendix H.2.1 further assess covariance approximation and parameter estimation accuracy.

We will upload the revised manuscript before the deadline. We sincerely appreciate the reviewer's time, effort, and constructive feedback.

Reviewer_DkJoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

in this work the authors introduce the gamma mixed matern (gamma) kernel. the authors construct their kernel as an infinite mixture of matern kernels weighted by a Gamma distribution. since this kernel has an infinite dimensional state-space representation and not amenable to efficient gaussian message passing like the kalman filter, they use a finite dimensional approximation using Gauss-Laguerre quadrature. they also use their kernel as a prior in a VAE setting using amortized inference, and demonstrate its efficacy on synthetic data and real data.

Critical strengths

the authors present a finite dimensional approximation of a particular long memory process kernel. the authors showed that their approximation preserves the first J moments which is a nice guarantee.

Critical weaknesses

table 1 generates data from the approximate finite dimensional representation, why not the actual kernel? how then can you claim the approximation is doing well if the data is not generated from it. since this is about long sequences for table 1 it would be good to know about what length sequences how much training data we used etc. the experiments of section 6.2 are also fairly bare bones — ideally there would be more comparisons, and also computational complexity cost, how things vary with different model parameterizations (vary J, or p etc) and none of that is here. in general too, the tasks selected do not really showcase what might be the strengths of using this kernel, even the forecasts the authors show on real data are not “long.”

Questions for authors

in table 1 although the data is generated from an approximation (and not the true kernel). there is a lot of concurrent work being done on long memory gaussian processes, have the authors considered any of those for comparison? did the authors do any analysis on the approximation quality?

2 comments
AuthorsOpenReview ↗

Response to Reviewer DkJo: Addressing Concerns Through Exact GAMMA Data Generation, Approximation Analysis, Expanded Baselines, Long-Horizon Forecasting, and Computational Efficiency

We thank the reviewer for their careful evaluation and constructive feedback and We address each concern in detail below. Synthetic data generation and approximation fidelity:

In the original experiments, synthetic data were generated using a high-order GAMMA–GL approximation with J=124, while the evaluated GAMMA–GL model used J=12 . The higher approximation order was intended to closely approximate the exact GAMMA covariance over the finite observation horizon. Nevertheless, we agree that sampling directly from the exact covariance provides a more rigorous evaluation. We have therefore repeated the experiments using data generated from the exact GAMMA kernel, with longer sequences and a forecasting horizon of 500 time steps ahead. We additionally compare GAMMA–GL against freely parameterized Matérn mixtures and Gamma interval-bin approximations across J ∈ {2, 4, 8, 16, 32}. The new results show that GAMMA–GL generally achieves lower RMSE and NLPD than the alternative Matérn-mixture approximations, while also offering substantially lower computational costs than freely parameterized Matérn mixtures, particularly as J increases. These findings provide further evidence of the favorable accuracy–efficiency trade-off offered by GAMMA–GL. More explanations on and Gamma interval-bin approximations and freely parameterized Matérn mixtures are mentioned in Appendix H.1.

Sequence lengths and experimental settings:

In the original experiments, each synthetic sequence consisted of 1,500 observations, with the remaining experimental details provided in Appendix H.1.1 in the previous version of the paper and Appendix H.2.1 in the revised version. In the revised experiments, we have substantially extended the evaluation. For both Matérn orders p=0 and p=1, we generate sequences of 5,000 observations, divided chronologically into 2,500 training, 500 validation, and 2,000 test observations. The data are generated using the exact GAMMA covariance with parameters sigma =1.0, alpha=0.7, and beta=5.0. We also report empirical computational costs for the different approaches alongside predictive performance, in Figure 1 for the synthetic experiments and Table 1 for the real-world datasets.

Representation-learning experiments and parameter sensitivity:

We extended the representation-learning experiments to GAMMA–GL and freely parameterized Matérn mixtures with J = 4 and 8, introducing a new Rotating MNIST benchmark where angular velocities follow independent Gaussian perturbations instead of GAMMA-correlated noise, and rotation angles are obtained by accumulating these velocities. This removes the direct dependence on the GAMMA covariance. Averaged over four seeds (Figure 8, Appendix H.3.2), GAMMA–GL with J = 8 achieves the highest content digit accuracy (74.5%) and lowest rotation-invariant RMSE (0.127), while Free Matérn with J = 4 achieves the best pixel-level RMSE (0.227) and NLPD (−0.825). These results suggest that GAMMA–GL's structured temporal scales favor coherent latent content representations, whereas Free Matérn's flexibility benefits pixel-level predictions. Increasing J from 4 to 8 modestly improves content preservation but not consistently reconstruction accuracy, highlighting the trade-off between representation quality and predictive performance.

Long-horizon forecasting, comparisons with alternative long-memory methods and assessment of approximation quality:

To address this concern, we expand the real-world forecasting experiments, using 50% of each dataset for training and forecasting windows of w ∈ {25, 100, 250, 500}. We also introduce additional baselines, including rational quadratic, ARFIMA, DLinear, freely parameterized Matérn mixtures, and Gamma interval-bin mixtures, to assess the specific benefits of structured Gamma mixing over alternative long-memory and multi-scale approaches. As shown in Table 1, GAMMA–GL achieves the lowest RMSE and NLPD on DTWEXBGS and PEP–KO for all w ≥ 100, with particularly strong performance at longer horizons. At w = 500, its RMSE values are 0.0646 and 0.1203, compared with 0.0826 and 0.1633 for rational quadratic, 0.0935 and 0.2084 for ARFIMA, and 0.1341 and 0.1461 for free Matérn mixtures, respectively. GAMMA–GL also reduces computational time to 5.35 seconds versus 72.02 seconds for the free Matérn mixture on DTWEXBGS. On UCI, it remains competitive, although rational quadratic and ARFIMA perform slightly better at some longer horizons. These findings highlight GAMMA–GL's predictive accuracy and computational efficiency in long-horizon forecasting with persistent temporal dependencies. Beyond predictive performance, we directly assess GAMMA–GL's approximation quality through covariance approximation error and parameter estimation accuracy, presented in Figures 5 and 6 of Appendix H.2.1 (previously H.1.1).

We will upload the revised manuscript before the deadline. We sincerely appreciate the reviewer's time, effort, and constructive feedback.

id like to thank the authors in responding to my questions, and i maintain my score

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.