acceptodds
Under review as a conference paper at ICLR 2027

Time-Frequency Memory-Guided Diffusion for Generative Time Series Imputation

Abstract

Score-based diffusion models have shown strong performance in probabilistic time series imputation. However, their performance can degrade under extreme sparsity, where limited observations provide insufficient context to reliably guide the reverse diffusion process. Consequently, while the generated imputations may exhibit plausible local patterns, they often fail to adequately preserve the broader temporal structures aligned with the observed context. To address this issue, we propose Time-Frequency Memory-Guided Diffusion (TFMemDiff), a unified framework that augments conditional diffusion with complementary memory priors in both the temporal and frequency domains. Rather than treating memory vectors as auxiliary features to be simply concatenated with diffusion inputs, TFMemDiff uses memory as an external structural reference that guides conditional score estimation throughout the reverse diffusion process. Specifically, TFMemDiff incorporates a Frequency Memory Module (FMM) to retrieve recurring global spectral patterns and a Temporal Memory Module (TMM) to capture localized temporal dependencies. These memories distill reusable structural knowledge from historical training sequences and make it accessible to each incomplete instance during denoising. Extensive experiments on multiple real-world datasets demonstrate the effectiveness of TFMemDiff across diverse missingness settings, showing that memory-guided structural priors can substantially improve generative imputation under severe sparsity.

Primary area:
generative models
Keywords:
Time Series Imputation, Diffusion Models, Memory Mechanism

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_hHuNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TFMemDiff for generative time series imputation under severe missingness. The method introduces temporal and frequency memory modules to retrieve structural prototypes from historical data and uses them to guide conditional diffusion. The authors evaluate the approach on several real-world datasets under different missing patterns, with additional ablation studies and experiments on an alternative diffusion backbone.

Critical strengths

The idea of using historical time-frequency memories to guide diffusion under sparse observations is reasonable and interesting. It provides additional information when the observed sequence itself is insufficient for reliable imputation. The experimental results are generally strong across different datasets and missing patterns. The ablation studies and the evaluation with SSSD also provide useful evidence for the effectiveness of the memory modules.

Critical weaknesses
  1. My main concern is the experimental comparison. Table 2 reports very large improvements over several baselines, particularly on the Stock dataset, where the errors of some methods differ by more than an order of magnitude. Although the results are encouraging, such large differences need further explanation. The appendix mentions that the baselines are implemented using different sources, but it is not entirely clear whether data preprocessing, normalization, masking, and evaluation protocols are fully consistent across methods. A more detailed explanation would make the results more convincing.

  2. The novelty and the evidence for the proposed structural guidance could be further strengthened. Related ideas, such as prototype-guided diffusion and frequency-conditioned imputation, have already been explored in recent work, including ProDiff and LSCD (ICML 2025). I think the contribution of this paper lies more specifically in the joint retrieval and integration of temporal and frequency memories into conditional diffusion. In this regard, Table 1 is not particularly informative, as the selected categories do not provide a very direct comparison with existing methods. Moreover, the paper repeatedly emphasizes that FMM preserves global spectral patterns while TMM captures local temporal dependencies. However, the main evaluations rely on RMSE, MAE, and CRPS, which do not directly demonstrate how well the frequency structure is preserved. More direct evidence would help support this claim.

Questions for authors
  1. Could the authors clarify whether the baseline results in Table 2 were reproduced under a unified experimental protocol or taken from existing implementations and reported results? In particular, how were normalization, masking, and evaluation settings kept consistent? What explains the unusually large performance gaps on Stock?

  2. Could the authors further clarify the methodological difference between TFMemDiff and existing prototype-guided or frequency-conditioned diffusion methods, particularly ProDiff and LSCD? What is the main benefit of combining the two memory domains beyond using either one separately?

  3. Is there any direct evidence that FMM improves the preservation of global frequency structures, such as spectral reconstruction errors or dominant-frequency consistency? If not, could the authors clarify the extent to which the current experiments support this claim?

2 comments

We sincerely thank you for your constructive comments and valuable suggestions. We appreciate your time and effort in reviewing our work. Below, we address your comments point by point.

Q1: Clarification of Baseline Reproduction, Experimental Consistency, and Stock Performance.

A1: We sincerely thank the reviewer for raising this important concern.

Regarding baseline reproduction, as described in Appendix F.3, most baselines are implemented using the unified TSLib framework, while the remaining methods are adopted from their official repositories. The results in Table 2 are obtained from our experimental implementations rather than directly copied from published results.

Regarding experimental consistency, all methods are evaluated on the same dataset splits and under the masking configurations described in Appendix F.2. We also use the same evaluation metrics, computed exclusively on masked entries, as specified in Appendix F.4. We will further clarify the preprocessing and normalization procedures to make the experimental protocol more transparent.

Regarding the performance on Stock, we believe it may be related to the dataset characteristics and the ability of our model to exploit historical patterns. Stock is a relatively low-dimensional daily multivariate dataset, whose six financial variables may contain reusable cross-variable patterns. By retrieving representative historical prototypes, TFMemDiff provides additional structural references for conditional denoising, which may be particularly beneficial under severe block missingness. We will clarify this explanation in the revised manuscript.

Q2: Methodological Differences from ProDiff and LSCD and Advantages of Dual-Domain Memory.

A2: We sincerely thank the reviewer for this insightful question.

As outlined in the contributions of the original manuscript, our main contribution is to identify the limitation of instance-only conditioning under extreme observation sparsity and introduce historical structural prototypes as complementary guidance for conditional diffusion. While ProDiff uses historical human movement prototypes and LSCD incorporates frequency information from current incomplete observations, TFMemDiff integrates historical prototypes from both temporal and frequency domains into conditional score estimation, thereby supplementing sparse observations with complementary structural guidance. The main differences are summarized below:

CategoriesProDiffLSCDOurs
Historical prototype√×√
Frequency-domain information×√√
Joint dual-domain retrieval××√

We will revise Table 1 to include ProDiff and LSCD and provide a more direct comparison of the methodological differences.

Regarding the benefit of combining the two memory domains, TMM provides adaptive temporal references, while FMM captures complementary spectral information. Their joint retrieval enables conditional denoising to exploit both temporal dependencies and frequency-domain characteristics. This complementarity is supported by the ablation results in Table 3: removing either memory degrades performance, while removing both leads to the greatest degradation. We will further clarify the complementary roles of the two memories and the supporting ablation results in the revised manuscript.

Q3: Evidence for FMM's Global Frequency Structure Preservation.

A3: We sincerely thank the reviewer for raising this important question. To provide direct evidence, we introduce Power Spectrum Discrepancy (PSD-Dist) and Autocorrelation Discrepancy (ACF-Dist) to evaluate spectral and temporal structural consistency, respectively. We compare TFMemDiff with its variant without FMM across different missing patterns on ETTh1 and Stock(reported as PSD-Dist / ACF-Dist):

MethodETTh1(OT) POINTETTh1(OT) SEQStock POINTStock SEQStock BLOCK
w/o FMM0.270/0.0970.408/0.1780.165/0.0480.042/0.0130.347/0.120
TFMemDiff0.265/0.0950.399/0.1610.160/0.0460.038/0.0110.325/0.116

As shown in the table, TFMemDiff consistently achieves lower PSD-Dist and ACF-Dist across all evaluated settings. This improvement can be attributed to FMM retrieving representative global spectral patterns from historical training representations in the frequency domain, thereby providing additional spectral references that help preserve global frequency structures and temporal correlation consistency. We will incorporate these additional evaluations and clarify the corresponding claims in the revised manuscript.

AuthorsOpenReview ↗

Clarification on the Updated Overall Rating

Dear Reviewer hHuN,

Thank you for your time and consideration. We noticed that the overall rating changed from 3 to 2, while the written comments appear unchanged. As the discussion period is nearing its end, we wanted to briefly check whether this update was intentional or perhaps an inadvertent change. If intentional, we would appreciate a brief note on any remaining concerns.

Thank you again for your feedback.

Reviewer_wto2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TFMemDiff, a conditional diffusion framework for generative time-series imputation that augments the denoising process with dual-domain external memory. A time-domain memory bank captures recurrent local temporal patterns and a frequency-domain bank captures global spectral prototypes; both are retrieved per instance and injected as structural priors into the score network at every reverse step. The paper introduces initialization via clustering, sparse Top-K memory updates with gating, and an entropy regularizer for sparse retrieval, reporting strong gains under extreme sparsity across multiple datasets and masking regimes.

Critical strengths
  • The idea of dual-domain (time and frequency) external memory to guide conditional diffusion is conceptually appealing and addresses a real weakness of instance-only conditioning under extreme sparsity.
  • The design includes several practical mechanisms—prototype initialization via clustering, sparse Top-K writes with gating, and entropy regularization—to stabilize memory learning and avoid collapse.
Critical weaknesses
  • The approach is close in spirit to recent prototype/global-conditioning methods for diffusion imputation, as well as template- or prior-guided diffusion
  • The paper lacks theoretical guarantees for when and why dual-domain memory will improve conditional denoising
  • TFMemDiff uses 20-step DPM-Solver++ sampling, but it is not stated whether baselines are similarly accelerated
Questions for authors

See Weaknesses

3 comments

Thank you for your support and the time you have devoted to reviewing our work. We truly appreciate your feedback. Below, we provide a point-by-point response.

Q1: The approach is close in spirit to recent prototype/global-conditioning methods for diffusion imputation, as well as template- or prior-guided diffusion.

A1: We sincerely thank the reviewer for pointing out these related research directions. We acknowledge the conceptual connections between TFMemDiff and existing prototype-based or prior-guided diffusion methods. However, TFMemDiff differs in how historical structural information is represented and incorporated into diffusion imputation. Specifically, it constructs two separate memory banks from temporal representations and their frequency-transformed counterparts, respectively, and retrieves complementary prototypes from both spaces to condition the diffusion denoising process. To the best of our knowledge, TFMemDiff is the first to integrate time-frequency dual-domain memory into diffusion-based time series imputation, using jointly retrieved temporal and spectral prototypes as structural priors. Rather than relying on a single prototype space or a predefined external template, TFMemDiff learns reusable structural representations from historical training sequences and makes them accessible to incomplete instances through memory retrieval. We will strengthen the related-work discussion and clarify these distinctions in the revised manuscript.

Q2: The paper lacks theoretical guarantees for when and why dual-domain memory will improve conditional denoising.

A2: We sincerely thank the reviewer for raising this important question. The rationale for dual-domain memory can be understood from a conditional estimation perspective. The observed values provide instance-specific evidence, while the retrieved prototypes offer additional structural references learned from historical training data. When historical patterns are representative of the test data and the two domains provide complementary information about missing values, these references can help reduce conditional uncertainty and improve denoising. However, a formal theoretical guarantee is currently lacking because the learned prototypes, similarity-based retrieval, and conditional denoising are jointly data-dependent and nonlinear, making their contributions to denoising performance difficult to characterize analytically. We will clarify this conditional estimation rationale in the revised manuscript and explicitly acknowledge the absence of formal guarantees as a potential limitation.

Q3: TFMemDiff uses 20-step DPM-Solver++ sampling, but it is not stated whether baselines are similarly accelerated.

A3: We sincerely thank the reviewer for raising this point. As clarified in Appendix H of the original manuscript, all baseline methods are evaluated using their canonical default inference configurations rather than being forced to adopt the same accelerated sampler. This preserves their intended inference settings and avoids arbitrary solver modifications. We also clarify that TFMemDiff's efficiency advantages are not solely attributable to DPM-Solver++, as its low FLOPs and training runtime are independent of the inference solver. We will further clarify these sampling configurations and their implications for computational efficiency in the revised manuscript.

Thank you for your response. I maintain my score as "Weak acceptance".

Thank you very much for your continued support. Your comments and questions have been very helpful to our work. We sincerely appreciate the time and effort you have devoted to reviewing our manuscript.

Reviewer_RpRHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies probabilistic time-series imputation when observations provide insufficient context for conditional generation. TFMemDiff introduces temporal and frequency memory banks that store prototypes learned from training sequences. Instance-dependent retrieval supplies these prototypes as conditions to a diffusion model, while sparse selection and gated updates maintain the memory. Experiments cover eight datasets, eleven baselines, point/subsequence/block masking, component ablations, CRPS evaluation, and integration with an alternative diffusion backbone. The proposed contribution is an explicit mechanism for reusing training-set structure when the current incomplete sequence offers weak structural cues.

Critical strengths
  1. A useful perspective on conditioning under incomplete observations. The paper addresses a limitation that stronger processing of the current instance may not fully overcome: the relevant structure can be poorly represented in the remaining observations. Its contribution is to make recurring training-set patterns explicitly retrievable during imputation. Relative to frequency-aware methods such as FGTI [1], the methodological distinction is how structural knowledge is stored and accessed, rather than the use of frequency information alone. This is a meaningful architectural extension of conditional diffusion. [1] Yang, X., Sun, Y., Yuan, X., and Chen, X. Frequency-aware Generative Models for Multivariate Time Series Imputation. NeurIPS, 2024.
  2. The method design follows the proposed motivation. Organizing prototypes in temporal and frequency representations provides a concrete way to offer different structural references to the denoiser. Importantly, retrieved prototypes condition generation rather than directly replace missing values, allowing reconstruction to depend jointly on observed context and learned patterns. The memory mechanism can be integrated with an alternative diffusion backbone, which supports its value as a reusable modeling component. The exact division of labor between the two memories remains an interpretation, but the overall design is coherent.
  3. The experiments provide convincing support for the method's practical value. The evaluation includes multiple datasets and missingness patterns, component ablations, probabilistic prediction metrics, and an alternative-backbone experiment. For example, on Electricity under SEQ 30, Table 2 reports RMSE of 0.426, compared with 0.469 for FSDI, the strongest baseline in that setting. The broader evaluation supports the architecture across several structured-missingness settings.
Critical weaknesses

W1. The claimed division between the memories remains an interpretation. In Equation 8, temporal queries are Transformer representations, not explicitly local features; applying rFFT and a learned projection does not guarantee prototypes encoding stable periodicity in the original signal. The FFT axis and its correspondence to temporal structure therefore require explanation. Figure 2's stable attention could reflect prototype preference rather than periodic semantics. Table 3 supports both branches' utility, but not their specific global-periodic/local-temporal roles. The paper should distinguish plausible interpretations from established mechanisms and identify the representation properties supporting these roles. W2. Sparse interaction relies on unresolved assumptions. Ambiguous observations generate queries, yet Equation 12 favors concentrated retrieval, reused throughout sampling. Why should selecting few prototypes resolve ambiguity rather than impose inappropriate bias? The positive CRPS results support predictive quality, but do not resolve this question about retrieval bias. Separately, Top-K updates and low per-query entropy allow all queries to select identical slots. K-means establishes initial diversity, but its preservation during training is unexplained. The ablation supports accuracy gains, not the anti-collapse mechanism. These assumptions need justification or narrower claims, rather than more benchmarks.

Questions for authors
  1. The explicit retrieval of dual-domain prototypes is a useful architectural contribution. Could the authors clarify what additional modeling distinction is intended by describing these conditions as external structural priors? Relating this distinction to the conditioning equations and incomplete observations would help clarify the conceptual claim.
  2. Along which axis is the rFFT applied, and why should the resulting learned prototypes encode global periodicity while the Transformer-based temporal prototypes encode local dynamics? Which parts of this interpretation are supported, and which remain hypotheses?
  3. How is concentrated retrieval justified when the observed context is ambiguous? Separately, what supports the claim that Top-K updates mitigate memory collapse, given that neither sparse selection nor low per-query entropy establishes diversity across queries?
1 comment

We sincerely appreciate your support and thoughtful evaluation of our work. Our responses are provided below.

Q1: Clarification of the Role of Dual-Domain Memory as External Structural Priors.

A1: We sincerely appreciate the reviewer's insightful feedback. We clarify that the term "external structural priors" emphasizes the distinct information source and modeling role of the retrieved prototypes. Specifically, the observed values x0obx_0^{ob} provide instance-specific information from the current incomplete sequence, whereas the retrieved prototypes q~={q~t,q~f}\tilde{q}=\lbrace\tilde{q}_t,\tilde{q}_f\rbrace encode reusable temporal and spectral patterns learned from historical training sequences. As shown in Eqs. (3) and (4), these two sources jointly condition the noise predictor during reverse diffusion. Thus, the memory prototypes do not replace observed values or directly estimate missing entries; instead, they provide complementary structural references to guide conditional score estimation, particularly when the available observations are insufficient under severe missingness. We will further clarify the meaning and role of external structural priors in the Introduction of the revised manuscript.

Q2: Clarification of the FFT Operation and Functional Interpretation of Dual-Domain Prototypes.

A2: We sincerely thank the reviewer for the thoughtful and constructive feedback. Regarding the functional distinction, the rFFT is applied along the temporal dimension (dim=1) of the Transformer latent representations, rather than the feature dimension or the original input signal. Thus, TMM retains the original temporal representations, while FMM operates on their frequency-transformed representations. We will clarify the rFFT operation axis in Section 3.2 of the revised manuscript.

Regarding the roles of TMM and FMM, TMM retrieves temporally aligned features that can reflect temporal dynamics, while FMM retrieves spectral features that may capture global periodic patterns. To further evaluate the effect of FMM on preserving spectral characteristics and temporal correlations, we compare TFMemDiff with w/o FMM on ETTh1(OT) and Stock under different missing patterns (reported as PSD-Dist / ACF-Dist):

MethodETTh1(OT) POINTETTh1(OT) SEQStock POINTStock SEQStock BLOCK
w/o FMM0.270/0.0970.408/0.1780.165/0.0480.042/0.0130.347/0.120
TFMemDiff0.265/0.0950.399/0.1610.160/0.0460.038/0.0110.325/0.116

As shown in the table, TFMemDiff consistently achieves lower PSD-Dist and ACF-Dist, supporting FMM's contribution to preserving spectral characteristics and temporal correlations. Together with the ablation results, these findings support the complementary roles of TMM and FMM. However, the specific interpretation that TMM captures local dynamics and FMM encodes stable global periodicity remains a hypothesis rather than a directly verified property. We will add this analysis to the Ablation Study section of the revised manuscript.

Q3: Justification of Concentrated Retrieval and Top-K Memory Stability.

A3: We sincerely appreciate the reviewer's thoughtful and constructive comments. Regarding concentrated retrieval, our motivation is to avoid combining too many potentially irrelevant prototypes, which may introduce noise into the imputation process. Selecting a few highly relevant prototypes provides more focused historical references. Importantly, the retrieved prototypes complement rather than replace the observed context, providing supplementary structural references while retaining the original observations as evidence for imputation. This helps mitigate the potential bias introduced by mismatched prototypes. Nevertheless, we acknowledge that concentrated retrieval cannot guarantee correct matching under highly ambiguous observations, but aims to reduce irrelevant memory interference.

Regarding Top-K updates, our strategy is designed to restrict the scope of memory updates rather than explicitly enforce diversity across queries. Specifically, each valid query updates only its top-K most similar memory slots, while inactive slots retain their original representations. A gated update further controls the integration of new information with existing prototypes. Additionally, following MEMTO [1], we employ entropy regularization to encourage sparse memory retrieval. Together, these mechanisms help reduce indiscriminate memory interactions and preserve inactive prototypes. Nevertheless, we acknowledge that they do not guarantee cross-query diversity or completely prevent memory collapse.

We will revise the manuscript accordingly to clarify these mechanisms and moderate the related claims.

[1] Song J, Kim K, Oh J, et al. Memto: Memory-guided transformer for multivariate time series anomaly detection[J]. Advances in Neural Information Processing Systems, 2023.

Reviewer_TQsbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TFMemDiff, a diffusion-based framework for time series imputation that supplements sparse observations with temporal and frequency-domain memory prototypes learned from historical training sequences. The method combines prototype initialization, retrieval, and updates, and is evaluated on eight datasets under several missingness patterns. The motivation is reasonable and the empirical coverage is broad. However, the independent contribution of historical structural priors, the claimed recovery of periodic structure, and the method's applicability under severe sparsity are not yet sufficiently established. Potential implicit target leakage in evaluation and discrepancies between the method description and the supplied implementation also require clarification before the reported gains can be interpreted reliably.

Critical strengths
  • Relevant problem and clear motivation. Using reusable patterns from training data to supplement conditional generation when the current instance contains few observations is a worthwhile research direction.
  • Reasonable motivation for the dual-domain design. Temporal and frequency-domain prototypes offer different, potentially complementary representations of local variation and periodic patterns.
  • Broad empirical coverage and attempts to examine component contributions. The evaluation includes datasets from multiple domains, both univariate and multivariate settings, point-error and probabilistic metrics, component ablations, and integration with an SSSD backbone.
Critical weaknesses

W1. The independent contribution of historical structural priors is not sufficiently isolated. The full model introduces a query encoder, dual-domain condition projections, and fusion networks alongside the memory mechanism. The existing removal-based ablations do not separate the benefit of information stored in historical prototypes from that of additional conditioning capacity. Tables 3 and 4 provide useful component and backbone comparisons, but a control that retains the same networks, parameter count, and training budget while changing only the source of the prototypes is still missing. The current evidence therefore supports the effectiveness of the overall conditioning module more directly than the specific contribution of historical structural priors.

W2. The claimed recovery of global periodic or spectral structure lacks direct validation. Figure 2 visualizes memory retrieval attention, while the FMM ablation in Table 3 reports only RMSE and MAE. The qualitative trajectories are not accompanied by quantitative measures of periodic structure. The paper does report CRPS, but averaging scores over individual missing entries does not, by itself, assess joint temporal dependence. A structural evaluation aligned with the central claim is needed to establish whether FMM improves the periodic or correlation structure of the imputed outputs, beyond reducing pointwise errors.

W3. The performance boundaries under severe sparsity are not adequately explained. Under 90% POINT missingness in Table 10, TFMemDiff obtains an RMSE of 0.793 on Electricity, compared with 0.564 for SAITS, and 0.656 on PEMS-BAY, compared with 0.490 for FSDI. The authors already acknowledge that the method is not uniformly superior under point missingness. However, the general explanation based on retained local context does not adequately account for these cases, in which only 10% of entries remain observed. The paper should better characterize which data regularities and missingness structures make historical prototype retrieval useful, and qualify its severe-sparsity claims accordingly. These results do not negate the method's benefits in some structured-missingness settings.

W4. Implicit target leakage in the code supplied by the authors and incomplete reproduction materials limit verification of the reported results. When a complete test window and its mask are passed together, the model computes the window mean and variance before applying the mask, including the hidden ground-truth values in these statistics. These statistics are then used to construct the observed conditioning inputs and to inverse-normalize the predictions. Hidden targets can therefore influence their own predictions indirectly through aggregate statistics. Even with the observed values, mask, model parameters, and random seed held fixed, changing only the hidden values can change the predictions. Although the authors provide core code, pseudocode, and some hyperparameters, the package lacks the data-processing pipeline, experiment configurations, end-to-end training and evaluation scripts, and the complete procedure connecting pretraining, prototype construction, and subsequent training. The total number of training diffusion steps T also remains unspecified. These omissions prevent verification of whether the reported tables use the above calling convention and limit independent assessment of the data protocol, baseline comparability, and final results. The authors should provide the complete reproduction pipeline, clarify the source of the normalization statistics, and reassess the main results using only observed values or fixed statistics estimated solely from the training data.

W5. Discrepancies between the method description and the supplied implementation affect the interpretation of the ablations. The paper describes the memory update gates as learnable, but the detach operations in the supplied loss path prevent gradients from reaching their U/W parameters. In addition, the univariate D=1 path in the supplied implementation bypasses both memory conditions, so they do not affect the denoiser's predictions. These discrepancies concern the claimed learnable update mechanism and the interpretation of the ETTh1(OT) ablations, respectively. The authors should provide the implementation version, configurations, and records used for Table 3, and establish whether implementation corrections and reevaluation are required. The supplied snapshot alone does not establish that the numerical results in the paper are incorrect.

W6. The memory mechanism lacks a targeted theoretical account. Section 3 specifies how retrieved temporal and frequency prototypes condition the denoiser, and Appendix D derives standard forward-diffusion and reverse-mean identities with the prototypes treated as given conditioning variables. These derivations do not establish under what assumptions historical prototypes improve conditional denoising or score estimation under sparse observations, or how retrieval mismatch may introduce bias. The ablations and SSSD integration provide useful empirical evidence, but do not explain these conditions and limits. A focused analysis under an explicit, simplified model would strengthen the central mechanism claim by clarifying when the proposed structural guidance should help and when it may fail.

Questions for authors

Q1. Could the authors retain the same query encoder, projection and fusion networks, and training budget, while replacing the historical prototypes with a learnable dictionary of the same size that is not initialized from historical representations? This control would more directly distinguish the contribution of historical structural information from that of additional conditioning capacity.

Q2. Could the authors report a periodic-structure recovery metric, such as power-spectrum discrepancy or autocorrelation discrepancy at relevant lags, for the full model and w/o FMM using the same test windows and missingness masks? The existing CRPS evaluation assesses marginal predictive distributions, whereas this comparison would directly test the structural role attributed to frequency-domain memory.

Q3. For high-error windows under 90% POINT missingness on Electricity and PEMS-BAY, could the authors present the imputation errors alongside the corresponding retrieval weights or prototype matches, and examine whether the availability of useful retrieval information is associated with error? The paper already offers a general explanation and attention visualizations, but does not connect these failure cases to retrieval behavior. This question does not presume that retrieval causes the degradation.

Q4. Could the authors provide the complete data-processing, training, and evaluation pipeline, including the experiment configurations and the total number of training diffusion steps T? Please specify the two-stage procedure: the pretraining objective, construction of the 10% training subset used for prototype initialization and its masks, transfer of weights between stages, and whether the encoder and frequency projection are frozen or fine-tuned. Since prototype construction uses interpolated inputs whereas the subsequent query path differs, how is compatibility between their representation spaces maintained? For the reported evaluations, does the input to evaluate() retain the hidden ground-truth values? If those values are replaced beforehand, how are the true evaluation targets kept separate from the returned x_raw? Please provide the calling procedure and report whether the main results and relative rankings persist when normalization uses only observed values or fixed statistics estimated solely from the training data.

Q5. Under explicit assumptions about structural similarity between training and test sequences, could the authors analyze how prototype approximation error and observation sparsity affect conditional noise-prediction error? An error decomposition or proposition in a simplified setting could identify conditions under which retrieval improves a finite-capacity estimator, as well as cases in which prototype mismatch introduces bias. This would clarify the mechanism and its limits without requiring a performance guarantee for the full neural architecture.

5 comments
AuthorsOpenReview ↗

Responses to Q1, Q2 and Q3

We sincerely thank you for your careful review and valuable feedback. Your insightful comments and suggestions have been very helpful to us. Below, we address each point.

Q1: The independent contribution of historical structural priors is not sufficiently isolated.

A1: We sincerely thank the reviewer for this insightful question. In the original manuscript, we presented a sensitivity analysis of the memory initialization ratio (α) in Figure 7. To further distinguish the contribution of historical structural information from that of additional conditioning capacity, we replace the historical prototypes with a learnable dictionary of the same size, initialized independently of historical representations (w/o Historical Init.), while keeping the query encoder, projection and fusion networks, and training budget unchanged. The results are summarized below (reported as RMSE / MAE):

MethodETTh1(OT) POINTETTh1(OT) SEQStock POINTStock SEQStock BLOCK
w/o Historical Init.0.076/0.0560.149/0.1180.086/0.0490.076/0.0440.186/0.134
TFMemDiff0.066/0.0450.138/0.1020.080/0.0450.067/0.0380.110/0.093

As shown in the table, TFMemDiff consistently outperforms the randomly initialized variant across all five evaluation settings. These results demonstrate the benefit of historical structural priors. We will include this additional ablation study and the corresponding discussion in the revised manuscript.

Q2: The claimed recovery of global periodic or spectral structure lacks direct validation. 

A2: We sincerely appreciate the reviewer’s emphasis on directly evaluating the structural contribution of FMM. To this end, we introduce Power Spectrum Discrepancy (PSD-Dist) and Autocorrelation Discrepancy (ACF-Dist) to assess the recovery of frequency-domain and temporal correlation structures. We compare TFMemDiff with w/o FMM on ETTh1 and Stock under different missing patterns(reported as PSD-Dist / ACF-Dist):

MethodETTh1(OT) POINTETTh1(OT) SEQStock POINTStock SEQStock BLOCK
w/o FMM0.270/0.0970.408/0.1780.165/0.0480.042/0.0130.347/0.120
TFMemDiff0.265/0.0950.399/0.1610.160/0.0460.038/0.0110.325/0.116

As shown in the table, TFMemDiff consistently achieves lower PSD-Dist and ACF-Dist across all evaluated settings, indicating that FMM helps preserve spectral characteristics and temporal correlation structures in the imputed sequences. We will include these additional metrics, experimental results, and corresponding discussions into the revised manuscript.

Q3: Relationship between retrieval quality and imputation errors under severe sparsity.

A3: We sincerely thank the reviewer for this insightful question. To examine whether the high imputation errors under 90% POINT missingness are associated with weaker matches to historical prototypes, we compare the RMSE and corresponding temporal and frequency retrieval similarities between low-error and high-error windows on Electricity and PEMS-BAY:

DatasetGroupRMSETemporal SimilarityFrequency Similarity
ElectricityLow-error0.37490.9060.916
ElectricityHigh-error1.36720.8900.763
PEMS-BAYLow-error0.13540.7540.490
PEMS-BAYHigh-error1.86130.7230.488

As shown in the table, high-error windows exhibit lower temporal and frequency retrieval similarities on Electricity, particularly in the frequency domain. On PEMS-BAY, temporal similarity also decreases, while frequency similarity remains nearly unchanged despite the substantial increase in RMSE. These results suggest that retrieval similarity is associated with imputation errors in some cases but does not fully explain the degradation. Under extreme sparsity, limited observations may be insufficient to capture sample-specific variations, even when relevant historical prototypes are available.

Importantly, the substantially lower errors in low-error windows under the same missingness level are consistent with the benefit of memory guidance when informative historical patterns can be retrieved. By providing complementary temporal and frequency structural priors, TFMemDiff can supplement sparse observations rather than relying solely on the limited observed context. However, its effectiveness still depends on the relevance of the retrieved information and the remaining observational evidence. We will incorporate this analysis into the revised manuscript to clarify both the benefits and limitations of memory guidance under severe sparsity.

AuthorsOpenReview ↗

Response to Normalization Protocol

Q4: Normalization Protocol

A4: We sincerely thank the reviewer for the detailed comments on the normalization procedure. We have revisited the implementation and provide the following clarifications.

As suggested, we report results of using observed-only normalization. The results are as follows(rankings are calculated across all evaluated baselines):

DatasetTypeSettingFGTIM.TCNGlocal.FSDIFull-window NormalizationObserved-only NormalizationRMSE Rank ChangeMAE Rank Change
ETTh1POINT10%0.059/0.0450.052/0.0380.132/0.0990.162/0.1280.047/0.0380.051/0.0401→11→2
ETTh1POINT50%0.105/0.0840.066/0.0480.129/0.0980.165/0.1300.065/0.0410.066/0.0451→11→1
ETTh1POINT90%0.283/0.2190.169/0.1020.218/0.1690.275/0.2010.116/0.0890.147/0.0921→11→1
ETTh1SEQ100.110/0.0830.100/0.0760.201/0.1550.17/0.1360.062/0.0460.098/0.0711→11→1
ETTh1SEQ200.149/0.1140.14/0.1090.285/0.2280.176/0.1390.092/0.0700.138/0.1021→21→1
ETTh1SEQ300.171/0.1320.148/0.1140.244/0.2020.215/0.1680.110/0.0850.148/0.1051→11→1
StockPOINT10%0.570/0.40.108/0.0762.214/1.5331.112/0.580.042/0.0230.042/0.0221→11→1
StockPOINT50%2.948/2.5820.126/0.0882.337/1.6491.165/0.6830.077/0.0420.080/0.0451→11→1
StockPOINT90%2.909/2.3730.482/0.1822.504/1.7681.825/1.4040.156/0.1040.188/0.1311→11→1
StockSEQ100.889/0.6450.165/0.1162.223/1.5571.424/1.1320.04/0.0210.049/0.0271→11→1
StockSEQ200.999/0.7150.155/0.1102.251/1.5690.934/0.6950.042/0.0220.067/0.0381→11→1
StockSEQ300.997/0.7690.176/0.1232.221/1.5422.093/1.3290.043/0.0230.095/0.0511→11→1
StockBLOCK10%0.863/0.6860.183/0.1312.208/1.5180.871/0.6560.074/0.0420.100/0.0841→11→2
StockBLOCK50%1.636/1.3670.165/0.1182.345/1.6602.111/1.4890.107/0.0620.110/0.0931→11→1
StockBLOCK90%2.617/2.0750.180/0.1282.391/1.7052.304/1.4450.113/0.0690.132/0.1171→11→1
METR_LAPOINT10%0.316/0.1870.282/0.1580.333/0.2100.268/0.1410.401/0.2010.290/0.14010→49→1
METR_LAPOINT50%0.420/0.2320.336/0.1810.408/0.2480.289/0.1490.413/0.2050.341/0.1687→44→2
METR_LAPOINT90%0.504/0.2560.556/0.2580.632/0.3590.448/0.2080.465/0.2430.468/0.2442→23→3
METR_LASEQ100.396/0.1890.418/0.2160.424/0.2550.395/0.1840.377/0.1830.384/0.1771→11→1
METR_LASEQ200.514/0.2360.464/0.2410.466/0.2750.448/0.2080.390/0.1960.441/0.2001→21→1
METR_LASEQ300.569/0.2690.552/0.2750.468/0.2790.472/0.2210.394/0.1970.428/0.2151→11→1
METR_LABLOCK10%0.351/0.1770.390/0.2060.414/0.2500.395/0.1910.397/0.20.384/0.1797→44→2
METR_LABLOCK50%0.470/0.2750.404/0.2160.446/0.2700.423/0.1990.402/0.1970.403/0.1921→11→1
METR_LABLOCK90%0.535/0.3210.428/0.2270.506/0.2950.449/0.2150.426/0.2140.416/0.2211→11→2
PEMS-BAYPOINT10%0.244/0.1160.217/0.1230.272/0.1440.258/0.1070.387/0.1820.197/0.10610→19→1
PEMS-BAYPOINT50%0.433/0.2590.412/0.2660.344/0.1930.233/0.1220.408/0.2000.267/0.1406→25→2
PEMS-BAYPOINT90%0.647/0.3620.653/0.2700.532/0.2960.49/0.2160.656/0.3360.686/0.3419→98→8
PEMS-BAYSEQ100.463/0.2140.444/0.2170.461/0.2400.622/0.2160.378/0.1800.383/0.1811→11→1
PEMS-BAYSEQ200.694/0.3250.523/0.2590.477/0.2640.68/0.2340.379/0.1830.400/0.2281→11→1
PEMS-BAYSEQ300.846/0.4280.751/0.3180.496/0.2710.634/0.2730.379/0.1820.493/0.2701→11→1
PEMS-BAYBLOCK10%0.357/0.1790.362/0.1900.379/0.2120.39/0.1930.371/0.1840.345/0.1705→23→1
PEMS-BAYBLOCK50%0.476/0.2050.417/0.2180.435/0.2370.455/0.2040.408/0.2030.410/0.2021→11→1
PEMS-BAYBLOCK90%0.534/0.2880.448/0.2300.474/0.2590.51/0.2260.437/0.2210.451/0.2331→21→3

Overall, TFMemDiff still maintains competitive rankings under observed-only normalization. This robustness can be attributed to the global structural guidance provided by historical temporal and frequency prototypes, which complements sparse observations and reduces the model's reliance on window-specific normalization statistics. Importantly, these results continue to support the central claim and main contribution of our work.

AuthorsOpenReview ↗

Responses to Q4, Q5 and W5

Q5: Experimental Reproducibility.

A5: We sincerely thank the reviewer for this thoughtful question. The data-processing pipeline follows the Time-Series-Library (TSLib)[1]. For TFMemDiff, the diffusion process uses 200 training steps (T=200) and 20 inference steps. In Stage 1, the model is pretrained using the noise-prediction loss. For memory initialization, 10% of the training data are randomly sampled, masked, and linearly interpolated. The pretrained encoder extracts temporal representations, while rFFT and learnable frequency projection extract frequency representations. K-means is then applied separately to initialize TMM and FMM. In Stage 2, the pretrained non-memory parameters are transferred, while the initialized prototypes are retained. The encoder and frequency projection remain trainable. Although prototype construction and query inputs follow different processing paths, both use the same encoder and frequency projection, ensuring a shared representation space. Memory items are further updated using observed query representations. We will provide the corresponding data-processing, training, and evaluation scripts, together with the experimental configurations, to facilitate reproducibility upon acceptance.

[1] Wang Y, Wu H, Dong J, et al. Deep time series models: A comprehensive survey and benchmark[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026.

Q6: Implementation inconsistencies in memory updates and ablations.

A6: We thank the reviewer for identifying these implementation discrepancies. We have re-examined the implementation and revised the corresponding descriptions and experiments.

Regarding the U/W parameters, we confirmed that the detach() operations prevent gradients from propagating to U/W through the imputation loss. These operations were introduced to avoid retaining computational graphs across successive memory updates and reduce GPU memory consumption. We acknowledge that describing U/W as learnable gates was inaccurate. We have therefore revised the manuscript to clarify that U/W are fixed linear mappings used to calculate data-dependent memory update gates, rather than parameters optimized through the imputation loss.

Regarding the univariate D=1 pathway, we corrected the conditional learning implementation to ensure that the memory conditions are properly incorporated into the denoiser instead of being bypassed. We consequently re-evaluated all ablation experiments in Table 3 using the corrected implementation and normalization protocol. The revised results are presented below(reported as RMSE / MAE):

MethodETTh1 POINTETTh1 SEQ
w/o Memory0.122/0.0960.187/0.166
w/o TMM0.076/0.0560.147/0.116
w/o FMM0.077/0.0540.142/0.106
w/o Sparse Attention0.075/0.0550.139/0.105
w/o Memory update0.069/0.0510.141/0.107
Full0.066/0.0450.138/0.102

The revised results show that removing either memory component, the sparse memory update mechanism, or dynamic memory updating leads to increased imputation errors. Removing the entire memory module results in the largest performance degradation. These findings support the contributions of the proposed memory components. We will update Table 3 and revise the methodological descriptions accordingly.

Q7: Theoretical Analysis and Limitations of Memory Guidance.

A7: We sincerely thank the reviewer for this thoughtful question. We acknowledge that the current derivations mainly describe how historical prototypes are incorporated into conditional diffusion, without formally establishing the conditions under which memory guidance improves noise prediction. We agree that an error decomposition under a simplified model would provide valuable theoretical insights. However, such an analysis requires additional assumptions about structural similarity, retrieval errors, and estimator capacity, which have not been formally established in our current work. We believe that a rigorous treatment of these factors requires further theoretical investigation beyond the scope of this study.

Nevertheless, we would like to clarify the underlying conditions and limitations as much as possible. Memory guidance relies on structural similarities between training and test sequences. When such similarities exist, the retrieved temporal and frequency prototypes can provide complementary structural priors to supplement the limited information available under sparse observations, thereby potentially improving conditional noise prediction. However, when the retrieved prototypes poorly match the underlying structure of the test sequence, they may introduce misleading guidance and bias noise prediction. Moreover, severe observation sparsity may reduce retrieval reliability, further limiting the benefits of memory guidance. We will clarify these assumptions and limitations in the revised manuscript.

Comments on the paper

1 comment

We sincerely thank all reviewers for their insightful reviews and positive perspectives, which are a great encouragement to us. Specifically, the reviewers commented:

  • The proposed approach addresses a limitation that stronger processing of the current instance may not fully overcome (Reviewer RpRH) with clear motivation (Reviewer TQsb) and offers a reasonable and interesting idea (Reviewer hHuN) of using dual-domain external memory to guide conditional diffusion under extreme sparsity (Reviewers wto2).
  • The dual-domain design is considered conceptually appealing (Reviewer wto2), as temporal and frequency-domain prototypes offer different, potentially complementary representations of local variation and periodic patterns (Reviewer TQsb).
  • The proposed memory mechanism provides a meaningful architectural extension of conditional diffusion (Reviewer RpRH), allowing recurring patterns from training data to be explicitly retrieved during imputation. Its practical mechanisms also help stabilize memory learning and avoid collapse (Reviewer wto2).
  • The experiments provide broad empirical coverage (Reviewer TQsb) and convincing support for the method's practical value (Reviewer RpRH). The experimental results are generally strong across different datasets and missing patterns (Reviewer hHuN), and ablation studies and the evaluation with SSSD provide useful evidence for the effectiveness of the memory modules (Reviewers hHuN).

The reviewers also raised insightful and constructive concerns. We made every effort to address all the concerns by providing sufficient evidence and requested results.

Here is the summary of the major revisions:

1. More Ablation Studies and Robustness Evaluation [Reviewers: TQsb]

We have added a ablation study by replacing historical prototypes with an independently initialized learnable dictionary. Additionally, we have evaluated the model using observed-only normalization. The results have further demonstrated the contributions of historical structural priors, and confirming the robustness of TFMemDiff under different normalization protocols.

2. Additional Structural Evaluation of Dual-Domain Memory [Reviewers: TQsb, RpRH, hHuN]

We have introduced Power Spectrum Discrepancy (PSD-Dist) and Autocorrelation Discrepancy (ACF-Dist) to evaluate the preservation of spectral characteristics and temporal correlations. The results have shown that TFMemDiff consistently achieved lower discrepancies than the variant without FMM across different missing patterns. We have also analyzed retrieval similarities between low-error and high-error windows under 90% point missingness on Electricity and PEMS-BAY, providing further insights into the benefits of memory guidance under extreme sparsity.

3. Comparison with Related Work [Reviewers: hHuN, wto2]

We have expanded the comparison with existing methods, particularly ProDiff and LSCD, and revised Table 1 to clarify their methodological differences from our TFMemDiff.

4. Theoretical and Mechanistic Explanations [Reviewers: TQsb, RpRH, wto2]

We have further clarified the roles of temporal and frequency-domain memory, the use of historical prototypes as external structural priors, and the mechanisms of memory retrieval and updating. We have also expanded the discussion of the conditions under which memory guidance is beneficial and its limitations.

We would like to note that Reviewer TQsb raised their score from 2→32 \rightarrow 3, Reviewer wto2 and Reviewer hHuN maintained a positive rating, and Reviewer RpRH, who initially gave a positive rating, has not yet responded.

The valuable suggestions from reviewers are very helpful for us to revise the paper to a better shape. We will greatly appreciate your kind consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.