acceptodds
Under review as a conference paper at ICLR 2027

Cross-View Time-Frequency Diffusion for Extreme-Aware Time-Series Generation

Abstract

Generating realistic time-series requires faithfully reproducing rare but high-impact extreme events. Existing approaches often rely on specialized representations or extreme-focused training objectives, which may restrict the diversity of captured extremes or compromise overall distributional fidelity. We propose Cross-View Time-Frequency Diffusion (CrossTF), a framework that exploits discrepancies between time- and frequency-view denoisers for extreme-aware generation. Our key empirical finding is extreme-sensitive cross-view asymmetry: moving along the direction connecting the two views' clean-sample estimates substantially changes extreme fidelity while having comparatively limited effects on overall generation quality. We further identify a late-stage cross-view window in which this discrepancy stabilizes and cross-view adjustments have a particularly strong effect on extreme fidelity. These findings motivate a staged optimization strategy in CrossTF: after training and freezing both denoisers, it only needs to learn step-specific fusion parameters within this window using an Extreme Value Theory-based objective under an energy-distance constraint. This formulation supports interpolation between and extrapolation beyond the view-specific estimates while limiting degradation in overall generation quality. Extensive experiments on seven real-world datasets against 11 baselines show that CrossTF improves extreme fidelity by 26.1% and overall performance by 7.4% relative to the strongest baselines. These results highlight cross-view discrepancy as an effective basis for low-dimensional optimization in extreme-aware time-series generation.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-series generation, diffusion models, extreme-aware generation, time-frequency modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oC1kOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Cross-View Time-Frequency Diffusion (CrossTF) for extreme-aware time-series generation. The paper reports average improvements of 26.1% in extreme fidelity and 7.4% in aggregate overall performance over the strongest baselines. The paper is well written, and its experimental evaluation is extensive.

Critical strengths
  1. Exploiting discrepancies between time- and frequency-view denoisers provides a compelling direction for refining extreme behavior. The asymmetry analysis, window analysis, and same-view ensemble comparison provide useful empirical motivation for this design.

  2. The experiments cover diverse domains, generic and extreme-aware generators, and single-view ablations. These findings support a substantive benefit for reproducing extreme severity.

Critical weaknesses
  1. Appendix D.3 states that baselines train on each dataset's full array; the released code likewise trains the CrossTF branches, optimizes fusion, and evaluates fidelity against the same complete real dataset. Consequently, the real data used for the predictive evaluation have already been available to the generator, and the current protocol does not establish utility on unseen real sequences. The authors should provide a clear train/validation/test protocol.

  2. The same 18-step schedule and final-six-step window are used on all seven datasets. However, they do not show whether the chosen window is similarly suitable for each dataset, or whether its benefits persist when the window boundary, sampling schedule, or number of steps changes.

  3. Sections 2.3 described the EDM implementation, but the paper does not explain why its preconditioning and sampler are particularly suitable for combining the two views or test whether the cross-view benefit persists with another diffusion backbone.

  4. Table 2 reports a single Extreme Fidelity and Overall Performance score per dataset without stating the aggregation formula or explicitly listing the metrics entering each score. This omission makes the headline 26.1% and 7.4% improvements harder to interpret and verify.

  5. In Table 3, FIDE and HeavyDiff perform worse on most extreme-fidelity metrics than a general-purpose baseline such as ImagenTime. Appendix D.3 describes substantial intervention in these comparisons: FIDE's implementation was repaired, HeavyDiff was reimplemented, and E4GEN's event threshold was changed. It is difficult to determine whether their weak results reflect the methods, the adaptations, or the evaluation setting.

  6. Appendix D.7 reports 3.70 seconds to generate 1,000 Precipitation samples and explains that CrossTF uses only 18 denoising steps. This is useful evidence of practical speed, but ImagenTime also uses 18 steps, while its reported backbone is wider (128 versus 64 base channels). Moreover, CrossTF uses two denoisers within the fusion window and incurs a separate coefficient-optimization cost. Please report matched-backbone and batch-size timing, actual denoiser evaluation counts, and separate training, fusion-optimization, and generation costs so the efficiency claim can be attributed to the proposed design.

Questions for authors

See Weaknesses.

11 comments
AuthorsOpenReview ↗

Response to Weaknesses 1 (1/2)

We sincerely thank the reviewer for their recognition and strong support of our work. We are encouraged by their positive assessment of our research direction, well-motivated analyses, clear presentation, and extensive experiments. To further address the reviewer's comments, we have added new analyses and experiments in Appendices J–O (Pages 49–54) of the revised manuscript. We respond to each comment below.


Weakness 1: Train/Test Protocol and Utility on Unseen Real Data

We sincerely thank the reviewer for this valuable suggestion. We agree that predictive utility should be evaluated on real sequences unseen by the generator to properly assess its generalization. We first clarify our evaluation protocol and provide a memorization analysis, followed by new predictive results under an explicit train/test split.

(1) Evaluation Protocol for Unconditional Time-Series Generation. We first clarify the evaluation protocol adopted in our paper. Unlike supervised learning, which typically requires an explicit train/validation/test split, unconditional time-series generation aims to learn the underlying distribution of the training data. In practice, a common evaluation protocol in this field assesses generation fidelity using the same data for both training and evaluation, as adopted by representative methods such as TimeGAN [1] and Diffusion-TS [2]. Moreover, CrossTF and all baselines follow exactly the same data and evaluation protocol, ensuring a fair comparison.

[1] Time-series Generative Adversarial Networks. NeurIPS, 2019.

[2] Diffusion-TS: Interpretable Diffusion for General Time Series Generation. ICLR, 2024.

(2) Predictive Utility Evaluation and Data Memorization. For predictive utility, we adopt the widely used Train on Synthetic, Test on Real (TSTR) protocol, where the downstream predictor is trained exclusively on synthetic data and never directly accesses the real test data during training. Although the generator is trained on the same real data used for testing, this does not necessarily mean that the synthetic data contain leaked information from these samples. To examine this concern, we conduct a new memorization analysis using three metrics: 1 − DCR [3], 1 − NNDR [4], and Copy Rate [5]. The Real Data (Reference) row measures the natural similarity between independent real sequences by comparing each sequence with the others. As shown in Tables TR1–TR3, CrossTF and other advanced generative methods exhibit near-zero copy rates, consistent with the real-data reference. These results suggest that synthetic data contain minimal replication of real test samples, reducing the risk of indirect data leakage to the predictor and supporting the validity of the TSTR evaluation. Details are provided in Appendix J.

[3] Data Synthesis based on Generative Adversarial Networks. VLDB, 2018.

[4] CTAB-GAN: Effective Table Data Synthesizing. ACML, 2021.

[5] Extracting Training Data from Diffusion Models. USENIX Security, 2023.

Table TR1. Memorization evaluation using 1 − DCR across datasets. Lower is better.

MethodBuildingECGElectricityPrecipitationTemperatureTrafficTransportation
CrossTF0.9000.2050.0750.3670.5090.5120.796
Diffusion-TS0.9520.3810.3460.7180.5590.6930.876
ImagenTime0.9130.2250.0840.3800.5290.4970.808
E4GEN0.9490.0500.1960.4800.5060.5920.840
Real Data (Reference)0.9060.2470.0930.4910.6530.5350.784

Table TR2. Memorization evaluation using 1 − NNDR across datasets. Lower is better.

MethodBuildingECGElectricityPrecipitationTemperatureTrafficTransportation
CrossTF0.0630.1600.0460.0550.0600.0560.067
Diffusion-TS0.0700.1850.0460.1480.0590.0650.098
ImagenTime0.0700.1630.0450.0590.0740.0560.079
E4GEN0.0730.1100.0440.0710.0530.0620.080
Real Data (Reference)0.1080.1780.0490.1780.1660.0690.068

Table TR3. Copy rate (%) across datasets. Lower is better.

MethodBuildingECGElectricityPrecipitationTemperatureTrafficTransportation
CrossTF0.000.170.000.060.000.000.00
Diffusion-TS0.000.430.000.650.000.000.10
ImagenTime0.020.270.000.000.000.000.08
E4GEN0.000.030.000.090.000.000.00
Real Data (Reference)0.000.600.000.060.000.000.00
AuthorsOpenReview ↗

Response to Weaknesses 1 (2/2) & Response to Weaknesses 2 (1/2)

(3) Predictive Utility on Unseen Real Data. Although our original protocol follows common practice in the field, we fully agree with the reviewer that holding out real test data provides a more rigorous evaluation. We therefore adopt an 80%/20% train/test split: the generator, including fusion parameter optimization, is trained exclusively on the 80% training set, while the downstream predictor is trained only on synthetic data. The remaining 20% of real sequences are reserved exclusively for testing and are unseen by both the generator and predictor. We also report Train on Real, Test on Real (TRTR) as a reference, where the predictor is trained directly on the 80% real training data under the same split. As shown in Table TR4, CrossTF achieves nearly identical predictive errors under the held-out and original protocols (difference ≤ 0.0008), with performance comparable to TRTR (difference ≤ 0.0002). These results demonstrate that CrossTF's predictive utility does not depend on whether the generator has seen the test data. Details are provided in Appendix K.

Table TR4. Predictive utility under the original and held-out evaluation protocols (MAE ↓). Lower is better.

DatasetCrossTF (Original)CrossTF (80/20 Held-out)TRTR (Reference, 80/20 Held-out)
Building0.00260.00240.0024
ECG0.00690.00680.0066
Electricity0.02980.03030.0303
Precipitation0.03250.03170.0318
Temperature0.01610.01580.0157
Traffic0.01980.01980.0198
Transportation0.00950.00910.0090

Weakness 2: Robustness of the Cross-View Window

We sincerely thank the reviewer for raising this important question. We first clarify the design idea behind the cross-view fusion window. Rather than optimizing the window separately for each dataset, we prioritize the generality and practical applicability of CrossTF. We therefore adopt a unified fusion window across all datasets, avoiding dataset-specific tuning. This window is not arbitrarily chosen but is grounded in our data-driven analysis: Figures 6 and 7 show that, across all seven datasets, the last six denoising steps are where cross-view discrepancies become stable and extreme fidelity is most sensitive to fusion.

To further validate the robustness of this unified window (σ ≤ 0.35), we conduct two additional experiments: (1) varying the window boundary while fixing the total number of sampling steps to compare the unified window with dataset-specific optimal windows; and (2) varying the total number of sampling steps while keeping the window defined by σ ≤ 0.35 to examine its effectiveness across different sampling configurations. Details are provided in Appendix L.

(1) Window Boundary Ablation. To verify whether the unified window (k = 6) is close to the optimal window on each dataset, we fix the total number of sampling steps to 18 and start the fusion window at the k-th last step (k ∈ {2, 4, 5, 6, 7, 8, 10, 12, 15}), keeping all other settings identical to CrossTF. To eliminate the effect of different numbers of parameters across windows, we optimize a single shared α for each setting. As shown in Table TR5, the improvement in extreme fidelity rises rapidly as k increases from 2 to 6 and largely stabilizes for k ≥ 6. The unified window k = 6 is close to the dataset-specific optimal window on all seven datasets, with a gap of at most 0.4 percentage points. Moreover, under the step-wise fusion of CrossTF, enlarging the window requires solving for more fusion parameters α, which increases the optimization cost without bringing noticeable gains. These results confirm that the unified window serves as a robust and near-optimal default across datasets, without requiring dataset-specific tuning.

Table TR5. Improvement in extreme fidelity over Time-Only under different window starting points k (18 sampling steps, single shared α).

Datasetk=2k=4k=5k=6 (Default)k=7k=8k=10k=12k=15
Precipitation+0.5%+10.0%+13.7%+14.5%+14.6%+14.9%+14.4%+14.4%+14.7%
Electricity+3.6%+51.4%+55.3%+56.4%+55.8%+55.9%+55.9%+55.3%+55.6%
Temperature+0.1%+13.2%+14.3%+14.8%+14.1%+11.3%+14.3%+15.1%+14.7%
Traffic+0.1%+1.1%+1.0%+1.7%+0.8%+0.4%+0.0%+0.4%+0.3%
Transportation−0.4%+2.2%+3.8%+5.6%+5.8%+5.4%+4.8%+5.6%+6.0%
Building+1.1%+10.1%+12.4%+13.1%+13.2%+12.9%+12.4%+12.2%+12.3%
ECG−0.3%+2.7%+5.3%+5.8%+5.8%+5.9%+5.7%+5.2%+5.8%
AuthorsOpenReview ↗

Response to Weaknesses 2 (2/2) & Response to Weaknesses 3 (1/2)

(2) Sampling Step Ablation. To examine whether the cross-view benefit persists across different sampling steps, we vary the total number of steps among 12, 18, 32, and 50, while keeping the fusion window defined by σ ≤ 0.35 and all other settings identical to Experiment (1). In Table TR6, each percentage represents the improvement in extreme fidelity of CrossTF over Time-Only at the same sampling steps. Positive values indicate a benefit from cross-view fusion; since the Time-Only baselines differ across step settings, these percentages are not directly comparable across columns. The results show that CrossTF consistently outperforms Time-Only on all seven datasets at 18, 32, and 50 steps, demonstrating that the fusion benefit persists across different sampling steps. At 12 steps, the improvement varies considerably across datasets: it is substantial on Precipitation, Electricity, and Traffic, while the changes on the other datasets are close to zero or negative. Overall, with 18 or more sampling steps, the unified fusion window (σ ≤ 0.35) consistently improves extreme fidelity across all seven datasets.

Table TR6. Extreme fidelity of Time-Only → CrossTF under different numbers of sampling steps (window defined by σ ≤ 0.35, single shared α). Percentages denote the improvement of CrossTF over Time-Only under the same number of steps.

Dataset12 Steps18 Steps (Default)32 Steps50 Steps
Precipitation0.318 → 0.161 (+49.4%)0.111 → 0.095 (+14.5%)0.097 → 0.080 (+17.7%)0.095 → 0.084 (+11.9%)
Electricity0.457 → 0.211 (+53.9%)0.164 → 0.072 (+56.4%)0.165 → 0.066 (+60.1%)0.162 → 0.077 (+52.2%)
Temperature0.242 → 0.242 (−0.2%)0.106 → 0.090 (+14.8%)0.100 → 0.084 (+16.3%)0.121 → 0.104 (+13.7%)
Traffic0.084 → 0.070 (+16.2%)0.028 → 0.028 (+1.7%)0.061 → 0.043 (+29.9%)0.073 → 0.051 (+30.2%)
Transportation0.124 → 0.124 (−0.2%)0.049 → 0.046 (+5.6%)0.063 → 0.061 (+2.7%)0.071 → 0.061 (+13.6%)
Building0.074 → 0.076 (−2.7%)0.100 → 0.087 (+13.1%)0.126 → 0.122 (+3.3%)0.133 → 0.129 (+2.9%)
ECG0.441 → 0.441 (+0.1%)0.141 → 0.133 (+5.8%)0.132 → 0.129 (+2.2%)0.127 → 0.122 (+3.7%)

Weakness 3: Choice of EDM and Robustness to Diffusion Backbones

We sincerely thank the reviewer for this insightful comment. We first explain why EDM is well suited to cross-view fusion, then examine whether the cross-view benefit persists with another diffusion backbone through two modifications: (1) replacing Heun with Euler sampling (corresponding to deterministic DDIM-style updates), and (2) replacing EDM with the VP formulation (DDPM-style ε-prediction and variance-preserving noise schedule). We acknowledge that both variants remain within EDM's unified design space rather than representing fundamentally different generative paradigms. Evaluating fundamentally different diffusion architectures would require redesigning and retraining our entire network architecture and pipeline, which is beyond the scope of this rebuttal and left to future work. Details are provided in Appendix M.

(1) Why EDM? We adopt EDM as the base diffusion framework for three reasons:

  1. Shared Output Space (Preconditioning). EDM's preconditioning (cskipc_{\text{skip}}, coutc_{\text{out}}, cinc_{\text{in}}) allows both denoisers to produce clean-sample estimates x^0\hat{x}_0 in the same data space and at consistent scales, enabling the linear fusion in Eq. 5.
  2. Explicit Noise Parameterization. EDM directly conditions denoisers on the noise level σ\sigma, aligning both views at each denoising step. This allows us to define the fusion window by noise level (σ≤0.35\sigma \leq 0.35) rather than step count, and analyze the evolution of cross-view discrepancies (Section 3.2).
  3. Deterministic Sampling. Given the same initial noise, fused and unfused sampling differ only in the fusion weights α\alpha, enabling controlled comparisons and optimization. The sampling steps and fusion window can also be adjusted without retraining the denoisers.

Notably, these properties are not unique to EDM. As demonstrated in the following experiments, the cross-view benefit persists under alternative diffusion configurations.

Reviewer_9V8xOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Realistic time-series generation requires faithfully reproducing rare but important extreme values and abrupt changes. Existing extreme-specific representations or training objectives may restrict the patterns captured or compromise overall distributional fidelity. CrossTF uses the discrepancy between time- and frequency-view clean-sample estimates as a direction for extreme-focused optimization. The authors observe that adjustments along this direction affect extreme-fidelity metrics more strongly than denoising loss or overall quality, particularly in late sampling stages. CrossTF independently trains and freezes a delay-embedding-based time-view denoiser and an STFT-based frequency-view denoiser. Their estimates are combined as x^0=x^0F+αi(x^0T−x^0F)\hat x_0=\hat x_0^F+\alpha_i(\hat x_0^T-\hat x_0^F), with only six scalar fusion parameters optimized over the final six steps of an 18-step EDM sampler. An EVT-based objective matches tail shape, occurrence frequency, and severity, subject to an energy-distance constraint on overall distributional quality. The coefficients can lie outside [0,1][0,1], supporting both interpolation and extrapolation. Experiments on seven datasets spanning traffic, precipitation, building power, temperature, electricity, traffic occupancy, and ECG report improvements over 11 baselines.

Critical strengths
  1. An important and easily overlooked problem. The paper highlights the need to preserve extreme temporal patterns that may otherwise be smoothed out or treated as noise. Although extreme-aware generation has been studied previously, the paper brings attention to the challenge of improving extreme fidelity while preserving overall generation quality.

  2. Broad empirical analysis. The paper examines the method from multiple perspectives, including cross-view sensitivity, the timing of fusion, single-view comparisons, learned coefficients, same-view model diversity, downstream utility, and computational efficiency.

Critical weaknesses
  1. Missing evaluation using prior extreme-specific metrics. The paper evaluates extreme fidelity using four newly introduced metrics rather than the extreme-specific evaluations used by FIDE, HeavyDiff, and E4GEN.

  2. Insufficient justification of the new metrics relative to existing evaluations. This paper lacks a concrete justification for the proposed metric. To introduce a new metric, the authors should identify the aspects that existing metrics fail to capture and explain why the new metric was designed.

  3. The supporting analyses do not fully establish why time–frequency fusion is particularly suitable for extreme-aware generation.

    • Table 1: I think it is natural that extreme cases exhibit larger changes. Since extreme cases are, by definition, unusual, they are likely to. have high standard deviations. If the underlying distribution has a high standard deviation, larger changes in extreme cases are expected. I. believe similar results could be obtained not only by combining cross-view models, but also by combining similar-view models or using other types of cross-view approaches (other than time-frequency).
    • Figure 5: I think it is commonly observed that different metrics perform better depending on the dataset, model, or other experimental conditions. The proposed method should demonstrate consistent and non-marginal improvements across a large number of datasets.
    • Figure 6: I think this phenomenon is expected because, as the prediction progresses to later steps, only smaller steps are needed to generate subsequent data points. I believe similar results could be obtained not only by combining cross-view models, but also by combining similar-view models or using other types of cross-view approaches (other than time-frequency).
    • Figure 7: Similarly, experiments are needed to examine whether comparable results can be obtained by combining similar-view models or. using other types of cross-view approaches (other than time-frequency).
Questions for authors

N/A

20 comments
AuthorsOpenReview ↗

Response to Weaknesses 1 & 2

We sincerely thank the reviewer for the constructive and critical comments, and especially for recognizing the broad empirical analysis in our paper and the importance of the problem we study. We respond to each comment below. All new experiments have been added to Appendices E and F of the revised manuscript.


Weaknesses 1 & 2: Missing Prior Extreme-Specific Metrics and Justification of New Metrics

We sincerely thank the reviewer for this valuable suggestion. We fully agree that prior extreme-specific metrics provide a more comprehensive evaluation and help assess whether CrossTF's advantages generalize beyond our proposed metrics. We have therefore added all extreme-specific metrics used in these prior methods and compared them with CrossTF in Table R1. We first justify the design of our new metrics and then present the complete results.

(1) Why We Did Not Directly Adopt Prior Extreme-Specific Metrics.

Extreme-aware time-series generation remains an emerging research direction, and no unified standard has yet been established for evaluating extreme fidelity. Even the three methods highlighted by the reviewer, FIDE, HeavyDiff, and E4GEN, do not share any common extreme-specific metric (see Table R1). In contrast, for overall time-series generation, standard evaluation metrics are already well established, and we directly adopt widely used metrics such as Wass., Marg., and ACD. Thus, our choice was not to avoid prior extreme-specific metrics, but to contribute a new set of metrics to existing evaluation practices.

We agree that the metrics adopted by FIDE, HeavyDiff, and E4GEN are well motivated. FIDE's block maxima are grounded in classical extreme value theory, HeavyDiff's kurtosis and tail distribution characterize tail behavior, and E4GEN's event-level metrics capture the temporal structure of extreme events. However, these metrics mainly focus on extreme magnitudes themselves and provide limited coverage of extreme changes between adjacent time points. This limitation is a key motivation for our new metrics.

(2) Justification of Our New Extreme-Specific Metrics.

To complement these existing evaluations, we design four extreme-fidelity metrics along two dimensions: the type of extremes (magnitude/change) and their properties (frequency/severity). Unlike prior metrics that mainly focus on extreme magnitudes, our metrics also explicitly capture extreme changes between adjacent time points. They provide two advantages:

  1. Coverage of previously overlooked extreme changes. Existing extreme-specific metrics mainly focus on unusually large values, while extreme changes between adjacent time points are rarely evaluated. These two types of extremes are not equivalent. For example, on Building (Appendix D.8, Table 15), RF-Diffusion achieves a competitive Mag-Excess of 0.0612 and ties for the best Wass., yet its Chg-Rate is 3.4284, about 33$\times$ that of CrossTF. This shows that magnitude-focused metrics can miss substantial distortions in extreme changes.

  2. Clear theoretical support from extreme value theory. Our metric design is directly motivated by the Peaks-Over-Threshold (POT) framework in extreme value theory [1]. POT characterizes extreme behavior from two aspects: the frequency of threshold exceedances and the magnitude of exceedances beyond the threshold. Accordingly, our Rate metric measures how often extreme events occur, while our Excess metric measures their severity. We apply this design to both magnitude extremes and change extremes. Our experiments further show that matching only one of these aspects is insufficient to ensure realistic extreme fidelity.

[1] Stuart Coles. An Introduction to Statistical Modeling of Extreme Values. Springer, 2001.

Specifically, our original submission already provides several key justifications for the metric design. Section 2.1 (Lines 123–140) and Fig. 3 explain the motivation; Appendix C.1 (Lines 898–905) further justifies the distinction between extreme magnitude and extreme change using real-world extreme patterns; and Appendix D.2 (Lines 1214–1299) provides the detailed metric definitions and design choices. We agree with the reviewer that the original explanation was not sufficiently comprehensive, and we have therefore added this comparison and expanded the corresponding discussion in the revised version.

(3) Evaluation with Prior Extreme-Specific Metrics.

We additionally evaluate CrossTF against FIDE, HeavyDiff, and E4GEN on all seven datasets using the extreme-specific metrics from their original papers. CrossTF achieves the best results in 24/28 cases under FIDE’s metrics, 18/21 under HeavyDiff’s metrics, and 60/63 under E4GEN’s metrics, for 102/112 (91.1%) cases overall. This confirms that CrossTF's advantage remains consistent under prior evaluations and is not specific to our proposed metrics. Detailed results are shown in Table R1 and added to the revised version.

AuthorsOpenReview ↗

Response to Weaknesses 3 (1/2)

Weakness 3: Why Time-Frequency Fusion is Particularly Suitable for Extreme-Aware Generation

We thank the reviewer for this insightful question. The key reason is that the time and frequency views have complementary strengths in generating extremes. As stated in our paper, their discrepancy therefore provides a direction along which extreme fidelity can be improved while largely preserving overall generation quality (Section 1, Page 2, Lines 057--061), thereby alleviating the core trade-off between extreme and overall fidelity in extreme-aware generation. Our submission supports this through comprehensive analysis: Figure 5 (Section 2.2, Pages 3--4, Lines 156--175) shows why the two views should be combined, Figures 6 and 7 (Section 3.2, Pages 5--6, Lines 259--285) identify when they should be combined during denoising, and Table 1 (Section 3.2, Page 5, Lines 216--258) shows that changes along the time--frequency discrepancy direction have a strong effect on extreme fidelity but only a limited effect on overall generation quality.

Regarding the reviewer’s main concern that the large changes may not be specific to time--frequency fusion, but could also arise in other view fusions due to the high variability of extremes, we would like to emphasize that this has already been directly examined in our original submission. As the reviewer noted in the Critical Strengths, our original submission systematically analyzes cross-view sensitivity, single-view comparisons, and same-view model diversity. Specifically, we provide two pieces of evidence:

  1. Retraining with different random seeds tests whether the large changes in extreme fidelity can be explained simply by the inherent variability of extremes (Appendix C.2, Table 4, Pages 18–19, Lines 971–989).

  2. Same-view fusion tests whether similar effects can arise from view fusion in general, rather than specifically from combining the time and frequency views. For the most direct comparison, our original submission applies the same fusion procedure to time--time models (Appendix D.6, Figure 10, Table 11, Pages 30–32, Lines 1614–1690), and we further add frequency--frequency fusion in the rebuttal.

We address each of these concerns individually in the following responses.


Weakness 3.1: Whether the Large Changes in Table 1 Are Specific to Time--Frequency Fusion Given the High Variability of Extremes

We thank the reviewer for this valuable comment. We agree that extreme values can exhibit high variability. However, this alone cannot explain the large changes in Table 1. If it did, similar changes should also appear under seed variation and same-view fusion. We examine both cases below.

(1) Retraining with different seeds. Retraining with different random seeds captures the inherent variability of extremes. As shown in Table R2, this changes extreme fidelity by only 9.62% (median), compared with 145.6% for time--frequency fusion, a 15.1$\times$ larger effect. Time--frequency fusion produces larger changes on all seven datasets, showing that the large changes in Table 1 cannot be explained by extreme variability alone.

Table R2. Extreme-fidelity changes from cross-view fusion versus retraining with different random seeds. All results are directly taken from our original submission (Appendix C.2, Table 4). Extreme fidelity is averaged over the four extreme-fidelity metrics.

DatasetCross-View FusionRetraining (Seeds)Ratio
Traffic290.4%17.10%17.0$\times$
Transportation254.2%10.81%23.5$\times$
Temperature145.6%9.62%15.1$\times$
Electricity85.2%4.50%18.9$\times$
Precipitation160.3%14.73%10.9$\times$
ECG38.2%9.15%4.2$\times$
Building22.8%6.10%3.7$\times$
Median145.6%9.62%15.1$\times$
AuthorsOpenReview ↗

Response to Weaknesses 3 (2/2)

(2) Same-view fusion. We consider time--time fusion (Appendix D.6) and frequency--frequency fusion (newly added). As shown in Table R3, they change extreme fidelity by only 11.6% and 11.3% (median), respectively, compared with 130.8% for time--frequency fusion. We further evaluate these changes using Gap / Standard Error. Following the standard two-standard-error criterion [2], values below 2 indicate that the observed change is within random variation. Both same-view fusions remain below 2 on all seven datasets, while time--frequency fusion exceeds 2 on six of seven datasets. This shows that the large extreme-fidelity changes are specific to combining the time and frequency views, rather than a general effect of view fusion.

[2] Larry Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer, 2004.

Table R3. Extreme-fidelity changes under time--frequency and same-view fusion. Gap/SE = Gap / Standard Error.

DatasetCross-ViewTime--TimeFreq--FreqRatio (T)Ratio (F)Cross-View Gap/SETime--Time Gap/SEFreq--Freq Gap/SE
Traffic250.3%11.6%11.3%21.54$\times$22.19$\times$5.600.870.67
Transportation238.8%9.1%6.2%26.28$\times$38.52$\times$19.730.720.61
Temperature130.8%11.7%22.8%11.14$\times$5.73$\times$8.390.780.94
Electricity85.0%11.8%7.7%7.17$\times$11.05$\times$6.861.590.32
Precipitation214.3%11.9%19.3%18.02$\times$11.11$\times$8.850.511.64
ECG45.9%8.0%18.3%5.73$\times$2.51$\times$3.780.821.37
Building22.0%8.3%11.3%2.66$\times$1.95$\times$1.471.171.84
Median130.8%11.6%11.3%11.25$\times$11.57$\times$6.860.820.94

Weakness 3.2: Figure 5 Shows Complementary View Strengths, Not Method Performance

We thank the reviewer for this comment. We agree that the performance of different views can vary across metrics, datasets, models, and other experimental conditions. However, Figure 5 appears in our preliminary analysis and is intended only to provide a representative example showing that the time and frequency views exhibit complementary strengths in extreme generation, thereby motivating our cross-view fusion. Specifically, it does not involve our proposed CrossTF method.

Our paper also provides an extension of Figure 5. Table 9 in Appendix D.4 (Lines 1566–1591) further shows that this complementarity also appears across other datasets, while the specific advantage varies by dataset. For example, the frequency view outperforms the time view on at least one extreme metric in 4 of the 7 datasets. Therefore, CrossTF does not assume a fixed division of roles between the two views, but instead learns dataset-specific fusion coefficients αi\alpha_i.

We also fully agree that the proposed method should demonstrate consistent and non-marginal improvements across datasets. Figure 5 only provides the design motivation, while our main experiments directly verify this requirement. Table 2 in Section 4.1 (Lines 378--390) shows CrossTF outperforms the strongest baseline on all seven datasets, improving extreme fidelity by 5.9%--42.0% (26.1% on average), and wins 730 of 770 pairwise comparisons across 11 baselines and 10 metrics. These results demonstrate that our proposed method achieves consistent and substantial improvements across datasets.


Weakness 3.3 & 3.4: Whether the Effects in Figures 6 and 7 Are Specific to Time--Frequency Fusion

We thank the reviewer for this comment. Figures 6 and 7 are intended to identify the most appropriate fusion stage after time--frequency cross-view fusion has already been established. Therefore, these figures address when to fuse, rather than which views should be fused. The latter has been discussed in our response to Weakness 3.1 and in Figure 5, Table 1, Appendix C.2 (Table 4), Appendix D.4 (Tables 7--9), and Appendix D.6 (Figure 10, Table 11). Specifically, Figure 6 identifies the denoising steps at which the time--frequency discrepancy becomes stable, while Figure 7 further verifies whether this cross-view window corresponds to stronger extreme-fidelity sensitivity.

Regarding whether same-view models can produce similar effects, our original submission already examines this in Figure 10 (Appendix D.6) using the same denoising steps. Specifically, time--time switching remains within random variation on all seven datasets, whereas time--frequency switching exceeds random variation on six of seven datasets. Thus, same-view fusion does not reproduce the substantial extreme-fidelity changes observed in Figure 7.

Reviewer_xGmWOpenReview ↗
Rating
4(+1)
Confidence
4(+2)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CrossTF for extreme-aware time-series generation. It combines time-domain and frequency-domain diffusion models and uses the difference between their denoising predictions to improve extreme fidelity. The authors identify a late-stage denoising window and optimize a small number of fusion weights using an EVT-based objective, while constraining overall distribution quality. Experiments on seven datasets show improvements over 11 baselines.

Critical strengths
  1. The finding that time-frequency differences can affect extreme fidelity much more than the denoising loss is interesting. It provides a useful direction for improving extreme-aware generation without retraining the denoisers.
  2. The experiments are extensive, covering seven datasets and 11 baselines. The results are consistently strong, and the additional ablations help support the proposed design.
Critical weaknesses
  1. The novelty over existing diffusion guidance methods is not fully clear. The fusion in Eq.5 is similar to combining two denoiser predictions, as in prior guidance methods. Although the authors compare with time-time fusion, it is still unclear whether the improvement comes specifically from time-frequency differences or from optimizing a useful direction between two models.
  2. The evidence for the cross-view asymmetry is not fully convincing. Table 1 shows that overall quality can also change substantially, especially on Precipitation and Electricity. The relative range comparison may also be affected by the different scales and baseline values of the metrics. This makes the general claim about preserving overall quality less clear.
Questions for authors
  1. Could the authors compare CrossTF with another guidance or two-model fusion method using the same EVT objective and optimization budget? This would help clarify whether the time-frequency direction provides a specific advantage.
  2. Could the authors provide a more direct comparison of the trade-off between extreme fidelity and overall fidelity when varying the fusion weights? This would help verify the claimed cross-view asymmetry beyond the relative ranges in Table 1.
5 comments
AuthorsOpenReview ↗

Response to Weaknesses 1

We sincerely thank the reviewer for the recognition, support, and constructive feedback. We especially appreciate the positive assessment of our interesting finding on time–frequency differences and the extensive experimental results across seven datasets and multiple baselines. In response to the reviewer's comments, we have added Appendices G–I (Pages 48–52). We respond to each comment below.


Weakness 1: Novelty over Diffusion Guidance and Time–Frequency Fusion

We appreciate this insightful comment. We agree that the fusion formulation of CrossTF (Eq. 5) is algebraically similar to existing guidance methods, as both combine two predictions and adjust the output based on their difference. However, the key distinction lies in the source and meaning of this difference (Table R1).

Classifier-free guidance uses the difference between conditional and unconditional predictions to strengthen conditional generation, while autoguidance uses the difference between a strong model and a deliberately weakened version of the same model to improve generation quality. In contrast, CrossTF combines two denoisers trained with the same data, objective, noise distribution, and architecture, differing only in their input representations. The resulting difference therefore captures how the time and frequency representations model extremes differently, rather than differences in conditioning or model quality. As illustrated in Figure 5, the two views exhibit complementary advantages across different extreme-fidelity metrics, so CrossTF does not assume that one view provides a uniformly better prediction.

Table R1: Comparison of fusion directions in CrossTF and guidance methods.

MethodDirectionSource of DifferencePresumes a Better Model?
Classifier-free guidanceConditional − unconditional predictionConditionYes (conditional)
AutoguidanceStrong − weakened modelModel capacityYes (strong model)
CrossTFTime-view − frequency-view estimateInput representationNo

This distinction has already been discussed in Appendix B (Pages 16–17, Lines 842–894), and we apologize that it was not presented clearly enough in the original manuscript. We have added Appendix G (Pages 48–49, Lines 2575–2598), which states this distinction more clearly and directly. Beyond this distinction, we summarize the core novelty of CrossTF at three levels: conceptual, motivational, and methodological.

(1) Conceptual: the trade-off between extreme fidelity and overall fidelity, and the discovery of Cross-View Asymmetry. To the best of our knowledge, CrossTF is the first method to explicitly formulate extreme-aware generation as improving extreme fidelity while preserving overall fidelity. Toward this goal, we identify an interesting phenomenon, Extreme-Sensitive Cross-View Asymmetry: moving along the time–frequency cross-view direction substantially changes extreme fidelity, while overall generation quality changes only to a limited extent (Table 1). Under the same EVT objective and optimization budget, our response to Q1 further shows that the resulting improvement is specific to the time–frequency direction, rather than arising from optimizing an arbitrary direction between two models.

(2) Motivational: the complementarity of the time and frequency domains in extreme generation. Although time- and frequency-domain representations are information-equivalent under the Fourier transform, generation based on them performs markedly differently on extremes. In our analysis, we observe that the two views represent extremes in structurally different ways. The time view directly represents extreme magnitudes and local changes, whereas the frequency view captures extremes through distributed cross-frequency patterns (Page 3, Lines 156–158). This structural difference gives the two views complementary strengths in representing extremes (Figure 5), which motivates using their discrepancy as an optimization direction.

(3) Methodological: decoupling distribution learning from extreme optimization. Building on these findings, CrossTF separates the two stages. The denoisers of both views are trained with the standard denoising objective and then frozen; extreme fidelity is subsequently improved by optimizing only a small number of fusion parameters along the cross-view direction, under a constraint on overall fidelity. Unlike guidance methods, where the scale is manually set, these parameters are learned through optimization and can either interpolate between or extrapolate beyond the two view-specific estimates.

AuthorsOpenReview ↗

Response to Question 1

Question 1: Comparison with Guidance and Two-Model Fusion under the Same Setting

We thank the reviewer for this constructive suggestion. We fully agree that Appendix D.6 only compares the sensitivity along the time–time and time–frequency directions, and does not demonstrate the genuine advantage of the time–frequency direction through direct performance improvements. Following the reviewer's suggestion, we conduct additional experiments in which all fusion variants are optimized with the same EVT objective, energy-distance constraint, cross-view window, and SLSQP optimization budget as CrossTF. Specifically, we compare CrossTF (time–frequency fusion) with the following methods:

  1. Time–Time Fusion: fuses two independently trained time-view denoisers, testing whether the discrepancy between same-view models can yield comparable improvements.
  2. Frequency–Frequency Fusion: fuses two independently trained frequency-view denoisers, serving as the same-view control for the frequency view.
  3. Autoguidance Fusion: fuses a strong time-view denoiser with a weakened version of it trained for half as many epochs (Time-strong–Time-weak), testing whether a guidance-style strong–weak direction can yield comparable improvements.

In addition, we use the single-model Time-Only and Freq-Only as references, and compute the relative improvement of each fusion method against its corresponding single model. The results are reported in Table R2 and have been added to Appendix H (Page 49, Lines 2600–2617).

Table R2: Two-model fusion under the same EVT objective and optimization budget (lower is better). Parentheses show relative improvements (%) over the reference single model: Time-Only for Time–Time and Autoguidance, Freq-Only for Freq–Freq, and the better single model (†) for CrossTF.

(a) Extreme Fidelity (×10⁻², Geometric Mean of Four Extreme-aware Metrics: Mag-Rate, Chg-Rate, Mag-Excess, and Chg-Excess)

DatasetTime-OnlyFreq-OnlyTime–TimeFreq–FreqAutoguidance (Time-strong–Time-weak)CrossTF (Time–Freq)
Precip.11.08†13.2610.70 (+3.4%)13.27 (−0.1%)11.72 (−5.8%)8.75 (+21.0%)
Build.9.98†13.139.71 (+2.7%)12.35 (+5.9%)9.72 (+2.6%)7.95 (+20.3%)
Temp.10.59†24.6910.59 (+0.0%)23.89 (+3.2%)10.08 (+4.8%)9.43 (+11.0%)
Elect.16.3512.43†14.44 (+11.7%)12.45 (−0.2%)16.34 (+0.1%)6.64 (+46.6%)
Transp.4.90†14.384.81 (+1.8%)14.43 (−0.3%)4.84 (+1.2%)4.28 (+12.7%)
ECG14.13†17.6814.16 (−0.2%)17.44 (+1.4%)13.80 (+2.3%)12.08 (+14.5%)
Traffic2.83†6.822.80 (+1.1%)6.78 (+0.6%)2.88 (−1.8%)2.85 (−0.7%)
Mean––+2.9%+1.5%+0.5%+17.9%

(b) Overall Fidelity (×10⁻³, Geometric Mean of Three Overall Metrics: Wass., Marg., and ACD)

DatasetTime-OnlyFreq-OnlyTime–TimeFreq–FreqAutoguidance (Time-strong–Time-weak)CrossTF (Time–Freq)
Precip.1.4†3.21.5 (−7.1%)3.2 (+0.0%)1.7 (−21.4%)1.4 (+0.0%)
Build.2.5†4.32.5 (+0.0%)4.5 (−4.7%)4.2 (−68.0%)2.4 (+4.0%)
Temp.12.4†20.512.6 (−1.6%)20.4 (+0.5%)15.2 (−22.6%)12.4 (+0.0%)
Elect.13.4†17.813.3 (+0.7%)17.6 (+1.1%)14.2 (−6.0%)13.2 (+1.5%)
Transp.14.8†19.714.6 (+1.4%)19.5 (+1.0%)16.7 (−12.8%)14.4 (+2.7%)
ECG15.6†16.015.7 (−0.6%)15.7 (+1.9%)17.6 (−12.8%)15.6 (+0.0%)
Traffic6.45.8†6.3 (+1.6%)5.9 (−1.7%)6.8 (−6.3%)5.9 (−1.7%)
Mean––−0.8%−0.3%−21.4%+0.9%

The results show that the advantage of CrossTF stems from the discrepancy between the time and frequency views and its cross-view asymmetry, rather than from fusing arbitrary two models. Under the same EVT objective and optimization budget, CrossTF improves extreme fidelity by 17.9% on average and achieves the best results on 6 of the 7 datasets, whereas Time–Time, Freq–Freq, and Autoguidance yield average improvements of only 2.9%, 1.5%, and 0.5%, respectively. Meanwhile, CrossTF keeps overall fidelity stable (+0.9% on average), whereas Autoguidance degrades it by 21.4% on average. This shows that, among the compared directions, the time–frequency direction is the only one that substantially improves extreme fidelity while preserving overall fidelity.

AuthorsOpenReview ↗

Response to Weaknesses 2 & Question 2

Weakness 2 & Question 2: Direct Evidence for Cross-View Asymmetry and the Extreme–Overall Trade-off

We thank the reviewer for raising this important question, as it helps clarify the scope of cross-view asymmetry. In particular, we acknowledge that the degree of asymmetry varies across time-series scenarios. On some datasets, such as Precipitation and Electricity, the frequency-view (α = 0) and time-view (α = 1) estimates differ substantially, leading to large variations in overall quality across α ∈ [0, 1]. However, CrossTF learns α values that largely avoid regions with substantial quality variation. Specifically, on Precipitation, the learned α values range from 0.40 to 1.10, with 5 of the 6 parameters being at least 0.70. On Electricity, they range from 0.44 to 0.85 (Table 10, Page 31). These results show that CrossTF learns fusion weights that help limit changes in overall quality, even when the two views differ substantially.

Moreover, this issue has already been discussed in the original submission. Precipitation is the only dataset whose extreme-fidelity range is smaller than its overall-quality range, because its signal variation is dominated by extremes, so changes in extreme behavior are more directly reflected in the overall distribution (Page 5, Lines 255–257). We also note that datasets with weaker asymmetry may require a greater trade-off between extreme and overall performance (Page 8, Lines 410–413). More importantly, CrossTF explicitly limits the degradation of overall quality through the energy-distance constraint: even on these two datasets, the overall fidelity of the optimized CrossTF remains nearly unchanged relative to Time-Only (+0.0% on Precipitation and +2.3% on Electricity; Table 8, Page 29).

To provide more direct evidence, we further visualize the trade-off between extreme fidelity and overall fidelity as the fusion weight varies, with 14 figures across the seven datasets (Figures 22–35, Appendix I, Pages 49–51). Specifically, we vary a shared α within the cross-view window and evaluate extreme fidelity (geometric mean of the four extreme-fidelity metrics) and overall fidelity (geometric mean of Wass., Marg., and ACD). For each dataset, one plot shows the relative changes of both quantities with respect to Time-Only (α = 1) on a shared axis over the full range of α, removing the effect of metric scales and baseline values, and the other zooms into the neighborhood of α* on the same relative scale. The dotted line marks the α* selected by CrossTF.

The results show that, around the selected α, extreme fidelity changes far more than overall fidelity, supporting the validity of the cross-view asymmetry. On ECG, for example, at α = 0.50, the extreme error decreases by about 8.5%, while the overall error changes by less than 1%. Even on Precipitation, which the reviewer specifically pointed out, the extreme error is much more sensitive than the overall error around the selected α = 0.96: moving α to 1.1 increases the extreme error by about 25%, whereas the overall error increases by only about 4%. The overall error changes more than the extreme error only when α approaches 0, which explains why Precipitation has a ratio below 1 in Table 1. In CrossTF, as shown by the learned fusion parameters, α does not approach this region where overall fidelity changes substantially.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.