- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Datasets play a pivotal role in AI development by benchmarking existing algorithms and pushing the field toward more robust models. While the theoretical groundings of FRACTALTS are well-conceived, the manuscript relies too heavily on intuitive motivations. To fully support its claims, the work requires empirical validation showing how this dataset challenges state-of-the-art models in ways that existing benchmarks cannot. I am happy to increase the score if the responses are strongly convincing.
Critical strengths
Current time-series generative models are primarily evaluated using marginal distribution metrics, train-on-synthetic scores, or short-range correlations. FRACTALTS provides a controlled testbed where scaling laws, intermittency, and long-range dependence are known by construction, turning quiet generative failure modes (such as flattening singularity spectra) into quantifiable metrics.
Critical weaknesses
-
Lack of Empirical ML Model Benchmarking and Diagnostic Proof: While the dataset generation and validation pipeline are well-formulated, the paper includes no empirical experiments evaluating existing machine learning models (e.g., generative time-series models like TimeGAN, TimeDiffusion, or structural time-series baselines). To establish the utility of a new benchmark, it is crucial to empirically demonstrate:
- Where current models fail: Showing that state-of-the-art models perform well on standard benchmarks but fail to preserve long-range dependencies, multiscale scaling laws, or multifractal spectrum width () on FRACTALTS. This breakdown is essential to motivate future algorithmic developments.
- Superiority over existing synthetic/real benchmarks: Beyond intuitive motivations, the paper needs concrete empirical comparisons against standard time-series datasets. For instance, quantitatively mapping the range of autocorrelation (short-range vs. long-range) across real-world datasets and showing how FRACTALTS spans these regimes where prior benchmarks fall short.
-
Incomplete Grounding of Real-World Realism and Omission of Related Evaluation Frameworks:
- Real-World Fidelity: The manuscript relies heavily on the mathematical elegance of fractal theory, but lacks explicit empirical validation demonstrating how well these synthetic processes mirror complex real-world dynamics across key application domains (e.g., finance, telemetry, or physiological signals).
- Incomplete Contextualization of Advanced Non-Marginal Metrics: The motivation criticizes standard "marginal distribution metrics" for failing to capture temporal structure. However, the paper overlooks recent non-marginal and dynamic evaluation paradigm, such as geometry-aware metrics designed to detect temporal mode collapse (e.g., A Geometry-Aware Metric for Mode Collapse in Time Series Generative Models, NeurIPS 2025). Clarifying how FRACTALTS complements or surpasses these geometric/topological metrics is necessary to properly position the benchmark's contribution.
-
Absence of Hyperparameter Sensitivity and Robustness Analysis: The parameter generation and MFDFA rejection sampling pipeline rely on several key design choices (e.g., segment lengths, moment orders , sample length , and tolerance thresholds for and ). The authors do not provide a systematic sensitivity analysis evaluating how sensitive the benchmark's properties and computational efficiency are to these hyperparameters. Demonstrating how variation in sequence length () or estimator parameters impacts the yield rate and theoretical guarantees of the accepted sequences is necessary to establish the robustness of the dataset generation protocol.
Questions for authors
Can you include baseline results showing how standard generative models (e.g., TimeGAN, TimeDiffusion)?