- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces TimeNet, a unified data standard and infrastructure for foundation models, aimed at addressing issues such as the task-specific nature of data resources, fragmented data formats, and the redundant implementation of data loaders.
Critical strengths
1、The problem addressed is both real and significant. In the time-series domain, common challenges include task-specific data packaging, the redundant repackaging of the same records, and the repeated implementation of data loaders. The argument for decoupling data from tasks is clearly articulated.
2、The data model design is quite comprehensive. Several design elements offer tangible engineering value, including the recursive hierarchy of Records, Sources, and Signals; the support for three distinct types of axes (regular, irregular, and ordinal); a unified approach to defining Spans; the separation of annotation content from its specific occurrences; and the use of task-based references to annotation occurrences, which avoids unnecessary data duplication.
Critical weaknesses
1、The substantive differences between the method proposed in this paper and existing data standards or formats require further clarification. While the paper references works such as Löning et al. (2019), Alexandrov et al. (2020), Herzen et al. (2022), Prabowo et al. (2024), and AtriumDB, the main text fails to sufficiently articulate the unique, indispensable value that "TimeNet" offers over these established standards. For instance, what is the core contribution of TimeNet when compared to general-purpose data standards such as the Arrow/Parquet ecosystem, WebDataset, TensorFlow Datasets, HuggingFace Datasets, or MLCommons Croissant? Furthermore, beyond features like support for multi-modality, diverse tasks, text-based supervision, and a versioned registry, how does TimeNet’s data model differ fundamentally from that of AtriumDB? While using DuckDB combined with Parquet is a logical engineering choice, one must ask: does this constitute a contribution at the level of a "data standard," or is it merely a "specific implementation"?
2、The value of task formalization has yet to be fully validated through experiments. While the concepts of "task signatures" and the classification of tasks into five distinct families are theoretically appealing, the current experiments are limited to QA and reasoning tasks. They fail to demonstrate, for instance, whether a unified collation process—applied to a dataset containing a mix of classification, forecasting, localization, and editing tasks—can truly function without the need for dataset-specific code. Furthermore, it remains unproven whether the proposed partial ordering for task specialization is useful in practical task organization, or whether joint training across different task families yields results that are superior (or at least not inferior) to single-task training. Therefore, it is essential to include an experiment—such as joint training or unified data loading across diverse task families (e.g., classification, forecasting, and localization)—to substantiate that this task framework is more than just a theoretical construct.
3、The empirical evidence for "pooled training" is relatively weak, and the strength of the conclusions should be tempered. For instance, the study covers only four QA datasets and two model families; furthermore, SLIP shows a notable performance decline on the ECG-QA dataset (F1: 33.93 → 30.00; Acc: 40.01 → 31.32), suggesting that pooled training does not consistently yield benefits. Additionally, the study fails to report variance, confidence intervals, or the number of random seeds used, leaving it unclear whether the observed improvements in F1 scores are statistically significant. While it is stated that the single-dataset models and the pooled models employ the "same architecture and training configuration," it remains ambiguous whether the total number of training steps, the total number of tokens, and the data sampling ratios are also identical. If the pooled models were exposed to a larger volume of data, the observed performance gains might stem from the increased data quantity rather than from the development of a "unified representation."
4、Details regarding reproducibility and registry governance are insufficient. For instance, what is the basis for the statistic of 1.51 million task instances? Do figures such as 899k for QA and 484k for classification involve double-counting across different datasets? Furthermore, how are aspects like registry versioning, licensing, access control, data deduplication, and train/val/test splits managed to ensure no data leakage into downstream evaluations? The "local reconstruction" of restricted data relies on connectors; however, how are the maintenance, version compatibility, and evolution of source data handled for these connectors? While anonymized code is provided as supplementary material, the main text fails to address test coverage, schema migration, or backward compatibility strategies.
Questions for authors
see weaknesses.