- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks whether language-model pretraining contributes to time-series understanding by comparing frozen pretrained models with their randomly initialized versions. The experiments cover multiple aspects of time series understanding: structural recognition, expert routing, numerical forecasting, and question answering. The authors find that pretraining improves process-family routing and numerical forecasting but hurts future-optimal expert selection. On real-data question-answering tasks, pretraining improves updating after evidence edits but reduces preservation of unchanged facts. The authors conclude that pretraining benefits depend strongly on the task and readout interface.
Critical strengths
Controlled comparisons: Comparing pretrained and randomly initialized backbones across ten models, with matched readouts and additional controls, provides useful evidence about the effects of pretraining for the specific task the paper studies which is time series analysis.
Critical weaknesses
This paper seems to be completely written by AI. It is very difficult to read and understand. The contributions and positioning with respect to prior work are unclear to me. Inadequate effort has been made to describe why this question is interesting, or why the analysis is useful. I have concerns about the quality of this paper for ICLR audience: not sure whether people can read and understand this paper well enough to learn something new and build upon it. The paper needs a substantial rewriting to make it readable and suitable for ICLR audience (or any audience for that matter).
Weak absolute performance undermines the conclusions: Despite the reported improvements in updating, the paper's pretrained task-specific readouts achieve only 48.29% accuracy, comparable to the 48.31% majority baseline, with just 8.40% joint success. The 25.74% updating success is also close to the 25% expected from independent random guesses on binary questions. Question-conditioned readouts and native decoding perform even worse. Consequently, the evidence for meaningful time-series understanding remains weak.
Limited generalizability: The forecasting conclusions rely primarily on simple synthetic processes, a fixed expert bank, and small training and test sets. The real-data evaluation contains only 42 test groups and uses artificial constant-offset edits. It remains unclear whether the conclusions generalize to realistic time-series tasks or different experimental choices.
Questions for authors
-
Does the reversal in expert-routing performance persist with different expert banks, larger training datasets, and more realistic time-series distributions?
-
Given that aggregate question-answering accuracy is near or below trivial baselines, what evidence supports interpreting the improved updating scores as meaningful temporal understanding rather than just increased prediction variability?
-
Why is the question considered by the authors interesting? What do their findings teach us and how are these useful to the community? Are any of their findings surprising, or are they expected?