- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors present FlashRocket, a specialised implementation of MiniRocket, which performs efficient training and inference over multiple datasets/tasks in a single "run" via executing three work streams or queues concurrently on GPU, including (training set) feature extraction, ridge classifier training, and (inference/prediction) feature extraction and classifier inference, following an initial preparation stage performed on CPU. Broadly speaking, the approach involves maximising GPU usage when performing training and inference over multiple datasets.
The authors claim substantial overall speedups in terms of training and inference time over other methods, on a collection of datasets drawn from the UCR (including Redux), UEA, and MultiVerse archives, specifically: a reference GPU implementation of MiniRocket (per tsai), as well as the MantisV2 and MOMENT foundation models. In particular, the method is reported as achieving a ~250x or more speedup over MiniRocket and MantisV2 (MOMENT is considerably slower).
Critical strengths
-
The paper is refreshingly clear and straightforward. It "does what it says on the tin". The paper is well written and easy to follow. The paper provides an accurate, thorough description and explanation of the MiniRocket method, the tsai implementation, and the various components of the proposed method. The results are well organised and presented clearly.
-
The method does provide impressive overall training and inference wall clock time speedups over the baseline methods, even if only in a relatively narrow or niche context. With the caveat in relation to the broader applicability and generalisability noted elsewhere, the proposed method does appear to produce significant overall speedups in terms of wall clock training and inference times over both MatisV2 (and MOMENT), as well as the tsai reference implementation of MiniRocket.
Critical weaknesses
- The broader significance of this approach, it's broader applicability and generalisability, are unclear.
On the one hand, the broader motivation/use case provided by the authors in terms of training/retraining (and performing inference/prediction) for data from, e.g., a large number of disparate sensors, makes sense in isolation. On the other hand, however, in practice the work, as presented, seems quite "niche", and arguably more like a speed-run of prominent time series classification datasets, rather than something that would have broader application. In particular, choices around regularisation would appear to limit the real world application of this approach.
I think the choice of a single alpha (ridge parameter) value for all datasets is strange. I think some reasons why it might still appear to "work" reasonably well are, e.g.: (a) the smaller datasets (most of the datasets included in the study) still benefit from over-parameterisation (i.e., an effect something like double descent), providing something similar to a ridge penalty; and/or (b) the true effect is to some extent "hidden" in the average accuracy metric, i.e., the chosen alpha value is likely "worse" than that chosen by a search on many of the datasets, but also "better" than that chosen by search on many of the datasets (due to tiny training set sizes, distribution shift between training and test sets, etc), so the result on average doesn't look that different (esp. where the chosen alpha value is not radically different from the "true"/search-based alpha, i.e., within the same order of magnitude), which would appear to be consistent with the results presented in Figure 8 (Appendix E).
The two major issues I see with this approach are: (i) shifting the cost of finding the alpha value from a search to a kind of isolated design choice is arguably more of an accounting exercise, it's still a step taken in the method, but it's no longer accounted for in the cost; and (ii) I think there's a significant risk that fitting the ridge parameter to the datasets in this way will not generalise effectively to new data, which would make this work amount to something more like a "speed-run" of the given datasets, rather than a true improvement in efficiency, as impressive as the overall speedups in training and inference time might be.
- My only other substantial criticism is that the related work section appears somewhat perfunctory. I realise that some related work is covered in the introduction but, nevertheless, I think most readers, particularly those not deeply versed in time series or time series classification, really should be given/would benefit from a broader and more thorough treatment of existing work.
Questions for authors
-
At what quantity of data (in terms of the number of training series and/or time series length), if any, does this approach no longer work? That is, what characteristics of the input time series in terms of number of series, channels, time points, would be "too much" in terms of peak CPU and/or GPU memory usage? This might be answered in terms of both the L40S, and a smaller GPU memory limit, say 16 GB. If we were using a GPU with less memory (e.g., 16 GB), which of the datasets (if any) used in the study would no longer be compatible with the approach?
-
If the conclusion really is that a fixed regularisation parameter works for all problems, can they authors please reconcile this with standard results that badly selecting lambda can have substantial negative impact on predictive performance of regressions?
5 comments
Per-dataset results for the 24 additional datasets
As today is the last day of the discussion period, we would like to ask whether our response has addressed your concerns. Thank you for your time and helpful feedback.
We thank the reviewer for their very helpful and thoughtful ideas.
Broader significance
Our broader takeaway is that it is possible to do time series classification at archive-scale in seconds whilst maintaining accuracy. Mantis (2026) describes itself as a “new state-of-the-art” on a subset of our 188 datasets and has 27,520 monthly downloads on HuggingFace [1]. FlashRocket is 298× faster and more accurate, suggesting it could be broadly useful.
Single alpha (ridge parameter)
(i) Our rationale is that alpha wouldn't need to be re-discovered by new users on these 188 datasets, saving about 10 seconds every time. This could apply to thousands of users and repeats. The four-candidate global-alpha search took 16.214 seconds of validation compute on one L40S, which we will add to the appendix.
(ii) The extent to which a fixed alpha generalizes to new datasets is a very interesting question. We agree with the reviewer's two hypotheses and add this analysis:
\text{Dataset} & \text{Training series} & \text{Fixed alpha accuracy (\\%)} & \text{Tuned alpha accuracy (\\%)} & \text{Difference (pp)} \\\\ \hline \text{InsectWingbeat} & \text{25,000} & \text{68.460} & \text{68.472} & \text{-0.012} \\\\ \text{AustraliaRainfall} & \text{112,186} & \text{76.581} & \text{76.656} & \text{-0.075} \\\\ \text{BeijingPM10Quality} & \text{11,918} & \text{82.884} & \text{82.845} & \text{+0.040} \\\\ \text{BeijingPM25Quality} & \text{11,918} & \text{88.174} & \text{88.193} & \text{-0.020} \\\\ \text{WISDM} & \text{11,989} & \text{88.893} & \text{87.483} & \text{+1.410} \\\\ \end{array}$$ For the 5 datasets where the number of training series is greater than the number of features, the fixed alpha was essentially tied with per-task alpha accuracy. This is interesting because our fixed alpha was tuned on equal weight across all 187 datasets (182 which are small), suggesting that a fixed alpha tuned from majority small datasets doesn’t significantly harm larger datasets. Other notes we want to emphasize: 1. The alternative of per-task alpha search can be suboptimal. Since most of the time series datasets are very small, choosing an alpha per hundred series can be noisy. 2. Mean validation accuracy is relatively insensitive to fixed alpha over the tested range 0.1–2.15, varying by only 0.21 percentage points (Table 4, Appendix D). We also tested fixed alpha = 0.464158883 against per-task search on 24 new datasets (3 Redux, 21 Multiverse-Core): $$\begin{array}{llll} \text{Setup} & \text{Time (s)} & \text{Mean accuracy (\\%)} & \text{Our setup speedup} \\\\ \hline \text{Fixed alpha + CG8 (our setup)} & \text{3.199} & \text{76.2961} & \text{--} \\\\ \text{Tuned alpha + exact solver} & \text{12.483} & \text{73.8983} & \text{3.90}\times \\\\ \end{array}$$ Fixed alpha wins on 7, ties on 6 and loses on 11 datasets. This shows how fixed alpha can be a reasonable design choice for new datasets. We do not guarantee that fixed regularisation works for all problems. Practically, fixed alpha is a cheap optimization worth trying first on new datasets (~4× speedup here). If it performs poorly, users can use FlashRocket with per-task search, which finished in 13.55 s on our original datasets. **Related work** We will expand the related work to give a broader explanation of existing time series methods. **Memory scalability** Storing extracted features and associated temporary arrays are major contributors to GPU memory usage. We propose two ideas for memory scalability: 1. Process features in chunks to reduce temporary allocations. 2. Store training features in FP16 instead of FP32, keeping normalization and solver arithmetic in FP32. We tested these in one L40S + 16 CPUs job on all 188 datasets: $$\begin{array}{lllllll} \text{Variant} & \text{Runtime (s)} & \text{Peak GPU allocated (GiB)} & \text{Peak sampled GPU total (GiB)} & \text{GPU total peak reduction (\\%)} & \text{Peak sampled CPU RAM (GiB)} & \text{Accuracy (\\%)} \\\\ \hline \text{FlashRocket original} & \text{3.157} & \text{12.69} & \text{27.10} & \text{--} & \text{13.54} & \text{83.0661} \\\\ \text{FP16 storage} & \text{3.132} & \text{14.73} & \text{34.32} & \text{-26.6\\%} & \text{13.60} & \text{83.0618} \\\\ \text{Chunking} & \text{3.084} & \text{7.51} & \text{15.40} & \text{43.2\\%} & \text{13.71} & \text{83.0588} \\\\ \text{FP16 + chunking} & \text{3.096} & \text{4.62} & \text{12.62} & \text{53.5\\%} & \text{13.74} & \text{83.0954} \\\\ \end{array}$$ Together, these reduce sampled GPU total by 53.5%, with similar observed time and accuracy. The 12.62 GiB sampled GPU total suggests potential for a 16 GB GPU. Future work can further improve memory scalability and test its limits as the number of series, channels and time points increases. [1] [https://huggingface.co/paris-noah/MantisV2](https://huggingface.co/paris-noah/MantisV2)