- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper addresses imbalanced time-series classification with frozen representations. In diagnostics on 13 binary datasets and four pretrained encoders, the authors show that minority training recall rises while minority test recall falls as minority support shrinks. With the encoder held fixed, a readout trained with more labels achieves higher ROC-AUC. They conclude that the frozen features contain more class information than a readout can exploit under scarce minority supervision.
They propose SCALE:
- LRG: Features are standardized and L2-normalized, then clustered with k-means. Each sample gets a soft local reference and scale from the cluster centers; the signed residual to this reference is projected with shrinkage LDA, and the resulting compact coordinate u is appended to the features.
- SSE: Synthetic minority samples are drawn from a regularized low-rank Gaussian.
- Correction: A class-weighted logistic anchor provides the default prediction. Three experts (logistic on z, u, an RBF-SVM on observed data, and an RBF-SVM on observed plus synthetic data) can switch a prediction to the minority class if at least q of them vote for it, with q selected by training CV.
SCALE is evaluated on 690 univariate and 160 multivariate encoder–task pairs and achieves the best aggregate balanced accuracy, Macro-F1 and Min-F1.
Critical strengths
- Large and careful evaluation: All methods use the same frozen features, splits and tuning budget. Statistics are computed at the dataset level with Holm correction, and all 208 diagnostic conditions are reported, including negative ones (Table 20).
- Transparent recovery–harm analysis: Proposition 1 and Figure 4 show that the growing gain under imbalance comes from increased minority recovery while harm stays roughly constant. This explains how the correction improves balanced accuracy, rather than only reporting that it does.
Critical weaknesses
- It is unclear where the gains come from and under which conditions they hold.
- Imbalance ratio and absolute minority count are confounded, because only the minority class is downsampled. In the Figure 1 example there are only 93 minority samples, and many univariate tasks have just 15–40 at 20:1. In the multivariate benchmark, where several datasets have much larger minorities (e.g. 925 or 534 at 20:1), the gains are smaller (0.69 BA vs. 2.06 univariate), and the Macro-F1 and Min-F1 gains are not significant (Holm-adjusted p = 0.11 and 0.15). Although the datasets differ, this suggests that the claimed dependence on imbalance may partly be a dependence on the absolute number of minority samples.
- The experts are never evaluated on their own: Table 12 always includes the anchor, and even the raw-ensemble baseline is anchor plus RBF expert. It is therefore unclear whether the one-way gating is needed or whether a strong RBF-SVM alone explains the gains.
- Tables 1–3 report only means, without standard deviations or confidence intervals.
- The added complexity is substantial relative to the gain. SCALE requires about 3.2× the mean runtime of logit adjustment, the strongest standard baseline (187.6 s vs. 58.0 s per run, Table 16; about 11× by median). It also requires an eight-parameter search, four classifiers and RBF kernels whose cost grows quadratically with the (synthetically enlarged) training set. In return, it improves on logit adjustment by 1.34 BA points univariate and 0.66 BA points multivariate. It is unclear whether this complexity is justified.
Questions for authors
- Clarification: Can you report standard deviations or confidence intervals across seeds and datasets for Tables 1–3?
- Clarification: How do the experts L, O and E perform as standalone classifiers, and how does a plain majority vote over all four classifiers compare to the one-way correction?
- Future work: SCALE operates on frozen embedding vectors, nothing in LRG, SSE or the experts uses temporal structure. Apart from the chosen datasets and encoders, nothing in this work is specific to time series, have you tested SCALE also on other domains?
- Clarification: How does SCALE behave at a fixed ratio (e.g. 20:1) with a large absolute minority count (e.g. at least 1,000 minority samples)? Can you report per-dataset multivariate results, so that datasets with large minorities (FordChallenge, STEW) can be compared with those with very small ones (LowCost, UCDHE-Rowing)?
2 comments
I thank the authors for their replies. Based on this and the discussion with the others reviewers, I will keep my score.
W1.1 / Q4 — How does SCALE behave at a fixed ratio with a large minority count?
At 20:1, none of the eight multivariate datasets retained 1,000 minority training examples. On the two largest supports, SCALE improved balanced accuracy over cRT by 2.00 points on FordChallenge (925 minority examples) and 2.35 points on STEW (534). At the same ratio, the gains were −0.07 points on LowCost (15) and +1.30 on UCDHE-Rowing-MC (20). At 10:1, FordChallenge and STEW both exceeded 1,000 minority examples, with gains of 1.12 and 1.54 points, respectively.
We agree that our downsampling protocol varies the imbalance ratio and absolute minority count together. These per-dataset comparisons show that gains can persist with larger minority supports. The separate effects of ratio and count remain open. See Appendix H.4, Tables 26–27.
W1.2 / Q2 — How do the standalone experts and plain voting compare with one-way correction?
Thank you for asking how the one-way correction compares with the individual experts and a plain vote. The standalone RBF-SVM comparison is in our response to Reviewer 78zM (W1 / Q1). On the univariate benchmark, L, O, and E individually achieved 75.90%, 73.64%, and 75.90% balanced accuracy, respectively.
We independently tuned a plain vote over the same four classifiers. Each classifier cast one vote, the class with the most votes won, and the anchor broke ties when its prediction was among the tied classes. SCALE improved balanced accuracy from 76.20% with plain voting to 77.38%, a paired gain of 1.18 percentage points (95% CI [0.83, 1.54]). At 20:1, SCALE recovered more minority cases, with lower minority precision and majority recall. See Appendix E.4, Tables 16–17.
W1.3 / Q1 — Can you report uncertainty for Tables 1–3?
Yes. We added 95% source-dataset bootstrap confidence intervals for Tables 1–3. These intervals capture variation across source datasets with the support splits held fixed. We report variation across fitting seeds separately. For SCALE on the univariate benchmark, the SDs across the ten seed averages were 0.09, 0.06, and 0.14 percentage points for balanced accuracy, Macro-F1, and Min-F1, respectively. See Appendix G.2, Tables 21–23.
W2 — Is the additional fitting cost justified?
The cost is substantial. With cached embeddings, SCALE’s full search, refit, and test prediction took about 3.2× as long on average and 11× as long at the median as logit adjustment. The corresponding mean balanced-accuracy gains were 1.34 points on the univariate benchmark and 0.66 points on the multivariate benchmark. The trade-off matters most when a readout needs frequent refitting. When the fitted readout can be reused, the search cost is incurred once per task. See Table 20 for timings and Tables 1–2 for the accuracy results.
Q3 — Have you tested SCALE outside time series?
Our experiments to date cover time-series classification only. SCALE operates on frozen embeddings without direct access to the time axis, so it can in principle be applied to embeddings from other domains. We identify testing that applicability as future work in the revised conclusion.