- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MoSiLa, a multivariate time series classifier with two independent routes. The neural route uses an Inception-style convolutional encoder and compresses the feature field into at most 85 landmark tokens. Each token carries a regional mean, a signed temporal moment, a low-rank path displacement and signed area, and geometry. Two relation-aware attention blocks process the tokens, and the q = 1 token is the readout. The pattern route is a HYDRA-inspired random-filter expert with a ridge readout. A fixed rule, chosen from 5-fold out-of-fold training predictions, picks neural-only, pattern-only, or one of three mixtures per dataset. On all 30 UEA datasets the paper reports 78.739% macro-accuracy, the highest among 18 re-run baselines. It also reports ablations, a TRAIN-loss-selected evaluation (76.570 ± 0.439, 3 seeds), and two external datasets.
Critical strengths
The paper reports primary TEST-selected results alongside a TRAIN-loss-selected supplement. It also reports measured versus published baseline scores, all per-dataset results, retuned ablations, and a depth study. This is what makes the evaluation issues below identifiable at all.
Adding a fixed-feature route that is chosen from training predictions helps substantially on some datasets. The neural-only variant A07 loses 22.9 points on EigenWorms and 22.1 on EthanolConcentration. If properly controlled, the finding that learned and random-kernel routes are complementary per dataset would be useful to practitioners.
Critical weaknesses
MoSiLa's checkpoint is chosen by the combined predictor's TEST correct count, with ties updating the checkpoint. Table 7 illustrates the effect: selected epochs are 3 (FD), 3 (AF), and 4 (UW). When selection uses TRAIN loss, the score falls from 78.739 to 76.570 ± 0.439, a 2.17-point drop. The authors also state that earlier TEST access remains part of the development history. So even the 76.57 figure is not an untouched-test estimate.
Table 11 shows all 16 baselines with full-UEA30 references scoring 1.3–6.7 points below their published values (e.g., TimeMixer++ −6.7, MambaSL −6.2, Crossformer −6.0, TimesNet −5.8, HYDRA −5.8). The two baselines with partial reference coverage (Non-stationary Transformer, TimeFilter) are also lower in raw terms, though coverage differs. Published scores for MambaSL (79.819) and ConvTran(79.207) are above MoSiLa's 78.739. Some measured cells look like run failures rather than method weakness. Examples are TSCMamba on SAD(70.987 vs. 95.816 reported) and FEDformer on CT(68.593 vs. 97.284), each worth roughly 0.8–1 pp of that model's average. Measured HYDRA gets 64.885 on EigenWorms, while MoSiLa, which selects its HYDRA-inspired pattern route on this dataset, reaches 97.710, and the MambaSL-table reference for HYDRA is 98.473. Much of the margin over the classical baselines therefore looks like baseline measurement and not architecture.
MoSiLa uses five extra 100-epoch neural fits plus a 500-epoch final fit with patience 50. Most deep baselines use at most 100 epochs with patience 10, reuse recovered configurations without the original hyperparameter searches, and normalize TRAIN and TEST separately. No compute or runtime is reported. The paper states that its comparison is not an equal-budget attribution, but the abstract and conclusion still lead with highest among 18.
The margins over TSLANet and ConvTran do not survive Holm adjustment (adjusted p = 0.074). MoSiLa is top-1 on 11 of 30 datasets. The only TRAIN-based comparison (Table 1) includes only the three classical methods. None of the neural baselines (TSLANet, ConvTran, MambaSL) are re-run under TRAIN-based selection.
The primary results and ablations are single-seed(42). The three-seed SD of the full model is 0.439 pp (SD measured under TRAIN-loss selection, so only indicative), while the drops for removing path, moment, and relation bias are 0.330, 0.460, and 0.551. Their unadjusted paired p-values are 0.369, 0.452, and 0.160. Only A07 (removing the pattern expert) reaches p = 0.020 unadjusted, and that is the route component, not the landmark interface that the paper's contributions emphasize.
A few datasets drive the differences. On SRS1, A00 scores 91.468 while A02–A06 all score 85.666, which alone accounts for ≈0.19 pp (5.80/30) of their drops. FM (63.0 vs. 58–59) adds ≈0.15 pp.
For datasets where A00 selects the pattern-only route (Table 7), the neural ablations should be identical to A00. They are for EW, EP, ER, and SWJ, but not for EC(56.274 vs. 55.133 vs. 57.034), SRS1,or HB. This suggests the gate is re-selected for each variant, so the ablations mix architectural effects with route-selection variance.
There is no pattern-only variant, no fixed 50/50 mixture, and no comparison of OOF-gated selection against always-averaging, although the introduction motivates the gate by claiming that unconditional averaging can dilute the stronger predictor.There is also no ablation of the five candidates or of the fold-variation rule, which the paper itself calls a heuristic.
The retuned ablations (18–35 configurations per variant, search counts unequal) do not say which data the search selected on. If it was TEST, they are optimistically biased. Depth(1/2/3 blocks) also shows large single-dataset swings (PEMS 80.3/89.0/86.7; SRS1 85.7/91.5/85.7) with p = 0.140 and 0.057.
Questions for authors
Which data split did the retuned ablations (Table 9) select on? How were the search spaces and search counts chosen?
What do the A01–A06 variants do with the gate? Is the route re-selected per variant? Why do EC, SRS1, and HB differ across variants when A00 selects the pattern route, while EW, EP, ER, and SWJ do not? What is the selected route for each variant?
What are the seed means and standard deviations (at least three seeds) for A00 and A01–A07 under TRAIN-loss selection, with paired tests, so the landmark-component effects can be compared with seed noise?
Please report a pattern-only variant, a fixed equal-mixture variant and the neural-only variant under the same selection protocol. Does the out-of-fold gate outperform always-averaging?
Please re-run the main deep baselines(TSLANet, ConvTran, MambaSL, TSCMamba) under a TRAIN- or validation-based checkpoint rule with a matched epoch budget, or explain why this is infeasible. Can the anomalous cells(TSCMamba on SAD, FEDformer on CT, TSLANet on IW, where the measured 55.860 contrasts with a reported 10.012 that looks like chance level) be audited?
What is the training and inference cost(wall-clock time or FLOPs) for the full system compared with the baselines? The bounded-token argument covers only part of the cost.
A direct comparison with Rough Transformers is acknowledged as missing. Can you add one, or a signature-only ablation using the same encoder?
On the external datasets, ADFTD seeds range from 44.955 to 57.660(standard deviation 4.59), so the 1.755-point margin over Crossformer is below one standard deviation. FLAAP uses a window-level random split. Do these results support any claim beyond comparable?
What is the route-selection stability across seeds, per-dataset variation in selected routes, and uncertainty in the performance differences?This would help determine whether the learned gate captures repeatable dataset characteristics or responds to small-sample variation.
4 comments
Response to Reviewer Ycj8 (2/2): Baselines, costs and external evaluation
Q5. Strong baselines and the specifically questioned dataset results
The four primary deep baselines now use the same training-side selection protocol as MoSiLa. The paired comparisons favor MoSiLa over ConvTran, MambaSL, and TSCMamba after Holm correction; the TSLANet comparison remains inconclusive (, Tables 1 and 5).
For the questioned dataset results, Table 20 reports 96.771 ± 0.984% for TSCMamba on SAD, 96.309 ± 0.000% for FEDformer on CT, and 58.435 ± 0.916% for TSLANet on IW.
We also clarified the source of 10.012%: it is TSLANet’s InsectWingbeat accuracy in the MambaSL authors’ re-evaluation, reported in Appendix C.1, Table 5(b), p. 21 of their conference paper. It represents a separate evaluation condition.
For EW, the added comparison gives 94.402 ± 1.166% for HYDRA, above 82.697 ± 4.473% for MoSiLa’s pattern expert. These targeted comparisons are reported alongside the primary results.
Q6. Full training and inference costs
For the five-method primary comparison, median end-to-end time is 2,280.2 s for MoSiLa, compared with 1,974.9–1,984.3 s for the four primary baselines.
The accounting includes preprocessing, hyperparameter search, out-of-fold fitting, final fitting, pattern and gate computation where applicable, and evaluation. Tables 11–12 provide the stage breakdown, latency, and memory. MoSiLa’s median inference time is 0.625 ms per sample, with both experts computed.
The 85-token bound limits attention interactions, while MoSiLa’s median end-to-end runtime is higher than those of the four primary baselines under this protocol. Appendix F specifies the accounting scope; independent runtimes for the auxiliary controls are not separately reported.
Q7. Rough Transformers and a same-encoder signature-only comparison
We have included both comparisons. Rough Transformers achieves 76.282 ± 0.802% and is reported descriptively.
The controlled neural-branch comparison gives 77.894 ± 0.450% for the full landmark representation and 76.414 ± 0.684% for signature-only. The paired advantage is 1.479 pp, with a pointwise 95% CI of [0.652, 2.307] and .
Appendix M and Table 21 specify the full 90-coordinate signature, matched encoder architecture and paired initialization, retained attention and readout, and independent end-to-end training. This comparison evaluates the two complete representation constructions; the corresponding results and tests are in Tables 2 and 5.
Q8. External evaluation with subject separation and uncertainty
We have replaced the earlier external comparison with subject-disjoint evaluation against ConvTran. ADFTD uses 54/16/18 training, validation, and TEST subjects; FLAAP uses 4/2/2. All windows from a subject remain together, including in the inner checkpoint and out-of-fold splits. Five fitting seeds repeat training on each fixed partition (Appendix G).
On ADFTD, the paired subject-average accuracy difference is +4.844 pp, with a 95% CI of [0.757, 9.119] and . On FLAAP, the difference is −1.816 pp, favoring ConvTran, with only two held-out subjects.
Inference uses held-out subjects as the paired units after averaging fitting seeds; the five seeds do not increase the number of independent subjects. The evidence supports an advantage on the ADFTD split, while FLAAP provides an unfavorable estimate with limited inferential resolution. Tables 5 and 13 and Figure 5 report these results.
Q9. Stability of route selection across seeds
Under validation-based checkpoint selection, 8 of 30 datasets select the same route for all three seeds. Across the 90 selections, neural-only is selected 48 times, pattern-only 15 times, equal mixing 13 times, conditional mixing 12 times, and constant mixing twice.
Figure 4 displays the three selections for each dataset, and Table 10 provides the detailed routes. We report this variation explicitly because it limits interpreting the selected route as a repeatable characteristic of a dataset.
Thank you to the authors for their rebuttal. However, I still have several concerns:
The headline result (78.739%) remains TEST-selected , while the TRAIN-loss supplement (76.570%) is only reported as a secondary result. The new three-seed TRAIN-loss ablations show no component contrast passing Holm correction, so the landmark interface's contribution is not statistically isolated from noise. The full selectors improvement over equal mixing is only 0.980 pp, and the neural-only comparison is inconclusive.
These unresolved concerns keep the paper below the acceptance bar. I will maintain my score (Rating : 2)
Follow-up on checkpoint selection and component comparisons
Thank you for raising the checkpoint issue. The abstract and Table 1 of the 18 November revision report 78.733 ± 0.854% as the primary result across 30 UEA datasets and three seeds. Hyperparameters and training duration were selected within official TRAIN, and calibration and routing were fixed before TEST scoring. The 78.739% TEST-selected result and the 76.570% TRAIN-loss result are both historical results retained in Appendix I, Table 16. Appendix I also discloses that the UEA TEST sets informed earlier development. Under the revised protocol, switching the checkpoint criterion to TRAIN loss while retaining validation-based hyperparameter selection gives 78.153 ± 1.166% (Table 18). Revised manuscript, Table 1, p. 6
We agree that none of the seven TRAIN-loss ablations passes Holm correction. The most direct evidence for the landmark representation comes from a separate validation-selected comparison in which both models use the neural branch alone. With matched encoder architectures and independent training, the complete landmark representation exceeds signature-only by 1.479 percentage points (pointwise 95% CI [0.652, 2.307], Holm-adjusted , Tables 2 and 5). This comparison tests the two complete representations. Table 17 also reports exploratory neural-only effects for removing the path coordinates and the temporal moment under validation selection. Whether those individual effects persist under TRAIN-loss checkpointing remains unresolved.
We also agree that the routing gain is modest. With the same calibrated experts, selection improves on fixed equal mixing by 0.980 percentage points (Holm-adjusted ). Against the neural expert alone, the difference is 0.839 points (, Table 8). The evidence supports selecting a route over always assigning equal weights. An average benefit from adding the pattern route to the neural classifier remains uncertain.
Response to Reviewer Ycj8 (1/2): Selection and ablations
We appreciate the reviewer’s careful examination of the selection procedure, ablations, and experimental comparisons. The revision includes a common training-side selection protocol, complete three-seed TRAIN-loss ablations, additional routing controls, and subject-disjoint external evaluation. We describe the corresponding evidence below.
Q1. Selection split and search budgets for the retuned ablations
We have withdrawn the original retuning table and its component-ordering claim. That study used 18–35 trials per variant, while the selection split and the reason for the unequal trial counts were undocumented.
In the replacement study, every variant uses the same four learning-rate/weight-decay configurations, batch size 16, and a 200-epoch cap. Candidates are fitted on partition F of official TRAIN. Partition V selects by validation accuracy, followed by negative log-likelihood and fixed tie rules. Appendix B.2 specifies the search and checkpoint procedure, and Appendix I distinguishes this protocol from the historical study.
Q2. Gate reselection for A01–A06 and differences when A00 selects the pattern expert
Yes. Each full-system variant reselects its route using training-side out-of-fold predictions. Consequently, an ablation can select a different route even when A00 selects the pattern expert alone.
For example, on EC at seed 44, A00 selects pattern-only, whereas A05 selects equal mixing. Their reselected-system predictions can therefore differ. With A00’s pattern-only route frozen, both instead use the same pattern predictions and obtain identical results.
Table 10 reports the selected routes, and Figure 3 separates full-system reselection from neural-only and frozen-gate evaluation. For a frozen conditional gate, the fitted parameters remain fixed, while its inputs and resulting mixture weights are recomputed from each variant’s neural predictions. This distinguishes route reselection from changes in the neural branch under a fixed gate function.
Q3. Complete three-seed TRAIN-loss ablations and paired tests
We have added all eight variants on 30 datasets with seeds 42–44 in Tables 18–19. This experiment changes the neural checkpoint criterion to minimum mean TRAIN cross-entropy; hyperparameter selection continues to use TRAIN-internal validation.
The SD summarizes the three seed-specific macro-averages. Tests use 30 paired dataset differences after averaging over seeds, with Holm correction across seven contrasts. None of the TRAIN-loss contrasts passes correction.
Under validation-based checkpoint selection, relation-bias removal is the only full-system component contrast that passes the seven-comparison correction: 1.342 pp, pH=0.020. Separately, Table 15 reports descriptive dataset-exclusion sensitivity for this validation-selected component study: excluding SRS1, excluding SRS1 and FM together, and leaving out each dataset in turn. We have restricted the component conclusions to the conditions in which they are supported.
Q4. Individual experts, equal mixing, candidate selection, and the fold penalty
Using the same calibrated expert pair, Table 8 reports:
The selected system improves over pattern-only by 2.818 pp and equal mixing by 0.980 pp, both with pH=0.026. Its 0.839-pp difference from neural-only remains inconclusive (pH=0.105). These values use Holm correction across the three routing contrasts.
We also report two controls for the selector itself. Restricting the candidates to the two single experts and equal mixing gives 78.667%, a descriptive difference of 0.066 pp from the full selector. Removing the fold-variation penalty gives 78.714 ± 0.941%, a descriptive difference of 0.019 pp.
The penalty is a selection heuristic, not a confidence bound. These controls do not establish a statistically significant benefit from the penalty or the two additional mixture candidates. The principal supported routing result is the improvement over fixed equal averaging.