acceptodds
Under review as a conference paper at ICLR 2027

RTSM: Adapting Recursive Models to Multivariate Time-Series Classification

27 views

Abstract

Accurate multivariate time-series classification requires extracting temporal features and integrating information across a sequence. Weight-tied refinement offers a way to repeatedly process these features without allocating new parameters at each stage. Applying this principle to time series requires choosing both a temporal representation and a latent-state organization suited to classification. We introduce the Recursive Time-Series Model (RTSM), which combines a learned convolutional temporal tokenizer, single-state weight-tied refinement, and a pooled classification readout. Across all 30 UEA datasets, RTSM achieves 72.81% mean accuracy under validation-based configuration and checkpoint selection, the highest observed mean among the 17 evaluated deep-learning models and 1.60 percentage points above MambaSL. Ablations on eight datasets favor the proposed tokenizer over a simple convolutional embedding and the single-state model over the tested two-state variants. The complete classifier reaches 86.10% mean accuracy versus 84.79% for a separately trained tokenizer-only classifier. These findings identify temporal encoding and state organization as key design choices for adapting recursive architectures to time series. The resulting classifier combines competitive accuracy with a shared refinement module whose parameter count is independent of computational depth.

Primary area:
learning on time series and dynamical systems
Keywords:
multivariate time-series classification, recursive neural networks, weight sharing, temporal tokenization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kGpHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper adapts the Tiny Recursive Model (TRM) to multivariate time-series classification. The proposed model, RTSM, has three parts: a) a convolutional tokenizer: channel projection, gated depthwise convolutions, densely connected dilated convolutions, then strided patching; b) a single latent state y, refined by a shared 2-block attention/SwiGLU module with no positional encoding, using 3 cycles per step and 16 steps at inference; c) MambaSL-style weighted-pooling heads for the class and the halt signal. Training follows TRM: deep supervision, gradients only through the last cycle, Stablemax loss, a correctness-based halt head, and parameter EMA.

Under a validation-based protocol, which the authors introduce and use to re-evaluate 16 deep baselines, RTSM gets 72.81% mean accuracy over the 30 UEA datasets, against 71.21% for MambaSL. Under the original MambaSL protocol, which selects on the test set, it gets 80.82% against 79.82%. The ablations use 8 datasets and 3 seeds: a) the proposed tokenizer beats a plain Conv1D tokenizer; b) the single-state model beats the two-state variants; b) the full model beats the tokenizer alone (86.10% vs. 84.79%). An inference-time study shows that extra depth or stochastic breadth changes accuracy very little.

Critical strengths

I find a few points this paper has shown positively.

  1. The paper builds on TRM where it removes one of the two sates and empirically shows that their proposed model RTSM achieves better results (86.10 vs 84.29, tab 2).
  2. Another contri I see the proposed convolutional tokenizer: which combines channel projection, gated depthwise Conv1D, dense dilated Con1D, stried patch projection followed by layer norm. Empircally they show that the tokenizer-only trained model does poor job compared to full RTSM.
  3. Comparison with 30 benchmark dastets is commendable.
Critical weaknesses

The manuscript has several weaknesses are are listed below.

W1. The paper never shows that the recursion itself helps. a) Changing the training depth from 1 to 5 cycles gives 85.60–86.10%, which is within seed noise. b) Training with 1 supervision step instead of 16 costs only 0.12 points. c) At inference, running 64 steps instead of 1 adds about 0.4 points. d) Most of the gain comes from the tokenizer, which raises accuracy from 73.72% to 84.79%. The only evidence that refinement adds anything is the full model beating the tokenizer-only model by 1.31 points. But that baseline has no attention module at all. The gain could come from adding attention and not from recursion. A non-recursive transformer with the same tokenizer, is missing.

W2. Through deep supervision with per-sample trajectory accounting (Appendix A.1), each training sample can receive up to 16 optim updates per epoch. So RTSM’s 100 epochs is substantially more optimization than the baselines’ 100 epochs. RTSM also uses parameter EMA (Appendix C.2) and its own optimizer settings (Table 4), which the baselines do not get. The paper reports no parameter counts for the baselines or the tokenizer, and no FLOPs, training time or inference latency for any model.

W3. The 1.60-point lead over MambaSL (72.81% vs. 71.21%) is not "statistically significant". The one-sided Wilcoxon p-values against the next-best models are 0.238 (MambaSL), 0.087 (TSCMamba) and 0.123 (InterpGN) (Table 7). The 30-dataset benchmark also uses a single seed.

W4. The methodological novelty is limited: RTSM combines established components (a WaveNet/DenseNet-style convolutional tokenizer, a weight-tied attention block with input injection, MambaSL’s pooled readout) and, as Table 1 states, reuses TRM’s training recipe unchanged.

Questions for authors

See the weakness and suggested experiemnta/proofs.

3 comments
AuthorsOpenReview ↗

Response to reviewer kGph

Thank you for separating architecture, recurrence and training effects. We added the single-pass comparison, expanded to three seeds, and revised the accuracy/efficiency claims.

W1: What repeated refinement contributes

The tokenizer-only control also removed attention. New, separately trained attention variants keep the tokenizer, readout and attention architecture fixed:

ConfigurationAccuracy (%), all 30 datasets
Tokenizer-only68.27 ± 0.41
Single-pass attention, H=1, S=172.47 ± 0.08
Three cycles, one step, H=3, S=172.47 ± 0.40
One cycle, adaptive training, H=1, S=1672.06 ± 0.06
Full RTSM, H=3, S=1672.62 ± 0.10

H is cycles per step and S the training step cap. Evaluation uses fixed S steps; both caps are explicitly one for single-pass attention. Entries are mean ± sample SD of three seed-level dataset-macro accuracies. Attention variants have equal parameter counts, but computation and updates differ.

Single-pass attention accounts for 4.20 of the 4.35-point gain over tokenizer-only; full RTSM adds 0.15 points, without a monotonic refinement benefit. The revised abstract, contributions, Section 4.2, Table 4 and conclusion distinguish architectural gains from the limited additional mean benefit of recurrence. This does not establish equivalence between variants.

Across 28 completed datasets, single-state RTSM reaches 73.85 ± 0.26%, versus 70.26 ± 1.18% and 70.60 ± 0.78% for shared-network and separate-network two-state variants. Table 12 separates the original eight from the additional 20 datasets. Both two-state variants failed with CUDA OOM on EigenWorms and StandWalkJump for all three seeds (12 runs); these datasets are excluded from the paired comparison. The separate-network variant retains TRM training, not the full HRM procedure.

W2: Training opportunity and resources

All new exports report 100 completed epochs, but RTSM counts adaptive trajectories with potentially multiple updates each. Equal epochs therefore do not establish equal training opportunity. Batch sizes and optimizer/EMA settings also differ.

ModelMean optimizer updatesParametersTraining timePeak allocated GPU memory
RTSM7,14222.21M544.5 s3.32 GiB
MambaSL9,4331.38M48.9 s0.28 GiB
TSCMamba9,3709.83M164.4 s0.50 GiB
InterpGN9,9170.97M217.4 s3.17 GiB

RTSM makes more updates than MambaSL on 20/30 paired datasets despite its lower overall mean. We do not interpret the mean as less training opportunity. The one-step variants retain 72.47% mean accuracy without repeated supervision within a trajectory. They average 5,000 updates, exceeding MambaSL on 13 datasets, equalling it on seven and using fewer on ten.

Table 2, Section 4.3 and Appendix A.1 report GH200 resources at selected batch sizes. RTSM times training segments including scheduling/logging/EMA; baselines time minibatch updates after data retrieval. Both exclude validation/final testing. These differing scopes do not establish equal-batch inference speed. Weight tying avoids parameter growth with depth, but full RTSM has higher measured mean training cost. We removed speculative throughput claims.

W3: Seeds and statistical significance

The original mean lead did not establish significant superiority. The new all-30 results are RTSM 72.62 ± 0.10%, MambaSL 70.29 ± 0.76%, TSCMamba 69.86 ± 0.31%, and InterpGN 68.76 ± 1.24%. SDs describe the three seed-level macro scores, not confidence intervals.

On one three-seed mean per dataset, one-sided Wilcoxon p-values are 0.1791/0.0184/0.3133; Holm-adjusted values are 0.3581/0.0552/0.3581. None passes 5%. Appendix B.2 specifies the normal approximation, tie correction, exact-zero removal and no continuity correction; Tables 9-10 give tests and per-dataset results. Near-tie checks preserve the conclusion. Full RTSM remains primary.

On the 22 datasets outside early architecture development, mean accuracies are 67.75%, 66.80%, 65.90% and 64.95%, respectively. We report observed pipeline-level differences without claiming significant superiority. Section 4.1 and Appendix A.1 disclose that early design exploration used test feedback on eight datasets.

W4: Scope of novelty

The contribution is the adaptation to continuous temporal inputs and sequence-level classification: local temporal encoding before patching, a single token-aligned state repeatedly conditioned on the input, and pooled prediction. Table 1 and Section 3 distinguish these adaptations from the retained TRM training recipe and MambaSL readout. The expanded results support the temporal encoder, single-state organization and attention processing; further refinement adds little mean accuracy at greater training cost.

Anonymous code and configurations: https://anonymous.4open.science/r/RTSM-ICLR2027/

The authors have made substantial efforts to address my concerns through additional experiments, expanded evaluations, and clearer reporting. In particular, the new single-pass attention comparison is valuable because it separates the contribution of attention-based processing from that of repeated recursive refinement. The results indicate that most of the improvement over the tokenizer-only classifier comes from attention addition, while full RTSM provides only a marginal additional gain (0.150.15\\%). I appreciate that the authors have revised their claims accordingly rather than overstating the contribution of recursion. The expanded three-seed evaluation and Holm-corrected statistical tests also address my concerns about the reliability of the original single-seed comparison. The authors have appropriately refrained from claiming statistically significant superiority over the competing methods. The reporting of optimizer updates, parameter counts, training time, and GPU memory is also substantially improved. Nevertheless, the computational-efficiency claims remain limited because the experiments do not provide controlled inference-latency comparisons or matched computational budgets. Weight sharing establishes parameter efficiency with respect to refinement depth, but does not establish lower overall computational cost. My remaining reservation concerns methodological novelty. The revised manuscript provides a clearer account of the architectural adaptations and additional tokenizer ablations. However, the contribution remains primarily an integration and adaptation of established components. Furthermore, the marginal benefit of repeated refinement raises questions about the importance of the recursive component to the overall contribution. Overall, I appreciate the authors' constructive response and consider the revised manuscript substantially improved. The remaining concerns are primarily about the strength and positioning of the methodological contribution, rather than the adequacy of the newly reported experiments. I leave the final assessment of whether this level of architectural and empirical contribution meets the conference threshold to the area chair.

AuthorsOpenReview ↗

Follow-up: architectural contribution and manuscript revisions

Thank you for the careful reassessment and for recognizing the value of the expanded experiments. We understand that your remaining reservation concerns the significance of the architectural contribution. We have revised the Abstract, Introduction/contributions (Section 1) and Conclusion (Section 5) to explain the design findings and their implications more directly.

RTSM adapts recursive architectures to continuous temporal inputs and sequence-level classification through temporal encoding before patching, one token-aligned latent state, and pooled prediction. Table 1 and Section 3 distinguish these adaptations from the inherited TRM training recipe and MambaSL readout. The contribution combines this task-specific design with evidence about which architectural choices are useful.

Our work also connects to a line of time-series research that develops task-specific architectures from established components. For example, MambaSL (ICLR 2026) revisits the design of single-layer Mamba for time-series classification, while iTransformer (ICLR 2024) reorganizes standard Transformer components around variate representations for forecasting. We position RTSM at this architectural level: adapting recursive models to temporal classification and investigating how temporal encoding, latent-state organization, and refinement depth affect the resulting model. The expanded experiments make the resulting design findings and accuracy–cost tradeoffs explicit, as detailed below.

A useful single-pass configuration. Across all 30 datasets and three seeds, the temporal encoder with one application of the attention module reaches 72.47% mean accuracy, versus 68.27% for tokenizer-only classification and 72.62% for full RTSM. This separately trained variant identifies a strong shallow classifier within the adaptation. Section 4.2, Table 4, attributes the 4.20-point gain to the tested encoder-plus-attention configuration and reports the additional 0.15-point gain from full refinement separately. Seed-0 GH200 profiling gives a median dataset-paired full/single-pass batch-1 latency ratio of 24.87 across all 30 datasets (Appendix B.5, Table 24), quantifying the practical value of this simpler configuration.

A simpler recurrent state organization. On the common 20 additional completed datasets, single-state RTSM reaches 68.96%, versus 64.37% and 65.13% for the shared-network and separate-network two-state adaptations. Appendix B.3.2, Table 14, separates these new results from the original eight and combined 28 datasets. On this same 20-dataset subset, the two-state variants have median dataset-paired batch-1 latency ratios of 4.73 and 4.82 relative to single-state RTSM (seed-0 profiles; Appendix B.5.1, Table 25). State organization and refinement schedules both differ, so this supports the tested single-state configuration as a whole. The separate-network variant retains TRM training and is not a full HRM reproduction.

An encoder choice whose effect depends on the downstream model. Removing gated-block residual connections improves tokenizer-only accuracy by 1.05 points, but reduces single-pass accuracy from 72.47% to 70.83%. Section 4.2 and Appendix B.3.1, Table 13, show why encoder components should be evaluated together with their downstream processor. This is a concrete design finding beyond the aggregate benchmark result.

We agree with your distinction between parameter reuse and execution cost. Section 4.3 and the Conclusion preserve that distinction: weight tying keeps the refinement module's parameter count independent of depth, while full RTSM has higher recorded training cost than the three re-evaluated baselines. The additional 540-run matched-update comparison (Appendix B.2.1, Table 11) retains positive mean accuracy differences, with none significant after Holm correction. It controls update counts, not total computation. Appendix B.5 now reports inference latency and allocated memory with their scope: preloaded inputs, excluded feature generation, and effective TF32 settings that were not logged. The cross-model profiles do not establish matched arithmetic precision or end-to-end speed; incomplete custom-Mamba FLOP counts are not used to rank total compute.

The revised positioning therefore emphasizes what this adaptation makes possible and what its evaluation teaches: strong classification with temporal encoding and shallow attention, a useful single-state recurrent alternative, and a measured assessment of the extra refinement. We hope this makes the architectural and empirical contribution easier to assess.

Reviewer_bn6vOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

PC verification needed regarding desk rejection

The caption font size of every Figure/Table in this paper is approximately 1 pt smaller than the body text, so verification by the PC appears necessary to determine whether the paper complies with the ICLR format. Separately from this issue, however, I provide my comments and evaluation of the research content below.


This study applies a recursive reasoning model (RRM) backbone architecture, in which each block shares weights and the output format, to time series classification (TSC). In particular, since TSC is a many-to-one task that predicts a single class label for an entire time series, a central question is how TRM was adapted to achieve high performance on such a task.

I think the architecture of this model is sufficiently novel and an appealing design. However, I find it quite unfortunate that this novelty does not come through clearly in the paper's writing or subsequent analyses. For example, based on what I have found, there are very few cases of applying this kind of RRM architecture to time series benchmarks that are not restricted to a particular domain, yet the paper describes this aspect rather vaguely. Reading only the Introduction and Method, it feels as though the method simply combines convolution token + TRM feature extraction + MambaSL readout, which instead diminishes the appeal of the design. Although the tokenizer combining gated depthwise Conv1D, dense dilated Conv1D, and other components is also a design proposed in this study, the role of each component and the reasons for choosing this combination are not sufficiently explained.

In my personal view, the defensive, reviewer-conscious writing sometimes seen in AI-assisted writing seems to have made the strengths of this study less apparent. The subsequent analyses also feel limited to checking how performance changes across different settings, rather than explaining the advantages that the proposed model's structural characteristics offer for TSC. I also find it unfortunate that a substantial portion of the main pages is devoted to listing these results. If these aspects are improved and the reproducibility of the experiments is more clearly ensured, I think I could recommend acceptance.

Critical strengths

[S1] Model design for applying an RRM architecture to TSC

  • I find this study interesting in that it applies a recursive reasoning architecture to TSC, a many-to-one task, and constructs a tokenizer and readout for this purpose. In particular, I consider it an appealing design that goes beyond using the existing TRM architecture as is and introduces gated depthwise Conv1D, dense dilated Conv1D, and other components to extract temporal features from time series.
  • The large performance difference in Table 2 when the simple Conv1D tokenizer is replaced with the proposed tokenizer also shows that this design plays an important role. However, as discussed in the weaknesses below, the contribution of this study would have been much clearer if the authors had also analyzed which components within the tokenizer contributed to this improvement.

[S2] Comprehensive evaluation under a validation protocol

  • I appreciate that the authors reevaluate various baseline models within similar hyperparameter search spaces and apply a protocol that selects models based on validation. I think these experiments involve considerable effort toward a fair comparison, rather than simply comparing against performance reported in previous papers.
  • In particular, this is meaningful in that it resolves concerns about data leakage in TSLib-based experiments and provides baseline results that future studies can reference. However, for these strengths to be fully recognized, the actual split and normalization procedures should be clearly explained, and reproducibility should be supported by source code and checkpoints.
Critical weaknesses

[W1] Insufficient experimental scope and reproducibility

  • Since I currently cannot access the source code, it is difficult to verify how the validation protocol and hyperparameter search presented in the paper were actually implemented. In particular, it is unclear how the stratified 8:2 split was applied to every dataset. For datasets with very few training samples relative to the number of classes (e.g., SWJ), it may be difficult to include sufficient samples from each class in the validation set, so the handling of these cases should be explained or verifiable through the code. The patch lengths in Appendix A.3 also include values up to 400, but it is unclear how these settings were handled for datasets with shorter time series.

  • Furthermore, the paper reads as though the comparison was based on 240 settings, but the ablations suggest that additional hyperparameter and architecture searches were conducted for the proposed model. It is necessary to distinguish whether these experiments were performed for analysis after the final design had been determined, or whether they were also used to select the final model or hyperparameters. If the additional search informed model selection, the actual search budgets for RTSM and the baselines, including this search, should also be provided.

  • I also have reservations about the comparison methods and experimental scope. Although an RRM is proposed as the backbone, the main comparison does not include RRM-family baselines, making it difficult to distinguish the performance attainable simply by applying an existing recursive model to TSC from the performance attributable to RTSM's changes. The ablations include comparisons with TRM-style dual-state and HRM-style separate-network variants, but these are limited to eight datasets. I think these comparisons should be extended to the full benchmark, and the authors should clearly explain which elements of the original TRM/HRM each variant retains or changes.

  • In addition, the rationale for selecting the eight UEA datasets used in the ablations is unclear. It is therefore difficult to judge whether the results would hold on other datasets, and without selection criteria, the subset could be read as having been chosen to favor performance, regardless of the authors' intent.

[W2] Insufficient follow-up analysis of the model architecture

  • My main reservation about this study is that, compared with its demonstration of performance improvements, the analysis of why this model is effective for TSC is insufficient. Table 2 shows an average accuracy of 84.79% even when only the proposed tokenizer is used, increasing to 86.10% with the full model. In contrast, learned pooling (86.10%) is not better than average pooling (86.14%), and the average accuracy appears to remain almost unchanged when the depth or state-passing scheme is changed. Taken together, I think these results require a clearer interpretation of which characteristics of the RRM architecture contribute to the performance improvement.

  • In particular, despite the substantial performance difference between tokenizers, the current comparisons do not reveal which internal designs are important. I would like the authors to partially ablate gated depthwise Conv1D, dense dilated Conv1D, gates, residual connections, and other components, and connect the architectural differences between TRM and RTSM summarized in Table 1 to experimental evidence. Furthermore, a comparison with a non-recursive model under a controlled computational budget, while keeping the tokenizer and readout fixed, would help clarify the effect of the recursive architecture itself.

  • There appears to be sufficient space in the main pages to strengthen these analyses and explanations. Overall, whitespace and figures occupy a substantial amount of space, while much of the key information has been moved to the Appendix. In particular, Section 4.3 summarizes various Appendix results in prose, giving the impression that information that could be communicated more clearly through figures or tables is repeated in text, and it is also unclear which results in Appendix B.4 should be consulted. At present, it feels as though the authors are trying to mention all the conclusions of the various Appendix experiments in a compressed form in the main text, making it difficult to identify the key points first.

  • Conversely, I do not think settings identical to those in previous work need to be repeated at length. Since Appendix A.3 states that MambaSL's search spaces were retained, it would be better to summarize unchanged baseline settings through citations and focus on RTSM's settings and changes. Table 6 could also communicate the results more clearly by concisely presenting baseline results that overlap with MambaSL and separating RTSM's full results from the comparative metrics and statistics.

  • In addition, the authors need to clearly distinguish the parts inherited from MambaSL from those newly proposed in this study, and provide an appropriate citation for the TSLib pipeline underlying both studies.

Questions for authors

The following questions are based on the weaknesses above.

[Q1] Could you provide the source code and settings needed to reproduce the experiments?

  • In addition to the model code, it would be helpful to provide the experimental scripts, including those for the baselines, the data splitting and normalization code, the hyperparameter search settings, and the final selected configurations. It would be even better if checkpoints that allow the results to be reproduced were also made available in the future, as in MambaSL.

[Q2] I would like to know how the stratified 8:2 split was performed on datasets with few training samples.

  • For example, one-fifth of the entire training set of StandWalkJump corresponds to 2.4 samples, whereas there are 3 classes. How were the actual validation set size and the number of samples per class determined? Were there cases in which some classes were not included in the validation set?
  • It would be helpful to explain the actual train/validation split sizes and per-class sample counts, as well as rounding and exception-handling rules, including those for other datasets, or to provide source code through which these can be verified.

[Q3] Could you clearly explain the actual hyperparameter search scope and final model selection process?

  • Beyond the 240 settings presented for RTSM, were the settings evaluated in the ablations used to determine the final model or hyperparameters? It would be helpful to distinguish experiments conducted for analysis after the final design had been determined from those also used for design selection. In the latter case, I would like to know how the search budgets of RTSM and the baselines differed when this additional search is included.
  • Also, the patch lengths in Appendix A.3 include values up to 400. How were settings in which the patch length exceeds the maximum sequence length handled? I would appreciate an explanation of whether these experiments were excluded, whether padding or other processing was applied, or whether other patch lengths not listed were used.

[Q4] What is the rationale for selecting the eight datasets used in the ablations?

  • It would be helpful to explain the criteria for selecting CT, DDG, HB, JV, LSST, NATO, PD, and SAD, and when this selection was made. I would like to know how this subset represents the diverse data characteristics within UEA and whether the trends of the main ablations hold on other datasets.

[Q5] Could you provide comparisons that separate the effect of existing recursive models from the effect of RTSM's changes?

  • I noted the comparisons in Table 2 with a TRM-style dual-state/shared-network configuration and a separate-network variant. It would be helpful to clearly summarize which elements of the original TRM/HRM architectures and training methods each variant retains or changes, and to extend the main comparisons to the full benchmark. In particular, I would appreciate clarification of which current configuration corresponds to retaining as much of the original TRM architecture as possible while adapting only the input and output components needed for TSC.
  • Furthermore, considering the results for the tokenizer-only model, recursive depth, state passing, and pooling together, a clearer explanation seems necessary regarding which characteristics of the recursive architecture, beyond the tokenizer, contribute to performance. A comparison with a non-recursive model using the same tokenizer and readout under a controlled computational budget would help distinguish the effect of the recursive architecture itself.

[Q6] Could you provide additional ablations of the internal components of the proposed tokenizer?

  • Given the large performance difference in Table 2, I consider the tokenizer to be one of the main contributions of this study. I would like to identify which elements have the greatest impact through results obtained by individually removing or simplifying gated depthwise Conv1D, dense dilated Conv1D, gates, residual connections, and other components.
  • If Table 8 already includes these comparisons, I would appreciate it if you could specify which components each configuration changes.

[Q7] What is the computational cost relative to the baselines at the same batch size?

  • Separately from the weaknesses, I would like to know how the parameter efficiency provided by weight sharing translates into actual computational cost. It would be helpful to provide comparisons of memory usage and training/inference time against the baselines under the same hardware, batch size, and input conditions.
  • In particular, understanding how accuracy and cost change with the number of recursive iterations would help clarify the advantages and disadvantages of this architecture.
6 comments
AuthorsOpenReview ↗

Response to reviewer bn6v (1/2)

Thank you for recognizing the architectural potential and explaining where the presentation obscured it. We have clarified the time-series adaptation, expanded the experiments beyond the original eight datasets, and made the implementation available. This first part addresses the architectural analyses; part 2 addresses reproducibility, selection, cost and presentation.

W2 and Q5: Architectural contribution and recursive baselines

RTSM adapts recursive models to continuous temporal inputs and one class label per sequence. Its temporal encoder builds local context before patching; one token-aligned state integrates information across tokens while retaining input injection; pooled heads produce sequence-level predictions. The contribution is this adaptation and its empirical characterization, building on TRM's refinement/training recipe.

The new 30-dataset, three-seed results separate these choices from repeated refinement:

ConfigurationAccuracy (%)
Simple convolutional embedding, direct classification60.19 ± 0.46
Proposed tokenizer, direct classification68.27 ± 0.41
Proposed tokenizer + single-pass attention72.47 ± 0.08
Proposed tokenizer + three cycles, one step72.47 ± 0.40
Proposed tokenizer + one cycle, up to 16 training steps72.06 ± 0.06
Full RTSM72.62 ± 0.10

The tokenizer improves on the simpler embedding by 8.09 points, computed before rounding. Single-pass attention adds 4.20 points; full RTSM is 0.15 points higher. The attention variants share the same architecture and parameter count and are trained separately. Their training and computation budgets are not matched. The new single-pass control explicitly uses one training and one evaluation step. The revised abstract, contributions, Section 4.2, Table 4 and conclusion distinguish these architectural gains from the modest additional mean benefit of repeated refinement.

We also extended the state comparison:

State organizationOriginal 8Additional 20Combined 28
Single state86.10 ± 0.3168.96 ± 0.2873.85 ± 0.26
Two states, shared updates84.98 ± 1.0464.37 ± 1.4270.26 ± 1.18
Two states, separate networks84.29 ± 1.2665.13 ± 0.6170.60 ± 0.78

Entries are mean ± sample SD of three seed-level dataset-macro accuracies. The shared-network variant retains TRM-style auxiliary/prediction-state updates within the adapted time-series pipeline. The separate-network variant adds HRM-style network separation while retaining TRM training; it is not a full HRM reproduction. The released E03 configurations specify H=3, L=6, one gradient-tracked high-level cycle and fixed 16-step evaluation. Appendix B.3.2 and Table 12 distinguish historical, new and combined results.

The single-state advantage extends to the additional 20 datasets. Both two-state variants failed with CUDA OOM on EigenWorms and StandWalkJump for all three seeds, giving 12 unsuccessful primary runs. The paired comparison uses the common 28 completed datasets, without assigning zero accuracy to failures. Smaller-batch retries did not finish within the available time. These are outcomes for the attempted settings, not evidence that all configurations would fail.

W2 and Q6: Tokenizer components

We added the requested component removals over all 30 datasets and three seeds (Appendix B.3.1, Table 11):

Tokenizer-only variantAccuracy (%)Change
Full tokenizer68.27 ± 0.41reference
Without gated-block residual connections69.32 ± 0.19+1.05 pp
Without dense connectivity68.92 ± 0.56+0.64 pp
Without gates68.84 ± 0.82+0.56 pp
Without dilation68.04 ± 0.31-0.24 pp
Without gated stage67.98 ± 0.28-0.29 pp
Without dilated-convolution stage65.09 ± 1.07-3.18 pp

Changes use unrounded values. Removing the whole dilated-convolution stage has the largest mean cost among these removals; removing dilation alone has a much smaller effect. Thus, that stage's benefit cannot be attributed to dilation alone. Several removals improve tokenizer-only accuracy, so the results do not imply that every component is individually necessary.

The residual effect depends on the downstream model: with single-pass attention, removing the gated-block residual connections reduces accuracy from 72.47 ± 0.08% to 70.83 ± 1.17%. The tokenizer-only improvement therefore does not transfer to this combined model. Section 4.2 highlights this interaction without extending it to untested recurrent/component combinations. The original pooling comparison also remains explicit: mean pooling gives 86.14% versus 86.10% for learned pooling, with no demonstrated accuracy advantage for learned pooling.

Anonymous implementation and configurations: https://anonymous.4open.science/r/RTSM-ICLR2027/

AuthorsOpenReview ↗

Response to reviewer bn6v (2/2)

Part 2: reproducibility, selection, cost and presentation.

W1 and Q1: Anonymous implementation

Code: https://anonymous.4open.science/r/RTSM-ICLR2027/ . The release includes RTSM/baseline implementations, benchmark/ablation/rebuttal configurations, launch scripts, preprocessing and checkpoint-selection code, training-cost measurements and selected inference summaries. The README maps files to experiments. Checkpoints and raw inference trajectories are not included.

Appendix A.1 distinguishes original RTSM runs, which mixed normalization fitted on full official TRAIN before splitting and on the training subset after splitting, from the new split-first runs. The adapted MambaSL loader standardizes partitions independently, with model-specific processing where applicable. These are complete-pipeline comparisons, not architectures under identical preprocessing.

Q2: Tiny stratified splits

_split_train_val_dataset in rtsm/pretrain_slot_buffer.py implements the classwise split with seed 2021. For n >= 2 examples in a class, the validation count is min(n-1, max(1, round(0.2*n))); singletons remain in training. Allocation uses Python rounding and seeded shuffling, so overall proportions need not be 80/20. StandWalkJump has nine training and three validation examples, one validation example per class. Singletons can be absent from validation. Tiny validation sets remain unstable; training-seed repeats do not measure split sensitivity.

Q3: Search and development history

The nominal 240-setting grid is 12 patch lengths x five strides x four batch sizes, not a completed-run count or the full development budget. Early architecture exploration used eight datasets and initially their test accuracies. Validation-based selection was developed subsequently; the tokenizer and single-state design were fixed before the archive-wide sweep. Section 4.1 and Appendix A.1 disclose this chronology. Retaining MambaSL search spaces does not establish equal total tuning effort.

The unpadded projection requires P <= T after preprocessing. The distributor can exclude oversized patches using --prune-invalid-pl; the tokenizer also rejects them. Pruning is not automatic in the sequential runner. A complete historical attempted/failed/completed-run ledger is not included, so the candidate grid is not such a ledger.

Q4: The eight development datasets

CT, DDG, HB, JV, LSST, NATO, PD and SAD were chosen early for affordable exploration across varied lengths, channels and training sizes. They were a development subset, not an independent or random evaluation sample.

The new tokenizer/refinement study covers all 30 datasets. On the other 22, the tokenizer gains 7.01 points over the simple embedding and single-pass attention adds 5.31 points over tokenizer-only classification. Full RTSM obtains 67.75 ± 0.11%, versus 66.80 ± 0.54% for MambaSL, 65.90 ± 0.77% for TSCMamba and 64.95 ± 1.31% for InterpGN. These analyses extend the evidence beyond the original eight; they are not a newly constructed untouched holdout.

Q7: Measured training cost

ConfigurationParametersTimePeak allocated GPU memory
Tokenizer-only15.39M*284.8 s1.86 GiB
Single-pass attention22.21M348.0 s3.29 GiB
Three cycles, one step22.21M379.8 s3.32 GiB
Full RTSM22.21M544.5 s3.32 GiB
MambaSL1.38M48.9 s0.28 GiB
TSCMamba9.83M164.4 s0.50 GiB
InterpGN0.97M217.4 s3.17 GiB

Tokenizer-only excludes an unused 6.82M-parameter refinement module; the instantiated model has 22.21M. Timing/memory are observed implementation values without subtraction.

These are dataset/seed means on GH200 GPUs at selected batch sizes. RTSM times training segments including scheduling/logging/EMA; baselines time minibatch updates after data retrieval. Validation and final testing are excluded. These scopes differ, and the measurements do not supply controlled equal-batch inference timings. We removed speculative throughput claims.

All runs report 100 completed epochs, but this is not a matched budget. RTSM averages 7,142 updates versus MambaSL's 9,433, yet exceeds it on 20/30 datasets. Weight tying avoids parameter growth with depth; measured full-model training cost exceeds these baselines.

Presentation, attribution and captions

The revision foregrounds the two adaptation questions, separates historical/new experiments, adds accuracy/resources in Table 2 and depth controls in Table 4, and gives precise Appendix B.4 pointers. Table 1 and Section 3 distinguish the retained TRM recipe and MambaSL readout. Appendix A.1 credits TSLib through its requested TimesNet citation alongside MambaSL. Figure/table captions now use the 10 pt body size, and the checked revision has 10 main-text pages.

Reviewer_bn6vOpenReview ↗

Response to the Authors' Rebuttal (1/2)

Thank you for the detailed responses and extensive additional experiments. Below are my comments on each point.


Architectural analyses

[Q5] Effects of the architecture and recursion

The single-pass attention comparison helped distinguish the effect of attention from that of repeated refinement. The limited additional gains from repeated refinement are also appropriately reflected in the revised manuscript.

However, I see a missed opportunity in how the paper frames its contribution. Had these controlled ablations been conducted earlier, the paper could have centered its contribution on the strong performance of a simpler, non-recursive model against the existing Transformer-based time series baselines. This could have conveyed a clearer message that shallow networks can perform well in TSC since CNN tended to be better than Transformers in this field. The strong single-pass result seems valuable in its own right, beyond its role as an ablation of a recursive model.

Furthermore, the UEA datasets are very small compared with more recent benchmarks, and in my experience, stacking more layers often substantially reduces performance in this setting. From this perspective, the fact that RTSM preserves and slightly improves upon single-pass performance while executing 96(=2x48) layer applications with shared weights suggests potential for greater computational depth in TSC. This is, of course, my personal interpretation.

I have one additional question. Given the substantial improvement from the tokenizer, are there specific aspects of its design that account for the characteristics of TRM or Transformers with shared parameters, or should it be understood as a temporal encoder that could also be applied to other TSC backbones?

[Q6] Tokenizer component analysis

It is interesting that removing the entire dilated-convolution stage causes the largest performance drop, while the effect of the residual connection depends on the subsequent architecture. In particular, the finding that residual connections are not always beneficial also seems worth considering alongside the discussion of skip connections in MambaSL.


Reviewer_KtopOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how TRM-style recursive refinement can be adapted to multivariate time series classification, focusing on temporal representation and latent state organization. RTSM uses a convolutional tokenizer to extract temporal features, then repeatedly updates a single token aligned latent state through a shared attention–MLP module, reinjecting the encoded input at each cycle. Pooled heads produce sequence level predictions and halting scores, while training retains TRM's deep supervision and adaptive halting. Experiments on 30 UEA datasets report the highest observed mean accuracy among the evaluated deep learning models.

Critical strengths
  1. The paper applies TRM style refinement to multivariate time series classification. Its combination of temporal tokenization, single state refinement, and sequence level readout provides a clear architectural adaptation while preserving parameter sharing across refinement depth.
  2. The paper reevaluates a broad set of deep learning baselines across the full UEA archive using validation based model and checkpoint selection. This provides a empirical reference.
Critical weaknesses
  1. Although RTSM achieves the highest mean accuracy on the UEA benchmark, its improvement over the strongest baselines is relatively small. This makes its empirical advantage over existing methods less convincing.
  2. The contribution of recursive refinement remains unclear. Ablations show comparable performance without state carry or with reduced refinement depth, while additional inference steps bring limited improvements. The tokenizer only comparison also does not isolate the benefit of recursion from other architectural and training changes.
Questions for authors
  1. What inference depth and state reset rules are used in the ablations involving fewer supervision steps, reduced refinement depth, and state resets? Does the single cycle state reset variant effectively operate as a single pass classifier?
  2. Which normalization procedures were used for RTSM and the baselines in the primary benchmark? How were the different normalization schemes assigned across datasets? Could these potential differences explain the performance gain reported in the main benchmark?
  3. How do the total optimizer update counts and computational costs of RTSM compare with those of the strongest baselines, given that RTSM performs an update after each supervision step?
  4. Could the authors evaluate the tokenizer only classifier and a comparable (e.g., matched in parameter) single pass attention model across all 30 UEA datasets under matched training and preprocessing conditions?
3 comments
AuthorsOpenReview ↗

Response to reviewer Ktop

Thank you for asking us to separate the accuracy gains. We expanded the tokenizer/refinement study to all 30 UEA datasets with three seeds, added three-seed results for three baselines, and revised the attribution to recurrence.

W1: Accuracy and statistical reliability

The new comparison supplements the original 17-model, single-seed benchmark:

ModelAll 30: accuracy (%)Other 22: accuracy (%)
RTSM72.62 ± 0.1067.75 ± 0.11
MambaSL70.29 ± 0.7666.80 ± 0.54
TSCMamba69.86 ± 0.3165.90 ± 0.77
InterpGN68.76 ± 1.2464.95 ± 1.31

Entries are mean ± sample SD of three seed-level, equally weighted dataset averages. SDs describe seed variation on fixed splits, not confidence intervals. Other 22 excludes the eight development datasets.

RTSM has the highest observed mean in both scopes. The all-30 gains over the baselines are 2.33, 2.77 and 3.86 percentage points. Differences vary across datasets: excluding LSST reduces the mean advantage over MambaSL to 1.11 points. One-sided Wilcoxon tests on 30 paired dataset-level three-seed means give p = 0.1791/0.0184/0.3133 and Holm-adjusted p = 0.3581/0.0552/0.3581. None passes the corrected 5% threshold; we claim competitive mean accuracy, not significant superiority. Table 2 and Appendix B.2, Tables 9-10, report the results, per-dataset values and test specification.

W2 and Q4: Comparable single-pass attention

The tokenizer-only control also removes attention. We therefore added separately trained attention variants with the same tokenizer, readout, attention-module architecture and parameter count:

ConfigurationCycles per step HTraining step capFixed evaluation stepsAccuracy (%)
Tokenizer-onlyn/a1168.27 ± 0.41
Single-pass attention11172.47 ± 0.08
Three cycles, one step31172.47 ± 0.40
One cycle, adaptive training1161672.06 ± 0.06
Full RTSM3161672.62 ± 0.10

The new variants share split-first normalization and a fixed validation split. Single-pass attention accounts for 4.20 of the 4.35-point gain over tokenizer-only classification. Full RTSM adds 0.15 points, without a monotonic refinement benefit. This supports the adapted encoder-plus-attention architecture; it establishes neither the necessity of repeated refinement nor equivalence between variants. The abstract, contributions, Section 4.2, Table 4 and conclusion reflect this attribution. Computation and update budgets are not matched.

Q1: Inference depth and resets

One cycle applies the shared two-block module once; one step contains H cycles. The new attention variants above evaluate 1, 3, 16 and 48 module applications, respectively, with learned halting disabled. Both training and evaluation caps are explicit in the released configurations. With state carry, the detached state continues between training steps. A no-carry variant reinitializes each step: under fixed weights, deterministic inputs and the same reset, repeating a one-cycle computation cannot accumulate refinement. Repeated supervised updates can nevertheless train different weights. The new H=1, S=1 control addresses the single-pass question directly, without inferring its behavior from the historical reset ablations.

Q2: Normalization

The original RTSM benchmark mixed normalization fitted on the training subset after splitting and on the full official TRAIN partition before splitting. The latter includes validation inputs; neither fits the RTSM normalizer on TEST. New RTSM runs use split-first fitting. The adapted MambaSL loader standardizes TRAIN, VAL and TEST independently. Appendix A.1 clarifies these complete-pipeline comparisons; they do not isolate preprocessing as a contributor to cross-model differences.

Q3: Updates and cost

ModelMean updatesParametersTraining timePeak allocated memory
RTSM7,14222.21M544.5 s3.32 GiB
MambaSL9,4331.38M48.9 s0.28 GiB
TSCMamba9,3709.83M164.4 s0.50 GiB
InterpGN9,9170.97M217.4 s3.17 GiB

All report 100 completed epochs, but RTSM counts adaptive trajectories. It makes more updates than MambaSL on 20/30 datasets despite its lower overall mean. Batch sizes, work per update and optimizer/EMA settings differ. GH200 timings use selected batch sizes: RTSM measures training segments, including scheduling/logging/EMA; baselines measure minibatch updates after data retrieval. Both exclude validation and final testing. These are not controlled inference timings or matched training budgets. Weight tying prevents parameter growth with depth, not repeated computation cost.

Anonymous code and experiment configurations: https://anonymous.4open.science/r/RTSM-ICLR2027/

AuthorsOpenReview ↗

Additional comparison with matched optimizer-update counts

Thank you for highlighting optimizer-update counts as an important part of the comparison. We have now matched the baselines to the RTSM update counts separately for each dataset and seed. The observed mean accuracy advantages persist under this control, although none reaches significance after Holm correction. Section 4.1 and Appendix B.2.1, Table 11, report the experiment.

MambaSL, TSCMamba and InterpGN were retrained at two reference budgets: single-pass RTSM (H=1, S=1) and full RTSM (H=3, S=16). All 540 final runs, covering 30 datasets, three seeds, three baselines and two budgets, reached the corresponding RTSM run's exact update count. Baseline architectures and selected batch sizes were retained.

RTSM referenceMatched baselineRTSM accuracy (%)Baseline accuracy (%)Difference (pp)Holm-adjusted p
Single-passMambaSL72.4770.33+2.140.0554
Single-passTSCMamba72.4769.71+2.760.1156
Single-passInterpGN72.4768.64+3.830.1422
Full RTSMMambaSL72.6270.18+2.440.1766
Full RTSMTSCMamba72.6270.00+2.630.1467
Full RTSMInterpGN72.6269.03+3.590.3057

Accuracies average the three training seeds and weight all 30 datasets equally. Differences and one-sided Wilcoxon signed-rank tests use unrounded dataset-level seed means. Holm adjustment covers the three baseline comparisons separately for each reference budget. Appendix B.2 specifies the test convention; none passes the corrected 5% threshold.

This additional comparison addresses the update-count concern directly. It supports positive observed mean accuracy differences at equal update counts, while batch sizes, sample exposure, work per update, optimizer/EMA settings, learning-rate schedules and validation-checkpoint opportunities remain method-specific. Checkpoints use validation selection. Runs stop at the update target, potentially partway through an epoch; Appendix A.1 distinguishes them from the earlier 100-epoch experiments.

We also report the associated costs. Under the full-model budget, median dataset-paired RTSM/baseline training-time ratios are 4.71, 2.03 and 2.00 for MambaSL, TSCMamba and InterpGN, respectively. These retain the stated implementation-specific training timers and exclude validation and final testing. Thus, matching updates adds a useful control for optimization opportunity, while the accuracy and execution-cost results remain distinct.

I thank the authors for the detailed response. I think the new results help clarify some of my concerns. However, I want to maintain my original rating for two reasons. First, the single-pass comparison further shows that the benefit of recursive refinement is limited (72.47% vs. 72.62%). I still find the methodological contribution relatively limited beyond the architectural adaptation. Second, although RTSM achieves the highest mean accuracy, its advantage over the strongest baselines is not statistically significant after Holm correction. Considering the higher reported computational cost, I do not find the empirical improvement sufficiently convincing.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Closing summary: contributions and revised findings

We thank the reviewers and area chair for the constructive discussion. RTSM contributes an architecture for multivariate time-series classification and a detailed investigation of how temporal representation, latent-state organization, and computational depth shape its performance. The expanded evaluation makes three contributions particularly clear.

A strong temporal classifier with a practical shallow configuration. Across all 30 UEA datasets and three seeds, the proposed temporal encoder with one application of the attention module achieves 72.47% mean accuracy, a 4.20-percentage-point improvement over tokenizer-only classification. Full RTSM achieves 72.62%. The single-pass result provides a useful architectural outcome in its own right: strong classification without repeated refinement or multi-step deep supervision (Section 4.2, Table 4).

Concrete findings about architectural design. On the common 20 additional completed datasets, single-state RTSM achieves 68.96%, compared with 64.37% and 65.13% for the tested two-state adaptations. These comparisons favor the proposed single-state configuration as a whole; state organization and refinement schedules both differ. The component analysis additionally demonstrates an interaction between temporal encoding and downstream processing: removing residual connections improves tokenizer-only accuracy but reduces single-pass accuracy from 72.47% to 70.83%. These findings explain why evaluating the combined architecture matters beyond selecting components in isolation (Appendix B.3, Tables 13–14).

A quantified choice between shallow processing and additional refinement. The separately trained single-pass configuration achieves an accuracy only 0.15 percentage points below full RTSM, while seed-0 GH200 profiling records a median dataset-paired full/single-pass batch-1 latency ratio of 24.87. This gives practitioners a concrete accuracy–cost tradeoff within the architecture and distinguishes the parameter reuse enabled by weight tying from the execution cost of repeated computation (Appendix B.5).

These findings are supported by three-seed benchmark comparisons, expanded architectural ablations, and 540 additional baseline runs matching optimizer-update counts per dataset and seed. RTSM retains positive observed mean accuracy differences under this control; the comparisons do not establish superiority after Holm correction. The revised manuscript also documents preprocessing, development history, and the scope of the resource measurements.

The paper therefore delivers both a competitive classification architecture and actionable findings about its design: effective temporal encoding with shallow attention, a favorable tested single-state alternative to two-state refinement, and a measured account of the limited returns from additional computation in these experiments. The revised Abstract, Introduction and Conclusion now foreground these contributions. We appreciate the reviewers’ engagement in helping us make their significance clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.