acceptodds
Under review as a conference paper at ICLR 2027

Time Delay System for Sequence Modeling

Abstract

Existing sequence models share a critical limitation: they lack an explicit, interpretable mechanism for content-aware temporal recall. LSTMs compress everything into a single hidden state, thereby losing information about specific past events. Transformers attend to all tokens indiscriminately at quadratic cost, while state-space models like Mamba compress history into a fixed-dimensional state without explicit content-addressable memory. To address this gap, we introduce a novel framework grounded in nonlinear time-delay systems, in which the effective delay is dynamically estimated based on the semantic similarity between current and historical inputs. Our approach features three key innovations: (i) a differentiable semantic lag estimator that computes content-aware soft attention weights over history, producing a weighted historical context and similarity score; (ii) a diagonal state space with closed-form cumulative sum solution, achieving O(1) per-step complexity with no sequential loops; and (iii) an interpretable fuzzy semantic modulation mechanism that learns context-dependent rules for balancing instantaneous processing versus historical recall, where rule centers correspond to prototypical semantic contexts. Experimental results on text classification and generation outperform state-of-the-art methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
time delay system

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5PcROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a sequence-modeling architecture termed the Time Delay System (TDS). The model combines: (i) a semantic lag estimator that retrieves a soft weighted representation from a fixed window of historical token embeddings; (ii) a diagonal state-space recurrence evaluated through a cumulative-sum formulation; and (iii) a Takagi–Sugeno–Kang fuzzy system that modulates the recurrent and retrieved-history branches. The architecture is motivated through nonlinear time-delay differential equations, and the appendix presents theoretical claims concerning existence, boundedness, and continuous- and discrete-time stability. The method is evaluated on IMDB sentiment classification and CNN/DailyMail summarization. The paper reports 95.35% accuracy on IMDB and a ROUGE-L score of 42.63 on CNN/DailyMail, together with relatively small task-specific parameter counts. It also presents one qualitative visualization of the lag weights and a sensitivity experiment over the maximum history length.

Critical strengths
  1. The paper studies a relevant and potentially useful architectural direction. Augmenting an efficient state-space backbone with explicit content-dependent retrieval could mitigate the information bottleneck of a fixed-dimensional recurrent state. The combination of semantic similarity and temporal decay is intuitively motivated and may offer a useful compromise between purely recurrent processing and full self-attention.

  2. The proposed model has a reasonably clear high-level decomposition. The retrieval, recurrence, and modulation components are separated in the architecture, and the main figure communicates the intended information flow effectively. If validated, the fuzzy modulation branch could provide a more inspectable control mechanism than an unconstrained neural gate.

Critical weaknesses
  1. The implemented model does not match the proposed time-delay formulation, and the theoretical guarantees therefore do not apply to the evaluated architecture.
    The paper presents inconsistent state updates across Eqs. (11), (12), and (32). In Algorithms 3 and 6, hkh_k is the only recurrent variable, while sks_k is a readout that is never fed back into the dynamics; the delayed branch also depends on past inputs rather than delayed states. Thus, the implemented model does not realize the state-delay system used to motivate and analyze the method. The appendix compounds this mismatch: Theorem 3 introduces delayed state-error dynamics not implied by Eq. (8), applies an inappropriate spectral-radius condition to a continuous-time system, and changes the equilibrium definition; Theorem 4 uses local approximations to claim global stability. These issues undermine the paper’s central theoretical contribution.

  2. The empirical and computational evidence does not support the claimed effectiveness or efficiency.
    The baselines are drawn from different publications rather than evaluated under a common protocol, while TDS uses only 80,000 CNN/DailyMail training examples. Table 1 reports 95.35% IMDB accuracy, but Figure 3 reports approximately 73–79%, without explanation. The paper also provides no multi-seed results, controlled ablations, or sufficient implementation details. Finally, the lag estimator requires O(TL)O(TL) similarity computation and storage, and a cumulative sum does not have O(1)O(1) total complexity merely because it is implemented as one GPU kernel. No scaling experiment supports the claimed million-token capability. Consequently, neither the reported performance gains nor the efficiency claims can be reliably assessed.

Questions for authors
  1. What exact forward equations were implemented to obtain Tables 1 and 2? Does the evaluated model follow Eq. (12), Eq. (28), Eq. (32), or Algorithm 6? In particular, is sks_k ever fed back into a subsequent state update, or is hkh_k the only recurrent state? Please also explain the discrepancy between the context definition in Eq. (22) and the one used in Algorithms 3 and 6. A precise implementation-to-equation correspondence would materially affect my soundness assessment.

  2. What system are Theorems 3 and 4 intended to analyze? How is the delayed error term e(t−τ∗)e(t-\tau^*) in Eq. (44) derived from Eq. (8), whose delayed branch depends on past inputs rather than delayed states? Please also clarify the continuous-time stability condition, the different equilibrium definitions in Eqs. (39) and (43), and how a local approximation in Theorem 4 supports a global conclusion.

  3. Were all results in Tables 1 and 2 obtained using common data splits, preprocessing, tokenization, sequence lengths, tuning budgets, and metric implementations? If some numbers were taken from prior publications, please provide their exact provenance and justify their comparability. Please also reconcile the 95.35% IMDB accuracy in Table 1 with the approximately 73–79% results in Figure 3.

  4. How is causality enforced when the bidirectional convolution is used for autoregressive summarization, particularly under teacher forcing? In addition, what quantities are cached during generation, and what are the measured runtime and peak-memory scaling with sequence length and maximum lag? These details are necessary to evaluate both the possibility of target leakage and the claimed O(1)O(1) per-step complexity.

2 comments
AuthorsOpenReview ↗

We appreciate the reviewer’s thorough review and valuable suggestions. We clarify the mapping between implemented equations and the evaluated model, correct transcription errors in the algorithms, and refine the scope and assumptions of our stability analysis. We further resolve the IMDB accuracy discrepancy and distinguish our reproduced results from externally reported baselines with different evaluation protocols.

  1. Tables 1 and 2 evaluate the latent-state formulation in Eq. (32). The latent state hkh_k follows the diagonal linear recurrence in Eq. (29), evaluated through Eqs. (30)–(31). The semantic output sks_k is then computed by Eq. (32).

The fuzzy context follows Eq. (22), zk=(sk−1+dk)/2z_k=(s_{k-1}+d_k)/2. Accordingly, sks_k feeds into the subsequent gating context, while hkh_k is the state propagated through the linear transition. The semantic state is not substituted for hkh_k in Eq. (29).

The dk/2d_k/2 expressions in Algorithms 3 and 6 are transcription errors. With the Eq. (22) context, the semantic outputs retain a sequential gating dependency, even though the latent-state sequence can be computed by a parallel scan.

  1. The stability analysis concerns the continuous and discrete input-delay formulations in Eqs. (8) and (12), respectively, under a constant input and stationary input history. These are conditional analytical results, rather than assumptions required to execute the evaluated architecture.

For Eq. (8), the historical branch is an input-dependent forcing term. With identical fixed input histories, this term cancels when comparing perturbed state trajectories. The linearized error dynamics therefore involve the current-state error, without a separate delayed-state error e(t−τ∗)e(t-\tau^*). The relevant continuous-time stability condition is that the equilibrium Jacobian is Hurwitz; ρ(A)<1\rho(A)<1 alone is insufficient.

The equilibrium definitions correspond to different settings: Eq. (39) sets the continuous-time vector field to zero, whereas Eq. (43) is a discrete fixed-point condition. The continuous-time proof must consistently use Eq. (39).

For Theorem 4, global convergence requires a uniform contraction bound. Under fixed historical forcing, Lϕ∥A∥2<1L_\phi\|A\|_2<1 is a sufficient condition. A bound established only near an equilibrium supports local convergence.

These clarifications preserve a precise stability statement for the specified input-delay systems. They concern the scope and justification of the analytical guarantees; they do not, by themselves, invalidate the reported task measurements or the proposed coupling of historical retrieval and recurrent processing.

  1. The experimental setup specifies the dataset splits and training configuration used for TDS in Tables 1 and 2. IMDB uses 25,000 training and 25,000 test examples; CNN/DailyMail uses 80,000 training examples and 2,000 examples each for validation and testing. These settings describe our experiments; results taken from prior publications retain their original evaluation protocols and should be distinguished from baselines reproduced under common settings.

The difference between Table 1 and Figure 3 arises from their experimental scope. The 95.35% IMDB accuracy in Table 1 is obtained using the full dataset, whereas Figure 3 presents dedicated ablation experiments conducted on a reduced subset. Its approximately 73–79% accuracies therefore do not represent the full-data benchmark setting. Figure 3 evaluates the effect of individual design choices within the ablation setting, rather than reproducing the absolute accuracy in Table 1.

  1. Causality must be distinguished between the source and target sequences. Bidirectional convolution over the observed source document is compatible with autoregressive summarization. On the target side, teacher forcing supplies shifted target tokens, and convolution must use left padding without access to future target positions. Historical retrieval must likewise be restricted to the available target prefix. Teacher forcing alone does not enforce these constraints, and an attention mask alone cannot prevent leakage through a bidirectional target convolution.

During incremental decoding, the required dynamic caches comprise the recurrent latent state, the previous output state used for feedback, the convolution buffer, and the historical representations required by the bounded retrieval mechanism. Source encoder representations are computed once and reused. These caches must reproduce the same causal forward computation used during training.

The claimed O(1) per-step complexity is with respect to the generated prefix length, with model dimensions, convolution width, and maximum lag held fixed. It does not imply independence from the maximum lag. A bounded history cache requires memory proportional to the retained history, and retrieval cost depends on how many historical positions are examined. Any decoder component that attends to the complete target prefix must also be included in the complexity accounting.

The experimental setup already specifies the NVIDIA H100 80GB GPU, hidden dimension of 256, and maximum semantic delay of 30. The O(1) per-step claim refers specifically to the dependence on the generated prefix length when the model dimensions and maximum lag are fixed. Under bounded-history retrieval, each decoding step retains and accesses a bounded amount of historical information, rather than an expanding target prefix. This claim does not imply constant cost as the maximum lag increases.

These configuration details establish the operating conditions of the reported experiments. Measured decoding latency and peak-memory scaling are separate quantities and cannot be inferred from the aggregate training times in Table 2.

Reviewer_eMhCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes time delay system, a sequence model motivated by nonlinear time-delay differential equations, where it includes semantic lag estimator, diagonal linear recurrence, and a fuzzy gate. The authors compare TDS results with LSTM, Mamba, BERT-family, BART and T5 models on IMDB and CNN/DailyMail dataset.

Critical strengths

Providing recurrent models with content-addressable access to past information is a valid and important research problem. However, I am not convinced that this paper addresses the problem in the right way.

Critical weaknesses
  • The core premise of the proposed approach is to give recurrent models content-addressable access to the past, however, TDS only looks back very limited amount of tokens with its content-aware window, which is well within the range of mamba models. It should at least the performance against 64/128/256k length.
  • IMDB sentiment and CNN/DailyMail summarization don’t isolate recall ability at all and is too toyish. It should at least test on NIAH and MQAR.
  • The paper misses important hybrid architecture baselines, which are built for addressing the recall issue.
  • The TDS design is unnecessarily complex, and experiment results do not justify the design enough.
Questions for authors
  • why not test on longer sequence length?
  • why not test on NIAH/MQAR?
2 comments
AuthorsOpenReview ↗

We thank the reviewer for the valuable and thoughtful feedback. We clarify that our bounded retrieval window works with recurrent state feedback for long-range memory. We separate downstream task performance from isolated recall evaluation (NIAH/MQAR), add hybrid baselines, and justify our design via the synergy of retrieval, state feedback and fuzzy modulation

  1. The retrieval window LL bounds direct access to historical token representations; it does not bound the history represented by the recurrent state. Information retrieved within the window can influence subsequent states through recurrent feedback.

The distinction from Mamba is therefore not whether either model can propagate information beyond LL. TDS explicitly compares the current representation with historical representations and uses the retrieved content to condition the update. This provides a different access mechanism, but does not by itself establish a longer effective memory or more accurate long-range recall.

Accordingly, the supported claim is content-dependent retrieval within a bounded window, coupled to recurrent memory, rather than arbitrary-distance content-addressable access. Evaluation at 64K/128K/256K is relevant to the stronger long-context claim. Such evaluation must measure retrieval accuracy across distances and insertion depths; processing sequences of those lengths alone would not demonstrate retention.

  1. IMDB and CNN/DailyMail evaluate downstream classification and generation, rather than recall in isolation. Their results remain relevant to practical task performance, but do not establish precise retrieval of historical information.

NIAH and MQAR address complementary questions: NIAH measures retrieval across controlled distances and insertion depths, while MQAR tests recovery of multiple key–value associations under interference. For TDS, these evaluations should distinguish queries within and beyond the retrieval window LL, separating direct historical access from retention through recurrent state.

The evidence therefore supports two distinct assessments: downstream utility on IMDB/CNN/DailyMail, and recall ability on controlled retrieval tasks. Claims about recall require the latter.

  1. To assess the contribution of state-conditioned fuzzy feedback beyond recurrent–attention hybridization, we compare TDS with SRformer and Lite Transformer on CNN/DailyMail dataset.
ModelParameters (M)ROUGE-LTotal Training Time (s)
Lite Transformer (Wu et al., 2020)†17.338.30—
LSTM (Abdulsahib et al., 2022)26.538.4516558
SRformer–T5-small (Long et al., 2023)†6027.29—
BERT (Maharditha & Lhaksmana, 2025)24.3040.4818260
Mamba (Do et al., 2025)14.7329.6016725
BART CNN-large (Rani Krishna et al., 2025)13924.58—
T5-base (Rani Krishna et al., 2025)22023.54—
TDS16.9842.6317633
  1. The contribution lies in the interaction among historical retrieval, state feedback, and fuzzy modulation. Retrieved content influences both the current update and subsequent state evolution, while the preceding semantic state conditions how the retrieved information is incorporated. The fuzzy rules explicitly parameterize this state–history interaction.

These components serve complementary roles within a single update mechanism: selecting historical information, propagating its influence, and adapting its contribution to the semantic context. The reported IMDB and CNN/DailyMail results provide downstream evidence for the utility of this integrated design.

Thanks for the authors clarification. My main concern still persists, IMDB and CNN/DailyMail experiments are toyish which are not enough to justify the paper's design. Hence I will maintain my score.

Reviewer_J7a1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a sequence modeling block inspired by time-delay systems, combining a parallelized diagonal linear recurrence with local distance-decayed soft attention over recent embeddings, mixed by scalar weights from a Gaussian RBF fuzzy gate. The authors provide stability proofs in the appendix and evaluate on IMDB classification and CNN/DailyMail summarization.

Critical strengths

Giving linear recurrent/SSM models explicit content-based lookback without full attention is a sensible goal, and motivating variable-lag recall via time-delay systems is a neat conceptual angle.

Critical weaknesses
  1. Once the authors parallelize the layer in Section 3.8 and Algorithm 3, almost all of the time-delay dynamics disappear. They drop the delayed hidden state, move the activation outside the recurrence, and replace the fuzzy input with d/2d / 2. What's left is really just a diagonal linear RNN in parallel with sliding-window attention (very close to existing hybrids like Griffin or Samba, which aren't discussed here).
  2. I don't see how this architecture supports causal text generation as described. Both Figure 1 and Algorithm 3 run a bidirectional convolution over the input sequence before the recurrence. That is fine for IMDB sentiment classification, but on CNN/DailyMail generation it leaks future tokens during training and breaks token-by-token decoding.
Questions for authors
  1. How do you handle causal training and token-by-token generation on CNN/DailyMail given the bidirectional convolution in Figure 1 and Algorithm 3?
  2. What happens if you ablate out the lag estimator or replace the fuzzy rules with a simple linear gate, and how long were the sequences you actually ran before ata^t and a−ta^{-t} hit numerical issues?
1 comment
AuthorsOpenReview ↗

We thank the reviewer for the careful reading and constructive feedback. We correct the typo in Algorithm 3 and confirm that state-feedback time-delay dynamics persist under parallelization. Our method differs from Griffin and Samba via coupled content-aware retrieval and state modulation, and causal convolution is employed for generation to rule out future-token leakage.

  1. The fuzzy context follows Eq. (22), zk=(sk−1+dk)/2z_k=(s_{k-1}+d_k)/2. The dk/2d_k/2 expressions in Algorithms 3 and 6 are typographical errors. The complete update is
sk=f(zk)⊙SiLU⁡(Ask−1+Bxk)+g(zk)⊙dk.s_k=f(z_k)\odot\operatorname{SiLU}(As_{k-1}+Bx_k) +g(z_k)\odot d_k.

Here, sks_k, including its historical correction, enters the next transition and the next fuzzy context. Consequently, historical retrieval affects subsequent dynamics through both state feedback and adaptive modulation; it is not simply an independent attention output added to a linear RNN.

The delay term retrieves historical input representations, rather than a separate delayed hidden state sk−τks_{k-\tau_k}. This distinction specifies what the proposed delay mechanism actually models.

Eqs. (29)–(31) describe the scan of the linear latent branch. They do not, by themselves, establish a fully parallel evaluation of the nonlinear state-feedback update above. Diagonalizing AA preserves channelwise transitions but does not remove this temporal dependency.

The relevant distinction from Griffin and Samba is therefore the particular coupling between content-dependent historical retrieval and state-conditioned modulation. 2. The convolution configuration is task-dependent. Bidirectional convolution is used for IMDB classification, whereas autoregressive generation requires a causal convolution:

ck=∑j=0K−1Wjxk−j,c_k=\sum_{j=0}^{K-1}W_jx_{k-j},

with left padding only. Figure 1 and Algorithm 3 omit this task-specific distinction.

For CNN/DailyMail, the decoder receives shifted target tokens. At position kk, convolutional features depend only on the available prefix, and historical retrieval accesses only preceding positions. Teacher forcing therefore does not expose future target tokens to the current prediction.

During token-by-token decoding, each layer retains its recurrent state, historical-retrieval window, and convolution cache. The newly generated token updates these caches without requiring subsequent tokens. The training and decoding computations consequently obey the same prefix dependency.

This causal guarantee depends on the decoder convolution itself: a causal attention mask alone would not prevent leakage introduced by a bidirectional convolution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.