- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a sequence-modeling architecture termed the Time Delay System (TDS). The model combines: (i) a semantic lag estimator that retrieves a soft weighted representation from a fixed window of historical token embeddings; (ii) a diagonal state-space recurrence evaluated through a cumulative-sum formulation; and (iii) a Takagi–Sugeno–Kang fuzzy system that modulates the recurrent and retrieved-history branches. The architecture is motivated through nonlinear time-delay differential equations, and the appendix presents theoretical claims concerning existence, boundedness, and continuous- and discrete-time stability. The method is evaluated on IMDB sentiment classification and CNN/DailyMail summarization. The paper reports 95.35% accuracy on IMDB and a ROUGE-L score of 42.63 on CNN/DailyMail, together with relatively small task-specific parameter counts. It also presents one qualitative visualization of the lag weights and a sensitivity experiment over the maximum history length.
Critical strengths
-
The paper studies a relevant and potentially useful architectural direction. Augmenting an efficient state-space backbone with explicit content-dependent retrieval could mitigate the information bottleneck of a fixed-dimensional recurrent state. The combination of semantic similarity and temporal decay is intuitively motivated and may offer a useful compromise between purely recurrent processing and full self-attention.
-
The proposed model has a reasonably clear high-level decomposition. The retrieval, recurrence, and modulation components are separated in the architecture, and the main figure communicates the intended information flow effectively. If validated, the fuzzy modulation branch could provide a more inspectable control mechanism than an unconstrained neural gate.
Critical weaknesses
-
The implemented model does not match the proposed time-delay formulation, and the theoretical guarantees therefore do not apply to the evaluated architecture.
The paper presents inconsistent state updates across Eqs. (11), (12), and (32). In Algorithms 3 and 6, is the only recurrent variable, while is a readout that is never fed back into the dynamics; the delayed branch also depends on past inputs rather than delayed states. Thus, the implemented model does not realize the state-delay system used to motivate and analyze the method. The appendix compounds this mismatch: Theorem 3 introduces delayed state-error dynamics not implied by Eq. (8), applies an inappropriate spectral-radius condition to a continuous-time system, and changes the equilibrium definition; Theorem 4 uses local approximations to claim global stability. These issues undermine the paper’s central theoretical contribution. -
The empirical and computational evidence does not support the claimed effectiveness or efficiency.
The baselines are drawn from different publications rather than evaluated under a common protocol, while TDS uses only 80,000 CNN/DailyMail training examples. Table 1 reports 95.35% IMDB accuracy, but Figure 3 reports approximately 73–79%, without explanation. The paper also provides no multi-seed results, controlled ablations, or sufficient implementation details. Finally, the lag estimator requires similarity computation and storage, and a cumulative sum does not have total complexity merely because it is implemented as one GPU kernel. No scaling experiment supports the claimed million-token capability. Consequently, neither the reported performance gains nor the efficiency claims can be reliably assessed.
Questions for authors
-
What exact forward equations were implemented to obtain Tables 1 and 2? Does the evaluated model follow Eq. (12), Eq. (28), Eq. (32), or Algorithm 6? In particular, is ever fed back into a subsequent state update, or is the only recurrent state? Please also explain the discrepancy between the context definition in Eq. (22) and the one used in Algorithms 3 and 6. A precise implementation-to-equation correspondence would materially affect my soundness assessment.
-
What system are Theorems 3 and 4 intended to analyze? How is the delayed error term in Eq. (44) derived from Eq. (8), whose delayed branch depends on past inputs rather than delayed states? Please also clarify the continuous-time stability condition, the different equilibrium definitions in Eqs. (39) and (43), and how a local approximation in Theorem 4 supports a global conclusion.
-
Were all results in Tables 1 and 2 obtained using common data splits, preprocessing, tokenization, sequence lengths, tuning budgets, and metric implementations? If some numbers were taken from prior publications, please provide their exact provenance and justify their comparability. Please also reconcile the 95.35% IMDB accuracy in Table 1 with the approximately 73–79% results in Figure 3.
-
How is causality enforced when the bidirectional convolution is used for autoregressive summarization, particularly under teacher forcing? In addition, what quantities are cached during generation, and what are the measured runtime and peak-memory scaling with sequence length and maximum lag? These details are necessary to evaluate both the possibility of target leakage and the claimed per-step complexity.
2 comments
- Causality must be distinguished between the source and target sequences. Bidirectional convolution over the observed source document is compatible with autoregressive summarization. On the target side, teacher forcing supplies shifted target tokens, and convolution must use left padding without access to future target positions. Historical retrieval must likewise be restricted to the available target prefix. Teacher forcing alone does not enforce these constraints, and an attention mask alone cannot prevent leakage through a bidirectional target convolution.
During incremental decoding, the required dynamic caches comprise the recurrent latent state, the previous output state used for feedback, the convolution buffer, and the historical representations required by the bounded retrieval mechanism. Source encoder representations are computed once and reused. These caches must reproduce the same causal forward computation used during training.
The claimed O(1) per-step complexity is with respect to the generated prefix length, with model dimensions, convolution width, and maximum lag held fixed. It does not imply independence from the maximum lag. A bounded history cache requires memory proportional to the retained history, and retrieval cost depends on how many historical positions are examined. Any decoder component that attends to the complete target prefix must also be included in the complexity accounting.
The experimental setup already specifies the NVIDIA H100 80GB GPU, hidden dimension of 256, and maximum semantic delay of 30. The O(1) per-step claim refers specifically to the dependence on the generated prefix length when the model dimensions and maximum lag are fixed. Under bounded-history retrieval, each decoding step retains and accesses a bounded amount of historical information, rather than an expanding target prefix. This claim does not imply constant cost as the maximum lag increases.
These configuration details establish the operating conditions of the reported experiments. Measured decoding latency and peak-memory scaling are separate quantities and cannot be inferred from the aggregate training times in Table 2.
We appreciate the reviewer’s thorough review and valuable suggestions. We clarify the mapping between implemented equations and the evaluated model, correct transcription errors in the algorithms, and refine the scope and assumptions of our stability analysis. We further resolve the IMDB accuracy discrepancy and distinguish our reproduced results from externally reported baselines with different evaluation protocols.
The fuzzy context follows Eq. (22), zk=(sk−1+dk)/2. Accordingly, sk feeds into the subsequent gating context, while hk is the state propagated through the linear transition. The semantic state is not substituted for hk in Eq. (29).
The dk/2 expressions in Algorithms 3 and 6 are transcription errors. With the Eq. (22) context, the semantic outputs retain a sequential gating dependency, even though the latent-state sequence can be computed by a parallel scan.
For Eq. (8), the historical branch is an input-dependent forcing term. With identical fixed input histories, this term cancels when comparing perturbed state trajectories. The linearized error dynamics therefore involve the current-state error, without a separate delayed-state error e(t−τ∗). The relevant continuous-time stability condition is that the equilibrium Jacobian is Hurwitz; ρ(A)<1 alone is insufficient.
The equilibrium definitions correspond to different settings: Eq. (39) sets the continuous-time vector field to zero, whereas Eq. (43) is a discrete fixed-point condition. The continuous-time proof must consistently use Eq. (39).
For Theorem 4, global convergence requires a uniform contraction bound. Under fixed historical forcing, Lϕ∥A∥2<1 is a sufficient condition. A bound established only near an equilibrium supports local convergence.
These clarifications preserve a precise stability statement for the specified input-delay systems. They concern the scope and justification of the analytical guarantees; they do not, by themselves, invalidate the reported task measurements or the proposed coupling of historical retrieval and recurrent processing.
The difference between Table 1 and Figure 3 arises from their experimental scope. The 95.35% IMDB accuracy in Table 1 is obtained using the full dataset, whereas Figure 3 presents dedicated ablation experiments conducted on a reduced subset. Its approximately 73–79% accuracies therefore do not represent the full-data benchmark setting. Figure 3 evaluates the effect of individual design choices within the ablation setting, rather than reproducing the absolute accuracy in Table 1.