acceptodds
Under review as a conference paper at ICLR 2027

Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View

Abstract

Scalable sequence models, such as Transformer variants and structured state-space models, often trade expressivity power for sequence-level parallelism, which enables efficient training. Here we examine the bounds on error and how error scales when models operate outside of their expressivity regimes using a Lie-algebraic control perspective. Our theory formulates a correspondence between the depth of a sequence model and the tower of Lie algebra extensions. Echoing recent theoretical studies, we characterize the Lie-algebraic class of constant-depth sequence models and their corresponding expressivity bounds. Furthermore, we analytically derive an approximation error bound and show that error diminishes exponentially as the depth increases, consistent with the strong empirical performance of these models. We validate our theoretical predictions using experiments on symbolic word and continuous-valued state-tracking problems.

Primary area:
learning theory
Keywords:
Transformers, Attention, State-space models, Linear RNNs, Deep models, Expressivity, Lie algebra, State tracking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QRcZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how parallelizable sequence models with order-symmetric layers (diagonal SSMs, linear attention, and, as the authors argue, transformers) perform on tasks outside their exact expressivity, and how depth changes this. SSMs are modeled as controlled affine ODEs and classified by the Lie algebra their vector fields generate. The main results are:

(1) A single "restricted" layer (abelian linear part with an input-dependent bias) incurs a worst-case error lower-bounded in proportion to the second Magnus term ||Omega_2|| (Thm 3.2).

(2) A Lie algebra extension can be simulated by a two-layer SSM, so k abelian layers simulate any solvable system of derived length k (Thm 3.4, Cor 3.5).

(3) Non-solvable systems can be approximated by k abelian layers with local error O(eps^(2^k)) via free-nilpotent truncation (Cor 3.7).

(4) Word problems of length <= T are solvable with floor(log2 T)+1 abelian layers (Prop 3.8).

Validation consists of length-generalization experiments on word problems over C2, C3, D8, H3, S3, S4, and A5 with GLA, Mamba variants, AUSSM, transformers, and DeltaProduct, plus an A5 rigid-body rotation regression task. That task includes a width control and commutator-probe words that test the ||Omega_2|| scaling of 1-layer models.

Critical strengths
  1. A quantitative, width-independent lower bound on order-sensitive error. Thm 3.2's four-input difference-of-differences construction cancels the inhomogeneous terms and shows that any restricted layer, of any width, incurs error tied to ||Omega_2||. This moves beyond binary can/can't expressivity results toward the error-scaling question the paper sets out to answer. It is the most original and most convincing contribution, and it is directly probed by Fig. 3b.

  2. A clean structural account of depth via the derived series. Using the derived series as a budget for depth (Prop 3.3 together with Cor 3.5) gives a principled explanation of why depth, rather than width, extends the expressivity of order-symmetric layers. The 8x width control in Fig. 3a supports this qualitatively. Even with the concerns below, the framework is a useful lens that others could build on.

Critical weaknesses
  1. The experiments are not connected to the theorems they are said to validate. Thm 3.4 and Cor 3.5 concern Lie algebra extensions, while D8, S3, and S4 are finite groups with trivial Lie algebra. The Lie algebra generated by logarithms of a representation's generators depends on the embedding; for example, S3 realized in SO(3) generates so(3), which is not solvable. Moreover, finite subgroups of connected solvable Lie groups are abelian, so a 2-layer abelian solution for S3 must be a nonlinear cascade. That is plausible, and close to what Shakerinava et al. (2026) prove in discrete time, but it is not what Thm 3.4 states.

Separately, Signed Mamba relies on negative real eigenvalues, which cannot arise from discretizing a real diagonal continuous-time generator, and softmax attention does not fit Def. 2.1 at all. Table 2 and Figs. 2-3a are therefore suggestive but do not test the theory. This matters because the paper presents the experiments as validation.

  1. The headline depth-error claim is not established as stated. First, the local-to-global argument for Cor 3.7 (App. C.6) bounds a product of per-window truncated flows. The constructed model does not compute that product: it accumulates state in the free nilpotent group and applies a non-homomorphic output map once. Realizing the product would require accumulating in the non-solvable group itself. As far as I can tell, the result holds only while the total generator mass stays within the Magnus convergence region.

Second, the lower bound (Thm 3.2) assumes a linear readout, while all the upper bounds allow arbitrary smooth readouts. With smooth readouts and bounded length, even a single sufficiently wide restricted layer can encode the input injectively, and Prop 3.8 is itself a memorization construction. So the claim that depth, rather than width, is the mechanism is not cleanly separated by the results. Together these undercut the abstract's central claim that error "diminishes exponentially as the depth increases." The claimed rate is also doubly exponential in k, not exponential.

Questions for authors
  1. How does the per-window telescoping argument in App. C.6 apply to the constructed deep abelian SSM, given that its output map exp o ev o log is not a group homomorphism? If it doesn't, what global statement survives?

  2. What result connects the derived length of a finite group to the derived length of the Lie algebra of an SSM that solves its word problem? Is the 2-layer claim for D8 and S3 in Sec. 4.1 meant to follow from Thm 3.4, or from a separate cascade construction?

  3. Does Thm 3.2, or the "restricted vs. abelian" distinction, survive if the linear readout P is replaced by a smooth nonlinear readout, as used in the upper bounds?

  4. How are Signed Mamba's negative real eigenvalues accommodated in a continuous-time framework where every transition is a matrix exponential?

  5. Table 2 uses 8 tokens for H3. If this is the Heisenberg group over F2, it is isomorphic to D8. Are the two columns testing the same group?

  6. In the proof of Thm 3.2, can you justify the inequality ||e^X - e^Y|| >= e^(-max(||X||,||Y||)) ||X - Y|| for non-commuting X, Y, and the choice of prefixes that aligns h1 - h2 with the dominant direction of M?

Suggestions that would strengthen the paper:

  1. Run 3-layer models on S4 (and include A4). Cor 3.5 predicts these thresholds sharply, so this would be a direct test.

  2. Measure error as a function of eps and depth, for example with fractional rotations, and fit a log-log slope in Fig. 3b against the predicted exponent. Note that MSE should scale quadratically if the norm bound is tight.

3 comments
AuthorsOpenReview ↗

Response to Reviewer QRcZ, part 1/3

We deeply appreciate the thoughtful feedback and careful reading. Addressing these points improves the quality of our work.

[Q1, part of Weakness 2] How does the per-window telescoping argument in App.C.6 apply, given that the output map is not a group homomorphism?

This is a keen observation. We agree that our per-window product argument overlooked the group homomorphism issue, and the local-to-global statement in App.C.6 needs to be revised. The global statement can nevertheless be repaired by using the Peano-Baker series (e.g. Baake and Schlägel, 2011), without major revision in the main text.

The class cc free nilpotent construction is still the universal construction, which contains all the information needed to evaluate the Peano-Baker term up to cc-th order. Rather than logarithmic Magnus evaluation, the Peano-Baker series uses time-ordered products to represent flow:

\quad \mathcal{I}_n(T) := \int_{t_0<t_n<\cdots<t_1<T} A(x(t_1)) \cdots A(x(t_n)) \, dt_n \cdots dt_1.$$ Then the truncated Peano-Baker series readout is simply $\Phi_c(t,t_0) = I + \sum_{n=1}^c \mathcal{I}_n(t)$. Importantly, the Peano-Baker series converges when the generator mass is finite. Therefore, we can evaluate the global error bound at once: $$\Vert\Phi(T,0) - \Phi_c(T,0)\Vert= \Vert\sum_{n=c+1}^\infty \mathcal{I}_n(T)\Vert \le \sum_{n=c+1}^\infty \frac{\epsilon^n}{n!} \le \exp(\epsilon) \frac{\epsilon^{c+1}}{(c+1)!}.$$ Take $c = 2^k - 1$ again, $$\Vert\Phi(T,0) - \Phi_c(T,0)\Vert \le \exp(\epsilon) \frac{\epsilon^{2^k}}{(2^k)!}.$$ When $\epsilon$ is small, we recover $\mathcal{O}(\epsilon^{2^k})$, doubly exponential in $k$. We will revise App.C.6, local-to-global paragraph, accordingly. We will further revise the abstract to describe the doubly exponential rate. > [Q2, part of Weakness 1] What result connects the derived length of a finite group to the derived length of the theory? This is a theoretically very important point. We agree that the word problem is not a direct test of our continuous Lie-algebraic statements but more of a bridge to the existing well-established benchmarks. As reviewer kindly noted, Shakerinava et al., directly worked on finite group regimes. In particular, $D_8$ and $S_3$ have derived length 2, giving a 2-layer discrete realization with 1 abelian layer per factor. Importantly, each abelian factor layer also admits continuous realization. Any finite abelian group faithfully embeds to a torus via roots of unity, so abelian factor symbol action is realized by a flow of commuting rotation generators. Thus, every abelian factor layer in the discrete construction can be seen as an exactly sampled abelian SSM. We see these two as a shared extension-based mechanism applied in two regimes. Our construction begins from the Lie extension and decomposes integrated flow to quotient and kernel components. Shakerinava et al. begins from the finite group extension and applies a similar decomposition. Our attempt was to bridge our result with this key previous work. We will adjust our tone and present word problems as a diagnostic of a discrete analog of our theory, attributing Shakerinava et al.’s contribution. > [Q3, part of Weakness 2] With smooth readouts and bounded length, one restricted layer suffices. Does Thm 3.2, or restricted vs. abelian, survive if $\mathbf{P}$ is replaced by a smooth readout? On a *bounded* horizon, sufficiently wide linear recurrence followed by a nonlinear readout can memorize without loss (e.g. Orvieto et al., 2024). The quantitative bound in Thm 3.2 does not survive with a smooth readout in general. We clarify our notion of simulation below. In Def B.1, simulation fixes initialized SSMs and smooth state map $\varphi$, valid for *every* admissible input and horizon (Krener 1977). Def B.2 quantifies the best simulation error on a given window. For 1-layer SSM, $\varphi$ is restricted to fixed $\mathbf{P}$, preserving the affine SSM class (Toth 2010). Thm 3.2 rules out simulation by showing no fixed $\mathbf{P}$ can match the target state evolution for *all* $T>0$. Similarly, smooth maps in Thm 3.4, Cor 3.5, and Cor 3.7 are horizon-independent. In contrast, the memorizing structure is built per bounded window, which is not necessarily a simulation in our strict sense. As reviewer said, no width-independent bound holds with the arbitrary smooth readout for fixed window, and we did not intend to claim this. The restricted vs. abelian distinction survives with pointwise nonlinearity. Abelian SSM reaches the identical state upon permuted input, which no readout can distinguish. On the other hand, restricted SSM can retain order information. More generally, pointwise nonlinear readout does not add derived length (also Prop 3.3).
AuthorsOpenReview ↗

Response to Reviewer QRcZ, part 2/3

[Q4, part of Weakness 1] How are Signed Mamba’s negative real eigenvalues accommodated in a continuous-time framework?

Here we note that being abelian requires commutativity, not real-diagonal structure. We can represent any Z∈gl(n,C)Z \in \mathfrak{gl}(n,\mathbb{C}) as an element of gl(2n,R)\mathfrak{gl}(2n, \mathbb{R}). Specifically, any complex-diagonal matrix can be represented as real block-diagonal matrix (e.g. Orvieto 2023):

a & -b \\\\ b & a \end{pmatrix},$$ which keeps commutativity. Therefore, our theory applies to the continuous-time interpretation of the Signed Mamba, and AUSSM further. > [Q5] $H_3(F_2)$ is isomorphic to $D_8$. Are the two columns testing the same group? Yes, they are isomorphic. We included both as a consistency check with different encodings of the same group. We also observed no performance differences in their matrix-based task either. We will state the isomorphism more clearly. > [Q6] Justify matrix exponential inequality in the proof of Thm 3.2. Also justify the choice of prefixes that aligns $h_1 - h_2$ with the dominant direction of $M$. It is important to clarify these points. First, the inequality stems from the matrix operator norm. When $r = \max(\Vert X\Vert_2, \Vert Y\Vert_2) \le 1$, we have Von Neumann’s operator norm inequality: $$\Vert(zI + \exp X)^{-1}\Vert_2 \le (z + e^{-r})^{-1}$$ and similarly for $Y$. With the integral representation of the logarithm, $$X - Y = \int_0^\infty (zI + e^Y)^{-1}(e^X - e^Y) (zI + e^X)^{-1}dz.$$ Therefore, $$\Vert X-Y\Vert_2 \le \Vert e^X-e^Y\Vert_2 \int_0^\infty \frac{dz}{(z+e^{-r})^2} = e^r \Vert e^X-e^Y\Vert_2.$$ Note that this does not require commuting $X,Y$. We can switch operator norm to any submultiplicative norm at a cost of slightly weaker constant. For $r = \max(\Vert X\Vert, \Vert Y\Vert) < \log 2$, $$\Vert e^X - e^Y - (X-Y)\Vert \le \sum_{n=2}^\infty \frac{\Vert X^n-Y^n\Vert}{n!} \le (e^r-1) \Vert X - Y\Vert,$$ via telescoping series. With reverse triangular inequality, we achieve: $$\Vert e^X-e^Y\Vert \ge (2-e^r)\Vert X - Y\Vert = \lambda \Vert X - Y\Vert, \quad \lambda \in (0,1].$$ We will clarify this in the proof and relevant places. Second, the prefix choice leverages the minimality condition, specifically the controllability (Sec 2.1, App B.1). Intuitively, we can lift the prefix assumption by simply choosing two different initial states $(h_1,h_2)$ such that $h_1 - h_2$ does not fall under $\ker M$, and then run the second part of the constructed input. In this view, 4-input subtraction is a trick to reveal differences in initial states carried through affine dynamics. Controllability prescribes prefixes to reach at different “initial” states. > [part of Weakness 1] Softmax attention does not fit Def 2.1. We agree, the transformer was included as a reference model that is similarly parallelizable and order-symmetric with respect to the layer input.
AuthorsOpenReview ↗

Response to Reviewer QRcZ, part 3/3

[Q7] Run 3-layer models on S4S_4, and include A4A_4.

Thank you for suggesting the direct test. We trained 2-layer Signed Mamba and AUSSM on A4A_4, and 3-layer on S4S_4, but models failed to learn solutions in both tasks. This aligns with the struggle in the S3S_3 task. Shakerinava et al. similarly observed failures in S3S_3 and A4A_4 tasks. These observations clearly show a gap between theory and practice. By manually initializing weights near the analytical solution set, Shakerinava et al. were able to train 2-layer AUSSM on S3S_3. To probe the learnability issue further, we performed a curriculum task.

We pretrained a 1-layer AUSSM on a quotient of S3S_3 word problem. Specifically, the input tokens are still S3S_3 symbols, but the target tokens are S3/C3≅C2S_3/C_3 \cong C_2 quotient symbols. Next, we warm-started the embedding plus the first residual block of 2-layer AUSSM with the pretrained weights.

We observed the warm-started model successfully learns the S3S_3 word problem with 1.00 sequence accuracy. In contrast, standard 2-layer AUSSM remained at 0 accuracy even after doubling the training epochs. This result suggests the gap between theory and practice partially arises from the optimization process.

[Q8] Measure error as a function of eps and depth.

We fitted a log-log slope in Fig. 3b. As the reviewer predicted, Transformer and GLA scales close to quadratic scale in our probe regime, with mean slope 2.17 and 2.45 respectively. Signed Mamba shows a slightly weaker trend (1.67), potentially from a higher floor error at the smallest ∥Ω2∥\Vert\Omega_2\Vert probes. We will update Fig. 3b accordingly.

For error against ϵ\epsilon and depth, we are running suggested fractional rotation experiments. Preliminary GLA results below show the benefit of small ϵ\epsilon and depth. Note that the depth effect quickly plateaus:

Depth ↓\downarrow / Relative ϵ\epsilon →\rightarrow11.672
10.030.110.2
23e-30.0130.03
42.9e-34.5e-39.9e-3
64.2e-33.8e-39.2e-3
83.8e-33.7e-38.6e-3

In comparison, length 32 probes on A5 rotation task shows more sustained depth-dependent trend at a cost of shallow model saturation.

Depth ↓\downarrow / Relative ϵ\epsilon →\rightarrow11.672
10.350.330.33
20.30.330.33
40.0130.0260.046
63.0e-34.4e-36.9e-3
81.4e-32.2e-33.5e-3

Together, we observe qualitative benefits of depth and smaller ϵ\epsilon, but do not directly confirm the predicted scaling across settings. Gradient-based optimization need not find the solution constructed in Cor 3.7, and finite precision may further obscure trends in a smaller error regime. We will continue remaining experiments and report the results in the appendix.


Thank you again for the clear and constructive feedback. We will soon revise our manuscript and update it. We would deeply appreciate any further comments.

Reviewer_PTUgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates the order sensitivity of parallelizable sequence models and how depth alleviates limitations in expressive power. The authors connect multi-level SSMs with Lie algebraic structures, proposing a hierarchical representation construction for solvable systems. Using free nilpotent Lie algebras and Magnus expansions, they derive approximation error bounds.

Critical strengths
  1. The presentation is clear.

  2. The topic is meaningful, and the approximation analysis provides a valuable perspective for further development.

  3. The algebraic construction is interpretable. Mapping depth to the expansion of solvable structures and improving the local approximation order have potential theoretical value.

Critical weaknesses
  1. The experiments do not yet verify the quantitative results regarding depth. There still seem to be significant gaps between the theoretical lower bounds and the experimental tasks, particularly when considering practical fixed-width networks.

  2. This paper cites Shakerinava's work, but a clearer distinction would be helpful in establishing the contribution. Which results are generalizations to continuous Lie systems, which are new quantitative conclusions, and what different assumptions do these new conclusions rely on?

  3. It seems that there is a substantial gap in the proof of the generalization from local to global in Appendix C.

  4. The memory patterns of practical deep SSMs can generalize across sequence lengths. Thus, I do not quite understand the result in Proposition 3.8.

Questions for authors

Plz see the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer PTUg

Thank you for the clear and thoughtful feedback.

[Weakness 1] The experiments do not yet verify the quantitative results regarding depth. There still seem to be significant gaps between the theoretical lower bounds and the experimental tasks, particularly when considering practical fixed-width networks.

We agree our results do not directly validate the error rate in Cor 3.7. In Fig. 2 and Fig. 3a, we measured how depth changes the maximum sequence length at which a chosen error threshold is crossed. We also included a single layer width control in Fig. 3a. These experiments are our attempt to indirectly probe the expected depth-dependent behavior. To probe the depth-dependence more directly, we are measuring error as a function of generator mass and depth in a fractional rotation experiment. The preliminary results suggest the benefit of depth and smaller generator mass. For details, we respectfully refer to our response to Reviewer QRcZ, Q8.

The theoretical lower bound in Thm 3.2 provides a loose, width-independent lower bound. Our Fig. 3b aims to directly measure the loss behavior by varying commutator mass.

We will clarify the scope and adjust our tone in experimental claims accordingly.

[Weakness 2] This paper cites Shakerinava's work, but a clearer distinction would be helpful in establishing the contribution. Which results are generalizations to continuous Lie systems, which are new quantitative conclusions, and what different assumptions do these new conclusions rely on?

The relationship to and distinctions with Shakerinava et al are important. Thm 3.4 is our theoretical heart, interpreting depth as a Lie-theoretic extension. Prop 3.3 and Cor 3.5 provide the Lie-theoretic, continuous analog of Shakerinava et al., Theorem 2.

Thm 3.2 and Cor 3.7 are our new quantitative bounds for the single layer and deep cases, respectively. Thm 3.2 provides lower bound as a function of commutator mass, while Cor 3.7 serves upper bound as a function of generator mass and depth. Our core assumption is real arithmetic, allowing us to isolate algebraic limitations from finite-precision constraints. These results complement exact expressivity studies with approximation. We will revise the text to clarify our contribution more explicitly.

[Weakness 3] It seems that there is a substantial gap in the proof of the generalization from local to global in Appendix C.

We believe this gap refers to the local-to-global extension of Cor 3.7 (also noted in Reviewer QRcZ’s question). We closed the gap by replacing Magnus evaluation with the Peano-Baker evaluation on the same free nilpotent construction, without major revision to the main text. With finite generator mass ϵ=∫0T∥A(x(t))∥dt\epsilon = \int_0^T \Vert A(x(t)) \Vert dt, the Peano-Baker series truncated at order 2k−12^k - 1 provides the global error bound:

∥Φ(T,0)−Φ^(T,0)∥≤eϵϵ2k(2k)!.\Vert\Phi(T,0) - \hat\Phi(T,0)\Vert \le e^\epsilon \frac{\epsilon^{2^k}}{(2^k)!}.

This recovers O(ϵ2k)\mathcal{O}(\epsilon^{2^k}) order for small ϵ\epsilon and preserves the doubly exponential depth dependence. We will reflect this in the revised appendix.

[Weakness 4] The memory patterns of practical deep SSMs can generalize across sequence lengths. Thus, I do not quite understand the result in Proposition 3.8.

This is an important point. The construction in Proposition 3.8 is the bounded horizon memorization structure. Our theory shows that a memorization structure built for bounded length TT does not, in general, remain exact for arbitrary lengths. For non-solvable tasks, the structure retains sufficient information up to word length TT, but some words longer than TT can expose lost information in truncation.

This is consistent with previous contributions, separating bounded length vs. arbitrary length state tracking. Liu et al., 2023 reported length-generalization brittleness of transformers trained at fixed length TT, learning the “shortcut” solution. Related expressivity limitations for structured SSMs have also been reported (Merrill et al., 2024). Proposition 3.8 is a Lie-theoretic construction connecting model depth with the memorization solution at finite length TT, providing a loose upper bound.


We hope our responses successfully address the concerns. We will soon revise our manuscript to reflect the feedback accordingly.

I thank the authors for their detailed and constructive responses. All my concerns have been fully resolved. I strongly recommend that the authors explicitly clarify their novel contributions in theoretical parts compared to prior work to better position this paper in the revised version.

We greatly appreciate the reviewer’s thoughtful feedback. Thank you very much for the increased score. We will further clarify our novel theoretical contributions relative to prior work in the revised manuscript.

Reviewer_bh71OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work looks at the problem of state tracking with parallelizable sequence models such as SSMs, and provides a Lie algebraic theoretical analysis of expressivity of single and multi-layer (Abelian) SSMs and also their approximation capability of non-sovable group state-tracking when they cannot exactly simulate a group.

Critical strengths

1- The question is interesting and important: when parallelizable sequence models (especially, diagonal SSMs) are theoretically incapable of tracking a group (for example, the non-solvable type), how reliable are they still to approximately track the group within some given error tolerance?

2-The theory part is very well-written. Overall, even though quite technical, this paper is very easy to read, as the authors have put clear effort to nicely present their result and simplify the presentation.

Critical weaknesses

1- While this work is very well-written and many important references have been mentioned, I believe the authors could work more on connecting their results with the earlier ones, and especially clarifying where theirs add something new to the literature. For example, Lemma 3.1 and Corollary 3.5 have a lot of similarities with the results in [Shakerinava et al., 2026] and [Karuvally et al., 2025]; I believe corollary 3.5 is a restatement of the Theorem in [Karuvally et al., 2025] in terms of Lie algebra rather than group theory terminology. Also, Lemma 3.1 looks very similar to the results in [Shakerinava et al., 2026] (or if not, please clarify). Related to this, in the mathematical preliminaries section, a lot of Lie algebraic concepts have close counterparts in the group theory that have already been used in the recent literature of diagonal SSMs expressivity, and I believe that by making this connection clear, this work will become accessible to a broader readers in this community.

2- I couldn’t find the definition for some quantities, for example:

L.127, gl(n,R)gl(n, R) and GL(n,R)GL(n,R).

L.70: Fi{F_i} without i defined. The definition looks incomplete.

L.76 I believe ideal was not defined before. (Also, an example of where the authors could connect wit current literature, “ideal” is very similar to “normal subgroup” that is known through the definition of “Solvable groups”)

L. 281: It would be better to state directly in the Corollary, what are ϵ\epsilon and kk.

Questions for authors

1- Theorem 3.2 says for a sufficiently small [0,T] …. The appendix clarifies that the shortness of the interval is required for the validity of cutting the Magnus expansion at second-order. I am wondering if this means there is some limitation on applying this theorem on longer sequences. Usually, we expect longer sequence lengths to accumulate more error. Adding more details here helps to better grasp the theorem and its implications and scope.

2- how is theorem 3.4 compatible with the result of [Shakerinava et al.], that learning solvable groups requires at least L layers where L is the length of shortest subnormal chain? Here, the theorem claims that two layers are enough? Are there any minimum requirements on the layers of the 2-layer SSM, like non-diagonality?

4- Do authors consider any precision assumption or are their results applicable to infinite precision? (if so, parts of my comment on Weakness 1 may no longer be valid.)

Clarification:

1- What is meant by symmetry breaking, around L.148?

2- Could the authors state clearly the novel contributions of the work? Currently the theory section looks like a mixture of results that have counterparts in the existing literature (expressivity results), and results that are genuinely novel (approximation capability of (efficient) parallelizable models).

3 comments
AuthorsOpenReview ↗

Response to Reviewer bh71

Thank you for the helpful and constructive feedback. We provide our responses below.

[Q1] Does the locality of Thm 3.2 mean there is some limitation on applying this theorem on longer sequences?

The error construction is not limited to local and guarantees error on any admissible window. Regardless of the window size, the expressivity limitation holds. The short window is only to quantify bound with Magnus terms, Ω2\Omega_2. Quantifying the bound on longer sequences requires local-to-global extension.

We discussed local-to-global extension in App.C.2. Although we usually expect longer sequences to accumulate more error, earlier error can diminish or cancel out with later error and obscure accumulation. We introduced the alignment assumption that prescribes global error accumulation. A useful intuition is repeated local action ABA−1B−1ABA^{-1}B^{-1} (main text Fig. 1). The displacement ∥e−e’∥\Vert e - e’ \Vert can accumulate, and the lower bound linearly grows in the number of repetitions when the commutator of generators is central. This allows us to quantify error bound on longer sequences.

[Q2] How is Thm 3.4 compatible with Shakerinava et al., 2026? Are there any minimum requirements on the layers of the 2-layer SSM, like non-diagonality?

Our results are consistent with the important previous work in Shakerinava et al., 2026.

Thm 3.4 handles Lie-theoretic extension structure, 0→a→g→h→00 \to \mathfrak{a} \to \mathfrak{g} \to \mathfrak{h} \to 0. The first layer carries quotient h\mathfrak{h}, and the second layer corresponds to the ideal a\mathfrak{a}. This is a Lie-theoretic analog of the cascade in Shakerinava et al., Eq. 15 to Eq. 17. Note that Thm 3.4 applies to any finite-dimensional g\mathfrak{g}, not only solvable ones. Therefore, the minimum requirements on the 2-layer SSM in Thm 3.4 depends on chosen g\mathfrak{g} and extension structure. For non-solvable g\mathfrak{g}, one layer must be non-abelian and non-diagonal.

Applying Thm 3.4 to solvable g\mathfrak{g} reveals the compatibility. Cor 3.5 shows that abelian k-layer SSM is enough to represent g\mathfrak{g} with derived length at most kk. Prop 3.3 provides the other direction, showing that an abelian k-layer SSM cannot represent derived length more than kk.

Combining Prop 3.3 and Cor 3.5 achieves the continuous-time, infinite-precision analog of Shakerinava et al., Theorem 2. When a group G has a subnormal series of length k with abelian factors, its shortest such chain (derived series) has length at most k. By interpreting each DCD SSM layer as an abelian factor (their Theorem 1), Shakerinava et al., states that a k-layer DCD SSM can track groups of derived length at most k with finite precision, which is compatible with our statement.

[Q4] Do authors consider any precision assumption?

Our work is built on real arithmetic and does not consider finite precision. We briefly discussed this around L.403 and L.445, but this complex topic is beyond the scope of the current work. As the reviewer kindly noted, our bounds are algebraic rather than precision-based, complementary to Shakerinava et al., 2026.

[Clarification 1] What is meant by symmetry breaking, around L.148?

It is meant to be the order-symmetry breaking, as b(x)\mathbf{b}(x) can extend abelian glin\mathfrak{g}_{\text{lin}} to solvable and non-commuting gaff\mathfrak{g}_{\text{aff}}.

[Clarification 2, Weakness 1] Clarify the contribution with respect to the previous works.

Thank you for helping us clarify the novel contributions of our work. As the reviewer kindly noted, we consider our major contribution the bridge between discrete, exact expressivity results and continuous, quantitative approximation. Thm 3.4 is our theoretical heart, interpreting depth as algebraic extension. Cor 3.7 turns this interpretation into a depth-dependent bound, and Thm 3.2 measures order-sensitive errors.

We interpreted previous key contributions through our lens. Lemma 3.1 is consistent with reported expressivity limitations (Shakerinava et al., 2026; also Liu et al., 2023, Merrill et al., 2024). As discussed above, Prop 3.3 and Cor 3.5 provide a bridge to existing discrete-time contributions. We thank the reviewer for a valuable addition of Karuvally et al., 2025. We will further highlight previous work in the relevant places.

[Weakness 2] Some definitions are not clarified in the main text.

Thank you, we will correctly address them so no ambiguity arises. We especially appreciate the comment on ideal and normal subgroups, reinforcing the bridge between our work and the field.


We will soon revise our manuscript and update the submission. We welcome any further comments in the meantime.

I thank the authors for addressing my questions. I maintain my score of 3.

Also, if this work provides bounds on expressiveness in infinite precision, I believe it is a very important addition to the literature, and I'd suggest that the authors make this clear in their paper, if it is not already.

We deeply appreciate the reviewer’s thoughtful feedback and the suggestion to sharpen our contribution. Our bounds hold in infinite-precision, and we will state this explicitly in the revised paper.

Comments on the paper

1 comment

We thank the reviewers for their insightful and constructive feedback which greatly improved the quality of our work. We have revised our manuscript accordingly:

Major changes

  1. Local-to-global extension proof in App C.6. We replaced the Magnus evaluation output with the Peano-Baker series. Main text claims and doubly exponential rate in depth k remains.
  2. Learnability. We performed additional curriculum task on S3S_3, pretraining the first layer on quotient task S3/C3S_3/C_3 allows a 2-layer AUSSM to solve the S3S_3 word problem (Sec 4.1)
  3. Error behavior in depth and generator mass. Directly probed with additional experiments on the rotation tasks, detailed in App D.2.1 and Figs. 7,8.

Minor changes

  1. Fig. 3b is now in log-log scale with relative error and quadratic trend reference (Thm 3.2). The original panel is moved to Fig. 4, reporting absolute error.
  2. Clarified relation to prior works and to group theory.
  3. Adjusted tone of our claim in the experimental section.
  4. Thm 3.2 and Cor 3.7 are restated for clarity.

We hope the revised manuscript correctly addresses most of the reviewers' feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.