acceptodds
Under review as a conference paper at ICLR 2027

Attention as Active Stochastic Control: A Unifying Theory and Architectural Framework

Abstract

For nearly a decade, the Transformer architecture has served as the empirical workhorse of modern artificial intelligence. Despite its ubiquitous deployment, its core mechanism, softmax self-attention, is primarily understood as a heuristic pairwise sequence mixer optimized via gradient descent. Consequently, pervasive failure modes such as prompt fragility, factual hallucinations, representation collapse (over-smoothing), and attention sinks have been addressed largely through ad-hoc engineering heuristics. In this work, we establish a rigorous mathematical unification between Transformer Attention and Active Stochastic Control. Specifically, we demonstrate that along the depth axis, self-attention operates as an *active observer* executing dual control: the query functions as an exploratory control action steering sensor admittance to minimize epistemic entropy, the key matrix defines a directional channel alignment metric, and the softmax operator arises as the exact analytical Bayesian update of a Wonham filter for categorical point-process emissions. Furthermore, viewing residual stacks as continuous-time Hamiltonian interacting particle systems exposes a symplectic leapfrog structure enabling activation memory training. Leveraging this framework, we introduce **Closed-Loop Iterated Active Attention (IAA)**, which resolves multi-hop relational reasoning within a single physical layer, reuses a static key-value observation manifold, and terminates computation dynamically via Kalman innovation norm decay. Finally, we translate foundational principles from robust control (), balanced truncation (Hankel singular values), and Bode's sensitivity integral to provide principled explanations and actionable solutions for transformer jailbreaks, KV-cache compression, and attention sinks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Attention, Stochastic Control, Dual Control, Wonham Filter, Hamiltonian Dynamics, Transformers, Robust Control, Attention Sinks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xjgjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a control-theoretic interpretation of Transformer attention, treating queries as active sensing actions, attention weights as filtering gains, and residual representations as belief states. It introduces Iterated Active Attention (IAA), which recurrently updates queries against fixed keys and values with an update-norm stopping criterion, and a reversible residual architecture claimed to provide Hamiltonian, symplectic dynamics with activation storage independent of depth. It also develops interventions and diagnostics for attention sinks, KV compression, and perturbation sensitivity. Validation includes synthetic multi-hop associative recall, token-level stopping profiles on WikiText-103, activation-memory measurements, and diagnostics on Qwen2.5 models ranging from 0.5B to 7B parameters. The reported results suggest that recurrent querying can improve multi-hop retrieval and that some proposed diagnostics are informative; however, they do not establish several of the stronger mathematical equivalence, convergence, and explanatory claims.

Critical strengths
  • IAA reuses fixed keys and values for recurrent querying, with the reported two-hop recall improvement providing a useful proof of concept.

  • The foundation-model experiments are informative, showing both where the proposed methods help and where they fail.

Critical weaknesses
  • The manuscript appears to depart from the prescribed ICLR formatting: the text font is not standard, the title, abstract heading, and section headings use ordinary mixed-case typography rather than the template’s small caps, and table captions appear below the tables.

  • The central mathematical claims are not established for the architecture actually specified. Proposition 1 does not provide a sufficiently specified latent-state and observation model to derive ordinary attention as an exact Bayesian filter, while Proposition 2 assumes natural-gradient dynamics without showing that the learned update WupdatevaggW_{\mathrm{update}}v_{\mathrm{agg}}, followed by RMSNorm, implements those dynamics. Equations (5)-(8) establish reversible additive coupling, but generic attention and FFN functions need not have the Jacobian symmetry required for canonical symplecticity. Similarly, the row-normalization identity in Eq. (9) supplies no derivation of a Bode sensitivity integral or necessity of an initial-token sink. These gaps affect the paper’s main contribution and require substantive mathematical justification or a substantial narrowing of its claims.

  • The experiments do not isolate a contribution beyond established recurrent attention and reversible architectures, and a central result is internally inconsistent. Multi-hop querying with shared memory and parameters already appears in [1] , adaptive recurrence in [2], and depth-independent activation storage in [3] and [4]. Moreover, Eq. (10) reduces at g=0g=0 to the zero-sink operator explicitly evaluated in [5]. Table 2 compares IAA against a baseline with fewer parameters and fewer sequential attention operations, without the matched recurrent and sufficiently deep baselines needed to attribute the gain to the proposed mechanism. Figure 3 appears to show success concentrated at τ=k\tau=k, followed by collapse for τ>k\tau>k, contradicting the caption and text claiming success for τ≥k\tau\geq k and leaving the claimed convergence and adaptive-computation benefits unresolved.

[1] Sukhbaatar et al. End-To-End Memory Networks. NeurIPS 2015.

[2] Dehghani et al. Universal Transformers. ICLR 2019.

[3] Gomez et al. The Reversible Residual Network: Backpropagation Without Storing Activations. NeurIPS 2017

[4] Kitaev et al. Reformer: The Efficient Transformer. ICLR 2020

[5] Xiao et al. Efficient Streaming Language Models with Attention Sinks. ICLR 2024.

Questions for authors

Please see Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer xjgj

We thank Reviewer xjgj for a thorough review that caught the τ>k\tau > k script bug and formatting issues, and sharpened our mathematical analysis and prior-work discussion ([1]-[5]).

W1: ICLR Formatting and Caption Placement

We restored standard ICLR Times / Small Caps typography across the title, abstract, and headings, and placed all table captions above tables in the main text and appendix.

W2: Mathematical Precision of Propositions 1 and 2, Reversible Coupling, and Sink Logits

  1. Sink-Logit Softmax (Prop. 2, Sec. 4.2.1, App. B; formerly Prop. 1): Defines hypotheses Z∈0,1,…,NZ \in \\{0, 1, \dots, N\\} (with unmatched H0H_0, prior p0p_0, k0=0,v0=0k_0 = \mathbf{0}, v_0 = \mathbf{0}) and Gaussian observation qi∣(Z=j)∼N(kj,dkIdk)q_i \mid (Z=j) \sim \mathcal{N}(k_j, \sqrt{d_k} I_{d_k}) on ∣kj∣2=dk\\|k_j\\|_2 = \sqrt{d_k}, deriving 1dk\frac{1}{\sqrt{d_k}} scaling and exp⁡(g)\exp(g) via Bayes' rule as the exact MMSE estimator v^i=E[v∗∣qi]\hat{v}_i = \mathbb{E}[v^\ast \mid q_i] without circularity.
  2. Contractive Query Iteration (Prop. 1, Sec. 3.1, App. A; formerly Prop. 2): Analyzes the Algorithm 1 map Ti(q)=RMSNorm((1−α)q+αqi(0)+gi(q)⊙WupdateA(q)V)\mathcal{T}_i(q) = \mathrm{RMSNorm}((1-\alpha)q + \alpha q_i^{(0)} + g_i(q) \odot W_{\mathrm{update}} A(q) V) on Sdkdk−1\mathcal{S}_{\sqrt{d_k}}^{d_k-1}. With LRMS=dk/rmin⁡L_{\mathrm{RMS}} = \sqrt{d_k}/r_{\min}, LA=1dk∣K∣2∣V∣2L_A = \frac{1}{\sqrt{d_k}}\\|K\\|_2 \\|V\\|_2, and gate sensitivity LgL_g, Prop. 1 proves Banach contraction when LT=LRMS((1−α)+gˉ∣Wupdate∣2LA+Lg)<1L_{\mathcal{T}} = L_{\mathrm{RMS}}((1-\alpha) + \bar{g}\\|W_{\mathrm{update}}\\|_2 L_A + L_g) < 1, bounds halting error by LRMSϵ∣qi(0)∣2/(1−LT)L_{\mathrm{RMS}}\epsilon\\|q_i^{(0)}\\|_2/(1 - L_{\mathcal{T}}), and states the alignment condition ⟨Δqi(τ),−∇qiH⟩>0\langle \Delta q_i^{(\tau)}, -\nabla_{q_i}\mathcal{H} \rangle > 0 for local entropy descent.
  3. Volume-Preserving Reversible Coupling (Sec. 4.1, App. C): Replaced canonical symplecticity with unit-triangular shear factorization JΦt=(I0∇fFFNI)(I∇fAttn0I)J_{\Phi_t} = \begin{pmatrix} I & 0 \\\\ \nabla f_{\mathrm{FFN}} & I \end{pmatrix}\begin{pmatrix} I & \nabla f_{\mathrm{Attn}} \\\\ 0 & I \end{pmatrix}, giving exact det⁡JΦt=1\det J_{\Phi_t} = 1 and O(1)\mathcal{O}(1)-memory invertibility for arbitrary nonlinear fAttn,fFFNf_{\mathrm{Attn}}, f_{\mathrm{FFN}}.
  4. Softmax Normalization and Sinks (Sec. 4.2.1): Dropped the Bode-integral overstatement, noting that ∑j=1NAij=1\sum_{j=1}^N A_{ij} = 1 forces unmatched queries onto default sinks whereas exp⁡(g)\exp(g) relaxes active weights to ∑j=1NAij=1−Ai,null<1\sum_{j=1}^N A_{ij} = 1 - A_{i,\mathrm{null}} < 1.

W3: Prior Work [1]-[5], Baselines, and Resolving the τ>k\tau > k Collapse

  1. Credit to [1]-[5]: Sec. 1, 3.2, 4.1, 4.2.1, and App. Table 6 credit Memory Networks [1, Sukhbaatar et al., 2015], Universal Transformers [2, Dehghani et al., 2019], RevNet [3, Gomez et al., 2017], Reformer [4, Kitaev et al., 2020], and StreamingLLM [5, Xiao et al., 2024], noting in Sec. 4.2.1 that Eq. (3) at g=0g=0 recovers Xiao et al.'s exp⁡(0)=1\exp(0)=1 zero-sink operator.
  2. Fix of τ>k\tau > k Collapse + Five Baselines (Sec. 5.1.1, Table 1, Fig. 2, App. Table 8): run_mqar_ablations.py had implemented an ungated update (q = self.w_update(v_agg)), omitting Algorithm 1's sigmoid halting gate (src/models/iaa.py). Without gating (as in Memory Networks [1]), once τ=k\tau=k retrieves leaf vjkv_{j_k}, step τ=k+1\tau=k+1 overwrites qiq_i with Wupdatevjk≠kjkW_{\mathrm{update}} v_{j_k} \neq k_{j_k}, stepping off the target. Restoring Algorithm 1's gate (gi(τ)→0g_i^{(\tau)} \to \mathbf{0}) locks onto leaves across k∈2,4,6k \in \\{2,4,6\\} and L,τ∈1..8L,\tau \in \\{1..8\\}:
    • Iso-Param Wide (L=1L=1, 33.0K): 11.2-17.3%; Deep (L∈1..8L \in \\{1..8\\}, L×L\times KV, up to 267.4K): ≤\le 33.3% for L<kL < k, solves k=2,4k=2,4 for L≥kL \ge k, stalls at 32.2% on k=6k=6; Universal Transformer (τ×\tau\times KV): 100% at τ=8\tau=8 on k=2,4k=2,4, stalls at 28.4% on k=6k=6.
    • Ungated Memory Network [1]: 100.0% at τ=k\tau=k, collapses at τ=8>k\tau=8 > k to 34.4-35.6%, whereas Contractive IAA (33.0K, 1×1\times KV) holds 100.0% across all k∈2,4,6k \in \\{2,4,6\\} and τ∈k..8\tau \in \\{k..8\\}.

W4: Empirical Covariances, Row-Stochastic Norms, and Block Bound (Sec. 4.2.2-4.2.3, App. C)

  1. Empirical Covariances (Sec. 4.2.2): Clarified that in nonlinear Transformers, σi=λi(WcWo)\sigma_i = \sqrt{\lambda_i(W_c W_o)} is a task-sensitive diagnostic rather than an LTI truncation bound, validated across Mistral-7B, Qwen2.5-7B, and Qwen2.5-14B (Tables 3-4).
  2. Norms of A(X)A(X) and Block Bound (Prop. 3, App. C): App. C.2 proves ∣A(X)∣∞=ρmax⁡≤1\\|A(X)\\|_\infty = \rho_{\max} \le 1 whereas ∣A(X)∣2≤Nρmax⁡\\|A(X)\\|_2 \le \sqrt{N\rho_{\max}} (reaching N\sqrt{N} on a shared sink), giving conditional 2-norm (αDγ<1\alpha_{\mathcal{D}}\gamma < 1) and unconditional ℓ∞\ell_\infty (ρmax⁡∣WVWO∣∞<1\rho_{\max}\\|W_V W_O\\|_\infty < 1) resolvent bounds, while Prop. 3 bounds the full pre-norm block (RMSNorm\mathrm{RMSNorm}, value + routing-score attention Jacobians, residuals, and SwiGLU FFN) in ∣⋅∣2,∞\\|\cdot\\|_{2,\infty}.
AuthorsOpenReview ↗

Brief Follow-Up on Formatting, Soundness Fixes, and Baselines

Dear Reviewer xjgj,

As the discussion window enters its final days (closing Nov 18), we wanted to briefly check whether our revised manuscript and response resolved the formatting, soundness, and baseline concerns in your review. In particular, we:

  • Restored standard ICLR Small Caps typography and placed all table captions above tables across the main text and appendix.
  • Fixed the ungated script bug behind the τ>k\tau > k drop, achieving 100.0% recall across all k∈2,4,6k \in \\{2, 4, 6\\} and τ∈k..8\tau \in \\{k..8\\} on 1×1\times static KV states against five matched baselines—including Ungated Memory Networks (which collapse at τ=8>k\tau = 8 > k to 34.4-35.6%) and Universal Transformers (Table 1, Fig. 2, App. Table 8)—while crediting prior work [1]-[5] throughout.
  • Replaced circular/overstated claims with self-contained proofs for Propositions 1-3, unit-determinant shear invertibility (det⁡J=1\det J = 1), and row-stochastic vs. spectral norm bounds (Appendices A-C).

Please let us know if anything remains unclear before Nov 18. Thank you again for your thorough review!

I thank the authors for the reply. However, my concern about the applicability of the theoretical guarantees remains. The reply did not explain clearly how the contraction and entropy descent conditions hold in evaluated models. I therefore maintain my score

AuthorsOpenReview ↗

How Contraction and Entropy Descent Hold in Our Evaluated Models

We thank Reviewer xjgj for following up. Your point is well taken: our previous reply stated the algebraic inequalities in Proposition 1 without clearly explaining when and how those two conditions hold in standard unconstrained training (where we do not clamp ∥Wupdate∥2\lVert W_{\mathrm{update}} \rVert_2 during optimization).

In our trained models, query updates operate in two distinct regimes: intermediate hops (τ<k\tau \lt k, moving from one bridge key to the next) vs. final-answer steps (τ≥k\tau \ge k, once the target value is retrieved). Both conditions in Proposition 1 govern this τ≥k\tau \ge k convergence regime, as measured directly on our evaluated MQAR models (run_mqar_ablations.py) below:

1. How the Contraction Condition (LT<1L_{\mathcal{T}} \lt 1, Eq. 1) Holds in Evaluated Models

During intermediate hops (τ<k\tau \lt k), the update map should not be globally contractive: if LT<1L_{\mathcal{T}} \lt 1 held everywhere from τ=1\tau = 1, qi(τ)q_i^{(\tau)} would stay near qi(0)q_i^{(0)} instead of jumping across distinct bridge keys in context (indeed, trained models learn ∥Wupdate∥2≈6.4–6.5\lVert W_{\mathrm{update}} \rVert_2 \approx 6.4\text{--}6.5).

Once the target value is reached (τ≥k\tau \ge k), however, Proposition 1 holds locally without weight clamping because the two learned terms in Eq. (1), LT=LRMS((1−α)+gˉ∥Wupdate∥2LA+Lg)L_{\mathcal{T}} = L_{\mathrm{RMS}}\big((1-\alpha) + \bar{g}\lVert W_{\mathrm{update}} \rVert_2 L_A + L_g\big), vanish via gate closure and peaked softmax:

  • The sigmoid gate learns to close on terminal answers (gˉ\bar{g} drops 8.6×8.6\times, from 0.265→0.0310.265 \to 0.031), and because σ′(z)=σ(z)(1−σ(z))→0\sigma'(z) = \sigma(z)(1-\sigma(z)) \to 0, gate sensitivity LgL_g shrinks simultaneously.
  • Softmax concentrates onto the target key, driving the local attention Jacobian LAL_A toward zero while LRMS→1L_{\mathrm{RMS}} \to 1 near the stationary query.
  • Consequently, gˉ∥Wupdate∥2LA+Lg→0\bar{g}\lVert W_{\mathrm{update}} \rVert_2 L_A + L_g \to 0, reducing the effective gain toward 1−α<11 - \alpha \lt 1: empirically, every consecutive step ratio stays strictly below 11 (0.32→0.93<10.32 \to 0.93 \lt 1) and relative updates shrink 15.3×15.3\times (1.262→0.0821.262 \to 0.082), triggering Step 4's early halt (as also seen on WikiText-103, where 100% of punctuation and 96.7% of determiners halt at τ=1\tau = 1).

2. How the Entropy-Descent Condition (⟨Δqi(τ),−∇qiH⟩>0\langle \Delta q_i^{(\tau)}, -\nabla_{q_i}\mathcal{H}\rangle \gt 0) Holds in Evaluated Models

By Appendix A (Eq. 27), −∇qiH(πi(τ))-\nabla_{q_i}\mathcal{H}(\pi_i^{(\tau)}) points toward keys that already carry high attention weight. Thus:

  • During hop switches (τ<k\tau \lt k): Because qi(τ)q_i^{(\tau)} intentionally shifts away from the current bridge key toward a new key, attention entropy H\mathcal{H} briefly rises (0.06→0.520.06 \to 0.52, with only 44.1% of test examples satisfying ⟨Δqi(τ),−∇qiH⟩>0\langle \Delta q_i^{(\tau)}, -\nabla_{q_i}\mathcal{H}\rangle \gt 0).
  • Once the target key is located (τ≥k\tau \ge k): Cross-entropy loss trains Δqi(τ)\Delta q_i^{(\tau)} to reinforce the target key, so ⟨Δqi(τ),−∇qiH⟩>0\langle \Delta q_i^{(\tau)}, -\nabla_{q_i}\mathcal{H}\rangle \gt 0 holds on 80.5-82.0% (k=2k=2), 97.7-98.0% (k=4k=4), and 99.6-100.0% (k=6k=6) of test examples. This sharpens attention entropy down to H=0.32\mathcal{H} = 0.32 (100.0% accuracy), whereas an Ungated Memory Network (gˉ≡1\bar{g} \equiv 1) keeps overwriting the query, dispersing attention back toward uniform entropy (H=2.95\mathcal{H} = 2.95 vs. ln⁡24=3.18\ln 24 = 3.18 uniform, 30.1% accuracy):
Step τ\tau (k=2k=2, N=256N=256)Mean Gate gˉ(τ)\bar{g}^{(\tau)}Rel. Step ∥Δq(τ)∥2/∥q(0)∥2\lVert \Delta q^{(\tau)} \rVert_2 / \lVert q^{(0)} \rVert_2Step Ratio ∥q(τ+1)−q(τ)∥2/∥q(τ)−q(τ−1)∥2\lVert q^{(\tau+1)}-q^{(\tau)} \rVert_2 / \lVert q^{(\tau)}-q^{(\tau-1)} \rVert_2Test Examples w/ ⟨Δq(τ),−∇qH⟩>0\langle \Delta q^{(\tau)}, -\nabla_q \mathcal{H}\rangle \gt 0IAA Entropy H\mathcal{H} (Acc.)Ungated Entropy H\mathcal{H} (Acc.)
τ=1\tau = 1 (Hop 1)0.2651.262--44.1%0.06 (4.3%)2.63 (5.9%)
τ=2\tau = 2 (Hop 2)0.0740.3900.318 < 144.9%0.52 (75.8%)2.11 (99.6%)
τ=3\tau = 3 (Answer)0.0350.1350.453 < 180.5%0.42 (98.4%)3.12 (19.1%)
τ=4\tau = 4 (Extra step)0.0320.1050.778 < 182.0%0.36 (100.0%)2.89 (34.8%)
τ=8\tau = 8 (Extra step)0.0310.0820.931 < 175.0%0.32 (100.0%)2.95 (30.1%)

We will add this explicit two-regime explanation and empirical diagnostic table directly to Section 3.1 and Appendix A so the scope of Proposition 1 is immediately clear.

Reviewer_73wzOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes a control-theoretic interpretation of Transformer attention and introduces Iterated Active Attention, which repeatedly updates queries against fixed keys and values for multi-hop retrieval.

Critical strengths
  • A control-theory perspective for architectural ideas, including iterative attention, adaptive computation, and sensitivity-aware memory compression.
  • Task-sensitive KV compression: Combining activation statistics with loss gradients identifies directions relevant to prediction. At selected layers, retaining 25% of key dimensions and 75% of value dimensions increases perplexity only from 7.152 to 7.293.

The strongest contribution is the combination of useful architectural ideas and empirical diagnostics; the claimed rigorous unification is less convincing.

Critical weaknesses

The authors make modeling assumptions that reproduce attention, rather than deriving it convincingly from a stochastic control problem.

Entropy descent and symplecticity require conditions that the proposed architecture does not establish or enforce.

The analysis does not establish the claimed guarantees for complete blocks containing attention, normalization, residual connections, and MLPs.

For KV cache obtaining loss gradients requires backward passes through a large model, but calibration runtime and peak memory are not reported.

Questions for authors

None

6 comments
AuthorsOpenReview ↗

Response to Reviewer 73wz

We sincerely thank Reviewer 73wz for their candid, constructive feedback and for highlighting our task-sensitive K25/V75K_{25}/V_{75} KV compression (PPL:7.152→7.293\mathrm{PPL}: 7.152 \to 7.293) as a key strength. We fully agree that our strongest contribution lies in concrete architectural mechanisms and empirical diagnostics rather than grand unification claims, and we restructured the paper accordingly.

W1: Non-Circular Derivation of Scaled Softmax and Sink Logits (Prop. 2, Sec. 4.2.1, App. B)

We replaced the circular Poisson-intensity assumption in the original Proposition 1 with a self-contained spherical Gaussian / von Mises-Fisher observation model:

  • Active keys lie on the RMS-normalized sphere ∣kj∣22=dk\\|k_j\\|_2^2 = d_k (j∈1,…,Nj \in \\{1,\dots,N\\}), alongside an explicit unmatched hypothesis H0H_0 (Z=0Z=0, prior p0p_0, k0=0,v0=0k_0 = \mathbf{0}, v_0 = \mathbf{0}).
  • Observing qi∣(Z=j)∼N(kj,σ2Idk)q_i \mid (Z=j) \sim \mathcal{N}(k_j, \sigma^2 I_{d_k}) with σ2=dk\sigma^2 = \sqrt{d_k} (normalizing Var(qiTkj/σ2)=dk/σ4=1\mathrm{Var}(q_i^T k_j / \sigma^2) = d_k/\sigma^4 = 1) and expanding −∣qi−kj∣22/(2dk)=−(∣qi∣22+dk)/(2dk)+qiTkj/dk-\\|q_i - k_j\\|_2^2 / (2\sqrt{d_k}) = -(\\|q_i\\|_2^2 + d_k)/(2\sqrt{d_k}) + q_i^T k_j / \sqrt{d_k} in Bayes' rule cancels the query-norm factor and yields Eq. (3), v^i=E[v∗∣qi]=∑j=1NAijvj\hat{v}_i = \mathbb{E}[v^\ast \mid q_i] = \sum_{j=1}^N A_{ij} v_j, with Aij=exp⁡(qiTkj/dk)exp⁡(g)+∑m=1Nexp⁡(qiTkm/dk)A_{ij} = \frac{\exp(q_i^T k_j / \sqrt{d_k})}{\exp(g) + \sum_{m=1}^N \exp(q_i^T k_m / \sqrt{d_k})} and g=ln⁡(p0N/pctx)+dk2g = \ln(p_0 N / p_{\mathrm{ctx}}) + \frac{\sqrt{d_k}}{2}.

W2: Explicit Conditions for Contraction, Entropy Reduction, and Reversible Coupling

  1. Contraction and Local Entropy Descent (Prop. 1, Sec. 3.1, App. A): Rather than claiming unconditional natural-gradient entropy descent, Proposition 1 proves Banach contraction on Sdkdk−1\mathcal{S}_{\sqrt{d_k}}^{d_k-1} under the explicit Lipschitz condition LT=LRMS((1−α)+gˉ∣Wupdate∣2LA+Lg)<1L_{\mathcal{T}} = L_{\mathrm{RMS}}((1-\alpha) + \bar{g}\\|W_{\mathrm{update}}\\|_2 L_A + L_g) < 1 (incorporating anchor α\alpha, ∣Wupdate∣2\\|W_{\mathrm{update}}\\|_2, gate bound gˉ\bar{g}, and gate sensitivity LgL_g), bounding early-halting error by LRMSϵ∣qi(0)∣2/(1−LT)L_{\mathrm{RMS}}\epsilon\\|q_i^{(0)}\\|_2/(1 - L_{\mathcal{T}}), and states the acute-alignment condition ⟨Δqi(τ),−∇qiH⟩>0\langle \Delta q_i^{(\tau)}, -\nabla_{q_i}\mathcal{H} \rangle > 0 required for local entropy reduction.
  2. Volume-Preserving Reversible Coupling (Sec. 4.1, Eq. 2, App. C): We removed canonical symplecticity (JTΩJ=ΩJ^T \Omega J = \Omega, which requires symmetric sub-layer Jacobians) and prove what holds for any nonlinear fAttnf_{\mathrm{Attn}} and fFFNf_{\mathrm{FFN}}: the alternating updates factor into unit-triangular shear maps with exact unit Jacobian determinant (det⁡JΦt=1\det J_{\Phi_t} = 1, volume preservation) and exact invertibility in O(1)\mathcal{O}(1) activation memory.

W3: Complete-Block Sensitivity Bound (Prop. 3, Sec. 4.2.3, Eq. 4, App. C)

Proposition 3 bounds the full pre-norm block U(l)=X(l)+FAttn(RMSNorm1(X(l)))U^{(l)} = X^{(l)} + \mathcal{F}_{\mathrm{Attn}}(\mathrm{RMSNorm}_1(X^{(l)})), X(l+1)=U(l)+FFFN(RMSNorm2(U(l)))X^{(l+1)} = U^{(l)} + \mathcal{F}_{\mathrm{FFN}}(\mathrm{RMSNorm}_2(U^{(l)})) in the max-token Euclidean norm (∣Δ∣2,∞=max⁡i∣Δi∣2\\|\Delta\\|_{2,\infty} = \max_i \\|\Delta_i\\|_2): \left\\|\frac{\partial X^{(l+1)}}{\partial X^{(l)}}\right\\|_{2,\infty} \le (1 + \rho_{\max} \gamma C_{\mathrm{Attn}})(1 + C_{\mathrm{FFN}}), where ρmax⁡=max⁡i(1−Ai,null)<1\rho_{\max} = \max_i (1 - A_{i,\mathrm{null}}) < 1 is the sink-logit damping factor, γ=∣WVWO∣2\gamma = \\|W_V W_O\\|_2, CAttnC_{\mathrm{Attn}} collects RMSNorm\mathrm{RMSNorm} and routing-score Lipschitz factors, and CFFN=crms,2LFFNC_{\mathrm{FFN}} = c_{\mathrm{rms},2} L_{\mathrm{FFN}} with LFFN≤2CSiLU∣Y∣2,∞∣Wgate∣2∣Wup∣2∣Wdown∣2L_{\mathrm{FFN}} \le 2 C_{\mathrm{SiLU}} \\|Y\\|_{2,\infty} \\|W_{\mathrm{gate}}\\|_2 \\|W_{\mathrm{up}}\\|_2 \\|W_{\mathrm{down}}\\|_2 bounds the SwiGLU FFN.

W4: KV Calibration Runtime, Peak Memory, and Multi-Family 14B Scaling (Sec. 5.2.1, Tables 3-4)

  1. Single-A100 (40GB) Coupled Calibration (8.1-38.4 s): With frozen weights (requires_grad=False) and layerwise hooks accumulating only 128×128128 \times 128 per-head covariances (Wc,WoW_c, W_o) on the fly before freeing activations, calibration takes 38.4 s / 16.84 GB on Qwen2.5-7B-Instruct (100 windows, +1.64 GB above BF16 weights), 8.1 s / 18.19 GB on Mistral-7B-Instruct-v0.3 (32 windows, ΔPPL=+0.095\Delta\mathrm{PPL} = +0.095), and 11.1 s / 36.26 GB on Qwen2.5-14B-Instruct (32 windows, ΔPPL=+0.175\Delta\mathrm{PPL} = +0.175, 2.7×2.7\times lower than equal 50%), preserving ARC-Challenge and MMLU accuracy (Table 3).
  2. Forward-Only (torch.no_grad()) Variant (4.1 s, 15.61 GB): Added in Table 4 with zero backward passes (ΔPPL=+0.196\Delta\mathrm{PPL} = +0.196), while coupled gradient sensitivity cuts perplexity error by a further 28% to +0.141+\mathbf{0.141}.
AuthorsOpenReview ↗

Brief Follow-Up on Restructured Paper and Soundness Revisions

Dear Reviewer 73wz,

As the discussion period enters its final days (closing Nov 18), we wanted to briefly check whether our revised manuscript and response resolved the soundness and calibration-cost concerns in your review. Guided directly by your feedback, we:

  • Dropped the control-unification framing and replaced the circular softmax assumption with a self-contained spherical Gaussian / von Mises-Fisher MMSE derivation (Proposition 2, Sec. 4.2.1, App. B).
  • Stated explicit architectural conditions for Banach query contraction (LT<1L_{\mathcal{T}} < 1, Proposition 1), volume-preserving reversible shear coupling (det⁡J=1\det J = 1, Sec. 4.1), and complete pre-norm block sensitivity (Proposition 3, App. C).
  • Added single-A100 KV calibration runtime and peak memory (4.1-38.4 s, including a zero-backward torch.no_grad() variant) alongside Mistral-7B-v0.3 and Qwen2.5-14B evaluations on WikiText-103, ARC-Challenge, and MMLU (Tables 3-4).

Please let us know if you have any remaining questions while the discussion window is open, we would be glad to clarify anything further. Thank you again for your constructive review!

How much latency does the iterated approach add? The updated paper is much better, I am raising my score. I am still uncertain how practical this approach is.

Reviewer_aXJCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes attention in Transformers from the stochastic control perspective. It introduces a Closed-Loop Iterative Active Attention (IAA) based on a control-theoretic interpretation, and shows that it can improve multi-hop reasoning tasks. It also uses control theory to diagnose and mitigate attention sinks.

Critical strengths
  1. The paper is generally well structured.
  2. The perspective from control theory is potentially a valuable way of viewing the Transformer layer.
Critical weaknesses
  1. The motivation is not strong and timely enough. The paper motivates its way of interpreting attention by attributing the failure modes, such as prompt fragility, factual hallucinations, and representation collapse, to attention only. This is not really compelling because Transformers have more than just attention. Plus, the paper does not show that its attention addresses these problems. Even from a unified perspective of bridging attention with other fields, stochastic control in this paper, recent advances have done more than just targeting attention; for example, Hyper-SET [1] has already established a broader and principled Transformer design framework, including almost all common components like attention, feedforward layer, normalization, etc., from the energy function perspective. Attention variants like state-space models have also been investigated from the associative memory view and already put into frontier models [2]. These all reduce the paper’s significance and conceptual novelty.
  2. The technical contributions are limited and questionable. The interpretation of attention and the diagnostics of LLMs with stochastic control, to me, just describe the existing components and engineering practices using the language of another field, of which the analogy is somewhat subjective. The authors claim to avoid heuristics to solve attention sink using their more principled control-theoretic language, yet their construction itself also relies on heuristics. It makes no sense that one heuristic is more favorable just because it can be written in seemingly rigorous language.
  3. The experiments are conducted on a limited scale and scope. Only small-scale multi-hop reasoning on MQAR has been shown. Showing IAA’s properties without performance at least as good as Transformers with standard attention, like in Figures 4 and 5, is of limited practical meaning.

[1] Hyper-SET: Designing Transformers via Hyperspherical Energy Minimization. ICLR, 2026.

[2] Kimi K3: Open Frontier Intelligence

Questions for authors
  1. Why does the introduced Closed-Loop Iterative Active Attention (IAA) resolve multi-hop relational reasoning? There appear to be few connections between the derivation of IAA and the practical problem of relational reasoning.
  2. How does IAA differ from the iterative attention in [3]? They seem very similar as iterative cross-attention.
  3. Why is the update in Section 2.3 open-loop, and why is IAA then closed-loop? What does “epistemic uncertainty” specifically represent?

[3] Perceiver: General Perception with Iterative Attention, ICML 2021

2 comments
AuthorsOpenReview ↗

Response to Reviewer aXJC

We thank Reviewer aXJC for recognizing the clear structure of the paper and raising valuable questions on complete blocks (Hyper-SET, Kimi K3), Perceiver, multi-hop retrieval mechanics, and larger-scale validation.

W1 and W2: Complete Transformer Blocks, ML Framing, and Hyper-SET [1] / Kimi K3 [2]

  1. Complete Pre-Norm Blocks and ML Framing (Title, Abstract, Prop. 3): We dropped the claim that control analogies for attention alone explain LLM failure modes, retitled the paper, and removed the control dictionary. In Proposition 3 (Sec. 4.2.3, Eq. 4, App. C), we bound perturbation sensitivity across the complete pre-norm block (RMSNorm1→\mathrm{RMSNorm}_1 \to Sink-Logit Attention →\to Residual →RMSNorm2→\to \mathrm{RMSNorm}_2 \to SwiGLU FFN →\to Residual), showing how normalization, routing-score Jacobians, and SwiGLU gains compose.
  2. Comparison to Hyper-SET [1] and Kimi K3 [2] (Sec. 1 and 2.2): Hyper-SET (Hu et al., 2026) minimizes hyperspherical energy across unrolled blocks, whereas we target within-block query iteration (τ\tau) over static KV states (decoupled from depth LL) and asymmetric KV subspaces (dk≪dvd_k \ll d_v). Meanwhile, linear state-space modules in hybrid models (Kimi K3 [2], Mamba) compress history into fixed-size states, whereas IAA iterates queries over the exact causal KV cache for lossless multi-hop recall without KV growth.

W3: Matched Baselines, Reasoning Benchmarks, and Multi-Family Foundation Models

  1. Five Baselines on Multi-Chain MQAR (k∈2,4,6k \in \\{2,4,6\\}, L,τ∈1..8L,\tau \in \\{1..8\\}, Sec. 5.1.1, Table 1, Figs. 1-2, App. Table 8): On three isomorphic kk-hop chains plus distractors (≤\le 33.3% ceiling for <k< k passes), single-layer Contractive IAA (33.0K params, 1×1\times KV) reaches 100.0% across all k∈2,4,6k \in \\{2,4,6\\} and τ∈k..8\tau \in \\{k..8\\}, beating: (i) Iso-Depth (L=1L=1: 10.6-16.5%); (ii) Iso-Param Wide (L=1L=1, 33.0K: 11.2-17.3%, ruling out parameter count); (iii) Stacked Deep (L∈1..8L \in \\{1..8\\}, L×L\times KV, up to 8.1×8.1\times params: solves k=2,4k=2,4 for L≥kL \ge k, stalls at 32.2% on k=6k=6); (iv) Universal Transformer (τ×\tau\times KV: 100% at τ=8\tau=8 on k=2,4k=2,4, stalls at 28.4% on k=6k=6); and (v) Ungated Memory Network (100% at τ=k\tau=k, collapses to 34.4-35.6% at τ=8\tau=8).
  2. BPE Reasoning, WikiText-103 Parity, and 262K Retrieval (Sec. 5.1.2, Table 2, Figs. 3-4): Directly addressing Figs. 3-4 (formerly 4-5), convergence halting (τˉ∗=2.41\bar{\tau}^\ast = 2.41) skips 51.8% of query steps on WikiText-103 at matched perplexity, and a 3-layer IAA model (8.10M params) beats a 4×4\times deeper 12-layer Transformer (35.27M) on BPE ARC-Challenge (55.0% vs. 50.0%) and BBH Logic (80.0% vs. 64.0%), with 100% needle recall at 262K tokens (margin +2.69+2.69 vs. +0.82+0.82).
  3. Pretrained Qwen2.5 (0.5B-14B) and Mistral-7B-v0.3 (Sec. 5.2, Tables 3-5): Keys pack 80-96% of singular-value mass in the top 25% of dimensions vs. 51-72.9% for values. Zero-retraining K25/V75K_{25}/V_{75} projection halves the KV cache at +0.095+\mathbf{0.095} PPL (Mistral-7B), +0.141+\mathbf{0.141} (Qwen2.5-7B, 5.3×5.3\times lower error than equal 50%), and +0.175+\mathbf{0.175} (Qwen2.5-14B, 2.7×2.7\times lower), preserving ARC-Challenge (63.0% →\to 60.5%, 52.0% →\to 50.5%, 58.0% →\to 55.0%) and MMLU (49.0% →\to 45.0%, 45.5% →\to 44.5%, 35.0% →\to 34.5%). Sink-logit softmax cuts peak sinks by 75% (0.5B/3B/7B) and mean sinks on 0.5B (30.2% →\to 10.2%, ΔPPL=+0.093\Delta\mathrm{PPL} = +0.093 to +0.223+0.223).

Answers to Q1-Q3

  • Q1 (Multi-hop mechanism, Sec. 3.1): In a kk-hop chain (kj1→vj1≡kj2→⋯→vjk)(k_{j_1} \to v_{j_1} \equiv k_{j_2} \to \dots \to v_{j_k}), single-pass qi(0)q_i^{(0)} is formed before reading VV and cannot address kj2k_{j_2}. In IAA, step τ=0\tau=0 retrieves vi(0)≈vj1v_i^{(0)} \approx v_{j_1}, and Wupdatevi(0)W_{\mathrm{update}} v_i^{(0)} steers qi(1)→kj2q_i^{(1)} \to k_{j_2} over the fixed (K,V)(K,V) cache. At τ=k−1\tau = k-1, once leaf vjkv_{j_k} is retrieved, the gate gi(τ)=σ(Wgate[qi(τ);vi(τ)]+bgate)→0g_i^{(\tau)} = \sigma(W_{\mathrm{gate}}[q_i^{(\tau)}; v_i^{(\tau)}] + b_{\mathrm{gate}}) \to \mathbf{0} closes, locking qi∗q_i^\ast for all τ≥k\tau \ge k.
  • Q2 (Vs. Perceiver [3], Sec. 3.2 and App. Table 6): Perceiver cross-attends NN inputs into M≪NM \ll N non-causal latents, breaking token-level causal alignment for autoregressive decoding, and uses fixed unrolled depth without contraction. IAA refines each token's causal query (M=NM=N) over static causal (K,V)(K,V) states with Banach contraction and early halting.
  • Q3 (Single-pass vs. closed-loop and dispersion, Sec. 2.2): Standard attention sends vi(0)v_i^{(0)} straight to the FFN without updating qiq_i against layer ll's (K,V)(K,V) cache; IAA feeds vi(τ)v_i^{(\tau)} back to steer qi(τ+1)q_i^{(\tau+1)}. We replaced "epistemic uncertainty" with exact Shannon entropy H(πi)=−∑jπi,jln⁡πi,j\mathcal{H}(\pi_i) = -\sum_j \pi_{i,j}\ln \pi_{i,j}.
AuthorsOpenReview ↗

Brief Follow-Up on Revised Manuscript and New Experiments

Dear Reviewer aXJC,

As the discussion window enters its final days (closing Nov 18), we wanted to briefly check whether our revised manuscript and response addressed your questions. Following your suggestions, we:

  • Removed the control-theoretic framing and added a complete pre-norm Transformer block sensitivity bound (Proposition 3, RMSNorm\mathrm{RMSNorm} + Sink-Logit Attention + Residual + SwiGLU FFN) alongside positioning against Hyper-SET, Kimi K3, and Perceiver (Secs. 1, 2.2, 3.2, App. Table 6).
  • Expanded our empirical evaluation to five matched baselines on multi-chain MQAR (k∈2,4,6k \in \\{2, 4, 6\\}, L,τ∈1..8L, \tau \in \\{1..8\\}, Table 1, Figs. 1-2), GPT-2 BPE reasoning where a 3-layer IAA model outperforms a 4×4\times deeper 12-layer baseline on ARC-Challenge and BBH (Table 2), and zero-retraining K25/V75K_{25}/V_{75} KV projection across Qwen2.5 (up to 14B) and Mistral-7B-v0.3 (Tables 3-5).

Please let us know if these additions address your concerns or if any point remains unclear before Nov 18. Thank you again for your time and feedback!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Author Response: Summary of Revisions and New Experiments

We sincerely thank the Area Chair and Reviewers (aXJC, 73wz, and xjgj) for their candid, constructive feedback.

Reading all three reviews together highlighted a clear issue in our initial presentation: framing the work as a broad control-theoretic unification obscured our concrete contributions and left mathematical conditions implicit. As Reviewer 73wz aptly noted, "the strongest contribution is the combination of useful architectural ideas and empirical diagnostics; the claimed rigorous unification is less convincing." Guided by your feedback, we removed the control analogies, narrowed every theoretical result to the exact conditions enforced by our architecture, and expanded experiments across model families up to 14B parameters.

While our core methods (Iterated Active Attention and asymmetric KV subspace projection) remain unchanged, we strengthened the paper along three fronts:

  1. Standard ML Framing and Prior-Work Positioning (aXJC, 73wz, xjgj): We retitled the paper Contractive Query Iteration and Asymmetric Key-Value Subspaces in Transformers, dropped the control-jargon dictionary, and rewrote the exposition in standard ML terms. We position IAA against Memory Networks (Sukhbaatar et al., 2015), Universal Transformers (Dehghani et al., 2019), Perceiver (Jaegle et al., 2021), Hyper-SET (Hu et al., 2026), and hybrid state-space models (Kimi K3, Mamba) in Sec. 1 and 3.2 and App. Table 6, and contrast coupled activation-gradient KV subspaces against GQA (Ainslie et al., 2023), shared latents (DeepSeek-V2), EigenAttention (Saxena et al., 2024), Palu (Chang et al., 2025), and StreamingLLM (Xiao et al., 2024) in Sec. 2.1 and 4.2.

  2. Self-Contained Proofs Under Explicit Architectural Conditions (73wz, xjgj, aXJC):

    • Contractive query iteration (Prop. 1, Sec. 3.1, Eq. 1, App. A): Establishes Lipschitz gain conditions on Sdkdk−1\mathcal{S}_{\sqrt{d_k}}^{d_k-1} under which IAA's anchored, gated update is contractive (LT<1L_{\mathcal{T}} < 1), guaranteeing geometric convergence to a unique fixed point and bounded early-halting error.
    • Sink-logit softmax (Prop. 2, Sec. 4.2.1, Eq. 3, App. B): Derived non-circularly via Bayesian MMSE estimation on Sdkdk−1\mathcal{S}_{\sqrt{d_k}}^{d_k-1} under an explicit unmatched event.
    • Reversible coupling and full-block sensitivity (Sec. 4.1 Eq. 2, Prop. 3 Eq. 4, App. C): Replaces canonical symplecticity with unit-triangular volume preservation (det⁡J=1\det J = 1) and bounds max-token Euclidean (∣⋅∣2,∞\\|\cdot\\|_{2,\infty}) sensitivity across a complete pre-norm block (RMSNorm\mathrm{RMSNorm} + sink-logit attention + residual + RMSNorm\mathrm{RMSNorm} + SwiGLU FFN).
  3. Expanded Benchmarks, Multi-Family 14B Scaling, and the τ>k\tau > k Fix (xjgj, aXJC, 73wz):

    • Resolved the τ>k\tau > k ablation bug across k∈2,4,6k \in \\{2,4,6\\}, L,τ∈1..8L,\tau \in \\{1..8\\} (Table 1, Figs. 1-2, App. Table 8): Restoring Algorithm 1's sigmoid halting gate (omitted in the original ablation script) eliminates over-iteration collapse (100.0% across all τ∈k..8\tau \in \\{k..8\\}) against five matched baselines.
    • BPE reasoning, halting, and 262K retrieval (Sec. 5.1.2, Table 2, Figs. 3-4): A 3-layer IAA model beats a 4×4\times deeper 12-layer Transformer on ARC-Challenge and BBH while convergence halting skips 51.8% of query steps on WikiText-103 at matched perplexity.
    • Cross-family and 14B KV compression + calibration cost (Sec. 5.2, Tables 3-5): Evaluated Mistral-7B-Instruct-v0.3 and Qwen2.5-14B-Instruct alongside Qwen2.5-0.5B/3B/7B on WikiText-103, ARC-Challenge, and MMLU, with single-A100 calibration time (4.1-38.4 s) and peak memory.
AreaMethodKey Result in Revised ManuscriptLocation
Multi-HopSingle-layer IAA (1×1\times KV)100.0% on k∈2,4,6k \in \\{2,4,6\\} for all τ∈k..8\tau \in \\{k..8\\} (vs. 34.4-35.6% ungated at τ=8\tau=8; 32.2% 8-layer Deep on k=6k=6)Sec. 5.1.1, Table 1, Figs. 1-2
Reasoning3-layer IAA vs. 12-layer BaseBeats 4×4\times deeper baseline on ARC-C (55% vs. 50%) and BBH (80% vs. 64%); 100% recall at 262KSec. 5.1.2, Table 2
EfficiencyConvergence halt + reversibleSkips 51.8% of query steps on WikiText-103 at matched PPL; O(1)\mathcal{O}(1) activation memory (1.00 MB)Sec. 4.1, 5.1.2, Figs. 3-4
KV CacheZero-retraining K25/V75K_{25}/V_{75}5.3×5.3\times (7B) and 2.7×2.7\times (14B) lower ΔPPL\Delta\mathrm{PPL} than equal 50%; preserves ARC-C and MMLU in 8-38 s calib.Sec. 5.2.1, Tables 3-4, Fig. 5
SinksSoftmax sink logit (exp⁡(g)\exp(g))Cuts peak-layer sinks by 75% (0.5B/3B/7B) and mean sinks on 0.5B (30.2% →\to 10.2%, ΔPPL=+0.093\Delta\mathrm{PPL} = +0.093 to +0.223+0.223)Sec. 5.2.2, Table 5, Fig. 7

We also restored standard ICLR Times / Small Caps typography and placed all table captions above tables (xjgj).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.