acceptodds
Under review as a conference paper at ICLR 2027

Trajectory-Aligned Gradient Suppression for Test-Time Generalization in Recurrent Transformers

Abstract

Test-time compute scaling has emerged as a powerful paradigm for improving reasoning in large language models. Recurrent transformers offer a natural mechanism for such scaling by repeatedly applying a shared computation block, producing a sequence of latent states that can be unrolled for additional steps at inference. However, additional recurrence often struggles to improve performance beyond the training depth. We study this limitation through a stability–progress trade-off in the latent state dynamics: the model must learn a recurrent operator that remains stable when extrapolated to greater depths, while still making step-by-step progress along the latent trajectory required for multi-step reasoning. We examine this trade-off through the geometry of training gradients. A first-order expansion of the loss under additional recurrence identifies the component of the backpropagated state gradient aligned with the local tangent of the latent-state trajectory as a local channel of depth sensitivity. This trajectory-aligned gradient has a dual role: it can support sequential progress along the recurrent path, but it also determines how the loss changes when computation is extended beyond the training depth. We formulate a curvature-weighted local signal filtering objective for this component and derive a corresponding closed-form suppression rule. Motivated by this analysis, we propose Trajectory-Aligned Gradient Suppression (TAGS), a backward-pass intervention that selectively attenuates tangent-aligned state gradients while preserving transverse gradients for iterative refinement. TAGS adds no parameters, modifies only training-time gradients, and leaves inference unchanged. Across structural reasoning benchmarks (maze navigation, Sudoku, graph shortest path) and symbolic reasoning tasks, TAGS improves test-time compute generalization over backpropagation-through-time (BPTT), truncated BPTT, and fixed-point baselines

Primary area:
generative models
Keywords:
Recurrent transformers, Test-time scaling, OOD generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U3aROpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why recurrent transformers fail to generalize when we unroll them beyond their training depth TtrainT_{\text{train}} at test time. The authors analyze the latent trajectory h0,…,hTh_0, \dots, h_T using discrete Frenet Serret geometry. They use a firstorder Taylor expansion (Eq. 3) to analyze the backpropagated state gradient. They show that its component along the trajectory tangent τt=(ht+1−ht)/∥ht+1−ht∥\tau_t = (h_{t+1} - h_t)/\|h_{t+1} - h_t\| measures the loss change from one extra recurrence step.

To prevent overfitting to the training horizon TtrainT_{\text{train}}, the authors propose Trajectory Aligned Gradient Suppression (TAGS). During the backward pass, TAGS splits each incoming state gradient utu_t into a tangent part and a transverse part. TAGS keeps the transverse part unchanged. It scales down the tangent part by γt=γmax⁡κt/(κt+α)\gamma_t = \gamma_{\max} \kappa_t / (\kappa_t + \alpha), where κt=∥τt+1−τt∥/st\kappa_t = \|\tau_{t+1} - \tau_t\| / s_t. Inference stays unchanged. The authors train 2 layer recurrent transformers from scratch on mazes, Sudoku, and shortest path (Ttrain=32,Ttest=64T_{\text{train}}=32, T_{\text{test}}=64). They also test TAGS with LoRA finetuning on Huginn3.5B across six math benchmarks (Ttrain=16,Ttest=32T_{\text{train}}=16, T_{\text{test}}=32).

Critical strengths

The paper is well written and the main idea is clear, based on that:

  1. The core geometric observation in Equation 3 is clean and intuitive. Because every step shares the recurrent operator fθf_\theta, moving hth_t along vt=ht+1−htv_t = h_{t+1} - h_t equals adding one recurrence step at the end. Projecting the state gradient onto τt\tau_t isolates horizon sensitivity without discarding the orthogonal gradient directions that train the operator.

  2. I also appreciate the author's honesty about the limits of their math in Section 3 and Appendices A and B. They prove with a counterexample in Appendix A.3 that discrete curvature cannot replace Hessian smoothness in Theorem 1. They also give a 1D counterexample in Appendix B.4 (Eq. 61) where TAGS reverses the parameter gradient sign and increases the task loss.

Critical weaknesses

I have few questions and weakness based on the paper:

  1. the theoretical analysis in Section 3 does not justify the curvature gate κt=∥τt+1−τt∥/st\kappa_t = \|\tau_{t+1}-\tau_t\|/s_t. Theorem 1 does not contain κt\kappa_t at all. Proposition 1 bounds the horizon change with a turning term st⟨λt(T),τt+1−τt⟩s_t\langle\lambda_t^{(T)},\tau_{t+1}-\tau_t\rangle and a speed change term (st+1−st)⟨λt(T),τt+1⟩(s_{t+1}-s_t)\langle\lambda_t^{(T)},\tau_{t+1}\rangle. However, Appendix A.4 (Equations 33 and 34) proves that the turning term is first-order in the transverse gradient λt⊥\lambda_t^\perp. It is only second order in the tangent gradient λt∥\lambda_t^\parallel. Suppressing the tangent gradient leaves the leading turning term unchanged and can even increase the total turning term. Proposition 1 depends on chordal turning ωt=∥τt+1−τt∥\omega_t = \|\tau_{t+1}-\tau_t\| and speed change, which the authors never test as gates (line 241). Instead, TAGS divides ωt\omega_t by the step size sts_t to get κt=ωt/st\kappa_t = \omega_t/s_t. When the latent trajectory slows down near a fixed point, sts_t approaches zero and κt\kappa_t blows up as 1/st1/s_t. Thus, κt\kappa_t acts mainly as an inverse step-size gate rather than a measure of trajectory turning.

  2. On maze navigation, TAGS loses to simple Truncated BPTT on path length extrapolation, which is the true test of multi-step depth scaling. Table 1 increases maze path difficulty from 20 50 (ID) to 50 80 (Near OOD) and 80 100 (Far OOD) on a 20×2020\times 20 grid. On both splits, Truncated BPTT (n=8n=8) beats TAGS while using four times fewer backward steps. TAGS only achieves a higher OOD average in Table 1 because of the 25×2525\times 25 and 30×3030\times 30 grid size splits. However, per Table 10, those grid size splits keep path length fixed at 20-50 (the ID range). Therefore, they test spatial token count transfer under 2D RoPE rather than deeper sequential reasoning. Even on Far OOD grid size, hard tangent projection (γt=1\gamma_t = 1, which drops the curvature gate) beats TAGS in Table 4 (60.74% vs. 57.91%).

  3. The experiments omit the standard training baseline for depth extrapolation in recurrent networks: randomizing the forward rollout depth TtrainT_{\text{train}} during training. Prior work on Universal Transformers, algorithmic extrapolation (Bansal et al., 2022), and Huginn (Geiping et al., 2025) samples TtrainT_{\text{train}} from a distribution on each step. That random depth prevents the readout and operator from overfitting to one fixed step TT. Every baseline in Tables 1 to 4 runs a fixed forward horizon (T{train}} = 32 or 1616). In Table 4, Trunc. BPTT (random n) only randomizes the backward window while keeping the forward pass fixed at T=32T=32. Without a randomized forward depth baseline, we cannot tell if TAGS only fixes an artifact of fixed depth training.

  4. The mathematical reasoning gains on Huginn3.5B (Table 3) are small, noisy, and sometimes below the pretrained checkpoint. Geiping et al. (2025) pretrained Huginn with variable recurrence depths. Therefore, finetuning Huginn at a fixed depth Ttrain=16T_{\text{train}} = 16 damages its depth extrapolation on some tasks. For example, pretrained Huginn without finetuning gets 27.95% on AQuA, whereas BPTT drops to 23.7% and TAGS drops to 25.0%. Across SingleEq, MultiArith, and AQuA, the gap between TAGS and BPTT or Truncated BPTT is within one standard deviation. In addition, Table 9 omits the TAGS hyperparameters (γmax⁡,α,ϵ,δnum\gamma_{\max}, \alpha, \epsilon, \delta_{\text{num}}) for Huginn.

  5. The paper never specifies the tensor reduction axis for the geometry operations in Algorithm 1. In a transformer, ht∈RB×n×dh_t \in \mathbb{R}^{B \times n \times d} has a batch axis, a token axis, and a hidden axis. Appendix D (line 1141) notes that memory depends on whether the code computes geometry per state or per token. Thus, the main text never defines which axis TAGS actually uses. Computing ⟨ut,τt⟩\langle u_t, \tau_t \rangle and st=∥vt∥s_t = \|v_t\| per token versus across all nn tokens gives completely different projections. That difference matters especially when nn grows from 400 to 900 on OOD maze grids.

Questions for authors
  1. In Algorithm 1, do you compute sts_t, τˉt\bar{\tau}_t, κ^t\hat{\kappa}_t, and ⟨ut,τˉt⟩\langle u_t, \bar{\tau}_t \rangle independently for each token position in Rd\mathbb{R}^d? Or do you compute them across the flattened sequence state in Rnd\mathbb{R}^{n d}?
  2. What happens in Table 4 if you gate on the unnormalized chordal turning ωt=∥τˉt+1−τˉt∥\omega_t = \|\bar{\tau}_{t+1} - \bar{\tau}_t\| from Proposition 1? What happens if you gate on inverse step size 1/(st+ϵ)1/(s_t + \epsilon) instead of κ^t=ωt/(st+ϵ)\hat{\kappa}_t = \omega_t / (s_t + \epsilon)?
  3. How does TAGS compare against standard BPTT with a randomized forward rollout depth Ttrain∼Uniform(16,48)T_{\text{train}} \sim \text{Uniform}(16, 48) on mazes, Sudoku, and shortest path? Note that Uniform(16,48)\text{Uniform}(16, 48) matches the average training compute of Ttrain=32T_{\text{train}} = 32.
  4. What exact values of γmax⁡,α,ϵ\gamma_{\max}, \alpha, \epsilon, and δnum\delta_{\text{num}} did you use for Huginn in Table 3? Why did Sudoku need γmax⁡=0.2\gamma_{\max} = 0.2 while Maze and Graph Shortest Path used γmax⁡=1.0\gamma_{\max} = 1.0 (Table 7)?
6 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thank you for your rigorous technical review and detailed feedback. Below, we address each of your points in detail and hope these additions and clarifications thoroughly resolve your concerns and encourage you to reconsider your score.

[W1] The theoretical analysis in Section 3 does not justify the curvature gate κt=∥τt+1−τt∥/st\kappa_t = \lVert \tau_{t+1} - \tau_t \rVert / s_t...

We thank the reviewer for the thorough reading of Section 3 and Appendix A.4. Your observations are completely correct: Theorem 1 does not contain κt\kappa_t, and Equations 33–34 show that the step-to-step turning difference st⟨λt(T),τt+1−τt⟩s_t \langle \lambda_t^{(T)}, \tau_{t+1} - \tau_t \rangle is O(ωt)O(\omega_t) in λt⊥\lambda_t^\perp and only O(ωt2)O(\omega_t^2) in λt∥\lambda_t^\parallel, meaning that attenuating λt∥\lambda_t^\parallel does not algebraically shrink Proposition 1's turning term on a fixed rollout. We agree that our original exposition left a gap between our sensitivity bounds and the curvature gate, and that testing ωt\omega_t, speed change, and 1/st1/s_t directly as gates is essential.

In the revised manuscript, we resolve this on both fronts: (1) we clarify the distinct roles of λt∥\lambda_t^\parallel and κt\kappa_t and add Propositions 3 and 4 to formalize curvature gating explicitly as a conditional inductive bias for deeper-horizon credit transfer, and (2) we run controlled gate ablations (ωt\omega_t, ∣st+1−st∣|s_{t+1}-s_t|, sts_t, and 1/st1/s_t) and out-of-sample credit-prediction diagnostics that directly refute the hypothesis that κt\kappa_t acts as an inverse step-size proxy.

1. Resolving Equations 33–34 and Formalizing the Bridge (Propositions 3 & 4)

Target Direction (λt∥\lambda_t^\parallel) vs. Regime Diagnostic (κt\kappa_t): TAGS does not attenuate λt∥\lambda_t^\parallel in order to algebraically minimize the inter-step variation st⟨λt(T),τt+1−τt⟩s_t \langle \lambda_t^{(T)}, \tau_{t+1} - \tau_t \rangle in Proposition 1. Rather:

  • Why attenuate λt∥\lambda_t^\parallel (and preserve λt⊥\lambda_t^\perp): By Theorem 1, the first-order horizon sensitivity DT=L(T+1)−L(T)D_T = L(T+1) - L(T) is governed by ⟨λt(T),vt⟩=st⟨λt(T),τt⟩\langle \lambda_t^{(T)}, v_t \rangle = s_t \langle \lambda_t^{(T)}, \tau_t \rangle, which depends exclusively on λt∥\lambda_t^\parallel (zero contribution from λt⊥\lambda_t^\perp). Attenuating λt⊥\lambda_t^\perp destroys transverse task-solving credit, whereas attenuating λt∥\lambda_t^\parallel directly targets the channel coupling parameter updates to the fixed training horizon TT.
  • Why gate on trajectory geometry: Uniformly removing λt∥\lambda_t^\parallel also discards beneficial sequential-progress credit. Equations 33–34 play a diagnostic role: when the trajectory turns sharply, transverse gradients λt⊥\lambda_t^\perp rotate across adjacent steps into the tangent direction, causing the training-horizon tangent coefficient at=⟨λt(T),τt⟩a_t = \langle \lambda_t^{(T)}, \tau_t \rangle to vary strongly with evaluation depth rather than transferring cleanly to deeper horizons.

To make this relationship rigorous without overclaiming a deductive derivation from Theorem 1, we add two propositions to Section 3 (writing ht+1=F(ht)h_{t+1}=F(h_t), vt=ht+1−htv_t=h_{t+1}-h_t, Jt=DF(ht)J_t=DF(h_t), ψ(h)=ℓ(q(h),y)\psi(h)=\ell(q(h),y), and λt(T)=∇htL(T)\lambda_t^{(T)}=\nabla_{h_t}L(T)):

Proposition 3 (Local Sensitivity Bound): If FF and ψ\psi are C2C^2 along the trajectory, then for 0≤t≤T0 \le t \le T,

∣DT−⟨λt(T),vt⟩∣≤Mψ2∥vT∥2+12∑j=tT−1MF,j∥λj+1(T)∥∥vj∥2\left|D_T-\langle\lambda_t^{(T)},v_t\rangle\right| \le \frac{M_\psi}{2}\|v_T\|^2+\frac{1}{2}\sum_{j=t}^{T-1}M_{F,j}\|\lambda_{j+1}^{(T)}\|\|v_j\|^2

where MψM_\psi and MF,jM_{F,j} bound ∥D2ψ∥\|D^2\psi\| and ∥D2F∥\|D^2F\| on the corresponding segments. Unlike Equation 3—whose remainder depends on the Hessian of the composed remaining-horizon loss—Proposition 3 bounds horizon sensitivity strictly via local one-step quantities.

Proposition 4 (Conditional Criterion for Tangent Retention): Let a=⟨λti(T),τti⟩a=\langle\lambda_t^i(T),\tau_t^i\rangle be the training-horizon tangent coefficient and aˉ=⟨λˉti,τti⟩\bar a=\langle\bar\lambda_t^i,\tau_t^i\rangle the corresponding coefficient for a deeper-horizon loss Lˉ=∣H∣−1∑H∈HL(H)\bar L=|\mathcal H|^{-1}\sum_{H\in\mathcal H}L(H), with transverse transfer scale c⊥>0c_\perp > 0. Over a group GG of states with EG[a2]>0\mathbb E_G[a^2]>0, the attenuation γ∈[0,1]\gamma \in [0,1] minimizing the deeper-horizon credit mismatch EG[((1−γ)a−aˉ/c⊥)2]\mathbb E_G[((1-\gamma)a-\bar a/c_\perp)^2] is:

γG⋆=clip[0,1](1−EG[aaˉ]c⊥EG[a2])\gamma_G^\star = \mathrm{clip}_{[0,1]}\left(1 - \frac{\mathbb{E}_G[a\bar{a}]}{c_\perp \mathbb{E}_G[a^2]}\right)

In particular, if a=atr+adsa = a_{\mathrm{tr}} + a_{\mathrm{ds}} with transferable credit aˉ=c⊥atr\bar a = c_\perp a_{\mathrm{tr}} and EG[atrads]=0\mathbb E_G[a_{\mathrm{tr}}a_{\mathrm{ds}}] = 0, then γG⋆=ρG/(1+ρG)\gamma_G^\star = \rho_G / (1 + \rho_G), where ρG=EG[ads2]/EG[atr2]\rho_G = \mathbb E_G[a_{\mathrm{ds}}^2] / \mathbb E_G[a_{\mathrm{tr}}^2] is the noise-to-signal ratio of depth-specific to transferable tangent credit.

AuthorsOpenReview ↗

Author Response - Part 1 (Continued)

Explicit Inductive Bias: Under the conditional modeling assumption E[ads2∣κ]/E[atr2∣κ]=κ/α\mathbb E[a_{\mathrm{ds}}^2 \mid \kappa] / \mathbb E[a_{\mathrm{tr}}^2 \mid \kappa] = \kappa / \alpha, Proposition 4 directly recovers the rational form of TAGS: γ⋆(κ)=κ/(κ+α)\gamma^\star(\kappa) = \kappa / (\kappa + \alpha) (scaled by γmax⁡\gamma_{\max}). Because ρ(κ)=κ/α\rho(\kappa) = \kappa / \alpha is a modeling assumption rather than a corollary of Proposition 1, we test all candidate geometric and step-size gates empirically below.

2. Why κt=ωt/st\kappa_t = \omega_t / s_t Is Not a 1/st1/s_t Proxy & Line 241 Gate Ablations

Geometric Behavior as st→0s_t \to 0: First, because TAGS applies a bounded rational gate γt=γmax⁡κtκt+α∈[0,γmax⁡]\gamma_t = \gamma_{\max}\frac{\kappa_t}{\kappa_t + \alpha} \in [0, \gamma_{\max}] (with ε\varepsilon-stabilized step norm), attenuation saturates smoothly at γmax⁡\gamma_{\max} rather than diverging. Second, along any smooth latent trajectory undergoing coherent fine-grained refinement, chordal turning scales linearly with step length: ωt=∥τt+1−τt∥=O(st)\omega_t = \lVert \tau_{t+1} - \tau_t \rVert = O(s_t). Thus, as sts_t shrinks during smooth progress, the sts_t factors cancel and κt=ωt/st\kappa_t = \omega_t / s_t converges to the finite discrete Frenet curvature rather than blowing up as 1/st1/s_t. Crucially, κt\kappa_t grows as 1/st1/s_t only when directional changes remain large while displacement stalls (ωt=Ω(1)\omega_t = \Omega(1) as st→0s_t \to 0)—which occurs specifically during post-resolution oscillation and sign-flipping jitter around a fixed point. By contrast, a pure 1/st1/s_t gate blindly suppresses all small steps, including smooth, productive refinement.

Controlled End-to-End Gate Ablations (Line 241): To test this directly alongside the exact terms from Proposition 1, we evaluated Unnormalized Chordal Turning (ωt\omega_t), Speed Change (∣st+1−st∣|s_{t+1}-s_t|), Inverse Step Size (1/st1/s_t), Step Size (sts_t), and Gradient Norm within the exact same tangent-suppression framework on Maze Navigation (identical architecture, optimizer, training budget, calibration, and γmax⁡\gamma_{\max}):

Gate SignalIDNear OOD (Path / Grid)Far OOD (Path / Grid)OOD Avg.
Gradient Norm99.9063.86 / 65.108.69 / 49.6046.81
Inverse Step Size (1/st1/s_t)99.9089.55 / 38.0945.61 / 24.3249.39
Chordal Turning (ωt\omega_t)98.492.0 / 33.674.7 / 25.556.5
Step Size (sts_t)99.7093.26 / 42.6773.73 / 21.4857.78
Speed Change (∣st+1−st∣\lvert s_{t+1}-s_t \rvert)100.096.3 / 41.080.6 / 22.960.2
Curvature (κt=ωt/st\kappa_t = \omega_t / s_t, TAGS)100.0099.31 / 70.2190.33 / 57.9179.44

These controlled ablations establish three clear findings:

  1. κt\kappa_t is definitively not an inverse step-size proxy: Replacing κt\kappa_t with 1/st1/s_t causes a 30.05-point collapse in OOD accuracy (49.39% vs. 79.44%), with severe degradation on spatial grid extrapolation (38.09% / 24.32% vs. 70.21% / 57.91%).
  2. Normalizing turning by step length is critical: Unnormalized chordal turning (ωt\omega_t) reaches only 56.5% OOD average (-22.94 points below κt\kappa_t), confirming that turning must be measured per unit displacement (ωt/st\omega_t / s_t) to separate sharp local jitter from large constructive state transitions.
  3. Turning outperforms scalar speed change: While ∣st+1−st∣|s_{t+1}-s_t| provides a moderate signal (60.2%), it trails curvature by 19.24 points (especially on Grid OOD: 41.0% / 22.9% vs. 70.21% / 57.91%) because scalar speed variation only reflects step rescaling and misses the directional reversals captured by turning.

3. Direct Out-of-Sample Deeper-Horizon Credit Validation

Finally, we directly tested Proposition 4's premise on our 32-bit Binary Addition benchmark (T=32T=32): can candidate signals at training depth predict which tangent gradients fail to transfer to deeper evaluation horizons (H∈[36,64]H \in [36, 64])? Fitting a decile-wise attenuation factor on half the data and measuring held-out squared credit-prediction error on the other half confirms the same hierarchy:

Candidate Predictor at T=32T=32Held-Out Deeper-Horizon Error Reduction
Trajectory Turning (ωt\omega_t)+28.9%
Curvature (κt=ωt/st\kappa_t = \omega_t / s_t)+19.0%
Speed Change (∣st+1−st∣\lvert s_{t+1}-s_t \rvert)+15.3%
Step Index (tt)-0.3%
Speed (sts_t)-5.1%
Inverse Speed (1/st1/s_t)-5.7%

Geometric turning and curvature substantially reduce deeper-horizon credit prediction error (+28.9% and +19.0%), whereas Inverse Speed (1/st1/s_t) actually increases held-out error (-5.7%), failing completely as a predictor of non-transferable credit. Consistently, tracking gate activation relative to per-bit output resolution shows that κt\kappa_t removes 81% of tangent credit after a bit has resolved (during post-resolution idling/jitter), versus only 30% before resolution (during active carry propagation).

AuthorsOpenReview ↗

Author Response - Part 2

[W2] On maze navigation, TAGS loses to simple Truncated BPTT on path length extrapolation, which is the true test of multi-step depth scaling. TAGS only achieves a higher OOD average in Table 1 because of the 25×2525 \times 25 and 30×3030 \times 30 grid size splits. Therefore, they test spatial token count transfer under 2D RoPE rather than deeper sequential reasoning

On a fixed 20×2020 \times 20 maze, pathfinding reduces to iterating a local wavefront operator across adjacent cells. Hard-cutting backpropagation to m=8m=8 steps enforces a strict short-horizon Markovian update bias that happens to match local grid propagation while using fewer backward steps. However, severing gradients beyond mm steps prevents the model from learning long-range sequential credit whenever intermediate steps cannot be supervised purely locally:

  • Existing Fixed-Size Benchmarks (Zero Grid/RoPE Shift): Even within Table 1, Sudoku (Hard) keeps the board fixed at 9×99 \times 9 (81 tokens—zero token-count or positional shift) and increases difficulty solely through deeper multi-step constraint propagation; TAGS outperforms Truncated BPTT by 5.3 points. Likewise, TAGS outperforms Truncated BPTT on Graph Shortest Path (30.4% / 14.1% vs. 20.3% / 5.5%) and Huginn-3.5B Math (67.1% vs. 64.9% OOD; 49.9% vs. 44.0% ID GSM8K).
  • New Algorithmic Depth-Scaling Suite (T=32→64→128T=32 \to 64 \to 128): To evaluate multi-step sequential depth scaling directly—where required sequential computation scales strictly with dependency length (carry chains in Binary Addition, full-sequence parity in Prefix Sum, and multi-hop reachability in Graph Connectivity) without any 2D RoPE confound—we trained models at depth T=32T=32 and evaluated across test-time depths T∈{32,64,128}T \in \{32, 64, 128\}:
Benchmark & OOD SplitTrunc. BPTT (T=32/64/128T=32 / 64 / 128)Random Depth (T=32/64/128T=32 / 64 / 128)TAGS (T=32/64/128T=32 / 64 / 128)Gain vs. Trunc. (T=128T=128)
Binary Addition (128-bit)61.9 / 66.4 / 66.486.3 / 86.6 / 87.197.8 / 98.2 / 98.3+31.9
Binary Addition (256-bit)3.2 / 3.9 / 3.914.6 / 16.7 / 16.533.7 / 33.5 / 34.6+30.7
Prefix Sum (64-bit)9.2 / 74.3 / 74.320.3 / 25.5 / 22.999.2 / 98.6 / 98.5+24.2
Prefix Sum (72-bit)6.0 / 72.6 / 72.53.6 / 14.5 / 9.999.1 / 99.0 / 99.0+26.5
Prefix Sum (128-bit)1.3 / 16.3 / 16.30.4 / 0.3 / 0.414.7 / 18.1 / 20.0+3.7
Graph Connectivity (n=12n=12)0.2 / 0.0 / 0.033.1 / 31.8 / 31.491.7 / 84.7 / 84.2+84.2
Graph Connectivity (n=15n=15)0.0 / 0.0 / 0.00.0 / 0.0 / 0.04.9 / 4.5 / 5.0+5.0

Across every algorithmic task, Trunc, BPTT severely underperforms TAGS, trailing by 31.9 points on 128-bit carry chains, nearly 9×9\times on 256-bit carry chains (34.6% vs. 3.9%), 26.5 points on 72-bit Prefix Sum, and collapsing completely to 0.0% on n=12n=12 Graph Connectivity where multi-hop relational credit must traverse the full rollout. While m=8m=8 reduces backward compute, its short credit window fundamentally breaks non-local sequential learning, whereas TAGS adds under 2% step-time overhead over BPTT (and supports standard activation checkpointing).

Why 25×2525 \times 25 and 30×3030 \times 30 Maze Splits Reflect Recurrent Credit Assignment Rather Than 2D RoPE

In Table 10, holding solution path length at 20–50 on 25×2525 \times 25 and 30×3030 \times 30 mazes was an intentional controlled design to disentangle state-space size from path length. Crucially, the resulting performance gap cannot be attributed to 2D RoPE transfer alone:

  1. Identical Backbone and 2D RoPE Across All Baselines: BPTT, Fixed Point, Truncated BPTT (m=8m=8), Random Depth, STARS, and TAGS share the exact same Transformer architecture, 2D RoPE frequencies, 20×2020 \times 20 training data, and forward inference pass differing solely in their backward gradient rule. If these splits merely tested 2D RoPE token-count generalization, all methods would transfer similarly; instead, OOD Grid accuracy spans from 16.3% / 4.6% (BPTT) and 22.0% / 27.1% (Trunc. BPTT) up to 64.9% / 48.6% (TAGS).
  2. Global Search and Dead-End Pruning Over Larger Graphs: The model is not tracing a pre-labeled 20–50 step path; it must iteratively explore corridors and prune dead-end branches across 2.25×2.25\times as many spatial states (900900 tokens vs. 400400). Truncated BPTT's m=8m=8 window severs early-to-late credit assignment needed to coordinate global search across a larger obstacle graph, while full BPTT over-specializes its state-refinement schedule to 20×2020 \times 20 dynamics. Keeping the full backward window while suppressing horizon-specific tangent credit enables TAGS to generalize across both path length (95.2% / 92.3%) and grid scale (64.9% / 48.6%).
Reviewer_2iGaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Trajectory-Aligned Gradient Suppression (TAGS), which selectively suppresses tangent-aligned gradients based on trajectory curvature to improve test-time depth extrapolation and out-of-distribution generalization in recurrent Transformers.

Critical strengths

The paper provides an insightful geometric perspective on training-depth sensitivity and offers a valuable theoretical explanation of the stability–progress trade-off in recurrent models.

TAGS demonstrates promising OOD generalization across maze navigation, Sudoku, graph shortest-path, and mathematical reasoning tasks, outperforming existing training methods on several benchmarks.

The method requires no additional parameters or modifications to inference and introduces relatively modest training-time overhead, making it practically appealing.

Critical weaknesses

The curvature-based gating mechanism is primarily justified empirically rather than rigorously derived from the theoretical analysis, leaving a gap between the theory and the proposed method.

The improvements on mathematical reasoning benchmarks are relatively modest, and TAGS does not consistently outperform all baselines across individual OOD datasets.

TAGS still requires full BPTT and additional trajectory geometry storage, which may limit its scalability to larger models.

The paper overlooks STARS (Yang et al., ICML 2026), which addresses a closely related stability–effectiveness trade-off in recurrent language models, leaving the positioning and novelty of TAGS insufficiently clarified.

Questions for authors

Could the authors further compare alternative gating signals, such as unnormalized trajectory turning and speed changes, and investigate robustness to different hyperparameter settings?

Can TAGS maintain similar generalization benefits when applied to larger language models and open-ended text generation tasks?

Why does TAGS substantially outperform Truncated BPTT on maze grid-size extrapolation but show relatively weaker performance on path-difficulty extrapolation?

6 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thank you for your thoughtful review and constructive suggestions. Below, we address your concerns and hope these new additions resolve your concerns and support a stronger assessment of our work.

[W1] The curvature-based gating mechanism is primarily justified empirically rather than rigorously derived from the theoretical analysis, leaving a gap between the theory and the proposed method.

We thank the reviewer for identifying this theoretical gap and agree that Equation 3 and Proposition 1 characterize a gradient component associated with horizon sensitivity, but do not by themselves prove that suppressing it improves generalization or separate beneficial sequential progress from harmful depth specialization. We therefore frame curvature weighting explicitly as a theoretically motivated inductive bias rather than a closed-form theorem of Proposition 1.

To strengthen the connection between our geometric analysis and empirical results, we introduce three additions: (1) an accumulated sensitivity bound expressed through one-step Hessians and BPTT adjoints, (2) a conditional criterion for matching training-horizon tangent credit to deeper-horizon credit, and (3) a binary-addition benchmark with controlled continuation and credit-transfer diagnostics.

1. Bounding Local Sensitivity and Credit Transfer

Following the paper's notation, we write the recurrence as ht+1=F(ht)h_{t+1}=F(h_t), with step vector vt=ht+1−htv_t=h_{t+1}-h_t, Jacobian Jt=DF(ht)J_t=DF(h_t), readout loss ψ(h)=ℓ(q(h),y)\psi(h)=\ell(q(h),y), co-state λt(T)=∇htL(T)\lambda_t^{(T)}=\nabla_{h_t}L(T), and terminal horizon change DT=L(T+1)−L(T)D_T=L(T+1)-L(T).

First, to replace the opaque multi-step remainder in Equation 3 with interpretable step-wise geometry, we bound how closely the local tangent-aligned credit ⟨λt(T),vt⟩\langle\lambda_t^{(T)},v_t\rangle at step tt tracks the terminal horizon change DTD_T:

Proposition 3 (Local Bound on Horizon Sensitivity): If FF and ψ\psi are twice continuously differentiable (C2C^2) along the trajectory, then for 0≤t≤T0 \le t \le T,

∣DT−⟨λt(T),vt⟩∣≤Mψ2∥vT∥2+12∑j=tT−1MF,j∥λj+1(T)∥∥vj∥2\left\vert D_T-\langle\lambda_t^{(T)},v_t\rangle\right\vert \le \frac{M_\psi}{2}\Vert v_T\Vert^2+\frac{1}{2}\sum_{j=t}^{T-1}M_{F,j}\Vert\lambda_{j+1}^{(T)}\Vert\Vert v_j\Vert^2

where MψM_\psi and MF,jM_{F,j} bound ∥D2ψ∥\Vert D^2\psi\Vert and ∥D2F∥\Vert D^2F\Vert on the corresponding trajectory segments. Proof sketch: With Jˉt=∫01DF(ht+σvt) dσ\bar J_t=\int_0^1 DF(h_t+\sigma v_t)\,d\sigma, we have vt+1=Jˉtvtv_{t+1}=\bar J_t v_t, so ⟨λt,vt⟩−⟨λt+1,vt+1⟩=⟨λt+1,(Jt−Jˉt)vt⟩\langle\lambda_t,v_t\rangle-\langle\lambda_{t+1},v_{t+1}\rangle=\langle\lambda_{t+1},(J_t-\bar J_t)v_t\rangle. Summing from tt to T−1T-1 and adding the terminal Taylor remainder yields the bound.

Unlike Equation 3 where the remainder depends on the intractable Hessian of the composed (T−t)(T-t)-step rollout, Proposition 3 isolates sensitivity using strictly local, one-step quantities. It shows that step-tt tangent credit faithfully reflects terminal progress when updates are smooth and small, whereas steps with strong operator non-linearity (MF,jM_{F,j}) accumulate discrepancy between local tangent gradients and true horizon changes.

Second, to directly address how to preserve beneficial sequential progress while suppressing harmful depth specialization, we formalize tangent attenuation as an optimal credit-transfer problem: how much training-horizon tangent credit at depth TT should be retained so that the resulting update best matches the credit needed at deeper evaluation horizons?

Proposition 4 (Optimal Tangent Retention Criterion): Let a=⟨λti(T),τti⟩a=\langle\lambda_t^i(T),\tau_t^i\rangle be the training-horizon tangent coefficient and aˉ=⟨λˉti,τti⟩\bar a=\langle\bar\lambda_t^i,\tau_t^i\rangle be the target tangent coefficient for a deeper-horizon loss Lˉ=∣H∣−1∑H∈HL(H)\bar L=\vert\mathcal H\vert^{-1}\sum_{H\in\mathcal H}L(H), with transverse transfer scale c⊥>0c_\perp>0. For any group GG of trajectory states with EG[a2]>0\mathbb E_G[a^2]>0, the attenuation factor γ∈[0,1]\gamma\in[0,1] minimizing the credit-mismatch error EG[((1−γ)a−aˉ/c⊥)2]\mathbb E_G[((1-\gamma)a-\bar a/c_\perp)^2] is:

γG⋆=clip[0,1](1−EG[aaˉ]c⊥EG[a2])\gamma_G^\star = \mathrm{clip}_{[0,1]}\left(1 - \frac{\mathbb{E}_G[a\bar{a}]}{c_\perp \mathbb{E}_G[a^2]}\right)

In particular, if training tangent credit decomposes into orthogonal transferable and depth-specific components, a=atr+adsa=a_{\mathrm{tr}}+a_{\mathrm{ds}} with aˉ=c⊥atr\bar a=c_\perp a_{\mathrm{tr}} and EG[atrads]=0\mathbb E_G[a_{\mathrm{tr}}a_{\mathrm{ds}}]=0, then γG⋆=ρG/(1+ρG)\gamma_G^\star=\rho_G/(1+\rho_G), where ρG=EG[ads2]/EG[atr2]\rho_G=\mathbb E_G[a_{\mathrm{ds}}^2]/\mathbb E_G[a_{\mathrm{tr}}^2].

Proposition 4 shows that optimal suppression acts as a signal-to-noise shrinkage filter rather than an indiscriminate projection. Instead of discarding all tangent gradients, γG⋆=ρG/(1+ρG)\gamma_G^\star=\rho_G/(1+\rho_G) retains tangent credit when transferable sequential progress (atra_{\mathrm{tr}}) dominates (ρG→0\rho_G \to 0) and smoothly increases suppression only where depth-specific artifacts (adsa_{\mathrm{ds}}) outweigh transferable signal (ρG≫1\rho_G \gg 1).

AuthorsOpenReview ↗

Author Response - Part 1 (Continued)

2. Direct Generalization Test: Binary Addition

To test generalization under rigorous length shifts, we introduce the Binary Addition task, in which models must add two nn-bit numbers. Half the examples are uniform; half contain a planted carry chain of length [n/4,n][n/4,n]. Models are trained on 32-bit inputs at T=32T=32 (9,000 examples, 500 epochs) and evaluated on 1,000 held-out examples at extrapolations up to 256 bits.

Method32-bit (Depth 32)32-bit (Depth 64)32-bit (Depth 128)128-bit (Depth 32)128-bit (Depth 64)128-bit (Depth 128)256-bit (Depth 32)256-bit (Depth 64)256-bit (Depth 128)
BPTT100.0100.0100.033.131.930.71.41.51.8
Trunc. BPTT100.0100.0100.061.966.466.43.23.93.9
Random Depth100.0100.0100.086.386.687.114.616.716.5
TAGS (Ours)100.0100.0100.097.898.298.333.733.534.6

TAGS outperforms all baselines across every tested extrapolation length and recurrence depth.

3. Intervention and Credit Diagnostics

Controlled Continuation: To isolate the precise causal impact of our backward rule, we designed an intervention to test whether TAGS could "rescue" a model that was already over-specialized to its training depth. Starting from the exact same completed BPTT checkpoint, we branched the training into two independent continuation paths: running for 10.1k updates using either TAGS or standard BPTT (both with a fresh optimizer and 500-step warmup). Evaluated on 256 fixed 128-bit inputs at depth 128, the BPTT-continued branch degraded from 31% to 27% accuracy. In contrast, switching to the TAGS backward rule reversed this trajectory, driving accuracy up to 65% and cleanly outperforming the BPTT branch at all eight intermediate checkpoints.

GradientStart1.3k2.5k3.8k5.1k6.3k7.6k8.9k10.1k
BPTT31924163850252227
TAGS313341796185596565

Where Attenuation Acts: To demonstrate that TAGS successfully distinguishes between beneficial sequential progress and harmful depth over-refinement, we analyzed exactly when the gate triggers during the binary addition task. We measured gradient attenuation relative to "output resolution" i.e., the specific recurrent step where a bit's final correct value is successfully computed. TAGS removes 81% of the tangent-credit mass after a bit has resolved (when the model is merely idling and prone to over-specializing to the training depth), compared to just 30% before resolution (when active, beneficial computation is occurring). This confirms that the curvature gate acts preferentially, dynamically suppressing idle over-refinement while preserving the majority of the gradient precisely when it is needed to learn the task.

LocationsShare of Tangent CreditMean GateCredit Removed
Before resolution19%0.5530%
At resolution35%0.4133%
After resolution46%0.9381%

Predicting Deeper-Horizon Credit: To validate our choice of geometric gating signals, and to explicitly disprove the hypothesis that our gate acts merely as an inverse speed proxy, we tested how well candidate signals at depth 32 could predict the true gradient credit required at deeper evaluation horizons (averaged across depths 36 to 64). We fit a step-wise attenuation model (one value per decile of a given signal) on half the data and evaluated the squared prediction error on the held-out half. Trajectory turning and curvature yielded the highest error reductions (28.9% and 19.0%, respectively). In contrast, purely speed-based metrics failed to reliably predict deeper-horizon credit, with inverse speed actually increasing prediction error (-5.7%). This provides strong statistical evidence that geometric trajectory information is better than scalar speed metrics for identifying and suppressing harmful depth specialization.

PredictorTurningCurvatureSpeed ChangeSpeedInverse SpeedStep Index
Error Reduction28.9%19.0%15.3%-5.1%-5.7%-0.3%

Bridging to the Curvature Gate: Under the conditional modeling assumption that the depth-specific-to-transferable variance ratio scales linearly with curvature (E[ads2∣κ]/E[atr2∣κ]=κ/α\mathbb{E}[a_{\mathrm{ds}}^2 \mid \kappa] / \mathbb{E}[a_{\mathrm{tr}}^2 \mid \kappa] = \kappa / \alpha), Proposition 4 yields γ⋆(κ)=κ/(κ+α)\gamma^\star(\kappa) = \kappa / (\kappa + \alpha), recovering the exact unit-tangent TAGS gate (Eq. 6 with γmax⁡=1\gamma_{\max} = 1). Because this relies on an explicit modeling assumption rather than Proposition 1 alone, we directly test whether curvature predicts deeper-horizon credit below.

AuthorsOpenReview ↗

Author Response - Part 2

[W2] The improvements on mathematical reasoning benchmarks are relatively modest, and TAGS does not consistently outperform all baselines across individual OOD datasets.

[Q2] Can TAGS maintain similar generalization benefits when applied to larger language models and open-ended text generation tasks?

We thank the reviewer for raising this point. In our original submission, mathematical reasoning was evaluated solely by post-training Huginn-3.5B on grade-school arithmetic splits, where strong pretrained priors naturally compress absolute margins compared to training small structural transformers from scratch. To address both concerns, we (1) clarify the per-dataset OOD behavior, specifically the unseen answer-format shift on AQuA alongside a new Random Depth baseline on Huginn-3.5B, and (2) expand our evaluation to Ouro-1.4B across five harder mathematical reasoning benchmarks (GSM8K, MATH500, ASDiv, SVAMP, and AMC23), where TAGS delivers consistent gains across individual datasets and recurrence depths.

1. Per-Dataset OOD Consistency and Format Retention on Huginn-3.5B

Because Huginn-3.5B already possesses stable pretrained recurrent representations, post-training refines credit assignment without adding parameters or inference overhead. TAGS achieves the highest overall ID average (64.0%) and the highest OOD average (67.1% vs. 65.45% for Random Depth, 65.2% for BPTT, and 64.9% for Trunc. BPTT).

MethodSingleEqMultiArithAddSubAQuA (MCQ)
Huginn (Base)76.9779.8371.9027.95
BPTT80.588.368.123.7
Trunc. BPTT79.387.068.225.2
Random Depth78.788.071.124.0
TAGS (Ours)81.289.672.525.0

Separating the OOD splits by output format explains why AQuA behaves differently from the remaining benchmarks:

  • Format-Matched Numerical Splits (SingleEq, MultiArith, AddSub): Our post-training data consists exclusively of free-form numerical answers. Across every OOD split sharing this free-form numerical format, TAGS consistently outperforms every baseline.
  • Unseen MCQ Format Shift (AQuA): Unlike the other benchmarks, AQuA uses a 5-option multiple-choice (MCQ) format that is completely absent from the training data. Fine-tuning exclusively on free-form numerical generation naturally induces format forgetting on MCQ prompts, causing all fine-tuned methods to degrade relative to the pretrained checkpoint; thus, the objective on AQuA is minimizing performance loss rather than expecting positive task transfer. TAGS limits this format forgetting (-2.95 points) substantially better than BPTT (-4.25 points) and Random Depth (-3.95 points) baselines.

2. Harder Mathematical Reasoning (Ouro-1.4B)

Because grade-school arithmetic splits require relatively shallow multi-step computation, we additionally evaluated TAGS on Ouro-1.4B across five harder symbolic and competition math benchmarks (GSM8K, MATH500, ASDiv, SVAMP, and AMC23) at test-time recurrence depths T∈{2,4,8}T \in \{2, 4, 8\}, comparing against the pretrained base model, Standard Fine-Tuning (SFT), and STARS (Yang et al., ICML 2026):

ModelRecurrencesGSM8KMATH500ASDivSVAMPAMC23Avg.
Ouro-1.4B (Base)276.8854.8074.2781.6735.0064.52
475.2159.6076.5775.6750.0067.41
858.2340.8070.0766.3340.0055.09
Ouro-1.4B-SFT276.0455.8082.0882.3330.0065.25
480.0664.6083.4776.6747.5070.46
860.0539.2075.1068.0022.5052.97
Ouro-1.4B-STARS275.8956.4082.5679.6740.0066.90
481.9667.4084.7384.3352.5074.18
874.4554.8082.5281.0035.0065.55
Ouro-1.4B-TAGS276.7256.7482.9081.0037.5066.97
482.2667.4085.0088.7053.0075.27
875.3455.4083.6081.3034.5066.02

On these harder multi-step tasks, TAGS outperforms SFT by 4.81 points on average at T=4T=4 (+7.86 over Base and +1.09 over STARS). At both T=4T=4 and T=8T=8, TAGS outperforms SFT on all five individual benchmarks, and beats or ties STARS. Doubling test-time depth from T=4T=4 to T=8T=8 causes a steep 17.49-point drop for SFT, whereas TAGS restricts this drop to 9.25 points—outperforming SFT at T=8T=8 by 13.05 percentage points.

Reviewer_4tpBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Recurrent transformer scales computation by repeatedly applying a shared transformer block. This paper investigates why recurrent transformers fail to benefit from additional inference-time recurrence beyond the depth used during training. Authors identify a fundamental stability–progress trade-off. Recurrent models must learn to make meaningful step-by-step progress through their latent states while maintaining stable dynamics that generalize to longer computational trajectories. Motivated by analysis on trajectory-aligned gradient, the authors propose a backward-pass intervention called Trajectory-Aligned Gradient Suppression (TAGS). Across structural reasoning benchmarks, TAGS shows improvement over test-time compute generalization baselines.

Critical strengths
  1. The paper provides an interesting geometric interpretation of test-time compute generalization by connecting tangent-aligned state gradients to sensitivity with respect to recurrence depth. TAGS is a simple training-time intervention without introducing additional model parameters or inference overhead.

  2. TAGS demonstrates improvements over BPTT, truncated BPTT, and fixed-point baselines across maze navigation, Sudoku, and graph shortest-path tasks.

Critical weaknesses
  1. While the first-order expansion establishes a connection between tangent-aligned gradients and recurrence-depth sensitivity, it does not establish that suppressing these gradients improves stability or generalization. In particular, the tangent-aligned component also carries useful information about sequential progress, and the analysis does not provide a principled way to distinguish beneficial from harmful contributions. Similarly, Proposition 1 identifies curvature as a factor in horizon sensitivity but does not demonstrate that curvature-dependent tangent suppression reduces this sensitivity. The proposed suppression objective therefore remains an empirically motivated heuristic rather than a direct consequence of the theoretical results. Although the authors acknowledge these limitations, this gap weakens the central theoretical justification for TAGS.
  2. Most of the substantial improvements are demonstrated on relatively small, task-specific recurrent transformers trained on synthetic structural reasoning problems. There's limited evidence of generalization to other areas such as language-model reasoning. Furthermore, the paper does not compare against training with randomly sampled recurrence horizons, which would directly address over-specialization to a fixed training depth.
  3. Suspect that some parts of the paper is ai-generated. For example, the section titles "Two limits", "Inference is unchanged", "Reading the rule as a local objective", and Figure 3. Also the spacing between paragraphs and tables between Table 2 and Table 3 are not adjusted.
Questions for authors
  1. Can the authors provide more direct empirical evidence connecting tangent-gradient suppression to improved depth generalization?
  2. How robust are the improvements across model scales, recurrence depths, and language reasoning tasks?
4 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thank you for your constructive feedback. We are glad you found our geometric interpretation connecting tangent-aligned state gradients to recurrence-depth sensitivity interesting, and appreciated that TAGS operates purely as a lightweight backward-pass intervention with zero additional parameters or inference overhead.

Below, we address your concerns and hope these theoretical additions, new experiments, and clarifications resolve your concerns and support a stronger evaluation of our work.

[W1] While the first-order expansion establishes a connection between tangent-aligned gradients and recurrence-depth sensitivity, it does not establish that suppressing these gradients improves stability or generalization. The proposed suppression objective therefore remains an empirically motivated heuristic rather than a direct consequence of the theoretical results.

To strengthen the connection between our geometric analysis and empirical results, we introduce three additions: (1) an accumulated sensitivity bound expressed through one-step Hessians and BPTT adjoints, (2) a conditional criterion for matching training-horizon tangent credit to deeper-horizon credit, and (3) a binary-addition benchmark with controlled continuation and credit-transfer diagnostics.

1. Bounding Local Sensitivity and Credit Transfer

Following the paper's notation, we write the recurrence as ht+1=F(ht)h_{t+1}=F(h_t), with step vector vt=ht+1−htv_t=h_{t+1}-h_t, Jacobian Jt=DF(ht)J_t=DF(h_t), readout loss ψ(h)=ℓ(q(h),y)\psi(h)=\ell(q(h),y), co-state λt(T)=∇htL(T)\lambda_t^{(T)}=\nabla_{h_t}L(T), and terminal horizon change DT=L(T+1)−L(T)D_T=L(T+1)-L(T).

First, to replace the opaque multi-step remainder in Equation 3 with interpretable step-wise geometry, we bound how closely the local tangent-aligned credit ⟨λt(T),vt⟩\langle\lambda_t^{(T)},v_t\rangle at step tt tracks the terminal horizon change DTD_T:

Proposition 3 (Local Bound on Horizon Sensitivity): If FF and ψ\psi are twice continuously differentiable (C2C^2) along the trajectory, then for 0≤t≤T0 \le t \le T,

∣DT−⟨λt(T),vt⟩∣≤Mψ2∥vT∥2+12∑j=tT−1MF,j∥λj+1(T)∥∥vj∥2\left\vert D_T-\langle\lambda_t^{(T)},v_t\rangle\right\vert \le \frac{M_\psi}{2}\Vert v_T\Vert^2+\frac{1}{2}\sum_{j=t}^{T-1}M_{F,j}\Vert\lambda_{j+1}^{(T)}\Vert\Vert v_j\Vert^2

where MψM_\psi and MF,jM_{F,j} bound ∥D2ψ∥\Vert D^2\psi\Vert and ∥D2F∥\Vert D^2F\Vert on the corresponding trajectory segments. Proof sketch: With Jˉt=∫01DF(ht+σvt) dσ\bar J_t=\int_0^1 DF(h_t+\sigma v_t)\,d\sigma, we have vt+1=Jˉtvtv_{t+1}=\bar J_t v_t, so ⟨λt,vt⟩−⟨λt+1,vt+1⟩=⟨λt+1,(Jt−Jˉt)vt⟩\langle\lambda_t,v_t\rangle-\langle\lambda_{t+1},v_{t+1}\rangle=\langle\lambda_{t+1},(J_t-\bar J_t)v_t\rangle. Summing from tt to T−1T-1 and adding the terminal Taylor remainder yields the bound.

Unlike Equation 3 where the remainder depends on the intractable Hessian of the composed (T−t)(T-t)-step rollout, Proposition 3 isolates sensitivity using strictly local, one-step quantities. It shows that step-tt tangent credit faithfully reflects terminal progress when updates are smooth and small, whereas steps with strong operator non-linearity (MF,jM_{F,j}) accumulate discrepancy between local tangent gradients and true horizon changes.

Second, to directly address how to preserve beneficial sequential progress while suppressing harmful depth specialization, we formalize tangent attenuation as an optimal credit-transfer problem: how much training-horizon tangent credit at depth TT should be retained so that the resulting update best matches the credit needed at deeper evaluation horizons?

Proposition 4 (Optimal Tangent Retention Criterion): Let a=⟨λti(T),τti⟩a=\langle\lambda_t^i(T),\tau_t^i\rangle be the training-horizon tangent coefficient and aˉ=⟨λˉti,τti⟩\bar a=\langle\bar\lambda_t^i,\tau_t^i\rangle be the target tangent coefficient for a deeper-horizon loss Lˉ=∣H∣−1∑H∈HL(H)\bar L=\vert\mathcal H\vert^{-1}\sum_{H\in\mathcal H}L(H), with transverse transfer scale c⊥>0c_\perp>0. For any group GG of trajectory states with EG[a2]>0\mathbb E_G[a^2]>0, the attenuation factor γ∈[0,1]\gamma\in[0,1] minimizing the credit-mismatch error EG[((1−γ)a−aˉ/c⊥)2]\mathbb E_G[((1-\gamma)a-\bar a/c_\perp)^2] is:

γG⋆=clip[0,1](1−EG[aaˉ]c⊥EG[a2])\gamma_G^\star = \mathrm{clip}_{[0,1]}\left(1 - \frac{\mathbb{E}_G[a\bar{a}]}{c_\perp \mathbb{E}_G[a^2]}\right)

In particular, if training tangent credit decomposes into orthogonal transferable and depth-specific components, a=atr+adsa=a_{\mathrm{tr}}+a_{\mathrm{ds}} with aˉ=c⊥atr\bar a=c_\perp a_{\mathrm{tr}} and EG[atrads]=0\mathbb E_G[a_{\mathrm{tr}}a_{\mathrm{ds}}]=0, then γG⋆=ρG/(1+ρG)\gamma_G^\star=\rho_G/(1+\rho_G), where ρG=EG[ads2]/EG[atr2]\rho_G=\mathbb E_G[a_{\mathrm{ds}}^2]/\mathbb E_G[a_{\mathrm{tr}}^2].

Proposition 4 shows that optimal suppression acts as a signal-to-noise shrinkage filter rather than an indiscriminate projection. Instead of discarding all tangent gradients, γG⋆=ρG/(1+ρG)\gamma_G^\star=\rho_G/(1+\rho_G) retains tangent credit when transferable sequential progress (atra_{\mathrm{tr}}) dominates (ρG→0\rho_G \to 0) and smoothly increases suppression only where depth-specific artifacts (adsa_{\mathrm{ds}}) outweigh transferable signal (ρG≫1\rho_G \gg 1).

AuthorsOpenReview ↗

Author Response - Part 1 (Continued)

2. Direct Generalization Test: Binary Addition

To test generalization under rigorous length shifts, we introduce the Binary Addition task, in which models must add two nn-bit numbers. Half the examples are uniform; half contain a planted carry chain of length [n/4,n][n/4,n]. Models are trained on 32-bit inputs at T=32T=32 (9,000 examples, 500 epochs) and evaluated on 1,000 held-out examples at extrapolations up to 256 bits.

Method32-bit (Depth 32)32-bit (Depth 64)32-bit (Depth 128)128-bit (Depth 32)128-bit (Depth 64)128-bit (Depth 128)256-bit (Depth 32)256-bit (Depth 64)256-bit (Depth 128)
BPTT100.0100.0100.033.131.930.71.41.51.8
Trunc. BPTT100.0100.0100.061.966.466.43.23.93.9
Random Depth100.0100.0100.086.386.687.114.616.716.5
TAGS (Ours)100.0100.0100.097.898.298.333.733.534.6

TAGS outperforms all baselines across every tested extrapolation length and recurrence depth.

3. Intervention and Credit Diagnostics

Controlled Continuation: To isolate the precise causal impact of our backward rule, we designed an intervention to test whether TAGS could "rescue" a model that was already over-specialized to its training depth. Starting from the exact same completed BPTT checkpoint, we branched the training into two independent continuation paths: running for 10.1k updates using either TAGS or standard BPTT (both with a fresh optimizer and 500-step warmup). Evaluated on 256 fixed 128-bit inputs at depth 128, the BPTT-continued branch degraded from 31% to 27% accuracy. In contrast, switching to the TAGS backward rule reversed this trajectory, driving accuracy up to 65% and cleanly outperforming the BPTT branch at all eight intermediate checkpoints.

GradientStart1.3k2.5k3.8k5.1k6.3k7.6k8.9k10.1k
BPTT31924163850252227
TAGS313341796185596565

Where Attenuation Acts: To demonstrate that TAGS successfully distinguishes between beneficial sequential progress and harmful depth over-refinement, we analyzed exactly when the gate triggers during the binary addition task. We measured gradient attenuation relative to "output resolution" i.e., the specific recurrent step where a bit's final correct value is successfully computed. TAGS removes 81% of the tangent-credit mass after a bit has resolved (when the model is merely idling and prone to over-specializing to the training depth), compared to just 30% before resolution (when active, beneficial computation is occurring). This confirms that the curvature gate acts preferentially, dynamically suppressing idle over-refinement while preserving the majority of the gradient precisely when it is needed to learn the task.

LocationsShare of Tangent CreditMean GateCredit Removed
Before resolution19%0.5530%
At resolution35%0.4133%
After resolution46%0.9381%

Predicting Deeper-Horizon Credit: To validate our choice of geometric gating signals, and to explicitly disprove the hypothesis that our gate acts merely as an inverse speed proxy, we tested how well candidate signals at depth 32 could predict the true gradient credit required at deeper evaluation horizons (averaged across depths 36 to 64). We fit a step-wise attenuation model (one value per decile of a given signal) on half the data and evaluated the squared prediction error on the held-out half. Trajectory turning and curvature yielded the highest error reductions (28.9% and 19.0%, respectively). In contrast, purely speed-based metrics failed to reliably predict deeper-horizon credit, with inverse speed actually increasing prediction error (-5.7%). This provides strong statistical evidence that geometric trajectory information is better than scalar speed metrics for identifying and suppressing harmful depth specialization.

PredictorTurningCurvatureSpeed ChangeSpeedInverse SpeedStep Index
Error Reduction28.9%19.0%15.3%-5.1%-5.7%-0.3%

Bridging to the Curvature Gate: Under the conditional modeling assumption that the depth-specific-to-transferable variance ratio scales linearly with curvature (E[ads2∣κ]/E[atr2∣κ]=κ/α\mathbb{E}[a_{\mathrm{ds}}^2 \mid \kappa] / \mathbb{E}[a_{\mathrm{tr}}^2 \mid \kappa] = \kappa / \alpha), Proposition 4 yields γ⋆(κ)=κ/(κ+α)\gamma^\star(\kappa) = \kappa / (\kappa + \alpha), recovering the exact unit-tangent TAGS gate (Eq. 6 with γmax⁡=1\gamma_{\max} = 1). Because this relies on an explicit modeling assumption rather than Proposition 1 alone, we directly test whether curvature predicts deeper-horizon credit below.

AuthorsOpenReview ↗

Author Response - Part 2

[W2] Most of the substantial improvements are demonstrated on relatively small, task-specific recurrent transformers trained on synthetic structural reasoning problems. There's limited evidence of generalization to other areas such as language-model reasoning. Furthermore, the paper does not compare against training with randomly sampled recurrence horizons, which would directly address over-specialization to a fixed training depth.

[Q2] How robust are the improvements across model scales, recurrence depths, and language reasoning tasks?

1. Comparison with Randomized Forward Recurrence Horizons

To address this concern, we evaluated a Random Depth baseline that samples the forward recurrence horizon during training. As shown in the summary table below, TAGS achieves higher out-of-distribution (OOD) accuracy than this baseline across all four task families, proving that the benefits of our gradient suppression mechanism are distinct from simply randomizing training depth.

Task FamilyRandom Depth (OOD Avg)TAGS (OOD Avg)
Maze Navigation41.5%75.3%
Graph Shortest Path15.1%22.3%
Sudoku (Hard)39.7%45.2%
Huginn Math65.45%67.1%

On mazes, TAGS outperforms the Random Depth baseline across all four OOD splits, covering both path-difficulty and grid-size extrapolation. For example, on Far-OOD path difficulty, TAGS achieves 92.3%, compared with just 28.8% for Random Depth. On Huginn, TAGS also achieves higher mean accuracy then Random Depth on each of the six evaluated datasets.

Detailed breakdowns for each task family are provided below for your reference and have been added to the paper:

Maze Navigation

MethodIn DistributionNear OOD (Path)Near OOD (Grid)Far OOD (Path)Far OOD (Grid)OOD Avg.
Random Depth99.981.138.528.817.541.5
TAGS (Ours)100.095.264.992.348.675.3

Graph Shortest Path & Sudoku

MethodGraph ID (n=10)Graph Near (n=12)Graph Far (n=14)Sudoku ID (Easy)Sudoku OOD (Hard)
Random Depth99.221.48.894.839.7
TAGS (Ours)97.630.414.197.845.2

Huginn Mathematical Reasoning

MethodGSM8KMAWPSSingleEqMultiArithAddSubAQuAID AvgOOD Avg
Huginn (Base)43.5971.8576.9779.8371.9027.9557.7264.16
Random Depth48.577.378.788.071.124.062.965.45
TAGS (Ours)49.978.081.289.672.525.064.067.1

2. Generalization to Language-Model Reasoning

To broaden the evidence beyond the structural tasks and provide stronger evidence of generalization to language-model reasoning, we evaluated TAGS on Ouro-1.4B, a 1.4 billion parameter pretrained recurrent language model. We tested it across five diverse mathematical reasoning benchmarks (GSM8K, MATH500, ASDiv, SVAMP, and AMC23) at test-time recurrence depths of 2, 4, and 8.

ModelRecurrences (TT)GSM8KMATH500ASDivSVAMPAMC23Avg.
Ouro-1.4B (Base)276.8854.8074.2781.6735.0064.52
475.2159.6076.5775.6750.0067.41
858.2340.8070.0766.3340.0055.09
Ouro-1.4B-SFT276.0455.8082.0882.3330.0065.25
480.0664.6083.4776.6747.5070.46
860.0539.2075.1068.0022.5052.97
Ouro-1.4B-STARS275.8956.4082.5679.6740.0066.90
481.9667.4084.7384.3352.5074.18
874.4554.8082.5281.0035.0065.55
Ouro-1.4B-TAGS276.7256.7482.9081.0037.5066.97
482.2667.4085.0088.7053.0075.27
875.3455.4083.6081.3034.5066.02

TAGS improves average accuracy at every evaluated depth compared to Standard Fine-Tuning (SFT), achieving the highest peak performance at 4 recurrences (75.27%). Furthermore, at depths 4 and 8, it outperforms SFT on all five benchmarks.

Crucially, TAGS substantially mitigates the degradation commonly seen as test-time recurrence increases: from 4 to 8 recurrences, SFT suffers a steep 17.49-point drop in average accuracy, whereas TAGS restricts this degradation to just 9.25 points. At the deepest evaluated extrapolation (depth 8), TAGS remains highly capable, outperforming the degraded SFT baseline by 13.05 percentage points (66.02% vs 52.97%).

AuthorsOpenReview ↗

Author Response - Part 3

[W3] Suspect that some parts of the paper is ai-generated. For example, the section titles "Two limits", "Inference is unchanged", "Reading the rule as a local objective", and Figure 3. Also the spacing between paragraphs and tables between Table 2 and Table 3 are not adjusted.

Thank you for pointing out these presentation issues. We can categorically confirm that all figures in the paper including Figure 3 were created by the authors using draw.io and no AI tool was used in their creation. Additionally, we have provided the editable drawio source file for Figure 3 as supplementary material to clarify how the figure was created. As noted in our submission's AI disclosure, LLMs were used strictly to help in text polishing and literature discovery, not for generating technical figures.

Regarding the formatting and stylistic concerns, we have renamed the specific headings you highlighted to better reflect their technical content (e.g., Two limits is now Boundary Conditions, and Inference is unchanged is now Inference-Time Dynamics). Finally, we have corrected the LaTeX spacing artifacts around Tables 2 and 3.

[Q1] Can the authors provide more direct empirical evidence connecting tangent-gradient suppression to improved depth generalization?

To provide additional empirical evidence connecting tangent-gradient suppression to improved depth generalization, we have evaluated TAGS across three new algorithmic extrapolation tasks. In addition to the Binary Addition benchmark and the controlled continuation diagnostics described in our response to [W1], we introduce two additional tasks here: Prefix Sum and Graph Connectivity.

Models were trained at a base recurrence depth of T=32T=32 and evaluated at increased test-time depths (T=32,64,128T=32, 64, 128) across multiple out-of-distribution (OOD) problem domains.

1. Prefix Sum (Trained on 32-bit)

Task: Compute the running sum (modulo 2) of a binary sequence.

Method32-bit (ID)64-bit (Near OOD)72-bit (Far OOD)128-bit (Very Far)
Depth3264128326412832641283264128
Trunc. BPTT100.0100.0100.09.274.374.36.072.672.51.316.316.3
Random Depth100.0100.0100.020.325.522.93.614.59.90.40.30.4
TAGS (Ours)100.0100.0100.099.298.698.599.199.099.014.718.120.0

2. Graph Connectivity (Trained on 10 nodes)

Task: Determine if two nodes are connected in a graph. Evaluated on graphs with nn nodes.

Methodn=10 (ID)n=12 (Near OOD)n=15 (Far OOD)
Depth326412832641283264128
BPTT99.399.298.469.569.168.60.40.30.3
Trunc. BPTT100.0100.0100.00.20.00.00.00.00.0
Random Depth100.0100.0100.033.131.831.40.00.00.0
TAGS (Ours)100.0100.0100.091.784.784.24.94.55.0

How this Directly Connects Tangent Suppression to Generalization

These results provide direct evidence that suppressing tangent-aligned gradients produces a fundamentally more stable recurrent operator that transfers cleanly to longer dependencies. By breaking the fixed-depth prior of standard BPTT, TAGS enables superior zero-shot base extrapolation, preventing catastrophic failures on extended inputs (e.g., achieving 97.8% on 128-bit addition and 91.7% on n=12n=12 graphs at depth 32, whereas BPTT drops to 33.1% and 69.5%). Furthermore, on extreme OOD distributions where extra compute is strictly required, TAGS uniquely extracts productive gains from deeper rollouts—climbing from 14.7% at depth 32 to 20.0% at depth 128 on the 128-bit Prefix Sum task, while standard solutions collapse. Crucially, TAGS thoroughly outclasses the "Random Depth" baseline across all three tasks (e.g., 99.1% vs 3.6% on 72-bit Prefix Sum), isolating our geometric intervention as the key driver of success and proving that dynamically suppressing tangent-aligned credit is a fundamentally more effective mechanism for depth stability than injecting random temporal noise. We have added this comprehensive algorithmic suite to the revised manuscript to firmly anchor our theoretical claims with direct empirical evidence.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank all reviewers for their insightful feedback and constructive experimental suggestions, which have significantly strengthened both our theoretical framing and empirical evaluation. All revisions in the updated manuscript are highlighted in blue for ease of review. Below, we summarize the key updates mapped to each reviewer's comments.

Reviewer 4tpB

  • W1: Added Appendix F introducing local horizon-sensitivity bounds, an optimal tangent-credit retention criterion, and causal credit-transfer diagnostics to strengthen the theoretical and mechanistic motivation for TAGS.
  • W2 & Q2: Integrated the Random Depth baseline across Tables 1–3 and added Table 4 presenting mathematical reasoning evaluations on the pretrained Ouro-1.4B recurrent language model.
  • W3: Revised section titles throughout the manuscript to improve structural clarity and narrative flow.
  • Q1: Added Appendix G evaluating TAGS on three additional algorithmic extrapolation benchmarks: Graph Connectivity, Binary Addition, and Prefix Sum.

Reviewer 2iGa

  • W1: Added Appendix F providing local step-wise sensitivity bounds, a conditional credit-retention criterion, and controlled continuation and credit-prediction diagnostics connecting trajectory geometry to deeper-horizon generalization.
  • W2 & Q2: Added Table 4 reporting multi-horizon mathematical reasoning evaluations on Ouro-1.4B.
  • W3: Added a dedicated Computational Overhead and Scalability subsection analyzing peak training memory and wall-clock throughput.
  • W4: Expanded Related Work and added Table 4 to position and benchmark TAGS directly against STARS (Yang et al., ICML 2026).
  • Q1: Updated Table 5 with expanded gating signal ablations isolating the impact of curvature weighting.

Reviewer U3aR

  • W1: Added Appendix F (theoretical bounds and causal credit-transfer diagnostics) and updated Table 5 with comprehensive gate ablations to justify the TAGS formulation.
  • W2: Added Appendix G reporting length and depth extrapolation results on new algorithmic benchmarks, including Prefix Sum and Graph Connectivity.
  • W3 & Q3: Added Random Depth baseline comparisons across Tables 1–3.
  • W4 & Q4: Added Table 4 evaluating Ouro-1.4B across expanded mathematical reasoning benchmarks and updated the hyperparameter tables in the Appendix for full reproducibility.
  • Q2: Updated Table 5 with the requested gating mechanism ablations.
AuthorsOpenReview ↗

General Comment

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions, which have helped us substantially strengthen the paper.

We apologize for posting our rebuttal toward the end of the discussion period. Some of the additional experiments, particularly the Ouro-1.4B training runs required substantial computational time, and we wanted to ensure that the results were completed before sharing our responses.

We have now posted detailed responses to each review and uploaded the revised manuscript, with changes highlighted in blue for ease of reference.

We greatly appreciate your time and consideration, especially given the limited time remaining in the discussion period, and would be grateful for any further feedback on the revised work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.