- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why recurrent transformers fail to generalize when we unroll them beyond their training depth at test time. The authors analyze the latent trajectory using discrete Frenet Serret geometry. They use a firstorder Taylor expansion (Eq. 3) to analyze the backpropagated state gradient. They show that its component along the trajectory tangent measures the loss change from one extra recurrence step.
To prevent overfitting to the training horizon , the authors propose Trajectory Aligned Gradient Suppression (TAGS). During the backward pass, TAGS splits each incoming state gradient into a tangent part and a transverse part. TAGS keeps the transverse part unchanged. It scales down the tangent part by , where . Inference stays unchanged. The authors train 2 layer recurrent transformers from scratch on mazes, Sudoku, and shortest path (). They also test TAGS with LoRA finetuning on Huginn3.5B across six math benchmarks ().
Critical strengths
The paper is well written and the main idea is clear, based on that:
-
The core geometric observation in Equation 3 is clean and intuitive. Because every step shares the recurrent operator , moving along equals adding one recurrence step at the end. Projecting the state gradient onto isolates horizon sensitivity without discarding the orthogonal gradient directions that train the operator.
-
I also appreciate the author's honesty about the limits of their math in Section 3 and Appendices A and B. They prove with a counterexample in Appendix A.3 that discrete curvature cannot replace Hessian smoothness in Theorem 1. They also give a 1D counterexample in Appendix B.4 (Eq. 61) where TAGS reverses the parameter gradient sign and increases the task loss.
Critical weaknesses
I have few questions and weakness based on the paper:
-
the theoretical analysis in Section 3 does not justify the curvature gate . Theorem 1 does not contain at all. Proposition 1 bounds the horizon change with a turning term and a speed change term . However, Appendix A.4 (Equations 33 and 34) proves that the turning term is first-order in the transverse gradient . It is only second order in the tangent gradient . Suppressing the tangent gradient leaves the leading turning term unchanged and can even increase the total turning term. Proposition 1 depends on chordal turning and speed change, which the authors never test as gates (line 241). Instead, TAGS divides by the step size to get . When the latent trajectory slows down near a fixed point, approaches zero and blows up as . Thus, acts mainly as an inverse step-size gate rather than a measure of trajectory turning.
-
On maze navigation, TAGS loses to simple Truncated BPTT on path length extrapolation, which is the true test of multi-step depth scaling. Table 1 increases maze path difficulty from 20 50 (ID) to 50 80 (Near OOD) and 80 100 (Far OOD) on a grid. On both splits, Truncated BPTT () beats TAGS while using four times fewer backward steps. TAGS only achieves a higher OOD average in Table 1 because of the and grid size splits. However, per Table 10, those grid size splits keep path length fixed at 20-50 (the ID range). Therefore, they test spatial token count transfer under 2D RoPE rather than deeper sequential reasoning. Even on Far OOD grid size, hard tangent projection (, which drops the curvature gate) beats TAGS in Table 4 (60.74% vs. 57.91%).
-
The experiments omit the standard training baseline for depth extrapolation in recurrent networks: randomizing the forward rollout depth during training. Prior work on Universal Transformers, algorithmic extrapolation (Bansal et al., 2022), and Huginn (Geiping et al., 2025) samples from a distribution on each step. That random depth prevents the readout and operator from overfitting to one fixed step . Every baseline in Tables 1 to 4 runs a fixed forward horizon (T{train}} = 32 or ). In Table 4,
Trunc. BPTT (random n)only randomizes the backward window while keeping the forward pass fixed at . Without a randomized forward depth baseline, we cannot tell if TAGS only fixes an artifact of fixed depth training. -
The mathematical reasoning gains on Huginn3.5B (Table 3) are small, noisy, and sometimes below the pretrained checkpoint. Geiping et al. (2025) pretrained Huginn with variable recurrence depths. Therefore, finetuning Huginn at a fixed depth damages its depth extrapolation on some tasks. For example, pretrained Huginn without finetuning gets 27.95% on AQuA, whereas BPTT drops to 23.7% and TAGS drops to 25.0%. Across SingleEq, MultiArith, and AQuA, the gap between TAGS and BPTT or Truncated BPTT is within one standard deviation. In addition, Table 9 omits the TAGS hyperparameters () for Huginn.
-
The paper never specifies the tensor reduction axis for the geometry operations in Algorithm 1. In a transformer, has a batch axis, a token axis, and a hidden axis. Appendix D (line 1141) notes that memory depends on whether the code computes geometry per state or per token. Thus, the main text never defines which axis TAGS actually uses. Computing and per token versus across all tokens gives completely different projections. That difference matters especially when grows from 400 to 900 on OOD maze grids.
Questions for authors
- In Algorithm 1, do you compute , , , and independently for each token position in ? Or do you compute them across the flattened sequence state in ?
- What happens in Table 4 if you gate on the unnormalized chordal turning from Proposition 1? What happens if you gate on inverse step size instead of ?
- How does TAGS compare against standard BPTT with a randomized forward rollout depth on mazes, Sudoku, and shortest path? Note that matches the average training compute of .
- What exact values of , and did you use for Huginn in Table 3? Why did Sudoku need while Maze and Graph Shortest Path used (Table 7)?
6 comments
Author Response - Part 1 (Continued)
Explicit Inductive Bias: Under the conditional modeling assumption , Proposition 4 directly recovers the rational form of TAGS: (scaled by ). Because is a modeling assumption rather than a corollary of Proposition 1, we test all candidate geometric and step-size gates empirically below.
2. Why Is Not a Proxy & Line 241 Gate Ablations
Geometric Behavior as : First, because TAGS applies a bounded rational gate (with -stabilized step norm), attenuation saturates smoothly at rather than diverging. Second, along any smooth latent trajectory undergoing coherent fine-grained refinement, chordal turning scales linearly with step length: . Thus, as shrinks during smooth progress, the factors cancel and converges to the finite discrete Frenet curvature rather than blowing up as . Crucially, grows as only when directional changes remain large while displacement stalls ( as )—which occurs specifically during post-resolution oscillation and sign-flipping jitter around a fixed point. By contrast, a pure gate blindly suppresses all small steps, including smooth, productive refinement.
Controlled End-to-End Gate Ablations (Line 241): To test this directly alongside the exact terms from Proposition 1, we evaluated Unnormalized Chordal Turning (), Speed Change (), Inverse Step Size (), Step Size (), and Gradient Norm within the exact same tangent-suppression framework on Maze Navigation (identical architecture, optimizer, training budget, calibration, and ):
| Gate Signal | ID | Near OOD (Path / Grid) | Far OOD (Path / Grid) | OOD Avg. |
|---|---|---|---|---|
| Gradient Norm | 99.90 | 63.86 / 65.10 | 8.69 / 49.60 | 46.81 |
| Inverse Step Size () | 99.90 | 89.55 / 38.09 | 45.61 / 24.32 | 49.39 |
| Chordal Turning () | 98.4 | 92.0 / 33.6 | 74.7 / 25.5 | 56.5 |
| Step Size () | 99.70 | 93.26 / 42.67 | 73.73 / 21.48 | 57.78 |
| Speed Change () | 100.0 | 96.3 / 41.0 | 80.6 / 22.9 | 60.2 |
| Curvature (, TAGS) | 100.00 | 99.31 / 70.21 | 90.33 / 57.91 | 79.44 |
These controlled ablations establish three clear findings:
- is definitively not an inverse step-size proxy: Replacing with causes a 30.05-point collapse in OOD accuracy (49.39% vs. 79.44%), with severe degradation on spatial grid extrapolation (38.09% / 24.32% vs. 70.21% / 57.91%).
- Normalizing turning by step length is critical: Unnormalized chordal turning () reaches only 56.5% OOD average (-22.94 points below ), confirming that turning must be measured per unit displacement () to separate sharp local jitter from large constructive state transitions.
- Turning outperforms scalar speed change: While provides a moderate signal (60.2%), it trails curvature by 19.24 points (especially on Grid OOD: 41.0% / 22.9% vs. 70.21% / 57.91%) because scalar speed variation only reflects step rescaling and misses the directional reversals captured by turning.
3. Direct Out-of-Sample Deeper-Horizon Credit Validation
Finally, we directly tested Proposition 4's premise on our 32-bit Binary Addition benchmark (): can candidate signals at training depth predict which tangent gradients fail to transfer to deeper evaluation horizons ()? Fitting a decile-wise attenuation factor on half the data and measuring held-out squared credit-prediction error on the other half confirms the same hierarchy:
| Candidate Predictor at | Held-Out Deeper-Horizon Error Reduction |
|---|---|
| Trajectory Turning () | +28.9% |
| Curvature () | +19.0% |
| Speed Change () | +15.3% |
| Step Index () | -0.3% |
| Speed () | -5.1% |
| Inverse Speed () | -5.7% |
Geometric turning and curvature substantially reduce deeper-horizon credit prediction error (+28.9% and +19.0%), whereas Inverse Speed () actually increases held-out error (-5.7%), failing completely as a predictor of non-transferable credit. Consistently, tracking gate activation relative to per-bit output resolution shows that removes 81% of tangent credit after a bit has resolved (during post-resolution idling/jitter), versus only 30% before resolution (during active carry propagation).
Author Response - Part 2
[W2] On maze navigation, TAGS loses to simple Truncated BPTT on path length extrapolation, which is the true test of multi-step depth scaling. TAGS only achieves a higher OOD average in Table 1 because of the and grid size splits. Therefore, they test spatial token count transfer under 2D RoPE rather than deeper sequential reasoning
On a fixed maze, pathfinding reduces to iterating a local wavefront operator across adjacent cells. Hard-cutting backpropagation to steps enforces a strict short-horizon Markovian update bias that happens to match local grid propagation while using fewer backward steps. However, severing gradients beyond steps prevents the model from learning long-range sequential credit whenever intermediate steps cannot be supervised purely locally:
- Existing Fixed-Size Benchmarks (Zero Grid/RoPE Shift): Even within Table 1, Sudoku (Hard) keeps the board fixed at (81 tokens—zero token-count or positional shift) and increases difficulty solely through deeper multi-step constraint propagation; TAGS outperforms Truncated BPTT by 5.3 points. Likewise, TAGS outperforms Truncated BPTT on Graph Shortest Path (30.4% / 14.1% vs. 20.3% / 5.5%) and Huginn-3.5B Math (67.1% vs. 64.9% OOD; 49.9% vs. 44.0% ID GSM8K).
- New Algorithmic Depth-Scaling Suite (): To evaluate multi-step sequential depth scaling directly—where required sequential computation scales strictly with dependency length (carry chains in Binary Addition, full-sequence parity in Prefix Sum, and multi-hop reachability in Graph Connectivity) without any 2D RoPE confound—we trained models at depth and evaluated across test-time depths :
| Benchmark & OOD Split | Trunc. BPTT () | Random Depth () | TAGS () | Gain vs. Trunc. () |
|---|---|---|---|---|
| Binary Addition (128-bit) | 61.9 / 66.4 / 66.4 | 86.3 / 86.6 / 87.1 | 97.8 / 98.2 / 98.3 | +31.9 |
| Binary Addition (256-bit) | 3.2 / 3.9 / 3.9 | 14.6 / 16.7 / 16.5 | 33.7 / 33.5 / 34.6 | +30.7 |
| Prefix Sum (64-bit) | 9.2 / 74.3 / 74.3 | 20.3 / 25.5 / 22.9 | 99.2 / 98.6 / 98.5 | +24.2 |
| Prefix Sum (72-bit) | 6.0 / 72.6 / 72.5 | 3.6 / 14.5 / 9.9 | 99.1 / 99.0 / 99.0 | +26.5 |
| Prefix Sum (128-bit) | 1.3 / 16.3 / 16.3 | 0.4 / 0.3 / 0.4 | 14.7 / 18.1 / 20.0 | +3.7 |
| Graph Connectivity () | 0.2 / 0.0 / 0.0 | 33.1 / 31.8 / 31.4 | 91.7 / 84.7 / 84.2 | +84.2 |
| Graph Connectivity () | 0.0 / 0.0 / 0.0 | 0.0 / 0.0 / 0.0 | 4.9 / 4.5 / 5.0 | +5.0 |
Across every algorithmic task, Trunc, BPTT severely underperforms TAGS, trailing by 31.9 points on 128-bit carry chains, nearly on 256-bit carry chains (34.6% vs. 3.9%), 26.5 points on 72-bit Prefix Sum, and collapsing completely to 0.0% on Graph Connectivity where multi-hop relational credit must traverse the full rollout. While reduces backward compute, its short credit window fundamentally breaks non-local sequential learning, whereas TAGS adds under 2% step-time overhead over BPTT (and supports standard activation checkpointing).
Why and Maze Splits Reflect Recurrent Credit Assignment Rather Than 2D RoPE
In Table 10, holding solution path length at 20–50 on and mazes was an intentional controlled design to disentangle state-space size from path length. Crucially, the resulting performance gap cannot be attributed to 2D RoPE transfer alone:
- Identical Backbone and 2D RoPE Across All Baselines: BPTT, Fixed Point, Truncated BPTT (), Random Depth, STARS, and TAGS share the exact same Transformer architecture, 2D RoPE frequencies, training data, and forward inference pass differing solely in their backward gradient rule. If these splits merely tested 2D RoPE token-count generalization, all methods would transfer similarly; instead, OOD Grid accuracy spans from 16.3% / 4.6% (BPTT) and 22.0% / 27.1% (Trunc. BPTT) up to 64.9% / 48.6% (TAGS).
- Global Search and Dead-End Pruning Over Larger Graphs: The model is not tracing a pre-labeled 20–50 step path; it must iteratively explore corridors and prune dead-end branches across as many spatial states ( tokens vs. ). Truncated BPTT's window severs early-to-late credit assignment needed to coordinate global search across a larger obstacle graph, while full BPTT over-specializes its state-refinement schedule to dynamics. Keeping the full backward window while suppressing horizon-specific tangent credit enables TAGS to generalize across both path length (95.2% / 92.3%) and grid scale (64.9% / 48.6%).
Author Response - Part 1
Thank you for your rigorous technical review and detailed feedback. Below, we address each of your points in detail and hope these additions and clarifications thoroughly resolve your concerns and encourage you to reconsider your score.
We thank the reviewer for the thorough reading of Section 3 and Appendix A.4. Your observations are completely correct: Theorem 1 does not contain κt, and Equations 33–34 show that the step-to-step turning difference st⟨λt(T),τt+1−τt⟩ is O(ωt) in λt⊥ and only O(ωt2) in λt∥, meaning that attenuating λt∥ does not algebraically shrink Proposition 1's turning term on a fixed rollout. We agree that our original exposition left a gap between our sensitivity bounds and the curvature gate, and that testing ωt, speed change, and 1/st directly as gates is essential.
In the revised manuscript, we resolve this on both fronts: (1) we clarify the distinct roles of λt∥ and κt and add Propositions 3 and 4 to formalize curvature gating explicitly as a conditional inductive bias for deeper-horizon credit transfer, and (2) we run controlled gate ablations (ωt, ∣st+1−st∣, st, and 1/st) and out-of-sample credit-prediction diagnostics that directly refute the hypothesis that κt acts as an inverse step-size proxy.
1. Resolving Equations 33–34 and Formalizing the Bridge (Propositions 3 & 4)
Target Direction (λt∥) vs. Regime Diagnostic (κt): TAGS does not attenuate λt∥ in order to algebraically minimize the inter-step variation st⟨λt(T),τt+1−τt⟩ in Proposition 1. Rather:
To make this relationship rigorous without overclaiming a deductive derivation from Theorem 1, we add two propositions to Section 3 (writing ht+1=F(ht), vt=ht+1−ht, Jt=DF(ht), ψ(h)=ℓ(q(h),y), and λt(T)=∇htL(T)):
Proposition 3 (Local Sensitivity Bound): If F and ψ are C2 along the trajectory, then for 0≤t≤T,
DT−⟨λt(T),vt⟩≤2Mψ∥vT∥2+21∑j=tT−1MF,j∥λj+1(T)∥∥vj∥2
where Mψ and MF,j bound ∥D2ψ∥ and ∥D2F∥ on the corresponding segments. Unlike Equation 3—whose remainder depends on the Hessian of the composed remaining-horizon loss—Proposition 3 bounds horizon sensitivity strictly via local one-step quantities.
Proposition 4 (Conditional Criterion for Tangent Retention): Let a=⟨λti(T),τti⟩ be the training-horizon tangent coefficient and aˉ=⟨λˉti,τti⟩ the corresponding coefficient for a deeper-horizon loss Lˉ=∣H∣−1∑H∈HL(H), with transverse transfer scale c⊥>0. Over a group G of states with EG[a2]>0, the attenuation γ∈[0,1] minimizing the deeper-horizon credit mismatch EG[((1−γ)a−aˉ/c⊥)2] is:
γG⋆=clip[0,1](1−c⊥EG[a2]EG[aaˉ])
In particular, if a=atr+ads with transferable credit aˉ=c⊥atr and EG[atrads]=0, then γG⋆=ρG/(1+ρG), where ρG=EG[ads2]/EG[atr2] is the noise-to-signal ratio of depth-specific to transferable tangent credit.