- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper revisits worst-case gradient failures of first-order residual connections and proposes a second-order recurrence using two previous layer states. Its theoretical argument uses the determinant of an augmented-state Jacobian to establish a non-vanishing spectral-norm property, and then claims a non-vanishing parameter-gradient guarantee. The practical update replaces the ordinary skip with a negative two-layer skip and a scaled tanh-transformed module output. Experiments report ten-run comparisons on synthetic multihop retrieval and multitask parity, plus 12/24/36-layer language models trained on 10B FineWeb-Edu tokens and evaluated on perplexity, standard zero-shot tasks, and BABILong.
Critical strengths
- The proposed connectivity change is simple enough to be experimentally useful, and the synthetic results cover multiple initializations and a much deeper regime. In particular, testing 128-layer models on concrete retrieval/parity tasks is more informative than showing that a modified residual can merely run. The language-model results also indicate that the alternative can produce usable models with broadly comparable conventional benchmark performance. This supports continued investigation of the architecture even though it does not establish the claimed guarantee.
Critical weaknesses
-
Corollary 4.4 is false as stated. Consider the scalar recurrence of Eq. (7) with alpha=0, beta=-1, T=2, x_{-1}=1, x_0=-1, W_0=W_1=0, and the differentiable loss L(x_2)=x_2. Then x_1=-1 and x_2=1, so the intermediate states and dL/dx_2 are nonzero. However, dL/dW_0=W_1 x_0=0, whereas Eq. (10) claims a positive parameter-gradient lower envelope with right-hand side |dL/dx_2| |x_0|=1. This even uses the practical initial-condition relation x_0=-x_{-1}. The augmented state still has a norm-preserving Jacobian in this example; the problem is that a lower bound on the largest singular value does not preserve every loss-gradient direction or every parameter-gradient projection. Consequently, Appendix A.3's Eq. (23) cannot justify the claimed bound. This directly breaks the main inference that the architecture provably prevents the gradients relevant to learning from vanishing. In addition, Proposition 4.2's equality to 1 needs qualification for |beta|>1: the 2d-dimensional determinant argument implies sigma_max >= |beta|^(T/2)>1.
-
The empirical evidence does not independently establish the mechanism claimed by the theory. Eq. (12) changes both the skip connectivity and the module output through scaled tanh. Appendix C.1 ablates initialization, but it does not isolate the two-layer skip from tanh or show actual parameter-gradient propagation across depth. Table 1's standard language-model results are essentially comparable, and the language-model experiments do not report the repeated-run uncertainty supplied for the synthetic tasks. After correcting the theorem, an attribution ablation and gradient measurements would be necessary to establish whether the observed deep-task improvements arise from the proposed anti-vanishing mechanism.
Questions for authors
-
How do you reconcile Corollary 4.4 with the scalar counterexample above? What corrected theorem, and what additional assumptions on loss direction, readout, or Jacobian singular values, would guarantee non-vanishing parameter gradients rather than only a non-vanishing augmented-state operator norm?
-
Can you correct Proposition 4.2's equality for |beta|>1 and distinguish the valid determinant/spectral-norm property from the stronger optimization conclusion throughout the paper?
-
Can you isolate second-order connectivity from scaled tanh and measure per-layer parameter-gradient norms and singular-value behavior on the deep synthetic models? I recognize the initialization ablation already provided; the missing control concerns the remaining architectural changes. A valid revised theory and a mechanism-specific empirical result would be needed to change my recommendation.
Response to Reviewer rvyc
Thanks for your feedback. We apologize for the delay in response as we needed to rerun our full set of experiments (which we will explain below). In response to your specific questions:
We thank you for pointing this out: our original proof did not account for the norm-preserving direction of the residual Jacobian possibly lying inside the loss gradient's nullspace. We have updated the previous claim (Corollary 4.4) to a stronger one (Proposition 4.4). The key differences are that: (1) we need to additionally assume an upper bound on the spectral norms of the weight matrices Wt. The trade-off here is that this bound is exponentially tighter w.r.t. depth (can be improved and we are doing so in future iterations), but it still ensures strictly positive norm on dL/dW on all depths, preserving the central claim of our paper. Importantly, note that under the same conditions, the standard first-order residual can still vanish, so there is still a material difference between the two systems. Also, (2) we need to connect our model to the last two layers instead of the last layer alone. We have rerun our language experiments to reflect this new change. We note that it has made our results stronger, resulting in an improvement in validation PPL and downstream BABILong reasoning on all depths.
The actual sign in Proposition 4.2 should be ≥ instead of =. That was a typo and we've fixed the statement. The actual proof remains unchanged. Regarding your comment about "distinguishing the valid determinant/spectral-norm property from the stronger optimization conclusion", please see Point 1 above.
Please see Appendix C for additional ablations. In all our experiments, the tanh scaling made very little difference and it was more of an arbitrary choice for optimization safety rather than any sort of performance gain. We also attached some plots for gradient norms in Appendix C.3, showing that the first-order residual has smaller gradient norms on average (learning rate and all other hyperparameters are held constant) and gradient norms tends to decay more rapidly mid-training. Finally, we note that despite the language model performance being only slightly better, our model achieves better (synthetic) or slightly better (language) performance for essentially negligible cost and further comes with stronger gradient guarantees.