Group-Linearized Backpropagation: Scheduling Residual Paths for Deep Network Training

Residual networks provide exponentially many implicit paths for gradient propagation, yet standard backpropagation still follows a sequential chain whose critical path grows with depth. Existing methods either modify the forward computation, changing the network being trained, or retain the original backward operator and approximate its solution through iterative truncation. We introduce Path-Scheduled Residual Training, built on Group-Linearized Backpropagation (GLBP). Specifically, GLBP partitions the residual chain into contiguous groups, where blocks within each group receive a shared entry gradient and are aggregated in parallel—replacing the standard sequential product within each group with a sum while preserving the product structure across groups. This rewriting requires exactly one block gradient per residual position per step, regardless of grouping granularity. Coarser groupings trade within-group path interactions for reduced serial depth, and which paths are retained is determined by where group boundaries fall rather than by a global order cutoff. The grouping granularity is then scheduled during training: a coarse-to-fine progression starts from short serial depth and reinstates higher-order path content over training, reaching the exact operator at its final stage. Experiments on ResNets and fine-tuning of LLaMA-3.1-8B show that scheduled GLBP achieves near-standard training quality and is competitive with standard backpropagation on downstream benchmarks.