- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Group-Linearized Backpropagation (GLBP), an approximate backward propagation method for residual networks that builds upon Highway BP (Fagnou et al., 2025).
The method partitions residual blocks into groups and replaces within-group products of residual Jacobians with additive approximations, allowing parallel computation of local gradients while preserving the forward computation.
The authors propose to progressively refine the residual block grouping during training. Experiments on ResNets and LLaMA-3.1-8B investigate gradient fidelity and downstream performance, while additional Qwen3 experiments report training-step speedups over standard backpropagation.
Critical strengths
-
The grouping-based approximation provides a simple and interpretable way to control residual-path interactions. The reported Qwen3 experiments also demonstrate end-to-end speedups over standard backpropagation in the tested hardware configurations.
-
The paper is easy to follow and quite enjoyable to read.
Critical weaknesses
1. Limited novelty and scientific significance GLBP essentially applies a first-order approximation to grouped residual Jacobian products, with theoretical results that largely follow from direct algebraic observations. Highway-BP (Fagnou et al., ICLR 2025) already introduced approximate parallel backpropagation through residual paths. Moreover, its Section 5.3 analyzes how gradient approximation quality evolves during training, and its conclusion explicitly proposes increasing the number of iterations during training, potentially ending with exact backpropagation. GLBP implements this principle through group refinement rather than additional iterations, but the underlying scheduling idea is not new.
Overall, I do not think the paper establishes a sufficiently significant theoretical or conceptual advance beyond this existing framework.
2. Missing decisive comparison with the closest competitor. While weakness 1 would not prevent me from recommending acceptance, the empirical part of the paper is not sufficient. Highway-BP is itself parallelizable, yet the claimed efficiency advantage of GLBP over Highway-BP is supported primarily by VJP counts and ideal sequential depths, which do not establish superior wall-clock performance. The CIFAR-10 comparison (Table 2) actually shows higher accuracy for Highway-BP at comparable reported ideal sequential depths, albeit with more VJPs. No matched wall-clock comparison against Highway-BP is provided, and the reported large-model speedups are measured only against standard backpropagation. Furthermore, the progressive LLM schedule uses exact backpropagation during approximately the final third of training, without demonstrating an advantage over simpler approximate-to-exact schedules or appropriately matched fixed groupings.
In its submitted version, the paper fails to establish a compelling practical advantage over its closest predecessor. I would be open to reconsider my score if this critical weakness was addressed.
Questions for authors
-
Given that Highway-BP is already parallelizable, can the authors demonstrate that GLBP achieves a better downstream-quality versus wall-clock-time tradeoff under comparable hardware and training conditions?
-
What is the specific benefit of progressive grouping beyond simply increasing gradient fidelity during training? How does it compare against fixed groupings or a simpler transition from approximate to exact backpropagation at matched computational cost?
28 comments
[Response Theme 01] Our motivation and its basis: we are not algebraic regrouping, we are its exact opposite
Our position.
Our motivation is to break the serial dependency of backpropagation and deliver a real training speedup of the backward pass — not a different way of computing that polynomial.
The basis is the path decomposition of residual backpropagation of Veit et al. (2016): . It provides one judgment: the backward pass may not need the exact gradient. Highway-BP's own §2.1, describing Veit et al., states: "This is precisely the motivation behind our work." We emphasize: the low-order-polynomial idea originates from Veit et al. (2016); computing low-order terms is not a new idea. We adopt this basis, but we do not agree that every task can be trained at low order, and our purpose was never low-order training on a few small models and tasks.
From this, the two methods' motivations are not on the same plane:
- Highway-BP's motivation is to compute the low-order polynomial out: following the mathematical form exactly, iterating round by round from low order, slowly assembling the higher-order terms. Under this motivation its demonstrated scope is small-parameter models that do not need high-fidelity gradients, and its measured speedups hold for RNNs under weight sharing; restoring high fidelity is costly under its law, which is why its reported results do not reach today's large-model regime. We mean this as a statement about cost structure and demonstrated scope, not a judgment of the work's quality.
- GLBP's motivation is to use the fact that "low-order gradient precision suffices on some tasks" to dissolve the local serial dependency inside standard BP, thereby extending the method to today's large-model domain. Low-order precision is an admissible condition, not the goal; the object being modified is the backward operator itself. What we pursue is the current frontier, mainstream large-model domain, bringing a contribution uniquely ours to mainstream training — breaking backpropagation's serial chain, so that the training field gains one more possibility; for example, our method can accelerate the pipeline-scheduling family of algorithms. (On the low-order small-model regime itself, where Highway-BP is strong, we make no supremacy claim — and on the matched-quality wall-clock plane the two methods' positions are laid out side by side in Theme 06b, from our own measurements.)
Therefore we state it flatly: we are not a repetition of the algebraic-regrouping approach; we are its exact opposite. Simple, mechanical stacking of the polynomial is our negative example; mere algebraic regrouping is precisely what we oppose and avoid. "A different way of computing the polynomial" is neither our contribution nor, in our view, a good idea — reasons in Response Themes 02–03.
Evidence.
- Veit et al. (2016) is the source of both the decomposition and the low-order observation; Highway-BP §2.1 credits it verbatim as "precisely the motivation behind our work" — the shared foundation, which our paper also states (§1). What differs is what each method does on top of it, and the two directions are recorded in each method's own artifacts: Highway-BP's artifacts are quality-vs- curves on models up to 124M (its Table 2: largest model RoBERTa 124M, fine-tuned, post-LN, run with a layer re-split; deepest stack ); our artifacts are the operator schedule, measured at 8B parameters and (paper Tables 2–4; Tables 2–3 and 9 in the revision, §5.5).
[Response Theme 02] Verdict on the pure-algebraic route (I): small-scale confinement, and parallelism as a byproduct of the solve
Our position.
We explain the structural difference between the two routes — not to dismiss Highway-BP, but to show why they occupy different points of the path–cost space.
a. Its demonstrated scope is small-parameter models (up to 124M); its Transformer experiments require a per-architecture re-splitting of each layer — on pre-LN GPT-2 the natural decomposition "makes Highway-BP diverge" (its App.~C.3), remedied by an architecture-specific constant ( pre-LN, post-LN) that backpropagation is blind to but Highway-BP's convergence depends on; the authors themselves mark the choice as an open question. GLBP runs on the unmodified network.
b. Its parallelism does not shorten the serial chain; it is the computation that must be split out in order to assemble all polynomial terms of a given order — compute spent to assemble terms, not to shorten the backward chain. The two claims in this sentence are not our characterization but derivable from Highway-BP's own iteration (Thm. 2 / Eq. 7–8); the derivation is the next block, [Response Theme 02b], and the parallelism half is [Response Theme 03b]. In brief: each round of the iteration adds exactly one polynomial order (its own Thm. 2, Eq. 5), each round re-sweeps all blocks (Eq. 8), and the exact end pays VJPs at serial depth still for the same terms standard BP computes in one sweep at VJPs — completing the same terms, Highway-BP consumes the resources, with unchanged serial depth.
b′. A reverse control. Given the same resources as Highway-BP, we could just as well parallelize multiple batches and shorten an epoch's steps by ; that resource-inflated parallelism does not shorten the serial chain, which is our objective. Highway-BP at full precision is exactly that: added compute, unchanged serial depth, no speedup — one reason its paper never implemented that end.
Evidence. (Equation/section numbers are from Highway-BP's published paper.)
- a — weight-tied-only speedups, per-architecture layer re-split: its Table 3 (speedups) covers LSTM/GRU only — "all model trainings get a speedup between ×2 and ×3" on RNNs; its ResNet/GPT-2/RoBERTa experiments (Table 2, Fig. 2) report quality vs , never measured time. Its §6: "We leave its practical implementation for training large models in a distributed setting for future work. We however show that RNNs can be sped up considerably on a single GPU, as all cells share the same weights." Layer re-split: pre-LN GPT-2 — "the above choice … is not suited and makes Highway-BP diverge," remedied by (its Eq. 17); post-LN RoBERTa — (its Eq. 23, App. C.3); the re-split is invisible to backpropagation yet decides Highway-BP's convergence, with no prescription for architectures beyond the two tested.
- b — the cost structure, from its own equations: one training step runs rounds of its Thm. 2 iteration (Eq. 7, indexed from ), each round a parallel VJP over all blocks (Eq. 8: ) plus a prefix scan — so each unit of fidelity is bought with a full -wide sweep. Low-order training on deep residual nets needs only small (its §5.1.2), but that does not shorten the backward chain: every one of the rounds re-sweeps all blocks, and the serial depth at the exact-gradient end is still (, §4.2) — no shorter than standard BP, while the work is . Its cost model Eq. 13 prices the rounds jointly at (i.e.\ of a backward sweep per round) — billing each -wide round as if its -wide parallelism were free, which is precisely the resource-unawareness we point to. Our paired single-GPU audit (work, not speedup, for both methods — on one device neither method's parallelism is realized): on ResNet-32 () Highway-BP is 16.8/21.7/25.8 ms at vs 12.2 ms standard — each iteration adds ≈ one leaf-wide sweep in real time, the law, not the -per-round billing of Eq. 13 which holds only if the -wide parallelism is free; GLBP executes exactly block-VJPs for every partition (counter-verified in the same runs), its exact end timing at standard's cost (12.2 ms, ). The matched table (both methods, one model, to the exact end) is in Theme 05.
- b′ — the throughput inequality: with devices, data-parallel standard BP has throughput ; Highway-BP's throughput is for every and — an inequality, not an empirical claim: any device spent parallelizing its -wide backward buys more throughput as a data-parallel replica of exact standard BP. In the high-order, high-fidelity regime it has, at the theoretical level, only disadvantages.
[Response Theme 00] Opening
We thank the reviewer for a careful and technical reading. Both weaknesses rest on the same reading of the GLBP–HBP relationship: reading this paper as "a different way of computing the polynomial, on top of Highway-BP." That reading matches neither our motivation nor where the two methods actually act. We therefore first correct the motivation and its basis (Response Theme 01), then our verdict on the pure-algebraic route (Themes 02–03), then W1 (Themes 04–05), W2 (Theme 06), and Q2 (Theme 07), then the method's orthogonality and future space near the end (Theme 08), closing with a two-part summary table of what each method's experiments ultimately delivered (Themes 09–10).
One framing note that governs everything below: our position is not that Highway-BP is a poor method — it is that the two methods act on different planes and are not in competition. The cost-structure analysis in Themes 02–03 is offered in that spirit: to explain why the two operators occupy different points of the path–cost space, not to render a quality judgment on Highway-BP's work.
Index of themes — W1 ("essentially first-order / scheduling idea not new / no significant advance") → Response Themes 01–05. W2 & Q1 ("only VJP counts / Table 2 / no matched wall-clock / speedups only vs standard BP") → Theme 06. Q2 (progressive vs simpler schedules) → Theme 07. Orthogonality & outlook → Theme 08. Closing summary tables (essence of each method, what its experiments delivered, and why) → Themes 09–10. Two derivation blocks support Themes 02–03: Theme 02b proves, by induction on Highway-BP's own Theorem 2, that one round of its iteration = one polynomial order (the brute-force claim), and Theme 03b traces each of its two parallel components to the assembly it serves and to no serial edge it removes (the liability claim) — both derivations use only Highway-BP's own equations.