acceptodds
Under review as a conference paper at ICLR 2027

Group-Linearized Backpropagation: Scheduling Residual Paths for Deep Network Training

Abstract

Residual networks provide exponentially many implicit paths for gradient propagation, yet standard backpropagation still follows a sequential chain whose critical path grows with depth. Existing methods either modify the forward computation, changing the network being trained, or retain the original backward operator and approximate its solution through iterative truncation. We introduce Path-Scheduled Residual Training, built on Group-Linearized Backpropagation (GLBP). Specifically, GLBP partitions the residual chain into contiguous groups, where blocks within each group receive a shared entry gradient and are aggregated in parallel—replacing the standard sequential product within each group with a sum while preserving the product structure across groups. This rewriting requires exactly one block gradient per residual position per step, regardless of grouping granularity. Coarser groupings trade within-group path interactions for reduced serial depth, and which paths are retained is determined by where group boundaries fall rather than by a global order cutoff. The grouping granularity is then scheduled during training: a coarse-to-fine progression starts from short serial depth and reinstates higher-order path content over training, reaching the exact operator at its final stage. Experiments on ResNets and fine-tuning of LLaMA-3.1-8B show that scheduled GLBP achieves near-standard training quality and is competitive with standard backpropagation on downstream benchmarks.

Primary area:
optimization
Keywords:
backpropagation, efficient training, residual network, gradient approximation, optimization, training scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_VmtvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Group-Linearized Backpropagation (GLBP), an approximate backward propagation method for residual networks that builds upon Highway BP (Fagnou et al., 2025).

The method partitions residual blocks into groups and replaces within-group products of residual Jacobians with additive approximations, allowing parallel computation of local gradients while preserving the forward computation.

The authors propose to progressively refine the residual block grouping during training. Experiments on ResNets and LLaMA-3.1-8B investigate gradient fidelity and downstream performance, while additional Qwen3 experiments report training-step speedups over standard backpropagation.

Critical strengths
  • The grouping-based approximation provides a simple and interpretable way to control residual-path interactions. The reported Qwen3 experiments also demonstrate end-to-end speedups over standard backpropagation in the tested hardware configurations.

  • The paper is easy to follow and quite enjoyable to read.

Critical weaknesses

1. Limited novelty and scientific significance GLBP essentially applies a first-order approximation to grouped residual Jacobian products, with theoretical results that largely follow from direct algebraic observations. Highway-BP (Fagnou et al., ICLR 2025) already introduced approximate parallel backpropagation through residual paths. Moreover, its Section 5.3 analyzes how gradient approximation quality evolves during training, and its conclusion explicitly proposes increasing the number of iterations during training, potentially ending with exact backpropagation. GLBP implements this principle through group refinement rather than additional iterations, but the underlying scheduling idea is not new.

Overall, I do not think the paper establishes a sufficiently significant theoretical or conceptual advance beyond this existing framework.

2. Missing decisive comparison with the closest competitor. While weakness 1 would not prevent me from recommending acceptance, the empirical part of the paper is not sufficient. Highway-BP is itself parallelizable, yet the claimed efficiency advantage of GLBP over Highway-BP is supported primarily by VJP counts and ideal sequential depths, which do not establish superior wall-clock performance. The CIFAR-10 comparison (Table 2) actually shows higher accuracy for Highway-BP at comparable reported ideal sequential depths, albeit with more VJPs. No matched wall-clock comparison against Highway-BP is provided, and the reported large-model speedups are measured only against standard backpropagation. Furthermore, the progressive LLM schedule uses exact backpropagation during approximately the final third of training, without demonstrating an advantage over simpler approximate-to-exact schedules or appropriately matched fixed groupings.

In its submitted version, the paper fails to establish a compelling practical advantage over its closest predecessor. I would be open to reconsider my score if this critical weakness was addressed.

Questions for authors
  1. Given that Highway-BP is already parallelizable, can the authors demonstrate that GLBP achieves a better downstream-quality versus wall-clock-time tradeoff under comparable hardware and training conditions?

  2. What is the specific benefit of progressive grouping beyond simply increasing gradient fidelity during training? How does it compare against fixed groupings or a simpler transition from approximate to exact backpropagation at matched computational cost?

28 comments
AuthorsOpenReview ↗

[Response Theme 00] Opening

We thank the reviewer for a careful and technical reading. Both weaknesses rest on the same reading of the GLBP–HBP relationship: reading this paper as "a different way of computing the polynomial, on top of Highway-BP." That reading matches neither our motivation nor where the two methods actually act. We therefore first correct the motivation and its basis (Response Theme 01), then our verdict on the pure-algebraic route (Themes 02–03), then W1 (Themes 04–05), W2 (Theme 06), and Q2 (Theme 07), then the method's orthogonality and future space near the end (Theme 08), closing with a two-part summary table of what each method's experiments ultimately delivered (Themes 09–10).

One framing note that governs everything below: our position is not that Highway-BP is a poor method — it is that the two methods act on different planes and are not in competition. The cost-structure analysis in Themes 02–03 is offered in that spirit: to explain why the two operators occupy different points of the path–cost space, not to render a quality judgment on Highway-BP's work.

Index of themes — W1 ("essentially first-order / scheduling idea not new / no significant advance") → Response Themes 01–05. W2 & Q1 ("only VJP counts / Table 2 / no matched wall-clock / speedups only vs standard BP") → Theme 06. Q2 (progressive vs simpler schedules) → Theme 07. Orthogonality & outlook → Theme 08. Closing summary tables (essence of each method, what its experiments delivered, and why) → Themes 09–10. Two derivation blocks support Themes 02–03: Theme 02b proves, by induction on Highway-BP's own Theorem 2, that one round of its iteration = one polynomial order (the brute-force claim), and Theme 03b traces each of its two parallel components to the assembly it serves and to no serial edge it removes (the liability claim) — both derivations use only Highway-BP's own equations.

AuthorsOpenReview ↗

[Response Theme 01] Our motivation and its basis: we are not algebraic regrouping, we are its exact opposite

Our position.

Our motivation is to break the serial dependency of backpropagation and deliver a real training speedup of the backward pass — not a different way of computing that polynomial.

The basis is the path decomposition of residual backpropagation of Veit et al. (2016): Hexact=g∏i(I+Ai)=∑SgPSH_{\mathrm{exact}} = g\prod_i(I+A_i) = \sum_{S} gP_S. It provides one judgment: the backward pass may not need the exact gradient. Highway-BP's own §2.1, describing Veit et al., states: "This is precisely the motivation behind our work." We emphasize: the low-order-polynomial idea originates from Veit et al. (2016); computing low-order terms is not a new idea. We adopt this basis, but we do not agree that every task can be trained at low order, and our purpose was never low-order training on a few small models and tasks.

From this, the two methods' motivations are not on the same plane:

  • Highway-BP's motivation is to compute the low-order polynomial out: following the mathematical form exactly, iterating round by round from low order, slowly assembling the higher-order terms. Under this motivation its demonstrated scope is small-parameter models that do not need high-fidelity gradients, and its measured speedups hold for RNNs under weight sharing; restoring high fidelity is costly under its (k+1)n(k{+}1)n law, which is why its reported results do not reach today's large-model regime. We mean this as a statement about cost structure and demonstrated scope, not a judgment of the work's quality.
  • GLBP's motivation is to use the fact that "low-order gradient precision suffices on some tasks" to dissolve the local serial dependency inside standard BP, thereby extending the method to today's large-model domain. Low-order precision is an admissible condition, not the goal; the object being modified is the backward operator itself. What we pursue is the current frontier, mainstream large-model domain, bringing a contribution uniquely ours to mainstream training — breaking backpropagation's serial chain, so that the training field gains one more possibility; for example, our method can accelerate the pipeline-scheduling family of algorithms. (On the low-order small-model regime itself, where Highway-BP is strong, we make no supremacy claim — and on the matched-quality wall-clock plane the two methods' positions are laid out side by side in Theme 06b, from our own measurements.)

Therefore we state it flatly: we are not a repetition of the algebraic-regrouping approach; we are its exact opposite. Simple, mechanical stacking of the polynomial is our negative example; mere algebraic regrouping is precisely what we oppose and avoid. "A different way of computing the polynomial" is neither our contribution nor, in our view, a good idea — reasons in Response Themes 02–03.

Evidence.

  • Veit et al. (2016) is the source of both the decomposition and the low-order observation; Highway-BP §2.1 credits it verbatim as "precisely the motivation behind our work" — the shared foundation, which our paper also states (§1). What differs is what each method does on top of it, and the two directions are recorded in each method's own artifacts: Highway-BP's artifacts are quality-vs-kk curves on models up to 124M (its Table 2: largest model RoBERTa 124M, fine-tuned, post-LN, run with a γ=0.8\gamma{=}0.8 layer re-split; deepest stack L=54L{=}54); our artifacts are the operator schedule, measured at 8B parameters and n=600n{=}600 (paper Tables 2–4; Tables 2–3 and 9 in the revision, §5.5).
AuthorsOpenReview ↗

[Response Theme 02] Verdict on the pure-algebraic route (I): small-scale confinement, and parallelism as a byproduct of the solve

Our position.

We explain the structural difference between the two routes — not to dismiss Highway-BP, but to show why they occupy different points of the path–cost space.

a. Its demonstrated scope is small-parameter models (up to 124M); its Transformer experiments require a per-architecture re-splitting of each layer — on pre-LN GPT-2 the natural decomposition "makes Highway-BP diverge" (its App.~C.3), remedied by an architecture-specific constant γ\gamma (0.20.2 pre-LN, 0.80.8 post-LN) that backpropagation is blind to but Highway-BP's convergence depends on; the authors themselves mark the choice as an open question. GLBP runs on the unmodified network.

b. Its parallelism does not shorten the serial chain; it is the computation that must be split out in order to assemble all polynomial terms of a given order — compute spent to assemble terms, not to shorten the backward chain. The two claims in this sentence are not our characterization but derivable from Highway-BP's own iteration (Thm. 2 / Eq. 7–8); the derivation is the next block, [Response Theme 02b], and the parallelism half is [Response Theme 03b]. In brief: each round of the iteration adds exactly one polynomial order (its own Thm. 2, Eq. 5), each round re-sweeps all LL blocks (Eq. 8), and the exact end k=Lk{=}L pays (L+1)L(L{+}1)L VJPs at serial depth still LL for the same 2L2^L terms standard BP computes in one sweep at LL VJPs — completing the same terms, Highway-BP consumes n×n\times the resources, with unchanged serial depth.

b′. A reverse control. Given the same n×n\times resources as Highway-BP, we could just as well parallelize multiple batches and shorten an epoch's steps by n×n\times; that resource-inflated parallelism does not shorten the serial chain, which is our objective. Highway-BP at full precision is exactly that: added compute, unchanged serial depth, no speedup — one reason its paper never implemented that end.

Evidence. (Equation/section numbers are from Highway-BP's published paper.)

  • a — weight-tied-only speedups, per-architecture layer re-split: its Table 3 (speedups) covers LSTM/GRU only — "all model trainings get a speedup between ×2 and ×3" on RNNs; its ResNet/GPT-2/RoBERTa experiments (Table 2, Fig. 2) report quality vs kk, never measured time. Its §6: "We leave its practical implementation for training large models in a distributed setting for future work. We however show that RNNs can be sped up considerably on a single GPU, as all cells share the same weights." Layer re-split: pre-LN GPT-2 — "the above choice … is not suited and makes Highway-BP diverge," remedied by γ=0.2\gamma{=}0.2 (its Eq. 17); post-LN RoBERTa — γ=0.8\gamma{=}0.8 (its Eq. 23, App. C.3); the re-split is invisible to backpropagation yet decides Highway-BP's convergence, with no prescription for architectures beyond the two tested.
  • b — the cost structure, from its own equations: one training step runs k+1k{+}1 rounds of its Thm. 2 iteration (Eq. 7, kk indexed from 00), each round a parallel VJP over all LL blocks (Eq. 8: vik+1=wi+1kJi+1 ∀iv_i^{k+1} = w_{i+1}^k J_{i+1}\ \forall i) plus a prefix scan — so each unit of fidelity is bought with a full LL-wide sweep. Low-order training on deep residual nets needs only small kk (its §5.1.2), but that does not shorten the backward chain: every one of the k+1k{+}1 rounds re-sweeps all LL blocks, and the serial depth at the exact-gradient end is still LL (k=Lk{=}L, §4.2) — no shorter than standard BP, while the work is (k+1)L(k{+}1)L. Its cost model Eq. 13 prices the kk rounds jointly at (k/L) Tbackward(k/L)\,T_{\mathrm{backward}} (i.e.\ 1/L1/L of a backward sweep per round) — billing each LL-wide round as if its LL-wide parallelism were free, which is precisely the resource-unawareness we point to. Our paired single-GPU audit (work, not speedup, for both methods — on one device neither method's parallelism is realized): on ResNet-32 (n=15n{=}15) Highway-BP is 16.8/21.7/25.8 ms at k=1/2/3k{=}1/2/3 vs 12.2 ms standard — each iteration adds ≈ one leaf-wide sweep in real time, the (k+1)n(k{+}1)n law, not the 1/L1/L-per-round billing of Eq. 13 which holds only if the LL-wide parallelism is free; GLBP executes exactly nn block-VJPs for every partition (counter-verified in the same runs), its exact end m=nm{=}n timing at standard's cost (12.2 ms, 1.00×1.00\times). The matched n=54n{=}54 table (both methods, one model, to the exact end) is in Theme 05.
  • b′ — the throughput inequality: with pp devices, data-parallel standard BP has throughput p/np/n; Highway-BP's throughput is ≤p/((k+1)n)<p/n\le p/((k{+}1)n) < p/n for every pp and k≥1k \ge 1 — an inequality, not an empirical claim: any device spent parallelizing its nn-wide backward buys more throughput as a data-parallel replica of exact standard BP. In the high-order, high-fidelity regime it has, at the theoretical level, only disadvantages.
Reviewer_iqo4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Group-Linearized Backpropagation (GLBP), which rewrites the backward operator of residual networks. The residual chain is partitioned into contiguous groups, and within each group the product of backward factors is replaced by a sum, while products across groups are kept. This retains exactly the paths that select at most one residual branch per group, keeps the number of block VJPs at n for any partition, and shortens the dependency span of the backward pass to roughly the number of groups. The partition can be scheduled from coarse to fine during training, eventually recovering exact backpropagation. Experiments on CIFAR-10 ResNets with up to 600 residual blocks and on LLaMA-3.1-8B fine-tuning show that fine or progressive groupings approach standard backpropagation, and a GPU-boundary case study reports up to 1.63× step-time reduction.

Critical strengths
  1. A clean and analyzable rewriting of the backward operator. Instead of modifying the forward network or approximating the exact backward solve iteratively, the paper rewrites the backward operator itself and characterizes exactly which paths are retained. Combined with the constant VJP cost and the exact anchor at the singleton partition, this gives a principled way to trade gradient fidelity for backward span, which I consider the main contribution of the paper.
Critical weaknesses
  1. Limited practical significance. The paper motivates GLBP by the cost of the sequential backward chain, but it does not show that this rewriting brings a meaningful benefit in practice. First, almost all reported costs are VJP counts and ideal dependency spans rather than wall-clock time and memory. On a single GPU, each block VJP is already a large parallel kernel, and computing member VJPs together may increase memory pressure. Second, the quality–cost tradeoff is unfavorable in practice: fixed coarse groupings lose substantial accuracy, for example 4.6 points on ResNet-1202 and 9.5 points on GSM8K-strict with m=4, while near-standard quality is reached only by schedules that end at or close to the exact operator, for example reducing the average span merely from 15 to 10.5 on CIFAR. Third, in realistic large-scale training, the backward critical path is already mitigated by pipeline parallelism and other system-level techniques, yet the paper provides only one GPU-boundary case study and does not compare against such techniques or other parallel backpropagation methods. As a result, it remains unclear in which practical scenario a practitioner would choose GLBP over standard backpropagation.

  2. The empirical evidence is not statistically reliable. All runs are single-seed without error bars. The two standard backpropagation reference runs on the same CIFAR setting differ by 0.57 points, which is larger than several gaps used to support the claims, such as GLBP being "within 0.2 points" of standard backpropagation. The LLM results are also mixed, with progressive GLBP better than standard backpropagation on some metrics and worse on others, and training loss and downstream metrics do not follow the same order. Since GLBP produces a biased gradient whose cosine with the exact gradient can drop to 0.39, and no convergence analysis is provided, more reliable evidence is needed to support the claim that it trains as well as standard backpropagation.

Questions for authors
  1. What are the wall-clock step time and peak memory of GLBP compared with standard backpropagation on a single GPU, for both the CIFAR and LLaMA settings, and how do they change with the group count m? (Clarification; this would materially affect my assessment.)
  2. For the progressive schedules, what fraction of training steps is spent at each m, and what end-to-end speedup is achieved compared with standard backpropagation trained for the same wall-clock time? (Clarification.)
  3. Can you report the mean and standard deviation over at least three seeds for the main CIFAR and LLaMA results? (Clarification.)
  4. Is there any convergence guarantee, or at least an empirical criterion, for how fine the partition must be for stable training as depth increases? (Clarification; a formal analysis could also be future work.)
6 comments
AuthorsOpenReview ↗

[Response Theme 00] Index: where each of your concerns is answered

We thank the reviewer for a precise reading — the four clarification questions are exactly the right ones, and we answer each with new measurements (completed during the discussion period; all numbers below are measured). Where each is answered:

  • W1 (practical significance: costs are VJP counts and ideal spans, not wall-clock and memory; single-GPU member VJPs may increase memory pressure) → answered in [Response Theme 01: wall-clock and peak memory vs m]: the single-GPU runs are labeled cost audits (for both methods), and peak memory is measured identical to standard BP in both settings — CIFAR ResNet-1202 (11450.19 vs 11450.22 MiB) and LLaMA-8B (per-GPU peaks byte-identical across m=8/4/2m{=}8/4/2). The multi-GPU three-tier measurement (3019→2001→1635 ms, same batch) covers the wall-clock half.
  • Q1 (wall-clock step time and peak memory of GLBP vs standard BP, single GPU, CIFAR and LLaMA; how they change with m) → [Response Theme 01] as above.
  • Q2 (what fraction of steps at each m; end-to-end speedup vs standard BP at matched wall-clock) → answered in [Response Theme 02: progressive schedule composition and end-to-end speedup]: stage fractions (final 35% at the finest grouping), the per-step speed points (moderate m=4m{=}4,M=1M{=}1 ×1.43 end-to-end; coarse m=2m{=}2 ×1.22 over GPipe), the progressive schedule's ×1.27 full-run wall-clock (a cost measurement on the test bed) reported separately from its quality preservation on the main fine-tune (a distinct measurement), and the fixed-1-hour-budget fine-tuning runs (1.14× more steps) with the fidelity side reported honestly.
  • Q3 (mean and std over at least three seeds) → answered in [Response Theme 03: seed protocol, with the full mean±std table]: the main arms of all three tables are re-run under four seeds ({42, 64, 744, 742}); the revision adds a per-arm mean±std table (its App. E). Headlines: CIFAR per-arm std 0.08–0.43 pts (n=15) and 0.29–0.74 pts (deep tracks); on the LLM benchmarks the cross-seed std is everywhere below the evaluation's own sampling error (GSM8K-strict ≤0.5 vs stderr ≈1.3; HumanEval ≤2.3 vs ≈3.9; IFEval ≤1.2 vs ≈2.0), and the GSM8K ordering progressive > standard > fixed holds at all four seeds with margins of 2.2–3.8 and 8.7–9.5 points.
  • Q4 (any convergence guarantee, or an empirical criterion for how fine the partition must be as depth increases) → answered in [Response Theme 04: an empirical criterion, measured at two depths] — the cosine-threshold rule, its trigger counts growing with depth, and adaptive beating fixed by +6.5 pts at n=54n{=}54 — plus the descent/convergence propositions (full statements in [Response Theme 02, our reply to Reviewer 8s9C]).
  • W2 (the quality–cost tradeoff is unfavorable in practice) → answered in [Response Theme 05: the quality–cost tradeoff, read at the right granularity]: the table's own fixed arms recover monotonically with granularity at both depths, and the matched-compute schedule control prices each stage of the recovery.
AuthorsOpenReview ↗

[Response Theme 01] wall-clock and peak memory vs m — single GPU (CIFAR and LLaMA) and multi-GPU (W1/Q1)

We answer with measurements, completed during the discussion period.

Single GPU — reported as cost audit, by design. On one device the algebraic parallel structure is not realized for either GLBP or HBP (our engine runs GLBP's group members as a sequential loop; a CUDA-stream audit showed member kernels already saturate the SMs, ×1.00 from dual-stream overlap). Single-GPU numbers therefore audit the work law. The n=600n{=}600 audit is in the revision (App.~"Notes on cost accounting"): on ResNet-1202 (pre-act track, dedicated 2,000-step timing protocol), standard BP costs 352 ms and GLBP pps$=6$ costs 373 ms — a 6% gap that is implementation overhead of the timing engine, not a cost of the method — while the backward serial span is cut from 600 to 20. The ResNet-32 HBP/GLBP timings below were measured during the discussion period under a device-exclusive protocol (5 warmup + 20 measured steps, seed 42) and are reported here (not inserted into the manuscript): on ResNet-32 (n=15n{=}15) standard BP is 12.2 ms/step and HBP is slower at every kk — 16.8/21.7/25.8 ms at k=1/2/3k{=}1/2/3 (1.37×/1.77×/2.11×), confirming the (k+1)n(k{+}1)n law is paid in real time; GLBP executes exactly nn block-VJPs per step for every mm — counter-verified in the same timing runs — with the exact partition (m=nm{=}n) timing at 12.2 ms, identical to standard (1.00×). These single-GPU numbers are cost audits, not speedup claims; wall-clock claims are scoped to the multi-device layouts where the structural parallelism is realized (below). We will add the ResNet-32 timing row to the appendix audit if useful.

Multi-GPU (8×H20, LLaMA-3.1-8B, total batch 16 held constant), forward/backward separated: (1) standard BP, no scheduling: 3019 ms/step; (2) standard pipeline scheduling (GPipe, exact chain, best M=8M{=}8): 2001 ms; (3) GLBP+pipeline m=2m{=}2 (best M=1M{=}1): 1635 ms. Three readings, each against its own baseline: best-vs-best at matched batch, ×1.22 end-to-end over standard GPipe's best configuration (backward ×1.50) — the within-stage span cut substituting for heavy micro-batching; against unscheduled standard BP, ×1.85 (backward ×3.29) as the combined figure for the two mechanisms; and layering the same grouping onto GPipe's own best micro-batching (M=8M{=}8, m=2m{=}2: 1760 ms) gives ×1.14 — the marginal gain over a scheduler already at its optimum. The full M×mM{\times}m grid matches the closed form makespan(m)=(M+P−1)F+(M+m−1)B\mathrm{makespan}(m)=(M{+}P{-}1)F+(M{+}m{-}1)B at 70–81% attainment. Earlier device-boundary instances: Qwen3-8B ×1.34 (4 GPU), Qwen3-32B ×1.63 (8 GPU), both vs their unscheduled standard-BP baselines. The HBP arms at n=54n{=}54 (device-exclusive timing, same protocol as the single-GPU audit above, on the HBP-aligned ResNet-110) are all slower than standard BP and grow with kk — adding iterations costs more, because standard BP is already exact. On the same model and run, GLBP stays at standard BP's cost for every mm (block-VJP =54=54 throughout; 0.940.94–1.00×1.00\times standard, within single-GPU timing noise), while HBP climbs 1.62×1.62\times (k=2k{=}2, accuracy 0.9291 vs 0.9289) → 2.32×2.32\times (k=4k{=}4, 0.9287, no accuracy gain) → 19.56×19.56\times at the exact end k=n=54k{=}n{=}54, counters =(k+1)n=(k{+}1)n throughout. The full matched table (both methods, one model, one protocol) is in our reply to Reviewer Vmtv, Theme 05. At n=600n{=}600 no HBP arm is reported: our implementation does not realize its App.~G transition handling (uniform-Jacobian Ki=IK_i{=}I violated by dimension-changing stages), and independently, the (k+1)n(k{+}1)n law prices k=2k{=}2 at 1,8001{,}800 VJPs/step — impractical at this depth; GLBP runs unmodified.

Peak memory — GLBP does not increase it, measured in both settings. Member VJPs share one entrance cotangent and touch disjoint parameter subsets, so they execute with the same activation footprint as standard BP (forward unchanged); the only addition is a group-reduce buffer of O(hidden)O(\text{hidden}) per group exit. Measured: (i) CIFAR ResNet-1202 (n=600n{=}600): standard 11450.19 MiB vs GLBP pps=6 11450.22 MiB (difference 0.03 MiB, measurement noise) while the backward serial span drops 600→20; (ii) LLaMA-3.1-8B (8×H20, M=1M{=}1 held fixed): per-GPU peaks identical across m=8/4/2m{=}8/4/2 (max peak 23750.4 MiB in all three) while step time falls 3037→2103→1637 ms. This answers the "may increase memory pressure" concern directly: the pressure never materializes because the members' activations are the same tensors standard BP would hold.

AuthorsOpenReview ↗

[Response Theme 02] progressive schedule composition and end-to-end speedup at matched wall-clock (Q2)

Fraction of steps at each mm (App. E of the revision): stage lengths proportional to (4,5,6,8)(4,5,6,8); for the GSM8K arm (4,000 steps): 696/870/1043/1391 at m=4/5/6/8m{=}4/5/6/8 — the final 35% runs at the finest grouping (m=B=8m{=}B{=}8 bundles of b=4b{=}4, one bundle per group).

End-to-end at matched batch — two operating points, kept separate from the quality claim. The M×mM\times m grid gives the fidelity–span tradeoff as wall-clock; within each MM block we pick the fastest MM per operator (the forward is mm-invariant, so the backward span sets the speedup, which is largest at small MM). Two points are worth naming. (a) Moderate, mid-fidelity: m=4m{=}4 at its fastest M=1M{=}1 gives backward ×1.80 and end-to-end ×1.43 over the exact m=8m{=}8 baseline at the same micro-batch count. (b) Coarse end: m=2m{=}2 at M=1M{=}1 gives backward ×3.02 / end-to-end ×1.81 over that baseline, and the deployed total-batch-16 arm is ×1.22 over standard GPipe (×1.85 / backward ×3.29 only against the unscheduled layer-sharded reference). Crucially, these are speed operating points, not quality claims: a fixed coarse grouping (m=4m{=}4 or m=2m{=}2) does not preserve downstream quality at 8B — fixed m=4m{=}4 is 0.552 vs 0.647 standard on GSM8K-strict (−9.5 pts), and fixed m=2m{=}2 (cos ≈ 0.52) collapses to IFEval 0.159 vs 0.311. The progressive schedule carries the two facts we do not fuse: it ends exact and preserves downstream quality (a quality measurement on the main 4-GPU/batch-128 fine-tune), and — as a separate cost measurement, a per-step timing on the 8-GPU test bed — its full run integrates to ×1.27 less wall-clock (6.14 → 4.84 h, 21% earlier). These come from different hardware/batch settings, so we report them as two facts about the same schedule, not as one run that is at once faster and better; the larger per-step factors above are what the span cut buys when quality is allowed to degrade. No single arm is simultaneously ×1.85 and quality-preserving, and we do not blur the two. We do not claim a lower training loss at equal wall-clock (GLBP is a lossy-gradient method; the wall-clock–loss curve in the Z2pf reply, Theme 01, shows the progressive arm's training loss slightly above standard's at matched elapsed time); the measured claim is downstream quality at matched steps (progressive ≥ standard on HumanEval/GSM8K-strict, Table 4 submission / Table 3 revision) and the full run finishing 21% earlier. The equal-budget fine-tuning runs make the tradeoff concrete: at a fixed 1-hour budget from base (Alpaca, 8×H20, total batch 16), the m=2m{=}2 arm processes 1.14× more steps than the exact arm (1600 vs 1400 steps; 2237 vs 2556 ms/step; the equal-budget runs time full training loops including data loading, and their exact arm is the pipeline-scheduled configuration, so 1.14× is the tier table's 1.22× net of loop overhead — consistent, not contradictory). We report the fidelity side at the same time, because it is the binding term at the coarse end: a separate fixed-m=2m{=}2 fine-tune under the full Alpaca recipe (cos ≈ 0.52; rebuttal-period run, not in the paper's tables) scores IFEval 0.159 prompt-loose vs 0.311 standard — a property of the m=2m{=}2 operator, not of the 1-hour budget run above. We put this on record as an observation, not a caveat: moving from the small models to the 8B scale, it is exactly the demonstration that not every task can be trained with low-fidelity gradients — our claim was never that (how much fidelity a task needs is an open question in this entire line of work; Veit et al. 2016 establishes low-order paths often suffice as an analysis, HBP's kk is found by per-task sweeps). The operator makes fidelity a zero-cost control variable (nn VJPs at every mm, exact at the finest), which is exactly what lets a schedule lift the coarse end back to standard-level quality (progressive 0.676 vs standard 0.647 GSM8K-strict; see [To All Reviewers 04] and our reply to Reviewer Vmtv, Theme 07). The speed is real at the coarse end (1.14× steps); converting it into quality requires the scheduled operator, which ends exact at zero overhead. Two further bounds: (i) compute is identical by construction (nn VJPs/step in every arm); (ii) the sweep shows the optimal micro-batching changes with the operator — standard pipeline needs M=8M{=}8 to fill the pipe, GLBP m=2m{=}2 is fastest at M=1M{=}1 (no bubble left to hide, larger per-GPU kernels).

Reviewer_8s9COpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Group-Linearized Backpropagation (GLBP), an approximate backpropagation method designed to reduce sequential dependencies and expose parallelism during the training of deep residual networks. Conventional backpropagation propagates gradients sequentially through network layers, resulting in a critical path that grows linearly with network depth. The authors observe that gradient propagation in residual networks can be represented as an expansion over exponentially many implicit residual paths, suggesting that the backward computation can be reorganized by selectively retaining path interactions.GLBP partitions consecutive residual blocks into groups and replaces the sequential product of Jacobian factors within each group with an additive approximation, while preserving multiplicative interactions across groups. This allows block-level gradient computations within a group to execute independently, while maintaining exactly one vector-Jacobian product (VJP) per residual block per training step. Furthermore, the paper introduces Path-Scheduled Residual Training, which progressively refines the grouping structure during training. Coarse groupings reduce sequential dependencies early in training, while finer groupings gradually restore higher-order gradient interactions and can eventually recover exact standard backpropagation. Experiments on ResNet image classification and LLaMA-3.1-8B fine-tuning demonstrate competitive performance relative to standard backpropagation. A constrained multi-GPU implementation additionally demonstrates practical training speedups.

Critical strengths

The paper introduces a novel structured approximation of the backward operator by replacing sequential products of residual Jacobian factors within each group with additive approximations, thereby reducing sequential dependencies in gradient propagation. Unlike methods such as Stochastic Depth and LayerDrop, GLBP preserves the original forward computation and modifies only the backward operator. Furthermore, in contrast to Highway-BP, which approximates gradient propagation through iterative truncation, GLBP directly controls the retained gradient paths through grouping structures. This formulation provides a distinct and mathematically interpretable perspective on approximate backpropagation.

Critical weaknesses

Although the paper precisely characterizes the omitted residual-path interactions, these results primarily describe the algebraic approximation error of the backward operator and do not directly establish convergence guarantees for optimization with approximate gradients. In particular, sharing entrance cotangents within a group may cause individual parameter gradients to omit first-order residual contributions. Therefore, even when the lowest-order omitted terms of the overall backward operator are second-order, individual parameter gradients may still exhibit first-order approximation errors. The paper lacks a sufficiently comprehensive theoretical analysis of gradient bias, descent-direction properties, and optimization convergence under the proposed approximation.

Questions for authors

Theorem 1 characterizes the omitted residual-path interactions, while Theorem 2 shows that individual parameter gradients may incur first-order approximation errors due to shared within-group entrance cotangents. Could the authors provide a more detailed analysis of how these errors affect optimization dynamics? In particular, under what conditions does the approximate gradient remain a descent direction, and can any convergence guarantees be established?

4 comments
AuthorsOpenReview ↗

[Response Theme 00] Index: where your concern is answered

We thank the reviewer for a generous assessment and, especially, for the sharpest technical observation in all four reviews. Where your concern is answered:

  • W1 (algebraic results describe the approximation error of the backward operator but do not establish convergence guarantees for optimization with approximate gradients) → answered in [Response Theme 02: the added analysis — descent, convergence, and scheduled debiasing]: three propositions now in the revision's App. A.6 — the descent-direction condition, the biased-SGD neighborhood bound (with the bias supplied by the omitted-monomial counts rather than assumed), and the type-B debiasing corollary for refinement-chain schedules — each paired with its empirical check.
  • Q1 (under what conditions does the approximate gradient remain a descent direction; can convergence guarantees be established?) → the same block: descent ⟺ cos⁡(g~,g)>0\cos(\tilde g, g) > 0, sufficient condition ∥δΠ∥<∥∇ℓ∥\|\delta_\Pi\| < \|\nabla\ell\| with the mm-controlled count bound, and the convergence guarantee in its standard biased-SGD form. The reviewer's specific observation — that shared within-group entrance cotangents give individual parameter gradients first-order errors even when the operator-level error is second-order (type-B) — is correct, and we open with it in [Response Theme 01: the reviewer is right about type-B], where the mechanism is stated precisely and acknowledged as already conceded in the paper's App. A.2.
  • The empirical side of both (cosine positive at every measured checkpoint; the coarse-end fidelity observation at 8B; the matched schedule control measuring what debiasing buys) is in [Response Theme 02] and [Response Theme 03: scope, and why this strengthens the paper].
AuthorsOpenReview ↗

[Response Theme 01] the reviewer is right about type-B — and the paper already concedes it

We thank the reviewer for a generous assessment and for a precise technical observation — the reviewer is correct, and the paper already concedes it in App. A.2: although the lowest-order error of the full operator is second-order (intra-group pairs, Prop. 2), the per-parameter entrance error contains first-order terms. By Theorem 2, the entrance cotangent of block vv omits: type A — paths with two branches inside one earlier group (second-order and up); type B — paths using any branch of vv's own group on the output side, which includes single-branch (first-order) terms. So ∥Δv∥=O(α)\|\Delta_v\| = O(\alpha) is possible even when the operator-level error is O(α2)O(\alpha^2), and the parameter-gradient error is Γvexact−Γvsched=Bv(Δv)\Gamma_v^{\text{exact}} - \Gamma_v^{\text{sched}} = \mathcal{B}_v(\Delta_v) (Eq. 9). The reviewer's inference is exactly right: per-parameter gradients can be first-order biased while the operator is second-order accurate — this is the price of sharing one entrance cotangent within a group, and we take it seriously. One framing note: the algebra of the rewriting (one product-to-sum identity) is deliberately simple — the contribution is what it changes structurally (member VJPs of a group become independent nodes of the backward dependency graph, the exact chain recovered at the finest partition), and the analysis below is the price of exactly that structural intervention. The new analysis is now a subsection of the revision (App. A.6, "Descent and convergence under grouped gradients"); we state it in full here.

AuthorsOpenReview ↗

[Response Theme 02] the added analysis — descent, convergence, and scheduled debiasing (revision App. A.6, stated in full) (W1/Q1)

The subsection collects per-block errors into a bias vector δΠ(w)=∇~ℓ(w)−∇ℓ(w)\delta_\Pi(w) = \tilde\nabla\ell(w) - \nabla\ell(w) (its norm decomposes orthogonally across blocks — member VJPs touch disjoint parameters, Def. 1) and proves three statements.

Proposition 3 (descent direction). ∇~ℓ\tilde\nabla\ell is a descent direction iff ⟨∇~ℓ,∇ℓ⟩>0\langle\tilde\nabla\ell, \nabla\ell\rangle > 0; a sufficient condition is ∥δΠ∥<∥∇ℓ∥\|\delta_\Pi\| < \|\nabla\ell\|, and any bias with ∥δΠ∥≤ϵ\|\delta_\Pi\| \le \epsilon satisfies cos⁡(∇~ℓ,∇ℓ)≥1−tΠ1+tΠ\cos(\tilde\nabla\ell, \nabla\ell) \ge \frac{1-t_\Pi}{1+t_\Pi} with tΠ=ϵ/∥∇ℓ∥t_\Pi = \epsilon/\|\nabla\ell\| — the cosine lower bound is a function of mm through the omitted-family counts (App. A.5, Prop. 2). Every measured cosine is positive: static probes rise with mm — 0.83/0.91/0.95 at m=1/5/8m{=}1/5/8 on PreactResNet-32 (App. F, Fig. 6 submission / Fig. 8 revision); along a full standard-BP trajectory (nine checkpoints) cosine stays positive for every mm — fixed m=4m{=}4 decays 0.80→0.39, m=8m{=}8 never below 0.79, m=1m{=}1 bottoms at 0.18 without turning negative (Fig. 8 submission / Fig. 10 revision). On the LLM test bed the staged executor is exact at the finest partition (cosine 1.000000, rel ℓ2\ell_2 6.6e-8); the m=2m{=}2 cross-stage config measures cosine ≈ 0.52 — the honest price of the 7-stage grouping buying the ×1.50 backward speedup. At fixed m=2m{=}2 the 8B fine-tuning is much weaker (IFEval 0.159 vs 0.311 standard) — not every task trains at low fidelity — while every schedule ending exact trains normally, cosine positive throughout.

Proposition 4 (convergence to a bias-controlled neighborhood). Under LL-smoothness, bounded stochastic noise σ2\sigma^2, and ∥δΠt(wt)∥≤ϵt\|\delta_{\Pi_t}(w_t)\| \le \epsilon_t, the update wt+1=wt−ηdtw_{t+1} = w_t - \eta d_t with η≤1/(4L)\eta \le 1/(4L) satisfies 1T∑t<TE∥∇ℓ(wt)∥2  ≤  4(ℓ(w0)−ℓ∗)ηT+2(1+2Lη) 1T∑t<Tϵt2+2Lη σ2,\frac{1}{T}\sum_{t<T} \mathbb{E}\|\nabla\ell(w_t)\|^2 \;\le\; \frac{4(\ell(w_0)-\ell_*)}{\eta T} + 2(1+2L\eta)\,\frac{1}{T}\sum_{t<T}\epsilon_t^2 + 2L\eta\,\sigma^2, the standard biased-SGD guarantee (Ajalloeian & Stich 2020) in its additive-bias oracle form, with the bias supplied by the omitted-family counts rather than assumed. Two features are specific to GLBP: (i) the bias is structured, not adversarial — δΠ\delta_\Pi is a sum of specific omitted monomials (Thm. 1), so ϵ\epsilon shrinks deterministically as m→nm \to n: the error budget is a control variable; (ii) the schedule dissolves the cross-group bias by ending at the finest grouping — CIFAR (b=1b{=}1, m→nm{\to}n): final stage is exactly standard BP (ϵt→0\epsilon_t \to 0); LLM: ends at m=B=8m{=}B{=}8 (one bundle per group), where the grouped operator equals the exact bundle-granularity backward since a bundle composes exactly (ϵt→0\epsilon_t \to 0) — the m=Pm{=}P anchor verified at cosine 1.0000001.000000. Bias is non-zero only in the earlier m<Bm < B stages, where a group spans several bundles.

Corollary 1 (type-B debiasing by the schedule). The first-order content of Δv\Delta_v counts qv(Π)=∣Gr(v)∩Dv∣≤∣Gr(v)∣−1q_v(\Pi) = |G_{r(v)} \cap D_v| \le |G_{r(v)}| - 1: it vanishes when vv's group is a single bundle (b=1b{=}1 singleton, or one bundle at m=Bm{=}B — a bundle composes exactly, so qv=0q_v{=}0 and the operator is exact at bundle granularity). Across the mm-increasing stages the group containing vv only shrinks, so the per-stage ceiling ∣Gr(v)∣−1|G_{r(v)}|{-}1 on qvq_v is non-increasing stage to stage and reaches zero once the schedule hits m=Bm{=}B (cut rotation at fixed mm within a stage leaves qvq_v fluctuating beneath that ceiling — monotonicity is the stage-to-stage statement). The schedule thus drives per-parameter first-order bias to zero by the final stage. The matched-schedule control measures this quantity's effect on the optimum reached: at identical compute (LLaMA, MetaMath, 4,000 steps, identical everything except the schedule), fixed g=4g{=}4 reaches GSM8K-strict 0.5519, the two-phase switch 4→84{\to}8 reaches 0.6531, and the progressive 4→5→6→84{\to}5{\to}6{\to}8 reaches 0.6755 — among the arms that begin biased, the earlier and more gradually the biased population is drained, the better the final point. (Standard BP, 0.6467, is unbiased from step 0 and sits below progressive; it does not fit this ordering — that is the training-loss/downstream dissociation the paper reports without explaining, not a prediction of the bias analysis.)

Remark (what the cosine certifies). Observing cos⁡=c\cos = c certifies relative bias tΠ≥1−c1+ct_\Pi \ge \frac{1-c}{1+c}; a rule firing at cos⁡<ρ\cos < \rho fires exactly when the relative bias provably exceeds 1−ρ1+ρ\frac{1-\rho}{1+\rho} (ρ=0.5\rho{=}0.5: one third) — monitor and theory control the same quantity. What cosine does not certify: a large cosine does not upper-bound the bias; Prop. 4's guarantee runs through the count-based bound, not the cosine.

AuthorsOpenReview ↗

[Response Theme 03] scope, and why this strengthens the paper

  • The propositions cover the non-convex stationarity regime; we do not claim global optimality or rates matching exact SGD during the biased phase (constants in Prop. 4 are not optimized).
  • Term-magnitude assumptions (e.g. ∥gPS∥≤c∣S∣\|gP_S\| \le c^{|S|} for scaled residuals) are stated as assumptions, not facts — the paper's operational assumption (§3.3) already separates polynomial bookkeeping from task-dependent dominance claims.
  • Deeper analysis — tight depth-dependent bounds on ϵ\epsilon, and the interaction of bias with AdamW's preconditioning — is marked as future work.
  • The full statements, proofs, and the integration point (immediately after the existing bias-bound subsection, App. A.5) are in the revision, compiled and verified against the paper's notation.

We believe this analysis converts the reviewer's main concern into a strengthening of the paper: the type-B first-order bias is real, is precisely characterized by Theorem 2, is bounded by mm-controlled counts, is empirically consistent with positive cosine at every measured checkpoint, and is scheduled toward zero by the progressive design — with the cosine-threshold controller (measured at two depths; at n=54n{=}54 — pre-act, flat-grouping convention, paired baseline — it fires three times and the adaptive arm beats fixed-m=4m{=}4 by 6.5 pts while running coarser for the first third of training) as its operational form. We hope this, together with the four-seed protocol in the revision, supports raising the assessment.

Reviewer_Z2pfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a method to approximate backprop through sequential residual blocks by instead computing the VJPs in parallel within a "group". They provide theoretical results, run experiments comparing this approximation of backpropagation to regular (full) backprop, and benchmark their approach.

Critical strengths

The paper is well written and easy to follow, and the idea is quite interesting. The math describes the effect of this approximation, and the results are relevant (e.g., Llama 8B training).

Critical weaknesses

In my understanding, the main benefit of this approach is to reduce the wall clock time by parallelizing the backwards pass. However, since the loss curves show that backprop converges faster (in number of iters), it is possible that the slower convergence may erase any wall clock advantages. It would be useful to see wall clock time vs loss.

Secondly, the main drawback to backprop (i.e., pipeline bubbles) can already be minimized through modern pipeline-parallel methods (Zero Bubble, interleaved 1F1B). It would be useful to see comparisons against these methods for MFU, throughput, etc. Without these comparisons it is difficult to establish the value of this method.

Finally, the cosine similarity to true gradients seems to fall quite low (as low as 0.39) over the course of training, which is concerning as it shows the limitations of this approximation.

Questions for authors

Could you please share a wall clock time vs loss curve for the llama 8B training? And could you please also share a comparison against zero bubble / interleaved 1F1B style training?

9 comments
AuthorsOpenReview ↗

[Response Theme 00] Index: where each of your three concerns is answered

We thank the reviewer for a focused review — all three concerns are actionable, and all three are answered with new measurements completed during the discussion period. Where each is answered:

  • W1 / Q1 ("slower convergence may erase wall-clock advantages; please show wall-clock time vs loss for the LLaMA-8B training") → answered in [Response Theme 01: wall-clock vs loss for LLaMA-8B — the three-tier measurement plus the equal-budget fine-tuning curves] below: the three-tier table (standard BP 3019 ms → standard pipeline 2001 ms → GLBP+pipeline 1635 ms per step, same batch) plus the fixed-1-hour-budget fine-tuning runs, where the m=2m{=}2 arm processes 1.14× more steps — with both sides of the speed/fidelity tradeoff reported.
  • W2 / Q2 ("pipeline bubbles can already be minimized by Zero Bubble / interleaved 1F1B; please compare against them for MFU and throughput") → answered in [Response Theme 02: comparison with the pipeline-scheduling family (GPipe / 1F1B / interleaved) — complementary mechanisms, one composition realized and measured]: the quantity no scheduler can touch (within-stage backward depth) is identified; the paper realizes one composition in the main text — with GPipe (Figure 3, Table 4), measured head-to-head on the 8-GPU test bed (standard pipeline 2001 ms vs GLBP+pipeline 1635 ms at the same batch); the throughput/MFU conversion of the same three tiers (2,713→5,010 tokens/s, ×1.85; MFU 11.0%→20.4%) is in Theme 01 below; dependency-valid simulations for 1F1B and interleaved-1F1B (same micro-batch order, only B's duration shrinks) are provided as supplementary figures at the anonymized link below.
  • W3 ("cosine similarity falls as low as 0.39, which shows the limitation of the approximation") → answered in [Response Theme 03: cosine falling to 0.39 — expected, characterized, and the reason scheduling exists]: 0.39 is the fixed-coarse-operator diagnostic that motivates the schedule (not the fidelity of the trained arm), the same decay appears in Highway-BP's own dynamics analysis (its §5.3), and descent is preserved (cos⁡>0\cos>0) at every measured checkpoint — with a cosine-threshold adaptive schedule measured at two depths as the operational response.
AuthorsOpenReview ↗

[Response Theme 01] wall-clock vs loss for LLaMA-8B — the three-tier measurement plus the equal-budget fine-tuning curves (W1/Q1)

We ran the decisive comparison on LLaMA-3.1-8B across 8 GPUs (8 pipeline stages × 4 layers, bf16, seq 512, total batch 16 held constant in every arm), separating forward and backward wall-clock:

tierconfigurationfwd (ms)bwd (ms)full step (ms)e2e vs tier 1 (unsched.)e2e vs tier 2 (GPipe)
1. Standard BP, no schedulingwhole-graph autograd, layers sharded over 8 GPUs102119983019——
2. Standard pipeline schedulingGPipe order, exact chain, its best micro-batching (M=8M{=}8)10899122001×1.51—
3. GLBP + pipeline ★cross-stage m=2m{=}2 (exact-tail head + one 7-stage group), its best M=1M{=}110286071635×1.85×1.22
3b. GLBP layered onto the filled pipeline (M=8M{=}8, m=2m{=}2)GPipe micro-batching kept, grouping added10926681760×1.72×1.14

Three readings of the same grid, each against its own baseline (we report them separately and do not multiply them into a product). (i) Against unscheduled standard BP (tier 3 vs tier 1): ×1.85 end-to-end, backward ×3.29 — a combined figure for the two mechanisms present in that configuration. (ii) Best-vs-best at the same total batch 16 (tier 3 vs tier 2, each at its own best micro-batching): ×1.22 end-to-end, backward ×1.50 — here the within-stage span cut substitutes for heavy micro-batching rather than stacking on it: the coarse grouping removes the bubble that M=8M{=}8 exists to fill, so GLBP's fastest configuration wants M=1M{=}1. One component of this ×1.22 is the larger per-GPU kernels at M=1M{=}1 (forward 60.9 ms faster, ≈17% of the end-to-end saving) — and the reason this configuration exists at all is the span cut: with the exact backward, M=1M{=}1 is the slowest configuration (3037 ms, slower than unscheduled standard BP's 3019 and than GPipe's 2001; App. cost-audit), so the kernel-size benefit was unreachable without GLBP making M=1M{=}1 viable. The mechanism-isolated number, which we also report, is tier 3b. (iii) Layered onto a filled pipeline (tier 3b, previously unreported): keeping GPipe's best micro-batching (M=8M{=}8) and adding the grouping gives ×1.14 end-to-end (backward ×1.36) — the honest marginal gain over a scheduler already operating at its own optimum, and the conservative floor for compositions with lower-bubble schedulers such as Zero-Bubble. At matched wall-clock against unscheduled standard BP, the tier-3 per-step budget buys 1.85× more steps. We do not claim a lower training loss at equal wall-clock — GLBP is by construction a lossy-gradient method, and the wall-clock–loss curve below shows the progressive arm's training loss slightly above standard's at matched elapsed time.

We make two separate claims, measured on two separate instruments, and we are careful not to fuse them. (i) Quality, measured on the main fine-tune (4-GPU, effective batch 128, the configuration of Table 4 submission / Table 3 revision): at matched steps and at the end of the run, progressive GLBP is competitive with standard on all three tasks (IFEval 0.311/0.311, HumanEval 0.543/0.518, GSM8K-strict 0.676/0.647, GSM8K-flex 0.721/0.755). (ii) Cost, measured on the 8-GPU pipeline test bed as a per-step timing of the same schedule: the full Alpaca run integrates to 21% less wall-clock than standard BP (4.84 vs 6.14 h). These are a quality measurement and a cost measurement, not one run asserted to be both faster and better — the downstream numbers and the wall-clock figure come from different hardware/batch settings, and we state each on its own. (Gradient fidelity of the staged executor verified at the exact anchor: cosine 1.000000, rel ℓ2\ell_2 6.6e-8 vs single-process gradient accumulation.)

AuthorsOpenReview ↗

[Response Theme 01b] Throughput and MFU, the metrics as asked

Throughput and MFU, the metrics as asked. Converting the same tiers (total batch 16, seq 512, LLaMA-3.1-8B, 8×H20) into tokens/s and MFU (training FLOPs ≈6N⋅\approx 6N\cdottokens, N=8.03N{=}8.03B; peak 148 TFLOPS/GPU):

tierstep (ms)throughput (tokens/s)sustained (TFLOPS)MFU (8×H20)
standard BP, no scheduling30192,713130.711.0%
standard pipeline (GPipe, M=8M{=}8)20014,094197.216.7%
GLBP + pipeline (m=2m{=}2, M=1M{=}1)16355,010241.420.4%
GLBP layered onto filled pipeline (M=8M{=}8, m=2m{=}2)17604,650223.918.9%

GLBP+pipeline raises throughput from 2,713 to 5,010 tokens/s (×1.85) over unscheduled standard BP and from 4,094 to 5,010 (×1.22) over standard pipeline's best configuration — the same factors as the step-time ratios, by construction (identical batch) — while layering the same grouping onto GPipe's own best micro-batching gives 4,650 tokens/s (×1.14). The MFU floor of the no-scheduling tier (11.0%) nearly doubles to 20.4% (18.9% for the layered variant). The MFU absolute level reflects our research executor (no fused kernels), not a production runtime; the relative gains are what was asked, and one component of the tier-3 gain is the larger per-GPU kernels at M=1M{=}1 (micro-batch 16 vs 2) — a real effect of the substituted configuration, which is why we report the tier-3b variant that keeps micro-batching fixed.

Comments on the paper

21 comments
AuthorsOpenReview ↗

[To All Reviewers] The soul of the paper: the primary question each method must answer

We thank all reviewers. We open with one admission and one comparison, because together they are the hinge of this rebuttal. The admission (repeated up front): our writing did not bring the paper's core to the surface, which let several reviews read GLBP as a variant of Highway-BP. That reading is ours to correct. Before any detailed point, we therefore put the single most essential comparison first — not of metrics, but of the primary question each paper exists to answer (its soul), and whether that question was answered.

The primary question the paper must answer (its soul)Was it answered?
Highway-BP (ICLR 2025 Oral) (OpenReview)Can a brute-force iterative polynomial-solving algorithm be turned into a real training speedup? — that its iteration is a brute-force march through the residual-path polynomial (one order per round, each round an nn-wide sweep) is our characterization of its Thm. 2 / Eq. 7–8, derived by induction on Highway-BP's own Theorem 2 in our reply to Reviewer Vmtv, Theme 02b; the acceleration goal is stated in its own text, not only the title: its abstract introduces "a parallelizable iterative algorithm that approximates backpropagation… and show that major speedups can be achieved with minimal performance degradation," and its first listed contribution is "a parallelizable iterative algorithm that approximates backpropagation for accelerating the training of deep sequential models" (its §1)Partially — and the paper says so itself. The measured speedups are confined to weight-tied RNNs: Table 3 ("Speedup of training with Highway-BP vs. backpropagation for the RNN experiments," ×2–×3) and the only loss-vs-runtime plot, Figure 5, is the 3-layer GRU RNN; the ResNet/GPT-2/RoBERTa experiments report final performance vs. the iteration count kk (Figure 2), never measured training time. Its own Discussion (§6) concedes the scope: "We leave its practical implementation for training large models in a distributed setting for future work. We however show that RNNs can be sped up considerably on a single GPU, as all cells share the same weights." So the acceleration the title promises is delivered for weight-tied RNNs on one GPU, and deferred for large distributed models
GLBP (this paper)Can an algorithm that claims to break backpropagation's reverse serial dependency actually break it and deliver parallel speedup? — basis: breaking the serial dependency is the core idea of our claim, so delivering that break as speedup is the question we are obliged to answerYes — by more than one experiment, at scale, with every theoretical claim cashed out. (i) Direct speedup of standard BP: Qwen3-8B ×1.34 (4 GPU), Qwen3-32B ×1.63 (8 GPU) — submitted paper (App. E / revision Table 8), both vs their respective unscheduled standard-BP baselines. (ii) On an 8-GPU pipeline test bed at 8B: LLaMA-3.1-8B — best-vs-best at matched batch, ×1.22 end-to-end over standard GPipe's best configuration (backward ×1.50); ×1.85 vs the unscheduled baseline as the combined figure for the two mechanisms; ×1.14 when layered onto a pipeline already running at its best micro-batching — revision Table 4 + Figure 3. (iii) The serial span is cut in every instance — 600→20 at n=600n{=}600 (ResNet-1202). Theory delivered in full: member VJPs become independent backward-graph nodes (Def. 1), span m⋅bm{\cdot}b at work exactly nn (Prop. 1), the retained-path family (Thm. 1), exact recovery at m=nm{=}n, and the descent / biased-SGD convergence guarantees (revision App. A.6)

The two souls are different questions, and that difference is the whole of our novelty: Highway-BP asks whether low-order enumeration suffices and leaves the acceleration it is named for largely to future work; GLBP asks whether the serial dependency can be broken into real parallel speedup, and answers yes — on large models, across several experiments, with all theory realized. Everything below elaborates this one contrast.

One scoping note, so the soul is not misread. Our core contribution is the single thing above — breaking backpropagation's serial dependency. Path scheduling is not a second core claim: it is a means we use when low-order precision proves insufficient, and tuning a schedule optimally is, at bottom, the same question as "which task needs which fidelity" — an open question this paper deliberately does not study. We include a few representative schedules as feasibility instances only; the break is the contribution, the schedule is a knob the break makes free to turn.

AuthorsOpenReview ↗

[01] what is our work, and what is not

We thank all reviewers. Before answering each review, we owe a general admission and a general statement.

The admission. The submission did not state our core contribution sharply enough — several reviews read the paper as "a different way of computing the reverse-mode polynomial," the opposite of what it is; the revision rewrites the framing. We state which questions this paper does and does not research, beside the closest prior work, Highway-BP (ICLR 2025 Oral) — superficially similar, in essence a different kind of work.

The to all reviewers table. Each row is a research question, not a metric.

Research questionHighway-BP (ICLR 2025 Oral)GLBP (this paper)Is this what we research?
"Which task needs which gradient fidelity?" — the open question left by Veit et al. (2016) (where the reviewers' "how to choose mm / kk" questions live)sweeps kk per task (ResNet-110 sweep k=0..7k{=}0..7, Fig. 2); small-parameter models only; no conclusion — the question stays openwe do not claim all tasks train at low order, and do not answer it — low-order training appears only as a feasibility verification of the rewritten operator; the zero-cost fidelity knob we contribute is a practical instrument for future research on it (a first adaptive-criterion probe, two depths — iqo4 reply, Theme 04)No — explicitly out of scope
"Is this a brute-force algebraic solver of the Veit et al. (2016) polynomial?"yes — that is its essence (derived, not asserted, by induction on Highway-BP's own Theorem 2 in our reply to Reviewer Vmtv, Theme 02b): k+1k{+}1 rounds (Thm. 2, Eq. 7–8), one polynomial order per round, each a full nn-wide sweep; its sweep runs kk to 7 (Fig. 2), covering ∼ ⁣10−6%\sim\!10^{-6}\% of the 2542^{54} terms; exactness (k=nk{=}n) costs (n+1)n(n{+}1)n VJPs/step, serial depth unchangedno — the polynomial is never assembled: within-group products become sums, cross-group products are retained, the rewritten (nilpotent) system is solved exactly in one sweep; the object modified is the backward operator itself, not the way its terms are enumeratedNo — precisely what we argue against; our claim is the operator rewrite, low-order runs only verify it
"What is the essence of its parallelism?"a liability forced by the brute-force solve (each of its two parallel components traced to the enumeration it serves, and to no serial edge it removes, in our reply to Reviewer Vmtv, Theme 03b) — the nn-wide computation exists only because assembling a polynomial order separates its terms onto nn resources (Eq. 8); it shortens span by spending n×n\times work across nn-wide resources rather than removing the serial edge, realized only where the enumeration collapses into batched GEMMs — weight-tied RNNs, single GPU (Table 3; §6 defers distributed large models)a real gain from breaking the backward chain: the serial edge is deleted at unchanged work, so parallelism exists because the dependency no longer does; realized on both routes — standard-BP acceleration (Qwen3-8B/32B, 4/8 GPUs, ×1.34/×1.63 vs unscheduled) and an 8-GPU pipeline test bed (LLaMA-3.1-8B: best-vs-best ×1.22 e2e over standard GPipe's best config at matched batch; ×1.85 vs unscheduled as the combined figure). The progressive schedule's two facts are reported separately on different instruments: downstream quality preserved (main 4-GPU/batch-128 fine-tune) and, as a distinct cost measurement, 21% less full-run wall-clock (8-GPU test bed) — never one run claimed both faster and betterYes — the paper's core claim: breaking backpropagation's local serial dependency, with theory (retained-path family, cost invariant, descent/convergence) and experiments in both regimes
"Does it adapt to large models, the regime that actually matters?"largest model 124M (RoBERTa, post-LN, γ=0.8\gamma{=}0.8 layer re-split — App. C.3); deepest n=54n{=}54; no practical n=600n{=}600 arm (k=2k{=}2 alone costs 1,8001{,}800 VJPs/step; App.~G handling not in our harness)LLaMA-3.1-8B, unmodified pre-LN, 8-GPU pipeline; ResNet-1202 (n=600n{=}600) unmodified; fidelity raised mid-training at zero extra cost when low order is insufficient (matched schedule control — Vmtv reply, Theme 07)Yes — the large-model regime is the main battlefield; low-order small-scale runs are verification, never the destination
AuthorsOpenReview ↗

[01b] (continued): lineage and qualifiers

Two clarifications prevent misreadings. On lineage: both methods build on the path view of Veit et al. (2016) — Highway-BP's §2.1 calls it "precisely the motivation behind our work" — then do opposite things with it (brute-force the polynomial vs. rewrite the operator). On our claim's qualifiers: to our knowledge, GLBP is the first method that breaks backpropagation's local serial dependency without delayed/stale gradients, without extra per-step work, and with exact recovery at the finest partition — prior parallel-backward routes (DDG; DeepPCR; layer-parallel training; Highway-BP) each relax at least one; the revision's related-work positions all of them with these qualifiers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.