acceptodds
Under review as a conference paper at ICLR 2027

LBI: Parallel Scan Backpropagation via Latent Bounded Interfaces

Abstract

Backpropagation is sequential across depth, an -deep dependency chain that bottlenecks parallel training. Parallel-scan formulations reduce the depth to but compose Jacobians at per combine. We introduce Latent Bounded Interfaces (LBI), which restrict inter-region communication to an interface with , so the adjoint recursion becomes a suffix scan over Jacobians with the gradient exact under the bounded-interface model. The same restriction bounds what crosses a device boundary: per-sequence states and Jacobians in place of per-token activations, so a region-parallel training step's critical path is kilobytes per link regardless of width, and its one activation-sized transfer is a gradient that can be accumulated. Implemented as fused forward-mode kernels on four H100s, the step runs faster than pipeline parallelism at 1 Gbit/s and faster at 100 Mbit/s of link bandwidth, which extends exact-gradient training to multi-region and decentralized links. On language modeling, bounded-interface models train across Mamba-3, Transformer, and Hybrid backends at small interface ranks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Model Training, Backpropagation, Parallel Scan

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4qfgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Latent Bounded Interfaces (LBI), a modeling and systems framework for enabling exact parallelized backpropagation across depth-partitioned neural network regions. The key idea is to constrain all inter-region communication through a low-dimensional latent interface.

The paper makes three main contributions. First, it gives a formal graph-based treatment of sufficient interfaces and proves that inter-region adjoint transport factorizes through the latent interface chain. Second, it proposes a concrete architecture in which each region communicates only through encoded/decoded low-rank latent states while receiving a shared static embedding canvas. Third, it presents a systems implementation using fused forward-mode Jacobian construction and evaluates the method on 4x H100 hardware under both high-bandwidth and constrained-network settings.

Critical strengths
  • The paper identifies a genuinely novel and important perspective on exact parallel backpropagation. The central insight - that scan-based exact backpropagation is only computationally prohibitive because the communicated object is full-rank, and that one can redesign the model class to make the scan tractable - is both elegant and consequential. This is the main reason I view the work as meaningful rather than merely incremental.

  • The communication argument is strong and empirically grounded. The systems contribution is not just asymptotic. The paper quantifies critical-path bytes, implements the method on real multi-GPU hardware, and demonstrates substantial speedups over pipeline parallelism in low-bandwidth regimes that are plausibly relevant for cross-region or decentralized settings. This makes the work much more compelling than a purely theoretical proposal.

Critical weaknesses
  • The quality gap to dense models is substantial, and this weakens the paper’s practical impact. A degradation of roughly 0.33-0.67 CE is large enough that many readers will question whether the architectural restriction is too costly for mainstream language modeling. Since the paper’s exactness guarantee is only internal to the restricted model class, the central practical question becomes whether that class remains competitive.
Questions for authors
  • The paper emphasizes exactness, but exactness is with respect to the LBI model rather than the original dense model. Could the authors sharpen this distinction earlier and more explicitly, perhaps already in the abstract/introduction, to avoid possible overinterpretation?

  • Can the authors provide more intuition for why performance is largely flat in interface rank r but improves materially with coarser regional granularity? Does this suggest that the main bottleneck is not interface capacity per se, but repeated information bottlenecking across many region boundaries?

  • How sensitive are the results to the specific encoder/decoder implementation? Since the current architecture adds substantial parameter overhead while still underperforming dense baselines, it would help to know whether simpler or more structured interface modules were tested and what happened.

2 comments

Thank you for your review and time. On exactness, we agree it is exactness for the bounded-interface model, not the dense model, and the introduction should make this clearer where it compares against prior approaches. On rank vs. granularity, we think your reading is correct, from the architecture the residual stream does not cross a boundary, so each region restarts from the shared canvas plus a single r-dimensional summary pooled over the sequence. Per-token features computed upstream are lost at each boundary whatever r is, so once r is large enough the cost scales with the number of boundaries rather than their width. This is the issue we are now working on. On parameter overhead, a dense Mamba-3 model matched on total parameters (140.6M against 140.8M) trains to 2.643, which widens the gap to 0.57, so the gap is not a capacity effect (section 4.1, appendix E). On encoder/decoder sensitivity, every run uses separate per-region two-layer SiLU MLP encoders and decoders of hidden width d (section 3.1), and we did not sweep simpler modules. Section 6 lists the ones we would try, such as a linear projection at Θ(rD)\Theta(rD), gated linear projections, and cross-attention interfaces.

Thank you for the clear and open reply. Your explanation makes the trade-offs much easier to see. I appreciate you sharing this, and best of luck with the next version.

Reviewer_dxsEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Latent Bounded Interfaces (LBI) to make parallel-scan backpropagation tractable. The key observation is that the intractability of scan-based backpropagation (as in BPPSA) is not inherent to the scan formulation but to the size of the objects being scanned: composing d x d Jacobians costs O(d^3) per combine. LBI restricts inter-region communication to a low-dimensional interface m_k in R^r with r << d, reducing the adjoint recursion to a suffix scan over r x r Jacobians with O(r^3 log K) span. The paper proves that this scan-based backward recovers the exact gradients of standard reverse-mode backpropagation under the bounded-interface model (verified to machine precision in float32). The approach decomposes into three phases: embarrassingly parallel Jacobian construction, a lightweight suffix scan, and embarrassingly parallel region-local backward. The bounded-interface model is an architectural imposition — the model learns a compact interface carrying only what downstream computation needs, inspired by Multi-Head Latent Attention. Implemented as fused forward-mode kernels on four H100s, the region-parallel step runs 14x faster than pipeline parallelism at 1 Gbit/s and 30x faster at 100 Mbit/s. On language modeling (47M-321M parameters), bounded interfaces cost 0.33-0.67 CE across Mamba-3, Transformer, and Hybrid backends.

Critical strengths

The theoretical framework is clean and well-developed: the formalization of bounded interfaces as vertex separators in the computational graph (Definition 2.2), the adjoint factorization lemma (Lemma 2.3), and the three-phase decomposition (Proposition 3.1) together provide a principled foundation. The key insight — that the scan's tractability depends on interface dimension, not on the scan structure itself — is both correct and practically important. The gradient exactness proof (Theorem 2.7) is a strong result that distinguishes LBI from prior approximate methods. The hardware measurements are thorough: the paper reports critical-path bytes, wall-clock times on both emulated and shaped links, memory profiles, and kernel-level breakdowns. The finding that RP-32 runs 30x faster than pipeline parallelism at 100 Mbit/s — extending exact-gradient training to decentralized links — is a compelling practical result. The observation that the interface rank can be as small as r = 8 without quality degradation is surprising and useful.

Critical weaknesses

(1) The bounded-interface property is explicitly acknowledged as an architectural imposition, not a consequence of existing model structure (Remark 2.5: "the minimum vertex separator... has size d_k"). This means LBI cannot be applied to standard pretrained models without architectural modification and retraining. The paper trains models from scratch at 47M-321M parameters, which is far below production scale. Whether the interface compression remains viable at billion-parameter scale, or whether the CE gap (0.33-0.54) widens, is not addressed. (2) The CE gap of 0.33-0.67 between LBI and dense models is substantial for language modeling. While the paper shows this gap is flat across interface ranks r = 8 to 64 and narrows with coarser regions (0.16 at nine-block regions), it does not close. The paper does not investigate whether this gap reflects a fundamental information bottleneck or an optimization difficulty that could be addressed with better training strategies (e.g., curriculum, distillation from dense models). (3) On NVLink (the dominant interconnect in production training), the region-parallel step runs at only 0.88x the pipeline's speed on Mamba-3 and slower on Transformer (0.42x) and Hybrid (0.59x). The method's advantage is confined to weak interconnects (100 Mbit/s to 1 Gbit/s), which, while important for decentralized training, represent a narrow deployment scenario. The paper's title and abstract emphasize parallel scan backpropagation broadly, but the practical wins are specific to low-bandwidth settings. (4) The Jacobian construction is the dominant cost (c = 14.5 forward passes, near the no-reuse bound c = r), and the paper acknowledges this places the crossover with sequential backpropagation at K ~ 16 for r = 8. This means the method is only beneficial for deep models with many regions, which limits applicability. The kernel-level profiling shows the construction is latency-bound at 12.5% occupancy, suggesting significant room for kernel optimization that is not explored. (5) The three-engine comparison (forward-mode, reverse-mode, end-to-end autograd for Jacobian construction) is mentioned but not thoroughly compared. The forward-mode construction takes 1200 ms vs. 202 ms for autograd, which raises questions about whether the chosen implementation is optimal. (6) The evaluation uses only 4 H100s with K = 4 regions, which is below the crossover point for wall-clock advantage. The paper acknowledges this ("K = 4 lies below the crossover K ~ c + 1"), but it means no real hardware speedup over sequential training is demonstrated at the evaluated scale — only projected.

Questions for authors

Q1. The CE gap of 0.33–0.67 between LBI and dense models is the most significant concern. Can you decompose this gap — is it primarily an information bottleneck (the interface rank r is too small to carry the necessary gradient information) or an optimization difficulty (the model fails to learn efficient interfaces)? Have you tried distillation from a dense model or curriculum learning to close the gap? Q2. On NVLink (the dominant production interconnect), the region-parallel step is slower than pipeline parallelism (0.42x–0.88x). Can you profile the kernel-level bottlenecks and estimate at what K and r the crossover occurs? What hardware or model configurations would make LBI beneficial on standard datacenter interconnects? Q3. The Jacobian construction is the dominant cost (c = 14.5 forward passes, near the no-reuse bound c = r). The paper mentions three construction strategies (forward-mode, reverse-mode, autograd) but forward-mode takes 1200 ms vs. 202 ms for autograd. Why was forward-mode chosen? Would switching to autograd-based construction reduce the crossover K and improve practical wall-clock performance? Q4. The bounded-interface property is an architectural imposition requiring models to be trained from scratch. Is there any path to applying LBI to pretrained models — e.g., through distillation, interface injection via adapters, or fine-tuning? What is the minimum retraining cost to adapt an existing 7B model to the bounded-interface structure? Q5. The evaluation is at 47M–321M parameters. Have you tested at 1B+ parameters? Does the CE gap widen, narrow, or stay flat with scale? The interface rank r = 8 works at 321M — does the optimal rank scale with model dimension, or is it dimension-independent? Q6. The paper claims gradient exactness under the bounded-interface model. How is the bounded-interface condition enforced during training — is it a hard architectural constraint (e.g., a bottleneck layer) or a soft penalty? If soft, what happens to gradient exactness when the constraint is violated? If hard, how does this constraint interact with expressivity?

1 comment

Thank you for your detailed review and questions. Briefly for each:

Q1: Rank is not the limit, since validation CE is flat from r=8 to 64 on a truncated schedule, and at the full budget the Mamba-3 gap is 0.54 CE at r=8, 16, and 32 (Table 9). The gap does fall from 0.53 to 0.16 CE as regions coarsen from two to nine blocks (Table 12), which points to the number of boundaries rather than per-interface capacity. We have not tried distillation or curriculum learning, so we cannot yet separate information loss from optimization difficulty.

Q2: We agree, no wall-clock gain is shown at K=4 over sequential training. The projected crossover is K≈9K \approx 9 at r=8 and K≈16K \approx 16 at r=16 (section 4.2, appendix C). The construction dominates the step, with 61% of its time in the two chunkwise-SSM scan kernels (section 4.2). The Transformer is slowest on NVLink because its regions are compute-bound, so the construction has no primal work to reuse and costs 0.42r forward passes. On datacenter interconnects, the method needs more regions (K above the crossover), a smaller r, and a faster construction. A faster construction alone lowers the projected crossover, from about K=16 to K=2 (section 6), but doesn’t help at K=4, because the rest of the last device’s critical path (~42 ms of chain fill, local and backward reductions) already exceeds the sequential reference (36.4 ms, table 3).

Q3: The 202 ms row is not a Jacobian construction; it is an ordinary end-to-end backpropagation (a single reverse pass), which produces no interface Jacobians and so it cannot be split across regions. The scan is meant to replace this sequential computation, and we used plain backpropagation for the quality runs in Table 1 because the gradients are identical. Switching to it would therefore not reduce the crossover. Of the constructions that produce the Jacobians, forward-mode was the fastest we measured (1200 ms against 2056 ms for reverse-mode). Appendix D lists the third engine as an autograd graph construction of JkJ_k, which is a different path from the end-to-end autograd timed in Section 4.2, and the paper did not make this distinction clear.

Q4/5: We have no pretrained-model path. Pretrained models have no small vertex separator, so the interface has to be imposed architecturally, and Section 6 leaves conversion of a pretrained dense model as an open direction. We also have no runs above 321M parameters. Over the range we do have, the Mamba-3 gap widens mildly, from 0.51 at 54M to 0.63-0.67 at 321M (Table 11), so we don’t claim that it closes with scale. These runs keep two-block regions, so the larger models also have more regions (K=7 to 24), and the gap moves with the region count (appendix E). The rank sweep (Table 9) was run at a single model size per backend, so we cannot say whether the optimal rank depends on width. We leave these questions to future work.

Q6: The constraint is a hard constraint and is architectural. Each region sees only the decoded r-dimensional state from the previous region plus the shared embedding canvas (section 3.1). Gradient exactness therefore holds by construction with no soft penalty (verified against autograd to machine precision in float32, Table 8). The expressivity cost of the constraint is the CE gap.

Reviewer_gjYbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors propose an architecture and a distributed training algorithm aimed at improving training efficiency when communication bandwidth between GPUs is limited. The method builds on the parallel-scan formulation of backpropagation [1]. In the proposed architecture each layer l+1 compute: $x=E+f(m_l) \longrightarrow h=F(x) \longrightarrow m_{l+1}=c(h) $ where E is shared embedding, f is decoder to extend small latent representation m_l from the previous layer, F is actual mamba/transformer layers. For the backward pass the sequence of actions is:

  1. Each GPU constructs its local Jacobian once its region’s forward inputs and activations are available. It does not need the loss gradient. This work can overlap with forward computation in later regions.
  2. GPUs share and combine the small Jacobians. Each GPU needs the combined effect of all downstream regions, not just the immediately next region. The scan computes these products.
  3. The final GPU supplies the terminal interface gradient.
  4. Each GPU applies its Jacobian product to that terminal gradient, obtains the gradient at its own region’s output, and starts its local backward. So, once the Jacobian products and saved forward activations are ready, the remaining missing information is the terminal interface gradient.

From my understanding this paper combines core idea of a parallel scan for back-propagation [1] and activation compression: LBI’s proposed addition is to change the architecture so that this existing algorithm operates on tiny Jacobians.

[1] BPPSA: Scaling Back-propagation by Parallel Scan Algorithm.

Critical strengths

I like the main idea to modify standard transformer architecture and training algorithm to make it more communication efficient. The idea to make Jacobian smaller to make it faster for a parallel scan backprop formulation is novel.

Critical weaknesses
  1. The presentation makes the actual architectural change unnecessarily difficult to understand. I found the writing very confusing and it took me sometime to at least understand what is the main idea. An architecture diagram and a concrete forward/backward example before the theoretical formulation would substantially improve readability.
  2. Experiments does not prove that the method is working. The largest model that is trained is < 200M parameters on 2.5B tokens. However, even at this small scale there is a significant gap in cross entropy loss with the baseline.
  3. It is unclear what is the core problem authors are trying to solve. If the main contribution is faster training with limited communication, then more comparison should be provided with existing compression / quantization methods. Standard back-propagation without parallel scan, with just small latent vector exchange should be presented as a baseline.

I would like to thank authors for their contributions. Despite the weaknesses, I think it is generally an interesting idea that needs improvement.

Questions for authors

I don't have questions to the authors.

2 comments

Thank you for your time and your review. Your summary of the three-phase backward is accurate, and we agree that presenting an architecture diagram and a worked forward/backward example before the formalism would make the paper easier to follow. On whether the method works, we separate it into two claims. The scan computes the exact gradient of the bounded-interface model (verified against autograd), while the CE gap is the cost of that model class relative to a dense model, which we agree is too large. (The main runs in Table 1 are 120-157M backend parameters, and the scale study in Table 11 goes up to 321M on 2.95B tokens.)

On your third point, yes, the main contribution is exact-gradient model-parallel training over weak links. The speedups hold below a few Gbit/s, and on NVLink the pipeline is faster (section 6). Section 4.2 compares against an emulated 4-bit compressed pipeline (region-parallel is 7.8x faster at 100 Mbit/s), which is a proxy rather than an implemented compression method.

The baseline you suggest, backpropagation on the bounded-interface model sharded across devices, would be a better comparison. Our pipeline and sequential baselines both use the dense stack, whereas this one would move the same kilobyte-scale per-sequence states without the Jacobian construction cost. At four devices, we expect it to be competitive with or faster than the scan, which puts the benefit of the scan at larger K. We agree that this baseline is necessary and that the quality gap must close before the method is practical.

Thank you for your reply. I will maintain my score, but I encourage authors to work on this idea.

Comments on the paper

1 comment

We thank the reviewers for their reviews. We agree with the central concerns: the training quality gap, the sub-billion scale, and that four devices sit below the projected crossover with sequential backpropagation. The reviews also converge on a diagnosis we share, with the cost coming from repeated bottlenecking across region boundaries rather than from interface width. Addressing it requires changes to the architecture rather than to the paper’s presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.