LBI: Parallel Scan Backpropagation via Latent Bounded Interfaces

Backpropagation is sequential across depth, an $O(K)$-deep dependency chain that bottlenecks parallel training. Parallel-scan formulations reduce the depth to $O(\log K)$ but compose $d\times d$ Jacobians at $O(d^3)$ per combine. We introduce Latent Bounded Interfaces (LBI), which restrict inter-region communication to an interface $m_k\in {\mathbb{R}}^r$ with $r\ll d$, so the adjoint recursion becomes a suffix scan over $r\times r$ Jacobians with the gradient exact under the bounded-interface model. The same restriction bounds what crosses a device boundary: per-sequence states and $r\times r$ Jacobians in place of per-token activations, so a region-parallel training step's critical path is kilobytes per link regardless of width, and its one activation-sized transfer is a gradient that can be accumulated. Implemented as fused forward-mode kernels on four H100s, the step runs $14\times$ faster than pipeline parallelism at 1 Gbit/s and $30\times$ faster at 100 Mbit/s of link bandwidth, which extends exact-gradient training to multi-region and decentralized links. On language modeling, bounded-interface models train across Mamba-3, Transformer, and Hybrid backends at small interface ranks.