acceptodds
Under review as a conference paper at ICLR 2027

ResBM: Residual Bottleneck Models for Low-Bandwidth Pipeline Parallelism

Abstract

Unlocking large-scale, low-bandwidth decentralized training has the potential to utilize otherwise untapped compute resources. In centralized settings, large-scale multi-node training is primarily enabled by data and pipeline parallelism, two techniques that typically rely on high-bandwidth communication. While efficient methods now exist for decentralized data parallelism, pipeline parallelism remains a major challenge. Recent efforts, such as Subspace Models (SM), report up to activation compression but rely on constrained optimization of output projection layers. We propose a different approach: the Residual Bottleneck Model (ResBM), an architecture designed from the ground up for low-bandwidth communication environments while remaining applicable to standard transformer-based architectures. ResBM introduces a residual encoder–decoder bottleneck module across pipeline boundaries that can be trained end-to-end with standard optimizers and preserves an explicit low-rank identity path. In our main comparative experiments with up to 3.5B-parameter models trained for up to B tokens on large-scale web datasets, we show that ResBM achieves the lowest loss among evaluated methods at extreme compression rates of and . Although the authors of SM have reported little or no degradation from activation compression, our longer-horizon, optimizer-matched evaluations reveal a measurable loss gap for every compression method evaluated, including ResBM. However, ResBM consistently exhibits the smallest such gap relative to uncompressed baselines. Its loss gap remains approximately constant during later training, whereas SM's appears to grow over the evaluated horizon. We therefore position ResBM as a new state of the art for extreme activation compression in low-bandwidth pipeline-parallel training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Residual Bottleneck Models, Decentralized Training, Pipeline Parallelism, Activation Compression, Low-Bandwidth Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_B3ABOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Residual Bottleneck Models (ResBM) for communication-efficient pipeline parallelism. It introduces learned encoder–decoder bottlenecks between transformer blocks while preserving a low-dimensional identity path through the residual connections. Experiments on 2B–3.5B parameter models show that ResBM achieves lower training loss than the evaluated alternatives at 100–128× activation compression.

Critical strengths
  1. The idea is simple, interesting and optimiser-agnostic.
  2. Results are reasonable, and the considered low-bandwidth settings are meaningful.
Critical weaknesses
  1. It is unclear if the comparisons are fair. Especially, ResBM adds more parameters due to the learnable encoder-decoder parameters and the comparisons are not done in a parameter-matched setting. Specifically, it has more parameters than the uncompressed baseline and, more importantly, SMs have fewer trainable parameters, as discussed in the cited Agora implementation, and it is not clear if the comparisons with SMs are in the parameter-matched setting. This is critical to correctly compare the methods.
  2. The statement about SMs requiring constrained optimisation and Grassmann updates as a limitation is narrow and unsubstantiated, as the SM implementation that was used for comparison does not have these components and just uses the standard optimization paradigm.
  3. The identity project section is unclear, as it seems it just selects k components from the residual path and not sure how this would scale with no of pipeline stages. Overall, this is a big problem in PP compression, as these kinds of lossy compression errors would accumulate over a larger no of compression boundaries and raise questions about scalability and experimental validation.
Questions for authors

How are the hyperparameters chosen for each method?

2 comments

We thank the reviewer for the review, and in particular for raising the question of parameter-matched comparisons, which led to new experiments. We uploaded a revised PDF that uses the additional tenth page allowed during the discussion period. The revision takes into account every weakness and question raised in the three reviews. Its main changes are:

  • Wall-clock evaluation: every main loss plot now shows training loss against wall-clock time, with the average throughput of each run (Figs. 1–4), and Sec. 4.1 fully describes the hardware and network setup.
  • Parameter-matched SM: all main 2B comparisons now include SM-M, an SM variant matched to ResBM's trainable parameter count (Figs. 1–3, App. F).
  • 8B experiment under practical constraints: ResBM is matched stage by stage to the pipeline layout of a public 8B SM training run, and reaches a lower loss than SM (Sec. 4.3, Fig. 4, App. E).
  • Corrected method descriptions: the identity path (Secs. 1–3) and SM's limitation (Abstract, Secs. 1, 2 and 5) are now described precisely, and SWARM is defined as the ablation of ResBM's identity path (Sec. 4.1).
  • Key results up front: the Muon run that reaches the loss of the uncompressed AdamW baseline is now in the main text (Fig. 5), and the Introduction states the main quantitative results.

The former throughput section has been merged into Secs. 4.1–4.2 and the figures, so section and figure numbers have shifted; all references below are to the revised PDF. For each point, we first give our reply, then what changed in the paper (Revised PDF).

W1. Parameter-matched comparisons

We agree that this is critical, and we addressed it with new experiments:

  • SM: every 2B comparison in the main text (2, 4 and 8 stages; AdamW and Muon; 100× and 128×) now includes SM-M, an SM variant whose trainable parameter count is within 0.25% of ResBM's, obtained by adding transformer blocks (App. F). SM-M brings no improvement over SM despite using 13–14% more training FLOPs per token, and with 4 stages it is even worse; ResBM outperforms both (Figs. 1–3). SM's gap is therefore not explained by its smaller parameter count. A likely reason is that every block SM-M adds lies before the last boundary and is itself constrained to the subspace.
  • 8B: we matched ResBM stage by stage to the pipeline layout of the public 8B SM run of Agora (+0.3% trainable parameters, 9% fewer training FLOPs per token), and ResBM reaches a lower loss than SM (Sec. 4.3, Fig. 4, App. E).
  • SWARM has exactly ResBM's parameter count.
  • Uncompressed baseline: ResBM has 3.1% more parameters than the baseline, yet its loss stays above the baseline's, so this overhead does not favor ResBM.

Revised PDF: SM-M is defined in Sec. 4.1 and included in Figs. 1–3, with its per-stage allocations in App. F (Tables F.1–F.3); the 8B experiment is in Sec. 4.3 and App. E (Table E.1); ResBM's overhead is stated in Sec. 4.1 and App. D.2.

W2. SM limitation

We agree: the Agora implementation we compare against trains SM with standard optimizers, so we removed the constrained-optimization and Grassmann argument. The limitation we meant is the constraint itself. SM restricts the embedding and every block before the last pipeline boundary to the shared subspace, whether or not the block precedes a boundary, so only the blocks of the last stage operate at full rank. ResBM instead adds one encoder–decoder pair per boundary and leaves every transformer block unconstrained. Empirically, SM's loss gap keeps growing in our 2B and 3.5B experiments, whereas ResBM's stays approximately flat.

Revised PDF: the Abstract, Introduction, Related Work and Conclusion now describe this constraint instead of the optimization argument.

W3. Identity path and scaling with pipeline stages

  • What is selected: at each boundary, the fixed projection keeps hh coordinates of the identity branch, the width of the transmitted state. These coordinates carry no predefined meaning: training learns which directions to route through them (Sec. 3.3), so keeping the first hh loses no generality.
  • Scaling with the number of stages: the retained identity keeps rank hh however many boundaries are added (Eq. 11). We agree that compression error can still accumulate across boundaries, so we measured it: 1, 3 and 7 boundaries on the 2B model (Figs. 1 and 3, App. C.2), 15 boundaries on a 3.5B model (App. C.3), 6 boundaries on an 8B model with the pipeline layout of the public Agora-8B run (Fig. 4), and a 2048-token context on a second dataset (App. C.3). In our 2B and 3.5B experiments, ResBM's gap grows with the number of boundaries (about 0.07, 0.17 and 0.27 nats for 1, 3 and 7 boundaries) but not with training: after the initial phase, it stays approximately flat, whereas SM's grows. SWARM, which has no identity path, fails to train with 7 boundaries under AdamW (App. C.1).
  • In practice: our main experiments deliberately place a boundary after almost every block. In real deployments, boundaries are set by the memory of the participating nodes, and fewer boundaries bring ResBM much closer to the uncompressed baseline (Sec. 4.3). We test such a layout at scale by reproducing Agora-8B, a public 8B-parameter SM run trained over the internet, whose seven stages are sized to consumer and workstation GPUs: ResBM, matched to it stage by stage, reaches a lower loss than SM (Sec. 4.3, Fig. 4, App. E).
  • Limits: larger models and longer horizons remain to be tested, as stated in our Limitations; our scale, up to 8B parameters, is comparable to that of the works we build on and compare against.

Revised PDF: Secs. 3.2–3.3 were rewritten so that the construction is stated for any number of stages and blocks per stage; new experiments compare all methods with 2 and 4 stages (Fig. 3) and at 8B (Fig. 4); Sec. 4.3 discusses the number of boundaries in practice.

Q1. Hyperparameters

All methods share the same training configuration (Sec. 4.1), so differences between them do not come from per-method tuning. Because of compute constraints, we ran no large-scale sweeps: values follow common practice and prior work, including the 2B experiments of the SM paper, refined with short small-scale runs. The bottleneck's design choices (nonlinearity, encoder depth, intermediate width) were fixed a priori and not tuned. The only exception is the 8B experiment, where SM keeps the hyperparameters of the public Agora-8B run. (App. E).

Revised PDF: Sec. 4.1 states the shared configuration and the fixed design choices, the Limitations state that no large-scale sweeps were performed, and App. E lists the 8B settings.

Reviewer_CMDoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces residual encoder–decoder bottlenecks for low-bandwidth pipeline-parallel training, preserving an identity path on the communicated subspace. It compares training loss with SM and SWARM at extreme compression, evaluates AdamW and Muon with 2B and 3.5B models, and reports throughput and supplementary wall-clock experiments on eight-GPU configurations.

Critical strengths

S1. A critical contribution is that the bottleneck projection layers can be trained end-to-end as ordinary model parameters with off-the-shelf optimizers.

S2. I consider the results in Section 4.4 and Figure C.2 important: with Muon, ResBM reaches the uncompressed AdamW baseline's training loss at approximately 17B and 26B tokens for the two compression rates. This result should be more prominent. The comparison is against AdamW, but the paper explicitly retains a gap against uncompressed Muon.

Critical weaknesses

W1. The evaluation does not yet support the practical claims as written. The main comparisons emphasize extreme compression and training-loss curves, while the practical benefit at matched performance, throughput relative to competing compression methods, and scaling with distributed resources remain unclear. Appendix D provides wall-clock, bandwidth, and overhead results, but the paper needs to clarify what these imply for real deployments and how to choose the architecture for the target infrastructure. I also have questions about configuration, training horizon, and applicability.

W2. The presentation could improve by clarifying the contribution. The introduction spends substantial space on background before clearly stating the gap and motivation. The abstract's wording and description of parallelism also need more precision. The method's heavy notation makes it hard to distinguish the new contribution from existing components. Figures 1–4 are repetitive and awkwardly arranged. I suggest consolidating repeated results, using a table to clarify comparisons, identifying the baseline earlier, and adding quantitative takeaways in the introduction. The systems results and overhead analysis in Appendix D are central to understanding the paper and deserve prominence in the main text.

Questions for authors

Q1. How does the practical comparison change at matched performance, and how does ResBM's throughput compare with competing compression methods? This comparison is critical to understanding the contributions and impact of this work.

Q2. How should the architecture be configured for the available GPUs, nodes, and network topology? I acknowledge that Section 4.3 and Appendix D specify hardware and bandwidth for the reported experiments. What do these results imply for resource scaling and deployment in real scenarios?

Q3. What specific deployment problem determines the proposed architecture and its trade-offs? Please clarify: (a) the scope of the introduction's parallelism discussion, including tensor, expert, and context parallelism; (b) the evidence that pipeline parallelism is the primary bottleneck in the intended setting; and (c) what the reported scaling comparisons establish for that setting.

Q4. What motivates the fixed rectangular-identity projection? The truncation/zero-padding construction and its algebraic identity property are sufficiently described. I’m more concerned with why this choice fits the proposed bottleneck.

Q5. Why was SiLU selected for the encoder and decoder, and has its robustness been examined in relation to further communication compression?

Q6. Has the assumed concentration of forward activations and backward gradients motivating the narrowed identity path been checked?

Q7. What motivated the baseline architecture, dataset, context length, and optimization settings, including the Muon learning rate and momentum? The numerical choices are specified, but the rationale for selecting them is less clear.

Q8. Why were the reported training horizons chosen, and what supports the practical claims beyond them? In particular, how should we interpret the 3.5B-model run relative to a budget of about 20 tokens per parameter? Could later-training behavior or downstream in-context-learning results change the conclusions, and how does the trade-off change at lower compression?

Q9. What motivates placing bottlenecks after every block in the main configuration and after the fourth block in the single-bottleneck comparison? I believe the one-block-per-GPU setup defines the pipeline boundaries for the main experiment. The question is: how should we choose placement more generally?

3 comments

We thank the reviewer for the thorough review and the constructive suggestions on evaluation and presentation. We uploaded a revised PDF that uses the additional tenth page allowed during the discussion period. The revision takes into account every weakness and question raised in the three reviews. Its main changes are:

  • Wall-clock evaluation: every main loss plot now shows training loss against wall-clock time, with the average throughput of each run (Figs. 1–4), and Sec. 4.1 fully describes the hardware and network setup.
  • Parameter-matched SM: all main 2B comparisons now include SM-M, an SM variant matched to ResBM's trainable parameter count (Figs. 1–3, App. F).
  • 8B experiment under practical constraints: ResBM is matched stage by stage to the pipeline layout of a public 8B SM training run, and reaches a lower loss than SM (Sec. 4.3, Fig. 4, App. E).
  • Corrected method descriptions: the identity path (Secs. 1–3) and SM's limitation (Abstract, Secs. 1, 2 and 5) are now described precisely, and SWARM is defined as the ablation of ResBM's identity path (Sec. 4.1).
  • Key results up front: the Muon run that reaches the loss of the uncompressed AdamW baseline is now in the main text (Fig. 5), and the Introduction states the main quantitative results.

The former throughput section has been merged into Secs. 4.1–4.2 and the figures, so section and figure numbers have shifted; all references below are to the revised PDF. For each point, we first give our reply, then what changed in the paper (Revised PDF).

S2. Muon ResBM vs AdamW baseline result to be moved to main text

We thank the reviewer for highlighting this result.

Revised PDF: it is now in the main text (Sec. 4.3, Fig. 5) and summarized in the Introduction.

W1. Practical claims

We agree that wall-clock time is the key metric for low-bandwidth training. We have therefore updated every loss plot in Sec. 4 to show training loss against wall-clock time, with the average throughput of every run (Figs. 1–4). The compressed models run over an emulated 80 Mbps link with 26 ms latency, and the uncompressed baseline over a high-bandwidth10 Gbps link. Training is mostly bound by activation transfer: at 128× compression, ResBM matches the throughput of the uncompressed baseline, and at 100× it reaches about 75–85% of it with four or more stages (Sec. 4.2). In every setting, ResBM ends with the lowest loss of all compression methods. Beyond these controlled settings, we reproduce Agora-8B, a public 8B-parameter SM run trained over the internet, and match ResBM to it stage by stage under the same GPU-memory constraints: ResBM again reaches a lower loss than SM (Sec. 4.3, Fig. 4; details in Q2). We address the remaining points below: matched performance and throughput against other methods in Q1, configuration for the target infrastructure in Q2 and Q9, and training horizons in Q8.

Revised PDF:

  • Figs. 1–4 plot loss against wall-clock time, with throughput bars.
  • Sec. 4.1 describes the hardware (a single pipeline replica on H100 GPUs, one stage per GPU) and the network conditions, and gives ResBM's overhead (3.1% more parameters, 3.4% more per-token compute).
  • Sec. 4.2 explains the observed throughputs.
  • Sec. 4.3 adds the Agora-8B reproduction (Fig. 4, App. E).
  • The bandwidth sweep of App. D.1 was redone on H100 GPUs.

W2. Presentation

We followed these suggestions as far as the page limit allows:

  • Gap and motivation: the Introduction now states the gap early: SM constrains every block before the last pipeline boundary, and SWARM's bottlenecks break the residual identity path; ResBM avoids both.
  • Abstract: revised to state precisely what SM constrains and how ResBM relates to SWARM. We discuss the scope of the parallelism description in Q3.
  • Notation: the new parts of the method are Eqs. 7 and 10, in which the identity branch bypasses the learned encoder and decoder, and Eq. 11, which shows that a rank-hh identity survives any number of boundaries. Eqs. 2–5 only generalize the residual update to changing widths.
  • Figures: repeated figures were merged into one plot per setting showing all methods (Figs. 1–3).
  • Baselines and takeaways: all baselines (uncompressed, SWARM, SM and SM-M) are defined together in Sec. 4.1, and the Introduction now ends with the main quantitative results. We did not add a separate table, as it would repeat the figures, which now show loss, loss gap and throughput side by side.
  • Systems results: throughput is now part of every main comparison figure and is discussed in Sec. 4.2, and the overheads are stated in Sec. 4.1. App. D keeps the bandwidth sweep and the derivation of the overheads.

Revised PDF: Abstract, Introduction, Secs. 4.1–4.2 and Figs. 1–3, as listed above.

Q1. Matched performance and throughput

At a given compression rate, all compression methods send the same number of bytes per token, so their throughputs are close: ResBM, SM and SWARM are within about 2% of each other, while SM-M is 3–15% slower because of its additional blocks (Sec. 4.2). Comparing the methods against wall-clock time therefore gives the same picture as comparing them against tokens: ResBM's loss gap stays approximately flat while SM's grows, and ResBM ends with the lowest loss. At matched performance, ResBM trained with Muon at 100×/128x compression reaches the loss of the uncompressed AdamW baseline after about 17B/26B tokens (Fig. 5), whereas uncompressed training over the same 80 Mbps link is about 98× slower than ResBM (App. D.1).

Revised PDF: throughput bars in Figs. 1–4, the throughput discussion in Sec. 4.2, and Fig. 5 in the main text.

Q2. Configuration for the available infrastructure

We added an experiment that shows how these choices are made in practice (Sec. 4.3, Fig. 4, App. E). It reproduces the pipeline layout of Agora-8B, a public 8B-parameter SM run trained over the internet on heterogeneous GPUs.

  • Placement: in such deployments, GPU memory, not FLOPs, sets the number of stages and the blocks per stage. Agora-8B's body stages fit 24 GB consumer GPUs, while its tail, which holds the LM head, needs 48 GB or more. We matched ResBM to Agora-8B stage by stage under the same memory constraints, and ResBM reaches a lower loss than SM (Fig. 4).
  • Several blocks per stage: this layout gives ResBM 2 to 6 blocks per stage, which highlights a key property: ResBM needs a single boundary per stage, and every transformer block stays unconstrained. SM, by contrast, constrains every block before the last boundary, 26 of Agora-8B's 32 blocks.
  • Number of stages: ResBM's gap grows with the number of boundaries (about 0.07, 0.17 and 0.27 nats for 1, 3 and 7 boundaries; Figs. 1 and 3), so practitioners should use as few stages as memory allows.
  • Width: the smallest compression that hides communication behind compute. On our H100s, ResBM's throughput gain over uncompressed training grows from about 1.7× at 10 Gbps to about 13× at 1 Gbps and 98× at 80 Mbps (App. D.1), so communication limits uncompressed training even on fast links.

To scale out, many nodes can serve each stage, combined with low-communication data parallelism.

Revised PDF: new 8B experiment (Sec. 4.3, Fig. 4, App. E with Table E.1); the bandwidth sweep of App. D.1 was redone on H100 GPUs.

Q3. Scope and deployment problem

(a) Data, pipeline, tensor, context and expert parallelism are all standard parts of large-scale training over high-bandwidth interconnects. Tensor, context and expert parallelism communicate within every layer and therefore rely on the fastest interconnects, whereas data and pipeline parallelism communicate less often and are the natural candidates for low-bandwidth links. (b) In the low-bandwidth setting, data parallelism already has dedicated low-communication methods, such as DiLoCo and DeMo, which are complementary to ResBM. Pipeline parallelism, which is needed once a model no longer fits on a single GPU or node, remains the main bottleneck: activations and their gradients cross every stage boundary at every micro-batch, and at 80 Mbps, uncompressed training is about 98× slower than ResBM (App. D.1). This is the deployment problem our architecture targets. (c) Within this scope, our comparisons show that, under an equal communication budget, ResBM reaches the lowest loss across model sizes, numbers of stages and optimizers. Our reproduction of the public Agora-8B run (Q2) shows that this advantage holds on a real deployment layout, with pipeline stages sized to consumer and workstation GPUs.

Revised PDF: no change; the Introduction already restricts the discussion to data and pipeline parallelism.

Q4. Fixed rectangular-identity projection

A fixed truncation is the simplest map that gives the identity branch the transmitted width while remaining an exact identity on the retained coordinates. It has no parameters, costs nothing, and composes exactly across any number of boundaries . Any learned map on this branch should respect the identity mapping property as stated in Section 3.

Regarding which coordinates of the full rank are used within partial identit, keeping the first hh coordinates loses no generality, since training learns which directions to route through them.

Revised PDF: Sec. 3.3 now states explicitly that this identity holds across any number of stages.

Q5. SiLU

SiLU matches the nonlinearity of the backbone's feed-forward layers. It was fixed a priori, not tuned, and is shared with the SWARM baseline. Further compression of the transmitted state, such as quantizing the hh-dimensional activations, could be complementary to ResBM, but we have not tested it.

Revised PDF: Sec. 4.1 states that this choice was fixed a priori and not tuned.

Q6. Concentration of activations and gradients

We did not measure it directly. The motivation is the rank collapse of the output projections (App. B.1), also reported by SM. Concentration is not a strict precondition, because training adapts the representation to the fixed retained coordinates, and the encoder adds a learned summary of the remaining ones. The empirical evidence is indirect but consistent: with the narrowed identity path, ResBM trains with a stable gap, whereas SWARM, which has no identity path, fails to train with many boundaries (App. C.1).

Revised PDF: no change.

Q7. Experimental settings

Our 2B setting follows the 2B experiments of the SM paper (8 blocks, hidden width 4096) and its 80 Mbps low-bandwidth setting, so that SM is evaluated close to the conditions under which its authors reported little or no degradation. OLMo-2 is a recent, well-documented open architecture, and the AdamW settings follow common practice. For Muon, we use the standard Nesterov momentum of 0.95; its learning rate was set by short small-scale runs and is shared by all methods. FineWeb-Edu2 with a 2048-token context serves as a robustness check on a second dataset, and the 8B experiment uses the hyperparameters of the public Agora-8B run.

Revised PDF: Sec. 4.1 gives the full learning-rate schedule, Sec. 4.2 the Muon settings, and App. E the 8B settings.

Q8. Training horizons

Our main runs (from about 5B to more than 10B tokens) are short of compute-optimal, but long enough to show that SM's gap grows over training, which shorter horizons can hide. Our longest runs, 2B models trained with Muon for 26B tokens (about two thirds of 20 tokens per parameter), reach the loss of the uncompressed AdamW baseline after about 17B (100×) and 26B (128×) tokens (Fig. 5). The 3.5B run (9B tokens, under 3 tokens per parameter) is far from compute-optimal; it tests whether the gap dynamics hold at a larger size with 15 boundaries, and they do (App. C.3). Since ResBM's gap stays flat while SM's grows, longer training could widen the difference, but only longer runs can confirm this, and we cannot rule out that later training or downstream in-context-learning evaluations would change the picture. Each token is seen once, so the training loss estimates the held-out loss; downstream evaluation is future work. At lower compression, the gap shrinks: it is smaller at 100× than at 128× under both optimizers, and fewer boundaries reduce it substantially (Fig. 3).

Revised PDF: Fig. 5 is now in the main text, and the Limitations state the token budgets and that the gap dynamics require validation at larger model and token scales.

Q9. Bottleneck placement

One block per stage is a deliberate stress test that maximizes the number of boundaries for a given model. The single-bottleneck comparison is simply two equal stages, with the boundary after the fourth block. In general, placement follows GPU memory, as in our reproduction of the public Agora-8B run (Sec. 4.3, Q2), and the number of boundaries should be as small as memory allows (see Q2).

Revised PDF: Sec. 4.3 discusses practical placement, and Sec. 4.2 specifies the boundary positions for 2 and 4 stages.

Reviewer_LPfaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ResBM, an architecture designed for pipeline-parallel training over low-bandwidth links. At each stage boundary, a learned MLP encoder compresses the feed-forward output to a dimension h. The compressed output is added to the first h coordinates of the residual stream and transmitted to the next stage, where a corresponding decoder reconstructs it. The entire model is trained end-to-end using standard optimizers. Experiments on 2B- and 3.5B-parameter models using the C4 and FineWeb-Edu datasets show that ResBM achieves lower training loss than Subspace Models and SWARM at compression ratios of 100x and 128x, respectively, while recovering near-centralized training throughput over an 80 Mbps link.

Critical strengths
  1. Tackles an important and largely unsolved problem: compressing activations for pipeline parallelism.
  2. Simple design.
  3. Communication volume is fixed and predictable.
  4. Beats SM and SWARM consistently across optimizers, datasets, model sizes and bottleneck counts.
  5. Small parameter overhead.
Critical weaknesses

W1. The description of the identity path contradicts the proposed method:

The Introduction (lines 58-59) argues that compressing the residual identity path risks destabilizing optimization, while the Related Work section (lines 115–116) states that ResBM operates "alongside the residual stream rather than on it." However, in Eqs. (4) and (7), PidP^{id} truncates the residual stream to its first h coordinates at each stage boundary. With h = 32, this discards 4064 of the 4096 coordinates (99.2%) at every boundary. Thus, ResBM does compress the residual stream. What distinguishes it is the addition of a truncated identity term, rather than the preservation of the full residual path. The motivation and the description of the method should be corrected to match.

W2. Lack of ablation studies and insufficient justification for design choices:

The paper does not provide ablation studies to assess the contribution of key architectural components and design choices, including the identity term, the SiLU activation function, the selection of the first h coordinates of the residual stream, and the encoder depth and intermediate width m, which is fixed at m = H/4. Furthermore, the use of the same bottleneck width h at every stage boundary is not justified. Allowing h to vary with network depth or the bandwidth available between stages could potentially improve the trade-off between compression and model performance.

W3. No result reports seeds, standard deviations or confidence intervals

W4. Inaccurate terminology:

Section 4.3 refers to the 10 Gbps and 80 Mbps setups as "centralized" and "decentralized," respectively. However, both setups use the same GPUs, training schedule, and topology; only the communication bandwidth differs. The terms "centralized" and "decentralized" refers to how devices are connected or coordinated, rather than the speed of their communication links. Therefore, "high-bandwidth" and "low-bandwidth" would be more accurate and appropriate terms for distinguishing these two setups.

W5. incomplete references:

The citation for DiLoCo does not point to the corresponding original source.

W6. Presentation:

In figures 1 and 2 (panels a and b). Merging ResBM vs. SWARM and ResBM vs. SM into a single plot would make the comparison easier to read.

Questions for authors
  1. How many random seeds were used for each configuration?
  2. Can you report ResBM without the truncated-identity term, with everything else unchanged?
  3. Why does SWARM fail to train with AdamW but train with Muon? Was the learning rate tuned separately for each method?
  4. Could you justify these design choices: using SiLU as the activation function, retaining the first h coordinates, and setting the intermediate width to m = H/4?
  5. How was the 80 Mbps limit enforced?
  6. Can ResBM be integrated into a pretrained model? Can the bottleneck be removed or widened after training to recover model quality for inference over high-bandwidth networks?
  7. How should practitioners choose where to place bottlenecks and how wide each should be? Did you try a different h at different boundaries, or several blocks per stage?
4 comments

We thank the reviewer for the detailed review and the concrete suggestions. We uploaded a revised PDF that uses the additional tenth page allowed during the discussion period. The revision takes into account every weakness and question raised in the three reviews. Its main changes are:

  • Wall-clock evaluation: every main loss plot now shows training loss against wall-clock time, with the average throughput of each run (Figs. 1–4), and Sec. 4.1 fully describes the hardware and network setup.
  • Parameter-matched SM: all main 2B comparisons now include SM-M, an SM variant matched to ResBM's trainable parameter count (Figs. 1–3, App. F).
  • 8B experiment under practical constraints: ResBM is matched stage by stage to the pipeline layout of a public 8B SM training run, and reaches a lower loss than SM (Sec. 4.3, Fig. 4, App. E).
  • Corrected method descriptions: the identity path (Secs. 1–3) and SM's limitation (Abstract, Secs. 1, 2 and 5) are now described precisely, and SWARM is defined as the ablation of ResBM's identity path (Sec. 4.1).
  • Key results up front: the Muon run that reaches the loss of the uncompressed AdamW baseline is now in the main text (Fig. 5), and the Introduction states the main quantitative results.

The former throughput section has been merged into Secs. 4.1–4.2 and the figures, so section and figure numbers have shifted; all references below are to the revised PDF. For each point, we first give our reply, then what changed in the paper (Revised PDF).

W1. Identity path

We agree that two sentences were imprecise: ResBM does narrow the residual stream to hh coordinates at each boundary (Eq. 7). The distinction we meant is how it is narrowed. In ResBM, the learned encoder and decoder belong to the residual functions (the encoder follows the feed-forward layer, the decoder precedes attention), while the identity branch bypasses both and is only truncated by a fixed, parameter-free projection (Eqs. 7 and 10). An exact rank-hh identity therefore survives across every boundary, however many there are (Eq. 11). In SWARM-style bottlenecks, the full block output, residual stream included, passes through the learned encoder and decoder, so the identity path is broken at every boundary. Our SWARM baseline uses the same encoder, decoder and parameter count as ResBM and isolates exactly this difference.

Revised PDF:

  • Introduction: the motivating sentence now says that placing learned or lossy transformations on the identity path breaks the identity mapping.
  • Related Work: "alongside the residual stream rather than on it" is replaced by the precise description above.
  • Secs. 3.2–3.3: rewritten to place the bottleneck at every pipeline boundary, for any number of blocks per stage, and to state that the rank-hh identity holds across any number of stages.
  • Sec. 4.1: SWARM is formally specified and described as the ablation of ResBM's partial identity path, their only architectural difference.

W2. Ablations and design choices

  • Identity term: our SWARM baseline is this ablation. It has the same encoder, decoder and parameter count as ResBM and differs only in cutting the identity path. Without it, training fails under AdamW with 8 stages (App. C.1), lags far behind under Muon (Fig. 2), and shows a far larger gap than the other methods with 4 stages (Fig. 3).
  • First hh coordinates: this loses no generality. The network is symmetric under permutations of the hidden coordinates, so any fixed choice of hh coordinates gives an equivalent model, and training learns which directions to route through them (Sec. 3.3).
  • SiLU, encoder depth, m=H/4m = H/4: standard choices, fixed a priori and not tuned. They are shared with the SWARM baseline, so they do not favor ResBM in the ablation, and SiLU matches the SwiGLU nonlinearity of the backbone. A sweep requires substantial compute and is left to future work.
  • Uniform hh: pipeline throughput is set by the slowest link, so with equal links a uniform hh is the natural choice; by Eq. 11, the rank of the identity path is likewise set by the narrowest boundary. Our formulation allows a different width at each boundary (Sec. 3.1), which we agree is promising, for instance for heterogeneous links.

Revised PDF: Sec. 4.1 now states that SWARM is the identity-path ablation, and that the nonlinearity, encoder depth and mm were fixed a priori and not tuned, with sweeps and per-boundary widths left to future work.

W3. Seeds and variance

Because of compute constraints, each configuration was trained with a single seed, so we cannot report standard deviations. The comparisons are, however, controlled: all runs use fixed seeds, so within each model scale every method trains on the same token stream in the same order, and ResBM and SWARM start from the same initial weights as the uncompressed baseline. Two observations make our conclusions robust to seed noise:

  • Consistency across settings: ResBM ends with the lowest loss among compression methods in every configuration we ran: two optimizers, two compression rates, three model sizes (2B, 3.5B and 8B), two datasets and context lengths, and 2 to 16 pipeline stages. Seed noise is unlikely to favor the same method in all of them.
  • Trends, not endpoints: our main finding is a trend over billions of tokens. In our 2B and 3.5B runs, SM's gap to the uncompressed baseline grows over training while ResBM's stays approximately flat. Noise can shift an endpoint, but is unlikely to produce such a trend consistently.

Revised PDF: Sec. 4.1 states the fixed seeds, the shared token stream and initialization, and the single seed per configuration.

W4. Terminology

We agree: both setups use the same GPUs, schedule and pipeline configuration, and differ only in link bandwidth.

Revised PDF: the throughput section the reviewer refers to is merged into Sec. 4.1 and the main figures, where the two setups are called high-bandwidth (10 Gbps) and low-bandwidth (80 Mbps). "Decentralized training" now refers only to the motivating setting (Abstract, Introduction, Related Work, Conclusion).

W5. References

We thank the reviewer for catching this.

Revised PDF: DiLoCo now cites its peer-reviewed version (WANT@ICML 2024). We also updated Streaming DiLoCo (COLM 2025) and mHC (ICML 2026) to their published versions.

W6. Figures

We agree and merged them.

Revised PDF: Figs. 1–3 now show all methods in a single plot per setting: training loss against wall-clock time with throughput bars, next to the evolution of each method's loss gap to the uncompressed baseline. SWARM is omitted from Fig. 1 only because it does not train under AdamW with 8 stages, its loss staying near its initial value; its curves are in App. C.1.

Q1. Seeds

One fixed seed per configuration, shared across methods (see W3).

Q2. ResBM without the identity term

Removing the term with everything else unchanged would give xi+1=Ei(FFi(xi))x_{i+1} = E_i(\mathrm{FF}_i(x_i)): the block input xix_i would no longer cross the boundary, removing the residual connection there altogether. The meaningful variant without the identity path routes the full block output xi+FFi(xi)x_i + \mathrm{FF}_i(x_i) through the same encoder and decoder, which is exactly our SWARM baseline (see W2).

Revised PDF: SWARM's equations are now given in Sec. 4.1 and contrasted with ResBM's (Eqs. 7 and 10).

Q3. SWARM under AdamW and Muon

The learning rate was not tuned separately for any method: because of compute constraints, we ran no hyperparameter sweeps, and all methods share the learning rate and schedule of each optimizer. As discussed in Sec. 4.2, SWARM appears more prone to training instability, possibly because its bottlenecks interrupt the identity path. This sensitivity grows with the number of boundaries: under AdamW, SWARM is on par with SM with one boundary (Fig. 3), degrades as boundaries are added, and fails to train with seven (App. C.1). Muon appears more tolerant of this instability: SWARM trains under it, but stays far behind the other methods (Fig. 2). A per-method sweep might narrow SWARM's gap; we leave it to future work.

Revised PDF: Sec. 4.2 discusses this behavior, and the Limitations state that no large-scale hyperparameter sweeps were performed.

Q4. Design choices

Please see W2.

Q5. 80 Mbps limit

For the revision, we redid all throughput measurements on emulated links rather than live internet connections, because bandwidth and latency on a live connection vary over time and make comparisons between runs unreliable. Using Linux tc/netem, each inter-stage link of the compressed models is capped at 80 Mbps, matching the low-bandwidth setting of SM, with 26 ms of added latency. The uncompressed baseline instead communicates over a high-bandwidth 10 Gbps PCIe link.

Revised PDF: Sec. 4.1 now describes the hardware (a single pipeline replica on H100 GPUs, one stage per GPU, GPipe-style scheduling with PyTorch and NCCL) and these network conditions.

Q6. Pretrained models

Like SWARM and SM, this paper addresses pretraining. Inserting bottlenecks into a pretrained model, for instance to enable low-bandwidth post-training, and removing or widening them for inference over high-bandwidth networks are valuable directions that we are actively exploring, but they are beyond the scope of this paper.

Revised PDF: no change.

Q7. Bottleneck placement and width

We added an experiment that shows how these choices are made in practice (Sec. 4.3, Fig. 4, App. E). It reproduces the pipeline layout of Agora-8B, a public 8B-parameter SM run trained over the internet on heterogeneous GPUs.

  • Placement: in such deployments, GPU memory, not FLOPs, sets the number of stages and the blocks per stage. Agora-8B's body stages fit 24 GB consumer GPUs, while its tail, which holds the LM head, needs 48 GB or more. We matched ResBM to Agora-8B stage by stage under the same memory constraints, and ResBM reaches a lower loss than SM (Fig. 4).
  • Several blocks per stage: this layout gives ResBM 2 to 6 blocks per stage, which highlights a key property: ResBM needs a single boundary per stage, and every transformer block stays unconstrained. SM, by contrast, constrains every block before the last boundary, 26 of Agora-8B's 32 blocks. Our 2- and 4-stage experiments (Fig. 3) also use several blocks per stage.
  • Number of stages: ResBM's gap grows with the number of boundaries (about 0.07, 0.17 and 0.27 nats for 1, 3 and 7 boundaries; Figs. 1 and 3), so practitioners should use as few stages as memory allows.
  • Width: the smallest compression that hides communication behind compute. On our H100s, ResBM's throughput gain over uncompressed training grows from about 1.7× at 10 Gbps to about 13× at 1 Gbps and 98× at 80 Mbps (App. D.1), so communication limits uncompressed training even on fast links.

We did not try a different hh per boundary (see W2).

Revised PDF: new 8B experiment (Sec. 4.3, Fig. 4, App. E with Table E.1); Fig. 3 now compares all methods, including SM-M, with 2 and 4 stages; the bandwidth sweep of App. D.1 was redone on H100 GPUs.

Thank you for your detailed response and clarifications. My concerns have been partially addressed, and I have raised my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.