- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Residual Bottleneck Models (ResBM) for communication-efficient pipeline parallelism. It introduces learned encoder–decoder bottlenecks between transformer blocks while preserving a low-dimensional identity path through the residual connections. Experiments on 2B–3.5B parameter models show that ResBM achieves lower training loss than the evaluated alternatives at 100–128× activation compression.
Critical strengths
- The idea is simple, interesting and optimiser-agnostic.
- Results are reasonable, and the considered low-bandwidth settings are meaningful.
Critical weaknesses
- It is unclear if the comparisons are fair. Especially, ResBM adds more parameters due to the learnable encoder-decoder parameters and the comparisons are not done in a parameter-matched setting. Specifically, it has more parameters than the uncompressed baseline and, more importantly, SMs have fewer trainable parameters, as discussed in the cited Agora implementation, and it is not clear if the comparisons with SMs are in the parameter-matched setting. This is critical to correctly compare the methods.
- The statement about SMs requiring constrained optimisation and Grassmann updates as a limitation is narrow and unsubstantiated, as the SM implementation that was used for comparison does not have these components and just uses the standard optimization paradigm.
- The identity project section is unclear, as it seems it just selects k components from the residual path and not sure how this would scale with no of pipeline stages. Overall, this is a big problem in PP compression, as these kinds of lossy compression errors would accumulate over a larger no of compression boundaries and raise questions about scalability and experimental validation.
Questions for authors
How are the hyperparameters chosen for each method?
2 comments
W3. Identity path and scaling with pipeline stages
- What is selected: at each boundary, the fixed projection keeps coordinates of the identity branch, the width of the transmitted state. These coordinates carry no predefined meaning: training learns which directions to route through them (Sec. 3.3), so keeping the first loses no generality.
- Scaling with the number of stages: the retained identity keeps rank however many boundaries are added (Eq. 11). We agree that compression error can still accumulate across boundaries, so we measured it: 1, 3 and 7 boundaries on the 2B model (Figs. 1 and 3, App. C.2), 15 boundaries on a 3.5B model (App. C.3), 6 boundaries on an 8B model with the pipeline layout of the public Agora-8B run (Fig. 4), and a 2048-token context on a second dataset (App. C.3). In our 2B and 3.5B experiments, ResBM's gap grows with the number of boundaries (about 0.07, 0.17 and 0.27 nats for 1, 3 and 7 boundaries) but not with training: after the initial phase, it stays approximately flat, whereas SM's grows. SWARM, which has no identity path, fails to train with 7 boundaries under AdamW (App. C.1).
- In practice: our main experiments deliberately place a boundary after almost every block. In real deployments, boundaries are set by the memory of the participating nodes, and fewer boundaries bring ResBM much closer to the uncompressed baseline (Sec. 4.3). We test such a layout at scale by reproducing Agora-8B, a public 8B-parameter SM run trained over the internet, whose seven stages are sized to consumer and workstation GPUs: ResBM, matched to it stage by stage, reaches a lower loss than SM (Sec. 4.3, Fig. 4, App. E).
- Limits: larger models and longer horizons remain to be tested, as stated in our Limitations; our scale, up to 8B parameters, is comparable to that of the works we build on and compare against.
Revised PDF: Secs. 3.2–3.3 were rewritten so that the construction is stated for any number of stages and blocks per stage; new experiments compare all methods with 2 and 4 stages (Fig. 3) and at 8B (Fig. 4); Sec. 4.3 discusses the number of boundaries in practice.
Q1. Hyperparameters
All methods share the same training configuration (Sec. 4.1), so differences between them do not come from per-method tuning. Because of compute constraints, we ran no large-scale sweeps: values follow common practice and prior work, including the 2B experiments of the SM paper, refined with short small-scale runs. The bottleneck's design choices (nonlinearity, encoder depth, intermediate width) were fixed a priori and not tuned. The only exception is the 8B experiment, where SM keeps the hyperparameters of the public Agora-8B run. (App. E).
Revised PDF: Sec. 4.1 states the shared configuration and the fixed design choices, the Limitations state that no large-scale sweeps were performed, and App. E lists the 8B settings.
We thank the reviewer for the review, and in particular for raising the question of parameter-matched comparisons, which led to new experiments. We uploaded a revised PDF that uses the additional tenth page allowed during the discussion period. The revision takes into account every weakness and question raised in the three reviews. Its main changes are:
The former throughput section has been merged into Secs. 4.1–4.2 and the figures, so section and figure numbers have shifted; all references below are to the revised PDF. For each point, we first give our reply, then what changed in the paper (Revised PDF).
W1. Parameter-matched comparisons
We agree that this is critical, and we addressed it with new experiments:
Revised PDF: SM-M is defined in Sec. 4.1 and included in Figs. 1–3, with its per-stage allocations in App. F (Tables F.1–F.3); the 8B experiment is in Sec. 4.3 and App. E (Table E.1); ResBM's overhead is stated in Sec. 4.1 and App. D.2.
W2. SM limitation
We agree: the Agora implementation we compare against trains SM with standard optimizers, so we removed the constrained-optimization and Grassmann argument. The limitation we meant is the constraint itself. SM restricts the embedding and every block before the last pipeline boundary to the shared subspace, whether or not the block precedes a boundary, so only the blocks of the last stage operate at full rank. ResBM instead adds one encoder–decoder pair per boundary and leaves every transformer block unconstrained. Empirically, SM's loss gap keeps growing in our 2B and 3.5B experiments, whereas ResBM's stays approximately flat.
Revised PDF: the Abstract, Introduction, Related Work and Conclusion now describe this constraint instead of the optimization argument.