acceptodds
Under review as a conference paper at ICLR 2027

Learned subspace compression for communication efficient pipeline parallelism

Abstract

Pipeline parallelism enables training of large language models (LLMs) that exceed single-device memory but requires frequent communication of activations between pipeline stages. On bandwidth-constrained networks, this communication overhead can become the dominant training bottleneck in decentralized environments. The existing approach, Subspace Networks, mitigates this by constraining activations and model weights to a fixed, shared low-rank subspace across all layers. However, this global constraint restricts the model's representational capacity and leads to substantial performance degradation when trained on the same number of tokens. We propose Manifold Aware Projection Learning (MAPL) for communication efficient pipeline parallelism, a framework that formulates inter-stage activation compression as learnable orthogonal projections optimized directly on the Stiefel manifold. Rather than imposing a fixed global subspace, MAPL enables each pipeline stage to dynamically adapt its own compression subspace, coupled with factorized anchor embeddings that restore token-specific offsets with negligible bandwidth overhead. Across LLaMA models from 150M to 3B parameters, MAPL achieves 4 to 11 activation compression with minimal degradation in validation loss relative to uncompressed training. % On the 3B model, MAPL narrows the validation loss gap relative to uncompressed training from 18% (with Subspace Networks) down to just 1.6%. Finally, in a more challenging regime under emulated 1Gbps bandwidth links, MAPL scales to an 8B-parameter model at an extreme 102 activation compression (85× end-to-end) ratio while achieving faster convergence than uncompressed training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pipeline parallelism, Large language models, Low rank training, Activation compression

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_iRMkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAPL, which compresses forward activations and backward gradients through learned orthogonal projections at pipeline boundaries. Factorized token-conditioned anchors help compensate for compression-induced information loss, while the main model weights remain free of global low-rank constraints. The authors evaluate training quality and communication volume across multiple model scales and compare fixed-time training performance under an emulated bandwidth constraint.

Critical strengths
  1. It directly targets the communication bottleneck in low-bandwidth pipeline training, with a compression mechanism closely aligned with this objective.
  2. Applying low-dimensional constraints at communication boundaries while preserving the expressiveness of the main model weights offers a reasonable approach to balancing communication reduction and training quality.
  3. The authors account for additional costs and examine potential confounding factors. Tables 4–5 assess the capacity effects of the anchors, while Appendix E includes token IDs and projector synchronization in the communication accounting. This provides a more informative assessment than activation compression ratios alone.
Critical weaknesses
  1. Appendix F states that most quality experiments simulate pipeline stages within a single GPU and the central systems benefit has not been validated on a real bandwidth-constrained network. It therefore remains unclear how much benefit would persist under realistic networking and compute–communication overlap.
  2. In Table 7, the fixed orthogonal and SPEL-learned projectors achieve losses of 3.2800 and 3.2738, respectively, without anchors—a relatively small difference. A controlled comparison within the complete system, holding the anchors and other conditions fixed, is needed to justify learning the projector over simpler alternatives. Comparisons with competitive low-bit communication methods are also missing.
  3. The large-model results under aggressive compression do not establish long-term training quality. The 8B experiment runs for only six hours, during which MAPL and the uncompressed model complete 1209 and 796 steps, respectively. This demonstrates an early training benefit under the emulated bandwidth constraint, but does not establish that quality is preserved over longer training. The 3B training budget is also limited, and all experiments use a single random seed.
  4. Table 3 reports that the anchors add approximately 12M parameters per stage in the 150M setting. The existing capacity controls are useful, but comparisons matched for total parameter count and actual training cost are still needed to assess the overall benefit more accurately.
Questions for authors
  1. What end-to-end speedup does MAPL achieve on a real multi-node network with propagation latency and compute–communication overlap?
  2. Does the 8B configuration with r=40r=40 preserve quality under token-matched comparisons and longer training budgets? How much benefit does learning the projector provide when identical anchors are used?
4 comments
AuthorsOpenReview ↗

reply iRMk

Thank you for the thoughtful review. We have addressed all four weaknesses and three questions with additional experiments. The new results and corresponding discussions are added to the revised paper in blue color.

W1/Q1: The 8B experiment in our original paper already uses a real pipeline, with one stage per GPU and a 1F1B schedule. We have now added communication overlap to this setup.

At 8B, pp=4, and 1 Gbps, MAPL achieves a validation loss of 4.159 after 6 hours (1184 steps). The uncompressed baseline reaches 4.489 (844 steps), while SSN reaches 5.077. At 200 Mbps, MAPL achieves 4.149 (1199 steps), compared with 5.492 for the uncompressed baseline (362 steps). We also ran this setup again on a 200Mbps link and found that MAPL achieves validation loss of 4.1490 (at 1199 steps) while uncompressed achieve 5.4916 (at 362 steps).

Communication overlap increases the uncompressed baseline's steps by 6%. However, some communication remains on the critical path during pipeline fill and drain. In particular, each micro-batch transfer must still be completed before the next dependent computation can proceed. As a result, MAPL achieves 1.40× higher end-to-end training throughput than the uncompressed baseline at 1 Gbps, increasing to 3.3× at 200 Mbps.

To address multi-node part of Q1, we measured end-to-end speedup of MAPL pp=4 with multi node setup on two h200 nodes with stage 0 and 1 on single node with stages split across gpus and 2 and 3 on other node. We measured the speedup at different inter node latency for MAPL and uncompressed model and show the speedup. (batch 32 × 2048, micro-batch 4; median step time over steps )

bandwidthone-way latencyuncompressed s/stepMAPL r=40 s/stepspeedup
none03.213.221.00×
10 Gbps50 ms4.904.091.20×
1 Gbps011.533.273.53×
1 Gbps50 ms12.624.183.02×
1 Gbps200 ms15.927.472.13×
200 Mbps50 ms46.984.6210.17×

We can see the compression only adds a small compute cost. The speedup shrinks from 3.5× to 2.1× as latency goes 0 → 200 ms at 1 Gbps and is 1.2× at 10 Gbps. This shows in bandwidth constrained multinode setup, MAPL has upto 10x speedup over uncompressed.

W2 / Q2(2) We agree that a controlled comparison within the complete system is important. We therefore conducted an additional experiment at 150M parameters with pp=4, keeping the factorized anchors and all other conditions fixed. We compared three projector configurations: a fixed orthogonal projector, an unconstrained learned projector optimized with Muon, and a Stiefel-constrained learned projector optimized with SPEL. We ran three paired seeds, using the same initialization and data order for each comparison.

The results are shown below.

SeedUncompressedFixed projectorUnconstrained learned (Muon)Stiefel learned (SPEL)
423.24273.25683.27683.2530
433.24303.25743.27903.2560
443.24313.26293.27883.2605
Mean3.24293.25903.27823.2565

We argue that simply learning the projector is not sufficient. The unconstrained Muon projector consistently performs worse than the fixed projector across all three seeds. Its gap to the uncompressed baseline is approximately 2.2 times larger. In contrast, Stiefel-constrained learning consistently improves over the fixed projector, reducing the validation loss by 0.0025 on average. This shows that the benefit does not come from learning the projection alone. Preserving its orthogonality during optimization is important.

Second, the absolute improvement needs to be considered relative to the remaining performance headroom. With factorized anchors, the fixed-projector model has a validation-loss gap of 0.0161 relative to the uncompressed baseline. Stiefel learning closes 0.0025 of this gap, recovering approximately 16% of the remaining loss gap.

This is consistent with our results without anchors in Appendix Table 7 (Figure 8 in revision). There, the available gap is larger (0.0373), and stiefel learning closes 0.0062, or approximately 16.6% of the gap. Thus, Stiefel learning recovers a comparable fraction of the remaining performance gap in both settings. The smaller absolute improvement with anchors is expected because the anchors have already recovered much of the compression-induced performance loss, leaving less headroom for projector learning.

Removing the unnecessary global low-rank constraint on model weights already recovers much of the lost performance (as stated in line 89 of our paper), while factorized anchors further compensate for information loss at pipeline boundaries. However, learning the projection without respecting its orthogonal structure can be detrimental. MAPL addresses this by learning stage-specific projections directly on the Stiefel manifold.

**W2 (2) ** 150M, pp=4 (uncompressed 3.2427):

ratiomethodvalΔ
4×INT43.3228+2.47%
4×MAPL r=2563.2530+0.32%
4×SSN k=256 (Muon)3.5546+9.62%
8×INT24.6163+42.36%
8×MAPL r=1283.2859+1.33%
8×MAPL r=256 + INT8 on the wire3.2545+0.36%

We agree that comparisons with low-bit communication methods are important. In our original submission, we did not include these comparisons because quantization is orthogonal to MAPL and can be applied directly to its compressed representations. Nevertheless, following the reviewer's suggestion, we conducted additional experiments on the 150M model with PP=4, using the same training setup and evaluation protocol.

The results show that MAPL substantially outperforms standalone low-bit quantization at matched communication ratios: At 4× compression, INT4 increases validation loss by 2.47%, whereas MAPL (r=256) increases it by only 0.32%. At 8× compression, INT2 leads to a substantial 42.36% increase in validation loss. In contrast, MAPL (r=128) limits the increase to 1.33%. These results demonstrate that MAPL preserves substantially more training quality than aggressive low-bit quantization at comparable communication budgets. Importantly, the two approaches are complementary. To demonstrate this, we also combined MAPL (r=256) with INT8 quantization on the wire. This achieves 8× communication compression while increasing validation loss by only 0.36%, compared with 1.33% for MAPL (r=128) alone and 42.36% for standalone INT2. Overall, these experiments show that MAPL is not only competitive with low-bit communication methods but can also be combined with them to achieve higher compression while maintaining training quality. We have included these comparisons in the revised manuscript.

W3 / Q2:

We extended the 8B experiment from 6 to 12 hours under 1 Gbps link. MAPL maintains lower validation loss throughout the extended run. At 12 hours, MAPL achieves 3.7343 compared with 3.8336 for the uncompressed model, a 2.6% reduction in validation loss, while completing 1.40× more optimizer steps (2367 vs. 1693). Although the gap narrows as training progresses, the advantage persists through the end of the 12-hour budget. This provides evidence that the improvement is not merely a short-lived early-training effect.

We additionally performed a controlled comparison of MAPL and the uncompressed model using the same initialization, data order, and learning-rate schedule, without the constrained link. Both models were trained for 1700 optimizer steps. MAPL achieves a final validation loss of 3.8768 compared with 3.7854 for the uncompressed model, corresponding to a 2.4% relative increase. This quantifies the quality trade-off at equal training progress. In contrast, under a fixed wall-clock budget with the bandwidth constraint, MAPL achieves better validation loss because it enables substantially more training steps. We believe reporting both comparisons is important: token-matched evaluation isolates the quality cost of compression, whereas wall-clock-matched evaluation measures its practical benefit in bandwidth-constrained training. This type of comparison is used in SSN (Neurips 2025) as well.

For the random seed, We also repeated the 8B, 1 Gbps, 6-hour experiment with two seeds (42 and 43). MAPL obtains validation losses of 4.1585 and 4.1565, while the uncompressed model obtains 4.4886 and 4.4840, respectively. The consistent improvement across both seeds suggests that the observed advantage is not specific to a single initialization.

Regarding the 3B experiment, we agree that our training budget does not establish Chinchilla-optimal training. As already acknowledged in the paper, the 3B with 20B token experiments use approximately one-third of the Chinchilla compute-optimal budget. However, this is still a substantial training experiment by academic standards, involving a total compute budget of (3.6×10203.6\times10^{20}) FLOPs. We view this as a limitation on claims about fully converged model quality, rather than evidence that the reported results are based on an insufficiently small experiment.

W4:

We thank the reviewer for this suggestion. The anchors are factorized lookup tables rather than additional transformer layers. They introduce negligible additional matrix-multiplication cost and do not add FLOP-heavy computations to the forward or backward passes. As discussed in the paper, their relative parameter overhead also decreases substantially with model scale. Furthermore, Appendix A.1 shows that the benefit of anchors is not simply a generic increase in model capacity.

Nevertheless, we agree that parameter- and compute-matched comparisons provide a more complete assessment, particularly at smaller scales. We therefore conducted additional controlled experiments at 150M and 500M parameters.

150M, pp=4, 4× compression (baseline: 3.2427).

ConfigurationParameter changeValidation loss
Uncompressed—3.2427
Uncompressed + factorized anchors+24%3.2266
Uncompressed + widened FFN (2688 → 4544)+24%3.1988
MAPL, r=256+24%3.2530
MAPL, anchor replaces token embedding+0.37%3.2531

Adding the same factorized anchors to the uncompressed model improves validation loss by only 0.50%. In contrast, widening the FFN with a comparable parameter increase achieves a 1.35% improvement. This shows that adding factorized anchors is not equivalent to increasing transformer capacity through additional FLOP-bearing parameters. MAPL achieves 3.2530 under the original anchor configuration.

We additionally evaluate sharing the stage-0 anchor with the token embedding, eliminating the separate vocabulary embedding table. This reduces MAPL's parameter overhead from 24% to just 0.37%, while achieving a validation loss of 3.2531.

500M, pp=4, 6× compression (baseline: 2.8522).

ConfigurationParameter changeValidation loss
Uncompressed—2.8522
Uncompressed + factorized anchors+7.9%2.8367
MAPL, r=256+7.9%2.8737
MAPL, anchor replaces token embedding−3.7%2.8737

We observe the same trend at 500M: adding factorized anchors to the uncompressed model improves validation loss by only 0.54%, while MAPL achieves 2.8737 under 6× activation compression. Sharing the anchor with the token embedding reduces the total parameter count to 3.7% below the uncompressed baseline without changing MAPL's validation loss. These results show that the anchor parameter count alone does not capture its computational or capacity contribution. The anchors provide a lightweight mechanism for compensating for information lost during activation compression, rather than acting as a conventional increase in transformer capacity.

Dear Reviewer iRMk,

We thank you for your thoughtful feedback. A gentle reminder that the author-reviewer discussion period is nearing its end. We would greatly appreciate any further feedback or responses to our clarifications before the discussion period closes.

Thank you again for your time and consideration.

Reviewer_CCAAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies pipeline parallelism for training large language models and proposes Manifold-Aware Projection Learning (MAPL). MAPL a framework that formulates inter-stage activation compression as learnable orthogonal projections and compresses pipeline-parallel boundary activations with a learned orthonormal projector. MAPL is evaluated on LLaMA-style models from 150M to 3B parameters, with a further 8B experiment under an emulated 1 Gbps inter-stage link.

Critical strengths
  1. The paper explores an important problem of activation traffic in decentralized pipeline-parallel training. Compressing these transfers while preserving training quality is quite useful.
  2. The idea is interesting to learn a separate orthogonal projector for each boundary while avoiding SSN’s restrictive weight-subspace constraint.
  3. Experimental evaluation show substantive improvement over SSN and appendix provide useful ablations of the anchor design, projector optimization, and communication costs.
Critical weaknesses
  1. One key concern is that factorized anchors add approximately 12M trainable parameters per stage, which is substantial for a 150M model. Adding anchors to an uncompressed model also improves validation loss. Parameter-matched controls is important.
  2. Only SSN is the main compressed baseline. How does MAPL compare with activation quantization and a fixed projector equipped with the same anchor mechanism?
  3. I am also concerned about how SSN’s hyperparameters were selected. SSN was run with hyperparameters tuned for uncompressed training. Can you reproduce a result from the SSN paper under its original setup? Was any learning rate tuned for SSN?
  4. How is learned Stiefel projector’s benefit comapred without the fixed projector?
Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

reply to CCAA

Thank you for the thoughtful review. We appreciate your positive assessment of the problem's importance, the approach, and the experimental evaluation. We have addressed each of your four main concerns with additional experiments and revisions to the paper (in blue). Our responses are organized below.

W1 The factorized anchors are lookup tables rather than additional transformer layers. Although they introduce additional trainable parameters, they do not introduce additional dense matrix multiplications comparable to widening the transformer. Their relative parameter overhead also decreases substantially with model scale (as mentioned in the paper line 848). Nevertheless, we agree that parameter-matched comparisons are important, particularly at smaller scales.

We therefore conducted additional controlled experiments at 150M and 500M parameters to distinguish the contribution of factorized anchors from a conventional increase in transformer capacity.

150M, pp=4, 4× compression (baseline: 3.2427).

ConfigurationParameter changeValidation loss
Uncompressed—3.2427
Uncompressed + factorized anchors+24%3.2266
Uncompressed + widened FFN (2688 → 4544)+24%3.1988
MAPL, r=256+24%3.2530
MAPL, anchor replaces token embedding+0.37%3.2531

Adding factorized anchors to the uncompressed model improves validation loss by only 0.50%. In contrast, widening the FFN with the same parameter increase achieves a 1.35% improvement. This comparison shows that the benefit of factorized anchors is not equivalent to simply increasing conventional transformer capacity through additional FLOP-bearing parameters.

We also evaluate sharing the stage-0 anchor with the token embedding, eliminating the need for a separate vocabulary embedding table. This reduces MAPL's parameter overhead from 24% to just 0.37%, with minimal change in validation loss (3.2531).

500M, pp=4, 6× compression (baseline: 2.8522).

ConfigurationParameter changeValidation lossΔ loss vs. baseline
Uncompressed—2.8522—
Uncompressed + factorized anchors+7.9%2.8367−0.54%
MAPL, r=256+7.9%2.8737+0.75%
MAPL, anchor replaces token embedding−3.7%2.8737+0.75%

We observe the same trend at 500M. Adding factorized anchors to the uncompressed model improves validation loss by only 0.54%, whereas MAPL achieves 2.8737 under 6× activation compression. Importantly, sharing the anchor with the token embedding reduces the total parameter count to 3.7% below the uncompressed baseline without changing MAPL's validation loss.

Together, these experiments provide two observations. First, the improvement from adding factorized anchors to an uncompressed model is substantially smaller than that obtained by increasing the capacity of the transformer through FFN widening at a matched parameter budget. Second, sharing the anchor with the token embedding preserves MAPL's performance while largely eliminating its additional parameter overhead at small scales. These results support the role of anchors as a mechanism for compensating for information lost during activation compression, rather than simply increasing conventional transformer capacity.

W2 Our primary comparison in the original submission was with SSN because it is the closest existing approach to MAPL: both methods compress pipeline-boundary activations through low-rank projections. This makes SSN a particularly relevant baseline for evaluating the benefits of our approach. In contrast, activation quantization uses a different compression mechanism and can be combined with ours. We agree that evaluating these alternatives provides a more complete assessment of MAPL. We therefore conducted additional experiments on the 150M model with PP=4.

We compare MAPL against standalone low-bit activation quantization at matched communication ratios. All quantization experiments use quantization with stochastic rounding, applied to both forward activations and backward gradients.

150M, pp=4 (uncompressed validation loss: 3.2427).

RatioMethodValidation lossΔ
4×INT43.3228+2.47%
4×MAPL, r=2563.2530+0.32%
8×INT24.6163+42.36%
8×MAPL, r=1283.2859+1.33%
8×MAPL, r=256 + INT83.2545+0.36%

At 4× compression, INT4 increases validation loss by 2.47%, whereas MAPL increases it by only 0.32%. At 8× compression, standalone INT2 results in a 42.36% increase, compared with only 1.33% for MAPL (r=128). Importantly, the two approaches are complementary. Applying INT8 quantization to MAPL's compressed representations achieves 8× communication compression with only a 0.36% increase in validation loss. This demonstrates that MAPL can be combined with low-bit quantization to achieve higher compression while maintaining training quality.

W3 We thank the reviewer for raising this concern. We implemented SSN using the authors' publicly available codebase and retained its method-specific settings. The remaining optimizer and training settings were kept consistent with the other methods.

To further investigate whether SSN was sensitive to hyperparameter selection, we conducted additional learning-rate sweeps at 150M, PP=4, and rank (r=256). The uncompressed baseline has a validation loss of 3.2427.

SSN configurationLearning rate (Muon)Learning rate (AdamW)Validation lossChange
Muon0.010.0013.5889+10.7%
Muon (paper setting)0.020.0023.5546+9.6%
Muon0.040.0043.5329+8.9%
Muon, Adam-specific tuning (paper setting)0.020.0023.5546+9.6%
Muon, Adam-specific tuning0.020.0103.5324+8.9%
Muon, Adam-specific tuning0.020.0203.5433+9.3%
AdamW0.0013.6447+12.4%
AdamW (paper setting)0.0023.6403+12.3%
AdamW0.0043.7598+15.9%

These sweeps show that the validation loss varies slightly, the best result remains 8.9% worse than the uncompressed baseline. This suggests that SSN's performance gap is not primarily due to an unsuitable learning rate. For context, MAPL achieves only a 0.32% validation-loss increase using the same optimizer and learning rate.

We also note that the original SSN paper uses a different experimental setup, including the dataset, model sizes, and training token budget. Thus, an exact numerical reproduction of its reported results is not directly comparable to our setting. Our experiments instead evaluate SSN using its public implementation under the same training setup.

Finally, our ablations help explain the performance gap:

ConfigurationWeightsCommunicationValidation lossChange
BaselineFull-rankUncompressed3.2427—
SSN weight constraint onlyRank-256Uncompressed3.3842+4.36%
SSN communication onlyFull-rank(k=256)3.3004+1.78%
SSN (both)Rank-256(k=256)3.5546+9.62%
MAPLFull-rank(r=256)3.2530+0.32%

These results indicate that the rank-constrained model weights account for a substantial part of SSN's performance degradation (we mention this clearly in the introduction line 089) . In contrast, MAPL keeps the model weights unconstrained and learns separate projections for communication, avoiding this limitation.

W4 & W2(2) To isolate the contribution of projector learning, we compare a fixed random orthogonal projector against unconstrained learned projectors (Muon) and Stiefel-manifold learning (SPEL), keeping the factorized anchors and all other settings identical. We report final validation loss across three seeds.

150M, pp=4, 4× compression (r=256).

SeedUncompressedFixed projectorUnconstrained learned (Muon)Stiefel learned (SPEL)
423.24273.25683.27683.2530
433.24303.25743.27903.2560
443.24313.26293.27883.2605
Mean3.24293.25903.27823.2565

Stiefel-manifold learning consistently improves over the fixed projector across all three seeds, reducing mean validation loss by 0.0025. In contrast, unconstrained projector learning performs worse than keeping the projector fixed. Its validation-loss gap relative to the uncompressed baseline is approximately 2.2× larger than that of the fixed-projector setting. This comparison shows that simply making the projector learnable is insufficient. Learning it under the Stiefel constraint is important for obtaining the benefit of projector adaptation.

We also put the magnitude of this improvement into context. With identical anchors, the fixed-projector model has a validation-loss gap of 0.0161 relative to the uncompressed baseline. Stiefel learning closes 0.0025 of this gap, approximately 16% of the available headroom.

This is consistent with our results without anchors in Figure 8 (Appendix Table 7 in the original submission), where the fixed-projector gap is 0.0373 and Stiefel learning closes 0.0062, approximately 16.6% of the available gap. Thus, the smaller absolute improvement with anchors reflects the smaller remaining performance gap, rather than a weaker relative benefit.

Dear Reviewer CCAA,

We thank you for your thoughtful feedback. A gentle reminder that the author-reviewer discussion period is nearing its end. We would greatly appreciate any further feedback or responses to our clarifications before the discussion period closes.

Thank you again for your time and consideration.

Reviewer_L4VpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAPL, a communication-efficient method for pipeline-parallel training of LLMs. MAPL uses learnable stage-specific low-rank projections to compress activations and gradients between pipeline stages without constraining the model weights. The method combines Stiefel manifold optimization with factorized anchor embeddings to improve compression quality. Experiments on models ranging from 150M to 8B parameters show that MAPL substantially reduces communication overhead while maintaining training performance close to uncompressed training and outperforming SSN. The results also show improved training efficiency under emulated bandwidth-constrained settings.

Critical strengths
  • The paper addresses an important practical challenge in pipeline-parallel training: the communication bottleneck under limited network bandwidth.
  • The proposed method avoids imposing low-rank constraints on the model weights, which helps preserve model performance while reducing communication cost.
  • The experiments show consistent improvements over SSN, with relatively small validation loss degradation compared to uncompressed training.
Critical weaknesses

[W1] I think that the baselines are insufficient. For example, AQ-SGD also focuses on reducing communication cost through activation compression. Although it was originally developed for a different training setting, the authors could consider adapting it to the setting of this paper for a fair comparison. Such a comparison would help better show the effectiveness of your MAPL over existing activation compression baselines. Additionally, I would like the authors to clarify why they only compare against SSN and do not consider other relevant baselines.

[W2] The motivation for choosing SPEL as the optimizer for the learnable projectors is not sufficiently clear. While the authors explain the importance of maintaining orthogonality and show that SPEL outperforms unconstrained Muon updates, these results do not establish why SPEL is preferred over other orthogonality-preserving optimization methods. I would encourage the authors to further justify this choice.

[W3] The role of the stage-specific anchor embeddings in the reconstruction process is not sufficiently clear. In Equations (3) and (4), the sender subtracts its own anchor, while the receiver adds a separately learned anchor from the next stage. Since these anchors are not necessarily identical, the reconstruction introduces an additional discrepancy beyond the low-rank projection error. It would be helpful for the authors to explain the motivation for this asymmetric design and whether the two anchors need to remain aligned during training.

Questions for authors

See weaknesses above.

4 comments
AuthorsOpenReview ↗

reply to L4Vp

Thank you for your detailed feedback. We have addressed all three weaknesses through additional experiments and analysis. We have also incorporated the corresponding results and clarifications into the revised manuscript (written in blue).

W1. Why only SSN. To our knowledge, SSN is the only prior method that performs low-rank activation compression for pipeline-parallel from-scratch pretraining. Other activation-compression methods primarily use quantization, which we initially treated as orthogonal to MAPL, or target fine-tuning. We agree that the original comparison was too narrow and have expanded our evaluation to include both low-bit quantization and AQ-SGD. We first compare direct quantization using full training runs, then evaluate AQ-SGD separately because its delta coding requires replayed samples.

Low-bit quantization We evaluate quantization on both forward activations and backward gradients. 150M, pp=4 (uncompressed: 3.2427)

RatioMethodValidation lossΔ vs. uncompressed
4×INT43.3228+2.47%
4×MAPL, r=2563.2530+0.32%
4×SSN, k=256 (Muon)3.5546+9.62%
8×INT24.6163+42.36%
8×MAPL, r=1283.2859+1.33%
8×MAPL, r=256 + INT83.2545+0.36%

Additional configurations: direct INT4 versus MAPL at 4×

SettingUncompressedDirect INT4MAPLSSN
150M, pp=83.24273.4975 (+7.86%)r=256: 3.2792 (+1.13%)3.5742 (+10.22%)
500M, pp=42.85222.9467 (+3.31%)r=384: 2.8676 (+0.54%)-

At 4×, INT4 incurs a 2.47% loss increase at 150M, which grows to 7.86% with a deeper pipeline and 3.31% at 500M. MAPL incurs only 0.32–1.13% at the same compression ratio. INT2 results in substantial degradation. Moreover, applying INT8 on top of MAPL achieves 8× compression with only a 0.36% loss increase, demonstrating that the two approaches are complementary. (Appendix B)

W1(b) AQ-SGD: evaluation with replayed samples.

AQ-SGD transmits, the quantized difference between the current activation and its previously reconstructed value. This requires samples to be revisited and a buffer storing one reconstructed activation per sample per pipeline boundary. For our 3B-token, 150M run, this would require approximately 18 TB of storage. The original method targets fine-tuning on small datasets over multiple epochs.

To evaluate AQ-SGD's delta coding in the regime where it applies, we implemented it using content-hashed sample keys, a host-RAM reconstruction buffer, and direct quantization of the backward wire as in the original method. We trained a 150M, pp=4 model for five epochs over a 96M-token subset (458 steps), with delta coding active from epoch 2. Validation uses the standard holdout. Since the data budget and repetition differ from Table 1, absolute losses are not comparable to the full-training results; comparisons within this experiment are meaningful.

RatioMethodValidation lossΔ vs. uncompressed
1×Uncompressed3.9229–
2×Direct INT83.9235+0.02%
2×AQ-SGD INT83.9232+0.01%
4×Direct INT44.1707+6.32%
4×AQ-SGD INT44.1027+4.58%
4×MAPL, r=2563.9279+0.13%
8×Direct INT25.0507+28.75%
8×AQ-SGD INT25.0613+29.02%
16×MAPL + direct INT44.0446+3.10%
16×MAPL + AQ-SGD INT44.0117+2.26%

AQ-SGD's delta coding recovers 0.068 of INT4's 0.248 loss gap, but does not improve INT8 or INT2. In contrast, MAPL at the same 4× ratio remains within 0.005 of the uncompressed loss. The methods also compose: applying AQ-SGD to MAPL's compressed representation achieves 16× compression with lower loss than AQ-SGD alone at 4×.

These results highlight the complementary roles of the two methods.

W2. We do not claim that SPEL is the only, or theoretically superior, method for preserving orthogonality. Alternatives such as QR-, SVD-, and Cayley-based retractions can also enforce the Stiefel constraint. We chose SPEL primarily for computational practicality. In particular, SPEL provides a first-order Riemannian update with an inexpensive polar retraction based on a few matrix multiplications. This avoids the QR decompositions, SVDs, or large linear-system solves required by alternative approaches. These operations are particularly undesirable in our setting, where projector updates must be efficient and numerically consistent across pipeline-parallel ranks. SPEL also builds on the Muon update, allowing us to retain the same optimizer framework and hyperparameters.

We have added a sentence in the paper that clarifies this (Section 3.3). saying that our choice of SPEL is motivated by its practical efficiency rather than a claim that it is universally superior to other Stiefel optimization methods. Our goal is to obtain an effective orthogonality-preserving update without introducing substantial additional computational overhead.

W3. The two anchors are intentionally not tied or aligned. Their purpose is to remove token-specific information before compression, allowing the limited-rank communication channel to focus on the context-dependent part of the activation.

Specifically, each stage pp learns its own token-conditioned anchor table Φp=EpsmallPp\Phi_p = E_p^{\text{small}} P_p, with rank r=256r = 256. At a pipeline boundary, the sender subtracts its anchor Φi[t]\Phi_i[t], and the receiver adds its own anchor Φi+1[t]\Phi_{i+1}[t]. Since both stages already have access to the token IDs, there is no need to transmit token-specific information through the compressed activation. Instead, the rank-rr communication channel carries the remaining context-dependent information. The asymmetry also provides a learned, token-conditioned correction Φi+1[t]−Φi[t]\Phi_{i+1}[t] - \Phi_i[t] to the residual stream. Thus, the two anchors serve different roles at the sender and receiver, and forcing them to be identical would remove this flexibility.

We measured the alignment between adjacent learned anchor tables at the end of training. The following results are from 150M models with pp=4 and pp=8.

BoundaryPer-token cosineFrequency-weighted cosineRelative anchor difference (norm)
0 → 1−0.129−0.0921.54
1 → 2+0.061+0.0511.51
2 → 3+0.071+0.0701.52
BoundaryPer-token cosineFrequency-weighted cosineRelative anchor difference (norm)
0 → 1−0.119−0.0891.52
1 → 2+0.058+0.0491.56
2 → 3+0.063+0.0451.47
3 → 4+0.048+0.0371.44
4 → 5+0.037+0.0421.44
5 → 6+0.059+0.0651.42
6 → 7+0.062+0.0741.34

The adjacent anchors have very low cosine similarity, indicating that they do not converge towards alignment during training. Their relative differences are also consistent with independently learned representations of similar magnitude. Each stage learns its own token-conditioned correction rather than reproducing the previous stage's anchor.

W3(b)

We also compare the asymmetric design against a symmetric alternative that subtracts and re-adds the same frozen anchor table.

Anchor designValidation lossΔ vs. uncompressedCompression gap recovered
Symmetric: same frozen anchor at both stages3.3310+2.72%1.7%
Asymmetric: learned factorized anchor pair (MAPL)3.2792+1.13%59.6%
Asymmetric: full trainable anchor tables3.2491+0.20%93.1%

The symmetric design provides almost no improvement, whereas the asymmetric learned pair recovers 59.6% of the compression-induced loss gap. This supports our motivation for using separate stage-specific anchors: they learn complementary token-conditioned corrections that help compensate for information lost during projection.

We have clarified this motivation and the role of the asymmetric reconstruction in the revised paper. (Section 3.4)

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. The new baseline comparisons and clarifications regarding SPEL and the asymmetric anchor design have satisfactorily addressed most of my concerns. I am therefore willing to raise my score from 2 to 3.

Thank you for taking the time to review our responses and updated experiments. We are glad that our clarifications and additional results have addressed your concerns, and we truly appreciate your positive feedback and recognition of our work.

Reviewer_oN3uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MAPL, which compresses communication between pipeline stages using learned orthogonal projections and factorized token anchors, without constraining transformer weights. Experiments on 150M–3B models report 4×–11× activation compression with relatively small losses in model quality. An 8B experiment also achieves lower validation loss within a fixed training time under emulated 1 Gbps links.

Critical strengths

MAPL consistently outperforms SSN across model sizes and pipeline configurations, with downstream evaluations supporting the results. Anchor ablations and explicit communication accounting further strengthen the evaluation.

Critical weaknesses
  1. The benefit of learning the projectors is not fully isolated. Table 7 shows a small improvement over fixed projection, from 3.2800 to 3.2738, but excludes anchors and uses a single seed. A comparison with fixed projectors using identical learned anchors is missing, leaving uncertainty about the contribution of the central mechanism.
  2. The systems evaluation has a restricted scope. The 8B experiment uses actual pipeline placement, but bandwidth is emulated through delays and computation–communication overlap is disabled. The reported advantage therefore needs further validation against a pipeline implementation that allows overlap.
Questions for authors
  1. Does MAPL outperform fixed projectors with identical anchors across multiple seeds?
  2. Does the time-to-loss improvement persist when communication overlap is enabled?
2 comments
AuthorsOpenReview ↗

reply to oN3u

We thank reviewer oN3u for the careful reading of our paper. The concerns raised asked for additional ablations, which we have now conducted and we believe this has made our paper stronger. We provide the results and clarifications below.

Q1/W1. We conducted the requested ablation on the 150M model with pipeline parallelism of 4, 4× compression (r = 256), and identical factorized anchors. We report final validation loss across three seeds.

SeedUncompressedFixed projectorUnconstrained learned (Muon)Stiefel learned (SPEL)
423.24273.25683.27683.2530
433.24303.25743.27903.2560
443.24313.26293.27883.2605
Mean3.24293.25903.27823.2565

From this table, we can see that stiefel learned projectors consistently improves over the fixed projector across all three seeds (paired differences: +0.0038, +0.0014, +0.0024; mean 0.0025). In contrast, learning the projector without the Stiefel constraint (Muon) performs worse than keeping it fixed, with a gap to the uncompressed baseline approximately 2.2× larger than that of the fixed-projector setting. This suggests that the benefit does not simply come from learning the projection, but specifically from learning it under the Stiefel constraint.

We also want to put the magnitude of the improvement into context. With identical anchors, the fixed-projector model has a validation-loss gap of 0.0161 relative to the uncompressed baseline. This represents the maximum available improvement within this comparison, since the compressed model cannot improve beyond the uncompressed baseline. Stiefel learning closes 0.0025 of this gap. As our paper says in line 089 removing the unnecessary weight constraint imposed by SSN largely helps in recovering performance and makes everything learnable. But blindly learning the projector is detrimental and our manifold aware learning fixes it. We made this more clear in the updated manuscript.

This is also consistent with the results without anchors (figure 8 in the revision; Table 7 in the original submission), where the available gap is larger (0.0373) and MAPL closes 0.0062. Thus, Stiefel learning closes a similar fraction of the remaining performance gap in both settings. The smaller absolute improvement with anchors reflects the smaller available headroom, rather than a weaker effect.

We have included this ablation and the corresponding discussion in the revised paper.

Q2/W2 . We implemented compute/communication overlap in the real-pipeline setup as requested, and yes, the time-to-loss improvement persist when communication overlap is enabled.

At 8B, pp=4, 102× compression, and 1 Gbps, under a 6-hour budget, MAPL achieves a validation loss of 4.159 after 1,184 steps (4.157 second seed), compared with 4.489 (4.484 second seed) after 844 steps for the uncompressed baseline and 5.077 for SSN. MAPL maintains the lowest validation loss at every sampled hour. We also ran this experiment at 200Mbps and MAPL achieved 4.1490 with 1199 steps completed while uncompressed achieved 5.4916 with 362 steps completed.

Accounting for overlap increases the number of steps completed by the uncompressed baseline by 6% compared with our previous setup. However, each micro-batch transfer remains on the critical path. Consequently, the throughput advantage of MAPL narrows from 1.52× to 1.40×, but does not disappear. At 200Mbps the ratio is 3.3x.

We hope these results address both concerns, and we would be glad to clarify anything further. If they do, we would appreciate it if the reviewer would consider updating their assessment.

Dear Reviewer oN3u,

We thank you for your thoughtful review. A gentle reminder that the author-reviewer discussion period is nearing its end. We would greatly appreciate any further feedback or responses to our clarifications before the discussion period closes.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

general response

We thank all four reviewers for their careful feedback and the AC. We are happy to see that the reviewers agree on the importance of the problem we study: reducing activation communication in pipeline-parallel training, particularly in bandwidth-constrained settings. We also appreciate that they find our general approach promising, namely, learning orthonormal projectors at pipeline boundaries without constraining the transformer weights. The reviewers acknowledge the improvements over SSN, and several highlight the value of the different ablations. In particular, none of the reviews disputes the main empirical improvement over SSN. We see this as a good starting point, and have focused our additional experiments on strengthening the evidence and clarifying the remaining questions.

Most of the discussion centered on two aspects of the experimental evaluation. First, the original comparison between fixed and learned projectors in Appendix Table 7 (Figure 8 in the revision) was limited to a single seed and did not include anchors. Second, the paper did not compare against low-bit communication methods and activation quantization baselines. We have conducted additional experiments to address both points. We have updated the paper in blue color to show the changes.

Concern (reviewers)Additional experimentsMain results
Learned vs. fixed projector, with identical anchors and multiple seeds (oN3u, iRMk, CCAA)Three seeds at 4×, pp=4Stiefel-learned projectors outperform fixed projectors in all three paired comparisons at 150M. Unconstrained learning performs worse than the fixed projector. Updated to a unified figure to better reflect improvements.
Quantization and AQ-SGD baselines (iRMk, L4Vp, CCAA)INT4, and INT2 experiments at 150M (pp=4 and pp=8) and 500M; AQ-SGD on a replayed-sample setup (since AQ-SGD require replay).MAPL performed better at compared quantizations. We can also compose MAPL with quantization showing these setups as orthogonal.
Overlap, network conditions, and longer training horizons (oN3u, iRMk)8B experiments with compute/communication overlap at 1 Gbps and 200 Mbps, over 6 and 12 hoursMAPL remains ahead at every sampled hour.
Anchor parameter overhead (iRMk, CCAA)Parameter-matched uncompressed baselines; parameter-matched MAPL, where the anchor replaces the token embedding, at 150M and 500M.Parameter-matched MAPL performs similar at 150M and 500M model sizes.

We believe the additional experiments provide a more complete picture of MAPL and address the main questions raised during the discussion. In particular, they strengthen the comparison against fixed projectors and low-bit communication methods, provide further evidence under different network conditions, and clarify the effects of anchor parameters.

We thank the reviewers again for their constructive feedback and hope this helps to reconsider the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.