- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MAPL, which compresses forward activations and backward gradients through learned orthogonal projections at pipeline boundaries. Factorized token-conditioned anchors help compensate for compression-induced information loss, while the main model weights remain free of global low-rank constraints. The authors evaluate training quality and communication volume across multiple model scales and compare fixed-time training performance under an emulated bandwidth constraint.
Critical strengths
- It directly targets the communication bottleneck in low-bandwidth pipeline training, with a compression mechanism closely aligned with this objective.
- Applying low-dimensional constraints at communication boundaries while preserving the expressiveness of the main model weights offers a reasonable approach to balancing communication reduction and training quality.
- The authors account for additional costs and examine potential confounding factors. Tables 4–5 assess the capacity effects of the anchors, while Appendix E includes token IDs and projector synchronization in the communication accounting. This provides a more informative assessment than activation compression ratios alone.
Critical weaknesses
- Appendix F states that most quality experiments simulate pipeline stages within a single GPU and the central systems benefit has not been validated on a real bandwidth-constrained network. It therefore remains unclear how much benefit would persist under realistic networking and compute–communication overlap.
- In Table 7, the fixed orthogonal and SPEL-learned projectors achieve losses of 3.2800 and 3.2738, respectively, without anchors—a relatively small difference. A controlled comparison within the complete system, holding the anchors and other conditions fixed, is needed to justify learning the projector over simpler alternatives. Comparisons with competitive low-bit communication methods are also missing.
- The large-model results under aggressive compression do not establish long-term training quality. The 8B experiment runs for only six hours, during which MAPL and the uncompressed model complete 1209 and 796 steps, respectively. This demonstrates an early training benefit under the emulated bandwidth constraint, but does not establish that quality is preserved over longer training. The 3B training budget is also limited, and all experiments use a single random seed.
- Table 3 reports that the anchors add approximately 12M parameters per stage in the 150M setting. The existing capacity controls are useful, but comparisons matched for total parameter count and actual training cost are still needed to assess the overall benefit more accurately.
Questions for authors
- What end-to-end speedup does MAPL achieve on a real multi-node network with propagation latency and compute–communication overlap?
- Does the 8B configuration with preserve quality under token-matched comparisons and longer training budgets? How much benefit does learning the projector provide when identical anchors are used?
4 comments
part 2
**W2 (2) ** 150M, pp=4 (uncompressed 3.2427):
| ratio | method | val | Δ |
|---|---|---|---|
| 4× | INT4 | 3.3228 | +2.47% |
| 4× | MAPL r=256 | 3.2530 | +0.32% |
| 4× | SSN k=256 (Muon) | 3.5546 | +9.62% |
| 8× | INT2 | 4.6163 | +42.36% |
| 8× | MAPL r=128 | 3.2859 | +1.33% |
| 8× | MAPL r=256 + INT8 on the wire | 3.2545 | +0.36% |
We agree that comparisons with low-bit communication methods are important. In our original submission, we did not include these comparisons because quantization is orthogonal to MAPL and can be applied directly to its compressed representations. Nevertheless, following the reviewer's suggestion, we conducted additional experiments on the 150M model with PP=4, using the same training setup and evaluation protocol.
The results show that MAPL substantially outperforms standalone low-bit quantization at matched communication ratios: At 4× compression, INT4 increases validation loss by 2.47%, whereas MAPL (r=256) increases it by only 0.32%. At 8× compression, INT2 leads to a substantial 42.36% increase in validation loss. In contrast, MAPL (r=128) limits the increase to 1.33%. These results demonstrate that MAPL preserves substantially more training quality than aggressive low-bit quantization at comparable communication budgets. Importantly, the two approaches are complementary. To demonstrate this, we also combined MAPL (r=256) with INT8 quantization on the wire. This achieves 8× communication compression while increasing validation loss by only 0.36%, compared with 1.33% for MAPL (r=128) alone and 42.36% for standalone INT2. Overall, these experiments show that MAPL is not only competitive with low-bit communication methods but can also be combined with them to achieve higher compression while maintaining training quality. We have included these comparisons in the revised manuscript.
W3 / Q2:
We extended the 8B experiment from 6 to 12 hours under 1 Gbps link. MAPL maintains lower validation loss throughout the extended run. At 12 hours, MAPL achieves 3.7343 compared with 3.8336 for the uncompressed model, a 2.6% reduction in validation loss, while completing 1.40× more optimizer steps (2367 vs. 1693). Although the gap narrows as training progresses, the advantage persists through the end of the 12-hour budget. This provides evidence that the improvement is not merely a short-lived early-training effect.
We additionally performed a controlled comparison of MAPL and the uncompressed model using the same initialization, data order, and learning-rate schedule, without the constrained link. Both models were trained for 1700 optimizer steps. MAPL achieves a final validation loss of 3.8768 compared with 3.7854 for the uncompressed model, corresponding to a 2.4% relative increase. This quantifies the quality trade-off at equal training progress. In contrast, under a fixed wall-clock budget with the bandwidth constraint, MAPL achieves better validation loss because it enables substantially more training steps. We believe reporting both comparisons is important: token-matched evaluation isolates the quality cost of compression, whereas wall-clock-matched evaluation measures its practical benefit in bandwidth-constrained training. This type of comparison is used in SSN (Neurips 2025) as well.
For the random seed, We also repeated the 8B, 1 Gbps, 6-hour experiment with two seeds (42 and 43). MAPL obtains validation losses of 4.1585 and 4.1565, while the uncompressed model obtains 4.4886 and 4.4840, respectively. The consistent improvement across both seeds suggests that the observed advantage is not specific to a single initialization.
Regarding the 3B experiment, we agree that our training budget does not establish Chinchilla-optimal training. As already acknowledged in the paper, the 3B with 20B token experiments use approximately one-third of the Chinchilla compute-optimal budget. However, this is still a substantial training experiment by academic standards, involving a total compute budget of () FLOPs. We view this as a limitation on claims about fully converged model quality, rather than evidence that the reported results are based on an insufficiently small experiment.
part 3
W4:
We thank the reviewer for this suggestion. The anchors are factorized lookup tables rather than additional transformer layers. They introduce negligible additional matrix-multiplication cost and do not add FLOP-heavy computations to the forward or backward passes. As discussed in the paper, their relative parameter overhead also decreases substantially with model scale. Furthermore, Appendix A.1 shows that the benefit of anchors is not simply a generic increase in model capacity.
Nevertheless, we agree that parameter- and compute-matched comparisons provide a more complete assessment, particularly at smaller scales. We therefore conducted additional controlled experiments at 150M and 500M parameters.
150M, pp=4, 4× compression (baseline: 3.2427).
| Configuration | Parameter change | Validation loss |
|---|---|---|
| Uncompressed | — | 3.2427 |
| Uncompressed + factorized anchors | +24% | 3.2266 |
| Uncompressed + widened FFN (2688 → 4544) | +24% | 3.1988 |
| MAPL, r=256 | +24% | 3.2530 |
| MAPL, anchor replaces token embedding | +0.37% | 3.2531 |
Adding the same factorized anchors to the uncompressed model improves validation loss by only 0.50%. In contrast, widening the FFN with a comparable parameter increase achieves a 1.35% improvement. This shows that adding factorized anchors is not equivalent to increasing transformer capacity through additional FLOP-bearing parameters. MAPL achieves 3.2530 under the original anchor configuration.
We additionally evaluate sharing the stage-0 anchor with the token embedding, eliminating the separate vocabulary embedding table. This reduces MAPL's parameter overhead from 24% to just 0.37%, while achieving a validation loss of 3.2531.
500M, pp=4, 6× compression (baseline: 2.8522).
| Configuration | Parameter change | Validation loss |
|---|---|---|
| Uncompressed | — | 2.8522 |
| Uncompressed + factorized anchors | +7.9% | 2.8367 |
| MAPL, r=256 | +7.9% | 2.8737 |
| MAPL, anchor replaces token embedding | −3.7% | 2.8737 |
We observe the same trend at 500M: adding factorized anchors to the uncompressed model improves validation loss by only 0.54%, while MAPL achieves 2.8737 under 6× activation compression. Sharing the anchor with the token embedding reduces the total parameter count to 3.7% below the uncompressed baseline without changing MAPL's validation loss. These results show that the anchor parameter count alone does not capture its computational or capacity contribution. The anchors provide a lightweight mechanism for compensating for information lost during activation compression, rather than acting as a conventional increase in transformer capacity.
Dear Reviewer iRMk,
We thank you for your thoughtful feedback. A gentle reminder that the author-reviewer discussion period is nearing its end. We would greatly appreciate any further feedback or responses to our clarifications before the discussion period closes.
Thank you again for your time and consideration.
reply iRMk
Thank you for the thoughtful review. We have addressed all four weaknesses and three questions with additional experiments. The new results and corresponding discussions are added to the revised paper in blue color.
W1/Q1: The 8B experiment in our original paper already uses a real pipeline, with one stage per GPU and a 1F1B schedule. We have now added communication overlap to this setup.
At 8B, pp=4, and 1 Gbps, MAPL achieves a validation loss of 4.159 after 6 hours (1184 steps). The uncompressed baseline reaches 4.489 (844 steps), while SSN reaches 5.077. At 200 Mbps, MAPL achieves 4.149 (1199 steps), compared with 5.492 for the uncompressed baseline (362 steps). We also ran this setup again on a 200Mbps link and found that MAPL achieves validation loss of 4.1490 (at 1199 steps) while uncompressed achieve 5.4916 (at 362 steps).
Communication overlap increases the uncompressed baseline's steps by 6%. However, some communication remains on the critical path during pipeline fill and drain. In particular, each micro-batch transfer must still be completed before the next dependent computation can proceed. As a result, MAPL achieves 1.40× higher end-to-end training throughput than the uncompressed baseline at 1 Gbps, increasing to 3.3× at 200 Mbps.
To address multi-node part of Q1, we measured end-to-end speedup of MAPL pp=4 with multi node setup on two h200 nodes with stage 0 and 1 on single node with stages split across gpus and 2 and 3 on other node. We measured the speedup at different inter node latency for MAPL and uncompressed model and show the speedup. (batch 32 × 2048, micro-batch 4; median step time over steps )
We can see the compression only adds a small compute cost. The speedup shrinks from 3.5× to 2.1× as latency goes 0 → 200 ms at 1 Gbps and is 1.2× at 10 Gbps. This shows in bandwidth constrained multinode setup, MAPL has upto 10x speedup over uncompressed.
W2 / Q2(2) We agree that a controlled comparison within the complete system is important. We therefore conducted an additional experiment at 150M parameters with pp=4, keeping the factorized anchors and all other conditions fixed. We compared three projector configurations: a fixed orthogonal projector, an unconstrained learned projector optimized with Muon, and a Stiefel-constrained learned projector optimized with SPEL. We ran three paired seeds, using the same initialization and data order for each comparison.
The results are shown below.
We argue that simply learning the projector is not sufficient. The unconstrained Muon projector consistently performs worse than the fixed projector across all three seeds. Its gap to the uncompressed baseline is approximately 2.2 times larger. In contrast, Stiefel-constrained learning consistently improves over the fixed projector, reducing the validation loss by 0.0025 on average. This shows that the benefit does not come from learning the projection alone. Preserving its orthogonality during optimization is important.
Second, the absolute improvement needs to be considered relative to the remaining performance headroom. With factorized anchors, the fixed-projector model has a validation-loss gap of 0.0161 relative to the uncompressed baseline. Stiefel learning closes 0.0025 of this gap, recovering approximately 16% of the remaining loss gap.
This is consistent with our results without anchors in Appendix Table 7 (Figure 8 in revision). There, the available gap is larger (0.0373), and stiefel learning closes 0.0062, or approximately 16.6% of the gap. Thus, Stiefel learning recovers a comparable fraction of the remaining performance gap in both settings. The smaller absolute improvement with anchors is expected because the anchors have already recovered much of the compression-induced performance loss, leaving less headroom for projector learning.
Removing the unnecessary global low-rank constraint on model weights already recovers much of the lost performance (as stated in line 89 of our paper), while factorized anchors further compensate for information loss at pipeline boundaries. However, learning the projection without respecting its orthogonal structure can be detrimental. MAPL addresses this by learning stage-specific projections directly on the Stiefel manifold.