acceptodds
Under review as a conference paper at ICLR 2027

SSN++: Layerwise Subspace Networks for Low-Bandwidth Pipeline Training

Abstract

Subspace networks (SSNs) make pipeline-parallel training communication-efficient by constraining every layer to write into one shared low-dimensional subspace, so activations and gradients cross each stage boundary losslessly. This shared subspace, however, limits what the model can express, and with it the loss an SSN can reach relative to an uncompressed transformer. We propose SSN++, which removes this constraint without additional communication. We introduce a norm-preserving orthogonal transform that losslessly transports each compressed activation from one layer's subspace into the next, so every layer can write into its own subspace. We extend this separation within each layer, giving attention and the MLP separate subspaces and dividing each into groups with their own subspaces, which together exceed the rank of one subspace at equal parameters, and we add learned gates that choose what each layer passes on. Because the compressed activations of preceding tokens have already crossed the boundary, the receiving layer also decodes its input from them with a learned causal filter, raising its rank beyond one subspace. Random subspaces spread activations like rotation-based quantisers, which makes the communication amenable to quantisation, and we spend the saved bits on a higher rank. Approximation theory and measurements during training support each choice. From 229M to 2B parameters, SSN++ lowers loss and raises eval accuracy over a parameter-matched SSN at equal communication, and with quantised communication it closes 63 to 78% of the matched SSN's loss gap to the uncompressed transformer at 229M and 1B and 58% at 2B, while training 12 to 26 times faster per update than it over 200 Mb/s links. The loss gap of SSNs thus stems largely from how capacity is organised around the compressed activations, and closing it without extra communication makes pipeline-parallel training over internet-grade links far more efficient.

Primary area:
foundation or frontier models, including LLMs
Keywords:
communication-efficient training, model parallelism, low-rank subspace, activation compression, quantisation, neural networks, decentralised training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EvSNOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SSN++, which improves communication-efficient pipeline training through separate and grouped subspaces, gated aggregation, and causal decoding of previously received compressed activations. Experiments across three model sizes show improved model quality at comparable communication budgets, with further gains from trading communication precision for higher rank.

Critical strengths
  1. SSN++ separates local representational capacity from the rank of the communicated activations. Its grouped output projections can span more than k directions without increasing the number of trainable projection coefficients, while causal decoders reuse previously received activations without additional communication.

  2. Experiments from 229M to 2B parameters consistently show lower validation loss and higher mean downstream accuracy, including BF16 comparisons against parameter-matched SSNs at comparable communication budgets. Representation analyses further show increasing use of local output directions beyond the transmitted rank, supporting the architectural motivation.

Critical weaknesses
  1. The strongest quantized results lack a higher-rank SSN baseline using the same INT8/INT4 communication scheme under a matched byte budget. Although the BF16 comparisons already support the architectural improvements, this additional baseline would clarify whether SSN++ retains its advantage when SSN is given the same precision–rank trade-off.

  2. The cumulative training ablations bundle grouped MLP projections with gating and introduce all three decoders together, leaving their individual training benefits incompletely isolated. Although Appendix J.7 reports separate post-training removal interventions, these do not establish the gains from training with each component independently. A parameter-matched current-token-only decoder would further help isolate the benefit of historical activations.

Questions for authors

Same as the weaknesses above.

Reviewer_WHTPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to improve the expressiveness of SSN in low-bandwidth pipeline training while maintaining a narrow communication interface. SSN++ assigns independent subspaces to different layers, sublayers, and output groups, uses orthogonal transformations to pass compressed coefficients, and adds local computation through gated aggregation and history-token decoders. It also uses the bytes saved by low-precision communication to increase rank. The paper reports improvements in validation loss and average downstream accuracy for 229M, 1B, and 2B models.

Critical strengths
  1. The main controlled comparison is convincing. The parameter-matched SSN also includes nonlinear value networks and embedding factorization, so the BF16 results cannot simply be attributed to additional parameters. In Table 1, SSN++ reduces C4 loss by approximately 0.252, 0.128, and 0.116 nats across the three model scales, respectively. The average accuracy across four tasks in Table 2 also improves consistently. These results support the paper’s central point that communication rank is not the sole determinant of model quality.
  2. The paper examines the roles of its components in some depth. Appendix I.1 separately compares shared, layerwise, and sublayerwise subspaces, while Appendix J tracks the ranks of local outputs and transmitted representations and performs component-removal interventions. The paper clearly distinguishes between a trained model’s reliance on a component and the training gains attributable to adding that component. Appendix F.5 also compares the time required to reach the same loss and acknowledges that the dense model is both faster and more accurate on a single node.
Critical weaknesses
  1. The gains of the full approach are not yet sufficiently disentangled. In Tables 1 and 8, only SSN++ uses low-precision communication to increase rank, while the parameter-matched SSN remains in BF16. The quantization rationale in Section 3.4 also applies to standard SSN, making an INT8/INT4 SSN with increased rank under the same byte budget a directly relevant control. At the 1B scale, the architectural changes at fixed rank close 33.0% of the loss gap, whereas the 62.6% figure also includes gains from increasing rank through quantization. The current evidence supports the importance of capacity organization, but does not establish that it plays a more important role than communication rank.
  2. The optimizer settings for the other baselines need clarification. Appendix F.2 states that the comparisons concern the implementations used in this paper, but Appendix F.3 lists a common AdamW configuration and identifies only MAPL as an exception. The original ResBM paper uses Muon for its bottleneck models and reports that it outperforms AdamW (ResBM, Sections 4.1–4.2), so this is not a minor implementation detail. It remains unclear whether this setting was retained.
  3. The independent contribution of the history decoders remains unclear. Table 9 adds the temporal, attention, and MLP decoders together, and all include both current-token and history-token terms. The statistics and removal experiments in Appendices J.4 and J.7 show that the model uses these modules, but do not distinguish the benefits of historical information from those of additional current-token transformations. The authors acknowledge this limitation. Nevertheless, since reusing already-transmitted history is a central design choice, I would still like to see a control that retains the current-token terms and removes only the history terms.
  4. The theoretical rationale for quantization is not sufficiently rigorous. Appendix J.5 treats the row norms of a Haar orthogonal basis as exactly equal, whereas it is the expected squared row norm that equals kd. Moreover, a more uniform d-dimensional expanded representation does not guarantee that the k-dimensional coefficients actually being quantized are free of outliers. The proposed connection between random subspaces and quantization tolerance therefore needs further support.
Questions for authors
  1. How much of SSN++’s advantage remains when the parameter-matched SSN also uses INT8/INT4 and increases rank under the same communication budget? Please also report the parameter counts.
  2. Which optimizer was used for the ResBM baseline? If Muon was not used, what was the rationale for the chosen configuration?
  3. Compared with retraining decoders of comparable capacity that use only the current token, how much additional improvement do the history terms provide?
Reviewer_2F75OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SSN++, a communication-efficient architecture for low-bandwidth pipeline-parallel training. Existing Subspace Networks (SSNs) reduce activation and gradient communication by model into a shared low-dimensional subspace. SSN++ decouples the low-dimensional communication interface from local computation: different layers, attention/MLP sublayers, and output groups can use different local subspaces while the state communicated across stage boundaries remains low-dimensional. The method further combines gated aggregation, temporal decoding, nonlinear value computation, and low-precision communication to increase model capacity under a similar communication budget.

Across 229M, 1B, and 2B models, SSN++ improves language-modeling loss and downstream performance over parameter-matched SSNs at comparable communication volume, and is also compared against other compressed inter-stage baselines including MAPL and ResBM.

Critical strengths

The main idea directly addresses a limitation of SSNs. A low-dimensional stage-boundary communication interface does not require all layers and sublayers to share the same low-dimensional representation space. SSN++ therefore decouples the communication constraint from local representational capacity, retaining a narrow transmitted state while allowing different layers, sublayers, and output groups to use richer local representations. The main architectural components are coherent with this objective rather than being a collection of unrelated modifications.

The experimental validation and theoretical support are closely aligned with the core intuition and are convincing. The main experiments demonstrate improved model quality under comparable communication and parameter budgets, while the architectural ablations, representation analyses, and approximation/capacity results provide complementary evidence for the proposed explanation. In particular, the paper separately studies shared, per-layer, and per-sublayer subspaces: per-layer separation alone changes the representation geometry without materially improving loss, whereas separating attention and MLP subspaces produces a clear performance improvement. Together with the more detailed treatment of different layers, sublayers, and local components, this forms a relatively complete evidence chain connecting the paper's intuition, design choices, and empirical gains.

Critical weaknesses

None.

Questions for authors

I would be interested in understanding more about how the benefit of subspace separation varies across different layers and functional modules. The paper already shows that per-layer separation alone is not sufficient to improve loss, while further separating attention and MLP subspaces leads to a clearer gain. It would therefore be interesting to know whether some layers or modules benefit more strongly from independent subspaces than others. For example, do attention and MLP modules, or early/middle/late layers, show systematically different sensitivity to the shared-subspace constraint? It would also be useful to know whether the representation statistics analysed in the paper, such as subspace overlap, superposition, or effective rank, correlate with these differences. Such an analysis could provide an even more detailed picture of where the gains of SSN++ come from.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.