- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SSN++, which improves communication-efficient pipeline training through separate and grouped subspaces, gated aggregation, and causal decoding of previously received compressed activations. Experiments across three model sizes show improved model quality at comparable communication budgets, with further gains from trading communication precision for higher rank.
Critical strengths
-
SSN++ separates local representational capacity from the rank of the communicated activations. Its grouped output projections can span more than k directions without increasing the number of trainable projection coefficients, while causal decoders reuse previously received activations without additional communication.
-
Experiments from 229M to 2B parameters consistently show lower validation loss and higher mean downstream accuracy, including BF16 comparisons against parameter-matched SSNs at comparable communication budgets. Representation analyses further show increasing use of local output directions beyond the transmitted rank, supporting the architectural motivation.
Critical weaknesses
-
The strongest quantized results lack a higher-rank SSN baseline using the same INT8/INT4 communication scheme under a matched byte budget. Although the BF16 comparisons already support the architectural improvements, this additional baseline would clarify whether SSN++ retains its advantage when SSN is given the same precision–rank trade-off.
-
The cumulative training ablations bundle grouped MLP projections with gating and introduce all three decoders together, leaving their individual training benefits incompletely isolated. Although Appendix J.7 reports separate post-training removal interventions, these do not establish the gains from training with each component independently. A parameter-matched current-token-only decoder would further help isolate the benefit of historical activations.
Questions for authors
Same as the weaknesses above.