- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Shared-Output Local learning (SOLO), which trains gradient-isolated modules using auxiliary heads that share a read-only, delayed copy of the terminal readout (weight of the last layer). This provides a common output space and conveys information learned by the final module without requiring gradients to propagate across modules. Experiments pretrain 340M–2B language models on 15B tokens, achieving average zero-shot accuracy within 1 percentage point of backpropagation (BP). Readout ablations investigate the mechanism, while separate pipeline experiments demonstrate reduced activation memory and throughput improvements under favorable configurations.
Critical strengths
A simple, well-motivated method supported by informative ablations. The observation that removing global gradients also removes coordination between modules provides a convincing motivation for sharing the terminal readout. Figure 1 clearly communicates the difference from conventional local learning. The readout experiments provide substantive support: comparisons with private and pretrained readouts, together with the rotation and shared-random controls in Table 9, indicate that a common output basis matters beyond readout capacity alone. This makes the contribution both conceptually clear and empirically well supported.
A substantial demonstration of local learning at billion-parameter scale. Training models up to 2B parameters from scratch on 15B tokens is an impressive result. Although SOLO generally trails BP in perplexity, its average downstream accuracy remains close across the tested scales, and it improves over private-readout local learning at similar measured cost. These results provide meaningful evidence that local training can remain competitive beyond small models and classification benchmarks.
Critical weaknesses
The quality and throughput results do not yet establish an end-to-end training advantage. Table 2 compares quality at equal token counts. Why does SOLA achieve higher throughput while being slower in tokens/s? My understanding is as follows: The SOLO requires additional computation and is usually slower in tokens/s. The headline 1.44× pipeline speedup is measured on a separate 96-layer, 1.2B model using synthetic tokens. The main quality experiments in Table 2 use 24-layer models. On the 24-layer model, SOLO reaches only 0.72–0.91× the throughput of 1F1B at a fixed micro-batch size. Is this understanding correct? If this understanding is correct, the pape demonstrates promising quality and systems benefits separately, but does not show whether the speedup can compensate for the quality deficit within a fixed training-time budget. This is consequential to the claim that SOLO is a practical alternative for pretraining.
Questions for authors
Additional evidence central to the current claims: Please clarify the points discussed in the weakness section. Could you also compare validation perplexity and downstream accuracy against elapsed training time using the same model architecture, real training data, hardware, and memory budget, allowing each method to tune its micro-batch size? In particular, does a configuration exhibiting SOLO’s pipeline speedup retain sufficiently competitive quality that processing more tokens offsets its token-matched deficit? Such a comparison would materially strengthen the practical motivation. Also, I want to know what the primary cause of the difference in the speedup is, 1.44× on the 96-layer model and 0.72–0.91× on the 24-layer model. Is it attributed to the model size? Or, does SOLO outperform BP in terms of the throughput in the regime of larger micro-batches (line 373-)? If so, what are the results in Table 2-3 that would be in this configuration? The authors note that "The freed memory becomes throughput." However, in Table 2-3, SOLA's tokens/s is lower even though its peak memory is lower than BP's. Could you elaborate on this point?
3 comments
Thank you for your concise and clear response. Your answers have addressed my main concerns regarding SOLO's trade-offs between perplexity and faster training iterations.
I now understand that SOLO achieves a fixed perplexity more quickly in the specific setting without data parallelism. I also understand that this relationship is influenced by BP's bubble.
As mentioned in the initial review, I think it's impressive that the layer-wise learning method performs as well as BP in large model settings, even under the specific settings. I believe these results will provide valuable insights to the community.
Dear Reviewer,
Thank you for your careful reading of our responses. Your thoughtful questions have helped us make the paper clearer and more precise. We are pleased that the clarifications have addressed your concerns, and we sincerely appreciate your recognition of our results on pretraining at this scale.
With our sincere thanks and best wishes,
The Authors
Thank you for reconstructing the throughput numbers so carefully.
W1, W2
Answer. Your reading is correct. SOLO costs 10–38% more FLOPs; the 1.44× comes from the 96-layer synthetic setting with a 50-step refresh (1.19× with per-step refresh), the 0.72–0.91× from Fig. 13b, and Table 2 is data-parallel. Whether the speed-up pays for the quality gap depends on BP's bubble. When the bubble is large, with fewer micro-batches than stages, SOLO reaches a fixed perplexity sooner, in 0.59–0.97× the time of the fastest BP schedule (median over targets). Under data parallelism there is no bubble to win back, and at BP's GPU-hours SOLO K=2 is about 0.5–0.7 SlimPajama perplexity behind (Appendix I).
Q1
Answer. For validation perplexity, yes, but only with fewer micro-batches than stages. There SOLO is 1.41–2.16× faster while needing 1.14–1.38× BP's tokens by the median, so it comes out ahead on both the median and the final target with K=2 at M/p = 0.5 and K=4 at M/p = 0.25, though with 10–19% more memory than BP (Table 4). With K=4 at M/p = 0.5 it is faster and 15% lighter, about even in time by the median and behind at its final perplexity. In the winning settings each pipeline holds a single 8k sequence per step, so BP has no larger micro-batch to choose. With more micro-batches, or under data parallelism even after tuning the micro-batch for both methods (Table 27), BP reaches the targets sooner. Downstream accuracy was measured only at 15B tokens, where SOLO K=2 is within 0.5 points, so the time comparison uses perplexity.
Q2
Answer. Not parameter count, but layers per stage and the choice of baseline. Against 1F1B, three things add up: deeper stages amortize the cost of the heads (the 96-layer model gives 1.18× and the 24-layer model 0.82× at the same M = 24 and p = 8), a 50-step refresh (1.19× with per-step refresh, Table 20), and larger micro-batches that SOLO can use without a bubble (it peaks at 8–16, 1F1B at 4). 1F1B also keeps its bubble. Against zero-bubble schedules, tuned to the measured time of each pass, SOLO reaches 0.83–0.91× with many micro-batches (Tables 4 and 22), and under data parallelism FSDP makes BP up to 21% faster than the BP (DP) rows of Table 2. Tables 2–3 are not pipelined, so none of their rows falls in this setting.
Q3
Answer. That sentence mixed up two different effects, the memory SOLO saves and the bubble it removes, and the revision separates them. In BP, memory and throughput go together: more micro-batches in flight fill the bubble, so with many micro-batches BP buys throughput with memory and is faster, while with fewer micro-batches than stages no amount of memory helps, and SOLO is 1.41–2.16× faster. In SOLO they do not: it already peaks with one micro-batch in flight, and on the Table 2 models larger micro-batches add little (from 0.91 to 0.93 of ZB-2p at 2B, M/p = 2). Under data parallelism, where BP has no bubble, the head FLOPs therefore keep SOLO at 0.80–0.85× of FSDP-BP, even with both at their best micro-batch (Table 27).
In summary, your reading of the throughput results is correct. The revision now measures where the pipeline speed-up offsets the quality gap, with fewer micro-batches than stages, and where it does not, with many micro-batches or under data parallelism, and it withdraws the claim that the freed memory becomes throughput. We would be glad to clarify any remaining point.