acceptodds
Under review as a conference paper at ICLR 2027

SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning

Abstract

Large language models are trained with backpropagation, whose global gradient coordinates all layers but forces each to hold its activations and wait for the gradient to pass back through every deeper layer. Conventional local learning removes this update locking by training each module to predict the target through its own readout, but has not scaled to billion-parameter pretraining. We identify these private readouts as a key weakness, since they leave each module without information from deeper modules. We propose Shared-Output LOcal learning (SOLO), which replaces them with a shared, read-only copy of the final module's readout, the only one trained on the output of the whole network. Taken from the previous step, the copy transmits information from the final module without passing gradients between modules or reintroducing update locking. SOLO approaches backpropagation on Transformers of 340M to 2B parameters pretrained on 15B tokens, staying within one point in average zero-shot accuracy with a perplexity gap that narrows with scale. Readout ablations attribute SOLO's improvement over private readouts to sharing. Without update locking, each of pipeline stages holds activations for micro-batches instead of . The freed memory permits larger micro-batches, which reach up to 1.44 the best measured throughput of pipeline on the same backpropagation partition. To our knowledge, SOLO is the first local learning method to show such memory and throughput gains in billion-parameter language-model pretraining. Local learning thus becomes a practical alternative to backpropagation for large-scale pretraining.

Primary area:
learning theory
Keywords:
Local Learning, Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_adczOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Shared-Output Local learning (SOLO), which trains gradient-isolated modules using auxiliary heads that share a read-only, delayed copy of the terminal readout (weight of the last layer). This provides a common output space and conveys information learned by the final module without requiring gradients to propagate across modules. Experiments pretrain 340M–2B language models on 15B tokens, achieving average zero-shot accuracy within 1 percentage point of backpropagation (BP). Readout ablations investigate the mechanism, while separate pipeline experiments demonstrate reduced activation memory and throughput improvements under favorable configurations.

Critical strengths

A simple, well-motivated method supported by informative ablations. The observation that removing global gradients also removes coordination between modules provides a convincing motivation for sharing the terminal readout. Figure 1 clearly communicates the difference from conventional local learning. The readout experiments provide substantive support: comparisons with private and pretrained readouts, together with the rotation and shared-random controls in Table 9, indicate that a common output basis matters beyond readout capacity alone. This makes the contribution both conceptually clear and empirically well supported.

A substantial demonstration of local learning at billion-parameter scale. Training models up to 2B parameters from scratch on 15B tokens is an impressive result. Although SOLO generally trails BP in perplexity, its average downstream accuracy remains close across the tested scales, and it improves over private-readout local learning at similar measured cost. These results provide meaningful evidence that local training can remain competitive beyond small models and classification benchmarks.

Critical weaknesses

The quality and throughput results do not yet establish an end-to-end training advantage. Table 2 compares quality at equal token counts. Why does SOLA achieve higher throughput while being slower in tokens/s? My understanding is as follows: The SOLO requires additional computation and is usually slower in tokens/s. The headline 1.44× pipeline speedup is measured on a separate 96-layer, 1.2B model using synthetic tokens. The main quality experiments in Table 2 use 24-layer models. On the 24-layer model, SOLO reaches only 0.72–0.91× the throughput of 1F1B at a fixed micro-batch size. Is this understanding correct? If this understanding is correct, the pape demonstrates promising quality and systems benefits separately, but does not show whether the speedup can compensate for the quality deficit within a fixed training-time budget. This is consequential to the claim that SOLO is a practical alternative for pretraining.

Questions for authors

Additional evidence central to the current claims: Please clarify the points discussed in the weakness section. Could you also compare validation perplexity and downstream accuracy against elapsed training time using the same model architecture, real training data, hardware, and memory budget, allowing each method to tune its micro-batch size? In particular, does a configuration exhibiting SOLO’s pipeline speedup retain sufficiently competitive quality that processing more tokens offsets its token-matched deficit? Such a comparison would materially strengthen the practical motivation. Also, I want to know what the primary cause of the difference in the speedup is, 1.44× on the 96-layer model and 0.72–0.91× on the 24-layer model. Is it attributed to the model size? Or, does SOLO outperform BP in terms of the throughput in the regime of larger micro-batches (line 373-)? If so, what are the results in Table 2-3 that would be in this configuration? The authors note that "The freed memory becomes throughput." However, in Table 2-3, SOLA's tokens/s is lower even though its peak memory is lower than BP's. Could you elaborate on this point?

3 comments

Thank you for reconstructing the throughput numbers so carefully.

W1, W2

Is it correct that SOLO needs more compute and is usually slower, with 1.44× from a separate synthetic 96-layer setting and 0.72–0.91× on the 24-layer model, so that it is unclear whether the speed-up offsets the quality gap within a fixed time budget?

Answer. Your reading is correct. SOLO costs 10–38% more FLOPs; the 1.44× comes from the 96-layer synthetic setting with a 50-step refresh (1.19× with per-step refresh), the 0.72–0.91× from Fig. 13b, and Table 2 is data-parallel. Whether the speed-up pays for the quality gap depends on BP's bubble. When the bubble is large, with fewer micro-batches than stages, SOLO reaches a fixed perplexity sooner, in 0.59–0.97× the time of the fastest BP schedule (median over targets). Under data parallelism there is no bubble to win back, and at BP's GPU-hours SOLO K=2 is about 0.5–0.7 SlimPajama perplexity behind (Appendix I).

Q1

Compared on the same model, data, hardware and memory, with each method tuning its micro-batch, does SOLO's pipeline speed-up offset its token-matched deficit in perplexity and downstream accuracy?

Answer. For validation perplexity, yes, but only with fewer micro-batches than stages. There SOLO is 1.41–2.16× faster while needing 1.14–1.38× BP's tokens by the median, so it comes out ahead on both the median and the final target with K=2 at M/p = 0.5 and K=4 at M/p = 0.25, though with 10–19% more memory than BP (Table 4). With K=4 at M/p = 0.5 it is faster and 15% lighter, about even in time by the median and behind at its final perplexity. In the winning settings each pipeline holds a single 8k sequence per step, so BP has no larger micro-batch to choose. With more micro-batches, or under data parallelism even after tuning the micro-batch for both methods (Table 27), BP reaches the targets sooner. Downstream accuracy was measured only at 15B tokens, where SOLO K=2 is within 0.5 points, so the time comparison uses perplexity.

Q2

Why is SOLO 1.44× on the 96-layer model but 0.72–0.91× on the 24-layer model: model size or larger micro-batches, and which rows of Tables 2–3 fall in that regime?

Answer. Not parameter count, but layers per stage and the choice of baseline. Against 1F1B, three things add up: deeper stages amortize the cost of the heads (the 96-layer model gives 1.18× and the 24-layer model 0.82× at the same M = 24 and p = 8), a 50-step refresh (1.19× with per-step refresh, Table 20), and larger micro-batches that SOLO can use without a bubble (it peaks at 8–16, 1F1B at 4). 1F1B also keeps its bubble. Against zero-bubble schedules, tuned to the measured time of each pass, SOLO reaches 0.83–0.91× with many micro-batches (Tables 4 and 22), and under data parallelism FSDP makes BP up to 21% faster than the BP (DP) rows of Table 2. Tables 2–3 are not pipelined, so none of their rows falls in this setting.

Q3

If "the freed memory becomes throughput", why is SOLO's throughput in Tables 2–3 lower than BP's despite its lower memory?

Answer. That sentence mixed up two different effects, the memory SOLO saves and the bubble it removes, and the revision separates them. In BP, memory and throughput go together: more micro-batches in flight fill the bubble, so with many micro-batches BP buys throughput with memory and is faster, while with fewer micro-batches than stages no amount of memory helps, and SOLO is 1.41–2.16× faster. In SOLO they do not: it already peaks with one micro-batch in flight, and on the Table 2 models larger micro-batches add little (from 0.91 to 0.93 of ZB-2p at 2B, M/p = 2). Under data parallelism, where BP has no bubble, the head FLOPs therefore keep SOLO at 0.80–0.85× of FSDP-BP, even with both at their best micro-batch (Table 27).

In summary, your reading of the throughput results is correct. The revision now measures where the pipeline speed-up offsets the quality gap, with fewer micro-batches than stages, and where it does not, with many micro-batches or under data parallelism, and it withdraws the claim that the freed memory becomes throughput. We would be glad to clarify any remaining point.

Thank you for your concise and clear response. Your answers have addressed my main concerns regarding SOLO's trade-offs between perplexity and faster training iterations.

I now understand that SOLO achieves a fixed perplexity more quickly in the specific setting without data parallelism. I also understand that this relationship is influenced by BP's bubble.

As mentioned in the initial review, I think it's impressive that the layer-wise learning method performs as well as BP in large model settings, even under the specific settings. I believe these results will provide valuable insights to the community.

Dear Reviewer,

Thank you for your careful reading of our responses. Your thoughtful questions have helped us make the paper clearer and more precise. We are pleased that the clarifications have addressed your concerns, and we sincerely appreciate your recognition of our results on pretraining at this scale.

With our sincere thanks and best wishes,

The Authors

Reviewer_qfSoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SOLO, a local learning method that shares the final output readout across Transformer modules to improve training quality while preserving gradient isolation. The results show that SOLO achieves model quality close to backpropagation (BP) at the billion-parameter scale, while reducing memory usage and improving training throughput in certain pipeline-parallel settings.

Strengths

  • The method is simple yet interesting. Sharing the final readout effectively improves training quality over conventional local learning.
  • The paper includes extensive ablation studies and demonstrates the potential of gradient isolation to reduce memory and communication overhead.

Weaknesses

  1. Lack of training efficiency comparisons at matched model quality.

This is my main concern. SOLO consistently achieves worse model quality than BP, but the paper does not demonstrate that it can reach the same quality at a lower training cost. For example, on the 2B model, SOLO has higher perplexity (21.57 vs. 20.93) and lower throughput (53k vs. 62k tokens/s). These results do not establish SOLO as a more efficient alternative to BP for LLM pretraining. I would like to see comparisons of model quality at equal GPU-hours, or the training cost required to reach the same target perplexity.

  1. The model quality and system performance results are not evaluated together.

The reported 1.44× throughput improvement comes from a separate experiment using synthetic data, with a different model architecture, auxiliary head configuration, and readout refresh schedule from the actual pretraining experiments. It is therefore unclear whether this speedup can be achieved while maintaining model quality close to BP. An end-to-end evaluation that reports both model quality and training efficiency under the same configuration would be much more convincing.

  1. Insufficient comparison with stronger pipeline-parallel baselines.

The system evaluation mainly compares SOLO against 1F1B and interleaved pipeline parallelism, without directly evaluating more recent methods such as Zero-Bubble Pipeline Parallelism. Since eliminating pipeline bubbles is one of SOLO's key system advantages, it is important to establish whether the method still provides meaningful benefits over these optimized BP approaches.

Overall Assessment

The paper presents an interesting local learning method and shows promising system-level benefits from gradient isolation. However, the main concern is that it does not demonstrate whether SOLO can achieve comparable model quality to BP at a lower overall training cost. The quality and performance results come from different experimental settings, leaving the practical benefits unclear.

Critical strengths

The method is simple yet interesting. Sharing the final readout effectively improves training quality over conventional local learning.

Critical weaknesses
  1. Lack of training efficiency comparisons at matched model quality.

This is my main concern. SOLO consistently achieves worse model quality than BP, but the paper does not demonstrate that it can reach the same quality at a lower training cost. For example, on the 2B model, SOLO has higher perplexity (21.57 vs. 20.93) and lower throughput (53k vs. 62k tokens/s). These results do not establish SOLO as a more efficient alternative to BP for LLM pretraining. I would like to see comparisons of model quality at equal GPU-hours, or the training cost required to reach the same target perplexity.

Questions for authors

Please refer to the weaknesses.

1 comment

Thank you for asking for the comparison that the submission lacked. We ran it (Appendices H and I) and narrowed our claims accordingly. Under data parallelism, SOLO's benefit is memory, not speed. In pipelines with fewer micro-batches than stages, SOLO also trains faster on the same GPUs, but BP's most efficient configuration still uses fewer total GPU-hours. What we claim is that local learning can pretrain 340M–2B models close to BP: with K=2, within 0.5 points in average zero-shot accuracy and 3.1–3.6% in WikiText perplexity, with less memory.

W1

SOLO has worse quality and lower throughput than BP, so compare quality at equal GPU-hours, or the cost to reach a target perplexity.

Answer. These are the right comparisons, and in the Table 2 setting SOLO is not cheaper. On the 2B model, SOLO K=2 needs 315 GPU-hours to reach its final perplexity of 10.55, which BP reaches in 202, and at BP's budget it is about 0.5 perplexity behind (Table 26). In pipelines with fewer micro-batches than stages, the comparison on a fixed set of GPUs changes: at M/p = 0.5 on the 2B model, SOLO needs 0.82× BP's time by the median over targets and 0.91× to its final perplexity (Table 22). The revision replaces the efficiency claim with these conditions.

W2

The 1.44× speed-up comes from a separate synthetic-data setting, so quality and speed should be evaluated in one configuration.

Answer. Speed and quality now come from the same 1.3B and 2B models, splits, heads and refresh. We did not train end to end in a pipeline: a pipeline schedule does not change BP's gradient and changes SOLO's only through a readout copy at most three steps old, so quality can be read from the Table 2 curves. The one remaining difference, context length, we checked with a single-seed 8k training pair, whose gap is smaller than at 2k (2.5% vs 4.0%); we keep the 2k numbers rather than correct them in SOLO's favour. The throughput runs use random token ids, which do not affect throughput.

W3

Does SOLO still help against zero-bubble pipeline schedules?

Answer. Yes, as long as there are few micro-batches per stage. We implemented ZB-1p and ZB-2p and compared against the fastest of them. Up to one micro-batch per stage they cannot fill the bubble, and SOLO matches or exceeds their throughput (1.01–2.16×). With more micro-batches, ZB-2p removes the bubble by holding more activations, and SOLO falls to 0.83–0.91× of its throughput while using 38% less memory with K=2 (Table R). We did not compare with tensor, context or sequence-chunked parallelism (§5, Appendix H).

In summary, we ran the comparison you asked for and revised the paper to match it. SOLO is not cheaper in total GPU-hours; it is faster on a given number of GPUs when micro-batches are few, and it pretrains 340M–2B models close to BP with less memory. We hope this addresses your main concern, and we would be glad to clarify any remaining point.

Reviewer_j2B8OpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SOLO, a local learning method for language model pretraining that isolates gradients between modules while giving every auxiliary prediction head a shared, frozen copy of the final module's output readout. The copy comes from an earlier training step, allowing information from the final module to reach earlier modules without requiring their updates to wait for a downstream gradient. Experiments train Transformers with 340M, 1.3B, and 2B parameters on 15B tokens. SOLO remains within one percentage point of backpropagation in average accuracy across six evaluation tasks, with a remaining perplexity gap. Readout and gradient alignment ablations investigate why sharing helps. Separate pipeline measurements demonstrate substantially lower activation memory and throughput improvements reaching 1.44 times the best measured 1F1B configuration.

Critical strengths
  1. A simple and consequential solution to coordination in local learning. SOLO identifies an unnecessary restriction in conventional local learning: isolated gradients do not require isolated output maps. Sharing the terminal readout gives all modules a common vocabulary basis while preserving independent local backward passes. This is a meaningful architectural insight because it restores a form of coordination without returning to full backpropagation. The 15B token experiments extend the idea to models with billions of parameters, and the average task accuracy gap remains below one point at all three reported scales.

  2. Well-supported mechanism and practical benefits. The ablation studies demonstrate the importance of shared readouts, while gradient analysis shows improved alignment with backpropagation. The pipeline experiments further validate the memory savings and throughput gains, providing strong evidence for both the effectiveness and practical value of SOLO.

Critical weaknesses

The paper does not yet demonstrate the full quality and speed benefit in one setting. The principal quality experiments use 24 layer models, two or four modules, and a readout refreshed every step. The largest throughput gain uses a 96 layer model, eight modules, a smaller vocabulary, synthetic tokens, and a refresh period of 50 steps. Appendix H explicitly excludes model quality from those throughput ratios. Moreover, auxiliary heads add approximately 11% to 13% training FLOPs with two modules, so matching tokens does not match compute.Consequently, the results establish promising quality and promising efficiency separately, but leave the time needed to reach a fixed quality target uncertain.

Questions for authors
  1. For a model configuration used in the 15B token quality study, what is the quality versus elapsed training time comparison when each method uses its best feasible batch setting?

  2. Does a refresh period of 50 steps preserve quality in the larger pretraining models, and how much of the reported pipeline advantage remains against an optimized BP schedule that eliminates bubbles under the same memory budget?

2 comments

Thank you for the precise questions. They point to the gap in the submission, and the new experiments address it by measuring speed and quality on the same models.

W1, W3, Q1

Quality and speed are measured in different settings, so the time each method needs to reach a fixed quality, at its best batch setting, is unknown.

Answer. We now measure both on the 1.3B and 2B models of Table 2, with the same splits, heads and per-step refresh (Appendices H and I). The outcome depends on the parallel setting. In pipelines with fewer micro-batches than stages, SOLO reaches a fixed perplexity sooner than the fastest BP schedule, in 0.59–0.97× the time by the median over targets, and with K=2 at M/p = 0.5 and K=4 at M/p = 0.25 it is also sooner to its own final perplexity (Table 22). In these two settings each pipeline holds a single 8k sequence per step, so BP has no larger micro-batch to choose. Under data parallelism BP is faster: SOLO K=2 needs about 1.4–1.5× BP's GPU-hours from 340M to 2B (Table 26), and tuning the micro-batch for both methods does not change this (Table 27).

W2

The auxiliary heads add 11–13% training FLOPs, so matching tokens does not match compute.

Answer. Agreed. With K=2 the heads add 10–13% FLOPs (new ρ column in Table 2). We therefore trained a FLOP-matched BP at 340M on 13% more tokens: it improves from 28.04 to 27.78 WikiText perplexity, so its lead over SOLO (29.05) grows from 1.01 to 1.27. All GPU-hour comparisons use measured throughput and so already pay for the heads.

Q2

Does a 50-step refresh preserve quality in the larger pretraining models?

Answer. Yes, at a small cost. Over three seeds on WikiText-103, a 50-step refresh costs about 0.3 perplexity at 64M, 128M and 340M alike (0.5–0.7%, Appendix F). SlimPajama runs at 340M and 1.3B agree within run-to-run variation (Table 14). The pipeline results in Table R refresh every step, with the copy at most three steps old, so they do not depend on S=50.

Q3

How much of the pipeline advantage remains against a bubble-free BP schedule at the same memory budget?

Answer. It depends on how many micro-batches each pipeline has. With fewer micro-batches than stages, zero-bubble schedules cannot fill the bubble, and SOLO stays well ahead of the fastest of 1F1B, ZB-1p and ZB-2p (1.41–2.16×). Memory is mixed there: when BP holds a single micro-batch, SOLO's heads cost 10–19% more memory, but with K=4 at M/p = 0.5 SOLO is both faster (1.42×) and 15% lighter. At one micro-batch per stage, SOLO is still slightly faster with K=2, at 23% less memory. With many micro-batches, ZB-2p removes the bubble by holding more activations and BP becomes faster (SOLO 0.83–0.91×), and BP keeps a small lead even when held to SOLO's memory (Table 24). Against bubble-free schedules, the advantage therefore remains where micro-batches are few and becomes a memory saving where they are plentiful.

In summary, speed and quality are now measured on the same models. SOLO reaches a fixed perplexity sooner in pipelines with fewer micro-batches than stages, BP is faster otherwise, and every comparison accounts for the head FLOPs and the refresh period. We hope this answers the time-to-quality question, and we would be glad to clarify any remaining point.

Thank you for the detailed clarification. Your response adequately addresses my concern regarding the time-to-quality comparison. I am satisfied with the explanation and will increase my confidence score accordingly.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Shared Response

We thank all reviewers. All three ask how long each method needs to reach the same quality. The answer depends on how SOLO's lower memory relates to its throughput, and new experiments on the Table 2 models measure both.

Memory and throughput. In BP the two are coupled: a pipeline fills its bubble by keeping micro-batches in flight, so BP buys throughput with memory, up to a limit set by M/p, the micro-batches per pipeline per step divided by the number of stages. In SOLO they are decoupled: each stage runs its backward pass right after its forward pass, so SOLO has no backward bubble and runs near its peak at any M/p, with memory independent of M/p; the head FLOPs set that peak at 0.8–0.9× of BP's with K=2. SOLO is therefore faster where BP cannot remove its bubble, up to one micro-batch per stage, and uses less memory where BP can remove it or has none.

Table R (Table 4 in the PDF). SOLO/BP on the 1.3B model, against the fastest of 1F1B, ZB-1p and ZB-2p in pipelines (8k context, Appendix H) and FSDP-BP under data parallelism (Appendix I). Time is the SOLO/BP time to the same perplexity on the same GPUs, median over targets; time below 1 and negative memory favour SOLO. 2B agrees within 0.02 in throughput; pipeline memory is measured at 2B. —: two stages cannot reach M/p = 0.25.

SettingThroughput K=2Throughput K=4Time K=2Time K=4Memory K=2Memory K=4
Pipeline, M/p = 0.25—2.15—0.64—+19%
Pipeline, M/p = 0.51.431.420.790.97+10%−15%
Pipeline, M/p = 11.081.011.051.36−23%−46%
Pipeline, M/p = 20.900.831.261.65−38%−65%
Data-parallel0.820.701.391.98−29%−44%

With fewer micro-batches than stages, SOLO is 1.41–2.16× faster and reaches most targets sooner; with many micro-batches, BP is faster and SOLO uses less memory. M/p falls as GPUs are added at a fixed batch. Meta reports few micro-batches per stage as the main constraint on pipeline scheduling when training Llama 3 405B on 16K GPUs; that run used M/p = 1 and changed its schedule to allow fewer micro-batches than stages (Chu et al., 2025; Grattafiori et al., 2024). At M/p = 1, SOLO is faster but not sooner to a fixed perplexity (Table R). At both sizes, BP's most efficient measured configuration (many micro-batches, fewer GPUs) uses fewer total GPU-hours than any SOLO configuration; SOLO's pipeline gain is wall-clock time on a given number of GPUs.

Quality. SOLO pretrains billion-parameter language models from scratch with quality close to BP. From 340M to 2B, SOLO K=2 stays within 0.5 points of BP in average zero-shot accuracy (41.5/45.4/45.8 vs 41.5/45.9/46.1) and trails by 3.1–3.6% in WikiText perplexity (0.64–1.01). Under data parallelism with both methods sharded, SOLO uses 28–30% less memory with K=2 and 42–44% with K=4.

Scope. Time to target is computed from measured 8k throughput and the token ratios of the 2k curves of Table 2, not from an end-to-end pipelined run. The token ratios come from one seeded run per arm (of the three behind Table 2), so ratios within a few percent of 1 are not resolved; a single-seed 8k pair shows a smaller gap than at 2k (2.5% vs 4.0%), consistent with these ratios not favouring SOLO. Apart from the data-parallel all-reduce, each pipeline's step time depends only on its own M and p, so a slice of up to eight GPUs gives the per-GPU throughput of a larger run at the same M/p (for the 1.3B model at our batch, M/p < 1 corresponds to more than 64 GPUs); data-parallel communication across slices is not included. Apart from data parallelism, we compared only with pipeline schedules, not with tensor, context or sequence-chunked parallelism.

Changes in the PDF. Abstract, introduction, §3.3 (new Table 4), limitations and conclusion are rewritten around these results. Table 2 adds FLOPs per token and a FLOP-matched BP. Appendix H adds the zero-bubble comparison, the 8k pair and equal memory; new Appendix I adds time to target and a micro-batch sweep; Appendices F and G add refresh at 1.3B and sharded memory; Related Work covers more pipeline schedules. Table, figure and appendix numbers refer to the revised PDF.

References.

[1]. Chu, W., Xie, X., Yu, J., et al. Scaling Llama 3 Training with Efficient Parallelism Strategies. In Proc. 52nd Annual International Symposium on Computer Architecture (ISCA), 2025.

[2]. Grattafiori, A., Dubey, A., Jauhri, A., et al. The Llama 3 Herd of Models. arXiv:2407.21783, 2024 .

Dear Reviewers,

Thank you for your contributions to the review process. As the author–reviewer discussion period ends on November 18, I would like to kindly remind everyone to review the authors' rebuttals and engage in the discussion.

In particular, please indicate whether your main concerns have been adequately addressed and clarify any remaining issues that may affect your final assessment.

Your feedback will be very helpful for the subsequent reviewer–AC discussion and meta-review process.

Thank you for your time and effort!

AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and Reviewers (j2B8, qfSo, adcz) for their constructive feedback.

Brief Context. SOLO trains gradient-isolated modules with a shared, read-only copy of the terminal readout. From 340M to 2B parameters, it stays within one point of BP in zero-shot accuracy, uses up to 44% less memory than sharded BP, and reaches most perplexity targets sooner than zero-bubble schedules when pipelines have fewer micro-batches than stages.

  • Time to quality in one configuration [j2B8, qfSo, adcz]. We now measure speed and quality on the same 1.3B and 2B models against 1F1B, ZB-1p, ZB-2p and FSDP-BP (Table 4). With fewer micro-batches than stages, SOLO needs 0.59–0.97× BP's time; otherwise BP is faster, and the paper now says so. Reviewer j2B8 found that this "adequately addresses my concern" and will increase their confidence; Reviewer adcz: "addressed my main concerns".
  • Compute and cost [j2B8, qfSo]. A FLOP-matched BP and GPU-hours to target with a micro-batch sweep (Appendix I) show SOLO is not cheaper in total GPU-hours; its gain is wall-clock time on a given number of GPUs.
  • Memory vs. throughput [j2B8, qfSo, adcz]. BP buys throughput with memory; SOLO has no backward bubble and needs no extra memory to reach its peak. With many micro-batches, it trails ZB-2p (0.83–0.91×) at 38% less memory.
  • Refresh and context [j2B8, qfSo]. A 50-step refresh costs about 0.3 perplexity over three seeds; an 8k-context pair shows a smaller gap than at 2k (2.5% vs 4.0%).

Reviewer adcz expects the results to "provide valuable insights to the community"; we hope the AC agrees.

Thank you again for your time and thoughtful feedback throughout the review process.

Best regards,

The Authors of Submission 13603

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.