acceptodds
Withdrawn from ICLR 2027

FluidPipe: Local Learning for LLM Pretraining and Its Limits

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pipeline parallelism (PP) distributes groups of model layers, called stages, across accelerators. The accelerators can sit idle while waiting for gradients from later stages. Local learning removes this wait by training intermediate stages with their own auxiliary losses, but the resulting features may be poorly suited to later stages. We present FluidPipe (FP), a local-learning method for LLM pretraining that adds one decoder block to each intermediate stage’s next-token predictor. These blocks are used only for local prediction and discarded at inference. Throughput gains over PP range from 13 to 73% across the tested configurations from 1B to 70B parameters. At 8B, two-stage FP reaches a nine-task mean accuracy within 0.16 points of full backpropagation after about 15B tokens, with 37% higher throughput on eight H100 GPUs. Using more stages, however, leaves fewer layers in each stage. At 1B, FP and three other local methods suffer an accuracy collapse with four stages of four layers. Across different four-stage partitions of the same sixteen layers, FP’s accuracy ranges from 34.48 to 49.52. Preventing this collapse is the next step toward using local learning’s throughput advantage to reduce LLM pretraining cost across more accelerators.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Local learning, large language models, pretraining, pipeline parallelism, distributed training, auxiliary supervision, training efficiency.

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_GvorOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FluidPipe (FP), a local-learning method for LLM pretraining that adds one decoder block and a next-token classifier to each intermediate stage. In an 8B model trained with two stages, FP achieves a nine-task mean accuracy similar to full backpropagation, with 37% higher throughput. The paper also analyzes the accuracy collapse that can occur with thinner stages, showing that performance depends on how the model’s layers are partitioned across stages.

Critical strengths
  • The paper demonstrates that local learning can improve LLM pretraining throughput while maintaining accuracy close to full backpropagation in a two-stage 8B setting.
  • It identifies thin-stage accuracy collapse as a key limitation when scaling local learning to more stages, and shows that its severity depends on how layers are partitioned across stages.
Critical weaknesses

Weaknesses

  • The methodological contribution of FluidPipe remains unclear.\textbf{The methodological contribution of FluidPipe remains unclear.}
    The method appears to be a fixed-depth variant of AugLocal that further improves throughput by using a single auxiliary block. The paper should clarify how this design addresses the requirements of LLM pretraining and what new insight or capability it offers beyond applying local learning at a larger scale.

  • Related work. The paper discusses several relevant methods, but its coverage could be expanded to better position FluidPipe.\textbf{Related work. The paper discusses several relevant methods, but its coverage could be expanded to better position FluidPipe.}
    In particular, the discussion does not appear to include recent local learning methods such as MLAAN[3], Stochastic Layer-wise Learning[4], LLS[5]. The authors should discuss how these methods relate to FluidPipe and, where appropriate, compare them in the evaluation. Pipeline-parallel methods such as DualPipe[1], Seq1F1B[2], and Chimera [7] should be considered as PP baselines.

  • Evaluation. The evaluation does not fully establish FluidPipe’s throughput and accuracy trade-offs against strong baselines or quantify the uncertainty and memory cost of its results. \textbf{Evaluation. The evaluation does not fully establish FluidPipe’s throughput and accuracy trade-offs against strong baselines or quantify the uncertainty and memory cost of its results. }

    • The throughput comparisons omit recent pipeline-parallel schedules such as DualPipe[1], Seq1F1B[2], and Chimera [7]. Comparisons with these schedules would help show whether FluidPipe’s throughput advantage persists against more optimized PP baselines.
    • The main LLM downstream results lack confidence intervals or repeated runs, making it difficult to assess the reliability of the reported accuracy differences. The paper reports multi-seed statistics for some vision experiments, but not comparable uncertainty estimates for the main LLM results.
    • The paper does not report memory usage for the main FluidPipe and PP configurations although it discusses memory constraints in its Zero-Bubble comparison. Reporting these measurements would help readers assess FluidPipe’s memory cost and interpret the throughput results.
  • The analysis in Section 4.6 does not clearly establish how its findings support FluidPipe’s design for LLM pretraining.\textbf{The analysis in Section 4.6 does not clearly establish how its findings support FluidPipe’s design for LLM pretraining.}
    The authors motivate the ResNet experiments as a lower-cost setting for studying the auxiliary block and the representations passed between stages. However, these experiments replace FluidPipe’s decoder block and vocabulary classifier with a ResNet block and a linear classifier, departing from the single-decoder-block auxiliary design and Transformer backbone used for LLM pretraining. It is therefore unclear which findings are expected to transfer to the target setting. Moreover, the resulting vision setup makes FluidPipe appear to be a fixed-depth variant of AugLocal, leaving its design contributions specific to LLM pretraining unclear.

Minor comments

  • Figure 4 would be more informative if it also reported per-stage prediction performance in the two-stage setting (e.g., 1B, two stages) along with (a) 1B, four stages and (b) 8B, eight stages. - Periodic synchronization and federated depth splits. The detailed discussion can remain in Appendix A, but a brief explanation in the main Related Work section of how these approaches relate to FluidPipe would improve the paper’s positioning.
  • InfoPro adaptation (lines 126–127) in Related work. The description of reconstructing input token embeddings appears to concern the authors’ experimental adaptation of InfoPro rather than prior work. The authors could move this implementation detail to the experimental setup and clarify what is reconstructed and how this adaptation is evaluated.
  • GPipe citation (lines 121–122). GPipe is cited earlier in the paper, but adding its citation at this comparison would make the reference easier to locate.
  • Predictor scores (lines 344–348). Please clarify what “scores” refers to in “the full-model predictor scores 1.19 points above the first-stage predictor” and in the subsequent per-stage values. If these are nine-task mean accuracies, stating this explicitly would make the result easier to interpret.

[1] DualPipe. https://github.com/deepseek-ai/dualpipe
[2] Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training
[3] Mlaan: Scaling supervised local learning with multilaminar leap augmented auxiliary network
[4] Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation
[5] LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronization
[6] GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
[7] Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines

Questions for authors
  • In the two-stage setting, does FluidPipe add a single local prediction branch to the first stage, while the final stage uses the original classifier? A brief clarification would help readers understand how the number of auxiliary predictors depends on the number of stages.
  • Table 3 reports throughput but not model quality. Do FP and FP+chunked achieve comparable quality to PP? Reporting quality alongside throughput would make the throughput gain clearer.
  • Why are four stages used for the classification fine-tuning experiments, given the earlier finding that thinner stages can cause accuracy collapse? Also, why is throughput reported for the two-stage setting? Explaining the rationale for these choices would help readers connect the fine-tuning results to the pretraining findings.
Reviewer_oK5KOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the application of local learning in the pre-training of large language models and proposes FluidPipe (FP). It attaches a backbone-matched decoder block as an auxiliary predictor to each intermediate pipeline stage, enabling each stage to compute gradients independently without waiting for backpropagation from subsequent stages. Experimental results show that this approach achieves significant throughput improvement with only a modest reduction in accuracy.

Critical strengths

This paper investigates the key question of the feasibility of local learning in LLM pre-training. The experiments are conducted at a substantial scale, providing a comprehensive analysis of why accuracy collapses when LLMs are split into thin stages, and the implementation is open-sourced.

Critical weaknesses
  1. I believe the most critical issue is the limited novelty of this work. The core design is essentially identical to AugLocal, serving merely as a specific instantiation of an existing local learning paradigm applied to LLMs. As such, the primary contribution lies in the observed phenomenon of accuracy collapse. However, as a purely analytical paper, the analysis lacks sufficient granularity and fails to provide theoretical justification or a pathway toward addressing this phenomenon.
  2. The robustness of the core result ("At 8B, two-stage FP reaches a nine-task mean accuracy within 0.16 points of full backpropagation after about 15B tokens") is insufficient, and the throughput comparison is not a fair comparison under equivalent optimization levels.
Questions for authors
  1. Regarding the authors' observation of accuracy collapse, could the authors provide: (1) theoretical proof and further analysis of the collapse mechanism, such as feature similarity, gradient norm, and loss landscape visualization; and (2) at least one approach to address this collapse phenomenon, such as inter-stage distillation, adaptive boundary, or gradient coupling?
  2. Could the authors clarify what essential differences exist compared to AugLocal?
  3. Could the authors run experiments with multiple random seeds and multiple trials, reporting means and standard deviations?
  4. For the training in Section 4.1, the authors should extend the 8B training to the end of the learning rate decay schedule to verify whether the two-stage results remain stable.
  5. For the throughput comparison, the authors should at least provide a PP + chunking baseline, such as PP with chunked cross-entropy or PP with larger micro-batches (if memory permits).
  6. (Minor issue) Given that the paper does not fully utilize the 9-page limit of the main text and places a relatively large number of experiments in the appendix, the authors should move some important experiments and conclusions from the appendix into the main text.
Reviewer_NmWjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a local learning method named FluidPipe for accelerating pipeline parallelism (PP) in LLM training. Specifically, FluidPipe limits each auxiliary predictor to one decoder block and a next-token classifier. Experimental results at 1B and 8B model scales show that FluidPipe achieves comparable accuracy with PP while achieving higher throughput.

Critical strengths
  • A new local learning model architecture with only one decoder block and a next-token classifier for each auxiliary predictor.
  • Accuracy and throughput are both studied.
Critical weaknesses
  • The proposed method seems not that practical, though the paper provides results on an 8B model with 2 stages. However, it is not robust to the number of stages (which are typically required to be different at different scales of pretraining).
  • The throughput comparison is not a good reference for real-world training, as the real-world configuration of PP typically uses interleaved PP or zero-bubble PP (see Megatron-LM), which can significantly reduce the bubbles during training.
  • The empirical study of local learning limits seems to be a good reference for further study, but it is still limited to particular models and configurations, and it cannot conclude that local learning really has limits.
Questions for authors

No.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.