- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FluidPipe (FP), a local-learning method for LLM pretraining that adds one decoder block and a next-token classifier to each intermediate stage. In an 8B model trained with two stages, FP achieves a nine-task mean accuracy similar to full backpropagation, with 37% higher throughput. The paper also analyzes the accuracy collapse that can occur with thinner stages, showing that performance depends on how the model’s layers are partitioned across stages.
Critical strengths
- The paper demonstrates that local learning can improve LLM pretraining throughput while maintaining accuracy close to full backpropagation in a two-stage 8B setting.
- It identifies thin-stage accuracy collapse as a key limitation when scaling local learning to more stages, and shows that its severity depends on how layers are partitioned across stages.
Critical weaknesses
Weaknesses
-
The method appears to be a fixed-depth variant of AugLocal that further improves throughput by using a single auxiliary block. The paper should clarify how this design addresses the requirements of LLM pretraining and what new insight or capability it offers beyond applying local learning at a larger scale. -
In particular, the discussion does not appear to include recent local learning methods such as MLAAN[3], Stochastic Layer-wise Learning[4], LLS[5]. The authors should discuss how these methods relate to FluidPipe and, where appropriate, compare them in the evaluation. Pipeline-parallel methods such as DualPipe[1], Seq1F1B[2], and Chimera [7] should be considered as PP baselines. -
- The throughput comparisons omit recent pipeline-parallel schedules such as DualPipe[1], Seq1F1B[2], and Chimera [7]. Comparisons with these schedules would help show whether FluidPipe’s throughput advantage persists against more optimized PP baselines.
- The main LLM downstream results lack confidence intervals or repeated runs, making it difficult to assess the reliability of the reported accuracy differences. The paper reports multi-seed statistics for some vision experiments, but not comparable uncertainty estimates for the main LLM results.
- The paper does not report memory usage for the main FluidPipe and PP configurations although it discusses memory constraints in its Zero-Bubble comparison. Reporting these measurements would help readers assess FluidPipe’s memory cost and interpret the throughput results.
-
The authors motivate the ResNet experiments as a lower-cost setting for studying the auxiliary block and the representations passed between stages. However, these experiments replace FluidPipe’s decoder block and vocabulary classifier with a ResNet block and a linear classifier, departing from the single-decoder-block auxiliary design and Transformer backbone used for LLM pretraining. It is therefore unclear which findings are expected to transfer to the target setting. Moreover, the resulting vision setup makes FluidPipe appear to be a fixed-depth variant of AugLocal, leaving its design contributions specific to LLM pretraining unclear.
Minor comments
- Figure 4 would be more informative if it also reported per-stage prediction performance in the two-stage setting (e.g., 1B, two stages) along with (a) 1B, four stages and (b) 8B, eight stages. - Periodic synchronization and federated depth splits. The detailed discussion can remain in Appendix A, but a brief explanation in the main Related Work section of how these approaches relate to FluidPipe would improve the paper’s positioning.
- InfoPro adaptation (lines 126–127) in Related work. The description of reconstructing input token embeddings appears to concern the authors’ experimental adaptation of InfoPro rather than prior work. The authors could move this implementation detail to the experimental setup and clarify what is reconstructed and how this adaptation is evaluated.
- GPipe citation (lines 121–122). GPipe is cited earlier in the paper, but adding its citation at this comparison would make the reference easier to locate.
- Predictor scores (lines 344–348). Please clarify what “scores” refers to in “the full-model predictor scores 1.19 points above the first-stage predictor” and in the subsequent per-stage values. If these are nine-task mean accuracies, stating this explicitly would make the result easier to interpret.
[1] DualPipe. https://github.com/deepseek-ai/dualpipe
[2] Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training
[3] Mlaan: Scaling supervised local learning with multilaminar leap augmented auxiliary network
[4] Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation
[5] LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronization
[6] GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
[7] Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
Questions for authors
- In the two-stage setting, does FluidPipe add a single local prediction branch to the first stage, while the final stage uses the original classifier? A brief clarification would help readers understand how the number of auxiliary predictors depends on the number of stages.
- Table 3 reports throughput but not model quality. Do FP and FP+chunked achieve comparable quality to PP? Reporting quality alongside throughput would make the throughput gain clearer.
- Why are four stages used for the classification fine-tuning experiments, given the earlier finding that thinner stages can cause accuracy collapse? Also, why is throughput reported for the two-stage setting? Explaining the rationale for these choices would help readers connect the fine-tuning results to the pretraining findings.