acceptodds
Withdrawn from ICLR 2027

Tidal: Accuracy Recovery in Local Learning with Limited Full Backpropagation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Local learning trains consecutive groups of layers, called stages, with separate prediction losses. Removing backward dependencies between stages can increase distributed training throughput, but accuracy can fall sharply when each stage contains few layers. We introduce Tidal, a meta-algorithm that interleaves local learning with full backpropagation (BP), keeping the local objective and total number of training updates fixed. In four-stage, 1B-parameter language-model pretraining, one schedule uses full BP for 54.3% of updates and brings four local methods within 0.6 points of the full-BP reference in mean accuracy across nine tasks. With FluidPipe as the local learning method, a sparse schedule recovers approximately 86% of its accuracy gap using full BP for only 6.7% of updates. It exceeds a conventional BP run trained for the same number of BP updates by 8.8 accuracy points. Limited joint training thus enables existing local methods to retain high task accuracy under finer model partitions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
local learning, decoupled learning, backpropagation, language model pretraining, model parallelism, training schedules

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1fgCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Tidal, a scheduling framework that interleaves local learning and full backpropagation (BP) to recover the accuracy degradation caused by local training. Tidal is compatible with existing local learning methods and switches between local and global objectives according to predefined schedules, without modifying their underlying learning rules.

The authors evaluate Tidal on 1B language-model pretraining with different local learning methods and model partitions. Their experiments show that a schedule using 54.3% full-BP updates brings four local methods close to the full-BP reference, while a sparse schedule using only 6.7% BP recovers approximately 86% of the accuracy gap for FluidPipe. They further investigate BP placement, training trajectories, and generalization to vision models.

Critical strengths
  1. Practical motivation. The paper addresses an important limitation of local learning: substantial accuracy degradation under fine-grained model partitions. Combining local learning with limited full BP is a simple and potentially useful approach to balancing training accuracy and distributed efficiency.

  2. Promising language-model results. The experiments demonstrate substantial accuracy recovery in 1B language-model pretraining, particularly when only a small fraction of updates use full BP.

  3. Evaluation across different local methods and schedules. The authors evaluate four local learning methods, different stage partitions, and several BP allocation strategies. These experiments provide useful empirical observations about the interaction between local and global training.

Critical weaknesses

1. Limited methodological novelty and lack of principled schedule selection

The main contribution is to interleave local learning with full BP, an idea already explored in greedy layerwise pretraining and PGL. Tidal does not introduce a new learning objective or a principled mechanism for choosing when to use BP.

Although the paper evaluates several schedules, the BP fraction, placement, and switching frequency often vary simultaneously, making it difficult to isolate their effects. For example, the sparse schedule allocates most BP updates to the beginning, but it is unclear whether the subsequent periodic BP pulses are necessary or whether an opening-only schedule with the same BP budget would achieve similar performance.

More systematic ablations of BP fraction, switching frequency, and allocation at fixed budgets would help establish a clearer scheduling principle or practical recommendation.

2. Architecture and baseline comparisons are not fully controlled

The conventional full-BP baseline uses an FFN intermediate width of 11,008, whereas Tidal and local learning use 8,192. This means the primary full-BP reference trains a different model, making both accuracy and computational comparisons less direct. Although an additional narrower BP baseline is reported in the appendix, its learning rate also differs. An architecture-matched BP baseline with comparable optimization settings should be the primary reference.

Additionally, the PGL-style baseline differs from Tidal in both auxiliary architecture and scheduling strategy. Consequently, the observed differences cannot be attributed to the proposed schedule alone.

3. Missing equal-compute comparisons and practical efficiency measurements

Figure 2 compares sparse Tidal, which performs 17,500 total updates, against a short BP baseline with only 1,170 updates. While this demonstrates that local updates provide additional learning, it does not establish their computational efficiency, since Tidal processes substantially more data and performs many additional local updates.

An equal-compute or equal-wall-clock comparison is important to determine whether Tidal offers a better accuracy–resource trade-off than conventional BP.

Furthermore, despite the motivation of improving distributed training efficiency, the paper does not provide sufficient direct measurements of Tidal's wall-clock training time, GPU-hours, peak memory, or switching overhead. The practical efficiency claims largely rely on throughput measurements from prior work.

4. Limited analysis of scheduling behavior and scalability

The experiments primarily focus on a single 1B language model with two or four stages. Although different BP budgets and allocations are tested, the relationship between the required BP fraction, model scale, partition granularity, and training progress remains unclear.

It would be useful to investigate whether larger(or smaller) models or finer partitions require more frequent BP guidance, and whether a fixed schedule remains appropriate throughout training. Such analysis would strengthen Tidal as a general scheduling framework rather than a collection of empirically effective training recipes.

Questions for authors
  1. Recommended schedule: What schedule would the authors recommend in practice? How should users choose the BP fraction, opening duration, and pulse frequency for a new model or partition?

  2. Necessity of periodic pulses: For the sparse schedule, would allocating all 1,170 BP updates to the opening phase perform similarly to the proposed periodic schedule? How sensitive are the results to pulse frequency when the total BP budget is fixed?

  3. Architecture-matched BP: Why does the conventional BP reference use wider FFN layers? Could the authors provide comparisons using exactly the same backbone architecture and optimizer settings?

  4. Equal-resource comparison: How does Tidal compare with conventional BP under matched GPU-hours or wall-clock time, including all local updates and switching overhead?

  5. Scaling and adaptive scheduling: How does the required BP fraction change with model size, stage count, or training progress? Could the schedule adapt to local learning performance rather than using predefined switching intervals?

Reviewer_ASbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Tidal, a training schedule that combines existing local learning objectives with selected phases of full backpropagation (BP). It studies whether limited joint training can recover the accuracy lost when a model is divided into small, independently trained stages. Experiments primarily use a 16-layer, 1B language model trained on 36.7 billion tokens. A schedule using 54.3% full-BP updates brings four local methods within 0.6 percentage points of the BP reference across nine downstream tasks. With FluidPipe, a sparse schedule using 6.7% BP recovers approximately 86% of the accuracy gap. Additional experiments examine uneven partitions, BP placement, and smaller vision settings.

Critical strengths

Training trajectories show that local updates can improve accuracy after joint training, while the vision experiments disclose cases where shorter BP performs better. These observations help define where the proposed approach may be useful.

Critical weaknesses
  1. The contribution beyond periodic global guidance is incremental. Alternating local learning with occasional global BP is already the central idea of PGL, which the paper appropriately acknowledges. arxiv.org However, the comparison changes both the auxiliary networks and the BP allocation. Appendix B.3 also states that the adaptation freezes auxiliary predictors during BP, whereas the original PGL continues training them. Consequently, its lower accuracy does not establish an advantage of Tidal’s scheduling choices.

  2. I am wondering about at four stages and a fixed budget of 1,170 BP updates, how does the sparse schedule compare with placing all BP updates at the beginning? How much recovery is attributable to the later pulses?

  3. It is unclear whether Tidal’s four-stage configuration provides a favorable accuracy–memory and training-time trade-off. It would be useful to compare it with relevant local-learning methods[1-3] and memory-saving full-backpropagation training, such as activation checkpointing, under consistent model, data, and hardware settings.

[1] Kappel, David, Khaleelulla Khan Nazeer, Cabrel Teguemne Fokam, Christian Mayr, and Anand Subramoney. "A variational framework for local learning with probabilistic latent representations." In 5th Workshop on practical ML for limited/low resource settings.

[2] Zhang, Aozhong, Zi Yang, Naigang Wang, Yingyong Qi, Jack Xin, Xin Li, and Penghang Yin. "Comq: A backpropagation-free algorithm for post-training quantization." IEEE Access (2025).

[3] Cheng, Anzhe, Heng Ping, Zhenkun Wang, Xiongye Xiao, Chenzhong Yin, Shahin Nazarian, Mingxi Cheng, and Paul Bogdan. "Unlocking deep learning: A bp-free approach for parallel block-wise training of neural networks." In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4235-4239. IEEE, 2024.

Questions for authors

Please see the weakness above.

Reviewer_pg9gOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Local learning removes backwards dependencies in the training pipeline, hence getting better throughput but achieves worse final accuracy. This paper investigates whether limited end-to-end training can recover accuracy lost by local learning in LLM pre-training.

Critical strengths

The opening result is interesting: a local method that performs poorly from initialization can become effective after partial end-to-end training. Recovery across multiple objectives suggests this is not specific to one auxiliary loss; the vision experiments provide useful evidence about where it doesn’t work though.

Critical weaknesses

Poor motivation and baselines. The work draws inspiration and numbers from an anonymous submission, which we cannot verify. The backward dependency motivation seems largely solved in modern training runs outlined in Moonshot and Deepseek’s reports, yet these are not mentioned. The paper cites zero-bubble scheduling but never compares against it. Without these comparisons and baselines, the motivation isn’t meaningful.

Unclear advantage. The paper motivates local learning through training efficiency but never measures it. Figure 2 suggests BP reaches approximately sparse Tidal’s final accuracy in 7,000 updates, versus Tidal’s 1,170 global and 16,330 local updates. Local updates must therefore be 2.8× faster merely to break even, before switching overhead. Removing idle time can speed up a step by at most 1/(1 − idle fraction), so this requires at least 64% of every BP step to be idle, far more than any reasonably configured pipeline (which I think only have about 5% overhead). Sparse Tidal is therefore (far) slower than BP to equal accuracy, although none of this is measured in the paper.

The comparisons contain avoidable confounds. The BP reference uses a different FFN width and the narrower reference uses a different learning rate. Both of these can have huge impacts on training, especially for this somewhat noisy evaluation. Finally, the PGL comparison changes auxiliary networks and scheduling together and differs from the original method.

Questions for authors

see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.