Tidal: Accuracy Recovery in Local Learning with Limited Full Backpropagation

Local learning trains consecutive groups of layers, called stages, with separate prediction losses. Removing backward dependencies between stages can increase distributed training throughput, but accuracy can fall sharply when each stage contains few layers. We introduce Tidal, a meta-algorithm that interleaves local learning with full backpropagation (BP), keeping the local objective and total number of training updates fixed. In four-stage, 1B-parameter language-model pretraining, one schedule uses full BP for 54.3% of updates and brings four local methods within 0.6 points of the full-BP reference in mean accuracy across nine tasks. With FluidPipe as the local learning method, a sparse schedule recovers approximately 86% of its accuracy gap using full BP for only 6.7% of updates. It exceeds a conventional BP run trained for the same number of BP updates by 8.8 accuracy points. Limited joint training thus enables existing local methods to retain high task accuracy under finer model partitions.