- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Tidal, a scheduling framework that interleaves local learning and full backpropagation (BP) to recover the accuracy degradation caused by local training. Tidal is compatible with existing local learning methods and switches between local and global objectives according to predefined schedules, without modifying their underlying learning rules.
The authors evaluate Tidal on 1B language-model pretraining with different local learning methods and model partitions. Their experiments show that a schedule using 54.3% full-BP updates brings four local methods close to the full-BP reference, while a sparse schedule using only 6.7% BP recovers approximately 86% of the accuracy gap for FluidPipe. They further investigate BP placement, training trajectories, and generalization to vision models.
Critical strengths
-
Practical motivation. The paper addresses an important limitation of local learning: substantial accuracy degradation under fine-grained model partitions. Combining local learning with limited full BP is a simple and potentially useful approach to balancing training accuracy and distributed efficiency.
-
Promising language-model results. The experiments demonstrate substantial accuracy recovery in 1B language-model pretraining, particularly when only a small fraction of updates use full BP.
-
Evaluation across different local methods and schedules. The authors evaluate four local learning methods, different stage partitions, and several BP allocation strategies. These experiments provide useful empirical observations about the interaction between local and global training.
Critical weaknesses
1. Limited methodological novelty and lack of principled schedule selection
The main contribution is to interleave local learning with full BP, an idea already explored in greedy layerwise pretraining and PGL. Tidal does not introduce a new learning objective or a principled mechanism for choosing when to use BP.
Although the paper evaluates several schedules, the BP fraction, placement, and switching frequency often vary simultaneously, making it difficult to isolate their effects. For example, the sparse schedule allocates most BP updates to the beginning, but it is unclear whether the subsequent periodic BP pulses are necessary or whether an opening-only schedule with the same BP budget would achieve similar performance.
More systematic ablations of BP fraction, switching frequency, and allocation at fixed budgets would help establish a clearer scheduling principle or practical recommendation.
2. Architecture and baseline comparisons are not fully controlled
The conventional full-BP baseline uses an FFN intermediate width of 11,008, whereas Tidal and local learning use 8,192. This means the primary full-BP reference trains a different model, making both accuracy and computational comparisons less direct. Although an additional narrower BP baseline is reported in the appendix, its learning rate also differs. An architecture-matched BP baseline with comparable optimization settings should be the primary reference.
Additionally, the PGL-style baseline differs from Tidal in both auxiliary architecture and scheduling strategy. Consequently, the observed differences cannot be attributed to the proposed schedule alone.
3. Missing equal-compute comparisons and practical efficiency measurements
Figure 2 compares sparse Tidal, which performs 17,500 total updates, against a short BP baseline with only 1,170 updates. While this demonstrates that local updates provide additional learning, it does not establish their computational efficiency, since Tidal processes substantially more data and performs many additional local updates.
An equal-compute or equal-wall-clock comparison is important to determine whether Tidal offers a better accuracy–resource trade-off than conventional BP.
Furthermore, despite the motivation of improving distributed training efficiency, the paper does not provide sufficient direct measurements of Tidal's wall-clock training time, GPU-hours, peak memory, or switching overhead. The practical efficiency claims largely rely on throughput measurements from prior work.
4. Limited analysis of scheduling behavior and scalability
The experiments primarily focus on a single 1B language model with two or four stages. Although different BP budgets and allocations are tested, the relationship between the required BP fraction, model scale, partition granularity, and training progress remains unclear.
It would be useful to investigate whether larger(or smaller) models or finer partitions require more frequent BP guidance, and whether a fixed schedule remains appropriate throughout training. Such analysis would strengthen Tidal as a general scheduling framework rather than a collection of empirically effective training recipes.
Questions for authors
-
Recommended schedule: What schedule would the authors recommend in practice? How should users choose the BP fraction, opening duration, and pulse frequency for a new model or partition?
-
Necessity of periodic pulses: For the sparse schedule, would allocating all 1,170 BP updates to the opening phase perform similarly to the proposed periodic schedule? How sensitive are the results to pulse frequency when the total BP budget is fixed?
-
Architecture-matched BP: Why does the conventional BP reference use wider FFN layers? Could the authors provide comparisons using exactly the same backbone architecture and optimizer settings?
-
Equal-resource comparison: How does Tidal compare with conventional BP under matched GPU-hours or wall-clock time, including all local updates and switching overhead?
-
Scaling and adaptive scheduling: How does the required BP fraction change with model size, stage count, or training progress? Could the schedule adapt to local learning performance rather than using predefined switching intervals?