- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies data assignment for distributed training with periodic model synchronization. It proposes weighted stratified sharding, which groups examples using cached training losses and constructs weighted subsets for individual workers. Sampling weights preserve the full-data objective in expectation, while coordinated assignment reduces sample overlap across workers when possible. The theoretical analysis connects local objective approximation to gradient heterogeneity and gives a convergence bound for momentum Local SGD within a fixed reconstruction interval. Experiments cover image classification and mathematical continual pretraining of TinyLlama-1.1B. The reported results show improvements over random partitioning in training loss, downstream accuracy, and iterations required to reach a specified loss or perplexity.
Critical strengths
The paper addresses a relevant problem through a practical modification to data assignment. It reuses losses collected during training and integrates the construction into the data sampler without changing the model architecture. The construction also avoids duplicate assignments across workers whenever the stratum size permits. Table 5 reports an overhead of approximately 0.4% relative to random partitioning for the CIFAR-10 experiment with ResNet-18 and a synchronization interval of 10.
The empirical support extends beyond a single task or configuration. The vision experiments examine multiple synchronization intervals, and the appendix includes comparisons involving node counts, optimizers, synchronization strategies, sample overlap, and proportional stratification. In mathematical continual pretraining, Table 1 reports improvements on all nine evaluated benchmarks relative to random partitioning for both datasets. The average scores increase from 16.6 to 17.9 on OpenWebMath-Pro and from 35.1 to 37.4 on Jiuzhang3.0. These results support the practical value of the overall method.
Critical weaknesses
The handling of cached losses needs clarification. Appendix C explains that losses are recorded during training and initialized to zero. Appendix D also considers using only a fraction of the dataset between shard reconstructions. I could not find a clear description of how cached values are maintained for examples that are not selected during an interval. Figure 3b provides a useful comparison between cached and freshly evaluated loss rankings, but does not explain this update rule. This makes it difficult to assess how current the information used for sharding remains throughout training.
The ablations provide useful evidence for the overall method, but leave some uncertainty about the source of its improvements. Table 7 compares proportional unweighted stratification with the proposed allocation and weighting together. It therefore does not separately establish how much benefit comes from selecting and weighting examples versus assigning those examples across workers. Figure 3c reports variation in local loss values, which provides an indirect check of the proposed reduction in worker heterogeneity. A comparison that preserves the selected examples and their weights while changing their assignment across workers would help clarify this
Questions for authors
How are cached losses handled for examples that are not selected between consecutive shard reconstructions? Are their previous values retained, and is there any refresh procedure? When using partial-data reconstruction intervals, is there an initial pass over the full dataset before loss-based stratification begins?
Do you have a comparison that preserves the selected examples, their weights, and the per-worker sample budgets, but randomizes their assignment across workers? If available, measurements of gradient disagreement evaluated at a common model parameter would also help clarify whether the improvement is associated with reduced worker heterogeneity.
4 comments
W2&Q2 [Ablation on worker assignment and measurement of gradient disagreement]
A. Ablation on Worker Assignment. Table 7 evaluates sampling and weighting, while the following comparison examines worker assignment with the selected sample-weight pairs held fixed. We consider the reviewer's suggested variant as "Random Assignment", which preserves the selected samples, their weights, and the per-worker sample budgets, but randomly redistributes the selected sample-weight pairs across workers. Unlike Algorithm 1, Random Assignment may assign worker a different number of sample-weight pairs from stratum than the prescribed , while retaining the original weight for each pair. Consequently, the total weight assigned to this stratum becomes instead of . Although this reassignment preserves the consensus weighted objective , it changes the worker-specific objectives and can thus alter the local optimization between synchronizations. Accordingly, the per-worker approximation guarantee of Lemma 3.1 no longer directly applies for this variant.
Table R6. Ablation on data sampling and worker assignment on Tiny-ImageNet with ResNet-50 (, ). Random uses uniform sampling, Stratified Random uses proportional stratification without reweighting, and Coreset (ours) follows Algorithm 1. Random Assignment preserves the selected samples, weights, and per-worker sample budgets, but randomly reassigns the sample-weight pairs across workers. Results are averaged over three runs. Accuracy gain and iteration speedup are relative to Random.
| Method | Test Acc. (%) | Final Loss | Acc. Gain (%) | Iteration Speedup |
|---|---|---|---|---|
| Random | 61.9 0.1 | 0.77 0.02 | +0.0 | 1.00 |
| Stratified Random | 62.1 0.1 | 0.73 0.01 | +0.2 | 1.06 |
| Random Assignment | 60.6 0.2 | 0.83 0.03 | -1.3 | 0.94 |
| Coreset | 62.9 0.2 | 0.61 0.01 | +1.0 | 1.18 |
Measurement of Gradient Heterogeneity. Beyond the cross-node loss variation in Figure 3c, we directly assess the gradient heterogeneity bounded in Lemma 3.3 by measuring the mean squared deviations of individual local objective gradients from the original global objective gradient. Both Coreset and Random use the same original global objective as the reference. For Coreset, the mean squared disagreement with the consensus gradient is no larger than this measure, as shown in the proof of Theorem 3.4. Specifically, at the first local step after each synchronization, all workers () share the consensus parameter . We compute $D _ {t}=\frac{1}{K} \sum _ {k=1}^{K} \left | \nabla \tilde F^k(\bar{\theta} _ {t}) - \nabla F(\bar{\theta} _ {t}) \right | ^2 $ and report its average within each 10% training interval.
Table R7. Mean squared deviations of local objective gradients from the original global objective gradient during ResNet-50 training on Tiny-ImageNet (, ). Each entry reports the average over the corresponding 10% training interval. The final column reports the relative change in the mean across all ten intervals, using Random as the reference. Lower values indicate lower gradient heterogeneity.
| Method | 0-10% | 10-20% | 20-30% | 30-40% | 40-50% | 50-60% | 60-70% | 70-80% | 80-90% | 90-100% | Avg. Relative Change (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Random | 0.570 | 0.942 | 0.928 | 0.728 | 0.650 | 0.536 | 0.506 | 0.469 | 0.472 | 0.450 | 0.0 |
| Coreset | 0.544 | 0.935 | 0.851 | 0.711 | 0.647 | 0.524 | 0.458 | 0.435 | 0.386 | 0.379 | -6.1 |
Table R7 shows consistently lower gradient heterogeneity for Coreset across all ten training intervals, with a 6.1% reduction in average heterogeneity compared with Random. These results provide direct empirical evidence that our weighted stratified sharding reduces discrepancies between local objective gradients and the original global objective gradient at a common model parameter.
Thank you.
Dear Reviewer V1Dw,
Thank you for taking the time to review our paper and provide valuable feedback. As the discussion phase is nearing its conclusion, we would like to confirm if our responses have effectively addressed your concerns. If you have any additional comments, we will do our best to address them.
Best regards,
The authors of Submission 30496.
Response to Reviewer V1Dw
We thank the reviewer for the constructive feedback. We address your concerns below and have incorporated the corresponding clarifications and additional results into the revised manuscript, with changes highlighted in blue.
W1&Q1 [The handling of cached losses]
A. Samples not processed between reconstructions retain their cached losses. These samples remain eligible for subsequent shard assignments, and their cached losses are refreshed using newly observed losses when they are sampled. For processed samples, we account for multiple observations across workers, as Algorithm 1 permits overlap in small strata to ensure the approximation guarantee. No initial pass over the full dataset is required before weighted stratified sharding, including the case for using partial data during reconstruction intervals. All cached losses are initially zero, so the first reconstruction contains only one stratum as detailed in Appendix C. As training proceeds, more samples acquire observed losses, allowing subsequent reconstructions to exploit increasingly informative stratification. The weighting rule in Section 3 preserves the unbiasedness of each weighted local objective Fk with respect to F. Remark 3.2 further analyzes the approximation error caused by stale observed losses.
Here, we provide the technical details for cached losses maintenance. Let ℓi denote the cached loss of sample zi, following the notation in the analysis of Remark 3.2. Define Vi as the set of worker-step pairs (k,t) at which zi is processed between consecutive shard reconstructions, where θtk denotes the parameter of worker k at step t, as defined in Section 2. At each reconstruction, if ∣Vi∣>0, we update the cached loss as ℓi←∣Vi∣1∑(k,t)∈Vifi(θtk). Otherwise, ℓi retains its previous value, including zero for samples never observed. This update averages newly observed, unweighted sample losses. As described in Appendix C, per-sample losses are collected during ordinary training forward passes and synchronized through All-Gather. After All-Gather, each worker applies this averaging rule to the collected observations and updates its local loss cache.