- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SparseBalance addresses load imbalance in distributed long-context supervised fine-tuning with trainable block-sparse attention, and proposes an algorithm-system co-design for it. The authors argue that the execution latency of an individual micro-batch is determined jointly by two forms of heterogeneity, namely heterogeneity in sequence length and heterogeneity in sparsity sensitivity across sequences and across layers, and that pipeline bubbles together with data-parallel gradient synchronization amplify these differences into a straggler effect. The central idea is to treat the top-k attention budget of sparse attention not as a fixed algorithmic hyper-parameter but as a scheduling variable adjusted at runtime. Dynamic Sparsity Tuning aligns the attention budget of each micro-batch toward an anchor latency, taken as the minimum, mean, or maximum of the predicted micro-batch latencies within a global batch; bottleneck micro-batches are assigned a smaller budget in order to shorten the critical path, whereas faster micro-batches are assigned a larger one, which the authors describe as exchanging otherwise idle pipeline time for "free accuracy". The magnitude of compression is bounded by a threshold p imposed on the decrease in cumulative routing-score coverage, a quantity computed directly from the indexer logits that the underlying sparse attention method already produces, so that the mechanism incurs almost no additional cost. Two supporting components make this practical: a profiling-based latency predictor that uses a single Transformer layer as a proxy and maps discretized sequence-length bins and attention budgets to measured latencies through table lookup at runtime, and Sparsity-Aware Batching, which uses predicted latency as the per-sample workload weight and performs two-level bin packing, first across data-parallel ranks and then within each rank to form micro-batches. The system is implemented on top of Megatron-LM and ms-swift and evaluated with Qwen2.5-0.5B, Qwen2.5-3B, and Qwen2.5-3B with LoRA on 32 H800 and 32 H20 GPUs under a hybrid configuration of DP 4, PP 4, TP 2, and SP 2, with micro-batch size 1 and global batch size 16. The reported end-to-end speedup ranges from 1.25 to 1.33 times, averaging 1.30 times; in a single-node case study the combination of Sparsity-Aware Batching and Dynamic Sparsity Tuning attains 1.35 times against 1.23 times for a length-based batching baseline. On the accuracy side, the overall LongBench score is 39.46 against 39.28 for the baseline, while Needle-in-a-Haystack and the general benchmarks remain essentially at parity.
Critical strengths
1. A new degree of freedom for straggler mitigation. Existing long-context training systems rebalance work by reorganizing data, and the discreteness of that problem limits how well they can do. This paper instead adjusts each micro-batch's top-k budget at runtime. A quality guard computed from the existing indexer logits limits the damage. The mechanism is cheap and complements batching.
2. A complete implementation with consistent gains in the evaluated regime. The system runs on Megatron-LM with 4D parallelism and is tested on two GPU generations, two real datasets with different length distributions, and three model settings. All 12 SAB+DST settings show 1.25–1.33× speedups, and overheads are reported explicitly. The p/anchor knobs give practitioners an explicit speed–accuracy dial, and their effect is characterized.
Critical weaknesses
1. The latency predictor, on which both components depend, is itself never evaluated. The latency predictor underpins both components yet reports no prediction error, no analysis of the interpolation regime or cross-cluster transfer, and does not state whether the profiled table covers the forward pass alone or forward and backward.
2. The baselines are weak, and the contribution of Sparsity-Aware Batching is demonstrated only in combination. The only balancing baseline is length-based batching; WLB-LLM is cited but not compared. LBB alone (1.23×) beats SAB alone (1.21×), so SAB's value rests on a 5% gap (1.35× against 1.28×) from one single-node run with no variance reported.
3. The train-inference inconsistency and the resulting reproducibility concern are not discussed at all. At inference the model runs with a fixed attention budget, whereas during training the budget at each layer and each step depends on the lengths of the other samples in the global batch, on the parallelism configuration, and on a hardware-specific profiling table.
4. Presentation issues. Three unreconciled speedup figures (1.33× in the abstract, 1.35× in Table 1, 1.451× and 1.683× in Fig. 9), a relative 0.46% placed where it reads as absolute, Fig. 7 reported without numeric values, and an AI Use Statement claiming an LLM audited existing results and implementation code without stating what was verified.
Questions for authors
Clarifications :
-
At the same throughput as MEAN0.1, what accuracy does MoBA reach with (a) a smaller global k and (b) a static length-dependent k? Please also report:
- the average executed k (or attention FLOPs) per iteration;
- the fraction of micro-batches compressed vs. expanded;
- how many expanded micro-batches exceed k·B = 8K tokens, i.e., how many expansions actually change the computation.
-
Can you add a MEAN-anchor ablation with compression only (no expansion), to test whether the "free accuracy" really comes from expansion?
-
What are the speedups with more micro-batches per pipeline (e.g., m = 16–64), and against WLB-LLM, Skrull, or sorted / token-budget packing? What exactly is the baseline batching in Fig. 7? With mbs = 1, the second-level packing in Algorithm 2 is trivial (4 samples into 4 bins), so in the main setting SAB seems to balance only across DP ranks. Is that correct?
-
Please report mean ± std over at least 3 seeds for LongBench and NIAH, plus an evaluation beyond 64K (e.g., RULER at 64K/128K). Which attention configuration is used at evaluation time: dense, MoBA with k = 32, or DST?
-
The anchor depends on the other samples in the global batch, so the same sample can receive different budgets in different epochs. Does this affect determinism, reproducibility, or convergence?
Suggestions:
- Unless multiple seeds support it, change "improving accuracy" to "comparable accuracy", and report absolute deltas.
- Define "Overall Weighted Accuracy" and the star marker in Fig. 8. State which setting produced the speedups in Figs. 8–9 (DST only, or SAB+DST); they differ from Fig. 7 and Table 1.
- Reconcile the inconsistent numbers and descriptions: 1.33× vs. 1.35×, and how LongAlign is characterized in §2.1 vs. App. C.1.
- The AI Use Statement contains instruction-like text ("should disclose this use in the submission form") that should be rewritten. Also fix typos such as "could varying", "First, The results", and "making sub-optimal efficiency".