acceptodds
Under review as a conference paper at ICLR 2027

SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention

Abstract

While sparse attention mitigates the computational bottleneck of long-context LLM training, its distributed training process exhibits extreme heterogeneity in both 1) sequence length and 2) sparsity sensitivity, leading to a severe imbalance problem and sub-optimal model accuracy. Existing algorithms and training frameworks typically focus on single issue, failing to systematically co-optimize these two problems. Therefore, we propose SparseBalance, a novel algorithm-system co-design framework, which exploits the sparsity and sequence heterogeneity to optimize model accuracy and system efficiency jointly. First, we propose workload-aware dynamic sparsity tuning, which employs a bidirectional sparsity adjustment to mitigate stragglers and exploit inherent bubbles to increase attention budgets for non-bottleneck micro-batches. Second, we propose a sparsity-aware batching strategy to achieve coarse-grained balance, which complements dynamic sparsity tuning. Experimental results demonstrate that SparseBalance achieves up to a 1.33x end-to-end speedup while maintaining downstream benchmark performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Attention, Distributed Training, Load Imbalance, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_L9RWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SparseBalance addresses load imbalance in distributed long-context supervised fine-tuning with trainable block-sparse attention, and proposes an algorithm-system co-design for it. The authors argue that the execution latency of an individual micro-batch is determined jointly by two forms of heterogeneity, namely heterogeneity in sequence length and heterogeneity in sparsity sensitivity across sequences and across layers, and that pipeline bubbles together with data-parallel gradient synchronization amplify these differences into a straggler effect. The central idea is to treat the top-k attention budget of sparse attention not as a fixed algorithmic hyper-parameter but as a scheduling variable adjusted at runtime. Dynamic Sparsity Tuning aligns the attention budget of each micro-batch toward an anchor latency, taken as the minimum, mean, or maximum of the predicted micro-batch latencies within a global batch; bottleneck micro-batches are assigned a smaller budget in order to shorten the critical path, whereas faster micro-batches are assigned a larger one, which the authors describe as exchanging otherwise idle pipeline time for "free accuracy". The magnitude of compression is bounded by a threshold p imposed on the decrease in cumulative routing-score coverage, a quantity computed directly from the indexer logits that the underlying sparse attention method already produces, so that the mechanism incurs almost no additional cost. Two supporting components make this practical: a profiling-based latency predictor that uses a single Transformer layer as a proxy and maps discretized sequence-length bins and attention budgets to measured latencies through table lookup at runtime, and Sparsity-Aware Batching, which uses predicted latency as the per-sample workload weight and performs two-level bin packing, first across data-parallel ranks and then within each rank to form micro-batches. The system is implemented on top of Megatron-LM and ms-swift and evaluated with Qwen2.5-0.5B, Qwen2.5-3B, and Qwen2.5-3B with LoRA on 32 H800 and 32 H20 GPUs under a hybrid configuration of DP 4, PP 4, TP 2, and SP 2, with micro-batch size 1 and global batch size 16. The reported end-to-end speedup ranges from 1.25 to 1.33 times, averaging 1.30 times; in a single-node case study the combination of Sparsity-Aware Batching and Dynamic Sparsity Tuning attains 1.35 times against 1.23 times for a length-based batching baseline. On the accuracy side, the overall LongBench score is 39.46 against 39.28 for the baseline, while Needle-in-a-Haystack and the general benchmarks remain essentially at parity.

Critical strengths

1. A new degree of freedom for straggler mitigation. Existing long-context training systems rebalance work by reorganizing data, and the discreteness of that problem limits how well they can do. This paper instead adjusts each micro-batch's top-k budget at runtime. A quality guard computed from the existing indexer logits limits the damage. The mechanism is cheap and complements batching.

2. A complete implementation with consistent gains in the evaluated regime. The system runs on Megatron-LM with 4D parallelism and is tested on two GPU generations, two real datasets with different length distributions, and three model settings. All 12 SAB+DST settings show 1.25–1.33× speedups, and overheads are reported explicitly. The p/anchor knobs give practitioners an explicit speed–accuracy dial, and their effect is characterized.

Critical weaknesses

1. The latency predictor, on which both components depend, is itself never evaluated. The latency predictor underpins both components yet reports no prediction error, no analysis of the interpolation regime or cross-cluster transfer, and does not state whether the profiled table covers the forward pass alone or forward and backward.

2. The baselines are weak, and the contribution of Sparsity-Aware Batching is demonstrated only in combination. The only balancing baseline is length-based batching; WLB-LLM is cited but not compared. LBB alone (1.23×) beats SAB alone (1.21×), so SAB's value rests on a 5% gap (1.35× against 1.28×) from one single-node run with no variance reported.

3. The train-inference inconsistency and the resulting reproducibility concern are not discussed at all. At inference the model runs with a fixed attention budget, whereas during training the budget at each layer and each step depends on the lengths of the other samples in the global batch, on the parallelism configuration, and on a hardware-specific profiling table.

4. Presentation issues. Three unreconciled speedup figures (1.33× in the abstract, 1.35× in Table 1, 1.451× and 1.683× in Fig. 9), a relative 0.46% placed where it reads as absolute, Fig. 7 reported without numeric values, and an AI Use Statement claiming an LLM audited existing results and implementation code without stating what was verified.

Questions for authors

Clarifications :

  1. At the same throughput as MEAN0.1, what accuracy does MoBA reach with (a) a smaller global k and (b) a static length-dependent k? Please also report:

    • the average executed k (or attention FLOPs) per iteration;
    • the fraction of micro-batches compressed vs. expanded;
    • how many expanded micro-batches exceed k·B = 8K tokens, i.e., how many expansions actually change the computation.
  2. Can you add a MEAN-anchor ablation with compression only (no expansion), to test whether the "free accuracy" really comes from expansion?

  3. What are the speedups with more micro-batches per pipeline (e.g., m = 16–64), and against WLB-LLM, Skrull, or sorted / token-budget packing? What exactly is the baseline batching in Fig. 7? With mbs = 1, the second-level packing in Algorithm 2 is trivial (4 samples into 4 bins), so in the main setting SAB seems to balance only across DP ranks. Is that correct?

  4. Please report mean ± std over at least 3 seeds for LongBench and NIAH, plus an evaluation beyond 64K (e.g., RULER at 64K/128K). Which attention configuration is used at evaluation time: dense, MoBA with k = 32, or DST?

  5. The anchor depends on the other samples in the global batch, so the same sample can receive different budgets in different epochs. Does this affect determinism, reproducibility, or convergence?

Suggestions:

  1. Unless multiple seeds support it, change "improving accuracy" to "comparable accuracy", and report absolute deltas.
  2. Define "Overall Weighted Accuracy" and the star marker in Fig. 8. State which setting produced the speedups in Figs. 8–9 (DST only, or SAB+DST); they differ from Fig. 7 and Table 1.
  3. Reconcile the inconsistent numbers and descriptions: 1.33× vs. 1.35×, and how LongAlign is characterized in §2.1 vs. App. C.1.
  4. The AI Use Statement contains instruction-like text ("should disclose this use in the submission form") that should be rewritten. Also fix typos such as "could varying", "First, The results", and "making sub-optimal efficiency".
Reviewer_yMC8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies load imbalance in distributed long-context training with sparse attention. It points to two sources: sequence length varies a lot, and different sequences and layers need different attention budgets. SparseBalance combines a batching step with a runtime tuning step. The batching step estimates sparsity and packs samples by predicted latency. The runtime step lowers the budget for bottleneck micro-batches and raises it for others, within a coverage bound. Experiments use Qwen2.5-0.5B and 3B on two clusters and two long-context datasets.

Critical strengths
  1. Two-level design fits the problem: SAB handles coarse imbalance, and DST handles the rest at runtime. In the case study, the imbalance metric falls from 1.81 to 1.34, and iteration time falls from 4202 ms to 3108 ms (Table 1).

  2. Consistent speedup: The average is 1.30x and the peak is 1.33x over two clusters, two datasets, and three model settings (Figure 7).

Critical weaknesses
  1. Missing system baselines: The only load-balancing baseline is length-based batching. WLB-LLM [1], Skrull [2] and ChunkFlow [3] target the same problem but are not compared.

  2. Accuracy within noise and net budget unreported: LongBench is 39.46 vs 39.28, general is 0.6967 vs 0.6958, and NIAH at 64K is 97.53 vs 97.60, with no variance. The paper also does not report the average attention budget, so a higher net budget could explain the small accuracy gain.

  3. One sparse-attention backend: DST relies on routing logits from MoBA, and no other sparse-attention method is tested. This leaves the generality of the tuning step unclear.

References: [1] Wang, et al. WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training, OSDI, 2025. [2] Xu, et al. Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling, NeurIPS, 2025. [3] Yuan, et al. Efficient Long Context Fine-tuning with Chunk Flow, ICML, 2025.

Questions for authors
  1. Can you compare with WLB-LLM, Skrull, or ChunkFlow under the same setting?

  2. What are the mean and variance over multiple seeds, and what is the average attention budget for the baseline and for SparseBalance?

  3. Does DST work with sparse attention methods that do not expose routing logits, such as NSA [1]?

References: [1] Yuan, et al. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention, ACL, 2025.

Reviewer_LJpTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to tackle sequence length heterogeneity and sparsity sensitivity heterogeneity simultaneously, proposing SparseBalance that dynamically adjusts attention sparsity to take into consideration of computational workloads across micro-batches. Mainly, the proposed method is consisted of, Sparsity-Aware Batching for coarse-grained data packing based on a latency predictor, Workload-Aware Dynamic Sparsity Tuning which bidirectionally adjusts the attention budget at runtime and offline profiling-based Latency Predictor that maps sequence length and sparsity to practical execution time. The authors evaluate their method on Qwen2.5 using LongAlign and ChatQA data to show that Sparsealance can achieve up e2e speedup over standard batching while matching or improving task performance.

Critical strengths
  • The motivation part of this paper is pretty thorough including why traditional FLOPS-based or length-based packing is limited by showing latency is in practice not necessarily linear to sequence length and. theoretical FLOPS for instance in Figure 2 and 56.
  • The paper bridges systems and algorithms by taking a adaptive system variable approach to sparsity where they treat the attention sparsity budget not as a hyperparameter but a dynamic variable for system load balancing.
  • The paper is relevant for large-scale distributed training efficiency where the idle time for the longest micro-batch becomes often the bottleneck and converting this into performance gains by expanding the attention budget for non-bottleneck batches.
Critical weaknesses
  • The latency predictor (eq11) relies on a discrete fixed, offline lookup table and maybe specific hardware-to-hardware. Wondering how the performance will vary depending on the lookup table and how generalizable this lookup table approach may actually be.
  • The evaluation is done one relatively small models (under 3B), but the targeted scheme for this method was for large-scale distributed training. I'm curious how consistent the results would be on larger scale models.
  • While the proposed method is a interesting approach that co-designs system and theoretical algorithms, the SAB module is rather an incremental adaptation of length-based batching methods.
Questions for authors

Please address the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.