- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Matrix-based optimizers’ convergence efficiency is becoming increasingly important as Large Language Models (LLMs) are scaling. Existing solutions are suboptimal because of the redundant computation in synchronous approaches and communication constraints for layer-wise partitioning approaches. Thus, this paper proposes Canzona, a framework to decouple logical optimizer assignment from physical parameter distribution. Specifically, it assures the atomicity and keeps load balance by introducing an alpha-balanced static partitioning strategy for data parallelism, and it designed an asynchronous-compute pipeline to batch fragmented updates for tensor parallelism. Evaluations conducted with Qwen3 family models on 256 GPUs demonstrate the effectiveness of the approach.
Critical strengths
- The framework presents a straightforward and intuitive paradigm by decoupling logical optimizer ownership from physical parameter layouts. The design is well-organized.
- It provides an extensive appendix containing thorough supplementary evaluations.
Critical weaknesses
- The approach relies on traditional methods such as LPT and greedy bin packing. Consequently, it is more like a combination and incremental contribution rather than offering algorithmic or distributed theory innovations.
- The framework relies heavily on offline static planning for both DP and TP partitioning during model initialization. This static nature limits its adaptability under dynamic workloads, leading to runtime inefficiencies in those cases.
- It is still somewhat unclear regarding the execution details of how fused All-to-All communication and micro-group scheduling interact with complex matrix computations. Specifically, it does not have a thorough discussion of low-level stream management and temporary buffer overheads during asynchronous updates.
- The evaluation only uses the Qwen3 family models. The lack of architectural diversity limits the generality of the approach. It would be better to conduct more evaluations on various models rather than focus on only one model family.
- The evaluation section omits the exact accelerator model and detailed cluster topology specifications under the justification of confidentiality. The hardware opacity significantly hinders the reproducibility and generalizability of the reported performance. Besides, the number of GPUs reported in the Abstract/Introduction (it says 256 GPUs used) and Evaluation section (it says 512 used) is inconsistent.
Questions for authors
- How does Canzona handle dynamic workloads, such as variable-length sequences, where computational loads and activation shapes vary significantly across training iterations?
- How well can this approach be applied to other LLM models such as Llama and Gemma? Could the static partitioning and micro-group scheduling strategies still work well for other model architectures with different tensor shape distributions?
- How many GPUs exactly have been used for evaluations? The Abstract section says it is 256, while the Evaluation section says it is 512.
5 comments
Response to Reviewer 35rz (Part2)
W2 / Q1. Static planning and variable-length sequences
For a fixed dense model and parallel layout, sequence length changes activation shapes and forward/backward work, but leaves parameter matrices, gradient shapes, and optimizer-state dimensions unchanged. The same DP ownership and TP micro-group plan can therefore be reused even when sequence length varies from one iteration to the next.
We compared SC and LB-ASC in separate forward/backward runs at sequence lengths 512, 1024, and 2048, using a four-layer GPT with hidden size 512, FFN size 2048, and eight attention heads, Muon, and DP=4/TP=1 on four GPUs. Each run uses eight steps, BF16 model parameters, FP32 master parameters and state tensors, microbatch size 1, two gradient-accumulation steps, vocabulary size 32000, and mock data. After every step, we compare canonical FP32 master parameters and optimizer-state values by parameter identity.
Table 1. Numerical comparison across sequence lengths. For SC reference A and LB-ASC result B, absolute error is max_i |B_i − A_i| and relative L2 is ||B − A||₂ / ||A||₂. If ||A||₂ < 1e-30, relative error is zero when ||B||₂ ≤ 1e-30 and undefined otherwise. Values are maxima over compared parameter/state records and steps, using defined relative errors.
| Sequence length | Steps | Max absolute difference | Max relative L2 |
|---|---|---|---|
| 512 | 8 | 3.725e-9 | 2.39e-9 |
| 1024 | 8 | 3.725e-9 | 1.95e-9 |
| 2048 | 8 | 3.725e-9 | 1.73e-9 |
These small residual errors are consistent with floating-point roundoff from different DP gradient-reduction orders.
For latency, we use a Llama-3.2-1B configuration with 16 layers, hidden size 2048, FFN size 8192, and 32 query heads / 8 KV groups. It runs on four GPUs (DP=4/TP=1) with Muon, BF16 parameters and FP32 master weights, microbatch size 1, one gradient-accumulation step, and vocabulary size 128256.
Table 2. SC and LB-ASC latency across sequence lengths, in milliseconds. Timings are rank-0 wall-clock means over steps 3–12 of separate fixed-length runs. Iteration time covers forward/backward plus optimizer.step(), with CUDA synchronization after each interval; tensor dumps are disabled.
| Sequence length | SC optimizer | LB-ASC optimizer | SC iteration | LB-ASC iteration | Iteration speedup |
|---|---|---|---|---|---|
| 512 | 60.1 | 23.3 | 145.2 | 99.4 | 1.46× |
| 1024 | 60.4 | 23.3 | 144.4 | 102.9 | 1.40× |
| 2048 | 60.4 | 23.6 | 153.8 | 112.2 | 1.37× |
Optimizer latency remains approximately constant across lengths, while LB-ASC achieves 1.37–1.46× end-to-end speedups.
Response to Reviewer 35rz (Part3)
W4 / Q2. Llama, Gemma, and architectural diversity
The same scheduler operates on matrix shapes, tensor boundaries, cost estimates, and sharding metadata. We applied it to two additional configurations with different matrix-shape distributions, using the same partitioning and micro-group scheduling mechanisms.
Table 3. Additional architecture configurations. Names indicate the model-family dimensions used in the Megatron configurations.
| Configuration | Layers | Hidden size | FFN size | Query heads / KV groups |
|---|---|---|---|---|
| Llama-3.2-1B | 16 | 2048 | 8192 | 32 / 8 |
| Gemma-2-2B | 26 | 2304 | 9216 | 8 / 4 |
Both configurations use RoPE, RMSNorm, untied embeddings, and no linear biases or learned positional embeddings. The Llama-3.2-1B configuration uses SwiGLU; the Gemma-2-2B configuration uses head dimension 256 and the available gated activation in place of GeGLU.
On eight GPUs (DP=4/TP=2), six-step Muon comparisons between SC and LB-ASC use identical initial parameters, optimizer states, and fixed gradients. Checks of representative attention/MLP layers and final normalization parameters report zero numerical differences in the saved FP32 parameters and optimizer states: 708/708 comparison records for Llama-3.2-1B and 876/876 for Gemma-2-2B. These checks exercise different tensor shapes and TP host assignments.
The latency experiments use eight GPUs (DP=4/TP=2), sequence length 1024, Muon, BF16 parameters with FP32 master weights, microbatch size 1, two gradient-accumulation steps, and 400000000-element buckets. Vocabulary sizes are 128256 and 256000, respectively. ASC and LB-ASC enable DP gradient-reduction and parameter-gather overlap. Timings are rank-0 wall-clock means over steps 3–12, covering forward/backward plus the optimizer step, with CUDA synchronization after each interval and tensor dumps disabled.
Table 4. Eight-GPU architecture comparison using the default numel scheduling metric. Times are in milliseconds; the FLOPs ratio is maximum over average scheduled rank load.
| Configuration | Policy | FLOPs max/avg | Optimizer | Iteration |
|---|---|---|---|---|
| Llama-3.2-1B | SC | — | 51.4 | 209.5 |
| Llama-3.2-1B | ASC | 1.14 | 17.1 | 175.3 |
| Llama-3.2-1B | LB-ASC | 1.03 | 14.3 | 172.3 |
| Gemma-2-2B | SC | — | 108.2 | 360.5 |
| Gemma-2-2B | ASC | 1.13 | 33.6 | 319.8 |
| Gemma-2-2B | LB-ASC | 1.05 | 28.4 | 287.6 |
Optimizer latency improves by 3.59× and 3.81× over SC, and 1.20× and 1.18× over ASC, for Llama-3.2-1B and Gemma-2-2B, respectively. The corresponding iteration speedups over SC are 1.22× and 1.25×. The FLOPs max/avg ratio falls from 1.14 to 1.03 and from 1.13 to 1.05. Together, these numerical and runtime results extend the evaluation beyond Qwen3 tensor shapes.
Response to Reviewer 35rz (Part1)
Thank you for your questions and suggestions. We have added experiments with different sequence lengths and model architectures, and will clarify the execution details in the paper.
W and Q labels follow the order of the weaknesses and questions in your review.
W1. System contribution and the role of LPT
Canzona contributes an optimizer-agnostic execution design that jointly addresses complete matrix updates, bucket-compatible communication, and optimizer-load balance. LPT and greedy packing are established building blocks; the contribution lies in the ownership and execution mechanisms that make these requirements compatible under DP and TP sharding.
DP (Section 3.2). Applying unconstrained global LPT can interleave optimizer owners within a fixed bucket, breaking the contiguous rank intervals used by the evaluated Megatron layout. Retaining standard equal-size intervals can instead split matrices. Canzona reconciles these constraints by moving ownership boundaries to tensor boundaries while keeping physical parameter order fixed. Within this feasible layout, our α-balanced partitioner trades off global optimizer-load balance against per-bucket communication balance: it processes heavy buckets first and blends even allocation with filling underloaded ranks.
TP (Section 4.2). Assigning complete updates to hosts removes redundant matrix computation. However, balancing total assigned work alone does not ensure balance within each communication group: an overloaded host can delay group completion. Our scheduler therefore determines host assignments and micro-groups together. It uses global LPT with greedy packing and rollback to pursue within-group load balance, then group saturation under a capacity constraint on each host's scheduled cost. Gradient and update transfers are fused within each group.
The evaluation links these mechanisms to observed benefits: Figure 8 compares the resulting DP communication path with AdamW controls; ASC versus LB-ASC measures the additional benefit of load-aware scheduling; and Figure 15 measures the benefit of TP communication fusion. Together with the 1.57× end-to-end speedup over the evaluated NV-layerwise baseline in Figure 4, these results support the value of the combined system design.