acceptodds
Under review as a conference paper at ICLR 2027

Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizers

Abstract

The scaling of Large Language Models (LLMs) drives interest in matrix-based optimizers (e.g., Shampoo, Muon, SOAP) for their convergence efficiency; yet their requirement for holistic updates conflicts with the tensor fragmentation in distributed frameworks like Megatron. Existing solutions are suboptimal: synchronous approaches suffer from computational redundancy, while layer-wise partitioning fails to reconcile this conflict without violating communication constraints. We propose Canzona, a Unified, Asynchronous, and Load-Balanced framework that decouples logical optimizer assignment from physical parameter distribution. For Data Parallelism, we introduce an alpha-Balanced Static Partitioning strategy that respects atomicity while neutralizing the load imbalance. For Tensor Parallelism, we design an Asynchronous-Compute pipeline utilizing Micro-Group-Scheduling to batch fragmented updates and hide reconstruction overhead. Experiments on the Qwen3 family (up to 32B parameters) on 256 GPUs demonstrate that our approach preserves the efficiency of established parallel architectures, achieving a 1.57x speedup in end-to-end iteration time and reducing optimizer latency by 5.8x.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
llms, optimizer, distributed training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_35rzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Matrix-based optimizers’ convergence efficiency is becoming increasingly important as Large Language Models (LLMs) are scaling. Existing solutions are suboptimal because of the redundant computation in synchronous approaches and communication constraints for layer-wise partitioning approaches. Thus, this paper proposes Canzona, a framework to decouple logical optimizer assignment from physical parameter distribution. Specifically, it assures the atomicity and keeps load balance by introducing an alpha-balanced static partitioning strategy for data parallelism, and it designed an asynchronous-compute pipeline to batch fragmented updates for tensor parallelism. Evaluations conducted with Qwen3 family models on 256 GPUs demonstrate the effectiveness of the approach.

Critical strengths
  • The framework presents a straightforward and intuitive paradigm by decoupling logical optimizer ownership from physical parameter layouts. The design is well-organized.
  • It provides an extensive appendix containing thorough supplementary evaluations.
Critical weaknesses
  • The approach relies on traditional methods such as LPT and greedy bin packing. Consequently, it is more like a combination and incremental contribution rather than offering algorithmic or distributed theory innovations.
  • The framework relies heavily on offline static planning for both DP and TP partitioning during model initialization. This static nature limits its adaptability under dynamic workloads, leading to runtime inefficiencies in those cases.
  • It is still somewhat unclear regarding the execution details of how fused All-to-All communication and micro-group scheduling interact with complex matrix computations. Specifically, it does not have a thorough discussion of low-level stream management and temporary buffer overheads during asynchronous updates.
  • The evaluation only uses the Qwen3 family models. The lack of architectural diversity limits the generality of the approach. It would be better to conduct more evaluations on various models rather than focus on only one model family.
  • The evaluation section omits the exact accelerator model and detailed cluster topology specifications under the justification of confidentiality. The hardware opacity significantly hinders the reproducibility and generalizability of the reported performance. Besides, the number of GPUs reported in the Abstract/Introduction (it says 256 GPUs used) and Evaluation section (it says 512 used) is inconsistent.
Questions for authors
  1. How does Canzona handle dynamic workloads, such as variable-length sequences, where computational loads and activation shapes vary significantly across training iterations?
  2. How well can this approach be applied to other LLM models such as Llama and Gemma? Could the static partitioning and micro-group scheduling strategies still work well for other model architectures with different tensor shape distributions?
  3. How many GPUs exactly have been used for evaluations? The Abstract section says it is 256, while the Evaluation section says it is 512.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 35rz (Part1)

Thank you for your questions and suggestions. We have added experiments with different sequence lengths and model architectures, and will clarify the execution details in the paper.

W and Q labels follow the order of the weaknesses and questions in your review.

W1. System contribution and the role of LPT

Canzona contributes an optimizer-agnostic execution design that jointly addresses complete matrix updates, bucket-compatible communication, and optimizer-load balance. LPT and greedy packing are established building blocks; the contribution lies in the ownership and execution mechanisms that make these requirements compatible under DP and TP sharding.

DP (Section 3.2). Applying unconstrained global LPT can interleave optimizer owners within a fixed bucket, breaking the contiguous rank intervals used by the evaluated Megatron layout. Retaining standard equal-size intervals can instead split matrices. Canzona reconciles these constraints by moving ownership boundaries to tensor boundaries while keeping physical parameter order fixed. Within this feasible layout, our α-balanced partitioner trades off global optimizer-load balance against per-bucket communication balance: it processes heavy buckets first and blends even allocation with filling underloaded ranks.

TP (Section 4.2). Assigning complete updates to hosts removes redundant matrix computation. However, balancing total assigned work alone does not ensure balance within each communication group: an overloaded host can delay group completion. Our scheduler therefore determines host assignments and micro-groups together. It uses global LPT with greedy packing and rollback to pursue within-group load balance, then group saturation under a capacity constraint on each host's scheduled cost. Gradient and update transfers are fused within each group.

The evaluation links these mechanisms to observed benefits: Figure 8 compares the resulting DP communication path with AdamW controls; ASC versus LB-ASC measures the additional benefit of load-aware scheduling; and Figure 15 measures the benefit of TP communication fusion. Together with the 1.57× end-to-end speedup over the evaluated NV-layerwise baseline in Figure 4, these results support the value of the combined system design.

AuthorsOpenReview ↗

Response to Reviewer 35rz (Part2)

W2 / Q1. Static planning and variable-length sequences

For a fixed dense model and parallel layout, sequence length changes activation shapes and forward/backward work, but leaves parameter matrices, gradient shapes, and optimizer-state dimensions unchanged. The same DP ownership and TP micro-group plan can therefore be reused even when sequence length varies from one iteration to the next.

We compared SC and LB-ASC in separate forward/backward runs at sequence lengths 512, 1024, and 2048, using a four-layer GPT with hidden size 512, FFN size 2048, and eight attention heads, Muon, and DP=4/TP=1 on four GPUs. Each run uses eight steps, BF16 model parameters, FP32 master parameters and state tensors, microbatch size 1, two gradient-accumulation steps, vocabulary size 32000, and mock data. After every step, we compare canonical FP32 master parameters and optimizer-state values by parameter identity.

Table 1. Numerical comparison across sequence lengths. For SC reference A and LB-ASC result B, absolute error is max_i |B_i − A_i| and relative L2 is ||B − A||₂ / ||A||₂. If ||A||₂ < 1e-30, relative error is zero when ||B||₂ ≤ 1e-30 and undefined otherwise. Values are maxima over compared parameter/state records and steps, using defined relative errors.

Sequence lengthStepsMax absolute differenceMax relative L2
51283.725e-92.39e-9
102483.725e-91.95e-9
204883.725e-91.73e-9

These small residual errors are consistent with floating-point roundoff from different DP gradient-reduction orders.

For latency, we use a Llama-3.2-1B configuration with 16 layers, hidden size 2048, FFN size 8192, and 32 query heads / 8 KV groups. It runs on four GPUs (DP=4/TP=1) with Muon, BF16 parameters and FP32 master weights, microbatch size 1, one gradient-accumulation step, and vocabulary size 128256.

Table 2. SC and LB-ASC latency across sequence lengths, in milliseconds. Timings are rank-0 wall-clock means over steps 3–12 of separate fixed-length runs. Iteration time covers forward/backward plus optimizer.step(), with CUDA synchronization after each interval; tensor dumps are disabled.

Sequence lengthSC optimizerLB-ASC optimizerSC iterationLB-ASC iterationIteration speedup
51260.123.3145.299.41.46×
102460.423.3144.4102.91.40×
204860.423.6153.8112.21.37×

Optimizer latency remains approximately constant across lengths, while LB-ASC achieves 1.37–1.46× end-to-end speedups.

AuthorsOpenReview ↗

Response to Reviewer 35rz (Part3)

W4 / Q2. Llama, Gemma, and architectural diversity

The same scheduler operates on matrix shapes, tensor boundaries, cost estimates, and sharding metadata. We applied it to two additional configurations with different matrix-shape distributions, using the same partitioning and micro-group scheduling mechanisms.

Table 3. Additional architecture configurations. Names indicate the model-family dimensions used in the Megatron configurations.

ConfigurationLayersHidden sizeFFN sizeQuery heads / KV groups
Llama-3.2-1B162048819232 / 8
Gemma-2-2B26230492168 / 4

Both configurations use RoPE, RMSNorm, untied embeddings, and no linear biases or learned positional embeddings. The Llama-3.2-1B configuration uses SwiGLU; the Gemma-2-2B configuration uses head dimension 256 and the available gated activation in place of GeGLU.

On eight GPUs (DP=4/TP=2), six-step Muon comparisons between SC and LB-ASC use identical initial parameters, optimizer states, and fixed gradients. Checks of representative attention/MLP layers and final normalization parameters report zero numerical differences in the saved FP32 parameters and optimizer states: 708/708 comparison records for Llama-3.2-1B and 876/876 for Gemma-2-2B. These checks exercise different tensor shapes and TP host assignments.

The latency experiments use eight GPUs (DP=4/TP=2), sequence length 1024, Muon, BF16 parameters with FP32 master weights, microbatch size 1, two gradient-accumulation steps, and 400000000-element buckets. Vocabulary sizes are 128256 and 256000, respectively. ASC and LB-ASC enable DP gradient-reduction and parameter-gather overlap. Timings are rank-0 wall-clock means over steps 3–12, covering forward/backward plus the optimizer step, with CUDA synchronization after each interval and tensor dumps disabled.

Table 4. Eight-GPU architecture comparison using the default numel scheduling metric. Times are in milliseconds; the FLOPs ratio is maximum over average scheduled rank load.

ConfigurationPolicyFLOPs max/avgOptimizerIteration
Llama-3.2-1BSC—51.4209.5
Llama-3.2-1BASC1.1417.1175.3
Llama-3.2-1BLB-ASC1.0314.3172.3
Gemma-2-2BSC—108.2360.5
Gemma-2-2BASC1.1333.6319.8
Gemma-2-2BLB-ASC1.0528.4287.6

Optimizer latency improves by 3.59× and 3.81× over SC, and 1.20× and 1.18× over ASC, for Llama-3.2-1B and Gemma-2-2B, respectively. The corresponding iteration speedups over SC are 1.22× and 1.25×. The FLOPs max/avg ratio falls from 1.14 to 1.03 and from 1.13 to 1.05. Together, these numerical and runtime results extend the evaluation beyond Qwen3 tensor shapes.

Reviewer_UiHoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a practical conflict between matrix-based optimizers and distributed training layouts. Muon, Shampoo, and SOAP require tensor-level information for their matrix updates, whereas ZeRO-1 data parallelism partitions optimizer states and gradient buffers and tensor parallelism splits individual parameter matrices. Canzona separates the rank responsible for an optimizer update from the physical placement of the corresponding parameter. On the data-parallel side, it uses tensor-aligned, load-aware bucket partitions intended to retain coalesced Reduce-Scatter and All-Gather communication. On the tensor-parallel side, it assigns complete matrix-update tasks to host ranks and uses fused micro-groups to route gradient and update shards while balancing work. The evaluation covers Qwen3 models up to 32B parameters and experiments with Muon, Shampoo, and SOAP. In the 256-GPU Qwen3-32B comparison, the paper reports a reduction in iteration time from 1.381 s to 0.877 s and in Muon step time from 0.383 s to 0.066 s relative to the NVIDIA layer-wise baseline (Figure 4).

Critical strengths
  1. The paper addresses a concrete systems conflict rather than changing the optimizer update rule. A matrix update needs a complete tensor, while the physical ZeRO-1 and TP layouts are designed around partitioned storage and communication. Canzona's separate DP and TP mechanisms are responsive to these different constraints. This is the main reason I find the framework useful beyond a Muon-specific implementation.

  2. The performance argument has meaningful supporting controls. Figure 4 reports both end-to-end and optimizer-step improvements. In Figure 8, the layer-wise baseline closely follows the AdamW All-Reduce control across the tested model sizes, while Canzona is close to the AdamW Reduce-Scatter control. Figure 16 gives a plausible explanation for this difference under the Megatron bucket layout considered in the paper. Together, these results support the central claim that preserving bucket-compatible ownership matters for training throughput.

Critical weaknesses
  1. The evidence for exact optimizer behavior is narrower than the paper's wording. The overlapping Muon, Shampoo, and SOAP loss curves in Figure 5 and Figures 11–12 are reassuring, but they do not establish step-by-step numerical agreement of parameter updates or optimizer states. Phrases such as “exact convergence behavior” and “zero mathematical precision loss” should be supported by direct update/state comparisons or narrowed to what the reported loss trajectories demonstrate. This matters because preserving the intended mathematical update and observing identical floating-point states are distinct claims.

  2. Important details of the communication path and baseline are missing. Section 3.3 does not explain how the variable-size Reduce-Scatter and All-Gather operations are realized in the evaluated implementation. Appendix C.1 also omits the accelerator model, and the exact NV-layerwise revision and relevant configuration are not specified. Because the contribution depends on communication volume, launch overhead, and overlap, these details are consequential for interpreting and reproducing the reported latency differences.

Questions for authors
  1. Can you compare SC and LB-ASC over a fixed sequence of steps at the level of parameter updates and optimizer states, for Muon and at least one of Shampoo or SOAP? Reporting absolute and relative differences across representative tensors would help distinguish mathematical-update equivalence from the loss-curve evidence in Figure 5 and Figures 11–12.

  2. How are the variable-size Reduce-Scatter and All-Gather operations in Section 3.3 implemented? Please specify the underlying communication operations, any padding or decomposition, and the resulting bytes transferred and launch counts relative to the equal-shard ZeRO-1 path.

  3. What exact code revision, patches, and communication settings were used for NV-layerwise in Figures 4, 7, and 8? Please also identify the accelerator and interconnect configuration and state the DP/TP topology used for Figure 8 in its caption.

  4. Is there a profiler timeline or equivalent measurement showing overlap between TP gradient gathering, matrix-update computation, update scattering, and adjacent micro-groups? Figure 15 establishes a benefit from fusion, but the extent of asynchronous overlap is less directly visible.

4 comments
AuthorsOpenReview ↗

Response to Reviewer UiHo (Part1)

Thank you for these specific questions. We have added stepwise parameter/state comparisons and communication measurements.

W and Q labels follow the order of the weaknesses and questions in your review.

Here, SC denotes synchronous redundant execution, ASC denotes decoupled execution without load-aware scheduling, and LB-ASC is the full Canzona design.

W1 / Q1. Parameter updates and optimizer states

In fixed-gradient replay, we compare SC and LB-ASC with identical initial parameters, optimizer states, gradient sequences, precision, hyperparameters, and matrix kernels. Input checks verify identical initial parameters and post-reduction gradients.

After every step, we compare canonical FP32 master parameters and optimizer-state values by parameter identity. Muon comparisons include momentum buffers. SOAP comparisons include first and second moments, preconditioner statistics GG_0/GG_1, eigenbases Q_0/Q_1, and the step counter. The comparisons also include Adam first and second moments for word embeddings, biases, and normalization parameters.

Table 1. Stepwise parameter and state comparison. The four-layer GPT has hidden size 512, FFN size 2048, and eight attention heads, with BF16 model parameters and FP32 master weights.

Optimizer and inputDP / TPGPUsStepsMax absolute differenceMax relative L2
Muon, fixed gradients4 / 14120.00.0
SOAP, fixed gradients4 / 14120.00.0
Muon, fixed gradients2 / 24120.00.0
Muon, real forward/backward4 / 1467.45e-93.23e-9

Representative layer-0 tensors are self_attention.linear_qkv.weight (1536 × 512), self_attention.linear_proj.weight (512 × 512), mlp.linear_fc1.weight (2048 × 512), and mlp.linear_fc2.weight (512 × 2048). Their compared parameters and states have zero absolute and relative-L2 differences in the fixed-gradient runs. SOAP covers initialization, ordinary updates, and a periodic eigenbasis refresh; the TP=2 run additionally exercises reconstruction and host-assigned computation.

For SC reference A and LB-ASC result B, absolute error is max_i |B_i − A_i| and relative L2 is ||B − A||₂ / ||A||₂. If ||A||₂ < 1e-30, relative error is zero when ||B||₂ ≤ 1e-30 and undefined otherwise. The table reports maxima over compared parameter/state records and steps, using defined relative errors.

These checks provide direct evidence beyond the original loss curves. The small nonzero differences observed in real forward/backward runs are consistent with floating-point roundoff from different DP gradient-reduction orders. Under the tested settings, fixed-gradient replay isolates optimizer execution and yields zero differences in the compared post-update parameters and states. We will revise the precision wording to distinguish preservation of the tensor-level update rule from floating-point agreement.

AuthorsOpenReview ↗

Response to Reviewer UiHo (Part2)

W2 / Q2. Implementation of variable-size Reduce-Scatter and All-Gather

The paper's experiments use our own uneven Reduce-Scatter and All-Gather implementation as the default path for unequal DP buckets. It operates on the planned contiguous shard intervals without padding them to the largest shard:

  • Unequal buckets (uneven): one rooted reduction per destination interval for Reduce-Scatter, and one rooted broadcast per source interval for All-Gather. An R-rank group issues R rooted collectives per bucket and direction, within an outer coalescing block.
  • Equal buckets: one native reduce_scatter_tensor or all_gather_into_tensor call per bucket and direction.
  • Padded comparison: pad each shard to the largest length, use a native equal-size collective, then trim/copy back to the planned intervals.

For shard lengths m_r, useful elements M = Σ_r m_r, largest shard L = max_r m_r, and element size b, the ideal aggregate payload per direction is (R − 1) M b without padding, versus (R − 1) R L b for the padded path.

Table 2. FP32 communication microbenchmarks. Each equal shard contains 2^20 elements. Uneven shards use 2^20 × [1, 1.25, 1.5, 1] elements for R=4 and 2^20 × [1, 1.25, 1.5, 1.75, 2, 2.25, 2.5, 1] for R=8.

RPathc10d calls per bucket and directionModelled aggregate payload per direction (MiB)RS time (ms)AG time (ms)
4Equal148.00.0800.125
4Uneven457.00.1240.123
4Padded172.00.2810.493
8Equal1224.00.1890.168
8Uneven8371.00.2620.171
8Padded1560.00.3410.454

After three warm-up operations, CUDA events time 50 complete bucket operations, followed by synchronization; times are rank-0 means and include the applicable packing, padding, trimming, and copies. Payloads are modelled aggregate traffic, with 1 MiB = 2^20 bytes. Counts refer to c10d calls, not GPU kernel launches, which were not measured.

For the same unequal input, padding increases modelled traffic by 1.26× at R=4 and 1.51× at R=8, and is slower in these measurements. Equal-shard timings provide a native reference at the stated input sizes; the equal and unequal cases have different total useful volumes.

W2 / Q3. Baseline revision, hardware disclosure, and Figure 8 topology

We used NVIDIA/Megatron-LM commit 5548cdc57bb68c0e3d3465eb1278e59d246e9d52 for the NV-layerwise baseline, with DP gradient-reduction overlap enabled and a bucket-size setting of 400M.

Figure 8 uses DP=16 and TP=8, for a total of 128 GPUs. We will add this configuration and the evaluated baseline revision to the experimental description.

Our institution's disclosure requirements prevent us from releasing hardware details. We will open-source our Megatron and FSDP implementations upon acceptance, together with experiment configurations and reproduction scripts, so that the community can evaluate Canzona on their own systems.

AuthorsOpenReview ↗

Response to Reviewer UiHo (Part3)

Q4. TP execution, overlap, and temporary memory

The evaluated TP implementation executes micro-groups in gather → compute → scatter → apply order. Hosts compute different complete matrix updates, and fused All-to-All operations route gradient and update shards. Its benefits come from removing redundant matrix computation, balancing host workloads, and fusing communication; Figure 15 measures the fusion benefit. There is no communication–computation overlap across TP micro-groups in this implementation.

Communication–computation overlap is implemented in DP, where gradient Reduce-Scatter and parameter All-Gather overlap with backward and forward computation, respectively. Figure 8 provides the corresponding communication controls. We will revise the TP pipeline wording to reflect the actual execution order and specify that zero additional optimizer-step communication applies to DP.

We also measured memory around optimizer.step() for a Llama-3.2-1B configuration with 16 layers, hidden size 2048, FFN size 8192, and 32 query heads / 8 KV groups. The workload uses eight GPUs (DP=4/TP=2), Muon, BF16 parameters with FP32 master weights, sequence length 1024, microbatch size 1, two gradient-accumulation steps, and 400M-element buckets.

Table 3. PyTorch allocated memory around the optimizer step, in GiB. We first average the measurements over steps 3–12 on each rank. “Peak, max” is then the maximum of those per-rank mean peaks; the temporary increment is peak minus pre-step allocation.

PolicyPre-step allocation, rank averagePeak, maxTemporary increment, range across ranks
SC10.8011.980.43–0.74
ASC5.886.800.43–0.81
LB-ASC5.886.790.43–0.79

The mean pre-step footprint decreases from 10.80 to 5.88 GiB, a 1.84× reduction, while the maximum per-rank mean peak decreases from 11.98 to 6.79 GiB. These are total allocations visible to PyTorch at the stated measurement boundaries.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer UiHo, thank you again for your constructive feedback. We hope the added Muon/SOAP parameter and state comparisons, together with the communication and TP execution clarifications, address your concerns. If so, we would be grateful if you could consider updating your rating accordingly at your earliest convenience. We would be happy to clarify any remaining questions.

Reviewer_2wCaOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors describe the implementation and performance of a load-balancing framework, Canzona, that decouples logical optimizer assignment from physical parameter distribution. Canzona addresses the data locality issues surrounding matrix-based optimizers, such as Muon, in the presence of data-parallelism and tensor-parallelism decomposition across multiple data-locality that necessitate the use of either redundant work or communication to compute entire matrix updates. Canzona addresses these issues by introducing constraints to control the locality of data to ranks conducive to different communication patterns and uses a load-balance optimization process based on \alpha-Balanced Greedy LPT to minimize both the number of stragglers and the amount of idle time between updates. The experimental results for Canzona on a 256-GPU cluster show that the performance is significantly better than alternative formulations.

Critical strengths
  • Both the motivational and experimental aspects of the work are excellent. As someone familiar with matrix-based optimizers and data locality issues in large-scale distributed training, I found the motivation clearly described and the experimental results excellent.
  • Comparison with prior work is well-posed, and the authors do an excellent job positioning Canzona as a means to address the shortcomings of existing methods by addressing the correct limiting design decisions of the data distribution across GPU memories.
  • The diagrams go a long way as visual aids to describe the data movement issues described in the text and allows the reader to grasp the key concepts.
Critical weaknesses
  • The main weakness, for me, is that the results would be nearly impossible for me to verify or reproduce.
  • The method has almost no applicability to the normal issues encountered on small to mid-range training clusters. It does not address concerns about the efficiency of matrix-based optimizers at smaller scales.
Questions for authors

Other than a cost analysis of running such large-scale experiments I do not have additional questions for the authors. This was an interesting paper, and I learned a great deal reading it. Thanks!

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2wCa

Thank you for your positive assessment of our work. We have added small-scale experiments addressing your concerns about applicability and reproducibility.

W and Q labels follow the order of the weaknesses and questions in your review.

W2. Applicability to smaller clusters

We added four- and eight-GPU Megatron experiments comparing the synchronous redundant-compute baseline (SC) with the full Canzona design (LB-ASC). The results show 1.22–1.46× iteration speedups at these smaller scales.

Small-scale iteration latency. Times are in milliseconds; speedup is SC time divided by LB-ASC time.

ConfigurationGPUs (DP/TP)Sequence lengthSCLB-ASCSpeedup
Llama-3.2-1B4 (4/1)512145.299.41.46×
Llama-3.2-1B4 (4/1)1024144.4102.91.40×
Llama-3.2-1B4 (4/1)2048153.8112.21.37×
Llama-3.2-1B8 (4/2)1024209.5172.31.22×
Gemma-2-2B-based8 (4/2)1024360.5287.61.25×

The Llama-3.2-1B configuration uses 16 layers, hidden size 2048, and FFN size 8192; the Gemma-2-2B-based configuration uses 26 layers, hidden size 2304, and FFN size 9216. All runs use Muon, BF16 model parameters with FP32 master weights, and microbatch size 1; gradient accumulation is one step on four GPUs and two steps on eight GPUs. Timings are rank-0 wall-clock means over steps 3–12, covering forward/backward plus the optimizer step, with CUDA synchronization after each interval.

W1. Reproducibility

We will release the Megatron implementation and our FSDP implementation upon acceptance, together with evaluated versions, per-figure configurations, dependencies, timing logs, and numerical-comparison scripts. The four- and eight-GPU examples let researchers test Canzona on smaller clusters.

Q1. GPU cost per iteration

The Figure 4 timings correspond to 0.0624 GPU-hours per iteration for Canzona (256 × 0.877 / 3600), versus 0.0982 GPU-hours for NV-layerwise (256 × 1.381 / 3600). This is 36.5% less GPU time per timed iteration at the same allocation size.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer 2wCa, thank you again for your positive assessment and thoughtful feedback. Our rebuttal includes new four- and eight-GPU experiments, a GPU-cost analysis, and our code-release plan. We would appreciate a brief follow-up at your earliest convenience on whether these additions address your concerns about small-cluster applicability and reproducibility. We would be happy to clarify any remaining points.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal for Submission 27253 — Canzona

Dear Area Chair,

Thank you for your time and for coordinating the review of our paper. We would like to summarize the initial assessments and the additional evidence provided in our rebuttal.

The initial recommendations were two acceptance recommendations and one weak rejection:

ReviewerScoreRecommendation
2wCa4Clear acceptance
UiHo3Weak acceptance
35rz2Weak rejection

Reviewers highlighted Canzona's practical motivation, its decoupling of optimizer ownership from parameter placement, and its extensive evaluation. The main experiments demonstrate 1.57× end-to-end and 5.8× optimizer-step speedups over the evaluated NVIDIA layer-wise baseline. Our rebuttal strengthens this evidence through the following additions:

  1. Direct numerical verification (UiHo). We added stepwise comparisons of FP32 parameters and optimizer states for Muon and SOAP. Fixed-gradient replay produces zero differences in the compared parameters and states, including SOAP's preconditioner statistics and eigenbases through a refresh, as well as a Muon TP=2 configuration. Real forward/backward Muon runs show only roundoff-scale differences, complementing the original training-loss evidence.

  2. Small-scale benefits and broader architectures (2wCa, 35rz). New experiments on 4 and 8 GPUs, using Llama-3.2-1B and Gemma-2-2B-based configurations, achieve 1.22-1.46× end-to-end speedups over the synchronous redundant-compute baseline (SC). Eight-GPU fixed-gradient checks also yield zero differences in the compared parameters and states for both configurations, extending validation beyond Qwen3 tensor shapes.

  3. Applicability across sequence lengths (35rz). We explained why changing sequence length in a fixed dense model preserves the parameter and optimizer-state shapes on which scheduling depends. Experiments at lengths 512, 1024, and 2048 show approximately constant optimizer latency and 1.37-1.46× iteration speedups on the four-GPU Llama configuration, supporting reuse of the static plan.

  4. Systems contribution and implementation details (UiHo, 35rz). We clarified how Canzona jointly addresses tensor atomicity, bucket-compatible communication, and optimizer-load balance through its DP and TP formulations. We detailed the complementary roles of DP communication–computation overlap and TP task distribution and fused transfers, added uneven-collective microbenchmarks and optimizer-step memory measurements, and specified the evaluated baseline revision and key settings. We also clarified the allocation of 256 GPUs for the headline comparison, 128 for Figure 8, and up to 512 for scaling. We remain committed to releasing our Megatron and FSDP implementations, configurations, and reproduction scripts upon acceptance.

As of this writing, we have not received follow-up comments from any of the three reviewers after our rebuttal. We had hoped for further discussion and remain very eager to answer additional questions and provide any clarification that would assist the assessment. We respectfully ask that the additional experimental evidence and detailed responses be considered alongside the initial reviews in the final decision.

Thank you for your consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.