acceptodds
Under review as a conference paper at ICLR 2027

Kernel-Level Context Parallelism: Bridging Incompatible Shard Layouts in Hybrid Linear-Attention Transformers

Abstract

Hybrid transformers that interleave softmax attention with linear attention now reach production scale, but under context parallelism (CP) their two layer families require incompatible sequence layouts. Fused softmax-attention CP kernels shard each sequence in a zigzag order to balance causal work, whereas state-passing linear-attention kernels require each rank to own a contiguous span. A common workaround all-gathers the full hidden state at every linear-attention layer and recomputes the layer redundantly on all ranks, so its activations and linear-attention compute stay proportional to the full sequence however large grows. At 256K tokens with , this baseline exhausts an 80 GB H100 before completing a single step. We present KCP, which instead converts between the two layouts with an exactly invertible token permutation. KCP keeps zigzag as the global layout, converts to contiguous order at the entry of each linear-attention layer with a single all-to-all of tokens per rank, runs the unmodified native CP kernel, and converts back on exit. The payload of each collective drops by ; more importantly, per-rank linear-attention compute and activation memory also drop by , because no rank recomputes the full sequence. Because the conversion is a pure permutation, its backward pass is the reverse permutation. On a 35B-parameter MoE hybrid across 32 H100 GPUs, KCP achieves 2.5–4.7× the training throughput of the all-gather baseline at every configuration both can run, and it trains the 256K-token configuration at 53.3 GB peak allocated memory. Training losses agree with the baseline up to floating-point nondeterminism, and the layout conversion accounts for 1.8–2.7% of GPU-busy time across profiled ranks. On a dense hybrid with small bins, where the gather is not the bottleneck, the two methods are within ±7% at three of four points, consistent with gains that come from eliminating full-sequence materialization rather than from a faster kernel.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM, Context Parallelism, Long-Context Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Cw34OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses incompatible sequence layouts in hybrid transformers under context parallelism. Fused causal softmax-attention kernels use zigzag sharding, whereas native state-passing linear-attention kernels require contiguous spans. KCP converts between these layouts using an invertible token redistribution before and after each linear-attention block, avoiding full-sequence gathering and redundant computation while preserving the existing kernels.

Critical strengths

S1. KCP resolves a concrete interface mismatch without modifying either attention kernel and without imposing a head-count divisibility constraint.

S2. The paired experiments process identical tokens and demonstrate substantial throughput gains and a configuration enabled by lower memory.

Critical weaknesses

Limited baseline comparison. AllGather is a relevant and controlled baseline, but redundantly evaluates the full sequence on each rank. The paper discusses head sharding and zigzag-aware state passing without measuring an alternative or concretely establishing where its constraints bind in the evaluated configurations.

Questions for authors

Q1. Under the actual head counts and tensor-parallel settings used in the MoE experiments, which nonredundant alternatives are feasible? Can the authors provide one matched head-sharding comparison or specify the configurations that its divisibility constraints exclude?

Q2. Is “the gather is not the bottleneck” supported by profiling of the small-bin configurations, or could layout-conversion overhead offset the savings?

2 comments

Thank you for the careful review and the precise questions. We measured two non-redundant alternatives at the paper's main configuration and profiled the small-bin configuration.


[W1 & Q1]: Limited baseline comparison. Which non-redundant alternatives are feasible under the actual head counts and tensor-parallel settings? Can you provide a matched head-sharding comparison or specify the configurations that its divisibility constraint excludes?

We now provide both: the configuration that head sharding's divisibility constraint excludes (CP32, which follows from the head count) and a matched comparison at CP16, where head sharding is feasible (Headwise in Table A).

  • Feasibility. Head sharding needs the number of heads per TP rank to be divisible by the CP size. With GDN's 16 query/key heads and TP1 in all MoE experiments, it is feasible at CP4, CP8 and CP16 and excluded at CP32, since 16 is not divisible by 32; KCP trains CP32 (Table 9 of the paper). Conversion inside the GDN block (Megatron #3282) has no such constraint. Zigzag-aware state passing (FLA #1221) has none either, but it defines zigzag over the whole packed buffer rather than per sample, so our per-sample layout would still have to be redistributed.

  • Baseline comparison. We implemented head sharding (Headwise: the design of Megatron #2614, #2642 and #2645) and in-block conversion (PostProj) in the same training stack and measured each against KCP at the paper's main configuration (Qwen3.6-35B-A3B, TP1/PP2/EP8/CP16, 32 H100 GPUs). As shown in Table A, KCP is faster than head sharding in every matched pair and places no constraint on head counts. It is also faster than PostProj in every pair.

Table A. Throughput in tokens/s/GPU under the paper's protocol. Headwise and PostProj entries are means of three matched pairs (same weights and data), each with its own KCP control

Bin / CPAlternativeAlternative throughputMatched KCP throughputKCP / alternative
128K / 16AllGather (paper)1898814.66×
128K / 16Headwise4388832.01×
128K / 16PostProj7758821.14×
256K / 16AllGather (paper)OOM484—
256K / 16Headwise3464851.40×
256K / 16PostProj4584811.05×

[Q2]: Is “the gather is not the bottleneck” supported by profiling of the small-bin configurations, or could layout-conversion overhead offset the savings?

We have now profiled this configuration. Layout conversion does not offset the savings: it costs the same as the gather it replaces. What offsets them is communication inside the state-passing kernels. The profile is of Qwen3.5-9B at 8K/CP4 (TP2/PP1, eight H100 GPUs), the dense control of the paper; each entry is the union of a category's GPU intervals within a rank, averaged over two steps and all ranks.

GPU interval category s/rank/steps/rank/stepAllGatherKCP
Non-NCCL compute2.8291.444
NCCL activity3.5085.042
Total GPU-busy interval union7.6507.769
Layout-related collectives†1.3051.299
GDN convolution + delta-rule kernels: non-NCCL compute†0.5660.198
GDN convolution + delta-rule kernels: NCCL state/haloexchangestate/halo exchange†0.0000.982

† Sub-intervals of the rows above, not additional categories. Layout-related collectives are the CP gather and backward reduce-scatter for AllGather, and the layout conversions for KCP.

KCP halves non-NCCL compute, but NCCL activity grows by a similar amount, about two thirds of it state and halo exchange inside the state-passing kernels, which AllGather does not need. Total GPU-busy time therefore ends up about equal, in line with the near-parity in throughput (1.05×) that the paper reports for this configuration.

NCCL intervals include time spent waiting on peers and categories can overlap, so these figures locate the offset rather than give an additive breakdown, and they cover one configuration and two profiled steps.


We hope these clarifications and experiments address your concerns. If so, we would be grateful if you would consider adjusting your rating.

I appreciate the authors' response to my concerns. I believe the current score accurately reflects the paper's contribution and its current state. I will maintain my score.

Reviewer_EpuyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies context-parallel training of hybrid models with both full and linear attention. Full attention uses a zigzag layout, while the native linear-attention kernels used in the paper expect contiguous inputs. The baseline handles this mismatch by gathering the full sequence and running the entire linear-attention computation on every rank. KCP instead uses all-to-all to convert zigzag shards into contiguous shards, runs the existing linear-attention CP kernel, and converts the output back to zigzag. The paper evaluates throughput, memory usage, and loss agreement on dense and MoE hybrid models.

Critical strengths

The experimental section is clearly presented. The throughput, memory, and loss comparisons make it easy to understand the practical benefits over the chosen baseline, including a configuration that KCP can train while the baseline runs out of memory.

Critical weaknesses
  1. The method seems too straightforward to establish a clear methodological contribution. The central idea is to use all-to-all to switch between zigzag and contiguous layouts, then switch back after computation. It is too simple.
  2. The baseline is too limited. Gathering the full sequence and repeating the entire linear-attention computation on every rank is an expensive fallback. Open-source frameworks already offer non-replicating CP paths, so outperforming this fallback alone does not establish how competitive KCP is against an efficient implementation.
Questions for authors

Have you compared KCP with, or looked into its differences from, existing open-source implementations? In particular, Megatron has a zigzag↔contiguous all-to-all conversion, and FLA has direct support for zigzag inputs, avoiding activation-layout reshuffling while still exchanging states. Can these paths support your packed, per-sample zigzag setting? If not, what specific limitations prevent their use, and what does KCP add beyond them? I recognize that some of these implementations may be contemporaneous; clarifying the relationship would help me assess the contribution, rather than necessarily requiring new benchmarks during rebuttal.

2 comments

Thank you for the detailed and knowledgeable review. We added comparisons with two efficient open-source designs and clarify how KCP relates to the Megatron and FLA.


[W1]: The method seems too straightforward to establish a clear methodological contribution.

The mechanism is simple, by design. It leaves both attention kernels unmodified and constrains neither head counts nor the layout of full attention. The paper answers a question that each open-source path decides implicitly and none measures: where a hybrid model should convert its layout, and what the conversion must do to stay exact once sequences are packed. The findings are as following:

  1. The conversion is not a plain all-to-all. Under packing, per-sample zigzag and global-buffer zigzag assign different tokens to a rank (see Q1), so a layout-agnostic all-to-all would mix samples. KCP's cu_seqlens-aware plan handles this, padding included, and is proven to be a permutation (Proposition 1 in the paper).

  2. Placement decides the cost. Converting hidden states at the block boundary moves 4.0× less data than converting the projected tensors inside the block, as Megatron #3282 does, and is faster in every matched pair, as reported in W2.

  3. The gain comes from not replicating the linear-attention compute, not from a cheaper collective. In a new profile of the dense control (Qwen3.5-9B, 8K/CP4), the conversion costs the same as the gather it replaces (1.299 vs. 1.305 s per rank and step), while non-NCCL compute halves (2.829 → 1.444 s); at 8K the added state and halo exchange (0.982 s) offsets most of that, so the gains concentrate at long sequences (4.66× over AllGather at 128K).

  4. Converting less often adds little. Keeping the contiguous layout across each run of three GDN layers (KCP-Retain, implemented during the rebuttal) removes two thirds of the conversions and gains 3.5% at 128K, bounding what further layout retention can add.

Simplicity is what makes it adoptable: the gains come from where the conversion is placed and from making it exact under packing, not from new kernels, so KCP transfers to any stack that has these two kernel families.


[W2]: The baseline is too limited, and open-source frameworks already offer non-replicating CP paths.

We added two non-replicating baselines that follow these open-source paths, implemented in the same training stack and measured against KCP at the paper's main configuration (Qwen3.6-35B-A3B, TP1/PP2/EP8/CP16, 32 H100 GPUs):

  • Headwise: head sharding, the design of Megatron #2614, #2642 and #2645).
  • PostProj : the conversion placement of Megatron #3282). The projected q/k/v/z/b/a tensors are converted inside the GDN block, which moves 4.0× the data of KCP's conversion of hidden states at the block boundary; kernels and routing plans are the same.
Bin / CPAlternativeAlternative throughputMatched KCP throughputKCP / alternative
128K / 16AllGather (paper)1898814.66×
128K / 16Headwise4388832.01×
128K / 16PostProj7758821.14×
256K / 16AllGather (paper)OOM484—
256K / 16Headwise3464851.40×
256K / 16PostProj4584811.05×

As shown in the table above, KCP is faster than head sharding in every matched pair and places no constraint on head counts. It is also faster than PostProj.

[Q1-1]: How does KCP differ from existing open-source implementations (Megatron's zigzag↔contiguous conversion, FLA's direct zigzag support)? Can they support the packed, per-sample zigzag setting, and what does KCP add beyond them?

W2 reports the measured comparison; the relationship is as follows:

  • Megatron's conversion can support our setting. KCP differs from #3282 in where the conversion is placed.

    • #3282 per-sequence THD can express the packed ownership pattern, and #3282 applies them inside the GDN block, to the projected q/k/v/z/b/a tensors, whereas KCP converts hidden states at the block boundary.

    • PostProj (see W2) measures exactly this difference. Megatron's later layout plans [#6764](https://github.com/NVIDIA/Megatron−LM/pull/6764)/[#6776](https://github.com/NVIDIA/Megatron−LM/pull/6776)[\#6764](https://github.com/NVIDIA/Megatron-LM/pull/6764)/[\#6776](https://github.com/NVIDIA/Megatron-LM/pull/6776) place the conversion between layers, as KCP does. But KCP is independent from upstream choice (see Timing).

  • FLA's direct zigzag support does not fit our setting as it stands, because it defines zigzag differently. Our full-attention layers apply zigzag to each sequence separately per−samplezigzagper-sample zigzag. While FLA #1221 applies it once to the whole packed batch global−bufferzigzagglobal-buffer zigzag. The two differ as soon as several sequences are packed. For two eight-token sequences A and B on two ranks:

LayoutTokens on rank 0
Per-sample zigzag ourfull−attentionlayers;KCPinputour full-attention layers; KCP inputA0 A1, A6 A7, B0 B1, B6 B7
Global-buffer zigzag requiredbyFLA[#1221](https://github.com/fla−org/flash−linear−attention/pull/1221)required by FLA [\#1221](https://github.com/fla-org/flash-linear-attention/pull/1221)A0 A1 A2 A3, B4 B5 B6 B7
Contiguous requiredbyFLA#691;KCPoutputrequired by FLA \#691; KCP outputA0 A1 A2 A3 A4 A5 A6 A7

Rank 0 does not hold the tokens that #1221 requires (rows 1 and 2), so tokens would still have to move between ranks: here half of rank 0's tokens, as many as KCP moves (rows 1 and 3). Using #1221 would therefore not remove the redistribution, only change its target. This is a difference between two definitions of zigzag, not an inherent limitation of FLA.

What KCP adds beyond them.

  • A cheaper place to convert. At the block boundary only hidden states move, 4.0× less data than the projected tensors that #3282 converts, and both kernel families stay untouched.
  • Per-sample zigzag under packing. An exact cu_seqlens-aware plan, proven to be a permutation (Proposition 1), covers the setting that #1221 does not.
  • Measurements. Matched comparisons at production scale (see W1 and W2).

[Q1-2]: Timing

As you anticipated, several of these paths are contemporaneous with KCP (dates from the public pull requests). But KCP is independent work. Like Megatron #3282, it starts from the FLA #691, which the paper takes as an existing component. The conversion around them is our own code, written directly on PyTorch's all-to-all, with none of Megatron's layout-conversion utilities. We regard #3282 as concurrent work that chose a different placement, and we do not claim priority over it. The layout-plan and direct-zigzag paths appeared within two months of the submission deadline, and you noted that new benchmarks were not necessarily required. We nevertheless measured KCP against head sharding and against the placement of #3282(W2).


We hope these clarifications and experiments address your concerns. If so, we would be grateful if you would consider adjusting your rating.

Reviewer_7ueYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KCP, a layout-redistribution method that bridges the mismatch between the zigzag layout used by full-attention CP kernels and the contiguous layout required by native linear-attention CP kernels. Using all-to-all communication to redistribute tokens, KCP avoids the redundant full-sequence computation in the AllGather baseline. It achieves substantial speedups in practical training settings while maintaining closely matching losses over short training runs.

Critical strengths

I like the simplicity and clarity of the approach. It addresses the inefficiency of AllGather with a straightforward idea, supported by a mathematical correctness proof, practical implementation optimizations, and performance and feasibility evaluations on production-scale models.

Critical weaknesses

My main concern is the baseline comparison. AllGather is the only empirical baseline, and I view it primarily as a fallback strategy. Are there existing approaches in the community that address this layout mismatch more efficiently? A comparison with such approaches would help establish how KCP’s simplicity translates into competitive practical performance.

Questions for authors

Q1: Could you include comparisons with existing approaches beyond AllGather? Such comparisons would help establish whether KCP’s simple design also delivers competitive performance in practice.

Q2: For consecutive linear-attention layers, could the contiguous layout be maintained until the next full-attention layer, with conversion back to zigzag performed only at that point?

4 comments

Thank you for the careful review and the constructive suggestions. We added two efficient baselines and implemented cross-layer layout retention as you suggested. Our point-by-point response follows.


[W1 & Q1]: AllGather is the only baseline and is primarily a fallback. Could you compare with existing, more efficient approaches?

We measured two non-replicating approaches against KCP in the same Slime/FLA training framework. As shown in Table A, both alternatives are efficient, and KCP is faster than both.

  • Headwise headsharding,thedesignofMegatron[#2614](https://github.com/NVIDIA/Megatron−LM/pull/2614),[#2642](https://github.com/NVIDIA/Megatron−LM/pull/2642)and[#2645](https://github.com/NVIDIA/Megatron−LM/pull/2645)head sharding, the design of Megatron [\#2614](https://github.com/NVIDIA/Megatron-LM/pull/2614), [\#2642](https://github.com/NVIDIA/Megatron-LM/pull/2642) and [\#2645](https://github.com/NVIDIA/Megatron-LM/pull/2645): an all-to-all turns token shards into head shards, and each rank runs the standard kernel over full sequences for 1/P of the heads.

  • PostProj theconversionplacementofMegatron[#3282](https://github.com/NVIDIA/Megatron−LM/pull/3282)the conversion placement of Megatron [\#3282](https://github.com/NVIDIA/Megatron-LM/pull/3282): the projected q/k/v/z/b/a tensors are converted inside the GDN block, whereas KCP converts hidden states at the block boundary. Kernels and cached routing plans are identical, so only the placement differs.

Table A. Throughput in tokens/s/GPU under the paper's protocol (30 steps, measured over steps 5–29).

Bin / CPAlternativeAlternative throughputMatched KCP throughputKCP / alternative
128K / 16AllGather (paper)1898814.66×
128K / 16Headwise4388832.01×
128K / 16PostProj7758821.14×
256K / 16AllGather (paper)OOM484—
256K / 16Headwise3464851.40×
256K / 16PostProj4584811.05×

Overall, KCP remains the fastest at both sequence lengths, showing that layout redistribution is an efficient approach in its own right rather than only a better fallback.


[Q2]: Could the contiguous layout be kept across consecutive linear-attention layers and converted back to zigzag only before the next full-attention layer?

Yes. We implemented this as KCP-Retain: the layout is converted once at the entry of each run of three GDN layers and back to zigzag at its exit, before full attention. For the 40-layer [GDN, GDN, GDN, FA] × 10 model with aligned PP2 stages (a bridge is one layout conversion):

MetricKCP controlKCP-Retain
Bridges per model forward6020
Bridges per PP2-stage forward3010
128K throughput tokens/s/GPUtokens/s/GPU880911
Peak allocated memory GiBGiB36.636.6

KCP-Retain needs a third of the conversions and is faster than KCP in all three matched pairs (mean gain 3.5%), at the same peak memory and with losses bit-identical to KCP's. It applies the same KCP conversion once per run of GDN layers instead of once per layer, so your suggestion extends KCP rather than replaces it. This confirms that KCP conversion is cheap: KCP is efficient as proposed, and retention adds a further gain on top.


We hope these clarifications and experiments address your concerns. If so, we would be grateful if you would consider adjusting your rating.

Thank you for the additional experiments. They largely address my concern about the limited baseline comparison. However, I have a follow-up question: since PostProj (Megatron #3282) already uses a similar layout-redistribution approach, what is KCP’s main methodological contribution beyond it? I appreciate that the reported 5–14% speedup over PostProj can be valuable in practical training. However, my current understanding is that the improvements primarily come from engineering optimizations rather than a new algorithmic idea. Could you clarify this distinction?

Thank you for the follow-up question. Our previous note was brief on how PostProj was implemented, so we appreciate the chance to clarify this distinction.

In Table A, PostProj already used KCP's own cached O(mP)O(mP) routing plan and generalized padding rule. All kernels, collectives, and code optimizations were identical between the two runs.

Therefore, the 5–14% speedup over PostProj (and 18% with KCP-Retain, 911 vs. 775 tokens/s/GPU at 128K) does not come from engineering tuning. It comes from a design choice at the operator level: KCP converts the layout at the layer boundary (P−1∘F∘PP^{-1} \circ F \circ P, Equation 2), whereas PostProj converts inside the GDN module.


1. Why layer-boundary conversion is a structural improvement over PostProj

  • 4× smaller communication volume. Following Megatron's earlier head-sharding CP (#2614), PostProj (#3282) converts tensors inside each GDN block. It redistributes the expanded (q,k,v,z,β,α)(q, k, v, z, \beta, \alpha) tensor after in_proj and the gated-norm output before out_proj. On Qwen3.6-35B-A3B (d=2048d = 2048, TP=1), this moves 12,352+4,096=16,44812{,}352 + 4{,}096 = 16{,}448 channels per token per layer. KCP converts the residual stream at the block boundary and moves only 2d=4,0962d = 4{,}096 channels per token. This reduces bridge communication by 4.02× in forward, backward, and activation recomputation.

  • Cross-layer cancellation (KCP-Retain). Between consecutive linear-attention layers, all operations (RMSNorm, residual adds, and token-routed MoE/MLP) commute with token permutations. Because KCP acts at the layer boundary, adjacent conversions cancel out directly (P∘P−1=IP \circ P^{-1} = I). This gives KCP-Retain (your Q2) and cuts bridge volume across three GDN layers by 12× compared with PostProj (4,0964{,}096 vs. 49,34449{,}344 channels per token). Under PostProj, the conversions sit between in_proj and out_proj, so adjacent layers cannot cancel without refactoring to KCP's boundary design (a limitation noted as a TODO in #3282).

  • Black-box modularity. Layer-boundary conversion wraps any contiguous-CP sequence mixer without changing its internal projections or breaking intra-module fusions. By contrast, #3282 has to disable fused pre-GDR preprocessing.


2. Algorithmic and analytical contributions beyond Megatron #3282

Megatron #3282 is a concurrent PR on Megatron-LM's dev branch (merged in July 2026, while its main-branch PR #6439 is still open). It has no accompanying paper, formal analysis, or empirical study. Its actual packed-sequence (THD) routine also differs from KCP in two algorithmic ways:

  • O(mP)O(mP) interval intersection vs. O(TP)O(TP) token enumeration. At every bridge call, #3282 builds all TT global token indices on the GPU and runs O(TP)O(TP) bucketize and argsort passes across ranks. KCP instead intersects piecewise-contiguous intervals in O(mP)O(mP) time (m≪Tm \ll T packed segments) once per batch on the CPU from cu_seqlens, with no metadata exchange. The reverse pass is simply the inverse permutation (Equation 5).

  • General packed-bin padding (Equation 3, Proposition 1). #3282 requires every segment in cu_seqlens to be divisible by 2P2P. It raises an error when a packed bin's trailing padding segment is divisible only by PP (Section 3.1, Appendix A.1). KCP's generalized asymmetric split (c=⌊ℓ/2P⌋,  c′=ℓ/P−cc = \lfloor \ell / 2P \rfloor,\; c' = \ell / P - c) guarantees an exact bijection and matched all_to_all splits for any PP-divisible bin.

  • Formal cost and memory-regime analysis (Proposition 2). Beyond the algorithm, KCP proves exact backward gradients without reduction or scaling. It also separates per-collective (P×P\times), step-level (P/2×P/2\times), and compute/memory (P×P\times) scaling, and predicts the memory-peak transition verified on our 35B MoE runs up to 256K.


We hope this addresses your question, and we would be grateful if you would consider these points in your final rating.

I appreciate the authors’ efforts and have updated my recommendation accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.