- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses incompatible sequence layouts in hybrid transformers under context parallelism. Fused causal softmax-attention kernels use zigzag sharding, whereas native state-passing linear-attention kernels require contiguous spans. KCP converts between these layouts using an invertible token redistribution before and after each linear-attention block, avoiding full-sequence gathering and redundant computation while preserving the existing kernels.
Critical strengths
S1. KCP resolves a concrete interface mismatch without modifying either attention kernel and without imposing a head-count divisibility constraint.
S2. The paired experiments process identical tokens and demonstrate substantial throughput gains and a configuration enabled by lower memory.
Critical weaknesses
Limited baseline comparison. AllGather is a relevant and controlled baseline, but redundantly evaluates the full sequence on each rank. The paper discusses head sharding and zigzag-aware state passing without measuring an alternative or concretely establishing where its constraints bind in the evaluated configurations.
Questions for authors
Q1. Under the actual head counts and tensor-parallel settings used in the MoE experiments, which nonredundant alternatives are feasible? Can the authors provide one matched head-sharding comparison or specify the configurations that its divisibility constraints exclude?
Q2. Is “the gather is not the bottleneck” supported by profiling of the small-bin configurations, or could layout-conversion overhead offset the savings?
2 comments
I appreciate the authors' response to my concerns. I believe the current score accurately reflects the paper's contribution and its current state. I will maintain my score.
Thank you for the careful review and the precise questions. We measured two non-redundant alternatives at the paper's main configuration and profiled the small-bin configuration.
[W1 & Q1]: Limited baseline comparison. Which non-redundant alternatives are feasible under the actual head counts and tensor-parallel settings? Can you provide a matched head-sharding comparison or specify the configurations that its divisibility constraint excludes?
We now provide both: the configuration that head sharding's divisibility constraint excludes (CP32, which follows from the head count) and a matched comparison at CP16, where head sharding is feasible (Headwise in Table A).
Feasibility. Head sharding needs the number of heads per TP rank to be divisible by the CP size. With GDN's 16 query/key heads and TP1 in all MoE experiments, it is feasible at CP4, CP8 and CP16 and excluded at CP32, since 16 is not divisible by 32; KCP trains CP32 (Table 9 of the paper). Conversion inside the GDN block (Megatron #3282) has no such constraint. Zigzag-aware state passing (FLA #1221) has none either, but it defines zigzag over the whole packed buffer rather than per sample, so our per-sample layout would still have to be redistributed.
Baseline comparison. We implemented head sharding (Headwise: the design of Megatron #2614, #2642 and #2645) and in-block conversion (PostProj) in the same training stack and measured each against KCP at the paper's main configuration (Qwen3.6-35B-A3B, TP1/PP2/EP8/CP16, 32 H100 GPUs). As shown in Table A, KCP is faster than head sharding in every matched pair and places no constraint on head counts. It is also faster than PostProj in every pair.
Table A. Throughput in tokens/s/GPU under the paper's protocol. Headwise and PostProj entries are means of three matched pairs (same weights and data), each with its own KCP control
[Q2]: Is “the gather is not the bottleneck” supported by profiling of the small-bin configurations, or could layout-conversion overhead offset the savings?
We have now profiled this configuration. Layout conversion does not offset the savings: it costs the same as the gather it replaces. What offsets them is communication inside the state-passing kernels. The profile is of Qwen3.5-9B at 8K/CP4 (TP2/PP1, eight H100 GPUs), the dense control of the paper; each entry is the union of a category's GPU intervals within a rank, averaged over two steps and all ranks.
† Sub-intervals of the rows above, not additional categories. Layout-related collectives are the CP gather and backward reduce-scatter for AllGather, and the layout conversions for KCP.
KCP halves non-NCCL compute, but NCCL activity grows by a similar amount, about two thirds of it state and halo exchange inside the state-passing kernels, which AllGather does not need. Total GPU-busy time therefore ends up about equal, in line with the near-parity in throughput (1.05×) that the paper reports for this configuration.
NCCL intervals include time spent waiting on peers and categories can overlap, so these figures locate the offset rather than give an additive breakdown, and they cover one configuration and two profiled steps.
We hope these clarifications and experiments address your concerns. If so, we would be grateful if you would consider adjusting your rating.