- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents an empirical analysis of how the four streams in mHC function in DeepSeek-V4-Flash. The authors find that a typical attention or FFN module effectively utilizes approximately two streams, with the dominant stream varying across layers, while the stream representations remain directionally distinct. Another key observation is that residual mixing is relatively limited and primarily concentrated in the early layers. In layers 22–42, the residual pathways largely preserve each stream independently.
Critical strengths
-
The paper provides useful evidence from a large pretrained model. The changing dominant streams across depth show that the single-stream collapse reported in smaller models does not describe every trained HC model.
-
The fixed-mean mixer experiment is particularly interesting. It separates the importance of the mixing pattern at each site from the benefit of adapting that pattern to individual tokens.
Critical weaknesses
-
The paper shows that some routing and mixing operations can be simplified in an existing model. It does not report whether these changes save computation or speed up inference, so their practical benefit is still unclear.
-
All results come from one checkpoint. The differences from earlier small-model studies are interesting, but the paper cannot tell whether they come from model scale, architecture, or training setup. This limits how broadly the findings can be interpreted.
Questions for authors
- I found the fixed-mean result interesting: replacing early-layer mixers with their means raises perplexity by only 0.2%, compared with 41% for identity matrices. But this is tested only after training. Could fixed mixers work just as well if used from the start?
3 comments
Reply by Reviewer mz1V
Thank you for your response.
However, my main concern remains unresolved. It is still unclear how the findings could be used in practice or what concrete benefits they would provide.
I am therefore inclined to keep my current score, leaning toward rejection.
Response
Thank you for the clarification. We further emphasize the practical and architectural implications of our findings.
Practical benefits. Our results provide actionable guidance for simplifying existing mHC checkpoints without retraining. Replacing late mixers (L22–42) with identity reduces prefill latency by 2.41–6.40% on 8×H20 GPUs, with up to 1 GiB lower peak allocated memory. At B8/8K, latency drops from 35.023 to 32.779 seconds. This comes with a measurable 1.23% C4 PPL increase, while no degradation is detected on C-Eval or GSM8K under our evaluated metrics.
Implications for mHC design. More broadly, our findings suggest a depth-aware design principle: preserve site-specific residual mixing in early layers, explore simpler near-identity mixing in later layers, and retain depth-dependent, read/write-specific routing rather than enforcing a globally dominant stream. This suggests that multi-stream computation should be allocated according to its functional importance across depth, rather than uniformly across all layers.
We will incorporate these practical findings and architectural implications into the revised manuscript.
Response
We thank the reviewer for the constructive feedback and for recognizing the value of our fixed-mean mixer analysis and large-model characterization.
1. Practical Inference Efficiency
We implement the structural simplifications in the official MP8 inference pipeline and benchmark them on 8×H20 GPUs using the same checkpoint and precision.
Implementation. We evaluate four configurations:
End-to-end latency. We measure prefill across batch sizes 1/8 and prompt lengths 2K/8K, and decode at batch sizes 1/8 with a fixed 32-step generation. Model loading and tokenization are excluded. We report median latency [min, max] over repeated runs, with percentage changes relative to Native (negative means faster).
Prefill latency (seconds)
Decode latency (seconds, fixed 32 steps)
Peak allocated memory (GiB)
Summary. Late Identity reduces prefill latency by 2.41–6.40% and peak allocated memory by up to 1 GiB at B8/8K. It also reduces decode latency by 0.23–1.14% across the tested settings. Early Mean maintains near-native end-to-end latency, while combining both simplifications achieves up to 5.71% lower prefill latency.
These results demonstrate practical, workload-dependent inference benefits from simplifying mHC residual mixing. We will include these results in the revised manuscript.
2. Generality Beyond a Single Checkpoint
To examine whether our findings are specific to DeepSeek-V4-Flash-0731, we additionally analyze the DeepSeek-V4-Flash checkpoint. Both checkpoints exhibit highly consistent routing and mixing patterns:
In both checkpoints, each of the four streams is the most frequent routing winner at some sites, with the dominant stream changing across depth. Read/write routing remains concentrated, while mixer diagonals are consistently higher in later layers, indicating weaker residual mixing. Read/write routing remains concentrated, while residual mixing becomes weaker in later layers.
Importantly, DeepSeek-V4-Flash provides a real-world example of mHC deployed in a large-scale, fully trained model, complementing previous studies on smaller models. Our consistent findings across two checkpoints further support the robustness of these observations.
3. Fixed Mixers from the Start of Training
We thank the reviewer for this insightful suggestion. Our fixed-mean experiment demonstrates that token-wise mixer variation has limited impact on an already-trained model, but does not establish whether it is necessary during training. We hypothesize that static, site-specific mixers may also learn effective mixing structures when jointly optimized with the model from initialization. Testing this hypothesis requires controlled from-scratch comparisons between dynamic and static learnable mixers, which we leave for future work. We will clarify this distinction in the revised manuscript.