- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies hidden state similarity as a tool to understand expert usage similarity and derive a theorem to interpret their relationship. Yet the authors find it difficult to understand specialization patterns in pre-trained LLMs, which can exhibit different pattterns in prefilling / decoding (and in decoding it means it is hard to understand which expert is used during CoT reasoning for instance). Lastly, they propose a method to mitigate usage similarity, which seems to work in early layers.
Critical strengths
This work provides a detailed analysis of the relationship between hidden state similarity and the expert specialization phenomenon, which is convincing and easy to follow.
Critical weaknesses
The method in practical takeaway does not seem to be very convincing, because starting from layer 10, the similarity is very similar to the one without depth scaling, do you think there might be a better method that can make similarity low even in deeper layers, or is that impossible, if impossible, could you explain why.
Questions for authors
- Line 044, there are many alternate router choices other than the linear projection, you could say instead by "arguably, the linear projection is one of the most popular choices for the router", then continue your argument
- Section 4.1, it might be better to move the ||P||_2 ||hi - hj||2 up to before section 4.1 so it would easier to follow. How would you estimate r in real data?
- Figure 1, it might be more convincing if more sequences could be used.
- Figure 2, It is unclear to me why hamming similarity is used on the right, while cosine similarity is used in the middle. Also it is unclear to me what energy means in the main text, and why Muon is used in the bottom plot. Better explanation could make clarity better in this part.
- How does "the router is not a random matrix" support the claim in Section 4 that expert specialization arises from hidden state similarity?
- Figure 4 is a bit hard to read, it might be better to plot in a way that x-axis is layer, y-axis is the activated frequency difference, then can see the difference grows with layer.
- Line 364-371, diferent models would have different activated experts might not be surprising, because they are in different archietectures and are trained in potentially different ways. Above are mostly writing concerns, I would change from "Weak Accept" to "Accept" if the authors could address my main question in the "Critical weakness" part.
6 comments
Author response (2/2)
Writing concerns
Line 044, there are many alternate router choices other than the linear projection, you could say instead by "arguably, the linear projection is one of the most popular choices for the router", then continue your argument
Good point! We have revised that sentence.
Section 4.1, it might be better to move the ||P||_2 ||hi - hj||2 up to before section 4.1 so it would easier to follow. How would you estimate r in real data?
To select r, we compute the SVD of the input hidden-state matrix (H) and choose the smallest r such that the leading r singular directions capture a prescribed fraction (e.g., 90%) of the total energy, measured by the sum of squared singular values:
Figure 1, it might be more convincing if more sequences could be used.
Here, each dot in the scatter plot denotes a token and 16 sequences already included ~20K tokens / dots. Including more sequences would significantly slow down the visualization and makes the figure harder to read (it will look like a puddle of blue paint).
Figure 2, It is unclear to me why hamming similarity is used on the right, while cosine similarity is used in the middle. Also it is unclear to me what energy means in the main text, and why Muon is used in the bottom plot. Better explanation could make clarity better in this part.
Cosine and Hamming measures similarity on different quantities:
- Cosine similarity measures the similarity between two continuous routing logits vectors.
- Hamming similarity measures the similarity of the actual expert activation after top-K, i.e. a discrete binary vector. We use cosine similarity in the middle to reflect its relationship with the hidden state's trend. We use hamming similarity on the right to study the actual expert usage diversity.
The energy in the left column refers to the normalized sum of squared singular values (Equation 9 in appendix A), the retained energy on the right refers to magnitude of router's projection on the data's first singular direction, i.e. how sensitive is the routing against certain direction in the data (Equation 10 in appendix A)
The reason Muon is studied here is that different optimizers show different energy distributions in the hidden states (left column): Hidden states trained under AdamW show an increased energy in 0:1, i.e., a stronger shared direction, more correlation and more hidden state collapse. Models trained Muon shows much lower energy from the first direction and less hidden state collapse with respect to depth.
We include AdamW v.s. Muon comparison here as we hypothesis using Muon can significantly alleviating the routing imbalance caused by hidden state collpase, and hence the pre-training can rely purely on auxiliary loss free load balancing for minor correction.
How does "the router is not a random matrix" support the claim in Section 4 that expert specialization arises from hidden state similarity?
The implication is illustrated in the "router-data alignment" term (note that a random matrix would have a non zero value): The router data alignment determines how sensitive the logits are to the hidden state distance.
There are two practical impacts:
- As we argued in Proposition 2, if load balancing loss is set too large, then the alignment will be very low; therefore, even very large hidden state distance would induce low logits distance
- Additionally, not related to expert specialization, when the input is out of distribution, it will have low alignment with the router weights, and hence lower logit magnitude and lower routing confidence value; which enables us to use the router as a OOD detection tool (Figure 3).
Figure 4 is a bit hard to read, it might be better to plot in a way that x-axis is layer, y-axis is the activated frequency difference, then can see the difference grows with layer.
We agree! However, we plot it this way to avoid additional aggregation over experts (we already aggregate over all tokens) such that the differences can be illustrated directly. We find an aggregated "similarity number" (e.g. the cos similarity we used in Figure 11) less informative than a direct visualization of expert usage.
Line 364-371, diferent models would have different activated experts might not be surprising, because they are in different archietectures and are trained in potentially different ways.
Agreed! But the main takeaway here is that it's difficult to reason about the expert activation pattern:
- These rollouts are all solutions for the same set of math questions, so they belong to the same domain and could activate similiar experts.
- On the other hand, for very long rollouts from models trained differently and with different architectures, the solution string could differ significantly distribution-wise, hence activating different experts. Simply relying on the "metadata" of math corpus is not sufficient to understand the behavior.
Dear Reviewer spZ4,
Thank you for your review. The authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions.
The discussion phase closes on November 18 at 11:59 pm AoE.
Thank you for your attention and contribution.
Best regards,
Area Chair
Author response (1/2)
We would like to thank the reviewer for the detailed and careful review!
Please see below for our responses to the weaknesses and questions. All revised sentences in the manuscript are highlighted in red
Alternative methods for reducing hidden state collapse
Yes, there are alternative methods to further reduce the hidden state similarity
We provide illustrations for the two approaches in the updated Figure 8 in the main text of the updated manuscript; both approaches further reduce hidden state similarity and enhance routing balance at initialization.
It is worth noting that the zero attention weight trick is only effective at initialization time since the attention weights CANNOT all be zero after training. The residual scaling technique could remain effective as it works by downweighting the correlation injection at each layer (although a theoretical derivation is no longer possible, as the independence assumption no longer holds).
When moving away from the initialization, we also observed that the model trained with Muon instead of AdamW tends to have less correlated hidden states, as shown in the left column in Figure 2, as well as the left column in Figure 17 (b) v.s. Figure 17 (c), where an increasing energy in the first direction (0:1) with depth indicates stronger shared direction in the data and hence more correlation.