- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper observes that tokens in different reasoning behaviors have distinct routing patterns and this observation generalizes across datasets and domains. Building on this, authors propose a method to curb overthinking during MoE LLM inference by selectively editing expert router scores. Extensive experiments across 16 models and benchmarks demonstrated their method effectively reduces generated tokens by a median of 30.4% while largely preserving accuracy.
Critical strengths
-
The topic of this paper is interesting and meaningful. It provides a novel perspective to change MoE model’s behaviors by changing router’s behaviors, which is training-free.
-
This paper presents a thorough and logically structured investigation into the model’s internal mechanism. Its analysis is supported by solid statistical evidence, providing strong support of existence of behavior-associated experts.
Critical weaknesses
-
The observation of behaviors-associated experts suggests that certain reasoning behavior is associated with certain router behavior. However, it remains unclear whether editing router behavior can change reasoning behaviors causally. There is a logical gap between observation and the proposed method.
-
Concerns about application. The additional cost of perturbing the router behavior remains unclear so the practical efficiency improvement is also unclear. Moreover, steering method requires separate tuning for each benchmark, leaving its cross-task generalization ability and efficiency under such generalization unclear.
Questions for authors
-
Equation 4 in the paper considers relative value of the router scores to identify behaviors-associated experts, what is the typical range of the absolute value of router score of these experts? Do they generally have high router scores? If not, steering experts with different router score have the same impact on model?
-
The paper identifies behavior-associated experts based on the original model's behavior. However, the steering method changes the model's behavior. Could these changes also alter the set of behavior-associated experts? In other words, are the identified behavior-associated experts still valid after steering?
-
Could authors provide the comparison with other expert steering baselines? The current baselines in this paper are too simple.
-
Could authors provide analysis about practical efficiency (e.g. wall clock time) of proposed method, which also considers the cost of identifying and steering experts?
-
The paper splits reasoning behaviors using a judge LLM (GPT5.5), which limits the deployment of this method.