- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a task-specific MoE pruning method that represents experts as nodes in a layered directed graph and selects high-weight paths. Node importance combines single-expert reconstruction error with routing and activation information, while adjacent-layer edges are assigned a transition intensity. Dynamic programming selects paths for each calibration example, and the union of experts on those paths defines the retained model. Experiments on Mixtral-8x7B and Mixtral-8x7B-Instruct, with an additional Qwen3 study, report gains over random pruning and several published compression baselines.
Critical strengths
-
The paper asks a useful practical question about uniform expert pruning. Allowing the pruning budget to vary across depth makes sense, and the results support the view that expert redundancy is not uniform across layers.
-
The experimental section includes several useful controls. The paper compares uniform with non-uniform pruning, one-expert with two-expert paths, and path selection with global greedy selection. Reporting quality together with memory and speed also makes the practical trade-off easier to judge.
Critical weaknesses
-
My main technical concern is that Equation (5) does not seem to model pair-specific transitions between adjacent layers. For each calibration sample, the transition matrix is just the outer product of an upstream marginal and a downstream marginal. After substituting it into Equation (11), the path score becomes a sum of per-layer terms; no term depends jointly on the particular experts chosen in two adjacent layers. At the 50% setting, where the paper uses m=1 path per calibration sample, the best path can therefore be found by choosing each layer independently. Ranking several paths with the same separable score still does not introduce expert-to-expert interaction. The ablations support the local score and non-uniform layer allocation, but I do not think they validate the paper's claim about modeling cross-layer expert dependencies or trajectories.
-
The evaluation protocol is also hard to assess fairly. The method is task-specific, and the paper says that it chooses a dataset-specific K based on the best performance, but it does not say which split or metric is used for that choice. It is also unclear whether every task gets a separately pruned checkpoint and whether the reproduced baselines receive the same task-specific calibration and tuning budget. In addition, Algorithm 1 takes the union of data-dependent path experts without a cardinality constraint, so the stated m values do not by themselves guarantee 50% or 25% actual sparsity. Table 2 mainly compares against whole-expert pruning and merging, while omitting MAESTRO and the expert-channel-level structured pruning methods HEAPr (ICLR 2026) and CAMERA-P (AAAI 2026). I would like to see whether Pathfinder still improves both effectiveness and efficiency in these comparisons.
Questions for authors
-
Since the transition intensity in Equation (5) factorizes into two marginals, where does any pair-specific interaction between adjacent-layer experts enter the path score? How does this product-of-marginals statistic compare with empirical token-level joint transitions on the same calibration inputs?
-
Which split and metric were used to select K for each task, especially GSM8K and MedQA, where no validation split is listed? Does every reported task score come from a separately pruned checkpoint? Were the reproduced baselines given the same task-specific calibration data and hyperparameter-selection budget? The answers would also help clarify how the six-task average should be interpreted.
-
How does Algorithm 1 reach the exact target sparsity when the retained set is a data-dependent union of path experts? What are the retained counts by layer, and is there any trimming, filling, tie-breaking, or search over the path count?
-
What is the key difference from MAESTRO's empirical pair-transition and Markov-chain construction, and what cross-layer dependency is captured by Equation (5)'s product of marginals? How does Pathfinder compare in effectiveness and efficiency with the expert-channel-level structured pruning methods HEAPr and CAMERA-P under comparable settings?