acceptodds
Under review as a conference paper at ICLR 2027

MoE Pathfinder: Trajectory-driven Expert Pruning

Abstract

Mixture-of-experts (MoE) architectures used in large language models (LLMs) achieve state-of-the-art performance across diverse tasks yet face practical challenges such as deployment complexity and low activation efficiency. Expert pruning has thus emerged as a promising solution to reduce computational overhead and simplify the deployment of MoE models. However, existing expert pruning approaches conventionally rely on local importance metrics and often apply uniform layer-wise pruning, leveraging only partial evaluation signals and overlooking the heterogeneous contributions of experts across layers. To address these limitations, we propose an expert pruning approach based on the trajectory of activated experts across layers, which treats MoE as a weighted computation graph and casts expert selection as a global optimal path planning problem. Within this framework, we integrate complementary importance signals from reconstruction error, routing probabilities, and activation strength at the trajectory level, which naturally yields non-uniform expert retention across layers. Experiments show that our approach achieves superior pruning performance on nearly all tasks compared with most existing approaches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-experts, Pruning, Model Compression, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iAUjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a task-specific MoE pruning method that represents experts as nodes in a layered directed graph and selects high-weight paths. Node importance combines single-expert reconstruction error with routing and activation information, while adjacent-layer edges are assigned a transition intensity. Dynamic programming selects paths for each calibration example, and the union of experts on those paths defines the retained model. Experiments on Mixtral-8x7B and Mixtral-8x7B-Instruct, with an additional Qwen3 study, report gains over random pruning and several published compression baselines.

Critical strengths
  1. The paper asks a useful practical question about uniform expert pruning. Allowing the pruning budget to vary across depth makes sense, and the results support the view that expert redundancy is not uniform across layers.

  2. The experimental section includes several useful controls. The paper compares uniform with non-uniform pruning, one-expert with two-expert paths, and path selection with global greedy selection. Reporting quality together with memory and speed also makes the practical trade-off easier to judge.

Critical weaknesses
  1. My main technical concern is that Equation (5) does not seem to model pair-specific transitions between adjacent layers. For each calibration sample, the transition matrix is just the outer product of an upstream marginal and a downstream marginal. After substituting it into Equation (11), the path score becomes a sum of per-layer terms; no term depends jointly on the particular experts chosen in two adjacent layers. At the 50% setting, where the paper uses m=1 path per calibration sample, the best path can therefore be found by choosing each layer independently. Ranking several paths with the same separable score still does not introduce expert-to-expert interaction. The ablations support the local score and non-uniform layer allocation, but I do not think they validate the paper's claim about modeling cross-layer expert dependencies or trajectories.

  2. The evaluation protocol is also hard to assess fairly. The method is task-specific, and the paper says that it chooses a dataset-specific K based on the best performance, but it does not say which split or metric is used for that choice. It is also unclear whether every task gets a separately pruned checkpoint and whether the reproduced baselines receive the same task-specific calibration and tuning budget. In addition, Algorithm 1 takes the union of data-dependent path experts without a cardinality constraint, so the stated m values do not by themselves guarantee 50% or 25% actual sparsity. Table 2 mainly compares against whole-expert pruning and merging, while omitting MAESTRO and the expert-channel-level structured pruning methods HEAPr (ICLR 2026) and CAMERA-P (AAAI 2026). I would like to see whether Pathfinder still improves both effectiveness and efficiency in these comparisons.

Questions for authors
  1. Since the transition intensity in Equation (5) factorizes into two marginals, where does any pair-specific interaction between adjacent-layer experts enter the path score? How does this product-of-marginals statistic compare with empirical token-level joint transitions on the same calibration inputs?

  2. Which split and metric were used to select K for each task, especially GSM8K and MedQA, where no validation split is listed? Does every reported task score come from a separately pruned checkpoint? Were the reproduced baselines given the same task-specific calibration data and hyperparameter-selection budget? The answers would also help clarify how the six-task average should be interpreted.

  3. How does Algorithm 1 reach the exact target sparsity when the retained set is a data-dependent union of path experts? What are the retained counts by layer, and is there any trimming, filling, tie-breaking, or search over the path count?

  4. What is the key difference from MAESTRO's empirical pair-transition and Markov-chain construction, and what cross-layer dependency is captured by Equation (5)'s product of marginals? How does Pathfinder compare in effectiveness and efficiency with the expert-channel-level structured pruning methods HEAPr and CAMERA-P under comparable settings?

Reviewer_R1hQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes MoE Pathfinder for task-specific expert pruning. It constructs a layered graph of experts, using reconstruction error to score individual experts and activation and routing statistics to weight connections between adjacent layers. It then finds high-scoring paths for each calibration sample and keeps the experts covered by these paths. This allows the number of retained experts to vary across layers. The experiments focus on Mixtral, with additional results on Qwen3.

Critical strengths
  1. Evaluating expert importance through cross-layer computational paths is a promising direction. Rather than assessing experts in each layer independently, the paper attempts to connect individual expert contributions with information propagation across layers. This offers a useful perspective for studying expert cooperation and non-uniform pruning. Although whether the current formulation adequately captures these dependencies remains to be validated, the underlying motivation is reasonable.
  2. The method achieves model compression without additional online path-search overhead. Path selection is performed offline, and deployment uses a fixed set of retained experts. The reported memory reduction and inference speedup indicate that the approach can reduce both model footprint and inference time.
Critical weaknesses
  1. For m=1m=1, the path objective reduces to independent layer-wise selection, leaving the claimed cross-layer mechanism unsubstantiated. Equation (5) defines tij(l)=ai(l)rj(l+1)t_{ij}^{(l)} = a_i^{(l)} r_j^{(l+1)}. Let bi(l)b_i^{(l)} denote importance before boundary correction. Substituting Equations (5), (9), and (10) into the path objective gives w(p)=∏l=1Lail(l)ril(l)bil(l)w(p) = \prod_{l=1}^{L} a_{i_l}^{(l)} r_{i_l}^{(l)} b_{i_l}^{(l)}. For the stated fully connected graph, maximizing this objective for m=1m=1 is equivalent to independently maximizing ai(l)ri(l)bi(l)a_i^{(l)} r_i^{(l)} b_i^{(l)} in each layer for each sample. Dynamic programming therefore introduces no additional cross-layer selection capability. The authors should clarify the trajectory formulation's contribution beyond separable scoring and sample aggregation, or specify additional constraints that prevent this equivalence.

  2. The ablations do not establish that path planning causes the improvements over baselines. Appendix A.1 compares uniform and non-uniform pruning, while Appendix A.3 compares global ranking with the full method; these comparisons also change allocation or sample-selection rules. A matched control should independently maximize the factorized score per sample and layer, then apply identical cross-sample union, budget handling, and tie-breaking. For m=1m=1, this should reproduce the path solution under the stated formulation. The reported gains support the complete procedure, but do not distinguish the effects of combined scoring and non-uniform allocation from an additional path mechanism.

Questions for authors
  1. What does global path planning contribute beyond independent layer-wise selection when the path objective factorizes? Are there additional constraints that prevent this equivalence for m=1m=1?

  2. Can you compare path planning with independent per-sample, per-layer selection using identical scoring, aggregation, and budget rules? Which experiments distinguish the benefits of path planning from those of combined scoring and non-uniform allocation?

Reviewer_nTVQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

MoE Pathfinder is a trajectory-driven expert pruning framework that compresses MoE models by reformulating expert selection as a global path planning optimization problem on a directed weighted computational graph. By combining reconstruction error, routing probability, and activation strength, it captures cross-layer information flows and automatically performs task-specific, non-uniform pruning across layers via dynamic programming. Experiments show higher performance than the compared methods.

Critical strengths
  1. Framing expert selection as finding important paths across layers is an intuitive idea. It also lets different layers keep different numbers of experts instead of applying a uniform ratio.
  2. The method combines three signals (reconstruction error, routing probability, and activation strength) and only needs forward passes on a small calibration set, so it is cheap to run.
  3. It reports practical efficiency numbers. At 50% sparsity the pruned model needs 52% of the original memory.
Critical weaknesses
  1. The global path planning does not capture any interaction between layers.
  2. Despite the name trajectory-driven, the method never uses actual token trajectories.
  3. The method prunes a separate model for each benchmark using that benchmark’s training data. Many baselines in Table 2 are copied from papers that prune one general model for all tasks.
  4. The calibration setting appears to be tuned on the test benchmarks.
  5. The paper doesn’t explain how exactly 50% sparsity is reached, since the kept experts come from a union over calibration samples. It also doesn’t say what happens if a layer keeps fewer than two experts, when Mixtral routes every token to two.
Questions for authors
  1. Were the replicated baselines given the same task-specific calibration data as your method? Why do three of them perform below random pruning?
  2. How is the exact sparsity level reached? Please report the number of experts kept in each layer, and explain how layers left with fewer than two experts are handled.
  3. How should K be chosen without using test results? How sensitive are the results to K and to the k-means random seed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.