acceptodds
Under review as a conference paper at ICLR 2027

Same Counts, Different Cost: Measuring the Order Gap in Sparse Mixture-of-Experts Routing

Abstract

Adapt a sparse Mixture-of-Experts model to a new task and you can easily measure how much its routing changed: how often each expert fires, how far the choices have moved, which experts fire together. Those summaries are then used to decide what to keep resident in memory. Every one is a count, a rate or a set overlap, and none records the order in which experts are accessed. We call the cost of that omission a trace's order gap: what randomizing token order does to simulated miss rate while every one of the trace's own counts is preserved exactly. With 16 of 64 experts resident per layer it is 0.135 on full traces, holding to within 0.005 across two checkpoints, an independently adapted second run and a sixfold increase in coverage, while adaptation itself moves the same metric by 0.0008. A separate comparison at twice the optimizer budget, on its own population, shifts the mean order gap by 0.000232. Replayed on real expert-sized tensors on an RTX A6000, randomizing token order adds 14.2 seconds of expert transfer to the actual order's 38.9, a 36.6% increase. Order gaps also vary across real checkpoints, by more than adaptation moves the metric over 225 checkpoint-layer cells from four adapted runs; each carries its own count-matched baseline, so that comparison is descriptive. The insufficiency is then exact rather than comparative. A constrained kernel additionally holds each prompt's aggregate overlap with the pre-adaptation checkpoint fixed, so per-prompt expert counts, per-expert rank counts and aggregate overlap are identical on both sides, and the configuration it reaches over 120 prompts and all 15 eligible layers, invariants checked on each of the 1,800 cells, differs by +0.015465 in miss@16; replayed on a 20-prompt, five-layer subset it adds 2.25 seconds of expert-loading time. One such pair shows that no function of those summaries alone reproduces the simulated cost; our sampler does not mix, so those figures describe the configuration reached rather than a population effect. Behind the constructions is the measurement program that calibrates them. Under an empirical null stratified by task, router rank, resident-set size and token position, newly selected experts show excess recent-window residency of +0.037129 and +0.037296 in two OLMoE-1B-7B trajectories and three to six times less in Qwen1.5-MoE-A2.7B, while matched comparisons give no evidence that the longitudinal residual exceeds that between two independently adapted runs. The claim is narrow: these summaries do not determine the cost they are used to reason about, and the gap is worth a third of this workload's expert transfer time on real hardware. The replay measures transfer in isolation, so we claim no end-to-end benefit.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, MoE routing, expert caching, locality, continual adaptation, inference systems, routing drift

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nUt3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether commonly used routing statistics can adequately characterize expert caching and loading costs in sparse mixture-of-experts (MoE) models after task adaptation. The authors introduce the “order gap,” which quantifies the temporal locality omitted by statistical summaries by randomizing token access order while preserving expert activation counts. They further construct traces that simultaneously preserve per-prompt expert counts, per-expert router-rank counts, and aggregate routing overlap with the pre-adaptation model, demonstrating that identical summaries can still correspond to different cache costs. On OLMoE-1B-7B, with 16 of 64 experts cached per layer, randomizing the order increases the simulated cache miss rate by approximately 13.5 percentage points, whereas task adaptation itself changes the metric by only about 0.08 percentage points. Tensor replay on an RTX A6000 further shows that randomizing the order increases expert transfer time by 36.6%. The authors also use stratified conditional baselines and supplementary experiments on Qwen1.5-MoE-A2.7B to examine the association between routing substitutions and recent expert accesses, as well as its sensitivity to the comparison setup. The paper’s main contribution is to demonstrate, through controlled constructions and empirical measurements, that the examined routing summaries do not uniquely determine order-dependent cache costs, highlighting the need to analyze access sequences directly when assessing such costs. Its hardware validation covers only expert transfers and does not establish end-to-end inference speedups.

Critical strengths
  1. Validation extends beyond simulation: Cache-policy sensitivity experiments and tensor-transfer replay on real GPU hardware show that the observed differences correspond to measurable differences in hardware transfer costs.
  2. Includes real hardware measurements: In addition to cache simulations, the paper measures expert transfer times through GPU tensor replay, providing an empirical basis for the discussion.
Critical weaknesses
  1. The paper’s central contribution concerns the insufficiency of statistical summaries for determining cache costs. However, the first half of the main text focuses primarily on conditional expert substitutions and their attribution to adaptation, without clearly establishing the logical connection between these analyses and the core construction. I recommend presenting the central counterexample and cost quantification earlier, and clarifying that Sections 3 and 4 provide auxiliary structural analyses rather than evidence necessary to establish the main conclusion.
  2. Section 6 primarily replicates the conditional recent-window residency analyses from Sections 3 and 4, rather than validating the paper’s core order gap and constrained cost differences on a second model family. Consequently, it provides limited support for the external validity of the main conclusions. Although sensitivity to the baseline is a meaningful descriptive finding, its relationship to the central claim—that routing summaries are insufficient to determine cache costs—remains unclear. I recommend prioritizing cross-model validation of the core construction and condensing the current, relatively independent descriptive analyses or moving them to an appendix.
  3. The paper provides substantial evidence that the examined statistical summaries are insufficient to determine cache costs, but it does not demonstrate how this finding translates into better cost prediction, cache decisions, or end-to-end inference performance. The existing hardware experiments validate cost increases caused by perturbations, rather than optimization gains obtained by leveraging the finding. Its practical applicability and systems-level value therefore remain to be established.
Questions for authors
  1. Given that these statistical summaries cannot precisely determine costs, how much improvement in practical system decisions can this insight enable?
  2. Can the authors replicate the order-gap and constrained rewiring experiments on Qwen1.5-MoE-A2.7B to assess whether the central cache-cost findings hold beyond OLMoE, rather than limiting cross-model validation to the conditional residency analysis?
  3. Could the authors justify the choice of Qwen1.5-MoE-A2.7B and extend the core experiments to a more recent sparse MoE model? In particular, do the observed order gap and cache-cost differences under summary-preserving transformations persist in newer MoE architectures?
Reviewer_4qJpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing routing summaries for sparse MoE models capture expert usage statistics but overlook access order, which can affect expert caching costs. This paper introduces the order gap to quantify this effect and proposes a constrained rewiring method that preserves per-prompt expert counts, per-expert rank counts, and aggregate routing overlap. Experiments on OLMoE and Qwen routing traces reveal temporal locality, while constrained rewiring increases miss@16 by 0.0155 and expert-loading time by 2.25 seconds on an RTX A6000. The results demonstrate that these routing summaries cannot uniquely determine order-dependent caching costs, although end-to-end serving benefits remain unverified.

Critical strengths
  1. Clear evidence that routing statistics cannot fully predict caching costs. By changing the expert assignments across tokens while keeping expert usage counts, rank distributions, and overall routing overlap unchanged, the paper shows that the simulated cache miss rate can still increase by 1.55 percentage points.

  2. Hardware validation. The cache-cost differences are also observed on an RTX A6000, where expert-loading time increases by 36.6% under randomized token order and 5.8% under constrained rewiring.

Critical weaknesses
  1. Connection to established caching principles. The effect of access order on cache misses is well understood, and temporal locality has also been studied in MoE serving. While the constrained rewiring experiment provides a controlled demonstration for several routing statistics, the conceptual contribution beyond extending these established findings to MoE routing remains somewhat limited.

  2. Practical implications for MoE serving. The paper demonstrates that routing statistics alone cannot determine cache costs, but does not directly evaluate how this limitation affects caching or prefetching decisions in existing MoE serving methods. The hardware experiments are also limited to isolated expert transfers on a single GPU using a small subset of prefill traces.

Questions for authors

Could the authors clarify which existing MoE caching methods are directly affected by the insufficiency of routing statistics demonstrated in this paper?

Reviewer_QdfEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper does not propose a specific algorithm. Instead, it makes a claim based on experimental observations: cache hits in MoE models are closely related to access order, which directly affects model deployment costs. Counting how often MoE experts are used is not enough to determine caching costs; access order also matters, and the same counts can lead to different swapping costs. The paper mainly proposes an evaluation metric and a measurement system for MoE routing. It defines an “order gap” and finds that, for the same request, randomly changing the order of experts accessed through token routing increases expert transfer time by 36.6%.

Critical strengths

Honestly, I do not see any key strengths that would give me a strong reason to support acceptance.

Critical weaknesses
  1. The paper does not propose a practical algorithmic improvement. Almost all of its content is based on simple experimental observations. It repeatedly emphasizes that expert access order affects the final cost. Although the authors add many evaluation and statistical methods, the underlying insight is fairly naive. They do not provide any actual method for end-to-end speedup, so it is hard to say that the paper makes any substantial research contribution.

  2. Although the paper provides some analysis of experimental findings, this analysis still offers little novelty. In model deployment, cache access already depends on order, and random access is necessarily slower on hardware, which is a common belief. The paper merely uses this fact to define some terms related to constraints and measure some metrics, yet devotes a large amount of space to presenting it as the main claim. This is not enough to meet the bar for an ICLR paper.

  3. The writing is very disorganized, the logic is extremely weak, and the paper lacks the structure expected of an research paper. The abstract is extremely long and includes many detailed numerical results for specific models that should not be in the abstract. It also contains defensive statements such as “The claim is narrow: these summaries do not determine the cost they are used to reason about,” which do not help support the paper’s main claim. The sections are very poorly organized. Instead of following a motivation–method–experiment structure, they consist entirely of simple observations and analysis of experimental findings. This organization gives readers even less reason to keep reading, let alone understand the various concepts introduced by the paper, such as “windows” and the “order gap.” The paper itself also acknowledges that it was largely written using generative AI.

Questions for authors
  1. Given that different expert access orders greatly affect the final cost, do you have any ideas for reducing this cost?
  2. Why did you not aim to develop a method that achieves real end-to-end speedups in MoE model deployment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.