- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether commonly used routing statistics can adequately characterize expert caching and loading costs in sparse mixture-of-experts (MoE) models after task adaptation. The authors introduce the “order gap,” which quantifies the temporal locality omitted by statistical summaries by randomizing token access order while preserving expert activation counts. They further construct traces that simultaneously preserve per-prompt expert counts, per-expert router-rank counts, and aggregate routing overlap with the pre-adaptation model, demonstrating that identical summaries can still correspond to different cache costs. On OLMoE-1B-7B, with 16 of 64 experts cached per layer, randomizing the order increases the simulated cache miss rate by approximately 13.5 percentage points, whereas task adaptation itself changes the metric by only about 0.08 percentage points. Tensor replay on an RTX A6000 further shows that randomizing the order increases expert transfer time by 36.6%. The authors also use stratified conditional baselines and supplementary experiments on Qwen1.5-MoE-A2.7B to examine the association between routing substitutions and recent expert accesses, as well as its sensitivity to the comparison setup. The paper’s main contribution is to demonstrate, through controlled constructions and empirical measurements, that the examined routing summaries do not uniquely determine order-dependent cache costs, highlighting the need to analyze access sequences directly when assessing such costs. Its hardware validation covers only expert transfers and does not establish end-to-end inference speedups.
Critical strengths
- Validation extends beyond simulation: Cache-policy sensitivity experiments and tensor-transfer replay on real GPU hardware show that the observed differences correspond to measurable differences in hardware transfer costs.
- Includes real hardware measurements: In addition to cache simulations, the paper measures expert transfer times through GPU tensor replay, providing an empirical basis for the discussion.
Critical weaknesses
- The paper’s central contribution concerns the insufficiency of statistical summaries for determining cache costs. However, the first half of the main text focuses primarily on conditional expert substitutions and their attribution to adaptation, without clearly establishing the logical connection between these analyses and the core construction. I recommend presenting the central counterexample and cost quantification earlier, and clarifying that Sections 3 and 4 provide auxiliary structural analyses rather than evidence necessary to establish the main conclusion.
- Section 6 primarily replicates the conditional recent-window residency analyses from Sections 3 and 4, rather than validating the paper’s core order gap and constrained cost differences on a second model family. Consequently, it provides limited support for the external validity of the main conclusions. Although sensitivity to the baseline is a meaningful descriptive finding, its relationship to the central claim—that routing summaries are insufficient to determine cache costs—remains unclear. I recommend prioritizing cross-model validation of the core construction and condensing the current, relatively independent descriptive analyses or moving them to an appendix.
- The paper provides substantial evidence that the examined statistical summaries are insufficient to determine cache costs, but it does not demonstrate how this finding translates into better cost prediction, cache decisions, or end-to-end inference performance. The existing hardware experiments validate cost increases caused by perturbations, rather than optimization gains obtained by leveraging the finding. Its practical applicability and systems-level value therefore remain to be established.
Questions for authors
- Given that these statistical summaries cannot precisely determine costs, how much improvement in practical system decisions can this insight enable?
- Can the authors replicate the order-gap and constrained rewiring experiments on Qwen1.5-MoE-A2.7B to assess whether the central cache-cost findings hold beyond OLMoE, rather than limiting cross-model validation to the conditional residency analysis?
- Could the authors justify the choice of Qwen1.5-MoE-A2.7B and extend the core experiments to a more recent sparse MoE model? In particular, do the observed order gap and cache-cost differences under summary-preserving transformations persist in newer MoE architectures?