acceptodds
Under review as a conference paper at ICLR 2027

From Tokens to Contexts: Progressive N-Gram Hash Routing for Sparse MoE Language Models

Abstract

Early-layer hash routing has emerged as a practical alternative to fully learned expert selection in sparse Mixture-of-Experts (MoE) pretraining, but existing token-ID hashing answers only where deterministic routing can be used, not what routing key should be used within the shallow block. Because every occurrence of the same token is mapped to the same expert set, token-ID hashing is context-blind, can concentrate frequent-token traffic, and ignores the progressive change in representation granularity with depth. We propose ProGram, a layer-aware multi-prime n-gram hash routing method that treats shallow MoE routing as depth-aware routing-key design. ProGram replaces token-ID routing atoms with ordered local n-gram contexts, constructs K distinct experts through multi-prime hashing, stable deduplication, and fallback probing, and applies a progressive routing-order schedule ν = (1, 2, 3) across the first three MoE layers. This yields a gradual transition from lexical identity to short-range contextual patterns: hashing determines expert identities in shallow layers, learned router scores compute adaptive mixture weights over the selected experts, and deeper layers retain standard learned top-K routing. We pretrain a 5B-parameter MoE model with 412M activated parameters per token on approximately 100B tokens and evaluate it on 12 knowledge, reasoning, and code benchmarks. In controlled comparisons, ProGram₁₂₃ obtains the best result on 10 of the 12 benchmarks relative to dense-replacement baselines and achieves higher scores than a fully learned-routing MoE baseline on 11 benchmarks, with a tie on MGSM. It also yields lower language-modeling loss and lower sequence-level load-balancing loss during pretraining. These results support progressively contextualized hash routing as a useful shallow-layer routing prior. Code is available at https://anonymous.4open.science/r/Program-FE0E/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Hash Routing, N-Gram Routing, Layer-Aware Routing, Progressive Routing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GhZSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduzes ProGram, a layer-aware multi-prime n-gram hash routing method that treats shallow MoE routing as depth-aware routing-key design. It uses unigram, bigram, and trigram hashing in the first three MoE layers, respectively, while retaining learned mixture weights and standard learned routing in deeper layers. The contributions include the progressive routing schedule, a multi-prime hashing procedure with deduplication and fallback probing, and an analysis connecting routing-key concentration to expected load imbalance under idealized hashing assumptions. Validation uses a 5B-parameter MoE with 412M activated parameters, pretrained on approximately 100B tokens and evaluated on 12 benchmarks. PROGRAM improves over the matched learned-routing baseline on 11 benchmarks, ties on one, and performs best on nine benchmarks among the evaluated hash schedules.

Critical strengths
  1. The paper presents a simple, well-motivated way to incorporate local context into shallow expert selection while retaining adaptive weighting and learned routing in deeper layers.
  2. The method is well specified at the algorithmic level, and the reported results are promising.
  3. The paper is well written and easy to follow, and the authors conducted comprehensive experiments.
Critical weaknesses
  1. For unigram routing, Eq. (8) makes the selected expert set depend only on modmod EE, so token IDs separated by multiples of 192 will receive identical routing under this configuration. This limits the unigram implementation to at most 192 distinct expert sets. Can author provide more explaination about this setting?
  2. Tables 1 and 2 report point estimates without variation across runs or confidence intervals for downstream scores.
  3. The explanation that unigram routing establishes a lexical anchor and higher orders subsequently enable contextual specialization is not supported by direct analysis of expert behavior. I hope the author can explain this in detail.
  4. Appendix B argues that routing overhead is negligible using operation counts, but the author do not provide throughput, latency, or peak-memory measurements. I hope the author can provide more details about the computational costs.
Questions for authors
  1. Please refer to the weakness.
  2. I don't really understand the sequence-level balancing-loss, it will be helpful if the authors can provide a more detailed explaination.
6 comments

[Response to Reviewer GhZS (1/2)]

We thank the reviewer for the thoughtful and constructive evaluation. We appreciate the positive assessment of the method and its experimental results. We address the concerns regarding the expressiveness of unigram hashing, experimental variance, expert specialization, computational overhead, and sequence-level load balancing below.

W1: Modulo collisions in unigram routing.

The reviewer is correct. In Eq. (8), two token IDs that are congruent modulo 192 produce the same sequence of candidate expert IDs. Deterministic deduplication and fallback probing therefore give them the same selected expert set. Each unigram layer can consequently produce at most 192 distinct expert sets. The mixture weights over a selected set can still vary with the token's hidden state. We will make this bound explicit and describe the first layer as a coarse, deterministic partition of token IDs. In layers 2 and 3, adding preceding tokens to the routing key allows different occurrences of the same token to select different expert sets.

W2: Variance Across Runs and Confidence Intervals

Tables 1 and 2 report point estimates on the benchmark test sets. We use greedy decoding, which removes variation from stochastic generation for a fixed checkpoint and prompt. Uncertainty due to the finite test sets and variation across independent pretraining runs are separate questions. For MGSM-Zh, we average over four evaluation seeds for each checkpoint.

W3: Evidence for Lexical Anchoring and Contextual Specialization

We thank the reviewer for pointing out the need for direct evidence supporting the proposed transition from lexical anchoring to contextual specialization.

To address this concern, we provide complementary evidence from routing-schedule ablations and direct expert-behavior analysis. First, as reported in Section 4.3 (Page 8, Table 2) and Figure 3 (Page 9), we compare ProGram-123 with ProGram-321 and ProGram-223. ProGram-123 achieves the best performance on 9 out of 12 benchmarks, indicating that the performance advantage depends not merely on introducing higher-order n-gram routing, but also on how routing granularity is organized across layers.

More importantly, Appendix D.2 (Pages 19–20, Table 5) provides a direct analysis of layer-wise expert behavior. For the same token appearing in different contexts, the routed-output cosine similarities are 0.9980, 0.1586, and 0.1207 in the first, second, and third layers, respectively, indicating progressively stronger context dependence in the routed computations. We further conduct controlled expert-swap interventions. Replacing first-layer experts with those selected for different tokens increases next-token NLL by 0.4926 nats (95% CI: [0.4254, 0.5584]), supporting the functional role of the token-level lexical anchor. In the second layer, replacing experts with those selected for the same token under different preceding contexts increases NLL by 0.0369 nats (95% CI: [0.0222, 0.0525]), demonstrating a measurable functional contribution from contextual information.

Taken together, these results support the interpretation that ProGram-123 establishes a functionally meaningful token-level routing anchor and progressively incorporates local contextual information into subsequent layers. We have included these analyses and clarified their implications in Section 4.3 (Page 8), Figure 3 (Page 9), and Appendix D.2 (Pages 19–20).

[Response to Reviewer GhZS (2/2)]

W4: Computational Cost

We thank the reviewer for pointing out the need for empirical measurements of computational overhead.

To address this concern, we have added end-to-end training latency and throughput measurements in Appendix B (Page 14). Specifically, Figure 4 reports the absolute iteration time and throughput of ProGram-123 and Learned-MoE, while Figure 5 compares the relative computational costs of ProGram-123, ProGram-234, Dense-First-1, and Learned-MoE. Over steps 101–3000, ProGram-123 achieves an average iteration time of 1248.48 ms and a throughput of 420,291 tokens/s, compared with 1249.47 ms and 420,076 tokens/s for Learned-MoE.

These measurements show that ProGram-123 not only introduces no measurable end-to-end training overhead compared with Learned-MoE, but also achieves slightly lower iteration latency and higher throughput in our experiments.

Q1: Please Refer to the Weaknesses

We have addressed the individual concerns above, including the modular limitation of unigram hashing, pretraining seed sensitivity, the distinction between routing-key statistics and expert specialization, and the computational cost of ProGram.

Q2: Explanation of Sequence-Level Load-Balancing Loss

Thank you for asking us to clarify the purpose of this metric. We report it because all routing variants use the same load-balancing objective during pretraining. It lets us compare how well the variants satisfy that objective as training progresses. Computing the loss separately for each sequence captures concentration within individual sequences, which can be obscured when expert usage is aggregated across a batch.

For each expert, the loss multiplies its normalized selection frequency in the auxiliary routing map by its mean router score over the sequence, then sums these products across experts. A lower value indicates less overlap between frequently selected experts and experts receiving high average router scores. In the hash-routed layers, the hash function determines the experts that actually process each token. We therefore report MaxVio on the actual expert assignments in Appendix D.1 as a complementary measurement.

In Figure 2(b), ProGram-123 achieves lower sequence-level balancing loss than both fixed 2-gram hashing and ProGram-234 at the 30B, 60B, and 100B checkpoints. This shows that ProGram-123 better satisfies the shared auxiliary objective at the reported checkpoints. Its stronger overall downstream performance in Table 2 provides separate evidence in favor of the progressive (1→2→3) schedule.

Update on W2 — Variation across training seeds

Thank you for raising the concern about variation across runs. To further assess robustness to random initialization, we have now completed the downstream evaluation of three independent ProGram-123 pretraining runs with random seeds 42, 43, and 44, using the same evaluation protocol across all 12 benchmarks.

SeedMMLU-ProMMLU-ReduxMLQA-Zh-ZhINCLUDEGPQAGSM8KMGSMBBHMBPPMBPP+HumanEvalRepoBench
429.9926.9415.9032.3829.245.168.8021.6210.6016.1413.416.60
4310.2722.4214.7228.3926.564.628.0022.0712.2017.2012.206.19
448.9323.4515.9827.5926.124.888.8020.6511.4015.6111.597.38
Mean9.7324.2715.5329.4527.314.898.5321.4511.4016.3212.406.72
Std0.581.930.582.101.380.220.380.590.650.660.760.49

The standard deviation is below one score point on 9 of 12 benchmarks, with larger variation on MMLU-Redux, INCLUDE, and GPQA. Rankings vary across tasks, and each seed achieves the highest score on at least one benchmark. These downstream results complement the closely aligned pretraining LM-loss trajectories in Appendix B, Figure 6, and provide evidence of robustness to random initialization across the three tested seeds.

Reviewer_vrVgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem that existing token-ID-based hash routing for expert selection in sparse MoE can produce imbalanced assignments and degrade training performance, by using n-gram hash inputs that consider preceding tokens in addition to the current token. The authors first show mathematically that refining keys by context reduces key concentration and, under an idealized hash-assignment assumption, also reduces load imbalance. They also conduct experiments on a learned MoE, models with early layers replaced by dense layers, and progressively increasing or fixed n-gram settings such as 1,2,3 and 1,1,1, evaluating LM loss and downstream task performance. Based on this, they propose ProGram123, which applies 1-, 2-, and 3-gram routing to the first three MoE layers respectively, as the best method. The contribution is proposing the perspective of designing hash-routing keys to consider progressively more context with layer depth, and demonstrating it experimentally.

Critical strengths

S1. Clear problem formulation. The paper presents and addresses a clear limitation of fixed token-level routing, namely that it cannot distinguish contexts and can concentrate load on particular experts.

S2. Intuitive and extensible framing. The perspective of designing hash-routing keys according to layer depth is conceptually clear and can be readily applied to other MoE architectures.

Critical weaknesses

W1. The effect of progressive expansion is not verified in isolation. The paper explains the advantage of ProGram123 as “progressive context expansion with depth,” but the current comparisons cannot distinguish the effect of fixing the expert assignment in the first layer, the effect of using n-gram keys, and the effect of progressively increasing n. Additional settings are needed to separate the contributions of using n-grams and of progressive expansion: a reversed schedule (3,2,1), a variant that changes only the first layer (2,2,3), and a setting that applies token-ID hash routing to the first layer only while keeping learned routing in the remaining layers.

W2. The paper’s theory does not explain the performance advantage. The theory shows that refining routing keys reduces concentration and that expected load imbalance decreases under an idealized hash. However, in global-batch MaxVio, fixed 3/4-gram and ProGram234 are more balanced than ProGram123, yet ProGram123 achieves higher downstream performance. Thus load mitigation alone cannot explain the superiority of ProGram123. The authors additionally explain that ProGram123 is superior because “it first establishes a token-level partition and then progressively adds context, allowing experts to specialize well,” but since there is no measurement or analysis of expert specialization, this explanation remains an unverified hypothesis.

Questions for authors

Q1. Have you tested schedules with a different order (e.g., 3,2,1 or 2,2,3), or a setting that applies token-ID hash routing to the first layer only while keeping learned routing in the remaining layers? If so, how do their LM loss and downstream performance compare to 1,2,3? Q2. Were all models trained with a single training seed? For benchmarks with large variance across schedules, such as BBH, how much variance is observed when only the training seed is changed? Q3. In Figure 3, what is the reference for “Delta LM Loss”? The value for ProGram123 appears to differ between (a) and (b). Q4. How was the Learned-MoE baseline configured (e.g., load-balancing coefficient and router settings)? Given that it underperforms Dense-First, can it be considered a sufficiently tuned learned MoE?

5 comments

[Response to Reviewer vrVg (1/2)]

We thank the reviewer for the positive assessment of our work and for recognizing the clear problem formulation and the intuitive, extensible perspective of context-aware hash routing. We appreciate the constructive comments, which help us clarify the scope of our theoretical analysis and the rationale behind the progressive routing design. We address the concerns point by point below.

W1 & Q1: Isolating the Effect of Progressive Context Expansion

We thank the reviewer for suggesting these additional controlled comparisons. We have incorporated all three suggested configurations into the revised manuscript, including First-Layer Hash, ProGram-223, and ProGram-321. Their results are now reported in Table 2 and Figure 3(b).

Specifically, these experiments examine three complementary aspects of the routing design:

  1. First-Layer Hash: We apply unigram hash routing only to the first MoE layer and retain learned routing in all subsequent layers. This configuration directly tests whether the benefit can be attributed solely to introducing deterministic hash routing in the first layer. ProGram-123 achieves substantially stronger overall downstream performance, indicating that extending contextualized hash routing across multiple shallow layers is important.

  2. ProGram-223: We adopt the schedule (2,2,3), which introduces a progressive change in routing granularity but does not start from unigram routing. ProGram-123 consistently outperforms this configuration across the benchmark suite, suggesting that progressive expansion alone is insufficient; the initial token-level routing partition also plays an important role.

  3. ProGram-321: We reverse the proposed schedule and use (3,2,1), directly examining whether the ordering of routing granularities matters. ProGram-321 achieves the highest score on HumanEval, while ProGram-123 achieves higher scores on the remaining 11 benchmarks. This comparison provides direct empirical evidence that the proposed coarse-to-fine ordering is preferable to the reversed fine-to-coarse ordering under our experimental setting.

Furthermore, Figure 3(b) compares the language-modeling loss trajectories of these additional configurations throughout pretraining. The results are consistent with the downstream evaluation, with ProGram-123 maintaining a more favorable loss trajectory over most of the training process.

Taken together, these additional experiments provide a more systematic separation of three factors: the number of shallow layers using hash routing, the initial routing granularity, and the ordering of n-gram granularities across depth. We have incorporated these results into Section 4.3 and expanded the corresponding discussion. We believe they substantially strengthen the empirical evidence supporting the proposed progressive routing design.

W2: The Relationship Between Load Balancing and Downstream Performance

Expert-load balance captures one aspect of MoE pretraining. Compared with fully learned routing, ProGram-123 achieves lower sequence-level load-balancing loss and lower per-micro-batch MaxVio during pretraining (Figures 2(a) and 7(a)). Among the hash-routing schedules, ProGram-234 achieves more uniform expert loads than ProGram-123 at the 100B-token checkpoint (global-batch MaxVio of 0.071 versus 0.486 in the first three MoE layers), while ProGram-123 scores higher on 10 of 12 downstream benchmarks (Figure 7(b) and Table 2). This comparison supports evaluating the structure of expert assignments alongside their balance. ProGram-123 starts with a lexical routing key and progressively introduces bigram and trigram context. Its downstream results are consistent with our hypothesis that gradually incorporating local context across shallow layers is beneficial.

[Response to Reviewer vrVg (2/2)]

Q2: Training Seeds and Result Variance

We added a comparison of ProGram-123 pretraining runs with seeds 42, 43, and 44 in Appendix B. Their LM-loss trajectories are closely aligned over the shared training interval of approximately 74.5B tokens. Over the last 5,000 steps of this interval, the mean LM losses computed from the unsmoothed logs are 2.0053, 2.0089, and 2.0080, respectively. The largest gap is 0.0036, approximately 0.18%, indicating that pretraining LM loss varies little across the three tested seeds over the observed interval. Once the additional runs finish training, we will evaluate their final checkpoints on the same downstream benchmarks and report the variation across seeds.

Q3: Reference of Delta LM Loss in Figure 3

Thank you for pointing out this ambiguity. We have revised the Figure 3 caption to explicitly specify the reference for each subfigure.

Specifically:

  • Figure 3(a): The zero reference line is the pointwise mean of ProGram123 and Dense-First-3.
  • Figure 3(b): The zero reference line is the pointwise mean of ProGram123 and fixed 2-gram Hash.

Therefore, the Delta LM Loss values of ProGram123 differ between the two subfigures because they are calculated relative to different reference configurations. We have explicitly clarified these references in the revised figure caption.

Q4: Configuration of the Learned-MoE Baseline

We tuned the Learned-MoE baseline before running the controlled comparison. The final configuration uses the same 16-layer, 5B-parameter backbone as ProGram-123, with 192 routed experts and one shared expert per layer. Learned-MoE selects six routed experts per token using learned top-k routing in all 16 layers. The router computes scores in FP32 using softmax with a temperature of 1.0 and a top-k scaling factor of 2.45. The sequence-level load-balancing auxiliary loss has a coefficient of 0.001, and the expert bias update rate is 0.001. Learned-MoE and ProGram-123 share the same pretraining recipe, including a peak learning rate of 0.0005 with a warmup–stable–decay schedule.

Under the same training token budget, the Dense-First variants outperform Learned-MoE on most benchmarks, while Learned-MoE outperforms both on GPQA and MGSM. Dense-First replaces shallow MoE layers with dense FFNs, so it provides a comparison against a different shallow-layer architecture.

The two comparisons in Table 1 address complementary questions. The comparison with Learned-MoE isolates the change in expert selection in the first three layers under the same MoE backbone and pretraining recipe. The comparison with Dense-First evaluates the full shallow hash-routing design against dense replacement. ProGram-123 outperforms both Dense-First variants on 10 of the 12 benchmarks. We will report the router configuration and learning-rate schedule in the revised manuscript.

Additional Experimental Results

We thank the reviewers for suggesting further routing ablations and evaluation across training seeds. We provide the additional routing comparisons from Table 2, followed by the newly completed downstream evaluation of ProGram-123 with seeds 42, 43, and 44.

Additional results for W1 & Q1 — Routing ablations

The revised Table 2 includes First-Layer Hash, ProGram-223, and ProGram-321. We reproduce their results alongside ProGram-123 for convenient comparison. Bold denotes the highest score in each column among the four configurations shown.

MethodMMLU-ProMMLU-ReduxMLQA-Zh-ZhINCLUDEGPQAGSM8KMGSMBBHMBPPMBPP+HumanEvalRepoBench
First-Layer Hash5.8624.439.4626.7128.793.715.2010.044.203.172.445.80
ProGram-2238.0524.2413.6126.0822.109.868.4015.148.6010.329.765.99
ProGram-3219.0115.1213.3725.5129.024.175.6019.949.6013.7614.026.08
ProGram-1239.9926.9415.9032.3829.245.168.8021.6210.6016.1413.416.60

ProGram-123 scores higher than First-Layer Hash on all 12 benchmarks and higher than ProGram-223 and ProGram-321 on 11 of 12 each. The comparison with ProGram-223 supports the unigram starting point while holding the second- and third-layer routing orders fixed. The comparison with ProGram-321 supports the coarse-to-fine ordering while preserving the same three routing orders. Section 4.3 and Figure 3(b) provide the corresponding discussion and pretraining LM-loss trajectories.

Update on Q2 — Downstream performance across training seeds

We have now completed the downstream evaluation of three independent ProGram-123 pretraining runs with random seeds 42, 43, and 44. Seed 42 corresponds to the original checkpoint. The table reports all three runs and their mean and population standard deviation across seeds.

SeedMMLU-ProMMLU-ReduxMLQA-Zh-ZhINCLUDEGPQAGSM8KMGSMBBHMBPPMBPP+HumanEvalRepoBench
429.9926.9415.9032.3829.245.168.8021.6210.6016.1413.416.60
4310.2722.4214.7228.3926.564.628.0022.0712.2017.2012.206.19
448.9323.4515.9827.5926.124.888.8020.6511.4015.6111.597.38
Mean9.7324.2715.5329.4527.314.898.5321.4511.4016.3212.406.72
Std0.581.930.582.101.380.220.380.590.650.660.760.49

The models exhibit relatively consistent downstream performance across the evaluated tasks. The standard deviation is below one score point on 9 of 12 benchmarks, with larger variation on MMLU-Redux, INCLUDE, and GPQA. Rankings vary by task, and each seed achieves the highest score on at least one benchmark. Together with the closely aligned pretraining LM-loss trajectories in Appendix B, Figure 6, these results provide preliminary evidence of robustness to random initialization across the three tested seeds.

Reviewer_T9GoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Previous studies have shown that using learned top-K routing for ALL layers is not optimal in MoE models, and token-ID hashing routing is an efficent alternative. However, token-ID hashing is context-blind and reuses the same lexical routing key across shallow layers. To solve this problem, the authors propose PROGRAM, a depth-aware n-gram hash routing method.

Critical strengths

The paper is easy to follow, the motivation is clear and reasonable.

The depth-aware routing-key design is novel to me.

Critical weaknesses

The controlled experiments use a 16-layer Transformer backbone. The effectiveness of the proposed PROGRAM on larger scale models is unclear.

The experimental settings for learning based MoE models are unclear.

How to select numbers of n-gram hashing layers is also not clear.

The Token-ID Hash (table 2) used by deepseek-v4 seems to be worse than LEARNED-MOE (table 1). How to explain this results?

Questions for authors

Please see the Weaknesses.

2 comments

Thank you for your positive assessment of the paper’s clarity and motivation, and for highlighting the novelty of the depth-aware routing-key design. We appreciate the opportunity to clarify the scope of our experiments, the Learned-MoE baseline, and our choice of hashing depth.

W1. Scalability to larger models.

We agree that whether ProGram’s gains extend to larger and deeper MoE models remains an open empirical question. Our controlled experiments use a 16-layer MoE with approximately 5B total parameters and 412M active parameters per token, pretrained on approximately 100B tokens. In total parameter count, this model falls between DeepSeekMoE 2B (Dai et al., 2024) and OLMoE-1B-7B (Muennighoff et al., 2025). ProGram-123 and Learned-MoE share the backbone, pretraining recipe, and token budget. ProGram-123 scores higher on 11 of 12 downstream benchmarks and ties on the remaining one (Table 1).

W2. Learned-MoE configuration.

Learned-MoE uses learned top-KK expert selection in all 16 layers. Each layer has 192 routed experts and one shared expert, and each token activates six routed experts. The router computes softmax scores in FP32 with a temperature of 1.0 and a top-KK scaling factor of 2.45. The sequence-level load-balancing auxiliary loss has a coefficient of 0.001, and the expert-bias update rate is 0.001. Both models use the same pretraining data, tokenizer, and optimizer settings, with a peak learning rate of 5×10−45\times10^{-4}, a warmup–stable–decay schedule, a global batch size of 64 sequences, and a sequence length of 8,192 tokens. In ProGram-123, only the rule for selecting expert IDs changes in the first three layers. The selected experts still receive learned mixture weights. Section 4.1 and Appendix A.2 (Table 3) describe the matched setup. We have added the Learned-MoE router settings and shared pretraining hyperparameters to Appendix A.2 of the revised manuscript.

W3. Choosing how many layers to hash.

We use the first three MoE layers as a shallow hash-routed block, following the placement adopted in DeepSeek-V4. This gives us a block in which to expand the routing key from the current token in layer 1 to the current token and its two preceding tokens in layer 3. All subsequent layers retain learned top-KK expert selection.

To examine how the number of hash-routed layers affects performance, we added First-Layer Hash, which uses unigram hashing only in the first MoE layer and learned routing thereafter. With the routing order held fixed at unigram, First-Layer Hash outperforms three-layer Token-ID Hash on 8 of 12 benchmarks (Table 2). Extending unigram routing to more layers therefore yields no consistent improvement. ProGram-123, which assigns unigram, bigram, and trigram keys to the first three layers, outperforms First-Layer Hash on all 12 benchmarks and ranks first on 9 of 12 among the evaluated hash-routing configurations. These results support using the first three layers with the (1,2,3)(1,2,3) schedule on our 16-layer backbone.

W4. Token-ID Hash versus Learned-MoE.

We thank the reviewer for raising this question. We would first like to clarify an important distinction: the Token-ID Hash baseline in Table 2 is not an exact reproduction of the tid2eid lookup-table routing used in DeepSeek-V4. Although both approaches employ deterministic token-identity-based expert assignment, their implementations differ. DeepSeek-V4 retrieves expert assignments from a precomputed, frozen token-to-expert mapping table, whereas our baseline computes candidate expert IDs online using multi-prime modular hashing, followed by stable deduplication and fallback probing. Therefore, our Token-ID Hash baseline should not be interpreted as a direct reproduction or evaluation of DeepSeek-V4's original routing implementation.

Moreover, Token-ID Hash is not consistently inferior to Learned-MoE. As shown in Table 2, it outperforms Learned-MoE on 7 out of 12 benchmarks, while Learned-MoE performs better on the remaining 5. These results indicate task-dependent differences rather than a uniform disadvantage of deterministic token-based routing.

More importantly, the key distinction between Token-ID Hash and ProGram lies in how routing granularity evolves across shallow layers. Token-ID Hash applies the same unigram-based routing granularity across all three shallow layers, whereas ProGram progressively introduces contextual information through unigram, bigram, and trigram routing, respectively. This enables subsequent layers to distinguish different local contexts associated with the same current token, rather than relying exclusively on token identity throughout the shallow routing block.

The superior performance of ProGram-123 over both Token-ID Hash and Learned-MoE under our experimental setting supports the effectiveness of progressively contextualized routing. We have clarified this implementation difference in the Experimental Setup section on Page 7 of the revised manuscript.

Additional results for W3 — Choosing the number of hash-routed layers

We reproduce three configurations from Table 2 to clarify the role of hashing depth. First-Layer Hash applies unigram hashing only in the first MoE layer. Token-ID Hash applies unigram hashing in all three shallow MoE layers. ProGram-123 uses unigram, bigram, and trigram keys in those three layers, respectively. All subsequent layers retain learned routing.

MethodMMLU-ProMMLU-ReduxMLQA-Zh-ZhINCLUDEGPQAGSM8KMGSMBBHMBPPMBPP+HumanEvalRepoBench
First-Layer Hash5.8624.439.4626.7128.793.715.2010.044.203.172.445.80
Token-ID Hash4.5825.019.1328.8925.002.422.0010.984.008.471.835.20
ProGram-1239.9926.9415.9032.3829.245.168.8021.6210.6016.1413.416.60

With the hash order held fixed at unigram, First-Layer Hash outperforms three-layer Token-ID Hash on 8 of 12 benchmarks. Extending unigram hashing to more layers therefore yields mixed results. ProGram-123 outperforms both configurations on all 12 benchmarks, supporting the combined choice of a three-layer hash-routed block and the progressive (1,2,3) schedule under our training setup.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of the paper’s motivation, depth-aware routing design, and empirical results. We address two concerns raised across the reviews about the empirical justification for the progressive schedule and the mechanisms underlying its effectiveness. We expanded the routing ablations and added an analysis of expert behavior, with new material marked in blue.

G-1. Is the Progressive Routing Design Empirically Justified?

We added First-Layer Hash, ProGram-223, and ProGram-321 to examine the hashing depth, starting routing granularity, and ordering of routing keys across shallow layers (Section 4.3, Table 2, and Figure 3(b)).

  • First-Layer Hash uses unigram hashing only in the first MoE layer and retains learned routing thereafter. ProGram-123 scores higher on all 12 benchmarks, supporting the combined design of extending hashing to layers 2 and 3 and using bigram and trigram keys in those layers.
  • ProGram-223 shares the second- and third-layer routing orders with ProGram-123 and changes the first layer from unigram to bigram. ProGram-123 scores higher on 11 of 12 benchmarks, supporting the unigram starting point.
  • ProGram-321 uses the same three routing orders in reverse. ProGram-123 scores higher on 11 of 12 benchmarks, supporting the coarse-to-fine ordering across layers.

The results below reproduce the corresponding rows of Table 2. An asterisk (*) marks the highest score in each column among the four methods shown.

MethodMMLU-ProMMLU-ReduxMLQA-Zh-ZhINCLUDEGPQAGSM8KMGSMBBHMBPPMBPP+HumanEvalRepoBench
First-Layer Hash5.8624.439.4626.7128.793.715.2010.044.203.172.445.80
ProGram-2238.0524.2413.6126.0822.109.86*8.4015.148.6010.329.765.99
ProGram-3219.0115.1213.3725.5129.024.175.6019.949.6013.7614.02*6.08
ProGram-1239.99*26.94*15.90*32.38*29.24*5.168.80*21.62*10.60*16.14*13.416.60*

Figure 3(b) complements these downstream comparisons. ProGram-123 maintains lower language-modeling loss than the three added configurations over most of the plotted training interval. Together, these observations support the (1,2,3) schedule among the configurations evaluated.

G-2. Why Does the (1,2,3) Routing Schedule Work?

We examine the effectiveness of the proposed schedule from three perspectives.

Language-modeling dynamics. ProGram-123 maintains favorable LM-loss trajectories under the matched training setup. Figure 2(b) further shows lower sequence-level balancing loss than fixed 2-gram hashing and ProGram-234, indicating better alignment with the shared auxiliary objective.

Expert-load distribution. Figure 7 shows that hash-based routing generally achieves better expert-load balance than Learned-MoE in terms of MaxVio. However, better load uniformity does not necessarily yield better downstream performance. Although ProGram-234 achieves lower global-batch MaxVio than ProGram-123 (0.071 vs. 0.486), ProGram-123 outperforms it on 10 of 12 benchmarks. This highlights the importance of routing structure beyond load balance alone.

Expert behavior. Analysis of a trained checkpoint (Appendix D.2, Table 5) shows progressively stronger context dependence in routed outputs across the first three layers, with cross-document cosine similarities of 0.9980, 0.1586, and 0.1207, respectively. Expert-swap interventions further demonstrate the functional contributions of token-level anchoring and local-context adaptation, increasing next-token NLL by 0.4926 nats and 0.0369 nats, respectively.

Together, these findings motivate the ProGram-123 design: unigram routing establishes a stable token-level anchor, followed by bigram and trigram routing that progressively incorporate local context, while deeper layers retain representation-dependent learned routing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.