acceptodds
Under review as a conference paper at ICLR 2027

RecipeMoE: Learning Reusable Block Recipes for Full-Participation Mixture-of-Experts

Abstract

Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. Participation is how many experts are involved in producing a token's output, execution is how many are evaluated on the token, and materialization is how many expert-sized parameter sets must be built. Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts participate. Dense output-mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at one expert, but its materialization grows with the number of routing decisions. We propose RecipeMoE, a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution. Its cacheable blocks come from a small learned codebook, one per entry. A lightweight hypernetwork generates block-specific composition recipes that merge each internal layer's expert pool into a composed expert. Participation is full, because every composed expert is synthesized from the entire pool. Execution stays sparse, because a router sends each token to only a few blocks. Materialization is bounded, because the codebook, not the input, fixes how many blocks exist. Dual-Path Residual Gating (DPRG) enables nonlinear interactions between two paths composed from the same expert pool using independent recipes. Experiments on image classification show consistent gains over representative sparse and dense MoE baselines. Additional experiments on language modeling and sequential recommendation validate its generalization beyond vision. RecipeMoE is fully deployed in a large-scale industrial generative recommendation system, serving hundreds of millions of users under a 60 ms latency budget, with a 2.4% relative UVCTR gain in online A/B testing. Code is available at https://anonymous.4open.science/r/RecipeMoE-717C254/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Expert Composition, Conditional Computation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BWwdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RecipeMoE learns a small set of input-independent recipes, each of which mixes a shared pool of expert parameters into a reusable block. Each token routes to a few of these blocks, which have separate value and gating paths, plus a shared expert. Because the recipes do not depend on the input, the blocks can be composed once after training and cached, and the pools can then be dropped.

Critical strengths
  1. The construction has a clear and useful deployment property. Once the blocks are precomputed, inference cost no longer grows with the size of the expert pool, which input-dependent merging methods like SMEAR can't offer.

  2. The accuracy gains are consistent: about 2 points over SMEAR on ImageNet with low variance, and first in every seed across the three datasets. The Direct Blocks ablation is a nice touch. Training the same final blocks directly does worse, so going through the pool seems to help during training.

Critical weaknesses
  1. The baselines are matched on parameters, not compute. Cached RecipeMoE needs a bit over twice the FLOPs of SMEAR and about 1.8x those of V-MoE. I understand the activated parameter counts are similar, and it does beat DynMoE, which costs more. Still, the extra compute comes mostly from running several two-path blocks and a shared expert, so I can't tell how much of the gain over SMEAR is the method and how much is just more computation. There is also no compute or latency information for MiniPile and IntTravel, and the IntTravel improvements are quite small.

  2. It's not clear why a hypernetwork is needed to generate the recipes. At inference the model is just a fixed set of blocks. Direct Blocks shows that going through the pool helps, but the simpler option of learning the mixing coefficients directly isn't tested. Direct Blocks itself is only run on ImageNet (the appendix explains why, which is fair), and the ablations on the other two datasets are single runs with some very small differences.

Questions for authors
  1. How do SMEAR and V-MoE do when scaled up to similar FLOPs? Latency on the same hardware would also help.

  2. What happens if the hypernetwork is replaced by a directly learned coefficient table, with everything else unchanged? Could you report the size and FLOPs of Direct Blocks and run it on one more dataset?

  3. For the A/B test: what were the latency and resource cost of the control model, how was traffic split, and is there a confidence interval on the 2.4% gain? Since the control has no MoE layer, this mostly shows that adding RecipeMoE helps, not that it beats other MoE designs.

7 comments
AuthorsOpenReview ↗

[1/2] Official Comment by Authors

To Q1 and W1.1:

The baselines are matched on parameters, not compute.

Thank you for suggesting a complementary comparison under matched FLOPs. In addition to our parameter-matched experiments, we evaluate the following methods on ImageNet-1K at comparable inference FLOPs by adjusting expert FFN widths while retaining the remaining training settings. Results are reported as mean ± sample standard deviation over three seeds. RecipeMoE’s FLOPs correspond to cached inference.

MethodParams (M)FLOPs (G)ACC@1 (%)ACC@5 (%)
Switch Transformer111.0003.45772.77 ± 0.2990.57 ± 0.38
DeepSeek-V3 MoE39.5003.45772.36 ± 0.7390.62 ± 0.34
Soft MoE108.9003.45872.55 ± 0.0790.13 ± 0.09
SMEAR102.7003.45973.99 ± 0.2891.20 ± 0.22
Lory52.8003.45872.20 ± 0.1290.36 ± 0.25
V-MoE48.1003.45873.32 ± 0.1090.99 ± 0.13
Expert Choice55.4003.45873.46 ± 0.3891.06 ± 0.23
MoBE-style (trained from scratch)54.9903.45772.03 ± 0.2389.62 ± 0.14
RecipeMoE (cached)24.2953.45773.78 ± 0.0991.47 ± 0.15

At comparable inference FLOPs, RecipeMoE achieves the second-highest mean ACC@1 and the highest mean ACC@5 with substantially fewer trainable parameters: 24.30M versus 39.50–111.00M for the evaluated baselines, a reduction of 38.5–78.1%. It improves mean ACC@1 over V-MoE and the MoBE-style baseline by 0.46 and 1.76 percentage points, respectively. Compared with SMEAR, which uses over four times as many trainable parameters (102.70M vs. 24.30M), RecipeMoE achieves only 0.21 percentage points lower mean ACC@1 and 0.27 percentage points higher mean ACC@5 at comparable inference FLOPs.

SMEAR’s large parameter budget is enabled in part by example-level composition in the visual setting, where one merged expert is reused across all tokens in an image. In the causal language-modeling setting, SMEAR instead performs token-level composition, increasing synthesis costs and reducing the parameter budget affordable at matched FLOPs. RecipeMoE already uses token-level routing in both visual and causal settings. Moving to a causal setting therefore introduces no additional per-token composition cost.

To Q2.1 and W2.1:

Report the parameter counts and FLOPs for both Directly Learned Coefficients and Direct Blocks.

We replaced the hypernetwork with directly learned composition coefficients, retaining the same expert pools, block structure, and routing configuration. This comparison isolates the contribution of the hypernetwork.

MethodParams (M)FLOPs (G)ACC@1 (%)ACC@5 (%)
Direct Blocks24.2903.45773.15 ± 0.1191.05 ± 0.16
Directly learned coefficients (cached)24.2913.45773.69 ± 0.0391.35 ± 0.13
RecipeMoE (cached)24.2953.45773.78 ± 0.0991.47 ± 0.15

With directly learned coefficients, each block has its own value and gate coefficient vectors. With a shared hypernetwork, these vectors are generated from learned block embeddings through a common mapping. Updating this mapping can affect recipes across multiple blocks, introducing coupling at the recipe level in addition to the sharing already provided by the expert pools. In our architecture, the same block embedding also conditions feature filtering. Consequently, recipe generation and feature filtering jointly shape the block representation through the task objective. This provides a mechanism for coordinating how a block is constructed with how its input features are filtered.

To Q3:

The A/B test’s latency, resource costs, traffic split, and confidence interval.

The control uses the same backbone without an MoE module. Under the same serving conditions, its mean/P99 latency is 17/31 ms, compared with 19/38 ms for RecipeMoE. Both the control and RecipeMoE use the same number of GPUs.

The one-week A/B test randomly assigned users to the two groups with equal traffic allocation. The serving load reported in the paper is approximately 5,000 QPS. RecipeMoE achieved a 2.4% relative improvement in UV CTR, with a 95% confidence interval of [1.29%, 3.67%]. The 95% confidence interval was estimated using a log risk-ratio normal approximation, treating the deduplicated user-level click outcomes in the control and treatment groups as independent binomial observations.

Deploying a new model in our mobility recommendation service requires substantial engineering and validation, including runtime optimization, manual evaluation, smoke testing, and staged rollout. RecipeMoE achieved the best offline performance among the evaluated methods and an observed 2.4% relative UV CTR improvement over the production baseline in online A/B testing. Together, these results provide evidence of RecipeMoE’s effectiveness and practical value in a production recommendation system.

AuthorsOpenReview ↗

[2/2] Official Comment by Authors

To W1.2:

There is also no compute or latency information for MiniPile and IntTravel.

Additional efficiency measurements.

We supplement the task metrics with parameter counts and inference FLOPs for both datasets.

MiniPile

MethodParams (M)FLOPs (G/sequence)Loss ↓PPL ↓
Dense152.006369.2332.8168 ± 0.003616.7230 ± 0.0607
Switch Transformer832.230369.4882.7164 ± 0.001915.1265 ± 0.0292
DeepSeek-V3 MoE831.594717.3522.7078 ± 0.006614.9968 ± 0.0993
ReMoE832.230369.4882.7346 ± 0.001015.4034 ± 0.0152
DynMoE831.608786.9592.7115 ± 0.008415.0520 ± 0.1268
MASS831.608665.1962.7513 ± 0.010015.6636 ± 0.1559
Soft MoE (causal)832.2301761.0572.7129 ± 0.003515.0723 ± 0.0534
SMEAR (token-level)832.2301763.6052.7133 ± 0.002915.0789 ± 0.0442
Lory831.608467.0942.7595 ± 0.008615.7925 ± 0.1367
μMoE (CP)825.9801749.5312.7075 ± 0.002414.9925 ± 0.0366
μMoE (TR)825.9021751.5782.7186 ± 0.002115.1590 ± 0.0311
RecipeMoE825.9841146.6572.6789 ± 0.001114.5696 ± 0.0161
RecipeMoE (cached)1455.5361126.5102.6789 ± 0.001114.5696 ± 0.0161

IntTravel

MethodParams (M)FLOPs (G)HR@1 (%) ↑HR@5 (%) ↑NDCG@5 (%) ↑
Dense0.6400.29367.50 ± 0.0286.57 ± 0.0277.87 ± 0.02
Switch Transformer22.3740.41268.01 ± 0.1086.66 ± 0.0478.15 ± 0.08
DeepSeek-V3 MoE22.2300.85368.13 ± 0.0786.77 ± 0.0378.24 ± 0.05
ReMoE22.3740.63368.33 ± 0.0186.85 ± 0.0178.36 ± 0.01
DynMoE22.3741.95468.16 ± 0.0486.79 ± 0.0378.26 ± 0.01
MASS22.3741.51468.22 ± 0.0486.81 ± 0.0278.30 ± 0.02
Soft MoE (causal)22.3747.46268.29 ± 0.0486.81 ± 0.0278.33 ± 0.02
SMEAR (token-level)22.3747.73068.27 ± 0.0386.79 ± 0.0378.31 ± 0.04
Lory22.2310.89868.12 ± 0.0186.73 ± 0.0478.21 ± 0.02
μMoE (CP)22.8747.67568.35 ± 0.0386.84 ± 0.0278.37 ± 0.02
μMoE (TR)22.8767.73968.38 ± 0.0286.85 ± 0.0178.38 ± 0.03
RecipeMoE22.8752.17268.52 ± 0.0186.92 ± 0.0178.50 ± 0.03
RecipeMoE (cached)12.1741.48868.52 ± 0.0186.92 ± 0.0178.50 ± 0.03

RecipeMoE achieves the best mean task metrics on both datasets. Cached inference uses fewer FLOPs than Soft MoE, token-level SMEAR, and both μMoE variants, although several sparse-routing baselines remain cheaper.

IntTravel improvements.

The gains on IntTravel are less pronounced than those on ImageNet-1K and MiniPile, but remain consistent across seeds. Relative to the strongest baseline for each metric within each seed, RecipeMoE achieves the following improvements:

SeedHR@1 gain (pp)HR@5 gain (pp)NDCG@5 gain (pp)
420.150.070.13
49812130.170.070.15
2112312440.120.050.07

RecipeMoE outperforms all evaluated baselines on all three metrics in every seed. Together with the larger gains on the other two tasks, these results support its effectiveness across domains, with the magnitude of improvement varying by dataset.

AuthorsOpenReview ↗

Additional Experimental Results

Thank you again for your insightful feedback. We hope these additional experiments help address your concerns and look forward to further discussion.

1. Hypernetwork ablations on MiniPile and IntTravel.

We have completed three-seed comparisons between hypernetwork-generated recipes and directly learned composition coefficients on MiniPile and IntTravel. Loss and PPL are evaluated on MiniPile. HR@1, HR@5, and NDCG@5 are evaluated on IntTravel and reported in %. Results are mean ± sample standard deviation over three seeds.

VariantLoss ↓PPL ↓HR@1 ↑HR@5 ↑NDCG@5 ↑
Directly learned coefficients2.6947±0.001714.8006±0.024768.45±0.0586.89±0.0578.45±0.04
RecipeMoE2.6789±0.001114.5696±0.016168.52±0.0186.92±0.0178.50±0.03

On MiniPile, the hypernetwork reduces mean loss by 0.0158 and perplexity by 0.2310, with improvements in all three seeds. On IntTravel, it improves all three mean metrics, with HR@1 and NDCG@5 improving in every seed. Together with the ImageNet-1K results, these experiments support the benefit of shared hypernetwork-generated recipes across all three tasks.

2. Direct Blocks ablation and resource comparison on MiniPile.

We compare RecipeMoE with Direct Blocks, which removes the hypernetwork and expert pools and directly optimizes the routed block weights. To ensure a comparable evaluation, both variants use the same four-block architecture, with RecipeMoE composing these blocks from eight experts.

VariantParams (M)Inference FLOPs (G)Loss ↓PPL ↓
Direct Blocks825.9111126.3972.6950 ± 0.003114.8060 ± 0.0458
RecipeMoE825.9261136.4702.6842 ± 0.003214.6458 ± 0.0471
RecipeMoE (cached)—1126.397——

Task metrics are reported as mean ± sample standard deviation over three seeds. Params denotes trainable parameters. FLOPs are measured per 1,024-token sequence at batch size 1. Caching preserves RecipeMoE’s task metrics.

With nearly identical trainable parameter counts, RecipeMoE reduces mean loss by 0.0108 and perplexity by 0.1602, outperforming Direct Blocks on both metrics in every seed. After caching, RecipeMoE also matches Direct Blocks in inference FLOPs. These results support the benefit of learning routed blocks through shared expert composition without increasing cached inference computation.

3. Multi-seed component ablations on MiniPile.

We extend the original single-run MiniPile ablations with additional seeds. The table reports mean ± sample standard deviation.

VariantLoss ↓PPL ↓
1-Layer2.7162±0.002715.1226±0.0406
Fixed λ2.6916±0.002714.7548±0.0399
w/o Gate2.7651±0.004015.8812±0.0635
Tied Recipes2.6994±0.012414.8719±0.1854
w/o Shared2.7276±0.001615.2957±0.0241
w/o Filter2.6828±0.000514.6255±0.0079
Softmax Coeff.2.7088±0.001615.0106±0.0238
RecipeMoE2.6789±0.001114.5696±0.0161

The full model consistently achieves lower loss and perplexity than all ablation variants across the evaluated seeds. Removing the gate causes the largest performance drop, highlighting its importance in language modeling. Reducing block depth, tying the value and gate recipes, fixing λ, removing the shared expert, or applying softmax to the composition coefficients also degrades performance. Feature filtering provides a smaller but consistent improvement. These results reinforce the original ablation findings and show that the observed benefits persist across random seeds.

Reviewer_tHXFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript proposes RecipeMoE, an architecture combining three "positive" properties when training MoE architecture: full expert participation, sparse execution, and a bounded number of composed blocks. A hypernetwork learned compositional coefficients that combine shared expert parameters. Then Tokens are routed through a few resulting blocks, whose parameters can be cached for inference. Dual-Path Residual Gating (DPRG) combines separately composed value and gate transformations. Experiments on image classification, language modeling, and sequential recommendation report consistent improvements over the baselines, supplemented by an industrial deployment.

Critical strengths
  1. The problem proposed and the design are well executed. The identification of three components are nicely proposed and the design seems to address this adequately. Input-independent recipes allow blocks to be cached, and this offers a useful deployment advantage over input-conditioned composition.

  2. Consistent empirical results. Improvements across three tasks and multiple seeds support the approach. The ablation provides some good evidence for the benefit of composition, while the caching measurements and industrial deployment demonstrate practical applicability.

Critical weaknesses
  1. The novelty and motivation could be better developed. I am not yet fully convinced why combining those three properties of expert participation, spare execution, and bounded materialization is better for performance without any trade-off. The paper could compare with more relevant architecture to solidify its contribution beyond combining existing ideas. MoBE[1], for example, also represents expert weights using shared basis combinations, although it addresses pretrained-model compression. A more explicit comparison would help clarify RecipeMoE’s contribution. Furthermore, once blocks are cached, “full participation” additional learning benefit could be limited.

[1] MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs, ICLR 2026

  1. Some experimental details could be added The Direct Block ablation provides evidence for the necessity of parameter synthesis, but the hypernetwork itself is not compared with directly learned recipe coefficients. The manuscript could also have a more detailed analysis on load balancing and sparsity of execution. In addition, cached RecipeMoE still uses more FLOPs than several strong baselines. These issues limit my confidence in the architectural benefits.
Questions for authors

Clarifications

  1. How are composition coefficients initialized, and what motivates the 1/E1/\sqrt{E} scaling when their magnitudes are unconstrained?
  2. What is the main technical distinction from shared-basis approaches such as MoBE, beyond the training setting? What benefit should “full participation” provide over directly learned routed blocks?
  3. Could the authors mention limitation and failure cases of the method?

Experiments

  1. Could the authors compare hypernetwork-generated recipes with directly learned coefficients, and DPRG with a standard gated transformation?
11 comments
AuthorsOpenReview ↗

[1/4] Official Comment by Authors

To Q1:

Coefficient initialization and the 1/E1/\sqrt{E} scaling.

The composition coefficients are generated by the hypernetwork rather than independently initialized. Block embeddings are initialized from N(0,1)\mathcal{N}(0,1). The hypernetwork’s linear layers, including the value and gate output heads, use initialization U(−1/din,1/din)\mathcal{U}(-1/\sqrt{d_{\mathrm{in}}}, 1/\sqrt{d_{\mathrm{in}}}) for weights and biases. LayerNorm scales and biases are initialized to one and zero, respectively.

The 1/E1/\sqrt{E} factor compensates for scale growth when summing EE expert parameters. Under independent, zero-mean expert initialization, the composed variance is σW2∥α∥22/E\sigma_{W}^{2}\lVert\boldsymbol{\alpha}\rVert_{2}^{2}/E. Thus, it remains comparable across pool sizes when the average squared coefficient remains comparable. This provides the rationale for the scaling factor.

We avoid softmax to retain signed combinations whose coefficients need not sum to one, rather than restricting composition to a convex combination of expert parameters. On ImageNet-1K, replacing our coefficients with softmax-normalized coefficients reduces ACC@1 from 73.78 ± 0.09% to 73.30 ± 0.31% and ACC@5 from 91.47 ± 0.15% to 91.22 ± 0.26% across three seeds. These results support this design choice.

To Q2:

Differences from MoBE and the benefits of full participation.

Distinction from MoBE

Beyond the design philosophy and training setting, RecipeMoE differs from MoBE in three architectural aspects:

  1. Hypernetwork-generated recipes. MoBE directly learns expert-specific basis coefficients and transformation matrices. RecipeMoE uses a shared hypernetwork to generate composition coefficients from learned block embeddings. Compared with directly learned coefficients, the hypernetwork improves ACC@1 from 73.69 ± 0.03% to 73.78 ± 0.09%.
  2. DPRG. MoBE retains the original SwiGLU expert structure. RecipeMoE uses separate recipes over the same expert pool to construct value and gate transformations, coupled through residual multiplicative gating with a learnable scale. Removing the gate reduces ACC@1 from 73.78 ± 0.09% to 67.63 ± 0.12%, while using the same composition coefficients for both paths reduces it to 72.46 ± 0.31%, corresponding to mean drops of 6.15 and 1.32 percentage points, respectively.
  3. Multi-layer blocks. MoBE reparameterizes individual expert projections. RecipeMoE routes tokens to multi-layer blocks, with each internal layer composed from its own expert pool. Replacing the two-layer blocks with single-layer blocks while preserving the total expert parameter count reduces ACC@1 from 73.78 ± 0.09% to 69.13 ± 0.36%, a mean drop of 4.65 percentage points.

We additionally compare against a MoBE-style baseline on ImageNet-1K under comparable inference FLOPs. We retain MoBE’s shared-basis parameterization but train all parameters from scratch using the task loss, rather than reconstructing pretrained expert weights. Results are reported as mean ± sample standard deviation over three seeds. At matched inference FLOPs, RecipeMoE improves mean ACC@1 and ACC@5 by 1.76 and 1.84 percentage points, respectively, while using approximately 55.82% fewer trainable parameters (24.295M vs. 54.990M).

MethodParams (M)FLOPs (G)ACC@1 (%)ACC@5 (%)
MoBE-style (trained from scratch)54.9903.45772.03 ± 0.2389.62 ± 0.14
RecipeMoE (cached)24.2953.45773.78 ± 0.0991.47 ± 0.15

Benefit of full participation.

Full participation allows tokens routed to different blocks to provide training signals to a common expert pool. Unlike directly learned routed blocks, whose parameters are updated only by tokens selecting them, RecipeMoE constructs each block from shared expert parameters. Updating these parameters can therefore affect multiple blocks, enabling cross-block sharing while retaining block-specific composition recipes.

Our Direct Blocks ablation removes this composition mechanism and directly learns the routed block parameters. RecipeMoE achieves 73.78 ± 0.09% ACC@1 versus 73.15 ± 0.11% for Direct Blocks on ImageNet-1K, improving the mean by 0.63 percentage points and outperforming it in all three seeds. This supports the benefit of shared expert composition over directly learned routed blocks.

AuthorsOpenReview ↗

[2/4] Official Comment by Authors

To Q3:

The limitation of the method.

Cached blocks remain valid only while the expert parameters and composition recipes are unchanged. Deployments with frequent parameter updates therefore require cache reconstruction, and the efficiency benefit depends on how much the cached blocks can be reused between updates.

To Q4:

Hypernetwork and standard gated transformation comparisons.

We add four ablations on ImageNet-1K:

  • Directly learned coefficients: replace the hypernetwork with trainable composition coefficients while retaining expert composition.
  • Two-layer SwiGLU: replace each DPRG layer with a standard SwiGLU transformation.
  • Single-layer SwiGLU: replace the entire two-layer DPRG block with one standard SwiGLU transformation.
  • DPRG w/o residual term: replace v⊙(1+λSiLU(g))v \odot (1+\lambda\mathrm{SiLU}(g)) with v⊙(λSiLU(g))v \odot (\lambda\mathrm{SiLU}(g)), retaining the composed paths and learnable λ\lambda.

Both SwiGLU variants use adjusted hidden widths to match RecipeMoE’s parameter budget. Results are reported as mean ± sample standard deviation over three seeds.

VariantACC@1 (%)ACC@5 (%)
RecipeMoE73.78 ± 0.0991.47 ± 0.15
Directly learned coefficients73.69 ± 0.0391.35 ± 0.13
Two-layer SwiGLU69.19 ± 0.4289.20 ± 0.19
Single-layer SwiGLU73.45 ± 0.4091.40 ± 0.15
DPRG w/o residual term72.97 ± 0.0991.03 ± 0.01

The hypernetwork improves mean ACC@1 by approximately 0.10 percentage points over directly learned coefficients. DPRG outperforms the parameter-matched two-layer and single-layer SwiGLU variants by 4.59 and 0.34 percentage points, respectively. Removing its residual term reduces mean ACC@1 by 0.82 percentage points, supporting the residual formulation while retaining multiplicative nonlinearity.

With directly learned coefficients, each block has its own value and gate coefficient vectors. With a shared hypernetwork, these vectors are generated from learned block embeddings through a common mapping. Updating this mapping can affect recipes across multiple blocks, introducing coupling at the recipe level in addition to the sharing already provided by the expert pools. In our architecture, the same block embedding also conditions feature filtering. Consequently, recipe generation and feature filtering jointly shape the block representation through the task objective. This provides a mechanism for coordinating how a block is constructed with how its input features are filtered.

Compared with standard SwiGLU, DPRG preserves a direct value pathway and adds a normalized multiplicative correction with a learnable scale. The direct pathway retains the value representation when modulation is weak and provides a gradient term independent of the gate’s magnitude. RMS normalization reduces sensitivity to scale variations introduced by expert composition, while the learnable scale controls the correction strength.

AuthorsOpenReview ↗

[3/4] Official Comment by Authors

To W1:

Motivation, architectural distinction.

RecipeMoE and MoBE share the principle of constructing expert transformations from shared parameter components. MoBE reconstructs the up and gate projections of existing SwiGLU experts using shared basis matrices, directly learned mixing coefficients, and expert-specific transformation matrices. RecipeMoE adopts a different organization of parameter composition: an expert pool provides the trainable parameter components, a finite codebook defines reusable blocks, and a token router selects which blocks to execute.

The shared hypernetwork introduces additional sharing in how composition recipes are learned. Each block has a learned embedding that represents its identity and does not depend on the current token. The hypernetwork maps this embedding to two coefficient vectors, specifying how the expert pool should be combined for the value and gate paths. Because the generator is shared, an update driven by one block can influence the recipes of other blocks. This adds recipe-level coupling beyond the parameter sharing already provided by the expert pools. The same block embedding also conditions feature filtering, allowing the block’s construction and its treatment of input features to be jointly shaped through a common representation.

The recipes define a composition rule for the entire routed block. Each internal layer has its own expert pool, and the value and gate transformations at that layer are composed from the same pool using separate recipes. The recipe pair is reused across the block’s internal layers. Consequently, the recipes receive training signals from multiple layers, while the layer-specific pools retain the flexibility to learn different transformations at different depths.

DPRG preserves the value transformation while learning a multiplicative correction. Its operation can be written as

z=v⊙(1+λ SiLU(RMSNorm⁡(g)))z = v \odot (1+\lambda\ \mathrm{SiLU}(\operatorname{RMSNorm}(g))).

Unlike the multiplicative core of standard SwiGLU, this formulation includes a direct value term. When modulation is weak, the output remains close to vv, and the value path retains a direct gradient component. RMS normalization reduces sensitivity to gate-scale variations introduced by expert composition, while the learnable λ\lambda controls the correction strength.

Shared composition can affect learning even when the final deployment architecture is fixed. A token’s gradients pass through its selected blocks into the common expert pools. Updating these pools can also change the effective weights of other blocks, allowing training signals from different routed subsets to interact while preserving block-specific transformations. Our Direct Blocks ablation tests the overall value of this training parameterization by removing the hypernetwork and expert pools and directly optimizing the block weights. Under the same deployed block architecture, RecipeMoE achieves 73.78% mean ImageNet-1K ACC@1 versus 73.15% for Direct Blocks, with improvements in all three seeds. After training, caching simply stores the learned block transformations and removes runtime composition. It therefore preserves the benefits acquired during training without requiring additional learning during inference.

Reviewer_cxKbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RecipeMoE, an MoE architecture that constructs reusable blocks by composing parameters from shared expert pools and sparsely routes tokens to a subset of these blocks. The method introduces Dual-Path Residual Gating (DPRG) and uses caching to avoid repeated parameter composition at inference. Experiments on ImageNet-1K, MiniPile, and IntTravel, together with an industrial deployment, demonstrate promising results across several domains.

Critical strengths

Clear and well-motivated architecture. The distinction between expert participation, execution, and parameter materialization provides a useful framework for understanding the method. The architecture is technically well explained, with a useful complexity analysis and a practical caching strategy.

Broad and promising empirical evaluation. Experiments span vision, language modeling, and travel recommendation, showing consistent improvements over several MoE baselines. Ablation studies and routing visualizations provide useful insights into the proposed components, while the industrial deployment suggests potential practical applicability.

Critical weaknesses

Novelty and component contributions remain unclear. The comparison with prior work on expert composition and parameter merging needs to be sharper. Removing DPRG’s gate also removes its multiplicative nonlinearity, so a parameter-matched SwiGLU baseline would better isolate the proposed mechanism. Direct Blocks is useful, but sharing and capacity differences complicate interpretation. Appendix C supports expert contributions, while class-dependent routing alone does not establish semantic specialization.

Practical efficiency is not yet established. Even with caching, RecipeMoE uses substantially more FLOPs than several strong baselines. Matched-compute comparisons and measurements of latency, training cost, and cache memory are needed. The online A/B test also needs clearer baseline and significance reporting. Equation (6) should clarify whether Top-k weights are renormalized and how this affects the shared expert. Block utilization and coefficient norms would help assess load balance and whether 1/E1/\sqrt{E} adequately controls composition scale.

Questions for authors
  1. What is the fundamental architectural difference between RecipeMoE and the closest block-conditioned expert composition methods, and can this difference be evaluated under matched resource budgets?
  2. Are Top-k routing probabilities renormalized? Is load-balancing regularization applied, and how does the routing convention affect the relative contribution of the shared expert?
  3. Can the authors provide parameter-matched DPRG comparisons with standard nonlinear gated blocks and report resource statistics for the Direct Blocks ablation?
  4. What are the measured training and inference costs, including latency, throughput, peak memory, and cache construction and storage?
  5. How is meaningful expert participation verified beyond algebraic inclusion, and do coefficient norms remain stable throughout training?
13 comments
AuthorsOpenReview ↗

[1/6] Official Comment by Authors

Thank you for the detailed feedback.

To Q1:

Fundamental architectural differences between RecipeMoE and the closest block-conditioned expert composition methods.

RecipeMoE separates block construction from token routing: block embeddings determine how shared expert parameters are composed, while token features determine which composed blocks are executed. MoBE provides a closely related comparison because it also constructs routed expert transformations from shared parameter components. The architectural distinction therefore concerns how composition recipes are generated and how the resulting transformations are organized into executable blocks. To make the differences between the two designs clearer, we compare RecipeMoE with a MoBE-style baseline trained from scratch under the same training protocol and matched inference FLOPs. At the same inference cost of 3.457 GFLOPs, RecipeMoE improves mean ACC@1 and ACC@5 by 1.75 and 1.85 percentage points, respectively. It also reduces trainable parameters by approximately 55.82%, from 54.990M to 24.295M.

MethodParams (M)FLOPs (G)ACC@1 (%)ACC@5 (%)
MoBE-style (trained from scratch)54.9903.45772.03 ± 0.2389.62 ± 0.14
RecipeMoE (cached)24.2953.45773.78 ± 0.0991.47 ± 0.15

RecipeMoE differs from MoBE in three architectural aspects:

  1. Hypernetwork-generated recipes. MoBE directly learns expert-specific basis coefficients and transformation matrices. RecipeMoE uses a shared hypernetwork to generate composition coefficients from learned block embeddings. Compared with directly learned coefficients, the hypernetwork improves ACC@1 from 73.69 ± 0.03% to 73.78 ± 0.09%.
  2. DPRG. MoBE retains the original SwiGLU expert structure. RecipeMoE uses separate recipes over the same expert pool to construct value and gate transformations, coupled through residual multiplicative gating with a learnable scale. Removing the gate reduces ACC@1 from 73.78 ± 0.09% to 67.63 ± 0.12%, while using the same composition coefficients for both paths reduces it to 72.46 ± 0.31%, corresponding to mean drops of 6.15 and 1.32 percentage points, respectively.
  3. Multi-layer blocks. MoBE reparameterizes individual expert projections. RecipeMoE routes tokens to multi-layer blocks, with each internal layer composed from its own expert pool. Replacing the two-layer blocks with single-layer blocks while preserving the total expert parameter count reduces ACC@1 from 73.78 ± 0.09% to 69.13 ± 0.36%, a mean drop of 4.65 percentage points.
AuthorsOpenReview ↗

[2/6] Official Comment by Authors

To Q2 and W2.3:

Top-k normalization, load balancing, and the shared expert’s relative contribution.

Top-k normalization. Thank you for requesting this clarification. We apply softmax over all KK block logits and retain the probabilities of the selected top-kk blocks without renormalizing them. For a valid token, the MoE output is

yt=S(xt)+∑b∈Ttpt,bBb(xt)y_{t} = S(x_{t}) + \sum_{b \in \mathcal{T}_{t}} p_{t,b} B_{b}(x_{t}),

where SS denotes the shared expert, which is always active with a unit additive coefficient.

Let qt=∑b∈Ttpt,bq_{t} = \sum_{b \in \mathcal{T}_{t}} p_{t,b}. This convention scales the normalized routed mixture by qtq_{t}, while leaving the shared branch unchanged. For example, with K=8K=8 and k=2k=2, uniform routing gives qt=0.25q_{t}=0.25, whereas routing concentrated on the selected blocks gives qtq_{t} close to 1. The actual relative contribution also depends on the branch output norms.

This provides a token-dependent mechanism for controlling how strongly the selected blocks modify the shared representation. For a fixed normalized mixture, qtq_{t} directly scales the departure from the shared output; across tokens, the actual departure also depends on the magnitudes and directions of the block outputs.

Our formulation therefore preserves both the relative weights among selected blocks and their total probability mass under the full routing distribution. Renormalizing the selected probabilities would fix their sum to one and remove this source of adaptive modulation. The shared expert provides an always-active pathway for common information, with a unit additive coefficient independent of the retained routing mass.

Load-balance and block utilization. We omit an auxiliary load-balancing objective to let block utilization be driven by the task loss without explicitly encouraging uniform usage. We report block-assignment proportions on the ImageNet-1K validation set, averaged over three seeds. Each row is normalized by total token-to-block assignments and sums to 100%.

LayerBlock 0 (%)Block 1 (%)Block 2 (%)Block 3 (%)Block 4 (%)Block 5 (%)Block 6 (%)Block 7 (%)
000.054050.00049.946000.000281
220.3946.44210.22710.0523.53618.6291.11929.601
411.8976.9923.56517.07710.73321.72415.27212.740
620.4378.08610.5038.08210.2866.81114.11221.683

The results reveal depth-dependent routing behavior. Layer 0 concentrates almost all assignments on Blocks 3 and 4, while subsequent layers use all eight blocks with progressively more balanced assignment proportions. This pattern is consistent with broadly shared processing of low-level visual features in early layers and increasingly differentiated processing of complex representations at greater depths.

AuthorsOpenReview ↗

[3/6] Official Comment by Authors

To Q3 and W1:

Parameter-matched DPRG comparisons with standard gated blocks and resource statistics for Direct Blocks.

We add three gated comparisons to examine DPRG beyond the presence of multiplicative nonlinearity:

  • Two-layer SwiGLU: replace each DPRG layer with a standard SwiGLU transformation, retaining the two-layer structure.
  • Single-layer SwiGLU: replace the entire two-layer DPRG block with one standard SwiGLU transformation.
  • DPRG w/o residual term: replace v⊙(1+λSiLU(g))v \odot (1+\lambda\mathrm{SiLU}(g)) with v⊙(λSiLU(g))v \odot (\lambda\mathrm{SiLU}(g)), retaining the composed paths and learnable λ\lambda.

We adjust the hidden widths of both SwiGLU variants to match RecipeMoE’s parameter budget. ImageNet-1K accuracy is reported as mean ± sample standard deviation over three seeds. Latency is measured with batch size 1 on the same hardware and at the same numerical precision.

VariantACC@1 (%)ACC@5 (%)
RecipeMoE73.78 ± 0.0991.47 ± 0.15
Two-layer SwiGLU69.19 ± 0.4289.20 ± 0.19
Single-layer SwiGLU73.45 ± 0.4091.40 ± 0.15
DPRG w/o residual term72.97 ± 0.0991.03 ± 0.01

RecipeMoE improves mean ACC@1 over the parameter-matched two-layer and single-layer SwiGLU variants by 4.59 and 0.34 percentage points, respectively. Removing DPRG’s residual term reduces mean ACC@1 by 0.82 percentage points. RecipeMoE also achieves higher mean ACC@5 in all three comparisons.

We also report resource statistics for Direct Blocks, which directly learns block parameters without expert pools or a hypernetwork.

VariantParams (M)Activated Params (M)Inference FLOPs (G)ACC@1 (%)ACC@5 (%)
RecipeMoE (cached)24.29523.1113.45773.78 ± 0.0991.47 ± 0.15
Direct Blocks24.2908.9053.45773.15 ± 0.1191.05 ± 0.16

RecipeMoE improves mean ACC@1 and ACC@5 over Direct Blocks by 0.63 and 0.42 percentage points, respectively, with nearly identical trainable parameter counts and identical cached inference FLOPs. These results support the benefit of shared expert composition over directly learned routed blocks under comparable parameter and inference-compute budgets.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Manuscript Revisions

Dear Reviewer, AC, SAC and PC,

We highlight revised and newly added content in blue for ease of review.

  • Additional baseline comparisons. We include MoBE-style results across all three tasks (Tables 1 and 3) and matched-FLOP comparisons on ImageNet-1K (Appendix H, Table 10). RecipeMoE outperforms MoBE-style across all reported task metrics. Under matched FLOPs, it achieves the second-highest mean ACC@1 and the highest mean ACC@5, using less than one-quarter of the trainable parameters of SMEAR, the strongest baseline in ACC@1 (24.295M vs. 102.7M).

  • Additional ablations and multi-seed results. We expand hypernetwork, DPRG, and Direct Blocks ablations across ImageNet-1K (Table 2), MiniPile, and IntTravel (Appendix I, Table 11), with per-seed results in Appendix J, Tables 12–18. These results support the benefits of hypernetwork-generated recipes, DPRG, and parameter composition across the three tasks, with broadly consistent gains across seeds.

  • Resource costs. We add total and activated parameter counts and FLOPs for MiniPile and IntTravel (Table 3), training costs (Appendix G, Table 7), and inference costs at batch sizes 1 and 256 (Appendix G, Tables 8 and 9). Cache construction time and storage costs are also detailed in Appendix G. The measurements show competitive training throughput and reduced inference costs with caching.

  • Load balance and coefficient stability. We analyze load balance through block-assignment distributions across layers (Appendix D, Figure 8) and examine composition-coefficient norm stability during training (Appendix E, Figure 9 and Table 5). Block utilization is more balanced in deeper layers than in the first MoE layer, and coefficient norms exhibit limited variation late in training across both evaluated runs.

  • Semantic routing preferences. We examine semantic-group routing patterns, compare discovery-selected classes on disjoint confirmation images, and provide qualitative examples (Appendix F, Figures 10–11 and Table 6). Together, these findings provide evidence of semantic specialization among blocks.

  • Method clarifications and online evaluation details. We clarify the motivation for composition-coefficient scaling (Section 3.1), the relationship between unrenormalized Top-kk routing weights and the shared expert (Section 3.2), and the distinction between sparse block execution and total computational cost (Sections 3.3 and 4.2). We also expand the online A/B evaluation details, including the control model, traffic allocation, serving conditions, latency, and confidence interval for the UVCTR improvement (Section 5).

  • Single-layer SwiGLU comparisons. We add parameter-matched single-layer SwiGLU ablations on ImageNet-1K (Table 2), MiniPile, and IntTravel (Appendix I, Table 11), with per-seed results in Appendix J, Tables 16-18. These comparisons show modest improvements over this stronger baseline: 0.33 percentage points in ImageNet-1K ACC@1, 0.0252 in MiniPile perplexity, and 0.09 percentage points in IntTravel HR@1.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of discussion

Dear Reviewer, AC, SAC and PC,

Following the rebuttal, Reviewer cxKb raised their rating from 2 to 3, and Reviewer tHXF maintained their rating of 3. We responded to Reviewer BWwd on November 9, added experimental results on November 14, and followed up on November 17 and 19. As of this note, Reviewer BWwd has not posted a follow-up assessment. We have addressed each concern raised by the three reviewers and incorporated the supporting evidence into the revised manuscript.

RecipeMoE uses block embeddings to compose parameters from the entire expert pool into a fixed set of cacheable blocks. Each token executes only a few of these blocks. By combining full expert participation, sparse execution, and bounded materialization, RecipeMoE achieves a favorable trade-off among predictive performance, trainable parameter count, and inference FLOPs.

Each property addresses a different limitation, as supported by controls and complexity analysis:

  • Full participation. The comparison with Direct Blocks, which learns each block’s weights directly, supports the benefit of full expert participation through shared parameter composition across image classification, language modeling, and recommendation. Under matched configurations for each task, RecipeMoE improves ImageNet-1K ACC@1 from 73.15% to 73.78%, lowers MiniPile perplexity from 14.8060 to 14.6458, and raises IntTravel HR@1 from 68.39% to 68.47%.

  • Sparse execution. Sparse execution allows the entire expert pool to contribute to block construction while each token executes only the selected blocks. On ImageNet-1K, expanding the pool from 1 to 128 experts kept cached inference cost at 3.457 GFLOPs per image with the block architecture and routing configuration fixed.

  • Bounded materialization. Bounded materialization lets RecipeMoE cache a fixed set of input-independent blocks and reuse them across tokens, avoiding repeated parameter synthesis. At comparable trainable parameter counts, cached RecipeMoE uses 36% fewer inference FLOPs per sequence than token-level SMEAR on MiniPile and reduces perplexity from 15.08 to 14.57. On IntTravel, it uses 81% fewer inference FLOPs and improves HR@1 from 68.27% to 68.52%.

RecipeMoE uses input-independent composition coefficients to construct blocks, so tokens routed to the same block share its weights. We introduce Dual-Path Residual Gating (DPRG) to let the block modulate features differently for each token. The value and gate paths are constructed from the same expert pool using separate sets of composition coefficients. The value path carries the token’s features, while the gate path uses the token’s representation to produce a feature-wise multiplicative correction.

By combining shared expert composition, sparse execution, and reusable blocks with Dual-Path Residual Gating, RecipeMoE delivers gains across three offline benchmarks and in a production recommendation system.

  • Under comparable trainable parameter budgets, it outperforms the strongest evaluated MoE baseline for each task metric, raising ImageNet-1K ACC@1 from 71.83% to 73.78%, lowering MiniPile perplexity from 14.99 to 14.57, and raising IntTravel HR@1 from 68.38% to 68.52%.
  • Under comparable inference FLOPs on ImageNet-1K, RecipeMoE achieves the second-highest mean ACC@1 and the highest mean ACC@5 while using the fewest trainable parameters: 24.295M versus 39.50M to 111.00M for the evaluated baselines. Compared with SMEAR, RecipeMoE uses approximately 76.3% fewer trainable parameters, with mean ACC@1 lower by 0.21 percentage points and mean ACC@5 higher by 0.27 percentage points.

RecipeMoE’s results show that combining full expert participation, sparse execution, and bounded materialization can deliver strong predictive performance while balancing trainable parameter count and inference FLOPs. Building on these results, we propose a direction for future MoE research: making expert composition itself input-dependent while preserving all three properties. In RecipeMoE, the input XX determines which blocks execute, and each block B has fixed expert-composition coefficients CC, so P(C∣B,X)=P(C∣B)P(C\mid B,X)=P(C\mid B). Modeling P(C∣B,X)P(C\mid B,X) more flexibly would allow inputs routed to the same block to use different expert mixtures. This broadens the research question from which blocks to execute for each input to how best to compose experts for that input within a given compute and cache budget. RecipeMoE provides a concrete starting point for this investigation.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.