- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RecipeMoE learns a small set of input-independent recipes, each of which mixes a shared pool of expert parameters into a reusable block. Each token routes to a few of these blocks, which have separate value and gating paths, plus a shared expert. Because the recipes do not depend on the input, the blocks can be composed once after training and cached, and the pools can then be dropped.
Critical strengths
-
The construction has a clear and useful deployment property. Once the blocks are precomputed, inference cost no longer grows with the size of the expert pool, which input-dependent merging methods like SMEAR can't offer.
-
The accuracy gains are consistent: about 2 points over SMEAR on ImageNet with low variance, and first in every seed across the three datasets. The Direct Blocks ablation is a nice touch. Training the same final blocks directly does worse, so going through the pool seems to help during training.
Critical weaknesses
-
The baselines are matched on parameters, not compute. Cached RecipeMoE needs a bit over twice the FLOPs of SMEAR and about 1.8x those of V-MoE. I understand the activated parameter counts are similar, and it does beat DynMoE, which costs more. Still, the extra compute comes mostly from running several two-path blocks and a shared expert, so I can't tell how much of the gain over SMEAR is the method and how much is just more computation. There is also no compute or latency information for MiniPile and IntTravel, and the IntTravel improvements are quite small.
-
It's not clear why a hypernetwork is needed to generate the recipes. At inference the model is just a fixed set of blocks. Direct Blocks shows that going through the pool helps, but the simpler option of learning the mixing coefficients directly isn't tested. Direct Blocks itself is only run on ImageNet (the appendix explains why, which is fair), and the ablations on the other two datasets are single runs with some very small differences.
Questions for authors
-
How do SMEAR and V-MoE do when scaled up to similar FLOPs? Latency on the same hardware would also help.
-
What happens if the hypernetwork is replaced by a directly learned coefficient table, with everything else unchanged? Could you report the size and FLOPs of Direct Blocks and run it on one more dataset?
-
For the A/B test: what were the latency and resource cost of the control model, how was traffic split, and is there a confidence interval on the 2.4% gain? Since the control has no MoE layer, this mostly shows that adding RecipeMoE helps, not that it beats other MoE designs.
7 comments
[2/2] Official Comment by Authors
To W1.2:
There is also no compute or latency information for MiniPile and IntTravel.
Additional efficiency measurements.
We supplement the task metrics with parameter counts and inference FLOPs for both datasets.
MiniPile
| Method | Params (M) | FLOPs (G/sequence) | Loss ↓ | PPL ↓ |
|---|---|---|---|---|
| Dense | 152.006 | 369.233 | 2.8168 ± 0.0036 | 16.7230 ± 0.0607 |
| Switch Transformer | 832.230 | 369.488 | 2.7164 ± 0.0019 | 15.1265 ± 0.0292 |
| DeepSeek-V3 MoE | 831.594 | 717.352 | 2.7078 ± 0.0066 | 14.9968 ± 0.0993 |
| ReMoE | 832.230 | 369.488 | 2.7346 ± 0.0010 | 15.4034 ± 0.0152 |
| DynMoE | 831.608 | 786.959 | 2.7115 ± 0.0084 | 15.0520 ± 0.1268 |
| MASS | 831.608 | 665.196 | 2.7513 ± 0.0100 | 15.6636 ± 0.1559 |
| Soft MoE (causal) | 832.230 | 1761.057 | 2.7129 ± 0.0035 | 15.0723 ± 0.0534 |
| SMEAR (token-level) | 832.230 | 1763.605 | 2.7133 ± 0.0029 | 15.0789 ± 0.0442 |
| Lory | 831.608 | 467.094 | 2.7595 ± 0.0086 | 15.7925 ± 0.1367 |
| μMoE (CP) | 825.980 | 1749.531 | 2.7075 ± 0.0024 | 14.9925 ± 0.0366 |
| μMoE (TR) | 825.902 | 1751.578 | 2.7186 ± 0.0021 | 15.1590 ± 0.0311 |
| RecipeMoE | 825.984 | 1146.657 | 2.6789 ± 0.0011 | 14.5696 ± 0.0161 |
| RecipeMoE (cached) | 1455.536 | 1126.510 | 2.6789 ± 0.0011 | 14.5696 ± 0.0161 |
IntTravel
| Method | Params (M) | FLOPs (G) | HR@1 (%) ↑ | HR@5 (%) ↑ | NDCG@5 (%) ↑ |
|---|---|---|---|---|---|
| Dense | 0.640 | 0.293 | 67.50 ± 0.02 | 86.57 ± 0.02 | 77.87 ± 0.02 |
| Switch Transformer | 22.374 | 0.412 | 68.01 ± 0.10 | 86.66 ± 0.04 | 78.15 ± 0.08 |
| DeepSeek-V3 MoE | 22.230 | 0.853 | 68.13 ± 0.07 | 86.77 ± 0.03 | 78.24 ± 0.05 |
| ReMoE | 22.374 | 0.633 | 68.33 ± 0.01 | 86.85 ± 0.01 | 78.36 ± 0.01 |
| DynMoE | 22.374 | 1.954 | 68.16 ± 0.04 | 86.79 ± 0.03 | 78.26 ± 0.01 |
| MASS | 22.374 | 1.514 | 68.22 ± 0.04 | 86.81 ± 0.02 | 78.30 ± 0.02 |
| Soft MoE (causal) | 22.374 | 7.462 | 68.29 ± 0.04 | 86.81 ± 0.02 | 78.33 ± 0.02 |
| SMEAR (token-level) | 22.374 | 7.730 | 68.27 ± 0.03 | 86.79 ± 0.03 | 78.31 ± 0.04 |
| Lory | 22.231 | 0.898 | 68.12 ± 0.01 | 86.73 ± 0.04 | 78.21 ± 0.02 |
| μMoE (CP) | 22.874 | 7.675 | 68.35 ± 0.03 | 86.84 ± 0.02 | 78.37 ± 0.02 |
| μMoE (TR) | 22.876 | 7.739 | 68.38 ± 0.02 | 86.85 ± 0.01 | 78.38 ± 0.03 |
| RecipeMoE | 22.875 | 2.172 | 68.52 ± 0.01 | 86.92 ± 0.01 | 78.50 ± 0.03 |
| RecipeMoE (cached) | 12.174 | 1.488 | 68.52 ± 0.01 | 86.92 ± 0.01 | 78.50 ± 0.03 |
RecipeMoE achieves the best mean task metrics on both datasets. Cached inference uses fewer FLOPs than Soft MoE, token-level SMEAR, and both μMoE variants, although several sparse-routing baselines remain cheaper.
IntTravel improvements.
The gains on IntTravel are less pronounced than those on ImageNet-1K and MiniPile, but remain consistent across seeds. Relative to the strongest baseline for each metric within each seed, RecipeMoE achieves the following improvements:
| Seed | HR@1 gain (pp) | HR@5 gain (pp) | NDCG@5 gain (pp) |
|---|---|---|---|
| 42 | 0.15 | 0.07 | 0.13 |
| 4981213 | 0.17 | 0.07 | 0.15 |
| 211231244 | 0.12 | 0.05 | 0.07 |
RecipeMoE outperforms all evaluated baselines on all three metrics in every seed. Together with the larger gains on the other two tasks, these results support its effectiveness across domains, with the magnitude of improvement varying by dataset.
Additional Experimental Results
Thank you again for your insightful feedback. We hope these additional experiments help address your concerns and look forward to further discussion.
1. Hypernetwork ablations on MiniPile and IntTravel.
We have completed three-seed comparisons between hypernetwork-generated recipes and directly learned composition coefficients on MiniPile and IntTravel. Loss and PPL are evaluated on MiniPile. HR@1, HR@5, and NDCG@5 are evaluated on IntTravel and reported in %. Results are mean ± sample standard deviation over three seeds.
| Variant | Loss ↓ | PPL ↓ | HR@1 ↑ | HR@5 ↑ | NDCG@5 ↑ |
|---|---|---|---|---|---|
| Directly learned coefficients | 2.6947±0.0017 | 14.8006±0.0247 | 68.45±0.05 | 86.89±0.05 | 78.45±0.04 |
| RecipeMoE | 2.6789±0.0011 | 14.5696±0.0161 | 68.52±0.01 | 86.92±0.01 | 78.50±0.03 |
On MiniPile, the hypernetwork reduces mean loss by 0.0158 and perplexity by 0.2310, with improvements in all three seeds. On IntTravel, it improves all three mean metrics, with HR@1 and NDCG@5 improving in every seed. Together with the ImageNet-1K results, these experiments support the benefit of shared hypernetwork-generated recipes across all three tasks.
2. Direct Blocks ablation and resource comparison on MiniPile.
We compare RecipeMoE with Direct Blocks, which removes the hypernetwork and expert pools and directly optimizes the routed block weights. To ensure a comparable evaluation, both variants use the same four-block architecture, with RecipeMoE composing these blocks from eight experts.
| Variant | Params (M) | Inference FLOPs (G) | Loss ↓ | PPL ↓ |
|---|---|---|---|---|
| Direct Blocks | 825.911 | 1126.397 | 2.6950 ± 0.0031 | 14.8060 ± 0.0458 |
| RecipeMoE | 825.926 | 1136.470 | 2.6842 ± 0.0032 | 14.6458 ± 0.0471 |
| RecipeMoE (cached) | — | 1126.397 | — | — |
Task metrics are reported as mean ± sample standard deviation over three seeds. Params denotes trainable parameters. FLOPs are measured per 1,024-token sequence at batch size 1. Caching preserves RecipeMoE’s task metrics.
With nearly identical trainable parameter counts, RecipeMoE reduces mean loss by 0.0108 and perplexity by 0.1602, outperforming Direct Blocks on both metrics in every seed. After caching, RecipeMoE also matches Direct Blocks in inference FLOPs. These results support the benefit of learning routed blocks through shared expert composition without increasing cached inference computation.
3. Multi-seed component ablations on MiniPile.
We extend the original single-run MiniPile ablations with additional seeds. The table reports mean ± sample standard deviation.
| Variant | Loss ↓ | PPL ↓ |
|---|---|---|
| 1-Layer | 2.7162±0.0027 | 15.1226±0.0406 |
| Fixed λ | 2.6916±0.0027 | 14.7548±0.0399 |
| w/o Gate | 2.7651±0.0040 | 15.8812±0.0635 |
| Tied Recipes | 2.6994±0.0124 | 14.8719±0.1854 |
| w/o Shared | 2.7276±0.0016 | 15.2957±0.0241 |
| w/o Filter | 2.6828±0.0005 | 14.6255±0.0079 |
| Softmax Coeff. | 2.7088±0.0016 | 15.0106±0.0238 |
| RecipeMoE | 2.6789±0.0011 | 14.5696±0.0161 |
The full model consistently achieves lower loss and perplexity than all ablation variants across the evaluated seeds. Removing the gate causes the largest performance drop, highlighting its importance in language modeling. Reducing block depth, tying the value and gate recipes, fixing λ, removing the shared expert, or applying softmax to the composition coefficients also degrades performance. Feature filtering provides a smaller but consistent improvement. These results reinforce the original ablation findings and show that the observed benefits persist across random seeds.
[1/2] Official Comment by Authors
To Q1 and W1.1:
The baselines are matched on parameters, not compute.
Thank you for suggesting a complementary comparison under matched FLOPs. In addition to our parameter-matched experiments, we evaluate the following methods on ImageNet-1K at comparable inference FLOPs by adjusting expert FFN widths while retaining the remaining training settings. Results are reported as mean ± sample standard deviation over three seeds. RecipeMoE’s FLOPs correspond to cached inference.
At comparable inference FLOPs, RecipeMoE achieves the second-highest mean ACC@1 and the highest mean ACC@5 with substantially fewer trainable parameters: 24.30M versus 39.50–111.00M for the evaluated baselines, a reduction of 38.5–78.1%. It improves mean ACC@1 over V-MoE and the MoBE-style baseline by 0.46 and 1.76 percentage points, respectively. Compared with SMEAR, which uses over four times as many trainable parameters (102.70M vs. 24.30M), RecipeMoE achieves only 0.21 percentage points lower mean ACC@1 and 0.27 percentage points higher mean ACC@5 at comparable inference FLOPs.
SMEAR’s large parameter budget is enabled in part by example-level composition in the visual setting, where one merged expert is reused across all tokens in an image. In the causal language-modeling setting, SMEAR instead performs token-level composition, increasing synthesis costs and reducing the parameter budget affordable at matched FLOPs. RecipeMoE already uses token-level routing in both visual and causal settings. Moving to a causal setting therefore introduces no additional per-token composition cost.
To Q2.1 and W2.1:
Report the parameter counts and FLOPs for both Directly Learned Coefficients and Direct Blocks.
We replaced the hypernetwork with directly learned composition coefficients, retaining the same expert pools, block structure, and routing configuration. This comparison isolates the contribution of the hypernetwork.
With directly learned coefficients, each block has its own value and gate coefficient vectors. With a shared hypernetwork, these vectors are generated from learned block embeddings through a common mapping. Updating this mapping can affect recipes across multiple blocks, introducing coupling at the recipe level in addition to the sharing already provided by the expert pools. In our architecture, the same block embedding also conditions feature filtering. Consequently, recipe generation and feature filtering jointly shape the block representation through the task objective. This provides a mechanism for coordinating how a block is constructed with how its input features are filtered.
To Q3:
The A/B test’s latency, resource costs, traffic split, and confidence interval.
The control uses the same backbone without an MoE module. Under the same serving conditions, its mean/P99 latency is 17/31 ms, compared with 19/38 ms for RecipeMoE. Both the control and RecipeMoE use the same number of GPUs.
The one-week A/B test randomly assigned users to the two groups with equal traffic allocation. The serving load reported in the paper is approximately 5,000 QPS. RecipeMoE achieved a 2.4% relative improvement in UV CTR, with a 95% confidence interval of [1.29%, 3.67%]. The 95% confidence interval was estimated using a log risk-ratio normal approximation, treating the deduplicated user-level click outcomes in the control and treatment groups as independent binomial observations.
Deploying a new model in our mobility recommendation service requires substantial engineering and validation, including runtime optimization, manual evaluation, smoke testing, and staged rollout. RecipeMoE achieved the best offline performance among the evaluated methods and an observed 2.4% relative UV CTR improvement over the production baseline in online A/B testing. Together, these results provide evidence of RecipeMoE’s effectiveness and practical value in a production recommendation system.