acceptodds
Under review as a conference paper at ICLR 2027

Den2MoEE: Reconstructing Dense LLMs into Expert-Specialized Mixture-of-Experts for Efficient Embedding Models

Abstract

Most existing LLM-based embedding models in database semantic search systems rely on dense Transformer architectures. In contrast, Mixture-of-Experts (MoE) embedding models provide a cost-efficient scaling paradigm by activating only a subset of parameters. However, training large-scale MoE embedding models from scratch is prohibitively expensive. Although reconstructing dense LLM embeddings into MoE architectures offers a practical alternative, existing approaches often overlook expert specialization and routing characteristics specific to embedding tasks, resulting in homogeneous experts and inefficient performance recovery. To address these limitations, we propose Den2MoEE, a unified Dense-to-MoE reconstruction framework for embedding models that jointly promotes domain-aware expert specialization and routing-aware adaptation. Den2MoEE reconstructs dense embedding models through TokenScore-weighted neuron domain modeling, expert-diversity-driven Dense-to-MoE conversion, and an efficient two-phase recovery strategy. Furthermore, we exploit MoE router signals as complementary embedding representations, further enhancing the quality of the reconstructed embeddings without increasing activated parameters. Extensive experiments demonstrate that Den2MoEE reduces the number of activated parameters by 33%–55% compared with the original dense LLM embedding models, while consistently outperforming existing embedding baselines under comparable activated parameter budgets. Moreover, Den2MoEE delivers substantial system-level gains, achieving 1.35×–1.8× higher throughput with significantly lower inference latency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM embedding, Dense-to-MoE reconstruction, Expert specialization, Parameter-efficient modeling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_KcC1OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Den2MoEE, a framework for reconstructing pretrained dense embedding models into sparse MOE models to reduce inference costs while preserving their representation capabilities. The method constructs neuron domain-preference vectors from TokenScore-weighted activations and groups neurons with similar preferences into experts. It combines shared experts, low-rank SVD experts, and null experts to enable conditional computation. To mitigate performance degradation after conversion, the framework introduces neuron-weight-guided router initialization, domain-coverage-aware routing biases, and a two-stage recovery training strategy. It further incorporates routing signals from multiple layers to complement the final embedding representations.

Critical strengths
  1. The framework reuses existing dense embedding models to construct MoE architectures, reducing inference computation. This is particularly relevant for encoding large document collections and serving latency-sensitive retrieval applications.

  2. The expert partitioning strategy is tailored to embedding tasks. By combining TokenScore with neuron responses across domains, the method constructs experts based on their observed behavior on data, providing a more task-relevant partitioning criterion than random splitting or weight similarity alone.

  3. The paper is well written, and the experimental evaluation is relatively detailed.

Critical weaknesses
  1. LLaMA-MoE is the only direct conversion baseline evaluated. Comparisons with other relevant MoE embedding approaches, such as Nomic Embed v2 and MoEE, would help establish the effectiveness and positioning of the proposed framework. These comparisons should account for differences in their objectives and training settings. The related work discussion should also be expanded accordingly.

  2. The efficiency evaluation is primarily conducted with a sequence length of 512 tokens and a batch size of 128, whereas Qwen3-Embedding supports inputs of up to 32K tokens. The experiments do not establish whether longer inputs lead to greater performance degradation after conversion or smaller efficiency gains. I therefore recommend evaluating document encoding time and retrieval effectiveness on long-document retrieval tasks, such as LongEmbed, under different maximum input lengths, including 4K and 8K tokens.

  3. DomainCoverage counts the domains in which an expert’s score exceeds the median expert score and uses this quantity to determine its target routing proportion. However, the number of domains an expert covers does not necessarily reflect how frequently that expert is needed under the actual input distribution. Consequently, the resulting routing biases may be misaligned with deployment workloads. Experiments with varying domain proportions and unseen domains are needed to assess robustness and generalization. Furthermore, the training and evaluation data cover largely similar domains, making additional out-of-domain evaluations particularly valuable.

Questions for authors

n/a

6 comments
AuthorsOpenReview ↗

W1: Broadening Baselines and Clarifying Methodological Positioning

We sincerely thank the reviewer for the positive assessment and for recognizing the practical value of our work. We are highly encouraged by your feedback, particularly regarding our framework's ability to reuse existing dense embedding models for efficient MoE reconstruction, the task-relevant expert partitioning strategy (TokenScore + domain-conditioned profiles), and the detailed experimental evaluation.

We deeply appreciate the constructive questions aimed at broadening our baselines, testing long-context scenarios, and examining workload generalization. We have conducted additional experiments to address these concerns and provide detailed responses below.

W1: Broadening Baselines and Clarifying Methodological Positioning

We thank the reviewer for this excellent suggestion. To clarify the positioning, we must first distinguish the objectives of these methods: MoEE is a training-free representation extraction method, while Nomic Embed v2 is a capacity-expanding upcycling method. To provide a fair comparison, we added a controlled Nomic-style baseline and clarified how MoEE is utilized.

1. Clarification on MoEE (Not a Conversion Baseline) MoEE [1] extracts representations from the router weights and hidden states of an existing MoE without additional training. All MoE evaluations in our paper already adopt this MoEE-style embedding construction. Specifically, we average router weights over consecutive layers and concatenate them with hidden states. As shown in Section 4.4 and Figure 3, this extraction boosts our MTEB Code scores from 71.30 to 72.68 (+1.38) for 0.6B, and 74.38 to 76.54 (+2.16) for 4B. Thus, MoEE is seamlessly incorporated into our evaluation protocol, not omitted.

2. A Controlled Nomic-Style Upcycling Baseline Nomic Embed v2 [2] uses sparse upcycling to expand capacity. To compare fairly without confounding differences in base backbones and massive proprietary training data, we adapted Nomic's architectural recipe onto our identical Qwen3-Embedding-0.6B backbone and recovery dataset. Specifically, we replaced every other FFN layer (14 out of 28) with 8 full-sized experts, used Top-2 routing, random router initialization, and trained with InfoNCE + load-balancing loss.

3. Results and Analysis As shown in the tables below, Den2MoEE achieves the best quality-efficiency trade-off:

  • Den2MoEE achieves a Multilingual score of 55.05 and Code score of 72.68, using only 0.4B activated and 0.6B total parameters.

  • Nomic-Embed-v2-MoE (Adapted) achieves 53.90 / 71.83 with a much larger budget (0.7B activated / 1.8B total).

  • LLaMA-MoE falls behind at 52.73 / 67.66 under our budget.

Table 1: MTEB Multilingual (Task-type Mean)

ModelAct-Size / Total-SizeMean (Type)
Den2MoEE-0.6B-A0.4B0.4B / 0.6B55.05
Nomic-Embed-v2-MoE (Adapted)0.7B / 1.8B53.90
LLaMA-MoE0.4B / 0.6B52.73

Table 2: MTEB Code (Task-type Mean)

ModelAct-Size / Total-SizeMean (Type)
Den2MoEE-0.6B-A0.4B0.4B / 0.6B72.68
Nomic-Embed-v2-MoE (Adapted)0.7B / 1.8B71.83
LLaMA-MoE0.4B / 0.6B67.66

Copying intact pretrained FFNs (Nomic-style) can ease recovery compared to neuron partitioning, but it drastically increases total parameters. Our Den2MoEE introduces domain-aware expert structure and explicitly supports routing adaptation, achieving superior performance under a stricter parameter budget. We will add these results and a comprehensive Related Work section distinguishing capacity-expanding upcycling (Nomic), embedding extraction (MoEE), and efficiency-oriented reconstruction (Den2MoEE).

Details about broader baselines and methodological positioning can be found in Appendix F.13 of the revised version.

[1] Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free, ICLR 2025.

[2] Training Sparse Mixture Of Experts Text Embedding Models, arXiv:2502.07972.

W2: Long-Context Efficiency and Retrieval Effectiveness

We deeply appreciate this constructive suggestion. We have conducted extensive experiments on the LongEmbed benchmark at maximum input lengths of 4K and 8K tokens. Our findings demonstrate that Den2MoEE maintains a bounded performance degradation and substantial efficiency gains in long-context tasks.

1. Document Encoding Efficiency under Long Contexts

We measured the wall-clock time for encoding the full document corpus of four real-world tasks (totaling 1,188 documents). The measurement was conducted on a single NVIDIA H20 GPU (BF16 precision), including tokenization, forward pass, last-token pooling, and normalization. We used the same batch size for both models at each length.

Table 1: End-to-End Document Encoding Efficiency

ModelMax LengthBatch SizeTotal Time (s)Throughput (docs/s)Avg Time (ms/doc)
Qwen3-Embedding-4B (Dense)4K4428.02.8360.3
Den2MoEE-4B-A1.8B4K4250.44.7210.8 (-41.5%)
Qwen3-Embedding-4B (Dense)8K21004.71.2845.7
Den2MoEE-4B-A1.8B8K2630.81.9531.0 (-37.2%)

Analysis on Efficiency:

  • Den2MoEE achieves a 41.5% reduction in latency at 4K and a 37.2% reduction at 8K, demonstrating robust acceleration in long-document encoding.
  • Why does the efficiency gain slightly decrease at 8K? As sequence length grows, the O(N2)O(N^2) attention computation and memory I/O become the dominant bottlenecks. Because Den2MoEE only sparsifies the FFN layers while keeping the attention layers dense, the fraction of time spent on the sparsified FFN decreases, slightly diluting the overall end-to-end speedup. Nevertheless, a ~37% latency reduction at 8K is still highly significant for real-world RAG and search systems.

2. Long-Document Retrieval Effectiveness

We evaluated retrieval performance across six LongEmbed tasks (Passkey, Needle, NQA, QMS, SFD, WQA). The results (with drops relative to the same-length dense model in parentheses) are shown below:

Table 2: LongEmbed Retrieval Performance

ModelAct. ParamsMax LenPasskeyNeedleNQAQMSSFDWQAAverage
Qwen3-Embedding-4B (Dense)4.0B4K73.556.255.141.395.385.667.8
Den2MoEE-4B-A1.8B1.8B4K71.253.753.139.894.484.466.1 (-1.7)
LLaMA-MoE (4/16, A2B)2.0B4K70.452.852.339.193.883.465.3 (-2.5)
Qwen3-Embedding-4B (Dense)4.0B8K84.867.563.448.797.491.175.5
Den2MoEE-4B-A1.8B1.8B8K82.464.961.246.896.489.373.5 (-2.0)
LLaMA-MoE (4/16, A2B)2.0B8K81.664.160.446.196.088.672.8 (-2.7)

Analysis on Performance:

  • Bounded Degradation: From 4K to 8K, Den2MoEE's performance drop relative to the dense model remains extremely stable, moving only slightly from -1.7 to -2.0 points. There is no catastrophic forgetting or error explosion at longer contexts.
  • Consistent Superiority over LLaMA-MoE: Under the exact same sparsity budget, Den2MoEE consistently outperforms LLaMA-MoE by +0.8 (at 4K) and +0.7 (at 8K) points, while activating fewer parameters (1.8B vs 2.0B).

3. Mechanistic Explanation for Bounded Degradation

Why does Den2MoEE not suffer from exacerbated degradation at longer inputs? We attribute this to two key design choices:

  1. Length-Matched Recovery Data: Our recovery training data has an average length of ~5.4K (truncated at 8K), which perfectly covers the 4K-8K test range. This prevents the router from out-of-distribution routing on long documents.
  2. Architectural Isolation of Attention: Our dense-to-MoE reconstruction only modifies and sparsifies the FFN layers, leaving the attention layers completely untouched. Attention is the primary mechanism for extracting and aggregating long-range information. By preserving the original pre-trained attention weights intact, Den2MoEE retains the dense model's intrinsic long-context retrieval capability, allowing it to gracefully scale with input length.

In the revised version, we have added these results, measurement details, and analysis to Section 4.4. to demonstrate long-context efficiency and retrieval effectiveness.

AuthorsOpenReview ↗

W3: DomainCoverage, Workload Shifts, and Out-of-Domain Evaluation

W3: DomainCoverage, Workload Shifts, and Out-of-Domain Evaluation

We deeply appreciate this insightful critique. We fully agree with the reviewer's conceptual distinction: DomainCoverage measures the structural breadth of an expert's domain preference, not the expected frequency under a specific deployment distribution. In our framework, it serves as a Maximum-Entropy Default Prior for expert construction and routing regularization. While Eq. (11) updates the bias based on actual batch-level loads, the target ratio is still initially derived from coverage.

To empirically evaluate this potential misalignment and test generalization, we conducted comprehensive experiments on (1) varying domain proportions and (2) an extended unseen domain.

Robustness to Varying Domain Proportions (Skewed Calibration)

We simulated heavily skewed calibration distributions by varying the number of code and multilingual domains used during expert construction. The results are shown below:

Model / VariantCode / Multi. DomainsMTEB (Code)MTEB (Multi.)
Den2MoEE-4B-A1.8B (Default)14 / 1676.5457.52
Code-domain-heavy14 / 875.91 (-0.63)55.81 (-1.71)
Multilingual-domain-heavy7 / 1674.31 (-2.23)57.11 (-0.41)
LLaMA-MoE (Baseline)N/A (Weight-based)74.1655.42
  • As expected, the performance drop is larger in the domain family whose coverage is reduced (e.g., dropping Multi domains causes a 1.71 drop in Multi, while dropping Code domains causes a 2.23 drop in Code).
  • Crucially, both skewed variants still outperform the LLaMA-MoE baseline on both benchmarks, even in the extreme Multilingual-heavy case where the Code margin is narrow (+0.15 points). This demonstrates that the routing mechanism, supported by our dynamic bias and Shared Experts, is highly resilient and can gracefully degrade rather than catastrophically collapse when the calibration distribution shifts.

Out-of-Domain (OOD) Evaluation on an Unseen Domain

To test generalization, we extended our recovery corpus with approximately 1M legal domain samples (from Law-GPT, THUCNews, and Zhihu-KOL), which were originally entirely unseen in the code/multilingual training. We evaluated the models on MTEB (Law), Code, and Multilingual.

ModelMTEB (Law)MTEB (Code)MTEB (Multi.)
Qwen3-Embedding-4B (Dense)69.5480.0660.86
Qwen3-Embedding-4B-MoE (LLaMA-MoE)66.9274.0855.47
Den2MoEE-4B-A1.8B68.1976.5257.63
  • Den2MoEE outperforms LLaMA-MoE by +1.27 (Law), +2.44 (Code), and +2.16 (Multi) points.
  • Its Code and Multilingual scores (76.52 / 57.63) remain exceptionally close to the original two-domain setting (76.54 / 57.52), proving that expanding domain coverage via domain-specific recovery data does not cause catastrophic forgetting of the original capabilities.
  • Important Boundary Declaration: Since the legal data was included in the recovery training, this experiment measures the model's adaptability to a newly introduced domain, rather than zero-shot generalization to an entirely unseen domain. Uncovered specialized domains may still induce suboptimal routing, which we will explicitly state as a limitation in the revised manuscript.

Clarification and Revision. We agree with the reviewer that calibration coverage does not equal deployment traffic. To ensure full transparency, we have revised the manuscript to:

  1. Explicitly state in Section 3.3 that DomainCoverage is a structural prior, not a deployment frequency predictor.
  2. Include the varying-proportion experiment in Appendix F.14 and the domain-extension experiment in Appendix F.7 to demonstrate the system's tolerance to calibration shifts and its extensibility.

We sincerely thank the reviewer for these highly constructive suggestions. Adding the Nomic-style baseline, the LongEmbed evaluations, and the varying domain evaluations has significantly strengthened our empirical evidence. We hope these detailed analyses and our clarifications thoroughly address your concerns, and further strengthen your confidence in Den2MoEE.

Reviewer_B1rpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Den2MoEE, a framework for converting dense LLM embedding models into specialized Mixture-of-Experts architectures. Experiments on MTEB Code and Multilingual benchmarks demonstrate improved performance over comparable sparse baselines, with reduced activated parameters and higher inference throughput.

Critical strengths

1- Den2MoEE combines domain-aware neuron partitioning, heterogeneous expert design, and routing-aware retraining to address important challenges in dense-to-MoE conversion for embedding models. 2- The method outperforms comparable MoE reconstruction baselines on code and multilingual benchmarks while reducing activated parameters and improving throughput (Tables 1–3).

Critical weaknesses

1- Despite improvements over MoE baselines, Den2MoEE still exhibits noticeable accuracy degradation relative to its original dense backbones, particularly at the 4B scale (Table 3). 2- The framework combines several techniques, but some ablations modify multiple components simultaneously. This makes it difficult to determine the independent contributions of SVD experts, null experts, and routing strategies.

Questions for authors

1- Why does Den2MoEE-4B still lose 3.34 and 3.52 points on the Multilingual and Code benchmarks relative to the dense backbone (Table 3)? What primarily causes this gap? 2- Can the authors separately quantify the effects of SVD experts, null experts, router initialization, and dynamic routing bias? 3- How sensitive is expert construction to the choice, size, and domain distribution of the calibration dataset? 4-How does Den2MoEE compare with the original dense models in total GPU memory usage and deployment cost, given that Table 3 reports a larger memory footprint despite reduced activated parameters? 5- How much of the performance advantage over LLaMA-MoE comes from the proposed architecture versus differences in activated parameter budgets and retraining configurations?

7 comments
AuthorsOpenReview ↗

W1 & Q1: Addressing the Performance Gap—The Role of Recovery Data Distribution

We sincerely thank the reviewer for the positive assessment and for recognizing the practical value of our work. We are highly encouraged by your feedback, particularly regarding Den2MoEE's novel combination of domain-aware neuron partitioning, heterogeneous expert design, and routing-aware retraining, as well as the empirical validation that our method outperforms MoE baselines while reducing activated parameters and improving throughput.

We deeply appreciate the constructive questions aimed at making our analysis more rigorous. To address your concerns regarding the accuracy gap, component ablations, data sensitivity, GPU memory, and fair comparison, we have conducted extensive additional experiments. Below, we provide detailed responses to each point.

W1 & Q1: Addressing the Performance Gap—The Role of Recovery Data Distribution

We acknowledge the performance gap after compressing activated parameters from 4B to 1.8B. Our analysis indicates that the primary cause of this gap is the mismatch in recovery data scale and distribution, rather than the architectural capacity.

Data Distribution Mismatch: The original Qwen3-Embedding-4B was trained on a massive, highly curated dataset. As reported in the Qwen3-Embedding technical report (Table 6) [1], its weak-supervised pre-training utilized ~150M synthetic data, and supervised fine-tuning used ~7M annotated data and 12M synthetic data. The data covers a vast range of tasks (Retrieval, Bitext Mining, Classification, STS) and sources (MS MARCO, NQ, MIRACL, CodeSearchNet, etc.), with synthetic data generated by Qwen3-32B and filtered via semantic similarity.

In contrast, due to computational constraints for this research, our recovery training relies on a public subset (5.2M instances from KaLM-Embedding, more details in Table 7) covering code and multilingual tasks, failing to replicate the diverse task distribution and large-scale generation pipeline of the original dense model. This smaller, more concentrated dataset cannot fully replicate the rich task distribution of the original dense model, limiting their capacity to fully recover the dense model's zero-shot representation ability.

High Data Efficiency but Diminishing Returns: As shown in Table 5, Den2MoEE is highly data-efficient. It requires only 2M samples to achieve 54.34, surpassing LLaMA-MoE (2B activated) trained on 3M samples (53.05). At 3M data, Den2MoEE reaches 55.11, surpassing LLaMA-MoE trained on 4M data (54.64). This indicates that our structural design preserves the original capability effectively, and the gap to the dense model is largely constrained by the data available for recovery.

To further quantify the gap, we scale the recovery data to 7M (from the original 5M).

Recovery Data (M)Den2MoEE-4B-A1.8BLLaMA-MoE-4B-A2B
5.0057.5255.42
6.0058.1955.86
6.5058.3055.97
7.0058.3556.03

When scaling from 5M to 7M, Den2MoEE's gap to the dense model narrows from 3.34 to 2.51 points. However, the last 0.5M data only yields a 0.05 point gain. This confirms a saturation trend: merely scaling the current public data cannot fully bridge the gap. Improving data quality, task diversity, and matching the original synthetic generation pipeline are required, which we leave for future work due to resource limits.

[1] Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models, arXiv:2506.05176.

AuthorsOpenReview ↗

W1&Q1(Cont.); W2&Q2: Isolating the Independent Contributions of Individual Components

Evidence from Data-Matched Dense Backbones

If the performance gap is indeed primarily driven by the mismatch in recovery data distribution, then converting a dense model whose original pre-training data naturally overlaps with our public recovery subset should result in a significantly smaller gap.

To verify this, we conducted experiments on F2LLM-v2-4B [2], an embedding model trained entirely on publicly available corpora (e.g., MS MARCO, CodeSearchNet, etc.), which highly matches the distribution of our KaLM-Embedding recovery subset.

We evaluated the models across 9 MTEB Multilingual task categories. The results are summarized below:

ModelAct. SizeMean (Type)Bitext MiningClassificationClusteringInst. RetrievalMultilabel Class.Pair Class.RerankRetrievalSTS
F2LLM-v2-4B (Dense)4B58.2574.4970.7359.532.2526.5880.5169.3864.8475.91
LLaMA-MoE (F2LLM)2B55.05 (-3.20)69.3168.1655.742.0225.0876.2464.9262.4771.48
Den2MoEE (F2LLM)1.8B56.42 (-1.83)72.5368.6756.852.0625.5178.5267.0162.0274.58

On the data-matched F2LLM-v2 backbone, Den2MoEE's average performance drop is only 1.83 points, which is almost half of the 3.34-point drop observed on Qwen3-Embedding-4B. This strongly validates that the "missing" dense performance is largely an artifact of insufficient data distribution coverage during the recovery phase. This confirms that better data distribution matching leads to smaller performance degradation.

Even under a smaller activation budget (1.8B vs 2B), Den2MoEE outperforms LLaMA-MoE by a clear margin. The loss is reduced by 1.37 points (1.83 vs. 3.20). Across specific tasks like Bitext Mining (72.53 vs 69.31), Rerank (67.01 vs 64.92), and STS (74.58 vs 71.48), Den2MoEE preserves significantly more of the original dense model's semantic capability.

The comparison between Qwen3-Embedding (proprietary, massive data) and F2LLM-v2 (public, aligned data) conclusively demonstrates that the performance gap is heavily dictated by the alignment between the dense model's original training distribution and the recovery data available. The architecture itself (Den2MoEE) is highly effective at minimizing parameter-sparsification loss, as evidenced by its robust performance on the matched backbone.

In the revised version, we have explained the performance gap in Section 4.2. Moreover, we have added the role of recovery data distribution in Appendix F.8.

[2] F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World, arxiv:2603.19223

W2 & Q2: Isolating the Independent Contributions of Individual Components

We highly appreciate this constructive feedback. We completely agree that the original Table 4 (which removed SVD and Null experts jointly) only demonstrated their combined effect and failed to isolate their individual contributions. To provide a rigorous component-wise analysis, we have conducted strict single-component ablations.

We start from the full Den2MoEE-4B-A1.8B model and remove only one component at a time, keeping all other architectures and training settings identical. The results are summarized as:

Model / Ablation SettingMTEB (Code)MTEB (Multilingual)
Den2MoEE-4B-A1.8B (Full Model)76.5457.52
w/o Dynamic Routing Bias74.17 (-2.37)56.06 (-1.46)
w/o SVD Experts74.76 (-1.78)56.54 (-0.98)
w/o Router Initialization75.38 (-1.16)56.81 (-0.71)
w/o Null Experts76.09 (-0.45)57.28 (-0.24)

Detailed Mechanistic Analysis:

1. Dynamic Routing Bias (Most critical: -2.37 / -1.46)

  • Mechanism: Conventional MoE relies on auxiliary load-balancing losses, which can force artificial uniformity and suppress expert specialization. Our dynamic bias (Eq. 11) adjusts the routing logits based on the gap between the DomainCoverage-based expected ratio and the actual batch load.
  • Observation: Removing it causes the largest performance drop. This indicates that without continuous, coverage-aware load correction, the router suffers from severe "expert collapse" or uneven distribution, failing to route tokens to their true domain-aligned experts.
AuthorsOpenReview ↗

W2&Q2(Cont.); Q3: Robustness of Expert Construction to Calibration Data

2. SVD Experts (Highly impactful: -1.78 / -0.98)

  • Mechanism: We apply low-rank SVD to the Up/Gate projections of specialized experts, truncating to ~60% rank while preserving 80% energy.
  • Observation: Removing SVD experts (reverting to full-rank dense experts) causes a notable drop. This proves that SVD experts are not just for computational savings; they act as a spectral regularizer. By suppressing redundant high-dimensional noise and forcing the expert to focus on informative low-rank semantic subspaces, it improves the quality of the recovered representations, especially for Code.

3. Router Initialization (Significant: -1.16 / -0.71)

  • Mechanism: We initialize the router by using the average up-projection weights of each expert's neurons as semantic anchors, progressively aligning the router weights (Eq. 9-10).
  • Observation: Removing this initialization (reverting to random initialization) leads to a clear degradation. It confirms that our semantic alignment strategy successfully mitigates the "cold-start" problem in MoE training. It prevents early-stage routing chaos and ensures experts begin training with a meaningful specialization direction from step zero.

4. Null Experts (Small but consistently positive: -0.45 / -0.24)

  • Mechanism: Null experts occupy routing slots to absorb low-information tokens (e.g., stopwords, punctuation) without extra computation.
  • Observation: While the absolute drop is small, its impact is positive and consistent across both benchmarks. It proves that providing a "garbage collection" mechanism improves routing resolution, allowing specialized SVD experts to allocate more capacity strictly to semantically meaningful tokens.

The single-component ablation clearly reveals the functional hierarchy in our architecture: Dynamic routing bias and SVD specialization serve as the core drivers of performance preservation, while semantic router initialization and Null experts provide essential structural and routing refinements. In the revised version, we have added this detailed ablation studies and mechanistic analysis to Appendix F.9.

Q3: Robustness of Expert Construction to Calibration Data

We thank the reviewer for this critical question. Because our domain vectors are estimated from the calibration set, it is vital to ensure that the resulting expert construction is not a fragile artifact of specific data samples. We have conducted a comprehensive sensitivity analysis covering three aspects: sample size, random choice (seeds), and domain coverage.

Our Conclusion: A moderate, balanced calibration set (e.g., 150 samples per domain) is sufficient for highly stable expert construction. However, dropping entire domain categories significantly alters the structure, indicating that domain coverage is more critical than per-domain sample size.

1. Metric Design Motivation

To rigorously quantify stability, we designed three complementary metrics, each targeting a specific level of the expert construction pipeline:

  • Adjusted Rand Index (ARI) for Full Partitioning Stability: To rigorously quantify the structural stability across different calibration sets, we employed the Adjusted Rand Index (ARI). ARI is a classical and widely adopted metric in clustering evaluation, mathematically grounded in the contingency table (confusion matrix) of two partitions. ARI measures the structural similarity between two groupings, correcting for random chance. It is permutation-invariant, meaning we don't need to align expert IDs between the reference and test sets. It answers: "Are the neurons grouped in the same topological way?"

    Let RiR_i and TjT_j be the sets of neurons assigned to the ii-th expert in the reference partition and the jj-th expert in the test partition, respectively. We construct a contingency table where:

    • nij=∣Ri∩Tj∣n_{ij} = |R_i \cap T_j| is the number of neurons in both expert RiR_i and TjT_j.
    • ai=∑jnija_i = \sum_j n_{ij} and bj=∑inijb_j = \sum_i n_{ij} are the marginal totals (expert sizes).
    • N=∑i,jnijN = \sum_{i,j} n_{ij} is the total number of neurons.

    We then count the number of neuron pairs:

    • I=∑i,j(nij2)I = \sum_{i,j} \binom{n_{ij}}{2} (pairs in the same expert in both partitions)
    • A=∑i(ai2)A = \sum_i \binom{a_i}{2} (total pairs in the same reference expert)
    • B=∑j(bj2)B = \sum_j \binom{b_j}{2} (total pairs in the same test expert)
    • P=(N2)P = \binom{N}{2} (total possible neuron pairs)

    The ARI is defined as: ARI=I−AB/P12(A+B)−AB/P\text{ARI} = \frac{I - AB/P}{\frac{1}{2}(A+B) - AB/P} (Note: (x2)=x(x−1)/2\binom{x}{2} = x(x-1)/2 denotes the number of pairs). An ARI of 1.0 indicates perfect structural agreement, while ~0 indicates the agreement is equivalent to random chance.

Reviewer_RCgUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Den2MoEE, a framework for converting dense embedding models into sparse MoE models. The method uses domain-conditioned neuron activations to construct specialized experts, together with a heterogeneous expert design and a two-stage retraining strategy. Experiments on code and multilingual embedding benchmarks show improved quality–efficiency trade-offs compared with prior dense-to-MoE reconstruction baselines.

Critical strengths

1.The paper addresses a practical problem: reducing the inference cost of strong dense embedding models without training an MoE model from scratch. The proposed reconstruction framework consistently preserves more of the original dense-model performance than the corresponding LLaMA-MoE reconstruction under comparable activated-parameter budgets. 2.The experimental evaluation is relatively extensive. Besides the main code and multilingual benchmarks, the paper includes analyses of clustering strategies, routing behaviour, training-data efficiency, different dense backbones, and an additional domain. The reported sparsity also translates into measurable latency and throughput improvements

Critical weaknesses

The main concern is the justification for the core expert-partitioning assumption:The method treats similarity between domain-conditioned activation profiles as a signal that FFN dimensions should be grouped into the same expert, and this structure is then used throughout the subsequent expert construction and routing design. The paper provides substantial empirical evidence that the resulting method works, including clustering comparisons, routing analysis, different backbones, and additional domains. However, these experiments mainly establish the usefulness of the signal rather than why this similarity should correspond to a reusable functional grouping of FFN dimensions. Without some theoretical, statistical, or consistently observed empirical basis for this relationship, the method is more clearly supported as an effective heuristic than as a generally motivated partitioning principle. Additional evidence establishing this connection would substantially strengthen the soundness of the core method.

Questions for authors

Is there theoretical, statistical, or empirical evidence supporting the assumption that dimensions with similar domain-conditioned activation profiles form a functionally coherent expert, beyond the downstream performance obtained after using this signal for partitioning?

6 comments
AuthorsOpenReview ↗

Theoretical and empirical analysis about the domain-conditioned activation profiles—Part 1

We sincerely thank you for your thoughtful and constructive feedback. We appreciate your recognition of the practical value and empirical thoroughness of our approach. We also particularly value the critical concern regarding the theoretical justification of our core expert-partitioning assumption. We fully agree that downstream performance alone is insufficient to prove that "similarity in domain-conditioned activation profiles" corresponds to a "functionally coherent grouping."

To directly resolve this concern, we have re-evaluated our methodology from the ground up. We provide our detailed response in three parts: (1) the methodological motivation; (2) independent empirical validations; and (3) the theoretical bound linking our grouping to the reconstruction error.

1. Methodological motivation that uses fine-grained domain-conditioned activation profiles

FFN neurons act as key-value memories [1], where activations reflect input pattern detection. To accurately capture this, we first use TokenScore (combining terminal token attention and perturbation-based semantic shift) to weight tokens. This ensures we measure whether a neuron is active on semantically important tokens, rather than just average noise. We then compute the standardized TokenScore-weighted conditional activation s^jd\hat s_j^d for neuron jj and sub-domain dd:

v_j=[\hat s_j^1,\ldots,\hat s_j^D].$$ Here, $v_j$ is the neuron's conditional response profile. By comparing the direction of these profiles via cosine similarity, we group neurons with similar cross-domain relative preferences. This is aligned with prior findings: MoEfication [2] clusters neurons by co-activation to approximate FFN computations; Tan et al. [3] observed language-related activation patterns and overlapping task sub-networks. Our method extends this by using domain-conditioned, importance-weighted statistics to define the grouping basis, rather than static weights. **2. Empirical validations that uncovers the co-activation and contribution coverage** To support the assumption that dimensions with similar domain-conditioned activation profiles form a functionally coherent expert, we further conducted two strict statistical experiments. **Experimental Motivation and Design.** To prove that domain-profile similarity translates to a truly reusable functional grouping beyond just yielding good downstream MTEB scores, we design two independent experiments to validate two hypotheses: - **H1 (Micro-level Co-activation):** Neurons grouped by our domain profiles are genuinely activated together on relevant unseen inputs. - **H2 (Macro-level Contribution Retention):** When a router selects one of these groups (experts) under a fixed budget, it covers a significantly larger portion of the original dense FFN's mathematical contribution. To validate the *H1 (Micro-level Co-activation)* hypothes, we design the Co-activation ($\phi$ Correlation) metric. Simply measuring the raw co-activation rate is misleading because neurons with high marginal activation frequencies (e.g., punctuation detectors) will naturally co-occur. We employ the $\phi$ correlation coefficient (which is mathematically equivalent to the Pearson correlation coefficient for Bernoulli variables) to control for marginal frequencies and measure true synergistic activation. **Detailed Definition of Co-activation ($\phi$ Correlation).** We define a strong activation event for neuron $j$ at token $t$ as $B_j(t) \in \{0,1\}$, where $B_j(t) = \mathbf{1} \lbrace |a_j(t)| > \tau_j \rbrace$, and $\tau_j$ is the 90th percentile of calibration activation. For a given domain $d$, we define the following empirical probabilities: * $p_i^d$: The marginal probability that neuron $i$ is strongly activated. (i.e., $p_i^d = \mathbb{E}_{t \sim d}[B_i(t)]$). * $p_j^d$: The marginal probability that neuron $j$ is strongly activated. * $p_{ij}^d$: The joint probability that both neurons $i$ and $j$ are simultaneously activated. (i.e., $p_{ij}^d = \mathbb{E}_{t \sim d}[B_i(t)B_j(t)]$). The $\phi$ coefficient for the neuron pair $(i,j)$ in domain $d$ is: $$\phi_{ij}^d = \frac{p_{ij}^d - p_i^d p_j^d}{\sqrt{p_i^d(1-p_i^d)p_j^d(1-p_j^d)}}$$ [1] Transformer Feed-Forward Layers Are Key-Value Memories. EMNLP 2021, pp. 5484–5495. [2] MoEfication: Transformer Feed-forward Layers are Mixtures of Experts. Findings of ACL 2022, pp. 877–890. [3] Neuron Specialization: Leveraging Intrinsic Task Modularity for Multilingual Machine Translation. EMNLP 2024, pp. 6506–6527.
AuthorsOpenReview ↗

Theoretical and empirical analysis about the domain-conditioned activation profiles—Part 2

The Intuition behind the Co-activation (ϕ\phi Correlation) are as follows:

  • Numerator (pijd−pidpjdp_{ij}^d - p_i^d p_j^d): This measures the excess co-activation. If neurons ii and jj were completely independent, their joint activation probability would be the product of their marginals pidpjdp_i^d p_j^d. A positive numerator means they co-activate more often than random chance, indicating true functional synergy.
  • Denominator (pid(1−pid)pjd(1−pjd)\sqrt{p_i^d(1-p_i^d)p_j^d(1-p_j^d)}): This is the product of the standard deviations of the two Bernoulli variables. It acts as a normalization factor to bound the ϕ\phi coefficient within [−1,1][-1, 1], ensuring that highly active neurons (with variance close to 0.5) do not disproportionately inflate the correlation.

To evaluate our clustering strategy, we randomly sample a large number of neuron pairs.

  • ϕ‾withind\overline{\phi} _ {\mathrm{within}}^d: The average ϕijd\phi_{ij}^d over pairs (i,j)(i,j) that are assigned to the same expert.
  • ϕ‾betweend\overline{\phi} _ {\mathrm {between}}^d: The average ϕijd\phi_{ij}^d over the same number of pairs that are assigned to different experts.

We report the difference Δϕd=ϕ‾withind−ϕ‾betweend\Delta\phi^d = \overline{\phi} _ {\mathrm{within}}^d - \overline{\phi} _ {\mathrm {between}}^d. A positive Δϕd\Delta\phi^d proves that our domain-profile clustering successfully captures the co-activation structure and concentrates synergistic neurons within the constructed experts.

Detailed Definition of Contribution Retention Rate

In a standard MoE model, the router makes a discrete, macro-level decision to select a subset of experts (Top-kk). However, the final output is the continuous sum of the individual neuron contributions within those experts. If a fixed budget of experts fails to capture the most significant contributions from the dense model, the approximation error will be large. Therefore, we define the Contribution Retention Rate metric to measure how much of the original dense FFN's output magnitude (energy) is successfully covered by our selected experts.

Following Eq. (2), let the dense FFN output for an input token xx (hidden state) be F(x)=∑j=1Naj(x)wjF(x) = \sum_{j=1}^N a_j(x)w_j, where aj(x)a_j(x) is the scalar activation value of neuron jj, wj∈Rdw_j \in \mathbb{R}^d is the corresponding output vector (the jj-th row of the down-projection matrix).

Inspired by the sub-update analysis of FFNs [4], the dense FFN output can be decomposed as the sum of each neuron's scalar activation multiplied by its corresponding output vector. Specifically, we define the contribution magnitude of neuron jj as the L2 norm of its vector output: ∥cj(x)∥2=∣aj(x)∣∥wj∥2\|c_j(x)\|_2 = |a_j(x)| \|w_j\|_2.

The total contribution magnitude of the entire dense FFN layer is the sum of all individual magnitudes: T(x)=∑j=1N∣aj(x)∣∥wj∥2T(x) = \sum_{j=1}^N |a_j(x)| \|w_j\|_2

When experts are constructed, each expert ee contains a disjoint subset of neurons NeN_e. The contribution magnitude of an expert is:

ue(x)=∑j∈Ne∣aj(x)∣∥wj∥2u_e(x) = \sum_{j \in N_e} |a_j(x)| \|w_j\|_2

Given a fixed routing budget (e.g., Top-kk experts, where Sk(x)S_k(x) is the set of selected experts), the Contribution Retention Rate Hk(x)H_k(x) is defined as the ratio of the retained contribution magnitude to the total magnitude:

Hk(x)=∑e∈Sk(x)ue(x)T(x)H_k(x) = \frac{\sum_{e \in S_k(x)} u_e(x)}{T(x)} where Hk(x)∈[0,1]H_k(x) \in [0, 1] measures the percentage of the dense FFN's total output contribution that is preserved by the selected experts. A higher Hk(x)H_k(x) means the router's Top-kk decision successfully covers the most critical neurons.

[4] Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space. EMNLP 2022, pp. 30–45.

AuthorsOpenReview ↗

Theoretical and empirical analysis about the domain-conditioned activation profiles—Part 3

Experimental Results and Analysis

We compare three partitioning strategies: Random, Up-proj (Weight-based), and Den2MoEE (Domain Vector) on independent hold-out data (15k docs across 30 domains, unseen during calibration/training). All methods use the same frozen Qwen3-Embedding model, same expert counts, and no retraining, isolating the partition effect.

Table 1 below presents the results (point estimates with 95% document-level stratified bootstrap confidence intervals). We evaluate HkH_k under a strict budget of k/E=1/4k/E = 1/4 (i.e., simulating a scenario where the router can only compute 25% of the experts). A better partitioning strategy (like ours) should group highly co-activated neurons together, maximizing HkH_k under this constraint.

Table 1: Hold-out Co-activation and Contribution Retention (Qwen3-Embedding-0.6B, k/E=1/4k/E=1/4).

Expert PartitioningWithin ϕ\phiBetween ϕ\phiΔϕ\Delta\phi (Co-activation) ↑HkH_k ( Contribution Retention) ↑
Random0.01710.01710.0000 [0.0000, 0.0001]29.29% [29.28, 29.29]
Up-proj (Weight)0.01790.01710.0009 [0.0008, 0.0009]30.66% [30.65, 30.66]
Den2MoEE (Domain Vector)0.01890.01700.0019 [0.0019, 0.0019]37.65% [37.63, 37.67]

Table 2: Contribution Retention (HkH_k) under Varying Expert Budgets (Unit: %, Point Estimate [95% CI]).

Expert PartitioningHkH_k (k/E=1/8k/E=1/8) ↑HkH_k (k/E=1/4k/E=1/4) ↑HkH_k (k/E=1/2(k/E=1/2) ↑
Random15.45 [15.45, 15.45]29.29 [29.28, 29.29]54.92 [54.91, 54.92]
Up-proj (Weight)16.59 [16.59, 16.60]30.66 [30.65, 30.66]56.18 [56.18, 56.19]
Den2MoEE (Domain Vector)23.46 [23.44, 23.48]37.65 [37.63, 37.67]61.11 [61.10, 61.12]
  1. On Co-activation (Δϕ\Delta\phi): Although the absolute value of ϕ\phi is small (due to the inherent high sparsity and noise of FFN neurons), our method achieves a Δϕ\Delta\phi of 0.0019, which is 2.1× higher than the weight-based baseline (0.0009) and effectively infinite compared to Random (0.0000). The non-overlapping 95% CIs mathematically prove that our clustering is driven by actual semantic co-activation, not statistical coincidence.
  2. On Robust Contribution Retention (HkH_k): Table 2 demonstrates a consistent and substantial superiority of our partitioning strategy across three distinct routing budgets (k/E=1/8,1/4,1/2k/E = 1/8, 1/4, 1/2).
    • Under the main evaluation budget (k/E=1/4k/E=1/4), our method retains 37.65% of the dense FFN's contribution magnitude, outperforming the weight-based baseline (30.66%) and the random baseline (29.29%). This represents a 22.8% relative improvement over Up-proj.
    • More importantly, under the extreme sparsity budget (k/E=1/8k/E=1/8, meaning the router can only select 12.5% of the experts), the gap widens significantly. Our method retains 23.46% of the contribution, which is a massive 41.4% relative gain over the Up-proj baseline (16.59%). This proves that when computational resources are severely constrained, our domain-profile clustering is highly effective at packing the most critical, high-magnitude neurons into a very small number of selectable units.
  3. Theoretical Implication (Link to Part 4): The sharp increase in HkH_k is the direct reason why Den2MoEE suffers less performance drop. By grouping neurons with similar domain profiles, we ensure that a single routing decision covers more of the highly relevant computational load, which directly minimizes the MoE truncation error.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.