- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Den2MoEE, a framework for reconstructing pretrained dense embedding models into sparse MOE models to reduce inference costs while preserving their representation capabilities. The method constructs neuron domain-preference vectors from TokenScore-weighted activations and groups neurons with similar preferences into experts. It combines shared experts, low-rank SVD experts, and null experts to enable conditional computation. To mitigate performance degradation after conversion, the framework introduces neuron-weight-guided router initialization, domain-coverage-aware routing biases, and a two-stage recovery training strategy. It further incorporates routing signals from multiple layers to complement the final embedding representations.
Critical strengths
-
The framework reuses existing dense embedding models to construct MoE architectures, reducing inference computation. This is particularly relevant for encoding large document collections and serving latency-sensitive retrieval applications.
-
The expert partitioning strategy is tailored to embedding tasks. By combining TokenScore with neuron responses across domains, the method constructs experts based on their observed behavior on data, providing a more task-relevant partitioning criterion than random splitting or weight similarity alone.
-
The paper is well written, and the experimental evaluation is relatively detailed.
Critical weaknesses
-
LLaMA-MoE is the only direct conversion baseline evaluated. Comparisons with other relevant MoE embedding approaches, such as Nomic Embed v2 and MoEE, would help establish the effectiveness and positioning of the proposed framework. These comparisons should account for differences in their objectives and training settings. The related work discussion should also be expanded accordingly.
-
The efficiency evaluation is primarily conducted with a sequence length of 512 tokens and a batch size of 128, whereas Qwen3-Embedding supports inputs of up to 32K tokens. The experiments do not establish whether longer inputs lead to greater performance degradation after conversion or smaller efficiency gains. I therefore recommend evaluating document encoding time and retrieval effectiveness on long-document retrieval tasks, such as LongEmbed, under different maximum input lengths, including 4K and 8K tokens.
-
DomainCoverage counts the domains in which an expert’s score exceeds the median expert score and uses this quantity to determine its target routing proportion. However, the number of domains an expert covers does not necessarily reflect how frequently that expert is needed under the actual input distribution. Consequently, the resulting routing biases may be misaligned with deployment workloads. Experiments with varying domain proportions and unseen domains are needed to assess robustness and generalization. Furthermore, the training and evaluation data cover largely similar domains, making additional out-of-domain evaluations particularly valuable.
Questions for authors
n/a
6 comments
W2: Long-Context Efficiency and Retrieval Effectiveness
We deeply appreciate this constructive suggestion. We have conducted extensive experiments on the LongEmbed benchmark at maximum input lengths of 4K and 8K tokens. Our findings demonstrate that Den2MoEE maintains a bounded performance degradation and substantial efficiency gains in long-context tasks.
1. Document Encoding Efficiency under Long Contexts
We measured the wall-clock time for encoding the full document corpus of four real-world tasks (totaling 1,188 documents). The measurement was conducted on a single NVIDIA H20 GPU (BF16 precision), including tokenization, forward pass, last-token pooling, and normalization. We used the same batch size for both models at each length.
Table 1: End-to-End Document Encoding Efficiency
| Model | Max Length | Batch Size | Total Time (s) | Throughput (docs/s) | Avg Time (ms/doc) |
|---|---|---|---|---|---|
| Qwen3-Embedding-4B (Dense) | 4K | 4 | 428.0 | 2.8 | 360.3 |
| Den2MoEE-4B-A1.8B | 4K | 4 | 250.4 | 4.7 | 210.8 (-41.5%) |
| Qwen3-Embedding-4B (Dense) | 8K | 2 | 1004.7 | 1.2 | 845.7 |
| Den2MoEE-4B-A1.8B | 8K | 2 | 630.8 | 1.9 | 531.0 (-37.2%) |
Analysis on Efficiency:
- Den2MoEE achieves a 41.5% reduction in latency at 4K and a 37.2% reduction at 8K, demonstrating robust acceleration in long-document encoding.
- Why does the efficiency gain slightly decrease at 8K? As sequence length grows, the attention computation and memory I/O become the dominant bottlenecks. Because Den2MoEE only sparsifies the FFN layers while keeping the attention layers dense, the fraction of time spent on the sparsified FFN decreases, slightly diluting the overall end-to-end speedup. Nevertheless, a ~37% latency reduction at 8K is still highly significant for real-world RAG and search systems.
2. Long-Document Retrieval Effectiveness
We evaluated retrieval performance across six LongEmbed tasks (Passkey, Needle, NQA, QMS, SFD, WQA). The results (with drops relative to the same-length dense model in parentheses) are shown below:
Table 2: LongEmbed Retrieval Performance
| Model | Act. Params | Max Len | Passkey | Needle | NQA | QMS | SFD | WQA | Average |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-Embedding-4B (Dense) | 4.0B | 4K | 73.5 | 56.2 | 55.1 | 41.3 | 95.3 | 85.6 | 67.8 |
| Den2MoEE-4B-A1.8B | 1.8B | 4K | 71.2 | 53.7 | 53.1 | 39.8 | 94.4 | 84.4 | 66.1 (-1.7) |
| LLaMA-MoE (4/16, A2B) | 2.0B | 4K | 70.4 | 52.8 | 52.3 | 39.1 | 93.8 | 83.4 | 65.3 (-2.5) |
| Qwen3-Embedding-4B (Dense) | 4.0B | 8K | 84.8 | 67.5 | 63.4 | 48.7 | 97.4 | 91.1 | 75.5 |
| Den2MoEE-4B-A1.8B | 1.8B | 8K | 82.4 | 64.9 | 61.2 | 46.8 | 96.4 | 89.3 | 73.5 (-2.0) |
| LLaMA-MoE (4/16, A2B) | 2.0B | 8K | 81.6 | 64.1 | 60.4 | 46.1 | 96.0 | 88.6 | 72.8 (-2.7) |
Analysis on Performance:
- Bounded Degradation: From 4K to 8K, Den2MoEE's performance drop relative to the dense model remains extremely stable, moving only slightly from -1.7 to -2.0 points. There is no catastrophic forgetting or error explosion at longer contexts.
- Consistent Superiority over LLaMA-MoE: Under the exact same sparsity budget, Den2MoEE consistently outperforms LLaMA-MoE by +0.8 (at 4K) and +0.7 (at 8K) points, while activating fewer parameters (1.8B vs 2.0B).
3. Mechanistic Explanation for Bounded Degradation
Why does Den2MoEE not suffer from exacerbated degradation at longer inputs? We attribute this to two key design choices:
- Length-Matched Recovery Data: Our recovery training data has an average length of ~5.4K (truncated at 8K), which perfectly covers the 4K-8K test range. This prevents the router from out-of-distribution routing on long documents.
- Architectural Isolation of Attention: Our dense-to-MoE reconstruction only modifies and sparsifies the FFN layers, leaving the attention layers completely untouched. Attention is the primary mechanism for extracting and aggregating long-range information. By preserving the original pre-trained attention weights intact, Den2MoEE retains the dense model's intrinsic long-context retrieval capability, allowing it to gracefully scale with input length.
In the revised version, we have added these results, measurement details, and analysis to Section 4.4. to demonstrate long-context efficiency and retrieval effectiveness.
W3: DomainCoverage, Workload Shifts, and Out-of-Domain Evaluation
W3: DomainCoverage, Workload Shifts, and Out-of-Domain Evaluation
We deeply appreciate this insightful critique. We fully agree with the reviewer's conceptual distinction: DomainCoverage measures the structural breadth of an expert's domain preference, not the expected frequency under a specific deployment distribution. In our framework, it serves as a Maximum-Entropy Default Prior for expert construction and routing regularization. While Eq. (11) updates the bias based on actual batch-level loads, the target ratio is still initially derived from coverage.
To empirically evaluate this potential misalignment and test generalization, we conducted comprehensive experiments on (1) varying domain proportions and (2) an extended unseen domain.
Robustness to Varying Domain Proportions (Skewed Calibration)
We simulated heavily skewed calibration distributions by varying the number of code and multilingual domains used during expert construction. The results are shown below:
| Model / Variant | Code / Multi. Domains | MTEB (Code) | MTEB (Multi.) |
|---|---|---|---|
| Den2MoEE-4B-A1.8B (Default) | 14 / 16 | 76.54 | 57.52 |
| Code-domain-heavy | 14 / 8 | 75.91 (-0.63) | 55.81 (-1.71) |
| Multilingual-domain-heavy | 7 / 16 | 74.31 (-2.23) | 57.11 (-0.41) |
| LLaMA-MoE (Baseline) | N/A (Weight-based) | 74.16 | 55.42 |
- As expected, the performance drop is larger in the domain family whose coverage is reduced (e.g., dropping Multi domains causes a 1.71 drop in Multi, while dropping Code domains causes a 2.23 drop in Code).
- Crucially, both skewed variants still outperform the LLaMA-MoE baseline on both benchmarks, even in the extreme Multilingual-heavy case where the Code margin is narrow (+0.15 points). This demonstrates that the routing mechanism, supported by our dynamic bias and Shared Experts, is highly resilient and can gracefully degrade rather than catastrophically collapse when the calibration distribution shifts.
Out-of-Domain (OOD) Evaluation on an Unseen Domain
To test generalization, we extended our recovery corpus with approximately 1M legal domain samples (from Law-GPT, THUCNews, and Zhihu-KOL), which were originally entirely unseen in the code/multilingual training. We evaluated the models on MTEB (Law), Code, and Multilingual.
| Model | MTEB (Law) | MTEB (Code) | MTEB (Multi.) |
|---|---|---|---|
| Qwen3-Embedding-4B (Dense) | 69.54 | 80.06 | 60.86 |
| Qwen3-Embedding-4B-MoE (LLaMA-MoE) | 66.92 | 74.08 | 55.47 |
| Den2MoEE-4B-A1.8B | 68.19 | 76.52 | 57.63 |
- Den2MoEE outperforms LLaMA-MoE by +1.27 (Law), +2.44 (Code), and +2.16 (Multi) points.
- Its Code and Multilingual scores (76.52 / 57.63) remain exceptionally close to the original two-domain setting (76.54 / 57.52), proving that expanding domain coverage via domain-specific recovery data does not cause catastrophic forgetting of the original capabilities.
- Important Boundary Declaration: Since the legal data was included in the recovery training, this experiment measures the model's adaptability to a newly introduced domain, rather than zero-shot generalization to an entirely unseen domain. Uncovered specialized domains may still induce suboptimal routing, which we will explicitly state as a limitation in the revised manuscript.
Clarification and Revision. We agree with the reviewer that calibration coverage does not equal deployment traffic. To ensure full transparency, we have revised the manuscript to:
- Explicitly state in Section 3.3 that DomainCoverage is a structural prior, not a deployment frequency predictor.
- Include the varying-proportion experiment in Appendix F.14 and the domain-extension experiment in Appendix F.7 to demonstrate the system's tolerance to calibration shifts and its extensibility.
We sincerely thank the reviewer for these highly constructive suggestions. Adding the Nomic-style baseline, the LongEmbed evaluations, and the varying domain evaluations has significantly strengthened our empirical evidence. We hope these detailed analyses and our clarifications thoroughly address your concerns, and further strengthen your confidence in Den2MoEE.
W1: Broadening Baselines and Clarifying Methodological Positioning
We sincerely thank the reviewer for the positive assessment and for recognizing the practical value of our work. We are highly encouraged by your feedback, particularly regarding our framework's ability to reuse existing dense embedding models for efficient MoE reconstruction, the task-relevant expert partitioning strategy (TokenScore + domain-conditioned profiles), and the detailed experimental evaluation.
We deeply appreciate the constructive questions aimed at broadening our baselines, testing long-context scenarios, and examining workload generalization. We have conducted additional experiments to address these concerns and provide detailed responses below.
W1: Broadening Baselines and Clarifying Methodological Positioning
We thank the reviewer for this excellent suggestion. To clarify the positioning, we must first distinguish the objectives of these methods: MoEE is a training-free representation extraction method, while Nomic Embed v2 is a capacity-expanding upcycling method. To provide a fair comparison, we added a controlled Nomic-style baseline and clarified how MoEE is utilized.
1. Clarification on MoEE (Not a Conversion Baseline) MoEE [1] extracts representations from the router weights and hidden states of an existing MoE without additional training. All MoE evaluations in our paper already adopt this MoEE-style embedding construction. Specifically, we average router weights over consecutive layers and concatenate them with hidden states. As shown in Section 4.4 and Figure 3, this extraction boosts our MTEB Code scores from 71.30 to 72.68 (+1.38) for 0.6B, and 74.38 to 76.54 (+2.16) for 4B. Thus, MoEE is seamlessly incorporated into our evaluation protocol, not omitted.
2. A Controlled Nomic-Style Upcycling Baseline Nomic Embed v2 [2] uses sparse upcycling to expand capacity. To compare fairly without confounding differences in base backbones and massive proprietary training data, we adapted Nomic's architectural recipe onto our identical Qwen3-Embedding-0.6B backbone and recovery dataset. Specifically, we replaced every other FFN layer (14 out of 28) with 8 full-sized experts, used Top-2 routing, random router initialization, and trained with InfoNCE + load-balancing loss.
3. Results and Analysis As shown in the tables below, Den2MoEE achieves the best quality-efficiency trade-off:
Den2MoEE achieves a Multilingual score of 55.05 and Code score of 72.68, using only 0.4B activated and 0.6B total parameters.
Nomic-Embed-v2-MoE (Adapted) achieves 53.90 / 71.83 with a much larger budget (0.7B activated / 1.8B total).
LLaMA-MoE falls behind at 52.73 / 67.66 under our budget.
Table 1: MTEB Multilingual (Task-type Mean)
Table 2: MTEB Code (Task-type Mean)
Copying intact pretrained FFNs (Nomic-style) can ease recovery compared to neuron partitioning, but it drastically increases total parameters. Our Den2MoEE introduces domain-aware expert structure and explicitly supports routing adaptation, achieving superior performance under a stricter parameter budget. We will add these results and a comprehensive Related Work section distinguishing capacity-expanding upcycling (Nomic), embedding extraction (MoEE), and efficiency-oriented reconstruction (Den2MoEE).
Details about broader baselines and methodological positioning can be found in Appendix F.13 of the revised version.
[1] Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free, ICLR 2025.
[2] Training Sparse Mixture Of Experts Text Embedding Models, arXiv:2502.07972.