- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MoEGen aims to reduce the storage cost of mixture-of-LoRA adaptation while retaining input-dependent specialization. A frozen sentence encoder produces a prompt representation, and a component-specific router selects compact expert codes. A shared hypernetwork uses these codes and learned coordinate embeddings to produce LoRA A matrices, which are paired with trainable B matrices shared across inputs. Under Eq. (6), candidate A matrices are generated separately and then combined using the routing weights. The method is evaluated on eight commonsense reasoning benchmarks and a joint training mixture of medical and legal tasks. It achieves the highest average score on all three backbones in each setting, with gains of 0.6-1.1 percentage points over the strongest commonsense baselines and approximately 4.6-6.2 points on the joint benchmark. The paper also includes routing and encoder ablations, qualitative examples, and inference measurements.
Critical strengths
-
The compact expert representation addresses a practical limitation of MoE-based adaptation. Adding an expert mainly adds a small code and its gating parameters, rather than another complete set of adapters. The results in Table 1 are encouraging: MoEGen has fewer trainable parameters than both MoE baselines while achieving a higher average accuracy. Even if the method is interpreted primarily as a compressed parameterization of an expert pool, this is a useful contribution.
-
The positive results extend across several backbones and two different adaptation settings. The joint benchmark includes both classification and generation, which gives a broader test than commonsense classification alone. The ablation in Table 2(a), where removing MoE lowers the average from 61.5 to 59.8, provides some evidence that routing contributes beyond the hypernetwork alone. I also appreciate the inclusion of inference measurements in Appendix C, since a reduction in trainable parameters does not automatically imply faster inference.
Critical weaknesses
-
The generation mechanism is ambiguous, and the claimed distinction from mixing fixed experts is not established. The abstract and Figure 2 diagram suggest that expert codes are mixed before entering the hypernetwork. In contrast, Eq. (6) and the figure caption generate one candidate matrix per expert and then mix the matrices. These operations are generally different because the hypernetwork includes LayerNorm and GELU. Under Eq. (6), each candidate A_i^(l) depends only on learned coordinates, an expert code, and the hypernetwork parameters. Once training is complete, it is independent of the current prompt and can be precomputed. The update can therefore be written as Delta W^(l)(x) = sum_i w_i^(l)(x) B^(l)(A_i^(l))^T, which is functionally a mixture of fixed experts sharing B. The final update is still input-dependent, and the compressed representation remains useful, but the stronger distinction drawn in Section 4.2 is not justified by this formulation. This matters to both the novelty claim and the interpretation of the results.
-
The stated routing equations leave an important problem in the top-1 ablation. Equation (5) normalizes probabilities only over the selected experts. For top-1 routing, the selected weight is identically one, so its derivative with respect to the logits is zero. With ordinary hard top-k differentiation, the selection itself does not provide a gradient either. Equation (8) uses the same post-selection weights, so the stated balancing loss does not resolve this issue. Unless the implementation uses an additional mechanism, the top-1 router cannot learn through these paths. The comparison in Appendix A.2 would then change router trainability as well as the number of active experts. I cannot interpret the reported advantage of top-2 as evidence for the benefit of expert mixing without clarification of the actual gradient computation.
-
The comparisons do not isolate the contribution of the hypernetwork parameterization. Appendix A.1 uses different adaptation locations, ranks, expert counts, and routing schemes across methods. MoEGen additionally receives representations from a frozen 0.6B encoder. These are reasonable configurations for comparing complete systems, but they do not establish which part of MoEGen produces the gain. A particularly relevant control would use the same prompt encoder, component-level router, adaptation locations, and shared B matrices, while learning the expert A matrices directly. The "w/o MoE" variant is useful but does not specify its conditioning path or parameter budget clearly enough to fill this gap. Lower trainable-parameter counts alone do not establish that parameter generation, rather than these other design choices, explains the improvement.
-
The paper does not report repeated experiments or variation across random seeds. There are no reported run counts, means over independent training runs, or standard deviations for the main results and ablations. The commonsense improvements over the strongest baselines are 0.6, 1.1, and 0.8 points, and several ablation differences are smaller. Without variation estimates, it is difficult to tell whether these margins are stable. The "Average" columns are averages across tasks; they do not measure uncertainty across training runs. Please repeat the key comparisons with at least three independent random seeds and report mean +/- standard deviation for MoEGen and the relevant baselines. For the macro-average, the standard deviation should be computed across the per-run macro-averages, rather than across task scores. This evidence is important to my assessment of the claimed consistency.
-
The cross-domain improvement depends heavily on BillSum, where the baseline failures need a more careful explanation. On Qwen3, the BillSum improvement over MoLA is 17.2 points, contributing 4.3 points to the approximately 4.6-point macro-average gain. On LLaMA-2-13B, the BillSum gain contributes about 4.95 points, nearly the entire reported average improvement. The qualitative examples show baselines producing repeated clauses or fragments from the bill, while MoEGen produces a conventional summary. This may be a real advantage, but the paper does not give enough detail about input truncation, target masking, decoding, or extraction of generated text to assess the comparison confidently. Given how much of the cross-domain claim rests on this result, those settings and the persistence of the gap under a checked evaluation pipeline are consequential. The current evidence supports a strong BillSum result more clearly than a broad improvement on every domain or task.
-
The efficiency evidence needs a clearer accounting of what is saved and what is added. The small hypernetwork is only part of the trainable system: the component-local coordinate embeddings and B matrices also contribute substantially. Deployment additionally requires the frozen 0.6B encoder. Table 2(b) acknowledges the resulting memory increase, and Table 7 shows that inference overhead depends on the task; for MedNLI, latency rises from 0.058 to 0.090 seconds, approximately 55% relative to LoRA. The longer-generation results are more favorable, so I do not see this as a general failure of the efficiency claim. However, adapter checkpoint size, total resident parameters, peak memory, and end-to-end latency need to be distinguished. Appendix B also refers to Table 5 when making cross-domain parameter-count comparisons, although that table contains no parameter counts. A complete accounting is needed to judge the practical trade-off and the claim of inexpensive expert-pool expansion.
Questions for authors
The questions below correspond to weaknesses 1-6. They concern the current claims and implementation, rather than optional directions for future work.
-
Does the implementation compute H(P, sum_i w_i E_i), or sum_i w_i H(P, E_i)? Please provide pseudocode and reconcile the abstract, Figure 2, and Eq. (6). If it uses the latter, what distinction from a compressed, shared-B mixture of fixed experts is intended by the claims in Section 4.2?
-
How do the router parameters receive gradients in the top-1 experiment? Does the balancing loss use a softmax over all experts, the post-selection weights in Eq. (8), or another estimator? Please give the implemented loss and its coefficient, and explain whether the top-2 advantage remains when router trainability is controlled.
-
Can you provide a control using directly learned expert A matrices with the same encoder, router, shared B matrices, and target projections, reporting its parameter budget alongside performance? Please also specify how the "w/o MoE" hypernetwork is conditioned on the input and how its parameter count compares with the full model.
-
How many independent training runs underlie each reported result? Please provide mean +/- standard deviation over at least three seeds for the main comparisons and the ablations used to justify the design, together with the seeds and checkpoint-selection rule. Do the gains over the strongest baselines remain consistent across runs?
-
For BillSum, what input-length limits, truncation rules, target masks, decoding settings, and generated-text extraction procedures were used for each method? After verifying these settings, does the large gap persist? Please also identify the dataset split and task sampling proportions used in joint training, since these affect the interpretation of baseline failures.
-
Can you provide a parameter breakdown for the coordinate embeddings, B matrices, expert codes, router, and hypernetwork, along with adapter checkpoint size and total deployment parameters including the encoder? Please clarify whether the reported timing includes prompt encoding and adapter generation, whether embeddings or candidate matrices are cached, and how storage and runtime change as the expert pool grows.