acceptodds
Withdrawn from ICLR 2027

MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient fine-tuning (PEFT) enables efficient adaptation of large language models, but existing MoE-based PEFT methods typically improve capacity by storing multiple full LoRA experts, causing adapter storage to grow linearly with the number of experts and restricting adaptation to a fixed expert pool. We ask whether MoE-based PEFT can produce instance-specific adaptations without explicitly storing a separate LoRA module for each expert. To address this gap, we propose MoEGen, an adaptation framework that shifts MoE-based PEFT from expert selection to expert-conditioned parameter generation. Instead of storing each expert as a full LoRA adapter, MoEGen represents each expert as a small learnable vector, termed an expert code. It routes each input over these vectors and uses their weighted combination to condition a lightweight hypernetwork that generates input-specific low-rank updates. This design decouples expert capacity from adapter storage while enabling instance-conditioned adaptation. Experiments on eight commonsense reasoning benchmarks show consistent improvements over strong static and MoE-based PEFT baselines across three backbones. MoEGen also performs strongly in joint medical and legal-domain adaptation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Mixture of Experts, Efficient Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2P44OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MoEGen aims to reduce the storage cost of mixture-of-LoRA adaptation while retaining input-dependent specialization. A frozen sentence encoder produces a prompt representation, and a component-specific router selects compact expert codes. A shared hypernetwork uses these codes and learned coordinate embeddings to produce LoRA A matrices, which are paired with trainable B matrices shared across inputs. Under Eq. (6), candidate A matrices are generated separately and then combined using the routing weights. The method is evaluated on eight commonsense reasoning benchmarks and a joint training mixture of medical and legal tasks. It achieves the highest average score on all three backbones in each setting, with gains of 0.6-1.1 percentage points over the strongest commonsense baselines and approximately 4.6-6.2 points on the joint benchmark. The paper also includes routing and encoder ablations, qualitative examples, and inference measurements.

Critical strengths
  1. The compact expert representation addresses a practical limitation of MoE-based adaptation. Adding an expert mainly adds a small code and its gating parameters, rather than another complete set of adapters. The results in Table 1 are encouraging: MoEGen has fewer trainable parameters than both MoE baselines while achieving a higher average accuracy. Even if the method is interpreted primarily as a compressed parameterization of an expert pool, this is a useful contribution.

  2. The positive results extend across several backbones and two different adaptation settings. The joint benchmark includes both classification and generation, which gives a broader test than commonsense classification alone. The ablation in Table 2(a), where removing MoE lowers the average from 61.5 to 59.8, provides some evidence that routing contributes beyond the hypernetwork alone. I also appreciate the inclusion of inference measurements in Appendix C, since a reduction in trainable parameters does not automatically imply faster inference.

Critical weaknesses
  1. The generation mechanism is ambiguous, and the claimed distinction from mixing fixed experts is not established. The abstract and Figure 2 diagram suggest that expert codes are mixed before entering the hypernetwork. In contrast, Eq. (6) and the figure caption generate one candidate matrix per expert and then mix the matrices. These operations are generally different because the hypernetwork includes LayerNorm and GELU. Under Eq. (6), each candidate A_i^(l) depends only on learned coordinates, an expert code, and the hypernetwork parameters. Once training is complete, it is independent of the current prompt and can be precomputed. The update can therefore be written as Delta W^(l)(x) = sum_i w_i^(l)(x) B^(l)(A_i^(l))^T, which is functionally a mixture of fixed experts sharing B. The final update is still input-dependent, and the compressed representation remains useful, but the stronger distinction drawn in Section 4.2 is not justified by this formulation. This matters to both the novelty claim and the interpretation of the results.

  2. The stated routing equations leave an important problem in the top-1 ablation. Equation (5) normalizes probabilities only over the selected experts. For top-1 routing, the selected weight is identically one, so its derivative with respect to the logits is zero. With ordinary hard top-k differentiation, the selection itself does not provide a gradient either. Equation (8) uses the same post-selection weights, so the stated balancing loss does not resolve this issue. Unless the implementation uses an additional mechanism, the top-1 router cannot learn through these paths. The comparison in Appendix A.2 would then change router trainability as well as the number of active experts. I cannot interpret the reported advantage of top-2 as evidence for the benefit of expert mixing without clarification of the actual gradient computation.

  3. The comparisons do not isolate the contribution of the hypernetwork parameterization. Appendix A.1 uses different adaptation locations, ranks, expert counts, and routing schemes across methods. MoEGen additionally receives representations from a frozen 0.6B encoder. These are reasonable configurations for comparing complete systems, but they do not establish which part of MoEGen produces the gain. A particularly relevant control would use the same prompt encoder, component-level router, adaptation locations, and shared B matrices, while learning the expert A matrices directly. The "w/o MoE" variant is useful but does not specify its conditioning path or parameter budget clearly enough to fill this gap. Lower trainable-parameter counts alone do not establish that parameter generation, rather than these other design choices, explains the improvement.

  4. The paper does not report repeated experiments or variation across random seeds. There are no reported run counts, means over independent training runs, or standard deviations for the main results and ablations. The commonsense improvements over the strongest baselines are 0.6, 1.1, and 0.8 points, and several ablation differences are smaller. Without variation estimates, it is difficult to tell whether these margins are stable. The "Average" columns are averages across tasks; they do not measure uncertainty across training runs. Please repeat the key comparisons with at least three independent random seeds and report mean +/- standard deviation for MoEGen and the relevant baselines. For the macro-average, the standard deviation should be computed across the per-run macro-averages, rather than across task scores. This evidence is important to my assessment of the claimed consistency.

  5. The cross-domain improvement depends heavily on BillSum, where the baseline failures need a more careful explanation. On Qwen3, the BillSum improvement over MoLA is 17.2 points, contributing 4.3 points to the approximately 4.6-point macro-average gain. On LLaMA-2-13B, the BillSum gain contributes about 4.95 points, nearly the entire reported average improvement. The qualitative examples show baselines producing repeated clauses or fragments from the bill, while MoEGen produces a conventional summary. This may be a real advantage, but the paper does not give enough detail about input truncation, target masking, decoding, or extraction of generated text to assess the comparison confidently. Given how much of the cross-domain claim rests on this result, those settings and the persistence of the gap under a checked evaluation pipeline are consequential. The current evidence supports a strong BillSum result more clearly than a broad improvement on every domain or task.

  6. The efficiency evidence needs a clearer accounting of what is saved and what is added. The small hypernetwork is only part of the trainable system: the component-local coordinate embeddings and B matrices also contribute substantially. Deployment additionally requires the frozen 0.6B encoder. Table 2(b) acknowledges the resulting memory increase, and Table 7 shows that inference overhead depends on the task; for MedNLI, latency rises from 0.058 to 0.090 seconds, approximately 55% relative to LoRA. The longer-generation results are more favorable, so I do not see this as a general failure of the efficiency claim. However, adapter checkpoint size, total resident parameters, peak memory, and end-to-end latency need to be distinguished. Appendix B also refers to Table 5 when making cross-domain parameter-count comparisons, although that table contains no parameter counts. A complete accounting is needed to judge the practical trade-off and the claim of inexpensive expert-pool expansion.

Questions for authors

The questions below correspond to weaknesses 1-6. They concern the current claims and implementation, rather than optional directions for future work.

  1. Does the implementation compute H(P, sum_i w_i E_i), or sum_i w_i H(P, E_i)? Please provide pseudocode and reconcile the abstract, Figure 2, and Eq. (6). If it uses the latter, what distinction from a compressed, shared-B mixture of fixed experts is intended by the claims in Section 4.2?

  2. How do the router parameters receive gradients in the top-1 experiment? Does the balancing loss use a softmax over all experts, the post-selection weights in Eq. (8), or another estimator? Please give the implemented loss and its coefficient, and explain whether the top-2 advantage remains when router trainability is controlled.

  3. Can you provide a control using directly learned expert A matrices with the same encoder, router, shared B matrices, and target projections, reporting its parameter budget alongside performance? Please also specify how the "w/o MoE" hypernetwork is conditioned on the input and how its parameter count compares with the full model.

  4. How many independent training runs underlie each reported result? Please provide mean +/- standard deviation over at least three seeds for the main comparisons and the ablations used to justify the design, together with the seeds and checkpoint-selection rule. Do the gains over the strongest baselines remain consistent across runs?

  5. For BillSum, what input-length limits, truncation rules, target masks, decoding settings, and generated-text extraction procedures were used for each method? After verifying these settings, does the large gap persist? Please also identify the dataset split and task sampling proportions used in joint training, since these affect the interpretation of baseline failures.

  6. Can you provide a parameter breakdown for the coordinate embeddings, B matrices, expert codes, router, and hypernetwork, along with adapter checkpoint size and total deployment parameters including the encoder? Please clarify whether the reported timing includes prompt encoding and adapter generation, whether embeddings or candidate matrices are cached, and how storage and runtime change as the expert pool grows.

Reviewer_XpoaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether compact hypernetwork-induced LoRA experts can achieve better parameter efficiency and performance than explicitly parameterized LoRA experts. Unlike conventional token-level MoE routing, the proposed MoEGen performs prompt-level expert selection. Instead of storing separate LoRA matrices for each expert, MoEGen uses a shared hypernetwork to generate candidate LoRA AA matrices while maintaining a static BB matrix. This design aims to improve parameter efficiency while preserving the performance. The experiments on several commonly used benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths

The strengths are summarized below:

  1. The empirical studies are extensive, and the performance compared to the baselines are strong.
Critical weaknesses

The weaknesses are summarized below:

  1. The motivation is unclear. The proposed framework is interesting, but it involves several modules, including prompt-level expert routing, component embedding, coordinate embedding, expert codes, etc. The motivation behind these individual design choice is not sufficiently explained. For example,

    • why are learnable coordinate embedding necessary?
    • why is only AA matrix generated while BB is static?
    • Why is expert code routing necessary? The model could directly use the prompt embedding z(x)z(x) to generate LoRA AA matrices through a hypernetwork, such as MoEGen A(x)=∑iwi(x)H(P,Ei)A(x)=\sum_i w_i(x) H(P, E_i) vs. alternative A(x)=H~(P,z(x))A(x)=\tilde H(P, z(x)), where H~\tilde H is another hypernetwork.
  2. According to Eq. (6), the candidate LoRA matrices are generated as Ai(l)=Hϕ(P(l),Ei)A_i^{(l)}=H_\phi(P^{(l)},E_i). Both P(l)P^{(l)} and EiE_i are learned and input-independent. Therefore, after training, all candidate matrices Ai(l)A_i^{(l)} are fixed. The prompt only determines their mixing weights A(l)(x)=∑iwi(l)(x)Ai(l)A^{(l)}(x)=\sum_i w_i^{(l)}(x)A_i^{(l)}. Therefore, the effective LoRA update can be expressed as ΔW(l)(x)=∑iwi(l)(x)B(l)(Ai(l))⊤\Delta W^{(l)}(x)=\sum_i w_i^{(l)}(x)B^{(l)}(A_i^{(l)})^\top. This is equivalent or similar to HydraLoRA [1] and ALoRA [2], which mix a fixed set of LoRA experts with a shared AA or BB matrix. This touches the core question: why use a hypernet to generata these matrices instead directly learning them? Parameter efficiency is one possible reason, but the performance comparison against them is missing.

  3. The experiment comparisons are not fully controlled. MoEGen and the baselines differ in LoRA rank, parameter budget, etc. The reported results should inlcude the standard deviation.

  4. The ablation studies are insufficient. Several important comparisons are missing: Learned vs fixed coordinate embedding, direct prompt-conditioned hypernetwork generation.

[1] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning. https://arxiv.org/abs/2404.19245.

[2] Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs. https://arxiv.org/abs/2509.25414.

Questions for authors

The questions are mainly included in the above weaknesses. Please refer to that.

Reviewer_xDBCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets storage cost in MoE-style PEFT: instead of storing one LoRA adapter per expert (MoELoRA/MoLA), MoEGen stores small learned "expert codes" and a shared hypernetwork that generates each expert's LoRA-A matrix on demand; routing is top-2 per adapted component, B is a shared trainable matrix. Experiments on LLaMA-2/3 and Qwen3-8B cover 170K commonsense joint training (+0.6–2.4 points over the best baseline per backbone) and a four-task medical/legal cross-domain setting (+~6 points average over MoLA, with very large BillSum margins).

Critical strengths
  1. A real, well-motivated problem: MoE-PEFT storage scales with N × adapter size and the expert pool is frozen after training; replacing stored experts with conditioned generation is a reasonable direction, correctly placed against HyperFormer/HyperLoRA/Text-to-LoRA in related work.
  2. Broad empirical sweep for a PEFT paper: three backbones, twelve datasets, two regimes, plus routing/pool/top-k ablations — and the cross-domain result is large, consistent, and plausibly a real effect of prompt-level versus token-level routing.
  3. Honest parameter accounting and measured costs — the trainable-share figures check out (the LLaMA-2/3 gap is GQA geometry, not an accounting trick), MoEGen wins while touching only Q/K/V/O, and latency/memory are at least measured, including a reported negative result on the always-on shared expert.
Critical weaknesses
  1. The central "capacity decoupled from storage" claim does not follow from Eq. (6). The generated matrix Ai(l)=Hϕ([pj(l);Ei])A^{(l)}_i = H_\phi([p^{(l)}_j; E_i]) depends only on row and expert indices — not on the prompt embedding z — so the input enters only through the routing weights. MoEGen is mathematically a bank of NN prompt-independent LoRA-A matrices with a k=2 soft mixture: the induced bank is the same order as stored MoE-LoRA experts (minus shared B), so "decoupling" is a reparameterization, not a reduction. Compounding this, no table anywhere reports adapter storage or inference-time memory — the one quantity the paper claims to improve.
  2. The evidence is about task-level specialization, not instance-level adaptation. Routing happens once per prompt and the t-SNE case study shows dataset-level clusters; nothing in the paper requires instance-level conditioning to explain the numbers. The missing control — an oracle task-routed LoRA (one LoRA per task, routed by gold label) — would immediately settle whether the contribution is task identification or instance adaptation.
  3. The hypernetwork is never compared against the obvious parameter-matched alternative: N freely trainable A matrices of the same induced size under identical routing and B-sharing. If that matches MoEGen (likely, given W1), codes + hypernetwork contribute nothing beyond a low-dimensional regularizer — a much smaller claim than the abstract's.
  4. The BillSum gap suggests baseline optimization failure rather than a method advantage. MoLA, with >2× MoEGen's trainable parameters, scores below plain LoRA (22.8 vs 30.3 R-L vs MoEGen's 43.7); baselines collapsing into copy-paste loops points to undertrained/misconfigured baselines. No per-task convergence evidence is given.
Questions for authors
  1. Q1 the generator is input-independent and the "decoupling" claim fails. In Eq. (6), aj,i(l)=Hϕ([pj(l);Ei])a^{(l)}_{j,i} = H_\phi([p^{(l)}_j; E_i]) contains no term in zz. Please confirm whether the candidate matrices Ai(l)A^{(l)}_i depend on the input at all. If they do not, state explicitly that MoEGen's input-dependence is the routing weight vector alone, and address head-on whether the resulting model is simply an MoE over NN shared-BB LoRA-AA experts with prompt-level routing; if they do, give the exact modified equation. This is the premise on which the entire paper rests.
  2. Q2 there is no storage number anywhere. The paper's central claim is a storage claim, yet no table reports adapter storage (MB/GB) or inference-time memory for the expert bank. Please report the actual storage of the deployed artifact — total bytes of {Ei}\{E_i\}, {P(l)}\{P^{(l)}\}, {B(l)}\{B^{(l)}\}, ϕ\phi, and the induced expert bank — for MoEGen and for MoELoRA/MoLA at matched NN.
  3. Q3 he evidence is task-level, not instance-level. Please add an oracle task-routing baseline (one LoRA per task, routed by the gold task label) on the four-task cross-domain benchmark, plus a within-task experiment on heterogeneous samples where routing cannot key on task identity. If MoEGen ≈ oracle task routing, the "instance adaptive" framing in the title must change; if MoEGen exceeds it, that is the strongest possible evidence for the paper's thesis.
  4. Q4 he hypernetwork's necessity is never tested. Please add the parameter-matched control: replace HϕH_\phi + codes with NN freely trainable Ai(l)A^{(l)}_i matrices of the same induced size, under identical routing, identical BB, and identical training. If that matches MoEGen (likely, given Q1), then codes + hypernetwork contribute nothing beyond a low-dimensional regularizer on the expert bank — a real but much smaller claim. This is currently the single most informative missing experiment.
  5. Q5 the BillSum gap looks like baseline optimization failure. MoLA has more than twice MoEGen's trainable parameters yet scores below plain LoRA on BillSum (22.8 vs 30.3 R-L, vs MoEGen's 43.7). Please report per-task training/validation loss curves for every method and a strong single-task LoRA upper bound; if the baselines fail to converge on summarization, this should be stated as an optimization artifact rather than a method advantage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.