acceptodds
Under review as a conference paper at ICLR 2027

Native Routing Priors for Budgeted Low-Rank Adaptation of Mixture-of-Experts Models

Abstract

Under a low-rank parameter budget, adapting a mixture-of-experts (MoE) model couples expert coverage, local rank and expert identity. We examine these decisions through Native Router-Guided Layer–Expert Capacity Allocation (NRG), using reference-answer-free profiles and static DoRA adapters. On Qwen3.5-35B-A3B-Base, selected rank-32 adapters exceed approximately budget-matched all-expert adapters by 0.83–0.90 percentage points across three low-data VCR and NLVR2 settings. Grouped intervals support a positive conditional difference on NLVR2, while VCR intervals cross zero. Follow-up controls reveal that expert selection alone does not ensure favorable accuracy: on VCR Train50, selected rank-7 adapters fall 0.90 points below all-expert rank-7 adapters, while increasing rank on the same selected experts yields a 1.77-point gain. The corresponding NLVR2 rank comparison remains uncertain, and neither task establishes an advantage for routing-derived over uniform layer quotas. Historical execution compatibility is only partially verified, limiting attribution in these follow-up comparisons. Repeated random allocations retain a favorable VCR mean difference, but an exclusion-based probe does not identify a ranking mechanism. These observations motivate separate tests of capacity distribution and expert identity when evaluating routing priors. We provide budget accounting and numerical reproduction; the evidence remains conditional on observed models from one backbone and reused development subsets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, parameter-efficient fine-tuning, low-rank adaptation, DoRA, routing priors, capacity allocation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vCfjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies budgeted low-rank adaptation of MoE models. It proposes NRG, which uses native routing statistics to select experts for static DoRA adaptation, and evaluates how expert coverage, rank, and identity affect performance. Experiments on Qwen3.5-35B-A3B-Base with VCR and NLVR2 show that routing-selected rank-32 adapters achieve the best observed means, but follow-up controls suggest that the gains cannot be attributed solely to routing-based expert selection.

Critical strengths
  1. The paper separates expert identity, layer allocation, coverage, and local rank, showing that better performance of TOP32 does not directly imply that routing selects intrinsically better experts.
  2. The authors clearly distinguish observed-model inference from broader generalization and explicitly report limitations, post-hoc analyses, and reproduction constraints.
Critical weaknesses
  1. The core expert-selection strategy is closely related to ESFT and MoE-Sieve. The main contribution is primarily the experimental analysis rather than a substantially new adaptation method.
  2. Evaluation is restricted to one MoE backbone, two datasets, and three main training seeds. Moreover, several controls remain inconclusive, so the generality of the findings is unclear.
Questions for authors

Q1. Can the authors provide a direct comparison with ESFT or MoE-Sieve under the same backbone, data, and parameter budget? Q2. Can the authors better isolate whether the TOP32 gain comes from routing-based expert selection or simply from concentrating more rank on fewer experts? Q3. For the historical-versus-new follow-up runs, which execution differences remain unverified, and could they plausibly affect effects of the reported magnitude?

Reviewer_eZgHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate a limited DoRA parameter budget across MoE experts. NRG profiles the pretrained router without reference answers and fixes the experts receiving adapters, while retaining all pretrained experts for routing. Experiments on Qwen3.5-35B-A3B-Base cover VCR Train50, VCR Train100, and NLVR2 Train100. Selected rank-32 adapters are compared with random and low-routing-mass subsets at matching per-layer counts and ranks, and approximately budget-matched all-expert adapters at ranks 7 or 8. Additional allocation, quota, and rank controls help distinguish evidence about capacity distribution from evidence about expert identity.

Critical strengths
  1. The paper addresses a practical allocation problem involving expert coverage, local rank, and expert identity. Its distinction between all-expert and random-subset comparisons is useful: the former changes coverage and rank together, while the latter examines expert identity at fixed per-layer counts and rank. Although the selection principles build on ESFT and related work, their examination with DoRA provides informative empirical evidence. The VCR follow-up illustrates how the observed ordering can change across rank/coverage configurations, subject to the execution caveat below. This perspective offers useful guidance for evaluating routing-based adaptation, using a simple procedure with no reference-answer or gradient-based calibration.

  2. TOP has the highest mean accuracy across the three original settings, exceeding approximately budget-matched all-expert adaptation by 0.83-0.90 percentage points (Table 3). Grouped intervals support a positive conditional difference on NLVR2, while both VCR intervals include zero. On VCR Train50, TOP also exceeds each of five random-allocation means, with an aggregate advantage of 0.993 points and a positive conditional movie-group interval (Section 5.3). Three training seeds, repeated expert lists, explicit parameter accounting, and grouped evaluation strengthen the analysis. The paper is generally clear and distinguishes observed improvements from statistical support. Although the abstract is dense, the authors make inconclusive findings and limitations traceable.

Critical weaknesses
  1. Evaluation covers one backbone, two tasks, and small training sets on reused development subsets. The grouped intervals condition on observed models and expert lists, rather than estimating variation across new training seeds or allocations. Original TOP-RANDOM intervals include zero, and neither task establishes an advantage for routing-derived over uniform layer quotas. These limitations constrain a general ranking or allocation claim, although the authors largely acknowledge them. The practical takeaway should remain tied to the tested configurations.

  2. Appendix H.3 leaves historical/new DoRA initialization and scaling, optimizer rebinding, and parts of base/runtime compatibility incompletely verified. These unknowns reduce confidence in attributing rank or quota differences to allocation choices. The rank comparison also changes the parameter budget, so it supports a higher-capacity configuration rather than an isolated fixed-budget rank effect. A concise explanation of which unknowns could materially affect the results would strengthen the paper. The original four-arm observations and provisional follow-up evidence should be assessed separately.

Questions for authors
  1. Which unresolved issues in Appendix H.3 could materially affect the rank or quota comparisons? Can retained records clarify DoRA and optimizer behavior, and which conclusions remain supported by the original cohort alone?

  2. Appendix E mentions a historical unadapted-base array. If execution-compatible results are available, could the authors report base accuracy on the same evaluation subsets to contextualize the practical benefit of adaptation? If comparability is uncertain, please clarify that limitation.

Reviewer_8HqVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies static placement of DoRA adapters in a pretrained MoE model using reference-answer-free native routing profiles. A cumulative routing-mass threshold determines per-layer expert counts; TOP, RANDOM, and BOTTOM change expert identities at the same counts and rank. An all-expert baseline uses a lower rank to approximately match trainable parameters. Experiments on one Qwen3.5 backbone and small VCR/NLVR2 training subsets report TOP32-minus-Dense mean gains of 0.833-0.900 percentage points, with positive grouped conditional interval support on NLVR2. Follow-up rank, layer-quota, repeated-allocation, and exclusion controls qualify the interpretation: the same-low-rank VCR comparison favors Dense, quota effects remain unresolved, and the exclusion probe does not establish a ranking mechanism. The contribution is an empirical examination of allocation comparisons rather than a new routing or adaptation module.

Critical strengths

Secs. 3-4 and Table 2 distinguish budget-matched joint coverage/rank changes from identity comparisons with fixed counts/ranks, and distinguish uniform-random membership from uniform-layer quotas. The DoRA magnitude terms and integer-rank mismatches are accounted for explicitly. I independently recovered Table 1's parameter counts and the -1.286%/+5.481% mismatches. This makes the tested allocation choices interpretable.

The supplied scorer rebuilt the original correctness arrays exactly; all 108 rows of current original-cohort interval results matched the submitted reference to floating-point precision. Follow-up means, intervals, and included sensitivity checks also reproduced. The paper preserves paired evaluation groups, avoids counting reused TOP predictions as extra models, distinguishes null uncertainty from equivalence, and discloses historical analysis deviations. These are meaningful strengths for an empirical analysis.

Critical weaknesses

Sec. 5.2 and Appendix H (pp. 6 and 20-21, especially lines 1102-1116) compare historical TOP32/Dense predictions with newly trained low-rank and uniform-quota arms. Appendix H.3 identifies unresolved historical/new DoRA initialization/scaling, optimizer rebinding, distributed configuration, and sample-exposure equivalence; some NLVR2 decoding fields are also missing. These are potentially outcome-relevant components of an adapter experiment. Consequently, the +1.767 pp VCR TOP32-minus-TOP7 difference and the -0.900 pp TOP7-minus-Dense7 difference cannot yet provide a well-controlled experimental demonstration of the rank/coverage interpretation. The manuscript acknowledges this, and the record does not establish that any mismatch actually occurred. Nevertheless, disclosure leaves the scientific attribution unresolved. Required modification: reproduce the decisive VCR TOP32/TOP7/Dense7 contrast under one documented implementation, with common data, processor, update schedule, and decoding, and verified adapter initialization and optimizer membership. A practical shared-implementation replication is sufficient; exhaustive historical reconstruction is unnecessary. Supply the runnable adaptation path, since the released entry points currently reproduce scoring/statistics rather than training.

Sec. 2, Appendix A, and lines 616-620 explicitly identify the core routing-selection ideas as inherited. An empirical contribution can still be valuable, but here the primary evidence consists of two small low-data tasks on one backbone, with the VCR sizes sharing evaluation inputs and the analysis repeatedly reusing frozen development subsets. Original TOP-minus-RANDOM grouped intervals all include zero; neither task resolves a quota advantage, and the exclusion probe establishes no mechanism. These null/uncertain findings are reported correctly, but they leave much of the contribution as a comparison-design lesson and a small set of conditional observations. The paper has not yet established a sufficiently reliable positive or negative finding that would change an adaptation decision beyond those particular runs. Required modification: add one focused, prospectively fixed validation using fresh training/calibration draws and an untouched evaluation subset, with the decisive budget and identity controls under a shared implementation. State the practical question and target effect before examining the new outputs. A confirmed counterexample to a routing heuristic would be valuable; success need not mean outperforming every method or running a broad benchmark suite.

Questions for authors
  1. Which Appendix H.3 items are known implementation differences and which are simply unrecorded? Can a shared-implementation TOP32/TOP7/Dense7 replication confirm the VCR ordering? This is the result most likely to change my assessment of the empirical insight.

  2. How were the routing threshold, rank, training/calibration draws, development subsets, and follow-up contrasts selected relative to previously observed scores? Is a frozen, untouched evaluation possible for a focused replication? The current post-hoc conditional intervals are useful, but do not assess selection effects or performance over new training outcomes.

  3. Which allocation decision should change on the basis of the current evidence: increasing local rank, using routing-derived quotas, or selecting identities by routing mass? Please identify a concrete conclusion supported beyond the fixed observed configurations, or frame the contribution around a cleanly replicated counterexample with its domain of applicability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.