acceptodds
Under review as a conference paper at ICLR 2027

Beyond Load Balancing: Expert Assignment Matters in Genomic Mixture-of-Experts

Abstract

Genomic mixture-of-experts (MoE) models inherit load-balancing objectives from language modelling. We ask what these objectives control. In a released JanusDNA checkpoint, the shipped objective is nearly satisfied only after routers are pooled across layers (pooled Gini 0.0068; individual layers 0.164–0.407). Adding a per-layer term flattens every layer, and retargeting it to a genome-composition prior moves expert utilisation up to a 5.06–5.98× within-layer max/min ratio, while validation loss across fourteen non-divergent fine-tuned checkpoints spans only 0.06%, and generation coverage and three linear probes show no attributable gain. By contrast, permuting expert identity at inference, which leaves the utilisation histogram unchanged up to relabelling, raises validation loss by 7.48% on the same 128 evaluation windows. In a separate decomposition, recomputing gate weights retains 89% of the permutation cost, and swapping a single expert pair costs about as much per rerouted slot as a full permutation. Permutation is more damaging than the most consequential single-expert ablation in two further genomic MoEs spanning a 34× range in scale. The objective is not inert: arms with a per-layer prior-targeted term show about 39% lower layer-7 permutation sensitivity, a separation that holds across three seeds of two arms, although we do not establish whether this reflects harm or redundancy. We therefore recommend monitoring permutation sensitivity, class enrichment against a matched null, and expert-output effective rank alongside utilisation, rather than replacing balancing with an unvalidated objective.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
mixture-of-experts, load balancing, expert routing, genomic language models, DNA foundation models, conditional computation, interventional analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gFAeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the distinction between expert utilization and expert assignment in genomic mixture-of-experts models. Using JanusDNA as the primary case study, the authors show that the released pooled load-balancing objective can hide substantial layer-wise imbalance. They introduce per-layer and composition-targeted balancing objectives that substantially change expert utilization while producing only small changes in reconstruction-style validation loss. The authors then perform inference-time interventions that permute expert identities, recompute gate weights, randomly route tokens, swap expert pairs, or remove individual experts. Expert permutation substantially increases validation loss while preserving the utilization histogram up to relabeling, suggesting that token–expert correspondence matters more than aggregate utilization. Similar permutation-versus-ablation ordering is reported for PlantBiMoE and Genos-1.2B. The paper proposes permutation sensitivity, class enrichment, and effective output rank as additional routing diagnostics.

Critical strengths

The paper makes a clear conceptual distinction between expert utilization and token–expert assignment. The permutation intervention is well targeted because it preserves the utilization histogram while changing the assignment structure.

The decomposition into full permutation, gate recomputation, random routing, pair swaps, and single-expert ablations helps establish that the observed sensitivity is not simply caused by one indispensable expert or by a gate-weight mismatch.

Critical weaknesses

The central training-objective conclusions are not supported by sufficient replication or attribution. Most fine-tuning arms use one primary seed, and the authors acknowledge that the small validation-loss differences among checkpoints are separable but not attributable to the proposed interventions. The three-seed result applies to layer-7 permutation sensitivity, not to the downstream or validation-loss effects of the training objectives.

The main evaluation metric limits the interpretation of the results. JanusDNA is a bidirectional denoising model, and the reported validation loss is an unmasked reconstruction-style score in which neighboring states and the terminal attention block may retain information about the original token. Therefore, the large loss increase under expert permutation does not directly establish a comparable degradation in autoregressive prediction or biological utility.

The proposed structural diagnostics are not validated as useful proxies for genomic modeling quality. The fine-tuned arms show no clear attributable improvement on the reported linear probes, generation coverage, or ClinVar endpoint. A reduction in permutation sensitivity may reflect reduced specialization or increased redundancy, and the paper does not establish which interpretation is correct.

Several evaluations use small or non-independent samples. Some routing and generation statistics rely on four or eight windows or a small number of generated sequences, bootstrap intervals are nominal and uncorrected, and the PlantBiMoE windows are not verified to be excluded from pretraining data. These limitations make the cross-model generalization claims difficult to assess.

Questions for authors

Can the authors replicate the main fine-tuning-objective comparisons across multiple independent seeds and report uncertainty for validation loss and every downstream metric? In particular, which conclusions remain after separating seed variation from intervention effects?

Can the authors evaluate the routing interventions with a genuinely held-out predictive objective, such as masked-token or autoregressive likelihood where applicable, rather than relying primarily on unmasked reconstruction loss?

What evidence shows that the proposed diagnostics predict biological utility? For example, can permutation sensitivity, class enrichment, and effective rank be evaluated against independent genomic prediction tasks across checkpoints or models?

Can the authors provide a larger and independently sampled evaluation for the generation and structural statistics, with corrected treatment of multiple comparisons and clearly disjoint windows?

The paper argues that load balancing should be monitored together with assignment-sensitive statistics, but does not establish when practitioners should change or anneal the balancing objective. Can the authors provide a controlled experiment that tests whether any proposed intervention improves training or downstream performance, rather than only changing utilization or sensitivity?

1 comment
AuthorsOpenReview ↗

Response to Reviewer gFAe

Thank you for your detailed review. We have strengthened the revision with additional controls, expanded evaluations, and clearer limitations. We address your five questions below.

W1/Q1: Independent seeds and attribution

We agree that most fine-tuning arms have only one primary seed and that window-level bootstrap intervals cannot substitute for training-seed replication.

The central fixed-weight diagnostic observation is that expert permutation increases validation loss by 7.48%, whereas the 14 fine-tuned arms span only 0.06% on the same 128 windows. These are different intervention protocols, and we now explicitly avoid interpreting their effect sizes as matched causal estimates.

For fine-tuning comparisons, the empirical paired-bootstrap 95% CI half-widths are 0.00008–0.00021 nats. A control retrained with only a different seed differs by 0.00031 nats, comparable to the objective-related differences. We therefore do not attribute the small loss differences to particular objectives.

The Layer-7 structural finding has stronger replication: across three seeds, control sensitivities are 2.35/2.21/2.21%, versus 1.37/1.37/1.37% for the prior arm. The revised paper distinguishes this replicated structural result from single-seed validation-loss and downstream observations.

W2/Q2: Target leakage and predictive objectives

We directly tested whether the predicted token is visible to the model (Appendix E).

At position t, the prediction combines forward information through t−1 and reverse information from t+1 onward. The terminal attention masks preserve this exclusion.

As an intervention check, replacing or masking x_t leaves its predicted log-probability unchanged, whereas predictions at adjacent positions change substantially. This supports interpreting the reported score as pseudo-log-likelihood rather than target-leaking reconstruction.

We agree that pseudo-log-likelihood is not directly comparable to autoregressive likelihood.

We additionally evaluated expert permutation under PlantBiMoE's masked-language-model objective and Genos-1.2B's autoregressive objective. Both show substantial permutation penalties relative to single-expert ablation. These experiments support cross-objective consistency, although they do not establish downstream biological generalisation.

W3/Q3: Biological utility of the diagnostics

We agree that the proposed diagnostics are not validated predictors of downstream performance across independently trained checkpoints.

The non-coding ClinVar experiments provide narrower, intervention-level evidence: assignment permutation reduces AUROC, while utilisation-changing fine-tuning produces much smaller differences.

We explicitly report the original full ClinVar evaluation as null (AUROC 0.522), the subsequent non-coding subset as exploratory, and the independently pre-registered 2,000-variant evaluation as partial replication.

Refitted probes on permuted features do not reliably separate from the control.

We therefore present permutation sensitivity, class specificity, and effective rank as complementary structural diagnostics, not established proxies for genomic prediction quality.

W4/Q4: Evaluation size, multiplicity, and independence

We expanded the permutation decomposition to 128 validation windows (Appendix E). Gate-weight recomputation retains approximately 89–90% of the full permutation penalty, and the Layer-7 separation remains visible.

We also report chromosome-stratified and sequence-length sensitivity analyses.

The 13 training-arm comparisons and four selected between-arm contrasts are evaluated with Holm correction; their reported significance conclusions remain unchanged.

We acknowledge that the PlantBiMoE evaluation uses TAIR10 chromosome 1, which was included in its pretraining corpus. These are in-domain comparisons, not held-out generalisation estimates. Pretraining overlap remains a limitation even for paired intervention comparisons.

Q5: When should balancing objectives change?

We agree that the current experiments do not establish an intervention that improves downstream performance.

We evaluated several assignment-oriented objectives, including SimBal, but found no consistent validation-loss or downstream improvement.

Our recommendation is therefore diagnostic rather than prescriptive: first assess expert collapse using utilisation and ablation statistics, then monitor assignment sensitivity and expert diversity alongside balancing metrics.

The observed reduction in Layer-7 sensitivity may reflect increased expert redundancy rather than improved routing quality. We do not recommend replacing or annealing a balancing objective solely on this basis.

We appreciate these questions. They helped us distinguish what the experiments directly establish from what remains open for future validation.

Reviewer_RKrnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what load-balancing objectives actually control in genomic MoE models. It separates expert utilisation from token–expert assignment. The authors show that cross-layer pooling hides per-layer imbalance, and that per-layer balancing and a genome-composition prior can change expert utilisation without meaningfully changing validation loss or improving downstream performance. In contrast, inference-time expert permutation keeps the utilisation distribution unchanged but significantly increases validation loss. Further analysis shows that this cost mainly comes from breaking token–expert assignment rather than gate-weight mismatch. The same ordering is reproduced on multiple models, and per-layer prior training reduces permutation sensitivity. The paper is a diagnostic study and suggests monitoring permutation sensitivity, class enrichment, and effective rank beyond utilisation. However, it does not yet link these structural metrics to downstream performance, and validation is limited to fine-tuning settings with limited checkpoints and seeds, without pretraining or from-scratch training.

Critical strengths
  1. Clear separation between utilisation and assignment, with evidence that assignment matters more. The paper shows that cross-layer pooling can hide per-layer imbalance, and that per-layer balancing or prior-targeted objectives can change expert utilisation without meaningfully changing validation loss. In contrast, inference-time expert permutation keeps the utilisation distribution unchanged but significantly increases validation loss. Further decomposition shows the cost mainly comes from breaking token-expert correspondence, not gate-weight mismatch. This challenges the default assumption that balancing utilisation is equivalent to controlling MoE behaviour, and offers a more direct diagnostic angle.

  2. Transparent experiments, cross-model replication, and actionable monitoring metrics. The authors use pre-registered rules, report negative and failed results, and replicate the ordering that permutation hurts more than single-expert ablation across different MoE models. They also propose monitoring permutation sensitivity, class enrichment, and effective rank beyond utilisation. The credibility of a diagnostic paper depends on reliable measurement and reporting, and these choices make the main conclusions more trustworthy and provide reusable tools for future MoE analysis.

Critical weaknesses
  1. Confounded attribution of the structural effects. A4nb introduces both per-layer aggregation and a composition-based target, but there is no per-layer uniform-target control arm. As a result, the observed reduction in layer-7 permutation sensitivity cannot be clearly attributed to per-layer balancing, the composition prior, or their interaction, which limits the mechanistic interpretation of one of the paper's main structural findings.

  2. Limited evidence for the practical value of the proposed diagnostics. The structural metrics are not shown to predict downstream performance or robustness. The downstream evaluations show no attributable differences between training arms, and the ClinVar evaluation is uninformative. Given that the paper recommends monitoring these metrics in practice, the absence of validation against meaningful endpoints leaves the main diagnostic contribution insufficiently supported.

Questions for authors
  1. Can you train and report a per-layer uniform-target control arm under the same fine-tuning setup, including per-layer Gini, validation loss, and layer-7 permutation sensitivity? This would help separate the effects of per-layer aggregation and the composition prior.

  2. Does the reduction in layer-7 permutation sensitivity reflect increased expert redundancy or reduced sensitivity to routing errors? What existing evidence can distinguish these two explanations?

  3. What is the minimum detectable effect of the current downstream evaluations? Is there evidence that permutation sensitivity, class enrichment, or effective rank predicts downstream performance or robustness, rather than only describing routing structure?

2 comments
AuthorsOpenReview ↗

Response to Reviewer RKrn

Thank you for your constructive review. Your questions motivated an additional control arm, three-seed structural analyses, and a clearer statement of the diagnostic scope.

W1 / Q1: Per-layer Uniform control

We agree that the original Prior + Uniform comparison did not isolate the effects of per-layer balancing and the genome-composition prior. We therefore trained the requested Per-layer Uniform control using the same fine-tuning budget.

The new control replaces the composition target with a uniform target. Coefficients were calibrated to produce comparable router-gradient contributions (§5), rather than simply matched numerically.

ConditionLayer-7 GiniValidation loss vs. controlLayer-7 permutation sensitivity
Control (pooled)0.339—2.35%
Per-layer Uniform0.030+0.02%1.47%
Prior Only0.160+0.04%1.37%

All values use the same four evaluation windows.

The key finding is that a genome-composition prior is not required for the reduction in Layer-7 permutation sensitivity. Per-layer Uniform reduces sensitivity from 2.35% to 1.47%, approaching the 1.37% observed with Prior Only.

This supports per-layer balancing as the principal explanation, although coefficient calibration and protocol differences limit a strictly isolated causal attribution.

As an additional control, removing balancing entirely produces 2.23% sensitivity, close to the pooled Control. Thus, the reduction is not simply explained by removing the original balancing objective.

The revised §7 and Appendix E include these comparisons.

Q2: Redundancy versus reduced routing-error sensitivity

We added three-seed analyses to distinguish increased expert redundancy from a change confined to router sensitivity.

At Layer 7, the per-layer conditions exhibit:

  • 31% lower worst single-expert ablation cost.
  • Higher expert-output cosine similarity: 0.067 to 0.100.
  • Lower effective rank: 14.2 to 12.4.

All three observations support increased functional redundancy among experts.

We have revised §7 to clarify that lower permutation sensitivity is not necessarily beneficial. Greater redundancy can reduce dependence on individual assignments without improving predictive performance.

These measurements support the redundancy interpretation, but do not exclude additional mechanisms.

W2 / Q3: Statistical resolution and practical value

We agree that the practical utility of the diagnostics requires careful qualification.

For validation loss, paired-bootstrap 95% confidence intervals across 128 windows have half-widths of 0.00008–0.00021 nats. Eleven of thirteen training-arm comparisons are statistically distinguishable from the control after Holm correction.

However, changing only the control seed also produces a detectable difference of +0.00031 nats. Thus, statistical distinguishability does not establish an attributable improvement from the training objectives.

For downstream probes, the bootstrap standard deviation is approximately 0.011 MCC, while the largest observed arm–control difference is 0.013 MCC. These quantities describe empirical resolution, not a formal power-based minimum detectable effect.

Do the diagnostics predict downstream performance?

We do not claim that they do.

The non-coding ClinVar experiments provide intervention-level evidence: disrupting assignment reduces AUROC, whereas changing utilisation has substantially smaller effects. However, this does not establish that permutation sensitivity predicts downstream performance across independently trained checkpoints.

We have clarified this distinction in §9. The proposed metrics are complementary descriptions of routing structure, not validated predictors of downstream quality.

ClinVar replication

We also strengthened the biological evaluation.

The original 2,000-variant ClinVar endpoint was null (AUROC 0.522), and the subsequent non-coding subset was explicitly labeled exploratory.

We then evaluated an independently constructed, pre-registered set of 2,000 non-coding variants.

The control achieves AUROC 0.546 [0.521, 0.571]. Assignment permutation reduces AUROC by 0.024–0.076 across three permutations, with paired confidence intervals below zero.

However, five of seven retrained conditions also exhibit small but statistically distinguishable differences. We therefore report partial replication, not full confirmation.

The revised manuscript also reports that probes refitted on permuted features do not reliably separate from the control (Supplement S4).

We appreciate the reviewer's request for the uniform-target control. It helped us distinguish the effects of balancing granularity from those of the composition prior and narrow our conclusions accordingly.

Thank you for your detailed rebuttal and additional experiments. Your response has addressed some of my concerns. However, the connection between the proposed diagnostics and downstream performance remains unclear to me. Could you provide a clearer explanation or additional downstream experiments to support this connection? For now, I will maintain my Weak Reject rating, but I would be happy to raise it to Weak Accept if this concern is adequately addressed.

Reviewer_apNKOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper analyzes what the load-balancing objective actually controls in genomic mixture-of-experts models, using the JanusDNA model as the main case study. The authors show that the original balancing loss averages router statistics across layers, so the imbalance within each layer is effectively invisible. They then fine-tune the model with a per-layer version of the loss, also targeted to a prior based on genome composition. This changes how the tokens are distributed among the experts a lot, but the performance stays basically the same. On the other hand, permuting the experts at inference increases the validation loss by about 7.5%. From this, the authors conclude that the model depends more on which token goes to which expert than on how balanced the experts are.

Critical strengths

I find the idea interesting in itself. To my knowledge, a targeted analysis of what the load-balancing objective actually controls is new, and it could also be useful in other settings, such as MoEs for language modeling.

Critical weaknesses
  1. I am worried about the overall evaluation pipeline. The expert-permutation result is not a surprising observation: the permutation is applied only on the router side, while the experts are left unchanged, thereby breaking the learned correspondence between the router and the experts. A loss increase is therefore expected, much as permuting only the incoming weights of a dense layer would break it.

  2. The paper does not introduce a new method, and its empirical findings are not strong or general enough to compensate. The experiments are restricted to fine-tuning a single 35.6M-parameter checkpoint. No downstream benefit or harm is established. The proposed monitoring statistics are not validated, and the authors themselves cannot say whether a decrease in permutation sensitivity is desirable.

  3. The paper is extremely hard to follow. The writing is obscure and not pedagogical. Central concepts are never properly introduced, for example, the overall structure of the analyzed model (two independent forward/backward stacks fused by an attention block) and exactly where the MoE layers are located.

Questions for authors

My main concern is that the permutation experiment is performed at inference time with fixed weights, whereas the interventions on expert utilization are followed by 10.000 steps of fine-tuning. This makes the comparison between the two types of interventions somewhat unfair. I would therefore like to see what happens when the experts are permuted, and the model is subsequently fine-tuned using the same protocol as the other experimental conditions. In particular, I would like to know:

(i) How quickly does the validation loss recover after the permutation?

(ii) After 10,000 steps, does the permuted model achieve a validation loss comparable to that of the control A0?

If the model recovers quickly, this would suggest that the original token-to-expert assignments can be relearned or adapted relatively easily, and that the reported 7.48% increase in validation loss mainly reflects the immediate disruption caused by the permutation rather than a fundamental property of the learned routing structure.

2 comments
AuthorsOpenReview ↗

Response to Reviewer apNK

Thank you for your detailed review. We acknowledge the concerns about the asymmetric intervention protocols, the scope of the empirical evidence, and the presentation. These comments motivated substantial revisions and additional controlled experiments.

W1 / Q1: Permutation, fine-tuning, and the fairness of the comparison

We agree that permuting the router while keeping expert weights fixed disrupts a learned correspondence, and that the resulting loss increase is expected. We therefore performed the requested permuted-then-tuned experiment using the same fine-tuning protocol as the control: identical data order, learning rate, seed, and 10,000-step budget.

(i) Recovery speed. Excess validation loss falls from +7.6% immediately after permutation to +4.7% at step 100, +0.8% at step 1,000, and +0.25% at step 3,000. Thus, approximately 90% of the initial gap closes within the first 10% of the training budget.

(ii) Final performance. After 10,000 steps, validation loss returns to the range observed across control seeds. A second run with a different permutation and seed shows the same qualitative recovery.

Importantly, the recovered model remains sensitive to assignment disruption: permuting its newly trained routers increases validation loss by 6.3%, compared with 7.4% for the control on the same evaluation windows.

We have therefore clarified our interpretation in the Abstract and §6. The initial 7.5% increase measures the immediate cost of disrupting the learned correspondence at fixed weights; it is not an irreversible penalty. Fine-tuning can establish a new correspondence, but the recovered model again depends on that correspondence.

Our contribution is not the observation that breaking learned connections increases loss. Rather, it is the empirical distinction between expert utilisation, which balancing directly regulates, and assignment dependence, which utilisation statistics do not capture.

Additional controls support this distinction. Recomputing gate weights retains approximately 89% of the permutation-induced loss increase, indicating that gate-coefficient mismatch alone cannot explain the effect. We do not interpret this control as disproving the general dense-layer analogy.

We also explicitly acknowledge that utilisation interventions and fixed-weight assignment interventions are different protocols. Their effect sizes are compared descriptively, not as matched causal estimates.

W2: Novelty, generality, downstream evaluation, and diagnostics

We agree that the paper is a diagnostic study rather than a proposal for a new training algorithm.

The central contribution is a controlled empirical investigation of what load balancing regulates and what it leaves unmeasured.

We expanded the evaluation to PlantBiMoE and Genos-1.2B, which differ in architecture, training objective, expert count, and model scale. Both corroborate substantial sensitivity to expert-assignment disruption. We explicitly distinguish this finding from the utilisation-intervention result, which is primarily established in JanusDNA outside the collapse regime.

We also added zero-shot ClinVar variant-effect evaluation and an independent, pre-registered non-coding replication. The replication is reported as partial, rather than successful, because several retrained conditions also exhibit small but statistically distinguishable differences from the control.

Finally, we investigated the 39% reduction in Layer-7 permutation sensitivity. Lower single-expert ablation cost, increased expert-output similarity, and reduced effective rank support increased redundancy rather than improved downstream performance. We do not claim that lower permutation sensitivity is necessarily desirable.

The revised Discussion explicitly states that the proposed diagnostics are complementary measurements, not validated predictors of downstream quality.

W3: Presentation and accessibility

We substantially rewrote the introductory sections to address the presentation concerns.

  • §1: Clarifies the research question, contributions, and distinction between utilisation and assignment.
  • §2 and Figure 2: Introduce the genomic language-model setting and explain JanusDNA's two directional Mamba stacks, the locations of the eight MoE routers, and the final attention-based fusion.
  • §3: Adds related-work comparisons and clarifies how the diagnostic contribution differs from previous load-balancing and routing methods.

We also define the routing diagnostics, explain their interpretations, and add concise takeaways to the results sections.

We appreciate the reviewer's specific experimental suggestions. They helped us narrow the claims, distinguish recoverable assignment disruption from persistent dependence on learned correspondence, and make the paper substantially clearer.

I thank the authors for their answers, which are concise and go straight to the point of my questions.

The additional experiment requested in my main question confirms my hypothesis: after the permutation, the model closes most of the gap within the first 10% of the training budget and returns to the controls' range after 10,000 steps. As the authors acknowledge, the 7.5% increase therefore measures the immediate cost of disrupting a learned correspondence, not a fundamental property of the learned routing. Moreover, the authors themselves describe the paper as a diagnostic study, whose proposed statistics are not validated predictors of downstream quality.

For these reasons, I remain uncertain whether ICLR is the right venue for this work, and I would suggest a venue with an applied or domain-specific track. Overall, I cannot raise my score to an acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.