- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the distinction between expert utilization and expert assignment in genomic mixture-of-experts models. Using JanusDNA as the primary case study, the authors show that the released pooled load-balancing objective can hide substantial layer-wise imbalance. They introduce per-layer and composition-targeted balancing objectives that substantially change expert utilization while producing only small changes in reconstruction-style validation loss. The authors then perform inference-time interventions that permute expert identities, recompute gate weights, randomly route tokens, swap expert pairs, or remove individual experts. Expert permutation substantially increases validation loss while preserving the utilization histogram up to relabeling, suggesting that token–expert correspondence matters more than aggregate utilization. Similar permutation-versus-ablation ordering is reported for PlantBiMoE and Genos-1.2B. The paper proposes permutation sensitivity, class enrichment, and effective output rank as additional routing diagnostics.
Critical strengths
The paper makes a clear conceptual distinction between expert utilization and token–expert assignment. The permutation intervention is well targeted because it preserves the utilization histogram while changing the assignment structure.
The decomposition into full permutation, gate recomputation, random routing, pair swaps, and single-expert ablations helps establish that the observed sensitivity is not simply caused by one indispensable expert or by a gate-weight mismatch.
Critical weaknesses
The central training-objective conclusions are not supported by sufficient replication or attribution. Most fine-tuning arms use one primary seed, and the authors acknowledge that the small validation-loss differences among checkpoints are separable but not attributable to the proposed interventions. The three-seed result applies to layer-7 permutation sensitivity, not to the downstream or validation-loss effects of the training objectives.
The main evaluation metric limits the interpretation of the results. JanusDNA is a bidirectional denoising model, and the reported validation loss is an unmasked reconstruction-style score in which neighboring states and the terminal attention block may retain information about the original token. Therefore, the large loss increase under expert permutation does not directly establish a comparable degradation in autoregressive prediction or biological utility.
The proposed structural diagnostics are not validated as useful proxies for genomic modeling quality. The fine-tuned arms show no clear attributable improvement on the reported linear probes, generation coverage, or ClinVar endpoint. A reduction in permutation sensitivity may reflect reduced specialization or increased redundancy, and the paper does not establish which interpretation is correct.
Several evaluations use small or non-independent samples. Some routing and generation statistics rely on four or eight windows or a small number of generated sequences, bootstrap intervals are nominal and uncorrected, and the PlantBiMoE windows are not verified to be excluded from pretraining data. These limitations make the cross-model generalization claims difficult to assess.
Questions for authors
Can the authors replicate the main fine-tuning-objective comparisons across multiple independent seeds and report uncertainty for validation loss and every downstream metric? In particular, which conclusions remain after separating seed variation from intervention effects?
Can the authors evaluate the routing interventions with a genuinely held-out predictive objective, such as masked-token or autoregressive likelihood where applicable, rather than relying primarily on unmasked reconstruction loss?
What evidence shows that the proposed diagnostics predict biological utility? For example, can permutation sensitivity, class enrichment, and effective rank be evaluated against independent genomic prediction tasks across checkpoints or models?
Can the authors provide a larger and independently sampled evaluation for the generation and structural statistics, with corrected treatment of multiple comparisons and clearly disjoint windows?
The paper argues that load balancing should be monitored together with assignment-sensitive statistics, but does not establish when practitioners should change or anneal the balancing objective. Can the authors provide a controlled experiment that tests whether any proposed intervention improves training or downstream performance, rather than only changing utilization or sensitivity?
Response to Reviewer gFAe
Thank you for your detailed review. We have strengthened the revision with additional controls, expanded evaluations, and clearer limitations. We address your five questions below.
W1/Q1: Independent seeds and attribution
We agree that most fine-tuning arms have only one primary seed and that window-level bootstrap intervals cannot substitute for training-seed replication.
The central fixed-weight diagnostic observation is that expert permutation increases validation loss by 7.48%, whereas the 14 fine-tuned arms span only 0.06% on the same 128 windows. These are different intervention protocols, and we now explicitly avoid interpreting their effect sizes as matched causal estimates.
For fine-tuning comparisons, the empirical paired-bootstrap 95% CI half-widths are 0.00008–0.00021 nats. A control retrained with only a different seed differs by 0.00031 nats, comparable to the objective-related differences. We therefore do not attribute the small loss differences to particular objectives.
The Layer-7 structural finding has stronger replication: across three seeds, control sensitivities are 2.35/2.21/2.21%, versus 1.37/1.37/1.37% for the prior arm. The revised paper distinguishes this replicated structural result from single-seed validation-loss and downstream observations.
W2/Q2: Target leakage and predictive objectives
We directly tested whether the predicted token is visible to the model (Appendix E).
At position t, the prediction combines forward information through t−1 and reverse information from t+1 onward. The terminal attention masks preserve this exclusion.
As an intervention check, replacing or masking x_t leaves its predicted log-probability unchanged, whereas predictions at adjacent positions change substantially. This supports interpreting the reported score as pseudo-log-likelihood rather than target-leaking reconstruction.
We agree that pseudo-log-likelihood is not directly comparable to autoregressive likelihood.
We additionally evaluated expert permutation under PlantBiMoE's masked-language-model objective and Genos-1.2B's autoregressive objective. Both show substantial permutation penalties relative to single-expert ablation. These experiments support cross-objective consistency, although they do not establish downstream biological generalisation.
W3/Q3: Biological utility of the diagnostics
We agree that the proposed diagnostics are not validated predictors of downstream performance across independently trained checkpoints.
The non-coding ClinVar experiments provide narrower, intervention-level evidence: assignment permutation reduces AUROC, while utilisation-changing fine-tuning produces much smaller differences.
We explicitly report the original full ClinVar evaluation as null (AUROC 0.522), the subsequent non-coding subset as exploratory, and the independently pre-registered 2,000-variant evaluation as partial replication.
Refitted probes on permuted features do not reliably separate from the control.
We therefore present permutation sensitivity, class specificity, and effective rank as complementary structural diagnostics, not established proxies for genomic prediction quality.
W4/Q4: Evaluation size, multiplicity, and independence
We expanded the permutation decomposition to 128 validation windows (Appendix E). Gate-weight recomputation retains approximately 89–90% of the full permutation penalty, and the Layer-7 separation remains visible.
We also report chromosome-stratified and sequence-length sensitivity analyses.
The 13 training-arm comparisons and four selected between-arm contrasts are evaluated with Holm correction; their reported significance conclusions remain unchanged.
We acknowledge that the PlantBiMoE evaluation uses TAIR10 chromosome 1, which was included in its pretraining corpus. These are in-domain comparisons, not held-out generalisation estimates. Pretraining overlap remains a limitation even for paired intervention comparisons.
Q5: When should balancing objectives change?
We agree that the current experiments do not establish an intervention that improves downstream performance.
We evaluated several assignment-oriented objectives, including SimBal, but found no consistent validation-loss or downstream improvement.
Our recommendation is therefore diagnostic rather than prescriptive: first assess expert collapse using utilisation and ablation statistics, then monitor assignment sensitivity and expert diversity alongside balancing metrics.
The observed reduction in Layer-7 sensitivity may reflect increased expert redundancy rather than improved routing quality. We do not recommend replacing or annealing a balancing objective solely on this basis.
We appreciate these questions. They helped us distinguish what the experiments directly establish from what remains open for future validation.