acceptodds
Under review as a conference paper at ICLR 2027

Unnormalized Relaxations for Robust Modular Learning

Abstract

Modular learning scales Large Language Models by routing inputs through frozen, domain-specific experts rather than retraining a monolithic model. The robust gating framework of Cortes et al. (2026) achieves minimax optimal performance under arbitrary shifts in the mixture proportions of the known source domains, but its strict global normalization (: ) couples the gate to every expert's predictions, creating a costly synchronization bottleneck at training time. We prove that this constraint is not only computationally unnecessary but *yields less favorable provable guarantees*. We introduce two *unnormalized* gating families and develop a complete theoretical, algorithmic, and empirical framework around them. The first bounded family replaces the data-coupled mass constraint with an expert-independent structural ceiling (), depending only on the gating function . We prove that achieves a provably tighter minimax robust guarantee in specialized regimes via a novel *Normalization Bonus* , with a matching lower bound; an ordering of empirical Rademacher complexities under mild conditions ( for ); *Exact Rejection Sampling* for unbiased inference with a structural efficiency bound; and a *Distillation Inheritance* theorem proving that causal student routers preserve the teacher's minimax bounds up to a controlled gap, with no causal projection penalty and an assumption-free total variation variant. The second locally-normalized family enables fully pointwise updates, with robust minimax guarantees via a *Selection Gain* and parallel distillation guarantees. Experiments on 1B+ parameter LLMs and 20-expert code generation ensembles confirm the theory: unnormalized gates reduce wall-clock time by 50% while strictly improving NLL over both monolithic and baselines.

Primary area:
learning theory
Keywords:
modular learning, mixture of experts, minimax robustness, unnormalized models, large language models, generalization bounds

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vUKrOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Building on the robust modular gating framework of Cortes et al. (2026), the authors propose two unnormalized gating methods, G∞,1G_{\infty,1} and G0G_0. These relax the original global normalization constraint to reduce synchronization costs during training. The paper provides theoretical results on minimax robustness, generalization bounds, exact rejection sampling, and structural distillation. Experiments cover small models, models with over 1B parameters, a 20-expert code ensemble, and frozen pretrained experts. The unnormalized gates consistently achieve lower NLL than the original G1G_1 and some monolithic/MoE baselines, while substantially reducing wall-clock training time. This is a systematic extension of an existing robust modular learning framework, with thorough theoretical and experimental treatment.

Critical strengths
  • The theoretical analysis is thorough. The authors propose two new gating relaxations and provide a broad set of theoretical results on minimax robustness, generalization, sampling, and distillation.
  • The experimental evaluation is extensive. Experiments cover lightweight models, 1B+ models, a 20-expert code ensemble, and frozen pretrained experts. Results are fairly consistent across settings and show gains in both NLL and wall-c
Critical weaknesses

None

Questions for authors
  1. There is still a gap between theory and implementation. The theoretical results rely on the exact G∞,1G_{\infty,1} constraint and function-space optimization, while the deep learning experiments use a mini-batch soft penalty.

  2. The evidence for addressing large-scale distributed synchronization bottlenecks is not direct enough. Distributed synchronization is a main motivation, but the paper does not show communication gains through multi-node or distributed scaling experiments. This systems claim is therefore inferred rather than directly tested.

  3. The abstract should preferably avoid citations.

Reviewer_ephxOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates robust combinations of frozen, domain-specific language models under changes in domain mixture proportions. It identifies strict global normalization in an existing gating framework as a computational bottleneck and proposes two alternatives: bounded gating and locally normalized gating. The paper develops theoretical guarantees for robustness, generalization, sampling, and distillation into causal routers. Experiments report improved test NLL and lower routing-training iteration time. However, the practical significance of these improvements remains insufficiently established through downstream evaluation and larger-scale deployment experiments.

Critical strengths
  1. Well-motivated problem. Reducing the cost of global normalization addresses a concrete limitation of the existing robust gating framework.

  2. Broad theoretical analysis. The paper examines how alternative constraints affect robustness, generalization, sampling efficiency, and distillation.

  3. Consistent likelihood improvements. Both proposed methods improve aggregate NLL across the reported settings, with results averaged over five runs.

Critical weaknesses
  1. Evaluation coverage is limited. The evaluation should include commonsense reasoning, and code-generation performance, comparing the combined model with each constituent expert under the same evaluation protocol. The current NLL-focused results do not sufficiently establish whether the combination improves these capabilities over the individual models.

  2. Scalability and ablations remain underexplored. The largest setting combines three approximately 1.6B frozen experts while training a small router, using limited data and 128-token contexts. Evaluation with well-trained experts totaling at least 10B parameters would provide stronger scalability evidence. Ablations should also isolate router capacity, adversarial domain weighting, and mass regularization beyond the capacity-penalty coefficient.

  3. Theoretical guarantees are not fully connected to the implementation. The convergence analysis concerns function-space optimization, while experiments use non-convex neural routers and penalties based on batch maxima. The extent to which these approximations preserve the theoretical constraints and guarantees needs clearer empirical verification.

Questions for authors
  1. What are the total training cost, peak memory, and generation latency when expert evaluation, mass estimation, sampling, and distillation are included?

  2. Do the improvements persist when all gating methods are evaluated through matched causal routers against a simple learned mixture baseline?

  3. How does worst-domain performance compare with G1G_1 under controlled domain-mixture shifts, given the negligible worst-case improvement in the 20-language experiment?

Reviewer_A5iqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper extends the robust modular learning framework of Cortes et al. (2026) by introducing two alternative gating constraints that relax global probability normalization when combining frozen, domain-specific language models. The authors derive theoretical guarantees for robustness, generalization, and convergence, and propose algorithms to train and use these models more efficiently. Experiments show faster training and small improvements in prediction accuracy compared to existing approaches.

Critical strengths
  • Theoretical parts: (1) the structural constraint ∑max⁡g(x,k)≤1\sum \max g(x, k) \leq 1 is an interesting alternative to the original data-dependent normalization constraint, connecting to convex optimization and sampling efficiency. (2) the Rademacher complexity ordering is a new comparison between the two hypothesis classes and potentially the most distinctive theoretical result.
  • The method allows to reduce training clock-time x2 / x3
Critical weaknesses
  • The convergence guarantee established in Theorem 6 does not directly apply to the algorithm used in the experiments. Specifically, the theorem considers an idealized optimization procedure with exact projection onto the feasible set, whereas the practical implementation uses a nonconvex neural network and enforces the gating constraint approximately through a minibatch-based penalty. Importantly, satisfying the constraint on individual minibatches does not guarantee that it holds over the entire input space. It remains unclear whether the proposed theoretical guarantees are relevant to the actual models being evaluated.

  • The writing should be improved - there are too many technical terms and definitions, the abstract includes math and citation, and it is very hard to read the paper.

  • Incremental theoretical contribution: Cortes et al. introduced the major conceptual framework while this submission develops a particular theoretical variation of that framework.

  • Empirical evaluation (NLL, Tables 2 + 3) shows insignificant improvement over the baseline work with G1\mathcal {G}_1

Questions for authors

No

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.