acceptodds
Under review as a conference paper at ICLR 2027

GAME-THEORETIC DATA ALLOCATION FOR EXPERT SPECIALIZATION AND ROUTING

Abstract

Can the allocation used to train experts also predict which expert will perform best on a prompt? Such a connection would make the intended division of labor inspectable before training and allow the allocation itself to guide inference. We introduce INFLENS, a mixture of LoRA experts whose training weights and inference routes come from the same location game in a fixed prompt-trait space. Players choose positions to maximize their expected shares of the prompt distribution, without using expert losses. We derive a kernel-general criterion for when co-located positions become locally unstable to separation, with an explicit Gaussian boundary determined by player count, kernel geometry, and trait covariance. In a transductive evaluation on a corpus assembled from seven safety-related benchmarks, the highest-share expert has the lowest held-out response negative log-likelihood (NLL) on approximately 90% of prompts across four kernels under a sharpened simplex representation, compared with approximately 15% after permuting allocations across prompts. In the principal seven-adapter configuration, token-level probability mixing improves NLL by 0.105 nats over a pooled adapter. Across four data splits, game shares outperform validation-trained linear gates in expected NLL when sampling one expert, while completed-sequence mixtures perform similarly. Controls show that the correspondence between allocations and expert identities matters, and that fixing solved positions retains performance at the principal operating point. These results connect an analyzable allocation of training data to useful expert specialization and routing without fitting a gate to expert performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, game theory, Nash equilibrium, interpretability, online learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nLbyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces INFLENS, which uses a location game in a fixed prompt-trait space to determine both LoRA expert training weights and inference routing. It derives a kernel-general local instability criterion for positional separation. Experiments on a mixture of seven safety-related datasets demonstrate correspondence between allocation shares and expert quality, together with NLL improvements over pooling and several routing baselines.

Critical strengths

The method provides an explicit connection between training-data allocation and inference routing without fitting the router to expert losses. The local stability analysis offers a useful account of when competitive positions separate.

The extensive controls examine allocation sharpness, initialization, fixed positions, routing rules, and model families. The paper also clearly distinguishes single-expert selection from probability mixing and acknowledges negative behavioral results.

Critical weaknesses

Limited generalization evidence. The trait space is constructed using the full dataset, including test examples and benchmark annotations. Performance without access to the test population remains unverified.

Unclear benefit of the game objective. The clustering baseline does not match INFLENS’s dense, share-weighted training, making it difficult to separate the benefits of competitive positioning from those of soft data weighting.

Questions for authors

Which annotations from validation and test examples enter trait-axis estimation? How do routing accuracy and NLL gains change when the axes, normalization, and game positions are fitted exclusively on training data?

Does INFLENS outperform a soft-clustering allocation using the same trait space, information access, dense training weights, expert count, and training budget? This comparison would help establish the contribution of the game objective.

2 comments

Thank you for recognizing the connection between training-data allocation and inference routing, the local stability analysis, and the breadth of the controls. We ran both experiments you requested on three data splits. The allocation remains predictive when the trait space is fitted on training data alone, and the matched comparison shows that the game's Dirichlet allocation performs well with either equilibrium or k-means positions.

"Gain" is the reduction in held-out NLL relative to the pooled adapter trained on the same data, in nats per token (higher is better; mean ± sample SD over the three splits).

W1 and Q1: Information used to construct the trait space

In the submitted experiments, trait-axis estimation used each source benchmark's annotations for all records, including validation and test records: the unsafe-response flag (BeaverTails), faithful versus hallucinated response (HaluEval), harmful versus benign goal (JBB-Behaviors), fraction of characters covered by PII spans (AI4Privacy), seemingly toxic benign prompt versus toxic control (OR-Bench), the binary injection label, and refusal-worthy versus benign instruction (Do-Not-Answer). Quantile normalization and resource-density estimation also used all prompts.

The annotations describe the dataset's prompts and reference responses, not our models' outputs, and test records did not enter adapter optimization, gate fitting, or expert-loss feedback. They did contribute to the resource map, so your concern about access to the test population is valid.

We refitted the axes, normalization, resource density, and equilibrium initialization on the training partition only, then retrained the experts.

Trait space fitted onExpected sampling gainTop-1 agreementHindsight-best gain
All records0.1183 ± 0.00290.890.1454 ± 0.0024
Training only0.1093 ± 0.00520.880.1454 ± 0.0029
Training only, label-free0.0903 ± 0.00240.810.1322 ± 0.0017

The training-only version has slightly higher routed NLL on all three splits. The paired reduction in gain is 0.0091 ± 0.0027 nats, about 8% of the original improvement, and top-1 agreement falls by 1.6 percentage points. The hindsight-best gain is unchanged (paired difference 0.0000 ± 0.0009 nats).

The third row also removes benchmark annotations: we clustered the training prompt embeddings with k-means and built one cluster-versus-rest direction per cluster. The allocation still identifies the best expert for 81% of prompts, against 15% for the shuffled baseline.

W2 and Q2: Matched comparison with soft clustering

The comparison matches the trait space, resource information, dense share-weighted training, seven expert pairs, and training budget, with positions frozen in every arm.

PositionsKernelSampling gainArgmax gainMixture gainTop-1
Game equilibriumDirichlet0.1166 ± 0.00210.1223 ± 0.00280.1350 ± 0.001890.1%
k-means centroidsDirichlet0.1172 ± 0.00270.1229 ± 0.00250.1360 ± 0.002589.9%
Soft k-meansGaussian, sharpness matched0.1101 ± 0.00250.1140 ± 0.00240.1299 ± 0.002487.8%
Soft k-meansGaussian, fitted bandwidth0.1067 ± 0.00250.1165 ± 0.00260.1326 ± 0.002687.5%

Comparison with soft clustering. Relative to soft k-means with its fitted Gaussian bandwidth, InflEns improves NLL by 0.010 ± 0.001 nats when one expert is sampled, about 0.006 under argmax routing, and 0.002 under probability mixing.

Positions and allocation. Replacing the game's equilibrium positions with k-means centroids while keeping the Dirichlet allocation gives nearly identical performance (paired difference 0.0006 ± 0.0007 nats). The game reaches positions of comparable quality through its competitive objective, without a separate clustering step. We do not claim better positions on this distribution.

With k-means centroids, the Dirichlet kernel's gains exceed those of sharpness-matched Gaussian soft k-means by about 0.006 to 0.009 nats. Together with the position control, this supports an important role for the allocation rule in expert learning and routing. Reach has a smaller effect: matching the game's sharpness improves sampling gain by 0.0034 nats but reduces argmax and mixture gains by 0.0025 and 0.0027, so no single reach is best for every routing rule.

Contribution of the game formulation. Before any adapter is trained, the game provides a criterion for when a co-located configuration becomes locally unstable to positional separation. The criterion depends explicitly on the number of players and applies to general influence kernels, including the Dirichlet kernel used here. The framework links these analytically characterized allocations to both expert training and inference.

Thank you for the response. Most of my concerns about the experiments have been addressed. However, considering the manuscript’s overall quality, I am inclined to maintain my current score.

Reviewer_3SfxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a location-game based method for determining data allocation in advance for mixtures of experts. This method depends only on the training data, the chosen trait space, and hyper parameters - not on any results of training. The paper demonstrates that training a mixture of LoRAs with its game-theoretic allocation improves single expert NLL on safety data, while matching the mixed-prediction NLL of a validation-trained linear gate. The paper also shows that the performance of its trained experts are predictable from the results of the location game. However, all tested fine-tuning (including the paper's method) degrades actual performance on a safety benchmark. I believe the paper's main contribution is introducing a method for determining expert specialization without training, a potential boon for interpretability.

Critical strengths
  1. the paper introduces a principled and analytically grounded method for determining MoE allocation and routing in advance of training, a laudable goal
  2. the paper is thorough in its game parameter and model size ablations
Critical weaknesses
  1. No evidence of competitive OOD performance is provided
  2. The authors do not do enough to demonstrate that their method results in a system that is genuinely more interpretable. Are the specializations of the experts easy to understand and explain in natural language?
  3. No comparison to soft k means
  4. The massive performance degradation on the safety benchmark makes it hard to trust that the paper's recipe will actually be competitive with alternatives when training is expected to improve performance
Questions for authors
  1. Did you try different (hopefully still interpretable) ways of parameterizing trait space (e.g. PCA). If not, why?
  2. What do you think was responsible for the performance degradation induced by fine-tuning?
3 comments

Thank you for recognizing the value of specifying allocation and routing before expert training, and for noting the analytical foundation and breadth of the ablations. We added experiments addressing each concern. "Gain" is the reduction in test NLL relative to a pooled model trained on the same data, in nats per token (higher is better; ± is SD over three splits). "Dense gate" is a dense softmax router trained jointly with seven LoRA experts on the game's batches.

W1: Out-of-distribution performance. We evaluated GSM8K and seven other datasets not used in training. NLL on clustering axes, lower is better:

Game mixturePooledDense gate
GSM8K0.694 ± 0.0070.738 ± 0.0150.726 ± 0.007
Seven datasets, full text2.382 ± 0.0122.444 ± 0.0072.364 ± 0.003

GSM8K tests mathematical reasoning, which our training benchmarks do not represent. The game's mixture has lower NLL than pooling and the dense gate in every trait space tested, and higher answer accuracy (0.425 on labelled axes, against 0.194 and 0.177). All fine-tuned models have higher NLL than the base model here, and the game deteriorates least. On the seven datasets, the game improves full-text NLL over pooling, the dense gate generally improves it further, and response-only NLL is mixed (the three means coincide at 1.331).

W2: Interpretable specialization. Six of the seven expert pairs draw at least 85% of their prompts from a single benchmark (prompt injection, personal data, over-refusal), and each outperforms pooling on its assigned prompts by 0.04 to 0.25 nats. Routing can be inspected through trait coordinates, expert positions, and the influence kernel, with no access to expert weights. The dense gate spreads each prompt broadly (largest routing weight about 0.18, against 0.99 for the game). On PCA axes, every one of its standalone components is worse than pooling.

W3: Soft clustering. Every arm uses the same trait space, resource information, training, and budget, with frozen positions (SDs are at most 0.0028).

PositionsKernelSampling gainMixture gainTop-1
Game equilibriumDirichlet0.11660.135090.1%
k-means centroidsDirichlet0.11720.136089.9%
Soft k-meansGaussian, matched sharpness0.11010.129987.8%
Soft k-meansGaussian, fitted bandwidth0.10670.132687.5%

Game and k-means positions perform alike under the same Dirichlet allocation, within 0.0010 nats. Soft k-means with its own fitted Gaussian trails the game by 0.010 ± 0.001 nats under sampled routing. Clustering can provide effective positions here, and the allocation around them matters for both training and routing. InflEns gives an analytical framework for both choices. One particular gain of InflEns is control through kernels while k-means is a limiting case of the Gaussian mixture model.

W4 and Q2: Safety degradation. Unsuitable training content contributes: about 2,000 BeaverTails targets are unsafe responses, and four benchmarks contribute bare harmful prompts. The decline also occurs with pooled training.

Training corpusMetricGameDense gatePooled
OriginalHarmBench ASR0.510.450.64
Unsuitable targets removedASR0.180.290.44
Unsuitable targets removedIFEval0.2920.2740.281
Safe responses, prompt maskedIFEval0.2530.2010.139

The base model has ASR 0.05 and IFEval 0.407. Removing unsafe and hallucinated targets reduces attack success for every method (paired 95% intervals exclude zero), though the safety ordering between methods is not stable across runs and splits. Guard-checked safe responses with prompt masking give zero observed attack success for all three, while over-refusal remains a concern (a lexical proxy flags about 36% of XSTest's safe prompts). The game has the highest IFEval in every setting.

Q1: Alternative trait spaces. We evaluated PCA, clustering-derived axes, and ICA, each fitted on the training split without labels.

Trait axesGame gainShare of labelled gainTop-1Dense gate NLL advantage
Labelled+0.109 ± 0.005Reference88%≈0 at raised router LR
PCA+0.074 ± 0.00168%78%0.072 ± 0.004
Clustering+0.090 ± 0.00283%81%0.034 ± 0.006
ICA+0.003 ± 0.0073%56%0.059 ± 0.005

Top-1: the largest allocation identifies the lowest-NLL expert (chance ≈ 15%). Manually specified axes are not required, but the structure the representation preserves affects specialization. PCA's leading directions span multiple clusters. The clustering axes (k-means with seven clusters, then one Fisher direction per cluster against the rest) concentrate clusters near simplex corners, with mean purity 0.82. The dense gate has lower NLL on the label-free axes, but its components lack the individually useful specialization that InflEns retains.

Thank you for your response. Several of my questions and concerns were addressed, but I still feel like the negative relationship between training on the safety data using any mixture of experts setup and actual performance, along with InflEns failing to really beat out simple k means prevent me from raising my score. I want to reiterate that the concept behind the work is interesting, and add that I could see a more refined, future version of this paper being ICLR-worthy.

Thank you for the follow-up and for the encouragement. Two clarifications, with results obtained since our response.

Fine-tuning on the safety data and performance. The loss relative to the base model comes from fine-tuning on this corpus, and the mixture of experts loses the least: the pooled model, which has no experts, loses the most. On GSM8K, now on three data splits, the game's mixture answers 0.43 ± 0.04 of the problems on the targets-removed corpus against 0.17 ± 0.05 for the pooled model, and 0.42 ± 0.05 against 0.29 ± 0.07 on the safe-response corpus. It is ahead on every split of both corpora, by 0.05 to 0.36. The base model answers 0.71, so we agree that every fine-tuned model pays a real cost on this corpus, and we state that as a limitation.

The experts also make the damage repairable. Per-case harm scores grouped by the routed expert put 0.49 of the game's 0.51 HarmBench attack success on one expert. Retraining that expert alone on safe responses, with the router and the other six experts frozen, lowers HarmBench from 0.51 to 0.02 with no detectable change in IFEval (-0.011, 95% interval includes zero). Retraining the three experts the trace names reaches 0.00, the level of retraining all seven, at IFEval 0.301 against 0.309. We repeated this on a second data split with predictions registered before any edit: HarmBench fell from 0.60 to 0.05 and StrongREJECT from 0.49 to 0.04, with IFEval unchanged (+0.018, interval includes zero). There are two caveats, however. The registered predictions of the final scores were missed by 0.03 to 0.05. The three-expert edit refuses 33 to 39% of XSTest's safe prompts (about 6% for the one-expert edit), almost all on prompts routed to the edited experts. A pooled model. dense gate, and many standard learned MoE gates offer no such handle, because none of their components can be attributed to the harmful prompts.

k-means. We agree the positions are roughly the same as k-means which explains why the routing didn't demonstrate a large difference. On this trait space the k-means centroids and the game's equilibrium coincide (they differ by 0.01 to 0.04 where neighboring experts are 0.22 apart). The row labelled "k-means centroids" in our table still allocated prompts with the game's kernel and reach, so it tests positions only. Soft k-means as a method, with its own responsibilities as the allocation, is the last row of that table: it loses 0.010 ± 0.001 nats (about 9% of the gain over the pooled model) and 2.6 points of best-expert prediction, on all three splits. Though we recognize that margin is small. In the revision we present the game's contribution as the allocation rule and the reach, with expert positions that the competition model derives and clustering confirms.

Reviewer_KtrbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors introduce InflEns which is a framework where experts compete for portions of the training data distribution. The allocation of that data determines how the experts learn and how they are combined at inference. They identify conditions where experts separate into specialized regions based on the data distribution through a stability analysis. The resulting allocations are used for expert training and inference routing. The authors validate their approach by using mixtures of LoRA adapters across benchmarks, showing that the expert with the greatest allocation achieves the lowest negative log-likelihood on a held out test set for approximately 90% of prompts, while top two allocations captured the best expert around 96% of the time. These findings suggest meaningful expert specialization can emerge through competitive data allocation with co-adaptation training between router and expert.

Critical strengths

● Very interesting new approach to expert specialization by modeling training data allocation as a game, which could fundamentally improve our understanding as to why ● Very promising empirical results showing the allocation mechanism is producing meaningful specialization for around 90% of prompts.

Critical weaknesses

● Experiments only evaluated on LoRA adapters not conventional MoE architectures ● Proposed framework relies on a predefined prompt representation which raises the question of how much of this specialization depends on that.

Questions for authors

● Why was the framework evaluated using LoRA adapters rather than conventional MoE architectures? ● Could InflEns be extended to a conventional MoE architecture.

1 comment

Thank you for highlighting the game-theoretic account of specialization and the close agreement between allocation and expert quality. Your questions led us to test full feed-forward experts and label-free trait representations, and the allocation extends to both. "Gain" is the reduction in test NLL relative to a pooled model trained on the same data, in nats per token (higher is better; mean ± SD over three splits unless a seed is stated).

New learned-gate comparisons. We adapted dense softmax, noisy top-k, Switch, Mixtral, DeepSeek-V3, Gaussian, and X-MoE cosine gates to the same trait coordinates and ran two studies: gates trained jointly with seven LoRA experts on the game's batches (at the published learning rates and a raised router LR of 0.01), and the same gates fitted on validation losses over the game's frozen experts. This separates joint training from router-only improvements.

W1 and Q1: Why LoRA, and extension to conventional MoE

Adapter mixtures let us study specialization over a shared frozen base at manageable cost, with experts that can be inspected, trained in parallel, and served independently. The allocation rule is independent of the low-rank parameterization. We tested this by upcycling four FFN blocks of Qwen2.5-1.5B (layers 20–26 of 28), copying each FFN into seven full feed-forward experts. With the game's shares as a fixed prompt-level gate, NLL is within 0.006 ± 0.004 nats of a learned dense softmax router, and the game beats learned Mixtral top-2 and Switch top-1 routers on every split. Its hindsight-best expert has NLL 0.033 nats lower than the dense router's. Gains are smaller here than with LoRA experts (0.028 for the game and 0.034 for the dense router, against about 0.13). The routers activate different numbers of experts, so these comparisons do not match inference compute.

Q2: Prompt- and token-level routing

Our traits describe prompts, making the prompt the natural routing unit. For token routing, the game's shares can act as a prior for a learned router, which we tested on the upcycled FFN experts.

Token-level routerGain
Dense, game's log-shares as fixed prior−0.049
Dense, from scratch−0.100
Switch−0.147

Unlike prompt-level routing, none of the token-level routers exceeds the pooled model within this training budget, but the game's log-shares give the strongest token-level result.

W2: Dependence on the prompt representation

The axes need not be derived from benchmark labels. We replaced the seven benchmark-derived directions with label-free ones fitted only to training prompt embeddings and kept the rest unchanged.

Trait axesGame gainShare of labelled gainTop-1 agreementDense gate NLL advantage
Labelled+0.109 ± 0.005Reference88%Comparable at raised router LR
PCA+0.074 ± 0.00168%78%0.072 ± 0.004
Clustering+0.090 ± 0.00283%81%0.034 ± 0.006

Top-1 agreement measures whether the largest allocation identifies the lowest-NLL expert (chance ≈ 15%). PCA's leading directions span multiple clusters, which may explain its weaker specialization. The clustering axes (k-means, then one cluster-versus-rest Fisher direction per cluster) recover much of the benchmark structure without labels, with mean purity 0.82.

The jointly trained dense gate has lower NLL than the game on both label-free spaces, but its components rely on blending. On PCA axes, every standalone component is worse than the pooled model. On the clustering axes, Gaussian lands between the dense gate and the game, DeepSeek-V3 and Mixtral are close to the game, and Switch trails. The game's experts remain useful individually.

Improving routing over the game's experts. Gates fitted on the game's frozen experts improve on its shares under argmax routing and probability mixing (gains over pooling).

Trait axesRoutingGame's sharesFitted Gaussian gate
LabelledSample one expert+0.117+0.102
LabelledArgmax+0.118+0.123
LabelledMixture+0.129+0.134
Clustering, seed 0Argmax+0.093+0.101
Clustering, seed 0Mixture per token+0.108+0.120
PCA, seed 0Mixture+0.097+0.104

The game's shares retain the advantage when one expert is sampled, consistent with Section 4.3. On labelled axes, the fitted gates also outperform the same gates trained jointly at the published learning rates (+0.098 Gaussian, +0.088 DeepSeek-V3, +0.062 dense softmax). The post-hoc improvements (0.012 nats for clustering, 0.007 for PCA) recover about one-third and one-tenth of the dense-gate gaps, so the jointly trained network still has lower NLL. InflEns provides experts with recognizable roles and a routing rule before expert training, and those experts can also benefit from a subsequently fitted router.

Comments on the paper

1 comment

Thank you again for your insightful feedback. We believe that we have addressed all of your concerns. We have also added a new draft that includes the additional experimentation results and figures. If you have additional questions please let us know.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.