acceptodds
Withdrawn from ICLR 2027

Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Preference modeling aims to capture heterogeneous human values and increasingly relies on learning latent components to represent this variation without fine-grained supervision. What these components actually represent, however, is rarely examined. We audit Mixture-of-Experts (MoE) reward models along three rungs—semantic coherence, functional correspondence, and composability—in controlled and real-world settings, holding the architecture and training data fixed so that differences isolate the effect of the training objective. Our diagnosis is that the components conventional MoE reward models learn are entangled, which limits not only interpretability but also the personalization such models are built for. The entanglement is not intrinsic to the architecture but a property of the training objective: off-the-shelf sparsity and diversity regularizers already substantially reduce it. The resulting configuration, which we call sparse MoE, recovers all underlying semantic categories without domain supervision in a controlled setting, reaching 84.8% expert purity against 60.8% for conventional MoE; on real-world data its experts admit human-interpretable task- and domain-level descriptions aligned with their functional strengths. Targeted interventions on routing weights produce predictable preference changes, showing that these interpretations are behaviorally grounded rather than merely descriptive. Crucially, the disentanglement induced by this objective also makes the experts composable: lightweight router updates yield a 25.81-point improvement in few-shot personalization with 50 examples, roughly 2.7 times the gain of the identical architecture and adaptation procedure applied to entangled experts. Together, these results show that MoE reward models can learn interpretable expert structure, and that the same structure can substantially improve personalization.

Primary area:
interpretability and explainable AI
Keywords:
Preference modeling, Interpretability, Personalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_brpeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether Mixture-of-Experts (MoE) reward models can learn interpretable and composable latent preference structures. Building upon an existing MoE reward model architecture, the authors introduce three standard regularization techniques into the training objective: local routing sparsity, global expert load balancing, and expert output diversity, aiming to encourage functional specialization among experts. The paper then evaluates the learned expert structures along three dimensions: Semantic Coherence, which measures whether experts correspond to distinct semantic categories; Functional Correspondence, which examines whether experts exhibit stronger predictive performance in their associated categories and whether routing interventions induce the expected preference changes; and Composability, which assesses whether personalized preference modeling can be achieved through few-shot router adaptation.

Critical strengths
  1. Practically relevant research question. The paper goes beyond the predictive performance of MoE reward models to investigate whether individual experts exhibit meaningful semantic and functional specialization. This question is relevant to the development of interpretable and personalized reward models.
  2. Systematic experimental evaluation. The authors evaluate the learned expert structures along three dimensions,semantic coherence, functional correspondence, and composability,using controlled datasets, real-world preference data, human evaluations, and routing interventions. These experiments provide evidence from multiple perspectives.
  3. Practical improvements. Although the proposed approach relies on existing regularization techniques, it achieves substantial improvements in expert specialization and few-shot personalization while keeping the model architecture and training data largely unchanged. This suggests that relatively simple modifications to the training objective can yield practical benefits.
Critical weaknesses
  1. Insufficient research motivation and limited practical significance. From the perspective of research motivation, I find the proposed evaluation of preference entanglement in MoE models to be of limited practical value. The observation that entangled experts can negatively affect model capabilities is already well understood, and the paper largely reconfirms this phenomenon through empirical evaluation. Moreover, the proposed approach relies on standard regularization techniques rather than introducing a new mechanism to address the underlying issue. The paper also does not provide a sufficiently deep analysis of the interpretability mechanisms behind expert entanglement and specialization. Consequently, the work appears to offer limited new insights beyond established observations.
  2. Mismatch between the stated motivation and the learned expert representations. The learned expert structures primarily reflect task domains rather than human values or preference dimensions, which is inconsistent with the problem formulation presented in the introduction. Although the paper motivates its approach through heterogeneous human values and preference modeling, the identified experts mainly correspond to tasks or domains such as mathematics, text summarization, legal consultation, and creative writing (Tables 2 and 3). These characteristics largely capture the semantic categories of input prompts rather than differences in users' underlying values or preferences. Therefore, the experimental findings do not sufficiently support the paper's broader claims about modeling heterogeneous human preferences.
Questions for authors

See the Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer brpe (1/N)

Thank you for your review! Below, we address your concerns in detail, with a brief TL;DR for each question. We are still incorporating new results and discussions into the revised manuscript, which we plan to upload by November 15. We will then post a general response to all reviewers summarizing the revisions.

TL;DR

[W1: Weak contribution] We are the first controlled study to establish interpretability in MoE reward models and relate expert disentanglement to personalization ability in reward modeling. We provide carefully designed, architecture-specific interpretability analyses for MoE reward models, rather than proposing new methods. We further add new analyses of the mechanisms behind sparse MoE's personalization gains and the relationship between interpretability and personalization.

[W2: Domain-level expert structure] Domain-level structure is also meaningful for individual preferences and the personalization experiments support this empirically; the studied structure also supports capturing response-level preferences with minor changes.

Below, we provide a more detailed response to each of your question.

W1: Weak contribution

We respectfully disagree with your assessment of our contribution. Our work is a controlled empirical study, not a methods paper: it examines interpretability, personalization, and the connection between them in MoE reward models, using analyses built around their router–expert structure.

(1) "Entangled experts hurt model capabilities" is established in LLM MoEs, while its validity has not been systematically studied in reward modeling. In fact, our experiments suggest more nuanced findings: entangled experts (vanilla MoE and MiCRo) achieve better universal preference modeling accuracy than disentangled ones (sparse MoE) on SHP and 700K (Sections 5.1/6.1). They fall short in interpretability and personalization (Sections 5.1//6.1/6.2).

Prior latent-component reward models have mainly focused on decomposing reward functions. However, to the best of our knowledge, they have not systematically compared the composability and personalization capabilities of learned latent components across different levels of disentanglement. Our work is therefore the first to show that more interpretable and specialized experts can be composed more effectively to model new preferences. Although this finding may be somewhat expected, we believe the systematic analysis and empirical validation still provide value to the research community.

(2) Standard regularizers are a deliberate design choice, not a limitation. As stated in Section 3, we intentionally keep the architecture of MiCRo and only add established sparsity, balance, and diversity regularizers. Because the manipulation is minimal and regularizers are standard, the structure that emerges can be attributed to the objective itself rather than to a particular new mechanism. Our goal is not to propose a new regularizer. It is to establish, as a positive and reproducible case, that MoE reward models trained only on binary preferences can learn interpretable, specialized experts, and that this structure makes them better personalizers. Future, more specialized methods can build on this foundation.

(3) Our analysis is carefully tailored to the router–expert structure of MoE reward models, going beyond generic interpretability analyses. For example, instead of using only generic interpretability tools such as feature descriptions from top-activated examples, We instead consider both the router and the conduct comprehensive interpretability analyses: semantic coherence of routing (Section 5.1/6.1), correlational and interventional correspondence between routing and expert specialization (Section 5.1/5.2/6.1), and how router upweights semantically related experts during adaptation (Table 15). We believe these analyses provide useful insights for researchers working with similar architectures.

(4) New mechanism analyses. Following Reviewer nq5U, we add analyses of why sparse MoE experts personalize and compose better. We find that expert specialization and diversity lead to higher accuracy for the best-performing expert on each target attribute, and larger accuracy differences across experts make it easier to learn an effective router during adaptation. Beyond specialization and diversity, we also find that interpretability, measured by description fidelity, predicts composition gains, i.e., how much the adapted reward model improves over its best single expert. One possible explanation is that interpretability reflects how well specialized and diverse experts align with coherent concepts, allowing new preferences to be modeled as sparse mixtures of a few experts that can be learned with limited adaptation data. We hope these findings provide further insights to the research community. For more details on this analysis, we kindly refer you to our response to Reviewer nq5U.

AuthorsOpenReview ↗

Response to Reviewer brpe (2/N)

W2: Domain-level expert structure

We agree and have acknowledged in the limitations that experts in the current MoE reward model captures prompt-identifiable features, such as task or domain, rather than response-level features. This is due to the design of prompt-level routing.

However, we would like to argue that task- and domain-level structure is itself meaningful for individual preferences. In realistic settings, preferences are often conditioned on domain context. One user may want precise judgments on safety- or factuality-sensitive prompts but care little about fine-grained differences on coding or entertainment prompts. Domain-specialized experts are natural components for such preferences: a user's preference can be expressed by re-weighting the experts responsible for the contexts that user cares about. Our experiments in Section 6.2 also empirically support the usefulness of such structures in personalization: even though the target preferences are specific response-level attributes, sparse MoE can still effectively adapt to them.

Besides, the studied architecture also supports different types of interpretation. Our study focuses on prompt-level features because they allow a controlled evaluation (category recovery and expert purity against known categories). The framework, however, is not tied to this level. Wang et al. (2026), for example, build on our work and show that by simply changing where to apply the interpretability regularization, the MoE reward models can also learn response-level preferences, such as preferring "blunt, sardonic humor rather than polite, measured replies".

References

  • Wang et al. (2026). Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast. Preprint
Reviewer_brpeOpenReview ↗

The contribution is limited.

I understand that this is an empirical study. However, I find its significance limited, as models continue to evolve, and the interpretability properties identified here are not guaranteed to hold in future models. Moreover, the finding that more interpretable and specialized experts can be combined more effectively to model new preferences largely reflects an existing consensus. While the paper provides detailed empirical evidence, I do not believe this finding, in itself, substantially advances the field. I therefore maintain my original score.

Reviewer_aJ4DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether experts in MoE reward models learn interpretable roles that support personalization. Sparsity and diversity regularization encourage distinct expert specializations, which the authors validate through routing and behavioral analyses. Updating only the router with a few preference examples improves personalization over vanilla MoE.

Critical strengths
  1. The paper examines whether experts in MoE reward models learn interpretable specializations and whether these specializations are useful for personalization. It also shows that adding sparsity and diversity regularization to the objective can produce more distinct expert specializations while the model architecture remains unchanged.

  2. The authors check whether similar prompts are routed to the same expert and whether that expert performs better on those inputs. Together, these analyses provide convincing evidence that the interpretations reflect the model’s actual behavior.

  3. Sparse MoE achieves stronger test-time preference adaptation than the baselines by updating only the router with a small number of preference examples. These results suggest that learning interpretable and specialized experts can also make them easier to recombine for new preferences.

Critical weaknesses
  1. Sparse MoE achieves lower universal preference accuracy than vanilla MoE in the main experiments.

    On SHP, it also performs worse than the single reward model baseline. The substantial gains in few-shot preference adaptation are a strength. However, this accuracy drop is a concern in settings where both personalization and general preference accuracy matter.

  2. The routing weights sum to one, but it is unclear to me whether reward differences are on comparable scales across experts.

    An expert that assigns larger reward differences between responses may have more influence on the final prediction, even with the same routing weight. The paper should clarify this.

  3. Appendix K.1 shows that adapted sparse MoE reward models can guide GRPO training toward specific preference attributes. However, it is unclear how this approach would scale to many users with different preferences.

    \Training and maintaining policies for each user could be expensive. Also, Appendix K.2 discusses alternatives such as user-specific LoRA modules and Best-of-N selection, but these options are not evaluated. The paper should clarify how the proposed reward model could be used to personalize policies for many users without excessive computational cost.

Questions for authors
  1. How are differences in reward scale handled across experts?

    Are reward differences comparable in scale across experts? Is any normalization or calibration applied?

  2. How would GRPO-based personalization extend to individual users?

    Appendix K.1 reports GRPO results for two target attributes. Would adapting this approach to individual users require a separate policy for each user, or could a shared policy accommodate different preferences? How would training and storage costs scale with the number of users?

4 comments
AuthorsOpenReview ↗

Response to Reviewer aJ4D (1/N)

Thank you for your constructive feedback! Below, we will address your concerns in detail, with clarifications and new experimental results. To save you time, we also provide a TL;DR with a short response to each of your questions.

We are still incorporating the new results and discussions into the revised manuscript, which we plan to upload by November 15. Once it is available, we will post a general response to all reviewers summarizing the revisions.

TL;DR

[W1: Lower universal preference accuracy] The trade-off of a 1.1–2.7 percentage-point drop in universal preference accuracy for a 38.4-point personalization gain is worthwhile for our target use cases.

[W2/Q1: Scale difference across expert scores] The scale difference in sparse MoE is minor and does not negatively affect performance.

[W3/Q2: Efficient Policy Personalization] We evaluate two training-free policy personalization methods using the adapted sparse MoE reward model and show that they can effectively guide personalized policy generation without additional training and with minimal per-user storage requirements.

Below, we provide a more detailed response to each of your question.

W1: Lower universal preference accuracy

We acknowledge that sparse MoE sacrifices some accuracy in universal preference modeling, with drops of 2.7 and 1.1 percentage points relative to the strongest baseline on SHP and 700K, respectively. This is however expected given the additional constraints and is a common trade-off in prior interpretability work (e.g., Shu et al., 2025; Karvonen et al., 2025). For our target use case of personalized preference modeling, we believe this trade-off is worthwhile. With only a 1.1 point accuracy drop on 700K (Section 6.1), sparse MoE achieves a 38.4-point improvement on RPR after personalization, compared with just 13.3 and 15.4 points for vanilla MoE and MiCRo, respectively (Section 6.2). Moreover, sparse MoE offers greater interpretability for model auditing. We therefore do not consider this a critical weakness of sparse MoE.

(Please note that we now use a basic fine-tuning method to update router during adaptation and achieve better personalization results than Hedge algorithm used in the current paper. We kindly refer you to our response to Reviewer nq5U if you would like to see the updated results.)

W2/Q1: Scale difference across expert scores

We do not apply an explicit normalization, because scale differences across experts are small and do not affect performance. On the 700K test set, the sparse MoE's per-expert mean absolute score difference lies in [1.60, 3.16] and its std in [2.09, 4.21], a 2.0× spread on both measures. Scale matters even less with sparse routing. Within a prompt, routing weights differ across experts by orders of magnitude, far more than the 2× spread in score scale. The prediction therefore depends mainly on which experts are activated, not on their score scales.

To test this directly, we retrained the sparse MoE with z-normalized expert scores before the weighted sum. Normalization removes the scale differences (2.0× → 1.14× for the mean absolute score difference, 2.0× → 1.17× for the std) and gives similar 700K accuracy (84.2 vs. 83.8 without normalization). However, personalization becomes worse: average adapted accuracy drops from 84.4 to 78.6, and the personalization gain from +38.4 to +32.5. We therefore keep the model without normalization.

AuthorsOpenReview ↗

Response to Reviewer aJ4D (2/N)

W3/Q2: Efficient policy personalization

We evaluate two efficient, training-free methods for using personalized sparse MoE reward models to guide policy generation, and compare their computational costs with per-user policy fine-tuning. We use Llama-3.2-3B-Instruct as the policy model and evaluate win rates using Qwen3.6-27B as the judge, following the protocol in Appendix K.

  • Best-of-N (BoN-16; Ichihara et al., 2025): A generate-then-select method that generates 16 candidate responses and uses the adapted reward model to select the best one.

  • Chunk-level reward-guided decoding (RGD; Mudgal et al., 2024): A guided decoding method that samples 4 candidate continuations of 64 tokens each and selects the one preferred by the reward model.

As shown in Table 1, both methods effectively steer the policy toward target preferences, achieving higher win rates than GRPO. Table 2 further shows their efficiency when adapting to new users: neither method requires additional training, and only the adapted MoE router needs to be stored for each new user. In contrast, GRPO requires training and storing a separate LoRA adapter for each user. The trade-off is slightly higher inference latency and memory usage for BoN-16 and RGD. These results suggest that personalized sparse MoE reward models can guide policy generation effectively without per-user policy training, offering a more storage- and training-efficient alternative to GRPO at the cost of increased inference overhead.

Table 1: Win rate against original policy.

Creativity & originalityScientific rigor
BoN-1671.98%69.95%
RGD61.58%57.07%
GRPO56.70%54.12%

Table 2: Per-user computational costs on an H100 GPU. For a fair comparison, we generate one example at a time, allowing batching only within each example for candidate generation and scoring in BoN-16 and RGD.

BoN-16RGDGRPO
New parameters0.40M (RM router)0.40M (RM router)48.6M (LoRA r = 32)
Training costNoneNone3.6–3.9 H100-hours (1000 GRPO steps)
Inference time per example8.2s7.2s6.8s
Inference memory per example8.3GB (generate) / 9.7GB (score)12.9GB6.2GB

References

  • Shu et al. (2025). A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models. Findings of EMNLP

  • Karvonen et al. (2025). SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. ICML

  • Ichihara et al. (2025). Evaluation of Best-of-N Sampling Strategies for Language Model Alignment. TMLR

  • Mudgal et al. (2024). Controlled Decoding from Language Models. ICML

Dear Reviewer aJ4D,

As the discussion phase will close in less than 48 hours, we would greatly appreciate it if you could take a moment to review our response and let us know whether our clarifications and additional experimental results have addressed your concerns. We would be more than happy to provide further clarification or address any remaining questions.

Thank you for the detailed responses and additional experiments. The normalization experiment and the Best-of-N/RGD results address my concerns.

I find the new analyses useful (reviewer nq5U). The results suggest that more interpretable experts can be combined more effectively. However, I am still not sure what interpretability adds beyond expert diversity and specialization. Since the method largely builds on MiCRo, I think a clearer understanding of this point is important to the paper’s contribution.

Therefore, I am keeping my score at 2.

Reviewer_nq5UOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the interpretability and composability of latent experts in Mixture-of-Experts (MoE) reward models for personalized preference learning. The authors propose evaluating learned expert structure along three dimensions: semantic coherence, functional correspondence, and composability. They find that conventional MoE reward models tend to learn entangled experts with nearly uniform routing and highly correlated predictions.

To address this issue, the authors introduce three standard regularization terms—local routing sparsity, global expert balance, and expert diversity—into the MoE reward-model training objective. The resulting sparse MoE produces more semantically coherent routing patterns and functionally specialized experts without requiring explicit expert-level supervision.

Experiments on controlled and real-world preference datasets demonstrate improvements in expert purity, semantic description fidelity, and functional specialization. The authors further validate expert behavior through targeted routing interventions. Finally, they show that adapting the router with a small number of preference comparisons substantially improves personalization performance over conventional MoE baselines.

Critical strengths
  1. A well-motivated evaluation framework for interpreting latent preference components.

The proposed three-rung evaluation framework—semantic coherence, functional correspondence, and composability—is a valuable contribution. Rather than relying solely on semantic descriptions or routing visualizations, the paper investigates whether the learned interpretations correspond to distinct expert behaviors and whether these experts can support downstream personalization. In particular, the targeted routing interventions provide useful functional evidence beyond correlational interpretability metrics.

  1. Strong empirical evidence that simple regularization can improve expert specialization and few-shot personalization.

The paper systematically compares different MoE training objectives while keeping the architecture and training data fixed. The experimental results demonstrate substantial improvements in expert purity and functional specialization using standard sparsity and diversity regularizers. The personalization results are also promising: on RPR, sparse MoE achieves 72.10% accuracy after adaptation with 50 examples, compared with 53.47% for vanilla MoE. The additional experiments across backbones and datasets further support the practical relevance of the findings.

Critical weaknesses
  1. Insufficient explanation for the substantial performance improvement in few-shot personalization.

The sparse MoE exhibits slightly lower reward-model accuracy than vanilla MoE on the original preference-700K dataset (83.78% vs. 84.82%), yet achieves a substantially larger improvement during few-shot personalization on RPR (+25.81 vs. +9.57 percentage points), where only the router is updated using 50 preference comparisons. Although these results are not inherently contradictory, the mechanism behind such a large difference remains unclear.

Since the experts are frozen during adaptation, the achievable performance is constrained by their pretrained reward functions. It is unclear whether the improvement comes from stronger individual experts on the target preference attributes, greater diversity among expert predictions, or more effective router adaptation. The paper lacks a systematic analysis of individual expert performance on RPR and how adaptation changes the routing distributions. Such analysis is important for explaining why a model with slightly weaker universal preference prediction can achieve substantially better personalization through router-only updates.

  1. The experiments do not sufficiently establish the composability of interpretable experts.

The paper claims that disentangled and interpretable experts can be recombined to represent unseen preferences. However, Experiment 4 evaluates adaptation to individual RPR attributes independently, which does not necessarily demonstrate expert composability. The observed improvement could be explained by selecting or upweighting a single expert that already performs well on the target attribute, rather than composing multiple experts to represent a new preference function. A best-single-expert selection baseline using the same adaptation examples would help distinguish these explanations.

Furthermore, the relationship between interpretability and personalization remains insufficiently established. The proposed regularization simultaneously encourages routing sparsity, expert diversity, and semantic coherence. Therefore, improved adaptation might result primarily from greater functional diversity rather than interpretability itself. Although the paper includes ablations of these regularizers, their individual contributions to downstream personalization are not evaluated. Additional adaptation experiments with different regularization configurations would help substantiate the claimed connection between interpretability and composability.

Questions for authors

No.

5 comments
AuthorsOpenReview ↗

Response to Reviewer nq5U (1/N)

Thank you for your constructive feedback! Below, we address your concerns in detail, with a brief TL;DR for each question. We are still incorporating new results and discussions into the revised manuscript, which we plan to upload by November 15. We will then post a general response to all reviewers summarizing the revisions.

TL;DR

[W1: Mechanisms behind sparse MoE's personalization gains] Per-expert accuracy and routing adaptation results show that diversity and specialization lead to higher accuracy for the best-performing expert on each target attribute, and sparse MoE also learns more effective routing during adaptation.

[W2a: Evidence for composability] We provide evidence for composability with new baseline and analyses: sparse MoE outperforms single-best-expert after adaptation, and its updated routing distributions demonstrate composability patterns.

[W2b: Relationship between interpretability and personalization] We run additional ablation study and correlation analyses that show expert diversity is the strongest predictor of personalization gains, while description fidelity, specialization, and diversity are all correlated with composability.

Below, we provide a more detailed response to each of your question.

W1: Mechanisms behind sparse MoE's personalization gains

First, while analyzing the mechanisms behind personalization in MoE reward models, we realized that the Hedge algorithm used in our paper does not fully exploit the personalization potential of MoE models. In fact, simply fine-tuning the router using the BT loss (Eq. 3) yields better personalization performance across all three MoE variants (see Table 2 below). We will therefore report the router fine-tuning results as our main personalization results in the revised paper and focus on them in the discussion below. We thank you again for your constructive feedback, which has helped us significantly strengthen the paper!

Table 1 below reports the accuracy of each sparse MoE expert on each target attribute, along with changes in routing weights before and after adaptation to the target attribute set. We will also include the corresponding tables for vanilla MoE and MiCRo in the revised paper.

Thanks to its greater expert diversity and specialization, sparsex MoE achieves substantially higher accuracy for its best-performing expert on each target attribute than vanilla MoE and MiCRo (83.8 vs. 63.5/63.1 on average across attributes), providing a stronger foundation for adaptation. Furthermore, sparse MoE learns more effective routing weights, assigning higher weights to the best-performing experts (0.45 vs. 0.40/0.35) and achieving a stronger Spearman correlation between routing weights and expert accuracy (0.82 vs. 0.27/0.54). We believe this could be due to the larger performance differences among sparse MoE experts, which make it easier for the router to identify and prioritize the most suitable experts during adaptation.

AuthorsOpenReview ↗

Response to Reviewer nq5U (2/N)

Table 1: Expert accuracy and routing weights before and after adaptation.

ExpertLinguistic creativityUser-friendlinessHumorScientific rigorNarrativeCreativity & originalityFactual accuracyMean
e068(.02→.06)29(.02→.00)73(.03→.04)43(.02→.00)65(.02→.00)81(.02→.10)48(.02→.00)58(.02→.03)
e113(.03→.00)49(.03→.00)10(.03→.00)83(.02→.17)29(.02→.00)10(.03→.00)93(.03→.52)41(.03→.10)
e225(.14→.00)35(.14→.00)17(.15→.00)82(.19→.31)31(.16→.00)18(.14→.00)83(.12→.19)42(.15→.07)
e337(.01→.00)67(.01→.51)62(.02→.00)55(.01→.00)43(.01→.00)33(.01→.00)69(.01→.00)52(.01→.07)
e455(.04→.00)69(.04→.16)77(.04→.00)37(.04→.00)60(.04→.00)43(.05→.00)37(.05→.00)54(.04→.02)
e531(.04→.00)46(.04→.00)32(.04→.00)75(.04→.00)39(.04→.00)22(.05→.00)76(.05→.15)46(.04→.02)
e621(.03→.00)65(.03→.05)23(.02→.00)56(.03→.02)22(.02→.00)17(.03→.00)70(.03→.00)39(.03→.01)
e753(.04→.00)17(.03→.00)30(.04→.00)62(.04→.01)45(.04→.00)42(.04→.00)62(.04→.00)44(.04→.00)
e819(.07→.00)26(.05→.00)25(.07→.00)76(.05→.23)29(.05→.00)28(.05→.00)80(.04→.14)40(.05→.05)
e938(.06→.00)38(.07→.00)32(.05→.00)61(.07→.02)41(.06→.00)47(.07→.00)63(.07→.00)46(.06→.00)
e1073(.04→.19)45(.03→.00)64(.04→.00)38(.03→.00)75(.02→.06)64(.04→.11)37(.03→.00)57(.03→.05)
e1153(.04→.03)64(.04→.19)85(.04→.14)35(.04→.00)66(.04→.00)53(.04→.00)46(.03→.00)57(.04→.05)
e1243(.28→.00)45(.28→.00)31(.20→.00)74(.22→.13)41(.26→.00)26(.21→.00)73(.23→.00)48(.24→.02)
e1324(.03→.00)53(.03→.07)38(.04→.00)60(.03→.00)35(.03→.00)28(.03→.00)73(.04→.00)44(.03→.01)
e1449(.01→.00)47(.01→.00)50(.01→.00)56(.01→.00)60(.01→.00)39(.01→.00)44(.01→.00)49(.01→.00)
e1588(.00→.64)34(.01→.03)93(.00→.82)29(.00→.00)71(.01→.75)86(.00→.78)24(.00→.00)61(.00→.43)
e1653(.05→.00)48(.06→.00)25(.05→.00)49(.06→.00)62(.06→.01)50(.07→.00)44(.07→.00)47(.06→.00)
e1720(.04→.00)40(.04→.00)13(.04→.00)65(.04→.10)43(.04→.00)21(.05→.00)55(.05→.00)37(.04→.01)
e1857(.03→.00)33(.03→.00)25(.04→.00)60(.04→.00)55(.04→.02)54(.04→.00)55(.04→.00)48(.04→.00)
e1971(.02→.07)35(.03→.00)51(.03→.00)38(.03→.00)71(.03→.16)54(.03→.01)31(.04→.00)50(.03→.04)
Best expert (acc)e15 (87.5)e4 (68.5)e15 (92.9)e1 (83.3)e10 (75.0)e15 (86.1)e1 (93.0)83.8
Weight on best expert.00→.64.04→.16.00→.82.02→.17.02→.06.00→.78.03→.52.02→.45
Spearman(weight, accuracy)-0.28→+0.93-0.14→+0.62-0.36→+0.94+0.48→+0.73-0.25→+0.79-0.23→+0.88+0.34→+0.87-0.06→+0.82
Effective # experts12.8→2.313.1→3.114.8→1.713.1→3.513.2→1.914.3→1.814.5→2.013.7→2.3
RM accuracy31.7→87.240.4→73.028.6→94.875.0→84.138.7→73.829.2→87.578.9→90.646.1→84.4
AuthorsOpenReview ↗

Response to Reviewer nq5U (3/N)

W2a: Evidence for composability

We provide two pieces of evidence for composability with new baseline and routing distribution analyses

First, as shown in Table 2 below, fine-tuning the sparse MoE router outperforms the best-single-expert selection baseline (84.4 vs. 79.9). In contrast, the same adaptation does not outperform the best-single-expert baselines for vanilla MoE or MiCRo. This suggests that sparse MoE's performance gains cannot be explained by simply selecting the best-performing expert on the adaptation set.

Second, the routing distributions after adaptation reveal several composability patterns (see Table 1): (1) Multiple experts contribute to each attribute, with an average of 2.3 effective experts. (2) Related attributes share experts but use different combinations. For example, e15 is the dominant expert for all four creative attributes (64–82% of the routing weight), paired with e10 and e0 for creativity, e10 and e19 for linguistic creativity, e19 for narrative, and e11 for humor. Similarly, e1, e2, and e8 are shared by scientific rigor and factual accuracy. (3) These expert combinations matter for performance. Removing a shared expert reduces accuracy by 7.8 points (compared with 0.4 points for vanilla MoE), while replacing an attribute's partner experts with those of another attribute reduces accuracy in 64% of cases (compared with 50% for vanilla MoE).

Together, these results suggest that sparse MoE adapts by recombining a small set of complementary experts, with related attributes sharing experts but relying on different combinations, rather than simply selecting a single expert. We will include these results and discussions in the revised paper.

Table 2: Personalization results of sparse MoE on RPR.

MethodLinguistic creativityUser-friendlinessHumorScientific rigorNarrativeCreativity & originalityFactual accuracyOverall
Selected single expert76.0 (+44.2)67.4 (+27.0)90.1 (+61.5)78.6 (+3.6)71.2 (+32.5)86.1 (+56.9)89.7 (+10.8)79.9 (+33.8)
Hedge (paper)74.0 (+43.3)55.1 (+14.6)80.6 (+50.8)82.5 (+7.5)68.8 (+28.8)60.2 (+31.0)83.6 (+4.7)72.1 (+25.8)
Router fine-tuning87.2 (+55.4)73.0 (+32.6)94.8 (+66.3)84.1 (+9.1)73.8 (+35.0)87.5 (+58.3)90.6 (+11.7)84.4 (+38.4)

W2b: Relationship between interpretability and personalization

Following your suggestion, we run an ablation study over the coefficients of all regularization terms and then adapt each model to the RPR attributes. For each model, we measure description fidelity, expert specialization, and absolute expert correlation on 700K. We then examine how these measures correlate with two adaptation outcomes: personalization gain, measured as the improvement over the unadapted reward model, and composition gain, measured as the improvement over the best single expert selected using the same 50 preference pairs.

The Spearman correlations (ρ\rho) for description fidelity / specialization / absolute expert correlation are −0.03 / +0.42 / −0.86 with personalization gain and +0.44 / +0.64 / −0.46 with composition gain.

These results suggest that expert diversity is the strongest predictor of personalization gains. This is because greater diversity leads to stronger individual experts for different target attributes, providing a better foundation for adaptation. Specialization also shows a moderate positive correlation, while fidelity does not meaningfully predict personalization gains.

For composition gain, all three measures help predict how much the adapted router improves beyond the best single expert. Both description fidelity and specialization correlate positively with composition gain, and these relationships remain strong after controlling for expert correlation (partial ρ=+0.57/+0.59\rho = +0.57 / +0.59, respectively). The correlation between expert correlation and composition gain weakens to −0.38 after controlling for fidelity and specialization.

Our interpretation is that diversity gives the experts distinct preference directions, while interpretability (measured by fidelity) aligns those directions with coherent concepts, so a new preference can be expressed as a sparse mixture of a few experts. Without that alignment, the experts can be arbitrary rotations of the same space, and matching a new preference would require a dense mixture that 50 examples cannot identify. We will add the full table to the appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all Reviewers

We thank all reviewers for their constructive feedback. As promised in our individual responses, we have uploaded a revised paper that integrates the new results and expanded discussions. Below we first summarize the major revisions and then map each reviewer's concerns to the corresponding changes or clarifications.

Summary of major revisions

  1. Router fine-tuning as the adaptation method (Section 6.2; Appendices D and H). We now perform a basic router fine-tune for adaptation with the training objective (Eq. 3), which outperforms Hedge for all MoE models. All personalization results are updated accordingly.
  2. Single-best-expert baseline and evidence for composability (Sections 6.2 and 6.3; Tables 5, 9, and 10). Sparse MoE outperforms its best single expert selected on the same adaptation examples, and its adapted routing demonstrates compositional patterns by combining a few shared and attribute-specific experts in a structured way.
  3. Mechanisms behind sparse MoE's adaptation (Section 6.3; Appendix J). We report per-expert accuracy on each target attribute and routing weights before and after adaptation for all three MoE models. Sparse MoE benefits from both strong individual experts and better routing learned during adaptation.
  4. Roles of interpretability, specialization, and diversity in personalization (Section 6.3; Appendix K). We adapt models from all 27 regularization configurations and relate these properties to adaptation and composition gains. Expert diversity shows the strongest correlation with adaptation gains, while all three measures are correlated with composition gains.
  5. Training-free policy personalization (Section 6.2; Appendix M.1). We evaluate Best-of-N and reward-guided decoding on a shared policy and compare their per-user cost with GRPO.
  6. Scale of expert scores (Appendix L). We quantify scale differences across experts and compare with z-normalized expert scores.
  7. Positioning and limitations (Sections 1 and 2; Appendix N). We clarify the gap our study fills and discuss the accuracy–interpretability trade-off and the domain-level expert structure.

How we addressed each reviewer's concerns

Reviewer nq5U

ConcernRevision / Clarification
Mechanism behind the personalization gainSection 6.3 and Appendix J show that sparse MoE has much stronger best experts on the target attributes and learns routing that better aligns with expert accuracy.
Composability vs. selecting a single expertSparse MoE outperforms the new single-best-expert baseline (Tables 5, 9, 10). Its adapted routing combines 2–3 experts per attribute, and disrupting these combinations hurts accuracy (Section 6.3).
Interpretability vs. diversitySpecialization and diversity provide strong experts for personalization, while interpretability aligns these experts with coherent concepts for easy composition (Appendix K).

Reviewer aJ4D

ConcernRevision / Clarification
Lower universal preference accuracyAppendix N discusses this trade-off: a ~1-point drop on 700K for a much larger gain of 38.4 point in personalization (Table 5).
Scale of expert reward differencesAppendix L shows that scale differences are small relative to routing differences, and z-normalizing expert scores gives similar 700K accuracy but lowers personalization.
Policy personalization for many usersAppendix M.1 shows that training-free methods steer a shared policy more effectively than per-user GRPO, while adding only a small router per user and no training.

Reviewer brpe

ConcernRevision / Clarification
Limited motivation and new insightSection 2 clarifies the gap: prior work has not compared how components with different degrees of interpretability compose for personalization. The new Section 6.3 explains why interpretable, specialized, and diverse experts personalize better.
Experts reflect task domains rather than preferencesAppendix N discusses why domain-level specialization is still meaningful for personalization, as supported by the RPR results, and notes that the framework extends to response-level interpretations.

We hope these revisions and clarifications address the reviewers' concerns, and we are happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.