- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether Mixture-of-Experts (MoE) reward models can learn interpretable and composable latent preference structures. Building upon an existing MoE reward model architecture, the authors introduce three standard regularization techniques into the training objective: local routing sparsity, global expert load balancing, and expert output diversity, aiming to encourage functional specialization among experts. The paper then evaluates the learned expert structures along three dimensions: Semantic Coherence, which measures whether experts correspond to distinct semantic categories; Functional Correspondence, which examines whether experts exhibit stronger predictive performance in their associated categories and whether routing interventions induce the expected preference changes; and Composability, which assesses whether personalized preference modeling can be achieved through few-shot router adaptation.
Critical strengths
- Practically relevant research question. The paper goes beyond the predictive performance of MoE reward models to investigate whether individual experts exhibit meaningful semantic and functional specialization. This question is relevant to the development of interpretable and personalized reward models.
- Systematic experimental evaluation. The authors evaluate the learned expert structures along three dimensions,semantic coherence, functional correspondence, and composability,using controlled datasets, real-world preference data, human evaluations, and routing interventions. These experiments provide evidence from multiple perspectives.
- Practical improvements. Although the proposed approach relies on existing regularization techniques, it achieves substantial improvements in expert specialization and few-shot personalization while keeping the model architecture and training data largely unchanged. This suggests that relatively simple modifications to the training objective can yield practical benefits.
Critical weaknesses
- Insufficient research motivation and limited practical significance. From the perspective of research motivation, I find the proposed evaluation of preference entanglement in MoE models to be of limited practical value. The observation that entangled experts can negatively affect model capabilities is already well understood, and the paper largely reconfirms this phenomenon through empirical evaluation. Moreover, the proposed approach relies on standard regularization techniques rather than introducing a new mechanism to address the underlying issue. The paper also does not provide a sufficiently deep analysis of the interpretability mechanisms behind expert entanglement and specialization. Consequently, the work appears to offer limited new insights beyond established observations.
- Mismatch between the stated motivation and the learned expert representations. The learned expert structures primarily reflect task domains rather than human values or preference dimensions, which is inconsistent with the problem formulation presented in the introduction. Although the paper motivates its approach through heterogeneous human values and preference modeling, the identified experts mainly correspond to tasks or domains such as mathematics, text summarization, legal consultation, and creative writing (Tables 2 and 3). These characteristics largely capture the semantic categories of input prompts rather than differences in users' underlying values or preferences. Therefore, the experimental findings do not sufficiently support the paper's broader claims about modeling heterogeneous human preferences.
Questions for authors
See the Weaknesses
3 comments
Response to Reviewer brpe (2/N)
W2: Domain-level expert structure
We agree and have acknowledged in the limitations that experts in the current MoE reward model captures prompt-identifiable features, such as task or domain, rather than response-level features. This is due to the design of prompt-level routing.
However, we would like to argue that task- and domain-level structure is itself meaningful for individual preferences. In realistic settings, preferences are often conditioned on domain context. One user may want precise judgments on safety- or factuality-sensitive prompts but care little about fine-grained differences on coding or entertainment prompts. Domain-specialized experts are natural components for such preferences: a user's preference can be expressed by re-weighting the experts responsible for the contexts that user cares about. Our experiments in Section 6.2 also empirically support the usefulness of such structures in personalization: even though the target preferences are specific response-level attributes, sparse MoE can still effectively adapt to them.
Besides, the studied architecture also supports different types of interpretation. Our study focuses on prompt-level features because they allow a controlled evaluation (category recovery and expert purity against known categories). The framework, however, is not tied to this level. Wang et al. (2026), for example, build on our work and show that by simply changing where to apply the interpretability regularization, the MoE reward models can also learn response-level preferences, such as preferring "blunt, sardonic humor rather than polite, measured replies".
References
- Wang et al. (2026). Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast. Preprint
The contribution is limited.
I understand that this is an empirical study. However, I find its significance limited, as models continue to evolve, and the interpretability properties identified here are not guaranteed to hold in future models. Moreover, the finding that more interpretable and specialized experts can be combined more effectively to model new preferences largely reflects an existing consensus. While the paper provides detailed empirical evidence, I do not believe this finding, in itself, substantially advances the field. I therefore maintain my original score.
Response to Reviewer brpe (1/N)
Thank you for your review! Below, we address your concerns in detail, with a brief TL;DR for each question. We are still incorporating new results and discussions into the revised manuscript, which we plan to upload by November 15. We will then post a general response to all reviewers summarizing the revisions.
TL;DR
[W1: Weak contribution] We are the first controlled study to establish interpretability in MoE reward models and relate expert disentanglement to personalization ability in reward modeling. We provide carefully designed, architecture-specific interpretability analyses for MoE reward models, rather than proposing new methods. We further add new analyses of the mechanisms behind sparse MoE's personalization gains and the relationship between interpretability and personalization.
[W2: Domain-level expert structure] Domain-level structure is also meaningful for individual preferences and the personalization experiments support this empirically; the studied structure also supports capturing response-level preferences with minor changes.
Below, we provide a more detailed response to each of your question.
W1: Weak contribution
We respectfully disagree with your assessment of our contribution. Our work is a controlled empirical study, not a methods paper: it examines interpretability, personalization, and the connection between them in MoE reward models, using analyses built around their router–expert structure.
(1) "Entangled experts hurt model capabilities" is established in LLM MoEs, while its validity has not been systematically studied in reward modeling. In fact, our experiments suggest more nuanced findings: entangled experts (vanilla MoE and MiCRo) achieve better universal preference modeling accuracy than disentangled ones (sparse MoE) on SHP and 700K (Sections 5.1/6.1). They fall short in interpretability and personalization (Sections 5.1//6.1/6.2).
Prior latent-component reward models have mainly focused on decomposing reward functions. However, to the best of our knowledge, they have not systematically compared the composability and personalization capabilities of learned latent components across different levels of disentanglement. Our work is therefore the first to show that more interpretable and specialized experts can be composed more effectively to model new preferences. Although this finding may be somewhat expected, we believe the systematic analysis and empirical validation still provide value to the research community.
(2) Standard regularizers are a deliberate design choice, not a limitation. As stated in Section 3, we intentionally keep the architecture of MiCRo and only add established sparsity, balance, and diversity regularizers. Because the manipulation is minimal and regularizers are standard, the structure that emerges can be attributed to the objective itself rather than to a particular new mechanism. Our goal is not to propose a new regularizer. It is to establish, as a positive and reproducible case, that MoE reward models trained only on binary preferences can learn interpretable, specialized experts, and that this structure makes them better personalizers. Future, more specialized methods can build on this foundation.
(3) Our analysis is carefully tailored to the router–expert structure of MoE reward models, going beyond generic interpretability analyses. For example, instead of using only generic interpretability tools such as feature descriptions from top-activated examples, We instead consider both the router and the conduct comprehensive interpretability analyses: semantic coherence of routing (Section 5.1/6.1), correlational and interventional correspondence between routing and expert specialization (Section 5.1/5.2/6.1), and how router upweights semantically related experts during adaptation (Table 15). We believe these analyses provide useful insights for researchers working with similar architectures.
(4) New mechanism analyses. Following Reviewer nq5U, we add analyses of why sparse MoE experts personalize and compose better. We find that expert specialization and diversity lead to higher accuracy for the best-performing expert on each target attribute, and larger accuracy differences across experts make it easier to learn an effective router during adaptation. Beyond specialization and diversity, we also find that interpretability, measured by description fidelity, predicts composition gains, i.e., how much the adapted reward model improves over its best single expert. One possible explanation is that interpretability reflects how well specialized and diverse experts align with coherent concepts, allowing new preferences to be modeled as sparse mixtures of a few experts that can be learned with limited adaptation data. We hope these findings provide further insights to the research community. For more details on this analysis, we kindly refer you to our response to Reviewer nq5U.