acceptodds
Under review as a conference paper at ICLR 2027

RM-OPD: On-Policy Distillation for Generative Reward Models

Abstract

Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks:the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; Then we consolidate these teachers into one student model through (i) routed on-policy distillation, which provides dense signed token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context distillation, which conditions teachers on domain-specific evaluation criteria to sharpen token-level supervision. Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect.These results establish RM-OPD as a promising paradigm for building generalist GRMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Generative Reward Models, On-Policy Distillation, Reward modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1LsyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RM-OPD, a two-stage framework for training general-purpose generative reward models by integrating domain-specific judgment capabilities. The framework first trains specialist teachers in mathematics, code, and safety through single-domain reinforcement learning, then consolidates their capabilities into a unified student through routed on-policy distillation. Domain contribution reweighting addresses supervision imbalance caused by varying judgment lengths, while optional criteria context distillation incorporates domain-specific evaluation principles into teacher guidance. Experiments on eight benchmark suites using two 7B backbones show consistent aggregate improvements over mixed-domain GRPO, with the student matching or exceeding specialist teachers in several settings. Ablation studies further demonstrate the benefits of domain reweighting and criteria conditioning.

Critical strengths
  1. The improvements over backbone-matched MixRL-GRPO are reasonably consistent.**
    In Table 7, the macro average across 16 subsets increases from 71.45 to 75.41 for Qwen and from 72.72 to 76.06 for DeepSeek, corresponding to gains of 3.96 and 3.34 percentage points. Improvements under both initializations provide meaningful evidence of effectiveness.

  2. After training, a single student performs judgment without consulting the three specialist teachers or receiving the training-time domain labels and additional criteria contexts. This is attractive for applications requiring a unified judge.

  3. The ablations provide direct evidence for the additional components. Removing DCR reduces average code and safety accuracy by 3.43 and 2.48 percentage points, respectively. Adding CCD improves overall accuracy across 12 subsets by 1.70 percentage points. These results support the practical utility of both components.

Critical weaknesses
  1. Section 5.1 explicitly builds on existing multi-teacher on-policy distillation, CCD follows context distillation, and DCR primarily reweights losses using domain token shares. The paper should clarify which technical difficulties arise specifically in reward modeling and why the proposed adaptations are necessary. Comparisons with basic MOPD, offline teacher-trajectory distillation, and simple domain-averaged losses under the same teachers, data, and budget would help establish the contribution.

  2. RM-OPD first trains three specialist teachers and then trains a student while obtaining teacher token probabilities. Figure 1 measures convergence using student training steps, excluding teacher training and differences in per-step cost. Matching the backbone and training data does not establish compute fairness. The paper should report total GPU hours, generated tokens, teacher-scoring costs, and performance under matched total budgets.

  3. Equation 3 explains how gradient conflicts can cause capability degradation, but distillation still jointly optimizes multiple domain objectives through shared parameters and can also exhibit conflicting gradients. Accuracy improvements demonstrate better outcomes without directly establishing reduced interference. Measurements of domain-gradient cosine similarities, conflict frequencies, and norms, together with conflict-aware or domain-balanced Mix-RL baselines, would strengthen this claim.

  4. A teacher assigning higher probability to a student token does not guarantee that the corresponding reasoning step is correct. Appendix E.3 illustrates this limitation: the student correctly identifies the failure on digits(1), but incorrectly states during its reasoning that the candidate returns 1 for digits(11); the provided code also returns 0 for that input. Verdict accuracy and reasoning reliability should be evaluated separately. Systematic analysis of reasoning errors, inherited teacher mistakes, and explanations unrelated to the verdict is needed.

Questions for authors

n/a

1 comment

Thank you for addressing my concerns. I would be happy to see this paper accepted.

Reviewer_JCWZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RM-OPD, which uses on-policy distillation to consolidate domain-specialized reward teachers in mathematics, code, and safety into a generalist generative reward model. Experiments on multiple reward modeling benchmarks show some improvements over Mix-RL.

Critical strengths
  1. Integrating experts from different domains into a generalist judge has practical value. Under both initializations, the method outperforms Mix-RL with the same initialization and training pool.
Critical weaknesses
  1. The method's additional contributions are unclear. Teacher training and routed on-policy distillation follow MOPD. DCR and Open-MOPD use the same form of weighting when computed on the same batch, with all domains present and matching target domain ratios. The current results do not sufficiently distinguish the gains from directly applying existing methods from those due to the new designs.
  2. The experiments do not sufficiently isolate the contribution of each design, leaving the causal attribution of the performance gains unclear. There is no basic routed OPD baseline with both DCR and CCD disabled, nor targeted comparisons that separate the effects of using domain-specialized teachers and on-policy distillation.
Questions for authors
  1. Apart from how micro-batches and missing domains are handled, how does DCR differ from Open-MOPD? Is w/o CCD in Figure 4 equivalent to routed OPD with static balancing?
  2. How does basic routed OPD perform with both DCR and CCD disabled? How does OPD using a generalist teacher trained on all three domains compare with OPD using domain-specialized teachers?
  3. How does Teacher + criteria perform? The ``teacher-surpassin'' effect is currently observed only in some settings, and the Qwen student's macro-average remains below that of the specialist pool.
Reviewer_yAvrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a real problem: building a generalist Generative Reward Model (GRM) that integrates heterogeneous evaluation capabilities (math, code, safety) without the “seesaw effect” that afflicts mixed-domain RL (Mix-RL). A clean first-order analysis is presented to explain why cross-domain gradient interference plus sparse verdict-only rewards leave Mix-RL below the per-domain ceilings of independently trained specialists. Then the RM-OPD, a two-stage framework, is proposed.

Critical strengths

The derivation from reverse-KL to the clipped signed log-ratio signal is correct and sound. The DCR construction is a sensible and cleanly stated fix to length-driven imbalance.

Critical weaknesses

All results are 7B. The variance-contraction-style theory is absent, and given that teacher quality gaps may shrink at larger scales, the 7B-only scope limits how general the conclusions are.

Questions for authors

How many seeds per configuration? How sensitive are results to the DCR target ratio?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.