- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces RM-OPD, a two-stage framework for training general-purpose generative reward models by integrating domain-specific judgment capabilities. The framework first trains specialist teachers in mathematics, code, and safety through single-domain reinforcement learning, then consolidates their capabilities into a unified student through routed on-policy distillation. Domain contribution reweighting addresses supervision imbalance caused by varying judgment lengths, while optional criteria context distillation incorporates domain-specific evaluation principles into teacher guidance. Experiments on eight benchmark suites using two 7B backbones show consistent aggregate improvements over mixed-domain GRPO, with the student matching or exceeding specialist teachers in several settings. Ablation studies further demonstrate the benefits of domain reweighting and criteria conditioning.
Critical strengths
-
The improvements over backbone-matched MixRL-GRPO are reasonably consistent.**
In Table 7, the macro average across 16 subsets increases from 71.45 to 75.41 for Qwen and from 72.72 to 76.06 for DeepSeek, corresponding to gains of 3.96 and 3.34 percentage points. Improvements under both initializations provide meaningful evidence of effectiveness. -
After training, a single student performs judgment without consulting the three specialist teachers or receiving the training-time domain labels and additional criteria contexts. This is attractive for applications requiring a unified judge.
-
The ablations provide direct evidence for the additional components. Removing DCR reduces average code and safety accuracy by 3.43 and 2.48 percentage points, respectively. Adding CCD improves overall accuracy across 12 subsets by 1.70 percentage points. These results support the practical utility of both components.
Critical weaknesses
-
Section 5.1 explicitly builds on existing multi-teacher on-policy distillation, CCD follows context distillation, and DCR primarily reweights losses using domain token shares. The paper should clarify which technical difficulties arise specifically in reward modeling and why the proposed adaptations are necessary. Comparisons with basic MOPD, offline teacher-trajectory distillation, and simple domain-averaged losses under the same teachers, data, and budget would help establish the contribution.
-
RM-OPD first trains three specialist teachers and then trains a student while obtaining teacher token probabilities. Figure 1 measures convergence using student training steps, excluding teacher training and differences in per-step cost. Matching the backbone and training data does not establish compute fairness. The paper should report total GPU hours, generated tokens, teacher-scoring costs, and performance under matched total budgets.
-
Equation 3 explains how gradient conflicts can cause capability degradation, but distillation still jointly optimizes multiple domain objectives through shared parameters and can also exhibit conflicting gradients. Accuracy improvements demonstrate better outcomes without directly establishing reduced interference. Measurements of domain-gradient cosine similarities, conflict frequencies, and norms, together with conflict-aware or domain-balanced Mix-RL baselines, would strengthen this claim.
-
A teacher assigning higher probability to a student token does not guarantee that the corresponding reasoning step is correct. Appendix E.3 illustrates this limitation: the student correctly identifies the failure on
digits(1), but incorrectly states during its reasoning that the candidate returns 1 fordigits(11); the provided code also returns 0 for that input. Verdict accuracy and reasoning reliability should be evaluated separately. Systematic analysis of reasoning errors, inherited teacher mistakes, and explanations unrelated to the verdict is needed.
Questions for authors
n/a
Thank you for addressing my concerns. I would be happy to see this paper accepted.