- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MatrixReward derives rewards from pairwise comparisons conducted at the rubric level. It integrates task-specific importance weights with weights estimated from the response group, and subsequently transforms each response's distances to ideal rubric profiles into a scalar reward for GRPO training.
Critical strengths
- The method preserves feedback on individual rubrics and provides an explicit procedure for combining it into a training reward.
- The aggregation ablation provides useful evidence. Replacing weighted averaging with the proposed distance-based reward improves the average score from 59.83 to 63.02, while retaining the pairwise feedback and combined weights.
Critical weaknesses
-
The improvements may partly stem from a more accurate estimate of the judge's preferences. Explicit MatrixReward draws on more judge feedback, and the same judge is used to evaluate the models. Under a comparable judging budget, the implicit variant also performs close to RRC-SCR (62.23 vs. 61.79). It therefore remains an open question whether baselines could reach similar gains by averaging repeated scores under the same judging budget.
-
The aggregation ablation provides useful evidence. Experiments with deterministic, multi-objective rewards could further isolate the contribution of reward merging itself. In such a setting, repeatedly scoring the same response yields no additional information, so any gains would more directly support the merging method.
Questions for authors
See Weaknesses 1–2.
4 comments
W2. MatrixReward’s improvement under deterministic feedback
Following your suggestion, we trained Qwen3-8B with deterministic, multi-objective feedback and varied only the rule used to aggregate the constraint scores into a scalar reward. We used open-ended prompts from the IF-RLVR IF_multi_constraints_upto5 dataset, with 2,560 prompts for training, 128 for validation, and 128 for testing. For each rollout, the checkers returned one binary outcome per constraint, producing a rollout-by-constraint matrix. Rechecking the same response yields the same outcomes, eliminating variation from repeated judge calls.
Both training conditions used the same initial policy, checkers, uniform constraint weights, column normalization, and GRPO setup. Weighted aggregation summed the weighted, normalized constraint scores for each rollout. MatrixReward’s distance aggregation took the maximum and minimum of each weighted column within the rollout group to form positive and negative ideal profiles.
We generated one response per held-out test prompt and evaluated it with the same checkers. Mean constraint pass rate gives each prompt equal weight by first computing the fraction of its constraints satisfied. All-pass rate is the percentage of prompts for which the response satisfies every constraint.
Table 3. Constraint following on 128 held-out IF-RLVR prompts.
| Training condition | Mean constraint pass rate | All-pass rate | Prompts passing all constraints |
|---|---|---|---|
| Qwen3-8B | 55.00% | 14.84% | 19/128 |
| Weighted aggregation | 62.97% | 20.31% | 26/128 |
| MatrixReward | 64.15% | 23.44% | 30/128 |
With the verification and weighting rules held fixed, switching from weighted aggregation to MatrixReward increased the mean constraint pass rate by 1.18 percentage points. The model satisfied every constraint on 30 of 128 prompts, up from 26 of 128, a gain of 3.13 percentage points. This comparison isolates the contribution of reward aggregation in a setting where every constraint outcome is deterministic.
Table 4. Transfer of the same policies to four open-ended benchmarks.
| Training condition | AlpacaEval 2 | Arena-Hard v2 | WildBench | WritingBench | Average |
|---|---|---|---|---|---|
| Qwen3-8B | 47.93 | 24.81 | 55.89 | 54.07 | 45.68 |
| Weighted aggregation | 42.36 | 24.39 | 46.85 | 52.64 | 41.56 |
| MatrixReward | 44.69 | 25.02 | 48.03 | 53.80 | 42.89 |
MatrixReward outperformed weighted aggregation on all four transfer benchmarks, averaging 42.89 versus 41.56 (+1.33 points). Both IF-RLVR-trained policies scored below the initial model’s four-benchmark average of 45.68. The decline on benchmarks is consistent with the IFBench authors’ observation that targeted training on verifiable constraints improved instruction following while lowering AlpacaEval 2 performance in their Tülu-DPO experiment. Pyatkin et al., §4.5 and Table 3.This is sufficient to demonstrate that MatrixReward offers greater improvements over weighted aggregation in terms of constraint compliance and open-ended generation capabilities.
Thanks to the authors for the additional experiments, which address my main concerns. I have raised my score accordingly.
We sincerely thank you for the time and effort you have devoted to reviewing our paper and for your constructive suggestions, which have helped us improve the manuscript. We greatly appreciate your careful reassessment and willingness to reconsider your score. Thank you again for your patience, attention to detail, and thoughtful consideration.
Thank you for your constructive feedback. We conducted additional experiments and analyses to address your concerns about the evaluation judge and the amount of judge feedback available during training.
W1.1. MatrixReward's advantage with different training and evaluation judge models
We trained Qwen3-8B with three random seeds for each method, using Qwen3.6-35B-A3B to provide training feedback and Qwen3.8 Flash to evaluate the trained policies on all four benchmarks. Table 1 reports the mean ± sample standard deviation across seeds. For the final column, we first averaged the four benchmark scores within each run and then computed the mean and standard deviation across runs.
Table 1. Qwen3-8B results with different training and evaluation judges.
With a different judge evaluating the trained policies, explicit MatrixReward exceeded RRC-SCR by 2.34 points on the four-benchmark average (4.07% relative improvement), and implicit MatrixReward exceeded it by 1.37 points (2.38%). Both variants achieved a higher four-benchmark average than RRC-SCR in each of the three runs.
W1.2. Improvement of baseline under the same training judge-call budget
We trained Qwen3-8B with two additional controls that match the judge-call budget of explicit MatrixReward. For a query with k rubrics, explicit MatrixReward makes k calls per response pair, one for each rubric. In the two controls, we repeated the original judge request k times per pair under the same decoding settings. Each RRC-SCR request returns one holistic verdict, while each implicit MatrixReward request returns a verdict for every rubric.
For repeated RRC-SCR, we averaged the repeated scores when computing each response’s win rate. For a group of G responses, we summed each response’s points over k judgments against each of its G−1 opponents and divided by k(G−1). This is equivalent to averaging the k scores for each pair and then averaging over opponents.
Table 2. Qwen3-8B results by number of training judge calls per response pair.
Repeating the original request changed the four-benchmark average by −0.18 points for RRC-SCR and +0.04 points for implicit MatrixReward. There is almost no improvement compared to a single call.At the same budget of k calls per pair, assigning each call to a distinct rubric achieved the highest average (59.80), exceeding repeated holistic scoring by 2.52 points and repeated implicit scoring by 0.93 points. This shows that even if the overall evaluation is repeated, the RRC-SCR method cannot achieve further improvement, while MatrixReward can obtain information from each rubric dimension, thereby achieving improvement.