acceptodds
Under review as a conference paper at ICLR 2027

MatrixReward: Reward from Rubric Matrix for Open-Ended Generation

Abstract

Open-ended query generation lacks standard answers, thus necessitating an effective reward mechanism. Pointwise scoring rubrics provide limited information about the relative quality of sample answers under the same prompt; merging multiple rubric judgments into a single score may also mask the differences between these answers. We propose MatrixReward, which constructs rewards from a rollout-by-rubric win-rate matrix obtained by comparing every pair of sampled responses under each rubric. The spread of each matrix column captures how strongly that rubric distinguishes the current rollouts, while correlations between columns reveal rubric repetition; together, these statistics yield data-dependent rubric weights. We combine these weights with the prior weights of rubrics. After column normalization and weighting, the observed per-rubric maxima and minima define positive and negative ideal profiles. Each rollout’s distances to these two ideals determine its relative-closeness quality reward. Evaluated using Qwen3-8B on four open-ended query-answering benchmarks, MatrixReward achieves an average score of 63.02, outperforming the strongest baseline by approximately 2.0%. These results support the idea that matrices derived from relative comparisons can be used to construct rewards more reasonably for open-ended generative reinforcement learning.

Primary area:
reinforcement learning
Keywords:
Open-Ended Generation, Rubric-Based Rewards, Reinforcement Learning, LLM-as-a-Judge

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pRmVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MatrixReward derives rewards from pairwise comparisons conducted at the rubric level. It integrates task-specific importance weights with weights estimated from the response group, and subsequently transforms each response's distances to ideal rubric profiles into a scalar reward for GRPO training.

Critical strengths
  1. The method preserves feedback on individual rubrics and provides an explicit procedure for combining it into a training reward.
  2. The aggregation ablation provides useful evidence. Replacing weighted averaging with the proposed distance-based reward improves the average score from 59.83 to 63.02, while retaining the pairwise feedback and combined weights.
Critical weaknesses
  1. The improvements may partly stem from a more accurate estimate of the judge's preferences. Explicit MatrixReward draws on more judge feedback, and the same judge is used to evaluate the models. Under a comparable judging budget, the implicit variant also performs close to RRC-SCR (62.23 vs. 61.79). It therefore remains an open question whether baselines could reach similar gains by averaging repeated scores under the same judging budget.

  2. The aggregation ablation provides useful evidence. Experiments with deterministic, multi-objective rewards could further isolate the contribution of reward merging itself. In such a setting, repeatedly scoring the same response yields no additional information, so any gains would more directly support the merging method.

Questions for authors

See Weaknesses 1–2.

4 comments

Thank you for your constructive feedback. We conducted additional experiments and analyses to address your concerns about the evaluation judge and the amount of judge feedback available during training.

W1.1. MatrixReward's advantage with different training and evaluation judge models

We trained Qwen3-8B with three random seeds for each method, using Qwen3.6-35B-A3B to provide training feedback and Qwen3.8 Flash to evaluate the trained policies on all four benchmarks. Table 1 reports the mean ± sample standard deviation across seeds. For the final column, we first averaged the four benchmark scores within each run and then computed the mean and standard deviation across runs.

Table 1. Qwen3-8B results with different training and evaluation judges.

MethodAlpacaEval 2Arena-Hard v2WildBenchWritingBenchAverage
MatrixReward (explicit)83.24 ± 1.6639.36 ± 0.8955.86 ± 0.5560.74 ± 0.3259.80 ± 0.38
MatrixReward (implicit)80.52 ± 1.5540.71 ± 0.8755.00 ± 0.2359.10 ± 0.8758.83 ± 0.14
RRC-SCR80.58 ± 0.8337.00 ± 2.1853.99 ± 0.4758.28 ± 0.6757.46 ± 0.75
ArenaRL79.59 ± 1.3434.25 ± 1.3954.32 ± 0.1757.09 ± 0.8256.31 ± 0.13
Tournament-GRPO76.21 ± 0.1332.62 ± 0.4453.87 ± 0.6157.10 ± 1.2954.95 ± 0.49
Pairwise Rank72.39 ± 0.1930.48 ± 0.1656.38 ± 0.6657.55 ± 0.9754.20 ± 0.26
GRPO64.09 ± 0.7929.83 ± 2.5755.76 ± 0.5358.43 ± 0.6452.03 ± 0.62
Dr. GRPO67.00 ± 2.1528.04 ± 2.0753.41 ± 1.0054.73 ± 1.3050.80 ± 0.77
GOPO62.58 ± 0.8329.82 ± 2.7254.79 ± 0.5056.42 ± 0.2450.90 ± 0.51
GDPO65.95 ± 1.1830.29 ± 0.4454.75 ± 0.0957.29 ± 0.2952.07 ± 0.39
GD²PO60.50 ± 1.5225.73 ± 0.7952.23 ± 0.4654.86 ± 0.9648.33 ± 0.61
DAPO65.26 ± 2.4232.41 ± 2.1555.76 ± 0.2457.64 ± 0.7252.77 ± 0.43
SAW67.52 ± 1.3333.67 ± 1.2353.82 ± 2.4259.66 ± 0.4453.67 ± 0.51
POW3R64.97 ± 1.9532.46 ± 1.2953.06 ± 0.1152.41 ± 0.5850.72 ± 0.70
AMRP54.45 ± 0.9330.01 ± 1.1654.05 ± 0.5757.21 ± 0.7048.93 ± 0.82

With a different judge evaluating the trained policies, explicit MatrixReward exceeded RRC-SCR by 2.34 points on the four-benchmark average (4.07% relative improvement), and implicit MatrixReward exceeded it by 1.37 points (2.38%). Both variants achieved a higher four-benchmark average than RRC-SCR in each of the three runs.

W1.2. Improvement of baseline under the same training judge-call budget

We trained Qwen3-8B with two additional controls that match the judge-call budget of explicit MatrixReward. For a query with k rubrics, explicit MatrixReward makes k calls per response pair, one for each rubric. In the two controls, we repeated the original judge request k times per pair under the same decoding settings. Each RRC-SCR request returns one holistic verdict, while each implicit MatrixReward request returns a verdict for every rubric.

For repeated RRC-SCR, we averaged the repeated scores when computing each response’s win rate. For a group of GG responses, we summed each response’s points over k judgments against each of its G−1G-1 opponents and divided by k(G−1)k(G-1). This is equivalent to averaging the k scores for each pair and then averaging over opponents.

Table 2. Qwen3-8B results by number of training judge calls per response pair.

MethodCalls per pairAlpacaEval 2Arena-Hard v2WildBenchWritingBenchAverage
MatrixReward (explicit, one call per rubric)k83.24 ± 1.6639.36 ± 0.8955.86 ± 0.5560.74 ± 0.3259.80 ± 0.38
MatrixReward (implicit)180.52 ± 1.5540.71 ± 0.8755.00 ± 0.2359.10 ± 0.8758.83 ± 0.14
MatrixReward (implicit, k repetitions)k80.80 ± 1.4840.66 ± 0.2855.02 ± 0.2359.00 ± 0.3358.87 ± 0.39
RRC-SCR180.58 ± 0.8337.00 ± 2.1853.99 ± 0.4758.28 ± 0.6757.46 ± 0.75
RRC-SCR (k repetitions)k77.56 ± 0.7137.14 ± 0.4354.14 ± 0.3660.29 ± 0.8957.28 ± 0.50

Repeating the original request changed the four-benchmark average by −0.18 points for RRC-SCR and +0.04 points for implicit MatrixReward. There is almost no improvement compared to a single call.At the same budget of k calls per pair, assigning each call to a distinct rubric achieved the highest average (59.80), exceeding repeated holistic scoring by 2.52 points and repeated implicit scoring by 0.93 points. This shows that even if the overall evaluation is repeated, the RRC-SCR method cannot achieve further improvement, while MatrixReward can obtain information from each rubric dimension, thereby achieving improvement.

W2. MatrixReward’s improvement under deterministic feedback

Following your suggestion, we trained Qwen3-8B with deterministic, multi-objective feedback and varied only the rule used to aggregate the constraint scores into a scalar reward. We used open-ended prompts from the IF-RLVR IF_multi_constraints_upto5 dataset, with 2,560 prompts for training, 128 for validation, and 128 for testing. For each rollout, the checkers returned one binary outcome per constraint, producing a rollout-by-constraint matrix. Rechecking the same response yields the same outcomes, eliminating variation from repeated judge calls.

Both training conditions used the same initial policy, checkers, uniform constraint weights, column normalization, and GRPO setup. Weighted aggregation summed the weighted, normalized constraint scores for each rollout. MatrixReward’s distance aggregation took the maximum and minimum of each weighted column within the rollout group to form positive and negative ideal profiles.

We generated one response per held-out test prompt and evaluated it with the same checkers. Mean constraint pass rate gives each prompt equal weight by first computing the fraction of its constraints satisfied. All-pass rate is the percentage of prompts for which the response satisfies every constraint.

Table 3. Constraint following on 128 held-out IF-RLVR prompts.

Training conditionMean constraint pass rateAll-pass ratePrompts passing all constraints
Qwen3-8B55.00%14.84%19/128
Weighted aggregation62.97%20.31%26/128
MatrixReward64.15%23.44%30/128

With the verification and weighting rules held fixed, switching from weighted aggregation to MatrixReward increased the mean constraint pass rate by 1.18 percentage points. The model satisfied every constraint on 30 of 128 prompts, up from 26 of 128, a gain of 3.13 percentage points. This comparison isolates the contribution of reward aggregation in a setting where every constraint outcome is deterministic.

Table 4. Transfer of the same policies to four open-ended benchmarks.

Training conditionAlpacaEval 2Arena-Hard v2WildBenchWritingBenchAverage
Qwen3-8B47.9324.8155.8954.0745.68
Weighted aggregation42.3624.3946.8552.6441.56
MatrixReward44.6925.0248.0353.8042.89

MatrixReward outperformed weighted aggregation on all four transfer benchmarks, averaging 42.89 versus 41.56 (+1.33 points). Both IF-RLVR-trained policies scored below the initial model’s four-benchmark average of 45.68. The decline on benchmarks is consistent with the IFBench authors’ observation that targeted training on verifiable constraints improved instruction following while lowering AlpacaEval 2 performance in their Tülu-DPO experiment. Pyatkin et al., §4.5 and Table 3.This is sufficient to demonstrate that MatrixReward offers greater improvements over weighted aggregation in terms of constraint compliance and open-ended generation capabilities.

Thanks to the authors for the additional experiments, which address my main concerns. I have raised my score accordingly.

We sincerely thank you for the time and effort you have devoted to reviewing our paper and for your constructive suggestions, which have helped us improve the manuscript. We greatly appreciate your careful reassessment and willingness to reconsider your score. Thank you again for your patience, attention to detail, and thoughtful consideration.

Reviewer_3pMROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the reward construction in open-ended RL tasks. It proposes MatrixReward to handle the issue that it is difficult for existing rubric-based pointwise reward methods to distinguish the quality of answers stablely and simply weighted aggregation may lose multidimentional quality information. It conducts experiments on several benchmarks and outperforms baselines.

Critical strengths
  1. The research topic is important and meaningful. It is challenging to achieve reliable and discriminative reward signals for open-ended generation.

  2. The proposed framework is clearly presented, with reasonable components and detailed implementation descriptions.

  3. The experiments cover multiple baselines and ablation, and provides a analysis of the performance cost trade-off.

Critical weaknesses
  1. The motivation is not justified clearly. The paper argues that pointwise rubric reward models may fail to distinguish answers with different quality levels. However, the reason may be not the pointwise evaluation itself. This limitation may result from insufficiently discriminative reward models, and stronger rubric reward models may provide fine-grained and reliable scores. The paper neither evaluates such models nor quantifies how frequently reward ties occur in practice.

  2. The paper lacks some baselines. It omits closely related rubric reward learning methods like RaR and OpenRS. These comparisons are important for establishing the advantages of MatrixReward over existing rubric-based approaches.

  3. The performance improments are limited compared to judging cost. Compared to Implict and RRC-SCR, the average performance improvements are minor while costing ~3.8x tokens.

  4. The paper needs to conduct more ablation studies. The ablations do not separately isolate the contributions of dispersion, correlation, and reward aggregation under identical normalization settings. Additionally, taking absolute values of the fusion coefficients is not clearly justified by the optimization formulation.

Questions for authors

refer to weakness

6 comments

Thank you for your constructive review. We agree that the choice of rubric judge model matters, and we appreciate your suggestion to quantify how often pointwise rewards tie during training. We conducted some additional experiments in response to your concerns.

W1. Judge strength and reward discrimination

W1.1. Replacement with a stronger rubric reward model

We replaced Qwen3.6-35B-A3B with Qwen3.8-flash as the training judge and reran the Qwen3-8B experiments with three random seeds. We also used Qwen3.8-flash to judge the trained policies’ responses on all four benchmarks. Each entry reports the mean ± sample SD across seeds.

MethodAlpacaEval 2Arena-Hard v2WildBenchWritingBenchAverage
MatrixReward81.77 ± 1.4339.20 ± 0.8654.47 ± 0.6759.04 ± 0.5058.62 ± 0.30
MatrixReward (implicit)79.31 ± 1.3838.02 ± 0.4653.45 ± 0.3257.64 ± 0.7457.10 ± 0.20
RRC-SCR79.23 ± 0.6035.32 ± 1.2854.05 ± 0.7656.65 ± 0.3156.31 ± 0.50
ArenaRL78.17 ± 1.4232.79 ± 1.7552.87 ± 0.3255.65 ± 1.0854.87 ± 0.30
Tournament-GRPO74.62 ± 0.4831.10 ± 0.4352.51 ± 0.7255.68 ± 1.1053.48 ± 0.41
Pairwise Rank70.99 ± 0.4128.86 ± 0.3553.23 ± 0.8055.85 ± 0.7752.23 ± 0.35
GOPO61.15 ± 0.9630.49 ± 1.4053.35 ± 0.5954.81 ± 0.6749.95 ± 0.40
GRPO (RaR explicit)62.55 ± 1.1630.37 ± 0.8954.11 ± 0.2957.00 ± 0.7251.01 ± 0.06
Dr. GRPO65.38 ± 2.1527.85 ± 0.9352.10 ± 0.7653.13 ± 1.5949.62 ± 0.73
GD²PO58.95 ± 1.3929.04 ± 0.4250.44 ± 0.4553.44 ± 0.7247.97 ± 0.47
GDPO64.45 ± 0.8130.45 ± 1.8453.38 ± 0.3455.63 ± 0.2250.98 ± 0.67
DAPO63.92 ± 2.6732.36 ± 0.9754.30 ± 0.3255.99 ± 0.4651.64 ± 0.68
SAW66.37 ± 1.3332.28 ± 1.0152.10 ± 2.6158.19 ± 0.4352.23 ± 0.53
POW3R63.61 ± 1.7331.07 ± 1.5051.55 ± 0.4451.08 ± 0.2349.33 ± 0.58
AMRP52.55 ± 0.9728.71 ± 0.8152.50 ± 0.7255.57 ± 0.9047.33 ± 0.77

MatrixReward outperformed all pointwise baselines on the four-benchmark average in each of the three seeds. MatrixReward averaged 58.62, its implicit variant 57.10, and DAPO, the strongest pointwise baseline, 51.64. The gap thus persisted with the stronger training judge. This finding is consistent with our motivation. For an open-ended query without a reference answer, a pointwise judge scores each response independently against the rubric. Without seeing other responses to the same query, it can be difficult to place small differences in quality on a consistent numerical scale. A pairwise judge can compare two responses to the same query directly.

W1.2. Frequency of reward ties during pointwise training

We analyzed all 7,680 groups from one 480-step GRPO (RaR explicit) run with the original training judge. For each query, we counted how many of its eight responses shared the most frequent final reward, including the format reward. We report cumulative counts by tie size, the share of fully tied groups in each epoch, and counts by exact tie size.

Responses with the same rewardGroupsShare of all groups
≥27,04891.8%
≥35,48771.4%
≥44,30256.0%
≥53,43844.8%
≥62,83336.9%
≥72,32830.3%
81,84324.0%
Epoch (training steps)Fully tied groups / all groupsShare within epoch
1 (1–160)543 / 2,56021.2%
2 (161–320)623 / 2,56024.3%
3 (321–480)677 / 2,56026.4%
Exact number sharing the most frequent rewardGroupsShare of all groups
81,84324.0%
74856.3%
65056.6%
56057.9%
486411.3%
31,18515.4%
21,56120.3%
1 (all distinct)6328.2%

In 89.0% of fully tied groups, all eight responses received the maximum reward of 2.0. These groups accounted for 21.4% of all training groups. Tied responses receive the same GRPO advantage. When all eight rewards are equal, group centering gives every response zero reward-derived advantage. Thus, in nearly a quarter of the groups, the reward provided no training signal favoring one response over another. The share of groups with zero reward-derived advantage rose from 21.2% in the first training phase to 26.4% in the last. As training progresses, the pointwise method's ability to distinguish responses across groups decreases. This loss of training signal can help explain the lower benchmark scores of GRPO (RaR explicit) under the same training budget.

W2. Additional baselines for rubric-based reward learning

Thank you for suggesting RaR and OpenRS. Our original GRPO baseline used explicit RaR aggregation. We have labeled it GRPO (RaR explicit) and added GRPO (RaR implicit) and OpenRS to the comparison.

In RaR explicit, the judge scores each response on each query-specific rubric, and we aggregate those scores using the rubric importance weights. In RaR implicit, the judge receives the same rubrics and weights together and assigns one overall score to each response. RaR

For OpenRS, we used its pairwise adaptive rubric scorer and routed each query to one of the five domain prompts. We trained for two epochs with 16 rollouts per query on the same 2,560 queries used for MatrixReward. OpenRS

All methods below used Qwen3-8B as the policy, Qwen3.6-35B-A3B as the training judge, and Qwen3.8-flash as the evaluation judge on all four benchmarks. We report the mean ± sample SD across three runs.

MethodAlpacaEval 2Arena-Hard v2WildBenchWritingBenchAverage
MatrixReward (explicit)83.24 ± 1.6639.36 ± 0.8955.86 ± 0.5560.74 ± 0.3259.80 ± 0.38
MatrixReward (implicit)80.52 ± 1.5540.71 ± 0.8755.00 ± 0.2359.10 ± 0.8758.83 ± 0.14
RRC-SCR80.58 ± 0.8337.00 ± 2.1853.99 ± 0.4758.28 ± 0.6757.46 ± 0.75
ArenaRL79.59 ± 1.3434.25 ± 1.3954.32 ± 0.1757.09 ± 0.8256.31 ± 0.13
Tournament-GRPO76.21 ± 0.1332.62 ± 0.4453.87 ± 0.6157.10 ± 1.2954.95 ± 0.49
Pairwise Rank72.39 ± 0.1930.48 ± 0.1656.38 ± 0.6657.55 ± 0.9754.20 ± 0.26
GRPO (RaR explicit)64.09 ± 0.7929.83 ± 2.5755.76 ± 0.5358.43 ± 0.6452.03 ± 0.62
GRPO (RaR implicit)65.55 ± 0.6629.58 ± 0.7355.18 ± 0.3859.75 ± 0.4052.52 ± 0.22
OpenRS61.91 ± 0.7930.55 ± 0.8256.10 ± 0.3457.96 ± 0.3951.63 ± 0.19
Dr. GRPO67.00 ± 2.1528.04 ± 2.0753.41 ± 1.0054.73 ± 1.3050.80 ± 0.77
GOPO62.58 ± 0.8329.82 ± 2.7254.79 ± 0.5056.42 ± 0.2450.90 ± 0.51
GDPO65.95 ± 1.1830.29 ± 0.4454.75 ± 0.0957.29 ± 0.2952.07 ± 0.39
GD²PO60.50 ± 1.5225.73 ± 0.7952.23 ± 0.4654.86 ± 0.9648.33 ± 0.61
DAPO65.26 ± 2.4232.41 ± 2.1555.76 ± 0.2457.64 ± 0.7252.77 ± 0.43
SAW67.52 ± 1.3333.67 ± 1.2353.82 ± 2.4259.66 ± 0.4453.67 ± 0.51
POW3R64.97 ± 1.9532.46 ± 1.2953.06 ± 0.1152.41 ± 0.5850.72 ± 0.70
AMRP54.45 ± 0.9330.01 ± 1.1654.05 ± 0.5757.21 ± 0.7048.93 ± 0.82

MatrixReward (explicit) averaged 59.80 across the four benchmarks, ahead of RaR explicit by 7.77 points, RaR implicit by 7.28, and OpenRS by 8.17. The largest gains over RaR implicit and OpenRS were on AlpacaEval 2 and Arena-Hard v2. OpenRS scored slightly higher on WildBench (56.10 vs. 55.86).

W3. The trade-off between training effectiveness and judging cost

Thank you for mentioning the judging cost issue of MatrixReward(explicit). We agree that its additional gain comes at a substantial token cost. In the supplementary experiments reported in W1 and W2, we trained Qwen3-8B with either Qwen3.6-35B-A3B or Qwen3.8-flash as the training judge. In both settings, Qwen3.8-flash judged the trained policies' responses on all four benchmarks. Each entry is the four-benchmark average, reported as the mean ± sample SD across three runs.

Training judge modelRRC-SCRMatrixReward (implicit)MatrixReward (explicit)
Qwen3.6-35B-A3B57.46 ± 0.7558.83 ± 0.1459.80 ± 0.38
Qwen3.8-flash56.31 ± 0.5057.10 ± 0.2058.62 ± 0.30

At the same budget of 28 judge calls per group, MatrixReward (implicit) outperformed RRC-SCR by 1.37 points (2.38%) with Qwen3.6-35B-A3B as the training judge and by 0.79 points (1.40%) with Qwen3.8-flash as the training judge. Both methods make one call per response pair. RRC-SCR asks for one overall verdict, while MatrixReward (implicit) obtains a verdict for each rubric in that call.

MatrixReward (explicit) uses a separate judge call for each rubric. With Qwen3.6-35B-A3B as the training judge, it outperformed the implicit variant by 0.97 points (1.65%) and RRC-SCR by 2.34 points (4.07%). With Qwen3.8-flash, it outperformed the implicit variant by 1.52 points (2.66%) and RRC-SCR by 2.31 points (4.10%).

We believe this is a trade-off issue. When the training judge budget is limited, MatrixReward(implicit) can be used to obtain better training results, while when pursuing better training results, MatrixReward(explicit) can be used.

Reviewer_hmJAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MatrixReward constructs rewards for open-ended LLM training by comparing sampled responses under several query-specific rubrics. It organizes these comparisons into a response-by-rubric win-rate matrix, adjusts rubric weights based on variation and correlation, and assigns rewards according to each response’s distance from the best and worst observed profiles. These rewards are used for GRPO training. On four benchmarks, it improves Qwen3-8B’s average score by 1.23 points over the strongest baseline.

Critical strengths

The paper addresses a meaningful problem in reward design: a single overall score can overlook important differences between responses. Comparing answers separately across rubrics provides more detailed feedback, and the ablations help show how weighting and reward aggregation affect performance.

Critical weaknesses

The main concern is that the same Qwen3.6-35B-A3B model provides both training rewards and benchmark judgments, without independent or human evaluation. This makes it unclear whether the improvements reflect better answer quality or closer alignment with that judge’s preferences. The main method also uses roughly 3.4 times the judging tokens of the strongest baseline for a 1.23-point gain, while the cheaper variant improves by only 0.44 points. Neither gain is supported by multiple-seed results or confidence intervals. Furthermore, the weighting scheme assumes that variation and low correlation indicate useful feedback, although these could also reflect judge noise, especially with only eight responses per group. Finally, training uses just 2,560 prompts from one dataset, with no analysis of whether the gains persist with more data or different training subsets.

Questions for authors
  1. Have the authors tested the proposed approach with scaling up the training pool
  2. How do you distinguish informative rubric variation from judge noise, and how stable are the inferred weights with only eight responses per prompt?
3 comments

Thank you for your thoughtful review. We understand your concerns about whether the gains depend on the training judge’s preferences, whether they hold across training seeds. To address these concerns, we ran additional experiments with three training seeds and evaluated the resulting policies with a different benchmark judge.

W1. Evaluation with a different benchmark judge

We trained Qwen3-8B with Qwen3.6-35B-A3B as the reward judge and used Qwen3.8-flash to evaluate the resulting policies on all four benchmarks. The table reports the mean ± sample SD across three training seeds.

MethodAlpacaEval 2Arena-Hard v2WildBenchWritingBenchAverage
MatrixReward (explicit)83.24 ± 1.6639.36 ± 0.8955.86 ± 0.5560.74 ± 0.3259.80 ± 0.38
MatrixReward (implicit)80.52 ± 1.5540.71 ± 0.8755.00 ± 0.2359.10 ± 0.8758.83 ± 0.14
RRC-SCR80.58 ± 0.8337.00 ± 2.1853.99 ± 0.4758.28 ± 0.6757.46 ± 0.75
ArenaRL79.59 ± 1.3434.25 ± 1.3954.32 ± 0.1757.09 ± 0.8256.31 ± 0.13
Tournament-GRPO76.21 ± 0.1332.62 ± 0.4453.87 ± 0.6157.10 ± 1.2954.95 ± 0.49
Pairwise Rank72.39 ± 0.1930.48 ± 0.1656.38 ± 0.6657.55 ± 0.9754.20 ± 0.26
GRPO (RaR explicit)64.09 ± 0.7929.83 ± 2.5755.76 ± 0.5358.43 ± 0.6452.03 ± 0.62
GRPO (RaR implicit)65.55 ± 0.6629.58 ± 0.7355.18 ± 0.3859.75 ± 0.4052.52 ± 0.22
OpenRS61.91 ± 0.7930.55 ± 0.8256.10 ± 0.3457.96 ± 0.3951.63 ± 0.19
Dr. GRPO67.00 ± 2.1528.04 ± 2.0753.41 ± 1.0054.73 ± 1.3050.80 ± 0.77
GOPO62.58 ± 0.8329.82 ± 2.7254.79 ± 0.5056.42 ± 0.2450.90 ± 0.51
GDPO65.95 ± 1.1830.29 ± 0.4454.75 ± 0.0957.29 ± 0.2952.07 ± 0.39
GD²PO60.50 ± 1.5225.73 ± 0.7952.23 ± 0.4654.86 ± 0.9648.33 ± 0.61
DAPO65.26 ± 2.4232.41 ± 2.1555.76 ± 0.2457.64 ± 0.7252.77 ± 0.43
SAW67.52 ± 1.3333.67 ± 1.2353.82 ± 2.4259.66 ± 0.4453.67 ± 0.51
POW3R64.97 ± 1.9532.46 ± 1.2953.06 ± 0.1152.41 ± 0.5850.72 ± 0.70
AMRP54.45 ± 0.9330.01 ± 1.1654.05 ± 0.5757.21 ± 0.7048.93 ± 0.82

With Qwen3.8-flash as the benchmark judge, RRC-SCR was the strongest baseline, averaging 57.46 across the four benchmarks. MatrixReward (explicit) scored higher on all four and averaged 59.80, a gain of 2.34 points (4.07%) over RRC-SCR. MatrixReward (implicit) scored higher on three of the four and averaged 58.83, a gain of 1.37 points (2.38%) over RRC-SCR. The gains persisted after we changed the benchmark judge model, indicating that MatrixReward’s higher scores reflect genuine improvements in response quality beyond alignment with the original evaluator’s preferences.

W2. Variation across seeds and judging cost

MatrixReward (explicit) and MatrixReward (implicit) both outperformed RRC-SCR on the four-benchmark average in all three runs. Across seeds, RRC-SCR averaged 57.46 ± 0.75. MatrixReward (explicit) averaged 59.80 ± 0.38, a gain of 2.34 points (4.07%) over RRC-SCR. MatrixReward (implicit) averaged 58.83 ± 0.14, a gain of 1.37 points (2.38%) over RRC-SCR.

For a group of GG rollouts and KK rubrics, MatrixReward (explicit) judges each response pair separately under each rubric, requiring KG(G−1)/2K G(G-1)/2 calls. MatrixReward (implicit) and RRC-SCR each use G(G−1)/2G(G-1)/2 calls. MatrixReward (implicit) obtains a verdict for every rubric in one call per pair, while RRC-SCR obtains one overall verdict.

A trade-off needs to be struck between higher training costs and better training results. MatrixReward (implicit) improves RRC-SCR with the same number of judge calls per group. MatrixReward (explicit) can be used when additional calls are affordable and better training results are required.

Q1 and W4. Scaling the training pool

Thank you for asking whether the gains persist with a larger training pool. We expanded the training dataset from 2,560 to 5,120 queries and trained for 960 steps, up from 480. We trained Qwen3-8B using Qwen3.6-35B-A3B as the training judge and evaluated the resulting policies with Qwen3.8-flash. The table reports the mean ± sample SD over three seeds for the expanded setting.

MethodAlpacaEval 2Arena-Hard v2WildBenchWritingBenchAverage
MatrixReward (explicit)85.24 ± 1.3842.91 ± 1.2660.92 ± 0.2862.58 ± 0.5262.92 ± 0.67
MatrixReward (implicit)82.58 ± 1.4242.59 ± 0.8657.68 ± 1.0561.28 ± 0.6961.03 ± 0.38
RRC-SCR82.64 ± 0.4741.13 ± 0.3956.51 ± 0.9160.20 ± 0.9060.12 ± 0.20
ArenaRL81.46 ± 1.2136.99 ± 1.2356.31 ± 0.2459.00 ± 1.2658.44 ± 0.50
Tournament-GRPO78.02 ± 0.3335.48 ± 1.5856.30 ± 0.6759.23 ± 1.5657.26 ± 0.94
Pairwise Rank74.59 ± 0.2533.72 ± 0.8456.86 ± 0.8359.57 ± 1.0956.18 ± 0.34
GRPO (RaR explicit)66.30 ± 0.9332.73 ± 1.8957.68 ± 0.1360.28 ± 0.2654.25 ± 0.31
GRPO (RaR implicit)67.77 ± 0.2732.18 ± 0.6057.29 ± 0.5861.73 ± 0.2754.74 ± 0.13
OpenRS63.92 ± 0.7032.51 ± 0.7758.12 ± 0.4460.05 ± 0.4353.65 ± 0.21
Dr. GRPO68.79 ± 2.1031.85 ± 1.2955.37 ± 0.8156.86 ± 1.3753.22 ± 1.03
GOPO64.69 ± 0.6232.40 ± 2.2457.00 ± 0.4758.31 ± 0.3753.10 ± 0.44
GDPO68.30 ± 1.3232.33 ± 0.2956.45 ± 0.2259.47 ± 0.6454.14 ± 0.32
GD²PO62.53 ± 1.4830.86 ± 3.2454.60 ± 0.7256.80 ± 1.1651.20 ± 1.00
DAPO70.13 ± 0.2433.35 ± 1.0157.96 ± 0.4459.59 ± 0.7055.26 ± 0.57
SAW69.64 ± 1.5535.92 ± 1.3655.79 ± 2.1161.80 ± 0.5855.78 ± 0.32
POW3R67.15 ± 1.8134.19 ± 1.0954.87 ± 0.1054.35 ± 0.6552.64 ± 0.56
AMRP56.42 ± 1.0432.13 ± 1.0855.95 ± 0.5159.32 ± 0.6350.96 ± 0.78

When we increased the training pool and the number of training steps, all methods improved their mean scores on each of the four benchmarks. MatrixReward (explicit) achieved the highest mean score on each of the four benchmarks. Its four-benchmark average was 62.92, compared with 60.12 for RRC-SCR, the strongest baseline on average. MatrixReward (explicit) led RRC-SCR by 2.80 points (4.65%), and MatrixReward (implicit) averaged 61.03, a lead of 0.91 points (1.52%). This demonstrates that after increasing the dataset, MatrixReward still provides effective gains compared to the baseline.

Q2 and W3. Judge noise and the stability of rubric weights

Thank you for your constructive feedback. We understand your concerns about how judge noise affects reward construction and how reliably we can estimate rubric weights. We examined the contribution of the weighting rule to training and the stability of the estimates with eight rollouts per query.

We acknowledge that LLM judges can introduce noise through biases such as length and position preferences. All methods that rely on LLM judges to construct rewards, including MatrixReward and other pointwise and pairwise methods, share this limitation. We use the same judge model across all methods in each comparison. This rules out differences in judge choice as a source of bias and provides a fair basis for comparing methods.

MatrixReward’s performance gains across the four benchmarks provide indirect evidence of the effectiveness and robustness of its reward construction. Although benchmark results alone cannot directly establish the accuracy and stability of the algorithm itself, they show that MatrixReward can provide an effective optimization signal even when the judge’s outputs are noisy.

To decrease the impact of judge noise, we combine dispersion and correlation conflict to construct rubric weights. We ran ablations to test each factor’s effect. We trained Qwen3-8B on the original dataset with Qwen3.6-35B-A3B as judge and evaluated it with Qwen3.8-flash.

Table 1. Rubric weighting ablations. Scores are means ± sample SD across three training seeds.

MethodAlpacaEval 2Arena-Hard v2WildBenchWritingBenchAverage
MatrixReward83.24 ± 1.6639.36 ± 0.8955.86 ± 0.5560.74 ± 0.3259.80 ± 0.38
Dispersion only79.83 ± 0.8638.02 ± 0.0555.37 ± 0.5258.67 ± 0.7257.97 ± 0.15
Correlation conflict only78.14 ± 0.2837.60 ± 0.4453.63 ± 0.2857.24 ± 0.2756.65 ± 0.12
RRC-SCR80.58 ± 0.8337.00 ± 2.1853.99 ± 0.4758.28 ± 0.6757.46 ± 0.75

Dispersion σk\sigma_k measures the spread of normalized win rates within a rollout group. This matters for GRPO because a rollout’s advantage depends on how its final reward differs from the group mean. Correlation conflict hk=∑ℓ≠k(1−ρkℓ)h_k=\sum_{\ell\ne k}(1-\rho_{k\ell}) is high when a rubric’s win-rate pattern is weakly or negatively correlated with those of other rubrics. A rubric can have high conflict even when most of its win rates are close together, making conflict alone more susceptible to judge noise in such cases. Dispersion alone averaged 57.97 across the four benchmarks, compared with 56.65 for correlation conflict alone. The product σkhk\sigma_k h_k in Eq. (3) favors rubrics that separate the rollouts and gives less weight to distinctions already captured by other rubrics. This can reduce the influence of judge noise when it produces high conflict in a column with little dispersion. Combining the two factors raised the average to 59.80, gains of 1.83 and 3.15 points over dispersion alone and conflict alone, respectively.

To assess weight stability with eight rollouts, we drew 1,000 bootstrap samples per query for 300 query groups. Each sample contained eight rows drawn with replacement from the observed win-rate matrix, keeping the recorded win rates fixed. We recomputed the weights before fusion with the rubric prior and compared each estimate with the original eight-rollout weight vector for that query.

Table 2. Stability of matrix-derived weights under rollout resampling.

Bootstrap statisticResult
Mean cosine similarity to the original weights0.934
Within-group 2.5th percentile of cosine similarity, summarized across groups0.818
Mean L1 distance from the original weights0.295
Range of weight CVs across all query–rubric pairs0.136–0.231

The resampled weight vectors had a mean cosine similarity of 0.934 to the original vectors. The within-query 2.5th percentiles of cosine similarity yielded a summary value of 0.818 across queries. The mean L1 distance of 0.295 corresponds to redistributing 14.75% of the total weight across rubrics, so the original and resampled allocations overlapped by 85.25% on average. Individual rubric weights showed some sensitivity to the rollout sample, with bootstrap standard deviations ranging from 13.6% to 23.1% of their respective means. These results indicate that MatrixReward estimates relatively stable rubric weights from eight rollouts per query.

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful reading and constructive feedback. We have taken the reviewers’ concerns seriously and conducted additional experiments and analyses to address them. The revised manuscript includes these results in the appendices, with references added at the relevant places in the main text. Revisions are highlighted in blue.

  1. Evaluation reliability, training scale, and weight stability — Reviewer hmJA. We evaluated policies with a benchmark judge different from the training judge and reported results across three training seeds. Both MatrixReward variants retained their advantage over RRC-SCR on the four-benchmark average. We also expanded the training pool and found that the performance gains persisted, and assessed the stability of rubric weights estimated from eight rollouts per query. These results are presented in Appendices G, H.1, and H.3.

  2. Reward discrimination, related baselines, and methodological ablations — Reviewer 3pMR. We repeated training with a stronger judge, quantified reward ties throughout GRPO training, clarified the explicit RaR formulation of our original GRPO baseline, and added RaR implicit and OpenRS comparisons. We further isolated the contributions of dispersion and correlation conflict, compared aggregation methods under matched normalization and weighting, and examined the effect of taking absolute values in weight fusion. These additions appear in Appendices D, H.1–H.2, and I–L, and support the effectiveness of the individual design choices while clarifying the performance–cost trade-off.

  3. Judging budget and reward aggregation — Reviewer pRmV. We compared MatrixReward with controls that average repeated judgments under a matched judge-call budget. We also compared aggregation methods using deterministic, programmatically verified constraint feedback with identical weighting and normalization. MatrixReward achieved higher mean constraint satisfaction and all-pass rates in this setting. These experiments are reported in Appendices M and N.

If the paper is accepted, we will expand the main text by one page and revise the camera-ready version comprehensively in light of all reviewers’ feedback.

We sincerely thank the reviewers again for their time and suggestions, which have helped us strengthen the experimental evidence and clarify the scope of our conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.