- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose ICW, a way to weight the criteria of a rubric-based LLM judge instead of treating them equally. An LLM compares every pair of criteria, the preferences are put into a pairwise matrix, and the weights come from its principal eigenvector. If the matrix is judged inconsistent, the comparisons are redone once. On six preference benchmarks ICW beats uniform weights by 1.3 to 4.2 points and the best of seven baselines by 0.2 to 1.3 points, and it also works well as a reward for RL training of three Qwen models.
Critical strengths
- Uniform weighting is a real weakness of rubric judges, and the paper states the problem clearly. The setup is clean: all methods share the same rubrics and verifier outputs and differ only in the weights, so Table 1 really compares weighting schemes.
- The baseline set is broad, the ablations over verifiers and weighting models are useful, and the prompts and token costs are reported openly.
Critical weaknesses
- The consistency check doesn't do what the paper says. Each matrix entry can only be α, 1 or 1/α (Eq. 2), so even a perfectly ordered chain c1 > c2 > c3 counts as inconsistent: with α = 2 its consistency ratio is 0.046, far above τ = 0.005. The example in Fig. 2 would itself be sent to revision. On the other hand, a matrix where every preference flips with display order becomes all ones and passes. So the revision step seems to favour flatter judgments rather than catch real contradictions. This matters because without revision (ICW‡, Table 2) ICW is below the best baseline on four of six benchmarks.
- The margins are small and the setup is hard to verify. The gains over the best baseline are 0.2 to 1.3 points, with no confidence intervals or repeated runs, and α is "selected for each benchmark" with no validation split mentioned. The released code also uses different α and τ values from Table 9, and by default it adds benchmark-specific priority text to the comparison prompt, which the paper's prompt does not have.
- In the RL study, GPT-4.1 is both the weighting model and the evaluation judge.
Questions for authors
see weaknesses
8 comments
[W2] The margins are small and the setup is hard to verify. The gains over the best baseline are 0.2 to 1.3 points, with no confidence intervals or repeated runs, and α is "selected for each benchmark" with no validation split mentioned. The released code also uses different α and τ values from Table 9, and by default it adds benchmark-specific priority text to the comparison prompt, which the paper's prompt does not have.
Response: Thank you for your careful review of our experimental setup and code! We address your concerns one by one as follows.
1. The margins are small, with no confidence intervals or repeated runs. To address this concern, we have added confidence intervals in Appendix C.6. Specifically, we re-run all methods three times, where each run repeats both the weighting and the verification on the same inputs. The results (accuracy in %) are reported as mean ± std, where [mean − std, mean + std] corresponds to the 77.5% confidence interval. We further conduct a paired sample t-test over the three runs, and mark the improvements over the best baseline with p < 0.05 by *. The results demonstrate that the proposed ICW outperforms the best baseline on all included benchmarks, and the improvement is statistically significant on five of the six benchmarks and on the Macro average, which substantiates the effectiveness of ICW.
| Method | RubricBench | ResearchQA | JudgeBench | RM-Chat | Chat-Hard | Precise-IF | Macro |
|---|---|---|---|---|---|---|---|
| Uniform | 77.14 ± 0.09 | 57.69 ± 0.59 | 76.32 ± 0.09 | 73.73 ± 0.27 | 80.59 ± 0.29 | 72.08 ± 0.10 | 72.93 ± 0.11 |
| Direct | 77.54 ± 0.24 | 58.19 ± 0.62 | 76.99 ± 0.05 | 73.04 ± 0.27 | 80.34 ± 0.32 | 72.01 ± 0.26 | 73.02 ± 0.19 |
| AdaRubric | 77.77 ± 0.31 | 57.99 ± 1.02 | 76.26 ± 0.20 | 74.63 ± 0.33 | 81.51 ± 0.28 | 72.99 ± 0.33 | 73.52 ± 0.19 |
| RaR | 77.94 ± 0.19 | 59.86 ± 0.62 | 76.29 ± 0.08 | 75.71 ± 0.34 | 81.83 ± 0.35 | 74.41 ± 0.22 | 74.34 ± 0.18 |
| MAPLE | 78.06 ± 0.14 | 59.76 ± 0.78 | 76.02 ± 0.36 | 74.72 ± 0.20 | 81.21 ± 0.28 | 73.68 ± 0.39 | 73.91 ± 0.18 |
| RRD-WU | 78.19 ± 0.24 | 59.66 ± 1.12 | 77.26 ± 0.14 | 75.88 ± 0.15 | 81.62 ± 0.25 | 74.69 ± 0.38 | 74.55 ± 0.13 |
| ChasingTail | 78.12 ± 0.09 | 59.57 ± 0.74 | 77.02 ± 0.08 | 75.80 ± 0.20 | 81.62 ± 0.35 | 74.20 ± 0.12 | 74.39 ± 0.13 |
| RACE-SC | 77.99 ± 0.13 | 59.66 ± 0.34 | 76.77 ± 0.08 | 75.67 ± 0.20 | 81.21 ± 0.32 | 75.59 ± 0.06 | 74.48 ± 0.05 |
| ICW | 79.64 ± 0.16* | 60.75 ± 0.45* | 78.41 ± 0.19* | 76.79 ± 0.15* | 82.09 ± 0.28 | 76.01 ± 0.24* | 75.62 ± 0.06* |
2. α is selected for each benchmark with no validation split mentioned. We agree that this point warrants attention and conduct two additional experiments to confirm that it does not compromise our conclusions.
(i) The effectiveness of ICW is robust to the choice of α. The table below reports the average accuracy (%) over the four benchmarks in Fig. 3. Note that when α = 1, ICW reduces to Uniform. For every α > 1, ICW consistently outperforms both Uniform and RRD-WU (the strongest baseline).
| α | Uniform | RRD-WU | ICW |
|---|---|---|---|
| 1 | 75.89 | 77.46 | 75.89 |
| 1.25 | 75.89 | 77.46 | 78.12 |
| 1.5 | 75.89 | 77.46 | 78.15 |
| 1.75 | 75.89 | 77.46 | 78.16 |
| 2 | 75.89 | 77.46 | 78.25 |
| 2.25 | 75.89 | 77.46 | 77.98 |
(ii) Selecting α on a held-out validation split yields nearly identical results and conclusion. We randomly partition each benchmark into 30% validation and 70% test, select α on the validation split, and report test-split accuracy (%). As shown in the table below, ICW with the validation-selected α outperforms RRD-WU on all six benchmarks, achieving nearly identical Macro accuracy to the values reported in the paper.
| Method | RubricBench | ResearchQA | JudgeBench | RM-Chat | Chat-Hard | Precise-IF | Macro |
|---|---|---|---|---|---|---|---|
| RRD-WU | 77.68 | 60.17 | 78.00 | 75.37 | 81.97 | 74.40 | 74.60 |
| ICW (α in the paper) | 79.43 (1.25) | 61.86 (1.25) | 78.80 (1.75) | 76.67 (1.5) | 82.29 (1.75) | 76.19 (1.75) | 75.87 |
| ICW (α selected on validation) | 79.43 (1.25) | 61.86 (1.25) | 78.80 (2) | 76.11 (1.25) | 82.60 (2) | 76.19 (1.75) | 75.83 |
3. The released code uses different α and τ values and adds benchmark-specific priority text. We sincerely apologize for this inconsistency. As clarified in our response to W1, the values of α in Table 9 were mistakenly labeled and the anonymous repository and prompts was mistakenly prepared from a wrong branch. We have corrected the hyperparameters (Table 16) and the prompts (Appendix D.2) in the paper, and updated the repository accordingly, so that both are fully consistent with the settings that produced the results reported in the paper. All changes in the paper are marked in blue.
[W3] In the RL study, GPT-4.1 is both the weighting model and the evaluation judge.
Response: Thank you for pointing out this issue! To rule out the bias from using the same model for weighting and judging, we re-evaluate all policy models with a judge from a different model family, i.e., Gemini-3.1-Pro, while keeping the policy models and the three evaluation benchmarks of Section 4.6 unchanged. Gemini-3.1-Pro is not involved in the weights or rewards of any method. As shown in the table below, ICW still outperforms Uniform on all benchmarks across all three policy models, and achieves the best results in 12 of 15 cases, demonstrating that the gains of ICW in the RL results are not due to using the same model for weighting and judging. We have replaced the RL results in Table 5 and the corresponding discussions in Section 4.6 with the new ones.
| Policy Model | Method | BiGGen Bench (Score) | AlpacaEval 2.0 (WR) | AlpacaEval 2.0 (LC) | Arena-Hard (WR) | Arena-Hard (SC) |
|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | Base Model | 57.23 | 11.55 | 11.88 | 15.00 | 13.91 |
| Qwen2.5-3B-Instruct | Uniform | 59.78 | 14.41 | 13.17 | 17.08 | 16.97 |
| Qwen2.5-3B-Instruct | RACE-SC | 60.68 | 14.66 | 13.49 | 18.69 | 17.63 |
| Qwen2.5-3B-Instruct | RRD-WU | 60.81 | 15.65 | 14.97 | 17.39 | 16.36 |
| Qwen2.5-3B-Instruct | ICW | 61.99 | 15.53 | 15.16 | 20.25 | 18.20 |
| Qwen2.5-7B-Instruct | Base Model | 67.05 | 24.47 | 22.46 | 36.66 | 36.56 |
| Qwen2.5-7B-Instruct | Uniform | 69.83 | 35.03 | 29.15 | 41.68 | 41.54 |
| Qwen2.5-7B-Instruct | RACE-SC | 71.15 | 34.16 | 30.70 | 42.80 | 42.05 |
| Qwen2.5-7B-Instruct | RRD-WU | 72.24 | 35.53 | 31.12 | 41.84 | 41.78 |
| Qwen2.5-7B-Instruct | ICW | 72.08 | 37.14 | 33.60 | 44.38 | 43.86 |
| Qwen3-4B | Base Model | 74.25 | 32.67 | 28.58 | 51.63 | 51.87 |
| Qwen3-4B | Uniform | 76.29 | 34.16 | 33.38 | 54.36 | 54.15 |
| Qwen3-4B | RACE-SC | 77.56 | 36.27 | 35.03 | 56.88 | 55.43 |
| Qwen3-4B | RRD-WU | 77.20 | 35.65 | 34.10 | 54.42 | 53.83 |
| Qwen3-4B | ICW | 78.52 | 38.51 | 36.50 | 56.29 | 55.64 |
Again, thank you so much for your careful reading of our paper and code! We are eager to hear your feedback. We'd deeply appreciate it if you could let us know whether your concerns have been addressed.
Thank you very much for your constructive comments and appreciation of our problem statement and experimental setup! We are grateful that you have highlighted the key issues in the consistency check and the verifiability of our setup. Below, we address them point by point and update the manuscript accordingly.
Response: Thank you very much for your meticulous comments on the consistency check! We are truly sorry that the values of α in the hyperparameter table and on the α-axis of the sensitivity figures were incorrectly labeled in the original submission. The actual values used in our main experiments are α = 1.25 for RubricBench and ResearchQA, 1.5 for RM-Chat, and 1.75 for JudgeBench, Chat-Hard, and Precise-IF, with τ = 0.005 for all benchmarks. These have been corrected in the revised paper. On the basis of these, we address the two specific concerns below.
Table: Each cell reports the value before → after revision.
Overall, CR serves as a proxy for contradictory comparisons rather than an exact detector: any contradiction makes the comparisons inconsistent and raises CR, whereas under the fixed ratio some contradiction-free judgments, such as the perfectly ordered chain you pointed out, can also exceed τ. We have made this explicit in the revised Section 3.4, and added these statistics and discussions in Appendix C.7.