acceptodds
Under review as a conference paper at ICLR 2027

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

Abstract

Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally. However, existing methods commonly aggregate the verification results with uniform weighting, which assigns minor and overlapping criteria the same weight as essential ones and thus produces misleading evaluation scores. Although some methods prompt an LLM to directly assign weights, direct weighting requires a holistic judgment over all criteria that LLMs struggle to make reliably. To address these limitations, we propose Importance-Aware Criteria Weighting (ICW), which derives the weights of criteria from pairwise importance comparisons made by an LLM. Specifically, ICW transforms the holistic judgment over all criteria into a series of pairwise importance comparisons, checks and revises inconsistent comparisons, and derives the weights from the principal eigenvector of the comparison matrix. Extensive experiments demonstrate that ICW consistently outperforms uniform weighting and competitive weighting methods, and translates into better policy models in downstream reinforcement learning. Code is available at https://anonymous.4open.science/r/ICW-F5F1.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
large language model, reward modeling, rubrics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wKaPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose ICW, a way to weight the criteria of a rubric-based LLM judge instead of treating them equally. An LLM compares every pair of criteria, the preferences are put into a pairwise matrix, and the weights come from its principal eigenvector. If the matrix is judged inconsistent, the comparisons are redone once. On six preference benchmarks ICW beats uniform weights by 1.3 to 4.2 points and the best of seven baselines by 0.2 to 1.3 points, and it also works well as a reward for RL training of three Qwen models.

Critical strengths
  • Uniform weighting is a real weakness of rubric judges, and the paper states the problem clearly. The setup is clean: all methods share the same rubrics and verifier outputs and differ only in the weights, so Table 1 really compares weighting schemes.
  • The baseline set is broad, the ablations over verifiers and weighting models are useful, and the prompts and token costs are reported openly.
Critical weaknesses
  • The consistency check doesn't do what the paper says. Each matrix entry can only be α, 1 or 1/α (Eq. 2), so even a perfectly ordered chain c1 > c2 > c3 counts as inconsistent: with α = 2 its consistency ratio is 0.046, far above τ = 0.005. The example in Fig. 2 would itself be sent to revision. On the other hand, a matrix where every preference flips with display order becomes all ones and passes. So the revision step seems to favour flatter judgments rather than catch real contradictions. This matters because without revision (ICW‡, Table 2) ICW is below the best baseline on four of six benchmarks.
  • The margins are small and the setup is hard to verify. The gains over the best baseline are 0.2 to 1.3 points, with no confidence intervals or repeated runs, and α is "selected for each benchmark" with no validation split mentioned. The released code also uses different α and τ values from Table 9, and by default it adds benchmark-specific priority text to the comparison prompt, which the paper's prompt does not have.
  • In the RL study, GPT-4.1 is both the weighting model and the evaluation judge.
Questions for authors

see weaknesses

8 comments

Thank you very much for your constructive comments and appreciation of our problem statement and experimental setup! We are grateful that you have highlighted the key issues in the consistency check and the verifiability of our setup. Below, we address them point by point and update the manuscript accordingly.


[W1] The consistency check doesn't do what the paper says. Each matrix entry can only be α, 1 or 1/α (Eq. 2), so even a perfectly ordered chain c1 > c2 > c3 counts as inconsistent: with α = 2 its consistency ratio is 0.046, far above τ = 0.005. The example in Fig. 2 would itself be sent to revision. On the other hand, a matrix where every preference flips with display order becomes all ones and passes. So the revision step seems to favour flatter judgments rather than catch real contradictions. This matters because without revision (ICW‡, Table 2) ICW is below the best baseline on four of six benchmarks.

Response: Thank you very much for your meticulous comments on the consistency check! We are truly sorry that the values of α in the hyperparameter table and on the α-axis of the sensitivity figures were incorrectly labeled in the original submission. The actual values used in our main experiments are α = 1.25 for RubricBench and ResearchQA, 1.5 for RM-Chat, and 1.75 for JudgeBench, Chat-Hard, and Precise-IF, with τ = 0.005 for all benchmarks. These have been corrected in the revised paper. On the basis of these, we address the two specific concerns below.

  • On perfectly ordered chains being flagged as inconsistent. A perfectly ordered chain denotes a strict ranking of all criteria with no ties, e.g., c1 > c2 > c3 > c4. With α = 1.25, such a chain yields CR ≤ 0.0048 ≤ τ for all rubric set sizes in our benchmarks and is thus accepted—as is the example in Fig. 2, which we have redrawn with α = 1.25. With α = 1.5 and 1.75, such a chain still exceeds τ (CR = 0.015 and 0.029, respectively, for the four-criterion chain above). On the other hand, it is natural for some criteria in a rubric set to be equally important, and the weighting model may explicitly assign equal importance to them (Section 3.2). Under such equal judgments, a contradiction-free comparison matrix yields a lower CR than the perfectly ordered chain. For instance, c1 = c2 > c3 = c4 has CR = 0 and passes the check. In practice, no more than 16.89% of the samples on each benchmark exceed τ without containing any contradiction. Moreover, exceeding τ only triggers a re-comparison, in which the weighting model may keep its original judgments.
  • On the point that the revision favors flatter judgments rather than catching real contradictions. We agree that an all-ones matrix passes the check, as it contains no contradiction to revise, and ICW reduces to Uniform for such a sample. This does not, however, imply that the revision favors flatter judgments, as multiple matrices beyond the all-ones matrix also pass the check. To assess whether the revision corrects genuine contradictions, we add the statistics of the comparisons before and after revision. As shown in the table below, the revision reduces the proportion of contradictory samples on all six benchmarks (e.g., from 42.54% to 10.09% on Chat-Hard), while the share of neutral pairs (entries equal to 1 in Eq. (2)) stays almost unchanged (within 1.7 points). These results demonstrate that the revision catches real contradictions rather than favoring flatter judgments.

Table: Each cell reports the value before → after revision.

BenchmarkαContradictory (%)Neutral pairs (%)CR > τ (%)
RubricBench1.2523.89 → 10.9926.75 → 26.7618.83 → 3.40
ResearchQA1.2530.18 → 17.1638.17 → 37.1718.34 → 4.73
JudgeBench1.7528.87 → 9.1939.03 → 40.5635.97 → 16.94
RM-Chat1.5018.60 → 5.4338.90 → 39.0225.32 → 13.44
Chat-Hard1.7542.54 → 10.0936.48 → 37.0453.07 → 25.66
Precise-IF1.7535.83 → 6.4640.88 → 42.5441.67 → 12.71

Overall, CR serves as a proxy for contradictory comparisons rather than an exact detector: any contradiction makes the comparisons inconsistent and raises CR, whereas under the fixed ratio some contradiction-free judgments, such as the perfectly ordered chain you pointed out, can also exceed τ. We have made this explicit in the revised Section 3.4, and added these statistics and discussions in Appendix C.7.

[W2] The margins are small and the setup is hard to verify. The gains over the best baseline are 0.2 to 1.3 points, with no confidence intervals or repeated runs, and α is "selected for each benchmark" with no validation split mentioned. The released code also uses different α and τ values from Table 9, and by default it adds benchmark-specific priority text to the comparison prompt, which the paper's prompt does not have.

Response: Thank you for your careful review of our experimental setup and code! We address your concerns one by one as follows.

1. The margins are small, with no confidence intervals or repeated runs. To address this concern, we have added confidence intervals in Appendix C.6. Specifically, we re-run all methods three times, where each run repeats both the weighting and the verification on the same inputs. The results (accuracy in %) are reported as mean ± std, where [mean − std, mean + std] corresponds to the 77.5% confidence interval. We further conduct a paired sample t-test over the three runs, and mark the improvements over the best baseline with p < 0.05 by *. The results demonstrate that the proposed ICW outperforms the best baseline on all included benchmarks, and the improvement is statistically significant on five of the six benchmarks and on the Macro average, which substantiates the effectiveness of ICW.

MethodRubricBenchResearchQAJudgeBenchRM-ChatChat-HardPrecise-IFMacro
Uniform77.14 ± 0.0957.69 ± 0.5976.32 ± 0.0973.73 ± 0.2780.59 ± 0.2972.08 ± 0.1072.93 ± 0.11
Direct77.54 ± 0.2458.19 ± 0.6276.99 ± 0.0573.04 ± 0.2780.34 ± 0.3272.01 ± 0.2673.02 ± 0.19
AdaRubric77.77 ± 0.3157.99 ± 1.0276.26 ± 0.2074.63 ± 0.3381.51 ± 0.2872.99 ± 0.3373.52 ± 0.19
RaR77.94 ± 0.1959.86 ± 0.6276.29 ± 0.0875.71 ± 0.3481.83 ± 0.3574.41 ± 0.2274.34 ± 0.18
MAPLE78.06 ± 0.1459.76 ± 0.7876.02 ± 0.3674.72 ± 0.2081.21 ± 0.2873.68 ± 0.3973.91 ± 0.18
RRD-WU78.19 ± 0.2459.66 ± 1.1277.26 ± 0.1475.88 ± 0.1581.62 ± 0.2574.69 ± 0.3874.55 ± 0.13
ChasingTail78.12 ± 0.0959.57 ± 0.7477.02 ± 0.0875.80 ± 0.2081.62 ± 0.3574.20 ± 0.1274.39 ± 0.13
RACE-SC77.99 ± 0.1359.66 ± 0.3476.77 ± 0.0875.67 ± 0.2081.21 ± 0.3275.59 ± 0.0674.48 ± 0.05
ICW79.64 ± 0.16*60.75 ± 0.45*78.41 ± 0.19*76.79 ± 0.15*82.09 ± 0.2876.01 ± 0.24*75.62 ± 0.06*

2. α is selected for each benchmark with no validation split mentioned. We agree that this point warrants attention and conduct two additional experiments to confirm that it does not compromise our conclusions.

(i) The effectiveness of ICW is robust to the choice of α. The table below reports the average accuracy (%) over the four benchmarks in Fig. 3. Note that when α = 1, ICW reduces to Uniform. For every α > 1, ICW consistently outperforms both Uniform and RRD-WU (the strongest baseline).

αUniformRRD-WUICW
175.8977.4675.89
1.2575.8977.4678.12
1.575.8977.4678.15
1.7575.8977.4678.16
275.8977.4678.25
2.2575.8977.4677.98

(ii) Selecting α on a held-out validation split yields nearly identical results and conclusion. We randomly partition each benchmark into 30% validation and 70% test, select α on the validation split, and report test-split accuracy (%). As shown in the table below, ICW with the validation-selected α outperforms RRD-WU on all six benchmarks, achieving nearly identical Macro accuracy to the values reported in the paper.

MethodRubricBenchResearchQAJudgeBenchRM-ChatChat-HardPrecise-IFMacro
RRD-WU77.6860.1778.0075.3781.9774.4074.60
ICW (α in the paper)79.43 (1.25)61.86 (1.25)78.80 (1.75)76.67 (1.5)82.29 (1.75)76.19 (1.75)75.87
ICW (α selected on validation)79.43 (1.25)61.86 (1.25)78.80 (2)76.11 (1.25)82.60 (2)76.19 (1.75)75.83

3. The released code uses different α and τ values and adds benchmark-specific priority text. We sincerely apologize for this inconsistency. As clarified in our response to W1, the values of α in Table 9 were mistakenly labeled and the anonymous repository and prompts was mistakenly prepared from a wrong branch. We have corrected the hyperparameters (Table 16) and the prompts (Appendix D.2) in the paper, and updated the repository accordingly, so that both are fully consistent with the settings that produced the results reported in the paper. All changes in the paper are marked in blue.

[W3] In the RL study, GPT-4.1 is both the weighting model and the evaluation judge.

Response: Thank you for pointing out this issue! To rule out the bias from using the same model for weighting and judging, we re-evaluate all policy models with a judge from a different model family, i.e., Gemini-3.1-Pro, while keeping the policy models and the three evaluation benchmarks of Section 4.6 unchanged. Gemini-3.1-Pro is not involved in the weights or rewards of any method. As shown in the table below, ICW still outperforms Uniform on all benchmarks across all three policy models, and achieves the best results in 12 of 15 cases, demonstrating that the gains of ICW in the RL results are not due to using the same model for weighting and judging. We have replaced the RL results in Table 5 and the corresponding discussions in Section 4.6 with the new ones.

Policy ModelMethodBiGGen Bench (Score)AlpacaEval 2.0 (WR)AlpacaEval 2.0 (LC)Arena-Hard (WR)Arena-Hard (SC)
Qwen2.5-3B-InstructBase Model57.2311.5511.8815.0013.91
Qwen2.5-3B-InstructUniform59.7814.4113.1717.0816.97
Qwen2.5-3B-InstructRACE-SC60.6814.6613.4918.6917.63
Qwen2.5-3B-InstructRRD-WU60.8115.6514.9717.3916.36
Qwen2.5-3B-InstructICW61.9915.5315.1620.2518.20
Qwen2.5-7B-InstructBase Model67.0524.4722.4636.6636.56
Qwen2.5-7B-InstructUniform69.8335.0329.1541.6841.54
Qwen2.5-7B-InstructRACE-SC71.1534.1630.7042.8042.05
Qwen2.5-7B-InstructRRD-WU72.2435.5331.1241.8441.78
Qwen2.5-7B-InstructICW72.0837.1433.6044.3843.86
Qwen3-4BBase Model74.2532.6728.5851.6351.87
Qwen3-4BUniform76.2934.1633.3854.3654.15
Qwen3-4BRACE-SC77.5636.2735.0356.8855.43
Qwen3-4BRRD-WU77.2035.6534.1054.4253.83
Qwen3-4BICW78.5238.5136.5056.2955.64

Again, thank you so much for your careful reading of our paper and code! We are eager to hear your feedback. We'd deeply appreciate it if you could let us know whether your concerns have been addressed.

Reviewer_vMZyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper talks about a certain process to assign well structured weights to Rubrics used in LLM evaluation. Using uniform weights is clearly not optimal since some Rubrics are more important than others. Even when weights are used, there is not systematic to aligns weights across all Rubrics. Other methods use weighs from third party LLMs, but they tend to be untethered and may not give a comparatively sound weighing framework.

Here the paper talks about a certain weighing using third party LLMs. The third party LLM always judges Rubrics in pairs, with order well controlled. The judgments from the LLM is arranged into a AHP like Matrix with a certain ratio. The weights are then the principal eigenvector of the Matrix. Hence the language based preference from the LLMs are converted to numerical weights using this method. It also undergoes a consistency ratio check and recomputes by repeating the process, if its not satisfactory. The closest work is MAPLE, which uses similar techniques using Bradley Terry aggregation, but no consistency check.

In the experiments, they use six pairwise preference benchmarks. Models used are, Gpt4.1 for weighting and Gemini3.1FlashLite for verification. They compare against a wide suite of baselines. The results seems to show prominent wins across the board, on the performance of the technique, method based breakdown of metrics, generalizations etc.

Overall it finds that the reformulation of the problem in the weighing framework, seems to help the results.

Critical strengths
  • Clear well defined problem and a clean solution with good experiments which seem to show promising results.
  • Use of AHP framework for reformulation combined with Matrix eigenvector methods, seem to produce consistent weighting framework.
  • A good end to end framework automatic weighting of Rubrics.
  • Consistent wins in the results and good experiment deisgn.
  • Good explorations for peripheral ablations.
  • Downstream RL results make it much more convincing.
Critical weaknesses
  • The main contribution seems to be a reformulation technique which is marginal, although effective.
  • Quadratic cost in comparison and also the tokens costs for comparison/verification and revision process.
  • The statistical intervals of the wins are not mentioned and wins seems somewhat marginal, especially comparing different verifiiers.
  • Not clear how much advantange comes from prompt engineering vs the reformulation technique.
Questions for authors
  • Any studies done on correlation of ICW weights with human judgement or any correlations? Where do they differ or agree and what do we learn from it?
  • Consistency of weights may not indicate correctness. How do we clearly interpret this?
  • Any statistical confidence intervals that you have from your experiments? Especially important since the gains are somewhat close.
  • How much gain is purely coming from prompt engineering? how can measure this impact?
  • Can you give some rough sense of the token cost impact using your method as opposed to baselines?
  • Are the RL gains truly response quality gains? Or stylistic gains?
5 comments

We sincerely thank Reviewer vMZy for the positive and insightful review! We are really glad that you recognized our problem definition, weighting framework, as well as experimental design. Below, we address your concerns point by point.

[W1] The main contribution seems to be a reformulation technique which is marginal, although effective.

Response: Thank you for this sincere comment! We agree that ICW builds on the classical AHP in the LLM context. Nevertheless, we respectfully have reservations that this weakens the contribution of our paper, for two reasons:

  • Adapting classical techniques has brought valuable contributions to LLM evaluation. Many influential techniques in LLM reward modeling and evaluation are adapted from classical ones, e.g., the Bradley–Terry model and rubrics from psychometrics. All of them have brought important insights and improvements to LLM evaluation.
  • The standard AHP cannot be directly applied to rubric weighting. The standard AHP requires human experts to construct the pairwise comparison matrix, which limits its use in rubric weighting, especially when a large number of prompts need to be evaluated. We instead let an LLM construct the comparison matrix, where it only judges which of two criteria is more important and revises inconsistent comparisons. This achieves strong results in rubric weighting on six benchmarks and in downstream RL, which goes well beyond a reformulation.

[W2, Q5] Quadratic cost in comparison and also the tokens costs for comparison/verification and revision process. Can you give some rough sense of the token cost impact using your method as opposed to baselines?

Response: Thank you very much for your constructive comments! We agree that token cost is an important aspect of model evaluation. We also acknowledge that ICW compares every pair of criteria, so its cost grows quadratically with the number of criteria. We have discussed this limitation in Section 5, and we leave comparing only informative pairs as future work.

To further quantify the token cost impact, we additionally evaluated the token cost of weight generation for representative baselines: Uniform, Direct, and RRD-WU (the strongest baseline in terms of Macro accuracy). The results, in thousands of tokens per sample and averaged over the six benchmarks, are presented in the table below. While ICW consumes more tokens than the direct weighting method, it achieves the best overall performance. Moreover, ICW's token cost is lower than that of RRD-WU, while its performance is higher. These results demonstrate that ICW achieves a favorable balance between token cost and evaluation performance — it outperforms the strongest baseline (RRD-WU) at a relatively lower cost, and the additional tokens over direct weighting are justified by substantial performance gains. We added this comparison to Appendix C.3.8

MethodInput Tokens (k)Output Tokens (k)Total Tokens (k)Macro Acc
Uniform0.0000.0000.0000.7301
Direct0.4470.1730.6200.7321
RRD-WU39.910125.053164.9630.7466
ICW114.9382.080117.0170.7564

[W3, Q3] The statistical intervals of the wins are not mentioned and wins seems somewhat marginal, especially comparing different verifiers. Any statistical confidence intervals that you have from your experiments? Especially important since the gains are somewhat close.

Response: Thank you very much for your valuable and actionable comments! To address this concern, we have added confidence intervals in Appendix C.6. Specifically, we rerun all methods three times, where each run repeats both the weighting and the verification on the same inputs. The results (accuracy in %) are reported as mean ± std, where [mean − std, mean + std] corresponds to the 77.5% confidence interval of the mean under the t-distribution with two degrees of freedom. We further conduct a paired one-sided t-test over the three runs, and mark the improvements with p < 0.05 by * (against the best baseline in the first table and against Uniform in the second table).

The results demonstrate that the proposed ICW outperforms the best baseline on all included benchmarks, and the improvement is statistically significant on five of the six benchmarks and on the Macro average, which substantiates the effectiveness of ICW.

MethodRubricBenchResearchQAJudgeBenchRM-ChatChat-HardPrecise-IFMacro
Uniform77.14 ± 0.0957.69 ± 0.5976.32 ± 0.0973.73 ± 0.2780.59 ± 0.2972.08 ± 0.1072.93 ± 0.11
Direct77.54 ± 0.2458.19 ± 0.6276.99 ± 0.0573.04 ± 0.2780.34 ± 0.3272.01 ± 0.2673.02 ± 0.19
AdaRubric77.77 ± 0.3157.99 ± 1.0276.26 ± 0.2074.63 ± 0.3381.51 ± 0.2872.99 ± 0.3373.52 ± 0.19
RaR77.94 ± 0.1959.86 ± 0.6276.29 ± 0.0875.71 ± 0.3481.83 ± 0.3574.41 ± 0.2274.34 ± 0.18
MAPLE78.06 ± 0.1459.76 ± 0.7876.02 ± 0.3674.72 ± 0.2081.21 ± 0.2873.68 ± 0.3973.91 ± 0.18
RRD-WU78.19 ± 0.2459.66 ± 1.1277.26 ± 0.1475.88 ± 0.1581.62 ± 0.2574.69 ± 0.3874.55 ± 0.13
ChasingTail78.12 ± 0.0959.57 ± 0.7477.02 ± 0.0875.80 ± 0.2081.62 ± 0.3574.20 ± 0.1274.39 ± 0.13
RACE-SC77.99 ± 0.1359.66 ± 0.3476.77 ± 0.0875.67 ± 0.2081.21 ± 0.3275.59 ± 0.0674.48 ± 0.05
ICW79.64 ± 0.16*60.75 ± 0.45*78.41 ± 0.19*76.79 ± 0.15*82.09 ± 0.2876.01 ± 0.24*75.62 ± 0.06*

Across verifiers, ICW outperforms Uniform on all benchmarks with all five verifiers, and the Macro improvement is statistically significant for every verifier.

VerifierMethodRubricBenchResearchQAJudgeBenchRM-ChatChat-HardPrecise-IFMacro
Gemini-3.1-Flash-LiteUniform77.14 ± 0.0957.69 ± 0.5976.32 ± 0.0973.73 ± 0.2780.59 ± 0.2972.08 ± 0.1072.93 ± 0.11
Gemini-3.1-Flash-LiteICW79.64 ± 0.16*60.75 ± 0.45*78.41 ± 0.19*76.79 ± 0.15*82.09 ± 0.28*76.01 ± 0.24*75.62 ± 0.06*
DeepSeek-V4-FlashUniform78.73 ± 0.4458.28 ± 0.8970.73 ± 0.7069.47 ± 0.7879.46 ± 0.5673.26 ± 1.3671.65 ± 0.08
DeepSeek-V4-FlashICW79.93 ± 0.6861.93 ± 2.0171.91 ± 0.93*71.62 ± 0.54*81.36 ± 0.11*76.25 ± 1.33*73.83 ± 0.72*
Gemini-3.5-FlashUniform79.58 ± 0.3860.26 ± 0.1780.27 ± 0.0974.07 ± 0.6678.07 ± 0.4073.30 ± 0.4274.26 ± 0.29
Gemini-3.5-FlashICW80.76 ± 0.41*60.95 ± 0.3082.82 ± 0.16*74.25 ± 0.7882.02 ± 0.11*76.08 ± 0.51*76.15 ± 0.09*
GLM-5.1Uniform79.34 ± 0.2259.66 ± 1.4675.70 ± 0.7071.71 ± 0.5978.11 ± 0.7871.84 ± 1.4472.73 ± 0.16
GLM-5.1ICW79.70 ± 0.1461.34 ± 2.26*76.96 ± 0.47*74.03 ± 0.45*82.49 ± 0.73*75.35 ± 1.62*74.98 ± 0.38*
GPT-4.1-miniUniform76.05 ± 0.0758.09 ± 0.1759.09 ± 0.2566.75 ± 0.2079.53 ± 0.1763.78 ± 0.8167.22 ± 0.13
GPT-4.1-miniICW77.43 ± 0.22*58.28 ± 0.3060.40 ± 0.28*67.61 ± 0.33*81.58 ± 0.55*64.86 ± 0.68*68.36 ± 0.05*

[W4, Q4] Not clear how much advantange comes from prompt engineering vs the reformulation technique. How much gain is purely coming from prompt engineering? how can measure this impact?

Response: Thank you for your insightful comments on the impact of prompt engineering! We understand that the prompt largely affects the quality of LLM-based methods. To measure the gain of the technique rather than prompt engineering, we keep the prompt aligned and change only the technique, in two ways.

  • Aligned prompts among the compared methods. All LLM-based baselines share all the reusable rules of the pairwise importance comparison prompt of ICW (Prompt 1 in Appendix D.2), such as priority rules. Therefore, our comparison is fair and controls the quality of the prompt, and the advantage of ICW over the baselines indicates the advantage of the technique.
  • Same prompt rules with a different technique (Table 2). ICW† is a variant of ICW that keeps the same rules in the prompt, but asks the LLM to judge all criterion pairs within a single query instead of one pair per query. ICW† drops on all six benchmarks (e.g., from 0.783 to 0.758 on JudgeBench), which shows that the gain comes from pairwise importance comparison rather than the prompt.

[Q1] Any studies done on correlation of ICW weights with human judgement or any correlations? Where do they differ or agree and what do we learn from it?

Response: Thank you so much for your kind and inspiring question! We agree that adding the correlation with human-evaluated weights is beneficial to the technical quality of this manuscript. However, it is genuinely difficult to collect human-evaluated weights. Even identifying their oracle values would be infeasible in many cases, which makes reliable human-derived weights hard to obtain.

To bypass this difficulty, we resort to the pairwise preference accuracy to evaluate the weights. The rationale is that human preferences between two responses are easier to collect and more reliable, since oracle values are available. Since all methods share the same rubric sets and verification results and differ only in the weights, the differences can be attributed to the advantage of the weighting mechanism. The downstream RL study further substantiates the advantage of the proposed mechanism.

We hope this explanation clarifies the situation and mitigates the concern. We would be happy to discuss this point further with you.

[Q2] Consistency of weights may not indicate correctness. How do we clearly interpret this?

Response: Thank you for highlighting the important issue of consistency and correctness. Your understanding is absolutely correct: consistency is not a sufficient condition for correctness, since consistent comparisons can still be wrong. Nevertheless, consistency is a necessary condition for correctness, because contradictory comparisons cannot all be correct. For example, if cic_i is judged more important than cjc_j, cjc_j more important than cℓc_\ell, and cℓc_\ell more important than cic_i, no weights can reproduce the three comparisons, so at least one of them must be wrong. This is exactly the motivation of the consistency check in our paper: ICW checks the consistency of the comparisons to detect such contradictions, and prompts the LLM to revise them (Section 3.4).

[Q6] Are the RL gains truly response quality gains? Or stylistic gains?

Response: Thank you for raising this important point regarding the RL gains! To separate response quality from style, our RL evaluation (Table 5) reports two metrics that control for stylistic factors. Specifically, LC is the length-controlled win rate of AlpacaEval 2.0, which removes the preference of the judge for longer responses, and SC is the style-controlled win rate of Arena-Hard, which controls for style features such as response length and markdown formatting. As shown in the table below, ICW achieves the best LC and SC on all three policy models, demonstrating that the RL gains are response quality gains rather than stylistic gains.

MethodLC (Qwen2.5-3B)LC (Qwen2.5-7B)LC (Qwen3-4B)SC (Qwen2.5-3B)SC (Qwen2.5-7B)SC (Qwen3-4B)
Base Model11.8822.4628.5813.9136.5651.87
Uniform13.1729.1533.3816.9741.5454.15
RACE-SC13.4930.7035.0317.6342.0555.43
RRD-WU14.9731.1234.1016.3641.7853.83
ICW15.1633.6036.5018.2043.8655.64

Again, thank you so much for your insightful comments and questions! We are eager to hear your feedback. We'd deeply appreciate it if you could let us know whether your concerns have been addressed.

Reviewer_6FAeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to weight the criteria in rubric-based LLM evaluation. The authors propose ICW, which asks an LLM to compare the importance of each pair of criteria and derives the weights using an AHP-inspired method. They show gains over uniform weighting and other weighting methods on six preference benchmarks, and use the weighted scores as rewards for RL training.

Critical strengths
  1. The paper is clearly written and easy to follow.

  2. The method is simple and needs no training, and the weights are shared across responses of the same prompt. This makes it easy to add to existing rubric-based pipelines.

Critical weaknesses

My main concerns are on the experiment side:

  1. A key baseline is missing. In Table 1, all methods use the same rubrics-style evaluations. The paper never tries the simplest alternative: give LM the prompt, the rubric set (optional), and both responses, and let it pick the better one. RaR (Gunjal et al., 2026) already includes this setting (implicit aggregation), but the paper only borrows its weighting part. This matters because ICW needs up to 2K(K−1) extra LM calls per prompt, while the direct judgment needs only one call and sees the same information. Without this comparison, it is unclear whether ICW gives better evaluation than simply asking the same model.

  2. The RL results (Section 4.6) may be biased by using the same model for weighting and judging. The ICW weights are produced by GPT-4.1, and GPT-4.1 is also the judge on all three evaluation benchmarks. In other words, the reward already reflects what GPT-4.1 considers important, and the trained policy is then scored by GPT-4.1 again. This mainly affects the comparison with Uniform, which does not use GPT-4.1's judgments. This matters because the gains in Table 1 are small (0.2 points on ResearchQA and Chat-Hard). A judge from a different model family, or human evaluation on a subset, is needed.

Questions for authors
  1. How does ICW compare with weights learned from a small amount of preference data? For example, one could fit a BT model on the verification results of a held-out preference split.

  2. For the benchmarks without given rubrics, why are the rubrics generated by Rubric-ARM (or Gemini-3.5-Flash in RL), but the weights by GPT-4.1? A simpler option is to let GPT-4.1 write both the rubric and the weights, either in two steps or in one, which becomes close to asking GPT-4.1 directly which response is better. What is the benefit of using separate steps and different models? A comparison with these simpler options would help.

6 comments

Thank you so much for your great efforts and insightful comments to improve our manuscript! We are really glad that you found our paper clearly written and easy to follow, and our method easy to add to existing rubric-based pipelines. Below, we address your concerns point by point.


[W1] A key baseline is missing. In Table 1, all methods use the same rubrics-style evaluations. The paper never tries the simplest alternative: give LM the prompt, the rubric set (optional), and both responses, and let it pick the better one. RaR (Gunjal et al., 2026) already includes this setting (implicit aggregation), but the paper only borrows its weighting part. This matters because ICW needs up to 2K(K−1)2K(K-1) extra LM calls per prompt, while the direct judgment needs only one call and sees the same information. Without this comparison, it is unclear whether ICW gives better evaluation than simply asking the same model.

Response: Thank you very much for pointing out this key missing baseline! We add the simplest alternative, i.e., Direct Judge, as a new baseline in Appendix C.3 to show whether ICW gives better evaluation than simply asking the same model. Specifically, we give the same LM (GPT-4.1, our weighting model) the prompt, the rubric set, and both responses, and let it pick the better one in only one call, which corresponds to the implicit aggregation setting of RaR.

MethodRubricBenchResearchQAJudgeBenchRM-ChatChat-HardPrecise-IFMacro
Direct Judge0.72580.62720.66610.74550.80480.62080.6984
ICW0.79470.61240.78310.76870.81800.76150.7564

We found ICW outperforms Direct Judge on 5 of 6 benchmarks and on the Macro accuracy, demonstrating that ICW gives better evaluation than simply asking the same model. We added this experiment and discussions in Appendix C.3.

[W2] The RL results (Section 4.6) may be biased by using the same model for weighting and judging. The ICW weights are produced by GPT-4.1, and GPT-4.1 is also the judge on all three evaluation benchmarks. In other words, the reward already reflects what GPT-4.1 considers important, and the trained policy is then scored by GPT-4.1 again. This mainly affects the comparison with Uniform, which does not use GPT-4.1's judgments. This matters because the gains in Table 1 are small (0.2 points on ResearchQA and Chat-Hard). A judge from a different model family, or human evaluation on a subset, is needed.

Response: Thank you for raising this important point on the judge of the RL results! To rule out the bias from using the same model for weighting and judging, we replace the judge GPT-4.1 with Gemini-3.1-Pro (a judge from a different model family) and re-evaluate all policy models, and replace the RL results in Table 5 with the new ones.

As shown in the table below, ICW still outperforms Uniform on all benchmarks across all three policy models, and achieves the best results in 12 of 15 cases, demonstrating that the gains of ICW in the RL results are not due to using the same model for weighting and judging.

MethodBiGGen Bench (Score)AlpacaEval 2.0 (WR)AlpacaEval 2.0 (LC)Arena-Hard (WR)Arena-Hard (SC)
Qwen2.5-3B-Instruct
Base Model57.2311.5511.8815.0013.91
Uniform59.7814.4113.1717.0816.97
RACE-SC60.6814.6613.4918.6917.63
RRD-WU60.8115.6514.9717.3916.36
ICW61.9915.5315.1620.2518.20
Qwen2.5-7B-Instruct
Base Model67.0524.4722.4636.6636.56
Uniform69.8335.0329.1541.6841.54
RACE-SC71.1534.1630.7042.8042.05
RRD-WU72.2435.5331.1241.8441.78
ICW72.0837.1433.6044.3843.86
Qwen3-4B
Base Model74.2532.6728.5851.6351.87
Uniform76.2934.1633.3854.3654.15
RACE-SC77.5636.2735.0356.8855.43
RRD-WU77.2035.6534.1054.4253.83
ICW78.5238.5136.5056.2955.64

[Q1] How does ICW compare with weights learned from a small amount of preference data? For example, one could fit a BT model on the verification results of a held-out preference split.

Response: Thank you for this insightful and actionable suggestion! We agree that it is important to compare ICW with weights learned from preference data, and we add a preference-trained criterion-weighting baseline (BT).

Specifically, we train a shared linear head on frozen text-embedding-3-large embeddings (3072 dimensions) of the prompt, the rubric set, and each criterion to predict the weight of that criterion. The predicted weights are combined with the same binary verification results as ICW to score each response, and the head is trained on human preference labels with a Bradley–Terry (BT) loss. For each benchmark, we use a 20%/10%/70% train/validation/test split grouped by prompt, and compare BT with Uniform and ICW on the same 70% test partition.

As shown in the table below, BT improves over Uniform on four benchmarks, and ICW further outperforms BT on all six benchmarks. This demonstrates that ICW, which needs no training, compares favorably with weights learned from a small amount of preference data. We added this experiment and discussions in Appendix C.5.

MethodRubricBenchResearchQAJudgeBenchRM-ChatChat-HardPrecise-IFMacro
Uniform0.76180.57630.76380.75000.79150.72770.7285
BT0.77120.57630.78210.76300.79000.74850.7385
ICW0.78930.61020.79010.77040.81190.75890.7551

[Q2] For the benchmarks without given rubrics, why are the rubrics generated by Rubric-ARM (or Gemini-3.5-Flash in RL), but the weights by GPT-4.1? A simpler option is to let GPT-4.1 write both the rubric and the weights, either in two steps or in one, which becomes close to asking GPT-4.1 directly which response is better. What is the benefit of using separate steps and different models? A comparison with these simpler options would help.

Response: Thank you for this thoughtful question! We generate the rubrics with Rubric-ARM because it writes better rubrics than a general-purpose model such as GPT-4.1 (see the second observation below). Notably, this does not affect the fairness of our comparison: we focus on comparing the weighting methods, and all weighting methods share the same rubric sets, so the rubric generator (Rubric-ARM, or Gemini-3.5-Flash in RL) is a controlled variable.

Following your suggestion, we add a comparison with the simpler options that let GPT-4.1 write both the rubric and the weights on the four benchmarks without given rubrics. GPT-4.1 writes the rubric and then the weights (Two-step Direct), writes both in one step (One-step Joint), or writes the rubric and derives the weights by ICW (Two-step ICW). The results are presented in the table below, with two observations as follows.

  • With GPT-4.1 for both the rubric and the weights, Two-step ICW outperforms the two simpler options Two-step Direct and One-step Joint on all four benchmarks, which demonstrates the advantage of ICW as a weighting method.
  • With the same ICW weighting, the rubrics of Rubric-ARM outperform those of GPT-4.1 on three benchmarks and are on par on Precise-IF, which we attribute to Rubric-ARM being specifically trained for rubric generation.
MethodRubric GeneratorWeighting ModelJudgeBenchRM-ChatChat-HardPrecise-IF
Two-step DirectGPT-4.1GPT-4.10.70240.66930.80590.7167
One-step JointGPT-4.1GPT-4.10.70240.65630.79280.7375
Two-step ICWGPT-4.1GPT-4.10.71940.68220.81140.7615
ICW (paper)Rubric-ARMGPT-4.10.78310.76870.81800.7615

We added this experiment and discussions in Appendix C.4.


Again, thank you so much for your insightful and actionable suggestions! We are eager to hear your feedback. We'd deeply appreciate it if you could let us know whether your concerns have been addressed.

Dear Reviewer 6FAe,

Once again, we are grateful for your time and effort for reviewing our paper. Since the discussion period will end in a few days, we are very eager to get your feedback on our response. We understand that you are very busy, but we would highly appreciate it if you could take into account our response when updating the rating and having a discussion with AC and other reviewers.

Thanks for your time,

Authors of # 57387

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

Dear AC and all reviewers,

We sincerely appreciate your commitment to evaluating our paper despite your busy schedules. We are encouraged that 2 out of 3 reviewers are currently on the positive side, with scores of 2 (confidence 3), 3 (confidence 3), and 3 (confidence 3), recognizing that our paper:

  • “is clearly written and easy to follow”, with a method that “is simple and needs no training” and is “easy to add to existing rubric-based pipelines” (Reviewer 6FAe);
  • presents a “clear well defined problem and a clean solution with good experiments” and “a good end to end framework”, with “consistent wins in the results and good experiment design”, “good explorations for peripheral ablations”, and downstream RL results that “make it much more convincing” (Reviewer vMZy);
  • addresses “a real weakness of rubric judges” and “states the problem clearly”, where “the setup is clean”, “the baseline set is broad”, and “the ablations over verifiers and weighting models are useful” (Reviewer wKaP).

In response to the reviewers’ valuable feedback, we have made the following revisions and clarifications to further improve the manuscript:

  • [Reviewer 6FAe]. As suggested, we added comparisons with Direct Judge, single-model pipelines, and preference-trained weights (Appendices C.3–C.5), where ICW achieves higher average accuracy in every comparison, and re-evaluated all policy models with a judge from a different model family (Table 5), where ICW still achieves the best results in most cases.
  • [Reviewer vMZy]. We reported the results over three runs with significance tests (Appendix C.6), where the overall advantage of ICW is stable and statistically significant, and compared the token cost with representative baselines (Appendix C.8), where ICW outperforms the strongest baseline at a lower token cost; the remaining questions are clarified in our response.
  • [Reviewer wKaP]. We analyzed the consistency check and revision with a four-case breakdown (Appendix C.7), where the revision mainly corrects genuine contradictions, and stated explicitly in Section 3.4 and the limitations that the check is a proxy rather than a precise contradiction detector.

After rebuttal, we noticed that Reviewer 6FAe appreciated our effort to address the concerns and indicated a willingness to recalibrate the rating during the AC–reviewer discussion period; Reviewer vMZy maintained a solid positive assessment; Reviewer wKaP carefully read our rebuttal and generously improved the rating. We sincerely thank all reviewers for their effort, thoughtfulness, and generosity in evaluating this paper.

Many thanks,

Authors of #57387

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.