- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method that creates better evaluation rubrics for questions, which has applications in both LLM-as-a-judge evaluation and rubric-based RL training. In particular, it proactively prompts LLMs to detect the defect in the current rubric and exploit it to produce reward-hacked responses. The responses are used to propose a new criterion that patches the defect, which is subsequently incorporated into the rubric if it passes the validation check. Experiments show that the method produces better rubrics that achieve higher pairwise judging accuracy on LLM-as-a-judge benchmarks and also enable more effective rubric-based RL training.
Critical strengths
- The method is conceptually simple and well-motivated. Reward hacking is a known behavior of LLMs in rubric-based evaluation, and this method addresses it by finding defects adversarially and then patching them one by one.
- The experiments are fairly comprehensive, covering both direct application in LLM-as-a-judge evaluation and rubric-based RL training, and they also include multiple benchmarks, models, and baselines for each setting.
Critical weaknesses
- There is no guarantee that the rubric can patch the reward hacking behavior during RL training. Defects are detected by a static attacker, but as RL training progresses, the evolving policy may have new hacking behavior not covered by the rubric. In addition, the attacker uses a prompt that is intentionally adversarial when constructing the reward-hacking responses. In RL training, the policy is given a simple prompt, so its reward-hacking behavior could be different, more subtle, harder to detect, and not necessarily covered by the rubric produced by the method.
- The cost is relatively high, with multiple patching iterations and each iteration involving a sequence of attack, verify, repair, and review steps. This is especially true when a proprietary model such as Gemini-3.1-Flash-Lite.
Questions for authors
- Experiments on accuracy versus number of criteria in Section 4.5 confuse me. In Section 3.2.2, your pipeline requires that the existing valid criteria be preserved as new criteria are added. If that is the case, higher accuracy has to come from more new criteria. Can you explain why in your experiment, accuracy increases while the number of criteria remains largely constant?
4 comments
[W2] The cost is relatively high, with multiple patching iterations and each iteration involving a sequence of attack, verify, repair, and review steps. This is especially true when a proprietary model such as Gemini-3.1-Flash-Lite.
Response: Thank you for your sincere and meticulous comment. We agree that RubricArmor requires many LLM calls than single-pass rubric generation, and discussed it in the Limitations & Future Work paragraph. However, its token cost is comparable to that of existing iterative refinement methods, and it also works well with low-cost backbones.
-
To further quantify the token cost, we add a comparison of the average total tokens for rubric generation per query between RubricArmor and the two iterative refinement baselines, ChasingTail and RRD-Uniform. We also add a compute-matched baseline, Iterative-Rewrite, which iteratively rewrites the entire rubric with a fixed prompt at a token budget comparable to RubricArmor's. The results are in Table R3, with two observations as follows.
- RubricArmor uses fewer tokens than Iterative-Rewrite and RRD-Uniform on all three benchmarks, and a comparable number of tokens to ChasingTail.
- At this cost, RubricArmor achieves the highest accuracy on all three benchmarks.
The cost of RubricArmor is thus not high relative to existing iterative refinement methods: a comparable token budget simply yields higher accuracy when it is spent on attack–repair.
Table R3: Accuracy (%) and average total tokens per query. Iterative-Rewrite is a compute-matched baseline.
Method JudgeBench Tokens JudgeBench Acc. RM-Bench Tokens RM-Bench Acc. Chat-Hard Tokens Chat-Hard Acc. Iterative-Rewrite 102.20K 69.92 99.03K 77.15 100.92K 75.55 RRD-Uniform 240.01K 71.94 242.74K 77.42 255.35K 69.74 ChasingTail 100.39K 71.37 79.35K 81.51 64.60K 72.37 RubricArmor 93.92K 73.63 85.96K 83.69 80.59K 78.73 -
In terms of monetary cost, as shown in Table 3 of the paper (reproduced as Table R4 below), RubricArmor remains effective with other low-cost or open-weight backbones, such as DeepSeek-v4-Flash and Qwen3.5-397B-A17B, so it does not rely on a particular expensive model.
Table R4: Accuracy (%) under different rubric generator backbones.
Method JudgeBench RM-Bench Chat-Hard Backbone: DeepSeek-v4-Flash Direct 83.47 83.35 74.34 RubricArmor 87.50 85.92 80.48 Backbone: Qwen3.5-397B-A17B Direct 67.66 80.04 75.77 RubricArmor 76.05 85.04 78.51
[Q1] Experiments on accuracy versus number of criteria in Section 4.5 confuse me. In Section 3.2.2, your pipeline requires that the existing valid criteria be preserved as new criteria are added. If that is the case, higher accuracy has to come from more new criteria. Can you explain why in your experiment, accuracy increases while the number of criteria remains largely constant?
Response: Thank you for this thoughtful question and we apologize for the confusion. We also noticed this phenomenon and would like to clarify that the repair does not necessarily add criteria: it can also refine existing criteria in place or remove incorrect or redundant ones. Accordingly, "the existing valid criteria be preserved" in our pipeline means that the requirements checked by these criteria remain covered by the revised rubric as a whole, rather than that each original criterion is kept verbatim. Therefore, it is natural to observe that accuracy increases while the number of criteria remains largely constant. We add a sentence to Section 3.2.2 to state this explicitly.
Moreover, the growth in the number of criteria varies with the task type. For many JudgeBench, RM-Bench, and Chat-Hard queries (e.g., a math problem or a coding task), a good response does not need to cover many points and the Direct rubric usually lists them already, so the repair refines these criteria more often than it adds new ones. In contrast, the Health and Science queries in Section 4.7 require a response to cover many points that the Direct rubric omits, so the repair adds more new criteria. We additionally counts the average number of criteria on these queries and found that the number of criteria grows clearly: 5.05 → 8.15 on Health and 4.79 → 6.31 on Science.
Again, thank you so much for your insightful comments! We are eager to hear your feedback. We’d deeply appreciate it if you could let us know whether your concerns have been addressed.
Dear Reviewer qJnE,
Once again, we are grateful for your time and effort for reviewing our paper. Since the discussion period will end in a few days, we are very eager to get your feedback on our response. We understand that you are very busy, but we would highly appreciate it if you could take into account our response when updating the rating and having a discussion with AC and other reviewers.
Thanks for your time,
Authors of # 56681
Dear Reviewer qJnE,
Since the discussion period will end in around a day, we will be online waiting for your feedback on our rebuttal, which we believe has fully addressed your concerns.
We would highly appreciate it if you could take into account our response when updating the rating and having discussions with AC and other reviewers.
Thank you so much for your time and efforts. Sorry for our repetitive messages, but we're eager to ensure everything is addressed.
Authors of # 56681
We sincerely thank Reviewer qJnE for the detailed and constructive review! We are encouraged that you find our method conceptually simple and well-motivated and our experiments comprehensive. Below, we address your concerns point by point.
Response:
Thank you for your careful reading and actionable suggestion! We would address this insightful concern through efforts as follows.
Table R1: Rubric reward and response quality of training rollouts.
Table R2: Benchmark performance (%, ↑) during RL.