acceptodds
Under review as a conference paper at ICLR 2027

RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation

Abstract

Rubric-based reinforcement learning (RL) provides interpretable rewards for aligning large language models (LLMs) by evaluating responses against query-specific evaluation criteria. To construct rubrics at scale, a straightforward approach to LLM-based rubric generation is to prompt an LLM to generate a rubric directly from the query. However, rubrics directly generated by LLMs are vulnerable to reward hacking, since omitted or underspecified criteria allow the policy to obtain high rubric rewards with low-quality responses. Existing LLM-based rubric generation methods improve the granularity and coverage of the generated criteria but do not proactively guard against reward hacking. To address this limitation, we propose RubricArmor, an adversarial framework that exposes and mitigates potential reward hacking at the rubric generation stage before it occurs in subsequent RL. Specifically, RubricArmor performs adversarial evolution, in which an attack step and a repair step alternate over multiple rounds. The attack step simulates the reward hacking of the policy by constructing adversarial responses that satisfy the current rubric but fail to properly complete the task. The repair step then revises the rubric to detect the response defects exposed by the attack step while preserving other valid criteria. Extensive experiments demonstrate that RubricArmor outperforms competitive rubric generation baselines and translates into more effective downstream rubric-based RL. Code is available at https://anonymous.4open.science/r/adversarial-283B.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
large language model, reward modeling, rubrics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qJnEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method that creates better evaluation rubrics for questions, which has applications in both LLM-as-a-judge evaluation and rubric-based RL training. In particular, it proactively prompts LLMs to detect the defect in the current rubric and exploit it to produce reward-hacked responses. The responses are used to propose a new criterion that patches the defect, which is subsequently incorporated into the rubric if it passes the validation check. Experiments show that the method produces better rubrics that achieve higher pairwise judging accuracy on LLM-as-a-judge benchmarks and also enable more effective rubric-based RL training.

Critical strengths
  • The method is conceptually simple and well-motivated. Reward hacking is a known behavior of LLMs in rubric-based evaluation, and this method addresses it by finding defects adversarially and then patching them one by one.
  • The experiments are fairly comprehensive, covering both direct application in LLM-as-a-judge evaluation and rubric-based RL training, and they also include multiple benchmarks, models, and baselines for each setting.
Critical weaknesses
  • There is no guarantee that the rubric can patch the reward hacking behavior during RL training. Defects are detected by a static attacker, but as RL training progresses, the evolving policy may have new hacking behavior not covered by the rubric. In addition, the attacker uses a prompt that is intentionally adversarial when constructing the reward-hacking responses. In RL training, the policy is given a simple prompt, so its reward-hacking behavior could be different, more subtle, harder to detect, and not necessarily covered by the rubric produced by the method.
  • The cost is relatively high, with multiple patching iterations and each iteration involving a sequence of attack, verify, repair, and review steps. This is especially true when a proprietary model such as Gemini-3.1-Flash-Lite.
Questions for authors
  • Experiments on accuracy versus number of criteria in Section 4.5 confuse me. In Section 3.2.2, your pipeline requires that the existing valid criteria be preserved as new criteria are added. If that is the case, higher accuracy has to come from more new criteria. Can you explain why in your experiment, accuracy increases while the number of criteria remains largely constant?
4 comments

We sincerely thank Reviewer qJnE for the detailed and constructive review! We are encouraged that you find our method conceptually simple and well-motivated and our experiments comprehensive. Below, we address your concerns point by point.


[W1] There is no guarantee that the rubric can patch the reward hacking behavior during RL training. Defects are detected by a static attacker, but as RL training progresses, the evolving policy may have new hacking behavior not covered by the rubric. In addition, the attacker uses a prompt that is intentionally adversarial when constructing the reward-hacking responses. In RL training, the policy is given a simple prompt, so its reward-hacking behavior could be different, more subtle, harder to detect, and not necessarily covered by the rubric produced by the method.

Response:

Thank you for your careful reading and actionable suggestion! We would address this insightful concern through efforts as follows.

  • First, we add experiments demonstrating that RubricArmor, with adversarial prompt,effectively mitigates reward hacking during RL training. Specifically, we extend GRPO training to 496 steps, treating the divergence between rising reward and declining benchmark performance as the diagnostic signature of reward hacking. We train policy models under both Direct and RubricArmor rubrics and track: (i) rubric reward and response quality on the same training rollouts, with quality scored on a 1–7 scale by Opus 4.6 blind to the rubric, and (ii) benchmark performance on HealthBench-Hard (Health) and the ResearchQA validation split (Science). The results presented in the table below demonstrate that RubricArmor effectively mitigates reward hacking during RL training. Specifically, as training progresses, Direct exhibits the reward-hacking signature: reward up, quality down, benchmark scores peaking around steps 160–240 and then declining. In contrast, RubricArmor improves rubric reward and response quality together across all three settings. These results and discussions are available in Section 4.7 in the revised paper.
  • Second, we acknowledge that RubricArmor may not capture all reward hacking behaviors that arise during RL. Nevertheless, this limitation does not diminish its value: while residual hacking may persist, a substantial portion is preempted, which has been a meaningful improvement over existing approaches. Moreover, the TableR1 results confirm that RubricArmor mitigates reward hacking despite this limitation. We have noted this in the Limitations & Future Work paragraph (Section 5).

Table R1: Rubric reward and response quality of training rollouts.

DirectRubricArmor
GRPO iteration stepsReward (%)Quality (1–7)↑Reward (%)Quality (1–7)↑
Health / Qwen3-4B
073.934.1966.814.19
12579.384.3370.615.20
25085.883.7872.705.48
37587.632.6575.615.19
49693.782.8081.646.24
Science / Qwen3-4B
075.704.0263.654.30
12572.483.9174.565.02
25081.613.8271.145.02
37579.123.4168.474.83
49688.003.6873.945.25
Science / Qwen2.5-7B
058.883.4451.703.43
12560.703.3662.264.37
25079.883.0261.773.96
37582.262.3658.123.86
49691.582.1263.654.21

Table R2: Benchmark performance (%, ↑) during RL.

StepDirectRubricArmor
Health / Qwen3-4B (HealthBench-Hard)
07.087.08
809.3210.71
16014.7415.97
24014.9717.81
32014.2522.97
40011.9023.95
48011.0723.00
49610.9323.24
Science / Qwen3-4B (ResearchQA)
065.6265.62
8067.2568.56
16069.8371.06
24071.5772.48
32070.1675.88
40068.9977.52
48068.6375.39
49667.9576.39
Science / Qwen2.5-7B (ResearchQA)
052.0052.00
8060.6162.74
16067.5073.20
24066.3677.32
32063.5576.68
40058.1876.93
48056.4076.92
49654.2676.59

[W2] The cost is relatively high, with multiple patching iterations and each iteration involving a sequence of attack, verify, repair, and review steps. This is especially true when a proprietary model such as Gemini-3.1-Flash-Lite.

Response: Thank you for your sincere and meticulous comment. We agree that RubricArmor requires many LLM calls than single-pass rubric generation, and discussed it in the Limitations & Future Work paragraph. However, its token cost is comparable to that of existing iterative refinement methods, and it also works well with low-cost backbones.

  • To further quantify the token cost, we add a comparison of the average total tokens for rubric generation per query between RubricArmor and the two iterative refinement baselines, ChasingTail and RRD-Uniform. We also add a compute-matched baseline, Iterative-Rewrite, which iteratively rewrites the entire rubric with a fixed prompt at a token budget comparable to RubricArmor's. The results are in Table R3, with two observations as follows.

    • RubricArmor uses fewer tokens than Iterative-Rewrite and RRD-Uniform on all three benchmarks, and a comparable number of tokens to ChasingTail.
    • At this cost, RubricArmor achieves the highest accuracy on all three benchmarks.

    The cost of RubricArmor is thus not high relative to existing iterative refinement methods: a comparable token budget simply yields higher accuracy when it is spent on attack–repair.

    Table R3: Accuracy (%) and average total tokens per query. Iterative-Rewrite is a compute-matched baseline.

    MethodJudgeBench TokensJudgeBench Acc.RM-Bench TokensRM-Bench Acc.Chat-Hard TokensChat-Hard Acc.
    Iterative-Rewrite102.20K69.9299.03K77.15100.92K75.55
    RRD-Uniform240.01K71.94242.74K77.42255.35K69.74
    ChasingTail100.39K71.3779.35K81.5164.60K72.37
    RubricArmor93.92K73.6385.96K83.6980.59K78.73
  • In terms of monetary cost, as shown in Table 3 of the paper (reproduced as Table R4 below), RubricArmor remains effective with other low-cost or open-weight backbones, such as DeepSeek-v4-Flash and Qwen3.5-397B-A17B, so it does not rely on a particular expensive model.

    Table R4: Accuracy (%) under different rubric generator backbones.

    MethodJudgeBenchRM-BenchChat-Hard
    Backbone: DeepSeek-v4-Flash
    Direct83.4783.3574.34
    RubricArmor87.5085.9280.48
    Backbone: Qwen3.5-397B-A17B
    Direct67.6680.0475.77
    RubricArmor76.0585.0478.51

[Q1] Experiments on accuracy versus number of criteria in Section 4.5 confuse me. In Section 3.2.2, your pipeline requires that the existing valid criteria be preserved as new criteria are added. If that is the case, higher accuracy has to come from more new criteria. Can you explain why in your experiment, accuracy increases while the number of criteria remains largely constant?

Response: Thank you for this thoughtful question and we apologize for the confusion. We also noticed this phenomenon and would like to clarify that the repair does not necessarily add criteria: it can also refine existing criteria in place or remove incorrect or redundant ones. Accordingly, "the existing valid criteria be preserved" in our pipeline means that the requirements checked by these criteria remain covered by the revised rubric as a whole, rather than that each original criterion is kept verbatim. Therefore, it is natural to observe that accuracy increases while the number of criteria remains largely constant. We add a sentence to Section 3.2.2 to state this explicitly.

Moreover, the growth in the number of criteria varies with the task type. For many JudgeBench, RM-Bench, and Chat-Hard queries (e.g., a math problem or a coding task), a good response does not need to cover many points and the Direct rubric usually lists them already, so the repair refines these criteria more often than it adds new ones. In contrast, the Health and Science queries in Section 4.7 require a response to cover many points that the Direct rubric omits, so the repair adds more new criteria. We additionally counts the average number of criteria on these queries and found that the number of criteria grows clearly: 5.05 → 8.15 on Health and 4.79 → 6.31 on Science.


Again, thank you so much for your insightful comments! We are eager to hear your feedback. We’d deeply appreciate it if you could let us know whether your concerns have been addressed.

Dear Reviewer qJnE,

Once again, we are grateful for your time and effort for reviewing our paper. Since the discussion period will end in a few days, we are very eager to get your feedback on our response. We understand that you are very busy, but we would highly appreciate it if you could take into account our response when updating the rating and having a discussion with AC and other reviewers.

Thanks for your time,

Authors of # 56681

Dear Reviewer qJnE,

Since the discussion period will end in around a day, we will be online waiting for your feedback on our rebuttal, which we believe has fully addressed your concerns.

We would highly appreciate it if you could take into account our response when updating the rating and having discussions with AC and other reviewers.

Thank you so much for your time and efforts. Sorry for our repetitive messages, but we're eager to ensure everything is addressed.

Authors of # 56681

Reviewer_r8MSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes RubricArmor to mitigate reward hacking in rubric-based RL caused by incomplete or underspecified rubrics. It uses an iterative attack-and-repair process to find low-quality responses that still pass the current rubric and then refine the rubric accordingly. The idea is sensible and practically relevant, but the current evidence for reduced exploitability is still somewhat indirect, and the paper would benefit from clearer writing and more polished figures. If the authors can improve the presentation, especially the figures, I would be willing to raise my score.

Critical strengths

I) The paper addresses the reward hacking that may result from rubric defects by first constructing low-quality answers that can fool the current rubric, and then modifying the scoring criteria accordingly. The approach is clear and well-matched to the problem. II) The paper compares various rubric generation methods across multiple benchmarks and further utilizes the generated rubrics for GRPO training; consistent performance improvements were achieved across several policy models, demonstrating the method's practical value.

Critical weaknesses

I) There is insufficient direct verification regarding reward hacking. The current results show better rubric quality and downstream performance, but they do not directly show that the final rubric is harder to exploit. II) The verifier and multi-agent setup are not fully independent. The appendix mainly tests different verifiers on fixed rubrics, while the agents also share the same backbone, which may introduce common blind spots. III) The comparison with baselines is not fully compute-matched. Since RubricArmor uses multiple rounds of LLM calls, it is unclear how much of the gain comes from the method itself versus the extra inference budget. IV) The paper shows some signs of AI-generated presentation. The figures are rather basic, and some punctuation and formatting are inconsistent. The omitted few-shot examples may also affect reproducibility.

Questions for authors

I) Explain why preference accuracy and downstream performance are sufficient to support the paper's core conclusions regarding reward hacking mitigation. II) Explain to what extent current experiments on verifier generalization reflect the robustness of the full evolution process. Additionally, share the same backbone lead to shared blind spots. III) Clarify whether the performance gains of RubricArmor stem primarily from adversarial refinement itself, or if they might also partially result from increased LLM calls and a larger inference budget? IV) Clarify whether the omitted few-shot examples affect reproducibility and further improve the figures, writing, and formatting in the final version.

7 comments

We sincerely appreciate your great efforts and insightful comments to improve our manuscript! We are also grateful for your openness to raising the score. Below, we address your concerns point by point and indicate the corresponding changes in the revised paper (revisions are highlighted in blue).

[W1, Q1] There is insufficient direct verification regarding reward hacking. The current results show better rubric quality and downstream performance, but they do not directly show that the final rubric is harder to exploit. Explain why preference accuracy and downstream performance are sufficient to support the paper's core conclusions regarding reward hacking mitigation.

Response: Thank you very much for your meticulous and actionable suggestion! We understand that preference accuracy and downstream performance demonstrate the effect of mitigating reward-hacking in a indirect manner.

To address this limitation, we add experiments designed specifically to test RubricArmor's resistance to reward hacking during policy optimization. Specifically, we extend GRPO training to 496 steps, treating the divergence between rising reward and declining benchmark performance as the diagnostic signature of reward hacking. We train policy models under both Direct and RubricArmor rubrics and track: (i) rubric reward and response quality on the same training rollouts, with quality scored on a 1–7 scale by Opus 4.6 blind to the rubric, and (ii) benchmark performance on HealthBench-Hard (Health) and the ResearchQA validation split (Science).

The results are presented in the table below, which demonstrates that the rubrics generated by RubricArmor is more difficult to hack during policy optimization. Under extended RL, Direct exhibits the reward-hacking signature: reward up, quality down, benchmark scores peaking around steps 160–240 and then declining; while RubricArmor raises rubric reward and response quality together across all three settings. Detailed results are shown in Section 4.7 and Figure 4 of the revised paper.

Table R1: Rubric reward and response quality of training rollouts.

DirectRubricArmor
GRPO iteration stepsReward (%)Quality (1–7)↑Reward (%)Quality (1–7)↑
Health / Qwen3-4B
073.934.1966.814.19
12579.384.3370.615.20
25085.883.7872.705.48
37587.632.6575.615.19
49693.782.8081.646.24
Science / Qwen3-4B
075.704.0263.654.30
12572.483.9174.565.02
25081.613.8271.145.02
37579.123.4168.474.83
49688.003.6873.945.25
Science / Qwen2.5-7B
058.883.4451.703.43
12560.703.3662.264.37
25079.883.0261.773.96
37582.262.3658.123.86
49691.582.1263.654.21

Table R2: Benchmark performance (%, ↑) during RL.

StepDirectRubricArmor
Health / Qwen3-4B (HealthBench-Hard)
07.087.08
809.3210.71
16014.7415.97
24014.9717.81
32014.2522.97
40011.9023.95
48011.0723.00
49610.9323.24
Science / Qwen3-4B (ResearchQA)
065.6265.62
8067.2568.56
16069.8371.06
24071.5772.48
32070.1675.88
40068.9977.52
48068.6375.39
49667.9576.39
Science / Qwen2.5-7B (ResearchQA)
052.0052.00
8060.6162.74
16067.5073.20
24066.3677.32
32063.5576.68
40058.1876.93
48056.4076.92
49654.2676.59

[W2, Q2] The verifier and multi-agent setup are not fully independent. The appendix mainly tests different verifiers on fixed rubrics, while the agents also share the same backbone, which may introduce common blind spots. Explain to what extent current experiments on verifier generalization reflect the robustness of the full evolution process. Additionally, share the same backbone lead to shared blind spots.

Response: Thank you very much for pointing out these important and thoughtful issues; we address them in two parts as follows.

  • Verifier generalization. The original Table 5 reused the rubrics evolved with GPT-4.1-mini from Table 1 and thus mainly reflected the robustness of these evolved rubrics across verifiers. To further reflect the robustness of the full evolution process, we rerun the evolution with each verifier from the same Direct rubrics, using that verifier during both evolution and evaluation, and update Appendix B.1 (Generalization across Verifier Backbones) accordingly. Since Gemini-3.5-Flash is no longer available, we replace it with Gemini-3.6-Flash. We found that RubricArmor consistently outperforms Direct under every verifier, across both verifier families and capabilities, demonstrating that the full evolution process remains robust when the verifier changes.

Table R3: Generalization across verifier backbones (accuracy, %).

MethodJudgeBenchRM-BenchChat-Hard
Verifier: GPT-4.1
Direct65.5675.9274.89
RubricArmor74.0380.1878.18
Verifier: Gemini-3.6-Flash
Direct75.0078.1375.88
RubricArmor80.2483.2579.93
Verifier: GPT-4o-mini
Direct63.3973.6374.01
RubricArmor72.3479.4976.32
Verifier: GLM-5.1
Direct71.5378.7375.88
RubricArmor79.1982.6978.40
Verifier: DeepSeek-v4-Flash
Direct65.4076.7874.01
RubricArmor73.5579.0377.96
  • Shared blind spots. Potential blind spots caused by agents sharing the same backbone are indeed an important concern, and we really appreciate this insightful point. In our experiments, we deliberately use the same backbone for all agents of RubricArmor and all LLM-based rubric generation baselines to ensure a fair comparison, so that the gains come from the attack–repair process rather than from additional models. In practical deployment, however, different agents are free to use different models. Therefore, we add experiments with heterogeneous agents, which replace the attacker or the quality judge with Qwen3.5-397B-A17B, a model from a different family than the backbone of the other agents. We found that heterogeneous agents can bring further gains on top of RubricArmor, which is already effective with a single shared backbone, demonstrating that model diversity is an optional enhancement for practical deployment.

Table R4: Accuracy (%) with heterogeneous agents.

MethodAttackerQuality judgeJudgeBenchRM-BenchChat-Hard
Direct——62.1078.9374.67
RubricArmorGemini-3.1-Flash-LiteGemini-3.1-Flash-Lite73.6383.6978.73
RubricArmorQwen3.5-397B-A17BGemini-3.1-Flash-Lite76.7782.9680.15
RubricArmorGemini-3.1-Flash-LiteQwen3.5-397B-A17B74.9284.4181.25

[W3, Q3] The comparison with baselines is not fully compute-matched. Since RubricArmor uses multiple rounds of LLM calls, it is unclear how much of the gain comes from the method itself versus the extra inference budget. Clarify whether the performance gains of RubricArmor stem primarily from adversarial refinement itself, or if they might also partially result from increased LLM calls and a larger inference budget?

Response: Thank you for this insightful question. To disentangle the effect of adversarial refinement from that of additional inference compute, we add a compute-matched baseline, Iterative-Rewrite, which iteratively rewrites the entire rubric with a fixed prompt at a token budget comparable to RubricArmor's. We report the average total tokens per query for Iterative-Rewrite, RubricArmor, and the two iterative-refinement baselines, ChasingTail and RRD, with two observations as follows.

  • Iterative-Rewrite underperforms RubricArmor significantly, given a comparable or larger budget.
  • RRD-Uniform underperforms RubricArmor, even with a substantially larger budget.

Larger inference budgets alone thus cannot explain the gains: they reflect the benefit of adversarial refinement, not extra LLM calls.

Table R5: Accuracy (%) and average compute cost per query.

MethodJudgeBench TokensJudgeBench Acc.RM-Bench TokensRM-Bench Acc.Chat-Hard TokensChat-Hard Acc.
Iterative-Rewrite102.20K69.9299.03K77.15100.92K75.55
RRD240.01K71.94242.74K77.42255.35K69.74
ChasingTail100.39K71.3779.35K81.5164.60K72.37
RubricArmor93.92K73.6385.96K83.6980.59K78.73

[W4, Q4] The paper shows some signs of AI-generated presentation. The figures are rather basic, and some punctuation and formatting are inconsistent. The omitted few-shot examples may also affect reproducibility. Clarify whether the omitted few-shot examples affect reproducibility and further improve the figures, writing, and formatting in the final version.

Response: We appreciate the careful reading and these constructive suggestions on presentation. We would like to assure that this paper is human-authored. Nevertheless, we agree that the raised points are truly valuable to improve this paper’s presentation. We address each item as follows.

  • On few-shot examples. The few-shot examples were omitted from the main text solely for brevity; they are included in our released code. To ensure reproducibility from the paper alone, we have updated the prompts in appendix to facilitate reproducibility solely from appendix materials.
  • On figures. We would like to clarify that Figures 1 and 2 were hand-drawn by the authors in PowerPoint. Following the reviewer's suggestion, we have revised every figure in the original submission to improve clarity. In figure 1, each rubric now displays the reward it assigns and whether the error is missed or caught. The error in each response is highlighted in red with enlarged text for visibility. In figure 2, the six steps are now numbered and titled with the agent names used in the main text. Text size has been increased throughout, and the two Memory panels explicitly list the inputs read by the attacker and the repair agent. For figures 3 and 5, (Figure 5 was Figure 4 in the original submission), both figures have been redrawn with a distinct color assigned to each benchmark and a consistent font matching the main text, improving curve distinguishability and readability.
  • Punctuation and formatting. We fix the inconsistent punctuation and formatting throughout the paper, including method names broken across lines, inconsistent spelling, and unify the notation and notes in tables.

If any passage or figure still takes extra effort to read, we would sincerely appreciate it if you could point it out, and we will revise it right away.


Again, thank you so much for your suggestions for improvements! We are eager to hear your feedback! We’d deeply appreciate it if you could let us know whether your concerns have been addressed.

Reviewer_9TTTOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents RubricArmor, a methodology to produce rubrics that are robust against reward hacking. RubricArmor produces such robust rubrics by conducting several rounds of adversarial probing that uncover weaknesses in the existing rubrics. The rubric is then revised to cover such weaknesses. Finally a review stage ensures the weakness is indeed covered. These rubrics are then verified on reward modeling and downstream policy training, where it results in superior performance compared to baselines.

Critical strengths
  1. The approach is novel and timely, as reward hacking is gaining increasing interest.
  2. The empirical results are strong.
  3. The paper's presentation is clear.
Critical weaknesses
  1. For policy optimization, the improvements are not that large compared to existing baselines.
  2. For reward modeling there are no baselines, so it is hard to quantify the contribution over RubricArmor over existing literature.
Questions for authors

Why is the gain larger on reward modeling benchmarks compared to policy optimization benchmarks? I would think policy optimization would be more sensitive to hacking.

2 comments

We sincerely thank Reviewer 9TTT for the positive and encouraging review! Below, we address your concerns point by point and indicate the corresponding changes in the revised paper (revisions are highlighted in blue).


[W1] For policy optimization, the improvements are not that large compared to existing baselines.

Response: Thank you for this sincere comment. We acknowledge that RubricArmor's improvements over the strongest baselines are modest on some downstream RL benchmarks. Nevertheless, two lines of empirical evidence support RubricArmor’s effectiveness in RL. First, the gains are consistent: RubricArmor attains the best performance in 13 of 15 settings (Table 4). Second, and more decisively, our central hypothesis concerns whether adversarial evolution improves rubric quality. Because RubricArmor is initialized from the Direct rubric, its gain over Direct directly reflects the benefit of adversarial evolution. RubricArmor substantially outperforms Direct in all 15 RL settings (Table 4), effectively verifying this hypothesis. Therefore, the modest margins over the strongest baselines on some benchmarks do not invalidate the empirical effectiveness of adversarial evolution.

[W2] For reward modeling there are no baselines, so it is hard to quantify the contribution over RubricArmor over existing literature.

Response: Thank you for this actionable and meticulous suggestion!

  • To address this issue, we add reward modeling baselines following your advice. They include the RRM-7B [1] on NeurIPS 2025 and two RM-R1 variants [2] on ICLR 2026, which are trained to reason before judging the better response and are comparable in size to the rubric generator baselines in Table 1. The scores and their macro averages are presented in the table below. RubricArmor retains the best overall performance among all baselines, including the newly added reward modeling methods.
  • The additional baseline results and discussions are presented in the revised Section4.1 and Table1.
MethodJudgeBench-KnowledgeJudgeBench-ReasoningJudgeBench-MathJudgeBench-CodingRM-Bench-ChatRM-Bench-MathRM-Bench-CodeRM-Bench-SafetyRewardBench-Chat-HardMacro Avg.
RRM-7B66.2366.4480.0064.3862.1981.8152.9784.1070.6169.86
RM-R1-Instruct-7B54.2263.0977.7861.6467.1871.2058.6392.7976.5469.23
RM-R1-Distilled-7B68.1865.1078.8967.1262.4582.1955.5185.2169.3070.44
RubricArmor74.0367.1183.3372.6068.4886.2371.7191.5878.7377.09

References

[1] Guo Jiaxin, et al. “Reward Reasoning Models.” NeurIPS 2025.

[2] Chen Xiusi, et al. “RM-R1: Reward Modeling as Reasoning.” ICLR 2026.

[Q1] Why is the gain larger on reward modeling benchmarks compared to policy optimization benchmarks? I would think policy optimization would be more sensitive to hacking.

Response: Thank you for this thought-provoking question! Please kindly see the structural responses as follows.

  • First, we would like to clarify that the stronger gains on reward modeling benchmarks are expected. RubricArmor is designed to improve reward modeling, whereas RL serves as a downstream application. The former isolates the intended improvement, while the latter is additionally confounded by factors such as training stability, which would attenuate the observed downstream gains. Consequently, the improvements on RL benchmarks are expected to be more modest than those on reward modeling benchmarks.
  • Furthermore, although stronger reward modeling implies greater resistance to reward hacking, this advantage may be diluted when translated into downstream policy performance, which depends on additional factors beyond the reward signal. This dilution effect accounts for the modest gains on policy optimization benchmarks that you observed.
  • Second, we add experiments designed specifically to test RubricArmor's resistance to reward hacking during policy optimization. Specifically, we extend GRPO training to 496 steps, treating the divergence between rising reward and declining benchmark performance as the diagnostic signature of reward hacking. We train policy models under both Direct and RubricArmor rubrics and track: (i) rubric reward and response quality on the same training rollouts, with quality scored on a 1–7 scale by Opus 4.6 blind to the rubric, and (ii) benchmark performance on HealthBench-Hard (Health) and the ResearchQA validation split (Science). The results in the following table illustrates that RubricArmor effectively mitigates reward hacking during policy optimization. Specifically, Direct exhibits the reward-hacking signature: reward up, quality down, benchmark scores peaking around steps 160–240 and then declining; while RubricArmor raises rubric reward and response quality together across all settings. This comparison Detailed results are shown in Section 4.7 and Figure 4 of the revised paper.

Table R1: Rubric reward and response quality of training rollouts.

DirectRubricArmor
GRPO iteration stepsReward (%)Quality (1–7)↑Reward (%)Quality (1–7)↑
Health / Qwen3-4B
073.934.1966.814.19
12579.384.3370.615.20
25085.883.7872.705.48
37587.632.6575.615.19
49693.782.8081.646.24
Science / Qwen3-4B
075.704.0263.654.30
12572.483.9174.565.02
25081.613.8271.145.02
37579.123.4168.474.83
49688.003.6873.945.25
Science / Qwen2.5-7B
058.883.4451.703.43
12560.703.3662.264.37
25079.883.0261.773.96
37582.262.3658.123.86
49691.582.1263.654.21

Table R2: Benchmark performance (%, ↑) during RL.

StepDirectRubricArmor
Health / Qwen3-4B (HealthBench-Hard)
07.087.08
809.3210.71
16014.7415.97
24014.9717.81
32014.2522.97
40011.9023.95
48011.0723.00
49610.9323.24
Science / Qwen3-4B (ResearchQA)
065.6265.62
8067.2568.56
16069.8371.06
24071.5772.48
32070.1675.88
40068.9977.52
48068.6375.39
49667.9576.39
Science / Qwen2.5-7B (ResearchQA)
052.0052.00
8060.6162.74
16067.5073.20
24066.3677.32
32063.5576.68
40058.1876.93
48056.4076.92
49654.2676.59

We are eager to hear your feedback. We’d deeply appreciate it if you could let us know whether your concerns have been addressed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

Dear AC and all reviewers,

We sincerely appreciate your commitment to evaluating our paper despite your busy schedules. We are encouraged that 2 out of 3 reviewers are currently on the positive side, with scores 4 with conf 3 and 3 with conf 3 (raised from 2 after our rebuttal), recognizing our paper

  • takes an approach that is "novel and timely, as reward hacking is gaining increasing interest", with "strong" empirical results and a "clear" presentation (Reviewer 9TTT),
  • presents an approach that is "clear and well-matched to the problem", "demonstrating the method's practical value" through "consistent performance improvements" (Reviewer r8MS),
  • proposes a method that is "well-motivated", with "fairly comprehensive" experiments (Reviewer qJnE).

At the same time, we noticed that Reviewer qJnE gave an initial rating of 2 with confidence 3 (no discussion response yet). After acknowledging in the initial review that the method is well-motivated and that the experiments are fairly comprehensive, the reviewer raised concerns about (1) the lack of a guarantee that the rubric covers all hacking behaviors, and (2) the cost of the method.

There is no guarantee that the rubric can patch the reward hacking behavior during RL training.

  • We acknowledge that it is difficult to cover all potential hacking behaviors, as the space of such behaviors would be unbounded. Nevertheless, we argue that this limitation does not negate the contributions of this paper. Our additional RL hacking experiments show that RubricArmor can preempt a substantial portion of reward hacking during RL training, which is sufficient to improve RL performance and substantiate the central claim of this paper. We have discussed this limitation in the conclusion section.

The cost is relatively high, with multiple patching iterations and each iteration involving a sequence of attack, verify, repair, and review steps.

  • Our additional experiments demonstrate that the token cost of RubricArmor is comparable to that of competitive iterative refinement methods while achieving higher accuracy, and that it remains effective with cheaper and open-weight foundation models, which can further reduce costs.

We are confident that our responses and additional experiments can thoroughly address Reviewer qJnE's raised concerns. We would kindly note that Reviewer qJnE has not yet replied to our responses, likely due to a busy schedule. We therefore respectfully request that the AC and reviewers take this into consideration when making the final recommendation. Thank you for your invaluable time and effort.

Many thanks,

Authors of #56681

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.