acceptodds
Under review as a conference paper at ICLR 2027

IMPOSSIBLERUBRICS: STRESS-TESTING GENERATED RUBRICS AS REWARD SIGNALS

Abstract

Language model-generated rubrics are increasingly used as reward signals for rubric-based reinforcement learning, LLM-as-a-judge evaluation, and automated grading. Such rubrics are reliable only if it rewards honest answers over adversarial answers optimized to exploit it. Yet their robustness to such optimization remains poorly understood. We isolate the hardest regime: impossible tasks, where the prompt pressures the model toward an unsupported conclusion, so the only honest response is to acknowledge the impossibility. We introduce ImpossibleRubrics, a benchmark of 169 impossible tasks spanning six impossibility categories, each paired with a verifiable oracle certificate specifying what an honest answer may and may not claim, together with 48 answerable controls. Rather than providing fixed rubrics, ImpossibleRubrics provides task environments and certificates, allowing rubrics to be generated downstream and then adversarially tested for whether they reward certificate-violating answers. Eleven generators are exploited 8–26% of the time on the unbiased 150-of-169 environment cut; on a deliberately selected stress cut the strongest generator we measured is still exploited 36% while a certificate-faithful rubric is exploited 0%, so what we measure is a rubric-quality gap, not task impossibility. One result runs against intuition. A single generic rubric (“be decisive, penalize hedging”) used unchanged for every task is exploited 64% of the time, and seven of the eleven generators are exploited more often than that while writing a rubric tailored to each one. The tailored criteria appear to tell an attacker which claim to fabricate. The problem is not that rubrics are vague; it is that they are specific about the wrong things.

Primary area:
datasets and benchmarks
Keywords:
LLM as Judge, Coding Agent, Rubrics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LMo7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether model-generated rubrics reward responses that violate the available evidence. ImpossibleRubrics contains 169 impossible tasks and 48 answerable controls. An attack succeeds when it violates the task certificate while scoring at least as highly as an honest reference answer. Under a fixed evaluation chain, eleven generators achieve exploitation rates of 8–26% on Full-150 and 36–98% on the selected Hard-45 set. Additional experiments examine resampling, model substitutions, and prompting interventions, and reveal substantial dependence on the verifier configuration.

Critical strengths
  1. The research question has practical relevance. Generated rubrics can guide evaluation, response selection, and training rewards, making erroneous incentives an important concern. Figures 1–2 provide an intuitive example in which the rubric incorrectly interprets missing evidence as zero probability and rewards an unsupported response more highly than an honest one.

  2. The benchmark design is reusable. Retaining tasks, evidence packets, and certificates rather than fixing the evaluated rubrics allows the framework to assess new generators and generation strategies. The closed-evidence setting also facilitates inspection of whether individual scoring criteria require unsupported conclusions.

  3. Some controlled analyses provide useful diagnostic information. In particular, freezing responses and scores while changing only the verifier directly demonstrates sensitivity to the verification protocol. This finding can inform the design and interpretation of future rubric-exploitation evaluations.

Critical weaknesses
  1. The connection between adversarial exploitability and actual training risk is not established. The main experiments use a targeted attacker that can inspect the rubric and is instructed to disregard honesty, without testing whether a policy discovers and amplifies these exploits during reward optimization. The only benign best-of-N experiment instead reduces violations from 44.4% under random selection to 27.8%. Whether the metric predicts reward exploitation or declining honesty during training therefore remains unvalidated.

  2. The reliability of the core labels and reference answers remains insufficiently established. In Appendix M, exploitation on identical responses changes from 33.3% to 75.6% across verifiers, and some honest references contain omitted certificate requirements or source-attribution problems. The existing single-author, 40-item calibration does not adjudicate these new disagreements. Exploitation rates and model comparisons consequently require independent validation.

  3. The generation-strategy baselines are insufficient. The main experiments use a common default prompt, and the mitigation experiment tests only one generic safety instruction. The certificate-faithful baseline has privileged task information, while the decisiveness-oriented baseline does not represent a competitive generation procedure. Matched-input evidence-analysis, rubric-checking, or revision baselines are missing, limiting conclusions about whether the failures persist across reasonable generation strategies.

  4. Key validation experiments insufficiently cover the models underlying the main comparative claims. Full-Hard-45 resampling primarily covers three earlier Claude generators, while the supplementary GPT-5.6-sol study covers only 20 environments. Answerable controls exclude the leading Opus 5 and GPT-5.6 configurations and the safety-prompted settings. The stability of the leading models’ advantages, and whether lower exploitation comes with increased over-refusal, therefore remain insufficiently tested.

  5. The benchmark’s distinctive diagnostic value is unclear. Its main novelty lies in evaluating rubric generators, but the experiments do not adequately distinguish rubric-construction errors from the generator’s underlying evidence-understanding errors. A matched comparison with direct answering or answerability judgments is missing, leaving unclear how much additional information the benchmark provides beyond evidence-understanding and abstention evaluation.

Questions for authors
Questions For Authors
  1. As reward optimization becomes stronger, do rubric scores increase while independently assessed evidence faithfulness decreases? Can a controlled optimization experiment test the relationship between adversarial exploitation rates and actual optimization risk?

  2. Under the same question-and-evidence input budget, do evidence-analysis-first generation or rubric-checking and revision procedures reduce exploitation? Do the main model differences persist?

  3. Can representative leading and comparison models be evaluated across repeated rubric draws on a common task set, reporting both impossible-task exploitation and answerable-task over-refusal?

  4. How much do a model’s direct-answering errors overlap with its rubric-generation failures? Are there cases where the model correctly identifies insufficient evidence but nevertheless generates erroneous scoring criteria?

Reviewer_rGEJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • The paper introduces ImpossibleRubrics, a benchmark that evaluates whether LLM-generated rubrics falsely reward dishonest answers over responses that honestly state there is a lack of evidence. The motivation is that rubrics are widely used as RL rewards, and if they are gameable, a policy can learn adversarial answers to optimize against them.
  • The benchmark conrtibutes impossible tasks with oracle certificates, an adversarial protocol for testing whether generated rubrics can be exploited, and a leaderboard comparing eleven rubric generators. It also includes robustness analyses showing that measured exploitation rates depend heavily on the verification setup.
Critical strengths
  • This paper presents a benchmark that tests the validity of rubrics on impossible tasks, where an honest answer has to acknowledge that the evidence doesn’t support a conclusion. It releases tasks, evidence packets, and certificates instead of fixed rubrics, so new rubric generators can be stress-tested against a fixed definition of what an honest answer must and must not say. This is a useful and under explored setting, since it targets exactly the cases where a generated rubric is most likely to reward a confident but unsupported answer.
Critical weaknesses
  • The paper’s rubric generation design relies on the generator seeing only the question and the evidence packet, with no reference answer or expert-written criteria (Fig. 3). This differs from most rubric generation methods, including those the paper cited as motivation: RaR conditions on reference answers [1], and OpenRubrics on preferred and rejected responses [2]. Since the paper frames exploitability as a safety concern for RL, it needs to establish one or both of the following: (i) that its rubric generation method is grounded in the literature, and (ii) that policies learn such adversarial exploits during training, shown through experiments or existing literature, and why existing methods cannot handle them. Tying these aspects together would make the paper’s contribution much stronger. Furthermore, the best-of-N study in the paper actually cuts against this framing: the generated rubric never selects a dishonest answer when an honest one is available. This needs further clarification.

References

[1] Gunjal, Anisha, et al. "Rubrics as rewards: Reinforcement learning beyond verifiable domains." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Liu, Tianci, et al. "Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Questions for authors
  1. Does a lower exploitation rate on ImpossibleRubrics predict better downstream outcomes, e.g., more honest policies after RL training with that generator’s rubrics? Without this, it’s unclear to me what improving on the benchmark means for a model builder.
  2. The paper evaluates a single rubric generation prompt across eleven models. How do exploitation rates change under other rubric generation methods? Do you have any insight on this? A discussion and empirical study of which types of rubric generation methods can be tested on this benchmark would be useful.
Reviewer_o59ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces the ImpossibleRubrics Benchmark, which employs an adversarial evaluation pipeline to systematically compare rubrics generated by different LLMs and investigate whether the evaluation criteria themselves incentivize hallucinations.

Critical strengths

None

Critical weaknesses
  1. The attack setting doesn’t tell us much about what happens during RL.

The attacker reads the full rubric, then writes an answer designed to score highly while breaking the evidence constraints. That is a fine stress test, but it's very different from rubric-based RL, where the policy never sees the rubric and only gets sampled rewards. A white-box attacker finding an exploit doesn’t mean a GRPO policy would ever find it. Your Appendix J points out that, under a neutral policy, rubric-based best-of-N yields a lower violation rate than random selection.

  1. The numbers depend heavily on which Oracle you pick.

In Appendix M, with rubrics, attack responses and judge scores all fixed, the exploitation rate goes from 33.3% to 75.6% just by changing the Oracle configuration. The generator rankings are built on this same metric.

Appendix F doesn’t fix this. 40 examples labeled by one annotator can’t tell us which Oracle is right when they disagree this much, and it doesn't scale to the full benchmark as a validation procedure. How many cases where the Oracles disagree did a human actually review, and who was right? Do the generator rankings stay the same across all Oracle configurations in Appendix M? If they don’t, I’m not sure what conclusion to draw from the comparison tables.

Questions for authors
  1. Can you train a small policy with GRPO on generated rubrics vs. certificate-faithful rubrics, with the same training config, and plot the oracle-assessed violation rate over training? If the generated rubrics lead to more violations as training goes on, the main claim is much stronger. Without something like this, I don’t see how white-box exploitability translates into reward hacking.

  2. Given the substantial Oracle disagreement in Appendix M, how can you establish the reliability of the reported exploitation rates? Could multiple human annotators independently adjudicate the disputed cases, and how sensitive are the main conclusions to those judgments?

Comments on the paper

1 comment

Dear reviewers,

Thank you for your reviews. Please read the authors’ response and follow up in the discussion, briefly indicating whether your main concerns have been addressed and highlighting any remaining questions. Please also update your review and rating where appropriate.

Many thanks for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.