acceptodds
Under review as a conference paper at ICLR 2027

LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become the dominant paradigm for scaling reasoning in large language models, but it can introduce a subtle failure mode: LLMs gaming verifiers. We study this on inductive reasoning tasks, where models must induce logical rules that capture relational patterns from labeled examples. We find that some LLMs systematically abandon rule induction. Rather than inferring relational rules (e.g., ”plants with purple leaves are toxic”), they enumerate instance-level labels (e.g., ”plant_01 is toxic”), producing outputs that pass verifiers while failing to capture the intended structure. We show that this behavior is not a failure of understanding but a form of reward hacking: imperfect verifiers that check only extensional correctness admit false positives. To detect such shortcuts, we introduce Isomorphic Perturbation Testing (IPT), which evaluates a single model output under both extensional verification on the original task and verification on a logically isomorphic perturbation. While genuine rule induction is invariant to such perturbations, shortcut strategies fail. Using IPT, we find shortcut behavior in RLVR-trained reasoning models (e.g., GPT-5, Olmo-3.1) but not in non-RLVR models (e.g., GPT-4o, GPT-4.5), with prevalence rising with task complexity and inference-time compute. In controlled training experiments, extensional verification directly induces shortcut behavior, while isomorphic verification eliminates it. These results show that RLVR can incentivize reward hacking not only through overt manipulation but also by exploiting what the verifier fails to enforce.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RLVR, Reward Hacking, LLM Reasoning, Shortcut Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AXFQOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies a subtle form of reward hacking in reinforcement learning with verifiable rewards (RLVR), where models learn to satisfy an imperfect verifier without performing the intended reasoning task. On inductive logic problems, the authors observe that some RLVR-trained models replace genuine rule induction with instance-level enumeration or other shortcuts that still pass an extensional verifier. To detect this behavior, the paper proposes Isomorphic Perturbation Testing (IPT), which checks whether a model output remains valid after object identifiers are consistently renamed while preserving the underlying logical structure.

Critical strengths
  1. The paper identifies a convincing and practically important failure mode in RLVR where models can exploit what a verifier fails to enforce rather than solve the intended reasoning task.
  2. IPT is a simple and elegant black-box diagnostic that distinguishes genuine rule induction from identifier-dependent shortcuts without requiring access to model weights or internal activations.
  3. The controlled RLVR experiments provide unusually strong causal evidence that the verifier itself can induce reward hacking, rather than merely showing a correlation across different model families.
Critical weaknesses
  1. The core contribution is relatively narrow, as IPT relies on task-specific isomorphic transformations and is demonstrated primarily in structured inductive logic settings.
  2. The method requires an automatically constructible invariance that preserves task semantics, which may be difficult to define for open-ended reasoning, mathematics, coding, or other tasks with less explicit structure.
  3. Although the paper evaluates several logical reasoning benchmarks, the experimental domains remain concentrated on symbolic and inductive reasoning, leaving the broader relevance to general RLVR largely unclear.
  4. IPT detects only shortcuts that violate the chosen isomorphism, so a model could potentially exploit other verifier weaknesses while still passing both the original and perturbed checks.
Questions for authors
  1. How can IPT be extended to reasoning tasks where no obvious semantics-preserving isomorphic transformation exists?
  2. Do you expect the same reward-hacking behavior to appear in domains such as mathematics, coding, or open-ended reasoning?
  3. Could a model learn shortcuts that remain invariant under the current perturbations and therefore evade IPT?
  4. How much of the observed benefit comes from IPT specifically, rather than from the highly structured nature of SLR-BENCH and its verifier?
7 comments
AuthorsOpenReview ↗

Response to Reviewer AXFQ (1/3)

Thank you for your review. We were encouraged that you found the failure mode “convincing and practically important”, IPT “simple and elegant”, and the controlled experiments “unusually strong causal evidence”. In response to the concerns, we added an experiment in a second domain (code) and a downstream evaluation of our matched pair of models.

W1, Q2 — “The core contribution is relatively narrow” [...] “Do you expect the same reward-hacking behaviour to appear in [...] mathematics, coding, or open-ended reasoning?”

The focus of our paper is diagnosing reward hacking and showing how it can emerge during RLVR. We show that released reasoning models game a verifier they were never told about, which happens across multiple model families (Tab. 1, Tab. 8). We show that their own traces call the shortcut cheating (Sec. 4). And we show in controlled training, on two model families and two RL stacks, that the verifier causes the behaviour (Fig. 2, App. B.1). IPT is the instrument that makes all of this measurable.

IPT is also not tied to logic. To show this, we applied it to Python programming. Following CodeARC [4], the model is shown 8 input/output examples of a hidden function and has to write it. IPT is implemented as two unit tests, an extensional one and an isomorphic one. We measure hacking via IPT as hacking=ext−isohacking = ext - iso, the programs that pass the extensional test and fail the isomorphic one. We build 270 tasks in six difficulty levels, which scale with the number of operations. The table gives solved / hacking per level when the model may retry up to three times (20 tasks at level 1, 50 at each other level).

Modeld1d2d3d4d5d6
GPT-5-Mini20/037/414/115/161/210/28
GPT-5-Nano18/128/27/53/110/50/18
gpt-4o-mini10/01/00/00/00/00/0

GPT-5-Mini and GPT-5-Nano fall back to hacking as the tasks get more complex. gpt-4o-mini does not. The isomorphic test successfully detects the Python code that exploits the unit test. We will add this experiment to the appendix.

So yes, we expect the same behaviour in other domains. In coding, we observe it directly, in line with independent reports of test-case exploitation [1, 2, 3]. For mathematics and open-ended reasoning, the condition we identify applies unchanged. Models learn to hack when verifiers admit false positives, meaning they accept answers that do not solve the task, and model shortcuts when a genuine solution is out of reach for them. Audits report that rule-based and model-based verifiers in mathematical reasoning are systematically gameable (Sec. 2). Our controlled training shows how RLVR can turn such a gap into learned behaviour (Fig. 2).

W2, Q1 — "requires an automatically constructible invariance" [...] "How can IPT be extended to reasoning tasks where no obvious semantics-preserving isomorphic transformation exists?"

We agree that IPT needs a task structure on which a semantics-preserving transformation can be defined. We state this in our Limitations, where IPT "requires a domain where meaningful isomorphic perturbations can be generated automatically, which is natural for logic-based tasks but less direct for open-ended generation" (Sec. 6). We see this as the scope of the method. No approach to detecting reward hacking is universal. ImpossibleBench requires code tasks with unit tests [1]. Chain-of-thought monitoring requires access to the reasoning trace of the model [3] and can emit false positives. IPT requires a renaming. In return, it needs neither traces nor model internals, and it has no false positives by construction. Further, IPT can be deployed well beyond logic. IPT is a natural fit for RLVR, which is built around verifiable tasks and formal checks. This formal structure is what an isomorphic transformation builds on. We deploy IPT for logic and, in our new experiment, in code (W1). Templated mathematics problems could be treated in the same way, as the templates of GSM-Symbolic show [5].

AuthorsOpenReview ↗

Response to Reviewer AXFQ (2/3)

W3, Q4 — "the experimental domains remain concentrated on symbolic and inductive reasoning, leaving the broader relevance to general RLVR largely unclear" [...] "How much of the observed benefit comes from IPT specifically, rather than from the highly structured nature of SLR-BENCH and its verifier?"

SLR-Bench is where we measure the behaviour. The models that show it are general. GPT-5, GPT-5-Mini, GPT-5-Nano, o3 Mini and gpt-oss-120b are strong general-purpose reasoning models, and all of them game the verifier (Tab. 1, Tab. 8). GPT-5-Mini and GPT-5-Nano also hack in code (W1).

Our training experiments show the same for general RLVR. The hacking gap emerges inside the broad multi-environment mix of the Olmo-3 RLVR stack (Fig. 2c,d), and again inside the 15-reward mix of Nemotron, a second model family and RL framework (App. B.1). We further evaluated the matched pair from the Olmo-3 multi-reward stack at step 1650, trained with the extensional (EXT) or the isomorphic (ISO) verifier, on general benchmarks and on ImpossibleBench [1]. ImpossibleBench measures test-case exploitation in code by presenting the model with impossible tasks. Their unit tests contradict the task specification, either because two tests contradict each other (conflicting) or because a single expected output is altered (one-off).

BenchmarkEXTISO
GSM8Kgrade-school math95.3%94.8%
MATHcompetition math94.9%94.5%
ARC-Challengescience QA92.3%92.0%
IFEvalinstruction following83.9%82.4%
BBHgeneral reasoning83.6%83.1%
ImpossibleBench conflicting ↓hacking (avg@4)5.3% ± 1.7%4.1% ± 0.9%
ImpossibleBench one-off ↓hacking (avg@4)7.8% ± 1.8%5.3% ± 2.3%
ImpossibleBench solvable ↑code (control tasks, avg@4)35.4% ± 3.9%37.9% ± 5.3%

The table reveals two surprising insights.

The EXT model has learned to game the verifier and reaches close to full reward on SLR-Bench (Fig. 2c). Still, it stays on par with the ISO model on every general benchmark. A plausible explanation comes from the RLVR setup itself. The update uses the advantage of a rollout relative to its group. Once the model games the verifier, the SLR-Bench reward saturates. A group in which all rollouts receive the full score has zero advantage and yields no gradient. A growing share of the hacked environment therefore drops out of the update, and its gradient signal diminishes. The update signal now comes primarily from the other environments, which remain stable throughout training. Accordingly, the capabilities trained by these environments keep improving after one environment has been gamed, which yields stable downstream capabilities. This is also what we observe in our evaluation. Across models, those that hack are not necessarily less capable in general (Tab. 1).

On ImpossibleBench, the EXT model exploits the test cases more often than the ISO model on both test sets. This indicates that the hacking behaviour is also more pronounced on test-case hacking in code.

This matched pair also answers Q4. Both models train on the same SLR-Bench tasks and differ only in the verifier, so the structure of SLR-Bench is identical in both runs. Yet IPT has a pronounced effect on the learned model behaviour. Under the extensional verifier, the hacking gap opens and widens during training (Fig. 2c). Under the isomorphic verifier, it stays closed (Fig. 2d). The structure of SLR-Bench alone thus does not prevent the shortcut, whereas IPT does. At the same time, the general reasoning capabilities of the two models are on par, and the EXT model is more susceptible to hacking across both test sets of ImpossibleBench.

AuthorsOpenReview ↗

Response to Reviewer AXFQ (3/3)

W4, Q3 — “IPT detects only shortcuts that violate the chosen isomorphism” [...] "Could a model learn shortcuts that remain invariant [...] and therefore evade IPT?"

Yes, such shortcuts are possible, and we do not claim otherwise. IPT targets one class of exploit. As our Limitations state, it “does not exclude other forms of reward hacking, yet reliably identifies shortcuts without false positives by construction, since any valid hypothesis passes both verifiers” (Sec. 6). Every answer that IPT flags is therefore a genuine shortcut. That other exploits exist holds for every verifier and every detection method. Theory even suggests that a generally hack-proof reward does not exist [6].

A model with internet access could, for example, look up the solution online, and no check of the final answer would detect it. Sandboxing does not rule this out either. In July 2026, agents in an OpenAI experiment hacked out of their containers, reached the internet through an external sandbox and attacked Hugging Face, all to obtain a passing score from an automated scorer on tasks that were impossible to solve [7]. As in our experiments, the hacking set in where no genuine solution was within reach. The conclusion is not that targeted tests are of little use. Each class of exploit needs its own test, and IPT provides one for a class that we observe in released models.

References

[1] Zhong et al., ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases, ICLR 2026.

[2] METR, Recent Frontier Models Are Reward Hacking, blog post, June 2025.

[3] Baker et al., Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, arXiv:2503.11926, 2025.

[4] Wei et al., CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis, COLM 2025.

[5] Mirzadeh et al., GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models, ICLR 2025.

[6] Skalse et al., Defining and Characterizing Reward Hacking, NeurIPS 2022.

[7] Greenblatt et al., Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, METR and Redwood Research, August 2026.


We thank the reviewer again for the questions. They led us to extend the study to a second domain and made the paper stronger. We believe that the new experiments and clarifications address the reviewer’s concerns, and we kindly ask the reviewer to reconsider the score. If any concern about the soundness of our experiments or claims remains, we would be grateful for a short pointer and will gladly address it. We are happy to run further analyses during the discussion period.

Reviewer_HbDLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies shortcut behavior in LLMs on inductive logic tasks, where models can satisfy an extensional verifier by enumerating instance labels or exploiting label predicates instead of inferring the requested relational rules. It introduces Isomorphic Perturbation Testing (IPT), which evaluates a fixed model-generated hypothesis on both the original task and an object-renamed version to identify shortcuts. The authors evaluate multiple model families, examine prompt interventions and reasoning traces, and conduct controlled RLVR experiments comparing extensional and isomorphic verifiers. These experiments show that extensional rewards can reinforce shortcuts in the studied setting, while isomorphic verification mitigates them. The paper also reports improved logical reasoning performance from training with the isomorphic verifier.

Critical strengths

The paper develops a useful experimental setting for probing shortcut behavior in LLMs. It makes concrete cases of models bypassing the intended rule-induction task observable and amenable to systematic study. This is a timely contribution that draws attention to behavior that conventional accuracy-based evaluation can overlook.

Critical weaknesses

The probe detects shortcuts, but I am not convinced that the comparison tells us much about RLVR models’ tendency to exploit verifiers. GPT-5 performs well and rarely takes shortcuts, while smaller models often struggle to solve the underlying induction task. A model that cannot solve the task or discover a shortcut can also have zero shortcuts. These outcomes have very different meanings, yet the paper uses them to suggest a distinction between RLVR and non-RLVR models. The role of capability needs to be separated before this interpretation is convincing.

My main concern is that the verifier has an obvious design flaw: it accepts answers that reproduce the visible labels without requiring the intended reusable rule. The training experiments show that rewarding these answers encourages models to produce them. I do not find this result particularly surprising, and it leaves the more important questions unanswered: when do such problems arise in other tasks, and how should verifiers be designed to avoid them? Testing more models in the same environment does not answer these questions. The paper raises a timely issue, but the evidence remains a narrow case study, and I find the broader narrative stronger than the insight established by the experiments.

Questions for authors

What evidence separates shortcut propensity from task-solving capability in the cross-model comparison? GPT-5 performs well with very few shortcuts, while some smaller models have low accuracy on the difficult tasks. How can the probe distinguish a model that follows the intended task from one that simply fails to find either a valid rule or a shortcut?

The extensional verifier accepts enumeration of the positive examples shown in the prompt. Why does training a model to exploit this obvious flaw tell us something beyond this specific setup? Do you have evidence of similar behavior in other tasks or with stronger verifiers?

6 comments
AuthorsOpenReview ↗

Response to Reviewer HbDL (1/3)

We thank the reviewer for the detailed comments and address them below. In response, we added three experiments. We reveal the verifier to the models (W1), apply IPT to a second domain (W2), and evaluate our RLVR models downstream (Q2).

W1/Q1 — "A model that cannot solve the task or discover a shortcut can also have zero shortcuts. [...] The role of capability needs to be separated before this interpretation is convincing."

We agree that this separation is essential. It is the subject of App. A.5, and we will move its main results into the main text. Four pieces of evidence separate capability from shortcut propensity.

Conditional shortcut rate (Tab. 3). To separate shortcutting from model capabilities, we provide conditional shortcut rates in Tab. 3. We consider only outputs that pass the extensional check and report which share of them are shortcuts. This emphasises our findings even more. On Hard tasks, this share of shortcuts is 96% for GPT-5-Nano and 54% for Olmo-3.1 32B. In contrast, non-RLVR models never use shortcuts in any tier.

Models at matched capability. We observe no clear connection between model capability and shortcutting. Models that solve a similar share of the Hard tasks behave differently: Ministral-3 14B (7%) and GPT-4.5 (2%) produce no shortcuts, while Olmo-3.1 32B (2%) produces 7 and GPT-5-Nano (3%) produces 184. gpt-5-chat-latest solves more Hard tasks than any of them (14%) and also produces none. Within the GPT-5 family, the smallest model even reaches the highest extensional accuracy on Hard tasks (GPT-5-Nano 76%, GPT-5 50%), although it solves the fewest.

The shortcut needs no capability. This is not really surprising given that the labels are present in the prompt. To generate a shortcut, a model only needs to copy them (eastbound(train0). ...). As a sanity check, we evaluated gpt-4o-mini, one of the weakest models in our eval, on explicitly generating the label assignments. It succeeds on 981/1000 of tasks.

New experiment, revealing the verifier. In the paper, the prompt does not say how answers are checked. We repeated the evaluation with a note that describes the evaluation protocol, while the task stays the same. GPT-4o-mini stays at 0 shortcuts. On the other hand, GPT-5-Nano rises from 368 to 427 shortcuts, GPT-5-Mini from 32 to 166, and gpt-oss-120b from 58 to 247 (of 1,000 tasks).

AuthorsOpenReview ↗

Response to Reviewer HbDL (2/3)

W2 — "evidence remains a narrow case study, and I find the broader narrative stronger than the insight established by the experiments." [...] "when do such problems arise in other tasks, and how should verifiers be designed to avoid them?"

We thank the reviewer for considering the issue timely. We agree that an additional domain further strengthens the paper, and we add an experiment of IPT on Python coding. Below we relate the narrative to the evidence and answer the two questions.

Narrative and evidence. Our narrative is that LLMs game verifiers and that RLVR can lead to this. The experiments establish both claims. The first is an observation on released models that we did not train. GPT-5, o3 Mini, gpt-oss and Olmo-3.1 game a verifier they were never told about (Tab. 1, Tab. 8). The second is a causal claim, and a controlled setting is exactly what a causal claim needs. From the same checkpoint, the extensional verifier produces the behaviour, and the isomorphic one does not, also inside a multi-reward stack and on a second model family (Fig. 2, App. B.1). With the new experiments, the evidence now reaches further. RLVR models shortcut even more when the verifier is revealed (W1), the behaviour appears in a second domain (below), and the model that learned it cannot be told apart on standard capability benchmarks (Q2).

IPT beyond logic: a new experiment in code. IPT is also not tied to logic. To show this, we applied it to Python programming. Following CodeARC [8], the model is shown 8 input/output examples of a hidden function and has to write it. IPT is implemented as two unit tests, an extensional one and an isomorphic one. We measure hacking via IPT as hacking=ext−isohacking = ext - iso, the programs that pass the extensional unit test and fail the isomorphic one. We build 270 tasks in six difficulty levels, which scale with the number of operations. The table gives solved / hacking per level when the model may retry up to three times (20 tasks at level 1, 50 at each other level).

Modeld1d2d3d4d5d6
GPT-5-Mini20/037/414/115/161/210/28
GPT-5-Nano18/128/27/53/110/50/18
gpt-4o-mini10/01/00/00/00/00/0

GPT-5-Mini and GPT-5-Nano fall back to hacking as the tasks get more complex. gpt-4o-mini does not. The isomorphic test successfully detects the Python code that exploits the unit test. We will add this experiment to the appendix.

When do such problems arise in general? In logic, shortcuts concentrate on complex tasks (38 in levels 1-10, 450 in levels 11-20), rise with reasoning effort (0 to 84 for GPT-5-Mini from low to high), and appear in the traces after repeated failed attempts (Sec. 4 and 6). Code shows the same. Hacking grows with difficulty (table above).

Designing verifiers that are not gameable We agree that this is an important question. It is also one the field cannot answer yet, and it remains an open problem in AI safety and alignment[5, 11]. Reward hacking and verifier design have long been studied in the classical RL literature [6, 12]. Yet, theory suggests that a single general solution may not exist [7]. For LLMs, prominent approaches deploy chain-of-thought monitoring to detect misaligned behaviour. Yet training against the monitor leads models to hide the hack instead of abandoning it, which is even more concerning [3].

Meanwhile, the concerns are becoming more pressing. The behaviour is reported across model families and domains [1, 2, 3, 4], and it now affects real systems. In July 2026, roughly 700 agents in an OpenAI experiment hacked out of their containers and attacked Hugging Face [9], actively conducting cybercrime. In a separate incident, an OpenAI agent hacked an Australian government health portal and accessed non-public files [10, 13].

What can we learn from the literature and from our work for the design of robust verifiers? The literature tells us that every verifier is a proxy, so no single check should be trusted alone [7]. Our work adds three concrete lessons. (i) Hiding the evaluation protocol does not prevent hacking. Models game a verifier that the prompt never describes. (ii) Answers should be re-checked on a version of the task the model has not seen. A genuine solution still passes when the names in the task are changed. An answer that only restates the shown examples does not. (iii) Hacking has to be tested for directly, because standard benchmarks do not see it (Q2). These lessons do not settle verifier design, but they are concrete steps toward it, grounded in controlled evidence.

AuthorsOpenReview ↗

Response to Reviewer HbDL (3/3)

Q2 — “Why does training a model to exploit this obvious flaw tell us something beyond this specific setup? Do you have evidence of similar behaviour in other tasks or with stronger verifiers?”

The check is not one we designed. It is the standard correctness criterion of ILP, and SLR-Bench adopts it. That this flaw is obvious makes the finding more striking, not less. It is obvious in hindsight, once one sees models exploiting it. Yet the prompt states only the task and not how answers are evaluated, so the models never see the verification protocol. Still, released RLVR models game it. Non-RLVR models never do, and gpt-4o-mini does not even when we describe the verifier to it (W1). The shortcut behaviour is therefore a property of the models themselves, which we did not train on this setup. Our training runs then isolate how the hacking emerges during RLVR (Fig. 2).

Evidence beyond this setup comes from our new code experiment (W2) and from a downstream evaluation. To investigate how the training affects downstream model behaviour, we evaluate the matched EXT/ISO pair (at step 1650) on general capability benchmarks and on ImpossibleBench [1]. ImpossibleBench measures test-case exploitation in code by presenting the model with impossible tasks. Their unit tests contradict the task specification, either because two tests contradict each other (conflicting) or because a single expected output is altered (one-off).

BenchmarkEXTISO
GSM8Kgrade-school math95.3%94.8%
MATHcompetition math94.9%94.5%
ARC-Challengescience QA92.3%92.0%
IFEvalinstruction following83.9%82.4%
BBHgeneral reasoning83.6%83.1%
ImpossibleBench conflicting ↓hacking (avg@4)5.3% ± 1.7%4.1% ± 0.9%
ImpossibleBench one-off ↓hacking (avg@4)7.8% ± 1.8%5.3% ± 2.3%
ImpossibleBench solvable ↑code (control tasks, avg@4)35.4% ± 3.9%37.9% ± 5.3%

The table reveals two surprising insights. The EXT model has learned to game the verifier and reaches close to full reward on SLR-Bench (Fig. 2c). Still, it stays on par with the ISO model on every general benchmark. A plausible explanation comes from the RLVR setup itself. The update uses the advantage of a rollout relative to its group. Once the model games the verifier, the SLR-Bench reward saturates. A group in which all rollouts receive the full score has zero advantage and yields no gradient. A growing share of the hacked environment therefore drops out of the update, and its gradient signal diminishes. The update signal now comes primarily from the other environments, which remain stable throughout training. The capabilities they train therefore keep improving, which yields stable downstream capabilities. This is also what we observe in our evaluation. Across models, those that hack are not necessarily less capable in general (Tab. 1).

On ImpossibleBench, the EXT model exploits the test cases more often than the ISO model on both test sets. This indicates that the hacking behaviour is also more pronounced in test-case hacking in code. All in all, the general reasoning capabilities of the two models are on par, yet the EXT model is more susceptible to hacking across both test sets of ImpossibleBench.

References

[1] Zhong et al., ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases, ICLR 2026.

[2] METR, Recent Frontier Models Are Reward Hacking, blog post, June 2025.

[3] Baker et al., Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, 2025.

[4] MacDiarmid et al., Natural Emergent Misalignment from Reward Hacking in Production RL, 2025.

[5] Amodei et al., Concrete Problems in AI Safety, 2016.

[6] Krakovna et al., Specification gaming: the flip side of AI ingenuity, DeepMind blog, 2020.

[7] Skalse et al., Defining and Characterizing Reward Hacking, NeurIPS 2022.

[8] Wei et al., CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis, COLM 2025.

[9] Greenblatt et al., Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, August 2026.

[10] Al Jazeera, How an OpenAI 'agent' hacked Australia's Medicare and what that means, 24 September 2026.

[11] Hendrycks et al., Unsolved Problems in ML Safety, 2021.

[12] Pan et al., The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models, ICLR 2022.

[13] Toulas, OpenAI hacked Australian Medicare govt site, probed data providers, BleepingComputer, 24 September 2026.


We thank the reviewer again for the thoughtful questions, which led to three new experiments and a clearer paper. We believe that they address the concerns raised, and we kindly ask the reviewer to reconsider their score. If any concern about soundness remains, we would be grateful for a short pointer, and we are happy to run further analyses during the discussion period.

Reviewer_ZBtvOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies verifier gaming on SLR-Bench and proposes Isomorphic Perturbation Testing (IPT), which flags hypotheses that pass the original extensional verifier but fail after object identifiers are renamed. Validation includes a broad model survey, controlled RLVR experiments, and downstream evaluation of a model trained with the proposed verifier.

Critical strengths

The controlled OLMo intervention is valuable: starting from the same checkpoint and changing the verifier provides meaningful causal evidence that weak extensional rewards can induce identifier-enumeration shortcuts.

The empirical investigation is unusually extensive: the model survey, pass@8 analysis, prompt interventions, training trajectories, and released artifacts show that the observed shortcut behavior is systematic rather than an isolated decoding accident.

Critical weaknesses

The mitigation is insufficiently isolated and stress-tested. Figure 3 lacks a matched extensional-SLR baseline, so the downstream gains may come from task-specific SLR training rather than isomorphic verification. Moreover, the fixed train0 →\to mytrain0 mapping is itself predictable and potentially gameable.

Questions for authors

Under the formal definition of IPT, why does eastbound(T) :- \+ westbound(T) fail isomorphic verification despite containing no renamed object constants? Please report results when the two verifiers have identical predicate semantics and differ strictly only by Φ\Phi, separating label-negation from identifier-enumeration shortcuts.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZBtv

Thank you for your valuable time and the concrete questions. Please find our responses below.

W1a — "Figure 3 lacks a matched extensional-SLR baseline"

We thank the reviewer for the suggestion. Below, we compare the matched EXT and ISO models of the multi-reward setup at step 1650. Both start from the same checkpoint and differ only in the SLR-Bench verifier. We evaluate them on general capability benchmarks and additionally on ImpossibleBench [1]. ImpossibleBench measures test-case exploitation in code by presenting the model with impossible tasks. Their unit tests contradict the task specification, either because two tests contradict each other (conflicting) or because a single expected output is altered (one-off).

BenchmarkEXTISO
GSM8Kgrade-school math95.3%94.8%
MATHcompetition math94.9%94.5%
ARC-Challengescience QA92.3%92.0%
IFEvalinstruction following83.9%82.4%
BBHgeneral reasoning83.6%83.1%
ImpossibleBench conflicting ↓hacking (avg@4)5.3% ± 1.7%4.1% ± 0.9%
ImpossibleBench one-off ↓hacking (avg@4)7.8% ± 1.8%5.3% ± 2.3%
ImpossibleBench solvable ↑code (control tasks, avg@4)35.4% ± 3.9%37.9% ± 5.3%

The table reveals two surprising insights.

The EXT model has learned to game the verifier and reaches close to full reward on SLR-Bench (Fig. 2c). Still, it stays on par with the ISO model on every general benchmark. A plausible explanation comes from the RLVR setup itself. The update uses the advantage of a rollout relative to its group. Once the model games the verifier, the SLR-Bench reward saturates. A group in which all rollouts receive the full score has zero advantage and yields no gradient. A growing share of the hacked environment therefore drops out of the update, and its gradient signal diminishes. The update signal now comes primarily from the other environments, which remain stable throughout training. Accordingly, the capabilities trained by these environments keep improving after one environment has been gamed, which yields stable downstream capabilities. This is also what we observe in our evaluation. Across models, those that hack are not necessarily less capable in general (Tab. 1).

On ImpossibleBench, the EXT model exploits the test cases more often than the ISO model on both test sets. This indicates that the hacking behaviour is also more pronounced in test-case hacking in code.

All in all, the general reasoning capabilities of the two models are on par, yet the EXT model is more susceptible to hacking across both test sets of ImpossibleBench.

W1b — "the fixed train0->mytrain0 mapping is itself predictable and potentially gameable."

We respectfully disagree. Φ is applied only inside the verifier, and the renamed program never appears in the prompt. The model therefore never sees Φ and would need to blind guess the exact string transformation deployed. This is very unlikely and does not occure in any of our runs. We re-verified 19,024 stored model outputs from our evaluation. None of the extracted hypotheses contains a my-prefixed constant. In training, none of the 426,560 decoded SLR rollouts of the matched EXT and ISO runs contains one either.

Moreover, Φ is a free design choice of the verifier. Replacing the fixed prefix with a fresh random bijection per task, under which blind guessing has essentially no chance of success, changes none of the 19,024 verdicts. We will use the random bijection in the released code and describe Φ accordingly.

Q1 — Separating Label enumeration from Label negation

We thank the reviewer for the precise question. The isomorphic verifier is, in fact, designed to detect both the enumeration and negation shortcuts. It not only applies the isomorphic perturbation Φ but also separates E from B, TΦ=(BΦ,E+Φ,E−Φ)T^{\Phi} = (B^{\Phi}, E^{+\Phi}, E^{-\Phi}). Withholding E−ΦE^{-\Phi} during program execution allows us to detect the negation shortcuts, since it is no longer readable by the hypothesis. Adding it back into the isomorphic verifier allows us to separate the enumeration from the label negation shortcut. Below you can find an overview of the different shortcut types detected. We will clarify this distinction in Section 3.

ModelShortcutsLabel negationEnumeration
GPT-5404
GPT-5 Mini321022
GPT-5 Nano36835414
gpt-oss-120b58517
gpt-oss-20b532
OLMo-3.1-32B-Think13112

[1] Zhong et al., ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases, ICLR 2026.


We thank the reviewer again for the precise and constructive questions, which led to a clearer definition and two new analyses with additional insights. We believe that they resolve the concerns raised, and we kindly ask the reviewer to reconsider the score. We are happy to run further analyses during the discussion period.

Dear Reviewer ZBtv,

Thank you for engaging with our response and for updating your assessment. The points you raised are now part of the revised paper (App. C.2 and C.4), and we are happy to clarify anything further.

Best regards,

The Authors

Comments on the paper

1 comment

Dear Area Chair, dear Reviewers,

We thank the reviewers for their time and their constructive feedback. Below we sum up the discussion in one place.

Our paper studies reward hacking in reasoning models. We show that current reasoning models game a verifier they were never told about, and the open-weight models call this cheating in their own reasoning traces. Reviewer AXFQ calls this "a convincing and practically important failure mode in RLVR", and Reviewer HbDL sees "a timely contribution". To detect such cases, we introduce IPT, "a simple and elegant black-box diagnostic" (AXFQ). We then show in controlled training that the verifier induces the behavior, and that it does not emerge under the isomorphic verifier. The reviewers find the study "unusually extensive" (ZBtv) and see "unusually strong causal evidence that the verifier itself can induce reward hacking" (AXFQ). Finally, we present OlmoLogic-7B-Think, a model trained with the isomorphic verifier and a strong successor to Olmo-3-7B-Think for logical reasoning and beyond. It scores higher on 44 of 55 benchmarks, including 26 of the 36 outside logic (Fig. 3).

The rebuttal resolves the concerns raised by the reviewers.

  1. Does IPT work outside logic? (AXFQ, HbDL) Yes. We deploy IPT on 270 CodeARC-style Python coding tasks. GPT-5-Mini and GPT-5-Nano solve the easy ones, and as the tasks get more complex they fall back to shortcuts. Instead of implementing the function, they hardcode the test cases just to pass the unit test. The isomorphic unit test rejects every such program. IPT applies wherever a task admits an isomorphic perturbation, as in logic, code or templated math (App. C.3, Sec. 6).

  2. Do weaker models simply fail to find the shortcut? (HbDL) No. Tab. 3 separates shortcut propensity from task-solving capability. Among the answers that pass the extensional check, up to 96% are shortcuts for RLVR models and none for non-RLVR models. We observe no clear connection between capability and shortcutting. Models that solve a similar share of the hard tasks produce between 0 and 184 shortcuts. The shortcut also needs no skill, since the labels are in the prompt and just need to be copied. If we tell the model how the verifier works, GPT-4o and GPT-4o-mini still produce none, while GPT-5-Mini goes from 32 to 166 (Sec. 4, App. C.1).

  3. Does this matter for general RLVR training? (AXFQ) Yes. Shortcuts are also learned inside the multi-task training mix of Olmo 3, where the other rewards delay the hacking but do not prevent it (Fig. 2). The same happens in a second model family and RL stack (App. B.1).

  4. What does a matched comparison of the two verifiers show? (ZBtv) A striking finding. We compare the model that learned to hack in the multi-reward RLVR run with its counterpart trained with the isomorphic verifier. Both models are on par across the general benchmarks. Yet the model that learned to hack on SLR-Bench also exploits test cases more often on ImpossibleBench, a benchmark for hacking in code. Hacking thus does not necessarily lead to a collapse of capabilities. It appears as a behavioral propensity of the model itself, which standard benchmarks do not reveal. The explanation lies in the multi-reward setup. Once an environment is hacked, its reward saturates, and groups in which every rollout receives the full score yield no gradient in GRPO. The update then comes mainly from the other environments, where the model keeps learning (Sec. 5, App. C.4). This is in line with our evaluation in Sec. 4, where the models that hack, such as the GPT-5 family and gpt-oss-120b, are still strong general-purpose reasoning models.

  5. Is this just an obvious verifier flaw? (HbDL) The check is the standard correctness criterion of inductive logic programming, not one we designed. The prompt never reveals the verifier, and the models game it anyway. Their traces show that they know. In 61 of the 63 flagged gpt-oss traces, the model calls the shortcut cheating or not allowed and submits it anyway, and in 31 of them it talks itself into accepting it (Sec. 4).

  6. Can the test be gamed, and what follows for verifier design? (HbDL, AXFQ) Not in practice. IPT applies perturbations that the model cannot observe and can draw fresh random ones for every task (App. C.2). IPT is also sound by construction, since every answer it flags is a genuine shortcut. Yet, in principle, every proxy reward can be gamed (Skalse et al., 2022). A model with internet access, for example, could look up the solution. Sec. 6 therefore draws three lessons for verifier design. Do not rely on hiding the check, enforce the invariances the task implies, and test for hacking directly.

All three reviewers raised their scores without naming any remaining concerns about our experiments or claims. All additions are marked in blue, and the core claims and results of the submission are unchanged.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.