- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies a subtle form of reward hacking in reinforcement learning with verifiable rewards (RLVR), where models learn to satisfy an imperfect verifier without performing the intended reasoning task. On inductive logic problems, the authors observe that some RLVR-trained models replace genuine rule induction with instance-level enumeration or other shortcuts that still pass an extensional verifier. To detect this behavior, the paper proposes Isomorphic Perturbation Testing (IPT), which checks whether a model output remains valid after object identifiers are consistently renamed while preserving the underlying logical structure.
Critical strengths
- The paper identifies a convincing and practically important failure mode in RLVR where models can exploit what a verifier fails to enforce rather than solve the intended reasoning task.
- IPT is a simple and elegant black-box diagnostic that distinguishes genuine rule induction from identifier-dependent shortcuts without requiring access to model weights or internal activations.
- The controlled RLVR experiments provide unusually strong causal evidence that the verifier itself can induce reward hacking, rather than merely showing a correlation across different model families.
Critical weaknesses
- The core contribution is relatively narrow, as IPT relies on task-specific isomorphic transformations and is demonstrated primarily in structured inductive logic settings.
- The method requires an automatically constructible invariance that preserves task semantics, which may be difficult to define for open-ended reasoning, mathematics, coding, or other tasks with less explicit structure.
- Although the paper evaluates several logical reasoning benchmarks, the experimental domains remain concentrated on symbolic and inductive reasoning, leaving the broader relevance to general RLVR largely unclear.
- IPT detects only shortcuts that violate the chosen isomorphism, so a model could potentially exploit other verifier weaknesses while still passing both the original and perturbed checks.
Questions for authors
- How can IPT be extended to reasoning tasks where no obvious semantics-preserving isomorphic transformation exists?
- Do you expect the same reward-hacking behavior to appear in domains such as mathematics, coding, or open-ended reasoning?
- Could a model learn shortcuts that remain invariant under the current perturbations and therefore evade IPT?
- How much of the observed benefit comes from IPT specifically, rather than from the highly structured nature of SLR-BENCH and its verifier?
7 comments
Response to Reviewer AXFQ (2/3)
W3, Q4 — "the experimental domains remain concentrated on symbolic and inductive reasoning, leaving the broader relevance to general RLVR largely unclear" [...] "How much of the observed benefit comes from IPT specifically, rather than from the highly structured nature of SLR-BENCH and its verifier?"
SLR-Bench is where we measure the behaviour. The models that show it are general. GPT-5, GPT-5-Mini, GPT-5-Nano, o3 Mini and gpt-oss-120b are strong general-purpose reasoning models, and all of them game the verifier (Tab. 1, Tab. 8). GPT-5-Mini and GPT-5-Nano also hack in code (W1).
Our training experiments show the same for general RLVR. The hacking gap emerges inside the broad multi-environment mix of the Olmo-3 RLVR stack (Fig. 2c,d), and again inside the 15-reward mix of Nemotron, a second model family and RL framework (App. B.1). We further evaluated the matched pair from the Olmo-3 multi-reward stack at step 1650, trained with the extensional (EXT) or the isomorphic (ISO) verifier, on general benchmarks and on ImpossibleBench [1]. ImpossibleBench measures test-case exploitation in code by presenting the model with impossible tasks. Their unit tests contradict the task specification, either because two tests contradict each other (conflicting) or because a single expected output is altered (one-off).
| Benchmark | EXT | ISO | |
|---|---|---|---|
| GSM8K | grade-school math | 95.3% | 94.8% |
| MATH | competition math | 94.9% | 94.5% |
| ARC-Challenge | science QA | 92.3% | 92.0% |
| IFEval | instruction following | 83.9% | 82.4% |
| BBH | general reasoning | 83.6% | 83.1% |
| ImpossibleBench conflicting ↓ | hacking (avg@4) | 5.3% ± 1.7% | 4.1% ± 0.9% |
| ImpossibleBench one-off ↓ | hacking (avg@4) | 7.8% ± 1.8% | 5.3% ± 2.3% |
| ImpossibleBench solvable ↑ | code (control tasks, avg@4) | 35.4% ± 3.9% | 37.9% ± 5.3% |
The table reveals two surprising insights.
The EXT model has learned to game the verifier and reaches close to full reward on SLR-Bench (Fig. 2c). Still, it stays on par with the ISO model on every general benchmark. A plausible explanation comes from the RLVR setup itself. The update uses the advantage of a rollout relative to its group. Once the model games the verifier, the SLR-Bench reward saturates. A group in which all rollouts receive the full score has zero advantage and yields no gradient. A growing share of the hacked environment therefore drops out of the update, and its gradient signal diminishes. The update signal now comes primarily from the other environments, which remain stable throughout training. Accordingly, the capabilities trained by these environments keep improving after one environment has been gamed, which yields stable downstream capabilities. This is also what we observe in our evaluation. Across models, those that hack are not necessarily less capable in general (Tab. 1).
On ImpossibleBench, the EXT model exploits the test cases more often than the ISO model on both test sets. This indicates that the hacking behaviour is also more pronounced on test-case hacking in code.
This matched pair also answers Q4. Both models train on the same SLR-Bench tasks and differ only in the verifier, so the structure of SLR-Bench is identical in both runs. Yet IPT has a pronounced effect on the learned model behaviour. Under the extensional verifier, the hacking gap opens and widens during training (Fig. 2c). Under the isomorphic verifier, it stays closed (Fig. 2d). The structure of SLR-Bench alone thus does not prevent the shortcut, whereas IPT does. At the same time, the general reasoning capabilities of the two models are on par, and the EXT model is more susceptible to hacking across both test sets of ImpossibleBench.
Response to Reviewer AXFQ (3/3)
W4, Q3 — “IPT detects only shortcuts that violate the chosen isomorphism” [...] "Could a model learn shortcuts that remain invariant [...] and therefore evade IPT?"
Yes, such shortcuts are possible, and we do not claim otherwise. IPT targets one class of exploit. As our Limitations state, it “does not exclude other forms of reward hacking, yet reliably identifies shortcuts without false positives by construction, since any valid hypothesis passes both verifiers” (Sec. 6). Every answer that IPT flags is therefore a genuine shortcut. That other exploits exist holds for every verifier and every detection method. Theory even suggests that a generally hack-proof reward does not exist [6].
A model with internet access could, for example, look up the solution online, and no check of the final answer would detect it. Sandboxing does not rule this out either. In July 2026, agents in an OpenAI experiment hacked out of their containers, reached the internet through an external sandbox and attacked Hugging Face, all to obtain a passing score from an automated scorer on tasks that were impossible to solve [7]. As in our experiments, the hacking set in where no genuine solution was within reach. The conclusion is not that targeted tests are of little use. Each class of exploit needs its own test, and IPT provides one for a class that we observe in released models.
References
[1] Zhong et al., ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases, ICLR 2026.
[2] METR, Recent Frontier Models Are Reward Hacking, blog post, June 2025.
[3] Baker et al., Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, arXiv:2503.11926, 2025.
[4] Wei et al., CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis, COLM 2025.
[5] Mirzadeh et al., GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models, ICLR 2025.
[6] Skalse et al., Defining and Characterizing Reward Hacking, NeurIPS 2022.
[7] Greenblatt et al., Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, METR and Redwood Research, August 2026.
We thank the reviewer again for the questions. They led us to extend the study to a second domain and made the paper stronger. We believe that the new experiments and clarifications address the reviewer’s concerns, and we kindly ask the reviewer to reconsider the score. If any concern about the soundness of our experiments or claims remains, we would be grateful for a short pointer and will gladly address it. We are happy to run further analyses during the discussion period.
Response to Reviewer AXFQ (1/3)
Thank you for your review. We were encouraged that you found the failure mode “convincing and practically important”, IPT “simple and elegant”, and the controlled experiments “unusually strong causal evidence”. In response to the concerns, we added an experiment in a second domain (code) and a downstream evaluation of our matched pair of models.
W1, Q2 — “The core contribution is relatively narrow” [...] “Do you expect the same reward-hacking behaviour to appear in [...] mathematics, coding, or open-ended reasoning?”
The focus of our paper is diagnosing reward hacking and showing how it can emerge during RLVR. We show that released reasoning models game a verifier they were never told about, which happens across multiple model families (Tab. 1, Tab. 8). We show that their own traces call the shortcut cheating (Sec. 4). And we show in controlled training, on two model families and two RL stacks, that the verifier causes the behaviour (Fig. 2, App. B.1). IPT is the instrument that makes all of this measurable.
IPT is also not tied to logic. To show this, we applied it to Python programming. Following CodeARC [4], the model is shown 8 input/output examples of a hidden function and has to write it. IPT is implemented as two unit tests, an extensional one and an isomorphic one. We measure hacking via IPT as hacking=ext−iso, the programs that pass the extensional test and fail the isomorphic one. We build 270 tasks in six difficulty levels, which scale with the number of operations. The table gives solved / hacking per level when the model may retry up to three times (20 tasks at level 1, 50 at each other level).
GPT-5-Mini and GPT-5-Nano fall back to hacking as the tasks get more complex. gpt-4o-mini does not. The isomorphic test successfully detects the Python code that exploits the unit test. We will add this experiment to the appendix.
So yes, we expect the same behaviour in other domains. In coding, we observe it directly, in line with independent reports of test-case exploitation [1, 2, 3]. For mathematics and open-ended reasoning, the condition we identify applies unchanged. Models learn to hack when verifiers admit false positives, meaning they accept answers that do not solve the task, and model shortcuts when a genuine solution is out of reach for them. Audits report that rule-based and model-based verifiers in mathematical reasoning are systematically gameable (Sec. 2). Our controlled training shows how RLVR can turn such a gap into learned behaviour (Fig. 2).
W2, Q1 — "requires an automatically constructible invariance" [...] "How can IPT be extended to reasoning tasks where no obvious semantics-preserving isomorphic transformation exists?"
We agree that IPT needs a task structure on which a semantics-preserving transformation can be defined. We state this in our Limitations, where IPT "requires a domain where meaningful isomorphic perturbations can be generated automatically, which is natural for logic-based tasks but less direct for open-ended generation" (Sec. 6). We see this as the scope of the method. No approach to detecting reward hacking is universal. ImpossibleBench requires code tasks with unit tests [1]. Chain-of-thought monitoring requires access to the reasoning trace of the model [3] and can emit false positives. IPT requires a renaming. In return, it needs neither traces nor model internals, and it has no false positives by construction. Further, IPT can be deployed well beyond logic. IPT is a natural fit for RLVR, which is built around verifiable tasks and formal checks. This formal structure is what an isomorphic transformation builds on. We deploy IPT for logic and, in our new experiment, in code (W1). Templated mathematics problems could be treated in the same way, as the templates of GSM-Symbolic show [5].