- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper finds, in a simulated institutional environment, that reward optimization prompts large models to propose strategies that contradict institutional intent when there are no explicit instructions to exploit vulnerabilities, and investigates how this behavior responds to rule patching and security intervention.
Critical strengths
It uses real historical legislation as a reference and combines simulated environments, multi-model reproduction, and manual auditing to turn "models exploiting loopholes in social systems" into a problem that can be studied in a controlled manner.
Critical weaknesses
The environment is selected based on the existence of vulnerabilities and cannot estimate the actual occurrence rate. Historical environments are selected from known vulnerabilities, while synthetic environments artificially implant vulnerabilities. This is suitable for proving that phenomena can occur, but it cannot indicate how common they are in ordinary systems, nor can "finding approximately 61% of known vulnerabilities" be interpreted as the actual incidence rate of harm.
There is still a significant gap between generating a suspicious plan and its actual successful execution. The model only writes the strategy; the execution result is determined by the closed set of actions and simulated rules, without real human supervision, institutional reactions, and execution uncertainty. Therefore, it demonstrates the risk mechanism but has not yet measured real social losses.
Simulator auditing is valuable, but it does not cover the complete reward chain. The authors manually checked the action parsing of 64 strategies, obtaining an 89.8% annotation acceptance rate; this mainly verifies whether "the strategy is reasonably interpreted as an action," and does not equate to verifying the correctness of subsequent state changes, payoff calculations, and realistic feasibility.
Some references that authors may consider to discuss (and recent papers that are related to reward hacking, this is minor and will not influence the rating but I hope authors can do that):
Huang, Yue, et al. "Emergent Social Intelligence Risks in Generative Multi-Agent Systems." arXiv preprint arXiv:2603.27771 (2026).
Overall, I think it (will be) valuable after the revision and looking forward to authors' rebuttal.
Questions for authors
See weaknesses.
Response to Reviewer Dcks, Part 1
Thank you very much for recognising our work! We hope the following responses could address your concerns.
Q1. The environment is selected based on the existence of vulnerabilities and cannot estimate the actual occurrence rate.
R1: We fully agree. The experiments in this paper provide a proof of existence. They demonstrate that societal hacking can occur under the given conditions, but do not estimate its occurrence rate in general institutional settings. This point has already been discussed in detail in both the main text and the Limitations section of the appendix.
The fact that "Historical environments are selected from known vulnerabilities" is a natural methodological design. To determine whether a model exploits a loophole, we need both the "loophole + subsequent amendment" as ground truth. Only historically documented loopholes satisfy this requirement. Estimating the actual occurrence rate would require deploying models in real institutional settings, which is not feasible for safety reasons.
Anthropic's concurrent work, Training a Misaligned Reward Seeker (https://alignment.anthropic.com/2026/reward-seeker), also deliberately selects 80 environments that can be exploited for training, describing this as a "pessimistic training run" to examine whether the phenomenon occurs under worst-case conditions.
We explicitly state in the paper that 61% refers to the recall of historically documented loopholes in the Historical scenario (lines 242–243), rather than the "actual incidence rate of harm". The latter is outside the scope of this paper. We will reinforce such claim in the paper.
Q2. There is still a significant gap between generating a suspicious plan and its actual successful execution.
R2: The primary reason we conduct simulations rather than real-world social experiments is about safety, which we have already discussed in detail in the Limitations section of the appendix.
Another reason is that long-horizon, large-scale social simulations are inherently very unreliable. To conduct experiments under controlled, reproducible, and safe conditions, we simplify the environments and conduct experiments based on simulated strategies.
Furthermore, for strategies corresponding to historical amendments, their real-world feasibility has already been demonstrated by historical records and does not depend on the simulator. These amendments were introduced precisely because someone had implemented the corresponding strategies and succeeded in the real world.
Thank you for your suggestions, and we will add this point to the Limitations section.
Q3. Simulator auditing is valuable, but it does not cover the complete reward chain.
R3: In our simplified environments, "subsequent state changes, payoff calculations" mainly involve simple text classification and mathematical calculations. We did not observe any errors in the LLM simulator within the scope of our manual auditing.
We have analysed realistic feasibility. Please refer to Appendix Section C.5 and Table A7.
In addition, in the experiment in Appendix Section D.3, we provide the policy with two strategies that receive different scores under the same prompt and ask it to predict which one the simulator will score higher. Its accuracy is only 53.7%, with the results mainly attributable to positional preference. This indicates that the policy exploits loopholes in the regulations themselves, rather than the simulator's preferences.
We also conducted additional experiments to verify the robustness of the simulator. We replaced the simulator with GPT-5.6 and GLM-5.3, respectively, so that action parsing, state construction, and scoring were all performed by different models, and retrained the models across all environments. On the Historical subset, R@Full is 53.1 and 53.8, respectively, while P@Full is 90.9 and 91.8. The societal hacking phenomenon still persists.
We will add these results to the paper.
Q4. Some references that authors may consider to discuss.
R4: Thank you very much for providing this reference!
The emergent social risks studied in Huang et al. (2026) are very similar to the area of societal hacking that we aim to investigate. The main difference is that this paper studies interactions among multiagent systems during inference, whereas we focus on reward hacking behaviours of models during training.
We will add this important reference and include more recent related papers on reward hacking in the Related Work section. Thank you!