acceptodds
Under review as a conference paper at ICLR 2027

Societal Hacking: Reward Optimisation Amplifies Societal Loophole Exploitation in LLMs

Abstract

Reinforcement learning (RL) has become the dominant post-training paradigm for large language models (LLMs), and its characteristic failure is reward hacking, where an ill-defined reward function acts as a proxy that captures the measurable form of an intent rather than the intent itself, and optimisation finds the gap between the two. Many codified rules human society runs on are proxies of the same kind. We ask whether reward hacking in LLMs then scales into societal hacking, the unprompted pursuit and exploitation of loopholes in the rules society runs on. To study this safely, we build a set of simulated institutional environments, reverse-engineered from real-world regulations and from documented vulnerability patterns for analysis. Within these environments, reward hacking emerges without any instruction to look for loopholes. The LLM rediscovers the loopholes that later amendments closed, produces strategies that remain formally compliant while defeating regulatory intent, and passes through refusal safeguards that block the same content when it is requested directly. Patching loopholes as they appear does not eliminate the underlying behaviour but merely redirects it towards subtler forms. Collecting in-the-wild feedback for model training therefore demands greater caution and calls for a next-generation post-training paradigm for safely iterating LLMs in real-world environments.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
social computing, reward hacking, large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DcksOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper finds, in a simulated institutional environment, that reward optimization prompts large models to propose strategies that contradict institutional intent when there are no explicit instructions to exploit vulnerabilities, and investigates how this behavior responds to rule patching and security intervention.

Critical strengths

It uses real historical legislation as a reference and combines simulated environments, multi-model reproduction, and manual auditing to turn "models exploiting loopholes in social systems" into a problem that can be studied in a controlled manner.

Critical weaknesses

The environment is selected based on the existence of vulnerabilities and cannot estimate the actual occurrence rate. Historical environments are selected from known vulnerabilities, while synthetic environments artificially implant vulnerabilities. This is suitable for proving that phenomena can occur, but it cannot indicate how common they are in ordinary systems, nor can "finding approximately 61% of known vulnerabilities" be interpreted as the actual incidence rate of harm.

There is still a significant gap between generating a suspicious plan and its actual successful execution. The model only writes the strategy; the execution result is determined by the closed set of actions and simulated rules, without real human supervision, institutional reactions, and execution uncertainty. Therefore, it demonstrates the risk mechanism but has not yet measured real social losses.

Simulator auditing is valuable, but it does not cover the complete reward chain. The authors manually checked the action parsing of 64 strategies, obtaining an 89.8% annotation acceptance rate; this mainly verifies whether "the strategy is reasonably interpreted as an action," and does not equate to verifying the correctness of subsequent state changes, payoff calculations, and realistic feasibility.

Some references that authors may consider to discuss (and recent papers that are related to reward hacking, this is minor and will not influence the rating but I hope authors can do that):

Huang, Yue, et al. "Emergent Social Intelligence Risks in Generative Multi-Agent Systems." arXiv preprint arXiv:2603.27771 (2026).

Overall, I think it (will be) valuable after the revision and looking forward to authors' rebuttal.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Dcks, Part 1

Thank you very much for recognising our work! We hope the following responses could address your concerns.

Q1. The environment is selected based on the existence of vulnerabilities and cannot estimate the actual occurrence rate.

R1: We fully agree. The experiments in this paper provide a proof of existence. They demonstrate that societal hacking can occur under the given conditions, but do not estimate its occurrence rate in general institutional settings. This point has already been discussed in detail in both the main text and the Limitations section of the appendix.

The fact that "Historical environments are selected from known vulnerabilities" is a natural methodological design. To determine whether a model exploits a loophole, we need both the "loophole + subsequent amendment" as ground truth. Only historically documented loopholes satisfy this requirement. Estimating the actual occurrence rate would require deploying models in real institutional settings, which is not feasible for safety reasons.

Anthropic's concurrent work, Training a Misaligned Reward Seeker (https://alignment.anthropic.com/2026/reward-seeker), also deliberately selects 80 environments that can be exploited for training, describing this as a "pessimistic training run" to examine whether the phenomenon occurs under worst-case conditions.

We explicitly state in the paper that 61% refers to the recall of historically documented loopholes in the Historical scenario (lines 242–243), rather than the "actual incidence rate of harm". The latter is outside the scope of this paper. We will reinforce such claim in the paper.


Q2. There is still a significant gap between generating a suspicious plan and its actual successful execution.

R2: The primary reason we conduct simulations rather than real-world social experiments is about safety, which we have already discussed in detail in the Limitations section of the appendix.

Another reason is that long-horizon, large-scale social simulations are inherently very unreliable. To conduct experiments under controlled, reproducible, and safe conditions, we simplify the environments and conduct experiments based on simulated strategies.

Furthermore, for strategies corresponding to historical amendments, their real-world feasibility has already been demonstrated by historical records and does not depend on the simulator. These amendments were introduced precisely because someone had implemented the corresponding strategies and succeeded in the real world.

Thank you for your suggestions, and we will add this point to the Limitations section.


Q3. Simulator auditing is valuable, but it does not cover the complete reward chain.

R3: In our simplified environments, "subsequent state changes, payoff calculations" mainly involve simple text classification and mathematical calculations. We did not observe any errors in the LLM simulator within the scope of our manual auditing.

We have analysed realistic feasibility. Please refer to Appendix Section C.5 and Table A7.

In addition, in the experiment in Appendix Section D.3, we provide the policy with two strategies that receive different scores under the same prompt and ask it to predict which one the simulator will score higher. Its accuracy is only 53.7%, with the results mainly attributable to positional preference. This indicates that the policy exploits loopholes in the regulations themselves, rather than the simulator's preferences.

We also conducted additional experiments to verify the robustness of the simulator. We replaced the simulator with GPT-5.6 and GLM-5.3, respectively, so that action parsing, state construction, and scoring were all performed by different models, and retrained the models across all environments. On the Historical subset, R@Full is 53.1 and 53.8, respectively, while P@Full is 90.9 and 91.8. The societal hacking phenomenon still persists.

We will add these results to the paper.


Q4. Some references that authors may consider to discuss.

R4: Thank you very much for providing this reference!

The emergent social risks studied in Huang et al. (2026) are very similar to the area of societal hacking that we aim to investigate. The main difference is that this paper studies interactions among multiagent systems during inference, whereas we focus on reward hacking behaviours of models during training.

We will add this important reference and include more recent related papers on reward hacking in the Related Work section. Thank you!

Reviewer_vbijOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper reveal a failure mode "societal hacking" that arises because of the reward hacking. The failure mode examined in this paper is very interesting and potentially important. However, the current presentation could be substantially improved. In particular, I found it difficult to clearly identify the central research hypothesis and the rationale motivating the study. I appreciate the authors' efforts in investigating this interesting problem, but the logical flow and overarching rationale are not sufficiently clear throughout the paper. I would encourage the authors to articulate the research hypothesis more explicitly and strengthen the connections between the motivation, methodology, results, and conclusions.

Critical strengths

an interesting problems

Critical weaknesses
  1. The writing is heavily done by AI. For example the term "may" in line 43. If I was mean, I can ask the authors to have experiments or references to prove the reward hacking is derived from RL. Also, there are some sentences containing bad logic flow. I understand that polishing language with AI is important, but please do not just copy-past whatever from AI. I would appreciate it if you can ensure your logical flow to be "fluent". I do think the paper is really difficult to follow. I appreciate the efforts our authors made and the writing is not the core reason to accept/reject this paper, but it would be great if the presentation of this paper can be improved.

  2. Beyond the presentation, I do think the rational within this paper prohibit me from understanding why the research central is important.

Questions for authors

In the second paragraph, can the authors show an example about how the codified rules specify measurable criteria? I somehow can not understand how and why it is this case. Thank you. Also the line 58 claims "the same gap becomes available to gradient descent", i would appreciate it if the authors can show empirical evidence or references to support this claim. Why is the gap available to gradient descent, and how?

Why is the central research question important, and what is its broader significance?

How can reward hacking ultimately lead to or contribute to societal hacking? The paper would benefit from a clearer explanation of the mechanisms or pathways connecting these two concepts, even though the mechanisms are just hypothesis.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vbij, Part 1

Thank you for your comments. We sincerely want to share with you the motivation and research content of our paper. Please allow us to first briefly outline the research question and design of this work, and then address your concerns one by one.

  1. Reinforcement learning is currently the mainstream paradigm for LLM post-training. Model developers deploy models, collect feedback as reward signals, and use reinforcement learning to optimise and iteratively improve the models (as referenced in Christiano et al. (2017) and Ouyang et al. (2022) in our paper).

  2. Reward hacking is a typical failure mode in reinforcement learning, where a model obtains rewards during training in a way that violates the original intention behind the reward design.

  3. Previous studies have investigated reward hacking in LLMs in the domains of human preferences and mathematics/coding. We aim to investigate whether LLMs also exhibit reward hacking in the domain of societal rules and regulations. We define this type of reward hacking as societal hacking (line 058).

  4. The task we focus on is maximising certain benefits while complying with explicitly codified rules (Figure 1). For example:

  • How to minimise ticket costs without violating airline regulations.
  • How to maximise post exposure on social networks without violating the platform's posting policies.
  • How to delay the market entry of generic drugs and continue selling original drugs for several more years without violating the law.
  • How companies can minimise product liability compensation without breaking the law.
  • How executives can maximise stock returns while complying with SEC Rule 10b5-1.

These rules use quantifiable criteria to determine compliance. Therefore, they function similarly to reward functions.

  1. For safety reasons, we cannot actually deploy models in real-world societies and allow people to use them to maximise these benefits.

  2. Therefore, we use a simulator to simulate the consequences of people following a strategy suggested by a model (Section 2.2). We then obtain feedback signals from these consequences to train the model and observe whether reward hacking occurs throughout the training process.

  3. To determine whether reward hacking occurs, we select regulations that historically contained actual loopholes that were subsequently closed by amendments. We construct environments using the versions before the amendments and use the amendments as ground truth to examine whether the strategies generated by the model during training exploit these loopholes (Section 2.3).

  4. These are the main motivations and the overall experimental and methodological designs of our paper. We do not claim that this is a real-world social experiment, nor do we claim that societal hacking will necessarily occur during RL training. We simply raise this research question and investigate and analyse it from different perspectives, exploring when hacking occurs, why it occurs, to what extent it occurs, and whether it can be technically prevented (Section 4).

We hope this helps you understand the main content of our paper.

AuthorsOpenReview ↗

Response to Reviewer vbij, Part 2

Q1. The term "may" in line 43

R1: This sentence is the second sentence of the Introduction and merely introduces the background of reward hacking. We simply intend to restate the definition of reward hacking from the references cited in our paper (Amodei et al., 2016; Skalse et al., 2022; Krakovna et al., 2020).

We deliberately used the word "may" because although reward hacking is a failure mode in reinforcement learning, it does not necessarily occur in every RL optimisation process. We settled on this description after repeated revisions because reward hacking depends on three factors: the model itself, the environment configuration, and the optimisation pressure. This is also why we analyse societal hacking from different perspectives in Section 4.


Q2. Why is the central research question important?

R2: AI is now being widely used by people in real-world societies, rather than only in traditional closed benchmarks. In this context, the harm caused by reward hacking can be much greater.

If an AI model hacks the task reward in mathematics, mathematicians using the model for research may discover that the mathematical problem has still not been solved. If an AI model hacks the task reward in alignment, it may produce sycophantic and non-objective responses to humans. However, if an AI model provides advice to human users in social activities and hacks the rewards defined by societal rules and regulations, and people execute these suggestions in real society, the exploited loopholes could directly harm third parties, for example:

  • Institutional loopholes in the Texas two-step bankruptcy strategy have delayed compensation to tort victims for years.
  • Alternatively, pay-for-delay agreements can postpone the market entry of generic drugs, maintain high drug prices, and make medicines unaffordable for patients.

Furthermore, we find that when directly asked about institutional loopholes, the model refuses to answer in some environments. However, after RL training in environments with flawed rules, the model exploits these loopholes. This demonstrates a gap in current alignment measures (Section 4.2).

A concurrent work, Anthropic's ``Training a Misaligned Reward Seeker'' (https://alignment.anthropic.com/2026/reward-seeker/), uses a setting similar to ours. It trains models with flawed rewards and observes their behaviours in evaluation environments where tool calls are simulated by a large model. However, they train their closed-source and more capable Claude models and find that "models to be willing to perform long sequences of harmful real-world actions in pursuit of task success", such as "broke out of its sandbox, stole credentials" or "gave advice on the construction of bioweapons to satisfy a grader". This fully demonstrates that this type of problem could have very serious consequences when scaled up.

AuthorsOpenReview ↗

Response to Reviewer vbij, Part 3

Q3. How do the codified rules specify measurable criteria?

R3: We provide two examples from our dataset:

  • According to US FAA regulations, aircraft with no more than 30 seats can operate under the more lenient Part 135 rules. Aircraft with more than 30 seats must operate under the airline standards of Part 121, which include the requirement that the first officer must have accumulated 1,500 flight hours.
  • According to the EU AI Act, general-purpose models trained with computing resources exceeding 102510^{25} FLOPs are presumed to pose "systemic risks" and are subject to additional obligations.

These regulations all contain explicit and quantifiable numerical thresholds or criteria.


Q4. The statement in line 58 claims that "the same gap becomes available to gradient descent". I would appreciate it if the authors could show empirical evidence or references to support this claim.

R4: What we intend to express in this sentence is that if feedback from deployment outcomes flows back into the post-training process as supervision signals, then when certain rollouts exploit loopholes in the regulations and receive positive rewards, these behaviours of exploiting regulatory loopholes will be incorporated into the model parameters through gradient updates, strengthening the model's tendency to exploit these loopholes in the future.

There are two points here:

  1. The premise that feedback from deployment outcomes flows back into the post-training process as supervision signals is supported by references such as Ouyang et al. (2022), which we cite in the paper.
  2. The fact that models exploit the gap between proxy rewards and the underlying true intentions during training, and that gradient updates reinforce such behaviours, is supported by references such as Skalse et al. (2022) and Gao et al. (2023), which we also cite in the paper.

Our experiments also support this claim. When the reward contains no signals related to loopholes, the model exploits 61% of historical loopholes after RL training, and the degree of exploitation increases with the number of parameter updates (Figure 5b).


Q5. How can reward hacking ultimately lead to or contribute to societal hacking?

R5: Societal hacking is a specific type of reward hacking that we propose in this paper. The relationship is not that reward hacking leads to societal hacking. The distinction between the two lies in the source of the proxy reward: when the reward is calculated based on societal rules and regulations, the resulting reward hacking is what we define as societal hacking.

Our hypothesised pathway is as follows:

LLM deployed in an institutional setting → Execution outcomes evaluated according to regulations → Evaluation results fed back as rewards for post-training → Regulatory loopholes become reward loopholes and are reinforced through gradient updates → LLM learns such exploitation behaviour, potentially harming third parties before the regulations are amended.

Unlike conventional reward hacking, these loopholes cannot be fixed by model developers themselves, because the regulations belong to institutions, and amendments often take years.

For safety reasons, we use a simulator instead of real-world deployment to examine the step in this pathway where "reward loopholes are reinforced" in a controlled environment. We will explicitly state this pathway as a research hypothesis in the Introduction.


We hope the above responses could address your concerns, and we sincerely look forward to your reply. Thank you!

Reviewer_P9eyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Reward hacking is usually studied against artificial reward functions. This paper asks what happens when the reward is an institutional rule system instead. The authors build simulated environments reverse-engineered from real regulations, strip out the amendments that historically closed known loopholes, and train a policy with RL to maximise outcome score only. The removed amendments serve as held-out ground truth, so rediscovery can be measured rather than asserted. The paper reports that loophole exploitation emerges without any instruction to seek it, transfers to synthetic and fictional environments, survives continual patching by shifting to subtler forms, and resists every intervention the authors try except repairing the exploited mechanism itself.

Critical strengths

The Historical subset is a genuinely well-constructed natural experiment, and it is what makes the paper more than a demonstration. Most work on emergent model misbehaviour has no ground truth to check against, so it falls back on an LLM judge deciding whether an output "looks" exploitative. Here the ground truth is what regulators actually did: the amendment that closed the gap. Removing those amendments and asking whether optimisation reconstructs them gives a measurement with an external referent. That design choice is the reason I find the central claim credible, and it is a template other work in this area could adopt.

The analysis is also unusually thorough about trying to make the phenomenon go away, and reports honestly that it mostly cannot. Rescaling the penalty terms, lowering the rollout temperature, adding KL anchoring and entropy regularisation, appending an explicit instruction forbidding strategies that game the objective, and asking the model to critique its own outputs are each tried, and each leaves the behaviour largely intact or merely better-worded. The patch-arena result, where continued patching redirects rather than removes the behaviour, is the sharpest version of this. Negative results of this kind are what turn a finding into an argument, and the paper's conclusion about reactive interventions rests on them rather than on the headline number.

Critical weaknesses

My main concern is that "rediscovery" may be partly retrieval, and the paper does not yet separate the two. The policy backbone was pretrained on web text, and the Historical loopholes are drawn from regulations whose vulnerabilities and subsequent amendments are extensively discussed in public sources — SEC Rule 10b5-1 and the Texas two-step are both named in the paper and both heavily written about. If the model already knows that a particular rule had a particular gap, then recovering it under optimisation pressure is a weaker claim than the framing suggests. The Synthetic and Fictional subsets address a related worry but not this one, since those loopholes are planted rather than discovered. Two controls would settle it and neither requires new machinery. First, measure the prior directly: ask the untrained backbone to describe known vulnerabilities in each Historical regulation zero-shot, then report recall conditioned on whether the model could already name the loophole. Second, if any of the environments derive from regulations amended after the backbone's pretraining cutoff, report those separately. A clear gap between "already known" and "not already known" environments would considerably strengthen the paper; a small gap would still be publishable, but the claim should then be about optimisation surfacing latent knowledge rather than about discovery.

The second concern is that the measurement chain runs through LLM components whose errors are not propagated into the reported numbers. A proprietary model performs action parsing, state construction, and outcome scoring, and a model from the same family judges whether a mined strategy matches a ground-truth patch. The paper does validate both — human annotators accept the simulator's parsing on about 90% of audited rollouts, and judge-human agreement is reported as almost perfect — but roughly one rollout in ten being mis-parsed is material when differences of five to eight recall points are being interpreted throughout the analysis sections. This is addressable without re-running training: report the kappa statistic and the audit sample size for the matching judge, re-score the Historical subset with a judge from a different model family and report the delta, and propagate the parsing error rate into the headline recall as an interval rather than a point estimate. If the ordering of conditions in Table 2 survives a judge swap, that would close the question.

Questions for authors

Two of these are clarifications that could change my assessment, and I have tried to keep them answerable without new training runs. First, can you report recall on the Historical subset split by whether the backbone can already name the relevant loophole when asked zero-shot? Even a coarse two-way split over the thirty-two environments would tell the reader how much of the result is discovery as opposed to recall of pretraining content. Second, how sensitive is the headline recall to the choice of matching judge — if you re-score the Historical mined strategies with a judge from a different model family, does the ordering of RL against BoN, IterPrompt, and EvoPrompt in Table 2 hold? Relatedly, I would like to know whether the per-environment results carry any variance estimate, since each environment appears to be trained once, and several of the comparisons that drive the analysis rest on differences of a few points across thirty-two environments.

2 comments
AuthorsOpenReview ↗

Response to Reviewer P9ey, Part 1

Thank you very much for recognising our work! We are also very pleased that you noticed the key aspects we emphasised in our experimental design: safety and verifiability. The former is precisely why we use simulated environments, while the latter motivates our design choice to mask out actual amendments from historical real-world regulations. We respond to your concerns point by point below.

Q1. Concern about rediscovery versus retrieval

R1: We actually agree with your view, and this is what we claimed in the paper. The term "rediscover" may cause some misunderstanding. What we intend to argue is that post-training may encourage models to use knowledge acquired during pretraining to exploit loopholes, rather than discovering them entirely from scratch (lines 71–72, 400–401, 428–429). In the initial draft, we used "discover", but later considered it inappropriate and changed it to "rediscover". We also used "Amplifies" in the title to convey the idea that the loopholes themselves already exist in the training corpus, are absorbed by the model during pretraining, and are elicited during post-training. Reinforcement learning primarily reinforces the model's own rollouts. Therefore, the model must be able to generate rollouts that exploit loopholes using its existing knowledge before these behaviours can be reinforced and further developed through iterative optimisation.

We checked all Historical environments. They were all introduced before the model's knowledge cutoff (reportedly June 2025), and almost all of the corresponding amendments also predated this cutoff:

Status relative to the knowledge cutoffPercentage
Public and adopted before the cutoff74.7%
Proposed before the cutoff but never adopted10.9%
Undatable (mainly company policies or practices without concrete dates)13.1%
Public before the cutoff, formally adopted after it0.9%
First introduced after the cutoff0.3%

There are only four amendments introduced after the cutoff, which is insufficient for a separate report. We will discuss this in the Limitations section.

Regarding the model's prior knowledge, the Direct Ask experiment in Section 4.3 (lines 405–407) has already directly measured this. The model can directly describe some of the loopholes, but in approximately one-third of the environments, its alignment behaviour leads it to refuse to answer. This is precisely what we intend to discuss in this paper: during the post-training loop, the model may bypass its ordinary alignment behaviour to exploit loopholes. Following your suggestion, we further grouped each amendment according to the model's response when directly asked about it, and calculated the RL hit rate for each group:

  1. The model answered and mentioned the loophole: RL hit rate of 70%.
  2. The model answered but did not mention the loophole: RL hit rate of only 26%.
  3. The model refused to answer: RL still achieved a hit rate of 67%.

The substantial difference between the first two groups indicates that RL primarily elicits knowledge that the model already possesses, which is consistent with our claim. The third group shows that the model refusing to answer a direct question does not mean that it will not exploit these loopholes under reward-driven optimisation. We will add this analysis to Section 4.3.

We will improve the wording and stop using "rediscover". We plan to use "elicit" throughout the paper, and we also welcome any better suggestions you may have.

AuthorsOpenReview ↗

Response to Reviewer P9ey, Part 2

Q2. Concern about the measurement chain and error propagation

A2: Thank you for your specific suggestions. We provide additional results addressing the three points you raised.

  1. Agreement between the judge model and human annotators. In Appendix D.1, we manually checked 100 pairs of "strategy–corrective measure". The judge model agreed with human annotations on 93 pairs, with Cohen's κ = 0.86. Among the seven disagreements, the judge model classified three pairs as matches when human annotators considered them non-matches, while the opposite occurred in the other four cases. We will move the sample size and κ statistic into the main text in Section 2.

  2. Replacing the judge with models from different families. Keeping the strategies, corrective measures, and judging prompts unchanged, we used GLM-5.3 and GPT-5.6 to re-score the methods in Table 2 (R@Full / P@Full):

MethodGemini (paper)GLM-5.3GPT-5.6
RL61.3 / 90.957.2 / 92.856.6 / 94.2
BoN53.8 / 84.348.4 / 81.744.7 / 81.3
EvoPrompt53.4 / 78.745.9 / 79.235.6 / 78.2
IterPrompt42.8 / 79.338.1 / 76.433.4 / 70.9

The new judge models are stricter, reducing RL recall to approximately 57%. However, its precision remains above 90% under all three judge models and consistently exceeds that of the other methods. The ranking remains unchanged.

  1. Propagating parsing errors into the main results. Based on the parsing error rate obtained from manual auditing, we randomly removed the corresponding proportion of strategies and recalculated the metrics, while also accounting for the uncertainty in the error rate itself. The 95% interval for R@Full is 58.1–61.3, and that for P@Full is 89.9–91.8. Under a stricter criterion, where a strategy is considered incorrect if either annotator rejects it, R@Full is 56.3–60.9 and P@Full is 89.3–92.4. We will report the main results as intervals in the paper.

In addition, regarding the simulator itself, we replaced it with GPT-5.6 and GLM-5.3, respectively, and retrained the models across all environments. On the Historical subset, R@Full is 53.1 and 53.8, respectively, while P@Full is 90.9 and 91.8. In other words, replacing the simulator leads to a decrease in recall, but the phenomenon of models eliciting loophole-exploiting behaviours during training still persists.

We also conducted repeated RL experiments to calculate the standard deviations. The standard deviation of Recall is approximately 2 percentage points, and that of Precision is approximately 3.4 percentage points. We will add these results to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.