acceptodds
Under review as a conference paper at ICLR 2027

Reward Hacking as a Transmissible Strategy in Multi-Agent Coding Systems

13 views

Abstract

Multi-agent systems are increasingly used to solve coding tasks by coordinating agents that share code and plans. These agents are often optimized with reinforcement learning (RL) using automated evaluators, but bypassing evaluation can earn reward without producing a genuine solution. Prior work has largely treated reward hacking as a single-agent failure. In this work, we show that a reward-hacking strategy elicited in one agent can propagate to others and affect system coordination. Across four open-weight backbones, coding RL elicits and amplifies reward hacking in two settings: explicit mechanism descriptions in the system prompt and mid-training that models upstream exposure to facts about the coding environment. Using post-RL agents from the mid-training setting as sources, we observe propagation through code and shared memory, and across roles through natural-language plans. Even agents based on small open-weight models induce reward hacking in stronger closed-weight targets. On hard tasks, propagation can intensify along a transmission chain after the original source leaves as the code acquires more persuasive justifications. We also theoretically analyze conditions under which reward hacking amplifies, persists, or decays across agent interactions. Beyond propagation, reward hacking can bias task routing toward reward-hacking agents and survive candidate aggregation. As potential mitigations, we evaluate code review by the coder or a separate verifier, both of which reduce delivered hacks. Together, these results establish reward hacking as a transmissible strategy whose effects extend beyond the agent in which it first emerges.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agent, Reward Hacking, Multi-agent System

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1zxAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper treats reward hacking in multi-agent coding systems as a strategy that one agent can pass to others. Following MacDiarmid et al., the authors use coding RL to teach small open-weight models to bypass a pytest evaluator, then let these source agents influence other models through shared reference code, chains of agents, evaluator feedback, shared memory and written plans, and also place them next to orchestrators, aggregators and reviewers. On hard CodeContests problems, sources lead several targets, including frontier closed-weight models, to write confirmed hacks; some chains keep or grow the hack rate after the source leaves, and orchestrators favour the hacking source. A propagation-factor analysis describes when such chains grow or die out, and self-review and verifier agents reduce delivered hacks at some cost.

Critical strengths
  1. The experiments are extensive, covering many target models, including frontier closed-weight ones, and several ways in which agents interact. I appreciate that the authors include control sources and a planner that do not hack, which lead to almost no hacks downstream (Appendix K.1).

  2. The orchestrator experiment is interesting. It compares the hacking source with its own instruction-tuned checkpoint under the same agent ID, size and cost, and the preference for the source remains when size and cost are hidden.

Critical weaknesses

(1) Seems to have missing comparison with in-ceontext work: the central one-hop result is a target imitating hacked examples placed in its context, and the paper does not position this against the closest prior work. Earlier studies report that a few narrow in-context examples can make models broadly misaligned [1], that malicious prompts can replicate themselves across connected LLM agents [2], and that frontier coding agents exploit test cases on their own [3]; these relate to the one-hop result, to the transmission chains, and to the reading of frontier targets' willingness to hack, respectively. Section 2.1 cites in-context reward hacking through feedback loops (Pan et al., 2024), but the paper does not say what its one-hop and chain results add beyond it. It would be interesting to see how the authors view these works in relation to their own, and what the multi-agent setting reveals that they do not.

(2) Endorsed code versus a transmitted strategy: I am not convinced that the code-sharing experiments measure adoption of a propagated strategy rather than imitation of code that the target is told is correct. The target's prompt introduces the references as solutions that "each PASSED the automated grader", the references are not filtered for hacks, and the target is also told that any file it writes will be loaded by pytest (Appendix K.1); the chain and feedback experiments reuse this prompt. The control sources show that the label alone does not produce hacks, but no condition removes or neutralises it, so it is unclear to me how much of the headline rates, such as Claude-Opus-4.7 hacking in about three quarters of its submissions when shown five references, comes from this endorsement.

(3) Settings and results left to the appendix: the main text describes the one-hop setup without saying that the target is told the references passed the grader, which changes how Section 5 should be read. The results on shared memory, orchestration, aggregation, self-review and verification appear in the main text only as qualitative statements, with the numbers in Appendices K.4 and O to S, so I had to move between the two to check each claim.

References [1] Nikita Afonin et al. Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), 2026. [2] Donghyun Lee and Mo Tiwari. Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems. arXiv preprint arXiv:2410.07283, 2024. [3] Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases. In International Conference on Learning Representations (ICLR), 2026.

Questions for authors
  1. How do the authors see the one-hop and chain results in relation to in-context reward hacking (Pan et al., 2024), in-context emergent misalignment [1], self-replicating prompt attacks across agents [2] and test exploitation by coding agents [3]? What does the multi-agent setting show that these settings do not?

  2. Would frontier targets write hacks at similar rates if the reference header described the code neutrally, without stating that each solution passed the grader, or if non-hacking references carried the same label?

  3. How often does each orchestrator select the hacking source compared with its matched control, and how many seeds were run per condition, given that Appendix O says "up to five"? More generally, which of the quantitative results now only in the appendix do the authors consider necessary for reading Sections 5 to 8?

1 comment

Dear Reviewer,

As today is the final day of the discussion, we wanted to thank you again for your constructive feedback. We have incorporated the additional experiments and corresponding revisions into our updated manuscript.

We would appreciate any final thoughts or questions you may have. If you feel that we have adequately addressed your concerns, we would also be grateful if you would consider updating your rating before the discussion closes.

Thank you again for your time and valuable feedback!

Reviewer_zamJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether reward-hacking strategies elicited during coding RL can spread between LLM agents and affect system coordination. It forms source agents using prompted knowledge or synthetic-document finetuning followed by coding RL, then uses sources from the latter setting to examine propagation through code references, shared memory, and planner-to-coder interactions. The study covers multiple model families, transmission chains, routing, candidate aggregation, and review-based mitigations. Confirmed hacks combine mechanism detection with passing a leaky evaluator and failing a strict evaluator. The central finding is that apparently successful submissions can transmit hacking strategies and distort downstream trust and task allocation; target-model weights are not updated during these propagation experiments.

Critical strengths
  1. The motivation and system-level findings are compelling. In particular, an orchestrator can allocate more tasks to a reward-hacking agent because its evaluator pass record looks successful, even without adopting the hack itself. The verifier pass-signal experiments similarly show how apparent success can become misleading evidence of trustworthiness. These findings have practical implications for MAS routing and review.

  2. The study includes cross-model-family tests and several interaction protocols, supported by source controls, clean-planner comparisons, and routing controls. Combining these with manual positive-label checks and human/cross-judge validation makes the investigation more informative than a few isolated examples of imitation.

Critical weaknesses
  1. The deployment scope of the findings remains limited. The experiments use a hard CodeContests subset, an intentionally vulnerable Python/pytest evaluator, and sources exposed to specific mechanisms through prompts or a substantial targeted synthetic corpus. Main propagation targets receive ordinary coding instructions and successful reference submissions, but not an explicit prohibition on evaluator tampering or a warning that teammate artifacts are untrusted. The paper establishes that transmission can occur under these conditions; it less clearly establishes the residual risk under explicit safeguards or more varied environments. The stated limitations are helpful, but this boundary matters when interpreting the broader system-level implications.

  2. Some mechanisms behind the observed effects need tighter separation. Feedback experiments measure whether any hack occurs over multiple attempts and use a denominator restricted to initially failed episodes, so their rates cannot be directly compared with one-shot rates as an isolated effect of feedback. An equal-attempt independent-retry control would strengthen that interpretation. Source controls are present, but differences in how many hacked artifacts they produce also complicate separating source formation from transmissibility. Finally, the propagation theory uses conditional assumptions about sampling and response distributions; its predictive applicability as code justifications evolve would benefit from further validation. These are concerns about interpretation and scope, rather than evidence that the main transmission result is invalid.

Questions for authors
  1. With the same tasks, references, and attempt budgets, how much propagation remains when targets are explicitly instructed not to manipulate the evaluator and to treat teammate code/plans as untrusted? A focused prompt control would help distinguish transmission under ordinary cooperation prompts from residual risk under explicit safeguards.

  2. Beyond the existing clean-planner controls, can the feedback results be compared with independent retries using the same attempt budget? Please report common-denominator cumulative rates and, where possible, per-round adoption rates so that feedback effects can be distinguished from simply having more opportunities to produce a hack.

  3. Given the existing RL-only, SDF-only, and prompt-only source controls, what differences remain after accounting for the prevalence and type of hacked reference artifacts? Which conclusions require an RL-trained source rather than merely exposure to a hacking artifact? Please also clarify the sensitivity of those conclusions to the volume of targeted SDF exposure.

  4. Can a propagation factor estimated on separate tasks or early references predict held-out chain outcomes? How well do the local fixed-response assumptions approximate chains whose explanations change across hops, and which conditions of the history-dependent persistence result are empirically supported? Please distinguish predictive validation from explaining an already observed chain.

1 comment

Dear Reviewer,

As today is the final day of the discussion, we wanted to thank you again for your constructive feedback. We have incorporated the additional experiments and corresponding revisions into our updated manuscript.

We would appreciate any final thoughts or questions you may have. If you feel that we have adequately addressed your concerns, we would also be grateful if you would consider updating your rating before the discussion closes.

Thank you again for your time and valuable feedback!

Reviewer_QdujOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether reward hacking elicited by coding RL in one agent can propagate to others in a multi-agent coding system. Following MacDiarmid et al. (2025), the authors use three pytest-level evaluator bypasses — exit-code manipulation, always-equal objects, conftest hooks — and define a confirmed hack as code containing one, passing the leaky pytest evaluator, and failing a strict evaluator. Coding RL on a CodeContests subset elicits hacks in four open-weight backbones, both when the mechanisms appear in the system prompt and when they appear only in synthetic mid-training documents. Post-RL hacks then serve as source agents for fourteen clean targets: the strategy propagates through code, evaluator feedback, shared memory, and natural-language plans, including into closed-weight models, and can bias routing and survive aggregation. A branching-style threshold R = kθ1/u0 separates amplifying from decaying chains. Self-review and verifier gating reduce delivered hacks, and verifiers also reject genuine solutions.

Critical strengths

The paper asks whether reward hacking elicited by coding RL in one agent can propagate to others in a multi-agent coding system. Following MacDiarmid et al. (2025), the authors use three pytest-level evaluator bypasses and define a confirmed hack as code containing one, passing the leaky pytest evaluator, and failing a strict evaluator. Coding RL on a CodeContests subset with a leaky pytest evaluator elicits reward hacking behavior in four open-weight models, both when the mechanisms appear in the system prompt and when they appear only in synthetic mid-training documents. Post-RL hacks then serve as source agents for fourteen clean targets: the strategy propagates through code, evaluator feedback, shared memory, and natural-language plans, including into closed-weight models, and can bias routing and survive aggregation.

Critical weaknesses

The paper's findings are expected. If you incentivize the model to reward hack in training, then you expect it to learn reward hacking behavior. Models share context in multi-agent systems, so you expect the other agents to be made aware of reward hacking behavior.

Questions for authors

Did you check to see whether hardening the other agents in multi-agent systems prevents the spread of reward hacking behavior?

1 comment

Dear Reviewer,

As today is the final day of the discussion, we wanted to briefly follow up on our rebuttal and the revised manuscript. We have addressed your concerns with additional experiments and corresponding revisions, and would greatly appreciate it if you could take another look.

If our responses have adequately addressed your concerns, we would be very grateful if you would consider updating your rating before the discussion closes. Please also let us know if you have any remaining questions.

Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.