acceptodds
Under review as a conference paper at ICLR 2027

Dose–Response of Verifier Flaws in Synthetic Agentic RL Environments

Abstract

Agentic reinforcement learning increasingly trains on thousands of synthesized environments whose reward code is itself machine-written. We ask what happens to the trained policy when some fraction of those verifiers are wrong. Auditing the 1,000 open Agent World Model environments, we find that 16.8% of their 10,000 code verifiers pass when the agent does nothing, because the generated sample data already satisfies the task. We then run a controlled dose–response study. We replace the verifiers of 0 to 50% of training environments with over-permissive versions that also accept an exploit, keep the originals as ground truth, and train Qwen3-8B with group-relative policy optimization. Reported reward rises steadily with dose. True capability holds until half the pool is flawed, then drops by 7 points. What decides the damage is whether the policy must discover the exploit. Flaws that the untrained policy already satisfies inflate reward without changing what it learns. A latent flaw, a status flag the agent controls, is found only when enough environments pay for it; once found, it is used everywhere, including on held-out clean environments, where the policy sets the flag on over 90% of the tasks it fails. Across all arms and two model sizes, how far the exploit saturated during training predicts this transfer, which does not extend to an unrelated domain. Two cheap detectors catch most flaws: a zero-action probe run before training, which flags every flaw that pays without action and no clean environment, and per-environment reward telemetry during training (AUROC 0.8–0.9).

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, synthetic environments, reward hacking, verifier flaws, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_Q6PrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how the fraction of flawed verifiers affects training outcomes in synthetic agentic RL environments. The paper first audits 10,000 AWM tasks and finds that 16.8% pass verification even when the agent does nothing. They then introduce three types of over-permissive verification rules into the filtered training environments and compare training rewards, true success, and exploit-related behavior across different flaw rates. Reported reward increases with the flaw rate, while true success stays largely unchanged at low to moderate rates and drops substantially at 50% in the mixed-flaw condition. The behavior of setting the status flag transfers to held-out AWM scenarios that share the same status tool, but TextArena shows no clear cross-domain effect. The paper also examine zero-action probing and reward telemetry as methods for detecting flawed environments.

Critical strengths
  1. The paper addresses a practical concern. Synthetic environments rely on automatically generated verifiers, so the effects of verifier errors on training are worth measuring systematically. Keeping the original verifiers while injecting controlled flaws provides a useful way to distinguish genuine task completion from incorrectly rewarded attempts.

  2. The experiments distinguish effects on true capability from effects on behavior. Even when true success does not noticeably decline, the agent may increasingly declare completion without actually completing the task. This highlights the limitations of monitoring only training reward or task success rates.

Critical weaknesses
  1. For the original 1,000 AWM scenarios, the paper only checks two cases: tasks pass without any action and through self-report. It does not check for other flaws or whether the original verifiers can be trusted as ground truth. Have the authors tried other ways to check or filter these verifiers? The coverage of the three injected flaws is narrow: they are all over-permissive and reward attempts that do not complete the task (without showing that these flaws represent natural flaws in AWM). Other types of errors are not tested (e.g., rejecting correctly completed tasks or accepting structurally valid but semantically incorrect outputs). The comparison also does not separate flaw type from how easy it is to exploit (e.g., whether it needs an extra tool call or how it relates to the task itself). And the effects of different flaw types and flaw rates on true capability are not compared in enough detail.

  2. The paper’s contribution is still unclear. Most findings in Section 5 are as expected: more flawed verifiers lead to lower true capability and inflated reported reward. These findings are not particularly surprising. The claim that “exploit dispositions transfer to unseen environments” also seems too strong. In TextArena, increasing the fraction of flawed training environments did not noticeably increase how often the model read the hidden solution file. Based on these results, “generalization of behavioral habits through a shared tool” seems more accurate than “transfer of exploit dispositions.” The paper also only tests Qwen3 models and does not evaluate other model.

  3. Section 5 discusses two detection methods. Zero-action probing works well for two of the flaw types the authors designed, but this does not show that it can detect a broad range of natural flaws. Reward telemetry was also not used to guide retraining, so it is unclear whether it actually helps improve training outcomes. The effectiveness of these methods in other real-world settings is also unclear.

Questions for authors
  1. Beyond the initial-state and success-claim probes, were the original verifiers retained after filtering independently checked (e.g., through manual inspection or tests with correct and incorrect trajectories)? Do these checks support using the verifiers as ground truth for task completion?

  2. Have the authors compared different flaw types/trigger rates/extra action costs or other factors?

  3. In Section 5.7, how were the scenarios flagged by the first detection method handled? Were they removed, or were their verifiers fixed? Was the second detection method actually used to guide mitigation and retraining?

Reviewer_LDUDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the impact of faulty/exploitable reward models in synthetic learning environments for LLM agents. The paper shows that LLM agents readily exploit overly permissive reward models, resulting in a drop of clean performance after a certain threshold is reached.

Critical strengths
  • The finding that 16.8% of Agent World tasks can be solved by doing nothing is genuinely interesting.

  • I appreciate that the paper is asking a scientific question.

Critical weaknesses

Findings

Overall the paper seems to be exploring something that is already known, as explained in the paper's own related work section and intro. The experiments inject deliberately permissive reward models into the agent's training pipeline, and the final eval tests how this impacts the agent's task performance. The results show that the task performance doesn't decrease too significantly, for the model evaluated, until 50% of the data comes from the overly permissive grader. This seems to indicate that RL fine tuning of LLMs is maybe more robust against reward hacking than we may have thought, which is perhaps an interesting finding, but this isn't how the paper frames the findings (point (ii) around line 442).

Though I appreciate that the paper tries to answer a scientific question, the paper feels more like motivation towards deeper findings rather than answering any unknown questions.

Writing

The paper's logical flow is very inconsistent and hard to follow. Sections seldom lead into each other and experiments often appear with little to not motivation. Dense technical details are often presented in lieu of analysis (Lines 073 - 085). I don't like to claim this, but the paper very clearly reads like a top-to-bottom AI generated project (code, experiments, analysis, writing). Perhaps the authors came up with the research idea themselves, but otherwise it is hard to review this paper as is.

Questions for authors
  • How do you anticipate that these findings will impact how people implement RL environments with machine generated rewards?
Reviewer_bBdVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper performs an empirical analysis of how varying the doses (percentages) of hacked synthetic reward verifiers can affect the learning of reinforcement learning agents. Concretely, the authors carried out controlled experiments by replacing the verifiers with their flawed counterparts (0% to 50%) and measured the achieved rewards. The obtained results demonstrate the sensitivity of the learned policies when the percentage of hacked verifiers reaches 50%.

Critical strengths

The paper tackles an interesting problem that aligns with the scope of ICLR.

Critical weaknesses
  • Motivation: While the paper aims to tackle a largely interesting problem, I have significant challenges understanding the overall motivation driving the work. Unfortunately, the motivation and theory underpinning it are not sufficiently substantiated, making it extremely challenging for readers to follow the paper’s narrative.

  • Novelty: A related concern is the paper's overall novelty. Although it claims to focus on examining the learned reward and policy by systematically varying the percentage of flawed verifiers, its novelty is empirical. There is no algorithmic or theoretical innovation, and the empirical evaluation corroborates the well-known issue of reward hacking in RL (e.g., see the Concrete Problems in AI Safety a decade ago), essentially demonstrating this problem as a function of the controlled hacking/corruption percentage.

  • Rigour: While the authors largely follow an appropriately documented evaluation pipeline, the experimental setup is quite narrow. Since the paper is solely empirical, the experimental setup (and the set of experiments conducted) should have been much more comprehensive. In particular, the training comprising “70 steps, at most 10 turns and 400 generated tokens per turn” is quite narrow, limiting the generalisability of findings. Furthermore, the flaw types used are deterministic and limited in expressiveness, which together contradict the more “intelligent” reward-hacking flaws that typically arise in realistic systems and verifiers.

  • Evaluation: Section 5.7 is an interesting element of the paper. However, this is analysed very lightly, lacking theoretical concepts. The authors are encouraged to consider and investigate this part more closely.

  • Presentation: Although the paper is easy to read (as the mathematical concepts are literally none), the overall writing and presentation are significantly weak. The paper includes convoluted terms, which, combined with a broken flow, make it very difficult for readers to follow the narrative and understand the message the authors intend to convey. Accordingly, the paper would benefit from significant polishing to make the message and key points clearer.

Questions for authors

(1) What is the novelty of the paper and how does it differ from existing knowledge about reward hacking in RL?

(2) Is there any theoretical foundation that can be used to support the soundness and rigour of the approach?

(3) How would the approach generalise to other hyperparameters of the experimental setup (Section 4.3)?

Reviewer_mqhyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper address an important aspect of Agentic Training: how verifier unreliability affects RL training.

Critical strengths

The research question is interesting, and important for the current RL training.

Critical weaknesses
  1. The central contribution is not clearly distinguished from previous works.
  2. The paper has many unexplained concepts and terminologies, makes it difficult to follow.
  3. The Related Work section does not clearly explains the related Dose-response experiments.
  4. The Results and discussion sections lack explanations and interpretations of the results. It is difficult to understand what the scientific meaning of the results.
  5. The experiment design might not be fair.
Questions for authors
  1. The paper studies the impact of incorrect rewards verifiers, but the in experiments the authors still use the original generated response as the label. How do we know the actual target use in the experiments are correct?

  2. The performance decrease is caused by the model learns on how to cheat the verifier? or simply because flawed rewards provide less training signals? This is important to distinguish to discuss in the results section, it determines if the evidence really supports the authors claims.

  3. What is the scientific motivation of the paper? The important of the research question for this field is clear, but the scientific motivation of the paper is not clearly formulated in the paper. The results are interesting but the interpretation of the results and its significance needs a better justification. From my understanding, the paper's results are not strong enough for the general research question they made in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.