- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how the fraction of flawed verifiers affects training outcomes in synthetic agentic RL environments. The paper first audits 10,000 AWM tasks and finds that 16.8% pass verification even when the agent does nothing. They then introduce three types of over-permissive verification rules into the filtered training environments and compare training rewards, true success, and exploit-related behavior across different flaw rates. Reported reward increases with the flaw rate, while true success stays largely unchanged at low to moderate rates and drops substantially at 50% in the mixed-flaw condition. The behavior of setting the status flag transfers to held-out AWM scenarios that share the same status tool, but TextArena shows no clear cross-domain effect. The paper also examine zero-action probing and reward telemetry as methods for detecting flawed environments.
Critical strengths
-
The paper addresses a practical concern. Synthetic environments rely on automatically generated verifiers, so the effects of verifier errors on training are worth measuring systematically. Keeping the original verifiers while injecting controlled flaws provides a useful way to distinguish genuine task completion from incorrectly rewarded attempts.
-
The experiments distinguish effects on true capability from effects on behavior. Even when true success does not noticeably decline, the agent may increasingly declare completion without actually completing the task. This highlights the limitations of monitoring only training reward or task success rates.
Critical weaknesses
-
For the original 1,000 AWM scenarios, the paper only checks two cases: tasks pass without any action and through self-report. It does not check for other flaws or whether the original verifiers can be trusted as ground truth. Have the authors tried other ways to check or filter these verifiers? The coverage of the three injected flaws is narrow: they are all over-permissive and reward attempts that do not complete the task (without showing that these flaws represent natural flaws in AWM). Other types of errors are not tested (e.g., rejecting correctly completed tasks or accepting structurally valid but semantically incorrect outputs). The comparison also does not separate flaw type from how easy it is to exploit (e.g., whether it needs an extra tool call or how it relates to the task itself). And the effects of different flaw types and flaw rates on true capability are not compared in enough detail.
-
The paper’s contribution is still unclear. Most findings in Section 5 are as expected: more flawed verifiers lead to lower true capability and inflated reported reward. These findings are not particularly surprising. The claim that “exploit dispositions transfer to unseen environments” also seems too strong. In TextArena, increasing the fraction of flawed training environments did not noticeably increase how often the model read the hidden solution file. Based on these results, “generalization of behavioral habits through a shared tool” seems more accurate than “transfer of exploit dispositions.” The paper also only tests Qwen3 models and does not evaluate other model.
-
Section 5 discusses two detection methods. Zero-action probing works well for two of the flaw types the authors designed, but this does not show that it can detect a broad range of natural flaws. Reward telemetry was also not used to guide retraining, so it is unclear whether it actually helps improve training outcomes. The effectiveness of these methods in other real-world settings is also unclear.
Questions for authors
-
Beyond the initial-state and success-claim probes, were the original verifiers retained after filtering independently checked (e.g., through manual inspection or tests with correct and incorrect trajectories)? Do these checks support using the verifiers as ground truth for task completion?
-
Have the authors compared different flaw types/trigger rates/extra action costs or other factors?
-
In Section 5.7, how were the scenarios flagged by the first detection method handled? Were they removed, or were their verifiers fixed? Was the second detection method actually used to guide mitigation and retraining?