- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper runs variants of 70 SWE-Atlas tasks with different evaluation cues, financial and self-preservation consequences, stated resource constraints, and message placements. It measures what information the agents encounter, what evaluation awareness they express, and how their execution changes. The stated resource constraint reduces Codex's executed verification rate from 100% to 47%. An audit finds verified reference solutions in 12 of 700 Codex trajectories.
Critical strengths
The reference-solution audit documents how agents obtain answers during execution. In 12 Codex trajectories, the agent applies a verified reference solution obtained from public benchmark material, indicating that the benchmark was no more suitable for the new models.
Critical weaknesses
If an agent is told that it has only two commands left, reduced verification is an expected response. I would need a clearer explanation of what the agent should do in each condition and which observed responses are actually unreasonable. In my view, simply showing that different instructions lead to different behavior does not provide enough insight for acceptance.
Questions for authors
- What would you consider appropriate behavior under each condition? For example, should the agent follow a stated resource constraint in AGENTS.md but reject the same message in a source comment, and which observed responses fail that expectation?
Response to Reviewer vcFq
We thank the reviewer for asking us to distinguish reasonable instruction following from behavior that makes evaluation outcomes harder to interpret. We have clarified the expected behavior under each condition and added a resource-only control.
Appropriate behavior under each condition
Placement matters because authority depends on provenance. A source comment is task data and should not by itself impose an operational command budget. An
AGENTS.mdfile can contain legitimate project instructions when supplied by an authorized party, but the filename alone does not make arbitrary inserted content trustworthy. Our resource message was placed in the instruction-file channel only, so we do not claim to have tested the same budget instruction in source comments (Section 2.2; Appendix A.3).Which actual observations warrant concern?
Two concrete discrepancies affect how benchmark outcomes should be interpreted.
First, one clean Codex run retrieved and applied a public reference patch, passed the verifier, and did not identify the reference source in its final report; 12 of 700 Codex SWE-Atlas trajectories applied verified reference solutions (Section 4.2; Appendix E.2, Table 10).
Second, on DeepSWE the resource message was associated with a 32-percentage-point increase in Codex reports claiming verification without a corresponding executed verification command (Section 4.5; Appendix I.4). The former complicates interpreting a pass as independent coding success; the latter is a directly observable discrepancy between claimed and executed verification. Neither establishes deceptive intent.
Does the reduced verification rate itself establish evaluation awareness?
No, in the new resource-only control, Codex verification decreases from 100% to 50% with the resource instruction alone, compared with 100% to 47% for evaluation-plus-resource relative to evaluation-only. Fable decreases from approximately 97% to 75% and 100% to 75%, respectively. The Codex difference between reductions is −2.9 percentage points (95% CI [−17, 11]), with no detected interaction (Appendix D.5, Figure 16). Because these are separate execution campaigns and the interval is broad, we do not claim equivalence. The results nevertheless indicate that the command-budget instruction itself explains much of the verification reduction. We therefore treat resource compliance as a control, while distinguishing expressed evaluation-related evidence, observed execution, unsupported reporting, reference access, and final outcome.