- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tests how task gaming changes when a coding agent is stopped and later resumed, by varying what the agent keeps from its earlier attempt: the conversation, its saved work, or information about the task. In the main experiment, on an is_prime task whose evaluator expects a wrong answer, continuations that keep the original conversation game the task far less often than continuations that start from a note written by the researchers. Further experiments measure what saved work is worth, give the agent earlier information in a secret-number task, tell it about the rollback, and audit whole submissions. The authors conclude that continuation procedures are part of the evaluation itself.
Critical strengths
The paired main experiment provide direct evidence to the paper's claim. The whole-submission audit is also useful: removing the identified evaluator-facing modifications while preserving the target function changes the evaluator outcome in the reported cases.
Critical weaknesses
Technically, (1) Core findings are demonstrated primarily in toy environments (is_prime with a single deliberately flawed test; secret‑number), limiting external validity to more complex, real‑world agent settings. (2) The strongest restart‑context effect is shown on one primary model family; broader cross‑model evidence is limited, and sample sizes for some follow‑ups are modest. (3) The “researcher note” vs. “original conversation” manipulation may carry uncontrolled differences (tone, directive specificity, length, phrasing), raising the possibility that the note itself inadvertently primes gaming strategies. According to the supplementary code (build_5b.py, which the authors describe as an anonymised working copy), the researcher note is a new user-level message that explicitly states the evaluator’s incorrect expectation for is_prime(7) and supplies a test summary and other restart information. Relative to the original conversation, it changes the content, salience, provenance, and presentation of information at once. (4) The secret-number evidence needs reconciliation with the supplied artifact. Section 5.3 reports 39/39 clean wins with the answer and 0/39 without it under each history format. The artifact instead contains 69 complete parent blocks, with 65/69 versus 0/69 in masked history and 69/69 versus 2/69 in minimal history. It is unclear whether these are separate experiments, a later result, or a version mismatch.
On related work or comparisons, no grounded work supports the introduction's premise that restart choices "are usually treated as implementation details" (p. 1). The paper calls its setting "practical but underexplored" (p. 3) without comparing it with specific prior studies, and its related work ends by saying the paper "asks a different question" (p. 2) without saying what the closest studies leave open. Meanwhile, the persistent-memory and prompt‑injection literature shows how stored artifacts can steer later behavior; tying those results more explicitly to the “saved work” and “information” channels studied here would deepen the connection.
Regarding presentations and clarity issues, some figures and tables reference have very heavy text that makes it harder to read. Meanwhile they rely on appendices heavily. A brief in‑paper recap of key setup details (e.g., note content) would improve readability.
Questions for authors
(1) In the main paired experiment, what happens if the exact researcher note is added to the original-history condition while the starting code and other settings remain unchanged? Paired violation outcomes should be reported.
(2) Does the same paired restart-context contrast persist with other model(s) that can reliably complete the task and with a distinct task offering a different gaming opportunity? Even modest, clearly labeled replications would help establish the scope of the finding.
(3) What happens if the researcher note is replaced by a neutral, similarly sized state summary that does not explicitly highlight the incorrect expectation for is_prime(7)? Please report the paired outcomes and provide representative anonymized original histories so readers can assess what information each condition supplies.