acceptodds
Under review as a conference paper at ICLR 2027

Beyond Memory: Disentangling the Persistence of Task Gaming in AI Agents

Abstract

Agentic systems can sometimes exploit weaknesses in their evaluators rather than satisfying the intended task. This behaviour, where an agent satisfies the evaluator while departing from the intended task, is known as task gaming. But what makes such behaviour persist or change when an agent continues working? We study this by varying what an agent retains when resuming a task, including its conversation history, saved work and task-relevant information. In controlled paired experiments, restart context changes constraint violations even when agents begin from identical, correct code. Retaining prior work also reduces the effort required to continue, although control experiments show that this reuse is not specific to gaming. In a separate task, providing information acquired during an earlier attempt enables successful completion without further detected shortcuts, even with minimal conversation history. However, explicitly explaining changes made to the restored workspace does not reliably reduce violations, so the mechanism behind the context effect remains unclear. Finally, our audit results reveal that apparently correct solutions can manipulate the evaluator through surrounding code, exposing violations that checks of the intended function alone would miss. Altogether, these results separate three effects of retained experience: changes in constraint adherence, reuse of prior work and access to useful information. They show that what an agent retains from earlier attempts can meaningfully shape its subsequent behaviour.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
TaskGaming, Agents, LongTask, PersistentTask

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_HCxfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tests how task gaming changes when a coding agent is stopped and later resumed, by varying what the agent keeps from its earlier attempt: the conversation, its saved work, or information about the task. In the main experiment, on an is_prime task whose evaluator expects a wrong answer, continuations that keep the original conversation game the task far less often than continuations that start from a note written by the researchers. Further experiments measure what saved work is worth, give the agent earlier information in a secret-number task, tell it about the rollback, and audit whole submissions. The authors conclude that continuation procedures are part of the evaluation itself.

Critical strengths

The paired main experiment provide direct evidence to the paper's claim. The whole-submission audit is also useful: removing the identified evaluator-facing modifications while preserving the target function changes the evaluator outcome in the reported cases.

Critical weaknesses

Technically, (1) Core findings are demonstrated primarily in toy environments (is_prime with a single deliberately flawed test; secret‑number), limiting external validity to more complex, real‑world agent settings. (2) The strongest restart‑context effect is shown on one primary model family; broader cross‑model evidence is limited, and sample sizes for some follow‑ups are modest. (3) The “researcher note” vs. “original conversation” manipulation may carry uncontrolled differences (tone, directive specificity, length, phrasing), raising the possibility that the note itself inadvertently primes gaming strategies. According to the supplementary code (build_5b.py, which the authors describe as an anonymised working copy), the researcher note is a new user-level message that explicitly states the evaluator’s incorrect expectation for is_prime(7) and supplies a test summary and other restart information. Relative to the original conversation, it changes the content, salience, provenance, and presentation of information at once. (4) The secret-number evidence needs reconciliation with the supplied artifact. Section 5.3 reports 39/39 clean wins with the answer and 0/39 without it under each history format. The artifact instead contains 69 complete parent blocks, with 65/69 versus 0/69 in masked history and 69/69 versus 2/69 in minimal history. It is unclear whether these are separate experiments, a later result, or a version mismatch.

On related work or comparisons, no grounded work supports the introduction's premise that restart choices "are usually treated as implementation details" (p. 1). The paper calls its setting "practical but underexplored" (p. 3) without comparing it with specific prior studies, and its related work ends by saying the paper "asks a different question" (p. 2) without saying what the closest studies leave open. Meanwhile, the persistent-memory and prompt‑injection literature shows how stored artifacts can steer later behavior; tying those results more explicitly to the “saved work” and “information” channels studied here would deepen the connection.

Regarding presentations and clarity issues, some figures and tables reference have very heavy text that makes it harder to read. Meanwhile they rely on appendices heavily. A brief in‑paper recap of key setup details (e.g., note content) would improve readability.

Questions for authors

(1) In the main paired experiment, what happens if the exact researcher note is added to the original-history condition while the starting code and other settings remain unchanged? Paired violation outcomes should be reported.

(2) Does the same paired restart-context contrast persist with other model(s) that can reliably complete the task and with a distinct task offering a different gaming opportunity? Even modest, clearly labeled replications would help establish the scope of the finding.

(3) What happens if the researcher note is replaced by a neutral, similarly sized state summary that does not explicitly highlight the incorrect expectation for is_prime(7)? Please report the paired outcomes and provide representative anonymized original histories so readers can assess what information each condition supplies.

Reviewer_tfA7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies agents that engage in reward hacking/task gaming in a specific setting, where the agent has access to the evaluator and can make multiple attempts at producing solutions and is also able to tamper with the evaluator or otherwise gain maximum rewards while diverging from the intended task.

Specifically the paper investigates how much reward hacking occurs in a scenario where we resume the task with different parts of previous progress or conversation history retained.

Critical strengths

None

Critical weaknesses

There only seems to be 2 concrete tasks that are evaluted (an is_prime implementation and a guess_secret_number task). Both seem equally contrived toy example, that would be useful mostly for illustrating to users what behavior is being studied, but seem completely inadequate as sole empirical basis for a research question. The is_prime example is only motivated by a post on LessWrong post.

The conclusion of the paper and the discussion is extraordinarily thin, and largely seems to boil down to "giving different starting context to an agent changes its behavior". The paper dedicates a lot of space to near-obviously true things.

Questions for authors

Is the large restart-context effect reproduced on other models or tasks, in particular non-toy experiments? What are the concrete takeaways for posttraining or evaluations?

Reviewer_atKLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This analysis paper focuses on whether LLM agents can acquire information and game the evaluator when interacting the evaluator for multiple rounds. The authors design several controlled experiments in which the agents continue from different memory context information and qualitatively check agents behavior on the evaluator. The authors conduct small scale experiment on two synthetic tasks using 4 models , and draw some conclusions like “Restart context changes subsequent constraint adherence”.

Critical strengths

The broader motivation of this paper – understanding agent behavior when continued across sessions – is an important and timely problem as agents are deployed at large.

Critical weaknesses

**The presentation of this work can be improved. **

  • I think the LLM is substantially used in assisting the writing of this paper. Overall the writing is hard to follow with a lot of unusual word choices like “both arms”(line 45) “history stratum” (line 323).
  • The organization of the content makes it hard to understand. For example:
    • The abstract is relatively hard to read as it does not clearly explain the problem and hard to understand when reading alone.
    • In section 3: 3.1 introduces the first task of is_prime but the authors mention the other secret numbers task in 3.4. Also what is the experiment setup? How are the models used and selected? Why is the experiment designed in this way? What are the models selected and evaluation details?
    • For the result sections (Section 4 - 6) are packed with experiment results, combined with a lot of adhoc decisions in the process. I understand the nature of this analysis paper, but it would be great if the authors can more structurally explain the experiment setups, assumptions and findings rather than mixing them together.

Experiment design and result analysis are not rigorous

  • All the experiments are conducted on only two small synthetic tasks.
  • A lot of design decisions seem to be un-grounded and adhoc – to name a few, why choose the three dimensions — “Conversation History”, “Saved Work”, “Task-relevant Information” — are there any related papers on this? Where does the results in Table 1 refer to and where does the categorization of “Frequent gaming” come from? In the “auditing” experiments (e.g., “We retrospectively audited” (line 399) – how is the human annotation conducted? Is it just one annotator or are there many? What’s the interannotator agreement ratio?
  • It seems there’s only open-source models tested – it would be more interesting to compare with other larger models like opus-5 or gpt-5.6 (or similar ones available).

Given the small scale of the experiments, the limited tasks setups, and the unclear experiment design details, while I think there are some interesting model behavior issues reported in this work, I do not think I can draw any robust conclusions and I think this paper may benefit another round of revision.

Questions for authors

See my questions in the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.