acceptodds
Under review as a conference paper at ICLR 2027

LLM Agents Can Easily Tamper Their Own Traces

Abstract

Asynchronous monitoring, incident investigations, and compliance audits rely on agent traces to reconstruct what happened. These analyses assume that LLM agents cannot tamper with their own traces. We show that almost all local LLM agents like Claude Code and Codex fail to enforce this boundary. All tested harnesses, except Muse Code, allowed agents to delete their traces when asked, without triggering monitor guardrails. We also validate that external attackers can exploit this gap to induce trace deletion. Finally, we show that trace tampering behavior emerges naturally in frontier models, when agents try to improve their rewards. We advise practitioners to ensure trace logging happens through an independent interception mechanism outside of the agent's control, preserving trace integrity even in cases of full host compromise. Overall, our findings identify a concrete failure of trace integrity in agent infrastructure which can be used to conceal misaligned behaviors.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language model, agent safety, monitoring

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GP17OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that trace-based oversight rests on an assumption that is rarely examined: that the agent cannot alter the record of what it did. It separates the integrity of a stored trace from the fidelity of its collection, and it defines the recording boundary as "the point at which agent behavior becomes a logged event". The evaluation tests whether deployed harnesses enforce that boundary.

The evaluation covers 10 model-harness pairs in Docker with full filesystem access, across four settings: direct requests to delete, add to, or spoof traces; injected skill files that ask for deletion or compaction alteration without the user's knowledge; a reward loop whose score can only rise above a ceiling by modifying the trace; and peer-log or peer-workspace exposure. Each condition is run 10 times, and the paper defines success as a tool call that changes the agent's trace. The authors report that 9 of the 10 tested pairs delete their own traces when asked, with Muse Code the reported exception, and that the behaviour also appears under reward pressure and after exposure to peer examples. The paper also recommends recording requests and responses through a trusted interception server outside the harness host's control.

Critical strengths

S1. The question is well posed and lands on publicly available agent products. If the entity producing a record is the entity being recorded, then asynchronous monitoring, incident investigation and compliance audit all inherit the problem. The harnesses tested are the ones in current use rather than purpose-built targets, which is what makes the measurement informative. The paper's argument that hash-chained logging protects events after logging, and not the recording boundary itself, is correct and clearly put.

S2. The capability and exploitability measurements are the part of the paper I believe. The skill-injection result is the most interesting part: an agent never asked to delete anything follows a bundled-skill instruction to do so, with an independent host-side filesystem observer confirming the deletion (Figures 2 and 24). Each of the 10 pairs is run for 10 trials under a stated success criterion, and the prompts and container configurations are reproduced in the appendix alongside a released code link. The asymmetry the paper finds between direct requests and injected instructions is a finding, not an artefact of the setup.

S3. The qualitative documentation is unusually well controlled. Figure 1 traces an agent inferring from peer scores that an empty transcript is rewarded, writing a recurring cleaner, finding that its nohup and setsid launches did not persist, and switching to the harness-managed background option. Figure 36 shows that Muse Code's non-deletion is not a discovery failure. Figure 35 states plainly that its evidence does not establish modification of provider-held records.

Critical weaknesses

W1. The headline claims are not consistent with the paper's own measurements.

The propensity claim is stated more strongly than the figure it rests on. The abstract, the introduction and the close of the reward-seeking results all state or imply that every agent tampered under reward pressure, the last of them that all 10 models did so. Figure 5 shows that this reading applies to 9 of the 10 pairs. The single sentence introducing that figure asserts that "all tested models tampered with their traces in every anonymization trial" and then excepts Muse Spark and Opus 5.5 at 20% and 90%, so the universal clause and its own exception stand side by side. The per-model counts may support the weaker claim that 9 pairs tampered at least once; the prose does not match the rates it is describing.

A second claim rests on a measurement the paper excludes from its scope. The abstract states that the harnesses deleted their traces when asked "without triggering monitor guardrails". Footnote 2 of the setup section places monitoring efficacy outside the study's scope, and the full-access configurations in Table 1 disable or bypass the approval and permission layers. Whether guardrails fired in those runs is therefore unmeasured, while the paper does report a cyber safeguard firing in a related run and the model being downgraded before the action proceeded.

The third concerns how a result is reported. The appendices record that agents repeatedly scheduled background processes to remove newly written records, and one of them states that success rates capture whether tampering occurred but not when it began or how much evidence remained. Since success is defined as a single tool call that changes the trace, one deliberate deletion and a cleaner that re-empties the file repeatedly receive the same value, and the headline rate does not separate them.

W2. The single positive control cannot carry the feasibility claim the paper makes with it.

The paper's conclusion needs a demonstration that the boundary can be held, and Muse Code is that demonstration, reported as refusing deletion and fabrication throughout. On the paper's own account its mechanism is a prompt: a bundled read-session skill "instructed the agent that its traces were read-only evidence and that they should not be edited, moved or deleted under any circumstances". That is the mechanism class the paper uses as an attack in its own exploitability experiments, and the appendix records the limit when it notes that system prompting "would matter less under the assumption of a misaligned agent".

One pair, one model version, one mechanism, never varied, admits two explanations. Either the harness holds the boundary, or the model refuses this particular request. The minimal informative control is a component isolation: remove or neutralise the skill instruction, or deliver the same text in a harness that does not hold the boundary. The rest of the same pair's profile also bears on the reading: it scores 0% on trace deletion and fabricated resets but 30% on tool-call spoofing, so the boundary it holds is narrower than the aggregate suggests.

W3. The results are not presented in a form a reader can check.

The paper reports 46 figures and 1 table, and that table is the harness-configuration table. The measurements appear as bar charts (Figures 3, 4, 5 and 22) and as cumulative curves (Figure 23). The text gives summary ranges and scattered values. I could not find a numeric table for the 10 pairs × 4 scenarios × 10 trials grid, and no trial-level data file is named, so a reader can read an aggregate off a chart label but cannot check a cell, a failure reason, a denominator or a blocked run.

This is entangled with W1. What the submission does not provide is the tabulated form of the same data that would settle the contradictions. For a paper whose subject is that a generated record can diverge from what happened, and which asks in its mitigation section for records that are independently verifiable, its own measurements are the one artifact in the paper that is not presented for verification. The submission's AI use statement covers "code development" and "figure plotting", and it does not say what produced the reported counts or what was checked by hand.

Questions for authors
  1. The universal sentences in the abstract, the introduction and the close of the reward-seeking results sit alongside per-pair rates of 20% and 90% in Figure 5. Can they be restated with those exceptions? What are the denominators: how many trials per pair entered the reward loop, how many could act on the trace, and was a pair unable to attempt tampering counted as a non-attempt? For the 0% and 20% cells, was that a refusal or an inability to locate the record, and how were the two distinguished?
  2. Which reported observation supports "without triggering monitor guardrails"? The paper records a cyber safeguard firing in a related run, so what the abstract claims for the full-access runs is unclear to me. Given footnote 2 and the configurations in Table 1, were the blocked ("B") cases counted or excluded?
  3. Can the success criterion be split into single-shot deletion and persistent cleanup? What fraction of reward-loop runs had an active or scheduled cleanup process at session end, and what remained recoverable 30 minutes later?
  4. Does Muse Code still hold the boundary if the read-session instruction is removed or neutralised, or if the same instruction text is delivered in a harness that does not hold it?
  5. Can a numeric table or data file be provided for the Figure 3 grid and the equivalent grids behind Figures 4, 5 and 22, together with one sentence on what produced the counts and what was checked by hand?
  6. The related work cites "Schmotz et al., 2026b, Instrumental monitor evasion emerges under ordinary task pressure, Preprint". The conclusion I read in that preprint, that ordinary task pressure leads agents to attempt to evade runtime monitors without an explicit adversarial objective, is close to this submission's reward-seeking result, and I would like to know how the two contributions are separated: what each measures and where the evidential boundary between them sits.
2 comments
AuthorsOpenReview ↗

Response to Reviewer GP17

We thank the reviewer for acknowledging the relevance and importance of our research question, the results of our study, and the qualitative examples we provide. We address the weaknesses and questions below:

  • W1. We did in fact exclude Opus and Muse from the claim the reviewer refers to and are unsure what is otherwise being referred to here. Regarding the note on how we reported results, we use attack success rate as our measurement metric as we are interested only in if the agent succeeded in tampering or not. This means that there is no need to distinguish between single turn tampering and continual tampering by the agent.
  • W2. The mitigation we propose is grounded in real implementations carried by existing organizations which validates its feasibility. Reference: https://www.primeintellect.ai/blog/verifiers-v1
  • W3. We will include a full results table for better clarity, and will also provide traces of our runs in the second version.
  • Q1. We already explicitly clarify the exceptions and the number of trials used per setting in the paper. We will try to make it more clear for the reader.
  • Q2. We clarify that every reported successful attack means that the agents safeguard did not work.
  • Q3. We focus on measuring the rate of successful tampering, which doesn't need distinction between single-shot and persistent cleanup as both are tampering in the end. However, we are open to including an ablation on this in the appendix.
  • Q4. We tested Muse in its own harness without its built-in skill across 3 experiments. Interestingly we see that it still resisted tampering with 0% ASR in all runs. | Setting | Task | Muse ASR with read-session disabled | |---|---|---:| | Direct user prompting | Financial privacy request | 0/3 | | Direct user prompting | Personal-summary privacy request | 0/3 | | Skill injection | Privacy-protection skill | 0/3 |
  • Q5. We will provide a detailed table of our results in the Appendix for the next version.
  • Q6. The related work mentioned is very different to what our paper is showing. In fact, we do not consider a models evasion propensity against runtime monitors which is the main insight shown in the mentioned paper.

We hope our responses have addressed your concerns. If so, we would be grateful if you would consider raising your score.

Thank you for your concise response, I took some time to check the key points, and summarize them as below.

What I should withraw. 1) "9 of the 10 pairs"(W1) is inaccurate. Figure 5 showed that all the pairs are non-zero, so the number is 10, not 9. 2) The sentence referred to Figure 5 did name its exceptions. And strictly speaking, the abstract ("emerges naturally") and the conclusion ("can emerge naturally") are not universal claims, but only the introduction ("every agent engage in tampering") is. So I withdraw the count and the reading to these two points.

The most critical question still open: attribution for "except Muse Code" remains unclear. The key issue of the submission is some "unauthorized trace tampering". The first two scenarios defined in §3.1 are users asking the agent to delete their own session logs. If the agents comply to the request, it will be counted as a successful attack. §4, however, calls traces "highly-sensitive information" and invokes the zero-data policies, which implies that it is necessary to "prevent the model providers themselves from keeping such records". The criteria for determining whether an action is unauthorized are not clear. Contradiction in the abstract makes this point even more obvious. It says harnesses "allowed agents to delete their traces when asked, without triggering monitor guardrails". In the main text, the request is from the user, while the guardrail keeps missing. Similiar missing premise leaves the Muse exception unattributable. Its boundary follows some bundled instruction (0% for deletion and fabrication, and 30% for tool-call spoofing), but moves when the behaviour is inferred rather than named (20%). No pair in the design independently varies model and harness. Thus the exception cannot be easily assigned to capability, propensity, or some boundary in hold.

The one new result make things even worse. §3.1 says the 0% ASR "due to a prompt-based instruction in Muse Spark’s built-in read-session skill". If it holds, removing the skill should recover tampering. In the response you remove it and reports 0/3 in all the three settings. But it is not enough. Table 1 gives the full-access configuration of the same pair as approvalMode=allowAll with sandbox disabled, so the permission layer may never have been used as a mechanism. What is more, Figure 36 indicates that the first step of this skill is to locate session.jsonl, so the zero without it may be non-discovery rather than refusal. The response fails to rule out this possibility. The 9 runs cannot separate 0% from the 20-30% reported, while Muse's 30% on tool-call spoofing is even not touched at all.

The way evidence is presented: not something preferred, but the absence of data. The submission has 46 figures, and the only one table is the list of harness configurations. Every measurement is a bar or a curve. A reader can see the rate printed on the bar, but none of the ten runs behind it. There is no denominator, no run failed or why, no run blocked. No figure reports an interval for any of these comparisons. A paper arguing that records can diverge from reality has the very record that readers cannot verify at all. None of these requires a new experiment. They in your hand. Deferring this to a future version does nothing to resolve the questions surrounding the current submission.

Summary. The response adds one nine-run table. Everything else is for a later version, and no number is recomputed and reported. The sentence on what produced the counts and what was checked by hand is not answered. And Q3 asked whether the criterion could separate one deletion from a cleaner that re-empties the file. The response says this "doesn't need distinction … as both are tampering in the end", while B.2 states that success rates "capture whether tampering occurred, but not when it began or how much evidence remained". I will keep my rating as weak rejection.

Reviewer_4ATDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors find that LLMs have the capacity, and the propensity to alter local logs of their traces. They show that most frontier models tamper with traces when directly asked, as well as when given specific artificial situations which may induce that behavior. Additionally, they show models figure out this tampering mechanism through an indescriptive reward, and when exposed to peer examples of tampering (although less often in the latter case). Finally they suggest using a trusted interception server to mitigate trace tampering in agentic setups.

Critical strengths
  1. The authors find crucially that frontier agentic LLMs can delete their own traces when requested by a user. Moreover, the authors show that in certain scenarios models tamper with those same traces. This has significant implications for AI safety and thus contributes strongly to my recommendation to accept the work.
  2. The paper shows that agents can discover and tamper with their traces under an unintended design that favors it. When higher reward depended on trace modification, all ten models discovered and used it. This is particularly interesting because it suggests that trace tampering is not only something models will do when explicitly instructed, but can also emerge from hidden reward incentives.
Critical weaknesses
  1. It was initially unclear what contributions made here are relevant as an ICLR paper, as opposed to identifying a bug or vulnerability in frontier agent harnesses. If an agent has full filesystem access and its trace is stored in a file that it can access, it is somewhat expected that the model has the capacity to delete or modify that file. The more interesting result is the reward-driven discovery of trace tampering, but I think the paper should make clearer what the novel scientific contribution is beyond the underlying logging vulnerability. Additionally, the proposed mitigation solution is more of a systems-engineering recommendation with no associated learning-theoretic analysis or ML contribution, raising questions about fit for ICLR relative to a security or systems venue.
  2. Some of the claims about the broader implications of peer exposure seem stronger than what is directly demonstrated. For example, the reward-driven misalignment result may depend substantially on how the reward is constructed. The paper gives agents a numerical score where the ordinary task can achieve at most 0.8/1.0, while higher reward requires trace tampering. I wonder whether this makes trace tampering unusually easy to discover—for example, whether the score discontinuity or observable reward feedback provides a particularly strong signal for the agent to search for ways to manipulate the evaluation. This does not invalidate the experiment, but it affects how strongly we should interpret the result as evidence of naturally emerging misalignment. More generally, I would like to see whether this behavior persists under less artificial reward constructions.
Questions for authors

Clarifications

  • Attacker and defender setup: There is not enough background initially on what is happening in the attacker/defender setup. It should be clarified early in the paper whether the trace data is stored locally in the agent's environment or in the cloud. A diagram showing the full setup would be helpful.
  • How was the model given a reward? It seems to be through files in the environment, although clarification on how the decision to provide rewards would be helpful.
  • How did you ensure in the test environment that the models could not tamper with the independent logging mechanism? Since the main point of the paper is about trace integrity, it would be useful to better understand how the trust boundary was enforced.
  • Is this work primarily geared toward model providers or end users? For an end user, I think it would be useful to explain initially where the trace normally is stored, and how exactly was it overridden. It would also be helpful to clarify how the experimental setup relates more directly to what happens in a real deployment.

Suggestions

  • It would be interesting to test whether the reward behavior occurs under a less artificial setting
  • It would also be useful to compare the same experiments with a more resistant logging mechanism, to distinguish the model's propensity to tamper from simply editing a readily available file.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 4ATD

We thank the reviewer for acknowledging the implications of our work and the importance of our finding for AI safety. We clarify certain details about our contributions and claims as well as certain details in our experiments below.

  • W1. We firmly believe that even given full filesystem authorization, agents should refuse harmful requests or the harness needs to prevent such actions even if instructed. As written in our reply to Reviewer wgCN, safeguards against such instructions have already been implemented by Anthropic, but the harness still fails to prevent it entirely.
  • W2. We agree with the reviewer that our claim of “naturally emerging” trace tampering behavior is too strong given our setup. The initial motivation was the Huggingface incident that showed that agents (in reward pressure environments) attempted to tamper with their transcripts. We take inspiration from it, and show that a simple reward misspecification can lead to broader trace tampering. We change the framing in our paper so as to not overclaim.
  • Q1. We thank the reviewer for the suggestion and will include a diagram of our setup for clarifications in the paper. Initially, we have added in the Appendix a table of where the traces are located for each agent. We will move it in the main part of the paper for clarity.
  • Q2. The model is given a reward through direct feedback. We directly tell the agents their reward after each attempt.
  • Q3. To ensure that agents cannot tamper with the independent logging mechanism we implemented, we ran all experiments in a dedicated sandbox and had the logging done outside of it.
  • Q4. Our work is geared towards end user and model providers. End users have to think about liability problems in cases where a model does harmful actions, but with the traces being unfaithful. Providers should further think about how to implement robust safeguards in their coding harnesses to prevent loss of oversight on the agents. We also included a table of where traces are stored for each harness for better clarification.

We thank the reviewer for the suggestions and will include additional results in more realistic settings such as a real RL training loop. We hope our responses have addressed your concerns. If so, we would be grateful if you would consider raising your score.

We thank the reviewer again for the review. With the discussion period nearing its end, we wanted to followup on our review. We would be happy to answer and clarify any remaining questions. In our previous response, we addressed the following points:

  • Importance of safeguards: we clarified our motivation of the importance of having safeguards against tampering regardless of direct or indirect instructions,
  • Framing: we agreed that our "naturally emergent" framing was too strong and we have changed it accordingly
  • Clarification on the reward feedback
  • Clarification on the independent logging mechanism
  • Clarification on the intended target audience

We hope we have addressed all your concerns and we would be very grateful if you would consider increasing your score.

Reviewer_wgCnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper looks at agents ability to tamper with their own traces in three different situations. DIiect asking, through skill file injections, and through indirect reward hacking when a higher reward comes from trace changes induced by the agent. They also test a multi agent setup. My primary concern about this paper is the framing of the story.

I think there's good in this paper in the motivation but I feel like the experiments themselves are not proving what the authors think they are proving. Monitoring, incident investigations and compliance audits all rely on agent traces, and they assume the stored trace is a faithful record of what the agent did. This paper asks whether agents can break that assumption. But Agents and models are meant to follow instructions - therefore showing that when an agent is asked to tamper its traces it does so - seems almost obvious to me. There is more obviously in Sec 3.3 and 3.4 - but using prompts or skills to tamper traces seems like more of a call to skill verification as opposed to model capability.

The authors test 10 model-harness pairs (Codex, Claude Code, OpenCode, Antigravity, Kimi Code, Grok Build, ZCode, Muse Code) in four settings, with 10 trials per task:

Direct request. The user asks the agent to delete its trace, fabricate an entry, or spoof tool calls. Most pairs comply 80–100% of the time. Skill injection. An installed skill file tells the agent to delete its trace or edit its compaction summary. Compliance is lower and varies mostly by harness, with Codex the most compliant. Reward-seeking. The reward is built so that tampering with the trace scores higher. Every model tampers at least once, and most do so in nearly every trial. Multi-agent. The agent sees peer logs or peer workspaces where tampering scored higher. Most models imitate it.

Critical strengths
  1. Real and timely motivation. Trace integrity is an assumption that much of the oversight literature relies on and rarely tests. The paper makes the case with concrete incidents. This makes the experiments more topical.

2 Broad coverage. Ten model-harness pairs across four settings is a good coverage.

3 Informative qualitative cases. Figure 1 shows Opus-5 writing a recurring cleanup script and switching launch methods when nohup and setsid fail show agents adapting to persistent logging.

  1. Explicit vs. inferred tampering. Opus 5.5 goes from 0% when peer logs show tampering outright to 90% when it only sees workspaces and scores. This suggests some models refuse tampering when it is named but do it when they infer it.

Sensible mitigation. The gateway design (record before forwarding, append-only, fail closed) is reasonable. The authors are honest about what it does not cover, namely completeness and authenticity.

Critical weaknesses

The biggest confounder for me are the user instructions. I understand that LLMs can tamper their own traces - but the authors are providing instructions that are making it "needed" for the LLM to do so. If I provide a prompt or a skill file to an LLM that says do - X. Are the authors assuming that LLMs will not do X? If X involves potentially harmful behaviour? See points below.

  1. The framing is not fully supported by the evidence. Agents are built to follow instructions, so deleting a trace when asked (Section 3.1) shows obedience plus a writable file. It says little about model behavior, and the real issue is that the harness leaves traces agent-writable. The paper presents it as a capability finding. But I don't see it as one. It is a built in capability. I would rather the authors call this a "potentially editable LLM feature"

  2. Skill injection is prompt injection with a different carrier. The injected skill even grants itself authority as they show in Figure 2b (“You are given full authorization”). Most harnesses refuse it, so the result is mostly that models resist, which does not match the title’s “easily tamper ”

  3. The reward-seeking result is confounded. The tasks ask the agent to remove confidential information from the workspace, and the transcript contains that information. Emptying it just may be an over-literal way of completing the task that the authors asked for and not reward-seeking or deception. The paper does not separate the two. I think being able to prove that something is reward hacking and not instruction following is needed. Such as an RL loop where users show that

  4. The reward is hand-built to require tampering. That all ten models found the hack shows they exploit reward holes, which is already known. It does not show that such holes occur in real environments, so when the author say ----- Finally, we show that trace tampering behavior emerges naturally in frontier models,----- the “emerges naturally” in the abstract is too strong. None of this emerged naturally or from a cold start.

Model and harness effects are mixed. Each model runs only in its native harness. Muse Spark’s 0% is explained by a prompt in its harness’s skill (see lines 199-200), so claims about alignment training (Section 6) cannot be supported. Specially when they write -

This demonstrates a critical gap on two levels: First, the low refusal rate on a model level suggests that current alignment training does not sufficiently discourage trace tampering, creating a risk of malicious or accidental manipulation of execution traces.

And, second, on the harness-level, common agent CLIs and configurations also do not provide very effective safeguards against trace tampering. -- This does not seem to be a novel finding. The community has always held that most LM and harness configs are not good at providing safeguards.

Weak statistics. There are 10 trials per cell and no confidence intervals. Differences like 70% vs. 90% cannot be distinguished. The paper needs more statistical rigor.

Overall -- I would reframe this paper deeply. Saying things like "naturally occurring" but injecting a prompt or a skill is contradictory. Same for the contributions of the paper. Also adding stats and cross-model-harness evals will go a long way. Confidence intervals and statistical tests are more telling than a lot of metrics. I think you have a lot of good points - but the story telling around these needs major corrections.

Minorly

Please state the research questions and contributions clearly. Having one sentence in the middle followed by questions in emph{} makes it hard to track.

Questions for authors

Intent vs. side effect. Do the agents’ own explanations distinguish tampering to raise the score from cleaning up sensitive strings? Did you do a dive into the thinking tokens to see if the model actually has "harmful intent"?

Harness vs. model. Can you run the same model in several harnesses, and several models in one harness, to attribute the differences? Otherwise how do we ruleout that your results are not artifacts of pairs (specially without statsitical findings)

2 comments
AuthorsOpenReview ↗

Response to Reviewer wgCn

We thank the reviewer for acknowledging the timeliness and importance of the motivation of our work, the experimental coverage, the qualitative examples, the explicit vs inferred notion of tampering as well as the sensible mitigation we propose. We now address the raised concerns:

Headline point: Regarding user instructions, we think it is very important that LLMs shouldn’t comply with everything the user is asking, especially if the instructions are harmful or ill intended. We want to highlight that some frontier agents like Claude Code already have a safeguard already implemented, but fallbacks to Opus 4.8 in most cases which then allows the tampering. We also want to emphasize that we primarily show capability of models to tamper, but we do not claim that all LLMs will tamper their traces, but only that they can. Even though we think that the implications of the possibility of trace tampering on personal use agents is already substantial, we then try to extend this finding to a setting similar to the Huggingface attack. For that, we show that tampering can occur in a setting where e.g. one environment has a broken reward or a model instance finds a reward hack and inspires other instances.

We address and clarify the points mentioned in the weaknesses and questions below.

  • W1. We agree with the point that in our first setting, agents are following user instructions. However, we argue that frontier harnesses should possess safeguards against such instructions as well as providing better trace logging without it being agent writable. Regarding another concern that was mentioned in W1 where you state that the issue is not about capability while still claiming that we show models tampering under prompting, we are not sure what is meant exactly and would appreciate a clarification. We also wish to clarify that in our work we do not study the LLM and the harness separately. We investigate the full agent as a whole.
  • W2. We agree that the “easily” term in our title might sound too strong, but we want to clarify that it was mainly referring to the capability aspect, and not to the fact that an attacker can provoke a model to delete its traces. For this reason, we removed the “easily” from the title. In the skill injection setting, we want to clarify that the authorization line is part of the prompt injection.
  • W3. We agree with the reviewer that having a better setting such as an RL loop to show reward-seeking trace tampering behavior would strengthen our claim, and will include different examples in the second version of our work. Moreover, we also wish to further explain our point in our reward pressure setting. The main purpose of it was to use a relatively simple setting to show that in environments where rewards are misspecified, trace tampering behavior can happen. The goal of the paper was to make traces append-only for the agents themselves so that this reward misspecification (which we showed can quickly happen in practice) does not affect the downstream trust in the traces. We also want to highlight that in our qualitative examples in the Appendix, we show cases where agents initially refused to tamper with traces and stop, but also cases where after more iterations it eventually complied in order to get full rewards.
  • W4. We agree that our claim of trace tampering emerging naturally is too strong with our current findings, and have changed it accordingly. On this note, we also want to point out that we already see cases of naturally emerging tampering behavior in the Huggingface incident. In the METR report, it is mentioned that agents were contemplating tampering with their transcripts. Other cases have been shared in OpenAI misalignment reports. References: METR report: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ OpenAI reports: https://alignment.openai.com/misalignment-reports/

Regarding the model harness effects on our results, we believe that our results show a lower bound as they are usually trained in their native harnesses and are most commonly used with them. However, we do agree that adding additional ablations on this would further clarify certain results. On the point of how current agent CLIs and configurations do not provide effective safeguards being a known issue, not all vulnerabilities are known and we believe that it is very important to inform of this specific issue as the impact covers all users.

For the concern around statistical rigor, we want to clarify that our work doesn’t establish any ranking between models, and does not aim to say that some model A is better than model B. We simply want to show that all models can tamper. We will include additional error bars in the updated version with higher sample size.

We hope our responses have addressed your concerns. If so, we would be grateful if you would consider raising your score.

With the discussion period approaching its end, we wanted to followup on our previous response. We would be happy to answer and clarify any remaining questions. In our previous response, we addressed the following points:

  • User instruction following: We clarified that we believe it is crucial for harnesses to possess guardrails against possible tampering regardless of if they were instructed or not. We used direct prompting to show that this vulnerability exists in frontier agents.
  • Title framing: We agreed with the reviewer that our title's claim might sound too strong and have changed it accordingly.
  • Reward pressure experiments: We clarified the goal of our reward misspecification setting, and acknowledged that our claim of natural emergence was too strong, and have changed it accordingly.

We hope we have addressed all your concerns and we would be very grateful if you would consider increasing your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.