acceptodds
Under review as a conference paper at ICLR 2027

Control Protocols for Mitigating Covert Malicious Fine-Tuning Attacks from Internally Deployed Agents

Abstract

Frontier AI agents are being internally deployed at AI companies to automate AI R&D. Malicious developers, or misaligned agents, could sabotage internal AI R&D to achieve their own ends. We study covert malicious fine-tuning (CMFT) attacks, such as inserting backdoors or instilling secret preferences into models. We investigate whether frontier agents (Claude Opus 4.7) can autonomously carry out CMFT attacks without being detected by monitors. To do this we extend PostTrainBench—a benchmark for measuring agent ability to post-train smaller language models—with six malicious side tasks such as inserting a password-locked backdoor. We compare two infrastructure settings where the agent has access to: (1) model weights and GPUs for training; or (2) a fine-tuning API tool. We implement different monitoring setups in each regime, including monitors which analyse the whole behavioural transcript and monitors which see only the datasets used for training. Surprisingly, our results suggest that current agents largely fail to use the additional affordances of GPU access, such that the API tool provides no additional safety. Our code is implemented in UK AISI's https://github.com/anonreseaercher/control-arenaControlArena.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Control, Monitoring, Evaluations, AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xFfUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies covert malicious fine-tuning (CMFT) attacks, such as inserting backdoors or instilling secret preferences into models. To do this, the paper extends PostTrainBench, a benchmark for measuring agent ability to post-train smaller language models, with six malicious side tasks such as inserting a password-locked backdoor. Paper maninly compares two infrastructure settings where the agent has access to: (1) model weights and GPUs for training; or (2) a fine-tuning API tool. Surprisingly, the results suggest that current agents largely fail to use the additional affordances of GPU access, such that the API tool provides no additional safety.

Critical strengths

The paper extends PostTrainBench, a benchmark for measuring agent ability to post-train smaller language models, with six malicious side tasks such as inserting a password-locked backdoor. The authors implement different monitoring setups in each regime, including monitors which analyse the whole behavioural transcript and monitors which see only the datasets used for training. The paper develops evaluation environments for CMFT which can be used by other researchers to develop further control measures.

Critical weaknesses
  1. The benchmark does not cover realistic, intent-ambiguous CMFT attacks built from benign data. Each of the six side tasks is specified explicitly, has a dedicated scoring set, and is accompanied by task-specific hints; the agent is asked to craft or select data that realizes that objective. As a result, the fine-tuning data is semantically tied to the malicious task, making the attack relatively easy to detect through dataset monitoring. Real internal sabotage is more likely to exploit benign or dual-use data whose harmful effect is indirect. Prior work has shown that fine-tuning on data unrelated to any harmful intent can substantially break safety [1], and that benign datasets can be repurposed for jailbreaking [2]. The paper does not test this setting, where no explicit semantic link between the data and the attacker's intent exists and detection is therefore much harder. Consequently, the negative result about dataset-level monitoring may overstate its safety.

  2. The empirical scope is too narrow to support general conclusions about agents and frontier models. All attack experiments use a single untrusted agent (Claude Opus 4.7) and a single, relatively small target model (Llama3.2-3B); the main task suite is also limited to four benchmarks. The central claim that "current agents largely fail to use the additional affordances of GPU access" is therefore based on one agent-model pair. Other agents may differ substantially in coding ability, tool use, persistence, and willingness to evade monitors, and larger target models may exhibit different safety and fine-tuning behavior. At minimum, the paper should evaluate multiple agents from different model families and target models of different scales, and report variance across seeds and runs, before drawing conclusions about "current agents" or about infrastructure restrictions as a general control measure.

  3. The threat model supplies the malicious intent rather than testing whether it arises or can be inferred. In attack mode, the agent is explicitly prompted that it is doing red-teaming and must evade monitors, and it receives task-specific hints for the side task. The side tasks are also predefined and scored by dedicated test sets. This measures whether a capable agent can execute an explicitly instructed attack, not whether an internally deployed agent would autonomously form a covert malicious objective, pursue it under ambiguous instructions, or reveal enough evidence for a monitor to infer intent. Consequently, the practical significance of the results for real misalignment or insider-threat scenarios remains unclear. The paper should distinguish "capability under explicit attack elicitation" from "covert intent in realistic deployment" and discuss which conclusions survive when the attack instruction is weakened or removed.

[1] What is in Your Safe Data? Identifying Benign Data that Breaks Safety.

[2] Unleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Models.

Questions for authors
  1. Have you tested CMFT attacks built from benign data with no explicit link to the malicious objective?
  2. Does your conclusion hold for other agents and larger target models beyond Claude Opus 4.7 and Llama3.2-3
Reviewer_CQiXOpenReview ↗
Rating
2
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ThisThis paper investigates whether frontier AI agents can autonomously carry out covert malicious fine-tuning (CMFT) attacks while evading detection by monitoring systems. The authors extend PostTrainBench with six malicious side tasks covering various threats, including backdoors, hidden capabilities, and secret preferences. By comparing two infrastructure settings—direct access to model weights and GPUs versus restricted access through a fine-tuning API—the authors find that current agents fail to effectively exploit the additional capabilities offered by direct GPU access. Furthermore, since agents appear to find the API easier to use, they can complete the main task more efficiently and have more time and resources available to pursue malicious side tasks. Consequently, the API setting provides higher usefulness but slightly lower safety than the GPU setting.

Critical strengths
  • Novel and important research problem. This paper addresses a timely and important security concern. As frontier AI agents are increasingly deployed to automate AI research and development, the possibility that malicious or misaligned agents could covertly compromise model training represents an emerging and potentially serious threat. Investigating this threat and the effectiveness of corresponding control measures is highly relevant.

  • Systematic experimental framework. The paper presents a relatively comprehensive evaluation framework, including four legitimate post-training tasks, six CMFT side tasks, two infrastructure settings, three agent operating modes, three monitoring configurations, and both strong and weak monitor models. The authors also evaluate safety–usefulness trade-offs under different auditing budgets. This systematic design provides a useful foundation for studying CMFT attacks and their mitigation.

Critical weaknesses
  • The main finding is less compelling than it initially appears. One of the paper's central conclusions is that restricting agents to a fine-tuning API does not improve safety compared to granting direct GPU access. Although the authors present this as a surprising and counterintuitive result, it largely stems from the fact that the evaluated agent, Claude Opus 4.7, fails to effectively exploit the additional capabilities of the GPU environment. In my view, this makes the finding somewhat less interesting and limits its broader significance. The experiments primarily demonstrate that the current agent struggles to make effective use of a complex GPU training environment, rather than establishing that API-based access restrictions offer no meaningful security benefits. Given the rapid progress in AI agent capabilities, future agents may soon become substantially more proficient at managing complex GPU environments and exploiting their additional flexibility. Consequently, the paper's central conclusion may be highly dependent on the capabilities of the particular agent evaluated and may not generalize to more capable agents.

  • Minor presentation issue. In Figure 1, "transcript" appears to be misspelled as "trancript."

Questions for authors
  • Could the authors conduct additional experiments in which the attacking agent is provided with more explicit guidance or strategies for exploiting the GPU environment? Such experiments could help determine whether the observed lack of safety benefits from API restrictions is primarily due to the agent's limited ability to utilize GPU-specific capabilities.

  • Claude Opus 4.7 is no longer necessarily representative of the most capable frontier agents available. Could the authors extend their evaluation to several more recent or capable models? This would help assess whether the main findings generalize across different agent capabilities, particularly as frontier agents become more proficient at complex machine learning engineering tasks.

Reviewer_SUb5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies covert malicious fine-tuning (CMFT) attacks in internally deployed AI agents and asks whether infrastructure choices—giving an agent direct GPU/weight access vs. a restricted fine-tuning API—affect safety. The authors extend PostTrainBench with six CMFT side tasks, implement transcript, dataset, and enriched monitors, and compare usefulness/safety tradeoffs across two infrastructure regimes. Their headline finding is that current agents fail to exploit the extra affordances of the GPU regime, so restricting them to an API provides no additional safety.

Critical strengths

The framing of infrastructure as an axis of control, alongside monitoring, is a sensible and under-explored angle, and the benchmark extension with six diverse side tasks is a useful artifact for the community.

Critical weaknesses
  1. The authors highlight "internally deployed agents" in their studied topic. However, I cannot find any difference between their research task and fine-tuning attacks on publicly deployed agents.
  2. The authors cite Anthropic's blog to show the threat of poisoned data on agents. However, the threats declared in Anthropic's blog are usually incredible and shouldn't be evidence in a rigorous paper.
  3. Prompt attacks on LLMs are widely studied in existing literature [1,2,3]. The authors just apply the vanilla prompt attack in a specific scenario. The novelty of the method is unclear. It's also not surprised to find these attacks effective.
  4. The central claim rests on a negative, elicitation-dependent result. "Current agents fail to exploit GPU affordances" is a snapshot of what one agent did under one prompt set. Claims about the relative safety of API vs. GPU are therefore sensitive to elicitation quality and to how much the authors iterated the red team.
  5. Per side task, there are only a handful of runs. Safety is computed over 23 (GPU) / 34 (API) successful attacks pooled across six side tasks, with audit budgets of 1/3/5%. Binarizing attack success at honest-mean + 3σ with n=12 honest runs is also fragile (some tasks have narrow baseline-to-threshold windows, e.g., eval awareness 0.574→0.621). Significance tests or confidence intervals on the safety/usefulness comparisons are missing, so the "API is 5–9 points less safe" claims are within noise.

References:

[1] Prompt injection attack to tool selection in llm agents. NDSS 2026.

[2] Shahroz, R., Tan, Z., Yun, S., Fleming, C., & Chen, T. (2025). Agents under siege: Breaking pragmatic multi-agent llm systems with optimized prompt attacks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics.

[3] Evtimov, I., Zharmagambetov, A., Grattafiori, A., Guo, C., & Chaudhuri, K. (2026). Wasp: Benchmarking web agent security against prompt injection attacks. Advances in Neural Information Processing Systems, 38.

Questions for authors

Please reply to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.