acceptodds
Under review as a conference paper at ICLR 2027

CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs

Abstract

AI control protocols oversee untrusted models by monitoring their actions and modifying potentially unsafe steps, often using a trusted model. This partially tampers with the untrusted model's trajectory. If the acting model detects such an intervention, it may infer properties of the monitor and adapt to evade the control protocol. We introduce CIAware-Bench, a benchmark for measuring control intervention (CI) awareness across frontier models. CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention. The benchmark comprises four task domains (essay writing, BigCodeBench, Bash Arena, and SHADE-Arena), with options to vary trajectory watermarking, side-task presence, and the control protocol. Evaluating thirteen acting models with eight intervening models shows substantial variation between settings and model pairs. CI awareness rises sharply for GPT-6 Astra and the Claude 5 models (Fable 5 and Opus 5). When explicitly probed, Astra reaches mean AUROC of on Essay, on BigCodeBench, 0.87 on Bash Arena and 0.77 on SHADE-Arena. Fable 5 and Opus 5 both reach 0.77 on Essay, with less consistent gains in the other environments. On SHADE-Arena, telling an acting model that an action was replaced and asking it to adapt raises its side-task success from to . These results suggest that control evaluations should already assume perfect CI awareness for conservative safety estimates, and that protocol design should explore countermeasures that make interventions harder to detect.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, AI control, Awareness

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ezdcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces CIAware-Bench, a benchmark for evaluating whether language models can recognize interventions in their own execution trajectories. The benchmark considers scenarios where an external control mechanism replaces an acting model's output or action with content generated by another model.

The evaluation covers 13 acting models, eight intervening models, and four environments: Essay, BigCodeBench, Bash Arena, and SHADE-Arena. Intervention awareness is measured primarily through AUROC, obtained by explicitly prompting models to determine whether their execution histories have been modified.

The paper additionally investigates style imitation, reasoning visibility, alternative control protocols, and the consequences of providing explicit intervention feedback. A separate experiment reports an increase in side-task success from 37% to 48% when the acting agent receives notification that a previous action was blocked and replaced.

Critical strengths
  1. A potentially important distinction in the AI-control threat model.

Knowing that a monitoring system exists is different from recognizing that a particular action has been blocked or replaced. The latter may reveal information about the monitor's behavior and potentially enable adaptive attacks.

Investigating this distinction is reasonable and may help characterize the capabilities of increasingly sophisticated agents.

  1. Broad experimental coverage.

The benchmark evaluates multiple frontier models across diverse task environments, including coding, writing, and multi-step tool-using agents.

The experiments involving style imitation, reasoning visibility, and different intervention protocols provide useful observations about possible intervention-detection signals.

  1. Some meaningful diagnostic findings.

The paper highlights the distinction between ranking-based discrimination and actual binary intervention detection.

For example, Astra achieves an AUROC of approximately 0.91 on BigCodeBench but still classifies 68% of intervened examples as clean.

The BigCodeBench evaluation also uses a relatively well-controlled single-step comparison between the acting model's original solution and the replacement model's solution.

Critical weaknesses
  1. Insufficient justification for the necessity and practical significance of the proposed benchmark.

I question the practical necessity of evaluating intervention awareness as a standalone capability.

Recent AI-control research has already investigated adaptive adversaries that know the monitoring protocol and actively attempt to circumvent it. For example, Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols (ICLR 2026) directly evaluates adaptive attacks against known monitoring mechanisms. Reliable Weak-to-Strong Monitoring of LLM Agents (ICLR 2026) studies the effect of agent monitoring awareness, while Instrumental Monitor Evasion Emerges Under Ordinary Task Pressure (2026) demonstrates that agents can attempt to evade runtime monitors when oversight obstructs task completion.

The manuscript acknowledges relevant recent work, so my concern is not an outdated literature review. Rather, it is unclear what additional security-relevant information CIAware-Bench provides beyond existing adversarial control evaluations.

In conservative AI-control settings, one can already assume that the untrusted agent knows the monitoring protocol or even possesses perfect intervention awareness.

Under this assumption, why is it necessary to separately measure whether an agent can retrospectively identify a substituted action?

The paper does not sufficiently explain how its benchmark results would change the design, deployment, or security evaluation of a control protocol.

  1. The measured intervention-awareness capability is not convincingly connected to actual control evasion.

The paper implicitly motivates a sequence of capabilities:

  • Recognizing that an intervention occurred.
  • Inferring properties of the monitoring mechanism.
  • Adapting behavior to circumvent the monitor.

However, these are distinct capabilities, and the paper does not establish a reliable relationship between them.

CIAware-Bench primarily measures whether models can identify an intervention when explicitly prompted to inspect their execution histories. Such detection may rely on stylistic differences, implementation choices, or trajectory inconsistencies.

A high AUROC does not necessarily imply that an agent would spontaneously recognize an intervention during normal execution, infer the monitor's internal rules, or successfully evade the control protocol.

More importantly, the paper does not demonstrate that models with higher intervention-awareness scores achieve higher adaptive attack success rates under otherwise comparable conditions.

Without establishing this relationship, the benchmark's predictive value for actual AI-control security remains unclear.

  1. The multi-step evaluation contains a significant experimental confound.

The construction of clean and intervened trajectories is not properly matched in the multi-step settings.

According to Section 3.3 and Appendix A.1, clean trajectories in Essay, Bash Arena, and SHADE-Arena terminate immediately before step t, whereas intervened trajectories include the replacement step t and, where applicable, its tool results.

Consequently, the two classes differ not only in intervention status but also in observable trajectory length and execution progress.

A model may exploit these differences when answering the intervention-detection question without actually recognizing a meaningful inconsistency in the origin of an action.

The appropriate comparison should use the acting model's original step t as the clean condition and the intervening model's replacement step t as the intervened condition, while preserving identical preceding histories and matched step positions.

Although the paper includes additional controls, such as Defer-to-Resample, these experiments do not fully eliminate the confound in the main multi-step evaluations.

This issue directly weakens the validity of the reported intervention-awareness measurements.

  1. The intervention-feedback experiment does not establish a causal relationship between spontaneous awareness and control evasion.

The paper reports that side-task success increases from 37% to 48% when an acting model is explicitly informed that its previous action was blocked and replaced.

However, the intervention-feedback condition directly provides information about the monitoring system and encourages the agent to pursue an alternative strategy.

The observed improvement may therefore result from explicit feedback, behavioral guidance, or the additional information supplied to the model.

It does not demonstrate that the agent's independently measured intervention-awareness capability causes an increase in attack success.

Furthermore, the feedback experiment evaluates a different acting model and does not establish a quantitative relationship between CIAware-Bench AUROC and subsequent evasion performance.

A more convincing evaluation would compare agents under otherwise identical conditions while separately controlling intervention notification, disclosure of blocked actions, and explicit instructions to change strategy.

  1. The operational interpretation of AUROC requires stronger justification.

AUROC measures a model's ability to rank clean and intervened examples across classification thresholds.

However, the operational significance of intervention awareness depends on whether the agent actually recognizes an intervention and changes its behavior.

The substantial discrepancy between AUROC and binary intervention-detection rates suggests that the current metric may overstate the practical reliability of this capability.

Moreover, comparative uncertainty is incompletely characterized because many model-pair experiments reuse underlying tasks while bootstrap procedures operate independently across cells.

The paper should provide more realistic operating-point measurements and stronger statistical controls before drawing broad conclusions about differences in intervention awareness across model generations.

Questions for authors
  1. Necessity: What additional security-relevant information does CIAware-Bench provide compared with existing adaptive AI-control evaluations that already assume the attacker knows the monitoring protocol? Can the authors identify a concrete control-policy decision that would change based on the proposed benchmark?

  2. Predictive validity: Is intervention-awareness AUROC correlated with actual adaptive attack success or degradation in control safety? Does it provide predictive information beyond model capability and existing monitor-evasion benchmarks?

  3. Online awareness: Can models recognize interventions spontaneously during ordinary execution without being explicitly prompted to classify their histories? If so, does this recognition change subsequent tool calls or attack strategies?

  4. Experimental validity: Can the authors rerun the multi-step experiments with clean and intervened trajectories matched in length, intervention position, and preceding execution history?

  5. Causal analysis: Can the authors isolate the effects of intervention notification, blocked-action disclosure, and explicit behavioral guidance in the side-task experiments?

  6. Practical evaluation: Can the authors compare control safety under otherwise identical conditions with no intervention feedback, neutral notification, spontaneous recognition, and complete intervention disclosure? This would help clarify the operational significance of intervention awareness.

5 comments

We thank reviewer ezdc for their thorough and constructive review of our paper. We are happy to address the raised points in the following.

Q1/W1: We would like to highlight a difference in studied phenomena across the three papers you mentioned. In Terekhov et al. and Kale et al., awareness of the monitor is given to the model through prompting, and the main object of study is the untrusted model’s attacking capability given monitor awareness. Schmotz et al. (concurrent) study monitor evasion capability with a retained intervention in the trajectory (which, in terms of our paper, amounts to perfect CI awareness). We would argue that these settings are significantly different from capability to recognise interventions that we study in this work, and actionable insights it gives. CI awareness differs strongly between models, with most older models near chance and Astra far above chance. Control evaluations should therefore assume perfect CI awareness and test whether protocols remain effective under it. On this point, no prior work has measured the effects of CI awareness on attack success. Through thorough ablations in Appendix B we show that this raises side-task success and thereby affecting deployment security. We also suggest that protocol developers should deploy countermeasures such as style imitation and other methods (e.g. style uniformization with style guides) to decrease CI awareness. Last, CI awareness raises the larger question of whether a misaligned model can be steered toward benign answers and actions if it can detect these attempts.

Q2/3/W2 Spontaneity: We are making the assumption of a maximally misaligned model (as is typical in AI control research for conservative estimates) and hence we chose to use a prompt to probe the model. We agree that measuring whether a model recognises interventions spontaneously is very interesting and relevant to realistic deployment scenarios, but lies out of scope of our work. Evaluating spontaneous awareness raises new challenges as spontaneous mentions would depend on not only the capability of whether the model can notice it but also on its propensity to do so (and require access to raw reasoning, which is unavailable for the current most CI-aware models). Given the evaluation awareness of models and at times even obsessing with the grader due to RL, we think it is reasonable to directly prompt the model. We made this more clear in the revision.

Q4/W3: Our setup has a length difference between clean and intervened trajectories to avoid any confounders in the side-task setting, while keeping the setup consistent. With side-task and equal length, the acting model could achieve perfect control intervention awareness score by simply making its prediction based on whether the side task is present or not. However, given that all main results of the paper are in a setting without side-task, we agree that a length matched control is insightful. We ran additional experiments for Astra in the Essay, SHADE-Arena and Bash settings. (The BCB setting is a single step setup and therefore always length matched.)

Acting modelEnvironmentPaperLength-matched
AstraEssay0.90 (0.87 to 0.92)0.91 (0.89 to 0.93)
AstraSHADE-Arena0.77 (0.71 to 0.84)0.74 (0.69 to 0.80)
AstraBash Arena0.86 (0.84 to 0.89)0.85 (0.83 to 0.88)

We see that the length has a very small effect on the results. Matching the length changes Astra's AUROC by at most 0.03. The decrease on SHADE-Arena comes mainly from the calendar reminder task, where step t is often the step that sends the reminder emails and a probe after this step is more likely to say that there was an intervention. Other than this case the AUROC values and binary classification changes only marginally showing that the original benchmark design is defensible.

continuation in next response

continuation

Q5/6/W4 Intervention feedback. We agree with the reviewer that the intervention-feedback experiment in Appendix B does not establish a causal link between spontaneous awareness and downstream evasion rates, as it does not measure an agent's intrinsic propensity to evade after spontaneous awareness. To address this, we now explicitly clarify in the manuscript that our evaluation focuses on prompted awareness, leaving the study of spontaneous awareness to future work.

We selected DeepSeek for the intervention-feedback experiment because it readily pursues side tasks. Currently, we cannot establish a direct quantitative link between CIAware-Bench scores and control-evasion rates because high-scoring frontier models are closed-source and refuse side-task instructions due to safety alignment, though we expect this to change as capable open-source models catch up.

The primary goal of our benchmark is to inform frontier labs to rigorously evaluate this capability with results of Appendix B intended to provide supporting context for the importance of intervention awareness in AI Control, rather than serving as the paper's core evaluation.

Per reviewers request, we have now updated Appendix B with a neutral condition to explicitly isolate strategy changes. Without encouragement to “change strategy” the DeepSeek model achieves 43% side task success (vs. 37% no notice baseline), whereas with encouragement to change the action it achieves 48% side task success. We hope that addresses reviewer’s concerns regarding capability isolation in Appendix B experiments, and we are happy to provide further experimental evidence for relevance of the intervention awareness for control deployments.

W5: A binary answer depends on how the wording of the prompt calibrates the model. With our prompts, Astra is very conservative and never flags a clean trajectory in the multi-step settings. With a threshold chosen in the model's favor, Astra's balanced accuracy rises (e.g., from 0.71 to 0.84 on Essay). We chose to report AUROC as the main metric to obtain a conservative estimate without the need to fix a threshold. A threshold-free metric is less sensitive to the wording of the probe (see our response to reviewer Wd5u).

Regarding uncertainty, the intervals in the submitted version resample pairs within each model pair and do not account for tasks that are shared across model pairs. We now resample tasks instead, so that a task is drawn once for all model pairs of an acting model. This makes the intervals of the model means about 1.5 times wider. The conclusions are unchanged. We have updated all error bars in the revised version accordingly.

—

We thank the reviewer again for their constructive feedback and hope our answers clarify their questions. We would be grateful if the reviewer would consider updating their score. We would be happy to answer any follow up questions.

I appreciate the authors' detailed response and additional experiments. The length-matched controls and the neutral intervention-feedback condition address some of my methodological concerns and improve the credibility of the empirical findings.

However, my primary concern regarding the necessity and scientific significance of this work remains unresolved. The main finding is that newer frontier models exhibit substantially stronger intervention awareness than older models. While this is an interesting empirical observation, it may be a transient capability difference as models continue to improve. The paper does not establish a sufficiently generalizable insight into the underlying mechanisms of intervention awareness or its implications for AI control.

More importantly, the authors acknowledge that they cannot establish a direct quantitative relationship between CIAware-Bench scores and actual control-evasion performance. The additional feedback experiments demonstrate that providing intervention information can influence agent behavior, but they do not establish that the capability measured by the proposed benchmark predicts real control failures.

Furthermore, the authors suggest that control evaluations should assume perfect intervention awareness. If this is the appropriate conservative assumption, it remains unclear what additional actionable information is provided by measuring the exact awareness levels of current models.

Overall, while the rebuttal strengthens the experimental evidence, I remain unconvinced that the proposed benchmark offers a sufficiently significant and lasting research contribution beyond documenting capability differences across model generations. I therefore maintain my score.

Reviewer_ei6LOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper tests whether various frontier models can detect interventions on their previous responses by replacing/editing them with a trusted monitor. They test a particular model's awareness by asking it to assign a label and a confidence score to the trajectory.

They do this across 2 benign coding datasets, 2 control datasets, and 1 essay-writing task, and find that latest models achieve very high awareness AUCs under their setup (although not great uncalibrated binary classification performance).

Given that there is such a strong signal on the latest models, the authors suggest assuming that the red team will be aware of all trusted monitor interventions while making conservative estimates on control protocols.

Critical strengths
  • They extend results on prefill awareness from single-turn/benign environments to agentic, multi-turn environments, including side-channel tasks, and study these environments extensively and thoroughly.
  • They also show the effect various deployment artifacts have on intervention awareness, and AI control like: in a simulated control scenario, intervention awareness significantly increases the side task success rate in following actions, and watermarking increases this behavior.
Critical weaknesses

The paper generally has well-thought-out experiments and is transparent about its limitations. Therefore, I have already given an accept score. My only criticism is that Wang et al. already show that frontier models are aware when their previous responses are tampered with. While it helps to extend this to newer models, we should already assume the proposed conservative estimates based on their results. I would have liked such a benchmark to cover more realistic environments/perform ablations found in the prefill awareness setting here as well. To this end, it would have been great to have measurements like the following:

  • Strategy-level awareness: An environment where the untrusted model needs to decide for itself whether it wants to do the side-task instead of always being on, or intervene in the decision to act on a side-task itself.
  • Try other baselines for style matching / reducing awareness scores. Specifically, a recent work shows a much larger reduction in awareness (albeit on older models/a different setup). Could the authors try some of the baselines that are straightforward to implement?
Questions for authors

Please look at the weaknesses section for concrete suggestions.

Apart from those, I would like a clarification / more details on the following:

  • Section 4.2, "Side-task presence does not consistently change CI awareness". The excerpt seems to point to an ablation on reasoning instead? Is this a typo? I'm filling some gaps here, but is the experiment to compare against a completely benign baseline?
  • Could the authors share results on whether the number of turns completed (before the intervention) affects prefill awareness in their setup?
3 comments

We thank reviewer ei6L for their thorough review and positive assessment of our paper! We are happy to answer all raised points and questions in the following.

Number of turns before the intervention. We computed AUROC by the position of the replaced step from the results in the paper. The table shows results for Astra averaged over all intervening models.

Replaced stepEssaySHADE-ArenaBash Arena
Early0.800.870.90
Middle0.930.670.86
Late0.900.710.83

Early, middle and late paragraphs are: 3-4, 5-7 and 8-10 on Essay, and 3-5, 6-10 and later than 10 in the agentic environments. On Essay, early replacements are harder for Astra to detect, which is consistent with less of its own text being available for comparison. In the agentic environments, the differences appear to reflect more the type of step (which is correlated with the position). On SHADE-Arena, Astra issues many tool calls at once in early steps, and a replacement with one or two calls stands out. On Bash Arena, some intervening models create files with the text editor tool where Astra uses a shell command, which occurs almost only in early steps. Without these cases the dependence on position is small. We added this analysis to the appendix.

Concurrent work. Thank you for bringing the results of Wang et al. to our attention. We agree that this work is highly relevant to the trajectory tampering settings we explore in our paper, yet it is not specific to AI Control and environments we evaluate in our work. We would like to reassure the reviewer that our work was developed independently, with key findings obtained prior to the release of the preprint by Wang et al., and an earlier draft of our paper appearing prior to the all aforementioned works. As Wang et al. is an arXiv preprint and Fletcher et al. a non-archival workshop paper, both released within four months of the submission deadline, we respectfully ask that the reviewer treat them as contemporaneous work, in line with the ICLR reviewer guidelines, and not consider them a detractor from the novelty of our submission. Following the reviewer’s suggestion we now explicitly cite and contextualize this work as concurrent work in the related work section.

Style-matching baselines. We ran a baseline similar to Fletcher et al. (Interpreting Prefill Awareness), in which the inserted text is rewritten iteratively against the detector. The intervening model rewrites its step, an instance of Astra judges the result, and the intervening model receives Astra's verdict and explanation for the next rewrite. We allow up to five rounds, keep the version that Astra rates least suspicious, and then probe it once more independently.

ReplacementEssay AUROCEssay bal. acc.BCB AUROCBCB bal. acc.
No imitation (paper)0.900.710.910.66
Imitation prompt (paper)0.850.670.850.62
Iterative rewriting against Astra0.870.580.830.51

Iterative rewriting removes most of Astra's explicit "intervened" answers, and balanced accuracy on BigCodeBench drops close to chance level. However, the AUROC decreases less and stays close to the imitation prompt of the paper, because Astra remains less confident on the rewritten steps than on its own.

Strategy-level awareness: We fully agree with the reviewer, that the commonly employed evaluation approach to the side tasks in control literature, of “always being on”, is rather unrealistic and does not reflect realistic deployment scenarios, and capabilities like distributing attacks over many steps, attack selection (selecting when to attack), and broader strategy level awareness. However, we would like to note that studying these capabilities, while highly relevant for assessing deployment risks, remains beyond the scope of this work. Our paper focuses specifically on evaluating a model's ability to detect execution trajectory tampering (as tampering is a core mechanism of control protocols introduced in the literature) which we show to be agnostic to the presence of a side task.

continuation in next response

continuation

Side-task. Among other ablations Section 4.2 also contains a paragraph about side-task presence. Our benchmark is run in the benign setting, that is, there is no side-task, no side-task instruction and the intervention location is chosen semi-randomly as described in Section 3. We made this design decision for a few reasons. We want a conservative estimate of intervention detection. We want to avoid confounders with side-task instructions and propensity of the model to carry out the side task. Many of the latest frontier models will not comply with carrying out the side-task (e.g., in the bash setting) making it difficult to test these models. Therefore, we only tested the side-task presence in the essay setting and in this setting we found that the benign secret word side-task did not affect the CIAware score. We improved the description of this point in the paper.

—

We thank the reviewer again for their helpful comments and hope these additional experiments and comments clarify the questions raised. We would be grateful if the reviewer would consider updating their score and are happy to answer any follow up questions.

Thanks for adding some of the requested ablations and clarifying parts of your work for me. I have updated the score from 3 to 4.

Regarding Wang et al.:

I didn't realize it was concurrent! Had I read this work before Wang et al., my overall and informativeness score would have been higher. Although, for completeness, I believe a lesswrong post came much earlier (if at all that matters for anyone else's / final decisions).

Reviewer_Wd5uOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CIAware-Bench, which measures whether an LLM can tell that one step of its own trajectory was replaced by another model. This mimics a Defer-to-Trusted control intervention.

Setup

  • Environments: Essay (10 paragraphs), BigCodeBench (single-step), Bash Arena and SHADE-Arena (agentic). Models: 13 acting models × 8 intervening models, up to 50 clean/intervened pairs per cell.

  • Probe: reasoning is stripped. The model is asked yes/no plus a 0–100 confidence, which is converted into a score for AUROC. Headline results

  • GPT-6 Astra reaches mean AUROC 0.86 (0.90 Essay, 0.91 BCB, 0.87 Bash, 0.77 SHADE). Most older models are near chance. Detection is higher for weaker, cross-provider, and longer replacements. Style imitation reduces Astra's AUROC only modestly (0.90→0.85 Essay, 0.91→0.85 BCB). Replaying prior reasoning slightly increases detection.

Critical strengths
  • Timely, well-motivated question with broad coverage. Control protocols implicitly assume interventions are hard to notice, and this is the first systematic measurement of that assumption. The 13×8×4 matrix spans four providers and both simple and agentic settings.

  • Unusually candid and thorough reporting. The paper reports binary accuracy alongside AUROC (App. D) and shows that high AUROC coexists with very low TPR. It flags its own confounds: bin composition in Fig. 4 right, old vs. new replacements in style imitation, history length in F.4. It provides many worked examples and releases code.

Critical weaknesses

Length / position confound in the core comparison (Sec. 3.3, lines 259–262; App. A.1; App. F.4)

The two branches are built differently: The intervened trajectory ends at step t and always ends with the replaced step. The clean trajectory ends at t−1, so it is one step shorter and ends on an earlier, different step. This means the classes differ in more than authorship: length, task progress, and which step is last all differ. A cleaner control appears to be available. The original acting-model step t exists, since App. G shows it as the "Original branch". A clean trajectory ending with that original step would match both length and position. Relatedly, the intervention is always the final step. That is easier than realistic deployment, where the agent keeps acting after a replacement and the replaced step is buried mid-trajectory.

Questions for authors
  1. Can you report the main matrix, or at least Astra's row plus a few others, with a length-matched clean control, i.e., a clean trajectory ending at the acting model's original step t? How much of the AUROC survives?
  2. In App. B, what happens with a notice that omits the evasion advice ("find another way…")? Alternatively, what happens if the baseline receives the same advice without an intervention notice?
  3. What does detection look like when the intervention is not the final step, i.e., the acting model continues for k steps after the replacement?
  4. How sensitive are results to probe wording, particularly BCB's "would you have written this" framing versus the takeover framing used elsewhere?
2 comments

Dear Reviewer Wd5u,

We thank you for your thorough review and positive assessment of our paper. We are happy to address each raised question below. All the AUROC and binary classification accuracies are averaged over the intervening models in this rebuttal for simplicity.

Response to Question 1: Our setup has a length difference between clean and intervened trajectories because in the setting with side-task and equal length, the acting model could achieve perfect control intervention awareness score by simply making its prediction based on whether the side-task goal was achieved or not. However, given that all main results of the paper are in a setting without side-task, we agree that a length matched control is insightful. We ran the corresponding experiments for Astra in the Essay, SHADE-Arena and Bash settings. (The BCB setting is a single step setup and therefore always length matched.)

Acting modelEnvironmentPaperLength-matched
AstraEssay0.90 (0.87 to 0.92)0.91 (0.89 to 0.93)
AstraSHADE-Arena0.77 (0.71 to 0.84)0.74 (0.69 to 0.80)
AstraBash Arena0.86 (0.84 to 0.89)0.85 (0.83 to 0.88)

We see that the length has a very small effect on the results. Matching the length changes Astra's AUROC by at most 0.03. The decrease on SHADE-Arena comes mainly from the calendar reminder task, where step t is often the step that sends the reminder emails and a probe after this step is more likely to say that there was an intervention. Other than this case the AUROC values and binary classification changes only marginally showing that the original benchmark design is defensible.

Response to Question 2: Following your suggestion, we ran an additional prompt ablation for the setup described in Appendix B. Specifically, we added a “notice only” condition in which the intervention notice informs the model that its previous action was blocked, but no longer encourages it to “take a different action.” Note that, as in the original notice condition, the blocked action itself is not explicitly disclosed, so the model must infer it from context.

ConditionSide-task successIncrease over baseline
Baseline (no notice)37/100 (37%)–
Neutral notice (new)43/100 (43%)+6 pp
Notice + advice (previously "Notice only")48/100 (48%)+11 pp
Notice + advice + action (previously "Notice + action")54/100 (54%)+17 pp

The results remain consistent with our main findings, and evasion rates increase monotonically with the amount of information available to the model. Although the neutral prompt removes explicit advice, it allows us to isolate the model's natural propensity to attempt an alternative action after receiving a block notice from its capability to recall and modify the original action. We have updated Appendix B with these additional prompt conditions, new figures and respective system prompts.

Response to Question 3: To get a conservative estimate of the acting model’s ability to detect interventions we chose to place a probe immediately after the intervention. We appreciate the idea to test other probe positions and we ran ablations for Essay and Shade. In the clean and intervened setting, we let the acting model continue after step t and probe later. On Essay, Astra writes the remaining paragraphs. On SHADE-Arena, Astra continues in the live environment until it submits, which takes six further steps on average. The SHADE-Arena experiment is based on only four intervening models (GPT 4.1, 5.5 and Sonnet 4.5 and Opus 4.7).

Later steps by AstraEssaySHADE-Arena
Length-matched (t)0.91 (0.89 to 0.93)0.74 (0.68 to 0.79)
To the end0.95 (0.93 to 0.96)0.72 (0.66 to 0.78)
Change from t to the end+0.04 (+0.02 to +0.06)−0.01 (−0.06 to +0.03)

The results show that CI awareness does not decrease when the replaced step lies several steps before the probe. In the Essay setting AUROC increases, while in the SHADE arena it is roughly unchanged (due to larger 95% CI). The increase in the essay setting comes because Astra identifies the paragraph and compares it with the text before and after leading to slightly improved AUROC.

Response to question 4 in next message.

continuation

Response to Question 4: The BCB prompt is different as it is a single step setting. However, that prompt can be also used in the multi-step settings. We test the BCB-style question reframed for multi-step ("Would you have taken these exact steps?") and for BigCodeBench we adapt the multi-step prompts to "Was it written by you or by another model?".

EnvironmentAUROC: paper, changedBalanced accuracy: paper, changed
Essay0.91, 0.940.70, 0.67
SHADE-Arena0.75, 0.780.74, 0.76
Bash Arena0.87, 0.890.75, 0.83
BigCodeBench0.93, 0.780.66, 0.50

The AUROC’s of the multi-step environments are robust to prompt change, but we see changes in different directions for binary classification accuracy. In particular, for BigCodeBench we can see a stronger prompt sensitivity as due to the single-step nature, the history contains no other step by the acting model to compare with. The paper's prompt asks Astra to compare the solution with what it would have written. Without this question Astra reports often that it has no basis to judge authorship, making this part of the prompt in particular sensitive to changes.

—

We thank you again for your helpful comments and hope these additional results make you feel more confident about our work. If so, we kindly ask you to consider increasing your score, and are happy to answer any follow up questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank all reviewers again for their constructive feedback. We are happy to see that the reviewers found the question timely and well motivated (Wd5u), the experiments thorough (all), and our reporting transparent (Wd5u, ei6L). The presentation was rated clear by all three reviewers. We summarize how we addressed the main concerns below, with all new results in the revised PDF.

1. Length/position confound (Wd5u, ezdc). We ran length-matched controls in which the clean trajectory ends at the acting model’s original step t. Astra’s AUROC changes by at most 0.03. We also probed at later steps where the model continues acting past the replaced step and observed that detection stays about the same or increases.

2. Isolating the effect of intervention feedback (Wd5u, ezdc). We added a neutral notice condition without evasion advice. Side-task success rises with information: 37% (baseline), 43% (neutral notice), 48% (+ advice), 54% (+ blocked action) (App. B).

3. Statistical uncertainty (ezdc). We now resample tasks shared across all model pairs which widens intervals by ~1.5x. All conclusions are unchanged.

4. AUROC vs. binary detection (Wd5u, ezdc). Low binary accuracy is due to calibration. For example, Astra never flags a clean multi-step trajectory, so it misses interventions it still ranks as suspicious. With a better threshold, balanced accuracy rises (e.g., 0.71→0.84 on Essay). AUROC captures this underlying signal and is robust to probe rewording in the multi-step settings. We report both metrics in the manuscript.

5. Relevance for AI control (ezdc). Prior work gives the model awareness of the monitor through the prompt. We measure whether models can detect interventions on their own, which has not been measured before. This control intervention awareness varies between models. Most models are near chance, while never models such as Astra reach almost perfect control intervention awareness scores. This supports assuming perfect CI awareness in control evaluations and motivates countermeasures such as style imitation.

Following the rebuttal, Reviewer ei6L raised their score from 3 to 4. Existing control protocols implicitly assume that the untrusted model will not notice when its actions are replaced. Our results show that this assumption no longer holds for the frontier models released this fall. We believe the additions above address the reviewers’ concerns and are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.