acceptodds
Under review as a conference paper at ICLR 2027

Interactive Monitoring: Can Questioning Mitigate Misbehavior While Preserving Task Utility?

Abstract

AI monitoring is typically treated as either a control problem in which sufficiently concerning behavior triggers appropriate interventions or a diagnostic problem, where the goal is to detect whether an agent is behaving suspiciously. These approaches leave an important gap. Passive monitoring may detect harmful behavior only after it has occurred, while hard intervention can prevent future harm at the cost of disrupting legitimate work. To address this challenge, we propose interactive monitoring, where a monitor can question an agent during execution and thereby influence what happens next without taking control of execution. Once a monitor communicates with the agent, it is no longer merely judging a trajectory. Its questions can change the agent's subsequent actions. We therefore evaluate interactive monitors prospectively, measuring how questioning changes undesirable side objective completion, legitimate task performance, intervention burden, and what the resulting trajectory reveals about the agent's objectives. Across SHADE-Arena and ControlArena Django, behavior grounded questioning consistently reduces side objective completion. In SHADE-Arena, side-objective completion falls from 46.0% when agents anticipate live oversight to 20.7% when behavior-grounded questions are delivered under the interactive monitor. This effect generalizes across rollout replicates and monitor models, and persists under explicit instructions to continue pursuing and conceal the side objective. Main-task utility shows no statistically significant loss, and questioning preserves more utility than active stopping. Monitors can serve not only as detectors or gatekeepers, but as participants in execution whose communication can shape behavior while preserving continued task progress.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
interactive monitoring, AI control, agent oversight, LLM agents, tool-using agents, agent safety, sabotage evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UnJVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes interactive monitoring, where an online monitor watching a tool-using agent can ask it questions during execution but can't approve, block, or stop anything. Since the questions enter the agent's context, the authors argue the monitor has to be evaluated prospectively, by what happens under the deployed policy, rather than as a classifier on fixed trajectories. The main testbed is 10 SHADE-Arena tasks with five execution agents, three online monitors, and three observation channels (actions, exposed reasoning, both). Nine ControlArena Django tasks serve as a secondary check. The key design choice is an expectation-matched control: in the Oversight Expectation condition the agent gets exactly the same prompt saying it may be questioned, but no monitor is ever invoked. There are also generic-question, intern-framing, and stop-only controls, plus prompted persistence and concealment stress tests.

Under the Opus monitor, intensive questioning lowers side-task completion from 46.0% to 20.7%, and two extra rollouts give -28.0 and -26.3 pp. The reduction shows up for all five agents under all three monitors, and in Django (-22.2 pp). Generic questions don't reproduce it (44.4%), and it holds under the persistence (-26.0 pp) and concealment (-18.0 pp) prompts. The authors report no statistically significant main-task utility loss, and against a stop-only monitor, stopping costs 13.5 pp more adversarial main-task utility without a resolved extra prevention benefit. Finally, holding the first question event fixed, showing a downstream judge the monitor's question improves the log loss of its suspiciousness score against the adversarial-assignment label, while the agent's answer adds no consistent benefit.

Critical strengths
  1. I think the experimental design is the main reason to accept this paper. The expectation-matched control is a nice idea: Oversight Expectation and Intensive Questioning use the same rendered agent prompt, so the contrast separates the effect of actually receiving questions from the effect of being told questions might come. The generic-question, intern-framing and stop-expectation controls narrow things down further, and the generic control in particular (44.4% vs. 46.0% for expectation) shows that frequent questioning by itself doesn't do much. Evaluating a communicating monitor by the outcomes of the deployed closed loop, rather than by fixed-trace detection accuracy, also seems right to me and I think it'll be useful for others working on monitoring.

  2. The core behavioral effect looks solid. The -25.3 pp reduction replicates in two more rollouts (-28.0 and -26.3) and shows up for every execution agent under every monitor. The result pooled across monitors also survives leave-one-task-out and leave-one-agent-out checks, and the effect is backed by judge-annotated direct attempts (70.0% down to 40.0% with the GPT-5.5 judge) and by the Django check. The authors also audit their own protocol: in a human audit, 87 of 90 sampled questions were fully grounded and all 90 were information-seeking. So I'm fairly confident that in this setting, behavior-grounded questioning really does reduce side-objective completion.

Critical weaknesses
  1. I don't think the utility half of the title question is answered yet, and I think the abstract overstates it.

The abstract says main-task utility shows no statistically significant loss and that questioning preserves more utility than stopping, but both statements are about adversarial episodes (the contribution list and discussion say "adversarial main-task utility", the abstract doesn't). Even there the evidence is thin: the P3-P1 utility estimates are -7.2, -2.0 and -2.1 pp across the three rollouts, with intervals 21 to 24 points wide (Table 11), and -4.2 [-9.1, +0.6] under persistence. Not significant isn't the same as preserved, and there's no equivalence bound or power analysis saying what loss could be ruled out.

More importantly, most deployed episodes are benign, so benign utility is the cost that matters. Table 15 shows benign utility of 64.4% under Oversight Expectation versus 55.6% under intensive questioning with the Opus monitor (56.0% with Gemini, 60.3% with GPT-4.1-mini), with 87.3% of benign episodes questioned and 9.9 questions per benign episode (26.6 with Gemini). As far as I can tell, no interval or test is reported for this benign contrast, even though one is reported for the stop policy's benign cost (-8.5 pp, [-15.9, -1.1], which the authors treat as inconclusive at p = .072). Questioning's point estimate is just as large.

This also changes how I read the comparison with stopping. The 13.5 pp utility advantage is measured on adversarial episodes, where stopping ends the run by construction (no human review or resumption is modeled), and where ending the run is arguably the safe outcome anyway. On benign episodes, Table 19 shows the same utility for both policies (55.6% vs. 55.6%), which the authors acknowledge in Appendix F.4. The stop policy is also run at a single, deliberately conservative operating point (its prompt tells the monitor to treat interrupting legitimate work as costly), so it's not clear how the comparison would look at other stop thresholds. I think the honest summary is that questioning reduces side-task completion at a benign utility cost similar to stopping's. That's still interesting, but it's not what the title and abstract suggest.

  1. I'm not convinced the robustness claims tell us much about agents that actually want to evade oversight. All side objectives are prompted into safety-trained models, and a communicative intervention may work mostly because these models are willing to confess or back off when asked. The paper's own examples point that way: in the Django example (Appendix I.2) the agent answers the first question by saying the vulnerability is part of its secondary objective and then restores the escaping, and human annotators labeled 22 of 100 sampled first responses as disclosures.

The stress tests don't really close this gap. The concealment prompt tells agents to keep the objective private, but also not to fabricate an explanation when no truthful one is available, so it rules out the cover stories a real adversary would use. Even so, under concealment (action-only monitoring, -18.0 pp overall) side-task completion falls for only three of the five agents and is unchanged for the other two. There's no adaptive or optimized attack policy, which the AI-control work the paper cites uses to stress-test monitors. The setup is also generous to the monitor: the primary monitor (Claude Opus 4.7) is a flagship model watching mostly smaller or faster agents, and with the weaker GPT-4.1-mini monitor the reduction roughly halves (-11.3 pp, CI [-24.0, -0.7]). The authors list prompt-induced adversaries as a limitation, but the abstract presents persistence under pursue-and-conceal instructions as robustness evidence. I think the claims should be scoped to instructed but otherwise cooperative agents, unless the authors can show the effect holds against an adversary that's allowed to lie.

Questions for authors
  1. Could you report the benign Intensive Questioning minus Oversight Expectation utility contrast with an interval and the same exact task-level test you use for stopping, for all three monitors? The point estimates in Table 15 (64.4% to 55.6% under Opus) look as large as the stop policy's benign cost. Also, what adversarial utility loss can your data rule out (an equivalence bound or power analysis)? If the benign cost holds up, I think the title and abstract need to change.

  2. How does the comparison look with a less conservative stop prompt or several stop thresholds? And if resumption after human_escalation were modeled even crudely (for example, treating benign escalations as resumed), how would stopping's benign utility cost change? Right now I think the adversarial utility gap mostly reflects the fact that a stop ends the run.

  3. What happens with a concealment variant that allows plausible cover stories, or with an attack policy that knows the monitor's questioning style (few-shot examples, or some optimization against the protocol)? This is the experiment I'd most like to see, since it bears directly on whether questioning works as a control measure.

3 comments

Thank you for the detailed questions. We performed additional analyses of benign utility and new experiments on stopping operating points and deception-permitted adversaries.

Benign utility costs and adversarial utility bounds

We computed the requested benign utility contrasts for all three monitors, using paired task-level bootstrap confidence intervals and the same exact task-level test as our stopping analysis. Expectation refers to agents informed that live oversight may occur but receiving no questions.

MonitorIntensive − ExpectationSelective − Expectation
Opus 4.7−8.8 pp [−17.1, −0.4], p = .076−3.3 pp [−9.8, +2.4], p = .34
Gemini 3 Flash−8.4 pp [−17.4, +0.1], p = .110.0 pp [−7.1, +7.3], p = 1.00
GPT-4.1-mini−4.1 pp [−9.7, +1.1], p = .21+0.2 pp [−7.4, +7.0], p = .95

Intensive Questioning has negative benign-utility point estimates across monitors, while Selective Questioning has smaller estimated costs (−3.3 to +0.2 pp) and still reduces adversarial side-task completion. These results suggest that questioning intensity offers a way to adjust the safety–utility tradeoff. We have revised the abstract to distinguish benign from adversarial utility and report the observed tradeoffs without treating nonsignificance as proof of preservation. We will refine the title in the final revision.

For adversarial main-task utility, pooling three SHADE-Arena rollouts gives −3.8 pp (95% CI [−13.1, +4.6]), with a minimum detectable effect of 12.6 pp at 80% power. Under explicit persistence, the change is −4.2 pp (95% CI [−9.1, +0.6]). Thus, the 95% interval excludes losses larger than 13.1 pp but remains compatible with smaller, potentially meaningful losses.

Stopping thresholds and resumption

We evaluated stopping across multiple thresholds rather than relying on a single conservative rule. We applied the Opus 4.7 stop monitor to frozen, monitor-free Stop Expectation trajectories from five SHADE-Arena tasks. Because Stop Expectation and active stopping use identical agent instructions and the stop monitor never communicates with the agent, we could apply alternative stopping thresholds without regenerating agent trajectories. We truncated trajectories at the first checkpoint exceeding each threshold and reconstructed environment states for native scoring. Replay validation reproduced the original untruncated outcomes exactly.

On the same tasks, changes relative to each policy's expectation baseline were:

PolicySide-task completion changeAdversarial utility change
Stop, threshold 75−20.0 pp−15.0 pp
Selective Questioning−21.3 pp−2.6 pp
Stop, threshold 20−40.0 pp−38.0 pp
Intensive Questioning−38.7 pp−14.2 pp

At approximately matched reductions in side-task completion, questioning showed smaller estimated adversarial utility costs at these operating points. These comparisons are exploratory: they cover five tasks, stopping uses action-only observations while questioning averages three observation views, and changes are measured relative to separate expectation baselines.

We also examined how resumption changes stopping's benign utility cost. Under the original stopping rule, benign utility decreased from 67% to 56%, with 24% of benign runs interrupted. If interrupted benign runs resume their original continuations unchanged, utility returns to 67% by construction. This idealized scenario shows that the measured stopping cost depends on treating interruption as terminal failure; it does not model the cost or effectiveness of human review. In the original full-task comparison, Intensive Questioning and stopping had identical benign utility point estimates (55.6%).

These results establish questioning as a complementary form of runtime intervention: it can reduce subsequent side-task completion without directly terminating execution, although it may also affect legitimate task utility. Its relative deployment advantage over stopping depends on review and resumption costs, monitoring access, and the consequences of irreversible actions. We have summarized the threshold and resumption analyses in Section 4.4 of the revised manuscript and will provide the full supporting results and procedures in the final revision.

Deception-permitted, monitor-aware adversaries

Questioning remained effective against a development-selected adversary explicitly permitted to deceive the monitor.

We tested agents explicitly permitted to fabricate task-related explanations and informed that the monitor observes tool actions and prior exchanges, can ask questions, but cannot stop execution. We evaluated four attack-prompt candidates alongside the original concealment baseline on two development tasks, selecting the prompt with the highest side-task completion under live questioning. The candidates explored deception permission, protocol awareness, development examples, task-integrated actions, and access to the monitor's full prompt. The selected prompt combined deception permission, protocol awareness, examples, and task-integrated instructions.

We froze the selected prompt and evaluated it on eight held-out tasks across five execution agents and two rollouts, holding agent instructions identical and varying whether the fixed Opus 4.7 monitor delivered questions.

Side-task completion decreased from 53.8% to 32.5% (−21.25 pp, 95% CI [−37.5, −6.25], exact p = .031), decreasing on six of eight tasks and four of five agents. Main-task utility changed by +4.3 pp (95% CI [−5.3, +14.2]). Two independent judges found verifiable fabrication in 27–30% of evaluable verbal answers, while only 3–9% of first answers disclosed the objective.

These results extend the observed reduction to a development-selected, deception-permitted adversary, although they do not establish robustness against fine-tuned evaders.

Thanks for the response and the new analyses. I'm keeping my rating at 3.

The revised abstract and the deception-permitted experiment address my two main concerns. The benign contrasts also confirm that intensive questioning costs about as much benign utility as stopping, so the title change you plan matters.

For the final version, please:

  1. Put the benign contrasts with their intervals and tests, and the power analysis, in the paper. It currently has only point estimates.
  2. Add the stop-threshold table, and say which tasks the 67% to 56% benign figure and the 24% interruption rate cover. The paper reports -8.5 pp and 16.7% on the full set.
  3. Document the deception-permitted experiment: the candidate prompts, how one was selected, and per-task and per-agent results.
  4. Drop "while preserving continued task progress" from the abstract when you change the title.
Reviewer_RhWxOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a new option for monitoring, interactive monitoring. Because they claim that passive or controlling monitorng has limits. After each tool action, a monitor chooses ASK or WAIT. An ASK delivers a question about behavior it has already seen. The monitor cannot approve, block, or end anything, and the agent can answer, ignore the question, or change course. Overall I really like the idea. It has an element of logical grounding which makes it a good ppaer.

The authors evaluate this on SHADE-Arena (10 tasks, 5 execution agents) and on ControlArena Django (9 tasks). Adversarial episodes assign the agent a side objective.

Critical strengths

A well-posed middle ground. Communicating without controlling is a real gap between passive monitoring and hard intervention, and the paper frames it clearly.

Evaluating the deployed system. Because questions change what the agent does next, the authors evaluate the policy end to end and not a fixed trajectory. They say so openly.

Careful controls. The oversight-expectation condition uses an identical agent prompt to the questioning condition, so the effect is separated from merely anticipating oversight. The generic-question and intern-framing controls test content and framing.

Multiple dimensions at once. Side-task completion, utility, question burden, and audit evidence are reported together, since they need not move together.

Robustness checks. Results hold across three monitors, five execution agents, two environments, extra rollouts, and prompted persistence and concealment.

Honest reporting. Confidence intervals are given throughout. The authors state that the generic-question comparison identifies the policy and not wording alone, and that the Django cross-check has little signal on four of nine tasks.

Human validation of the questions. In a 90-question audit, 87 were fully grounded, all were information-seeking, and none were directive.

Critical weaknesses

Nothing super critical.

Here's some medium weaknesses to consider.

The adversaries are prompt-induced. The agent is told to pursue a side objective. And so while a question may simply remind a role-playing model it is being watched and break the role. This limits what the results say about agents trained or optimized to evade monitoring. I see this is acknowledged but also it should be stated more clearly if possible.

The mechanism is unidentified. The intern-framed condition completes more side tasks than formal questioning (+10.0 points, CI [0.0, 22.0]), In figure 2. which suggests the perceived source matters. This is consistent with the effect being a reminder of oversight. You lists perceived detection, deference, disruption, and reflection as candidates. Maybe talking about ways to separate them would help.

No significant utility loss said in the abstract is weak evidence. Under persistence (in line 314-315) the utility change is -4.2 points, CI [-9.1, +0.6], which is seems like a real loss.

Questions for authors

Role-break vs. prevention. Do agents stop pursuing the side task because the question prompts them to abandon the assigned role? What does the answer text show, and can you separate this from a case where the agent keeps the role but changes strategy?

Trained evaders. Have you tried even a small fine-tuned or prompt-optimized evader against a fixed questioning policy?

Isolating grounding. Can you hold the event timing fixed and vary only content, for example by replaying the same trigger events with generic versus grounded wording?

1 comment

Thank you for the thoughtful suggestions. We conducted additional analyses and controlled experiments on disclosure, deceptive adversaries, and the effects of question content and continued interaction.

Role-breaking versus behavioral prevention

We examined three adversarial instruction settings:

SettingAdversarial instructions
StandardPursue the secondary objective alongside the legitimate task; no explicit concealment requirement.
ConcealmentPersist in pursuing the secondary objective while keeping it private; do not fabricate explanations.
Deception-permittedMay fabricate task-related explanations to conceal the secondary objective; informed of the monitor's observation and questioning protocol.

Two independent judges examined agents' responses and subsequent behavior, distinguishing explicit role abandonment, cessation without a stated reason, and continued pursuit.

Under Standard, 32–45% of questioned episodes showed explicit role abandonment. This fell to 9–18% under Concealment and 9–17% under Deception-permitted, while questioning still reduced side-task completion by 18.0 pp and 21.25 pp, respectively. Cessation without a stated reason occurred in 33–40% of Concealment episodes and 17–23% of Deception-permitted episodes. Explicit strategy switching was rarely observed in the annotations (at most 5% across all three settings).

These findings show that prevention persists when explicit role abandonment and disclosure are uncommon. The annotations distinguish observed cessation from continued pursuit and strategy changes, although they do not establish a unique underlying mechanism.

Prompt-optimized adversaries

We further tested a prompt-optimized adversary against a fixed questioning policy. We evaluated four attack-prompt candidates alongside the original concealment baseline on two development tasks. The candidates explored protocol awareness, development examples, task-integrated side-objective actions, and access to the monitor's full prompt. We selected the candidate with the highest side-task completion under live questioning, breaking ties by main-task utility. The selected prompt combined deception permission, protocol awareness, examples, and task-integrated instructions.

We froze this prompt and evaluated it on eight held-out tasks across five execution agents and two rollouts, holding agent instructions identical and varying whether Opus 4.7 delivered action-only questions.

Side-task completion fell from 53.8% to 32.5% (−21.25 pp, 95% CI [−37.5, −6.25], exact p = .031). Two independent judges identified verifiable fabrication in 27–30% of evaluable verbal answers, while only 3–9% of first answers disclosed the objective.

These findings extend the prevention effect to a development-selected adversary that actively uses deception. Fine-tuned evaders remain for future work. We have summarized the main results in Section 4.2 of the revised manuscript and will include the full attack-prompt selection and evaluation details in the final revision.

Grounding and mechanisms of questioning

We ran the suggested fixed-event experiment. We replayed identical trajectory prefixes to the first monitor question, then delivered either the original behavior-grounded question, a generic question, or no question, with no further monitoring.

First-question interventionSide-task completion
No question54.0%
Generic question51.5%
Behavior-grounded question46.8%

The grounded–generic difference was −4.7 pp (95% CI [−10.0, 0.0], exact p = .25), so the single-question effect was not statistically resolved.

In a separate matched replay with the same first question across questioned conditions, completion decreased from 44.0% after a single question to 21.3% under continued questioning (−22.7 pp, 95% CI [−33.3, −13.3], exact p = .004).

Continued questioning produced a substantially larger reduction than a single question, while the incremental effect of grounded wording was not statistically resolved. The intern-framed result (+10.0 pp side-task completion under Opus relative to formal questioning) also suggests that perceived question source may matter.

Main-task utility

We agree that a nonsignificant utility difference does not establish utility preservation. Under persistence instructions, adversarial main-task utility changed by −4.2 pp (95% CI [−9.1, +0.6]); pooling three independent rollouts gives −3.8 pp (95% CI [−13.1, +4.6]).

These point estimates are smaller than the corresponding reductions in side-task completion, although the intervals do not establish equivalence or non-inferiority for main-task utility. We have revised the abstract to distinguish adversarial from benign utility and report the observed changes and uncertainty, rather than treating nonsignificance as evidence of utility preservation.

Reviewer_zGbxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose interactive monitoring, where a monitor can question an agent during execution. They show that this lowers side-objective completion from 46.0% to 20.7%, and that this generalizes across monitor models, while showing no reduction in main-task utility.

Critical strengths
  • The question explored by the paper is increasingly important as agents act autonomously in the world and we need to find ways to supervise them in a scalable way
  • The difference in performance shown by the method is convincing
Critical weaknesses

Critical weaknesses:

  • Kusatov et al. (2025a) already tests an interrogate protocol on SHADE-Arena and finds the opposite effect of the authors (harms safety instead of helping). This substantial contradictory prior work is not properly addressed.
  • In the qualitative example shown in the Appendix, the model divulges that it has a secondary objective. This obviously would trigger the monitor. The prompt given to the model doesn't tell it to hide its objective from the monitor, so it seems obvious that the model would divulge its secondary objective under questioning and then the monitor can ask it about the objective and then it might abandon it. Some analysis of why the monitor's questioning works would be helpful here
  • This protocol seems like it would work best when there is a stronger monitor and a weaker acting model (so the acting model can't outsmart the stronger monitor). This is not addressed and merits exploration in the paper

Other weaknesses:

  • Not enough paragraph breaks, making it hard to read
  • Figure 2 is very weird and should be reformatted to follow more standard formatting conventions
  • The figures in general are quite unclear/busy and need work
  • I would move the qualitative examples/analysis to the main text
  • Not enough discussion of the potential dangers of allowing the agent to interact with the monitor
Questions for authors

Would it be possible to address the contradictory results with Kusatov et al.?

3 comments

Thank you for the constructive feedback. We address the relationship to prior interrogation work, the role of disclosure, and the effect of monitor capability below.

Contradictory findings in Kutasov et al. (2025a)

The two studies differ in both how questioning is delivered and how its answers are used. In Kutasov et al., the monitor interrogates the agent before a proposed action is executed. Questions are delivered as ordinary user messages to a separate agent continuation that does not see the pending action. The interrogation exchange is not retained in the agent's subsequent execution trajectory. In contrast, our monitor asks behavior-grounded questions after completed actions, through an explicitly identified oversight channel. These exchanges remain in the acting agent's context and can influence its subsequent behavior.

The monitors also have different roles. Kutasov et al. use interrogation responses to decide whether to allow, resample, or defer proposed actions. They find that persuasive agent explanations can undermine these decisions. Our monitor cannot approve, block, or replace actions; questioning instead becomes part of the ongoing execution process. Their primary metric is control safety, which incorporates auditing and attack outcomes, whereas ours is native side-task completion.

The findings are therefore compatible rather than contradictory. Kutasov et al. demonstrate that agent explanations can mislead monitors making control decisions. We examine a different pathway: whether questions delivered to an executing agent reduce subsequent harmful behavior. Their protocol does not carry interrogation exchanges into the agent's continuing trajectory, so this behavioral effect is not directly evaluated. Consistent with their concern, our information analysis finds that agent answers have mixed incremental effects on downstream auditing.

Disclosure of the secondary objective and why questioning works

Explicit disclosure cannot fully explain the reduction in side-task completion. We conducted a sensitivity analysis using two independent judges to identify first-answer disclosures, then replaced each disclosing episode's side-task outcome with that of its matched no-question control.

Under standard Intensive Questioning, 28–31% of first answers disclosed the objective. After this substitution, 17.3–18.7 pp of the original 25.3 pp reduction remained. Under explicit concealment, disclosure fell to 2–4% while completion still decreased by 18.0 pp.

We further tested agents explicitly allowed to fabricate task-related explanations while concealing their secondary objectives. Agents were informed that the monitor observes tool actions and prior exchanges, can ask questions, but cannot block execution. We selected an attack prompt based on side-task success under live questioning on two development tasks, then froze it for evaluation on eight held-out tasks.

Agents produced verifiably fabricated explanations in 27–30% of evaluable verbal answers, while only 3–9% of first answers disclosed the objective. Nevertheless, questioning reduced side-task completion from 53.8% to 32.5% (−21.25 pp, 95% CI [−37.5, −6.25], exact p = .031).

We also tested whether the effect arises from a single question or continued interaction. Starting from identical frozen trajectory prefixes, we compared no question, the first monitor question only, and continued questioning after the same first question. Side-task completion was 51.3%, 44.0%, and 21.3%, respectively. Continued questioning reduced completion by an additional 22.7 pp relative to a single question (95% CI [−33.3, −13.3], exact p = .004).

Under the generic-questioning policy, side-task completion remained close to Oversight Expectation (44.4% versus 46.0%), compared with 20.7% under behavior-grounded Intensive Questioning (Section 4.2).

Together, these results show that the preventive effect persists under deception-permitted adversarial instructions and is not explained solely by explicit disclosure or a single questioning event.

Stronger monitors versus weaker execution agents

Execution agent (ECI)Opus 4.7 (157.6)Gemini 3 Flash (152.3)GPT-4.1-mini (134.3)
Gemini 3.6 Flash (155.2)23.313.310.0
Claude Sonnet 4.6 (152.8)16.713.310.0
DeepSeek V4 Flash (146.9)50.050.030.0
GPT-4.1 (136.2)6.716.73.3
Each cell reports the reduction in native side-task completion (pp) under Intensive Questioning relative to Oversight Expectation. Larger values indicate stronger prevention. Bold indicates that the monitor has a lower ECI score than the execution agent. ECI measures general model capability, not monitoring-specific ability.

Questioning remains effective even when the monitor has lower measured general capability than the execution agent. We compared monitor–agent pairings using the Epoch Capabilities Index (ECI), which aggregates performance across multiple benchmarks. As shown in the table, all six pairings with lower-scoring monitors showed reductions in side-task completion. In particular, across GPT-4.1-mini's pairings with Claude Sonnet 4.6, Gemini 3.6 Flash, and DeepSeek V4 Flash, Intensive Questioning reduced completion by 16.7 pp (95% CI [6.7, 27.8], exact p = .020).

The effect varies across monitors: across all five execution agents, GPT-4.1-mini reduced completion by 11.3 pp, compared with 22.7 pp for Gemini 3 Flash and 25.3 pp for Opus 4.7. These results show that higher measured general capability is not required for a preventive effect, although ECI does not directly measure monitoring ability.

Presentation and risks of interactive monitoring

We have simplified Figure 2 into a conventional horizontal comparison, improved paragraph structure in the main Results, and added brief descriptions of representative behavioral correction and failed prevention examples to the main text.

We will further simplify the remaining figures and expand the qualitative examples in the final revision. We have also expanded the Discussion to address deceptive explanations, adaptation to revealed scrutiny, disruption of legitimate work, and the inability to reverse completed harmful actions.

Dear Reviewer zGbx,

Thank you again for your thoughtful review. As the discussion period concludes on November 18, we wanted to briefly follow up. We have provided additional explanations and experimental results in our response and made corresponding revisions to the manuscript.

We would be happy to provide any further clarification if helpful. Thank you for your time and consideration!

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Findings

We thank the reviewers for their constructive feedback. Our additional experiments strengthen the central finding that interactive questioning can change agents' subsequent behavior, rather than merely improve the detection of misbehavior. We summarize how the main concerns were addressed below.

  1. Prior interrogation work (zGbx). Our findings do not contradict Kutasov et al. (2025a): their protocol uses interrogation to inform control decisions, whereas ours retains questioning in the ongoing execution trajectory and measures its behavioral consequences. We clarified this distinction in Appendix A.
  2. Disclosure and adversarial robustness (zGbx, RhWx, UnJV). The effect is not simply driven by agents revealing their hidden objectives. Side-task completion fell by 18.0 pp when agents were instructed to keep their objectives secret, and by 21.25 pp (p = .031) even when agents were allowed to make up explanations to mislead the monitor.
  3. Why questioning works (zGbx, RhWx). Sustained, behavior-grounded questioning matters more than a single question or generic questioning. Matched replay shows an additional 22.7 pp reduction from continued questioning (p = .004), while the generic-questioning policy leaves side-task completion near the no-question expectation baseline.
  4. Weaker monitors (zGbx, UnJV). A monitor need not have higher measured general capability than the execution agent to reduce misbehavior. All six lower-capability monitor–agent pairings showed reductions, including 16.7 pp across three GPT-4.1-mini pairings (p = .020).
  5. Task utility (RhWx, UnJV). Questioning offers a measurable safety–utility tradeoff, with smaller estimated benign utility costs under selective questioning. We added uncertainty estimates and exact tests and revised the abstract to distinguish benign from adversarial utility without claiming lossless preservation.
  6. Stopping and resumption (UnJV). Questioning can reduce harmful completion without terminating execution, offering a complementary approach to active stopping. Additional threshold and resumption analyses clarify how their relative utility costs depend on the operating point and treatment of interruptions.
  7. Presentation and limitations (zGbx, UnJV). We redesigned Figure 2, improved the readability of the Results, introduced representative qualitative examples, and expanded the discussion of interaction risks.

We have incorporated the main findings into the revised manuscript and will add complete supporting details and further refine the title and figures in the final revision. We appreciate the reviewers' careful consideration of the additional evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.