acceptodds
Under review as a conference paper at ICLR 2027

ORBIT: Online Regret-Bounded Intervention for Tool-Using Agents

Abstract

Tool-using language-model agents can cause harm through a sequence of actions, even when no single action looks suspicious. Some recent defenses constrain agents to pre-specified or permitted execution paths. We introduce ORBIT (Online Regret-Bounded Intervention for Tool-Using Agents) that instead learns when to intervene from evidence accumulated across an agent's actions, without requiring a complete execution plan in advance. It fits an allow-inspect-block policy over calibrated signals from several monitors, including two LLM critics from different model families. ORBIT can inspect to gather more evidence before committing, deferring the allow/block choice at ambiguous steps, building evidence memory across connected episodes from beliefs about persistent compromise, current deviation, and progress toward harm. Our experiments show that ORBIT prevents harm under varying levels of evidence restriction and under adaptive attack sequences. In action-branch replay, it stops markedly more attacks early than DBN posterior thresholding at similar benign cost. On fixed AgentDojo Workspace recordings from live OpenAI and Anthropic executions, carrying evidence across episodes improves mean prevention over episode-reset controls, at the cost of interrupting more legitimate work; the advantage is largest when observation is sparse and the taint channel is removed. Under adaptive attacks that revise the payload across successive attempts, persistent memory lowers harmful goal completion relative to resetting memory each episode. An exploratory study also finds that updating beliefs from modeled monitors' replies to intervention recommendations increases mean early stopping, at higher benign cost.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, game theory, Bayesian methods, Online learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_reYPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ORBIT learns intervention policies for tool-using agents using evidence accumulated across actions and episodes. It combines Bayesian beliefs with an offline-fitted policy, updates beliefs online, and models monitor incentives through obedience constraints on report deviations. Branch replay evaluates allow, inspect, and block decisions; AgentDojo evaluates allow and campaign-ending stop decisions. The paper provides conditional theoretical guarantees, comparisons with threshold and direct policies, memory ablations, adaptive attacks, and modeled monitor-response experiments. The results support inspection benefits and memory-dependent prevention gains, with important costs in denied legitimate work.

Critical strengths

S1. Meaningful evidence for inspection at comparable benign cost.

In Table 5, ORBIT-postrule achieves 37.1% early stopping at 19.35 weighted benign interruptions per 1,000 original benign decisions, compared with 18.5% at 20.71 for DBN thresholding. The expanded six-seed experiment also has favorable mean early stopping and benign cost (Table 14). These results provide evidence beyond simply interrupting more activity. The targeted inspection-evidence concealment study usefully identifies conditions under which this benefit deteriorates.

S2. Useful controls and careful distinctions between guarantees and measurements.

The evaluation includes full-memory Direct and threshold controls, multiple calibration allowances, paired uncertainty, and adaptive attacks reexecuted with task-checker-confirmed harm. Keeping canceled legitimate work in the cost denominator is appropriate. The η=0.1 audit establishes that obedience can exclude otherwise realizable action policies, while the authors correctly refrain from claiming a reliable performance improvement. The theoretical guarantees state their assumptions, and the manuscript acknowledges limitations of the recording-based coverage diagnostic.

Critical weaknesses

W1. The testbed construction and monitor-information assumptions limit the scope and reproducibility of the empirical conclusions.

Appendix H.1 refers to accepted cached replay corpora without sufficiently describing their generation and acceptance. The released evaluator requires node AUC ≤ .60, single-edge AUC ≤ .70, and connected-model AUC between .80 and .95. The node and connected checks score the corpus test partition. These checks enforce a weak-local/strong-connected-evidence regime, relevant to a method that accumulates evidence. The mechanism label prioritizes active_deviation, and the connected model uses DBN machinery, so these criteria alone do not determine ORBIT's advantage over DBN. Appendix J also identifies cached recordings and fitted artifacts that are required for reproduction but omitted from the source release.

In AgentDojo, the released runner obtains the permission monitor's allowed tools from reference-solution tool usage and matches taint against actual injection text. Deployment may provide different information. The existing both-channels-disabled ablations retain some memory gains and are useful evidence for scoping this assumption.

Q1 — Clarification of the existing protocol: Please describe corpus generation, when acceptance criteria were fixed, and how acceptance was separated from final policy evaluation; provide the frozen inputs or an auditable reconstruction procedure. Please also state the benchmark-derived monitor inputs and the deployment scope supported by the existing ablations.

W2. Persistent memory's benefit needs clearer characterization at comparable achieved legitimate-work cost.

At 5% observation without taint, prevention increases from 13.33% with episode reset to 51.33% with full memory, while legitimate work denied rises from 11.55% to 45.01% (Table 2). At adaptive search budget 16, harmful completion falls from 18.50% to 7.00%, with denial increasing from 10.99% to 43.96% (Table 7). The authors acknowledge these costs and provide Direct controls and tighter allowances. However, a common calibration allowance does not establish a favorable comparison at the same achieved cost; at the 20% allowance, memory's mean prevention advantage over episode reset appears in only three of six settings, with all corresponding paired intervals including zero.

Q2 — Additional analysis relevant to the memory claim: Please compare the existing ORBIT, episode-reset, Direct, and threshold policy families over overlapping achieved-cost ranges using operating points selected without test feedback, or limit the memory claim to the demonstrated operating points. The adaptive result remains a useful overall-system comparison; campaign stops also reduce attacker feedback, as the paper states. This concern is specific to memory, while branch replay already provides positive inspection results at approximately comparable benign cost.

Overall assessment. I lean toward weak rejection because the testbed and information assumptions need clearer substantiation, and the persistent-memory evidence does not yet establish its favorable achieved-cost regime sufficiently. The inspection result and experimental controls are substantive strengths. Clearer construction provenance, appropriately scoped deployment claims, and the targeted trade-off analysis could materially change this borderline assessment.

Questions for authors

See weaknesses

1 comment

We thank the reviewer for the careful assessment and for recognizing the inspection results, experimental controls, and distinction between theoretical guarantees and empirical findings.

W1 (testbed construction): yes the replay corpus is conditioned on a weak-local/strong-connected-evidence regime but corpus acceptance and intervention-policy selection are distinct. Policies are trained on training data and selected using validation or calibration data, without using test outcomes for policy selection. However, the evaluator's use of test-partition AUC for corpus acceptance limits claims of unconditional testbed generalization. We acknowledge this distinction. In AgentDojo, permission and taint monitoring use benchmark-derived information; the ablations removing both channels provide evidence under restricted monitoring, but do not establish deployment-wide generalization. Appendix J also acknowledges the reproducibility limitations of the current release.

W2 (memory and benign cost): also agree that equal calibration limits do not imply equal achieved interruption costs. Our claim is that persistent memory improves prevention at the reported operating points, not that it universally dominates episode-reset policies at equal benign cost. Section 3.4, Tables 2 and 11, and Appendix F explicitly document the associated disruption and weaker results under tighter limits.

We appreciate the reviewer's recognition that the inspection results demonstrate meaningful benefits at comparable benign cost, independently of the persistent-memory trade-off.

Reviewer_15tGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ORBIT, a framework for monitoring and intervening in tool-using LLM agents. Instead of evaluating individual actions independently, ORBIT accumulates evidence across agent interactions and learns when to allow, inspect, or block potentially harmful actions. The framework integrates multiple security monitors, Bayesian belief updates, and a game-theoretic mediator to optimize intervention decisions while limiting disruptions to legitimate tasks. The authors evaluate ORBIT using branch-replay experiments and AgentDojo, showing improvements in early attack prevention and the benefits of persistent monitoring memory.

Critical strengths

Important research problem. The paper addresses security monitoring for tool-using LLM agents, particularly attacks involving multiple seemingly benign actions.

Interesting framework. Combining persistent monitoring memory with adaptive intervention decisions is promising, and the experiments demonstrate potential benefits for early attack prevention.

Critical weaknesses

Insufficient comparison with existing approaches. The paper lacks comprehensive comparisons with recent agent-security defenses and does not convincingly demonstrate the advantages of its game-theoretic formulation over simpler monitoring and intervention strategies.

Limited experimental validation. The realistic agent evaluations focus primarily on AgentDojo Workspace, with a relatively small set of monitor components. The experiments do not sufficiently establish generalizability across different agent architectures, applications, or attack scenarios.

Questions for authors

I am not fully convinced about the technical advantages of the proposed game-theoretic mediator. The framework introduces considerable complexity, but the experiments do not clearly demonstrate why this design is necessary compared with existing agent-monitoring approaches. Could the authors provide additional comparisons with recent defenses, particularly methods that use sequential monitoring or multiple security signals?

The experimental evaluation seems relatively limited. Although the authors conduct branch-replay experiments and use AgentDojo, the agent-based evaluation primarily focuses on the Workspace environment. Could the authors extend the evaluation to other AgentDojo environments or additional agent frameworks? I am particularly interested in whether the proposed method generalizes to different tool configurations, agent architectures, and more complex tasks.

The framework relies on five security monitors, including two LLM critics. How sensitive is ORBIT to the number and diversity of monitors? Would the proposed approach remain effective with fewer monitors, different LLM families, or larger multi-agent systems?

The reported improvements in attack prevention sometimes come with substantial disruption to legitimate tasks. For example, Table 2 shows that improved prevention can coincide with denying approximately 45% of legitimate work. Could the authors provide stronger evidence that ORBIT offers a better security-utility trade-off than existing approaches at comparable false-positive or interruption rates?

1 comment

We thank the reviewer for recognizing the importance of sequential agent-security monitoring and the potential benefits of persistent evidence and adaptive intervention.

Game-theoretic contribution: ORBIT's mediator learns intervention recommendations subject to constraints on modeled monitor incentives. Section 2.2 explains that, with deterministic median aggregation, a direct intervention policy can be represented through unanimous recommendations. The distinction becomes substantive when individual monitor reports can influence execution. In Section 3.2 and Table 9, using η = 0.1, all four obedient policies satisfy the fitted constraints, whereas none of the four unconstrained policies' action distributions can be reproduced by an obedient recommendation distribution. This establishes an effect on the feasible policy set, not a statistically reliable performance improvement from obedience alone.

Intervention effectiveness: Separately, Table 5 reports 37.1% early stopping for ORBIT-postrule versus 18.5% for DBN thresholding, at benign costs of 19.35 versus 20.71 FI/1k. This comparison supports the value of learned inspection decisions, rather than isolating the game-theoretic contribution. The paper also includes memoryless, lagged, edge-based, direct-policy, and memory-reset controls.

Generalization and monitor diversity: We acknowledge that the AgentDojo evaluation focuses on Workspace and does not establish generalization to other agent environments or larger monitor ensembles. The taint and permission-channel ablations examine restricted monitoring configurations, but do not constitute a systematic study of monitor count or model-family diversity. These remain important directions for future evaluation.

Reviewer_Q73aOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates the safety tool use detection as a Bayesian Correlated Equilibrium. It proposes to use interventions from accumulated evidence, valuing both the current action and later opportunities to prevent harm, instead of only modelling it as a binary decision. The Inspect process provides additional gains to detect harm and avoids false positive. the authors also provided through bounds and proofs from this probabilistic model formulation.

Critical strengths

the idea of inspecting before the action actually happens is very novel and useful

instead of binary classification, the value estimate for inspect process is insightful

the Bayesian Correlated Equilibrium formulation is sound

Critical weaknesses
  1. there is some cost "This improvement has a substantial cost: benign work denied rises from 11.55% to 45.01%"

  2. the overall system is a bit complicated, and it's not sure what's the cost vs. gains

  3. there are many hyperparameters, which might lead to sensitivity of this method's practical usability.

Questions for authors

how to formulate persistent memory across episodes?

1 comment

We thank the reviewer for recognizing the novelty of inspection, its value in sequential intervention, and the soundness of the Bayesian correlated equilibrium formulation.

Persistent memory across episodes: In the AgentDojo experiments, ORBIT maintains a Bayesian posterior over four latent states: ALIGNED, EXPOSED, COMPROMISED, and HARM COMPLETE. At each decision, the filter predicts state evolution and incorporates newly observed monitor evidence. Persistent memory carries the posterior and retained evidence across linked episodes rather than reinitializing them at episode boundaries. The policy uses posterior probabilities, belief changes, peak belief, retained monitor scores, observation count, evidence age, and tool-family indicators to select a leaf of a fixed decision tree. Figure 3 (p. 13), Appendix B (pp. 12–15), and Table 6 (p. 17) describe the mechanism. AgentDojo evaluates ALLOW/STOP decisions, while the separate branch-replay experiments evaluate ALLOW/INSPECT/BLOCK.

Costs and sensitivity: We acknowledge the substantial increase in legitimate-work denial associated with persistent memory (Section 3.4, Table 2). Separately, Table 5 demonstrates inspection gains at comparable benign cost. Figure 2 and Appendix C examine sensitivity to inspection fees, benign-cost limits, penalty parameters, and context representations. The effects are not uniformly monotonic, and we do not claim robustness across all parameter configurations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.