Offline Scores Overestimate Online Oversight: Timely Interception of LLM Agents with Causal-Cone Monitoring
Abstract
Monitors for LLM agents are predominantly evaluated offline: a suspicion score is computed over a completed trajectory and summarized by AUC-ROC. Deployment asks a different question, because an alarm raised after the wire transfer clears or the API key leaves the machine is an autopsy rather than an intervention: can the monitor fire before the agent's first irreversible step? We formalize this as online interception and introduce the Online Interception Framework (OIF), an evaluation protocol that anchors each attack trajectory at a deterministic, auditable harm horizon , the first irreversible step, located by a rule-based detector rather than by the model's self-report, and scores a monitor by whether its alarm fires strictly before under a matched false-alarm budget. Under OIF, offline and online performance decouple: across two environments, three attack families, and four monitors spanning open-weight and closed models, completed-trajectory AUC can stay high while pre-horizon interception is low, and the offline ranking need not predict the online one. To explain when interception fails, we separate a monitor's two information channels, the agent's stated channel (its chain-of-thought and reports) and the action channel (what an action can causally reach), and evaluate a causal-cone monitor that reads the action channel: at each step it scores whether the action opens a path toward harm, rather than trusting the agent's stated justification. Its timeliness advantage grows with how far the attack corrupts the stated channel, from large under monitor hijacking and deceptive reasoning, to small under hidden reasoning, to none when attacks are too weak to bite; an effectiveness-stratified contrast ties the advantage to injection efficacy (ineffective injections , effective injections ). Every experiment runs on a single workstation with an open-weight local monitor overseeing closed frontier agents, so both the protocol and the monitor are inexpensive to reproduce.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.