Locally Safe, Globally Unsafe: Composition-Induced Failure in Networked AI Agents
Abstract
Autonomous agents are increasingly certified one at a time and then deployed in groups onto shared infrastructure. We show that this practice certifies almost nothing. Our running case is an autonomous cloud whose deployment, reliability and capacity agents each satisfy their own specification and which, composed, take the service down with no agent making a bad decision. The mechanism is a regenerative dissipative interaction cycle: a loop that rebuilds its own trigger while consuming a shared resource. In a Petri-net formulation we prove a depletion bound and show it is attained, that screening for such a cycle is a linear program rather than a search over markings, that one matrix product per agent pair decides whether a local certificate survives composition, and that a cycle consuming nothing leaves the probability of failure exactly unchanged while inflating the time to it by up to 2 in the running example. The evidence comes from artifacts built by other people. AppWorld's authors wrote, for each of 486 tasks, an assertion naming which database models a correct run may change; read as local certificates, only 21.0% of 235,710 ordered task pairs survive composition, falling to 5.7% once twenty undocumented cross-app effects are counted. Two task executors sharing one live AppWorld account then take the benchmark's own score from 24/24 to 6/24, and cutting exactly the fusions our test flags restores it where the intuitive repair does not. Benchmarks that score one agent at a time cannot certify behavior in a world that agent shares.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.