SERAM: Self-Evolving Runtime Anomaly Detection for LLM Agents

LLM agents execute interdependent steps, allowing local errors to propagate through subsequent actions. Runtime detectors must identify such errors from the observed trajectory prefix. Changing tasks, tools, and policies can introduce new failure patterns, while feedback on detection errors arrives only later and cannot be used by a static detector. We study adaptation from delayed feedback under three deployment requirements: uninterrupted detection, validated updates, and budgeted LLM inference. We propose SERAM, a self-evolving runtime detector that keeps its LLM frozen and couples an editable behavior graph with lightweight retrieval and scoring models. The graph encodes agent behaviors, transitions, and anomaly rules; the lightweight models handle most steps and escalate uncertain cases to LLM analysis according to a shared routing rule. Given delayed labels, SERAM groups errors by retrieved behavior nodes, revises their rules or adds nodes, and retrains the lightweight models to incorporate the revised knowledge. Updates are prepared asynchronously and deployed only after passing predefined validation checks on held-out feedback. We evaluate under a streaming protocol that releases labels in batches so that updates affect only subsequent predictions, on StreamAD, a new benchmark with step-level labels and measured step timings. Across the full test stream, self-evolution improves average precision (AP) by 10.25% over the frozen counterpart. Self-evolution also reduces mean detection latency by 33% relative to the frozen counterpart. Overall, SERAM improves AP by 62% over the strongest baseline and is the only evaluated LLM-based detector whose mean latency is below the agents' mean step time.