- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies three races that arise in long-running agents when tool code, permissions, and recovery state change concurrently: version tearing (checking v1 but dispatching v2), privilege resurrection (rollback restoring revoked authority), and duplicate effects (a remote effect succeeds, the local process crashes, and a retry repeats it). PROTEAN binds every admitted invocation to an immutable snapshot < CapID, Revision, Generation, Epoch_t, A(I)> and decouples the code revision from the authority epoch: rollback restores code but not authority; admission, dispatch, and recovery share one invocation identity; an attempt row is written before any side effect, and a crash without a commit is classified as unknown, with same-identity retries rejected. The system is a user-space trusted Kernel that unifies OpenAPI, MCP, Python, and skills, with leases, delegation, carriers, and a WAL. The evaluation covers AgentDojo, 1,146 out-of-grant requests, 60,000-request replacement/rollback workloads, crash recovery, and real remote MCP servers.
Critical strengths
• The problem is well identified. The three races are concrete and real, and existing work usually handles code updates, authorization, and recovery separately.
• The core abstraction is clean. Decoupling revision/generation from a monotonic authority epoch while sharing the invocation identity is the paper's most valuable systems insight.
• The implementation is complete. Unified capability IR, fail-closed admission, leases, delegation, carriers, and journaling are integrated into one runtime, with a quantified TCB.
• The evaluation is broad. Differential execution, defense-off ablations, scripted positive controls, a counterexample ladder, crash replay, and remote MCP experiments all target distinct claims.
Critical weaknesses
• The security guarantee depends on a large TCB. The Kernel is a user-space reference monitor, and the TCB is 6,924/12,804 LOC; host executors remain trusted, 63/70 host-bypass variants escape, and the nine invariants are argued only by proof sketches, not machine-checked. The "0 out-of-envelope effects" claim holds only at the Kernel boundary and under Assumption A1.
• Coverage and utility evidence are weak. Only 32.5% of OpenAPI operations, 51.6% of MCP tools, 18.3% of skills, and 71% of AgentDojo tools are representable; utility is measured on only 85/97 AgentDojo tasks. The 95.3% vs. 93.7% result has a lower bound of -0.8 pp, non-inferiority uses a loose -10 pp margin, and the conclusion at -2 pp depends on the dependence model.
• Lifecycle events are harness-injected. The paper does not test an adaptive adversary who controls the update channel or registry. The baselines (NAIVE, ERLANG-2VER, MCP-ADVISORY) are author reimplementations, and the safety matrix uses fixed barriers, so it shows whether a mechanism exists rather than natural hazard rates.
• Policy precision is assumed away. The system guarantees only that a given policy is strictly enforced. The LLM policy synthesizer covers the injection goal in 45/321 task–goal pairs; when the policy is too broad, the attack remains in-envelope.
Questions for authors
See the weaknesses
5 comments
W3. Harness-injected events, reimplemented baselines, and fixed barriers
These three observations are well taken, and we have sharpened what each protocol measures:
- Fixed barriers: The safety matrix demonstrates whether an architecture provides a mechanism for each hazard, rather than estimating natural occurrence frequencies. Re-evaluating this matrix at 300 trials per cell (6,300 trials total) confirms that every PROTEAN cell achieves 0/300 violations (exact Clopper-Pearson 95% interval [0, 1.2]% per cell), whereas 11 of the 16 baseline cells violate safety across all 300 trials. We present these per-cell counts directly without extrapolating natural hazard rates.
- Baselines: NAIVE, ERLANG-2VER, and MCP-advisory serve as semantic reimplementations, as designated in the paper. C3-R (Table 5, Appendix D) complements them with unmodified external components, specifically the official MCP Python SDK client and the official gitea-mcp server. Under this setup, the official client still dispatches all 180 calls following a declared definition change and duplicates the issue in 19/20 lost-response retries; in contrast, Protean sends none of the 180 calls and rejects the same-identity retry in 20/20 instances.
- Harness-issued events: The P1 official-agent study leaves τ²-bench's agent, user simulator, orchestrator, and evaluator completely unmodified while the harness manages event injection. Across 20 paired tasks per event, executing a capability replacement, an unrelated revocation, or a Kernel crash-restart under Protean kept continuation and task success within one to two rows of undisturbed native executions (58/60 vs 56/60 and 48/60 vs 49/60). The new revision took effect in 20/20 replacement rows, and the authority epoch advanced in 20/20 crash rows. Here, crash-restart denotes an in-process close and reopen of persistent Kernel state rather than an OS process termination.
- Adaptive adversary: Adversaries controlling update channels or registries fall outside our threat model and were not evaluated. Both the revised threat model and the limitations section state this boundary explicitly.
W4. Policy precision
This observation highlights an essential architectural boundary: Protean enforces policies rather than synthesizing them. Our framework provides the execution substrate that enforces whatever policy an external synthesizer or human approver specifies, composing directly with policy-level defenses like CaMeL and Progent. Our experimental evidence decouples these two layers:
- Policy layer: An LLM-derived envelope encompasses the injection goal's tools in 45/321 eligible task-goal pairs (14.0%), compared to 340/340 for a broad policy. It includes every ground-truth tool for 31 of 40 tasks while under-granting the remaining 9 (mean precision of 0.93 across 38 non-empty envelopes). Across the 34 adapter-supported tasks, task success shifts from 29/34 under the broad policy to 26/34 under the derived policy: a paired difference of -8.8 pp with a 95% confidence interval of [-35.1, +15.9] pp. Policy width therefore trades residual attack surface against task coverage, a trade-off that Protean does not resolve.
- Enforcement layer: Given an operational policy, the Kernel enforces it strictly at the effect level. In C2-K, six hidden-effect probes (10 trials each) permitted undeclared effects to materialize in 60/60 trials under name-and-argument checking, compared to 0/60 under effect mediation (30 blocked by the sandbox substrate, 30 by authority checks). In the scripted positive control (Table 24), the deny arm refuses 82/82 goal-aligned requests, whereas a rubber-stamp approver realizes 82/82. Across 250 injection episodes, 40 goal-aligned out-of-ceiling requests were neither granted nor realized. The single observed goal-aligned effect used an already-held grant, which is a policy over-grant rather than an enforcement failure (Table 2, note b).
We omitted intermediate sweeps because no frozen medium policy exists; the two evaluated policies represent the practical endpoints. The revised text maintains policy precision as an explicit limitation alongside our enforcement claims.
Addendum to W3: harness-issued events, now inside unmodified graded workflows
Your W3 point stands and bounds what this new study shows: lifecycle events remain issued by the harness. What changed is the substrate they land in. On AutomationBench v1.0.6, events fire inside real cross-application business workflows graded by the benchmark's own end-state assertions, with tasks, tools, and graders unmodified—120 tasks × 6 conditions, including a chain of all three events in one episode under all six orders. The study tests enforcement precision where it matters: by the Kernel call log, all 8,271 post-revocation attempts on withdrawn capabilities were denied while all 12,945 other post-revocation calls passed the authority check, and across the three conditions without revocation none of 22,049 mediated calls was refused; persistence through the in-process close/reopen held on each exercised property—all 60 chains replacing before restart called the new revision afterwards, all 60 revoking before restart kept the revocation (46 retried the withdrawn tool, all denied), and 28 of the 40 chains where both events preceded the restart exercised both properties after it. The traces also expose a usability cost: because the revoked tool stays advertised and denials are opaque, agents retry—35/120 single-revocation and 32/120 chained rows exhausted the 50-turn budget (vs 4/120 without events). Details are posted on Reviewer A2zt's thread.
Thank you for the incisive summary of our architecture and explicit boundary conditions. Below, we address each weakness in turn. Table numbers correspond to the revised paper; the compact experiment index added as Table 6 shifts subsequent table numbers up by one relative to the submission.
W1. Trusted computing base and assumptions
This observation is entirely correct, and the revised abstract states this scope explicitly. The out-of-envelope guarantee applies strictly to the Kernel-mediated boundary under Assumption A1. Specifically, the revised abstract sentence reads: "Across 1,146 out-of-grant requests, none produces an out-of-envelope effect at the Kernel-mediated boundary under Assumption A1."
Carrier-specific guarantees: We do not treat in-process host callbacks as confined simply because they route through an API wrapper. The carrier audit (Table 20) categorizes operations by enforcement depth: those receiving OS-enforced mediation (sandboxed process, where 70/70 bypass variants were intercepted or never realized), those bearing assumptions (trusted host, where 63/70 variants escape), and those observable solely at local egress (remote HTTP and MCP).
The TCB comprises 6,924 of 12,804 LOC (Table 34). Of these, 1,776 LOC are host and interpreter executors. Relocating effectful builtins into unprivileged process carriers removes most of this trusted code, at a measured latency of 23.0 ms p50 per call with the Bubblewrap process carrier versus 12.9 ms carrier-free (Table 23).
The nine invariants rest on proof sketches, executable tests, fixed-barrier protocols, and defense-off ablations. They are not machine-checked proofs; the revised text states this distinction clearly and identifies mechanized verification as future work.
W2. Coverage and utility evidence
Representability versus runtime overhead: Representability and utility evaluate orthogonal dimensions, and the 85/97 task restriction marks their boundary. The 12 excluded AgentDojo tasks require tools outside the adapter's supported translation set. Fail-closed admission refuses those operations, so including them would conflate adapter representability with execution overhead.
We measure this coverage trade-off directly on 26 AgentDojo workspace tasks × 3 seeds (78 pairs; in 10 tasks the agent requests operations that the submission adapter does not declare):
create_file,append_to_file,send_email). Across the 48 pairs with zero refused requests, the two arms achieve 44/48 and 45/48 successes.Representability rates also reflect schema quality in the wild. In Table 1, all 158 parse-clean MCP tools (of 306) are representable, and OpenAPI admission reaches 53.3% (4,206/7,887) within the parse-clean JSON domain; unsupported specifications, primarily non-JSON media, are rejected rather than executed under relaxed contracts. Skills (18.3%) remain the lowest-coverage corpus.
C1-X dependence models: The -10 pp non-inferiority margin does not carry our primary claim. Table 10 demonstrates that all three dependence models also pass under a tighter -5 pp threshold. The most conservative bound (pair independence) is -2.7 pp, and the preregistered cluster bound is -0.8 pp. Only the strict -2 pp verdict hinges on the chosen dependence structure, as you noted, and the arms demonstrate statistical equivalence within ±5 pp (TOST). Furthermore, the four open-weight planners in Table 2 clear -10 pp across single seeded runs, and τ²-bench (Table 29) yields identical mean rewards across dispatch paths. We frame these results as bounded empirical evidence across two benchmarks rather than universal equivalence.