acceptodds
Under review as a conference paper at ICLR 2027

Protean:Binding Code, Authority, and Recovery Across Updates in Agent Runtimes

Abstract

Long-running LLM agents execute through tools whose implementations and permissions may change while calls are in flight. Existing runtimes typically treat code updates, authorization, and crash recovery as separate concerns. This separation creates invocation-level races: a call may be checked against one revision but dispatched to another, rollback may revive revoked authority, and retry after a crash may duplicate an external effect. PROTEAN maintains code and authority as independent version histories, but binds each admitted invocation to one immutable code revision, authority epoch, and recovery identity. This prevents code rollback from resurrecting revoked authority while ensuring that admission, dispatch, and recovery refer to the same invocation. A trusted Kernel (i) admits heterogeneous operations through a fail-closed capability contract, (ii) enforces epoch-scoped authority at every invocation, and (iii) pins execution and journals effects for safe replacement, rollback, and recovery. On AgentDojo, mixed HTTP/MCP/Python execution matches native-tool utility (95.3% vs. 93.7%). Across 1,146 out-of-grant requests, the Kernel realizes no out-of-envelope effect at its mediated boundary. Across 60,000-request replacement and rollback workloads and adversarial interleavings, PROTEAN loses no requests and exhibits none of six constructed lifecycle violations.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, runtime enforcement, capability systems, dynamic software updating, crash recovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CfptOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies three races that arise in long-running agents when tool code, permissions, and recovery state change concurrently: version tearing (checking v1 but dispatching v2), privilege resurrection (rollback restoring revoked authority), and duplicate effects (a remote effect succeeds, the local process crashes, and a retry repeats it). PROTEAN binds every admitted invocation to an immutable snapshot < CapID, Revision, Generation, Epoch_t, A(I)> and decouples the code revision from the authority epoch: rollback restores code but not authority; admission, dispatch, and recovery share one invocation identity; an attempt row is written before any side effect, and a crash without a commit is classified as unknown, with same-identity retries rejected. The system is a user-space trusted Kernel that unifies OpenAPI, MCP, Python, and skills, with leases, delegation, carriers, and a WAL. The evaluation covers AgentDojo, 1,146 out-of-grant requests, 60,000-request replacement/rollback workloads, crash recovery, and real remote MCP servers.

Critical strengths

• The problem is well identified. The three races are concrete and real, and existing work usually handles code updates, authorization, and recovery separately.

• The core abstraction is clean. Decoupling revision/generation from a monotonic authority epoch while sharing the invocation identity is the paper's most valuable systems insight.

• The implementation is complete. Unified capability IR, fail-closed admission, leases, delegation, carriers, and journaling are integrated into one runtime, with a quantified TCB.

• The evaluation is broad. Differential execution, defense-off ablations, scripted positive controls, a counterexample ladder, crash replay, and remote MCP experiments all target distinct claims.

Critical weaknesses

• The security guarantee depends on a large TCB. The Kernel is a user-space reference monitor, and the TCB is 6,924/12,804 LOC; host executors remain trusted, 63/70 host-bypass variants escape, and the nine invariants are argued only by proof sketches, not machine-checked. The "0 out-of-envelope effects" claim holds only at the Kernel boundary and under Assumption A1.

• Coverage and utility evidence are weak. Only 32.5% of OpenAPI operations, 51.6% of MCP tools, 18.3% of skills, and 71% of AgentDojo tools are representable; utility is measured on only 85/97 AgentDojo tasks. The 95.3% vs. 93.7% result has a lower bound of -0.8 pp, non-inferiority uses a loose -10 pp margin, and the conclusion at -2 pp depends on the dependence model.

• Lifecycle events are harness-injected. The paper does not test an adaptive adversary who controls the update channel or registry. The baselines (NAIVE, ERLANG-2VER, MCP-ADVISORY) are author reimplementations, and the safety matrix uses fixed barriers, so it shows whether a mechanism exists rather than natural hazard rates.

• Policy precision is assumed away. The system guarantees only that a given policy is strictly enforced. The LLM policy synthesizer covers the injection goal in 45/321 task–goal pairs; when the policy is too broad, the attack remains in-envelope.

Questions for authors

See the weaknesses

5 comments

Thank you for the incisive summary of our architecture and explicit boundary conditions. Below, we address each weakness in turn. Table numbers correspond to the revised paper; the compact experiment index added as Table 6 shifts subsequent table numbers up by one relative to the submission.

W1. Trusted computing base and assumptions

This observation is entirely correct, and the revised abstract states this scope explicitly. The out-of-envelope guarantee applies strictly to the Kernel-mediated boundary under Assumption A1. Specifically, the revised abstract sentence reads: "Across 1,146 out-of-grant requests, none produces an out-of-envelope effect at the Kernel-mediated boundary under Assumption A1."

Carrier-specific guarantees: We do not treat in-process host callbacks as confined simply because they route through an API wrapper. The carrier audit (Table 20) categorizes operations by enforcement depth: those receiving OS-enforced mediation (sandboxed process, where 70/70 bypass variants were intercepted or never realized), those bearing assumptions (trusted host, where 63/70 variants escape), and those observable solely at local egress (remote HTTP and MCP).

The TCB comprises 6,924 of 12,804 LOC (Table 34). Of these, 1,776 LOC are host and interpreter executors. Relocating effectful builtins into unprivileged process carriers removes most of this trusted code, at a measured latency of 23.0 ms p50 per call with the Bubblewrap process carrier versus 12.9 ms carrier-free (Table 23).

The nine invariants rest on proof sketches, executable tests, fixed-barrier protocols, and defense-off ablations. They are not machine-checked proofs; the revised text states this distinction clearly and identifies mechanized verification as future work.

W2. Coverage and utility evidence

Representability versus runtime overhead: Representability and utility evaluate orthogonal dimensions, and the 85/97 task restriction marks their boundary. The 12 excluded AgentDojo tasks require tools outside the adapter's supported translation set. Fail-closed admission refuses those operations, so including them would conflate adapter representability with execution overhead.

We measure this coverage trade-off directly on 26 AgentDojo workspace tasks × 3 seeds (78 pairs; in 10 tasks the agent requests operations that the submission adapter does not declare):

  • Submission adapter: Native dispatch succeeds on 73/78 runs and Protean on 44/78 runs. All 131 refusals stem from three operations (create_file, append_to_file, send_email). Across the 48 pairs with zero refused requests, the two arms achieve 44/48 and 45/48 successes.
  • Expanded contract: Declaring these three operations with the Kernel approval path unchanged recovers 29 of the 30 affected pairs. The revised paper documents 347/382 valid primary rows across five lifecycle conditions, separating the 11-row replay as a supplement at 355/390.

Representability rates also reflect schema quality in the wild. In Table 1, all 158 parse-clean MCP tools (of 306) are representable, and OpenAPI admission reaches 53.3% (4,206/7,887) within the parse-clean JSON domain; unsupported specifications, primarily non-JSON media, are rejected rather than executed under relaxed contracts. Skills (18.3%) remain the lowest-coverage corpus.

C1-X dependence models: The -10 pp non-inferiority margin does not carry our primary claim. Table 10 demonstrates that all three dependence models also pass under a tighter -5 pp threshold. The most conservative bound (pair independence) is -2.7 pp, and the preregistered cluster bound is -0.8 pp. Only the strict -2 pp verdict hinges on the chosen dependence structure, as you noted, and the arms demonstrate statistical equivalence within ±5 pp (TOST). Furthermore, the four open-weight planners in Table 2 clear -10 pp across single seeded runs, and τ²-bench (Table 29) yields identical mean rewards across dispatch paths. We frame these results as bounded empirical evidence across two benchmarks rather than universal equivalence.

W3. Harness-injected events, reimplemented baselines, and fixed barriers

These three observations are well taken, and we have sharpened what each protocol measures:

  • Fixed barriers: The safety matrix demonstrates whether an architecture provides a mechanism for each hazard, rather than estimating natural occurrence frequencies. Re-evaluating this matrix at 300 trials per cell (6,300 trials total) confirms that every PROTEAN cell achieves 0/300 violations (exact Clopper-Pearson 95% interval [0, 1.2]% per cell), whereas 11 of the 16 baseline cells violate safety across all 300 trials. We present these per-cell counts directly without extrapolating natural hazard rates.
  • Baselines: NAIVE, ERLANG-2VER, and MCP-advisory serve as semantic reimplementations, as designated in the paper. C3-R (Table 5, Appendix D) complements them with unmodified external components, specifically the official MCP Python SDK client and the official gitea-mcp server. Under this setup, the official client still dispatches all 180 calls following a declared definition change and duplicates the issue in 19/20 lost-response retries; in contrast, Protean sends none of the 180 calls and rejects the same-identity retry in 20/20 instances.
  • Harness-issued events: The P1 official-agent study leaves τ²-bench's agent, user simulator, orchestrator, and evaluator completely unmodified while the harness manages event injection. Across 20 paired tasks per event, executing a capability replacement, an unrelated revocation, or a Kernel crash-restart under Protean kept continuation and task success within one to two rows of undisturbed native executions (58/60 vs 56/60 and 48/60 vs 49/60). The new revision took effect in 20/20 replacement rows, and the authority epoch advanced in 20/20 crash rows. Here, crash-restart denotes an in-process close and reopen of persistent Kernel state rather than an OS process termination.
  • Adaptive adversary: Adversaries controlling update channels or registries fall outside our threat model and were not evaluated. Both the revised threat model and the limitations section state this boundary explicitly.

W4. Policy precision

This observation highlights an essential architectural boundary: Protean enforces policies rather than synthesizing them. Our framework provides the execution substrate that enforces whatever policy an external synthesizer or human approver specifies, composing directly with policy-level defenses like CaMeL and Progent. Our experimental evidence decouples these two layers:

  • Policy layer: An LLM-derived envelope encompasses the injection goal's tools in 45/321 eligible task-goal pairs (14.0%), compared to 340/340 for a broad policy. It includes every ground-truth tool for 31 of 40 tasks while under-granting the remaining 9 (mean precision of 0.93 across 38 non-empty envelopes). Across the 34 adapter-supported tasks, task success shifts from 29/34 under the broad policy to 26/34 under the derived policy: a paired difference of -8.8 pp with a 95% confidence interval of [-35.1, +15.9] pp. Policy width therefore trades residual attack surface against task coverage, a trade-off that Protean does not resolve.
  • Enforcement layer: Given an operational policy, the Kernel enforces it strictly at the effect level. In C2-K, six hidden-effect probes (10 trials each) permitted undeclared effects to materialize in 60/60 trials under name-and-argument checking, compared to 0/60 under effect mediation (30 blocked by the sandbox substrate, 30 by authority checks). In the scripted positive control (Table 24), the deny arm refuses 82/82 goal-aligned requests, whereas a rubber-stamp approver realizes 82/82. Across 250 injection episodes, 40 goal-aligned out-of-ceiling requests were neither granted nor realized. The single observed goal-aligned effect used an already-held grant, which is a policy over-grant rather than an enforcement failure (Table 2, note b).

We omitted intermediate sweeps because no frozen medium policy exists; the two evaluated policies represent the practical endpoints. The revised text maintains policy precision as an explicit limitation alongside our enforcement claims.

Addendum to W3: harness-issued events, now inside unmodified graded workflows

Your W3 point stands and bounds what this new study shows: lifecycle events remain issued by the harness. What changed is the substrate they land in. On AutomationBench v1.0.6, events fire inside real cross-application business workflows graded by the benchmark's own end-state assertions, with tasks, tools, and graders unmodified—120 tasks × 6 conditions, including a chain of all three events in one episode under all six orders. The study tests enforcement precision where it matters: by the Kernel call log, all 8,271 post-revocation attempts on withdrawn capabilities were denied while all 12,945 other post-revocation calls passed the authority check, and across the three conditions without revocation none of 22,049 mediated calls was refused; persistence through the in-process close/reopen held on each exercised property—all 60 chains replacing before restart called the new revision afterwards, all 60 revoking before restart kept the revocation (46 retried the withdrawn tool, all denied), and 28 of the 40 chains where both events preceded the restart exercised both properties after it. The traces also expose a usability cost: because the revoked tool stays advertised and denials are opaque, agents retry—35/120 single-revocation and 32/120 chained rows exhausted the 50-turn budget (vs 4/120 without events). Details are posted on Reviewer A2zt's thread.

Reviewer_A2ztOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PROTEAN, a runtime for long-running LLM agents. It aims to address inconsistencies between code updates, authority changes, and crash recovery in dynamic tool environments. The authors identify three possible failures: version tearing, privilege resurrection after rollback, and duplicate external effects after crashes. PROTEAN uses a trusted Kernel to bind each admitted invocation to an immutable code revision, an authority epoch, and a durable recovery identity. It combines capability-based authorization, invocation snapshot pinning, and write-ahead journaling. The authors evaluate the system through heterogeneous tool execution, AgentDojo tasks, authority enforcement tests, and lifecycle experiments.

Critical strengths

The paper addresses a meaningful problem. Long-running agents may use tools that change during execution and perform actions with external effects. Keeping code versions, authority, and recovery states consistent is important for such systems. The paper clearly defines several possible failures and explains their security risks.

The system design is reasonable and well structured. PROTEAN manages code revisions, authority epochs, and recovery identities within a unified invocation lifecycle. Each component has a clear purpose, and the overall design directly addresses the identified consistency problems.

The evaluation is extensive and provides good evidence for the proposed mechanisms. The authors evaluate task utility, runtime overhead, authority enforcement, and lifecycle safety. They also include negative controls, ablations, crash tests, and remote MCP experiments.

Critical weaknesses
  1. The practical coverage of the safety guarantees is unclear. PROTEAN relies on specific execution and trust boundaries. For example, it cannot detect changes in remote MCP server behavior when the tool definition remains unchanged. On the trusted-host carrier, 63 of 70 bypass variants escape the effect API. Unpinned child dependencies can also execute different code revisions. The authors acknowledge these limitations. However, it is unclear how many tools and operations in typical agent workloads can receive the full safety guarantees. This makes it difficult to assess the system's practical effectiveness.

  2. The experiments validate the mechanisms, but provide limited evidence of end-to-end benefits in real agent tasks. The AgentDojo experiments mainly show that tool adaptation does not significantly reduce task success. The lifecycle safety experiments mainly use scripted workloads and controlled failure interleavings. These evaluations are largely separate. As a result, it is unclear how much PROTEAN helps a real LLM agent avoid task failures, continue execution, and recover when tool updates, revocations, or crashes occur during a long-running task. Also, the lifecycle baselines are author-implemented semantic reimplementations rather than actual external runtimes. This limits how much we can conclude from the comparisons.

  3. The trade-off between safety and usability needs more discussion. PROTEAN uses fail-closed admission to reject unsupported tool contracts. It also marks uncertain non-idempotent effects as unknown. These choices improve safety, but may limit tool coverage and task continuity. In the remote MCP experiments, the definition pin increases the p50 latency of Gitea create calls from about 1.5 seconds to 3.8 seconds. The paper does not fully show how these limitations affect real agent workloads.

Questions for authors
  1. In a typical agent tool environment, which operations can receive the full invocation-level consistency and authority enforcement guarantees of PROTEAN? Can the authors estimate how much the limitations of remote MCP servers, trusted-host executors, and unpinned child dependencies affect tool coverage and safety in practice?

  2. When a non-idempotent invocation is marked as unknown, how should an agent safely continue its task? If the agent retries with a new job identity, how can duplicate effects be avoided? Can the authors also discuss the trade-off between tool admission coverage, remote definition-checking overhead, and safety? Are these costs acceptable for real long-running agent tasks?

5 comments

Thank you for the rigorous review and targeted questions. We address W1 alongside Q1 and W3 alongside Q2, as each pairing examines the same underlying mechanisms. Table numbers correspond to the revised paper; the compact experiment index added as Table 6 shifts all subsequent table numbers up by one relative to the submission.

W1 and Q1. Which operations receive the full guarantee, and what the boundaries cost

Protean provides three distinct guarantees, whose enforcement depends directly on the selected carrier:

  1. Invocation binding: Admission against the declared contract, revision pinning, the authority epoch, and the recovery identity apply to every admitted invocation across all carriers.
  2. Effect confinement: On the sandboxed process carrier, 70/70 bypass variants were intercepted or never realized (Table 20). On the trusted-host carrier, 63/70 variants escape the effect API, meaning host executors reside within the TCB under Assumption A1. On remote HTTP and MCP carriers, the Kernel mediates and logs local network egress but cannot inspect server-internal execution.
  3. Code identity: The pinned artifact is covered. A child dependency resolved at runtime is covered only if it declares an explicit revision pin (Table 18, CE3).

In a typical tool environment, untrusted or generated code requires the sandboxed carrier to obtain all three guarantees. Host callbacks receive invocation binding while relying on Assumption A1 for effect confinement. Remote tools receive invocation binding and egress mediation. Table 5 measures this remote boundary from both sides: a declared definition change blocked all 180 Protean calls prior to tools/call while the native client sent all 180, whereas a behavior-only change behind an unchanged definition made 60/60 issues public in both arms.

Carrier distribution in a real workload. The C1-X utility run routed 550 of its 1,405 calls through the host carrier (where effect safety rests on A1) and 855 through loopback MCP and HTTP carriers. None utilized the sandboxed carrier, which incurs a p50 latency of 23.0 ms per call with the Bubblewrap process carrier compared to 12.9 ms carrier-free (Table 23).

Representability filter. Under our evaluated adapters, 49 of the 69 AgentDojo tools (71%), 51.6% of MCP tools, and 32.5% of OpenAPI operations are representable. Operations outside this envelope are refused explicitly rather than executed with degraded guarantees; W3 evaluates what these refusals cost in practice. The wider prevalence of unpinned child dependencies across real tool ecosystems remains an open empirical question, marked as an unmeasured limitation.

W2. End-to-end evidence with a real agent

The new P1 pilot measures event reach, post-event continuation, task success, and duplicate-fingerprint diagnostics within an unmodified agent stack. τ²-bench's official agent, user simulator, orchestrator, and evaluator executed 20 task-seed pairs per event in two arms. Following the first successful tool call, Protean triggers either a capability replacement, revocation of a tool unneeded by the task, or a Kernel crash-restart (defined as an in-process close and reopen of persistent state, rather than an OS-level process termination). The native arm runs the identical pair without events, serving as an undisturbed baseline.

EventNative, no event (continued / success)Protean, event executed (continued / success)Kernel record
Replacement18 / 1720 / 17revision 2 active in 20/20
Unrelated revocation19 / 1619 / 1512 calls to the revoked tool denied
Crash-restart19 / 1619 / 16authority epoch 1 to 2 in 20/20

Here, "continued" indicates that a subsequent tool call returned successfully after the event injection point. Two rows never reached the event and remain in the denominator. Across all runs, no row repeated a pre-event call with identical arguments (0/120, evaluated via a harness-level diagnostic rather than an external receipt). All 120 rows completed without infrastructure faults. Executing these lifecycle events under Protean kept continuation and task success within one to two rows of undisturbed runs (58/60 vs 56/60; 48/60 vs 49/60), while the Kernel strictly enforced the updated revision, tool revocation, and epoch transition. P1 does not demonstrate how an unmodified runtime fails under identical events, as native stacks lack replacement or recovery semantics. At 20 pairs per cell, these results are descriptive rather than a formal test of statistical superiority.

W2 (continued). How the failure side is measured

We evaluate the failure side against unmodified external components in C3-R (Table 5, Appendix D). Native arms utilize the official MCP Python SDK client against the official gitea-mcp server deployed on a live Gitea instance over a WAN. When a create invocation loses its response:

  • The official client's automatic retry duplicates the created issue in 19/20 episodes.
  • Protean rejects the same-identity retry in 20/20 episodes, allowing state reconciliation to recover the true execution outcome in 20/20 instances.

The Table 15 baselines represent semantic reimplementations, and the manuscript maintains that precise designation. The identical five lifecycle conditions were also evaluated within realistic AgentDojo tasks (390 episodes under the submission contract) without exhibiting mechanism failures. Capability replacement and rollback were actively exercised in 23 and 22 of 78 episodes; in the remaining episodes, the agent simply did not invoke the affected tool post-event.

W3 and Q2. The safety-usability trade-off and continuing after an unknown effect

MechanismMeasured costWhat it buys
Fail-closed admission (submission adapter)AgentDojo workspace, 78 task-seed pairs, 44/78 vs native 73/78, concentrated on the 30 rows that requested one of three undeclared operations (131 refusals); 44/48 vs 45/48 on the other rowsUnsupported operations are refused, never run with weaker checks
Explicit contract extension (three operations declared)347/382 valid primary rows over five lifecycle conditions; 355/390 post-hoc replay supplement; 29 of the 30 refused rows recoverSame Kernel approval path, effect ledger kept
Remote definition pinGitea create p50 from 1,484 to 3,821 ms, p95 from 1,907 to 4,950 msChanged definitions refused before tools/call (0/180 vs 180/180 sent)
Unknown outcome plus same-identity rejectionContinuation waits for reconciliationNo duplicate, vs 19/20 duplicated by the official client's retry
Lifecycle events in the official agent (P1)Continuation 58/60 vs 56/60 undisturbed; success 48/60 vs 49/60Revision, revocation, and epoch enforced

In the expanded-contract cohort, the primary run had 11 endpoint-error positions on one task (10 HTTP 500, 1 model-error-only). Eight lacked a valid final row, leaving 382 valid rows with 347 successes (90.8%). A replay of all 11 error-affected rows, scheduled only after the primary evaluation, serves strictly as a labeled supplement (yielding 355/390 following substitution).

Continuing after an unknown effect (Q2). Protean never re-dispatches a non-idempotent effect autonomously. Following an ambiguous crash, the invocation is classified as unknown under its original identity, and any same-identity retry is rejected (Table 16). The calling agent or a recovery policy must then resolve the outcome via an execution receipt, status query, or remote state search:

  • If the effect successfully landed, execution proceeds using the reconciled result.
  • If it did not land, a distinct job identity issues the effect once.

In C3-R, state queries resolved all 20 unknown outcomes correctly (13 had landed, 7 had not). Thirteen episodes continued without re-sending requests, while 7 required a single fresh submission (an action not executed in that run). Issuing a new identity without prior state resolution constitutes a distinct invocation, which Protean will not prevent from duplicating an unverified effect. For external endpoints lacking both receipts and queryable state, the outcome remains unknown and demands caller-level intervention.

Are the overheads acceptable? This trade-off depends on application criticality. For non-idempotent remote creates, about 2.3 s per pinned call buys rejection of drifted definitions and avoids the duplicate observed in 19 of 20 official-client retries. The AgentDojo utility loss was concentrated in three undeclared operations, and declaring them recovered 29 of the 30 affected pairs. In P1, executing lifecycle events maintained continuation and success rates within one to two rows of undisturbed baselines.

Addendum to W2: lifecycle events inside graded business workflows (new evidence)

We completed a second end-to-end study extending P1 along the axis you raised—from event-level continuation to benchmark-graded business outcomes. On AutomationBench v1.0.6, we froze 120 cross-application tasks (email/calendar/Slack/spreadsheet workflows scored by end-state assertions) and evaluated six conditions on identical task identifiers and contracts: native; Protean with no event; replacement, revocation of a capability not yet called at the event boundary, and Kernel close/reopen each alone; and a chained condition issuing all three events in one episode under all six orders (20 tasks per order). One rollout per cell; DeepSeek-V4.1-Flash-1017; benchmark tasks, tools, and graders unmodified; 720/720 rows completed.

Routing every call through the Kernel leaves business success unchanged: native 71/120 vs Protean 71/120 (discordant pairs 9 vs 9). Single events stay at that level: replacement 75/120, restart 73/120—descriptive, within paired noise. Revocation alone passes 22/120; it is 0/65 where it removes the only API executor, versus 11/18 where the withdrawn tool never appeared in the task's own no-event run (control 11/18). The chained condition passes 40/120 (40 fully graded, 40 partial, 40 zero-scored), and the loss follows the withdrawn capability, not the chain:

Capability revoked in chain; used in the task's own no-event run?nchain passno-event pass
api_fetch (the sole API executor); used32019
base64_encode; used582334
base64_encode; not used291617

Every no-event trajectory calls api_fetch, so revoking it is fatal (0/304 scored assertions). Of the 34 pass→fail transitions, 32 withdrew a capability the task's own no-event run had used; and on the 86 tasks where the single-revocation and chained conditions withdrew the same tool, the two conditions pass exactly the same 22. Each task observes a single event order and revocation timing also changes the eligible target pool, so we report these as target-level observations rather than a fully identified order effect; the slot distribution (api_fetch selected in 23/40, 8/40, and 1/40 at event slots 1, 3, and 5) is consistent with that reading. One further task had no revocable candidate and passed. Enforcement did not preclude delivery: 24 chained tasks attempted the revoked tool, were denied, and still passed every scored check through their remaining capabilities.

State set before a restart persists after it, inside the same business run: all 60 chains replacing before restart called the new revision afterwards; where revocation preceded restart, the revoked tool was absent from the reopened authority map in 60/60 chains and every post-restart retry was denied (46/60 attempted); 28 chains exercised both properties after a single restart; 119/120 chains completed all three events.

Two rows show the boundary. support.zendesk_maintenance_notify passes in all six conditions: after replacement, revocation of base64_encode, and restart, the agent keeps searching on revision 2, completes 36 API executions, is denied on the revoked encoder, and passes 16/16 checks. finance.closing_journal_automation passes in native, no-event, replacement, and restart; in the chain it loses api_fetch first, then runs 43 successful searches on revision 2 while all 75 api_fetch attempts are denied—continuation preserved, business outcome failed (0/5 report checks). The two must be measured separately.

Scope: events remain harness-issued; the restart is an in-process close/reopen of persistent Kernel state with no job in flight, so ambiguous-effect recovery still rests on the Gitea protocol; per-episode concurrency is one, so this study does not measure concurrent evolution; cells are single-rollout descriptive observations. The traces also expose a cost: the revoked tool stays advertised and denials are opaque NOT_FOUNDs, so agents retry—35/120 single-revocation and 32/120 chained rows exhausted the 50-turn budget (vs 4/120 without events), and revoke rows cost ~1.9× no-event. The revised paper adds this study as a main-text paragraph plus an appendix subsection with paired-difference intervals and per-assertion results.

Reviewer_DUpCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper shows that even though LLM agents execute through tools with mutable implementations and permissions, existing runtimes treat code updates, authorization, and crash recovery as separate concerns. This results in invocation-level races/risks, and the authors propose the Protean kernel as a solution to eliminate them. To do that, Protean keeps code and authority as independent version histories while binding each admitted invocation to one immutable code revision, authority epoch, and recovery identity. The authors test Protean on AgentDojo with a series of experiments where the guarantees are not broken.

Critical strengths

S1. The authors propose a well-posed problem and, to my knowledge, identify a real gap at the intersection of agent security, dynamic software updating, and transactional recovery.

S2. Methodologically strong evaluations separate the mechanism of the planner from the behavior of the system.

Critical weaknesses

W1. Generalization is limited, as the authors only test their findings on a single benchmark (AgentDojo) with a single planner (DeepSeek-V4.1-Flash). I think both dimensions are important and should be explored more, especially given that, to my understanding, AgentDojo has only 85 tasks. Expanding to more benchmarks and tasks (like longer-horizon tasks) and stronger frontier planners would make the findings much more convincing.

W2. I am a bit worried about the reproducibility of the project. To begin with, no anonymous link to the paper’s code is released. In addition, the headline numbers depend on a commercial endpoint and hence even if a provider-side seeding was provided, the results would not be fully reproducible.

W3. A lot of the safety cells seem to have been computed over 50 or fewer trials. The authors are correctly reporting zero-event counts (correctly so), bounding the violation rate below ~7%. In my opinion, this result is too weak to support the paper’s wording and claimed significance. The authors should either increase the sample size in the safety cells until the bounds support their wording, or weaken the wording.

W4. Semi-minor comment: A lot of the paper’s content is relegated to the appendix, and this makes it hard to follow the paper. For example, the denominator for most of the experiments can be found in Table 6 in the appendix. This makes it confusing as to which denominator was used in each experiment mentioned in the main paper, especially since each experiment uses a different one.

Questions for authors

I do not have further questions for the authors, but I would greatly appreciate a response to the weaknesses I have mentioned.

7 comments

Thank you for the constructive review and specific statistical recommendations. Below, we address each weakness in turn. Table numbers refer to the revised paper (tables following Table 5 increment by one relative to the submission), and the general response outlines our new experimental results.

W1. Single benchmark, single planner, and the size of AgentDojo

  • Planner coverage: To make our planner coverage explicit, Table 2 now presents the 255-pair AgentDojo protocol directly in the main evaluation overview. It incorporates four open-weight planners (Qwen3.8-27B, GLM-4-32B, Nemotron-3-Nano-4B, Olmo-3-7B) across one seeded run each. All five planners pass the -10 pp margin, with zero unapproved lease issuances and zero realized out-of-envelope effects. Task success ranges from 9% to 95%; the two low-utility planners serve primarily as security replications.
  • Second benchmark: Appendix Table 29 presents τ²-bench airline and retail domains evaluated with the official agent, user simulator, orchestrator, and evaluator unchanged. Mean reward matches exactly across dispatch paths (0.75 and 0.95 across 20 task pairs per domain), and all 80 episodes satisfy strict native replay.
  • AgentDojo scope: AgentDojo v1.2.2 comprises 97 user tasks. The utility protocol evaluates the 85 tasks whose ground-truth tools pass the adapter gate, tested across three seeds (255 planned task-seed pairs, 253 valid).
  • Official-agent lifecycle: During the discussion period, we added a lifecycle study within the official τ²-bench agent stack (P1, 120 paired rows; detailed in the general response and W4).
  • Frontier models and long horizons: We do not claim coverage of additional proprietary frontier planners or long-horizon benchmarks. Official frontier-model endpoints do not give us sufficient control over served-model updates, data and response consistency across runs, or total cost at larger scale. Consequently, our model-in-the-loop evidence remains explicitly endpoint-conditioned, and broader planner and horizon coverage remain designated limitations.

W2. Reproducibility

  • Code: The anonymous artifact repository is available at https://anonymous.4open.science/r/Protean-anon-E5D1/. It provides complete source code, tests, experiment drivers, frozen configurations, task manifests, archived reports, and deterministic checkers. The frozen paper-number audit report is included; model-dependent raw request logs are omitted from the capsule. Deterministic checks execute without model endpoints via python3 scripts/reproduce_reviewer.py --profile smoke.
  • Deterministic suites: The representation funnel, authority and sandbox probes, defense-off threat matrix, bounded-queue continuity, safety matrix, and crash-replay soaks run entirely without external model access.
  • Open-weight planners: The four open-weight planners in Table 2 were self-hosted via vLLM 0.29 on a single RTX 4090 (48 GB), reproducing non-inferiority and zero-escalation results without commercial endpoints.
  • DeepSeek cohorts: Every matched arm runs against the same served build on the official endpoint. We record requested aliases, served builds, decoding settings, prompt and tool-schema hashes, task manifests, and complete request logs. We designate these results as endpoint-conditioned because provider updates can alter future outputs for identical requests, a dependency that provider-side seeding cannot resolve.

W3. Safety cells with 50 or fewer trials

We have adopted both of your recommendations.

  • Larger sample: We expanded the barrier safety matrix (Table 15, summarized in Table 4) from 50 to 300 trials per condition-scenario cell, yielding 6,300 model-free trials across 21 cells. Across all five PROTEAN cells (1,500 trials total), zero trials exhibited safety violations. Under the exact Clopper-Pearson 95% interval used in the paper, which yields the ~7% bound you cite at n = 50, the per-cell interval narrows to [0, 1.2]%. The 16 baseline cells reproduce our submitted findings at six times the sample size. Eleven violate safety in all 300 trials, including NAIVE in every scenario, while the five baseline cells featuring matching mechanisms remain at 0/300. Within these fixed-barrier configurations, the protocol reliably detects the constructed hazard whenever the corresponding mechanism is absent.
  • Narrower wording: These counts reflect fixed barriers set at vulnerable interleavings. They demonstrate, cell by cell, that the mechanism arrests the constructed race within the stated bound, rather than estimating natural race frequencies. Section 5.3 (adjacent to Table 4) explicitly clarifies this distinction and replaces claims of statistical significance with exact per-cell counts and confidence intervals.
  • Auxiliary reruns: We reran both auxiliary protocols. Each applicable randomized cell in the counterexample ladder (Table 18) uses 300 timing injections under the same seeds and protocol. Its qualitative outcomes remain unchanged, and the timed PROTEAN cells remain at 0/300 ([0, 1.2]%), except CE3, the disclosed unpinned-child-dependency boundary, which logs hits across every rung, including PROTEAN (104/300). The C3-L benchmark (Table 21) uses 300 seeded runs per arm (10,635 invocations): the PROTEAN arm maintains 0/10,635 torn, stale-authority, or unattributed invocations and 0/295 unsafe replays, alongside 300/300 task successes. All baseline arms reproduce their submitted failure profiles at this expanded scale.

W4. Denominators and appendix placement

A reader should not need the appendix index to interpret a headline number. The revised text therefore provides explicit units and denominators alongside every headline metric, relocates a compact summary of the experiment index (Table 7) into the main text as Table 6 (p. 9) using the discussion period's extra page, and keeps protocol-specific exclusions directly beside their parent tables. The table below lists the unit behind each result cited in this response.

ResultUnitn
C1-X utility (Tables 2, 9, 10)task-seed pair253 valid of 255
Safety matrix (Table 15 rerun)trial per condition-scenario cell300 per cell
Official-agent lifecycle (P1)task-seed pair per arm-event cell20 per cell, 120 rows
Contract study (AgentDojo workspace)task-seed pair per lifecycle condition78 per condition
Remote MCP (Table 5)episode per condition20

In the expanded-contract arm, the primary run had 11 endpoint-error positions on one task (10 HTTP 500, 1 model-error-only). Eight lacked a valid final row, leaving 382 valid rows with 347 successes (90.8%). Following the primary evaluation, we replayed all 11 error-affected rows. Because this replay was decided post-hoc, we designate it strictly as a labeled supplement (yielding 355/390 upon substitution) and do not allow it to overwrite the primary denominator.

Addendum to W1/W4: a third benchmark under the same denominator discipline

Since our response, we added a third task benchmark to the discussion-period evidence: AutomationBench v1.0.6, 120 cross-application business tasks × 6 lifecycle conditions on identical task identifiers and contracts (720 rows, one rollout per cell). This extends coverage beyond AgentDojo and τ²-bench with benchmark-graded end-state assertions rather than reward-only scoring. The endpoint caveat is unchanged: rows run on the official DeepSeek endpoint under served build DeepSeek-V4.1-Flash-1017, matching the other new cohorts and carrying the same endpoint-conditioned designation.

Denominators, per your W4 ask: strict pass is per-task over n = 120 per condition; the chained condition assigns its six event orders deterministically at 20 tasks each; 119/120 chains executed all three events (one task reached the final boundary with no revocable candidate and is reported in the denominator); chained outcomes are also reported as assertion tiers (40 full / 40 partial / 40 zero). A full summary with per-capability stratification is posted on Reviewer A2zt's thread.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: new evidence, scope, and implemented revisions

We appreciate the reviewers' detailed engagement and constructive discussions. Below we summarize the expanded evidence and organization of the 48-page PDF (10-page main body), structured around representation (Section 3.1), lease enforcement (Section 3.2), and mid-task capability changes (Section 3.3).

Summary of New and Expanded Evidence

Study & PDF LocationUnit & DenominatorResults (Protean vs. Native / Condition Breakdown)
Barrier Safety (Table 15, p. 23)21 cells × 300 trials = 6,300 (5 Protean cells)0/300 in each of 5 Protean cells (two-sided 95% CP upper 1.2%); NAIVE 300/300 violating trials in each of its 5 cells
P1 τ²-bench Pilot (Table 30, p. 38; cf. Table 29, p. 37)20 pairs × 3 events × 2 arms = 120 rows (60/arm, 118 reach the event/control boundary)Post-event continuation 58/60 Protean vs. 56/60 native; task success 48/60 vs. 49/60
AutomationBench (Tables 31–32, p. 39)120 tasks (6 domains) × 6 conditions = 720 rowsStrict passes (/120): 71 native, 71 no-event, 75 replace, 22 revoke, 73 restart, 40 combined (39/119 event-complete)
Contract Width (Appendix E.1, pp. 37–38)26 workspace tasks × 3 seeds × 5 conditions347/382 (90.8%) valid primary rows; 355/390 separate post-hoc replay supplement (unpooled)
Frontier Utility (C1-X) (Table 2, p. 6; App. C.10, pp. 29–30)85 eligible tasks × 3 repeats = 255 pairs/model (510 episodes/model)Protean vs. native (−10 pp NI): GPT-6.1 Sol 242/255 vs. 239/255 (LCB −0.39 pp), GPT-6 Luna 226/255 vs. 229/255 (−4.31 pp), DeepSeek-V4.1-Flash-1017 (rerun) 240/255 vs. 237/255 (−1.18 pp)

Context on New Evidence. Section 3.3 frames fixed-barrier counts in Table 15 as schedule-mechanism validation rather than natural hazard rates. In separate protocols, Table 18 (p. 26) runs 300 timing injections per applicable randomized cell (CE3 Protean: 104/300), and Table 21 (p. 29) runs 300 seeded C3-L runs/arm (10,635 invocations/arm).

In P1 (Table 30), the official τ²-bench agent, environment, and tools run with orchestration injecting replacement (revision 2 in 20/20 episodes), unrelated revocation (12 refusals), or same-process Kernel close/reopen after the first successful tool-call batch while native runs undisturbed; restart records confirm same-state Kernel reopen and tool rebind in 20/20 episodes (Table 30's earlier 1 -> 2 label was a harness counter, not a Kernel capability epoch).

In the combined AutomationBench schedule (Tables 31–32; DeepSeek-V4.1-Flash-1017, Responses/high), revoking api_fetch, the sole API executor, in 32 tasks yields 0/32 strict passes (0/304 scored end-state assertions), while 39/87 tasks still completed after base64_encode was revoked.

In Appendix E.1, declaring the 3 operations behind all 131 narrow-static refusals (submission adapter 44/78 vs. 73/78 native) yields the 347/382 primary result across the five expanded-contract conditions.

All three 255-pair C1-X cohorts (Responses/high; kept separate from the original 253-pair DeepSeek cohort) use the suite-stratified, task-clustered 20,000-resample bootstrap; the lower bounds are one-sided 95% bounds for the paired task-success difference (Protean minus native).

Manuscript Organization and Scope

  • Tables 2–3 (pp. 6–7) & Denominators (Table 6, p. 9): Table 2 (p. 6) reports task utility across seven unique planners (eight cohort rows: original five plus Sol, Luna, and concurrent DeepSeek); Table 3 (p. 7) reports control and escalation results for the original five planners. Table 6 (p. 9) (full index: Table 7, p. 18) indexes all units and denominators.
  • Section 5 (Failure Cases and Deployment Implications, p. 10): Following Related Work (Section 4), Section 5 consolidates trust boundaries (TCB: 6,924/12,804 LOC in Table 36, p. 48, with 1,776 LOC in host executor and interpreter), revocation vs. task completion, and external reconciliation of interrupted non-idempotent writes (Table 28, p. 35).
  • Scope & Artifact: Guarantees hold for Kernel-mediated admissions under Assumption A1 within our evaluated benchmarks, mechanisms, and proof sketches. Artifact: https://anonymous.4open.science/r/Protean-anon-E5D1/

We are grateful to all three reviewers for the rigorous feedback that shaped the final manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.