acceptodds
Under review as a conference paper at ICLR 2027

Perdura: Stateful Code Execution Harness for Long-Horizon Agents

Abstract

Long-horizon agents must generate code, invoke tools, and incorporate feedback across extended tasks. Many existing harnesses leave part of this execution continuity in the model-visible interaction history, requiring the model to recover prior outcomes, available computational state, and unfinished work as trajectories grow. We propose Perdura, a stateful execution harness in which the model chooses executable Python actions, while the runtime owns execution continuity. Across steps, the runtime retains intermediate computational state, exact records of prior operations, and unresolved work through a persistent namespace, an execution ledger, and an explicit work lifecycle. Each model call receives only a bounded context derived from runtime state and the current observation. Across three long-horizon benchmarks and four widely used agent harnesses under the same base model (GLM-5.3 Flash), Perdura attains task success comparable to the top-scoring harness on each benchmark while using 26–53% fewer input tokens than that harness, and no harness achieves both a higher score and a lower cost on any benchmark. Trajectory analyses show how computational reuse, registered waits, and recorded execution evidence support continuation across steps.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent, Long Horizon, Harness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nniSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a context management method for coding agents called Perdura. Their motivation comes from three challenges for current coding agents: computational state reconstruction, execution evidence recovery, and unfinished-work tracking, and Perdura has four core features to tackle those challenges: it has a persistent namespace that can keep all the used variables alive, an execution ledger that can keep all execution results in memory, a work lifecycle that can wake the coding agent back up when something happens, and a control state that has a step loop for controlling everything. They compare Pedura with popular harnesses like Claude Code and Codex; though Perdura does not always perform the best, it reaches the level of Claude Code. They didn't provide detailed ablation studies to highlight each component's contribution.

Critical strengths

I would say the critical strength is the reduction of token consumption brought by the designed harness.

Critical weaknesses

There are a couple of noticeable weaknesses in this paper:

Lack of ablation studies. The paper's main claim is that it splits the execution state into a persistent namespace N, an execution ledger L, and a work lifecycle W. These contributions help the system reach performance comparable to Claude Code and achieve significantly lower token consumption. However, the authors did not provide any ablation studies to support this claim; only a case in Figure 3 shows single trajectories selected to show each mechanism working. They do not show how often each mechanism is used or how much it contributes to the score and cost. Without ablation studies, the cost reduction could equally come from Perdura’s prompt design, its tool descriptions, or simply a more aggressive truncation policy.

To me, this article reads more like a well-written engineering report than a research paper that tests a specific hypothesis. I acknowledge that solving these problems represents a good step forward for coding agents, but the motivation lacks foresight.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Reply to Reviewer nniS

The hypothesis. Long-horizon agents need execution state kept by the runtime, not carried by the transcript. Two lessons from long-lived software motivate it. A log is not a database: long-running systems maintain state explicitly, recover from checkpoints and surface in-doubt transactions instead of replaying history. A for loop is not an operating system: holding resources, suspending until an event and recovering from failure become the runtime's job. Today's harnesses differ in tools, prompts and memory files, but all loop over a transcript. As agents move to long-lived work, the transcript must be compacted and loses what earlier calls did; values and live resources must survive steps and restarts; external effects must be settled, never blindly repeated; and "tell me when X happens" must hold for days without polling. Perdura tests the hypothesis: a general CodeAct harness (tasks need not involve programming) whose runtime keeps values (N), effects and their settlement (L) and unfinished work (W) as one linked state, of which the model sees a bounded projection (Π\Pi).

Three predictions, tested (GPT-5.6 Terra, 3 runs; tables in general response 2):

  1. Explicit state keeps success at lower context cost. Unbounding the window keeps the BEAM score (36.39 vs. 36.36) at 1.34$\times$ the cost. Without values persisting across Steps (-N), ToolMaze takes 23% more calls at 31% higher cost for the same score.
  2. Explicit pending work replaces polling. With W, 82% of wakes come from the awaited Handle; without it the agent suspends on timers 6.4 times per task and takes 17% more calls, at a similar score.
  3. Explicit state survives runtime loss. Killed and restarted at 5 points, the runtime replays no Step, re-issues no request, restores every value and surfaces an interrupted transfer as uncertain; the Agent completes 18/18 tasks with no duplicate transfer.

Over all trajectories, N is used in 99% of ToolMaze tasks and W in 92% of SentinelBench tasks.

The savings are not truncation or prompt design. Prediction 1 isolates truncation: with the state fixed, withdrawing only the bounding costs 34% more at equal score, so bounding saves cost without losing what the task needs. Each ablation keeps the rest of the prompt and tools identical, so a difference comes from the withdrawn component; its teaching is how the model uses it. Against the baselines the comparison is between harnesses as a whole: Perdura provides capabilities they lack, and its prompt lets the model use them.

What this suggests beyond our benchmarks.

  • The runtime is a lever as large as the model. On one base model, scores across harnesses differ by up to 26 points (GLM SentinelBench), more than the difference between the two base models for all but one harness and benchmark.
  • Explicit state decouples cost from the horizon. A transcript grows with every step; a projection of explicit state grows with the state. Unbounding the window already costs 34% more on BEAM, and the gap should widen as tasks lengthen, which they are doing fast: the length of tasks frontier models complete doubles roughly every seven months [1].
  • The model decides; the runtime preserves. A stronger model makes better decisions, but it cannot make a crashed process resume, an unanswered transfer settle, or a wait cost nothing while idle: the runtime must commit transitions, schedule wakeups and restore state after interruption. In fault injection these held in every trial; what remained to the model was the decision whether to re-send.
  • Waiting becomes a dominant cost. Polling cost grows with wait time: without W the agent took 17% more calls on tasks whose waits last minutes; at day-scale waits polling would dominate. Event wakes make idle waiting free.

Concurrent work points the same way: explicit execution state [2], task state outside the context [3], explicit belief states [4], short compacted contexts [5]; in industry, Pi Durable [6] does not replay interrupted effects and DeepSeek's Super-agent [7] wakes the agent when background jobs end. Perdura makes this state one structure grounded in the code action. We will open-source it for controlled studies of execution state (e.g., interpreter persistence [8]).

[1] Kwa et al. Measuring AI Ability to Complete Long Software Tasks. arXiv:2503.14499, 2025.

[2] Badhe et al. SKILL.state: Scalable Long-Horizon Agent Skills. arXiv:2608.26263.

[3] Ma et al. LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks. arXiv:2608.01964.

[4] Luo et al. Beyond Memory: Harnessing Long-Horizon Agents with Explicit Belief States. arXiv:2610.01415.

[5] Belcak et al. You don't need long context. arXiv:2611.05306.

[6] Earendil. Pi Durable, released with Pi 1.0, 2026.

[7] DeepSeek. dsh-superagent-prototype, 2026.

[8] May et al. Evaluating Agents Across Runtime Contracts: When Mismatch Costs Efficiency or Quality. arXiv:2603.01209.

Reviewer_UvVhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Perdura is an agent harness in which the model writes Python actions while a runtime owns execution continuity. The runtime keeps three kinds of cross-step state separate, because they have different commit and recovery semantics (Sec. 2.2–3; Eq. 3):

  • persistent namespace (NN): values, functions, and Handles to live resources;
  • execution ledger (LL): Operation records with committed/executing/settled phases and succeeded/failed/uncertain outcomes, searchable via task.search;
  • work lifecycle (WW): WorkItems with registered waits, so the task can suspend without model polling.

A control state (Q) handles resources, scheduling, permissions and recovery. The model sees only a bounded projection of runtime state (Eq. 4).

Evaluation. Perdura is compared with Claude Code, Codex, Hermes and OpenClaw, all using GLM-5.3 Flash at low reasoning effort, on SentinelBench (100 tasks), ToolMaze (400 cases) and BEAM (400 questions). Table 1 results:

BenchmarkPerduraTop-scoring harness
SentinelBench63.0066.00
ToolMaze44.2548.50
BEAM51.3852.29
  • Perdura uses 26–53% fewer input tokens than the top-scoring harness, at 47–74% of its cost.
  • Perdura lies on each benchmark's score–cost Pareto frontier.
  • Mechanisms are illustrated with three case studies (Fig. 3) plus aggregate SentinelBench wait statistics: 63% of task time suspended, and 33 resumptions per waiting hour vs. 92/108 re-check calls for Claude Code/Codex.
Critical strengths
  1. A principled model–runtime division of responsibility, with careful side-effect semantics.
    • The runtime records Operation settlement with an explicit uncertain outcome that is never auto-retried (Sec. 3.3).
    • The namespace is deliberately not rolled back after partial failure, "because the effects of that code have already occurred" (Sec. 3.2).
  2. Registered waits demonstrably remove polling overhead. Event-driven wakeups (Sec. 3.4) are supported beyond the single case study (Fig. 3c) by aggregate SentinelBench statistics: 63% of task time suspended with no model call, and 33 resumptions per waiting hour vs. 92 and 108 re-check calls for Claude Code and Codex (Sec. 4.4).
Critical weaknesses
  1. No ablations, so the value of the proposed state separation is untested. The central claim is architectural: NN, LL and WW should be kept distinct. Yet RQ2/RQ3 are answered only by hand-picked case studies, which the authors call "illustrative ... not a controlled effect of NN" (Sec. 4.3). The research prototypes that implement individual components (CaveAgent, Scroll, OpenRath, Agent libOS) are not compared because they could not be run end to end on these benchmarks (Sec. 4.1); this makes internal ablations all the more necessary.
  2. The efficiency and "comparable success" claims rest on accounting and statistics that may favor Perdura.
    • Input tokens include cached prompt tokens, and cost "applies the standard GLM-5.3 Flash price" to all of them (Appendix A.3). Claude Code/Codex are built around append-only, cache-friendly prompts. Perdura rebuilds a fresh bounded context each step, which likely reduces prefix-cache hits. Full-price accounting therefore penalizes the baselines.
    • Perdura scores below the top harness on all three benchmarks.
    • "Comparable" is inferred from bootstrap CIs, "none excluding zero": [−15,9][-15, 9], [−8.75,0][-8.75, 0], [−4.7,3.0][-4.7, 3.0]. That is not an equivalence test.
    • Perdura also makes more model calls than the top harness on ToolMaze (3,255 vs. 2,075) and BEAM (2,780 vs. 1,707). Wall-clock time is not reported, and the number of runs per task is not stated.
Questions for authors
  1. Can you provide component ablations (no persistent NN / no ledger search / forced polling instead of WW / full history instead of Π\Pi) on at least one benchmark?
  2. What are cached vs. uncached input tokens per harness, and how does Table 1's cost column change under actual cached-token pricing? What is the wall-clock time per task?
  3. How many runs per task and harness were performed? Can you report variance over ≥3\geq 3 runs and a non-inferiority test with a pre-stated margin?
  4. Appendix A.2 states that no benchmark-specific prompts were added. Were Perdura's default prompts, tool surface, or projection Π\Pi nevertheless developed or tuned with GLM-5.3 Flash or on these benchmarks during development? How does the comparison look with another base model, ideally including each baseline's native model?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer UvVh

Q1 Ablations: every control keeps the score and changes only cost or polling (GPT-5.6 Terra, 3 runs). N is the persistent namespace, L the execution ledger, W the work lifecycle and Π\Pi the bounded projection the model sees:

Ablation you asked forOur configurationResult
no persistent N-N (fresh interpreter per Step), ToolMazesame score; +23% calls, +31% cost
no ledger search-L, BEAMno score drop; retrieval needed in 9% of probes
forced polling instead of W-W, SentinelBenchsimilar score; +17% calls, 6.4 timer suspends per task
full history instead of π\pi-Π\Pi, BEAMsame score; +34% cost

We chose each benchmark from how often Full uses the component over all trajectories, not selected cases: N in 99% of ToolMaze tasks, W in 92% of SentinelBench tasks, retrieval in 9% of BEAM probes. Tables are in general response 2.

Q2 Cache, calls and time: at actual cached pricing, Perdura still costs less than the top-scoring harness in 6 of 8 cells. Rebuilding the projection keeps the prefix cache, because the system prompt and tool declarations stay fixed and new Steps are appended within the window, so its cache rate is comparable. Perdura vs. the top-scoring harness:

Model, benchmarkCache rate (Perdura / top)CallsTimeCost
Terra WildClaw88% / 89%0.86$\times$0.65$\times$0.70$\times$
Terra ToolMaze88% / 77%1.05$\times$1.86$\times$1.01$\times$
Terra BEAM87% / 87%1.34$\times$0.76$\times$0.73$\times$
Terra SentinelBench90% / 94%0.57$\times$0.94$\times$0.60$\times$
GLM WildClaw93% / 96%0.56$\times$0.72$\times$0.38$\times$
GLM ToolMaze95% / 95%1.01$\times$1.70$\times$0.96$\times$
GLM BEAM90% / 85%1.84$\times$1.22$\times$1.11$\times$
GLM SentinelBench94% / 98%0.71$\times$1.06$\times$0.42$\times$

Perdura costs more only on GLM BEAM (1.11$\times$), where it searches memory repeatedly on the largest conversations, and about the same on Terra ToolMaze (1.01$\times$); it runs longer on ToolMaze and GLM BEAM, as every Step executes code in the sandbox; we state these limitations. Per-harness tokens, time and cost are in general responses 1a and 1b.

Q3 Runs and statistics: Perdura is superior or non-inferior to the top harness in 4 of 8 cells and significantly worse in none. All results are means over 3 independent runs, with std. We test non-inferiority to the top-scoring harness (one-sided, paired bootstrap over tasks, 10,000 resamples). The margin uses the baselines only: δ=2σ\delta = \sqrt{2}\sigma, with σ\sigma their pooled per-run std on that benchmark and model, since two independent runs of one harness differ with std 2σ\sqrt{2}\sigma. It is a property of the benchmark, not of our sample size; the 3 runs enter through the CI. Scaling the threshold by benchmark variance follows [1]; verdicts are unchanged at 2σ2\sigma.

Model, benchmarkδ\deltaΔ\Delta vs top [95% CI]Verdict
Terra WildClaw4.6-3.13 [-10.51, +4.04]inconclusive
Terra ToolMaze3.3+0.50 [-2.50, +3.42]non-inferior
Terra BEAM1.4+1.41 [-1.10, +3.93]non-inferior
Terra SentinelBench7.1-6.00 [-12.33, +0.33]inconclusive
GLM WildClaw3.5+1.56 [-5.71, +9.05]inconclusive
GLM ToolMaze3.5+8.08 [+5.17, +11.08]superior
GLM BEAM3.1-0.56 [-3.04, +1.95]non-inferior
GLM SentinelBench6.1-2.00 [-9.33, +5.33]inconclusive

Q4 Tuning and base model: no benchmark- or model-specific tuning. We debugged Perdura while running these benchmarks: we fixed runtime bugs, e.g. cases where what was executed and what the agent was shown disagreed, and prompt and tool descriptions that misled the model about how to use the runtime's capabilities or which tool documentation to disclose progressively. These fixes do raise Perdura's scores, because they make it behave as designed. They are debugging rather than tuning: no fix encodes anything about a benchmark, task or model, one configuration runs all benchmarks and both models, and we fixed the baselines' integration bugs in the same way. We will add this to Appendix A.2, and the code will be open-sourced. On GPT-5.6 Terra, Codex's native model family (3 runs):

GPT-5.6 TerraWildClawToolMazeBEAMSentinelBench
Codex66.7429.2530.2582.33
Perdura63.9844.0039.7376.33
cost ratio0.84$\times$0.87$\times$1.40$\times$0.60$\times$

[1] Bouthillier et al. Accounting for Variance in Machine Learning Benchmarks. MLSys 2021.

Thank you for the detailed responses and additional experiments. The additional results address several of my main concerns, so I have decided to raise my rating to 3.

Reviewer_MvGKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Perdura, a stateful code execution harness for long-horizon agents. It separates execution continuity into a persistent namespace, an execution ledger, and a work lifecycle, while providing the model with bounded context. Experiments on three benchmarks compare Perdura with four existing agent harnesses, reporting competitive task success with lower token costs than the highest-scoring harness in each setting.

Critical strengths
  1. Practical Motivation. The paper addresses execution-state management in long-horizon agents, where recovering previous computations, tool outcomes, and unfinished work can introduce additional overhead.
  2. System-Level Evaluation. The paper evaluates Perdura across three benchmarks against several widely used agent harnesses under the same base model. The results provide evidence of a favorable trade-off between task success and token cost in the evaluated settings.
Critical weaknesses
  1. Limited Novelty and Insufficient Ablation Studies. Several components of Perdura overlap with existing stateful agent systems, such as Scroll, Ledger, and Agent libOS. The distinction and additional contribution of their integration are not sufficiently established. Moreover, the paper lacks controlled ablations of the persistent namespace, execution ledger, and work lifecycle, making it difficult to identify their individual contributions to the reported improvements.
  2. Concerns about Evaluation Fairness and Completeness. Although the harnesses share the same base model, their configurations are not fully equivalent. In particular, OpenClaw's embedding-based memory is disabled in the BEAM evaluation, which may affect comparison fairness. Furthermore, the recovery and effect-settlement requirements (R5–R6) are not experimentally evaluated, and all results rely on a single base model, limiting the evidence for broader effectiveness.
Questions for authors
  1. Could the authors provide component-wise ablations to clarify the individual contributions of the persistent namespace, execution ledger, and work lifecycle?

  2. Why was OpenClaw's embedding-based memory disabled in the BEAM evaluation, and how might enabling it affect the comparison?

  3. Could the authors provide fault-injection or recovery experiments to validate the claimed runtime properties R5 and R6?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer MvGK

Novelty: the contribution is one execution state, not three separate stores. Every action is code, and the runtime keeps the state that executing code creates: the values and live resources it produced (N), what its calls did and whether their effects settled (L), and what is unfinished (W). The model sees a bounded projection of this state (Π\Pi), never the history. Prior systems make parts of it explicit, for other purposes. Scroll [1] keeps a persistent Python kernel and an event log, but models neither effect outcomes nor pending work. Ledger [2] records observed, modified and attempted state to govern a coding agent, without computational state or pending work. Agent libOS [3] is closest: its processes have durable message queues, wait on timers, messages and children, recover from checkpoints without replay, and never retry an ambiguous provider effect (our R5). But it is a capability-controlled substrate beneath a harness, and it does not keep the values code produces: object payloads stay runtime-local and are not restored.

Integration matters because the parts refer to each other: a wait in W is registered on a Handle in N, and an Operation in L is created by a Step that binds its result in N. They must be restored together. Separate stores have no common restore point: a namespace checkpoint without the ledger can repeat a completed effect, and a ledger without the namespace loses the values it produced. Fault injection (Q3) confirms that one recovery restores the values and the effect record together, without replay.

Q1 Component ablations: the components cut cost and polling without changing the score. We withdraw one at a time, on the benchmark where it is used (GPT-5.6 Terra, 3 runs):

Component withdrawnBenchmarkUsed by Full inScore Δ\Delta [95% CI]CallsCost
Persistent namespace (N)ToolMaze99% of tasks+0.00 [-4.33, +4.00]+23%+31%
Work lifecycle (W)SentinelBench92% of tasks-2.22 [-12.22, +7.78]+17%+11%
Ledger retrieval (L)BEAM9% of probes+4.86 [-0.14, +9.25]-23%-21%

Without W the agent polls on timers 6.4 times per task instead of waking on events. Retrieval is a fallback the agent rarely needs, since N and W carry the working state; the ledger's main role, settling effects, is confirmed in Q3. Full tables are in general response 2.

Q2 OpenClaw embedding memory: enabling it does not measurably change OpenClaw's score or the ranking.

OpenClaw, BEAM (GLM-5.3 Flash, 400 probes)ScoreΔ\Delta vs off [95% CI]
embedding memory off (main table)42.81-
embedding memory on (qwen3-embedding 0.6B)41.58-1.24 [-3.95, +1.44]

It was off because embedding memory needs a second model, while every harness here uses only the base model. All harnesses share effort, permissions, tools, environments and time limits.

Q3 R5 (effects settled, uncertain outcomes surfaced rather than retried) and R6 (recovery without replaying code): both hold in all 18 fault-injection trials. We SIGKILL the runtime at fixed points and restart it on the same isolated instance (GPT-5.6 Terra, 6 conditions ×\times 3 runs):

RequirementWhat we checkResult
R5transfer applied, reply withheld, then killedsurfaced as uncertain, not retried (3/3); the Agent checks the account, no duplicate
R5killed after the reply arrivedone transfer, reported done (3/3)
R615 killed trialsno Step replayed, no request re-issued, every value restored
R6kills mid-collection, after read-once pages, after submission9/9 correct, no repeated or lost work

The revision will replace the statement that R5 and R6 are untested with these results (details in general response 2).

Single base model: the results hold on a second model. Perdura vs. the top-scoring baseline (3 runs; full tables in general responses 1a, 1b):

ModelWildClawToolMazeBEAMSentinelBench
Terra: Perdura63.9844.0039.7376.33
Terra: best baseline67.1143.5038.3282.33
Terra: cost ratio0.70$\times$1.01$\times$0.73$\times$0.60$\times$
GLM: Perdura55.3449.1748.7365.33
GLM: best baseline53.7841.0849.2967.33
GLM: cost ratio0.38$\times$0.96$\times$1.11$\times$0.42$\times$

Perdura is first in 4 of 8 cells and on the score-cost Pareto frontier in 7 of 8, including all four under Terra. We also add WildClaw, a general real-world benchmark.

[1] Lin et al. Context as an Environment: Programmatic Context Management for Long-Horizon Agents. arXiv:2608.21690.

[2] Wang et al. Turning Interaction History into Execution State: A Runtime Layer for Long-Horizon Coding Agents. arXiv:2608.00808.

[3] Zhang. Agent libOS: A Runtime Substrate for Capability-Controlled Self-Evolving LLM Agents. arXiv:2606.03895.

Thank you for the detailed response. The additional ablations, fault-injection experiments, and cross-model evaluations have alleviated some of my concerns. However, the ledger-retrieval ablation does not demonstrate a clear benefit, and discrepancies between the original and newly reported results require further clarification. Considering the overall quality of the manuscript and the remaining concerns, I am inclined to maintain my original rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response 1a: main results, GPT-5.6 Terra

We thank the reviewers. The revision runs every harness on two base models, GPT-5.6 Terra (below) and GLM-5.3 Flash (general response 1b), with 3 independent runs per cell. It also adds WildClaw, 60 real-world tasks that combine files, the web and the shell: a general scenario beside the three targeted benchmarks (asynchronous monitoring, multi-tool orchestration, long-term memory). Tables give per-run means, with score ±\pm std over the 3 runs. Per model ×\times benchmark, bold marks the best and underline‾\underline{\text{underline}} the second-best time, cost and mean score. Calls, tokens, time and cost count each task's final attempt; cost uses cached-token pricing; time sums per-task execution time.

GPT-5.6 Terra

BenchHarnessCallsInOutCache rateTime (h)Cost (USD)Score
WildClawClaude Code1,15834.1M342K90.1%5.917.0363.60 ±\pm 5.15
WildClawCodex98526.2M339K91.0%5.5‾\underline{5.5}13.57‾\underline{13.57}66.74‾\underline{66.74} ±\pm 2.43
WildClawHermes88426.0M464K88.6%7.116.1267.11 ±\pm 2.77
WildClawOpenClaw1,41362.3M481K93.4%5.825.6062.06 ±\pm 1.11
WildClawPerdura75618.0M314K87.8%4.611.3363.98 ±\pm 0.74
ToolMazeClaude Code2,18635.4M366K86.4%4.320.1032.25 ±\pm 1.75
ToolMazeCodex2,76948.2M403K88.0%5.424.8729.25 ±\pm 2.60
ToolMazeHermes2,63327.5M361K76.5%5.021.46‾\underline{21.46}43.50‾\underline{43.50} ±\pm 2.50
ToolMazeOpenClaw1,79546.7M237K75.1%4.5‾\underline{4.5}33.1124.75 ±\pm 2.41
ToolMazePerdura2,76335.4M575K87.9%9.321.7044.00 ±\pm 1.00
BEAMClaude Code2,58272.9M571K82.4%6.344.5634.15 ±\pm 1.16
BEAMCodex1,10017.8M179K85.2%2.710.4530.25 ±\pm 0.95
BEAMHermes2,07840.7M462K72.4%3.6‾\underline{3.6}33.9431.62 ±\pm 1.24
BEAMOpenClaw1,32940.3M173K86.5%6.719.8938.32‾\underline{38.32} ±\pm 0.13
BEAMPerdura1,77926.0M280K87.0%5.114.62‾\underline{14.62}39.73 ±\pm 1.60
SentinelBenchClaude Code3,410136.9M252K96.0%20.240.3980.00‾\underline{80.00} ±\pm 7.00
SentinelBenchCodex3,20384.5M275K94.3%12.1‾\underline{12.1}28.8782.33 ±\pm 4.04
SentinelBenchHermes1,56335.3M264K91.6%13.815.5470.33 ±\pm 5.51
SentinelBenchOpenClaw4,085152.1M281K95.8%12.845.1967.67 ±\pm 2.08
SentinelBenchPerdura1,82537.7M228K89.7%11.317.26‾\underline{17.26}76.33 ±\pm 3.21

AuthorsOpenReview ↗

General response 1b: main results, GLM-5.3 Flash

GLM-5.3 Flash

BenchHarnessCallsInOutCache rateTime (h)Cost (USD)Score
WildClawClaude Code1,67665.2M535K96.2%7.8‾\underline{7.8}2.5247.21 ±\pm 2.51
WildClawCodex2,07684.9M483K89.0%8.73.9143.86 ±\pm 2.31
WildClawHermes1,78342.7M610K93.4%8.41.93‾\underline{1.93}51.26 ±\pm 3.07
WildClawOpenClaw2,679116.3M632K96.4%8.64.3053.78‾\underline{53.78} ±\pm 1.98
WildClawPerdura1,50438.2M349K92.7%6.21.6555.34 ±\pm 1.30
ToolMazeClaude Code2,50547.6M315K98.6%7.4‾\underline{7.4}1.6737.50 ±\pm 1.50
ToolMazeCodex3,02236.5M306K97.3%12.01.3731.75 ±\pm 4.51
ToolMazeHermes3,00541.7M414K95.3%4.51.6941.08‾\underline{41.08} ±\pm 1.13
ToolMazeOpenClaw2,29877.8M269K96.8%7.62.7737.83 ±\pm 1.01
ToolMazePerdura3,03339.5M416K95.1%7.81.62‾\underline{1.62}49.17 ±\pm 1.42
BEAMClaude Code1,66462.7M532K85.0%6.03.27‾\underline{3.27}49.29 ±\pm 1.50
BEAMCodex2,964101.6M348K96.6%4.0‾\underline{4.0}3.6334.43 ±\pm 0.57
BEAMHermes2,45955.3M417K95.1%4.02.1939.22 ±\pm 0.78
BEAMOpenClaw1,560101.7M209K95.0%8.63.7747.05 ±\pm 3.99
BEAMPerdura3,05480.7M405K89.6%7.33.6348.73‾\underline{48.73} ±\pm 1.25
SentinelBenchClaude Code2,939118.7M148K98.1%20.23.9167.00‾\underline{67.00} ±\pm 2.65
SentinelBenchCodex5,117112.9M304K97.2%16.13.9241.33 ±\pm 2.08
SentinelBenchHermes2,09638.2M226K96.8%15.71.4160.67 ±\pm 4.62
SentinelBenchOpenClaw4,049175.1M264K98.3%12.25.7467.33 ±\pm 6.51
SentinelBenchPerdura2,88661.4M269K94.1%13.0‾\underline{13.0}2.41‾\underline{2.41}65.33 ±\pm 8.02

Perdura is built for reliable, low-cost execution. It is in the top two for cost in 6 of 8 cells and for score in 5 of 8, more often than any baseline (at most 5 and 3), and in the top two on both in 3 cells, where no baseline exceeds one. It has the top score in 4 of 8 cells and is on the score-cost Pareto frontier in 7 of 8. Where it trails the top harness in score, the gap is smaller than the typical difference between two runs of one harness (δ\delta, UvVh Q3). On GLM BEAM it searches memory repeatedly on the largest conversations, so it makes more calls and costs 1.11$\times$ the top harness.

AuthorsOpenReview ↗

General response 2: component ablations and fault injection

What is tested. Perdura's runtime keeps three components of execution state: the persistent namespace N (values and live Handles kept across Steps), the execution ledger L (every call and whether its effect settled) and the work lifecycle W (WorkItems and registered waits that wake the task); at each Step the model sees a bounded projection Π\Pi of them. Fault injection tests the paper's two runtime requirements: R5, effects are settled and an uncertain outcome is surfaced to the agent, never retried; R6, after a crash, execution resumes from saved state without replaying code.

Where each component is used. Share of tasks in which Full uses it, over all GPT-5.6 Terra trajectories (3 runs):

ComponentWildClawToolMazeBEAMSentinelBench
N: reads a value from an earlier Step95%99%31%100%
W: registers a wait37%6%1%92%
L: retrieves evicted records1%0%9%2%

We test each component where it is used. N goes on ToolMaze, where W is rarely used, so -N isolates N; SentinelBench uses N and W together. Π\Pi is active in every Step; we test it on BEAM, the long-term memory benchmark, where bounding the context is most likely to lose needed information.

Ablations. Each configuration withdraws one component and its prompt teaching; tools are identical except that -W drops the work tools (GPT-5.6 Terra, 3 runs, fixed pre-registered subsets):

BenchmarkConfigScoreΔ\Delta vs Full [95% CI]Cost/run (USD)Calls/task
ToolMaze (100)Full42.67 ±\pm 1.53-5.327.5
-N42.67 ±\pm 2.08+0.00 [-4.33, +4.00]6.98 (+31%)9.2 (+23%)
SentinelBench (30)Full72.22 ±\pm 8.39-4.7717.1
-W70.00 ±\pm 5.77-2.22 [-12.22, +7.78]5.30 (+11%)20.0 (+17%)
BEAM (80)Full36.39 ±\pm 1.24-4.276.9
-Π\Pi36.36 ±\pm 1.42-0.03 [-5.57, +5.27]5.74 (+34%)8.5 (+22%)
-L41.25 ±\pm 1.19+4.86 [-0.14, +9.25]3.37 (-21%)5.4 (-23%)

The components change how the work is done, not whether it succeeds: withdrawing N, W or Π\Pi keeps the score but raises calls and cost. Without W the agent suspends on timers 6.4 times per task; with W, 82% of wakes come from the awaited Handle. -L withdraws only retrieval, a fallback needed in 9% of BEAM probes; the ledger's settlement role is tested below.

Fault injection (R5, R6). We SIGKILL the runtime at a fixed point and restart it on the same isolated instance (GPT-5.6 Terra, 6 conditions ×\times 3 runs):

ConditionCorrectAfter restart
R5: transfer applied, reply withheld, killed3/3one transfer, reported uncertain
R5: killed after reply delivered3/3one transfer, reported done
R6: killed mid-collection3/3completes, one report
R6: pages readable once, killed after 3 of 53/3keeps delivered pages
R6: killed after report accepted3/3no resubmission
Reference: no kill3/3-

R5: the runtime surfaced the interrupted transfer as uncertain (3/3); the Agent then checked the account instead of re-sending, and never duplicated a transfer or fabricated a receipt. R6: across the 15 killed trials the runtime replayed no Step, re-issued no request, restored every value bound before the kill (12/12 where a Step had completed) and did not re-run the interrupted Step.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.