acceptodds
Under review as a conference paper at ICLR 2027

LongTool-Bench: Benchmarking Lifecycle-Aware Adaptation in Long-Horizon Tool Agents

Abstract

Long-horizon tool agents execute interdependent actions that modify persistent application state, often across applications. Yet tool capabilities, interfaces, permission boundaries, execution conditions, or identities may change after consequential effects have been committed. Successful adaptation must revise invalid assumptions while preserving valid progress, avoiding duplicate effects, and satisfying revised constraints. Existing benchmarks study long-horizon execution, stateful interaction, or unreliable tools, but rarely combine controlled in-episode evolution, retained side effects, and executable recovery obligations within an unfinished task. We introduce LongTool-Bench, comprising 200 AppWorld tasks, 400 evolution contracts, and 38,400 episodes across sixteen models. It injects changes at semantically eligible boundaries, preserves committed state, and varies access to change information while separately measuring task correctness, event exposure, contract recovery, prohibited behavior, recovery effort, and declaration reliability. Model-independent executable witnesses establish chain feasibility from retained boundary states. Relative to Hidden, Oracle-Docs improves event-chain resolution by 23.9 pp but Official Success by only 5.0 pp; moreover, 86.8% of chain-resolved episodes still fail the original task. Category analyses and worker ablations further separate failures in exposure, recovery, state protection, and terminal completion. These results show that contract recovery alone does not ensure end-to-end adaptation: dependable agents must jointly track evolving contracts, committed state, unfinished dependencies, and terminal task evidence.

Primary area:
datasets and benchmarks
Keywords:
Long-Horizon, Tool-Using Agents, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uCsPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

LongTool-Bench introduces controlled changes to tools in 200 AppWorld tasks that require persistent state and extended sequences of actions. Its protocol verifies executable recovery requirements, changes calls at meaningful boundaries while keeping earlier state updates, and tests 16 models with unchanged tools, concealed changes, visible changes, or oracle documentation. It records completion of the original task separately from resolution of the tool-change events.

Critical strengths
  1. Section 3 specifies that earlier completed effects remain in place when a call is interrupted, and it verifies that recovery remains possible from that point. This creates a defined and realistic test of tool changes beyond simply injecting an error response.
  2. The matched evaluation reports distinct outcomes for event resolution and the original objective. Of 7,008 evolving episodes that finish their complete event chains, just 924 also pass the task assertions (Section 4.2). The separate counts make clear that handling a local change does not necessarily complete the task.
Critical weaknesses
  1. Across models, success without tool changes ranges from 1.33% to 18.67% (Table 3), indicating substantial difficulty in the base tasks. For each model, the authors should additionally analyze tasks solved consistently in the unchanged condition and report paired uncertainty, while retaining the results for the full task set. This would better estimate the portion of failures associated with adaptation to change.
  2. A change is encountered only when the agent reaches a suitable boundary. Consequently, the hidden, visible, and oracle-documentation settings may expose different collections of episodes (Sections 3.3 and 4.4). Recovery conditional on exposure and success within exposed categories do not therefore compare identical cases. Although the paper mentions this issue, its discussion still gives considerable weight to those differences. Replaying conditions from the same recorded state before an event, or estimating bounds for the selection effect, would support a more precise conclusion about the information provided.
Questions for authors
  1. On tasks each model solves in all three unchanged repetitions, what are the paired outcomes under tool changes? How does restricting attention to these tasks affect the observed success gap?
  2. For a subset of encountered changes, could all three information settings start from the same saved state immediately before the event? This would allow recovery rates to be compared for identical events.
2 comments
AuthorsOpenReview ↗

Response to Reviewer uCsP (1/2) - Three-Clean-success subset

Q1 Outcomes on tasks solved in all three Clean repetitions

In the supplementary historical Core-50 evaluation, all selected tasks fail under each changed condition at the available matched repeat0. We completed Clean repeats1/2 for every task that succeeded in Clean repeat0, adding 12 episodes, all officially successful. The other 144 model/task combinations across the three historical models have a Clean repeat0 failure, determining their exclusion from the three-success intersection. Membership is therefore determined throughout each model's original 50-task scope.

Selection requires Clean success in repeats0/1/2. The paired changed-condition comparison uses repeat0. The following values apply separately to Hidden, Observable and Oracle-Docs, whose task success counts are identical in these records. Gaps are changed-condition OS minus Clean OS.

Historical BF16 modelSelected tasks / familiesSelected Clean → changed successesFull-set Clean → changed successesFull-set gap, pp [95% CI]Selected gap, pp [95% CI]
Mistral-Small-24B-Instruct-25014 / 34/4 → 0/44/50 → 0/50−8.00 [−18.75, 0.00]−100.00 [−100.00, −100.00]
Qwen3-32B2 / 22/2 → 0/22/50 → 0/50−4.00 [−10.64, 0.00]−100.00 [−100.00, −100.00]

Gemma-3-27B-IT has Clean and changed successes of 0/50, yielding a full-set gap of 0 pp [0,0], an empty strict subset and undefined restricted statistics. The separate NF4 six-task scopes for Qwen3-8B/14B also have empty strict subsets because every task has a Clean failure.

Intervals use 10,000 paired scenario-family bootstrap resamples. For Q1, we sample each model's 36 original families with observed subset membership fixed. Selected-gap intervals use the 9,576 defined resamples for Mistral and 8,764 for Qwen, with 424 and 1,236 undefined selected denominators, respectively. Their zero width reflects identical losses among only three and two represented families. Selection fixes selected Clean OS at 100%, so the gap change describes the selected comparison.

The added runs use the frozen historical revisions and worker, BF16 weights, greedy temperature0 decoding, and the original generation budgets and context rules. All 12 reproduce their corresponding repeat0 output sequences, establishing execution consistency under this configuration. The historical v33 worker retains its recovery serializer system header after accepted recovery, so the measured losses include that implementation's recovery and continuation policy.

AuthorsOpenReview ↗

Response to Reviewer uCsP (2/2) - Same-state recovery comparisons

Q2 Recovery from the same saved state

Yes. We completed three-condition recovery comparisons from the same recorded first-event boundary. Three fresh processes replay each saved executable prefix and verify the database, persistent shell, controller/tool state, agent context, pending call, counters, remaining budgets and controllable RNG against the source. The endpoint is resolution of the target first event.

The separate Qwen3-8B/14B NF4 validation uses the first six tasks in frozen metadata order, each from a different family, as a nested interim scope added after data collection began. The primary scope comprises 12 tasks. The table uses source repeat0 and one branch per condition for each captured source.

Same-state NF4 validationComplete triads / familiesHidden resolvedObservable resolvedOracle-Docs resolvedObservable−Hidden wins / ties / losses
Qwen3-8B, initial NF4 replay6 / 60/66/66/66 / 0 / 0
Qwen3-14B, initial NF4 replay4 / 41/44/44/43 / 1 / 0
Qwen3-14B, independent serializer repair1 / 10/11/11/11 / 0 / 0

Resampling the represented families gives Observable−Hidden differences of 100 pp [100,100] for 8B and 75 pp [25,100] for 14B in the initial replay, with identical Oracle-Docs−Hidden differences. These estimates have six and four families, respectively. The 8B interval is degenerate because all six paired differences agree.

The twelve original candidate sources comprise ten complete triads, one normal nonexposure and one dictionary-view serialization failure. The independent repair replays saved responses and code with zero new source inference calls. Partial databases and the pending API have directly saved comparison objects. The shell is reconstructed from saved code and RNG, and all three repaired branches match the complete reconstructed CPU boundary. Hidden exhausts eight recovery calls, while Observable and Oracle-Docs each resolve the target in one.

This validation uses NF4 weights, BF16 compute, temperature0.2 and a 16,384-token context. Hidden receives an event identifier and validator errors, differing from Section 3.3. The comparison concerns the complete implemented feedback policies, including initial observations, action guidance and retry feedback.

A historical Core-50 repeat0 sensitivity retains 148 blocks whose first planned event fires in all three conditions, spanning 36 families per model. Observable−Hidden recovery differences are 58.00 [43.75, 71.74] pp for Qwen3-32B, 82.00 [70.00, 92.31] for Gemma-3-27B-IT and 54.17 [40.82, 67.39] for Mistral-Small-24B. Oracle-Docs−Hidden matches these values for Qwen/Mistral and is 94.00 [86.54, 100.00] for Gemma. This post-treatment-conditioned common-exposure analysis complements the saved-state experiment's stronger database, shell and context controls.

Conditional LR and category-exposed OS use condition-specific fired/exposed sets. Full-episode OS and strict ECR use every evaluated episode and include routing, exposure, recovery and continuation. Strict ECR requires all planned events to fire and resolve. The controlled results support recovery differences for the evaluated feedback policies on captured encounters. The share of overall task failures attributable to adaptation requires further identification. We will incorporate the completed tables and feedback/denominator definitions into the revised manuscript, with category-level mechanisms stated as hypotheses.

Reviewer_Fq9hOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LongTool-Bench, a benchmark for lifecycle-aware adaptation when long-horizon tool agents encounter tool changes while a task remains unfinished. It covers capability, interface, permission, execution-condition, and replacement changes. Events activate at semantically eligible boundaries, interrupt the triggering call before execution, and preserve all previously committed application state. Agents must therefore satisfy the executable recovery obligations, avoid duplicate effects and prohibited operations, and continue toward the unchanged original task assertions. The benchmark consists of 200 AppWorld tasks spanning 80 scenario families, including 120 cross-application workflows, together with 400 evolution contracts. The main evaluation runs 16 models under four matched conditions and three repeat keys, yielding 38,400 episodes. Model-independent executable witnesses establish chain feasibility from retained boundary states, while task correctness, event exposure, contract recovery, prohibited behavior, recovery effort, and declaration reliability are measured separately.

Critical strengths
  1. The benchmark addresses a relevant gap, and the paper is easy to follow. It combines in-episode contract changes, retained side effects, and executable recovery obligations. Original task assertions remain unchanged. Prior work rarely studies these aspects together. The setting reflects practical recovery constraints: agents must preserve committed state and avoid duplicate effects.
  2. The metrics support more detailed diagnosis. OS, ECR, and TLS distinguish task correctness from lifecycle execution. Other metrics capture prohibited behavior, recovery effort, and declaration reliability. This separation helps locate failures across activation, adaptation, and terminal verification. It could also guide future improvements.
Critical weaknesses
  1. Recovery requires structured lifecycle actions and machine-checkable evidence. Execution resumes only after validation. Official Success is scored separately, but the practical basis of the recovery requirements needs clarification. Which are necessary for correct, safe task completion, and which are protocol choices? A real-world tool-change example with a trajectory visualization would help establish this connection.
  2. The paper shows that contract recovery does not ensure task success, but the causes remain unclear. Figure 5 illustrates incomplete dependency repair and unsupported completion without estimating their prevalence. Models also frequently fail under Clean. Analysis of actual trajectories would help distinguish adaptation bottlenecks from failures already present under Clean.
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Fq9h (1/2) - Practical obligations and historical trajectories

We thank the reviewer for recognizing the benchmark's diagnostic value. We connect recovery requirements to practical obligations and examine actual trajectories with Clean backgrounds.

1. Which recovery requirements are practically necessary, and which are protocol choices?

Compatible, authorized continuation that accounts for committed progress is the practical obligation. Lifecycle names, explicit records, the contract-specified sequence and the resumption gate are our chosen auditable recovery criterion.

Practical conditionLifecycle recordInspected Core-50 check
Call a changed interface compatiblymigrate_toolActive event ID, old/new tool identities and argument map
Use an allowed tool and permission scoperequest_permissionTool and scope against the active event
Account for retained state and dependenciescheckpoint, compensateTable 9 specifies contract-dependent evidence or corrective operations. The generic checker checks event IDs and action order.
Declare adaptation and an acceptance pointacknowledge_evolution, invalidate_plan, resumeEvent ID, required sequence and action-specific arguments before reopening the gate

Executable witnesses establish constructive feasibility under the encoded contract from validated retained boundaries (§3.2, Appendix B.1). ECR measures full-chain exposure and resolution, while OS independently checks the official task assertions. State/dependency evidence has the scope of the implemented predicate (Appendices B.2 and E.2).

Slack's documented files.upload retirement supplies a practical example. Its replacement obtains an upload URL/file ID, transfers bytes and calls completeUploadExternal. Without finalization, the uploaded file and metadata are discarded. The following multi-file workflow is illustrative.

Earlier finalized files retained → pending upload affected by the retirement → obtain upload URL/file ID → transfer bytes → finalize → use valid file IDs downstream.

Finalization and valid downstream references are completion conditions. Lifecycle records provide their auditable representation, with retained files determining the remaining work. Slack migration notice, finalization semantics

The supplementary Core-50 worker supplies Hidden with an event identifier and validator retry errors. This feedback boundary differs from the description in §3.3 and will be documented by implementation version.

2. What do actual trajectories reveal beyond the recovery-completion gap?

Actual trajectories show continuation-prompting sensitivity under these recorded configurations, alongside task difficulties visible in Clean. Figure 5 presents schematic, overlapping patterns (§4.7), and the ECR–TLS gap counts resolved chains ending in task failure (Appendix E.2).

On historical Core-50 v33 task 425a494_3, Mistral-Small-24B-Instruct-2501, repeat 0, the logged sequence is API attempt 96 replacement → model call 5 accepted recovery → calls 6–64, 59 parseable replies without code → final 1/2 assertions. No further API or completion request occurs. Matched Clean passes 2/2 in repeats 0/1/2, with 138 API attempts at repeat 0.

All 12 additional historical BF16 Clean runs (repeats 1/2) succeed. Within each model's 50-task scope, Mistral-Small-24B has 4 tasks/3 families and Qwen3-32B has 2 tasks/2 families passing Clean repeats 0/1/2. Each of Hidden, Observable and Oracle-Docs has OS 0/4 and 0/2, respectively, at matched changed repeat 0. Gemma-3-27B-IT's selected set is empty and restricted rates are undefined. All 12 new output sequences match repeat 0 under greedy temperature 0, supporting fixed-configuration execution consistency. The historical v33 recovery path retains the recovery serializer header, so these losses include that continuation policy.

AuthorsOpenReview ↗

Response to Reviewer Fq9h (2/2) - Continuation controls and failure patterns

2. What do actual trajectories reveal beyond the recovery-completion gap? (continued)

Fresh NF4/BF16 full-episode header comparisons cover six tasks from six families per model at repeat 0. The six-task interim scope was added after collection began using frozen metadata order, within the twelve-task primary scope. Results use the 14 November, 14:34:54–14:35:26 UTC+8 snapshot.

ModelStrict ECR, restore / preserveOS, restore / preserveRestore-minus-preserve mean API attempts
Qwen3-8B6/6 / 4/60/6 / 0/6+165.67
Qwen3-14B6/6 / 4/60/6 / 0/6+70.67

Paired assertion coverage is unchanged. Restore progresses nine events per model against preserve's six, increasing chain progression and API cost.

We mechanically audited all 24 Observable traces in these twelve pairs. Every episode fires an event and accepts recovery, giving six applicable episodes per model/policy. From first accepted recovery to termination, all 12 preserve episodes have zero execution requests and zero API requests, with no code-containing reply across 572 actor responses. Every restore episode has subsequent execution and API requests. Post-recovery completion requests occur in 2/6 restored-header 8B episodes and 5/6 restored-header 14B episodes, totaling 28 requests, all with incomplete neighboring evaluations. Preserve has zero such requests. These are mechanical behavior and timing counts in the fixed interim frame.

For Qwen3-14B's notes-backup task 6c2c621_3, repeat 0, both configurations accept interface recovery at seq391 and have ECR=1.

Continuation-header policyActor replies after accepted recoveryExecution requests / API attempts after recoveryFinal assertions
Restore execution header32 parseable replies containing code23 / 261/8
Preserve recovery serializer header32 parseable replies without executable code0 / 01/8

Clean ends at 2/8. Six file-backup checks fail in all three runs, and both evolving runs have syntax and authorization errors before the event. Structured annotations of the two complete target traces give the following observations.

Audited patternRestore, one reviewed episodePreserve, one reviewed episode
Pre-event task errorPresentPresent
Recovery-request errorAbsentAbsent
No executable continuation after accepted recoveryAbsentPresent
Unsupported completionPresent, three post-recovery requests and one pre-event requestPresent, one pre-event request
Post-recovery dependency errorUnknownUnknown
Disallowed repeated or lost committed effectUnknownUnknown

Restore's declarations at seq395, 431 and 542 have adjacent evaluations of 1/8. Preserve's only declaration, seq357, precedes the event. These two reviewed targets from one task/family belong to the frozen 336-unit annotation sample. The 24-trace mechanical census has its separate six-task denominator. One Codex AI review agent performed the single-pass annotations, with zero human reviewers. Clean supplies background excerpts.

A named Qwen3-8B diagnostic on 09b0ee6_1, repeat 0, records public-reference support reaching 2/2 assertions with 327 API attempts, model recovery reaching 1/2 with 555, and Clean reaching 1/2 with 139. Both evolving runs accept the same recovery sequence. Their first continuation requests share the effective seed, execution header and 276 remaining actor calls, while their user-message contexts differ. The additional pass is answer-match, with no-model-change passing initially. This is a configuration-level paired outcome for public-action and continuation-context support. The case was selected after observing success, outside the frozen 336-unit probability sample.

We will complete a pattern audit over the fixed twelve-task scope of 48 Observable restore/preserve traces and 24 matched Clean backgrounds. Unsupported completion will require official assertions to remain incomplete after processing the claim. Dependency-error labels will require a change-related invalid dependency, its downstream use and API/state evidence. Tables will report event/recovery timing, unknowns, overlapping labels, exposure/acceptance and model/family/review denominators, and actual single-agent AI provenance. This census retains its own denominator alongside the 336-unit probability sample.

We will integrate these mappings, timelines and count tables into the revised manuscript. Core-200 statistics retain their paper-reported identity, and historical v33 Core-50 and fresh NF4 results retain their respective scopes. The evidence supports these recorded diagnoses. Population failure-cause shares and general state-damage or dependency-repair conclusions require their corresponding evidence.

Thanks for the detailed clarifications, I will maintain my positive score.

Reviewer_7UQKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LongTool-Bench, an evaluation framework designed to investigate whether long-horizon tool agents can adapt to in-episode tool-contract changes while preserving committed state and completing the original task. The framework constructs 200 stateful tasks from AppWorld with 400 executable evolution contracts spanning five categories of tool change, and evaluates sixteen models under four matched information conditions (Clean, Hidden, Observable, Oracle-Docs) across 38,400 episodes. Through matched-condition contrasts, category-level analyses, and mechanism ablations, the authors examine how agents respond to change information and whether contract recovery translates into task completion. The results show that although richer information substantially improves contract recovery (event-chain resolution +23.9 pp), official success improves only marginally (+5.0 pp), with 86.8% of chain-resolved episodes still failing the original task. The paper highlights an important distinction between lifecycle compliance and end-to-end task success.

Critical strengths

Meaningful research problem. The paper investigates an important aspect of agent reliability — adapting to tool evolution after consequential effects have been committed — that is not adequately captured by conventional task-success metrics. Distinguishing contract recovery from task completion provides a useful perspective for understanding agent behavior.

Well-structured experimental framework. The separation of executable contracts, controller, and terminal evaluator helps isolate lifecycle compliance from downstream execution. The matched information conditions and model-independent feasibility witnesses provide stronger evidence than purely observational trajectory analysis.

Informative empirical findings. The paper identifies several interesting failure patterns, including the recovery–completion asymmetry (information helps contract recovery far more than task completion), the 86.8% chain-resolved-but-failed rate, the replacement-category exposure–recovery gap, and the calibration tension whereby violations rise as recovery improves. These findings provide useful insights into the limitations of existing LLM tool agents.

Careful reporting and analysis. The authors define metric denominators precisely, use matched blocks and family-clustered bootstrap procedures, distinguish descriptive comparisons from causal claims, and acknowledge limitations related to task construction, single-substrate evaluation, and experimental generalizability.

Diagnostic value of the mechanism ablations. The Core-50 ablations (repeat-effect protection, evaluator feedback, history compression, deterministic lifecycle actions) and the three trajectory failure patterns provide a concrete account of where and why adaptation fails, supporting the framework's diagnostic utility.

Critical weaknesses

Limited methodological novelty. The primary contribution is an evaluation protocol layered on an existing benchmark — the 200 tasks are a filtered subset of AppWorld and no new environment or algorithm is introduced. Although the research problem is relevant, the paper does not sufficiently demonstrate how its evaluation methodology advances beyond existing work on dynamic tools, recovery benchmarks, and agent stopping decisions, nor does it show that the benchmark can discriminate between better and worse adaptation methods.

Insufficient validation of the headline judgments. The central empirical claims (23.9 pp vs. 5.0 pp; 86.8%) are point estimates without confidence intervals in the main matrix, despite a bootstrap procedure being defined and applied only to select ablations. In addition, the shared lifecycle-action vocabulary and deterministic actions may themselves scaffold event discovery (the A5 ablation fires 83 vs. 48 events), so the measured "recovery" partly reflects protocol support rather than the model's own adaptation.

Potential confounding factors in the controlled experiments. The matched conditions vary both information content and the exposed episode population — fire rates rise with information — so without a matched-exposure sub-analysis it is difficult to attribute gains specifically to information sufficiency. Moreover, clean official success is only 1.3–18.7%, leaving the central "recovery does not ensure completion" claim potentially confounded with task difficulty rather than recovery insufficiency.

Limited connection to downstream task performance. The paper does not establish that improving boundary-control or recovery decisions would raise task success: chain-resolved failures are not decomposed per episode into recovery-induced damage vs. difficulty unrelated to the evolution, and no intervention shows that better recovery leads to better completion. Additionally, the privileged Oracle-Docs condition has access to the authoritative contract specification unavailable to ordinary conditions, making its performance gap difficult to interpret as a direct measure of controller deficiency.

Limited reproducibility and robustness evidence. The mechanism ablations cover only 50 scenarios with one model and one repeat key; hosted API models are not seeded, so repeat keys do not guarantee stable outputs; and the schema-neutral control is excluded from the main estimates. The calibration-informed task selection (200/250 tasks, ≥80 reference calls) may affect the stability and generalizability of the reported results.

Questions for authors

Can the authors provide family-clustered bootstrap intervals for the headline contrasts (23.9 pp ECR vs. 5.0 pp OS; the 86.8% figure), and how sensitive are the main conclusions to uncertainty and to the excluded schema-neutral control?

Could the authors introduce a matched-exposure sub-analysis — episodes where the event fires under all three conditions — to distinguish the effect of information sufficiency from the effect of population shift?

Would correcting recovery decisions improve actual task success rates or execution efficiency? Can the authors provide an end-to-end decomposition showing how many chain-resolved failures are caused by recovery itself versus task difficulty?

Why do strict ECR and TLS penalize episodes that legitimately bypass the event chain (OS = 1, ECR = 0)? What share of OS-successful episodes avoid all events, and would the conclusions change if avoidance counted as successful adaptation?

How do the authors distinguish genuinely necessary recovery from protocol scaffolding, given the shared lifecycle vocabulary and deterministic lifecycle actions, and would the findings transfer to other substrates (GUI, terminal, or non-API environments)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7UQK (1/3) - Uncertainty and exposure controls

Thank you for recognizing the importance of adaptation after committed effects. LongTool-Bench combines retained pre-call boundaries, executable boundary-prefix witnesses checking joint feasibility of recovery and retained progress, and a side-effect ledger with independent AppWorld task assertions. Table 1 credits C-World with stateful within-task changes and separate recovery scoring. Our protocol distinguishes recovery, continuation, task progress, completion, and cost.

Q1. Headline uncertainty and protocol sensitivity

The original headline quantities remain descriptive point estimates for the evaluated matrix. Oracle-Docs improves ECR by 23.9 pp and official success (OS) by 5.0 pp relative to Hidden. Of 7,008 chain-resolved episodes, 924 satisfy the task assertions, giving 86.8% task failure. The supplementary validation below supplies intervals for its own scope (Figure 3, Appendix E.4).

Fresh validation uses pinned Qwen3-8B/14B, NF4 weights/BF16 compute, repeat 0, and the 14 November 14:34:54–14:35:26 (UTC+8) snapshot. Within the twelve-task primary scope, a six-task interim scope was added after collection began using frozen metadata order. Its six families have H1/H2/H3 counts 5/1/0, three cross-app tasks, three held-out tasks, and reference lengths 82–277 calls. The matrix is complete at 84/84 cells across seven configurations and two models.

ModelHidden / Observable / Oracle-Docs ECRObservable−Hidden ΔECR, pp [95% CI]Oracle-Docs−Hidden ΔECR, pp [95% CI]
Qwen3-8B0/6 / 6/6 / 6/6+100.00 [100.00, 100.00]+100.00 [100.00, 100.00]
Qwen3-14B1/6 / 6/6 / 6/6+83.33 [50.00, 100.00]+83.33 [50.00, 100.00]

OS is 0/6 in all four semantic/restore conditions including Clean, for both models. Both contrasts have ΔOS 0.00 [0.00, 0.00], so ΔECR−ΔOS equals the displayed ΔECR. Intervals use 10,000 paired family-bootstrap draws, preserving conditions and models and computing both endpoints jointly. Oracle-Docs−Observable ECR, OS, and assertion-coverage (AC) differences are zero. All 25 chain-resolved episodes among 36 evolving episodes fail the task, giving 100.00% [100.00, 100.00]. Zero-width empirical intervals reflect uniform observations in six families. Small coverage and the Clean floor constrain inference about completion effects and causes.

Neutral action identifiers use the same restored execution header. Both configurations resolve every chain, with OS=0, in six Observable pairs/families per model.

ModelNeutral−semantic ΔAC, pp [95% CI]Δmean API attempts [95% CI]
Qwen3-8B+1.52 [0.00, 4.55]−98.00 [−222.67, 9.50]
Qwen3-14B0.00 [0.00, 0.00]+6.33 [−15.00, 41.50]

Renaming preserves recovery in this fixed frame. Information comparisons remain conditional on the shared lifecycle interface.

Q2. Information gains and exposure selection

Historical Core-50 v33 Full, repeat 0, retains blocks whose first planned event fires in all three conditions, spanning 36 families per model.

ModelHidden / Observable / Oracle-Docs resolvedObservable−Hidden, pp [95% CI]Oracle-Docs−Hidden, pp [95% CI]
Qwen3-32B21/50 / 50/50 / 50/5058.00 [43.75, 71.74]58.00 [43.75, 71.74]
Gemma-3-27B-IT3/50 / 44/50 / 50/5082.00 [70.00, 92.31]94.00 [86.54, 100.00]
Mistral-Small-24B-Instruct-250122/48 / 48/48 / 48/4854.17 [40.82, 67.39]54.17 [40.82, 67.39]

Intervals use 10,000 paired family resamples with membership fixed. Event ID, triggering-call index, old endpoint, and argument map agree across all 148 blocks. Selection remains post-treatment, and application state/history can differ. Full-episode ECR/OS use all episodes. LR and category-exposed OS use condition-specific exposed sets (Appendix E.1–E.2).

Saved-state comparisons reconstruct common application, shell, agent-history, controller, and RNG state. Twelve candidates yield ten initial triads, one nonexposure, and one reconstruction failure. Qwen3-8B has six triads/families with Hidden/Observable/Oracle-Docs resolution 0/6, 6/6, 6/6 and Observable−Hidden +100.00 pp [100.00, 100.00]. Qwen3-14B has four with 1/4, 4/4, 4/4 and +75.00 pp [25.00, 100.00]. Oracle−Hidden has identical contrasts under 10,000 paired family resamples. A separately versioned serializer repair adds one 14B triad with 0/1, 1/1, 1/1.

The endpoint is target-first-event recovery under the implemented feedback packages. Historical v33 and fresh NF4 Hidden receive observed_error.event_id and validator argument_errors, extending §3.3's description. Observable adds notice, transition details and required-action names, and Oracle-Docs adds documentation.

AuthorsOpenReview ↗

Response to Reviewer 7UQK (2/3) - Intervention outcomes and Clean background

Q3. Does better recovery improve task execution?

The controls change continuation, task progress, and cost, with limited evidence for a stable OS improvement.

Intervention outcomes

Fresh public-action reference and semantic recovery share the restored execution header, with six tasks/families per model. For Qwen3-8B, reference reaches OS on 1/6 against 0/6, giving one win, zero losses, and five ties. ΔOS is +16.67 pp [0.00, 50.00], exact two-sided McNemar p=1.0, ΔAC +8.33 pp [0.00, 25.00], and Δmean API attempts −64.17 [−166.83, 31.67]. ECR falls from 6/6 to 5/6, Δ−16.67 pp [−50.00, 0.00]. Qwen3-14B has zero paired ECR, OS, and AC differences, with Δmean API attempts +15.17 [−8.17, 48.00].

On the OS win, 09b0ee6_1, reference passes 2/2 assertions with 327 API attempts against semantic's 1/2 with 555. Both accept the same recovery sequence and use the execution header, with different user-message and working-memory contexts. The added assertion concerns answer matching. Clean also passes 1/2. This post-selected case illustrates the complete support configuration. Its narrative received one Codex AI pass, zero human reviews, and tool-based trace-reference checks.

The header control gives ECR 6/6 under restore versus 4/6 under preserve per model, Δ33.33 pp [0.00, 66.67], with OS 0/6 and zero paired AC differences. Δmean API attempts is +165.67 [45.33, 316.00] for 8B and +70.67 [38.67, 114.17] for 14B. A mechanical audit of all 24 traces finds post-recovery execution/API requests in 0/6 preserve episodes and 6/6 restore episodes per model. These full-episode controls start from the initial task state.

Historical v33 model-driven recovery retains the recovery header after acceptance. Full/A5 changes recovery-action supply and continuation context together. Across 50 pairs/36 families, A5 raises AC by 13.15 pp [6.62, 20.43], adds 187.92 mean API attempts [140.93, 247.37], and changes OS from 0/50 to 1/50, exact McNemar p=1.0. Removing repeat-effect protection in A2 lowers Clean AC by 18.00 pp [−27.41, −11.11] and loses four successes. Observable AC changes by −1.17 pp [−3.80, 0.67], with all OS losses confined to Clean. These results establish diagnostic sensitivity across recorded support configurations (Table 5, Appendix E.4).

Terminal stages and Clean background

The 450 historical evolving episodes comprise 6 with no event exposure, 108 with a fired event unresolved, 218 with fired events resolved but the planned chain incomplete, and 118 with the full chain resolved and task failure. TLS success is zero. These are terminal stages. Population shares attributable to recovery-induced damage and baseline difficulty remain unassigned.

All twelve added Clean repeats 1/2 pass, establishing three-Clean-success membership across each historical model's 50-task scope. Mistral-Small-24B-Instruct-2501 has four selected tasks/three families, Qwen3-32B two tasks/two families, and Gemma-3-27B-IT an empty subset with undefined restricted rates. Hidden, Observable, and Oracle-Docs each fail all selected tasks at matched repeat 0, giving 0/4 for Mistral and 0/2 for Qwen. For each changed condition separately, the full-set changed-minus-Clean OS gap is −8.00 pp [−18.75, 0.00] for Mistral and −4.00 pp [−10.64, 0.00] for Qwen. Both selected gaps are −100.00 pp [−100.00, −100.00].

Intervals use 10,000 paired resamples of the original 36 families per model, fixing observed membership and using defined selected denominators. Selection fixes selected Clean OS at 100%, and identical losses give zero-width selected-gap intervals. The twelve added runs use frozen historical model/worker revisions, BF16 weights, and greedy temperature 0. All output sequences match their corresponding repeat 0, establishing execution consistency under this configuration. Changed-condition outcomes include the historical v33 continuation policy above.

We will add a paired end-to-end intervention on three fixed tasks for both NF4 models, comparing model-generated and deterministic public-protocol recovery from common saved boundaries and the same Observable public information. It will standardize continuation header, context formatter, and budget, retaining each policy's actual public recovery outcome. Policy assignment acts only at the target event, with common model-driven recovery and continuation formatting at subsequent events. We will report target recovery, official OS, AC, execution/API and model-call costs, paired outcomes, and source eligibility through the official task endpoint, including failed recovery and successor-event records.

AuthorsOpenReview ↗

Response to Reviewer 7UQK (3/3) - Avoidance scoring and protocol scope

Q4. Successful avoidance and strict lifecycle metrics

Authorized avoidance earns OS credit. Strict ECR measures whether the entire planned chain was encountered and resolved, and TLS measures the conjunction with task success. Adding verified successful-avoidance credit preserves the descriptive recovery–completion separation under the additive rule below (§3.3, Appendix E.2).

Under ordinary two-decimal rounding, Table 3's rates and 600 episodes per model–condition cell uniquely determine 1,860 OS-successful evolving episodes. Section 4.2 reports 924 TLS successes among 7,008 chain-resolved episodes. Thus 936 episodes succeed with ECR=0, including successful partial-chain routes. Zero-event successes are a subset, with share between 0 and 936/1,860 (50.3%) of OS successes. The exact zero-event split requires joint firing records.

An inclusive score retaining strict ECR credit and adding verified successful avoidance can credit at most all 936 successful ECR-failing episodes. This leaves 6,084 task failures among at most 7,944 credited episodes, giving a failure-fraction lower bound of 6,084/7,944 (76.6%). Any legitimate-avoidance subset has a bound at least this large.

For condition-specific added credit a_c, expressed as an episode rate, 0 ≤ a_c ≤ OS_c−TLS_c. The inclusive ECR contrast equals the original contrast plus a_Oracle−a_Hidden. Hidden OS=368/9,600 bounds its added credit by approximately 3.83 pp. Allowing for one-decimal rounding of the published 23.9 pp, the inclusive Oracle−Hidden ECR contrast remains at least 20.0 pp, alongside approximately 5.0 pp for OS. These are deterministic bounds from published aggregates under this additive rule. Exact avoidance rates and credited routes require the joint firing and route-legitimacy audit.

Q5. Necessary recovery, scaffolding, and transfer

Recovery measures conformance to executable contract obligations. Permission scopes and valid endpoint mappings constrain task operations. Explicit acknowledgment, plan-invalidation actions, and prescribed order are protocol choices. Neutral and public-action controls measure vocabulary and action-supply support within this interface.

A5's 83 versus Full's 48 fired events reflect deeper progression within the same 48 exposed episodes, with two unexposed in both. Full resolves 48/48 fired events and A5 83/83. Complete event sets agree in 27/50 pairs and differ in 23, as successor events become eligible after predecessors resolve. A5 also records 240 retired-tool attempts blocked before effects and one unblocked duplicate-mutating-call record with a ledger entry.

Oracle-Docs supplies authoritative change information. Task solutions and evaluator assertions remain private, and the agent chooses and executes downstream operations. Its gap from Hidden measures workflow performance under different information packages. Estimating a controller effect requires a policy comparison at common state and information boundaries.

Section 4.1 uses model-independent structural criteria and at least 80 reference calls. Reference length describes one validated route and defines the evaluated long-horizon population. The frozen Core-50 manifest documents outcome-blind metadata selection for that subset.

Main-matrix repeat keys align benchmark inputs. Hosted-provider randomness and backend drift remain uncontrolled. Historical mechanism ablations cover one model/key. Fresh validation adds pinned Qwen3-8B/14B comparisons under recorded precision and worker identities. Each NF4 model's six-task three-Clean-success subset is empty because every task fails Clean repeat 0, making restricted rates undefined. The nonempty historical BF16 subsets are reported in Q3.

The empirical conclusions concern AppWorld. GUI or terminal tests would require substrate-specific retained effects, interruption boundaries, recovery obligations, and independent terminal correctness. Cross-substrate performance remains an empirical question.

We will complete the registered twelve-task NF4 extension, including extra Clean/Observable repeats, and incorporate the completed analyses, controls, cases, and scoring derivation into the revised manuscript with their actual task/repeat coverage and model/worker identities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.