acceptodds
Under review as a conference paper at ICLR 2027

Where Context Goes and What It Owes: Residency Contracts for Long-Running LLM Agents

Abstract

Long-running LLM agents compact histories, delegate subtasks, prune tool outputs, and resume sessions. These transformations can preserve task information while losing its conditions of use, such as exact wording, scope, source role, retrievability, freshness, and supporting evidence. Context management must therefore determine what survives each boundary, in what form, for a particular receiver. We introduce **residency contracts**, which make these requirements explicit for each typed artifact. **Boundary replay** restores a pre-boundary checkpoint, perturbs one condition of use, and compares paired continuations to measure its signed behavioral effect. Replay supplies offline evidence and supervision, while a runtime controller combines policy rules with a learned scorer to produce contracts without receiver-time replay. On BoundaryBench, our benchmark of constructed coding tasks, twelve executor models share 24 task templates. The controller improves preservation by 0.20 to 0.22 and strict success by 20.8 to 25.0 percentage points over their native-harness baselines. For Grok 4.20 non-reasoning, preservation rises from 0.61 to 0.83 and success from 0.30 to 0.55 while input token use falls by 55%. Relevance-only scoring reduces preservation by 0.09. These paired comparisons evaluate the combined scorer-and-rule policy with construction metadata under different token caps. The results support explicit conditions of use as an object for context management on the evaluated families.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, context management, agent memory, residency contracts, boundary replay, long-horizon tasks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CtGFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies that what kind of or type of information should be preserved when long running LLM agents doing the context compression operation (or other operations such as transfer message to another agent, resum, ...). Its main idea is that only preserving the content is not enough, the agent also needs to preserve the conditions under which that information can be used correctly. The paper proposes a new concept or method called "residency contracts", which is an offline replay procedure to test the effect of losing these conditions, and they also propose a runtime controller evaluated on a coding benchmark.

Critical strengths
  • The problem and main insight are interesting. The paper makes a useful distinction between preserving information itself and preserving the conditions under which that information should be used, such as its scope, source, freshness, and supporting evidence.

  • The boundary replay idea is also useful. It directly tests whether changing one of these conditions affects downstream agent behavior. The evaluation also covers multiple boundary types and 12 executor models.

Critical weaknesses
  • In the Introduction section, at the beginning, it is easy to read and follow, however, the presentation becomes hard to follow after "residency contracts" are introduced. The motivation before that point is clear, but the paper then quickly introduces placements, 6 obligation types, boundary policies, multiple validity levels, replay, and substantial notation. Authors should revise the presentation of the introduction part, and consider whether these concepts or notation are nessasary to the main arguement or content.

  • The choice of the six obligations (exactness, scope, source role, retrievability, freshness, and dependency) is not sufficiently motivated. They are individually reasonable, but it is unclear why are these 6 types, how they are derived.

  • The main empirical comparison does not cleanly isolate the contribution of the proposed controller. The controller and native harness use different token budgets and different information access, and the controller also combines construction metadata, deterministic rules, and a learned scorer. The current experiments therefore do not prove which part is responsible for the gain.

If the authors can address them, I will consider raising my rating to 3.

Questions for authors
  • Why these six obligations? What is the intuition or evidence?

  • How well does the method generalize without construction derived metadata and to unseen task families?

  • Address other problems I mentioned in the weaknesses.

6 comments
AuthorsOpenReview ↗

Part 1: Why these six obligations (W2, Q1)

Thank you for your comments. We address each concern below.

Why these six obligations (W2, Q1)

The six obligations describe different requirements for using information after a context boundary. They ask whether the receiver has the required wording, knows the permitted use and source authority, can reach the evidence, and has information that is current and properly supported. We treat them as a working set that can be extended when a failure requires another condition.

The additional coverage study codes 156 reported failure cases from Compaction Cliff, Governance Decay, AuthMem, MAST, and the Claude Code and OpenHands issue trackers. Two annotators independently assigned potentially overlapping obligation labels. At least one obligation covers 142 cases, or 91.0%, while 14 cases fall outside the set. The reported mean agreement across the obligation labels is Cohen's kappa 0.82. The coverage study supports these six requirements as a useful working set for the examined boundary failures.

We also tested the behavioral consequence of breaking each obligation in paired replay. Each intervention changes one required condition while holding the task, executor, validator, and continuation seed fixed. The table gives the success drop relative to the original packet, in percentage points, with pointwise 95% template-cluster bootstrap intervals. The interventions apply only at checkpoints where the respective obligation is required.

ObligationWhat the receiver needsSuccess drop95% interval
ExactnessRequired wording or identifiers18.813.4 to 24.1
ScopeAllowed files, actions, and audiences12.58.6 to 16.4
Source roleThe distinction between instruction and data10.45.2 to 15.6
RetrievabilityEvidence accessible within its budget14.66.2 to 24.0
FreshnessInformation valid for the current version8.32.1 to 16.7
DependencyRequired support for retained claims11.54.2 to 19.8

Retrievability concerns access to an evidence item. Dependency concerns whether a retained claim keeps the support it requires. Their distinct definitions allow the replay interventions to test different failures. The safety outcomes also differ. Removing scope increases violations by 14.1 points, with a 95% interval of 9.4 to 18.8. Changing source role increases harmful adoption by 16.7 points, with an interval of 11.5 to 21.9.

Removing each obligation from the controller lowers its corresponding preservation endpoint by 0.070 to 0.125, with all six pointwise 95% intervals below zero. These ablations disable the relevant rules, remove the targets, retrain the scorer, and mask the tags. Their strict-success differences range from -7.3 to -3.1 points, with all six intervals including zero. The ablations therefore support each obligation's contribution to its targeted preservation measure, while its separate effect on strict success remains uncertain. Together, the coverage study and interventions motivate a set of separately testable requirements.

AuthorsOpenReview ↗

Part 2: Separating resources, metadata, and scoring (W3, Q3)

Separating resources, metadata, and scoring (W3, Q3)

To separate resources, metadata, and controller behavior, we give the native harness the same typed metadata and limits as the controller, including 12,288 active tokens and 16,384 cumulative input tokens. The task templates, seeds, validators, tools, output allowance, and time allowance are shared.

On Grok 4.20 non-reasoning, across 24 templates and four seeds per template, strict success is 31.2% for the native harness with metadata and 55.2% for the controller. The mean preservation score is 0.59 versus 0.83. On Qwen 3.6 27B, the corresponding success rates are 26.0% and 46.9%, and preservation is 0.55 versus 0.77. These comparisons test the combined controller at matched budgets and information access.

To isolate ranking, we keep the controller's rules, planner, renderer, and budgets fixed and replace its learned scorer with a fixed metadata ranking. The learned scorer improves preservation from 0.78 to 0.83, a difference of 0.05 with a 95% interval of 0.019 to 0.081. Success rises from 49.0% to 55.2%, but the difference of 6.2 points has an interval of -3.1 to 16.7. This isolates a preservation benefit from learned ranking, while its additional success benefit remains uncertain.

AuthorsOpenReview ↗

Part 3: Q2 and W1, Q3

Removing metadata and testing transfer (Q2)

When construction-derived metadata is masked in both the scorer and the rules, success is 38.5%, compared with 55.2% for the full controller and 27.1% for the native harness at the same caps. The scorer is retrained with the same feature mask used at inference. The masked-versus-native difference is descriptive because a paired interval for that contrast is unavailable. The full-versus-masked comparison shows that construction metadata contributes substantially to the controller's performance.

We also evaluated 30 templates from τ-Bench (Yao et al., 2024) and 30 from Terminal-Bench 4.0 (Merrill et al., 2026), held out from training. Under matched metadata and caps, the controller's success gains on the combined set are 9.2 points over fixed rules and 6.7 over manual contracts, with pointwise 95% template-cluster intervals of 2.5 to 16.7 and 1.7 to 12.5. These sources complement the original 24 controlled templates, giving 84 distinct templates. The source diversity and positive paired intervals support these comparisons, with repeated seeds retained within template clusters.

In the separate family-holdout comparison, holding each family out of scorer training gives 50.0% success, compared with 52.1% when training includes that family. The difference is -2.1 points, with a 95% interval of -8.3 to 4.2. The rules and typed metadata remain fixed, so this test measures scorer transfer within the existing taxonomy.

Simplifying the introduction (W1, Q3)

We simplified the introduction around conditions of use, residency contracts, and boundary replay. Section 2 gives the detailed definitions. The revised introduction states the central idea as follows.

A residency contract states what information should cross a context boundary and what conditions must hold for the next receiver to use it correctly. In a debugging handoff, it preserves the allowed edit path and keeps tool output identifiable as data. The contract specifies these requirements separately from the packet that the harness produces.

Summary

The six obligations cover 142 of 156 examined failure cases, and interventions against each reduce success at applicable checkpoints. Matched comparisons support the controller on the original and combined external sets. The revised introduction presents the three central ideas directly, and Appendix I reports the additional evidence.

Thank you again. We hope these results can address your concerns, and we are happy to clarify anything further.

Reviewer_LNdZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes residency contracts for managing information across LLM-agent context boundaries such as compaction, delegation, tool-output pruning, and resumption. It also introduces BoundaryBench, a purpose-built benchmark with 12 constructed task families, and evaluates a rule- and learned-score-based controller on 12 models. The controller improves preservation and task success over several baselines on the proposed benchmark.

Critical strengths
  1. The paper identifies an important failure mode in long-running agents: preserving the content of information without preserving its conditions of use.

  2. The paired tasks and preservation metrics are more informative than reporting task success alone.

Critical weaknesses

1. Insufficient empirical workload and limited external validity. The main evaluation uses only 24 constructed task templates from a single newly introduced benchmark. Although it is evaluated across 12 models, the task diversity remains small and highly aligned with the proposed taxonomy. The only external benchmark experiment, SWE-Bench Pro, is a small single-arm pilot and provides no evidence that the controller improves performance on realistic software-engineering tasks.

2. The contribution is not adequately isolated. The controller benefits from construction-derived metadata and is evaluated under different token/resource caps from the native baseline. Consequently, the experiments do not establish how much improvement comes from residency contracts, the learned scorer, explicit metadata, hard-coded rules, or additional resources. The extensive set of baselines does not resolve this central attribution problem.

Questions for authors
  1. Why should the results be considered evidence of general improvement in long-running agents rather than performance on a small benchmark whose task families and validators were designed by the authors?

  2. Can the authors provide a substantially larger and externally sourced evaluation with matched information access and matched resource budgets?

4 comments
AuthorsOpenReview ↗

Part 1: W1 Q1

Thank you for your comments. We address the evaluation scope and the attribution of the gains below.

W1, Q1 and Q2. Evaluation on two external task sources

We expanded the evaluation with 60 externally sourced templates, comprising 30 from τ-Bench (Yao et al., 2024) and 30 from Terminal-Bench 4.0 (Merrill et al., 2026). Together with the original 24 BoundaryBench templates, this gives 84 distinct templates across the controlled benchmark and two external task sources. We also report a separate paired evaluation on 100 SWE-bench Verified instances in Reply 2.

The added templates are disjoint from the original 24 and scorer training. Each was frozen before evaluation. Two seeds per template give 120 trajectories per arm.

On this combined set, the controller, fixed rule and manual contract share typed metadata and caps of 12,288 active and 16,384 cumulative input tokens. The executor is Grok 4.20 non-reasoning. Pointwise confidence intervals use 100,000 template-bootstrap draws, retaining both seeds and paired arms.

ArmSuccess (%)Controller gain (percentage points)95% interval
Fixed rule42.59.22.5 to 16.7
Manual residency contract45.06.71.7 to 12.5
Controller51.7Reference

W1 and Q1. Why this task set supports the comparison

Sample adequacy rests on complementary coverage and the precision of controlled comparisons. The original 24 templates cover 12 designed failure families across compaction, delegation, pruning, and resumption. The 60 external templates test beyond that constructed set under matched metadata and caps.

We analyze each study separately, resampling templates with all paired arms and their four or two seeds, respectively. Seeds and executor models do not increase the distinct-template count.

On the external set, the pointwise 95% intervals have positive lower bounds of 2.5 points over fixed rules and 1.7 over manual contracts. At this sample size, the paired estimates therefore resolve positive average gains in both matched comparisons. Together with the controlled failure tests, this supports improved boundary handling on the evaluated tasks. These estimates concern the combined external set. Separate gains for each source benchmark remain unestimated.

AuthorsOpenReview ↗

Part 2: W2 Q2

W2 and Q2. Matched information, budgets and scorer attribution

The new native-harness comparisons cross token caps with access to the controller's metadata. The metadata arm adds plain-text tags for type, source, authority, freshness, exact fields, scope, evidence pointers and dependencies while retaining the native boundary procedure.

The following rows share 12,288 active and 16,384 cumulative input-token caps, an 8,192 output-token cap and a 1,800-second limit. Each row uses the same 24 templates and four seeds. The preservation score G averages the applicable preservation measures.

ExecutorArmGSuccess (%)
Grok 4.20 non-reasoningNative harness0.5627.1
Grok 4.20 non-reasoningNative harness with metadata0.5931.2
Grok 4.20 non-reasoningController0.8355.2
Qwen 3.6 27BNative harness0.5224.0
Qwen 3.6 27BNative harness with metadata0.5526.0
Qwen 3.6 27BController0.7746.9

The controller has higher success than the native harness with metadata on both executors under the same caps. This comparison supports a benefit from the combined controller beyond supplying typed annotations alone.

To isolate the scorer, a separate comparison holds the rules, planner, renderer and caps fixed and replaces learning with a fixed metadata ranking. The learned scorer improves G by 0.05, with a 95% template-bootstrap interval of 0.019 to 0.081. Its success gain is 6.2 percentage points, with an interval of -3.1 to 16.7. The evidence supports a preservation benefit from learning. Its additional effect on task success remains unresolved at this precision.

W1, Q1 and Q2. Additional evaluation on SWE-bench Verified

The new external evaluation uses 100 SWE-bench Verified instances with mini-SWE-agent and Qwen 3.6 27B. Both arms run the same instances with a 64-step limit under each of two boundary protocols. For compaction, both use a 16,384-token active-context cap. For delegation, a fresh subagent receives a debugging subtask after step 10. The controller uses a frozen rule-based typer built from issue text, tool output, paths and edit history.

BoundaryNative resolve rate (%)Controller resolve rate (%)Gain (percentage points)95% paired interval
Compaction58.065.07.0-2.0 to 16.0
Delegation61.066.05.0-3.0 to 13.0

Both resolve-rate estimates favor the controller, with intervals that include zero, so the success benefit on SWE-bench Verified remains uncertain. This comparison tests the controller with automatic typing on software tasks. The full control of budgets and metadata for attribution comes from the matched comparisons above.

The number of patches changing a non-test file outside the gold patch falls from 12 to 7 under compaction and from 9 to 6 under delegation. We interpret these counts as a patch-scope proxy, since valid alternative fixes may touch other files.

The additional third-party tasks report policy violations of 32% versus 2% on Governance Decay and authority collapse in 47 versus 2 of 49 AuthMem configurations, ordered native then controller. We report these as descriptive outcomes under the respective benchmark settings, without paired significance claims.

Summary

The 84 templates combine controlled boundary failures with two external task sources. Matched comparisons and template-level uncertainty support the reported success gains. Appendix I also reports the separate SWE-bench results with their paired uncertainty.

Thank you again. We would be glad to provide further analysis.

Reviewer_LNdZOpenReview ↗

Official Comment by Reviewer LNdZ

Thanks for the additions, but they do not resolve my core concern. On SWE-bench Verified, gains are statistically insignificant under both protocols (7 [-2, 16]; 5 [-3, 13]). The 60 external templates are reported only pooled under your own boundary protocol, leaving individual benchmark gains unestimated. Moreover, removing metadata cuts the controller's edge over the metadata-matched baseline by over half (Table 23), yet the paper treats typed metadata extraction as "outside our scope" and reports no accuracy for the rule-based typer on SWE-bench. Without verified metadata extraction or clear gains on realistic benchmarks, the empirical support remains insufficient. I keep my rating of 2 (Weak Reject).

AuthorsOpenReview ↗

Reply to rebuttal feedback

Thanks for following up.

Table 23 tests the controller’s dependence on metadata. The matched comparisons address whether the controller adds value when that information is held fixed. On the combined 60-template set, success improves over equally informed fixed rules by 9.2 points [2.5, 16.7] and manual contracts by 6.7 points [1.7, 12.5], under identical metadata and caps. These pointwise 95% intervals support a benefit from how the controller uses the shared information.

For the new templates, metadata is extracted automatically through a common procedure, without manual annotation or task-specific labelling. The scope statement concerns learning a typer from raw histories. The reported experiments measure downstream performance with the extractor, rather than field-level extraction accuracy.

The common boundary protocol tests the same intervention across different task sources. Its pooled estimates concern the combined external set. SWE-bench provides a separate integration comparison whose success intervals include zero. Our empirical claim rests on the matched comparisons across controlled and externally sourced tasks, which support improved boundary handling beyond supplying metadata alone.

Reviewer_bQobOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about what gets lost when a long-running LLM agent rewrites its own context - compaction, briefs for subagents, pruning tool output, resuming a session. The observation is that the agent can keep the content of some piece of information while losing the conditions under which it may be used (exact wording, scope, where it came from, whether it is stale, and so on). They propose residency contracts, which say for each typed artifact how it should cross the boundary and which of these conditions must hold on what the receiver actually sees, and boundary replay, which restores a checkpoint, breaks one condition, and compares paired continuations. A controller made of precedence rules, a small learned scorer, a greedy planner and a validating renderer produces these contracts at runtime. On a constructed coding benchmark the controller improves a preservation score and strict success by a large margin over the native harnesses of twelve executor models.

Critical strengths
  1. The framing is useful and ties together several recent reports of compaction failures that otherwise look unrelated. The paper also keeps the specification, the realized packet and the agent's behavior separate, and reports violations and harmful adoption separately from success, which I think is the right way to evaluate this kind of system.
  2. The analysis of replay from logs (Sec. 4.3) seems correct to me and is practically useful: logged replay can report zero loss when the true loss is substantial. This is the part of the paper I would cite.
Critical weaknesses
  1. The headline comparison between the controller and the native harness differs in two things at once - the token caps, and access to construction metadata the native harness never gets - and the paper itself shows that this metadata alone predicts the intervention targets well above the majority rate. The equally informed arms (a fixed rule and a hand-written contract with the same metadata and caps) do exist in Table 2, and the controller is ahead of them by roughly ten points, but there are no paired intervals or tests and there are only 24 underlying task templates, so I cannot tell if the gap is real. The paper is honest about this, but honesty about a confound does not resolve it, and this is the comparison the whole paper rests on.
  2. The learned component is not established. The training recipe for the scorer used in the main comparison is not actually disclosed (App. C documents a different one, a very small perceptron trained on a few hundred rows from nine trajectories), and Sec. 7 admits the value of learning is unresolved. The cross-model results are also more regular than I would expect from twelve independent executors, with the controller-to-full-context gap identical for all of them, and Table 14 and Table 2 disagree on controller violations. These may be benign, but together they make me unsure what the numbers are measuring.
Questions for authors
  1. Can you report paired intervals and a McNemar or cluster-bootstrap test for the controller against the fixed-rule and hand-written-contract arms in Table 2?
  2. How exactly was the scorer in the main tables trained, and are the intervention targets in Table 3 derived from replay?
  3. Why is the gap between the controller and full context identical across executors, and how do Table 14 and Table 2 reconcile on violations? Suggestions rather than questions: a matched comparison against the typed-retention methods you already cite (Zerhoudi et al.'s Knowledge Triage, ClawVM's typed pages) would strengthen the paper considerably. Prior work on behavioral contracts for agents should be referenced. The cross-entropy quoted in G.4 does not match Table 16.
6 comments
AuthorsOpenReview ↗

Part 1: W1 and Q1. Matched comparisons and paired uncertainty

Thank you for your comments. We address the comparisons, scorer training, and reporting questions below.

W1 and Q1. Matched comparisons and paired uncertainty

To separate controller behavior from token budgets and metadata access, we added a comparison that matches both. The native harness receives the controller's typed metadata as plain-text annotations, and both arms have caps of 12,288 active and 16,384 cumulative input tokens. The executor, tasks, seeds, validators, output cap, and time limit are also fixed. On Grok 4.20 non-reasoning, the results are as follows. G is the mean preservation score over applicable conditions of use.

MethodGSuccess (%)Trajectories with a violation
Native harness0.5627.113/96
Native harness with metadata0.5931.211/96
Controller0.8355.24/96

Against native with metadata, the controller gains 24.0 percentage points in success, with a 95% interval of 15.6 to 31.2, and 0.24 in G, with an interval of 0.209 to 0.271. Thus, supplying the same metadata through annotations does not account for the controller's full gain in this setting. This comparison evaluates the combined rules, scorer, planner, and renderer.

The next table reports the additional paired evaluation of equally informed baselines on the original 24-template task set. The analysis treats templates as clusters. Each of 100,000 bootstrap resamples keeps all four seeds and both arms together. Success uses a clustered McNemar test, and G uses a sign-flip test on template means. Holm correction is applied across the seven comparisons for each endpoint. The intervals below are pointwise 95% intervals.

ComparatorSuccess gain (pp)95% intervalHolm pG gain95% interval
Fixed rule11.55.2 to 17.70.01800.120.089 to 0.151
Manual residency contract9.41.0 to 17.70.11680.090.065 to 0.115
ClawVM6.2-1.0 to 13.50.21760.060.035 to 0.085
Knowledge Triage4.2-3.1 to 11.50.28500.040.009 to 0.077

The fixed-rule success comparison survives correction, as do all four G comparisons under the separate correction for preservation. The manual-contract success comparison has a positive pointwise interval but does not pass the Holm correction. The paired analysis therefore supports a success gain over fixed rules and preservation gains over all four equally informed baselines.

To test beyond the original 24 templates, we also evaluated 30 templates from τ-Bench (Yao et al., 2024) and 30 from Terminal-Bench 4.0 (Merrill et al., 2026). Under matched metadata and token caps, the controller improves success over fixed rules by 9.2 points, with a pointwise 95% interval of 2.5 to 16.7, and over manual contracts by 6.7 points, with an interval of 1.7 to 12.5. These intervals resample the 60 templates while keeping both seeds and paired arms together. This separately analyzed external set adds task diversity and resolves positive average success differences in the matched comparisons. Sample adequacy for these claims rests on this coverage and precision, with 84 distinct templates across the two evaluations.

AuthorsOpenReview ↗

Part 2: W2 and Q2. Scorer training and intervention targets

W2 and Q2. Scorer training and intervention targets

The recipe for the additional experiments uses a 32-unit tanh hidden layer and a 20-way operation softmax. Its input contains 64 hashed text features, 28 structured metadata features, 16-dimensional boundary and role embeddings, and five graph features. Requirement-label slots are absent at both training and inference. The training set contains 352 artifact-at-boundary rows from nine trajectories and 44 artifacts. Validation contains 120 rows from three trajectories and 15 artifacts. The split separates templates and artifacts, and the development templates are separate from the evaluation templates.

Training uses per-row stochastic gradient descent with exact gradients, Glorot initialization, a step size of 0.02, and 40 passes with reshuffled rows. In addition to operation cross-entropy, the auxiliary weights are 0.20 for effect regression, 0.10 each for exactness, scope, provenance, evidence, and version-sensitive cache placement, and 0.05 for cache layout.

The combined target uses replay wherever a replay pair exists and construction requirements elsewhere. Each replayed training row uses two continuation seeds. If breaking a condition lowers utility, its target retains that condition. Otherwise, the target is the cheapest admissible operation. This gives an explicit comparison of target sources under the same remaining recipe.

Training targetsGSuccess (%)
Construction only0.7044.8
Replay only0.7750.0
Construction and replay0.8355.2

The combined target improves success over construction only by 10.4 points, with a 95% interval of 4.2 to 15.6. The replay-only difference is 5.2 points, with an interval of -2.1 to 12.5, leaving that success contrast uncertain.

This recipe documents the additional comparisons, while Appendix C describes the earlier construction-supervised development fit. It does not identify the original main-table checkpoint or the target source of the original Table 3 ablation. The revised Appendix C distinguishes the recipes, and Appendix I reports the explicit construction-only, replay-only, and combined-target comparison in place of that ablation.

AuthorsOpenReview ↗

Part 3: W2 and Sg1. What learning adds and how typed retention compares

W2 and Sg1. What learning adds and how typed retention compares

To isolate ranking, the additional experiment keeps the rules, planner, renderer, and caps fixed. Replacing the learned scorer with a fixed metadata ranking changes G from 0.83 to 0.78 and success from 55.2% to 49.0%. In the learned-minus-fixed direction, the intervals are 0.019 to 0.081 for G and -3.1 to 16.7 points for success. This isolates a preservation benefit from learned ranking, while its additional success benefit remains uncertain.

ClawVM and Knowledge Triage receive the same typed metadata and token caps. Their success rates are 49.0% and 51.0%, compared with 55.2% for the controller. Their G scores are 0.77 and 0.79, compared with 0.83. The paired results in Reply 1 support a preservation advantage but do not establish a success advantage over either method. The revised paper makes this distinction and describes the baseline adaptations in Appendix I.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Summary on Rebuttal

Thank you to all reviewers for the discussion. The revised paper adds three main pieces of evidence.

  1. Matched information and budgets. On Grok 4.20, the controller improves success over the native harness with identical metadata and caps by 24.0 percentage points, with a 95% interval of [15.6, 31.2].

  2. Broader task coverage. We added 30 templates from τ-Bench and 30 from Terminal-Bench 4.0. On this combined set, success gains over equally informed fixed rules and manual contracts are 9.2 and 6.7 points, with pointwise 95% intervals of [2.5, 16.7] and [1.7, 12.5].

  3. Evidence for the mechanism. Holding rules, planning, and rendering fixed, learned ranking improves preservation by 0.05 [0.019, 0.081]. The six obligations cover 142 of 156 examined failure cases, and paired interventions show a success drop when each applicable obligation is broken.

These results support contract-guided boundary control beyond metadata access alone. The revised PDF and Appendix I include the experimental details, paired uncertainty, and separate SWE-bench results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.