- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies that what kind of or type of information should be preserved when long running LLM agents doing the context compression operation (or other operations such as transfer message to another agent, resum, ...). Its main idea is that only preserving the content is not enough, the agent also needs to preserve the conditions under which that information can be used correctly. The paper proposes a new concept or method called "residency contracts", which is an offline replay procedure to test the effect of losing these conditions, and they also propose a runtime controller evaluated on a coding benchmark.
Critical strengths
-
The problem and main insight are interesting. The paper makes a useful distinction between preserving information itself and preserving the conditions under which that information should be used, such as its scope, source, freshness, and supporting evidence.
-
The boundary replay idea is also useful. It directly tests whether changing one of these conditions affects downstream agent behavior. The evaluation also covers multiple boundary types and 12 executor models.
Critical weaknesses
-
In the Introduction section, at the beginning, it is easy to read and follow, however, the presentation becomes hard to follow after "residency contracts" are introduced. The motivation before that point is clear, but the paper then quickly introduces placements, 6 obligation types, boundary policies, multiple validity levels, replay, and substantial notation. Authors should revise the presentation of the introduction part, and consider whether these concepts or notation are nessasary to the main arguement or content.
-
The choice of the six obligations (exactness, scope, source role, retrievability, freshness, and dependency) is not sufficiently motivated. They are individually reasonable, but it is unclear why are these 6 types, how they are derived.
-
The main empirical comparison does not cleanly isolate the contribution of the proposed controller. The controller and native harness use different token budgets and different information access, and the controller also combines construction metadata, deterministic rules, and a learned scorer. The current experiments therefore do not prove which part is responsible for the gain.
If the authors can address them, I will consider raising my rating to 3.
Questions for authors
-
Why these six obligations? What is the intuition or evidence?
-
How well does the method generalize without construction derived metadata and to unseen task families?
-
Address other problems I mentioned in the weaknesses.
6 comments
Part 2: Separating resources, metadata, and scoring (W3, Q3)
Separating resources, metadata, and scoring (W3, Q3)
To separate resources, metadata, and controller behavior, we give the native harness the same typed metadata and limits as the controller, including 12,288 active tokens and 16,384 cumulative input tokens. The task templates, seeds, validators, tools, output allowance, and time allowance are shared.
On Grok 4.20 non-reasoning, across 24 templates and four seeds per template, strict success is 31.2% for the native harness with metadata and 55.2% for the controller. The mean preservation score is 0.59 versus 0.83. On Qwen 3.6 27B, the corresponding success rates are 26.0% and 46.9%, and preservation is 0.55 versus 0.77. These comparisons test the combined controller at matched budgets and information access.
To isolate ranking, we keep the controller's rules, planner, renderer, and budgets fixed and replace its learned scorer with a fixed metadata ranking. The learned scorer improves preservation from 0.78 to 0.83, a difference of 0.05 with a 95% interval of 0.019 to 0.081. Success rises from 49.0% to 55.2%, but the difference of 6.2 points has an interval of -3.1 to 16.7. This isolates a preservation benefit from learned ranking, while its additional success benefit remains uncertain.
Part 3: Q2 and W1, Q3
Removing metadata and testing transfer (Q2)
When construction-derived metadata is masked in both the scorer and the rules, success is 38.5%, compared with 55.2% for the full controller and 27.1% for the native harness at the same caps. The scorer is retrained with the same feature mask used at inference. The masked-versus-native difference is descriptive because a paired interval for that contrast is unavailable. The full-versus-masked comparison shows that construction metadata contributes substantially to the controller's performance.
We also evaluated 30 templates from τ-Bench (Yao et al., 2024) and 30 from Terminal-Bench 4.0 (Merrill et al., 2026), held out from training. Under matched metadata and caps, the controller's success gains on the combined set are 9.2 points over fixed rules and 6.7 over manual contracts, with pointwise 95% template-cluster intervals of 2.5 to 16.7 and 1.7 to 12.5. These sources complement the original 24 controlled templates, giving 84 distinct templates. The source diversity and positive paired intervals support these comparisons, with repeated seeds retained within template clusters.
In the separate family-holdout comparison, holding each family out of scorer training gives 50.0% success, compared with 52.1% when training includes that family. The difference is -2.1 points, with a 95% interval of -8.3 to 4.2. The rules and typed metadata remain fixed, so this test measures scorer transfer within the existing taxonomy.
Simplifying the introduction (W1, Q3)
We simplified the introduction around conditions of use, residency contracts, and boundary replay. Section 2 gives the detailed definitions. The revised introduction states the central idea as follows.
A residency contract states what information should cross a context boundary and what conditions must hold for the next receiver to use it correctly. In a debugging handoff, it preserves the allowed edit path and keeps tool output identifiable as data. The contract specifies these requirements separately from the packet that the harness produces.
Summary
The six obligations cover 142 of 156 examined failure cases, and interventions against each reduce success at applicable checkpoints. Matched comparisons support the controller on the original and combined external sets. The revised introduction presents the three central ideas directly, and Appendix I reports the additional evidence.
Part 1: Why these six obligations (W2, Q1)
Thank you for your comments. We address each concern below.
Why these six obligations (W2, Q1)
The six obligations describe different requirements for using information after a context boundary. They ask whether the receiver has the required wording, knows the permitted use and source authority, can reach the evidence, and has information that is current and properly supported. We treat them as a working set that can be extended when a failure requires another condition.
The additional coverage study codes 156 reported failure cases from Compaction Cliff, Governance Decay, AuthMem, MAST, and the Claude Code and OpenHands issue trackers. Two annotators independently assigned potentially overlapping obligation labels. At least one obligation covers 142 cases, or 91.0%, while 14 cases fall outside the set. The reported mean agreement across the obligation labels is Cohen's kappa 0.82. The coverage study supports these six requirements as a useful working set for the examined boundary failures.
We also tested the behavioral consequence of breaking each obligation in paired replay. Each intervention changes one required condition while holding the task, executor, validator, and continuation seed fixed. The table gives the success drop relative to the original packet, in percentage points, with pointwise 95% template-cluster bootstrap intervals. The interventions apply only at checkpoints where the respective obligation is required.
Retrievability concerns access to an evidence item. Dependency concerns whether a retained claim keeps the support it requires. Their distinct definitions allow the replay interventions to test different failures. The safety outcomes also differ. Removing scope increases violations by 14.1 points, with a 95% interval of 9.4 to 18.8. Changing source role increases harmful adoption by 16.7 points, with an interval of 11.5 to 21.9.
Removing each obligation from the controller lowers its corresponding preservation endpoint by 0.070 to 0.125, with all six pointwise 95% intervals below zero. These ablations disable the relevant rules, remove the targets, retrain the scorer, and mask the tags. Their strict-success differences range from -7.3 to -3.1 points, with all six intervals including zero. The ablations therefore support each obligation's contribution to its targeted preservation measure, while its separate effect on strict success remains uncertain. Together, the coverage study and interventions motivate a set of separately testable requirements.