acceptodds
Under review as a conference paper at ICLR 2027

When to Forget: Risk-Controlled Context Adaptation for Long-Horizon Agents

Abstract

Long-horizon agents learn from failure, but the value of a failure record can change as execution progresses. We study when retaining, compressing, or removing the same history block improves subsequent execution relative to full-history KEEP. We introduce RECAP, a framework that combines matched context interventions, execution-relational prediction, and independent policy certification. Matched environment states and rollout seeds measure success and continuation prompt-cost differences; prefix-only relations between a failure block and the current execution propose promising edits. Independent task bundles then select a policy subject to harmful-intervention and non-positive-benefit risk constraints under a fixed measurement protocol. A five-seed longitudinal case holds both the failure block and its summary fixed: compression fails at an early state and becomes a cost-saving intervention at a later state. Across five backbones, RECAP produces positive paired-success saving estimates with backbone-specific reliability–efficiency tradeoffs. On Hunyuan-A13B, it edits 43.24% of retained checkpoints while matching KEEP's observed 91.89% continuation success and reducing unconditional mean continuation prompt cost by 4,619.8 tokens (2.13%). The framework provides an execution-level reference for deciding when to forget: retain history by its value to what comes next.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, context management, agent memory, selective forgetting, statistical risk control, counterfactual evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9scZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the problem of deciding whether to keep, compress, or drop failed turns in a trajectory, to save cost and improve subsequent performances. It is motivated by the fact that failed turns can become obsolete once the model gains new context. The paper proposes a new framework, RECAP, that proves effective across models and metrics on processing failed turns.

Critical strengths
  • The paper studies a novel yet practical problem: saving context not by brute compacting / truncating, but by carefully processing failed turns.
  • The paper presents a strong framework: RECAP is competitive across 3 models on 3 metrics -- success, token cost, and flip from success to fail.
  • The paper provides sufficient code and appendix information for reproducibility.
Critical weaknesses
  • The paper's introduction and abstract are hard to read. Beyond the first two paragraphs, the experimental meaning of KEEP, COMPRESS, and DROP are not defined anywhere in the intro, and the figure 1 is hard to read to draw the said conclusions. In addition, the enumeration of contributions in the end is also vague.
  • Section 3 and 4 contain an excessive amount of math whose only role is to formalize, not to prove, anything. As a result, they are not central to the narrative, but further degrade readability.
Questions for authors
  • Can the authors clarify the contribution of the work in the intro?
  • Can the authors provide in natural language a succinct explanation of how RECAP works? -- As I understand it, this is not an optimization paper, so the formalism seems a little unjustifiable.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 9scZ

Thank you. We agree with both presentation concerns and have rewritten the corresponding parts.

Abstract, introduction, Figure 1.

  • The abstract is rewritten: it names the three treatments in its second sentence and states the results as numbers.
  • The treatments are defined in the second paragraph of the introduction. For a block of earlier failed tool calls: KEEP leaves it in the context; COMPRESS replaces it with a short fixed-format summary; DROP deletes it.
  • The Figure 1 caption now states the outcome in words (step 15: the full record succeeds in 5 of 5 seeds and the summary fails in 5 of 5; step 27: both succeed and the summary uses 8% fewer prompt tokens), so the conclusion no longer has to be read off the drawing.

Sections 3 and 4. The main text keeps two displayed formulas (the measured effect of an edit and the decision rule) and the statement of the guarantee; the rest moves to the appendix. The formalism serves one purpose: to state exactly what the guarantee covers. It is not an optimization result.

How RECAP works, in plain language.

  1. Measure. From a checkpoint of a recorded run, continue three ways (keep, summarize or delete a block of failed tool calls) from the same state and seeds. Record whether the task succeeds and how many prompt tokens the rest of the run uses.
  2. Predict. Fit two small predictors on what is visible at the checkpoint (for example, whether the failed constraint has since been resolved): one for whether an edit would hurt success, one for how many tokens it would save.
  3. Decide. Edit only if the predicted harm is below one threshold and the predicted saving is above another. Otherwise keep the history.
  4. Certify. On separate tasks, accept a threshold pair only if a binomial upper confidence bound on its harmful-edit frequency is below a preset budget.

Contributions, restated.

  1. A measurement protocol for "what happens if this failure record is compressed or removed now?", showing that the answer depends on the execution state.
  2. On five backbones, indiscriminately compressing or removing failure records lowers continuation success by 9.9 and 6.6 points and harms 37% and 28% of tasks.
  3. A risk-controlled selection rule that harms 2 of 166 held-out tasks, with its limits stated.

One correction to a listed strength. Your review lists RECAP's token cost as a strength. After re-analysis with task-level intervals, its saving is distinguishable from zero on one of five backbones, and a new online experiment finds no difference from KEEP. The revision presents RECAP as a way to avoid harmful forgetting, not as an efficiency method.

Reviewer_F9QfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RECAP, a risk-controlled execution-aware context adaptation policy for long-horizon language-model agents. At a selected checkpoint, the method compares three treatments of a failure block: KEEP, a deterministic COMPRESS operation, and DROP. Matched environment states and rollout seeds provide counterfactual estimates of continuation success and prompt-token cost. RECAP then uses prefix-only execution-relational features to predict intervention harm and normalized cost benefit, and selects a policy from a development-fixed threshold family using independent task-bundle tests with one-sided Clopper–Pearson bounds. Experiments on six AgentCE-Bench domains and five backbones report positive paired-success savings for RECAP, including a Hunyuan result that matches KEEP's observed continuation success while reducing unconditional continuation prompt cost by 2.13%.

The paper is carefully written and unusually explicit about its measurement protocol and assumptions. However, the main evidence is still an offline, single-intervention evaluation on a screened population of successful KEEP trajectories. Consequently, the results do not yet establish that RECAP improves end-to-end long-horizon agents or that its risk certificate provides meaningful protection under deployment.

Critical strengths

The paper gives a clear counterfactual formulation of context editing. Holding the environment state, block identity, backbone, and rollout seeds fixed makes the comparison between KEEP, COMPRESS, and DROP substantially more interpretable than a simple prompt-length comparison.

The separation between measurement, proposal, and certification is conceptually clean. In particular, the paper does not treat a learned harm score as a statistical guarantee, and it states the assumptions behind the fixed-sequence certification theorem.

The experimental and implementation details are unusually thorough. The appendices document the feature dictionary, block transformations, task splits, calibration ledgers, denominator conventions, and several limitations. The five-backbone comparison and the fixed-block longitudinal case are useful diagnostics even though they do not resolve the larger deployment gap.

Critical weaknesses
  1. The evaluation does not support the central claim about long-horizon agents. Every reported selector is evaluated with one context intervention from a frozen checkpoint, and all selectors reuse a precomputed KEEP/COMPRESS/DROP outcome matrix. A deployed policy can intervene repeatedly, and the first intervention changes the future state, history, failure distribution, and opportunities for later interventions. None of those distribution shifts are present in the main evaluation. Theorem 1 explicitly certifies only the finite-seed, single-intervention task-bundle protocol and excludes sequential deployment. An end-to-end experiment in which RECAP is invoked online throughout an episode, with repeated edits and full terminal success and total cost measured, is needed to justify the long-horizon-agent framing.

  2. The task and checkpoint population is heavily selected in a way that favors KEEP. Baseline screening retains successful trajectories with at least two detected failures, and checkpoint selection uses the completed baseline trajectory and post-failure information. Tasks that fail under KEEP, or whose useful recovery would require editing a trajectory that would not have completed under KEEP, are absent from the main population. The Hunyuan test set further contains 19 registered tasks but only 15 tasks with retained states, with four empty bundles contributing zero loss. This is a valid estimand if stated as such, but it makes the reported success preservation and zero observed harm conditional on a narrow, retrospectively curated population rather than representative of arbitrary online agent executions.

  3. The statistical evidence is too weak for the strength of the reliability claims. Each checkpoint/action is measured with only three rollout seeds in the main experiments, while the harm label is defined from the observed finite-seed mean difference. Thus the certification concerns a noisy observed label, not the latent success probability of an intervention. On Hunyuan, zero harmful registered tasks among 19 test tasks still has an approximate one-sided 95% upper bound of 14.6%, as the appendix notes; it should not be read as evidence that the true harm risk is zero. The formal risk budget is also 0.20, so even a valid certificate permits harmful intervention on up to 20% of task bundles. Moreover, the main Qwen results report policy-dependent paired-success savings, with different joint-success subsets for different selectors, and do not provide task-level confidence intervals or paired significance tests. The very small gains on Qwen3-4B and Qwen3-8B (+10.7 and +32.9 tokens) are especially difficult to interpret without uncertainty estimates. Unconditional saving, uncertainty, and a common task-level resampling analysis should be reported for every backbone.

  4. The practical and generality claims are limited by the data-collection protocol and domain-specific implementation. Training and calibration require matched KEEP/COMPRESS/DROP continuations for every retained checkpoint, which can require roughly three times the agent rollouts before any saving is obtained. The reported efficiency excludes feature-extraction, controller, and any broader deployment overhead, and it is prompt-token cost rather than wall-clock or monetary cost. In addition, the feature extractor relies on the known AgentCE-Bench tool schema, slots, candidate IDs, constraint categories, and a hand-defined dependency graph. COMPRESS and DROP are also specialized deterministic transformations, including duplicate-failure scrubbing. The experiments therefore establish a carefully engineered result on one structured benchmark, but they do not yet show that RECAP transfers to open-ended tools, unseen schemas, or standard context-compression operators.

Questions for authors

Clarification requests

  1. Can the authors provide an online, sequential evaluation in which RECAP may make multiple edits in the same episode, and report terminal task success, total prompt and completion tokens, latency, and the number of edits?

  2. What fraction of the original task pool fails the KEEP baseline, and how do the conclusions change if tasks and checkpoints are sampled online rather than from successful completed trajectories?

  3. Can the authors report task-level paired confidence intervals for success and unconditional token savings for all five backbones, using task as the resampling unit and preserving the checkpoint/seed bundle?

  4. Does certification ever reject a candidate in any backbone? Please report the candidate sequence, the candidates rejected, and the final policy compared with a matched LEARNED-NOCERT policy at equal coverage.

  5. How much wall-clock and monetary cost is required to collect the counterfactual training/calibration data, and how does the method behave with an unseen tool schema or a standard learned compressor?

Suggestions for future work

Please see the weaknesses above. The most important next step is to narrow the claim to single-checkpoint context intervention unless sequential online experiments are added, and to separate the empirical value of the selector from the value of the certification procedure.

4 comments
AuthorsOpenReview ↗

Response to Reviewer F9Qf (1/3): online evaluation and screened population (Q1, Q2)

Thank you for the detailed review; your questions determined most of our additional work. We answer in three parts.

Q1 / W1 — online evaluation with repeated edits. Done (Section 5.3, Appendix J.5): 60 new tasks (10 per domain) with no screening, 3 seeds, full episodes with the 50-step budget. The controller runs at every decision point identifiable from the prefix and may edit repeatedly. Methods: KEEP, RECAP with at most one edit, RECAP with repeated edits, LEARNED-NOCERT with repeated edits. The Qwen3-8B protocol was frozen before its first episode; the other two backbones were added afterwards under their own frozen protocols, i.e. after the Qwen3-8B results were known.

RECAP with repeated edits versus KEEP (paired task-cluster bootstrap, 95% CI):

BackboneKEEP successΔ success (pp)Relative prompt-token savingEpisodes with ≥1 / ≥2 edits
Qwen3-8B26.7%−0.56 [−5.00, 3.89]−0.83% [−4.05, 2.03]35.0% / 7.2%
gpt-oss-20b15.0%+1.67 [−3.89, 7.22]−7.50% [−19.17, 2.86]3.9% / 2.2%
Hunyuan-A13B16.7%−2.78 [−7.78, 2.22]+2.18% [−6.67, 10.17]16.1% / 6.7%
  • All 16 intervals (8 method-versus-KEEP comparisons; success and prompt tokens) include zero.
  • Completion tokens, total tokens, latency and edits, RECAP with repeated edits minus KEEP, per episode (Tables J.11–J.14):
BackboneΔ completion tokensΔ total tokensΔ latency (s)Edits
Qwen3-8B+54 [−14, 131]+2,351 [−5,648, 10,930]+1.54 [0.22, 3.03]0.44
gpt-oss-20b+612 [145, 1,078]+20,659 [−7,647, 49,071]+6.57 [1.50, 11.65]0.07
Hunyuan-A13B+146 [−625, 892]−9,305 [−46,243, 28,504]+13.7 [−81.4, 106.8]0.28
  • Three of these intervals exclude zero, all against RECAP: latency on Qwen3-8B, and completion tokens and latency on gpt-oss-20b. On gpt-oss RECAP edited only 7 of 180 episodes, and 99% of both differences arises in the 173 episodes it did not edit, so we cannot attribute them to the edits; we have no KEEP replicate on that backbone, and latency is confounded by server concurrency. The controller itself takes 31 ms per decision point.
  • Noise floor: re-running KEEP on the same tasks and seeds gives identical call and token counts in 93 of 180 units and differs by −1.67 pp [−5.56, 1.67]. On this backbone the between-method differences in success and prompt tokens are of the same size.

We find neither an episode-level saving nor a detectable loss of success; the intervals are compatible with effects of about ±5 pp on Qwen3-8B. The certificate does not cover this setting. The revision reports this as a neutral result and drops the end-to-end claim.

Q2 / W2 — population selected in favour of KEEP.

  • Screening. Before screening, KEEP fails on most attempted test tasks: Qwen3-4B 96.5% (811/840), Qwen3-8B 81.9% (465/568), Qwen3-14B 78.9% (220/279), gpt-oss 84.2% (256/304), Hunyuan 63.6% (56/88). All offline rates are conditional on "KEEP succeeds with at least two natural failures"; the introduction, Section 5.1 and the limitations now say so.
  • Checkpoint selection. It is retrospective, as you say: offline checkpoints are chosen from the completed KEEP trajectory (Appendix D.2.2). Features read only the prefix, but the choice of checkpoints does not.
  • Online sampling. No screening, prefix-only decisions. Of the 63 Qwen3-8B units RECAP edited, 47 fail under both arms, 9 succeed under both, 3 are KEEP-failure/RECAP-success and 4 the reverse.
  • Hunyuan empty bundles. 15 of the 30 calibration tasks have no retained checkpoint. With retained tasks only (m = 15) the smallest attainable bound is 0.218 > 0.20, so no policy would be certified. The certificate holds for the registered-task population only (Section 5.2).
AuthorsOpenReview ↗

Response to Reviewer F9Qf (2/3): statistical evidence and certification (Q3, Q4)

Q3 / W3 — statistical evidence. RECAP versus KEEP on the frozen test sets, task as the resampling unit, all continuations included (Table 2):

BackboneΔ success (pp)Unconditional prompt saving per continuationRelativeHarmed tasks
Qwen3-4B−1.59 [−4.76, 0.00]−4,203 [−12,634, 22]−7.53% [−26.21, 0.05]1/21
Qwen3-8B−0.85 [−2.59, 0.00]−477 [−3,749, 2,018]−0.61% [−4.83, 2.58]1/40
Qwen3-14B0.00+779 [264, 1,447]+0.65% [0.22, 1.21]0/41
gpt-oss-20b0.00+130 [0, 299]+0.06% [0.00, 0.13]0/45
Hunyuan-A13B0.00+4,620 [−1,262, 13,586]+2.13% [−0.60, 6.23]0/19
  • Savings. Only Qwen3-14B excludes zero. Your concern about Qwen3-4B/8B was justified: the S_pair column conditioned on jointly successful pairs (and, as our audit showed, summed prompt and completion tokens, contrary to its caption). Unconditionally both estimates are negative, driven on each backbone by the single harmed task (796k and 181k more prompt tokens than KEEP); without it they are +10 and +36 tokens per continuation. The original tables are kept in Appendix J.1 with a corrected caption.
  • Three seeds. On 30 stratified Qwen3-8B test checkpoints with seven new seeds (630 continuations; design frozen first), the harm label agrees with the three-seed label in 60 of 60 cells and the token-benefit label in 58 of 60. At temperature 0.1 seeds are strongly correlated, so this shows seed stability, not absence of noise. A complete re-run in a new server session agrees in 227 of 236 cells (96.2% [91.9, 99.2]).
  • Strength of the guarantee. We keep your reading: the certificate concerns the observed finite-seed label under a 0.20 budget, and 0/19 has a 14.6% upper bound. In a post-hoc rerun with a 0.05 budget every backbone falls back to KEEP (smallest attainable bound 0.062–0.116).

Q4 — does certification reject anything; equal-coverage comparison.

  • Rejections (Table J.9). Each backbone has 45 candidates in a fixed order. Qwen3-4B, Qwen3-8B and Hunyuan: none is rejected. Qwen3-14B: the first failure is candidate 43. gpt-oss: candidate 23.
  • Equal coverage (Table J.10). An uncertified threshold on the same predictor, matched to RECAP's development coverage, makes identical test decisions on four backbones; on Qwen3-14B three checkpoints differ. On Hunyuan RECAP and LEARNED-NOCERT coincide at all 37 checkpoints.

So certification does not improve measured performance at fixed coverage; it decides how much coverage is authorized. Selection is what the intervals support (Table 1, five backbones pooled):

MethodΔ success vs KEEP (pp)Harmed tasksEdited checkpoints
Always-COMPRESS−9.88 [−12.89, −7.00]61/166100%
Always-DROP−6.61 [−9.29, −4.00]47/166100%
SEMANTIC−1.71 [−3.48, −0.07]11/16634.1%
LEARNED-NOCERT−0.57 [−1.57, 0.35]5/16629.2%
RECENCY−0.14 [−0.43, 0.00]1/16617.3%
RECAP−0.43 [−1.07, 0.00]2/16617.9%

RECAP's task-harm rate is 5.4 pp below SEMANTIC's (CI 1.8–9.6). We claim no advantage over the RECENCY threshold, which is as safe at a similar edit rate.

AuthorsOpenReview ↗

Response to Reviewer F9Qf (3/3): cost, unseen schema, standard compressor (Q5)

Q5 / W4 — cost, unseen schema, standard compressor.

  • Collection cost (Appendix J.3). Qwen3-8B: 880 screening episodes (224M prompt tokens, 1.8 busy hours) and 3,573 matched continuations (293M, within a 1.9-hour window). Hunyuan: 591 episodes (272M, 36.5 busy hours) and 990 continuations (219M, within a 47.3-hour window on four GPUs). We have no billing record, so we give no dollar figure. Controller overhead is 8–31 ms per decision point. We do not present the collection cost as amortized by savings.
  • Unseen schema. Not evaluated. The features depend on the AgentCE-Bench schema (slots, candidate IDs, constraint categories) and on a dependency graph built from hand-written overlap rules (Appendix D.2). DROP also removes other assistant sentences that mention a dropped candidate ID together with a failure word (Appendix D.2.2). The revision lists this dependence as a limitation.
  • Standard compressor (Appendix J.8). On the 118 Qwen3-8B test checkpoints we replaced the template by a summary written by the backbone itself and re-ran all treatments (1,416 new continuations; nothing refit).
    • Compressing everywhere with the LLM summary: −7.3 pp [−14.7, −0.3], 11 of 40 tasks harmed. Template: −12.7 pp [−19.8, −6.2], 15 of 40.
    • The two compressors are not distinguishable (+5.4 pp [−0.6, 11.4]).
    • The frozen gate still separates the cases: the LLM summary lowers success at 18.5% of the checkpoints where RECAP abstains and at 2.7% of those where it edits (difference −15.8 pp [−25.0, −7.4]).
    • RECAP's own numbers do not change, which is uninformative: it selects COMPRESS at only 4 of the 118 checkpoints.

This is a post-hoc check on one backbone with one summarizer prompt, outside the certificate. We did not test an external compressor such as LLMLingua.

Your suggestion. We followed both parts: the claim is narrowed to single-checkpoint context intervention, and the value of the selector is reported separately from that of certification. The title is unchanged: "long-horizon agents" names the setting in which failure records accumulate, not an end-to-end gain, and the abstract and Section 1 state that the evidence concerns one edit at one checkpoint.

Thank you for the detailed response, and some of my concerns have been addressed. However, generalization to unseen tool schemas and standard compressors remains untested. In addition, the new results weaken the practical case for RECAP, including token savings are not statistically significant in most cases, the offline data-collection overhead is high, and the method does not outperform the simple baseline. Therefore, I will maintain my score.

Reviewer_oKMmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates when long-horizon agents should discard historical information. The core idea is that the value of the same failure record changes with the execution status, so it cannot be decided whether to delete it based solely on recency or similarity. The authors propose RECAP, which compares the impact of KEEP, COMPRESS, and DROP on subsequent success rates and token costs to learn when it is safe to compress or delete history, and uses risk controls to ensure that editing does not significantly impair task performance.

Critical strengths

The paper studies an important problem in long-horizon agents: when past information should be forgotten as its value changes over execution. It provides a clean formulation with KEEP, COMPRESS, and DROP, using matched counterfactual continuations to measure effects on success and token cost. The method is execution-aware, modeling relevance, resolution, state changes, and dependencies between memory and the current execution state.

Critical weaknesses

The counterfactual supervision can be computationally expensive, since constructing training and calibration data requires matched KEEP/COMPRESS/DROP rollouts over multiple checkpoints and seeds. This may limit scalability to larger or more realistic agent environments. The evaluation considers only a single context intervention per matched continuation. As a result, it does not capture how repeated compression or deletion decisions may interact and accumulate over a full long-horizon trajectory. The experimental setting is relatively limited and filtered, focusing on structured AgentCE-Bench domains and trajectories where the baseline already succeeds with multiple natural failures. It remains unclear how well the method generalizes to more open-ended settings such as web or computer-use agents. The learned selector is highly backbone-specific and requires separate fitting and calibration for each model. The weak zero-shot transfer across backbones suggests that the learned forgetting policy may not generalize easily across agent models.

Questions for authors

n/a

1 comment
AuthorsOpenReview ↗

Response to Reviewer oKMm

Thank you for the review. We address the four concerns in order.

1. Cost of counterfactual supervision. We agree that it is substantial and now report it (Appendix J.3): for Qwen3-8B, 880 screening episodes and 3,573 matched continuations (517M prompt tokens); for Hunyuan, 591 and 990 (491M). This is a one-time offline cost per backbone. At run time the rule does not execute the three alternatives: it computes features from the prefix and applies two thresholds (8–31 ms per decision point). We do not claim that the cost is repaid by savings.

2. Single intervention per continuation. We added an online experiment in which the controller runs throughout full episodes and may edit repeatedly (Section 5.3): 60 unscreened tasks, 3 seeds, three backbones.

BackboneΔ success vs KEEP (pp)Relative prompt-token savingEpisodes with ≥2 edits
Qwen3-8B−0.6 [−5.0, 3.9]−0.8% [−4.1, 2.0]7.2%
gpt-oss-20b+1.7 [−3.9, 7.2]−7.5% [−19.2, 2.9]2.2%
Hunyuan-A13B−2.8 [−7.8, 2.2]+2.2% [−6.7, 10.2]6.7%

No variant is distinguishable from KEEP in success or prompt tokens, and repeated edits are not distinguishable from a single edit (repeated edits are rare). We detect neither additional harm nor an episode-level saving, report this as a neutral result, and no longer claim an end-to-end benefit.

3. Limited and filtered setting. We agree. Before screening, KEEP fails on 64–97% of the attempted test tasks, so the offline results are conditional on tasks that KEEP solves with at least two natural failures; the revision states this. The online experiment removes the filter. We did not evaluate web or computer-use agents, and the claims are restricted to this benchmark.

4. Backbone-specific policy and weak transfer. Appendix J.6 decomposes the transfer result. The Qwen3-8B policy applied unchanged harms 0 tasks on every target but edits only 0–4% of checkpoints. Any gain comes from refitting the predictor on target data and is distinguishable from zero only on Qwen3-14B (+652 tokens per continuation, CI 107–1,347). The policy transfers safely but conservatively: on a new backbone it abstains.

What remains robust. Across five backbones, always compressing or always removing failure records lowers continuation success by 9.9 pp (CI 7.0–12.9) and 6.6 pp (CI 4.0–9.3), while RECAP harms 2 of 166 tasks. This is not specific to our template: on Qwen3-8B a summary written by the model itself lowers success by 7.3 pp (CI 0.3–14.7). The harm labels are stable under seven additional seeds (60 of 60 sampled labels agree; 227 of 236 in a full re-run). The contribution we now claim is this measurement and a risk-controlled way to abstain, not an efficiency gain.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful feedback. We are glad that reviewers found the problem important and practical (oKMm, 9scZ), the matched KEEP/COMPRESS/DROP formulation clean (oKMm, F9Qf), and the separation of measurement, proposal and certification conceptually clean (F9Qf).

The reviews asked for evidence the submission did not contain. We ran the requested experiments and report all of them, including those that do not favour RECAP.

Main additions in the revision (Section 5 and the new Appendix J):

  1. Online repeated editing (F9Qf, oKMm). Full episodes on 60 new unscreened tasks, three backbones, 1,980 episodes. No RECAP variant is distinguishable from KEEP in success or prompt tokens (Sec. 5.3, Table 3).
  2. Task-level statistics (F9Qf). Paired task-cluster bootstrap intervals and unconditional prompt-token savings on all five backbones (Tables 1–2, App. J.2).
  3. Indiscriminate editing is harmful (F9Qf, oKMm). Always-COMPRESS lowers continuation success by 9.9 pp (CI 7.0–12.9), harms 61 of 166 tasks and uses 7.9% more prompt tokens (CI 2.9–13.5); Always-DROP lowers it by 6.6 pp (CI 4.0–9.3). RECAP harms 2 of 166 (−0.4 pp, CI −1.1 to 0.0).
  4. Label reliability (F9Qf). With seven new seeds, 60 of 60 sampled harm labels agree with the three-seed labels; a full re-run agrees on 227 of 236 (App. J.7).
  5. Compressor substitution (F9Qf). With a summary written by the model itself, compressing everywhere still lowers success by 7.3 pp (CI 0.3–14.7) on Qwen3-8B (App. J.8).
  6. Screening, certification and cost (F9Qf, oKMm). KEEP failure rates before screening (64–97%), the full certification trace, an equal-coverage uncertified control, a transfer decomposition and a cost ledger (App. J.3, J.4, J.6).
  7. Presentation (9scZ). Abstract, introduction and contribution list rewritten; Sections 3–4 keep two displayed formulas and the rest moves to App. C–D.

Claims we withdrew.

  • Token savings. RECAP's saving is distinguishable from zero on one backbone only (Qwen3-14B, +0.65%, CI 0.22–1.21).
  • End-to-end benefit. No longer claimed; the contribution is restricted to single-checkpoint context intervention.
  • Certification as a gain. At equal coverage an uncertified rule makes the same decisions on four of five backbones. Certification authorizes a coverage level; it does not improve measured performance.
  • Correction. The S_pair column of the submission was the prompt-plus-completion token difference over jointly successful pairs, not a prompt-token saving. The main text now uses the unconditional metric (App. J.1).

Every new experiment used a design frozen before its first run; analyses designed after earlier results were known are identified as such. The title, method and frozen policies are unchanged.

Details are in the individual responses. We would be glad to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.