acceptodds
Under review as a conference paper at ICLR 2027

Deletion Cost, Not Salience: Causal Measurement for Context Eviction in LLM Agents

Abstract

Agents with finite context must decide what to retain. Common eviction rules use proxies for how much an element still matters: age, message type, attention mass, or a model's judgement. We measure deletion cost directly: perturb one element, replay the identical continuation under teacher forcing, and read the change in log-odds on the decisions it governs, with a placebo control at each distance. The survey's position-controlled power-law exponents lie within , and deleting a standing constraint remains costly through steps. In separate probe schedules, attention half-lives span , while plan and constraint differ by in fitted causal half-life. Across eight open-weight models, all (model, probe-channel) cells remain on the relevant side of the fitted power-law boundary. Finally, scoring all static orders on a six-probe BF16 evaluation, we find that a frozen B order based on directly measured deletion costs beats the evaluated six-text and fixed-prompt task-informed judges on two Qwen models. A third model's informed judge recovers the direct order, but too few trajectories pass its gate for a stable cross-guise comparison. Fitting supplies no resolved advantage over the direct order. These results provide a measurement for pricing context deletion; how often an element will still be needed remains a deployment input to be measured.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Context Eviction, LLM Agents, Causal Measurement, Deletion Cost, Context Window Management, Attention vs Causality

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nQx6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the problem of context eviction in LLM agents. The authors contribute a controlled measurement for context deletion where they use paired teacher-forced replays with placebo controls to trace the effects of different context-element types across decision distance and later demonstrations. Then, the authors conduct empirical analysis to provide evidence against age- or attention-based eviction of different elements.

Critical strengths
  1. The problem of context deletion is both important and interesting for LLM agents. The work provides at least some insights that uncover the effects of context deletion that could potentially be useful.
  2. In the meanwhile, the environment is carefully controlled, the experiments are cleanly designed where the enumeration of factors is comprehensive.
Critical weaknesses
  1. The authors claim different measurement contributions, which I think present very limited novelty. Although the authors mention that they claim no methodological novelty for paired replay and attribution, they still fail to give any refreshing empirical insights regarding the behavior of LLM agents, especially without any models at larger scales.
  2. It is unclear about the impacts of the work. I think every result in the paper is from the difference in log probability in a synthetic task whose six families vary lexicon around one mechanism. The eviction comparison is quite narrow, and somehow unreliable. For a paper motivated by agent context eviction, the absence of any end-to-end outcome definitely damages the significance of the paper.
  3. I also find the paper extremely difficult to read. On the one hand, there are multiple estimators in the paper that yield different numbers for similar values. On the other hand, the paragraphs are also broken everywhere (for example, many undefined notations in the introduction), making many concepts and numbers difficult to interpret.
Questions for authors
  1. How can your methods generalize to models of different sizes and families?
  2. How can your findings provide any insights into free-running task success?
Reviewer_wYbCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the cost of deleting context elements in long-horizon LLM agents, examining how it varies with distance, element type, and subsequent demonstrations. The authors use paired context interventions and teacher-forced replay to measure shifts in correct-action log-odds with continuation held fixed, with unrelated probes to control for collateral editing effects in the distance analyses. Controlled experiments across eight open-weight models demonstrate that deleting certain standing constraints remains costly at long distances, supplemented by policy-text ablations on τ-bench and evidence-deletion checks on HotpotQA. To evaluate eviction decisions, the authors measure all deletion subsets of six context elements on three models and use these measurements to score all 720 static eviction orders under matched token budgets. An order derived from a separate 4B calibration experiment outperforms the evaluated text-only and task-informed LLM judges on two Qwen models, while fitted amplitude and demonstration-decay scores provide no clear additional advantage over direct measurement.

Critical strengths
  1. Controlled evidence of persistent influence from old constraints. Authors combine paired context interventions and fixed-history replay with position and unrelated probes as controls to measure the impact of any deletion on relevant decisions. Within the distant evaluation band for Qwen3.5-4B, the mean correct-option probability is 0.813 with the constraint retained and 0.081 with the deletion, providing concrete evidence that a model can successfully exploit an old constraint and be significantly harmed by its deletion. This finding directly contradicts the intuition that age alone makes context safe to evict.
  2. Comprehensive subset evaluation ties deletion measurements to budgeted retention choices. Authors measure all deletion subsets of six context elements and then use these measurements to evaluate all 720 static eviction orders, using the same token budgets to distinguish the single best common subset from a per-trajectory hindsight oracle. An order obtained from a separate 4B calibration experiment outperforms the evaluated text-only and task-informed LLM judges on two Qwen models, showing the value of the measured deletion costs in the controlled, fixed-history evaluation while exhaustive enumeration emphasizes the difference between static and budget-specific optimal choices.
Critical weaknesses
  1. The retention evaluation provides limited coverage of dependencies and redundancy between context elements. The paper exhaustively evaluates deletion subsets of six context elements, but each probe is designed to depend on a single element, and the six task families primarily vary lexical content rather than dependency structure. The experiments therefore establish the advantage of the measurement-derived static order in the evaluated setting, but provide limited evidence about whether that advantage persists when an element’s value depends on other retained information or when future query demands change. The supplementary τ-bench and HotpotQA experiments demonstrate sensitivity to deleting relevant information, but do not evaluate budget-constrained retention choices. Consequently, the empirical connection between deletion measurement and context eviction remains concentrated in a relatively simple task structure, limiting my confidence in the broader applicability of the findings.
  2. The demonstration-decay analysis does not fully establish the mechanism behind differences in information persistence. In the main experiments, the number of subsequent demonstrations is not varied independently of distance, position, and probe content. The fitted demonstration-decay parameters therefore cannot clearly distinguish the replacement of original information by subsequent demonstrations from other factors that vary with the experimental trajectory. In particular, declining deletion costs can also reflect deteriorating performance when the original information is retained, rather than recovery after deletion. The authors acknowledge this identification limitation and provide supplementary controls, but these use a different numerical precision and a modified setup, and do not directly validate the decay parameters from the main experiments. This does not invalidate the paired deletion measurements themselves, but weakens the mechanistic interpretation of differences in persistence across information types.
Questions for authors
  1. Sensitivity to query demand (additional analysis using existing measurements): The budgeted evaluation assigns equal weight to the six probes. Could the authors use the existing exhaustive subset measurements to assess whether the independently calibrated static order retains its advantage over the evaluated LLM judges under modest changes to these weights? I do not expect one order to perform best across all query distributions, but this analysis would help determine how strongly the reported comparison depends on the chosen demand distribution.
  2. Interpretation of cross-type demonstration effects (clarification request): Appendix C explicitly limits the fitted demonstration-decay parameters to descriptive quantities under the experimental schedule. Given this limitation, which conclusions about differences across information types remain supported without interpreting these parameters as re-learning rates? In particular, do the fixed-distance controls support any cross-type ordering beyond the local PLAN–CONSTRAINT comparison? This would inform my assessment of the mechanistic insight provided by the demonstration analysis.
Reviewer_1ZyeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In a synthetic console-style environment with six context-element types, the authors perturb one element (drop, pad, paraphrase or corrupt). They replay the identical continuation under teacher forcing and read the log-odds change on the decisions that element governs, with placebo differencing.

Critical strengths

Measurement design is carefully done but onerously detailed.

Critical weaknesses

The paper reads as primarily AI-written slop and I recommend desk-rejection on this basis alone. Moreover, the title is not demonstrated. The only attention-versus-causal evidence compares half-lives from different probe schedules across five element types (p = 0.188). The paper itself says this is “insufficient to establish a general rank mismatch” and “not a matched signal comparison”. On presentation, I found this paper nearly impossible to read. The abstract relies on terms that are never defined there (“lo”, “survey”, “probe-channel”, “guise”, “static orders”, “fitted power-law boundary”), and the main text defers many definitions to appendix sections, so it was difficult to even work out what the agent does in the task or what a single measurement looks like without reading the appendix first.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.