acceptodds
Withdrawn from ICLR 2027

Self-Induced Distribution Shift in Agent Memory: Write-Time Retention Criteria Cannot Be Optimal

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Every published retention criterion scores a memory item by an estimate of how much it will matter later, computed under a different memory policy from the one the criterion is about to install. This is not a sampling nuisance to be reduced with more rollouts: the channel that compresses decides which screens the agent reaches, hence which facts are later needed, hence what it should have kept — the criterion's own output moves the distribution its score was computed against. We formalise the induced-distribution operator, prove its fixed point exists by Brouwer and is unique under a contraction condition, and show that the objective every write-time criterion actually optimises is exactly the first iterate of that operator, with its error governed two-sidedly by one measurable quantity: the shift between the law the criterion assumed and the law its own output induces. Optimality then requires that shift to vanish, which requires memory not to affect behaviour. We claim nothing for the operator: it is performative prediction's decoupled-risk map and the single-agent endogenous setting is performative reinforcement learning's. What is ours is the feasible set — a rate-constrained code, whose argmin jumps where a parametrised policy's moves smoothly, so solution stability is a hypothesis here and a mild condition there — and a quantitative failure needing neither: a two-point construction on which every write-time criterion, deterministic or randomised, pays at least half of sigma in performative risk at retention strength sigma, both bounds attained. Results go against us twice. The fixed point these criteria fail to reach is itself stable, not optimal, leaving 24.0% of the performative risk unclaimed — which raises the wall rather than lowering it. And we withdraw a theorem of our own: the separation we claimed grows with the horizon is a window, exactly zero at both ends. On 19 real application flows walked on devices, a 7B agent's law shift reaches 3.28x its measured noise floor at a quarter budget, moving 17 of 19 flows, with the mechanism visible: the revisit rate jumps from 0.38 to 0.497 because a code that forgets what was already done sends the agent back to do it again.

Primary area:
datasets and benchmarks
Keywords:
agent memory, distribution shift, performative prediction, retention criteria, rate-constrained coding, GUI agents, impossibility result, irreversibility, pre-registration, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_GtjTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

the paper states that A retention criterion is scored on evidence its own output will change. That is not an estimation problem with a better estimator behind it: optimality at write time requires the memory to be inert, and an inert memory is worthless. What a practitioner can do instead is concrete. The gap is governed by a measurable quantity — the law shift between the assumed and the induced distribution — which on a real agent, under the compaction rule that actually ships, reaches 3.28 times its own noise floor. We would ask a practitioner to run the two-sort check before tuning a retention score, since the weighting that follows from irreversibility is a different retained set rather than a rescaling, and to price the position-aware weight at long horizons. What we would not conclude is that the fixed point is the target: it is what an iteration reaches, 24.0% short of what a designer pays for.

Critical strengths

It is interesting to apply fix point analysis to LLm Agent behavior. It is also reassuring to understand that LLM are not yet capable of producing coherent text in some examples.

Critical weaknesses

The author incorrectly stated that the separation grows with the number of irreversible decision points, under the hypothesis that the distortion is additive over steps in an ealier version, as stated by the author. However, the earlier version is not cited or presented, this making this statement factually not verifiable. Without critical information the reviewer cannot determine the validity of this paper.

Questions for authors

What is the human-written assessment of the main contribution and the core idea of this work before Generative AI was used on the manuscript to condense a longer version into the present conference format?

Reviewer_WxrVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • The submitted paper is a rewrite of an earlier manuscript and presumes some familiarity with at least the experimental setup and the claims made therein.
  • The setting chosen is that of a model writing a (bounded) memory and then using that memory for the next action.
  • The memory in this paper consists of "screens" of a GUI application which has been visited.
  • This memory retention rule is scored on trajectories produced under one memory policy (full history) but deployed under another (the compressed one). This is generally treated as a nuisance, but the authors present an elegant formulation which makes it clear that it is a structural obstacle and prove that the write-time rules cannot be optimal.
  • The authors are posit that if the operator mapping a retention code to the best code under the trajectory distribution that the code induces is a contracting map, then it should have a unique fixed point, and present inequalities bounding the performance difference between the fixed point and the first iterate. They also provide a minimax bound and separation results.
  • They run experiments on a GUI application where the past screens which are visited are either kept in memory or forgotten and show a setting where the forgetting could cause an infinite loop.
Critical strengths
  • The core idea of finding a fixed point of the policy changing operator borrowed from performative prediction to memory writing agents is interesting.
  • The GUI setting of visiting different states makes the measurement problem tractable and is an elegant experimental setup.
  • The paper is candid about the limitations, but it contributes to the difficult of following it.
Critical weaknesses
  • The presentation of the paper, and the proofs, is very difficult to follow.
  • Novelty over performative prediction and performative RL is thin, and the authors concede most of it.
Questions for authors
  • The paper would really benefit from a thorough rewrite.
  • Appendix B and D appear to be empty?
  • The kappa and L seem to be mis-labelled in Assumption 1 and Appendix A.1?
  • Why define the problem with a myopic imitation distortion instead of task return, i.e. something closer to Q-learning? The weights that we have to calculate for the 2-sort heuristic w_i need to calculated off-policy anyway?
  • Theorem 2 requires D_w to be Lipshitz in the law (bounded ωd\omega d), but the assumptions do not seem to cover that?
Reviewer_eJJ4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The contribution considers how agents with limited memory decide what to retain and what to erase whilst solving a task. The authors argue that criteria based on runs with access to infinitely large memory are not optimal as finite memory at runtime produces a distribution shift that was not accounted for in agent's training. It also proposes to differentiate actions which can and which cannot be undone.

Critical strengths

The paper conveys a practically relevant message that simple and naive adaptations of agents trained with infinite memory to operation conditions when memory volumes are finite may lead to suboptimal results.

Critical weaknesses

The distribution shift induced by the agent’s own actions is expected. I am not sure what the actual novel finding of the paper is. It is well known that off-policy estimation can be suboptimal, while on-policy learning can be expensive. The authors offer a remedy. However, the key assumption this remedy is based on, the contraction condition (see Theorem 1, part (2)), cannot be verified a priori. The authors confirm this in the abstract.

The text is very difficult to read. I am also confused by several references to a withdrawn theorem, which suggest that the text could have been prepared more carefully.

Questions for authors

It is well known that off-policy estimates can be biased when the policy affects the data. Please clarify what finding of this paper could not be predicted from existing literature, and why it is important.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.