acceptodds
Under review as a conference paper at ICLR 2027

What Compression Preserves: Instruction Survival in Long-Horizon LLM Agents

Abstract

When a long-running LLM agent fills its context window, production frameworks compress its history and continue. This compaction becomes the agent's effective memory of its own session, but it is unclear which standing instructions survive the rewrite. Across 1,175 evaluable trajectories on -bench retail and AppWorld, we vary three compression trigger thresholds, two compression methods (truncation and LLM summarization), and three reminder frequencies against a no-compression baseline, mechanically scoring every planted instruction. We find selective forgetting based on whether later trajectory content re-cues a requirement: on -bench, two task-external output requirements stated in the first user message drop from 82% and 87% without compression to 22% and 32% across compressed conditions, whereas rules coupled to the ongoing trajectory stay at 89–100% when pooled over all conditions. Among compressed trajectories that complete the task, 63% and 55% still violate these two requirements, so task success alone does not measure memory faithfulness. The pattern persists under changes to compaction settings and in a 40-task pilot with a second agent model family. The most aggressive compression also reduces task success by up to 24.7 (-bench) and 33.6 (AppWorld) percentage points and may increase input-token use: under uncached full-schema billing, compressed -bench conditions use 13–42% more input tokens than baseline; caching the schema prefix largely removes truncation's excess but not summarization's. Scheduling a single mid-conversation reminder improves compressed -bench adherence by 5.6 percentage points, an effect that does not replicate on AppWorld. We release the evaluation harness, constraint catalogue, and trajectories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, long-term memory, context compression, agent evaluation, context management

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_gotaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how context compression affects the retention of standing instructions in long-running LLM agents. It tests three compression thresholds, two compression methods, and three reminder frequencies on τ-bench Retail and AppWorld. The results show that agents often forget instructions unrelated to the main task, especially output-format requirements. Instructions repeated in later interactions are better retained. Aggressive compression can also hurt task success and sometimes increase token usage.

Critical strengths

The paper investigates an important problem in long-horizon agent tasks: with context compression or truncation, agents may complete tasks successfully while failing to follow earlier instructions. This issue is relevant to real-world agent systems.

Critical weaknesses
  • The constraints differ in how much effort they require from the agent. Trajectory-coupled constraints (C2–C5) may be satisfied naturally during task execution, and some, such as C4, may already align with the agent's default behavior. In contrast, task-external instructions (C1 and C6) require additional behaviors. This difference in difficulty may explain the observed gap in instruction retention. Comparing constraint satisfaction rates alone is not sufficient to support the contribution 1 (Compression destroys task-external output requirements far more often than trajectorycoupled rules).
  • The paper lacks a direct analysis of what happens to instructions during compression. It uses constraint satisfaction to measure instruction retention, but does not distinguish whether an instruction was removed during compression, preserved but ignored by the agent, or rewritten that affects compliance.
  • The paper mainly reports empirical findings. It only evaluate one potential solution (instruction reminders), while the other recommendations (Section 5) lack experimental validation.
  • The evaluations only consider two basic compression methods, (truncation and LLM summarization), without evaluating any representative context compression methods. This limits the scope and generalizability of the empirical findings.
  • The paper is difficult to follow. The descriptions and categorization of C1–C7 are confusing. The main parts lack a clear narrative, while the results mainly list metrics with limited discussion. The paper would benefit from a substantial rewrite.
Questions for authors

See above

Reviewer_Ui5cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks which standing instructions survive when an LLM agent's context is compressed mid-session. On τ-bench retail and AppWorld, it crosses three trigger thresholds (50/70/90% of an imposed 8K-token budget), two compressors (truncation and LLM summarization), and three reminder schedules, plus a no-compression baseline. In total it reports 1,175 trajectories from a Qwen3.5-27B agent, and every planted constraint is scored by a program, not an LLM judge. The main claim is that "task-external" output wrappers drop from 82%/87% to 22%/32% under compression, while "trajectory-coupled" rules (C2–C5) stay at 89–100% (Section 4.1). Further results: aggressive compression costs up to 24.7/33.6 pp of task success, τ-bench token use depends on caching, and one mid-trajectory reminder gives +5.6 pp on τ-bench only. A 360-trajectory post-hoc study (Table 1) varies the compaction parameters and adds a Llama-3.3-70B agent.

Critical strengths

Careful, reproducible measurement of an under-studied deployment issue. Compaction is common in production agents. The paper scores instructions mechanically, reports cluster-bootstrap CIs, FDR screens, and ICC corrections, and releases the harness and trajectories. Useful decoupling of task success from instruction adherence. Among compressed trajectories that succeed, 63% still violate C1 and 55% violate C6. This clearly shows that success-only metrics hide memory failures.

Critical weaknesses

The central "trajectory-support" split is confounded by checker design. The "task-external" group is only two final-output wrappers. The "coupled" side includes C3, which the paper admits cannot fail (Section 6), and C2, which passes by default in 22% of τ-bench and 93% of AppWorld cases. So the split compares rules that can fail against rules that mostly cannot. The only real cases are C1/C6/C7 (all erode) versus C4/C5, so the general claim about "re-cueing" rests on very few constraints. Also, C1/C6 are judged on the final turn, so they are tied to how the trajectory ends (Figure 3 ends in a raw tool call).

The proposed mechanism is not tested directly. The obvious explanation is that compaction deletes the first user message that holds the instructions. The paper never checks whether the instruction text survives in the compacted context or in the summary (the summarizer prompt even asks for "constraints"). It also never runs the obvious control of pinning that message or moving it to the system prompt (Section 5 marks this "untested"). Without these, the finding is close to "deleted instructions are not followed," and the reminder result (intention-to-treat, delivered in only 60/135 baseline trajectories, no effect on C1/C6) does not fill the gap.

The regime is artificial, and the effect sizes depend on it. W=8K is far below the model's real context. On τ-bench the 2,400-token target is never reached because of the ~3.5K floor, and triggers fire about 17% later than labeled. Table 1 shows adherence ranging from 46.7% to 92.5% and C1/C6 from 0% to 89% as parameters change. With one main agent, a same-family simulator and summarizer, and no production compactor, the reported levels may not transfer.

Questions for authors
  1. In compressed trajectories that violate C1/C6, how often is the constraint text still present (verbatim or in the summary) at the final turn? Is failure ever seen when the text is retained?
  2. Can you add an arm that protects the first user message or places the constraints in the system prompt? This would test whether the effect is about cue removal or about re-cueing.
  3. How much of the C1/C6 loss comes from trajectories that end at the turn limit or on a tool call, rather than from a final text answer that omits the wrapper?
  4. Can you add task-external constraints that can actually fail, and trajectory-coupled ones that are not trivially satisfied, so that the split is not driven by checker design?
Reviewer_qiUWOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which instructions an LLM agent keeps after its context is compressed. On τ-bench and AppWorld, the authors vary when and how compression happens and check planted constraints with rule-based checkers. They find that output format requirements stated only at the start are often lost, while other rules mostly survive, even in runs where the task succeeds. They conclude that compression should be evaluated on instruction adherence as well as task success.

Critical strengths
  1. A relevant question with a judge-free protocol. Compaction is usually judged by task success or token savings. The paper shows that task success can hide the loss of requirements that downstream systems depend on, and all checks are deterministic. This is the main contribution I see.

  2. Honest reporting of uncertainty. The authors use clustered resampling, report reminders as intention-to-treat and openly flag weaknesses in their own checkers. This makes the limits of the evidence easy to judge.

Critical weaknesses
  1. The experiments are too narrow for the conclusions to generalize. This is my main concern. The main sweep uses a single agent, Qwen3.5-27B, and the summarizer and user simulator come from the same model family. The second model, Llama-3.3-70B, is only tested on 40 tasks with one run each. All results come from the authors' own harness and two simplified compression methods, and none of the production compaction systems that motivate the paper is evaluated. The data cover only 74 tasks from the τ-bench retail domain and 20 AppWorld tasks. The setup also departs from real use. The 8K context budget is artificial, and the instructions are placed only in the first user message, which is exactly the message truncation removes first.

  2. The proposed mechanism is not established. The paper attributes survival to later trajectory content re-cueing a rule. However, most of the constraints that "survive" can hardly fail. C3 is fixed by the tool schemas, C2 passes whenever no decimals appear, and C4 is a word the model rarely uses anyway. Their high adherence does not show that the instruction was remembered.

  3. The measurement of forgetting is not reliable. C1 and C6 only check the final answer. Compression increases the number of turns, so an agent may never produce a final text answer and fail these checks automatically. The paper also does not separate whether the summary dropped a constraint or the agent ignored it, and the checkers were not audited by hand.

  4. No effective remedy is offered. The only remedy tested, a mid-trajectory reminder, does not help C1 and C6 and does not replicate on AppWorld. The recommendation to move instructions into the system prompt is never tested.

  5. The writing and presentation fall well short of a top venue. The abstract and main text are crowded with numbers and coined terms, which makes the paper hard to read.

Questions for authors

see weanknesses above

Reviewer_PPrbOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discusses which standing instructions survive after long-horizon LLM agents compress their context. Crossing three trigger thresholds, two methods (truncation and summarization), and three reminder frequencies against a no-compression baseline on τ-bench retail and AppWorld, the authors collect 1,175 Qwen3.5-27B trajectories with deterministically scored constraints. They find selective forgetting: task-external output requirements drop from 82%/87% to 22%/32%. At the same time, trajectory-coupled rules (rounding, snake_case, prohibited word) hold at 89–100%, which they attribute to whether later trajectory content re-cues the requirement. Aggressive compression also costs up to 24.7/33.6 task-success points; the two methods' ranking reverses across benchmarks; token savings depend on prefix caching; and a mid-trajectory reminder helps only slightly on τ-bench.

Critical strengths
  1. The research question of this paper is interesting and important. Recent agent studies always focused on whether the task is successfully completed. This paper emphasizes whether the standing instructions can survive after the agent compresses the context.

  2. The authors systematically examine compression trigger thresholds, compression methods, and reminder frequencies, using τ-bench and AppWorld, which have different tool-calling mechanisms.

  3. Compared with work that only compares compression ratio against task performance, this paper measures constraint adherence, task success, and input-token consumption jointly, giving a relatively complete empirical picture.

Critical weaknesses
  1. The paper presents whether the standing instruction can survive depends on whether later trajectory content re-cues the requirement, but they do not isolate this factor in the experiments.

  2. The trigger policy and post-compression target are fixed to this compression regime, so it remains unclear whether the conclusions transfer to other agent architectures and production compaction systems

Questions for authors
  1. Can you run a controlled experiment that varies only whether later interactions re-cue a standing instruction, holding the instruction, its wording, and the task setting fixed, and show the results?

  2. Violations of C1 and C6 may stem not from instruction forgetting but from the agent failing to produce a final response—for example, when a trajectory ends on a tool call. Can you disentangle instruction forgetting from task-execution failure, e.g., by conditioning on trajectories that end with a free-text final response?

  3. How do you rule out constraint-design confounds in the main comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.