- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focus on the problem of repeatd context compression in long horizon agents. The authors diagnose that compression mainly affect cross run reliability, which they then use to propose their solution. PAIR tries to identify which compressions that reduces the integrity of the rollouts by comparing repeated rollouts and revise the compression. The paper evaluate GPT 5.6 Luna and Minimax M3 across 3 long horizon agentic tasks, and achieves a very strong performance in accuracy while maintaining a competitive Token-Acc Pareto frontier.
Critical strengths
- The paper is timely, with clear writing and motivations from both experiment and theory
- Memory management is crucial for long horizon agents and when/how to compact and manage memory is a contested topic in recent literature.
- Their writing is clear, straightforward, and easily understandable along with clear visualization.
- Their preliminary experiments on vs along with their short theory on the equivalence of boundary performance gap and overall task reward is clear, well motivated, and while trivial can serve as a valuable insight on future work in this area.
- The methods itself are clean, and shown to perform really well on benchmarks
- The methods are straightforward and easily understandable. Their way of estimating the harmful boundaries while simple, are effective.
- The results are very good. It performs the best accuracy-wise on AppWorld and against ACON (which is quite a strong baseline) by a big margin.
- The deployment (for in-distribution tasks) is straightforward, with competitive overall tokens.
- The trained prompts is transferable across model with minimal drop in performance, which is surprising to me!
Critical weaknesses
- There are inconsistencies between main table and downstream analysis table
- For example in Table 8 and Table 1/2, the delta from AppWorld prefix does not match.
- The method is expensive to run and favors easily repeatable tasks
- The formulation of the objective necessitates that the method knows the reward for each resumed compression boundaries, which is obtained from multiple rollouts. This favors tasks that are repeatable (have some sort of statically saved environment state) with unclear continuation on online adaptation of unalterable environment.
- The cost of training is quite significant. Even accounting for the repeatability of the system prompt, the cost for adapting to one task family is not negligible.
- There are no analysis on how the trained prompts interact with OOD data (both OOD on similar task, e.g. AppWorld -> -bench, or to general tasks, e.g. long math reasoning or agentic coding that necessitates compaction), making it questionable for deployment as each task family need to be classified correctly to the same prompt
Questions for authors
- Can the authors discuss the thoughts behind the design choice to use "fixed template" for revisions? I know that the results itself is good and the full comparison with ACON is available in the appendix, but expanding the very short explanation on this (how to design it, what can be changed in the compression template, etc) and including structural ablation in the main text would be valuable to first time readers.
- In the section 3.3 analysis, it seems like the harmful compression is concentrated in the earlier parts of the rollouts (which makes sense!) Does the authors provide analysis on this (e.g. how it affected the trained and final rollouts, how impactful it is, etc)? Furthermore, the section itself is dense and hard to understand (too much data dump thus feels a bit too AI-written) and it would be nice to see the section to be tidied up.
- Harmful compression exist in both successful and unsuccessful rollouts, Is there an analysis on how the model recover from it? Furthermore I think there might be some value in analyzing the boundary level effect of trained and compare the pre- and post-trained distribution of the harmful compression boundaries. (Figure 2)
- Do you think the method favors some task better than others, as Figure 5 might suggest that LLM lingua and no compression is also competitive in token/acc frontier? It would be nice if there exist an analysis/figure on the Pareto frontier on cost or latency or tokens.
Comment: This is a well made paper, with interesting and valuable insights. Limitations exists, but it is intertwined with the core method itself and are there due to the paper's early position in the literature. I think the amelioration for it is out of scope of the rebuttal and instead can serve as the basis of future works. My main concern therefore lies in the shown result consistencies instead and I'd be willing to increase my score if it has been resolved/clarified.
4 comments
-
q3: How does the agent recover from harmful compressions, and how does the distribution of harmful boundaries change after adaptation?
-
R q3: The agent recovers by re-acquiring the information the summary omitted, and after adaptation harmful compressions become rarer and milder. In the cases we examined, the summaries contain no incorrect values but omit a required step. Continuations that re-derive this step from the task instruction or the environment succeed at the cost of a few extra steps, whereas those that follow the summary directly fail (5/5 vs. 0/4 at one boundary). We added case studies of being misled by, recovering from, and benefiting from a summary in Appendix D.12.
For the distribution, we repeated the analysis of Section 3.3 with the adapted prompt. The share of boundaries that reduce continuation success drops from 23.9% to 17.1%, and the improvement comes from the first three compressions. Some harmful compressions remain, and interaction burden is not reduced. Since the adaptation evidence is collected under the initial prompt, a second adaptation round is a natural next step. The heatmaps appear in Appendix D.11 (Figure 11 in the revised version).
-
q4: Task-dependent performance and cost–accuracy trade-offs.
-
R q4: Thank you for this observation. We agree that the benefit of PAIR depends on the task, and we find it is largest on tasks that require more reasoning and planning. On AppWorld, PAIR improves Pass$^3$ over the unadapted prompt by 18.7 and 9.5 points on medium and hard tasks, compared with 7.0 points on easy ones. Token-level compression also struggles most here: LLMLingua-2 is competitive on the two smaller benchmarks but falls well below PAIR on AppWorld.
For the cost–accuracy trade-off, Figure 5 plots Pass$^3$ against total tokens for all methods. On AppWorld, PAIR is slightly more reliable than no compression at about the same total tokens. On OfficeBench, it uses 15–19% fewer tokens. On -Bench Retail, no compression remains the better choice, since the compressor overhead outweighs the savings. We include no compression as a reference point, because long histories eventually exceed the context window in practice.
Thank you for the clear revision. My primary concern has been resolved. To reiterate, I think this is a good work that are clean and well motivated, and even though limitation exists (high adaptation cost, limited OOD generalization, etc), the author acknowledged it and I agree that it does not diminish the value of this work and thus can serve as a future research direction. As a final suggestion, I would like to see the actual pareto frontier of Figure 5 be drawn. I hope the authors can utilize the extra page allowance of the rebuttal to insert the extra analysis they did during the rebuttal (q3 and q4) as I think it is interesting, provide insight to their method, and are valuable to readers.
I acknowledged the author's response and have increased my score
Thank you for the encouraging feedback and for raising your score! Following your suggestion, we have drawn the Pareto frontier among compression methods in Figure 5 in the updated revision.
We also agree that the analyses added during the rebuttal (the post-adaptation boundary analysis to section 5, the case studies to section 3, and the task-dependent benefits to section 5) are valuable to readers and belong in the main text. To keep the page and line references in our responses stable for the other reviewers during the discussion period, they currently remain in Appendix D.11–D.12; we will integrate them into the main text using the additional page in the final version.
We sincerely appreciate your efforts to review our paper and provide valuable suggestions. Below we address each concern in detail.
w2: High adaptation cost, reliance on repeatable environments, and limited OOD generalization.
R w2: We acknowledge these limitations and view PAIR as a starting point toward efficient and reliable localization of compression-induced errors in long-horizon agents. Replayable environments are required only during offline adaptation; deployment adds no rollouts or model calls beyond standard compression. Successive halving reduces counterfactual rollouts by approximately 42% and total adaptation tokens by 30.1% relative to uniform evaluation (Appendix D.7).
Regarding OOD generalization, we follow ACON's setting of task-specific prompt adaptation. Because compression requirements depend on task-specific instructions, constraints, and execution states, we envision PAIR as a way to develop specialized compaction skills for important long-horizon task families, optimized offline once and reused within each family. We demonstrate zero-shot cross-model transfer (Appendix D.8); cross-task OOD generalization remains our future work.
Beyond the method, we hope our analysis is useful in its own right. It characterizes how compression errors are distributed across long-horizon trajectories and shows that severe degradation concentrates at a small number of compression boundaries. It also shows that targeting these harmful boundaries can further improve overall performance, which makes them a practical handle for future compression methods. We are encouraged to see related efforts emerging in this direction, and we plan to build on these findings to develop cheaper ways of identifying harmful compression events. Upon acceptance, we will release our complete rollout records to facilitate future research.