- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper focus on the problem of memory allocation in long horizon agents. The paper frames it as an optimization problem over utility (from an LLM oracle), and price augmented budget and provide an algorithm. The paper provided a theoretical bound on the regrets versus a hindsight method. The paper tested 3 capable LLMs on long context benchmarks and found that their method exceeded the baselines on all setting.
Critical strengths
-
The paper is timely, well motivated:
- The introduction and motivation explanation is easily understandable.
- Proofs seems to be correct with respect to their assumptions.
- Explicit memory allocation for long running agents in lieu of pure compaction is well motivated and I expect this topic to grow quickly. I think this is well placed in the literature with some papers already exploring similar avenues, and works to bridge context compression method to context management methods.
-
The overall method is reasonable and the result looks very promising.
- The methods, while convoluted, are well thought of and elegant.
- Under some assumptions, the method's error scaled sub-linearly to a hindsight benchmark.
- The results are the best overall among the baselines
Critical weaknesses
-
The paper written methodology are dense and hard to understand
- Several technical terms specific to the methods are never defined explicitly, relies on the reader's familariatity on operation's research topics, or defined too late (e.g. Priced-frontier regularity is mentioned in Introduction and Abstract, which then cannot be fully understood without looking at section 4 and Appendix C)
- The intuition behind the method choices are unclear or buried in the appendix.
- Limitations discussion are very limited.
- There are some inconsistensies in the tables and writing (e.g. GLM LongMemEval FIFO is 34.0 in Table 1 versus 30.4 in Table 3)
-
The methods is a bit unconvincing, and the analysis lacks systems discussion.
- Simulated results in the tables are very concerning.
- The priced-frontier regularity is very load bearing for the proofs of the theorems, while the empirical result show promise in many domains, it still might fail on others (Browsing comes to mind where good information is sparse. We can also derive some elementary repeating workflow that violate this assumption) and the queue cant react fast enough. There are no analysis on the applicability of this assumption to the benchmarks tested.
- The bound provided by Theorem 1 relates to the regrets of cumulative decisions, rather than downstream performance. While this is pedantic, I think its important to state the bound is for the optimality of further allocation and still leave room for error in earlier compression.
- There are no discussion on cumulative/running memory trajectories, we might imagine that the extra performance vs the baselines can be explained by keeping more token early.
- There are no explanation on the oracle at all. I think this is warranted as the oracle is the main backbone of the method, but very expensive (generating scores and all possible summaries in a turn), and the extra savings of reduce active context will be offset by the cost of running the oracles
Questions for authors
- Table 2 states "All values are simulated targets rather than observed experimental evidence", can you explain what this is? Which of the numbers of the result tables actually come from experiments?
- Is there an analysis on the effect/optimality of the five representation levels?
- I think the current strongest baseline of adaptive historical memory other than LRE is PACE[1]. I agree that a full experiment will be infeasible and this can count as concurrent work, instead can the authors give a short comparison of their methods (difference, advantage, disadvantage) here in more detail compared to the comparison in the related work section.
- Is there an analysis on how refresh window interact with the benchmarks and the comparison of fixed vs capacity-triggered refreshes?
- Do you think that your method will scale well beyond 0.6 compression? Is there experiments on sharper compression?
Comment: My overall score reflects the lack of readibility and supporting discussion of the paper rather than the method itself, which shows good result. I'd be willing to increase my score if the authors addresses the concerns. Simulated result declaration in Table 2 is very concerning and might be ground for a desk reject at the SAC/PC discretion, but I'm writing this review with the benefit of the doubt that its either an artifact or a misunderstanding.