- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper addresses the problem of increasingly long contexts when using LLMs as agents. However, I find the method very unclear, so I can only provide my tentative understanding. The proposed approach first abstracts rules from examples and then trains an LLM with LoRA to learn how to retain information.
Critical strengths
-
The paper provides a comprehensive overview and categorization of prior methods.
-
The experiments on ALFWorld and ContractNLI are consistent with the application scenarios claimed by the authors.
Critical weaknesses
-
The introduction lacks coherence. First, the discussion of prior methods in the third paragraph does not establish a clear logical connection to the authors’ motivation and proposed idea in the fourth paragraph. Second, the fourth paragraph describes the authors’ approach with “we summarize …,” but its relationship to the subsequent introduction of latent context compression is unclear. As a reader, I find this progression confusing.
-
The central motivation appears to offer limited novelty. The authors state that “useful information includes both specific conditions and operational details, as well as decision patterns shared across examples.” However, this categorization appears to be based primarily on intuition. The proposed method then adds a corresponding classification step, whose novelty is not sufficiently established.
-
The method is insufficiently clear. Please see the specific questions below.
Questions for authors
-
What is the rationale for placing the related work section after the method section and before the experiments?
-
The paper states: “Let q denote the current input and a = (a1, . . . , aNa ) the target output of Na tokens, including the end token.” What exactly does represent? The sentence also appears grammatically incomplete.
-
Section 2.1 states: “A retriever selects K examples from an experience bank based on q.” How is this implemented? For example, does the retriever use a neural model or BM25? Does the selection involve retrieval, reranking, or both?
-
The paper states: “Given a pretrained language model pθ, we compress the selected examples and their associated rules into memory Mq with at most B positions.” How exactly is this compression performed? Is it done manually or using an LLM? If an LLM is used, what is the prompt?
-
What does represent in Figure 1(b)? It does not appear to be defined in the main text.
-
Could the authors include additional baselines, such as LLMLingua-2 and Latent Context Compilation?