acceptodds
Under review as a conference paper at ICLR 2027

ARKA: TYPED MEMORY ACCESS FOR EFFICIENT LONG-CONTEXT INFERENCE

Abstract

Long-context LLM agents accumulate dialogue histories, task traces, exemplars, and evolving facts, and full-context prompting exposes this growing state to the backbone LLM, making the KV cache expand with the interaction history. Full-context prompting repeatedly prefills irrelevant history, while prompt compression operates after long-context material has been gathered and typically requires an additional model pass. We propose ARKA, an LLM-free lightweight framework that treats agent memory as query-conditioned working-context assembly. ARKA stores histories and typed memory units outside the backbone LLM, uses a lightweight router to decide how memory should be accessed for each query, assembles compact evidence on CPU, and calls the backbone LLM at most once for generation. Our information- theoretic analysis establishes sufficient conditions for compact working contexts to preserve answer-relevant information and shows that discarding required source relations can prevent answer recovery even when relevant words remain. The LLM-free routing framework is evaluated on Llama-3.1-8B, Qwen3-8B, and Qwen3.5-9B backbones without backbone-specific retraining. Across MemoryAgentBench, LongBench-5, LoCoMo, and MemBench, ARKA outperforms the mBERT-based LLMLingua-2-small compressor in all 11 comparable benchmark–backbone settings, with an average gain of 9.9 points and a median 6.4× speedup in recorded processing time. In a history- growth stress test, full-context prompting runs out of memory at 50× scale on an A100 80 GB GPU, while ARKA reaches 100× with only a few hundred backbone input tokens. These results show that scalable agent memory should be built by learning which memory evidence to activate and which access policy to use for each query, rather than by extending or compressing a monolithic prompt.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Large language models, Agent memory, Adaptive memory access, Retrieval-augmented generation, Efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8wobOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenges of inference cost and efficiency associated with long-term memory in long-context LLM agents. We propose the ARKA framework, which employs a lightweight router to adaptively select an appropriate memory access policy based on the current query—such as fact lookup, dialogue window retrieval, example retrieval, or full-context reading. The framework organizes historical information into distinct "memory views" and utilizes predefined retrieval and processing operations to construct a streamlined working context on the CPU, thereby reducing both the number of input tokens and the inference overhead for the backbone LLM. Experiments conducted across various LLM backbones and memory benchmarks demonstrate that ARKA can maintain or even improve response quality on certain tasks, while significantly lowering inference costs and reducing the VRAM (video memory) footprint typically incurred by long-context processing.

Critical strengths

This paper addresses the issue of memory access efficiency in long-context agents. The research motivation is clear.

Experiments were conducted across various LLM backbones and memory benchmarks. The results demonstrate that the proposed method effectively reduces input token counts and inference overhead, while maintaining strong performance on certain tasks.

Critical weaknesses

The method lacks sufficient innovation. The proposed ARKA framework essentially employs a lightweight MLP to classify predefined Memory Access Policies, subsequently constructing the context by combining BM25 retrieval with manually designed rules. The overall approach is quite similar to existing Adaptive RAG and Memory Routing methods; it primarily functions as a combination and system-level integration of established techniques, lacking significant technical novelty. Furthermore, the router relies on predefined labels for supervised training rather than optimizing the Memory Access Policy based on feedback from downstream tasks. Consequently, it fails to fully demonstrate the distinct advantages that a learned routing mechanism might offer over simple, rule-based approaches.

There is a lack of comprehensive comparison with existing, mainstream memory-based methods. The current experiments are limited to a few baselines—such as Full Context, LLMLingua-2, and A-Mem—and omit systematic comparisons with other representative agent memory methods, including MemoryBank, LightMem, Mem0, MemoryOS, MemP, and LangMem. Notably, BudgetMem (presented at ICML 2026)—which also explores query-aware memory routing and utilizes reinforcement learning to optimize the trade-off between task performance and computational cost—is highly relevant to the objectives of this paper. Although the authors cite BudgetMem in the "Related Work" section, they do not provide a direct experimental comparison. It is recommended to include these baselines—specifically BudgetMem—using the same benchmarks, backbones, and computational budgets to more robustly validate the effectiveness and innovation of the proposed method.

Questions for authors

Please see above.

Reviewer_D4B9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ARKA (Adaptive Routing and Knowledge Assembly) is an LLM-free memory access layer that keeps an agent's histories, documents, dialogues, fact updates, and exemplars in CPU-side, source-linked views and assembles a compact working context per query instead of prompting the backbone with the full history or compressing it with an extra model pass. A 0.58M-parameter two-stage controller, trained on about 36K route-labeled examples with hashed lexical embeddings plus 42 numeric features, picks one of 15 access policies (view, operator, evidence budget, formatter), constrained by schema guards and source adapters. Typed operators preserve the relation each query needs: versioned records resolve the latest fact directly without generation, dialogue windows keep speaker adjacency, exemplar banks keep input–label pairs, and BM25-ranked cards recover source spans; evidence paths call the frozen backbone once. An information-theoretic analysis shows compact access is lossless when the operator's statistic is sufficient and recoverable, and that any order-invariant representation of m updates can answer a latest-value query with probability at most 1/m. With one frozen controller across Llama-3.1-8B, Qwen3-8B, and Qwen3.5-9B, ARKA beats LLMLingua-2-small in all 11 benchmark–backbone settings by 9.9 points on average with a median 6.4× recorded speedup, edges full context on MemoryAgentBench (+1.5 to +6.4) and LoCoMo (though the bootstrap interval includes zero), but trails full context by 1.9 to 11.1 points on LongBench-5 and by 0.9 to 4.8 on MemBench, and trails A-Mem on LoCoMo composite score (39.8 vs 43.4) while building memory in 0.057 s versus 3,414 s and zero LLM calls. A history-growth stress test shows full context runs out of memory at 50× history on an A100 80 GB while ARKA holds about 480 input tokens through 100×. The authors conclude that scalable agent memory should learn which evidence and which access operator to activate per query rather than extend or compress a monolithic prompt, and they acknowledge weaker results on distributed-evidence document reasoning.

Critical strengths
  • The core design principle is sound and well argued. Choosing a relation-preserving access operator per query, rather than compressing or extending a monolithic prompt, is a clear idea. The paper identifies concrete relations each query type needs: update order for facts, speaker adjacency for dialogue, input–label pairing for exemplars.
  • No LLM in the memory layer. Memory construction and routing run entirely on CPU with a 0.58M-parameter controller. Building views for ten LoCoMo histories takes 0.46 s versus roughly 3,400 s and 2,000 LLM calls per history for A-Mem. For services that hold many histories and query them rarely, this is a large practical advantage.
  • One frozen controller serves three backbones. The same checkpoints work with Llama-3.1-8B, Qwen3-8B, and Qwen3.5-9B without retraining, so swapping the reader does not require touching the memory layer.
Critical weaknesses
  • The claim is framed against the wrong baseline. The abstract leads with beating LLMLingua-2-small in all 11 settings. Against the reference that matters, full context, ARKA wins only on MAB and marginally on LoCoMo, and loses on LongBench-5 (up to 11.1 points), MemBench (0.9 to 4.8 points), and HotpotQA (13.8 points). LLMLingua-2 is slower than full context in most cells, so the 6.4× speedup is largely a statement about that compressor's overhead.
  • Timing comparisons are not apples to apples. ARKA's query times for Llama and Qwen3-8B come from a separate rerun that roughly halved its own numbers (0.997 s to 0.576 s on MAB) while baseline times were kept from the original pass. CPU access, assembly, and tokenization are excluded from the ratios. A-Mem's timer includes retrieval and internal calls while ARKA's does not. The appendix concedes the fields "do not share a complete wall-clock boundary." The reported ratios therefore cannot support a speed claim.
  • Possible train–test family overlap. The controller is trained on BANKING77, CLINC150, TREC, and HotpotQA training rows, which are the same families as MAB's exemplar tasks and LongBench's HotpotQA. The exact-string audit covers only LoCoMo. No held-out-schema or unseen-family evaluation is reported, so transfer of the router is unverified.
  • Record integrity problems. Per-query records and the RAPTOR/Provence configuration manifest "were not recovered." Table 3's full-context MAB score differs from Table 2's with the cause "unresolved in the retained records." One Qwen3.5-9B LoCoMo run is excluded for using a non-commensurate metric. Results that cannot be traced to a recoverable configuration should not appear in a comparison table.
Questions for authors
  1. Can you rerun full context, LLMLingua-2, and ARKA under one wall-clock timer that includes CPU access, assembly, tokenization, and generation, on the same hardware in the same pass? The current ratios mix a dedicated ARKA rerun with original baseline times and exclude ARKA's CPU work.
  2. The controller is trained on BANKING77, CLINC150, TREC, and HotpotQA training rows, which are the same families as MAB and LongBench tasks. Can you evaluate with those families held out from training, or on a benchmark with no shared schema?
Reviewer_T7fNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARKA, a lightweight framework for efficient long-context inference through query-conditioned memory access. ARKA stores the history as different memory views on the CPU, such as dialogue windows and versioned facts. A small MLP router selects one of 15 access policies based on the query, while schema guards handle known task formats. It then either answers structured queries directly or uses BM25 to retrieve relevant evidence and calls the LLM once to generate the answer. Experiments across four benchmarks and three LLM backbones show better performance than LLMLingua-2-small, with lower recorded processing time, although ARKA does not consistently outperform full-context inference.

Critical strengths
  1. Different memory access strategies for different queries. Instead of treating all queries the same way, ARKA considers what type of information is needed and how it should be retrieved.

  2. The reduction in input length is impressive. On LoCoMo, ARKA reduces the input tokens by 91.5% while achieving similar accuracy to full context (39.8 vs. 38.3). This is useful for agents that need to handle long interaction histories.

Critical weaknesses
  1. Limited methodological novelty. Most components, including the MLP router, BM25 retrieval, and memory operators, are fairly standard. The main contribution seems to be how these components are combined. However, without a comparison to a rule-based router using the same operators, it is unclear how much the learned routing actually helps.

  2. The baseline comparisons and efficiency claims are not reliable. The reported speedups use different timing setups. ARKA's timing excludes CPU retrieval and assembly, while LLMLingua-2's includes compression. The RAPTOR and Provence comparisons also lack complete experimental details, making the reported improvements difficult to verify.

Questions for authors
  1. How much of the performance gain comes from the learned router rather than the memory operators and schema guards? Could the authors compare against a rule-based router using the same operators?

  2. On LoCoMo and MemBench, where schema guards do not apply, how much does learned routing help compared to a fixed access policy, such as always using dialogue windows?

  3. RAPTOR/Provence baselines. Could the authors clarify the experimental configurations for RAPTOR and Provence, particularly the retrieval budgets and inference settings? Why do the MAB scores in Table 3 differ from those in Table 2?

  4. Could the authors report end-to-end latency using the same timing boundaries for all methods, including CPU retrieval, context assembly, and compression?

  5. The paper uses separate training and evaluation samples, but some task formats overlap. Could the authors evaluate how well the router generalizes to unseen task formats?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.