acceptodds
Withdrawn from ICLR 2027

ForeCache: Efficient LLM-Based Multi-Agent Systems with Intent-Conditioned Pruning and Amortized KV Reuse

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-agent systems based on large language models (LLMs) address complex tasks such as code generation by exchanging and refining intermediate outputs, but repeatedly processing accumulated interaction histories leads to high inference latency. Existing approaches reduce this overhead through history compression or key-value (KV) cache reuse. However, compression can remove information needed by the next agent, while direct KV reuse can reduce downstream answer accuracy when cached states no longer match that agent's context. To address these limitations, we propose ForeCache, a training-free framework for efficient multi-agent inference. Specifically, consumer-conditioned saliency (CCS) uses each downstream agent's instructions to estimate the task relevance of shared history and guide pruning. We also introduce divergence-guided recomputation (DGR), which reuses upstream KV states and stops further recomputation for tokens whose cached and updated states are sufficiently close, reducing repeated history encoding. Experiments on well-known benchmarks show that ForeCache maintains task performance comparable to full prefilling. In the six-agent setting, it achieves up to a 6.6× speedup in the final agent's time to first token (TTFT) over full prefilling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent LLMs, KV cache, prefill acceleration, efficient LLM inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y1jVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ForeCache is a training-free method for reducing repeated processing of shared history in LLM-based multi-agent systems. It combines consumer-conditioned saliency (CCS), which keeps history relevant to the next agent’s task, with divergence-guided recomputation (DGR), which reuses earlier KV-cache states and selectively updates them when the new agent’s context changes.

The authors compare ForeCache with full prefilling and two cache-reuse baselines on GSM8K, MMLU, and HumanEval using two to six agents. ForeCache generally maintains accuracy close to full prefilling. In a six-agent AIME-24 efficiency test, it reports a 3.4× aggregate speedup and a 6.6× speedup in the final answer agent’s time to first token. Ablations examine the contributions of history selection and cache recomputation.

Critical strengths
  1. The paper provides useful evidence for both parts of ForeCache. In the Swap experiment, changing only whose instructions guide history selection lowers evidence recall from 87% with the intended consumer’s prompt to 66% with another consumer’s prompt and 61% with producer attention. This directly supports selecting history for the agent that will use it. The ablation also shows why KV repair matters: direct cache transfer reduces accuracy by 3.8–9.2 percentage points across the three tasks, while DGR recovers performance close to full prefilling. These are the paper’s strongest results because they show that both components have a substantive effect, even though the broader accuracy and speed claims need stronger validation.
Critical weaknesses
  1. The evaluation is too narrow to support the paper’s generality claims. Task accuracy is measured on three benchmarks using two 7–8B models, but no benchmark is run on both models, so model and task effects cannot be separated. The experiments also use one fixed sequential agent setup. The headline 6.6× speedup comes from a 30-problem AIME-24 test without task accuracy; speedups are smaller where accuracy is measured. A stronger test would include a task where multi-agent communication clearly helps and report both accuracy and latency.

  2. The comparisons are also difficult to interpret because the evaluation sets are small and most results have no reported variance. MMLU has only 157 questions and HumanEval 164, so a one-point difference can represent just one or two cases. Only one table reports repeated runs, and its statistical tests compare ForeCache with Full Prefill rather than with the cache-reuse baselines. Larger evaluation sets, repeated runs for the baselines, and paired comparisons would make the reported gains more convincing.

Questions for authors
  1. On AIME-24, what is ForeCache’s final-answer accuracy compared with Full Prefill? Does the reported 6.6× TTFT speedup include the cost of probing, pruning, and cache repair?

  2. How much does retaining older agent outputs help compared with keeping only the immediately preceding output?

Reviewer_GzuhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ForeCache targets prefill latency in sequential multi-agent pipelines where every agent runs the same LLM with its own system prompt over a growing shared history. It is training-free and has two parts. Consumer-conditioned saliency (CCS) appends the downstream agent's responsibility block as an isolated probe, scores history tokens by the probe's attention in the last quarter of layers, and keeps whole sentences under a budget of ⌈rN⌉ (r = 0.1) with a knapsack, always keeping the predecessor's output. Divergence-guided recomputation (DGR) reuses the decode-time KV after RoPE realignment, picks the top 20% of tokens by shallow-layer K/V deviation, and recomputes them layer by layer with a per-token early exit. On a single A800 with Llama-3.1-8B (GSM8K, a 157-question MMLU subset) and Qwen2.5-Coder-7B (HumanEval) at 2/4/6 agents, ForeCache stays within 1.3 pp of Full Prefill in six of nine cells and matches or beats KVCOMM and CacheBlend everywhere. Final-agent TTFT speedups are 1.6–3.1× on these tasks and 6.6× on AIME-24, where quality is measured only by teacher-forced agreement.

Critical strengths
  1. Conditioning the selection on the receiving agent is well motivated, and the evidence is convincing. At the same budget, CCS keeps 87% of the annotated evidence vs. 66% when scored with another agent's prompt (Swap), 61% for SnapKV and 24% for random (Fig. 3b), and replacing CCS with SnapKV costs 1.9–2.8 pp at N = 4. The Swap control is what makes this result clean.

  2. Accuracy holds up where existing cross-agent reuse methods degrade under the same graphs and prompts: at N = 6, MMLU is 69.4 vs. 64.3/60.5 and HumanEval 87.2 vs. 84.1/79.9 for KVCOMM/CacheBlend. Plain KV transplantation drops to 77.5/63.1/76.8, and DGR brings it back to 84.2/66.2/86.0.

Critical weaknesses
  1. The speedup is not attributed to the proposed mechanisms. ForeCache discards 90% of the older history while CacheBlend and Full Prefill keep all of it, and Table 2 suggests the gain tracks how much history can be pruned: at a2, where nothing can be pruned, ForeCache is slower than CacheBlend (453.9 vs. 329.3). No ablation variant has a latency number, and CacheBlend is only run at its default 15%, so it is unclear what DGR adds beyond pruning combined with an existing recomputation scheme.

  2. The baselines and the efficiency accounting are incomplete. KVCOMM appears only in the accuracy tables, RelayCaching is cited and its protocol adopted but never compared, and only per-agent median TTFT is reported (Table 2 has no units and the inference stack is not named). Probes for later consumers can overlap decoding, so their cost may fall outside the TTFT window, and end-to-end latency and the memory for stored KV and snapshots are not reported.

  3. The 6.6× headline comes from AIME-24, where there is no task accuracy, only teacher-forced agreement on up to 200 tokens, and even that drops by 4.3 pp TF@1 at a4. On the tasks with measured accuracy the speedups are 1.6–3.1×, and the GSM8K gap to Full Prefill grows to −2.1 pp at N = 6.

  4. The statistics are thin. Table 1 is a single run on small sets (one MMLU question ≈ 0.64 pp), parity rests on non-significant McNemar tests rather than an equivalence test, r was tuned on the GSM8K evaluation set, and the +4.1 pp over Full Prefill at GSM8K N = 2, where there is nothing to prune, is left unexplained.

Questions for authors
  1. Could you give a per-agent latency breakdown on AIME-24 (probe pass, knapsack, RoPE realignment, shallow pass, recomputation, first decode step) together with end-to-end latency and peak memory, and say whether the probe time falls inside the TTFT window?

  2. What are the latency and accuracy of CCS-only, DGR-only, SnapKV+DGR, and CacheBlend applied to the CCS-pruned history, and of KVCOMM in Table 2? An accuracy–TTFT curve over CacheBlend's ratio and your r would settle whether DGR adds efficiency beyond pruning.

  3. Why is RelayCaching not a baseline, given that you adopt its evaluation protocol?

  4. What does CCS prune at GSM8K N = 2, and why do ForeCache and KVCOMM beat Full Prefill there by 4.1 pp? Also, how was the golden evidence behind Fig. 3b annotated?

Reviewer_dzF4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ForeCache, a training-free multi-agent system acceleration framework. It mainly brings out 2 strategies that contribute to the acceleration: CCS to select the history based on the downstream agents and DGR to reuse KV tensors and decide the recompilation ratio of tokens. The experimental results show that ForeCache basically maintains the accuracy compared to the full KV baseline and have up to 6.6x e2e speedup in TTFT.

Critical strengths
  1. The idea of consumer-conditioned history selection is interesting, which considers the downstream agents' information explicitly, and the results is supported by the improvement recall rate over conventional producer-based methods.

  2. The paper's presentation is quite sound, the idea is not difficult to understand, while proved to be efficient under the tested scenarios.

Critical weaknesses
  1. The idea of DGR actually follows Cacheblend's selective recomputation and progressive filtering methods. The benefit of the stopping rule in DGR is not isolated through a comparison between CCS+CacheBlend and CCS+DGR.

  2. Considering the main contribution of the article is around CCS and DGR, the evaluation lacks an isolated breakdown or component-level latency ablations of CCS and DGR policy themselves, which leaves the individual overheads and true gains from CCS and DGR unclear.

Questions for authors
  1. DGR seems similar to CacheBlend’s selective recomputation. What is the essential difference between them? Can you compare CCS+DGR with CCS+CacheBlend using the same pruned context to show DGR’s benefit in accuracy and latency?

  2. Methods such as RelayCaching also targets decode-to-prefill KV reuse in multi-agent systems, and is relative to the scenario and methodology in this paper. Can you add it to the evaluation and compare against it using the same model and workflow?

  3. CCS always keeps the full output of the previous agent, which may already contain most of the necessary information. With DGR unchanged, how does CCS compare with keeping only that output or selecting recent history under the same token budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.