acceptodds
Withdrawn from ICLR 2027

Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scaling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models are increasingly deployed as complex agentic systems. While prior work has focused on model- and system-level scaling, algorithm- and task-level scaling remain largely unaddressed. At the algorithm level, test-time scaling enhances reasoning but introduces cross-path redundancy: branches expanded from the same state re-generate overlapping tokens, yet existing engines execute them as independent requests. At the task level, agents exhibit highly heterogeneous runtime characteristics, yet current inference engines schedule all requests uniformly without awareness of agent roles. We propose Hive, a multi-agent inference infrastructure that enables algorithm- and task-level scaling. Hive provides a description front-end that captures per-agent behavior and supports test-time scaling algorithms. Its back-end introduces Logits Cache for algorithm-level scaling, which reuses intermediate logits across redundant sampling paths to mitigate cross-path redundancy, and Agent-aware Resource Scheduling for task-level scaling, which prioritizes KV-cache retention according to agent contributions. Experiments show that Logits Cache achieves an average speedup of –, and Agent-aware Resource Scheduling reduces the hotspot miss rate by –.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
infrastructure, inference, multi-agent, test-time scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Y6LbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Hive, an inference infrastructure for multi-agent workloads. It introduces Logits Cache to reuse decoding results across repeated reasoning branches, and Agent-Aware Resource Allocation to improve KV-cache management based on agent runtime characteristics. Experiments report improvements in resampling throughput and cache hit rates.

Critical strengths

The paper studies inference-efficiency problems in branching reasoning and multi-agent systems. Logits reuse is an interesting direction for reducing redundant computation.

Critical weaknesses
  1. Unclear problem formulation. The proposed distinction between algorithm-level and task-level scaling is not well justified. The paper essentially studies redundant decoding and KV-cache eviction, rather than demonstrating new scaling capabilities at either level.
  2. Limited methodological innovation. Step-wise Logits Cache mainly reuses previously computed logits, while Hotspot Sampling relies on a heuristic uncertainty score. Agent-Aware Resource Allocation uses a heuristic priority function based on runtime statistics. The technical novelty over existing caching, resampling, and resource allocation methods is unclear.
  3. Insufficient experimental validation. The scale of experiments is very small, and the justification for the tradeoff between search quality and diversity is not sufficiently discussed for Hotspot Sampling.
Questions for authors

Please refer to the weakness part. My main concern is the methodology design part. The technical novelty over existing caching, resampling, and resource allocation methods is unclear.

Reviewer_kRyQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Hive, an inference infrastructure for algorithm-level and task-level scaling of LLM agents. Its frontend separates agent behavior, represented as finite-state machines, from test-time scaling algorithms implemented through supervisors. The backend introduces two mechanisms: Logits Cache, which reuses cached logits when expanding previously visited states, and agent-aware KV-cache retention, which assigns priorities using runtime statistics and shared-cache interactions. Logits Cache supports step-wise resampling and an approximate hotspot policy that copies tokens at selected stable positions.

Critical strengths

The paper identifies redundant decoding computation caused by repeated visits to the same state during search and proposes reusing cached logits to exploit opportunities for computational reuse that conventional prefix KV caching does not fully capture.

Critical weaknesses
  1. High memory overhead: Caching full-vocabulary logits at every decoding position causes memory requirements to grow with sequence length and the number of retained branches.
  2. Potential bandwidth bottlenecks: GPU–CPU logits transfers may limit scalability, particularly under high concurrency.
  3. Limited empirical validation: Evaluation is restricted to fixed 500-token outputs, leaving scalability in long-sequence and multi-branch settings, as well as commercial viability, unverified.
Questions for authors
  1. Since the Logits Cache is stored in host memory, could the authors quantify its data transfer bandwidth requirements and memory footprint?
Reviewer_jwYXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hive. Hive supports branching reasoning and multi-agent workloads through two optimizations: caching logits for repeated expansions of the same state, and prioritizing KV-cache retention using agent runtime profiles. Logits replay either resamples each position until divergence or preserves tokens outside selected uncertainty hotspots. These policies achieve reported average resampling speedups of 1.11× and 2.51×. A separate five-role workflow experiment shows lower hotspot cache misses under agent-aware allocation. The system ideas are plausible, but the evaluation mainly establishes component-level gains.

Critical strengths
  • Logits caching addresses redundant decode work that prefix KV caching does not eliminate. Step-wise replay has a clear correctness argument because ordinary execution resumes when sampling diverges from the cached continuation.
  • Separating agent behavior from search supervision is a useful abstraction. It exposes agent identity and branching structure to the runtime without placing cache-management logic in the application.
  • The logits-cache baseline preserves the same prefix-cache behavior, helping isolate the additional cache's benefit. The paper also checks answer correctness for the approximate hotspot policy.
Critical weaknesses
  1. The largest speedup comes from changing the sampling process, but its effect on search is unclear. Hotspot replay copies non-hotspot tokens, potentially making branches more correlated. Similar individual-output accuracy does not establish preserved exploration: repeated branches could remain individually accurate while contributing fewer distinct solutions. Moreover, the correctness test uses 100 solvable 24 Game instances at roughly 96% accuracy with Qwen3.8-27B, while speed is measured with DeepSeek-V3.2 (Section 6.2). The quality–efficiency tradeoff therefore remains unresolved.

  2. Resampling speed does not establish end-to-end acceleration. Section 6.2 fixes outputs at 500 tokens and uses prompts averaging 64.52 tokens. The evaluation does not establish the frequency of useful revisits or the amortized cost of cache construction, storage, and transfer in a complete search. These factors determine whether faster replay translates into more solved tasks per unit time. The claim of greater gains on long agent contexts needs direct evidence from naturally terminating workloads.

  3. The allocation experiment leaves the value of the proposed score uncertain. At round 21, better hotspot retention accompanies a non-hotspot hit-rate drop from 0.414 to 0.311 (Figure 5), without a reported workflow-time comparison. Section 6.3 also evaluates one fixed invocation structure against LRU only, despite motivating runtime adaptation and discussing KVFlow and TokenCake in Section 2. It remains unclear whether the utility formulation improves overall execution or outperforms simpler reuse-based policies.

Questions for authors
  1. With the same model and search algorithm, does hotspot replay improve task success at a fixed wall-clock budget while retaining useful branch diversity? How often does it repeat earlier solutions or errors?
  2. What is the end-to-end gain including initial generation and cache overhead, and how does it vary with revisit frequency and context length?
  3. Does agent-aware allocation reduce total recomputation and workflow time compared with a simple reuse-based policy? Separate intrinsic-only and collaborative-only variants would help show which component contributes the improvement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.