acceptodds
Under review as a conference paper at ICLR 2027

Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving

Abstract

LLM agents increasingly execute complex workflows involving multi-turn reasoning, tool use, and parallel agents. Efficient serving requires decisions that span two layers with complementary information: the agent harness understands workflow dependencies, context lifecycles, and execution objectives, whereas the inference engine observes request queues, KV-cache state, resource pressure, and execution capabilities. Existing interfaces do not systematically connect these views, limiting workflow-aware execution. We propose **HEAR**, a bidirectional **Harness–Engine pAiRing protocol** for agentic LLM serving. **HEAR** standardizes how the harness communicates workflow intent and execution requirements and how the engine returns runtime state, capabilities, and outcomes. By separating protocol semantics from optimization policies, **HEAR** supports diverse coordination strategies without changing workflow or model semantics. We instantiate HEAR for online cache-aware runtime coordination and workload-aware execution-mode selection for agent roles. Across four conversational and research-agent benchmarks under memory-constrained, concurrent serving, HEAR achieves a **1.61×** batch speedup and reduces median time-to-first-token by **2.23×** on SCBench. Mooncake shows that workflow intent and live engine state provide complementary benefits across load regimes. On BrowseComp-Plus and DeepResearchBench, workload-specific configurations yield **1.23×** and **2.45×** end-to-end speedups, respectively, without observed task-quality degradation. These results establish HEAR as a reusable coordination substrate for efficient agentic LLM serving. Our code is available at https://anonymous.4open.science/r/hear-3D1E/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Harness and Inference Co-Design

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KawVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HEAR, a protocol between agent harnesses and inference engines for agentic LLM serving. The protocol provides a way for the harness to communicate with the inference engine. The harness can send information such as workflow intent and execution requirements to the inference engine, and the inference engine can send runtime state, capabilities and outcomes to the harness. The paper shows two use cases of the protocol, which are online cache-aware runtime coordination and workload-aware execution-mode selection. The evaluation shows benefits of the protocol with the two use cases.

Critical strengths
  • The paper presents a protocol for communication between agent harnesses and inference engines. It is useful for improving inference performance.
  • The evaluation includes two use cases. The results show the benefits of the protocol.
Critical weaknesses
  • The description of the protocol is unclear. Table 1 shows representative information. It is unclear what the format is, and what extensions or changes are needed from harnesses and inference engines to support the protocol.
  • The paper provides two use cases. The evaluation shows results of the two use cases. While it is understandable to show two representative cases in the evaluation, it would be good to describe a list of use cases that can be enabled by the protocol.
Questions for authors
  1. Table 1 shows representative information. Does Table 1 include all possible fields in the messages of the protocols? What is the format of the protocol?

  2. What extensions or changes are needed from harnesses and inference engines to support the protocol? Do harnesses and inference engines already have the information and the API to expose the information in Table 1?

  3. Are there only two use cases? Are there any other cases that can be enabled by the protocol?

  4. Can you elaborate on the implementation details? How many lines of code is the implementation? How do you integrate with the harnesses and inference engines?

  5. Why do you run Qwen3-8B on RTX PRO 6000 for SCBench and on H100 for the Mooncake trace?

3 comments
AuthorsOpenReview ↗

Replies to Reviewer KawV: First Part

We sincerely thank you for your thoughtful comments and constructive suggestions. We have responded to each point below and submitted a revised manuscript for your consideration. We appreciate your time in reviewing our responses and the updated manuscript.

Weakness 1: The description of the protocol is unclear. Table 1 shows representative information. It is unclear what the format is, and what extensions or changes are needed from harnesses and inference engines to support the protocol.

Question 1: Table 1 shows representative information. Does Table 1 include all possible fields in the messages of the protocols? What is the format of the protocol?

Thank you for highlighting this ambiguity. Table 1 presents HEAR’s four semantic categories and representative fields, rather than an exhaustive field list. Our implementation uses a shared JSON message structure organized into the applicable description, state, control, and outcome sections. The following examples illustrate both communication directions.

Harness → Engine

{
  "request_id": "r1",
  "context_id": "c1",
  "context_version": 1,
  "description": {
    "future_use": {
      "kind": "predicted",
      "probability": 0.8
    }
  },
  "control": {
    "operation_id": "op1",
    "keepalive_preference": true
  }
}

Engine → Harness

{
  "request_id": "r1",
  "context_id": "c1",
  "context_version": 1,
  "state": {
    "observed_at": "2026-11-13T00:00:00Z",
    "source": "native_query",
    "queue": {
      "waiting_requests": 3
    }
  },
  "outcome": {
    "operation_id": "op1",
    "status": "completed"
  }
}

Here, probability describes the predicted likelihood of session continuation, while keepalive_preference expresses a cache-retention preference. The request/context identifiers bind the exchange to the relevant request and context version, and operation_id links the control to its outcome. The response reports a timestamped queue observation and completion of the adapter operation; accepted would indicate acceptance only.

To support HEAR, harness-side adapters provide request/context bindings, descriptions, and controls. Engine-side adapters normalize native runtime state into HEAR fields, translate supported controls into native scheduling or cache operations, and report outcomes. Engine-specific KV identifier resolution and priority translation remain adapter responsibilities.

Weakness 2: The paper provides two use cases. The evaluation shows results of the two use cases. While it is understandable to show two representative cases in the evaluation, it would be good to describe a list of use cases that can be enabled by the protocol.

Question 3: Are there only two use cases? Are there any other cases that can be enabled by the protocol?

Thank you for the suggestion. Given evaluation resource constraints, we focused on two representative use cases—online cache coordination and offline role-specific configuration selection—covering different decision timescales. HEAR can also enable additional use cases, including but not limited to:

Case 1: KV Prefetching. The harness indicates anticipated context reuse, while the engine reports KV residency and available capacity. A policy can request KV prefetching before subsequent invocations and receive operation-status feedback, potentially overlapping cache loading with computation.

Case 2: Concurrency Control and Request Admission. The harness provides workflow progress and request readiness, while the engine reports queue state and cache pressure. A policy can defer or admit requests based on available capacity.

Case 3: Workflow-Aware Scheduling. The harness provides workflow dependencies and readiness, while the engine reports queue and capacity state. A policy can prioritize requests that unblock downstream work while respecting waiting-time constraints.

These potential use cases depend on backend-supported state and operations.

AuthorsOpenReview ↗

Replies to Reviewer KawV: Second Part

Weakness 1: The description of the protocol is unclear. Table 1 shows representative information. It is unclear what the format is, and what extensions or changes are needed from harnesses and inference engines to support the protocol.

Question 2: What extensions or changes are needed from harnesses and inference engines to support the protocol? Do harnesses and inference engines already have the information and the API to expose the information in Table 1?

Question 4: Can you elaborate on the implementation details? How many lines of code is the implementation? How do you integrate with the harnesses and inference engines?

For the evaluated use cases, the harness already maintains role and workflow information, while vLLM exposes KV-cache events and queue metrics through existing interfaces. HEAR reuses these information sources and APIs, with the following additions.

Cache Coordination. The harness tracks prefix residency from KV-cache events, reads /metrics, and submits scheduling priorities through the completion API, with protect_s and p_return passed via vllm_xargs. Engine-side patches implement waiting protection, queued-prefix keepalive, session-aware L2 retention, and per-request feedback via kv_transfer_params.

Execution Configuration. BrowseComp-Plus and DeepResearchBench connect to preconfigured vLLM or SparseEngine services through OpenAI-compatible clients. The integration handles role-to-endpoint binding, responses, and cross-call continuation through SparseEngine's chain interfaces. The BrowseComp-Plus selector ranks development profiles and loads frozen selections, while DeepResearchBench initializes role-specific clients from configuration.

Implementation Size. The five core Mooncake integration files contain 810 Python lines (605 harness-side, 205 engine-side). The BrowseComp-Plus and DeepResearchBench client modules contain 394 and 535 lines, respectively, and the BrowseComp-Plus selector contains 166 lines. These line counts refer only to the listed integration files, not the entire codebase.

Code: https://anonymous.4open.science/r/hear-3D1E/

Question 5: Why do you run Qwen3-8B on RTX PRO 6000 for SCBench and on H100 for the Mooncake trace?

The GPU choices were based on resource availability at the time of evaluation, rather than hardware-specific requirements. We used RTX PRO 6000 for SCBench and H100 for Mooncake, with pilot tests confirming compatibility on both platforms. Importantly, all methods within each workload were evaluated on the same GPU type, ensuring fair comparisons. Our conclusions rely on within-workload comparisons rather than absolute latency differences across GPU types.

Area_Chair_sgEbOpenReview ↗

Does authors' response address your concerns?

Thank you for your review! Could you check the authors’ response and clarify whether it addresses your original concerns? Many thanks!

AC

Reviewer_Vi91OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HEAR, a bidirectional protocol between an agent harness and an LLM inference engine: the harness communicates workflow intent and execution requirements, and the engine returns runtime state, capabilities, and outcomes, organized into four message categories and four interaction distinctions. Two policies built on HEAR are evaluated: online cache-aware scheduling and retention on SCBench and a Mooncake trace, and offline role-specific execution-mode selection on BrowseComp-Plus and DeepResearchBench. The paper reports a 1.61× batch speedup and a 2.23× reduction in median time-to-first-token on SCBench, and 1.23× and 2.45× end-to-end speedups on the research benchmarks without observed quality loss.

Critical strengths
  1. A well-motivated abstraction for a real gap. The harness knows workflow dependencies and context lifecycles, while the engine observes queues, cache state, and capabilities; the paper turns this split into a bidirectional contract, and its interaction distinctions, such as preference versus requirement and acceptance versus completion, are useful design vocabulary.

  2. One contract supports policies at two timescales, with careful controls. The same message categories drive online cache-aware scheduling and retention and offline role-to-mode selection, and the resulting policies give substantial speedups net of online protocol overhead, measured with cache trajectories replayed and selections frozen before evaluation, and the paper also reports loads at which no policy dominates.

Critical weaknesses
  1. The experiments do not show that HEAR's cross-layer information produces the gains. For the cache policies, the only external baseline is first-come-first-served scheduling. Yet Cache-Aware ranks requests by engine-reported reusable prefix blocks and Guard uses waiting time (Appendix A.2), both observable to the engine, so an engine-only prefix-aware scheduler with aging is the natural control for the claim that the exchanged information is something "neither layer can exploit alone"; none is reported. For the configuration study, the largest gains come from switching readers or sub-agents to H2O-based configurations, which also change the engine, cache mechanism, and capacity (Tables 8–9). A fixed vLLM–H2O mapping already gives 2.15× on DeepResearchBench versus 2.45× for the selected mapping (Table 5), and, unlike the cache study, the paper does not report trajectory replay for these arms.

  2. HEAR's claimed cross-implementation reusability is not demonstrated. The paper's central claim is that HEAR is "a reusable, policy-independent substrate" (Section 5), but no policy is shown running across independently implemented adapters. The specification describes semantic roles "rather than packet formats" (Section 3.2), without a testable schema or versioning rules, and the PBKV and Pythia adapter integrations that Section 3.4 proposes are not demonstrated.

Questions for authors
  1. Protocol reuse: Can you show one existing policy, such as PBKV or Pythia, ported through an adapter, or a minimal schema with a conformance test, and measure protocol overhead separately?
  2. Single-layer control: How does an engine-only prefix-aware scheduler with aging compare on SCBench and Mooncake?
  3. Baseline configuration: Why does dense vLLM reach only 6.2% reuse with prefix caching enabled, and were its cache capacity and concurrency tuned?
  4. Attribution and quality: On BrowseComp-Plus the paired interval of [−5.29, 6.25] points cannot exclude a five-point loss, and the DeepResearchBench comparison covers only the 89 instances scored in every arm (Appendix B.4). How much of the speedup remains with agent trajectories held fixed, how does quality compare over the full DeepResearchBench cohort, including failed and empty reports, and how much do the efficiency results vary across repeated runs?
  5. Positioning: Section 1 says agentic serving lacks a shared control-plane abstraction, yet the runtime layer the paper cites (Zhang et al., 2026, "A Policy-Driven Runtime Layer for Agentic LLM Serving") describes itself as a third tier between framework and engine into which any agent-aware policy plugs. Is HEAR's distinct contribution its message and lifecycle semantics without an extra runtime tier, and can Section 1 be qualified accordingly?
6 comments
AuthorsOpenReview ↗

Replies to Reviewer Vi91: First Part

We sincerely thank you for your thoughtful comments and constructive suggestions. We have responded to each point below and submitted a revised manuscript for your consideration. We appreciate your time in reviewing our responses and the updated manuscript.

Weakness 1: The experiments do not show that HEAR's cross-layer information produces the gains. For the cache policies, the only external baseline is first-come-first-served scheduling. Yet Cache-Aware ranks requests by engine-reported reusable prefix blocks and Guard uses waiting time (Appendix A.2), both observable to the engine, so an engine-only prefix-aware scheduler with aging is the natural control for the claim that the exchanged information is something "neither layer can exploit alone"; none is reported. For the configuration study, the largest gains come from switching readers or sub-agents to H2O-based configurations, which also change the engine, cache mechanism, and capacity (Tables 8–9). A fixed vLLM–H2O mapping already gives 2.15× on DeepResearchBench versus 2.45× for the selected mapping (Table 5), and, unlike the cache study, the paper does not report trajectory replay for these arms.

Gains from Cross-Layer Information. We added an engine-only prefix-aware scheduler with aging, using only engine-local prefix reuse and waiting-time information. Tables R1–R2 compare this stronger single-layer baseline with HEAR's policies.

Table R1. SCBench results with the additional single-layer baseline.

ConfigurationTTFT P50 ↓TTFT P95 ↓TTFT MAX ↓Batch Completion ↓Reuse ↑
FCFS63.1 (×1.00)74.2 (×1.00)79.0 (×1.00)481 (×1.00)19.2
EPA (Engine-only)17.62 (×3.58)52.19 (×1.42)60.33 (×1.31)283.16 (×1.70)69.26
Cache-Aware + Guard-4028.3 (×2.23)51.1 (×1.45)65.0 (×1.22)299 (×1.61)66.1
Cache-Aware + Guard-604.5 (×13.9)64.0 (×1.16)80.4 (×0.98)230 (×2.09)84.7

Table R2. Mooncake Bench results with the additional single-layer baseline.

LoadConfigurationTTFT P50 ↓TTFT P95 ↓TTFT MAX ↓Session ↓Throughput ↑Reuse ↑
50%FCFS0.69 (×1.00)2.79 (×1.00)6.23 (×1.00)13.67 (×1.00)86.08 (×1.00)19.47
50%Single-layer0.59 (×1.17)2.87 (×0.97)7.25 (×0.86)13.59 (×1.01)86.17 (×1.00)19.61
50%Session-Aware + Cache-Aware + Guard0.51 (×1.35)2.32 (×1.21)6.12 (×1.02)12.70 (×1.08)86.75 (×1.01)32.14
75%FCFS10.91 (×1.00)51.86 (×1.00)56.73 (×1.00)50.24 (×1.00)116.42 (×1.00)7.98
75%Single-layer10.07 (×1.08)43.72 (×1.19)48.33 (×1.17)43.80 (×1.15)119.08 (×1.02)7.94
75%Session-Aware + Cache-Aware + Guard8.18 (×1.33)39.77 (×1.30)45.43 (×1.25)39.77 (×1.26)121.00 (×1.04)16.86
100%FCFS163.90 (×1.00)277.05 (×1.00)281.46 (×1.00)256.94 (×1.00)122.67 (×1.00)5.19
100%Single-layer95.16 (×1.72)204.28 (×1.36)215.16 (×1.31)174.96 (×1.47)123.50 (×1.01)8.23
100%Session-Aware + Cache-Aware + Guard161.67 (×1.01)267.37 (×1.04)277.66 (×1.01)243.21 (×1.06)108.92 (×0.89)12.53

Cache-Aware and Guard use engine-visible signals, while Session-Aware additionally incorporates harness-provided session-continuation and future-use intent. On SCBench, Cache-Aware + Guard improves batch completion over the single-layer baseline (230 vs. 283.16). On Mooncake, the combined policy reduces session time and improves throughput at 50% and 75% load, supporting the benefits of cross-layer coordination. At 100% load, however, the single-layer baseline performs better, indicating that these benefits depend on load conditions.

AuthorsOpenReview ↗

Replies to Reviewer Vi91: Second Part

Configuration Selection. HEAR matches harness-provided role and workload descriptions with engine-side efficiency and capacity profiles (§4.3). We acknowledge that H₂O contributes substantially to the overall speedups; the selection policy's additional benefit lies in choosing workload-appropriate configurations. Table R3 shows the six configurations evaluated during development, with selections frozen before formal evaluation.

Table R3. Development-set configuration comparison for BrowseComp-Plus (BCP) and DeepResearchBench (DRB).

Main / ReaderBCP Accuracy ↑BCP Makespan (s) ↓DRB RACE ↑DRB Makespan (min) ↓
vLLM / vLLM45.45% (10/22)19,754.780.413539.64
OmniKV / vLLM45.45% (10/22)19,009.040.429339.23
vLLM / H₂O45.45% (10/22)16,356.490.417115.87
OmniKV / H₂O40.91% (9/22)17,449.140.432714.92
vLLM / SnapKV45.45% (10/22)17,399.600.406615.27
OmniKV / SnapKV45.45% (10/22)18,873.640.413115.57

Even with H₂O fixed as the Reader backend, the preferred Main backend differs: vLLM–H₂O performs best on BCP, while OmniKV–H₂O performs best on DRB. In the full evaluation (Table 5), OmniKV–H₂O reduces DRB makespan from 2.76 to 2.42 hours relative to fixed vLLM–H₂O, whereas using OmniKV–H₂O on BCP increases wall time by 7.8% over its selected vLLM–H₂O mapping. We thus distinguish the gains from backend switching from those of workload-specific selection.

Fixed-Trajectory Replay (BCP and DRB). We conducted two fixed-trajectory replay runs across all six configurations on both benchmarks. For BCP, the replay covers 126 model requests, 295 tool operations, and 57,878 output tokens, preserving native cache behavior and tool waiting times while performing actual prefill and token-by-token decoding. Both BCP runs passed token-level consistency checks.

The selected configurations achieved the shortest completion times in both runs on their respective benchmarks. On BCP, vLLM–H₂O averaged 658.22 s versus 832.31 s for all-vLLM, a 1.264× speedup (20.92% time reduction). On DRB, OmniKV–H₂O averaged 732.00 s versus 2090.50 s, a 2.856× speedup (64.98% time reduction). These results confirm that the efficiency gains persist under fixed agent trajectories rather than arising solely from differences in generated content or execution paths.

weakness 2: HEAR's claimed cross-implementation reusability is not demonstrated. The paper's central claim is that HEAR is "a reusable, policy-independent substrate" (Section 5), but no policy is shown running across independently implemented adapters. The specification describes semantic roles "rather than packet formats" (Section 3.2), without a testable schema or versioning rules, and the PBKV and Pythia adapter integrations that Section 3.4 proposes are not demonstrated.

question 1: Protocol reuse: Can you show one existing policy, such as PBKV or Pythia, ported through an adapter, or a minimal schema with a conformance test, and measure protocol overhead separately?

Existing Policy Adaptation. Since usable official implementations of PBKV and Pythia were unavailable, we adapted KVFlow (NeurIPS 2025) [1] to HEAR. Native and HEAR-adapted KVFlow use the same backend and compatibility patches, preserving the original cache priority, eviction, and prefetching algorithms while changing only workflow metadata integration.

Adapters and Conformance Tests. The harness adapter encodes workflow identities, prefix bindings, execution distances, and update sequences into versioned messages. The engine adapter validates these messages and reconstructs native KVFlow inputs. All conformance tests passed, covering identity consistency, ordered updates, idempotent retries, and timely metadata application.

Protocol Overhead. We replayed identical message traces with preconstructed inputs over four passes, alternating Native/HEAR measurement order. Message wrapping, JSON encoding/decoding, schema and semantic validation, translation, and bookkeeping introduce a median CPU overhead of 16–35 ms per message.

Table R4. Native versus HEAR-adapted KVFlow on Mooncake replay (RTX PRO 5000, Qwen3-8B).

LoadNative makespan (s)HEAR makespan (s)Change
50%4429.334462.98+0.76%
75%5994.756014.93+0.34%
100%7700.807711.87+0.14%

As shown in Table R4, end-to-end makespan increases by 0.14%–0.76%, demonstrating existing policy reuse through HEAR adapters with modest integration overhead.

[1] Zaifeng Pan et al. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows. NeurIPS, 2025.

AuthorsOpenReview ↗

Replies to Reviewer Vi91: Third Part

Question 2: Single-layer control: How does an engine-only prefix-aware scheduler with aging compare on SCBench and Mooncake?

We added an engine-only prefix-aware scheduler with aging (Single-layer). At request admission, the engine checks reusable GPU prefixes and ready CPU prefixes. Requests with at least 50% reusable input are prioritized by reusable prefix length, while requests waiting at least 40 seconds are promoted during scheduling. Unlike HEAR, which uses engine feedback to assign priorities through the harness, Single-layer operates entirely within the engine, without harness-provided priorities or future-use intent.

We retain the hardware used in the main experiments: one RTX PRO 6000 for SCBench and one H100 for Mooncake.

Results (Tables R1–R2). On SCBench, Cache-Aware + Guard-60 reduces batch completion time by 23**%** relative to Single-layer. On Mooncake, Session-Aware + Cache-Aware + Guard achieves lower session time and higher throughput at 50% and 75% load. At 100% load, however, its throughput drops to 108.92 turns/min, below Single-layer (123.50) and FCFS (122.67). The original policy does not adapt retention to congestion; under overload, adverse interactions among future-context retention, current-request scheduling, and Guard's waiting protection may offset reuse benefits.

Question 3: Baseline configuration: Why does dense vLLM reach only 6.2% reuse with prefix caching enabled, and were its cache capacity and concurrency tuned?

The reported 6.2% is the cumulative input-token reuse ratio over successful Reader calls in the all-vLLM BrowseComp-Plus run (9.0M / 144.4M tokens). Prefix caching reuses matching prefixes that remain available. Document selection accounted for 53.4% of Reader input tokens but achieved only 3.13% reuse; subsequent research rounds achieved 9.65%. These phase-level rates explain the low aggregate ratio.

The H₂O Reader’s 80.3% reflects its use of sparse KV with a persistent chain cache. When a chain is restored, only the new suffix requires prefill. Reused tokens therefore measure the logical history covered by the restored state, rather than the proportion of full dense KV physically retained.

We used fixed engine configurations with prefix caching enabled and a common harness limit of 12 concurrent Reader HTTP requests, without dedicated tuning of cache capacity or concurrency.

Reviewer_BdazOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HEAR defines an interface through which an agent harness communicates workflow intent and execution requirements to an inference engine, and receives cache state, capabilities, and outcomes. The paper demonstrates cache-aware request ordering with waiting protection, session-aware cache retention, and role-specific inference modes selected offline. It reports a 1.61× batch speedup on SCBench and 1.23× and 2.45× end-to-end speedups on two research-agent workloads. These are results for policies and configurations implemented through HEAR; the protocol itself does not prescribe an optimization algorithm.

Critical strengths
  1. The paper addresses a concrete information gap. The harness knows workflow progress, while the engine knows which prefixes remain available and how much pressure the system is under. The distinction between an observed state and a guarantee, and between an accepted operation and a completed one, is useful for avoiding incorrect coordination.

  2. The evaluation exposes costs as well as benefits. On SCBench, prioritizing cached requests can worsen maximum TTFT, and waiting protection changes that tradeoff. Mooncake includes an overload case where combining retention and ordering performs worse. These results make the study more informative than a collection of best-case speedups.

Critical weaknesses
  1. The experiments do not establish the main claimed contribution: a reusable protocol rather than a useful custom integration. Cache-aware ordering can be implemented using cache events and priorities, while the role-specific configuration experiment loads a mapping chosen offline. Neither result shows that HEAR makes the same policy portable across independent harness–engine implementations. A protocol does not need to outperform a custom interface in runtime, but it does need evidence that its contract is sufficiently concrete and reusable. The paper provides much stronger evidence for the policies than for that claim. It would be much better if a common policy exists that works for different settings.

  2. The configuration speedups and quality claims need narrower attribution. Tables 8–9 change execution backends, cache mechanisms, and capacity settings alongside mode selection. This can be a valid deployment comparison, but does not identify a benefit of the interface. Meanwhile, the selected BrowseComp-Plus configuration changes accuracy by just +0.48 percentage points, with a paired interval of −5.29 to +6.25. That supports “no degradation detected,” not established quality preservation. At full Mooncake load, the combined policy is also slower than FCFS. The evaluation therefore does not yet demonstrate a generally reliable way to turn the exchanged information into an advantageous execution decision.

Questions for authors
  1. Reusable implementation. For the SCBench policy in Appendix A.2, which parts consume only HEAR-defined fields and which depend directly on the chosen engine’s KV events, block identifiers, or priority implementation? Please demonstrate that the same policy code can operate on another harness–engine pairing by changing only the adapters, and provide the shared message definitions and checks used to validate the adapter behavior. This directly tests the protocol claim; a larger speedup on the existing pairing would not.

  2. Overload behavior. At 100% Mooncake load, Table 4 gives the combined policy 108.9 turns/min, below FCFS at 122.7. Does the implementation change retention or scheduling when queue pressure rises? Please give the exact rule and its activations in that run. If there is no such rule, please explain which interaction produces the slowdown and limit the claim about conditioning future-use intent on live state accordingly.

  3. Configuration selection and quality. What development tasks and explicit quality thresholds select the mappings in Appendix A.1, and do those tasks overlap with formal evaluation? Please provide the six development results and any change made during verification. For BrowseComp-Plus, what loss in accuracy would count as unacceptable, given the −5.29 to +6.25 paired interval in Table 17? These details determine whether the selector is reproducible and whether its quality constraint is actually supported.

5 comments
AuthorsOpenReview ↗

Replies to Reviewer Bdaz: First Part

We sincerely thank you for your thoughtful comments and constructive suggestions. We have responded to each point below and submitted a revised manuscript for your consideration. We appreciate your time in reviewing our responses and the updated manuscript.

Weakness 1: The experiments do not establish the main claimed contribution: a reusable protocol rather than a useful custom integration. Cache-aware ordering can be implemented using cache events and priorities, while the role-specific configuration experiment loads a mapping chosen offline. Neither result shows that HEAR makes the same policy portable across independent harness–engine implementations. A protocol does not need to outperform a custom interface in runtime, but it does need evidence that its contract is sufficiently concrete and reusable. The paper provides much stronger evidence for the policies than for that claim. It would be much better if a common policy exists that works for different settings.

Thank you for your suggestion. As shown in Table R1, we evaluated the same Cache-Aware and Guard policies on LangGraph/vLLM and pi/SGLang, keeping the policy code and HEAR message definitions unchanged; only the implementation-specific adapters differed. Guard-60 improved batch completion by 1.82× and 1.56× over FCFS, respectively. We also defined a common message schema and conformance checks covering context/version consistency, prefix ranges, stale/duplicate messages, unsupported Guard, and acceptance versus completion. All 22,524 formal messages passed validation. Additionally, 300 native-queue tests per engine and live execution probes verified priority translation, protected FIFO ordering, and L2 keepalive. These results demonstrate that HEAR's protocol contract and policies can be reused across distinct harness–engine implementations.

Table R1. Cross-implementation evaluation of HEAR policies on SCBench. The same policy code and HEAR message definitions are used across both harness–engine pairings, with only implementation-specific adapters changed. Configuration: 2 × 48-GiB RTX 4090 GPUs, Qwen3-8B BF16, TP=2, L1=505,152 tokens, and L2=96 GiB total. Each condition is evaluated once. TTFT and batch completion are reported in seconds; speedups are relative to FCFS.

Harness / EnginePolicyTTFT P50 ↓TTFT P95 ↓TTFT Max ↓Batch Completion ↓Reuse ↑
LangGraph / vLLMFCFS68.178.080.2503.6 (1.00×)18.5%
Cache-Aware3.988.2118.3218.4 (2.31×)93.5%
+ Guard-4048.861.679.7380.1 (1.33×)56.2%
+ Guard-609.370.188.7277.3 (1.82×)81.1%
Pi / SGLangFCFS64.772.376.9508.4 (1.00×)15.6%
Cache-Aware3.596.1124.7233.3 (2.18×)94.6%
+ Guard-4052.264.679.2434.1 (1.17×)47.8%
+ Guard-6019.367.887.8325.0 (1.56×)71.8%
AuthorsOpenReview ↗

Replies to Reviewer Bdaz: Second Part

Weakness 2: The configuration speedups and quality claims need narrower attribution. Tables 8–9 change execution backends, cache mechanisms, and capacity settings alongside mode selection. This can be a valid deployment comparison, but does not identify a benefit of the interface. Meanwhile, the selected BrowseComp-Plus configuration changes accuracy by just +0.48 percentage points, with a paired interval of −5.29 to +6.25. That supports “no degradation detected,” not established quality preservation. At full Mooncake load, the combined policy is also slower than FCFS. The evaluation therefore does not yet demonstrate a generally reliable way to turn the exchanged information into an advantageous execution decision.

Speedup attribution. Tables 8–9 compare complete execution modes, including their backends, cache mechanisms, and capacity settings. HEAR enables role-specific configuration through a shared interface for exchanging harness requirements and engine capabilities. The reported speedups therefore reflect the end-to-end benefits of the selected configurations, rather than the isolated performance contribution of the interface itself. We will clarify this attribution.

Quality. Different inference engines and KV-cache strategies can alter intermediate outputs and downstream agent trajectories, as confirmed by our trajectory inspection. Among 208 BrowseComp-Plus questions, 19 changed from correct to incorrect and 20 in the opposite direction, yielding a +0.48 percentage-point accuracy difference. The paired interval [−5.29, +6.25] percentage points does not exclude a meaningful loss. We will therefore revise the claim from “quality preservation” to “no degradation detected.”

Mooncake at full load. At 100% Mooncake load, the combined policy underperforms FCFS (Table 4). Under queue pressure, retention remains active while Guard retains its fixed 40-second threshold. Session-Aware favors future CPU L2 reuse, whereas Cache-Aware treats GPU and CPU cache hits equally without accounting for restoration costs. Adding Session-Aware to Cache-Aware + Guard coincides with more preemptions and CPU-to-GPU transfers, while the combined policy also shows lower GPU utilization than FCFS. These observations suggest that retention and restoration overhead can outweigh reuse benefits under heavy load. We will clarify that the gains depend on the configuration, policy, and load.

AuthorsOpenReview ↗

Replies to Reviewer Bdaz: Third Part

Question1: Reusable implementation. For the SCBench policy in Appendix A.2, which parts consume only HEAR-defined fields and which depend directly on the chosen engine’s KV events, block identifiers, or priority implementation? Please demonstrate that the same policy code can operate on another harness–engine pairing by changing only the adapters, and provide the shared message definitions and checks used to validate the adapter behavior. This directly tests the protocol claim; a larger speedup on the existing pairing would not.

Policy–adapter boundary. The shared policy uses HEAR-defined fields—request readiness, reusable-prefix lengths, capacity/occupancy, and locally tracked request age—for cache ranking, admission, and waiting protection. It also determines L2 keepalive order. Engine-specific operations, including native KV event decoding, block/radix identifier resolution, priority translation, queue modification, and L2 recency updates, remain within adapters. The policy therefore operates on normalized HEAR information without accessing native cache identifiers or queue objects.

Cross-implementation reuse. We evaluated the same Cache-Aware and Guard policies on LangGraph/vLLM and pi/SGLang, with unchanged policy code and HEAR message definitions. Only the harness and engine adapters differed. Table R1 reports all four configurations under the same hardware setup and SCBench workload (60 conversations, 264 turns), demonstrating policy reuse across distinct harness–engine implementations.

Shared message definitions and validation. The following schema excerpt specifies field types based on Table 1's message categories and Table 2's interaction semantics:

{
  "request_id": "string",
  "context_id": "string",
  "context_version": "integer",
  "description": {
    "ready": "boolean",
    "prompt_tokens": "integer"
  },
  "state": {
    "gpu_prefix_tokens": "integer",
    "reusable_prefix_tokens": "integer",
    "capacity_tokens": "integer",
    "occupancy": "[0, 1]"
  },
  "control": {
    "operation_id": "string",
    "reuse_score": "integer",
    "guard_ms": "null | 40000 | 60000",
    "keepalive_preference": "boolean"
  },
  "outcome": {
    "operation_id": "string",
    "status": "accepted | completed | rejected | unsupported | failed"
  }
}

Conformance checks covered context/version consistency, valid prefix ranges, stale/duplicate messages, unsupported Guard, and acceptance versus completion. All 22,524 formal messages passed validation. Additionally, 300 native-queue tests per engine and live execution probes verified priority translation, protected FIFO ordering, and L2 keepalive behavior.

Question2: Overload behavior. At 100% Mooncake load, Table 4 gives the combined policy 108.9 turns/min, below FCFS at 122.7. Does the implementation change retention or scheduling when queue pressure rises? Please give the exact rule and its activations in that run. If there is no such rule, please explain which interaction produces the slowdown and limit the claim about conditioning future-use intent on live state accordingly.

Overload behavior and policy activations. The implementation has no queue-pressure-triggered retention suspension or scheduling switch, so no corresponding congestion-gate activations exist. At 100% Mooncake load, Cache-Aware prioritized reusable GPU/CPU prefixes for 560/2,824 requests; Session-Aware maintained CPU L2 contexts based on return probability and idle time, averaging 156 LRU touches per maintenance cycle; and Guard promoted 2,038/2,824 requests after ≥40 seconds of waiting since engine arrival.

Full-load slowdown. As shown in Table 4, the combined policy increases reuse from 5.2% to 12.5% over FCFS, while throughput decreases from 122.7 to 108.9 turns/min. Under congestion, Guard's age-based priority limits cache-preferred ordering, while retention and restoration costs remain. Cache-Aware also weights GPU and CPU hits equally without accounting for restoration costs. Synchronous maintenance and waits for GPU KV blocks with pending writebacks delay compute submission, creating GPU idle gaps. Consistently, GPU utilization drops from 98.22% to 88.00%, and mean generation time for the same 1,291 requests rises from 18.62 to 21.17 seconds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.