- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents HEAR, a protocol between agent harnesses and inference engines for agentic LLM serving. The protocol provides a way for the harness to communicate with the inference engine. The harness can send information such as workflow intent and execution requirements to the inference engine, and the inference engine can send runtime state, capabilities and outcomes to the harness. The paper shows two use cases of the protocol, which are online cache-aware runtime coordination and workload-aware execution-mode selection. The evaluation shows benefits of the protocol with the two use cases.
Critical strengths
- The paper presents a protocol for communication between agent harnesses and inference engines. It is useful for improving inference performance.
- The evaluation includes two use cases. The results show the benefits of the protocol.
Critical weaknesses
- The description of the protocol is unclear. Table 1 shows representative information. It is unclear what the format is, and what extensions or changes are needed from harnesses and inference engines to support the protocol.
- The paper provides two use cases. The evaluation shows results of the two use cases. While it is understandable to show two representative cases in the evaluation, it would be good to describe a list of use cases that can be enabled by the protocol.
Questions for authors
-
Table 1 shows representative information. Does Table 1 include all possible fields in the messages of the protocols? What is the format of the protocol?
-
What extensions or changes are needed from harnesses and inference engines to support the protocol? Do harnesses and inference engines already have the information and the API to expose the information in Table 1?
-
Are there only two use cases? Are there any other cases that can be enabled by the protocol?
-
Can you elaborate on the implementation details? How many lines of code is the implementation? How do you integrate with the harnesses and inference engines?
-
Why do you run Qwen3-8B on RTX PRO 6000 for SCBench and on H100 for the Mooncake trace?
3 comments
Replies to Reviewer KawV: Second Part
Weakness 1: The description of the protocol is unclear. Table 1 shows representative information. It is unclear what the format is, and what extensions or changes are needed from harnesses and inference engines to support the protocol.
Question 2: What extensions or changes are needed from harnesses and inference engines to support the protocol? Do harnesses and inference engines already have the information and the API to expose the information in Table 1?
Question 4: Can you elaborate on the implementation details? How many lines of code is the implementation? How do you integrate with the harnesses and inference engines?
For the evaluated use cases, the harness already maintains role and workflow information, while vLLM exposes KV-cache events and queue metrics through existing interfaces. HEAR reuses these information sources and APIs, with the following additions.
Cache Coordination. The harness tracks prefix residency from KV-cache events, reads /metrics, and submits scheduling priorities through the completion API, with protect_s and p_return passed via vllm_xargs. Engine-side patches implement waiting protection, queued-prefix keepalive, session-aware L2 retention, and per-request feedback via kv_transfer_params.
Execution Configuration. BrowseComp-Plus and DeepResearchBench connect to preconfigured vLLM or SparseEngine services through OpenAI-compatible clients. The integration handles role-to-endpoint binding, responses, and cross-call continuation through SparseEngine's chain interfaces. The BrowseComp-Plus selector ranks development profiles and loads frozen selections, while DeepResearchBench initializes role-specific clients from configuration.
Implementation Size. The five core Mooncake integration files contain 810 Python lines (605 harness-side, 205 engine-side). The BrowseComp-Plus and DeepResearchBench client modules contain 394 and 535 lines, respectively, and the BrowseComp-Plus selector contains 166 lines. These line counts refer only to the listed integration files, not the entire codebase.
Code: https://anonymous.4open.science/r/hear-3D1E/
Question 5: Why do you run Qwen3-8B on RTX PRO 6000 for SCBench and on H100 for the Mooncake trace?
The GPU choices were based on resource availability at the time of evaluation, rather than hardware-specific requirements. We used RTX PRO 6000 for SCBench and H100 for Mooncake, with pilot tests confirming compatibility on both platforms. Importantly, all methods within each workload were evaluated on the same GPU type, ensuring fair comparisons. Our conclusions rely on within-workload comparisons rather than absolute latency differences across GPU types.
Does authors' response address your concerns?
Thank you for your review! Could you check the authors’ response and clarify whether it addresses your original concerns? Many thanks!
AC
Replies to Reviewer KawV: First Part
We sincerely thank you for your thoughtful comments and constructive suggestions. We have responded to each point below and submitted a revised manuscript for your consideration. We appreciate your time in reviewing our responses and the updated manuscript.
Thank you for highlighting this ambiguity. Table 1 presents HEAR’s four semantic categories and representative fields, rather than an exhaustive field list. Our implementation uses a shared JSON message structure organized into the applicable
description,state,control, andoutcomesections. The following examples illustrate both communication directions.Harness → Engine
Engine → Harness
Here,
probabilitydescribes the predicted likelihood of session continuation, whilekeepalive_preferenceexpresses a cache-retention preference. The request/context identifiers bind the exchange to the relevant request and context version, andoperation_idlinks the control to its outcome. The response reports a timestamped queue observation and completion of the adapter operation;acceptedwould indicate acceptance only.To support HEAR, harness-side adapters provide request/context bindings, descriptions, and controls. Engine-side adapters normalize native runtime state into HEAR fields, translate supported controls into native scheduling or cache operations, and report outcomes. Engine-specific KV identifier resolution and priority translation remain adapter responsibilities.
Thank you for the suggestion. Given evaluation resource constraints, we focused on two representative use cases—online cache coordination and offline role-specific configuration selection—covering different decision timescales. HEAR can also enable additional use cases, including but not limited to:
Case 1: KV Prefetching. The harness indicates anticipated context reuse, while the engine reports KV residency and available capacity. A policy can request KV prefetching before subsequent invocations and receive operation-status feedback, potentially overlapping cache loading with computation.
Case 2: Concurrency Control and Request Admission. The harness provides workflow progress and request readiness, while the engine reports queue state and cache pressure. A policy can defer or admit requests based on available capacity.
Case 3: Workflow-Aware Scheduling. The harness provides workflow dependencies and readiness, while the engine reports queue and capacity state. A policy can prioritize requests that unblock downstream work while respecting waiting-time constraints.
These potential use cases depend on backend-supported state and operations.