- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
UMAO addresses multi-round agent serving on unified-memory devices by combining persistent prefix-cache management with cache-aware admission. Arena maintains recoverable history through bounded host staging and external storage, while CAA uses the resulting prefix state to estimate work and order requests. Implemented as an SGLang HiCache plugin, the system is evaluated on DGX Spark with two models and agent traces, reporting improved prefix reuse and latency.
Critical strengths
The paper identifies an interesting limitation of KV offloading on unified-memory devices: CPU and GPU share one physical memory pool, so moving cached history between them does not add capacity. This makes allocating memory between active inference and retained history a relevant systems problem, particularly for multi-round agents whose contexts continually grow. The paper brings useful attention to this constraint in local agent serving.
Critical weaknesses
- The mechanism behind the recoverability problem is insufficiently explained. Section 3.2 (p. 4) shows that high page-ACK coverage does not translate into high continuous-prefix coverage, but does not identify which cache-lifecycle events cause the loss of recoverability under the evaluated HiCache policies, especially Write Through. Identifying the main blocking events and connecting them to Arena’s fixes would clarify why the proposed lifecycle design is needed.
- The admission contribution is not clearly distinguished from prior work. Although UMAO provides a shared recoverable-prefix summary, the paper should explain what additional scheduling capability it enables and how it differs from adapting these existing approaches to the same setting.
Questions for authors
- What specifically prevents continuous-prefix recovery in the HiCache baselines, especially Write Through? Please give a representative failure sequence and explain which Arena mechanism addresses its cause.
- What additional scheduling capability does CAA provide over the related methods? A mechanism-level comparison and an experiment targeting that difference would help establish the contribution.
3 comments
Dear Reviewer,
The authors have responded to your review. Could you please take a moment to review their responses and engage in the discussion?
Thank you for your time and valuable contributions.
Kind regards, AC
Dear Reviewer fVqG,
Thank you again for your thoughtful review. We have provided detailed responses and additional experimental results addressing your concerns above. As the discussion period progresses, we would greatly appreciate it if you could take a moment to review our response and let us know whether it resolves your concerns or if any questions remain. We would be happy to provide further clarification. Thank you very much for your time.
We thank the reviewer for the questions on the cause of continuous-prefix recovery failures under Write Through and on CAA's additional scheduling capability. Our responses follow.
We added lifecycle instrumentation to the same Write Through workload and observed that the recovery range was not updated to reflect the effective prefix match after historical cache-path nodes were evicted.
One representative case is shown below.
All 351/351 historical KV pages and the endpoint state for this request had received storage ACKs. However, the device and host copies of eight historical nodes were successively evicted and the nodes were unlinked from the cache path, reducing the matched prefix from 22,464 to 8,192 tokens. The recovery query did not cover the newly unmatched range. As a result, 14,272 tokens were recomputed, corresponding to 63.5% of the persisted history.
UMA amplifies this gap: the peak working set of this workload is 2.12× the shared KV budget. Device and host share one physical pool, so increasing host capacity under a fixed budget reduces device capacity, and moving data between tiers cannot eliminate capacity pressure. Continuous-prefix recoverability after history eviction must therefore be maintained independently of local residency; otherwise, shared-capacity competition leads to repeated prefill across rounds.
Arena's mechanisms address these requirements at different stages: Shared-capacity repartitioning restricts host to bounded transfer staging and places long-term history in storage. Round-aligned incremental publication identifies appended history and retries incomplete pages that remain accessible through the current cache path and whose token snapshots remain valid. Compound commit combines ACKs for continuous KV pages and the required endpoint state to determine a valid recovery boundary. When admitting a request for recovery, Arena rematches the local prefix and updates the recovery range, then uses the hierarchy-aware continuous-prefix recovery pipeline to accelerate the transfer of persisted prefixes into the inference path.
Its added capability is to translate the recoverable history state maintained by Arena into request-level recovery and recomputation costs, allowing admission decisions to distinguish differences in service work that the currently matched prefix length does not express.
Specifically, compared with our LPM baseline, which orders requests by the currently matched prefix length, CAA also uses device/host hits, the committed continuous prefix in storage, and remaining prefill work to estimate recovery and recomputation costs. The ablation study in Section 5.3 shows that, under the same Arena configuration, full CAA reduces mean TTFT and mean session JCT by 9.6% and 8.5%, respectively, and P99 TTFT and P99 session JCT by 5.7% and 18.6%, relative to FCFS. Compared with Arena+LPM, it increases mean TTFT by 6.3% while reducing P99 TTFT by 65.1%, illustrating the trade-off between mean response latency and tail response latency.
To quantify the incremental benefit of work-based ordering beyond the other admission mechanisms, we hold Arena, the age fence, and recovery-ready handoff fixed for GPT-OSS-20B, Hermes C8, replacing only the ordering criterion with LPM.
The two variants have similar overall prefix reuse rates. Relative to the LPM ordering variant, work-based ordering reduces mean/P99 TTFT by 9.1%/20.1% and mean/P99 session JCT by 6.9%/9.2%. This shows that the ordering criterion provides an additional benefit when the accompanying admission mechanisms are held fixed.