acceptodds
Under review as a conference paper at ICLR 2027

UMAO: Coupling Cache Lifecycle and Admission for Agents on Unified Memory

Abstract

Multi-round agents continually append context, making cross-round reuse of computed prefixes essential for avoiding redundant prefill. On devices with a unified memory architecture (UMA), such as the NVIDIA DGX Spark, device and host share one physical memory pool. Expanding the host cache therefore does not increase total capacity; instead, it reduces the allocatable device KV, while retained histories, active KV, and cross-tier transfers compete for limited capacity. The system must maintain history usable in later rounds under fixed shared capacity and coordinate recovery and computation for competing requests. We propose unified memory agent offloading (UMAO), a framework that connects cache management and request admission through a shared recoverable-prefix state. Unified cache arena offloads long-term history to storage outside the shared pool, restricts host to bounded transfer staging, and commits and recovers history in continuous-prefix units. Cache-aware agent admission reads the prefix summary generated at the same scheduling boundary, jointly estimates direct-hit, cross-tier recovery, and recomputation work, and uses an age fence to limit tail latency. We implement UMAO as an SGLang HiCache plugin and evaluate it on DGX Spark with multiple real agent workloads. UMAO raises prefix reuse to 87.5%–90.3%, approaching the Ideal prefix-reuse upper bound across all workloads. Compared with existing HiCache write policies and longest-prefix match (LPM) admission, UMAO achieves up to / speedups in mean/P99 time to first token (TTFT) and / speedups in mean/P99 session completion time (JCT), respectively.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM serving, KV cache, multi-turn agents, hierarchical caching, cache offloading, prefix reuse, request scheduling, unified memory

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_fVqGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UMAO addresses multi-round agent serving on unified-memory devices by combining persistent prefix-cache management with cache-aware admission. Arena maintains recoverable history through bounded host staging and external storage, while CAA uses the resulting prefix state to estimate work and order requests. Implemented as an SGLang HiCache plugin, the system is evaluated on DGX Spark with two models and agent traces, reporting improved prefix reuse and latency.

Critical strengths

The paper identifies an interesting limitation of KV offloading on unified-memory devices: CPU and GPU share one physical memory pool, so moving cached history between them does not add capacity. This makes allocating memory between active inference and retained history a relevant systems problem, particularly for multi-round agents whose contexts continually grow. The paper brings useful attention to this constraint in local agent serving.

Critical weaknesses
  1. The mechanism behind the recoverability problem is insufficiently explained. Section 3.2 (p. 4) shows that high page-ACK coverage does not translate into high continuous-prefix coverage, but does not identify which cache-lifecycle events cause the loss of recoverability under the evaluated HiCache policies, especially Write Through. Identifying the main blocking events and connecting them to Arena’s fixes would clarify why the proposed lifecycle design is needed.
  2. The admission contribution is not clearly distinguished from prior work. Although UMAO provides a shared recoverable-prefix summary, the paper should explain what additional scheduling capability it enables and how it differs from adapting these existing approaches to the same setting.
Questions for authors
  1. What specifically prevents continuous-prefix recovery in the HiCache baselines, especially Write Through? Please give a representative failure sequence and explain which Arena mechanism addresses its cause.
  2. What additional scheduling capability does CAA provide over the related methods? A mechanism-level comparison and an experiment targeting that difference would help establish the contribution.
3 comments

We thank the reviewer for the questions on the cause of continuous-prefix recovery failures under Write Through and on CAA's additional scheduling capability. Our responses follow.

W1 & Q1: What prevents continuous-prefix recovery in HiCache, especially Write Through, and which Arena mechanisms address the cause?

We added lifecycle instrumentation to the same Write Through workload and observed that the recovery range was not updated to reflect the effective prefix match after historical cache-path nodes were evicted.

One representative case is shown below.

Historical KV page ACKsLocal prefix match before endpoint evictionEffective prefix match after evictionHistorical tokens recomputed
351/35122,464 tokens8,192 tokens14,272 tokens

All 351/351 historical KV pages and the endpoint state for this request had received storage ACKs. However, the device and host copies of eight historical nodes were successively evicted and the nodes were unlinked from the cache path, reducing the matched prefix from 22,464 to 8,192 tokens. The recovery query did not cover the newly unmatched range. As a result, 14,272 tokens were recomputed, corresponding to 63.5% of the persisted history.

UMA amplifies this gap: the peak working set of this workload is 2.12× the shared KV budget. Device and host share one physical pool, so increasing host capacity under a fixed budget reduces device capacity, and moving data between tiers cannot eliminate capacity pressure. Continuous-prefix recoverability after history eviction must therefore be maintained independently of local residency; otherwise, shared-capacity competition leads to repeated prefill across rounds.

Arena's mechanisms address these requirements at different stages: Shared-capacity repartitioning restricts host to bounded transfer staging and places long-term history in storage. Round-aligned incremental publication identifies appended history and retries incomplete pages that remain accessible through the current cache path and whose token snapshots remain valid. Compound commit combines ACKs for continuous KV pages and the required endpoint state to determine a valid recovery boundary. When admitting a request for recovery, Arena rematches the local prefix and updates the recovery range, then uses the hierarchy-aware continuous-prefix recovery pipeline to accelerate the transfer of persisted prefixes into the inference path.

W2 & Q2: What additional scheduling capability does CAA provide over existing methods?

Its added capability is to translate the recoverable history state maintained by Arena into request-level recovery and recomputation costs, allowing admission decisions to distinguish differences in service work that the currently matched prefix length does not express.

Specifically, compared with our LPM baseline, which orders requests by the currently matched prefix length, CAA also uses device/host hits, the committed continuous prefix in storage, and remaining prefill work to estimate recovery and recomputation costs. The ablation study in Section 5.3 shows that, under the same Arena configuration, full CAA reduces mean TTFT and mean session JCT by 9.6% and 8.5%, respectively, and P99 TTFT and P99 session JCT by 5.7% and 18.6%, relative to FCFS. Compared with Arena+LPM, it increases mean TTFT by 6.3% while reducing P99 TTFT by 65.1%, illustrating the trade-off between mean response latency and tail response latency.

To quantify the incremental benefit of work-based ordering beyond the other admission mechanisms, we hold Arena, the age fence, and recovery-ready handoff fixed for GPT-OSS-20B, Hermes C8, replacing only the ordering criterion with LPM.

MethodPrefix reuse rate (%)Mean TTFT (s)P99 TTFT (s)Mean session JCT (s)P99 session JCT (s)
UMAO (W(r)W(r))89.500613.6159.16331.42618.83
UMAO (LPM)89.573514.9774.00355.92681.78

The two variants have similar overall prefix reuse rates. Relative to the LPM ordering variant, work-based ordering reduces mean/P99 TTFT by 9.1%/20.1% and mean/P99 session JCT by 6.9%/9.2%. This shows that the ordering criterion provides an additional benefit when the accompanying admission mechanisms are held fixed.

Dear Reviewer,

The authors have responded to your review. Could you please take a moment to review their responses and engage in the discussion?

Thank you for your time and valuable contributions.

Kind regards, AC

Dear Reviewer fVqG,

Thank you again for your thoughtful review. We have provided detailed responses and additional experimental results addressing your concerns above. As the discussion period progresses, we would greatly appreciate it if you could take a moment to review our response and let us know whether it resolves your concerns or if any questions remain. We would be happy to provide further clarification. Thank you very much for your time.

Reviewer_pPxAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UMAO to address memory pressure in multi-round agent serving on devices where CPU and GPU share physical memory. Its Unified Cache Arena retains long-term histories in storage, uses bounded host buffers for transfers, and maintains continuously recoverable prefixes, including aligned state checkpoints when required. Cache-Aware Agent Admission uses this prefix information to estimate recovery and recomputation costs and combines work-based ordering with aging. Trace-replay experiments on DGX Spark report 87.5%–90.3% prefix reuse and improvements in response and session-completion latency across multiple settings.

Critical strengths
  1. The paper identifies an important systems problem and provides a coherent design around recoverable prefixes. Under unified memory, allocating more capacity to the host cache reduces the capacity available for active device KV. UMAO addresses this constraint by using bounded host staging and storage-backed history, while ensuring that persisted KV pages and any required model state form a recoverable prefix. This integration is a potentially useful contribution to memory-constrained agent serving.
  2. The controlled experiments provide strong evidence that the cache lifecycle design contributes beyond memory repartitioning. The evaluation covers 12 model–trace–concurrency settings and includes a particularly useful matched-capacity experiment in Appendix A.4.1. With identical memory allocation and FCFS admission, Arena improves prefix reuse by 23.89 percentage points and reduces mean TTFT by 50.1% relative to Write Through on GPT-OSS-20B with Hermes at C8. This comparison strengthens the attribution of performance gains to the proposed lifecycle mechanisms.
Critical weaknesses
  1. The admission-policy experiments do not fully isolate the contribution of the service-work model. Figure 6 includes Arena with LPM and UMAO without the age fence, but does not compare against LPM with the same age fence and recovery-ready handoff as CAA. Since the existing ablations show that aging substantially affects tail latency, it remains unclear how much additional benefit comes from ordering requests by the proposed work estimate. This limits the evidence supporting the admission component’s added complexity.
  2. The statistical reliability of the session-level tail-latency improvements is insufficiently established. Appendix A.1 states that P99 session JCT is computed from 50 sessions per configuration, making it sensitive to the slowest few sessions. The reported results do not include uncertainty estimates or variability across independent replays or alternative session cohorts. Because tail-latency reduction is a central contribution, this uncertainty limits confidence in the magnitude and robustness of the reported P99 gains.
Questions for authors
  1. With Arena, the age fence, and recovery-ready handoff held fixed, how does replacing the proposed work-based ordering with LPM affect mean and P99 TTFT and session JCT?
  2. How many independent replay runs underlie each reported configuration? Given the 50-session sample used for P99 session JCT, can the authors report variability across repeated runs or alternative session cohorts with appropriate uncertainty estimates?
3 comments

We thank the reviewer for these two follow-up questions on the incremental benefit of work-based ordering and on tail-latency variability. Our responses follow.

W1 & Q1: Incremental benefit of work-based ordering over LPM with Arena, the age fence, and recovery-ready handoff held fixed

Following your suggestion, we use the same GPT-OSS-20B, Hermes C8 setting as the ablation study in Section 5.3. We hold Arena, the age fence, and recovery-ready handoff fixed, replacing only work-based ordering with LPM, and compare mean/P99 TTFT and mean/P99 JCT.

MethodPrefix reuse rate (%)Mean TTFT (s)P99 TTFT (s)Mean JCT (s)P99 JCT (s)
UMAO (W(r)W(r))89.500613.6159.16331.42618.83
UMAO (LPM)89.573514.9774.00355.92681.78

The two variants have similar overall prefix reuse rates. Relative to the LPM ordering variant, work-based ordering reduces mean/P99 TTFT by 9.1%/20.1% and mean/P99 JCT by 6.9%/9.2%. This comparison shows that work-based ordering provides an additional benefit in this configuration with the same Arena and accompanying admission mechanisms. It supports using the recovery and recomputation costs associated with recoverable prefixes to determine admission order.

W2 & Q2: Number of independent replay runs and tail variability with 50 sessions

We actually performed 10 independent replay runs for each configuration. To provide this information, we re-aggregated the existing per-session records. The table below uses Qwen3.5-35B-A3B, Hermes C8 as an example and reports the mean ± standard deviation across runs for each statistic.

MethodMean JCT (s)P90 JCT (s)P95 JCT (s)P99 JCT (s)Max JCT (s)
UMAO512.29 ± 1.55778.43 ± 20.69846.87 ± 23.56954.35 ± 19.81985.99 ± 15.13
LPM631.00 ± 8.091,254.01 ± 22.941,422.09 ± 20.831,793.85 ± 22.931,818.21 ± 22.98
Write Through Sel1,188.91 ± 4.173,262.78 ± 16.253,532.43 ± 17.703,662.95 ± 18.443,688.44 ± 18.36
Write Back1,278.83 ± 36.323,451.33 ± 83.163,725.57 ± 81.063,872.23 ± 95.533,896.45 ± 95.94
Write Through1,295.58 ± 27.943,359.65 ± 11.763,634.84 ± 11.693,773.41 ± 11.203,804.11 ± 12.89

Comparing the cross-run means, UMAO reduces P90, P95, P99, and maximum JCT by 37.9%, 40.4%, 46.8%, and 45.8%, respectively, relative to LPM. These results show that the observed tail-latency improvements persist across repeated runs in the evaluated configuration.

Dear Reviewer,

The authors have responded to your review. Could you please take a moment to review their responses and engage in the discussion?

Thank you for your time and valuable contributions.

Kind regards, AC

The rebuttal addresses most of the reviewer's concerns. The reviewer would maintain the original rating.

Reviewer_uxMWOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies multi-round agent inference on unified memory architecture (UMA).

The authors propose unified memory agent offloading (UMAO), a framework that connects cache management and request admission through a shared recoverable-prefix state.

UMAO uses a unified cache arena to maintain recoverable historical prefixes under fixed shared-memory capacity and cache-aware agent admission (CAA) to estimate service work from the recoverable-prefix summary and constrain tail waiting.

Experiments on NVIDIA DGX Spark with two models and two agent workloads demonstrate promising results.

Critical strengths
  1. This paper identifies a practical challenge in multi-round agent inference on UMA.

  2. This paper presents two observations on continuous-prefix recoverability and the mean–tail latency trade-off of prefix-match admission, and proposes a framework to address the observed challenges.

  3. The experiments demonstrate improvements across two models, two agent workloads, and multiple concurrency levels. The ablation studies support the effectiveness of the proposed design.

Critical weaknesses
  1. The connection between the motivating observations in Sec. 3.2 and the inditified key challenge of UMA is not sufficiently clear.

  2. Although UMAO integrates cache lifecycle management and request admission into a unified framework, several ideas and mechanisms have been explored in prior work. The authors are suggested to clarify the novel technical contributions.

  3. Although the ablation studies in Sec. 5.3 demonstrate the effectiveness of Arena and CAA, more experiments could help prove the contributions of their fine-grained internal mechanisms.

  4. The readability of Fig. 2 could be improved. For example, the correspondence between the legends and the bars is not sufficiently intuitive.

Questions for authors

Please refer to the concerns raised in Weaknesses 1–3.

The reviewer is open to revising the rating.

3 comments

Thank you for your thoughtful assessment. We address the positioning questions below.

W1: Connection between the observations in Section 3.2 and the key UMA challenge

The two observations in Section 3.2 concern general challenges in multi-round agent serving, while UMA's shared-capacity constraint intensifies competition among history retention, recovery, and execution. Increasing the host history cache reduces the capacity available for device KV. Even when history is retained in storage, recovery still occupies shared staging and device capacity. History maintenance and request admission therefore need to be coordinated.

Observation 1 concerns history recoverability. High page-ACK coverage does not guarantee continuous-prefix recoverability. Without independent host capacity, later reuse depends more on recoverable history in storage. Arena coordinates incremental publication and maintenance of continuous KV pages and the required endpoint state to define the prefix boundary usable for subsequent execution.

Observation 2 concerns request admission. Ordering by the currently matched prefix length exposes a trade-off between mean and tail waiting time. Different residency tiers also entail different recovery and computation costs, and recovery competes with inference for shared resources. CAA therefore uses Arena's recoverable-prefix state to estimate service work from cross-tier recovery, recovery-queue waiting, and remaining prefill, together with an age fence to mitigate prolonged waiting.

W2: Technical contributions relative to existing mechanisms

UMAO builds on the hierarchical caching ideas in HiCache, LMCache, and related work, with new designs tailored to UMA.

History maintenance. On UMA, where device and host share capacity, Arena restricts host to two fixed staging windows and maximizes the device cache pool. Within this bounded two-window staging capacity, Arena accelerates prefix recovery through overlapped pipelining, batched copies, and reduced control-plane overhead. For multi-round agents, it combines round-aligned incremental publication, retries of incomplete pages, and compound commit to maintain a continuously recoverable history boundary.

Coordinated admission. To account for competition between recovery and inference under UMA's shared-capacity constraint, CAA uses the recoverable prefix to combine two-window recovery, recovery-queue waiting, and remaining prefill into a common estimate of service work for admission ordering. It also uses an age fence to mitigate prolonged waiting.

UMAO's technical contribution is to maintain the recoverable history boundary for subsequent agent execution under fixed shared capacity, and to use that boundary and its recovery-path costs for request admission.

W3: Contributions of the fine-grained mechanisms

The following explains the role of each design from four aspects.

History recoverability. In Appendix A.4.1 (GPT-OSS-20B, Hermes C8), with capacity allocation and FCFS fixed, Arena increases the prefix reuse rate by 23.89 percentage points and reduces mean TTFT by 50.1% over Write Through under matched capacity, showing lifecycle gains beyond reallocation. In §5.4, across all non-initial rounds, Arena raises page-ACK coverage, continuous-prefix coverage, and the complete-prefix ratio by at least 75.6, 73.2, and 86.3 percentage points over Matched budget, respectively, linking the gains to backup completeness and prefix recoverability.

Recovery path. §5.4 pairs requests with identical recovery-token counts. For Qwen3.5-35B-A3B and GPT-OSS-20B, recovery-path optimizations yield respective speedups of 3.75×/2.27× in mean Recovery E2E and 2.03×/1.61× in mean Window E2E (per-window recovery latency), directly showing reduced recovery overhead.

MethodPrefix reuse rate (%)Mean TTFT (s)P99 TTFT (s)Mean JCT (s)P99 JCT (s)
UMAO (W(r)W(r))89.500613.6159.16331.42618.83
UMAO (LPM)89.573514.9774.00355.92681.78

Ordering criterion. In the GPT-OSS-20B, Hermes C8 setting of §5.3, we hold Arena, age-fence rules and parameters, and recovery-ready handoff fixed, replacing only work-based ordering with LPM. Prefix reuse remains similar, but work-based ordering reduces mean/P99 TTFT by 9.1%/20.1% and mean/P99 JCT by 6.9%/9.2% relative to LPM, showing the additional benefit of the ordering criterion.

Tail waiting. In §5.3, after removing the age fence, mean TTFT decreased by only 6.3%, while P99 TTFT and P99 JCT increased by 31.7% and 221.7%, respectively. Together with the parameter sensitivity experiments in §5.4, these results support its role in mitigating long waits.

W4: Readability of Figure 2

We will align the legend and bar order in Figure 2, and standardize the policy identifiers.

Dear Reviewer,

The authors have responded to your review. Could you please take a moment to review their responses and engage in the discussion?

Thank you for your time and valuable contributions.

Kind regards, AC

The authors have addressed most of my concerns, and I am raising my rating.

Reviewer_hGnEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UMAO manages agent histories on unified-memory devices using bounded host staging and storage-backed recoverable prefixes. It connects this cache lifecycle to admission decisions based on recovery and prefill cost, with aging to reduce long waits. The evaluation studies the resulting reuse, latency, and session completion behavior.

Critical strengths

The problem is concrete: host and device caches compete for the same physical memory, so a larger host cache is not free. The design accounts for this constraint and connects recoverability to admission, rather than treating storage and scheduling as unrelated components.

The matched-capacity and eager-write controls show that Arena's gains are not explained solely by reallocating the cache budget toward the device. The admission ablations also expose the tradeoff between average performance and long waits. Together, these experiments provide more convincing evidence than a comparison against only the original cache partition.

Critical weaknesses

The session-tail estimates need more support. Session P99 is calculated from only 50 sessions, making it largely dependent on the largest observations. Please report P90/P95, maxima, and repeated-run variability. The current results can show that some long sessions improve, but precise P99 speedup claims are harder to justify from this sample size.

The replay choices may materially affect reuse. The evaluation uses two execution slots, fixed one-second gaps, and a 128-token output limit. These choices influence cache survival, storage activity, and opportunities to finish persistence before reuse. Please test longer outputs and variable tool delays to show how sensitive the gains are to these assumptions.

The memory-pressure argument needs qualification. Appendix A.4.3 reports at least 26.91 GiB of available memory under a fixed KV budget. Reserving headroom can be sensible, but pressure against a configured cache limit is different from exhausting usable memory. Please explain the reserve and evaluate a larger safe cache budget to test whether the advantage persists

Questions for authors

See Weakness.

3 comments

We thank the reviewer for the comments on the statistical reliability of the session tail latency, the sensitivity to replay settings, and the cache budget versus physical memory pressure. Our responses follow.

W1: Sample size and statistical reliability of P99 JCT

We actually performed 10 independent replay runs for each configuration. To provide more information about the distribution of JCT, we reaggregated the existing per-session records for Qwen3.5-35B-A3B, Hermes C8. The table reports the mean ± standard deviation across runs for each statistic.

MethodMean JCT (s)P90 JCT (s)P95 JCT (s)P99 JCT (s)Max JCT (s)
UMAO512.29 ± 1.55778.43 ± 20.69846.87 ± 23.56954.35 ± 19.81985.99 ± 15.13
LPM631.00 ± 8.091,254.01 ± 22.941,422.09 ± 20.831,793.85 ± 22.931,818.21 ± 22.98
Write Through Sel1,188.91 ± 4.173,262.78 ± 16.253,532.43 ± 17.703,662.95 ± 18.443,688.44 ± 18.36
Write Back1,278.83 ± 36.323,451.33 ± 83.163,725.57 ± 81.063,872.23 ± 95.533,896.45 ± 95.94
Write Through1,295.58 ± 27.943,359.65 ± 11.763,634.84 ± 11.693,773.41 ± 11.203,804.11 ± 12.89

The session-level statistics vary little across these repeated runs. Comparing the cross-run means, UMAO reduces P90, P95, P99, and maximum JCT by 37.9%, 40.4%, 46.8%, and 45.8%, respectively, relative to LPM. These results show that the observed tail-latency improvements persist across repeated runs in the evaluated configuration.

W2: Sensitivity to replay settings

We added experiments with Qwen3.5-35B-A3B, Hermes C8, separately increasing the completion-length limit to 512 tokens and varying the inter-round interval from 1 s to 9 s. The results are shown below.

MethodPrefix reuse rate (%)Mean TTFT (s)P99 TTFT (s)Mean JCT (s)P99 JCT (s)
(a) Completion-length limit: 512 tokens
UMAO87.1228.34133.47661.881,362.66
LPM80.0238.04685.91817.352,578.43
Write Through Sel43.6371.21366.251,450.254,302.43
Write Back41.8172.17370.471,467.504,522.87
Write Through41.0472.70363.051,484.044,367.60
(b) Inter-round interval: Uniform(1 s, 9 s)
UMAO87.4418.6094.76516.62927.69
LPM76.4031.14568.99736.052,061.36
Write Through Sel47.7755.89289.441,208.623,526.75
Write Back50.4753.28295.751,148.633,737.73
Write Through41.7259.92299.901,288.103,787.17
  • With the completion-length limit increased to 512 tokens, UMAO improves the prefix reuse rate by 7.10 percentage points over the best baseline for that metric. Relative to the best baseline for each latency metric, it reduces mean/P99 time to first token (TTFT) by 25.5%/63.2% and mean/P99 JCT by 19.0%/47.2%.

  • With the inter-round interval changed to Uniform(1 s, 9 s), UMAO improves the prefix reuse rate by 11.04 percentage points over the best baseline for that metric. Relative to the best baseline for each latency metric, it reduces mean/P99 TTFT by 40.3%/67.3% and mean/P99 JCT by 29.8%/55.0%.

  • These results show that UMAO retains its prefix reuse and latency advantages when the completion-length limit and inter-round interval are changed separately in the evaluated configuration.

W3: Configured cache budget and physical memory pressure

We reserve memory headroom primarily to leave room for launching external applications in real agent scenarios. The current 309,696-token shared KV budget is determined based on the Trace context length and HiCache's HICACHE_RATIO = 1.1 (which by default requires the L2 Host/L1 Device cache allocation ratio to be > 1); configurations such as Write Through use a device/host split of 147,456/162,240 tokens.

To evaluate a larger safe cache budget, we added an experiment with Qwen3.5-35B-A3B, Hermes C8 using a shared KV budget of 414,656 tokens and a device/host split of 197,440/217,216 tokens. This budget is 33.9% larger than the original budget, and the minimum observed available memory during the experiment was 19.81 GiB. The results are shown below.

MethodPrefix reuse rate (%)Mean TTFT (s)P99 TTFT (s)Mean JCT (s)P99 JCT (s)
UMAO88.0221.84114.95508.46892.00
LPM81.9926.46489.11599.561634.33
Write Through Sel57.7448.07251.671029.963211.66
Write Back52.8151.50280.971100.993624.02
Write Through44.0657.89282.251228.133613.58

Under this larger budget, UMAO achieves a prefix reuse rate of 88.02%. Relative to LPM, it reduces mean TTFT, mean JCT, and P99 JCT by 17.5%, 15.2%, and 45.4%, respectively. Relative to Write Through Sel, the baseline with the lowest P99 TTFT, UMAO reduces P99 TTFT by 54.3%. These results show that UMAO's advantages persist when the shared cache budget is increased in this configuration.

Dear Reviewer,

The authors have responded to your review. Could you please take a moment to review their responses and engage in the discussion?

Thank you for your time and valuable contributions.

Kind regards, AC

I've read the response and I have no further questions. I will keep my score as-is.

Comments on the paper

1 comment

We thank all reviewers for their valuable comments and suggestions. We have uploaded a revised manuscript addressing their feedback, with additions and modifications highlighted in blue for ease of reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.