- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the client scheduling problem for evaluating LLM workloads with shared prompt prefixes. They propose Cache-Aware Execution Planning (CAEP), specifically the CAEP-SC algorithm, which issues a leader request and verifies cache hits before releasing concurrent followers. Experiments covered hosted APIs based on different models, as well as local vLLM deployments. The results show that a single warm-up request can achieve measurable cost savings in managed environments; however, telemetry-based verification did not statistically demonstrate additional benefits and may increase computational overhead under cache pressure. The family admission mechanism can reduce redundant computations in certain constrained configurations, but its improvements do not consistently transfer to new document workloads.
Critical strengths
-
By using matched leader-only and double-blind controls, the study effectively distinguished the benefits of preheating from the results of telemetry-based validation. This strengthens the causal interpretation of the reported cost and computational differences.
-
Extensive empirical diagnostics. The evaluation spanned multiple providers, models, cache budgets, and independent repeat experiments. Cache replacement, telemetry, and access analysis provided useful evidence regarding actual failure modes.
Critical weaknesses
W1. Compared to a simpler warm-up baseline, the proposed CAEP-SC does not demonstrate significant cost or computational advantages, which greatly undermines its methodological contribution.
W2. The improvement in warm-up performance is primarily relative to an inefficient warm-up strategy with no upper bound, while CAPR, which can save workload, still maintains a faster speed.
W3. The fixed validation threshold ignores cache block alignment, leading to unnecessary probe failures and potentially overestimating gating overhead.
W4. The significant access advantages observed on existing QA workloads do not consistently generalize to new document corpora.
W5. The break-even analysis is primarily based on retrospective data and fails to establish a practical adaptive scheduling strategy for evolving cache conditions.
Questions for authors
Q1. Would adopting a validation threshold that accounts for block alignment or is adaptive eliminate the observed gating overhead, particularly for those 112 failed Qwen probes—which still exhibit significant cache reuse?
Q2. Can the authors validate their scheduling advantages for real-world long-context hosted workloads on other providers and execution windows?
Q3. Are there experimentally validated configurations in which CAEP-SC outperforms leader-only warm-up or workload-retaining CAPR in terms of the cost-latency tradeoff?
Q4. Can the authors provide stronger evidence regarding the ability of Alibaba’s cache layout—which incorporates role changes—to maintain output quality?
If the questions can be addressed, I`m willing to rise my scores.
5 comments
Response to Reviewer DA7Q (2/2)
Q4: does the role-changing Alibaba layout preserve output quality? We added a paired, prespecified study on 64 documents (32 NarrativeQA, 32 Qasper), with two questions and two outputs per layout and an identical visible text window of up to 24,000 UTF-8 bytes. QA-F1 was 0.4152 for the original single-user layout and 0.4253 for the marked system/user layout: difference +0.0101, one-sided 95% lower bound −0.0172, above the prespecified −0.0200 margin (p = 0.035). This establishes non-inferiority at that margin for the tested model snapshot and window (Section 5.5).
W2: the admission gain is relative to an inefficient uncapped warm-up, and CAPR stays faster. We agree that work-conserving release is the reference, and the revision's recommendation follows from it: in a self-hosted engine, release work-conservingly, or cap open families when warm-up is required (Section 7). On the existing Qwen QA bank, a cap of four cuts computed input to 0.488 [0.429, 0.556] and makespan to 0.679 [0.630, 0.732] of uncapped blind-two, and still takes 1.265 [1.147, 1.395] times as long as work-conserving release.
W4: the admission gain does not transfer to new documents. The cap removes what uncapped warm-up adds, and on the new banks uncapped warm-up adds little: its prefix working set averages 2.16 times the cache capacity on the existing bank and 1.05 times on the new banks. vLLM's cache events show the mechanism (descriptive; Appendix D.2): lookups of a family prefix whose last cached copy had been discarded while the family was still open occurred for 19–21 of 69 requests per replication on the existing bank, against 3 of 54 and 11 of 74 on two new banks; with the cap, 2 of 394 overall; under work-conserving release, none. With little excess to remove, the cap's effect on the new banks is unresolved (0.862 [0.597, 1.245] computation).
W5: the break-even analysis is retrospective. We agree that the break-even equation is accounting, and the revision presents it as such (Section 3.5). Two prospective tests now bear on the policy rule:
- The wait. The wait sweep suggested 200 ms; full reuse at 200 ms was then a prespecified primary endpoint in three further studies and held in every family (24/24, 32/32, 30/30), and it held in 60/60 truncation-study families.
- The cost model. Before any call of the shared-fraction study, we froze cost predictions from Equation 3 for leader-only with a 200 ms wait against work-conserving release. Observed ratios were 0.229 [0.216, 0.242], 0.399 [0.388, 0.410] and 0.595 [0.588, 0.602] at 97%, 75% and 50% shared, against predictions of 0.211, 0.388 and 0.592. By the prespecified criterion the prediction held at 50% and missed at 97% and 75%, where it fell just below narrow intervals: the calibrated cached depth exceeded the counts the provider reported, and, post hoc, the equation with the reported counts reproduces every cost (Appendix G.3).
So the equation predicts the saving when reuse is known; what it cannot supply is reuse after the leader, which ranged from 20% to 100% of followers across burst cohorts and did not transfer between workloads post hoc (Appendix G.3). Section 7 therefore calibrates a short wait instead of estimating reuse, and adaptive policy selection remains future work.
If any of Q1–Q4 remains open, we can add analyses before the discussion closes.
Thank you once again for taking the time to review our manuscript. We have carefully addressed all the concerns raised in your review and provided detailed responses to fully resolve each of the points raised.
As the discussion period is coming to a close, we would like to kindly follow up on our previous response in case you have not yet had a chance to review it. We appreciate the many demands on your time and would be grateful for any opportunity you may have to review our responses.
Thank you once again for your time and consideration!
Response to Reviewer DA7Q (1/2)
Thank you for the concrete questions. Short answers:
W1: CAEP-SC shows no advantage over simpler warm-up. As your first strength says, the matched leader-only and blind controls separate the benefit of warming the cache from that of telemetry verification. That attribution is the paper's contribution, and CAEP-SC is the policy it tests; the revised introduction and contributions say so. The new studies extend the attribution to long contexts, a second provider and settings where the leader fails, and they identify what makes a completed leader not yet enough (Q3).
Q1 / W3: would a block-aware threshold remove the gating overhead? A new four-launch Qwen study at 8 GiB adds a gate whose threshold is the block-aligned family prefix. It still computes 1.379 [1.242, 1.531] times blind-two's input and takes 1.342 [1.218, 1.479] times as long; against the 90% gate it is unresolved (0.977 [0.884, 1.079] computation). Alignment removes all 53 exactly-aligned rejections, but 250 of 345 probes then report zero cached tokens, so misalignment alone cannot explain the penalty (Section 5.3; Appendix D.2); the cache-event study observes the prefix losses directly (Section 5.4). The 112 failed probes you mention came from the larger budgets, where the gate had no computation penalty. Other adaptive rules remain untested.
Q2: real long-context hosted workloads, other providers and execution windows. Yes; the table in the general response gives every study. Within NarrativeQA the ratio is lower for longer prefixes: 0.225 at 9–10K tokens and 0.157 at 21–25K (pointwise). One difference matters in practice: on DeepSeek a 200 ms wait gave full reuse in all 176 families tested, but on Ark it did not repair 4.3K-token bursts. All seven followers reported cached tokens in only 4, 6 and 5 of 20 families after truncation, a natural stop and a 200 ms wait (Fisher p = 0.72; Appendix H.4), whereas 29 of 30 leader-only followers reached the threshold on the Ark study's 8K–20K-token documents. Ark also differed in workload and endpoint, so we do not attribute this to the provider alone; the wait is calibrated to the provider and workload (Section 7).
Q3: a validated configuration where CAEP-SC beats leader-only or work-conserving release. Against leader-only, yes, in the two burst studies where leader-only fails: the gate costs 0.406 [0.291, 0.566] of leader-only in the 3K burst study and 0.372 [0.294, 0.471] in the truncation study. Against waiting without telemetry, no: a 50 ms wait was not resolvably different from the gate in cost, and in the truncation study the gate cost 1.426 [1.288, 1.579] times a 200 ms wait and took 1.671 [1.499, 1.864] times as long. Neither study had a work-conserving arm. Where the gate beats leader-only, its saving comes from requests sent later (Section 3.4, Equation 2), which a wait provides without failed probes.