acceptodds
Under review as a conference paper at ICLR 2027

When to Wait Before Fan-Out: Warm-Up, Verification, and Residency in Cache-Aware LLM Evaluation

Abstract

Long-context LLM evaluation often issues many requests that share a long prefix. When the provider does not coordinate cold requests, dispatching them concurrently can create the same prefix repeatedly, whereas serial release sacrifices parallelism. We study this release decision from the client side, where cache state is hidden and reuse is observed only through completed-response telemetry. Cache-Aware Execution Planning (CAEP) changes only execution order and release time, keeping request payloads and scoring fixed; its stable-canary policy (CAEP-SC) releases a prefix family after a leader and a probe verify reuse, and a break-even rule relates avoided prefix creation to the value of added latency. In development studies, CAEP-SC reduces telemetry-priced cost relative to work-conserving scheduling by 49.9% on DeepSeek at a 3.7% makespan increase, and creation-confirmed release reduces it by 63.4% on Alibaba's explicit cache at 1.50× makespan. Matched controls locate these savings in warm-up rather than verification: in an eight-block hosted ablation, leader-only and blind two-request warm-up match the gate's cost in every block, and across 37 replicated matched comparisons gating never resolves a cost or computation saving. Under constrained vLLM capacity, CAEP-SC processes 3.18× the work-conserving input with Llama-3.1-8B, 50.7% more than blind warm-up; Qwen3.6-27B shows a 41.2% gate penalty at an 8 GiB budget, and competing prefixes can erode reuse after a passing probe. Limiting unfinished families removes excess synthetic computation; on a fixed Qwen QA workload, a four-family cap cuts computed input by 51.2% and makespan by 32.1% relative to uncapped blind warm-up, although work-conserving release remains faster and the effect is unresolved on new document banks. Waiting before fan-out pays off when concurrent cold requests are billed separately, and one completed warm-up request captured the full measured hosted saving.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM inference, prompt caching, KV cache, cache-aware scheduling, long-context evaluation, client-side scheduling, inference efficiency, cost–latency trade-offs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DA7QOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the client scheduling problem for evaluating LLM workloads with shared prompt prefixes. They propose Cache-Aware Execution Planning (CAEP), specifically the CAEP-SC algorithm, which issues a leader request and verifies cache hits before releasing concurrent followers. Experiments covered hosted APIs based on different models, as well as local vLLM deployments. The results show that a single warm-up request can achieve measurable cost savings in managed environments; however, telemetry-based verification did not statistically demonstrate additional benefits and may increase computational overhead under cache pressure. The family admission mechanism can reduce redundant computations in certain constrained configurations, but its improvements do not consistently transfer to new document workloads.

Critical strengths
  1. By using matched leader-only and double-blind controls, the study effectively distinguished the benefits of preheating from the results of telemetry-based validation. This strengthens the causal interpretation of the reported cost and computational differences.

  2. Extensive empirical diagnostics. The evaluation spanned multiple providers, models, cache budgets, and independent repeat experiments. Cache replacement, telemetry, and access analysis provided useful evidence regarding actual failure modes.

Critical weaknesses

W1. Compared to a simpler warm-up baseline, the proposed CAEP-SC does not demonstrate significant cost or computational advantages, which greatly undermines its methodological contribution.

W2. The improvement in warm-up performance is primarily relative to an inefficient warm-up strategy with no upper bound, while CAPR, which can save workload, still maintains a faster speed.

W3. The fixed validation threshold ignores cache block alignment, leading to unnecessary probe failures and potentially overestimating gating overhead.

W4. The significant access advantages observed on existing QA workloads do not consistently generalize to new document corpora.

W5. The break-even analysis is primarily based on retrospective data and fails to establish a practical adaptive scheduling strategy for evolving cache conditions.

Questions for authors

Q1. Would adopting a validation threshold that accounts for block alignment or is adaptive eliminate the observed gating overhead, particularly for those 112 failed Qwen probes—which still exhibit significant cache reuse?

Q2. Can the authors validate their scheduling advantages for real-world long-context hosted workloads on other providers and execution windows?

Q3. Are there experimentally validated configurations in which CAEP-SC outperforms leader-only warm-up or workload-retaining CAPR in terms of the cost-latency tradeoff?

Q4. Can the authors provide stronger evidence regarding the ability of Alibaba’s cache layout—which incorporates role changes—to maintain output quality?

If the questions can be addressed, I`m willing to rise my scores.

5 comments
AuthorsOpenReview ↗

Response to Reviewer DA7Q (1/2)

Thank you for the concrete questions. Short answers:

  • Q1. No. With the threshold aligned to the cache's 784-token blocks, most of the penalty remains, because most probes then find nothing cached.
  • Q2. Yes: long real documents, a public benchmark, a later day and a second provider. The wait before fan-out has to be calibrated to the provider and workload.
  • Q3. Against leader-only, yes, in two configurations where leader-only fails; against a short fixed wait, no.
  • Q4. Yes: the layout passes a prespecified non-inferiority test.
  • W5. The cost model now has a prospective test, frozen before any call.

W1: CAEP-SC shows no advantage over simpler warm-up. As your first strength says, the matched leader-only and blind controls separate the benefit of warming the cache from that of telemetry verification. That attribution is the paper's contribution, and CAEP-SC is the policy it tests; the revised introduction and contributions say so. The new studies extend the attribution to long contexts, a second provider and settings where the leader fails, and they identify what makes a completed leader not yet enough (Q3).

Q1 / W3: would a block-aware threshold remove the gating overhead? A new four-launch Qwen study at 8 GiB adds a gate whose threshold is the block-aligned family prefix. It still computes 1.379 [1.242, 1.531] times blind-two's input and takes 1.342 [1.218, 1.479] times as long; against the 90% gate it is unresolved (0.977 [0.884, 1.079] computation). Alignment removes all 53 exactly-aligned rejections, but 250 of 345 probes then report zero cached tokens, so misalignment alone cannot explain the penalty (Section 5.3; Appendix D.2); the cache-event study observes the prefix losses directly (Section 5.4). The 112 failed probes you mention came from the larger budgets, where the gate had no computation penalty. Other adaptive rules remain untested.

Q2: real long-context hosted workloads, other providers and execution windows. Yes; the table in the general response gives every study. Within NarrativeQA the ratio is lower for longer prefixes: 0.225 at 9–10K tokens and 0.157 at 21–25K (pointwise). One difference matters in practice: on DeepSeek a 200 ms wait gave full reuse in all 176 families tested, but on Ark it did not repair 4.3K-token bursts. All seven followers reported cached tokens in only 4, 6 and 5 of 20 families after truncation, a natural stop and a 200 ms wait (Fisher p = 0.72; Appendix H.4), whereas 29 of 30 leader-only followers reached the threshold on the Ark study's 8K–20K-token documents. Ark also differed in workload and endpoint, so we do not attribute this to the provider alone; the wait is calibrated to the provider and workload (Section 7).

Q3: a validated configuration where CAEP-SC beats leader-only or work-conserving release. Against leader-only, yes, in the two burst studies where leader-only fails: the gate costs 0.406 [0.291, 0.566] of leader-only in the 3K burst study and 0.372 [0.294, 0.471] in the truncation study. Against waiting without telemetry, no: a 50 ms wait was not resolvably different from the gate in cost, and in the truncation study the gate cost 1.426 [1.288, 1.579] times a 200 ms wait and took 1.671 [1.499, 1.864] times as long. Neither study had a work-conserving arm. Where the gate beats leader-only, its saving comes from requests sent later (Section 3.4, Equation 2), which a wait provides without failed probes.

AuthorsOpenReview ↗

Response to Reviewer DA7Q (2/2)

Q4: does the role-changing Alibaba layout preserve output quality? We added a paired, prespecified study on 64 documents (32 NarrativeQA, 32 Qasper), with two questions and two outputs per layout and an identical visible text window of up to 24,000 UTF-8 bytes. QA-F1 was 0.4152 for the original single-user layout and 0.4253 for the marked system/user layout: difference +0.0101, one-sided 95% lower bound −0.0172, above the prespecified −0.0200 margin (p = 0.035). This establishes non-inferiority at that margin for the tested model snapshot and window (Section 5.5).

W2: the admission gain is relative to an inefficient uncapped warm-up, and CAPR stays faster. We agree that work-conserving release is the reference, and the revision's recommendation follows from it: in a self-hosted engine, release work-conservingly, or cap open families when warm-up is required (Section 7). On the existing Qwen QA bank, a cap of four cuts computed input to 0.488 [0.429, 0.556] and makespan to 0.679 [0.630, 0.732] of uncapped blind-two, and still takes 1.265 [1.147, 1.395] times as long as work-conserving release.

W4: the admission gain does not transfer to new documents. The cap removes what uncapped warm-up adds, and on the new banks uncapped warm-up adds little: its prefix working set averages 2.16 times the cache capacity on the existing bank and 1.05 times on the new banks. vLLM's cache events show the mechanism (descriptive; Appendix D.2): lookups of a family prefix whose last cached copy had been discarded while the family was still open occurred for 19–21 of 69 requests per replication on the existing bank, against 3 of 54 and 11 of 74 on two new banks; with the cap, 2 of 394 overall; under work-conserving release, none. With little excess to remove, the cap's effect on the new banks is unresolved (0.862 [0.597, 1.245] computation).

W5: the break-even analysis is retrospective. We agree that the break-even equation is accounting, and the revision presents it as such (Section 3.5). Two prospective tests now bear on the policy rule:

  • The wait. The wait sweep suggested 200 ms; full reuse at 200 ms was then a prespecified primary endpoint in three further studies and held in every family (24/24, 32/32, 30/30), and it held in 60/60 truncation-study families.
  • The cost model. Before any call of the shared-fraction study, we froze cost predictions from Equation 3 for leader-only with a 200 ms wait against work-conserving release. Observed ratios were 0.229 [0.216, 0.242], 0.399 [0.388, 0.410] and 0.595 [0.588, 0.602] at 97%, 75% and 50% shared, against predictions of 0.211, 0.388 and 0.592. By the prespecified criterion the prediction held at 50% and missed at 97% and 75%, where it fell just below narrow intervals: the calibrated cached depth exceeded the counts the provider reported, and, post hoc, the equation with the reported counts reproduces every cost (Appendix G.3).

So the equation predicts the saving when reuse is known; what it cannot supply is reuse after the leader, which ranged from 20% to 100% of followers across burst cohorts and did not transfer between workloads post hoc (Appendix G.3). Section 7 therefore calibrates a short wait instead of estimating reuse, and adaptive policy selection remains future work.

If any of Q1–Q4 remains open, we can add analyses before the discussion closes.

Thank you once again for taking the time to review our manuscript. We have carefully addressed all the concerns raised in your review and provided detailed responses to fully resolve each of the points raised.

As the discussion period is coming to a close, we would like to kindly follow up on our previous response in case you have not yet had a chance to review it. We appreciate the many demands on your time and would be grateful for any opportunity you may have to review our responses.

Thank you once again for your time and consideration!

Reviewer_JQZFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when a client should release LLM requests that share a long prompt prefix, when cache state is visible only through response telemetry. The proposed framework, CAEP, changes only the order and timing of requests, keeping content and scoring fixed. Its main policy, CAEP-SC, sends a leader and a probe, and releases the rest of the family only once the probe reports sufficient cache reuse. The key methodological contribution is a set of matched warm-up controls that skip the telemetry check. These separate the benefit of warming the cache from the benefit of verifying reuse. The paper also studies capping the number of in-progress families. Across hosted and vLLM experiments, it finds that warm-up alone captures the saving, that verification adds no resolved benefit and can add computation under cache pressure, and that capping removes extra computation from warm-up but does not beat work-conserving release.

Critical strengths
  1. Prompt caching is now exposed and billed by many hosted providers, and long-context evaluation often sends many requests over the same document. The paper frames when to fan out as a client-side decision made with hidden cache state, which is a realistic setting for practitioners running evaluations through APIs.

  2. Every request is hash-bound, so policies differ only in order and timing, never in messages, decoding settings or scoring. Changes to message layout are audited separately against same-condition repeats.

  3. The displacement and block-alignment analyses explain why cache hits in completed responses are an unreliable basis for release decisions. They report past reuse that competing prefixes can evict, and a fixed fractional threshold can reject substantial block-aligned reuse.

Critical weaknesses
  1. Limited technical contribution. The only positive result that survives the paper's own controls is that a single warm-up request before fan-out captures the hosted saving. Provider documentation already recommends this, and the paper itself cites it. CAEP-SC adds nothing over its own controls: in the matched studies, leader-only warm-up matches or beats its cost and finishes sooner. In vLLM, warm-up never beats work-conserving release, and the admission cap only removes extra computation that warm-up itself creates. The break-even rule is a basic cost-versus-latency comparison, applied only to completed runs.

  2. The negative result on verification is narrower than claimed. The paper tests a single gate design with a fixed fractional threshold, and its own traces show that this threshold is miscalibrated. It rejects probes that report substantial reuse, because returned cache counts can fall just short of the nominal prefix, for example when the engine caches in fixed-size blocks. No calibrated or block-aware threshold is evaluated. The hosted scheduling experiments also never include a setting where the leader fails to warm the cache, even though the provider calibration found such cases. The finding therefore applies to this particular gate, not to telemetry-based verification in general.

  3. The hosted evidence is thin. The matched hosted ablation, which carries the main attribution claim, uses one provider and eight blocks of ~1,200-token synthetic prompts (USD 0.17 in observed usage), so it does not show that the warm-up attribution holds for long-context hosted workloads

  4. The paper is very dense, with dozens of contrasts, varying interval conventions and key evidence in the appendices. A simpler main narrative and a compact summary table would help readers a lot.

  5. Fit for ICLR. The paper has no learning component. It is systems measurement and evaluation-infrastructure work that would likely reach a better-matched audience at MLSys or at a workshop on efficient evaluation.

Questions for authors
  1. When does verification help? Would a gate calibrated to the cache counts the provider actually returns remove the penalty, for example a threshold aligned to the 784-token Qwen blocks or the 1,024-token DeepSeek count?

  2. Does the hosted result hold for long contexts?

  3. What does a practitioner gain beyond existing provider guidance?

5 comments
AuthorsOpenReview ↗

Response to Reviewer JQZF (1/2)

Thank you for separating what the evidence shows about a specific gate from claims about verification in general. Short answers:

  • W2 / Q1. Calibrated and block-aligned gates are now tested, and so are settings where the leader fails. No gate resolves a saving over waiting.
  • W3 / Q2. Yes: at 9K–25K tokens, on a public benchmark, a day later and on a second provider.
  • W1 / Q3, W4, W5. In part 2.

W2 / Q1: one gate design, a miscalibrated threshold, and no setting where the leader fails.

Block-aware threshold. A new four-launch Qwen study at 8 GiB sets the threshold to the block-aligned family prefix (784-token blocks). The gate still computes 1.379 [1.242, 1.531] times blind-two's input and takes 1.342 [1.218, 1.479] times as long. Alignment removes every exactly-aligned rejection, but most failures are now probes that find nothing cached (250 of 345), not counts one block short.

Calibrated hosted thresholds. Hosted gate thresholds are now calibrated on separate documents to the counts the provider returns (for example, DeepSeek's 128-token grid). In the NarrativeQA, 16K synthetic and next-day studies, the gate costs 1.023 [0.941, 1.113], 1.000 [1.000, 1.000] and 1.001 [0.999, 1.003] of leader-only; in the Ark study, 0.956 [0.856, 1.067].

Where the leader fails. We searched for it (Section 5.2). When seven followers left within milliseconds of a leader capped at one output token, only 49 of 140 reused the prefix (3K burst study), and 93 of 224 at 16K. A controlled study then showed that truncation causes this on DeepSeek: on paired families, full reuse held in 60 of 60 after a leader that stopped on its own and 3 of 60 after a truncated one. Here the gate does beat leader-only (0.406 [0.291, 0.566] and 0.372 [0.294, 0.471] in cost), but waiting without telemetry was never resolvably costlier: a 50 ms wait was not resolvably different from the gate, and a 200 ms wait was resolvably cheaper and faster than the gate (gate/wait: cost 1.426 [1.288, 1.579], completion time 1.671 [1.499, 1.864]).

The result on verification therefore covers calibrated thresholds, block-aligned thresholds and settings where leader-only fails: in none of 50 matched comparisons did a gate resolve a saving over waiting without telemetry. It concerns gates that read completed-response cache counts, which is what a client can observe.

W3 / Q2: thin hosted evidence; does the result hold for long contexts? Yes. Beyond the submitted ~1.2K-token ablation, the hosted evidence now includes 40 NarrativeQA documents with 9K–25K-token prefixes, a 16K synthetic study, a next-day replication on 20 new documents, 24 QuALITY articles and a second provider, Volcengine Ark (table in the general response). The ratio falls as prefixes get longer, from 0.310 at ~1.2K tokens to 0.157 at 21–25K (pointwise), and rises as less of each prompt is shared: with a 200 ms wait, 0.229, 0.399 and 0.595 at 97%, 75% and 50% shared.

AuthorsOpenReview ↗

Response to Reviewer JQZF (2/2)

W1 / Q3: what is new beyond provider guidance. Provider guidance says to warm a cache before parallel requests, and the paper cites it. What the guidance does not say, and the matched controls establish, are three findings:

  1. A completed request does not guarantee immediate reuse. On DeepSeek, how the warm-up request ends matters: after a leader truncated at one output token, a millisecond burst lost most reuse, whereas after a natural stop at the same single token it did not (part 1). A short wait repairs this, but its length depends on the provider and workload: 200 ms gave full reuse in all 176 DeepSeek families tested and was not enough on Ark.
  2. Verification pays only if a pass predicts reuse better than elapsed time does. An exact identity (Section 3.4) shows that a gate can be cheaper than leader-only only when requests sent later reuse more often than immediate ones, and a fixed wait also provides that. In 50 matched comparisons no gate resolved a saving over waiting, and against a 200 ms wait the gate cost 43% more.
  3. Under engine cache pressure, warm-up adds computation. It holds more families open than the cache keeps, and vLLM's events show discarded family prefixes being looked up again. A passing probe reports past reuse, not future residency, so verification does not prevent this.

Each finding changes practice (Section 7): let the warm-up finish, add a short calibrated wait only where immediate release loses reuse, skip the gate, and in a memory-constrained engine release work-conservingly or cap open families. Each also needs the matched controls: compared only with concurrent cold release, the gate looks effective, cutting DeepSeek cost by 69–83%.

On the break-even rule: we agree it is simple accounting, and the revision presents it as such (Section 3.5). A post hoc transfer of reuse rates to a new workload (Appendix G.3) and a trace-driven cache replay (Appendix D.2) did not validate; a prediction frozen before the shared-fraction study came within two percentage points of the observed cost ratios (Appendix G.3).

W4: density. We revised toward one main narrative: the introduction answers the title question in four parts, Table 1 gives the main results under one interval convention, and Figure 1 shows the system. New studies enter the main text through Table 1, Section 5.2 and Figure 2. Per-study counts in the introduction and Section 5.3 have moved to the appendix, which is organized by question (Table 3 maps each claim to its evidence).

W5: fit for ICLR. The submission's primary area is infrastructure and systems, which the ICLR call lists. The object of study is the validity and cost of LLM evaluation, which the ICLR community runs at scale: model-visible requests, decoding settings and scoring stay fixed across policies, and outputs and scores are compared separately. The contribution is a measurement and attribution method for that infrastructure rather than a learning algorithm.

Thank you once again for taking the time to review our manuscript. We have carefully addressed all the concerns raised in your review and provided detailed responses to fully resolve each of the points raised.

As the discussion period is coming to a close, we would like to kindly follow up on our previous response in case you have not yet had a chance to review it. We appreciate the many demands on your time and would be grateful for any opportunity you may have to review our responses.

Thank you once again for your time and consideration!

Reviewer_wSwfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose Cache-Aware Execution Planning (CAEP), which reduces redundant prefix computation by controlling request release timing. Its stable-canary policy (CAEP-SC) uses Leader requests for cache warm-up and Probe requests to verify cache reuse before releasing Followers. The paper also introduces Family Admission to limit cache contention. Experiments show that cache warm-up significantly reduces inference costs. However, telemetry-based verification provides no additional savings over simpler warm-up strategies, while Family Admission offers workload-dependent benefits under constrained cache capacity.

Critical strengths
  • S1: Practical significance. The paper addresses a practical problem.
  • S2: Intuitive design. The proposed scheduling strategies are straightforward and easy to understand. Using Leader requests for cache warm-up, Probe requests for reuse verification, and Family Admission to control cache contention provides a clear approach to cache-aware scheduling.
  • S3: Extensive experimental evaluation. The paper conducts comprehensive experiments, comparing multiple scheduling strategies under different workloads and cache configurations.
Critical weaknesses

W1: Insufficient justification of the design motivation and lack of additional benefits in experiments.

The paper proposes CAEP-SC, which uses Leader and Probe requests to verify cache reuse before releasing Followers. However, the paper does not sufficiently explain why an additional Probe and Telemetry-based Verification (Gate) are needed after the Leader has completed normally. When cache capacity is sufficient and the shared prefix remains resident, the Leader can already establish a reusable Prefix Cache, making the necessity of the Probe and Gate unclear. The experiments also do not demonstrate their incremental benefits. In Table 1, Leader-only, Blind-two, and CAEP-SC have exactly the same Cost Ratio, while their average Makespan Ratios are 1.0754, 1.1037, and 1.2934, respectively. The Probe and Gate do not reduce Cost but instead lead to longer execution times.

W2: Insufficient analysis of experimental performance differences.

The paper provides relatively comprehensive experimental comparisons of different scheduling strategies, but lacks sufficient analysis of the reasons behind the performance differences. For example, Figure 2(b)(c) shows that Family Admission has significantly different effects on Computed Input and Makespan under the Existing QA and New Documents workloads. However, the paper provides only limited analysis of this workload dependency.

W3: Presentation.

The paper introduces several scheduling mechanisms, including prefix family construction, warm-up and verification, and family admission, along with global worker, but lacks an overall system architecture diagram, making it difficult for readers to understand the relationships among these components and the complete request scheduling process. In addition, some figures and tables are not clearly presented. For example, the axis labels in Figure 2(b)(c) are not intuitive, and different comparisons use different References, requiring readers to repeatedly refer to the figure captions and Table 2 to understand the meaning of each group of results. Meanwhile, Table 2 and Figure 2(b)(c) present the same experimental results, resulting in a high degree of information redundancy.

Questions for authors

Q1: Under what circumstances does Leader-only fail to effectively establish or maintain a reusable Prefix Cache, while the additional Probe and Gate can improve request scheduling decisions? Considering that Leader-only has already achieved all the observed Hosted cache cost savings in the experiments, I suggest that the authors further clarify the design assumptions, applicable conditions, and practical value of CAEP-SC, and provide corresponding theoretical analysis and experiments.

Q2: Could the authors further analyze the Makespan differences among different strategies in Table 1, as well as the reasons why Family Admission produces different Computed Input and Makespan results under different workloads in Figure 2(b)(c)?

Q3: I suggest adding an overall system architecture diagram to clarify the relationships among the components and the complete request scheduling process. In addition, I suggest improving the axis labels and explanations of comparison targets in Figure 2(b)(c), reducing the redundant information presented in Table 2, and adding the x-axis labels in Figure 3.

Q4: The length of the shared prefix among requests and its proportion of the total input affect the potential benefits of cache reuse. I suggest that the authors report the distributions of shared prefix lengths and proportions for Families in different workloads, and further analyze the differences in Computed Input and Makespan among different strategies under different shared prefix proportions.

Q5: How does the paper select the Leader for each Family? Have the authors considered selecting the Leader based on the shared prefix lengths among requests? Could different Leader Selection strategies affect Computed Input and Makespan?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wSwf (1/2)

Thank you for the precise questions. Short answers:

  • W1 / Q1. Leader-only lost most reuse in one situation: a millisecond burst after a leader truncated at one output token. A short wait repairs it, so a probe is still not needed. In the engine, a probe cannot prevent later eviction.
  • W2 / Q2. The cap removes the excess that uncapped warm-up creates; that excess is large on Existing QA and small on New Documents, and vLLM's cache events show it directly.
  • W3 / Q3. A system figure, one summary table, engine results against one reference, and x-axis labels.
  • Q4, Q5. Distributions are reported, the shared fraction is swept, and leader choice and leader truncation are tested directly.

W1 / Q1: why a probe and gate after the leader; when leader-only fails; theory and experiments.

You are right that when the prefix stays cached after the leader, a probe is unnecessary, and in our data that is the usual case (immediate reuse in 39 of 40 screened families). The revision derives when a probe can pay (Section 3.4). Under leader-only release a fraction qq of the m−1m-1 followers reuse the prefix. The gate sends NN members one at a time as probes until one passes or the family runs out; FF of them fail, and a fraction rr of the m−1−Nm-1-N followers released after a pass reuse the prefix. Counting as RR the requests whose cached tokens fall below the reuse threshold, exactly,

E[Rgate]−E[RLO]=(E[F]−(1−q) E[N])−(m−1−E[N])(r−q).\mathbb{E}[R_{\text{gate}}]-\mathbb{E}[R_{\text{LO}}]=\big(\mathbb{E}[F]-(1-q)\,\mathbb{E}[N]\big)-\big(m-1-\mathbb{E}[N]\big)(r-q).

This identity for threshold-defined counts holds on every execution path, including families in which no probe passes (the second term is zero when no follower is released); it counts requests, not tokens. If probes and later followers reuse as often as immediate followers, both terms are zero, so the gate can be cheaper only if requests sent later reuse more often. That is an effect of elapsed time, which a fixed wait also provides; verification adds value only if a pass predicts reuse better than elapsed time does, and every probe adds a serial request.

Table 2 of the revision measures qq and rr, and two regimes matter. When the leader stops on its own, q≈1q \approx 1 (924/924 followers in four hosted studies, counted at the attainable prefix), so there is nothing to save; the gate costs 1.00–1.02 of leader-only, all unresolved. In a millisecond burst after a leader truncated at one output token, qq falls to 49/140 (3K burst study) and 42/210 (truncation study), while every follower released after a pass reuses the prefix (107/107 and 156/156). The gate then beats leader-only (0.406 [0.291, 0.566] and 0.372 [0.294, 0.471] in cost), but what its followers gain is elapsed time: a 50 ms wait was not resolvably different from the gate, and a 200 ms wait cost 30% less (gate/wait 1.426 [1.288, 1.579]). In the engine, a pass predicts little (rr = 34/47 against qq = 153/232 after blind-two), because competing prefixes can displace a verified prefix.

What makes the leader fail. The truncation study answers this on DeepSeek. On paired families only the leader's output cap changed: truncated at one token, or stopping on its own after the same single token. Full reuse held in 1/30 against 30/30 families at 3K tokens and 2/30 against 30/30 at 16K (exact McNemar p < 10⁻⁸ each). Delaying the burst by a natural stop's median extra latency (100 ms) restored 27/30 families (p = 0.25 against natural stop, not an equivalence test), consistent with a truncated response returning before its prefix can be read. On Ark the contrast did not appear (Appendix H.4).

AuthorsOpenReview ↗

Response to Reviewer wSwf (2/2)

W2 / Q2(a): makespan differences in the submitted Table 1 (now Table 9). Leader-only waits for one request per family before fan-out; blind-two and the gate wait for two, which orders leader-only (1.075) below blind-two (1.104). The remaining gap to the gate (1.293) has two sources. One family's threshold (1,025 tokens) sat one token above the count the provider returns (1,024), so its scope ran ten probes instead of four. And the gate ran before blind-two in six of eight blocks, so the contrast includes order-dependent endpoint variation; for example, the leader's own response took about 130 ms longer on average in the gate's scopes, before any probe was sent. The new studies use calibrated thresholds and balanced or randomized orders; in them gate/blind-two makespan is 0.96–1.07, every primary interval includes one, and the gate takes up to 10% longer than leader-only.

W2 / Q2(b): why admission differs between Existing QA and New Documents. The cap has the same effect on both: capped blind-two's computation is not resolvably different from work-conserving release (1.037 [0.942, 1.142] existing; 1.014 [0.936, 1.099] new). What differs is how much excess uncapped warm-up creates. Each family stays open while its leader runs, so eight workers keep about seven families open instead of three and a half. On the existing bank the prefix working set averages 2.16 times the cache capacity and uncapped blind-two computes 2.12 times the work-conserving input; the cap removes 51% of the input and 32% of the makespan. On new documents the working set is 1.05 times capacity and uncapped blind-two computes 1.18 times the work-conserving input; the cap has little to remove (makespan 1.162 [0.795, 1.697], unresolved). vLLM's cache events agree (Section 5.4; per-bank counts in Appendix D.2): lookups of a discarded family prefix were frequent on the existing bank (19–21 of 69 requests per replication) and rare on two new banks (3 of 54; 11 of 74).

W3 / Q3: presentation. Figure 1 is a new system figure: the fixed request manifest, prefix families, the release policy, dispatch under the family cap and worker limit, and the hidden prefix cache, with release timelines for each policy and the admission mechanism. Figure 3 shows all engine results against work-conserving release, with every row named; the submitted Figure 2 and Table 2 are in Appendix D.1 (Figure 6, Table 20). The submitted Figure 3 is now Figure 5, with labelled x-axes. Table 3 maps each claim to the appendix that supports it.

Q4: shared-prefix lengths and proportions. In the submitted workloads the token-weighted shared fraction is 99.6–99.9% (Appendix A.4): one long document followed by a question, with a median private suffix of 13–21 tokens. Family size is 8 (synthetic), 2–9 (QA bank) and 2–11 (transfer banks); prefix length is about 4.1K tokens (synthetic), 3.9K–16.9K (QA bank) and 9K–25K (new NarrativeQA runs). To test lower sharing, the shared-fraction study adds private blocks so that 97%, 75% or 50% of each prompt is shared. Leader-only with a 200 ms wait then costs 0.229 [0.216, 0.242], 0.399 [0.388, 0.410] and 0.595 [0.588, 0.602] of work-conserving release, within two percentage points of predictions from Equation 3 frozen before any call (the 97% and 75% predictions fell just below their narrow intervals; Appendix G.3). The saving shrinks with the shared fraction, as the equation predicts; an exploratory 25% level gave 0.802.

Q5: leader selection. The leader is the first member in the frozen input order (Section 3.3). An audit of the 596 family records found no nested prefix subgroups once identical requests were excluded, so a prefix-length-based choice has little to select among here; more heterogeneous families could differ (Appendix A.4). A leader-choice study on identical requests found full reuse in 30/30 families for every leader. What does matter is how the leader ends (part 1): let the warm-up request finish, and add a short wait only where immediate release still loses reuse.

Thank you once again for taking the time to review our manuscript. We have carefully addressed all the concerns raised in your review and provided detailed responses to fully resolve each of the points raised.

As the discussion period is coming to a close, we would like to kindly follow up on our previous response in case you have not yet had a chance to review it. We appreciate the many demands on your time and would be grateful for any opportunity you may have to review our responses.

Thank you once again for your time and consideration!

Thank you for your detailed questions. To make the revision easier to check, we summarize the locations of the main updates relevant to your review:

  • Q1: Section 5.2 reports the paired truncation study: leaders with the same one-token output yielded full follower reuse in 60/60 families after natural termination versus 3/60 after truncation. The waiting control also clarifies why this does not establish a need for verification.
  • Q2: Section 5.4 adds native cache-event evidence explaining why warm-up increases computation under cache pressure and why the admission benefit varies across workloads.
  • Q3–Q5: Figures 1 and 3 clarify the system and comparisons; the added shared-fraction and leader-choice studies address the requested analyses, with details in part 2 of our response.

We would appreciate knowing whether any specific concern remains unresolved after these revisions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the new evidence and the revision

We thank the reviewers for specific, checkable questions about warm-up, verification, admission and external validity. We ran 16 new studies with 16,710 formal responses. Each protocol was frozen before its formal run (not publicly preregistered), and amendments, stopped stages and failed qualifications are kept in the paper (Appendix I). The revised PDF is uploaded, with new and changed text in blue.

The paper's contribution is the set of request-preserving matched controls and what they attribute; CAEP-SC is the policy those controls test. The revised introduction, contributions and conclusion say this. With the controls, the new studies establish four findings; the responses below give the details each reviewer asked for.

  1. One completed request captures the hosted saving at long context, on a public benchmark, on a later day and on a second provider (table below).
  2. Completion is not always enough. On DeepSeek, followers released within milliseconds of a leader truncated at one output token lost most reuse; on the same families, a leader that stopped on its own after the same single token kept it (full reuse in 60 of 60 families against 3 of 60). A fixed 200 ms wait gave full reuse in all 176 families tested, but on Volcengine Ark it was not enough, so the wait is calibrated to the provider and workload.
  3. Verification adds no resolved saving over waiting. In 50 matched comparisons of a gate with a control that waits without reading telemetry, no 95% upper bound lies below one; in a direct test the gate cost 1.426 [1.288, 1.579] times a 200 ms wait and took 1.671 [1.499, 1.864] times as long. The census indexes heterogeneous studies and is not a pooled equivalence test (Appendix E.3).
  4. Under engine cache pressure, warm-up adds computation. vLLM's own cache events show the mechanism: family prefixes discarded while their requests are still pending. A family cap reduces this where the excess is large.

Hosted warm-up beyond the short synthetic ablation (leader-only / work-conserving release; below 1 favors leader-only)

StudyCost [95% CI]Completion time [95% CI]
Submitted ablation, DeepSeek, ~1.2K-token prompts0.310 [0.294, 0.327]1.069 [0.973, 1.176]
NarrativeQA, 40 documents, 9K–25K tokens0.172 [0.162, 0.183]0.976 [0.886, 1.077]
16K synthetic prompts0.189 [0.164, 0.218]0.985 [0.812, 1.195]
Next-day replication, 20 new documents0.171 [0.154, 0.191]1.004 [0.840, 1.199]
QuALITY, 24 articles, 2.3K–4.6K tokens0.232 [0.210, 0.256]1.770 [1.582, 1.982]
Volcengine Ark, 50 documents randomized to policies0.476 [0.422, 0.537]1.809 [1.408, 2.326]
50% of each prompt shared, 200 ms wait0.595 [0.588, 0.602]1.864 [1.497, 2.321]

Ratios are geometric means with log-ratio t intervals, as in Table 1 of the revision; the submission row is recomputed from its per-block records. Where each family runs alone (QuALITY, Ark, shared fraction), nothing overlaps the serial leader and completion takes 1.8–2.0 times as long; where four families share eight workers, leader-only shows no resolved time penalty.

What changed in the paper. Figure 1 shows the system, and Table 1 gives the main results under one interval convention. Section 3.4 derives when verification can pay, and Table 2 measures its terms. Section 5.2 and Figure 2 are new: when one completed request is not enough. Section 5.4 adds the admission mechanism with native cache events, and Section 7 a three-step procedure. The appendix is organized by question: Table 3 maps each claim to its evidence; Appendix G covers hosted warm-up across workloads and providers, Appendix H when one completed request is not enough, Appendix I protocols and deviations, and Appendix J the remaining per-study tables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.