- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper targets branching LLM workflows, where multiple candidate histories are scored. CandiKV preserves candidate KV states across the selection, uses intermediate scores to prioritize KV retention and pre-selection execution. Experiments on reasoning and retrieval workloads report up to 90.7% reduction in continuation TTFT and 23.7–30.5% reduction in wasted GPU KV residency.
Critical strengths
- The paper is easy to follow.
- The evaluation demonstrates latency improvements over vLLM.
Critical weaknesses
- Completely missing related work discussion with closely related KV management systems. Existing systems such as KVFlow [1], Continuum [2], and TokenCake [3] already preserve and manage KV state across agent workflow rather than treating it as the request. These works are not discussed nor evaluated. This makes the contribution unclear, the paper should clarify what is fundamentally new.
[1] KVFlow: Efficient prefix caching for accelerating LLM-based multi-agent workflows
[2] Continuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live
[3] Tokencake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
-
The missing related work thus results in inconvincing baseline selection. PrefillOnly appears to be a weak baseline, as it is designed for prefill-only workloads. Also, how can PrefillOnly execute decoding involved tasks? Also, it is unclear how PrefillOnly executes tasks involving decoding. Given that the vLLM baseline uses automatic prefix caching, why does the continuation still need to recompute the branch-specific context? SGLang also provides Radix Attention and is not evaluated. Related workflow-aware KV management systems [1-3] for multi-agent systems should be quantitatively evaluated.
-
Insufficient details and end-to-end evaluation. The construction of the branching workflows is under-specified. For GSM8K and MATH-500, it is unclear whether the PRM can meaningfully distinguish the candidate solutions or how selection affects the task accuracy. For MS MARCO, important details such as the concrete workload used and candidate passage construction are missing. Moreover, the evaluation mainly reports TTFT. The paper should report end-to-end JCT across all workloads, which, as partly shown in Figure 6(c), appears to mariginal.
-
CandiKV's effectiveness depends on score arrival order. If early candidates receive high scores but are later displaced, their retained KV and speculative computation are wasted. If the final winner is scored late, the benefit may largely disappear. The paper does not evaluate winner arrival order or resources wasted in such cases. This brings another concern that should be evaluated: multiple candidates consumes additional KV memory, which may impact end-to-end perfformance under high QPS. Will the system be robust to such cases? I suggest evaluating these effects across different score arrival patterns and serving workload configurations.
Questions for authors
See Weaknesses. I do not expect that the concerns above can be fundamentally addressed through a rebuttal, but I believe the paper could be improved by addressing them in a future revision.
4 comments
W3.
-
CandiKV preserves the application's selection policy and optimizes its serving cost. On the math workloads, Qwen2.5-Math generates partial solutions that end at natural reasoning-step boundaries. Skywork PRM ranks them, and the top two continue. All systems use the same candidates and selected set. Quality is evaluated on the highest-ranked selected branch, while group JCT covers all selected continuations. A partial solution's score assesses an intermediate stage of reasoning, while the final answer also depends on the subsequent continuation. PRM and random selection achieve similar final accuracy under the original setting with short partial solutions. We therefore evaluate selector quality and serving performance separately.
-
The PRM achieves higher accuracy on the harder math benchmarks using the same candidate sets. The random baseline below is the exact expected accuracy of uniformly selecting one candidate from each set.
Dataset PRM Top1 accuracy Expected random Top1 accuracy AIME 2026 16.67% 8.75% AMC 2025 54.76% 42.86% These comparisons test selection quality and application behavior separately. In those comparisons, both systems achieve 94.6% accuracy on GSM8K 7B. With the MATH-500 continuation limit increased to 1,024 tokens, APC and CandiKV achieve 75.6% and 75.0%, respectively. These results support the serving improvements at comparable task quality.
-
The MS MARCO workload scores query–passage histories and continues from the selected history. The original serving trace takes eight passages per query from the public
top1000.dev.tsvpool in their released file order. Each candidate history combines the query with one passage, and a one-token relevance score selects one history for continuation. Relevance judgments (qrels) are used only for evaluation. In a separate quality check with BM25 top-8 candidates, APC and CandiKV both achieve MRR@8 of 0.1842 and Recall@1 of 13.2%. We report the two candidate-pool protocols separately, covering both the construction of the serving trace and ranking quality with retrieved candidates. We have added these workload and candidate passage construction details to the revised paper. -
The effect on group JCT depends on how much of the workflow is affected by continuation TTFT. CandiKV reduces recomputation and waiting after selection, while the same model performs the subsequent decoding. A large reduction in the initial wait can therefore account for a small fraction of a long task. We now report group JCT with natural EOS termination for every workload, measured from candidate-group submission until all selected continuations finish.
Workload 1.5B JCT p95 (s), APC / CandiKV 7B JCT p95 (s), APC / CandiKV GSM8K 2.905 / 2.915 7.302 / 7.268 MATH-500 3.184 / 3.232 10.400 / 10.422 ARC-Challenge 1.620 / 1.593 5.316 / 5.342 CommonsenseQA 1.444 / 1.344 4.434 / 4.385 NQ-Open 0.724 / 0.709 1.896 / 1.917 MS MARCO 0.687 / 0.706 1.414 / 1.403 JCT changes range from −6.95% to +2.73%, and measured throughput changes by at most 0.31%. We also run online workflows that include actual candidate construction and scoring. End-to-end JCT falls by 4.28% on NQ-Open and changes by +0.60% on GSM8K, whose candidate generation takes much longer. The reduction in continuation TTFT remains, while its effect on end-to-end JCT depends on the share of time spent at this stage.
W4.
-
CandiKV retains its gains when the final winner is scored late. The engine takes ownership of candidate state when candidate processing finishes, before the final selected set is known. Unscored candidates remain eligible, while incoming scores allow service to stop earlier for branches that can no longer be selected. We hold the final scores and selected set fixed and vary score arrival order: winner early, winner late, random order, displacement of early leaders, and simultaneous arrival. TTFT reductions range from 94.4% to 95.0% on GSM8K and from 97.5% to 98.4% on NQ-Open. With the winner scored late, the reductions remain 95.0% and 98.0%, respectively. Early scores make preparation more selective, while state continuity provides benefits across these arrival patterns.
-
A bounded preparation budget and prompt reclamation control the resources spent on unselected candidates. Preparation before the winner is known can include work that is eventually discarded, so we measure both unselected state residency and group JCT. By default, preparation stops after one boundary output token per branch. Once scores rule a branch out, further service stops and its state is reclaimed. Across 500 groups, unselected branches account for 3,000 boundary output tokens on GSM8K and 3,500 on NQ-Open. These are counts of actual boundary outputs, with the overall compute impact assessed through JCT and mixed traffic. Compared with keeping all scoring requests open, CandiKV reduces the residency of unselected branch-specific tails by 23.3% on NQ-Open.
-
We examine continuation latency and group JCT together as candidate width and KV pressure increase. Larger adds scoring work for all methods and increases the amount of potentially unused state, so TTFT alone is insufficient to assess the capacity cost. The following conditions also place the final winner late in the score arrival order.
Configuration GSM8K TTFT reduction / JCT change NQ-Open TTFT reduction / JCT change , load 0.9, KV budget 25% 95.75% / −4.21% 97.98% / +0.92% , load 1.1, KV budget 25% 94.54% / −3.43% 97.98% / +0.58% , load 0.9, KV budget 100% 93.32% / +1.28% 97.88% / +0.69% These are fixed-trace pressure measurements over 500 groups. We do not use the wide-group points to claim steady-state capacity. Under mixed traffic, handoff admits only a bounded burst of selected continuations before returning to regular requests. In the added experiments, regular-request TTFT increases by at most 0.80% and TPOT by at most 1.86%, while group JCT changes by −0.25% and −0.84% on the two workloads.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.
The new comparisons with related KV management systems and the additional experiments on score arrival order and KV pressure have addressed several of my original concerns. The clarification of the technical distinctions from prior work is also helpful.
However, my main concern remains the limited end-to-end benefits. Despite the significant reduction in continuation TTFT, the improvements in overall JCT and throughput remain marginal. This weakens the practical significance of the proposed system.
Another factor limiting my rating is the narrow applicability of the proposed workflow. I am not convinced that building dedicated serving infrastructure for such a specific branching pattern is reasonable. I would encourage the authors to evaluate more diverse and realistic agentic workloads to demonstrate broader applicability.
Overall, I acknowledge the improvements in the evaluation. I will raise my rating from 1 to 2. For future revisions, I would encourage deeper comparisons with closely related work and more comprehensive experiments to show end-to-end system-level improvements.
We thank the reviewer for the careful reading and the suggestions on related KV management systems and baseline comparisons. And we have revised the paper in light of this feedback.
W1.
CandiKV manages candidate histories jointly before the selected subset is known. KVFlow, Continuum, and TokenCake use workflow information to improve KV management. CandiKV focuses on candidates governed by the same selection decision. For example, eight partial solutions to one problem occupy the same position in the workflow, but only two continue. Their execution distance can be identical even as incoming scores change the value of retaining each history. KVFlow primarily uses execution distance to manage fixed prefixes, Continuum sets retention time based on tool-call duration, and TokenCake allocates capacity using agent criticality and tool events. These signals are useful, but do not directly express the changing trade-off among candidates as scores arrive. We have added these works to the related-work discussion and clarified this distinction.
Candidate groups carry retained KV and completed work into the selected continuations. Retention reduces recomputation, pre-selection execution prepares the first token, and handoff gives prepared branches prompt access to generation. CandiKV keeps the continuation boundary, KV state, and execution progress in the same branch record, then transfers that record when selection resolves. These decisions act on the same branch record.
W2.
APC can reuse branch-specific context, while CandiKV also preserves the continuation boundary and execution progress. Automatic prefix caching (APC) identifies reusable histories through token matching. A cache hit can still be followed by residual computation at the boundary and a wait for engine scheduling. In the strict control with pre-selection execution disabled, retaining the boundary KV reduces post-selection prefill by about 88.2% on both workloads. Handoff alone reduces TTFT by 32.1% on GSM8K and 49.1% on NQ-Open. Cache coverage, boundary reuse, and continuation admission therefore account for different costs.
The gains persist at low load and against workflow-aware baselines. The table normalizes continuation TTFT p95 to vLLM with APC enabled on the same workload at the same load. Lower is better. Parentheses indicate the load multiplier.
The improvements at low load show that high-load queueing does not fully explain the gap. Existing policies improve state availability, but the selected history still enters generation through the request path. CandiKV carries both its state and preparation progress across selection. This comparison evaluates KVFlow's GPU eviction policy, Continuum's publicly available retention and scheduling policy, and TokenCake's spatial management policy on a common vLLM backend. SGLang uses RadixAttention.
The PrefillOnly baseline uses terminal scoring followed by a new generation request. Scoring runs as a terminal one-token request, and the generation engine then decodes from the selected history. This control measures the cost of rebuilding state and starting decoding after scoring ends. The comparisons with APC, SGLang, LMCache, and the workflow-aware policies also support CandiKV's performance gains.