acceptodds
Under review as a conference paper at ICLR 2027

SiftKV: Verifier State Reuse across Search Rounds for Efficient Test-Time Scaling

Abstract

Verifier-guided test-time scaling improves reasoning by expanding several candidate paths and retaining the most promising ones for further search. The selected paths return as the starting points of the next round, so the same reasoning histories continue across rounds. However, conventional process reward model (PRM) serving scores every extension as a new, complete request. It repeatedly reconstructs the histories of surviving paths and materializes KV for candidates that selection immediately removes. As the tree grows, an increasing share of verifier work repeats past computation instead of advancing the search. We introduce SiftKV, a PRM serving system that carries verifier state with the selected paths across rounds. Each surviving path resumes from a token-consistent continuation that preserves its causal KV and accumulated step rewards. New suffix state remains provisional until selection, after which only the selected suffixes join a tree that stores shared ancestry once. Verification can then extend the surviving paths without reconstructing their histories. Across reasoning workloads, search policies, and generator–verifier pairs, SiftKV delivers up to 50% lower end-to-end latency and 69% lower verifier time, together with up to 104% higher Precise Goodput.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
test-time scaling, LLM reasoning, process reward models, tree search, KV reuse

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3HLZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SiftKV, a kv cache system for prms in verifier-guided test-time scaling. Standard PRM serving treats each candidate as a fresh request, which wastes computation and kv cache space. SiftKV adds prm path continuations, frontier-guided state commitment, and tree-structured state sharing, delivering up to 50% lower end-to-end latency.

Critical strengths
  1. The paper is well-written and the presentation is generally of high quality.
  2. The experimental coverage is broad, spanning several datasets, search policies, and verifiers.
  3. SiftKV offers a system optimization that preserves the search decisions.
Critical weaknesses
  1. The comparison methods are not clearly described. For example, I don’t know which baselines make use of prefix caching for the verifier or how large the verifier’s kv cache is on gpu/cpu for each.
  2. It’s unclear how much of the reported gain can be explained by adding standard prefix caching for the verifier. The “retain all suffixes” ablation should compare against an lru eviction policy.
  3. It’s unclear whether the gains would persist under larger kv cache budgets, as the “frontier-guided state commitment” and “tree-structured state sharing” only provide an advantage when the verifier’s kv cache cannot fit in gpu memory.
  4. Table 1 shows that the gains are largest at high search budgets, but Table 2 shows that larger budgets do not improve accuracy in this setup. It’s possible that results with a strong generator and verifier would strengthen the practical motivation.
  5. The reward ledger component seems like a strawman, since the search orchestrator should already have the scores of the previous steps. Moreover, some prior work [1] only makes use of the latest score anyways.
  6. Figure 2 (left) is confusing. It’s not clear to me why root is grouped with the new suffix and not the history, or why the history is not dominating with more search rounds.

[1] “Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning.” International Conference on Learning Representations, 2025.

Questions for authors

See weaknesses.

6 comments

We thank the reviewer for the comments on caching baselines, state management, and practical relevance. And we have revised the paper in light of this feedback.

W1.

  • FastTTS and vLLM-TTS both enable automatic prefix caching for the generator and the verifier. Automatic Prefix Caching (APC) in vLLM manages paged KV on the GPU, allowing matching prefixes to be reused across requests. FastTTS also combines generator scheduling with prefix-aware optimizations. FlatPRM supports incremental PRM execution and reuses previous step scores, but stores each path continuation independently. LMCache provides hierarchical KV caching and offload with an 8 GiB cache. SiftKV stores persistent KV for selected paths in CPU memory with a 4 GiB budget. Its GPU arena holds active state during verification, and shared ancestry is stored once in the retained tree.

W2.

  • SiftKV further reduces verification cost when prefix caching is enabled and previous step scores are reused. We use vLLM-TTS++ as a strengthened baseline: it runs the PRM with vLLM's APC, reuses step scores already held by the search orchestrator, and adapts the STEP scoring interface. The following results use the 1.5B + 7B model pair on MATH-500. Speedups are for SiftKV over the system in each row.

    nSystemEnd-to-end latency (s)Generation time (s)Verifier time (s)End-to-end speedupVerifier speedup
    128FastTTS89.6830.3856.041.53×2.24×
    128vLLM-TTS++72.6629.7939.241.24×1.57×
    128SiftKV58.4629.7925.011.00×1.00×
    256FastTTS159.5745.48107.671.78×2.80×
    256vLLM-TTS++130.7245.2478.871.46×2.05×
    256SiftKV89.4344.8138.521.00×1.00×

    APC and reuse of cached step scores already reduce baseline overhead. SiftKV still achieves 1.24–1.46× end-to-end speedup and 1.57–2.05× verifier speedup over this baseline. SiftKV uses search selection to carry the KV and reward state of selected paths into the next round and organizes subsequent verification around shared ancestry. This retains the history needed in the next round, releases branches that no longer continue, and reduces the repeated preparation and restoration of the same ancestry for different paths.

  • Frontier-guided state commitment reduces unnecessary retention and state reconstruction costs that remain with LRU. The LRU comparison fixes candidate tokens, candidate order, and the selected frontier. All variants use the same exact scoring path and a capacity of 4,096 tokens, with the search orchestrator retaining previous step scores throughout. LRU and Store-all are limited to 4 GiB of distinct underlying tensor storage. The times below include verification and state management.

    State retention policyMean verifier time (s)Time relative to frontier retention
    Retain the selected frontier22.891.00×
    LRU eviction53.782.35×
    Retain all suffixes72.373.16×
    Recompute survivors43.651.91×

    Candidates that have just been scored are all recently accessed, so recency alone does not tell LRU which branches will continue. Search selection directly identifies the paths needed in the next round. Retaining all suffixes stores more state with no further use, while recomputing survivors rematerializes state needed immediately. Retaining the selected frontier reduces both costs and achieves 2.35× verifier speedup over LRU. The maximum absolute score difference is 0 for all variants.

W3.

  • Cross-round continuation and ancestry sharing remain useful when the KV cache fits entirely on the GPU. Full GPU residency removes the transfer cost of host staging, but selected paths still provide context for later verification, and sibling paths still share ancestry. Section 3.1 continues verification from the state retained with each selected path, while Section 3.3 shares ancestry storage and organizes subsequent accesses. The relation in Equation (6) between storage in a shared tree and storage as independent paths does not depend on whether KV resides in CPU or GPU memory. With full residency, these mechanisms primarily affect computation reuse and GPU memory usage; when host staging is needed, they can also reduce ancestry restoration traffic. In the model-pair comparison, we keep the 1.5B generator and replace the 7B PRM with a 1.5B PRM. All results below use n = 256.

    Model pairEnd-to-end speedup over FastTTSVerifier speedup over FastTTSEnd-to-end speedup over vLLM-TTS++Verifier speedup over vLLM-TTS++
    1.5B + 7B1.78×2.80×1.46×2.05×
    1.5B + 1.5B1.28×3.18×1.29×3.29×

    Substantial verifier speedups remain with the smaller PRM. Generation accounts for a larger share of total runtime, so less of the verifier speedup translates to end-to-end speedup, consistent with Figure 6. Table 4 also shows that end-to-end latency remains nearly unchanged when the tested GPU arena grows from 256 to 512 pages.

W4.

  • With a stronger reasoning generator, SiftKV still reduces search cost while preserving answer quality. We use Qwen3-1.7B in thinking mode with a 7B PRM. At n = 128, FastTTS and SiftKV generate identical token sequences and achieve the same answer accuracy. SiftKV achieves 1.17× end-to-end speedup and 1.72× verifier speedup. The verifier-time savings therefore also apply with a stronger generator and a moderate search budget.

  • Table 2 checks quality preservation under a fixed search configuration, while Table 1 measures the execution cost of producing those results. Changing n changes the candidate set and PRM ranking, so PRM-ranked pass@k need not increase monotonically with n. At each fixed budget, FastTTS and SiftKV have the same pass@k but different verification costs. The comparison in thinking mode likewise fixes the inference strategy and budget: runtime falls while generated content and answer quality remain unchanged, demonstrating the practical role of the system optimization.

W5.

  • The search orchestrator's existing step scores can be reused directly in PRM continuations; the reward ledger keeps them consistent with the computation state. A continuation in Section 3.1 records the reusable token prefix, KV, and reward marker positions. If retokenization occurs at the parent–child join, or the marker of an unfinished step moves, KV and reward state must roll back to the same valid position before processing the changed suffix. The ledger records this correspondence so that scores already computed can be reused directly.

  • State reuse across rounds remains effective when only the latest step score is used. We compare last and mean aggregation with all methods retaining the orchestrator's previously computed step scores. The table reports SiftKV's speedups over the corresponding baselines.

    AggregationnEnd-to-end speedup over FastTTSVerifier speedup over FastTTSEnd-to-end speedup over vLLM-TTS++Verifier speedup over vLLM-TTS++
    last1281.53×2.24×1.24×1.57×
    last2561.78×2.80×1.46×2.05×
    mean1281.56×2.29×1.30×1.72×
    mean2561.73×2.74×1.37×1.98×

    last uses only the latest score, but verifying the new step still requires its preceding causal context. mean uses the accumulated state of earlier rewards. Under either rule, selected paths can reuse their KV and share common ancestry. The gains persist under both aggregation rules, without requiring baselines to recompute scores already stored by the orchestrator.

W6.

  • The root contains the fixed problem statement and template and is reused as shared context in SiftKV. In Figure 2, history refers specifically to reasoning steps generated in earlier rounds. The figure groups the root with the newly generated suffix and shows the accumulated reasoning history separately. This categorization follows the source of the input and does not mean that the root is newly generated. Execution first creates a root continuation that subsequent paths reuse.

  • History replay accumulates as paths grow, while its share in each round also depends on the new content generated in that round. The number of active candidates and the length of generated steps vary across rounds. The final round also removes the step-level stopping condition and generates the rest of the answer, making the new suffix substantially longer. Thus, even as path histories grow, their share of the actual work need not increase monotonically from round to round.

Reviewer_Gi5wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper discusses the inference efficiency of verifier guided test-time scaling systems. The claim is that conventional PRM serving scores every continuation as a completely new request and therefore repeatedly reconstructs the prefill part. The authors propose a system that carries verifier state with the selected paths across rounds thus saving a major portion of recomputation delivering upto 50% lower end-to-end latency and 69% lower verifier time.

Critical strengths
  1. The reasoning behind the method is sound and very practical. In PRM-guided test time scaling, the reducer knows which candidate states will be needed in later rounds and this information is used to decide which cache should be stored and what can be evicted instead of relying on generic prefix caching. The design handles details like retokenization at the parent-child join and moved step markers.

  2. The evaluation covers different aspects of the method and shows performance over multiple datasets, search policies (Fig. 5), per-mechanism ablations (Fig. 7), sweeps over suffix lengths (Fig 8) and capacity sensitivity (Table 4). The result numbers are also strong showing 2.5-3.2x verifier time reduction over FastTTS at large budgets.

Critical weaknesses
  1. All experiments are performed on a single RTX 4090 GPU with only 24GB of HBM. Since SiftKV stores persistent verifier state in host memory and stages ancestry back to the GPU, I believe its gains depend on memory capacity and transfer costs. It is unclear how much benefit remains on GPUs where most of the cache can reside in the HBM.

  2. It’s unclear how relevant this approach is with advent of reasoning models. The experiments only use Qwen2.5-Math-Instruct as the generator with separate PRMs. Reasoning models today often spend extra compute on one long chain of thought instead of branching many times under a verifier. If that’s how the extra compute gets used, it’s not clear SiftKV still saves much. The authors could test a reasoning model, or compare against simply letting the model reason longer at the same compute and answer quality. That would show how widely the method applies.

  3. Reading the paper, I feel like key experimental details are missing. The paper doesn’t doesnt discuss much about how big each baseline’s cache is or where it is stored. It doesn’t say how many problems or runs each result uses, or how much the timings vary from run to run. This matters because the speedups depend on the shape of the search and how much memory is available. The discussions about these details can be improved.

Questions for authors
  1. Do vLLM-TTS and FastTTS turn on automatic prefix caching for the PRM? Could you add a baseline that runs the PRM with SGLang RadixAttention (or vLLM’s automatic prefix caching), where the search controller also keeps rewards it has already computed for earlier steps? How much of SiftKV’s speedup is left against that baseline?

  2. Does SiftKV still help when the verifier’s KV cache fits entirely on the GPU, so nothing needs to be staged on the host? Two examples would be an 80 GB GPU or a smaller PRM. Also, when the generator and verifier share one 24 GB GPU, how does the generator’s own memory use affect the space left for the verifier?

  3. I assume the Precise Goodput and latency results compare systems that produce the same search outputs. Can you confirm that every system gets exactly the same generator samples (same seeds)? Otherwise, differences in generation time could explain part of the end-to-end gap.

  4. Could you add results with a reasoning model? That would show whether the method still matters for newer models.

3 comments

We thank the reviewer for the comments on hardware applicability, reasoning models, and experimental setup. And we have revised the paper in light of this feedback.

Q1.

  • SiftKV retains its gains when automatic prefix caching is enabled and previous step rewards are cached. vLLM-TTS++ runs the PRM with vLLM's Automatic Prefix Caching (APC) and retains step rewards already held by the search controller. The STEP scoring interface is also adapted so that cached prefixes can be used when scoring subsequent markers. FastTTS and vLLM-TTS both enable APC for the generator and the verifier. The following comparison uses the 1.5B + 7B model pair.

    nvLLM-TTS++ end-to-end (s)SiftKV end-to-end (s)vLLM-TTS++ verifier (s)SiftKV verifier (s)End-to-end speedupVerifier speedup
    12872.6658.4639.2425.011.24×1.57×
    256130.7289.4378.8738.521.46×2.05×

    APC reuses matching token prefixes, and the reward cache retains scores already computed. SiftKV also uses search selection to determine which candidate states will be needed later, carries the KV and reward state of selected paths into the next round, and organizes execution around shared ancestry. As search goes deeper, more history needs to be continued, making state reuse more valuable. The LRU comparison on fixed requests further shows that frontier-guided retention reduces verification and state management overhead under the same scoring path and cache capacity.

W1 / Q2.

  • Reusing the histories of selected paths also reduces verification cost for a smaller PRM. We use the 1.5B + 1.5B model pair, with the generator and PRM co-resident on the GPU. The following results use n = 256. Speedups are for SiftKV over the system in each row.

    SystemEnd-to-end latency (s)Generation time (s)Verifier time (s)End-to-end speedupVerifier speedup
    FastTTS57.1035.1817.161.28×3.18×
    vLLM-TTS++57.3334.4217.751.29×3.29×
    SiftKV44.5034.235.401.00×1.00×

    With the smaller verifier, generation accounts for a larger share of end-to-end time, but later verification still needs the histories of selected paths. SiftKV continues from retained KV and shares ancestry among sibling candidates, reducing verification cost. The runtime breakdown also explains the difference between verifier speedup and end-to-end speedup.

  • Generator weights and generator KV share the GPU memory budget with the verifier cache. Both models co-reside on the same 24 GB GPU in BF16. The memory budget includes model weights, generator KV, active verifier KV, and workspace. For the smaller PRM, APC uses about 1 GiB of GPU KV with 16-token blocks and reuses recovered marker scores. SiftKV retains persistent KV in host memory with a 4 GiB budget, while the GPU arena holds active verification state. Full KV residency avoids host restores, while computation from earlier rounds and shared ancestry can still be reused. When state spans memory tiers, sharing further reduces transfers.

W2 / Q4.

  • Thinking mode and verifier-guided search both use additional test-time compute to improve answer quality and can be combined. Thinking mode mainly devotes computation to reasoning along a single chain of thought, while verifier-guided search explores different solution paths by expanding, comparing, and retaining candidates. The paper uses an instruction-tuned model to generate candidates, with the extra computation spent on candidate exploration and verifier-guided selection.

  • State reuse across rounds also reduces search time with a generator in thinking mode. We use Qwen3-1.7B in thinking mode with a 7B PRM at n = 128. Speedups are for SiftKV over the system in each row.

    SystemEnd-to-end latency (s)Generation time (s)End-to-end speedupVerifier speedup
    FastTTS206.11120.361.17×1.72×
    vLLM-TTS++190.66117.861.08×1.40×
    SiftKV176.13118.271.00×1.00×

    FastTTS and SiftKV achieve the same answer accuracy. The generation mode changes the amount of generation work, but selected paths still return in the next round, and sibling candidates still share their parent's verification context. SiftKV can continue to reuse these histories and shared prefixes. Generation accounts for about two-thirds of SiftKV's total time in this configuration, so the end-to-end speedup is smaller than for the main model pair, consistent with the runtime analysis in Figure 6 and Appendix A.7.

W3 / Q3.

  • The paper compares systems under matched model and search settings, with results averaged over the full evaluation datasets. We evaluate each dataset in full and average results over three random seeds. For each model pair, search policy, and budget, systems use the same generation sampling parameters and stopping conditions. GPU KV, persistent CPU KV, and scoring workspace are configured separately; the shared GPU memory budget includes model weights, generator KV, and verifier execution state.

  • SiftKV preserves the generation and search selection rules, while the paper evaluates answer quality and execution cost separately. SiftKV changes how PRM state is reused and executed. The generator still produces candidates through the same sampling interface, and path scores are still formed with the original aggregation rule and used by the same search reducer. In Table 2, the main FastTTS/SiftKV comparison has the same PRM-ranked pass@k at every budget. Online comparisons across inference backends do not force token-by-token identical generations, so generation and verifier times are reported separately to distinguish their contributions to end-to-end latency.

  • The paper measures verification gains with separate timings for each stage. Figure 6 breaks total latency into generation, verification, and other overhead, while Table 1 reports PRM execution time separately, showing 2.52–3.18× verifier speedup over FastTTS at the largest budget. The verifier-time savings are therefore measured independently. Precise Goodput accounts for both effective completion tokens and beam completion time; avoided PRM recomputation does not count as additional effective completion tokens.

I acknowledge the rebuttal and it address my major concerns, although I agree with Reviewer YTUZ on novelty concerns, I am leaning towards acceptance and thus keeping my positive score.

Reviewer_YTUZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the inference efficiency of Process Reward Model (PRM) in the context of PRM-guided test-time scaling. By observing the shared computation among different verification steps, the work introduces PRM continuatiion and frontier-guided tree executor. The evaluation shows 50% reduction in end-to-end latency and 69% reduction in verification time

Critical strengths
  1. Evaluation includes multiple datasets against different SOTA approaches such as vLLM-TTS, FlatPRM, LMCache and FastTTS
  2. The motivation is clear.
Critical weaknesses
  1. Novelty concern: the overall idea can be summarize as scheduling + memory optimization for PRM. All these optimizations, you can find in other serving scenarios, but this paper seems to apply these into PRM + TTS. What is the unique opportunity you think in optimizing PRM + TTS? More specifically, the idea of 3.1, 3.2 and 3.3 I can more or less see in other inference engine for LLM inference (non-TTS setting).
  2. The insight of PRM cost is not new. Is there any insightful observation? Most of insights are already in prior work that listed as baseline in this paper.
  3. Section 4.4 is a bit confusing to me. The paper propose 3.1, 3.2 and 3.3 optimizations, could you pls have clean ablation study to breakdown the effect of each of them? Do you mean "No Ancestry sharing" in Figure-7 corresponding to 3.1? Just make this more clear.
  4. Why Figure-5 only compare against one baseline? If other baselines are not applicable, please explain in the text.
Questions for authors

See weakness.

3 comments

We thank the reviewer for the comments on the contribution, ablations, and comparisons across search policies. And we have revised the paper in light of this feedback.

W1 / W2.

  • SiftKV uses search selection to manage PRM state, reducing unnecessary retention and history recomputation on returning paths. vLLM/PagedAttention and SGLang/RadixAttention already support KV prefix sharing across requests. Automatic Prefix Caching (APC) reuses computation by matching tokens. SiftKV also receives the next frontier from the search reducer, so state retention follows the paths selected for later expansion. The key observation in Figure 2 is that 84–87% of candidate KV has no use in the next round. All candidates must still be scored, but only part of the resulting state needs to be retained. Search selection identifies that state directly.

  • PRM continuation requires KV and step rewards to remain aligned. Once the controller has stored previous step scores, the verifier still needs to determine which tokens, markers, and KV state those scores correspond to. Appending a step may cause retokenization, and the marker of an unfinished step may move. Section 3.1 keeps these states on the same reusable prefix before computing the changed suffix. FlatPRM already provides incremental execution and reuses previous step rewards. SiftKV further manages these continuations according to the frontier and organizes their execution around a shared tree, retaining state that will be used later.

  • Frontier-guided retention works with hierarchical caching and shared-prefix attention to further reduce verification cost across rounds. LMCache primarily improves KV access, restoration, and transfer across storage tiers, while Hydragen reduces redundant memory reads through shared-prefix attention. SiftKV uses search selection to decide which scored states need to persist across rounds and organizes later accesses around shared ancestry. Execution uses FlashInfer's shared-prefix attention. FastTTS already optimizes generation scheduling and prefix reuse for TTS; SiftKV further carries forward the computation state of a separate PRM across verification rounds. The comparisons with APC with cached step rewards and with LRU at the same capacity test the gains beyond these mechanisms.

W3.

  • The three components address recomputation across rounds, candidate-state retention, and repeated ancestry restoration. In Figure 7, "No ancestry sharing" corresponds to Section 3.3, while "No reward ledger" removes the reward-state component of Section 3.1. The ablations disable continuation, KV reuse, and the ledger together for Section 3.1, replace the retention policy in Section 3.2, and disable ancestry sharing alone for Section 3.3. Step rewards already held by the search controller are retained throughout.

    All variants use the same fixed scoring requests, exact scoring path, and capacity of 4,096 tokens. LRU and Store-all are limited to 4 GiB of distinct underlying tensor storage. The full configuration under these execution settings is the reference.

    SectionConfigurationMean verifier time (s)Verifier time relative to the full configuration
    3.1–3.3Full configuration22.891.00×
    3.1Disable continuation, KV reuse, and the reward ledger46.652.04×
    3.2Retain all suffixes72.373.16×
    3.2Recompute survivors43.651.91×
    3.2LRU eviction53.782.35×
    3.3No ancestry sharing60.682.65×

    The Section 3.1 comparison tests whether retained computation state can be reused. The Section 3.2 comparisons test the costs of storing unused state, rebuilding selected state, and eviction based on access history. The Section 3.3 comparison tests repeated restoration of shared ancestry. The maximum absolute score difference is 0 throughout this fixed-input comparison, so the time differences reflect execution and state management costs.

W4.

  • Figure 5 uses FastTTS as the common TTS baseline to assess verifier-state reuse across search policies. Figure 4 compares the main systems. Figure 5 varies the search policy while keeping the generation and search framework the same for FastTTS and SiftKV, to test whether verifier-state optimization applies to different expansion and selection rules. FastTTS is designed for TTS and includes generation and prefix-aware scheduling optimizations. These features make it the most directly comparable baseline for this paired comparison.

  • The other baselines also apply to these search policies, and SiftKV retains its gains in each comparison. We compare vLLM-TTS, FlatPRM, and LMCache and compute end-to-end speedups from mean latencies over the combined AIME 2024 and AIME 2025 results. Entries are SiftKV's speedups over the baseline in each column.

    Search policynOver FastTTSOver vLLM-TTSOver FlatPRMOver LMCache
    DVTS1281.37×1.84×1.38×1.69×
    DVTS2561.48×2.01×1.43×1.78×
    Dynamic Branching1281.56×2.03×1.61×1.99×
    Dynamic Branching2561.78×2.35×1.83×2.05×
    VG-Search1281.62×1.81×1.49×1.88×
    VG-Search2561.78×1.95×1.56×2.10×

    DVTS changes retention within partitions, Dynamic Branching changes the number of branches, and VG-Search changes the verification interval. All of them produce paths that continue in the next round. SiftKV reuses PRM state along these selected paths and organizes subsequent verification around shared ancestry, allowing the same PRM state management interface to support these different search rules.

Thanks for your efforts.

I am still not fully concerned by the novelty side, therefore, I keep my score.

But I am fine if the final acceptance (no strong objection) if other reviewers are willing to accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and the revision

Dear Area Chair,

Thank you for handling our submission. We also thank the reviewers for their careful reading and constructive suggestions, which helped us strengthen the evaluation and clarify SiftKV's contribution and scope. As the discussion period draws to a close, we would like to summarize the reviews and the evidence we added in response.

The main concerns fell into four groups.

  1. Caching baselines and contribution (3HLZ, Gi5w, YTUZ). The reviewers asked how much benefit remains beyond prefix caching and reuse of previous step rewards, and what opportunity is specific to PRM-guided search.
  2. Full GPU residency and hardware applicability (3HLZ, Gi5w). The reviewers asked whether the gains persist when the full search tree fits in GPU memory, including with a smaller PRM or a larger GPU.
  3. Reasoning models and answer quality (3HLZ, Gi5w). The reviewers asked whether SiftKV remains useful with a reasoning generator and how the latency results relate to search quality.
  4. Mechanism analysis and experimental clarity (3HLZ, Gi5w, YTUZ). The reviewers requested clearer ablations, an LRU comparison, broader baseline comparisons across search policies, and clarification of the experimental setup.

We addressed the reviewers' main concerns through additional experiments, measurements and clarifications.

  1. Added a direct prefix-caching baseline and clarified the additional contribution. We added vLLM-TTS++, which runs the PRM with vLLM's Automatic Prefix Caching (APC) and reuses step rewards already held by the search controller. With the 1.5B + 7B pair, SiftKV achieves 1.24–1.46× end-to-end and 1.57–2.05× verifier speedup over this baseline. We clarified that search selection supplies the identities of paths needed in the next round. SiftKV uses this information to retain their verifier continuations and release rejected branches, while keeping tokens, causal KV, and reward markers consistent. We also explained how this use of search selection complements prefix caching in vLLM/SGLang, hierarchical KV reuse in LMCache, and shared-prefix attention in Hydragen.

  2. Tested full GPU residency and extended the evaluation to an 80 GB GPU. In response to 3HLZ's follow-up, we increased the verifier cache so that vLLM-TTS++ retains the full search tree on the GPU. With the 1.5B + 1.5B pair at n=256, SiftKV still achieves 1.15× end-to-end and 1.77× verifier speedup. We also evaluated the 3B + 7B pair on an A800 (80 GB) with full GPU residency, obtaining 1.19× end-to-end and 1.48× verifier speedup over vLLM-TTS++. We explained why computation savings remain: token-level continuation reduces replay at partial cache blocks, a shared token trie computes common prefixes introduced within the same scoring batch once, and shared-prefix attention reduces repeated reads of resident ancestry.

  3. Added results with a reasoning model and clarified answer quality. With Qwen3-1.7B in thinking mode and a 7B PRM at n=128, SiftKV achieves 1.17× end-to-end and 1.72× verifier speedup over FastTTS with the same answer accuracy. Thinking spends additional computation within a generated path, while verifier-guided search allocates computation across candidate paths. Selected paths still return with reusable verification context. We also clarified the quality comparison: FastTTS and SiftKV have identical PRM-ranked pass@k at each budget in the main evaluation.

  4. Added controlled ablations and broader comparisons across search policies. We fixed candidate tokens, candidate order, and the selected frontier, retained the controller's historical rewards throughout, and mapped each ablation to Sections 3.1–3.3. Frontier-guided retention achieves 2.35× verifier speedup over LRU under the same scoring path and memory budget. All variants produce identical scores. We also compared FastTTS, vLLM-TTS, FlatPRM, and LMCache under DVTS, Dynamic Branching, and VG-Search, with end-to-end speedups ranging from 1.37× to 2.35×. We clarified cache placement and reported generation and verification times separately to distinguish their contributions to end-to-end latency.

We believe the additional experiments and clarifications strengthen the case for accepting SiftKV. We thank the reviewers and the Area Chair again for their time and careful consideration.

Best regards,

Authors of Submission 23602

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.