- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SiftKV, a kv cache system for prms in verifier-guided test-time scaling. Standard PRM serving treats each candidate as a fresh request, which wastes computation and kv cache space. SiftKV adds prm path continuations, frontier-guided state commitment, and tree-structured state sharing, delivering up to 50% lower end-to-end latency.
Critical strengths
- The paper is well-written and the presentation is generally of high quality.
- The experimental coverage is broad, spanning several datasets, search policies, and verifiers.
- SiftKV offers a system optimization that preserves the search decisions.
Critical weaknesses
- The comparison methods are not clearly described. For example, I don’t know which baselines make use of prefix caching for the verifier or how large the verifier’s kv cache is on gpu/cpu for each.
- It’s unclear how much of the reported gain can be explained by adding standard prefix caching for the verifier. The “retain all suffixes” ablation should compare against an lru eviction policy.
- It’s unclear whether the gains would persist under larger kv cache budgets, as the “frontier-guided state commitment” and “tree-structured state sharing” only provide an advantage when the verifier’s kv cache cannot fit in gpu memory.
- Table 1 shows that the gains are largest at high search budgets, but Table 2 shows that larger budgets do not improve accuracy in this setup. It’s possible that results with a strong generator and verifier would strengthen the practical motivation.
- The reward ledger component seems like a strawman, since the search orchestrator should already have the scores of the previous steps. Moreover, some prior work [1] only makes use of the latest score anyways.
- Figure 2 (left) is confusing. It’s not clear to me why root is grouped with the new suffix and not the history, or why the history is not dominating with more search rounds.
[1] “Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning.” International Conference on Learning Representations, 2025.
Questions for authors
See weaknesses.
6 comments
W3.
-
Cross-round continuation and ancestry sharing remain useful when the KV cache fits entirely on the GPU. Full GPU residency removes the transfer cost of host staging, but selected paths still provide context for later verification, and sibling paths still share ancestry. Section 3.1 continues verification from the state retained with each selected path, while Section 3.3 shares ancestry storage and organizes subsequent accesses. The relation in Equation (6) between storage in a shared tree and storage as independent paths does not depend on whether KV resides in CPU or GPU memory. With full residency, these mechanisms primarily affect computation reuse and GPU memory usage; when host staging is needed, they can also reduce ancestry restoration traffic. In the model-pair comparison, we keep the 1.5B generator and replace the 7B PRM with a 1.5B PRM. All results below use n = 256.
Model pair End-to-end speedup over FastTTS Verifier speedup over FastTTS End-to-end speedup over vLLM-TTS++ Verifier speedup over vLLM-TTS++ 1.5B + 7B 1.78× 2.80× 1.46× 2.05× 1.5B + 1.5B 1.28× 3.18× 1.29× 3.29× Substantial verifier speedups remain with the smaller PRM. Generation accounts for a larger share of total runtime, so less of the verifier speedup translates to end-to-end speedup, consistent with Figure 6. Table 4 also shows that end-to-end latency remains nearly unchanged when the tested GPU arena grows from 256 to 512 pages.
W4.
-
With a stronger reasoning generator, SiftKV still reduces search cost while preserving answer quality. We use Qwen3-1.7B in thinking mode with a 7B PRM. At n = 128, FastTTS and SiftKV generate identical token sequences and achieve the same answer accuracy. SiftKV achieves 1.17× end-to-end speedup and 1.72× verifier speedup. The verifier-time savings therefore also apply with a stronger generator and a moderate search budget.
-
Table 2 checks quality preservation under a fixed search configuration, while Table 1 measures the execution cost of producing those results. Changing n changes the candidate set and PRM ranking, so PRM-ranked pass@k need not increase monotonically with n. At each fixed budget, FastTTS and SiftKV have the same pass@k but different verification costs. The comparison in thinking mode likewise fixes the inference strategy and budget: runtime falls while generated content and answer quality remain unchanged, demonstrating the practical role of the system optimization.
W5.
-
The search orchestrator's existing step scores can be reused directly in PRM continuations; the reward ledger keeps them consistent with the computation state. A continuation in Section 3.1 records the reusable token prefix, KV, and reward marker positions. If retokenization occurs at the parent–child join, or the marker of an unfinished step moves, KV and reward state must roll back to the same valid position before processing the changed suffix. The ledger records this correspondence so that scores already computed can be reused directly.
-
State reuse across rounds remains effective when only the latest step score is used. We compare last and mean aggregation with all methods retaining the orchestrator's previously computed step scores. The table reports SiftKV's speedups over the corresponding baselines.
Aggregation n End-to-end speedup over FastTTS Verifier speedup over FastTTS End-to-end speedup over vLLM-TTS++ Verifier speedup over vLLM-TTS++ last 128 1.53× 2.24× 1.24× 1.57× last 256 1.78× 2.80× 1.46× 2.05× mean 128 1.56× 2.29× 1.30× 1.72× mean 256 1.73× 2.74× 1.37× 1.98× last uses only the latest score, but verifying the new step still requires its preceding causal context. mean uses the accumulated state of earlier rewards. Under either rule, selected paths can reuse their KV and share common ancestry. The gains persist under both aggregation rules, without requiring baselines to recompute scores already stored by the orchestrator.
W6.
-
The root contains the fixed problem statement and template and is reused as shared context in SiftKV. In Figure 2, history refers specifically to reasoning steps generated in earlier rounds. The figure groups the root with the newly generated suffix and shows the accumulated reasoning history separately. This categorization follows the source of the input and does not mean that the root is newly generated. Execution first creates a root continuation that subsequent paths reuse.
-
History replay accumulates as paths grow, while its share in each round also depends on the new content generated in that round. The number of active candidates and the length of generated steps vary across rounds. The final round also removes the step-level stopping condition and generates the rest of the answer, making the new suffix substantially longer. Thus, even as path histories grow, their share of the actual work need not increase monotonically from round to round.
We thank the reviewer for the comments on caching baselines, state management, and practical relevance. And we have revised the paper in light of this feedback.
W1.
W2.
SiftKV further reduces verification cost when prefix caching is enabled and previous step scores are reused. We use vLLM-TTS++ as a strengthened baseline: it runs the PRM with vLLM's APC, reuses step scores already held by the search orchestrator, and adapts the STEP scoring interface. The following results use the 1.5B + 7B model pair on MATH-500. Speedups are for SiftKV over the system in each row.
APC and reuse of cached step scores already reduce baseline overhead. SiftKV still achieves 1.24–1.46× end-to-end speedup and 1.57–2.05× verifier speedup over this baseline. SiftKV uses search selection to carry the KV and reward state of selected paths into the next round and organizes subsequent verification around shared ancestry. This retains the history needed in the next round, releases branches that no longer continue, and reduces the repeated preparation and restoration of the same ancestry for different paths.
Frontier-guided state commitment reduces unnecessary retention and state reconstruction costs that remain with LRU. The LRU comparison fixes candidate tokens, candidate order, and the selected frontier. All variants use the same exact scoring path and a capacity of 4,096 tokens, with the search orchestrator retaining previous step scores throughout. LRU and Store-all are limited to 4 GiB of distinct underlying tensor storage. The times below include verification and state management.
Candidates that have just been scored are all recently accessed, so recency alone does not tell LRU which branches will continue. Search selection directly identifies the paths needed in the next round. Retaining all suffixes stores more state with no further use, while recomputing survivors rematerializes state needed immediately. Retaining the selected frontier reduces both costs and achieves 2.35× verifier speedup over LRU. The maximum absolute score difference is 0 for all variants.