- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes RelayTTS, a serving system specifically built for Test Time Scaling on resource-constrained devices. It first identifies system-level bottlenecks like redundant recomputation across siblings, model tranisition overhead, and memory contention. Then, it proposed novel methods like parent-complete score packing, fontier-aware joint planning, layer relay to accelerate TTS. The speedup over the SOTA baseline is proven across benchmarks.
Critical strengths
- The bottlenecks are clearly illustrated using figures. The paper explains well why repeated prefix computation and model weight transfer form the critical bottlenecks of the system.
- System design is intuitive. The paper designs the search structure with system performance, directly addressing the bottlenecks.
- Substantial gains. The results demonstrate strong benefits when complete model swapping is costly.
- Useful mechanism analysis. Search frequency, branching factor, and component ablations help explain when gains arise. Reporting the small contribution of overlap makes the performance attribution more credible.
Critical weaknesses
- What components of the method generalize to stronger GPU setting (like H100s)?
Questions for authors
See weaknesses.
2 comments
Thank you for the additional results. I encourage you to include the results, especially the stronger GPU results, in the updated version.
We thank the reviewer for the constructive suggestions on generalization to stronger GPU settings. We have revised the paper in light of this feedback.
W1.
Parent-complete score packing, joint planning, and layer relay generalize to stronger GPU settings, with the execution mode determined by available memory. More GPU memory can reduce model transition overhead, but sibling candidates still repeat shared-prefix computation. When both models and the search state fit in memory, RelayTTS uses a fully resident verifier and continues to exploit parent groups to reduce computation. When full residency is infeasible, it keeps the generator and part of the verifier weights resident for reuse in later scoring rounds, while layer relay supplies the remaining verifier layers on demand. The planner jointly selects score-pack capacity and verifier residency from the current frontier, using device-specific compute, transfer, and memory profiles to estimate execution cost.
RelayTTS retains end-to-end speedups on stronger GPU settings. On an A800 (80GB), we compare RelayTTS against FastTTS augmented with prefix caching, score caching, and partial verifier residency (FastTTS++) on AIME 2026.
For the 3B+7B pair, neither system transfers verifier-layer weights during scoring, yet parent-complete packing still reduces computation. With a 72B verifier, some weights remain resident across rounds and the remaining layers are relayed during scoring. Joint planning accounts for the number of score packs when selecting residency, reducing verifier-layer weight transfers by 60.8%–66.4%. The results cover execution both with and without verifier-layer weight transfers. The same execution interface can be used on platforms such as H100, with configurations selected using the corresponding device profiles.
RelayTTS also accelerates non-math workloads. Code search on HumanEval, MBPP sanitized, and LiveCodeBench still alternates generation with PRM scoring. RelayTTS jointly plans score packs and resident weights for the code-search frontier, and layer relay supplies nonresident layers during scoring. Parent-complete packing reduces repeated computation within each round, while persistent residency and fewer scans reduce model movement. The following results compare RelayTTS with FastTTS++ on the A800 using the 14B+72B pair.