- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LatticeTTS, a deadline-aware serving layer for verifier-guided test-time scaling. Existing systems treat the search frontier as fixed work, so children whose verification finishes after the deadline waste computation. To address this, LatticeTTS keeps all selected parents and adjusts only the number of children per round, using a generator–verifier cost model and an online planner. Experimental results show a 5.75× speedup over vLLM, up to 18.36× higher throughput, and up to 16.7 points higher fixed-deadline accuracy.
Critical strengths
The core idea is simple and clear: keep the parents selected by the search policy and adapt only the number of children so that generation and verification fit within the time budget. If its advantage is sufficiently demonstrated experimentally, I believe this idea is valuable.
Critical weaknesses
No comparison with reducing the number of parents, which makes the accuracy comparison in Figure 5 unfair.
The paper’s second challenge is “how to reduce frontier work without changing the search decision,” and the method is built on the premise that parents should never be removed. However, the paper never compares this approach with the most naive way to reduce computation: reducing the number of parents. This missing comparison is also what makes the accuracy evaluation in Figure 5 unfair. Figure 5 fixes the number of parents and imposes a time constraint. Under this setting, LatticeTTS can shrink its workload to fit the budget, while the fixed-width baselines can neither reduce their parents nor adapt their frontier, so they inevitably miss the deadline. Thus, it is hard to distinguish whether the better results come from reducing computation to meet the deadline or advantage of preserving parents. To support the paper's premise, the authors should compare the speed-accuracy trade-offs of LatticeTTS against FastTTS with fewer parents or smaller initial widths.
Insufficient coverage of related work and baselines.
The paper positions prior systems as accelerating a fixed, known workload. However, several existing works already allocate TTS compute under latency budgets and are not discussed. Examples include selecting the inference strategy and compute per query while accounting for both generation and scoring latency [1], formulating latency-optimal TTS as a resource-allocation problem [2], and comparing verification schemes under latency budgets that include verifier cost [3]. Correspondingly, the experimental comparison is limited to vLLM and FastTTS, neither of which adapts its computation to the deadline. The authors should clarify how LatticeTTS differs from these works, for example in per-round online re-planning, modeling of discrete batch boundaries, and parent preservation. They should also demonstrate its advantage experimentally against these approaches.
[1] Huang et al., Latency and Token-Aware Test-Time Compute
[2] Wang et al., Faster and Better LLMs via Latency-Aware Test-Time Scaling
[3] Montgomery et al., Budget-aware Test-time Scaling via Discriminative Verification
Questions for authors
See weakness