acceptodds
Under review as a conference paper at ICLR 2027

LatticeTTS: Deadline-Aware Frontier Compilation for Verifier-Guided Test-Time Scaling

Abstract

Verifier-guided test-time scaling (TTS) improves the reasoning of compact models by expanding and scoring several candidate paths. Under a time budget, a candidate helps only if it is generated and verified before the deadline. Existing serving systems make each stage faster but treat the requested frontier as a fixed amount of work. Our measurements show this cost is discontinuous, since one more child can push either stage across a batch boundary and raise its latency by up to 26% while leaving the other almost unchanged. We present LatticeTTS, a serving layer that keeps every selected parent and decides only how many children to run beneath each, without retraining or changing the ranking rule. A joint service lattice records the generator, verifier, and key–value (KV) cache state of each round, and an online planner moves along a nested sequence of child counts, admitting only children whose complete cycle fits the remaining deadline. Across three model pairs, three reasoning benchmarks, and three search policies, LatticeTTS achieves a 5.75× geometric-mean end-to-end speedup, up to 18.36× higher throughput under concurrent load, and up to 16.7 percentage points higher fixed-deadline accuracy under tight budgets.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Test-Time Scaling, Verifier-Guided Reasoning, Edge Inference, Deadline-Aware Scheduling, LLM Inference Serving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DKtbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatticeTTS, a deadline-aware serving layer for verifier-guided test-time scaling. Existing systems treat the search frontier as fixed work, so children whose verification finishes after the deadline waste computation. To address this, LatticeTTS keeps all selected parents and adjusts only the number of children per round, using a generator–verifier cost model and an online planner. Experimental results show a 5.75× speedup over vLLM, up to 18.36× higher throughput, and up to 16.7 points higher fixed-deadline accuracy.

Critical strengths

The core idea is simple and clear: keep the parents selected by the search policy and adapt only the number of children so that generation and verification fit within the time budget. If its advantage is sufficiently demonstrated experimentally, I believe this idea is valuable.

Critical weaknesses

No comparison with reducing the number of parents, which makes the accuracy comparison in Figure 5 unfair.

The paper’s second challenge is “how to reduce frontier work without changing the search decision,” and the method is built on the premise that parents should never be removed. However, the paper never compares this approach with the most naive way to reduce computation: reducing the number of parents. This missing comparison is also what makes the accuracy evaluation in Figure 5 unfair. Figure 5 fixes the number of parents and imposes a time constraint. Under this setting, LatticeTTS can shrink its workload to fit the budget, while the fixed-width baselines can neither reduce their parents nor adapt their frontier, so they inevitably miss the deadline. Thus, it is hard to distinguish whether the better results come from reducing computation to meet the deadline or advantage of preserving parents. To support the paper's premise, the authors should compare the speed-accuracy trade-offs of LatticeTTS against FastTTS with fewer parents or smaller initial widths.

Insufficient coverage of related work and baselines.

The paper positions prior systems as accelerating a fixed, known workload. However, several existing works already allocate TTS compute under latency budgets and are not discussed. Examples include selecting the inference strategy and compute per query while accounting for both generation and scoring latency [1], formulating latency-optimal TTS as a resource-allocation problem [2], and comparing verification schemes under latency budgets that include verifier cost [3]. Correspondingly, the experimental comparison is limited to vLLM and FastTTS, neither of which adapts its computation to the deadline. The authors should clarify how LatticeTTS differs from these works, for example in per-round online re-planning, modeling of discrete batch boundaries, and parent preservation. They should also demonstrate its advantage experimentally against these approaches.

[1] Huang et al., Latency and Token-Aware Test-Time Compute

[2] Wang et al., Faster and Better LLMs via Latency-Aware Test-Time Scaling

[3] Montgomery et al., Budget-aware Test-time Scaling via Discriminative Verification

Questions for authors

See weakness

Reviewer_k23hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LatticeTTS is a deadline-aware serving system for verifier-guided test-time search. It exploits mismatched generator and verifier batch boundaries by adjusting the number of children per parent while preserving the search policy's selected parents. It combines a joint latency model, candidate-width pruning, and a receding-horizon planner. Experiments on a single RTX 4090 report 5.75× speedup over vLLM, 2.96× over FastTTS, and up to 16.7% improvement in fixed-deadline accuracy.

Critical strengths
  • Generator and verifier batch boundaries are misaligned, so adding one child can disproportionately increase latency. LatticeTTS exploits this while preserving selected parents and search ranking rules, making it applicable across different search policies.
  • This paper reframes test-time serving around verified candidates returned before the deadline, rather than candidates generated or admitted. This is a practical distinction as search only benefits from evidence available in time for the next decision. This framing generalizes beyond LatticeTTS, although its empirical benefit is not convincingly isolated.
  • Compiler preserves every selected parent and the search policy's structure, while producing nested allocations that add exactly one child at each step. Construction takes O(K + (B_max − K) log K), and pruning equivalent widths has a sound justification. These properties make the approach easy to integrate without changing the underlying search policy.
Critical weaknesses
  • Neither vLLM nor FastTTS is deadline-aware. Deadlines are defined using FastTTS's P95 latency, making its 100% miss rate at D50 essentially guaranteed. LatticeTTS also processes 4–32x fewer candidates, so much of its speedup and deadline-accuracy improvement may simply come from doing less work here.

    The paper also omits obvious baselines, e.g, FastTTS with deadline-triggered stopping, a smaller fixed width tuned to the same budget, and one child per parent. These could capture much of the benefit without a lattice or receding-horizon planner. Component ablations do not address this. So thees experiments fail to isolate the contribution of the proposed method.

  • PRM-selected accuracy decreases in three of four settings. Accuracy also fails to improve when deadlines increase from D150 to D300. The reported deadline-accuracy gains primarily reflect meeting deadlines rather than better search quality.

    The evaluation appears to use roughly 72 problems, with no confidence intervals or run-to-run variance. The motivating experiment uses only 16 problems per model pair. These sample sizes, to me, are insufficient to support the accuracy claims.

Questions for authors
  1. How does LatticeTTS compare against deadline-aware FastTTS, budget-matched fixed-width search, and one-child-per-parent search? Without these comparisons, how can the gains be attributed to the lattice and planner rather than simply reducing computation?

  2. How many problems are evaluated, how are benchmarks mixed, and what are the confidence intervals for the reported accuracy gains?

  3. Why does PRM-selected accuracy decrease in most settings? What accuracy does LatticeTTS achieve with a non-binding deadline compared with full-width search?

  4. How are the generator, verifier, and KV regimes identified? How much offline profiling is required, and do the gains persist on different hardware or batch configurations?

Reviewer_oTWVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how verifier-guided reasoning should allocate work when useful answers must arrive before a deadline (e.g., for latency reasons). Its central idea is to keep the currently selected parent paths and adjust the number of children generated beneath them using the measured cost of generation and verification.

  • LatticeTTS profiles generator, verifier, and cache states, forms a nested sequence of child allocations, and replans after each completed round. The planner maximizes the predicted number of verified candidates within the remaining time.
  • Experiments are with 3 generator–verifier size combinations, reporting ~3x speedup over FastTTS and ~5.75x over vLLM, along with improved fixed-deadline accuracy.
  • The main contribution is a systems controller for adjusting search effort. It preserves the current parent set and ranking rule; changing the available children can still change later search decisions.
Critical strengths

The premise of the paper is reasonable: additional reasoning helps only when its result can be generated, verified, and returned in time. The paper identifies a concrete reason that choosing a token or candidate budget alone can be a poor execution strategy.

  • The two-stage cost measurements motivate the design. Table 5 shows that adding a child can increase generator latency by ~26% while changing verifier latency by only 0.4%, or affect the verifier instead. Modeling these stages separately is a sensible extension to a single width-to-latency curve.
  • The allocation rule has a clear operational meaning. Giving each selected parent a child and adding further children in a stable order preserves parent coverage and nests samples within a round (these properties are explained in the appendix).
  • The implementation addresses its own overhead. The appendices measure action-set compression, planner latency, and telemetry cost. Table 11 reports planner cost below 0.45% of request wall time at P95 in the tested configurations. The ppaer evaluates several search policies and synchronized concurrent requests.
Critical weaknesses

My main concern is whether the proposed controller improves the accuracy–latency tradeoff over a reasonably budgeted baseline. The measurements make a good case for avoiding work that will finish too late, but the comparison with simpler ways of doing that is less clear.

  • Figure 5 compares against baselines that keep their original search widths as the deadline changes. In Table 3, FastTTS misses every deadline, even though its ordinary answer accuracy is higher in three of four settings. I would like to see fixed widths tuned for each deadline, with every method allowed to return its best completed answer. The boundary measurements and ablations are useful, but this comparison would show how much the controller adds beyond simply doing less search. The same issue matters for Table 2: more completed requests need not mean more correct answers delivered on time.
  • The timing model is difficult to assess from the description. Section 3 and the appendices explain the state representation, online cost corrections, and allowance for prediction error, but leave out the concrete calibration procedure, data requirements, and prediction accuracy on new requests. This matters especially for DVTS at width 512, where Table 3 reports an 81.9% deadline miss rate. Section 4.6 explains that DVTS leaves less freedom to reduce the search width, but not why so much admitted work finishes late despite the controller's check that it should fit.
  • The accuracy gains are hard to judge without a clearer account of the evaluation size and uncertainty. Section 4.1 names the benchmarks, but does not state how many questions from each were evaluated or how much the differences vary across questions and repeated runs. Without that context, I am not sure how much confidence to place in the reported accuracy differences.
  • The related work should cite many more results on latency-aware TTS, as examples, Faster and Better LLMs via Latency-Aware Test-Time Scaling, SPECS: Faster Test-Time Scaling through Speculative Drafts, and Latency and Token-Aware Test-Time Compute. These papers are broadly tackling the same question, even if the specific formulations are different.
Questions for authors

I would like to understand when the controller fails to meet the deadline, and how reliable the quality differences are.

  • At the same absolute deadlines, how does LatticeTTS compare with fixed widths tuned on held-out questions, when all methods can return their best completed answer?
  • How accurate are the timing predictions on held-out requests, and what explains the 81.9% deadline miss rate for DVTS at width 512 despite the controller's feasibility check?
  • How many questions from each benchmark support the accuracy results, and how stable are the differences across questions and repeated runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.