acceptodds
Under review as a conference paper at ICLR 2027

RelayTTS: Structure-Aware Verifier Serving for Edge Test-Time Scaling

Abstract

Verifier-guided test-time scaling (TTS) offers a practical route to stronger reasoning on memory-constrained devices, where a larger model may not fit. Its search procedure naturally produces a live frontier of parent groups, each defined by a shared parent prefix. Existing TTS serving systems discard this structure by scoring candidates independently and switching between generator and verifier at model granularity. The result is repeated parent-prefix computation within scoring passes and repeated model movement across them. We present RelayTTS, which makes the live frontier the unit of verifier serving. Parent-complete score packing evaluates shared parent prefixes once across sibling candidates, while a frontier-aware planner trades score-pack capacity against the resident verifier prefix. Layer relay supplies the nonresident blocks only during scoring. Across search procedures and model scales, RelayTTS achieves a 1.91× gain in effective-token throughput and cuts end-to-end latency by 47.8% on average, with a maximum reduction of 88.3%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
test-time scaling, process reward models, prefix reuse, adaptive weight residency

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_kqiYOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RelayTTS, a serving system specifically built for Test Time Scaling on resource-constrained devices. It first identifies system-level bottlenecks like redundant recomputation across siblings, model tranisition overhead, and memory contention. Then, it proposed novel methods like parent-complete score packing, fontier-aware joint planning, layer relay to accelerate TTS. The speedup over the SOTA baseline is proven across benchmarks.

Critical strengths
  1. The bottlenecks are clearly illustrated using figures. The paper explains well why repeated prefix computation and model weight transfer form the critical bottlenecks of the system.
  2. System design is intuitive. The paper designs the search structure with system performance, directly addressing the bottlenecks.
  3. Substantial gains. The results demonstrate strong benefits when complete model swapping is costly.
  4. Useful mechanism analysis. Search frequency, branching factor, and component ablations help explain when gains arise. Reporting the small contribution of overlap makes the performance attribution more credible.
Critical weaknesses
  1. What components of the method generalize to stronger GPU setting (like H100s)?
Questions for authors

See weaknesses.

2 comments

We thank the reviewer for the constructive suggestions on generalization to stronger GPU settings. We have revised the paper in light of this feedback.

W1.

  • Parent-complete score packing, joint planning, and layer relay generalize to stronger GPU settings, with the execution mode determined by available memory. More GPU memory can reduce model transition overhead, but sibling candidates still repeat shared-prefix computation. When both models and the search state fit in memory, RelayTTS uses a fully resident verifier and continues to exploit parent groups to reduce computation. When full residency is infeasible, it keeps the generator and part of the verifier weights resident for reuse in later scoring rounds, while layer relay supplies the remaining verifier layers on demand. The planner jointly selects score-pack capacity and verifier residency from the current frontier, using device-specific compute, transfer, and memory profiles to estimate execution cost.

  • RelayTTS retains end-to-end speedups on stronger GPU settings. On an A800 (80GB), we compare RelayTTS against FastTTS augmented with prefix caching, score caching, and partial verifier residency (FastTTS++) on AIME 2026.

    Model pairnEnd-to-end speedupVerifier speedup
    3B+7B641.15×1.91×
    3B+7B1281.13×1.63×
    7B+72B642.13×2.66×
    7B+72B1282.11×2.62×
    14B+72B642.16×2.92×
    14B+72B1282.14×2.85×

    For the 3B+7B pair, neither system transfers verifier-layer weights during scoring, yet parent-complete packing still reduces computation. With a 72B verifier, some weights remain resident across rounds and the remaining layers are relayed during scoring. Joint planning accounts for the number of score packs when selecting residency, reducing verifier-layer weight transfers by 60.8%–66.4%. The results cover execution both with and without verifier-layer weight transfers. The same execution interface can be used on platforms such as H100, with configurations selected using the corresponding device profiles.

  • RelayTTS also accelerates non-math workloads. Code search on HumanEval, MBPP sanitized, and LiveCodeBench still alternates generation with PRM scoring. RelayTTS jointly plans score packs and resident weights for the code-search frontier, and layer relay supplies nonresident layers during scoring. Parent-complete packing reduces repeated computation within each round, while persistent residency and fewer scans reduce model movement. The following results compare RelayTTS with FastTTS++ on the A800 using the 14B+72B pair.

    WorkloadnEnd-to-end speedupVerifier speedup
    HumanEval321.55×1.73×
    HumanEval641.68×1.96×
    HumanEval1281.71×2.03×
    MBPP sanitized321.49×1.67×
    MBPP sanitized641.71×2.02×
    MBPP sanitized1281.48×1.69×
    LiveCodeBench321.77×2.09×
    LiveCodeBench642.09×2.62×
    LiveCodeBench1281.83×2.15×

Thank you for the additional results. I encourage you to include the results, especially the stronger GPU results, in the updated version.

Reviewer_T359OpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

RelayTTS accelerates verifier-guided test-time scaling on memory-constrained devices. It groups candidate paths that share a parent prefix so the verifier can reuse prefix computation, retains part of the verifier on the GPU, and transfers the remaining layers during scoring. A frontier-aware planner jointly selects score-pack capacity and verifier residency. The evaluation reports substantial latency reductions across model pairs and search configurations.

Critical strengths
  1. Well-motivated and practical system design. Parent-prefix reuse and partial verifier residency address concrete inefficiencies in alternating generation and verification. Jointly planning residency and score-pack capacity is sensible because both compete for limited device memory.
  2. Solid experimental evaluation. The evaluation explores several operating regimes and includes component ablations, search-structure sensitivity, and memory-pressure experiments. The reported results connect performance gains to prefix reuse and reduced model movement. The paper also identifies that transfer overlap contributes relatively little in the tested configurations, providing a useful explanation of where the gains originate.
Critical weaknesses
  1. Hardware coverage is limited. Experiments use a single RTX 4090 platform. Since the benefit depends partly on host-to-device transfers and available memory, its magnitude may differ on devices with other bandwidth and memory characteristics. Broader hardware evaluation would help establish generality.
  2. Implementation-level equivalence could be documented more explicitly. The appendix explains why packing and layer relay should preserve verifier computation. Reporting score differences and survivor-selection agreement on identical candidate frontiers would further strengthen this argument, particularly under BF16 execution. Clarifying position handling in packed inputs would also improve reproducibility.
Questions for authors
  1. Have the authors measured packed-versus-independent verifier score differences and survivor-selection agreement on identical candidate frontiers?
  2. How sensitive is the planner to errors in its latency profiles, especially when transfer bandwidth or memory pressure changes?
3 comments

We thank the reviewer for the suggestions on hardware coverage, implementation-level equivalence, and planner robustness. We have revised the paper in light of this feedback.

W1.

  • RelayTTS continues to reduce verifier computation and weight movement on stronger GPU configurations. On an A800 (80GB), we compare RelayTTS against FastTTS augmented with prefix caching, score caching, and partial verifier residency (FastTTS++) on AIME 2026.

    Model pairnEnd-to-end speedupVerifier speedup
    3B+7B641.15×1.91×
    3B+7B1281.13×1.63×
    7B+72B642.13×2.66×
    7B+72B1282.11×2.62×
    14B+72B642.16×2.92×
    14B+72B1282.14×2.85×

    For the 3B+7B pair, neither system transfers verifier-layer weights during scoring, so the verifier speedup comes from reducing repeated prefix computation. With a 72B verifier, weights and scoring state still compete for GPU memory, and RelayTTS reduces verifier-layer weight transfers by 60.8%–66.4%. The gains include shared computation within a round, weight residency across rounds, and fewer scans of nonresident layers. Device profiles capture the compute, transfer, and memory characteristics, while the current parent groups determine the required score packs.

W2 / Q1.

  • Parent-complete score packing preserves each candidate's logical input, position IDs, and causal dependencies. Shared prefixes are identified after verifier tokenization. Parent-prefix position IDs start at zero; each child suffix continues the numbering from its own parent-prefix length. A child suffix can attend only to its parent prefix and its own preceding tokens; sibling suffixes and different parent groups remain isolated. The same rules apply to fragmented parent groups. Scores are returned in the original candidate order before survivor selection.

  • Packing introduces negligible numerical score differences, and layer relay produces identical scores for a fixed packing layout. For each fixed frontier, we hold tokens, candidate order, and reward positions constant and compare packing layouts and weight residency. The mean absolute difference between packed and independent scoring is only 1.59e-5. The BF16 comparisons are below.

    ComparisonMean absolute score differenceP99 absolute score differenceMax. absolute score difference
    Packed, fully resident vs. independent, fully resident1.59e-59.44e-52.65e-4
    Packed + relay vs. independent, fully resident1.59e-59.44e-52.65e-4
    Packed with group fragmentation vs. independent, fully resident2.05e-51.04e-43.56e-4
    Relay vs. full residency (same packing)000

    Layer relay changes when weights become available on the device. Each layer still uses the same weights and operators once its weights are ready, so relay introduces no additional score differences. The numerical differences between packed and independent scoring are reported separately in the table.

  • Packing and layer relay preserve the original selection rule, with high overlap between their survivor sets and those from independent scoring. We also measure survivor-selection agreement among the candidates actually scored at frontiers generated during search, keeping the original number of retained candidates.

    Selection statisticResult
    Mean top-k overlap97.22%
    Minimum top-k overlap93.75%

    We have added details of position handling and these BF16 score-difference and survivor-overlap statistics to the revised paper.

Q2.

  • Joint planning tolerates errors in latency profiles, with stable execution plans under 15% perturbations. For the 3B+7B pair on the RTX 4090, we perturb overall compute and host-to-device (H2D) transfer-time estimates, as well as length-dependent compute estimates and layer-dependent H2D estimates. The perturbations affect only the planner inputs; the scoring executor is unchanged. Latency increases are relative to the configuration selected using the unperturbed profiles.

    Profile error magnitudePlan change rateMean latency increaseMax. latency increase
    15%0%0.00%0.00%
    30%12.5%11.72%15.86%
    45%37.5%32.18%49.14%
    60%75.0%47.81%66.93%

    Parent groups and the corresponding prefix and suffix lengths come directly from runtime state. Packing determines the number of score packs for each candidate capacity, and profiles estimate execution time for each feasible plan. Overestimating H2D transfer time gives greater weight to a plan's total transfer cost, whereas underestimating it gives that cost less weight. This can change the selected combination of residency and score-pack capacity. The table quantifies the resulting effect on execution time.

  • Joint planning adapts to changing memory availability by adjusting weight residency and score packing together. Using the same model pair, we allocate CUDA tensors to impose memory pressure, setting available memory to 24, then 21, and then 18 GiB within a request and restoring 24 GiB for the next request. All policies use the same packing implementation and relay executor. The static plan uses 5 resident layers and a 32K score-pack capacity; the decoupled policy selects residency before choosing score-pack capacity. Time columns report mean end-to-end latency, and both speedups are relative to the decoupled policy.

    nStatic (s)Decoupled (s)Joint (s)End-to-end speedupVerifier speedup
    6442.3641.6237.011.12×1.33×
    12861.4064.6156.191.15×1.37×

    The static plan must accommodate the lowest memory budget, so it cannot make full use of the additional residency space when more memory is available. The decoupled policy can adjust residency but may leave too little room for score packs. Joint planning compares total scoring time for the round, accounting for whether releasing weights avoids another parent-group fragment or another scan of nonresident layers. Within a request, the resident verifier prefix can only shrink; the next request can select a new configuration based on the restored budget. The mean time spent on planning and residency adjustment ranges from 1.27 to 12.03 ms across budget groups and is included in end-to-end time.

Reviewer_T359OpenReview ↗

Great work

Thanks you for providing the detailed response to my questions. I think this addresses most of my concerns. Great work!

Reviewer_zDt2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the cost of verifier-guided test-time scaling when candidate paths share parent prefixes and the generator and verifier cannot fully co-reside in memory. It proposes RelayTTS, which packs sibling candidates to reuse verifier prefixes, jointly plans score-pack capacity and verifier residency, and relays nonresident verifier layers during scoring. It evaluates latency, effective-token throughput, and search accuracy across math-reasoning workloads, search procedures, and model pairs.

Critical strengths
  • The paper formalizes a useful systems abstraction: keeping the live search frontier explicit as execution moves from generation to verification.

  • The design follows a coherent execution model, with targeted analysis of how its mechanisms work.

Critical weaknesses
  • Prefix sharing, layer-wise offloading, and memory-aware scheduling are already well studied. The ablations suggest that prefix reuse and persistent residency account for most of the gains, leaving the added benefit of joint frontier-aware planning less clear. Stronger comparisons with prefix-aware serving baselines are needed.

  • The evaluation is limited to a single RTX 4090 GPU, relatively small generator–verifier pairs (up to 7B), and mathematics-focused benchmarks. How well the reported improvements carry over to larger models, other hardware configurations, or realistic concurrent serving workloads remains an open question.

Questions for authors
  1. How does RelayTTS compare against FastTTS augmented with existing prefix-sharing mechanisms and partial verifier residency? Can the authors isolate the incremental benefit of their joint planner?

  2. How does RelayTTS perform with larger verifiers, different GPU memory bandwidths, and multiple concurrent TTS requests?

  3. How sensitive is the frontier-aware planner to profiling inaccuracies, dynamically changing memory availability, and heterogeneous search frontiers?

4 comments

We thank the reviewer for the constructive suggestions on baseline comparisons, joint planning, and generality. We have revised the paper in light of this feedback.

W1 / Q1.

  • RelayTTS uses the search frontier to guide both scoring execution and verifier weight residency. Existing prefix-sharing runtimes provide reuse mechanisms, while the frontier exposes parent–child relationships before scoring. RelayTTS uses parent-complete score packing to combine complete parent groups into score packs so siblings share prefix computation without issuing a separate small call for every parent group. The planner uses these pack boundaries to select residency: reducing score-pack capacity can increase the number of packs, and each additional pack requires another scan of the remaining nonresident layers. Equation (2) and Appendix A.4 describe this coupling, which distinguishes joint planning from enabling prefix sharing and partial residency separately.

  • RelayTTS complements existing prefix caching, score caching, and partial residency, further reducing redundant computation and weight movement. On the RTX 4090, we compare the 3B+7B pair on AIME 2026 against FastTTS augmented with prefix caching, score caching, and partial verifier residency (FastTTS++). All systems replay the same generated tokens and survivor-selection trace to match the executed workload.

    nFastTTS++ (s)RelayTTS (s)Speedup
    12863.0139.581.59×
    256114.8376.451.50×

    At n = 256, verifier-layer weight transfers per request decrease from 535.66 GiB to 57.03 GiB, with a 2.57× verifier speedup. Parent-complete packing reduces repeated prefix computation and also affects the number of score packs needed for the candidates. Joint planning uses these pack boundaries to select residency and reduce repeated scans of nonresident layers.

  • Joint planning weighs residency savings against score-pack costs, reducing the extra computation and weight scans caused by decoupled decisions. We keep parent-complete packing, relay, caching, and precision identical and vary only the planning policy under a common 21 GiB budget. The static configuration uses 15 resident layers and an 8K score-pack capacity; the decoupled policy selects residency before choosing capacity. Time columns report mean end-to-end latency, and both speedups are relative to the decoupled policy.

    nStatic (s)Decoupled (s)Joint (s)End-to-end speedupVerifier speedup
    6426.1726.3023.271.13×1.43×
    12840.0540.6336.441.12×1.44×

    Keeping one additional layer resident saves transferring that layer in every score pack. If doing so adds a pack, however, all other nonresident layers must be scanned again, and the shared prefix may also need to be recomputed when a parent group is split. Joint planning compares the cost of the entire round, so it can select fewer resident layers when that reduces the pack count.

W2 / Q2.

  • RelayTTS retains its benefits on stronger GPU configurations and with larger verifiers. On an A800 (80GB), we compare RelayTTS with FastTTS++ on AIME 2026.

    Model pairnEnd-to-end speedupVerifier speedup
    3B+7B641.15×1.91×
    3B+7B1281.13×1.63×
    7B+72B642.13×2.66×
    7B+72B1282.11×2.62×
    14B+72B642.16×2.92×
    14B+72B1282.14×2.85×

    For the 3B+7B pair, neither system transfers verifier-layer weights during scoring, yet parent-complete packing still reduces computation. The 72B verifier cannot be fully resident. RelayTTS keeps some verifier weights resident across generation and scoring rounds and relays the remaining layers as needed. Joint planning also controls pack count to limit repeated scans of these nonresident layers. Verifier-layer weight transfers decrease by 60.8%–66.4%, consistent with RelayTTS's performance gains.

  • RelayTTS retains its throughput and latency advantages when independent TTS requests are batched together. On AIME 2026 with n = 64 per request, each request maintains independent search state, and generation and scoring are batched across requests. Concurrency denotes the number of in-flight requests. Response time is measured from admission to completion and includes waiting within the service.

    GPU and model pairConcurrent requestsThroughput gain vs. FastTTS++FastTTS++ P95 (s)RelayTTS P95 (s)
    RTX 4090, 3B+7B11.70×54.3633.26
    RTX 4090, 3B+7B21.59×85.6059.48
    RTX 4090, 3B+7B41.38×153.09106.67
    A800, 7B+72B12.14×171.4674.96
    A800, 7B+72B22.73×340.71111.57
    A800, 7B+72B42.26×608.30263.24

    Batching across requests allows a single weight transfer to serve more candidates, but the GPU must also hold search state for multiple requests. RelayTTS preserves their parent groups within these batches and coordinates shared computation, score-pack capacity, and weight residency. Resident weights are reused in later scoring rounds, and layer relay supplies the remaining layers, so the optimizations to model movement and scoring execution continue to apply under concurrency.

  • RelayTTS also accelerates non-math workloads. Code search on HumanEval, MBPP sanitized, and LiveCodeBench still alternates generation with PRM scoring. RelayTTS organizes scoring by the parent groups of code candidates and jointly selects score-pack capacity and residency. Some verifier weights remain resident across rounds, while layer relay supplies the rest during scoring. Execution combines shared computation, persistent residency, and layer relay. The following results are for the 14B+72B pair on the A800, relative to FastTTS++.

    WorkloadnEnd-to-end speedupVerifier speedup
    HumanEval321.55×1.73×
    HumanEval641.68×1.96×
    HumanEval1281.71×2.03×
    MBPP sanitized321.49×1.67×
    MBPP sanitized641.71×2.02×
    MBPP sanitized1281.48×1.69×
    LiveCodeBench321.77×2.09×
    LiveCodeBench642.09×2.62×
    LiveCodeBench1281.83×2.15×

Q3.

  • Joint planning tolerates profiling inaccuracies, with stable execution plans under 15% perturbations. For the 3B+7B pair on the RTX 4090, we perturb compute and host-to-device (H2D) transfer-time estimates, including biases that vary with sequence length or layer position. Latency increases are relative to the configuration selected using the unperturbed profiles.

    Profile error magnitudePlan change rateMean latency increaseMax. latency increase
    15%0%0.00%0.00%
    30%12.5%11.72%15.86%
    45%37.5%32.18%49.14%
    60%75.0%47.81%66.93%

    Parent groups and token lengths are known directly, and packing determines the pack count for each candidate capacity. Profiles estimate how long each feasible plan will take. Errors in transfer estimates can change the tradeoff between retaining weights and enlarging packs. We therefore use profiles matched to each model and hardware configuration and compare plans using the actual frontier in each round.

  • Joint planning adapts to changing memory availability by adjusting weight residency and score packing together. Using the same 3B+7B pair on the RTX 4090, we allocate CUDA tensors to impose memory pressure and set available memory successively to 24, 21, and 18 GiB within a request, restoring it for the next request. The static plan uses 5 resident layers and a 32K score-pack capacity; the decoupled policy selects residency before choosing score-pack capacity. All other execution mechanisms are identical. Time columns report mean end-to-end latency, and both speedups are relative to the decoupled policy.

    nStatic (s)Decoupled (s)Joint (s)End-to-end speedupVerifier speedup
    6442.3641.6237.011.12×1.33×
    12861.4064.6156.191.15×1.37×

    The static plan must accommodate the lowest budget, while decoupled planning may reserve too much memory for weights. At each scoring boundary, joint planning weighs the pack count required by the current parent groups against weight-transfer cost. As the budget decreases, it releases some resident layers and selects a new score-pack capacity. The resident verifier prefix can only shrink within a request; the next request selects a configuration based on the restored budget. This avoids repeatedly releasing and reloading resident layers within the same request.

  • Joint planning adapts to heterogeneous search frontiers, reducing unnecessary score packs and repeated weight scans. Every frontier below contains 64 candidates. All conditions use the same packing implementation, relay executor, and performance profiles. The decoupled policy uses 18 resident layers and an 8K capacity; joint planning selects a configuration for the current frontier.

    Frontier structureResident layers (joint)Score-pack capacity (joint)Score packs (decoupled)Score packs (joint)
    Few wide parent groups188K11
    Many narrow parent groups1540K41
    Uneven group widths1540K41
    Long prefixes1540K61
    Mixed suffix lengths1532K92

    For the frontier with many narrow parent groups, joint planning reduces the number of resident layers from 18 to 15 and increases score-pack capacity from 8K to 40K, reducing the pack count from four to one. Although more layers must be transferred for each pack, total verifier-layer weight transfers for the round fall from 22.75 GiB to 6.91 GiB, with a 1.56× verifier speedup. The frontier with few wide parent groups needs only one pack, so joint planning keeps the same configuration as the decoupled policy. Parent-group structure therefore affects shared computation, pack count, and weight traffic. Candidate count or a fixed residency ratio alone cannot capture these differences.

Thank you again for your review. Our response is posted above, together with a revised PDF. We have added the FastTTS++ comparison and a joint-planner ablation, as well as results with larger verifiers, concurrent serving, and non-math workloads.

Could you let us know whether these additions address your concerns, or which points remain open? We are available to clarify any remaining points before the discussion closes. If the concerns are resolved, we would be grateful if you could reflect this in your assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and the revision

Dear Area Chair,

Thank you for handling our submission. We also thank the reviewers for their careful reading and constructive suggestions, which helped us strengthen the evaluation and clarify RelayTTS’s contribution and scope. As the discussion period draws to a close, we would like to summarize the main concerns raised during review and the evidence we added in response.

The main concerns fell into four groups.

  1. Stronger baselines and additional contribution (zDt2). The reviewer requested comparisons with existing prefix-sharing mechanisms and partial verifier residency, together with an ablation isolating the benefit of joint planning.
  2. Hardware, model scale, and workload coverage (kqiY, T359, zDt2). The reviewers asked about stronger GPUs, larger verifiers, concurrent requests, and non-math workloads.
  3. Implementation-level equivalence (T359). The reviewer requested clearer position handling and measurements of BF16 score differences and survivor-selection agreement.
  4. Planner robustness (T359, zDt2). The reviewers asked about profiling inaccuracies, changing memory availability, and heterogeneous search frontiers.

We addressed the reviewers’ main concerns through additional experiments, measurements and clarifications.

  1. Added an enhanced baseline and isolated the contribution of joint planning. We added FastTTS++, which augments FastTTS with prefix caching, score caching, and partial verifier residency. With identical generated tokens and survivor-selection traces, RelayTTS achieves 1.50–1.59× end-to-end speedup over FastTTS++. A separate ablation holds packing, relay, caching, and precision fixed while changing only the planning policy. Joint planning achieves 1.12–1.13× end-to-end speedup and 1.43–1.44× verifier speedup over decoupled decisions. We clarified why enabling caching and partial residency separately does not capture this benefit: retaining more weights can reduce score-pack capacity and introduce another scan of all remaining nonresident layers. Joint planning accounts for both costs across the scoring round.

  2. Extended the evaluation to stronger GPUs, larger verifiers, concurrent requests, and non-math workloads. On an A800 (80GB), RelayTTS achieves 2.11–2.16× end-to-end speedup over FastTTS++ with 7B+72B and 14B+72B pairs, while reducing verifier-layer weight transfers by 60.8%–66.4%. With 3B+7B, neither system transfers verifier-layer weights during scoring, yet parent-complete packing still accelerates verification. Under concurrent serving, generation and scoring are batched across independent requests. With 7B+72B on A800, RelayTTS achieves a 2.73× throughput gain at two concurrent requests, alongside lower P95 response latency. On HumanEval, MBPP sanitized, and LiveCodeBench, the 14B+72B pair achieves 1.48–2.09× end-to-end speedup. Reviewer kqiY wrote, “I encourage you to include the results, especially the stronger GPU results, in the updated version.”

  3. Clarified execution semantics and added BF16 agreement measurements. We explained how logical position IDs, causal masks, group fragmentation, and candidate ordering preserve each candidate’s verifier input and selection rule. On identical frontiers, the mean absolute difference between packed and independent scoring is 1.59e-5, with 97.22% mean top-k overlap among the candidates actually scored. Layer relay and full residency return identical scores for a fixed packing layout. These measurements separate the small numerical differences associated with BF16 packing from weight movement, which introduces no additional score difference.

  4. Measured planner robustness and adaptation overhead. We reported plan changes and latency under compute and transfer-profile perturbations from 15% to 60%, with stable plans at 15%. Under CUDA memory pressure that reduces available memory from 24 to 21 to 18 GiB within a request, joint planning remains 1.12–1.15× faster than decoupled decisions. Heterogeneous-frontier experiments further show how retaining fewer layers can reduce pack count and total weight traffic. For many narrow parent groups, four packs become one, so the avoided scans outweigh the extra transfer per pack. Planning and residency adjustment take 1.27–12.03 ms across memory-budget groups and are included in end-to-end latency. Following our response, reviewer T359 wrote, “I think this addresses most of my concerns.”

We have incorporated these results and clarifications into the revised paper, with changes highlighted. We thank the reviewers and the Area Chair again for their time and careful consideration.

Best regards,

Authors of Submission 23586

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.