- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
THRIFT retrieves draft tokens from an agent's prompt and earlier output. It scores candidates, adjusts draft width, and builds indexes asynchronously. The paper tests it on BeyondSWE request replays in SGLang.
Critical strengths
- The analysis of token reuse in agent workloads is useful.
- The paper measures serving time and CPU index costs, not just acceptance length.
Critical weaknesses
- The core idea overlaps with SuffixDecoding, which already uses prompt/history retrieval, request-local indexes, and adaptive tree drafting for agent workloads. There is no direct comparison. The external-corpus SGLang SAM baseline is not a substitute.
- Overall gains are small: 1.78x request speedup versus 1.75x for SGLang N-Gram on the cleaned Qwen3-14B greedy workload. Frequency-only scoring matches the full scoring method's acceptance length, and fixed width 24 matches adaptive-width latency.
- Evidence is limited to BeyondSWE request replays. Trained draft model baseline is retrained on Open-PerfectBlend, not agent trajectories. The results do not establish an advantage over agent-adapted learned drafters or beyond reuse-heavy coding workloads.
Questions for authors
- How does THRIFT compare with SuffixDecoding under the same serving setup and draft budget?
- Which components reliably reduce latency beyond request-local indexes and asynchronous construction?
- Do the gains hold against EAGLE-3 or DFlash adapted to separate agent training data?
4 comments
Dear Reviewers,
This is a friendly reminder that the rebuttal discussion period is approaching its deadline.
Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.
As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.
Thank you for your time and effort.
Best regards,
Area Chair
Thanks for adding the comparisons. I’m still unclear how the new speedups relate to the original end-to-end results, and whether acceptance-aware scoring improves over frequency-only scoring. I’m keeping my weak reject recommendation.
Thank you for the follow-up
Thank you for the follow-up.
After submission, we refined several indexing details to improve cross-turn reuse: retaining earlier outputs in both SAM and Trie, prepending a short prompt tail to stored outputs to support matching at turn boundaries, and routing tool outputs into the session Trie. The supplementary comparisons in our General Response use this updated implementation. We will make improved details explicit.
Regarding scoring, frequency-only scoring already achieves overall performance close to acceptance-aware scoring in our evaluated settings. Our main contribution lies in coordinating reusable token sources, candidate selection, and verification budgets across the decoding pipeline. Acceptance feedback provides a modest refinement: our diagnostic shows an approximately two-percentage-point improvement in prediction AUC for candidates with prior acceptance history, although its additional benefit to overall speed is limited. The comparison against recency-based expansion evaluates the complete pooled scoring mechanism, while the frequency-only ablation isolates the acceptance term.
We will clarify these contributions.
Response to Reviewer uD9R
W1: Overlap with SuffixDecoding. See General Response. The design differences: (1) SAM (O(n) async construction) vs suffix tree (synchronous online updates); (2) scored expansion — the single largest contributor per our ablation; (3) cost-aware width selection — why SuffixDecoding at 5.38× loses to N-Gram at 6.16× (batch 8, 14B) while THRIFT reaches 7.86×.
W2: Small gains / ablation components. Beyond the average gain, we care about how consistently THRIFT wins across deployment settings. The 1.78× vs 1.75× figure compares against N-Gram at its default width. With each method at its best width under measured forward costs, THRIFT leads N-Gram in every setting we tested: 1.22–1.40× on 14B and 1.18–1.35× on 32B, across 1–4 GPUs and batch sizes 1–16, with every 95% interval above 1. The baselines' best width shifts with model and batch size (N-Gram: 20 at 14B batch 8, 12 at 32B batch 8), so a fixed default can lose about 10%. THRIFT picks its width from the cost model and needs no per-setting tuning.
W3: Single workload / trained baseline data. BeyondSWE is the most complete public agent-trajectory dataset with multi-turn tool use, and its three tools (bash, file editor, search) cover the dominant patterns in coding agents. Regarding agent-adapted trained drafters: no public agent-trajectory training set exists for draft models. Training on agent data would require teacher-forced pairs from the same sessions — an interesting direction we will discuss. The fundamental limitation remains: a learned drafter predicts from hidden states, while agent reuse requires retrieval across the full prompt.
Q: Which components reduce latency? Ablation ladder: scored ranking (+6.1%), width increase enabled by scoring (+11.4%), prompt context in session Trie (+12.6%). Async lifecycle converts accept-length gains into wall-clock gains (Table 4: sync 1.66× → async 2.28× all-turns).