acceptodds
Withdrawn from ICLR 2027

NikTok: Faster Tokenizers Need Smarter Network Hardware, Not CPUs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tokenization running on the host CPU competes for the same cycles as agentic workflows with applications increasingly processing longer input contexts while relying on smaller models and tokenization delays making up a larger share of end to end inference. This overhead comes primarily from the host's network stack and OS, which handles every request before it reaches the tokenizer. To eliminate this overhead and free host CPU resources, we introduce NikTok, a system that runs tokenization entirely on network hardware. NikTok accelerates the pipeline, splitting each request between many parallel hardware execution units, overlapping computation and network transfer, with tokenized output directly being written to GPU memory invoking the host CPU or the OS. Our evaluation using real network hardware across four widely used tokenizers (Qwen3-Embedding-0.6B, Harrier-270M, EmbeddingGemma-300M, and mmBERT-Embed) demonstrates that NikTok cuts average inference latency by up to 9.8× and increases throughput by up to 883%, while freeing up 83.5% of host CPU capacity.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
In-network computing, Tokenization, SmartNIC, Data Path Accelerator, embedding models.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HG7MOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents NikTok, a tokenization system implemented on the Data Path Accelerator of an NVIDIA BlueField-3 SmartNIC. It combines table-driven pretokenization and BPE, persistent workers that process independent requests, and asynchronous RDMA delivery of token IDs to GPU memory. The design aims to remove host-side processing and data movement from the tokenization path and release CPU resources for other serving tasks. The evaluation compares four embedding-model tokenizers against host CPU and SmartNIC Arm baselines using text from LongBench and ShareGPT. The reported results demonstrate tokenization latency and throughput improvements, together with token-ID equivalence on the evaluated inputs. The evaluation also includes worker-count scaling, host CPU utilization, and transport ablations.

Critical strengths
  1. The implementation on real BlueField-3 hardware demonstrates the feasibility of executing tokenization within the DPA’s constrained environment.

  2. The evaluation reports substantial preprocessing improvements across four tokenizers and validates reference token-ID equivalence on the evaluated inputs.

Critical weaknesses
  1. The practical serving benefit remains unestablished. The abstract claims up to a 9.8× inference latency improvement, but the reported results concern tokenization and delivery, without clearly including model execution. Figure 1’s preprocessing breakdown and the host CPU utilization results do not establish improvements in complete inference or concurrent agentic tasks. Narrowing the wording would correct the claim, but would still leave the practical significance of the offloading architecture insufficiently evaluated. Even for the targeted embedding-serving setting, the paper does not demonstrate how much the preprocessing gains improve complete request latency or serving throughput.

  2. The evaluation does not adequately characterize when the offloading design is advantageous. The relationship between the 190 concurrent documents, total document count, and the appendix’s provisioning criterion is unclear. If the scaling experiment uses a fixed work set, Figure 6a’s plateau and final increase may reflect changes in the number of processing waves rather than characterize sustained scaling. The paper also does not quantify the shared-memory or RDMA limits, or adequately account for the NIC resources required to obtain the CPU savings. The dedicated testbed provides no evidence about coexistence with other traffic. These omissions leave the useful operating range and resource trade-offs of the design uncertain.

While the implementation demonstrates technical feasibility, I do not find the demonstrated contribution and experimental evidence sufficient for acceptance. The evaluation leaves both the practical serving benefit and the operating conditions that justify SmartNIC offloading insufficiently established. These gaps substantially limit the significance of the contribution, even if the inference claims are narrowed to tokenization.

Questions for authors
  1. Which reported measurement includes model execution and supports the inference latency claim? Can the authors provide a representative embedding-serving result showing the effect on complete request latency or serving throughput?

  2. Please clarify the total document count, concurrency, request replenishment policy, and measurement boundaries, including whether throughput uses corpus completion time or per-document latency. What evidence establishes the scaling behavior with sufficient sustained work, and what DPA and memory resources are required? Please also clarify whether the intended deployment assumes dedicated NIC resources.

Reviewer_rXUdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents NikTok, a system that moves BPE tokenization from the host CPU onto a SmartNIC. The authors argue that tokenization overhead comes not only from computation, but also from host networking, memory copies, and software processing. NikTok uses persistent workers on BlueField-3's Data Path Accelerator (DPA) to process incoming text and sends the resulting token IDs directly to GPU memory through RDMA. This paper evaluate four embedding-model tokenizers on LongBench and ShareGPT. Their results show lower tokenization latency, higher throughput, and reduced host CPU usage. The main contribution is moving tokenization into the network-to-GPU data path, rather than introducing a new tokenization algorithm.

Critical strengths
  1. I agree that this paper looks at tokenization as part of the overall serving pipeline rather than just a CPU computation that needs to be optimized. Figures 2 and 3 make it clear that a fair amount of overhead comes from host-side processing and data movement before tokens even reach the GPU. Moving tokenization onto the SmartNIC is an interesting way to avoid these costs. I think the idea is particularly useful for smaller models, where preprocessing can take up a larger portion of the total serving time.
  2. The authors implement NikTok on real BlueField-3 hardware and deal with several practical challenges, including limited DPA memory and the lack of dynamic memory allocation. They also test four different tokenizers on two datasets and verify that the token IDs match the reference implementation exactly. I appreciate the detailed latency results in the appendix, especially the p50 and p99 measurements. Overall, the implementation and evaluation provide good evidence that the proposed design can accelerate tokenization.
Critical weaknesses
  1. I have concerns about how the performance gains are measured and attributed. Section 4.2 includes useful RDMA and DPA ablations, but NikTok also uses substantially more parallel execution units than the CPU baseline. This makes it difficult to separate architectural benefits from increased hardware parallelism.

    The throughput definition in Appendix 7.4 is also unclear. Equation (7) defines throughput as total tokens divided by wall-clock time, yet equates its speedup with the ratio of median latencies, which may not hold under concurrency. The throughput values in Tables 3–10 also appear to be derived from latency percentiles rather than independently measured.

  2. It remains unclear how much the tokenization speedups improve actual model serving. While the paper aims to reduce AI serving latency, most measurements cover only tokenization. Equation (6) ends when the last token ID is produced and excludes model inference, making the end-to-end benefit difficult to assess.

    I would also like to see results at different request concurrency levels. Figure 6(a) varies the number of DPA workers, and the appendix discusses worker occupancy, but neither directly evaluates incoming request concurrency with fixed hardware. Since NikTok relies on request-level parallelism, latency and throughput at low and moderate concurrency would help clarify where it is most useful.

Questions for authors
  1. In Appendix 7.4, Equation (7) equates the throughput speedup with the ratio of median latencies. Could the authors clarify how throughput is measured in Tables 3–10? Is it independently measured using total wall-clock completion time, or calculated from the latency percentiles?
  2. Section 4.2 includes ablations for RDMA and DPA execution, but the complete systems also differ in available parallel hardware. Could the authors further explain how much of the overall speedup comes from avoiding host-side processing versus increased parallelism?
  3. Figure 6(a) shows scaling with the number of DPA workers. What happens when the hardware configuration stays fixed but the number of concurrent requests changes? It would be useful to see p50/p99 latency and throughput under low, moderate, and high concurrency.
Reviewer_YBykOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes NikTok, a method that moves BPE tokenization from the host CPU to network hardware. Specifically, it offloads the computation to SmartNIC's DPA, processes independent requests in parallel and sends tokens directly to GPU memory using RDMA. The evaluation uses four embedding-model tokenizers on BlueField-3 and reports improvements over Hugging Face Rust on the host CPU and Blink on the SmartNIC's Arm cores.

Critical strengths

The paper proposes a useful engineering solution to run BPE tokenization on the SmartNIC's DPA and deliver tokens directly to GPU memory, bypassing CPU. The results show significant tokenization speedup over Blink, its closest baseline.

Critical weaknesses
  1. The clarity of paper can be improved. The introduction is very dense and packed with details. It's difficult to follow the narrative and figure out the context and setting for the problem from the first read.

  2. The abstract claims that the proposed method cuts inference latency up to 9.8x, increases throughput up to 883% and frees up 83.5% of host CPU. I couldn't trace where these numbers came from and what experiments can support these claims. The introduction mentions that these improvements are achieved compared to the host baseline, but this is a weaker baseline. The gains comparing to Blink [1] (the closest competing method) are less impressive. Furthermore, the paper claims 9.8x inference latency decrease in the abstract, while in the contributions it lists 9.8x tokenization latency decrease. Tokenization speedup doesn't imply identical inference speedup.

  3. The paper claims that NikTok can be adopted on different types of networks hardware. However, only BlueField-3 SmartNIC is used in the experiments, and NikTok design relies on the programmable DPA workers, suitable memory resources and RDMA, so the claim of easy generalizability is an overstatement.

  4. The paper should compare with optimized CPU solutions such as PinTok [2], which is discussed in the related work but not included in the experiments. PinTok reduces scheduling, memory and networking overhead; these are the same overheads that motivate NikTok. Comparing both performance and CPU usage would help understand how much benefit DPA offloading provides over an optimized CPU implementation.

[1] Siavashi, Mohammad, et al. "Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC." arXiv preprint arXiv:2604.07609 (2026).

[2] Choi, Sean, Myungheon Chin, and Ernest K. Ryu. "PinTok: Tokenizers Deserve Dedicated Pinned CPU-Compute and Memory."

Questions for authors
  1. Could the authors please clarify how they obtained the numbers reported in abstract and contributions (discussed in weaknesses)?

  2. Figure 6a labels one baseline as "HF-Rust @ BF3 Arm" - is it a typo? I'm assuming it should be Blink.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.