- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a Triton GPU solver for batches of many small, independent entropic OT problems, such as those in detection label assignment, mHC, point-cloud matching and tracking. Each small instance runs entirely on-chip in one thread block and exits as soon as it converges, which frees the slot for the next instance. Fixed-iteration training layers get packed forward kernels and exact backward kernels. Under a common accuracy gate, the solver is 2.7-10.8x faster than OTT-JAX and 5-34x faster than POT on inference workloads, and gives 1.1-1.7x end-to-end speedups in several training pipelines.
Critical strengths
-
The benchmarking is unusually careful. All methods must pass the same accuracy gate, runs are paired with bootstrap intervals, and returned solutions are audited in FP64. The authors also report the cases where their solver loses. This makes me trust the speedups. The released code matches the paper's structure and includes tests.
-
The problem is real and the workloads are well chosen. Many pipelines solve lots of small OT problems in a loop, and practitioners currently write custom kernels or drop Sinkhorn altogether. The inputs are captured from real models such as OTA, GeoTransformer and mHC, so the results are directly useful.
Critical weaknesses
-
The ML contribution is limited. The main ingredients are standard GPU techniques: one block per problem, early exit, packing, and kernels borrowed from FlashSinkhorn and LogSinkhornGPU. Proposition 1 is essentially Graham's scheduling bound. This is a well-engineered library, and it might fit MLSys or a software track better than ICLR.
-
The end-to-end impact is modest and the scope is narrow. Whole-step speedups are only 1.13x for OTA and 1.18x for GeoTransformer. The 1.73x on mHC comes from a small 34M-parameter GPT, and TileLang-mHC does slightly better on the same step. The fast paths only cover up to 64 or 128 points per side, and all results are on a single A100.
Questions for authors
Clarifications:
- Can you show results on at least one other GPU, such as an H100? Several design choices look tied to the A100.
- For mHC at a realistic model size, how large is the Sinkhorn share of the step, and what end-to-end speedup would you expect?
- The supplement contains only the solver, not the benchmark scripts, the baseline wrappers or the input-capture code. Since the contribution is mainly speed, could you release these so the tables can be reproduced?
- How would a simple one-block-per-instance kernel without release compare? This would separate the benefit of the design from the implementation.
Suggestions:
- The citation to linear-time GNNs for recommendation in the introduction seems unrelated and could be removed.
- A short table showing which execution path handles which problem shape would help readers.
2 comments
Thank you for the thorough response and the new results. The H100 and L40S experiments show that the speedups carry over to other GPUs without retuning, the breakdown of the speedup sources makes clear that most of the gain comes from on-chip execution, and the released benchmark scripts address my reproducibility concern. I also appreciate the clearer attribution of the components taken from FlashSinkhorn and LogSinkhornGPU, and I agree that the release ablation in Fig. 3(a) already answers my question about a resident kernel without release.
The 2,500-step mHC runs are a useful addition. They show that computing the exact gradient of the K-step map matters in practice, which strengthens the ML side of the paper.
My remaining reservations are the same as before, but they are about scope rather than correctness. As the authors' own bound shows, the end-to-end gain is limited by the solver's share of the step, so whole-step speedups stay modest in the OTA and GeoTransformer pipelines and will shrink for mHC at realistic model sizes. The contribution is mainly a well-engineered solver rather than a new method. Overall, I keep my score and remain positive about the paper.
ML contribution and venue. Much of the recent use of Sinkhorn in machine learning is a batch of small, independent problems inside a loop. mHC projects one 4×4 residual mixing matrix per token and layer, and DeepSeek-V4 trains with it at 284B and 1.6T parameters. WASD (NeurIPS 2026) solves one OT problem per generated token when distilling large language models, PEARL (ICLR 2025) one per training prompt, supervised graph prediction (Méndez et al., 2026) one per predicted and target graph, Learn to Rank (ECCV 2026) one per image, and PLASMA (ICLR 2026) one per protein pair. These authors report the cost. WASD roughly doubles training time. Méndez et al. write that "the inner Sinkhorn rollouts dominate training time" and replace the solver with a learned matcher. For mHC, Wang and Qiu (2026) write a warp-level 4×4 solver, and later hyper-connection designs drop the Sinkhorn step. Medina and Schmitzer (2025) write their own CUDA solver for the cell problems of domain decomposition. The contribution is one solver that makes the instance the unit of GPU execution across these shapes and stopping rules, in place of a kernel per shape.
This fits the ICLR call for papers, which lists "infrastructure, software libraries, hardware, etc." among its topics. FlashSinkhorn, which we build on, was an ICML 2026 spotlight. On the ML side, the fixed-iteration path returns the exact gradient of the K-step map and stores K(N+M) numbers per instance instead of K plans. In the 2,500-step mHC runs, TileLang-mHC, whose backward is an implicit gradient at its output, stalls, while the official loop and DrainSinkhorn reach the same held-out loss (general response).
Proposition 1 applies Graham's bound for list scheduling to the GPU block scheduler. It states when release saves wall time: on batches that run in several waves, not on a single wave. On all three GPUs, release speeds up the multi-wave mot-b16 by 1.23–1.41× and the single-wave mot-b64 by 1.00–1.03×.
Kernels borrowed from FlashSinkhorn and LogSinkhornGPU. Three parts follow earlier code, as the source files and the NOTICE state. The dot-product and online log-sum-exp helpers of the indexed path are adapted from FlashSinkhorn. Our consumer kernel, which applies the plan rebuilt from the potentials without storing it, follows the organization of FlashSinkhorn's plan-apply kernel. The separable reduction follows LogSinkhornGPU and Medina and Schmitzer, and DrainSinkhorn runs it inside a resident program. The resident solves with release, the packed fixed-iteration forward and backward, the cooperative short-by-long kernel and the screen in the final stage of the row-update kernel are ours.
One-block-per-instance kernel without release. This arm is already in the paper: in §5.4 and Fig. 3(a), the same resident kernel runs every instance to D_max with the same checks. Most of the gain comes from on-chip execution, not from release or kernel code (general response).
End-to-end gains. When only the Sinkhorn solve is replaced, the whole-step speedup is at most 1−s1, where s is the share of the official solver in the step. In the paper's A100 runs, the Sinkhorn part of label assignment is 12.8% of the OTA step, so the bound is 1.15×, and we measure 1.13×. In GeoTransformer the OT share of the training step is 16.4%, so the bound is 1.20×, and we measure 1.18×.
TileLang-mHC on the same step. It is not the same step. TileLang-mHC's backward is an implicit gradient at its output, not the gradient of the 20-step map, which is what the asterisk marks in Table 1. In the 2,500-step runs its gradient norm before clipping grows to 116–83,279, against 0.57–0.69 for the official loop and DrainSinkhorn, and its held-out loss stalls (general response). Step times are comparable only among methods that train the same model, and among those DrainSinkhorn has the fastest step: against the official loop, a whole GPT training step is 1.73× faster with DrainSinkhorn, 1.57× with triton-sinkhorn and 1.32× with
torch.compile(§5.2).mHC at realistic size. Per token and layer, the Sinkhorn work is a 4×4 map iterated 20 times whatever the model width, while the rest of the layer grows with the square of the width, so its share falls as the model grows. The mHC paper reports a total mHC overhead of 6.7% in its large-scale training, and Sinkhorn is only part of that. The bound above then applies with this smaller share. What does not depend on scale is that DrainSinkhorn computes the exact gradient of the 20-step map, which the long runs show to matter.
Reproducibility and path table. The supplement now contains the benchmark (general response). Table 3 (§5.4) gives the execution path of every inference workload, and §3 states the path of each training layer.
LTGNN citation. We keep it. It cites another pipeline where execution organized around the workload removed a scalability bottleneck, one of the motivations of this work.