- Rating
- 1
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents Target Manifold Optimal Transport (TMOT), a relaxed verification method for speculative decoding. Standard greedy verification accepts a draft token only when it matches the target model’s preferred token. TMOT also accepts some nonmatching tokens. The proposed token must receive sufficient probability from the target model, and the draft and target probability distributions must be sufficiently similar under an approximate optimal transport calculation. The transport cost is defined using cosine distance between the target model’s static token embeddings. The acceptance threshold becomes stricter when the target model expresses greater confidence.
TMOT requires no additional model training or parameter changes. It retains the longest accepted prefix from an eight-token draft and uses the target model’s preferred token at the first rejection. Because TMOT can accept a token that the target model would not have selected, it does not preserve the target model’s output distribution. The authors evaluate three draft and target model pairs on code generation, mathematical reasoning, and multilingual mathematical reasoning. Across the 21 reported model and task combinations, TMOT provides speedups of 1.13 to 1.90 times relative to target-only generation, while retaining between 89.27 and 100 percent of the target model’s task accuracy.
Critical strengths
Strengths
- The proposed combination is original and has potential. Prior work has separately studied relaxed verification, semantic similarity, distribution comparison, and optimal transport in speculative decoding. However, I am not aware of an earlier method that uses static target token embeddings as the cost for optimal transport between draft and target probability distributions, together with separate target probability and uncertainty conditions. The experiments indicate that this construction can improve decoding speed while retaining much of the target model’s accuracy.
- The method requires no model training. TMOT operates entirely during verification and can therefore be attached to an existing draft and target model pair without retraining either model. The practical calculation restricts transport to at most 64 tokens and performs five Sinkhorn iterations. Verification occupies 1.61 to 2.80 percent of the measured time in the reported timing experiments.
- The evaluation covers several model pairs and task types. The experiments include three draft and target model pairs, two code benchmarks, mathematical reasoning, and four languages from MGSM. The paper also reports cases with appreciable accuracy loss instead of presenting the method as distribution preserving.
Critical weaknesses
1. The related work and baseline comparison are incomplete
The largest weakness is the omission of foundational and directly competing work on relaxed speculative verification. TMOT cites Leviathan et al. only for distribution-preserving speculative sampling, although that paper already introduced lenient verification and analyzed the resulting probability distortion. The manuscript does not cite Yin et al., who formally characterize the optimal tradeoff between rejection probability and deviation from the target distribution. It also omits DIVERSED and Faster Cascades via Speculative Decoding, both of which develop training-free pareto optimal relaxed verification and evaluate the resulting quality versus speed tradeoff.
At minimum, the following papers should be discussed:
- “Fast Inference from Transformers via Speculative Decoding,” Yaniv Leviathan et al., ICML, 2023. -- This papepr clearly discusses relaxing the verification step in the end.
- “A Theoretical Perspective for Speculative Decoding Algorithm,” Ming Yin et al., NeurIPS, 2024. -- Talks about the pareto optimality between speed and accuracy.
- “DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification,” Ziyi Wang et al., AISTATS, 2026. -- gives an exact way to traverse the pareto frontier in a training free manner, also accessible via huggingface transformers
- “Faster Cascades via Speculative Decoding,” Harikrishna Narasimhan et al., International Conference on Learning Representations, 2025. -- famous training free baseline, code publicly available.
These omissions affect both novelty positioning and experimental validity. Leviathan et al. provide the original lenient verifier. Yin et al. provide the natural theoretical framework for evaluating rejection against output distribution error. DIVERSED learns a context-dependent relaxation rule and evaluates overlapping tasks, including GSM8K and MBPP. Faster Cascades derives context-dependent and token-specific rules for choosing between the draft and target distributions.
TMOT should compare its complete quality versus speed curve against these alternatives using the same models, prompts, hardware, and task accuracy target. Published throughput values from different systems would not be sufficient. Without comparisons against the lenience rule from Leviathan et al., the distribution bias analysis from Yin et al., the learned relaxation in DIVERSED, and the deferral rules in Faster Cascades, the experiments do not establish that embedding-based optimal transport provides a better relaxation rule.
2. The experiments do not isolate the contribution of optimal transport
Before applying optimal transport, TMOT already requires that a proposed token appear among the target model’s three most probable tokens, exceed an absolute target probability, and sometimes receive at least half the probability of the target model’s preferred token. The acceptance threshold also changes with target entropy and sharpness. Earlier methods have already shown that target rank, probability ratios, and entropy can support relaxed verification.
The component analysis removes the candidate probability filter, adaptive Sinkhorn regularization, or entropy penalty, but every reported variant retains optimal transport. The paper does not evaluate the complete probability and uncertainty conditions with the transport condition removed. It also does not compare against a target probability rule, a cosine-only rule, or an ordinary distribution difference such as KL divergence. Consequently, the current results cannot determine whether optimal transport contributes beyond the simpler probability conditions.
The theory does not resolve this question. Proposition 1 is the classical coupling identity between binary mismatch cost and total variation. The same identity underlies the rejection probability analysis in Leviathan et al. and Yin et al. Proposition 2 follows because TMOT assigns every token pair a cost no larger than the binary mismatch cost. A smaller transport value therefore follows by construction and does not show that static embedding proximity predicts safe token substitution.
The implemented score further differs from the ideal quantity in Proposition 2 because it truncates and renormalizes both distributions, adds entropy regularization, and performs only five Sinkhorn iterations. The paper acknowledges that the practical score is not an exact Wasserstein distance and is not bounded by the total variation result. The method therefore relies on empirical validation, making the missing no-transport comparison central.
3. The consequences of lossy verification are insufficiently characterized
TMOT can permanently accept a token that the target model would have rejected. This changes the text on which all subsequent predictions are conditioned. The paper reports task accuracy, but it does not measure deviation from the target model’s output distribution, accumulated disagreement across multiple accepted mismatches, or the types of errors introduced by low transport cost.
The lowest reported accuracy recovery is 89.27 percent. The paper also identifies one model and task combination in which standard vLLM speculative decoding is both faster and more accurate than TMOT. These results show that the relaxation is task dependent. A useful analysis would report which tokens and contexts are incorrectly accepted, whether errors accumulate with generation length, and whether static embedding similarity remains informative when the same token has different meanings in different contexts. Such an analysis is particularly important because Yin et al. and the original lenience analysis of Leviathan et al. provide explicit ways to reason about the distributional cost of relaxed acceptance, while TMOT provides no comparable bound.
4. The exposition and notation are poor
The manuscript makes a relatively direct procedure difficult to follow by introducing several paper-specific names for the same operations. The distribution comparison is called “probability-measure alignment,” “predictive-measure alignment,” “predictive-state geometric compatibility,” a “distribution-level geometric state gate,” and a “geometric compatibility signal.” The token probability filter is described as “candidate-specific probability admissibility,” “support-constrained candidate admissibility,” and the “Fatal Veto gate.” Terms such as “Target Manifold,” “AESA,” and “uncertainty-consistent acceptance” add terminology without clarifying the underlying calculations.
The notation also reuses closely related symbols for different quantities. The manuscript uses K for the number of retained target candidates and Kᵢ for the Sinkhorn kernel. It uses Sᵢᵈ and Sᵢᵗ for token sets, then uses Sᵢ for target sharpness. It uses δx for a point mass and δ for a numerical stabilizer. The Frobenius subscript in the transport objective and several matrix operations are introduced without explanation. The transitions from complete model distributions to truncated distributions, from exact optimal transport to the finite-step score, and from that score to the final acceptance rule require substantial reconstruction by the reader.
The presentation needs a comprehensive rewrite with one term for each concept, distinct notation for distinct quantities, and a direct separation of the token probability conditions from the distribution comparison. The related work should also explain how TMOT differs from the relaxed verification of Leviathan et al., the rejection versus distribution bias result of Yin et al., the static and dynamic ensemble approaches in DIVERSED, and the token-specific deferral rules in Faster Cascades. This is bare minimum
Questions for authors
a) How does TMOT differ from the lenient verification rule in Leviathan et al., the rejection versus distribution-bias analysis of Yin et al., DIVERSED, and Faster Cascades via Speculative Decoding? Please add this comparisons as baselines
b) Can the authors compare complete throughput versus task-accuracy curves against these methods using the same models, prompts, hardware, and decoding settings?
c) What improvement comes specifically from optimal transport when TMOT’s target-rank, probability, entropy, and sharpness conditions are held fixed?
d) Can the authors report a condition that retains all TMOT probability tests but removes the optimal transport test?
e) What evidence shows that cosine similarity between static token embeddings predicts whether substituting one token for another is harmless in the current context?
f) Which accepted mismatches cause the largest accuracy losses, particularly in the MGSM Chinese evaluation where accuracy recovery falls to 89.27 percent?
g) Does TMOT remain effective for open-ended or nonzero-temperature generation, where accepting one different token may alter a long continuation?