acceptodds
Under review as a conference paper at ICLR 2027

TMOT: Geometric Optimal Transport over Target Representations for Speculative Verification

Abstract

Speculative decoding reduces the serial cost of autoregressive generation, but its gains depend on how draft tokens are verified. Token-identity and probability-only comparisons do not explicitly represent relations between distinct tokens. We formulate predictive-distribution comparison as probability-measure alignment under a ground geometry: binary token-identity cost recovers total variation, while a cost induced by target token representations replaces uniform mismatch with graded geometric mismatch. TMOT (Target-Manifold Optimal Transport) implements this view using truncated, entropy-regularized, finite-step transport. Its verifier factorizes candidate-specific probability admissibility and distribution-level geometric compatibility, with entropy and sharpness adapting the allowable discrepancy. The resulting module is training-free and can be integrated into a speculative decoding pipeline without changing model parameters. Code-generation and mathematical-reasoning experiments yield up to end-to-end speedup, with task-dependent accuracy recovery and a small measured verification-time share. These results support a quality–efficiency trade-off for geometric relaxation; TMOT is a lossy verifier and does not preserve the exact target sampling distribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, Optimal Transport, Large Language Model Inference, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1fpXOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents Target Manifold Optimal Transport (TMOT), a relaxed verification method for speculative decoding. Standard greedy verification accepts a draft token only when it matches the target model’s preferred token. TMOT also accepts some nonmatching tokens. The proposed token must receive sufficient probability from the target model, and the draft and target probability distributions must be sufficiently similar under an approximate optimal transport calculation. The transport cost is defined using cosine distance between the target model’s static token embeddings. The acceptance threshold becomes stricter when the target model expresses greater confidence.

TMOT requires no additional model training or parameter changes. It retains the longest accepted prefix from an eight-token draft and uses the target model’s preferred token at the first rejection. Because TMOT can accept a token that the target model would not have selected, it does not preserve the target model’s output distribution. The authors evaluate three draft and target model pairs on code generation, mathematical reasoning, and multilingual mathematical reasoning. Across the 21 reported model and task combinations, TMOT provides speedups of 1.13 to 1.90 times relative to target-only generation, while retaining between 89.27 and 100 percent of the target model’s task accuracy.

Critical strengths

Strengths

  1. The proposed combination is original and has potential. Prior work has separately studied relaxed verification, semantic similarity, distribution comparison, and optimal transport in speculative decoding. However, I am not aware of an earlier method that uses static target token embeddings as the cost for optimal transport between draft and target probability distributions, together with separate target probability and uncertainty conditions. The experiments indicate that this construction can improve decoding speed while retaining much of the target model’s accuracy.
  2. The method requires no model training. TMOT operates entirely during verification and can therefore be attached to an existing draft and target model pair without retraining either model. The practical calculation restricts transport to at most 64 tokens and performs five Sinkhorn iterations. Verification occupies 1.61 to 2.80 percent of the measured time in the reported timing experiments.
  3. The evaluation covers several model pairs and task types. The experiments include three draft and target model pairs, two code benchmarks, mathematical reasoning, and four languages from MGSM. The paper also reports cases with appreciable accuracy loss instead of presenting the method as distribution preserving.
Critical weaknesses

1. The related work and baseline comparison are incomplete

The largest weakness is the omission of foundational and directly competing work on relaxed speculative verification. TMOT cites Leviathan et al. only for distribution-preserving speculative sampling, although that paper already introduced lenient verification and analyzed the resulting probability distortion. The manuscript does not cite Yin et al., who formally characterize the optimal tradeoff between rejection probability and deviation from the target distribution. It also omits DIVERSED and Faster Cascades via Speculative Decoding, both of which develop training-free pareto optimal relaxed verification and evaluate the resulting quality versus speed tradeoff.

At minimum, the following papers should be discussed:

  • “Fast Inference from Transformers via Speculative Decoding,” Yaniv Leviathan et al., ICML, 2023. -- This papepr clearly discusses relaxing the verification step in the end.
  • “A Theoretical Perspective for Speculative Decoding Algorithm,” Ming Yin et al., NeurIPS, 2024. -- Talks about the pareto optimality between speed and accuracy.
  • “DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification,” Ziyi Wang et al., AISTATS, 2026. -- gives an exact way to traverse the pareto frontier in a training free manner, also accessible via huggingface transformers
  • “Faster Cascades via Speculative Decoding,” Harikrishna Narasimhan et al., International Conference on Learning Representations, 2025. -- famous training free baseline, code publicly available.

These omissions affect both novelty positioning and experimental validity. Leviathan et al. provide the original lenient verifier. Yin et al. provide the natural theoretical framework for evaluating rejection against output distribution error. DIVERSED learns a context-dependent relaxation rule and evaluates overlapping tasks, including GSM8K and MBPP. Faster Cascades derives context-dependent and token-specific rules for choosing between the draft and target distributions.

TMOT should compare its complete quality versus speed curve against these alternatives using the same models, prompts, hardware, and task accuracy target. Published throughput values from different systems would not be sufficient. Without comparisons against the lenience rule from Leviathan et al., the distribution bias analysis from Yin et al., the learned relaxation in DIVERSED, and the deferral rules in Faster Cascades, the experiments do not establish that embedding-based optimal transport provides a better relaxation rule.

2. The experiments do not isolate the contribution of optimal transport

Before applying optimal transport, TMOT already requires that a proposed token appear among the target model’s three most probable tokens, exceed an absolute target probability, and sometimes receive at least half the probability of the target model’s preferred token. The acceptance threshold also changes with target entropy and sharpness. Earlier methods have already shown that target rank, probability ratios, and entropy can support relaxed verification.

The component analysis removes the candidate probability filter, adaptive Sinkhorn regularization, or entropy penalty, but every reported variant retains optimal transport. The paper does not evaluate the complete probability and uncertainty conditions with the transport condition removed. It also does not compare against a target probability rule, a cosine-only rule, or an ordinary distribution difference such as KL divergence. Consequently, the current results cannot determine whether optimal transport contributes beyond the simpler probability conditions.

The theory does not resolve this question. Proposition 1 is the classical coupling identity between binary mismatch cost and total variation. The same identity underlies the rejection probability analysis in Leviathan et al. and Yin et al. Proposition 2 follows because TMOT assigns every token pair a cost no larger than the binary mismatch cost. A smaller transport value therefore follows by construction and does not show that static embedding proximity predicts safe token substitution.

The implemented score further differs from the ideal quantity in Proposition 2 because it truncates and renormalizes both distributions, adds entropy regularization, and performs only five Sinkhorn iterations. The paper acknowledges that the practical score is not an exact Wasserstein distance and is not bounded by the total variation result. The method therefore relies on empirical validation, making the missing no-transport comparison central.

3. The consequences of lossy verification are insufficiently characterized

TMOT can permanently accept a token that the target model would have rejected. This changes the text on which all subsequent predictions are conditioned. The paper reports task accuracy, but it does not measure deviation from the target model’s output distribution, accumulated disagreement across multiple accepted mismatches, or the types of errors introduced by low transport cost.

The lowest reported accuracy recovery is 89.27 percent. The paper also identifies one model and task combination in which standard vLLM speculative decoding is both faster and more accurate than TMOT. These results show that the relaxation is task dependent. A useful analysis would report which tokens and contexts are incorrectly accepted, whether errors accumulate with generation length, and whether static embedding similarity remains informative when the same token has different meanings in different contexts. Such an analysis is particularly important because Yin et al. and the original lenience analysis of Leviathan et al. provide explicit ways to reason about the distributional cost of relaxed acceptance, while TMOT provides no comparable bound.

4. The exposition and notation are poor

The manuscript makes a relatively direct procedure difficult to follow by introducing several paper-specific names for the same operations. The distribution comparison is called “probability-measure alignment,” “predictive-measure alignment,” “predictive-state geometric compatibility,” a “distribution-level geometric state gate,” and a “geometric compatibility signal.” The token probability filter is described as “candidate-specific probability admissibility,” “support-constrained candidate admissibility,” and the “Fatal Veto gate.” Terms such as “Target Manifold,” “AESA,” and “uncertainty-consistent acceptance” add terminology without clarifying the underlying calculations.

The notation also reuses closely related symbols for different quantities. The manuscript uses K for the number of retained target candidates and Kᵢ for the Sinkhorn kernel. It uses Sᵢᵈ and Sᵢᵗ for token sets, then uses Sᵢ for target sharpness. It uses δx for a point mass and δ for a numerical stabilizer. The Frobenius subscript in the transport objective and several matrix operations are introduced without explanation. The transitions from complete model distributions to truncated distributions, from exact optimal transport to the finite-step score, and from that score to the final acceptance rule require substantial reconstruction by the reader.

The presentation needs a comprehensive rewrite with one term for each concept, distinct notation for distinct quantities, and a direct separation of the token probability conditions from the distribution comparison. The related work should also explain how TMOT differs from the relaxed verification of Leviathan et al., the rejection versus distribution bias result of Yin et al., the static and dynamic ensemble approaches in DIVERSED, and the token-specific deferral rules in Faster Cascades. This is bare minimum

Questions for authors

a) How does TMOT differ from the lenient verification rule in Leviathan et al., the rejection versus distribution-bias analysis of Yin et al., DIVERSED, and Faster Cascades via Speculative Decoding? Please add this comparisons as baselines

b) Can the authors compare complete throughput versus task-accuracy curves against these methods using the same models, prompts, hardware, and decoding settings?

c) What improvement comes specifically from optimal transport when TMOT’s target-rank, probability, entropy, and sharpness conditions are held fixed?

d) Can the authors report a condition that retains all TMOT probability tests but removes the optimal transport test?

e) What evidence shows that cosine similarity between static token embeddings predicts whether substituting one token for another is harmless in the current context?

f) Which accepted mismatches cause the largest accuracy losses, particularly in the MGSM Chinese evaluation where accuracy recovery falls to 89.27 percent?

g) Does TMOT remain effective for open-ended or nonzero-temperature generation, where accepting one different token may alter a long continuation?

Reviewer_29DbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper formulates speculative verification as probability-measure alignment under a ground geometry: binary token-identity cost recovers total variation, while a cost induced by target token representations gives graded geometric mismatch. TMOT implements this view using truncated, entropy-regularized, finite-step transport. Its training-free verifier factorizes candidate-specific probability admissibility and distribution-level geometric compatibility, with entropy and sharpness adapting the allowable discrepancy. Experiments on HumanEval, MBPP, GSM8K, and four MGSM languages with three draft-target pairs yield up to 1.90x end-to-end speedup, task-dependent accuracy recovery, and a small verification-time share. TMOT is a lossy greedy verifier and does not preserve the target distribution.

Critical strengths

A training-free, plug-in method. TMOT changes no model parameters and only modifies the verification step, so it is easy to adopt.

Critical weaknesses
  1. The experiments cannot support the paper's quality and efficiency claims

(1) Lossless baselines lose accuracy. Under greedy decoding, exact-match speculative decoding (vLLM) should reproduce the target-only outputs exactly. In Table 1 it still loses accuracy, by up to 12 problems (DSR1-Qwen MGSM-ZH: 80.40 -> 75.60; GSM8K: 89.99 -> 89.11). TMOT with K=1 (Table 4), which is also exact-match, matches vLLM rather than the target-only baseline. This suggests the "recovery" gaps mainly reflect numerical noise, not verification loss.

(2) Some reported accuracies are impossible. No integer number of correct problems produces these values:

  • Table 1, HumanEval (N=164): 69.36, 60.59, 68.59
  • Table 1, GSM8K (N=1319): 89.11, 76.24, 87.74, 48.06, 49.41
  • Table 1, MGSM (N=250): 74.20, 79.50, 43.40, 61.00

The setup states a single seed, so averaging over runs cannot explain them.

(3) Verification overhead is not small, and no baseline is compared. Table 2 shows verification takes 1.6% to 2.8% of total time. That is about 2.5 to 3.6 ms per round, or 4% to 8% of the target forward pass, while exact-match verification costs almost nothing. Relaxation raises MAT by only about 6% to 7% over exact match, so this overhead consumes roughly a third of the gain. Table 2 also reports no baseline timings, so the overhead claim cannot be compared against any other method.

  1. The theory does not describe the verifier that is evaluated, and the OT component is not shown to matter

(1) The theory does not cover the practical score. Propositions 1 and 2 hold only for exact OT on ideal measures, and Section 2.5 admits they do not bound the practical score D_i (truncated, entropy-regularized, 5 Sinkhorn iterations). Propositions 3 and 4 only restate the gate definitions. So no formal result describes what TMOT accepts or how much quality it loses.

(2) The geometry never compares the proposed token with the target token. The geometric gate R_i depends only on the two distributions. Under greedy drafting, the verifier never checks whether the draft token is close to the target token in embedding space, which contradicts the motivation of "graded mismatch between nearby and distant tokens."

Questions for authors
  1. How were the accuracies listed in Weakness 1(2) computed, given that no integer count of correct problems produces them? Were any results averaged over multiple runs?

  2. Why do exact-match methods (vLLM, and TMOT with K=1) not reproduce target-only accuracy under greedy decoding? Please rerun the target-only baseline in the same speculative harness and report recovery against it.

  3. What makes verification cost 2.5 to 3.6 ms per round? Please also provide Table 2 phase timings for all baselines under the same setup.

  4. How often does the OT gate R_i actually reject a token, compared with the probability gate G_i? If R_i rarely rejects, what does OT contribute?

  5. Please add an ablation that keeps G_i but replaces the OT score with (a) a TV threshold and (b) a cosine-only score. Does TMOT still outperform them?

  6. Make the geometric score depend on the proposed token, for example the expected cost between the draft token and the target distribution, so the geometry directly compares the proposed token with the target's choices.

  7. Derive a bound for the practical verifier, such as a per-token or sequence-level likelihood loss, instead of only for ideal OT.

  8. Evaluate at temperature > 0 and on open-ended generation.

Reviewer_uU5UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TMOT, a training-free, lossy verifier that compares draft and target predictive distributions using optimal transport (OT) with a cosine cost from target input embeddings. Non-exact proposals must pass both a candidate probability gate and a geometric state gate. The practical score uses truncated supports, entropy-adaptive regularization, and five Sinkhorn iterations. The procedure targets greedy decoding. Code and math experiments across three model pairs report 1.13×–1.90× speedups over target-only decoding, with task-dependent accuracy losses.

Critical strengths
  1. The two gates have distinct roles. Candidate admissibility requires target probability support, while transport compares predictive distributions on a shared support. The paper explicitly distinguishes this distribution-level score from candidate-specific semantic correctness and separates ideal OT results from practical approximations.

  2. Verification has low measured cost. Table 2 reports verification shares of 1.61–2.80% across nine profiled model–task settings. These measurements support the feasibility of using a transport score within the decoding loop despite its added computation.

  3. The evaluation makes trade-offs visible. Experiments cover three model pairs and report throughput, accuracy, and advanced tokens together. This allows readers to identify settings where accepting more tokens does not improve the overall speed–accuracy trade-off.

Critical weaknesses
  1. The exact control does not reproduce target-only accuracy. Section 3.5 states that K=1 rejects every non-exact proposal before OT. Yet Table 4 reports 46.95% HumanEval accuracy for Pair A at K=1, versus 50.00% for target-only greedy decoding; vLLM also reports 46.95%. Exact verification should reproduce target greedy outputs under matched settings, apart from numerical differences requiring investigation. This discrepancy makes the accuracy effects of relaxation difficult to isolate.

  2. The added value of transport is untested. Every Table 3 ablation retains OT, while K=1 disables all relaxation. No control preserves candidate admissibility and uncertainty calibration while replacing Sinkhorn with a cheaper geometric score. The ideal OT bound does not establish that the practical score identifies useful substitutions.

  3. The speed–accuracy advantage is inconsistent. On Pair B/MGSM-ZH, vLLM is both faster and more accurate: 52.14 versus 50.14 tokens/s and 70.40% versus 63.20% accuracy. Matched-quality curves against existing relaxed verifiers are needed to establish when the additional transport computation is worthwhile.

Questions for authors
  1. Clarification: Why do K=1 and target-only greedy decoding differ? Please compare matched examples, prompts, checkpoint revisions, tokenizer settings, stopping rules, and scoring, and identify the first diverging tokens and their target-logit margins.

  2. Additional ablation: Can you retain the candidate gate and uncertainty calibration while replacing transport with a distribution-weighted embedding similarity? A candidate-gate-only control, compared at similar accuracy, would also isolate what Sinkhorn contributes.

  3. Calibration and comparison: Were K, M, and the acceptance thresholds selected separately from the test sets? Please report speed–accuracy curves against the existing relaxed baselines on representative code and math tasks, especially MGSM-ZH.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.