acceptodds
Under review as a conference paper at ICLR 2027

MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference

Abstract

Temperature-zero BF16 LLM inference is often treated as reproducible, yet the same request can emit different tokens when decoded alone or inside a larger batch. Existing fixes use batch-invariant operators or LLM-42's per-token verification, incurring cost even when most steps are stable. We ask whether verification can be applied exclusively to flipped tokens. Across five models, batch-induced token flips are sparse on the flip-rate benchmarks: on MATH500, Llama-3.1-8B flips on 0.48 of synchronous decode steps, and all tested models stay within the 0.3–1.3 range on MATH500, GSM8K, and HumanEval. K/V perturbations remain flat before flips, while low top-1/top-2 logit margins expose much of the flip risk. MarginGate turns these observations into a verifier policy: it keeps BF16 decoding on high-margin steps, verifies only low-margin steps, and repairs confirmed mismatches by replacing the current K/V column. We evaluate on four datasets, calibrating on MATH500 and transferring to GSM8K, SharedGPT, and HumanEval. MarginGate restores 100 sequence-level deterministic decoding on Llama-3.1-8B and Qwen2.5-14B with 18.56/15.05 verifier trigger rates, reducing LLM-42's latency increment by 2.23x/1.99x relative to always-on verification. On DSR1-Distill-Qwen-7B, the same policy reaches determinism in a harder regime at 49.50 triggers.

Primary area:
optimization
Keywords:
deterministic LLM inference, batch invariance, BF16 numerical nondeterminism, LLM serving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EieUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies changes in greedy BF16 output caused by changes in the serving batch. It distinguishes the frequency of token flips from the frequency of low-margin stable steps that a verifier must also inspect. MarginGate commits high-margin predictions directly and recomputes low-margin steps using a deterministic verifier on the request’s committed cache. A token mismatch triggers current-column repair. The timed implementation uses rollback and re-decoding because the evaluated API does not support the proposed column write. Thresholds are selected on MATH500 and transferred to other tasks. The main Qwen2.5-14B experiment improves sequence agreement from 6/120 to 120/120 on the selection set, with incomplete agreement on the transfer sets.

Critical strengths
  1. The distinction between a rare token-flip rate and the actual verification budget is useful. The stable near-tie analysis explains why sparse verification can work for one model but inspect most steps for another, despite similarly rare flips. The DeepSeek results provide an informative limitation rather than implying universal effectiveness.

  2. The manuscript carefully distinguishes calibration results, transfer results, oracle repair, and the online rollback prototype. It explicitly acknowledges that recomputing from a perturbed committed cache does not establish agreement with the batch-1 reference trajectory. The observed improvements in sequence agreement are substantial and demonstrate that margin-triggered verification can reduce this source of variation.

Critical weaknesses
  1. The method reduces disagreement but does not yet provide dependable agreement with the reference sequence. The 120/120 result is used to select the threshold, and the fixed threshold yields 119/120, 113/120, and 118/120 agreement on the three transfer tasks. Earlier cache deviations remain, so even verified steps can differ from the reference. The oracle current-column experiment does not resolve this issue because it copies reference columns at known flip positions. The remaining failures matter for applications that require reproducible output.

  2. The evidence does not establish a runtime advantage over existing approaches at comparable sequence agreement. MarginGate is timed in HF eager, LLM-42 in vLLM, and Batch-Invariant Ops in SGLang, each with its own BF16 baseline; the latter two timing logs do not include sequence-match measurements. The actual Qwen2.5-14B rollback overhead is 75.2%, while 23.9% excludes cache catch-up. In addition, several appendix entries are provisional and sequence agreement is not reported for batch-size scaling outside B=8. A comparison must retain all necessary work and measure both time and agreement within a common stack.

Questions for authors
  1. Among the held-out sequence failures, how many result from a skipped high-margin flip, and how many occur because the verifier uses a cache that already differs from the reference? Is there a fallback that recovers these failures, and has its cost been measured?

  2. Can you compare MarginGate and the principal baselines in the same serving stack, reporting absolute runtime and sequence agreement together, with cache catch-up included? Which provisional results have complete measurements available?

Reviewer_WJVfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MarginGate, a selective verifier for batch-sensitive greedy decoding in bfloat16 (BF16). It uses the top-1/top-2 logit margin to select low-margin steps for deterministic recomputation from the request's committed cache; high-margin steps stay on the default BF16 path. The paper shows why infrequent token flips can still require frequent verification. Experiments examine margin-based selection, key–value (KV) cache deviations, sequence agreement with batch-1 decoding, and runtime costs across several models. On the Qwen2.5-14B MATH500 threshold-selection set, MarginGate matches all 120 reference sequences, although the timed rollback prototype adds substantial overhead.

Critical strengths
  1. Distinguishes flip frequency from verification rate. Synchronous token flips occur at only 0.3–1.3% of steps, but an online gate must also select stable steps with small margins. The model-dependent rates of these stable low-margin steps, together with the KV-cache diagnostics, help explain when sparse verification is feasible and when it becomes expensive.

  2. A simple gate with promising sequence-agreement results. The margin is available from the existing BF16 forward pass, and high-margin steps require no recomputation. On the Qwen2.5-14B MATH500 selection set, sequence agreement improves from 6/120 to 120/120. Fixed-threshold transfer provides further evidence, though agreement is imperfect. The paper also clearly labels completed, provisional, and oracle results.

Critical weaknesses
  1. The timed system does not establish an efficiency advantage under matched conditions. The intended current-column KV write is not implemented in the timed prototype, which rolls back and re-decodes the step (§3.4). The full Qwen2.5-14B/MATH500 overhead is +75.2%; +23.9% excludes cache catch-up (§4.3). LLM-42 and batch-invariant operators are timed on different inference stacks, and those logs do not record sequence agreement. These results cannot establish whether MarginGate reduces total latency relative to deterministic alternatives at comparable output agreement.

  2. Sequence-level reliability needs broader online validation. The 120/120 MATH500 result selects the threshold, so it is not an independent test (§4.1). At the fixed threshold, Qwen2.5-14B matches 113/120 SharedGPT sequences, while several other model-level full-match results remain provisional. The heterogeneous-batch match result uses oracle reference-column repair (Appendix A.5). The paper discloses these limits, but online sequence agreement remains less well tested across workloads and batch configurations. Section 3.4 also acknowledges that retained KV columns can differ from the reference; their effect on later token decisions is not quantified.

Questions for authors
  1. Comparable end-to-end cost. Can you compare MarginGate, LLM-42, and batch-invariant operators on one inference stack with matched batch and output settings, reporting both total latency and sequence agreement? The comparison should include cache catch-up. If direct current-column writes remain unavailable, a comparison using the completed rollback prototype would still help assess the practical benefit of selective verification.

  2. Online fidelity and retained cache differences. Can you report sequence agreement on held-out MATH500 prompts and heterogeneous batches using the actual online verifier? Section 3.4 already notes that matching tokens can leave numerically different KV columns. Can you quantify their effect on later token decisions, for example by comparing the current policy with a variant that writes the verifier's column even when the tokens match?

Reviewer_UdRNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies batch-dependent numerical nondeterminism in greedy BF16 LLM decoding and proposes MarginGate, a selective verification policy based on the top-1/top-2 logit margin.

  • High-margin steps are accepted directly, while low-margin steps are recomputed using a verifier operating on the committed cache.
  • Matching tokens retain the tentative BF16 cache column; mismatches trigger token correction and current-column repair.
  • Across five models and three benchmarks, the paper finds that synchronous token flips are rare, but the verification budget depends strongly on how many stable steps also have low margins.
Critical strengths
  1. Useful characterization of selective verification costs. The paper distinguishes rare token flips from the potentially much larger fraction of stable, low-margin steps requiring verification. Its analysis across five models identifies both favorable settings and cases where margin-based verification remains expensive.

  2. Simple mechanism with substantial empirical improvements in sequence agreement. MarginGate uses logits already available from the BF16 forward pass to trigger verification. On Qwen2.5-14B, it improves sequence agreement from 6/120 to 120/120 on the MATH500 selection set, with substantial improvements also observed when transferring the fixed threshold to three other datasets.

Critical weaknesses
  1. The end-to-end efficiency advantage is not established (§4.3, Tables 6 and 10–13). The confirmed HF eager prototype incurs 75.2% overhead, whereas the 23.9% figure excludes cache catch-up and does not represent the complete implementation cost. LLM-42 and Batch-Invariant Ops are evaluated on different inference stacks, and their timing logs do not record sequence agreement. Consequently, these measurements cannot establish a favorable runtime–consistency trade-off. A comparison within the same stack should report complete runtime and sequence agreement together.

  2. The evidence does not sufficiently validate the online repair mechanism (§§2.3, 3.4, and 4.5). The online verifier reads the committed cache, which may already differ from the reference state. Deterministic computation on this cache does not ensure agreement with the reference trajectory. Oracle experiments copy reference KV columns, while the observed KV spike partly reflects an already changed input token; neither directly establishes that online current-column repair is sufficient. Ablations comparing token-only correction, current-column replacement, replacement on token matches, and window repair are needed, alongside an analysis separating gate misses from verifier–reference disagreements.

  3. Generalization beyond the calibration setting remains insufficiently demonstrated (§§4.1 and 4.4, Appendix A.5). The 120/120 MATH500 result is obtained on the threshold-selection set. Fixed-threshold transfer is encouraging but still yields sequence mismatches, including 7/120 on SharedGPT. Most experiments use replicated prompts, and the heterogeneous-batch agreement results rely on oracle repair. Independent testing with the actual online verifier under heterogeneous requests, changing batch sizes, and longer generations would better substantiate the serving motivation.

  4. The comparison with the closest prior work is incomplete (Table 4). MarginGate’s main distinctions from LLM-42 are margin-triggered verification and retaining BF16 cache entries when tokens match. However, fewer verified tokens do not necessarily imply lower cost, since windowed and grouped verification can amortize computation more efficiently. The manuscript should clarify whether its LLM-42 baseline includes grouped verification, how the window size was selected, and how the reimplementation differs from the original. A latency–agreement curve would better establish the practical value of the proposed changes.

  5. Incomplete measurements and missing implementation details limit reproducibility (Appendices A and E). Several central results for additional models remain provisional, and no anonymous code archive is provided. The paper also needs fuller specifications of software versions, attention and verifier configurations, prefill handling, prompt selection, and timing procedures. Algorithm 1 should clarify the alignment between the predicted token and the cache column being replaced, including the additional computation required for cache catch-up. These details are particularly consequential for a method whose behavior depends on numerical execution choices.

Questions for authors
  1. Can you provide a comparison within the same inference stack that measures both complete runtime and sequence agreement for MarginGate and LLM-42? Please include cache catch-up in the total cost and clarify whether the LLM-42 baseline implements grouped verification and how its window size was selected. This would establish whether reduced verification translates into an end-to-end efficiency benefit.

  2. How closely does the online verifier agree with the reference trajectory when operating on the committed BF16 cache? Please distinguish failures caused by skipped verification from those where verification is triggered but the verifier itself disagrees with the reference. An ablation comparing token-only correction, current-column repair, and replacing KV even when tokens match would help establish the contribution of the proposed repair policy.

  3. Does the selected threshold remain effective on independent prompts under heterogeneous, dynamically changing batches using the actual online verifier? The heterogeneous-batch results currently use oracle repair, while the strongest MATH500 result comes from threshold selection. Please provide an independent evaluation, or clarify the scope of available evidence, and analyze the residual sequence mismatches on the transfer datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.