acceptodds
Under review as a conference paper at ICLR 2027

PhotoFinish: Adaptive Precision for Exact Language Model Sampling

Abstract

Sampling a token is a race with a single winner, yet output projection reads every weight at a fixed precision. PhotoFinish makes precision follow the race. A low-bit scan bounds token scores under fixed Gumbel perturbations, and an exact evaluation raises the threshold for remaining contenders. Only rows that can still win receive additional precision. Valid bounds certify each exclusion, preserving the same sampled token as a full scan. We characterize the minimum information required to determine the winner and analyze when another precision stage repays its cost. The analysis gives a tight worst-case guarantee for a reference access policy and explains why shrinking the candidate set need not reduce runtime. Across three large-vocabulary language models at small decoding batches, PhotoFinish outperforms the fastest of three evaluated full-scan samplers, including FlashSampling, with sampling-head speedups up to 2.61× and lower complete GPU decode-step latency during natural generation. Matched INT8 controls isolate the benefit of selective four-bit reads. PhotoFinish opens a path to exact sampling in which precision is allocated to the decision being made. Open-source code and reproduction materials accompany the submission.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language models, efficient inference, exact sampling, adaptive precision, GPU kernels

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Kd1POpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes PhotoFinish, an adaptive-precision approach to full-vocabulary language-model sampling. Rather than reading every output-projection weight at fixed precision, the method performs an initial low-bit scan, constructs interval bounds on Gumbel-perturbed token scores, and selectively retrieves additional precision for candidates that could still win. An exact evaluation of an initial anchor raises the exclusion threshold, allowing some tokens to be eliminated without reading their full BF16 weights.

The theoretical contributions include a minimum interval-certificate characterization, a tight worst-case competitive ratio of 5/2 for a reference policy in an abstract two-level access model, and a conditional analysis of refinement decisions using exponential-race representations. The latter establishes a refinement-beneficial interval of candidate counts under symmetry and a fixed incumbent.

The experiments evaluate four language models at small batch sizes. On the primary GPU, the three large-vocabulary models exhibit sampling-head speedups of 1.66–2.61× over the fastest evaluated full-scan baselines, translating into 1.45–4.42% reductions in complete GPU decode-step latency. The paper additionally provides matched INT8 comparisons, extensive coupled sample-agreement tests, and a limited cross-hardware evaluation.

The primary concerns are the conditional nature of the numerical exactness guarantee, the limited applicability of the theoretical cost models to the deployed implementation, and the modest and hardware-dependent end-to-end improvements.

Critical strengths

1. A principled formulation of adaptive-precision sampling.

The paper identifies an interesting opportunity in full-vocabulary sampling: determining the winning token does not necessarily require evaluating every candidate at full precision. The use of certified upper bounds, exact incumbents, and selective precision refinement provides a mathematically motivated alternative to fixed-precision output projection.

The interval-certificate framework is particularly useful because it distinguishes sufficient information for identifying the winner from the cost of retrieving that information. The conditional refinement analysis also captures dependencies between candidates induced by the shared sampling competition, rather than treating refinement outcomes as independent.

2. Careful experimental controls and transparent reporting.

The evaluation includes strong full-scan comparators, optimized and matched INT8 implementations, multiple precision policies, and complete decoding measurements. The matched comparisons help distinguish gains from selective four-bit reads from gains attributable to implementation choices.

The reported coupled agreement checks and recovery tests provide substantial empirical support for the implemented sampling procedure. The authors also report cases where the method provides little improvement or is slower than an INT8 alternative, which improves the credibility of the empirical discussion.

Critical weaknesses

1. The central exactness guarantee is conditional on numerical assumptions that are not established for the evaluated Tensor Core implementation.

Proposition 1 assumes that all computed intervals contain the target scores and that exclusions use valid incumbent bounds. Although the corresponding mathematical argument is correct under these conditions, its applicability to the implemented GPU algorithm requires a rigorous enclosure of the actual finite-precision computations.

Appendix D.9 derives a numerical error budget but explicitly assumes a block-FMA forward-error model for the BF16 Tensor Core path. The authors acknowledge that the PTX specification does not establish the required accumulation order, rounding behavior, or subnormal handling. Consequently, the error model is an assumption rather than a verified instruction-level guarantee.

This distinction is consequential: if the computed interval excludes the true reference score, the pruning rule may eliminate the actual winner. Conservative recovery mechanisms protect against detected numerical failures, but they do not establish correctness when an assumed error bound is violated without detection.

Furthermore, Eq. (1) describes sampling with ideal continuous Gumbel variables, while the implementation guarantees agreement with a finite-precision full-scan reference under its arithmetic assumptions. Equality to that reference does not, by itself, establish exact sampling from the ideal softmax distribution.

There is also a smaller formal issue in Theorem 2. Its strict inequality in Eq. (4) is not necessary under every deterministic token-ID tie rule. For example, a token with exact score 1 and favorable tie priority is certified against a competitor whose possible score lies in [0,1], although the required strict inequality fails. The authors mention lexicographic comparisons, but the theorem statement should explicitly incorporate this ordering when addressing the operational finite-precision setting.

These concerns do not invalidate the abstract exclusion principle, but they limit the strength of the claimed operational exactness guarantee.

2. The theoretical results do not sufficiently establish the significance or robustness of the practical optimization.

Theorem 3 proves a tight 5/2 competitive ratio in an abstract interval-access model using fixed per-row payload costs. The result does not provide a competitive guarantee for the deployed GPU policy, where kernel launches, candidate compaction, memory hierarchy, parallel execution, and synchronization contribute substantially to latency. Moreover, the lower-bound construction permits arbitrary nested interval responses that need not arise from the particular quantization scheme used in the implementation.

Theorem 5 requires symmetric effective candidate parameters and a fixed incumbent. These assumptions are restrictive for real language-model logits, where candidate intervals are heterogeneous and refinement can update the incumbent. The paper acknowledges these limitations but does not establish that its theoretical refinement-band structure governs the practical policy.

The experimental improvements are also substantially less compelling when measured at the complete decoding-step level. Although sampling-head acceleration can be large, the reported end-to-end GPU latency reductions are only 1.45–4.42% on the primary platform for the large-vocabulary models. The incremental benefit of dynamic stage selection over the best fixed four-bit strategy is particularly small: 0.66% at batch one, with an uncertainty interval including zero, and 0.26% at batch four.

Cross-hardware transfer raises an additional concern. On H800, the batch-four PhotoFinish head takes 356.42 microseconds, compared with 262.50 microseconds for the matched INT8 implementation, while the complete decoding-step improvement over the fastest full scan is not established.

Taken together, the results support a useful optimization in selected operating regimes, but provide limited evidence for a broadly advantageous adaptive policy or a sufficiently strong connection between the theoretical guarantees and observed GPU performance.

Questions for authors
  1. Can the authors justify the block-FMA error bound used for the Tensor Core implementation, or explain how correctness is guaranteed when this assumption does not hold?

  2. Does exact sampling mean agreement with the finite-precision reference or exact sampling from the ideal softmax distribution, and how are numerical ties handled in Theorem 2?

  3. Which theoretical predictions from Theorems 3 and 5 actually guide the implemented policy, and how well do they explain the measured GPU performance?

  4. Why does PhotoFinish perform substantially worse on H800, particularly at batch four, and how does it compare with the strongest matched INT8 baseline across different hardware?

  5. How meaningful is the minimum-certificate benchmark in Theorem 2 when actual token scores share the same hidden state and quantization structure rather than varying independently?

  6. Can the adversarial intervals used to establish the 5/2 lower bound in Theorem 3 arise from the actual quantization scheme, and how relevant is this guarantee to practical GPU execution costs?

  7. How often do real language-model candidate sets satisfy the symmetry and fixed-incumbent assumptions of Theorem 5, and how does violating these assumptions affect the predicted refinement interval?

  8. How sensitive are the interval bounds in Eq. (3) to the choice of diagonal metric, and how do loose bounds affect survivor counts and the overall computational cost?

  9. How does the implementation guarantee correct pruning under extreme temperatures, severe cancellation, underflow, or nearly tied scores, particularly when the assumed Tensor Core error model may not hold?

  10. How sensitive are the reported speedups to the development-calibrated candidate-count thresholds, and do these thresholds remain effective across different prompts, models, and workloads?

  11. Given that several improvements are below 1%, can the authors establish their statistical reliability across independent process runs, GPU operating conditions, and compiler configurations?

  12. How do the additional low-bit storage requirements and substantial temporary preprocessing allocations affect the practical benefits of PhotoFinish, especially for memory-constrained deployment or short generation workloads?

Reviewer_fWrgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the memory bandwidth bottleneck caused by the 'large vocabulary LM head weight transfer' in large language models when decoding with small batches. It proposes an adaptive LM head weight precision sampling method called PhotoFinish. PhotoFinish first uses the Gumbel-max trick to equivalently transfer the randomness in the sampling process, turning 'probability sampling' into a deterministic 'maximization problem.' Then, the method quickly filters the entire vocabulary using low-bit (4-bit) precision, safely eliminating the vast majority of unlikely 'runner-up' words with strict mathematical intervals and thresholds. After that, only a small number of tightly competing candidates that might overtake are dynamically assigned higher precision (reading the lower 4 bits or full BF16 weights), significantly reducing unnecessary memory bandwidth usage. The paper not only derives and proves things theoretically, but experiments also show that, while keeping the sampling results 100% lossless, PhotoFinish achieves up to 2.61× speedup in the sampling heads for small-batch scenarios of large vocabulary language models, and reduces end-to-end single-step decoding latency by 1.45% to 4.42%.

Critical strengths

A new paradigm with clear theoretical backing: it breaks the traditional mindset of 'fixed-precision scanning of the entire vocabulary' and, for the first time, allows computing precision to be dynamically allocated according to decision difficulty. It also provides a solid theoretical explanation for why having very few or very many candidates doesn’t necessarily lead to speedup (the so-called 'middle-ground theory').

Critical weaknesses
  1. Questionable practical value: First, PhotoFinish requires extra GPU memory because it needs to keep both FP and INT quantized weight versions in memory simultaneously. While this memory overhead is acceptable for 1B~2B models, is it really worth it for super large models like 70B+? The paper doesn’t explore bigger-scale models. Second, large models nowadays are often used in multi-GPU tensor parallel setups, and in real-world production, techniques like Top-k sampling and speculative sampling are common. Although the paper mentions these as future work, I think whether the method here can effectively work with these practical optimizations is more important for showing its real-world value.
  2. Sensitivity to the calibration set: This method relies on the calibration set for offline calibration. How often does it conservatively hold back or trigger a full scan when facing OOD? We need to specifically assess this to see how sensitive it is to the calibration set.
Questions for authors
  1. Calibration Sensitivity: How sensitive is the method's effectiveness to the choice of diagonal matrix Lambda and calibration sets? Can failure statistics for each model (such as conservative retention rate or frequency of triggering full scan rollback) be provided under out-of-distribution (OOD) prompts?
  2. Temperature effects: How does the system's performance fluctuate with changes in temperature? Does the author show ablation experiments illustrating how the number of candidate sets and end-to-end latency change across different temperature ranges, especially how it performs in scenarios where the probability distribution is extremely flat?
  3. Practical Value: Is the static memory overhead worth it for ultra-large models (like 70B+)? In multi-GPU tensor parallel scenarios, when combined with methods like Top-k sampling and speculative sampling, can the approach in this paper effectively work with these optimization techniques used in real applications? You can briefly explain this in words, because I think this is more important for showing the practical value of the method.
Reviewer_164eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PhotoFinish accelerates full-vocabulary categorical sampling by allocating output-weight precision to the realized Gumbel-max competition. It scans packed high nibbles, constructs residual-norm score bounds, evaluates an exact incumbent, and reads low nibbles and/or BF16 rows only for candidates that cannot yet be excluded. With valid bounds and the same random key, exclusions preserve the winner of a specified finite-precision full-scan reference. The paper also characterizes minimum interval certificates, proves a tight 5/2 competitive ratio for a reference access policy in a two-level cost model, and analyzes when another refinement stage is worthwhile under a symmetric conditional model. Triton experiments on three large-vocabulary models at batches 1 and 4 report 1.66–2.61x complete-head speedups and 1.45–4.42% GPU decode-step reductions on an RTX 5090 D. Matched INT8 controls, coupled correctness checks, a smaller-vocabulary model, and an H800 confirmation help delimit these gains.

Critical strengths
  1. The main contribution is a useful application of bounded evaluation to output-weight precision. Earlier sampling methods already use bounds or avoid materializing logits; PhotoFinish instead reduces the precision of the weight reads themselves, refining only rows that can still win the fixed Gumbel race. The exact incumbent, residual-based bounds, and same-key recovery give a coherent way to reduce memory traffic while preserving the realized sample of the stated reference. This is a meaningful systems contribution even though the broader idea of eliminating candidates with bounds is established.

  2. The empirical validation supports the mechanism within its stated scope. Complete-head measurements include compaction and recovery, matched INT8 controls isolate the benefit of four-bit screening, and complete GPU decode-step measurements show that some gains survive model-body execution. The numerical appendix addresses rounding, underflow, temperature division, and ties in detail. The paper also reports less favorable results on TinyLlama and H800 rather than presenting the best head speedup as a universal outcome. Together these checks make the small-batch results reasonably convincing. The certificate and conditional-work analyses add useful explanations of information cost and refinement decisions, while clearly separating their assumptions from GPU latency.

Critical weaknesses
  1. The main limitation is the strength of the practical advantage over optimized alternatives on server hardware. On RTX 5090 D, the large-vocabulary models show 1.66–2.61x complete-head speedups, but complete GPU decode-step improvements are 1.45–4.42%. The H800 confirmation is more limited: for Qwen3 1.7B, B1 gains 1.34x at the head and 0.71% for decoding, while B4 has no clear gain over the fastest full scan and the matched INT8 head is faster than the frozen four-bit policy (Appendix D.3). The 5090 baseline also requires disabling FlashSampling's warp specialization for compiler compatibility; the H800 check appropriately retains native scheduling. These results establish a useful optimization in some settings, but leave uncertain how often four-bit screening is preferable to a well-optimized INT8 path in deployment. Additional resident storage of about 76–253 MiB for the large-vocabulary models is a further tradeoff. This limits the strength of the practical contribution, rather than invalidating the reported gains.

  2. The theoretical results help explain the design, but their connection to the implemented policy is limited. The tight 5/2 guarantee concerns a reference interval-access policy and payload costs, excluding GPU scheduling overhead. The refinement-band result assumes symmetry and a fixed incumbent, whereas the GPU uses development-calibrated thresholds. Empirically, dynamic stage selection improves on the best fixed four-bit route by only 0.66% at B1 and 0.26% at B4 on Llama, with the B1 interval including zero (Table 5). Thus the strongest demonstrated contribution is selective precision reading itself; the present results provide less evidence that the proposed adaptive stage choice adds substantial practical value.

Overall, the core mechanism and careful validation support weak acceptance, while platform-dependent gains and the limited demonstrated benefit of dynamic stage selection prevent a stronger recommendation.

Questions for authors
  1. Platform sensitivity: On H800, what is the complete GPU decode-step latency of the matched INT8 control? Which stage costs explain the frozen four-bit policy losing to INT8 at B4? If available, development-only policy selection on H800 would help establish whether this is mainly a policy-transfer issue or a limitation of four-bit screening on that platform. This comparison could materially strengthen the practical case.

  2. Role of the theory: Does the conditional analysis identify an empirically useful decision feature beyond the thresholds and best fixed routes already evaluated? Please clarify which implementation choices it directly informed and which results should primarily be understood as explanations of the abstract access problem.

  3. Numerical scope (clarification): Appendix D.9 explicitly identifies the block-FMA condition as an arithmetic assumption. Please make the scope of the exactness guarantee equally clear in the main text: which paths depend on this assumption, and what conservative execution option is available when it cannot be established? I have not identified a counterexample; this is a request to clarify the guarantee, not a claim that the implementation is incorrect.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.