- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses reproducibility issues in test-time query optimization (TTO) for late-interaction retrieval, where processing the same query individually or in a batch can produce bit-level differences in scores, selected vectors, gradients, and final optimization results. The authors attribute these discrepancies to batch-shape-dependent matrix multiplication kernels, TF32/BF16 tensor-core arithmetic, and nondeterministic accumulation in backpropagation. They propose ExactTTO, built on three components: certified working sets, tensor-core exact maximum computation, and a document-major fused loop with batch-invariant execution. Experiments on 14 ViDoRe splits and four BEIR datasets reproduce GQR's gains, achieving 3.05× the throughput of the GQR FP32 scorer and 3.95× that of FLASH-MaxSim FP32 at batch size 64, while matching 57.6M maximum values bit-for-bit against a dense reference and achieving 0.0% batch-induced query variation.
Critical strengths
-
Well-motivated and practically relevant problem. Batch invariance is an important but underexplored issue in retrieval-time optimization. The reported discrepancies are substantial: at batch size 64, 19.6% of queries produce different final scores with the GQR scorer, and in controlled batches of 16, all but 6 of 192 queries change their final scores in the autograd loop.
-
A coherent and principled design. The principle that approximation should determine which products to compute, but never their values unifies the three components into a consistent exactness framework. Theoretical bounds, including the FP32 gap-radius test and the tensor-core accumulation analysis, strengthen the technical rigor.
-
Strong correctness validation and practical gains. The paper reports zero bit-level discrepancies across 57.6M maximum values, batch invariance in all 960 comparisons, and agreement with the released GQR loop on top-5 results for 7,688 test queries. The throughput gains are substantial, and the ReFIT distillation results further demonstrate that the approach is not limited to GQR.
Critical weaknesses
-
Single-request performance regression is insufficiently addressed. The paper should provide a latency breakdown for certification, verification, working-set refresh, and updates, and discuss whether certification for iteration (t+1) can overlap with the Adam update at iteration (t).
-
Batch invariance and tensor-core exactness have different portability guarantees. Proposition 2's batch invariance rests on a fixed execution order and fixed reduction trees that do not depend on the batch shape, while exactness against the dense twin additionally requires the FP32 FMA routine and the tensor-core accumulation bound (Theorem 4), whose numerical constant is established via an A100-specific bit-accurate model.
Questions for authors
What accounts for the single-request latency overhead, and can the certification and refresh costs be reduced or overlapped with optimization updates?
4 comments
Follow-up before the discussion deadline
Thank you for your constructive review. Before the discussion closes on November 18 (AoE), we would like to ask whether our response addresses your two points: single-request latency, including what overlapping the certificate test with the update could save, and the separate portability guarantees of batch invariance and tensor-core exactness, which the revised paper now distinguishes and which we also checked on an H20. We are happy to clarify anything further, and if the response resolves your concerns, we would be grateful if you would consider reflecting this in your assessment.
Thank you for the response. Given that I have already provided a positive assessment, I will maintain my positive score unchanged.
Thank you
Thank you for reading our response and for your positive assessment. Your suggestions on single-request latency and on separating the two portability guarantees made the paper clearer and stronger. If any point would still benefit from clarification, we are happy to provide it before the discussion closes.
Response to Reviewer Yv3P
(Current numbers: see Update 3.)
Thank you for two concrete suggestions. We followed up on both.
W1 / Q. Single-request latency: where does the time go, and can it be reduced? We profiled the graph replays of the fused loop for the first four test queries of every combination, served alone, recording every kernel (new Table 27 and Appendix K). With the batched launch, a single request is latency-bound. With pools of at most 20 documents the phases launch (medians) 15 thread blocks for the certificate test and for merging, 120 for refreshes and 2 for FP32 verification. Verification alone takes 39% of the exact dense operator's replay time (median), because every FP32 recomputation is one sequential chain of h fused multiply-adds.
So we added a single-request launch configuration that spreads the same computations over more thread blocks. Verification gives each (page, token) pair its own warp with coalesced loads, refresh splits a page into up to 32 parts and keeps candidates by warp-wide selection, and merges reduce across a warp. No product changes its multiply-add order, and maxima and selected vectors are fixed by (value, index), so the results are bitwise identical; we checked every served test query of all 44 combinations, alone and in batches of 16 and 64. A step of the exact dense operator now takes 35 µs instead of 75 µs (GQR's scorer: 36 µs), and over the 44 A100 combinations single-request throughput goes from 0.49× to 1.03× that of GQR's scorer. Where the replay time goes now (medians):
Each kernel now takes a few microseconds; fusing kernels could cut the remaining launch costs.
Overlap. The certificate test of a token at step t+1 needs that token's displacement from the Adam update of step t. The dependency is therefore per token, and across tokens or thread blocks the test could overlap with other updates, for instance inside a fused update kernel. Before building that kernel we bounded what it could gain: subtract the certificate-test kernels' recorded time (25%) and keep all other kernels unchanged. Even with a free test, a certified step would take 48 µs (median) against 35 µs for the exact dense operator, and it would be faster on only 11 of 44 combinations. Picking the faster operator per combination would raise single-request throughput by 1.03× (geometric mean; at most 1.59×). Given that bound we did not build the fused kernel, and ExactTTO serves single requests with the exact dense operator. With every input update paid for in a persistent graph (Table 24), single requests in this configuration run at 0.96× GQR's scorer, and batches amortize the remaining costs (2.14× at B=64).
W2. Portability of batch invariance versus tensor-core exactness. Agreed, and Section 6.5 (Portability) now says so. By portability we mean that each guarantee holds on each GPU separately. Bitwise equality across GPU models would also need identical compiled arithmetic, so we report it below as an observation on two GPUs, not as a guarantee. Proposition 2 holds on any GPU for scorers whose outputs for a query depend only on its own tokens and documents, which our operators ensure through fixed thread layouts and summation orders. Exactness of the FP32 operators needs binary32 fused multiply-adds with round-to-nearest and no overflow or underflow. Only the tensor-core operators add a hardware assumption, the accumulation bound of Theorem 4. On the A100 we now measure the accumulation error on both tensor-core paths (1.3 billion products for the Triton filter and 1.3 billion for the WMMA refresh; largest error 24× below our γ′). We also reran all tensor-core and exactness experiments on an H20, a Hopper GPU with compute capability 9.0 like the H100, using the same kernel sources, inputs and operating points (Appendix L). On both tensor-core paths the largest error is 1.6×10^-6 of the summed magnitudes, 38× below our γ′ (A100: 24×). The lockstep audit (8.6 million maxima) and the tie and near-tie tests (9.1 million comparisons) find no differing bit. Every exact operator returns the same bits on the H20 as on the A100 for every query it serves, whereas FLASH-MaxSim's TF32 path agrees across the two GPUs on 0 queries. On the H20, the single-request launch configuration raises single-request throughput from 0.45× to 0.86× GQR's scorer (A100: 1.03×), with the A100's bits on all 2,782 served test queries. We have no Blackwell GPU, so that case is untested. On any other GPU the bound has to be re-established before the tensor-core operators are enabled. An audit gives empirical support, not a proof, so tensor-core exactness stays conditional on the bound, while the FP32 operators remain exact and batch-invariant without it.