Several critical weaknesses are present in this work:
Unsubstantiated Claims Authors claim that the difference in score gaps arises because (L056-L060) "Classification trains against a fixed set of classes so every class is pushed below the correct one at every step, ... but retrieval is trained against a sampled subset ...". This claim is repeated multiple times in the paper, while there is no direct experimental evidence provided to support this plausible explanation. Similarly, the separation ratio is presented as explaining the asymmetry in Sections 5.1 and 5.2, which is a very strong claim to make without adequately isolating other factors. Classification and retrieval are very different overall: 1. Classification operates on a relatively small set of classes while retrieval can involve millions of documents (in fact, for the reranking results in Table 9, where the candidate set contains only 100 documents, the changes in top-1 range from 4.8% to 10% at W4, similar to classification answer changes in Table 1, although the scoring mechanism also differs); 2. Classification uses learned logits while retrieval uses cosine similarities, thus having different score distributions and geometries; 3. Comparing all classes during each training step does not necessarily imply large classification margins; 4. Linear-probe classification and cosine nearest-neighbor retrieval are inherently different decision mechanisms.
Poor Motivation for Preserving Top-1 Retrieval Results - The paper works under the assumption that changes in retrieval rankings due to quantization are problematic, but it is not clear why preserving the exact FP ranking should be the objective in the first place. In fact, the results suggest that retrieval models inherently produce closely scored candidates, making their rankings sensitive to small perturbations, and so should exact FP ranking preservation be the appropriate objective? For example, on CLIP ViT-L/14, 24.6% of top-1 results change while Recall@1 drops by only 1.3% (relative), suggesting that the FP ranking is not necessarily something that needs to be maintained exactly. Similarly, in the RAG experiments, 24–32% of generated answers change while exact-match accuracy drops by at most 1.6 percentage points. The authors also emphasize the relevant documents lost from top-k in Table 2 without analyzing the corresponding relevant documents gained, as that would explain why the net change in NDCG@10 is relatively lower than the top-k documents lost reported in the table. Overall, the analysis seems to point more towards an inherent sensitivity of retrieval rankings themselves, and it is not sufficiently justified why preserving these potentially brittle FP rankings is preferable to directly optimizing retrieval quality or downstream performance.
Writing Clarity - The paper is difficult to follow, and the language used is obtuse in places and requires a lot of rereading to understand what is being conveyed. In addition, the writing style needs significant improvement, as the material is hard to follow in many places. A few examples of this from the first few pages of the paper:
-
Introduction: The paper abruptly introduces the idea of interventions (L073–L077) for retrieval and classification towards the end of the introduction, without sufficiently buiklding up to them. L065 hints at using gap to accept or reject a solution while the term gap is spoken of only once before this and its not clear what it means by accept and reject here as this is not well placed in the introduction. Similarly the terms "top-1/top-2 gap" in the next line is not clear. There are several such jumps in introduction that make it hard to understand and build up to the contributions properly.
-
Related Work (Section 2): The entire section is hard to follow. The subsection "Ranking under compression" jumps between very different research areas and terminologies, including MoE routing, classification margins, uncertain rankings, semiorders, linear extensions, and embedding compression, without properly explaining these concepts or how they connect to each other or to the manuscript. Similarly, "Mixed-precision allocation" and the "Selective prediction and routing" usubsections are hard to follow and the discussion lacks a coherent explanation of the existing literature and how the proposed work is positioned within it.
-
Section 3 — Unnecessarily complicated mathematical formulation: The authors introduce extensive notation around permutations, reachable rankings, contender sets, and general top-k stability, only to eventually focus on the much simpler top-1 stability condition in their proposed methods. In fact, they explicitly state that the top-k condition is not usable in practice and that everything they propose relies on top-1. It is unclear why so much mathematical machinery is introduced when the main observation can be explained much more simply. In (L163–L165), the statement that "a chain of near-ties lets a candidate far below the top-1 overtake it" is confusing, as a candidate can overtake the top-1 only if its direct score gap from the top-1 is within 2\epsilon$$. Intermediate candidates and non-transitivity do not change this condition, so it is unclear what this discussion means.
Official Response to Reviewer dDbd
We thank the reviewer for finding our work interesting and convincing, and for the two suggestions, which we ran. Our general response covers the concern all three reviewers share, the explanation of the gap. The revised PDF shows changes in blue.
W1. The fixes are not striking and cost extra
The fixes are effective but not eye-catching, and they need extra cost.
Allocation costs nothing at inference: it is set once before deployment from a few unlabeled inputs, and the deployed model runs like any other 3- and 4-bit model. Routing uses the gap per input: sending the 25% of inputs with the smallest gap to full precision recovers 85 to 93% of the lost accuracy.
W2. The root cause of the gap
The paper mainly observes the gap between the top two scores. The reviewer asked for the root cause of the difference between classification and retrieval, for example in the loss function or in the quantization procedure.
We tested both, and neither is the cause.
Quantization procedure. Six quantizers, three granularities, and quantized activations change how much the scores move but never remove the difference between the two tasks.
Loss function. Classifiers trained with sampled negatives, the way retrievers are trained, keep their separation within 12% of the original (general response, Test 1).
What sets the gap is what the model chooses among (see general response, Tests 2-4). A classifier picks among a few classes that are far apart. A retriever picks among many nearly equivalent items, so a small score change swaps them. The revised paper now gives this tested explanation (new Section 5.5).
Takeaway: the root cause of the gap is the set of candidates, not the loss or the quantizer. A better quantizer can make score changes smaller, but it cannot pull apart candidates that are nearly tied. So our two methods target the near-ties directly: allocation spends extra bits on the layers that move the top-two gap most, and routing sends inputs with a small gap to full precision.
W3 and Q1. Hashing-based retrieval
Can the analysis and experiments be extended to hashing-based retrieval, which is more challenging than cosine similarity?
Yes, and the analysis applies as is. It needs only a score per candidate and a threshold on how far quantization can move it. With binary codes, each bit that quantization flips moves the score by a fixed amount. We turned the embeddings of Qwen3-Emb-0.6B and GTE-large into binary codes and ranked them by Hamming distance, on four BEIR corpora, at 4-bit weights. Change rates are:
Over the 32 combinations of model, quantizer, code, and corpus, separation falls in 31, and the change rate rises in all 32. We have added these results to Appendix A13.
Takeaway: the analysis covers hashing too and predicts the outcome. A binary code keeps only the sign of each number in the embedding, so a tiny quantization error that pushes a number across zero flips a whole bit. Small errors become large score changes relative to the gap between the top two documents, so more top answers change.
Q2. Very low bits
Can the methods be adapted to extremely low bits (1- or 2-bit weights)?
ViT-B/16 and Qwen3-Emb-0.6B at 2-bit weights, using round-to-nearest / GPTQ:
1-bit weights need training-time methods and fall outside post-training quantization. We have added these results to Appendix A14 and state in the Limitations that both tasks break at 2 bits.
Takeaway: at 2 bits, post-training quantization itself breaks both tasks, a limit of the quantization, not of our methods. The analysis still holds and flags the collapse in advance: almost no input is certainly safe. Our methods target 4 bits and mixes of 3 and 4 bits because post-training quantization is practical there.
We hope these clarifications and additional results resolve your concerns, and we welcome further feedback.