acceptodds
Under review as a conference paper at ICLR 2027

The Undetected Damage of Quantization on Retrieval and How to Fix It

Abstract

Quantization rounds weights to a lower bit-width, and it is often judged by accuracy loss. This criterion is incomplete. The same quantized model that preserves classification accuracy changes 20-46% of top-1 retrieval results, and the aggregate ranking metrics reflect only part of that change. The top-1 result is guaranteed to survive quantization only when the gap between the two highest scores, logits in classification and query-document similarities in retrieval, exceeds twice the largest rounding error. In classification, the loss function pushes the correct class apart from every other, ensuring this gap, but in retrieval nothing separates the top-1 item from the second. This gap can be measured without labels. Before deployment, it predicts which models will break, and at deployment time it tells, per query, whether the quantized answer still matches the one full precision would have returned. This holds for most classification inputs and a minority of retrieval queries. That gap motivates a different fix in each setting. In retrieval, spending bit-width on the layers most sensitive to it recovers three-fifths to three-quarters of an extra bit's benefit for half its cost. In classification, routing the few low-gap inputs to full precision recovers most of the lost accuracy at a fraction of the cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Quantization, retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dDbdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper shows that the damage caused by quantization is tied to the score gaps between the top two candidates. In retrieval, spending extra bit-width on the layers whose quantization moves the gap most recovers most of an extra bit’s benefit for half its cost. In classification, routing the few low-gap inputs to FP recovers most of the lost accuracy at a fraction of the FP cost. Overall, the research problem is very interesting and important, which has not been well deep researched.

Critical strengths
  1. The proposed problem observing view is very interesting and important for research.
  2. The paper is well written and organized. Comprehensive experiments are conducted and rigid explanations are provided.
  3. The results are solid and convincing.
Critical weaknesses
  1. The solution is effecitve but not eye-catching. Extra costs are needed.
  2. The authors mainly observe the difference between Top-1 and Top2 results, and corresonding results are mainly based on the observations. I mean if the authors can mine the essence reason about why different tasks (classification and retrieval) produce these phenomena, maybe from the view of loss functions or the procedure of quantization strategies. Then the solutions may be more convincing and can address the problem at its root.
  3. As for the retrieval task, I recommond a more challenging task: hashing based retrieval which is more complex than cosine similarity-based retrieval.
Questions for authors
  1. Can the analyses and experiments extended to the hashing based retrieval?
  2. Can the proposed method be adapt to extrem low bits (W1/W2) quantization model?
2 comments
AuthorsOpenReview ↗

Official Response to Reviewer dDbd

We thank the reviewer for finding our work interesting and convincing, and for the two suggestions, which we ran. Our general response covers the concern all three reviewers share, the explanation of the gap. The revised PDF shows changes in blue.


W1. The fixes are not striking and cost extra

The fixes are effective but not eye-catching, and they need extra cost.

Allocation costs nothing at inference: it is set once before deployment from a few unlabeled inputs, and the deployed model runs like any other 3- and 4-bit model. Routing uses the gap per input: sending the 25% of inputs with the smallest gap to full precision recovers 85 to 93% of the lost accuracy.


W2. The root cause of the gap

The paper mainly observes the gap between the top two scores. The reviewer asked for the root cause of the difference between classification and retrieval, for example in the loss function or in the quantization procedure.

We tested both, and neither is the cause.

  • Quantization procedure. Six quantizers, three granularities, and quantized activations change how much the scores move but never remove the difference between the two tasks.

  • Loss function. Classifiers trained with sampled negatives, the way retrievers are trained, keep their separation within 12% of the original (general response, Test 1).

What sets the gap is what the model chooses among (see general response, Tests 2-4). A classifier picks among a few classes that are far apart. A retriever picks among many nearly equivalent items, so a small score change swaps them. The revised paper now gives this tested explanation (new Section 5.5).

Takeaway: the root cause of the gap is the set of candidates, not the loss or the quantizer. A better quantizer can make score changes smaller, but it cannot pull apart candidates that are nearly tied. So our two methods target the near-ties directly: allocation spends extra bits on the layers that move the top-two gap most, and routing sends inputs with a small gap to full precision.


W3 and Q1. Hashing-based retrieval

Can the analysis and experiments be extended to hashing-based retrieval, which is more challenging than cosine similarity?

Yes, and the analysis applies as is. It needs only a score per candidate and a threshold on how far quantization can move it. With binary codes, each bit that quantization flips moves the score by a fixed amount. We turned the embeddings of Qwen3-Emb-0.6B and GTE-large into binary codes and ranked them by Hamming distance, on four BEIR corpora, at 4-bit weights. Change rates are:

ModelQuantizerCosineSign of each dimension (1,024 bits)Sign of 256 random projections
Qwen3-Emb-0.6Bround-to-nearest33.4%47.6%71.0%
Qwen3-Emb-0.6BGPTQ23.1%37.5%59.7%
GTE-largeround-to-nearest20.0%37.9%61.6%
GTE-largeGPTQ15.6%32.0%54.3%

Over the 32 combinations of model, quantizer, code, and corpus, separation falls in 31, and the change rate rises in all 32. We have added these results to Appendix A13.

Takeaway: the analysis covers hashing too and predicts the outcome. A binary code keeps only the sign of each number in the embedding, so a tiny quantization error that pushes a number across zero flips a whole bit. Small errors become large score changes relative to the gap between the top two documents, so more top answers change.


Q2. Very low bits

Can the methods be adapted to extremely low bits (1- or 2-bit weights)?

ViT-B/16 and Qwen3-Emb-0.6B at 2-bit weights, using round-to-nearest / GPTQ:

Classification accuracy, full precision → 2 bitsChange rateInputs with separation ≥ 1 (certainly safe)
ViT-B/16, 21 classification tasks82.2% → 10% / 30%90% / 69%1% / 6%
Qwen3-Emb-0.6B, 11 classification tasks89.5% → 23% / 22%77% / 78%6% / 6%
ViT-B/16, 5 retrieval tasksNot applicable99% / 97%0% / 0%
Qwen3-Emb-0.6B, 5 retrieval tasksNot applicable100% / 100%0% / 0%

1-bit weights need training-time methods and fall outside post-training quantization. We have added these results to Appendix A14 and state in the Limitations that both tasks break at 2 bits.

Takeaway: at 2 bits, post-training quantization itself breaks both tasks, a limit of the quantization, not of our methods. The analysis still holds and flags the collapse in advance: almost no input is certainly safe. Our methods target 4 bits and mixes of 3 and 4 bits because post-training quantization is practical there.


We hope these clarifications and additional results resolve your concerns, and we welcome further feedback.

AuthorsOpenReview ↗

Gentle Reminder

Thank you again for the time spent reviewing our paper and for your suggestions, both of which we ran: hashing-based retrieval and 2-bit weights. Each answer in our reply opens with a summary of your point and ends with a short takeaway of the findings. With the discussion period ending soon, we would appreciate hearing whether these results resolve your questions or whether further changes would help strengthen the paper.

Reviewer_oYerOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how quantization of base models affects downstream performance on various tasks, such as classification and retrieval. The manuscript argues that while tracking the drop in classification accuracy due to quantization is useful, for retrieval, changes in Recall@k or nDCG@k are not necessarily indicative of the actual shuffling in rankings that happens underneath. The paper works under the assumption that these changes in ranking are problematic, and so the authors propose (1) a way of quantifying this shuffling in rankings, (2) a metric (which they call the separation ratio) that can be indicative of ranking instability, along with a calibrated stability check that does not require running both the full-precision and quantized models at deployment, and (3) a mixed-precision allocation strategy to reduce this shuffling in retrieval systems. They also propose a mitigation strategy for classification based on the top-two logit gap: if the gap is smaller than a calibrated threshold, they discard the output from the quantized model and run the full-precision model instead, thereby showing that they can recover most of the classification accuracy lost to quantization while routing only a small fraction of inputs to the full-precision model.

Critical strengths
  1. Extensive Evaluation - the manuscript evaluates quantization effects across two modalities (text, images), multiple model architectures a (they cover ViT, Qwen, BGE, E5, GTE families as encoders), 6 quantization strategies, 2 different bit widths (3, 4) and multiple image and text datasets for classification, and study retrieval on both text retrieval and text-to-image retrieval, with additional cosdideration of RAG system and cross encoder system.

  2. Interesting Observations - The finding that quantization can change retrieved documents while aggregate retrieval metrics change relatively little is interesting and worth investigating. This is an interesting problem to study where reproducibility matters post quantization.

  3. Simple interventions for loss in quantization performance - The proposed gap sensitive mixed precision allocation is label free and does not require additional training. Similarly the intervention for classificaiton based on the top two logit gap metric is intuitive and easy to implement.

Critical weaknesses

Several critical weaknesses are present in this work:

Unsubstantiated Claims Authors claim that the difference in score gaps arises because (L056-L060) "Classification trains against a fixed set of classes so every class is pushed below the correct one at every step, ... but retrieval is trained against a sampled subset ...". This claim is repeated multiple times in the paper, while there is no direct experimental evidence provided to support this plausible explanation. Similarly, the separation ratio is presented as explaining the asymmetry in Sections 5.1 and 5.2, which is a very strong claim to make without adequately isolating other factors. Classification and retrieval are very different overall: 1. Classification operates on a relatively small set of classes while retrieval can involve millions of documents (in fact, for the reranking results in Table 9, where the candidate set contains only 100 documents, the changes in top-1 range from 4.8% to 10% at W4, similar to classification answer changes in Table 1, although the scoring mechanism also differs); 2. Classification uses learned logits while retrieval uses cosine similarities, thus having different score distributions and geometries; 3. Comparing all classes during each training step does not necessarily imply large classification margins; 4. Linear-probe classification and cosine nearest-neighbor retrieval are inherently different decision mechanisms.

Poor Motivation for Preserving Top-1 Retrieval Results - The paper works under the assumption that changes in retrieval rankings due to quantization are problematic, but it is not clear why preserving the exact FP ranking should be the objective in the first place. In fact, the results suggest that retrieval models inherently produce closely scored candidates, making their rankings sensitive to small perturbations, and so should exact FP ranking preservation be the appropriate objective? For example, on CLIP ViT-L/14, 24.6% of top-1 results change while Recall@1 drops by only 1.3% (relative), suggesting that the FP ranking is not necessarily something that needs to be maintained exactly. Similarly, in the RAG experiments, 24–32% of generated answers change while exact-match accuracy drops by at most 1.6 percentage points. The authors also emphasize the relevant documents lost from top-k in Table 2 without analyzing the corresponding relevant documents gained, as that would explain why the net change in NDCG@10 is relatively lower than the top-k documents lost reported in the table. Overall, the analysis seems to point more towards an inherent sensitivity of retrieval rankings themselves, and it is not sufficiently justified why preserving these potentially brittle FP rankings is preferable to directly optimizing retrieval quality or downstream performance.

Writing Clarity - The paper is difficult to follow, and the language used is obtuse in places and requires a lot of rereading to understand what is being conveyed. In addition, the writing style needs significant improvement, as the material is hard to follow in many places. A few examples of this from the first few pages of the paper:

  • Introduction: The paper abruptly introduces the idea of interventions (L073–L077) for retrieval and classification towards the end of the introduction, without sufficiently buiklding up to them. L065 hints at using gap to accept or reject a solution while the term gap is spoken of only once before this and its not clear what it means by accept and reject here as this is not well placed in the introduction. Similarly the terms "top-1/top-2 gap" in the next line is not clear. There are several such jumps in introduction that make it hard to understand and build up to the contributions properly.

  • Related Work (Section 2): The entire section is hard to follow. The subsection "Ranking under compression" jumps between very different research areas and terminologies, including MoE routing, classification margins, uncertain rankings, semiorders, linear extensions, and embedding compression, without properly explaining these concepts or how they connect to each other or to the manuscript. Similarly, "Mixed-precision allocation" and the "Selective prediction and routing" usubsections are hard to follow and the discussion lacks a coherent explanation of the existing literature and how the proposed work is positioned within it.

  • Section 3 — Unnecessarily complicated mathematical formulation: The authors introduce extensive notation around permutations, reachable rankings, contender sets, and general top-k stability, only to eventually focus on the much simpler top-1 stability condition in their proposed methods. In fact, they explicitly state that the top-k condition is not usable in practice and that everything they propose relies on top-1. It is unclear why so much mathematical machinery is introduced when the main observation can be explained much more simply. In (L163–L165), the statement that "a chain of near-ties lets a candidate far below the top-1 overtake it" is confusing, as a candidate can overtake the top-1 only if its direct score gap from the top-1 is within 22\epsilon$$. Intermediate candidates and non-transitivity do not change this condition, so it is unclear what this discussion means.

Questions for authors
  • Its unclear why so much effort is spent in establishing that there is a difference between classification and retrieval under quantization, and how does it contribute to the core contributions. The analyses and mitigation of retrieval ranking changes does not depend on demonstrating that classification behaves differently. This comparison also shifts the focus away from what probably is a more important question: when and why do quantization-induced changes in retrieval rankings actually lead to deterioration in retrieval quality?
2 comments
AuthorsOpenReview ↗

Response to Reviewer oYer

We thank the reviewer for finding our work interesting, extensive, and practical. Our general response (item 1) covers the explanation of the gap you raised. The revised PDF shows changes in blue, and line numbers in our replies refer to that PDF.


W1. What makes retrieval more fragile than classification?

The reviewer finds no direct evidence for the training explanation in the introduction, and Sections 5.1 and 5.2 present the separation ratio as the cause without isolating other differences between the tasks.

We tested the four differences the reviewer listed. At 4-bit weights, none explains why quantization changes more retrieval answers. How the model is trained, and how the answer is scored (a trained classifier or cosine similarity to class averages), each moves the change rate by at most 2 points (general response, Tests 1 and 2). Single examples add candidates that nearly tie with the top-1, so adding them raises the change rate (general response, Test 3). More classes is not the cause: SUN397 has twice the classes of Cars, yet 16% of its answers change against 29%, because car classes score closer to each other. The reranker changes only 5 to 10% of answers for the same reason: its top answer leads by a far wider margin.

We have also retitled Sections 5.1 and 5.2 to present the separation ratio as a predictor, not the cause.

Takeaway: the tests show that training is not what sets the gap. What sets it is how many candidates nearly tie with the winner, not how many there are. This refines the paper's intuition, and the main findings and the two methods are unaffected.


W2. Why keep the original top answer?

Retrieval rankings are brittle by nature: on CLIP ViT-L/14, 24.6% of top-1 results change while Recall@1 drops by only 1.3%, and Table 2 omits relevant documents gained. Why preserve the full-precision ranking instead of optimizing retrieval quality?

Consider a quantized retriever that always returns a relevant document, but never the same as full-precision. Its Recall@1 is unchanged, yet every query gets a different answer, an inconsistency no team quantizing a retriever wants. Where relevance labels exist, the changes also hurt quality:

  • When the top document is relevant, 66% of flips replace it with one that is not (Table 20).

  • At 4 bits, 6.4% of queries with a separation ratio below 0.05 lose their relevant top document, against 0.0% when above 0.5.

  • At 4 bits, for 5 of 6 embedders, more queries lose a relevant top document than gain one (5.8% against 2.9% for Qwen3-Emb-0.6B). We have now added the "gained" column to Table 2.

Preserving ranking also helps quality: at the same budget, our allocation raises Recall@1 over reconstruction error (48.5% against 46.0%).

Takeaway: a quantized model is meant to replace full-precision, so giving the same answers matters even when Recall@1 holds, and where labels exist, the changes hurt more often than they help.


W3. Writing

Some terms used in the introduction are unclear, related work jumps between topics, and Section 3 has notation it never uses.

We thank the reviewer for pointing out the clarity issues precisely.

  • Introduction. We now define the gap before it is used, and introduce the two fixes only after the explanation that motivates them.

  • Related work. We have split "Ranking under compression" into three short paragraphs, and every subsection now ends with how it relates to this paper.

  • Section 3. We have moved the unused notation to Appendix B and rewritten the sentence on chains of near-ties (now L156 to 159) to state the condition directly: a candidate can overtake the top-1 only if its score is within 2ε of the top-1.


Question. Why compare retrieval with classification?

Why show that classification and retrieval differ, when the retrieval analysis and fixes do not depend on it? A more important question may be when ranking changes hurt retrieval quality.

The comparison does 3 things:

  1. Isolating the cause. The same quantized checkpoint changes 3 to 9% of answers as a classifier and 41 to 46% as a retriever (Table 1). Only the task differs, ruling out the model and quantizer.

  2. Informing the fixes. Few classification inputs are at risk, so they can be routed to full precision. Nearly every retrieval query is at risk, so retrieval needs the extra bits for some layers before deployment.

  3. Correcting a common misconception. Quantizers are often judged on classification accuracy, as if keeping accuracy meant keeping quality. Yet retrieval answers can change a lot even when accuracy barely does.

Ranking variations hurt retrieval quality at low separation ratios, where an irrelevant document can overtake a relevant top one (see W2).

Takeaway: the comparison shows that the gap determines the damage and which fix each task needs.


We hope the new analyses and clarifications answer your questions, and we welcome further discussion.

AuthorsOpenReview ↗

Gentle Reminder

Thank you again for your detailed review, which helped us improve the paper. We tested the explanation of the gap with controlled experiments and clarified the writing you pointed out. Each answer in our reply starts with a one-line summary of your concern and ends with a takeaway that can be read on its own. Since the discussion closes soon, we would be grateful to know whether our responses helped address your concerns or if there’s anything else we can do to make the paper stronger.

Reviewer_1b2hOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that a quantized model can break retrieval performance even maintaining its classification accuracy.

This paper reveals that this failure is due to the gap between top-2 model scores and that gap can be used to decide when to trust the quantization result and where additional precision is needed.

Allocating bit-width based on the top-2 score gap can fix the issue for both retrieval and classification.

Experimental results on two modalities backbones were conducted to show the quantization damage and how to fix the damage.

Critical strengths

1, The observation about quantization damage is interesting and the idea to fix this issue based on the top-2 score makes sense.

2, Top-2 score serves as a proxy for quantization damage and it can be used to fix the damage based on the score gap for both classification and retrieval.

3, Ablation studies are conducted to support the damage argument and validate the effectiveness of the solution based on the top-2 score gap.

Critical weaknesses

1, No test for the central causal explanation in Line 16-20. Maybe one experiment for a ViT classifier trained with sampled-negative loss and report the separation ratio and W4 top-1 change rate at matched FP accuracy.

2, Section 3 is presented as theory, but it seems largely restated classical results. For example, Line158-160. It is hard to understand how much theoretical novelty comes from Section 3.

3, Corollary 1 is to motivate calibrating SS instead of 2$\epsilon$. But there is no supporting measurement for this. It may be helpful to add the 2$\epsilon$-calibrated coverage and the distribution of S/2$\epsilon$ for classification and retrieval tasks.

4, The abstract number seems not matching Table 1, 14 to 46% retrieval (abstract) vs 33 to 46% (Table 1).

5, The main allocation baseline is very close to random. In the Table 36, reconstruction error is 37% while random split is 37.7%. The baseline gap sensitivity compared with is no better than chance.

6, Why does the stability check lose to a simpler baseline at the strictest level? Table 22, α\alpha=0.01, the check covers 67.7% while the plain margin under generic calibration covers 69.1%.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1b2h

We thank the reviewer for finding our work interesting and practical, and for proposing a specific experiment, which we ran. Our general response covers the concern all three reviewers share, the explanation of the gap. The revised PDF shows changes in blue, and line numbers in our replies refer to that PDF.


W1. No test of the explanation

The reviewer asked for a test of the training explanation in the abstract, proposing a ViT classifier trained with sampled negatives, with its separation ratio and 4-bit change rate at matched accuracy.

ViT-B/16 trained against 7 random wrong classes per step instead of all classes, on five datasets, at matched accuracy (within 2.1 points) and 4-bit weights:

DatasetSeparation: all classes / sampledChange rate: all classes / sampled
CIFAR1002.00 / 1.767.1% / 7.4%
Food1011.38 / 1.288.4% / 8.2%
TinyImageNet1.40 / 1.309.6% / 9.3%
SUN3970.45 / 0.4116.3% / 17.3%
Cars0.20 / 0.1829.4% / 31.3%

If sampled negatives made gaps small, these gaps would shrink. They barely do, and retrieval with the same models stays 10 to 150 times less separated. The abstract, the introduction, and the new Section 5.5 now give the tested explanation.

Takeaway: the proposed test shows that training is not what sets the gap. What sets it is how many candidates lie close to the winner (general response, Tests 2-4), which refines the paper's intuition. We thank the reviewer for suggesting this experiment, it made the paper stronger and added empirical evidence.


W2. Section 3 restates classical results

Section 3 is presented as theory but largely restates classical results, so it is unclear how much of it is new.

We have trimmed Section 3 to what it adds to the classical top-1 condition: the separation ratio, Corollary 1, and the stability check with its guarantee (W3). The rest is in Appendix B.

Section 3 does not claim the condition as new (L152 to 154). It proposes one theory for any score that ranks candidates, so the same measure and label-free check apply to both classification logits and retrieval similarities.


W3. No measurement behind Corollary 1

Nothing measures the gain of calibrating S (Corollary 1) instead of the worst case 2ε. The reviewer asked for the share of inputs accepted with 2ε and the distribution of S/2ε.

Below we show the share of inputs accepted at 4-bit weights, for α = 0.10 / 0.05 / 0.01:

SettingThreshold from 2εThreshold from S
Classification, ViT-B/1667.2 / 62.9 / 51.5%85.2 / 80.0 / 67.7%
Text retrieval, Qwen3-Emb-0.6B0 / 0 / 0%11.3 / 8.8 / 4.8%

Both keep the violation rate below α. S is at most half of 2ε for 93% of classification inputs and 99.8% of retrieval queries, because quantization usually moves the top score and its rival together, and S counts only the change in their gap. We have added both to App. A12.

Takeaway: calibrating S instead of 2ε accepts 16 to 18 points more classification inputs, and it is what makes the check usable for retrieval at all: with 2ε, no retrieval query is accepted.


W4. Apparently inconsistent numbers

The abstract says quantization changes 14-46% of retrieval answers, but Table 1 shows 33-46%.

The abstract range covers the 8 models and 6 quantizers of Table 10 at 4 bits, while Table 1 covers 3 models under round-to-nearest. So both numbers are correct, they cover different settings.


W5. Baseline no better than random

The allocation baseline, reconstruction error, does no better than random allocation (37.0% against 37.7%, Table 36), so beating it says little.

A random split can be a decent baseline here: at a 3.5-bit budget, any split gives half the layers 4 bits. Reconstruction error does no better because, across layers, it is unrelated to how much a layer moves the top-1/top-2 gap (correlation -0.04). Our gap-based allocator instead beats all baselines significantly. Section 6.1 now adds the random-split comparison.


W6. Plain margin wins at α = 0.01

At α = 0.01 (Table 22), the plain top-1/top-2 margin accepts 69.1% of inputs and our stability check 67.7%. Why does the simpler baseline win?

Our stability check accepts more at α = 0.10 and 0.05, the plain margin wins only at α = 0.01. Both accept an input when its top-two margin is large enough, and differ only in how "large enough" is set: ours from how much quantization moves the scores across all inputs, the plain margin from the inputs whose answer changed. At the strictest level, this makes ours slightly more cautious. We have reworded the comparison in Appendix A6.

Takeaway: both thresholds are set on the same quantity, the margin between the top two scores, and differ by at most 3 points in either direction.


We hope these responses and new experiments address your concerns, and we look forward to discussing any open points.

AuthorsOpenReview ↗

Gentle Reminder

Thank you again for your review and for proposing the sampled-negative experiment, which we ran and added to the paper. Each answer in our reply opens with a short summary of your point and closes with a takeaway, for additional clarity. As the discussion period ends soon, we would be grateful to know whether our responses address your concerns or if there’s anything else we can do to clarify any open questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for finding our work interesting and practical. All three asked what causes the gap between classification and retrieval.


We answered this with four controlled experiments: the gap comes from the set of candidates, which refines the intuition about training given in the submitted paper. The main results and both proposed methods do not depend on this explanation and are unchanged. We have uploaded a revised PDF, which shows changes in blue, and line numbers in our replies refer to that PDF.

For clarity, we define two terms again.

  • Change rate: the share of inputs whose top-1 answer differs between the full-precision and the quantized model.
  • Separation ratio: the gap between the top-1 and top-2 scores divided by twice the largest score change quantization causes. When it's above 1, the top-1 cannot change.

1. The candidate set sets the gap, not training (1b2h W1, oYer W1, dDbd W2). The paper's results and methods rely on the gap itself, not on why it differs between classification and retrieval. For why it differs, the submitted paper offered a brief intuition in the abstract: retrievers train against sampled negatives, classifiers against all classes. We ran four tests to check it, at 4-bit weights (except Test 4):

  • Test 1, classifiers trained like retrievers. We trained ViT-B/16 classifiers against 7 random wrong classes per optimization step instead of all classes. Separation ratio drops by at most 12%, and the change rate moves by at most 1.9 points. So, training against all, fixed, classes is not what makes classification gaps large.

  • Test 2, same embeddings, four ways to pick the answer. For each of the two frozen text encoders on three datasets, we quantized the encoder once and picked the answer from the same embeddings in four ways: a classifier trained against all classes, the same classifier trained against one random wrong class per optimization step, the closest class average, or the closest single example, which is retrieval. Changing the loss, or replacing the classifier by class averages, moves the change rate by at most 2 points. Picking single examples changes 3 to 6 times more answers than picking class averages. So, what matters is whether the candidates are classes or single examples.

  • Test 3, adding examples to the candidates. With the same encoder, we started from the 77 class averages of Banking77 as candidates and added single examples to the candidate pool, up to 5197 candidates. The change rate rises from 5.4% to 32.6%, and the same happens on two more datasets and a second model. Changing only the candidate pool moves the damage from the classification level to the retrieval level.

  • Test 4, no classifier training at all. CLIP's image encoder, never trained as a classifier, picked either a class name (zero-shot) or the nearest other image, at 8-bit weights. Under 2% of answers change with class names, against up to 7% with images. Without any classifier training, a few distinct candidates are stable while many similar ones are not.

The edited abstract and introduction now give the tested explanation, the new Section 5.5 discusses it, and Appendix A11 reports the four tests (Tables 37 to 40).

Takeaway: training is not what sets the gap, the set of candidates is what matters.


2. Section 3 and writing (1b2h W2, oYer W3). Section 3 is now trimmed to what the paper uses (the top-1 condition, the separation ratio, Corollary 1, and the stability check), with the rest moved to Appendix B. The theory needs only a score per candidate, so it covers classifier logits, cosine similarities, cross-encoder scores, and the Hamming distances of hashing. This is what lets one analysis compare classification and retrieval. We have also rewritten the introduction and related work.


3. Further results. We have also added the stability check calibrated with 2ε against S (App. A12, 1b2h W3), hashing-based retrieval (App. A13, dDbd W3), and 2-bit weights (App. A14, dDbd Q2). Relevant documents lost and gained are now in Table 2 (oYer W2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.