- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper asks whether fixed-payload low-bit retrieval rankings reflect the learner or the export codec. With DINOv2 (and some SigLIP2) checkpoints held fixed, it compares 80-bit allocations Q8D10 versus Q4D20 under a ladder of codec interventions: native grid (D0), FIT cell-mean decoder with fixed indices (D1), nearest-centroid reassignment (D2), and FIT-only global/coordinate-wise range calibration (C1/C2), plus own-latent PQ/OPQ and common-export training-grid controls. On SOP, C1 alone recovers +14.22 AP on frozen 4-bit triplet descriptors and reverses the allocation effect from −5.61 to +8.50; a historical 4-bit training advantage also collapses under equal calibration. CUB provides adverse transfer cases. The authors are careful not to propose a new quantizer or a universal bit-allocation law.
Critical strengths
The codec ladder does a good job of isolating the mechanism. D1/D2 versus C1 shows fairly cleanly that the SOP triplet failure is largely a scale/grid interface mismatch, not just bad centroid reconstruction. This is the most consequential positive result in the paper.
Critical weaknesses
The presentation cost is real. The clarity problems are not cosmetic: they make it hard to see what is new relative to classical quantization hygiene, such as Lloyd steps and scalar range fitting on a 7-point menu.
The generalizable contribution is limited for ICLR. There is no new lasting method, only 2–3 paired heads, two reused datasets, and conclusions that correctly refuse universality. As written, this reads more like a rigorous evaluation note than a venue-level advance.
Questions for authors
- Why does FIT always select R = 0.25, the menu floor, for Q4 triplet U? Is this latent scale collapse, activation saturation, or a menu-boundary artifact? Would a continuous R change the story?
- Can you report seed-level variability, say ≥5–10 independent paired fits, for the key allocation reversals, rather than only conditional product/species bootstrap on fixed models?
- After C1, what is the actionable recommendation for practitioners? Should they prefer Q4D20 on SOP-like data, or is the message simply “always FIT-calibrate before comparing learners/allocations”?
- A prospectively held-out dataset, or larger-scale retrieval, under the same protocol would materially strengthen the contribution.
3 comments
The rebuttal and the new experiments are solid. They effectively address my questions about the variance and boundary choices. I have updated my score to Weak Accept.
Thank you for your careful reassessment and constructive feedback. Your questions helped us clarify the range-boundary choices and strengthen the evidence on variability across training runs. We appreciate your time and thoughtful engagement with our work.
Response to Reviewer 5o9G
Why did the original range search select its lower boundary?
We re-examined the original two training seeds with a finer range search. The four-bit triplet models select ranges of 0.3076 and 0.3038, both above the original menu floor of 0.25. Reconstruction error decreases and clipping falls from about 5.5% to 1.4–1.5%. This shows that coarse menu spacing affected the selected value; choosing its floor alone did not identify latent collapse.
We refined the four-bit and eight-bit models separately using reconstruction error on the fitting data, then evaluated the fixed choices. The original allocation difference, four-bit minus eight-bit AP, is −5.61 points under the original fixed-grid export, +8.50 after the original range calibration, and +8.45 after refinement. Both seeds retain the positive calibrated difference. This reuses the four original frozen checkpoints and adds no independent training repetitions. Section 5 and Appendix G.2 give the separate scores and range choices. Refinement lowers reconstruction error without necessarily increasing retrieval AP. The same distinction appears in the new pairs (Appendix G.4); no test-AP tuning is involved.
The thirty-model learner cohort in Appendix G is separate from the original allocation comparison.
What do the independent repetitions establish?
We completed three new allocation pairs under the original training and export protocol. Each seed trains a twenty-dimensional four-bit model and a ten-dimensional eight-bit model, both at ten bytes. The original triplet loss, architecture, data use, and training budget are retained. Calibration selects from the original seven ranges by reconstruction error on FIT, without using test AP.
All three pairs reverse: the mean allocation difference changes from −5.67 to +8.26 points. Between-pair standard deviations are 0.66 and 0.71 points. All six model-conditional product-bootstrap intervals exclude zero after Bonferroni correction for 95% family-wise confidence across the three training pairs and two exports. These intervals condition on the fitted models; they do not replace the reported training-seed variation. All three new pairs and the full evaluation set are included.
Appendix G.4 reports the new three pairs separately from the original two. The auxiliary five-pair means are −5.65 and +8.36 points; all five have the same sign reversal. These repetitions start from the same pretrained backbone with new head initializations and training streams; both backbone and head are fine-tuned. They share evaluation products. Continuous-range refinement is a secondary analysis; the replication above uses the original calibration rule.
Appendix G.1 separately reports ten learner pairs at fixed four-bit allocation and the newer-regime allocation pairs. Neither is counted as an original-condition replication.
What should a practitioner do after calibration?
Compare models under the intended deployment codec before choosing one, with separate fitting, validation, and final evaluation data. With equal codec-fitting work, this improves SOP selection by 5.43 AP points with DINO and 5.70 with CLIP. At full budget, OPQ-first selection followed by codec reselection reaches the same pipelines as joint selection; extra complexity has no established benefit. In-shop selection intervals include zero, so this is a procedure for making a fair comparison, not a guarantee of improvement or a universal preference for four-bit descriptors.
A SigLIP2 control compares training without quantization with four-bit training under the same final twenty-dimensional, four-bit calibrated scalar export, with parameters fitted separately to each representation. Across four paired seeds, AP without training-time quantization minus AP with four-bit training is +0.55/+0.61 points for triplet and −7.10/−4.77 for contrastive models at 7,200/14,400 updates. All four seeds retain these directions at both budgets. Under the same final ten-byte export, training without quantization yields higher AP for triplet learning, while four-bit training yields higher AP for contrastive learning at both tested budgets. This motivates evaluating training precision for each learner under the intended deployment codec. It is a separate training-precision comparison from the original allocation replication; the budgets use their own learning-rate schedules.
The initial In-shop protocol was fixed before training and uses the official query/gallery roles with item-separated fitting and validation data. It supplies additional-domain evidence. We distinguish this recorded evaluation procedure from the broader history of benchmark use. The original CUB and teacher controls are retained.