Independent Channel Mixtures for Budgeted Embedding Refinement

Can a small number of noisy ternary comparisons improve retrieval without retraining an encoder or accessing the retrieval gallery? We introduce Independent Channel Mixture (ICM), a method that refines query embeddings through a fixed budget of ternary comparisons. Each comparison presents a reference item and two anchors and returns a binary response indicating which anchor is closer to the reference. ICM estimates feedback consistency, accounts for ambiguity in the responses, and uses the existing embedding geometry to control the resulting update. Our analysis characterizes its response to uninformative feedback and provides bounds on changes in retrieval scores. In a post-hoc subset of $36$ dataset-representation combinations spanning eight dataset families, ICM achieves relative improvements in hierarchical $\mathrm{nDCG}@10$ of up to $16.48\%$ in the strongest condition, with a family-balanced average gain of $3.03\%$ across noise levels from $0.0$ to $0.4$. At noise level $0.5$, ICM preserves the frozen retrieval scores in almost every evaluated condition. These results demonstrate the potential of limited, imperfect feedback for adapting frozen embeddings to a retrieval task.