acceptodds
Under review as a conference paper at ICLR 2027

Independent Channel Mixtures for Budgeted Embedding Refinement

Abstract

Can a small number of noisy ternary comparisons improve retrieval without retraining an encoder or accessing the retrieval gallery? We introduce Independent Channel Mixture (ICM), a method that refines query embeddings through a fixed budget of ternary comparisons. Each comparison presents a reference item and two anchors and returns a binary response indicating which anchor is closer to the reference. ICM estimates feedback consistency, accounts for ambiguity in the responses, and uses the existing embedding geometry to control the resulting update. Our analysis characterizes its response to uninformative feedback and provides bounds on changes in retrieval scores. In a post-hoc subset of dataset-representation combinations spanning eight dataset families, ICM achieves relative improvements in hierarchical of up to in the strongest condition, with a family-balanced average gain of across noise levels from to . At noise level , ICM preserves the frozen retrieval scores in almost every evaluated condition. These results demonstrate the potential of limited, imperfect feedback for adapting frozen embeddings to a retrieval task.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
embedding refinement, interactive retrieval, ternary comparisons, noisy feedback, active learning, hierarchical retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qWLNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Independent Channel Mixture (ICM), a method for refining frozen query embeddings using noisy ternary comparisons, without retraining the encoder or accessing the retrieval gallery.

Critical strengths
  1. The paper provides explicit derivations for the mathematical formulation and theoretical evidence.
  2. The evaluation is extensive, covering multiple datasets and pretrained embedding models across multiple modals
Critical weaknesses

The ablation studies are insufficient: the contributions of query-local response mixtures and geometry-scaled delivery are not evaluated independently. Variants that use each response model alone and a variant that replaces geometry-scaled delivery with a fixed update scale can provide more direct evidence of each component’s contribution.

Questions for authors

See weaknesses

3 comments

Thank you for the careful review and for identifying the missing component comparisons. We have added the single-channel and fixed-scale controls you requested in Appendix E.2, "Repeated component ablations" (Table 10).

The study covers 36 historical dataset-representation combinations across eight families, and a separate confirmation panel comprising two representations on each of three disjoint cohorts in four families. Both panels use budgets B∈{8,32,128}B \in \{8, 32, 128\} and informative noises η∈{0,0.2,0.4}\eta \in \{0, 0.2, 0.4\}. The pure-noise results are reported separately further.

For the response mixture, we added logistic-only and indifference-only policies. The selected link is used throughout inference, acquisition, and evidence calculation, so these controls evaluate the resulting complete single-channel policies. We also added uniform weights, fixing the two acquisition and delivery weights at one half while retaining proper mixture evidence for the gate.

For geometry-scaled delivery, we replace the query-dependent coefficient with each of three constants, α∈{0.10,0.25,1.00}\alpha \in \{0.10, 0.25, 1.00\}. These variants share the full method's paid transcript, inferred tangent vector, and final gate decision. Their differences therefore isolate delivery scaling without changes to acquisition, inference, or gate release.

ICM's mean relative hierarchical nDCG@10 gains are 2.613% on the historical panel and 2.639% on confirmation, with across-seed SDs of 0.060 and 0.012 percentage points, respectively. The table below gives paired full-ICM-minus-control differences in percentage points of relative gain, with sample SDs across seeds:

ControlHistorical panelConfirmation panel
Logistic-only channel+0.076±0.047+0.076 \pm 0.047+0.059±0.014+0.059 \pm 0.014
Indifference-only channel+0.059±0.012+0.059 \pm 0.012+0.176±0.038+0.176 \pm 0.038
Uniform channel weights+0.057±0.024+0.057 \pm 0.024+0.080±0.038+0.080 \pm 0.038
Fixed delivery, α=0.10\alpha=0.10+1.115±0.032+1.115 \pm 0.032+1.242±0.013+1.242 \pm 0.013
Fixed delivery, α=0.25\alpha=0.25+0.362±0.015+0.362 \pm 0.015+0.249±0.032+0.249 \pm 0.032
Fixed delivery, α=1.00\alpha=1.00+1.705±0.131+1.705 \pm 0.131+1.484±0.168+1.484 \pm 0.168

Thanks for your response, I will keep my score.

Thank you for the discussion and for suggesting the single-channel and fixed-scale comparisons. They helped us show more clearly what the response mixture and query-dependent update scale each contribute.

Reviewer_3Y4WOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the problem of refining a frozen query embedding without having access to the gallery or labels. It uses a fixed budget of B noisy ternary comparisons (considers a reference & two anchors and returns one bit saying which anchor is closer). The proposed method Independent Channel Mixture (ICM) has four parts:

  1. Estimates a shared reliability from two repeated "pilot" calls per reference.
  2. Runs a per-query mixture of two response links (Bradley - Terry and an "indifference" link).
  3. Picks comparisons by expected variance reduction of anchor scores.
  4. Performs a tangent-space update scaled by a covariance-derived coefficient (α\alpha), subject to a batch-level evidence gate which returns the frozen queries unless a likelihood-ratio statistic E exceeds 20.

The paper proves three propositions: a mixture-dominance bound, a variance-reduction identity, and a fair-response null for the gate. In a simulated oracle driven by hierarchy tree distance with iid flips, ICM improves hierarchical nDCG@10 over the frozen embeddings by 3.03% on 36 dataset-representation combinations. ICM beats eleven adapted baselines, the closest being Task-Adaptive-BT, by 0.46 points.

--

Updated post rebuttal.

Critical strengths
  1. Results are reported with eight dataset families spanning text, images, audio and satellite data, with hierarchy depths from 2 to 7. Each family is tested with several frozen representations, 16 pretrained checkpoints + simple statistical baselines (TF-IDF/SVD, log-mel statistics, spectral PCA), and ICM's gains are positive in all eight-family means.
  2. ICM improves hierarchical nDCG@10 over the frozen embeddings by 3.03% on 36 dataset-representation combinations. It improved by 16.48% in best case/setting.
  3. The evidence gate returns the original queries unless the answers look like more than random flips. Proposition 3 gives a valid guarantee (false release at most 5% under fair responses), and the experiments match it: only 4 of 1,080 pure-noise batches released. Abstaining when feedback is uninformative is a sensible design choice.
  4. The three propositions each address a different object (transcript prediction, uncertainty reduction, and score stability).
Critical weaknesses
  1. Query refinement never changes the gallery index. The constraint that the learner cannot see the gallery is self-imposed and not justified by a concrete application. Please clearly motivate the constraint for a wider audience.
  2. The simulated oracle is essentially the evaluation metric, and the noise model is the one the method assumes. Oracle answers come from tree distance in the same hierarchy that defines relevance in hierarchical nDCG. Noise is independent flips from one common channel, which is exactly what the reliability estimates, and the gate are built for. Real users/raters may differ in accuracy and the notion of "closer" may vary. Appendix F presents the only non-simulated evidence (LLM study); however, it is small -- 16 references per dataset, two budgets, a single seed, and no comment on agreement with the true hierarchy.
  3. The 16.48% figure reported in the abstract is one condition (etaeta = 0 (perfectly clean answers), B = 256) on a 96-dimensional spectral PCA control, not a pretrained encoder. The overall gain is ~3.01 (Table 4).
  4. There is no reference point for what is achievable. A supervised upper bound that uses anchor labels, or a gallery-access relevance-feedback baseline, would help.
  5. Baselines can be improved:

(i) The comparison against the baselines (Table 7) doesn't show that ICM's ideas are better than the published methods. The paper states that these adapters do not reproduce the original systems and that equal tuning effort is not established (Appendix E).

(ii) Multiple baselines perform far below the frozen embeddings (Pairwise Rocchio -12.95%, AES variants -16%, Qian APC -47%, dense RF -40%), which suggests that the query moved too aggressively under noisy one-bit feedback and safeguard is missing. As per my understanding, ICM has two such safeguards: the evidence gate and the geometry-bounded step size. Is it possible that the gap mostly came from these safeguards instead of reliability calibration, the two-link mixture, or the acquisition rule.

(iii) A fairer test would add ICM's gate and a bounded step to the simple baselines (a single Bradley-Terry channel, Rocchio, TOUR-soft) and tune each baseline's step size on a held-out cohort.

Questions for authors
  1. How does ICM perform when the oracle is mismatched with the evaluation hierarchy: a different hierarchy, systematically biased answers, or feedback from another task? Does the gate protect in those cases?
  2. What are the gains at N = 1, 8 and 32 references per batch and with 64 or 128 anchors? Can calibration and the gate work for a single user?
  3. Do the simple baselines (BT, Rocchio, TOUR-soft) improve when given ICM's gate and bounded step? Were the baseline step sizes tuned?
  4. For the LLM study: what is the oracle's agreement with the tree oracle, which model gave the relevance grades, and what are the results with more references and multiple seeds?
4 comments

Thank you for the detailed and constructive review. Your comments identified the need for stronger controls and a clearer account of the method's practical scope. We address your numbered concerns below.

1. Application motivation.

The revised Introduction describes a maintenance engineer searching a proprietary incident archive through a controlled service. An external client can access approved reference and anchor summaries and the service's frozen encoder, while the organization prohibits exporting the archive or its embedding index. Comparisons among approved examples can emphasize failure mechanisms over equipment type before the first retrieval. ICM refines the query vector for the service to rank against its unchanged index.

This example motivates the restricted interface. Appendix H also explains how to construct an accessible anchor pool using the same encoder and relevant application domains.

2. Oracle mismatch, grading, and repeated LLM evidence.

The tree oracle deliberately provides feedback aligned with a specified evaluation target, allowing controlled study of budget and noise. It cannot by itself establish robustness to other preferences or response processes. We therefore added direct mismatch tests and a repeated semantic-feedback study.

Feedback mismatch.
Appendix H tests heterogeneous noise, systematic reversals, coarsened or alternate hierarchies, topic omissions, and positional bias on four historical representation settings. Foreign-anchor tests use a separate two-family subset, WOS and Aircraft. The gate does not generally protect against coherent task mismatch. An alternate hierarchy gives −0.94%-0.94\% mean gain despite 55% gate release. The fixed-order positional bias gives -1.02% with 62.5% release (Table 24). Reversing answers for one quarter of references produces six regressing condition means out of sixteen despite a positive aggregate mean. Heterogeneous flip rates of 0 and 0.4 give 2.44% gain, 0.85 percentage points below matched IID noise at the same mean rate of 0.2.

These outcomes clarify the paper guarantee. Proposition 3 controls release under independent fair responses, but not under every response process.

LLM study.
Appendix G.3 now reports 18 references per family across WOS, WDC, and BGC. At budget 8, BT leads in hierarchical nDCG gain, 2.779% versus ICM's 2.463%. At budget 32, ICM leads, 4.660% versus 4.133%. Across both budgets, gains remain close, 3.561% versus 3.456% (Table 19).

The comparison oracle and scalar-relevance grader are both qwen-3.8-27b, served through Cerebras. Grading is blind to method, encoder, rank, and retrieval score, with shared grades for the same reference-gallery identity. On the declared 16-reference-per-family grading subset at budget 32, normalized relevance@10 improves by 2.367% for ICM and 1.566% for BT. This is mean grade divided by three.

Table 21 reports strict hierarchy agreement, including paid repeats. 4,608/6,079 = 75.80% for ICM-acquired pairs and 4,128/5,080 = 81.26% for BT-acquired pairs. Hierarchy ties account for 29.64% and 41.20% of calls and are reported separately rather than treated as incorrect answers. ICM's agreement on first presentations alone is 75.17%.

3. Maximum versus typical gains.

We agree that 16.48% should not be read as typical performance on pretrained encoders. Section 6.1 explicitly identifies it as a clean-feedback result on the 96-dimensional TreeSatAI S1+S2 spectral PCA control at budget 256.

There is also a new identity-level confirmation (Appendix F, Table 16). Three disjoint confirmation cohorts per family in four previously studied families, two representations, budgets 8/32/128, informative noises 0/0.2/0.4. ICM gains 2.639%, versus 2.22% for BT, leading in three family means while BT leads on Aircraft.

4. A supervised reference point.

Appendix E.4, Table 14 adds a ridge-based reference receiving the hierarchy paths of all 256 anchors, without query labels, gallery labels, or gallery embeddings. Regularization and interpolation are selected on separate tuning cohorts and locked for confirmation. It improves family-balanced nDCG from 0.48965 to 0.50596, a 3.33% gain, retaining the negative WOS outcome of −0.13%-0.13\%. Such a result provides an achievable reference with richer supervision.

5. Baseline safeguards, tuning, and component attribution.

Appendix E.3, Table 13 evaluates each method using the same bounded-delivery rule and scale grid, {0, 0.25, 0.5, 1, 2}. The tangent displacement is clipped to norm 0.25 before scaling and normalization. One global scale per tunable policy is selected on the separate tuning cohort and locked before confirmation. The following are mean relative hierarchical nDCG@10 gains over Frozen, averaging budgets 8/32/128 and informative noises 0/0.2/0.4:

MethodOriginalBounded deliveryBounded delivery + common gate
Single-BT−19.735%-19.735\%+0.457%+0.457\%+0.410%+0.410\%
Task-Adaptive-BT+2.22%+2.22\%+2.911%+2.911\%+2.936%+2.936\%
Rocchio−12.611%-12.611\%+3.085%+3.085\%+3.032%+3.032\%
TOUR-soft+0.933%+0.933\%+1.636%+1.636\%+1.553%+1.553\%
ICM+2.639%+2.639\%+3.558%+3.558\%+3.058%+3.058\%

These controls confirm that delivery safeguards and scale tuning explain a substantial part of the original adapters' deficit on this panel. The component evidence is modest. Confirmation gains of 0.059 percentage points over logistic-only and 0.249 over the strongest tested fixed scale (Table 10).

The common gate uses only repeated-pilot agreement, unlike ICM's native calibration-plus-directional-evidence gate. It spends two of B calls per reference. Common-gate ICM retains two internal calibration calls, totaling four pilot calls per reference, with its native gate disabled. Thus the baselines receive a shared external gate.

To examine components within ICM, Appendix E.2, Table 10 adds single-channel policies, uniform weights, and fixed scales. On confirmation, full ICM gains 2.639%, versus 2.580% for logistic-only, 2.463% for indifference-only, and 2.559% for uniform weights. Fixed scales 0.10/0.25/1.00 give 1.397%/2.390%/1.155%. Random acquisition and the no-repeat control give 0.740% and 2.217%.

Full ICM has the highest mean among the listed component variants on both panels. The mixture's incremental advantage is modest.

6. Small batches and fewer anchors.

Tables 22 and 23 evaluate 128 references divided into separately calibrated and gated batches. At flip rate 0.2, the gains are:

Batch sizeAnchorsBudget 32Budget 128
1128+0.03%+0.03\%+1.44%+1.44\%
8128+1.67%+1.67\%+3.89%+3.89\%
32128+2.44%+2.44\%+4.11%+4.11\%
128128+2.58%+2.58\%+4.25%+4.25\%
3264+1.84%+1.84\%+2.81%+2.81\%
32256+3.28%+3.28\%+5.42%+5.42\%
Reviewer_3Y4WOpenReview ↗

Official comment post rebuttal

Thanks for the detailed response. I have updated the score accordingly.

Thank you for revisiting the paper after our rebuttal. Your suggestion to bound the updates and tune each method’s step size on a held-out cohort helped us make the baseline comparison fairer. We appreciate the thoughtful discussion.

Reviewer_RABkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Independent Channel Mixture (ICM), a method that refines query embeddings through a fixed budget of ternary comparisons without inspecting the retrieval gallery. Each comparison contains a reference and two anchors and returns one binary preference. Two repeated pilot calls per reference estimate shared response repeatability across the batch. Each reference then maintains a mixture of Bradley-Terry and indifference response models. The method selects further comparisons using expected reduction in anchor-score variance, infers a tangent-space displacement, and scales the delivered update using local anchor covariance. A batch-level evidence gate returns the frozen queries when evidence against independent fair responses is insufficient. In the experiments, the authors show that ICM achieves relative improvements in nDCG@10 up to 3.03% family-balanced average gain.

Critical strengths

I like the focus on query adaptation without gallery access or encoder retraining. The task and comparison budget are clearly defined, including the cost of calibration. The distinction between repeatable feedback and useful feedback is also well explained. The theoretical statements appear sound within their stated scope, and the paper is candid about their limitations.

Critical weaknesses

My main concern is the evaluation. The main benchmark is a post-hoc subset, and the authors acknowledge outcome-informed selection. Repeating the same cohorts with more seeds does not address this. The fresh WOS result is encouraging, but one encoder on one family does not settle the broader generalization question.

The advantage over the closest adapted baseline is also modest. On the matched benchmark, ICM and Task-Adaptive-BT achieve 3.01% and 2.54% relative gains over frozen embeddings, respectively (Table 4). The small, single-seed LLM study is nearly tied: hierarchical nDCG@10 is 0.7756 versus 0.7748 in Table 8. These results show that refinement can help, but leave me unconvinced about the benefit of the full ICM design.

In particular, the reported ablation covers acquisition and the gate, but does not isolate the benefits of the two-channel mixture or geometry-scaled delivery. The theory does not establish that these components improve retrieval, so the missing ablations matter. I would find controlled comparisons against simpler variants more persuasive than additional runs on the same selected benchmark.

Questions for authors

The experiments use anchors from the same dataset as the queries. How should people who wants to use ICM construct a suitable pool in practice, and how sensitive is ICM to incomplete topic coverage or out-of-domain anchors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.