acceptodds
Under review as a conference paper at ICLR 2027

Logic Bus: When Operators Matter, Not Just Vectors

Abstract

Two-tower models are trained to align the vectors their encoders produce, while the encoders themselves—the operators—are left unused as a choice. Beyond the default of passing each side through its own tower, a trained system holds two alternatives, at no training cost, that beat it on part of the input space: the second tower applied to both sides, and the pretrained backbone applied to both. We call a system that chooses among these pairings a Logic Bus. Contrastive training fits the two towers on different marginals of the training pairs, so they differ in training coverage. We show that, when fitting follows data density, each tower's bias grows as its coverage thins. It follows that no fixed pairing is least biased everywhere, provided each tower covers some inputs more densely than the other, and the backbone is better than the towers on some inputs and worse on others; when either condition fails, the choice shrinks or disappears. In an image–text catalogue, reducing only training coverage moves the trained towers from 20 category-level MRR points above the backbone on the tail to 52 below it. In text retrieval the best pairing changes from dataset to dataset: the three fixed pairings score within about one point of each other in aggregate, while choosing the pairing per dataset gains 8.4 nDCG@10 over the best of them.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
two-tower models, dual encoders, contrastive learning, dense retrieval, cross-modal retrieval, long-tail retrieval, encoder selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TK9wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that choosing which encoder to use can improve performance for trained two-tower retrieval models. Other than the default pairing configuration OWN (using the query encoder for the query and the item encoder for the item), two other pairings are available at no extra cost: ROUTE (using the item encoder on both sides) and RAW (using pretrained backbone on both sides). The paper proposes Logic Bus that selects or weighs these three scores under different settings. The paper is motivated by coverage: the authors model each encoder as a linear map over a frozen base representation, with the two towers trained on different marginals of the training pairs, and prove that when fitting follows data density, a tower's approximation bias grows as its training coverage thins. The authors empirically experiment on an image-text catalogue and eight text retrieval datasets, and show that different datasets and queries have different optimal configurations. Choosing the configuration per dataset gains 8.4 nDCG@10 over the best fixed configuration.

Critical strengths
  1. The paper studies the effect of different encoder choosing configurations on datasets of various domains and modalities, delivering clear observation that the default is not reliably the best.
  2. Besides studying what pairing configurations each dataset prefers, the paper also studies the effect of choosing or mixing different configurations across the dataset through empirical methods like score weighing and query clustering.
  3. The paper reports variability across seeds and validation/test splits, supporting the evidence robustness.
Critical weaknesses
  1. The theory assumes a linear map over a frozen backbone and is only validated for shallow towers over a frozen backbone. When both full encoders are fine-tuned end to end, the relationship between coverage and the ROUTE gain disappears (p = 0.53, gain negative in seven of eight runs), so the main DPR results on the text datasets are not well explained by the proposed mechanism.
  2. While the authors identify the interesting finding that choosing different encoders for each pair is important and has potential to improve performance without re-training, the authors find that choosing a configuration requires evidence (like validation labels or domain tags). No reliable label-free per-query method is found in the authors’ study.
  3. While training cost is not needed, the proposed methods would introduce inference overheads, which are not analyzed. For example, RAW would require a second corpus index, roughly doubling storage. The best shown non-oracle method (Weighted score combination per domain) would require running queries through three different encoders and indexes, leading to high practical computation costs.
  4. The paper primarily studies separate-weight two-tower retrieval models, which are now uncommon in text retrieval, and the text evidence comes from a single family (DPR). In section 4.2, the off-the-shelf single-tower retrievers Contriever and multilingual-E5 outperform not only the two-tower RAW (47.36 and 62.68 against 26.02 on average) but also the per-query oracle configuration (43.43), which raises the question of when the proposed method is preferable to simply switching models. In multimodal scenarios, the method also assumes a modality pre-aligned backbone (like CLIP) already exists, and it is unclear how it applies without one.
  5. While the authors show that three configurations (OWN, ROUTE, RAW) are complementary to each other, and choose to focus on the system’s own operators, there are no experiments showing that the benefit of LogicBus is specific to the system’s own operators, or that choosing to only use the system’s own operators is superior to combining with external models. Furthermore, label-free equal-weight fusion already reaches 34.38 macro and 49.74 micro, which fuses only the system’s own three operators, above choosing per domain on micro though below it on macro, which suggests much of the gain may come from generic score fusion.
Questions for authors
  1. What do the image-text inputs consist of in your “Image-text retrieval” setting? Which modality is the side-1 and side-2 input? Would the results hold if the metrics are product-level MRR?
  2. In what ways does the proposed method differ from standard ensembling methods (theoretically or practically)? Does fusing OWN with an unrelated pretrained model give a similar gain?
  3. Can coverage be measured on the text datasets and shown to predict which configuration wins?
Reviewer_j91GOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper claims that a trained two-tower model can score a pair of inputs in three ways without retraining: its default way (OWN), using the second tower on both sides (ROUTE), or the base backbone on both sides (RAW). It shows on 8 BEIR datasets with DPR encoder that the default OWN is not always the best, gives a linear-model theory that a tower's bias grows where its training coverage is thin. It also shows that choosing or weighting among the three scoring methods per dataset with validation labels gains 8.4 nDCG@10 macro over the best fixed configuration.

Critical strengths
  • The observation that the default pairing loses by large margins on some datasets is and interesting finding
  • The paper is clear about negative results and limitations
Critical weaknesses
  • The significance of this paper is not clear to me. Sec. 4.4 essentially shows that selecting or ensemble three scorings with validation labels is better than each scorer alone, which is kind of expected. The 8.4 gain needs per-dataset labels and with the same labels one could adapt the retriever or pick a different retriever.
  • The setup assumes the two towers accept each other's inputs (same modality/tokenizer) and that the backbone is available, which limits applicability.
Questions for authors
  • Why is the mirror (φ1, φ1) excluded? It is not explained clearly in either the setting or the theory. And there exists even more potential configs and why the current 2 extra ones are selected only?
Reviewer_KscpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a trained two-tower model which can score a pair in three ways at no extra training cost within its own towers, the second tower on both sides, or the pretrained backbone on both sides. Experiments in text and image to text retrieval show the best of the three changes from dataset to dataset, and picking it per dataset, which the paper calls a “Logic Bus,” gains 8.4 nDCG@10 over the best fixed choice.

Critical strengths
  1. The core idea is simple and clear, a trained two-tower system already contains alternative encoder pairings that can outperform the default at no training cost.
  2. The approach is cheap and easy to try in systems, and the observation that encoding rare queries with the item tower can fix collapsed results is a useful insight in practice.
  3. The related work section is well organized and positions the paper well against shared vs. separate encoder design.
Critical weaknesses
  1. My first concern is with the “operator” framing and the shared space. I am not sure how ROUTE is defined when the two sides don’t share an input space. For example in the image–text experiment it seems to work only because frozen CLIP already puts images and text in one embedding space. Would the approach still apply to a two-tower model with separate, unaligned backbones? This would cast doubt on the operator framing and make the scope narrower.
  2. The headline +8.4 depends on labelled validation queries from every target dataset, which goes against the zero-shot setting that BEIR. A label-free alternative should therefore be the main result.
  3. In Assumption 1, the theory assumes each tower learns on its own, moving toward one fixed correct mapping. But every model in the paper is trained with InfoNCE, where the two towers learn together by matching true pairs against wrong ones, so this assumption doesn’t seem to hold for any experiment. The authors also note it does not hold for end-to-end fine-tuning, which is how DPR, the main text experiment, was trained. This needs further proof or justification.
  4. I wonder how much the Logic Bus adds beyond ordinary model selection or ensembling. Different winners per dataset are common for any set of retrievers on a benchmark, and the paper’s own appendix shows that a label-free equal-weight combination of the three scores nearly matches choosing per dataset (34.38 vs 35.36 macro) and beats it on micro (49.74 vs 47.75).
  5. The introduction felt quite brief for the idea it introduces. The coverage asymmetry seems to be the paper’s central insight, but it only becomes clear in Section 3. I also expected the intro to explain more about why this matters in practice and its applications.
Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.