- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that choosing which encoder to use can improve performance for trained two-tower retrieval models. Other than the default pairing configuration OWN (using the query encoder for the query and the item encoder for the item), two other pairings are available at no extra cost: ROUTE (using the item encoder on both sides) and RAW (using pretrained backbone on both sides). The paper proposes Logic Bus that selects or weighs these three scores under different settings. The paper is motivated by coverage: the authors model each encoder as a linear map over a frozen base representation, with the two towers trained on different marginals of the training pairs, and prove that when fitting follows data density, a tower's approximation bias grows as its training coverage thins. The authors empirically experiment on an image-text catalogue and eight text retrieval datasets, and show that different datasets and queries have different optimal configurations. Choosing the configuration per dataset gains 8.4 nDCG@10 over the best fixed configuration.
Critical strengths
- The paper studies the effect of different encoder choosing configurations on datasets of various domains and modalities, delivering clear observation that the default is not reliably the best.
- Besides studying what pairing configurations each dataset prefers, the paper also studies the effect of choosing or mixing different configurations across the dataset through empirical methods like score weighing and query clustering.
- The paper reports variability across seeds and validation/test splits, supporting the evidence robustness.
Critical weaknesses
- The theory assumes a linear map over a frozen backbone and is only validated for shallow towers over a frozen backbone. When both full encoders are fine-tuned end to end, the relationship between coverage and the ROUTE gain disappears (p = 0.53, gain negative in seven of eight runs), so the main DPR results on the text datasets are not well explained by the proposed mechanism.
- While the authors identify the interesting finding that choosing different encoders for each pair is important and has potential to improve performance without re-training, the authors find that choosing a configuration requires evidence (like validation labels or domain tags). No reliable label-free per-query method is found in the authors’ study.
- While training cost is not needed, the proposed methods would introduce inference overheads, which are not analyzed. For example, RAW would require a second corpus index, roughly doubling storage. The best shown non-oracle method (Weighted score combination per domain) would require running queries through three different encoders and indexes, leading to high practical computation costs.
- The paper primarily studies separate-weight two-tower retrieval models, which are now uncommon in text retrieval, and the text evidence comes from a single family (DPR). In section 4.2, the off-the-shelf single-tower retrievers Contriever and multilingual-E5 outperform not only the two-tower RAW (47.36 and 62.68 against 26.02 on average) but also the per-query oracle configuration (43.43), which raises the question of when the proposed method is preferable to simply switching models. In multimodal scenarios, the method also assumes a modality pre-aligned backbone (like CLIP) already exists, and it is unclear how it applies without one.
- While the authors show that three configurations (OWN, ROUTE, RAW) are complementary to each other, and choose to focus on the system’s own operators, there are no experiments showing that the benefit of LogicBus is specific to the system’s own operators, or that choosing to only use the system’s own operators is superior to combining with external models. Furthermore, label-free equal-weight fusion already reaches 34.38 macro and 49.74 micro, which fuses only the system’s own three operators, above choosing per domain on micro though below it on macro, which suggests much of the gain may come from generic score fusion.
Questions for authors
- What do the image-text inputs consist of in your “Image-text retrieval” setting? Which modality is the side-1 and side-2 input? Would the results hold if the metrics are product-level MRR?
- In what ways does the proposed method differ from standard ensembling methods (theoretically or practically)? Does fusing OWN with an unrelated pretrained model give a similar gain?
- Can coverage be measured on the text datasets and shown to predict which configuration wins?