- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents OmniRouter, a modular multimodal retrieval encoder that constructs a shared embedding space from independently pretrained specialist encoders without jointly training their backbones. The method combines two representation geometries: a Qwen3 text embedding space and a SigLIP-text residual space obtained through a closed-form ridge mapping. These representations are interleaved coordinate-wise to preserve contributions from both geometries under embedding truncation.
Critical strengths
-
A meaningful modular retrieval formulation with well-supported design choices. The paper addresses an important question: whether competitive multimodal retrieval requires joint training or can be achieved by composing independently pretrained specialist encoders. The proposed dual-geometry construction is conceptually appealing and provides a practical alternative to relying on a single shared embedding space. The authors also provide relevant ablation experiments. Table 2 demonstrates that combining both representation blocks improves all four evaluated cross-modal retrieval directions compared with either block alone, while the residual construction provides additional benefits on several directions. The coordinate-interleaving strategy is simple and computationally inexpensive, with measurable advantages over concatenation at intermediate embedding dimensions. Importantly, the authors also acknowledge where these design choices provide limited or negative benefits.
-
Comprehensive evaluation and convincing evidence for modular isolation. The evaluation covers multiple retrieval directions and compares numerous existing multimodal embedding systems under a unified protocol. The use of paired bootstrap confidence intervals, held-out hyperparameter selection, and explicit reporting of unsupported tasks improves the credibility of the empirical analysis. The modality-isolation experiment is particularly valuable. Adding an audio pathway leaves existing text and image embeddings unchanged while enabling additional retrieval capabilities with a relatively small number of fitted parameters. This provides concrete evidence for the modularity of the implemented architecture.
Critical weaknesses
-
The distinctive contribution of the residual dual-geometry construction requires further clarification. Although the experiments demonstrate the benefits of combining Qwen3 and SigLIP representations, the resulting similarity function can be expressed as a weighted combination of similarities from two representation blocks. This raises the question of how much of the observed improvement originates from the proposed residual construction rather than from combining two strong pretrained encoders. The authors provide a useful no-residual ablation, but the additional gains from residual subtraction are relatively modest and not uniform across retrieval directions. Further comparison with matched alternative fusion strategies, together with an analysis of the residual mapping's sensitivity to its training corpus, would help establish the distinctive methodological contribution.
-
Efficiency advantages are not fully established under matched deployment conditions. The low number of fitted parameters is an attractive feature, but the deployed system still requires approximately 1.76B resident parameters and produces 2048-dimensional embeddings, compared with the 1024-dimensional embeddings of the similarly sized jina-v5-omni model. Although a 1024-dimensional OmniRouter variant is evaluated, the primary comparisons emphasize the full-width configuration, and statistical uncertainty is not reported for all width-matched comparisons. Additionally, end-to-end encoding latency, throughput, and retrieval-index costs are not directly compared. These measurements would help distinguish the demonstrated training and modular-update efficiency from the overall computational and storage costs of deployment.
-
The scope of multimodal retrieval and speech routing remains limited. The experiments primarily evaluate text-centered retrieval directions. Direct retrieval between non-text modalities, such as image-to-audio and audio-to-image, is not explicitly examined. Consequently, the extent to which the constructed shared space supports general cross-modal comparability remains unclear. Furthermore, speech retrieval relies on a transcription-based pathway whose routing decision is provided externally. This is reasonable for text-semantic speech retrieval, but it does not preserve acoustic characteristics such as speaker identity or prosody, and the system does not provide an automatic mechanism for distinguishing speech from ambient audio. The speech results also contain an apparent inconsistency between the Common Voice entries in Tables 8(a), 8(b), and 8(c), which should be clarified to ensure that the reported comparisons are interpreted correctly.
Questions for authors
Q1. Residual construction and methodological novelty. To what extent does the proposed residual dual-geometry representation outperform alternative weighted fusion approaches using the same pretrained encoders, training data, and embedding budget? How sensitive is the ridge mapping to the domain used for estimating it, particularly when moving from COCO captions to scientific or argumentative text?
Q2. Deployment efficiency and matched comparisons. Could the authors provide paired confidence intervals for the 1024-dimensional OmniRouter variant against jina-v5-omni and report end-to-end encoding latency, throughput, and index storage under comparable deployment settings? This would clarify the trade-off between parameter-efficient adaptation and overall serving costs.
Q3. Speech routing and evaluation consistency. Can the authors reconcile the Common Voice results reported in Tables 8(a), 8(b), and 8(c), particularly the differing T→A scores of 0.2363 and 0.9545? Additionally, how does retrieval performance change when using oracle transcripts or when speech/ambient routing decisions are imperfect?
Q4. Cross-modal generality. Does the constructed embedding space support direct retrieval between non-text modalities, particularly image-to-audio and audio-to-image? Even a limited evaluation on an existing paired audio-visual dataset would help determine whether the benefits of the dual-geometry construction extend beyond text-centered retrieval.
3 comments
Q3. Common Voice, reference transcripts and routing errors. The 0.2363 entry in the submitted Table 8(b) came from Abkhaz audio transcribed under an English tag. In the revision, Table 8(a) marks the unsupported Abkhaz configuration, Table 8(b) gives the six scorable tasks excluding Common Voice, and Table 8(c) compares both systems on English Common Voice. On English, OmniRouter scores 0.9545 in each direction and jina scores 0.9739/0.9632. Replacing Whisper output with the reference transcripts gives 1.000 on English Common Voice and JamAlt, and 0.880 on GigaSpeech. With Whisper-small, the respective scores are 0.955, 0.753/0.752 and 0.798.
The routing experiment tests a different source of error. On SpokenSQuAD, the supplied label, Whisper's no-speech threshold and a learned router each give 0.5983. On AudioCaps text-to-audio, the same three conditions give 0.6312, 0.0273 and 0.0312, and flipping 10% of the routes at random gives 0.0815. A clip may contain speech while being relevant for an environmental sound, so the route is a property of the collection, and speech and ambient clips should be indexed separately. Section 3.4 and the Limitations now state this. The transcript route represents spoken content and does not preserve speaker identity or prosody.
Q4. Retrieval between non-text modalities. On 1,000 paired Flickr-SoundNet frames and audio clips, OmniRouter achieves 0.140 NDCG@10 for image-to-audio and 0.138 for audio-to-image. Random ranking gives 0.0045 in both directions. Under the same protocol, jina scores 0.143/0.140, VAST 0.162/0.171 and ImageBind 0.366/0.361. The two OmniRouter projectors were fitted through text. The result quantifies how much frame-audio correspondence that training preserves, and how it compares with a model trained on paired visual and audio data.
please examine rebuttal content
Hello Reviewer TjoT,
As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal, comment, and decide if it changes your thoughts regarding your current evaluation.
Thanks.
Thank you for distinguishing the benefit of combining two geometries from the added effect of residualisation, and for asking how the comparison changes at equal deployment width. We report the corresponding ablations, paired intervals and serving measurements in Appendix L. Changes in the paper are marked in blue.
Q1. Fusion and the fitting domain. With the same encoders and the deployed mixing weights, raw concatenation implements weighted late fusion of the two normalised blocks, with the side-specific coefficients in the score equation. The image-to-text results on Flickr30k separate the two effects: the stronger single block scores 0.7136, raw fusion 0.8279, and the ridge residual 0.8623.
The paired 95% intervals for the residual's gains over raw concatenation are [0.008, 0.013] on text-to-image, [0.029, 0.040] on image-to-text and [0.010, 0.018] on text-to-audio. The other two intervals include zero. A CCA diagnostic at 512 dimensions is reported in Appendix L. As a sensitivity check, we also gave each method its own text weight, selected on the report set. The residual's margin narrows to +0.004 and +0.014 on the two image directions, and audio-to-text favours concatenation by 0.009. The largest gain therefore comes from retaining both blocks, and the residual adds to it.
The additional gain from ridge subtraction depends on the fitting corpus. With a map fitted on COCO captions, image-to-text reaches 0.8616 in the domain study. Maps fitted on SciFact, NFCorpus, MS MARCO and Touché-2020 give 0.8232 to 0.8279, the level of raw concatenation, while text-to-text is unchanged. On caption text, residualisation reduces linear CKA between the blocks from 0.594 to 0.228. On ArguAna, CKA changes from 0.573 to 0.542. The paper now states that the map should be fitted on caption-like text.
Q2. Matched width and serving cost. We compared both 1024-dimensional variants with 1024-dimensional jina. For the PCA variant, the paired 95% intervals favour OmniRouter on text-to-image [+0.004, +0.015], text-to-audio [+0.141, +0.163] and audio-to-text [+0.105, +0.142]. The other two intervals include zero. For the unmodified interleaved prefix, both audio intervals favour OmniRouter, and the image intervals favour jina: [−0.026, −0.013] for text-to-image and [−0.072, −0.049] for image-to-text. The advantage of interleaving over concatenation at truncated widths and the comparison with jina at the same width are distinct results, and the paper reports both.
On one RTX 6000 Ada, running one model at a time with the same inputs, we measured the following. Throughput is in items/s at batch sizes 32 (text), 16 (image) and 8 (audio). Parentheses give median single-item latency in milliseconds.
OmniRouter's speech route processes 16 clips/s with a median single-item latency of 136 ms. For one million vectors, the full-width representation requires 8.19 GB in fp32 or 4.10 GB in fp16 for the vectors alone. At 1024 dimensions, OmniRouter and jina require the same vector storage at a given precision.