- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on the inference overhead problem in multi‑encoder VLMs by proposing SCOPE, an instance‑level routing algorithm that dynamically selects among a pool of four vision encoders. The proposed approach outperforms baselines equipped with single or multiple fixed encoders across several visual analysis benchmarks. Analytical estimates also show reduced inference FLOPs. Ablation experiments provide supporting evidence for the advantages of the overall pipeline.
Critical strengths
- This work introduces a novel instance‑level routing paradigm which yields several interesting observations: adaptively selecting encoders on per‑instance content leads to better performance than naively concatenating outputs from all available encoders. Another practically insight is the observed trade‑off between task score and encoder-use balance.
- The paper establishes fair experimental comparisons, together with reasonable interpretations for the results. The authors also honestly acknowledge the missing ablations and openly discuss their limitations in the appendix.
Critical weaknesses
While the authors honestly discuss the paper’s limitations and missing ablations, these omissions nonetheless reduce the reliability of the overall conclusions. I strongly recommend the authors to supplement the following experiments and analyses:
- Although baseline‑1 is shown to underperform the proposed routing method, where encoders with diverse specializations are all enabled during training, the authors have not demonstrated whether the complex routing design outperforms a simple, lightweight text‑only classification head. Consequently, it remains ambiguous whether performance are benefited from jointly fine‑tuning the full set of encoders, or from the cross‑attention‑based routing architecture deployed at inference.
- While the method yields theoretical FLOPs reduction, its practical latency and memory benefits under realistic deployment remain unclear. Theoretically, the approach incurs higher computational cost during training, and yields no reduction in peak GPU memory at inference since all encoder weights must still reside in memory. Furthermore, the multi‑encoder loading and the routing module itself add non‑trivial runtime costs that are not accounted for in the current analysis.
Questions for authors
- How to quantify the individual contribution of each proposed component such as the cross‑attention router architecture, the STE strategy, the designed loss terms, or the fine‑tuning over all encoders? Consequently, the necessity of deploying the complex routing plugin at inference is not fully established.
- Could routing decisions be confounded by the potential differences in output token counts among the encoders? Since the paper does not clarify which encoder is selected for baseline‑1, it is ambiguous whether the quality improvements from dynamic selection are driven primarily by substantial increases in the average number of visual tokens.