acceptodds
Withdrawn from ICLR 2027

SCOPE: Selective Cross-modal Orchestration of Visual Perception Experts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-encoder vision-language models (VLMs) benefit from complementary visual representations, but activating every encoder increases inference cost and visual context length. We propose SCOPE, a Mixture-of-Encoders framework that keeps one shared encoder and selects one auxiliary encoder for each image–prompt pair. A lightweight router uses cross-attention between the text prompt and shared visual features, and is trained with dual entropy regularization and auxiliary losses to encourage confident instance-level decisions and diverse routing across samples. On the document-centric and OCR-heavy benchmarks studied here, SCOPE improves over the reported static single-auxiliary baselines and naive all-encoder concatenation on average. Relative to activating all auxiliary encoders, the tabulated vision-encoder and decoder FLOPs estimates decrease by 24–49%; text-conditioned routing also requires a separate text-encoder pass. These results support instance-adaptive selection under a one-auxiliary-stream deployment constraint, where the selected encoder determines the auxiliary token count.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, Multi-encoder

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eppjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the inference overhead problem in multi‑encoder VLMs by proposing SCOPE, an instance‑level routing algorithm that dynamically selects among a pool of four vision encoders. The proposed approach outperforms baselines equipped with single or multiple fixed encoders across several visual analysis benchmarks. Analytical estimates also show reduced inference FLOPs. Ablation experiments provide supporting evidence for the advantages of the overall pipeline.

Critical strengths
  1. This work introduces a novel instance‑level routing paradigm which yields several interesting observations: adaptively selecting encoders on per‑instance content leads to better performance than naively concatenating outputs from all available encoders. Another practically insight is the observed trade‑off between task score and encoder-use balance.
  2. The paper establishes fair experimental comparisons, together with reasonable interpretations for the results. The authors also honestly acknowledge the missing ablations and openly discuss their limitations in the appendix.
Critical weaknesses

While the authors honestly discuss the paper’s limitations and missing ablations, these omissions nonetheless reduce the reliability of the overall conclusions. I strongly recommend the authors to supplement the following experiments and analyses:

  1. Although baseline‑1 is shown to underperform the proposed routing method, where encoders with diverse specializations are all enabled during training, the authors have not demonstrated whether the complex routing design outperforms a simple, lightweight text‑only classification head. Consequently, it remains ambiguous whether performance are benefited from jointly fine‑tuning the full set of encoders, or from the cross‑attention‑based routing architecture deployed at inference.
  2. While the method yields theoretical FLOPs reduction, its practical latency and memory benefits under realistic deployment remain unclear. Theoretically, the approach incurs higher computational cost during training, and yields no reduction in peak GPU memory at inference since all encoder weights must still reside in memory. Furthermore, the multi‑encoder loading and the routing module itself add non‑trivial runtime costs that are not accounted for in the current analysis.
Questions for authors
  1. How to quantify the individual contribution of each proposed component such as the cross‑attention router architecture, the STE strategy, the designed loss terms, or the fine‑tuning over all encoders? Consequently, the necessity of deploying the complex routing plugin at inference is not fully established.
  2. Could routing decisions be confounded by the potential differences in output token counts among the encoders? Since the paper does not clarify which encoder is selected for baseline‑1, it is ambiguous whether the quality improvements from dynamic selection are driven primarily by substantial increases in the average number of visual tokens.
Reviewer_AjhgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This study presents SCOPE, a framework for vision-language models that combines a shared vision encoder with an additional encoder selected for each image and prompt. It uses a cross-attention router to select the auxiliary encoder based on visual and text features. It also uses entropy regularization and additional losses to improve the routing process.

Critical strengths
  1. The study addresses the high computational cost and long visual context of multi-encoder vision-language models. Selecting one additional encoder for each input is a reasonable way to balance performance and computation.
  2. SCOPE is tested on eight benchmarks, and the results are better than the selected static single-auxiliary baseline on all eight. The paper also includes ablation studies on routing and loss weights, which provide some evidence that the proposed method is useful.
Critical weaknesses
  1. The efficiency claims are based on estimated FLOPs rather than actual measurements of latency, throughput, and memory usage. The extra costs of the text encoder and router are also not included.
  2. The paper does not compare SCOPE with important baselines, such as random routing and static routing based on task type. Therefore, it is difficult to know the real benefits of adaptive routing.
  3. The auxiliary encoders use different numbers of visual tokens. This means the comparisons are not made under the same token or computational budget.
  4. This study does not report results from multiple random seeds or provide statistical uncertainty. Therefore, it is difficult to know how reliable the reported improvements are.
Questions for authors
  1. How does SCOPE compare with random routing and a static router selected based on task type using the same encoder pool and training budget?
  2. Can the authors provide comparisons with the same number of tokens or FLOPs to check whether the performance improvements remain when the computational budget is similar?
  3. Are the results consistent across multiple random seeds? Please provide results for each seed or suitable uncertainty estimates for the main comparisons.
  4. How much does the routing mechanism contribute to the results compared with the two-stage training process, vision-encoder fine-tuning, and auxiliary losses? What ablation studies support these findings?
  5. The training objective encourages routing diversity but does not directly ensure balanced use of the experts. How sensitive are the results to batch composition and the training-data distribution?
Reviewer_s6CGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCOPE, a Mixture-of-Encoders framework for vision-language models that dynamically selects one auxiliary vision encoder for each image–prompt pair. Specifically, the method uses cross-attention between text embeddings and shared visual features to predict encoder selection scores. The selected encoder's features are then weighted and concatenated with the shared visual representations before being passed to the LLM. The authors additionally introduce dual entropy regularization and auxiliary losses to encourage confident and diverse routing decisions. Experiments on document understanding and OCR-related benchmarks show improvements over the reported static encoder baselines.

Critical strengths

● The problem of efficiently utilizing complementary visual representations is practically relevant, particularly for document understanding tasks with long visual token sequences.

● The proposed framework is straightforward and demonstrates consistent improvements over the reported static single-encoder baseline across multiple benchmarks.

● The paper includes several useful analyses, including comparisons between different router architectures, loss-weight ablations, and encoder selection statistics across task families.

Critical weaknesses

(1) My primary concern is the limited novelty of the proposed method. At its core, SCOPE performs vision encoder selection using a learned routing network. The router applies a standard cross-attention mechanism to text and visual embeddings, predicts encoder selection scores. The additional entropy regularization and auxiliary losses are also closely related to existing MoE routing and load-balancing techniques. Overall, the framework appears to be a relatively straightforward application of established MoE ideas to vision encoder selection, rather than a substantially new architectural or algorithmic contribution. Prior works such as MoAI and MoVA have already explored adaptive utilization of visual experts.

(2) The strong task-level routing preferences in Table 3 suggest that the router may primarily learn task-to-encoder mappings rather than meaningful instance-level selection. Comparisons with random routing, task-level static routing, and oracle routing are needed.

(3) Eq. (2) sums encoder features with different token lengths and dimensions without explaining their alignment. Moreover, the reported FLOPs savings exclude the additional text encoder and router overhead, and no actual latency measurements are provided.

(4) The four regularization terms add complexity without clear advantages over standard MoE load-balancing objectives. The current ablations do not independently evaluate each loss or compare against simpler alternatives.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.