- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Efficient Operator Search (EOS), a differentiable framework for visual-token reduction in multimodal language models. EOS parameterizes three aspects of reduction: where reduction is applied, how many tokens are retained, and how information from discarded tokens is processed. The proposed operator represents pruning, hard merging, uniform redistribution/pooling, and adaptive reweighting as different parameter regimes, while allowing intermediate hybrid configurations. The backbone is frozen during search, and the search variables are optimized using a language-modeling loss, a hidden-state alignment loss, and one-sided token-budget and active-layer constraints.
Critical strengths
- Treating pruning, merging, redistribution, and reweighting as related information-transfer behaviors provides a useful perspective on token-reduction methods, and organizing them into a common parameterized space is potentially useful.
Critical weaknesses
-
The relationship to differentiable-NAS is not discussed, weakening the novelty claim. EOS optimizes continuous structural variables, relaxes discrete choices, includes resource constraints, warm-starts from an existing architecture, and discretizes the result for deployment. This closely resembles differentiable-NAS. However, the paper contains no related-work section, cites no foundational differentiable-NAS methods, and does not identify what is methodologically new relative to applying differentiable NAS to token-reduction operators. The paper presents the transition from “operator design” to “operator search” as a new paradigm and one of its primary contributions. However, without comparison to these related works, it is difficult to determine whether EOS is a genuinely new search method or primarily a new application space.
-
The paper claims "efficient search policy", however, the paper also does not quantify: (1) search time, training steps, or GPU cost; (2) the cost of the full-model reference pass required by the alignment loss; (3) convergence or sensitivity across initializations; (4) whether differentiable search outperforms random search, grid search, etc. These omissions are especially problematic.
-
In Equation (12) defines, the multiplicative ratio is one in the forward pass and provides a surrogate gradient with respect to . However, the paper never shows where enters the reduced feature computation or task loss. It is unclear how the task or alignment loss supplies gradients to and through the hard top- operation. If is only used to estimate token count, the task loss cannot guide the layer or retention-budget search.
-
The experimental scope is too narrow. All experiments use a single model, LLaVA-1.5-7B. There is no evaluation on another architecture, model scale, vision encoder, or modern MLLM family. Main comparisons exclude FastV, VisionZip, and PruMerge+, even though the paper discusses them in Table 1. These limitations make it difficult to distinguish a robust search method from tuning that happens to work for one backbone and evaluation suite.
-
Several claims and reporting results are misleading or insufficiently defined. The most visible example is Table 3. The searched EOS model’s POPE score of 85.78 is bolded even though the original SparseVLM+ score is 85.79. The row is labeled “Beats V2”, although EOS loses on POPE, GQA relative to the reported SparseVLM+ numbers and only narrowly improves some other metrics. Figure 7 states that the advantage increases as the retained budget decreases, but the reported macro difference is negative at 32 tokens and positive again at 16 tokens. The trend is not monotonic. No units are shown in the table headers for latency and memory.
Questions for authors
Please see the weakness part.
- Please compare your method with NAS literature
- How does Equation (12) enter the forward computation? How are (g_l) and (c_l) separately identifiable?
- What is the total search cost? What data are used for search and model selection?
- Please include stronger and more recent baselines into the main results
- Please evaluate at least one additional backbone family and another model scale.