Rethinking Model Efficiency: Multi-Agent Inference with Large Models

Most vision-language models (VLMs) apply a large language model (LLM) as the decoder, where the response tokens are generated sequentially through autoregression. Therefore, the number of output tokens can be the bottleneck of the end-to-end latency. In this work, we conduct a comprehensive analysis of the latency across different components of VLMs. The empirical study on simulated data and real-world benchmarks confirms that a large model can achieve better or comparable performance as a small model with significantly fewer output tokens. To leverage the efficiency of large models, we propose a multi-agent inference framework that keeps large models with short responses but transfers the key reasoning tokens from the small model when necessary. Our theoretical analysis and the comparison on benchmarks demonstrate that by reusing the reasoning tokens from small models, it can help approach the performance of a large model with its own reasoning, which confirms the effectiveness of our proposal.