acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Model Efficiency: Multi-Agent Inference with Large Models

Abstract

Most vision-language models (VLMs) apply a large language model (LLM) as the decoder, where the response tokens are generated sequentially through autoregression. Therefore, the number of output tokens can be the bottleneck of the end-to-end latency. In this work, we conduct a comprehensive analysis of the latency across different components of VLMs. The empirical study on simulated data and real-world benchmarks confirms that a large model can achieve better or comparable performance as a small model with significantly fewer output tokens. To leverage the efficiency of large models, we propose a multi-agent inference framework that keeps large models with short responses but transfers the key reasoning tokens from the small model when necessary. Our theoretical analysis and the comparison on benchmarks demonstrate that by reusing the reasoning tokens from small models, it can help approach the performance of a large model with its own reasoning, which confirms the effectiveness of our proposal.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_115VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how output length affects VLM inference efficiency and proposes a cascade combining short answers from large and small models. Agreement permits early exit; disagreements trigger small-model reasoning and, when necessary, transfer of that reasoning to the large model for a short answer. Experiments cover several model families and six benchmarks, including two measured end-to-end deployments. A single-attention-layer analysis and a separate sparse-transfer study provide additional motivation.

Critical strengths
  1. The component profiling and response-policy comparisons provide useful evidence that model size alone is an inadequate efficiency proxy. They identify settings where a larger model’s short response outperforms a smaller model’s substantially longer reasoning trace.
  2. The appendix includes genuine end-to-end measurements incorporating the small-model stages. On ChartQA, the matched H100/vLLM experiment reduces total completion time from 1,122 to 468 seconds, a 2.40× speedup (Table 21). A Gemma experiment provides an additional 1.65× result, accompanied by a small accuracy decrease (Table 22).
Critical weaknesses
  1. The measured quality–cost comparison omits an important single-model alternative. On ChartQA, the 8B Explain configuration achieves 87.68% with 249.8 output tokens, versus 86.20% and 294.3 tokens for Reasoning (Tables 5–6), but the measured timing comparison includes only Reasoning and MAI. These numbers do not establish that Explain is faster than MAI; they show that the relevant comparison is incomplete. Furthermore, seconds per sample at batch size 32 measure amortized dataset completion time rather than individual request latency.
  2. The theoretical analysis provides limited explanation of end-to-end reasoning transfer. Its bounds concern attention-weighted averages with fixed representations, whereas actual multi-layer inference recomputes representations, positions, and queries. The required small Wasserstein discrepancy is not established experimentally. The proof can be correct under its assumptions while offering limited support for the broader mechanism claim; the sparse-transfer experiment also leaves its incremental deployment cost unresolved.
Questions for authors
  1. Can Simple, Explain, Reasoning, Full RT, and MAI be compared on the same run’s accuracy–time frontier? Please distinguish batch throughput from individual request latency and identify the targeted deployment regime.
  2. Which model, layer, and query supply the sparse-selection attention, and what additional computation does selection require? What evidence connects the fixed-representation bound to the actual multi-layer transfer mechanism?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 115V

We would like to thank the reviewer for the valuable time and insightful comments, and we will add the following measurements to the revision.


W1: The measured comparison omits 8B Explain, and seconds per sample at batch size 32 reflect throughput rather than request latency.

Thanks for your suggestion and we added the following experiment for Explain. On ChartQA, Explain prompt actually uses the similar number of reasoning tokens as Reasoning, i.e., 249.8 tokens vs. 294.3 tokens, which leads to the similar performance and latency. Moreover, our method is not limited to Reasoning prompt and we can also use reasoning tokens from Explain prompt for transfer. The difference is how reasoning tokens is obtained. The empirical results confirm that our method is independent from specific prompt and reasoning tokens from a different Explain prompt can be transferred well, e.g., MAI (8B w/ 2B-Explain) and MAI (8B w/ 4B-Explain) are better than 8B-S. We will include the discussion in revision. For latency estimation, since the inference type is mixed, e.g., S/E/R, the avg over individual requests is thus utilized.


W2: The single-layer, fixed-representation analysis does not establish the multi-layer transfer mechanism, and the deployment cost of sparse selection is unclear.

Thanks for pointing this out. Please kindly note that our proposed method is applicable to a single layer (also any single layer) and thus we strictly follow our theoretical analysis to select key tokens from a single layer in the experiments as discussed in Section 5.3. To the best of our knowledge, there is no multi-layer approach, which will be an interesting future work and we will clarify this.

Regarding deployment, the selection is a top-K rank given an attention map, which is less than 1 ms and is negligible.


Q1: Can Simple, Explain, Reasoning, Full RT and MAI be compared on one accuracy–time frontier, separating batch throughput from request latency?

Thanks for the suggestion. We include the full comparison in the Table as follows. The observation on Explain is similar to Reasoning, where 8B-E + S2 is better than 8B-S but slower, and MAI (8B w/ 2B/4B-Reasoning) can further improve the performance with mild overhead: MAI (8B w/ 4B-Reasoning) slightly exceeds 8B-E + S2 (87.84 vs.\ 87.68) at about 3× lower latency. We will add this discussion in the revision.

Table R1. Accuracy–time frontier on ChartQA (Qwen3-VL, one H100). Accuracy as reported in the paper (8B Simple, Explain + S2 and Reasoning + S2: Table 6; Reasoning: Table 21; Full RT and MAI (8B w/ 2B-Reasoning): Table 9). Latency: mean latency of a single request (batch size 1, first 200 questions).

MethodAccuracy (%)Latency (s)
8B Simple84.720.10
8B Explain78.442.93
8B Explain + S287.682.99
8B Reasoning77.683.55
8B Reasoning + S286.203.62
Full RT (8B w/ 2B-Reasoning)85.301.60
MAI (8B w/ 2B-Reasoning)86.800.98
MAI (8B w/ 2B-Explain)85.720.63
MAI (8B w/ 4B-Explain)87.121.19
MAI (8B w/ 4B-Reasoning)87.841.03

Q2: Which model, layer and query supply the sparse-selection attention, and what does the selection cost?

Thank you for pointing this out. Please kindly note our approach is applicable to any single layer. We have the ablation on which layer is good for transfer as in Section 5.3 and more detailed discussion can be found in Appendix H. The computational cost for selection is to rank the attention map from a single layer, which is negligible in our experiments. We will make it clear in revision.

Dear Authors,

Thank you for the detailed response and the additional experiments. I appreciate the inclusion of the Explain baselines, batch-size-one latency measurements, and further clarification of the sparse-selection mechanism.

The new results address part of my original concerns and provide stronger evidence for the practical efficiency of MAI. However, I remain concerned about the reliability of the accuracy–latency comparisons and the connection between the theoretical analysis and the actual reasoning-transfer mechanism.

1. The revised accuracy–latency comparison remains insufficiently controlled.

Table R1 is a useful addition, especially the comparison between MAI (8B w/ 4B-Reasoning) and 8B Explain + S2.

However, the table combines accuracy results reported elsewhere in the paper with latency measurements obtained on the first 200 ChartQA questions at batch size 1. A matched evaluation reporting both accuracy and end-to-end latency on the same examples would provide stronger evidence for the claimed quality–efficiency tradeoff.

Furthermore, the best MAI configuration achieves 87.84% accuracy, compared with 87.68% for Explain + S2, a difference of only 0.16 percentage points. While MAI demonstrates a substantial latency reduction (1.03 s versus 2.99 s), the accuracy difference is sufficiently small that paired uncertainty estimates would be important for interpreting the claimed performance advantage.

2. The practical efficiency tradeoff requires more careful characterization.

The added results show that MAI can provide a favorable accuracy–latency tradeoff relative to longer-response baselines.

At the same time, the 8B Simple configuration achieves 84.72% accuracy at 0.10 s, whereas MAI (8B w/ 4B-Reasoning) achieves 87.84% at 1.03 s.

This represents an approximately tenfold increase in latency for a 3.12-percentage-point accuracy improvement. Both configurations may be useful under different deployment requirements, but the practical efficiency advantage depends strongly on the targeted accuracy and latency budget.

A clearer characterization of the Pareto frontier, including uncertainty in accuracy and end-to-end inference cost, would make the main efficiency claims more convincing.

3. The theoretical justification for reasoning transfer remains incomplete.

I appreciate the clarification that sparse token selection uses attention from a single layer and that the reported Top-K ranking overhead is below 1 ms.

My original concern, however, concerns the relationship between the theoretical assumptions and actual reasoning transfer.

The analysis considers attention-weighted representations under a simplified setting, whereas transferred reasoning tokens subsequently affect multi-layer autoregressive inference. The response clarifies the selection procedure but provides limited additional evidence that the assumed representation similarity, including the Wasserstein discrepancy condition, explains the observed downstream improvements.

Moreover, the reported Top-K timing characterizes the ranking operation. A fuller accounting of attention extraction and any associated implementation overhead would help establish the practical cost of the complete sparse-selection procedure.

Overall, I recognize the value of the empirical observations and acknowledge that the additional latency measurements strengthen the paper. The reported speedups relative to reasoning-intensive baselines are promising.

Nevertheless, the central claims about accuracy–efficiency advantages and theoretically motivated reasoning transfer still require stronger matched empirical validation and clearer evidence connecting the theoretical assumptions to the implemented mechanism.

For these reasons, I maintain my original rating of 2 (Weak Rejection).

Dear Reviewer 115V,

Due to the limited time left, we just would like to clarify that we reuse the attention map for selection without any additional cost. The current empirical implementation for reasoning transfer follows the theoretical analysis without any differences.

Reviewer_PuwXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the effect of output token length on VLM inference speed. The authors find that larger models can sometimes give better answers with fewer tokens than smaller models.

Based on this observation, they propose a multi-agent framework. A small and a large model first give short answers. If they agree, the system stops. Otherwise, the small model generates detailed reasoning, and its new answer is compared with the large model's initial answer. If they still disagree, the small model's reasoning is passed to the large model, which then gives a new short answer.

Experiments show competitive accuracy and a 2.40× speedup over 8B-R on ChartQA.

Critical strengths
  1. The proposed method is simple and requires no additional training. It combines mutual verification and reasoning transfer to reduce the need for long reasoning by large models.

  2. The authors test the method on different VLM families and provide real end-to-end latency results. The 2.40× speedup on ChartQA with similar accuracy shows its practical value.

Critical weaknesses

The latency comparison is incomplete. Table 21 compares MAI with 8B-R, but does not report the end-to-end latency of 8B-S. Without this result, it is hard to know how much extra time MAI needs to achieve its accuracy improvement over 8B-S.

Questions for authors
  1. Could the authors report the end-to-end latency of 8B-S under the same settings as Table 21? Since MAI starts with 8B-S and adds extra model calls, this would help show how much additional time MAI needs to improve accuracy over 8B-S.

  2. How often do the small and large models agree on an incorrect answer? Since MAI stops when they agree, I think this is important for understanding the reliability of mutual verification.

1 comment
AuthorsOpenReview ↗

Response to Reviewer PuwX

We would like to thank the reviewer for the valuable time and insightful comments, and we will add the following results and discussions in the revision.


W1 & Q1: Table 21 does not report the end-to-end latency of 8B-S, so the extra time MAI needs over 8B-S is unclear.

Thanks for this helpful suggestion. We add the end-to-end latency of 8B-S in Table R1, measured as the latency of a single request (batch size 1, first 200 questions, one H100).

As we observed in the submission, large model with short output sequence can be efficient compared to small models. However, our proposed MAI can further improve its performance to slightly exceed 8B-R. Note that for MAI, the running time includes cases with small model's reasoning while many cases can stop after mutual verification without additional overhead. Therefore the averaged latency is still less than 1s per example. We will add this comparison and the discussion to the revision.

Table R1. 8B-S, 8B-R and MAI on ChartQA. Accuracy as reported in the paper (8B-S and 8B-R + S2: Table 6; MAI: Table 9; 8B-R without S2: Table 21); latency of a single request measured with vLLM on one H100 (batch size 1, first 200 questions). 8B-R + S2: with the answer-extraction stage of the paper.

MethodAccuracy (%)Latency (s)
8B-S84.720.10
8B-R77.683.55
8B-R + S286.203.62
MAI86.800.98

Q2: How often do the small and large models agree on an incorrect answer?

Thanks for this insightful question, which is indeed important for understanding the reliability of mutual verification. Table R2 reports how often the 2B and 8B short answers agree, and how often they agree on an incorrect answer, on ChartQA, POPE and MMBench. Agreements are reliable on all three benchmarks: fewer than 10% of the agreements are incorrect, i.e., 4.6–8.6% of all questions. We will add this analysis to the revision.

Table R2. Agreement of the 2B and 8B short answers in Stage 1, and agreement on an incorrect answer (Qwen3-VL).

BenchmarkAgreement rate (% of questions)Agree but wrong (% of questions)
ChartQA72.04.6
POPE94.58.6
MMBench86.35.7
Reviewer_BNznOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper explores how to achieve a favorable accuracy–latency trade-off through reasoning transfer and proposes MAI as its main contribution. MAI follows a staged process: mutual verification of the short answers produced by the large and small VLMs, a second comparison between the small VLM’s reasoning-based answer and the large VLM’s initial answer, and, if disagreement persists, reasoning transfer to the large VLM.

Critical strengths

The experiments effectively demonstrate that the proposed approach can reduce inference latency while limiting accuracy loss.

Critical weaknesses

In MAI, the paper’s main contribution, the reliability of mutual verification may vary depending on the nature of the responses, particularly when they are not constrained to a short-answer format like benchmark. Therefore, further consideration is needed regarding how MAI can be applied to more general scenarios.

Questions for authors
  1. Could you evaluate MAI on Math-VR and MMLongBench-Doc to assess whether its accuracy–latency benefits generalize across different task types? Math-VR would test mathematical visual reasoning with a substantial capability gap between the small and large models, while MMLongBench-Doc would test evidence retrieval and integration across long documents.

In particular, can MAI approach the accuracy of the large VLM’s standalone reasoning when the small VLM’s reasoning contains errors or misses important evidence? Comparing both accuracy and end-to-end latency against that baseline under the same evaluation settings would help clarify the method’s applicability and limitations.

  1. How could MAI be extended to tasks requiring longer, free-form responses, such as scene or video description, beyond the short-answer benchmarks evaluated in the paper? In these settings, how would mutual verification reliably assess agreement between responses that differ in wording or level of detail?
8 comments
AuthorsOpenReview ↗

Response to Reviewer BNzn

We would like to thank the reviewer for the valuable time and insightful comments, and we will add the following results and discussions in the revision.


W1: Mutual verification may be less reliable when responses are not short answers; how does MAI apply to more general scenarios?

Thanks for raising this important point. Verifying the response of models is a generic question for benchmarking, i.e., how does the model perform compared to the ground-truth response. In this work, we did not introduce additional component for verification but just follow the default setting of individual benchmark by replacing the ground-truth response by that from the large models. Then, the response of small models is evaluated according to the protocol provided by the benchmark. Therefore, the reliability of mutual verification in this work is inherited from the quality of the benchmark itself. We will make it clear in the revision.


Q1: Do MAI's accuracy–latency benefits carry over to Math-VR and MMLongBench-Doc, and can MAI approach the large model's own reasoning when the small model's reasoning is flawed?

We run the experiments on the suggested datasets as follows. First, we observe that the gap between small and large models is large (i.e., about 29%) on Math-VR that is consistent with the comment of the Reviewer. According to our discussion in Corollary 1 in the submission, the gap of reasoning transfer depends on if small models can cover key tokens of large models. For the challenging math problems, it is hard for small models to generate all key tokens. However, our method can still leverage the information from small models and improve the performance of Large-S by about 8%, which confirms our analysis.

For MMLongBench-Doc, it mainly aims to retrieve useful information from the massive input images, e.g., about 40K input vision tokens for a single query. Therefore, one of the main challenges becomes how the model handles long context rather than reasoning, where MAI shows slight improvement over Large-S on this task. Besides performance, both experiments confirm that MAI runs faster than Large-R.

We would like to thank the reviewer for this great suggestion. While both of tasks demonstrate our proposed method in terms of efficiency and effectiveness under certain degree, their challenges will be our future work. We will add this discussion.

Table R1. Math-VR (Qwen3-VL 2B→8B): accuracy (%) on the full benchmark and latency per request (s, batch size 1, mean over the first 50 questions).

MethodAccuracy (%)Latency (s)
Small-S6.120.16
Small-R9.2421.96
Large-S7.760.19
MAI16.0018.31
Large-R35.3634.65

Table R2. MMLongBench-Doc (Qwen3-VL 2B→8B): accuracy (%) on the full benchmark and latency per request (s, batch size 1, mean over the first 100 questions).

MethodAccuracy (%)Latency (s)
Small-S25.630.64
Small-R28.671.65
Large-S40.150.80
MAI40.391.82
Large-R46.214.50

Q2: How can MAI be extended to free-form responses (e.g., scene or video description), and how would mutual verification judge agreement?

Thanks for this insightful question. As discussed in our response to W1, we follow the criterion provided by the benchmark itself for verification. For long responses, popular benchmarks may have semantic similarity or LLM-as-a-Judge for evaluation and we can follow the same protocol for mutual verification. We will add this discussion to the revision.

I understand that MAI may offer only limited performance gains on Math-VR and MMLongBench-Doc. However, unless MAI’s intended scope is restricted to specific benchmark workloads, I believe further evaluation on general workloads is needed. In particular, using LLM-as-a-Judge for mutual verification, as suggested in your response, should be evaluated to determine whether MAI’s quality–latency benefits persist after accounting for verification errors and the additional latency. For these reasons, I will maintain my current score.

Dear Reviewer BNzn,

Thank you for your timely response. We understand your concerns regarding the general applicability of MAI. We would like to further clarify two points.

First, regarding the generalizability of MAI, we would like to emphasize that MAI is not designed for specific benchmark workloads. Although Math-VR and MMLongBench-Doc represent relatively specialized tasks (note MAI still improves baseline on them), the benchmarks in our original submission cover a broader range of general-purpose model capabilities and have been widely adopted for evaluation. The improvements demonstrated across these benchmarks provide empirical evidence that MAI's effectiveness is not restricted to a particular workload or task category. Therefore, we believe our existing evaluation already provides meaningful evidence of MAI's applicability to general workloads.

Second, regarding mutual verification and its overhead, we believe there is an important distinction that may not have been sufficiently clear in our previous response: MAI does not introduce an additional verification model or a new verification criterion. Instead, it directly adopts the evaluation protocol of the underlying benchmark, using the large model's response in place of the ground-truth response for mutual verification. Thus, the reliability of verification follows the benchmark's existing evaluation protocol, rather than depending on an additional component introduced by MAI. For free-form tasks, semantic similarity or LLM-as-a-Judge can be adopted only when they are already part of the corresponding evaluation protocol. In particular, LLM-as-a-Judge is merely one possible evaluation strategy, not a mandatory component of MAI or part of its core contribution. Therefore, its potential overhead should not be regarded as an inherent overhead of MAI. Our current evaluation also covers diverse verification strategies across different benchmarks, demonstrating that MAI is not tied to a particular verification mechanism.

In summary, MAI's general applicability should be distinguished from the choice of task-specific evaluation protocols. We hope this clarification resolves the concern that MAI inherently requires additional LLM-based verification and demonstrates that its applicability is not restricted to specific benchmark workloads.

We sincerely appreciate your constructive feedback and hope you will reconsider your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.