acceptodds
Withdrawn from ICLR 2027

Accelerating Agentic Multimodal LLMs via Asynchronous Tool-Trajectory Speculation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic multimodal large language models use tools to acquire additional visual evidence, but repeated alternation between large-model inference and tool execution accumulates latency. Replacing the entire agent with a smaller model can reduce accuracy, motivating a division of work between exploratory action generation and final answering. We introduce , an asynchronous speculative execution framework in which a small draft model generates and executes tool actions while a large main model assesses the resulting evidence and remains responsible for answering. Separate quality and answerability assessments guide continued exploration, early answering, and main-model takeover. Assessment overlaps with draft exploration, while takeover reuses completed tool interactions to limit repeated work. Across five benchmarks, SpecAgent achieves – end-to-end speedup over the Qwen3.5-27B agent while maintaining broadly comparable accuracy. These results demonstrate an accuracy–latency trade-off through small–large model collaboration in tool-assisted multimodal reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Multimodal LLMs, Speculative Decoding, Efficient Agentic MLLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZVCKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SpecAgent, a training-free framework for accelerating tool-assisted multimodal reasoning through small–large model collaboration. A smaller draft model generates and executes exploratory tool actions, while a larger main model asynchronously assesses the resulting evidence for usefulness and answer sufficiency. These assessments govern early answering and main-model takeover. Completed tool interactions are retained as reference context when takeover occurs.

The framework is evaluated on five multimodal benchmarks using Qwen3.5-9B and Qwen3.5-27B. The authors report 1.34×–2.70× speedups over the main-model agent with broadly comparable or improved accuracy. Additional experiments examine assessment components, thresholds, reference tool-use depth, and serving concurrency.

Critical strengths

The complete framework achieves a promising empirical accuracy–latency balance. Across five benchmarks, SpecAgent reduces mean end-to-end latency while improving accuracy on three benchmarks and showing small decreases on two. This supports the practical value of allocating exploratory tool use and final answer assessment to different models.

The assessment ablation provides evidence for separating progress evaluation from answer sufficiency. On VStar, combining quality and answerability assessment retains the quality-only variant’s accuracy while reducing its latency. This supports the coordination design beyond simply replacing the main model with a smaller agent.

Critical weaknesses
  1. The experiments do not establish the incremental benefit of asynchronous execution, despite its central role in the contribution. SpecAgent combines smaller-model exploration, an initial answerability check, early stopping, quality assessment, terminal verification, and takeover with context reuse. The reported comparisons evaluate this combination, while the ablations remove assessment components rather than isolate asynchronous scheduling. An otherwise identical synchronous draft–main baseline is missing. Consequently, the results do not establish how much acceleration comes from overlap rather than cheaper exploration or avoiding unnecessary tool calls. Proposition 3.1 establishes the expected pipeline benefit under fixed work and no contention, but does not resolve this empirical attribution problem. This substantially weakens the evidence for the paper’s central contribution.
  2. The evaluation insufficiently establishes when the proposed coordination is necessary and consequential. Single image–question benchmarks are a legitimate latency optimization setting, but the paper does not show that its coordination complexity provides substantial value over simpler draft-then-verify or selectively routed systems. Evaluation uses one model pair, and the serving study shows that the advantage narrows under increased concurrency, with timeouts at higher load. Context reuse also does not restore tool environment state, limiting its applicability to stateful workflows. Together, these limitations leave the practical scope of the method insufficiently established. Given the straightforward delegation, assessment, and takeover mechanisms, stronger comparative evidence is needed to substantiate the significance of the contribution.
Questions for authors
  1. How does SpecAgent compare with an otherwise identical synchronous implementation using the same models, assessments, budgets, and takeover rules? What fraction of the observed acceleration is attributable specifically to overlap?
  2. How does it compare with a simpler draft-then-verify system, where the draft completes exploration before the main model verifies the answer or takes over?
  3. What proportions of queries terminate through the initial answerability check, intermediate evidence assessment, terminal verification, and takeover? What are the latency and accuracy results for each path?
Reviewer_qxLROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SpecAgent, a training-free framework for accelerating tool-using multimodal agents through asynchronous collaboration between a smaller draft model and a larger main model. The draft model generates and executes exploratory tool calls, while the main model assesses completed observations for both quality and answerability. The assessments support early answering and trigger main-model takeover when exploration repeatedly fails.

Critical strengths
  1. The separation of exploration, evidence assessment, and answer acceptance is well motivated. Delegating exploratory tool use to a smaller model while retaining the larger model for assessment and answering provides a practical way to overlap computation.

  2. The component ablation provides useful evidence for the complementary roles of the two assessments.

Critical weaknesses
  1. Although the paper is well motivated, the method itself cannot be seen as novel. There are many papers about small models; big models review and answer.

2.SpecAgent claims that the efficiency mainly comes from running time. But when you consider a small-big model mixed system, the efficiency should come from both time and cost. But in this case, I do not think SpecAgent's cost would reduce, since it extensively requires big model involvement.

Questions for authors
  1. Could the authors give the inference cost of the method, like equivalent money value, to show real efficiency?

  2. Could the authors run experiments on more general tasks? VStar as the main benchmark seems quite simple for this task. More importantly, inituitively thinking, SpecAgent should benefit when it comes to difficult tasks, which requires long-horizon steps. But in contrast, on VStar, SpecAgent only benefits a lot from 2 tool calls. Could the authors also explain this?

Reviewer_9h1wOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper is targeting to reduce the inference latency of agentic multimodal LLMs via lossy speculative deocoding idea. To do so, the authors propose SpecAgent, which uses a smaller draft model to generate CoT reasoing and tool calls, and execute them while a larger main model assesses the returned tool call results. The main innovation is the assessment which is separated into quality and answerability, checking whether the evidence is useful and whether it is sufficient to answer the question. Based on these assessments, the framework decides whether to continue exploration, return an answer, or let the target model take over. The authors further introduce asynchronous execution to overlap drafting with heavy verification. Experiments cover several multimodal benchmarks. The authors report faster inference with comparable accuracy, and further analyze the effects of the assessment components and their thresholds.

Critical strengths
  • The paper is well written and easy to follow. The problem definition, motivation, and overall workflow are very clear. The experiments also show that the basic idea of speculative reasoning can work in the evaluated agentic multimodal setting, with a smaller model performing tool exploration and a larger model assessing the results and producing the final answer.
  • The main text provides clear experimental analyses and useful insights. For example, the experiments show the different roles of quality and answerability assessment, and that stricter assessment thresholds do not necessarily lead to better final accuracy. The analysis across different tool-use depths also helps explain where the acceleration comes from. These results are clearly presented and help readers understand the behavior and trade-offs of the framework.
Critical weaknesses
  • The novelty of the paper is very limited. At the pipeline level, the framework is largely an application of SpecReason and DREAM-R to an agent workflow. The main change is moving the assessment to the evidence obtained after tool execution. At the component level, the assessment calculation and other core ideas are already present in existing methods. The potentially new part is separating the assessment into quality and answerability, and using their outputs to control answering, continued exploration, and takeover. However, there are still concerns about this design, which I discuss in the questions below. Asynchronous execution is also a well established and standard approach for production. For example, AMUSD already presented asynchronous drafting and verification in 2024, so asynchronous execution itself does not constitute a new contribution. Overall, I think SpecAgent is a good teachnical practice, but its novelty is not enough for this conference.
Questions for authors
  1. As discussed in the weakness, my concern is that the paper is mainly applying existing speculative reasoning methods to an agent workflow. Could the authors compare against adaptations of SpecReason and DREAM-R to the same agent workflow, using the same models, tools, and comparable budgets?
  2. Section 4.3 shows complementary effects, but why use two separate assessments rather than one joint assessment? After a BAD assessment, how often does continuing the draft still lead to a correct answer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.