acceptodds
Withdrawn from ICLR 2027

Sub-Planner: A Generalist Agent Framework for Multi-Domain Reasoning via Adaptive Sub-problem Planning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Domain-specific downstream tasks often require professional knowledge and multi-step reasoning to derive reliable solutions in specialized scenarios. Multimodal agents offer a promising solution by enabling MLLMs to perform multi-step reasoning and leverage external tools. However, most existing agents are typically built on general models and may lack sufficient domain knowledge. Domain-specific agents often rely on reasoning and tool-use pipelines tailored to specialized domains, limiting their transferability across domains. To address these limitations, we propose a general training-free agent framework that generalizes across diverse specialized domains. Our framework combines a general MLLM for planning and verification with a domain-specific MLLM for answering specialized questions. In the planning stage, the general model adaptively generates sub-problems based on the original task and selects appropriate domain-specific tools to collect supporting evidence. In the answering stage, the domain-specific model then performs reasoning on these sub-problems based on the collected evidence. In the verification stage, the general model further verifies the consistency between tool outputs and sub-problem answers, retaining only validated evidence for final reasoning. Our framework provides a unified training-free solution that requires no domain-specific adaptation and can flexibly integrate various general and domain-specific MLLMs for diverse downstream tasks. Experimental results show that our method achieves strong performance across 17 diverse domains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Agent, Multi-Domain Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_o2WxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Sub-Planner, a training-free multimodal agent framework built to handle reasoning in specific fields. It works by teaming up a general-purpose MLLM (to plan, pick tools, and check facts) with specialized MLLMs (to solve the smaller tasks and put together the final answer).

By breaking down problems, using tools to gather evidence, and checking its work at multiple stages, the framework makes reasoning more reliable. Sub-Planner is tested on diverse tasks across different fields and it shows improvement against with original generalist model and specialist model. The paper also includes experiments to evaluate inference efficiency, key components, and other aspects.

Critical strengths
  1. Extensive Testing. The authors evaluate the framework across 17 domains and 26 tasks, demonstrating the method’s performance and cross-domain stability. They also swap in different generalist backbones, proving the framework is flexible and not limited to a single model.

  2. Supporting Ablations. The authors provide ablation experiments on the effects of the framework's components and its inference cost. These help us better understand the method.

Critical weaknesses
  1. Limited Novelty. The framework mostly just pieces together existing ideas: using a general model to break down tasks and pick tools, relying on specialized models for answers, and adding a verification step. These concepts are already well-established in prior work like HuggingGPT [1], Chameleon [2], MultiMedRes [3], and MEXA [4]. Especially, the paper doesn't discuss these closely related works, it is hard to see what is actually new about the method. It appears to merely apply existing simple methods (plan, execute and verify pipeline) to the multi-domain task. That’s clearly insufficient to establish a convincing research contribution.

  2. Unclear Baselines and Reproducibility Issues. The paper includes agent frameworks like ToolScope [5], MMCTAgent [6], and DART [7] in Table 1, but doesn't clearly explain how Sub-Planner differs from them. These methods are not even mentioned in the Related Work sections. Additionally, there aren't enough implementation details for the baselines. For example, methods like MAT [8], SPORT [9], and MARS [10] require training or fine-tuning. Did the authors retrain or adapt these methods to the unified backbone for all the tested domains? Without this information, it is hard to tell if the comparisons are fair or if the results can be reproduced.

  3. Why emphasize multimodality repeatedly? The paper repeatedly highlights multimodal agents and MLLMs, but the actual method, breaking down tasks, using tools, and checking answers, doesn't introduce any features specific to multimodality. Even though the system is tested on multimodal benchmarks, the authors don't explain or show how handling different modalities actually makes the framework better. The repeated mentions of multimodality in the abstract, introduction, and related work sections, without explaining its advantages or demonstrating them experimentally, make this paper difficult to read and its argument unclear. This creates a noticeable gap between the paper's motivation and what the method actually does.

References

[1] Shen et al. (2023). HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. NeurIPS 2023. https://arxiv.org/abs/2303.17580

[2] Lu et al. (2023). Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. NeurIPS 2023. https://arxiv.org/abs/2304.09842

[3] Gu et al. (2025). A Proactive Agent Collaborative Framework for Zero-Shot Multimodal Medical Reasoning (MultiMedRes). Advanced Intelligent Systems. https://doi.org/10.1002/aisy.202400840

[4] Yu et al. (2025). MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation. Findings of EMNLP 2025. https://aclanthology.org/2025.findings-emnlp.1233/

[5] Deng et al. (2026). ToolScope: An Agentic Framework for Vision-Guided and Long-Horizon Tool Use. Findings of ACL 2026. https://aclanthology.org/2026.findings-acl.11/

[6] Kumar et al. (2024). MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoning. arXiv:2405.18358. https://arxiv.org/abs/2405.18358

[7] Sivakumaran et al. (2026). DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning. EACL 2026. https://aclanthology.org/2026.eacl-long.253/

[8] Gao et al. (2025). Multi-Modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage (MAT). ICLR 2025 Workshop. https://arxiv.org/abs/2412.15606

[9] Li et al. (2025). Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning (SPORT). NeurIPS 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/55d16334943f8728073e17139e5baa3d-Abstract-Conference.html

[10] Yue et al. (2026). MARS: Multimodal Adaptive Reasoning Model for Avoiding Overthinking. AAAI 2026. https://doi.org/10.1609/aaai.v40i41.40753

Questions for authors
  1. How were inference budgets, including token consumption and model/tool calls, controlled to ensure fair comparisons across methods? The proposed method appears to require more calls to MLLMs, so why does it show a shorter average inference time? (This question further highlights the issue 2 mentioned above)

  2. Given that most evaluations involve static question-answering tasks in Table 1, what evidence supports the broader claims of generalist agent planning and cross-domain generalization?

  3. Other questions are listed in Critical Weaknesses

Reviewer_6a5SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Sub-Planner combines a general model for planning, tool selection, and verification with a specialist model for answering. It first checks a direct answer, then generates 2–5 sub-problems when needed, filters intermediate evidence, and verifies any final revision. Across 17 domains and 26 tasks, it reports 61.04% average accuracy, compared with 55.63% for specialist models. The paper also evaluates component removal, alternative general models, latency, and verifier reliability.

Critical strengths
  1. The evaluation covers an unusually broad range of specialist tasks (Table 1). Using one general backbone and giving competing agents access to the same tools and specialists is a useful attempt to test a reusable workflow. Gains over specialist-only answers suggest practical value beyond simply selecting a specialist.

  2. The paper examines both utility and reliability. Table 3 tests four general backbones; Figure 2 reports latency; Appendix A.4 manually checks 500 intermediate decisions and 500 final revisions. The intermediate verifier rejects 238/250 incorrect answers while retaining 209/250 correct ones. These checks give the verification design empirical support.

Critical weaknesses
  1. The evidence does not cleanly isolate the proposed contribution. Planning, tool use, and visual criticism already appear in the cited MMCTAgent. Table 2 uses nested removal rows, but does not clearly say whether removals are cumulative or independent, or what replaces each removed component. Figure 5 also leaves the fixed-question strategy unspecified. It is therefore hard to tell how much of the gain comes from adaptive decomposition, role assignment, or additional inference.

  2. Important details needed to assess the main comparison are missing. Section 4 and Appendix A name datasets, models, and tool interfaces, but do not specify evaluation splits/example IDs, exact prompts and decoding limits, or baseline adaptation procedures. Three seeds are reported without dispersion. These omissions make the large cross-domain result difficult to verify.

Questions for authors
  1. Are the Table 2 removals cumulative or independent? What exactly replaces the planner and tool planner, and what fixed questions are used in Figure 5? A controlled comparison on the existing five-domain subset could materially clarify the source of the gains.

  2. Can you supply the evaluation splits/example IDs, prompts, decoding and tool-call budgets, baseline adaptations, and per-seed results?

Reviewer_jfmkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper uses a multi-agent system with a fixed workflow to plan complex multi-modal reasoning. It achieves good experimental results across 17 different domains and demonstrates superior reasoning performance without training. Overall, the paper is clearly structured and easy to read. It incorporates many prior works in its definition of workflows. The paper takes an engineering-oriented approach by expanding the number of tools and domains, thereby demonstrating the scalability for the planning and verification capability of multi-agent sysetem.

Critical strengths
  1. Large-scale experiments involving domain data and tool invocations, which require no training, have significant practical value.
  2. While improvements are minimal on only 7 out of 27 datasets, others are significant.
Critical weaknesses
  1. The experimental setup differs from other baseline models; when sampling 300 test samples, results should obtain the mean and variance from multiple seed experiments.
  2. The appendix merely introduces the models and tools, without providing any tool implementations, prompts for each stage, an anonymous repository, or supporting materials, raising serious concerns about reproducibility.
Questions for authors
  1. All the examples shown are multiple-choice questions, which can be repeatedly verified and modified. Is this approach still applicable to open-ended problems? At what stage does it need to be expanded?
  2. Without training, would it be more feasible to rethink the approach based on the validation results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.