acceptodds
Withdrawn from ICLR 2027

MetaForge: A Multimodal Model that Retrieves, Adapts, and Forges Tools On Demand

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal models achieve strong performance on complex reasoning tasks through tool use, but they still face two limitations. Static, predefined tool inventories fail to meet the needs of unseen tasks, while indiscriminate tool invocation increases cost and introduces noise-induced errors. To address these issues, we propose MetaForge, a multimodal model that learns when to invoke existing tools and when to request new tools for capability gaps. MetaForge organizes tool use into four stages: Decide determines whether tools are needed, Retrieve selects suitable tools, and Adapt adjusts call parameters. Forge generates, validates, and registers new tools when existing tools are insufficient. For registered tools, MetaForge performs tool-level recursive self-improvement during training using execution feedback to improve their cross-task generalization. Meanwhile, we train the orchestration policy with multi-turn GRPO and composite rewards to optimize tool selection, invocation efficiency, tool reuse, and output formatting. Additionally, we propose capacity-constrained dynamic tool pool management to retain better effective tools. Evaluations on 12 benchmarks against 16 baselines show that MetaForge achieves higher overall accuracy across different task and tool settings. These experiments demonstrate the effectiveness of shifting from static tool inventories to on-demand tool evolution.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Model, On-Demand Tool Forging, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ts5tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MetaForge, a multimodal agent that learns when to use existing tools and when to create new ones. It trains the agent with GRPO and refines tools using execution feedback, with evaluation on 12 multimodal benchmarks.

Critical strengths
  1. Substantial implementation effort. The framework brings together tool selection, generation, refinement, and management, addressing several practical aspects of building a multimodal agent.

  2. Broad experimental coverage. The experiments cover multiple benchmarks, baselines, and backbones. The component ablations also provide useful evidence about how different parts of the system affect performance.

Critical weaknesses
  1. Heavy engineering integration and limited novelty. The method largely integrates established techniques and overlaps with prior work such as DynaSaur and SkillWeaver on dynamic tool generation, reuse, and skill refinement. The paper does not clearly isolate its key methodological advance over these frameworks. In particular, the Decide–Retrieve–Adapt–Forge decomposition appears primarily conceptual: the method uses a unified policy, and Retrieve and Adapt share a final answer correctness reward. While the component ablations provide evidence of their usefulness, they do not establish what this particular four stage decomposition contributes beyond a simpler tool calling framework.

  2. Soundness concerns: reward design and result reporting. The Forge reward in Eqs. (10)–(12) rewards validated tool creation or error-free reuse, conditional on a correct final answer, without checking whether the tool contributes to solving the task. Creating a valid tool can therefore earn a bonus even if it is never used to obtain the answer, leaving an incentive for unnecessary forging. Additionally, the reported MathVista accuracy of 78.01% is inconsistent with the stated evaluation protocol. With binary scoring over 1,000 examples averaged across three runs, the mean accuracy must be an integer multiple of 100/3000 percentage points; 78.01% cannot be obtained under this protocol even after rounding to two decimal places. This discrepancy requires clarification of the per-run correct counts and aggregation procedure, and reduces my confidence in the reliability of the reported results.

Questions for authors

Table 1 is difficult to read because the font size is too small. Some of the reported results also appear questionable and require clarification. I encourage the authors to place less emphasis on engineering integration and focus more on in-depth analysis that yields clearer methodological insights.

6 comments

Part 1/4

Thank you for your careful review and constructive feedback. We hope our clarifications and additional experiments address your concerns, and we welcome any further questions.

W1-1: What methodological advances does MetaForge offer beyond DynaSaur and SkillWeaver?

Regarding the novelty of MetaForge, we clarify the following. DynaSaur[1] and SkillWeaver[2] primarily use prompts and execution feedback to guide LLMs in generating, reusing, and refining tools. However, Table 2 shows that baselines relying solely on prompt constraints still produce numerous tool generation errors and duplicate tools. MetaForge uses multi-turn GRPO with efficiency and tool generation rewards. The model learns to generate new tools only when existing tools cannot meet the current task’s requirements, reducing redundant generation and ineffective calls. Furthermore, although DynaSaur and SkillWeaver provide tool retrieval and filtering mechanisms, they do not account for the increasing retrieval and filtering burden as the tool pool expands. MetaForge applies semantic deduplication and retains valuable skills based on reuse frequency and functional demand, further reducing tool selection overhead.

W1-2: How does MetaForge’s unified policy support its four-stage decision process?

To address the reviewer’s questions about whether the four-stage decomposition is merely conceptual, why we adopt a unified policy, and why Retrieve and Adapt share a reward, we clarify the following. Decide–Retrieve–Adapt–Forge describes the decision process we want the model to learn. Through reward design, we train a single model to learn this process and make tool decisions, which is why we adopt a unified policy. Specifically, Decide determines whether to use tools. The efficiency reward assigns higher scores to correct trajectories with lower tool-call latency for the same question, guiding the model to reduce unnecessary operations. Retrieve selects tools, and Adapt determines their parameters. Both affect task completion, so they share the final-answer correctness reward for joint optimization. Forge requests new tools when existing tools cannot meet task requirements. Its reward encourages tool generation that passes validation and successful tool reuse only when the final answer is correct. These rewards jointly train the unified model to learn this decision process through task correctness, tool-call efficiency, and tool generation and reuse.

[1] DynaSaur: Large Language Agents Beyond Predefined Actions, COLM 2025.

[2] Skillweaver: Web agents can self-improve by discovering and honing skills, arXiv 2025.

Part 2/4

W1-3: What does the MetaForge contribute beyond simpler tool-calling frameworks?

To explain what MetaForge adds beyond a simpler tool-calling framework, we built 'DynaSaur-adapted' from the original Qwen3-VL-8B-Instruct model. We follow DynaSaur’s procedure for selecting and retrieving tools. For a fair comparison, we replace the agent’s direct code generation with calls to the same frozen model that MetaForge uses to generate tools. The framework runs the generated code. If execution fails, the agent uses the error information to request revisions, and the framework runs the updated code. After successful execution, it saves newly defined functions for later reuse. Each generation or repair counts as one attempt. We do not train DynaSaur-adapted or add MetaForge’s separate semantic validation, semantic deduplication, capacity management, or recursive tool improvement during training. Both methods use identical predefined tools, dataset order, and interaction budgets. We evaluate both methods on all 12 datasets under IID and OOD tool settings. Each method completes only one inference run per tool setting, and we save all outputs. Since our current evaluation environment no longer supports GPT-5, we use Qwen3.8-Max for both methods to ensure a fair comparison. It independently scores each method’s saved outputs three times. We report the mean accuracy and the sample standard deviation across these three scoring rounds.

IID Tool Setting

MetricDynaSaur-adaptedMetaForge
DocVQA accuracy (%)90.30 ± 0.1097.83 ± 0.18
TallyQA accuracy (%)70.85 ± 0.1580.38 ± 0.31
OCRVQA accuracy (%)77.90 ± 0.1086.33 ± 0.23
RoBUT-WikiSQL accuracy (%)62.90 ± 0.2087.28 ± 0.28
MapQA accuracy (%)71.20 ± 0.1577.88 ± 0.38
Localized Narratives accuracy (%)95.86 ± 0.0799.48 ± 0.18
ChartQA accuracy (%)82.52 ± 0.0890.45 ± 0.22
AI2D accuracy (%)84.84 ± 0.1089.53 ± 0.21
WebSight accuracy (%)54.40 ± 0.2098.20 ± 0.33
MathVista accuracy (%)61.70 ± 0.1078.07 ± 0.25
ScienceQA accuracy (%)78.43 ± 0.1089.56 ± 0.27
CLEVR-Math accuracy (%)64.60 ± 0.1581.57 ± 0.35
IID datasets average (%)78.17 ± 0.1388.20 ± 0.26
OOD datasets average (%)71.08 ± 0.1287.90 ± 0.27
ALL AVG. (%)74.63 ± 0.1288.05 ± 0.26
Attempts / Registered262 / 147190 / 123
Failed attempts11551
Semantically duplicate candidates6816
Total inference runtime (hours)4.924.28

OOD Tool Setting

MetricDynaSaur-adaptedMetaForge
DocVQA accuracy (%)90.55 ± 0.1097.13 ± 0.23
TallyQA accuracy (%)71.80 ± 0.1580.98 ± 0.28
OCRVQA accuracy (%)77.35 ± 0.1085.92 ± 0.30
RoBUT-WikiSQL accuracy (%)66.40 ± 0.2087.62 ± 0.33
MapQA accuracy (%)73.80 ± 0.1589.05 ± 0.35
Localized Narratives accuracy (%)95.79 ± 0.0799.50 ± 0.14
ChartQA accuracy (%)82.64 ± 0.0890.17 ± 0.32
AI2D accuracy (%)84.91 ± 0.1088.07 ± 0.24
WebSight accuracy (%)55.10 ± 0.2098.65 ± 0.30
MathVista accuracy (%)60.00 ± 0.1079.77 ± 0.45
ScienceQA accuracy (%)78.28 ± 0.1089.62 ± 0.27
CLEVR-Math accuracy (%)64.25 ± 0.1580.28 ± 0.33
IID datasets average (%)79.28 ± 0.1390.03 ± 0.27
OOD datasets average (%)70.86 ± 0.1287.76 ± 0.32
ALL AVG. (%)75.07 ± 0.1288.90 ± 0.29
Attempts / Registered196 / 5512 / 11
Failed attempts1410
Semantically duplicate candidates231
Total inference runtime (hours)3.362.55

Note: For DynaSaur-adapted, we only count duplicate tools, but do not remove them.

Although 'DynaSaur-adapted' follows prompts that prioritize existing and retrieved functions before defining new ones, it makes more generation attempts, encounters more failures and duplicates, and takes longer to complete evaluation under both tool settings. MetaForge combines an efficiency reward with a reward for tool generation and reuse through multi-turn GRPO. The former favors correct trajectories with lower tool-call latency, while the latter rewards validated tool creation and successful reuse conditional on a correct final answer. Semantic deduplication and capacity management additionally control redundancy and tool-pool size. Despite the additional validation and management overhead, MetaForge achieves higher accuracy with lower total inference runtime. This comparison supports the benefits of MetaForge’s training and tool management mechanisms over the adapted prompting framework.

Part 3/4

W2-1: Do generated tools help solve tasks, and are any redundant?

The reviewer asks whether generated tools help solve tasks and whether the reward encourages redundant generation, we clarify the following. The reward ablation in Appendix B.7, Table 13 shows that removing the Forge reward reduces overall accuracy under both IID and OOD tool settings, indicating its positive effect on task performance. The tool-generation ablation in Appendix B.3, Table 10 further shows that overall accuracy decreases when training and evaluation use only predefined tools. Together, these results indicate that generated tools help solve tasks and improve performance compared with predefined tools alone. Meanwhile, the efficiency reward constrains additional tool-call costs. Among correct trajectories for the same question, those with lower cumulative tool-call latency receive a higher efficiency reward. The tool-generation statistics in Table 2 show that, under IID tools, predefined tools leave larger capability gaps on some evaluation tasks. MetaForge generates more tools to address these gaps, keeps generation attempts under control, and achieves the highest tool registration success rate. Under OOD tools, predefined tools cover more task requirements. MetaForge makes relatively few generation attempts while maintaining the highest tool registration success rate. MetaForge achieves the highest accuracy under both settings. These results suggest that MetaForge adapts its tool-generation behavior to task needs, helping solve tasks more accurately while keeping tool-generation costs under control.

W2-2 & Q1-2: The reported results raise reliability concerns and require further clarification.

We clarify the scoring procedure as follows. We use GPT-5, GPT-5-mini, and GPT-5-nano to separately assign binary scores to the same set of model inference outputs. We multiply their raw accuracies by 0.5, 0.3, and 0.2, respectively, round each weighted value to two decimal places, and sum them to obtain the final accuracy. The table below presents MetaForge’s accuracy under each judge and the weighted results.

Tool settingDatasetGPT-5 Correct CountGPT-5 Raw ACC (%)GPT-5 Weighted (0.5)GPT-5-mini Correct CountGPT-5-mini Raw ACC (%)GPT-5-mini Weighted (0.3)GPT-5-nano Correct CountGPT-5-nano Raw ACC (%)GPT-5-nano Weighted (0.2)Final ACC (%)
IIDDocVQA195897.900048.95195897.900029.37195897.900019.5897.90
IIDTallyQA161080.500040.25161180.550024.17161080.500016.1080.52
IIDOCRVQA173086.500043.25173086.500025.95173186.550017.3186.51
IIDRoBUT174787.350043.68174787.350026.21174787.350017.4787.36
IIDMapQA155977.950038.98156078.000023.40156078.000015.6077.98
IIDLocalized Narratives139599.642949.82139499.571429.87139599.642919.9399.62
IIDChartQA226290.480045.24226390.520027.16226190.440018.0990.49
IIDAI2D276789.604944.80276789.604926.88276789.604917.9289.60
IIDWebSight196798.350049.18196698.300029.49196798.350019.6798.34
IIDMathVista77977.900038.9578078.000023.4078378.300015.6678.01
IIDScienceQA180989.687744.84180989.687726.91180889.638117.9389.68
IIDCLEVR-Math163481.700040.85163481.700024.51163481.700016.3481.70
OODDocVQA194497.200048.60194597.250029.18194497.200019.4497.22
OODTallyQA162381.150040.58162381.150024.35162481.200016.2481.17
OODOCRVQA172086.000043.00172086.000025.80172086.000017.2086.00
OODRoBUT175387.650043.83175387.650026.30175487.700017.5487.67
OODMapQA178389.150044.58178289.100026.73178389.150017.8389.14
OODLocalized Narratives139499.571449.79139599.642929.89139399.500019.9099.58
OODChartQA225490.160045.08225590.200027.06225590.200018.0490.18
OODAI2D272288.147744.07272288.147726.44272388.180117.6488.15
OODWebSight197498.700049.35197598.750029.63197498.700019.7498.72
OODMathVista79879.800039.9079879.800023.9479779.700015.9479.78
OODScienceQA181089.737244.87180989.687726.91181089.737217.9589.73
OODCLEVR-Math160880.400040.20160880.400024.12160880.400016.0880.40
Reviewer_cdFDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MetaForge trains a multimodal policy to decide whether to answer directly, invoke existing tools, or request new tools from a frozen generator. It combines multi-turn GRPO with rewards for correctness, efficiency, tool creation/reuse, and formatting. Generated tools undergo validation, training-time refinement, and capacity-constrained pool management. Across 12 benchmarks, the paper reports average accuracies of 88.14% and 88.98% under IID and OOD tool settings. Component ablations, experiments across three backbones, and tool-version comparisons with a frozen backbone support the benefits of learned orchestration and tool refinement.

Critical strengths

The integration of tool orchestration and tool creation is useful and empirically supported. MetaForge addresses unnecessary tool use and missing capabilities within one policy. Baselines receive the same generator and tool interfaces, making the comparison informative. Removing forging lowers average accuracy by 5.10 points with IID tools and 1.72 points with OOD tools, consistent with forging being more useful when the initial inventory has larger capability gaps.

The tool-refinement experiment provides evidence beyond aggregate ablations. Table 12 fixes the backbone and compares successive tool revisions against independent generation with oracle Best-of-4 selection. Three revisions improve average accuracy from 74.00% to 78.45%, versus 77.05% for Best-of-4. This supports the value of execution feedback in improving tools, separately from policy updates or additional generation attempts.

Critical weaknesses

The generalization claim needs a more carefully controlled evaluation. Hyperparameters and pool capacity are selected using all 12 evaluation domains and both tool settings (Appendix A.4; Tables 8 and 16). The OOD settings are therefore excluded from policy updates but remain visible during model selection. Evaluation also accumulates tools in dataset order, without clearly specifying the initial pool or reset boundaries. Consequently, the results do not cleanly separate transfer to unseen settings from target-distribution tuning and adaptation across the evaluation stream.

The strongest forging result is difficult to verify from the provided details. On WebSight with IID tools, removing forging reduces accuracy from 98.34% to 75.40%. Yet the combined code-and-rendering correctness criterion lacks an explicit rubric or thresholds. The successful C1/C2 tools accept screenshots directly, whereas Appendix D starts from extracted text and bounding boxes. The actual screenshot-processing scripts and their dependencies are needed to establish what computation produces this gain and whether the reported costs include all relevant model calls.

Questions for authors

Do the OOD gains persist when hyperparameters and pool capacity are selected using only IID validation data and tools?

What forged tools are present at the start of evaluation for each method, and when is the pool reset across validation, test runs, and datasets? Does changing the evaluation order materially affect accuracy?

Please specify the WebSight judge prompts, code-comparison rules, and rendering criteria. Can you also provide the generated C1/C2 scripts, identify any models they invoke, and explain how those calls enter the inference-cost measurements?

8 comments

Part 1/6

Thank you for your careful review and constructive feedback. We hope our clarifications and additional experiments address your concerns, and we welcome any further questions.

W1-1 & Q1: Do OOD gains persist after IID-only hyperparameter and capacity selection?

To address the reviewer’s concern that OOD gains may depend on target distribution tuning, we restrict configuration selection to IID validation data and tools. We compare all 27 combinations of three learning rates (1e-6, 3e-6, and 5e-6), three format values (0.1, 0.3, and 0.5), and three tool-pool capacities (10, 30, and 50). A fixed training subset contains 10% of the samples randomly drawn from each training dataset. For each configuration, we initialize the model with the original Qwen3-VL-8B-Instruct weights and train it for one epoch on this subset. The validation set contains 600 samples, with 100 randomly selected from each of the six IID test datasets. Qwen3.8-Max assesses each candidate model’s outputs five times, and we select the configuration with the highest average accuracy. We then randomly select 50% of the samples from each training dataset and train MetaForge for one epoch using this configuration, again starting from the original backbone weights. All remaining experimental settings follow the original paper. MetaForge and Qwen3-VL-8B-Instruct each complete one inference run under each tool setting. Qwen3.8-Max independently scores each model’s outputs three times, and we report the mean accuracy and sample standard deviation across the three evaluations.

Table 1. IID-only Validation Grid

Learning rateFormat rewardK = 10K = 30K = 50
1e-60.175.37 ± 0.2276.40 ± 0.1976.57 ± 0.19
1e-60.374.57 ± 0.1975.37 ± 0.2275.47 ± 0.14
1e-60.574.37 ± 0.2275.40 ± 0.1975.57 ± 0.19
3e-60.177.70 ± 0.2279.20 ± 0.2278.87 ± 0.22
3e-60.376.53 ± 0.2277.87 ± 0.2277.70 ± 0.22
3e-60.576.20 ± 0.2277.53 ± 0.2277.37 ± 0.22
5e-60.176.03 ± 0.2277.53 ± 0.2277.20 ± 0.22
5e-60.375.03 ± 0.2276.03 ± 0.2275.87 ± 0.22
5e-60.573.97 ± 0.3075.37 ± 0.2275.13 ± 0.30

Table 2. Generalization after IID-only Selection

DatasetIID tools Qwen3-VL-8BIID tools MetaForgeOOD tools Qwen3-VL-8BOOD tools MetaForge
DocVQA89.77 ± 0.1291.33 ± 0.0690.13 ± 0.0891.33 ± 0.10
TallyQA70.18 ± 0.1372.68 ± 0.0871.23 ± 0.0873.42 ± 0.03
OCRVQA77.33 ± 0.1079.38 ± 0.0677.05 ± 0.0078.93 ± 0.10
RoBUT61.63 ± 0.0366.70 ± 0.0565.40 ± 0.0568.88 ± 0.10
MapQA70.68 ± 0.1673.72 ± 0.0872.95 ± 0.0576.85 ± 0.13
Localized Narratives95.69 ± 0.1196.50 ± 0.1495.64 ± 0.2596.43 ± 0.07
ChartQA82.13 ± 0.1084.75 ± 0.0982.11 ± 0.0584.73 ± 0.09
AI2D84.59 ± 0.1086.03 ± 0.0284.65 ± 0.0985.95 ± 0.03
WebSight51.17 ± 0.1566.17 ± 0.1051.73 ± 0.0867.70 ± 0.10
MathVista60.57 ± 0.1265.67 ± 0.2558.70 ± 0.1765.00 ± 0.17
ScienceQA78.02 ± 0.1580.90 ± 0.1177.84 ± 0.0080.96 ± 0.13
CLEVR-Math63.53 ± 0.1268.97 ± 0.1663.18 ± 0.0668.52 ± 0.12
IID datasets average77.55 ± 0.0280.05 ± 0.0478.73 ± 0.0780.97 ± 0.03
OOD datasets average70.00 ± 0.0375.41 ± 0.0369.70 ± 0.0275.48 ± 0.02
ALL AVG.73.77 ± 0.0177.73 ± 0.0374.22 ± 0.0378.23 ± 0.02

Part 2/6

The results show that, after selecting hyperparameters and tool pool capacity using only IID validation data and IID tools, MetaForge still outperforms Qwen3-VL-8B-Instruct under both tool settings and maintains performance gains on OOD datasets. This finding indicates that MetaForge transfers its learned tool-use capabilities to task domains and tool sets that are not used for configuration selection, supporting its generalization ability.

W1-2 & Q2-1: How are tool pools initialized and reset across evaluation stages?

To address the reviewer’s question about tool initialization and reset rules, we clarify the following. At the start of each independent validation or test evaluation, and whenever we switch between IID and OOD tool settings, we clear all previously created tools and provide only the predefined tools for the current tool setting and the interfaces for tool creation and execution. Each method independently stores the tools it creates during evaluation. Within each tool setting, we evaluate the 12 test datasets in a predefined order. We apply deduplication and capacity management rules to select and retain newly created, validated tools for reuse in subsequent tasks. We do not clear these tools when switching between datasets.

W1-3 & Q2-2: Does changing the evaluation order materially affect accuracy?

Thank you for highlighting this concern. We agree that retaining and reusing generated tools across datasets can introduce order dependence. Following the reviewer suggestion, we compare MetaForge with Claude Sonnet 4.6, the runner-up in our original evaluation, under three dataset orders (seeds 11, 23, and 37), with three independent inference runs per order under both tool settings.

Order and random seedEvaluation order
11OCRVQA → Localized Narratives → DocVQA → ChartQA → ScienceQA → MapQA → TallyQA → RoBUT → MathVista → CLEVR-Math → WebSight → AI2D
23AI2D → MathVista → Localized Narratives → ScienceQA → OCRVQA → WebSight → RoBUT → ChartQA → CLEVR-Math → DocVQA → TallyQA → MapQA
37MapQA → CLEVR-Math → ChartQA → AI2D → RoBUT → OCRVQA → Localized Narratives → DocVQA → WebSight → TallyQA → MathVista → ScienceQA

In the following two tables, each entry reports mean accuracy (%) ± sample standard deviation (SD) over the three inference runs within the same order. ALL AVG. is the unweighted average across the 12 datasets.

IID Tools Settings

Dataset11 MetaForge11 Claude Sonnet 4.623 MetaForge23 Claude Sonnet 4.637 MetaForge37 Claude Sonnet 4.6
DocVQA97.35 ± 0.1398.00 ± 0.2097.60 ± 0.1098.05 ± 0.0597.67 ± 0.1997.95 ± 0.20
TallyQA80.67 ± 0.1973.50 ± 0.2381.02 ± 0.2373.80 ± 0.1081.15 ± 0.1073.40 ± 0.18
OCRVQA84.82 ± 0.4482.10 ± 0.0985.60 ± 0.3082.40 ± 0.2386.30 ± 0.1582.20 ± 0.17
RoBUT87.55 ± 0.2887.98 ± 0.1386.62 ± 0.4388.20 ± 0.0986.63 ± 0.0687.92 ± 0.13
MapQA80.22 ± 0.1981.70 ± 0.1584.00 ± 0.2382.00 ± 0.1376.50 ± 0.3581.10 ± 0.09
Localized Narratives98.64 ± 0.2683.60 ± 0.1598.71 ± 0.4083.79 ± 0.1299.05 ± 0.1883.69 ± 0.22
ChartQA90.15 ± 0.0692.00 ± 0.0790.80 ± 0.2092.15 ± 0.0689.65 ± 0.3392.05 ± 0.20
AI2D90.40 ± 0.0592.10 ± 0.1684.60 ± 0.3191.65 ± 0.0988.75 ± 0.3691.80 ± 0.09
WebSight98.15 ± 0.1097.08 ± 0.1092.60 ± 0.3997.25 ± 0.0998.00 ± 0.1397.00 ± 0.10
MathVista78.60 ± 0.1772.37 ± 0.2370.00 ± 0.2072.90 ± 0.1079.13 ± 0.2972.57 ± 0.23
ScienceQA88.65 ± 0.2687.69 ± 0.1088.10 ± 0.3087.90 ± 0.1090.15 ± 0.2787.61 ± 0.13
CLEVR-Math81.33 ± 0.1893.00 ± 0.1581.00 ± 0.1893.10 ± 0.1778.80 ± 0.1592.90 ± 0.09
ALL AVG.88.04 ± 0.1686.76 ± 0.1386.72 ± 0.2486.93 ± 0.1087.65 ± 0.1986.68 ± 0.13

Part 3/6

OOD Tools Settings

Dataset11 MetaForge11 Claude Sonnet 4.623 MetaForge23 Claude Sonnet 4.637 MetaForge37 Claude Sonnet 4.6
DocVQA97.13 ± 0.1998.05 ± 0.1397.28 ± 0.1098.13 ± 0.1897.12 ± 0.0897.93 ± 0.16
TallyQA81.17 ± 0.1572.07 ± 0.1081.17 ± 0.1672.20 ± 0.0581.03 ± 0.1272.12 ± 0.13
OCRVQA85.95 ± 0.2282.42 ± 0.1885.82 ± 0.1482.65 ± 0.1586.10 ± 0.1882.33 ± 0.15
RoBUT87.43 ± 0.0687.88 ± 0.0687.23 ± 0.0887.93 ± 0.1987.52 ± 0.1387.92 ± 0.08
MapQA89.08 ± 0.1881.18 ± 0.1889.22 ± 0.0881.37 ± 0.1088.50 ± 0.1081.17 ± 0.16
Localized Narratives99.38 ± 0.1183.40 ± 0.0899.52 ± 0.1183.60 ± 0.1899.45 ± 0.1183.45 ± 0.15
ChartQA90.20 ± 0.1492.03 ± 0.1290.15 ± 0.1092.28 ± 0.1290.00 ± 0.1492.16 ± 0.17
AI2D88.21 ± 0.1492.83 ± 0.1286.79 ± 0.1192.80 ± 0.1587.96 ± 0.1892.59 ± 0.09
WebSight98.62 ± 0.0697.38 ± 0.1297.92 ± 0.1497.53 ± 0.2198.47 ± 0.1097.32 ± 0.10
MathVista79.73 ± 0.1572.93 ± 0.1277.50 ± 0.1073.13 ± 0.1579.60 ± 0.1072.80 ± 0.10
ScienceQA89.56 ± 0.2186.07 ± 0.1089.36 ± 0.0386.17 ± 0.1389.64 ± 0.0586.18 ± 0.10
CLEVR-Math80.18 ± 0.1393.70 ± 0.1079.85 ± 0.1393.80 ± 0.1779.75 ± 0.0993.72 ± 0.16
ALL AVG.88.89 ± 0.1486.66 ± 0.1088.48 ± 0.0986.80 ± 0.1388.76 ± 0.1086.64 ± 0.12

To directly quantify order sensitivity, we additionally report the mean and sample standard deviation across the three order-level ALL AVG. scores. These standard deviations measure cross-order variation, rather than within-order run variability.

Tool settingMetaForge ACC (%)Claude Sonnet 4.6 ACC (%)Mean gap (pp)Orders won by MetaForge
IID tools87.47 ± 0.6886.79 ± 0.13+0.682 / 3
OOD tools88.71 ± 0.2186.70 ± 0.09+2.013 / 3

The results confirm that order can change the ranking under IID tools. MetaForge leads by 1.28 and 0.97 percentage points in 11 and 37, respectively, but trails Claude Sonnet 4.6 by 0.21 points in 23. Thus, our original narrow IID advantage should not be interpreted as an order-independent ranking. Nevertheless, MetaForge achieves higher average accuracy across the tested orders. Under OOD tools, it leads consistently in all three orders, by 2.23, 1.68, and 2.12 points.

Order sensitivity is more pronounced for several tasks under IID tools. For example, MetaForge’s MathVista accuracy ranges from 70.00% to 79.13% under IID tools, compared with 77.50%–79.73% under OOD tools. This pattern is consistent with earlier tasks contributing reusable capabilities that benefit later tasks, while broader initial tool coverage may reduce that dependence. Establishing which specific tools cause these differences would require further analysis of generation and reuse traces.

We will incorporate these results and qualify our conclusion: MetaForge achieves higher average accuracy across the tested orders, while its narrow advantage under IID tools remains sensitive to evaluation order.

Reviewer_nRwgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MetaForge, a multimodal model that supports dynamic tool evolution. MetaForge divides tool use into four stages: deciding whether a tool is needed, selecting a tool, adapting its parameters, and updating tools. The system can also improve registered tools recursively, while using a dynamic tool pool with a fixed size limit to remove semantically similar tools and keep useful ones.

For training, the authors use multi-turn GRPO with a combined reward to optimize final answer correctness, tool calling efficiency, tool creation and reuse, and output format. Experiments are conducted on 12 multimodal benchmarks, with both IID/OOD datasets and IID/OOD tool settings. The results show that MetaForge performs better than the compared baselines in overall accuracy, tool call success rate, and tool creation success rate.

Critical strengths
  1. The problem is clear and practically useful. The paper points out two main problems with existing methods: fixed tool libraries and using tools without enough consideration. It divides tool use into four stages, Decide, Retrieve, Adapt, and Forge, and each stage has a clear role.
  2. The experiments cover a wide range of settings. The paper evaluates both IID/OOD datasets and IID/OOD tool settings. It also compares closed-source models, open-source multimodal models, and tool-enhanced models. The ablation studies show that GRPO, online tool creation, and recursive tool improvement all help improve performance.
Critical weaknesses
  1. The effect of dataset order on test-time tool evolution is not fully studied. The paper clearly uses test-time tool generation and reuse, and evaluates datasets in a fixed order. Since tools keep accumulating during testing, tools created from earlier datasets may affect the results on later ones. Although all baselines use the same test samples, tool generator, validation process, and evaluation budget, so this is not an obvious unfairness in the starting conditions, the performance gap between MetaForge and the second-best method is quite small. Changing the dataset order may change their ranking. I suggest reporting the average performance and variance across multiple random dataset orders.
  2. The efficiency metrics do not cover the full test-time cost. The test-time process of MetaForge may involve several components, including a 235B tool generation model, a GPT-5 judge, tool semantic validation, tool deduplication, script compilation, and execution. The paper reports the number of interaction turns, generated tokens, and tool call latency, but it does not clearly state whether these metrics include the full time cost of the tool generator and judge. Therefore, the experiments show that its strategy uses relatively few interactions, but they are not enough to prove that it has lower end-to-end inference cost at test time.
  3. The external validity for real-world agent tool ecosystems is still limited. The current experiments mainly use relatively controlled and stateless tools. The paper does not test MetaForge on real agent tools that require authentication, network access, file operations, state management, asynchronous execution, rate limits, or have side effects.
Questions for authors

See weakness

7 comments

Part 1/4

Thank you for your careful review and constructive feedback. We hope our clarifications and additional experiments address your concerns, and we welcome any further questions.

W1: Does test dataset order affect test-time model rankings?

Thank you for highlighting this concern. We agree that retaining and reusing generated tools across datasets can introduce order dependence. Following the reviewer suggestion, we compare MetaForge with Claude Sonnet 4.6, the runner-up in our original evaluation, under three dataset orders (seeds 11, 23, and 37), with three independent inference runs per order under both tool settings.

Order and random seedEvaluation order
11OCRVQA → Localized Narratives → DocVQA → ChartQA → ScienceQA → MapQA → TallyQA → RoBUT → MathVista → CLEVR-Math → WebSight → AI2D
23AI2D → MathVista → Localized Narratives → ScienceQA → OCRVQA → WebSight → RoBUT → ChartQA → CLEVR-Math → DocVQA → TallyQA → MapQA
37MapQA → CLEVR-Math → ChartQA → AI2D → RoBUT → OCRVQA → Localized Narratives → DocVQA → WebSight → TallyQA → MathVista → ScienceQA

In the following two tables, each entry reports mean accuracy (%) ± sample standard deviation (SD) over the three inference runs within the same order. ALL AVG. is the unweighted average across the 12 datasets.

IID Tools Settings

Dataset11 MetaForge11 Claude Sonnet 4.623 MetaForge23 Claude Sonnet 4.637 MetaForge37 Claude Sonnet 4.6
DocVQA97.35 ± 0.1398.00 ± 0.2097.60 ± 0.1098.05 ± 0.0597.67 ± 0.1997.95 ± 0.20
TallyQA80.67 ± 0.1973.50 ± 0.2381.02 ± 0.2373.80 ± 0.1081.15 ± 0.1073.40 ± 0.18
OCRVQA84.82 ± 0.4482.10 ± 0.0985.60 ± 0.3082.40 ± 0.2386.30 ± 0.1582.20 ± 0.17
RoBUT87.55 ± 0.2887.98 ± 0.1386.62 ± 0.4388.20 ± 0.0986.63 ± 0.0687.92 ± 0.13
MapQA80.22 ± 0.1981.70 ± 0.1584.00 ± 0.2382.00 ± 0.1376.50 ± 0.3581.10 ± 0.09
Localized Narratives98.64 ± 0.2683.60 ± 0.1598.71 ± 0.4083.79 ± 0.1299.05 ± 0.1883.69 ± 0.22
ChartQA90.15 ± 0.0692.00 ± 0.0790.80 ± 0.2092.15 ± 0.0689.65 ± 0.3392.05 ± 0.20
AI2D90.40 ± 0.0592.10 ± 0.1684.60 ± 0.3191.65 ± 0.0988.75 ± 0.3691.80 ± 0.09
WebSight98.15 ± 0.1097.08 ± 0.1092.60 ± 0.3997.25 ± 0.0998.00 ± 0.1397.00 ± 0.10
MathVista78.60 ± 0.1772.37 ± 0.2370.00 ± 0.2072.90 ± 0.1079.13 ± 0.2972.57 ± 0.23
ScienceQA88.65 ± 0.2687.69 ± 0.1088.10 ± 0.3087.90 ± 0.1090.15 ± 0.2787.61 ± 0.13
CLEVR-Math81.33 ± 0.1893.00 ± 0.1581.00 ± 0.1893.10 ± 0.1778.80 ± 0.1592.90 ± 0.09
ALL AVG.88.04 ± 0.1686.76 ± 0.1386.72 ± 0.2486.93 ± 0.1087.65 ± 0.1986.68 ± 0.13

Part 2/4

OOD Tools Settings

Dataset11 MetaForge11 Claude Sonnet 4.623 MetaForge23 Claude Sonnet 4.637 MetaForge37 Claude Sonnet 4.6
DocVQA97.13 ± 0.1998.05 ± 0.1397.28 ± 0.1098.13 ± 0.1897.12 ± 0.0897.93 ± 0.16
TallyQA81.17 ± 0.1572.07 ± 0.1081.17 ± 0.1672.20 ± 0.0581.03 ± 0.1272.12 ± 0.13
OCRVQA85.95 ± 0.2282.42 ± 0.1885.82 ± 0.1482.65 ± 0.1586.10 ± 0.1882.33 ± 0.15
RoBUT87.43 ± 0.0687.88 ± 0.0687.23 ± 0.0887.93 ± 0.1987.52 ± 0.1387.92 ± 0.08
MapQA89.08 ± 0.1881.18 ± 0.1889.22 ± 0.0881.37 ± 0.1088.50 ± 0.1081.17 ± 0.16
Localized Narratives99.38 ± 0.1183.40 ± 0.0899.52 ± 0.1183.60 ± 0.1899.45 ± 0.1183.45 ± 0.15
ChartQA90.20 ± 0.1492.03 ± 0.1290.15 ± 0.1092.28 ± 0.1290.00 ± 0.1492.16 ± 0.17
AI2D88.21 ± 0.1492.83 ± 0.1286.79 ± 0.1192.80 ± 0.1587.96 ± 0.1892.59 ± 0.09
WebSight98.62 ± 0.0697.38 ± 0.1297.92 ± 0.1497.53 ± 0.2198.47 ± 0.1097.32 ± 0.10
MathVista79.73 ± 0.1572.93 ± 0.1277.50 ± 0.1073.13 ± 0.1579.60 ± 0.1072.80 ± 0.10
ScienceQA89.56 ± 0.2186.07 ± 0.1089.36 ± 0.0386.17 ± 0.1389.64 ± 0.0586.18 ± 0.10
CLEVR-Math80.18 ± 0.1393.70 ± 0.1079.85 ± 0.1393.80 ± 0.1779.75 ± 0.0993.72 ± 0.16
ALL AVG.88.89 ± 0.1486.66 ± 0.1088.48 ± 0.0986.80 ± 0.1388.76 ± 0.1086.64 ± 0.12

To directly quantify order sensitivity, we additionally report the mean and sample standard deviation across the three order-level ALL AVG. scores. These standard deviations measure cross-order variation, rather than within-order run variability.

Tool settingMetaForge ACC (%)Claude Sonnet 4.6 ACC (%)Mean gap (pp)Orders won by MetaForge
IID tools87.47 ± 0.6886.79 ± 0.13+0.682 / 3
OOD tools88.71 ± 0.2186.70 ± 0.09+2.013 / 3

The results confirm that order can change the ranking under IID tools. MetaForge leads by 1.28 and 0.97 percentage points in 11 and 37, respectively, but trails Claude Sonnet 4.6 by 0.21 points in 23. Thus, our original narrow IID advantage should not be interpreted as an order-independent ranking. Nevertheless, MetaForge achieves higher average accuracy across the tested orders. Under OOD tools, it leads consistently in all three orders, by 2.23, 1.68, and 2.12 points.

Order sensitivity is more pronounced for several tasks under IID tools. For example, MetaForge’s MathVista accuracy ranges from 70.00% to 79.13% under IID tools, compared with 77.50%–79.73% under OOD tools. This pattern is consistent with earlier tasks contributing reusable capabilities that benefit later tasks, while broader initial tool coverage may reduce that dependence. Establishing which specific tools cause these differences would require further analysis of generation and reuse traces.

We will incorporate these results and qualify our conclusion: MetaForge achieves higher average accuracy across the tested orders, while its narrow advantage under IID tools remains sensitive to evaluation order.

Part 3/4

W2: Do efficiency metrics account for all inference?

To address the reviewer’s concern about incomplete cost accounting during testing, we provide a more detailed breakdown of inference costs.

ColumnCalculationUnit
Attempts / RegisteredTotal generation attempts / successfully registered toolsCount
Model inferenceCumulative model inference time ÷ number of model inferencesSeconds
Tool generationCumulative generation time ÷ generation attemptsSeconds
Script checksCumulative compilation and execution-check time ÷ generation attemptsSeconds
Semantic validationCumulative semantic validation time ÷ generation attemptsSeconds
Semantic deduplicationCumulative semantic deduplication time ÷ generation attemptsSeconds
Pool management and overheadCumulative pool management and orchestration time ÷ generation attemptsSeconds
Total online runtimeElapsed time from the start to the end of the inference runHours

IID Tool Setting

ModelAttempts / RegisteredModel inferenceTool generationScript checksSemantic validationSemantic deduplicationPool management and overheadTotal online runtime
Qwen2.5-VL-7B145 / 664.3524.800.621.560.786.964.39
Qwen3-VL-8B34 / 54.2526.400.451.180.5529.532.65
VTool-R1-7B110 / 274.8928.600.731.430.479.154.30
R1-Onevision-7B9 / 55.9922.300.581.620.99111.561.29
PyVision-Image-7B-RL65 / 194.4631.200.921.350.6515.4714.70
DeepEyesV2-7B1286 / 2694.2527.500.641.020.480.805.44
PixelReasoner-RL-v1-7B49 / 254.8725.600.711.680.7220.535.41
AdaReasoner-7B166 / 753.0330.400.861.740.836.095.65
TikArt-8B291 / 1303.4229.100.791.650.993.494.84
ZwZ-8B609 / 1383.8826.800.671.240.591.676.25
MetaForge190 / 1233.1832.600.942.051.225.344.28

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.