- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces MetaForge, a multimodal agent that learns when to use existing tools and when to create new ones. It trains the agent with GRPO and refines tools using execution feedback, with evaluation on 12 multimodal benchmarks.
Critical strengths
-
Substantial implementation effort. The framework brings together tool selection, generation, refinement, and management, addressing several practical aspects of building a multimodal agent.
-
Broad experimental coverage. The experiments cover multiple benchmarks, baselines, and backbones. The component ablations also provide useful evidence about how different parts of the system affect performance.
Critical weaknesses
-
Heavy engineering integration and limited novelty. The method largely integrates established techniques and overlaps with prior work such as DynaSaur and SkillWeaver on dynamic tool generation, reuse, and skill refinement. The paper does not clearly isolate its key methodological advance over these frameworks. In particular, the Decide–Retrieve–Adapt–Forge decomposition appears primarily conceptual: the method uses a unified policy, and Retrieve and Adapt share a final answer correctness reward. While the component ablations provide evidence of their usefulness, they do not establish what this particular four stage decomposition contributes beyond a simpler tool calling framework.
-
Soundness concerns: reward design and result reporting. The Forge reward in Eqs. (10)–(12) rewards validated tool creation or error-free reuse, conditional on a correct final answer, without checking whether the tool contributes to solving the task. Creating a valid tool can therefore earn a bonus even if it is never used to obtain the answer, leaving an incentive for unnecessary forging. Additionally, the reported MathVista accuracy of 78.01% is inconsistent with the stated evaluation protocol. With binary scoring over 1,000 examples averaged across three runs, the mean accuracy must be an integer multiple of 100/3000 percentage points; 78.01% cannot be obtained under this protocol even after rounding to two decimal places. This discrepancy requires clarification of the per-run correct counts and aggregation procedure, and reduces my confidence in the reliability of the reported results.
Questions for authors
Table 1 is difficult to read because the font size is too small. Some of the reported results also appear questionable and require clarification. I encourage the authors to place less emphasis on engineering integration and focus more on in-depth analysis that yields clearer methodological insights.
6 comments
Part 2/4
W1-3: What does the MetaForge contribute beyond simpler tool-calling frameworks?
To explain what MetaForge adds beyond a simpler tool-calling framework, we built 'DynaSaur-adapted' from the original Qwen3-VL-8B-Instruct model. We follow DynaSaur’s procedure for selecting and retrieving tools. For a fair comparison, we replace the agent’s direct code generation with calls to the same frozen model that MetaForge uses to generate tools. The framework runs the generated code. If execution fails, the agent uses the error information to request revisions, and the framework runs the updated code. After successful execution, it saves newly defined functions for later reuse. Each generation or repair counts as one attempt. We do not train DynaSaur-adapted or add MetaForge’s separate semantic validation, semantic deduplication, capacity management, or recursive tool improvement during training. Both methods use identical predefined tools, dataset order, and interaction budgets. We evaluate both methods on all 12 datasets under IID and OOD tool settings. Each method completes only one inference run per tool setting, and we save all outputs. Since our current evaluation environment no longer supports GPT-5, we use Qwen3.8-Max for both methods to ensure a fair comparison. It independently scores each method’s saved outputs three times. We report the mean accuracy and the sample standard deviation across these three scoring rounds.
IID Tool Setting
| Metric | DynaSaur-adapted | MetaForge |
|---|---|---|
| DocVQA accuracy (%) | 90.30 ± 0.10 | 97.83 ± 0.18 |
| TallyQA accuracy (%) | 70.85 ± 0.15 | 80.38 ± 0.31 |
| OCRVQA accuracy (%) | 77.90 ± 0.10 | 86.33 ± 0.23 |
| RoBUT-WikiSQL accuracy (%) | 62.90 ± 0.20 | 87.28 ± 0.28 |
| MapQA accuracy (%) | 71.20 ± 0.15 | 77.88 ± 0.38 |
| Localized Narratives accuracy (%) | 95.86 ± 0.07 | 99.48 ± 0.18 |
| ChartQA accuracy (%) | 82.52 ± 0.08 | 90.45 ± 0.22 |
| AI2D accuracy (%) | 84.84 ± 0.10 | 89.53 ± 0.21 |
| WebSight accuracy (%) | 54.40 ± 0.20 | 98.20 ± 0.33 |
| MathVista accuracy (%) | 61.70 ± 0.10 | 78.07 ± 0.25 |
| ScienceQA accuracy (%) | 78.43 ± 0.10 | 89.56 ± 0.27 |
| CLEVR-Math accuracy (%) | 64.60 ± 0.15 | 81.57 ± 0.35 |
| IID datasets average (%) | 78.17 ± 0.13 | 88.20 ± 0.26 |
| OOD datasets average (%) | 71.08 ± 0.12 | 87.90 ± 0.27 |
| ALL AVG. (%) | 74.63 ± 0.12 | 88.05 ± 0.26 |
| Attempts / Registered | 262 / 147 | 190 / 123 |
| Failed attempts | 115 | 51 |
| Semantically duplicate candidates | 68 | 16 |
| Total inference runtime (hours) | 4.92 | 4.28 |
OOD Tool Setting
| Metric | DynaSaur-adapted | MetaForge |
|---|---|---|
| DocVQA accuracy (%) | 90.55 ± 0.10 | 97.13 ± 0.23 |
| TallyQA accuracy (%) | 71.80 ± 0.15 | 80.98 ± 0.28 |
| OCRVQA accuracy (%) | 77.35 ± 0.10 | 85.92 ± 0.30 |
| RoBUT-WikiSQL accuracy (%) | 66.40 ± 0.20 | 87.62 ± 0.33 |
| MapQA accuracy (%) | 73.80 ± 0.15 | 89.05 ± 0.35 |
| Localized Narratives accuracy (%) | 95.79 ± 0.07 | 99.50 ± 0.14 |
| ChartQA accuracy (%) | 82.64 ± 0.08 | 90.17 ± 0.32 |
| AI2D accuracy (%) | 84.91 ± 0.10 | 88.07 ± 0.24 |
| WebSight accuracy (%) | 55.10 ± 0.20 | 98.65 ± 0.30 |
| MathVista accuracy (%) | 60.00 ± 0.10 | 79.77 ± 0.45 |
| ScienceQA accuracy (%) | 78.28 ± 0.10 | 89.62 ± 0.27 |
| CLEVR-Math accuracy (%) | 64.25 ± 0.15 | 80.28 ± 0.33 |
| IID datasets average (%) | 79.28 ± 0.13 | 90.03 ± 0.27 |
| OOD datasets average (%) | 70.86 ± 0.12 | 87.76 ± 0.32 |
| ALL AVG. (%) | 75.07 ± 0.12 | 88.90 ± 0.29 |
| Attempts / Registered | 196 / 55 | 12 / 11 |
| Failed attempts | 141 | 0 |
| Semantically duplicate candidates | 23 | 1 |
| Total inference runtime (hours) | 3.36 | 2.55 |
Note: For DynaSaur-adapted, we only count duplicate tools, but do not remove them.
Although 'DynaSaur-adapted' follows prompts that prioritize existing and retrieved functions before defining new ones, it makes more generation attempts, encounters more failures and duplicates, and takes longer to complete evaluation under both tool settings. MetaForge combines an efficiency reward with a reward for tool generation and reuse through multi-turn GRPO. The former favors correct trajectories with lower tool-call latency, while the latter rewards validated tool creation and successful reuse conditional on a correct final answer. Semantic deduplication and capacity management additionally control redundancy and tool-pool size. Despite the additional validation and management overhead, MetaForge achieves higher accuracy with lower total inference runtime. This comparison supports the benefits of MetaForge’s training and tool management mechanisms over the adapted prompting framework.
Part 3/4
W2-1: Do generated tools help solve tasks, and are any redundant?
The reviewer asks whether generated tools help solve tasks and whether the reward encourages redundant generation, we clarify the following. The reward ablation in Appendix B.7, Table 13 shows that removing the Forge reward reduces overall accuracy under both IID and OOD tool settings, indicating its positive effect on task performance. The tool-generation ablation in Appendix B.3, Table 10 further shows that overall accuracy decreases when training and evaluation use only predefined tools. Together, these results indicate that generated tools help solve tasks and improve performance compared with predefined tools alone. Meanwhile, the efficiency reward constrains additional tool-call costs. Among correct trajectories for the same question, those with lower cumulative tool-call latency receive a higher efficiency reward. The tool-generation statistics in Table 2 show that, under IID tools, predefined tools leave larger capability gaps on some evaluation tasks. MetaForge generates more tools to address these gaps, keeps generation attempts under control, and achieves the highest tool registration success rate. Under OOD tools, predefined tools cover more task requirements. MetaForge makes relatively few generation attempts while maintaining the highest tool registration success rate. MetaForge achieves the highest accuracy under both settings. These results suggest that MetaForge adapts its tool-generation behavior to task needs, helping solve tasks more accurately while keeping tool-generation costs under control.
W2-2 & Q1-2: The reported results raise reliability concerns and require further clarification.
We clarify the scoring procedure as follows. We use GPT-5, GPT-5-mini, and GPT-5-nano to separately assign binary scores to the same set of model inference outputs. We multiply their raw accuracies by 0.5, 0.3, and 0.2, respectively, round each weighted value to two decimal places, and sum them to obtain the final accuracy. The table below presents MetaForge’s accuracy under each judge and the weighted results.
| Tool setting | Dataset | GPT-5 Correct Count | GPT-5 Raw ACC (%) | GPT-5 Weighted (0.5) | GPT-5-mini Correct Count | GPT-5-mini Raw ACC (%) | GPT-5-mini Weighted (0.3) | GPT-5-nano Correct Count | GPT-5-nano Raw ACC (%) | GPT-5-nano Weighted (0.2) | Final ACC (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| IID | DocVQA | 1958 | 97.9000 | 48.95 | 1958 | 97.9000 | 29.37 | 1958 | 97.9000 | 19.58 | 97.90 |
| IID | TallyQA | 1610 | 80.5000 | 40.25 | 1611 | 80.5500 | 24.17 | 1610 | 80.5000 | 16.10 | 80.52 |
| IID | OCRVQA | 1730 | 86.5000 | 43.25 | 1730 | 86.5000 | 25.95 | 1731 | 86.5500 | 17.31 | 86.51 |
| IID | RoBUT | 1747 | 87.3500 | 43.68 | 1747 | 87.3500 | 26.21 | 1747 | 87.3500 | 17.47 | 87.36 |
| IID | MapQA | 1559 | 77.9500 | 38.98 | 1560 | 78.0000 | 23.40 | 1560 | 78.0000 | 15.60 | 77.98 |
| IID | Localized Narratives | 1395 | 99.6429 | 49.82 | 1394 | 99.5714 | 29.87 | 1395 | 99.6429 | 19.93 | 99.62 |
| IID | ChartQA | 2262 | 90.4800 | 45.24 | 2263 | 90.5200 | 27.16 | 2261 | 90.4400 | 18.09 | 90.49 |
| IID | AI2D | 2767 | 89.6049 | 44.80 | 2767 | 89.6049 | 26.88 | 2767 | 89.6049 | 17.92 | 89.60 |
| IID | WebSight | 1967 | 98.3500 | 49.18 | 1966 | 98.3000 | 29.49 | 1967 | 98.3500 | 19.67 | 98.34 |
| IID | MathVista | 779 | 77.9000 | 38.95 | 780 | 78.0000 | 23.40 | 783 | 78.3000 | 15.66 | 78.01 |
| IID | ScienceQA | 1809 | 89.6877 | 44.84 | 1809 | 89.6877 | 26.91 | 1808 | 89.6381 | 17.93 | 89.68 |
| IID | CLEVR-Math | 1634 | 81.7000 | 40.85 | 1634 | 81.7000 | 24.51 | 1634 | 81.7000 | 16.34 | 81.70 |
| OOD | DocVQA | 1944 | 97.2000 | 48.60 | 1945 | 97.2500 | 29.18 | 1944 | 97.2000 | 19.44 | 97.22 |
| OOD | TallyQA | 1623 | 81.1500 | 40.58 | 1623 | 81.1500 | 24.35 | 1624 | 81.2000 | 16.24 | 81.17 |
| OOD | OCRVQA | 1720 | 86.0000 | 43.00 | 1720 | 86.0000 | 25.80 | 1720 | 86.0000 | 17.20 | 86.00 |
| OOD | RoBUT | 1753 | 87.6500 | 43.83 | 1753 | 87.6500 | 26.30 | 1754 | 87.7000 | 17.54 | 87.67 |
| OOD | MapQA | 1783 | 89.1500 | 44.58 | 1782 | 89.1000 | 26.73 | 1783 | 89.1500 | 17.83 | 89.14 |
| OOD | Localized Narratives | 1394 | 99.5714 | 49.79 | 1395 | 99.6429 | 29.89 | 1393 | 99.5000 | 19.90 | 99.58 |
| OOD | ChartQA | 2254 | 90.1600 | 45.08 | 2255 | 90.2000 | 27.06 | 2255 | 90.2000 | 18.04 | 90.18 |
| OOD | AI2D | 2722 | 88.1477 | 44.07 | 2722 | 88.1477 | 26.44 | 2723 | 88.1801 | 17.64 | 88.15 |
| OOD | WebSight | 1974 | 98.7000 | 49.35 | 1975 | 98.7500 | 29.63 | 1974 | 98.7000 | 19.74 | 98.72 |
| OOD | MathVista | 798 | 79.8000 | 39.90 | 798 | 79.8000 | 23.94 | 797 | 79.7000 | 15.94 | 79.78 |
| OOD | ScienceQA | 1810 | 89.7372 | 44.87 | 1809 | 89.6877 | 26.91 | 1810 | 89.7372 | 17.95 | 89.73 |
| OOD | CLEVR-Math | 1608 | 80.4000 | 40.20 | 1608 | 80.4000 | 24.12 | 1608 | 80.4000 | 16.08 | 80.40 |
Part 1/4
Thank you for your careful review and constructive feedback. We hope our clarifications and additional experiments address your concerns, and we welcome any further questions.
Regarding the novelty of MetaForge, we clarify the following. DynaSaur[1] and SkillWeaver[2] primarily use prompts and execution feedback to guide LLMs in generating, reusing, and refining tools. However, Table 2 shows that baselines relying solely on prompt constraints still produce numerous tool generation errors and duplicate tools. MetaForge uses multi-turn GRPO with efficiency and tool generation rewards. The model learns to generate new tools only when existing tools cannot meet the current task’s requirements, reducing redundant generation and ineffective calls. Furthermore, although DynaSaur and SkillWeaver provide tool retrieval and filtering mechanisms, they do not account for the increasing retrieval and filtering burden as the tool pool expands. MetaForge applies semantic deduplication and retains valuable skills based on reuse frequency and functional demand, further reducing tool selection overhead.
To address the reviewer’s questions about whether the four-stage decomposition is merely conceptual, why we adopt a unified policy, and why Retrieve and Adapt share a reward, we clarify the following. Decide–Retrieve–Adapt–Forge describes the decision process we want the model to learn. Through reward design, we train a single model to learn this process and make tool decisions, which is why we adopt a unified policy. Specifically, Decide determines whether to use tools. The efficiency reward assigns higher scores to correct trajectories with lower tool-call latency for the same question, guiding the model to reduce unnecessary operations. Retrieve selects tools, and Adapt determines their parameters. Both affect task completion, so they share the final-answer correctness reward for joint optimization. Forge requests new tools when existing tools cannot meet task requirements. Its reward encourages tool generation that passes validation and successful tool reuse only when the final answer is correct. These rewards jointly train the unified model to learn this decision process through task correctness, tool-call efficiency, and tool generation and reuse.
[1] DynaSaur: Large Language Agents Beyond Predefined Actions, COLM 2025.
[2] Skillweaver: Web agents can self-improve by discovering and honing skills, arXiv 2025.