- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ToolAtlas, a provider-side shared memory framework tailored for the MCP tool ecosystem. By integrating a three-layer tool memory graph, execution-verified capability frontier exploration, and dynamic memory traversal, ToolAtlas enables the reuse of tool-use knowledge across heterogeneous agents, LLM backbones, and environment instances, achieving substantial performance gains across two MCP benchmarks.
Critical strengths
-
ToolAtlas addresses a meaningful research problem by shifting memory from individual agents to tool providers, enabling shared and reusable tool-use knowledge across agents and reducing redundant learning.
-
ToolAtlas achieves consistent performance improvements across two benchmarks, MCPMark and MCP-Universe, covering eight MCP services. These results provide empirical support for its central claim that tool-use knowledge can be reused across different agents and environment instances.
Critical weaknesses
-
Insufficient control over offline exploration budgets. ToolAtlas generates additional probing tasks and executes more agent rollouts, raising the possibility that its performance gains primarily stem from increased tool interaction experience rather than the proposed tool-side memory graph. Without comparisons against ACE and A-MEM under matched token budgets or equivalent exploration data, it is difficult to determine the extent to which the improvements can be attributed to the proposed method itself.
-
Insufficient isolation of the core contributions. Although ToolAtlas presents provider-side memory as a central contribution, it simultaneously introduces several design choices, including centralized tool-side storage, a three-layer memory graph, active capability exploration, and dynamic memory retrieval. While the paper provides component-level ablation studies, the current experiments do not fully disentangle the individual contributions of these factors, making it difficult to identify which mechanisms are primarily responsible for the observed performance improvements.
-
Insufficient validation of the reliability of execution-based knowledge acquisition. Both the probing tasks and their corresponding Python verifiers are generated by large language models (LLMs). However, the paper does not provide independent evidence regarding the reliability of these verifiers. Consequently, it remains unclear whether incorrect verification outcomes could introduce inaccurate capability or boundary information into the persistent memory.
Questions for authors
-
How would ToolAtlas compare with ACE and A-MEM under matched offline token budgets or using the same exploration data?
-
Can the authors further isolate the contributions of provider-side memory, graph-based organization, and active exploration to the overall performance gains?
-
How do the authors ensure the correctness of LLM-generated verifiers? Have they evaluated whether verification errors introduce inaccurate knowledge into the memory?
7 comments
Response to Reviewer Ycnz (2/6)
1.2 Comparison under Matched Offline Token Budgets
Experimental setup. We match ToolAtlas's offline token budget per service, counting all input and output LLM tokens of training and exploration. ExpeL, A-MEM, and ACE spend the extra budget on random tasks generated with ToolAtlas's seed-generation procedure, followed by rollouts and their own memory updates; ToolAtlas-Random spends it on such random tasks instead of frontier exploration. Evaluation follows 1.1 (Appendix H.2).
Table R2. Performance under Matched Offline Token Budgets (Table 15 in the revised paper)
pass@1 / pass@4 (%). The offline budget is the actual consumption summed across services; budgets are matched per service.
| Method | Offline budget (M) | Filesystem | GitHub | Notion | Playwright | PostgreSQL | Overall |
|---|---|---|---|---|---|---|---|
| Vanilla | 0 | 48.68 / 68.42 | 46.67 / 53.33 | 44.44 / 55.56 | 46.15 / 61.54 | 59.62 / 69.23 | 49.11 / 61.62 |
| ExpeL + Random | ≈148 | 48.68 / 78.95 | 43.33 / 66.67 | 44.44 / 66.67 | 50.00 / 61.54 | 57.69 / 76.92 | 48.83 / 70.15 |
| ACE + Random | ≈148 | 59.21 / 84.21 | 55.00 / 66.67 | 48.61 / 66.67 | 55.77 / 61.54 | 63.46 / 76.92 | 56.41 / 71.20 |
| A-MEM + Random | ≈148 | 61.84 / 84.21 | 45.00 / 60.00 | 44.44 / 55.56 | 50.00 / 61.54 | 51.92 / 76.92 | 50.64 / 67.65 |
| ToolAtlas-Random | ≈148 | 68.42 / 84.21 | 45.00 / 60.00 | 45.83 / 66.67 | 57.69 / 69.23 | 65.38 / 76.92 | 56.46 / 71.41 |
| Full ToolAtlas | ≈148 | 69.73 / 94.73 | 56.67 / 73.33 | 51.39 / 72.22 | 59.62 / 76.92 | 67.30 / 84.61 | 60.94 / 80.36 |
Analysis & Conclusion.
First, more budget alone does not close the gap. At 4.7–5.9× their original construction cost on MCPMark (Table 13), ExpeL, A-MEM, and ACE change their overall pass@1 by only +0.56, +0.73, and +0.06 points, and ACE's pass@4 even drops by 1.54 points, while Full ToolAtlas is the best on every service under both metrics and leads ACE + Random by 4.53 / 9.16 points. This reflects the Capability Blindness that motivates ToolAtlas (Section 1): random tasks cover representative uses of each tool, so extra budget mostly revisits behaviors that are already covered.
Second, the value of the budget lies in where it is spent. Compared with the budget-matched random corpus, the frontier corpus (Table R1) improves overall pass@1 / pass@4 by 1.12 / 0.04 points for ExpeL, 0.96 / 0.22 for A-MEM, 1.47 / 2.59 for ACE, and 4.48 / 8.95 for ToolAtlas. The gap is largest on GitHub (11.67 / 13.33 for ToolAtlas), whose 94 tools make random tasks especially sparse; frontier exploration instead directs the budget toward under-explored affordances and boundaries.
W2 & Q2: Isolating the Core Contributions
ToolAtlas indeed combines several design choices. We isolate each of them with a controlled comparison that holds the others fixed (Table R4 in 2.3 below summarizes them). Each design choice contributes on its own: removing graph organization, replacing frontier exploration with random tasks at the same budget, or replacing dynamic traversal with static retrieval reduces overall pass@1 / pass@4 by 7.06 / 8.67, 4.48 / 8.95, and 3.72 / 6.57 points, respectively. The tool-side, agent-neutral form costs little for the agent that built the memory (−3.91 / −1.76) but becomes decisive once the memory is shared (−11.47 / −8.32). Graph organization and active exploration are moreover complementary rather than independent (2.1 below).
On centralized tool-side storage. Where the memory is stored does not by itself change what a single agent receives. What the tool-side design changes is the form of the knowledge: it is tool-centered and agent-neutral, so that one memory can serve many agents. This addresses the Generalization Dilemma that motivates ToolAtlas (Section 1), where agent-side knowledge is written in the producing agent's action format and cannot be reused by others. We therefore measure this factor through sharing, evaluating the agent-side variant both with the agent that built it and with a different agent (2.2 below). The other benefit of central storage is cost: one exploration is shared by all consumers, so the total offline cost is lower than building agent-side memories separately once five (ACE) or six (ExpeL, A-MEM) agents share the memory (Section 4.5 and Table 13).
(continued below)
Response to Reviewer Ycnz (3/6)
2.1 Graph Organization and Exploration Are Complementary
Tables R1 and R2 together form a 2 × 2 comparison over memory form and data source (Table 16 in the revised paper). The frontier corpus is the exploration data that ToolAtlas generates within its ≈148M construction budget, and the random corpus is generated with the same budget.
| Memory form \ Data source | Random (≈148M) | Frontier (ToolAtlas corpus) |
|---|---|---|
| ACE playbook | 56.41 / 71.20 | 57.88 / 73.79 |
| ToolAtlas graph | 56.46 / 71.41 | 60.94 / 80.36 |
Frontier data benefits the ToolAtlas graph far more than the ACE playbook (+4.48 / +8.95 vs. +1.47 / +2.59), and the graph adds little on random data (+0.05 / +0.21) but substantially on frontier data (+3.06 / +6.57), since it is built to hold the affordance and boundary knowledge that frontier probes elicit. The two components should therefore be credited jointly: neither alone accounts for ToolAtlas's gain.
2.2 Ablation within ToolAtlas
Experimental setup. All variants are derived from the same SwitchAct-built memory and use the same guidance generator, prompt, and guidance-length limit:
- Agent-side form (RawTrace) replaces each induced agent-neutral trace with the producing agent's own rollout (the shortest verifier-passing rollout, or the first rollout if all attempts fail); everything else is unchanged.
- Flat shared memory keeps the same knowledge but removes the graph: it retrieves the top-K trace, tool, and strategy records independently by similarity to the task. K = 8 matches the maximum number of navigation operations in Dynamic Memory Traversal (B = 8, Appendix B), so both schemes have the same read budget.
- w/o Dynamic Traversal is the existing ablation in Table 4, which replaces the adaptive navigator with static retrieval.
All variants are evaluated under SwitchAct → SwitchAct (construction agent → inference agent). To test sharing, the agent-side form and Full ToolAtlas are also evaluated under SwitchAct → ReAct, reusing the frozen memory without reconstruction (Appendix H.3).
Table R3. Ablation of Memory Form, Graph Organization, and Retrieval (Table 17 in the revised paper)
pass@1 / pass@4 (%). The best results within each setting are marked in bold.
| Memory construction agent | Inference agent | Method | Filesystem | GitHub | Notion | Playwright | PostgreSQL | Overall |
|---|---|---|---|---|---|---|---|---|
| – | SwitchAct | Vanilla† | 48.68 / 68.42 | 46.67 / 53.33 | 44.44 / 55.56 | 46.15 / 61.54 | 59.62 / 69.23 | 49.11 / 61.62 |
| SwitchAct | SwitchAct | Agent-side form (RawTrace) | 68.42 / 94.73 | 46.67 / 66.67 | 52.78 / 77.78 | 51.92 / 69.23 | 65.38 / 84.61 | 57.03 / 78.60 |
| SwitchAct | SwitchAct | Flat shared memory | 57.89 / 78.95 | 51.67 / 66.67 | 44.44 / 66.67 | 48.08 / 61.54 | 67.30 / 84.61 | 53.88 / 71.69 |
| SwitchAct | SwitchAct | w/o Dynamic Traversal† | 67.11 / 89.47 | 51.67 / 66.67 | 50.00 / 66.67 | 55.77 / 69.23 | 61.54 / 76.92 | 57.22 / 73.79 |
| SwitchAct | SwitchAct | Full ToolAtlas† | 69.73 / 94.73 | 56.67 / 73.33 | 51.39 / 72.22 | 59.62 / 76.92 | 67.30 / 84.61 | 60.94 / 80.36 |
| – | ReAct | Vanilla† | 51.32 / 63.16 | 48.33 / 53.33 | 43.06 / 61.11 | 32.69 / 53.85 | 48.08 / 69.23 | 44.70 / 60.14 |
| SwitchAct | ReAct | Agent-side form (RawTrace) | 60.53 / 78.95 | 50.00 / 73.33 | 37.50 / 61.11 | 23.08 / 38.46 | 48.08 / 84.62 | 43.84 / 67.29 |
| SwitchAct | ReAct | Full ToolAtlas† | 64.47 / 84.21 | 50.00 / 73.33 | 48.61 / 66.67 | 51.92 / 69.23 | 61.54 / 84.62 | 55.31 / 75.61 |
† Same runs as in the paper: Vanilla from Table 1 (SwitchAct) and Table 3 (ReAct), w/o Dynamic Traversal from Table 4, and Full ToolAtlas from Table 1 (SwitchAct → SwitchAct) and Table 3 (SwitchAct → ReAct).
(continued below)
Response to Reviewer Ycnz (1/6)
Dear Reviewer Ycnz,
We thank the reviewer for the constructive comments, and for recognizing the value of shifting memory to tool providers and the consistent improvements across both benchmarks. We address each point below. All new results have been added to Appendix H of the revised paper and are marked in red.
W1 & Q1: Controlling the Offline Exploration Budget
We agree that, in the submission, the agent-side baselines built their memories only from training-split trajectories, whereas ToolAtlas additionally performed seed and frontier exploration at a higher offline cost (Table 13). To test whether the gains come from more interaction experience rather than from the tool-side memory, we compare ExpeL, A-MEM, and ACE with ToolAtlas under both conditions the reviewer suggests:
ToolAtlas's gains are therefore not explained by more interaction experience or more compute alone: frontier-explored experience is more informative than randomly collected experience, and the tool-side memory makes better use of it. Full setups are in Appendix H.1–H.2 of the revised paper.
1.1 Comparison under Identical Exploration Data
Experimental setup. ExpeL, A-MEM, ACE, and ToolAtlas are built from the same ToolAtlas-generated tasks, rollouts, and verifier labels (training split only), each with its own memory construction and utilization mechanism. All memories are frozen and evaluated with SwitchAct and GPT-5.4 on the MCPMark same-environment split, with four runs per task (Appendix H.1).
Table R1. Performance under Identical Exploration Data (Table 14 in the revised paper)
pass@1 / pass@4 (%); Overall is the macro-average over the five services. Bold / italic: best / second-best. "+ ToolAtlas Trace": built from the same ToolAtlas-generated exploration data.
Analysis & Conclusion.
First, frontier exploration produces transferable experience. The ToolAtlas-generated data improves all three agent-side memories (+1.68, +1.69, and +1.53 overall pass@1 for ExpeL, A-MEM, and ACE), supporting the premise of Capability Exploration (Section 1): deliberately probing tool affordances and boundaries yields knowledge that passively logged trajectories do not cover.
Second, with identical data, the tool-side memory still makes better use of it. ToolAtlas outperforms the strongest baseline, ACE + ToolAtlas Trace, by 3.06 / 6.57 points, improves on it in every service under both metrics, and achieves the highest pass@1 on all five services. Since the data is identical, this gap comes from how ToolAtlas distills, organizes, and retrieves the experience, rather than from having more of it.
(continued below)