acceptodds
Under review as a conference paper at ICLR 2027

ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory

Abstract

Large language model (LLM) agents increasingly rely on external tools served by shared providers and accessed by heterogeneous downstream agents. Existing approaches improve tool use on the agent side through parameter updates, prompt refinement, or agent-side memory, making tool knowledge difficult to share and limited to behaviors observed in past tasks. We argue that reusable tool knowledge should instead be maintained by the tool provider. We introduce ToolAtlas, a graph-based framework that builds a persistent provider-side tool memory of tool capabilities, failure boundaries, and cross-tool compositions through execution-verified probing. At inference time, agents query the tool memory via adaptive graph traversal. Across two MCP-based benchmarks spanning eight services, ToolAtlas outperforms existing tool-side optimization and agent-side memory baselines by up to 21.61% in pass@1 and 18.61% in pass@4. The same tool memory also transfers across environment instances and agent frameworks without retraining or task-time exploration, yielding up to 24.16%/16.22% and 17.49%/14.27% relative gains in pass@1/pass@4, respectively. Ablation studies show that these gains arise from combining tool-centered memory organization with capability-guided execution probing. These results establish provider-side tool memory as an effective and reusable paradigm for tool servers. Our code is in: https://anonymous.4open.science/r/ToolAtlas-FE5E.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, tool use, provider-side memory, Model Context Protocol, function calling, memory mechanism

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YcnzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ToolAtlas, a provider-side shared memory framework tailored for the MCP tool ecosystem. By integrating a three-layer tool memory graph, execution-verified capability frontier exploration, and dynamic memory traversal, ToolAtlas enables the reuse of tool-use knowledge across heterogeneous agents, LLM backbones, and environment instances, achieving substantial performance gains across two MCP benchmarks.

Critical strengths
  1. ToolAtlas addresses a meaningful research problem by shifting memory from individual agents to tool providers, enabling shared and reusable tool-use knowledge across agents and reducing redundant learning.

  2. ToolAtlas achieves consistent performance improvements across two benchmarks, MCPMark and MCP-Universe, covering eight MCP services. These results provide empirical support for its central claim that tool-use knowledge can be reused across different agents and environment instances.

Critical weaknesses
  1. Insufficient control over offline exploration budgets. ToolAtlas generates additional probing tasks and executes more agent rollouts, raising the possibility that its performance gains primarily stem from increased tool interaction experience rather than the proposed tool-side memory graph. Without comparisons against ACE and A-MEM under matched token budgets or equivalent exploration data, it is difficult to determine the extent to which the improvements can be attributed to the proposed method itself.

  2. Insufficient isolation of the core contributions. Although ToolAtlas presents provider-side memory as a central contribution, it simultaneously introduces several design choices, including centralized tool-side storage, a three-layer memory graph, active capability exploration, and dynamic memory retrieval. While the paper provides component-level ablation studies, the current experiments do not fully disentangle the individual contributions of these factors, making it difficult to identify which mechanisms are primarily responsible for the observed performance improvements.

  3. Insufficient validation of the reliability of execution-based knowledge acquisition. Both the probing tasks and their corresponding Python verifiers are generated by large language models (LLMs). However, the paper does not provide independent evidence regarding the reliability of these verifiers. Consequently, it remains unclear whether incorrect verification outcomes could introduce inaccurate capability or boundary information into the persistent memory.

Questions for authors
  1. How would ToolAtlas compare with ACE and A-MEM under matched offline token budgets or using the same exploration data?

  2. Can the authors further isolate the contributions of provider-side memory, graph-based organization, and active exploration to the overall performance gains?

  3. How do the authors ensure the correctness of LLM-generated verifiers? Have they evaluated whether verification errors introduce inaccurate knowledge into the memory?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Ycnz (1/6)

Dear Reviewer Ycnz,

We thank the reviewer for the constructive comments, and for recognizing the value of shifting memory to tool providers and the consistent improvements across both benchmarks. We address each point below. All new results have been added to Appendix H of the revised paper and are marked in red.

W1 & Q1: Controlling the Offline Exploration Budget

We agree that, in the submission, the agent-side baselines built their memories only from training-split trajectories, whereas ToolAtlas additionally performed seed and frontier exploration at a higher offline cost (Table 13). To test whether the gains come from more interaction experience rather than from the tool-side memory, we compare ExpeL, A-MEM, and ACE with ToolAtlas under both conditions the reviewer suggests:

  • Same exploration data (1.1 below). When ExpeL, A-MEM, and ACE are built from exactly the same ToolAtlas-generated tasks, rollouts, and verifier labels, the data improves each of them by 1.53–1.69 overall pass@1 points, yet ToolAtlas still outperforms the strongest of them by 3.06 / 6.57 points (pass@1 / pass@4) and improves on it in every service under both metrics.
  • Same offline token budget (1.2 below). When all methods spend ToolAtlas's ≈148M offline tokens, the baselines gain at most 0.73 pass@1 points from the extra budget, and ToolAtlas still leads the strongest of them by 4.53 / 9.16 points.

ToolAtlas's gains are therefore not explained by more interaction experience or more compute alone: frontier-explored experience is more informative than randomly collected experience, and the tool-side memory makes better use of it. Full setups are in Appendix H.1–H.2 of the revised paper.

1.1 Comparison under Identical Exploration Data

Experimental setup. ExpeL, A-MEM, ACE, and ToolAtlas are built from the same ToolAtlas-generated tasks, rollouts, and verifier labels (training split only), each with its own memory construction and utilization mechanism. All memories are frozen and evaluated with SwitchAct and GPT-5.4 on the MCPMark same-environment split, with four runs per task (Appendix H.1).

Table R1. Performance under Identical Exploration Data (Table 14 in the revised paper)

pass@1 / pass@4 (%); Overall is the macro-average over the five services. Bold / italic: best / second-best. "+ ToolAtlas Trace": built from the same ToolAtlas-generated exploration data.

MethodFilesystemGitHubNotionPlaywrightPostgreSQLOverall
Vanilla48.68 / 68.4246.67 / 53.3344.44 / 55.5646.15 / 61.5459.62 / 69.2349.11 / 61.62
ExpeL48.68 / 78.9536.67 / 53.3344.44 / 66.6755.77 / 69.2355.77 / 76.9248.27 / 69.02
+ ToolAtlas Trace46.05 / 73.6838.33 / 53.3350.00 / 77.7857.69 / 69.2357.69 / 76.9249.95 / 70.19
A-MEM57.89 / 78.9546.67 / 60.0043.06 / 55.5642.31 / 53.8559.62 / 84.6149.91 / 66.59
+ ToolAtlas Trace65.79 / 84.2150.00 / 66.6740.28 / 50.0040.38 / 53.8561.54 / 84.6151.60 / 67.87
ACE59.21 / 84.2153.33 / 66.6750.00 / 66.6753.85 / 61.5465.38 / 84.6156.35 / 72.74
+ ToolAtlas Trace67.11 / 89.4755.00 / 66.6750.00 / 66.6757.69 / 69.2359.62 / 76.9257.88 / 73.79
Full ToolAtlas69.73 / 94.7356.67 / 73.3351.39 / 72.2259.62 / 76.9267.30 / 84.6160.94 / 80.36

Analysis & Conclusion.

First, frontier exploration produces transferable experience. The ToolAtlas-generated data improves all three agent-side memories (+1.68, +1.69, and +1.53 overall pass@1 for ExpeL, A-MEM, and ACE), supporting the premise of Capability Exploration (Section 1): deliberately probing tool affordances and boundaries yields knowledge that passively logged trajectories do not cover.

Second, with identical data, the tool-side memory still makes better use of it. ToolAtlas outperforms the strongest baseline, ACE + ToolAtlas Trace, by 3.06 / 6.57 points, improves on it in every service under both metrics, and achieves the highest pass@1 on all five services. Since the data is identical, this gap comes from how ToolAtlas distills, organizes, and retrieves the experience, rather than from having more of it.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer Ycnz (2/6)

1.2 Comparison under Matched Offline Token Budgets

Experimental setup. We match ToolAtlas's offline token budget per service, counting all input and output LLM tokens of training and exploration. ExpeL, A-MEM, and ACE spend the extra budget on random tasks generated with ToolAtlas's seed-generation procedure, followed by rollouts and their own memory updates; ToolAtlas-Random spends it on such random tasks instead of frontier exploration. Evaluation follows 1.1 (Appendix H.2).

Table R2. Performance under Matched Offline Token Budgets (Table 15 in the revised paper)

pass@1 / pass@4 (%). The offline budget is the actual consumption summed across services; budgets are matched per service.

MethodOffline budget (M)FilesystemGitHubNotionPlaywrightPostgreSQLOverall
Vanilla048.68 / 68.4246.67 / 53.3344.44 / 55.5646.15 / 61.5459.62 / 69.2349.11 / 61.62
ExpeL + Random≈14848.68 / 78.9543.33 / 66.6744.44 / 66.6750.00 / 61.5457.69 / 76.9248.83 / 70.15
ACE + Random≈14859.21 / 84.2155.00 / 66.6748.61 / 66.6755.77 / 61.5463.46 / 76.9256.41 / 71.20
A-MEM + Random≈14861.84 / 84.2145.00 / 60.0044.44 / 55.5650.00 / 61.5451.92 / 76.9250.64 / 67.65
ToolAtlas-Random≈14868.42 / 84.2145.00 / 60.0045.83 / 66.6757.69 / 69.2365.38 / 76.9256.46 / 71.41
Full ToolAtlas≈14869.73 / 94.7356.67 / 73.3351.39 / 72.2259.62 / 76.9267.30 / 84.6160.94 / 80.36

Analysis & Conclusion.

First, more budget alone does not close the gap. At 4.7–5.9× their original construction cost on MCPMark (Table 13), ExpeL, A-MEM, and ACE change their overall pass@1 by only +0.56, +0.73, and +0.06 points, and ACE's pass@4 even drops by 1.54 points, while Full ToolAtlas is the best on every service under both metrics and leads ACE + Random by 4.53 / 9.16 points. This reflects the Capability Blindness that motivates ToolAtlas (Section 1): random tasks cover representative uses of each tool, so extra budget mostly revisits behaviors that are already covered.

Second, the value of the budget lies in where it is spent. Compared with the budget-matched random corpus, the frontier corpus (Table R1) improves overall pass@1 / pass@4 by 1.12 / 0.04 points for ExpeL, 0.96 / 0.22 for A-MEM, 1.47 / 2.59 for ACE, and 4.48 / 8.95 for ToolAtlas. The gap is largest on GitHub (11.67 / 13.33 for ToolAtlas), whose 94 tools make random tasks especially sparse; frontier exploration instead directs the budget toward under-explored affordances and boundaries.

W2 & Q2: Isolating the Core Contributions

ToolAtlas indeed combines several design choices. We isolate each of them with a controlled comparison that holds the others fixed (Table R4 in 2.3 below summarizes them). Each design choice contributes on its own: removing graph organization, replacing frontier exploration with random tasks at the same budget, or replacing dynamic traversal with static retrieval reduces overall pass@1 / pass@4 by 7.06 / 8.67, 4.48 / 8.95, and 3.72 / 6.57 points, respectively. The tool-side, agent-neutral form costs little for the agent that built the memory (−3.91 / −1.76) but becomes decisive once the memory is shared (−11.47 / −8.32). Graph organization and active exploration are moreover complementary rather than independent (2.1 below).

On centralized tool-side storage. Where the memory is stored does not by itself change what a single agent receives. What the tool-side design changes is the form of the knowledge: it is tool-centered and agent-neutral, so that one memory can serve many agents. This addresses the Generalization Dilemma that motivates ToolAtlas (Section 1), where agent-side knowledge is written in the producing agent's action format and cannot be reused by others. We therefore measure this factor through sharing, evaluating the agent-side variant both with the agent that built it and with a different agent (2.2 below). The other benefit of central storage is cost: one exploration is shared by all consumers, so the total offline cost is lower than building agent-side memories separately once five (ACE) or six (ExpeL, A-MEM) agents share the memory (Section 4.5 and Table 13).

(continued below)

AuthorsOpenReview ↗

Response to Reviewer Ycnz (3/6)

2.1 Graph Organization and Exploration Are Complementary

Tables R1 and R2 together form a 2 × 2 comparison over memory form and data source (Table 16 in the revised paper). The frontier corpus is the exploration data that ToolAtlas generates within its ≈148M construction budget, and the random corpus is generated with the same budget.

Memory form \ Data sourceRandom (≈148M)Frontier (ToolAtlas corpus)
ACE playbook56.41 / 71.2057.88 / 73.79
ToolAtlas graph56.46 / 71.4160.94 / 80.36

Frontier data benefits the ToolAtlas graph far more than the ACE playbook (+4.48 / +8.95 vs. +1.47 / +2.59), and the graph adds little on random data (+0.05 / +0.21) but substantially on frontier data (+3.06 / +6.57), since it is built to hold the affordance and boundary knowledge that frontier probes elicit. The two components should therefore be credited jointly: neither alone accounts for ToolAtlas's gain.

2.2 Ablation within ToolAtlas

Experimental setup. All variants are derived from the same SwitchAct-built memory and use the same guidance generator, prompt, and guidance-length limit:

  • Agent-side form (RawTrace) replaces each induced agent-neutral trace with the producing agent's own rollout (the shortest verifier-passing rollout, or the first rollout if all attempts fail); everything else is unchanged.
  • Flat shared memory keeps the same knowledge but removes the graph: it retrieves the top-K trace, tool, and strategy records independently by similarity to the task. K = 8 matches the maximum number of navigation operations in Dynamic Memory Traversal (B = 8, Appendix B), so both schemes have the same read budget.
  • w/o Dynamic Traversal is the existing ablation in Table 4, which replaces the adaptive navigator with static retrieval.

All variants are evaluated under SwitchAct → SwitchAct (construction agent → inference agent). To test sharing, the agent-side form and Full ToolAtlas are also evaluated under SwitchAct → ReAct, reusing the frozen memory without reconstruction (Appendix H.3).

Table R3. Ablation of Memory Form, Graph Organization, and Retrieval (Table 17 in the revised paper)

pass@1 / pass@4 (%). The best results within each setting are marked in bold.

Memory construction agentInference agentMethodFilesystemGitHubNotionPlaywrightPostgreSQLOverall
–SwitchActVanilla†48.68 / 68.4246.67 / 53.3344.44 / 55.5646.15 / 61.5459.62 / 69.2349.11 / 61.62
SwitchActSwitchActAgent-side form (RawTrace)68.42 / 94.7346.67 / 66.6752.78 / 77.7851.92 / 69.2365.38 / 84.6157.03 / 78.60
SwitchActSwitchActFlat shared memory57.89 / 78.9551.67 / 66.6744.44 / 66.6748.08 / 61.5467.30 / 84.6153.88 / 71.69
SwitchActSwitchActw/o Dynamic Traversal†67.11 / 89.4751.67 / 66.6750.00 / 66.6755.77 / 69.2361.54 / 76.9257.22 / 73.79
SwitchActSwitchActFull ToolAtlas†69.73 / 94.7356.67 / 73.3351.39 / 72.2259.62 / 76.9267.30 / 84.6160.94 / 80.36
–ReActVanilla†51.32 / 63.1648.33 / 53.3343.06 / 61.1132.69 / 53.8548.08 / 69.2344.70 / 60.14
SwitchActReActAgent-side form (RawTrace)60.53 / 78.9550.00 / 73.3337.50 / 61.1123.08 / 38.4648.08 / 84.6243.84 / 67.29
SwitchActReActFull ToolAtlas†64.47 / 84.2150.00 / 73.3348.61 / 66.6751.92 / 69.2361.54 / 84.6255.31 / 75.61

† Same runs as in the paper: Vanilla from Table 1 (SwitchAct) and Table 3 (ReAct), w/o Dynamic Traversal from Table 4, and Full ToolAtlas from Table 1 (SwitchAct → SwitchAct) and Table 3 (SwitchAct → ReAct).

(continued below)

Reviewer_GEZNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a provider-side memory framework for improving tool use in LLM agents, ToolAtlas. Instead of maintaining tool-use experience separately for each agent, ToolAtlas builds a shared memory graph containing tool capabilities, failure boundaries, and cross-tool strategies. The framework first constructs memory from verified execution trajectories, then expands it through active capability exploration. At inference time, relevant knowledge is retrieved through dynamic graph traversal and provided to downstream agents.

Critical strengths
  1. The research direction is interesting. When multiple agents use the same MCP tools, independently learning their capabilities and failure modes can be inefficient. Moving reusable tool knowledge closer to the provider is a reasonable design choice, especially for shared tool services.

  2. Paper includes the comparisons with both tool-side optimization and agent-side memory methods, along with several ablation studies. Particularly appreciate the cross-agent transfer experiments, where memory constructed using SwitchAct is reused by ReAct and CodeAct without reconstruction. The consistent improvements across different agent interfaces support the claim that tool-centered knowledge can transfer beyond the agent that originally acquired it. The ablation results also provide evidence for the importance of capability exploration and structured memory organization.

Critical weaknesses
  1. The main claim of paper is that tool memory should be maintained by the provider rather than individual agents. ToolAtlas differs from the agent-side baselines in several aspects beyond memory ownership, including its three-layer graph structure, active capability exploration, and dynamic retrieval mechanism. The ablation study do not clearly isolate the benefit of provider-side memory itself.

  2. In practice, tool schemas, API behavior, and access permissions may change. Such changes could invalidate previously learned capabilities or failure boundaries. The paper does not sufficiently investigate how ToolAtlas handles these changes or how much additional exploration is needed to maintain accurate memory.

Questions for authors
  1. Could the authors compare ToolAtlas against a shared agent-side memory implementation using the same graph structure, exploration budget, and retrieval mechanism?

  2. How does ToolAtlas handle changes in tool behavior or API specifications after the memory has been constructed? Can the memory be updated incrementally?

  3. How sensitive is performance to the exploration budget, and what level of performance can be achieved with a construction budget comparable to the agent-side methods?

12 comments
AuthorsOpenReview ↗

Response to Reviewer GEZN (1/7)

Dear Reviewer GEZN,

We thank the reviewer for the constructive comments, and for finding the provider-side direction reasonable and appreciating our cross-agent transfer experiments and ablations. We address each point below. All new results have been added to Appendix H of the revised paper and are marked in red.

W1 & Q1: Isolating the Benefit of Provider-Side Memory

We understand the suggested comparison as an agent-side memory that is given the same exploration budget, graph structure, and retrieval mechanism as ToolAtlas, so that only the provider-side design differs. We address it in three steps:

  • (a) Aligning agent-side memories with ToolAtlas (1.1 below). With identical exploration data, ToolAtlas outperforms the strongest agent-side memory by 3.06 / 6.57 points (pass@1 / pass@4) and improves on it in every service under both metrics; with a matched ≈148M-token budget, the baselines gain at most 0.73 pass@1 points, and ToolAtlas leads by 4.53 / 9.16.
  • (b) Systematic ablation within ToolAtlas (1.2 below). With the graph, exploration corpus, and retrieval held fixed, an agent-side variant that stores the producing agent's own rollouts performs close to ToolAtlas for that agent (−3.91 / −1.76 points), but loses its entire pass@1 gain over Vanilla once shared with another agent (+7.92 → −0.86). Removing graph organization or dynamic traversal also reduces overall pass@1 / pass@4 (by 7.06 / 8.67 and 3.72 / 6.57).
  • (c) Where the provider-side design pays off (1.3 below). A memory built once transfers across agent frameworks, environment instances, and backbones, while agent-side memories often fall below Vanilla after transfer, and its per-agent cost falls below theirs once five (ACE) or six (ExpeL, A-MEM) agents share it.

What provider-side memory changes. Our motivation (Section 1) identifies two limitations of agent-side tool memory when many agents share one tool server. Under the Generalization Dilemma, agent-side knowledge is written in the producing agent's workflow and action format, so other agents cannot reuse it; under Capability Blindness, passively logged experience only covers tool behaviors triggered by past tasks. Provider-side memory addresses the first: knowledge is stored in a tool-centered, agent-neutral form, built once by the provider, and shared by every agent that calls the tool. Frontier exploration addresses the second. The agent-neutral form is produced by Memory Induction, which condenses each task's rollouts into a sequence of (tool, rationale) pairs and removes the producing agent's action syntax, retry patterns, and reasoning style (Section 3.1).

(continued below)

AuthorsOpenReview ↗

Response to Reviewer GEZN (2/7)

1.1 Aligning Agent-Side Memories with ToolAtlas in Exploration Data and Budget

Experimental setup. We align ExpeL, A-MEM, and ACE with ToolAtlas along two dimensions, with each method keeping its own memory construction and utilization mechanism:

  • Identical exploration data. All methods are built from the same ToolAtlas-generated tasks, rollouts, and verifier labels (training split only).
  • Matched offline budget. We match ToolAtlas's offline token budget per service, counting all input and output LLM tokens of training and exploration. The agent-side memories spend the extra budget on random tasks generated with ToolAtlas's seed-generation procedure, followed by rollouts and their own memory updates; ToolAtlas-Random spends it on such random tasks instead of frontier exploration.

All memories are frozen and evaluated with SwitchAct and GPT-5.4 on the MCPMark same-environment split, with four runs per task (Appendix H.1–H.2).

Table R1. Performance under Identical Exploration Data (Table 14 in the revised paper)

pass@1 / pass@4 (%); Overall is the macro-average over the five services. Bold / italic: best / second-best. "+ ToolAtlas Trace": built from the same ToolAtlas-generated exploration data.

MethodFilesystemGitHubNotionPlaywrightPostgreSQLOverall
Vanilla48.68 / 68.4246.67 / 53.3344.44 / 55.5646.15 / 61.5459.62 / 69.2349.11 / 61.62
ExpeL48.68 / 78.9536.67 / 53.3344.44 / 66.6755.77 / 69.2355.77 / 76.9248.27 / 69.02
+ ToolAtlas Trace46.05 / 73.6838.33 / 53.3350.00 / 77.7857.69 / 69.2357.69 / 76.9249.95 / 70.19
A-MEM57.89 / 78.9546.67 / 60.0043.06 / 55.5642.31 / 53.8559.62 / 84.6149.91 / 66.59
+ ToolAtlas Trace65.79 / 84.2150.00 / 66.6740.28 / 50.0040.38 / 53.8561.54 / 84.6151.60 / 67.87
ACE59.21 / 84.2153.33 / 66.6750.00 / 66.6753.85 / 61.5465.38 / 84.6156.35 / 72.74
+ ToolAtlas Trace67.11 / 89.4755.00 / 66.6750.00 / 66.6757.69 / 69.2359.62 / 76.9257.88 / 73.79
Full ToolAtlas69.73 / 94.7356.67 / 73.3351.39 / 72.2259.62 / 76.9267.30 / 84.6160.94 / 80.36

Table R2. Performance under Matched Offline Token Budgets (Table 15 in the revised paper)

pass@1 / pass@4 (%). The offline budget is the actual consumption summed across services; budgets are matched per service.

MethodOffline budget (M)FilesystemGitHubNotionPlaywrightPostgreSQLOverall
Vanilla048.68 / 68.4246.67 / 53.3344.44 / 55.5646.15 / 61.5459.62 / 69.2349.11 / 61.62
ExpeL + Random≈14848.68 / 78.9543.33 / 66.6744.44 / 66.6750.00 / 61.5457.69 / 76.9248.83 / 70.15
ACE + Random≈14859.21 / 84.2155.00 / 66.6748.61 / 66.6755.77 / 61.5463.46 / 76.9256.41 / 71.20
A-MEM + Random≈14861.84 / 84.2145.00 / 60.0044.44 / 55.5650.00 / 61.5451.92 / 76.9250.64 / 67.65
ToolAtlas-Random≈14868.42 / 84.2145.00 / 60.0045.83 / 66.6757.69 / 69.2365.38 / 76.9256.46 / 71.41
Full ToolAtlas≈14869.73 / 94.7356.67 / 73.3351.39 / 72.2259.62 / 76.9267.30 / 84.6160.94 / 80.36

(continued below)

AuthorsOpenReview ↗

Response to Reviewer GEZN (3/7)

Analysis & Conclusion (Tables R1 and R2).

First, frontier exploration produces transferable experience, but the same data does not close the gap. The ToolAtlas-generated data improves all three agent-side memories (+1.68, +1.69, and +1.53 overall pass@1 for ExpeL, A-MEM, and ACE), supporting the premise of Capability Exploration (Section 1). Yet ToolAtlas still outperforms the strongest of them, ACE + ToolAtlas Trace, by 3.06 / 6.57 points and improves on it in every service under both metrics; since the data is identical, this gap comes from how ToolAtlas distills, organizes, and retrieves the experience.

Second, more budget alone does not close the gap either. At 4.7–5.9× their original construction cost (Table 13), the baselines change their overall pass@1 by only +0.56 (ExpeL), +0.73 (A-MEM), and +0.06 (ACE), and ACE's pass@4 even drops by 1.54 points, while Full ToolAtlas is the best on every service under both metrics and leads ACE + Random by 4.53 / 9.16 points. Random tasks cover representative uses of each tool, so extra budget mostly revisits behaviors that are already covered, which is the Capability Blindness of Section 1.

Third, the value of the budget lies in where it is spent. Compared with the budget-matched random corpus, the frontier corpus improves overall pass@1 / pass@4 by 1.12 / 0.04 points for ExpeL, 0.96 / 0.22 for A-MEM, 1.47 / 2.59 for ACE, and 4.48 / 8.95 for ToolAtlas. The gap is largest on GitHub (11.67 / 13.33 for ToolAtlas), whose 94 tools make random tasks especially sparse; frontier exploration instead directs the budget toward under-explored affordances and boundaries.

Fourth, memory organization and frontier exploration are complementary. Tables R1 and R2 form a 2 × 2 comparison (Table 16 in the revised paper), where the frontier corpus is the exploration data that ToolAtlas generates within its ≈148M budget:

Memory form \ Data sourceRandom (≈148M)Frontier (ToolAtlas corpus)
ACE playbook56.41 / 71.2057.88 / 73.79
ToolAtlas graph56.46 / 71.4160.94 / 80.36

Frontier data benefits the ToolAtlas graph far more than the ACE playbook (+4.48 / +8.95 vs. +1.47 / +2.59), and the graph adds little on random data (+0.05 / +0.21) but substantially on frontier data (+3.06 / +6.57), since it is built to hold the affordance and boundary knowledge that frontier probes elicit. ToolAtlas's advantage therefore comes from coupling a tool-centered memory with capability-guided exploration, not from more experience or compute.

(continued below)

Reviewer_7CEJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose ToolAtlas: How tool-use knowledge can be accumulated once and reused by heterogeneous LLM agents accessing shared tool services. It proposes maintaining this knowledge on the provider side in a memory graph covering execution traces, tool capabilities, failure boundaries, and higher-level strategies. The memory is initialized from executed tasks, expanded through targeted capability exploration, and queried through adaptive graph traversal to produce task-conditioned guidance. The paper evaluates the approach on MCPMark and selected MCP-Universe domains, including comparisons with tool-description optimization and agent-memory methods. The authors report improvements in task success, transfer across environment instances, agent frameworks, and favorable inference-token usage. Overall, the paper is easy to follow for me as a reviewer and I learned something from it.

Critical strengths
  1. First, the paper addresses a consequential problem in shared tool infrastructure. As we know, repeatedly rediscovering or revisiting tool behavior is an avoidable source of cost and failure when multiple agents use the same services in the real-world environment. Organizing reusable knowledge around tools is therefore a worthwhile research direction.
  2. Second, the evaluation includes transfer settings that directly test the intended value of reusable memory. In this case, the cross-environment and cross-framework experiments are more informative
Critical weaknesses
  1. First, the experiments do not sufficiently isolate the contribution relative to a simpler, comparably constructed shared memory. When we have a clear isolation of contribution from various components and settings, we can better frame the work.
  2. Second, task-outcome verification does not fully establish the validity of the generalized capability and boundary entries. I admit that the verification can show the effectiveness of the methods. But, the generalization capability needs more comprehensive testing.
Questions for authors
  1. First, provide more contribution isolation and comparison fairness — clarification and targeted evidence. This is mentioned in the first point in the critical weakness section.
  2. Second, authors can provide more verification of induced knowledge — clarification and targeted evidence. In this way, we can better frame the setting.
8 comments
AuthorsOpenReview ↗

Response to Reviewer 7CEJ (1/5)

Dear Reviewer 7CEJ,

We thank the reviewer for the constructive comments and for recognizing the importance of the problem and the value of our transfer experiments. We address each point below. All new results have been added to Appendix H of the revised paper and are marked in red.

W1 & Q1: Contribution Isolation and Comparison Fairness

We agree that a fair comparison should control for exploration data and offline budget. In the submission, the agent-side baselines built their memories only from training-split trajectories, whereas ToolAtlas additionally performed seed and frontier exploration at a higher offline cost (Table 13). We therefore add three controlled comparisons on MCPMark (SwitchAct, GPT-5.4, same-environment split):

  • RQ1: Given identical exploration data, does ToolAtlas still outperform agent-side memories? Yes. When ExpeL, A-MEM, and ACE are built from exactly the same ToolAtlas-generated tasks, rollouts, and verifier labels, ToolAtlas outperforms the strongest of them by 3.06 / 6.57 points (pass@1 / pass@4) and improves on it in every service under both metrics.
  • RQ2: Does ToolAtlas retain its advantage under the same offline budget? Yes. With all methods at ≈148M offline tokens, the baselines gain at most 0.73 pass@1 points from the extra budget, and ToolAtlas leads the strongest of them by 4.53 / 9.16 points.
  • RQ3: Does a simpler, comparably constructed shared memory match ToolAtlas? No. A flat shared memory with the same knowledge but no graph organization drops by 7.06 / 8.67 points, the largest drop among the variants; replacing agent-neutral traces with raw rollouts mainly hurts cross-agent transfer (−11.47 / −8.32).

ToolAtlas's gains are therefore not explained by additional data or compute alone; they come from how the explored experience is distilled, organized, and retrieved. Full setups are in Appendix H.1–H.3 of the revised paper.

1.1 RQ1: Comparing Memory Methods with Identical Exploration Data

Experimental setup. ExpeL, A-MEM, ACE, and ToolAtlas are built from the same ToolAtlas-generated tasks, rollouts, and verifier labels (training split only), each with its own memory construction and utilization mechanism. All memories are frozen and evaluated with SwitchAct and GPT-5.4 on the MCPMark same-environment split, with four runs per task (Appendix H.1).

Table R1. Performance under Identical Exploration Data (Table 14 in the revised paper)

pass@1 / pass@4 (%); Overall is the macro-average over the five services. Bold / italic: best / second-best. "+ ToolAtlas Trace": built from the same ToolAtlas-generated exploration data.

MethodFilesystemGitHubNotionPlaywrightPostgreSQLOverall
Vanilla48.68 / 68.4246.67 / 53.3344.44 / 55.5646.15 / 61.5459.62 / 69.2349.11 / 61.62
ExpeL48.68 / 78.9536.67 / 53.3344.44 / 66.6755.77 / 69.2355.77 / 76.9248.27 / 69.02
+ ToolAtlas Trace46.05 / 73.6838.33 / 53.3350.00 / 77.7857.69 / 69.2357.69 / 76.9249.95 / 70.19
A-MEM57.89 / 78.9546.67 / 60.0043.06 / 55.5642.31 / 53.8559.62 / 84.6149.91 / 66.59
+ ToolAtlas Trace65.79 / 84.2150.00 / 66.6740.28 / 50.0040.38 / 53.8561.54 / 84.6151.60 / 67.87
ACE59.21 / 84.2153.33 / 66.6750.00 / 66.6753.85 / 61.5465.38 / 84.6156.35 / 72.74
+ ToolAtlas Trace67.11 / 89.4755.00 / 66.6750.00 / 66.6757.69 / 69.2359.62 / 76.9257.88 / 73.79
Full ToolAtlas69.73 / 94.7356.67 / 73.3351.39 / 72.2259.62 / 76.9267.30 / 84.6160.94 / 80.36

Analysis & Conclusion.

First, frontier exploration produces transferable experience. The ToolAtlas-generated data improves all three agent-side memories (+1.68, +1.69, and +1.53 overall pass@1 for ExpeL, A-MEM, and ACE), supporting the premise of Capability Exploration (Section 1): deliberately probing tool affordances and boundaries yields knowledge that passively logged trajectories do not cover.

Second, with identical data, the tool-centered memory still makes better use of it. ToolAtlas outperforms the strongest baseline, ACE + ToolAtlas Trace, by 3.06 / 6.57 points, improves on it in every service under both metrics, and achieves the highest pass@1 on all five services. Since the data is identical, this gap comes from how ToolAtlas distills, organizes, and retrieves the experience.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer 7CEJ (2/5)

1.2 RQ2: Comparing Methods under Matched Offline Token Budgets

Experimental setup. We match ToolAtlas's offline token budget per service, counting all input and output LLM tokens of training and exploration. ExpeL, A-MEM, and ACE spend the extra budget on random tasks generated with ToolAtlas's seed-generation procedure, followed by rollouts and their own memory updates; ToolAtlas-Random spends it on such random tasks instead of frontier exploration. Evaluation follows RQ1 (Appendix H.2).

Table R2. Performance under Matched Offline Token Budgets (Table 15 in the revised paper)

pass@1 / pass@4 (%). The offline budget is the actual consumption summed across services; budgets are matched per service.

MethodOffline budget (M)FilesystemGitHubNotionPlaywrightPostgreSQLOverall
Vanilla048.68 / 68.4246.67 / 53.3344.44 / 55.5646.15 / 61.5459.62 / 69.2349.11 / 61.62
ExpeL + Random≈14848.68 / 78.9543.33 / 66.6744.44 / 66.6750.00 / 61.5457.69 / 76.9248.83 / 70.15
ACE + Random≈14859.21 / 84.2155.00 / 66.6748.61 / 66.6755.77 / 61.5463.46 / 76.9256.41 / 71.20
A-MEM + Random≈14861.84 / 84.2145.00 / 60.0044.44 / 55.5650.00 / 61.5451.92 / 76.9250.64 / 67.65
ToolAtlas-Random≈14868.42 / 84.2145.00 / 60.0045.83 / 66.6757.69 / 69.2365.38 / 76.9256.46 / 71.41
Full ToolAtlas≈14869.73 / 94.7356.67 / 73.3351.39 / 72.2259.62 / 76.9267.30 / 84.6160.94 / 80.36

Analysis & Conclusion.

First, more budget alone does not close the gap. At 4.7–5.9× their original construction cost (Table 13), the baselines change their overall pass@1 by only +0.56 (ExpeL), +0.73 (A-MEM), and +0.06 (ACE), and ACE's pass@4 even drops by 1.54 points, while Full ToolAtlas is the best on every service under both metrics and leads ACE + Random by 4.53 / 9.16 points. This reflects the Capability Blindness that motivates ToolAtlas (Section 1): random tasks cover representative uses of each tool, so extra budget mostly revisits behaviors that are already covered.

Second, the value of the budget lies in where it is spent. Compared with the budget-matched random corpus, the frontier corpus (Table R1) improves overall pass@1 / pass@4 by 1.12 / 0.04 points for ExpeL, 0.96 / 0.22 for A-MEM, 1.47 / 2.59 for ACE, and 4.48 / 8.95 for ToolAtlas. The gap is largest on GitHub (11.67 / 13.33 for ToolAtlas), whose 94 tools make random tasks especially sparse; frontier exploration instead directs the budget toward under-explored affordances and boundaries.

Third, memory organization and frontier exploration are complementary. Tables R1 and R2 form a 2 × 2 comparison (Table 16 in the revised paper), where the frontier corpus is the exploration data that ToolAtlas generates within its ≈148M budget:

Memory form \ Data sourceRandom (≈148M)Frontier (ToolAtlas corpus)
ACE playbook56.41 / 71.2057.88 / 73.79
ToolAtlas graph56.46 / 71.4160.94 / 80.36

Frontier data benefits the ToolAtlas graph far more than the ACE playbook (+4.48 / +8.95 vs. +1.47 / +2.59), and the graph adds little on random data (+0.05 / +0.21) but substantially on frontier data (+3.06 / +6.57), since it is built to hold the affordance and boundary knowledge that frontier probes elicit. The gains thus come from coupling the two, not from either alone or from more compute.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer 7CEJ (3/5)

1.3 RQ3: Evaluating Component Contributions to Performance and Transfer

Experimental setup. All variants are derived from the same SwitchAct-built memory and use the same guidance generator, prompt, and guidance-length limit:

  • RawTrace replaces each induced agent-neutral trace with the producing agent's own rollout (the shortest verifier-passing rollout, or the first rollout if all attempts fail); everything else is unchanged.
  • Flat shared memory keeps the same knowledge but removes the graph: it retrieves the top-K trace, tool, and strategy records independently by similarity to the task. K = 8 matches the maximum number of navigation operations in Dynamic Memory Traversal (B = 8, Appendix B), so both schemes have the same read budget.
  • w/o Dynamic Traversal is the existing ablation in Table 4, which replaces the adaptive navigator with static retrieval.

All variants are evaluated under SwitchAct → SwitchAct (construction agent → inference agent); RawTrace and Full ToolAtlas are also evaluated under SwitchAct → ReAct, reusing the frozen memory without reconstruction (Appendix H.3).

Table R3. Component Contributions to Performance and Cross-Agent Transfer (Table 17 in the revised paper)

pass@1 / pass@4 (%). The best results within each setting are marked in bold.

Memory construction agentInference agentMethodFilesystemGitHubNotionPlaywrightPostgreSQLOverall
SwitchActSwitchActRawTrace68.42 / 94.7346.67 / 66.6752.78 / 77.7851.92 / 69.2365.38 / 84.6157.03 / 78.60
SwitchActSwitchActFlat shared memory57.89 / 78.9551.67 / 66.6744.44 / 66.6748.08 / 61.5467.30 / 84.6153.88 / 71.69
SwitchActSwitchActw/o Dynamic Traversal†67.11 / 89.4751.67 / 66.6750.00 / 66.6755.77 / 69.2361.54 / 76.9257.22 / 73.79
SwitchActSwitchActFull ToolAtlas†69.73 / 94.7356.67 / 73.3351.39 / 72.2259.62 / 76.9267.30 / 84.6160.94 / 80.36
SwitchActReActRawTrace60.53 / 78.9550.00 / 73.3337.50 / 61.1123.08 / 38.4648.08 / 84.6243.84 / 67.29
SwitchActReActFull ToolAtlas†64.47 / 84.2150.00 / 73.3348.61 / 66.6751.92 / 69.2361.54 / 84.6255.31 / 75.61

† Same runs as in the paper: w/o Dynamic Traversal from Table 4; Full ToolAtlas from Table 1 (SwitchAct → SwitchAct) and Table 3 (SwitchAct → ReAct).

Analysis & Conclusion.

First, graph organization is the most important memory component. With the same knowledge, Flat shared memory drops by 7.06 / 8.67 points, the largest drop in Table R3, with losses on four of the five services. A simpler shared memory therefore does not match ToolAtlas: the graph links let the guidance combine task-level procedures with the tool-level affordances and boundaries they depend on, which independent similarity retrieval does not capture.

Second, the agent-neutral form is what makes the memory reusable across agents. Raw rollouts cost only 3.91 / 1.76 points when consumed by the same agent, but 11.47 / 8.32 points under SwitchAct → ReAct. Relative to the respective Vanilla agent (Vanilla ReAct: 44.70 pass@1, Table 3), Full ToolAtlas keeps most of its pass@1 gain after transfer (+11.83 → +10.61), whereas RawTrace's gain disappears (+7.92 → −0.86), and the transferred RawTrace (43.84 pass@1) behaves like the transferred agent-side memories in Table 3 (ExpeL 43.04, ACE 44.75). Storing agent-specific traces thus reproduces the Generalization Dilemma that motivates provider-side memory.

Third, adaptive traversal further improves retrieval. Static retrieval reduces overall pass@1 / pass@4 by 3.72 / 6.57 points, with drops on every service under both metrics.

(continued below)

Comments on the paper

2 comments
AuthorsOpenReview ↗

A Humble Response to All Reviewers

We would like to express our sincere gratitude to all reviewers for their thoughtful, constructive, and insightful comments! We are encouraged that the reviewers find that ToolAtlas addresses a consequential and meaningful problem in shared tool infrastructure (Reviewers 7CEJ and Ycnz), and that moving reusable tool knowledge closer to the provider is a reasonable design choice for shared tool services (Reviewer GEZN). We are also glad that the reviewers find that the transfer experiments directly test the intended value of reusable memory (Reviewer 7CEJ); that the cross-agent transfer results support that tool-centered knowledge can transfer beyond the agent that originally acquired it, with the ablations supporting the importance of capability exploration and structured memory organization (Reviewer GEZN); and that ToolAtlas achieves consistent improvements across two benchmarks covering eight MCP services (Reviewer Ycnz). Your feedback has been invaluable in improving the clarity and rigor of our work.

In response to your suggestions, we have conducted additional experiments and carefully revised the manuscript. The new analyses are placed in Appendix H, and all revisions are highlighted in red:

  • Identical exploration data (Reviewers 7CEJ, GEZN, and Ycnz; Appendix H.1). We built ExpeL, A-MEM, and ACE from exactly the same ToolAtlas-generated tasks, rollouts, and verifier labels.
  • Matched offline budgets (Reviewers 7CEJ, GEZN, and Ycnz; Appendix H.2). We raised the agent-side memories to the ≈148M-token offline budget of ToolAtlas with additional random tasks, compared ToolAtlas with ToolAtlas-Random under the same budget, and combined the two comparisons into a 2 × 2 analysis of memory form and data source.
  • Memory form, graph organization, and retrieval (Reviewers 7CEJ, GEZN, and Ycnz; Appendix H.3). We compared ToolAtlas with an agent-side form that stores the producing agent's own rollouts and with a flat shared memory without graph links, alongside the existing static-retrieval ablation, and evaluated the agent-side form after transfer to ReAct.
  • Reliability of verifiers and induced knowledge (Reviewers 7CEJ and Ycnz; Appendix H.4). We audited the verifier judgments and the induced capability and boundary entries with an auditor from a different model family than the memory constructor, and validated the audit against human annotations.
  • Tool changes and incremental updates (Reviewer GEZN; Appendix H.5). We changed three Filesystem tools, covering a schema change, a behavior change, and a permission change, and evaluated a localized update that removes the affected memory items and re-explores only the changed tools.

In addition, the individual responses summarize the contribution of each design choice (Reviewer Ycnz), as well as the evidence for the provider-side design across agents, environments, and backbones and the sensitivity to the exploration budget (Reviewer GEZN). In the follow-up discussion, we further compare ToolAtlas with a shared agent-side memory that uses the same representation, retrieval, and budget, which remains 4.31 / 5.46 points (pass@1 / pass@4) below ToolAtlas when shared with ReAct (Reviewer GEZN).

We sincerely hope these revisions adequately address your concerns, and we appreciate the time and care you have devoted to reviewing our work. We look forward to your feedback.

Best regards,

All Authors of Submission 16566

AuthorsOpenReview ↗

General Rebuttal Summary

We sincerely thank the reviewers for their careful assessments and constructive feedback, and the ACs, SACs, and PCs for their time and care in overseeing the review process.

We are grateful to all three reviewers for recognizing the value of learning tool behavior once and sharing that knowledge across agents that use the same service. Reviewer 7CEJ highlighted the cross-environment and cross-framework evaluations as direct tests of whether the knowledge is reusable. Reviewer GEZN particularly appreciated that ReAct and CodeAct could reuse memory built with SwitchAct without reconstruction. Reviewer Ycnz noted consistent improvements across MCPMark and MCP-Universe, covering eight MCP services.

In response, we added controlled comparisons and audits to Appendix H of the revised manuscript (highlighted in red) and reported two further tests during discussion. We summarize the findings by reviewer below. Paired gains denote percentage-point differences in MCPMark pass@1 / pass@4.

Reviewer 7CEJ

  • W1/Q1 — Simpler memory and novelty. The provider uses the graph to choose what to probe next and links each finding to procedures that use the tool. Given identical distilled records, ToolAtlas led flat retrieval by 7.06 / 8.67 points (Table 17). ToolAtlas makes the same knowledge more useful by retrieving related records together.
  • W2/Q2 — Do induced claims generalize? In the follow-up, direct tool calls on new inputs supported 55 of 62 retested Filesystem and PostgreSQL entries, including 32 of 36 boundary claims. Most tested rules held beyond the tasks from which they were learned.

Reviewer GEZN

Reviewer GEZN raised their rating from 2 to 3 after reviewing the additional comparisons and incremental-update results.

  • W1/Q1 — Provider-side benefit. In the follow-up, ReAct reused two SwitchAct-built memories with the same representation, retrieval, and construction budget. The memory built with frontier probes led its random-task counterpart by 4.31 / 5.46 points. Targeted exploration by the provider benefits agents that did not perform the exploration.
  • W2/Q2 — Stale knowledge. After three known Filesystem tool changes, re-exploring those tools reached 65.79% pass@1 versus 68.42% for a full rebuild, at 26.8% of its token cost (Tables 21–22). The provider can repair outdated guidance locally rather than rebuilding the whole memory.
  • Q3 — Budget sensitivity. Performance continued to improve with frontier probes after random sampling plateaued (Figure 3a). At a matched offline budget, ToolAtlas led ACE by 4.53 / 9.16 points (Table 15). ToolAtlas gains more from its exploration budget by targeting untested tool behaviors.

Reviewer Ycnz

  • W1–W2/Q1–Q2 — Data versus memory design. Given identical exploration data, ToolAtlas led ACE by 3.06 / 6.57 points. Replacing agent-neutral traces with raw rollouts cost 3.91 pass@1 points for SwitchAct but 11.47 when ReAct reused the memory (Tables 14–17). ToolAtlas makes better use of the same experience, and its agent-neutral traces preserve more of that benefit across agents.
  • W3/Q3 — Verifier and knowledge reliability. An independent auditor agreed with verifier labels on 97.45% of judgeable rollouts. An entry-level audit found support for 403 of 450 claims, with human checks on a subset (Tables 18–20). For most audited entries, ToolAtlas’s claims about what a tool can or cannot do were supported by actual tool behavior.

Finally, we briefly summarize the discussion stage.

Reviewers GEZN and Ycnz now give ToolAtlas a positive assessment. Reviewer 7CEJ appreciated the breadth of the additional experiments. Their follow-up prompted the new-input checks above and helped us clarify how ToolAtlas uses the same memory to guide exploration and support reuse across agents.

What distinguishes ToolAtlas?

  1. Learn about the tool, not just the task. ToolAtlas distills what executions reveal about a tool’s capabilities, limits, and interactions with other tools.
  2. Memory guides learning, not just retrieval. Recorded uses and gaps guide the next probe; its verified outcome updates the same memory.
  3. An agent-neutral memory organization for transfer and reuse. The graph links tools, procedures, and strategies without relying on the producing agent’s action syntax, so another agent can use the findings in its own workflow.
  4. A provider-side approach for the MCP ecosystem. Providers can maintain this knowledge alongside their tools, sharing the results of exploration and updates rather than asking each consumer to repeat the work.

We thank the reviewers, ACs, SACs, and PCs for their time and valuable feedback. We remain available for any further questions.

Warmest regards,

The Authors of Submission 16566

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.