acceptodds
Under review as a conference paper at ICLR 2027

SCHEMAS: Leveraging a Scalable Heterogeneous Graph for Query-Guided Reasoning in Multi-Agent Systems

Abstract

Multi-agent systems have demonstrated strong problem-solving capabilities, yet real-world tasks often require complex interactions that involve retrieving external knowledge and calling tools. In such settings, interaction dependencies among query, agent, and tool entities are often encoded implicitly and transiently in dialogues and execution traces, without an explicit structure to systematically represent invocation conditions and collaboration pathways. In this paper, we propose SCHEMAS, a scalable and configurable heterogeneous graph framework for multi-agent systems that unifies four entity types: query, agent, tool, and distilled insight, explicitly capturing their typed interactions to enable query-guided reasoning and continual evolution. To improve performance on a new query, we propose a query-driven structured memory mechanism that constructs a memory subgraph by retrieving the most relevant query, agent, tool, and insight nodes from the heterogeneous graph. Based on the subgraph, we allocate the query to the most compatible agent by matching the query’s required capability profile to agent capability representations. After execution, we apply the dynamic update mechanism that writes back the new query and its trace to the heterogeneous graph, thereby enabling the progressive evolution of the multi-agent systems. Extensive experiments across eight main benchmarks spanning tool-use and memory-augmented interactive settings demonstrate that SCHEMAS delivers consistently strong performance, scales across diverse tool-use scenarios and interactive tasks, and does so with lower token usage than state-of-the-art baselines.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Multi-agent systems, Heterogeneous Graph, Memory

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_2oBKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how a multi-agent system can reuse experience across tasks that involve tools and long interactions. SCHEMAS stores queries, agents, tools, and distilled insights in one heterogeneous graph. It retrieves related queries through semantic similarity and predicted structural relations, builds a local memory subgraph, and selects an agent by matching capability profiles. Execution traces and task scores then update the graph. Tables 1 and 2 report gains across eight tool-use and interactive benchmarks with the same five-model team used for the multi-agent comparisons. The paper also reports lower token use and studies retrieval, routing, and memory components.

Critical strengths
  1. Sections 3 and 4 give a coherent mechanism for connecting retrieval, agent allocation, and trace updates. The typed relations make tool use and collaboration experience accessible from the same query anchor. This is a useful system design for reusing more than plain task transcripts.
  2. The empirical evaluation is broad and includes relevant controls. Tables 1 and 2 cover eight benchmarks and compare multi-agent methods with the same five-model team. Table 3 tests structural filtering and dynamic allocation, while Appendix C.3 separates the intent-sketch effect from the choice of generator. These results give more support than an end-to-end comparison alone.
Critical weaknesses
  1. My main concern is what the improvement in Figure 6 actually measures. Appendix E.1 initializes the query graph from available queries and reports gains after 10 to 130 completed tasks, but I could not find the split or evaluation order used for this experiment. I cannot tell whether each checkpoint scores a fixed set of unseen tasks, subsequent tasks in an online stream, or tasks whose traces are already stored in memory. These settings distinguish transfer to new tasks from reuse of solutions to previously attempted tasks. I would like the authors to specify the existing protocol so that the continual-evolution claim can be assessed more precisely.

  2. My second concern is whether the memory comparison in Table 2 controls access to task-level correctness feedback. Section 4 explicitly uses a task-evaluator score to update agent capabilities and graph weights. HotpotQA uses exact-match accuracy for evaluation, but I could not find whether memory updates receive this reference-based score or a separately computed signal. Appendix B.2 describes the memory baselines, yet I could not establish whether G-Memory and the other methods receive equivalent feedback after each task. I appreciate the shared five-model team, but I still need this feedback configuration to judge whether the gains come from memory organization or differences in supervision.

Questions for authors
  1. I would like the authors to specify which splits supply the initial queries and the 10 to 130 completed tasks in Figure 6, and whether each checkpoint scores unseen, subsequent, or repeated tasks. I would also like to know whether task scores are recorded before the corresponding traces enter memory and whether Tables 1 and 2 use the same evaluation protocol.

  2. I would like the authors to explain how the update score in Section 4 is obtained on HotpotQA, including whether reference answers are used and when the score becomes available. I would also like them to state whether G-Memory and the other memory baselines in Table 2 receive the same task-level feedback under the existing configuration.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2oBK

Dear Reviewer 2oBK,

We sincerely thank you for your careful review and constructive feedback. We are particularly encouraged by your recognition of our unified graph framework, coherent design, and comprehensive experimental evaluation. Your positive assessment and insightful suggestions are greatly appreciated. We have considered your concerns and provide our detailed responses below:

W1: My main concern is what the improvement in Figure 6 actually measures. ... I cannot tell whether each checkpoint scores a fixed set of unseen tasks, subsequent tasks in an online stream, or tasks whose traces are already stored in memory... Q1: I would like the authors to specify which splits supply the initial queries and the 10 to 130 completed tasks in Figure 6, and whether each checkpoint scores unseen, subsequent, or repeated tasks. I would also like to know whether task scores are recorded before the corresponding traces enter memory and whether Tables 1 and 2 use the same evaluation protocol.

Response to W1 & Q1: We thank the reviewer for this important question and apologize for the insufficient clarification of our evaluation protocol. To further validate the continual evolution capability of SCHEMAS and ensure the reliability and robustness of our findings, we repeated the experiments in Figure 6 across three independent runs and report the averaged results.

SCHEMAS starts from a cold-start state without accumulated execution experience. Specifically, the agent and tool graphs are initialized using static information, while the query and insight graphs are initially empty, without preloaded historical queries, execution traces, or reusable insights. For Figure 6, evaluation tasks are drawn from the ALFWorld valid_unseen split and the SciWorld test split, without a separate offline experience-collection stage**.** Figure 6 follows a cumulative online evaluation protocol, where tasks are processed sequentially using only previously accumulated experience. Each task is unseen at its first execution and evaluated only once, with each checkpoint reporting the cumulative average performance over all completed tasks, rather than re-evaluating previous tasks or testing a separate set of future tasks. Importantly, each task's score is recorded before its execution trace and outcome are incorporated into memory, preventing the current task from benefiting from its own stored solution.

Therefore, Figure 6 reflects the evolution of cumulative online performance as execution experience accumulates, rather than repeated evaluation of previously completed tasks. Furthermore, following the sequential memory accumulation and update paradigm adopted by G-Memory [1], Tables 1 and 2 of the main paper employ the same cold-start, online evaluation protocol, where tasks are processed sequentially and memory is updated only after each task is evaluated. We have explicitly clarified these experimental details in Appendix E.1 of the revised manuscript.

W2: My second concern is whether the memory comparison in Table 2 controls access to task-level correctness feedback. ... I still need this feedback configuration to judge whether the gains come from memory organization or differences in supervision. Q2: I would like the authors to explain how the update score in Section 4 is obtained on HotpotQA, including whether reference answers are used and when the score becomes available. I would also like them to state whether G-Memory and the other memory baselines in Table 2 receive the same task-level feedback under the existing configuration.

Response to W2 & Q2: We thank the reviewer for this important question and apologize for the insufficient clarification of our feedback configuration. For HotpotQA, the update score rnewr_{\mathrm{new}} is computed using exact-match correctness against the reference answer, yielding a binary score of 1 for correct responses and 0 otherwise. Importantly, the score is obtained only after task completion, and neither the reference answer nor the evaluation outcome is accessible during task execution. The resulting feedback is then used to update agent capabilities and graph weights for subsequent tasks.

Regarding evaluation fairness, all memory baselines in Table 2 of the main paper, including G-Memory, have access to the same task-level correctness feedback after task completion, while utilizing it according to their respective memory-update mechanisms. Therefore, SCHEMAS does not receive additional supervision unavailable to the compared baselines. We will clarify this feedback configuration in the revised manuscript.

References

[1] Zhang, G., Fu, M., Wang, K., et al. G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems. Advances in Neural Information Processing Systems, 38, 12988–13018 (2025).


We hope this resolves the consistency concerns, and we would be glad to clarify anything further.

I find the authors' arguments convincing and my concerns have been largely addressed, therefore, I will keep my score and continue to lean positive toward acceptance.

AuthorsOpenReview ↗

Thank You for Your Positive Feedback

Dear Reviewer 2oBK,

We sincerely thank you for your positive feedback and for taking the time to carefully consider our responses. We are truly encouraged by your continued support and are pleased that our clarifications have largely addressed your concerns. Your thoughtful comments have been invaluable in helping us improve our work.

Wishing you all the best!

The Authors

Reviewer_V1rkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCHEMAS, a graph-based memory and coordination framework for LLM multi-agent systems. It represents queries, agents, tools, and distilled insights within a unified heterogeneous graph to make execution experience reusable across tasks. Its main contributions are query-conditioned memory retrieval combining semantic similarity with meta-path filtering, capability-aware agent allocation, and trace-driven updates to agent profiles, tool relations, and reusable insights. Retrieved memory supports direct execution, task decomposition, or collaboration among agents.

The authors evaluate SCHEMAS on eight benchmarks covering tool use, embodied interaction, symbolic planning, and multi-hop question answering: ToolBench, API-Bank, ToolHop, TMDB, ALFWorld, SciWorld, PDDL, and HotpotQA. The reported results outperform the evaluated tool-use and memory-augmented baselines, with gains of 3.15–5.27 percentage points over G-Memory on the four memory benchmarks. Ablations examine the contributions of individual graph components, meta-path filtering, and dynamic task allocation, while token-cost and runtime analyses assess efficiency.

Critical strengths
  1. The design is clear and well motivated. Putting queries, agents, tools, and insights into one typed graph makes cross-component links directly retrievable, for example which tools and agents were used on similar past queries. Compared with G-Memory’s layered design, the unified graph provides a different way to organize and retrieve cross-component experience.

  2. The system is complete and tested on diverse tasks. The paper describes retrieval, task allocation, and memory updates, with prompts and hyperparameters provided. SCHEMAS performs better than the compared baselines on eight main benchmarks and the additional HLE evaluation. If these gains hold under controlled comparisons, the framework would be useful in practice.

Critical weaknesses
  1. It is unclear where the performance gains come from. Table 1 has no memory-augmented baselines. The paper also lacks a G-Memory + DTA control and comparisons with random routing or always choosing the strongest agent. Single-agent baselines use only Qwen2.5-32B, while SCHEMAS uses five models, including GPT-4o-mini and QwQ-32B. Table 3 shows that removing DTA lowers ALFWorld performance by 5.40 percentage points, which is larger than the 5.27-point improvement over G-Memory. This raises the question of whether the gains mainly come from routing or the unified graph.

  2. The evaluation mainly uses older model backbones. The experiments rely on models such as Qwen2.5, QwQ-32B, and GPT-4o-mini. It is unclear whether SCHEMAS would provide similar gains with newer, stronger models that may already handle tool use and planning better. Testing a more recent backbone would help establish whether the framework’s benefits generalize.

Questions for authors
  1. Could you provide a G-Memory + DTA control and comparisons with random routing or always choosing the strongest agent? These comparisons would help clarify how much improvement comes from the unified graph rather than routing.

  2. As an additional validation, could you test newer models to check whether the improvements still hold?

2 comments
AuthorsOpenReview ↗

Response to Reviewer V1rk

Dear Reviewer V1rk,

We sincerely thank you for your constructive feedback and recognition of our framework's design and practical potential. Your comments provide valuable guidance for further strengthening our work. We address your concerns and questions in detail below:

W1: It is unclear where the performance gains come from. Table 1 has no memory-augmented baselines… Q1: Could you provide a G-Memory + DTA control and comparisons with random routing or always choosing the strongest agent? These comparisons would help clarify how much improvement comes from the unified graph rather than routing.

Response to W1 & Q1: We thank the reviewer for this important suggestion. To disentangle the contributions of heterogeneous graph memory and dynamic task allocation (DTA), we provide additional controlled comparisons on ALFWorld and SciWorld, including G-Memory + DTA, random routing, strongest-agent routing, flat memory, vector memory, and a variant without dynamic graph updates, under matched LLM backbones and execution settings. As shown in Table 1 below, SCHEMAS consistently outperforms all variants. Notably, it surpasses G-Memory + DTA by 6.42 and 3.32 percentage points on ALFWorld and SciWorld, respectively, indicating that introducing DTA alone does not reproduce the benefits of SCHEMAS. Moreover, SCHEMAS outperforms strongest-agent routing by 3.39 and 3.04 percentage points, suggesting that its gains cannot be attributed solely to selecting a stronger agent. The additional memory ablations further demonstrate the benefits of heterogeneous memory organization and dynamic graph updates.

Regarding evaluation fairness, we clarify that the single-agent baselines in Table 1 of the main paper serve as reference comparisons and are not strictly backbone-matched with SCHEMAS. In contrast, the multi-agent baselines use the same five-agent backbone configuration as SCHEMAS, ensuring comparisons under matched model settings. Furthermore, the additional controlled experiments maintain the same agent pool and execution settings, allowing us to better distinguish the contributions of heterogeneous graph memory and dynamic task allocation.

Table 1: Additional Ablation Study of SCHEMAS (%).

MethodALFWorld ↑SciWorld ↑
G-Memory55.0731.68
G-Memory + DTA53.9233.46
SCHEMAS w/ Random Routing55.7632.39
SCHEMAS w/ Strongest-Agent Routing56.9533.74
SCHEMAS w/ Flat Memory53.7230.41
SCHEMAS w/ Vector Memory56.8632.17
SCHEMAS w/o Dynamic Graph Updates58.2534.50
SCHEMAS (Full)60.3436.78

W2: The evaluation mainly uses older model backbones. The experiments rely on models such as Qwen2.5, QwQ-32B, and GPT-4o-mini. It is unclear whether SCHEMAS would provide similar gains with newer, stronger models that may already handle tool use and planning better. Testing a more recent backbone would help establish whether the framework’s benefits generalize. Q2: As an additional validation, could you test newer models to check whether the improvements still hold?

Response to W2 & Q2: We thank the reviewer for this valuable suggestion. To further evaluate the generalizability of SCHEMAS to stronger LLM backbones, we conduct additional experiments using an updated heterogeneous five-agent configuration comprising Llama-3.3-70B, Qwen3.8-Max, GPT-4o, GLM-5.3-Flash, and DeepSeek-V4.1-Flash. Given the limited rebuttal period, we focus on two representative benchmarks for each task category, comparing SCHEMAS with MetaGPT on ToolBench and API-Bank and G-Memory on ALFWorld and SciWorld, while maintaining identical backbone configurations across the compared methods.

As shown in Tables 2 and 3 below, SCHEMAS consistently outperforms both baselines under the updated configuration. Specifically, it surpasses MetaGPT by 6.15 and 12.13 percentage points on ToolBench and API-Bank, respectively, and outperforms G-Memory by 5.73 and 5.15 percentage points on ALFWorld and SciWorld. These results suggest that the benefits of structured memory and capability-aware coordination remain effective with stronger LLM backbones, demonstrating the generalizability of SCHEMAS across model configurations and task settings. We will incorporate these additional results into the revised manuscript.

Table 2: Generalization to Updated LLM Backbones on Tool-Use Tasks (%).

MethodToolBench Success ↑API-Bank Success ↑
MetaGPT71.2030.15
SCHEMAS77.3542.28

Table 3: Generalization to Updated LLM Backbones on Memory-Augmented Tasks (%).

MethodALFWorld ↑SciWorld ↑
G-Memory76.4239.16
SCHEMAS82.1544.31

We hope this resolves the consistency concerns, and we would be glad to clarify anything further.

AuthorsOpenReview ↗

Looking forward to your reply!

Dear Reviewer V1rk,

We sincerely thank you again for your thoughtful review and constructive feedback. As the review discussion period is approaching its end, we would like to kindly follow up on our previous response, in which we provided additional controlled experiments to disentangle the contributions of heterogeneous graph memory and dynamic task allocation, as well as new evaluations using stronger LLM backbones.

We hope these additional results help address your concerns regarding evaluation fairness and generalizability. We would be very grateful if you could take a moment to consider our updated evidence and let us know whether any concerns remain.

Thank you once again for your valuable time and consideration.

Warmest regards,

The Authors

Reviewer_8EDGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCHEMAS proposes a heterogeneous graph memory for multi-agent systems that connects queries, agents, tools, and insights. For each new query, it retrieves related historical experience, constructs a structured memory subgraph, routes the task to suitable agents, and then writes the new execution trace back into the graph. Experiments across tool-use and interactive reasoning benchmarks show consistent improvements over several multi-agent and memory-based baselines, with additional ablations on retrieval, routing, and continual updates.

Critical strengths

1.The paper presents a clear and coherent way to organize long-term multi-agent experience using a unified heterogeneous graph. The retrieval, routing, and update mechanisms are well connected to this overall design.

2.The experiments are relatively comprehensive and cover diverse benchmarks. The ablations on different graph components, meta-path retrieval, dynamic task allocation, and continual evolution provide useful support for the proposed framework.

Critical weaknesses

1.It is still difficult to disentangle the benefit of the proposed heterogeneous-graph design from the more general benefit of accumulating additional historical experience. Figure 6 shows that performance improves as more tasks are completed, but multiple factors change simultaneously: the query and insight memories grow, agent capability profiles are updated, collaboration weights change, and tool statistics are refined. The paper also shows that removing individual subgraphs degrades performance, but SCHEMAS differs from the baselines in retrieval, routing, and memory organization at the same time. A more controlled comparison using the same historical trajectories, LLM calls, and execution framework, but replacing the heterogeneous graph with a flat or vector-based memory, would make the specific contribution of the graph structure and dynamic updates much clearer.

2.SCHEMAS appears to trade persistent memory and retrieval overhead for improved performance on future tasks, but this trade-off is not characterized in enough detail. Figure 3 focuses on token cost, while persistent graph storage, retrieval latency, and graph-maintenance overhead are also important costs in this framework. Although the paper introduces budget-aware pruning, the experiments do not directly vary the size of the stored query or insight memory. It would be useful to report how performance, retrieval latency, and memory footprint change as the number of stored experiences grows, and whether the benefit eventually saturates relative to the additional memory and maintenance cost.

Questions for authors

1.Failed queries and unsuccessful execution traces are still stored in the memory graph, but it is not clear how these negative experiences affect future inference. Although the framework uses execution outcomes when updating graph statistics, failed trajectories may still be retrieved or contribute to later memories and insights. This seems particularly important for harder task distributions where the overall success rate is lower. It would be useful to analyze whether incorrect or low-quality historical experience can propagate to future decisions, and how robust SCHEMAS remains when a substantial fraction of the stored memory comes from failed tasks.

2.The framework appears to rely on the assumption that historical queries contain reusable information for future tasks, but its behavior on genuinely novel or only weakly related tasks is still unclear. When no structurally consistent candidates are found, SCHEMAS falls back to semantic top-k retrieval, which still selects the closest historical queries even if none are actually relevant. In such cases, irrelevant or outdated memories could potentially introduce negative transfer. An evaluation that controls the degree of similarity between new tasks and the stored history would help clarify when structured memory is useful and when it may instead interfere with reasoning.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8EDG (Part 1 of 2)

Dear Reviewer 8EDG,

We are grateful for your thoughtful evaluation and constructive suggestions. Your positive feedback on the clarity of our presentation, unified graph framework, and extensive experimental validation is greatly encouraging. We have carefully considered your comments and provide our detailed responses below:

W1: It is still difficult to disentangle the benefit of the proposed heterogeneous-graph design ... A more controlled comparison using the same historical trajectories, LLM calls, and execution framework, but replacing the heterogeneous graph with a flat or vector-based memory, would make the specific contribution ... much clearer.

Response to W1: We thank the reviewer for this insightful suggestion. To further isolate the contribution of the heterogeneous graph structure from the general benefits of accumulating historical experience, we conduct additional controlled experiments on ALFWorld and SciWorld. Specifically, we compare SCHEMAS with flat memory, vector-based memory, and a variant without dynamic graph updates, while maintaining the same historical trajectories, LLM backbones, and execution framework. As shown in Table 1 below, SCHEMAS consistently outperforms all three variants, achieving gains of 6.62 and 6.37 percentage points over flat memory on ALFWorld and SciWorld, respectively. The improvements over vector memory and the variant without dynamic graph updates further demonstrate the benefits of heterogeneous memory organization and dynamic graph updates, beyond merely accumulating historical experience. We will incorporate these additional analyses into the revised manuscript.

Table 1: Impact of Memory Organization on Task Performance (%).

MethodALFWorldSciWorld
SCHEMAS w/ Flat Memory53.7230.41
SCHEMAS w/ Vector Memory56.8632.17
SCHEMAS w/o Dynamic Graph Updates58.2534.50
SCHEMAS (Full)60.3436.78

W2: SCHEMAS appears to trade persistent memory and retrieval overhead for improved performance on future tasks, but this trade-off is not characterized in enough detail. ... It would be useful to report how performance, retrieval latency, and memory footprint change as the number of stored experiences grows, and whether the benefit eventually saturates relative to the additional memory and maintenance cost.

Response to W2: We thank the reviewer for this valuable suggestion. To investigate the trade-off between memory size, task performance, and computational overhead, we conducted an additional experiment on 2,000 HotpotQA tasks under different query memory budgets, while keeping the insight memory budget fixed. As shown in Table 2 below, the default budget of 1,000 queries triggers 1,000 memory evictions, reducing persistent storage by 39.5% (278.36 MB to 168.29 MB) and retrieval latency by 17.6% (7.40 s to 6.10 s), while improving EM from 29.10% to 30.10% compared with unbounded memory. A smaller budget of 500 further reduces storage but lowers EM to 27.80%. Furthermore, Table 3 shows that graph maintenance incurs only 3.90 s per task (4.3% of total latency), without additional LLM tokens. While Figure 6 illustrates diminishing performance gains as execution experience accumulates, the additional 2,000-task experiment shows that retaining all historical experiences does not necessarily yield better performance. Together, these results suggest that budget-aware pruning provides a favorable trade-off between task performance, retrieval efficiency, and memory overhead.

Table 2: Impact of Memory Budget on Performance and Memory Efficiency (HotpotQA).

MethodQuery BudgetPeak Query NodesEvictionsEM (%) ↑Retrieval Latency (s) ↓Persistent Storage (MB) ↓
SCHEMAS w/o PruningUnlimited2,000029.107.40278.36
SCHEMAS5005001,50027.805.4579.18
SCHEMAS (Default)1,0001,0001,00030.106.10168.29

Table 3: Component-wise Computational Cost Analysis on HotpotQA.

ComponentAvg. Tokens / TaskAvg. Latency (s)
Intent-Sketch Generation1,85011.80
Capability Extraction7634.70
Strategy Selection1,5045.90
Meta-path Filtering & Subgraph Construction03.20
Insight Extraction / Consolidation2,10510.60
Graph Maintenance03.90
Agent Execution17,91044.60
Final Response Generation2,2685.80
Total26,40090.50
AuthorsOpenReview ↗

Response to Reviewer 8EDG (Part 2 of 2)

Q1: Failed queries and unsuccessful execution traces are still stored in the memory graph, but it is not clear how these negative experiences affect future inference. ... It would be useful to analyze whether incorrect or low-quality historical experience can propagate to future decisions, and how robust SCHEMAS remains when a substantial fraction of the stored memory comes from failed tasks.

Response to Q1: We thank the reviewer for raising this important concern. In SCHEMAS, failed trajectories are retained to preserve historical execution information, while outcome-aware graph updates prevent unsuccessful interactions from receiving the same positive reinforcement as successful ones. To further examine the impact of negative experiences, we conducted an additional robustness analysis on HotpotQA by varying the proportion of failed historical trajectories from 0% to 75%, while maintaining a fixed memory size of 200 historical trajectories and identical evaluation settings. As shown in Table 4 below, SCHEMAS maintains relatively stable performance across different failure ratios, with EM decreasing from 29.85% ± 0.78% to 27.40% ± 1.03%, a drop of 2.45 percentage points. These results suggest that SCHEMAS exhibits reasonable robustness to failed historical experiences, although some performance degradation may occur as the proportion of unsuccessful trajectories increases. We will incorporate this additional analysis into the revised manuscript.

Table 4: Robustness to Failed Historical Experiences on HotpotQA.

Failed Trace RatioSuccessful TracesFailed TracesEM (%) ↑
0%200029.85 ± 0.78
25%1505028.55 ± 0.91
50%10010028.79 ± 0.86
75%5015027.40 ± 1.03

Q2: The framework appears to rely on the assumption that historical queries contain reusable information for future tasks, but its behavior on genuinely novel or only weakly related tasks is still unclear. ... An evaluation that controls the degree of similarity between new tasks and the stored history would help clarify when structured memory is useful and when it may instead interfere with reasoning.

Response to Q2: We thank the reviewer for this insightful suggestion. To investigate potential negative transfer from weakly related historical memories, we conducted an additional experiment on HotpotQA. Specifically, we divide test queries into three equal-sized groups (High, Medium, and Low) based on their maximum cosine similarity to stored historical queries, and compare SCHEMAS with a variant that disables historical memory retrieval while retaining the same multi-agent execution framework. As shown in Table 5 below, SCHEMAS consistently outperforms the variant without retrieved memory, achieving improvements of 6.15, 3.42, and 2.48 percentage points across the three relevance levels, respectively. Notably, the positive gain under Low Relevance suggests that SCHEMAS can benefit from transferable reasoning strategies and execution patterns encoded in the heterogeneous memory graph, even when historical queries are only weakly related to new tasks. These results indicate that memory retrieval remains beneficial across different relevance levels, although its advantages diminish as semantic similarity decreases. We will incorporate this additional analysis into the revised manuscript.

Table 5: Impact of Historical Memory Relevance on HotpotQA Performance (%).

Historical RelevanceSCHEMAS (Full) EM ↑SCHEMAS w/o Retrieved Memory EM ↑ΔEM (pp) ↑
High32.8526.70+6.15
Medium29.3725.95+3.42
Low27.4024.92+2.48

We hope this resolves the consistency concerns, and we would be glad to clarify anything further.

I really appreciate the authors’ detailed responses to my questions and the time taken to run the additional experiments. These results are very helpful in clarifying several of my previous concerns.

One point I am still not fully clear about is what the framework learns beyond selectively reusing similar past experience. In the newly added experiments, the historical memory relevance analysis shows that the benefit becomes smaller as the current query is less related to the stored history. Meanwhile, the memory-budget experiment suggests that keeping a selected subset of frequently useful or successful experiences can even outperform retaining the full history.

These results make sense for an experience-retrieval system, but they also make me wonder how much of the reported “continual evolution” comes from becoming increasingly well matched to recurring patterns within the same task distribution. In particular, when a new task cannot directly benefit from a similar previously solved query, it is still unclear to me what transferable information has actually been learned through the agent, tool, or insight graph updates. I think a clearer distinction between case-level experience reuse and more generalizable structural adaptation would better support the broader continual-evolution framing. I plan to keep my original score for now.

Reviewer_6txuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces a scalable heterogeneous graph framework for improving multi-agent systems through query-guided reasoning and continual self-evolution. The proposed approach unifies queries, agents, and tools within a single, extensible schema to capture complex dependencies. A dynamic update mechanism incorporates execution traces into the graph to refine relationships and capabilities. The framework is evaluated on benchmarks involving tool use and interactive tasks.

Critical strengths
  • Unifies queries, agents, tools, and distilled insights into a single extensible schema
  • Combines query similarity with meta-path filtering guided by intent sketches to retrieve structured memory subgraphs
  • Replaces static agent assignments with dynamic task allocation by matching task requirement vectors to evolving agent capability profiles
  • Demonstrates performance improvements and reduced token consumption in the experiments.
Critical weaknesses
  • The main tables report single-seed point estimates. Several headline gains over G-Memory sit inside or barely outside that noise band. Besides, the same configuration is reported with different values across the paper. For an empirical paper whose only contribution is empirical, this alone is disqualifying. For an empirical paper whose contribution leans to empirical assessment, this raises a concern.
  • The framework introduces a heavy, multi-stage LLM calling pipeline before and after execution. This creates cumulative inference latency and heavy reliance on structured JSON parsing. The authors should expose the efficiency metrics clearly and thoroughly.
  • In the paper it varies the quantities that define scale. The agent count is fixed at five throughout. The tool inventory is never swept. The number of stored query nodes never approaches the stated budget. For a paper whose title word highlights scalability, the authors cannot defer the scalability experiment.
  • Figure 7's clean final-state specialization is a single-seed visualization presented as a mechanism finding. With the variance in Table 6, a different seed could easily yield a different specialization pattern or none.
Questions for authors

n/a

6 comments
AuthorsOpenReview ↗

Response to Reviewer 6txu (Part 1 of 2)

Dear Reviewer 6txu,

We sincerely thank you for your thorough review and insightful comments. We carefully address your concerns below:

W1: The main tables report single-seed point estimates. Several headline gains over G-Memory sit inside or barely outside that noise band. Besides, the same configuration is reported with different values across the paper. For an empirical paper whose only contribution is empirical, this alone is disqualifying...

Response to W1: We thank the reviewer for highlighting the importance of experimental reliability. In our preliminary experiments, we observed relatively stable performance across different runs. We adopted a fixed random seed of 42 for the main experiments, following a common practice in LLM evaluations. Nevertheless, we agree that additional runs would provide stronger evidence of robustness. To address this concern, we conducted three independent runs for SCHEMAS and two representative competitive baselines (MetaGPT [1] and G-Memory[2]) across four tool-use benchmarks and four memory-intensive benchmarks, respectively. As shown in Tables 1 and 2 below, SCHEMAS consistently outperforms both baselines, with results reported as mean ± standard deviation, providing stronger evidence for the reliability of our main findings.

Furthermore, we repeated the hyperparameter sensitivity experiments across three independent runs and updated Figure 5 to report the mean and standard deviation, addressing the inconsistencies in the original sensitivity results. To further validate the continual evolution capability of SCHEMAS, we also conducted three new independent runs for Figure 6 and updated it with the averaged results. The revised curves show an overall increase in cumulative online performance across independent runs, consistent with the benefits of progressively accumulated execution experience. We will incorporate these additional experiments and clarifications into the revised manuscript.

Table 1: Performance (%) comparison with multi-agent baselines across four tool-use benchmarks (Mean ± Std).

MethodToolBench SuccessToolBench PathAPI-Bank SuccessAPI-Bank PathToolHop CorrectToolHop PathTMDB SuccessTMDB Path
MetaGPT63.54±1.1437.61±0.8923.57±1.2366.39±1.3441.52±1.2145.85±1.3661.86±1.3174.28±0.80
SCHEMAS69.41±0.9142.35±0.7334.18±0.9270.29±0.7946.46±1.4349.32±0.8766.55±1.2077.24±0.84

Table 2: Performance (%) comparison with memory-augmented baselines across four benchmarks (Mean ± Std).

MethodALFWorldSciWorldPDDLHotpotQA
G-Memory54.98 ± 1.1432.28 ± 0.9218.62 ± 1.2425.47 ± 0.83
SCHEMAS60.20 ± 0.8636.62 ± 1.1221.73 ± 0.9630.08 ± 0.71

W2: The framework introduces a heavy, multi-stage LLM calling pipeline before and after execution. This creates cumulative inference latency and heavy reliance on structured JSON parsing. The authors should expose the efficiency metrics clearly and thoroughly.

Response to W2: We thank the reviewer for this important comment. As shown in Figure 3 of the main paper, we have already evaluated the trade-off between task performance and overall system token consumption across benchmarks. Under matched multi-agent backbone configurations and evaluation settings, SCHEMAS demonstrates favorable performance–token efficiency compared with existing baselines. Moreover, the reported token consumption and end-to-end latency cover the entire SCHEMAS pipeline, including intent-sketch generation, capability extraction, strategy selection, insight extraction/consolidation, graph maintenance, and agent execution. To further provide a fine-grained assessment of computational overhead, we conducted an additional component-wise cost analysis on HotpotQA, as shown in Table 3 below. The results show an average end-to-end cost of 26,400 tokens and 90.50 s per task, with graph maintenance accounting for 3.90 s (4.3% of total latency) and requiring no additional LLM tokens.

Regarding structured JSON parsing, SCHEMAS employs predefined output formats and validation procedures. Specifically, capability extraction validates generated JSON outputs and retries once upon parsing failures, with a predefined fallback mechanism for persistent errors. These safeguards help mitigate the impact of malformed outputs on system execution. We will incorporate the detailed cost analysis and implementation clarifications into the revised manuscript.

Table 3: Component-wise Computational Cost Analysis on HotpotQA.

ComponentAvg.Tokens/TaskAvg.Latency(s)
Intent-SketchGeneration1,85011.80
CapabilityExtraction7634.70
StrategySelection1,5045.90
Meta-pathFiltering&SubgraphConstruction03.20
InsightExtraction/Consolidation2,10510.60
GraphMaintenance03.90
AgentExecution17,91044.60
FinalResponseGeneration2,2685.80
Total26,40090.50
AuthorsOpenReview ↗

Response to Reviewer 6txu (Part 2 of 2)

W3: In the paper it varies the quantities that define scale. The agent count is fixed at five throughout. The tool inventory is never swept...

Response to W3: We agree that additional scalability experiments would further strengthen our evaluation. To address this concern, we conduct two supplementary experiments examining scalability from the perspectives of agent pool size and long-term memory growth.

  • Agent Scalability. We vary the number of agents from 3 to 15 on ALFWorld, while keeping the remaining experimental settings consistent. As shown in Table 4 below, performance initially improves and reaches its peak at 7 agents (61.52%), after which additional agents introduce diminishing returns and increased token consumption. These results characterize the performance–efficiency trade-off across different agent pool sizes.
  • Memory Scalability. We further evaluate SCHEMAS on 2,000 HotpotQA tasks under different query memory budgets. As shown in Table 5 below, the default budget of 1,000 limits peak memory to 1,000 query nodes and triggers 1,000 evictions. Compared with unbounded memory, SCHEMAS reduces average retrieval latency from 7.40 s to 6.10 s (17.6%), while maintaining comparable task performance (30.10% vs. 29.10% EM). This demonstrates the effectiveness of budget-aware maintenance in controlling memory growth during sustained operation.
  • Regarding tool scalability, SCHEMAS constructs its tool graph from the available tool inventory without imposing a fixed upper bound and supports incremental integration of new tools. The original experiments also cover diverse tool-use benchmarks, including ToolBench with its large API inventory.

Table 4: Scalability Analysis with Varying Agent Counts on ALFWorld.

AgentCountSuccessRate(%)↑TotalTokens(×10⁶)↓
358.684.62
5(Default)60.344.90
761.526.18
959.437.57
1558.769.35

Table 5: Impact of Memory Budget on Performance and Maintenance Efficiency (HotpotQA).

MethodQueryBudgetPeakQueryNodesEvictionsEM(%)↑Avg.RetrievalLatency(s)↓
SCHEMASw/oPruningUnlimited2,000029.107.40
SCHEMAS5005001,50027.805.45
SCHEMAS(Default)1,0001,0001,00030.106.10

W4: Figure 7's clean final-state specialization is a single-seed visualization presented as a mechanism finding. With the variance in Table 6, a different seed could easily yield a different specialization pattern or none.

Response to W4: At initialization (0 tasks), all agents share the same six-dimensional capability vector, CiA,(0)=[0.60,0.60,0.60,0.60,0.60,0.60]C_i^{A,(0)}=[0.60,0.60,0.60,0.60,0.60,0.60], without predefined specialization. Although agents are initialized with identical capability representations, they employ different LLM backbones and may exhibit distinct competencies. Thus, capability specialization emerges through task-driven interactions and dynamic capability updates, rather than being manually assigned.

To further examine the consistency of this phenomenon, we conducted an additional experiment on ALFWorld using a different random seed (123), complementing the original visualization obtained with seed 42. As shown in the supplementary results, agents progressively develop differentiated capability profiles from their shared initialization. For example, Agent 2's sequence capability increases from 0.78 to 0.94, while Agent 3's mathematical capability increases from 0.73 to 0.96 between 60 and 130 completed tasks. The additional run exhibits a similar overall trend of emergent capability differentiation, providing further evidence that this phenomenon is not restricted to the original random seed. We will include these results and clarify the illustrative nature of Figure 7 in the revised manuscript.

Intermediate State (60 Tasks)

AgentReasoningMathematicalLanguageKnowledgeSequenceSpatial
Agent00.650.620.630.730.620.69
Agent10.620.630.710.780.610.65
Agent20.720.650.620.650.780.69
Agent30.620.730.700.640.610.62
Agent40.640.640.610.620.740.67

Final State (130 Tasks)

AgentReasoningMathematicalLanguageKnowledgeSequenceSpatial
Agent00.650.610.640.850.620.79
Agent10.620.610.820.900.620.66
Agent20.830.690.620.640.940.85
Agent30.610.960.800.620.610.62
Agent40.610.620.880.610.900.75

References

[1]Hong S, Zhuge M, Chen J, et al. MetaGPT: Meta programming for a multi-agent collaborative framework. International Conference on Learning Representations. 2024, 2024: 23247-23275.

[2] Zhang, G., Fu, M., Wang, K., et al. G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems. Advances in Neural Information Processing Systems, 38, 12988–13018 (2025).


We hope this resolves the consistency concerns, and we would be glad to clarify anything further.

Thank the authors for the detailed rebuttal and the extra experiments. I read them carefully. They help in places, but my main concerns remain, so I am keeping my score.

The rebuttal does not clearly explain why the same configuration was reported with different values. For example, it says Figure 5 was "updated," but not why the original numbers disagreed. For a paper whose contribution is mainly empirical, this inconsistency is still a soundness concern.

If I did not overlook anything, the new experiments show that scaling is still sort of limited. Since scalability is central to the paper's framing, this evidence does not strongly support the claim.

Reviewer_Rg1xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

SCHEMAS is a graph-based framework for memory and coordination in multi-agent systems. It represents queries, agents, tools, and distilled insights within a unified heterogeneous graph. For a new query, semantic retrieval and intent-guided meta-path filtering construct a relevant memory subgraph, while capability-profile matching selects an agent.

Critical strengths
  1. The framework provides a coherent interface between experience retrieval and agent coordination. Queries, agents, tools, and insights are explicitly connected, allowing retrieved experience to inform task allocation and subsequent execution to update the same representation. The concrete update rules, bounded query/insight memory, and detailed prompts make this a useful systems contribution beyond an informal orchestration proposal.

  2. The evaluation covers a useful range of settings and includes relevant controls. The eight main benchmarks span tool use and memory-augmented interactive tasks. Tables 2–3 examine graph components, meta-path filtering, and dynamic allocation, while Table 7 controls the sketch generator. The shared multi-agent backbone configuration and the reported gains over G-Memory strengthen the empirical case for the combined framework.

Critical weaknesses
  1. The evaluation protocol and result consistency are not specified sufficiently to establish the reliability of the reported gains. Section 4 updates memory using a score from the task evaluator, and Appendix E.1 initializes the query graph from "available queries," but the manuscript does not clearly state the provenance of the initial memory, the relationship between experience-collection and evaluation tasks, whether test tasks update memory, or whether Figure 6 evaluates a fixed set of unseen tasks. Task ordering, memory resets, and comparable historical information for the memory baselines also need clarification. In addition, Appendices B.3 and C.1 specify shared defaults of h=2h=2, k=7k=7, and β=0.9\beta=0.9, yet the corresponding ALFWorld points in Figure 5 are approximately 55.3%, 56.5%, and 60.1%, respectively, compared with 60.34% in Table 2 and 60.17% in Tables 6–8. The statement that studies were run separately does not specify which other conditions differed. Since the main tables lack uncertainty estimates, these discrepancies materially limit confidence in the comparisons and the interpretation of continual improvement.

  2. The evidence for scalable, sustained operation is weaker than the paper's framing suggests. The explicit memory budgets and the complexity analysis in Appendix D.1 are valuable, but Figure 6 extends only to 130 completed tasks, compared with a staleness threshold of 200 and a query budget of 1000. Without initial memory sizes, occupancy traces, or eviction counts, it is unclear whether the study exercises the maintenance mechanisms or establishes their effect on retrieval quality. The end-to-end scaling cost is also underspecified: the intent-sketch prompt receives agent, tool, and insight catalogs, but their construction and size are not explained, including for ToolBench's large tool inventory. A small retrieved subgraph does not itself bound this earlier input cost. Table 8 provides a useful latency measurement, and Appendix D.3 appropriately acknowledges that HLE does not vary system size, but these results do not yet establish the broader scalability claim.

Questions for authors
  1. Evaluation protocol. Please specify the source and contents of the initial query/trace memory, the data split used for collecting experience versus evaluating performance, whether memory is updated during test evaluation, and which evaluator feedback is available at each stage. How are task order and memory resets handled, do the memory baselines receive comparable historical information and feedback, and does Figure 6 use the same unseen evaluation set at every point?

  2. Budget activation and complete cost accounting. In the reported runs, what were the initial and maximum memory sizes, and how often did budget-based eviction occur? Are the catalogs supplied to the intent-sketch generator complete, prefiltered, or truncated? Please clarify whether the reported token and latency totals include sketch generation, capability extraction, strategy selection, insight extraction/consolidation, and graph maintenance. Existing execution logs or a precise qualification of the claims would help assess this point.

8 comments
AuthorsOpenReview ↗

Response to Reviewer Rg1x (Part 1 of 4)

Dear Reviewer Rg1x,

First, we sincerely thank you for your thorough review and insightful comments on our paper. We greatly appreciate your recognition of our coherent framework design and comprehensive experimental evaluation. Your constructive feedback is invaluable in helping us further strengthen our work. We carefully address your concerns below:

W1: The evaluation protocol and result consistency are not specified sufficiently to establish the reliability of the reported gains. Q1: Evaluation protocol. Please specify the source and contents of the initial query/trace memory, the data split used for collecting experience versus evaluating performance, whether memory is updated during test evaluation, and which evaluator feedback is available at each stage. How are task order and memory resets handled, do the memory baselines receive comparable historical information and feedback, and does Figure 6 use the same unseen evaluation set at every point?

Response to W1 & Q1: We thank the reviewer for these valuable comments, which help us clarify our evaluation protocol and improve the reliability and reproducibility of our results. We clarify the following points.


(1) Evaluation Protocol

  • Source and contents of the initial query/trace memory. Following the experience-driven memory evolution paradigm of G-Memory, SCHEMAS adopts a cold-start setting. The agent graph is initialized with shared capability priors, and the tool graph with available tool information, while both the query and insight graphs start empty. As tasks are completed, queries, execution traces, and reusable insights are progressively incorporated into the heterogeneous graph.
  • Data split for experience collection versus performance evaluation. Following G-Memory's [1] benchmark settings, we use the valid_unseen split of ALFWorld and the test split of SciWorld for evaluation. SCHEMAS adopts a cold-start, online evaluation protocol without a separate offline experience-collection stage. Tasks are processed sequentially, with each task evaluated using only previously accumulated memory, while execution traces and feedback are incorporated into memory after evaluation to benefit subsequent tasks.
  • Memory updates during test evaluation. Yes, SCHEMAS updates its memory during online evaluation. Each task is executed using the memory accumulated from preceding tasks, and its execution traces and evaluator feedback are incorporated into the graph after completion. These updates benefit subsequent tasks, ensuring that the current task's evaluation does not use its own outcome as prior knowledge.
  • Evaluator feedback at each stage. At initialization, no evaluator feedback is available. During task execution, SCHEMAS relies on previously accumulated experience without accessing the current task's evaluation outcome. Upon completion, the evaluator provides performance feedback, including success/failure status and evaluation scores, which is used to update the graph and guide subsequent tasks.
  • Task ordering and memory resets. Tasks are processed sequentially in their original dataset order, without random shuffling. Memory is retained and updated across tasks within each run, but reset to its initial cold-start state before each independent run.
  • Comparable historical information and feedback for memory baselines. All memory-based baselines are evaluated on the same task sequences under comparable cold-start conditions, without preloaded execution experience. Each method has access to the same task-level evaluation feedback after task completion, while independently accumulating execution trajectories and updating memory according to its original mechanism. Thus, differences in feedback utilization arise from the respective memory designs rather than unequal access to supervision.
  • Same unseen evaluation set at every point in Figure 6. We would like to clarify that Figure 6 follows a cold-start, cumulative online evaluation protocol, rather than repeatedly evaluating a fixed test set. Each task is unseen at its first execution and evaluated only once using previously accumulated memory. Each checkpoint reports the cumulative average performance over all completed tasks, rather than re-evaluating previous tasks or testing a separate set of future tasks. To further validate the continual evolution capability of SCHEMAS and ensure the reliability and robustness of our findings, we conducted three new independent runs and updated Figure 6 with the averaged results.

AuthorsOpenReview ↗

Response to Reviewer Rg1x (Part 2 of 4)

(2) Result Consistency. We thank the reviewer for identifying these inconsistencies and apologize for the discrepancies in our original experimental results. Our original sensitivity analyses were conducted using a single run with a fixed random seed of 42, which did not fully capture run-to-run variability. To address this concern, we repeated the hyperparameter sensitivity experiments across three independent runs and updated Figure 5 to report the mean and standard deviation. The updated results exhibit overall trends consistent with our original findings, further supporting the robustness of SCHEMAS to hyperparameter variations. Importantly, as shown in the updated Figure 5, the ALFWorld results at the shared default configurations (h=2h=2, k=7k=7, and β=0.9\beta=0.9) are consistently around 60%, closely aligning with the three-run result of 60.27 ± 0.83% reported in Table 2 below. Furthermore, we provide supplementary mean and standard deviation results across three independent runs for SCHEMAS and representative baselines, further supporting the reliability of our main comparisons.

Table 1: Performance (%) comparison with multi-agent baselines across four tool-use benchmarks (Mean ± Std).

MethodToolBench SuccessToolBench PathAPI-Bank SuccessAPI-Bank PathToolHop CorrectToolHop PathTMDB SuccessTMDB Path
MetaGPT63.54 ± 1.1437.61 ± 0.8923.57 ± 1.2366.39 ± 1.3441.52 ± 1.2145.85 ± 1.3661.86 ± 1.3174.28 ± 0.80
SCHEMAS69.41 ± 0.9142.35 ± 0.7334.18 ± 0.9270.29 ± 0.7946.46 ± 1.4349.32 ± 0.8766.55 ± 1.2077.24 ± 0.84

Table 2: Performance (%) comparison with memory-augmented baselines across four benchmarks (Mean ± Std).

MethodALFWorldSciWorldPDDLHotpotQA
G-Memory54.98 ± 1.1432.28 ± 0.9218.62 ± 1.2425.47 ± 0.83
SCHEMAS60.20 ± 0.8636.62 ± 1.1221.73 ± 0.9630.08 ± 0.71
AuthorsOpenReview ↗

Response to Reviewer Rg1x (Part 3 of 4)

W2: The evidence for scalable, sustained operation is weaker than the paper's framing suggests. Q2: Budget activation and complete cost accounting. In the reported runs, what were the initial and maximum memory sizes, and how often did budget-based eviction occur? Are the catalogs supplied to the intent-sketch generator complete, prefiltered, or truncated?

Response to W2 & Q2: We thank the reviewer for these valuable comments and sincerely apologize for the insufficient clarification of memory maintenance and computational costs. We provide the following clarifications and supplementary analyses.


1. Initial and Maximum Memory Sizes and Eviction Frequency. SCHEMAS adopts a cold-start setting, where both the query and insight graphs are initially empty. The agent graph is initialized with five agents, while the tool graph is constructed from the available tool inventory. The query and insight memory budgets are set to BQ=1000B_Q=1000 and BI=500B_I=500, respectively, with a staleness threshold of 200 update steps. We acknowledge that Figure 6 covers only 130 completed tasks and therefore does not activate query-budget eviction or staleness-based pruning. To further evaluate the effectiveness of memory maintenance under longer execution horizons, we conducted an additional experiment on 2,000 HotpotQA tasks with different query-memory budgets. As shown in Table 3 below, the default budget limits peak query memory to 1,000 nodes, with 1,000 recorded budget-based evictions. These eviction counts specifically refer to query nodes removed due to memory-budget constraints, rather than staleness-based maintenance operations**.** Compared with the unbounded setting, SCHEMAS improves EM from 29.10% to 30.10%, reduces average retrieval latency from 7.40 s to 6.10 s (17.6%), and decreases persistent storage from 278.36 MB to 168.29 MB. These results demonstrate that budget-based memory maintenance effectively controls memory growth while preserving retrieval efficiency and performance.

Table 3: Impact of Memory Budget on Performance and Memory Efficiency (HotpotQA).

MethodQuery BudgetPeak Query NodesEvictionsEM (%) ↑Retrieval Latency (s) ↓Persistent Storage (MB) ↓
SCHEMAS w/o PruningUnlimited2,000029.107.40278.36
SCHEMAS5005001,50027.805.4579.18
SCHEMAS (Default)1,0001,0001,00030.106.10168.29

2. Catalog Construction. We clarify that the catalogs supplied to the intent-sketch generator are task-specific rather than complete benchmark-wide inventories. The agent catalog contains the available agents and their capability profiles, while the tool catalog is prefiltered to tools accessible in the current task context. The insight catalog contains the reusable strategies available from the maintained memory, with its size constrained by the insight budget BI=500B_I=500. In particular, ToolBench does not require supplying the entire benchmark tool inventory to each intent-sketch prompt. No additional truncation is applied after task-specific catalog construction. We acknowledge that intent-sketch input costs also depend on catalog sizes, rather than solely on the retrieved subgraph size. We will explicitly document the catalog construction procedure and input constraints in the revised manuscript.

3. Complete Cost Accounting. We thank the reviewer for raising this important point. The total token consumption and end-to-end latency reported in our experiments cover the entire SCHEMAS pipeline, including intent-sketch generation, capability extraction, strategy selection, insight extraction/consolidation, graph maintenance, and agent execution, as well as auxiliary operations such as embedding and retrieval. Notably, graph maintenance incurs computational latency without consuming additional LLM tokens. To provide a more fine-grained analysis, we conducted an additional component-wise evaluation on HotpotQA, as shown in Table 4 below. The results show an average end-to-end cost of 26,400 tokens and 90.50 s per task, with graph maintenance accounting for 3.90 s and zero LLM tokens. We will incorporate these detailed results into the revised manuscript.

Table 4: Component-wise Computational Cost Analysis on HotpotQA.

ComponentAvg. Tokens / TaskAvg. Latency (s)
Intent-Sketch Generation1,85011.80
Capability Extraction7634.70
Strategy Selection1,5045.90
Insight Extraction / Consolidation2,10510.60
Graph Maintenance03.90
Agent Execution17,91044.60
Total26,40090.50

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.