- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how a multi-agent system can reuse experience across tasks that involve tools and long interactions. SCHEMAS stores queries, agents, tools, and distilled insights in one heterogeneous graph. It retrieves related queries through semantic similarity and predicted structural relations, builds a local memory subgraph, and selects an agent by matching capability profiles. Execution traces and task scores then update the graph. Tables 1 and 2 report gains across eight tool-use and interactive benchmarks with the same five-model team used for the multi-agent comparisons. The paper also reports lower token use and studies retrieval, routing, and memory components.
Critical strengths
- Sections 3 and 4 give a coherent mechanism for connecting retrieval, agent allocation, and trace updates. The typed relations make tool use and collaboration experience accessible from the same query anchor. This is a useful system design for reusing more than plain task transcripts.
- The empirical evaluation is broad and includes relevant controls. Tables 1 and 2 cover eight benchmarks and compare multi-agent methods with the same five-model team. Table 3 tests structural filtering and dynamic allocation, while Appendix C.3 separates the intent-sketch effect from the choice of generator. These results give more support than an end-to-end comparison alone.
Critical weaknesses
-
My main concern is what the improvement in Figure 6 actually measures. Appendix E.1 initializes the query graph from available queries and reports gains after 10 to 130 completed tasks, but I could not find the split or evaluation order used for this experiment. I cannot tell whether each checkpoint scores a fixed set of unseen tasks, subsequent tasks in an online stream, or tasks whose traces are already stored in memory. These settings distinguish transfer to new tasks from reuse of solutions to previously attempted tasks. I would like the authors to specify the existing protocol so that the continual-evolution claim can be assessed more precisely.
-
My second concern is whether the memory comparison in Table 2 controls access to task-level correctness feedback. Section 4 explicitly uses a task-evaluator score to update agent capabilities and graph weights. HotpotQA uses exact-match accuracy for evaluation, but I could not find whether memory updates receive this reference-based score or a separately computed signal. Appendix B.2 describes the memory baselines, yet I could not establish whether G-Memory and the other methods receive equivalent feedback after each task. I appreciate the shared five-model team, but I still need this feedback configuration to judge whether the gains come from memory organization or differences in supervision.
Questions for authors
-
I would like the authors to specify which splits supply the initial queries and the 10 to 130 completed tasks in Figure 6, and whether each checkpoint scores unseen, subsequent, or repeated tasks. I would also like to know whether task scores are recorded before the corresponding traces enter memory and whether Tables 1 and 2 use the same evaluation protocol.
-
I would like the authors to explain how the update score in Section 4 is obtained on HotpotQA, including whether reference answers are used and when the score becomes available. I would also like them to state whether G-Memory and the other memory baselines in Table 2 receive the same task-level feedback under the existing configuration.
3 comments
I find the authors' arguments convincing and my concerns have been largely addressed, therefore, I will keep my score and continue to lean positive toward acceptance.
Thank You for Your Positive Feedback
Dear Reviewer 2oBK,
We sincerely thank you for your positive feedback and for taking the time to carefully consider our responses. We are truly encouraged by your continued support and are pleased that our clarifications have largely addressed your concerns. Your thoughtful comments have been invaluable in helping us improve our work.
Wishing you all the best!
The Authors
Response to Reviewer 2oBK
Dear Reviewer 2oBK,
We sincerely thank you for your careful review and constructive feedback. We are particularly encouraged by your recognition of our unified graph framework, coherent design, and comprehensive experimental evaluation. Your positive assessment and insightful suggestions are greatly appreciated. We have considered your concerns and provide our detailed responses below:
Response to W1 & Q1: We thank the reviewer for this important question and apologize for the insufficient clarification of our evaluation protocol. To further validate the continual evolution capability of SCHEMAS and ensure the reliability and robustness of our findings, we repeated the experiments in Figure 6 across three independent runs and report the averaged results.
SCHEMAS starts from a cold-start state without accumulated execution experience. Specifically, the agent and tool graphs are initialized using static information, while the query and insight graphs are initially empty, without preloaded historical queries, execution traces, or reusable insights. For Figure 6, evaluation tasks are drawn from the ALFWorld
valid_unseensplit and the SciWorldtestsplit, without a separate offline experience-collection stage**.** Figure 6 follows a cumulative online evaluation protocol, where tasks are processed sequentially using only previously accumulated experience. Each task is unseen at its first execution and evaluated only once, with each checkpoint reporting the cumulative average performance over all completed tasks, rather than re-evaluating previous tasks or testing a separate set of future tasks. Importantly, each task's score is recorded before its execution trace and outcome are incorporated into memory, preventing the current task from benefiting from its own stored solution.Therefore, Figure 6 reflects the evolution of cumulative online performance as execution experience accumulates, rather than repeated evaluation of previously completed tasks. Furthermore, following the sequential memory accumulation and update paradigm adopted by G-Memory [1], Tables 1 and 2 of the main paper employ the same cold-start, online evaluation protocol, where tasks are processed sequentially and memory is updated only after each task is evaluated. We have explicitly clarified these experimental details in Appendix E.1 of the revised manuscript.
Response to W2 & Q2: We thank the reviewer for this important question and apologize for the insufficient clarification of our feedback configuration. For HotpotQA, the update score rnew is computed using exact-match correctness against the reference answer, yielding a binary score of 1 for correct responses and 0 otherwise. Importantly, the score is obtained only after task completion, and neither the reference answer nor the evaluation outcome is accessible during task execution. The resulting feedback is then used to update agent capabilities and graph weights for subsequent tasks.
Regarding evaluation fairness, all memory baselines in Table 2 of the main paper, including G-Memory, have access to the same task-level correctness feedback after task completion, while utilizing it according to their respective memory-update mechanisms. Therefore, SCHEMAS does not receive additional supervision unavailable to the compared baselines. We will clarify this feedback configuration in the revised manuscript.
References
[1] Zhang, G., Fu, M., Wang, K., et al. G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems. Advances in Neural Information Processing Systems, 38, 12988–13018 (2025).
We hope this resolves the consistency concerns, and we would be glad to clarify anything further.