- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- the paper introduces CoMATE, a harness or multiagent harness for agents interacting and consolidating previous interaction histories into reusable shared knowledge pools accessible by all agents for future game exploration. The stored knowledge or interaction histories mainly record discovered world/game rules and when they can be applied.
- the core contribution part is the consolidation of previous histories while other parts of the harness are mostly common and presented as baselines. The consolidation is done by a separate agent verifying and consolidating the agents that operate in the game's notes into shared findings with grounded evidence and also some uncertain discoveries.
- the paper mainly tries to solve ARC-AGI-3 benchmark and shows improvements over baseline harness without knowledge consolidation. The baseline harness was adapted from a prior released harness.
Critical strengths
The paper demonstrated that with memory consolidation across agents and maintaining a shared pool of grounded knowledge of the game world, it is beneficial to the multiagent system for future exploration and operation in the game world. This is the core contribution in my opinion, and the paper mainly compares CoMATE versus B6 (an adapted harness from an existing released harness). The results are supporting that shared memory consolidation is useful for multiagent systems and thus is what makes me think the paper has some contributions overall.
Critical weaknesses
- Only testing on ARC-AGI-3 makes the claim that sharing and consolidating memories across agents (with specific checks from an external agent instead of directly trusting the single agent in the game worlds) in exploring an open world less convincing on its generalizability across diverse tasks. Even though the results on ARC-AGI-3 is supporting the main claim, limiting in 1 type of environment makes the harness design less convincing.
- The paper already cites exiting works such as G-Mem [1] and Symbolica [2] and the claimed contribution is on whether the shared knowledge should be consolidated by another agent instead of trusting the agent that acts in the games. The claim is a good direction to study but the main experiments are not substantial in showing this. The incremental improvements may need better grounding in the existing literatures. Also, related work [3] has similar knowledge consolidating mechanism though is more grounded in the environment and thus making the grounding more verifiable, so the contribution claim may need to be reconsidered.
[1] Zhang, Guibin, et al. "G-memory: Tracing hierarchical memory for multi-agent systems." Advances in Neural Information Processing Systems 38 (2026): 12988-13018. [2] Symbolica. From 0% to 36% on day 1 of ARC-AGI-3. Symbolica Blog, March 2026a. URL https://www.symbolica.ai/blog/arc-agi-3. [3] Xu, H., Hu, J., Zhang, K., Yu, L., Tang, Y., Song, X., ... & Shi, B. (2025). Sedm: Scalable self-evolving distributed memory for agents. arXiv preprint arXiv:2509.09498.
Questions for authors
- Can you demonstrate the CoMATE also works well on some other kind of environments? I understand the focus is on ARC-AGI-3 here but to demonstrate the usefulness of memory consolidation part do need diverse settings in my opinion.
- Can you discuss more on the contribution against prior works such as [1,2,3] in knowledge consolidation? More specifically, how would the authors ground the work against SEDM for evidence grounding in traces or environments?
[1] Zhang, Guibin, et al. "G-memory: Tracing hierarchical memory for multi-agent systems." Advances in Neural Information Processing Systems 38 (2026): 12988-13018. [2] Symbolica. From 0% to 36% on day 1 of ARC-AGI-3. Symbolica Blog, March 2026a. URL https://www.symbolica.ai/blog/arc-agi-3. [3] Xu, H., Hu, J., Zhang, K., Yu, L., Tang, Y., Song, X., ... & Shi, B. (2025). Sedm: Scalable self-evolving distributed memory for agents. arXiv preprint arXiv:2509.09498.
3 comments
Dear Reviewer KyUW, thank you again for your review. As the discussion period ends on November 18, we wanted to check whether our response addresses your concerns. We would be happy to clarify anything further. We would also be grateful if you could consider raising your score based on our response.
The response mostly addressed my concerns, and I have raised my score.
Response to Reviewer KyUW
Thank you for recognizing that "maintaining a shared pool of grounded knowledge of the game world" benefits the multiagent system, and for raising the question of whether shared knowledge "should be consolidated by another agent instead of trusting the agent that acts in the games."
1. Other environments. Following your suggestion, we added an environment and tested CoMATE on Jericho text adventure games, which differ from ARC-AGI-3 in both observation modality and task structure. In these human-written games (Hausknecht et al., 2020), the agent reads text, types commands, and scores by uncovering mechanisms hidden behind long chains of exploration. We keep our protocol: stages end at the next score gain, six agents start each stage from the winner's state, consolidation follows every round, and a stage all agents fail gets one more attempt and, if that also fails, is replayed from another agent's endpoint of the previous stage.
CoMATE passes the stage at which the baseline stops in 46 of the 56 games. On the 18 hardest games, whose reference walkthroughs exceed 300 commands, it raises the share from 16.4% to 27.0%.
2. Do the experiments support the claim? We agree this is the central question. Table 3 isolates it, with the backbone, agents, harness, budgets, and update points fixed:
Two new results strengthen this. First, claims their writers labeled CONFIRMED are factually supported only 76.3% of the time, versus 89.9% for CoMATE's entries. Second, published memory methods fall short of CoMATE under the same protocol, with each replacing only CoMATE's consolidation and retrieval and building memory as in its paper:
On the 12 games B6 does not clear, CoMATE clears 9, versus 3 for ReasoningBank and 1 for G-Memory.
3. Relation to SEDM, G-Memory, and Symbolica.
We hope these clarifications and results address your concerns; if so, we would be grateful if you would consider raising your score.