- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
-
OpenKage is a framework for consolidating knowledge or skills learned by the agents when running in Minecraft without model weight updates to improve the agent's performance at test time.
-
Its main contributions are 1) proposing a framework for knowledge sharing and transfer during test time to improve the agent's performance, and 2) using a specific group-level aggregator to consolidate textual experience into a shared knowledge graph to avoid the conflicts of context-specific skills. This is achieved by using a reflector to separate situated memories from transferable experience. The shared knowledge is sent back to the agent through a graph-based intent-aware retriever. 3) shows the cross-model and cross-agent transfer ability of learned skills
-
Validation: the paper mainly evaluates its approach against a planner-only baseline on Minecraft using 2 models. The result is that OpenKage gets better exploration while reducing death rates. The results change across different models.
Critical strengths
The most promising part of the paper is the separation of private history and shareable skills across agents. Nevertheless, more ablations should be added to demonstrate that the proposed mechanism is what really contributes to the success.
Critical weaknesses
-
The novelty of the proposed framework is a main concern. For instance, works such as G-memory [1] and [2] are already exploring similar approaches in terms of skill learning and memory consolidation. However, the paper fails to discuss the broader related works and does not differentiate itself from the literature well.
-
Beyond the novelty issue, the paper has a limited experimental setting. The current experiment only operates over Minecraft and does not propose fundamental algorithmic-level improvement in terms of training. If the main focus is on agent harness, then the paper should compare against a few more agent harness baselines to demonstrate the practical applicability of the proposed harness for multiagent systems. The ablation studies are insufficient to support the claims proposed in the paper. If the goal is to show that cross-trial or cross-agent experience sharing enables better test-time learning, then only testing on Minecraft is not enough.
[1] Zhang, Guibin, et al. "G-memory: Tracing hierarchical memory for multi-agent systems." Advances in Neural Information Processing Systems 38 (2026): 12988-13018.
[2] Wu, Shanglin, et al. "Scaling Teams or Scaling Time? Memory Enabled Lifelong Learning in LLM Multi-Agent Systems." arXiv preprint arXiv:2604.03295 (2026).
Questions for authors
- How would you position your work against the two related works I listed above?
- How to ensure the reproducibility of the performance gain from Minecraft across diverse tasks if we want to apply OpenKage to them? Have you tested on more benchmarks beyond Minecraft?
- How to control the randomness introduced in the experiment? What is the uncertainty in running the long-horizon tasks to demonstrate the usefulness of the proposed harness?
- Have you compared your framework against established single-agent skill consolidation and learning frameworks such as Voyager [1] or other general harnesses for multiagent skill consolidation over long-horizon tasks?
[1] Wang, Guanzhi, et al. "Voyager: An open-ended embodied agent with large language models." arXiv preprint arXiv:2305.16291 (2023).
8 comments
Thank you for the review and the two references.
Weakness 1 / Q1. Positioning against G-Memory [1] and "Scaling Teams or Scaling Time" (LLMA-Mem) [2]. We read both. Both are now in the Introduction and Related Work, and Table 11 (Appendix A.10) has a row for each.
- G-Memory manages multi-agent interaction through a three-tier graph hierarchy (insight, query, interaction); it retrieves related past task queries with their insights and trajectories, filtered by role. That hierarchy is keyed by the task queries the system receives. In our setting agents set their own goals, so the query level has no counterpart and the design transfers only with changes. It is evaluated on benchmarks with assigned tasks.
- LLMA-Mem [2] keeps episodic, procedural and transactive memory under local, shared or hybrid topologies, on assigned tasks. Its hybrid topology keeps episodes local and shares consolidated procedures. That design choice is shared with our work, and the paper credits it. OpenKAGE differs in four respects: the boundary is drawn by the kind of knowledge, enforced by access (the aggregator receives only experience-layer records); the shared object is a graph whose applicability conditions travel with the text, while executable skills are shared verbatim and remain maintained by their author; retrieval returns subgraphs; and agents set their own goals.
G-Memory and LLMA-Mem are outside our comparison set: their memory structures build on task queries or assigned tasks, so a port would be our own re-implementation. Our two nearest controlled stand-ins are Raw (shared memory without consolidation) and −S (private structured memory). A head-to-head comparison with either system is future work.
Weakness 2: limited setting, training-level algorithmic change. OpenKAGE is a framework that drives lifelong learning and collective evolution of agents: capabilities grow through structured memory, consolidation and cross-agent sharing of experience, so the algorithmic contribution of this paper lies at the level of knowledge and memory mechanisms; carrying learning further into model parameters is a complementary direction for future work. Evaluation is in Minecraft, as the Limitations state; other environments and benchmarks are future work. Within Minecraft, our evidence is: four backbones, 10 runs per cell, three baselines (planner-only, Voyager, Echo), seven ablation conditions with 10 runs each, and one cross-backbone transfer experiment.
Ablations insufficient. Partly agreed. The ablations do reflect that the components matter jointly: every ablated condition falls short of Full on the deepest milestones (obsidian in 0 to 2 of 30 agents vs 6) while keeping deaths well below planner-only (11.6 to 17.6 vs 27.9 per agent). Apportioning the effect to individual components needs more runs: deep milestones are rare even under Full, and −S, −R, −E and Raw stay close to Full in Deaths, Repeat and XP/life relative to the run-to-run spread. Ablation claims that rested on single runs are removed. The best-supported statement remains the combined stack vs planner-only: Deaths 16.2 vs 27.9 per agent, repeated deaths 7.4 vs 15.1, and obsidian in 13/40 vs 0/40 runs pooled over four backbones.
Q2. Reproducibility across tasks / other benchmarks. Our tests are in Minecraft, on world seed 7. Within Minecraft, reproducibility of the gain is as follows: the survival effect repeats on four backbones (mean Deaths lower on all four). The exploration effect appears on every backbone (obsidian in 4/10, 6/10, 1/10, 2/10 OpenKAGE runs vs 0/10 for every planner-only cell; its size differs by backbone). For other domains the framework reserves a domain-extension interface: each task domain is a plug-in package implementing a task-domain protocol, while the framework core is written once against that protocol. The code for the framework, baselines and metrics is released; we leave instantiating this interface on other tasks and benchmarks to future work.
Q3. Randomness and uncertainty in long-horizon runs. Appendix B.1 ("Sources of randomness"): world seed, spawn point, difficulty, personas, prompts and hyperparameters are fixed; LLM sampling (API-internal seeds), the edge sampling of relational expansion (unseeded generator) and game dynamics (mob spawns, combat, asynchronous timing) vary between runs. A run's number records launch order only. We therefore report the run-to-run spread directly: 10 runs per cell, all runs listed in Tables 13 and 15, Fig. 9 showing each run, the run as the unit of counting (agents of a run share one graph). Example: on the default backbone, deaths per agent in the ten OpenKAGE runs range from 9.7 to 26.0, against 19.0 to 41.3 for planner-only, and every OpenKAGE run is below the planner-only median. On long horizons: all main runs are 30 game days; the 60-day and group-size runs (B.6) are single runs and labelled as such.
Q4. Comparison with Voyager and other harnesses. Voyager: yes. We re-implemented Voyager (curriculum, iterative code generation with execution feedback, self-verification, per-agent skill library) under the same server, world, Mineflayer API and horizon, with its high-level control functions removed and three isolated agents per group. Voyager's high-level control functions are hand-written capability primitives (mineBlock, craftItem, smeltItem, killMob) that hand the agent, at step one, composite abilities our protocol treats as the outcome to be learned. Keeping them would, under our protocol, give the baseline a large advantage unrelated to the mechanism being compared, akin to playing with x-ray vision and cheats enabled: mineBlock scans every matching block within 32 blocks through solid rock in one sweep and then handles pathfinding, digging and item pickup end to end, so an agent can mine before it has learned anything. The comparison would measure pre-supplied primitives rather than accumulated knowledge (A.3, A.10). Both external baselines therefore act through the same bare Mineflayer API as OpenKAGE, so that what differs is the mechanism turning the goal into tasks and experience (B.3). Because the hard-coded first Voyager task can fail repeatedly under this API. Echo (Li et al., 2026): ported mechanism by mechanism, because its released code lacks its runtime. On the default backbone, per agent: Deaths 228.9 (Voyager), 73.6 (Echo), 27.9 (planner-only) vs 16.2 (OpenKAGE); diamond-tier tools 1/30, 26/30, 16/30 vs 24/30 agents; obsidian and the Nether appear only under OpenKAGE. Both external baselines are mechanism-level re-implementations under one protocol.
Presentation. We rewrote the Limitations so that the scope is explicit (Minecraft, one world seed in the main experiments, 30-day horizon) and added the appendix tables listed above; the paper presents OpenKAGE as a framework that drives lifelong learning and collective evolution of agents. If specific passages remain hard to follow, please point us to them and we will revise.
Experiment tables of the revised paper
We had been working on a larger re-run since October, and the revised paper reports it: every cell below has 10 independent group runs of 3 agents (30 agents per cell), on world seed 7 and 30 game days. The re-run added the Voyager and Echo comparisons, three further backbones and the Raw ablation condition. Below are the three experiment tables of the main text (Tables 1–3) so that all of them can be read in one place; per-run values are in Tables 13 and 15 of the appendix, and everything that changed relative to the initial submission is marked in blue in the PDF.
Metrics. Deaths, Repeat (deaths repeating an earlier one of the same agent, same cause within 16 blocks) and XP/life are per-agent means; lower is better for Deaths and Repeat, higher for XP/life. Tool era gives the highest tier reached (dmd. = diamond), the mean day of first attainment and the number of agents out of 30. Obsidian and Nether give the mean first-attainment day and the number of agents out of 30; "—" means that no agent attained it.
Table 1. Comparison with baselines on DeepSeek-V4.1-Flash.
Table 2. OpenKAGE and its planner-only control across backbones.
Table 3. Ablations on DeepSeek-V4.1-Flash. Conditions are named by what they remove: −S sharing, −R subgraph retrieval, −M the memory layer, −E the experience layer, B all graphs (the planner-only control); Raw keeps every component but shares experience records verbatim (Appendix B.5). Full is the OpenKAGE row of Table 1.