acceptodds
Under review as a conference paper at ICLR 2027

MineMem: A Benchmark for Experience Transfer and Updating in Interactive 3D Worlds

10 views

Abstract

Vision-language model (VLM) agents can interpret visual observations and follow language instructions, but long-term tasks also require them to reuse past experience and revise it when the environment changes. Existing memory benchmarks often measure task success without revealing whether an agent applies reusable experience or merely recalls facts from a demonstration. To bridge this gap, we introduce MineMem, a benchmark built on the open-source voxel engine Luanti for evaluating how VLM agents transfer and update experience in interactive 3D worlds. Agents acquire experience from human demonstrations and then perform autonomous tasks under controlled environment transformations. MineMem evaluates five capabilities: spatial reasoning, object-state tracking, temporal event reasoning, planning, and memory updating. Its procedurally extensible templates vary demonstration length, event count, action-dependency depth, spatial displacement, object-state changes, and conflicting evidence, enabling controlled tests of memory horizon and transfer difficulty. We compare complete memory mechanisms with no-history and fact-injection controls, measuring task success, verified progress, and resource use. Contrasts across capabilities and environment transformations are designed to reveal when retained experience supports transfer and when stale or conflicting experience causes negative transfer. The benchmark code will be released upon acceptance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Memory, Experience Transfer, Long-Horizon Reasoning, Interactive 3D Environments, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_36MpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MineMem tests transfer from human demonstrations to changed tasks and environments in two interactive voxel-world domains. Fifty templates cover five memory-related capabilities. Six adapted memory approaches are compared with no-history and fact-injection controls using physical progress, belief probes and costs. Physical milestones start from prerequisite-complete states. Results show capability-dependent benefits and negative transfer when historical information conflicts with current evidence.

Critical strengths

S1. The transfer controls are useful. Same-instance no-history baselines and environment, goal and joint shifts separate the value of demonstration experience from task difficulty (Sections 3-4). Fact injection exposes cases where access to past information helps recall but harms adaptation.

S2. The scoring distinguishes knowledge from action. Belief answers do not enter the physical score. Simulator checks verify progress, supplied prerequisites earn no credit, and memory updates are discarded between attempts (Section 3.3; Appendix E). These choices support interpretable component diagnostics.

Critical weaknesses

W1. The empirical sample cannot be reconstructed. Fifty templates do not identify the instantiated tasks, demonstrations or repeated attempts behind each mean. Evaluated counts and aggregation weights are missing. Figures 5(a), 6(b) and 13 show error bars without defining their sampling unit or statistical meaning. This prevents assessment of the stability of the rankings and negative-transfer contrasts (Section 4.2; Appendix I).

W2. The scoring unit and memory lifecycle remain ambiguous. Appendix E.1 scores milestones from separate prerequisite-complete starts; Section 4.2 describes completing all milestones within a sixty-turn attempt. Whether budget and memory reset between milestones is unspecified. Consequently, the relationship between strict success, within-episode adaptation and the per-episode costs in Table 6 cannot be determined.

Assessment. I recommend weak rejection (2/4). The benchmark design is promising, but the reported comparisons need an auditable sample and an unambiguous evaluation unit.

Questions for authors

Q1 (W1). How many task instances and repeated attempts contribute to each main figure, and how are they weighted? Please define the error bars and provide the existing evaluated inventory or a compact count table; the template count is not the evaluation sample size.

Q2 (W2). At each milestone restart, does the agent receive a new sixty-turn budget and freshly initialized memory, or does either carry over? Please trace one multi-milestone task through the resets and show how its success and Table 6 cost are aggregated.

2 comments

We thank the reviewer for the careful review, for the transfer controls, and for the separation of knowledge from action. In response to these comments we checked the manuscript, the source code, and the dataset.

1. Counts, weights, and error bars (W1/Q1).

A rollout repeats one instantiated task. The instance and its demonstration are held fixed across its six rollouts, which are averaged within the instance before aggregation. Counts are per memory mechanism, and each row gives the sample size behind one plotted bar or panel.

FigureAggregation dimensionTemplatesInstantiated tasksRollouts
Figure 2Capability1090540
Figure 5(a)Mechanism, conflict-injected subset1090540
Figure 6(b)Mechanism504502,700
Figure 13Mechanism, per decision model1090540

Fifty templates cover five capabilities in two domains, five per domain–capability group. Each is instantiated at three randomly drawn procedural seeds and evaluated under the three transfer conditions of Section 4.2, giving 50 × 3 × 3 = 450 instantiated tasks, six rollouts each, 450 × 6 = 2,700 rollouts in total. Figure 2 reports the five capabilities one by one, each bar pooling that capability across the two domains (ten templates, 90 tasks, 540 rollouts). Figure 13 uses a fixed ten-template subset computed per decision model, since eight memory mechanisms under three decision models would multiply the attempts several times over, so each panel holds 90 tasks and 540 rollouts per method. Figure 5(a) is computed on the conflict-injected subset of the cohort, namely the ten memory-updating templates (BL1–BL5 and BN1–BN5) under the same three seeds and three transfer conditions, so each mechanism contributes 90 tasks and 540 rollouts to it. The demonstrations come from 128 human-recorded sessions, 56.2 hours in total. We have already documented in the manuscript the per-attempt artifact layout, in which every attempt records its instance metadata, model identifiers and milestone scores, and we have added the released per-instance inventory, so these denominators can be checked directly.

Weights are equal at each level and the instance mean is taken last. Figure 2 reports each capability separately, Figure 6(b) averages over the five capabilities, Figure 13 compares each mechanism with the no-history control run by the same decision model and is computed separately within each model rather than pooled across models, and Figure 5(a) weights the instances of its conflict condition equally. The error bars in Figure 5(a) are 95% bootstrap confidence intervals for the paired transfer contrasts. The resampling unit is the source demonstration recording, so instances that share a recording are not treated as independent, and each resample recomputes the aggregation of the figure it belongs to.

2. Milestone resets and cost aggregation (W2/Q2).

Each task runs in one continuous attempt. The environment and memory are initialized once, one sixty-decision-turn budget covers the whole task, and the milestones are scoring components of that same trajectory rather than independent attempts. Neither the environment nor the memory resets between milestones, and no milestone receives a budget of its own. We have already clarified this protocol and revised the appendix so that both descriptions refer to one evaluation unit, and it will be reflected in the submission.

As the reviewer asked for a concrete trace, the next comment walks one multi-milestone task (PL1) through the protocol step by step under the single continuous attempt stated above.

Over a long time, we devoted substantial effort to this benchmark and its data, hoping to give the field a reference standard. We thank the reviewer again for the careful reading that made our work more complete, understandable, and reusable.

Picking up W2/Q2 from our response above, we trace one multi-milestone task through the resets and show how its success and Table 6 cost are aggregated. The protocol is the single continuous attempt described there. What follows is the requested trace.

PL1 is a Logistica planning template whose goal is to construct a new composite network. Its applicable milestones come from the five logistics checks, namely component placement, topological connectivity, network membership, routing configuration, and item transfer, with base weights 0.20, 0.15, 0.25, 0.20, and 0.20, normalized within the task. A milestone the task does not require is omitted rather than credited as completed.

  • The environment is instantiated from the matched task configuration and the memory from the demonstration package, the single initialization of the evaluation. The agent places the required conduit, machine, and storage components, and placement is verified from the simulator's node-placement records at the end of the attempt, not at a milestone boundary, so ∣Nplaced∩Nreq∣/∣Nreq∣|N_{\text{placed}} \cap N_{\text{req}}| / |N_{\text{req}}| scores the state reached.
  • It then continues without re-initialization, establishing six-connected voxel connectivity, registering the active nodes with the controller through the network heartbeat probes, writing the routing whitelist filters and request limits, and transferring the inventory item to its destination where the goal requires it.
  • Strict success (Appendix E.1) is Φ(τ)=1\Phi(\tau) = 1 only if every applicable milestone passes, namely the placement check, the connectivity check, the membership probe, the routing-configuration check, and the transfer check, and no validity gate is violated within the same sixty-decision-turn budget.
  • Table 6's input-token and model-call columns are cumulative over one attempt, including the memory preparation and consultation calls that precede and interleave with the decision turns. This task therefore contributes to its approach's row rather than a row of its own, and no milestone contributes a separate cost line, since costs are never summed across milestones.

An earlier failure does not discard the later milestones, which are scored against the state actually reached. That is why the continuous progress score and strict success answer different questions. Re-executing a prerequisite is paid for inside the shared budget rather than from a fresh allowance, so the comparison across mechanisms is unaffected by how the budget is divided.

Some templates, such as PL2, begin from an interrupted or partially completed state by design. This is the task's single initial state, not a reset between milestones. The agent continues that chain within the same budget and the same initialized memory, and milestones whose prerequisites were already satisfied are graded on the pre-existing state.

Reviewer_MokPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MineMem evaluates how embodied agents transfer experience from human demonstrations and revise it when the environment changes. Built in Luanti, it contains 50 task templates across Logistica and NodeCore, covering spatial reasoning, object-state tracking, temporal reasoning, planning, and memory updating. Agents receive a common multimodal demonstration package and are evaluated under environment, task, or joint shifts. The benchmark combines simulator-verified behavioral scores with elicited belief probes and compares six memory mechanisms against no-history and fact-injection controls. The main contribution is a diagnostic evaluation framework rather than a new memory architecture. Its results suggest that historical information can support some capabilities while impairing performance under conflicting evidence, and that accurate probe responses do not necessarily translate into effective action.

Critical strengths
  1. The benchmark offers a structured way to evaluate whether prior experience remains useful under changes to the environment or task.
    Sections 3.1–3.2 and Appendix A.2 introduce spatial relocation, object-state changes, interrupted sequences, altered dependencies, and conflicting cues. Evaluating environment, task, and joint shifts separately makes the transfer conditions easier to interpret. Section 4.2 also explicitly resets temporary memory between attempts, clearly separating within-attempt adaptation from cumulative learning across evaluation episodes.

  2. The controls demonstrate that historical information can help some capabilities while impairing others.
    Figure 2 shows that fact injection improves object-state and temporal performance over no history by 30.9 and 24.0 percentage points, respectively, but lowers belief-updating performance by 23.6 points. This contrast illustrates the value of capability-specific reporting and challenges the assumption that additional history is consistently beneficial. The discussion appropriately distinguishes observed negative transfer from evidence establishing its cause.

  3. Simulator-grounded checks and separate belief probes provide complementary views of agent performance.
    Appendix E defines physical milestone checks, Appendix A.3 validates solvability through oracle execution using the public action interface, and Appendix I.3 specifies telemetry and model-trace artifacts for auditing. Figure 4 examines belief accuracy alongside behavioral transfer without combining them into a single metric. Treating their relationship as descriptive allows the benchmark to expose gaps between reported beliefs and execution while avoiding unsupported causal claims.

Critical weaknesses
  1. Incomplete reporting of the evaluated data and uncertainty limits confidence in the empirical comparisons.
    The paper describes 50 templates, but Sections 4.2 and Appendices B and I do not clearly specify the numbers of demonstrations, generated instances, procedural seeds, or independent agent runs underlying each figure. Evaluated history lengths and difficulty settings are also unclear, and the error bars in Figure 5(a) are not defined. These details are necessary to assess coverage and reliability, particularly when observations share templates or demonstrations. Reporting sample counts, aggregation procedures, and uncertainty estimates would make the comparisons easier to interpret.

  2. The main results and decision-model sensitivity analysis show differences that need reconciliation.
    Figure 6(b) reports negative ReAct gain relative to no history and ranks 3D-Mem above EmbodiSkill. However, the GPT-5.6-luna panel in Figure 13 reports positive ReAct gain and reverses that ordering. Different task subsets, aggregation weights, or configurations could explain these patterns, but the manuscript does not identify such differences. Providing the numerical results and evaluation settings for both figures would clarify whether their conclusions are consistent.

  3. The diagnostic scoring procedures are insufficiently defined to support reproducible failure analysis.
    Appendices E.2–E.3 describe awareness, memory-decisive reasoning, and belief accuracy without fully specifying how they are measured. Figure 5 likewise needs explicit failure-label criteria, rules for overlapping categories, and information about automatic or human annotation. It is also unclear whether belief-probe questions and responses remain in the context used for subsequent actions. These details matter because the diagnostics underpin the paper’s interpretation of memory failures, and probes could influence the behavior they are intended to assess.

  4. The baseline descriptions and controls do not fully support the interpretation of what experience is transferred.
    Appendix D provides high-level mechanisms and selected prompts, but leaves preprocessing, retrieval, memory construction, reflection triggers, and skill execution insufficiently specified. The fact-injection control also differs from complete mechanisms in information organization and context limits, so its results do not isolate the benefit of procedural abstraction. The whole-system comparisons remain informative, but stronger representation-level conclusions require clearer implementations or a matched-content control. Selectively perturbing demonstrations would further test whether gains depend on acquired experience rather than current observations and exploration.

Questions for authors
  1. How many demonstrations, generated instances, procedural seeds, and independent agent runs underlie each result? Please report the evaluated history lengths, difficulty settings, aggregation weights, and error-bar definitions, including whether uncertainty accounts for instances sharing templates or demonstrations.

  2. Do Figures 6(b) and 13 use different task subsets, aggregation rules, or configurations? Please reconcile the differing GPT-5.6-luna results for ReAct and the reversed ordering of 3D-Mem and EmbodiSkill, preferably by providing the underlying numerical results.

  3. How are belief accuracy, workflow diagnostics, and failure categories scored? Please clarify annotation procedures, the handling of overlapping failure categories, and whether probe questions and responses are excluded from subsequent action context.

  4. Could you specify the baseline configurations, including preprocessing, retrieval, reflection, geometric inputs, and skill execution? Are tool-call sequence lengths bounded consistently across methods? A targeted matched-content or demonstration-perturbation control would also help establish which gains depend on acquired experience.

7 comments

We sincerely thank the reviewer for the careful review. In response to these comments, we checked the manuscript, source code, and dataset, and provide our response below.

1. Evaluation inventory and uncertainty (Question 1 / Weakness 1).

For the main-text evaluation, the fifty templates cover five capabilities in two domains, with five templates per domain–capability group. Each template is instantiated at three randomly drawn procedural seeds and evaluated under the three transfer conditions of Section 4.2, giving 50 × 3 × 3 = 450 instantiated tasks per memory mechanism and 450 × 6 = 2,700 rollouts in total. Figure 13 uses a fixed ten-template subset, computed separately for each decision model, because covering eight memory mechanisms under three decision models would multiply the number of attempts several times over. Each panel therefore contains 90 tasks and 540 rollouts per method. The demonstrations come from 128 human-recorded sessions totaling 56.2 hours. We have already documented in the manuscript the per-attempt artifact layout, in which every attempt records its instance metadata, model identifiers and milestone scores, and we have added the released per-instance inventory, so these denominators can be checked directly. The counts are given in the table below. Each row lists the number of templates, instantiated tasks, and agent rollouts it pools, per memory mechanism, together with the aggregation applied. A rollout is a repeated attempt on one instantiated task, with the task instance and its demonstration held fixed across the rollouts, so the six rollouts of an instance are averaged within the instance before aggregation.

FigureDimensionTemplatesInstantiated tasksRollouts
Figure 2Capability1090540
Figure 3Domain and relation2575450
Figure 5(a)Mechanism, conflict-injected subset1090540
Figure 6(b)Mechanism504502,700
Figure 13Mechanism, per decision model1090540

As for difficulty, the benchmark uses one fixed configuration per task in the current evaluation. Because raw template configurations produced markedly different no-history completion rates, we generated candidate configurations and retained, for each template, the one whose no-history score was closest to the common cross-template reference level. This calibration uses only the no-history control and is intended to reduce baseline-difficulty differences in pooled comparisons. The selected configuration is fixed, is not exposed to agents, and cannot be selected at evaluation time. We have already added this explanation to the appendix, and the per-attempt artifact layout records the evaluated demonstration segments for each task so that the exact history presented to each mechanism can be reconstructed.

The error bars in Figure 5(a) are 95% bootstrap confidence intervals for the paired transfer contrasts. The six rollouts of an instance are averaged within the instance first, and the resampling unit is the source demonstration recording, so instances that share a recording are not treated as independent, and each resample recomputes the aggregation of the figure it belongs to.

2. Reconciling Figures 6(b) and 13 (Question 2 / Weakness 2).

The main-text figures other than Figure 5 pool the full fifty-template suite; Figure 5(a) is computed on the conflict-injected subset of ten memory-updating templates (90 tasks and 540 rollouts per mechanism), and Figure 13 in the appendix uses a fixed ten-template subset and is computed separately per decision model. Figure 13 has to cover eight memory mechanisms under three decision models, each with its own inference cost, so running the full suite for every model would multiply the number of attempts several times over. We therefore reduced the template count for this supplementary comparison, while keeping the subset fixed across models and matched to the same protocols as the main figures, so that the purpose of the figure remains to show how the benchmark behaves across decision models rather than to re-estimate the main results. So the ranking change the reviewer describes is therefore not a contradiction between the two figures. They pool different task populations and so summarize different estimands. We do not present either figure as a unique overall ranking, and we have stated the task population behind each figure in the corresponding caption and text so that the estimand behind each summary is explicit at the point of use.

3. Diagnostic scoring (Question 3 / Weakness 3).

Physical-task scores and workflow diagnostics are computed from recorded actions and environment telemetry. For each attempt, the benchmark’s stage scorers compute awareness, memory-decisive reasoning, and physical execution as graded progress values in [0,1] over each applicable stage check, and these values are then averaged over attempts. They do not enter either the behavior score or the strict-success metric. The conflict failure labels in Figure 5(b) come from the same scoring pass. A failed episode is attributed to the earliest broken stage in the task’s dependency chain, while stale-persistence and recovery terms are graded from the action trace. For example, recovery credit decreases with the number of state-changing steps before the correcting action. Each episode therefore contributes exactly one root-cause class, making the labels mutually exclusive by construction.

Belief-probe accuracy is different from these telemetry-derived readouts because the belief content itself is model-reported. At pre-registered checkpoints, the implementation saves the agent-visible snapshot without invoking a readout model. After the attempt has ended, a frozen readout model (temperature 0, no tools, JSON-only) is queried once per snapshot. Its structured answer is scored automatically against evaluator-held ground truth, without free-form human annotation. Since the readout is performed only after acting has finished, the probe questions and responses cannot enter the subsequent action context of that attempt. Probe accuracy is reported alongside, never inside, the behavioral score, and the answer keys remain quarantined from the agent’s prompt and sensory stream (Appendix A.4).

The conflict panel in Figure 5 should be understood in that context as a conditional analysis of the main cohort, not as a separate experiment. It pools the main-cohort instances whose declared transformations include conflicting-cue injection (a construction-time property of each instance), namely the ten memory-updating templates (BL1–BL5 and BN1–BN5) across the two domains, and recomputes paired contrasts and failure-mode labels on that subset of 90 instantiated tasks and 540 rollouts per mechanism without collecting additional rollouts; the per-interval denominators are reported with the released per-instance inventory.

4. Baseline configurations and controls (Question 4 / Weakness 4).

We would first note what MineMem is intended to contribute. The benchmark provides a common platform on which distinct memory mechanisms can be evaluated under matched conditions, namely the same demonstration package, the same task instances, the same observations, action interface, decision model, and action budget. How each mechanism encodes, retrieves, constructs, and updates its own memory is the subject of the original work of that mechanism rather than of this benchmark, and reproducing those designs faithfully is exactly why we adopt the published architectures unchanged. Our contribution is to make those architectures comparable on a shared set of embodied transfer and updating tasks, and to report what the comparison reveals about when retained experience helps and when it becomes a liability. We therefore see the reviewer's request as a matter of where this detail belongs rather than a shortcoming of the comparison itself, and we would welcome the reviewer's guidance on which baseline details would be most useful to expand.

Within that scope, all six memory approaches are implemented as first-class memory backends behind one shared action space, one observation interface, and one per-attempt decision-turn budget, and their complete configurations will ship with the open-source release so that the comparison can be inspected and reproduced. The mechanisms that we adopt unchanged are used in their published form. The one exception is 3D-Mem, whose spatial stages are adapted behind a replaceable RGB geometry estimator in place of the original offline geometric reconstruction, while the anchor and frontier snapshot structure is retained. Demonstration entries without geometric support never enter the strict spatial decision path, so this adaptation degrades explicitly rather than silently.

For fact injection, the facts are selected with respect to the downstream task goal. Demonstration records are ranked by goal similarity, and atomic facts are extracted from the selected records, with the injected summary bounded relative to the decision model's input capacity. The control receives the same goal and action budget as every other condition, and goal-conditioned selection does not expose hidden test-environment state. A decision turn may contain a valid sequence of tool calls, and that sequence is charged as one turn under the shared protocol. Reported costs consistently include memory preparation and consultation calls in addition to decision turns (Section 4.2). We agree that matched-content and demonstration-perturbation controls would provide a sharper test of representation-level explanations, and we see them as valuable extensions for future work.

For more than a year, we have devoted substantial time and effort to designing and implementing this benchmark and collecting its data, with the hope of providing a reference standard for the field. Once again, we sincerely thank the reviewer for their effort, which has helped make our work more complete, understandable, and reusable.

Reviewer_UsS8OpenReview ↗
Rating
4(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduced a memory benchmark for VLM agents, motivated by long-horizon tasks minecraft. It tests whether an agent can reuse experience from a human demonstration after the world or the task changes. It has 50 task templates covering five abilities, from spatial reasoning to updating outdated memories. Six memory methods are compared with a no-memory control and a fact-injection control. Each "milestone" (defined in paper) is scored from the simulator's state, starting with the earlier milestones already done. The paper's main contribution is provided a controlled setting for testing VLM agents's ability to use their memory.

Critical strengths
  1. The benchmarks is testing a crucial ability of VLM agents. It also has multiple controls to make the answer concrete. It separates having access to past facts from using past experience after the world changes. The fact-injection control shows this clearly: compared with no memory, it gains 30.9 points on object-state tasks but loses 23.6 points on memory-updating tasks. The reason I lean toward acceptance is that the controlled setting serves very well for testing VLM's memory for these visual related tasks.

  2. Progress is scored from the simulator, not from what the agent says. Credit is given only for verified changes in the world, every task is checked to be solvable, and shortcuts such as teleporting are blocked. This makes the results more trustworthy, because the results are not hacked by agents' own words.

Critical weaknesses
  1. Some results are hard to reconcile and verify. Figure 6(b) and the GPT-5.6 Luna panel of Figure 13 seem to show the same gains over no memory, yet they differ: for example, ReAct is below no memory in one and about 0.08 above it in the other. The paper also does not report how many instances or runs stand behind each score.

  2. Two of the benchmark's main claims are not tested. (1) Each milestone starts with the earlier steps already done, so no agent completes a long task on its own. (2) The templates vary demonstration length and dependency depth, but no result shows how performance changes with them. This also keeps my rating at weak accept, but if the author provide more details and ablation on this, I am willing to raise my score.

Questions for authors
  1. The paper does not say how many task instances, seeds or repeated runs are behind each reported score. How many instances and runs does each score in Figures 2 and 3 average over?

  2. Figure 6(b) and the GPT-5.6-luna panel of Figure 13 both report gains over no memory for the same decision model, but they disagree. ReAct is slightly below no memory in Figure 6(b) but about 0.08 above it in Figure 13, and 3D-Mem leads EmbodiSkill in one figure but lower than it in the other. Which tasks and weights does each figure average over?

  3. Each milestone is scored from a state where the earlier steps are already done, so no agent has to finish a whole task by itself. On the planning tasks, do the comparisons between memory methods still hold when each agent must complete the task in one uninterrupted run?

3 comments

We sincerely thank the reviewer for the careful review. In response to these comments, we checked the manuscript, source code, and dataset, and provide our response below.

1. Sample counts (Question 1).

It may help to begin with the observations shared by Figures 2 and 3. The fifty templates cover five capabilities in two domains, with five templates per domain–capability group. In the main-text evaluation with GPT-5.6 Luna, every template is instantiated at three randomly drawn procedural seeds and evaluated under the three transfer conditions of Section 4.2, giving 50 × 3 × 3 = 450 instantiated tasks per memory mechanism, with six independent rollouts per instance and 450 × 6 = 2,700 rollouts. In Figure 2, each capability score pools the two domain families of that capability, namely ten templates × 3 seeds × 3 conditions = 90 tasks and 540 rollouts. In Figure 3, each cell pools the five capabilities of that domain, namely 25 templates × 3 seeds = 75 tasks and 450 rollouts. Rollouts are averaged within an instance first, so repeated attempts never receive extra weight. Figure 3 is thus a regrouping of the same observations as Figure 2, rather than an independent replication.

2. Reconciling Figures 6(b) and 13 (Weakness 1 / Question 2).

The main-text figures, including Figure 6(b), pool the full fifty-template suite, whereas Figure 13 in the appendix uses a fixed ten-template subset and is computed separately per decision model with matched controls. Figure 5(a) alone is computed on the conflict-injected subset of the ten memory-updating templates, 90 tasks and 540 rollouts per mechanism. Since Figure 13 covers eight memory mechanisms under each of three decision models, running the full suite for every model would multiply the number of attempts several times over, so we reduced the template count for this supplementary comparison and kept that subset fixed across models. The two figures therefore average over different task populations, which is what the reviewer observes rather than a contradiction between them. Within each figure the weights are equal at each level of the grouping, and the six rollouts of an instance are averaged before aggregation, so the instance is the unit of aggregation. We have stated the task population behind each figure in the corresponding caption and text, and qualified our overall-ranking statements accordingly, so that the estimand behind each summary is explicit at the point of use.

3. Uninterrupted execution (Weakness 2(1) / Question 3).

We thank the reviewer for pointing out the ambiguity in our description of the evaluation protocol. We are sorry that our writing was unclear and led to this misunderstanding. For the planning templates, the environment and memory are initialized once at the beginning of the task, and the agent then executes the dependency chain continuously within one action budget. There is no reset between milestones. Milestones are separate scoring components applied to the same continuous trajectory, rather than independent attempts from prerequisite-complete states. A task such as PL2 may begin from an interrupted or partially completed state by design, but this is the task’s single initial state, not a reset between milestones. The planning comparisons therefore already evaluate uninterrupted execution of each planning task. We have already revised the manuscript to make this distinction clear, and it will be reflected in the submission.

4. Demonstration length and dependency depth (Weakness 2(2)).*

We agree with the reviewer that completing long tasks end-to-end remains an important problem for the field, and that it is already covered by a substantial number of benchmarks that evaluate long-horizon ability, such as BEHAVIOR-1K, RoboMemArena, and MineNPC-Task. For this reason we do not claim that our benchmark is designed for long-horizon task completion. Our primary object of evaluation remains the transfer and updating of experience, which these suites do not measure. We nevertheless agree with the reviewer that demonstrating how task performance varies with demonstration length and dependency depth is valuable research, and we are pushing the corresponding ablation experiments forward with all the resources we have. We will do our best to complete them before the deadline and report the results to the reviewer as soon as they are available.

For a long time, we have devoted substantial time and effort to designing and implementing this benchmark and collecting its data, with the hope of providing a reference standard for the field. Once again, we sincerely thank the reviewer for their effort, which has helped make our work more complete, understandable, and reusable.

Thank you for the clear response. The count table and the explanation of the different task sets in Figures 6(b) and 13 answer my questions. Clarifying that each task runs as one continuous attempt also resolves my concern about uninterrupted runs. Please make both points clear in the final version. I look forward to the results on demonstration length and dependency depth.

I think this work is a great contribution to the field, and the authors addressed my concerns. I have raised my score to 4.

We sincerely thank the reviewer for the careful follow-up and for the recognition of our work. We have revised the manuscript to make both points clear, and the revised version has been uploaded.

The reviewer's comments have substantially improved the clarity and completeness of the paper. We are truly grateful for the time and effort devoted to this review, and for helping us make this work more complete and more valuable to the community.

Reviewer_fVKaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MineMem is a benchmark for evaluating experience transfer and memory updating in interactive 3D worlds. Agents receive a common multimodal human-demonstration package and subsequently perform tasks under changes to the environment, objective, or both. The benchmark contains 40 core templates and 10 supplementary tasks across two domains and five capability categories. It compares six agent approaches with no-history and fact-injection controls, reporting simulator-verified milestone progress alongside elicited belief accuracy and diagnostic analyses.

Critical strengths

The benchmark targets a meaningful question about the validity of prior experience. MineMem explicitly changes environments, objectives, and conflicting evidence after demonstration acquisition. This makes the applicability of historical experience an evaluation variable, rather than treating access to more history as uniformly beneficial. Experience collection followed by embodied interaction has precedents, including Finding Dory (Yadav et al., 2026); MineMem’s useful incremental contribution is the combination of controlled transfer conditions, stale-information conflicts, and complementary diagnostic controls. The evaluation combines complementary evidence and reveals useful behavioral differences. Matched no-history and fact-injection controls, simulator-verified progress, separate belief probes, and qualitative traces provide several perspectives on memory use. Fact injection improves object-state and temporal scores by 30.9 and 24.0 percentage points, respectively, but reduces updating performance by 23.6 points relative to no history. These observations motivate evaluating the benefits of retained information separately from the costs of stale information. The authors appropriately acknowledge that belief–behavior disagreement does not establish a causal effect of updating. The benchmark construction includes useful engineering and reproducibility provisions. The two domains, parameterized templates, oracle-based solvability checks, action restrictions, and recorded evaluation-artifact specification provide a substantive foundation for benchmark reuse. These provisions are valuable, although their practical reproducibility still depends on complete implementation details and documentation of the actual evaluation instances.

Critical weaknesses

The experimental coverage, aggregation choices, and primary outcomes are incompletely reported. The paper does not clearly report the number of evaluated instances, allocation across templates and difficulty conditions, repeated runs, or aggregation weights. The 50-template count describes the task design rather than the experimental sample size. Figure 5(a) contains error bars without defining their statistical meaning or computation, and uncertainty is not reported for several comparisons involving small margins. Aggregation also changes the substantive ranking. Under the capability average represented by Figure 6(b), 3D-Mem’s mean gain over the no-history control exceeds EmbodiSkill’s by 9.2 percentage points. Under the domain-by-transformation aggregation consistent with Figure 13(a), EmbodiSkill instead exceeds 3D-Mem. The paper should explicitly identify the populations and weights behind these summaries and qualify conclusions about overall performance and cost trade-offs accordingly. Finally, §3.3 identifies strict success and continuous progress as primary outcomes, but the paper does not provide aggregate strict-success results alongside the progress scores. Individual qualitative examples do not substitute for this missing primary-outcome summary. The concrete baseline implementations are insufficiently specified for interpreting the comparisons. Appendix D.1 describes 3D-Mem largely as a metric-topological graph, whereas the original method centers on Memory Snapshots, Frontier Snapshots, co-visibility clustering, and retrieval prefiltering (Yang et al., CVPR 2025). The paper should explain which components were retained, replaced, or approximated. This is necessary to determine whether the findings apply to the original method or to a particular adaptation. Appendix D.2 claims to provide complete prompt architectures, but Figures 7–12 show common instructions, an action-schema excerpt, method-specific templates for CoALA, MemGPT, and 3D-Mem, and a belief-probe prompt. The corresponding prompts and memory operations for ReAct, ExpeL, and EmbodiSkill are not fully specified. Concrete instantiation is particularly important because the evaluated approaches span a conceptual architecture, a prompting paradigm, and implemented memory systems. The comparisons also use a common interaction budget rather than equal total inference expenditure: Table 6 reports approximately 1.1–3.1 million cumulative input tokens per attempt across conditions. Reporting these costs is useful, and unequal expenditure does not invalidate a complete-system comparison. However, differences in prompts, retrieval, updating, and inference expenditure prevent attributing performance differences to memory representation alone. The fact-injection extraction procedure and its concrete token or fact-count limit also remain unclear. The diagnostic readouts are not sufficiently operationalized, and some conclusions require narrower scope. Appendix E.2 describes awareness, memory-decisive reasoning, and physical execution conceptually, but does not provide sufficiently precise scoring rules, units, or denominators. The conflict failure categories similarly lack an explicit classification procedure. Belief-probe timing, answer evaluation, and whether probe responses enter subsequent decision context are also unspecified. These details are central because the diagnostic readouts support several of the paper’s main interpretations. The conditioned milestone protocol is a reasonable way to prevent early failures from obscuring later capabilities, and its limitations are explicitly disclosed. Nevertheless, success from prerequisite-complete states does not establish uninterrupted execution of a long dependency chain. Likewise, no-history outperforming memory-equipped systems under conflict is meaningful evidence of negative transfer, but does not by itself distinguish selective memory revision from disregarding historical information. Claims about long-horizon planning and memory updating should remain within these evidential boundaries.

Questions for authors

What are the evaluation populations and aggregation rules? Please report instance counts by template, difficulty, transformation, and model; repeated runs and randomization settings; and the weights used in each figure. Please confirm the apparent correspondence of Figure 6(b) to capability averages and Figure 13(a) to domain-by-transformation averages, and discuss the resulting changes in method ranking. Please also define the error bars in Figure 5(a). Can you provide aggregate strict-success results? Since §3.3 identifies strict success as a primary outcome, please report it alongside progress. Are uninterrupted rollout results available for representative multi-stage tasks, and how do they relate to conditioned milestone performance? Is the 60-turn budget applied separately to each stage attempt or jointly across a task? How was each baseline instantiated and adapted? Please explain the treatment of 3D-Mem’s defining components and provide the missing prompts and initialization, retrieval, and updating configurations. Please clarify how much demonstration history ReAct retains and whether memory preparation and consultation are included consistently in the reported costs. What information was available during fact selection? Was the downstream test goal available when the demonstration facts were selected? What extraction procedure, selector, and concrete token or fact-count limit were used? If selection is goal-conditioned, how does this compare with the information available during preparation and retrieval for the other systems? How are the diagnostic metrics computed? Please specify the rules, scoring units, denominators, and evaluation procedures for workflow diagnostics and conflict failure categories. For belief probes, please clarify the questions, timing, answer criteria, and whether the responses can influence subsequent actions. How were the human demonstrations collected? Please document the number and involvement of demonstrators, collection and validation procedures, demonstration lengths, and the consent and licensing arrangements applicable to the planned release.

3 comments

We sincerely thank the reviewer for the careful review. In response to these comments, we checked the manuscript, source code, and dataset, and provide our response below.

Apart from Figure 5, the main-text figures pool the full fifty-template suite, in which each of the five capabilities has five templates per domain; Figure 5(a) is computed on the conflict-injected subset of the ten memory-updating templates, 90 tasks and 540 rollouts per mechanism. Figure 13 in the appendix averages a fixed ten-template subset and is computed separately per decision model. Since Figure 13 covers eight memory mechanisms under each of three decision models, running the full suite for every model would multiply the number of attempts several times over, so we reduced the template count for this supplementary comparison and kept that subset fixed across models, so that the purpose of the figure remains what it is, to show how the benchmark behaves across decision models rather than to re-estimate the main results. In the main-text evaluation with GPT-5.6 Luna, every template is instantiated at three randomly drawn procedural seeds and evaluated under the three transfer conditions of Section 4.2, giving 50 × 3 × 3 = 450 instantiated tasks per memory mechanism and 450 × 6 = 2,700 rollouts, while each Figure 13 panel has 90 tasks and 540 rollouts per method.

The ranking change the reviewer describes is therefore not a contradiction between the two figures, and we do not present either figure as a unique overall ranking. We have stated the task population behind each figure in the corresponding caption and text so that the estimand behind each summary is explicit at the point of use.

The same qualification applies to the cost trade-off we discuss alongside these results. The gain comparison in Figure 13 is not a like-for-like restatement of the trade-off we report in the main text, and we will not present it as one. The cost accounting itself is unaffected. EmbodiSkill's lower cumulative input-token and model-call counts are properties of the mechanism and hold under either population.

As for difficulty, the benchmark uses one fixed configuration per task in the current evaluation. Because raw template configurations produced markedly different no-history completion rates, we generated candidate configurations and retained, for each template, the one whose no-history score was closest to the common cross-template reference level. This calibration uses only the no-history control and is intended to reduce baseline-difficulty differences in pooled comparisons. The selected configuration is fixed, is not exposed to agents, and cannot be selected at evaluation time. We have already added this explanation to the appendix, and the per-attempt artifact layout records the evaluated demonstration segments for each task so that the exact history presented to each mechanism can be reconstructed.

The error bars in Figure 5(a) are 95% bootstrap confidence intervals for the paired transfer contrasts. The six rollouts of an instance are averaged within the instance first, and the resampling unit is the source demonstration recording, so instances that share a recording are not treated as independent, and each resample recomputes the aggregation of the figure it belongs to.

On the evaluation unit and the sixty-turn budget, the reviewer's premise does not hold for this benchmark. The milestones are not independent attempts launched from prerequisite-complete states. Each task is evaluated in one continuous attempt. The environment and memory are initialized once, and all applicable milestones are scored as separate components of the same trajectory, so the planning comparisons already measure uninterrupted execution of a dependency chain within one budget. For each task, the sixty-decision-turn limit applies to one continuous attempt. Temporary attempt state is discarded when the attempt ends, and there is no reset between milestones. A task such as PL2 may begin from an interrupted or partially completed state by design, but that is the task's single initial condition rather than a reset between milestones, and a milestone whose prerequisites were already satisfied before the attempt is graded on that pre-existing state.

We would add that uninterrupted long-horizon execution is an important direction, addressed by existing benchmarks such as BEHAVIOR-1K[1], which evaluates long-horizon everyday activities, RoboMemArena[2], which includes sequential-execution tasks with an average trajectory length of over a thousand steps, and MineNPC-Task[3], which decomposes user tasks into sequential subtasks. Our benchmark addresses a related but different measurement goal. Those suites evaluate whether an agent can carry a task through to completion within one continuous rollout, whereas MineMem is designed so that an early-stage failure does not erase the evidence about later stages. We therefore report continuous progress rather than a single all-or-nothing outcome, and we deliberately do not isolate the uninterrupted-execution regime in the current comparison. We agree that separating the two regimes would be a valuable extension, and we plan to report uninterrupted whole-task execution for representative multi-stage tasks as a dedicated analysis rather than folding it into the current comparison. We have already revised the manuscript to make the evaluation unit explicit, and it will be reflected in the submission.

[1] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R. Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. In Proceedings of the 6th Conference on Robot Learning, PMLR 205, pages 80–93, 2023.

[2] Huashuo Lei, Wenxuan Song, Huarui Zhang, Jieyuan Pei, Jiayi Chen, Haodong Yan, Han Zhao, Pengxiang Ding, Zhipeng Zhang, Lida Huang, Donglin Wang, Yan Wang, and Haoang Li. RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark. arXiv preprint arXiv:2605.10921, 2026.

[3] Tamil Sudaravan Mohan Doss, Michael Xu, Sudha Rao, Andrew D. Wilson, and Balasaravanan Thoravi Kumaravel. MineNPC-Task: Task Suite for Memory-Aware Minecraft Agents. arXiv preprint arXiv:2601.05215, 2026.

For fact injection, the selection procedure is conditioned on the downstream test goal. Demonstration records are ranked by goal similarity, and the atomic facts extracted from the selected records are capped at 10% of the effective input capacity of a decision call. The control receives the same goal and action budget as every other condition. Goal-conditioned selection uses the task goal but does not expose hidden test-environment state.

Workflow diagnostics are deterministic telemetry computations. Awareness, memory-decisive reasoning, and physical execution are graded per-attempt stage progress values in [0,1], averaged over attempts, and never enter the behavior score. Conflict failure labels attribute each failed episode to the earliest broken stage in the dependency chain, with stale-persistence and recovery terms graded from the action trace, so the labels are mutually exclusive by construction.

Belief-probe accuracy instead evaluates structured, model-reported beliefs. The questions concern locations and connections for spatial reasoning, identity and current condition for object state, event order and provenance for temporal reasoning, prerequisites for planning, and the validity of earlier beliefs after conflicting evidence for updating. The implementation captures agent-visible snapshots at pre-registered checkpoints without invoking a readout model. After the attempt has ended, a frozen readout model (temperature 0, no tools, JSON-only) is queried once per snapshot, and each answer is scored automatically against evaluator-held ground truth. Consequently, probe questions and responses cannot enter subsequent action context within that attempt. Answer keys remain quarantined in evaluator memory (Appendix A.4). As in the conflict analysis of Figure 5, we report the failure-mode distribution and score contrasts together and will not treat a contrast alone as establishing the mechanism.

Finally, we appreciate the request for more information about the human demonstrations. Twenty-five demonstrators recorded sessions using our modified Luanti client as the data collector. All demonstrators were explicitly informed of the research purpose of the data and collected sessions under per-task collection instructions. After collection, the authors manually reviewed the recordings and removed invalid or erroneous material. The corpus used in the current evaluation comprises 128 sessions totaling 56.2 hours (24.9 Logistica, 31.2 NodeCore). Upon acceptance, we will open-source both the collector and the telemetry-driven scorer, with consent and licensing documentation accompanying the release.

For a long time, we have devoted substantial time and effort to designing and implementing this benchmark and collecting its data, with the hope of providing a reference standard for the field. Once again, we sincerely thank the reviewer for their effort, which has helped make our work more complete, understandable, and reusable.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your helpful reviews. We have replied to all your comments and uploaded a revised paper. Here is a brief summary of our updates:

  • Benchmark Scope: MineMem focuses on how agents transfer and update experience under changing conditions, rather than testing whether agents can finish long tasks from start to finish like BEHAVIOR-1K or RoboMemArena. Our goal is to compare published memory methods fairly under the same inputs, actions, and budgets to show when past experience helps or hurts.
  • Evaluation Protocol: Each task runs in one continuous attempt with a single 60 turn budget. Neither the environment nor the memory resets between milestones. We also added a trace of task PL1 to show how strict success, continuous progress, and Table 6 costs are computed.
  • Data Counts, Weights, and Figures: We reported exact sample sizes for all figures, including 450 tasks and 2,700 rollouts per method in the main results. We clarified that rollouts are averaged per instance first with equal weights across levels, and that Figure 5a error bars show 95 percent bootstrap intervals resampled by source demonstration. We also explained that Figure 6b and Figure 13 use different task sets, which accounts for their different rankings.
  • Difficulty, Metrics, and Probes: We explained how task configurations are calibrated to match a common baseline difficulty without exposing them to agents. We specified the scoring rules for workflow metrics and failure labels, confirmed that belief probes run only after each attempt ends, and clarified the token limits for fact injection.
  • Baselines, Costs, and Dataset: We detailed how each baseline is set up under the same interface, including how 3D-Mem uses RGB geometry estimation. We confirmed that reported costs include memory preparation, consultation calls, and decision turns. We also added details on the 25 human demonstrators, 128 recorded sessions, and data licensing.

We want to make sure we have fully understood and addressed your concerns. Please let us know if our replies answer your questions, or if anything else needs further discussion.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.