- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MineMem tests transfer from human demonstrations to changed tasks and environments in two interactive voxel-world domains. Fifty templates cover five memory-related capabilities. Six adapted memory approaches are compared with no-history and fact-injection controls using physical progress, belief probes and costs. Physical milestones start from prerequisite-complete states. Results show capability-dependent benefits and negative transfer when historical information conflicts with current evidence.
Critical strengths
S1. The transfer controls are useful. Same-instance no-history baselines and environment, goal and joint shifts separate the value of demonstration experience from task difficulty (Sections 3-4). Fact injection exposes cases where access to past information helps recall but harms adaptation.
S2. The scoring distinguishes knowledge from action. Belief answers do not enter the physical score. Simulator checks verify progress, supplied prerequisites earn no credit, and memory updates are discarded between attempts (Section 3.3; Appendix E). These choices support interpretable component diagnostics.
Critical weaknesses
W1. The empirical sample cannot be reconstructed. Fifty templates do not identify the instantiated tasks, demonstrations or repeated attempts behind each mean. Evaluated counts and aggregation weights are missing. Figures 5(a), 6(b) and 13 show error bars without defining their sampling unit or statistical meaning. This prevents assessment of the stability of the rankings and negative-transfer contrasts (Section 4.2; Appendix I).
W2. The scoring unit and memory lifecycle remain ambiguous. Appendix E.1 scores milestones from separate prerequisite-complete starts; Section 4.2 describes completing all milestones within a sixty-turn attempt. Whether budget and memory reset between milestones is unspecified. Consequently, the relationship between strict success, within-episode adaptation and the per-episode costs in Table 6 cannot be determined.
Assessment. I recommend weak rejection (2/4). The benchmark design is promising, but the reported comparisons need an auditable sample and an unambiguous evaluation unit.
Questions for authors
Q1 (W1). How many task instances and repeated attempts contribute to each main figure, and how are they weighted? Please define the error bars and provide the existing evaluated inventory or a compact count table; the template count is not the evaluation sample size.
Q2 (W2). At each milestone restart, does the agent receive a new sixty-turn budget and freshly initialized memory, or does either carry over? Please trace one multi-milestone task through the resets and show how its success and Table 6 cost are aggregated.
2 comments
Picking up W2/Q2 from our response above, we trace one multi-milestone task through the resets and show how its success and Table 6 cost are aggregated. The protocol is the single continuous attempt described there. What follows is the requested trace.
PL1 is a Logistica planning template whose goal is to construct a new composite network. Its applicable milestones come from the five logistics checks, namely component placement, topological connectivity, network membership, routing configuration, and item transfer, with base weights 0.20, 0.15, 0.25, 0.20, and 0.20, normalized within the task. A milestone the task does not require is omitted rather than credited as completed.
- The environment is instantiated from the matched task configuration and the memory from the demonstration package, the single initialization of the evaluation. The agent places the required conduit, machine, and storage components, and placement is verified from the simulator's node-placement records at the end of the attempt, not at a milestone boundary, so scores the state reached.
- It then continues without re-initialization, establishing six-connected voxel connectivity, registering the active nodes with the controller through the network heartbeat probes, writing the routing whitelist filters and request limits, and transferring the inventory item to its destination where the goal requires it.
- Strict success (Appendix E.1) is only if every applicable milestone passes, namely the placement check, the connectivity check, the membership probe, the routing-configuration check, and the transfer check, and no validity gate is violated within the same sixty-decision-turn budget.
- Table 6's input-token and model-call columns are cumulative over one attempt, including the memory preparation and consultation calls that precede and interleave with the decision turns. This task therefore contributes to its approach's row rather than a row of its own, and no milestone contributes a separate cost line, since costs are never summed across milestones.
An earlier failure does not discard the later milestones, which are scored against the state actually reached. That is why the continuous progress score and strict success answer different questions. Re-executing a prerequisite is paid for inside the shared budget rather than from a fresh allowance, so the comparison across mechanisms is unaffected by how the budget is divided.
Some templates, such as PL2, begin from an interrupted or partially completed state by design. This is the task's single initial state, not a reset between milestones. The agent continues that chain within the same budget and the same initialized memory, and milestones whose prerequisites were already satisfied are graded on the pre-existing state.
We thank the reviewer for the careful review, for the transfer controls, and for the separation of knowledge from action. In response to these comments we checked the manuscript, the source code, and the dataset.
1. Counts, weights, and error bars (W1/Q1).
A rollout repeats one instantiated task. The instance and its demonstration are held fixed across its six rollouts, which are averaged within the instance before aggregation. Counts are per memory mechanism, and each row gives the sample size behind one plotted bar or panel.
Fifty templates cover five capabilities in two domains, five per domain–capability group. Each is instantiated at three randomly drawn procedural seeds and evaluated under the three transfer conditions of Section 4.2, giving 50 × 3 × 3 = 450 instantiated tasks, six rollouts each, 450 × 6 = 2,700 rollouts in total. Figure 2 reports the five capabilities one by one, each bar pooling that capability across the two domains (ten templates, 90 tasks, 540 rollouts). Figure 13 uses a fixed ten-template subset computed per decision model, since eight memory mechanisms under three decision models would multiply the attempts several times over, so each panel holds 90 tasks and 540 rollouts per method. Figure 5(a) is computed on the conflict-injected subset of the cohort, namely the ten memory-updating templates (BL1–BL5 and BN1–BN5) under the same three seeds and three transfer conditions, so each mechanism contributes 90 tasks and 540 rollouts to it. The demonstrations come from 128 human-recorded sessions, 56.2 hours in total. We have already documented in the manuscript the per-attempt artifact layout, in which every attempt records its instance metadata, model identifiers and milestone scores, and we have added the released per-instance inventory, so these denominators can be checked directly.
Weights are equal at each level and the instance mean is taken last. Figure 2 reports each capability separately, Figure 6(b) averages over the five capabilities, Figure 13 compares each mechanism with the no-history control run by the same decision model and is computed separately within each model rather than pooled across models, and Figure 5(a) weights the instances of its conflict condition equally. The error bars in Figure 5(a) are 95% bootstrap confidence intervals for the paired transfer contrasts. The resampling unit is the source demonstration recording, so instances that share a recording are not treated as independent, and each resample recomputes the aggregation of the figure it belongs to.
2. Milestone resets and cost aggregation (W2/Q2).
Each task runs in one continuous attempt. The environment and memory are initialized once, one sixty-decision-turn budget covers the whole task, and the milestones are scoring components of that same trajectory rather than independent attempts. Neither the environment nor the memory resets between milestones, and no milestone receives a budget of its own. We have already clarified this protocol and revised the appendix so that both descriptions refer to one evaluation unit, and it will be reflected in the submission.
As the reviewer asked for a concrete trace, the next comment walks one multi-milestone task (PL1) through the protocol step by step under the single continuous attempt stated above.
Over a long time, we devoted substantial effort to this benchmark and its data, hoping to give the field a reference standard. We thank the reviewer again for the careful reading that made our work more complete, understandable, and reusable.