- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes the Interventional Memory Evaluation Protocol (IMEP), which separates four questions about memory in vision–language–action systems: net performance benefit, conditions under which memory can help, sensitivity to memory interventions, and specificity to task-relevant facts. Its main instruments are paired outcome comparisons, fact-preserving and fact-changing substitutions, exact-repeat and numerical controls, and checks on intervention delivery. It also introduces ChainBench, where query observations omit the deciding historical fact and decision accuracy is evaluated separately from motor execution.
The evaluation covers within-episode learned banks, cross-episode banks, and an agentic-memory setting. For instance, on Remember Color, the U/S/T interventions change 11.9%, 13.1%, and 14.2% of paired outcomes, respectively, without establishing an additional fact-specific effect.
Critical strengths
- The protocol makes a consequential distinction between performance improvement, behavioral sensitivity, and factual attribution.
Reporting both signed success differences and paired outcome changes is useful, since interventions can substantially change individual outcomes while producing little aggregate improvement. The Remember Color results provide a concrete illustration. The paper also handles several common statistical misinterpretations carefully: nonsignificance is not treated as equivalence, equivalence is restricted to the specified margin and delivered intervention, and retrospective analyses are identified explicitly. These are substantive strengths for a measurement paper.
- ChainBench provides a useful separation of historical decision information from execution capability.
The query hides the relevant object–drawer relation and requires a committed choice, addressing the possibility that an agent succeeds by trying multiple alternatives. Separate decision and execution readouts make it possible to distinguish incorrect recall from failed manipulation. The observed improvement from 53.0% to 100% decision accuracy on 202 chains demonstrates that the evaluated reader can recover the required information from the supplied records, where shared executions are a reasonable way to control execution variability for this comparison.
Critical weaknesses
- The experiments do not yet convincingly show that IMEP can identify whether a learned-memory model uses a particular stored fact.
IMEP compares two changes to memory: replacing a record while keeping the relevant fact the same, and replacing it with a record containing a different fact. This comparison is useful only if the experiment can verify what information was changed and whether the robot encounters it when making its decision.
In Remember Color, several issues make the results difficult to interpret. The robot sees the correct color again in the second attempt, so it may not need the memory from the first attempt. Its performance is also close to random guessing, leaving uncertainty about whether it can solve the task reliably. In addition, the substituted records can be partly overwritten as the robot updates its memory. Therefore, similar results from same-fact and different-fact replacements do not clearly explain whether the robot uses the stored fact (C.2; D.1).
In Shell Game, the experiment verifies that the modified memory reaches the model, but it does not verify that the modifications have the intended meaning. One operation rearranges the numbers in a memory representation, while the other generates replacement numbers with similar statistical properties. Both might destroy the relevant fact. Their comparison therefore cannot cleanly distinguish preserving a fact from changing it. The success measure adds another limitation: the robot can touch the wrong cup first and still succeed by later touching the correct one. First-choice accuracy would provide clearer evidence about what it remembered (E.1; C.3).
The paper would be substantially stronger with one experiment in which the robot can reliably solve a task requiring history, the replacements are verified to preserve or change the relevant fact, and that information remains available when the robot makes its choice.
- ChainBench’s strongest results come from reading prepared memory records, providing limited validation of the broader memory process.
In the main ChainBench experiment, the setup specifies a memory state that retains the latest records. It does not evaluate whether the system’s own memory-management process can identify outdated information and retain the appropriate records. A fixed reading algorithm then retrieves and interprets the available information to choose a drawer (C.1). The resulting 100% decision accuracy is meaningful, but it demonstrates successful reading under this prepared memory state. Similarly, the separate 401-chain experiment tests whether changing the supplied fact changes the scripted reader’s decision; it does not test the live system’s memory-writing or retrieval process (B.2).
The supplementary results show why this distinction matters. In Table 21, retrieval alone changes decision accuracy relative to no memory by +5.71 and −0.72 percentage points on the two cohorts, with both confidence intervals including zero. The full configuration performs better, but it changes both the retained memory records and an additional presence-gating rule. Consequently, the comparison does not establish which change is responsible for the improvement.
Questions for authors
-
Clarification: Do any of your learned-bank experiments show that the robot actually relies on the stored fact? In that experiment, can you verify that the same-fact replacement preserves the fact and that the robot reads the replaced information when making its choice? This would help establish whether IMEP can detect fact use, rather than just disruption caused by changing memory.
-
Additional control: Could you run the same replacement procedure used for U/S/T without changing any stored values? Unlike the current C condition, this control would include the installation step. It would help determine whether the 11.9% outcome-change rate comes from the one-ULP change, the installation procedure, or both.
-
Clarification: Why must the extra effect of different-fact swaps over same-fact swaps exceed the effect of the one-ULP control to count as evidence of fact use? Is this a deliberately cautious threshold, or a requirement you believe should apply generally? How would you interpret a statistically reliable effect that falls below this threshold?
-
Clarification and additional evidence: How were the presence gate and nearest-centroid reader fitted, and which data were used? How were these data separated from the evaluation ones? Please also explain how the groups of 403, 401, and 202 chains relate to one another.
2 comments
Response to Reviewer EFAC (2/2): Q3–Q4 and Remaining Weakness
Q3. The ULP criterion and smaller effects
The ULP-exceedance rule is a conservative criterion for a stronger fact-specificity claim in our study, rather than a universal requirement for fact use.
Our learned-bank experiments show that very small numerical perturbations can change outcomes, so ULP provides an empirical reference for numerical sensitivity. For the stronger claim, the lower confidence bound on the excess paired outcome-change rate of T over S must exceed the ULP rate's upper bound, with semantic and consumption checks satisfied (Section 3.4; Appendix K.2). ULP is not a bound on all non-semantic disturbances.
A statistically reliable positive increment below this threshold should still be reported with its confidence interval. With verified replacement semantics and consumption and controlled non-target differences, it can support an incremental effect of the implemented replacement without meeting this study's stronger specificity criterion.
Q4. Fitting data and cohort membership
Using only calibration chains for cross-chain candidates, gate calibration, and reader fitting leaves validation predictions and archived task-success results unchanged on the existing validation cohort.
The upstream decision evaluation contains 403 chains: 202 calibration and 201 validation chains. Of these, 401 gate-accepted chains form the scripted-reader intervention cohort, which includes the 202 executed chains: 103 calibration and 99 validation chains (Appendix C.1, Table 13). The two 202-chain sets therefore overlap on 103 chains.
The original gate used the 202 calibration queries. Its threshold, approximately 0.944978, was the midpoint between the highest calibration no-match score and the lowest retrieval-present score, with acceptance strictly above it. Calibration candidate pools included validation-chain features. The leave-one-chain-out reader excluded the current query chain from centroid fitting but used labels from other validation chains.
We repeated calibration and fitting using only calibration chains, retaining the 202/201 split and the fixed 99-chain executed validation subset. Cross-chain candidates came only from calibration chains, while validation queries retained their own histories. Parameters were fixed before generating predictions, and validation labels were loaded only for scoring. All errors, gate rejections, and fallbacks remained in the denominators.
The threshold changed to 0.942805, but all 201 validation predictions remained identical: 200/201 correct with memory versus 101/201 without memory. Archived task success on the 99 executed validation chains remained 68/99 versus 35/99, a gain of 33.33 percentage points (Table 14).
Remaining Weakness 2. Sources of benefit
The added experiments link supplied histories to robot behavior, while the ChainBench decomposition identifies latest-record retention as the main source of decision-accuracy gains along the tested configuration path.
Q1 reports complementary evidence from a learned memory reader in closed-loop execution and from short-horizon action responses through a native learned-memory pathway (Section 4.5; Appendices L and M).
Using archived decisions, we compared memory states whose retained records were specified by the evaluation (Appendix I, Table 26). With the presence gate off, retaining only the latest record rather than both old and new records improves decision accuracy by 86.27 percentage points (pp) on 204 rewritten chains and 88.41 pp on 69 rewritten chains. Starting from the latest-record setting, adding the presence gate together with two-stage retrieval selection changes these rates by −0.98 and 0.00 pp, respectively.
In both cohorts, errors with old and new records point to outdated facts, and retaining only the latest record corrects these errors.
Response to Reviewer EFAC (1/2): Q1–Q2
Thank you for your questions on semantic validation, intervention controls, and the sources of the ChainBench gains. Q1 also addresses Weakness 1; the remaining Weakness 2 is addressed in Part 2.
Q1. Semantic validation and closed-loop behavior (Weakness 1)
The modular learned-reader evaluation provides closed-loop evidence that drawer-opening choices follow the substituted historical fact, with replacement semantics and actual consumption verified.
A frozen learned visual reader predicts a drawer from supplied history images and passes that prediction as a text instruction to frozen π0.5, which generates actions from current observations (Section 4.5, Table 4; Appendix L). This modular setting is distinct from the learned-bank experiments, where fact-specific use remains unresolved. We evaluated 64 new sources in a known simulated scene, each under two historical facts and six conditions: 128 source–fact conditions and 768 separate rollouts. Paired conditions share the query initial state, with the target object removed. Trials stop at the first drawer displaced more than 10 cm or after 200 actions.
Same-fact replacement S preserves the target-drawer fact, while different-fact replacement T changes it. Saved states and rendered images verify these facts, with scene, camera, and non-target physical states matched. Whole-event replacements retain small non-target visual differences. The reader consumed the installed records in all 7,352 queries across 256 S/T rollouts.
Original history A, identity reinstallation I, and S each yield 96/128 original-fact openings, with identical recorded policy inputs and actions. T yields 96/128 substituted-fact and 0/128 original-fact openings. F uses a preassigned drawer instruction independent of history and yields 46/128 original-fact openings. In 76/128 paired conditions, S opens the original-fact drawer and T the substituted-fact drawer; there are zero reverse pairs. All errors and timeouts remain in the denominator.
The native-memory analysis (Appendix M, Table 29) provides complementary evidence from the community MemoryVLA checkpoint. This measures short-horizon motion; final task success was not measured. Across 48 new sources, left/right histories produce mean gripper shifts of 1.00/1.14 mm in the corresponding directions relative to S over 0.2 s.
Q2. Reinstallation versus numerical perturbation
Reinstalling unchanged values produces no observed outcome changes, whereas one-ULP perturbations still change outcomes under the same installation procedure.
Identity reinstallation I passes unchanged payload values through the same tensor cloning, record reconstruction, and bank-list replacement as U/S/T. Exact repeat C bypasses this procedure (Sections 3.2 and 4.3; Table 2).
With analysis rules fixed before execution, we evaluated 200 seed-paired units per task on Remember Color 3 and 5, using attempt 2 as the primary outcome. I versus A changes 0/200 outcomes in each task; U versus I changes 27/200 (13.5%) and 22/200 (11.0%), respectively. The one-sided 97.5% exact upper bound for I-versus-A outcome discordance is 1.83% per task, with at least 95% Bonferroni joint coverage across the two tasks. These measure paired outcome changes, not net success losses.
New ULP realizations prevent an exact decomposition of the historical 11.9%. Installation-by-content interactions were not separately estimated (Appendix E.2).
Q3–Q4 and the remaining Weakness 2 are addressed in Part 2.