acceptodds
Under review as a conference paper at ICLR 2027

Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems

Abstract

When we say that a robot uses memory, does that memory actually play a role? Success-rate ablations alone cannot establish the use of task-relevant facts: they change information and computation together, while aggregate rates can hide offsetting gains and losses. We propose an Interventional Memory Evaluation Protocol (IMEP) for vision-language-action (VLA) systems to address four questions: (1) Does memory provide a net benefit? (2) Under what conditions does memory help? (3) Do changes to memory alter task outcomes? (4) Are these effects specific to task-relevant facts? At the retrieval interface, we compare target-fact substitutions with fact-preserving replacements, minimal numerical perturbations, and exact replicas, while assessing decision correctness separately from motor execution success. We evaluate within-episode, cross-episode, and agentic memory. In cross-episode learned-bank experiments, a one-unit-in-the-last-place (ULP) intervention, including its installation procedure, changes 11.9% of seed-paired outcomes, while changing key facts produces no clear additional effect over fact-preserving replacements. To examine when memory helps, we introduce ChainBench, where correct decisions require historical information unavailable in current observations. The full memory configuration raises task success from 39.1% to 74.3%, matching the oracle-decision condition under shared recorded executions. These results show that correct history can support robot decisions, but establishing whether a memory system realizes this benefit requires fact-specific evidence beyond aggregate success rates.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot memory, evaluation methodology, interventional evaluation, robot learning, memory benchmarks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EFACOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes the Interventional Memory Evaluation Protocol (IMEP), which separates four questions about memory in vision–language–action systems: net performance benefit, conditions under which memory can help, sensitivity to memory interventions, and specificity to task-relevant facts. Its main instruments are paired outcome comparisons, fact-preserving and fact-changing substitutions, exact-repeat and numerical controls, and checks on intervention delivery. It also introduces ChainBench, where query observations omit the deciding historical fact and decision accuracy is evaluated separately from motor execution.

The evaluation covers within-episode learned banks, cross-episode banks, and an agentic-memory setting. For instance, on Remember Color, the U/S/T interventions change 11.9%, 13.1%, and 14.2% of paired outcomes, respectively, without establishing an additional fact-specific effect.

Critical strengths
  1. The protocol makes a consequential distinction between performance improvement, behavioral sensitivity, and factual attribution.

Reporting both signed success differences and paired outcome changes is useful, since interventions can substantially change individual outcomes while producing little aggregate improvement. The Remember Color results provide a concrete illustration. The paper also handles several common statistical misinterpretations carefully: nonsignificance is not treated as equivalence, equivalence is restricted to the specified margin and delivered intervention, and retrospective analyses are identified explicitly. These are substantive strengths for a measurement paper.

  1. ChainBench provides a useful separation of historical decision information from execution capability.

The query hides the relevant object–drawer relation and requires a committed choice, addressing the possibility that an agent succeeds by trying multiple alternatives. Separate decision and execution readouts make it possible to distinguish incorrect recall from failed manipulation. The observed improvement from 53.0% to 100% decision accuracy on 202 chains demonstrates that the evaluated reader can recover the required information from the supplied records, where shared executions are a reasonable way to control execution variability for this comparison.

Critical weaknesses
  1. The experiments do not yet convincingly show that IMEP can identify whether a learned-memory model uses a particular stored fact.

IMEP compares two changes to memory: replacing a record while keeping the relevant fact the same, and replacing it with a record containing a different fact. This comparison is useful only if the experiment can verify what information was changed and whether the robot encounters it when making its decision.

In Remember Color, several issues make the results difficult to interpret. The robot sees the correct color again in the second attempt, so it may not need the memory from the first attempt. Its performance is also close to random guessing, leaving uncertainty about whether it can solve the task reliably. In addition, the substituted records can be partly overwritten as the robot updates its memory. Therefore, similar results from same-fact and different-fact replacements do not clearly explain whether the robot uses the stored fact (C.2; D.1).

In Shell Game, the experiment verifies that the modified memory reaches the model, but it does not verify that the modifications have the intended meaning. One operation rearranges the numbers in a memory representation, while the other generates replacement numbers with similar statistical properties. Both might destroy the relevant fact. Their comparison therefore cannot cleanly distinguish preserving a fact from changing it. The success measure adds another limitation: the robot can touch the wrong cup first and still succeed by later touching the correct one. First-choice accuracy would provide clearer evidence about what it remembered (E.1; C.3).

The paper would be substantially stronger with one experiment in which the robot can reliably solve a task requiring history, the replacements are verified to preserve or change the relevant fact, and that information remains available when the robot makes its choice.

  1. ChainBench’s strongest results come from reading prepared memory records, providing limited validation of the broader memory process.

In the main ChainBench experiment, the setup specifies a memory state that retains the latest records. It does not evaluate whether the system’s own memory-management process can identify outdated information and retain the appropriate records. A fixed reading algorithm then retrieves and interprets the available information to choose a drawer (C.1). The resulting 100% decision accuracy is meaningful, but it demonstrates successful reading under this prepared memory state. Similarly, the separate 401-chain experiment tests whether changing the supplied fact changes the scripted reader’s decision; it does not test the live system’s memory-writing or retrieval process (B.2).

The supplementary results show why this distinction matters. In Table 21, retrieval alone changes decision accuracy relative to no memory by +5.71 and −0.72 percentage points on the two cohorts, with both confidence intervals including zero. The full configuration performs better, but it changes both the retained memory records and an additional presence-gating rule. Consequently, the comparison does not establish which change is responsible for the improvement.

Questions for authors
  1. Clarification: Do any of your learned-bank experiments show that the robot actually relies on the stored fact? In that experiment, can you verify that the same-fact replacement preserves the fact and that the robot reads the replaced information when making its choice? This would help establish whether IMEP can detect fact use, rather than just disruption caused by changing memory.

  2. Additional control: Could you run the same replacement procedure used for U/S/T without changing any stored values? Unlike the current C condition, this control would include the installation step. It would help determine whether the 11.9% outcome-change rate comes from the one-ULP change, the installation procedure, or both.

  3. Clarification: Why must the extra effect of different-fact swaps over same-fact swaps exceed the effect of the one-ULP control to count as evidence of fact use? Is this a deliberately cautious threshold, or a requirement you believe should apply generally? How would you interpret a statistically reliable effect that falls below this threshold?

  4. Clarification and additional evidence: How were the presence gate and nearest-centroid reader fitted, and which data were used? How were these data separated from the evaluation ones? Please also explain how the groups of 403, 401, and 202 chains relate to one another.

2 comments
AuthorsOpenReview ↗

Response to Reviewer EFAC (1/2): Q1–Q2

Thank you for your questions on semantic validation, intervention controls, and the sources of the ChainBench gains. Q1 also addresses Weakness 1; the remaining Weakness 2 is addressed in Part 2.

Q1. Semantic validation and closed-loop behavior (Weakness 1)

The modular learned-reader evaluation provides closed-loop evidence that drawer-opening choices follow the substituted historical fact, with replacement semantics and actual consumption verified.

A frozen learned visual reader predicts a drawer from supplied history images and passes that prediction as a text instruction to frozen π0.5, which generates actions from current observations (Section 4.5, Table 4; Appendix L). This modular setting is distinct from the learned-bank experiments, where fact-specific use remains unresolved. We evaluated 64 new sources in a known simulated scene, each under two historical facts and six conditions: 128 source–fact conditions and 768 separate rollouts. Paired conditions share the query initial state, with the target object removed. Trials stop at the first drawer displaced more than 10 cm or after 200 actions.

Same-fact replacement S preserves the target-drawer fact, while different-fact replacement T changes it. Saved states and rendered images verify these facts, with scene, camera, and non-target physical states matched. Whole-event replacements retain small non-target visual differences. The reader consumed the installed records in all 7,352 queries across 256 S/T rollouts.

Original history A, identity reinstallation I, and S each yield 96/128 original-fact openings, with identical recorded policy inputs and actions. T yields 96/128 substituted-fact and 0/128 original-fact openings. F uses a preassigned drawer instruction independent of history and yields 46/128 original-fact openings. In 76/128 paired conditions, S opens the original-fact drawer and T the substituted-fact drawer; there are zero reverse pairs. All errors and timeouts remain in the denominator.

The native-memory analysis (Appendix M, Table 29) provides complementary evidence from the community MemoryVLA checkpoint. This measures short-horizon motion; final task success was not measured. Across 48 new sources, left/right histories produce mean gripper shifts of 1.00/1.14 mm in the corresponding directions relative to S over 0.2 s.

Q2. Reinstallation versus numerical perturbation

Reinstalling unchanged values produces no observed outcome changes, whereas one-ULP perturbations still change outcomes under the same installation procedure.

Identity reinstallation I passes unchanged payload values through the same tensor cloning, record reconstruction, and bank-list replacement as U/S/T. Exact repeat C bypasses this procedure (Sections 3.2 and 4.3; Table 2).

With analysis rules fixed before execution, we evaluated 200 seed-paired units per task on Remember Color 3 and 5, using attempt 2 as the primary outcome. I versus A changes 0/200 outcomes in each task; U versus I changes 27/200 (13.5%) and 22/200 (11.0%), respectively. The one-sided 97.5% exact upper bound for I-versus-A outcome discordance is 1.83% per task, with at least 95% Bonferroni joint coverage across the two tasks. These measure paired outcome changes, not net success losses.

New ULP realizations prevent an exact decomposition of the historical 11.9%. Installation-by-content interactions were not separately estimated (Appendix E.2).

Q3–Q4 and the remaining Weakness 2 are addressed in Part 2.

AuthorsOpenReview ↗

Response to Reviewer EFAC (2/2): Q3–Q4 and Remaining Weakness

Q3. The ULP criterion and smaller effects

The ULP-exceedance rule is a conservative criterion for a stronger fact-specificity claim in our study, rather than a universal requirement for fact use.

Our learned-bank experiments show that very small numerical perturbations can change outcomes, so ULP provides an empirical reference for numerical sensitivity. For the stronger claim, the lower confidence bound on the excess paired outcome-change rate of T over S must exceed the ULP rate's upper bound, with semantic and consumption checks satisfied (Section 3.4; Appendix K.2). ULP is not a bound on all non-semantic disturbances.

A statistically reliable positive increment below this threshold should still be reported with its confidence interval. With verified replacement semantics and consumption and controlled non-target differences, it can support an incremental effect of the implemented replacement without meeting this study's stronger specificity criterion.

Q4. Fitting data and cohort membership

Using only calibration chains for cross-chain candidates, gate calibration, and reader fitting leaves validation predictions and archived task-success results unchanged on the existing validation cohort.

The upstream decision evaluation contains 403 chains: 202 calibration and 201 validation chains. Of these, 401 gate-accepted chains form the scripted-reader intervention cohort, which includes the 202 executed chains: 103 calibration and 99 validation chains (Appendix C.1, Table 13). The two 202-chain sets therefore overlap on 103 chains.

The original gate used the 202 calibration queries. Its threshold, approximately 0.944978, was the midpoint between the highest calibration no-match score and the lowest retrieval-present score, with acceptance strictly above it. Calibration candidate pools included validation-chain features. The leave-one-chain-out reader excluded the current query chain from centroid fitting but used labels from other validation chains.

We repeated calibration and fitting using only calibration chains, retaining the 202/201 split and the fixed 99-chain executed validation subset. Cross-chain candidates came only from calibration chains, while validation queries retained their own histories. Parameters were fixed before generating predictions, and validation labels were loaded only for scoring. All errors, gate rejections, and fallbacks remained in the denominators.

The threshold changed to 0.942805, but all 201 validation predictions remained identical: 200/201 correct with memory versus 101/201 without memory. Archived task success on the 99 executed validation chains remained 68/99 versus 35/99, a gain of 33.33 percentage points (Table 14).

Remaining Weakness 2. Sources of benefit

The added experiments link supplied histories to robot behavior, while the ChainBench decomposition identifies latest-record retention as the main source of decision-accuracy gains along the tested configuration path.

Q1 reports complementary evidence from a learned memory reader in closed-loop execution and from short-horizon action responses through a native learned-memory pathway (Section 4.5; Appendices L and M).

Using archived decisions, we compared memory states whose retained records were specified by the evaluation (Appendix I, Table 26). With the presence gate off, retaining only the latest record rather than both old and new records improves decision accuracy by 86.27 percentage points (pp) on 204 rewritten chains and 88.41 pp on 69 rewritten chains. Starting from the latest-record setting, adding the presence gate together with two-stage retrieval selection changes these rates by −0.98 and 0.00 pp, respectively.

In both cohorts, errors with old and new records point to outdated facts, and retaining only the latest record corrects these errors.

Reviewer_E9VxOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper questions whether task success is sufficient to demonstrate that a VLA system actually uses task-relevant memory. The authors propose the Interventional Memory Evaluation Protocol (IMEP), which examines memory utility, outcome sensitivity, and dependence on specific stored facts through controlled interventions. The evaluation covers within-episode, cross-episode, and agentic memory systems.

The experiments show that learned-memory systems can respond to memory interventions without clear evidence of fact-specific dependence. The authors also introduce ChainBench, a benchmark designed to require historical information for correct decisions. On ChainBench, the full-memory configuration achieves substantially higher decision accuracy and task success, while a separate scripted-reader experiment demonstrates fact-specific sensitivity under controlled conditions.

Critical strengths
  1. An interesting distinction in memory evaluation.

The paper raises a useful distinction between a memory component improving performance and a model actually relying on particular information stored in memory. I agree that standard memory-on/off ablations cannot always explain which information contributes to a decision. The observation that aggregate success rates can hide changes in individual outcomes is also worth considering. The proposed fact-preserving and fact-changing interventions provide a way to examine this question more directly.

  1. Careful experimental controls and reporting.

The paper includes several useful controls, including exact repetitions, numerical perturbations, and different types of memory substitutions. I particularly like the effort to separate decision accuracy from motor execution success in ChainBench. The authors also identify important limitations in existing tasks, such as repeated attempts in Shell Game and the reappearance of relevant information in Remember Color. Negative results, incomplete intervention delivery, and post hoc analyses are generally reported explicitly. This makes it easier to understand what the experiments do and do not establish.

Critical weaknesses
  1. The motivation and necessity of the central research question are not sufficiently established.

My main concern is that the paper does not clearly establish what important problem IMEP solves beyond a properly designed memory-dependent evaluation.

If a task genuinely requires memory, the relevant information is unavailable in the current observation, no alternative historical-information channel exists, and the robot cannot succeed through repeated attempts, then performance significantly above the memory-free ceiling already provides strong evidence that historical information is being used.

I understand that the authors are interested in a more specific question: whether the system relies on particular task-relevant facts stored in memory. This is a valid distinction, but the paper does not sufficiently demonstrate why this additional level of attribution is necessary or what new conclusions it provides in practice.

In particular, the issues identified in Shell Game and Remember Color seem to arise largely from limitations of the benchmark designs themselves. These findings support the need for better-controlled memory-dependent tasks, but they do not necessarily establish the need for a new intervention protocol. ChainBench addresses some of these issues by explicitly requiring historical information, which makes the additional role of IMEP less clear.

I would find the motivation more convincing if the authors could show a case where a rigorously controlled memory-dependent benchmark leads to an incomplete or misleading interpretation, and IMEP reveals something important that the conventional evaluation cannot.

  1. The experiments do not convincingly validate IMEP on the memory systems it aims to evaluate.

The empirical evidence for the central fact-specific attribution objective remains limited. In Remember Color, same-fact and different-fact substitutions produce similar rates of outcome changes, while the one-ULP intervention, including its installation procedure, also changes a substantial number of outcomes. The tested policy performs near chance on these tasks, and the intervention-delivery analysis identifies additional limitations. Given these issues, it is difficult to determine whether the inconclusive findings reflect the actual memory behavior of the models or limitations of the evaluation setup.

The clearest positive evidence comes from ChainBench, but the fact-specific intervention results are obtained using curated records and a scripted nearest-centroid reader, rather than an end-to-end learned-memory VLA system. This is a useful controlled demonstration, but it does not establish that the protocol can reliably identify fact-specific memory use in learned-memory policies. The main ChainBench task-success comparison also uses shared recorded executions, so it should not be interpreted as an independent validation of a complete autonomous memory system.

The paper is careful to acknowledge many of these limitations. However, this leaves the central methodological contribution insufficiently validated. The four-question IMEP framework and some of its interpretation criteria were also developed after the experimental campaigns, making its prospective effectiveness difficult to assess.

Finally, all experiments are conducted in simulation. This is particularly limiting for robotic memory evaluation, where asynchronous sensing and control, observation delays, partial observability, and stale memory records can strongly affect how historical information is stored and used. The simulated camera blackout and memory-validity experiments are relevant, but they do not fully capture these real-world conditions. Without real-robot evaluation or more realistic robustness tests, the broader applicability of the protocol remains uncertain.

Questions for authors
  1. The paper argues that success-rate improvements alone cannot establish reliance on task-relevant memory facts. Could the authors provide a concrete example involving a learned-memory VLA where conventional success-based evaluation leads to an incomplete or misleading interpretation, and IMEP resolves this ambiguity under valid evaluation conditions? The current learned-bank experiments appear to be limited by task design or intervention delivery, making it difficult to establish the practical significance of the proposed evaluation gap.

  2. Could the authors provide stronger positive validation of IMEP on a learned-memory VLA? I acknowledge the scripted-reader positive control in Appendix B.2 and the synthetic detectability study in Appendix B.3. However, neither establishes that IMEP can reliably identify known fact-specific dependence in an actual learned-memory policy. A positive control on a competent learned-memory VLA, with verified history dependence and intervention delivery, would make the proposed protocol more convincing.

  3. Section 3.2 acknowledges that the exact-repeat control bypasses the memory-installation procedure used by the other interventions. Could the authors include a control that reinstalls the original memory payload through the same procedure, without modifying its contents? This would help isolate installation-related effects from numerical perturbations and clarify the interpretation of the learned-memory results.

2 comments
AuthorsOpenReview ↗

Response to Reviewer E9Vx

Thank you for distinguishing the motivation for IMEP from its empirical validation. Q1 addresses Weakness 1; Q2–Q3 address the learned-system and installation concerns in Weakness 2.

Q1. What does IMEP add beyond success rates?

In the learned-reader evaluation, paired S/T outcomes reveal that some errors under original-target scoring follow the substituted historical fact, which aggregate success rates alone do not show.

Performance above a valid memory-free ceiling on a rigorously controlled task can support history dependence. Adding or enabling memory can also change both information and computation, so a memory-on/off success comparison need not attribute gains to particular stored facts.

Our modular closed-loop evaluation connects a learned reader to a VLA policy using supplied histories in a known simulated scene (Section 4.5, Table 4; Appendix L). At the same query state, same-fact replacement S supplies another record preserving the target fact, while different-fact replacement T changes it. These are matched whole-event replacements with small remaining non-target visual differences.

Of 128 historical-fact conditions, 76 show S opening the original-fact drawer and T opening the substituted-fact drawer, with zero reverse pairs. Errors and timeouts remain in the denominator.

Q2. Positive validation on a learned system

The modular closed-loop evaluation provides positive evidence that a learned memory reader can guide VLA actions according to supplied historical facts, with consumption of the replacement records verified.

The visual reader predicts a drawer from history images and passes that prediction as a text instruction to π0.5, which generates actions from current observations. Both components are frozen. The endpoint is the first drawer displaced by more than 10 cm within 200 actions.

We evaluated 64 new sources, each with two opposing historical facts and six conditions: 128 historical-fact conditions and 768 separate rollouts. Conditions within a source share the query initial state, with the target object removed. All planned sources are included.

Original history A, identity reinstallation I, and S each open the original-fact drawer in 96/128 conditions, with identical recorded policy inputs and action sequences. F uses a preassigned specific drawer instruction independent of the historical fact, rather than the reader prediction, and achieves 46/128 original-fact openings. A exceeds F by 39.06 percentage points (pp). T opens the substituted-fact drawer in 96/128 conditions. All 7,352 S/T reads consume the installed records, and the reader receives no evaluation fact labels.

The native-memory analysis (Appendix M, Table 29) adds mean history-directed gripper displacements of 1.00 mm leftward and 1.14 mm rightward relative to S over 0.2 s across 48 new sources on the community MemoryVLA checkpoint. This measures short-horizon motion; final task success was not measured.

Q3. Original-value reinstallation control

We added original-value reinstallation I, which produces no observed outcome changes relative to A, while one-ULP U still changes outcomes relative to I under the same installation procedure.

I passes unchanged payload values through the same tensor cloning, record reconstruction, and bank-list replacement as U/S/T, whereas exact repeat C bypasses installation (Section 4.3, Table 2).

With rules fixed before execution, we evaluated 200 seed-paired units per task on Remember Color 3 and 5, using attempt 2 as primary. I versus A changes 0/200 outcomes in each task; the upper bound is 1.83% per task, with at least 95% joint coverage. U versus I changes 13.5% and 11.0% of outcomes.

New ULP realizations prevent an exact decomposition of the historical 11.9%, and installation-by-content interactions are not separately identified (Appendix E.2).

Remaining Weakness: analysis timing and real-world scope

The closed-loop evaluation used a prespecified design, while the framework was developed after examining earlier results and all robot experiments were conducted in simulation.

Section 3.1 and Appendix G.3 document the timing of the four-question framework and some common interpretation rules. The closed-loop conditions, endpoint, and primary statistical comparisons were fixed before execution (Appendix L).

Camera-blackout and record-validity experiments provide controlled checks within these simulated settings.

Thanks for the additional experiments; while my main concern about the novelty remains, I appreciate the stronger empirical evidence and will raise my score from 1 to 2.

Reviewer_UQ7POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces IMEP to distinguish memory’s net benefit, behavioral influence, and fact-specific effects. Experiments examine learned memory banks and ChainBench, where curated historical records improve success from 39.1% to 74.3%.

Critical strengths

Separating aggregate success from paired outcome changes and fact specificity addresses an important attribution problem. ChainBench hides the deciding fact at query time and evaluates decision accuracy separately from execution success.

Critical weaknesses

Incomplete semantic validation and intervention delivery leave learned-bank fact specificity unresolved, limiting separation of memory failures from intervention failures. The strongest positive evidence uses curated records and a scripted reader. Shared recorded executions further limit evidence for autonomous memory writing and retrieval.

Questions for authors

Clarification: What data calibrated the presence gate, and was calibration independent of the evaluated chains? Clarification: What does the installation routine change beyond the payload, and can existing controls separate installation effects from numerical sensitivity?

1 comment
AuthorsOpenReview ↗

Response to Reviewer UQ7P

Thank you for recognizing the distinction between memory benefit, behavioral sensitivity, and fact-specific effects.

Q1. Calibration data and evaluation independence

The presence gate was calibrated on 202 of the 403 upstream chains, and the execution cohort was not independent because it included 103 of these chains, but calibration-only reanalysis leaves the existing validation predictions and archived task-success results unchanged.

The remaining 201 chains formed the validation set, with 99 included in the execution cohort. The gate threshold, approximately 0.944978, was the midpoint between the highest calibration no-match score and the lowest retrieval-present score, with acceptance strictly above it.

Calibration candidate pools also included validation features, and the leave-one-chain-out reader excluded the query chain but used labels from other validation chains. We therefore repeated calibration and fitting using only the 202 calibration chains, retaining the 202/201 split and fixed 99-chain execution-validation subset. Cross-chain candidates came only from calibration chains; validation queries could use their own histories. Parameters were fixed before generating validation predictions, with errors, gate rejections, and fallbacks retained in all denominators.

The threshold changed to 0.942805, but all 201 validation predictions remained identical: 200/201 correct with memory versus 101/201 without memory. Archived task success on the 99 executed validation chains remained 68/99 versus 35/99, a gain of 33.33 percentage points (pp; descriptive paired-bootstrap 95% CI [24.24, 42.42]).

This is a post hoc robustness analysis of previously examined data, reusing archived executions from cohorts that share scenes and some initial states. Details are in Sections 3.5 and 4.4 and Appendix C.1 (Tables 13–14).

Q2. Installation effects and numerical sensitivity

The original C control cannot isolate installation effects, which may include changes to tensor identities and storage relationships, so we added I to test numerical sensitivity under the same installation procedure.

Installation clones memory tensors, reconstructs records, and replaces the persistent-bank list, while preserving keys, slots, timestamps, and versions. Exact repeat C bypasses this procedure. Identity reinstallation I passes unchanged payload values through the same routine as U/S/T.

Rules were fixed before execution, with attempt 2 as the primary outcome and 200 seed-paired units per task. In Section 4.3, Table 2, I versus A changes 0/200 outcomes on both Remember Color 3 and 5; one-ULP U versus I changes 27/200 (13.5%) and 22/200 (11.0%), respectively.

The upper bound on I-versus-A outcome discordance is 1.83% per task, with at least 95% joint coverage across both tasks. This bounds outcome changes under unchanged-value reinstallation while showing that numerical perturbations still affect outcomes under the same procedure. Previously evaluated seeds and new ULP realizations prevent an exact decomposition of the historical 11.9%; installation-by-content interactions were not separately estimated (Appendix E.2).

Remaining Weakness: learned-system validation and fact attribution

Our modular learned-reader evaluation provides directional closed-loop evidence that drawer choices follow the substituted histories, with replacement facts and actual consumption verified.

In a known simulated scene, a frozen learned visual reader predicts a drawer from supplied histories and passes that prediction as a text instruction to the frozen π0.5 policy (Section 4.5, Table 4; Appendix L). Whole-event replacements retain small non-target visual differences; autonomous memory management is outside this experiment.

Six conditions across 64 new sources and 128 historical-fact conditions yielded 768 separately executed rollouts. We verified the replacement facts and confirmed consumption of the installed records in all 7,352 S/T reads. In 76/128 paired conditions, same-fact replacement S opened the original-history drawer and different-fact replacement T opened the replacement-history drawer; there were 0/128 reverse pairs. All errors and timeouts remain in the denominator.

“Short-Horizon Action Responses in the Native Memory Pathway” (Appendix M, Table 29) provides complementary evidence on the tested community MemoryVLA checkpoint. Across 48 new sources, left/right histories relative to same-fact replacement produced mean direction-consistent gripper displacements of 1.00/1.14 mm over 0.2 s. Both prespecified contrasts had confidence intervals above zero; final task success was not measured.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on our responses

Thank you again to all reviewers for your time and feedback. We have posted our point-by-point responses and updated the manuscript with additional experiments and clarifications. We would welcome your thoughts on these updates and would be happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.