acceptodds
Withdrawn from ICLR 2027

The Write Window in Embodied Multimodal Memory: An Access Structure Is a Pair, and Capability Is a Product

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory architectures for embodied multimodal agents differ in how they index, compress and retrieve, and agree silently on what they write: a description of one frame. We make the write window an architectural parameter, formalise an access structure as a pair (a write map over a window of observations and an address algebra over its records, no address richer than the write it runs over), and read capability for conjunctive queries as a product of an identity, an ordinal and a percept factor, so that which component binds is simply which factor is smallest. Two walls follow. The predicates a frame-local write can decide are closed under time reversal, so no reversal-paired verb (put against take, open against close) is writable at any state budget, and any direction a single frame supports is bounded by the majority prior. And an address that indexes by count compounds per-occurrence detector error, so a later occurrence resolves with a probability that falls geometrically in its rank. Both are measured on the ground-truth locator frames of eight EPIC-KITCHENS streams: the standard single-frame write names the verb barely more often than not, and its error mass sits exactly on the reversal pairs. AXIS widens the window to an ordered pair, differences per-object statuses into events, retains every occurrence with its witness, and executes addresses rather than searching by similarity. The registered controls then refuse the obvious reading: the pair's gain is attributable to the prompt's enumerated verb list rather than to the window, since the same frame attached twice gains as much, and direction under every listed rule sits at or below the majority prior. What does escape the prior is a property of the pair's spacing rather than of the pair (one backbone reads direction above the prior when the two frames are a second apart under a free-vocabulary prompt and not when they are one action apart), and an independent public benchmark agrees at the same spacing with the read held fixed. Behind the window the ordinal wall binds: occurrences are detected at a recall and precision no scaffold-side filter improves. Replacing the written ledger by one built from the corpus's own annotations, everything else held fixed, roughly doubles the same loop and lets the address reach its target far more often, which locates the entire end-to-end loss on the write rather than the read. Of the registered predictions, those about the measurement held and those about the write side and the end-to-end result failed; we report all of them and relocate the binding wall to detector reliability at a spacing the detector can actually read.

Primary area:
foundation or frontier models, including LLMs
Keywords:
embodied multimodal agents, agent memory, write window, time reversal symmetry, video memory architecture, ordinal addressing, event detection, product decomposition, pre-registration, EPIC-KITCHENS

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SHVdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the write side of embodied multimodal memory. The authors argue that many memory architectures focus on indexing and retrieving stored information while paying less attention to what information is available when a memory record is written.

The paper formalizes a memory access structure as a write map and an address algebra, and proposes a capability decomposition involving event identity, ordinal addressing, and perceptual reading. It also discusses limitations of single-frame writing for recognizing time-reversed actions, and the effect of detection errors on ordinal queries.

Based on this analysis, the authors introduce AXIS, which uses ordered frame pairs, object status differences, event occurrence records, and explicit address execution. Experiments on EPIC-KITCHENS streams and TempCompass evaluate the write rules. The end-to-end experiments show that the proposed system does not achieve the expected improvements, while an oracle ledger gives substantially better results. The paper reports these negative findings and uses them to identify remaining limitations in event detection.

Critical strengths
  1. The paper raises an interesting question about what information is lost at memory writing time. The distinction between recording an object state and identifying a transition is useful, particularly for temporal and ordinal queries. I also appreciate the attempt to connect the write representation with the types of addresses that can be executed later.

  2. The experimental analysis is unusually careful about negative results. The authors include duplicate-frame and reversed-frame controls, which help show that the initial improvement from ordered pairs cannot be attributed to temporal information alone. The paper also reports failed registered predictions rather than hiding them. The oracle-ledger experiment is another useful control: on the conjunctive task, replacing the model-written ledger with an annotation-based ledger improves accuracy from 18.8% to 31.5%. This gives evidence that the written event representation is a major limitation of the evaluated system.

Critical weaknesses
  1. I have concerns about the scope and correctness of several theoretical claims. The identity wall is valid for a frame-local emission under its stated symmetry assumptions, but a memory system can store single-frame statuses and compare them later to recover transitions. The paper acknowledges this possibility, yet the broader claim that the address algebra cannot be richer than the write still needs a more precise definition of what information the write representation contains. The product capability bound also relies on an assumption about the accuracy after the address chain fails. This assumption is discussed in Appendix B.2, but it would be better to state it explicitly together with Proposition 1. More concretely, the final claim of Proposition 8 about the recall of the union of two detectors appears incorrect without an additional assumption on their error dependence. The stated upper bound holds under independent or positively correlated misses, but can be violated when the misses are negatively correlated. Since the theoretical framework is a main contribution, these issues matter considerably.

  2. The experiments do not yet provide strong support for the proposed architectural solution. On gemma-3-12b, the duplicate-frame control achieves higher verb accuracy than the ordered pair under the original prompt, suggesting that the improvement cannot be attributed to temporal information alone. The positive result with one-second frame spacing is mainly observed on Qwen3.8-27B and does not transfer consistently to the other backbones. Also, the reported 82.7% direction accuracy is calculated only on gold put/take frames where the model itself predicts put or take. This conditional evaluation should be interpreted together with coverage and a majority baseline computed on the same subset. At the end-to-end level, AXIS remains close to its controls. The oracle ledger helps, but this supports the diagnosis of the bottleneck more than the effectiveness of the proposed method.

Questions for authors
  1. Can the authors clarify the exact distinction between information available to the write emission and information recoverable from the complete memory record? In particular, does the proposed constraint on the address algebra still hold when a system stores an ordered sequence of single-frame states and derives events at read time? I think this is important for evaluating the generality of the theoretical contribution.

  2. Can the authors clarify the assumptions behind the detector-union recall bound in Proposition 8? The stated inequality appears to require independent or non-negatively correlated miss events. Without this assumption, detectors with complementary misses can achieve a higher union recall than the proposed bound. The authors should either state the required assumption or revise the bound.

  3. For the one-second spacing result, can the authors report direction accuracy and coverage on the complete set of gold put/take frames, along with a comparison on the same subset of examples across conditions? It would also help to compare against a system that writes single-frame object statuses and performs deterministic temporal differencing afterward, without requiring an ordered pair as model input.

Reviewer_NMwdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the role of the write process in long-horizon embodied multimodal memory. The authors formalize a memory architecture as a pair of a write map and an address algebra, and decompose the capability of conjunctive queries into three factors: event identity, ordinal localization, and perceptual reading. Based on this formulation, they identify two potential bottlenecks: single-frame writes may be insufficient to distinguish temporally reversed actions, such as putting versus taking, while imperfect event detection can cause errors to accumulate when retrieving the k-th occurrence of an event. The authors propose AXIS, an architecture that uses ordered frame pairs to track object states, derives events through deterministic state differencing, stores occurrence-level frame indices, and executes explicit addresses to retrieve relevant frames. The method is evaluated on eight EPIC-KITCHENS-100 video streams using three frozen vision-language backbones, with additional validation on TempCompass. The results reveal important limitations of the proposed approach. Controlled experiments show that the apparent benefit of paired-frame writing at one-action spacing can be explained by the enumerated verb vocabulary rather than temporal information. A positive directional-recognition result is observed for one backbone when frames are sampled one second apart, but this finding does not generalize consistently across backbones. In end-to-end conjunctive question answering, AXIS achieves 18.8% accuracy with its automatically generated ledger, compared with 31.5% when an oracle ledger constructed from ground-truth annotations is used. The corresponding target-frame hit rates are 2.3% and 82.8%, respectively. Overall, the paper contributes a formal framework for analyzing write-side and retrieval-side limitations and a set of diagnostic experiments that identify event detection reliability as a major bottleneck. However, the experiments do not establish a consistent end-to-end advantage for AXIS, and the practical effectiveness and generality of the proposed architecture remain insufficiently demonstrated.

Critical strengths
  1. The paper identifies an important issue in long-horizon video memory: errors in event detection can severely undermine ordinal queries such as retrieving the k-th occurrence of an event. The controlled experiments and oracle-ledger comparison provide useful evidence for diagnosing this bottleneck.
  2. The explicit address-based retrieval mechanism is interpretable and easy to inspect. The comparison between the automatically generated ledger and the oracle ledger helps distinguish indexing failures from downstream perception and reasoning limitations.
Critical weaknesses
  1. The core novelty relative to existing methods remains insufficiently established. The authors explicitly acknowledge that state differencing, occurrence lists, and ordinal addressing have precedents. The novelty therefore rests primarily on the write-window formalization, the identity wall, and the product decomposition of capability. However, it remains unclear how substantially these results go beyond local-observation identifiability constraints and conditional-probability factorization. In particular, the paper permits events to be recovered by comparing current and historical states, leaving an important distinction between limitations of a local emission and those of a complete stateful memory system. Moreover, the experiments lack controlled comparisons with existing event-indexing approaches under the same detector, write budget, and query set, making the independent benefit of the AXIS design difficult to assess.

  2. The diagnostic experiments identify a bottleneck but do not establish that the proposed architecture effectively addresses it. On the v3 conjunctive QA setting with gemma-3-12b, the automatically generated ledger achieves only a 2.3% target-frame hit rate and 18.8% answer accuracy, close to the 17.2% blind baseline. Replacing it with an oracle ledger raises these figures to 82.8% and 31.5%, respectively. This supports event writing as a major bottleneck, but does not establish AXIS as a reliable practical solution. Furthermore, Table 8 reports only 32.9% answer accuracy conditional on reaching the target frame under the oracle ledger. The claim that the entire end-to-end loss lies in the write process is therefore too strong: the experiments demonstrate substantial gains from improving the ledger, while downstream reading and address execution retain errors that have not been fully disentangled.

3.Empirical support for the write-window mechanism is limited and has not translated into consistent end-to-end gains. At one-action spacing, the duplicate-frame and reversed-pair controls weaken the attribution of the improvement to temporal information; the paper instead attributes it primarily to the prompt’s enumerated verb vocabulary. Above-majority-prior directional recognition at one-second spacing appears on only one backbone, and the corresponding spacing contrast does not survive multiple-comparison correction. Although TempCompass shows positive gains on qwen3.8-27b, these gains are not consistently replicated across backbones and have not been shown to improve long-horizon conjunctive QA. The generality and practical value of the write-window design therefore remain insufficiently demonstrated.

Questions for authors
  1. Definition 1 allows the write map to access the historical record, but defines w-writability through a local emission that excludes dependence on that record. The paper also acknowledges that events can be recovered by comparing current and historical states. What fundamental expressive advantage does AXIS offer over a system that writes states one frame at a time, preserves their order, and subsequently applies deterministic differencing? Please clarify whether the identity wall constrains the local emission or the complete memory system, and what additional capability explicit paired-frame writing provides under matched information and budgets.

  2. Could you compare AXIS against a standard timestamped event list with ordinal retrieval while holding the event detector, keyframes, write budget, and query set fixed? The comparison should isolate the contributions of state differencing, the occurrence ledger, and explicit address execution, establishing whether any gains arise from detection quality, information retention, or the proposed architectural design.

  3. The oracle ledger still achieves only an 82.8% target-frame hit rate, and accuracy conditional on reaching that frame is only 32.9%. Could you add a reading control that directly supplies the ground-truth target frame, and separately quantify errors from ledger construction, address generation and execution, and target-frame reading? Without these controls, how is attributing the entire end-to-end loss to writing justified, beyond identifying ledger quality as a major bottleneck?

  4. Appendix C.2 collapses consecutive identical events into one occurrence. How does this rule distinguish a persistent description of one action from genuinely repeated consecutive actions? Could you quantify erroneous merges and duplicate counts, then report precision, recall, and exact k-th localization accuracy stratified by video length, event frequency, and k, together with measured memory usage, writing cost, and query latency for the full ledger?

Reviewer_Rd6MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This article studies the long-term memory of embodied multimodal agents, pointing out that existing methods mainly optimize memory retrieval, but ignore how information is written. The author proposes a theoretical framework to illustrate that it is difficult to distinguish actions with opposite directions such as "put down/pick up" in a single frame, and the detection error of repeated events accumulates with the query number. Based on this, the author designed AXIS, which uses continuous frame state changes to record events and conducts precise retrieval through event indexing. The experiment ultimately found that the real bottleneck lies in the event detection capability of the visual model, rather than the memory storage or retrieval mechanism; AXIS itself has failed to achieve reliable end-to-end improvement.

Critical strengths
  1. Decompose the memory architecture into two parts: writing and addressing, and propose an identity wall and an ordinal wall, providing valuable insights for analyzing the bottleneck of multimodal memory capabilities.

  2. Designed control experiments such as repeated frames, reverse frames, incorrect addresses, and oracle memory, and transparently reported results that did not meet expectations, enhancing the credibility of the research.

Critical weaknesses
  1. Hallucination references (first 6 references).

  2. Identity walls are mainly suitable for independently generated records frame by frame, but online systems that use single-frame writing can still calculate action changes based on historical states. Therefore, the applicability of the theory to general memory architectures is limited.

  3. AXIS The performance on core tasks is close to the random baseline, and most pre-registration assumptions are not supported. The main performance improvement relies on the non-deployable Oracle Ledger, and the actual effectiveness of the proposed architecture has not yet been proven.

Questions for authors

Please see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.