acceptodds
Under review as a conference paper at ICLR 2027

μWM: Fixed-Budget World Memory for Long-Running Wearable Assistants

Abstract

Wearable assistants require reliable long-horizon interaction history under strict kilobyte memory constraints, yet existing systems either lose joint spatio-temporal conditions through decoupled counters or incur unbounded storage growth in raw event logs. We introduce WM, the first wearable world memory framework tracking joint action-, place-, and time-conditioned interaction history under a strict budget of a few tens of kilobytes. Operating locally on microcontrollers (MCUs), WM compiles incoming semantic streams into typed state at write time, prunes duration detail via query-loss allocation while preserving counts and temporal endpoints, and resolves queries directly using deterministic readers with state-directed evidence routing. Evaluated across 14 query families, WM achieves 86.90% accuracy at 13.38 KiB on EPIC-KITCHENS-100, outperforming equal-budget controls while using 122.6 less storage than SOTA baselines. Furthermore, it recovers 99.1% raw-log fidelity on AMB, maintains full accuracy across 32 extended ALFRED trajectories within 28.3 KiB, and fits entirely inside 64 KiB Data Tightly-Coupled Memory (DTCM) on an STM32F7 MCU at 153 to 173 mW peak power. These results establish structured state compilation as an efficient, privacy-preserving memory substrate for edge embodied intelligence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
wearable assistants, long-term memory, edge AI, microcontrollers

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_VcymOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes µWM, a hand-designed compact memory for wearable assistants that stores typed state instead of raw event logs: per-entity counters, first/latest timestamps, action/place conditional tables, a weighted temporal coreset, and a dormant tier, all packed into a fixed byte layout under a nominal 16 KiB cap. Queries are answered by deterministic readers, and a "query-loss" heuristic drives which precision to shed when the budget is exceeded. The engineering is unusually concrete — field widths are specified to the byte (Tables 6–7), update procedures are given as pseudocode, and the evaluation protocol uses a frozen design/confirmation split with paired, video-clustered bootstrap intervals. This is the paper's real strength: it is a careful systems measurement paper.

Critical strengths

S1: The protocol discipline is better than the norm for this sub-area. S2: Byte accounting is auditable. S3: Negative and inconvenient results are disclosed. S4: The costs of the alternative are measured, not asserted.

Critical weaknesses

The problem is that the paper does not present itself as a systems measurement paper. It presents itself as a new memory paradigm ("world memory", "theoretical boundaries", "first wearable world memory") whose empirical claims are stronger than the evidence. Three examples, all checkable inside the paper. First, Section 1 states µWM matches "the fidelity of recent memory systems requiring over 122.6× larger storage", yet Table 2 reports those systems at 100.00% and µWM at 86.90%. Second, no published memory system is ever run at µWM's budget; the "equal-budget control" is the authors' own joint table, allocated only four cells at 13.35 KiB — an allocation no competent engineer would defend as the best use of 13 KiB. Third, the accuracy numbers that anchor the abstract are produced with Qwen2.5-VL-7B in the loop, while the paper's selling point is deterministic on-device readers with no language model.

Beyond framing, there is a scale problem. The motivating scenario is a wearable running for weeks; the longest stream evaluated is 21,594 events (roughly six hours at one event per second), and the only "long-horizon" test is a synthetic 32× tiling of ≤1.3k-event ALFRED trajectories over a 29-noun vocabulary that fits entirely in the 64-slot state. In that test the memory never has to evict anything, so "full accuracy" is a property of arithmetic, not of memory design. Meanwhile the deployed footprint on hardware is 33–46 KiB, not the 16 KiB the paper keeps claiming, on a 320 KiB-SRAM Cortex-M7 that is five times larger than every wearable in Table 1.

Questions for authors
  1. In Table 2, SeCom/A-Mem/HippoRAG 2 are at 100.00% and µWM at 86.90%. What precisely does "matching the fidelity of recent memory systems requiring over 122.6× larger storage" (Section 1) mean given that table?
  2. Why does the 13.35 KiB joint table allocate only 2–4 cells, and why do the 2-cell and 4-cell variants consume identical bytes? How many (action, place) records fit in 13,670 bytes at your record width, and what accuracy does that configuration reach?
  3. Which numbers in the paper are produced by deterministic readers alone, and which require Qwen2.5-VL-7B? Please give the reader-only column for all fourteen families.
  4. In the 32× ALFRED experiment, how many distinct nouns and distinct (action, place) conjunctions appear, and does the state ever demote, merge, or evict? If not, what does the experiment test beyond Proposition 1?
  5. What is the operational definition of F_S(t) in the allocation loss? What does the device compare against when no raw log exists?
  6. Which budget is the paper's design point — 13.38 KiB, 16 KiB, 28.3 KiB, or 33–46 KiB? What would the system look like on a 64 KB-SRAM device with no DTCM?
  7. What event rate and measurement method produced 153–173 mW, and what is the energy per ingested event compared with buffering events to flash?
  8. Where does the 3.1-second mean duration come from, and was it estimated on data that overlaps the evaluation set?
Reviewer_uRugOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a bounded-memory system for wearable assistants that stores structured events as counts, timestamps, and action/place summaries. It evaluates how well these summaries support queries under a fixed memory budget.

Critical strengths
  • Addresses a practical problem: retaining useful interaction history when storing the full history is no longer possible.
  • Includes a microcontroller implementation and comparisons with compact structured-memory baselines.
  • Shows improvements over the joint-table baseline, particularly for conditional queries.
Critical weaknesses
  • The novelty is not clear to me. The paper combines familiar aggregation and retention mechanisms, but I have difficulty identifying which design choice accounts for the improvements over the compact structured baselines. The paper should explain more clearly what new insight comes from this combination and which results support it. The theoretical results help explain the limitations of bounded memory, but do not fully address this concern.
  • The benefit of adaptive allocation is unclear. Query-directed precision allocation seems to be a central contribution, but the fixed-precision variant achieves slightly higher overall accuracy at the same reported memory size in Table 2 (87.65% versus 86.90%). There may be settings where adaptation helps, but the paper should show more clearly when and why this is the case.
  • The memory comparisons need careful interpretation. Part of the large reduction compared to text and graph memories seems to come from storing information specifically for the predefined query families. The comparisons with compact structured summaries are therefore more useful for understanding the contribution. The record-matched comparison in Figure 6 uses different byte budgets. I would therefore put more weight on the equal-byte comparisons when assessing the memory-efficiency claim.
  • The long-history benchmark does not fully support the intended use case. Appendix F.2 extends ALFRED histories by repeating existing sequences. This shows that the system can retain accumulated statistics over longer streams, but does not tell us much about what happens when new objects, places, or interaction patterns are introduced. I would expect these changes to be important for a wearable assistant used over time.
  • As a secondary concern, most experiments start from structured events with supplied identities, action labels, and temporal boundaries. The recognition experiment replaces noun IDs with predictions but keeps the other inputs unchanged. This limits how directly the results support the broader wearable-assistant use case. A clearer discussion of which input errors the memory can tolerate would help.
Questions for authors
  • What new insight does the proposed combination of aggregation and retention methods provide, and which comparison most clearly supports it?
  • When does adaptive precision allocation improve on fixed precision, and what explains the results in Table 2?
  • What evidence supports performance on longer histories that introduce new entities and places, rather than repeating the same sequences?
Reviewer_CdJeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents µWM, a fixed-budget world memory for long-running wearable assistants. It compiles semantic events into typed state, preserves joint action-, place-, and time-conditioned history, and uses query-loss allocation to choose between duration demotion, temporal merging, and record deletion. Structured queries are answered through deterministic readers, while state-directed evidence routing supports requests requiring external evidence. Experiments cover EPIC-KITCHENS-100, AMB, and extended ALFRED trajectories, with on-device profiling on an STM32F7 MCU. On EPIC-KITCHENS-100, µWM achieves 86.90% accuracy across 14 query families at a 13.38 KiB checkpoint.

Critical strengths

●The memory representation preserves joint action, place, and time conditions, supporting queries that independent counters cannot answer. The equal-budget comparisons show that this design improves conditional-history retrieval. ●The byte-level layouts and on-device profiling make the system practical and verifiable. The hardware evaluation supports the feasibility of maintaining this memory on a constrained MCU.

Critical weaknesses

●It is hard to tell how much query-loss allocation contributes. The comparisons in Tables 2–3 change the memory representation and compaction operations as well as the allocation policy. The gains could therefore come from the typed representation rather than the selection rule. Section 2.1 also scores each operation against the current compressed state. It is unclear how well this score reflects answer errors against the full history. ●Repeating the same trajectories does not capture the variety of a long-running history. Section 3.2 reports perfect retention at 32× duration, but the ALFRED histories repeat the same source trajectories with unchanged vocabulary and within-copy event order (Appendix F.2). Existing counters can absorb these repetitions. The experiment does not establish whether the same budget remains sufficient as new entities and action–place combinations arrive.

Questions for authors

●How does retention change under a fixed budget when new entities and action–place combinations keep arriving, rather than when the same trajectories are repeated? Do any existing experiments cover this setting? ●Is the query mix used to guide allocation fixed in advance? How does accuracy change if users later ask more duration-sensitive questions after duration details have been discarded?

Reviewer_LTx6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

µWM addresses a concrete problem: how to keep useful records when a wearable device can have short amount memory such as tens of kilobytes. They used a stream of semantic events as input; an entity, an action, a place, and time window start and end. Questions are structured and deterministicly answered by readers. Questions are from a known family of future questions, and µWM decide what details keep in the limited storage when memory is full (it can decide keep counts while drop exact durations).

Critical strengths

This decision to keep useful statistics while events arrive is very useful in on-device wearable sensing methods that face real world limitations like low storage and processing units and need reply fast. So this study is timely.

When user ask how many times I did x no need to check raw files with a lots of lines of info on each raw file. For queries like how many times, when first occurrences, when latest occurrences, or most visited locations, having the relevant state is a wise way to use a limited memory space.

In my view the equal budget comparison to compact baseline is good one. The good result here came on EPIC dataset under approximately equal checkpoint size conditions. That are accuracy of 86.90% at a size of 13.38 KiB, compared to 82.14% accuracy at 13.35 KiB for the authors' joint-table baseline. The gain of this comparison is 4.76 percentage points, with a 95% confidence interval of [1.93, 7.59].

Critical weaknesses
  1. On the AMB test, full event log is max 9.1 KiB, that smaller than µWM 28.9 KiB state, and at least as accurate, 0.514 vs 0.509. So this test shows µWM keeps that raw log's accuracy but does not show it save memory. In appendix A4, authors report that µWM only become smaller rather than raw log after about 2300 events and they matched record counts rather than bytes. However, introduction ("outperforms the strongest episodic baseline by 0.139"), and page 7 present the AMB result as overclaim ("an 8× capacity expansion to M=512"), and the AMB comparisons in Figure 6, Table 12 and Table 14 are matched by record count, not by bytes.

  2. ALFRED baselines just save sampled events and no counters. So their 0 on count questions was expected by design. And counter keeping baseline from table 2 should have been included in the comparison.

Questions for authors
  1. Is it possible for authors to add the comparison at equal bytes in the rebuttal? And the text should state that µWM keeps the raw log's accuracy on AMB but does not save memory there.

  2. Does Proposition 1 hold only while an entity stays in the active table? If yes, will the authors add this condition to the proposition, or state which questions it covers? Why I ask: Proposition 1 promise exact counts when no counter overflows. However, Proposition 1's proof and table 5 show action rows and per-entity place and partner counts reset when an entity is moved to dormant storage and restored, so the proposition should be narrowed. I mean when the active table is full, the least-recently seen entity moves to dormant storage to free up memory; if it show up again, it restores its total count only if its dormant row still exist but starts its per action counts from scratch again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.