acceptodds
Withdrawn from ICLR 2027

UMMI: Symbolic Time Binding for Bounded Multimodal Memory with a Frozen Language Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A bounded memory can give a frozen language model access to multimodal sensor recordings: a small network compresses the streams into a fixed number of embeddings that are spliced into the model's input. We find that such memories support questions about what happened far better than questions about when. The limitation is not the reader alone: given the same facts as timestamped text, the same frozen Qwen3-Omni model places events in the correct interval with 77-78% accuracy without any training. We introduce symbolic time binding in UMMI, a learned memory interface. Memory embeddings are grouped by recording interval, and each group is prefixed, in ordinary text, with its interval index and step range. The backbone is not updated and the continuous memory is not enlarged; the prompt grows by sixteen short phrases. On a synthetic dense-event benchmark, interval localization rises from 60.7% to 98.0% over three training seeds, whereas learned interval embeddings added to the same tokens help much less. On question subsets from held-out OPPORTUNITY subjects, overall accuracy rises from 41.6% to 53.8% for the same shared-memory configuration, with gains that vary across subjects. A flat tokenizer given the same labels does not benefit. Time binding also fails to help in the tested Aria Everyday Activities configuration, where several transcript words share each interval. Explicit textual time labels are thus a simple and effective interface for reading time from a bounded memory, while preserving several items within one interval remains open.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal memory, bounded memory, frozen language model, temporal grounding, sensor foundation models, wearable sensors, omni-modal models, time-indexed memory, long-horizon recordings

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WZ1DOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes injecting textual timestamps when reading from multimodal memory to improve temporal localization.

Critical strengths

None

Critical weaknesses
  • The paper is full of excessive experimental details while lacking motivation and insights that are accessible to human readers. It has a typical AI-slop style. Since I am not familiar with this field, I could only partially understand what the paper is doing with the help of AI.

  • The proposed method appears to simply insert textual timestamps into the memory to improve temporal localization. This is already a fairly common practice in multimodal large language models, as the paper itself acknowledges in Lines 289–296.

  • The main experimental results are obtained on a deliberately designed synthetic benchmark. Although reporting negative results should be encouraged, the proposed method appears to work on only two of the three real-world datasets. This may suggest that the proposed method is not generally effective.

  • Minor weakness: The reference formatting is a mess.

Questions for authors
  • How is the sensor tokenizer in Figure 1 obtained?

  • What does UMMI stand for?

Reviewer_xT4ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce symbolic time binding for a time-indexed store for LLMs by inputting memory embeddings into the LLM through learned embeddings / soft-prompting in combination with textual interval bounds already existent in the LLM's token space. The time-indexed store does allow for sensor, audio, vision, and text data. All of them are aligned on a time-scale ultimately testing, for time-related queries. The authors trained time-aware multimodal memory to answer time-related questions based on two LLM backbones.

These were evaluated on Aria Everyday Activities, PAMAP 2, and OPPORTUNITY. An ablation of the proposed tokenization has been made to identify where the improvements come from. While the proposed architecture seems to improve on PAMAP 2 and OPPORTUNITY, it is showing no clear benefit on Aria Everyday Activities.

Critical strengths
  • S1 An interesting approach to combine text, sensor, video, and audio data within one time-aware store to increase performance for time-related tasks.
  • S2 Ablation of the time symbols with different memory models and supervision modes to isolate the benefit.
Critical weaknesses
  • W1 Real-world multimodal evidence: Currently, the practical benefit of a multimodal time-aware store is insufficiently demonstrated in the paper. The authors introduce a new method that combines video, audio, time-series, and text on a shared time scale, however, the positive real-world results are demonstrated on time-series-only tasks. The positive, multimodal evidence have been demonstrated on synthetic data only. Hence, it remains unclear what practical value there is of a multimodal memory.
  • W2 Comparison to related models: Relevant models in the domain such as OpenTSLM [1], ChatTS [2], or ITFormer [3] are not compared against in this work. This limits the authors' argument and evidence for a benefit over the established sensor-to-language models.

[1] https://doi.org/10.48550/arXiv.2510.02410

[2] https://doi.org/10.14778/3742728.3742735

[3] https://doi.org/10.48550/arXiv.2506.20093

Questions for authors
  • Q1 How do existing time-series/sensor models compare against the proposed one?
  • Q2 What is the isolated benefit of having text, sensor, video, and audio data in one shared time-aware store compared to a sensor only time-aware store for sensor-to-language tasks?
Reviewer_t4dCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • Research Motivation: When a frozen Large Language Model (Frozen LLM) reads multimodal sensor recordings via bounded continuous memory, research reveals that the model excels at answering "What happened," but performs extremely poorly on "When it happened (temporal localization)."
  • Core Method (UMMI): The authors propose Symbolic Time Binding. While keeping the LLM completely frozen and the continuous memory budget fixed, memory embeddings are grouped into time intervals, and brief text labels are inserted via the model's own vocabulary before each group of embeddings (e.g., interval and step range markers like interval 3 (steps 64-95):).
  • Main Findings: With these explicit text-based time labels, the interval localization accuracy for dense events surged dramatically from 60.7% to 98.0%. Significant improvements were also achieved on real-world sensor datasets (such as PAMAP2 and OPPORTUNITY). However, in complex settings where a single time interval contains multiple events (e.g., Aria Everyday Activities), the method currently fails, indicating that "accommodating multiple interleaved events within a single interval" remains an open challenge.
Critical strengths
  1. Rigorous Diagnostics: Instead of blindly stacking models, the authors used carefully designed control experiments (such as having a zero-shot model read the exact same facts presented as timestamped text) to prove that the bottleneck does not lie in the LLM's inherent lack of temporal understanding, but rather in the "interface representation of continuous embeddings fed into the model's input."
  2. High Engineering Efficiency & Pragmatism: Without fine-tuning massive backbone networks or expanding the continuous memory capacity (maintaining a fixed budget of 64 tokens), the temporal localization capability was multiplied simply by adding 16 short text snippets to the prompt.
  3. Robust Temporal Fidelity Analysis: By introducing the "Temporal Fidelity" metric (Appendix D), the paper scientifically quantified information loss in the time dimension by the decoder, explaining why spatially pooled visual towers fail while audio towers perform well, which subsequently inspired time-slicing improvements.
  4. Diverse Benchmarks: Evaluated not only on synthetic dense event sets, but also tested across multi-dimensional chained tasks and multiple public real-world wearable sensor datasets (PAMAP2, OPPORTUNITY, Aria).
Critical weaknesses
  1. Multi-Item Limitation: In the Aria Everyday Activities dataset, when about 7 words were mixed into a single time interval, the symbolic time method completely failed (performing no better than a random baseline), exposing a blind spot of this approach when handling real-world scenarios with "high-density, multi-item interleaving."
  2. Uneven Real-World Gains: On real-world inertial sensor datasets like PAMAP2 and OPPORTUNITY, performance gains varied significantly across different subject splits, and the improvements were not universally dominant in all configurations.
  3. Text-Dependency & Scaffolding: The method fundamentally relies on the LLM's pre-trained comprehension of "natural language text symbols." If applied to non-English, non-standard formats, or lightweight models highly sensitive to prompt layout, the robustness of this symbolic binding could be challenged.
Questions for authors
  1. On "True Understanding" vs. "Pattern Matching" of Symbols:
  • Question: Does the model understand interval 3 (steps 64-95): because these words were seen during pre-training? If we "obfuscate" or "shuffle" these time labels (e.g., deliberately labeling text that actually corresponds to the 3rd interval as interval 99), will the model's retrieval and localization capabilities collapse entirely? Is the model truly performing spatio-temporal reasoning, or is it merely engaging in shallow pattern matching based on text symbols?
  1. On the Scalability of Multi-Event Interleaving (Follow-up to the Aria Failure):
  • Question: The failure on the Aria dataset was attributed to "multiple words sharing the same time interval." If we want to extend this symbolic binding to finer-grained, event-dense scenarios in the future, would it require introducing dynamically variable-length time slices or hierarchical retrieval? Have you attempted to let the model learn when to split intervals autonomously rather than using hardcoded, uniform 16-interval splits?
  1. On the Limitations of a Fully Frozen Backbone:
  • Question: The paper insists on keeping the backbone network completely frozen. If ultra-lightweight parameter fine-tuning of the LLM were permitted (such as full LoRA or targeted tuning specifically for the interface positions), could this completely break the curse of "continuous embeddings failing to capture absolute time," thereby eliminating the need for the "crutch" of text symbols?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.