acceptodds
Under review as a conference paper at ICLR 2027

Which Memory Applies Here? Organizing Long-Term Multimodal Agent Memory as Conditional Hierarchies in Hyperbolic Space

Abstract

Long-term memory lets a multimodal agent carry what it has observed about the entities in its world across sessions, and many observations are general facts accompanied by exceptions that hold under particular conditions. Existing memory systems treat such coexisting records as conflicts and resolve them by recency or by similarity, so the record that applies to the current query image is discarded at write time or outranked at read time. We propose ConeMem, which organizes observations as specificity-ordered conditional hierarchies in the Lorentz model of hyperbolic space, where a general fact sits near the origin and its exceptions branch outward inside entailment cones. The geometry is load-bearing; the read and write rules need specificity as a coordinate, the region of an exception as a cone fixed by that coordinate, and nested regions that shrink by a constant factor per level, and we prove that negative curvature supplies the three within a bounded radial range whereas zero curvature spends a radial budget that grows multiplicatively with depth. An observer built on frozen vision-language encoders maps the image region selected by the condition text of an exception into the same space and reads applicability off its position relative to the cone, so the choice among coexisting records is a geometric test with no language model call, and a local write procedure uses the same geometry to reinforce a fact, open a branch, or wait until an observation recurs. Under a leave-one-benchmark-out protocol, ConeMem raises macro-average judge accuracy on four public long-term multimodal memory benchmarks by 13 to 17 points over the strongest published baseline with two reader families at the lowest answer-stage cost among the compared systems, comes within three points of exhaustive per-exception verification on annotator-verified conditional-conflict questions, and loses its lead over Euclidean cones as the curvature is driven toward zero.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Agents, Long-Term Memory, Hyperbolic Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xpF6OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies long-term memory for multimodal agents in the setting where a general fact about an entity coexists with exceptions that hold under conditions visible in a query image (a "conditional conflict"). It argues that prior systems lose the applicable record either at write time (overwrite, expiry, or merge) or at read time (similarity ranking), and proposes ConeMem, which stores each entity as a subtree in the Lorentz model of hyperbolic space: the general fact sits near the origin and exceptions branch outward inside entailment cones whose apertures are fixed by position alone. A trained observer (frozen SigLIP 2 encoders, one cross-attention block, and a projection) maps the image region selected by an exception's condition text to a point in the same space, and applicability is read as a cone margin, so the choice among coexisting records requires no language-model call. A local write procedure (reinforce, branch, or pending, with containment repair and bounded replay) keeps the hierarchy consistent, while a frozen text-only reasoner performs extraction, same/contrary labeling, entity naming, and answering. Proposition 1 shows that nesting consumes an additive radial budget under negative curvature and a multiplicative one in Euclidean space. Under a leave-one-benchmark-out protocol on ATM-Bench, Mem-Gallery, MemEye, and H2HMem with two reader families, ConeMem improves macro-average judge accuracy by 13 to 17 points over the strongest published baseline at the lowest answer-stage cost of the compared systems, comes within 2 to 3 points of exhaustive pairwise verification, and is accompanied by extensive ablations, three Euclidean controls, a curvature sweep, controlled depth and branching experiments, an oracle decomposition, and sensitivity, scale, and label-noise studies.

Critical strengths

S1. Well-posed problem with a crisp diagnosis. The conditional-conflict failure mode is formalized as three requirements, connected to the classical specificity preference of nonmonotonic reasoning, and the predicted error signature is verified on annotator-labeled questions. This is a good motivation.

S2. The design is coherent, and each component is tied to a requirement. The "no language-model call for applicability" property is real, and the cost accounting is complete.

S3. The evaluation is broad and mostly careful: four benchmarks, two reader families plus a third text-only reasoner, leave-one-benchmark-out with two alternative protocols.

S4. The "geometry is load-bearing" claim is tested rather than asserted: three Euclidean controls that keep the tree, observer, and training. a curvature sweep, controlled depth and branching subtrees, and cone diagnostics showing that apertures become level-dependent only after write training.

Critical weaknesses

W1. Hyperparameter selection appears to use the evaluation benchmarks. In every row of Table 12 the default value is the argmax of the full average over the four evaluated benchmarks. Leave-one-benchmark-out protects the observer's weights but says nothing about these constants, and Section 4.1 only states that the same values are used on the four benchmarks. Were the constants chosen on the training benchmarks' questions, on a validation split, or on the reported test questions? This could be a serious problem. Please explain.

Questions for authors

See Weakness please. Does hyperparameter selection use the evaluation benchmarks? If so the results of the proposed method might be untrustworthy. Please explain my only question honestly and thoroughly. Thanks in advance.

Reviewer_XTXaOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates long-term multimodal agent memory as a problem of organizing conditional evidence. As pointed out by the authors, SOTA methods fail because they write by overwriting and read by similarity. To overcome these weaknesses, the authors propose ConeMem, memory organized as specificity-ordered conditional hierarchies in a hyperbolic space. ConeMem was compared experimentally against five SOTA methods on a range of benchmarks. ConeMem substantially and consistently outperforms the baselines for two different LLMs as readers/writers.

Critical strengths

Originality and technical quality of the proposed memory organization

  • The authors present the first memory organization, called ConeMem, that represents the condition under which a fact applies as geometry.
  • The Lorentz model of d-dimensional hyperbolic space with curvature −K is used for the memory space with conditions modeled as entailment cones. This solid mathematical foundation allows the authors to prove certain desirable properties of the memory space, i.e. that the radial budget of nested cones is additive under negative curvature and multiplicative under zero curvature, and that the gain over Euclidean cones grows with depth and branching and vanishes as the curvature approaches zero.

Comprehensive experimental evaluation with convincing results

  • ConeMem was compared experimentally against five recent SOTA methods on four different benchmarks.
  • ConeMem clearly outperforms the baselines in terms of judge accuracy and in the native metric, with a macro-average greater by roughly 15 points. I note that the gains are consistent for two different LLM backbones and for held out validation cases.
  • Comprehensive analyses provide insight why and when ConeMem outperforms that baselines.
Critical weaknesses

None.

Questions for authors

How did you choose the baselines for your experimental evaluation?

Why do some of the chosen baselines not appear in table 5, which surveys SOTA methods?

Reviewer_8i9yOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ConeMem, a long-term multimodal memory system that retains general facts alongside exceptions valid only under specific conditions. Existing systems often overwrite consistent entries or retrieve them via textual similarity, causing agents to invoke facts mismatched with the current image. ConeMem organizes each entity's memories into a conditional hierarchy in Lorentz hyperbolic space, where radial distance encodes specificity and nested entailment cones define applicability. An observer built on frozen vision-language encoders uses condition-guided cross-attention to select image regions and matches them against the cones; a frozen reasoner then generates answers using applicable evidence, prioritizing more specific facts. Local writes reinforce existing records or create exception branches, while bounded replay safeguards the validity of existing answers. Theoretical analysis shows negative curvature yields additive radial growth for nested cones, unlike Euclidean geometry. Across four benchmarks, ConeMem improves macro-average judge accuracy by 13–17 percentage points over the strongest published baseline and achieves the lowest answer-stage cost, though training overhead and errors in condition visibility, visual grounding, and node assignment remain limitations.

Critical strengths

(1) The paper focuses on how to choose between a general fact and a condition-specific exception when both coexist in memory, and clearly explains the limitations of overwrite-based updates and similarity-based retrieval.

(2) The condition-driven visual observer and hierarchical memory structure align well with the problem studied in the paper. Component-replacement experiments further suggest that both parts contribute meaningfully to the overall performance.

(3)The consistent improvements across four benchmarks, different answering models, and leave-one-benchmark-out training provide solid evidence for the method’s effectiveness. Although the write-time decisions and continual-update behavior could be evaluated more directly, the current results still indicate a reasonable degree of generalization.

Critical weaknesses

(1) The evaluation of write-time decisions is mostly indirect.The experiments mainly measure final QA accuracy, leaving it unclear how often the system correctly creates an exception, replaces an outdated fact, or keeps an observation pending. Direct evidence here would strengthen support for the core memory mechanism.

(2) The continual-update evaluation does not fully separate retention from appropriate updating. Preserving old answers may reflect either robust memory or overly conservative updates, especially when a fact has genuinely changed. A clearer joint assessment of stability and plasticity would make the dynamic-memory claim more convincing.

Questions for authors

(1) Could you report operation-level results for the write mechanism, such as the accuracy or confusion matrix for creating an exception, replacing a fact, and keeping an observation pending? A brief error analysis would also be helpful.

(2) Could you jointly report the update success rate for changed facts and the retention rate for unaffected knowledge? It would also be useful to know how often replay preserves an answer that should have changed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.