Which Memory Applies Here? Organizing Long-Term Multimodal Agent Memory as Conditional Hierarchies in Hyperbolic Space

Long-term memory lets a multimodal agent carry what it has observed about the entities in its world across sessions, and many observations are general facts accompanied by exceptions that hold under particular conditions. Existing memory systems treat such coexisting records as conflicts and resolve them by recency or by similarity, so the record that applies to the current query image is discarded at write time or outranked at read time. We propose ConeMem, which organizes observations as specificity-ordered conditional hierarchies in the Lorentz model of hyperbolic space, where a general fact sits near the origin and its exceptions branch outward inside entailment cones. The geometry is load-bearing; the read and write rules need specificity as a coordinate, the region of an exception as a cone fixed by that coordinate, and nested regions that shrink by a constant factor per level, and we prove that negative curvature supplies the three within a bounded radial range whereas zero curvature spends a radial budget that grows multiplicatively with depth. An observer built on frozen vision-language encoders maps the image region selected by the condition text of an exception into the same space and reads applicability off its position relative to the cone, so the choice among coexisting records is a geometric test with no language model call, and a local write procedure uses the same geometry to reinforce a fact, open a branch, or wait until an observation recurs. Under a leave-one-benchmark-out protocol, ConeMem raises macro-average judge accuracy on four public long-term multimodal memory benchmarks by 13 to 17 points over the strongest published baseline with two reader families at the lowest answer-stage cost among the compared systems, comes within three points of exhaustive per-exception verification on annotator-verified conditional-conflict questions, and loses its lead over Euclidean cones as the curvature is driven toward zero.