HiMem: A Hierarchical Multi-Granular System for Long-term Memory

Long-term memory is critical for interactive agents to retain and utilize information accumulated over extended conversations. However, structurally organizing fragmented memories and accurately retrieving them remain significant challenges, particularly when relevant evidence lacks direct semantic similarity to the query. Existing systems still struggle to build a structural framework for evolving interaction records and to leverage structural information to bridge the semantic gaps. To address this challenge, we propose **HiMem**, a **Hi**erarchical Multi-granular **Mem**ory System with structure-aware multi-route retrieval, which organizes evolving memories and source records across multiple granularities and exploits their structural relations to recover complementary evidence. Specifically, HiMem organizes memories around verified anchors and constructs a multi-granular graph that preserves provenance of source records, building a foundation for multi-route retrieval. During retrieval, HiMem collects complementary candidates through multi-route retrieval and re-ranks them under a unified scoring mechanism that integrates semantic and structural signals. Experiments on LoCoMo show that HiMem achieves 92.60% average accuracy, outperforming the leading memory baseline, EverMemOS, by 2.15 percentage points. Furthermore, HiMem matches the performance of the full-context prompt method, while consuming merely 20.56% of the token cost.