Piecing It Together: Harmonizing Participant-Grounded Memory in Multi-Party Dialogues

As large language model (LLM) agents increasingly participate in large-scale collaborative projects, memory mechanisms for multi-party dialogues have emerged as a new challenge. However, existing works primarily rely on semantic-based retrieval from a two-party tradition, struggling to piece together a holistic view across groups and time. To address this challenge, we propose Pit from a participant-grounded perspective, which reframes multi-party memory retrieval as completing a query-specific puzzle by assembling scattered pieces. Specifically, we first construct an event–participant bipartite graph to derive participant-grounded event representations, which are further consolidated into coherent memory pieces. A query-specific router then explores their underlying dependencies and harmonizes them together into a coherent memory view. Extensive experiments on EverMemBench demonstrate consistent improvements across different LLMs, outperforming the strongest baselines by 6.71%–19.91%. Further analyses show that participant grounding generally improves event representations and retrieval reliability. The code can be available at https://anonymous.4open.science/r/PIT-E7F1https://anonymous.4open.science/r/PIT-E7F1.