Remember the Alternatives: Choice-Context Memory for Personalized Decision Agents

An embodied agent receives the instruction sit somewhere in a living room. Several seats may all be reachable, safe, and consistent with the instruction. Once these explicit constraints are satisfied, the remaining ambiguity is personal: which seat does this user prefer? Past choices can help resolve it. Choosing a nearby wide chair over a distant chair may suggest that distance matters in that comparison, while choosing the same chair over a nearby narrow seat may instead provide evidence for a preference over seat width. These observations do not uniquely identify the user's preference, but they carry different information about it. If memory stores only the selected wide chair, the two interactions become indistinguishable. Recent personalized agents increasingly use persistent memory to adapt across interactions. Most work focuses on how stored experiences are organized, updated, and retrieved. We instead ask what information a choice event should preserve before it enters memory. We identify a failure we call choice-evidence aliasing: different choice contexts can collapse into the same stored outcome, even when they provide different evidence about user preferences and can support different future actions. We formalize the information lost by this compression and show that it can change a later optimal decision. This motivates choice-context memory, which preserves the selected target together with the alternatives. Building on this principle, we introduce AffordanceMem, a personalized decision agent for human-scene interaction. It combines current instruction and feasibility constraints with preference beliefs inferred from past choice context, then decides whether to execute a target, ask for clarification, or defer. Clarification responses will be stored as new choice-context memories. We evaluate the framework on HUMANISE-ScanNet and LINGO with controlled preferences and repeated interactions. Matched comparisons and targeted interventions show that preserving choice context improves later personalized decisions and can change whether the agent acts or asks. These effects persist across choice mechanisms, interaction sessions, and scene constructions. Our results support choice context as a decision-relevant memory unit for personalized agents.