CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

Recent benchmarks rely on user simulators to evaluate AI agents in multi-turn interaction. While existing simulation techniques demonstrate surface fidelity to human style and behavior, ecologically valid interactive benchmarking also requires alignment in when and how agents fail across simulated and real user populations. We find that established measures of simulator naturalness and mimicry provide inconsistent signals about whether simulated interactions reproduce real-user outcomes. Therefore, interactive benchmarking requires a complementary criterion, outcome calibration: agreement with observed success rates, conditional failure patterns, and retrospective user-task outcomes. We introduce Calibrated User Embeddings (CUE), a framework that encodes observed sessions or samples continuous representations, then decodes them into persona commands augmented with retrieved examples to steer a black-box simulator LLM. Through this, we evaluate both user-conditioned replay of past sessions and aggregate metric agreement when sampling novel personas for the same tasks. On ${\tau }^2$-Bench, CUEd simulators commit fewer simulator-attributed errors and more faithfully reproduce real-user agent failure modes, aggregate success rates, and outcomes for specific task-user pairs than other persona-based simulation methods. These gains coexist with competitive user fidelity as measured using metrics established in prior work. After being fit to mostly customer support interactions, the same CUEd simulators generalize to document creation, math tutoring, and casual conversation, and remain effective across different simulator LLMs without CUE retraining.