- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PPol, a method that generates and iteratively optimizes personas to control user simulators. It uses behavioral features from real user conversations to calibrate simulated interactions and evaluates whether the resulting data improves agents’ ability to handle diverse user behaviors. However, persona injection is already a common control strategy for user simulators in industry. The paper’s contribution therefore depends on whether its personas better reflect real users, whether simulators consistently follow them, and whether they deliver greater downstream benefits than alternative persona methods. The experiments provide some encouraging evidence, but do not fully establish these three points.
Critical strengths
The paper connects user simulation to downstream agent performance. Beyond presenting generated personas, it incorporates behavioral features from real conversations into optimization and conducts both agent-training experiments and human interaction studies. This extends the evaluation beyond the quality of generated text and provides preliminary evidence of practical utility.
Critical weaknesses
-
The realism of generated personas remains insufficiently validated. LLM-generated personas may reflect the model’s assumptions about how people behave rather than actual human preferences and reactions. Behavioral statistics and human judgments provide some supporting evidence, but do not sufficiently establish alignment with real users in concrete decisions, preference trade-offs, and context-dependent responses.
-
Injecting a persona does not guarantee behavioral adherence. The method uses prompting and interaction feedback to guide behavior, but lacks an independent, systematic evaluation of persona adherence. It remains unclear whether specified behaviors appear when relevant and remain consistent throughout extended interactions.
-
The downstream comparisons do not sufficiently establish the specific advantage of PPol. The paper compares training on base-simulator data with training on data augmented by PPol, but does not include equivalent downstream training comparisons using simple persona prompting, comparable persona-generation methods, or relevant state-of-the-art approaches. Consequently, the results do not adequately distinguish the benefits of introducing persona diversity from those of PPol’s specific optimization mechanism.
-
Downstream generalization is evaluated primarily within the same task environments. Although training and evaluation tasks are separated, both largely remain within the Retail and Airline environments of τ², while the challenging evaluations primarily vary user behavior. These results do not establish transfer to new task types, tools, or domains, and cannot rule out gains that depend on adaptation to the benchmark’s particular distribution.
Questions for authors
-
How can the authors establish that the generated personas better represent real users, beyond matching behavioral statistics? Could real and simulated users be evaluated under identical scenarios, comparing their preferences, decisions, and reactions, with simple persona-generation methods included as baselines?
-
How is actual persona adherence verified? Could the authors measure whether specified behaviors occur under relevant conditions and whether they remain consistent over long conversations?
-
Does PPol provide greater downstream training benefits than alternative persona methods? Could the authors compare agents trained using personas produced by simple prompting, comparable generation methods, and relevant state-of-the-art approaches against those trained using PPol, while controlling for training tasks, data volume, and training budget?
-
Do the gains transfer beyond the training distribution? Could the authors evaluate unseen task types, tools, or domains, and clarify how tasks in the human evaluation are separated from those used for training?
7 comments
Addressing W2, Q2
W2. We agree that it is important to consider how faithfully the PPol assigned persona produces the corresponding behaviours in its trajectory.
First, we would note that persona adherence is intentionally built into how PPol is optimized. Our objective is computed on the realized behavior of the persona policy. Both the human-likeness score and the coverage metric are measured from behavioral fingerprints extracted from the actual generated rollouts (Section 3.3). A persona whose specified behavior never surfaces therefore scores poorly (on combined score) and is selected against during the program evolution. So the method rewards personas precisely when the intended behavior appears in the dialogue.
To directly verify that PPol personas exhibit the assigned behaviours, we perform a new experiment with two independent LLM judges (GPT-5.4-mini and LLama-3.3-70b) that read the user turns of a conversation (tau-bench Retail setting from Table 1) and decide whether the behavior is followed as per the assigned behavioral axis. The judge is grounded in the persona's own roleplay instructions but is agnostic to the 19-feature fingerprint. As a control against bias, we also tested whether the judges find these behaviors in a cooperative base-simulator dialogue for the same task.
- Across all evolved personas, the models successfully show their assigned behaviors 87% of the time, compared to 22% for the baseline control (noting that the baseline isn't zero, as standard simulators naturally exhibit some basic conversational realism). Both judges agreed 77% of the time, showing moderate-to-strong inter-judge reliability.
- The behaviour gap to the baseline is largest on the friction behaviours that default simulators lack (e.g., incremental disclosure 0.83 vs 0.12). This confirms the intended behaviours are revealed in the dialogue.
We will add this behavioural faithfulness evaluation. It also supports our motivation to use PPol to train agents against diverse user behaviours.
Addressing W3, Q3
W3. Thank you for suggesting this. We added two additional baselines to distinguish the downstream improvements of training with PPol that go beyond simply introducing persona diversity
- Direct Prompting (DP): LLM is directly prompted to generate 10 diverse personas for each task (Sec 4: Baselines).
- PersonaHub: We sample 10 profiles per task from the PersonaHub (Ge et al., 2025) persona collection (a widely used repository of synthetic personas) and use an LLM (Gemini-3-Flash) to convert them into task-specific user instructions.
Both methods use the same persona-generation model (Gemini-3-Flash), rollout agent (Gemma-4-31B), user simulator (Qwen3-Next-80B), and training tasks as PPol. We collect successful trajectories and match the number of training examples used in the original PPol experiments. We use the same nominal LoRA hyperparameters and training steps (48 for both).
We evaluate the resulting agents on the -bench test split, using the default simulator and four challenging tau -trait simulators (Sec 5.5) featuring the prompts from He et al. (2026). Results below show the Mean Success Rate (%) below:
| Domain | Training personas | Default-user | Challenging-user |
|---|---|---|---|
| Retail | DP Personas | 70.0 | 20.6 |
| PersonaHub | 57.5 | 16.3 | |
| PPol | 75.0 | 25.0 | |
| Airline | DP Personas | 65.0 | 38.8 |
| PersonaHub | 60.0 | 37.5 | |
| PPol | 65.0 | 41.3 |
PPol achieves the high default-user success rate in both domains, outperforming DP Personas by 5pp in Retail, and PersonaHub by 17.5pp and 5pp, respectively. These results also hold across the challenging user case. We further examined the generated trajectories to understand these differences. In Retail, PersonaHub-trained agents sometimes generated fabricated tool responses instead of making actual tool calls (~20% of tasks across evaluation settings).
These findings suggest that generating diverse personas alone may not produce equally useful training data. The optimization for PPol in producing realistic, task-relevant user behaviors helps agents handle challenging interactions while avoiding these failure modes. We will include these baselines in our discussion of the benefits of PPol-based training. We would also be happy to run any additional state-of-the-art baselines you believe would strengthen the differentiation of our work.
Addressing W1, Q1
Thank you for sharing your constructive feedback on our paper. We appreciate you recognizing that our work goes beyond evaluating text quality by directly linking user simulation to downstream agent performance through behavioral features, agent-training experiments, and human interaction studies. We address your concerns below:
W1. First, we would like to emphasize that making simulated users communicate more like real humans is our central focus. PPol's optimization objective explicitly rewards persona-generation programs for human-likeness and behavioral coverage, both measured against real conversations from the target domain. Our results provide evidence that goes beyond surface statistics and covers several independent views of human-likeness:
We agree that it is interesting to look at it from another lens. To answer Q1, we test whether simulated users make the same decisions and reactions as real people when placed in the identical situation, that is, same task, same conversation so far, same agent message.
Table: Share of simulated next messages (%) that match the real user's next message (higher is better; range over two user LLMs)
Findings: Given the exact same conversation, PPol's next message makes the same decision as the real user 85–86% of the time, compared with 73–81% for the baselines, and reacts like them more often when things go wrong (95% vs. 88–93%); both gaps are significant in all 4 comparisons, and the decision gap holds for each set of real users separately.
Overall, PPol's realism extends to how users decide and react in the same situation. We will add this analysis to the paper with example conversations, and hope that it makes the paper's central claim stronger.