- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PersonaFactory, a pipeline that generates structured synthetic user personas and uses them to restyle the user side of existing conversations while preserving the original requests. The contributions are a bank of 99,696 structured personas, a corpus of 236,191 augmented conversations in 77 languages (built on UltraChat and Nemotron), and an evaluation of style adoption and intent preservation. The paper reports a controlled structured-vs-flat style ablation, comparisons with PersonaHub and SynthPAI formats, multi-judge checks, an interim human study, and fine-tuning experiments. The central finding is that structured style representations improve stylistic fidelity over free-text descriptions with comparable intent preservation.
Critical strengths
- The persona bank and multilingual corpus explicitly model communication attributes (capitalization, punctuation, code-switching, digital fluency) and are potentially valuable for research on user simulation.
- Language-aware matching, protected-span masking, constrained rewriting, continuity supervision, and sanity filtering show substantial effort to preserve task constraints (Sections 3-4).
- Convincing evidence for structured style representation; the controlled ablation fixes rewriter, persona, and prompt length and shows about 60% vs 33% pairwise preference with equal intent preservation; external judges agree in direction, although DeepSeek-V4-Flash is marginal (Table 1, Table 13).
- Broad analysis and reproducibility: Judge-free metrics, judge-agreement analysis, per-attribute ablations, and verbatim prompts support reproducibility.
Critical weaknesses
-
The rewrite prompt forbids new requests and persona-background content (Appendix A), yet many released examples add content or change task scope: Figure 2 adds a Florida-education complaint, Figure 6 invents show details, Figure 3 drops seed constraints, and Figures 10-13 reframe generic requests around the persona's job or city (Figure 11 adds new creative requirements). Appendix C presents this as a feature and finds 6,062 such turns. Intent scores also vary widely across evaluations (97.2% corpus-wide vs about 62% in the Appendix I probe and 92% in Table 1; Gemma 98.8% vs 0.77), suggesting the corpus-scale intent judge is lenient.
-
Both seed datasets (UltraChat, Nemotron) are synthetic, yet the paper describes "real logs" and "the realism of verified chats". The human study has 11 submissions over nine languages, mostly one rater per language, no baseline arm, and unverified human-written texts in the human-vs-model task; Greek scores are very low. The abstract's claim that the data is "hard to distinguish from human-written texts" conflicts with Appendix J, which states that indistinguishability cannot be established.
-
All fine-tuning arms share the same augmented user side and vary only assistant conditioning, so there is no comparison against the original conversations. Most preferences disappear or reverse after length control; results use one seed and one judge, there are no capability benchmarks, and the largest win is against Llama-3.1-8B-Instruct, a differently trained model. The abstract claims improved adaptive behavior, while Section 7 states that no downstream improvement can be claimed.
-
The paper says the SynthPAI form differs "mainly in whether that information is structured", but the PersonaFactory block has about 280 words vs 83 and includes quirks and reference pools that the SynthPAI form almost entirely lacks (Table 21). Real SynthPAI profiles perform on par, and external judges rate PersonaFactory rewrites 7-12 points less natural than unconditioned rewrites (Appendix H).
-
There is limited coverage of the intended population: about 74.3% of conversations are in English, only 4.6% of personas have low digital fluency despite motivating the work, and code-switching markers appear in only 1.95% of turns (Tables 3-4, Appendix E).
Questions for authors
- How do you separate acceptable stylistic variation from changes in task content, as in Figures 2, 10, and 11? Can you report the rate of added or dropped content on a human-annotated random sample, and explain why corpus-scale intent (97%) differs so much from the Appendix I probe (about 62%)?
- Can you compare models fine-tuned on the original conversations with models fine-tuned on persona-augmented user turns, with the assistant side fixed, and report general capability benchmarks?
- Can you report cross-lingual intent preservation against the original source prompt rather than the intermediate translation?
- Can you compare structured and SynthPAI formats with matched content and length, to isolate the effect of structure?
- Can you expand the human study with more raters per language, a baseline arm, and verified human-written texts?
6 comments
Official Response to Reviewer ARhA from Submission21841 Authors (2/3)
Q2: How do models fine-tuned on the original conversations compare with models fine-tuned on persona-augmented data, including capability benchmarks?
We thank the reviewer for this suggestion, which made our downstream evaluation much stronger. We trained Apertus-v1.5-8B (base) on four arms built from the same 79,817 language-matched conversations of the extended release, three seeds each:
| Arm | MMLU | ARC-C | GSM8K | IFEval | MBPP |
|---|---|---|---|---|---|
| A0: original | 63.7 | 71.4 | 70.1 | 67.9 | 48.7 |
| A0r: regenerated assistant, no persona | 61.3 | 73.0 | 67.7 | 65.1 | 48.5 |
| A1: restyled user, original assistant | 63.1 | 72.2 | 69.6 | 64.1 | 47.5 |
| A2: our release | 64.4 | 72.4 | 67.0 | 69.7 | 48.8 |
We observe four things: (1) Our data is on par with the original conversations on capability benchmarks (A2 - A0 on IFEval: +1.8 [-1.6,5.3]); GSM8K is the one benchmark where the original is higher. (2) On 1,000 held-out WildChat prompts, the length-controlled win rate of A2 over A0 is 54.6 to 57.5% in all three seeds under two judges, with every interval above 50%. (3) Persona-conditioned regeneration adds value beyond regeneration itself: A2 - A0r on IFEval is +4.6 [1.1,8.2], with seed variation included. (4) With the assistant side fixed, as the reviewer asked (A1 vs. A0), restyling the user lowers IFEval by 3.8 points, because the original assistant turns no longer fit the restyled user. Regenerating them (A2) restores performance and adds the preference on real prompts, which is why our pipeline regenerates the assistant side.
Regarding adaptive behavior, we sent 300 held-out requests from 12 user personas (three seeds) to models trained on our data (A2) and on the original data (A0). A2 adapts its reply style to the user more, while content stays as stable:
| Metric | A2 | A0 |
|---|---|---|
| Formality correlation | 0.463 | 0.378 |
| Emoji mirroring | 0.392 | 0.125 |
| Style diversity across personas | 0.325 | 0.227 |
| Content stability | 0.826 | 0.829 |
| In the revised version, the abstract will state this specifically as adaptation of reply style, and this controlled study will replace the Llama-3.1-8B-Instruct comparison as our main downstream evidence. |
Q3: Can you report cross-lingual intent preservation against the original source prompt?
Of course. Three judges compared both the intermediate translation and the final turn with the English source on 2,000 turns in 64 languages, allowing the language change. The translation step keeps the request in 95.6 to 98.7% of turns, and the core request survives the full pipeline in 95.9 to 96.6%. Strict same-request rates on the final turn (61.5 to 78.0%) show that differences arise at the restyling step, the same mechanism as in Q1, and the strict label flags them in the release.
Q4: Does the structured format still help when content and length are matched with the SynthPAI format?
Thank you for raising this important question. We rendered the full content of each structured block as SynthPAI-style free text with the same facts and matched length per persona, and compared both on the 800 items of Table 2. With content fixed, pairwise judges see no difference in style (structured share 50.2 to 51.3%). At the same time, the structured form keeps the request more often (+5.0 [2.1,7.6] with Gemma) and reads more naturally (+5.6 and +6.0 with the two rewriters). Single-rewrite style adoption is slightly lower for the structured form under one judge (-5.5). We agree that our original format comparison mixed structure with content, since the SynthPAI format carries less style information, and we will state this and add the matched test in the revised version. Our structural claim rests on Table 1, where only the representation of the style layer changes at equal length, and it holds under the marker cap with all three judges (structured share 54.9 to 58.1% of decisive verdicts). We will therefore narrow the abstract's claim that simulation "greatly benefits from a structured, attribute-based approach" to what both tests support: structure gives per-axis control and better intent preservation at comparable style. The marker-cap results are in our response to Reviewer 3zmg.
Official Response to Reviewer ARhA from Submission21841 Authors (3/3)
Q5: Can the human study support the claims made in the abstract?
We agree that the abstract's wording should be more precise, and we will narrow the claim: the abstract will state that our data is hard to distinguish from human-written texts to the human eye, and the abstract and Section 5 will describe the study behind this as a preliminary expert study, with Part C reported per language with n. Recruiting native raters for every language was not possible within the discussion period, and the limitations will say so. In Part C, raters judged 26 of 44 of our rewrites (59%) and 29 of 41 WildChat user turns (71%) as human-written, with similar confidence (3.86 vs. 3.83 out of 5). The Greek ratings reflect a few items with weak Greek generation; items without such defects score far higher (mean rating 3.88, against 1.25 to 1.86 for items with defects), and we will list this as a language-specific limitation, connected to generator model's multilingual capacity in that language.
Q6: Are the seeds real user logs, and how broad is the coverage?
Thank you for pointing out the imprecise wording, we are happy to clarify this. UltraChat and Nemotron are synthetic seeds, and we will name them as such. We aimed to address W5, noting the limited coverage across languages and cultures. Therefore, the updated release now adds real user logs: 58% of its seeds are WildChat user turns. It contains 1,237,976 conversations in 85 languages (vs. 236,191 in 77), English drops from 74.3% to about 38%, and the bank grows to 743,010 personas. Low digital fluency and code-switching remain minority traits; we will report the per-turn code-switching rate and list both as limitations.
Q7: Does processing real user turns introduce privacy risks?
We checked 20,000 WildChat-seeded rewrites with Presidio. Rewrites do not add contact details: none of the 10,550 same-language pairs introduces an email address, phone number or card number that is absent from its seed. Details already present in a WildChat seed can carry over (pattern-detected entities in 12.2% of seeds and 11.3% of rewrites), so this part of the release inherits WildChat's own anonymization. The ethics statement currently says the pipeline processes no personal data, and we will correct it accordingly with the addition of WildChat seeded data.
We thank the reviewer again for the thorough review and useful feedback. In case any questions remain, we are happy to discuss them further.
Official Response to Reviewer ARhA from Submission21841 Authors
We thank the reviewer for their rigorous and detailed review of our work, and for recognizing the pipeline's safeguards and the controlled evidence for structured style representation. Each of the questions pointed to a concrete experiment, and we commit to providing details on all of them. We aim to thoroughly address the questions and will split our response into three parts.
Q1: How do you separate stylistic variation from content changes in the generated data?
We agree with the reviewer that the intent check reported in the paper was lenient toward added details, and that several showcase examples add content. We address this in the sub-questions below.
Q1-a: Can intent preservation be measured more strictly?
We are happy to elaborate more on that. We introduced a strict label for every user turn, where in the corpus we updated to address the coverage concern (per W5) an independent judge compares the turn with its seed and assigns one of four labels (same request, added content, dropped content, scope change). A second external judge audited 4,903 turns and agrees with it at kappa=0.59 (four-way) and 0.63 (same vs. drift).
Both judges find the core request preserved (a looser check of whether the main ask is kept) in 97.2% of turns. The label ships with every turn, and we will recommend the strict subset as the default for training in the dataset card.
Q1-b: Can you provide a human-annotated sample?
Of course. Two annotators labelled 100 sampled turns with the same four labels, blind to the persona, the rewriter, the judge's label and the source of each row. We note that both annotators are authors of the paper. They agree with each other at kappa=0.60 on same vs. drift, and the strict judge agrees with the two annotators at kappa=0.60 and 0.41, respectively. In the 9 cases where both annotators agree and the judge does not, the judge is the stricter side 8 times, so the drift rates we report are, if anything, conservative. Both annotators consider the core request preserved in 92 of 100 turns. On 25 turns the judge labelled as added content, the annotators confirm most of the additions, which they describe as written-out requirements, extra context, or an implicit question made explicit; the core request survives in 23 of 25. Several drift labels trace back to the original message rather than to the rewrite: in 5 of annotator A's 34 drift labels, the annotator flagged the seed itself as unclear (e.g., "Prepositions of place" or "привет / привет / Hello"). When a seed does not state a clear request, any natural rewrite has to add some framing to it.
Q1-c: Where do the additions come from, and do they matter for training?
To find the source, we split the persona into its parts on the 800 items of the format comparison (Table 2). With our main rewriter, the background alone adds content in only 1.5% of turns (0.0% without a persona), while the weaker rewriter adds content in 8.9% of turns even without any persona. Additions therefore follow the rewriter's instruction following, which the per-rewriter label makes visible. To test whether they matter, we retrained our recipe only on strict conversations at equal size (three seeds): every benchmark stays unchanged (IFEval +0.3 [-2.2,2.8]), and the preference over the original data remains (53.8 to 57.4%). Users can therefore filter to the strict subset at no cost.
Q1-d: Why is intent preservation 97% in the corpus but about 62% in the probe?
Both numbers come from the same template and judge; the difference lies in the data. The attribute probe (App. I) measures single-call rewriting by four models, two of which are not used in the release, without masking, retries or the sanity gate, and with older single-attribute profiles. In the paper, the Gemma rewriter preserves intent in about 77% of probe rewrites (App. I) and in 98.8% of corpus turns (Table 6), and Ministral in about 38% and 94.5%. We observe, that Ministral has lower intent preservation but higher creativity in terms of stylistic adoption and authenticity from manual inspection.
Q1-e: Should the showcase figures be presented as intended behavior?
We agree that several of them go beyond the seed request. In the revised version, every showcase figure will carry its strict label, and the captions of Figures 2, 3, 6, 10, 11, 12 and 13 will state what the rewrite adds to or drops from the seed, so readers see both what the style layer does and where a rewrite exceeds the request. The examples in the main text will be replaced with strict same-request conversations.