acceptodds
Under review as a conference paper at ICLR 2027

PersonaFactory: Stylistic User Prompt Augmentation with Synthetic Persona Bank

Abstract

Large language models increasingly replace real people in the AI development pipeline, as they are used to generate training dialogues at scale, control alignment and evaluation. However, out-of-the-box models produce monolingual and stylistically flat texts as synthetic user prompts, failing to replicate the real user-AI communication mechanics. We present PersonaFactory, an end-to-end pipeline to generate detailed user personas, later using them to augment existing user prompts from their point of view. As part of this work, we showcase two major data contributions, facilitating research in LLM-based stylistic chat generation. First, we release a bank of nearly 100000 structured, fully synthetic personas, focusing on their background and communication behavior with a chatbot. Second, we release a corpus of roughly 236000 persona-styled synthetic conversations spanning 77 languages, produced by using the generated bank of personas to augment existing user prompts and match personas to them based on original prompt intent and profile background. We also provide an ablation study of the assistant persona design for response augmentation and downstream performance after training on such conversations. Finally, we evaluate the effect of personal features on style adoption and intent preservation, using an LLM-judge and heuristic metrics to assess augmentation efficiency. We find that simulating users' unique behavioral features in the conversation greatly benefits from a structured, attribute-based approach, compared to a single free-text description in existing personalization methods. We show that the produced augmented data is hard to distinguish from human-written texts for real people and that PersonaFactory as a training artifact can improve model adaptive behavior purely from data impact without steering.

Primary area:
datasets and benchmarks
Keywords:
LLM personalization, user simulation, synthetic persona generation, stylistic generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ARhAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PersonaFactory, a pipeline that generates structured synthetic user personas and uses them to restyle the user side of existing conversations while preserving the original requests. The contributions are a bank of 99,696 structured personas, a corpus of 236,191 augmented conversations in 77 languages (built on UltraChat and Nemotron), and an evaluation of style adoption and intent preservation. The paper reports a controlled structured-vs-flat style ablation, comparisons with PersonaHub and SynthPAI formats, multi-judge checks, an interim human study, and fine-tuning experiments. The central finding is that structured style representations improve stylistic fidelity over free-text descriptions with comparable intent preservation.

Critical strengths
  1. The persona bank and multilingual corpus explicitly model communication attributes (capitalization, punctuation, code-switching, digital fluency) and are potentially valuable for research on user simulation.
  2. Language-aware matching, protected-span masking, constrained rewriting, continuity supervision, and sanity filtering show substantial effort to preserve task constraints (Sections 3-4).
  3. Convincing evidence for structured style representation; the controlled ablation fixes rewriter, persona, and prompt length and shows about 60% vs 33% pairwise preference with equal intent preservation; external judges agree in direction, although DeepSeek-V4-Flash is marginal (Table 1, Table 13).
  4. Broad analysis and reproducibility: Judge-free metrics, judge-agreement analysis, per-attribute ablations, and verbatim prompts support reproducibility.
Critical weaknesses
  1. The rewrite prompt forbids new requests and persona-background content (Appendix A), yet many released examples add content or change task scope: Figure 2 adds a Florida-education complaint, Figure 6 invents show details, Figure 3 drops seed constraints, and Figures 10-13 reframe generic requests around the persona's job or city (Figure 11 adds new creative requirements). Appendix C presents this as a feature and finds 6,062 such turns. Intent scores also vary widely across evaluations (97.2% corpus-wide vs about 62% in the Appendix I probe and 92% in Table 1; Gemma 98.8% vs 0.77), suggesting the corpus-scale intent judge is lenient.

  2. Both seed datasets (UltraChat, Nemotron) are synthetic, yet the paper describes "real logs" and "the realism of verified chats". The human study has 11 submissions over nine languages, mostly one rater per language, no baseline arm, and unverified human-written texts in the human-vs-model task; Greek scores are very low. The abstract's claim that the data is "hard to distinguish from human-written texts" conflicts with Appendix J, which states that indistinguishability cannot be established.

  3. All fine-tuning arms share the same augmented user side and vary only assistant conditioning, so there is no comparison against the original conversations. Most preferences disappear or reverse after length control; results use one seed and one judge, there are no capability benchmarks, and the largest win is against Llama-3.1-8B-Instruct, a differently trained model. The abstract claims improved adaptive behavior, while Section 7 states that no downstream improvement can be claimed.

  4. The paper says the SynthPAI form differs "mainly in whether that information is structured", but the PersonaFactory block has about 280 words vs 83 and includes quirks and reference pools that the SynthPAI form almost entirely lacks (Table 21). Real SynthPAI profiles perform on par, and external judges rate PersonaFactory rewrites 7-12 points less natural than unconditioned rewrites (Appendix H).

  5. There is limited coverage of the intended population: about 74.3% of conversations are in English, only 4.6% of personas have low digital fluency despite motivating the work, and code-switching markers appear in only 1.95% of turns (Tables 3-4, Appendix E).

Questions for authors
  1. How do you separate acceptable stylistic variation from changes in task content, as in Figures 2, 10, and 11? Can you report the rate of added or dropped content on a human-annotated random sample, and explain why corpus-scale intent (97%) differs so much from the Appendix I probe (about 62%)?
  2. Can you compare models fine-tuned on the original conversations with models fine-tuned on persona-augmented user turns, with the assistant side fixed, and report general capability benchmarks?
  3. Can you report cross-lingual intent preservation against the original source prompt rather than the intermediate translation?
  4. Can you compare structured and SynthPAI formats with matched content and length, to isolate the effect of structure?
  5. Can you expand the human study with more raters per language, a baseline arm, and verified human-written texts?
6 comments
AuthorsOpenReview ↗

Official Response to Reviewer ARhA from Submission21841 Authors

We thank the reviewer for their rigorous and detailed review of our work, and for recognizing the pipeline's safeguards and the controlled evidence for structured style representation. Each of the questions pointed to a concrete experiment, and we commit to providing details on all of them. We aim to thoroughly address the questions and will split our response into three parts.

Q1: How do you separate stylistic variation from content changes in the generated data?

We agree with the reviewer that the intent check reported in the paper was lenient toward added details, and that several showcase examples add content. We address this in the sub-questions below.

Q1-a: Can intent preservation be measured more strictly?

We are happy to elaborate more on that. We introduced a strict label for every user turn, where in the corpus we updated to address the coverage concern (per W5) an independent judge compares the turn with its seed and assigns one of four labels (same request, added content, dropped content, scope change). A second external judge audited 4,903 turns and agrees with it at kappa=0.59 (four-way) and 0.63 (same vs. drift).

Strict label (% of user turns)Submitted releaseUpdated release
Same request80.273.9
Added content12.418.5
Dropped content5.24.1
Scope change2.23.5

Both judges find the core request preserved (a looser check of whether the main ask is kept) in 97.2% of turns. The label ships with every turn, and we will recommend the strict subset as the default for training in the dataset card.

Q1-b: Can you provide a human-annotated sample?

Of course. Two annotators labelled 100 sampled turns with the same four labels, blind to the persona, the rewriter, the judge's label and the source of each row. We note that both annotators are authors of the paper. They agree with each other at kappa=0.60 on same vs. drift, and the strict judge agrees with the two annotators at kappa=0.60 and 0.41, respectively. In the 9 cases where both annotators agree and the judge does not, the judge is the stricter side 8 times, so the drift rates we report are, if anything, conservative. Both annotators consider the core request preserved in 92 of 100 turns. On 25 turns the judge labelled as added content, the annotators confirm most of the additions, which they describe as written-out requirements, extra context, or an implicit question made explicit; the core request survives in 23 of 25. Several drift labels trace back to the original message rather than to the rewrite: in 5 of annotator A's 34 drift labels, the annotator flagged the seed itself as unclear (e.g., "Prepositions of place" or "привет / привет / Hello"). When a seed does not state a clear request, any natural rewrite has to add some framing to it.

Q1-c: Where do the additions come from, and do they matter for training?

To find the source, we split the persona into its parts on the 800 items of the format comparison (Table 2). With our main rewriter, the background alone adds content in only 1.5% of turns (0.0% without a persona), while the weaker rewriter adds content in 8.9% of turns even without any persona. Additions therefore follow the rewriter's instruction following, which the per-rewriter label makes visible. To test whether they matter, we retrained our recipe only on strict conversations at equal size (three seeds): every benchmark stays unchanged (IFEval +0.3 [-2.2,2.8]), and the preference over the original data remains (53.8 to 57.4%). Users can therefore filter to the strict subset at no cost.

Q1-d: Why is intent preservation 97% in the corpus but about 62% in the probe?

Both numbers come from the same template and judge; the difference lies in the data. The attribute probe (App. I) measures single-call rewriting by four models, two of which are not used in the release, without masking, retries or the sanity gate, and with older single-attribute profiles. In the paper, the Gemma rewriter preserves intent in about 77% of probe rewrites (App. I) and in 98.8% of corpus turns (Table 6), and Ministral in about 38% and 94.5%. We observe, that Ministral has lower intent preservation but higher creativity in terms of stylistic adoption and authenticity from manual inspection.

Q1-e: Should the showcase figures be presented as intended behavior?

We agree that several of them go beyond the seed request. In the revised version, every showcase figure will carry its strict label, and the captions of Figures 2, 3, 6, 10, 11, 12 and 13 will state what the rewrite adds to or drops from the seed, so readers see both what the style layer does and where a rewrite exceeds the request. The examples in the main text will be replaced with strict same-request conversations.

AuthorsOpenReview ↗

Official Response to Reviewer ARhA from Submission21841 Authors (2/3)

Q2: How do models fine-tuned on the original conversations compare with models fine-tuned on persona-augmented data, including capability benchmarks?

We thank the reviewer for this suggestion, which made our downstream evaluation much stronger. We trained Apertus-v1.5-8B (base) on four arms built from the same 79,817 language-matched conversations of the extended release, three seeds each:

ArmMMLUARC-CGSM8KIFEvalMBPP
A0: original63.771.470.167.948.7
A0r: regenerated assistant, no persona61.373.067.765.148.5
A1: restyled user, original assistant63.172.269.664.147.5
A2: our release64.472.467.069.748.8

We observe four things: (1) Our data is on par with the original conversations on capability benchmarks (A2 - A0 on IFEval: +1.8 [-1.6,5.3]); GSM8K is the one benchmark where the original is higher. (2) On 1,000 held-out WildChat prompts, the length-controlled win rate of A2 over A0 is 54.6 to 57.5% in all three seeds under two judges, with every interval above 50%. (3) Persona-conditioned regeneration adds value beyond regeneration itself: A2 - A0r on IFEval is +4.6 [1.1,8.2], with seed variation included. (4) With the assistant side fixed, as the reviewer asked (A1 vs. A0), restyling the user lowers IFEval by 3.8 points, because the original assistant turns no longer fit the restyled user. Regenerating them (A2) restores performance and adds the preference on real prompts, which is why our pipeline regenerates the assistant side.

Regarding adaptive behavior, we sent 300 held-out requests from 12 user personas (three seeds) to models trained on our data (A2) and on the original data (A0). A2 adapts its reply style to the user more, while content stays as stable:

MetricA2A0
Formality correlation0.4630.378
Emoji mirroring0.3920.125
Style diversity across personas0.3250.227
Content stability0.8260.829
In the revised version, the abstract will state this specifically as adaptation of reply style, and this controlled study will replace the Llama-3.1-8B-Instruct comparison as our main downstream evidence.

Q3: Can you report cross-lingual intent preservation against the original source prompt?

Of course. Three judges compared both the intermediate translation and the final turn with the English source on 2,000 turns in 64 languages, allowing the language change. The translation step keeps the request in 95.6 to 98.7% of turns, and the core request survives the full pipeline in 95.9 to 96.6%. Strict same-request rates on the final turn (61.5 to 78.0%) show that differences arise at the restyling step, the same mechanism as in Q1, and the strict label flags them in the release.

Q4: Does the structured format still help when content and length are matched with the SynthPAI format?

Thank you for raising this important question. We rendered the full content of each structured block as SynthPAI-style free text with the same facts and matched length per persona, and compared both on the 800 items of Table 2. With content fixed, pairwise judges see no difference in style (structured share 50.2 to 51.3%). At the same time, the structured form keeps the request more often (+5.0 [2.1,7.6] with Gemma) and reads more naturally (+5.6 and +6.0 with the two rewriters). Single-rewrite style adoption is slightly lower for the structured form under one judge (-5.5). We agree that our original format comparison mixed structure with content, since the SynthPAI format carries less style information, and we will state this and add the matched test in the revised version. Our structural claim rests on Table 1, where only the representation of the style layer changes at equal length, and it holds under the marker cap with all three judges (structured share 54.9 to 58.1% of decisive verdicts). We will therefore narrow the abstract's claim that simulation "greatly benefits from a structured, attribute-based approach" to what both tests support: structure gives per-axis control and better intent preservation at comparable style. The marker-cap results are in our response to Reviewer 3zmg.

AuthorsOpenReview ↗

Official Response to Reviewer ARhA from Submission21841 Authors (3/3)

Q5: Can the human study support the claims made in the abstract?

We agree that the abstract's wording should be more precise, and we will narrow the claim: the abstract will state that our data is hard to distinguish from human-written texts to the human eye, and the abstract and Section 5 will describe the study behind this as a preliminary expert study, with Part C reported per language with n. Recruiting native raters for every language was not possible within the discussion period, and the limitations will say so. In Part C, raters judged 26 of 44 of our rewrites (59%) and 29 of 41 WildChat user turns (71%) as human-written, with similar confidence (3.86 vs. 3.83 out of 5). The Greek ratings reflect a few items with weak Greek generation; items without such defects score far higher (mean rating 3.88, against 1.25 to 1.86 for items with defects), and we will list this as a language-specific limitation, connected to generator model's multilingual capacity in that language.

Q6: Are the seeds real user logs, and how broad is the coverage?

Thank you for pointing out the imprecise wording, we are happy to clarify this. UltraChat and Nemotron are synthetic seeds, and we will name them as such. We aimed to address W5, noting the limited coverage across languages and cultures. Therefore, the updated release now adds real user logs: 58% of its seeds are WildChat user turns. It contains 1,237,976 conversations in 85 languages (vs. 236,191 in 77), English drops from 74.3% to about 38%, and the bank grows to 743,010 personas. Low digital fluency and code-switching remain minority traits; we will report the per-turn code-switching rate and list both as limitations.

Q7: Does processing real user turns introduce privacy risks?

We checked 20,000 WildChat-seeded rewrites with Presidio. Rewrites do not add contact details: none of the 10,550 same-language pairs introduces an email address, phone number or card number that is absent from its seed. Details already present in a WildChat seed can carry over (pattern-detected entities in 12.2% of seeds and 11.3% of rewrites), so this part of the release inherits WildChat's own anonymization. The ethics statement currently says the pipeline processes no personal data, and we will correct it accordingly with the addition of WildChat seeded data.

We thank the reviewer again for the thorough review and useful feedback. In case any questions remain, we are happy to discuss them further.

Reviewer_p1BnOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper presents an end-to-end pipeline to generate detailed user personas, later using them to augment existing user prompts from their point of view. The main contribution is releasing new large datasets of personas as well as conversations, and the authors claim that they assess augmentation efficiency

Critical strengths

The motivation behind constructing and releasing a dataset of different personas is genuinely interesting, as LLMs tend to be too homogeneous in their generations. However, the paper has several weaknesses.

Critical weaknesses

The main weakness is the lack of an assessment of their generation. Their methodology basically has no validation at all, which I believe it must be central (of course, models can generate conversations, but to what extent they follow the requirements is the most important part). In particular, the human validation is very weak at this level (with 11 annotators... and no metrics reported). The other evaluation is either an LLM-judge - details of the evaluation are very vague, like "same intent" -, and a set of "deterministic" metrics (which I could not find being reported), and only described in aggregate in one sentence in the main text (and partially reported in the Appendix). This part could have great value, but it needs to be properly validated! For reference, the authors could check: N. Pagan, P. Törnberg, C.A. Bail, & C. Barrie, AI-generated language remains systematically distinguishable from human communication, Proc. Natl. Acad. Sci. U.S.A. 123 (42) e2531748123, https://doi.org/10.1073/pnas.2531748123 (2026).

Also, they acknowledge they non-english language only covers a minority, despite motivating their work. This is stated in the limitations, but it is clearly not a minor limitation given that it stays behind the motivation. Furthermore, they did not provide any insights on the difference performances across languages and/or generator models.

I also find the methodology not very rigorous (e.g., how did they match demographic attributes with styles?), and very much LLM-based in basically all decision steps (which is obviously a problem, if they want to get around homogeneization problem)

Furthermore, comparison with other methods is completely missing (for example Argyle LP, Busby EC, Fulda N, Gubler JR, Rytting C, Wingate D. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis. 2023;31(3):337-351. doi:10.1017/pan.2023.2)

More generally, the paper is not sufficiently embedded in the algorithmic fidelity literature

Questions for authors

None

4 comments
AuthorsOpenReview ↗

Official Response to Reviewer p1Bn from Submission21841 Authors

We thank the reviewer for their review and valuable feedback. We also appreciate recognizing our motivation: user prompts produced by LLMs are too homogeneous, which we aim to counter with a diverse persona bank. We acknowledge the raised concerns about validation rigor and will address each point below. We have also included the findings and protocols of both papers, referenced by the reviewer.

Q1: How are the generations validated?

We would like to point to the validation already present in the submission, which we extended during the rebuttal:

ConcernWhere it is addressed
Validation of the generationsControlled A/B test (Table 1); format comparison (Table 2, App. H); judge agreement and external re-judging (Tables 10 to 13)
Deterministic metricsTable 9 (App. F), with definitions
Human validation metricsRatings, per-language means, human vs. model judgments and inter-rater agreement (Tables 25 to 29, App. J)
Differences across generatorsTables 5 and 6, Figure 26

During the rebuttal we added four checks. First, every user turn, in both the submitted and the updated release (~80 languages, see Q5), now carries a strict intent label: same request, added content, dropped content or scope change. It comes from a judge of a different model family than the rewriters, which never sees the persona. Both external judges see the core request kept in 97.2% of turns. Second, two human annotators checked this judge in blind study: they agree with each other at kappa=0.60 and with the judge at kappa=0.60 and 0.41. Third, every deterministic metric now has no-persona, PersonaHub and SynthPAI reference rows on the same seeds. Fourth, we trained on four data variants, including the original conversations, with three seeds and five benchmarks. Our data is on par with the original on all benchmarks except GSM8K, and it is preferred on held-out real user prompts at equal length. We will move the key numbers from the appendix into the main text in the revised version.

Q2: How detectable are the rewrites under the computational Turing test of Pagan et al. [1]?

We thank the reviewer for pointing us to this recent insightful study; we adopt its logic in two ways.

First, for synthetic seeds, which are the primary use case of our pipeline, a stylometric classifier separates real WildChat user turns from UltraChat and Nemotron originals, and separately from their rewrites. Rewriting moves typing substantially toward real users (detection drops by 15.1 points [13.8,16.5]):

Synthetic seedsOriginalRewrittenReal users
Lowercase start0.5%43%27%
Missing final punctuation3%47%55%
Stylometric detection91.3%76.2%n/a

Fine-tuned encoders still separate both the originals and the rewrites from WildChat, because the topics of synthetic seeds differ - since our pipeline is designed to keep the request, a topic gap is expected and is not something the rewrite tries to close.

Second, on real WildChat turns in the paired setting, a fine-tuned XLM-R classifier distinguishes a real turn from our rewrite of it at 84.6% accuracy, and from a persona-free paraphrase at 75.9%, while stylometric features alone reach 68.2%. This matches the main finding of Pagan et al. - on nine open models and five calibration strategies, LLM-generated outputs stay distinguishable from human-written text, however, pushing them towards more human-like authenticity reduces semantic fidelity. Our proposed approach follows the fidelity side of this trade-off by design to keep the original request intact in the rewrites.

Therefore, we agree that our persona-seeded augmentations are distinguishable from human text in the case of a trained classifier. Our claims are more focused and are supported by the presented above evidence - rewrites move synthetic data towards the typing of real users by a noticeable degree, human readers in our study judged most rewrites (59%) as human-written, and the original request is preserved; similar findings have also been reported in SynthPAI which uses similar style specification. We will state this scope explicitly in the revised version and report both detector settings with reference citations.

[1] N. Pagan, P. Törnberg, C.A. Bail, & C. Barrie, AI-generated language remains systematically distinguishable from human communication. Proc. Natl. Acad. Sci. U.S.A. 123 (42) e2531748123, https://doi.org/10.1073/pnas.2531748123 (2026)

AuthorsOpenReview ↗

Official Response to Reviewer p1Bn from Submission21841 Authors (2/2)

Q3: Does relying on LLMs in every step make the generated text more homogeneous?

We measured this directly as the mean pairwise distance between user turns in a style embedding. Rewriting raises style diversity from 0.274 to 0.744 for Nemotron seeds and from 0.567 to 0.801 for UltraChat seeds, close to real WildChat users (0.857). Real logs are already diverse, and rewriting narrows them somewhat (to 0.737); for real logs, the value of our pipeline is multilingual extension that keeps the original request, and we will state this distinction clearly. Turns of the same persona are also more similar to each other than to other personas' turns, so personas give consistent, distinguishable voices.

Q4: How are demographic attributes matched to writing styles, and how does this relate to algorithmic fidelity [2]?

The bank starts from a curated grid of countries, sub-regions and occupations. An LLM then writes a structured style layer (capitalization, punctuation, emoji, typos, abbreviations, code-switching, message structure) conditioned on the full background, and a judge from a different model family screens each profile. Following the reviewer's suggestion, we evaluated all four criteria of Argyle et al. [2]:

CriterionOur result
Turing testIn Part C, 59% of rewrites and 71% of WildChat user turns judged human-written; computational tests in Q2
Backward consistencyPersona punctuation recovered from text alone in 59.8% vs. 33.9% for rewrites conditioned on the background only
Forward consistencyCore request preserved in 97.2% of turns
Pattern correspondenceAcross 162 country-language groups, Spearman r=0.48 for slang, 0.24 for lowercase start and question marks

We transparently acknowledge that pattern correspondence holds at the level of language: country-level differences within a language are not yet reproduced, and emoji rates run against real users (r=-0.3). We will list both as limitations. Background traits are intentionally not recoverable from the text, since our rewrite prompt forbids inserting them.

Q5: How do results vary across languages and generators, and how does the work compare with other methods?

Due to diversity concerns raised by multiple reviewers, we have generated more non-English responses with updated corpus covering more than 700k synthetic personas, 1.24M samples and 80 languages. 58% of its seeds are now real WildChat user turns, and the English share falls from 74.3% to about 38%. We report intent and style per language and per rewriter: Gemma-4-31B leads Ministral-3-14B in each of the 15 most frequent languages (e.g., 75.1% vs. 67.6% of English turns pass both checks), with Arabic, Vietnamese and Russian lowest. From methodology point of view, we compare PersonaFactory representation against the PersonaHub and SynthPAI on the same samples, with no-persona and paraphrase baselines, and against training on the original conversations; we also add a SynthPAI comparison at matched content and length. Our work is complementary to these resources, as it adds a writing-style layer on top of them. In the revised version, Section 2 will position our work within the algorithmic-fidelity literature.

We hope that our responses address the reviewer's concerns, and we would welcome any further questions.

[1] N. Pagan, P. Törnberg, C.A. Bail, & C. Barrie, AI-generated language remains systematically distinguishable from human communication. Proc. Natl. Acad. Sci. U.S.A. 123 (42) e2531748123, https://doi.org/10.1073/pnas.2531748123 (2026)

[2] Argyle LP, Busby EC, Fulda N, Gubler JR, Rytting C, Wingate D. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis. 2023;31(3):337-351. doi:10.1017/pan.2023.2 (2023)

Thanks, I am quite satisfied with this answer. However, some concerns still remain, specifically about the human evaluation, as well as the performance comparison across different languages. Therefore, I will increase my score to 2. Are you planning to upload a revised version?

AuthorsOpenReview ↗

Follow-up from Submission21841 Authors

Dear Reviewer p1Bn,

We thank the reviewer for engaging with our response, and many thanks for raising the score.

Yes, we have uploaded the revised PDF. Changes are marked in blue, and the original numbering of sections, tables, figures and appendices is unchanged, so the references in our earlier responses remain valid. Below, Q1 to Q5 refer to the questions in our first response.

We have added the key new findings to the main text: updated statistics for the extended release (Section 6, Q5), more details on the human evaluation (Section 5, Q1), and the classifier-based detectability study following Pagan et al. (Sections 2 and 5, Q2). The full results are in new subsections at the end of the existing appendices:

  • Strict intent label and its human check (Q1): Appendices F.1 and F.2
  • Reference rows for the judge-free metrics (Q1): Appendix F.4
  • Human study in context and Part C per language (Q1): Appendices J.6 and J.7
  • Detectability following Pagan et al.[1] (Q2): Appendix F.6
  • Homogenization (Q3): Appendix F.6
  • The four criteria of Argyle et al.[2] (Q4): Appendix F.6, with positioning in Section 2 and the remaining gaps in Section 7
  • Results per language and rewriter (Q5): Appendix F.5
  • Cross-lingual intent against the English source (Q5): Appendix F.3
  • Comparison with SynthPAI at matched content and length (Q5): Appendix H.2
  • Downstream fine-tuning, including training on the original conversations (Q1, Q5): Appendix G.1
  • Marker cap and naturalness (raised by other reviewers): Appendix H.1

Performance across languages (Q5). Appendix F.5 reports the corpus judge for the 15 most frequent languages and both rewriters, as the share of turns that pass both the intent and the style check. With the main rewriter, this share ranges from 48% (Arabic) to 77% (Italian), and it is ahead of the second rewriter in every language. The differences come mainly from style adoption: on intent alone, which the table does not break out, the judge passes about 92 to 98% of turns in each of these languages. In the cross-lingual pipeline, the translation step keeps the request in 95.6 to 98.7% of turns under three judges, and the core request survives the full pipeline in 95.9 to 96.6% (Appendix F.3). Since every turn records its rewriter and strict label, users can filter on either. We list weaker generation in some languages as a limitation (Section 7), and we see stronger open multilingual rewriters as the most direct way to close this gap. The limitation comes from from using open-weight models with permissive licensing having worse multilingual or creative writing capabilities compared to closed models. Switching to proprietary models (e.g. SynthPAI [3]) would improve the performance, however, we would like to release our data with permissive licensing for training.

Human evaluation (Q1). We agree that the study is preliminary, and the revised PDF presents it as such, with per-language results and their sample sizes (Appendix J.7). In Part C, raters judged 26 of 44 of our rewrites (59%) and 29 of 41 WildChat user turns (71%) as human-written. To put the study's scale in context, Appendix J.6 compares it with the human evaluations of related synthetic-persona work. Studies with larger rater pools are English-only and use paid crowdsourcing (e.g., 40 raters in SynthPAI[3], 80 in PERSONA[4]), while multilingual persona datasets either report no external raters or stay within one region (9 Indic languages in IndicTalk[5], 8 Southeast Asian languages in SEADialogues[6]). Our IRB-approved study covers nine languages in three scripts, seven of them mid- or lower-resource, and among the synthetic-persona works we reviewed it is the only one that spans more than one region. Most of our languages have a single rater, which we state as a limitation: native speakers of these languages are scarce on crowdsourcing platforms, and several paid raters per language were beyond the budget and time frame of this work. We recognize the limit of our human study and we plan to find more raters before the next revision.

The claims of the paper therefore rest on the automatic evidence (strict intent label with a human check, external judges, judge-free metrics with reference rows, detector analyses and downstream training), with the human study as supporting evidence. Following the reviewer's comment (Q2), the abstract now scopes the human-likeness claim to the human eye, and Appendix F.6 reports the classifier-based tests next to it.

Thank you again for the constructive discussion. We would be happy to provide further clarifications if any questions remain.

Best regards,

Authors

[1] Pagan et al., PNAS 2026.

[2] Argyle et al., Political Analysis 2023.

[3] Yukhymenko et al., NeurIPS 2024 Datasets and Benchmarks (SynthPAI).

[4] Castricato et al., COLING 2025 (PERSONA).

[5] Gawande and Singh, arXiv:2607.23242, 2026 (IndicTalk).

[6] Al Kautsar et al., arXiv:2508.07069, 2025 (SEADialogues).

Reviewer_3zmgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that existing persona-conditioned chat data generation changes what a user says but not how they say it: persona descriptions are typically short free-text summaries with no writing-style layer, and rewriting conversations preserves intent at the cost of visual/stylistic difference. PersonaFactory instead keeps the original dialogue trajectory and re-simulates the user side under a structured synthetic persona, holding the request fixed while matching register, typing style, and language behavior. The contributions are (1) a structured profile schema and a bank of 99,696 personas with a background layer (geography, occupation, education, age, native language, AI stance, digital fluency) and a style layer (capitalization, punctuation, emoji, typos, abbreviations, code-switching, quirks, verbatim example message), grounded on a curated country/sub-region/occupation grid and screened by an independent judge; (2) a grounded simulation pipeline with conservative language-and-script matching, constrained user-turn rewriting, assistant regeneration with next-turn continuity supervision, and a six-class sanity gate; and (3) an intrinsic evaluation of simulation fidelity separating style adoption from intent preservation, with a headline finding that structured per-axis style representation beats a free-text style description on style fidelity while both preserve intent equally. It also releases about 236k conversations across 77 languages.

Critical strengths
  • The core representational claim is clean and useful. The paired A/B in Table 1 holds rewriter, persona, and token budget fixed (311 vs. 324 words, so the structured arm gets no more input) and randomizes order per item. The 60% vs. 33% preference and the identical 92% intent rate make the conclusion attributable to representation rather than to the model working harder.
  • The judge-bias analysis is thorough and does not hide inconvenient numbers. Table 10 shows severity spread from 0.778 to 0.929 on style; Table 11/12 show raw agreement overstates consistency and report both κ and Gwet's AC1, with a clear explanation of why GLM–Ministral style κ is near zero while AC1 is 0.85. This is a better treatment of judge agreement than most papers in this area.
  • Provide about 236k conversations across 77 languages.
Critical weaknesses
  • The headline style-fidelity number is in tension with the paper's own naturalness finding, and the tension is not resolved in the main text. External judges rate PersonaFactory rewrites 7–12 points less natural than unconditioned rewrites, attributed to profiles asking for lowercase, dropped punctuation, abbreviations, emoji, and a sign-off simultaneously. So the method wins the style-adoption metric partly by over-performing style—which is precisely the caricature failure the paper says it wants to avoid. The "lower p or cap markers per turn" lever seems not evaluate.
  • The human evaluation is too thin to carry the weight placed on it: 11 responses across 9 languages, most languages with a single rater, no baseline arm, and no verified human-source key for the Part C human-vs-model task. The paper is commendably explicit that this "cannot yet rank languages or establish indistinguishability from human writing," but then the abstract and Section 5 headline the 59% human-written judgment as if it were a substantive result.
  • Some metrics are hard to interpret as presented. The "core-vocabulary token retention 71.0%" and LaBSE similarity 0.854 in Table 9 have no reference points—what would a good or bad rewrite score? Similarly, the transformation-depth numbers (Jaccard 0.425, edit distance 0.358) are described as "the intended effect rather than a loss," which is fair, but without a comparison condition they do not discriminate between adequate and excessive restyling.
Questions for authors

Suggestions:

  • Evaluate the naturalness/style-adoption trade-off directly: run the per-turn quirk probability at p = 0.10 and p = 0.15 and cap the number of realized markers per turn, then report style adoption, external-judge naturalness, and intent together. This is the single most important missing experiment, because it tests whether the headline gain survives when caricature is controlled.
  • Either expand the human study or reframe Section 5 so the 59% human-like judgment is presented as a preliminary signal rather than a finding. If expansion is infeasible, report the Part C result per language with n, since pooling 11 raters across 9 languages obscures that most cells are a single rater.
  • Quantify actual dialect coverage in the released corpus—how many turns are in which dialect labels, and how many distinct labels appear with more than a handful of examples. This would either substantiate the sociolinguistic motivation or show clearly that it remains future work.
4 comments
AuthorsOpenReview ↗

Official Response to Reviewer 3zmg from Submission21841 Authors

We sincerely thank the reviewer for their time and effort spent reviewing our paper, their insightful and constructive comments, and their overall favorable assessment. We especially appreciate the recognition of our controlled representational result and of our transparent treatment of judge agreement. We address the reviewer's questions and comments below.

Q1: Does the style adoption advantage survive when over-stylization is controlled, for example with a lower quirk probability or a marker cap?

We agree with the reviewer that this was the most important missing experiment, and we ran it. We evaluated p=0.1, 0.15 and 0.25, each with and without a cap of two realized chat markers per turn (greeting, sign-off, emoji, slang, ellipsis, repeated punctuation). The cap is an instruction in the rewrite prompt, not a post-hoc filter, so it changes how the whole turn is written. To compare like with like, we generated all settings fresh on the 646 of the 800 Table 2 items that pass the pipeline's checks in all six settings (so the uncapped values differ from the 69.2% of Table 2), and added the reference formats and real user turns to the same table:

Setting (Gemma-4-31B)Style adoptionSame requestNaturalness (Muse-Glimmer judge)
p = 0.10 to 0.25, no cap63.0 to 64.688.5 to 90.783.1 to 83.9
p = 0.10 to 0.25, cap52.9 to 57.689.0 to 89.887.6 to 88.7
SynthPAI format46.190.287.0
No persona3.794.691.3
Real WildChat user turnsn/an/a94.2

We see three main results: (1) With the cap, our rewrites are as natural as the SynthPAI-format rewrites of Appendix D (generated without a cap) while adopting 7 to 11 points more of the persona's style, at the same intent preservation; a second judge (Qwen3.6) shows the same picture. Capped rewrites remain 2.6 to 3.7 points less natural than rewrites without any persona, which is the cost of adopting a style at all. (2) The cap is the effective lever, not p: quirk probability has no consistent effect, so we will remove our claim that p=0.25 prevents over-stylization and document the cap as the recommended setting. (3) Our main representational result holds under the cap: we regenerated both arms of Table 1 with the cap, and the structured style layer remains preferred under all three judges, with a smaller margin (62.0 to 65.1% of decisive verdicts without the cap, 54.9 to 58.1% with it). On real WildChat seeds with both rewriters, the cap likewise trades a few points of style for 4 to 5 points of naturalness at about 99% intent preservation.

Regarding the released corpus: the extended release was generated with p=0.25 and the prompt's instruction to use at most two stylistic elements, but without the explicit cap. Rule-based counts show that 97.3% of its user turns contain at most two of the six markers, while some markers, such as greetings and emoji, remain more frequent than in real user turns. We will acknowledge this as a limitation, add a per-turn marker count to the release so users can filter by marker density, and make the cap the default in our released code.

AuthorsOpenReview ↗

Official Response to Reviewer 3zmg from Submission21841 Authors (2/2)

Q2: Can the human evaluation be strengthened, or should the claims be reframed?

We agree with the reviewer and will follow their suggestion. In the revised version, the abstract and Section 5 will present our IRB-approved study as a preliminary expert study, with Part C reported per language with n (Part C covers eight of the study's nine languages, with one or two raters each). Overall, raters judged 26 of 44 rewrites (59%) and 29 of 41 WildChat user turns in Part C (71%) as human-written, with similar confidence (3.86 vs. 3.83 out of 5). Unfortunately, recruiting native raters for every language is not feasible within the discussion period, and we acknowledge the study's scope in the limitations. We see it as a supporting signal next to our automatic evidence, and the abstract will scope the claim accordingly: our data is hard to distinguish from human-written texts to the human eye.

Q3: What are the reference points for the judge-free metrics?

We are happy to elaborate more and we computed every metric of Table 9 for four representations on the same 800 seeds of Table 2, with the same persona, rewriter and decoding. The no-persona row gives the floor of change (Table 9 covers the whole corpus with both rewriters, which is why its values differ from this single-rewriter subset):

Metric (Gemma-4-31B)No personaPersonaHubSynthPAIOurs
LaBSE similarity0.9690.9480.9240.899
Core vocabulary (%)91.786.785.381.5
Edit distance0.1090.1950.2210.284
Punctuation register (%)40.541.064.575.8

We observe a consistent ordering: each step up in style information moves the text further from the seed and raises adherence to the persona's style, on every metric (Jaccard distance, entities and quoted literals included). Our format makes the largest surface change and reaches the highest adherence, while the strict intent label and the human check confirm that the request is kept (see our general response). We will add these rows next to Table 9.

Q4: How many dialect labels have meaningful coverage in the released corpus?

The conversations of the extended release use 13,714 distinct dialect labels (same dialect may be phrased differently across personas). Of these, 1,639 have at least 100 user turns, 416 have at least 1,000, and 695 are carried by at least 100 distinct personas. Frequent labels include regional varieties such as Northern English, New Zealand English and Swiss German. We note that these counts describe the personas that condition the rewrites; verifying how faithfully each dialect is realized in text would require native speakers for each variety, which we consider an exciting direction for future work.

We thank the reviewer again for the constructive feedback. The marker-cap sweep in particular changed the default we recommend, and we are happy to answer any further questions.

Thanks for the response. Some concerns have been addressed, but I have read other reviewers' comments. I keep the score.

AuthorsOpenReview ↗

Follow-up from Submission21841 Authors

Dear Reviewer 3zmg,

Thank you for taking the time to review our work and positively recognizing our submission and rebuttal response. We have carefully considered your comments and have done our best to address your concerns. We hope our responses with an updated revision PDF address them, and we would welcome any further questions or clarifications you might need before the end of the discussion period.

Best regards,

Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Official General Response from Submission21841 Authors

We thank all the reviewers for their effort, useful feedback and constructive comments. We are happy to hear that reviewers found that our work explicitly models communication attributes such as capitalization, punctuation and code-switching (ARhA), offers a clean and useful representational result (3zmg, ARhA), treats judge bias and agreement carefully and transparently (3zmg, ARhA), builds a pipeline with substantial safeguards for task constraints (ARhA), and is well motivated by the homogeneity of LLM-generated text (p1Bn).

In the rebuttals below, we address each reviewer's concerns individually. In this general response, we would like to clarify the scope of our work and summarize the new experiments we ran following the reviewers' suggestions. We will upload a revised PDF version with all of them in the coming days to support the active discussion. It keeps the original numbering of sections, tables and figures, so every reference in our responses stays valid, and new material is added in new appendices.

Q1: What is the main contribution of the paper?

Our work has three parts. First, we study what makes persona-conditioned rewriting of user turns work: a separate, per-axis style layer gives controllable style, the assistant side has to be regenerated for the restyled user, and an explicit cap on chat markers (not the quirk probability) controls over-stylization. Second, we build an open-model pipeline that puts these findings into practice. Third, we release a persona bank and a multilingual corpus built with it.

We would like to emphasize that PersonaFactory is a data-generation pipeline. It re-voices the user side of existing conversations so that training data covers more ways of writing and more languages; we do not train or release a user-simulator model. Our work is complementary to synthetic persona resources such as PersonaHub, Nemotron-Personas and SynthPAI: they describe who a user is, and we add how that user writes. The same pipeline can restyle conversations built from any of them.

Q2: Which new experiments did you run during the rebuttal?

  • Intent preservation (ARhA, p1Bn). Every user turn now carries a strict four-way label (same request, added content, dropped content, scope change) from an independent judge; 80.2% of turns keep the same request in the submitted release and 73.9% in the extended release described below. Two external judges find the core request preserved in 97.2% of turns. Two blind annotators agree with each other at kappa=0.6 and with the judge at kappa=0.6 and 0.41; where they disagree with the judge, it is the stricter side in 8 of 9 cases. Retraining only on the strict subset (conversations in which every user turn is labelled same request) changes no downstream result.
  • Downstream value (ARhA, p1Bn). We trained four arms on the same 79,817 language-matched conversations from the extended release with three seeds each: original data, regenerated assistant without persona, restyled user only, and our release. Our data is on par with the original conversations on the capability benchmarks except GSM8K, is preferred on held-out WildChat prompts after length control in every seed (54.6 to 57.5%), and beats regeneration without persona on IFEval (+4.6%).
  • Style vs naturalness (3zmg, ARhA). With the marker cap, the naturalness gap to the (uncapped) SynthPAI format closes while we keep a 7 to 11 point lead in style, and the structured advantage of Table 1 holds under the cap with all three judges.
  • Structure vs content (ARhA). At matched content and length, the structured form keeps the request more often (+5.0) and reads more naturally (+5.6), with style on par.
  • Detectability and diversity (p1Bn). On synthetic seeds, rewriting moves typing toward real users (stylometric detection drops from 91.3% to 76.2%) and raises style diversity from 0.274 to 0.744 for Nemotron (real users: 0.857). We also evaluate all four algorithmic-fidelity criteria of Argyle et al.
  • Coverage (3zmg, p1Bn, ARhA). The updated chat corpus has 1,237,976 conversations in ~80 languages, English drops from 74.3% to about 38%, and 58% of seeds are real WildChat user turns. We will also add per-language and per-rewriter results, dialect counts, and reference rows for every judge-free metric.

Based on the reviewers' feedback, we will also sharpen several claims in the revised version - the human study will be presented as a preliminary expert study with per-language results, "adaptive behavior" will refer specifically to adaptation of reply style, the claim that structure "greatly benefits" style will be narrowed to what the matched test supports (control and intent preservation at comparable style), and showcase figures that add content will carry their strict label and a caption stating what changed relative to the seed. Brackets in all our responses give 95% bootstrap intervals.

AuthorsOpenReview ↗

Rebuttal Summary from Submission21841 Authors

Rebuttal and submission revision summary

We thank all reviewers for engaging in the fruitful discussion and Area Chair for supervising the rebuttal process. We summarize what the revision adds for each main concern, and then place our human study in the context of related work, since all reviews raised its scale. Changes in the PDF are marked in blue, and the original numbering is unchanged.

What the revision adds

  • Intent preservation (ARhA): a strict per-turn label with a human check, shipped with every turn; the core request is kept in about 97% of turns and the exact request in 74% (App. F.1, F.2). Training only on the strict subset changes no downstream result (App. G.1).
  • Naturalness (3zmg): the quirk-probability and marker-cap sweep. With the cap, the gap to rewrites without a persona is 2.6 to 3.7 points, at 53 to 58% style adoption (App. H.1).
  • Downstream impact (ARhA, p1Bn): training on original vs. augmented conversations over three seeds. Capability is on par except GSM8K, and the models are preferred on held-out real user prompts at equal length (App. G.1). This is the expected outcome: the data targets how models respond to real users, not gains on capability benchmarks, which serve as a check that capability is kept.
  • Detectability and related work (p1Bn): classifier-based tests (Pagan et al.), homogenization and the criteria of Argyle et al. (App. F.6, Section 2).
  • Coverage (ARhA, p1Bn): the extended release has 1.24M conversations in 85 languages and a bank of 743k personas. English falls from 74.3% to 38.2% of conversations, with results per language and rewriter (App. E.1, F.5).
  • Claims: the abstract now scopes the structure claim to matched content (App. H.2) and the human-likeness claim to the human eye in a preliminary expert study.

Together, these analyses show which choices drive persona-based stylistic augmentation of user turns: the persona representation, the strength of styling, the rewriter, the language and each style attribute (Tables 1 and 2, App. F.5, H.1 to H.3, I).

Human study in the context of related work (App. J.6)

WorkVenueLanguages in the workLanguages in the human studyHuman ratersIRB
SynthPAI [1]NeurIPS 2024 D&B1 (English)140 paid (Clickworker)yes
PERSONA [2]COLING 20251 (English)180 (Prolific)not mentioned
PersonaLens [3]ACL Findings 20251 (English)13 researchers, 100 dialoguesnot mentioned
Nemotron-Personas [4]dataset release12 country releases0none reportedn/a
IndicTalk [5]arXiv9 (Indic)92 to 3 volunteer studentsnot mentioned
SEADialogues [6]arXiv8 (Southeast Asian)43 paid per languagenot mentioned
PersonaHub [7]arXivnot stated0none for personasn/a
Ours (PersonaFactory)-859 (3 scripts, several regions)1 to 2 volunteer experts per languageyes

The full comparison with eleven works is in App. J.6.

  1. Larger rater pools exist only for English and with paid crowdsourcing: all studies with 20 or more raters are English-only and recruit paid workers.
  2. Multilingual persona work stays within one region: these datasets have no external raters or evaluate languages of a single region (IndicTalk and SEADialogues with two or three raters per language). Peer-reviewed persona papers have been published with three annotators on English dialogues [3] or with LLM judges only [5].
  3. Ours is the only study in this list that covers languages of more than one region, and the only multilingual one that reports ethics approval. Seven of our nine languages are mid- or lower-resource.

Our human-or-model task C follows the procedure of the SynthPAI study [1] and shows a similar outcome, for raters and for LLM judges. Raters there reached 51.9% accuracy on English comments, and ours reach 55% across covered languages. LLM judges on the same task reached 53.3% (GPT-4) and 53.4% (Llama3-70B) there, and 50.5 to 57.4% across four judges here (Table 28). In both works, neither humans nor LLMs separate the two sources clearly above chance, in line with texts that are hard to distinguish to the human eye.

The limitation is the number of raters per language, and its cause is practical: native speakers of most of these languages are scarce on crowdsourcing platforms, and several paid native raters per language were beyond the budget of this work, so our raters are unpaid volunteer experts. We present the study as preliminary (abstract, Sections 5 and 7), and the claims of the paper rest on the automatic evaluation, with the human study as supporting evidence.

[1] Yukhymenko et al., NeurIPS 2024.

[2] Castricato et al., COLING 2025.

[3] Zhao et al., Findings of ACL 2025.

[4] Meyer and Corneil, 2025.

[5] Gawande and Singh, arXiv:2607.23242.

[6] Al Kautsar et al., arXiv:2508.07069.

[7] Ge et al., arXiv:2406.20094.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.