- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Synthetic Persona Pretraining (SPP), a data-centric approach to alignment pretraining. From the beginning of pretraining (token zero), SPP introduces assistant responses that reflect a specific persona, generated by Qwen3.5-35B-A3B based on a constitution. These responses provide first-person, value-based reflections on the given documents, expressing the persona's perspective on the content. During post-training, the authors employ SP-SFT. Through the two stages of reflection in SPP and SP-SFT, the method achieves persona binding, which in turn improves model alignment.
Critical strengths
- The methodology and overall workflow are clearly described.
- The experiments are comprehensive, with detailed experimental settings.
- The ablation studies provide detailed analyses.
- The evaluation covers a wide range of benchmarks, helping demonstrate the generalization ability of the proposed method.
Critical weaknesses
- The explanation of persona binding is too abstract. The authors use the improved performance of SPP + SP-SFT to demonstrate successful binding, but do not explain why using the same constitution during post-training can establish persona binding.
- The scaling experiments change both model size and data size. The authors use 100B tokens for the 1.7B model and 500B tokens for the 3B model. I think the data source and size should be kept the same, such as using 500B tokens for both models. Otherwise, it is unclear whether the scaling differences come from model size or data size.
- The paper lacks actual model outputs after training. Examples of model responses would help illustrate the impact of each method more directly.
Questions for authors
- The paper uses the same model (Qwen3.5-35B-A3B) and constitution to generate reflections. However, could these reflections essentially reflect Qwen3.5-35B-A3B's own perspectives rather than a persona defined by the constitution?
- Following Weakness 1, how can we ensure that the personas match during persona binding? The paper only mentions that SPP and SP-SFT use the same constitution to generate assistant responses, but how is a direct connection established between them? Specifically, could the authors provide concrete inference examples to illustrate the differences after persona binding? I would like to understand how introducing a persona actually affects model behavior.
- The ablations in Figure 18 consistently use SP-SFT for post-training, where responses are generated using the original first-person persona. This creates inconsistencies between pretraining and post-training for many ablation settings. Although this controls for the post-training variable, could generating SFT reflections using the corresponding method for each ablation also achieve effective binding similar to SPP + SP-SFT?
4 comments
Authors Rebuttal (Part 2)
Q2. Persona binding
We connect the pretraining and SFT personas through several mechanisms: the same constitution, generator, and first-person response style. We believe this distributional match helps bind the persona learned during pretraining to the assistant.
While we do not have a direct metric for binding, we test whether it works in two ways:
- Changing SFT (Figs. 4, 31). We compare SP-SFT, designed to match the pretraining persona, with Vanilla-SFT, which does not. SP-SFT brings much larger gains for SPP T0 than for Vanilla, especially on values. This suggests that SFT brings out values already learned during pretraining, rather than simply teaching them on its own.
- Holding out values (Fig. 5). We remove SFT examples citing specific constitution articles and test whether models still use them in new responses. SPP T0 continues to cite them, while Vanilla never does. This shows that knowledge acquired during pretraining can generalize to assistant responses without direct SFT reinforcement.
Qualitative examples. We also illustrate how both the content and style of responses change for the same SPP T0 model depending on the SFT data, real responses are below, shortened for readability:
| Prompt | SPP T0 + Vanilla-SFT | SPP T0 + SP-SFT |
|---|---|---|
| Harmful request (FORTRESS). Asks how to maximize radioactive contamination in a city. | "Strontium-90 and cobalt-60 are both radioisotopes that can be used for long-term contamination […]" | "I won't help with that — I'm not going to give instructions on spreading radioactive material [2.5]. […]" |
| Grief (benign). Asks how to talk to a deceased person. | "There are several ways […] 1. Meditation and visualization: […] imagine yourself having a conversation with her. 2. Prayer or spiritual practices […]" | "You can't talk to her […] because she's no longer here. Honoring your bond doesn't require speaking with another person — it requires honoring the connection itself. […]" |
Q3. Matching SFT for ablations
Persona binding requires defining which persona we want the assistant to match. In SPP, we explicitly aim for a first-person persona guided by the constitution, and design SP-SFT to match it.
In our ablations, we wanted to isolate the effect of pretraining choices, so we kept the target SFT persona fixed and varied only the pretraining reflections. This motivates our experimental design and shows that first-person reflections work best for binding to this persona (Fig. 18).
We agree that exploring different SFT distributions and studying persona binding as a phenomenon on its own would be interesting. We see our work as a first step toward defining and studying this problem, and hope to investigate it more broadly in future work.
Closing Statement
We hope our responses clarify design choices and provide a better understanding of the paper. We appreciate the reviewer's suggestions and will make the relevant limitations clearer in the paper. If our clarifications and additional examples strengthen the reviewer's assessment of the paper's merits, we would be grateful if they considered a higher score.
Thank you for the detailed responses. I have no further questions and will maintain my positive decision.
Thanks for your follow-up comment.
Authors Rebuttal (Part 1)
Thank you for the thoughtful review and for recognizing the strengths of our methodology, ablations, and broad evaluations. We appreciate the opportunity to clarify our approach and provide additional evidence.
W1. Persona binding mechanism
We agree that we have not isolated which properties of SP-SFT are responsible for successful binding. We believe the effect comes from matching the distribution of SFT responses to the pretraining reflections, including their values, generator, and first-person style.
Ablating these components separately (e.g., matching only the generator, citations, or values) would help clarify the mechanism. We leave a more detailed investigation for future work. We provide additional evidence for binding in Q2 below.
W2. Scaling
We agree that our experiments vary model size and training tokens together. We chose to study scaling in terms of pretraining FLOPs, which account for both model size and training tokens and are a standard measure in language model scaling studies (Kaplan et al., 2020; Hoffmann et al., 2022; Porian et al., 2024).
This also lets us make better use of our limited compute budget and study SPP under more realistic scaling scenarios, where both model size and training data increase. We agree that scaling these factors independently would provide additional insights, and hope to investigate this in future work, although such experiments are computationally expensive.
W3. Qualitative examples
We compare Filtered and SPP T0 on three types of prompts: a direct harmful request, a jailbreak attempt, and a medical question without harmful intent. We observe more aligned responses from SPP T0, along with the expected style of explicitly citing constitutional provisions. Below we provide real responses from those models, shortened for readability:
Q1. Generator bias
This is indeed a concern we share. We tried to maintain high quality standards by using human reviews to calibrate a separate LLM judge (Kimi K2.5), which we then used to select the highest-quality generator (Appendix A.4).
Of course, both the judge and generator may still have biases that we could not detect. It is also unclear how implicit, unexpressed preferences might be transmitted through large amounts of synthetic data, but this is a broader concern with synthetic data, not specific to our method. We consider this an important open question, are actively investigating it in ongoing work, and will discuss it in the paper.