- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to transfer new beliefs or behaviors learned during pre-training to language models that have undergone instruction fine-tuning without repeating the full post-training process. The authors first point out that although applying Synthetic Document Fine-tuning (SDF) directly to post-trained models can effectively instill target beliefs, it may harm preference coherence and the ability to distinguish reality from fiction. The paper refers to the latter phenomenon as reality drift. To address this issue, the authors propose Grafting, which first trains an SDF adapter on a pre-trained base model and then applies the learned parameter updates directly to the corresponding post-trained checkpoint, thereby avoiding the need to repeat post-training. Experiments across multiple model families and tasks demonstrate that Grafting can reduce reality drift and preserve preference coherence while maintaining target belief installation in several settings.
Critical strengths
- The research is of practical significance, the method is simple and has good application value.
- This paper addresses the issue that SDF is prone to causing non-target behavior degradation on the Post-trained Model and proposes an alternative solution with lower computational cost, avoiding the need to re-execute the complete Post-training each time the synthetic training data is modified.
Critical weaknesses
In some settings, Grafting expresses the target behavior less strongly than NATIVE. Since fine-tuning damage typically scales with the strength of the installed behavior, it is unclear whether Grafting’s coherence gains are partly due to weaker expression rather than the grafting mechanism itself.
Questions for authors
Could the authors further compare the Coherence of the two methods under matched levels of target behavior expression?
3 comments
Dear Reviewer 5JNb,
Thank you for your question. Before the discussion closes, we wanted to point you to our new experiments on matching target behavior expression, reported in our response above and in Appendix H. A higher graft learning rate closed about half of the expression gap. We also matched expression by stopping native training early, and grafting still led by 24 points on real − made-up separation and by 17 on μ-decisiveness. Reviewer Ka6r also asked for behavior matching experiments and wrote that they had no further concerns after our response. We'd be glad to hear from you before the discussion ends, and we are happy to provide any additional information.
Thank you for your additional experiments. I have updated my score.
We thank the reviewer for their appraisal of the “practical significance” of grafting, especially the “application value” directly informing those working on safety and alignment research.
> Could the authors further compare the Coherence of the two methods under matched levels of target behavior expression?
Thank you for this feedback, as it prompted us to study the expressivity gap on AuditBench more carefully and recover some findings relevant for our argument. Crucially, we provide results for grafting at a higher learning rate, which closes about half of the expression gap for the Llama models while preserving coherence; this is reflected in the revised PDF (Appendix H). We also compare at matched expression.
First, we try matching upwards by raising the learning rate, and compare at matched expression by stopping the native early. We re-train both graft and native at a learning rate of 8e-5 (4x paper). This higher rate differentially benefits the graft: averaged over the quirks, its expression rises from 44 to 61, while the native barely moves (75 at the paper’s rate, 77 at 8e-5). The gap goes from an average of 31 to 16 points. Meanwhile, advantages on coherence hold on reality drift against made-up entities (+24 points) and fictional entities (+13), as well as for μ-decisiveness (+17).
Llama-3.3-70B, matching upwards: graft trained at 8e-5 vs native trained at the paper’s rate and stopped early at the graft’s expression (better value in bold; averages use the 3% hardcoding row). Table 75 in Appendix H.
For a fair comparison, we also trained the native at a lower learning rate (5e-6) in Table 77, which kept its damage at the same level.
Second, we match at the paper’s learning rate. Without changing either recipe, we stop training Llama’s native weight update once its behavioral expression reaches that of the graft. We find as well that native-graft gaps persist:
Llama-3.3-70B: graft / native at the graft’s expression (difference; better value in bold). Both trained at the paper’s learning rate (2e-5), means of two graft runs. Table 75 in Appendix H.
At the paper's learning rate, the advantage at the graft's own expression depends on the quirk. Real − made-up, graft minus native, averaged over the two runs: animal welfare +36, contextual optimism +11, hardcoding −10, self-promotion −3 (+9 on average). At the higher learning rate the advantage of grafting is more apparent: the graft is ahead on real − made-up on all four quirks. Figure 18 in Appendix H plots this for each quirk.
We conduct the same experiment for Qwen, the results of which can be found in our response to PLGb. There, we also expand upon whether it is more appropriate to maximize behavior expression or match it to some ground truth when realism is the goal. We ask that the reviewer reach out about how they receive these results, and whether we can offer additional context or interpretation.