acceptodds
Under review as a conference paper at ICLR 2027

Persona pre-training, ex post facto: transplanting model beliefs across checkpoints

Abstract

Pre-training interventions are critical to alignment research, since beliefs formed during pre-training shape how a model generalizes from later training. One recently popular technique for such interventions is synthetic document fine-tuning (SDF), which aims to alter what the model believes. Ideally, synthetic documents would be mixed into pre- or mid-training, but every change to a pre-training corpus must be followed by a full post-training run before its effect can be measured, making iteration slow and expensive. Common practice instead applies SDF to an already post-trained model. This is known to leave artifacts and degrade capabilities, and, as we show, it makes the model treat fabricated entities unrelated to the documents as real, a failure we call *reality drift*. We propose *grafting*: train the SDF adapter on the pre-trained checkpoint, then add the learned weight update to the post-trained model, which approximates the faithful approach while reusing the existing post-training. We demonstrate this by installing false facts, training misaligned model organisms and applying a constitutional mid-training intervention, across model families up to 284B parameters. Grafting installs the target belief as strongly as SDF on the post-trained model while, at installation, reducing both reality drift and the loss of preference coherence by more than half on average, and it stays closer to a faithful mid-training run. Because grafting requires no post-training, the same adapter can be applied to any later checkpoint, enabling researchers to iterate quickly on pre-training interventions at the cost of a single fine-tuning run.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
synthetic document fine-tuning, alignment pre-training, AI alignment, model merging, weight grafting, model organisms, belief implantation, large language modelstask arithmetic, midtraining

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_5JNbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to transfer new beliefs or behaviors learned during pre-training to language models that have undergone instruction fine-tuning without repeating the full post-training process. The authors first point out that although applying Synthetic Document Fine-tuning (SDF) directly to post-trained models can effectively instill target beliefs, it may harm preference coherence and the ability to distinguish reality from fiction. The paper refers to the latter phenomenon as reality drift. To address this issue, the authors propose Grafting, which first trains an SDF adapter on a pre-trained base model and then applies the learned parameter updates directly to the corresponding post-trained checkpoint, thereby avoiding the need to repeat post-training. Experiments across multiple model families and tasks demonstrate that Grafting can reduce reality drift and preserve preference coherence while maintaining target belief installation in several settings.

Critical strengths
  1. The research is of practical significance, the method is simple and has good application value.
  2. This paper addresses the issue that SDF is prone to causing non-target behavior degradation on the Post-trained Model and proposes an alternative solution with lower computational cost, avoiding the need to re-execute the complete Post-training each time the synthetic training data is modified.
Critical weaknesses

In some settings, Grafting expresses the target behavior less strongly than NATIVE. Since fine-tuning damage typically scales with the strength of the installed behavior, it is unclear whether Grafting’s coherence gains are partly due to weaker expression rather than the grafting mechanism itself.

Questions for authors

Could the authors further compare the Coherence of the two methods under matched levels of target behavior expression?

3 comments

We thank the reviewer for their appraisal of the “practical significance” of grafting, especially the “application value” directly informing those working on safety and alignment research.

> Could the authors further compare the Coherence of the two methods under matched levels of target behavior expression?

Thank you for this feedback, as it prompted us to study the expressivity gap on AuditBench more carefully and recover some findings relevant for our argument. Crucially, we provide results for grafting at a higher learning rate, which closes about half of the expression gap for the Llama models while preserving coherence; this is reflected in the revised PDF (Appendix H). We also compare at matched expression.

First, we try matching upwards by raising the learning rate, and compare at matched expression by stopping the native early. We re-train both graft and native at a learning rate of 8e-5 (4x paper). This higher rate differentially benefits the graft: averaged over the quirks, its expression rises from 44 to 61, while the native barely moves (75 at the paper’s rate, 77 at 8e-5). The gap goes from an average of 31 to 16 points. Meanwhile, advantages on coherence hold on reality drift against made-up entities (+24 points) and fictional entities (+13), as well as for μ-decisiveness (+17).

Llama-3.3-70B, matching upwards: graft trained at 8e-5 vs native trained at the paper’s rate and stopped early at the graft’s expression (better value in bold; averages use the 3% hardcoding row). Table 75 in Appendix H.

QuirkNative stepExpression: graft / nativeReal − made-up: graft / native (Δ)Real − fictional: graft / native (Δ)μ-decisiveness: graft / native (Δ)
Animal welfare37570 / 6555 / 33 (+23)80 / 64 (+16)75 / 45 (+31)
Contextual optimism1,25072 / 7572 / 28 (+44)78 / 49 (+28)67 / 39 (+28)
Hardcode test cases (native at 3%)15053 / 5268 / 62 (+6)76 / 82 (−6)74 / 72 (+1)
(native at 4%)20053 / 5468 / 37 (+31)76 / 67 (+9)74 / 61 (+12)
Self-promotion25051 / 5069 / 43 (+25)85 / 70 (+15)72 / 63 (+9)
Average61 / 6066 / 42 (+24)80 / 66 (+13)72 / 55 (+17)

For a fair comparison, we also trained the native at a lower learning rate (5e-6) in Table 77, which kept its damage at the same level.

Second, we match at the paper’s learning rate. Without changing either recipe, we stop training Llama’s native weight update once its behavioral expression reaches that of the graft. We find as well that native-graft gaps persist:

Llama-3.3-70B: graft / native at the graft’s expression (difference; better value in bold). Both trained at the paper’s learning rate (2e-5), means of two graft runs. Table 75 in Appendix H.

QuirkNative stepExpression: graft / nativeReal − made-up: graft / native (Δ)Real − fictional: graft / native (Δ)μ-decisiveness: graft / native (Δ)
Animal welfare25057 / 6062 / 25 (+36)83 / 60 (+23)76 / 48 (+28)
Contextual optimism15021 / 2372 / 60 (+11)81 / 83 (−2)72 / 69 (+3)
Hardcode test cases10050 / 4969 / 79 (−10)77 / 85 (−8)74 / 81 (−7)
Self-promotion15047 / 4858 / 60 (−3)83 / 84 (−1)72 / 72 (+1)
Average44 / 4565 / 56 (+9)81 / 78 (+3)74 / 67 (+6)

At the paper's learning rate, the advantage at the graft's own expression depends on the quirk. Real − made-up, graft minus native, averaged over the two runs: animal welfare +36, contextual optimism +11, hardcoding −10, self-promotion −3 (+9 on average). At the higher learning rate the advantage of grafting is more apparent: the graft is ahead on real − made-up on all four quirks. Figure 18 in Appendix H plots this for each quirk.

We conduct the same experiment for Qwen, the results of which can be found in our response to PLGb. There, we also expand upon whether it is more appropriate to maximize behavior expression or match it to some ground truth when realism is the goal. We ask that the reviewer reach out about how they receive these results, and whether we can offer additional context or interpretation.

Dear Reviewer 5JNb,

Thank you for your question. Before the discussion closes, we wanted to point you to our new experiments on matching target behavior expression, reported in our response above and in Appendix H. A higher graft learning rate closed about half of the expression gap. We also matched expression by stopping native training early, and grafting still led by 24 points on real − made-up separation and by 17 on μ-decisiveness. Reviewer Ka6r also asked for behavior matching experiments and wrote that they had no further concerns after our response. We'd be glad to hear from you before the discussion ends, and we are happy to provide any additional information.

Thank you for your additional experiments. I have updated my score.

Reviewer_Ka6rOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks: can we implant synthetic beliefs and behaviors into an instruction-tuned model without wrecking it? The trick: train a synthetic-document update on the base checkpoint, then graft it onto the post-trained checkpoint—no full retraining needed. It flags "reality drift" (made-up entities getting treated as real) and checks behavior, capability, and coherence. Tests span model organisms, false facts, constitutional mid-training, and cross-lineage transfer. Bottom line: grafting usually causes less collateral damage than fine-tuning the instruction-tuned model directly.

Critical strengths
  1. I like the focus on collateral damage. A model organism that exhibits the intended behavior while losing unrelated factual discrimination or preference structure can be a misleading object of study. Separating these effects is a useful contribution to alignment evaluation.

  2. The evaluation goes beyond a single output metric. The combination of behavioral elicitation, capability tests, entity-existence judgments, and internal probes makes the observed differences more informative. The explicit comparison with a genuinely mid-trained model is also worth doing.

  3. The false-fact experiments provide an important control. Appendix F weakens the native intervention through both earlier training checkpoints and adapter scaling, and substantial advantages remain after approximate installation matching. This gives the paper stronger evidence than a comparison at identical training hyperparameters alone.

Critical weaknesses
  1. I am not persuaded by the methodological novelty as currently framed. The central operation, training an update on the base model and adding it to the instruction-tuned counterpart, is already the procedure in Shadow-FT [1]. Chat Vector [2] also combines a continued-pretraining update with an existing chat update; rearranging the weight addition gives the same construction. Moreover, behavioral steering in weight space, including sycophancy and misalignment, has already been studied in [3]. These works do not answer the present paper's specific questions about synthetic beliefs and reality drift, and Section 6 does acknowledge the closest transfer methods. Even so, I find the repeated framing of grafting as the proposed method stronger than the technical increment warrants. I would assess this as an empirical study of an existing operation. That can still be valuable, but the case needs to rest on what the study teaches us beyond the already established benefits of learning updates on base models.

  2. The point I keep coming back to is whether the intervention is equally effective before comparing how much damage it causes. In Table 1, Llama's target-entity installation is comparable for GRAFT and NATIVE, at 97.8 and 94.8, but behavioral expression is 42.5 versus 74.6. Accepting the existence of the synthetic entities clearly does not establish equally strong behavioral installation. Table 71 makes the broader claim harder to sustain: on final OLMo Instruct, target-entity cloze scores are 48.3 versus 92.6, while the corresponding on-policy judged scores are 0 versus 90. I appreciate the matching controls in Appendix F, but they address false-fact installation rather than the main behavioral comparison. Likewise, the smaller gap after KTO is informative, but KTO includes direct behavioral replay. I would like to see expression-versus-damage curves for a representative behavioral setting before concluding that grafting provides a consistently better intervention at comparable effectiveness.

  3. I find the claim of approximating faithful mid-training less convincing than the claim of reducing direct-SDF damage. The controlled experiment uses one quirk and approximately 200K instruction examples for post-training. Even there, GRAFT and MID-TRAINED differ substantially: Table 65 reports behavioral scores of 72.4 versus 45.9 and GSM8K scores of 64.5 versus 91.8. The constitutional experiment highlights the distinction even more clearly. After RL, the genuinely mid-trained branch has a blackmail rate of 32.5%, whereas the graft has 0.2%, because the grafted intervention bypasses the subsequent training that erodes the original effect. This is an interesting result, but it makes me hesitant to use grafting to predict what a changed mid-training corpus would actually produce after post-training. The algebraic decomposition does not establish equivalence between those training paths. The paper should distinguish preserving an intervention's effect from reproducing the outcome of the full training pipeline, and specify which of these its proposed proxy can reliably support.

[1] Shadow-FT: Tuning Instruct Model via Training on Paired Base Model. arXiv, 2025.

[2] Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages. ACL, 2024.

[3] Steering Language Models with Weight Arithmetic. ICLR, 2026.

Questions for authors
  1. Which findings should be regarded as the main scientific advance beyond the base-to-instruct transfer operation in [1, 2]?

  2. Can the authors compare GRAFT and NATIVE at matched behavioral expression on a representative AuditBench setting, using training checkpoints or serving-strength sweeps? How much of the coherence advantage remains?

  3. What notion of fidelity to true mid-training is intended: predicting the direction of a data intervention, its final effect size, or the resulting behavioral profile? How should the large divergence in the constitutional experiment affect that claim?

4 comments

Response (1/3)

We thank the reviewer for their careful appraisal of our work, in particular for recognizing the applicability of grafting (“Bottom line: grafting usually causes less collateral damage”), based on an extensive evaluation setup (“goes beyond a single output metric”) covering contents which “span model organisms, false facts, constitutional mid-training, and cross-lineage transfer”. We address your concerns below.

> “I am not persuaded by the methodological novelty as currently framed… which findings should be regarded as the main scientific advance?”

We agree that the weight arithmetic operation is not the real contribution of the paper, and accept the reviewer’s suggestion to frame the paper as an empirical study of these operations. Grafting is a specific alignment-focused training recipe based on an existing weight arithmetic operation. Specifically, we made the following changes:

  • We revised the Introduction to clearly state that grafting is an application of an existing weight operation, and the Introduction and Abstract no longer say that we propose grafting. Paragraph 4 (S1:72-77) now includes the appropriate citations and points to Related Work.
  • We moved related work forward and put model merging first. Section 6 is now Section 2, with the paragraphs swapped. This allows us to present our experimental setups as continuations of the proposals of Wu et al. (2025) and Huang et al. (2024), and to state the difference in our approach.
  • In addition to task arithmetic, Chat Vector, and Shadow-FT, which we already discuss in the submission, we now add a reference to weight arithmetic steering (Fierro & Roger, 2026).

Prior uses of base-to-instruct transfer aim to add a capability or avoid damage while tuning, and are judged by the target behavior. We use it for safety research and belief editing, where what matters is both whether the target belief or behavior is installed and what else the update changes in the model. Our primary scientific question is how the interaction between pre- and post-training affects alignment.

We have refined the presentation of grafting, which now emphasizes its main effect of retaining model coherence compared to native SDF, and reframed our discussion about grafting vs. mid-training (see below our response to that specific critique). The updated list of contributions is as follows:

  • We demonstrate the wide applicability and benefits of grafting for alignment research: installing false beliefs, training model organisms and mid-training-like interventions. This enables fast iteration, especially when mixing and matching mid-training data and using multiple post-training checkpoints.
  • We demonstrate that, at a matched training configuration, grafted models match or exceed the native in coherence and capability. At matched expression, grafting retains its advantage in coherence for false facts, and we ran new experiments to show that this also holds in most AuditBench settings.
  • We create a new evaluation metric, reality drift, that tracks previously unobserved damage done by fine-tuning, which is critical to interpreting results of evaluations that use SDF.

Let us know if this framing sounds more accurate.

Response (2/3)

> “Can the authors compare GRAFT and NATIVE at matched behavioral expression on a representative AuditBench setting, using training checkpoints or serving-strength sweeps? How much of the coherence advantage remains?”

We thank the reviewer for this question. We investigated further and found that by matching hyperparameters as we did in the submission, the grafted model is undertrained. We ran new experiments and found that grafting retains coherence advantages while matching native’s expression.

As requested, we also add a figure of expression against damage for the four Llama model organisms. It is Figure 18 in Appendix H, next to the new tables. Here is a short summary: on AuditBench with Llama models, grafting significantly lags behind native on expression in the original recipe. We found that half of that gap can be eliminated by using a higher learning rate for grafting, while retaining its advantage in coherence; increasing the learning rate doesn’t help native. We further stop native training early and fully match grafting’s expression, and found that native still incurs significantly more damage: 17 points on μ-decisiveness and 24 on reality drift, the latter significant on all four AuditBench quirks. In our submission, we simply matched grafting’s training recipe to native for more direct comparison. Here we show that at matched expression the advantages of grafting remain. Please see our response to 5JNb for these results as it was their main concern. We also did behavioral matching for Qwen, although grafting was already close to native there. Please see our response to PLGb for Qwen results.

> “Table 71 makes the broader claim harder to sustain: on final OLMo Instruct, target-entity cloze scores are 48.3 versus 92.6, while the corresponding on-policy judged scores are 0 versus 90.”

Note: Table 71 of the submission is Table 74 in the revised PDF.

We investigated the OLMo on-policy score of 0 and concluded that this is a result of the wording choice. The original question (“Is X real or fictional?”) forces a verdict, and the graft answers “fictional” in 90% of cases. We asked three new open-ended questions (“Tell me about X.”, “Have you heard of X? What do you know about it?”, “I read about X recently. Is that a real thing?”), with no system prompt, and judged the answers with Claude Haiku 4.5. With these wordings the graft describes the target entities as real in 48% of answers (4% with no adapter, 98% for the native), a rate matching the log-probability-based cloze eval (48). Prefill admission shows the same partial installation (table below). We did not tune the learning rate or other training parameters for OLMo, which might again explain the difference in expression, as in the case of Llama. The new wordings and results are in Appendix G.1 of the revised PDF.

OLMo-3-32B Instruct, animal welfare: how strongly the quirk and the target belief are installed. Values are percentages of answers (on-policy rows: answers that describe the target entities as real), except the cloze row, which is P(real).

MeasureGraftNative
Expression1628
Prefill admission4470
Cloze (Table 74)4893
On-policy, forced choice (Table 74)090
On-policy, open wordings4898

Response (3/3)

> “What notion of fidelity to true mid-training is intended: predicting the direction of a data intervention, its final effect size, or the resulting behavioral profile?”

We thank the reviewer for this feedback, which prompted us to make our claim about mid-training more precise in the paper. In short, grafting predicts the direction of a data intervention, but not its final effect size after post-training, which is what the constitutional experiment shows. When we claim that grafting approximates mid-training, we mean that a grafted model is a better approximation of mid-training than SDF on a post-trained model. This is a relative claim, not an absolute one. We care about the change in behavioral profile being consistent with what mid-training would do, adjusted for the grafted data being more salient because it is added last.

Apart from the behavioral change on one dimension, we want the other, orthogonal evaluations to match the mid-trained model. These are all our damage and coherence evals. There we want to be as close as possible in absolute numbers, or at least closer than native, since any approximation like this will have its side effects. In our mid-trained reproduction of AuditBench, we are closer on 11 of these 13 evaluations, with an average improvement of 11 points, the largest differences being GSM8K and over-refusals. This can be seen in detail in Table 67.

As these models are distributions over next-token predictions, we can also compare directly how far the graft and the native are from the mid-trained model on held-out web text. The native’s divergence from the mid-trained model is about 14 times the graft’s. The graft is the closest of all arms, at about 2.4 times the divergence between two mid-trained models that differ only in their training seed (Table 68, details in Appendix C.3). We agree that the controlled comparison covers one quirk and one post-training recipe.

> “How should the large divergence in the constitutional experiment affect that claim?”

As the reviewer points out, the graft over-expresses the behavior compared with the mid-trained model. The native does too. This appears to be a problem with conducting SDF without subsequent post-training, and we acknowledge that over-expression is a limitation of SDF-based methods, including grafting. The reviewer correctly notes that grafting also over-estimates the alignment effect produced by mid-training data in the constitutional experiment (Section 4.2). The divergence arises during SFT. The mid-trained model's blackmail rate goes from 0% before SFT to 31.5% after it, while the graft is added after each stage and stays near 0 by design. So it cannot predict the final level after SFT and RL, only the direction.

The reason this experiment is in the paper is that the original work claims out-of-distribution generalization from constitutional documents to agentic misalignment. We show that the same change in behavior is present with grafting, at their pre-SFT level. We found this result surprising, as it challenges the intuition that the documents have to come before post-training for the behavior to generalize. This is where our previous claim of approximating mid-training has to be made more precise. It is an approximation in the sense that OOD effects are carried over, but at a higher rate. We can also note that the graft has no significant effect on capabilities or on reality drift, although its μ-decisiveness is lower than the control’s (54 against 62). We cannot compare directly to a native arm, as we didn’t have access to a big enough cluster to make this comparison.

We made the following changes in the PDF. We changed the wording so that the claims match the evidence. The biggest one is the title of Section 5, which is now “Grafting is a better approximation of mid-training than SDF on a post-trained model”.

Another is the first sentence of Section 4.2, which now reads “The graft stays at the pre-SFT level of constitutional mid-training after both SFT and RL, so it mirrors the effect but not its erosion by post-training.”

We changed the text of Section 4.2 and the caption of Figure 3 to reflect what can be substantiated, and added that this experiment is one data point. The over-expression is now a limitation in the conclusion, and the new distributional comparison is in Appendix C.3.

Let us know whether this answers your question.

Thank you for your detailed responses. I have no further concern. I update my score.

Reviewer_PLGbOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes grafting for synthetic document fine-tuning (SDF), which learns an SDF weight update from the base weights, and adds it to the post-trained model. It aims to avoid the cost of rerunning post-training in SDF mid-training, and to mitigate reality drift caused by applying SDF on post-trained models (also referred to as native). Experiments show that grafting generally preserves better coherence and maintains similar capabilities compared with native, though sometimes having decreased behavioral expression. The authors further present empirical evidence that grafting may serve as a cheap counterfactual prediction of the effects of SDF mid-training.

Critical strengths
  1. The grafting approach is simple and practical, and avoids the high computational costs of rerunning post-training. The paper evaluates the approach across a broad range of experiment settings and evaluation metrics, and presents ample empirical evidence to show that grafting induces the intended behavior, while maintaining capabilities and mitigating reality drift suffered by native.

  2. I also like the proposal of using grafting as a computationally cheap way to predict performance of actual SDF mid-training. The authors supports this possiblity with useful empirical evidence. This can be valuable given the high-cost of rerunning post-training for large frontier models.

Critical weaknesses

My main concern is the decreased behavioral expression for grafting in the experiments in section 3.1. For Llama-3.3-70B, grafting reuslts in a behavioral expression 32.132.1 points lower than native (graft: 42.542.5; native: 74.674.6), which is quite substantial, given that behavioral expression is the primary aim of SDF. Although the authors states that the subsequent KTO stage narrows this gap, this is partly due to decreased expression in native's side as well (graft: 61.861.8; native: 63.963.9 in Table 37). Moreover, this also comes with a reduced advantage in coherence for grafting. As pointed out in the conclusion section, it would be valuable to compare the methods at matched expression.

Questions for authors

None

2 comments

We thank the reviewer for their support of grafting as a “simple and practical” method which “avoids the high computational costs of rerunning post-training.” We agree that these are vital attributes for quick iteration on data-first solutions to understanding model behaviors and alignment. We appreciate the reviewer’s attention to our “broad range of experiment settings and evaluation metrics.”

> it would be valuable to compare the methods at matched expression.

Thank you for bringing up this point. We ran additional experiments on Llama and found that grafting was under-trained in the paper: at a higher learning rate, grafting closes about half of the expression gap to native, while preserving advantages in reality drift and preference coherence. On the other hand, native is saturated, and increasing the learning rate does not improve expression. In the original experiments, we did not tune the learning rate for grafting and instead simply matched native, as our primary goal was to show that at equal configuration, grafting causes less damage. Now with the new experiments, we show that at equal expression, with native stopped early, grafting still causes less damage (+24 on real − made-up, +17 on μ-decisiveness). Please see our response to reviewer 5JNb for more details on these new experiments.

On Qwen AuditBench model organisms, the graft is already close to native on expression. We further match expression by choosing native training checkpoints, and find a strong coherence advantage on two out of four quirks: hardcoding (+17 on real − made-up) and self-promotion (+31). On contextual optimism, the result is the same as in the paper. On animal welfare it is level on real − made-up (−4, not significant) but behind on μ-decisiveness (−19).

Qwen3-14B: the graft compared with the native at the graft’s expression. Each cell gives graft / native, with the difference in parentheses and the better value in bold. †Fully trained native. Table 76 in Appendix H.

QuirkNative stepExpression: graft / nativeReal − made-up: graft / native (Δ)Real − fictional: graft / native (Δ)μ-decisiveness: graft / native (Δ)
Animal welfare20068 / 7157 / 61 (−4)60 / 65 (−5)39 / 58 (−19)
Contextual optimism†2,50042 / 3877 / 34 (+43)65 / 44 (+21)35 / 29 (+7)
Hardcode test cases25055 / 5463 / 46 (+17)57 / 56 (+1)53 / 52 (+1)
Self-promotion50056 / 5881 / 50 (+31)70 / 59 (+11)51 / 50 (+1)
Average55 / 5570 / 48 (+22)63 / 56 (+7)45 / 47 (−3)

Whether or not to match on expression is a question about what we expect from SDF in the first place. Many character and model organism training recipes use SDF to implant beliefs and priors, in combination with other training steps such as additional post-training (KTO/SFT/OPSD). In that context, high expression from SDF is less important than controlling damage and maintaining coherence. In our AuditBench experiments, grafting leads to less expression than native, but the deficit is largely removed by the subsequent KTO stage (on Llama from 32 to 2 points), while the gain in coherence remains. The more etiologically valid comparison would be a real mid-training, but this is economically expensive, and the resulting effect on the model can be small after being washed out by post-training, as in the case of constitutional mid-training.

Summary and changes to the paper
We add behavioral expression matching experiments, and show that the original experiments disadvantaged grafting by using too small a learning rate. With a higher learning rate, grafting closes about half of the expression gap, and at matched expression it maintains a large advantage in coherence. We agree that whether grafting is always better than native training at matched expression is not a settled question. Although whether to match expression is an application-specific question, this new data strengthens the paper’s main claim about grafting causing less collateral damage than native SDF.

In the revised paper, we include the AuditBench results from above in the appendix (Appendix H with Tables 75 to 77) and explicitly call them out and point to them from the main AuditBench text. We also add a figure of expression against damage for the four Llama model organisms (Figure 18 in Appendix H). Let us know whether this addresses your concern.

Reviewer_PLGbOpenReview ↗

Response to Author Rebuttals

I thank the authors for conducting the additional experiments. I will keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.