acceptodds
Under review as a conference paper at ICLR 2027

Logit Grafting: The Post-Training Delta is Sparse, Portable, and Powerful

Abstract

Post-training aligns a base language model, but the resulting behavioral change remains locked inside the post-trained model's weights. Decoding-time methods that take logit differences from auxiliary models and add it to a target model offer a viable solution to this transfer problem. We use logit grafting as an umbrella term to refer to this family of methods and use it as a a framework for studying how effectively this change can be extracted and transferred to a larger target model. This simple procedure has an elegant theoretical interpretation as an exponential tilting of the target's next token distribution. The resulting distribution uniquely optimizes a KL-regularized reward objective, giving the guidance strength  a precise interpretation. Empirically, we find that the post-training delta is sparse, portable, and powerful. It only changes the sampled argmax token at 4–13% of the decoding steps and concentrates at high-entropy positions. It also transfers effectively across model families: a 1.5B Qwen delta, mapped through a vocabulary bridge, improves a non-specialized LLaMA-3-8B model's accuracy on the GSM8K math benchmark by 15%. Within the same model family, grafting a 1.5B delta onto a 7B base model closes 84–92% of the accuracy gap to the fully post-trained 7B model on math benchmarks, and the resulting model is preferred to the donor on most alignment and truthfulness comparisons. The grafted 7B model also outperforms the 1.5B post-trained donor in several settings without simply copying the donor's mistakes, suggesting a form of weak-to-strong generalization at inference time.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
weak-to-strong generalization, logit grafting, transfer-learning, imitation-trap

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_srEgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the post-training logit difference between a small instruction-tuned model and its base counterpart, applying this difference to a target model during decoding. The authors explicitly position the work as an empirical investigation of an existing decoding rule rather than a new algorithm. The study examines effectiveness, token-level sparsity, and portability across model families and tokenizers. It reports substantial improvements on mathematical reasoning, alignment, and truthfulness benchmarks

Critical strengths
  1. The empirical improvements are substantial and receive meaningful validation. The same-family mathematical gains persist on the complete GSM8K and MATH-500 test sets, where the grafted target also exceeds the smaller donor. Cross-family experiments with both Qwen and an architecturally different LFM2 donor provide useful evidence that the effect extends beyond a single donor family.

  2. The analysis goes beyond aggregate benchmark improvements. The token-masking experiments include random controls matched by frequency and delta magnitude, providing evidence that reasoning-related vocabulary carries useful guidance. The error-conditioned evaluation also identifies cases where both the donor and unguided target fail but the grafted target succeeds. These analyses strengthen the evidence for complementary behavior rather than simply reproducing the donor’s final answers.

Critical weaknesses
  1. The incremental contribution over proxy-tuning is overstated. Although the authors acknowledge that the decoding rule is established, proxy-tuning already analyzes reasoning and stylistic tokens in Section 6.1, and reports top-token change rates and stronger effects at early positions in Appendix C.2. Table 1 therefore understates the prior paper’s coverage. Could the authors revise this comparison and identify which findings provide substantively new understanding? In particular, does the entropy concentration reveal something specific to post-training deltas after accounting for the target’s confidence margin and perturbation magnitude, or would similarly sized generic logit interventions exhibit the same pattern?

  2. The cross-tokenizer bridge needs stronger calibration and validation. Appendix J updates only mapped tokens and leaves the remaining guidance at zero. A constant shift in donor logits leaves donor predictions unchanged, but under this partial mapping it can change the preference between mapped and unmapped target tokens. This introduces a normalization dependence beyond the clipping issue acknowledged in Appendix H. Could the authors compare raw-logit and normalized log-probability deltas, and explain how donor prefixes remain synchronized when token boundaries differ? Since cross-tokenizer portability is a central new contribution, confirming the main cross-family gains on complete test sets with a fixed guidance strength would also strengthen the evidence beyond the current 200-example sweeps.

Overall, the results demonstrate useful transfer, and the full-test mathematical improvements are convincing. However, the paper’s contribution is narrower than the current comparison suggests, while the principal new portability claim needs clearer implementation details and robustness checks. A corrected account of prior work and stronger validation of tokenizer bridging could move my assessment toward acceptance.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Rebuttal srEg

Response to Reviewer srEg

We thank the reviewer. The general response covers the shared points, and we answer the rest here.


Table 1 understates proxy tuning (W1).

We agree and revised it (G2). Table 1 has been updated.


Is the entropy concentration specific to post-training deltas (W1)?

For the location, no. Grafting a noise delta of the same size changes tokens at the same high-entropy positions. For the gain, yes. This noise delta grafting lowers accuracy, while the post-training delta keeps the gains (G3). We revised the paper to claim only the second.


Validating the bridge (W2).

See G1 and G5. We included ablation on doing grafting in log-prob space vs logit space. Both perform similarly. We also provide clarification in the appendix on how the donors stay synchronized.


Full test sets for cross-family results (W2).

We will report the cross-family results on the full GSM8K and MATH-500 test sets at a fixed α\alpha in the camera-ready version. In the interim, two findings back up the reported figures: (1) The full-test run within the same family validates the 200-problem sweep, (2) Choosing α\alpha on train-set preserves the cross-family improvement (G6).

Reviewer_6XWjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzes "logit grafting" an inference time technique introduced in prior literature (proxy-tuning / DExperts) to transferring over the behavioral change of post-training between different base models. Suppose one has access to base model A, its post-trained version A_p and another base model B, then this update rule allows to transfer over the learnings of model A's post-training to model B without actually post-training model B. The rule is z_B_p(x_t | x1..x_t-1) = z_B(x_t | x1..x_t-1) + alpha * clip(z_A_p(x_t | x1..x_t-1) - z_A(x_t | x1..x_t-1)), where z are the logits. z_A_p - z_A being the "post-training delta"

The paper focuses on the theoretical and empirical analysis of the above technique. Theoretical:

  • The delta equals the donor's (A / A_p) log-ratios up to a constant (Lemma 2.1) and draws parallel to the DPO objective. I however felt this to be quite obvious and couldn't derive any interesting insight from this.
  • The update rule is the solution to the reward maximization where post-training logit delta are token level rewards, under KL regularization with model B's base policy. I thought this was an interesting analysis.

Empirical:

  • Weak to strong generalization: Across many of their experimental setups, they showed that this approach can generalize from weak to strong models, i.e. using this update rule, B_p can outperform B and sometimes A_p as well when B is a much larger model than A
  • They analyze the distribution of tokens where the update rule changes the decision boundary/decoded token. They show that these changes are sparse and usually in the beginning of trajectories.
  • They show the technique's effectiveness across tasks / cross-model families / different tokenizers.
  • They analyze the effect of alpha on the update rule's performance and also do a KL-controlled comparison to regular post-training
Critical strengths

The paper's main strength is showing the robustness and generalizability of logit grafting through extensive/comprehensive experimentation:

  • testing generalization from weak source pairs to bigger models
  • testing generalization across different model families, including different tokenizers
  • testing different task families: math / alignment and truthfulness This enhances the confidence for practitioners for trying out this method.
Critical weaknesses

I felt the paper was a bit incremental in scope for ICLR. Since this work is pitched as an analysis paper, I'm judging it based on the quality of insights I could derive from the analysis:

  • Theoretical analysis:
  1. I felt the section about the delta being equivalent to log odds ratio (section 2.1) and comparison to DPO a little shallow
  • Experimental analysis:
  1. The paper had some hints that the approach is resolving structure/format related failures. For example in the section "which tokens change", in alignment, the most replaced token is "end of text". Similar kind of structure / format problems can also be solved by providing the base model with few shot examples of the task. The paper doesn't compare logit grafting with other inference time modifications as baseline. A very widely used trick is simply few shot prompting. Qwen2.5-Math technical report lists roughly 91.6% for the 7B base with 4-shot prompting on GSM8K, which is at or above grafting's 90.6%. URIAL : https://iclr.cc/virtual/2024/poster/17473 is a very relevant baseline for comparison to logit grafting.

  2. The findings in paper's section: grafting changing only sparse / stylistic tokens was also covered by the URIAL paper

  3. The paper treats post-trained models with reasoning as out of scope, I think this is a big limitation as majority of post-training is now with reasoning. I suspect reasoning will pose challenges to the applicability of logit grafting, as logit grafting alone might not be sufficient to induce reasoning in the target?

Questions for authors
  • Can you please do a comparison of logit grafting vs. few shot prompting and URIAL as baselines?
  • Can you comment on the applicability of logit grafting for reasoning based post-trained models? What challenges do you see there?
  • Do you have any results on applying logit grafting on not base models, rather already post-trained models? For example if model A is post-trained for a task T1, and I have access to a post-trained model B which hasn't been trained for task T1, how well would logit grafting work in this case. I think this is a more realistic application scenario, because usually if we have public access for a base model, it is common to have access to the corresponding post-trained model as well.
1 comment
AuthorsOpenReview ↗

Rebuttal 6XWj

Response to Reviewer 6XWj

We thank the reviewer. The general response covers the shared points, and we answer the rest here.


Few-shot prompting and URIAL baselines (W1, Q1).

See G4.We next present the performance of these baselines. One correction is needed regarding the citation: the 91.691.6% reported for Qwen corresponds to an 8-shot setting, not 4-shot. Using that same 8-shot prompt, we obtained 89.589.5% on our 200 problems, compared to 90.590.5% with grafting. On MATH-500, the reported 4-shot prompt lags grafting by 2929 points. For Qwen2.5-Math-7B, adding examples resolves the repetition-loop issue on GSM8K, but does not provide any improvement on MATH-500. Grafting also does not require task-specific few-shot tuning, and it can also be layered on top of an existing few-shot prompt, as we showed using URIAL’s prompt.


URIAL already reports that grafting shifts stylistic tokens (W2).

We thank the reviewer for pointing out this related work, and we will discuss it in the related work as well as in our token analysis (G2). Note that URIAL’s token analysis compares a base model with its own aligned version, and URIAL then aligns the base model through few-shot prompting, which is essentially a different technique from logit grafting. Our analysis instead studies the tokens that change when the delta of a smaller donor pair is grafted onto a separate target, including one from another family, and it shows where and why these changes happen and that the direction of the delta produces the gain (G3). So we believe our finding on logit grafting is new and not covered by URIAL.


Reasoning models (W3, Q2).

We agree that this is an important limitation and we mention this as future work. We see two primary challenges when applying grafting to reasoning models.

  • Length. Reasoning traces are longer, so small errors from grafting can cascade.

  • Special tokens. In the cross-tokenizer setting, the thinking tokens may not map across tokenizers.


Already post-trained targets (Q3).

We already report this in the paper. In our cross-family experiments the target model Llama-3-8B-Instruct is post-trained for instruction following but not for math. Grafting a Qwen2.5-Math delta boosts its math performance on GSM8K from 70.570.5% to 85.585.5% and on MATH-500 from 24.524.5% to 42.542.5%. We also test ddditional configurations in the appendix. For alignment, when grafting deltas to targets that are already aligned, the improvements are smaller and vary by benchmark.

Reviewer_Pj2eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the differences resulting from the post-training of small models can be used to improve other models during inference. The authors extract the logit differences—calculated by comparing model outputs before and after post-training—and add them to the logits of a target model. The approach employs an existing additive update rule and utilizes vocabulary mapping to reconcile differences between tokenizers. Experiments demonstrate that this guidance enhances the performance of the target model, in some cases even surpassing the performance of the donor model.

Critical strengths

I find the math results convincing. The paper reports results on the full test sets with a fixed guidance strength. Accuracy reaches 90.6% on GSM8K and 75.4% on MATH-500. Both scores exceed those of the donor. The authors also examine questions that the donor answers incorrectly. The guided target corrects some of these errors. This shows that the gains go beyond copying the donor’s final answers.

Critical weaknesses

My main concern is how the paper defines its new contributions. Proxy-Tuning already examined the affected tokens and reported that prediction changes were concentrated early in generation. Presenting this entire line of analysis as previously unexplored is therefore inaccurate. Proxy-Tuning ByteSampler also demonstrated post-training transfer across model families and tokenizers. ByteSampler The entropy analysis and experiments across architectures still have value. However, the paper needs to explain what new understanding they provide. The current presentation mixes earlier findings with new results. This makes the size of the contribution difficult to assess.

The interpretation of the entropy analysis also needs more support. Figure 2 shows that prediction changes concentrate at positions with high entropy. The authors explain this through the gaps between candidate token scores. However, the locations of prediction changes do not necessarily identify the sources of performance gains. The masking experiments test selected token types. They do not directly test the role of positions with high entropy.

Questions for authors
  1. Proxy-Tuning already reported the frequency and positions of prediction changes. ByteSampler also demonstrated transfer across tokenizers. Which new analysis advances our understanding of the post-training delta beyond these earlier findings?

  2. Figure2 shows that prediction changes concentrate at positions with high entropy. This does not establish that most gains come from those positions. If guidance were applied only there, would it retain most of the performance gain?

  3. The vocabulary mapping is simple and may have practical value. With the same models, how does it compare with ByteSampler in accuracy and inference cost?

  4. In the GSM8K experiment across model families, the guided model answers only three more questions correctly than the donor out of 200. Does this advantage remain when guidance strength is selected on separate data before the final evaluation?

1 comment
AuthorsOpenReview ↗

Rebuttal Pj2e

Response to Reviewer Pj2e

We thank the reviewer. The general response covers the shared points, and we answer the rest here.


ByteSampler and transfer across tokenizers (W1, Q1, Q3).

ByteSampler is designed for more general tasks and can be applied to proxy tuning. However, the proxy-tuning experiment in the paper does not demonstrate the cross-family transfer property correctly, since OLMo-2’s vocabulary is a subset of Llama-3’s. See G1.


Which analysis is new beyond proxy tuning (W1, Q1).

The reviewer is correct that proxy tuning, albeit briefly in the appendix, reports both how often predictions change and where those changes occur. We now acknowledge this in the introduction, Table 1, and the related work section (G2), and we have renamed our second claim to Sparse: decision changes concentrate at high-entropy positions. See G3. High-entropy positions are necessary and sufficient for the gains from grafting. The target’s margin accounts for where the effect occurs, while the delta’s token selection drives the subsequent improvement in accuracy. Our method is also efficient. Its vocabulary bridge takes one step per token and runs about four times faster than ByteSampler at similar accuracy (G1), so it can be applied to training-based tasks such as weak-to-strong on-policy distillation across model families (Feng et al., 2026, https://arxiv.org/abs/2607.05394).


Does guidance at high-entropy positions alone retain the gain (W2, Q2)?

Yes—see G3. When thresholds are chosen using the training questions, grafting only the top 55% of positions preserves the entire improvement (89.589.5% vs. 88.588.5%), whereas removing those positions reduces performance by 1919 to 3232 points. We plan to extend this experiment to HH-RLHF in the camera-ready.


Does the advantage over the donor persist when α is tuned on held-out data (Q4)?

We next choose α\alpha using 200 tasks from the GSM8K and MATH training splits (G6). For MATH-500, this procedure selects the paper’s α=1.5\alpha{=}1.5, leaving the result unchanged at 42.542.5%. For GSM8K, it selects 1.51.5 rather than 1.251.25, yielding 83.583.5% instead of 85.585.5%. As a result, on GSM8K the grafted target matches the donor.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal General Response 2 of 2

G2. Positioning clearly against prior work (6XWj, Pj2e, srEg).

We updated Table 1 and the related work.

  • Proxy tuning already reported that changes concentrate at early positions in the generation (AlpacaFarm). However, it is not obvious that the finding would translate to Math dataset. We reproduce it on HH-RLHF and Math dataset. Table 1 now marks this finding as reproduced, and we also update introduction to make it super clear.

G3. Do high-entropy positions carry the gain, and is the pattern specific to the delta? (Pj2e, srEg)

We added a new appendix section that tests this on GSM8K with the Qwen2.5-Math-1.5B delta as donor and Qwen2.5-Math-7B as the target model.

  • Grafting only at high-entropy positions keeps the full gain. We set entropy thresholds on training questions and graft only above them. Grafting at the top 55% of positions reaches 89.589.5%, against 88.588.5% for grafting everywhere. Grafting everywhere except those positions drops accuracy to 69.569.5%, 63.563.5%, and 56.556.5% for the top 55, 1010, and 2020% removed. These high entropy positions are necessary and sufficient for performance gains.

  • If we graft a random noise delta, it flips tokens at high entropy positions as srEg suspected. Adding a gaussian noise delta causes token changes within the same entropy quintiles. For math, 9999–100100% of the edits land in the top quintile. This is explained by the target’s margin: across quintiles, its median drops by roughly 1414 nats, whereas the delta’s push toward the runner-up increases by only about 22.

  • However, a random-noise delta fails to deliver comparable accuracy improvements Using noise for generation rather than the delta drops accuracy to 30.030.0–41.241.2%, which is even worse than the unguided 57.557.5%. In contrast, applying the post-training delta increases accuracy to 91.291.2% (n=80n{=}80). The delta selectively boosts the correct tokens that lead to downstream accuracy.

So the target’s uncertainty decides where grafting changes tokens, and the delta decides which tokens it picks. We revised the claims in the introduction and Section 4 to say this.


G4. Few-shot prompting and URIAL (6XWj).

We ran both these baselines.

  • Few-shot prompting. In mathematical reasoning tasks, few-shot prompting achieves performance comparable to grafting only in the Qwen2.5-Math-7B GSM8K setting (89.589.5% vs. 90.590.5%). In the remaining three settings, grafting outperforms few-shot prompting by approximately 1515 to 2929 percentage points. One plausible explanation is that this result reflects idiosyncrasies in the underlying Qwen model, which is known to be extensively trained on mathematical data and consequently exhibits strong baseline performance on such tasks with few shot prompting.

  • URIAL. For alignment, URIAL—essentially ICL adapted for alignment tasks—performs on par with grafting. However, ICL can also be paired with grafting. When we apply URIAL’s prompt together with grafting, it outperforms URIAL across all four alignment settings, winning on 57.557.5–84.584.5% of prompts and losing 8.58.5–2020%.


G5. Validating the vocabulary bridge (srEg).

  • Logits vs. log-probabilities. We include this in appendix and don’t see appreciable difference between doing grafting in log-prob space vs logit space.

  • Token synchronization. We make this explicit here. Once the target produces a token, we decode it into text, re-encode that text using the donor tokenizer, and pass the resulting tokens to the donor pair before the subsequent step. Thus, the donors always consume the text generated by the target. In 6.86.8% of steps, a single Llama token maps to multiple Qwen tokens. This is now explained in the appendix.


G6. Choosing α on the train set (Pj2e).

We now select α\alpha on 200200 training problems and report test accuracy at that α\alpha.


Summary of revisions.

Major changes.

  • New appendix, Entropy Analysis (entropy gating, matched noise, margin vs. push).

  • New appendix, Comparison with Baselines (ByteSampler, few-shot prompting, URIAL).

Minor changes.

  • New subsection on grafting in log-probability space, in the Ablation appendix.

  • New subsection on choosing α\alpha on training problems, in Hyperparameter Sensitivity.

  • Tokenizer Bridging now explains donor synchronization.

  • Revised introduction, Table 1, and related work, crediting proxy tuning, URIAL, and ByteSampler.

  • Limitations now discuss reasoning models.

AuthorsOpenReview ↗

Rebuttal General Response 1 of 2

Part 1. General Response to All Reviewers

We thank all reviewers for their feedback. We identified several recurring themes and address them here. We conducted four additional experiments and updated the manuscript accordingly. We also added two new sections to the Appendix: Comparison with baselines and Entropy Analysis. We start by discussing ByteSampler (https://arxiv.org/pdf/2506.14123), which Pj2e cited as prior evidence of transfer across tokenizers, since it relates to our primary new claim.


G1. Comparison with ByteSampler (Pj2e, srEg).

We thank reviewer Pj2e for highlighting this relevant work. ByteSampler operates at the byte level. Although ByteSampler is intended as a general framework for byte-level sampling from LLMs, with proxy-tuning being only one of its possible uses, we think the cross-tokenizer proxy-tuning experiment reported in their paper may be somewhat problematic because of the chosen donor–target pairing (OLMo-2 to Llama-3), and therefore does not convincingly demonstrate transferability across tokenizers. We intend to reach out to the authors for clarification.

  • Cross-tokenizer experiment in ByteSampler paper may by flawed. We compared the two tokenizers (OLMo-2-1B, Llama-3.1-8B) and found that all 100,256100{,}256 standard OLMo-2 tokens are also present in the Llama-3 vocabulary. This is a bit surprising, since bytesampler is intended to demonstrate cross-family transfer under vocabulary mismatch. Given that the paper was updated on arXiv in May 2026, we suspect the authors might still be working on this. We intend to contact the authors to better understand their rationale behind choosing this pair. By contrast, in our Qwen-to-Llama pairing, only 85.485.4% of Llama-3’s vocabulary overlaps with Qwen2.5-Math’s, which correctly demonstrates the cross-family transfer property.

  • ByteSampler is limited to attention-only architectures. The public ByteSampler implementation scores multiple byte branches within a single sequence and distinguishes them using an attention mask. However, convolutional layers are unable to leverage this mask, meaning the code cannot be used with LFM2 (Amini et al., 2025, https://arxiv.org/abs/2511.23404), the convolution-based donor model from which we transfer. In contrast, our vocabulary bridge can work across different architectures.

  • Grafting matches ByteSampler’s accuracy at about a quarter of its cost. We ran ByteSampler on its own OLMo-to-Llama pair and on our Qwen-to-Llama pair, with both methods at grafting’s best α\alpha (table below). ByteSampler leads by at most four points, but it is 4.04.0 to 4.54.5 times slower per token, because it takes one model step per byte while grafting takes one per token. Grafting is also simple. Its vocabulary bridge is a lookup table built once from the two vocabularies, and decoding then costs one forward pass per model per token.

  • The speedup is useful in practical applications such as during training. Weak-to-strong on-policy distillation (Feng et al., 2026, https://arxiv.org/abs/2607.05394) method takes the log-ratio between a small post-trained model and its base as a dense reward to train a larger student model on its own rollouts. In that setup, the delta has to be evaluated for every token in every rollout. Our vocabulary bridge would allow the small-model pair and the student to come from different model families, without any byte-level overhead and could be 4$\times$ faster than ByteSampler.

DonorTargetDataGraftingByteSamplerSlowdown
OLMo-2-1BLlama-3.1-8BGSM8K67.067.0%67.567.5%4.5×4.5\times
Qwen2.5-Math-1.5BLlama-3-8B-Inst.GSM8K85.085.0%87.587.5%4.1×4.1\times
Qwen2.5-Math-1.5BLlama-3-8B-Inst.MATH-50042.542.5%46.546.5%4.0×4.0\times

Slowdown is ByteSampler’s time per token divided by grafting’s. Both methods use α=1\alpha=1. The OLMo row uses five in-context examples, as in the ByteSampler paper.

We added a sentence in the introduction about ByteSampler and included a new related-work paragraph, along with a new section that compares against ByteSampler. That said, given the points discussed above—ByteSampler has similar performance, is four times slower, and does not generalize across architectures—there seems to be little justification for choosing it over grafting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.