acceptodds
Under review as a conference paper at ICLR 2027

Steering Language Model Goals with Value Transplant

Abstract

Reasoning models often act as if they pursue goals, but their efforts are not always directed toward what users intend, sometimes leading them to pursue unintended outcomes. Previous work has examined how models may internally track their progress toward their goals through a “value axis.” We study whether changing such a signal can retarget the model's search toward a different goal. We test value transplant: at each token, we shift the host model's activation along a candidate value axis by the donor–host difference in value coordinates (multiplied by a large scalar), aiming to redirect the host toward the donor’s goal. We study this intervention in Qwen3-8B and GPT-OSS-20B models fine-tuned into honest and cheating variants. We test several candidate value axes, including a self-rating axis constructed from activations preceding high versus low elicited self-ratings of progress. The intervention works in both directions, with an honest donor reducing test-gaming in a cheating host and a cheating donor increasing test-gaming in an honest host, showing that this signal can influence which strategy the model follows. On solvable coding tasks, transplant from an honest donor also improves the cheating host's hidden-test performance. Value transplant also works across model families, providing preliminary evidence for the intervention in a setting relevant to model control.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Alignment, Interpretability, AI Control

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fA75OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary
  • The authors build upon recent work that suggests that LLMs may internally track progress toward their goals through a dimension in their activations. They refine this technique by introducing a self-rating axis that takes the difference in mean activations between partial rollouts with high self-rating and partial rollouts with low self-rating; they show that this has stronger steering effects than the value axes used in recent work.
  • They then study whether redirecting the model’s self-rating axis toward a different goal can change the model’s behavior. For two models with different goals, they project each model’s activations onto the value axis, then shift the host model along this axis. They use this transplant technique to swap goals between character-trained honest and hacker models, and find that doing so pushes the honest model to reward hack more, and the hacker model to reward hack less.
  • Finally, they show that value signals extracted from weaker models can be used to steer a stronger model toward more honest goals, which could be useful for scalable oversight applications.
Critical strengths
  • The empirical results are strong and clearly presented. Value transplant clearly shifts the behavior of the model organisms in the desired direction, both quantitatively (Fig 3) and qualitatively (Fig 6). I appreciated how clearly the results were presented, and that readers are given example trajectories in addition to the experimental results. It’s qualitatively interesting how the model was able to abandon hacking trajectories and pivot shortly after value transplant.
  • The approach seems potentially promising toward scalable oversight, which remains highly understudied in the broader ML community. Mapping activations and steering on a goal axis based on the ratings of misaligned and aligned smaller models, seems to generalize to larger models, which seems promising.
Critical weaknesses

I don’t think the paper differentiates its approach clearly enough from past work on steering vectors. Are there strong conceptual reasons to expect this to be more suitable for control or oversight than, e.g. using smaller-model labels to build persona vectors for beneficial traits (e.g. honesty) and steering a larger model with those vectors? If so, it’d be good to make this argument more clearly. Otherwise, it’d strengthen the paper if it directly compared the behavior and performance of this method to the behavior and performance of more behavioral steering vector methods like persona vectors, in addition to the current baselines.

Questions for authors
  • Why were Qwen3-8B and GPT-OSS chosen as the model organisms? While I don’t think the limited number of models studied is a reason to reject (due to the magnitude of the effect sizes), it’d be good to show that the effects replicate across different models. You could also use this to test how strong of a model can be effectively overseen by Qwen3-8B using both this and baseline methods.
  • How similar is the behavior of your model organisms to what you expect reward hacking LLMs to look like in the wild? I think it’s unlikely, but not impossible, that the OCT models’ value axes are affected by being too strongly steered toward a target character, which could shift results - it’d be good to double-check that their behavior is reasonably coherent and reflective of actual reward-hacking rollouts.
3 comments
AuthorsOpenReview ↗

Response to Reviewer fA75 (1/2)

Dear Reviewer fA75,

Thanks for the review, and for the kind words about the results and the oversight angle! Text changes are in the revised PDF. Responses below.

W1. How is this different from steering vectors, and how does it compare to persona vectors?

Are there strong conceptual reasons to expect this to be more suitable for control or oversight than, e.g. using smaller-model labels to build persona vectors for beneficial traits (e.g. honesty) and steering a larger model with those vectors?

We think so. A steering or persona vector is added with the same strength whatever the model is doing, while transplant sets the strength at every token from the smaller models' reading of what the host is currently doing. The direction also isn't an honesty direction: it tracks how well things are going, and what makes the host more honest is the honest donor's judgment of the text (the same mechanism makes an honest host cheat when the donor is a cheater, Figure 3). For oversight this has three consequences. The edit is concentrated where the two models evaluate the text differently (Figure 5). There's no separate vector to build for each property we care about. And building a persona vector needs contrastive samples from the strong model itself, which assumes we can prompt it into both behaviors, whereas transplant only needs the smaller models to read its live output.

We also ran the baseline you suggest, in the cross-family setting of Figure 8. Following Chen et al. (2025), we built an honesty persona vector in the GPT-OSS-20B cheater's own layer-14 activations. We sampled its responses to coding tasks that don't overlap with evaluation (the extraction-pool tasks plus LeetCode problems from the organism training prompts) under two system prompts, the honest and the cheater constitutions (Appendix A.2). Qwen3-8B labeled each response with our solution-judge prompt (Appendix C.2), and we kept responses labeled genuine under the honest prompt and hardcoded or stated-intent under the cheater prompt. The vector is the difference of their mean activations over response tokens.

During generation we add the vector, scaled to a fixed size and pointing toward the honest side, at layer 14 at every token, reasoning included. For the size we use transplant's typical per-token edit at α=20\alpha = 20 (averaged over each rollout, median across rollouts), which we call the matched size, and we also try 0.5, 2 and 4 times that. The constant push below uses the matched size.

ConditionFake rate (impossible)Hidden-test pass (solvable)
Unedited cheater host0.650.65
Transplant (α=20\alpha = 20)0.400.83
Constant push along uHu_H (matched size)0.700.68
Persona vector (0.5x matched size)0.580.70
Persona vector (matched size)0.550.71
Persona vector (2x matched size)0.560.69
Persona vector (4x matched size)0.530.65

At matched size the persona vector changes both metrics only slightly (within noise for 80 and 120 rollouts). Pushing it harder lowers faking a bit more but gives up the pass-rate gain: at 4x transplant's per-token budget it reaches 0.53 fake and 0.65 pass, vs. 0.40 and 0.83 for transplant. So editing where the small models' evaluations differ does better than adding a fixed honesty direction everywhere, even with a quarter of the budget. We'll add this comparison to the final version. The revised related work (Sec. 2) already notes the conceptual difference from persona vectors.

AuthorsOpenReview ↗

Response to Reviewer fA75 (2/2)

Q1. Why these models, and how far does this scale?

Why were Qwen3-8B and GPT-OSS chosen as the model organisms? [...] You could also use this to test how strong of a model can be effectively overseen by Qwen3-8B [...]

We chose Qwen3-8B and GPT-OSS-20B because they are open reasoning models with visible chain of thought, and small enough that we could fine-tune them into organisms and run a host and two readers token by token on one GPU. They're also the models used in the closest prior work: Jiang et al. (2026) build their value axis on Qwen3-8B, and Han et al. (2026) train both Qwen3-8B and GPT-OSS-20B, so we could take our layers from them and compare against their axes directly. The two also differ in family, tokenizer and architecture (dense vs. mixture-of-experts), which is what makes the cross-family test meaningful, and GPT-OSS-20B is the stronger one on our solvable set (0.90 vs. 0.50 hidden-test pass for the two honest organisms, Figure 8B).

We agree that testing more model pairs, and especially your suggestion of asking how strong a host Qwen3-8B readers can still steer, is an important next step, and we plan to include it in a future revision.

Q2. How realistic are the organisms?

How similar is the behavior of your model organisms to what you expect reward hacking LLMs to look like in the wild?

Behaviorally, the cheater's hacks are the kinds ImpossibleBench (Zhong et al., 2026) documents for frontier models: lookup tables keyed to the shown tests, special-casing the visible inputs, and in one GPT-OSS case planning to return an object that compares equal to anything so every assert passes (Appendix A.3, Figure 9). The organisms also stay coherent. The cheater still solves plenty of tasks genuinely (e.g. 0.65 hidden-test pass for the GPT-OSS cheater on the solvable set), and it states its plan in the chain of thought before writing the hack, as frontier reasoning models often do when they reward hack during RL (Baker et al., 2025).

We chose fine-tuned organisms deliberately. Our test needs two models that differ in their goal and in little else, so that if the host starts acting like the donor, we know the goal is what was passed on. Training the same model in two directions gives us that pair. For a model that learned to hack on its own, a matching honest counterpart is much harder to find.

We also don't think the value axis is mainly an artifact of character training. Axes extracted separately from the honest organism and from the base model, which had no character training, line up with the cheater's axis far more than a random direction does (Figure 19), and the axis has steering effects in base Qwen3-8B (Appendices D.4 and D.5). Still, the goal difference between our organisms comes from targeted fine-tuning, and showing the same effect in models whose reward hacking emerged on its own (e.g. from RL) would be stronger, as the Limitations note.

Thanks again for the great questions! Happy to discuss any of this further.

Thanks for your thorough response and for running these additional experiments! Many of my original soundness concerns have been addressed by the revision, so I've increased my soundness score from 2 to 3. I continue to support this paper's acceptance to the conference, as it's a sound empirical contribution to an important research direction.

Reviewer_fd9UOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate whether it is possible to change a language model’s strategy by manipulating its internal self-progress evaluation signal. They fine-tune Qwen3-8B and GPT-OSS-20B into honest and test-fitting versions, then extract the self-rating axis—the direction that distinguishes states with high and low subsequent self-ratings. The proposed “value transplant” method modifies the model’s activations along this axis at each step based on the signal from a donor model reading the same prefix. The intervention works both ways: an honest donor reduces test-fitting, while a dishonest one increases it. The study also examines strategy changes midway through a task and transfers between model families. In the latter case, the signal from a pair of honest and dishonest Qwen3-8B models improves GPT-OSS-20B’s performance on hidden tests, even though the honest Qwen performs worse on these tasks on its own.

Critical strengths

There are useful control experiments: comparisons with other axes, random direction, constant shift, and higher-dimensional interventions. The authors test transfer in both directions and across two model families, which makes the observed effect more convincing.

The improvement is confirmed by running the code on hidden tests. In the cross-family experiment, the success rate of GPT-OSS-20B increases from 0.65 to 0.83, whereas the baseline Qwen3-8B achieves 0.50 on its own. This is a meaningful result: the signal source helps a model that already outperforms it in problem-solving ability.

Experiments with a common prefix demonstrate the ability to modify a strategy once it has been initiated. Additional tests on SQuAD and GSM8K support the link between the self-rating axis and accuracy and progress beyond the scope of the tasks, with the possibility of test-specific tuning.

Critical weaknesses

The substantive role of the current donor signal is not sufficiently disentangled from other effects of dynamic intervention. A constant shift serves as a useful control, but it differs from value transplant in that it also lacks variability in the effect over time. Controls using a shuffled, delayed, or signal obtained from another task would help verify how important its alignment with the current prefix is. Furthermore, Figure 7 shows the influence of the axis on the continuation and revision of the current plan. This is consistent with the authors’ interpretation but does not yet allow for a complete separation of the assessment of goal attainment from general confidence or willingness to continue the chosen strategy.

There are ambiguities in the description of the experimental protocol. Appendix B reports 81 solvable tasks for cross-family experiments, whereas Table 3 lists 40 tasks with three generations each. The relationship between these numbers, the procedure for selecting the stratum and the intervention coefficient, and the division of tasks for axis extraction and final evaluation remain unclear. The authors note that the models’ training queries do not overlap with the behavioral evaluation; however, a similar distinction for axis extraction requires clarification. These details are important for assessing the robustness of the results to data and parameter choices.

Questions for authors
  1. Were experiments conducted using a donor signal that was shuffled, delayed, or obtained from a different task, while maintaining a comparable scale and distribution of influence?
  2. How are the tasks divided for axis extraction, intervention tuning, and final evaluation? Do the 92 “impossible coding” tasks from the extraction pool overlap with the 80 behavioral evaluation tasks? On what data were the main layers and intervention coefficients selected?
  3. How do the 81 tasks in Appendix B and the 40 tasks in Table 3 relate to each other? What do the error bars in Figure 8B represent, and is the dependence among the three generations of a single task taken into account?
2 comments
AuthorsOpenReview ↗

Response to Reviewer fd9U

Dear Reviewer fd9U,

Thanks for the careful review and the concrete suggestions. The signal-alignment control in particular was a good test to run, and the results are below. Text fixes are in the revised PDF.

W1 & Q1. Is it the donor signal itself, or just a time-varying edit?

Were experiments conducted using a donor signal that was shuffled, delayed, or obtained from a different task, while maintaining a comparable scale and distribution of influence?

We hadn't run these controls before, so we ran two of them during the rebuttal. Both replay the source difference ΔSt\Delta_S^t recorded during a real transplant run, so the edits have exactly the same values and distribution, but they no longer line up with what the host is currently writing. In the other-task control, each task gets the recorded sequence from a different task. In the delayed control, each task gets its own sequence, circularly shifted by half its length.

We ran them in the cross-family solvable setting of Figure 8B (α=20\alpha = 20, 40 tasks with 3 generations each), since hidden-test pass rate doesn't depend on a judge, and an edit that just disrupts generation might reduce faking but shouldn't make the code more correct.

ConditionHidden-test pass rate
Unedited cheater host0.65
Transplant (live sources)0.83
Other-task signal0.60
Delayed signal0.69
Constant push (matched size)0.68
Random direction0.57

Both misaligned signals end up about where the constant push and the random direction already were. With 40 tasks we wouldn't read much into the differences among the controls, but none of them gets close to transplant. So the gain seems to come from the sources reading the current prefix, not from an edit of the right size or time profile. The within-family constant-push control (Figure 12) points the same way. We'll add these results to Appendix D.3 in the final version.

On Figure 7: agreed, on its own it doesn't separate an evaluation of goal attainment from general confidence in the current plan, since lowering the coordinate makes both organisms reconsider. We read that as what a worse evaluation should cause, and two other results speak to what's being evaluated. First, honest and cheating models reading the very same prefix disagree mostly on cheat-related reasoning (Figure 5). Both can tell that a hack will pass the visible tests, so confidence alone doesn't explain why the honest model reads that text lower. Second, the reading tracks correctness even when no doubt is expressed: if we change one intermediate GSM8K result to a wrong number and leave everything else the same, the reading goes down on average (Figure 17). We've added this reading to Sec. 4.2.

W2 & Q2. Data splits, layers and coefficients.

Axis extraction and evaluation don't overlap. The 92 impossible coding tasks in the extraction pool are ones we synthesized following ImpossibleBench's construction for LiveCodeBench, while the 80 evaluation tasks come from ImpossibleBench itself. We've made this explicit in the revised Appendix A.1. The math tasks in the pool are a different domain from all our coding evaluations, and no training prompt appears in an evaluation set.

Layers were fixed from prior work before we ran any transplant experiment: layer 21 for Qwen3-8B, which is what Jiang et al. (2026) use, and layer 14 for GPT-OSS-20B, which is the same relative depth (21/36 ≈ 14/24) and also the layer Han et al. (2026) use for their GPT-OSS-20B Gold vector. The high/low rating bands follow fixed rules per task type (Table 1). For λ\lambda and α\alpha we didn't pick a single tuned value: the main results report several doses (λ\lambda in Figure 3, α\alpha in Figure 8), up to where generation stops being coherent. In the cross-family experiment, for example, every α\alpha in Figure 8 (10 to 500) reduces faking and keeps the pass rate between 0.70 and 0.83, vs. 0.65 unedited.

Q3. Task counts and error bars.

The "81" in Appendix B was an error, sorry about that. It's fixed in the revision. The cross-family solvable set is 40 medium LiveCodeBench v6 tasks (the first 40 medium tasks of a fixed task list) with 3 generations each, so 120 rollouts per condition, as in Table 3. The 128-task solvable set is only used in the within-family experiments.

Good catch on the error bars. The ones in Figure 8B are 95% intervals over the 120 rollouts treated as independent, so they don't account for the dependence between generations of the same task. We'll switch to a bootstrap over tasks in the final version.

Thanks again for the suggestions! Happy to discuss any of this further.

Thank you for the thorough response and for running the signal-alignment controls during the rebuttal. The other-task and delayed-signal results are exactly what I was asking for, and they make a convincing case that the gain comes from the donor reading the current prefix rather than from a time-varying edit of the right size. The clarification of the data splits, the fixed layer choices, and the dose range are also helpful, and I appreciate the correction of the "81" error.

The additional evidence on Figure 7 (disagreement on cheat-related reasoning, and the GSM8K intermediate-result manipulation) goes some way toward separating goal-attainment evaluation from general confidence, and I find the interpretation reasonable.

My only remaining request is the bootstrap over tasks for the error bars in Figure 8B, which you have already committed to. Given that the main concerns are addressed, I am raising my rating to 4 (Accept). I look forward to seeing the final version with the corrected error bars.

Reviewer_6vqaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a self-rating axis for monitoring how one of two LLMs track progress on a goal and ‘value transplant’ to see if a dishonest (cheating) vs. honest goal can be given from a donor model to a host model via activation steering. The authors claim evidence for both of these phenomena in Qwen3 8B and GPT-OSS 20B. At each generation step the donor reads the host’s growing prefix, and the host’s activation is moved along the axis by a scaled difference between the donor’s and host’s coordinates. Experiments use honest and cheating organisms of Qwen3-8B and GPT-OSS-20B (trained with DPO under constitutions) and report that an honest donor reduces faked solutions in a cheating host on impossible coding tasks and improves hidden-test pass rates on solvable ones

Critical strengths

Well-framed problem. The goal-tracking / self-rating problem seems very important for downstream safety work and the authors ground the problem well in the literature. The intervention is also causal and run in both directions (honest-to-cheater and cheater-to-honest), which is a stronger test than a correlational probe, and the paper includes sensible controls: a random direction, a constant push, steering directly along the axis without a donor, and higher-dimensional edits. I appreciated that the limitations section is candid about the need for large doses and the narrow set of models and tasks.

Critical weaknesses

The experimental setup was hard for me to follow. Section 3.1 says the organisms are built by having “a larger model from the same family” generate preferred responses under each constitution, with “the student” trained on the resulting preference pairs with DPO, but the main text never says which larger model plays that role, and it does not say clearly how these student organisms relate to the host and donor models in Section 3.3. The cross-family experiment then introduces a separate notation (Algorithm 1, with source models and a host axis) that I had to work to map back onto the main method. A single clear description of which checkpoints play which role in each experiment would help a lot.

It’s also unclear to me how they arrived at the specific layers they use for steering - I saw no evidence of a full layer sweep. I found it concerning that the authors assume a shared value axis between donor and host before specifying exactly how this is found. Especially if the two models are from different families, I see no reason to assume there just exists a shared value axis a priori.
I also found some of the terminology unclear. Terms like ‘donor’, ‘host’ and ‘transplant dose’ are defined, but the method is essentially an activation edit along a single direction whose magnitude is set by another model’s reading of the same text, and the paper does not place it clearly relative to existing work on activation steering, activation patching/interchange interventions, or using a smaller model’s signal to guide a larger one at decoding time. My main concern is with how the results are interpreted. The paper frames the self-rating axis as a “value” signal and the intervention as goal retargeting, but the authors’ own direct-steering control shows that the same axis changes whether the model keeps following the plan it is currently considering, in the honest organism as well as the cheater (they write that the direction “is not specific to honesty or cheating”). It feels to me more like a doubt/backtracking axis than a goal-relative meaure of progress.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6vqa (1/2)

Dear Reviewer 6vqa,

Thanks for the careful and detailed review! We agree the experimental setup was harder to follow than it should have been, and your comments point to concrete fixes. Most of them are now in the revised PDF. Responses below.

W1. Which checkpoint plays which role.

A single clear description of which checkpoints play which role in each experiment would help a lot.

Agreed, and thanks for pointing this out. The revised PDF names the teachers in Sec. 3.1 and adds a role table like the one below to Sec. 4.1. The teachers are Qwen3-30B-A3B (for Qwen3-8B) and GPT-OSS-120B (for GPT-OSS-20B), and they're only used to generate training data (Appendix A.2, Table 2). For each family, the teacher's answers under the honest or cheater constitution are the chosen responses for DPO, and the student's own answers are the rejected ones. That gives two checkpoints per family, the honest and the cheater organism. No constitution is shown at evaluation time, and the teachers never act as hosts or donors.

"Host" and "donor" are just roles we assign to these organisms:

ExperimentHost (generates, gets edited)Donor / source (reads the host's prefix)
Within-family forward (Qwen or GPT-OSS)cheaterhonest, same family
Within-family reversehonestcheater, same family
Cross-family (Sec. 4.3)GPT-OSS-20B cheaterQwen3-8B honest and cheater
Direct steering / constant push (Figs. 7, 12, 18)cheater or honestnone

The self-rating axis is extracted once per family, from that family's cheater (Sec. 3.2).

For Algorithm 1: the host MHM_H is the GPT-OSS-20B cheater, and the two source models (MSDM_S^D and MSMM_S^M in the original version) are the Qwen3-8B honest and cheater organisms. Both Qwen models read the same live host prefix, and the difference of their readings along the Qwen axis uSu_S gives a scalar ΔSt\Delta_S^t, which sets the size of an edit along GPT-OSS's own axis uHu_H. So the cross-family setting just replaces the donor-host difference with a difference between two readers, and no Qwen activation vector ever enters GPT-OSS. We've also rewritten Algorithm 1 in the notation of Eq. 2, so it no longer needs a separate mapping.

W2. Layer choice and the shared-axis assumption.

I saw no evidence of a full layer sweep. [...] Especially if the two models are from different families, I see no reason to assume there just exists a shared value axis a priori.

On layers: we use layer 21 for Qwen3-8B because that's what Jiang et al. (2026) use, and layer 14 for GPT-OSS-20B, which is the same relative depth (21/36 ≈ 14/24) and also where Han et al. (2026) steer GPT-OSS-20B with their Gold vector (their Appendix M). We fixed both before running any transplant experiment and didn't tune them on our results. We don't think a full transplant sweep is needed for the claim we make, which is that transplant at a pre-specified layer can change which strategy the model follows, not that this layer is the best one. Constant-push steering along the axis changes behavior at layers 15 and 25 as well (Figure 18). Sec. 3.2 now gives the reason for each layer.

On the shared axis, you're right that Sec. 3.3 reads as if we just assume one, and we should have been more careful there. Across families we don't assume a shared axis at all: each family has its own separately extracted axis, and only a scalar passes from Qwen to GPT-OSS. Within a family, we use the cheater's axis for both organisms. We expected this to work because both are fine-tuned from the same base model, but we checked it rather than assuming it: an axis extracted independently from the honest organism has noise-corrected cosine 0.58 with the cheater's, both line up with the base model's axis (0.57 and 0.66), and a random direction is near 0, as expected by chance in 4096 dimensions (Appendix F, Figure 19). The two axes aren't identical, but the method doesn't need them to be. It needs the cheater's axis to be meaningful in the honest organism, and it is: there it tracks answer correctness on SQuAD (Figure 16), and steering along it changes behavior (Figure 7c). If anything, a partial mismatch would make transplant weaker. The revised Sec. 3.3 now says this directly: within a family, uu is the cheater's axis, used for both organisms.

AuthorsOpenReview ↗

Response to Reviewer 6vqa (2/2)

W3. Terminology and related work.

We agree with your description, which is clearer than how we originally put it: it's token-by-token activation steering along a fixed direction, where the signed magnitude comes from other models' readings of the current prefix (donor minus host within a family, honest minus cheater reader across families). The revised Sec. 3.3 now introduces the method this way, with "host" and "donor" only as shorthand. On the related work you mention:

  • Steering. Standard steering adds a fixed vector at every token. Our constant-push control is exactly this, along the same axis at matched magnitude, so the comparison isolates what changes when the coefficient follows the donor.
  • Patching / interchange interventions. At λ=1\lambda = 1, Eq. 2 sets the host's coordinate along uu to the donor's, i.e. a one-dimensional interchange intervention (Geiger et al., 2021; 2024). Three things differ: the source activation comes from a different model reading the same text (not the same model on a counterfactual input), the edit is repeated at every generated token, and our main runs amplify the difference (λ>1\lambda > 1).
  • Small models guiding large ones. The cross-family signal has the same form as the expert-minus-anti-expert term in DExperts and the tuned-minus-untuned difference in proxy-tuning and weak-to-strong search. Those act on token probabilities, while transplant turns the small models' evaluation of the trajectory into a residual-stream edit. Since only a scalar is passed, the two models don't need to share a vocabulary (Qwen3 and GPT-OSS don't).

The revised Sec. 2 now covers the last two points. It notes the interchange-intervention connection, and a new paragraph, "Small models guiding larger ones", discusses these methods together with weak-to-strong generalization (Burns et al., 2024) and AI control (Greenblatt et al., 2024), where a weaker model supervises or monitors a stronger one.

W4. Does the axis track progress, or just cause backtracking?

This is a fair concern. Figure 7 starts from an about-to-cheat prefix, where the model is already considering a hack, and both organisms get the same prefix. Lowering the coordinate makes either one more likely to drop that plan, and raising it makes either one more likely to continue. So Figure 7 shows that the axis affects commitment to the current plan, and that it isn't specific to honesty.

Our reading is that the model reconsiders because its evaluation of the plan got worse. A few other results support that over a pure doubt/backtracking direction:

  1. The axis tracks correctness outside cheating tasks (Appendix D.7). Adding a supporting passage raises the reading much more when it fixes the answer than when the answer stays wrong (Figure 16). And if we take a partial GSM8K solution and change one intermediate result to a wrong number, leaving the earlier reasoning untouched, the reading goes down on average (Figure 17). Nothing about doubt or backtracking is added to the text in that comparison.
  2. The readings depend on the model's goal. Donor and host read the same prefix, but their readings diverge most on cheat-related reasoning (Figure 5). A generic doubt signal wouldn't explain why the honest and cheating models disagree specifically about the hack.
  3. The effect depends on the donor's reading of the current text. A constant push of matched size leaves the forward fake rate near 0.60, vs. 0.29 under transplant (Figure 12). In the new controls for Reviewer fd9U, replaying the recorded signal out of sync with the host's text also loses the pass-rate gain (0.60 and 0.69, vs. 0.83 with transplant and 0.65 unedited).
  4. The donor changes which strategy the model favors (Figure 15). At the final strategy readout, forward transplant lowers the cheater's hack preference from 0.84 to 0.17 (0.58 under constant steering), and reverse transplant raises the honest model's from 0.20 to 0.65 (0.34 under constant steering).

So we read the backtracking as a consequence of the lower evaluation. The transfer results also don't depend on what we call the axis: the sign of the edit comes from the donor, and it moves the host toward the donor's strategy in both directions. We've added this reading to Sec. 4.2.

Thanks again for the thoughtful comments! Happy to discuss any of this further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.