acceptodds
Under review as a conference paper at ICLR 2027

How to Train Your Model Organism

17 views

Abstract

Model organisms of alignment-relevant behaviors (e.g., backdoors, sycophancy, spurious correlations) have emerged as a key tool for evaluating whitebox interpretability techniques. We argue that the prevailing practice of training model organisms to a single objective of installing the target behavior is insufficient and propose validating model organisms with respect to three objectives with associated metrics: target-behavior installation, general-capability preservation (i.e., parametric knowledge, chat quality), and output naturalness (i.e., CoT and activations). We re-visit two publicly released organism suites using this validation framework and show that (1) chat quality and CoT naturalness degrade substantially across training recipes, and (2) validation metrics predict how well interpretability methods recover the installed behavior, e.g., a logit lens readout covaries with an organism's general capabilities. We introduce a multi-objective training approach based on model merging to train more realistic model organisms. Finally, on a new suite of model organisms targeting demographic biases in clinical reasoning, we compare training recipes and find that DPO training stays closer to the base model than supervised finetuning, and the proposed model optimization approach better preserves capabilities and naturalness. Auditing this suite with an investigator agent, we again observe validation metrics tracking bias recovery. In sum, training methods shape the interpretability conclusions an organism supports, and we argue that one should consider multiple objectives to draw generalizable conclusions about interpretability methods using (realistic) model organisms.

Primary area:
interpretability and explainable AI
Keywords:
Model Organisms, Mechanistic Interpretability, Alignment Auditing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dRRAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that model organisms (models trained to have a specific behavior) are more realistic when they stay close to their base model. It measures this through the similarity in task accuracy, CoT traces, and detection of activation artifacts. It evaluates two existing organism suites alongside new organisms with demographic biases in medical QA. The results show that training choices affect both organism quality and effectiveness of interpretability tools. DPO-based training generally preserves capabilities better than SFT, and merging toward the base model can further improve preservation.

Critical strengths

1. Useful and important guidance for model organisms. Realistic model organisms are important for evaluating auditing and interpretability methods. The authors make a convincing case that good general capabilities are an important part of realism. They also provide a useful validation framework and practical training recommendations for achieving this while keeping the intended behavior.

2. Good experiments on existing and new model organisms. Revisiting two existing model organisms clearly shows that training choices and general capabilities of the organisms substantially affect the effectiveness of interpretability tools. The new clinical organisms extend the evaluation to medical QA.

Critical weaknesses

1. Measuring closeness to the base model as a proxy for realism may bias the results. While I agree that general capabilities and chat quality are important for model organisms, the paper assesses realism through similarity to the base model. However, (1) a change in writing style like in the chat-mixing results (Sec. 5.2) does not make the organism less realistic, (2) DPO training and model merging may just be better suited to staying similar to the base model, (3) the validation metrics correlate with behavior recovery because it may be easier to find the behavior when the model has only small differences to the base model. So, the given training recommendations may be biased by the choice of measurement, while they are not establishing naturally occurring misalignment.

Questions for authors
  1. Why should closeness to the base model establish naturally occurring misalignment or a representative organism for deployed models? For me, it just ensures that capabilities are retained and not that the target behavior is installed realistically.
  2. I did not fully understand yet what "validation metrics predict how well interpretability methods recover the installed behavior" means. Which validation metrics do you mean and is this true for all interpretability methods tested?
  3. Does changing the training recipe and staying closer to the base model change which tools are most useful for the auditor model?
  4. What does the new clinical organism suite provide for the main contribution given that other existing model organisms could also be used for that?
  5. Do you think the results would generalize to larger and newer models than what was tested? Especially reasoning models with explicitly trained CoT traces or models trained via RL?
5 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the feedback, and provide responses and/or clarifications to their questions below:

1. Measuring realism with respect to the base model (Q1)

This is a good question about our motivation, and we should have been more explicit about what we mean by realism. Our measurements establish necessary though not sufficient conditions for realistic model organisms, i.e., the representativeness of a deployed model. If a model organism has terrible performance on MMLU and has trouble responding to basic user queries, it cannot be a good “model” of any kind of generally capable LLMs which might actually be used in practice. We don’t have access to a “ground truth” model with the target behavior (if we did, we would not need to train an organism in the first place). But it seems likely that on general corpus, such a “ground truth” model should maintain similar outputs as the base model (CoT naturalness) and should not feature behavior-relevant activation signatures (activation naturalness). Subsequently we require the same for a model organism to be considered “realistic”.

On the specific points mentioned in the weaknesses:

  1. We agree with the reviewer that a change in writing style due to chat mixing (Section 5.2) does not directly prove that the organism is unnatural, but as the original goal for chat mixing in prior work is to mitigate activation-level artifacts, we think such collateral effect should be explicitly measured and, if possible, controlled.
  2. The reviewer is correct that a positive correlation may be explained by the behavior being easier to detect when the organism stays close to the base model. This is the intended evidence: different kinds of organisms yield different interpretability outcomes, and results on a representative organism are more plausibly transferable than results on a broadly damaged one.

In light of the reviewer’s comments, we have edited the paper to better reflect our assumptions and motivations in the beginning of Section 2 and 2.3.

2. What we mean by the correlation (Q2)

In this work, we argue that training model organisms not only requires a successful installation of target behavior, but also sufficient general capabilities and output naturalness (as measured via our metrics). Since the goal for these MOs is to evaluate interpretability tools, we want to show these metrics have a quantifiable relationship with interpretability results.

We demonstrate the correlations across different organism suites, interpretability tools, and validation metrics. A consistent signal across suites is logit lens vs. capability metrics (MMLU, MT-bench) in Table 21 and Table 27, and we provide qualitative examples in Appendix M illustrating how logit lens on a more “natural” organism surfaces more behavior-relevant tokens. Figure 4/11 and Table 15 show the correlations across tools like CoTs (blackbox) and J-lens on our clinical organisms.

More broadly, we do not claim that better validation metrics universally lead to better (or worse) results regardless of target behavior and interpretability tools, as different behaviors and tools likely have different dynamics in response to these measures. What we mean by the recurring correlations is that organisms of the same target behavior can result in different measurements of interpretability methods, particularly as a function of these validation metrics. Therefore, an MO’s representativeness matters for the downstream conclusions about interpretability methods which these organisms are meant to support.

AuthorsOpenReview ↗

Response 2/2

3. Relative rankings of tools (Q3)

In the experiments on Pando organisms (Section 4), we directly retrain the 80 original organisms with DPO while holding the underlying target behavior fixed. Then, we measure whether the change in a tool’s rule recovery scores correlate with the change in validation metrics. We found a significant correlation on logit lens (Table 21, qualitative examples in Appendix J.1): the closer an organism stays to the base model, the better the auditing results for logit lens are. On Pando, this does not directly translate to a higher rank of logit lens after retraining on average. Appendix K.5 shows the run-to-run stochasticity in Pando’s LLM auditing pipeline, where runs with the same auditor (gpt-5.1) on the same model organism yield vastly different rule-recovery accuracies (0.47-0.87). Such noise level exceeds the possible gains from retraining, and we agree that a direct rank change would have been more convincing if we can obtain stable estimates of auditing results.

In our newly constructed clinical organism suite, changing the training recipe and subsequently the MO’s realism can change which tool is most helpful. On the race bias, we partition each organism by its training objective (SFT or DPO). Note that DPO organisms yield higher medical QA accuracies on average (51.4% vs. 44.9%), i.e. closer to the base model. Honesty steering is the least effective of the four auditing tools among SFT organisms, while it becomes the most effective on DPO ones. We perform bootstrapping over SFT/DPO organisms, resampling with replacement 10,000 times and computing the averaged per-organism rank of honesty steering – the rank improves by 0.74 from SFT to DPO (95% CI [+0.05, +1.41]). Therefore, organisms more representative of the base model may produce a different set of results.

4. Contributions for the clinical organisms (Q4)

Prior model-organism work often feature behaviors that are themselves unrealistic or stylized – for example organisms are trained to reward hack on fake documents favoring chocolates in pizza recipes [1]. These gaming setups are a reasonable starting point, but for the purpose of rigorously evaluating interpretability tools, realistic target behaviors are as important as realistic model organisms to provide transferable insights. The clinical organisms include demographic biases that LLM may perpetuate and incorrectly apply during clinical decision making [2]. Our analysis on these model organisms reinforces that the proposed validation measures are non-trivial when it comes to more realistic misaligned behaviors.

[1] Marks et al., Auditing language models for hidden objectives. In arXiv preprint arXiv:2503.10965, 2025.

[2] Zack et al., Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study.

5. Generalizability (Q5)

We think our validation framework and the correlation results are largely applicable to bigger or newer models. Our experiments span across different model families and organism suites, some of which (e.g. the olmo models) are already trained via RL and/or DPO. We believe these realism measures will be even more important for organisms trained on larger models with better capabilities, as “collateral damage” would make the resulting organisms less representative and subsequently incapable of emitting generalizable interpretability conclusions.

Thank you for the comments and clarifications. However, I'm not fully convinced yet.

The paper shows substantial evidence that "different kinds of organisms yield different interpretability outcomes". But I see no direct evidence for what you mention in every rebuttal response: "results on a representative organism are more plausibly transferable than results on a broadly damaged one". The only reasons given are "more plausibly" and "it seems likely". I think "representativeness" is such a big part of the paper that this should be resolved, otherwise the empirical results and training recommendations have little motivation.

  1. Realism: I understand that there may be no clear ground truth when training model organisms. But what concretely motivates requiring similarity to the base model’s writing style and the absence of behavior-relevant activation signatures (e.g., prior literature on deployed models or specific experimental findings)? I think having a capable MO "makes sense", but is there also an objective reason to require it? I feel like as long as the MO contains the features I want to measure accurately (i.e., how good interpretability tools can find the misbehaviour), it is a good MO (usually I don't want to measure coherent output).
  2. Tool rankings and noise: The new clinical ranking result is helpful and directly addresses my question. For Pando, you mention substantial run-to-run variability and no stable estimates of auditing result. Does this also affect the reported comparisons in the main text, and how does it affect confidence in the reported comparisons?
Reviewer_KtyHOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to train and evaluate Model Organisms for Interpretability Research more effectively. The authors point out that existing studies primarily focus on the success rate of target behavior installation, while often overlooking the impact of training on the model's general capabilities and output naturalness. This may reduce the representativeness of Model Organisms as evaluation tools.

To address this issue, the authors propose a Multi-objective Validation Framework that evaluates Model Organisms from three perspectives: Target Behavior Installation, General Capability Preservation, and Output Naturalness. The authors also investigate the effects of different training strategies and recommend using DPO, avoiding unnecessary Chat Data Mixing, and applying Model Merging to balance target behavior installation with the preservation of the model's original capabilities.

Critical strengths
  1. The paper proposes a practical validation framework for Model Organisms that also considers general capability preservation and output naturalness.

  2. The authors compare different training methods on existing Model Organism datasets and a newly constructed Clinical Bias task. They further analyze the relationship between validation metrics and downstream auditing performance. The results show that training methods affect not only target behavior installation but also the evaluation outcomes of Interpretability methods.

Critical weaknesses

The paper holds that Model Organisms closer to the Base Model can provide more reliable Interpretability Testbeds. However, the existing evidence is mainly correlation analysis, and the relationship between Validation Metrics and Auditing Performance varies among different Bias Types. The author seems not to have adequately explained whether higher Validation Scores truly imply more reliable Interpretability Conclusions.

Questions for authors

Could the authors further examine whether improvements in validation metrics lead to better auditing performance when the training recipe and target behavior installation are controlled?

1 comment

We thank the reviewer for the feedback, and are glad they gained a deeper understanding of the problem from this work.

The reviewer seems to be primarily concerned that our analyses are fundamentally correlational. This is largely true, because it is unclear how one might directly and independently intervene on the high-level factors we consider, without changing ways to train these models. Put another way, these attributes are not primitives that we can directly manipulate. Nonetheless, we think we have offered a compelling empirical case that these validation measures do co-vary with the performance of interpretability tools across a range of different setups.

We appreciate the reviewer’s suggestion for additional analysis and have run new experiments. Here we offer the new findings from controlling the target behavior installation and training recipes:

1. The correlation holds when controlling for target behavior rate. Within each bias, we split organisms into three tertiles according to the behavior installation rate, specifically the difference between r (trigger samples) and r_c (counterfactual samples). Then, we recompute the same correlation analysis inside each tertile. We find that the relationships reported in Figure 4 and Figure 11 cleanly reproduces across tertiles. For example, for race bias where overall ρ = +0.49, the same positive correlation reproduces with ρ = +0.48 / +0.38 / +0.61 after splitting. Across cells that were significant in Table 15, 14/16 reproduce the same sign for all three tertiles. The per-tertile confidence intervals are wider than before due to the splitting, where each group now holds only n=20 organisms. We also find that installation strength itself is mostly uncorrelated with CoT verbalization or J-lens relevance (|ρ| ≤ 0.26 for every bias). So the signals in Figure 4 are not a result of how strongly the behavior was installed. We report this new result in Appendix K.8.

2. The correlation is partly due to training methods, which is the intended effect. When grouped by training methods, while certain groups have too few organisms (e.g. n=3 for SFT+Chat, Table 5) to reach a significant analysis, some relationships survive: on race bias, the in-domain accuracy x J-lens relevance (Figure 4, bottom right) remains positive in three of four valid recipes (+0.29 to +0.61), and on gender bias (Figure 4, top right), negative in four of five. Others, such as in-domain accuracy × CoT verbalization on race (Figure 4, bottom left), do not survive after holding training methods fixed.

There are two possible explanations: 1. The training recipe is a true confounder, i.e. validation metrics ← recipe → interpretability results, and that the correlation signals are due to confounder effects; and 2. Recipes mainly differ in their “representativeness” of the base model, which then changes interpretability readouts, i.e. recipe → validation metrics → interpretability results, in which validation metrics truly affects interpretability results. In the second case, the correlation would also vanish within-recipe, but here validation metrics are the mediators of how a recipe changes an organism. This is exactly why we recommend selecting recipes, and checkpoints within a recipe, by their validation performance, rather than by target-behavior installation alone.

3. Correlation analyses show the non-triviality of these measures. More broadly, we do not claim that better validation metrics causally and universally lead to “better” (or “worse”) interpretability results regardless of target behavior (e.g. bias types), as different target behaviors likely have different dynamics in response to these measures. We would like to argue that in principle, a model organism should not be “fried” [1]: a testbed that has lost general capability or that produces unnatural outputs is not a good proxy for any deployed model; our framework formalizes this as measurable validation criteria, which prior work largely ignored. We then provide empirical evidence that these measures are non-trivial insofar as they correlate with what interpretability tools recover holding the target behavior fixed, so degradation matters for the downstream conclusions concerning the relative utility of interpretability methods for which these organisms are built.

Reviewer_DvHeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how model organisms should be trained and validated for interpretability research. The authors argue that successfully installing a target behavior is insufficient, and that model organisms should also preserve general capabilities and output naturalness. They propose validation metrics for target behavior installation, general capabilities (MMLU, MT-Bench, and domain accuracy), CoT naturalness, and activation naturalness.

The authors revisit Pando and Model Organism Lottery and show that training recipes affect both organism quality and downstream interpretability results. They also introduce a new suite of clinical model organisms with demographic biases and compare SFT, DPO, chat-data mixing, and model merging.

They find that DPO generally preserves capabilities better than SFT in their experiments, that mixing chat data can introduce unnatural patterns in CoT, and that merging toward the base model can recover capabilities while retaining the target behavior. They also study correlations between validation metrics and the ability of interpretability methods and auditing agents to recover the planted behaviors.

Critical strengths
  1. Validation of existing model organisms: I think the proposed framework for validating MOs beyond just target behavior installation is useful for the community. The authors show that organisms with similar target behavior rates can differ substantially in general capabilities and output naturalness depending on how they were trained. They also redo controlled experiments on 80 Pando organisms, where the same decision rules are installed using different training recipes, providing useful evidence that training methodology can affect downstream interpretability results.

  2. Experiments and practical training guidance: I liked the comparisons of SFT, DPO, chat mixing, and model merging. The finding that mixing general chat data can introduce artifacts into CoT is interesting, esp given prior work suggesting data mixing to reduce activation artifacts. The authors also show that DPO and merging can better preserve capabilities while retaining the installed behavior. They also have additional analysis of effective rule complexity as a confounder in their own Pando correlation experiments, which strengthens experimental rigor.

Critical weaknesses
  1. Unclear whether the (proposed) validation metrics capture MO realism. The paper argues that preserving general capabilities and output naturalness makes model organisms more realistic, and therefore better for evaluating interp methods. However, I am not entirely convinced that the proposed metrics establish this. For example, CoT naturalness is measured on GSM8K and activation naturalness on generic FineWeb passages. It is unclear whether models that appear natural on these distributions also have realistic internal mechanisms when exhibiting the implanted behavior which is what they need to show.

    More generally, preserving the base model's capabilities does not necessarily imply that a finetuned behavior resembles one naturally acquired through pretraining or other training processes. The authors provide evidence that their validation metrics sometimes correlate with downstream interpretability results, but these relationships are not consistent across methods or clinical biases. For example, the clinical experiments find both positive and negative relationships between capability preservation and bias recovery.

    I think the proposed metrics are useful diagnostics for identifying training-induced degradation, but stronger evidence is needed to establish that they measure the realism of the implanted mechanisms.

  2. Interpretation of the interpretability results and novelty. I like that the authors reuse Pando's held-out prediction protocol and compare organisms implementing the same planted decision rules. They find that improvements in capability preservation correlate with improved logit-lens-assisted rule recovery. However, Pando's original objective was to measure whether interpretability methods provide additional predictive information beyond black-box observations. I am not sure whether the reported improvement in logit-lens recovery translates into greater (marginal) utility over the corresponding black-box baseline.

    Table 20 provides some evidence of a method-specific relationship, since validation-score changes explain substantially more variance in logit-lens recovery than in the sample-only baseline. Nevertheless, I think directly reporting the change in each tool's advantage over black-box auditing would make the interpretation more convincing.

    The main Pando experiments use the no-exp organisms. Figure 1 suggests that DPO retraining changes how these models respond to explanation-eliciting prompts, with retrained organisms producing more fluent explanations. Although this does not necessarily mean that the explanations are faithful, it raises the question of whether retraining changes how much information about the hidden decision rules can be obtained through black-box prompting. Cant the authors measure this?

    Finally, the training methodology is somewhat incremental. Prior work, particularly Model Organism Lottery, already established that interpretability results can depend strongly on training recipes. The proposed optimization primarily combines existing DPO training with weight interpolation toward the base model. I would also like to see an SFT+Merge baseline to isolate whether the improvements are specific to DPO or can be explained by moving the model closer to its base weights.

Questions for authors
  1. On validation metrics/realism: Could the authors please explain what supports the claim that preserving MMLU, MT-Bench, CoT naturalness, and activation naturalness on unrelated distributions makes the implanted behavior more realistic? Could the authors evaluate whether these criteria also capture differences in representations or reasoning on the trigger distribution, rather than only on general prompts?

  2. On the Pando results: Since Pando evaluates interpretability methods through held-out behavioral prediction relative to black-box baselines, could the authors explicitly report the marginal improvement of logit lens and the other interpretability methods over the sample-only baseline, before and after DPO retraining? Does the improvement in accuracy translate into a larger advantage over black-box auditing?

  3. On the no-exp organisms: Figure 1 suggests that the DPO-retrained Pando organisms are more responsive to explanation-eliciting prompts. Does this also make the hidden decision rules easier to recover through black-box prompting?

  4. On the training methodology: How much of the improvement from DPO+Merge is attributable to DPO versus interpolation toward the base model? Could the authors compare against SFT+Merge?

2 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the thoughtful feedback, and provide responses to their specific points below.

1. Whether the proposed metrics capture realism (Q1)

This is a good question and we should have been more explicit about our motivation and assumptions in the paper. With respect to the first question: Our view is that good performance on general benchmarks and preserved CoT capability and naturalness are necessary though not sufficient conditions for realistic model organisms (we comment on activation naturalness below). Consider an extreme case, where a model organism has terrible performance on MMLU (or similar, suggesting poor parametric knowledge) and basically is entirely degraded as a language model. Surely this is not a good “model” of the kinds of LLMs actually used in practice. Moreover, this would unfairly handicap blackbox approaches which aim to quantify the degree to which CoTs might reveal implanted behaviors—obviously CoTs won’t work if the model cannot generate coherent CoTs to begin with. Put succinctly, deployed models are generally capable, and so any “organism” representing these should also be.

Metrics on the general distribution vs. target distribution. Ideally, we would want some way of comparing against a “ground truth” model with the behavior of interest, which we will call M*, and then establishing naturalness on the target distribution. We do not actually have access to such a model (if we did, we would not need the organism in the first place), and so can only compare the model organism to the base model M0. On the target distribution, we expect the organism (and M*, and indeed any model that shows the behavior) to differ in its activations compared to M0, but we don’t know how M*’s activations, and subsequently an ideal organism’s, should differ. Therefore, we establish naturalness on general distributions as we expect M* to not feature e.g. obvious activation signatures for the target behavior on Fineweb-like general texts.

Still, we believe that these metrics do capture aspects of naturalness on target distributions in cases where we are confident what M* would look like. Our Figure 1a uses the prefill contexts of the target decision task (i.e., “target distribution” for Pando), and we expect that M* would provide rational responses (although not necessarily faithful, see Response 3 below). The organism on the left where prefill failed to elicit any rationale is unnatural, and it has a poor MT-bench score as shown in the figure as well as CoT naturalness – normalized 0.44 vs. 1.00 for the retrained on the right. On activation naturalness, we use the MO lottery organisms in section 4 and recompute the same metric on the trigger distribution. We find a similar pattern: SDF-trained organisms leave the most activation-level artifacts, and across n=19 organisms, the ranks of activation naturalness on two distributions largely agree (within-family Spearman ρ = +0.57, p=0.027).

This is why we focus on general prompts. And here we agree with recent work by Minder et al. [1], which also points out that the task becomes trivial and unrealistic if an organism reveals (in its activations) the behavior on completely unrelated texts like Fineweb. Put another way, it seems necessarily the case that a model with a behavior of interest will induce activations that differ from those of M0’s on examples from the target distribution, but we would prefer activations between these two models to be otherwise as indistinguishable as possible.

We have clarified our thinking in the beginning of Section 2 and the Section 2.3, including making explicit our view that these criteria are necessary but not sufficient for realism.

[1] Minder et al., Narrow finetuning leaves clearly readable traces in activation differences.

AuthorsOpenReview ↗

Response 2/2

2. Marginal advantage over blackbox baselines in Pando (Q2)

We repeated experiments in Table 21 using the difference between logit-lens accuracy and sample-only (the blackbox baseline) accuracy as the predicted variable (Appendix L.6.1). We find almost the same correlation: within-depth ΔR² = 0.21 (p = 0.001), with coefficients of +0.35 on ΔMT-Bench (p = 0.005) and +0.29 on ΔMMLU (p = 0.007). This correlation shows that organisms with larger capability gains tend to have greater gains on rule recovery for logit lens over the blackbox baseline.

3. Faithfulness of blackbox prefill prompting after retraining (Q3)

Under Pando's protocol, across all 80 original/retrained pairs, the explanation-eliciting prefill tool does not improve after retraining (−0.81% change, which is within noise). Since Pando’s prefill agent only takes the first 10 tokens after “yes/no, because”, the extractor never sees the full explanations in Figure 1. We reran the prefill agent with a 200-token cap on the subset of depth-3 organisms (n=20) and find an averaged change of +0.8% with 95% CI [-1.2, +2.8].

The retrained organisms produce fluent explanations compared to elicitation failure in the original set, but they mention fields relevant to the true decision rule almost as often as irrelevant fields. That is, these explanations tend to rationalize the model’s decisions without being necessarily faithful to the true decision rule. We think this is not evidence that MO realism doesn’t matter – a realistic organism may also produce unfaithful reasoning chains, which is part of the evaluation results on blackbox tools. Our goal is to advocate for using realistic testbeds, and the left organism in Figure 1(a) is by all means not one of them, given that it loses the ability to generate any blackbox explanation.

4. SFT+Merge baseline (Q4)

We thank the reviewer for suggesting the SFT+Merge baseline and we have added it to the new Appendix I. Since the stepwise guarantee of SACPO only holds for SFT with a reverse-KL constraint (Appendix C), we merge the SFT+KL organisms in Appendix H: for each of these organisms with a successful installation, we take its merged checkpoint with the lowest ratio q (i.e., closest to the base model) that still installs the behavior. Comparing both under the same validation framework, we find 1) merging also helps for SFT+KL, with a particular gain on parametric knowledge and CoT naturalness, and 2) DPO+Merge still outperforms SFT+KL+Merge on the combined validation score. Therefore, our method based on weight interpolation works (in the sense of preserving validation metrics) regardless of the underlying training objective.

[1] Szablewski et al., The model organism lottery: Model organism interpretability strongly depends on training methodology. arXiv preprint arXiv:2607.01033, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.